Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models
Authors: Andy Zhou
Jindong Wang
Yu-Xiong Wang
Haohan Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: robustness
knowledge distillation
adversarial training
data augmentation
generalization
TL;DR: We distill out-of-distribution robustness from foundation models by generating adversarial examples from the teacher

Abstract: We propose a conceptually simple and lightweight framework for improving the robustness of vision models through the combination of knowledge distillation and data augmentation. We address the conjecture that larger models do not make for better teachers by showing strong gains in out-of-distribution robustness when distilling from pretrained foundation models. Following this finding, we propose Discrete Adversarial Distillation (DAD), which leverages a robust teacher to generate adversarial examples and a VQGAN to discretize them, creating more informative samples than standard data augmentation techniques. We provide a theoretical framework for the use of a robust teacher in the knowledge distillation with data augmentation setting and demonstrate strong gains in out-of-distribution robustness and clean accuracy across different student architectures. Notably, our method adds minor computational overhead compared to similar techniques and can be easily combined with other data augmentations for further improvements.

Distilling Out-of-Distribution Robustness from Vision-Language Foundation Models
Andy 3 Jindong Wang', Yu-Xiong Wang', and Haohan Wang!
Univeesiity Illinois at Urbana-Champaign
Microsoft Research
UIUC
(andyz3, yxv, hachana)@illinoii.. edu, jindong. wangdmicrosoft.ccm
Abstract
We propose a conceptually simple and lightweight framework for improving the robustness of vision models through the combination of knowledge distillation and data augmentation. We address the conjecture that larger models do not make for better teachers by showing strong gains in -uttof-distribution robustness when distilling from pretrained foundation models. Following this finding, we propose Discrete Adversarial Distillation (DAD), which leverages robust teacher to generate adversarial examples and a VQGAN to discretize them, creating more informative samples than standard data augmentation techniques, We provide theoretical framework for the use of a robust teacher in the knowledge distillation with data augmentation setting and demonstrate strong gains in out-of-diitriiution robustness and clean accuracy across different student architectures. Notably, our method adds minor computational overhead compared similar techniques and can be easily combined with other data augmentations for further improvements.
Introduction
One of the goals of machine learning is to develop systems that can generalize effectively across diverse populations and environments. much like human intelligence. Despite the impressive ad- vancements in neural networks that surpass human performance various tasks in computer vision, their generalization capabilities remain inadequate when faced with out-of-distribution data, such as adversarial perturbations [60], unusual colors and textures [17, 58, 68), or challenging contexts [29].
One major line of research addresses this issue with more sophisticated training strategies [37], including adversarial training [39], data augmentation [18, 28, 79], or other regularizations [35, 46, 61, 73, 69]. In this paper, focus adversarial tiapiaa---raiinnnbbee based data augmentation [75], which can enhance the quantity and diversity of training data. In addition, theoretical work suggests achieving high robustness requires significantly more samples than clean accuracy [56]. This has also been shown empirically [3, 24], most recently with transformers [14] which achieve robustness on a variety of computer vision tasks. In addition to weak inductive bias, powerful model capacity, and grounding with language, these models are often trained with large-scale datasets [12, 14, 48, 76], up to billions of images [57] that encompass many real-world distribution shifts. As result, these foundation models [1] exhibit remarkable zero-shot generalization, especially on natural distribution shifts such artistic renderings, but require large amounts of compute and heavily parameterized models.
In this paper, we aim to connect these two lines of work. We investigate it is possible to improve robustness by introducing foundation model as teacher to distill robust representations and help generate a diverse data augmentation. We conduct our analysis without requiring the teacher's 's large- scale dataset and focus nut-of-distribution robustness by introducing image-to-imas generative model to discretize optimized perturbations. We aim to leverage in-distribution data to a greater
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

extent and conduct our investigation with CLIP [48]. This marks a departure from existing work in knowledge distillation (KD). which tends to focus on smaller models and datasets. In fact, prior work [8] has called into question the utility of distilling from stronger teachers over training from scratch altogether. However, we find that although this model capacity gap can impair improvements in clean accuracy, distilling from robust teachers improves out-of-distre -uttr---ditributionn robustness, even when leveraging only in-distribution data. Surprisingly, distilling on clean ImageNet images from CLIP with the original KD objective [32] results in more robust ResNet34 than training with state-of-the-art regularization methods, despite a parameter difference 13.7x.
However, it is currently unclear in what settings teacher's 's robustness can reliably transfer to a student and how to best combine distillation with data augmentation. We aim answer this question both theoretically and empirically. We view adversarial training and data augmentation in the lens of domain generalization and prove that the diversity of the augmented samples leads to improved robustness. Our findings further suggest that foundation models make for strong teachers due to their diverse training distribution.
Building upon these findings, we introduce discrete adversarial distillation (DAD), a KD framework that further distills the robustness of a teacher model by leveraging the adversarial examples of the teacher discretized by a VQGAN [15] as data augmentation. Notably, these samples are generated in an offline fashion, adding minor computational overhead compared to standard adversarial training. Intuitively, foundation model will produce more diverse adversarial samples than a teacher trained on the same distribution, and we provide a theoretical framework using Wasserstein distance to formalize this proposition. Empirically, when distilling CLIP to ViT-B, we achieve robust accuracy of 46. 1 % on ImageNet-Sketchl [68] and 65. 1% on ImageNet- Renditinn [26]. improving on the state of the art by 17.8% and 11.3% respectively. DAD can also be freely combined with existing regularization techniques, resulting in improvements in clean accuracy. In summary, our contributions are
1. Establishing the KD for out-of-distribution robustness setting and proposing novel KD objective based on data augmentation
2. Providing theoretical framework in KD for the choice of a teacher based on the diversity of the data augmentation
3. Proposing a novel data augmentation DAD that outperforms both adversarial training and distillation techniques natural distribution shifts
2 Related Work
We define out-of-distribute accuracy/robustnes as a model's performance on non-adversarial distribution shifts, adversarial accuracy/robustness the case of robustness of adversarial examples, and clean accuracy as evaluation on dataset drawn from the same distribution.
Data augmentation. Data augmentation is frequently used as regularization [75] by expanding the quantity and diversity of training data. This often achieved through simple transformations such as rotations image crops more advanced techniques such as image mixing [79, 28], reinforcement learning [10, ] or adversarial training [21, 31, 41] to find the optimal transformation.
Adversarial training (AT) was initially introduced to enhance model robustness by training with adversarial examples [39]. Although effective for defending against adversarial attacks, several works [64, 78, 70] have indicated a trade-off between adversarial clean accuracy in AT, limiting its effectiveness as general data augmentation. Considerable efforts [47, 49] have been made to minimize this trade-off and directly use adversarial examples data augmentation [50, 74], but there still a considerable gap in out-of-distribution performance compared to foundation models like CLIP [48].
One line of work has recently been adapted this issue. The model-based robustness paradigm [4, 52] leverages the disentangled latent representations of pretrained generative models to improve or validate out-of-distribution robustness [80], and can be used to improve adversarial examples. Most similar to our work is [23, 2, 41], which use GAN or VAE [15, 34, 67] to discretize or discover adversarial examples. However, we leverage both a pretrained discretizer and foundation model, and adapt the AT objective to a knowledge distillation setting.
code at https: //tgttttttttttb http:///wwtaaaaaeeeeeeeeearrrrra

---

Knowledge distillation. Knowledge Distillation (KD) is a technique for training a student model with guidance from stronger teacher model, widely applied in vision and language tasks [6, 32, 55, 71]. Most works focus on improving the KD objective with different knowledge transfer objectives, such as feature distance [7, 54, 63], attention [77], distribution [45], activation boundary [30], and sample distance [38, 44, 66]. [8] raises the model capacity gap issue, where training suffers when the size of the student and teacher models differ, but we find that there still benefit to distilling from a robust teacher. Another line of work, defensive distillation, aims to distill adversarial robustness from an adversarially trained teacher [20, 82, 83]. We have a similar goal, but for out-of-distribution robustness and propose a loss objective not previously explored in prior works.
3 Method
We consider standard dataset ((x,)))==1 ~ PN where instances and their labels (In, yn) are drawn from a distribution P and are used for training the student model 0. We also consider discretizer, Q, and a teacher model ø and we use ((x) to denote the output of model given the sample denotes function that applies data augmentation æ, also known transformation function. Additionally, a E A, the class all such functions.
3.1 Setup
Invariance is a desirable property where the model will have the same representation of an input after a transformation is applied. model, 0 is said to be invariant if ((x) = ((x') for all E U, where U is the set of all images that can be obtained by transformation of by E A, which includes the identity transformation. a ranges from worst-case imperceivable perturbations to real-world distribution shifts like artistic sketches [68]. this paper, we focus on the latter, denoted as and Á. We can consider à to represent an individual distribution P and Å to be drawn from the distribution of distributions P. Our ultimate goal is to train to be invariant to transformations in Å. A model that maintains the same representation under transformations distribution shifts of is said to be robust, which we define as the worst-case expected risk where
max T Twesst P,.) P' PPSc
((aa)) 1(0(x),)).
where is the loss function and depends anchor distribution P, and e, the maximum deviation allowed under the Wasserstein's 's distance metric. Similarly, we can define the expected robustness and expected risk in terms of an arbitrary distribution, including the training distribution.
r r(P, e) = EprPPs(Pr P)sc E (((()))))
(2)
r(P) (( 1(0(z).))
(3)
The robustness of the resulting model is highly dependent on x', P. and the choice of data aug- mentation. It is also susceptible to adversarial attacks, where x'' is worst-case perturbation of x. Adversarial robustness can be improved with adversarial training, which couples the outer minimiza- tion objective from (3) with inner maximization objective in the following
min E<E))>P [z(0(z),)) y) + max (1))),
where x' = + is the perturbation, and / is the cross-entropy loss. This achieves adversarial robustness, but cannot generalize well to real-world domain shifts in Á. To address this, we consider generative model, Q, trained on P and can model Á. Passing an input through Q in the maximization objective applies a worst-case transformation from Å. This modifies (4) to minimize the empirical semantic adversarial risk,
min E(C.y)~ [1(((z)))) + max 1(0(Q(x)))
3.2 Distillation from a robust teacher
Next, we consider a setting where we also have access to pretrained & invariant to distribution shifts in A. This enables us to leverage knowledge distillation (KD). This modifies (3) in the following
min ((),))))(((((()) v) ((())))
(6)
3

---

CC (x+
NVSON
Distil
(rabbit)
Image
(..0))....0.00
CLIP
parameters
The overall framework of discrete adversarial distillation (DAD). We leverage foundation model generate and distill adversarial examples after discretization by VQGAN.
Figure 1:
where 4 is the classification loss, the cross-entrops loss, and l is distance loss between ((x) and ((x), the KI. divergence. (6) can be approximated by the empirical risk where
min 4 (0(x)) v) + (2((x)), o(x)).
Following theoretical work [43], distilling from robust teacher with improves generalization due to minimizing the population risk, which has lower variance. In this formulation, the output of the teacher acts distribution classes. This allows the student to learn the teacher representations on in-distribution data, but experiments show that this is inadequate for out-of-distribution robustness, when using a robust teacher. To address this, we combine (5) and (7) to also distill the representations ø on augmented samples,
robust supervisory signal than the label, encompassing the probability
min 4 (f(x),) + (2(0(x)) ((x)) + 12 (2(((((())) (())(((
(8)
The teacher is more robust, and is able "solve" the perturbation for the student through distillation. Like [41] and [72], we train our models with both the original and augmented samples, expanding the size of the dataset and maintaining the original information path for x. We do not the cross- entropy loss or y labels for x', as these labels may be wrong and could limit the expressiveness the data augmentation. This allows
adversarial samples of the teacher in novel maximization objective and obtain stronger empirical results.
3.3 Discrete Adversarial Distillation
The goal of our method, discrete adversarial distillation (DAD), is to distill from a large-scale vision- language model using only ImageNet [13] data. In the practical setting, we use approximations of an ideal discretizer and robust teacher. For we use CLIP [48], which was trained a large-scale dataset and achieves impressive zero-shot generalization on a variety of natural distribution shifts.
For Q, we use pretrained VQGAN [15], following [41], which also finds minimal improvements with stronger discretizer. The VQGAN consists an encoder, decoder, and quantization, where the encoder learns a latent representation of the input the quantization maps the representation to visual codebook entry, and the decoder reconstructs æ. We denote this process as Q(x)). In adversarial training setting, Q discretizes æ, worst-case perturbation is added by approximating the maximization objective to obtain z', and the resulting image then discretized by Q again. To improve the transfer of robustness from the teacher, hypothesize better augmentation is more informative and exposes more teacher knowledge. We make this rigorous in the following section.
To generate these examples, we adopt adversarial training and modify the maximization objective of (5) to use the worst-case transformations of the teacher. We hypothesize teacher trained on more diverse data will have more informative adversarial examples. To ensure the correctness of the perturbation, we only use samples that are still classified correctly by the teacher after generation. use the teacher as an "oracle", allowing it to distill correct representations of the transformed image. Additionally, we generate these samples in an offline manner asynchronously from the pretrained teacher and add them to the original dataset during the training of the student. These examples

---

only need to be generated once for each teacher and be reused as additional data. This adds minor additional training costs compared to online adversarial training or DAT [41], which has 11x and 3.5x the cost of standard training, respectively [41]. Our full objective is described as,
min & (0(x), (2((()), ((x)) + +22(((((2))) (Q(())))
(9)
where x'
((Qxz))) y- || Se
max
((Q()))
3.4 Theoretical Investigation
We aim to investigate how to best distill robustness from robust teacher trained large-scale dataset. We find that robust performance can be connected to the distance between the training and test distributions. A data augmentation can represent a new distribution, and the robustness of a model trained on this distribution can be quantified by its diversity and closeness to the test distributions. Although its representation on in-distribution samples due to being closer to the test distribution, it is more effective to leverage the discretized adversarial examples of the teacher than the student as data augmentation of choice.
distill a degree of robustness,
show that
We begin with some assumptions.
For arbitrary data pair (3r, y), transformations in À do alter the semantics of the data. We can also say we consider an ideal labeling function where any (x. y) pair can be correctly mapped. y f())
Assumption 1.
Assumption 2. Any arbitrary distributions P and P' we compare possess smooth probability densities controlled by two constants and depending the smoothness of P and P' where cwr( P, Pty and с > and is only dependent on a.
Assumption 3. For function 7(0),, n, parameterized by hypothesis space |0), number of samples
n, and the probability when the bound holds 3, if the samples are i.i.d, T(0), n, 3) = (R(C) +
/ (log 1/3)/2n where R()) stands for Rademacher complexity and C = (1 0 E θ}, where lo is the loss function corresponding 8. Additionally, if is finite, U(,-)) is zero-one loss, and samples are i..dd then 7(0)), n, 3) = (log((()
log( log(1/B))/2n
Lemma 3.1. Given Assumptions 1 and 2 and variational divergence tv, for arbitrary distributions P and P' with corresponding density functions 8 and 8', r(P') S r(PP) + w(P', P).
Lemma 3.2. Given Assumption 3, Lemma 3.1, and probability at least I,
r(P') S (r((,,)) + w(P', + E(T(X,Y) 0,1')
where Tx,Y), denotes the number of sample sizes in the finite dataset (X, Y) and & is vanilla term that connects ,) and with the generalization error bound.
Proof. We apply conventional generalization analysis through uniform convergence to Lemma 3.1. We leave the full proof of Lemma 3.1 in Sec. E in the Appendix.
This results in intuitive conclusion: empirical robustness performances depends the divergence between the training and testing distributions, as well as two additional elements. The first is the empirical error term on the training distribution, which can be quantified, and the second is technical term influenced by the sample size and hypothesis space. The exact manner which & depends on these parameters is contingent the particular convergence analysis employed.
Therefore, the decisive factor for robustness is the degree of deviation between the training and testing distributions. Therefore, using diverse data augmentations close to the testing distribution will lead to the largest gains in robustness. This intuitive understanding suggests that training with distributions generated from the teacher will be more advantageous, the teacher, having been trained large dataset, encapsulates more diverse distributions.
There findings are applicable to any arbitrary distributions P P P. Nevertheless, this doesn't inherently encapsulate the characteristics of foundation models that trained on data from across the internet, composed of a variety of semantic distributions from Å.
5

---

Therefore, we use P E Á to denote a set of m distributions, i.e., P P P2. P2..... Pm and we consider a pretrained foundation model trained with such a set of distributions. To facilitate forthcom- ing discussions, we extend the notation of w to encompass sets, defining wo(P P) as the average divergence between distributions within the set. Thus, wPP',P)) E. w(P', P.)/m, VP, P.P
Lemma 3.3. Given distribution P. we generate a new distribution P E Å using the discretized worst-case adversarial samples of a model 0. Training with adversarial training is equivalent to training f with empirical risk minimzation P where w(P.P") S.
We leave the full proof of Lemma 3.3 in Sec. E of the Appendix. Finally, let's denote the model trained over distribution P as (PP) and the adversarial data augmentation process that results in a new semantic data distribution as D. We aim to compare w D (P)())). ) PP) and ((((((P)) P"). In this context, P' is any arbitrary testing distribution, P is a specific training distribution, and P represents the set of distributions used for training foundation models.
Lemma 3.4. Assuming P is continuous and has finite expected value, for two sets of distributions P1 and P2, assuming there is at least one distribution in the intersection of P1 and P2, for a fixed testing distribution P', we have
Bp 10 ((((((())))))))))))))) -
S 2Epl
sup w (',,P')+ + sup w(P, w(), PEP2
PEP,
We leave the full proof of Lemma 3.4 in Sec. of the Appendix. Our findings provide a comparison of the differences in the training mechanisms for various models é, each trained with distinct data sets (P1 and P2) and subjected to adversarial training. The methodology can easily be simplified to compare the bounded results between adversarial training based on the teacher model and standalone adversarial training by setting one of the training datasets to consist of a single distribution.
In the scope of our investigation, we compare DAD and discrete adversarial training based on the teacher model, referred to as Pt. with DAT [41] and the traditional approach of adversarial training based on the student model, referred to as P2. Our results suggest two key implications:
1. Given a fixed P2. a more diverse P1 potentially results in greater variations in performance. We show visualizations that support this in Sec. in the Appendix. In other words, the use of larger, more diverse pretrained datasets for the teacher model or more diverse data augmentations for the student model increases the likelihood of creating a robust final model. This is also been shown empiricially in prior work investigating the source of robustness in foundation models [16, 62]. However, in practice, the efficacy of distilling from this teacher depends on variety of factors, including student-teacher architecture, training objective, and student capacity.
2. For a fixed P2. the greater the distance between the testing dataset P' and P2. the more likely it is that the performance gains will be realized by distilling the teacher model trained on a more extensive set of training data. To put intuitively, if the testing dataset closely resembles the training dataset (i.e., w: (P, P') is small), then adversarial training based on the teacher model might not yield significant performance improvements. However, if the testing dataset differs considerably from the training dataset, then adversaria training based on a teacher model that has been trained on a larger dataset is more likely to yield superior performance gains. This observation aligns with our empirical results.
4 Experimental Results
4.1 Experimental Setup
Models. We focus primarily ResNet50 [25] and ViT-B/16 [14]. We distill from frozen pretrained CLIP-VIT-L/1 [48], trained on 224 x 224 resolution images with a patch size of 14.
Datasets. We train our models on ImageNet-1K [13]. We use several evaluation datasets. For in-distribution performance, we evaluate on ImageNet and ImageNet-V2 [51], a replication of ImageNet's evaluation set. We focus our study on natural distribution shifts and evaluate on ImageNet- [29], a set of adversarially filtered natural images misclassified by a ResNet50, ImageNet-Skeen [68] which contains artistic sketches of objects, and ImageNet-Renditee [26] which contains abstract

---

Table Main results on natural distribution shifts and in in-distribution ImageNet. Baseline models are ViT-B/16 (top half) and ResNet50 (bottom half) trained on 224 x 224 images. The CLIP teacher is ViT-I/14. DAD variants have the best average performance for both types of distributions.
Method
Rendition
Sketch
Avg
Method
ImageNet
V2
Avg
CLIP [48)
87.7
61.6
64.2
71.2
CLIP (48)
79.9
72.9
76.4
VIT [14]
27.1
17.3
[14]
72.8
58.7
65.8
Adyprop [74]
43.5
31.7
Adyprop [74]
74.1
Fast Adyprop [42]
29.4
Fast Advprop [42]
79.0
73.0
Debiased
40.3
29.4
Debiased
79.3
67.6
73.5
AugReg-ViT [59]
39.5
29.2
29.2
AugReg-Vii VIT [59]
79.9
67.9
73,9
Pyramid AT
47.7
36.8
23.0
35.8
Pyramid AT [31]
81.7
70.3
76.0
DAT [41]
47.3
34.8
30.2
37.4
DAT[41]
81.5
70.8
76.2
DAD (Ours)
65.1
46,1
31.8
47.7
DAD (Ours)
79.6
69.9 9
74.8
DAT DAD (Ours)
53.2
39.3
28.2
40.2
DAT DAD (Ours)
81.9
71.7
76.8
ResNet50 [25]
36.1
24.0
0.0
20.0
ResNe150 [25]
76.1
63.2
69.7
Advprop [74]
38.8
25.5
4.3
22.9
Advprop [74]
77.6
65.5
35.6
Pyramid AT [31]
38.9
23.8
3.0
21.9
Pyramid AT [31]
75.5
62.5
71.6
Debiased [36]
40.8
28.4
3.5
24.2
Debiused [36]
76.9
65.0
71.0
DAT [41]
42.0
27.3
DAT [41]
76.5
70.8
DAD (Ours)
51.6
35,8
7.7
31.7
DAD (Ours)
75.7
65.0
70.4
DAT DAD (Ours)
47.7
33.3
29.0
DAT DAD (Ours)
77.8
71.9
or rendered objects. To observe performance on distributions that are out-of-distribution for the CLIP
teacher, evaluate on synthetic distribution shifts ImageNet- [27] which applies corruptions (snow, blur, noise, etc.) to ImageNet, and ImageNet [17], which processes ImageNet with style transfer from a source image.
4.2 Baselines
DAD consists of both data augmentation and knowledge distillation objective. We compare to both types of methods our experiments.
Common data augmentations. For the simplest baseline, we follow [59] and train with common data augmentations Mixup [79], which combines images and labels, and Randaugment [11], which learns policy over common transformations such brightness or shear.
DAT. We also compare against the state-of-the-arr data augmentation, DAT [41], which uses VQGAN [15] to discretize adversarial examples in adversarial training. DAT uses the standard adversarial training objective 5.
Knowledge distillation. We compare against other logit-based knowledge distillation objectives, which only distill the output logits of the teacher. We consider standard knowledge distillation 7 and DIST [33], which aims to address the model capacity gap issue by distilling logit class relationships. Neither method natively supports distillation on augmented samples, so also compare defensive distillation objectives ARD [20] and RSLAD [83]. ARD modifies to use the KI. divergence between the student logits the augmented sample with the teacher logits the normal sample. RSLAD an extension of ARD that replaces the cross-entropy terms with a KL divergence loss. For fair comparison, we use DAD the data augmentation.
4.3 Main Experimental Results on VIT-B/16 and ResNet50
ImageNet-1K. Tab. 1 shows results on ImageNet- 1K and its distribution shifts. We compare against ViT-B/16 and ResNet50 models without data augmentation and with the sfate-of-the-as data augmentation approaches, PyramidAT [31] and DAT [41]. We combine DAD with the data augmentations used AugReg [59], MixUp [79] and RandAugment [11]. find that DAD has the best average performance across datasets for both models. For ViT-B we find that DAD has competitive in-distribution performance, but greatly improves performance on natural distributions. Compared to Pyramid AT and DAT, DAD also generalizes well to ResNet50, This suggests that the

---

Table 2: Main results on synthetic distribution shifts, which is out-of-distribution for the CLIP teacher. Models are ViT-B/16 (left) and ResNet50 (right) trained on 224 x 224 images. The CLIP teacher is ViT-I/14, Compared DAT, DAD tends perform worse on ViT-B but better on ResNet50.
Method
C (4)
Stylized
CLIP [48]
60.2
18.5
Method
C (4)
Stylized
VIT [14]
74.0
6.4
ResNet50 [25]
76.7
7.4
Advprop [74]
51.5
19.2
Advprop [74]
70.5
8.0
Fast Advprop [42]
53.3
18.4
Pyramid AT [3
76.4
10.4
Debiased [36]
49.8
22.4
Debiased [36]
67.6
17.4
AugReg- ViT
Rigg [59]
54.5
16.6
DAT [41]
74.2
10.8
Pyramid AT [31]
45.0
19.1
DAD
67.4
13.1
DAT [41]
44.7
23.1
DAT + DAD
65.2
14.6
DAD
53.2
22.4
DAT DAD
47.5
22.6
Table Comparison to distillation objectives on ImageNet. We use DAD the data augmentation of ARD and RSLAD for fair comparison. All students are ViT-B/16 trained on 224 x 224 images and all teachers are ELIP-ViT-L/14 We find that our distillation objective is the best at distilling out-of-distribution robustness from CLIP.
In-distribution
Synthetic
Natural
Method
ImageNet
V2
C (L)
Stylized
Rendition
Sketch
A
Avg
KD [32]
78.6
67.2
61.5
16.2
51.5
34.7
16.0
43.2
DIST [33]
76.6
63.9
65.8
12.7
40.8
26.9
11.2
38.0
ARD [20]
80.1
70.3
52.1
22.2
55.6
38.6
27.3
48.9
RSLAD [83]
79.9
69.3
55.6
20.8
55,9
39.8
25.5
47.9
DAD (Ours)
79.6
69.9
53.2
22.4
65.1
46.1
31.8
51.7
DAD data augmentation can be used across student architectures and that due to distillation, DAD is especially effective when training smaller models.
We also demonstrate DAD can be combined with existing approach DAT for stronger in-distribution performance. We add our distillation objective to the DAT objective and train the student on both the teacher's is and its own adversarial samples. However, this comes at the cost of lower performance natural distribution shifts, although do observe that DAD DAT still outperforms the prior state-of-thc-att on ImagcNet-Sketch and ImageNet-Rendition
Synthetic distribution shifts. We also evaluate models on synthetic distribution shifts composed of generated transformations Tab. 2. Since the diverse training distribution of CLIP mostly composed of natural distribution shifts, it has weaker zero-shot generalization to synthetic distribution shifts, and this performance inherited the student model. In fact, zero-shot CLIP is already However, for
outperformed by some compared methods ImageNet-C. and ResNet50 DAD also has the best ImageNet-C performance, likely due to compared methods being specialized for certain distribution shifts [36, 74] architectures [31].
Distillation. In Tab. 3 we compare DAD to knowledge distillation objectives. KD [32] and DIST [33] vanilla distillation approaches without data augmentation AT. ARD [20] and RSLAD [83] are defensive distillation objectives that use a adversarially robust teacher to encourage invariance CLIP- CLIP-ViT-L/14 ViT-L/14 the teacher and discretize the
to perturbations. For a fair comparison, adversarial examples. We find that our distillation objective outperforms vanilla and defensive distillation objectives. We note that all methods can transfer robustness the student, even methods without data augmentation.
Net-2k. In Tab. 4 we show further gains in robustness from applying DAD to ViT-B/16 pretrained on ImageNet-21K. We fine-tune this model with our method using only our method. Despite the baseline model performing worse than the variant trained with AugReg, DAD achieves

---

Table 4: Main results from pre-training on ImageNet-21K and fine-tuning on ImageNet- All columns report top-1 accuracy except ImageNet-C which reports mean Corruption Error (mCE) where lower is better. All models are ViT-B/16 trained on 224 224 images. We find that pretraining on ImageNet-21 K results in larger robustness improvements.
In-distribution
Synthetic
Natural
Method
ImageNet
V2
C ())
Stylized
Rendition
Sketch
A
Avg
ViT
77.5
65.7
61.9
17.7
41.5
16.4
23.1
40.0
DAT [41]
83.1
73.2
43.6
24.8
55.2
41.7
36.7
53.0
DAD (Ours)
79.8
70.9
52.0
23.4
72.1
51.2
40.3
55.1
Table Results other student/teacher architectures, on ImageNet-IK. All experiments use DAD
for the knowledge distillation objective and data augmentation. Using the best CLIP model as the teacher tends to result in the highest overall performance, but some teachers better for some shifts.
Student
Teacher
IM
A
C (1)
V2
Rendition
Sketch
Stylized
Avg
ViT-B
CLIP-RN 101 [48]
81.2
24.3
49.6
69.3
48.7
34.5
17.3
46.5
VIT-B
CLIP-ViT-L [48]
79.6
31.8
53.2
69.9
65.1
46.1
22.4
51.7
RNSO
76.1
0
76.7
63.2
36.1
24.0
7.4
32.9
RNSO
VIT-B DAT [41]
80.4
10.1
65.6
68.8
40.4
29.6
8.5
38.9
RN50
DrViT-S [40]
78.5
5.5
67.4
66.2
42.0
30.1
11.5
38.1
RN50
CLIP-RNI01 [48]
76.4
5.4
70.2
64.5
47.7
32.2
9.6
37.9
RN50
CLIP-ViT-L [48]
75.7
7.7
67.4
65.0
51.6
35,8
13.1
40.2
ViT-$
77.8
11.9
63.9
66.0
36.9
25.3
12.0
38.0
VIT-S
VIT-B + DAT [41]
77.8
11.9
67.1
66.0
36.9
25.3
12.0
37.5
VIT-S
CLIP-RNI01 [48]
73.4
9.0
65.2
62.1
38.8
23.9
12.0
36.3
VIT-S
CLIP-ViT-L [48]
73.8
18.0
63.1
64.0
52.9
35.8
17.3
42.7
RN34
66.5
3.0
94.5
54.7
32.4
21.0
5.6
27.0
RN34
RN50 AugMix [28]
68.9
1.8
82.9
56.2
37.2
24.1
9.9
30.7
RN34
DrVIT-S [40]
68.2
79.5
55.6
37.0
23.0
10.5
31.0
RN34
ViT-B + DAT [41]
69.2
2.2
79.0
56.6
38.7
25.0
11.0
32.0
RN34
CLIP-RNIO [48]
65.4
2.5
85.3
53.5
42.5
26.3
8.5
30.5
RN34
CLIP-VIT-L [48]
63.6
4.5
82.0
53.7
46.0
29.1
11.7
32.4
higher relative and absolute gains robustness. We hypothesize the larger training distribution equips the student with useful inductive biases that let more easily learn the more outtrf-distribution adversarial examples generated from CLIP. We note that the CLIP training set is still ~~8. 2..2x larger this does not contradict theory, but it may also be beneficial to train pretrain the student more diverse dataset for a smoother distillation process.
4.4 Ablations
Other student and teacher architectures. Although our method and theory is adapted for foundation models, we investigate efficacy on other models and teachers in Tab. 5. We consider a different large-scale teacher, CLIP-RNI01, as well as teachers trained on ImageNet-1K that achieve out-of- distribution robustness through methods besides large-scale training, like Discrete ViT [40] or ViT-B [14] trained with DAT [41]. We also consider smaller students like ResNet34 [25] and ViT-S.
We find that distilling robustness in our setting depends on several factors, but most crucially, the robustness of the teacher. Like other distillation techniques, we find that our method can transfer representations between various student/teacher architectures. We find that our method is also susceptible to the model capacity gap, with lower clean accuracy on ResNet34 when distilling from CLIP than training from scratch. However, using CLIP results the best performance on natural distribution shifts. Despite the more similar architecture, distilling from CLIP-RNIO) across students less effective than distilling from the more robust CLIP-VIT-L.. We include similar results with vanilla knowledge distillation in Sec. A of the Appendix.

---

Table Comparisons to data augmentation approaches. All columns report top-1 accuracy except ImageNet-C which reports mean Corruption Error (mCE) where lower is better. All models are ViT-B/16 trained on 224 224 images. We remove the distillation terms and use DAD samples as standard data augmentation. All methods are based on AugReg and use Mixup and Randaugment.
Model
IM
A
C
(()
V2
Rendition
Sketch
Stylized
Avg
AugReg-Vi ViT [59]
79.9
19.0
54.5
67.9
39.5
29.2
16.6
42.5
+ Pyramid AT [31]
81.7
23.0
45.0
70.3
47.7
36.8
19.1
47.7
DAT [41]
81.5
30.2
44.7
70.8
47.3
34.8
23.1
49.0
+ DAD (Ours)
80.2
24.6
53.0
69.8
51.7
36.9
22.1
47.5
Table 7: Aikeerariil-trainnnn-baaaar based data augmentations and their training budget. Cost is based on
training a ResNet50 from scratch for 100 epochs. While still more expensive than standard training. DAD is significantly cheaper than other techniques due to reusing precomputed adversarial examples.
Method
Attack Steps
Training Budget
ImageNet
0
1x
Adversarial Training [22]
10
11x
AdvProp [74]
5
7x
Fast AdvProp [42]
I
3x
DAT [41]
1
3.5x
DAD (Ours)
1
2x
Pure data augmentation. We study in Tab. 6 the effect of training on the DAD adversarial examples purely as a data augmentation technique, without distillation. Although DAD remains competitive, we find significant drops in performance, suggesting that it is difficult for the student to learn robust representations of these images on its own. However, we continue to observe improvements on natural distribution shifts, suggesting these samples are closer to CLIP's training distribution. However, training with DAD samples is significantly cheaper than DAT and Pyramid AT, making it more efficient in practice.
Computational cost analysis. Since DAD uses adversarial examples generated from a frozen teacher, there is need regenerate them during training. This amortizes the otherwise significant cost of adversarial training. We compare the cost of DAD with other adversarial data augmentation approaches in Tab. 7. By avoiding the need continuously generate new adversarial examples, the only remaining cost for DAD is training on larger dataset, making it cheaper than similar methods.
Additional ablations choice of generative model, use of gradients, and transfer to adversarial robustness can be found in Sec. B in the Appendix.
5 Conclusion and limitations
We conduct the first study on distilling out-of out-of-distribution robustness. We develop framework for the use of foundation models in this setting and empirically and theoretically validate their advantages as teacher. We propose discrete adversarial distillation (DAD) which uses the discrete adversarial examples of the teacher as more diverse data augmentation and directly distill its most diverse representations. However, we find that DAD tends to be biased towards the performance of the CLIP teacher, exhibiting improvements mostly on natural distribution shifts. In practice, these shifts tend to be the most useful, and with the small computational cost of using DAD, we encourage practitioners to adopt it when training small models. We hope the development and release of improved foundation models and generative models will further demonstrate the effectiveness of our method.
We encourage further work to understand the limitations of machine vision models in out-of- distribution settings. More robust models carry the potential risk of automation bias, i.e., an undue trust in vision models. However, even if models are robust against corruptions distribution datasets, they might still quickly fail on the massive space of semantic transformations in real-world data. Understanding under what conditions model decisions can be deemed reliable is still an open research question.
finite out-of-
10

---

Acknowledgements
This work was supported in part by NSF Grant 2106825 and NIFA Award 2020-67021-32799.
References
[1] Rishi Bommasani, Drew Hudson, Ehsan Adeli, et al. On the opportunities and risks of foundation models, arXiv:2108.072:: 07258, 2022.
[2] Dan Andrei Calian, Florian Stimberg, Olivia Wiles, Sylvestre-Alvise Rebufti, András György. Timothy Mann, and Sven Gowal. Defending against image corruptions through adversarial augmentations. In ICLR. 2022. 2
[3] Yair Carmon, Aditi Raghunathan, Ludwig Schmidt, John C. Duchi, and Percy $ Liang. Unlabeled data improves adversarial robustness. In NewrIPS, 2019.
[4] Junbum Cha, Kyungjae Lee, Sungrae Park, and Sanghyuk Chun. Domain generalization by mutual- information regularization with pre-trained models. In ECCV, 2022.
[5] Minwoo Chac and Stephen G. Walker. Wasserstein upper bounds of the total variation for smooth densities. Statistics Probubility Letters, 163: 2020. 18
[6] Guobin Chen, Wongun Choi, Xiang Yu, Tony Han, and Manmohan Chandraker. Learning efficient object detection models with knowledge distillation. In NeurIPS, 2017.
[7] Liqun Chen, Zhe Gan, Dong Wang, Jingjing Liu, Ricardo Henao, and Lawrence Carin. Wasserstein contrastive representation distillation. CVPR, 2020. 3
[8] Jang Hyun Cho and Bharath Hariharan. On the efficacy of knowledge distillation. In JCCV, 2019. 2, 3
[9] Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. ICMI,, 2020. 17
[10] Ekin D. Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quac V. Le. Autoaugment: Learning augmentation strategies from data. In CVPR, June 2019. 2
[11] Ekin Dogus Cubuk, Barret Zoph, Jon Shiens, and Quoc Lc. Randaugment: Practical automated data augmentation with reduced search space. In Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, NeurlPS, 2019. 7
[12] Mostafa Dehghani, Josip Djolonga, Basil Mustafa, al. Scaling vision transformers 22 billion parameters. ICML, 2023.
[13] Jin Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image datahase. In CVPR, 2009. 4,
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner. Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words:
Transformers for image recognition scale, arXiv:2010.1192,, 2020. 1, 6. 7, 8, 15
[15] Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In CVPR, 2021. 2, 4, 7, 15
[16] Alex Fang, Gabriel Ilharco, Mitchell Wortsman, Yuhao Wan, Vaishaal Shankar, Achal Dave, and Ludwig Schmidt. Data determines distributional robustness in contrastive language image pre- traininng (clip). In ICML, 2022.
[17] Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A Wichmann, and Wieland Brendel. Imagenet-trained cnns biased towards texture; increasing shape bias improves accuracy and robustness. In ICLR, 2019. 1,7
[18] Golnaz Ghiasi, Hugo Touvron, Tom Deiries Tsung- Yi Lin, Yann LcCun, and Roozbeh Mottaghi. Augmax: Adversarial composition mixtures for robust data augmentation. arXiv:2106. 005822 2021.
[19] Evarist Giné and Richard Nickl. Mathematical foundations of infimite-dimensional statisticul models. Cambridge university press, 2021. 18
[20] Micah Goldblum, Liam Fowl, Soheil Feizi, and Tom Goldstein. Adversarially robust distillation. AAAI. 2020. 7.8

---

[21] Chengyue Gong. Tongzheng Ren, Mao Ye, and Qiang Liu. Maxup: A simple way to improve generalization of neural network training. In CVPR, 2021. 2
[22] lan J Goodfellow, Jonathon Shiens, and Christian Szegedy. Explaining and hameessing adversarial examples. arXiv:1412.6572, 2014. 10,
[23] Sven Gowal, Chuan Qin, Po-Sen Huang, Taylan Cemgil, Krishnamurthy Dvijotham, Tim Mann, and Pushmeet Kohli. Achieving robustness in the wild via adversarial mixing with disentangled representations. In CVPR, 2020. 2
[24] Sven Gowal, Sylvestre-Alvise Rehuffi, Olivia Wiles, Florian Stimherg, Dan Andrei Calian, and Timothy A Mann. Improving rohustness using generated data. NeurIPS, 2021.
[25] Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016. 6, 7, 8.9
[26] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-o--distribution generalization. In ICCV, 2021. 2, 6
[27] Dan Hendryeks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In ICLR, 2019. 7
[28] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. ICLR, 2019. 1, 2,
[29] Dan Hendrycks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In CVPR, 2021. 6
[30] Byeongho Heo, Minsik Lee, Sangdoo Yun, and Jin Young Choi. Knowledge transfer via distillation of activation boundaries formed by hidden neurons. AAAI, 2019.
[31] Charles Herrmann, Kyle Sargent, Lu Jiang, Ramin Zabih, Huiwen Chang, Ce Liu, Dilip Krishnan, and Deqing Sun. Pyramid adversarial training improves vit performance. In CVPR, 2022. 2, 7, 8, 10
[32] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. arXiv:1503.02531 2015. 2, 8, 15
Distilling the knowledge in a neural network.
[33] Tao Huang, Shan You, Fei Wang, Chen Qian, and Chang Xu. Knowledge distillation from a stronger teacher. In NeurlPS, 2022. 7,8
[34] Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019. 2
[35] Woo Jae Kim, Yoonki Cho. Junsik Jung, and Sung-Eui Yoon. Feature separation and recalibration for adversarial robustness. CVPR, 2023.
[36] Yingwei Li. Qihang Yu, Mingxing Tan, Jieru Mei, Peng Tang. Wei Shen, Alan Yuille, et al. Shape-texture debiased neural network training. In ICLR, 2020. 7.8
[37] Haoyang Liu, Maheep Chaudhary, and Haohan Wang. Towards trustworthy and aligned machine learning: A data-centric survey with causality perspectives. 2307 16851, 2023. 1
[38] Yufan Liu, Jiajiong Cao, Bing Li, Chunfeng Yuan, Weiming Hu, Yangxi Li, and Yungiang Duan. Knowledge distillation via instance relationship graph. In CVPR, 2019. 3
[39] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In ICLR, 2018. 1,2, 17
[40] Chengzhi Mao, Lu Jiang, Mostafa Dehghani, Carl Vondrick, Rahul Sukthankar, and Irfan Essa. Discrete representations strengthen vision transformer robustness. In ICLR, 2022. 9
[41] Xiaofeng Mao, Yuefeng Chen, Ranjie Duan, Yao Zhu, Gege Qi. Shaokai Ye, Xiaodan Li, Rong Zhang, and Hui Xue. Enhance the visual representation via discrete adversarial training. In NeurlPS, 2022. 2, 4, 5, 6, 7, 8, 9, 10, 17
[42] Jieru Mei, Yucheng Han, Yutong Bai, Yixiao Zhang, Yingwei Li, Xianhang Li, Alan Yuille, and Cihang Xie. Fast advprop. In ICLR, 2022. 7, 10
12

---

[43] Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, Seungyeon Kim, and Sanjiv Kumar. A statistical perspective on distillation. In JCML, 2021. 4
[44] Wonpyo Park, Dongju Kim, Yan Lu, and Minsu Cho. Relational knowledge distillation. In CVPR, 2019. 3
[45] Nikolaos Passalis and Anastasios Tefas. Learning deep representations with probabilistic knowledge transfer. In ECCV, 2018.
[46] C. Qin, Martens, Gowal, D. Krishnan, Dvijotham, A. Fawzi, De, R. Stanforth, and Kohli. Adversarial robustness through local lincarization. In NeurlPS, 2019. 1
[47] Rahul Rade and Scyed-Mohsn Mohsen Moosavi- Reducing excessive margin to achieve a better accuracy vs. robustness trade-off. In ICLR, 2021. 2
[48] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, 2021. 1, 2, 4, 6, 7, 8, 9, 15
[49] Aditi Raghunathan. Sang Michael Xie, Fanny Yang. John C. Duchi, and Percy Liang. Understanding and mitigating the tradeoff between robustness and accuracy. ICML, 2020.
[50] Sylvestre-Alvis Rebufti, Francesco Croce, and Sven Gowal. Revisiting adapters with adversarial training. In ICLR, 2023. 2
[51] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenct classifiers generalize to imagenet? In /CML, 2019.
[52] Alexander Robey, George J. Pappas, and Hamed Hassani. Model-based domain generalization. In NeurlPS, 2021. 2
[53] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolttion image synthesis with latent diffuusion models. In CVPR, 2022. 15
[54] Adriana Romero, Nicolas Ballas, Samira Ebrahimi Kahou, Antoine Chassang, Carlo Gatta, and Yoshua Bengio. Fitnets: Hints for thin deep nets. In ICLR, 2015.
[55] Victor Sanh, Lysandre Debut, Julien Chaumond. and Thomas Wolf. Distilbert, distilled version bert: smaller, faster, cheaper and lighter. arXiv:1910.01108 108, 2020. 3
[56] L. Schmidt, S. Santurkar, D. Tsipras, K. Talwar, and A. Madry. Adversarially robust generalization requires more data. In NeurlPS, 2018.
[57] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kun- durthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. Laion-5b: An open large-scale dataset for training next generation image-text models. In NewrlPS, 2022.
[58] Ali Shahin Shamsabadi, Ricardo Sanchez-Martee Matiila,, and Andrea Cavallaro. Colorfool: Semantic adversarial colorization, In CVPR, 2020.
[59] Andreas Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer. How to train your vit? data, augmentation, and regularization in vision transformers. arXiv:2106. 10270, 2022. 8, 10, 15, 16
[60] Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, D. Erhan, Ian J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In ICLR, 2014.
[61] Jihoon Tack, Sihyun Yu, Jongheon Jeong, Minseon Kim, Sung Hwang, and Jinwoo Shin. Consistency regularization for adversarial robustness. In AAAI, 2022.
[62] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and I Ludwig Schmidt. Measuring robustness to natural distribution shifts in image classification. arXiv:2007.0064 2020. 6
[63] Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive representation distillation. In ICLR, 2020. 3,
16
[64] Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robust- may be at odds with accuracy. In JCML, 2019. 2
13

---

[65] Zhuozhuo Tú, Jingwei Zhang, and Dacheng Tao. Theoretical analysis of adversarial learning: A minimax approach. In NeurlPS, 2019. 19
[66] Frederick Tung and Greg Mori. sumilarity-preserviie knowledge distillation. In CVPR, 2019. 3
[67] Aaron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu. Neural discrete representation learning. In NeurIPS, 2017.
[68] Haohan Wang, Songwei Ge, 7achary Lipton, and Eric P Xing. Learning robust glohal representations by penalizing local predictive power. In NeurlPS, 2019. 2, 6
[69] Haohan Wang, Zeyi Huang, Xindi Wu, and Eric Xing. Toward learning robust and invariant representations with alignment regularization and data augmentation. In Proceedings of the 28th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2022.
[70] Haohan Wang, Xindi Wu, Zeyi Huang, and Eric P Xing. High-frequency component helps explain the generalization of convolutional neural networks. In CVPR, 2020. 2
[71] Huan Wang, Yijun Li, Yuehai Wang. Haoji Hu, and Ming-Hsuan Yang. Collaborative distillation for ultra-resolution universal style transfer. In CVPR, 2020.
[72] Huan Wang, Suhas Lohit, Mike Jones, and Yun Fu. What makes "good" data augmentation in knowledge distillation statistical perspective. NewrlPS, 2020.
[73] D. Wu, S.-T. Xia, and Y. Wang. Adversarial weight perturbation helps robust generalization. In Neur/PS, 2020.
[74] Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V. Le. Adversarial examples improve image recognition. In CVPR, 2020. 2, 7,8,, 8, 10
[75] Suorong Yang, Weikang Xiao, Mengeheng Zhang, Suhan Guo, Jian Zhao, and Furao Shen. Image data augmentation for deep learning: A survey, MxXiv:22040086/1 2022. 1,2
[76] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv:2205.01917, 2022.
[77] Sergey Zagoruyko and Nikos Komodakis. Paying attention attention: Improving the performance of convolutional neural networks attention transfer. /CLR, 2017.
[78] Hongyang Zhang, Yaodong Yu. Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoreti- cally principled trade-off between robustness and accuracy. In ICML, 2019,
[79] Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. Paz. Mixup: Beyond empirical risk minimization. ICLR, 2018. 1, 2.7
[80] Peiyan Zhang, Haoyang Liu, Chaozbuo Li, Xing Xie, Sunghun Kim, and Haohan Wang. Foundation orieeeod robustness: Robust image model evaluation with pretrained models, arXiv:2308.10633 10632, 2023. 2
[81] Xinyu Zhang, Qiang Wang. Jian Zhang, and Zhao Zhong. Adversarial autoaugment. In ICLR, 2019.
[82] Shiji Zhao, Jie Yu, Zhenlong Sun, Bo Zhang, and Xingxing Wei. Enhanced accuracy and robustness via multi -eather adversarial distillation. In ECCV, 2022. 3
[83] Bojia Zi, Shihao Zhao, Xingjun Ma. and Yu-Gang Jiang. Revisiting adversarial robustness distillation: Robust soft labels make student better. ICCV, 2021. 3, 7.8
14

---

Appendix
The appendix is organized as follows. First, in Sec. A, show additional results using the original knowledge distillation objective. In Sec. we show the results of additional ablations on the generative model, use of gradients, and transfer adversarial robustness, In Sec. C we provide additional hyperparameler and implementation details. In Sec. D we show visualizations of DAD and the diversity of its data augmentation support our theory. In Sec. E provide full proofs from our theory. Finally, in Sec. F we provide visualizations of generated DAD samples compared to standard and DAT samples.
A Additional results vanilla knowledge distillation
Table 8: Knowledge distillation can improve robustness. The teacher is CLIP-ViT-L/14 @ 224px [48] We use the original knowledge distillation objective [32]. ViT-B [14] and VīT-S [14] are trained with AugReg [59]. Top half of the table is the original performance. Bottom half is the distilled performance. We find that distilling from CLIP can transfer robustness, even on in-distribution data.
Model
ImageNet
A
C (
V2
Rendition
Sketch
Stylized
CLIP [48]
79.9
64.2
60.2
72.9
87.7
61.6
18.5
ViT-B [59]
79.9
19.0
54.5
67.9
39.5
29.2
16.6
ViT-S
77.8
11.9
63.9
66.0
36.9
25.3
12.0
ResNet50
76.1
0.0
76.7
63.2
36.1
24.1
7.4
ResNet34
66.5
3.0
94.5
54.7
32.4
21.0
5.6
ViT-B
78.6
16.0
61.5
67.2
51.5
34.7
16.2
ViT-S
79.3
18.1
59.1
68.8
45.9
30.6
14.3
ResNet50
77.8
7.4
69.0
67.6
47.0
32.3
8.7
ResNet34
74.5
3.6
77.1
63.2
41.2
28.5
9.3
Average Change
+2.1
+2.8
-5.7
+3.75
+10.2
+6.63
+1.73
Table 9: Results on other student/teacher architectures with the original KD objective, on ImageNet- IK. Robustness can be distilled from variety of robust teachers.
Model
Teacher
IM
A
C (L)
V2
Rendition
Sketch
Stylized
RN50 RN50
VIT-B + DAT
80.0
8.1
66.3
68.5
40.9
29.4
8.6
DIVITS
79.3
8.2
67.4
68.4
41.7
30.0
8.9
RN34
RN50 + AugMix
72.6
1.9
80.2
61.5
37.9
25.6
8.6
RN34
DiVIT-S
74.2
2.5
77.4
62.1
38.2
25.4
8.7
RN34
VIT-B + DAT
74.3
2.5
77.1
62.3
37.8
8.4
RN34
CLIP-RNIOI
72.4
3.9
79.8
61.2
45.4
29.8
8.5
In Tab. 8 we find that surprisingly, distilling from CLIP on only in-distribution data able to transfer robust representations, but is generally outperformed by DAD. This works especially well smaller models, like ResNet34. In fact, can also improve clean accuracy compared to training from scratch, for all the models test except ViT-B. In Tab. 9, find that distilling from CL.PP generally results in the highest average robust performance, especially for natural distribution shifts. However, any robust teacher distill robustness in this setting, including ResNet50 trained with AugMix the only robustness intervention. This matches our results for Tab. 5.
B Additional ablations
B.1 Choice of generative model
We use VQGAN [15] as our generative model of choice due to nature as image-to-image model, making it suitable as discretizer. To justify this, we also experiment with Stable Diffusion [53],
15

---

Table 10: We ablate the use of VQGAN by using Stable Diffusion with DAD. The results are significantly worse, indicating the need to use a image-to-image model to discretize adversarial examples.
ImageNet
V2
Rendition
Sketch
Avg
Stable-Diffusion VQGAN
79.1
67.8
45.9
33.4
22.0
49.6
79.6
69.9
65.1
46.1
31,8
69.5
Table 11: VQGAN sampling baseline comparison. To show the importance of using model gradients to discover a diverse data augmentation, we sample from the VQGAN without optimizing for perturbation. The results are significantly worse than DAD.
ImageNet
V2
Sketch
Avg
VQGAN Sample VQGAN Grad
80.9
70.1
49.3
34.9
24.0
51.8
79.6
69.9
65.1
46.1
31.8
69.5
text-to-image generative model. We the generic prompt "A photo of object". We observe in Tab. significant decrease in performance when trained using DAD compared to VQGAN. This suggests the importance of using discretizer for DAD. Perhaps modifying the text prompt for could boost performance and be an interesting avenue for future work, especially since CLIP also requires text prompt.
B.2
Use of gradients
DAD is based on adversarial training and uses gradients to find the most diverse and useful data augmentations. show importance of using teacher gradients to generate adversarial examples, we implement a sampling-based baseline where we discretize images without the added perturbation. The results in Tab. are significantly worse than DAD, indicating the need to use gradients to discover diverse samples. This is also supported by our theoretical analysis that indicates more diverse adversarial examples are better for robustness. Higher in-distribution performance also suggests the samples are less diverse.
B.3 Transfer to adversarial robustness
Although we center our study on out-of-distie of-iisiribttii robustness, it is natural to examine the effect of DAD adversarial robustness due to the use of adversarial training. Tab. 12 attack our trained models with adversarial attacks of various difficulty. We observe a small improvement in adversarial robustness for simpler attacks, but neither DAT or DAD is robust to AutoAtttac. This is because the perturbation is discretized and no longer represents the original adversarial example. observe that DAT is stronger than DAD for ViT-B, Unlike out-of-distribution robustness, since adversarial robustness is based on perturbations generated with gradients from the base model, DAT models are trained on images closer to these perturbations than DAD models (which were trained on perturbations generated with CLIP gradients). However, for ResNet50, DAD is better even for adversarial robustness as distillation is able to help smaller capacity models learn discrete adversarial examples. We also observe higher ResNet50 performance in general in Tab. 1.
C Implementation details
adopt official hyperparameter settings for fair comparison for our baselines. For knowledge distillation, we use a temperature of / for all models and a = 0.5, following [63]. For DAD, we also weight the second -ivdeieeeeeece term by a. All ViT models are trained with the AugReg [59] hyperparameter and data augmentation configurations.
16

---

Table 12: Comparison of adversarial attack methods. We use pretrained models and attack with FGSM [22], PGD [39], and AutoAttack [9]. Models trained on discretized adversarial examples are somewhat robust but fail on stronger attacks.
Method
FGSM
PGD
AutoAttack
ResNet50
23.5
1.0
0.0
ResNet50 DAT
33.0
5.9
0.0
ResNet50 DAD
43.5
12.6
0.0
VIT-B
49.4
24.7
0.0
VÎT-B DAT
64.9
26.2
0.0
VIT-B DAD
47.2
25.0
0.0
Following [41]. we use the pretrained VQGAN weights from the official GitHub 2. The VQGAN with f= 8, d = 4 and == 16384 is used for main experiments.
We one iteration for the adversarial attack, and an attack learning rate of 0.1.
For the DAT + DAD variant, we add an additional cross entropy loss term with the student adversarial example to the overall training objective, and weight by
We conduct all of our experiments on 32GB NVIDIA V100 GPUs.
D Wasserstein distance comparisons
To justify our theoretical framework and empirical results we investigate the relationship between Wasserstein distance and model performance. We use pretrained ResNet50 and calculate Wasserstein distance from batch norm statistics on different distributions using 1000 mini-batchi distributions are data augmentations generated with the respective methods. 2a we find that DAD tends to have better performance the larger the distribution shift. This is likely due to the help of distillation letting the model learn robust representations. Additionally, in 2b, we find that relative to the Wasserstein distance between clean ImageNet images and distribution shift and baseline models, our method has higher performance.
These
Coepperioon Degree of Dissitation Parfermanne
Inspphot DAT -4p D4O m]
Comperison of Degree of Datribution Roldtiee Performance
-d- DVT
-Bh D/0 (Am))
Pesion Aufureni
0.2
6.5
(a) The relationship between Wasserstein distance and performance. DAD is the only method that im- proves performance when the generated data aug- mentation is more diverse.
(b) The relative relationship between Wasserstein dis- tance and performance to the baseline model. DAD has stronger performance compared to DAT over the standard model.
E Proofs
Proof of Lemma 3.1.
Butp::/ggituaaccommo/oom/itttinn Sranaa transformers
17

---

Proof. Variational divergence to measures the divergence between distributions, where B is the set of measurable subsets in P and p
tu(P, P') - sup
BEB Pr[B] - Pr[BI
r(P" = - + -(P) r(P)
r + +|r(P) r(P)
r + 16(x) (x)) dr
< < + to(P,
Following [5] and from Assumption 2, we can bound total variation with Wasserstein's distance. We take K R - R, kernel satisfying suitable moment condition, for any coupling of P and Pt,
[T, - TI(KIl W.,PPP).
|| kk x K xpple S sup aft 88
by the Jensen's 's inequality. Assume that p: p'p' E (RR... Let {Øm} be the orthonormal system in I.2 ([-1,,1) of Legendre polynomials defined by
2m +
dm
Φ0(x) = 2-1/2 $ 1),
6m(x) =
m! dr [(x2 - 1[xx s 1),
2
for x E IR and m E N. Define
K(x) = Σ
om (x)
m==
Then, by Propositions 4.1.5 and 4.1.6 of [19].
Kh p pll & || h2 and |Kh + -' < "i
Since max (l [x)] V 16 (z)]) is bounded by a constant depending only on m. where &'m is the derivative of op and V is the maximum of and b,
[[Ts(mm) Te(dm)ll1= = /
10m( - 8) Φmlx - (d
10m(x s) 6m (x t)ldr 2/s -| max
I@m (x)(
=E[-1.1]
{10 e/V x-1/513
< 4|s t max 10 (z) + 2|s - max
16m (x)1
xEl xE(-1.1)
xE(-1.1]
< 1s tl.
Thus,
[T,(KN) Ti(K))li 1 T. Σ dm (0) / 4m
I
-
|s
om
dx
h
where @m (0) and depends only on a. By the triangle inequality, have
lp p'lla I |p
Kh * pll1 + |Kh x Kh * p'lll + [Kh 1''h'''
x ||p|| F + (P P')
+ 1
h
If we take
(P,
1/(a+1)
[pll + 17
the proof is complete.
18

---

Proof of Lemma 3.3.
Proof. Let (Z, dz) be metric space where Z X and dz defined as: dz(2, z0) dz v), (To, (o)) (dx (2, va) + dr (v. v0)) where dx and dy represent the metric in the feature space and label space, respectively. Then we can define the Wasserstein distance between P and p*
(P, =
inf M)
((,,00)- M (dz 20)].
where [(P, P*) denotes the collection of all measures on Z x Z with marginals P and P on the first and second factors, respectively.
Following [65], we use the minimax approach, considering the worse-case P in the Wasserstein ball B. of radius centered around P where
BTt = {PP' uupP,PP)) 5
Next define transport map T: Z 7 push P to P as follows:
= (z, y) + (x'.y)
where arg E 1(0(z)), 3). By the definition of dz, dz u), (2*,")) dx (z, x²). Finally, let P = To# P, the pushforward of P by To. then we have R(P,c) R(P"). By the definition, we have
R(P,E) ( P [max ((0(zo).
P [[(((=)),)) v)
(( [(0(z)), v)l
Therefore, r(P,c) =r())) This lets us establish upper bound on the worst-case in the Wasserstein ball and bound the adversarial expected risk. Next define the radius of the adversary constrained by B as EB == supzen dx (z. 0). For any hypothesis h and the corresponding P = To P. we have w(P. P*) S En. By the definition of the Wasserstein distance, we have
((P. P") Erldz (Z
=xplx(xxx=))
(EB),
where the last inequality uses the translation invariant property of dx. Therefore, we have w(P, P*) EB.
Proof of Lemma 3.4
Proof. We use to denote the (at least) one distribution in the intersection of P, and P2-
Ep [p[w(D(((P)), -((D())))))))))(((()))))))))) P')]
I [u[[[(P Pı) (PP,, P')) - [. [w([] P2) w(P, P'))
= Ep (Po Pı) ) w(P. P2)] + Ep P1, P') + ww( (P P')
= EE sup Tworst (P. E) Tworst (P, PEP
(P,)) e)] + Bp (P, P') + w (P P')
< sup F Twersi (P,c) Tworra P.. E, sup w P. P') + w (P, P') PEP
PeP:
2E, sup w P') +
PEP.
where is positive constant.
Similarly, we can have
(p[w(D()(P2)) P³) -w((ww())))) P')] c
2-2R 2Rp sup w(P. P').
PeP:
19

---

Visualizations
(a) Clean
(b) Pixel AT
(c) DAT
(d) DAD
Figure 3: Additional visualizations of generated images. To highlight the difference, we use adver- sarial examples that classified differently by the base model. Using CLIP in DAD results in more diverse adversarial example than vanilla ResNet50. Adversarial examples in pixel-space are imperceptible.
20


Reviews:
[{"review_id": "6D3CR7i6kW", "reviewer": "Reviewer_GKv1", "summary": "This paper presents a knowledge distillation framework for vision model, by leveraging a teacher model (CLIP). In the setup, a discretizer (VQGAN) model is introduced to the adversarial examples from the teacher model. Adversarial training (AT) objective is adapted in the knowledge distill setting. The proposed discrete adversarial distillation (DAD) demonstrate improved performance in out-of-distribution tasks, comparing to other AT approaches. ", "questions": "1. How come DAD+DAT performs worse than DAT in some cases?\n2. Are there any qualitative results? what are the drawbacks from previous model(s) that are better addressed by the newly proposed DAD?\n3. Have you try training the teacher model in parallel with the student model, not just provide adversarial samples from offline manner?", "limitations": "Please see weakness.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper present an interesting approach for knowledge distillation for large scale vision model. The experiment is thoroughly conducted with various of SOTA approaches compared, and comprehensive ablation study. The writing is easy to follow and the idea is well established.", "weaknesses": "1. One main concern is on comparing the proposed method with DAT. In DAT, the idea of discretizer is introduced and VQGAN was also considered in their framework. In their paper, various tasks were evaluated, including image classification, self-supervised learning, and object detection. To me, DAD seems to be applying the similar methodology, but on a OOD setup. \n\n2. In the experiment, it seems like DAT has achieved competitive results in ImageNet-1K, and even better results in ImageNet-21K. What is the key advantage of DAD over DAT? ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments and positive review. We are glad the reviewer found our idea well established, experiments thorough and comprehensive, and writing easy to follow. We address the concerns about the comparisons and results below.\n\\\n\\\n*One main concern is on comparing the proposed method with DAT...To me, DAD seems to be applying the similar methodology, but on a OOD setup.*\n\nAlthough DAD draws upon the use of a VQGAN for data augmentation, the methodology is distinct in three key ways. \n1. *Setting.* DAT follows a standard data augmentation setting to improve robustness. We are the first to also introduce a robust teacher and use knowledge distillation to further improve robustness. We focus on OOD-robustness due to practical usefulness and the specialization of large-scale models on natural distribution shifts. These models exhibit strong robustness due to their diverse training data, and we are the first to leverage these representations as an additional form of regularization\n2. *Objective.* We introduce a novel knowledge distillation objective for the OOD-robustness setting where we add a second KL-divergence term between the student and teacher predictions on the augmented image. Previous works in defensive distillation use the teacher predictions on the normal image for this second term, but this is not adaptive to semantic transformations.\n3. *Data augmentation.* This is the aspect of DAD that is adapted from DAT. However, our key novelty here is using the teacher’s gradients to generate adversarial examples. We find that this results in more diverse adversarial examples closer to the teacher’s representations, and our theoretical framework is based on this idea. This also makes DAD much cheaper than DAT since we only generate the adversarial examples once.\n\n\\\n*In the experiment, it seems like DAT has achieved competitive results in ImageNet-1K, and even better results in ImageNet-21K. What is the key advantage of DAD over DAT?*\n\nAlthough DAT performs well, we observe distinct improvements in DAD across distributions and especially on natural distribution shifts. \n\n- On average, there is an improvement of 2.7% for ViT-B and 4.2% for ResNet50 on ImageNet-1K across 7 distribution shifts. \n- Also following Reviewer Y3Hr’s suggestion, we updated Table 2 with the average performance for clarity. *DAD has higher results than DAT on ImageNet-21K as well.* We attached the updated table in the uploaded PDF.\n- DAD has a significant improvement of (avg +10.3% for ViT-B and avg +7.1% for ResNet50) on natural distribution shifts  (IM-A, IM-R, IM-Sketch). The key advantage of DAD over DAT is that it is able to leverage the robustness of a pretrained foundation model. This is also the likely cause for worse performance on synthetic distribution shifts. In fact, the CLIP teacher performs worse than DAT training on ImageNet-C (-15.5%) and ImageNet-Stylized (-4.6%) due to being trained only on natural images, making it difficult to surpass DAT on these distributions with a distillation-based approach. \n\n\\\n*How come DAD+DAT performs worse than DAT in some cases?*\n\nWe found that combining DAD with DAT resulted in higher in-distribution accuracy and accuracy on synthetic distribution shifts, but at the cost of lower performance on natural distribution shifts than DAD. In practice, DAD targets natural distribution shifts while DAT performs better on synthetic corruptions. Due to CLIP’s diverse training set, our theoretical framework in Sec 3.4 shows CLIP’s adversarial examples encode similar representations. Combining them may dilute the specialization that leads to their strong individual performance. We do note that DAD+DAT still outperforms DAT on natural distribution shifts.\n\n\\\n*Are there any qualitative results? what are the drawbacks from previous model(s) that are better addressed by the newly proposed DAD?*\n\nYes we included qualitative results:\n1. We add visualizations of the generated images for DAD in the uploaded PDF. Both DAT and DAD adversarial examples are computed with the same budget. We select sets where the DAT image is classified correctly but the DAD image is classified incorrectly to highlight the difference. DAD images appear to be more graphic, displaying a greater variety of distribution shifts than the corresponding DAT image.\n2. We kindly point the reviewer to the Appendix where we provide graphs in Figures A-C on page 15 describing the connection to our theoretical framework. Here we demonstrate the relationship between Wasserstein distance of the evaluation distribution and performance. In figure C, we show that for a test distribution, DAD has a larger improvement over standard ImageNet training than DAT the larger the Wasserstein distance between the test distribution and training distribution is. This suggests DAD can better address harder distribution shifts like natural distribution shifts (IM-A, IM-R, IM-Sketch), which are also closer real-world use cases than simple synthetic modifications to the base image (IM-C, IM-Style). \n3. DAD is also cheaper than DAT and AT. We find that DAT is only 2x as expensive as standard training, compared to 3.5x for DAT and 11x for AT.\n\n\\\n*Have you try training the teacher model in parallel with the student model, not just provide adversarial samples from offline manner?*\n\nThis is an interesting suggestion and something we started to try before deciding not to pursue further. Joint training certainly has the potential to discover harder and more diverse examples, but at a higher computational cost of updating two models. In addition, it is still unclear what the best way to apply adversarial training to CLIP is, especially on only ImageNet data. We found that training the teacher led to instability, but this is definitely an idea that could lead to further improvements.\n\n\\\nWe hope our response addressed the reviewer’s questions and concerns. We are happy to answer any further questions.\n\n\n\n"}, {"review_id": "NxqGEPCNaX", "reviewer": "Reviewer_gGrQ", "summary": "This paper introduces a novel approach called discrete adversarial distillation (DAD) to train small, robust models using large-scale models as teachers. The authors establish the knowledge distillation framework for out-of-distribution robustness and provide a theoretical framework for utilizing large-scale models as teachers. DAD achieves state-of-the-art performance on natural distribution shifts, surpassing adversarial training and traditional distillation techniques.", "questions": "Please refer to the weakness. In addition, why the authors did not provide the code for review but answered the *Reproducibility* as \"yes\"? Please provide the code during the rebuttal period, using Anonymous GitHub is a viable option. Also, the authors claim that the proposed only brings negligible computational overhead, but the reviewer could not find the part that addresses computing resources; what exactly are the computational costs for the proposed framework?", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "Overall, this paper is also well-written and clearly structured; the authors provide both theoretical analysis and empirical experiments to verify their approach. The reviewer would like to list some strengths of the work.\n\nFirstly, it extends the knowledge distillation framework to the domain of out-of-distribution robustness, opening up new possibilities for training small, robust models that can generalize effectively across diverse populations and environments.\n\nSecondly, this paper highlights the advantages of leveraging large-scale models as teachers, providing a theoretical framework to support this approach. This understanding can inform the development and utilization of robust models in practical applications.\n\nLastly, the experimental results showcase state-of-the-art performance on natural distribution shifts, surpassing existing techniques. This signifies the potential of the proposed DAD method in addressing the challenges of robustness and generalization in ML models.", "weaknesses": "The reviewer found it hard to understand the main idea at first glance, the paper could improve its clarity in figures, as well as in presenting the assumptions and theoretical analysis, and its connection to the proposed method. Explicitly linking the theoretical framework to the practical implementation of the DAD method would help readers grasp the underlying principles and motivations behind the proposed approach more effectively.\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the constructive comments and positive review. We are glad the reviewer agreed with the core contributions of our setting and the importance of small, robust models.\n\\\n\\\n*The reviewer found it hard to understand the main idea at first glance, the paper could improve its clarity in figures, as well as in presenting the assumptions and theoretical analysis, and its connection to the proposed method. Explicitly linking the theoretical framework to the practical implementation of the DAD method would help readers grasp the underlying principles and motivations behind the proposed approach more effectively.*\n\n1. We have attached an updated main figure in the PDF. \n2. In Sec 3.4, our goal is to establish the conditions where robustness is achieved through data augmentation / adversarial training. We find that the diversity of an adversarial sample (measured by Wasserstein distance from the training distribution) depends on the diversity of the model’s training distribution, so we propose using the large-scale teacher’s gradients to generate this. This can be mapped to L163 in Eq. 9 if we substitute α(ϕ(P1)) for DAD and α(ϕ(P2)) for DAT in Lemma 3.4. We will revise the section with this explicit notation.\n3. For a qualitative result linking our theory and implementation, we kindly point the reviewer to the Appendix, where we provide figures that demonstrate the relationship between the magnitude of the distribution shift to performance. In figure B, we find that contrary to standard ImageNet training and DAT, DAD has improved performance when the Wasserstein distance of the evaluation distribution is higher. This is also shown in figure C, where we show that the improvement over standard training with DAD is higher than with DAT. This reflects our theoretical conclusions that DAD adversarial examples are more informative. We moved this result to the Appendix due to space, but agree with the reviewer that demonstrating this connection is important and will add this to the main paper in the revised version. \n\n\\\n*In addition, why the authors did not provide the code for review but answered the Reproducibility as \"yes\"? Please provide the code during the rebuttal period, using Anonymous GitHub is a viable option*\n\nThanks for bringing this to our attention, we will attach the code following rebuttal guidelines.\n\\\n\\\n*Also, the authors claim that the proposed only brings negligible computational overhead, but the reviewer could not find the part that addresses computing resources; what exactly are the computational costs for the proposed framework?*\n\nWe apologize for not being clear about this claim. We have attached a table with the results of an empirical analysis following the methodology in FastAdvProp [1] of the full computational cost of DAD. To clarify, what we mean by “negligible computational overhead” is the cost of generating adversarial examples, which is the largest cost of adversarial training. Compared to standard adversarial training and DAT, since we generate our samples from a frozen teacher and reuse them, the cost of generation over a full training run becomes negligible. The only major additional cost over standard training is training on the new examples, which is also shared by other forms of adversarial training. We will update the claim with the precise statement in the revised paper. \n\n| Method               | Attack Steps | Training Budget |\n|----------------------|--------------|-----------------|\n| ImageNet | 0           | 1x             |\n| Adversarial Training | 10           | 11x            |\n| AdvProp              | 5            | 7x              |\n| AdvProp              | 1            | 3x              |\n| DAT                  | 1            | 3.5x            |\n| DAD                  | 1            | 2x              |\n\n\n\\\nWe hope our response addressed the reviewer’s questions and concerns. We are happy to answer any further questions.\n\n[1] Jieru Mei, Yucheng Han, Yutong Bai, Yixiao Zhang, Yingwei Li, Xianhang Li, Alan Yuille, and Cihang Xie. Fast advprop. *ICLR*, 2022\n\n"}, {"review_id": "nh3fWbZgtc", "reviewer": "Reviewer_Y3Hr", "summary": "The paper introduces a new method to improve the robustness of vision models through knowledge distillation by leveraging a large-scale pre-trained teacher model (CLIP) and a VQ-GAN discretizer. The paper does a great job motivating the method and arguing for knowledge distillation when tackling out of distribution robustness. It also includes extensive theoretical justifications. And finally, the paper introduces and evaluates a novel distillation objective (\"DAD\"). In the proposed method (\"DAD\") the student model is trained to minimize a loss made up of a standard cross-entropy loss, a KL to the teacher's predictions (i.e. standard knowledge distillation) and an additional KL between the teacher and student's predictions on additional (teacher-adversarial) data points (which is novel). The paper also includes a reasonable evaluation section for the method with overall positive results, while highlighting some considerable performance improvements on natural distribution shifts over baselines.", "questions": "- At a high level, looking at equation (9) of the full model loss, I'm trying to understand how each component affects downstream robustness. So, I'd like to able to map the contribution of each of the two \"l_2\" losses and the way in which the adversarial example is computed to specific results. This seems like a pretty straightforward and important ablation. Table 3 and Table 5 present just a slice of this ablation as far as I can tell. Could you please clarify how one can reconstitute this ablation from the results presented in the paper?\n\n- The adversarial examples could also not be computed but rather perturbations could be sampled instead (while still using the VQGAN), or random data augmentation methods (or just an identity function used, i.e. which would correspond to just dropping the last l_2 loss in Eq 9). (From Table 6 it seems that more iterations actually results in worse performance across many metrics.) Have you experimented with this?\n\n- In Table 1, does the row \"DAT + DAD (Ours)\" refer to training with DAT & DAD on top of AugReg-ViT[55]? If yes, please mark it with a leading \"+\" for clarity.\n\n- In line 303, the paper states \"These adversarial examples are generated with DAD\" -- was \"DAT\" intended?", "limitations": "The authors adequately addressed the method's limitations.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is relatively well written and easy to follow. The paper introduces a relatively novel and interesting knowledge distillation loss which should have increasingly wider applicability and grow in significance as more and more of the ML world is moving towards leveraging pre-trained foundation models.", "weaknesses": "- Missing paragraph detailing DAT: since a large proportion of the strongest results seem to be relying on combining DAD with DAT. The paper should also include a description of DAT's loss and the \"DAT + DAD\" loss written out (in the appendix if no space available in the main body).\n- The paper mentions the method has \"negligible computational overhead\" -- I strongly disagree. Computing adversarial examples for a CLIP model that also passes them through VQGAN is not negligible. One can say that the computation cost is amortized though -- as you only have to do it once and then read it from disk.\n- Missing \"Avg\" column in Table 2 and 3, and best results are not shown in bold in Table 2.\n- Loss is unclear (Eq 9) -- please expand and clarify this to show which variables are being optimized over, at which stage, which are the constraints exactly, and specify the losses (what is l_1 and l_2 precisely - these are only mentioned in passing in line 125-126 being \"typically\"/\"usually\" CE and KL); and also please use or remove \"n\", as currently it is not used to index anything.\n\n- Missing references, e.g., on [1] (on worst case risk over a population of distributions) and [2] (for a related formulation of a similar set of assumptions and similar theoretical conclusion):\n\n[1] Arjovsky et al., Invariant Risk Minimization.\n\n[2] Calian et al., Defending against image corruptions through adversarial augmentations.\n\n- Figure 1 does not seem to be faithful depiction of the method as described in the paper, so it should ideally be improved significantly (the caption could also give more details towards explaining the method).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the extensive and positive review and helpful suggestions. We are glad the reviewer found our proposed method novel and relevant. \n\\\n\\\n*Missing paragraph detailing DAT: since a large proportion of the strongest results seem to be relying on combining DAD with DAT...*\n\nThank you for bringing this to our attention, we will formally write out the objectives for all compared methods in the revised paper. DAT follows the standard adversarial training objective, but with discretized samples: \n\\\n\\\n$CE(x, y) + CE(x’,y)$\n\\\n\\\nWe combine DAT with DAD by adding the second L1 objective from DAT to the DAD objective. The full loss is:\n\\\n\\\n$CE(x, y) + KL(\\theta(x), \\phi(x)) + KL(\\theta(x’), \\phi(x’)) + CE(x’,y)$\n\\\n\\\n\\\n*The paper mentions the method has \"negligible computational overhead\" -- I strongly disagree. Computing adversarial examples for a CLIP model that also passes them through VQGAN is not negligible. One can say that the computation cost is amortized though -- as you only have to do it once and then read it from disk.*\n\nWe apologize for not making the claim clearer. We agree that generating adversarial examples with a VQGAN is not negligible. But as the reviewer suggests, the cost over a full training run is amortized. We generate the set of adversarial examples once from the frozen teacher and reuse it for all of our training. For training a ViT-B model for 300 epochs, this cost is essentially reduced by a factor of 300. We also use the straight-through gradient applied in DAT, so the actual cost of generation is the same, but because DAT generates these examples every epoch, DAD is significantly cheaper. For a more formal analysis, we follow the methodology in FastAdvProp [1]. The only major additional cost is training on the new adversarial examples. We will also update the claim in the revised paper to make the statement more explicit.\n\n| Method               | Attack Steps | Training Budget |\n|----------------------|--------------|-----------------|\n| ImageNet | 0           | 1x             |\n| Adversarial Training | 10           | 11x            |\n| AdvProp              | 5            | 7x              |\n| AdvProp              | 1            | 3x              |\n| DAT                  | 1            | 3.5x            |\n| DAD                  | 1            | 2x              |\n\n\\\n*Missing \"Avg\" column in Table 2 and 3, and best results are not shown in bold in Table 2.*\n\nWe have uploaded an updated table in the PDF and will include this in the revised paper.\n\\\n\\\n*Loss is unclear (Eq 9)...and also please use or remove \"n\", as currently it is not used to index anything.*\n\nWe apologize for the confusion. We will remove “n” from this loss. We provide the loss here.\n\n$CE(x, y) + KL(\\theta(x), \\phi(x)) + KL(\\theta(x’), \\phi(x’))$\n\\\n\\\n*Missing references...*\n\nThank you for bringing these references to our attention. We will cite them in the revised paper.\n\\\n\\\n*At a high level, looking at equation (9) of the full model loss, I'm trying to understand how each component affects downstream robustness...Could you please clarify how one can reconstitute this ablation from the results presented in the paper?*\n\nWe apologize for not making the objectives of the baselines and ablations explicit. We will add this to the revised paper. Yes, the results from ablating different components of DAD can be reconstruction from our experiments. \n\\\n\\\n$L1$ \n\n$CE(x, y)$\n\nThis corresponds to standard training on ImageNet and is the simplest baseline we compare to in Table 1.\n\\\n\\\n$L1 + L2_1$ \n\n$CE(x, y) + KL(\\theta(x), \\phi(x))$\n\nThis corresponds to standard knowledge distillation. We observe the results of this objective on various student/teacher architectures in Table 7. We find that this simple objective is enough to transfer a degree of robustness from a variety of teachers.\n\\\n\\\n$L1 + L2_1 + L2_2$ \n\n$CE(x, y) + KL(\\theta(x), \\phi(x)) + KL(\\theta(x’), \\phi(x’))$\n\nThis is our proposed method DAD. \n\\\n\\\n$CE(x, y) + CE(x’, y)$\n\nTo separate L2_2 and see the direct effect of using DAD adversarial examples, we point the reviewer to Table 5. Here we use the DAT objective with precomputed DAD samples. We find that training on DAD adversarial examples without distillation can also improve performance on natural distribution shifts but to a smaller extent.\n\\\n\\\n*The adversarial examples could also not be computed but rather perturbations could be sampled instead (while still using the VQGAN), or random data augmentation methods (or just an identity function used, i.e. which would correspond to just dropping the last l_2 loss in Eq 9)...*\n\nWe thank the reviewer for the interesting suggestion. In fact, we already provide results using common data augmentations. This can be observed in Table 7 in the Appendix. Here we use Mixup and RandAugment, which are components of AugReg, and demonstrate the results of distilling from CLIP without using the generative model. Since these augmentations replace the base image and do not add additional samples, the two L_2 losses are combined. The results are much worse, with an average -13.2% decrease in performance on natural distribution shifts. We will provide the results of sampling the VQGAN in the discussion phase.\n\\\n\\\n*In Table 1, does the row \"DAT + DAD (Ours)\" refer to training with DAT & DAD on top of AugReg-ViT[55]?*\n\nYes, that is correct, DAT + DAT is also on top of AugReg. We will update the table with the “+”\n\\\n\\\n*In line 303, the paper states \"These adversarial examples are generated with DAD\" -- was \"DAT\" intended?*\n\nNo, the original statement is correct. The goal of this ablation is to see how DAD examples improve over DAT examples.\n\\\n\\\nWe hope our response addressed the reviewer’s questions and concerns. We are happy to answer any further questions.\n\nJieru Mei, Yucheng Han, Yutong Bai, Yixiao Zhang, Yingwei Li, Xianhang Li, Alan Yuille, and Cihang Xie. Fast advprop. ICLR, 2022\n"}, {"review_id": "vKy2pFppZt", "reviewer": "Reviewer_qyW3", "summary": "This paper proposes a simple and lightweight framework for improving the robustness of vision models through knowledge distillation. This paper applies pre-trained teacher model to generate adversarial examples and apply VQGAN as a data augmentation method to generate more informative adversarial samples. This paper provides a theoretical framework for applying robust teacher in the knowledge distillation with data augmentation.", "questions": "see the weakness", "limitations": "see the weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+The paper applies a large pre trained model based on knowledge distillation to enhance the model's out-of-distribution robustness with data augmentation, this idea is novel.\n\n+The paper provides a theoretical analysis of how to best distill robustness from a robust teacher trained on a large-scale dataset.\n\n+The experiment of the paper can prove the effectiveness of the method.\n", "weaknesses": "-The motivation for applying VQGAN is not so clear. The article claims that VQGAN can discretize adversarial examples, but can other data augmentation technology such as common data augmentation methods (translation, cropping), other variants of GAN, or Stable Diffusion achieve better performance? This part seems to be a direct application, and I think a discussion of the necessity should be provided.\n\n-Although this paper has a pretty performance of out-of-distribution robustness, I wonder about the performance of adversarial robustness (such as PGD or AutoAttack).\n\n-The frame diagram is somewhat rough and crude (this will not affect the final rating, but I hope the author can make some modifications).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful suggestions and positive review. We are glad the reviewer found our idea novel and think the experiments and theory prove its effectiveness. \n\\\n\\\n*The motivation for applying VQGAN is not so clear. The article claims that VQGAN can discretize adversarial examples, but can other data augmentation technology such as common data augmentation methods (translation, cropping), other variants of GAN, or Stable Diffusion achieve better performance? This part seems to be a direct application, and I think a discussion of the necessity should be provided.*\n\nThe motivation for applying VQGAN is to ensure perturbations are semantic to better match real-world transformations. This allows us to focus on an out-of-distribution robustness setting. This is motivated by several works that use generative models to construct data augmentations; we adapt the choice VQGAN from the prior SOTA DAT, which finds VQGAN performs best. The VQGAN can apply a semantically meaningful transformation on the image to better capture a useful distribution shift than standard pixel-level perturbations. \n\nThe effects of not applying a generative model and using common data augmentations can be observed in Table 7 in the Appendix. Here we use Mixup and RandAugment which are components of AugReg and demonstrate the results of distilling from CLIP without using the generative model / adversarial examples. The results are much worse, with an average -13.2% decrease in performance on natural distribution shifts.\n\nFollowing the reviewer's suggestion, we also provide the results on Stable Diffusion, a newer model since DAT was proposed. We use the generic prompt \"A photo of an {object}\". We observe a significant decrease in performance when trained using DAD compared to VQGAN. We hypothesize that VQGAN is better suited for the image-to-image task of discretizing images than a text-to-image model. Perhaps modifying the text prompt for could boost performance and be an interesting avenue for future work, especially since CLIP also requires a text prompt.\n\n| | ImageNet | V2 | Rendition | Sketch | A | Avg |\n| --- | --- | --- | --- | --- | --- | --- |\n| Stable-Diffusion | 79.1 | 67.8 | 45.9 | 33.4 | 22.0 | 49.6 |\n| VQGAN | 79.6 | 69.9 | 65.1 | 46.1 | 31.8 | 69.5 |\n\n\\\n\\\n*Although this paper has a pretty performance of out-of-distribution robustness, I wonder about the performance of adversarial robustness (such as PGD or AutoAttack).*\n\nThis is an interesting observation. We evaluate our ResNet50 checkpoint on FGSM. We observe an improvement on adversarial robustness compared to both standard training and DAT. We will provide results on DamageNet and ViT-B in the discussion phase.\n\n\n| Method       | FGSM  |\n|--------------|-------|\n| ResNet50     | 23.5% |\n| ResNet50-DAT | 33.0% |\n| ResNet50-DAD | 43.5% |\n\n\\\n*The frame diagram is somewhat rough and crude (this will not affect the final rating, but I hope the author can make some modifications).*\n\nThank you for pointing this out, we have uploaded an updated figure in the uploaded PDF.\n\\\n\\\nWe hope our response addressed the reviewer’s questions and concerns. We are happy to answer any further questions or provide additional results.\n\n"}, {"review_id": "8xiL1wwTbY", "reviewer": "Reviewer_ktZ5", "summary": "The authors propose a framework that improves the image classification model's robustness by distilling CLIP models and augmenting adversarial learning with pre-trained generative models. The authors follow classical adversarial learning to generate perturbed examples and then input the examples to VQ-GAN. Lastly, they distill a CLIP model into a smaller network with both normal and augmented adversarial examples. While I have some concerns about the results, extensive experiments verify the effectiveness of the proposed method in many settings.", "questions": "1. The performance could be better, even though it outperforms prior work. In Table 1, neither the baselines nor the proposed method achieves the CLIP baseline performance. Maybe one should look into cross-modality instead of augmenting query datasets for distillation.\n\n2. What does it mean by *DAD (no distill)*? I recommend the authors refer the readers to the corresponding loss terms.\n\n3. The proposed method is similar to leveraging more data to perform distillation (Eq. 8). The author may like to explore using different kinds of data (e.g., ImageNet + CIFAR-10 or ImageNet + ImageNet with augmentation), which could even outline the contribution of the proposed method.", "limitations": "1. The authors only consider distilling CLIP models.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "1. The extensive experiments cover various settings, such as different architectures for teacher and student models and variations of ImageNet.\n2. The authors provide some theory explaining that a more diverse distribution could be helpful for distillation.", "weaknesses": "1. The research problem is unclear. In L37, the authors claim to \"train small robust models with a large-scale model as a teacher, but without access to the teacher's training data.\" However, they focus more on out-of-distribution robustness in the later parts and the experiments.\n \n2. The proposed method needs to be better justified. In L118, the authors suddenly introduce generative models to augment adversarial examples and claim that it helps generalizability without any reasoning. If possible, the authors should provide more motivation and insights into it and conduct some experiments.\n\n3. Many notations are not defined. For example, the definition of $\\theta(x, x\\prime)$ is missing. What exactly does $Q(x\\prime)$ work? \n\n4. The novelty is limited. The main contribution seems solely to leverage VQ-GAN to enhance distillation. The reason for choosing VQ-GAN is also missing.\n\n5. The research scope is limited. The authors claim in L45 that distilling large-scale models into smaller models improves adversarial robustness. However, the experiments only consider CLIP models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the insightful comments. We will address the concerns about the novelty and justification for our method.\n\\\n\\\n*The research problem is unclear...they focus more on out-of-distribution robustness in the later parts and the experiments.*\n\nThank you for pointing this out. To clarify, the goal of our DAD is to improve out-of-distribution robustness on smaller models and datasets by distilling from a large-scale teacher. We point out the distinction between different forms of robustness on L67-69, but agree with the reviewer that this should be clear from the beginning and will revise the statement in the updated paper.\n\\\n\\\n*The proposed method needs to be better justified...If possible, the authors should provide more motivation and insights into it and conduct some experiments.*\n\nThe choice of data augmentation is one of the key components of our proposed method. Since we focus on out-of-distribution robustness, we adopt a generative model to generate data augmentations adversarially to ensure the resulting transformation is semantically meaningful. The VQGAN can apply a semantically meaningful transformation on the image to better capture a useful distribution shift than standard pixel-level perturbations. The effects of not applying a generative model and using common data augmentations can be observed in Table 7 in the Appendix. Here we only used Mixup and RandAugment and demonstrate the results of distilling from CLIP without using the generative model / adversarial examples. The results are much worse, with an average -13.2% decrease in performance on natural distribution shifts. \n\\\n\\\n*Many notations are not defined. For example, the definition of theta(x,x’) is missing. What exactly does Q(x’) work?*\n\nX refers to the base input (L101) and x’ refers to the augmented input (L104). Theta(x, x’) means passing the image or augmented image through the student model. We will add this notation to the revised paper. Q(x’) indicates the process of discretizing an input image, which we describe in more detail in L145-152. To summarize, after we apply a normal perturbation x -> x’, the VQGAN encodes input x’ into a latent representation which is then decoded, applying a semantic transformation.\n\\\n\\\n*The novelty is limited. The main contribution seems solely to leverage VQ-GAN to enhance distillation. The reason for choosing VQ-GAN is also missing.*\n\nWe kindly emphasize our key contributions below:\n1. *Setting.* We are the first to also introduce a robust teacher and use knowledge distillation to further improve robustness. We focus on OOD-robustness due to practical usefulness and the specialization of large-scale models on natural distribution shifts. These models exhibit strong robustness due to their diverse training data, and we are the first to leverage these representations as an additional form of regularization\n2. *Objective.* We introduce a novel knowledge distillation objective for the OOD-robustness setting where we add a second KL-divergence term between the student and teacher predictions on the augmented image. Previous works in defensive distillation like ARD use the teacher predictions on the normal image for this second term, but this is not adaptive to semantic transformations.\n3. *Data augmentation.* We indeed use a VQGAN for the data augmentation. However, our key novelty here is using the teacher’s gradients to generate adversarial examples. We find that this results in more diverse adversarial examples closer to the teacher’s representations, and our theoretical framework is based on this idea. We choose VQGAN in particular following the prior SOTA DAT, which finds this to lead to the best result. We also try using the newer Stable Diffusion model in the response to Reviewer qyW3, and find that it results in worse performance.\n\n\\\n*The research scope is limited. The authors claim in L45 that distilling large-scale models into smaller models improves adversarial robustness. However, the experiments only consider CLIP models.*\n\nWe focus our analysis on CLIP because of its accessibility and performance. At the time of submission, there aren’t any open-source large-scale vision models better than CLIP. However, we note that DAD is agnostic to the choice of teacher. In fact, we provide results in Table 4 using different teacher models. Generally we find that it is better to use CLIP. We hope to use newer foundation models as they become available.\n\\\n\\\n*The performance could be better, even though it outperforms prior work...Maybe one should look into cross-modality instead of augmenting query datasets for distillation.*\n\nWe propose to use CLIP as an additional source of regularization in lower compute regimes, rather than improving upon CLIP. Data augmentation is easy to apply and can be done across a variety of settings, including in cross-modal training. DAD also improves upon prior data augmentations, especially on natural distribution shifts. \n\\\n\\\n*What does it mean by DAD (no distill)?...*\n\nThis means training with only the L1 cross-entropy loss but with DAD samples, which corresponds to standard training using previously computed DAD samples for augmentation. We will make this clearer in the revised paper.\n\\\n\\\n*The proposed method is similar to leveraging more data to perform distillation (Eq. 8). The author may like to explore using different kinds of data...*\n\nIndeed we do try simpler data augmentations in Table 7 in the Appendix. Here we use Mixup and RandAugment and find the results are worse. The goal of using the VQGAN and teacher-gradient-based adversarial examples is to generate semantically meaningful and diverse perturbations. We find that these simpler forms of data do not represent difficult real-world transformations as well as DAD. Combining different datasets is certainly an interesting direction to consider.\n\\\n\\\nWe hope our response addressed the reviewer’s questions and concerns. We are happy to answer any further questions.\n"}]

Author rebuttal:
We are thankful for the generally positive reviews and useful feedback. We are glad reviewers found our idea novel (Reviewers gGrQ, Y3Hr, qyW3), setting significant (Reviewers gGrQ, Y3HR), and analysis convincing (Reviewers GKv1, qyW3, ktZ5). We provide an updated main figure, updated Table 2, and additional visualizations of DAD adversarial examples. To address specific concerns, we also provide additional experimental results on using Stable Diffusion, computational cost, and adversarial robustness.

Meta-review: The recommendation is based on the reviewers' comments, the area chair's personal evaluation, and the post-rebuttal discussion. 

This paper studies a new way of generating discrete adversarial examples with the aid of CLIP models. All reviewers find the studied setting novel and the results provide new insights. The authors’ rebuttal has successfully addressed the major concerns of reviewers, and the reviewers' comments and suggestions have helped sharpen the results and presentations. All reviewers are in favor of accepting this submission. Therefore, I recommend acceptance of this submission. I also expect the authors to include the new results and suggested changes during the rebuttal phase to the final version.
Decision: Accept (poster)