Rethinking the Effect of Data Augmentation in Adversarial Contrastive Learning
Authors: Rundong Luo
Yifei Wang
Yisen Wang
Venue: ICLR 2023 poster (2023)
Keywords: adversarial training
contrastive learning
adversarial contrastive learning

Abstract: Recent works have shown that self-supervised learning can achieve remarkable robustness when integrated with adversarial training (AT). However, the robustness gap between supervised AT (sup-AT) and self-supervised AT (self-AT) remains significant. Motivated by this observation, we revisit existing self-AT methods and discover an inherent dilemma that affects self-AT robustness: either strong or weak data augmentations are harmful to self-AT, and a medium strength is insufficient to bridge the gap. To resolve this dilemma, we propose a simple remedy named DYNACL (Dynamic Adversarial Contrastive Learning). In particular, we propose an augmentation schedule that gradually anneals from a strong augmentation to a weak one to benefit from both extreme cases. Besides, we adopt a fast post-processing stage for adapting it to downstream tasks. Through extensive experiments, we show that DYNACL can improve state-of-the-art self-AT robustness by 8.84% under Auto-Attack on the CIFAR-10 dataset, and can even outperform vanilla supervised adversarial training for the first time. Our code is available at \url{https://github.com/PKU-ML/DYNACL}.

Published as a conference paper at ICLR 2023
RETHINKING THE EFFECT OF DATA AUGMENTATION IN ADVERSARIAL CONTRASTIVE LEARNING
Rundong Luo Yifei Wang Yisen Wang3.4f
1 "Scoool of EECS, Peking University
Schooo of Mathematical Sciences, Peking University
"National Key Lab. of General Artificial Intelligence.
School of Intelligence Science and Technology, Peking University
4 Institute for Artificial Intelligence, Peking University
ABSTRACT
Recent works have shown that self-supervised learning can achieve remarkable robustness when integrated with adversarial training (AT). However, the robustness gap between supervised AT (sup-AT) and self-supervised AT (self-AT) remains significant. Motivated by this observation, we revisit existing self-AT methods and discover an inherent dilemma that affects self-AT robustness: either strong or weak data augmentations are harmful to self-AT, and medium strength is insuf- ficient to bridge the gap. To resolve this dilemma, we propose a simple remedy named DYNACL (Dynamic Adversarial Contrastive Learning). In particular, we propose an augmentation schedule that gradually anneals from strong augmenta- tion to a weak one to benefit from both extreme cases. Besides, we adopt fast post-processing stage for adapting it to downstream tasks. Through extensive ex- periments, we show that DYNACL can improve state-of-the-art self-AT robustness by 8.84% under Auto-Attack on the CIFAR- 10 dataset, and can even outperform vanilla supervised adversarial training for the first time. Our code is available hattp:://qituuu com wwt.///////////OEEA
INTRODUCTION
Learning low-dimensional representations of inputs without supervision is one of the ultimate goals of machine learning. As a promising approach, self-supervised learning is rapidly closing the performance gap with respect supervised learning (He al., 2016 Chen al. (020b) in downstream tasks. However, for whatever supervised and self-supervised learnin models, adversarial vulnerability remains a widely-concerned security issue, i.e., natural inputs injected by small and human imperceptible adversarial perturbations can fool the deep neural networks (DNNs) into making wrong predictions (Goodfellow et al. 2014).
In supervised learning, the most effective approach enhance adversarial robustness is adversarial training (sup-AT) that learns DNNs with adversarial examples (Madry et al.) 2017 Wang et al. 2019
Zhang et al. 2019; Wang et al. 2020 Wang & Wang 2022). However, sup- AT requires groundtruth
labels to craft adversarial examples. In self-supervised learning, recent works including RoCL (Kim
et al. (020). ACL (Jiang et al.) 2020), and AdvCL (Fan et al.) 2021) explored some adversarial training counterparts (se.f-AT). However, despite obtaining certain degree of robustness, there is still a very large performance gap between sup-AT and self-AT methods. As shown in Figure [[a] sup-AT obtains 46.2% robust accuracy while itate-of-ie oatee--f-th--nn the-art self AT method only gets 37.6% on CIFAR- 10, and the gap 8%. As reference, in standard training (ST) using clean examples, the gap in classification accuracy between sup-ST and self-ST is much smaller (lower than 1% on CIFAR- 10, see da Costa et al. (2022)). This phenomenon leads to the following question:
What is the key fuctor that prevents self-AT from obtaining comparable robustness to sup-AT?
To answer this question, we need to examine the real difference between sup-AT and self-AT. As they share the same minimax training scheme, the difference mainly lies in the learning objective.
"Equal Contribution.
I Corresponning Author: Yisen Wang (yisen, wang www.@wku.edd........

---

Published as a conference paper at ICLR 2023
Original
Sup-AT
SaffAT
Bandard amnca
Roust scursey Af
PbG.
ACL
Acv0L
CHAAGDDMA.,11
sugrentzion strngg
(a) robustness comparison
(b) contrastive learning
(c) data augmentations
Figure I: Experiments on CIFAR-I0: (a) Comparison supervised AT (vanilla PGD-AT (Madry et al.
2017)) and five self- self-supervised AT methods: RoCL, ACL, AdvCL, our DYNACL and DYNACL+ ++
with ResNet-18 backbone. (b) Performance of standard contrastive learning (Chen et al. 2020a) and adversarial contrastive learning (Jiang et al. (020) using different augmentation strengths. (c) Illustrative examples of data augmentations adopted by sup-AT and self-AT. We can that self-AT
adopts much more aggressive augmentation than sup-AT.
Different from sup-AT relying on labels, self-AT methods (RoCI, ACI, and AdvCL. mostly adopt contrastive learning objectives, which instead rely matching representations under (strong) data augmentations to learn meaningful features. As noted by recent theoretical understandings (HaoChen) et al. 202T] Wang et al., 2022), the core mechanism of contrastive learning is that it leverages strong augmentations to create support overlap between intra-class samples, and result, the alignment between augmented positive samples could implicitly cluster intra-class samples together. Weak augmentations, instead, not capable of generating enough overlap, and thus lead to severe performance degradation (Figure [bb) red line). In the meanwhile, we also find that strong data augmentations can be very harmful for ad adversarial robustness (Figure [[b) blue line). This reveals critical dilemma of self-AT, that (default) strong augmentations, although useful for standard accuracy, also severely hurt adversarial robustness. To understand why, we draw some augmented samples and observe that self-AT augmentations induce significant semantic shifts original images (Figure [(e]. This suggests that strong augmentations create a very large distribution shift between training and test data, and the local Ep robustness on training examples becomes less transferable to test data. Notably, some augmented samples from different classes can even become inseparable (the bird and the cat samples in red squares), and thus largely distort the decision boundary. The necessity and harmfulness of data augmentations form a fundamental dilemma in self-AT.
Therefore, as data augmentations play an essential role in self-AT, we need to strike a balance between utilizing strong augmentations for representation learning and avoiding large image distortions for good robustness. Although it seems impossible under the current static augmentation strategy, we notice that we can alleviate this dilemma by adopting a dynamic augmentation schedule. In particular, we could learn good representations with aggressive augmentations at the beginning stage, and gradually transfer the training robustness to the test data by adopting milder and milder augmentations at later stages. We name this method Dynamic Adversarial Contrastive Learning (DYNACL). In this way, DYNACL could benefit from both sides, gaining both representation power and robustness aligned with test distribution. Built upon DYNACL, we further design a fast post- processing stage for bridging the difference between the pretraining and downstream tasks, dubbed DYNACL++. As a preview of the results, Figure [(a) shows that the proposed DYNACL and DYNACL bring a significant improvement over state-of-the-art self-AT methods, and achieve comparable, even superior robustness, to vanilla Sup-AT. Our main contributions are:
We reveal the reason behind the robustness gap between self-AT and sup-AT, i.e., the widely adopted aggressive data augmentations in self-supervised learning may bring the issues of training-test distribution shift and class inseparability.
We propose dynamic augmentation strategy along the training process to balance the need for strong augmentations for representation and mild augmentations for robustness, called Dynamic Adversarial Contrastive Learning (DYNACL) with its variant DYNACL++.
Experiments show that our proposed methods improve both clean accuracy and robustness over existing self-AT methods by a large margin. Notably, DYNACL+ improves the robustness of ACI (Jiang et al. 2020) from 37.62% to 46.46% on CIFAR-10, which is even
2

---

Published as a conference paper at ICLR 2023
slightly better than vanilla supervised AT (aadry et al. 2017). Meanwhile, it is also more computationally efficient as requires less training time than ACL.
2 BACKGROUND AND RELATED WORK
Sup-AT. Given labeled training dataset Dj = f E R", E [K]}, to be resistant to fp- bounded adversarial attack, supervised adversarial training (sup-AT) generally adopts the following min-max framework (Madry et al. 2017):
Csup-Ar AT = Bxy max (cE(f(ī 6z). (CE(f().)) = log exp(f()lll AEED
(I)
Σ exp(f()[k])
Here the classifier R" R is learned the adversarially perturbed data (3 + 8z, y), and = =( ||8|p S E denotes the feasible set for adversrial perturbation.
Contrastive Learning. For each X, we draw two positive samples x; xt from the augmentation distribution A(-1z), and draw M independent negative samples {am J== from the marginal distribu-
tion A(-) (-Iz). Then, we train an encoder q : X + 2 by the widely adopted InfoNCE loss using the augmented data pair (az, xt, {mm)) (Oord et al. 2018
min (NCE(g) Bg,z (am (NC xt {mm)ig)
exp(sim(g(z), ((=)))))(((((
(2)
(NCE(1, xt, (mm):)) = log
Σn exp(sim (g(z), (ac))g((z))) (<m))//T)
Here sim((-,) -) is the cosine similarity between two vectors, and T is a temperature hyperparameter. Surged from InfoNCE (Oord et al. 2018), contrastive learning undergoes rapid growth (Tian et al., 2019; Chen et al.] 2020a He al., 2020 Wang et al. (222]) and has demonstrated state-of art performance on self-supervised tasks (Chen et al. 2020b | Kong et al. 2020). Nevertheless, several works also point out contrastive learning is still vulnerable to adversarial attack when transfer the
learned features to the downstream classification (Ho & Nvasconcelos 2020 Kim et al.) 2020).
AeIf-AT. To enhance the robustness of contrastive learning, adversarial training was similarly adapted to self-super settings (self-AT), Since there are no available labels, adversarial examples are generated by maximizing the contrastive loss (Eq, 2 w.r.f. all input samples,
CAdwNCE(F, (x)):)))
max 5,5 (-,6)))
(NCE(E + 5, + + 8+, (x, + 1:9),
(3)
Several previous works, including ACL (Jang et al. 2020), RoCL (Kim et al. 2020), and CLAE
(Ho & Nvasconcelos] 2020), adopt Eq. 3 to perform self-AT. In addition, ACL (Jiang et al.) 2020) further incorporates the dual-BN technique (Xie et al. 2020) for better performance. Specitically, given a backbone model like ResNet, they duplicate its BN modules and regard the encoder as two different branches (all parameters are shared except for BN): the clean branch Je and the adversarial branch 9a are trained by clean examples and adversarial examples respectively:
fACL (ACL( (x, g=)):g) (NCE(", xt, {=m)ige) + (AdvNCE(x; x {=m):go))
After training, they use the adversarial branch 9n as the robust encoder. They empirically show that ACL obtains better robustness than the single branch version. Built upon ACL, AdvCL (Fan al., 2021)) further leverages an additional ImageNet- preetriind model to generate pseudo-labels for sup-AT AT via clustering. Very recently, DeACL (Zhang et al., (002) proposes a two-stage method. that is to distil standard pretrained encoder via adversarial training. In this paper, following the vein of RoCL,, ACL, and AdvCL, we study how to perform adversarial contrastive learning from scratch.
RETHINKING THE EFFECT OF DATA AUGMENTATIONS IN SELF-AT
Different from the case of supervised learning, data augmentation is an indispensable ingredient in contrastive self-supervised learning. Existing self-AT methods inherit the aggressive augmenta- tion strategy adopted in standard contrastive learning (e.g.. SimCLR (Chen et al. 2020a)), i.e., a composition of augmentations like random resized crop, color jitter, and grayscale. However, this strategy not necessarily optimal for adversarial training. In this section, we provide an in-depth investigation of the effect of this default data augmentation strategy on self-supervised adversarial training (s)lf-AT).
3

---

Published as a conference paper at ICLR 2023
digince of talliingg
6:15
schornan raqunill
Mup-AT
800
Advensent
RUD-AI
(a) training-lest distribution gap
(b) classwise distance
(c) worst-case samples
Figure 2: (a & b) Comparison of different augmentation strategies CIFAR-10, where "clean" denotes no augmentation; "adversarial" denotes adversarial examples generated by PGD attack; "sup-AT" and "self-AT" denote the corresponding default data augmentations. (c) Training examples from different classes CIFAR-10 that could yield pure black or pure white augmented views.
3.1 PROBLEMS OF AGGRESSIVE DATA AUGMENTATIONS IN SELF-AT
As shown in Figure T(c) the default augmentations in self-AT are much more aggressive than that of sup-AT, which leads to two obvious drawbacks: large training-test distribution gap and class inseparability, as we quantitatively characterize below. For comparison, we evaluate kinds of data augmentation methods CIFAR-10 10:0 1) clean (no augmentation). 2) pre-generated adversarial augmentation by a pretrained classifier, 3) the default augmentations (padding-crop and horizontal flip) adopted by sup-AT (Madry et al. 2017), and 4) the default augmentations (random resized crop, color jitter, grayscale, and horizontal flip) adopted by self-AT (Jiang et al. 2020 Kim al., 2020).
Large Training-test Distribution Gap. Although training and test samples are supposed to follow the same distribution, random augmentations often adopted in the training stage, which creates training-test distribution gap. To study quantitatively in practice, we measure this distribution gap with Maximum Mean Discrepancy (MMD) (Gretton et al. 2012) calculated between the augmented training set and the raw test set of CIFAR- 10 (details Appendix B. As shown in Figure ((a) "adversarial" and "sup-AT" augmentations only produce a small distribution gap, while the aggressive "self-AT" augmentations yield much larger gap (6 times) than 'sup-ATT. Therefore, the self-AT AT training data could be very distinct from test data, and the robustness obtained w.r.f. augmented training data could be severely degraded when transferred to the test data. Due to the limit of space, theoretical j justification of this point is included in Appendix E
Class Inseparability, Yang et al. 2020) found that under supervised augmentations and adversarial perturbations with 8/255, the minimal loo distance between training samples from different classes, namely the classwise distance, larger than
16/255. In other words, CIFAR- 10 data are adversarially separable in sup-AT. AT. However, we notice that this no longer holds in self-AT. As shown in Figure [(b) self-AT augmentations have much smaller classwise distance that is close zero. One possible cause is those CIFAR- 10 samples with complete black or white regions, which could result in identical augmentations after aggressive random cropping (Figure 2(c)). As result of class inseparability, self-AT will mix adversarial examples from different classes together, and these noisy examples will degrade the robustness of the learned decision boundary.
3.2 THE DILEMMA OF SELF-AT UNDER STATIC AUGMENTATION STRENGTH
The discussion above shows that aggressive augmentations are harmful to self-AT, which motivates us to quantitatively study the effect of different augmentation strengths on self-AT.
Quantifying Augmentation Strength. order give quantitative measure of the augmentation strength, we design a set of data augmentations Tss) that vary according to a hyperparameter [0, 1): 1) when 0, mild and contains only horizontal flip; when s 1, it is the aggressive augmentations of self-AT; and 3) when s E (0, 1), we linearly interpolate the augmentation hyperparameters to create middle degree of augmentation. For example, in random resized cropping,
As adversarial perturbations are defined in the input space, measure the distance in the input space evaluate adversarial vulnerability. We include latent-space results in Appendix H.2 which show similar trends.
4

---

Published as a conference paper at ICLR 2023
s=1
$=0.8
s=0.6
s=0.4
s=0.2
t=0 s(t==1
t-0.2"T
T= s.))=0.6
t=0.6T s(1)=0.4
t=0.8TT s(t)=0.22
t= (terminated)
=.8
s()==
Figure 3: Illustration of augmenting the same image of cat with different augmentation strengths
8 E [0, 1], where a smaller 8 (left to right) indicates milder image distortion.
R/gTWetrton Merngh
468 Cheigh
dengh
(a) training test distribution gap
(b) classwise distance
(c) accurncy and robusinsss
training process
Figure 4: Quantitative results CIFAR- 10 of self-AT (baseline method is ACL Jiang al. 2020 with different augmentation strengths.
a larger will crop to smaller region in an image, which amounts to larger training-test distribution gap, as shown in Figure 3 See detailed configuration of (T() in Appendix A.
Static Augmentation Cannot Fully Resolve the Dilemma of Self-AT. The discussion in Section 3.1 reveals that strong augmentations are harmful to adversarial training while being critical for representation learning (Figure l(b).. The augmentation strength thus becomes critical dilemma of self-AT. Here, we study whether we could resolve this issue by tuning the augmentation strength for an optimal tradeoff.. As shown in Figures 4(a) and 4(b) decreasing augmentation strength (left right) indeed brings smaller training-test distribution gap and eliminates class inseparability, which benefits adversarial robustness (blue line Figure 4(c) and hurts standard accuracy at the same time (the red line in Figure 4(c)). Roughly, 0.5 seems good trade-off between accuracy and robustness, while 41.79% robustness still much lower than the 46.23% robustness of sup-AT (orange line in Figure 4 This suggests that the dilemma of augmentation strength cannot be fully resolved by tuning a static augmentation strength. The training process Fig) presents that strong or weak augmentations have their own defects while medium augmentations only bring a limited gain (green line). In the next section, we will introduce resolve this dilemma (a preview of performance is shown the red line Figure (dd).
dynamic augmentation strategy
4 DYNAMIC ADVERSARIAL CONTRASTIVE LEARNING (DYNACL)
In this section, we propose simple but effective method DYNACL resolve the dilemma of self-AT augmentations. In Section 4.1 we introduce our dynamic pretraining method for self-AT, named further propose improved version of DYNACL with fast post- processing stage to mitigate the gap between pretraining and downstream tasks, named DYNACL++ Figure 5 in Appendix
DYNACL. In Section 4.2
demonstrates the framework of the proposed DYNACI.
A.3
4.1 DYNACL: DYNAMIC ADVERSARIAL CONTRASTIVE LEARNING
In view of the dilemma of static augmentation strength (Section 3.2 we propose new dynamic training recipe for AT. named Dynamic Adversarial Contrastive Learning (DYNACL), which consists of dynamic schedule for data augmentation and dynamic objective for model training.
5

---

Published as a conference paper at ICLR 2023
Dynamic Augmentation. Different from prior works that all adopt static augmentation strength, we propose to treat the augmentation strength s as a dynamic parameter s(t) that varies along different epochs t. In this way, at earlier training stages, we can enforce self-AT to learn meaningful representations by adopting aggressive augmentations with a large augmentation strength s(l). Afterwards, we could gradually mitigate the training- test distribution gap and class inseparability by annealing down the augmentation strength s(t) (see Figures 4(a) and 4(b).. With this dynamic augmentation schedule, we could combine the best of both worlds: on the one hand, we can benefit from the representation ability of contrastive learning at the beginning stage; on the other hand, we can achieve better adversarial robustness by closing the distribution gap the later stage.
Following the analysis above, we design the following piecewise decay augmentation schedule:
(1)) 1
t R T 1 1=
K
where T is the total number of training epochs, K is the decay period, and [-] the floor operation. this schedule s(t) starts from strong augmentations with 1 at the start
As illustrated in Figure 3 epoch, and gradually decays to weak augmentation with
by reducing K/T e/Tevery K epochs.
Dynamic ACL Loss. Following the same idea above, we can also design a dynamic version of the Instead of adopting an equal average of the clean and adversarial contrastive losses, we can learn meaningful representations in the beginning stage with the clean contrastive loss ENCE (Eq.2). Later, we can gradually turn our focus to adversarial robustness by down-weighting the clean contrastive loss and up- up-weighting the adversarial loss CAd (Eq. B) Following this principle, we propose a dynamic ACL loss with a dynamic reweighting function w(() for the t-th epoch:
ACL loss in Eq. 4
LDYNACL tg: (1 C s(t)) + (1 + w CAuNNCA s(t).
(6)
For simplicity, we assign the dynamic reweighting function w(t) as a function of the dynamic augmentation schedule st)::
( (t) =
t o-....,,T1
where À E [0.1] 1] denotes the reweighting rate. In this way, when the training starts (t = 0), we adopt an equal combination of the clean and adversarial losses as in ACI. As training continues, the augmentation strength stt) anneals down, and the reweighting coefficient wtt) becomes larger. Accordingly, the clean loss ÉNCE will have smaller weight while the adversarial loss AadvNCE will have a larger weight. At last (t T), the weight of clean loss is decreased to 1 - X, and the weight of adversarial loss is increased to 1 + A. A larger reweighting rate À indicates more aggressive reweighting. Our dynamic ACL loss degenerates to vanilla ACL loss with À = 0 as a special case.
4.2 DYNACL++: DYNACL WITH FAST POST-PROCESSING
The main dynamic pretraining phase proposed above can help mitigate the training-test data distribu- tion gap. Nevertheless, there remains a training-tes task discrepancy lying between the contrastive pretraining and downstream classification tasks: the former is instance-level discriminative task, while the latter is a class-level discriminative task. Therefore, the pretrained instance wise robustness might not transfer perfectly to the classwise robustness to be evaluated at test time.
To achieve this, we propose simple post-processing phase with Pseudo Adversarial Training (PAT) to mitigate this "task gap"" Given a pretrained encoder with a clean branch ge and an adversarial branch Ia. we simply generate pseudo labels (ūs) } by k-means clustering using the clean branch ge. which natural accuracy is relatively high because it is fed with natural samples during training. Inspired by the Linear Probing then full Tinninuning (LP-FT) strategy (Kumar et al.) 2022). we propose to learn a classifier with Linear Probing then Adversarially full Fine (LP-AFT) for better feature robustness: we first use standard training to train linear head h : 1T + R (k is the number of pseudo-classes) with fixed adversarial branch 9a. and then adversarially finetune the entire classifier h Ia on generated pseudo pairs (a, 9) using the TRADES loss (Zhang et al. 2201)) In practice, we only perform PAT for few epochs (e.g., 25), which is much shorter than the pretraining stage with typically 1000 epochs. Thus, we regard this as a fast post-processing stage of the learned features to match the downstream classification task and name the improved version DYNACL++.
Discussions on Previous Works. Our DYNACL built upon the dual-stream framework of ACL (Jiang et al. 2020). Compared to ACL with static augmentation strength and learning objective. DYNACI adopts dynamic augmentation schedule together with dynamic learning objective.

---

Published as a conference paper at ICLR 2023
Further, + recycles the clean branch of ACL to further bridge the task gap with a fast post-processing stage. Also built upon ACL, AdvCL (Fan et al., 2021) incorporates third view for high-frequency components, and incorporates the clustering process into the entire pretraining phase. Thus, AdvCL requires much more training time than ACL, while our dynamic training brings little computation overhead. Additionally, AdvCL generates pseudo labels by an ImageNet- pretrained model before pretraining, while DYNACL++ generates pseudo labels by the pretrained model itself. Moreover, even with ImageNet labels, AdvCL. still performs inferior to our DYNACL. and DYNACL++ (Section 5.I). Lastly, we devise Phoearr-robbng--oo -reeion--bveersaareeee Adversarial- -eweesaria--ull------ Sall--uul--iiin Full-Finetuning (LP-AFT) strategy exploit pseudo labels, while AdvCL. simply use them as pseudo supervision.
5 EXPERIMENTS
In this section, we evaluate DYNACL and + under the benchmark datasets: CIFAR- 10, CIFAR-100 (Krizhessky) 2009), and STL-10 (Coates al. 2011), compared with baseline methods:
RoCL Kim et al. 2020). : ACL (Jang et al. 2020). and AdvCL (Fan et al. 2021). We provide additional experimental details in Appendix
Pretraining. We adopt ResNet-18 18 (He et al., 2016 as the encoder following existing self-AT
methods (Kim et al. 2020 Jiang et al. 2020 Fan et al. (021). We set the decay period K = 50, and reweighting rate À 2/3. As for other parts, mainly follow the training recipe of ACL (Jiang (et al., 2020), while we additionally borrow two training techniques from modern contrastive learning variants: the stop gradient operation and the momentum encoder. Both are proposed by BYOL (Grill al. 2020) and later incorporated in MoCo-v3 (Chen al. 202I) and many other methods. An ablation study on these two components can be found in Appendix [.3] where we show that the improvement of DYNACL mainly comes from the proposed dynamic traming strategy.
Evaluation Protocols. We evaluate the learned representations with three protocols: standard linear finetuning (SLF), adversarial linear finetuning (ALF), and adversarial full finetuning (AFF). The former two protocols freeze the learned encoder and tune the linear classifier using cross-entropy loss with natural (SLF) or adversarial (ALF) samples, respectively. As for AFF, we adopt the pretrained encoder as weight initialization and train the whole classifier following ACL (Jiang al. 2020).
5.1 BENCHMARKING THE PERFORMANCE OF DYNACL
Robustness on Various Datasets. In Table 1 evaluate the robustness of sup-AT and self-AT methods on CIFAR-10, CIFAR- 100, and STL-TO. As we can see, DYNACL outperforms all previous self-AT methods in terms of robustness without using additional data Specifically, under the auto-attack benchmark, DYNACL brings a big leap of robustness by improving previous otate-of- the-art self-AT methods by 8.84% (from 37.62% to 46.46%) on CIFAR-1O, 4.37% (from 15.68% to 20.05%) on CIFAR-100, and 1.95% on STL-10 (from 45.26% CIFAR- 10, DYNACL++ achieves even higher AA accuracy than the supervised vanilla AT (with heavily tuned default hyperparameters (Pang al. 2020 It is the first time that self-supervised AT method outperforms supervised counterpart.
47.21%) 1)). Furthermore, on
Robustness under Different Evaluation Protocols. Table@]showe that DYNACL and DYNACL+ obtain ofte-ooftthe-rt robustness among self-AT methods across different evaluation protocols. In particular, under ALF settings, DYNACL++ outperforms all existing self-AT methods and even the sup-AT method. Furthermore, under AFF settings, DYNACL could improve the heavily tuned sup-AT baseline (Pang al. 2020) by 1.58% AA accuracy (48.96% - 50.54%). and is superior to other pretraining methods. In conclusion, DYNACL and DYNACL++ have achieved state-of-the- art performance across various evaluation protocols.
Performance under Semi-supervised Settings. Following Diang et al.] (2020), we evaluate our proposed +YNACL+ under semi-supervised settings. Baseline methods are sfate-of-the-art semi- supervised AT method UAT++ (Uesaoo al. 2019), and self-supervised method ACL Jiang et al.
"Note that AdvCL incorporates extra data as utilizes an ImageNet--rettrined model to generate pseudo labels. For a fair comparison, we replace AdvCL's ImageNet model with model pretrained on the training dataset itself. We use AdvCL to denote this ablated version and refer to the original version as "AdvCL (+ (+TmageNet)".
UAT++ (Uesato et al. 2019) did not release the training code pretrained ResNet- 18 model. We copied the RA and SA results from ACT. (Jiag et al. 2020).
7

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of supervised and self-supervised adversarial training methods CIFAR-10, 10, CIFAR-106 1000 and STL-10. SA and AA stand for standard accuracy and robust accuracy under Auto- Attack (Croce & Hein, 2020). AdvCL (+ImageNet) uses additional ImageNet data. N/A: AdvCL
does not provide ImageNet based labels for STL-10.
CIFAR-10
CIFAR-100
STL.-10
Pretraining Method
AA(%)
SA(%)
AA(%)
SA(%)
AA(%)
SA(%)
Sup-AT
46.23
84.35
23.27
58.98
29.21
49.38
RoCL
26.12
77.90
8.72
42.93
26.51
78.19
ACL
37.62
79.32
15.68
45.34
33.24
71.21
AdvCL
37.46
73.23
15,45
37.58
45.26
72.11
DYNACL (ours)
45.04
77.41
19.25
45.73
46.59
69.67
DYNACL++ (ours)
46.46
79.81
20.05
52.26
47.21
70.93
AdvCI. (+ImageNet)
42.57
80.85
19.78
48.34
N/A
N/A
Table 2: Performance comparison on CIFAR-10 with three evaluation protocols: SLF (standard linear finetuning), ALF (adversarial linear finetuning), and AFF (adversarial full finetuning).
SLF
ALF
AFF
Pretraining Method
AA(%)
SA(%)
AA(%)
SA(%)
AA(%)
SA(%)
Sup-AT
46.23
84.35
47.00
83.22
48.96
80.23
RoCL
26.12
77.90
29.69
75.62
45.02
78.51
ACL
37.62
79.32
40.91
76.57
49.46
82.11
AdvCL
37.46
73.23
37.28
73.15
48.58
82.31
DYNACL (ours)
45.04
77.41
45.72
72.87
50.54
81.84
DYNACL+ (ours)
46.46
79.81
47.95
78.84
50.31
81.94
AdvCL (+ImageNet)
42.57
80.85
42.54
79.41
49.97
83.27
The results are shown in Table 3 We observe that our DYNACL+ demonstrate fairly good performance even when only 1% of labels are available.
2020).
5.2
EMPIRICAL UNDERSTANDINGS
In this part, we conduct comprehensive experiments on CIFAR-10 have a deep understanding of the proposed DYNACL and DYNACL++. More analysis results are deferred to Appendix D
Ablation Studies. We conduct an ablation study of our two major contributions (augmentation annealing Section 4.1 and post-processing in Section 4.2) Table 4 We can see that both annealing and pest-procesiig can significantly boost the model's robustness, which demonstrates the superiority of method.
Robust Overfitting. A well-known pitfall of sup-AT is Robust Overfitting (Rice et al. 2020): when training for longer epochs, the test robustness will dramatically degrade. Surprisingly, we find that this phenomenon does not exist in DYNACL. In Figure (a) we compare DYNACL and sup-AT for very long (pre)training epochs. We can see that sup-AT overfits quickly after the 100th epoch and suffers 12% test robustness decrease eventually. Nevertheless, the test robustness of DYNACL keeps increasing along the training process even under 2000 epochs.
Loss Landscape. Previous studies (Prabhu et al. 2019: Yu et al. 2018 Wu et al. 2020) have shown that flatter weight landscape often leads better robust generalization. Motivated by that discovery, we visualize the loss landscape ACL and DYNACL+ following (Li et al. 2018). Figures (bb) and (ndd(cc)ss show that DYNACL++ has much flatter landscape than ACL, explaining the superiority of DYNACL+ in robust generalization.
Training Speed. We evaluate the total training time of self-AT methods with single RTX 3090 GPU, Specifically, the total training time is 32.7, 105.0, 29.4, and 30.3 hours for ACL,, AdyCL. DYNACI,
8

---

Published as a conference paper at ICLR 2023
Table 3: Performance under semi-supervised settings CIFAR-1O. RA stands for robust accuracy under the PGD-20 attack.
Label Ratio
UAT++
ACI.
DYNACL++ (ours)
AA(%)
RA(%)
SA(%)
AA(%)
RA(%)
SA(%)
AA(%)
RA(%)
SA(%)
1% labels
N/A
30.46
41.88
45.65
50.46
74.76
46.95
51.30
76.77
10% labels
N/A
50.43
70.79
45.47
50.01
75.14
48.56
53.00
78.34
Table 4: Ablation study two key designs of DYNACL+ on CIFAR-10. 10. Baseline method is ACL (Jiang et al. 2020), and last two lines represent DYNACL and DYNACL++, respectively.
Annealing
Post-processing
AA(%)
RA(%)
SA(%)
37.62
40.44
79.32
43.24 (45.62)
45.48 (45.04)
77.40 (-192)
X
45,04 (+7.42)
48.40 (47.96)
77.41 (-191)
46.46 (+8.84)
49.21 (48.77)
79.81 (+0.49)
and DYNACL++, respectively. Built upon ACL., DYNACL's dynamic strategy (augmentation and loss) introduces little computation overhead, and it spends even less time due to the stop gradient on the target branch. Moreover, since the post-processing only lasts a few epochs, DYNACL++ is also faster than ACL. In comparison, AdvCL is much slower and even triples the training time of ACL. These statistics show that our DYNACL and DYNACL+ are both efficient and effective.
Effect of Annealing Schedule. We evaluate DYNACL++ with different decay period K Table ((a) We observe that our dynamic schedule significantly outperforms the constant one. Also, notice that the step-wise annealing strategy performs slightly better. This may be because over-frequent data reloading (i.e., distribution shift) may make the model fail to generalize well under each distribution.
Effect of Reweighting Rate. We also study the effect of annealing loss (Eq. with different reweighting rates Table 5(b) shows that compared to no annealing (A = 0), the annealing loss (A 0) consistently improves both accuracy and robustness for around 1%, and the best robustness obtained with moderate reweighting rate
2/3.
Table 5: Performance of DYNACL+ CIFAR-10 with alternative annealing schedules and decay (a), and with different reweighting rate (in Eq. 7 (b).
periods K (in Eq. 5
(a) decay schedule and decay period K
(b) rewcighting rate À
schedule
K
AA(%)
SA(%)
X
AA(%)
SA(%)
constant
40.76
70.83
0
44.81
77.83
45.77
80.26
1/3
45.63
79.55
25
45.99
79.62
1/2
46.23
79.85
linear
50
46.46
79.81
2/3
46.46
79.81
100
45.79
79.79
1
46.01
80.46
CONCLUSIONS
In this paper, we observe dilemma about the augmentation strength in self-supervised adversarial training (self-AT) that either weak or strong augmentations degrade the model robustness through in- depth quantitative investigations. Going beyond the simple trade-off by selecting a medium constant augmentation strength, we propose to adopt dynamic augmentation schedule that gradually anneals the strength from strong to weak, named DYNACL. Afterward, we devise fast clustering- based post- processing technique adapt the model for downstream finetuning, named DYNACL++. Experiments show that the proposed DYNACL and DYNACL+ successfully boost self-AT's robustness and even outperform its vanilla supervised counterpart CIFAR-10 under Auto-Attack.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
Yisen Wang is partially supported by the National Key R&D Program of China (2022/D0160304)) the National Natural Science Foundation of China (62006153), Open Research Projects of Zhejiang Lab (No. 2022RC0AB05), and Huawei Technologies Inc.
REFERENCES
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In ICML, 2020a.
Ting Chen, Simon Kornblith, Kevin Swersky, Mohammad Norouzi, and Geoffrey Hinton. Big self-supervised models are strong semi-supervised learners. arXiv preprint arXiv:2006.10029, 2020b.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. arXiv preprint WrXiv:2004020557 2021.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single-layer networks in unsupervised feature learning. In AISTATS, 2011.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with ensemble of diverse parameter- free attacks. In ICML, 2020.
Victor Guilherme Turrisi da Costa, Enrico Fini, Moin Nabi, Nicu Sebe, and Elisa Ricci. solo-learn: A library of self-supervised methods for visual representation learning. JMLR, 2022.
Lijie Fan, Sijia Liu, Pin- Yu Chen, Gaoyuan Zhang, and Chuang Gan. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? In NewrIPS, 2021.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6577 2014.
Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A
kernel two-sample test. JMLR, 2012.
Jean-Bastien Grill, Florian Strub, Florent Altché, C. Tallec, Pierre H. Richemond, Elena Buchatskaya, C. Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, B. Piot, K. Kavukcuoglu, Rémi Munos, and Michal Valko. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS, 2020.
Jeff Z HaoChen, Colin Wei, Adrien Gaidon, and Tengyu Ma. Provable guarantees for self-supervised deep learning with spectral contrastive loss. In NeurIPS, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, 2016.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, 2020.
Chih-Hui Ho and Nuno Nyasconcelos. Contrastive learning with adversarial examples. In NeurIPS. 2020.
Ziyu Jiang, Tianlong Chen, Ting Chen, and Zhangyang Wang. Robust pre-training by adversarial contrastive learning. In NeurlPS, 2020.
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning. In NeurIPS, 2020.
Lingpeng Kong, Cyprien de Masson d' Autume, Wang Ling, Lei Yu, Zihang Dai, and Dani Yogatama. A mutual information maximization perspective language representation learning. ICLR, 2020.
Alex Krizhevsky, Learning multiple layers of features from tiny images. 2009.
10

---

Published as a conference paper at ICLR 2023
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint urXiv:2202. 10054, 2022.
Hao Li, Zheng Xu, Gavin Taylor, Christoph Studer, and Tom Goldstein. Visualizing the loss landscape of neural nets. In NeurIPS, 2018.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 170::1700.000 1706.0608, 2017.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03745 2018.
Tianyu Pang, Xiao Yang. Yimpeng Dong. Hang Su, and Jun Zhu. Bag of tricks for adversarial training. arXiv preprint arXiv:2010.00667, 2020.
Vinay Uday Prabhu, Dian Ang Yap, Joyce Xu, and John Whaley, Understanding adversarial robustness through loss landscape geometries. arXiv preprint arXiv:1907.09061, 2019.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In ICML, 2020.
Aman Sinha, Hongscok Namkoong, Riccardo Volpi. and John Duchi. Certifying some distributional robustness with principled adversarial training. arXiv preprint arXiv: 10571, 2017.
Yonglong Tian, Dilip Krishnan, and Phillip Isola. Contrastive multiview coding. arXiv preprint arXiv:1906.05849, 2019.
Jonathan Uesato, Jean- Ban-tBttiste Alayrac, Po-Sen Huang, Robert Stanforth, Alhussein Fawzi, and Pushmeet Kohli. Are labels required for improving adversarial robustness? In NeurIPS, 2019.
Hongjun Wang and Yisen Wang. Self-ensemble adversarial training for improved robustness. In ICLR, 2022.
Yifei Wang, Zhengyang Geng, Feng Jiang, Chuming Li, Yisen Wang, Jiansheng Yang, and Zhouchen Lin. Residual relaxation for multi-view representation Icarning. In NeurIPS, 2021.
Yifei Wang, Qi Zhang, Yisen Wang, Jiansheng Yang. and Zhouchen Lin. Chaos is ladder: A new theoretical understanding of contrastive learning via augmentation overlap. In ICLR, 2022.
Yisen Wang, Xingjun Ma, James Bailey, Jinfeng Yi, Bowen Zhou, and Quanquan Gu. On the convergence and robustness of adversarial training. In ICMI., 2019.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingiun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In ICLR, 2020.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust general- ization. In NeurIPS, 2020.
Cihang Xie, Mingxing Tan, Boqing Gong, Jiang Wang, Alan Yuille, and Quoc V Le. Adversarial examples improve image recognition. In CVPR, 2020.
Yao-Yuan Yang, Cyrus Rashtchian, Hongyang Zhang, Russ R Salakhutdinov, and Kamalika Chaud- huri. A closer look at accuracy vs. robustness. In NeurlPS, 2020.
Fuxun Yu, Chenchen Liu, Yanzhi Wang, Liang Zhao, and Xiang Chen. Interpreting adversarial robustness: A view from decision surface in input space. arXiv preprint arXiv: 1810001110000144 2018.
Chaoning Zhang, Kang Zhang, Chenshuang Zhang, Axi Niu, Jiu Feng, Chang D. Yoo, and In So Kweon. Decoupled adversarial contrastive learning for supervipe adversarial robustness. In ECCV, 2022.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent El Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In ICML, 2019.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features perceptual metric. In CVPR, 2018
11

---

Published as a conference paper at ICLR 2023
A ALGORITHM DETAILS
A.1 DETAILED CONFIGURATION OF DATA AUGMENTATIONS
We formulate the augmentation configuration T(s) with respect augmentation strength in the following PyTorc-sttyle code:
from torchvision import transforms
def get_transforms (strength))
ena_color_jitten jitter transforms RandomApply [itransfoors [tremsffcss.ooooriittter
cna_color___
0.4 strength, 0.4 strength, 0.4 strength, 0.1 strength) p=0.8 strength)
rnd_gray transforms heeeemmaseea (p=0. (p=0.2 strength) transform transforms. cransforms.Compose..
transtorms transtorms. Randomeesizeedrrgg
32, scale= (1.0 0.9 strength, 1.8)), $ No need to decay horizental flip
transforms. Random/iandmmiiiiiic...oooo aanddnttorrcommllllippppp.... rnd _olor_jitter,
rnd_gray,
transforms. ToTensor 1)
return transform
A.2 PSEUDO CODE OF DYNACL AND DYNACL+ ÁLGORITHM
Algorithm 1: DYNACL algorithm
input Encoder g and linear classification head h. Function of the augmentation set T(-),
number of pretraining epoch T, and post-processing epoch T' output Pretrained and post-processed encoder g.
Phase 1:
Momentum contrastive pretraining with augmentation annealing (DYNACL)
forall E 21, T) do
for sampled mini-batch X' do
augment samples from X' by augmentation set ((s(t))
C LDYNNNACLA (g)t)
3
update parameters in g to minimize C
/* Phase 2:
Pseudo label based post processing (DYNACL++) */
Extract the normal route of the momentum encoder, and denote it by Jc
{ús} = k mcans((2;)::4
Post-procee
phase 1, tune the head
Extract and freeze the adversarial route of the momentum encoder, denote by 9a
forall epoch E (1, .......... .T' do
L Standard SonnddrrdTraiiing (h 9a, { (0c))
/* Post-processing phase 2, tune the whole model
= Unlock the parameters in 9a
12 forall epoch E (1,2, ... T") do TRADES (h Ja, )xx), (ū:))
Algorithm A demonstrates the pseudo-code our proposed DYNACL and DYNACL++ Note that DYNACL+ adds fast and simple post-processing phase to DYNACL and enjoys higher robustness.
A.3 PIPELINE OF DYNACL AND DYNACL+
Figure 5
demonstrates the pipeline of the proposed DYNACL and DYNACL++ For DYNACL (upper part of the figure), adopt proposed augmentation annealing strategy and the momentum
12

---

Published as a conference paper at ICLR 2023
Dyanamic Adversarial Contrastive Pretraining
f1 Tistrengthi
EnEDORT
projection
Info-NCL
Argnestaiio
Advarsarial NCE
encarder
prajection
TRADES Advensarial Training
Unlabeled images
means Clustering
pacudo lahels
projection
Weighted
Nornaal Brasc
encodee
cioseriiation
IRADES
Advemariial Branch
Pacudo Adversarial Training
Figure 5: Pipeline of DYNACL.
technique (He et al. 2020) in the pretraining phase. Projection heads are added on the top of both branches of the ResNet backbone. For simplicity, this architecture is not explicitly shown in the figure. For DYNACL++, we add a simple and fast post-processing phase DYNACL (bottom part of the ligure). Specifically, we replace the projection head with a linear classification head and adopt PAT algorithm. Note that both two phases require NO true labels.
B DETAILS OF THE MEASUREMENT OF AUGMENTATION EFFECTS
B. CALCULATION PROTOCOLS
Calculation of minimal classwise distance. We randomly augment each sample from the training set 50 times and calculate the distance in terms of foo norm in the input space. We believe that the input space class separability is vital indicator of adversarial robustness. Typical adversarial perturbation budget is defined the input space, e-g., 8/255 under -orm.. Thus, given an augmented dataset D, whether the minimal input space classwise distance is smaller than 2 8/255 = 16/255 decides whether this dataset is adversarially separable, i.c., whether there exists a classifier that achieves 100% training robust accuracy.
Calculation of Maximum Mean Discrepancy (MMD). Denote the whole training set by Xer and the whole test set by Xts. We first transform each set into batch of tensors and then flatten them into two dimensions. Finally, we adopt Radical Basis Function as kernel function with bandwidth 10, 15, 20, 50 calculate the MMD between X,, and X..-
B.2 LATENT SPACE DISTRIBUTION GAP AND MINIMAL CLASSWISE DISTANCE
In Section 3.1
we have demonstrated the input space distribution gap between the training set and test set and found that stronger data augmentation will result in wider distribution gap. To further justify our findings, we calculate the statistics using the distance of sample features in the latent space. Specifically, we utilize the Perceptual Similarity proposed in (Zhang et al. 2018) to calculate latent space distance.
Latent Space Distribution Gap. Specifically, for MMD, given perceptual similarity function f(x v)
which takes two images y as inputs, the kernel function in MMD is defined as 1 f(z y). Note that the perceptual similarity is lower for similar images and 0 for identical images. Figure ((a) demonstrates the latent space MMD between the augmented training set and clean test set. Similar to input space MMD, stronger data augmentation will bring a bigger distribution gap, thus harming robustness generalization.
Latent Space Minimal Classwise Distance. Similarly, we can calculate the classwise distance using latent space distance. Specifically, augment each sample from the training set 20 times and use
13

---

Published as a conference paper at ICLR 2023
training- seesst MMD (relative)
24-
classwise distance (bg)
0.5
18
0.3
0.1
0.0
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
augmentation strength
augmentation strength
(a) latent space distribution gap
(b) latent space minimal classwise distance
Figure 6: (a) Latent space relative MMD between the augmented training set and clean test set. (b)
Latent space minimal classwise distance on the training set.
a pretrained AlexNet to extract its latent space features. After normalization, We calculate the & distance between augmented samples from different classes and record the minimum. The results are shown in Figure [(b) We can see that the latent classwise distance will also become significantly smaller under stronger augmentations (larger s), which generally agrees with the trend in the input space.
C EXPERIMENTAL DETAILS
C.I PRETRAINING SETTINGS
Except that we adopt the momentum backbone and stop-gradient operation (Grill et al.) 2020). we follow ACL on other pretraining configurations. Specifically, we train the backbone for 1000 epochs with LARS optimizer and cosine learning rate scheduler. The projection MLP has layers. For DYNACL pretraining. we set the decay period
50, and reweighting rate À 2/3. For DYNACL++, we generate pseudo labels for training data by K-means clustering, First, we tune the classification head only by standard training for 10 epochs, then tune both the encoder and the classification head by TRADES (Zhang et al. 2019 for 25 epochs.
C.2 EVALUATION PROTOCOLS
As for SLF and ALF we train the linear classifier for 25 epochs with an initial learning rate of 0.01 CIFAR-10 10 and 0.1 CIFAR- 100 and STL-10 10 (decays at the 10th and 20th epoch by 0.1) and batch size 512. For AFF, we employ the TRADES (Zhang et al. (219) except that in DYNACL and DYNACL++, we feed the clean samples to the normal encoder when generating adversarial perturbations. Finetuning lasts 25 epochs with an initial learning rate 0.1 (decays by 0.1 at the 15th and 20th epoch) and batch size 128. For all baselines and DYNA,I, we report the last result in the evaluation phase.
with default parameters,
the original implementation AdvCL (Fan et al. 202T), they incorporate extra data from ImageNet to train a feature extractor based on SimCLR. For fair comparison, we replace AdvCL's ImageNet- pretrained model with hyperparameters and official implementations and reaches a clean accuracy of 88.15% 15% on CIFAR-10 and 65.78% on CIFAR-100.
model. This model is trained following the default
D ADDITIONAL EXPERIMENTS
D.1 TRAINING DYNAMICS
Figure 7 demonstrates the training dynamics of ACL, DYNACL, and supervised-AT (all with cosine learning rate schedule). We observe that the robust accuracy (PGD-20) of both DYNACL and ACL increases continuously throughout the training process, while that of sup-AT suffers a significant decrease in the later period of training.
14

---

Published as a conference paper at ICLR 2023
robust accuracy (PGD-20)
50
1 Mmmmmm
45
35
30
DYNACL (stop annealing 700 epach)
DYNACL tfull annealing)
ACL
20
400
800
1000
epoch
Figure 7: Training dynamics of ACL,, DYNACI, and supervised AT.
Figure 7 also depicts the effectiveness of annealing the augmentation strength & to O. To further examine this point, consider alternative schedule, where we threshold the augmentation strength when it reaches 8 0.3 at the 700-th epoch, and adopt a constant augmentation strength s 0.3 afterwards. Table 6 shows that the thresholding strategy performs little bit worse than the original strategy.
Table Comparison between the threshold strategy and the original strategy.
AA(%)
SA(%)
original (1 - 0)
46.46
79.81
threshold 0.3
45,89
80.14
D.2 DATA AUGMENTATION STRENGTH IN SUP-AT
Popular sup-AT methods Zhang al., 2019 Pang et al. 2020) typically adopt very weak data augmentations, including only horizontal flip and padding-crop As shown in Figure & aggressive data augmentations detrimental to the model's performance. For self-AT, this phenomenon also holds, and the corresponding results are shown (C)
81
robust accuracy (Auto-Attack)
standard accuracy
79
45
77-
43
75-
41
standard accuracy
robust accuracy (auto-attack)
0.0
0.2
0.4
0.6
0.8
1.0
augmentation strength
Figure 8: Performance of sup-AT with respect augmentation strength
D.3 ABLATION OF MOMENTUM ENCODER AND STOP GRADIENT OPERATION
Here conduct ablation study the momentum encoder and stop-gradient operation that is additional adopted by DYNACL. We can see without both techniques. DYNACL still achieves very high AA robustness (44.9%) that is significantly better than ACL (Jiang et al. 2020). As for the two techniques, we notice that the stop gradient alone has limited improvement, also it helps improve training efficiency. The momentum encoder, instead, help improves the standard accuracy by nearly 2%, while attains similar AA robustness the vanilla DYNACL (ACL dynamic strategy).
15

---

Published as a conference paper at ICLR 2023
Table 7: Ablation study of momentum backhone and stop gradient operation the CIFAR-10 dataset. Our DYNACL (last line) uses both momentum encoder and stop gradient operation following (Grill al. 2020 Baseline method (first line) is ACL (Jiang al. 2020).
Dynamic Strategy
Stop Gradient
Momentum
AA(%)
RA(%)
SA(%)
X
37.62
40.44
79.32
X
44.92
48.65
75.76
X
45.02
48.57
75.00
45.04
48.40
77.41
55-
carad (SSES cantl, mi-0eo111
expuff eonens state
02
42
550
Synies epoch
22
203
460
600
600
1000 12001400 200000002000
<poch
(a) robustness v.s. training progress
(b) loss landscape of ACL
(c) loss landscape of DYNACL+
Figure (a) Comparison of Robust Overfitting between sup-AT and self-AT (DYNACL) with different pretraining schedules (without post-processing) on CIFAR-I0. (b), (c): loss landscape visualization of ACL and DYNACL++. Note that DYNACL+ enjoys flatter loss landscape compared with ACL.
D.4 ROBUST OVERFITTING AND LOSS LANDSCAPE
We provide illustrations on robust overfitting and loss landscape in Figure 9 Please refer to Sec. 5.2 for detailed discussions.
E THEORETICAL JUSTIFICATION
In this section, following the thcoretical analysis of adversarial training (Sinha et al. 2017), provide solid theoretical justification on our analysis. Specifically, we show that large training-test distribution gap caused by aggressive data augmentations will indeed induce larger test errors.
Setup. Consider input data space X' and bounded loss function ((:; () satisfying [[x;; 0) < M. For the case theoretical exposure, we consider the Wasserstein distance for measuring the distribution gap. In particular, the Wasserstein distance between P and Q is
Wa(P,Q) MEN(P..)
inf EM [c(X,X')
(8)
where I1(P, Q) denotes the couplings of P and Q, and c: X - X defines non-negative, lower semi- continuous cost function satisfying c(x, 2) 0, x E Accordingly, we adopt a robust surrogate objective Φ. 10; := supzex (É(0;z) 7e (z, xo)}, which allows adversarial perturbations of the data in the Wasserstein space, modulated by the penalty 7. Considering the equivalence of norms in the finite-dimensional space, our discussion here in the Wasserstein space also sheds light on E, -nomm bounded adversarial training.
Suppose the original data distribution (without augmentation) Ptest, and the augmented data distribution is Pang- Denote the empirical augmented training data distribution Pang with n samples. Below, we are ready to establish theoretical guarantees on the cuttof-distribution (OOD) generalization of robust training the augmented data, i.e., Pany to the original test data distribution Prest-
16

---

Published as a conference paper at ICLR 2023
Theorem For any and fixed t >> 0, the following inequality holds with probability at least e
simultaneously for all 0 E P 0, 2 0:
sup P:Wa(P SS
E P..... [e(0: 5xp +
P Paux
[0: + c..)t.
(9)
Here, en == 7b1 V class, (F,,e, I]) is the corresponding cover number, and by and b2 are numerical constants.
M
n So Vlgg N (F, (11== + b2M b2M where F is the hypothesis
Analysis. From Theorem [ we can see that controls the distribution gap between the original data distribution and the augmented data distribution, and larger p indeed introduces larger emror term the upper bound of the test performance. Specifically, when p very large, egg., with aggressive augmentations, the test performance guarantees to be poor. This shows that theoretically, too large training- test distribution gap introduced by aggressive augmentation will indeed contribute worse generalization, which echoes with our empirical investigation Section 3
Consequently. if we gradually anneal the augmentation strength from 1 to 0, as we have done in the proposed DYNACL, the distribution discrepancy p will gradually shrink (as it nearly 0 when s = at last), leading to a smaller upper bound. Thus, our augmentation annealing strategy will indeed help bridge the distribution gap and improve generalization to test data.
E. PROOF THEOREM
We first state the following lemma from Sinha et al. (Sinaa et al. 2017) that gives a duality result between the objectives on any two distributions P and QQ
Lemma 2 Let θ x X + and e X x X - R.,b continuous. Let Φn, (0; z0) supzex (E(0:2) yc (z, 20)} be the robust surrogate (2 b). For any distribution Q and any p 0.
sup [p([(0; X)] = inf {Tp + EQ (0;
P.W.(P.Q)SP
720
for any 7 2 0, we have
sup
[p](((; - = Eq
p
Applying Lemma 2 our problem, we have the following deterministic result
sup
Ep [(;0 S 7P + E [Ø, X)]
Preat (PePe Pang]Sp
for all > 0, distributions Paug. and y 2 O. Next, we show that E lo X) concentrates around its population counterpart at the usual rate. Also remind that we have that d., (0; z) E -M, M), because -M < E(0; æ) I (0; x) S sup, ((0;z) x) S M. Thus, the functional -> F, (0) satisfies bounded differences, and applying standard results on Rademacher complexity and entropy integrals gives the result.
17


Reviews:
[{"review_id": "CKv4H5kSWI", "reviewer": "Reviewer_UQhW", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper first reveals a dilemma about the augmentation strength that either strong or weak data augmentations are harmful to self-supervised adversarial training (self-AT). To resolve the dilemma,  the paper proposes a simple remedy named DynACL (Dynamic Adversarial Contrastive Learning). DynACL adopts a dynamic augmentation schedule along the training process that gradually anneals the strength from strong to weak, then adopts a fast post-processing stage for adapting it to downstream tasks. ", "summary_of_the_review": "The paper is well written, the contribution is clearly presented and the experimental results are thoroughly executed. I would tend to accept this paper.", "strength_and_weaknesses": "Strength:\n\na)\tThe paper reveals the reason behind the robustness gap between self-AT and sup-AT and investigates the effect of data augmentation strategy on self-AT.\n\nb)\tThe proposed DynACL shows a significant improvement in clean accuracy and robustness over existing self-AT methods.\n\nc)\tThe DynACL is also more computationally efficient.\n\nd)\tThis paper is well written.\n\nWeaknesses:\n\na)\tIn Table 1, the standard accuracy (SA%) of DynACL on STL-10 is significantly lower than other self-AT methods, which is not explained in the paper.\n\nb)\tThe metric RA(%) in Table 2 is not explained.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "C80-6o1yeor", "reviewer": "Reviewer_umnn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper suggests dynamic adversarial contrastive learning which gradually anneals from a strong augmentation to a weak augmentation. Further, the authors propose fast post-processing stage for adapting it to classification tasks which boost the robustness. From this simple and effective strategy, DynACL reduces the gap between supervised AT and selfsup AT. ", "summary_of_the_review": "Overall, I recommend acceptance to this paper. This is well written and has intuitive motivation with empirical results.", "strength_and_weaknesses": "**Strength**\n\n- This paper proposes a simple and effective scheduling method for selfsup-AT which could surprisingly improve the robustness.\n- The paper is well-written and shows convincing motivations.\n- This paper suggests novel aspects of the relationship between augmentation and adversarial training in selfsup-AT.\n- This paper demonstrates extensive experimental results including diverse ablation experiments and semi-supervised settings.\n\n**Concerns and Questions**\n- I hope the authors describe the difference between the DynACL++ and AdvCL post-processing which pseudo labels the unlabeled examples with k mean clustering. I think it is overemphasized claims that fast post-processing is a novel idea.\n- Does proposed technique also can be adapted to RoCL? And also boost the performance of this method too?\n- Does dynamic scheduling also benefit the performance of vanilla simCLR?\n- The proposed dynamic scheduling which gradually decreases the strength of the augmentation seems a somewhat empirical approach seems bit simple approach without any convincing motivation or evidence. It is understandable that we should control the strength of augmentation but I am not sure proposed method is the most effective design of data augmentation. Because if there is some standard to find the adequate strength of the augmentation at a certain training stage of self-AT (i.e., classwise distance, or MMD), an adaptive strength controller could show better robustness than the gradual scheduling.\n- I am not quite sure this approach could be generalized to different architectures, or different frameworks. Since the approach is a naive approach, hyperparameter search may be always needed for optimal performance. And the current gradual scheduling could not be the best option for some circumstances (where may need a longer step size). I think adaptive data augmentation scheduling [1,2] could also be applied to self-AT which could improve the current approach more intuitively and make the approach to be more universal to diverse circumstances.\n\n[1] ADAAUG: LEARNING CLASS- AND INSTANCE- ADAPTIVE DATA AUGMENTATION POLICIES, ICLR 2022\n\n[2] MetaAugment: Sample-Aware Data Augmentation Policy Learning, AAAI 2021", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}, {"review_id": "7qNV-dVy05T", "reviewer": "Reviewer_Kni5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the effects of data augmentation in adversarial contrastive learning (ACL). The authors show that stronger data augmentation can result in a larger distribution gap between training data and test data, and a smaller classwise distance. Therefore, the authors propose a piecewise decay augmentation schedule for ACL, namely DynACL. The empirical results validate that DynACL can achieve comparable and even better adversarial robustness than supervised adversarial training (Madry et al., Towards deep learning models resistant to adversarial attacks, 2017).", "summary_of_the_review": "Overall, the empirical results solidly support the effectiveness of the proposed method. However, the analysis of some investigation (e.g., why a more aggressive data augmentation can incur a smaller classwise distance?) is not very clear. Therefore, I tend to borderline accept.", "strength_and_weaknesses": "Strength\n+ This paper provides an interesting and insightful investigation of the effects of data augmentation in ACL. The empirical investigation validates that stronger data augmentation can be harmful to ACL. \n+ The proposed method is compatible with variants of ACL method and has no extra computational consumption. The experiments validate DynACL can significantly improve adversarial robustness compared to previous self-supervised adversarial training methods.\n\nWeaknesses\n- The investigation totally depends on the empirical phenomenon. Say the classwise distance. The paper only shows the phenomenon. But it seems that the paper does not provide more (theoretically) analyses of the reason for this phenomenon.  \n- The proposed dynamic scheduling is somewhat naïve. And, it is strange that the training procedure still uses the most aggressive data augmentation at the final stage, which has been pointed out as harmful to ACL.\n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster