Exploring The Forgetting in Adversarial Training: A Novel Method for Enhancing Robustness
Authors: Xianglu Wang
Hu Ding
Venue: ICLR 2025 Poster (2025)
Keywords: Adversarial training
Continual learning
Catastrophic forgetting
Primary area: alignment, fairness, safety, privacy, and societal considerations

Abstract: In recent years, there has been an explosion of research into developing robust deep neural networks against adversarial examples. As one of the most successful methods, Adversarial Training (AT)  has been widely studied before, but there is still a gap to achieve promising
clean and robust accuracy for many practical tasks. In this paper, we consider the AT problem from a new perspective which connects it to catastrophic forgetting in continual learning (CL). Catastrophic forgetting is a phenomenon in which neural networks forget old knowledge upon learning a new task. Although AT and CL are two different problems, we show that they actually share several  key properties in their training processes. Specifically, we conduct an empirical study and find that this forgetting phenomenon indeed occurs in adversarial robust training across multiple datasets (SVHN, CIFAR-10, CIFAR-100, and TinyImageNet) and perturbation models ($\ell_{\infty}$ and $\ell_{2}$). Based on this observation, we propose a novel method called Adaptive Multi-teachers Self-distillation (AMS), which leverages a carefully designed adaptive regularizer to mitigate the forgetting by aligning model outputs between new and old ``stages''. Moreover, our approach can be used  as a unified method to enhance multiple different AT algorithms. Our experiments demonstrate that our method can significantly enhance robust accuracy and meanwhile preserve high clean accuracy, under several popular adversarial attacks (e.g., PGD, CW, and Auto Attacks). As another benefit of our method, we discover that it can largely alleviate the robust overfitting issue of AT in our experiments.

Published as a conference paper at ICLR 2025
EXPLORING THE FORGETTING IN ADVERSARIAL TRAINING: A NOVEL METHOD FOR ENHANCING Ro- BUSTNESS
Xianglu Wang", Hu Ding"*
School of Artificial Intelligence and Data Science, School of Computer Science and Technology University of Science and Technology of China [wxlu]email. ustc. en, [uading)@uster edu.. en
ABSTRACT
In recent years, there has been an explosion of research into developing robust deep neural networks against adversarial examples. As one of the most successful methods, Adversarial Training (AT) has been widely studied before, but there is still gap to achieve promising clean and robust accuracy for many practical tasks. In this paper, we consider the AT problem from a new perspective which connects it to catastrophic forgetting in continual learning (CL). Catastrophic forgetting is phenomenon in which neural networks forget old knowledge upon learning a new task. Although AT and CL are two different problems, we show that they actually share several key properties in their training processes. Specifically, we conduct an empirical study and find that this forgetting phenomenon indeed occurs ad- versarial robust training across multiple datasets (SVHN, CIFAR-10, CIFAR- 100, and TinyImageNer and perturbation models (Eos and f2). Based on this observa- tion, we propose novel method called Adaptive Multi-teachers Self-distillation (AMS), which leverages a carefully designed adaptive regularizer to mitigate the forgetting by aligning model outputs between new and old "stages". Moreover, our approach can be used as a unified method to enhance multiple different AT algorithms. Our experiments demonstrate that method significantly hance robust accuracy and meanwhile preserve high clean accuracy, under several popular adversarial attacks (e.g., PGD, CW, and Auto Attacks). As another benefit of our method, we discover that it can largely alleviate the robust overfitting issue of AT in our experiments.
INTRODUCTION
Deep Neural Networks (DNNs) LeCun et al. in number of fields, such as computer vision (He 2016) and natural language processing (De- vlin et al.) 2019). But DNNs have been shown to be vulnerable to adversarial perturbation examples. Such examples are crafted by making small changes natural data, but these changes are often im- perceptible to human observers (Goodfellow et al. 2015: Szegedy et al. 2014). This vulnerability of DNNs raises significant security concerns regarding their practicality security-sensitie appli- cations, such as face recognition (Parkhi et al. 2015) and autonomous driving (Chen et al.) 2015).
2015) have demonstrated state-of-the-art performance
To address the security concerns, a great deal of defense methods have been developed to improve the adversarial robustness of DNNs (Buckman al. 2018 Guo et al.] 2018 Song et al. 2018 Xie
et al., 2018: Madry et al., 2018 Zhang et al., [019b] Yang et al. 2024 Dedeoglu et al. 2024). Among existing defense strategies, Adversarial Training (AT) (Madry al. 2018 Zhang al.
2019b) has been demonstrated as one of the most effective methods to defend against adversarial examples (Pang et al. 2021 Maini et al. 2020). The idea of AT is utilizing adversarial samples to train the model in each training epoch, which can be formulated as a "minimax robust optimization" problem, searching for the best solution to the worst-case scenario. However, the robust accuracies achieved by AT are still not that satisfying for many tasks. For instance, as representative AT method, if we perform TRADES (Zhang al. 2019b) with the for norm on CIFAR- 10. yields
"Corresponding author.

---

Published as a conference paper at ICLR 2025
robust accuracy less than 50% under Auto Attack. A complete introduction of existing AT methods is provided in Section 2
In this paper, conduct deep investigation on the learning process of adversarial samples, because whether the model can effectively learn adversarial examples greatly determines the final adversarial training performance. In particular, Continual learning (CL), an area that has been extensively studied recently (Wang et al., (024). Our inspiration comes from the following observation: during the adversarial training process, the distribution of adversarial examples varies across different training epochs, a phenomenon quite similar learning different distributions in CL (each distribution corresponds unique learning task), Specifically. the standard DNNs training process implicitly assumes that the data are drawn independently and identically distributed (i.i.d.) from the same probability distribution. However, in the scenario of AT, the generated adversarial samples often do not hold this assumption. Note that the adversarial samples each epoch are usually generated by gradient information determined by current model parameter (Madry et al. 2018). Therefore, the model parameter updates during the training process, the distribution of ad adversarial samples should continuously shift over time. Thus, be regarded unique task (similar with CL
build the seemingly unusual connection between AT and
each stage of the alternating training procedure scenario). In addition, unlike conventional machine learning models built on static data distribution, CL is characterized by learning from dynamic data distributions and often suffers from catastrophic forgetting (Kirkpatrick et al., 2017). Specifically, the catastrophic forgetting is a phenomenon in which neural networks forget old knowledge upon learning a new task. Considering the similarity between the AT and CL problems, it is natural to ask the following questions:
Does the forgetting phenomenon also occur in adversarial training process? If so, is it possible to improve the performance of adversarial training through miligating the forgetting?
1.1 OUR MAIN CONTRIBUTIONS
Firstly, we conduct an empirical study to answer the first question. Our experimental results uncover that the forgetting phenomenon indeed appears in adversarial training process. Specifically, when the model is trained the later epochs, is possible to forget the information from adversarial samples learned in earlier epochs. For instance, as shown in Figure I we continuously monitor the classification probability changes of an adversarial sample with the true label "ship" during the adversarial training process (the total epochs are evenly divided into 5 stages). We find that during stages 1 to 3, the sample was correctly classified and the confidence score reached 41. 79% in stage 2. However, the sample was misclassified in stages 4 and 5, and the confidence had dropped to merely 7.15% stage 5. In our empirical study, we can also observe this phenomenon on multiple datasets beyond CIFAR-10, 10, such as SVHN, CIFAR-100, and TinylmageNet. It is worth noting that the previous work (Gupta et al. 2020> illustrated another experiment, using the adversarial samples generated at different training stages to test the final built model. As we know, an adversarial sample is generated based on the current parameters and gradient information of the training stage it belongs to, so their experimental purpose is not about uncovering the "forgetting" issue of a fixed adversarial example across different training stages. On the other hands, our used adversarial sample in Figure generated at the beginning, and then is always fixed to test across all the stages (so to illustrate the forgetting towards this fixed adversarial sample).
33
02
(a) stage 1 (V)
(b) stage (V) (c) stage 3
(d) stage (x)
(e) stage 5 (x)
Figure 1: Demonstration of a real adversarial example being learned and correctly classified by the model in Stage 1, but forgotten and misclassified in stage 5. The true label of this adversarial example is "ship". The AT method is TRADES (Zhang et al. 2019b). The horizontal axis represents the ten classes in CIFAR-10 and the vertical axis represents classification probabilities. Probabilities corresponding to the true label are shown in purple, while all others are shown in orange. "/" means "classified correctly", and "x" means "classified wrongly".
2

---

Published as a conference paper at ICLR 2025
Following these empirical findings, we try explore that whether the performance of AT can be enhanced by mitigating the forgetting issue. In the line of research on CL, there have been nu- merous methods emerged for addressing the phenomenon of catastrophic forgetting, such as the replay-based methods (Buzzega et al., 2020; Boschini et al., 2022) and the regularization- methods (Sun al. 2023: Rebuff al., 2017). However, most of these methods are designed for standard CL tasks and cannot be directly applied to AT, due to the following two major challenges. (1) For AT, the model should capture much more subtle information from each adversarial example, compared to ordinary image classification problems, because an adversarial sample is usually gener- ated by adding small and carefully crafted changes. For example, perturbations may target specific features such as a cat's tail, rather than altering the entire sample's classification from cat to dog. (2) As we known, adversarial training is inherently much harder than training standard DNN, as theoretically proved in (Zhang et al.) 2019b). For example, adversarial training models usually have a low classification accuracy (c.g., roughly less than 40% on -IFAR-100), which may lead to trans- ferring& a certain extent of incorrect information to the subsequent "stages". Thus, how to transfer the previous correct information and avoid the negative impact from the incorrect information, is another obstacle that hinders directly applying existing CL methods to enhance AT.
To address these challenges, we propose a novel method named "Adaptive Multi-teacher Self- distillation (AMS)". which combines multi-teacher self-distillation method and adaptive weight mechanism to alleviate the forgetting issue in AT, Our initial idea comes from Knowledge distil- lation (Hinton al., 2015), which is a method that can capture more refined and subtle feature information by aligning the model soft output between old and new tasks, However, the knowledge distillation methods employed in CL (e.g., Lwf (Li & Hoiem 2017) and iCaRL (Rebulli al.) 2017)) may not be sufficiently sensitive to certain types of perturbations, leading to suboptimal defense ef- fectiveness these situations. For example, one "stage" of AT may be sensitive to pixel-leved noise, while another may be sensitive to geometric transformations or color shifts. To address this issue, we propose a multi-teacher self-distillation method designed to capture diverse feature representations, thereby enhancing the model's ability to cope with various forms of perturbations. As explained before, another issue is that the information transferred from carlier stages to later stages is not al- ways reliable. So, design "Reweighting-basee Loss Correction" mechanism to adaptively eliminate the negative effects caused by "bad" teachers. Specifically, we utilize the softmax outputs of the teacher models determine the weight for each sample based the probability of the true class, Higher probabilities indicate a greater confidence of correct classification, and thus we assign higher weights to these samples and lower weights those misclassified ones. Overall, this mech- anism ensures that misclassified samples do not significantly affect the loss function of distillation module.
We validate the effectiveness of our proposed AMS method through the experiments on range of datasets (CIFAR-10/100, SVHN, and TinylmageNet) and model architectures (PreActResNet- 18 and WideResNett34/28-11 of-the-art adversarial training methods, improving adversarial robustness accuracy across various datasets and architectures.
The experimental results suggest that our method can enhance state-
2 BACKGROUND
2.1 ADVERSARIAL TRAINING
Let D = {(x,, y: denote a C-classes training dataset with n samples from the original data distribution P. where x; E IRd is a natural sample in the d-dimensional input space and yı E (1...., .C} is its ground true label. Vanilla adversarial training can be formulated as minimux robust optimization problem (Madry et al. 2018):
min E Exp
max (x,,)
(x)o(x)) v/)
(I)
where ho: .Rd - C is a DNN classifier with parameter 0, (x': ys) is an adversarial example of (xi, v.). and B, (x = {x X - x:Ipp < <) is an adversarial region centered at x; with radius 0 under the E, norm-bounded perturbation (e.g., foc (Rice et al.] 2020) and l2 (Carlini et al.) 2019) norms), [(he(x)), y:) is the cross-entropy loss function on the adversarial example (x,, yi). As explained in (Madry et al.) 2018), the inner maximization problem is to find the worst-case samples for the given model, while the outer minimization problem aims to train a model robust to adversarial examples. To solve this problem, Madry et al. (2018) employed multi-step "Projected
3

---

Published as a conference paper at ICLR 2025
Gradient Descent (PGD)" method, which starts at randomly initialized point x² in (x e) and iteratively updates the adversarial example by
x (x.() (x] + a sign (Vx, C (hx(22), 9.)))
(2)
where a is the step size, (I(.) is the projection operator. We refer to this inner maximization problem with K steps "PGD-K" Numerous studies have been proposed based PGD, including (Jia eet al. 2022 Kannan et al. 2018 Chen et al., 2022 Rice et al. 2020).
Besides PGD, another representative AT method is TRADES (Zhang et al.) 2019b), which balances the trade-off between robust and clean accuracy by minimizing a different adversarial loss
min Ex,eP [(ha((x)) wi) + 0
max (x..)
Dko,
|| he hx())))
(3)
where (- || .) is the Bullbak--Leibler divergence. The first term in Eq. contributes to clean accuracy, and the second term with Syper--arametee 3 (set 3 = 6 as default) can be seen as a reg- ularization for adversarial robustness that balances the outputs of clean and adversarial examples. Furthermore, a substantial body of related work based on TRADES has been proposed, includ- ing (Wu et al. 2020 Cui al. 2021 Jin et al. 2023).
In addition to PGD-based and TRADES RRseeS-based methods, several enhancements for AT have been made by employing other strategies, including mussuperviseddssf--speeeiii
learning (Alayrac al. 2019 Chen et al.] 2020b a Naseer et al.) 2020), data augmentation (Lee et al., [2200] Rebuffi et al,
2021b Gowal et al. 2021 Rice al. 20200] [u al.) 2020), and generative model (Dong al. Pang et al., 2022; Wang et 2023). 3.. Furthermore, due to the high computational cost of AT, various efforts have been made to accelerate the training process. These include reusing computa-
2020
tions Shafahi et al. 2019 /Zang et al. (019a) and adopting one-step training methods (Wong et al.,
2020
Jia et al.
2024 Vivek Babu 2020).
2.2
CONTINUAL LEARNING AND KNOWLEDGE DISTILLATION
Continual learning (CL) (Wang al. 2024 Lopez-Paz & Ranzato
2017). a rapidly developing
field in deep learning, aims to develop systems capable of learning continuously from sequential or streaming data while retaining previously acquired knowledge. encounters is "catastrophic forgetting" (McClelland et al. 1995 McCloskey & Cohen 1989), where DNNs forget old knowledge upon learning new information. Various strategies have been proposed to mitigate this phenomenon in CL., primarily focusing on regularization-basel methods and replay- based methods (De Lange et al. (022I) Roughly speaking, the replay-based methods method uti- lizes reservoir sampling (Vitter 1985) to maintain historical data (c.g., ER (Chaudhry et al., 2019). GCR (Tiaari et af., 2022) and DGC-ER (Lin et al.) 2024)) or logits (e.g., DER (Buzzega et al,, 2020h) in the memory buffer, then extract new incoming training data with random samplings for learning the current task. Another way to solve continual learning is through deliberately designed regularizer terms. For example, the method EWC (Kirkpatrick al., (017) adds a quadratic penalty in the loss function, which penalizes the variation of each parameter depending on its contribution to old tasks: Lwf (Li & Hoiem 2017) learns new training samples while using their predictions from the output head of the old tasks to compute the distillation loss.
significant challenge that CL
method that distills knowledge from larger deep neural network into small network (Li et al. 2020 Polino et al. 2018 Zhang et al.. 2018a b). KD learning schemes (Gou al.) (02T) can be divided into three main categories based on whether the teacher model is updated simultaneously with the student model or not: offline distillation (Hin- ton et al. 2015 Mirzadeh et al. 2020), online distillation (Zhang et al. [01 8b Wu & Gong 2021). ) and self self-distillation (Zhang et al. [2019] Hou et al. 2019; Yang et al. 2019). In self-distillation, the same network serves as both the teacher and the student models, enabling direct application in CL
Knowledge distillation (KD) (Hinton al., 2015)
scenarios, e.g., iCaRL (Rebufhi et al. 2017) and Lwf (Li & Hoiem 2017).
3 CATASTROPHIC FORGETTING IN AT
In this section, we first explain that why AT can be regarded as "CL-style" problem, from the per- spective of data distribution shifts during the training process. Then, we investigate the phenomenon of forgetting in AT through empirical study.
Regard AT as a CL-style problem. We are aware that AT involves the model adapting to contin- ually changing distributions of adversarial samples, and will explain that it shares some similar
4

---

Published as a conference paper at ICLR 2025
properties with CL scenario that adapts the model to new tasks or changing environments. Recall from Eq. that vanilla adversarial training is formulated as a minimax robust optimization prob- lem, utilizzig the original data distribution P and DNNs classifier hg. In each epoch of the training process, the parameter is updated by a set of adversarial samples, denoted by pade, based on the inner maximization problem, where pade can be viewed distribution by adding adversarial perturbations to the original distribution. To distinguish the differ- ent pados at different epochs, we add subscript to it; for example, Padv denotes the sample pads at the f-th epoch. The knowledge required to solve the outer minimization problem (i.e., minimizing the loss L on Prdd)) different from the knowledge needed for optimizing on Pr-y. In other words, the tasks training on P and pade are in fact different. The sequence of changing adversarial samples (Pad")?", induce a sequence of tasks for the classifier heg Since the inner maximization optimal problem epoch t can only generate the sample Prdde the model at this current epoch cannot revisit the knowledge from previous tasks, which is similar to a CL scenario.
sample set selected from an implicit
Does catastrophic forgetting exist in AT? To answer this question, we further conduct an empirical study to verify the forgetting phenomenon in AT. We first define "stage" and "stage sequence" within the context of AT, which are similar "task" and "task sequence" in CL. Let Peder represent the adversarial samples generated in the t-th epoch. Suppose there are I epochs in total. We divide the training process into multiple stages with each stage containing m epochs:
Pado
padv m
padv Padv
pady
m+1
Thus, we define the stage sequence as S = (S.....,(EE)
11Z1).
The experiment is designed as fol-
lows. We first train a model by an AT method and collect the adversarial samples. Specifically, collect correctly classified adversarial samples in the s-th stage and form a new "correctly classified adversarial dataset", which is denoted by Cadv, This process continues until the end, producing collection of correctly classified adversarial dataset denoted as Cade = {Crde, ,..., C,á 1 }- After
completing the training process. we evaluate the classification accuracy on each collected dataset Code Meanwhile, we employ the Final Average Accuracy (FAA) (Douillard et al. 2020 Hou et al.,
2019) and Final Forgetting (FF) (Chaudhry et al., 2019) to assess the forgetting degree. These two metrics are both widely used for continual learning.
Table I: Verification results of the forgetting phenomenon in adversarial training across different datasets and perturbation threat models. The total epochs are evenly divided into 5 stages. Note that the "adversarial datasets" collected at each stage are initially classified correctly, resulting in an initial accuracy of 100% for each dataset. Let dk.y (for k 2 j) denote the classification accuracy evaluated on the testing set of the stage j after learning stage k. The FAA is defined as FAA & + Σj= a: the FF is defined as FF --T EJ= (ma.xe(l.....TTi 1I)..,, - ar.j). These two metrics quantify the degree of forgetting, where higher FAA and lower FF indicate less forgetting of previously learned knowledge.
Robust Test Accuracy (%)
Dataset
Norm stage
FAA (%)
FF ( (%)
stage
stage
stage
stage 5
Eoo 88.36
91.43
89.21
90.65
90.21
89.97
10.03
CIFAR-10
89.34
90.67
90.56
91.84
90.63
90.61
9.39
8. 77.74
87.87
88.33
89.25
89.35
86.51
13.49
C1FAR-100
f2 78.89
89.34
88.21
88,50
90.17
87.02
12.98
8oo 88.61
92.52
92.45
92.91
90.18
91.33
8.67
SVHN
E 86.72
92.66
90.57
90.47
91.81
90.45
9.55
Exo 58.98
81.53
80.70
81,05
75.83
75.62
24.38
TinyImageNet
82
65.18
81.85
79.70
82.37
82.73
78.37
22.63
In Table I we observe that forgetting occurs across a variety of datasets (e.g.. SVHN 1I, CIFAR- 100 and TinylmageNet), perturbation threat models (e.g., & and fxo) and different stages of training, indicating that might be a general property in adversarial training. We can find a significant gap between the robust test performances at the stage when the datasets are collected
5

---

Published as a conference paper at ICLR 2025
and the final robust test performance at the end of training. Notably, under the Lo threat model, FAA decreased significantly from 100% to 86.51% for CIFAR-100 and from 100% to 75.62% for TinyImageNet. We also investigate the phenomenon of forgetting in adversarial training via some other supplemental experiments Appendix B.1 and B.2
OUR METHODOLOGY
For case of understanding, we briefly introduce our high-level idea below, and then elaborate on each technical part.
Stages
Targel Label
Reweighting based Loss Convetion (w)
Old Legins
Adayrive Multi-teach hell--tccchrr Self- distiliation Law (U.rn))
Snop Grudiient
Logits
Current
Muti teacher aif--ditillatioon Loss (l.,.)
Te
Curreut Logis
Altack
Training Loss (Pagts)
Time Stump
Model for Curvent
Target Label
Backpaspagation
Framework of our AMS. The upper module (in yellow block) consists of the model for previous stages, with the aim of maintaining knowledge of previous stages. The lower module (in red block) the vanilla adversarial training process.
Figure 2:
Overview. Our goal is to design a method to enhance robust accuracy through alleviating forgetting. We consider using multi-teacher self-distillation technology, which combines multi-tcacher distil- lation architecture and self-distillation algorithm to transfer the subtle feature information learned from previous "stages". However, since adversarial training models usually do not have high classifi- cation accuracy (say, > 90%), which may lead transferring more previous incorrect information the subsequent "stages". To alleviate the negative effects caused by the incorrect information from previous "stages", we propose an adaptive sample reweighting mechanism called "Reweighting- based Loss Correction" to correct the loss function in distillation module. The overall framework of our method can be seen in Figure and the complete algorithm is presented in Algorithm [
Multi-teacher Self-distillation. A direct way to employ multi-teacher self-distillation technique to treat each training epoch in the adversarial training process separate "stage". However, since the adversarial training process requires a large number of epochs (e.g., 200 as in (Madry) al. 2018)). directly applying this method could result in rapid increase in both training time and memory as training progresses. So we take every epochs of the training process as one stage, as illustrated Section before.
Now we integrate the multi multi-teacher self-distillation approach into the PGD framework to transfer knowledge learned from previous stages, thereby mitigating forgetting and improving the model's robust accuracy. Generally, assuming we have model ha, parameterized by O, at epoch within stage 8, the multi-teacher self-distillation approach can formulated as:
Lia (he (x() || he, (x()),
(4)
where ( (. -) is the Kullback-Leibler divergence. (0 represent the model parameters from all stages before stage s, and x; is an adversarial sample epoch t. Thus, the training objective of AT with multi-teacher self-distillation can be expressed
X
Ladv
Lms:
(5)

---

Published as a conference paper at ICLR 2025
Algorithm 1 Adversarial training with AMS
Input: Training datasets D (x,, 1 perturbation bound e, learning rate step size a, number of iterations K in inner optimization, network architecture parameterized by 0, memory buffer M, regularization parameter X, interval parameter m, and the number of epoch T. Output: Robust network with parameter Or
1: Initialize 0
2: 8 - 0, M <- 0
3: f 1........ T do
4: Sample x; from D
5: for = ......n.. do
6: x < x; 0.001 (0
7: for k = -....., do
8:
X (x.e)) (xf + sign (Vx, (hx(x)), u)))
9:
end for
10:
for 0g M do
11:
W [softmax(ho, (x()) /" Reweightng
braeeihh-g-baeedd based Loss Correction */
end for
13:
0.+1= <- 0; T E V(Lude + Ei mas j=1 Waj" DKI, (xf) ho, (x()))
14:
1 Adaptive Multi-teacher Self-distillation %/
15:
end for
16:
if mt m) == 0 then
17:
M - M U (0,} / Save model parameters
18:
++
19: end if
end for
where the regularization parameter À controls the plasticity-stabiliy trade-off. Too much focus on stability will hinder the model's s ability learn new adversarial samples, whereas excessive plasticity will lead to greater forgetting of previously learned knowledge. The fraction s-T is used to calculate the average of all distillation losses. The term "Ludo" is the adversarial loss of current epoch t and can be formulated as
Ladv
max L(ho, (x)).
(6)
(x,,,(x..e)
Ideally, we look for parameters that fit the current stage well while retaining the correct knowledge learned in the previous stages, thereby enhancing the model's ability to cope with various forms of perturbations. However, during knowledge distillation, large amount incorrect information made by previous stage models can be transferred to current stage model along with parameters. This occurs due to the low robust accuracy of AT models. For example, the best robust accuracy on CIFAR-10 is less than 60%.
Reweightng-based Loss Correction (RLC). To resolve the above "transferring-incorre information" problem, we propose a Reweighting--asse Loss Correction mechanism to adaptively eliminate the negative effects caused by "bad" teachers. Specifically, for a teacher model he, we first use the softmax function to compute the probability output of adversarial samples that are gen- erated in the current stage. Then, we extract the probability corresponding to the true class for each adversarial sample, denoted as w.j- This probability w., quantifies the degree of correct informa- tion that can be transferred from the teacher model. Therefore, for each adversarial sample x', we employ w. as distillation weight to correct the loss function L.s (see Eq.(4). Mathematically. the probability of an adversarial sample x with true label y/ can be expressed as:
Wijj = sgoftmax(ho (x))
As a consequence, we modify our approach to be an "adaptive" way, where the loss function of Eq. (4) is changed to be:
A
Lams
Wis DKL (ho, (x)) || he, (x)))
>j=1 Wi.j
f=1
7

---

Published as a conference paper at ICLR 2025
5 EXPERIMENTS
In this section, we first discuss the hyper-parameter sensitivity of our method, detailed in Section 5.1 We then evaluate the robustness on benchmark datasets in Section 5.2 Section 5.3 presents our findings that our method can mitigate the robust overfitting issue to certain degree. Furthermore, in Section[3.4] we compare our approach with various CL methods. Additional experimental results are explained in Section 5.5 Finally, we conduct the ablation study in Section 5.6
Experimental Setting. We default using PreActResNet-118 (He et al. (016) for & and E~ threat models on CIFAR-10/100 (Krizhevsky 2009). TinyImageNet and SVHN (Netzer et al., 2011). In addition, we also use WideReesNet-34-
(Zagoruyko & Komodakis, 2016) for CIFAR- 10/100 with
lo threat model (for a proper comparison to the results reported for TRADES (Zhang et al. 2019b).
and use WideRessNet-28- 10 for TinyImageNe (Tinylmagenet dataset is a subset of ImageNet (Deng train with the SGD optimizer with Nesterov momentum (Nes- terov 1983 0.9 using a batch size of 128, a step-wise learning rate initially at O. and weight decay 5 x 10 For SVHN, we use the same parameters except with starting learning rate of 0.01 in- stead. For TinylmageNet, we use the cyclic learning rate schedule with cosine annealing (Smith) & Topin 2019). where the initial learning rate is set to 0.2. We adopt the widely used adversarial training setting [Madry et al. 2018 Zhang et al. 2019b). For the foo threat model, we using radius c= 8/255 and step size a 2/255; For the Eg threat model, we using radius c 128/255 and step size a = 15/255. For vanilla adversarial training, the training examples are generated with 10 steps. We also adopt the widely used data augmentation technology, such as random horizontal flip and 32 x 32 random crop with 4-pixel padding. We report the test accuracy on the best checkpoint that achieves the highest robust validation accuracy under PGD-20. We implement and train our model with PyTorch on 64-bit Linux machine with 8 NVIDIA RTX A6000 Ada GPUs.
et al. ))09)). For CIFAR- 10/100,
Evaluation Setting. We evaluate the model robustness against wihte-box attacks, black-box attack and Auto Attack. For white-box attacks, we utilize PGD-20 and CW-20 (Carlini & Wagner 2017) to evaluate the trained models. For the black-box attacks (Papernot et al. (017). we generate ad- versarial examples by attacking local substitute model trained with the vanilla adversarial training method. These adversarial examples are then applied to the defense model to evaluate its perfor- mance. For the black-box attacks, utilize PGD-20 and CW-20 the attack methods. As one of the strongest attacks for evaluating model robustness, Auto Attack (Croce & Hein (0205) con- tains an ensemble of diverse attacks, including three wihte-box attacks (APGD-CE (Croce & Hein 2020b), APGD-DLR (Croce & Hein, (2020b) and FAB (Croce & Hein 2020a)) and one black-box attack (Square Attack (Andriushchenko al. 2020
5.1 SENSITIVITY OF HYPER- ENTER-PARE PARAMETER
we need to determine two parameters, the regularization parameter and the epoch interval parameter m. We conduct the numerical experiments to illustrate how these hyper-parameterr influence the performance of our algorithm. For the regularization parameter X, we train the model with À E {0.05. 0.1, 0.2, 0.3, 0.5, 1.0, 3.0}. For the interval parameter m, we train the model with m E {10, 20, 50..
In our proposed Algorithm !
we observe that as the regularization parameter À increases, both the clean accuracy and robust accuracy initially go up and then down. Taking both metrics into account, we take À = 0.5 as proper choice for our method. Meanwhile, we evaluate the interval parameters m comprehensively from four aspects: memory usage, time consumption, clean accuracy, and robust accuracy, illustrated in Figure in Appendix B.3 We find that
From Table in Appendix B.3
20 is an appropriate setting for our approach. So, in the following experiments, we set À = 0.5 and m = 20 as the default parameters.
5.2 COMPARISON WITH SOTA ON WIDERESNET
In Table 2. we compare our method with the state-of-the-arr methods on WideResNet across CIFAR- 10/100, SVHN and TinyImmgeNNe under the loo norm with e = 8/255. The state- offee---------rrrr methods
include AVMixup Lee et al. 2020), MART (Wang et al. 2019), ES (Rice et al. 2020). FAB (Zhang
ct al.
2020), LS (Pang et al. 2021). LBGAT (Cui et al. 2021). 1). O (Jin et al.
2022). RAT (Jin et al.
2023
AWP (Wu al. 2020), EMA (Gowal et al. 2020).
(Sehwag et al.
2022 (Gowal al.
8

---

Published as a conference paper at ICLR 2025
2021). (Rebuffi et al., (022 and Wang et al. )023). We integrate AMS with TRADES, (Sehwag et al. ).22). and (Wang et al. )023). and assess their performances in different datasets. The results Table suggest that our method consistently improves robust accuracy across different datasets and models. Especially for TinylmageNet under Auto Attack, our method achieves a robust test accuracy of 31.52%.
Table 2: Test accuracy (%) of the proposed methods and current state-of-the-arr methods on CIFAR- 10/100, SVHN and TinylmageNet under the &o norm with = 8/255. We highlight the best results means under PGD-10 attack. The
in bold. Note that means under PGD-40 attack, and numerical results of the baseline methods are quoted from their papers. "_" indicates that the result not reported in the original paper. The notations "(t A)" and "(L A)" indicate percentage increases and decreases compared to the baseline, respectively. The term "IM" denotes the utilization one million images generated by class-conditional Denoising Diffusion Probabilistic Model (Ho al.
2020
Dataset
Method
Architecture
Clean
PGD-20
CW-20
AVMixup
Lee cl
2020
WideResNet 14-10
92.56
59:75
$4.34
39.70
Gowal ct
2020
WideRcsNet-70-11
85.29
58.22"
$7.20
MART
Wang ct
2019
WidcRosNea-44-00
83.51
58.31
$4.33
51.10
WideResNet-44-20
FAB
Zhang
2020
Wid:ResNe-34-10
84.52
53.51
CIFAR-10
LS
Pang ct al
202
Wid:RcsNat-34-20
86.43
$7.91*
$4.39
Jin et
1022
WidcResNet-34-2
36.01
61.12
57.93
$5.90
Dia
WideResNett34-U
56.13
S
Jin
10223
WideResNet.3411
85,58
59.43
55.66
$3.93
M'S
WideResNet
85.83 (t 0,25)
61 29 (t 1.86)
57.78 (1 2.12)
$5.92 (f 1.59)
Wang et al.
2023
(IM)
WideResNer:34.1
91.18
68.11
65,20
63.31
Wang et
AMM
WideResNett34-1
90.79
0.39)
69.32 (1 1,21)
(1 0.83)
63.97 0.66)
TRADES hong
20196
WideResNe-:3-10
56.68
$4,49
TRADES+EMA (Gowal
2020
WidcResNet-4410
84.78
57.23
$5.12
53.76
TRADES+AMS
Wid:RcsNet-34-10
85,37 (1 0.72)
$8.76 (t 2.08)
$6.43 (1 1.94)
$4.31 (1 1.31)
TRADES+AWP (Wa al. 2020
WidcResNet-34-10
$4.99
59.67
57.41
56.17
TRADES-AWP-AMS
Wid-RoNNes34-10
0.22))
61.45 1.778)
(1 1.14)
57.36 1.19)
Gowal 2020
WideResNen-70-16
60.86
31.47"
30.03
Kchalh al MiMl (LM)
WiddekeNNet28-10
62.41
32.06
LBGAT Connet 2021
Wid:RcaNet-54-11
60,43
35.50
31,50
29.34
CIFAR-100
RAT
WideResNest
33.36
29.61
TRADES /hang
WideMosNen.5411
32.11
28.93
26
TRADES+EMA (Gowal
2020
WidcResNet-54-10
61,43
32.76
29.41
27.19
TRADES+AMS
Wid:ResNe::31-11
62.56 (1 2.34)
33.81
1.70)
30.47
1.54)
28.25 (1 1.35)
Schwag et al.
2022
(1M)
WidcResNNr:34-11
65.76
36.33
32.97
31.20
Schwag
(IMHAMS
WideResNen:34-0
(6.42
33.56 (1 0.50)
31,46 0.26)
et
M21
WideResNer
92.87
56.83
Gowal al
M21
(1M)
WideeResNe-28-10
94.15
60.90
SVIIN
Xeballi al
202
(1M)
Wid::ccsNat288-10
94.39
61.09
Wang al
(MM)
WidcResNett28-1
95.08
65.38
62.98
61.73
Wang
(1M)+MS
WideMesNet-22
95.26
1.29)
64.21 (1 1.23)
Gowal al.
2021
WideResNer:28-11
28-10
51,56
21.56
Tiny Image Net
Gowal et
212
(IM)
WideResNet:29.11
60.95
26.66
Wang et al.
2024
(M)
WidekesNett28-11
64,83
32.21
$1.65
30.76
Wang ct al.
2023
(IMAAMS
WideResNet-28-10
64.20 (1 0.63).
33.46 (7 1,25)
32.54 (1 0.89)
31.52 (1 0.76)
5.3 ROBUST OVERFITTING
In our experiments, we also observe an interesting phenomenon that our method can alleviate robust overfitting (Rice et al., 2020), though it is not the main focus in our paper. As common issue in adversarial training, the property of adversarial robust overfitting is that after the first and second learning rate decay, further training will continue to substantially decrease the robust test accuracy. To illustrate this issue, we show the comparisons on test accuracy with the basclines in Appendix B.4 (Table from both the best checkpoint, which achieves the highest robust test accuracy under PGD. 20 (denoted as "Best"), and the final checkpoint (denoted as "Final"), as well as the differential larger value of "Diff" indicates a more
between these two checkpoints (denoted as "Diff"). pronounced occurrence of robust overfitting. We observe that the accuracy difference between the best and final test results from our method is reduced to around 0.5%, while the gaps for TRADES significantly larger, at 3.28% under the PGD-20 attack on CIFAR-10. This indicates that method has the potential to mitigate the robust overfitting issue. Meanwhile, we also compared with some related works that mitigate robust overfitting in Appendix [.4 (Table 7)

---

Published as a conference paper at ICLR 2025
5.4 COMPARISON WITH DIFFERENT CL METHODS IN APPENDIX B.5
In Table 8 and Table 9 we compare our method against various CL methods, such as replay-based methods (e.g.. ER (Chaudhry et al. 2019), DER (Buzzega al., 2020 DGC-ER (Lin al. 2024). GCR (Tiwari et al. (2022), rggularization-baae
methods (e.g.. EWC (Kirkpatrick et al. 2017
Lwf (Li Hoiem (017)). The experimental parameter settings for these CL methods are provided in the Appendix B.S From the results in Table 8 we observe that almost all the CL methods achieve certain extent improvements on robust accuracy. For example, Lwf (Li & Hoiem 2017) has the increased best test robust accuracy from 47.70% to 50.45% under Auto Attack. Compared to other CL methods, our approach has larger improvement robust accuracy, while maintaining clean accuracy without decline; for example, the best test robust accuracy increases from 47.70% to 51.63% (+3.93%) under Auto Attack.
5.5 OTHER EMPIRICAL RESULTS IN APPENDIX B.6
We also compare our method with PGD MMadry al. 2018) using PreActResNet-18 on CIFAR- 10/100 and SVHN, under the & norm with = 128/ 255, as detailed in Table I0 Additionally, we evaluate the performance of our method under black- box attacks in Table IT These results also indicate that our method can enhance robust accuracy to certain extent, while maintaining clean accuracy without any decrease. Furthermore, as illustrated in Table we have empirically validated the effectiveness of our method in mitigating forgetting. Specifically, compared to the validation results of the vanilla AT method presented in Table [ the results our approach reveal significant improvements in FAA and reductions in FF. Finally, we apply AMS to other structures (VGG16 [Simonyan] 2014) and VIT (Dosovitskiy et al. (202T)) with normal adversarial training; the results shown in Table 13 illustraee that our AMS method is effective these architectures well.
5.6 ABLATION EXPERIMENT.
Given that our proposed method incorporates a "Reweighting- Loss Correction" (RLC) tech- nique, a natural question arises: how effective is it in improving the final performance? To address this question, we conducted an ablation experiment, and the results are shown in Table 3 The re- sults demonstrate that the use of RLC can enhance robust accuracy to a certain extent. For example, when RLC is removed, the robust accuracy on CIFAR- 100 decreases from 28.05% to 27.62% with TRADES+AMS, and from 28.31% % to 27. 27.58% with PGD- 10+AMS, highlighting the contribution of the RLC component to improving adversarial robustness.
Table Test accuracy (%) of TRADES+ and 1GD-10+AMS with or without (w/o) the "RLC" term under the Lo norm with 8/255 based on the PreActResNet-18 architecture.
Method TRADES
CIFAR-10 (clean)
CIFAR-10 (AA)
CIFAR-100 (clean) CIFAR-100 (AA)
84.86
51.22
60.03
28.05
TRADES w/o RLC
84,72
50.08
60.11
27.62
PGD-I0+AMS
84.45
51.63
58.87
28.31
TED-I0+AMS w/o RLC
85.66
50.31
58.83
27.58
6 CONCLUSION AND OUTLOOK
In this paper, we address the problem of AT from a novel perspective by linking it to the forget- ting phenomenon in CL. We first demonstrate that the phenomenon of forgetting indeed occurs in AT. Following this observation, we propose a novel method called Adaptive Multi-teacher Self- distillation (AMS), which employs carefully designed adaptive regularizer to mitigate forgetting by aligning model outputs between successive stages. Our approach can be seamlessly integrated with several existing AT methods, leading to substantial improvements our experimental results indicate that our method enjoys another benefit, where it can significantly alleviates the issue of robust overfitting for AT. So we conjecture that robust overfitting might be partly caused by the forgetting issue. As the future work, we think it is deserved to explore the connection between forgetting and robust overfitting in greater depth, from both the theoretical and empirical aspects.
robust accuracy. In addition,
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
The authors would like thank the anonymous reviewers for their valuable comments and sugges- tions. This work was partially supported by the National Natural Science Foundation of China (No. 62272432, No. 62432016), the National Key Research and Development Program China (No. 2021YFA1000000) and the Natural Science Foundation of Anhui Province (No. 2208085MF163).
REFERENCES
Jean-Bare Sann-Baptiste Alayrac, Jonathan Uesato, Po-Sen Huang, Alhussein Fawzi, Robert Stanforth, and Pushmeet Kohli. Are labels required for improving adversarial robustness? Advances Neural Information Processing Systems, 32, 2019.
Maksym Andriushchenko, Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square at- tack: a query-efficient black-box adversarial attack via random search. European conference computer vision, pp. 484 501.-00..50 Springer, 2020.
Matteo Boschini, Lorenzo Bonicelli, Pietro Buzzega, Angelo Porrello, and Simone Calderara. Class- incremental continual learning into the extended der-verse. IEEE Transactions on Pattern Analy- sis and Machine Intelligence, 2022.
Jacob Buckman, Aurko Roy, Colin Raffel, and Ian Goodfellow. Thermometer encoding: One hot way to resist adversarial examples. In International conference on learning representations, 2018.
Pietro Buzzega, Matteo Boschini, Angelo Porrello, Davide Abati, and Simone Calderara. Dark ex- perience for general continual learning: strong, simple baseline. Advances in neural information processing systems, 33:1 1155599011993 2020.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 ieee symposium on security and privacy (sp). pp. 59-57. Iece, 2017.
Nicholas Carlini, Anish Athalye, Nicolas Papernot, Wieland Brendel, Jonas Rauber, Dimitris Tsipras, Ian Goodfellow, Aleksander Madry, and Alexey Kurakin. On evaluating adversarial robustness. arXiv preprini urxiv::902.00 1902.06702.. 2019.
Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny. Thalaiyasingam Ajanthan, Puneet K Dokania, Philip HS Torr, and Mare' Aurelio Ranzato. On tiny episodic memories in continual learning. arXiv preprint arXiv: 1902. 10486, 2019.
Chenyi Chen, Ari Seff, Alain Kornhauser, and Jianxiong Xiao. Deepdriving: Learning affordance for direct perception in autonomous driving. Proceedings the IEEE international conference computer vision, pp. 2722-2730, 2015.
Jinghui Chen, Yu Cheng, Zhe Gan, Quanquan Gu, and Jingjing Liu. Efficient robust training via backward smoothing. In Proceedings of the AAAI Conference on Artificial Intelligence, vol- ume 36, pp. 6222 6230, 2022.
Kejiang Chen, Yuefeng Chen, Hang Zhou, Xiaofeng Mao, Yuhong Li, Yuan He, Hui Xue, Weim- ing Zhang, and Nenghai Yu. Self-supervised adversarial training. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2218-2222. IEEE, 2020a.
Tianlong Chen, Sijia Liu, Shiyu Chang, Yu Cheng, Lisa Amini, and Zhangyang Wang. Adversarial robustness: From self-supervised pre-training to fine-tuning. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 699-708, 2020b.
Francesco Croce and Matthias Hein, Minimally distorted adversarial examples with fast adaptive boundary attack. International Conference on Machine Learning, pp. 2196-2205. PMLR,
2020a.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks. In International conference on machine learning, pp. 2206- 2216. PMLR, 2020b.
11

---

Published as a conference paper at ICLR 2025
Jiequan Cui, Shu Liu, Liwei Wang, and Jiaya Jia. Learnable boundary guided adversarial training. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 15721-15730, 2021.
Matthias De Lange, Rahaf Aljundi, Mare Masana, Sarah Parisot, Xu Jia, Aleš Leonardis, Gregory Slabaugh, and Tinne Tuytelaars. A continual learning survey: Defying forgetting in classification tasks. IEEE transactions on pattern analysis and machine intelligence, 44(7):3366-33 -38853356,633,, 2021.
Enes Dedeoglu, Himmet Toprak Kesgin, and Mehmet Fatih Amasyali. A robust optimization method for label noisy datasets based on adaptive threshold: Adaptive-k. Frontiers Comput. Sci., 18(4),
2024.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEFE conference computer vision and pattern recognition, pp. 248-25822522 Icee, 2009.
Jacob Devlin, Ming-Woo Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre- are-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Association for Computational Linguistics, 2019.
Short Papers), pp. 4171-4188
Yinpeng Dong, Zhijie Deng, Tianyu Pang, Jun Zhu, and Hang Su. Adversarial distributional training for robust deep learning. Advances in Neural Information Processing Systems, 33:8270-8283, 2020.
Yinpeng Dong, Ke Xu, Xiao Yang. Tianyu Pang. Zhijie Deng, Hang Su, and Jun Zhu. Exploring memorization in adversarial training. In International Conference on Learning Representations, 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image worth 16x16 words: Transformers for image recognition at scale. In 9th International Conference Learning Representutions, ICLR 2021, Virtual Event, Austria, May 3-7, 2021, 2021.
Arthur Douillard, Matthieu Cord, Charles Ollion, Thomas Robert, and Eduardo Valle. Podnet: Pooled outputs distillation for small-tasks incremental learning. In Computer vision-ECCV 2020: 16th European conference, Glasgow, UK, August 23-28, 2020, proceedings, part XX 16, pp. 86- 102. Springer, 2020.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and hamnessing adversarial examples. In International Conference on Learning Representations, 2015.
Jianping Gou, Baosheng Yu, Stephen J Maybank, and Dacheng Tao. Knowledge distillation: A
survey. International Journal of Computer Vision (9(6))1789-188) 2021.
Sven Gowal, Chongli Qin, Jonathan Uesato, Timothy A. Mann, and Pushmeet Kohli. Uncovering the limits of adversarial training against norn-bounded adversarial examples. CoRR, abs/2010.03593. 2020.
Sven Gowal, Sylvestre-Alvise Rebuffi, Olivia Wiles, Florian Stimberg, Dan Andrei Calian, and Timothy A. Mann. Improving robustness using generated data. In Advances in Neural Information 2021.
Processing Systems 34, pp.
Chuan Guo, Mayank Rana, Moustapha Cisse, and Laurens van der Maaten. Countering adversarial images using input transformations. In International Conference Learning Representations, 2018.
Sidharth Gupta, Parijat Dube, and Ashish Verma. Improving the affordability of robustness training for dnns. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 780-781, 2020.
12

---

Published as a conference paper at ICLR 2025
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey E. Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the knowledge in neural network. arXiv preprint (rti::5503.25555 2015.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-653 68511 2020.
Yuenan Hou, Zheng Ma, Chunxiao Liu, and Chen Change Loy. Learning lightweight lane detection cnns by self attention distillation. In Proceedings of the IEEE/CVF international conference computer vision, pp. 1013-1021, 2019.
Lang Huang, Chao Zhang, and Hongyang Zhang. Self-adaptive training: beyond empirical risk minimization. Advances in neural information processing systems, 33: 13:19565-13376 2020.
Xiaojun Jía, Yong Zhang, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. Las-at: adversarial training with learnable attack strategy. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13398-13400 2022.
Xiaojun Jia, Yong Zhang, Xingxing Wei, Baoyuan Wu, Ke Ma, Jue Wang, and Xiaochun Cao. Improving fast adversarial training with prior-guided knowledge. IEEE Transactions Patlern Analysis and Machine Intelligence, 2024.
Gaojie Jin, Xinping Yi, Wei Huang, training with second-order statistics of weights. In Proceedings of the IEEE/CVF conference computer vision and patlern recognition, pp. 15273-15283, 2022.
Sven Schewe, and Xiaowei Huang. Enhancing adversarial
Gaojie Jin, Xinping Yi, Dengyu Wu, Ronghui Mu, and Xiaowei Huang. Randomized adversarial training via taylor expansion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16447-16457, 2023.
Harini Kannan, Alexey Kurakin, and Ian Goodfellow. Adversarial logit pairing. arXiv preprint arXiv:1803.06373 2018
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Grabska-Barwar and Agnieszka. Overcom- ing catastrophic forgetting in neural networks. Proceedings of the national academy of sciences, 114(13):351-33566 2017.
Alex Krizhevsky. Learning multiple layers features from tiny images, Master's thesis, University of Tront, 2009.
Yann LeCun, Yoshua Bengio, and Geoffrey Hinton. Deep learning. nature, 521(7553):436 2015.
Sachyung Lee, Hyungyu Lee, and Sungroh Yoon. Adversarial vertex mixup: Toward better adver- sarially robust generalization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 272-281, 2020.
Tianhong Li, Jianguo Li, Zhuang Liu, and Changshui Zhang. Few sample knowledge distillation for efficient network compression. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14639-14647, 2020.
Zhizhong Li and Derek Hoiem. Learning without forgetting. IEEE transactions on pattern analysis and machine intelligence, 40(12):2935-294 2017.
Weichen Lin, Jiaxiang Chen, Ruomin Huang, and Hu Ding. An effective dynamic gradient calibra- tion method for continual learning. In Forty-first International Conference Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024, 2024.
David Lopez-
Paz Paz and Mare' Aurelio Ranzato. Gradient episodic memory for continual learning. Advances in neural information processing systems, 30, 2017.
13

---

Published as a conference paper at ICLR 2025
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018.
Pratyush Maini, Eric Wong, and Zico Kolter. Adversarial robustness against the union of multiple perturbation models. In International Conference on Machine Learning, pp. 6640-6650. PMLR, 2020.
James L McClelland, Bruce L McNaughton, and Randall C C''Reilly, Why there are complementary learning systems in the hippocampus and neocortex: insights from the successes and failures of connectionist models of learning and memory. Psychological review, 102(3):419, 1995.
Michael McCloskey and Neal Cohen. Catastrophic interference in connectionist networks: The sequential learning problem. In Psychology of learning and motivation, volume 24, pp. 109-165. Elsevier, 1989.
Seyed Iman Mirzadeh, Mehrdad Farajtabar, Ang Li, Nir Levine, Akihiro Matsukawa, and Hassan Ghasemzadeh. Improved knowledge distillation via teacher assistant. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 5191-5198, 2020.
Muzammal Nascer, Salman Khan, Munawar Hayat, Fahad Shahbaz Khan, and Fatih Porikli. A self- supervised approach for adversarial robustness. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 262-271, 2020.
Yurii Nesterov. A method for solving the convex programming problem with convergence rate (1/k2). In Dokl akad nauk Sssr, volume 269, pp. 543, 1983.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, Ng, and Andrew Y. Read- ing digits in natural images with unsupervised feature learning. In NIPS workshop on deep learn- ing and unsupervised feature learning, volume 2011, pp. Granada, 2011.
Tianyu Pang, Xiao Yang, Yinpeng Dong, Hang Su, and Jun Zhu. Bag of tricks for adversarial training. In International Conference on Learning Representations, 2021.
Tianyu Pang, Min Lin, Xiao Yang. Jun Zhu, and Shuicheng Yan. Robustness and accuracy could be reconcilable by (proper) definition. International Conference on Machine Learning, pp- 17258-17277. PMLR, 2022.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM Asia conference computer and communications security, pp. 506-519, 2017.
Omkar Parkhi, Andrea Vedaldi, and Andrew Zisserman. Deep face recognition. In BMVC 2015- Proceedings of the British Machine Vision Conference 2015. British Machine Vision Association, 2015.
Antonio Polino, Razvan Pascanu, and Dan Alistarh. Model compression via distillation and quan- tization. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings, 2018.
Sylvestre-Alviss Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H Lampert. icarl: Incremental classifier and representation learning. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp. 2001-2010, 2017.
Sylvestre-Alvise Rebuffi, Sven Gowal, Dan A. Calian, Florian Stimberg, Olivia Wiles, and Timothy A. Mann. Fixing data augmentation to improve adversarial robustness. CoRR, 00.22103.01946. 2021a.
Sylvestre- Alvise Rebuffi, Sven Gowal, Dan Andrei Calian, Florian Stimberg, Olivia Wiles, and Timothy A Mann. Data augmentation can improve robustness. Advances in Neural Information Processing Systems, 34:29935-29944 2021b.
Leslie Rice, Eric Wong, and Zico Kolter. Overfitting in adversarially robust deep learning. In International conference machine learning, pp. 8093-8104, PMLR, 2020.
14

---

Published as a conference paper at ICLR 2025
Vikash Sehwag. Saced Mahloujifar, Tinashe Handina, Sihui Dai, Chong Xiang, Mung Chiang, and Prateek Mittal. Robust learning meets generative models: Can proxy distributions improve ad- versarial robustness? In International Conference Learning Representations, 2022.
Ali Shafahi, Mahyar Najbi,, Mohammad Amin Ghiasi, Zheng Xu, John Dickerson, Christoph Studer. Larry S Davis, Gavin Taylor, and Tom Goldstein. Adversarial training for free! Ad- vances in neural information processing systems, 32, 2019.
Karen Simonyan. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
Lestie N Smith and Nicholay Topin. Super-convergence: Very fast training of neural networks using large learning rates. In Artificial intelligence and machine learning for multi-domain operations applications, volume 1006, pp. 369-386. SPIE, 2019.
Yang Song, Taesup Kim, Sebastian Nowozin, Stefano Ermon, and Nate Kushman. Pixeldefend: Leveraging generative models to understand and defend against adversarial examples. In Inter- national Conference on Learning Representations, 2018.
Zhicheng Sun, Yadong Mu, and Gang Hua. Regularizing second-order influences for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 20166-20175, 2023.
Christian Szegedy. Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Rishabh Tiwari, Krishnateja Killamsetty, Rishabh Iyer, and Pradeep Shenoy. Ger: Gradient coreset based replay buffer selection for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 99-108, 2022.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9111), 2008
Jeffrey S Vitter. Random sampling with a reservoir. ACM Transactions on Mathematical Software (TOMS), 11(1):37-57, 1985.
BS Vivek and R Venkatesh Babu. Single-step adversarial training with dropout scheduling. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 947-956. IEEE, 2020.
Liyunn Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: theory, method and application. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
Yisen Wang, Difan Zou, Jinfeng Yi, James Bailey, Xingjun Ma, and Quanquan Gu. Improving adversarial robustness requires revisiting misclassified examples. In International conference on learning representations, 2019.
Zekai Wang, Tianyu Pang, Chao Du, Min Lin, Weiwei Liu, and Shuicheng Yan. Better diffusion models further improve adversarial training. In International Conference Machine Learning, pp. 36246-36263. PMLR, 2023.
Eric Wong, Leslie Rice, and J Zico Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2020.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener- alization. Advances in neural information processing systems, 33:2958-209 29699, 2020.
Guile Wu and Shaogang Gong. Peer collaborative learning for online knowledge distillation. In Proceedings of the AAAI Conference on artificial intelligence, volume 35, pp. 10302-10310, 2021.
15

---

Published as a conference paper at ICLR 2025
Cihang Xie, Jianyu Wang, Zhishuai Zhang. Zhou Ren, and Alan Yuille. Mitigating adversarial effects through randomization. In International Conference on Learning Representations, 2018.
Chenglin Yang, Lingxi Xie, Chi Su, and Alan L Yuille. Snapshot distillation: Teacher-student optimization in one generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2859-2868, 2019.
Yuting Yang, Pei Huang, Juan Cao, Jintao Li, Yun Lin, and Feifei Ma. A prompt-based approach to adversarial example generation and robustness enhancement. Frontiers Comput. Sci., 18(4),
2024.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Richard C. Wilson, Edwin R. Hancock, and William A. P. Smith (eds.), Proceedings of the British Machine Vision Conference 2016, BMVC 2016, York, UK, September 19-22, 2016. BMVA Press, 2016.
Dinghuai Zhang, Tianyuan Zhang, Yiping Lu,, Zhanxing Zhu, and Bin Dong. You only propagate once: Accelerating adversarial training via maximal principle. Advances in neural information processing systems, 32, 2019a.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent EI Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International conference on machine learning, pp. 7472-7482. PMLR, 2019b.
Jingfeng Zhang, Xilie Xu, Bo Han, Gang NIu, Lizhen Cui, Masashi Sugiyama, and Mohan Kankan- halli. Attacks which do not kill training make adversarial learning stronger. In International 11287 PMLR, 2020.
conference on machine learning,
Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma. Be your own teacher: Improve the performance of convolutional neural networks via self distillation. In Proceedings of the IEEF/CVF international conference computer vision, pp. 3713-3722, 2019c.
Xiangyu Zhang, Xinyu Zhou, Mengxiao Lin, and Jian Sun. Shufflenet: An extremely efficient convolutional neural network for mobile devices. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6848-6856, 2018a
Ying Zhang, Tao Xiang. Timothy M Hospedales, and Huchuan Lu. Deep mutual learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4320-4328, 201 8b.
16

---

Published as a conference paper at ICLR 2025
A ALGORITHM
In this section, we provide the vanilla AT algorithm (Madry et al. 2018 Zhang et al. 2019b) in Subsection A. Algorithm We also summarize the continual learning combine adversarial training implementations, such as replay-based method (i.e., DER (Buzzega et al. 2020)). regularization-
based method (i.e., Lwf (Li & Hoiem 2017) in Subsection A.2 and Subsection A.3
respectively.
A.1 VANILLA AT ALGORITHM
In the following, provide the vanilla adversarial training Algorithm 2
Algorithm Vanilla Adversarial Training
Input: Training datasets D = (x,, Dr) perturbation bound learning rate step size number of epochs in inner optimization, network architecture parameterized by 0
Output: Robust network with parameter 0r
1: Initialize 0
2: 1, T do
3: Sample x; from D
4: for 1...., n do
5: x <- x; - 0.001 (N
6: for k = =1,... -,.... K do
7:
x;- = (x.e( (x) + a sign (Vx, (ho(x)), v)))
8:
end for
9:
0,+1 < 0; T D Vade
end for
11: end for
A.2 ADVERSARIAL TRAINING WITH DER ALGORITHM
Adversarial training with DER. Dark Experience Replay (DER) is a replay-based continual leam- ing method that aligns the network's ' logits sampled throughout the optimization trajectory and stored memory buffer, thereby promoting consistency with its previous behavior. Specifically, the al- gorithm employs the Reservoir Sampling strategy (Vitter 1985) for buffer insertion. Below, we present Algorithm which integrates DER with AT.
A.3 ADVERSARIAL TRAINING WITH LWF ALGORITHM
Adversarial training with Lwf. Learning without Forgetting (Lwf) is regularization--asee con- tinual learning method that trains the network exclusively on new task data while preserving its original capabilities. Below, present Algorithm 4 which integrates Lwf with AT.
17

---

Published as a conference paper at ICLR 2025
Algorithm 3 Adversarial training with DER
Input: Training datasets (x,, yi y:) perturbation bound e, learning rate т, step size a, number of epochs K in inner optimization, network architecture parameterized by 0, memory buffer M, regularization parameter À
Output: Robust network with parameter Or
Initialize 0
2: M 0
3: for ,.... T do
4: Sample x, from D
5: for = I.....n do
6:
xf <- x; 0.001 (O, I)
7:
for k = -....., do
8:
X- (x..e) (xl + a- sign (Vx, (ha(x)))
9:
end for
(x), )g) sample((
11:
2; f hog (x )
12:
reg 2|z he ( ||
0 - 0 T
(Ladv + reg)
14: M < reservior (M, 2, v))
(J,,
15:
end for
16: end for
Algorithm Adversarial training with Lwf
Input: Training datasets = (x,, . perturbation bound e, learning rate т, step size a, number of epochs K in inner optimization, network architecture parameterized by 0, regularization parameter
Output: Robust network with parameter 0r
1: Initialize 0
2:
fo 1,.., T do
3: Sample x; from D
4: for I..... do
5:
x <- x; - 0.001 (O,
6:
for l 1...., do
7:
x [I (x. k-1 a sign (Vx, L (ho ).))))
8:
end for
9:
Live = DKI.
(ho,-) DKI,(h (x:) (x!))
10: 0,41 <- 0, T Σい V(Ladu + Livil
11: end for
12:
end for
18

---

Published as a conference paper at ICLR 2025
B FULL EXPERIMENTS
In this section, we first introduce another method to verify catastrophic forgetting in adversarial training (AT) Subsection B.1 We then employ t-SNE analysis (Van der Maaten & Hinton 2008) to visually demonstrate the dynamic nature of adversarial sample distributions during training in Subsection B.2, Additionally, we present the experimental results for the sensitivity of hyperparam- eters in Subsection B.3] for robust overfitting in Subsection [.44 and for continual learning (CL) in
Subsection B.5
Finally, we provide other empirical details and results in Subsection B.6
B.1 ANOTHER METHOD TO VERIFY CATASTROPHIC FORGETTING
In this part, we introduce another method verify the phenomenon of forgetting in adversarial training. The specific experimental process is as follows:
Step 1. We train model by an AT method (e.g.. PGD TRADES) benchmark dataset. We divide the training process into stages with each stage containing m epochs. In the final epoch of the s-th stage, collect all adversarial samples and form new "adversarial dataset", which denoted by Dedv, This process continues until the end of the training pe- riod, resulting in collection of adversarial datasets denoted Dade {DD
D )
Step 2. After collecting the adversarial dataset, initialize the AT model. Then, train the initialized model to convergence on each Dadv in sequence. Importantly, we do not include samples from Dade while training D After training dataset Dady, evaluate the accuracy of generated examples that correctly classified by the classifier Dade
100
90-
Test Accuracy (1g)
80-
Test Accuncy (%)
80
70
50
40
50
Task
Task
(a) CIFAR-10 (E
(b) SVHN (E
100
80
(%4
Test Accumncy (%)
40-
20
20-
Tusk
(c) CIFAR- (P.o)
(d) TinyImageNet (f.c)
Figure 3: Catastrophic forgetting verification in the AT scenario datasets CIFAR-10 (loo), SVHN (Loo), CIFAR- 100 (loo), TinyImageNet (l The horizontal axis represents a timestamp, where each number indicates the order of the current task.
19

---

Published as a conference paper at ICLR 2025
From Figure 3 is obvious that the phenomenon of forgetting is very clear, with the classifier abil- ity on 2t sharply decreasing after fine-tuning on Dr For instance, the test accuracy of Didr dropped from 93.30% to 42.24% at the end of training on CIFAR-10. We consistently find that for- getting occurs across a variety of datasets (i.e., SVHN, CIFAR-10, CIFAR-100, and Tany indicating that it is general property of the adversarial training.
B.2 CONNECTIONS BETWEEN AT AND CL
To visually demonstrate the dynamic nature of adversarial sample distributions during training, employed t-SNE analysis (Van der Maaten & Hinton (008), a technique that effectively captures the complexity of high-dimensional data in reduced-dimensions space. Specifically, trained PGD-10 model on the MNIST dataset using the PreActResNet-118 architecture under the for norm with = 8/255, We then performed t-SNE visualizations on test adversarial samples at 20-epoch intervals, from the beginning to the end of training. The results, shown in Figure reveal distinct shifts in the data distribution of adversarial samples across epochs. This progression is similar to the challenges faced in continual learning, where models must adapt to evolving data distributions without forgetting previous states. This analysis visually substantiates the similarities between AT and continual learning processes.
(a) Clean samples
(b) Epoch 20
Epoch 40
(d) Epoch 60
Epoch 80
(f) Epoch 100
(g) Epach 120
(h) Epoch 140
(i) Epoch 160
(i) Epoch 180
Figure 4: The t-SNE visualization of sample distributions, showing clean samples in (a) and ad- versarial samples from epoch 20 (b) to epoch 180 (j). Each plot illustrates the dynamic changes in the representation of adversarial samples as training progresses. we trained PGD-10 model on the MNIST dataset under the foc norm with 8/255, using the PreActResNet- 18 architecture.
B.3 ADDITIONAL EXPERIMENT RESULTS FOR SENSITIVITY HYPER-PAR SUPEE-PARAMETER
In Table 4 we provide the test accuracy of our method (AMS) with different regularization parame- ter À = 0.05, 0.2, 0.3, 0.5, 1.0, 2.0, 3.0 and interval parameter m = 10, 20, 30, 50 on CIFAR-10 10 with L and 8/255 threat model using PreActResNet--B 18. In Table 5 we present the robust curacy, clean accuracy, memory usage, and computation time for various values of m. Additionally, the corresponding curves shown in Figure
0.2.
20

---

Published as a conference paper at ICLR 2025
Table 4: Test accuracy (%) of the proposed method with different regularization parameter 0.05, 0.1, 0.2, 0.3, 0.5, 1.0, 2.0, 3.0 and interval parameter 10, 20, 30, 50 on CIFAR-10 with to and = 8/255 threat model for PreActResNet-188
highlight the best results in bold.
m
Clean
PGD-20
CW-20
AA
A
Clean
PGD-20
CW-20
AA
0.05
83.45
55.32
52.30
50.31
0.5
83.78
55.67
53.43
51.32
0.1
83.76
55.21
52.77
51.06
1.0
83.63
53.41
53.31
49.82
0.2
84.25
55.36
52.64
51.03
2.0
83.34
55.21
53.14
49.79
0.3
83.91
55.38
52.76
50.96
3.0
83.20
55.01
52.99
49.63
0.05
83.98
55.13
52.23
50.30
0.5
84.86
55.59
53.31
51.22
0.1
84.12
55.20
52.70
51.03
1.0
84.45
55.06
53.24
49.78
20
0.2
84.50
55.28
52.53
50.87
2.0
84.67
55.11
53.08
50.17
0.3
84.93
55.41
52.78
50.95
3.0
84.21
54.78
52.89
49.94
0.05
84.03
55.01
52.15
50.27
0.5
84.91
55.36
53.01
50.97
0.1
84.15
55.18
52.63
50.89
1.0
84.20
54.82
52.91
49.33
30
0.2
84.58
55.19
52.66
50.95
2.0
84.36
54.93
52.87
49.36
0.3
85.01
55.28
52.50
50.63
3.0
84.30
54.47
52.35
49.06
0.05
83.21
54.88
52.07
50.16
0.5
85.12
55.34
53.16
51.04
0.1
84.13
55.08
52.53
50.67
1.0
84.58
55.03
53.00
49.52
0.2
84.38
54.96
52.34
50.59
2.0
84.35
54.82
52.79
49.25
0.3
84.97
55.04
52.58
50.49
3.0
84.30
54.63
52.51
49.57
Table Robust and clean accuracy, memory usage, and computation time for different values of m.
771
AA (%)
Clean (%)
Memory (MB)
Time (h)
10
51.32
83.78
853.6
23.12
20
51.22
84,86
426.8
17.56
30
50.97
84.91
256.08
15.77
50
51.04
85.12
170.72
13.01
90
84
1000-20 25
Accuracy (%6)
Clemm Accumcy (%)
53
Momory (MB)
Time (h)
52
68
600
51
84
50
400
19
48
0
10
40
50
60
Figure 5: Evaluation of interval parameter from: memory usage, time consumption, clean accuracy, and robust accuracy.
21

---

Published as a conference paper at ICLR 2025
B.4 ADDITIONAL EXPERIMENT RESULTS FOR ROBUST OVERHITTING
Comparison with Related Works. In Table 7 we compare the performance of our method in terms of robust overfitting with a number of related works, such as AWP (Wu et al. 2020), SAT (Huang et al.) 2020).TE TE (Dong et al. (022), and LS (Pang et al., 2021). To empirically compare with these methods, we conduct experiments on CIFAR-IO with the PreActResNet-IB 18 network under &oc. threat model. The results are presented in Table from both the best checkpoint, which achieves the highest robust test accuracy under PGD-20 (denoted as "Best"), and the final checkpoint (denoted as "Final"), well the differential between these two checkpoints (denoted as "Diff"). A larger value of "Diff" indicates a more pronounced occurrence of robust overfitting. We observe that our AMS method has the potential to alleviate robust overfitting by addressing forgetting.
Table 6: Test accuracy (%) of the proposed methods and two baseline methods with regularization paramcter À = 0.5 and interval parameter m = 20 on CIFAR-10/100 and SVHN with Loo and = 8/255 threat model for PreActResNet-188 18. We highlight the best results in bold.
Clean
PGD-20
CW-20
AA Final
Dataset
Method
Best
Final
Diff
Best
Final
Diff
Best
Final
Diff' Best
Diff
TRADES
83.25 84.36 -1.11 53.29 50.01 3.28 49.98 48.03 1.95 49.11 46.37
2.74
CIFAR-10
TRADES+AMS 84.86 85.27
-0.41 55.59 55.01 0.58 53.31 52.86 0.45 51.22 50.57
0.65
TRADES
58.26
59.83 -1.57 31.07 28.84 2.23 27.85 26.02
1.83 25.93
24.36
1.57
CIFAR-100
TRADES+AMS 60.03
60.67 -0.64 33.21 32.84 0.37 29.47 28.96
0.51 28.05 27.31
0.74
TRADES
91.02
91.56
-0.54 60.23
58.31
1.92 53.33 50.11
3.22 41.56
39.07
2.49
SVIIN
TRADES+AMS
90.67
90.82
-0.15
60.45
59.84
0.61 56.07
55.67
0.40
52.64
51.77
0.87
Table 7: Test accuracy (%) of our proposed method and related works on the CIFAR-10 dataset
using a PreActResNet- 18 model within an loo threat model and = 8/255. This experiment was conducted under the TRADES framework. We chose the best checkpoint based on the highest robust accuracy achieved on the test set under PGD-20. The best results are highlighted in bold.
Clean
PGD-20
CW-20
AA
Method
Best
Final
Diff
Best
Final
Diff
Best
Final
Diff
Best
Final
Diff
TRADES (Zhang et al.
2019b
83.25
84.36
-1.11 53.29 50.01
3.28 49.98 48.03
1.95 49.11 46.37
2.74
AWP Wu et al. 2020
83.27
84.88
-0.61 54.33 53.02
1.31 51.74 50.12
1.62 50.39
49.27
1.12
SAT Huang al. 2020
82.97
82.36
0.61 53.24
52.86
0.38
52.35
51.88
0.47 50.46
49.87
0.59
TRADES+TE Dong et al.
2022
82.65
83.21 -0.56
55.07 54.11 0.96 52.64
52.16
0.48 50.83
49.82
1.01
TRADES+LS
Pang et al. 2021
83.05
84.14
-1.09
53.86
51.65
2.21
51.35
50.24
1.11
49.76
47.82
1.94
TRADES+AMS
84.86
85.27
-0.41
55,59
55.01
0.58
53.31
52.86
0.45
51.22
50.57
0.65
B.5 ADDITIONAL EXPERIMENT DETAILS AND RESULTS FOR CL
Details for experimental setup. Most of the comparison methods (CL) in our experiments use the implementation of Ma 11 DER (Buzzega al.; 2020), GCR (Tiwari the memory size to 500,000, which allows us to store 300,000 samples. For regularization methods such EWC (Kirkpatrick et al. 2017) and Lwf (Li & Hoiem (017).
For replay-based method such as ER Chauddhyy et al. 2019). al.) 2022 and DGC-ER (Lin et al.) (2024). set
set the interval parameter m to 20, the same as our method. In Table 8 and Table we present the test accuracy of our proposed method and other CL methods on CIFAR-10 with an Lox and = 8/255 threat model using PreActResNet-I1 18 within the PGD and TRADES frameworks.
https:///ittub..com///
22

---

Published as a conference paper at ICLR 2025
Table 8: Test accuracy (%) of the proposed methods and other continual learning methods on CIFAR-10 with an loo and = 8/255 threat model for PreActResNet-18 under the PGD-10 frame- work. The best results are highlighted in bold.
Method
Clean
PGD-20
CW-20
AA
PGD-10
85.29
52.31
51.67
47.70
PGD-I0+ER Chaudhry et al.
2019
82.72
52.44
50.85
47.81
PGD-10+DER Buuzega et al. 2020
82.78
52.51
50.93
48.02
PGD-10+GCR (iwari et al. 2022
82.21
51.39
49.85
47.11
PGD-10+EWC Kirkpatrick et al.
2017)
83.21
53.87
51.24
49.53
PGD-10+Lwf (Li & Hoiem
2017
83.84
54.76
52.82
50.45
PGD-IO+DGC-ER (Lin 2024
82.95
52.58
51.10
48.11
PGD-10+AMS
84.45
55.83
53.17
51.63
Table 9: Test accuracy (%) of the proposed methods and other continual learning methods on CIFAR-10 10 with for and 8/255 threat model for PreActResNet--1 under the TRADES frame- work. The best results are highlighted in bold.
Method
Clean PGD-20
CW-20
AA
TRAEDS
83.25
53.29
49.98
49.11
TRAEDS+ER Chaudhry al.
2019
80.86
53.47
50.33
49.82
TRAEDS+DER (Buzzeaa et al.
2020
81.01
53.55
50.47
50.05
TRAEDS+GCR Tiwari et al. 2022)
80.92
53.31
49.84
48.64
TRAEDS+EWC Kirkpatrick et al.
2017)
83.75
54.15
51.73
50.37
TRAEDS+LWI (i & Hoiem
2017
84.32
54.99
52.86
50.68
TRAEDS+DGC (Lin et al. 2024
81.17
53.69
50.53
50.12
TRAEDS+AMS
84.86
55,59
53.31
51.22
B.6 OTHER EMPIRICAL DETAILS AND RESULTS
Adversarial training with & threat model. In Table 10 we present the test accuracy of the pro- posed method and PGD-10 on the CIFAR-10, CIFAR- 100, and SVHN datasets under an & threat model with 128/255, using the PreActResNett-B architecture within the PGD framework.
Robustness under black-box attacks. We train a PreActResNet-188 model under the foc threat model on the CIFAR-10, CIFAR-100, and SVHN datasets. Black-box adversarial examples are generated using a surrogate adversarial training model with identical settings, employing PGD-20 and CW-20 attacks. As shown in Table II the results demonstrate that our method can effective defense black-box attacks.
Alleviate forgetting. In Table 12 validate the effectiveness of our method in mitigating forget- ting. Compared to Table it evident that our method significantly alleviates forgetting.
Different structures. In Table 13 2014) and ViT (Dosovitskiy et al. 2021). The results illustrate that our AMS method effective on these structures well.
apply AMS to other structures, including VGG16 (Simonyan,
23

---

Published as a conference paper at ICLR 2025
Table 10: Test accuracy (%) of the proposed methods and PGD-10 on CIFAR-10/100 and SVHN with &2 and 128/255 threat model for PreActResNet-18. The results of our methods are in bold.
Datasct
Method
Clean
PGD-20
PGD-10 (Madry et al.
2018
87.85
70.28
CIFAR-10
1ED-I0+AMS
88.21
72.95
PGD-10 (Madry al.
2018
60.05
43.52
CIEAR-100
PGD I0+AMS
64.88
45.02
PGD-10 (Madry al.
2018
92.95
72.85
SVHN
PGD- 10+AMS
93.25
73.10
Table 11: Black-box test accuracy (%) of the proposed methods and PGD-10 10 CIFAR- 10/100 and SVHN with an foc and c = 8/255 threat model for PreActResNet-118 The results of our methods in bold.
CIFAR-10
CIFAR-100
SVHN
Method
Clean
PGD
CW
Clean
PGD
CW
Clean
PGD
CW
PGD-10 Madry ct al. 2018
85.29
58.52
56.31
58.12 12
36.15 33.07
58.92 54.46 90.77
89.12
PGD-10+AMS
85.45
62.38 60.47
58.87
39.73 37.56
66.23 63.64
Table 12: Verification results of the forgetting phenomenon in adversarial training across different datasets and perturbation threat models, integrating with our AMS approach. is important to note that the adversarial datasets accumulated at each phase are initially classified with perfect accuracy, thereby presenting an initial accuracy rate of 100% for each dataset.
Robust Test Accuracy (%)
Dataset
Norm
FAA (%)
FF (%)
Stage
Stage 2
Stage 3
Stage 4
Stage 5
too 92.54
94.28
95.77
97.82
98.20
95.72
4.28
CIFAR-10
Eg 90.83
96.71
97.84
95.82
96.25
95.49
4.51
8o 86.17
96.59
96.35
93.13
95.17
93.48
6.52
CIFAR-100
Eg 87.75
95.73
95.34
96.40
97.65
94.57
5.43
foo 92.17
97.36
97.67
98.20
97.89
96.66
3.34
SVHN
f2 94.62
94.42
97.64
97.74
98. 14
96.51
3.49
foo 72.53
86.81
83.18
87.59
88.56
83.73
16.27
TinyImageNet
82 75.37
86.90
85.63
88.93
92.23
85.81
14.19
Table 13: Test accuracy (%) on CIFAR-IO with foo and e
-8/255 threat model for VIT and VGG16.
Architecture
Method
Clean
AA
TRADES Zhang al.
2019b
85.11
47.04
VIT
TRADES+AMS
85.36
49.62
TRADES 7hang et al. TRADES+AMS
2019h
79.66
44.19
VGG16
81.03
45.79
24


Reviews:
[{"review_id": "EeHKNV2CZX", "reviewer": "Reviewer_Ezci", "summary": "The paper studied the connection between forgetting (studied in the continual learning domain) and lack of adversarial robustness. The idea is that generated adversarial examples at every epoch has its own distribution (akin to tasks in continual learning literature), and learning on the new distribution (task) in the next epoch causes forgetting. The paper has demonstrated the forgetting phenomenon in adversarial training through experiments, and based on these observations, proposed a method called Adaptive Multi-teacher Self-distillation (AMS), which utilizes self-knowledge distillation from multiple checkpoints of the model throughout training to overcome forgetting.", "questions": "1. The main question is, if forgetting is the problem in adversarial training (AT), why does having more data (such as synthetic data) boost robust accuracy? If the number of samples increases, there are greater differences in the distributions of samples (tasks) from one epoch to another, which should potentially increase forgetting. However, in reality, robust accuracy improves. The explanation for this phenomenon appears to be missing from the paper, especially given that almost all recent SOTA methods utilize generated data, and even in the experimental section of this paper, a version with 1M extra generated samples is employed.\n\n2. Regarding the lack of ablation study on the loss formulation, the hyperparameter sensitivity experiments indicate that the size of the interval parameter \"m\" has minimal effect on robustness performance, especially with larger values. If distribution shifts from one epoch to another cause forgetting, we would expect a decline in performance with larger values of \"m,\" as less forgetting would be recovered; however, this trend does not appear in Figure 4. Why is this the case?\n\n\nI have listed my concerns and questions in the weaknesses and questions sections and would appreciate it if the authors could provide precise justifications for them. However, in its current format, I am leaning toward rejection.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of studying forgetting in adversarial training is interesting and might be of interest to researchers in the field.  \n2. The paper is well-written, and it is easy to follow the text along with the experimental results.  \n3. Experimental results cover standard benchmarks in the field.  \n4. It is indeed interesting that the proposed method has been shown to be effective in mitigating robustness overfitting, and studying this direction could deepen researchers' understanding in the field.", "weaknesses": "1. The proposed loss function has multiple components, and each of them needs to be justified with ablation studies. In Section 4, where the loss function and algorithms are discussed, it is argued that \"Reweighting-based Loss Correction\" is needed; however, this is not demonstrated in the paper with experimental results.  \n2. From a memory perspective, multiple checkpoints of the model need to be stored and used for knowledge distillation, which is costly. This could be acceptable if the performance gain is notable; however, previous studies have shown higher performance gains by using only extra synthetic data (see point 5).\n3. In the \"Experimental Setting,\" it is stated that accuracy is reported for the best checkpoint of the model with the highest \"test\" accuracy, which is quite problematic.  \n4. The accuracy of the proposed method in Table 2 has been bolded, but for CIFAR-10, three of the baselines have higher accuracy than the proposed method; similarly for CIFAR-100.  \n5. Related to the previous point, other recent benchmarks, such as \"Better Diffusion Models Further Improve Adversarial Training\" (https://arxiv.org/abs/2302.04638), are missing in the table. For example, with 1M extra adversarial examples, this baseline achieves much higher robust accuracy than the proposed method with the same number of extra examples.  \n6. The study misses a comparable forgetting analysis in natural training. Although experiments have shown that forgetting occurs in adversarial training, it may not be specific to AT, and performing knowledge distillation only for the clean loss in TRADES might be sufficient. An experimental analysis to distinguish this could provide further insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zwAethmLDy", "reviewer": "Reviewer_THQc", "summary": "The authors approach AT from a continual learning perspective, treating adversarial samples generated with different parameters during training as tasks with distinct distributions, as models undergoing AT tend to forget adversarial examples learned in earlier stages, similar to the forgetting phenomenon in continual learning. This work introduces an Adaptive Multi-teacher Self-distillation method, which mitigates forgetting by aligning model outputs across training stages. Experiments on multiple datasets and under various adversarial attacks demonstrate that AMS enhances both clean and robust accuracy, while also alleviating robust overfitting.", "questions": "1.\tIs the classification result shown in Figure 1 based on the training set? If so, this result may not fully illustrate the catastrophic forgetting problem; it may only indicate that newly generated samples were successfully attacked, especially given that an untargeted attack was used. If the result is on the test set, it would more convincingly suggest similarity to catastrophic forgetting in continual learning.\n2.\tCatastrophic overfitting[1] is a common issue in fast adversarial training, with many works focusing on its mitigation[2][3]. Have you explored whether the proposed method could address catastrophic overfitting as well, given its similarity to catastrophic forgetting in continual learning?\n3.\tDid the author consider dividing the epochs into stages unevenly, with each stage containing a different number of training epochs?\n4.\tWhile the method is efficient for certain datasets, it remains computationally demanding for larger datasets like the full ImageNet. Do the authors have plans to further reduce computational requirements in future work?\n\n[1]. Wong E. et.al “Fast is better than free: Revisiting adversarial training” ICLR, 2020  \n[2]. Zhang Y. et.al “Revisiting and advancing fast adversarial training through the lens of bi-level optimization” PMLR, 2022  \n[3]. Wang Z. et.al “Preventing Catastrophic Overfitting in Fast Adversarial Training: A Bi-level Optimization Perspective” ECCV, 2024", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "1.\tThe approach of viewing AT as a continual learning problem is innovative, offering a new perspective on enhancing model robustness within AT.\n2.\tThe paper is well-structured and clearly presents the progression from problem formulation to solution, supported by extensive experiments on diverse datasets that validate the proposed method’s robustness.", "weaknesses": "1.\tThe authors approach AT through the lens of continual learning but do not sufficiently establish the connection between the two, either experimentally or theoretically. For example, demonstrating how adversarial samples generated with different parameters follow distinct data distributions could strengthen the argument. Some statements appear subjective, despite the proposed method's effectiveness. \n2.\tThe AMS method requires tracking knowledge across multiple stages, which increases memory and computational costs, particularly with large datasets and models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9QqNqn07C4", "reviewer": "Reviewer_VMqL", "summary": "The paper presents a novel perspective on mitigating catastrophic forgetting during adversarial training. The paper proposes Adaptive Multi-teachers Self-distillation that uses an adaptive regularizer to align the knowledge differences across stages. The evaluation is performed on multiple target models under various gradient-based attacks.", "questions": "Please address my concerns stated in the weakness section. Note that, although there are concerns raised, I believe none of them is severely affecting the validity of the paper. Hence I rate the initial version of the submission as a borderline accept. My final rating would be conditioned on the soundness of the authors' responses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is well-written with a clear storyline and a sound motivation.\n2. The proposed method is reasonable and fair.\n3. The experiments on three attacks are comprehensive.", "weaknesses": "1. The proposed framework shallowly touches on the training efficiency of adversarial training. It is appreciated that authors mention effective variants of adversarial training that have high training efficiency (e.g., Free-AT [1], YOPO [2]). Yet the paper's algorithm design does not fully address how such efficiency techniques are involved in the framework. Also, Line 6 of Algorithm 1 adds an initial random perturbation to the source image, which is similar to [3]. Authors are expected to elaborate on the design choice of the $0.001$ strength used in this line.\n\n2. The scope of target models is limited. The paper only evaluates CNN-based architectures (e.g., ResNet variants), while more modern vision models based on Vision Transformers are not sufficiently evaluated. It will help to validate the framework if experiments on ViT also support the existing findings. \n\n[1] Adversarial Training for Free. NeurIPS 2019.\n\n[2] You Only Propagate Once: Accelerating Adversarial Training via Maximal Principle. NeurIPS 2019.\n\n[3] Fast is better than free: Revisiting adversarial training. ICLR 2020.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "DnZr2eKl1q", "reviewer": "Reviewer_XV9q", "summary": "This paper studies the adversarial learning problem from a continual learning perspective. In particular, the authors found a novel connection between catastrophic forgetting; a well-studied problem in continual learning, and adversarial training (where the model at later stages forgets the learnt adversaries from earlier stages). Based on this connection, the authors propose a novel method that distills the knowledge from multiple teachers during adversarial training. Each teacher is a previously saved checkpoint obtained through the adversarial training. Through extensive experiments on different datasets and threat models, the proposed orthogonal method shows consistent performance improvement.", "questions": "Please refer to the weaknesses section. I am happy to raise my score if my questions and concerns are resolved.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This paper has the following strengths:\n\n1- The connection between adversarial training and continual learning is the part that I like the most in this paper. This should open the door for further research on the intersection between these two fields.\n\n2- The method proposed in this paper is intuitive, and more importantly is orthogonal. That is, if a stronger training algorithm is developed in the future, one would expect that this algorithm + AMS would perform better.\n\n3- The experiments are comprehensive spanning multiple datasets, adversarial training schemes, and threat models.", "weaknesses": "Despite the strengths in this paper, there are still remaining concerns that need to be resolved.\n\n1- Concern regarding experiments: In Table 2, it seems that S$^2$O performs better than TRADES+AMS in every aspect on CIFAR-10 (both clean accuracy and robust accuracy under all considered attacks). This then begs for the following two experiments:\n\n1a. Can yo provide the performance of S$^2$O on the other considered datasets (e.g. CIFAR100, SVHN, TinyImageNet)?\n\n1b. Can you provide the results of S$^2$O + AMS? Generally, how is the performance of other robust training methods when combined with AMS?\n\nThere are also a couple of other experiments missing in this paper; namely:\n\n1c. Experiments on CIFAR10 using the 1M generated images (with and without AMS).\n\n1.d Comparison agains AWP [A] seems to be missing in this paper.\n\n2- Concern regarding presenting results in table: I think it is quite confusing to not make the best results in bold rather than making the proposed method's results in bold. I think an easier way to trace the results is to distinguish the best results and the runner up. Alternatively, you could present each method as (Method in one row, Method + AMS in another row highlighting the performance gain under AMS).\n\n3- Concern about the proposed method: The results in the appendix (Table 5) show that AMS can suffer from significant performance drop under different choices of hyper parameters (namely $\\lambda$). Is the choice of hyper parameters at least generalizable under different robust training methods? A simple experiment to address this concern is to fix the hyper parameters for the suggested experiment in 1.b to be $m=20, \\lambda=0.5$.\n\nMinor comments:\n\n4- Performance gain under AMS is marginal (often ~1%). \n\n5- How is the performance of AMS in the continual learning setting? I am curios to see if it actually outperforms the CL methods such as LwF and ER.\n\n6- Missing reference: I find the motivational experiments of this paper (experiments on adversarial examples from previous checkpoints) are quite similar to the work of Gupta et.al [B]. I believe that the related work in this paper will benefit from a discussion on the relation between the results in Figure 1 in this work and the results in Figure 1 in [B].\n\n[A] Adversarial Weight Perturbation Helps Robust Generalization, NeurIPS 2020\n\n[B] Improving the affordability of robustness training for DNNs, CVPRW 2020", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uYqInAyqY2", "reviewer": "Reviewer_rCAv", "summary": "This work proposes an enhanced adversarial training algorithm that tackles this problem from the perspective of continual learning and catastrophic forgetting. It first demonstrates a forgetting phenomenon with an experiment. Then, a methodology named AMS that distills from weighted teachers (checkpoints from earlier training epochs) is proposed. Empirical experiments on SVHN, CIFAR-10/100, and Tiny ImageNet show improved robustness when adding AMS upon e.g. Madry's adversarial training and TRADES. Robust overfitting can also be mitigated with the help of AMS.", "questions": "Please see the questions in weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The work is presented in a clear and smooth way. Viewing adversarial training from a continual learning and forgetting perspective is interesting and could be of significance if it's novel (which I'm not entirely sure since I'm away from adversarial training literature for a while). It starts by exposing the forgetting problem with a concrete experiment, which justifies the validity of employing the forgetting view to some extent. The method is also presented well, with Figure 2 being quite informative. Experiments seem quite extensive.", "weaknesses": "1. AMS will inevitably incur larger time and space cost. Beyond increasing $m$, is there any other more intelligent method that can help reduce the number of teachers? For example, could there be a metric that helps identify the most informative teachers, so that throughout the training one can only keep say the most valuable $k$ teachers?\n\n2. One straightforward approach for aggregating or maintaining learned information from previous models is Exponential Moving Average (EMA), which has been shown to help with adversarial training [1]. EMA should be considered as a baseline in my opinion.\n\n3. An ablation study should be presented to validate the proposed reweighting-based loss correction.\n\n4. In Table 6, it seems that distilling from more teachers (having smaller $m$) hurts clean accuracy but not robustness. Any idea why?\n\n\n[1] Uncovering the Limits of Adversarial Training against Norm-Bounded Adversarial Examples", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)