Sustainable Self-evolution Adversarial Training

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-05 18:35:30

Abstract: With the wide application of deep neural network models in various computer vision tasks, there has been a proliferation of adversarial example generation strategies aimed at exploring model security deeply. However, existing adversarial training defense models, which rely on single or limited types of attacks under a one-time learning process, struggle to adapt to the dynamic and evolving nature of attack methods. Therefore, to achieve defense performance improvements for models in long-term applications, we propose a novel Sustainable Self-evolution Adversarial Training (SSEAT) framework. Specifically, we introduce a continual adversarial defense pipeline to realize learning from various kinds of adversarial examples across multiple stages. Additionally, to address the issue of model catastrophic forgetting caused by continual learning from ongoing novel attacks, we propose an adversarial data replay module to better select more diverse and key relearning data. Furthermore, we design a consistency regularization strategy to encourage current defense models to learn more from previously trained ones, guiding them to retain more past knowledge and maintain accuracy on clean samples. Extensive experiments have been conducted to verify the efficacy of the proposed SSEAT defense method, which demonstrates superior defense performance and classification accuracy compared to competitors.

Main paper: 
 introduction: 
 Deep learning has been widely applied in computer vision tasks such as image classification [8, 65] and object detection [6, 37] , resulting in significant advancements. However, the vulnerability of deep learning models to adversarial attacks [16, 26, 49] has become a critical concern. Adversarial examples involve intentionally crafted small perturbations that deceive deep learning models, leading them to produce incorrect outputs. This poses a serious threat to the reliability and security of these models in real-world applications. Consequently, research on defense mechanisms [17, 22, 34] has become increasingly essential and urgent.Nowadays, researchers study various model defense methods to address highly destructive adversarial samples, including input sample denoising [21, 24] and attack-aware detection [14, 15] . Among these defense methods, adversarial training [12, 67] stands out as one of the most effective defense strategies. Adversarial training is a game-based training approach aimed at maximizing perturbations while minimizing adversarial expected risk. Its core idea is to integrate generated adversarial examples into the training set, enabling the model to learn from these examples during training and enhance its robustness.Current adversarial training strategies often rely on one or limited types of adversarial examples to achieve robust learning, and focus on improving the defense performance against attacks and the efficiency of the adversarial training process. However, in realworld applications, as researchers delve deeper into model defense and security, various new attack strategies continue to emerge (According to our incomplete statistics, over the past five years, more than 200 papers on adversarial attack algorithms have been published in top journals and conferences every year across various fields including multimedia, artificial intelligence, and computer vision), existing adversarial training strategies obviously struggle to address such complex scenario. Therefore, it is essential for deep models to achieve sustainable improvement in defense performance for long-term application scenarios, as shown in Fig. 1 . This has brought about the following challenges: In this study, to address the above three challenges for the defense model against the ongoing generation of new adversarial examples, we propose a novel and task-driven Sustainable Selfevolution Adversarial Training (SSEAT) framework to ensure the model maintains its accuracy and possesses robust and continuous defense capabilities. Our SSEAT defense method comprises three components: Continual Adversarial Defense (CAD), Adversarial Data Reply (ADR), and Consistency Regularization Strategy (CRS). To achieve sustainability in the adversarial training strategy (Challenge (1)), drawing inspiration from the continue learning paradigm, we propose a CAD pipeline, which learns from one type of adversarial example at each training stage to address the continual generation of new attack algorithms. To address the issue of catastrophic forgetting when continuously learning from various attacks (Challenge (2)), we introduce an ADR module to establish an effective re-learning sample selection scheme, advised by classification uncertainty and data augmentation. Meanwhile, to realize the trade-off between the model's robustness against adversarial examples and accuracy on clean data (Challenge (3)), We designed a CRS module to help the model not overfit to current attacks and prevent the model from losing knowledge on clean samples. Overall, our SSEAT method effectively addresses a range of defense challenges arising from continuously evolving attack strategies, maintaining high classification accuracy on clean samples, and ensuring lifelong defense performance against ongoing new attacks.We summarize the main contributions of this paper as follows:  
related work 2.1 adversarial attacks: 
 The impressive success of deep learning models in computer vision tasks [6, 37, 62] has sparked significant research interest in studying their security. Many researchers are now focusing on adversarial example generation [16, 26, 49] . Adversarial examples add subtle perturbations that are imperceptible to the human eye on clean data, causing the model to produce incorrect results. Depending on the access rights to the target model and data, attacks can be divided into black-box attacks [1, 38, 45, 47] and white-box attacks [4, 19, 40, 46] . Most white-box algorithms [19, 31, 40] Adversarial training [19] is a main method to effectively defend against adversarial attacks. This approach involves augmenting the model's training process by incorporating adversarial examples, thus the data distribution learned by the model includes not only clean samples but also adversarial examples. Many adversarial training research mainly focuses on improving training efficiency and model robustness [11, 32, 57, 67] . For example, Zhao [66] uses FGSM instead of PGD during training to reduce training time and enhance efficiency, Dong [12] investigates the correlation between network structure and robustness to develop more robust network modules, Chen [7] uses data enhancement or generative models to alleviate robust overfitting, and Lyu [39] adopts regularization training strategies, such as stopping early to smooth the input loss landscape. Meanwhile, the trade-off between robustness and accuracy has attracted much attention [34, 43, 51] . TRADES [63] utilizes Kullback-Leibler divergence (KL) loss to drive clean and adversarial samples closer in model output, balancing robustness and accuracy.In addition, some studies [48, 64] try to use curriculum learning strategies to improve robustness while reducing the decrease in accuracy on clean samples. Most current adversarial training approaches rely on a single or limited adversarial example generation algorithm to enhance model robustness. However, in real-world scenarios, existing defense methods struggle to address the ongoing emergence of diverse adversarial attacks. Inspired by the continue learning paradigms, we aim to enhance defense capabilities by enabling the adaptive evolution of adversarial training algorithms for long-term application scenarios.Continue learning [2, 28, 58] aims at the model being able to continuously learn new data without forgetting past knowledge. Continue learning can be mainly divided into three categories. One is based on the regularization of model parameters [27, 61] by preserving important parameters from the past while updating less critical ones. However, this method's performance is not ideal when applied to scenarios involving a large number of tasks. One is based on knowledge playback [3, 9, 35, 42, 50] , where important past samples are stored in memory and used for training when encountering new tasks. Another approach involves dynamically expanding model parameters [18, 23, 33] to assign different parameters to different tasks. This method helps alleviate catastrophic forgetting and enhances model performance, but it requires significant memory and computational resources. Nevertheless, continue learning models are also susceptible to adversarial examples. Wang et. al. [54] attempt to combine adversarial training with continue learning paradigms, but they do not consider the defensive performance of the current model against old attacks in long-term application scenarios. Differently from previous work, our focus is on ensuring that when the model continues to encounter new adversarial attacks, it can maintain robustness against past adversarial examples and improve resilience against new adversarial attacks. 
methodology 3.1 task defination and framework overview: 
 Continuous Defense Setting. As deep learning models become widely used in fields like healthcare, manufacturing, and military, ensuring their security has become a primary focus for researchers.Investigating adversarial examples with high transferability provides valuable insights into deep models. Consequently, there has been a constant influx of diverse new attack methods [16, 26, 49] in recent years. EQUATIONwhere L 𝑎𝑑𝑣 represents the loss function, and we use the standard cross-entropy loss to design the loss function L 𝑎𝑑𝑣 .In our practical CDS, the adversarial training model will continue to encounter adversarial examples generated in various ways. Thus, we design a novel Continuous Adversarial Defense (CAD) pipeline, at each stage of CAD, the model is exposed to a batch of new attack samples for training to adapt continuously to new environments and data distributions. In the initial stage, the model is trained on original clean data, and the trained model is denoted as 𝑓 𝜃 0 . Additionally, we generate multiple sets of adversarial examples targeting the same original sample using different attack methods, and each adversarial example set corresponds to a specific attack method. During the CAD, as more and more attack examples are incorporated into training, the model increasingly struggles to avoid catastrophic forgetting, hindering its ability to maintain sustainable defense capabilities in long-term application scenarios. Thus, we introduce a novel Adversarial Data Reply (ADR) strategy to realize an effective rehearsal sample selection scheme, enhance adversarial example diversity, and obtain high-quality replay data.High-quality sample data should accurately reflect their class attributes and demonstrate clear distinctions from other classes in the feature space. We consider samples located at the distribution center to be the most representative, while those at the classification boundary are the most distinctive. Therefore, based on these two characteristics, we select diverse and representative replay data within the feature space. However, accurately computing the relative position of samples in the feature space requires significant computational resources and time. Therefore, we utilize our classification model to infer the uncertainty of samples, thereby indirectly revealing their relative positions in the feature space. In practical implementation, we perform various data augmentations to obtain augmented samples. Subsequently, we calculate the variance of the model's output results for these samples subjected to different data augmentations to assess their uncertainty. We think that when the model's predictions for a sample are more certain, the sample may be closer to the core of the class distribution; conversely, when uncertainty in predictions increases, the sample may be closer to the class boundary.First, we define the learning samples for each round as,EQUATION)where D 𝑡 represents the sample set of the 𝑡 𝑡ℎ learning stage in CAD.We assume that the prior distribution of samples 𝑝 ( x |𝑥) is a uniform mixture of various data augmentations, where x represents the augmented samples generated via color jitter, shear, or cutout. We utilize the Monte Carlo method to approximate the uncertainty of the sample distribution 𝑝 (𝑦 = 𝑐 |𝑥). Then, we measure the relative distribution of samples based on the uncertainty of model outputs.The derivation process is as follows,EQUATIONwhere 𝑍 signifies the number of augmentation methods utilized.The distribution D represents the data distribution defined by x. 𝑝 (𝑦 = 𝑐 | x𝑡 ) denotes the probability of the augmented sample x𝑡 having the label 𝑐. Specifically, the augmented sample x is generated by a random function 𝑔 𝑘 (•),EQUATIONwhere 𝛼 𝑘 represents a hyper-parameter signifying the stochastic component of the 𝑘 𝑡ℎ perturbation.The prior distribution 𝑝EQUATIONwhere the random variable 𝜔 𝑘 is selected from a categorical binary distribution. We assess the sample's uncertainty in relation to the perturbation by,EQUATIONEQUATIONwhere We allocate memory for a replay buffer of size 𝐾 for each learning iteration. We sort all samples D 𝑡 based on the computed uncertainty 𝑟 (𝑥), and sample the examples with an interval of |D 𝑡 | /𝐾. We have diversified the replay samples by sampling perturbed samples of varying intensities, ranging from robust to fragile ones, which can broaden the scope of memories, encompassing a wide range of scenarios.In practical application models, in addition to achieving sustainable defense against attack examples, it is crucial to maintain high recognition accuracy on original clean samples. To prevent the model's learned data distribution from straying too far from the space of clean sample data, we leverage the knowledge distillation method and propose a novel Consistency Regularization Strategy (CRS), to ensure that the same sample fed into both previous model 𝑓 𝜃 𝑡 , where 𝑧 𝜃 𝑡 (𝑥) is the logit value of 𝑓 𝜃 𝑡 (𝑥) before the softmax operation. Therefore, we obtain the regularization loss as follows:EQUATIONwhere 𝐽𝑆 (• ∥ •) denotes the Jensen-Shannon divergence. Since augmentations are randomly sampled at each training step, minimizing the proposed objective ensures that adversarial examples remain consistently predicted regardless of augmentation selection. Additionally, in adversarial training, due to the relatively low confidence of predictions (i.e., maximum softmax value), using a smaller temperature helps ensure a sharper distribution to address this issue. By ensuring consistency between the predictions of the previous model and the current training model under different data augmentation schemes, we can ensure that the previous model retains the knowledge learned from the previous training. This CRS approach not only helps improve the model's robustness to adversarial samples but also maintains accuracy on clean samples.The overarching training objective, denoted as 𝐿 𝑡𝑜𝑡𝑎𝑙 , integrates adversarial training objectives with consistency regularization losses. Initially, we deliberate on averaging the inner maximization objective 𝐿 𝑎𝑑𝑣 across two distinct augmentations, 𝐴 1 and 𝐴 2 , sampled from the augmentation set 𝐴. This choice stems from the equivalence of minimizing over the augmentation set 𝐴 to averaging over 𝐴 1 and 𝐴 2 .EQUATIONSubsequently, we integrate our regularizer into the averaged objective mentioned above, introducing a hyperparameter denoted as 𝜆. Therefore, the final training objective 𝐿 𝑡𝑜𝑡𝑎𝑙 can be expressed as follows:EQUATIONOur regularization method stands apart from the selection of the adversarial training objective, rendering it universally applicable to diverse established adversarial training approaches. To illustrate, if we were to adopt the conventional adversarial training loss, the resulting overall objective would be formulated as: 7)EQUATION1: for 𝑡 in 1, • • • , 𝑇 do 2:in 𝐷 𝑡 𝑎𝑑𝑣 , apply adversarial training, according to Eq. ( 10), to train the model 3:in 𝐷𝑟 𝑡 , apply adversarial training and consistency regularization strategy, according to Eq. ( 12), to train the model 𝑓 𝜃 𝑡Combine replay samples with adversarial samples of the current stage : 𝐷 𝑡 = (𝑥, 𝑦) |, (𝑥, 𝑦) ∈ 𝐷𝑟 𝑡 ∪ D 𝑡 𝑎𝑑𝑣 , and select replay samples 𝐷𝑟 𝑡 +1 according to Eq. ( 7) 5: end for 6: return 𝑓 𝜃 𝑇 
experimental 4.1 experimental settings: 
 Datasets. We evaluate our SSEAT model over the CIFAR-10 dataset [29] ,which is commonly used for adversarial attack and defense research. It contains 50,000 images for training and 10,000 images for testing, covering 10 different categories of objects.Our method only uses 1000 images for training and all 1000 images for testing. In each stage, the training and test data are converted into attack according to the corresponding attack algorithm. The converted training part of the data is used for SSEAT training, and the test part is only used for the final black box test. To better verify the efficacy of our SSEAT method, we also conduct more experiments over different datasets in the supplementary materials. Attack Algorithms. Under the CDS task, we use various attack algorithms, i.e., FGSM [19] , BIM [30] , PGD [40] , RFGSM [53] , MIM [13] , NIM [36] , SIM [36] , DIM [60] , VNIM [55] , and VMIM [55] , to generate adversarial samples under 𝑙 ∞ for training our SSEAT model and further evaluate the robustness against the attacks. The perturbation amplitude of all attacks is set to 𝜖 = 8/255 and the attack step size to 𝛼 = 2/255. In our experiments, we conduct four different attack orders for the CDS task: (1) Order-I: FGSM, PGD, SIM, DIM, VNIM; (2) Order-II: BIM, RFGSM, MIM, NIM, VMIM; (3) Order-III: MIM, PGD, FGSM, SIM, BIM; (4) Order-IV: FGSM, BIM, PGD, RFGSM, NIM, SIM, DIM. We designed experiments involving Implementation Details. We summarize the training procedure of our SSEAT framework in Alg. 1. The model uses resnet18 [20] as the classification network structure. We implement Torchattacks [25] to generate 100 adversarial images per category for each adversarial attack strategy, for a total of 1000 images. On the CIFAR10 dataset, before training, each image is resized to 32×32 pixels, and underwent data augmentation, which included horizontal flipping and random cropping. For the training hyperparameters of experiments, we use a batch size of 8, the epoch for training clean samples is set to 40, and the epoch for training adversarial samples is set to 20. We train the model using the SGD optimizer with momentum 0.9 and weight decay 5 × 10 -4 . In addition, we set the memory buffer size to 1000. Competitors. We compare our SSEAT method with several adversarial training works, such as the PGD-AT [40] , TRADES [63] , MART [56] , AWP [59] , RNA [12] , and LBGAT [10] . Evaluation Metrics. To comprehensively assess the model's performance in both robustness and classification within the CDS task, we employ two indicators: (1) The model's classification accuracy across all kinds of adversarial examples after completing all adversarial training stages, which is more practical and differs from the 'classification accuracy against each attack after every adaptation step' [54] ; (2) The model's classification accuracy on the original clean data after completing all adversarial training stages.We have conducted extensive experiments over the CIFAR-10 dataset with various attack orders in the CDS task, compared to several competitors and baseline. All attack results are reported under the black-box condition. For the CDS task, our SSEAT model can achieve the best robustness against ongoing generated new adversarial examples. To better evaluate our SSEAT model under the CDS task, we generate 10 kinds of attacks and organize them as three different sets of adversarial sample sequences. We report model robustness over various attack orders compared to several competitors. As shown in Tab. 1, Tab. 2, and Tab. 3, our SSEAT method can beat For the CDS task, our SSEAT model can maintain competitive classification accuracy over the clean data. For real-life applications, in addition to continuously improving the defense performance against new attack methods, the model also needs to have good recognition effects on clean samples. Thus, we report the classification accuracy over original data in Tab. 1, Tab. 2, and Tab. 3. The results show our SSEAT method not only achieves strong robustness under complex changes, but also has good performance on the original task. Achieving a balanced trade-off between robustness and accuracy greatly enhances the practicality of the adversarial training strategy in real scenarios.To verify the role of each module in our SSEAT, we conducted extensive ablation studies on the following variants: (6) Comparing the '+CRS', '+ADR', and '+ADR+CRS (Simple)', the results demonstrate the ADR and CRS modules can complement each other on both classification accuracy and defense robustness; (7) Comparing '+ADR+CRS (Simple)' and '+ADR+CRS (Ours)', the results show our SSEAT model can achieve the best performance. The reason is that the model of the previous stage has knowledge of the data in memory, but does not have knowledge of the data in the current stage. If you move the model's output closer to it on the current data, it is likely to have a counterproductive effect and mislead the update of the current model; if you move the model's output closer to it on the memory data, it will guide the model to retain the knowledge of past samples. (8) As shown in Fig. 3 , as training is completed, our SSEAT model can achieve a gradual improvement in defense performance and maintain classification accuracy. Our SSEAT model can be adapted to more challenging CDS tasks. (1) Considering the complexity of real-world application scenarios, we extended the attack sequence to be longer. As shown in Tab. 5, thanks to the proposed data replay module and knowledge distillation strategy for alleviating the problem of catastrophic forgetting, our SSEAT model still shows the best defense and classification ability. (2) Additionally, we intentionally disrupted the original attack sequence to further validate the effectiveness of our method. As shown in Tab. 6, we show the results under re-ordering the attack sequence for Order-I in the CDS task, the defense performance still does not fluctuate excessively, which fully demonstrates that the SSEAT method can defend against a variety of adversarial samples. (3) Furthermore, to be more realistic, we designed a different setting in which the types of attack examples in each stage are not unique. As shown in Tab. 7, in the three stages of training with different numbers of types of attack samples, the model trained by our method maintains excellent accuracy on adversarial samples and clean samples. The results fully prove the versatility of our method and will not be affected by too many constraints. Overall, the biggest difference from other competitors is that our method model has the ability to evolve independently and can continue to learn and defend against more adversarial attacks. Our approach is able to adapt to complex and diverse attack samples and achieve broad and general model robustness. 
conclusion: 
 To realize the adaptive defense ability of deep models with adversarial training when facing the continuously generated diverse new adversarial samples, we proposed a novel and task-driven Sustainable Self-evolution Adversarial Training (SSEAT) method. Inspired by the continue learning, the SSEAT framework can continuously learn new kinds of adversarial examples in each training stage, and realize the consolidation of old knowledge through the data rehearsal strategy of high-quality data selection. At the same time, a knowledge distillation strategy is used to further maintain model classification accuracy on clean samples. We have verified the SSEAT model efficacy over multiple continue defense setting orders, and the ablation experiments show the effectiveness of the components in the SSEAT method. 
 
 Appendix:  


Decision: The review comments are only lukewarm. Some reviewers complained about inadequacy of the rebuttal in addressing the original issues.

The proposed approach has some merits, but the issues probably outweigh the strengths. So I would vote for Reject, but will be open to a different outcome.

Reviews: review: paper_summary: paper_summary: This paper extends the adversarial training in the continuous learning setting. The proposed method, called SSEAT, involves an adversarial data reply module to select relearning data and a consistency regularization to retain past knowledge and maintain accuracy on clean data. Experiments on CIFAR-10 dataset demonstrate superior defense performance over the baselines.
strength_weakness: strength_weakness: -Exploring model robustness in the long-term learning process is encouraged.

-The paper is easy to read.
limitations: limitations: This paper explores continuous adversarial defense in multiple-stage learning, which is interesting and encouraged. However, the effect of the proposed method is limited and the experiments are insufficient.

-The adversarial examples in the paper are all gradient-based. Since there exist many adversarial attack methods, i.e., learning-based and optimization-based, they should be also explored in the continuous defense.

-The experiments are only conducted on CIFAR-10 dataset. More complex datasets should be tested.

-Most of the compared methods are one-time learning, applying different training methods in the CAD setting is more appropriate.

-I'm curious about the distribution of the adversarial examples selected in the replay memory module. Since the adversarial examples are all gradient-based, would the strongest adversarial examples (PGD) always be selected?
paper_summary: paper_summary: The paper introduces continual adversarial defense which combines continue learning and adversarial training to provide a novel Sustainable Self-evolution Adversarial Training framework. To incorporate continue learning into adversarial training, the authors introduce  consistency regularization strategy and adversarial data replay to retain past knowledge and tackle model catastrophic forgetting. The experiments demonstrate the effectiveness of the proposed algorithm.
strength_weakness: strength_weakness: 1. The proposed consistency regularization strategy and adversarial data replay seem natural for the combination of continue learning and adversarial training.
2. The experiment results demonstrate the superiority of the proposed algorithm over other baselines.
3. The ablation studies demonstrate the effectiveness of each component.
limitations: limitations: 1. There is no evaluation of different attack strengths, such as PGD attacks with more attack iterations and larger perturbation sizes.
2. There is no evaluation of black-box attacks.
3. More evaluation on multiple datasets is recommended.
paper_summary: paper_summary: The authors proposed an adversarial continual learning framework termed Sustainable Self-evolution Adversarial Training, which aims to improve defense performance for models in long-term applications.
strength_weakness: strength_weakness: This paper is well-written, and the training process is clearly explained.

This work is instructive as a robust model should have a long-term learning paradigm.
limitations: limitations: This work suggests that a robust model should also incorporate a continual learning framework to face growing malicious attacks. However, one difficulty in the traditional continual learning framework is handling new tasks or new categories, which may cause catastrophic forgetting.

In this work, using new attack methods as an analogy seems inappropriate. As far as current attack methods are concerned, the difference between the adversarial samples of new attacks and those of old attacks is not significant. For example, a model trained with samples generated by DI-FGSM can still defend against adversarial samples generated by PGD.

Additionally, setting the attack budget single to 8/255 makes it difficult to distinguish the disturbances caused by different attacks.
paper_summary: paper_summary: This paper proposes a continual learning framework-based adversarial training method to guarantee the sustainable robustness of deep learning models toward novel adversarial attacks. The overall idea in this paper is simple and straightforwardly illustrated. However, the limitations of this paper include errors and ambiguities in notation, non-rigorous assertions, and the absence of somehow important experimental illustrations.
strength_weakness: strength_weakness: 1. The problem related to the continuous robustness toward novel adversarial attacks is interesting and promising. 

2. The core idea of this paper is straightforward and easy to understand. 

3. The comparative experiment results in this paper are relatively sufficient.
limitations: limitations: 1. There are some confused notations in section 3.2, especially with $P(y=c \mid x)$ from lines 462 - 485. In line 416, (x,y) is defined as an input-label pair with a certain mapping. I understand there is uncertainty from $x$ to $\tilde x$. Does it mean $P(f(g_k(x,\alpha _k))=y\mid x)$?. 

2. The challenge mentioned in the introduction line 126 "How to balance the model’s robustness on adversarial examples and accuracy on clean data." is interesting. How to solve it by SSEAT? Could you give a brief explanation? 

3.  A key problem SSEAT aims to solve is the robustness to tackle nove adversarial attacks. However, in the experimental part, from Tables 1 to 3, all attack algorithms are included in the training process of SSEAT. The performance of SSEAT under unknown adversarial attacks is important, yet this result is absent in the paper. Moreover, there is a small mistake in reporting the performance of PGD-AT under PGD attack. This method achieves SOTA performance but is not bold.