# Improving Adversarial Robustness via Distillation-Based Purification
###### Abstract  
Despite the impressive performance of deep neural networks on many different vision tasks, they have been known to be vulnerable to intentionally added noise to input images. To combat these adversarial examples (AEs), improving the adversarial robustness of models has emerged as an important research topic, and research has been conducted in various directions including adversarial training, image denoising, and adversarial purification. Among them, this paper focuses on adversarial purification, which is a kind of pre-processing that removes noise before AEs enter a classification model. The advantage of adversarial purification is that it can improve robustness without affecting the model's nature, while another defense techniques like adversarial training suffer from a decrease in model accuracy. Our proposed purification framework utilizes a Convolutional Autoencoder as a base model to capture the features of images and their spatial structure. We further aim to improve the adversarial robustness of our purification model by distilling the knowledge from teacher models. To this end, we train two Convolutional Autoencoders (teachers), one with adversarial training and the other with normal training. Then, through ensemble knowledge distillation, we transfer the ability of denoising and restoring of original images to the student model (purification model). Our extensive experiments confirm that our student model achieves high purification performance(i.e., how accurately a pre-trained classification model classifies purified images). The ablation study confirms the positive effect of our idea of ensemble knowledge distillation from two teachers on performance. adversarial robustness; adversarial attacks; adversarial purification; knowledge distillation; image classification; convolutional autoencoders +
Footnote †: journal:  
model more robust so that it can classify well even when these AEs are mixed into the input. There are many types of adversarial attacks, and one popular technique is to add noise to an image based on gradients (FSGM, PGD, etc.) [8; 9; 10]. Other methods include generating AEs that minimize a loss function over the input [11], changing only one of the most critical pixels in the image [12], and combining multiple methods of creating AEs [13]. The effectiveness of the attack usually depends on the value of the parameter \(\epsilon\), which controls the amount of noise added to images. Adversarial defense strategies to combat these attacks have also been actively studied. Representative areas include adversarial training [8], which uses AEs together to train a model, and image denoising, which tries to remove noise from the input AEs [14; 15]. Image denoising aims to restore the AEs as close as possible to the original image by removing the noise in the image, and among them, adversarial purification aims to remove the noise by assuming that the noise is definitely an adversarial purturbation caused by adversarial attacks. However, both methods have limitations in that their robustness performance decreases with different types of attacks, and their accuracy for normal inputs decreases. In this paper, we propose a novel purification technique that can improve adversarial robustness. The main idea of the proposed method is to transfer the knowledge of two Convolutional Autoencoder [16] models (one with adversarially trained and the other with normally trained) to a student model through ensemble knowledge distillation [17]. Convolutional Autoencoder is an image-friendly structure that replaces the multi-layer perceptron (MLP) with Convolutional layers in the original MLP-based Autoencoder. It has shown good performance in image restoration and generation because it can capture the local features and spatial structures of images better than MLP.