# Improving Adversarial Robustness via Distillation-Based Purification
## 3 Method
### Training Purifier via Knowledge Distillation  
We next build a purifier model as a student to distill the knowledge of the two previously trained teacher models. The purifier also uses a Convolutional Autoencoder with the same structure as the teacher models. Overall, The purifier learns to effectively denoise the AEs, just like the AT teacher model. The purifier is trained based on the ensemble knowledge distillation framework: by distilling the restoration ability of the NT teacher model to the purifier, we hope that the denoised part of the purified image will be restored similar to the original image; by distilling the denoising ability of the AT teacher model to the purifier, we expect the purifier's denoising ability to be improved.  
Figure 3 depicts our process of learning a purifier based on knowledge distillation. As we introduced, the AT teacher model is given AEs generated by the PGD attack to purify them, and the NT teacher model is given pure images to restore them. The purifier takes AEs (each denoted by \(x^{\prime}\)) as input and tries to remove the noise. Then, the difference between the purified image \(f_{s}(x^{\prime})\) and the original image \(x\) is defined as the reconstruction loss function \(L_{s}\) for our purifier, which is computed as follows:  
\[\begin{split} L_{s}&=MSE\big{(}(f_{s}(x^{\prime}),\ x)\\ &=\frac{1}{n}\sum_{i=1}^{n}(f_{s}(x^{\prime})_{i},\ x_{i})^{2} \end{split} \tag{4}\]  
Another loss function of our purifier, the ensemble knowledge distillation loss function \(L_{kd}\), consists of the Kullback-Leibler distance and the mean square error between the outputs of multiple teacher models \(f_{T_{j}}\) and student models \(f_{s}\), as follows:  
\[L_{kd}=\frac{1}{M}\sum_{j=1}^{M}\Big{(}KL\Big{(}g\big{(}f_{T_{j}}(x^{\prime}) \big{)},\ g\big{(}f_{s}(x^{\prime})\big{)}\Big{)}+MSE\Big{(}f_{T_{j}}\big{(}x^ {\prime}),\ f_{s}\big{(}x^{\prime}\big{)}\Big{)}\Big{)} \tag{5}\]  
where \(M\) is the number of teacher models (in our case, \(M=2\)) and \(T_{j}\) is each teacher model. The Kullback-Leibler divergence function \(KL()\) computes the difference between  
Figure 3: The proposed framework for learning a purification model based on knowledge distillation.  
the output distribution of the teacher model and that of the student model, where each probability distribution is computed by the Softmax function \(g\).  
As a result, the purifier learns the denoising ability and image restoration ability of the two teacher models, respectively, and is simultaneously optimized by the Kullback-Leibler divergence and the mean square error, which can reduce both the distribution difference between the student and teacher models and the output image difference. The final loss function \(L\) for training the purifier is configured as follows:  
\[L=\beta\cdot L_{s}+\gamma\cdot L_{kd} \tag{6}\]  
where \(\beta\) and \(\gamma\) control the importance of the reconstruction loss \(L_{s}\) and the knowledge distillation loss \(L_{kd}\), respectively. For simplicity, we assume that the two loss terms have equal importance and set \(\beta=\gamma=0.5\).