# Exploring Synergy of Denoising and Distillation: Novel Method for Efficient Adversarial Defense
## 2 Related Works
### Adversarial Defense
#### 2.3.2 Defensive Distillation  
Defensive distillation [33] is an approach that implements knowledge distillation to find off adversarial attacks, thereby increasing the complexity of grasping the model's gradient and executing attacks. The essence of knowledge distillation is to amplify the gradient through temperature values [21]. This study employs knowledge distillation as a fundamental technique for constructing an efficient model to counter adversarial attacks. Therefore, we analyzed the concept of knowledge distillation and subsequently scrutinized defensive distillation. Knowledge distillation is a technique that facilitates the transfer of knowledge from a pretrained network, often called a teacher network. This knowledge is subsequently incorporated into a smaller network known as the student network, which is designed for eventual user applications [21]. Therefore, the purpose of the knowledge distillation process is to extract superior performance from a smaller model by acquiring intricate features from a larger network that are too intricate to learn from a smaller one. The student model designed for the ultimate user application underwent proficient training through a learning process that consisted of two loss computations. \[L_{\text{soft distillation}}=(1-\lambda)L_{\text{CE}}(\psi(Z_{s}),y)+\lambda \tau^{2}\text{KL}(\psi(Z_{s}/\tau),\psi(Z_{t}/\tau)) \tag{5}\]  
\[Softmax(Z_{i})=\frac{e^{Z_{j}}}{\sum\limits_{j=1}^{K}e^{z_{j}}},Softmax(z_{i} )with\tau=\frac{e^{Z_{j}}/\tau}{\sum\limits_{j=1}^{K}e^{z_{j}}/\tau} \tag{6}\]  
The soft distillation loss function in Equation (5) is a crucial component of knowledge distillation. The student model, which is subjected to a two-phase learning process, utilizes two distinct loss functions. The proportional weight assigned to each loss function is represented by \(\lambda\) in Equation (5). The student model's supervised learning from labels aligns with the function \(L_{CE}\) in Equation (5), where the logits output from the student model is denoted by \(\psi(Z_{s})\). Here, \(y\) denotes the label, thereby enabling traditional supervised learning. The function \(KL\) represents the Kullback-Leibler divergence function, which is a measure of dissimilarity between two probability distributions. The learning process aims to minimize this divergence. Within the \(KL\) function, the two distributions correspond to the student model output (\(\psi(Z_{s}/\tau)\)) and the teacher model output (\(\psi(Z_{t}/\tau)\)). The logit value from each model is divided by \(\tau\) in the label smoothing process to optimize the learning effect of the soft labels. This value, referred to as 'temperature \(\tau\)' by [PERSON] et al. [21], is used in the softmax function, as depicted in Equation (6). Equation (5) provides a comprehensive understanding of how the student model absorbs the weight information from the teacher model. This led to the development of several subsequent models, including data-efficient image transformers (DeiT) [34] and DeiT3 [35], all of which incorporate the KD principle of knowledge distillation. Many studies have substantiated the information transfer efficacy of knowledge distillation, which, in turn, validates its utility in enhancing and lightening models [21]. In addition, several studies have confirmed the effectiveness of knowledge distillation in object detection [36; 37; 38], and there have been numerous in-depth analyses of its successful implementation in natural language processing [39; 40]. Previous studies established that knowledge distillation extends across various domains beyond image processing. Particularly in the context of AI security, [PERSON] et al. [41] demonstrated that knowledge distillation can be a potent method in adversarial defense as it allows for the partial transfer of the defense performance of the teacher model. There have also been initiatives to counter adversarial attacks by directly using knowledge distillation.