# Defense Against Adversarial Attacks in Deep Learning
## 4 Defense Mechanism Based on Knowledge Transfer
### Training Method Based on Distillation  
By observing the denoised image and the original image, we found there are still subtle differences between the two images. The differences cause the denoised image to be recognized incorrectly. To further improve the ability to resist adversarial samples, we need to improve the robustness of the model to denoised pictures. For improving the applicability of the defense strategy, our proposed strategy should minimize the modification of the model network structure and maintain the speed of the network. After applying the defense strategy, the model could accurately identify the denoised image that is very close to the original image and maintain the accuracy of the recognition. We propose a training strategy based on Distillation. The specific strategy is shown in Figure 4.  
Figure 4: Defense mechanism based on Distillation. The mechanism consists of a teacher model and a student model. The student is trained with the output classification probability of the teacher.  
In our proposed defense strategy, we trained the student model by using the teacher model. Teacher and Student model have the same network structure. We used the denoised image and the corresponding ground truth to train the teacher model, where the ground truth refers to the one-hot label. The Student model was then trained using the image and the corresponding output of the Teacher model, which represents the correlation of the image with other types of images and finer image information. The output probability was used to train student model, which can improve the robustness and accuracy of the model. After the training, the student model is used as a model for external access. The denoised image recognition result can be obtained by processing with the student model. Therefore, our defense strategy can maintain the speed of the model. The method prevents the model from fitting closely to the data and helps to generalize the training data. The parameter T controls the knowledge extracted by the action of Distillation. The larger T is, the larger the output probability value for recognition class is. The overall defense strategy is shown in Figure 5. The overall defense mechanism is divided into three phases. In the first stage, noise images are first obtained from clean pictures and corresponding attack images. The noise samples are used to train the GNR to achieve the reconstruction of the noise. The trained GAN is used to generate the corresponding adversarial image for the clean image. The more image pairs are obtained, and the trained dataset is expanded. In the second stage, the already expanded dataset is used to train the deep denoising network UDDN, and the denoised image can be obtained through accessing the trained UDDN. In the classification model, relatively clean images can be correctly identified with a high probability. To further improve the resilience of the model, in the third stage, the training strategy based on knowledge transfer is proposed to train the model. The teacher model is trained with the denoised images. After that, the student model is trained with the denoised images and the output of the teacher model. Finally, the student model is used for external access. The training method can enhance the robustness of the model and improve the resilience of the model. Our proposed defensive strategy applies to any deep neural network model, and the overall strategy does not significantly increase the computation of the network and maintains the speed of the network.