# Defense Against Adversarial Attacks in Deep Learning
## 3 Deep Denoising Network Based on U-Net: UDDN
### Network Structure  
The denoising autoencoder (DAE) is one of the popular denoising models. A DAE was used as a multi-layered perceptual network to defend against adversarial attacks. Experiments of defensive strategies are often performed on some relatively simple datasets, such as the MNIST dataset. For high resolution images, the strategy does not work well. DAE has a bottleneck structure between the encoder and the decoder. The network structure may limit the propagation of fine-scale information required to reconstruct high-resolution images. To overcome this problem, we incorporated the idea of U-Net in the DAE. In our method, the decoder layer can fuse the feature map of the encoder layer, which can better locate and recover the details of the image. Unlike U-Net, the decoder layer fuses feature images of the same scale at the encoder layer. The learning target of UDDN is to obtain noise instead of reconstructing the entire image, which is different from DAE. This residual learning method can help to train the model more easily and obtain more accurate noise data. By subtracting the noise from the adversarial image, the clean image can be obtained. The specific structure of UDDN is shown in Figure 3.  
Figure 3: Deep denoising neural network (UDDN) network structure. The structure consists of an encoder and a decoder. The output of the decoder is added to the adversarial images to obtain the denoised image. The whole model is composed of C2, C3, and F units, the structures of which are shown on the right side of the figure. As can be seen from Figure 3, a DAE can be obtained by removing the horizontal connection in UDDN. C stands for a basic module that consists of a three-layer network. The first layer is a 3 \(\times\) 3 convolution followed by a BN layer and a ReLU activation function layer. Ck stands for k consecutive Cs. In C2, we added a connection at the second C, forming a residual structure. In C3, we added numerous connections at the second and third C positions, forming a structure similar to the densenet block. The F module is the same fusion module as U-Net, which combines the same-scale feature map of the encoder layer and the decoder layer into a new feature map. The overall network also consists of a contracting path and an expanding path. The encoder consists of five modules, one C2 and four C3. The step size of the first convolutional layer in C3 is two, and step size in other layers is one. The decoder consists of four modules and a 1*1 convolutional layer. Each module receives a feedback input from the decoder and a lateral input from the encoder. It first upsamples the feedback input to the same size as the horizontal input. Then, Ck processes the concatenation of the feedback input and the lateral input. From top to bottom, three C3 and one C2 were used. The output of the last module is converted to negative noise \(-\mathrm{d}\hat{x}\) by a \(1\times 1\) convolutional layer. \(x^{*}\) represents the noise image, and \(\hat{x}\) represents the output of the network. Then, the final output is the sum of the negative noise and the input image, as shown in Equation (4). \[\hat{x}=x^{*}-d\hat{x} \tag{4}\]  
The difference between the adversarial image and the clean image is negligible. However, this small perturbation is gradually amplified by the deep neural network and produces erroneous predictions. Even though the noise reducer can significantly suppress pixel-level noise, the residual noise can still distort the response of the target model. To overcome this problem, we replaced the pixel-level loss function with the reconstruction loss of the model output. More specifically, given a target neural network, we extracted the representation of x and x from the Lth layer, and defined the loss function as the L1 norm of their differences:  
\[\mathrm{L}=\parallel f_{l}(\hat{x})-f_{l}(x)\parallel \tag{5}\]  
The supervisory signals come from certain top layers of the model, as well as guidance information related to image classification.