# Stylized Pairing for Robust Adversarial Defense
## 3 Material and Methods
### Stylized Pairing Training  
As shown in Figure 1, the proposed method mainly consists of two unique parts: stylized pre-processing and stylized pairing loss. The stylized network generates the stylized images, and the pairing loss is utilized to force the content images and stylized images to be similar. **Stylized Pre-Processing:** Stylized pre-processing takes a content image \(c\) and a style image \(s\), and it creates an output image that combines the content of the \(c\) with the style of the \(s\). In this paper, we follow the original settings of [20] (the code and pre-trained models are available at [[https://github.com/xunhuang1995/AdaIN-style](https://github.com/xunhuang1995/AdaIN-style)]([https://github.com/xunhuang1995/AdaIN-style](https://github.com/xunhuang1995/AdaIN-style)) (accessed on 1 September 2022)) and utilize a single encoder-decoder architecture to realize style transfer. The first several layers of encoder \(f(\cdot)\) (up to \(relu4\_1\)) are fixed to that of a pre-trained VGG-19, and the decoder \(g\) is used to generate stylized images. The AdaIN layer aligns the mean and variance of the content feature map to those of the style feature map to produce the target feature map \(t\). It can be expressed as follows:  
\[t=AdaIN(f(c),f(s)). \tag{3}\]  
In the training phase, the decoder \(g\) is learned to map the \(t\) back to the image space to generate the stylized image \(T(c,s)\):  
\[T(c,s)=g(t). \tag{4}\]  
It is worth mentioning that the function \(AdaIN(\cdot,\cdot)\) performs style transfer in the latent space by transferring the feature channel-wise mean and variance as shown in Equation (1). Because \(AdaIN\) has no learnable affine parameters, it adaptively computes the affine parameters from the style input. Due to its low overhead to generate style transfer images, we utilize it as a part of our method to generate stylized images. In Figure 2, we show the stylized images on three datasets: CIFAR-10, CIFAR-100, and ImageNet, respectively. **Stylized Pairing Loss:** [PERSON] et al. [28] proposed the concept of adversarial logit pairing, which encourages the logit of adversarial example and corresponding normal examples to be similar. Inspired by this idea, we propose a novel method that encourages the stylized images and their corresponding clear images to be similar. For a deep neural network \(F(\cdot)\), it takes an input \(x\) to compute an output \(z=F(x)\), and the stylized loss can be formulated as follows:  
Figure 2: Stylized image examples on three dataset: CIFAR-10, CIFAR-100, and ImageNet. \[\mathcal{L}_{s}=\lambda L(F(x),F(x^{\prime})), \tag{5}\]  
where \(x^{\prime}\) is the stylized image generated by the stylized network, \(\lambda\) is a regularization parameter to determine the strength of the stylized pairing penalty, and \(L(\cdot,\cdot)\) is the loss function to measure the similarity. In the proposed method, we use mean squared loss for \(L\), which measures the mean squared error (squared \(L_{2}\) norm) between two elements. **Stylized pairing Training:** Stylized loss pairing matches the DNN's output from a normal image \(x\) and its corresponding stylized image \(x^{\prime}\). In the traditional training scheme, they treat stylized images as data augmentation. The model is trained to assign both \(x\) and \(x^{\prime}\) to the same output class label, but the model has no access to receiving any information to indicate that \(x^{\prime}\) is more similar to \(x\) compared to another example with the same class with \(x\). Therefore, we propose to add a stylized pairing loss on the base of the normal training with stylized images. Let us denote with \(J(M,\theta)\) the cost function of normal training with stylized images where a model with parameters \(\theta\) is trained on a mini-batch \(M\) including normal images \(\{x_{1},x_{2},\dots,x_{m}\}\) and corresponding stylized images \(\{x^{\prime}_{1},x^{\prime}_{2},\dots,x^{\prime}_{m}\}\).