Adversarial Noise Layer: Regularize Neural Network By Adding Noise

In this paper, we introduce a novel regularization method called Adversarial Noise Layer (ANL) and its efficient version called Class Adversarial Noise Layer (CANL), which are able to significantly improve CNN’s generalization ability by adding carefully crafted noise into the intermediate layer activations. ANL and CANL can be easily implemented and integrated with most of the mainstream CNN-based models. We compared the effects of the different types of noise and visually demonstrate that our proposed adversarial noise instruct CNN models to learn to extract cleaner feature maps, which further reduce the risk of over-fitting. We also conclude that models trained with ANL or CANL are more robust to the adversarial examples generated by FGSM than the traditional adversarial training approaches. Although Convolutional Neural Networks (CNNs) are powerful and widely used in various computer vision tasks, they suffer from over-fitting due to the excessive amount of parameters (Srivastava et al. 2014). The initial development of the neural network was inspired by the mechanism of human brain (Rosenblatt 1957) which does not work as precisely as the computer. Inspired by the difference, we infer that adding noise into the process of training could instruct CNNs to learn more robust feature representations to against the effect of noise, thereby reducing the risk of over-fitting. Many regularization methods (Zhong et al. 2017; Simonyan and Zisserman 2015; Krizhevsky, Sutskever, and Hinton 2012) have been proposed to prevent over-fitting by adding noise into the training data. Besides, methods like DisturbLabel (Xie et al. 2016) randomly changed the label of a small subset of samples to incorrect value each iteration, thereby regularizing the CNNs on loss layer. Sankaranarayanan et al. proposed a regularization method called Layerwise Adversarial Training (LAT) (Sankaranarayanan et al. 2018) which uses the gradients of the previous batch to generate noise for the current batch during training. Different from data augment methods, LAT adds the perturbations not only to the input images, but also the intermediate layer activations. In this work, we propose a variant of LAT and its efficient version, we call them Adversarial Noise Layer (ANL) and Class Adversarial Noise Layer (CANL) respectively. Unlike LAT, ANL generates the noise base on the current batch gradients. CANL further reduces additional time costs by caching the gradients based on image category. ANL and CANL are well compatible with various CNN architectures and can be injected without changing the design philosophy. ANL and CANL are only embedded in the model during the training, so that it actually takes no extra computation in the inference process. Our main contributions are summarized as follows: • We introduce a CNN regularization method called ANL. The empirical results show that ANL can significantly improve the performance of various mainstream deep convolution neural networks on popular datasets (FashionMNIST, CIFAR-10, CIFAR-100, ImageNet). • We introduce the CANL algorithm, which is an efficient version of ANL. Compared to ANL, CANL takes less training time, but at the cost of slight degeneration in regularization performance. • We demonstrate that ANL and CANL provide stronger regularization compared to LAT and Dropout. We also verified that ANL and CANL can improve the robustness of the CNN models, comparable to the traditional adversarial training approach, under the attack of Fast Gradient Sign Method (Goodfellow, Shlens, and Szegedy 2015).