SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization

1 INTRODUCTION . Machine learning has achieved state-of-the-art ( SOTA ) performance in many fields , especially in computer vision tasks . This success can mainly be attributed to the deep architecture of convolutional neural networks ( CNN ) that typically have 10 to 100 millions of learnable parameters . Such a huge number of parameters enable the deep CNNs to solve complex problems . However , besides the powerful representation ability , a huge number of parameters increase the probability of overfitting when the number of training examples is insufficient , which results in a poor generalization of the model . In order to improve the generalization ability of deep learning models , several data augmentation strategies have been studied . Random feature removal is one of the popular techniques that guides the CNNs not to focus on some small regions of input images or on a small set of internal activations , thereby improving the model robustness . Dropout ( Nitish et al. , 2014 ; Tompson et al. , 2015 ) and regional dropout ( Junsuk & Hyunjung , 2019 ; Terrance & Graham , 2017 ; Golnaz et al. , 2018 ; Singh & Lee , 2017 ; Zhun et al. , 2017 ) are two established training strategies where the former randomly turns off some internal activations and later removes and/or alters random regions of the input images . Both of them force a model to learn the entire object region rather than focusing on the most ∗Department of Computer Science & Engineering , Kyung Hee University , South Korea . †Corresponding author . Published as a conference paper at ICLR 2021 T ar ge t Im ag e So u rc e Im ag e Dog - 80 % ? Cat - 20 % ? Augmented Image Augmented Label Dog - 80 % ? Cat - 20 % ? LabelOriginal Image Dog Cat T a rg e t Im ag e So u rc e Im ag e Dog - 80 % ? Cat - 20 % ? Augmented Image Augmented Label Dog - 80 % ? Cat - 20 % ? LabelOriginal Image Dog Cat T ar g et I m ag e S o u rc e I m ag e Dog - 80 % ? Cat - 20 % ? Augmented Image Augmented Label Dog - 80 % ? Cat - 20 % ? LabelOriginal Image Dog Cat important features and thereby improving the generalization of the model . Although dropout and regional dropout improve the classification performance , this kind of feature removal is undesired since they discard a notable portion of informative pixels from the training images . Recently , Yun et al . ( 2019 ) proposed CutMix , that randomly replaces an image region with a patch from another training image and mixes their labels according to the ratio of mixed pixels . Unlike Cutout ( Devries & Taylor , 2017 ) , this method can enjoy the properties of regional dropout without having any blank image region . However , we argue that the random selection process may have some possibility to select a patch from the background region that is irrelevant to the target objects of the source image , by which an augmented image may not contain any information about the corresponding object as shown in Figure 1 . The selected source patch ( background ) is highlighted with a black rectangle on the source image . Two possible augmented images are shown wherein both of the cases , there is no information about the source object ( cat ) in the augmented images despite their mixing location on the target image . However , their interpolated labels encourage the model to learn both objects ’ features ( dog and cat ) from that training image . But we recognize that it is undesirable and misleads the CNN to learn unexpected feature representation . Because , CNNs are highly sensitive to textures ( Geirhos et al. , 2019 ) and since the interpolated label indicates the selected background patch as the source object , it may encourage the classifier to learn the background as the representative feature for the source object class . We address the aforementioned problem by carefully selecting the source image patch with the help of some prior information . Specifically , we first extract a saliency map of the source image that highlights important objects and then select a patch surrounding the peak salient region of the source image to assure that we select from the object part and then mix it with the target image . Now the selected patch contains relevant information about the source object that leads the model to learn more appropriate feature representation . This more effective data augmentation strategy is what we call , ” SaliencyMix ” . We present extensive experiments on various standard CNN architectures , benchmark datasets , and multiple tasks , to evaluate the proposed method . In summary , SaliencyMix has obtained the new best known top-1 error of 2.76 % and 16.56 % for WideResNet ( Zagoruyko & Komodakis , 2016 ) on CIFAR-10 and CIFAR-100 ( Krizhevsky , 2012 ) , respectively . Also , on ImageNet ( Olga et al. , 2015 ) classification problem , SaliencyMix has achieved the best known top-1 and top-5 error of 21.26 % and 5.76 % for ResNet-50 and 20.09 % and 5.15 % for ResNet-101 ( He et al. , 2016 ) . In object detection task , initializing the Faster RCNN ( Shaoqing et al. , 2015 ) with SaliencyMix trained model and then fine-tuning the detector has improved the detection performance on Pascal VOC ( Everingham et al. , 2010 ) dataset by +1.77 mean average precision ( mAP ) . Moreover , SaliencyMix trained model has proved to be more robust against adversarial attack and improves the top-1 accuracy by 1.96 % on adversarially perturbed ImageNet validation set . All of these results clearly indicate the effectiveness of the proposed SaliencyMix data augmentation strategy to enhance the model performance and robustness . 2 RELATED WORKS . 2.1 DATA AUGMENTATION . The success of deep learning models can be accredited to the volume and diversity of data . But collecting labeled data is a cumbersome and time-consuming task . As a result , data augmentation has been introduced that aims to increase the diversity of existing data by applying various transformations e.g. , rotation , flip , etc . Since this simple and inexpensive technique significantly improves the model performance and robustness , data augmentation has widely been used to train deep learning models . Lecun et al . ( 1998 ) applied data augmentation to train LeNet for hand written character recognition . They performed several affine transformations such as translation , scaling , shearing , etc . For the same task , Bengio et al . ( 2011 ) applied more diverse transformation such as Gaussian noise , salt and pepper noise , Gaussian smoothing , motion blur , local elastic deformation , and various occlusions to the images . Krizhevsky et al . ( 2012 ) applied random image patch cropping , horizontal flipping and random color intensity changing based on principal component analysis ( PCA ) . In Deep Image ( Wu et al. , 2015 ) , color casting , vignetting , and lens distortion are applied besides flipping and cropping to improve the robustness of a very deep network . Besides these manually designed data augmentations , Lemley et al . ( 2017 ) proposed an end-to-end learnable augmentation process , called Smart Augmentation . They used two different networks where one is used to learn the suitable augmentation type and the other one is used to train the actual task . Devries & Taylor ( 2017 ) proposed Cutout that randomly removes square regions of the input training images to improve the robustness of the model . Zhang et al . ( 2017 ) proposed MixUp that blends two training images to some degree where the labels of the augmented image are assigned by the linear interpolation of those two images . But the augmented images look unnatural and locally ambiguous . Recently , Cubuk et al . ( 2019 ) proposed an effective data augmentation method called AutoAugment that defines a search space of various augmentation techniques and selects the best suitable one for each mini-batch . Kim et al . ( 2020 ) proposed PuzzleMix that jointly optimize two objectives i.e. , selecting an optimal mask and an optimal mixing plan . The mask tries to reveal most salient data of two images and the optimal transport plan aims to maximize the saliency of the revealed portion of the data . Yun et al . ( 2019 ) proposed CutMix that randomly cuts and mixes image patches among training samples and mixes their labels proportionally to the size of those patches . However , due to the randomness in the source patch selection process , it may select a region that does not contain any informative pixel about the source object , and the label mixing according to those uninformative patches misleads the classifier to learn unexpected feature representation . In this work , the careful selection of the source patch always helps to contain some information about the source object and thereby solves the class probability assignment problem and helps to improve the model performance and robustness . 2.2 LABEL SMOOTHING . In object classification , the class labels are usually represented by one-hot code i.e. , the true labels are expected to have the probability of exactly 1 while the others to have exactly 0 . In other words , it suggests the model to be overconfident which causes overfitting to training dataset . As a result , the models have low performance on unseen test dataset . To alleviate this problem , label smoothing allows to relax the model confidence on the true label by setting the class probability to a slightly lower value e.g. , lower than 1 . As a result , it guides the model to be more adaptive instead of being over-confident and ultimately improves the model robustness and performance ( Szegedy et al. , 2016 ) . Our method also mixes the class labels and enjoys the benefit of label smoothing . 2.3 SALIENCY DETECTION . Saliency detection aims to simulate the natural attention mechanism of human visual system ( HVS ) and can be classified into two main categories . The first one is a bottom-up approach ( Cheng et al. , 2014 ; Zhu et al. , 2014 ; Li et al. , 2015 ; Zhou et al. , 2015 ; Achanta et al. , 2009 ; Li et al. , 2013 ; Hou & Zhang , 2007 ; Qin et al. , 2015 ; Peng et al. , 2016 ; Lei et al. , 2016 ; Montabone & Soto , 2010 ) that focuses on exploring low-level vision features . Some visual priors that are inspired by the HVS properties are utilized to describe a salient object . Cheng et al . ( 2014 ) utilized a contrast prior and proposed a regional contrast based salient object detection algorithm . Zhu et al . ( 2014 ) introduced a robust background measure in an optimization framework to integrate multiple low level cues to obtain clean and uniform saliency maps . Li et al . ( 2015 ) optimized the image boundary selection by a boundary removal mechanism and then used random walks ranking to formulate pixel-wised saliency maps . Zhou et al . ( 2015 ) proposed a saliency detection model where the saliency information is propagated using a manifold ranking diffusion process on a graph . In addition , some Published as a conference paper at ICLR 2021 Saliency Detection Peak Salient Region Source Image Saliency Detection Peak Salient Region Target ImageSource Patch Augmented Image Source Image Saliency Map of the Source Image Selecting the Peak Salient Region of the Saliency Map Target Image Selecting the Source Patch Based on the Peak Salient Region Augmented Image Mixing the Source Patch with the Target Image traditional techniques are also introduced to achieve ima e saliency detection , such as frequency domain analysis ( Achanta et al. , 2009 ) , sparse representation ( Li et al. , 2013 ) , log-spectrum ( Hou & Zhang , 2007 ) cellular automata ( Qin et al. , 2015 ) , low-rank recovery ( Peng et al. , 2016 ) , and Bayesian theory ( Lei et al. , 2016 ) . Hou & Zhang ( 2007 ) proposed a spectral residual method that focuses on the properties of background . Achanta et al . ( 2009 ) proposed a frequency tuned approach that preserves the boundary information by retaining sufficient amount of high frequency contents . Montabone & Soto ( 2010 ) introduced a method that was originally designed for a fast human detection in a scene by proposing novel features derived from a visual saliency mechanism . Later on this feature extraction mechanism was generalized for other forms of saliency detection . The second one is a top-down approach which is task-driven and utilizes supervised learning with labels . Several deep learning based methods have been proposed for saliency detection ( Deng et al. , 2018 ; Liu et al. , 2018 ; Zhang et al. , 2018a ; b ; Qin et al. , 2019 ) . Deng et al . ( 2018 ) proposed a recurrent residual refinement network ( R3Net ) equipped with residual refinement blocks ( RRBs ) to more accurately detect salient regions . Contexts play an important role in the saliency detection task and based on that Liu et al . ( 2018 ) proposed a pixel-wise contextual attention network , called PiCANet , to learn selectively attending to informative context locations for each pixel . Zhang et al . ( 2018a ) introduced multi-scale context-aware feature extraction module and proposed a bi-directional message passing model for salient object detection . Zhang et al . ( 2018b ) focused on powerful feature extraction and proposed an attention guided network which selectively integrates multi-level contextual information in a progressive manner . Recently , Qin et al . ( 2019 ) proposed a predict and refine architecture for salient object detection called boundary-aware saliency detection network ( BASNet ) . The author introduced a hybrid loss to train a densely supervised Encoder-Decoder network . However , despite the high performance of top-down approach , there is a lack of generalization for various applications since they are biased towards the training data and limited to the specific objects . In this study , we require a saliency model to focus on the important object/region in a given scene without knowing their labels . As a result , we rely on bottom-up approach which are unsupervised , scale-invariant and more robust for unseen data . It is worth noting that the training based saliency methods can also be applied where the quality and quantity of data for training the saliency methods may be correlated with the effectiveness of data augmentation . Section 3.3 further explains the effects of different saliency detection algorithm on the proposed data augmentation method .

Summary: This paper proposes a new augmentation method based on CutMix. The authors find out that randomly selecting may mix background textures and this will mislead the model. So, they propose to use saliency maps to control the selection of mixed patches, which is called SaliencyMix. This idea seems easy and reasonable, many experiments are conducted to prove the effectiveness of the proposed method. However, the experiments’ results fail to show the ability of the method, and some explanation is missed.