SaliencyMix: A Saliency Guided Data Augmentation Strategy for Better Regularization
Authors: A F M Shahab Uddin
Mst. Sirazam Monira
Wheemyung Shin
TaeChoong Chung
Sung-Ho Bae
Venue: ICLR 2021 Poster (2021)
Keywords: SaliencyMix
Saliency Guided Data Augmentation
Data Augmentation
Regularization

Abstract: Advanced data augmentation strategies have widely been studied to improve the generalization ability of deep learning models. Regional dropout is one of the popular solutions that guides the model to focus on less discriminative parts by randomly removing image regions, resulting in improved regularization. However, such information removal is undesirable. On the other hand, recent strategies suggest to randomly cut and mix patches and their labels among training images, to enjoy the advantages of regional dropout without having any pointless pixel in the augmented images. We argue that such random selection strategies of the patches may not necessarily represent sufficient information about the corresponding object and thereby mixing the labels according to that uninformative patch enables the model to learn unexpected feature representation. Therefore, we propose SaliencyMix that carefully selects a representative image patch with the help of a saliency map and mixes this indicative patch with the target image, thus leading the model to learn more appropriate feature representation. SaliencyMix achieves the best known top-1 error of $21.26\%$ and $20.09\%$ for ResNet-50 and ResNet-101 architectures on ImageNet classification, respectively, and also improves the model robustness against adversarial perturbations. Furthermore, models that are trained with SaliencyMix, help to improve the object detection performance.  Source code is available at \url{https://github.com/SaliencyMix/SaliencyMix}.

Published as a conference paper at ICLR 2021
SALIENCYMIX: A SALIENCY GUIDED DATA AUG- MENTATION STRATEGY FOR BETTER REGULARIZA- TION
A. M. Shahab Uddin uddin@khu ar
Mst. Sirazam Monira monira@khu.ac.
Wheemyung Shin wheemi @khu. ac kr
TaeChoong Chung tcchungekhu.ac.. ac.
Sung-Ho Bae
shbae@khu.ac.uo
ABSTRACT
Advanced data augmentation strategies have widely been studied to improve the generalization ability of deep learning models. Regional dropout is of the popular solutions that guides the model to focus on less discriminative parts by randomly removing image regions, resulting in improved regularization. How- ever, such information removal is undesirable. On the other hand, recent strate- gies suggest to randomly cut and mix patches and their labels among train- ing images, to enjoy the advantages of regional dropout without having any pointless pixel in the augmented images. We argue that such random selec- tion strategies of the patches may not necessarily represent sufficient informa- tion about the corresponding object and thereby mixing the labels according that uninformative patch enables the model to learn unexpected feature repre- sentation. Therefore, we propose SaliencyMMix that carefully selects a repre- sentative image patch with the help of saliency map and mixes this indica- tive patch with the target image, thus leading the model to learn more appro- priate feature representation. SaliencyMix achieves the best known top-1 error of 21.26% and 20.09% for ResNet-50 and ResNet-101 architectures on ImageNet classification, respectively, and also improves the model robustness against ad- versarial perturbations. Furthermore, models that are trained with SaliencyMix help to improve the object detection performance. Source code is available https:/ / com Sali htt.://ww.....oeaeeeeeee///aaaea
INTRODUCTION
Machine learning has achieved state-of-the-art (SOTA) performance in many fields, especially in computer vision tasks. This success can mainly be attributed to the deep architecture of convolu- tional neural networks (CNN) that typically have 10 to 100 millions of learnable parameters. Such huge number of parameters enable the deep CNNs solve complex problems. However, besides the powerful representation ability. huge number of parameters increase the probability of overfitting when the number of training examples insufficient, which results in a poor generalization of the model.
In order to improve the generalization ability of deep learning models, several data augmentation strategies have been studied. Random feature removal is one of the popular techniques that guides the CNNs not to focus on some small regions of input images or on a small set of internal activations, thereby improving the model robustness. Dropout (Nitish et al. 2014 Tompson et al. 20155) and regional dropout (Junsuk & Hyunjung 2019 Terrance & Graham 2017 Golnaz et al. 2018 [iigh & Lee 2017: Zhun | 2017) are two established training strategies where the former randomly turns off some internal activations and later removes and/or alters random regions of the input im- ages. Both of them force model to learn the entire object region rather than focusing on the most
"Department of Computer Science & Engineering, Kyung Hee University, South Korea.
1 "Coresponniig author.

---

Published as a conference paper at ICLR 2021
Target Image
Source Image
Augmented Image
20%
Mixed label for randomly mixed images
Dog 80% & Cat 20% ? Dog 80% & Cat 20% ?
Problem of randomly selecting image patch and mixing labels according to it. When the selected source patch does not represent the source object, the interpolated label misleads the model to learn unexpected feature representation.
Figure| 1:
important features and thereby improving the generalization of the model. Although dropout and regional dropout improve the classification performance, this kind of feature removal is undesired since they discard a notable portion of informative pixels from the training images.
Recently, Yun al. (2019) proposed CutMix, that randomly replaces image region with a patch from another training image and mixes their labels according to the ratio of mixed pixels. Unlike Cutout (Devries & Taylor 2017). this method can enjoy the properties of regional dropout without having any blank image region. However, we argue that the random selection process may have some possibility select patch from the background region that is irrelevant to the target objects of the source image, by which an augmented image may not contain any information about the corresponding object as shown in Figure [ The selected source patch (background) is highlighted with a black rectangle on the source image. Two possible augmented images are shown wherein both of the cases, there no information about the source object (cat) the augmented images the target image. However, their interpolated labels encourage the
despite their mixing location model to learn both objects' features (dog and cat) from that training image. But we recognize that it is undesirable and misleads the CNN learn unexpected feature representation. Because, CNNs are highly sensitive to textures (Geirhos et al. 2019) and since the interpolated label indicates the selected background patch the source object, it may encourage the classifier to learn the background as the representative feature for the source object class.
We address the aforementioned problem by carefully selecting the source image patch with the help of some prior information. Specifically, we first extract a saliency map of the source image that highlights important objects and then select patch surrounding the peak salient region of the source image to assure that we select from the object part and then mix it with the target image. Now the selected patch contains relevant information about the source object that leads the model learn more appropriate feature representation. This more effective data augmentation strategy is what we call, "Si... We present extensive experiments on various standard CNN architectures, benchmark datasets, and multiple tasks, to evaluate the proposed method. In summary, SaliencyMix has obtained the new best known top-1 error of 2.76% and 16. 5.% for WideResNet (Zagoruyko & Komodakis 2016) on CIFAR-10 10 and CIFAR- 100 (Krizhevsky) 2012). respectively. Also, Ima- geNet (Olga et al. 2015) classification problem, SaliencyMis has achieved the best known top- I and top-5 error of 21. 21.26% and 5.76% for ResNet-50 and 20.09% and 5.15% for ResNet-101 (He al., 2016). In object detection task, initializing the Faster RCNN (Shaoqing et al. (015) with Salien- cyMix trained model and then fine-tuning the detector has improved the detection performance Pascal VOC (Everingham et al. 2010) dataset by +1.77 mean average precision (mAP). Moreover. SaliencyMix trained model has proved be more robust against adversarial attack and improves the top-1 accuracy by 1.96% on adversarially perturbed ImageNet validation set. All of these re- sults clearly indicate the effectiveness of the proposed SaliencyMix data augmentation strategy to enhance the model performance and robustness.
RELATED WORKS
2.1
DATA AUGMENTATION
The success of deep learning models can be accredited to the volume and diversity of data. But col- lecting labeled data is cumbersome and time-consuming task. As result, data augmentation has
2

---

Published as a conference paper at ICLR 2021
been introduced that aims to increase the diversity of existing data by applying various transforma- tions e.g., rotation, flip, etc. Since this simple and inexpensive technique significantly improves the model performance and robustness, data augmentation has widely been used to train deep learning models. Lecun et al. 1998) applied data augmentation to train LeNet for hand written character recognition. They performed several affine transformations such translation, scaling, shearing, etc. For the same task. [engio et al. (201T) applied more diverse transformation such as Gaussian noise, salt and pepper noise, Gaussian smoothing, motion blur, local elastic deformation, and various occlusions to the images. Krizhevsky et al. (2012) applied random image patch cropping, horizontal flipping and random color intensity changing based on principal component analysis (PCA). Deep Image (Wu et al.) 2015), color casting, vignetting, and lens distortion are applied besides flipping and cropping to improve the robustness of a very deep network.
Besides these manually designed data augmentations, Lemley et al. (017)) proposed an end-to-end learnable augmentation process, called Smart Augmentation. They used two different networks where one is used to learn the suitable augmentation type and the other one is used train the actual task. Devries & Taylor 2017) proposed Cutout that randomly removes square regions of the input training images to improve the robustness of the model. Zhang et al. (2017) proposed MixUp that blends two training images to some degree where the labels of the augmented image are assigned by the linear interpolation of those two images. But the augmented images look unnatural and locally ambiguous. Recently, Cubuk et al.] (2019) proposed an effective data augmentation method called AutoAugment that defines search space of various augmentation techniques and selects the best suitable one for each mini-batch. Kim et al. (2020) proposed PuzzleMix that jointly optimize two objectives i.c., selecting an optimal mask and an optimal mixing plan. The mask tries to reveal most salient data of two images and the optimal transport plan aims to maximize the saliency of the revealed portion of the data. Yun et al. (2019) proposed CutMix that randomly cuts and mixes image patches among training samples and mixes their labels proportionally to the size of those patches. However, due to the randomness the source patch selection process, it may select a region that does not contain any informative pixel about the source object, and the label mixing according to those uninformative patches mislcads the classifier to learn unexpected feature representation.
In this work, the careful selection of the source patch always helps to contain some information about the source object and thereby solves the class probability assignment problem and helps to improve the model performance and robustness.
2.2 LABEL SMOOTHING
In object classification, the class labels are usually represented by one-hot code i.e., the true labels are expected to have the probability of exactly I while the others to have exactly 0. In other words, it suggests the model to be overconfident which causes overfitting to training dataset. As a result, the models have low performance on unseen test dataset. To alleviate this problem, label smoothing allows to relax the model confidence on the true label by setting the class probability to a slightly lower value e.g., lower than 1. As result, it guides the model to be more adaptive instead of being over-confidess and ultimately improves the model robustness and performance (Szegedy et al. 2016). Our method also mixes the class labels and enjoys the benefit of label smoothing.
2.3 SALIENCY DETECTION
Saliency detection aims to simulate the natural attention mechanism of human visual system (HVS) and can be classified into two main categories. The first one is a bottom-up approach (Cheng et al.,
2014 Zhu et al. 2014 Li et al. 2015 Zhou et al. 2015 Achanta et al.) 20099] Li et al. 2013 Hou
& Zhang 2007 Qin et al. 2015 Peng et al. 2016: Lei et al. 2016 Montabone & Soto 2010) that
focuses on exploring lowelevel vision features. Some visual priors that are inspired by the HVS properties are utilized to describe salient object. [Cheng et al. (2014) utilized a contrast prior and proposed a regional contrast based salient object detection algorithm. Zhu et al. (2014) introduced robust background measure in optimization framework to integrate multiple low level cues to obtain clean and uniform saliency maps. [Li et al. (2015) optimized the image boundary selection by boundary removal mechanism and then used random walks ranking to formulate pixel-wised saliency maps. Zhou et al.] (2015) proposed a saliency detection model where the saliency infor- mation is propagated using manifold ranking diffusion process a graph. In addition, some
3

---

Published as a conference paper at ICLR 2021
Iagge
Image
Salumty Source Image
Selecting the Peak Rleginn of the Saliency Map
Sclorting the Source Punch Brsed Region
Mining Source Patdi Augmented Image Target Iniage
Figure 2:
The proposed SaliencyMix data augmentation. We first extract the saliency map of the source image that highlights the regions of interest. Then we select a patch around the peak salient pixel location and mix it with the target image.
traditional techniques are also introduced to achieve image saliency detection, such as frequency domain analysis (Achanta et al., 2009), sparse representation (Li et al. 2013), log-spectrum (Hou & Zhang. 2007) cellular automata (Qin et al. (015)) low-rank recovery (Peng et al. 2016). and Bayesian theory (Lei et al. 2016 Hou & Zhang (2007) proposed a spectral residual method that focuses on the properties of background. Achanta et (2009) proposed a frequency tuned approach that preserves the boundary information by retaining sufficient amount of high frequency contents. Montabone & Soto (2010) introduced a method that was originally designed for a fast human detec- tion in scene by proposing novel features derived from a visual saliency mechanism. Later on this feature extraction mechanism was generalized for other forms of saliency detection.
The second one is a top-down approach which is task-driven and utilizes supervised learning with labels. Several deep learning based methods have been proposed for saliency detection (Deng et al.) 2018 Liu et al., 20188 Zhhang et al. 2018a Qin al.) 2019). Deng et al. (2018) proposed recur- rent residual refinement network (R3Net) equipped with residual refinement blocks (RRBs) to more accurately detect salient regions. Contexts play an important role in the saliency detection task and based on that Liu et al. (2018) proposed pixel-wise contextual attention network, called PiCANet, to learn selectively attending to informative context locations for each pixel. Zhang et al. 20) in- troduced multi-scale context-aware feature extraction module and proposed bi bi-directional message passing model for salient object detection. Zhang et al. (2018b) focused on powerful feature extrac- tion and proposed attention guided network which selectively integrates multi-level contextual information in a progressive manner. Recently, [Qin et al. (2019) proposed a predict and refine ar- chitecture for salient object detection called boundary-aware saliency detection network (BASNet). The author introduced hybrid loss to train a densely supervised Encoder-Decoder network.
However, despite the high performance of top-down approach, there is a lack of generalization for various applications since they are biased towards the training data and limited to the specific objects. In this study, we require a saliency model to focus on the important object/region in given scene without knowing their labels. As result, we rely bottom-up approach which are unsupervised, scale-ing inver-inva and more robust for unseen data. methods can also be applied where the quality and quantity of data for training the saliency methods may be correlated with the effectiveness of data augmentation. Section 3.3 further explains the effects of different saliency detection algorithm on the proposed data augmentation method.
worth noting that the training based saliency
3 PROPOSED METHOD
Similar to Yun et al. 2019), we cut a source patch and mix it to the target image and also mix their labels proportionally to the size of the mixed patches. But in order to prevent the model from learning any irrelevant feature representation, the proposed method enforces to select a source patch in way that it must contains information about the source object. It first extracts a saliency map of the source image to highlight the objects of interest and then selects a patch surrounding the peak salient region to mix with the target image. Here we explain the process in detail.
3.1 SELECTION OF THE SOURCE PATCH
The goal of saliency detection is to find out the pixels regions that attractive to the HVS and to assign them with higher intensity values (Cong et al. 2019). A saliency detection method produces the visual saliency map, gray-scale image, that highlights the objects of interest and thereby mostly
4

---

Published as a conference paper at ICLR 2021
CIPAR10
Thyy InigeWes
CIFARIO
Tiny Imogehod
3.0
Momatrr 8
Frogestyy Turee
Wonstone Soie
Grite to Canter
Non AI Kon-dal
Canaleatiio
Praqertrr
Nar Salto
Salto
Da
Nan 1o
Barth
Sperr
0.6
399
5189
41.00
Salienzy Detecbon Method
Sabency Detecton Method
Sourc and Targe: Putch Mixing Style
Source and Tarpet Patch Mautea Style
Figure 3: The effect of using different saliency detection methods in the proposed data augmentation on (a) CIFARIO and (b) Tiny-ImageN classification tasks. Different ways of selection and mixing of the source patch with the target image and their effects on (c) CIFAR 10 and (d) Tiny- Imy-ImageNet classification tasks. Performance reported from the average of five and three runs for CIFARIO and TingeemaeeNet, respectively.
focuses on the foreground. Let I, E R is randomly selected training (source) image with label y/ from which a patch will be cut. Then its saliency map detection can be represented as
Jos = J(Ta;
where Ly E 1gW'H represents the visual saliency map of the given source image as shown in Figure 2 where the objects of interest have higher intensity values and ʃ(-) represents a saliency detection model. Then we search for a pixel Iid in the saliency map that has the maximum intensity
value. The i, represent the z and y coordinates of the most salient pixel and can be found
j.j argmax((Tss)
Then select a patch, either by centering the 1'-j th pixel if possible, keeping the 1 - th pixel on the selected patch. ensures that the patch is selected from the object region, not from the background. The size of the patch is determined based combination ratio À which is sampled from the uniform distribution (0, 1) to decide the percentage of an image to be cropped.
3.2 MIXING THE PATCHES AND LABELS
Let J E W is another randomly selected training (target) image with label yt, where the source patch will be mixed. SaliencyMix partially mixes 1 and I, to produce a new training sample La the augmented image, with label yo- The mixing of two images can be defined as
La - M o M' O.,
where La denotes the augmented image. M {0, [ represents binary mask, M' is the complement of M and 3 represents dementtwwse multiplication. First, the source patch location is defined by using the peak salient information and the value of X and then the corresponding location of the mask M is set to 1 and others to O. The element-wise multiplication of M with the source image results with an image that removes everything except the region decided to keep. In contrast, M' performs in opposite way of M i.e., the element-wise multiplication of M' with the target image keeps all the regions except the selected patch. Finally, the addition of those two creates a new training sample that contains the target image with the selected source patch in it (See Figure 2 Besides mixing the images we also mix their labels based on the size of the mixed patches as
Va = Ay. + (1 A)y.:
(4)
where Ya denotes the label for the augmented sample and À is the combination ratio. Other ways of mixing investigated in Section 3.4
3.3 IMPACT OF DIFFERENT SALIENCY DETECTION METHODS
Here we investigate the effect of incorporating various saliency detection methods our Salien- cyMix data augmentation technique. We use four well-recognized saliency detection algorithms
5

---

Published as a conference paper at ICLR 2021
Table 1: Classification performance (average of five runs) of SOTA data augmentation methods on CIFAR-10 and CIFAR-100 datasets using popular standard architectures. An additional "+" sign after the dataset name indicates that the traditional data augmentation techniques have also been used during training.
METOOD
CIFAR-10
CIFAR-10+
TOP-1
ERROR [56]
CIFAR-100
CIFAR-
1E (BASELINE)
10.03 0.26
1,72
22,462 0,55
REANET-18 CUTOUT
9.31+0018
3.9940.13
34.98+0.29
21.96++0.24
RESNET-18
9.44:10.34
3.784-0.12
34.4240.27
19.42-10.23
RESNET-18- SALIENCYMIX
7.59.10.22
3.6510.110
28.73...13
19.29.10.21
RESNET-SU (BFFF/SSBBAELLI (BASELINE)
12.1410.955
1.9810111
36.48100.50
21.5830.43
RESNET-50- CUTOUT
0.84:0.77
3.86200.25
32.97:00.74
31.38:20.0
REsNET-50- CUTMiX
9.16:10.38
3.6110.13
51.65:10.61
18.72:10.23
RESNET-50 + MALIENYMIXX
6.81:20.30
3.46:20.08
24.89-10.39
18.57:10.29
WIDERESNET- 10 (BASELINE)
0.97100.22
3.8710008
26, 26.06:00.12
18. 50:10.08
WIERRISNET-28-1 CUTOUT
5.54:00.88
3 3.06+-0116
23.9440.15
18 18.41+0.277
WIDERESNET AUTOAUGMENT
2.60::0.10
17.10:10.30
WimERISNNTT-22 28- 10 PUZZz XMIX (200 (POCHS)
16.23
WInERISNETT28-111 CUTMIX
5.18-+0.0
2.87+00.16
23.21+0.20
16. 06.66-+-.200
WIDERESNEET28- SALIECYMIX
4.04::0.33
2.76#00007
19.45:10.22
16, 16.56-20.17
(Montabone & Soto 2010; Hou & Zhang 2007
Achanta et al. 2009: Qin et al. 2019), and perform
experiments using ResNet- 18 as a baseline model on CIFAR- 10 dataset and ResNet-50 as a baseline model on tical saliency models (Montabone & Soto 2010 Hou & Zhang| |007| Achanta et al.] (009) work on any size of images. But the learning based model i.e.. Jiin et al. (20T9) are not scale invariant and it should resizes the input image to 224 x 224 and the resulting saliency map is scaled back to the original size. Figure[]a-b) show that Montabone & Soto (2010) performs better on both the datasets and the effects are identical on CIFAIO and ImageNet datasets. As a result, Montabone & Soto 2010) used extract the saliency map in the proposed data augmentation technique.
dataset for 200 epochs and 100 epochs. respectively. Note that the statis-
3.4 DIFFERENT WAYS OF SELECTING AND MIXING THE SOURCE PATCH
There several ways select the source patch and mix it with the target image. In this section, explore those possible schemes and examine their effect on the proposed method. We use ResNet- 18 and ResNet-50 architectures with SaliencyMix data augmentation and perform experiments on CIFAR-10 10 and Tiny-ImageNer datasets, respectively. We consider five possible schemes: (i) Salient to Corresponding, that selects the source patch from the most salient region and mix it to the corre- sponding location of the target image; (ti) Salient to Salient, that selects the source patch from the most salient region and mix it to the salient region of the target image; (iii) Salient to Non- Kon-Salient, that selects the source patch from the most salient region but mix it to the non- non-salient region of the target image; (iv) Non-Salient to Salient, that selects the source patch from the non- non-salient region of the source image but mix it to the salient region of the target image; and (v) Non-Salient to Non- Salient, that selects the source patch from the non-salient region of the source image and also mix it to the non-salient region of the target image. To find out the non-salient region, use the least important pixel of an image.
Figure Ee-d show the classification performance of the proposed SaliencyMix data augmentation with the above mentioned selection and mixing schemes. Similar to the Section 3.3 the effects of the proposed method are similar for CIFAR10 and Tiny-ImageNet datasets. Both the Non-Salient to Salient and Non-Salient to Non-Saliemi select the source patch from the non-salient region of the source image that doesn't contain any information about the source object and thereby produce large classification error compared to the other three options where the patch is selected from the most salient region of the source image. It justifies our SaliencyMix i.e.. the source patch should be selected in such way so that it must contain information about the source object. On the other hand, Salient to Salient covers the most significant part of the target image that restricts the model from learning its most important feature and Salient to Non-Salient may not occlude the target object which is necessary to improve the regularization. But Salient to Corresponding keeps balance by changeably occluding the most important part and other based on the orientation of the source and target object. Consequently, it produces more variety of augmented data and thereby achieves the lowest classification error. Also, it introduces less computational burden since only the source image saliency detection is required. Therefore, the proposed method uses Salient to Corresponding as the default selection and mixing scheme.

---

Published as a conference paper at ICLR 2021
Performance comparison (the best performance) of SOTA data augmentation strategies on ImageNet classification with stan- dard model architectures.
Table 2:
Table 3: Impact of SaliencyMix trained model on transfer learning to object detec- tion task. The results are reported from the average of three runs.
Tor-1 EXROR (%)
ToP-5
ERROR (%)
RESNET-50 (BASELINE)
23.68
7.03
RESNET-50 + CUrOUT
22.93
6.66
RESNET-50 STOPH
22.46
6.27
IMAGENET
DETECTION
RESNET-500 MIXUP
RESNET-50 MANIFOLD MIXUP
22.50
22.58
6.21
6.40
BACKBONE NETWORK
ToP-1 (%)
CLS. ERR.
(F-RCNN)
(MAP)
RESNETI-50 DRUPBLOCK
RESNET-50 AUTOAUGMENT
RESNET-50 CUTMIX
22.40
21 40
21.87
6.20
5,98
5.92
RESNET-50 (BASELINE)
CUTOUT-TRAINE TRAINED
22.93
23.68
77.17 (+0.46)
76.71 (+0.00)
RESNET-50+ PUZZLEMIX
21.24
MIXUP-TRAINEL
77.98 (+1.27)
RESNET-50- SALIENCYMIX
21.26
S.76
CUTMIX TRAINED
21.40
78.31 (+1.60)
RESNET-IO1 (BASBLINE)
21.87
6,29
SALIENCY MIX-TRAINER
78.48 (+1.77)
RESNET-101 CUTOUT
20.72
RESNET-101- + MIXUP
20.52
5.28
RESNET-101 CUTMIX
20.17
5.24
RESNET-101 SALIENCYMIX
20.09
5.15
EXPERIMENTS
We verify the effectiveness of the proposed SaliencyMix data augmentation strategy on multiple tasks. We evaluate our method on image classification by applying it on several benchmark image recognition datasets using popular SOTA architectures. We also use the SaliencyMir trained model and fine-tune it for object detection task to verify its usefulness in enhancing the detection perfor- mance. Furthermore, we validate the robustness of the proposed method against adversarial attacks. All experiments were performed on PyTorch platform with four NVIDIA GeForee RTX 2080 Ti GPUs
4.1
IMAGE CLASSIFICATION
4.1.1
CIFAR-10 AND CIFAR-100
There are 60, 000 color images of size 32 x 32 pixels both the CIFAR-10 and CIFAR- 100 datasets Krizhevsky 2012) where CIFAR-10 10 has 10 distinct classes and CIFAR- 100 has 100 classes. The number of training and test images in each dataset is 50, 000 and 10, 000, respectively. We apply several standard architectures: a deep residual network (He et al. 2016) with a depth of 18 (ResNet- 18) and 50 (eesNet-50), and a wide residual network (Zagoruyko & Komodakis 2016) with a depth of 28, widening factor of 10, and dropout with drop probability of p 0.3 in the convolutional layers (WideResNeet22-100) We train the networks for 200 epochs with a batch size of 256 using stochastic gradient descent (SGD), Nesterov momentum of 0.9, and weight decay of 5e 4. The initial learning rate was 0.1 and decreased by factor of 0.2 after each of the 60, 120, and 160 epochs. The images are normalized using per-channel mean and standard deviation. We perform experiments with and without a traditional data augmentation scheme where the traditional data augmentation includes padding, random cropping, and horizontal flipping.
Table D presents the experimental results on CIFAR datasets where the results are reported on five runs average. It can be seen that for each of the architectures, the proposed SaliencyMix data aug- mentation strategy outperforms all other methods except PuzzleMix (Kim et al., 2020). It is worth noting that PuzzleMix (Kim et al. 2020) and Autonng (Cubuk et al. 2019) require additional optimization process find out the best augmentation criterion, thereby introduces computational burden. On the other hand, rest the methods do not require such process. The proposed method achieves the best known top-1 error of 2.76% and 16.56% for WideResNet-28-- on CIFAR- 10 and CIFAR- 100 datasets, respectively. Moreover, Saliency Mix shows significant performance improve- ment over CutMix (Yun al. 2019) when applied without any traditional augmentation technique. reduces the error rate by 1.85% % 2.35%, and 1.14% on CIFAR-IO dataset when applied with ResNet- 18, ResNet-50 and Net architectures, respectively. Using the same architectures, reduces the error rate by 5.69%, 6. 76%, and 3. 3.76% on CIFAR- 100 dataset, respectively.
4.1.2 IMAGENET
ImageNet (Olga et al. 2015) contains 1.2 million training images and 50, 000 validation images of 1000 classes. To perform experiments ImageNet dataset, apply the same settings used in
7

---

Published as a conference paper at ICLR 2021
Bastinee (ReshNet5))
Cutout
Mixup
Cutbix
SaiiencyMis
Towpe Toapet
Augietted
Mouraa Tor:
Grldnn Ratherr
CAM for Taltoee
Marmr
"Tgertat
Tape Cil
Kiaup
Cuookt
SaleagMia
Visia
(b) CAM visualizations on augmented images. Left most column shows the original images, top row shows the input augmented images by different meth- ods, middle and bottom rows show the CAM for "Golden Retriever', and 'Tiger Cat' class, respectively. The proposed method and Yun et al. (2019) improves the localization ability of the model.
(a) CAM visualizations on un un- augmented images. The proposed data augmentation method guides the model precisely focus on target object.
Figure 4: Class activation map (CAM) for models that trained with various data augmentation techniques. The images are randomly taken from ImageNet (Olga et al. 2015) validation set.
Yun et al. (2019) for a fair comparison. We have trained our SaliencyMix for 300 epochs with an initial learning rate of 0.1 and decayed by a factor of 0.1 at epochs 75, 150, and 225, with a batch size of 256. Also, the traditional data augmentations such as resizing. cropping, flipping, and jitters have been applied during the training process. Table presents the ImageNet experimental results where the best performance of each method is reported. SaliencyMix outperforms all other methods in comparison and shows competitive results with PuzzleMix (Kim et al. (020). It drops top-1 error for ResNet-50 by 1.66%, 1,31, and 0. 1.14 over Cutout, Mixup and CutMix data augmentation, respectively. For ResNet- 101 architecture, SaliencyMix achieves the new best result of 20.09% top-1 error and 5.15% top-5 error.
4.2 OBJECT DETECTION USING PRE-TRAINED SALIENCYMIX
In this section, we use the SaliencyMix trained model to initialize the Faster RCNN (Shaoqing et al,, 2015) that uses ResNet-50 backbone network and examine its effect object detection task. The model is fine-tuned on Pascal VOC 2007 and 2012 datasets and evaluated on VOC 2007 test data using the mAP metric. We follow the fine-tuning strategy of the original method (Shaoqing et al. 2015). The batch size, learning rate, and training iterations are set to 8, 4e 3, and 41K, respectively and the learning rate is decayed by factor of 0.1 33K iterations. The results are shown in Table Pre-training with CutMix and SaliencyMix significantly improves the performance of Faster RCNN. This is because in object detection, foreground information (positive data) is much more important than the background (Lin et al. 2017). Since SaliencyMix helps the augmented image to have more foreground or object part than the background, it leads to better detection performance. It can be seen that SaliencyMix trained model outperforms other methods and achieves a performance gain of +1.77 mAP.
4.3 CLASS ACTIVATION MAP (CAM) ANALYSIS
Class Activation Map (CAM) Zhou et al. 2016) finds out the regions of input image where the model focuses to recognize an object. To investigate this, we extract CAM of models that trained with various data augmentation techniques. Here we use a vanilla ResNet-50 model equipped with various data augmentation techniques and trained on ImageNet (Olga et al. 2015). Then we extract CAM for un-augmented images as well as for augmented images. Figure 4 presents the experimental results. Figure [4a shows that the proposed data augmentation technique guides the model to precisely focus on the target object compared to others. Also, [ the similar effect when we search for a specific object in a scene with multiple objects. It can be seen that Mixup (Zhang et al., 2017) has severe problem of being confused when trying to recognize an object because the pixels are mixed and it is not possible to extract class specific features. Also, Cutout (Devries & Taylor 2017) suffers disadvantages due to the uninformative image region. On the other hand, both the CutMix (Yun et al., 2019) and Saliency effectively focuses on the corresponding features and precisely localizes the two objects in the scene.
8

---

Published as a conference paper at ICLR 2021
Table 4: Performance comparison on adversar- ial robustness. Top-1 accuracy (%) of various data augmentation techniques on adversarially perturbed ImageNet validation set.
Table 5: Training time comparison of various data augmentation techniques using ResNer- 18 architecture on CIFAR-10 10 dataset.
BASELINE
CUROUr
MixUe
CCTMix
SALIENCTAR
BASELIKE
CUIGUT
MixUP
CorMix
SALBNEYMIX
TIME (HoUs)
0.83
0.84
0.87
0.89
82
11.3
244
31.0
32.96
4.4 ROBUSTNESS AGAINST ADVERSARIAL ATTACK
Deep learning based models vulnerable to adversarial examples i.e., they be fooled by slightly modified examples even when the added perturbations are small and unrecognizable (Szegedy al. 2014 Goodfellow et al., 2015 Madry al. 2017). Data augmentation helps to increase the robustness against adversarial perturbations since it introduces many unseen image samples during the training (Madry al., 2017). Here we verify the adversarial robustness of model that is trained using various data augmentation techniques and compare their effectiveness. Fast Gradient Sign Method (FGSM) (Madry al. 2017) is used to generate the adversarial examples and ImageNet pre-trained models of each data augmentation techniques with ResNer-50 architecture is used in this experiment. Table 4 reports top-1 accuracy of various augmentation techniques on adversarially attacked ImageNet validation set. Due to the appropriate feature representation learn- ing and focusing on the overall object rather than a small part, SaliencyMix significantly improves the robustness against the adversarial attack and achieves 1.96% performance improvement over the nearly comparable method CutMix (Yun et al. 2019)
4.5 COMPUTATIONAL COMPLEXITY
We investigate the computational complexity of the proposed method and compare it with other data augmentation techniques in terms of training time. All the models are trained on CIFAR- 10 dataset using ResNet-18 18 architecture for 200 epochs. be seen that SaliencyMix requires a slightly longer traming time compared to others, due to saliency map generation. But considering the performance improvement, it can be negligible.
presents the training time comparison. It can
5 CONCLUSION
We have introduced an effective data augmentation strategy, called SaliencyMix, that is carefully designed for training CNNs to improve their classification performance and generalization ability. The proposed SaliencyMix guides the models to focus on the overall object regions rather than small region of input images and also prevents the model from learning in-appropriate feature representation by carefully selecting the representative source patch. It introduces a little compu- tational burden due to saliency detection, while significantly boosts up the model performance and strengthen the model robustness on various computer vision tasks. Applying SaliencyMix with WideResNet achieves the new best known top-1 error of 2. 2.76% and 16.56% on CIFAR- and CIFAR-100 100, respectively. On ImageNet classification, applying SaliencyMix with Re and ResNet-101 obtains the new best known top-1 error of 21.26% and 20.099 09%. respectively. On ob- ject detection, using the SaliencyMix trained model to initialize the Faster RCNN (ResNet-50 backbone network) and fine-tuning leads to a performance improvement by +1.77 mAP. Further- more, SaliencyMix trained model is found to be more robust against adversarial attacks and achieves 1.96% accuracy improvement on adversarially perturbed ImageNet validation set compared to the nearly comparable augmentation method. Considering more detailed and/or high level semantic information for data augmentation will be our future work.
ACKNOWLEDGMENTS
This research was supported by Basic Science Research Program through the National Re- search Foundation of Korea (NRF) funded by the Ministry of Science, ICT & Future Planning PO0ISRICIB3008111 and Basic Science Research Program through the National Research Foun- dation of Korea (NRF) under Grant (NRF-020R 1F 1F A1050014].

---

Published as a conference paper at ICLR 2021
REFERENCES
R. Achanta, S. Hemami, F. Estrada, and S. Susstrunk. Frequency-tunee salient region detection. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1597-1604, June 2009. doi: 10. R..1009CCVRR.2009520066
Yoshua Bengio, Frédéric Bastien, Arnaud Bergeron, Nicolas Boulanger-Lewanddoski, Thomas Breuel, Youssouf Chherawala, Moustapha Cisse, Myriam Côté, Dumitru Erhan, Jeremy Eu- stache, Xavier Glorot, Xavier Muller, Sylvain Pannetier Lebeuf, Razvan Pascanu, Salah Rifai, François Savard, and Guillaume Sicard. Deep learners benefit more from out-of-distribution ex- amples. In Proceedings of the Fourteenth International Conference on Artificial Intelligence and Statistics, volume 15 of Proceedings of Machine Learming Research, pp. 164-172, Fort Laud- erdale, FL, USA, 11-13 Apr 2011. PMLR. URL http::/ppocceedinggg/ ml press/
y15/bengiolis html
Ming-Ming Cheng, Niloy J Mitra, Xiaolei Huang, Philip HS Torr, and Shi-Min Hu, Global contrast based salient region detection. IEEE transactions on pattern analysis and machine intelligence, 37(3):569-582, 2014.
R. Cong. J. Lei, H. Fu, M. Cheng, W. Lin, and Q. Huang. Review of visual saliency detection with comprehensive information. IEEE Transactions on Circuits and Systems for Video Technology, 29(10):2941-2955 Oct 2019. ISSN 1558-2205. doi: 10. (009//CSSTT2088228770866
F. D. Cubuk, B. Zoph, D. Mané, V. Vasudevan, and Q V. Lee Autoaugment: Learning augmentation strategies from data. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 113-123, 2019. doi: (0.1199/CVPR.201900000
Zijun Deng, Xiaowei Hu, Lei Zhu, Xuemiao Xu, Jing Qin, Guoqiang Han, and Pheng-Am Heng. R3net: Recurrent residual refinement network for saliency detection. Proceedings of the 27th International Joint Conference Artificial Intelligence, pp. 684- AAAI Press, 2018
Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. ArXiv, 000/7708.4455 2017.
Mark Everingham, Luc Van Gool, Christopher Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303- 338. 06 2010. doi: 10. 1007/s 1263-009-0275-4,
Robert Geirhos, Patricia Rubisch, Claudio Michaelis, Matthias Bethge, Felix A. Wichmann, and Wieland Brendel. Imagenet-trained CNNs are biased towards texture; increasing shape bias im- proves accuracy and robustness. In International Conference on Learning Representations, 2019.
Ghiasi Golnaz, Lin Tsung- Yi, and V. Le Quoc. Dropblock: A regularization method for convolu- tional networks. In Neural Information Processing Systems (NeurIPS), 2018
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy, Explaining and harnessing adversarial examples. arXiv preprint, 01 2015.
http///ss//111...1 11.. 6572
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Paltern Recognition (CVPR), pp. 770-778, June 2016. doi: 10.1109/CVPR.2016.20
X. Hou and L. Zhang. Saliency detection: A spectral residual approach. In 2007 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1 1-8, June 2007. doi: 00.1009/CVPR..200.3333333
Choe Junsuk and Shim Hyunjung. Attention-based dropout layer for weakly supervised object localization. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2219-2228, 2019.
Jang- Kim, Wonho Choo, and Hyun Oh Song. Puzzle mix: Exploiting saliency and local statistics for optimal mixup. 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
10

---

Published as a conference paper at ICLR 2021
Alex Krizhevsky, Ilya Sutskever, and Geoffrey Hinton. Imagenet classification with deep convo- lutional neural networks. In Neural Information Processing Systems (NeurIPS), pp. 1097-1105, January 2012. doi: 10. 10.1145/3065386.
Y. Lecun, L. Bottou, Y. Bengio, and P. Haffner. gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278 222-2224,,,,, Nov 1998. ISSN 1558-2256. doi: 10.1 10.1109/5.72679.
Jianjun Lei, Bingren Wang, Yuming Fang, Weisi Lin, Patrick Le Callet, Nam Ling, and Chunping Hou. A universal framework for salient object detection. IEEE Transactions on Multimedia, 18 (9):1783-1795, 2016.
Joseph Lemley, Shabab Bazrafkan, and Peter Corcoran. Smart augmentation learning an optimal data augmentation strategy. IEEE Access, 5:5858-5866 988988 2017.
Changyang Li, Yuchen Yuan, Weidong Cai, Yong Xia, and David Dagan Feng. Robust saliency detection via regularized random walks ranking. In Proceedings of the IEEE conference com- puter vision and pattern recognition, pp. 2710-2717, 2015.
Xiaohui Li, Huchuan Lu, Lihe Zhang, Xiang Ruan, and Ming Hsuan Yang. Saliency detection via dense and sparse reconstruction. In Proceedings of the IEEE international conference on computer vision, pp. 2976-2983, 2013.
Lin, P. Goyal, R. Girshick, K. He, and P. Dollár, Focal loss for dense object detection. In 2017 IEEE International Conference Computer Vision (ICCV), pp. 2999-3007, Oct 2017. doi: 10.1 D0.110//CCV.2011..322
Nian Liu, Junwei Han, and Ming-Hsuan Yang. Picanet: Learning pixel-wise contextual attention for saliency detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3089-3098, 2018
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant adversarial attacks. ArXiv, abs/1706.0608. 2017.
Sebastian Montabone and Alvaro Soto. Human detection using a mobile platform and novel features derived from visual saliency mechanism. Image and Vision Computing, 28(3):39-442,, 2010.
Srivastava Nitish, Hinton Geoffrey, Krizhevsky Alex, Sutskever Ilya, and Salakhutdinov Rus- lan. Dropout: A simple way to prevent neural networks from overfitting. Journal of Ma- chine Learning Research, 15:1929-1958, 2014. URL http:// /mmb::://mmr...// wtggpapprs/vls..
Sri arivastaval4p html
Russakovsky Olga, Deng Jia, Su Hao, Krause Jonathan, Satheesh Sanjeev, Ma Sean, Huang Zhi- heng, Karpathy Andrej, Khosla Aditya, Bernstein Michael, C. Berg Alexander, and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision, 115 (3):211-252, 2015.
Houwen Peng, Bing Li, Haibin Ling, Weiming Hu, Weihua Xiong, and Stephen J Maybank. Salient object detection via structured matrix decomposition. IEEE transactions pattern analysis and machine intelligence, 818 11-8888.222. 2016.
Xuebin Qin, Zichen Zhang, Chenyang Huang, Chao Gao, Masood Dehghan, and Martin Jagersand. Basnet: Boundary-awarr salient object detection. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Yao Qin, Huchuan Lu, Yiqun Xu, and He Wang. Saliency detection via cellular automata. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 110-119, 2015.
Ren Shaoqing, He Kaiming, Girshick Ross, and Sun Jian. Faster r-cnn: Towards real-time object detection with region proposal networks, In Neural Information Processing Systems (NeurIPS), 2015.
11

---

Published as a conference paper at ICLR 2021
K. K. Singh and Y. Lee. Hide-and-seek: Forcing network be meticulous for weakky-supeevee object and action localization. In 2017 IEEE Interaational Conference on Computer Vision (ICCV), pp. 3544-3553,201 2017.
C. Szegedy, V. Vanhoucke, Ioffe, Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In 2016 IEEE Conference on Computer Vision and Pallem Recognition (CVPR), pp. 2818-2826, 2016.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. In International Conference Learning Reprexentations, 2014. URL http:tpp///aaxi..orr/a///////// 6199
DeVries Terrance and W Taylor Graham. Inproveddeegllarzzaaioooo convollti neural networks with cutout. arXiv preprint, 2017. URL http:/tss/aaxxti.crg/aaa///////s 04555 04552
Jonathan Tompson, Ross Goroshin, Arjun Jain, Yann Lecun, and Christoph Bregler. Efficient object localization using convolutional networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 448-656, 06 2015. doi: 10.1009/CVRR22015.79666
Ren Wu, Shengen Yan, Shan, Qingqing Dang, and Gang Sun. Deep image: Scaling up image recognition. arXiv preprint, 01 2015. httpp//s/aaa....o///////////////
02876
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. Interna- fional Conference on Computer Vision (ICCV), 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. Procedings of the British Ma- chine Vision Conference 2016, 2016. doi: 10.5244/c.30.87. URL http::/tpx.xooo doi. orl/10.
2244//.30.87
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond em- pirical risk minimization. arXiv preprint, 2017. URL fttp:://aarxiv.or//bbb//// 09412
Lu Zhang, Ju Dai, Huchuan Lu, You He, and Gang Wang. A bi-directional message passing model for salient object detection. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 1741-1750, 2018a.
Xiaoning Zhang, Tiantian Wang. Jinqing Qi, Huchuan Lu, and Gang Wang. Progressive attention guided recurrent network for salient object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 714-722, 2018b
B. Zhou, A. Khosla, A. Lapedriza, A. Oliva, and A. Torralba. Learning deep features for discrimina- tive localization. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2921-2929, June 2016. doi: 10. 19.100922V..2
Li Zhou, Zhaohui Yang. Qing Yuan, Zongtan Zhou, and Dewen Hu. Salient region detection via inte- grating diffusion-based compactness and local contrast. IEEE Transactions on Image Processing, 24(11):3308- -3200, 2015.
Wangjiang Zhu, Shuang Liang, Yichen Wei, and Jian Sun. Saliency optimization from robust back- ground detection. In Proceedings of the IEEE conference computer vision and pattern recog- nition, pp. 281 2-22,, 2014.
Zhong Zhun, Zheng Liang, Kang Guoliang, Li Shaozi, and Yang Yi. Random erasing data augmen- tation. arXiv preprint, 2017. URL httph:tt///// /ats://///xiv..// www.wwwww.................oooeee hwwg//bs/iii 04896
12


Reviews:
[{"review_id": "rW4xKETaz4_", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "*Summary and contributions:*\nThis paper proposes a new data augmentation strategy to train image classifiers and object detectors. The key insight is to use an image saliency signal to guide where to crop-and-paste images when mixing them. The paper includes an exploration of the design space of such approach, and multiple experimental results showing the empirical superiority of the proposed approach compared to existing data augmentation strategies.\n\n\n*Significance:*\nThe paper is interesting because it provides a new trick in the bag of tricks that is both simple to understand, reasonable to argue for, and (now) has good empirical support (for classification, detection, and adversarial attack robustness).\n\n\nOriginality: \nLimited. Although no previous work provides the experimental results presented here, the results are expected. This work is good A+B incremental work.\n\n\n*Strengths:*\n* Overall the paper read easily. The general argumentation, method description, and experiments are all reasonably well described.\n* Simple ideas that provide good results. In retrospectively it might seem obvious, yet not explored before.\n* Widely applicable for all methods using pre-trained image classifiers.\n \n*Weaknesses:*\n* Although the experimental section is good, some elements are missing. E.g. adding non-data-augmented as reference point in the plot, or considering CAM as a saliency strategy.\n* Some sections of the text would benefit from revisiting the English.\n* The method implicitly relies on having “simple images” with one dominant foreground object  like the ones in CIFAR and ImageNet. The saliency / “object of interest”  detection method depends on these characteristics. Ideally the paper would be more upfront on these assumptions. Especially in the context of the “Rethinking ImageNet Pre-training”, ICCV 2019 work.\n* The related work section for saliency is very partial.\n* Some of the saliency methods evaluated use training data, even the ones that do not have been tuned using additional data. The paper would benefit from a discussion of this additional information.\n\n\n*Correctness:* Paper seems correct. There are minor shortcomings in the experimental protocol, but nothing that would foreseen invalidating the main conclusions of the paper.\n\n*Clarity:* Paper presentation is clear.\n\n\n\n\n\n\n*Relation to prior work:*\nRelated work section has a reasonable extent. \nRegarding data augmentation, the paper compares with the main methods.\nThe text mentions Lemley 2017, however I think it would be also worth mentioning AutoAugment Cubuk 2019; and justifying why it is not included in the results comparison.\n\nThe saliency detection is very partial, and does not cover the main works in the area. \nFor one the text does not clarify “which kind” of saliency is considered (where will a human look ? which are the main objects of the scene ? which is the main object of the scene ?). Depending on which one, discussing the main performers in the related benchmarks (e.g. table in Qin 2019 paper) seem relevant. \nFrom what I grasp, the proposed method would actually want to have as input a weakly supervised class-conditional segmentation. And “saliency” is used as a proxy for this. Discussing the relation to (image-level labels) weakly supervised segmentation would also be welcome.\n(I would guess it could provide even better results, but at a much higher computational and system complexity cost). In particular CAM is discussed in section 4.3. Would that not be a task-specific way to obtain the desired “semantically important regions”  ?\n\n\nWhen preparing the camera ready, please consider discussing the concurrent work of https://arxiv.org/pdf/2009.06962.pdf which seems related (seems recent enough, ICML 2020, to give the benefit of presumed concurrency).\n\n\n*Reproducibility:*\nThe overall algorithm is simple to understand and re-implement.\nThe selected saliency method “Wand and Dudek, 2014” seems to be a video saliency method. From a quick inspection of that paper it is not immediately clear to me how to transpose it for single image saliency. Since this method is used in most experiments, providing more details of the implementation and its parameters are necessary to reproduce the key results.\n\n\n*Specific per-section feedback:*\n\nSection 1:\n- every field: is too broad of a statement, remove/rephrase the first sentence.\n- extremely complex -> complex\n- generalizability -> generalization\n- undesired to the CNN since -> undesired since\n- does not allow … to have any uniformative pixel: double negative, consider simplifying.\n- semantically important region: these are task dependent. How do you ensure the saliency to match the task / semantics ?\n\n\nSection 2.1:\n- comes into account that aims: unclear, please rephrase.\n\n\nSection 2.2:\n- unable to … 1 or 0: this is not true, please rephrase.\n- intermediate values: you mean closer to 0.5 ?\n- Thereby, helps: unsure if this is proper English.\n\n\nSection 2.3: See comments above regarding related work.\n- Which kind of saliency is considered here ? Which are the relevant benchmarks ? Which training data / evaluation data are these methods bringing (indirectly) into the system ?\n- BAsNet for example, was trained on 10k images. Why not simply include these (and their mask) as part of the pretraining when considering some of the baselines ?\nSection 3.1:\n- selected training image -> selected training (source) image. In general do give hints for the subscript meaning of I_s, I_vs, etc.\n- Why only one pixel with maximum intensity ? What happens if (due to quantization) two pixels have the same value ?\nThis is clarified later in the text, but some context would be welcome in the mention here.\nFigure 3:\n- Add non-augmented result bar as reference point. \n- Consider showing the five points per bar, or some hint for the variance in these results. From the plot it is left unclear if the fluctuations across methods are significant or minor (since not reference point, nor sense of the variance).\nSection 3.3:\n- Tiny-Imagenet: saliency methods tend to _not_ be scale invariant (in particular trained methods like BasNet). How is this handled ? Why would one expect Tiny-Imagenet to provide conclusive data ?\n- What about CAM, or any other class-conditioned saliency / weakly supervised segmentation method ?\nSection 3.4:\n- found out -> consider\n- What about “Salient to Random“ ? That seems a reasonable option too ?\n- are identical -> are similar\n\n\nSection 4.1:\n- SOTA top-1 error: there are 20 methods that claim better results in https://paperswithcode.com/sota/image-classification-on-cifar-10 and https://paperswithcode.com/sota/image-classification-on-cifar-100 that claim better results. Maybe temperate the “SOTA” claim, with phrasings like “best known results for model X”  or similar. One specific model result, far from best known, does not constitute “state of the art”  in my understanding.\n\n\nSection 4.2:\n- Because in -> This is because in\n\n\n*Updates after reviews and authors feedback:*\n\nThe updates from the author are appreciated and make the arguments of the paper clearer.\nAfter reading the other reviews and discussions, I have downgraded my score to \"7: Good paper, accept\".\n\n(Please note that in the current pdf the table at the top of page 7 has formatting issues.)\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "XsNAkufofid", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper is based on an interesting observation that previous data augmentation tricks cut&mix may select regions that do not contain useful information. Instead, this paper use saliency models to detect the salient regions first and then cut and mix these salient regions in a source image.\n\nThis observation is interesting and worth trying. However, the contribution might be too limited for a ICLR conference paper:\n1. as cut&mix cuts bigger blocks in an image, the target object is more likely to be selected. Also for most image classification dataset, the images are quite iconic, so the improvement on classification tasks are limited (as shown in Tab. 1, C10+ and C100+). This might help with detection as it may train models to focus on the most discriminative part of the image, but recent works show that there is no direct correlation between between the performance of the same backbone on detection tasks and classification tasks. In addition, the author didn't provide analysis on what causes the 1.8% improvement on detection tasks (better on smaller objects?) So it's not clear how helpful this trick is.\n2. While the method is simple, I expect either some mathematic proof or this method works well on various tasks. The paper didn't have any proof or statistical analysis. This paper didn't either show if the proposed method will work on more tasks (for example segmentation or GAN? the detection provided in this paper is using the backbone initialized from classification, during training faster-rcnn it seems that the trick is not used).\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "HlJaeFasEco", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a new augmentation method based on CutMix. The authors find out that randomly selecting may mix background textures and this will mislead the model. So, they propose to use saliency maps to control the selection of mixed patches, which is called SaliencyMix. This idea seems easy and reasonable, many experiments are conducted to prove the effectiveness of the proposed method. However, the experiments’ results fail to show the ability of the method, and some explanation is missed.\n \nPositive:\n1.     The idea is simple and clear, the paper is well organized and easy to follow.\n2.     The experiments are comprehensive, including classification and transfer learning.\n \nWeakness:\n1.     The main concern is the effectiveness of the proposed method. According to the authors’ experiments, the improvement over CutMix is very limited on all datasets. Especially on ResNet-101, the promotion over CutMix is only 0.08.\n2.     According to the authors’ ablation study in sec. 3.3, only using fast self-tuning background subtraction produces better results than CutMix. Why other methods even worse than CutMx? What’s the core reason for the improvement of using fast self-tuning background subtraction?\n3.     The authors use batchsize=256, lr=0.1 for CIFAR training, while usually batchsize=128, lr=0.1 is used in previous works (Cutout). And as described in [1], the learning rate should be increased linearly with batchsize. This change of hyperparameters may need further explanation.\n \n[1] Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.\n\n\n============Post Rebuttal====================\nAfter reading the feedback from authors, I still have my concerns. The novelty of this paper is too limited for ICLR. I really do not think a combination of CutMix with existing saliency detection method is a novel method. Moreover, the improvement over CutMix is diminishing. These main concerns are not addressed by the authors. So, my final recommendation is still rejection.", "extra_scores": {}, "extra_text": {}}, {"review_id": "AEPHpQ662J", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 9, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a method called SaliencyMix. They improve a method that augments images by adding random patches from other images. The innovation is that they select these patches using a saliency map.\n\nThis paper has an excellent discussion and critique of previous work. They discuss the existing work with a nice summary and then discuss reasons why selecting random patches can have issues. There is a clear argument for their method over selecting patches randomly. They make clear claims that this approach improves performance over randomly selecting patches. The experiments support this with sufficient related work (Cutout, Cutmix) and exploration of other design decisions and aspects about the idea.\n\nThe idea is relatively straightforward and is inline with existing literature. The paper is well executed so there is not much to complain about.\n\nThe availability of the source code is not clear from the text.\n\nA potentially interesting analysis (but not required) is an analysis of the increased runtime in practice.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "8in6Q-dDGIs", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes an improvement on the cutmix strategy of data augmentation, where the source patch is selected not randomly but based on saliency. Results show improvements w.r.t mixup and other related strategies on Imagenet, CIFAR 10/100 and also transfer to object detection\n\nPros:\n- The approach is intuitive and makes sense (which is more than can be said for the baselines of CutMix and MixUp). I think this approach probably starts to get to the heart of why these previous strategies work: they are probably less effective ways of doing what this paper suggests.\n- The results seem quite promising, and the improvements seem significant.\n\nCons:\n- I am a bit surprised that the best strategy is Sal + Corr. I understand the author's reasoning, but I find it strange that it has that big of an effect. If the author's reasoning is correct, I would assume that a random placement of the cut patch would be just as effective. Could the authors try Sal + Random instead?\n- It is well known in the saliency literature that saliency has a center bias. This suggests a baseline where the source patch is always cut from the center. I would suggest the authors add this baseline.\n- I am not sure about the point of using CAM visualizations on augmented images. Perhaps a better visualization might be CAM visualization of the models trained with each kind of visualization on the unaugmented images?", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes an approach to data augmentation to train image recognition models called SaliencyMix, which involves pasting salient regions (as judged by some saliency detector) of one image into another, and mixing the two labels accordingly. Most reviewers generally agreed that the proposed approach is simple -- it is easy to understand the method and its motivation, especially in the context of related augmentation approaches like CutMix -- and has solid experimental results demonstrating its effectiveness.

The main objection the more negative reviewers had to the work is a perceived lack of novelty. In my view it is a new method (even if similar to prior work like CutMix), and as AR5 argues: "this approach probably starts to get to the heart of why these previous strategies work: they are probably less effective ways of doing what this paper suggests." The improvements in Table 1, columns 1 & 3 (CIFAR-10 & CIFAR-100) especially speak to this -- these improvements with traditional augmentation disabled are quite substantial, even though the differences become marginal when moving to the "+" augmented versions of the dataset (as well as in ImageNet). So although the method is indeed similar to CutMix, I agree that it offers valuable insight into why these previous methods work. Besides which the results *do* show improvements over similar methods, even if the improvements are marginal.

Overall, I recommend accepting the paper as it provides useful insight into why prior methods work and proposes a new one that in practice works slightly better. Minor comments for the camera-ready version:

Please revise the writing based on AR4's good suggestions.

Highlighting a comment from AR4:
> BAsNet for example, was trained on 10k images. Why not simply include these (and their mask) as part of the pretraining when considering some of the baselines ?

I recommend including discussion of this important point in the final version of the paper. The learning-based approaches are effectively using additional training data. It's good that a non-learning-based method happens to perform best so that the results remain comparable with prior work, but this should nonetheless be discussed if the learning-based approaches are to be included.

Please remove the blue text coloring (if not already planned -- I'm not sure if this was done as a "diff" for the response).

> Figure 3(a-b) show that Montabone & Soto (2010) performs better on both the datasets and the effects are identical on CIFAR10 and ImageNet datasets

I do not see how Figure 3 shows this. Is "OpenCV Saliency" in the figure using the method from Montabone & Soto (2010)? Please clarify this by making the connection between the bar labels in the figure and the discussion in the text clear for the camera-ready version of the paper.
Decision: Accept (Poster)