Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: This paper proposes an improvement on the cutmix strategy of data augmentation, where the source patch is selected not randomly but based on saliency. Results show improvements w.r.t mixup and other related strategies on Imagenet, CIFAR 10/100 and also transfer to object detection

Pros:
- The approach is intuitive and makes sense (which is more than can be said for the baselines of CutMix and MixUp). I think this approach probably starts to get to the heart of why these previous strategies work: they are probably less effective ways of doing what this paper suggests.
- The results seem quite promising, and the improvements seem significant.

Cons:
- I am a bit surprised that the best strategy is Sal + Corr. I understand the author's reasoning, but I find it strange that it has that big of an effect. If the author's reasoning is correct, I would assume that a random placement of the cut patch would be just as effective. Could the authors try Sal + Random instead?
- It is well known in the saliency literature that saliency has a center bias. This suggests a baseline where the source patch is always cut from the center. I would suggest the authors add this baseline.
- I am not sure about the point of using CAM visualizations on augmented images. Perhaps a better visualization might be CAM visualization of the models trained with each kind of visualization on the unaugmented images?
review_title: Interesting approach; some baselines would be useful

final_score: rating: 7: Good paper, accept
review_content: *Summary and contributions:*
This paper proposes a new data augmentation strategy to train image classifiers and object detectors. The key insight is to use an image saliency signal to guide where to crop-and-paste images when mixing them. The paper includes an exploration of the design space of such approach, and multiple experimental results showing the empirical superiority of the proposed approach compared to existing data augmentation strategies.


*Significance:*
The paper is interesting because it provides a new trick in the bag of tricks that is both simple to understand, reasonable to argue for, and (now) has good empirical support (for classification, detection, and adversarial attack robustness).


Originality: 
Limited. Although no previous work provides the experimental results presented here, the results are expected. This work is good A+B incremental work.


*Strengths:*
* Overall the paper read easily. The general argumentation, method description, and experiments are all reasonably well described.
* Simple ideas that provide good results. In retrospectively it might seem obvious, yet not explored before.
* Widely applicable for all methods using pre-trained image classifiers.
 
*Weaknesses:*
* Although the experimental section is good, some elements are missing. E.g. adding non-data-augmented as reference point in the plot, or considering CAM as a saliency strategy.
* Some sections of the text would benefit from revisiting the English.
* The method implicitly relies on having “simple images” with one dominant foreground object  like the ones in CIFAR and ImageNet. The saliency / “object of interest”  detection method depends on these characteristics. Ideally the paper would be more upfront on these assumptions. Especially in the context of the “Rethinking ImageNet Pre-training”, ICCV 2019 work.
* The related work section for saliency is very partial.
* Some of the saliency methods evaluated use training data, even the ones that do not have been tuned using additional data. The paper would benefit from a discussion of this additional information.


*Correctness:* Paper seems correct. There are minor shortcomings in the experimental protocol, but nothing that would foreseen invalidating the main conclusions of the paper.

*Clarity:* Paper presentation is clear.






*Relation to prior work:*
Related work section has a reasonable extent. 
Regarding data augmentation, the paper compares with the main methods.
The text mentions Lemley 2017, however I think it would be also worth mentioning AutoAugment Cubuk 2019; and justifying why it is not included in the results comparison.

The saliency detection is very partial, and does not cover the main works in the area. 
For one the text does not clarify “which kind” of saliency is considered (where will a human look ? which are the main objects of the scene ? which is the main object of the scene ?). Depending on which one, discussing the main performers in the related benchmarks (e.g. table in Qin 2019 paper) seem relevant. 
From what I grasp, the proposed method would actually want to have as input a weakly supervised class-conditional segmentation. And “saliency” is used as a proxy for this. Discussing the relation to (image-level labels) weakly supervised segmentation would also be welcome.
(I would guess it could provide even better results, but at a much higher computational and system complexity cost). In particular CAM is discussed in section 4.3. Would that not be a task-specific way to obtain the desired “semantically important regions”  ?


When preparing the camera ready, please consider discussing the concurrent work of https://arxiv.org/pdf/2009.06962.pdf which seems related (seems recent enough, ICML 2020, to give the benefit of presumed concurrency).


*Reproducibility:*
The overall algorithm is simple to understand and re-implement.
The selected saliency method “Wand and Dudek, 2014” seems to be a video saliency method. From a quick inspection of that paper it is not immediately clear to me how to transpose it for single image saliency. Since this method is used in most experiments, providing more details of the implementation and its parameters are necessary to reproduce the key results.


*Specific per-section feedback:*

Section 1:
- every field: is too broad of a statement, remove/rephrase the first sentence.
- extremely complex -> complex
- generalizability -> generalization
- undesired to the CNN since -> undesired since
- does not allow … to have any uniformative pixel: double negative, consider simplifying.
- semantically important region: these are task dependent. How do you ensure the saliency to match the task / semantics ?


Section 2.1:
- comes into account that aims: unclear, please rephrase.


Section 2.2:
- unable to … 1 or 0: this is not true, please rephrase.
- intermediate values: you mean closer to 0.5 ?
- Thereby, helps: unsure if this is proper English.


Section 2.3: See comments above regarding related work.
- Which kind of saliency is considered here ? Which are the relevant benchmarks ? Which training data / evaluation data are these methods bringing (indirectly) into the system ?
- BAsNet for example, was trained on 10k images. Why not simply include these (and their mask) as part of the pretraining when considering some of the baselines ?
Section 3.1:
- selected training image -> selected training (source) image. In general do give hints for the subscript meaning of I_s, I_vs, etc.
- Why only one pixel with maximum intensity ? What happens if (due to quantization) two pixels have the same value ?
This is clarified later in the text, but some context would be welcome in the mention here.
Figure 3:
- Add non-augmented result bar as reference point. 
- Consider showing the five points per bar, or some hint for the variance in these results. From the plot it is left unclear if the fluctuations across methods are significant or minor (since not reference point, nor sense of the variance).
Section 3.3:
- Tiny-Imagenet: saliency methods tend to _not_ be scale invariant (in particular trained methods like BasNet). How is this handled ? Why would one expect Tiny-Imagenet to provide conclusive data ?
- What about CAM, or any other class-conditioned saliency / weakly supervised segmentation method ?
Section 3.4:
- found out -> consider
- What about “Salient to Random“ ? That seems a reasonable option too ?
- are identical -> are similar


Section 4.1:
- SOTA top-1 error: there are 20 methods that claim better results in https://paperswithcode.com/sota/image-classification-on-cifar-10 and https://paperswithcode.com/sota/image-classification-on-cifar-100 that claim better results. Maybe temperate the “SOTA” claim, with phrasings like “best known results for model X”  or similar. One specific model result, far from best known, does not constitute “state of the art”  in my understanding.


Section 4.2:
- Because in -> This is because in


*Updates after reviews and authors feedback:*

The updates from the author are appreciated and make the arguments of the paper clearer.
After reading the other reviews and discussions, I have downgraded my score to "7: Good paper, accept".

(Please note that in the current pdf the table at the top of page 7 has formatting issues.)

review_title: Simple and useful idea, well presented.

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: The paper presents a method called SaliencyMix. They improve a method that augments images by adding random patches from other images. The innovation is that they select these patches using a saliency map.

This paper has an excellent discussion and critique of previous work. They discuss the existing work with a nice summary and then discuss reasons why selecting random patches can have issues. There is a clear argument for their method over selecting patches randomly. They make clear claims that this approach improves performance over randomly selecting patches. The experiments support this with sufficient related work (Cutout, Cutmix) and exploration of other design decisions and aspects about the idea.

The idea is relatively straightforward and is inline with existing literature. The paper is well executed so there is not much to complain about.

The availability of the source code is not clear from the text.

A potentially interesting analysis (but not required) is an analysis of the increased runtime in practice.

review_title: Solid paper

final_score: rating: 3: Clear rejection
review_content: This paper proposes a new augmentation method based on CutMix. The authors find out that randomly selecting may mix background textures and this will mislead the model. So, they propose to use saliency maps to control the selection of mixed patches, which is called SaliencyMix. This idea seems easy and reasonable, many experiments are conducted to prove the effectiveness of the proposed method. However, the experiments’ results fail to show the ability of the method, and some explanation is missed.
 
Positive:
1.     The idea is simple and clear, the paper is well organized and easy to follow.
2.     The experiments are comprehensive, including classification and transfer learning.
 
Weakness:
1.     The main concern is the effectiveness of the proposed method. According to the authors’ experiments, the improvement over CutMix is very limited on all datasets. Especially on ResNet-101, the promotion over CutMix is only 0.08.
2.     According to the authors’ ablation study in sec. 3.3, only using fast self-tuning background subtraction produces better results than CutMix. Why other methods even worse than CutMx? What’s the core reason for the improvement of using fast self-tuning background subtraction?
3.     The authors use batchsize=256, lr=0.1 for CIFAR training, while usually batchsize=128, lr=0.1 is used in previous works (Cutout). And as described in [1], the learning rate should be increased linearly with batchsize. This change of hyperparameters may need further explanation.
 
[1] Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour.


============Post Rebuttal====================
After reading the feedback from authors, I still have my concerns. The novelty of this paper is too limited for ICLR. I really do not think a combination of CutMix with existing saliency detection method is a novel method. Moreover, the improvement over CutMix is diminishing. These main concerns are not addressed by the authors. So, my final recommendation is still rejection.
review_title: Simple idea but results are not strong enough

final_score: rating: 3: Clear rejection
review_content: This paper is based on an interesting observation that previous data augmentation tricks cut&mix may select regions that do not contain useful information. Instead, this paper use saliency models to detect the salient regions first and then cut and mix these salient regions in a source image.

This observation is interesting and worth trying. However, the contribution might be too limited for a ICLR conference paper:
1. as cut&mix cuts bigger blocks in an image, the target object is more likely to be selected. Also for most image classification dataset, the images are quite iconic, so the improvement on classification tasks are limited (as shown in Tab. 1, C10+ and C100+). This might help with detection as it may train models to focus on the most discriminative part of the image, but recent works show that there is no direct correlation between between the performance of the same backbone on detection tasks and classification tasks. In addition, the author didn't provide analysis on what causes the 1.8% improvement on detection tasks (better on smaller objects?) So it's not clear how helpful this trick is.
2. While the method is simple, I expect either some mathematic proof or this method works well on various tasks. The paper didn't have any proof or statistical analysis. This paper didn't either show if the proposed method will work on more tasks (for example segmentation or GAN? the detection provided in this paper is using the backbone initialized from classification, during training faster-rcnn it seems that the trick is not used).


review_title: Interesting trick but not enough for ICLR

Meta-review: This paper proposes an approach to data augmentation to train image recognition models called SaliencyMix, which involves pasting salient regions (as judged by some saliency detector) of one image into another, and mixing the two labels accordingly. Most reviewers generally agreed that the proposed approach is simple -- it is easy to understand the method and its motivation, especially in the context of related augmentation approaches like CutMix -- and has solid experimental results demonstrating its effectiveness.

The main objection the more negative reviewers had to the work is a perceived lack of novelty. In my view it is a new method (even if similar to prior work like CutMix), and as AR5 argues: "this approach probably starts to get to the heart of why these previous strategies work: they are probably less effective ways of doing what this paper suggests." The improvements in Table 1, columns 1 & 3 (CIFAR-10 & CIFAR-100) especially speak to this -- these improvements with traditional augmentation disabled are quite substantial, even though the differences become marginal when moving to the "+" augmented versions of the dataset (as well as in ImageNet). So although the method is indeed similar to CutMix, I agree that it offers valuable insight into why these previous methods work. Besides which the results *do* show improvements over similar methods, even if the improvements are marginal.

Overall, I recommend accepting the paper as it provides useful insight into why prior methods work and proposes a new one that in practice works slightly better. Minor comments for the camera-ready version:

Please revise the writing based on AR4's good suggestions.

Highlighting a comment from AR4:
> BAsNet for example, was trained on 10k images. Why not simply include these (and their mask) as part of the pretraining when considering some of the baselines ?

I recommend including discussion of this important point in the final version of the paper. The learning-based approaches are effectively using additional training data. It's good that a non-learning-based method happens to perform best so that the results remain comparable with prior work, but this should nonetheless be discussed if the learning-based approaches are to be included.

Please remove the blue text coloring (if not already planned -- I'm not sure if this was done as a "diff" for the response).

> Figure 3(a-b) show that Montabone & Soto (2010) performs better on both the datasets and the effects are identical on CIFAR10 and ImageNet datasets

I do not see how Figure 3 shows this. Is "OpenCV Saliency" in the figure using the method from Montabone & Soto (2010)? Please clarify this by making the connection between the bar labels in the figure and the discussion in the text clear for the camera-ready version of the paper.
Decision: Accept (Poster)