Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: The authors propose a novel sailency-guided data augmentation method that alleviates some of the drawbacks arising with recent Mixup-based augmentation approaches.  Specifically, the authors propose sailency thresholding for region selection (instead of maximum sailent region), stochastic sampling of sailent patches, and sailency-based label mixing. The results show clear improvements over competitors, and an ablation study shows the performance gains of each of the design choices discussed previously.

The paper is pleasant to read and all decissions are properly motivated. The main contributions are fairly novel, the formulation sound and the experiments well designed. 

The claim of robustness of data corruption of the model seems like an over-statement. Even though the authors show that Sailency Grafting performs well under data corruption, one may attribute the robustness feature to the AugMix method that the grafting acts upon. The results of table 5. just show that grafting sailent corrupted image patches improves over just training on corrupted patches, which is something to expect given the previously reported improvements.  

Early in the paper the authors claim a difference wrt. previous work that is the uniform thresholding of the salient region rather than selecting the maximum, to mitigate selection bias. I missed a section in the ablation study demonstrating the specific improvement of this decision. Also the authors briefly mention the approach to select the thresholding as the mean of the normalized sailency map. I'm curious to know if any other method of threshold selection (or patch sampling) has been explored. 
review_title: Well written. Clear and effective method. State-of-the art results with thorough ablation study.

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper advances the line of research in data augmentation following Mixup paper. Cutmix is a image-specific variant of mixup that pastes a rectangular region from a donor image to a target image; however, it does this in a completely random fashion not paying attention to whether the discriminative parts of either image are retained. Recent methods alleviate this problem by using saliency guided region selection either preserving the most discriminative parts of the donor image (Attentive Cutmix) or both images (PuzzleMix). However, in doing so they replace the random nature of Cutmix with a deterministic approach potentially reducing sample diversity. The first contribution of the paper is a saliency guide stochastic augmentation method which combines the best of both worlds. The second contribution is in the generation of more meaningful labels for the newly created image by taking into account the saliency of the regions used from both images rather than blindly using the mixing ratio. The contributions of the paper are somewhat incremental but are reasonable. The validity of using saliency in determining the label of the new sample is clear. The experimental results are mixed. Most experiments demonstrate minor improvements over previous methods. On the other hand the top-5 performance of CutMix is better than the proposed method in Table 2. With such small differences and no standard deviations on accuracy reported it is not clear whether the improvements are really significant. 

In the data scarcity experiments, I find the statement “ Note that the performance of CutMix deteriorates as the number of data per class decreases due to their randomness occurring
label mismatching.” misleading. This is true of every method in that table not just CutMix, and the detoriation amounts are very similar. 

Clarity: Overall I found the paper well written and easy to read. 

The authors have provided standard errors which improves the confidence that the improvements in the experiments are significant. 
review_title: borderline paper

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper propose a novel method to address the lack of diversity in mixed samples created by the saliency-based Mixup strategies such as Puzzle Mix. To attain this goal, the proposed approach incorporates a random process in the saliency regions selection process to give all salient regions equal chance, and accordingly adjust the modeling target to match the saliency of the mixed image. The paper presents an interesting extension to the existing work. Nevertheless, I would like to see more evidence of why the existing methods lack of diversity and how it negatively impacts the model’s performance. I also think the experiments in the current form is weak. My major comments are as follows. 

Regarding the proposed method:

1.	The motivation of generating diverse mixed images through selecting diverse salient features is interesting to me. But I think the current justification is weak. It is true that the Puzzle Mix will select saliency regions with high intensity, but I wonder if the random mixing ratio \lambda in PuzzleMix would help to generate diverse mixed samples due to the random sampling process. Another factor for the diversity here is that even using similar saliency regions of the input pair for the mixed images, the mixed labels are different due to the random selection of the mixing ratio. In this sense, I think how severe this lack of diversity issue is and how it negatively impacts  the performance of the mode are not very clear to me.  It would be very beneficial if the paper could show it somehow. 
2.	The way that the proposed method generates the mixed modeling target for the mixed input reminds me of this paper: nonlinear mixup: out-of-manifold data augmentation for text classification (AAAI2020). In the nonlinear Mixup method, the mixed label for a mixed input in Mixup is also computed based on the input pair, although the method is evaluated using text classification tasks. I think it would be useful to discuss the difference and adjust the content such as Table1 and Section3 to make the statements in the paper more accurate. 

3.	Some parts in the proposed method deserve further discussion and justification. For example, the Threshold for the normalized saliency map. How sensitive is the threshold value to the performance of the proposed model? It would be useful to include some analysis either theoretically or experimentally. 

Regarding the experiments:

1.	I am surprised to see from Table2 that the PuzzleMix degraded the performance of the baseline model. I wonder why this happened. In this sense, I would like to see experiments with some other benchmarking datasets such as Cifar10, SVHN, or TinyImagenet. Also, for these experiments, it would be useful to also provide the deviation of the 3 runs. 

2.	For Cifar100 using WRN28-10 (also for ImageNet with ResNet-50), the difference between the proposed method and the PuzzleMix is very small. This suggests that it would useful to experiment on other network architectures such as ResNet-18 or ResNet-50, which are used in the original Mixup and the PuzzleMix papers. 

Minor issues:

1.	The paper discusses the harmful samples potentially generated by Mixup in several places without citation. I think it would be useful to include some citations such as the manifold intrusion issue raised in the AdaMixup paper (AAAI2019) or the noise image issue as discussed in the PuzzleMix paper (ICML2020).

 In short, I think the paper would benefit from further justifying the problem of lack of mixed sample diversity in the PuzzleMix. Also, the current experimental results show minor improvement over PuzzleMix, which also makes the contribution of the paper less significant. 

review_title: Would like to see strong evidence of why Puzzle Mix lacks of diversity and how it negatively impacts the model’s performance

Meta-review: Overall, this paper has been on the very borderline. All reviewers agree that the motivation and the idea of the paper are reasonable (although somewhat incremental) and make an interesting extension of mixup-type data augmentation. However, one expert reviewer raised some concerns which are unfortunately not fully resolved despite the intensive interaction. 

The first one is the issue of diversity claimed in the paper. The authors explanation is mostly qualitative, and I, as an AC, also felt a logical jumping here, although I do understand that the proposed method somehow results in better generalization ability w.r.t. the number of generated samples as shown in Fig.3 in the appendix. This point is important because if the better generalization is coming from the label modification rather than the diversity in image space, then the novelty of the paper is limited.
Another concern is that there are some inconsistencies in the scores of previous methods implemented by the authors and in the original papers. We understand that the exact score is not always easy to reproduce, but at least it is desirable to follow the original setting (such as the number of epochs) for each method as far as possible, because the accuracy should be the most important criterion as the goal of data augmentation is better generalization. Then, authors may separately discuss computational efficiency. 

Based on the discussion with reviewers as above, I conclude that the paper should be further polished and completed before publication. Thus I recommend rejection for this time.
Decision: Reject