Saliency Grafting: Innocuous Attribution-Guided Mixup with Calibrated Label Mixing
Authors: Joonhyung Park
June Yong Yang
Jinwoo Shin
Sung Ju Hwang
Eunho Yang
Venue:  (2021)
Keywords: Deep learning
Data augmentation
Input attribution

Abstract: The Mixup scheme of mixing a pair of samples to create an augmented training sample has gained much attention recently for better training of neural networks. A straightforward and widely used extension is to combine Mixup and regional dropout methods: removing random patches from a sample and replacing it with the features from another sample. Albeit their simplicity and effectiveness, these methods are prone to create harmful samples due to their randomness. In recent studies, attempts to prevent such a phenomenon by selecting only the most informative features are gradually emerging. However, this maximum saliency strategy acts against their fundamental duty of sample diversification as they always deterministically select regions with maximum saliency, injecting bias into the augmented data. To address this problem, we present Saliency Grafting, a novel Mixup-like data augmentation method that captures the best of both ways. By stochastically sampling the features and ‘grafting’ them onto another sample, our method effectively generates diverse yet meaningful samples. The second ingredient of Saliency Grafting is to produce the label of the grafted sample by mixing the labels in a saliency-calibrated fashion, which rectifies supervision misguidance introduced by the random sampling procedure. Our experiments under CIFAR and ImageNet datasets show that our scheme outperforms the current state-of-the-art augmentation strategies not only in terms of classification accuracy, but is also superior in coping under stress conditions such as data corruption and data scarcity. The code will be released.

Under review as a conference paper at ICLR 2021
SALIENCY GRAFTING: INNOCUOUS ATTRIBUTION- GUIDED MIXUP WITH CALIBRATED LABEL MIXING
Anonymous authors Paper under double-blind review
ABSTRACT
The Mixup scheme of mixing a pair of samples to create an augmented training sample has gained much attention recently for better training of neural networks. A straightforward and widely used extension is to combine Mixup and regional dropout methods: removing random patches from a sample and replacing with the features from another sample. Albeit their simplicity and effectiveness, these methods are prone create harmful samples due to their randomness. recent studies, attempts to prevent such phenomenon by selecting only the most infor- mative features are gradually emerging. However, this maximum saliency strat- egy acts against their fundamental duty of sample diversification as they always deterministically select regions with maximum saliency. injecting bias into the augmented data. To address this problem, Mixup-like data augmentation method that captures the best of both ways. By stochastically sampling the features and 'grafting' them onto another sample, our method effectively generates diverse yet meaningful samples. The second ingredi- ent of Saliency Grafting produce the label of the grafted sample by mixing the labels in a saliency- calibrated fashion, which rectifies supervision misguidance in- troduced by the random sampling procedure. Our experiments under CIFAR and ImageNet datasets show that our scheme outperforms the current state-of-the-art augmentation strategies not only terms of classification accuracy, but is also su- perior in coping under stress conditions such as data corruption and data scarcity.
present Saliency Grasting, a novel
saliency-calibrated


Reviews:
[{"review_id": "koItAPIRQH", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper propose a novel method to address the lack of diversity in mixed samples created by the saliency-based Mixup strategies such as Puzzle Mix. To attain this goal, the proposed approach incorporates a random process in the saliency regions selection process to give all salient regions equal chance, and accordingly adjust the modeling target to match the saliency of the mixed image. The paper presents an interesting extension to the existing work. Nevertheless, I would like to see more evidence of why the existing methods lack of diversity and how it negatively impacts the model’s performance. I also think the experiments in the current form is weak. My major comments are as follows. \n\nRegarding the proposed method:\n\n1.\tThe motivation of generating diverse mixed images through selecting diverse salient features is interesting to me. But I think the current justification is weak. It is true that the Puzzle Mix will select saliency regions with high intensity, but I wonder if the random mixing ratio \\lambda in PuzzleMix would help to generate diverse mixed samples due to the random sampling process. Another factor for the diversity here is that even using similar saliency regions of the input pair for the mixed images, the mixed labels are different due to the random selection of the mixing ratio. In this sense, I think how severe this lack of diversity issue is and how it negatively impacts  the performance of the mode are not very clear to me.  It would be very beneficial if the paper could show it somehow. \n2.\tThe way that the proposed method generates the mixed modeling target for the mixed input reminds me of this paper: nonlinear mixup: out-of-manifold data augmentation for text classification (AAAI2020). In the nonlinear Mixup method, the mixed label for a mixed input in Mixup is also computed based on the input pair, although the method is evaluated using text classification tasks. I think it would be useful to discuss the difference and adjust the content such as Table1 and Section3 to make the statements in the paper more accurate. \n\n3.\tSome parts in the proposed method deserve further discussion and justification. For example, the Threshold for the normalized saliency map. How sensitive is the threshold value to the performance of the proposed model? It would be useful to include some analysis either theoretically or experimentally. \n\nRegarding the experiments:\n\n1.\tI am surprised to see from Table2 that the PuzzleMix degraded the performance of the baseline model. I wonder why this happened. In this sense, I would like to see experiments with some other benchmarking datasets such as Cifar10, SVHN, or TinyImagenet. Also, for these experiments, it would be useful to also provide the deviation of the 3 runs. \n\n2.\tFor Cifar100 using WRN28-10 (also for ImageNet with ResNet-50), the difference between the proposed method and the PuzzleMix is very small. This suggests that it would useful to experiment on other network architectures such as ResNet-18 or ResNet-50, which are used in the original Mixup and the PuzzleMix papers. \n\nMinor issues:\n\n1.\tThe paper discusses the harmful samples potentially generated by Mixup in several places without citation. I think it would be useful to include some citations such as the manifold intrusion issue raised in the AdaMixup paper (AAAI2019) or the noise image issue as discussed in the PuzzleMix paper (ICML2020).\n\n In short, I think the paper would benefit from further justifying the problem of lack of mixed sample diversity in the PuzzleMix. Also, the current experimental results show minor improvement over PuzzleMix, which also makes the contribution of the paper less significant. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ILKHmGL3wx", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors propose a novel sailency-guided data augmentation method that alleviates some of the drawbacks arising with recent Mixup-based augmentation approaches.  Specifically, the authors propose sailency thresholding for region selection (instead of maximum sailent region), stochastic sampling of sailent patches, and sailency-based label mixing. The results show clear improvements over competitors, and an ablation study shows the performance gains of each of the design choices discussed previously.\n\nThe paper is pleasant to read and all decissions are properly motivated. The main contributions are fairly novel, the formulation sound and the experiments well designed. \n\nThe claim of robustness of data corruption of the model seems like an over-statement. Even though the authors show that Sailency Grafting performs well under data corruption, one may attribute the robustness feature to the AugMix method that the grafting acts upon. The results of table 5. just show that grafting sailent corrupted image patches improves over just training on corrupted patches, which is something to expect given the previously reported improvements.  \n\nEarly in the paper the authors claim a difference wrt. previous work that is the uniform thresholding of the salient region rather than selecting the maximum, to mitigate selection bias. I missed a section in the ablation study demonstrating the specific improvement of this decision. Also the authors briefly mention the approach to select the thresholding as the mean of the normalized sailency map. I'm curious to know if any other method of threshold selection (or patch sampling) has been explored. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "2TMur92pcNI", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper advances the line of research in data augmentation following Mixup paper. Cutmix is a image-specific variant of mixup that pastes a rectangular region from a donor image to a target image; however, it does this in a completely random fashion not paying attention to whether the discriminative parts of either image are retained. Recent methods alleviate this problem by using saliency guided region selection either preserving the most discriminative parts of the donor image (Attentive Cutmix) or both images (PuzzleMix). However, in doing so they replace the random nature of Cutmix with a deterministic approach potentially reducing sample diversity. The first contribution of the paper is a saliency guide stochastic augmentation method which combines the best of both worlds. The second contribution is in the generation of more meaningful labels for the newly created image by taking into account the saliency of the regions used from both images rather than blindly using the mixing ratio. The contributions of the paper are somewhat incremental but are reasonable. The validity of using saliency in determining the label of the new sample is clear. The experimental results are mixed. Most experiments demonstrate minor improvements over previous methods. On the other hand the top-5 performance of CutMix is better than the proposed method in Table 2. With such small differences and no standard deviations on accuracy reported it is not clear whether the improvements are really significant. \n\nIn the data scarcity experiments, I find the statement “ Note that the performance of CutMix deteriorates as the number of data per class decreases due to their randomness occurring\nlabel mismatching.” misleading. This is true of every method in that table not just CutMix, and the detoriation amounts are very similar. \n\nClarity: Overall I found the paper well written and easy to read. \n\nThe authors have provided standard errors which improves the confidence that the improvements in the experiments are significant. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Overall, this paper has been on the very borderline. All reviewers agree that the motivation and the idea of the paper are reasonable (although somewhat incremental) and make an interesting extension of mixup-type data augmentation. However, one expert reviewer raised some concerns which are unfortunately not fully resolved despite the intensive interaction. 

The first one is the issue of diversity claimed in the paper. The authors explanation is mostly qualitative, and I, as an AC, also felt a logical jumping here, although I do understand that the proposed method somehow results in better generalization ability w.r.t. the number of generated samples as shown in Fig.3 in the appendix. This point is important because if the better generalization is coming from the label modification rather than the diversity in image space, then the novelty of the paper is limited.
Another concern is that there are some inconsistencies in the scores of previous methods implemented by the authors and in the original papers. We understand that the exact score is not always easy to reproduce, but at least it is desirable to follow the original setting (such as the number of epochs) for each method as far as possible, because the accuracy should be the most important criterion as the goal of data augmentation is better generalization. Then, authors may separately discuss computational efficiency. 

Based on the discussion with reviewers as above, I conclude that the paper should be further polished and completed before publication. Thus I recommend rejection for this time.
Decision: Reject