Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper elucidates the importance of instance-dependent transformation in augmenting data and proposes InstaAug that learns the input-dependent transformation which could preserve the information of original input while maintaining sufficient transformation diversity. The invariance module $\phi$ is jointly trained with the classifier $f$, which can predict the parameters of transformation distribution based on a given input. The authors also propose a location-related parameterization technique to parameterize cropping augmentations.
strength_and_weaknesses: (+) The paper is well-written and the proposed method is simple and straightforward.

(+) The location-related parameterization (LRP) for cropping seems quite novel to me.

(-) It seems the flow of the draft needs to be substantially modified. Most parts of the draft describe the necessity of instance-wise augmentation in comparison to global augmentations. As the idea of instance-wise (or class-wise) augmentation is not new [1,2], the differences and empirical comparisons with these works should be more intensively addressed in the paper. Only brief and limited descriptions can be 
found in the related works section. The authors claim that these prior works focus on choosing ‘which type of transformation to apply’ while InstaAug keeps the type of transformation fixed and learns instance-specific parameters. However, [1] searchs not only the type of transformation but also the magnitude, and in this regard, the advantage of ‘keeping the transformation fixed’ is unclear given that 
a set of hyperparameters have to be found for each transformation.

(-) Some of the motivations for the proposed approach are not convincing. The authors explain the limitations of global augmentation approaches using Figure 1. In Figure 1 (b), it seems the global augmentation could rather boost the generalization performance since the model has to exploit other features (e.g. texture, shape) for correct prediction without simply relying on color. I believe this is why 
Cutmix and Cutout have been successful as it prevents the model from predicting only highly discriminative features. Meanwhile, the proposed method might fortify the inductive bias of the classifier as the $\phi$ is trained to simply reduce the classification error of the classifier. It would be great if the authors can provide evidence/analysis to further strengthen their claim.

(-) Section 3.2 and Figure 3 are weak to support the authors’ claim. Of course, extreme global transformations will generate incorrect samples (‘red’ region in Figure 3). But it seems the proposed method also suffers from a similar issue and probably this is the reason why the authors introduce the upper bound $H_{max}$ in their constraints. Wouldn't this problem be solved by relaxing the type and 
intensity of transformation in global augmentation as well? (why should Figure 3 be an illustration rather than the visualizations of the actual decision boundary of Global Augmentation and InstaAug?)

(-) This work aims to propose an augmentation that can conduct information-preserving transformations while keeping a sufficient diversity of transformations. Inspired by goals similar to this work, saliency-based augmentations [3, 4, 5, 6] are recently proposed for information-preserving augmentations. [5, 6] also consider the diversity of augmented data. Although previous related works did not compare with these works, for this work, empirical comparisons with these saliency augmentations are required given the highly similar goals (at least combining InstaAug with these saliency augmentations). Comparing the method against baselines such as Cutmix and Mixup is 
insufficient.

---
### Questions
- Does this method work well for data-deficient conditions? Since data augmentation is most needed when data is scarce, I wonder if the invariance module can learn the proper local invariance even when there is insufficient data.

- Gradient-based saliency [5,7] (or other saliency methods) can detect instance-specific regions without relying on predefined sets. What are the advantages of LRP over these?

- Although this fact doesn't affect my score, is there a reason to experiment with only one dataset per augmentation type? (e.g. TinyImageNet can be used also for testing color jitter). It would be good to experiment with commonly used datasets such as CIFAR and SVHN.

- (minor) The tuning for $H_{min}$ and $H_{max}$ is necessary when practitioners deploy this method to their own data modality or augmentations. It would be nice if the insights of determining $H$ are explained in the main paper

[1] Cheung, Tsz-Him, and Dit-Yan Yeung. "AdaAug: Learning Class-and Instance-adaptive Data 
Augmentation Policies." International Conference on Learning Representations. 2021.

[2] Zhou, Fengwei, et al. "Metaaugment: Sample-aware data augmentation policy 
learning." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 35. No. 12. 2021.

[3] Kim, Jang-Hyun, Wonho Choo, and Hyun Oh Song. "Puzzle mix: Exploiting saliency and local 
statistics for optimal mixup." International Conference on Machine Learning. PMLR, 2020.

[4] Uddin, A. F. M., et al. "Saliencymix: A saliency guided data augmentation strategy for better 
regularization." arXiv preprint arXiv:2006.01791 (2020).

[5] Park, Joonhyung, et al. "Saliency grafting: Innocuous attribution-guided mixup with calibrated label 
mixing." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 36. No. 7. 2022.

[6] Kim, JangHyun, et al. "Co-Mixup: Saliency Guided Joint Mixup with Supermodular
Diversity." International Conference on Learning Representations. 2020.

[7] Simonyan, Karen, Andrea Vedaldi, and Andrew Zisserman. "Deep inside convolutional networks: 
Visualising image classification models and saliency maps." arXiv preprint arXiv:1312.6034 (2013)
clarity,_quality,_novelty_and_reproducibility: The paper is well-written and easy to follow. Some components are quite novel, but overall the idea of instance-adaptive augmentation is not new. Differences from related works are not well described in the draft, and experimental comparisons are too limited. The authors submitted their code, so I believe this work is reproducible.


summary_of_the_review: Overall, I think this work needs further refinement for acceptance due to the concerns mentioned above (please see Strength And Weakness section). I understand that it is no longer possible for the authors to revise their draft, but I would like to see the authors' response within the remaining period.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposes an algorithm to learn instance-specific augmentation for images, which substantially improves computer vision models over a wide range of scenarios.
strength_and_weaknesses: Strength:
1. The algorithm is simple and effective.
2. It applies to a wide range of scenarios, including supervised learning and contrastive learning.

Weakness:
1. The two principles suggested at the beginning of Section 2, while intuitively correct, are not theoretically justified in the paper. In addition to empirical successes, is there a reason why these two principles lead to better representation? The authors may need to explain them from an information-theoretical perspective (since the loss function includes entropy) or from a learning-theoretical perspective.
2. The proposed method introduces too many hyperparameters. For example, the algorithm introduces lower/upper limits of the entropy and the regularizer in the loss function. Furthermore, the algorithm requires a set of hyperparameters for each type of augmentation. Therefore, it is not easy to adopt the algorithm to a new scenario (without a tedious hyperparameters search).
3. The current method only works for vision models since the augmentation types are mostly vision-specific. However, the proposed method should also work in other domains, particularly natural language processing. It will make the technique more general/impactful by demonstrating its capabilities other than computer vision. 
clarity,_quality,_novelty_and_reproducibility: The paper is original as far as I know and clearly written. 
summary_of_the_review: The paper proposed a simple and effective instance-specific algorithm that generally improves the performance of computer vision models. However, there are still some limitations, as suggested in Strength And Weaknesses.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: Data augmentations with common input transformations is a commonplace in ML training, and is known to improve performance. This work argues that example-agnostic common transformations are too restrictive and that example specific augmentations can allow much wider transformation. The proposed methods of learning instance specific augmentations is shown to improve performance when training on Tiny-Imagenet, test-time augmentations and for contrastive learning. 
strength_and_weaknesses: Strengths.
1. Need for instance specific augmentation is well-motivated. I particularly like Section 5.1 and Figure 5. 
2. Significant empirical gains in Table 1 and 2. 


Weakness/questions.
1. The proposed method is simple, which is nice. However, in (2b), I do not see what pushes the per-instance transformation entropy to be more than H_min. In other words, what part of the loss encourages diversity in the distribution of transformations?
2. Comparison with per-label (instead of per-instance) augmentations should also be presented. 
3. In Table 1, InstaAug (without input) but with LRP is worse than global random crop, which is unexpected. I expected them to be at least comparable, why do we see a difference? 
4. Section 5.3 does not describe in detail how \phi is re-learned and on what data.  Technical details on how \phi is parameterized is also missing. Section 3.3 is also not very clear. 
5. With regard to test-time augmentations of Section 5.3, the improvement in performance with InstaAug is impressive. However, InstaAug can only learn augmentations that the trained model is already robust toward, I do not see the source of improvement in Table 2. Could you explain/comment? 
6. (Minor) Figure 7 contains some legends but not their plots making it look incomplete.

clarity,_quality,_novelty_and_reproducibility: Clarity: Some technical details are missing, but it is clear otherwise. 
Code is not released although they provided link to a codebase that they started from. 
summary_of_the_review: I am not yet fully convinced that the paper is foolproof. The motivation and problem setup is clear, but their method and technical details are not. I request the authors to address my concerns listed above and I will make a more decisive evaluation after looking at authors’ responses and based on other reviews. 

--------
**Edit after response.**    
I thank the authors for patiently addressing all the question. Most of my question/confusion have now been resolved. I thank the authors again for adding the comparison with per-label augmentations. After reading other reviews and author comments, I have the following outstanding concerns. 

* Reviewer eZc6 pointed to several related work. I agree with eZc6 that the paper should have elaborated more on their existence and comparison. I find the reasons the authors provide for not comparing with MetaAug/AdaAug unconvincing for the following reasons: (a) "MetaAug/AdaAug do not search over parameterisation of augmentations", which could perhaps be fixed simply by extending the search space to include augmentations with different parameterisations. (b) "they require validation dataset", given that train loss is a good proxy for InstaAug, it could also be a good proxy for MetaAug/AdaAug. Why not simply replace their need for validation dataset with train dataset? I agree that it is not trivial to use them for comparisons on test-time augmentations though. I did not go over the related work before and got the impression (after reading the draft) that instance specific augmentations is novel when it is not. The paper thus may need to be restructured to carefully communicate their contributions. 
* Reviewer 4op9 raised an important point about evaluation on the running examples of digit rotations. The authors' response to this question is concerning. If instance-specific augmentations are not useful in this simple example, I do not understand how it is helping for other (complex/real-world) datasets. In that regard, the paper should present a more faithful explanation/intuition of their method. 

Existing work on learning effective augmentation stratgies (including instance-specific augmentations) all use policy search based methods and therefore InstaAug is more desirable since it learns the augmentation policy end-to-end. LRP and parameterising of augmentations are all interesting contributions, which make InstaAug readily available for examples beyond training data such as in test-time augmentations. Although I am excited and recognise these merits of InstaAug, I am discourcaged by problems in exposition and comparison. The paper is an interesting contribution (especially given their performance gains) at some point, but the current draft needs further work. To make the acceptance decision more easier, I am now lowering my earlier score.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes an additional module that performs data-specific augmentation. Specifically, location-related parametrization taking account of spatial location has been proposed. Experimental results show the effectiveness of the proposed method on image datasets in supervised and self-supervised representation learning tasks.
strength_and_weaknesses: Strengths

+ The task of capturing local invariances by learning data-specific augmentation is interesting.

+ The proposed method is simple but looks effective; by looking at the description, simply parameterizing data augmentations and optimizing its entropy is enough.

Weaknesses

- It seems prior works are not properly discussed. Based on the description in Section 5.2, the uniform parameterization is the same as Augerino, which is proposed by [Benton et al.]. Also, I believe test-time augmentation is also used in [Benton et al.]. Indeed, the performance of Augerino and the proposed method without LRP looks almost the same by looking at Table 1. Please do side-by-side comparison with closely related works like [Benton et al.].

- The first intuitive example in Figure 1(a) is digit recognition, but it is not appeared in the experimental results. I wonder if the range of rotation for 6 and 9 is indeed trained to be narrow, such that they are distinguishable.

- Similar to supervised learning, authors could also compare the proposed method with mixup-variants, which also have shown to be effective in many prior works, such as [Lee et al.] and [Shen et al.]. Also, I believe mixup-variants become a default choice to boost the performance of both supervised and self-supervised learning, so I wonder if the proposed method can be combined with them.

[Lee et al.] i-Mix: A Domain-Agnostic Strategy for Contrastive Representation Learning. In ICLR, 2021.

[Shen et al.] Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning. In AAAI, 2022.

**post-rebuttal**

The authors addressed my concerns mostly well, but an exception is the MNIST example in the motivative figure: I think their demonstration is indeed not proper and should be either completely removed or thoroughly discussed with experimental results to avoid any confusion.

Other reviewers found some critical points I was not initially aware of. For example, regarding AdaAug/MetaAug, if they indeed share the same spirit with the proposed method, then I think it is worth making some comparison (maybe by adapting the proposed method to the prior works' setting).

All in all, I do not change my initial rating, as I think other reviewers' concerns seem to be valid.
clarity,_quality,_novelty_and_reproducibility: Writing is clear and well-written in general, except for the comparison with closely related works. Parametrizing augmentations is not novel, but the specific way of location-related parametrization seems novel. I believe this work is reproducible with the provided code.
summary_of_the_review: I am mostly happy with this paper, except for the unclear comparison with prior works. Please answer my concerns above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper proposes a method called InstaAug that learns input-specific augmentation directly from training data. A total of 5 reviewers reviewed this paper; 2 reviewers suggested borderline accept and the remaining 2 suggested borderline reject. Most of the reviewers agreed that the paper is clearly well written and has the advantage of proposing a simple yet effective algorithm. However, despite the authors' rebuttal, this AC and the reviewers were not completely convinced of the following two issues, and in the additional ac-reviewer meeting, we also pointed out them as critical problems of this paper:

- lack of technical novelty : considering two previous works i) Augerino learning augmentation strategy only through training objective/data without valuation set and ii) papers like AdaAug (mentioned by Reviewer eZc6) learning the augmentation method instance-wisely. The proposed method can be understood as the simple combination of two approaches. In particular, this paper seems to overemphasize the instance-wise augmentation learning as the original contribution of this paper.

- lack of strong experimental evidence : i) the effect of  instance specific aug is not clear if we look at table 1 (as mentioned by reviewer 4op9). LRP may be novel but it alone is not good enough for the acceptance. ii) missing lots of recent baselines for augmentations. The proposed method also has the validation set to set its hyperparameters or as AdaAug did in its paper, parts of the training data can be used as validation.  

For those reasons, even positive reviewers did not strongly insist on accepting this paper. I would like to encourage to resubmit it to other venue after addressing the above issues especially regarding on experiments.
justification_for_why_not_higher_score: see 'summery of ac-reviewer meeting'
justification_for_why_not_lower_score: n/a
Decision: Reject