Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper illustrates an adaptive based data augmentation method named as AdaAug that searches adaptive augmentation policies in a class-dependent and potentially instance-dependent manner to improve the generalisation capability of deep learning models. This paper proposes an efficient exploition-exploration workflow to search for an augmentation policy that optimizes the generalization performance. The empirical studies on some datasets shows that the performance increase with AdaAug.
main_review: Strengths

1. Adaptive augmentation by learning a class-dependent and potentially instance-dependent augmentation policy for each data instance is a nice idea with strong possibility in augmentation
2. Empirical studies has been conducted on different scenarios.
3. The paper is well-written and easy to follow.

Weaknesses
Not enough experiment. From the experimental results, it gives me an impression that the results are happen to be successful. Multiple trials with mean and variants should be given. 
summary_of_the_review: I will give marginally acceptation for this paper. The idea is clear and novel but the novelty and contribution is not quite sounding. I can not decline that this is an interesting paper for this community.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper introduces a data augmentation method AdaAug that learns adaptive augmentation policies in a class-dependent and potentially instance-dependent manner to improve the generalisation capability of deep learning models. Concretely, it proposes an efficient exploition-exploration workflow to search for an augmentation policy that optimizes the generalization performance. Experimental results on datasets with transfer and direct settings show the efficacy of this approach.
main_review: **Strengths**
1. The motivation of this work is clear and the idea of searching data-dependent augmentation policies seems reasonable.
2. Empirical investigation is conducted on different scenarios. 
3. The paper is well-written and easy to follow.

**Weaknesses**
1. There are some related works this paper omitted. For example, [1] has also considered individual sample variations in augmentation policy searching and it proposed a meta-learning approach to learn the augmentation policies. This paper can compare with this in terms of the idea and performance, so that we will know which approach is better under which scenarios. Also, regarding the performance, the authors mention AdaAug has achieved state-of-the-art performance on the CIFAR-10, CIFAR-100, and SVHN datasets. However, according to [1], it seems AdvAA [2] performs better than the baselines in this paper. Did the authors ever compare with this method? What are the results?
2. The performance improvement of the proposed approach seems marginal. Some prior works such as MetaAug [1] and AutoAug [3] run multiple times for the model and present the mean and standard deviation. Did the authors run multiple times for each model and what about presenting the mean and standard deviation etc?
3. Results on different neural network architectures are missed. Table 3 only presents results on the Wide-Resnet model. What is the performance on different models such as Shake-Shake (26 2x96d) and PyramidNet+ShakeDrop as shown in [1,2,3]?

**Other questions**
* As this work automatically learns the policy, we are curious about the issue of convergence during training. Is the convergence guaranteed on both training and validation data? Is it possible to do a theoretical convergence analysis?

**References**

[1] Zhou et al., MetaAugment: Sample-Aware Data Augmentation Policy Learning, AAAI 2021

[2] Zhang et al., Adversarial Autoaugment, ICLR 2020

[3] Cubuk et al., AutoAugment: Learning Augmentation Strategies from Data, CVPR 2019

summary_of_the_review: The topic of this paper is interesting and worth exploring. Overall, the paper is well-written and the idea is novel. However, there are some questions regarding the experiments and analyses. We would like to see the feedback of the authors to the questions above. Also, if the performance improvement of the proposed model is not significant, we would like to see some theoretical analyses on the advantages of this work.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a search algorithm for instance conditional data augmentation. The contributions are two-fold: 1) it could be the first trial to automatically learn class-dependent or instance-dependent augmentations; 2) an efficient workflow is designed that not only realizes gradient-based augmentation search but also extends the search space of augmentation magnitudes to continuous.
main_review: **Strengths:**

- **S1.** **[reasonable and well-motivated]** The basic idea of this work that makes the augmentation conditional on data itself seems reasonable and worth exploring. Examples like "flip-invariance in digit classification" discussed in the paper motivates this work well.

- **S2.** **[relatively novel]** It is not a new idea to explore the class or subgroup conditional augmentations as introduced in the related work, but the originality may not be overwhelmed since this work may be the first trial to *automatically* learn instance-dependent augmentations.

- **S3.** **[enlightening]** The policy search phase (the exploration pass) in this method is elegant and extends the augmentation search space: it mixes the activations of different augmented images so that it avoids discretizing the augmentation magnitudes (while much prior work did this). This is not a major strength but a favorable bonus that allows better exploration for augmentation search, thus is appreciated.


**Weaknesses:**

- **W1.** **[missing ablation]** Some prior work like AutoAug [1] presents their search-free results to prove the augmentation policy does benefit from the search phase. If it is possible, these results should be shown to substantiate the benefits of the augmentation search. A straightforward way is to evaluate a randomly initialized $h_\gamma$ (not trained ever) on downstream experiments while keeping other *diversity parameters* the same.

- **W2.** **[unclear performance improvements]** The performance reported in Table 5. shows that large $\delta$ (*e.g.*, 0.4) would bring substantial improvements. The large-$\delta$ setting makes the augmentation policy close to a random one given the augmentation magnitude ranges from 0 to 1. This can raise the suspicion that the performance gains mainly come from the rich randomness of augmentation magnitudes, not the augmentation search algorithm. Reporting the variance (*e.g.*, standard deviation) of the augmentation magnitudes $\lambda$ and comparing it with the value of 0.4 would help further clarify this.


**Open questions:** (not weakness)

- **O1.** As a regularization technique, data augmentation almost always improves the loss, *i.e.*, augmented data have larger loss values than original ones. Thus, it is somewhat surprising that the augmentation policy does not collapse to "no augmentation" (*i.e.*, all magnitudes turn into zero) when we minimizing the loss of augmented data in the exploration pass. Could the authors explain this?


- **O2.** The authors claim that this method approximately reduces the distance between the densities of **augmented $D_{valid}$** and **$D_{train}$**. In contrast, I feel "reduces the distance between the densities of **augmented $D_{train}$** and **$D_{valid}$**" could be more natural and reasonable, as in most deep learning cases we apply data augmentation only on training data and hope these augmented samples could be like validation data.
    
    - In order to "reduce the distance between the densities of **augmented $D_{train}$** and **$D_{valid}$**", we need to use $D_{valid}$ to train $f_{\alpha}$ and $g_{\beta}$, and use augmented $D_{train}$ to optimize $h_{\gamma}$.
    
      - [use $D_{valid}$ to train $f_{\alpha}$ and $g_{\beta}$]: the exploitation pass does not perform any augmentations since we want to use $D_{valid}$ itself to train $f_{\alpha}$ and $g_{\beta}$. Thus, this pass could be understood as "encouraging $f_{\alpha}$ and $g_{\beta}$ to memorize the distribution of $D_{valid}$".
    
      - [use augmented $D_{train}$ to optimize $h_{\gamma}$]: the exploration pass could be regarded as "making augmented training data more close to validation data via minimizing the loss of augmented training data".
     
    - How do the authors think of this view?


-------------------

[1] Cubuk, Ekin D., et al. "Autoaugment: Learning augmentation strategies from data." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.





summary_of_the_review: I incline to give a marginal accept now given the originality and enlightening contributions. However, there are some questions towards the experiments that may affect my judgment. See main review for more details.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: - This paper proposes AdaAug, an Automated Data Augmentation (AutoDA) method to learn a class/instance-dependent augmentation policy efficiently. The key ideas of AdaAug are two-fold. First, it uses a hidden feature of the original input to adapt the augmentation for each instance. Second, it alternates exploit-and-explore procedures for efficiently updating the augmentation policy. The authors demonstrate the empirical effectiveness on two scenarios; 1) transfer and 2) direct. In transfer setup, AdaAug significantly outperforms other AutoDA baselines. In direct setup, AdaAug shows comparable performance with the baselines.
main_review: **Pros.**

- **Clarity**. Overall, the writing is clear and easy to follow
- **Well motivated problem and novel approach**. Instance-awareness of AutoDA is an essential but not explored aspect for better performance on the learned dataset and transferability to other datasets. The proposed idea that utilizes the hidden feature of the original input seems to be reasonable and novel.

**Cons.**

- **Experiments/Ablation for diversity parameters**. The authors introduce the three diversity parameters for further adapting the learned policy. However, the proposed method itself is also possible to adapt the augmentation based on the feature map of each instance. Why these additional diversity parameters are required? Is it necessary? Also, for a precise ablation, the results without these diversity parameters should be provided in a direct setup (Table2), and the comparison of distribution for resulting augmentations is required (e.g., with/without the diversity parameters, how the learned augmentation is adapted from CIFAR-100 to each target dataset?)
- **ImageNet experiments**.
    - First, it is quite confusing as the experimental results on ImageNet are presented in Section 4.1 (Transfer setting). Is it right or should it be moved to Section 4.2.?
    - Next, although the proposed method is quite worsen than other automatic augmentation learning methods in Table 2, I believe that the advantage of the proposed method is task-wise adaptability which more benefit from the transfer learning as done in Table 1 (as fine-tuning of the model from ImageNet pre-trained model is one of the most promising transfer learning nowadays) Can the proposed method outperform other fixed augmentation when we finetune the ImageNet pre-trained model?
- **Weighted summation within exploration**. According to Equation (2), the weighted summation of differently augmented samples is used to generate a prediction for updating augmentation policy. Hence, as the one sampled augmentation is only used at the exploitation, there is a discrepancy between exploration and exploitation in the way for using augmentation. Is the weighted summation is essential for exploration, or can it be replaced by another method? For example, one can use a Gumbel-softmax trick at exploration.

**Other comments.**

- **Illustration of the augmentation policy magnitude $\lambda$**. In Figure 2, the authors show the learned probability p to provide empirical evidence of class/instance-dependent augmentation. Since the magnitude $\lambda$ is also an important learnable parameter, it would be helpful to validate the proposed method if the corresponding illustration is additionally provided.
- **Qualitative results for better transferability with instance-awareness**. In Table 1, the proposed AdaAug significantly outperforms the other baselines by automatically adapting the augmentation. If the distribution of used augmentation can be compared qualitatively as done in Figure 2, it would enhance the empirical justification. (i.e., how the distribution of augmentation is **quantitatively** changed for the different tasks, instead of **qualitatively** as done in Figure 3?)
summary_of_the_review: Although the target problem is important and the proposed method is novel, the empirical significances and analysis are quite insufficient currently. If the authors can provide that 1) experimental results fine-tuned from ImageNet model and 2) the empirical analysis how the augmentation parameters are actually changed by proposed method, especially in transfer scenario, I'll raise my score.
review_title: null

Meta-review: Reviewers agreed that this work is well-motivated and presents a novel approach for data augmentation around the adaptive augmentation policies. There were some concerns around the lack of ablation studies and unclear performance improvements, which were addressed well by the authors’ responses. Thus, I recommend an acceptance.
Decision: Accept (Poster)