Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: A more "automated" augmentation search method is proposed. The main contributions are:

- Default transformations like flipping and cropping are regarded as new augmentation candidates which could be optimized, making the algorithm more automated.
- The gradient matching formulation is borrowed and seems to be effective for augmentation search.
- MC sampling and a progressive search pipeline are used to reduce the computational complexity.
main_review: **Pros:**

- **P1.** **[more automated]** This work takes a further step towards automated machine learning by absorbing some default image transformations (flipping and cropping) into the operation set. The idea is natural, reasonable, and worth exploring.

- **P2.** **[relatively novel; enlightening]** The formulation of gradient matching is radically different from previous bilevel-optimization based works. Though it is not a new idea to perform gradient matching in meta-learning, this opens the avenue for automated data augmentation search, and could potentially be useful for other scenarios like self-supervised learning.


**Cons:**

- **C1.** **[missing details on other default operations]** Currently there are only two default operations (flipping and cropping) being searched. But there are also some other defaults like pixel value scaling (normalization) and color jittering. Are they still used by default in DeepAA? Or are there maybe some existing experimental results that include these two (norm and colorjitter) default augmentations?

- **C2.** **[correctness of statements]** A few statements are not well-supported, or require small changes to be made correct: The authors claim that DeepAA outperforms existing automated data augmentation methods, but it seems that some recent work is missing [1, 2]. The authors should add the comparisons to the paper for correctness. Though this might make the performances of DeepAA look weaker, it wouldn’t devalue the novelty and contributions.

- **C3.** **[reliability of new metric]** This work uses a new metric rather than the validation accuracy for augmentation search. However, the reliability of this metric is not well assessed. A straightforward way is to analyze the *correlation* between gradient similarities and final performances, *e.g.*, to evaluate different augmentation policies and to see whether the policies with higher gradient similarities would have better performances.

- **C4.** **[unclear experimental details]** I got confused with some details and have some questions regarding them:
  
  - **C4.1.** In Sec. 4, the paper introduces that sub-datasets are used to train the networks. Does this mean that the CNN networks are pretrained on these sub-datasets before conducting the augmentation search (like [3])? If this is true, why not to use the full set since the pretraining seems not so expensive?
  
  - **C4.2.** In Sec. 4.1, the paper indicates the policy is updated for 512 iterations. Does this mean that only 512 images are used to search augmentation policies? Since Eq. (12) shows only one single data item is used for each policy update. If it's true, why not to use more training samples to further boost the policy? It would be insightful if a discussion is included on the choice of this hyper-parameter.

**Minor Problems:**

- [Sec. 1.] "an effective *metrics*" -- "*metric*"
- [Sec. 3.1.] "where the augmentation policies *is* shallow" -- "*are* shallow".
- [Sec. 3.3.3.] "transformations that *performs* good" -- "*perform*"
- Figure 2. and 3. are not vectorized and look blurry. Using vector graphics (like Figure 1.) could be better.

---------------------------

[1] Zhang, Xinyu, et al. "Adversarial AutoAugment." International Conference on Learning Representations. 2019.

[2] Tian, Keyu, et al. "Improving Auto-Augment via Augmentation-Wise Weight Sharing." Advances in Neural Information Processing Systems 33 (2020).

[3] Lim, Sungbin, et al. "Fast autoaugment." Advances in Neural Information Processing Systems 32 (2019): 6665-6675.





summary_of_the_review: I tend to give a rejection for now. This work is well motivated and has a natural goal. The formulation of gradient matching brings something new to this area. However, there are some flaws like claims with minor issues (**C2**) and missing details (**C1, C4**). Some claims needs to be updated and some details need further explanations. Furthermore, I have some questions about the reliability (**C3**) which may affect my recommendation as well. If all of these issues are solved, I'll raise my score.

--------
*After rebuttal: my main concerns are well addressed, hence I recommend to accept this work. Besides, the manuscript can be improved by providing more discussion and analysis. See the response to authors.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposed a learned data augmentation method. The proposed method is able to search for the augmentation policy with multiple layers by searching the next transformation operation conditioned on all previous augmentations. The authors also proposed to use cosine similarity between the gradients of the original and augmented data as guidance rather than e.g. the more commonly used classification validation loss.
main_review: ## Strength
- The proposed method reduces the searching time for multi-layered augmentation policy by a large margin while achieving good performance.
- It is interesting to see that the latter augmentation layers converge to identify mapping (Fig3), which is a good indicator for the optimal number of augmentation layers.


## Weakness
- It is not well justified in the experiment section why formulating data augmentation search as a gradient matching problem is favorable. Even though the authors provide the similarity scores after stacking more augmentation layers in Tab6, the comparison of the cosine similarity of gradient between the proposed method and other augmentation methods is missing. Furthermore, the reason that higher cosine similarity of gradients leads to better final performance is missing. There exist a correlation between these two but the reviewer expect more analysis and insights for the causality.
- It is unclear whether the proposed method generalized to other tasks (eg. object detection)
summary_of_the_review: One major weakness of this work is that the main idea of using cosine similarity of gradients as guidance for policy search is not well justified in the experiment section. Except for that, based on the experimental results that (1) the proposed method reduces the policy search time by a large margin while maintaining good performance compared to other SoTA augmentation methods, and that (2) the proposed method is applicable to multi-layer augmentation, overall the reviewer considers it a good paper.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper provides an interesting finding that instead of injecting experts prior to hand-pick augmentations at the top layers, the entire search can be formulated end-to-end in a multi-layer (deep) way without hand-picked transformation optimized with gradient matching. The independent sampling at each layer with estimated mean gradient of training data is used to stabilize the training and make the optimization process more efficient. Empirical results on CIFAR-10/100 and ImageNet outperform previous best performing DAS methods.
main_review: Strengths: 
+ Relaxing the strong human assumption on data augmentation for specific data is novel and well motivated. 
+ The proposal of using cosine similarity between the augmented training data and validation data as a reward signal to optimize the sampling probability of the corresponding data augmentation transformations is technically sound. 
+ Table 1, 2 shows its better performance than SOTA DAS methods. Table 4, 5 further shows its increasing performance given an increasing number of layers and its stable performance given multiple runs. 
+ Figure 3 shows in the top (deep) layers the augmentations are not dataset-specific and therefore justify its motivation. 

Weakness: 
- Isn’t the optimal similarity between train/validation to transform validation data to training data? Could the author[s] provide some intuition on why using gradient matching for the search. If it will cause overfitting issues on training data? 
- Fig.3, 4 and 5 show different operation selections at different stages for DeepAA. Could the author[s] show the difference between previous works? 
- Since this work is to advocate less human-constraints on operator selections, it would make this work better if the author[s] can show experiments on data-diversity cases where there is no prior knowledge about the data transformation. Showing extra results on some transfer learning experiments on different datasets will be highly appreciated.
summary_of_the_review: Nice motivation and an interesting approach. Results on standard benchmarks are strong compared to other DAS methods. Experiments can be further expanded to different datasets to show the effectiveness of the proposed method. Can also add more comparisons to the previous methods in Fig. 3/4/5.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper proposes a novel automatic augmentation method. That is a method that learns based on a dataset, how to augment that dataset during training of a classifier.

The proposed method works by matching gradients based on cosine similarity on fully trained networks and uses this approach to build augmentation policies with multiple augmentations applied after one another. Similar to the recent usage of very deep RandAugment for EfficientNetV2s for example.
main_review: Strengths
- Getting rid of default augmentations is an interesting step
- Approach to get around the need of RL in previous work
    - Problem: Little misalignment with final objective, 1 sample vs full mean
- Nice visualization of the policy choice of DeepAA
- Repeated experiments with confidence intervals

Not Clear
- Do you optimize all augmentation layers at once or in separate optimization steps?

Weaknesses
- The statement about SOTA in the abstract is _not true_. The SOTA for C-10 with WRN-28-10 is at least 84.33 as reported in TrivialAugment, which is stronger than the 84.02 of this method. Similarly, TrivialAugment is forgotten in the comparison for ImageNet, even though here Deep AA is stronger but not as much as stated.
- I believe stating to use 139 augmentations is a little misleading, as you treat the same augmentation with different strengths as different augmentations
- Another experiment on the relatively standard SVHN datasets would be interesting, as this has very different images.
- Claim of not using default augmentations is weakened by the fact that the search space is changed for datasets that require different default augmentations
- Comparison to TrivialAugment is a little unfair, as TrivialAugment’s standard setting is stronger than the results presented, and TA actually includes a run with CIFAR-100 and ShakeShake
- Missing true baseline. I would be more sure of the method's performance if there was a simple baseline implemented on the same search space and in your codebase, either UA or TA: This could actually work out very well for you, too, if you include another baseline of a trivial baseline without standard augmentations, too.
- A baseline for Table 6 would be good to answer whether the cosine similarity increase with any augmentation application? That could be done by applying RA with different numbers of layers or applying TA multiple times.

Very minor: there are some typos here and there.

summary_of_the_review: Overall, I believe the proposed method is interesting: it builds interesting policies in an intuitive way and requires less per-dataset adaptions than previous work.

I do believe, though, that the results simply are not strong enough for publication and a few comparisons in the paper are unfair to baselines. It might be possible to improve upon this method in some ways though, for example with Batch Augmentation ("Augment Your Batch:.." by Hoffer et al.), as this is closer to \theta’s objective.
review_title: null

Meta-review: We appreciate the authors for engaging in discussions with the reviewers and providing further experimental results to clarify and address the concerns raised by them in their original reviews, leading to changes in some of the recommendations.

While the (revised) paper with the clarifications and new results incorporated are more ready for publication, some outstanding concerns should preferably be addressed to further enhance its quality.

The authors are highly recommended to take into consideration all the comments and suggestions of the reviewers to further revise their paper to make it a scholarly work to contribute to the ICLR and the more general ML community.
Decision: Accept (Poster)