Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper proposes to completely remove even the two hyperparameters in RandAugment by full random selection of augmentation policies without any searching and achieve the comparable performances by simple multi-stage complexity driven augmentation strategy.

Pros.
It is interesting that the fully random selection of augmentation policies during training can produce competitive performances in comparison to the found augmentation policies by the previous augmentation search algorithms. In specific, with multi-stage training from hard augmentations to easy augmentations, it can improve the overall performances even when incorporating the mix-based augmentations which cannot be handled by the previous AutoAugment algorithms.
Experimental results on various tasks including not only image classification but also face recognition and text detection show that it consistently obtains improved performances over the baselines.

Cons.
It seems that the contribution of this work from the perspective of the algorithm novelty would be marginal from RandAugment. For example, the tuning for epoch allocation for each stage in the proposed method can be compromised with the hyperparameter search in RandAugment. And, the proposed manual three-stage augmentation scheduling can be compared with automatic augmentation scheduling such as PBA, AdvAA, and OHL-Auto-Aug (C. Lin, et al., Online Hyper-parameter Learning for Auto-Augmentation Strategy, ICCV 2019). Especially, compared with online augmentation learning in OHL-Auto-Aug that uses just 8 or 4 parallel trainings, the tuning for epoch allocation in the proposed multi-stage approach would not be computationally efficient, and it seems to largely affect the performances for each task, maybe network and dataset, etc.

Why the obtained results in Table 1 are different from Table 3 and Table 4 (97.97/80.27 vs. 97.75/79.89)?

It seems that the performance improvements by the use of mix-based augmentations are marginal. 
review_title: Official Blind Review #2

final_score: rating: 3: Clear rejection
review_content: This paper aims to provide an effective augmentation strategy without the need for a separate search. The resulting method is called NOSE Augment, which is presented as a substitute for the previous AutoAugment type methods (e.g. Fast AutoAugment, Population Based Augmentation, RandAugment, Adversarial AutoAugment etc.) In parallel to this goal, the authors propose adding the mixing-based augmentation operations Mixup, Cutmix, and Augmix into the list of operations used in AutoAugment. Finally, authors also employ a curriculum of augmentation strength during training. 

While the cost of search for AutoAugment-type policies has been significantly reduced (e.g. PBA and Fast AA), it is still a worthwhile goal to want to remove the search phase completely, for further reduction of computational cost as well as convenience. While RandAugment does not use a separate search phase, it still has two hyperparameters that need to be optimized, similar to learning rate or weight decay, as the authors of this submission correctly point out. Thus I find the goal of wanting to remove the 2 hyperparameters that RandAugment requires worthwhile. However, I do not believe that the results shown in the paper indicate that NOSE Augment has achieved this goal, for reasons detailed below. Thus, I do not believe this work presents an improvement over previous work.

1)  RandAugment paper found that the optimal strength of augmentation depends on model size and dataset size, and found the optimal strength to be especially different for small ImageNet models such as Resnet-50 vs. large ImageNet models such as EfficientNet-B7. Since NOSE Augment is only evaluated on models that similarly sized to each other, it is hard for us to know if NOSE Augment would also do well on larger models such as EfficientNet-B7, or much smaller datasets that were explored in RandAugment. Looking at the optimal magnitude that was reported in RandAugment for the models NOSE Augment was evaluated on, it would not be surprising that random AutoAugment policies would do well. The authors should evaluate their method on a larger model such as EfficientNet-B7 or on a small dataset such as a small subset of CIFAR-10 to see the performance of NOSE Augment on different model sizes and dataset sizes. 

2) This paper adds several different components to their method at once, without any ablations on where the improvements are coming from. Previous work has seen that combining AutoAugment and Mixup can be helpful (e.g. see  "Compounding the Performance Improvements of Assembled Techniques in a Convolutional Neural Network" by Jungkyu Lee et al.) In the case of NOSE Augment, it is not clear how much of their performance results from adding different methods on top of each other vs. other factors such as the curriculum. 

3) It is also not clear to me what the main contribution of this paper is. Random AutoAugment policies have been evaluated both in AutoAugment and RandAugment papers. AutoAugment paper found that 25 random subpolicies did somewhere between Cutout and AutoAugment. RandAugment paper found that uniformly randomly sampled magnitudes do as well as constant magnitude on cifar-10. If the main contribution of NOSE Augment is the curriculum they use for augmentation, the authors should provide experiments that show that the curriculum alone gives improvements, and not the addition of new operations such as Mixup, Cutmix, and Augmix. PBA and RandAugment papers have both tried curriculum for augmentation, and it seems like RandAugment without curriculum can get as good results as PBA or RandAugment with curriculum.

4) For a paper on augmentation strategies, the results should only focus on improvements due to the augmentation strategy, and not any other model decisions such as training protocol or architecture. Looking at Table 1 of the paper, it is not clear to me if the authors have run their own baselines for each architecture, and what accuracy their own baselines got. The baseline column in Table 1 matches the baseline results that were reported in the AutoAugment paper to 4 significant figures for every row. This makes me think that the numbers reported as baseline are not actually the accuracies that the authors would necessarily get if they ran their experiments with just the baseline augmentation. For example, we know that ResNet-50 can get much higher accuracy than 76.3% on ImageNet, without any advanced augmentation, if other regularization methods are employed (e.g. label smoothing etc.) I would urge the authors to report the results of their own baseline models in all of the relevant tables, so that the reader can directly see how much of the reported performance is due to the proposed augmentation strategy.


review_title: Review of NOSE Augment

final_score: rating: 5: Marginally below acceptance threshold
review_content: The authors propose a method for learning an augmentation pipeline for image recognition. As opposed to recent existing approaches such as AutoAugment or RandAugment, the authors do not seek for the augmentation pipeline iteratively. Instead they use a stochastic approach, where augmenters are split to three categories based on their complexity to be used by curriculum learning.

The exact description of the selection method is not clear. Despite reading it several times, I did not fully understand the procedure. Moreover, the method and the split into the three categories (BaseAug, AdAug, SuperAug) seem somewhat arbitrary, and questions the validity of the results. 

The results seem appropriate, and suggest that the proposed method can reach the state of the art; outperforming AutoAugment and Adversarial AutoAugment with a fraction of computational cost. However, from a practical standpoint, I consider all results in Table 1 almost equal. The minor differences in accuracy are not justifying the use of the proposed method. Instead, the authors should emphasize even stronger the computational savings; against all methods, not just two.

Finally, it is critically important that the authors specify what is the baseline (also called "standard augmentation").
review_title: Stochastic augmentation at three stages

Meta-review: All reviewers generally admit that the motivation of realizing search-free autoaugment is reasonable and important. However, they also raised many concerns regarding the experimental evaluation to validate the practical effectiveness of the method. In particular, unclear discussion with respect to ablation studies, and the lack of the baselines implemented by the authors were the central issues that obscure the essential effect of the core contribution of the work.  The authors made great efforts to conduct additional experiments and did address some of those issues, however some experiments are not yet ready such as the baseline implementation on ImageNet and testing on large models. After the discussion phase, all reviewers decided to keep their initial scores toward rejection, and the AC agreed with their opinions. 

In summary, the paper focuses on an important problem and the proposed method is potentially very useful, but the paper in its current form should be further polished and completed before publication, thus I recommend rejection this time. 
 
Decision: Reject