Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper proposes a novel method called augMix, which creates synthetic samples by mixing multiple augmented images. Coupled with a Jensen-Shannon Divergence consistency loss, the proposed method has been experimentally, using CIFAR10, CIFAR100, and ImageNet, shown to be able to improve over some augmentation methods in terms of robustness and uncertainty.
 
The paper is very well written and easy to follow. The idea of the approach is simple, and should be easy to be implemented. The evaluation of the proposed method is currently based on experimental evidence. Nevertheless, the empirical studies in its current form could be further improved. Please see my detailed comments below.

1. The proposed approach relies on a chain of augmented methods. In this sense, experimental studies on the sensitivity for how the augmentation methods in the chain (e.g., augmentation operations) and their chain structure (e.g., length of the chain) impact the performance of the augMix should be provided. This is in particular relevant because the authors did mention that “adding even more mixing is not necessarily beneficial” on page 8.

2. Since the proposed method mixes multiple augmented images, a more appropriate comparison baseline would be a method involving creating synthetic data with multiple images. For example, the n-fold Mixup method as discussed in Guo AAAI2019 (Mixup as Locally Linear Out-Of-Manifold Regularization).

3. Some experimental results/observations deserve further discussions. For example, on page 8, the authors mention that “applying augMix on top of Mixup increases the error rate to 13.3%”. I wonder if the authors could provide any insights or hypothesis on why the proposed model behaviors in this way? 

4. Would that be any manifold intrusion issue as discussed in Guo’s AAAI2019 paper? That is, would it be possible to generate images that are very close to some real images but with different labels? For example, by looking at the bottom-center image in the Appendix B, the synthetic image created seems to be close to some birds with other categories. 

5. Does the method work for other network architectures such as DenseNet?


*********new  comment**********
During the rebuttal period, the paper has been improved with additional experimental results, analyses, and observations. I therefore have adjusted my evaluation score accordingly. 


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: The paper discusses a new  data augmentation method which improves the accuracy of the network for several specific shifted domain scenarios. The main goal of the paper is to increase the robustness of the deep model trained on the augmented data to generalize well beyond the data corruption like the  rotation, translation, noise,.... For each input, they apply  $k$ different operation of image shift and make the weighted combination of them. The weight vector is generated randomly from Dirichlet distribution with the parameter $\alpha$.  The weighted combined images would be added to the original image in convex combination. The convex weights are generated from distribution Beta with parameter $\beta$. Later they train the network with adding the Jensen-Shannon divergence for the posterior distributions of augmented images as the consistency regularizer.  They show this data augmentation will increase the accuracy of the model for shifted and non-shifted domains and also it leads to more calibrated model for domain shift problem.

Pros: 
the paper is well-written with clear implementation details. The level of experiments are wide and cover different aspects. The experiments shows the significant improvement compared to several baselines. The authors conducted the experiments for a wide range of model-datasets to show the validity of their ideas. 

Cons: 
1- The title of this work is a strong claim that is not supported in the paper. In this paper, it is mentioned that AugMix is a data augmentation method that generates data to add to the training set and after training with data augmentation, the method would be more robust to other distortions that can be added to the datasets. Generally, the definition of domain shift is wider than just adding perturbation to the dataset.  To support the claim, the paper should also report the results for similar tasks datasets such as CIFAT10-STL10- or MINIST-SVHN for different models and with different domain adaptation methods. The claim about the improvement of uncertainty also is not supported well by the experiments. The method should be tested for many model-datasets specifically, to support improving the  uncertainty under the domain shift idea like the paper [1].  

2- The novelty of the work is limited. The generating method of distorted  images is the combination of previously proposed methods like [2] and [3].  The motivation of why the proposed method is working well is not clear. How this objective function can improve the robustness to the image perturbation but it does not lose the accuracy is not discussed. It would be better if the proposed method were supported by theory and also the intuition and explained why it should get better results than previous data augmentation methods such as AutoAugment [3].

3-  Fine-tuning the parameters like $k$, $\alpha$ and $\beta$ is not discussed at all.

4- To show the robustness of the proposed method to domain shift, the paper compares the proposed method to other data augmentation methods that are not designed for domain shift which seems unfair.  

References:
[1] Ovadia, Yaniv, et al. "Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift." arXiv preprint arXiv:1906.02530 (2019).
[2] Zhang, Hongyi, et al. "mixup: Beyond empirical risk minimization." arXiv preprint arXiv:1710.09412 (2017).
[3] Cubuk, Ekin D., et al. "Autoaugment: Learning augmentation policies from data." arXiv preprint arXiv:1805.09501 (2018).

review_title: Official Blind Review #3

final_score: rating: 8: Accept
review_content: This paper proposes a method called AugMix, which is intended to improve model robustness to data distribution shift. AugMix appears fairly simple to implement. Several new images are created by augmenting an original image through chains of sequentially applied transformations (the "Aug" part of AugMix), then the augmented images are combined together, along with the original image, via a weighted sum (the "Mix" part of AugMix). Additionally, a Jensen-Shannon Divergence consistency loss is applied during training to encourage the model to make similar predictions for all augmented variations of a single image. This technique is shown to achieve state-of-the-art performance on standard robustness benchmarks without loss of clean test accuracy, and is also shown to improve calibration of model confidence estimates.

Overall, I would tend to vote for accepting this paper. The method is simple yet effective, the paper is very well written and easy to follow, and experiments are extensive and, for the most part, convincing.

Questions:
1) The one main concern I have with the training procedure is the amount of time the models were trained for. It is known that model trained with aggressive data augmentation schemes often require much longer training than normal in order to fully benefit from the stronger augmentations. For example, AutoAugment trains ImageNet models for 270 epochs [1], while CutMix trains for 300 epochs [2]. However, the ImageNet experiments in this paper claim to follow the procedure outlined in [3], which only trains for 90 epochs. This is reflected in the clean test accuracy, where AutoAugment only appears to provide a 0.3% gain over standard training, while we might expect 1.3% improvement (according to [2]). The AllConvNet and WideResNet in CIFAR-10 and CIFAR-100 experiments were also trained for only 100 epochs each, where 200 is more conventional. Again this shows in the reported numbers: on WideResNet for CIFAR-10, Mixup only has a 0.3% gain were as we might expect 1% improvement instead [4], and AutoAugment has 0.4% improvement, were as we might expect 1.3% gain if trained longer [1]. My question then is, how much does training time affect results? Do AugMix, and other techniques such as Mixup, CutMix, and AutoAugment, achieve better robustness when models are trained for longer, or do they become more brittle as training time is extended? 

2) For the Jensen-Shannon divergence consistency, how much worse does it perform when using JS(Porig;Paugmix1) versus JS(Porig;Paugmix1;Paugmix2)? What might cause this behaviour?

3) Patch Gaussian is changed to Patch Uniform to avoid overlap with corruptions in ImageNet-C. How does Patch Uniform compare to Patch Gaussian in terms of performance for non-Gaussian noise corruptions?

4) How does AugMix perform as an augmentation technique in terms of clean test accuracy compared to other SOTA techniques? Is there a trade-off between clean test accuracy and robustness, or does AugMix improve performance in both domains? Can AugMix be combined with other augmentation techniques or does this destroy robustness properties?

Things to improve the paper that did not impact the score:
5) It would be nice if the best result in each column could be bolded in Tables 2-4.

References:
[1] Cubuk, Ekin D., Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V. Le. "Autoaugment: Learning augmentation policies from data." CVPR (2019).

[2] Yun, Sangdoo, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. "Cutmix: Regularization strategy to train strong classifiers with localizable features." ICCV (2019).

[3] Goyal, Priya, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. "Accurate, large minibatch sgd: Training imagenet in 1 hour." arXiv preprint arXiv:1706.02677 (2017).

[4] Zhang, Hongyi, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. "mixup: Beyond empirical risk minimization." ICLR (2018).
review_title: Official Blind Review #2

Meta-review: This paper tackles the problem of learning under data shift, i.e. when the training and testing distributions are different. The authors propose an approach to improve robustness and uncertainty of image classifiers in this situation. The technique uses synthetic samples created by mixing multiple augmented images, in addition to a Jensen-Shannon Divergence consistency loss. Its evaluation is entirely based on experimental evidence.

The method is simple, easy to implement, and effective. Though this is a purely empirical paper, the experiments are extensive and convincing. 

In the end, the reviewers didn't show any objections against this paper. I therefore recommend acceptance.
Decision: Accept (Poster)