Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper provides a systematic study of data augmentation in image classification problems with deep neural networks and argues that data augmentation could replace some common explicit regularizers like the weight decay and dropout. The data augmentation techniques are also shown to be insensitive to hyper parameters, so easier to use than explicit regularizers when changing architectures.

It is good to have a systematic study of data augmentations, however, the materials in this paper in the current state might not be a strong ICLR publication. The paper could potentially be made more interesting or solid if some of the followings could be investigated:

- considering a wider range of different problems apart from image classification, and investigate the effectiveness of domain specific data augmentation and general data augmentation
- systematically study each of the data augmentation techniques separately to see which is more important (as oppose to only having 'light' and 'heavy' scheme); potentially also study other less-traditional augmentation schemes such as adversarial examples, etc.
- propose novel data augmentation schemes
- more analysis of the interplay with Batch Normalization, why the results for BN vs no-BN is not presented for WRN?
- carefully designed synthetic (or real) data / task to verify the statements. For example, the explicit regularizers are thought to unnecessarily constraint the model too much. Can you measure the norm (or other complexity measures) of the models learned with explicit regularizers vs models learned with data augmentation?


review_title: Systematic study of data augmentation in image classification problems

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents an empirical study of whether data augmentation can be a substitute for explicit regularization of weight decay and dropout.  It is a well written and well organized paper.  However, overall I do not find the authors’ premises and conclusions to be well supported by the results and would suggest further investigations.  In particular:

a) Data augmentation is a very domain specific process and limits of augmentation are often not clear.  For example, in financial data or medical imaging data it is often not clear how data augmentation should be carried out and how much is too much.  On the other hand model regularization is domain agnostic (has to be tuned for each task, but the methodology is consistent and well known).  Thus advocating that data augmentation can universally replace explicit regularization does not seem correct.

b) I find the results to be somewhat inconsistent.  For example, on CIFAR-10, for 100% data regularization+augmentation is better than augmentation alone for both models, whereas for 80% data augmentation alone seems to be better.  Similarly on CIFAR-100 the WRN model shows mixed trends, and this model is significantly better than the All-CNN model in performance.  These results also seem inconsistent with authors statement “…and conclude that data augmentation alone - without any other explicit regularization techniques - can achieve the same performance to higher as regularized models…”

review_title: Interesting empirical study but unclear outcome and results do not seem to support the conclusions

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper proposes data augmentation as an alternative to commonly used regularisation techniques like weight decay and dropout, and shows for a few reference models / tasks that the same generalization performance can be achieved using only data augmentation.

I think it's a great idea to investigate the effects of data augmentation more thoroughly. While it is a technique that is often used in literature, there hasn't really been any work that provides rigorous comparisons with alternative approaches and insights into its inner workings. Unfortunately I feel that this paper falls short of achieving this.

Experiments are conducted on two fairly similar tasks (image classification on CIFAR-10 and CIFAR-100), with two different network architectures. This is a bit meager to be able to draw general conclusions about the properties of data augmentation. Given that this work tries to provide insight into an existing common practice, I think it is fair to expect a much stronger experimental section. In section 2.1.1 it is stated that this was a conscious choice because simplicity would lead to clearer conclusions, but I think the conclusions would be much more valuable if variety was the objective instead of simplicity, and if larger-scale tasks were also considered.

Another concern is that the narrative of the paper pits augmentation against all other regularisation techniques, whereas more typically these will be used in conjunction. It is however very interesting that some of the results show that augmentation alone can sometimes be enough.

I think extending the analysis to larger datasets such as ImageNet, as is suggested at the end of section 3, and probably also to different problems than image classification, is going to be essential to ensure that the conclusions drawn hold weight.



Comments:

- The distinction between "explicit" and "implicit" regularisation is never clearly enunciated. A bunch of examples are given for both, but I found it tricky to understand the difference from those. Initially I thought it reflected the intention behind the use of a given technique; i.e. weight decay is explicit because clearly regularisation is its primary purpose -- whereas batch normalisation is implicit because its regularisation properties are actually a side effect. However, the paper then goes on to treat data augmentation as distinct from other explicit regularisation techniques, so I guess this is not the intended meaning. Please clarify this, as the terms crop up quite often throughout the paper. I suspect that the distinction is somewhat arbitrary and not that meaningful.

- In the abstract, it is already implied that data augmentation is superior to certain other regularisation techniques because it doesn't actually reduce the capacity of the model. But this ignores the fact that some of the model's excess capacity will be used to model out-of-distribution data (w.r.t. the original training distribution) instead. Data augmentation always modifies the distribution of the training data. I don't think it makes sense to imply that this is always preferable over reducing model capacity explicitly. This claim is referred to a few times throughout the work.

- It could be more clearly stated that the reason for the regularising effect of batch normalisation is the noise in the batch estimates for mean and variance.

- Some parts of the introduction could be removed because they are obvious, at least to an ICLR audience (like "the model would not be regularised if alpha (the regularisation parameter) equals 0").

- The experiments with smaller dataset sizes would be more interesting if smaller percentages were used. 50% / 80% / 100% are all on the same order of magnitude and this setting is not very realistic. In practice, when a dataset is "too small" to be able to train a network that solves a problem reliably, it will generally be one or more orders of magnitude too small, not 2x too small.

- The choices of hyperparameters for "light" and "heavy" motivation seem somewhat arbitrary and are not well motivated. Some parameters which are sampled uniformly at random should be probably be sampled log-uniformly instead, because they represent scale factors. It should also be noted that much more extreme augmentation strategies have been used for this particular task in literature, in combination with padding (for example by Graham). It would be interesting to include this setting in the experiments as well.

- On page 7 it is stated that "when combined with explicit regularization, the results are much worse than without it", but these results are omitted from the table. This is unfortunate because it is a very interesting observation, that runs counter to the common practice of combining all these regularisation techniques together (e.g. L2 + dropout + data augmentation is a common combination). Delving deeper into this could make the paper a lot stronger.

- It is not entirely true that augmentation parameters depend only on the training data and not the architecture (last paragraph of section 2.4). Clearly more elaborate architectures benefit more from data augmentation, and might need heavier augmentation to perform optimally because they are more prone to overfitting (this is in fact stated earlier on in the paper as well). It is of course true that these hyperparameters tend to be much more robust to architecture changes than those of other regularisation techniques such as dropout and weight decay. This increased robustness is definitely useful and I think this is also adequately demonstrated in the experiments.

- Phrases like "implicit regularization operates more effectively at capturing reality" are too vague to be meaningful.

- Note that weight decay has also been found to have side effects related to optimization (e.g. in "Imagenet classification with deep convolutional neural networks", Krizhevsky et al.)

REVISION: I applaud the effort the authors have put in to address many of my and the other reviewers' comments. I think they have done so adequately for the most part, so I've decided to raise the rating from 3 to 5, for what it's worth.

The reason I have decided not to raise it beyond that, is that I still feel that for a paper like this, which studies an existing technique in detail, the experimental side needs to be significantly stronger. While ImageNet experiments may be a lot of work, some other (smaller) additional datasets would also have provided more interesting evidence. CIFAR-10 and CIFAR-100 are so similar that they may as well be considered variants of the same dataset, at least in the setting where they are used here.

I do really appreciate the variety in the experiments in terms of network architectures, regularisation techniques, etc. but I think for real-world relevance, variety in problem settings (i.e. datasets) is simply much more important. I think it would be fine if additional experiments on other datasets were not varied along all these other axes, to cut down on the amount of work this would involve. But not including them at all unfortunately makes the results much less impactful.
review_title: Detailed study of data augmentation is a great idea, but more variety is needed to draw meaningful conclusions

Meta-review: The reviewers agree that the authors have made an interesting contribution studying the effect of data augmentation, but they also agree that the claims made by the paper require a broader empirical study beyond the limited number of tasks surveyed in the current revision.  I urge the authors to follow this advice and see what they find.
Decision: Reject