Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper demonstrates that for regularization, data augmentation usually works better than explicit regularization methods such as weight decay and dropout. The experiments are detailed and also includes theory explanation of why data augmentation works.

Data augmentation (or increasing the size of training data) and explicit regularization are standard methods to overcome overfitting. From my understanding, they are two different methods to tune the model quality, and they can combined to further improve the model performance, as discussed in [1]. So this paper is not well motivated. 

[1] DeVries, Terrance, and Graham W. Taylor. "Improved regularization of convolutional neural networks with cutout." arXiv preprint arXiv:1708.04552 (2017).

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper questions the conventional wisdom of using explicit regularization methods (e.g., L2, dropout) in training neural networks. The authors compare data augmentation with explicit regularization on several image classification datasets, architectures and amount of data, concluding using data augmentations is enough to reach a on-par performance with using explicit regularization. I do have several concerns about the paper.

1. My most worrying concerns are about the experiments.
(1) The ImageNet experimental setting is not that convincing since it follows a different resolution than the literature. The results obtained (e.g., 17% top-5 error) are too far from state-of-the-art. 

(2) The weight decay and dropout are used together, but not separately studied. In fact, in state-of-the-art CIFAR and ImageNet models, dropout are often not used. Though the reason is that they already use data augmentations, so dropout is typically no longer helpful (WRN is an exception). I think L2 alone is more worth studying, since it is probably known that dropout doesn't help upon a conventional augmentation.

(3) The hyperparameters used for WD+dropout in the experiments are "as specified in the original papers". But the original papers assume the conventional data augmentation. If you use new data augmentation schemes (light/heavier), the regularization hyperparameters should be tuned accordingly. I believe if the strengths of weight decay is properly tuned then it should help even with data augmentation, by a noticeable margin.

(4) If we only see WRN and DenseNet on CIFAR datasets (All-CNN is probably outdated and performs poorly, and ImageNet is not convincing as said in (1)), we notice that actually WD+dropout do provides a small increase on the augmentation schemes. This does not support the main claim of the paper.

(5) More experiments on other domains (e.g., NLP) can be used to strengthen the paper, since the title does not specify a modality.

2. "Explicit regularization techniques ... they blindly reduce the effective capacity of the model, introduce sensitive hyper-parameters and require deeper and wider architectures to compensate for the reduced capacity". I cannot agree the regularization schemes just "blindly" reduce the capacity. Take L2 weight decay as example, it does not reduce the theoretical representation power of the network, all it does is to encourage simpler solutions. Also, data augmentation schemes involve a lot of hyper-parameters too, and possibly requires deeper/wider architectures to fully exploit its advantage. In my opinion, data augmentation does not solve the possible inconvenience brought by explicit regularizations.


3.The definitions of explicit and implicit regularization in Section 2 a bit vague. Under this two definitions, I can see dropout actually falls in both categories, despite slightly more similar to the explicit one. On one hand it is specifically restricting the model's capacity by sampling a smaller model in each iteration, and on the other hand it also changes "the learning algorithm" and "characteristics of the network architecture". Similar thing holds for "Stochastic Depth". Also, injecting noise in intermediate activations is very similar to dropout since dropout is actually injecting noise by randomly removing a portion of the activations. However I can see under these two definitions injecting noise is implicit while dropout is implicit. I think it helps to list at least 5 or 6 examples for each category right there.


In summary, the claims are not well supported by the experiments, and I tend to reject the paper. 
--------------------------------------------------------------

I appreciate the detailed author response and have some quick comments:

I couldn't agree with the arguments about hyperparameters of DA. They are meaningful but their influence to the network training still needs to be tuned, possibly for different architectures, to best optimize performance.

It is also not justified that "increasing resolution" won't help in this case since your top-5 error is a bit too high compared with recent results. Especially for a strong argument in the title I would expect a more standard setting evaluated.

The DenseNet paper only uses dropout in absence of data augmentation, and in my personal experience if DA is used dropout is not helpful anymore, for ResNet as well. In standard ImageNet augmentation scheme, no one uses dropout in popular models of recent years (ResNet, DenseNet, SENet, etc.) but all use weight decay. 

I'm happy with some other parts of the response, e.g., about the title, update of definitions. Some other points seem more like opinions and I might have a different opinion from the authors, e.g., whether weight decay constrains hypothesis space.
review_title: Official Blind Review #3

final_score: rating: 1: Reject
review_content: This paper proposed to adopt data augmentation over explicit regularization (weight decay and dropout), supported by a comparative study of 3 ConvNets (all-conv, wide-resnet and densenet) on three image classification benchmarks (ImageNet, CIFAR-10 and CIFAR-100), as well as variants of experiments such as training with smaller training set.

Most of the observations presented here are largely known to the community, though I think at least a few of them are worth emphasizing:

1. One major downside of explicit regularization the requirement to tune hyper-parameters on each different scenarios. For example, the paper shows that the default hyper-parameter becomes suboptimal when the architecture becomes either deeper or shallower.
2. Given strong enough data augmentation, sometimes turning off weight decay / dropout could give better test performance (i.e. the optimal hyper-parameters are zeros in those cases).

That being said, I think this the paper at the current state does not contain sufficient message to stand as a full paper. I list a few areas that I think could potentially improve the paper if properly addressed:

1. Proper definition of explicit / implicit regularization: one goal this paper is trying to achieve is an unambiguous definition of explicit / implicit regularization. However, the current definition given by the paper is not any less ambiguous than prevision conventions. If I understand correctly, the paper defines explicit regularization as mechanism that is explicitly designed to reduce the model capacity, and implicit one as anything else (that happens to improve generalization). There is still a lot ambiguity here: for example, one technique, say, random left-right flip of the inputs, is considered implicit regularization because it improves CIFAR-10 classification performance. However, applying it to wrong data, e.g. speech recognition, would potentially hurt the test performance. Would this technique sometimes be an implicit regularization and sometimes not? Another question is regarding dropout, which is classified as explicit regularization by the paper. Yet, if dropout is applied not to some intermediate layer, but to the input layer, does it suddenly become more similar to data augmentation (think of, e.g. cutout augmentation), and therefore an *implicit* regularization? I think a if clear, mathematical and formally verifiable definition of different types of regularization, if possible, will definite make the paper stronger.

2. Theoretical analysis: currently the paper shows the definition of the Rademacher complexity and some handwavy discussions. It does not provide any additional insights apart from what the definition says: explicit regularization constrains the capacity and data augmentation increase the number of training examples. The discussion mentioned that the augmented data are non i.i.d. which would be an interesting topic for providing theoretical insights, but it is ruled "out of the scope of the paper". Besides, the definition of the Rademacher complexity captures the capacity of the hypothesis via the mathematical sup operator, and is completely independent of the underlying algorithm used to compute the sup. Therefore, data augmentation, which is part of the training procedure, does not change the Rademacher complexity, unless we completely re-define the hypothesis space to be a complicated notion that captures something like "all the functions achievable by SGD with this augmentation and that hyper-parameters, etc.". But this will turn the "implicit regularization" nature of "data augmentation" into the regime of "explicit regularization", because it is now confining the hypothesis space into a subset (so reducing the capacity).

In summary, data augmentation clearly improves the generalization performance, but to formally characterize it, one needs an alternative approach from the default Rademacher complexity way. It would make the paper much stronger if a viable approach is proposed, and even stronger if the non-i.i.d. nature of augmented data points could be discussed under the proposed framework.

3. Empirical studies: I think the paper could still be strong even if without any theoretical characterization, if it contains strong supporting experiments. However, currently the paper only studies image classification tasks and a few convolutional neural networks. Moreover, even within those domain and tasks, there are many well known techniques, such as the cutout augmentation, that are not studied. It would be of a much useful paper to the community if the paper could provide a comprehensive survey of existing explicit / implicit regularization techniques across multiple tasks and domains (e.g. the paper mentioned vision, speech recognition, NLP, etc.). Ideally, pros and cons of each regularization technique could be discussed and if the message that "data augmentation is better than explicit regularization" could hold across multiple domains and tasks, then this will definitely be delivering a strong message. Even if that is not the case, this paper could still serve as practical guides to practitioners for choosing between different approaches of regularization techniques. 

Other potential improvements are:

a. hyper-parameter tuning for each regularization technique. The paper acknowledged that the hyper-parameters are suboptimal when they change settings for explicit regularization. This is demonstrating a good point that explicit regularization is inconvenient. However, it would be better if in parallel with the default hyper-parameter, we could learn what would be the best results if we re-tune the best hyper-parameters. Because sometimes people would like to achieve the best performance in practice with all the resources and hammers they could get their hands on. So, for example, with strong data augmentation, would the (re-tuned) optimal dropout rate actually perform even better? (if true, this is also contradicting with the main message of the paper)

b. to some extent data augmentation also have "hyper-parameters", which control how to augment the data. It would be interesting to see, for example, controlled experiments on the effects on test performance when the "wrong" type or magnitudes of augmentation is applied to the data. 

minor things:

* all bars in the plots are red / purplish colors. Maybe more diverse colors could be used to make it easier to distinguish which is which?
* Why is there no DenseNet results on ImageNet? (e.g. Figure 1 missing one row?)
* As a paper that studies augmentation, it would be great to provide full details on all the details (including the hyper parameters on the magnitudes of all perturbations) of the data augmentation used, especially for the "heavier augmentation" variant.

Finally, I'm not sure if this violate the policy of anonymous submission --- the acknowledgements section mentioned explicit names when thanking for feedback. 
review_title: Official Blind Review #1

Meta-review: The paper explores the setting of *just* using data augmentation without an additional regularization term included.  The submission claims that comparatively good performance can be achieved with data augmentation alone.  The reviewers unanimously felt that the submission was not suitable for publication at ICLR.  The reasons included skepticism that augmentation without regularization is a useful setting to explore, as well as concerns about the experiments used to support the conclusions in the paper.  In particular, there were concerns that the experiments do not match best practice and that the error rates were too high.  Finally, there were concerns about the clarity of definitions of "implicit" and "explicit" regularization.
Decision: Reject