Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: This paper shows that a penalty term called rugosity captures the implicit regularization effect of deep neural networks with ReLU (and piecewise affine in general) activation. Roughly, rugosity measures how far the function parametrized as a deep network deviates from a locally linear function. 

The paper starts by showing that the amount of training loss increased from adding data augmentation is upper bounded in terms of (roughly) a Monte Carlo approximate to a Hessian based measure of rugosity. It then formally derives this measure of rugosity for networks with continuous piecewise affine activations. Finally, experimental evaluation for classification tasks on MNIST, SVHN and CIFAR shows that data augmentation indeed reduces the rogusity by a significant amount particularly when using the ResNet structure. A somehow surprising message is, however, that if one imposes explicit regularization with rugosity in lieu of data augmentation, then the better generalization usually seen from data augmentation no longer presents, though one does get a network with smaller rugosity.

Comments:

It is quite interesting to see that the rugosity measure proposed in the paper captures at least some aspects of the implicit regularization effect of data augmentation both in terms of theory (i.e. Theorem 1) and practical observations. My feeling is that rugosity is mostly a measure of the smoothness of the function parametrized by the neural network. From that perspective, how is the rugosity as a smoothness measurement for neural networks with piecewise affine activations different from the Lipschitz constant for general neural networks? My guess is that data augmentation also decreases the Lipschitz constant of a neural network near the training data points, but regardless of whether this is true or not, it is not clear if and how rugosity is better than Lipschitz constant for characterizing the implicit regularization of data augmentation. 

In addition, there have been many recent studies on showing that gradient penalty / Lipschitz regularization are useful for achieving better generalization and adversarial robustness, see e.g. [a,b,c]. The results in this paper on showing that regularizing rugosity does not improve accuracy seem to contradict with the conclusion of these prior studies. It is unclear to me whether this is caused by insufficient experimentation or if there is any fundamental difference between rugosity and Lipschitz regularization that I am missing.

[a] Finlay et al., Lipschitz regularized deep neural networks generalize and are adversarially robust
[b] Gouk et al., Regularisation of Neural Networks by Enforcing Lipschitz Continuity
[c] Thanh-Tung et al., Improving generalization and stability of GANs





review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: The paper aims to explain the regularization and generalization effects of data augmentation commonly used in training Neural Networks. 
It suggests a novel measure of "rugosity" that measures a function's diversion from being locally linear and explores the connection between data augmentation and the decrease in rugosity.
It further suggests the explicit use of rugosity measure as a regularization during training to replace need for data augmentation.
The paper is very well written and both the positive and negative findings are clearly presented and discussed. 
Cons:
- The main contribution of the paper, in my view, is the suggestion of using rugosity as a explicit regularization for training Neural Networks. Nevertheless, all the results in the paper show a negative impact of this on the test accuracy which is contradicting to the proposition. 
This result has been discussed in section 5 but without much evidence to the explanations mentioned. The connection is very interesting but I believe further work is needed to explain those negative results on test accuracy. 
- The difference in finding (Table 1) between the CNN and ResNet networks can be more discussed. 
- Additional tasks (like regression) or even toy examples can be useful in further explaining the connection between rugosity and generalization to test data. 
 
review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: This paper shows (theorem 1) that data augmentation (DA) induces a reduction of rugsity on the loss function associated to the model. Here rugosity is defined as a measure of the curvature (2nd order) of the function. However, the two concepts seems to be different because the authors empirically show that directly reducing the rugosity of a network does not improve generalization (in contrast to DA).

I lean to reject this paper because the contributions, even if interesting, do not lead to any new understanding of the topic. More in detail, data augmentation improves the generalization on deep learning models. This paper shows that DA induces rugosity (theorem 1), but rugosity does not improve generalization (empirically). Thus, rugosity is not responsible for generalization, which is the interesting property that we care about.

The paper is well written and easy to follow, however I found the actual contribution limited because:
- The definition of rugosity is an extension of (Donoho & Grimes (2003)) in which the extension is not really improving anything or used anywhere in the paper.
- The Hessian-based rugosity analysis of DA is correct, but it does not help to understand the generalization performance or any other useful property of DA.
 
Additional Comments:
- In 3.4 second paragraph the authors suggest that reducing rugosity can improve generalization as DA, but later we see that this is not the case.
- The entire paper seems written with the idea of using rugosity as a surrogate of DA, but at the end it does not work

review_title: Official Blind Review #1

Meta-review: This paper aims to study the effect of data augmentation of generalization performance. The authors put forth a measure of rugosity or "roughness" based on the tangent Hessian of the function reminiscent of a classic result by Donoho et. al. The authors show that this measure changes in tandem with how much data augmentation helps. The reviewers and I concur that the rugosity measure is interesting. However, as the reviewer mention the main draw back of this paper is that this measure of rugosity when made explicit does not improve generalization. This is the main draw back of the paper. I agree with the authors that this measure is interesting in itself. However, I think in its current form the paper is not ready for prime time and recommend rejection. That said, I believe this paper has a lot of potential and recommend the authors to rewrite and carry out more careful experiments for a future submission.
Decision: Reject