Venue: ICLR 2017 conference

Reviews:

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: This paper offers a very interesting empirical observation regarding the memorization capacity of current large deep convolutional networks. It shows they are able to perfectly memorize full training-set input-to-label mapping, even with random labels (i.e. when label has been rendered independent of input), using the same architecture and hyper-parameters as used for training with correct labels, except for a longer time to convergence. 
Extensive experiments support the main argument of the paper. 

Reflexions and observations about finite-sample expressivity and implicit regularization with linear models fit logically within the main theme and are equally thought-provoking.

While this work doesn’t propose much explanations for the good generalization abilities of what it clearly established as overparameterized models,
it does compel the reader to think about the generalization problem from a different angle than how it is traditionally understood.

In my view, raising good questions and pointing to apparent paradox is the initial spark that can lead to fundamental progress in understanding. So even without providing any clear answers, I think this work is a very valuable contribution to research in the field.

Detailed question: in your solving of Eq. 3 for MNIST and CIFAR10, did you use integer y class targets, or a binary one-versus all approach yielding 10 discriminant functions (hence a different alpha vector for each class)?

review_title: null

final_score: rating: 10: Top 5% of accepted papers, seminal paper
review_content: the authors of this work shed light on the generalization properties of deep neural networks. Specifically, the consider various regularization methods (data augmentation, weight decay, and dropout). They also show that quality of the labels, namely label noise also significantly affects the generalization ability of the network.

There are a number of experimental results, most of which are intuitive. Here are some specific questions that were not addressed in the paper:
1. Given two different DNN architectures with the same number of parameters, why do certain architectures generalize better than others? In other words, is it enough to consider only the size (# of parameters) of the network and the size of the input (number of samples and their dimensionality), to be able to reason about the generalization properties of a given network?

2. Does it make sense to study the stability of predictions given added dropout during inference?

Finally, provided a number of experiments and results, the authors do not draw a conclusion or offer a strong insight into what is going on with generalization in DNNs or how to proceed forward. 
review_title: Memorization, overfitting, generalization

final_score: rating: 9: Top 15% of accepted papers, strong accept
review_content: This paper presents a set of experiments where via clever use of randomization and noise addition authors demonstrate the enormous modeling (memorization) power of the deep neural networks with large enough capacity.  Yet these same models have very good generalization behavior even when all obvious explicit or implicit regularizers are removed.  These observations are used to argue that classical theory (VC dimension, Rademacher complexity, uniform stability) is not able to explain the generalization behavior of deep neural networks, necessitating novel theory.

This is a very interesting and thought provoking body of work and I am in complete accord with the observations and conclusions of the paper.  The classical generalization theory is indeed often at a loss with complex enough model families.  As the authors point out, once model families reach a point where they have capacity to memorize train sets, the classical theory does not yield useful results that could give insight into generalization behavior of these models, leaving one to empirical studies and observations.

A minor clarification comment: On page 2, “ … true labels were replaced by random labels.”  Please state that random labels came from the same set as the true labels, to clarify the experiment.
review_title: null

final_score: rating: null
review_content: 1) Your experiments consider classification error, but the loss being optimized is always a different surrogate loss (e.g. conditional log likelihood). Did you look at the effect on the actual training objective being optimized? Can you comment on that?

2) Regarding the result on effective finite sample expressivity. This results relies on the ability to uniquely identify each of your n d-dimensional training data points, after projection, as a single unique real number. Isn’t this feasible primarily because they are « true » real values. Would it not break at some point with finite-limited-precision values?

review_title: What about the optimizaiton objective? And finite-limited-precision?

Meta-review: The authors report the experimental findings of a fascinating inquiry on the ability of the deep neural networks to fit randomly labelled data. The investigation is sound, enlightening, and inspiring. The authors propose both a) a theoretical example showing that a simple shallow network with a number of parameters large enough wrt sample size yields perfect finite-sample expressivity; b) a systematic extensive experimental evaluation to support the findings and claims. The experimental evaluation is a model of thoroughness. 
 
 This is definitely groundbreaking work, which will inspire many works in the coming years.
Decision: Accept (Oral)