Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 8: Accept
review_content: The paper studies influence of different hyperparameters of neural networks: architecture, width, depth, initialization, optimizer, etc. on the generalization/memorization trade-off.

To do this, paper propose a clever trick: train a model to mimic identity function, so that output should be exactly the same, as input. This allows rich visualization and easy evaluation via correlation or MSE. Moreover, authors showed that it is possible to do the study on _single_ training example. Yet, in my opinion, the idea of using identity as training objective is even bigger contribution that the study itself.

The number of experiments is really terrific and lots of interesting observation is been made.  E.g, showing that over-parameterization by increasing width does not lead to overfitting, but the opposite is true. In the same time, it is hard for deeper networks to learn identity function regardless the width used.

I definitely vote this to be the oral presentation and even for the best paper award. 

Questions: 
    - Fig 14 (and similar): how it is possible, that for high train-test correlation, similarity of network outputs is high BOTH for constant prediction and identity? Or, might be I am just reading such visualization wrong. In that case, could you please make it more clear?
    - It would be interesting to see, how resnets behave in such setup. 
    - It is surprising that He init was inferior to Xavier. Could you also try orthonormal init (Saxe et.al, ICLR 2014 https://arxiv.org/abs/1312.6120) and LSUV init (Mishkin and Matas, ICLR 2016 https://arxiv.org/abs/1511.06422)?


****
I would like to thank authors for the updated version. I agree with R2 that paper is relying too much on the appendix, but hope that authors could fix this somehow. 
Now I am even more convinced that paper should be accepted and is award-quality.
    
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper studies the inductive bias of neural nets by considering the toy example of learning an identity map through a single data point (and hence the NNs are always overparametrized). The authors compare CNNs versus FCNs, and find that CNNs tend to “generalize” in terms of actually learning the concept of an identity, whereas FCNs are prone to memorization. The authors also present results under various different settings such as changing the filter size or the number of hidden channels of CNNs. The conclusion is that the simpler the network architecture is, the better it generalizes. Another observation is that deep CNNs exhibit extreme memorization.

Overall, this is a well-written paper with an interesting set of experiments. However, I do have several concerns regarding the generality of the observed phenomenon in this paper. The first one is that the authors have chosen the comparison between CNNs and FCNs. In this case, I think a more fair comparison is to restrict the number of links (number of nonzero entries in the weight matrices) for both to be the same. In the current setup, however, the authors seem to consider the same number of hidden neurons, which naturally grants advantages to CNNs as they are of lower complexity.

Second, the training data is always a simple image from MNIST, and I am unsure how much this setting can generalize to other tasks, such as different data (say, music or text) or more complicated images (how do these experiments compare when we use a single data from ImageNet or CIFAR to train?). For instance, since CNNs are designed to capture invariances in natural images, it is unsurprising that they can generalize better on image data, but it would be quite astonishing if the same still holds true for acoustic data. In that case, the conclusion of this paper can be strengthened from “CNNs generalize better on image data” to “CNN generalize better”. Given the scope of the current paper, however, the best we can conclude is that “CNNs generalize better on MNIST”.

Last, again regarding the fair comparison, when comparing deep CNNs versus shallow ones, it is also of interest to see that, when restricted to the same number of parameters, if the deep CNNs still exhibit worse generalization. Otherwise, if some network complexities keep growing, we cannot really tell whether it is the network architecture that induces the inductive bias or it is simply the effect of complexities.

I also would like to suggest a future direction based on the idea in this paper: Comparing the inductive bias of GD versus SGD, a subject of intense study in the current literature. Since the authors considered a single training data, the results in this paper are always for GD. Now, say let us use 5 data, and compare the training of CNNs with different batch-size. Do the results differ? I think such a thought experiment would shed some light on the mysterious behaviors of the first-order algorithms that are widely used in practice.

Finally, a question: another submission to ICLR2020 [1] seems to suggest that optimization methods do not play a role in generalization, which is the opposite observation of this paper. Do the authors have any insight towards this contradiction?

[1] Fantastic Generalization Measures and Where to Find Them https://openreview.net/forum?id=SJgIPJBFvH
review_title: Official Blind Review #1

final_score: rating: 6: Weak Accept
review_content: This paper studies the inductive bias in deep neural networks. The authors train several FF image recognition networks and many CNN variants on a single image, and observe that most networks fall into one of two categories: either memorizing the output of the single training sample, or learning the identity function and generalizing to new, unseen images. The paper is clearly written, present a broad set of experiments, and provides interesting insights, that are somewhat surprising.

As someone from outside the area, my main concern with the paper is that I somehow missed the bigger picture. The authors present multiple different pieces of evidence that demonstrate the different conditions on which different model variants learn the different functions (memorization and generalization), but do not provide high level intuitions about what can be done with this information, and what are potential takeaways for the community. I would have expected to see some discussion after section 3, rather than jumping straight to the conclusions.

Aside from that, I think the paper puts too much of its content in the appendix (about 3 times as much content as in the main paper). 

Minor:
-- section 3.2: "... if *the a* layer ..."
review_title: Official Blind Review #2

Meta-review: The paper studies the effect of various hyperparameters of neural networks including architecture, width, depth, initialization, optimizer, etc. on the generalization and memorization. The paper carries out a rather through empirical study of these phenomena. The authors also rain a model to mimic identity function which allows rich visualization and easy evaluation.  The reviewers were mostly positive but expressed concern about the general picture. One reviewer also has concerns about "generality of the observed phenomenon in this paper". The authors had a thorough response which addressed many of these concerns. My view of the paper is positive. I think the authors do a great job of carrying out careful experiments. As a result I think this is a good addition to ICLR and recommend acceptance.
Decision: Accept (Poster)