Venue: ICLR 2017 workshop

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper tries to argue the incorrectness of the existing claim by Zhang that deep neural network learns to memorizing training data. 

Firstly, as pointed out by the first reviewer, the term memorization in the two papers are different. The claims are not completely contradict. For example, in Zhang's paper, they claim that "neural networks are able to capture the remaining signal in the data, while at the same time fit the noisy part using brute-force". In this paper, the authors claims "we believe that DNNs first learn and then refine simple patterns, which are shared across examples, in order to quickly drive down training loss, and only incorporate more case-by-case memorization as a later resort". Basically, the learning of DNNs are capable of identifying patterns in structured data, but with random noise where no shared patterns could be extracted across target classes, sample dependent hypotheses are learnt given enough capacity. Hence, the two papers are claim the same DNN learning behavior.

On the experimental setup, the datasets used for both papers have only a small amount of target classes. Even when randomizing the labels, it's still labeled as one of the 10 classes. DNNs learn to fit to the training data, which guides DNNs to find shared patterns within samples of the same target class. A naive thought if there are the same number of classes as the number of samples in the training data, how does DNNs learn. If DNNs still learn to model the boundaries, there will still be some generalization capability. Otherwise, if DNNs learn by memorization, it should no generalize at all. Because the target labels are not pushing the model to extract shared patterns. Just random thought not sure useful or not.

Regarding the claim "DNNs trained on real data examples learn simpler functions than when trained with noise data, as measured by the sharpness of the loss function at convergence". Personally, I believe that relates more to the data, as with noise data, true function DNNs try to learn are very different from the clean data. Hence, it's not all about how DNNs learn. The difference in the target function DNNs have to learn also affects. 

The experiments in this papers do shown different perspectives from what Zhang reported. It would be of interest of many readers, but it would be a more stronger case if clearer definitions of terms such as memorization are defined and more experiments to support the claims. 

review_title: Interesting observations but not convincing enough 

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper reads as a rebuttal to the recent work by Zhang. In this paper, it was shown that neural nets are in principle able to shatter the input data in the conducted experiments, which means that any labelling can be represented by the network:

"The experiments we conducted emphasize that the effective capacity of several successful neural network architectures is large enough to shatter the training data. Consequently, these models are in principle rich enough to memorize the training data."

This paper, however defines memorization as a failure to generalize.

Hence, both papers are based on different definitions of what memorization means. I felt it would have been much better if the authors of this paper had followed along the definition of Zhang et al. My fear is that this confusion is actually hurting, as it suggest a conflict where – in my opinion – none exists.


Nevertheless, the paper has some interesting experiments.
review_title: No consensus on what memorization means

Decision: Accept