Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The authors consider the general problem of "structure" in datasets--particularly, what are the features of datasets that govern the learning dynamics of neural networks trained to classify that data.  They approach this problem by looking at combinations of [iid gaussian, structure] inputs and [teacher, latent] tasks (for particular choices of "teacher", "latent", and "structure).  Finally, they identify that "structure" in the input space, and a notion of "latent"-ness in the task seem crucial for a synthetic dataset to recapitulate the learning dynamics of a real-world dataset.

The experiments, exposition, and motivation are all exceedingly clear.  My only reservations are about the scope of the experimentation / strength of conclusions of the paper for generally structure data.

Thus, I suggest a Weak Reject of this paper (though I would likely increase my rating to Weak Accept given my comments below).

The primary weakness of this paper is the over-reliance on the MNIST dataset, which is very nearly linearly separable.  Thus, I strongly worry that any notions of latentness that work for MNIST might not transfer at all to more complicated data regimes---i.e., while I believe the authors have identified and patched an interesting gap between the learning dynamics of iid data and MNIST, I'm not sure if there still isn't a gap between something like MNIST and, say, CIFAR-10.  I would raise my score to an accept if the authors carried out their analysis on (at least) CIFAR-10 as well, and even higher if the authors greatly expanded their experiments.
review_title: Official Blind Review #1

final_score: rating: 1: Reject
review_content: This paper studies influences of data structures on neural network learning. The data structures discussed in this paper are structured inputs (concentrating on a low-dimensional manifold) versus unstructured ones, as well as the teacher task (labels are obtained as a function of high-dimensional inputs) versus the latent task (labels are obtained as a function of the lower-dimensional latent representation of inputs). The introduced model, the hidden manifold model, which is a latent task with structured inputs, is claimed to reproduce two features found in learning of the MNIST data set, whereas the teacher task with unstructured inputs does not.

The observation that typical real-world datasets are concentrated on a lower-dimensional manifold is not novel, and it is also well expected that networks trained with such a dataset would exhibit different behaviors for inputs outside such a lower-dimensional manifold. The other observation that in real-world learning tasks one rarely encounters plateaux is not novel either. The possible novelty of this paper would thus be in the proposal of the hidden manifold model, but I am not convinced with the significance of the latent task. Because of these, I would judge possible contributions of this paper rather weak, so that I would not be able to recommend acceptance of this paper.

I think that the main difference between the authors’ “teacher task” and “latent task” lies in realizability of the underlying function: The teacher task is certainly realizable once the number of hidden units exceeds that of the teacher, whereas we are not sure about the realizability of the latent tasks. There might even be different levels of unrealizability which can affect learning. Anyway, the teacher versus latent distinction of learning tasks, as introduced in this paper, should be best regarded, at least in its current status, as a working hypothesis which would need more investigation. I would agree that this paper puts a step forward, but does not arrive at any decisive conclusion yet.

Page 3, line 26: The assumption that g should act componentwise does not seem needed because in equation (1) it acts on a scalar.
Page 4, line 16: there exist(s) a student network
Page 6, line 20: gradient descent methods such (as) natural gradient descent
Page 7, line 7: I do not understand what is meant by “by dividing all entries by the covariance of the entire matrix”. An entry should be a real number, whereas the covariance should be a matrix.
Page 7, line 12: cf. (left -> right) of Fig. 1
 
review_title: Official Blind Review #2

final_score: rating: 1: Reject
review_content: The paper studies how different settings of data structure affect learning of neural networks and how to mimic the behavior of neural networks seen on real datasets (e.g. MNIST) when learning on a synthetic one.
I would recommend rejecting the paper due to several issues pointed out below.
1. The paper abuses blanket citation making it difficult to identify and verify contribution and conduct a comparison with existing literature. Related work section amounts only to one paragraph in size. It looks questionable, that nobody ever treated the problem of the generalization ability of neural networks from a point of the data manifold properties. After a quick search, for example, [1] provides an in-depth analysis and generalization bounds for two-layered neural networks and provides a data complexity measure that can discriminate between random labels (which are equivalent to the outputs of randomly initialized fixed teacher networks in this work) and true labels on structured datasets like MNIST and CIFAR. The paper fails to cite this work as well. 
2. The paper claims to experimentally identify key differences in the training dynamics of neural networks in teacher-student setup and on an MNIST task (binary classification into even and odd numbers).  One of the differences is the presence of plateaus in the learning curves in the vanilla teacher-student setup, however, as the paper states itself - this is a well established and studied characteristic of the setup, not something unexpected and new.
3. Overall, I have yet to see actionable development in this paper as it consists of observations that have been noticed and studied previously and presents no attempt at explanation or rigorous analysis.

As for the experiments:
The setting of the experiment in section 3.1 leaves space for improvement. For example, it would be interesting to see whether two neural networks learned in the vanilla teacher-student setup on the iid random inputs agree on the MNIST inputs (i.e. inverting the experiment in 3.1) as a sanity check for other factors interplay since MNIST inputs would be an example of the out of distribution inputs for the networks learned on iid random examples used in the experiment. As another pointer, [2] shows that even when training input is from a standard normal distribution, the problem can have spurious local minima, implying that even on unstructured training datasets, neural networks from different initializations yield diverse outputs for out of distribution inputs, not agreeing among each other.

I would also like to see concrete examples when and how the hidden manifold model may benefit theoretical understanding or practical knowledge on, for example, how to cook a dataset or check if the dataset admits/affects learning.

References
[1] Arora, Sanjeev, et al. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks.
[2] Safran, Itay, and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks.
review_title: Official Blind Review #3

Meta-review: The paper examines the idea that real world data is highly structured / lies on a low-dimensional manifold. The authors show differences in neural network dynamics when trained on structured (MNIST) vs. unstructured datasets (random), and show that "structure" can be captured by their new "hidden manifold" generative model that explicitly considers some low-dimensional manifold.

The reviewers perceived a lack of actionable insights following the paper, since in general these ideas are known, and for MNIST to be a limited dataset, despite finding the paper generally clear and correct.

Following the discussion, I must recommend rejection at this time, but highly encourage the authors to take the insights developed in the paper a bit further and submit to another venue. E.g. trying to improve our algorithms by considering the inductive bias of structure of the hidden manifold, or developing a systematic and quantifiable notion of structure for many different datasets that correlate with difficulty of training would both be great contributions.
Decision: Reject