Modelling the influence of data structure on learning in neural networks
Authors: S. Goldt
M. Mézard
F. Krzakala
L. Zdeborová
Venue:  (2020)
Keywords: Neural Networks
Generative models
Synthetic data sets
Generalisation
Stochastic Gradient descent

Abstract: The lack of crisp mathematical models that capture the structure of real-world
data sets is a major obstacle to the detailed theoretical understanding of deep
neural networks. Here, we first demonstrate the effect of structured data sets
by experimentally comparing the dynamics and the performance of two-layer
networks trained on two different data sets: (i) an unstructured synthetic data
set containing random i.i.d. inputs, and (ii) a simple canonical data set such
as MNIST images. Our analysis reveals two phenomena related to the dynamics of
the networks and their ability to generalise that only appear when training on
structured data sets. Second, we introduce a generative model for data sets,
where high-dimensional inputs lie on a lower-dimensional manifold and have
labels that depend only on their position within this manifold. We call it the
*hidden manifold model* and we experimentally demonstrate that training
networks on data sets drawn from this model reproduces both the phenomena seen
during training on MNIST.

Under review as a conference paper at ICLR 2020
MODELLING THE INFLUENCE OF DATA STRUCTURE ON LEARNING IN NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
The lack of crisp mathematical models that capture the structure of real-world data sets is a major obstacle to the detailed theoretical understanding of deep neural networks. Here, we first demonstrate the effect of structured data sets by experimentally comparing the dynamics and the performance of two-layer networks trained on two different data sets: (i) an unstructured synthetic data set containing random i.i.d. inputs, and (ii) a simple canonical data set containing MNIST images. Our analysis reveals two phenomena related to the dynamics of the networks and their ability to generalise that only appear when training on structured data sets. Second, we introduce a generative model for data sets, where ligh-dimensional inputs lie a lower-dimensional manifold and have labels that depend only on their position within this manifold. We call it the hidden manifold model and we experimentally demonstrate that training networks on data sets drawn from this model reproduces both the phenomena seen during training on MNIST.
INTRODUCTION AND RELATED WORK
major impediment for understanding the effectiveness of deep neural networks our lack of mathematical models for the data sets on which neural networks are trained. This lack of tractable models prevents us from analysing the impact of data sets on the training of neural networks and their ability to generalise from examples, which remains an open problem both in statistical learning theory (Vapnik] 2013) Mohri et al. (2012), and in analysing the average-case behaviour of algorithms in synthetic data models (Seung et al. 1992 Engel & Van den Broeck 2001 Zdeborová & Krzakala 2016).
Indeed, most theoretical results on neural networks do not model the structure of the training data, while some works build on a setup where inputs are drawn component-wise i.i.d. from some probability distribution, and labels are either random or given by some random, but fixed function of the imputs. Despite providing valuable insights, these approaches are by construction blind to key structural properties of real-world data sets.
Here, we focus on two types of data structure that can both already be illustrated by considering the simple canonical problem of classifying the handwritten digits in the MNIST database using a neural network N (LeCun & Cortes) 1998). The input patterns are images with 28 x 28 pixels, so a priori we work the high-dimensional 784, However, the inputs that may be interpreted as handwritten digits, and hence constitute the "world" of our problem, span but a lower-dimensional manifold within 784 which is not easily defined. Its dimension can nevertheless be estimated to be around D = 14 based on the neighbourhoods of inputs in the data set (Grassberger & Procaccial 1983, Costa & Hero 2004 Levina Bickel 2004; Facco et al.. 2017 Spigler et 2019). The intrinsic dimension being lower than the dimension of the input space a property expected to be common to many real data sets used in machine leanring. We should not consider presenting / with input that is outside of its world (or maybe should train it to answer that the "input is outside of my world" in such cases). We will call inputs structured if they are concentrated on a dower-dimensional manifold and thus have dine--iineniionl latent representation.
The second type of the structure concerns the function of the inputs that is to be learnt, which we will call the learning task. We will consider two models: the teacher task, where the label is obtained as function of the high-dimensional input; and the latent task, where the label is a function of only the lower-dimensional latent representation of the input.

---

Under review as a conference paper at ICLR 2020
structured inputs
inputs that are concentrated on fixed, lower-dimensional man- ifold in input space
latent representation
for structured input, its coordinates in the lower-dimensions manifold
task
the function of the inputs be learnt
latent task
for structured inputs, labels are given function of the latent representation only
teacher task
for all inputs, labels are obtained from a random. but fixed func- tion of the nigh-dimensional input without explicit dependence on the latent representation, if it exists
MNIST task
discriminating odd from even digits in the MNIST database
vanilla teacher-student setup
Generative model due to Gardner & Derrida 1989). where data sets consist of component-wist i.i.d. inputs with labels given by a fixed, but random neural network acting directly on the input
hidden manifold model (HMF)
Generative model introduced in Sec. 4 for data sets consisting of structured inputs (Eq. 6) with latent labels (Eq.
Table 1: Several key concepts used/introduced this paper.
We begin this paper by comparing neural networks trained on two different problems: the MNIST task, where one aims to discriminate odd from even digits in the in the MNIST data set; and the vanilla teacher-student senup, where inputs are drawn as vectors with i.i.d. component from the Gaussian distribution and labels are given by random, but fixed, neural network acting the high-dimensional inputs. This model is an example of a teacher task on unstructured inputs. was introduced by Gardner Derrida (1989) and has played major role in theoretical studies of the generalisation ability of neural networks from an average-case perspective, particularly within the framework of statistical mechanics (Seung et al. 1992, Watkin et al.. 1993. Engel & Van den Broeck & Saxe 2017] Aubin et al. [2018] Barbier et al.) 2019
2001 Zdeborová & Krzakala
Goldt al.) 2019] Yoshida al. (2019), and also in recent statistical learning theory works, e.g. (Ge et al. 2017 Li & Y. 2017 Mei & Montanari, 2019: Arora et al.) (019). We choose the MNIST data set because it is the simplest widely used example of a structured data set on which neural networks show significantly different behaviour than when trained on synthetic data of the vanilla teacher-student setup.
Our reasoning then proceeds in two main steps:
We experimentally identify two key differences between networks trained in the vanilla teacher- student setup and networks trained the MNIST task (Sec. 3). i) Two identical networks trained on the same MNIST task, but starting from different initial conditions, will achieve the same test error on MNIST images, but they learn globally different functions. Their outputs coincide in those regions of input space where MNIST images tend to lie the "world" of the problem, but differ significantly when tested on Gaussian inputs. In contrast, two networks trained on the teacher task learn the same functions globally to within a small error. ii) In the vanilla teacher-student setup, the test error of a network is stationary during long periods of training before sudden drop-off. These plateaus are well- features of this setup (Saad & Solla, 1995] Engel & Van den Broeck [001). but are not observed when training on the MNIST task nor on other datasets used commonly in machine learning.
Our main contribution: We introduce the hidden manifold model (HMF), a probabilistic model that generates data sets containing high-ddmensional inputs which lie on a lower-dimensional manifold and whose labels depend only on their position within that manifold (Sec. f). this model, inputs are thus structured and labels depend on their lower-dimensional latent representation. We experimentally demonstrate that training networks on data sets drawn from this model reproduces both behaviours observed when training MNIST. We also show that the structure of both, input space and the task to be learnt, play an important role for the dynamics and the performance neural networks.
2

---

Under review as a conference paper at ICLR 2020
Other related work Several works have compared neural networks trained from different initial conditions on the same task by comparing the different features learnt in vision problems (Li et al. 2013 Raghu et al., 2017 Morcos et al. 2018). but these works did not compare the functions learned by the network. On the theory side, several works have appreciated the need to model the inputs, and go beyond the simple component- wise i.i.d. modelling (Bruna & Mallat, 2013 Patel al. 2016) Mézard (017] Gabrié et al. 20188) Mossel 2018;; Saxe et al. 2019). While we will focus on the ability of neural network to generalise from examples, two recent papers studied a network's ability to store inputs with lower-dimensional structure and random labels: Chung et al. (2018) studied the linear separability of general, finite-dimensie dimensionl manifolds, Rotondo et al. (2019) extended Cover's argument (Cover, 1965) to count the number of learnable dichotomies when inputs are grouped in tuples of inputs with the same label.
Accessibility and reproducibility The full code of our experiments can be accessed via ht //orive.google, google. com/ open ? wrld==1LLOOOORTTYY XXMXXI XXITXI MMw.IIu2LEEVaaUL._. necessary parameter values to reproduce our figures beneath each plot. For ease of reading, we adopt the notation from the textbook by Goodfellow et al. (2016).
We give
SETUP
In order to proceed on the question of what is a suitable model for structured data, we consider the setup of a feedforward neural network with one hidden layer with a few hidden units, as described below. We chose this setting because it is the simplest one found where were able to identify key differences between training in the vanilla teacher-student setup and training on the MNIST task. So throughout this work, focus on the dynamics and performance of fully-connected two-layer neural networks with K hidden units and first- and second-layer weights W E R and v E R* resp. Given an input RR-V the output of network with parameters 0 (W, v) is given by
K
oxx; 0) Σ DE (uEx/VV))
where wk is the kth row of W, and g RR R is the non-linear activation function of the network. We will focus sigmoidal networks with g(z) arf(=//2), ReLU networks where g(x) = max(0, x) (see Appendix
We will train the neural networks on data sets with P inpput-oupput pairs (x,, y/ ). I,... P. where we use the starred S to denote the true label of an input H,, We train networks by minimising the quadratic training error F(0)) 1/2 Σ A? with A; = [xx;; 0) yē' using stochastic gradient
descent (SGD) with constant learning rate 17,
@u+1 0t vi (0)eer .
Initial weights for both layers of sigmoidal networks were always taken component wise i..d. from the normal distribution with mean O and variance 1. The initial weights of ReLU networks were also taken from the normal distribution, but with variance 10-6 to ensure convergence.
The key quantity of interest is the test error or generalisation error of a network, for which we compare its predictions the labels given in test that is composed of p* input-output pairs (x;, ).), 1, 1,..., p* that are not used during training.
1 Σ 2P
mac 0
10(x;, v.'²
The test set might be composed of MNIST test images or generated by the same probabilistic model that generated the training data. For binary classification tasks with y" = +1, this definition is easily amended to give the fractional generalisation error frae 0 α Σi θ [-<((x; (x,, O)y,), where O(-) is the Heaviside step function.
g
2.1 LEARNING FROM REAL DATA OR FROM GENERATIVE MODELS?
We want to compare the behaviours of two-layer neural networks Eq. ①I trained either on real data sets on unstructured tasks. As example of a real data set, we will use the MNIST image database
3

---

Under review as a conference paper at ICLR 2020
0.5
0.06
0.4
e (L.id. Gaussian)
0.3
0.04
9
free
0.2
efrc (structured)
0.02
fac (i..dd Gaussian)
0.1
0.0
0.00
2
6
8
1
2
3
6
7
8
K
Figure 1: (Left) Networks trained independently on MNIST achieve similar performance, but learn different functions. For two networks trained independently cation task, we show the averaged final fractional test error, Eg the fraction of Gaussian i..dd. inputs and MNIST test images the networks classify differently after training (green diamonds and orange crosses, resp.). (Right) Training independent networks on teacher task with i.i.d. inputs does not reproduce this behaviour. We plot the results of the same experiment, but for Gaussian i.i.d. inputs with teacher labels y (Eq. M 4). For both plots, g(x) = erf (z/ (x/v2) . = 0.2, P = 76N, N = 784.
the MNIST odd-even classifi- frac (blue dots). We also plot frre (5).
of handwritten digits (LeCun Cortes 1998) and focus on the task of discriminating odd from even digits. Hence the inputs I; will be the MNIST images with labels y, 11 -1 for odd and even digits, resp. The joint probability distribution of input-output pairs (æ,, v,) for this task inaccessible, which prevents analytical control over the test error and other quantities of interest. To make theoretical progress, networks for data arising from probabilistic generative model.
therefore promising to study the generalisation ability of neural
classic model for data sets is the vanilla teacher-studen setup (Gardner & Derrida 1989), where unstructured i.i.d. inputs are fed through random neural network called the teacher. We will take the teacher to have two layers and M hidden nodes. We allow that M + and we will draw the components of the teacher's weights l (v" IR W E IR MxN) i..d. from the normal distribution with mean zero and unit variance. Drawing the inputs i.i.d. from the standard normal distribution Nxx; 0, IN), will take
V = o(xx, 0*)
(4)
for regression tasks, or 3/ sgn(o(x;, 0))) for binary classification tasks. This is hence an example of teacher task. In this setting, the network with K hidden units that trained using SGD Eq, 2 is traditionally called the student. Notice that, K M, there exists a student network that has zero generalisation error, the one with the same architecture and parameters as the teacher.
3 TWo CHARACTERISTIC BEHAVIOURS OF NEURAL NETWORKS TRAINED ON STRUCTURED DATA SETS
We now proceed to demonstrate experimentally two significant differences in the dynamics and the performance of neural networks trained on realistic data sets and networks trained within the vanilla muccher-ttuent setup.
3.1 INDEPENDENT NETWORKS ACHIEVE SIMILAR PERFORMANCE, BUT LEARN DIFFERENT FUNCTIONS WHEN TRAINED ON STRUCTURED TASKS
We trained two sigmoidal networks with K hidden units, starting from two independent draws of initial conditions to discriminate odd from even digits in the MNIST database. We trained both networks using SGD with constant learning rate 7,, eq. (2) until the generalisation error had converged stationary value. We plot this asymptotic fractional test error o
frac blue circles the left
4

---

Under review as a conference paper at ICLR 2020
Vanilla teacher/student
Teacher task
10
MNIST task
10
Latent task
10-1
10
10
100
10a
10a
10a
104
10
10
10a
102
10a
104
steps / /
steps / /K/N
Figure (Left) Extended periods with stationary test error during training ("plateaus") appear in the vanilla teacher-student setup, not MNIST. We plot the generalisation error ig
ease (3) of a network trained on Gaussian i.i.d inputs with teacher labels (Eq, 4 M = 4, blue)
and when learning discriminate odd from even digits in MNIST (orange). We trained either the first layer only (dashed) or both layers (solid). Notice the log scale on the x-axes. (Right) Both structured inputs and latent labels are required to remove the plateau for synthetic data. Same experiment, but now the network is trained on structured inputs (Bq. 6) (f(z) sgn(x)), with teacher labels y (Eq. 4 blue) and with latent labels V/ (Eq. 7 orange). In both plots, g(x) erf (a/v2). 76N, = 3, = 0.2.
in Fig. (the averages are taken over both networks and over several realisations of the initial conditions). We observed the same qualitative behaviour when we employed the early-stopping error evaluate the networks, where take the minimum of the generalisation error during training (see Appendix(C).
First, we note that increasing the number of hidden units in the network decreases the test error on this task. We also compared the networks to one another by counting the fraction of inputs which the two networks classify differently,
P
1,2
frac (01, 02) = 2P Σ 0 0 01)0(x, 02)]
(5)
This is a measure of the degree to which both networks have learned the same function Φ(x, 0). Independent networks disagree the classification of MNIST test images at rate that roughly corresponds to their test error for K > 3 (orange crosses). However, even though the additional parameters of bigger networks are helpful the discrimination task (decreasing Eg). both networks learn increasingly different functions when evaluated over the whole of R using Gaussian inputs as the network size K increases (green diamonds). The network learned the right function on the lower-dimensional manifold on which MNIST inputs concentrate, but outside of it.
This behaviour is not reproduced if we substitute the MNIST data set with a data set of the same size drawn from the vanilla teacher-student setup from Sec. 2.1 with M 4. leaving everything else the same (right of Fig. []. The final test error decreases with K, and as soon as the expressive power of the network zero, since the data set is large enough for the network to recover the teacher's weights to within a very small error, leading to a small generalisation error. We also computed the 61,2 using Gaussian i.i.d. inputs (green diamonds). Networks with fewer parameters than the teacher find different approximations to that function, yielding finite values of 61,2. If they have just enough parameters (K" = M), they learn the same function. Remarkably, they also learn the same function when they have significantly more parameters than the teacher. The vanilla teacher-student setup is thus unable reproduce the behaviour observed when training MNIST.
least equal to that of the teacher, i.e. K 2 M, the asymptotic test error goes to
frac evaluated
5

---

Under review as a conference paper at ICLR 2020
3.2 THE GENERALISATION ERROR EXHIBITS PLATEAUS DURING TRAINING ON L.I.D INPUTS
We plot the generalisation dynamics, i.e. the test eтror as a function of training time, for neural networks of the form T in Fig. 2 For a data set drawn from the vanilla teacher-student setup with M = 4, (blue lines in the left-hand plot of Fig. 2 we observe that there is an extended period of training during which the test error Eg remains constant before a sudden drop. These "platcaus" well-known in the literature for both SGD, where they appear a function of time (Bichl & Schwarze 1995 Saad & Solla 1995 Bichl et al. 1996), and in batch learning, where they appear as a function of the training
size (Schwarze 1993 Engel & Van den Broeck 2001). Their appearance is related to different stages of learning: After a brief exponential decay of the test error at the start of training, the network "believes" that data are linearly separable and all her hidden units have roughly the same overlap with all the teacher nodes. Only after longer time, the network picks up the additional structure of the teacher and "specialises", each of its hidden units idcally becomes strongly correlated with one and only one hidden unit of the teacher before the generalisation error decreases exponentially to its final value.
In contrast, the generalisation dynamics of the same network trained on the MNIST task (orange trajectories on the left of Fig. ( shows no plateau. In fact, plateaus are rarely seen during the training of neural networks (note that during training, we do not change any of the aaper-parameter, e.g. the learning rate 17-)
It has been an open question how to eliminate the plateaus from the dynamics of neural networks trained in the teacher-student setup. The use of second-order gradient descent methods such as natural gradient descent (Yang & Amari 1998) can shorten the plateau (Rattray et al. 1998), but we would like to focus on the more practically relevant case of first-order SGD. Yoshida et al. (2019) recently showed that length and existence of the plateau depend on the dimensionality of the output of the network, but we would like a model where the plateau disappears independently of the output dimension.
4 THE HIDDEN MANIFOLD MODEL
We now introduce new generative probabilistic model for structured data sets with the aim of reproducing the behaviour observed during training on MNIST, but with a synthetic data set. The main motivation for such model is that closed-form solution of the learning dynamics is expected to be accessible. To generate a data set containing P inputs in dimensions, we first choose D feature vectors in N dimensions and collect them feature matrix F E PxN Next we draw P vectors C, with random i.i.d. components and collect them in the matrix C e RPxD The vector c gives the coordinates of the ith input on the lower-dimensional manifold spanned by the feature vectors in F. We will call c, the latent representation of the input X,, which is given by the ith row of
x= (CF//DD RRNNN
where f is a non-linear function acting component-wise. In this model, the "world" of the data on which the true label can depend is D-dimensional manifold, which is obtained from the linear subspace of RN generated by the D lines of matrix F, through a folding process induced by the nonlinear function f. As we discuss in Appendix A the exact form of f does not seem to be important, as long as it is a nonlinear function.
The latent labels are obtained by applying a two-layer neural network with weights θ- = (W* e R MMD V E R within the unfolded hidden manifold according to
M
V = o(c,, -=)= = Σ unn wi,, c c//VD).
Eong m
We draw the weights in both layers component-wise i.i.d. from the normal distribution with unity variance, unless we note it otherwise. The key point here is the dependency of labels T on the coordinates of the lower-dimensional manifold C rather than the high- high-dinensional data X. We believe that the exact form of this dependence is not crucial and we expect several other choices to yield similar results to the ones we will present in the next section.

---

Under review as a conference paper at ICLR 2020
0.4
0.06
2,2 (structured)
(Li.d. Gaussian)
0.3
0.04
0.2
E (structured)
<t (i.idd Gaussian)
0.1
0.00
0.0
4
K
Figure A latent task on structured inputs makes independent networks behave like networks trained on MNIST. (Left) For two networks trained independently on binary classification task with structured inputs (6) and latent labels 87 (Eq. 7 M = 1), we plot the final fractional test error, frre (blue dots). also plot 1122 (5). the fraction of Gaussian i.i.d. inputs and structured inputs Cg the networks classify differently after training (green diamonds and orange crosses, resp.). (Right) In the same experiment, structured inputs with teacher labels y
44 (M 4) fail to reproduce the behaviour observed on MNIST (ef. Fig. 1b. both plots, f(x) sgn(r).g(z) = erf (2/v2), 10, 1 0.2.
In the following, we choose the entries of both C and F to be i.i.d. draws from the normal distribution with mean zero and unit variance. To ensure comparability the data sets for different data-generating function f(x), matrix from all components and we rescale inputs by dividing all entries by the covariance all the entries in the matrix before training.
always center the input matrix X by subtracting the mean value of the entire
4.1 THE IMPACT OF THE HIDDEN MANIFOLD MODEL ON NEURAL NETWORKS
We repeated the experiments with two independent networks reported in Sec. using data sets generated from the hidden manifold model with D 10 latent dimensions (see Appendix D). On the right of Fig. 3] we plot the asymptotic performance of network trained on structured inputs which lie on manifold (6) with a teacher task: the labels are function of the high-dimensional inputs and do not explicitly take the latent representation c; of an input into account, y = o(x,, 0*). The final results are similar to those of networks trained on data from the vanilla teacher-student setup (ef. right of Fig. [D: given enough data, the network recovers the teacher function if the network has at least as many parameters as the teacher. Once the teacher weights are recovered by both networks, test error (blue circles) and they agree the classification of random Gaussian inputs because they do implement the same function.
they achieve
The left plot of Fig. shows network performance when trained on the same inputs, but this time with latent task where the labels are function of the latent representation of the inputs: 7 ((c,, 0*). The asymptotic performance of the networks then resembles that of networks trained on MNIST: after convergence, the two networks will disagree on structured inputs at rate that is roughly their generalisation error, but as K increases, they also learn increasingly different functions, up the point where they will agree on their classification random Gaussian input in just half the cases. The hidden manifold model thus reproduces the behaviour of independent networks trained on MNIST.
look at the right-hand plot Fig. reveals that this model the plateaus are absent. Again, we repeat the experiment of Sec. 3.2 but we train networks structured inputs X sgn(CF) with clear from these plots that the plateaus only appear for the teacher task. In Appendix B we demonstrate that the lack of plateaus for latent tasks in Fig. is not due to the fact that the network in the latent task asymptotes at a higher generalisation enror than the teacher task.
teacher (y₂) and latent labels (y ). respectively.
7

---

Under review as a conference paper at ICLR 2020
0.04
# inputs æ;
task
0.03
(1) i..d. Gaussian
teacher task, Eq. (4)
10-1
10"1
10
10
(2) i.i.d. Gaussian
teacher trained on MNIST, Eq. (4)
0.02
>Mapk 10
6g
tree
(i.id. Gaussian)
structured, Eq. (6)
teacher task, Eq. (4)
structured, Eq. (6)
latent task, Eq. (7)
0.00
1
3
5
8
Figure 4: (Left) Same plot as the right plot of Fig. with Gaussian i.i.d. inputs æ, and labels of (4) provided by teacher network with M = hidden units that was pre-trained on the MNIST task, reaching ~ 5% on the task. Inset: Typical generalisation dynamics networks where we train the first or both layers (dashed and solid, resp.). g(z) = erf (a/v2) 77 = 0.2, N = 784, M K = 4, P 76N. (Right) Four different setups for synthetic data sets supervised learning problems.
4.2 LATENT TASKS, STRUCTURED INPUTS ARE BOTH NECESSARY TO MODEL REAL DATA SETS
Our quest to reproduce the behaviour of networks trained on MNIST has led us to consider three different setups so far: the vanilla teacher- sctup, i.e. teacher task on unstructured inputs; and teacher and latent tasks on structured inputs. While it is not strictly possible to test the case of approximate this setup by training network
latent task with unstructured inputs, MNIST task and then using the resulting network as a teacher to generate labels y 4 for inputs drawn i.i.d. component-wise from the standard normal distribution. To test this idea, trained both layers sigmoidal networks with M = hidden units using vanilla SGD on the MNIST task, where they reach generalisation error of about 5%. They have thus clearly learnt some of the structure of the MNIST task. However, as we show on the left of Fig. independent students trained on data set with i.i.d. Gaussian inputs æ; and true labels Df given by the pre-trained teacher network behave similarly to students trained in the vanilla teacher-student setup of Sec. 3.1 Furthermore, the learning dynamics of a network trained this setup display the plateaus that we observed the vanilla teacher-student setup (inset of Fig. 4..
the
the right Fig. we summarise the four different setups for synthetic data sets in supervised learning problems that we have analysed in this paper. Only the hidden manifold model, consisting of a latent task on structured inputs, reproduced the behaviour neural networks trained on the MNIST task, Icading us conclude that model for realistic data sets has to feature both, structured inputs and a latent task.
5 CONCLUDING PERSPECTIVES
We have introduced the hidden manifold model for structured data sets that is simple to write down, yet displays some of the phenomena that observe when training neural networks on real-world inputs. We saw that the model has two key ingredients, both of which are necessary: (1) high- dimensional inputs which lie on lower-dimensional manifold and (2) latent labels for these inputs that depend on the inputs' position within the low dimensional manifold. We hope that this model is a step towards more thorough understanding of how the structure we find in real-world data sets impacts the training dynamics of neural networks and their ability to generalise.
We see two main lines for future work. On the one hand, the present work needs to be generalised to multi-layer networks to identify how depth helps to deal with structured data sets and to build a model capturing the key properties. On the other hand, the key promise of the synthetic hidden manifold model is that the learning dynamics should be amenable to closed-form analysis in some limit. Such analysis and its results would then provide further insights about the properties of learning beyond what is possible with numerical experiments.
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
M.S. Advani and A.M. Saxe. High-dimensional dynamics of generalization error in neural networks. arXiv:1710.03667 2017.
S. Arora, N. Cohen, W. Hu, and Y. Luo. Implicit Regularization in Deep Matrix Factorization. In Advances in Neural Information Processing Systems 33, arXiv: 1905.1355, 2019.
B. Aubin, A. Maillard, J. Barbier, Krzakala, N. Macris, and L.. Zdeborová. The committee machine: Computational to statistical gaps in learning a two-layers neural network. In Advances in Neural Information Processing Systems 31, pp. 3227-3238, 2018.
J. Barbier, F. Kızakala, N. Macris, L. Miolane, and L.. 7deborová. Optimal errors and phase transitions in high-dimenniional generalized linear models. Proceedings of the National Academy of Sciences, 116(12)5451-5466 2019.
M. Biehl and H. Schwarze. Learning by on-line gradient descent. J. Phys. A. Math. Gen., 28(3): 643-656, 1995.
M. Biehl, P. Riegler, and C. Wöhler. Transient dynamics of on-line learning in two-layered neural networks. Journal of Physics A: Mathematicaal and General, 29(16), 1996.
J. Bruna and S. Mallat. Invaraint scattering convolution networks. IEEE Transactions on Pattern Analysis and Machine Intelligence, (35):1872-1886, 2013.
S. Chung, Daniel D. Lee, and H. Sompolinsky. Classification and Geometry of General Perceptual Manifolds. Physical Review X, 8(3):31003, 2018.
J.A. Costa and A.O. Hero. Learning intrinsic dimension and intrinsic entropy of high-dimensional datasets. In 2004 12th European Signal Processing Conference, pp. 369-372, 2004.
T.M. Cover. Geometrical and Statistical Properties Systems of Linear Inequalities with Applications in Pattern Recognition. IEEE Transactions on Electronic Computers, EC-14(3):326-334 1965.
A. Engel and C. Van den Broeck. Statistical Mechanics of Learning. Cambridge University Press, 2001.
Elena Facco, Maria D'Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific Reports, 7(1):1-8, 2017.
M. Gabrié, A. Manoel, C. Luneau, J. Barbier, N. Macris, F. Krzakala, and L. Zdeborová. Entropy and mutual information in models of deep neural networks. In Advances in Neural Information Processing Systems 31, pp. 1836. 2018.
E. Gardner and B. Derrida. Three unfinished works on the optimal storage capacity of networks. Journal of Physics A: Mathematical and General, 1289294993-1199, 1989.
R. Ge, J.D. Lee, and T. Ma. Learning one-hidden-layer neural networks with landscape design. arXiv preprint arXiv:1711.0050/ 2017.
S. Goldt, M.S. Advani, A.M. Saxe, F. Krzakala, and L. Zdeborová. Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setup. to appear. In Advances in Neural Information Processing Systems 33, arXiv:1906.08632. 2019.
I. Goodfellow, Y. Bengio, and A. Courville. Deep learning. MIT Press, 2016.
Grassherger and I. Procaccia. Measuring the strangeness of strange attractors. Physica D: Nonlinear Phenomena, (1-2):189-200 1983.
Y. LeCun and C. Cortes. The MNIST database of handwritten digits, 1998.
E. Levina and P.J. Bickel. Maximum likelihood estimation of intrinsic dimension. In Advances in Neural Information Processing Systems 17, 2004.
Y. Li and Y. Y. Convergence analysis of two-layer neural networks with relu activation. In Advances in Neural Information Processing Systems, pp. 597-607, 2017.

---

Under review as a conference paper at ICLR 2020
Y. Li, J. Yosinski, J. Clune, H. Lipson, and J. Hopcroft. Convergent Learning: Do different neural networks learn the same representations? In D. Storcheus, A. Rostamizadeh, and S. Kumar (eds.), Proceedings of the 1st International Workshop on Feature Extraction: Modern Questions and Challenges at NIPS 2015, volume 44 of Proceedings of Machine Learning Research, pp. 196-212. PMLR, 2015.
S. Mei and A. Montanari. The generalization enror of random features regression: Precise asymptotics and double descent curve. arXiv preprint arXiv:1908.088 0059988.0535, 2019.
M. Mézard. Mean-field message passing equations in the Hopfield model and its generalizations. Phys. Rev. E, (95):022177, 2017. 2017.
M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of Machine Learning. MIT Press, 2012.
A.S. Morcos, M. Raghu, and S. Bengio. Insights on representational similarity in neural networks with canonical correlation. In Advances in Neural Information Processing Systems 31, pp. 5727-5736, 2018.
E. Mossel. Deep learning and hierarchical generative models. arXiv preprint arXiv: /61209957, 2018.
A.B. Patel, M.T. Nguyen, and R. Baraniuk. A probabilistic framework for deep learning. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 2556-2566. Curran Associates, Inc., 2016.
M. Raghu, J. Gilmer, J. Yosinski, and J. Sohl-Dickstein. SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretability, In Advances in Neural Information Processing Systems 30, pp. 6076-6085. Curran Associates, Inc., 2017.
M. Rattray, D. Saad, and S.-I. Amari. Natural Gradient Descent for On-Line Learning. Physical Review Letters, 81(24):54615444 1998.
P. Rotondo, M. Cosentino Lagomarsino, and M. Gherardi. Counting the learnable functions of structured data, arXiv: 1903. /2021, 2019.
D. Saad and S.A. Solla. Exact Solution for On-Line Learning Multilayer Neural Networks. Phys. Rev. Lett., 74(21):4337-4444 1995.
A.M. Saxe, J.L. McClelland, and S. Ganguli. A mathematical theory of semantic development in deep neural networks. Proceedings of the National Academy of Sciences, 116(33:11537-11555 2019.
H. Schwarze. Learning rule in a multilayer neural network. Journal of Physics A: Mathematical and General, 26(21):5781-5794 1993.
H. S. Seung, H. Sompolinsky, and N. Tishby. Statistical mechanics of learning from examples. Physical Review A, 450)),6056-6091, 1992.
Spigler, M. Geiger, and M. Wyart. Asymptotic learning curves kernel methods: empirical data v.s. Seacher-Student paradigm. 190i::1905.00... 2019.
V. Vapnik. The nature of statistical learning theory. Springer science & business media, 2013.
T.L.H. Watkin, Rau, and M. Biehl. The statistical mechanics of learning a rule. Reviews of Modern Physies, 65(2):499-5566 1993.
H.H. Yang and S.-I. Amari. The Efficiency and the Robustness of Natural Gradient Descent Learning Rule. In M Jordan, MJ Kearns, and A Solla (eds.), Advances in Neural Information Processing Systems 10, pp. 385-391, 1998.
Y. Yoshida, R. Karakida, M. Okada, and S.-I. Amari. Statistical mechanical analysis of learning dynamics of two-layer perceptron with multiple output units. Journal of Physics A: Mathematical and Theoretical, 52(18), 2019.
L. Zdeborová and F. Krzakala. Statistical physics of inference: thresholds and algorithms. Adv. Phys., 65(5):453555, 2016.
10

---

Under review as a conference paper at ICLR 2020
A THE EXACT FORM OF THE DATA-GENERATING FUNCTION f() IS NOT IMPORTANT, AS LONG AS IT IS NON-LINEAR
Two questions arise when looking at the way we generate inputs in our data sets, X (C(CC//DD)
the non-linearity necessary f(-) necessary, and is the choice of non-linearity important?
To answer the first question, we plot the results in Sec. 4.1 The setup is exactly the same, except that we now take inputs to be
experiment with independent networks described
X = CF,
(8)
i.e inputs are just linear combination of the feature vectors, without applying non-linearity. In this case, two networks trained in the vanilla teacher-student setup will learn globally different functions, as can be seen from the fractional generalisation error between the networks (5) (green diamonds), which is 1/2, i.c not better than chance. This is direct consequence of using f(x) to perfectly generalise with respect to the teacher, it is thus sufficient to learn only the D components of the teacher weights w in the direction F. Thus the weights of the network in the weight space orthogonal to the directions F are unconstrained, and by starting from random initial conditions, will converge to different values for each network.
We also checked that the qualitative behaviour of neural networks trained on the hidden manifold model does not depend on the data-generating non-linearity f(z). In Fig. 6 we therefore show the results of the same experiment described in Sec, 4.1 but this time using
X max (0, CF).
where the application of the non-linearity is again component-wise. Indeed, the results mirror those when we used the sign function f(x) sgn(x).
0.5
0.5
0.4
dLy (structured)
(structured)
< (i..d.
eas Gaussian)
0.2
0.2
0.1
0.1
Figure 5: The input-generating function must be non-linear. We repeat the plots of Fig. 3 where we plot the fractional test errors of networks trained on labels generated by teacher with M hidden units acting on the inputs (Left) and on the coefficients (Right), only that we take the inputs to be X = CF, i.c we choose linear data-generating function f(z) Notably, even networks trained within the vanilla teacher-student setup will disagree on Gaussian inputs. M = 1, = 0.2, D 10, 8% 1.
B THE EXISTENCE OF PLATEAUS DOES NOT DEPEND ON THE ASYMPTOTIC GENERALISATION ERROR
We have demonstrated on the right of Fig. 2 that neural networks trained on data drawn from the hidden manifold model (HMF) introduced here do not show the plateau phenomenon, where the generalisation error stays stationary after an initial exponential decay, before dropping again. Upon closer inspection. HMF asymptotes at higher generalisation error than the student trained the vanilla teacher-student setup. This is not the case, demonstrate in Fig. 7 observe plateau in sigmoidal network
might think that this is due to the fact that the student trained on data from the
11

---

Under review as a conference paper at ICLR 2020
Structured inputs, latent task
0.25
10a
Vanilla teacher/Student
62 (structured)
2 (Lidd Geussian)
10-2
7
100
10a
10a
steps INN
Figure 6: The qualitative behaviour of inde- pendent students trained on the hidden man- ifold model does not depend on our choice of lata-generating non-linearity (xx). Same plot
Figure 7: The plateau in the vanilla teacher- student setup have larger generalisation error than the asymptotic error a latent task structured inputs. Generalisation dy- namics of a sigmoidal network where we train only the first layer (i) structured inputs X max(0, CF) with latent labels j/ 7 (blue, D = 10) and (ii) the vanilla teacher-student setup (Sec. 2 orange). In both cases, M 5, K 6, = 0.2, 76N, um
Fig. 3 with X max(0, CF). M 1,
0.2, 10, m = 1.
trained on data from the HMF even that network asymptotes at a generalisation error that is, within fluctuations, the same as the generalisation error of network of the same sized trained in the vanilla teacher-student setup and which shows a plateau.
C EARLY-STOPPING YIELDS QUALITATIVELY SIMILAR RESULTS
In Fig. 8 we reproduce Fig. where we compare the performance of independent neural networks trained on the MNIST task (Left), or trained on structured inputs with latent task (Center) and teacher task (Right), respectively. This time, we the early-stopping generalisation error zfre rather than the asymptotic value at the end of training. We define zlrac the minimum of < during the whole of training. Clearly, the qualitative result of Sec, 4.1 unchanged: although we use structured inputs (6) in both cases, independent students will learn different functions which agree on those inputs only when they are trained on a latent task (7) /Center), but not when trained on vanilla teacher task (4) (Right). Thus structured inputs and latent tasks sufficient to reproduce the behaviour observed when training on the MNIST task.
D DYNAMICS WITH LARGE NUMBER OF FEATURES D ~ N
It is of independent interest to investigate the behaviour of networks trained on data from the hidden manifold model when the number of feature vectors / is the same order the input dimension N. We call this the regime of extensive D. It is different regime from MNIST, where experimental studies consistently find that inputs lie on a dineennnnii manifold dimension D 14. which
is much smaller than the input dimension N = 784 (Costa & Hero 2004 Spigler al. 2019).
Levina & Bickel 2004
We show the results of our numerical experiments with N 500, D = 250 in Fig. where we reproduce Fig. 3 for the asymptotic (top row) and the early-stopping (bottow row) generalisation enror. The behaviour of networks trained a teacher task with structured inputs (right column) unchanged w.r.t. to the case with D = 10. For the latent task, increasing the number of hidden units however increases the generalisation error. indicating severe over-fitting, which is only partly mitigated by early stopping. The generalisation error on this task is generally much higher than the low-D regime and clearly, increasing the width of the network not the right way learn latent
12

---

Under review as a conference paper at ICLR 2020
0.30-
0.25
0.06
(ntractured)
0.04
[R..6 Gaussinn
0.2
677 (stractored)
0.15
0.02
0.1
G/ Gaussan)
0.00
0.0
0.00
2
Figure 8: Measuring early stopping errors does not affect the phenomenology of latent and teacher tasks. (Left) Performance of independent sigmoidal students on the MNIST task as evaluated by the carly-stopping generalisation error. (Center and Right) We reproduce Fig. of of the main text, but this time we plot the early-stopping generalisation error glte for two networks trained independently on a binary classification task with structured inputs (6) and latent labels i7 (Eq. M 1, Center) and teacher labels y/ (M 4) (Left). In both plots, f(x) sgn(z).g(a)) erf (x/V2), D 10, 17 0.2.
task; instead, it would be intriguing to analyse the performance of deeper networks on this task where finding a good intermediate representation for inputs is key. This an intriguing avenue for future research.
E INDEPENDENT STUDENTS WITH RELU ACTIVATION FUNCTION
We also verified that the behaviour of independent networks observed MNIST with sigmoidal students persists when training networks with ReLU activation function and that the hidden manifold model able reproduce for these networks. We show the results of our numerical experiments in Fig. 10 To that end, we trained both layers of a network o(æ, 0) with g(x) (a, 0) starting from small initial conditions, where we draw the weights component wise i.i.d. from normal distribution with variance 10
We see that the generalisation error of ReLU networks on the MNIST task (Left Fig. 10 decreases with increasing number of hidden units, while the generalisation error on MNIST inputs of the two independent students with respect each other comparable less than the generalisation error of each individual network on the MNIST task.
structured inputs with teacher task (Right of Fig. 10), where labels were generated by a teacher with . = 4 hidden units, the student recovers the teacher such that its generalisation error is less than -3 for > 4, and both independent students learn the same function, as evidenced by their generalisation errors with respect each other. This is the same behaviour that see in Fig. for sigmoidal networks. The finite value of the generalisation error for K M = 4 is due to two out of ten runs taking very long time to converge, longer than simulation lasted for. Finally, that for latent task on structured inputs, the generalisation error of the two networks with respect each other increases beyond the generalisation error on structured inputs each of them, observed on MNIST. Thus we have recovered the phenomenology that we described for sigmoidal networks in ReLU networks, too.
13

---

Under review as a conference paper at ICLR 2020
trac
e (struc.)
f122 (i.i.d))
0.06
f (structured)
e (ii.d. Gaussian)
0.4
0.2
0.02
0.00
00000
0.0
20
40
60
80
100
8
16
24
32
0.05
0.25
T (structured)
0.04
0.20
e (ii.d. Gaussian)
0.15
0.02
0.10
0.01
0.05
0.00
00000
8
16 K
24
32
8
16 K
24
32
Figure 9: Performance of independent networks trained latent task with inputs in many latent directions D = N/2. (Top Left) For two networks trained independently on binary classifi- cation task with structured inputs (6) and latent labels 8/ (Eq, 7 M 1), we plot the final fractional test error, Eg (blue dots). We also plot frec 5). the fraction of Gaussian i.i.d. inputs and structured inputs the networks classify differently after training (green diamonds and orange crosses, resp.).
(Top Right) Same experiment, but with structured inputs and teacher labels yf (4) (M 4). (Botlom
row) Same plots in the top row, but this time for the early-stopping error é afrac (see Sec. C In all
plots, f(x) sgn(æ), (xx) erf (2/v2),4 N 500, D 250, 77 0.2.
Intracunet
0.99
4
0.02
0.1
0.0
0.00
2
Figure 10: Behaviour of independent students with ReLU activation functions. (Left) Asymp- totic generalisation error independent students with ReLU activation function g(x) = max (0,z)) on the MNIST task. (Center and Right) We reproduce Fig. 3f the main text for two networks with ReLU activation trained independently on a binary classification task with structured inputs (6) and latent labels D, (Eq. 7 M 1) (Center) and teacher labels v/ (4) (M 4 Right). In both plots, f(x) = sgn(x), (x)) max(0.x), D 10, = 0.1.
14


Reviews:
[{"review_id": "Skxrfl2ptB", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper studies how different settings of data structure affect learning of neural networks and how to mimic the behavior of neural networks seen on real datasets (e.g. MNIST) when learning on a synthetic one.\nI would recommend rejecting the paper due to several issues pointed out below.\n1. The paper abuses blanket citation making it difficult to identify and verify contribution and conduct a comparison with existing literature. Related work section amounts only to one paragraph in size. It looks questionable, that nobody ever treated the problem of the generalization ability of neural networks from a point of the data manifold properties. After a quick search, for example, [1] provides an in-depth analysis and generalization bounds for two-layered neural networks and provides a data complexity measure that can discriminate between random labels (which are equivalent to the outputs of randomly initialized fixed teacher networks in this work) and true labels on structured datasets like MNIST and CIFAR. The paper fails to cite this work as well. \n2. The paper claims to experimentally identify key differences in the training dynamics of neural networks in teacher-student setup and on an MNIST task (binary classification into even and odd numbers).  One of the differences is the presence of plateaus in the learning curves in the vanilla teacher-student setup, however, as the paper states itself - this is a well established and studied characteristic of the setup, not something unexpected and new.\n3. Overall, I have yet to see actionable development in this paper as it consists of observations that have been noticed and studied previously and presents no attempt at explanation or rigorous analysis.\n\nAs for the experiments:\nThe setting of the experiment in section 3.1 leaves space for improvement. For example, it would be interesting to see whether two neural networks learned in the vanilla teacher-student setup on the iid random inputs agree on the MNIST inputs (i.e. inverting the experiment in 3.1) as a sanity check for other factors interplay since MNIST inputs would be an example of the out of distribution inputs for the networks learned on iid random examples used in the experiment. As another pointer, [2] shows that even when training input is from a standard normal distribution, the problem can have spurious local minima, implying that even on unstructured training datasets, neural networks from different initializations yield diverse outputs for out of distribution inputs, not agreeing among each other.\n\nI would also like to see concrete examples when and how the hidden manifold model may benefit theoretical understanding or practical knowledge on, for example, how to cook a dataset or check if the dataset admits/affects learning.\n\nReferences\n[1] Arora, Sanjeev, et al. Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks.\n[2] Safran, Itay, and Ohad Shamir. Spurious local minima are common in two-layer relu neural networks.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "HkeSXiWb9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The authors consider the general problem of \"structure\" in datasets--particularly, what are the features of datasets that govern the learning dynamics of neural networks trained to classify that data.  They approach this problem by looking at combinations of [iid gaussian, structure] inputs and [teacher, latent] tasks (for particular choices of \"teacher\", \"latent\", and \"structure).  Finally, they identify that \"structure\" in the input space, and a notion of \"latent\"-ness in the task seem crucial for a synthetic dataset to recapitulate the learning dynamics of a real-world dataset.\n\nThe experiments, exposition, and motivation are all exceedingly clear.  My only reservations are about the scope of the experimentation / strength of conclusions of the paper for generally structure data.\n\nThus, I suggest a Weak Reject of this paper (though I would likely increase my rating to Weak Accept given my comments below).\n\nThe primary weakness of this paper is the over-reliance on the MNIST dataset, which is very nearly linearly separable.  Thus, I strongly worry that any notions of latentness that work for MNIST might not transfer at all to more complicated data regimes---i.e., while I believe the authors have identified and patched an interesting gap between the learning dynamics of iid data and MNIST, I'm not sure if there still isn't a gap between something like MNIST and, say, CIFAR-10.  I would raise my score to an accept if the authors carried out their analysis on (at least) CIFAR-10 as well, and even higher if the authors greatly expanded their experiments.", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJePTrbx9H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper studies influences of data structures on neural network learning. The data structures discussed in this paper are structured inputs (concentrating on a low-dimensional manifold) versus unstructured ones, as well as the teacher task (labels are obtained as a function of high-dimensional inputs) versus the latent task (labels are obtained as a function of the lower-dimensional latent representation of inputs). The introduced model, the hidden manifold model, which is a latent task with structured inputs, is claimed to reproduce two features found in learning of the MNIST data set, whereas the teacher task with unstructured inputs does not.\n\nThe observation that typical real-world datasets are concentrated on a lower-dimensional manifold is not novel, and it is also well expected that networks trained with such a dataset would exhibit different behaviors for inputs outside such a lower-dimensional manifold. The other observation that in real-world learning tasks one rarely encounters plateaux is not novel either. The possible novelty of this paper would thus be in the proposal of the hidden manifold model, but I am not convinced with the significance of the latent task. Because of these, I would judge possible contributions of this paper rather weak, so that I would not be able to recommend acceptance of this paper.\n\nI think that the main difference between the authors’ “teacher task” and “latent task” lies in realizability of the underlying function: The teacher task is certainly realizable once the number of hidden units exceeds that of the teacher, whereas we are not sure about the realizability of the latent tasks. There might even be different levels of unrealizability which can affect learning. Anyway, the teacher versus latent distinction of learning tasks, as introduced in this paper, should be best regarded, at least in its current status, as a working hypothesis which would need more investigation. I would agree that this paper puts a step forward, but does not arrive at any decisive conclusion yet.\n\nPage 3, line 26: The assumption that g should act componentwise does not seem needed because in equation (1) it acts on a scalar.\nPage 4, line 16: there exist(s) a student network\nPage 6, line 20: gradient descent methods such (as) natural gradient descent\nPage 7, line 7: I do not understand what is meant by “by dividing all entries by the covariance of the entire matrix”. An entry should be a real number, whereas the covariance should be a matrix.\nPage 7, line 12: cf. (left -> right) of Fig. 1\n ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper examines the idea that real world data is highly structured / lies on a low-dimensional manifold. The authors show differences in neural network dynamics when trained on structured (MNIST) vs. unstructured datasets (random), and show that "structure" can be captured by their new "hidden manifold" generative model that explicitly considers some low-dimensional manifold.

The reviewers perceived a lack of actionable insights following the paper, since in general these ideas are known, and for MNIST to be a limited dataset, despite finding the paper generally clear and correct.

Following the discussion, I must recommend rejection at this time, but highly encourage the authors to take the insights developed in the paper a bit further and submit to another venue. E.g. trying to improve our algorithms by considering the inductive bias of structure of the hidden manifold, or developing a systematic and quantifiable notion of structure for many different datasets that correlate with difficulty of training would both be great contributions.
Decision: Reject