Modelling the influence of data structure on learning in neural networks

1 INTRODUCTION AND RELATED WORK . A major impediment for understanding the effectiveness of deep neural networks is our lack of mathematical models for the data sets on which neural networks are trained . This lack of tractable models prevents us from analysing the impact of data sets on the training of neural networks and their ability to generalise from examples , which remains an open problem both in statistical learning theory ( Vapnik , 2013 ; Mohri et al. , 2012 ) , and in analysing the average-case behaviour of algorithms in synthetic data models ( Seung et al. , 1992 ; Engel & Van den Broeck , 2001 ; Zdeborová & Krzakala , 2016 ) . Indeed , most theoretical results on neural networks do not model the structure of the training data , while some works build on a setup where inputs are drawn component-wise i.i.d . from some probability distribution , and labels are either random or given by some random , but fixed function of the inputs . Despite providing valuable insights , these approaches are by construction blind to key structural properties of real-world data sets . Here , we focus on two types of data structure that can both already be illustrated by considering the simple canonical problem of classifying the handwritten digits in the MNIST database using a neural network N ( LeCun & Cortes , 1998 ) . The input patterns are images with 28 × 28 pixels , so a priori we work in the high-dimensional R784 . However , the inputs that may be interpreted as handwritten digits , and hence constitute the “ world ” of our problem , span but a lower-dimensional manifold within R784 which is not easily defined . Its dimension can nevertheless be estimated to be around D ≈ 14 based on the neighbourhoods of inputs in the data set ( Grassberger & Procaccia , 1983 ; Costa & Hero , 2004 ; Levina & Bickel , 2004 ; Facco et al. , 2017 ; Spigler et al. , 2019 ) . The intrinsic dimension being lower than the dimension of the input space is a property expected to be common to many real data sets used in machine leanring . We should not consider presenting N with an input that is outside of its world ( or maybe we should train it to answer that the “ input is outside of my world ” in such cases ) . We will call inputs structured if they are concentrated on a lower-dimensional manifold and thus have a lower-dimensional latent representation . The second type of the structure concerns the function of the inputs that is to be learnt , which we will call the learning task . We will consider two models : the teacher task , where the label is obtained as a function of the high-dimensional input ; and the latent task , where the label is a function of only the lower-dimensional latent representation of the input . We begin this paper by comparing neural networks trained on two different problems : the MNIST task , where one aims to discriminate odd from even digits in the in the MNIST data set ; and the vanilla teacher-student setup , where inputs are drawn as vectors with i.i.d . component from the Gaussian distribution and labels are given by a random , but fixed , neural network acting on the high-dimensional inputs . This model is an example of a teacher task on unstructured inputs . It was introduced by Gardner & Derrida ( 1989 ) and has played a major role in theoretical studies of the generalisation ability of neural networks from an average-case perspective , particularly within the framework of statistical mechanics ( Seung et al. , 1992 ; Watkin et al. , 1993 ; Engel & Van den Broeck , 2001 ; Zdeborová & Krzakala , 2016 ; Advani & Saxe , 2017 ; Aubin et al. , 2018 ; Barbier et al. , 2019 ; Goldt et al. , 2019 ; Yoshida et al. , 2019 ) , and also in recent statistical learning theory works , e.g . ( Ge et al. , 2017 ; Li & Y. , 2017 ; Mei & Montanari , 2019 ; Arora et al. , 2019 ) . We choose the MNIST data set because it is the simplest widely used example of a structured data set on which neural networks show significantly different behaviour than when trained on synthetic data of the vanilla teacher-student setup . Our reasoning then proceeds in two main steps : 1 . We experimentally identify two key differences between networks trained in the vanilla teacherstudent setup and networks trained on the MNIST task ( Sec . 3 ) . i ) Two identical networks trained on the same MNIST task , but starting from different initial conditions , will achieve the same test error on MNIST images , but they learn globally different functions . Their outputs coincide in those regions of input space where MNIST images tend to lie – the “ world ” of the problem , but differ significantly when tested on Gaussian inputs . In contrast , two networks trained on the teacher task learn the same functions globally to within a small error . ii ) In the vanilla teacher-student setup , the test error of a network is stationary during long periods of training before a sudden drop-off . These plateaus are well-known features of this setup ( Saad & Solla , 1995 ; Engel & Van den Broeck , 2001 ) , but are not observed when training on the MNIST task nor on other datasets used commonly in machine learning . 2 . Our main contribution : We introduce the hidden manifold model ( HMF ) , a probabilistic model that generates data sets containing high-dimensional inputs which lie on a lower-dimensional manifold and whose labels depend only on their position within that manifold ( Sec . 4 ) . In this model , inputs are thus structured and labels depend on their lower-dimensional latent representation . We experimentally demonstrate that training networks on data sets drawn from this model reproduces both behaviours observed when training on MNIST . We also show that the structure of both , input space and the task to be learnt , play an important role for the dynamics and the performance of neural networks . Other related work Several works have compared neural networks trained from different initial conditions on the same task by comparing the different features learnt in vision problems ( Li et al. , 2015 ; Raghu et al. , 2017 ; Morcos et al. , 2018 ) , but these works did not compare the functions learned by the network . On the theory side , several works have appreciated the need to model the inputs , and to go beyond the simple component-wise i.i.d . modelling ( Bruna & Mallat , 2013 ; Patel et al. , 2016 ; Mézard , 2017 ; Gabrié et al. , 2018 ; Mossel , 2018 ; Saxe et al. , 2019 ) . While we will focus on the ability of neural network to generalise from examples , two recent papers studied a network ’ s ability to store inputs with lower-dimensional structure and random labels : Chung et al . ( 2018 ) studied the linear separability of general , finite-dimensional manifolds , while Rotondo et al . ( 2019 ) extended Cover ’ s argument ( Cover , 1965 ) to count the number of learnable dichotomies when inputs are grouped in tuples of k inputs with the same label . Accessibility and reproducibility The full code of our experiments can be accessed via https : //drive.google.com/open ? id=1L0UOtOoRTYSHZtTxMxKIQuZLEuVaoJl_ . We give necessary parameter values to reproduce our figures beneath each plot . For ease of reading , we adopt the notation from the textbook by Goodfellow et al . ( 2016 ) . 2 SETUP . In order to proceed on the question of what is a suitable model for structured data , we consider the setup of a feedforward neural network with one hidden layer with a few hidden units , as described below . We chose this setting because it is the simplest one we found where we were able to identify key differences between training in the vanilla teacher-student setup and training on the MNIST task . So throughout this work , we focus on the dynamics and performance of fully-connected two-layer neural networks with K hidden units and first- and second-layer weightsW ∈ RK×N and v ∈ RK , resp . Given an input x ∈ RN , the output of a network with parameters θ = ( W , v ) is given by φ ( x ; θ ) = K∑ k vkg ( wkx/ √ N ) , ( 1 ) where wk is the kth row of W , and g : R → R is the non-linear activation function of the network . We will focus on sigmoidal networks with g ( x ) = erf ( x/ √ 2 ) , or ReLU networks where g ( x ) = max ( 0 , x ) ( see Appendix E ) . We will train the neural networks on data sets with P input-output pairs ( xi , y∗i ) , i = 1 , . . . , P , where we use the starred y∗i to denote the true label of an input xi . We train networks by minimising the quadratic training error E ( θ ) = 1/2 ∑P i=1 ∆ 2 i with ∆i = [ φ ( xi , θ ) − y∗i ] using stochastic gradient descent ( SGD ) with constant learning rate η , θµ+1 = θµ − η∇θE ( θ ) |θµ , xµ , y∗µ . ( 2 ) Initial weights for both layers of sigmoidal networks were always taken component-wise i.i.d . from the normal distribution with mean 0 and variance 1 . The initial weights of ReLU networks were also taken from the normal distribution , but with variance 10−6 to ensure convergence . The key quantity of interest is the test error or generalisation error of a network , for which we compare its predictions to the labels given in a test set that is composed of P ∗ input-output pairs ( xi , y ∗ i ) , i = 1 , . . . , P ∗ that are not used during training , mseg ( θ ) ≡ 1 2P ∗ P∗∑ i [ φ ( xi , θ ) − y∗i ] 2 . ( 3 ) The test set might be composed of MNIST test images or generated by the same probabilistic model that generated the training data . For binary classification tasks with y∗ = ±1 , this definition is easily amended to give the fractional generalisation error fracg ( θ ) ∝ ∑P∗ i Θ [ −φ ( xi , θ ) y∗i ] , where Θ ( · ) is the Heaviside step function . 2.1 LEARNING FROM REAL DATA OR FROM GENERATIVE MODELS ? . We want to compare the behaviours of two-layer neural networks Eq . ( 1 ) trained either on real data sets or on unstructured tasks . As an example of a real data set , we will use the MNIST image database 1,2 ( 5 ) , the fraction of Gaussian i.i.d . inputs and MNIST test images the networks classify differently after training ( green diamonds and orange crosses , resp. ) . ( Right ) Training independent networks on a teacher task with i.i.d . inputs does not reproduce this behaviour . We plot the results of the same experiment , but for Gaussian i.i.d . inputs with teacher labels y∗i ( Eq . 4 , M = 4 ) . For both plots , g ( x ) = erf ( x/ √ 2 ) , η = 0.2 , P = 76N , N = 784. of handwritten digits ( LeCun & Cortes , 1998 ) and focus on the task of discriminating odd from even digits . Hence the inputs xi will be the MNIST images with labels y∗i = 1 , −1 for odd and even digits , resp . The joint probability distribution of input-output pairs ( xi , y∗i ) for this task is inaccessible , which prevents analytical control over the test error and other quantities of interest . To make theoretical progress , it is therefore promising to study the generalisation ability of neural networks for data arising from a probabilistic generative model . A classic model for data sets is the vanilla teacher-student setup ( Gardner & Derrida , 1989 ) , where unstructured i.i.d . inputs are fed through a random neural network called the teacher . We will take the teacher to have two layers and M hidden nodes . We allow that M 6= K and we will draw the components of the teacher ’ s weights θ∗ = ( v∗ ∈ RM , W ∗ ∈ RM×N ) i.i.d . from the normal distribution with mean zero and unit variance . Drawing the inputs i.i.d . from the standard normal distribution N ( x ; 0 , IN ) , we will take y∗i = φ ( xi , θ ∗ ) ( 4 ) for regression tasks , or y∗i = sgn ( φ ( xi , θ ∗ ) ) for binary classification tasks . This is hence an example of a teacher task . In this setting , the network with K hidden units that is trained using SGD Eq . ( 2 ) is traditionally called the student . Notice that , if K ≥M , there exists a student network that has zero generalisation error , the one with the same architecture and parameters as the teacher .

Summary: The authors consider the general problem of "structure" in datasets--particularly, what are the features of datasets that govern the learning dynamics of neural networks trained to classify that data.  They approach this problem by looking at combinations of [iid gaussian, structure] inputs and [teacher, latent] tasks (for particular choices of "teacher", "latent", and "structure).  Finally, they identify that "structure" in the input space, and a notion of "latent"-ness in the task seem crucial for a synthetic dataset to recapitulate the learning dynamics of a real-world dataset.