Modelling the influence of data structure on learning in neural networks
The lack of crisp mathematical models that capture the structure of real-worlddata sets is a major obstacle to the detailed theoretical understanding of deepneural networks.Here, we first demonstrate the effect of structured data setsby experimentally comparing the dynamics and the performance of two-layernetworks trained on two different data sets: an unstructured synthetic dataset containing random i.i.d. inputs, and a simple canonical data set suchas MNIST images.Our analysis reveals two phenomena related to the dynamics ofthe networks and their ability to generalise that only appear when training onstructured data sets.Second, we introduce a generative model for data sets,where high-dimensional inputs lie on a lower-dimensional manifold and havelabels that depend only on their position within this manifold.We call it the*hidden manifold model* and we experimentally demonstrate that trainingnetworks on data sets drawn from this model reproduces both the phenomena seenduring training on MNIST.
We demonstrate how structure in data sets impacts neural networks and introduce a generative model for synthetic data sets that reproduces this impact.The paper studies how different settings of data structure affect learning of neural networks and how to mimic behavior on real datasets when learning on a synthetic one.