Effects of Data Geometry in Early Deep Learning

1 INTRODUCTION . The capacity of Deep Neural Networks ( DNNs ) to approximate arbitrary functions given sufficient training data in the supervised learning setting is well known ( Cybenko , 1989 ; Hornik et al. , 1989 ; Anthony & Bartlett , 1999 ) . Several different theoretical approaches have emerged that study the effectiveness and pitfalls of deep learning . These studies vary in their treatment of neural networks and the aspects they study range from convergence ( Allen-Zhu et al. , 2019 ; Goodfellow & Vinyals , 2015 ) , generalization ( Kawaguchi et al. , 2017 ; Zhang et al. , 2017 ; Jacot et al. , 2018 ; Sagun et al. , 2018 ) , function complexity ( Montúfar et al. , 2014 ; Mhaskar & Poggio , 2016 ) , adversarial attacks ( Szegedy et al. , 2014 ; Goodfellow et al. , 2015 ) to representation capacity ( Arpit et al. , 2017 ) . Some recent theories have also been shown to closely match empirical observations ( Poole et al. , 2016 ; Hanin & Rolnick , 2019b ; Kunin et al. , 2020 ) . One approach for studying DNNs is to examine how the underlying structure , or geometry , of the data interacts with learning dynamics . The manifold hypothesis states that high-dimensional real world data typically lies on a low dimensional manifold ( Tenenbaum , 1997 ; Carlsson et al. , 2007 ; Fefferman et al. , 2013 ) . Studies have shown that DNNs are highly effective in deciphering this underlying structure by learning intermediate latent representations ( Poole et al. , 2016 ) . The ability of DNNs to “ flatten ” complex data manifolds , using composition of seemingly simple piece-wise linear functions , appears to be unique ( Brahma et al. , 2016 ; Hauser & Ray , 2017 ) . DNNs with piecewise linear activations , such as ReLU ( Nair & Hinton , 2010 ) , divide the input space into linear regions , wherein the DNN behaves as a linear function ( Montúfar et al. , 2014 ) . The density of these linear regions serves as a proxy for the DNN ’ s ability to interpolate a complex data landscape and has been the subject of detailed studies ( Montúfar et al. , 2014 ; Telgarsky , 2015 ; Serra et al. , 2018 ; Raghu et al. , 2017 ) . The work by Hanin & Rolnick ( 2019a ) on this topic stands out because they derive bounds on the average number of linear regions , as opposed to worst case bounds , and verify the tightness of these bounds empirically for deep ReLU networks . Hanin & Rolnick ( 2019a ) conjecture that the number of linear regions correlates to the expressive power of randomly initialized DNNs with piecewise linear activations . However , they assume that the data is uniformly sampled from the Euclidean space Rd , for some d. By combining the manifold hypothesis with insights from Hanin & Rolnick ( 2019a ) , we are able to go further in estimating the the number of linear regions and the average distance from linear boundaries . We derive bounds on how the geometry of the data manifold affects the aforementioned quantities . To corroborate our theoretical bounds with empirical results , we design a toy problem where the input data is sampled from two distinct manifolds that can be represented in a closed form . We count the exact number of linear regions on these two manifolds that a randomly initialized neural network splits them into . We also observe the average distance to the boundaries of linear regions . We demonstrate how the number of linear regions varies for two distinct manifolds in our setting . These results show that the number of linear regions on the manifold do not grow exponentially with the dimension of input data . Our experiments do not provide estimates for theoretical constants , as is the case in deep learning theory , but demonstrate that the number of linear regions change as a consequence of these constants . We also study high dimensional data that lies on low dimensional manifolds with unknown structure and how the number of linear regions vary on and off this manifold , which is a more realistic setting . To achieve this we present experiments performed on the manifold of natural images . We sample data from the image manifold using a generative adversarial network ( GAN ) ( Goodfellow et al. , 2014 ) trained on the curated images of paintings . Specifically , we generate images using the pre-trained StyleGAN ( Karras et al. , 2019 ; 2020b ) trained on the curated MetFaces dataset ( Karras et al. , 2020a ) . We also assign random labels to the images in the dataset and train a deep ReLU network in a supervised manner , a scenario in which it would overfit ( Zhang et al. , 2017 ) . We generate curves on the image manifold of faces , using StyleGAN , and show how overfitting is reflected in the density of linear regions of the aforementioned deep ReLU network . 2 PRELIMINARIES AND BACKGROUND . Our goal is to understand how the underlying structure of real world data matters for deep learning . We first provide the mathematical background required to model this underlying structure as the geometry of data . We then provide a summary of previous work on understanding the approximation capacity of deep ReLU networks via the complexity of linear regions . For the details on how our work fits into the theory of DNNs we refer the reader to Appendix C . 2.1 DATA MANIFOLD AND DEFINITIONS . We use the example of the MetFaces dataset ( Karras et al. , 2020a ) to illustrate how data lies on a low dimensional manifold . The images in the dataset are 1028 × 1028 × 3 dimensional . By contrast , the number of realistic dimensions along which they vary are limited , e.g . painting style , artist , size and shape of the nose , jaw and eyes , background , clothing style ; in fact , very few 1028× 1028× 3 dimensional images correspond to realistic faces . We illustrate how this affects the possible variations in the data in Figure 1 . A manifold formalises the notion of limited variations in high dimensional data . One can imagine that there exists an unknown function f : X → Y from a low dimensional space of variations , to a high dimensional space of the actual data points . Such a function f : X → Y , from one open subset X ⊂ Rm , to another open subset Y ⊂ Rk , is a diffeomorphism if f is bijective , and both f and f−1 are differentiable , also referred to as smooth . Therefore , a manifold is defined as follows . Definition 2.1 . Let k , m ∈ N0 . A subset M ⊂ Rk is called a smooth m-dimensional submanifold of Rk ( or m-manifold in Rk ) iff every point x ∈ M has an open neighborhood U ⊂ Rk such that U ∩M is diffeomorphic to an open subset Ω ⊂ Rm . A diffeomorphism ( i.e . differentiable mapping ) , f : U ∩M → Ω is called a coordinate chart of M and the inverse , h : = f−1 : Ω→ U ∩M is called a smooth parametrization of U ∩M . For the MetFaces dataset example , suppose there are 10 dimensions along which the images vary . Further assume that each variation can take a value continuously in some interval of R. Then the smooth parametrization would map f : Ω∩R10 →M ∩R1028×1028×3 . This parametrization and its inverse are unknown in general , and computationally very difficult to estimate in practice . There are similarities in how geometric elements are defined for manifolds and Euclidean spaces . A smooth curve , on a manifold M , γ : I → M is defined from an interval I to the manifold M as a function that is differentiable for all t ∈ I , just as is done for Euclidean spaces . The shortest such curve between two points on a manifold is no longer a straight line , but is instead a geodesic . One recurring geometric element , which is unique to manifolds and stems from the definition of smooth curves , is that of a tangent space , defined as follows . Definition 2.2 . LetM be anm-manifold in Rk and x ∈M be a fixed point . A vector v ∈ Rk is called a tangent vector of M at x if there exists a smooth curve γ : I →M such that γ ( 0 ) = x , γ̇ ( 0 ) = v where γ̇ ( t ) is the derivative of γ at t. The set TxM : = { γ̇ ( 0 ) |γ : R→M is smoothγ ( 0 ) = x } , of tangent vectors of M at x is called the tangent space of M at x . In simpler terms , the plane tangent to the manifold M at point x is called the tangent space and denoted by by TxM . Consider the upper half of a 2-sphere , S2 ⊂ R3 , which is a 2-manifold in R3 . The tangent space at a fixed point x ∈ S2 is the 2D plane perpendicular to the vector x and tangential to the surface of the sphere that contains the point x . We refer to these concepts in the text that follows . For additional background on manifolds we refer the reader to Appendix B . 2.2 LINEAR REGIONS OF DEEP RELU NETWORKS . We consider a neural network , F , which is a composition of activation functions . Inputs at each layer are multiplied by a matrix , referred to as the weight matrix , with an additional bias vector that is added to this product . We limit our study to ReLU activation function ( Nair & Hinton , 2010 ) , which is piece-wise linear and one of the most popular activation functions being applied to various learning tasks on different types of data like text , images , signals etc . We further consider DNNs that map inputs , of dimension nin , to scalar values , i.e . values in R. Therefore , F : Rnin → R is defined as , F ( x ) = WLσ ( BL−1 +WL−1σ ( ... σ ( B1 +W1x ) ) ) , ( 1 ) where Wl ∈Mnl×nl−1 is the weight matrix for the lth hidden layer , nl is the number of neurons in the lth hidden layer , Bl ∈ Rnl is the vector of biases for the lth hidden layer , n0 = nin and σ : R→ R is the activation function . For a neuron z in the lth layer we denote the pre-activation of this neuron , for given input x ∈ Rnin , as zl ( x ) . For a neuron z in the layer l we have z ( x ) = Wl−1 , zσ ( ... σ ( B1 +W1 , zx ) ) , for l > 1 ( for the base case l = 1 we have z ( x ) = W1 , zx ) where Wl−1 , z is the row of weights , in the weight matrix of the lth layer , Wl , corresponding to the neuron z . We use Wz to denote the weight vector for brevity , omitting the layer index l in the subscript . We also use bz to denote the bias term for the neuron z. Neural networks with piecewise linear activations are piecewise linear on the input space ( Montúfar et al. , 2014 ) . Suppose for some fixed y ∈ Rnin as x→ y if we have z ( x ) → −bz then we observe a discontinuity in the gradient ∇xσ ( bz +Wzz ( x ) ) at y . Intuitively , this is because x is approaching the boundary of the linear region of the function defined by the output of z . Therefore , the boundary of linear regions , for a feed forward neural network F , is defined as : BF = { x|∇F ( x ) is not continuous at x } . Hanin & Rolnick ( 2019a ) argue that an important generalization for the approximation capacity of a neural network F is the ( nin − 1 ) −dimensional volume density of linear regions defined as volnin−1 ( BF ∩ K ) /volnin ( K ) , for a bounded set K ⊂ Rnin . This quantity serves as a proxy for density of linear regions and therefore the expressive capacity of DNNs . Intuitively , higher density of linear boundaries means higher capacity of the DNN to approximate complex non-linear functions . The quantity is applied to lower bound the distance between a point x ∈ K and the set BF , which is distance ( x , BF ) = min neurons z |z ( x ) − bz|/||∇z ( x ) || , which measures the sensitivity over neurons at a given input . The above quantity measures how “ far ” the input is from flipping any neuron from inactive to active or vice-versa . Informally , Hanin & Rolnick ( 2019a ) provide two main results for a randomly initialized DNN F , with a reasonable initialisation . Firstly , they show that E [ volnin−1 ( BF ∩K ) volnin ( K ) ] ≈ # { neurons } , meaning the density of linear regions is bound above and below by some constant times the number of neurons . Secondly , for x ∈ [ 0 , 1 ] nin , E [ distance ( x , BF ) ] ≥ C # { neurons } −1 , where C > 0 depends on the distribution of biases and weights , in addition to other factors . Meaning that the distance to the nearest boundary is bounded above and below by a constant times the inverse of the number of neurons . These results stand in contrast to earlier worst case bounds that are exponential in the number of neurons . Hanin & Rolnick ( 2019a ) also verify these results empirically to note that the constants lie in the vicinity of 1 throughout training .

Summary: The authors study average-case representation capacity of random ReLU neural networks, as measured through their linear regions, as in Hanin and Rolnick 2019 -- the difference in the authors' setting is that they consider the input space to be defined on a low-dimensional manifold, whereas Hanin and Rolnick considered inputs on the solid cube. The authors derive analogues of each of the results of Hanin and Rolnick in the manifold setting -- these results include formulas for the "average volume density" of linear regions, a proxy for the exact number of linear regions, and the average distance to the boundary of the linear regions -- where the dimension dependence is on the intrinsic manifold dimension rather than the ambient dimension, and additional constants appear that depend on properties like curvatures of the manifolds. They provide toy experiments to verify the theory (involving testing linear region properties on a pair of 1D manifolds with different curvatures), as well as an experiment on 'manifolds of faces' generated by the StyleGAN model -- in both cases, the linear regions are studied throughout training rather than just at initialization (as the theory pertains to).