S-System, Geometry, Learning, and Optimization: A Theory of Neural Networks

1 INTRODUCTION . The recent development in the algorithm family of neural networks ( NN ) ( LeCun et al . ( 2015 ) ) that aim to solve high dimensional perception problems , has led to results that sometimes outperform humans in particular datasets , e.g. , vision ( He et al . ( 2015 ) ) . It is a computational imitation of biological NNs ( Rosenblatt ( 1958 ) Fukushima ( 1980 ) Ruineihart et al . ( 1986 ) ) . From a theoretical view , however , it has arguably progressed for six decades as a blackbox function approximator . Researchers of various background have been intrigued by theoretical understanding of NNs . From the perspective of physics , we have Mehta & Schwab ( 2014 ) Lin & Tegmark ( 2017 ) Choromanska et al . ( 2015a ) ; from that of applied mathematics , we have Mallat ( 2016 ) ; from that of information theory , we have Amari ( 1995 ) Shwartz-Ziv & Tishby ( 2017 ) ; from that of theoretical computer science , we have Arora et al . ( 2014 ) ; and from the machine learning perspective , we have Anselmi et al . ( 2015 ) Anselmi et al . ( 2016 ) Jeffrey Pennington ( 2017 ) Ankit B. Patel et al . ( 2016 ) etc . This work is also motivated for a better theoretical understanding of NNs . Our investigation in this work has led to a measure-theoretical theory of NNs . To the best of our knowledge , we do not find works that are intimately close to ours , and a very detailed discussion on related works is written in appendix H. Our main contributions are summarized as follows . • Built on the formalism of probability coupling theory , we derive an algorithm framework , named Hierarchical Measure Group and Approximate System ( HMGAS ) , nicknamed S-System , that is designed to learn the complex hierarchical , statistical dependency in the physical world , of which the hierarchical structure is formulated as measure-theoretical assumptions . • We show that NNs are special cases of S-System when the probability kernels assume certain exponential family distributions . Activation Functions are derived formally . We further endow geometry on NNs through information geometry , and quantitatively show NNs implement Renormalization Group ( RG ) , of which the large scale property to be renormalized is dual Bregman divergence , or informally semantic difference , and completes the analog between NNs and RG . • S-System shows NNs are inherently stochastic , and under a set of realistic boundedness and diversity conditions , it enables us to prove that for large size nonlinear deep NNs with a class of losses , including the hinge loss , all local minima are global minima with zero loss errors , and regions around the minima are flat basins where all eigenvalues of Hessians are concentrated around zero . This part is almost a complete work on its own right , and most of the contents have to be put in the appendices . This unusual fact is because we feels like we need to solve a known hard problem to convince ourselves and the community the plausibility of S-System proposed . We summarize insights of our theoretical results as follows . • Inherent stochasticity of NNs and renormalization/coarse-graining on semantic difference/geometry on stochastic manifolds of NNs ( summary of section 4 ) . Activations in the intermediate layers are function of estimated realization of random variables , of which the true probability measure is transported from the input data . Activation functions used in practice actually assume the exponential family distributions , which explains why NNs learn templates , since the mean of an exponential family distribution uniquely determines its distribution . Forward propagation is maximization of expected data likelihood . A stochastic manifold structure is endowed on the intermediate space of NNs , where the “ distance ” is defined to characterize the semantic difference between events/samples . As the layers go deeper , the semantic difference potentially becomes gradually coarse-grained to reflect the higher level semantic difference , e.g. , dogs vs cats , while ignoring lower level variations , e.g. , textures ( theorem 4.1 ) . This is true due to the information monotony phenomenon : by blocking half of the information from propagating ( using ReLU as an example ) , information related to irrelevant variations could potentially be discarded . The semantic difference is the physical quantity to renormalize to produce large scale properties , which is missing in the incomplete analog between NNs and RG ( Mehta & Schwab ( 2014 ) Lin & Tegmark ( 2017 ) ) . Lastly , the event spaces of samples are the objects to study if one wants to study the geometry of NNs , e.g. , symmetry in the geometry . For example , robustness to deformation in images can be characterized as a close “ distance ” between two event collections where one is obtained by deforming all events in the other . An example is provided in example 4.1 . • Optimization in NNs ( summary of section 5 and appendix F ) . The stochasticity identified enables us to analyze NNs stochastically in its full complexity . It enables us to characterize the optimization behaviors of NNs in theorem 5.1 described in the contributions above . It explains the optimization myths of NNs that though being non-convex NNs can optimize the loss to zero , and why learning progresses slowly when approaching the minima . Informally , a huge number of cooperative yet diverse neurons can group samples into arbitrary groups corresponding to labels . The assumptions made in the theorem are sufficient practice-guiding preconditions instead of unrealistic assumptions made to make the proof work . It explains why centering of neuron activation ( Glorot & Bengio ( 2010 ) ) is helpful , for it helps to let the eigenvalues of the Hessian of the risk function be symmetric w.r.t . y-axis ( theorem F.2 ) , thus guaranteeing the existence of negative eigenvalues to provide loss-minimizing directions ; why normalization of neuron activation ( Ioffe , Sergey and Szegedy ( 2015 ) ) is helpful , for the boundedness and diversity conditions 5.1 5.2 ask the correlation between neurons , formulated as cumulants , to be small , and normalization of standard deviation possibly maintains the conditions throughout the training ( appendix F.3 ) ; why the larger the network is , the easier for it to reach zero error , for our results on optimization is a probably-approximately-correct type result where the error is controlled by the size of the network — the underlying reason is complicated , and we refer readers to appendix F.2 . • Functional purpose of NNs ( summary of section 3 ) . This is the most important one , but also perhaps the hardest one to get . S-System shows NNs work by grouping samples/events into groups — which emerges through optimizing an objective function — and estimate/approximate their true probability measure through empirical observations . The group is a formalization of semantics , and explains how discrete labels emerge from continuous samples , i.e. , a label identifies a group of events/samples . A proper implementation of the above process applied hierarchically creates an adaptive complex system that consists of a huge number of neurons . The neurons represent different groups of events of low mutual correlation , which preconditions the optimization results . 2 NEURAL NETWORK , A POWERFUL INFERENCE MACHINE OF PLAUSIBILITY OF EVENTS IN THE PHYSICAL WORLD . This section aims to give an intuitive description of the formal definition of NN given and its behaviors without delving into mathematical details ; and it also serves as the paper outline . The rest of the paper characterizes the informal description in this section formally . If an agent wants to interact with the world , for whatever reasons , it first needs to perceive it . A way to perceive the world is to measure certain physical objects of the world , which could be implemented as sensors of the agent , e.g. , a camera measuring the spatial configuration of photon intensity . However , the measurement data ( formalized in definition 1 ) record many events that happen in the same spatial and temporal span , and those events are entangled in the measurements ( formalized in assumption 3.5 ) . To perceive events happening in the world , e.g. , a lion is nearby , a mechanism is needed to recognize it from measurement data , e.g. , sensed spatial photon patterns . This leads to the problem of the structure of the physical world , which the still unknown mechanism at least needs to relate to if it aims to recognize events in the world well . This leads us to Complex System ( Bar-Yam ( 1997 ) Nicolis & Nicolis ( 2012 ) Newman ( 2009 ) ) . The research in the field allows us to safely say that one of the most important structures of the physical world is hierarchy ( Amderson ( 1972 ) ) . Atoms form molecules ; molecules form organism , and inorganic material ; organisms form creatures , which form ecology system ; inorganic material forms planets , then solar systems , then galaxies . Of the scales across the hierarchy of the world , the events at a lower scale interacting with a particular way form events at a higher scale . The hierarchy in nature is formulated as assumptions 3.1 3.2 3.3 3.4 3.5 . In section 3 , S-System is introduced in definition 3 to recognize and represent the hierarchy of events from measurement data , formulated measure-theoretically and based on probability coupling theory ( Thorisson ( 2000 ) ) . S-System formalizes the idea that a creature is not going to reproduce an impartial representation of the world , it only captures the events that cater to its need , e.g. , the survival need to identify lion , and within its reach and capacity , i.e. , the amount of measurements it can gather . S-System recognizes a hierarchy of events from the measurements , not exactly in the sense of physical reality — if a creature never measured/saw a black swan , it does not mean there aren ’ t any — but in the sense of a manually created hierarchical groups of events , and each group is perhaps given a tag/name . For examples , some low level object groups are named as edges ; a higher level , named textures ; a higher level , named body parts ; an even higher one , named body , e.g. , lion . In section 4 , a NN is shown to be an implementation of an S-system ( shown in definition 4 ) , when the measure is being approximated with compositional exponential family distributions ( defined in definition 6 ) . Activation Function is derived formally in definition 5 . The geometry structure of the representation built by NNs is defined in definition 7 as stochastic manifolds , and benefits of hierarchy are shown quantitatively as the coarse graining on “ distance ” in theorem 4.1 , which is the large scale physical quantity to renormalize in RG . The section shows that a NN is an inference system to infer how plausible groups of events forming a hierarchy have occurred . We have formally defined NNs and given the benefits of hierarchy , but the question remains is , “ is it practical ? ” . This is addressed in section 5 . The hierarchical organization of NNs to recognize and represent the hierarchical physical world has made itself a complex system . Contrary to existing shallow models , or simple models , it is exactly the complexity built by a hierarchy that makes NNs the most powerful inference model . A large collection of cooperative yet autonomous neurons , formalized as assumptions 5.1 5.2 , gives NNs the ability to partition events into arbitrary groups and infer the plausibility of any groups of events ( proved in theorem 5.1 ) , which is the emergent behavior emerging from the disorder in the NN complex system . Lastly , we note the notation used . All scalar functions are denoted as normal letters , e.g. , f ; bold , lowercase letters denote vectors , e.g. , x ; bold , uppercase letters denote matrices , e.g. , W ; normal , uppercase letters denotes random elements/variables , all the remaining symbols are defined when needed , and should be self-clear in the context . r.e . and r.e.s are short for random element , random elements respectively , so are r.v . and r.v.s for random variable and random variables . To index entries of a matrix , following Erdős et al . ( 2017 ) , we denote J = [ N ] : = { 1 , . . . , N } , the ordered pairs of indices by I = J × J . For α ∈ I , A ⊆ I , i ∈ J , given a matrix W , wα , WA , Wi : ,W : i denotes the entry at α , the vector consists of entries at A , the ith row , ith column ofW respectively . Given two matrix A , B , the curly inequality between matrices , i.e. , A B , means B − A is a positive definite matrix . Similar statements apply between a matrix and a vector , and a matrix and a scalar . is defined similarly . κ denotes cumulant , whose definition and norms , e.g. , |||κ||| , are reviewed at appendix F.2 . : = is the “ define ” symbol , where x : = y defines a new symbol x by equating it with y. tr denotes matrix trace . dg ( h ) denotes the diagonal matrix whose diagonal is the vector h .

Summary: The paper provides a new framework "S-System" as a generalization of hierarchal models including neural networks. The paper shows an alternative way to derive the activation functions commonly used in practice in a principled way. It further shows that the landscape of the optimization problem of neural networks has nice properties in the setting where the number of input/hidden units tending to infinity and the neurons satisfy certain diversity conditions.