Barcodes as summary of objective functions' topology

1 INTRODUCTION . The learning via finding minima of objective functions is the principal strategy underlying majority of learning algorithms . For example , in Neural Network training , the objective function ’ s input is model parameters ( weights ) and the objective function ’ s output is the loss on training dataset . The graph of the loss function , often called loss surface , typically has complex structure ( e.g . see loss surface visualisations by Li et al . ( 2018 ) ) : non-convexity , many local minima , flat regions , steep slopes . These obstacles harm exploration of the loss surface and complicate searching for optimal network weights . The optimization of modern neural networks is based on the gradient descent algorithm . The global topological characteristics of the gradient vector field trajectories are captured by the Morse complex via decomposing the parameter space into cells of uniform flow , see Barannikov ( 1994 ) ; Le Roux et al . ( 2018 ) and references therein . The invariants of Morse complex called `` canonical forms '' ( or barcodes ) constitute the fundamental summary of the topology of the gradient vector field flow . The `` canonical forms '' , or barcodes , in this context are decompositions of the change of topology of the sublevel sets of objective function into simple `` birth-death '' phenomena of topological feautures of different dimensions . The calculation of the barcodes for different functions constitutes the essence of the topological data analysis . The currently available software packages for the calculation of barcodes of functions , also called `` sublevel persistence '' , are GUDHI , Dionysus , PHAT , and TDA package which incorporates all three previous packages B.T.Fasy et al . ( 2014 ) . They are based on the algorithm , described in Barannikov ( 1994 ) , see also appendix and e.g . Bauer et al . ( 2014 ) and references therein . This algorithm which has complexity of O ( n3 ) . These packages can currently handle calculations of barcodes for functions defined on a grid of up to 106 points , and in dimensions two and three . Thus all current packages have the scalability issues . We describe a new algorithm for computations of the barcodes of functions in lowest degree . Our algorithm works with functions defined on randomly sampled or specifically chosen point clouds . Point cloud based methods are known to work better than grid based methods in optimization related problems ( Bergstra and Bengio ( 2012 ) ) . We also use the fact that the definition of the barcode of lowest degree can be reformulated in geometrical terms ( see definition 1 in section 2 ) . The previously known algorithms were based on the more algebraic approach as in definition 3 . Our algorithm has complexity of O ( n log ( n ) ) . It was tested in dimensions up to 16 and with number of points of up to 108 . In this work , we develop a methodology to describe the properties of the loss surface of the neural network via topological features of local minima . We emphasize that the value of the objective function at the minimum can be viewed as only a part of its topological characteristic from the “ canonical form ” ( barcode ) . The second half can be described as the value of objective function at the index-one saddle , which can be naturally associated with each local minimum . The difference between the values of objective function at the associated index-one saddle and at the local minimum is a topological invariant of the minimum . For optimization algorithms this quantity measures , in particular , the obligatory penalty for moving from the given local minimum to a lower minimum . The main contributions of the paper are as follows : Applying the one-to-one correspondence between local minima and 1-saddles to exploration of loss surfaces . For each local minimum p there is canonically defined 1-saddle q ( see Section 2 ) . The 1-saddle associated with p can be described as follows . The 1- saddle q is precisely the point where the connected component of the sublevel set Θf≤c = { θ ∈ Θ | f ( θ ) ≤ c } containing the minimum p merges with another connected component of the sublevel set whose minimum is lower . This correspondence between the local minima and the 1-saddles , killing a connected component of Θf≤c , is one-to-one . The segment [ f ( p ) , f ( q ) ] is then the “ canonical form ” invariant attached to the minimum p. The set of all such segments is the barcode ( `` canonical form '' ) of minima invariant of f . It is a robust topological invariant of objective function . It is invariant in particular under the action of homeomorphisms of Θ . Full “ canonical form ” invariants give a concise summary of the topology of objective function and of the global structure of its gradient flow . Algorithm for calculations of the barcodes ( canonical invariants ) of minima . We describe an algorithm for calculation of the canonical invariants of minima . The algorithm works with function ’ s values on a a randomly sampled or specifically chosen set of points . The local minima give birth to clusters of points in sublevel sets . The algorithm works by looking at neighbors of each point with lower value of the function and deciding if this point belongs to the existing clusters , gives birth to a new cluster ( minimum ) , or merges two or more clusters ( index one saddle ) . A variant of the algorithm has complexity of O ( n log ( n ) ) , where n is the cardinality of the set of points . Calculations confirming observations on behaviour of neural networks loss functions barcodes . We calculate the canonical invariants ( barcodes ) of minima for small fully-connected neural networks of up to three hidden layers and verify that all segments of minima ’ s barcode belong to a small lower part of the total range of loss function ’ s values and that with the increase in the neural network depth the minima ’ s barcodes descend lower . The usefulness of our approach and algorithms is clearly not limited to the optimization problems . Our algorithm permits really fast computation of the canonical form invariants ( persistence barcodes ) of many functions which were not accessible until now . These sublevel persistence barcodes have been successfully applied in different discipline , to mention just a few : cognitive science ( M. K. Chung and Kim ( 2009 ) ) , cosmology ( Sousbie et al . ( 2011 ) ) , see e.g . Pun et al . ( 2018 ) and references therein . Our viewpoint should also have applications in chemistry and material science where 1-saddle points on potential energy landscapes correspond to transition states and minima are stable states corresponding to different materials or protein foldings ( see e.g . Dellago et al . ( 2003 ) , Oganov and Valle ( 2009 ) ) . The article is structured as follows . First we describe three definitions of barcodes of minima . After that our algorithm for their calculation is described . In the last part we give examples of calculations , including the loss functions of simple neural nets . 2 TOPOLOGY OF LOSS SURFACES VIA CANONICAL FORM INVARIANTS . The “ canonical form ” invariants ( barcodes ) give a concise summary of topological features of functions ( see Barannikov ( 1994 ) , Le Roux et al . ( 2018 ) and references therein ) . These invariants describe a decomposition of the change of topology of the function into the finite sum of “ birth ” – “ death ” of elementary features . We propose to apply these invariants as a tool for exploring topology of loss surfaces . In this work we concentrate on the part of these canonical form invariants , describing the “ birth ” – “ death ” phenomena of connected components of sublevel sets of the function . However it should be stressed that this approach works similarly also for “ almost minima ” , i.e . for the critical points ( manifolds ) of small indexes , which are often the terminal points of the optimization algorithms in very high dimensions . We give three definitions of the “ canonical form ” invariants of minima . DEFINITION 1 : MERGING WITH CONNECTED COMPONENT OF A LOWER MINIMUM The values of parameter c at which the topology of sublevel set Θf≤c = { θ ∈ Θ | f ( θ ) ≤ c } changes are critical values of f . Let p be one of minima of f . When c increases from f ( p ) − to f ( p ) + , a new connected component of the set Θf≤c is born ( see fig 1a , the connected components S1 , S2 , S3 of sublevel set are born at the blue , green and red minima correspondingly . If p is a minimum , which is not global , then , when c is increased , the connected component of Θf≤c born at p merges with a connected component born at a lower minimum . Let q is the merging point where this happens . The intersection of the set Θf < f ( q ) with any small neighborhood of q has two connected components . This is the index-one saddle q associated with p. Also these two subsets of small neighborhood of q belong to two different connected components of the whole set Θf < f ( q ) . The 1-saddles of this type are called “ + ” ( “ plus ” ) or “ death ” type . The described correspondence between local minima and 1-saddles of this type is one-to-one . In a similar way , the 1-saddle q associated with p can be described also as follows . Proposition 2.1 . Consider various paths γ starting from the local minimum p and going to a lower minimum . Let mγ ∈ Θ is the maximum of the restriction of f to such path γ . Then 1-saddle q which is paired with the local minimum p is the minimum over the set of all such paths γ of the maxima mγ : q = arg min γ : [ 0,1 ] →Θ γ ( 0 ) =p , f ( γ ( 1 ) ) < f ( p ) [ max t f ( γ ( t ) ) ] DEFINITION 2 : NEW MINIMUM ON CONNECTED COMPONENTS OF SUBLEVEL SETS The correspondence in the opposite direction can be described analogously . Let q is a 1-saddle point of such type that the two branches of the set Θf≤f ( q ) − near q belong to two different connected components of Θf≤f ( q ) − . A new connected component of the set Θf≤c is formed when c decreases from f ( q ) + to f ( q ) − . The restriction of f to each of the two connected components has its global minimum . Proposition 2.2 . Given a 1-saddle q , the minimum p which is paired with q is the new minimum of f on the connected component of the set Θf≤c which is formed when c decreases from f ( q ) + to f ( q ) − . The two branches of the set Θf≤f ( q ) − near q can also belong to the same connected components of this set . Then such saddle is of “ birth ” type and it is naturally paired with index-two saddle of “ death ” type ( see theorem 2.3 ) . DEFINITION 3 : INVARIANTS OF FILTERED COMPLEXES Chain complex is the algebraic counterpart of intuitive idea representing complicated geometric objects as a decomposition into simple pieces . It converts such a decomposition into a collection of vector spaces and linear maps . A chain complex ( C∗ , ∂∗ ) is a sequence of finite-dimensional k-vector spaces and linear operators → Cj+1 ∂j+1→ Cj ∂j→ Cj−1 → . . .→ C0 , which satisfy ∂j ◦ ∂j+1 = 0 . The j−th homology of the chain complex ( C∗ , ∂∗ ) is the quotient Hj = ker ( ∂j ) /im ( ∂j+1 ) . A chain complex C∗ is called R−filtered if C∗ is equipped with an increasing sequence of subcomplexes ( R−filtration ) Fs1C∗ ⊂ Fs2C∗ ⊂ . . . ⊂ FsmaxC∗ = C∗ , indexed by a finite set of real numbers s1 < s2 < . . . < smax . Theorem 2.3 . ( Barannikov ( 1994 ) ) Any R−filtered chain complex C∗ can be brought by a linear transformation preserving the filtration to “ canonical form ” , a canonically defined direct sum of R−filtered complexes of two types : one-dimensional complexes with trivial differential ∂j ( ei ) = 0 and two-dimensional complexes with trivial homology ∂j ( ei2 ) = ei1 . The resulting canonical form is uniquely determined . The full barcode is a visualization of the decomposition of an R−filtered complexes according to the theorem 2.3 . Each filtered 2-dimensional complex with trivial homology ∂j ( ei2 ) = ei1 , 〈ei1〉 = F≤s1 , 〈ei1 , ei2〉 = F≤s2 describes a topological feature in dimension j which is `` born '' at s1 and which `` dies '' at s2 . It is represented by segment [ s1 , s2 ] in the degree-j barcode . And each filtered 1-dimensional complex with trivial differential , ∂jei = 0 , 〈ei〉 = F≤r describes a topological feature in dimension j which is `` born '' at r and never `` dies '' . It is represented by the half-line [ r , +∞ [ in the degree-j barcode . The proof of the theorem is given in Appendix . Essentially , one can bring an R−filtered complex to the required canonical form by induction , starting from the lowest basis elements of degree one , in such a way that the manipulation of degree j basis elements does not destroy the canonical form in degree j − 1 and in lower filtration pieces in degree j . Let f : Θ→ R is smooth , or more generally , piece-wise smooth continuous function such that the sublevel sets Θf≤c = { θ ∈ Θ | f ( θ ) ≤ c } are compact . One filtered complex naturally associated with function f and such that the subcomplexes FsC∗ compute the homology of sublevel sets Θf≤s is the gradient ( Morse ) complex , see e.g . Barannikov ( 1994 ) ; Le Peutrec et al . ( 2013 ) and references therein . Without loss of generality the function f can be assumed smooth here , otherwise one can always replace f by its smoothing . By adding a small perturbation such as a regularization term we can also assume that critical points of f are non-degenerate . The generators of the gradient ( Morse ) complex correspond to the critical points of f . The differential is defined by counting gradient trajectories between critical points when their number is finite . The canonical form of the gradient ( Morse ) complex describes a decomposition of the gradient flow associated with f into standard simple pieces . Let p be a minimum , which is not a global minimum . Then the generator corresponding to p represents trivial homology class in the canonical form , since the homology class of its connected component is already represented by the global minimum . Then p is the lower generator of a two-dimensional complex with trivial homology in the canonical form . I.e . p is paired with an index-one saddle q in the canonical form . The segment [ f ( p ) , f ( q ) ] is then the canonical invariant ( barcode ) corresponding to the minimum p. The full canonical form of the gradient ( Morse ) complex of all indexes is a summary of global structure of the objective function ’ s gradient flow . The total number of different topological features in sublevel sets Θf≤c of the objective function can be read immediately from the barcode . Namely the number of intersections of horizontal line at level c with segments in the index j barcode gives the number of independent topological features of dimension j in Θf≤c . The description of the barcode of minima on manifold Θ with nonempty boundary ∂Θ is modified in the following way . A connected component can be also born at a local minimum of restriction of f to the boundary f |∂Θ , if gradf is pointed inside manifold Θ . The merging of two connected components can also happen at an index-one critical point of f |∂Θ , if gradf is pointed inside Θ .

Summary: This paper introduces the notion of barcodes as a topological invariant of loss surfaces that encodes the "depth" of local minima by associating to each minimum the lowest index-one saddle. An algorithm is presented for the computation of barcodes, and some small-scale experiments are conducted. For very small neural networks, the barcodes are found to live at small loss values, and the authors argue that this suggests it may be hard to get stuck in a suboptimal local minimum.