Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks

1 Introduction . In recent years , deep neural networks ( DNNs ) have become the de facto machine learning tool and have revolutionized a variety of fields such as natural language processing [ DCLT18 ] , image perception [ KSH12 , RBH+21 ] , geometry processing [ QSMG17 , ZBL+20 ] and 3D vision [ DBI18 , GLW+21 ] . Despite their widespread use , little is known about their theoretical properties . Even now the top-performing DNNs are designed by trial-and-error , a pesky , burdensome process for the average practitioner [ EMH+19 ] . Furthermore , even if a top-performing architecture is found , it is difficult to provide performance guarantees on a large class of real-world datasets . This lack of theoretical understanding has motivated a plethora of work focusing on explaining what , how , and why a neural network learns . To answer many of these questions , one naturally examines the generalization error , a measure quantifying the differing performance on train and 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) test data since this provides significant insights into whether the network is learning or simply memorizing [ ZBH+21 ] . However , generalization in neural networks is particularly confusing as it refutes the classical proposals of statistical learning theory such as uniform bounds based on the Rademacher complexity [ BM02 ] and the Vapnik–Chervonenkis ( VC ) dimension [ Vap68 ] . Instead , recent analyses have started focusing on the dynamics of deep neural networks . [ NBMS17 , BO18 , GJ16 ] provide analyses on the final trained network , but these miss out on critical training patterns . To remedy this , a recent study [ SSDE20 ] connected generalization and the heavy tailed behavior of network trajectories–a phenomenon which had already been observed in practice [ SSG19 , ŞGN+19 , SZTG20 , GSZ21 , CWZ+21 , HM20 , MM19 ] . [ SSDE20 ] further showed that the generalization error can be linked to the fractal dimension of a parametric hypothesis class ( which can then be taken as the optimization trajectories ) . Hence , the fractal dimension acts as a ‘ capacity metric ’ for generalization . While [ SSDE20 ] brought a new perspective to generalization , several shortcomings prevent application in everyday training . In particular , their construction requires several conditions which may be infeasible in practice : ( i ) topological regularity conditions on the hypothesis class for fast computation , ( ii ) a Feller process assumption on the training algorithm trajectory , and that ( iii ) the Feller process exhibits a specific diffusive behavior near a minimum . Furthermore , the capacity metrics in [ SSDE20 ] are not optimization friendly and therefore can ’ t be incorporated into training . In this work , we address these shortcomings by exploiting the recently developed connections between fractal dimension and topological data analysis ( TDA ) . First , by relating the box dimension [ Sch09 ] and the recently proposed persistent homology ( PH ) dimension [ Sch20 ] , we relax the assumptions in [ SSDE20 ] to develop a topological intrinsic dimension ( ID ) estimator . Then , using this estimator we develop a general tool for computing and visualizing generalization properties in deep learning . Finally , by leveraging recently developed differentiable TDA tools [ CHU17 , CHN19 ] , we employ our ID estimator to regularize training towards solutions that generalize better , even without having access to the test dataset . Our experiments demonstrate that this new measure of intrinsic dimension correlates highly with generalization error , regardless of the choice of optimizer . Furthermore , as a proof of concept , we illustrate that our topological regularizer is able to improve the test accuracy and lower the generalization error . In particular , this improvement is most pronounced when the learning rate/batch size normally results in a poorer test accuracy . Overall , our contributions are summarized as follows : • We make a novel connection between statistical learning theory and TDA in order to develop a generic computational framework for the generalization error . We remove the topological regularity condition and the decomposable Feller assumption on training trajectories , which were required in [ SSDE20 ] . This leads to a more generic capacity metric . • Using insights from our above methodology , we leverage the differentiable properties of persistent homology to regularize neural network training . Our findings also provide the first steps towards theoretically justifying recent topological regularization methods [ BGND+19 , CNBW19 ] . • We provide extensive experiments to illustrate the theory , strength , and flexibility of our framework . We believe that the novel connections and the developed framework will open new theoretical and computational directions in the theory of deep learning . To foster further developments at the the intersection of persistent homology and statistical learning theory , we release our source code under : https : //github.com/tolgabirdal/PHDimGeneralization . 2 Related Work . Intrinsic dimension in deep networks Even though a large number of parameters are required to train deep networks [ FC18 ] , modern interpretations of deep networks avoid correlating model over-fitting or generalization to parameter counting . Instead , contemporary studies measure model complexity through the degrees of freedom of the parameter space [ JFH15 , GJ16 ] , compressibility ( pruning ) [ BO18 ] or intrinsic dimension [ ALMZ19 , LFLY18 , MWH+18 ] . Tightly related to the ID , Janson et al . [ JFH15 ] investigated the degrees of freedom [ Ghr10 ] in deep networks and expected difference between test error and training error . Finally , LDMNet [ ZQH+18 ] explicitly penalizes the ID regularizing the network training . Generalization bounds Several studies have provided theoretical justification to the observations that trained neural networks live in a lower-dimensional space , and this is related to the generalization performance . In particular , compression-based generalization bounds [ AGNZ18 , SAM+20 , SAN20 , HJTW21 , BSE+21 ] have shown that the generalization error of a neural network can be much lower if it can be accurately represented in lower dimensional space . Approaching the problem from a geometric viewpoint , [ SSDE20 ] showed that the generalization error can be formally linked to the fractal dimension of a parametric hypothesis class . This dimension indeed the plays role of the intrinsic dimension , which can be much smaller than the ambient dimension . When the hypothesis class is chosen as the trajectories of the training algorithm , [ SSDE20 ] further showed that the error can be linked to the heavy-tail behavior of the trajectories . Deep networks & topology Previous works have linked neural network training and topological invariants , although all analyze the final trained network [ FGFAEV21 ] . For example , in [ RTB+19 ] , the authors construct Neural Persistence , a measure on neural network layer weights . They furthermore show that Neural Persistence reflects many of the properties of convergence and can classify weights based on whether they overfit , underfit , or exactly fit the data . In a parallel line of work , [ DZF19 ] analyze neural network training by calculating topological properties of the underlying graph structure . This is expanded upon in [ CMEM20 ] , where the authors compute correlations between neural network weights and show that the homology is linked with the generalization error . However , these previous constructions have been done mostly in an adhoc manner . As a result , many of the results are mostly empirical and work must still be done to show that these methods hold theoretically . Our proposed method , by contrast , is theoretically well-motivated and uses tools from statistical persistent homology theory to formally links the generalization error with the network training trajectory topology . We also would like to note that prior work has incorporated topological loss functions to help normalize training . In particular , [ BGND+19 ] constructed a topological normalization term for GANs to help maintain the geometry of the generated 3d point clouds . 3 Preliminaries & Technical Background . We imagine a point cloud W = { wi ∈ Rd } as a geometric realization of a d-dimensional topological space W ⊂ W ⊂ Rd . Bδ ( x ) ⊂ Rd denotes the closed ball centered around x ∈ Rd with radius δ . Persistent Homology From a topological perspective , W can be viewed a cell complex composed of the disjoint union of k-dimensional balls or cells σ ∈ W glued together . For k = 0 , 1 , 2 , . . . , we form a chain complex C ( W ) = . . . Ck+1 ( W ) ∂k+1−−−→ Ck ( W ) ∂k−→ . . . by sequencing chain groupsCk ( W ) , whose elements are equivalence classes of cycles , via boundary maps ∂k : Ck ( W ) 7→ Ck−1 ( W ) with ∂k−1◦∂k ≡ 0 . In this paper , we work with finite simplicial complexes restricting the cells to be simplices . The kth homology group or k-dimensional homology is then defined as the equivalence classes of k-dimensional cycles who differ only by a boundary , or in other words , the quotient group Hk ( W ) = Zk ( W ) /Yk ( W ) where Zk ( W ) = ker ∂k and Yk ( W ) = im ∂k+1 . The generators or basis of H0 ( W ) , H1 ( W ) and H2 ( W ) describe the shape of the topological spaceW by its connected components , holes and cavities , respectively . Their ranks are related to the Betti numbers i.e.βk = rank ( Hk ) . Definition 1 ( Čech and Vietoris-Rips Complexes ) . For W a set of fine points in a metric space , the Čech cell complex Čechr ( W ) is constructed using the intersection of r-balls around W , Br ( W ) : Čechr ( W ) = { Q ⊂ W : ∩x∈QBr ( x ) 6= 0 } . The construction of such complex is intricate . Instead , the Vietoris-Rips complex VRr ( W ) closely approximates Čechr ( W ) using only the pairwise distances or the intersection of two r-balls [ RB21 ] : Wr = VRr ( W ) = { Q ⊂ W : ∀x , x′ ∈ Q , Br ( x ) ∩Br ( x′ ) 6= 0 } . Definition 2 ( Persistent Homology ) . PH indicates a multi-scale version of homology applied over a filtration { Wt } t : = VR ( W ) : ∀ ( s ≤ t ) Ws ⊂ Wt ⊂ W , keeping track of holes created ( born ) or filled ( died ) as t increases . Each persistence module PHk ( VR ( W ) ) = { γi } i keeps track of a single k-persistence cycle γi from birth to death . We denote the entire lifetime of cycle γ as I ( γ ) and its length as |I ( γ ) | = death ( γ ) − birth ( γ ) . We will also use persistence diagrams , 2D plots of all persistence lifetimes ( death vs. birth ) . Note that for PH0 , the Čech and VR complexes are equivalent . Lifetime intervals are instrumental in TDA as they allow for extraction of topological features or summaries . Note that , each birth-death pair can be mapped to the cells that respectively created and destroyed the homology class , defining a unique map for a persistence diagram , which lends itself to differentibility [ BGND+19 , CHN19 , CHU17 ] . We conclude this brief section by referring the interested reader to the well established literature of persistent homology [ Car14 , EH10 ] for a thorough understanding . Intrinsic Dimension The intrinsic dimension of a space can be measured by using various notions . In this study , we will consider two notions of dimension , namely the upper-box dimension ( also called the Minkowski dimension ) and the persistent homology dimension . The box dimension is based on covering numbers and can be linked to generalization via [ SSDE20 ] , whereas the PH dimension is based on the notions defined earlier in this section . We start by the box dimension . Definition 3 ( Upper-Box Dimension ) . For a bounded metric space W , let Nδ ( W ) denote the maximal number of disjoint closed δ-balls with centers inW . The upper box dimension is defined as : dimBoxW = lim sup δ→0 ( log ( Nδ ( W ) ) /log ( 1/δ ) ) . ( 1 ) We proceed with the PH dimension . First let us define an intermediate construct , which will play a key role in our computational tools . Definition 4 ( α-Weighted Lifetime Sum ) . For a finite set W ⊂ W ⊂ Rd , the weighted ith homology lifetime sum is defined as follows : Eiα ( W ) = ∑ γ∈PHi ( VR ( W ) ) |I ( γ ) |α , ( 2 ) where PHi ( VR ( W ) ) is the i-dimensional persistent homology of the Čech complex on a finite point set W contained inW and |I ( γ ) | is the persistence lifetime as explained above . Now , we are ready to define the PH dimension , which is the key notion in this paper . Definition 5 ( Persistent Homology Dimension ) . The PHi-dimension of a bounded metric spaceW is defined as follows : dimiPHW : = inf { α : Eiα ( W ) < C ; ∃C > 0 , ∀ finite W ⊂ W } . ( 3 ) In words , dimiPHW is the smallest exponent α for which Eiα is uniformly bounded for all finite subsets ofW .

Summary: The relationship between generalized loss of NN and Intrinsic Dimension by Persistent homology is mathematically proved. They also propose a normalization term based on the theory presented. In addition, we experimentally verify the theoretical results and confirm the effectiveness of the proposed algorithm.