Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks

1 Introduction . In recent years , deep neural networks ( DNNs ) have become the de facto machine learning tool and have revolutionized a variety of fields such as natural language processing [ DCLT18 ] , image perception [ KSH12 , RBH+21 ] , geometry processing [ QSMG17 , ZBL+20 ] and 3D vision [ DBI18 , GLW+21 ] . Despite their widespread use , little is known about their theoretical properties . Even now the top-performing DNNs are designed by trial-and-error , a pesky , burdensome process for the average practitioner [ EMH+19 ] . Furthermore , even if a top-performing architecture is found , it is difficult to provide performance guarantees on a large class of real-world datasets . This lack of theoretical understanding has motivated a plethora of work focusing on explaining what , how , and why a neural network learns . To answer many of these questions , one naturally examines the generalization error , a measure quantifying the differing performance on train and 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) test data since this provides significant insights into whether the network is learning or simply memorizing [ ZBH+21 ] . However , generalization in neural networks is particularly confusing as it refutes the classical proposals of statistical learning theory such as uniform bounds based on the Rademacher complexity [ BM02 ] and the Vapnik–Chervonenkis ( VC ) dimension [ Vap68 ] . Instead , recent analyses have started focusing on the dynamics of deep neural networks . [ NBMS17 , BO18 , GJ16 ] provide analyses on the final trained network , but these miss out on critical training patterns . To remedy this , a recent study [ SSDE20 ] connected generalization and the heavy tailed behavior of network trajectories–a phenomenon which had already been observed in practice [ SSG19 , ŞGN+19 , SZTG20 , GSZ21 , CWZ+21 , HM20 , MM19 ] . [ SSDE20 ] further showed that the generalization error can be linked to the fractal dimension of a parametric hypothesis class ( which can then be taken as the optimization trajectories ) . Hence , the fractal dimension acts as a ‘ capacity metric ’ for generalization . While [ SSDE20 ] brought a new perspective to generalization , several shortcomings prevent application in everyday training . In particular , their construction requires several conditions which may be infeasible in practice : ( i ) topological regularity conditions on the hypothesis class for fast computation , ( ii ) a Feller process assumption on the training algorithm trajectory , and that ( iii ) the Feller process exhibits a specific diffusive behavior near a minimum . Furthermore , the capacity metrics in [ SSDE20 ] are not optimization friendly and therefore can ’ t be incorporated into training . In this work , we address these shortcomings by exploiting the recently developed connections between fractal dimension and topological data analysis ( TDA ) . First , by relating the box dimension [ Sch09 ] and the recently proposed persistent homology ( PH ) dimension [ Sch20 ] , we relax the assumptions in [ SSDE20 ] to develop a topological intrinsic dimension ( ID ) estimator . Then , using this estimator we develop a general tool for computing and visualizing generalization properties in deep learning . Finally , by leveraging recently developed differentiable TDA tools [ CHU17 , CHN19 ] , we employ our ID estimator to regularize training towards solutions that generalize better , even without having access to the test dataset . Our experiments demonstrate that this new measure of intrinsic dimension correlates highly with generalization error , regardless of the choice of optimizer . Furthermore , as a proof of concept , we illustrate that our topological regularizer is able to improve the test accuracy and lower the generalization error . In particular , this improvement is most pronounced when the learning rate/batch size normally results in a poorer test accuracy . Overall , our contributions are summarized as follows : • We make a novel connection between statistical learning theory and TDA in order to develop a generic computational framework for the generalization error . We remove the topological regularity condition and the decomposable Feller assumption on training trajectories , which were required in [ SSDE20 ] . This leads to a more generic capacity metric . • Using insights from our above methodology , we leverage the differentiable properties of persistent homology to regularize neural network training . Our findings also provide the first steps towards theoretically justifying recent topological regularization methods [ BGND+19 , CNBW19 ] . • We provide extensive experiments to illustrate the theory , strength , and flexibility of our framework . We believe that the novel connections and the developed framework will open new theoretical and computational directions in the theory of deep learning . To foster further developments at the the intersection of persistent homology and statistical learning theory , we release our source code under : https : //github.com/tolgabirdal/PHDimGeneralization . 2 Related Work . Intrinsic dimension in deep networks Even though a large number of parameters are required to train deep networks [ FC18 ] , modern interpretations of deep networks avoid correlating model over-fitting or generalization to parameter counting . Instead , contemporary studies measure model complexity through the degrees of freedom of the parameter space [ JFH15 , GJ16 ] , compressibility ( pruning ) [ BO18 ] or intrinsic dimension [ ALMZ19 , LFLY18 , MWH+18 ] . Tightly related to the ID , Janson et al . [ JFH15 ] investigated the degrees of freedom [ Ghr10 ] in deep networks and expected difference between test error and training error . Finally , LDMNet [ ZQH+18 ] explicitly penalizes the ID regularizing the network training . Generalization bounds Several studies have provided theoretical justification to the observations that trained neural networks live in a lower-dimensional space , and this is related to the generalization performance . In particular , compression-based generalization bounds [ AGNZ18 , SAM+20 , SAN20 , HJTW21 , BSE+21 ] have shown that the generalization error of a neural network can be much lower if it can be accurately represented in lower dimensional space . Approaching the problem from a geometric viewpoint , [ SSDE20 ] showed that the generalization error can be formally linked to the fractal dimension of a parametric hypothesis class . This dimension indeed the plays role of the intrinsic dimension , which can be much smaller than the ambient dimension . When the hypothesis class is chosen as the trajectories of the training algorithm , [ SSDE20 ] further showed that the error can be linked to the heavy-tail behavior of the trajectories . Deep networks & topology Previous works have linked neural network training and topological invariants , although all analyze the final trained network [ FGFAEV21 ] . For example , in [ RTB+19 ] , the authors construct Neural Persistence , a measure on neural network layer weights . They furthermore show that Neural Persistence reflects many of the properties of convergence and can classify weights based on whether they overfit , underfit , or exactly fit the data . In a parallel line of work , [ DZF19 ] analyze neural network training by calculating topological properties of the underlying graph structure . This is expanded upon in [ CMEM20 ] , where the authors compute correlations between neural network weights and show that the homology is linked with the generalization error . However , these previous constructions have been done mostly in an adhoc manner . As a result , many of the results are mostly empirical and work must still be done to show that these methods hold theoretically . Our proposed method , by contrast , is theoretically well-motivated and uses tools from statistical persistent homology theory to formally links the generalization error with the network training trajectory topology . We also would like to note that prior work has incorporated topological loss functions to help normalize training . In particular , [ BGND+19 ] constructed a topological normalization term for GANs to help maintain the geometry of the generated 3d point clouds . 3 Preliminaries & Technical Background . We imagine a point cloud W = { wi ∈ Rd } as a geometric realization of a d-dimensional topological space W ⊂ W ⊂ Rd . Bδ ( x ) ⊂ Rd denotes the closed ball centered around x ∈ Rd with radius δ . Persistent Homology From a topological perspective , W can be viewed a cell complex composed of the disjoint union of k-dimensional balls or cells σ ∈ W glued together . For k = 0 , 1 , 2 , . . . , we form a chain complex C ( W ) = . . . Ck+1 ( W ) ∂k+1−−−→ Ck ( W ) ∂k−→ . . . by sequencing chain groupsCk ( W ) , whose elements are equivalence classes of cycles , via boundary maps ∂k : Ck ( W ) 7→ Ck−1 ( W ) with ∂k−1◦∂k ≡ 0 . In this paper , we work with finite simplicial complexes restricting the cells to be simplices . The kth homology group or k-dimensional homology is then defined as the equivalence classes of k-dimensional cycles who differ only by a boundary , or in other words , the quotient group Hk ( W ) = Zk ( W ) /Yk ( W ) where Zk ( W ) = ker ∂k and Yk ( W ) = im ∂k+1 . The generators or basis of H0 ( W ) , H1 ( W ) and H2 ( W ) describe the shape of the topological spaceW by its connected components , holes and cavities , respectively . Their ranks are related to the Betti numbers i.e.βk = rank ( Hk ) . Definition 1 ( Čech and Vietoris-Rips Complexes ) . For W a set of fine points in a metric space , the Čech cell complex Čechr ( W ) is constructed using the intersection of r-balls around W , Br ( W ) : Čechr ( W ) = { Q ⊂ W : ∩x∈QBr ( x ) 6= 0 } . The construction of such complex is intricate . Instead , the Vietoris-Rips complex VRr ( W ) closely approximates Čechr ( W ) using only the pairwise distances or the intersection of two r-balls [ RB21 ] : Wr = VRr ( W ) = { Q ⊂ W : ∀x , x′ ∈ Q , Br ( x ) ∩Br ( x′ ) 6= 0 } . Definition 2 ( Persistent Homology ) . PH indicates a multi-scale version of homology applied over a filtration { Wt } t : = VR ( W ) : ∀ ( s ≤ t ) Ws ⊂ Wt ⊂ W , keeping track of holes created ( born ) or filled ( died ) as t increases . Each persistence module PHk ( VR ( W ) ) = { γi } i keeps track of a single k-persistence cycle γi from birth to death . We denote the entire lifetime of cycle γ as I ( γ ) and its length as |I ( γ ) | = death ( γ ) − birth ( γ ) . We will also use persistence diagrams , 2D plots of all persistence lifetimes ( death vs. birth ) . Note that for PH0 , the Čech and VR complexes are equivalent . Lifetime intervals are instrumental in TDA as they allow for extraction of topological features or summaries . Note that , each birth-death pair can be mapped to the cells that respectively created and destroyed the homology class , defining a unique map for a persistence diagram , which lends itself to differentibility [ BGND+19 , CHN19 , CHU17 ] . We conclude this brief section by referring the interested reader to the well established literature of persistent homology [ Car14 , EH10 ] for a thorough understanding . Intrinsic Dimension The intrinsic dimension of a space can be measured by using various notions . In this study , we will consider two notions of dimension , namely the upper-box dimension ( also called the Minkowski dimension ) and the persistent homology dimension . The box dimension is based on covering numbers and can be linked to generalization via [ SSDE20 ] , whereas the PH dimension is based on the notions defined earlier in this section . We start by the box dimension . Definition 3 ( Upper-Box Dimension ) . For a bounded metric space W , let Nδ ( W ) denote the maximal number of disjoint closed δ-balls with centers inW . The upper box dimension is defined as : dimBoxW = lim sup δ→0 ( log ( Nδ ( W ) ) /log ( 1/δ ) ) . ( 1 ) We proceed with the PH dimension . First let us define an intermediate construct , which will play a key role in our computational tools . Definition 4 ( α-Weighted Lifetime Sum ) . For a finite set W ⊂ W ⊂ Rd , the weighted ith homology lifetime sum is defined as follows : Eiα ( W ) = ∑ γ∈PHi ( VR ( W ) ) |I ( γ ) |α , ( 2 ) where PHi ( VR ( W ) ) is the i-dimensional persistent homology of the Čech complex on a finite point set W contained inW and |I ( γ ) | is the persistence lifetime as explained above . Now , we are ready to define the PH dimension , which is the key notion in this paper . Definition 5 ( Persistent Homology Dimension ) . The PHi-dimension of a bounded metric spaceW is defined as follows : dimiPHW : = inf { α : Eiα ( W ) < C ; ∃C > 0 , ∀ finite W ⊂ W } . ( 3 ) In words , dimiPHW is the smallest exponent α for which Eiα is uniformly bounded for all finite subsets ofW .

Summary: This paper proposes a measure of generalization based on persistence homology, a standard technique of topological data analysis.  The proposed measure, the persistence homology dimension (PHD), is computed on optimization trajectories during training.  A theoretical connect between PHD and generalization is made using existing results in the literature: first the equivalence of PHD and the box-dimension [KLS06, Sch19], and second that the generalization error may be bounded under the box dimension (under the assumption that the optimization dynamics follow a Feller process).  From these main result of the paper, Proposition 1, then follows under assumption (H1) .  The authors then proceed to remove the Feller requirement in Proposition. 2 using a recent result in the mathematics literature [Sch20].  Based on these results, an algorithm for computing PHD is proposed.  The authors proceed to perform an empirical evaluation of how well PHD predicts generalization for select deep network architectures (alexnet, cnn-9, fcn-5, fcn-7), datasets (mnist, cifar-10, cifar-100), and training hyperparameters (learning rate, batch size).  The authors visualize these results in Figures 2 and 3, where in Figure 3 the authors claim that PHD "directly" correlates with generalization error across experimental settings. Linear trends are evident in the plots, however no measures of goodness of fit are given.  In addition, a novel loss regularizer is proposed based on their PHD measure. The effect on generalization of his regularizer is studied, but small differences are shown.  Ablation studies are additionally carried out.