Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology

While many approaches to make neural networks more fathomable have been proposed , they are restricted to interrogating the network with input data . Measures for characterizing and monitoring structural properties , however , have not been developed . In this work , we propose neural persistence , a complexity measure for neural network architectures based on topological data analysis on weighted stratified graphs . To demonstrate the usefulness of our approach , we show that neural persistence reflects best practices developed in the deep learning community such as dropout and batch normalization . Moreover , we derive a neural persistencebased stopping criterion that shortens the training process while achieving comparable accuracies as early stopping based on validation loss . 1 INTRODUCTION . The practical successes of deep learning in various fields such as image processing ( Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Hu et al. , 2018 ) , biomedicine ( Ching et al. , 2018 ; Rajpurkar et al. , 2017 ; Rajkomar et al. , 2018 ) , and language translation ( Bahdanau et al. , 2015 ; Sutskever et al. , 2014 ; Wu et al. , 2016 ) still outpace our theoretical understanding . While hyperparameter adjustment strategies exist ( Bengio , 2012 ) , formal measures for assessing the generalization capabilities of deep neural networks have yet to be identified ( Zhang et al. , 2017 ) . Previous approaches for improving theoretical and practical comprehension focus on interrogating networks with input data . These methods include i ) feature visualization of deep convolutional neural networks ( Zeiler & Fergus , 2014 ; Springenberg et al. , 2015 ) , ii ) sensitivity and relevance analysis of features ( Montavon et al. , 2017 ) , iii ) a descriptive analysis of the training process based on information theory ( Tishby & Zaslavsky , 2015 ; Shwartz-Ziv & Tishby , 2017 ; Saxe et al. , 2018 ; Achille & Soatto , 2018 ) , and iv ) a statistical analysis of interactions of the learned weights ( Tsang et al. , 2018 ) . Additionally , Raghu et al . ( 2017 ) develop a measure of expressivity of a neural network and use it to explore the empirical success of batch normalization , as well as for the definition of a new regularization method . They note that one key challenge remains , namely to provide meaningful insights while maintaining theoretical generality . This paper presents a method for elucidating neural networks in light of both aspects . We develop neural persistence , a novel measure for characterizing neural network structural complexity . In doing so , we adopt a new perspective that integrates both network weights and connectivity while not relying on interrogating networks through input data . Neural persistence builds on computational techniques from algebraic topology , specifically topological data analysis ( TDA ) , which was already shown to be beneficial for feature extraction in deep learning ( Hofer et al. , 2017 ) and describing the complexity of GAN sample spaces ( Khrulkov & Oseledets , 2018 ) . More precisely , we rephrase deep networks with fully-connected layers into the language of algebraic topology and develop a measure for assessing the structural complexity of i ) individual layers , and ii ) the entire network . In this work , we present the following contributions : - We introduce neural persistence , a novel measure for characterizing the structural complexity of neural networks that can be efficiently computed . - We prove its theoretical properties , such as upper and lower bounds , thereby arriving at a normalization for comparing neural networks of varying sizes . - We demonstrate the practical utility of neural persistence in two scenarios : i ) it correctly captures the benefits of dropout and batch normalization during the training process , and ii ) it can be easily used as a competitive early stopping criterion that does not require validation data . 2 BACKGROUND : TOPOLOGICAL DATA ANALYSIS . Topological data analysis ( TDA ) recently emerged as a field that provides computational tools for analysing complex data within a rigorous mathematical framework that is based on algebraic topology . This paper uses persistent homology , a theory that was developed to understand highdimensional manifolds ( Edelsbrunner et al. , 2002 ; Edelsbrunner & Harer , 2010 ) , and has since been successfully employed in characterizing graphs ( Sizemore et al. , 2017 ; Rieck et al. , 2018 ) , finding relevant features in unstructured data ( Lum et al. , 2013 ) , and analysing image manifolds ( Carlsson et al. , 2008 ) . This section gives a brief summary of the key concepts ; please refer to Edelsbrunner & Harer ( 2010 ) for an extensive introduction . Simplicial homology The central object in algebraic topology is a simplicial complex K , i.e . a high-dimensional generalization of a graph , which is typically used to describe complex objects such as manifolds . Various notions to describe the connectivity of K exist , one of them being simplicial homology . Briefly put , simplicial homology uses matrix reduction algorithms ( Munkres , 1996 ) to derive a set of groups , the homology groups , for a given simplicial complex K. Homology groups describe topological features—colloquially also referred to as holes—of a certain dimension d , such as connected components ( d = 0 ) , tunnels ( d = 1 ) , and voids ( d = 2 ) . The information from the dth homology group is summarized in a simple complexity measure , the dth Betti number βd , which merely counts the number of d-dimensional features : a circle , for example , has Betti numbers ( 1 , 1 ) , i.e . one connected component and one tunnel , while a filled circle has Betti numbers ( 1 , 0 ) , i.e . one connected component but no tunnel . In the context of analysing simple feedforward neural networks for two classes , Bianchini & Scarselli ( 2014 ) calculated bounds of Betti numbers of the decision region belonging to the positive class , and were thus able to show the implications of different activation functions . These ideas were extended by Guss & Salakhutdinov ( 2018 ) to obtain a measure of the topological complexity of decision boundaries . Persistent homology For the analysis of real-world data sets , however , Betti numbers turn out to be of limited use because their representation is too coarse and unstable . This prompted the development of persistent homology . Given a simplicial complex K with an additional set of weights a0 ≤ a1 ≤ · · · ≤ am−1 ≤ am , which are commonly thought to represent the idea of a scale , it is possible to put K in a filtration , i.e . a nested sequence of simplicial complexes ∅ = K0 ⊆ K1 ⊆ · · · ⊆ Km−1 ⊆ Km = K. This filtration is thought to represent the ‘ growth ’ of K as the scale is being changed . During this growth process , topological features can be created ( new vertices may be added , for example , which creates a new connected component ) or destroyed ( two connected components may merge into one ) . Persistent homology tracks these changes and represents the creation and destruction of a feature as a point ( ai , aj ) ∈ R2 for indices i ≤ j with respect to the filtration . The collection of all points corresponding to d-dimensional topological features is called the dth persistence diagram Dd . It can be seen as a collection of Betti numbers at multiple scales . Given a point ( x , y ) ∈ Dd , the quantity pers ( x , y ) : = |y − x| is referred to as its persistence . Typically , high persistence is considered to correspond to features , while low persistence is considered to indicate noise ( Edelsbrunner et al. , 2002 ) . 3 A NOVEL MEASURE FOR NEURAL NETWORK COMPLEXITY . This section details neural persistence , our novel measure for assessing the structural complexity of neural networks . By exploiting both network structure and weight information through persistent homology , our measure captures network expressiveness and goes beyond mere connectivity properties . Subsequently , we describe its calculation , provide theorems for theoretical and empirical bounds , and show the existence of neural networks complexity regimes . To summarize this section , Figure 1 illustrates how our method treats a neural network . 3.1 NEURAL PERSISTENCE . Given a feedforward neural network with an arrangement of neurons and their connections E , let W refer to the set of weights . SinceW is typically changing during training , we require a function φ : E → W that maps a specific edge to a weight . Fixing an activation function , the connections form a stratified graph . Definition 1 ( Stratified graph and layers ) . A stratified graph is a multipartite graph G = ( V , E ) satisfying V = V0 ⊔ V1 ⊔ . . . , such that if u ∈ Vi , v ∈ Vj , and ( u , v ) ∈ E , we have j = i + 1 . Hence , edges are only permitted between adjacent vertex sets . Given k ∈ N , the kth layer of a stratified graph is the unique subgraph Gk : = ( Vk ⊔ Vk+1 , Ek : = E ∩ { Vk × Vk+1 } ) . This enables calculating the persistent homology of G and each Gk , using the filtration induced by sorting all weights , which is common practice in topology-based network analysis ( Carstens & Horadam , 2013 ; Horak et al. , 2009 ) where weights often represent closeness or node similarity . However , our context requires a novel filtration because the weights arise from an incremental fitting procedure , namely the training , which could theoretically lead to unbounded values . When analysing geometrical data with persistent homology , one typically selects a filtration based on the ( Euclidean ) distance between data points ( Bubenik , 2015 ) . The filtration then connects points that are increasingly distant from each other , starting from points that are direct neighbours . Our network filtration aims to mimic this behaviour in the context of fully-connected neural networks . Our framework does not explicitly take activation functions into account ; however , activation functions influence the evolution of weights during training . Filtration Given the set of weightsW for one training step , let wmax : = maxw∈W |w| . Furthermore , letW ′ : = { |w|/wmax | w ∈ W } be the set of transformed weights , indexed in non-ascending order , such that 1 = w′0 ≥ w′1 ≥ · · · ≥ 0 . This permits us to define a filtration for the kth layer Gk as G ( 0 ) k ⊆ G ( 1 ) k ⊆ . . . , where G ( i ) k : = ( Vk ⊔ Vk+1 , { ( u , v ) | ( u , v ) ∈ Ek ∧ φ′ ( u , v ) ≥ w′i } ) and φ′ ( u , v ) ∈ W ′ denotes the transformed weight of an edge . We tailored this filtration towards the analysis of neural networks , for which large ( absolute ) weights indicate that certain neurons exert a larger influence over the final activation of a layer . The strength of a connection is thus preserved by the filtration , and weaker weights with |w| ≈ 0 remain close to 0 . Moreover , since w′ ∈ [ 0 , 1 ] holds for the transformed weights , this filtration makes the network invariant to scaling , which simplifies the comparison of different networks . Persistence diagrams Having set up the filtration , we can calculate persistent homology for every layer Gk . As the filtration contains at most 1-simplices ( edges ) , we capture zero-dimensional topological information , i.e . how connected components are created and merged during the filtration . These information are structurally equivalent to calculating a maximum spanning tree using the Algorithm 1 Neural persistence calculation Require : Neural network with l layers and weightsW 1 : wmax ← maxw∈W |w| ▷ Determine largest absolute weight 2 : W ′ ← { |w|/wmax | w ∈ W } ▷ Transform weights for filtration 3 : for k ∈ { 0 , . . . , l − 1 } do 4 : Fk ← G ( 0 ) k ⊆ G ( 1 ) k ⊆ . . . ▷ Establish filtration of kth layer 5 : Dk ← PERSISTENTHOMOLOGY ( Fk ) ▷ Calculate persistence diagram 6 : end for 7 : return { ∥D0∥p , . . . , ∥Dl−1∥p } ▷ Calculate neural persistence for each layer weights , or performing hierarchical clustering with a specific setup ( Carlsson & Mémoli , 2010 ) . While it would theoretically be possible to include higher-dimensional information about each layer Gk , for example in the form of cliques ( Rieck et al. , 2018 ) , we focus on zero-dimensional information in this paper , because of the following advantages : i ) the resulting values are easily interpretable as they essentially describe the clustering of the network at multiple weight thresholds , ii ) previous research ( Rieck & Leitte , 2016 ; Hofer et al. , 2017 ) indicates that zero-dimensional topological information is already capturing a large amount of information , and iii ) persistent homology calculations are highly efficient in this regime ( see below ) . We thus calculate zero-dimensional persistent homology with this filtration . The resulting persistence diagrams have a special structure : since our filtration solely sorts edges , all vertices are present at the beginning of the filtration , i.e . they are already part of G ( 0 ) k for each k. As a consequence , they are assigned a weight of 1 , resulting in |Vk×Vk+1| connected components . Hence , entries in the corresponding persistence diagramDk are of the form ( 1 , x ) , with x ∈ W ′ , and will be situated below the diagonal , similar to superlevel set filtrations ( Bubenik , 2015 ; Cohen-Steiner et al. , 2009 ) . Using the p-norm of a persistence diagram , as introduced by Cohen-Steiner et al . ( 2010 ) , we obtain the following definition for neural persistence . Definition 2 ( Neural persistence ) . The neural persistence of the kth layer Gk , denoted by NP ( Gk ) , is the p-norm of the persistence diagram Dk resulting from our previously-introduced filtration , i.e . NP ( Gk ) : = ∥Dk∥p : = ( ∑ ( c , d ) ∈Dk pers ( c , d ) p ) 1 p , ( 1 ) which ( for p = 2 ) captures the Euclidean distance of points in Dk to the diagonal . The p-norm is known to be a stable summary ( Cohen-Steiner et al. , 2010 ) of topological features in a persistence diagram . For neural persistence to be a meaningful measure of structural complexity , it should increase as a neural network is learning . We evaluate this and other properties in Section 4 . Algorithm 1 provides pseudocode for the calculation process . It is highly efficient : the filtration ( line 4 ) amounts to sorting all n weights of a network , which has a computational complexity of O ( n log n ) . Calculating persistent homology of this filtration ( line 5 ) can be realized using an algorithm based on union–find data structures Edelsbrunner et al . ( 2002 ) . This has a computational complexity of O ( n · α ( n ) ) , where α ( · ) refers to the extremely slow-growing inverse of the Ackermann function ( Cormen et al. , 2009 , Chapter 22 ) . We make our implementation and experiments available under https : //github.com/BorgwardtLab/Neural-Persistence .

Summary: This paper proposes to analyze the complexity of a neural network using its zero-th persistent homology. Each layer is considered a bipartite graph with edge weights. As edges are being added in a monotonically decreasing order, each time a connected component is merged with others will be recorded as a new topological feature. The persistence of each topological feature is measured as the weight difference between the new edge and the maximal weight (properly normalized). Experiments show that by monitoring the p-norm of these persistence values one can stop the training a few epochs earlier than the validation-error-based early stopping strategy, with only slightly worse test accuracy.