Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology

While many approaches to make neural networks more fathomable have been proposed , they are restricted to interrogating the network with input data . Measures for characterizing and monitoring structural properties , however , have not been developed . In this work , we propose neural persistence , a complexity measure for neural network architectures based on topological data analysis on weighted stratified graphs . To demonstrate the usefulness of our approach , we show that neural persistence reflects best practices developed in the deep learning community such as dropout and batch normalization . Moreover , we derive a neural persistencebased stopping criterion that shortens the training process while achieving comparable accuracies as early stopping based on validation loss . 1 INTRODUCTION . The practical successes of deep learning in various fields such as image processing ( Simonyan & Zisserman , 2015 ; He et al. , 2016 ; Hu et al. , 2018 ) , biomedicine ( Ching et al. , 2018 ; Rajpurkar et al. , 2017 ; Rajkomar et al. , 2018 ) , and language translation ( Bahdanau et al. , 2015 ; Sutskever et al. , 2014 ; Wu et al. , 2016 ) still outpace our theoretical understanding . While hyperparameter adjustment strategies exist ( Bengio , 2012 ) , formal measures for assessing the generalization capabilities of deep neural networks have yet to be identified ( Zhang et al. , 2017 ) . Previous approaches for improving theoretical and practical comprehension focus on interrogating networks with input data . These methods include i ) feature visualization of deep convolutional neural networks ( Zeiler & Fergus , 2014 ; Springenberg et al. , 2015 ) , ii ) sensitivity and relevance analysis of features ( Montavon et al. , 2017 ) , iii ) a descriptive analysis of the training process based on information theory ( Tishby & Zaslavsky , 2015 ; Shwartz-Ziv & Tishby , 2017 ; Saxe et al. , 2018 ; Achille & Soatto , 2018 ) , and iv ) a statistical analysis of interactions of the learned weights ( Tsang et al. , 2018 ) . Additionally , Raghu et al . ( 2017 ) develop a measure of expressivity of a neural network and use it to explore the empirical success of batch normalization , as well as for the definition of a new regularization method . They note that one key challenge remains , namely to provide meaningful insights while maintaining theoretical generality . This paper presents a method for elucidating neural networks in light of both aspects . We develop neural persistence , a novel measure for characterizing neural network structural complexity . In doing so , we adopt a new perspective that integrates both network weights and connectivity while not relying on interrogating networks through input data . Neural persistence builds on computational techniques from algebraic topology , specifically topological data analysis ( TDA ) , which was already shown to be beneficial for feature extraction in deep learning ( Hofer et al. , 2017 ) and describing the complexity of GAN sample spaces ( Khrulkov & Oseledets , 2018 ) . More precisely , we rephrase deep networks with fully-connected layers into the language of algebraic topology and develop a measure for assessing the structural complexity of i ) individual layers , and ii ) the entire network . In this work , we present the following contributions : - We introduce neural persistence , a novel measure for characterizing the structural complexity of neural networks that can be efficiently computed . - We prove its theoretical properties , such as upper and lower bounds , thereby arriving at a normalization for comparing neural networks of varying sizes . - We demonstrate the practical utility of neural persistence in two scenarios : i ) it correctly captures the benefits of dropout and batch normalization during the training process , and ii ) it can be easily used as a competitive early stopping criterion that does not require validation data . 2 BACKGROUND : TOPOLOGICAL DATA ANALYSIS . Topological data analysis ( TDA ) recently emerged as a field that provides computational tools for analysing complex data within a rigorous mathematical framework that is based on algebraic topology . This paper uses persistent homology , a theory that was developed to understand highdimensional manifolds ( Edelsbrunner et al. , 2002 ; Edelsbrunner & Harer , 2010 ) , and has since been successfully employed in characterizing graphs ( Sizemore et al. , 2017 ; Rieck et al. , 2018 ) , finding relevant features in unstructured data ( Lum et al. , 2013 ) , and analysing image manifolds ( Carlsson et al. , 2008 ) . This section gives a brief summary of the key concepts ; please refer to Edelsbrunner & Harer ( 2010 ) for an extensive introduction . Simplicial homology The central object in algebraic topology is a simplicial complex K , i.e . a high-dimensional generalization of a graph , which is typically used to describe complex objects such as manifolds . Various notions to describe the connectivity of K exist , one of them being simplicial homology . Briefly put , simplicial homology uses matrix reduction algorithms ( Munkres , 1996 ) to derive a set of groups , the homology groups , for a given simplicial complex K. Homology groups describe topological features—colloquially also referred to as holes—of a certain dimension d , such as connected components ( d = 0 ) , tunnels ( d = 1 ) , and voids ( d = 2 ) . The information from the dth homology group is summarized in a simple complexity measure , the dth Betti number βd , which merely counts the number of d-dimensional features : a circle , for example , has Betti numbers ( 1 , 1 ) , i.e . one connected component and one tunnel , while a filled circle has Betti numbers ( 1 , 0 ) , i.e . one connected component but no tunnel . In the context of analysing simple feedforward neural networks for two classes , Bianchini & Scarselli ( 2014 ) calculated bounds of Betti numbers of the decision region belonging to the positive class , and were thus able to show the implications of different activation functions . These ideas were extended by Guss & Salakhutdinov ( 2018 ) to obtain a measure of the topological complexity of decision boundaries . Persistent homology For the analysis of real-world data sets , however , Betti numbers turn out to be of limited use because their representation is too coarse and unstable . This prompted the development of persistent homology . Given a simplicial complex K with an additional set of weights a0 ≤ a1 ≤ · · · ≤ am−1 ≤ am , which are commonly thought to represent the idea of a scale , it is possible to put K in a filtration , i.e . a nested sequence of simplicial complexes ∅ = K0 ⊆ K1 ⊆ · · · ⊆ Km−1 ⊆ Km = K. This filtration is thought to represent the ‘ growth ’ of K as the scale is being changed . During this growth process , topological features can be created ( new vertices may be added , for example , which creates a new connected component ) or destroyed ( two connected components may merge into one ) . Persistent homology tracks these changes and represents the creation and destruction of a feature as a point ( ai , aj ) ∈ R2 for indices i ≤ j with respect to the filtration . The collection of all points corresponding to d-dimensional topological features is called the dth persistence diagram Dd . It can be seen as a collection of Betti numbers at multiple scales . Given a point ( x , y ) ∈ Dd , the quantity pers ( x , y ) : = |y − x| is referred to as its persistence . Typically , high persistence is considered to correspond to features , while low persistence is considered to indicate noise ( Edelsbrunner et al. , 2002 ) . 3 A NOVEL MEASURE FOR NEURAL NETWORK COMPLEXITY . This section details neural persistence , our novel measure for assessing the structural complexity of neural networks . By exploiting both network structure and weight information through persistent homology , our measure captures network expressiveness and goes beyond mere connectivity properties . Subsequently , we describe its calculation , provide theorems for theoretical and empirical bounds , and show the existence of neural networks complexity regimes . To summarize this section , Figure 1 illustrates how our method treats a neural network . 3.1 NEURAL PERSISTENCE . Given a feedforward neural network with an arrangement of neurons and their connections E , let W refer to the set of weights . SinceW is typically changing during training , we require a function φ : E → W that maps a specific edge to a weight . Fixing an activation function , the connections form a stratified graph . Definition 1 ( Stratified graph and layers ) . A stratified graph is a multipartite graph G = ( V , E ) satisfying V = V0 ⊔ V1 ⊔ . . . , such that if u ∈ Vi , v ∈ Vj , and ( u , v ) ∈ E , we have j = i + 1 . Hence , edges are only permitted between adjacent vertex sets . Given k ∈ N , the kth layer of a stratified graph is the unique subgraph Gk : = ( Vk ⊔ Vk+1 , Ek : = E ∩ { Vk × Vk+1 } ) . This enables calculating the persistent homology of G and each Gk , using the filtration induced by sorting all weights , which is common practice in topology-based network analysis ( Carstens & Horadam , 2013 ; Horak et al. , 2009 ) where weights often represent closeness or node similarity . However , our context requires a novel filtration because the weights arise from an incremental fitting procedure , namely the training , which could theoretically lead to unbounded values . When analysing geometrical data with persistent homology , one typically selects a filtration based on the ( Euclidean ) distance between data points ( Bubenik , 2015 ) . The filtration then connects points that are increasingly distant from each other , starting from points that are direct neighbours . Our network filtration aims to mimic this behaviour in the context of fully-connected neural networks . Our framework does not explicitly take activation functions into account ; however , activation functions influence the evolution of weights during training . Filtration Given the set of weightsW for one training step , let wmax : = maxw∈W |w| . Furthermore , letW ′ : = { |w|/wmax | w ∈ W } be the set of transformed weights , indexed in non-ascending order , such that 1 = w′0 ≥ w′1 ≥ · · · ≥ 0 . This permits us to define a filtration for the kth layer Gk as G ( 0 ) k ⊆ G ( 1 ) k ⊆ . . . , where G ( i ) k : = ( Vk ⊔ Vk+1 , { ( u , v ) | ( u , v ) ∈ Ek ∧ φ′ ( u , v ) ≥ w′i } ) and φ′ ( u , v ) ∈ W ′ denotes the transformed weight of an edge . We tailored this filtration towards the analysis of neural networks , for which large ( absolute ) weights indicate that certain neurons exert a larger influence over the final activation of a layer . The strength of a connection is thus preserved by the filtration , and weaker weights with |w| ≈ 0 remain close to 0 . Moreover , since w′ ∈ [ 0 , 1 ] holds for the transformed weights , this filtration makes the network invariant to scaling , which simplifies the comparison of different networks . Persistence diagrams Having set up the filtration , we can calculate persistent homology for every layer Gk . As the filtration contains at most 1-simplices ( edges ) , we capture zero-dimensional topological information , i.e . how connected components are created and merged during the filtration . These information are structurally equivalent to calculating a maximum spanning tree using the Algorithm 1 Neural persistence calculation Require : Neural network with l layers and weightsW 1 : wmax ← maxw∈W |w| ▷ Determine largest absolute weight 2 : W ′ ← { |w|/wmax | w ∈ W } ▷ Transform weights for filtration 3 : for k ∈ { 0 , . . . , l − 1 } do 4 : Fk ← G ( 0 ) k ⊆ G ( 1 ) k ⊆ . . . ▷ Establish filtration of kth layer 5 : Dk ← PERSISTENTHOMOLOGY ( Fk ) ▷ Calculate persistence diagram 6 : end for 7 : return { ∥D0∥p , . . . , ∥Dl−1∥p } ▷ Calculate neural persistence for each layer weights , or performing hierarchical clustering with a specific setup ( Carlsson & Mémoli , 2010 ) . While it would theoretically be possible to include higher-dimensional information about each layer Gk , for example in the form of cliques ( Rieck et al. , 2018 ) , we focus on zero-dimensional information in this paper , because of the following advantages : i ) the resulting values are easily interpretable as they essentially describe the clustering of the network at multiple weight thresholds , ii ) previous research ( Rieck & Leitte , 2016 ; Hofer et al. , 2017 ) indicates that zero-dimensional topological information is already capturing a large amount of information , and iii ) persistent homology calculations are highly efficient in this regime ( see below ) . We thus calculate zero-dimensional persistent homology with this filtration . The resulting persistence diagrams have a special structure : since our filtration solely sorts edges , all vertices are present at the beginning of the filtration , i.e . they are already part of G ( 0 ) k for each k. As a consequence , they are assigned a weight of 1 , resulting in |Vk×Vk+1| connected components . Hence , entries in the corresponding persistence diagramDk are of the form ( 1 , x ) , with x ∈ W ′ , and will be situated below the diagonal , similar to superlevel set filtrations ( Bubenik , 2015 ; Cohen-Steiner et al. , 2009 ) . Using the p-norm of a persistence diagram , as introduced by Cohen-Steiner et al . ( 2010 ) , we obtain the following definition for neural persistence . Definition 2 ( Neural persistence ) . The neural persistence of the kth layer Gk , denoted by NP ( Gk ) , is the p-norm of the persistence diagram Dk resulting from our previously-introduced filtration , i.e . NP ( Gk ) : = ∥Dk∥p : = ( ∑ ( c , d ) ∈Dk pers ( c , d ) p ) 1 p , ( 1 ) which ( for p = 2 ) captures the Euclidean distance of points in Dk to the diagonal . The p-norm is known to be a stable summary ( Cohen-Steiner et al. , 2010 ) of topological features in a persistence diagram . For neural persistence to be a meaningful measure of structural complexity , it should increase as a neural network is learning . We evaluate this and other properties in Section 4 . Algorithm 1 provides pseudocode for the calculation process . It is highly efficient : the filtration ( line 4 ) amounts to sorting all n weights of a network , which has a computational complexity of O ( n log n ) . Calculating persistent homology of this filtration ( line 5 ) can be realized using an algorithm based on union–find data structures Edelsbrunner et al . ( 2002 ) . This has a computational complexity of O ( n · α ( n ) ) , where α ( · ) refers to the extremely slow-growing inverse of the Ackermann function ( Cormen et al. , 2009 , Chapter 22 ) . We make our implementation and experiments available under https : //github.com/BorgwardtLab/Neural-Persistence .

Summary: The paper proposes the notion of "neural persistence", i.e., a topological measure to assign scores to fully-connected layers in a neural network. Essentially, a simplicial complex is constructed by considering neurons as 0-simplices and connections as 1-simplices. Using the (normalized) connection weights then facilitates to define a filtration. Persistent homology (for 0-dim. homology groups) then provides a concise summary of the evolution of the 0-dim. features over the filtration in the form of a barcode. The p-norm of the persistence diagram (containing points (1,w_i)) is then used to define the "neural persistence"  NP(G_k) of a layer G_k; this measure is averaged over all layers to obtain one final neural persistence score. Thm. 1 establishes lower and upper bounds on N(G_k); Experiments show that neural persistence, measured for small networks on MNIST, aligns well with previous observations that batch-norm and dropout are benefical for generalization and training. Further, neural persistence it can be used as an early stopping criterion without having to rely on validation data.