Characterizing and Measuring the Similarity of Neural Networks with Persistent Homology 

1 INTRODUCTION . Machine learning practitioners can train different neural networks for the same task . Even for the same neural architecture , there are many hyperparameters , such as the number of neurons per layer or the number of layers . Moreover , the final weights for the same architecture and hyperparameters can vary depending on the initialization and the optimization process itself , which is stochastic . Thus , there is no direct way of comparing neural networks accounting for the fact that neural networks solving the same task should be measured as being similar , regardless of the specific weights . This also prevents one from finding and comparing modules inside neural networks ( e.g. , determining if a given sub-network does the same function as other sub-network in another model ) . Moreover , there are no well-known methods for effectively characterizing neural networks . This work aims to characterize neural networks such that they can be measured to be similar once trained for the same task , with independence of the particular architecture , initialization , or optimization process . To characterize neural networks and measuring their similarity , we assume two different similarities that should be related if the characterization is correct , namely , 1. similarity between the tasks a model is trained on , and 2. similarity between neural networks themselves considered as mathematical objects .. We do not provide a mathematical definition for the first one , rather an intuitive one , and understand is as qualitative similarity between semantically close tasks . For instance , we think that MNIST ( 10 labels ) is closer to MNIST restricted to 2 labels than to language identification , and that MNIST is closer to MNIST restricted to 8 labels than to MNIST restricted to 2 labels . For the second one , in this work we propose a similarity measure . Ideally , this similarity measure should then capture 1. the semantic similarity between the underlying tasks neural networks have been trained on ( understanding that networks performing similar tasks should be similar , regardless of their specific weights and architectures ) , and 2. intrinsic properties of the neural network should also be captured to some degree . Thus , we consider two scenarios , that is , comparing NNs trained for similar tasks , using the mentioned task similarity , and comparing neural networks trained on identical tasks but with different architectures . We focus on Multi-Layer Perceptrons ( MLPs ) for the sake of simplicity . We start by observing that we can represent a neural network as a directed weighted graph to which we can associate certain topological concepts.1 Considering it as a simplicial complex , we obtain its associated Persistent Diagram . Then , we can compute distances between Persistent Diagrams of different neural networks . 1See Jonsson ( 2007 ) for a complete reference on graph topology . The proposed experiments aim to show that the selected structural feature , Persistent Homology , serves to relate neural networks trained for similar problems and that such a comparison can be performed by means of a predefined measure between the associated Persistent Homology diagrams . To test the hypothesis , we study different classical problems ( MNIST , Fashion MNIST , CIFAR-10 , and language identification and text classification datasets ) , different architectures ( number and size of layers ) as well as a control experiment ( input order , to which neural network similarity should be invariant ) . The code and results are fully open source in the Supplementary Material under a MIT license . In summary , the main contributions of this work are the following : 1 . We propose an effective graph characterization strategy of neural networks based on Persistent Homology . 2 . Based on this characterization , we suggest a similarity measure of neural networks . 3 . We provide empirical evidence that this Persistent Homology framework captures valuable information from neural networks and that the proposed similarity measure is meaningful . The remainder of this paper is organized as follows . In Section 2 , we go through the related work . Then , in Section 3 we describe our proposal and the experimental framework to validate it . Finally , in sections 4 and 5 we report and discuss the results and arrive to conclusions , respectively . 2 RELATED WORK . One of the fundamental papers of Topological Data Analysis ( TDA ) is presented in Carlsson ( 2009 ) and suggests the use of Algebraic Topology to obtain qualitative information and deal with metrics for large amounts of data . For an extensive overview of simplicial topology on graphs , see Giblin ( 1977 ) ; Jonsson ( 2007 ) . Aktas et al . ( 2019 ) provide a thorough analysis of PH methods . More recently , a number of publications have dealt with the study of the capacity of neural networks using PH . Guss & Salakhutdinov ( 2018 ) characterize learnability of different neural architectures by computable measures of data complexity . Donier ( 2019 ) propose the concept of spatial capacity allocation analysis . Konuk & Smith ( 2019 ) propose an empirical study of how NNs handle changes in topological complexity of the input data . Rieck et al . ( 2019b ) introduce the neural persistence metric , a complexity measure based on TDA on weighted stratified graphs . This work suggests a representation of the neural network as a multipartite graph . They perform the filtering of the Persistent Homology diagrams independently for each layer . As the filtration contains at most 1-simplices ( edges ) , they only capture zero-dimensional topological information , i.e . connectivity information . When consecutive layer analysis is used , the global topology of the network is not taken into account making the strong assumption that the NNs encode the learned information layer pairwise exclusively . Additionally , there are trivial global transformations of a NN that are not captured by analyzing pairs of layers : • Superfluous cycle insertions : for example , add two neurons and connect their input to a single neuron in a lower layer and their two outputs to a single output neuron in an upper layer with opposite weights . • Identity layer insertions : for instance , insert an intermediate identity layer with neurons and trivially connect to the next layer . • Non-planar neural networks analysis : the analysis of neural networks that use multiple connections between non-consecutive layers require higher order topological analysis . In terms of pure neural network analysis , there are relevant works , like Hofer et al . ( 2020 ) , that study topological regularization . Clough et al . ( 2020 ) introduce a method for training neural networks for image segmentation with prior topology knowledge , specifically via Betti numbers . Corneanu et al . ( 2020 ) try to estimate ( with limited success ) the performance gap between training and testing via neuron activations and linear regression of the Betti numbers . This type of representation depend on the input data and not only on the NN function under study . Instead , we are interested in characterising and comparing NNs as functions , independently of the data to which they are applied . On the other hand , topological analysis of decision boundaries has been a very prolific area . Ramamurthy et al . ( 2019 ) propose a labeled Vietoris-Rips complex to perform PH inference of decision boundaries for quantification of the complexity of neural networks . Naitzat et al . ( 2020 ) experiment on the PH of a wide range of point cloud input datasets for a binary classification problems to see that NNs transform a topologically rich dataset ( in terms of Betti numbers ) into a topologically simpler one as it passes through the layers . They also verify that the reduction in Betti numbers is significantly faster for ReLU activations than hyperbolic tangent activations . Regarding neural network representations , one of the most related works to ours , Gebhart et al . ( 2019 ) , focuses on topological representations of neural networks . They introduce a method for computing PH over the graphical activation structure of neural networks , which provides access to the task-relevant substructures activated throughout the network for a given input . Interestingly , in Watanabe & Yamana ( 2020 ) , authors work on neural network representations through simplicial complexes based on deep Taylor decomposition and they calculate the PH of neural networks in this representation . In Chowdhury et al . ( 2019 ) , they use directed homology to represent MLPs . They show that the path homology of these networks is non-trivial in higher dimensions and depends on the number and size of the network layers . They investigate homological differences between distinct neural network architectures . As far as neural network similarity measures are concerned , the literature is not especially prolific . In Kornblith et al . ( 2019 ) , authors examine similarity measures for representations ( meaning , outputs of different layers ) of neural networks based on canonical correlation analysis . However , note that this method compares neural network representations ( intermediate outputs ) , not the neural networks themselves . Remarkably , in Ashmore & Gashler ( 2015 ) , authors do deal with the intrinsic similarity of neural networks themselves based on Forward Bipartite Alignment . Specifically , they propose an algorithm for aligning the topological structures of two neural networks . Their algorithm finds optimal bipartite matches between the nodes of the two MLPs by solving the well-known graph cutting problem . The alignment enables applications such as visualizations or improving ensembles . However , the methods only works under very restrictive assumptions,2 and this line of work does not appear to have been followed up . Finally , we note that there has been a considerable growth of interest in applied topology in the recent years . This popularity increase and the development of new software libraries,3 along with the growth of computational capabilities , have empowered new works . Some of the most remarkable libraries are Ripser Tralie et al . ( 2018 ) ; Bauer ( 2021 ) , and Flagser Lütgehetmann et al . ( 2019 ) . They are focused on the efficient computation of PH . For GPU-Accelerated computation of Vietoris-Rips PH , Ripser++ Zhang et al . ( 2020 ) offers an important speedup . The Python library we are using , Giotto-TDA Tauzin et al . ( 2020 ) , makes use of both above libraries underneath . We have seen that there is a trend towards the use of algebraic topology methods for having a better understanding of phenomena of neural networks and having more principled deep learning algorithms . Nevertheless , little to no works have proposed neural network characterizations or similarity measures based on intrinsic properties of the networks , which is what we intend to do . 3 METHODOLOGY . In this section , we propose our method , which is heavily based on concepts from algebraic topology . We refer the reader to the Supplementary Material for the mathematical definitions . In this section , we also describe the conducted experiments . Intrinsically characterizing and comparing neural networks is a difficult , unsolved problem . First , the network should be represented in an object that captures as much information as possible and then it should be compared with a measure depending on the latent structure . Due to the stochasticity of both the initialization and training procedure , networks are parameterized differently . For the same task , different functions that effectively solve it can be obtained . Being able to compare the trained networks can be helpful to detect similar neural structures . We want to obtain topological characterizations associated to neural networks trained on a given task . For doing so , we use the Persistence Homology ( from now on , PH ) of the graph associated to a 2For example , the two neural networks `` must have the same number of units in each of their corresponding layers '' , and the match is done layer by layer . 3https : //www.math.colostate.edu/~adams/advising/appliedTopologySoftware/ neural network . We compute the PH for various neural networks learned on different tasks . We then compare all the diagrams for each one of the task . More specifically , for each of the studied tasks ( image classification on MNIST , Fashion MNIST and CIFAR-10 ; language identification , and text classification on the Reuters dataset ) ,4 we proceed as follows : 1 . We train several neural network models on the particular problem . 2 . We create a directed graph from the weights of the trained neural networks ( after changing the direction of the negative edges and normalising the weights of the edges ) . 3 . We consider the directed graph as a simplicial complex and calculate its PH , using the weight of the edges as the filtering parameter , which range from 0 to 1 . This way we obtain the so-called Persistence Diagram . 4 . We compute the distances between the Persistence Diagrams ( prior discretization of the Persistence Diagram so that it can be computed ) of the different networks . 5 . Finally , we analyze the similarity between different neural networks trained for the same task , for a similar task , and for a completely different task , independently of the concrete architecture , to see whether there is topological similarity . As baselines , we set two standard matrix comparison methods that are the 1-Norm and the Frobenius norm . Having adjacency matrix A and B , we compute the difference as norm ( A−B ) . However , these methods only work for matrices of similar size and thus , they are not general enough . We could also have used the Fast Approximate Quadratic assignment algorithm suggested in Vogelstein et al . ( 2015 ) , but for large networks this method becomes unfeasible to compute .

Summary: In this work, the authors propose to characterize neural networks with Topological Data Analysis, more precisely with its main descriptor, the so-called persistence diagram, in order to be able to compare neural networks with different numbers of layers, different numbers of neurons, or trained on different data sets. More specifically, they show that the computational graphs corresponding to the neural networks can be filtered using the edge weights that are learnt during training, in order to produce filtered flag complexes, from which persistent homology can be computed. Then, the authors interpreted the distances between the persistence diagrams obtained from networks with varying parameters (number of layers, neurons, labels), and showed that, often, the distances have intuitive correlation with the complexity of the networks.