Persistent Homology Captures the Generalization of Neural Networks Without A Validation Set

1 INTRODUCTION . Generalization is what makes a machine learning model useful ; the uncertainty of its behaviour with unseen data is what makes it potentially dangerous . Thus , understanding the generalization error of a model can be considered one of the holy grails of the entire machine learning field . Machine learning practitioners typically monitor some metrics of the model to estimate its generalization error and stop the training even before the numerical convergence to prevent the overfitting of the model . Usually , the error measure or the metric relevant to the task is computed for a holdout set , the validation set . Since these data have not been directly used for updating the parameters , it is assumed that the performance of the model on the validation set can be used as a proxy of the generalization error , provided it is representative of the data that will be used in inference . One can , though , potentially overfit to this holdout set if is repeatedly used for guiding a hyperparameter search . Instead of relying on an external set , though , the question of whether it could be possible to estimate the generalization error with some intrinsic property of the model is highly relevant , and it has been barely explored in the literature . On the other hand , Algebraic Topology has recently been gaining momentum as a mathematical tool for studying graphs , machine learning algorithms , and data . In this work , we have the goal of , once having characterized neural networks as weighted , acyclic graphs , represented as Algebraic Topology objects ( following previous works ) , computing distances between consecutive neural network states . More specifically , we can calculate the Persistent Homology ( PH ) diagram distances between a give state ( i.e. , when having a specific weights during the training process ) and the next one ( i.e. , after having updated the weights in a training step ) , as depicted in Figure 1 . We observe that during the training procedure of neural networks we can measure this distance in each learning step , and show that there exists a high correlation with the corresponding validation accuracy of the model . We do so in a diverse set of deep learning benchmarks and model hyperparameters . This shines light on the question of whether the generalization error could be estimated from intrinsic properties of the model , and opens the path towards a better theoretical understanding of the dynamics of the training of neural networks . In summary , our contributions are as follows : • Based on principles of Algebraic Topology , we propose measuring the distances ( Silhouette and Heat ) between the PH persistence diagrams obtained from a given state of a neural network during the training procedure and the one in the immediately previous weights update . • We empirically show that the evolution of these measures during training correlate with the accuracy in the validation set . We do so in diverse benchmarks ( MNIST , CIFAR10 , CIFAR100 , Reuters text classification ) , and models ( MLPs in MNIST and Reuters , MLPs and CNNs in CIFAR100 and CIFAR100 ) . • We thus provide empirical proof of the fact that valuable information related to the learning process of neural networks can be obtained from PH distances between persistence diagrams ( we will call this process homological convergence ) . In particular , we show that homological convergence is related to learning process and the generalization properties of neural networks . • In practice , we provide a new tool for monitoring the training of neural networks , and open the path to estimating their generalization error without a validation set . The remainder of this article is as follows . In Section 2 we describe the theoretical background of our proposal in terms of Algebraic Topology , while in Section 3 we go through the related work . Then , in Section 4 we formalize our method . Finally , in sections 6 and 7 we present and discuss our empirical results , respectively . 2 BACKGROUND . In this section we introduce the mathematical foundations of this paper . A detailed mathematical description is included in the Supplementary Material . A simplicial complex is a set composed of points , line segments , triangles , and their n-dimensional counterparts , named simplex ( K ) . In particular , a simplicial complex must comply with two properties : 1 . Every face of a simplex is also in the simplicial complex ( of lower dimension ) . 2 . The non-empty intersection of any two simplices contained on a simplicial complex is a face of both . 0,1,2,3-simplex and non simplex examples are shown in Figure 2 . We can associate to an undirected graph , G = ( V , E ) , a simplicial complex where all the vertices of G are the 0-simplex of the simplicial complex and the complete subgraphs with i vertices , in G corresponds to a ( i−1 ) -simplex . This type of construction is usually called a complex clique on the graph G , and is denoted by Cl ( G ) . Figure 3 shows a graph clique complex Cl ( G ) example . The boundary function is defined as a map , from an i-simplex to an ( i− 1 ) -simplex , as the sum of its ( i− 1 ) -dimensional faces . A boundary function sample is shown in Figure 4 . In algebraic topology , a k-chain is a combination of k-simplices ( sometimes symbolized as a linear combination of simplices that compose the chain ) . The boundary of a k-chain is a ( k−1 ) chain . It is the linear and signed combination of chain element boundary simplices . The space of i-chains is denoted by Ci ( K ) . There are two special cases of chains that will be useful to define homology group : • Closed chain or i-cycle : i-chain with empty boundary . An i-chain c is an i-cycle if and only if ∂ic = 0 , i.e . c ∈ ker ( ∂i ) . This subspace of Ci ( K ) is denoted as Zi ( K ) . • Exact chain or i-boundary : An i-chain c is an i-boundary if there exists an ( i+ 1 ) -chain d such that c = ∂i+1 ( d ) , i.e . c ∈ im ( ∂ i+1 ) . This subspace of Ci ( K ) , the set of all such i-boundaries forms , is denoted by Bi ( K ) . Now , if we consider i-cycles not bounding an ( i+1 ) -simplicial complex , this is the definition of an i-th homology group of the simplicial complex K. The precise definition is the quotient group of Bi ( K ) module Zi ( K ) ( i.e . Bi ( K ) /Zi ( K ) , see Supplementary Material ) . The number of non equivalent i-cycles ( Figure 5 ) is the dimension of the homology group Hi ( K ) , also named Betti numbers . It is possible to compare two PDs using specific distances ( Wasserstein and Bottleneck ) . To efficiently perform this operation , due to the size of these diagrams , it is sometimes necessary to simplify them by means of a discretization process ( such as Weighted Silhouette and Heat vectorizations ) . 3 RELATED WORK . Algebraic Topology and Machine Learning The use of Algebraic Topology in the fields of data science and machine learning has been gaining momentum in recent years ( see Carlsson ( 2009 ) ) . Specifically in the case of neural networks , some works have applied topology for improving the training procedure of the models Hofer et al . ( 2020 ) ; Clough et al . ( 2020 ) , or pruning the model afterwards Watanabe & Yamana ( 2020b ) . Other works have focused on analyzing the capacity of neural networks Guss & Salakhutdinov ( 2018a ) ; Rieck et al . ( 2019b ) ; Konuk & Smith ( 2019 ) or the complexity of input data Konuk & Smith ( 2019 ) . Furthermore , recent works have provided topological analysis of the decision boundaries of classifiers based on PH and Betti numbers Ramamurthy et al . ( 2019 ) ; Naitzat et al . ( 2020 ) . Graph and topological representations of neural networks Gebhart et al . ( 2019 ) suggest a method for computing the PH over the graphical activation neural networks , while Watanabe & Yamana ( 2020a ) propose representing neural networks via simplicial complexes based on Taylor decomposition , from which one can compute the PH . Chowdhury et al . ( 2019 ) show that directed homology can be used to represent MLPs . Anonymous ( 2021 ) concurrently show neural networks , when represented as directed , acyclic graphs , can be associated to an Algebraic Topology object , specifically to a directed flag complex . By computing the PH diagram , one can effectively characterize neural networks , and even compute distances between two given neural networks , which can be used to measure their similarity . This is unlike other works Corneanu et al . ( 2019 ) ; Guss & Salakhutdinov ( 2018b ) approximating neural networks representations with regard to the input space . Relevant to our work , in Rieck et al . ( 2019b ) authors propose a complexity measure of neural networks based on persistent homology . However , we will see that their representation does not fulfill our requirements in Section 4 . Estimating the generalization and studying the learning process We are , though , specifically interested in the use of PH for analyzing the learning process , especially with the goal of estimating generalization . In this regard , the literature is perhaps more limited . Jiang et al . ( 2019 ) work on understanding what drives generalization in deep networks from a Bayesian of view . Neyshabur et al . ( 2017 ) study the generalization gap prediction from the training data and network parameters using a margin distribution , which are the distances of training points to the decision boundary . In Li et al . ( 2020 ) , authors propose an alternative to cross-validation for model selection based on training once on the whole train set , without any data split , deriving a validation set with data augmentation . Corneanu et al . ( 2020 ) try to estimate the performance gap between training and testing using PH measures . They claim . However , one can observe some caveats . The first one is that their regression fitted to predict the test error has a considerably high error , making it not usable in practice . The second caveat is that for fitting the regression one needs at least part of the sequestered testing set . In this work , motivated by the interest of having a better understanding of whether it would be possible to estimate the generalization of neural networks without a holdout set , we suggest using the topological characterization and distances concurrently proposed in Anonymous ( 2021 ) but , crucially , measured between consecutive weight updates . We will show that the evolution of this distance is similar to the one of the validation accuracy . Unlike Li et al . ( 2020 ) , we do not use any data at all . Unlike Corneanu et al . ( 2020 ) , we do not build a statistical or machine learning model ( linear regression ) for predicting the testing error . Instead , we propose a new measure , and we empirically show that it highly correlates with the validation accuracy . Note that in this work we do not work with any input data and activations , but with the parameters of the neural network themselves . The code and outputs are fully available in the Supplementary Material under a MIT License .

Summary: This paper proposes a topological data analysis approach to investigate the inner workings of a deep network. In particular, the generalization performance of a deep network is measured without a holdout set by analyzing the topological change of the network during training. Multiple experiments conducted over various datasets demonstrate that the method performs reasonably for various learning rates, batch sizes, and dropout probabilities.