Topologically Regularized Data Embeddings

1 INTRODUCTION . Motivation Modern data often arrives in complex forms that complicate their analysis . For example , high-dimensional data can not be visualized directly , whereas relational data such as graphs lack the natural vectorized structure required by various machine learning models ( Bhagat et al. , 2011 ; Kazemi & Poole , 2018 ; Goyal & Ferrara , 2018 ) . Representation learning aims to derive mathematically and computationally convenient representations to process and learn from such data . However , obtaining an effective representation is often challenging , for example , due to the accumulation of noise in high-dimensional biological expression data ( Vandaele et al. , 2021b ) . In other examples such as community detection in social networks , graph embeddings struggle to clearly separate communities with interconnections between them . In such cases , expert prior knowledge of the topological model may improve learning from , visualizing , and interpreting the data . Unfortunately , a general tool for incorporating prior topological knowledge in representation learning is lacking . In this paper , we introduce such tool under the name of topological regularization . Here , we build on the recently developed differentiation frameworks for optimizing data to capture topological properties of interest ( Brüel-Gabrielsson et al. , 2020 ; Solomon et al. , 2021 ; Carriere et al. , 2021 ) . Unfortunately , such topological optimization has been poorly studied within the context of representation learning . For example , the used topological losses are indifferent to any structure other than topological , such as neighborhoods , which may be useful for learning . Therefore , topological optimization often destructs natural and informative properties of the data in favor of the topological loss . Our proposed method of topological regularization effectively resolves this by learning an embedding representation that incorporates the topological prior . As we will see in this paper , these priors can be directly postulated through topological loss functions . For example , if the prior is that the data lies on a circular model , we design a loss function that is lower whenever a more prominent cycle is present in the embedding . By extending the previously suggested topological losses to fit a wider set of models , we show that topological regularization effectively embeds data according to a variety of topological priors , ranging from clusters , cycles , and flares , to any combination of these . Related Work Certain methods that incorporate topological information into representation learning have already been developed . For example , Deep Embedded Clustering ( Xie et al. , 2016 ) simultaneously learns feature representations and cluster assignments using deep neural networks . Constrained embeddings of Euclidean data on spheres have also been studied by Bai et al . ( 2015 ) . However , such methods often require extensive development for one particular kind of input data and topological model . Contrary to this , incorporating topological optimization into representation learning provides a simple yet versatile approach towards combining any data embedding method with topological priors , that generalizes well to any input data as long as the output is a point cloud . Topological autoencoders ( Moor et al. , 2020 ) already combine topological optimization with a data embedding procedure . The main difference here is that the topological information used for optimization is obtained from the original high-dimensional data , and not passed as a prior . While this may sound as a major advantage—and certainly can be as shown by Moor et al . ( 2020 ) — obtaining such topological information heavily relies on distances between observations , which are often meaningless and unstable in high dimensions ( Aggarwal et al. , 2001 ) . Furthermore , certain constructions such as the α-filtration obtained from the Delaunay triangulation—which we will use extensively and is further discussed in Appendix A—are expensive to obtain from high-dimensional data ( Cignoni et al. , 1998 ) , and are therefore best computed from the low-dimensional embedding . Our work builds on a series of recent papers ( Brüel-Gabrielsson et al. , 2020 ; Solomon et al. , 2021 ; Carriere et al. , 2021 ) , which showed that topological optimization is possible in various settings and developed the mathematical foundation thereof . However , studying the use of topological optimization for data embedding and visualization applications , as well as the new losses we develop therefore and the insights we derive from them in this paper , are to best of our knowledge novel . Contributions We include a sufficient background on persistent homology—the method behind topological optimization—in Appendix A ( note that all of its concepts important for this paper are summarized in Figure 1 ) . We summarize the previous idea behind topological optimization of point clouds ( Section 2.1 ) . We introduce a new set of losses to model a wider variety of shapes in a natural manner ( Section 2.2 ) . We show how these can be used to topologically regularize embedding methods for which the output is a point cloud ( Equation ( 1 ) ) . We include experiments on synthetic and real data that show the usefulness and versatility of topological regularization , and provide additional insights into the performance of data embedding methods ( Section 3 & Appendix B ) . We discuss open problems in topological representation learning and conclude on our work ( Section 4 ) . 2 METHODS . The main purpose of this paper is to present a method to incorporate prior topological knowledge in a point cloud embedding E ( dimensionality reduction , graph embedding , . . . ) of a data set X . As will become clear below , these topological priors can be directly postulated through topological loss functions Ltop . Then , the goal is to find an embedding that that minimizes a total loss Ltot ( E , X ) : = Lemb ( E , X ) + λtopLtop ( E ) , ( 1 ) where Lemb is a loss that aims to preserve structural attributes of the original data , and λtop > 0 controls the strength of topological regularization . Note that , X itself is not required to be a point cloud , or reside in the same space as E , which is especially useful for representation learning . In this section , we mainly focus on topological optimization of point clouds , that is , the loss Ltop . The basic idea behind this recently introduced method—as presented by Brüel-Gabrielsson et al . ( 2020 ) —is illustrated in Section 2.1 . We show that direct topological optimization may neglect important structural information such as neighborhoods , which can effectively be resolved through ( 1 ) . Hence , as we will also see in Section 3 , while representation learning may benefit from topological losses for incorporating prior topological knowledge , topological optimization itself may also benefit from other structural losses to represent the topological prior in a more truthful manner . Nevertheless , some topological models remain difficult to naturally represent through topological optimization . Therefore , we introduce a new set of topological losses , and provide an overview of how different models can be postulated through them in Section 2.2 . Experiments with and comparisons to topological regularization of embeddings through ( 1 ) will be presented in Section 3 . 2.1 BACKGROUND ON TOPOLOGICAL OPTIMIZATION OF POINT CLOUDS . Topological optimization is performed through a topological loss function evaluated on one or more persistence diagrams ( Barannikov , 1994 ; Carlsson , 2009 ) . These diagrams—obtained through persistent homology as formally discussed in Appendix A—summarize all from the finest to coarsest topological holes ( connected components , cycles , voids , . . . ) in the data , as illustrated in Figure 1 . While methods that learn from persistent homology are now both well developed and diverse ( Pun et al. , 2018 ) , optimizing the data representation for the persistent homology thereof only gained recent attention ( Brüel-Gabrielsson et al. , 2020 ; Solomon et al. , 2021 ; Carriere et al. , 2021 ) . Persistent homology has a rather abstract mathematical foundation within algebraic topology ( Hatcher , 2002 ) , and its computation is inherently combinatorial ( Barannikov , 1994 ; Zomorodian & Carlsson , 2005 ) . This complicates working with usual derivatives for optimization . To accommodate for this , topological optimization makes use of Clarke subderivatives ( Clarke , 1990 ) , whose applicability to persistence builds on arguments from o-minimal geometry ( van den Dries , 1998 ; Carriere et al. , 2021 ) . Fortunately , thanks to the recent work of Brüel-Gabrielsson et al . ( 2020 ) and Carriere et al . ( 2021 ) , powerful tools for topological optimization have been developed for software libraries such as PyTorch and TensorFlow , allowing their application without deeper knowledge of these subjects . Topological optimization optimizes the data representation with respect to topological information summarized by its persistence diagram ( s ) D. We will use the approach by Brüel-Gabrielsson et al . ( 2020 ) , where ( birth , death ) tuples ( b1 , d1 ) , ( b2 , d2 ) , . . . , ( b|D| , d|D| ) in D are first ordered by decreasing persistence dk − bk . The points ( b , ∞ ) , usually plotted on top of the diagram such as in Figure 1b , form the essential part of D. The points with finite coordinates form the regular part of D. In case of point clouds , one and only one topological hole , i.e. , a connected component born at time α − 0 , will always persist indefinitely . Other gaps and holes will eventually be filled ( Figure 1 ) . Thus , we only optimize for the regular part in this paper . This is done through a topological loss function , which for a choice of i ≤ j ( which , along with the dimension of topological hole , will specify our topological prior as we will see below ) and a function g : R2 → R , is defined as Ltop ( D ) : = j∑ k=i , dk < ∞ g ( bk , dk ) , where d1 − b1 ≥ d2 − b2 ≥ . . . . ( 2 ) It turns out that for many useful definitions of g , Ltop ( D ) has a well-defined Clarke subdifferential with respect to the parameters defining the filtration from which the persistence diagram D is ob- tained . In this paper , we will consistently use the α-filtration as shown in Figure 1a ( see Appendix A for its formal definition ) , and these parameters are entire point clouds ( in this paper embeddings ) E ∈ ( Rd ) n of size n in the d-dimensional Euclidean space . Ltop ( D ) can then be easily optimized with respect to these parameters through stochastic subgradient algorithms ( Carriere et al. , 2021 ) . As it directly measures the prominence of topological holes , we let g : R2 → R : ( b , d ) 7→ µ ( d− b ) be proportional to the persistence function . By ordering the points by persistence , Ltop is a function of persistence , i.e. , it is invariant to permutations of the points in D ( Carriere et al. , 2021 ) . The factor of proportionality µ ∈ { 1 , −1 } indicates whether we want to minimize ( µ = 1 ) or maximize ( µ = −1 ) persistence , i.e , the prominence of topological holes . Thus , µ determines whether more clusters , cycles , . . . , should be present ( µ = −1 ) or missing ( µ = 1 ) . The loss ( 2 ) then reduces to Ltop ( E ) : = Ltop ( D ) = µ j∑ k=i , dk < ∞ ( dk − bk ) , where d1 − b1 ≥ d2 − b2 ≥ . . . . ( 3 ) Here , the data matrix E ( in this paper the embedding ) defines the diagram D through persistent homology of the α-filtration of E , and a persistence ( topological hole ) dimension to optimize for . For example , consider ( 3 ) with i = 2 , j = ∞ , µ = 1 , restricted to 0-dimensional persistence ( measuring the prominence of connected components ) of the α-filtration . Figure 2 shows the data from Figure 1 optimized for this loss function for various epochs . The optimized point cloud quickly resembles a single connected component for smaller numbers of epochs . This is the single goal of the current loss , which neglects all other structural structural properties of the data such as its underlying cycles ( e.g. , the circular hole in the ‘ R ’ ) or local neighborhoods . Larger numbers of epochs mainly affect the scale of the data . While this scale has an absolute effect on the total persistence , thus , the loss , the point cloud visually represents a single connected topological component equally well . We also observe that while local neighborhoods are preserved well during the first epochs simply by nature of topological optimization , they are increasingly distorted for a larger number of epochs .

Summary: The paper develops a way to construct low-dimensional embeddings that combine an embedding cost (e.g. PCA, t-SNE, UMAP, etc.) with the topological cost, where the latter forces the embedding to have a particular topological structure as represented by the persistent homology. As an example, one can construct a data embedding and force it to be a circle, or a figure-8, or have three clusters.  The paper is based on a series of several 2020-2021 papers, with which I am unfamiliar, so it was difficult for me to judge on the novelty. I found the paper interesting, and the method seems to work well. My biggest criticism is that the goal of the method appears somewhat artificial: low-dimensional embeddings are typically done for data exploration, but one does not want to enforce any particular structure for the purposes of data exploration, and the topological structure is typically not known a priori. That's why I am giving a borderline score.