Delaunay Component Analysis for Evaluation of Data Representations

1 INTRODUCTION Quality evaluation of learned data representations is gaining attention in the machine learning community due to the booming development of representation learning techniques . One common approach is to assess representation quality based on their performance on a pre-designed downstream task ( Bevilacqua et al. , 2021 ; Li et al. , 2020 ) . Typically , a classification problem is used to evaluate either the ability of a model to recover labels of raw inputs , or the transferability of their representations to other domains , as done in state-of-the-art unsupervised representation learning methods ( Chen et al. , 2020b ; Ermolov et al. , 2021 ) . However , in many scenarios such straightforward downstream classification task can not be defined , for instance , because it does not represent the nature of the application or due to the scarcity of labeled data as often occurs in robotics ( Chamzas et al. , 2021 ; Lippi et al. , 2020 ) . In these scenarios , representations are commonly evaluated on hand-crafted downstream tasks , e.g. , specific robotics tasks . However , these are time consuming to design , and potentially also bias evaluation procedures and consequentially hinder generalization of representations across different tasks . Recently , more general evaluation methods such as Geometry Score ( GS ) ( Khrulkov & Oseledets , 2018 ) , Improved Precision and Recall ( IPR ) ( Kynkäänniemi et al. , 2019 ) and Geometric Component Analysis ( GeomCA ) ( Poklukar et al. , 2021 ) have been proposed . ∗Correspondence to Petra Poklukar , poklukar @ kth.se . These methods analyze global geometric and topological properties of representation spaces instead of relying on specific pre-designed downstream tasks . These works assume that a set of evaluation representations E is of high quality if it closely mimics the structure of the true data manifold captured by a reference set of representationsR . This reasoning implies thatR andE must have similar geometric and topological properties , such as connected components , their number and size , which are extracted from various approximations of the data manifolds corresponding to R and E. For example , GS and GeomCA estimate the manifolds using simplicial complexes and proximity graphs , respectively , while IPR leverages a k-nearest neighbour ( kNN ) based approximation . However , as we discuss in Section 2 , neither of these approaches provides a reliable manifold estimate in complex arrangements of R and E , for instance , when points form clusters of varying shape and density as well as in the presence of outliers . Moreover , the informativeness of the scores introduced by these methods is limited . In this work , we address the impediments of evaluation of learned representations arising from poor manifold approximations by relying on a more natural estimate using Delaunay graphs . As seen in Figure 1 , edges ( solid lines ) in a Delaunay graph connect spacial neighbours and thus vary in length . In this way , they naturally capture local changes in the density of the representation space and thus more reliably detect outliers , all without depending on hyperparameters . We propose an evaluation framework called Delaunay Component Analysis ( DCA ) which builds the Delaunay graph on the union R ∪ E , extracts its connected components , and applies existing geometric evaluation scores to analyze them . We experimentally validate DCA on a variety of setups by evaluating contrastive representations ( Section 4.1 ) , generative models ( Section 4.2 ) and supervised models ( Section 4.3 ) . Furthermore , we exploit the natural neighbourhood structure of Delaunay graphs to evaluate a single query representation . This is crucial in applications with continuous stream of data , for example , in interactions of an intelligent agent with the environment or in the assessment of the visual quality of individual images generated by a generative model as also explored by Kynkäänniemi et al . ( 2019 ) . In these cases , existing representation spaces need to be updated either by embedding novel query points or distinguishing them from previously seen ones . In Delaunay graphs , this translates to analyzing newly added edges to the given query point . We demonstrate various possible analyses in Section 4 using aforementioned experimental setups . 2 STATE-OF-THE-ART METHODS AND THEIR LIMITATIONS . We review the state-of-the-art methods for evaluation of learned data representations , namely GS ( Khrulkov & Oseledets , 2018 ) , IPR ( Kynkäänniemi et al. , 2019 ) and GeomCA ( Poklukar et al. , 2021 ) , which compare topological and geometrical properties of an evaluation set E with a reference set R representing the true underlying data manifold . We discuss differences in their manifold approximations , visualized in Figure 2 , as well as informativeness of their scores . The pioneering method in this area , GS , constructs witness simplicial complexes on randomly sampled subsets of R and E ( panel ( a ) ) , and compares their connected components using tools of computational topology ( Zomorodian & Carlsson , 2004 ) . The result is an average over several iterations summarized either in the form of histograms , which are cumbersome for quantitative comparison , or as a single un-normalized score , which is in many cases not sufficiently informative . Moreover , GS depends on four hyperparameters , which can be difficult to understand and tuned by practitioners unfamiliar with computational topology . In contrast , IPR obtains manifold approximations by enlarging each point in R ( or E ) with a hypersphere of radius equal to the distance to its kNN in that set ( panel ( b ) ) . It defines two scores : precision PI which counts the number of E points contained on the approximated R manifold , and vice versa for recall RI . While the method depends only on one hyperparameter k , it is highly affected by outliers which induce overly large spheres and dense areas of the space which yield too conservative coverage as visualized in panel ( b ) . Moreover , it is the only method expecting R and E to be of equal size , thus , often requiring subsampling of one of them . Both GS and IPR have been primarily developed to assess generative adversarial networks ( GANs ) ( Goodfellow et al. , 2014 ) . The most recent and generally applicable method , GeomCA , extends GS and IPR in two ways : i ) it provides functionality to analyze individual connected components of the approximated manifold , hence enabling one to examine local areas of representation spaces where inconsistencies arise , and ii ) characterizes the captured geometry in four global scores : precisionPG and recallRG , which are similar to IPR , as well as network consistency cG and network quality qG , which are also the basis of the GeomCA local evaluation scores ( see Section 3 for recap ) . GeomCA estimates the manifolds using ε-proximity graphs where the hyperparameter ε defines the maximum distance between two points connected by an edge and is estimated from distances among points in R. However , in practice , representations often form clusters of different shape and density which makes it impossible to select a single value of ε that adequately captures such variations ( see examples in panel ( c ) ) . Moreover , GeomCA reduces the number of points by performing geometric sparsification that extracts a subset of points from each of R and E being pairwise at least distance δ apart , where δ is estimated from ε . While the authors argue that this step does not affect their scores as it preserves the topology , we show that it nevertheless can bias the manifold estimation ofR and E. An example is illustrated in the bottom component of panels ( c ) and ( d ) , where the sparsification removes a large portion of the denseE subset and artificially increases the precision . We demonstrate the occurrence of this scenario in the evaluation of a GAN model in Section 4.2 . Our DCA framework utilizes Delaunay graphs ( panel ( e ) ) to address the discussed limitations in manifold approximations of the existing methods , and employs evaluation scores introduced by Poklukar et al . ( 2021 ) to maximize its informativeness . Moreover , it extends the existing methods by additionally providing a general framework for quality evaluation of single query representations . 3 METHOD . We propose Delaunay Component Analysis ( DCA ) algorithm for evaluation of data representations consisting of three parts : i ) manifold approximation which approximates the Delaunay graph on the given sets of representations , ii ) component distillation which distills the graph into connected components , and lastly iii ) component evaluation which outputs the evaluation scores summarizing the geometry of the data . We provide an outline of our DCA framework in Algorithm 1 found in Appendix A . Moreover , by exploiting the nature of Delaunay graphs , DCA can be efficiently implemented ( Section 3.2 ) and extended for evaluation of individual representations ( Section 3.1 ) . Phase 1 : Manifold approximation As mentioned in Section 1 , the unique property of Delaunay graphs is the definition of neighbouring points that are connected by an edge . For example , in an ε-graph , two points are adjacent if they are at most ε distance apart . In a kNN graph , a point is connected to all points that are closer than the k-th smallest distance of that point to any other . In contrast , in a Delaunay graph ( Figure 1 ) , a point is adjacent to any point that is its spatial neighbour , regardless of the actual distance between them or the number of its other spatial neighbours . We refer to such spatial neighbours as natural neighbours of a point and rigorously define them through Voronoi cells ( depicted as dashed lines in Figure 1 ) in the following . Definition 3.1 Given a set W ⊂ RN we define the Voronoi cell Cell ( z ) associated to a point z ∈ W as the set of points in RN for which z is the closest among W : Cell ( z ) = { x ∈ RN ∣∣ ∥x− z∥ ≤ ∥x− zi∥∀zi ∈W } . The Delaunay graph GD ( W ) = ( V , E ) built on the set V = W is then defined by connecting points whose Voronoi cells intersect , i.e. , E = { ( zi , zj ) ∈W ×W | Cell ( zi ) ∩ Cell ( zj ) ̸= ∅ , zi ̸= zj } . We consider a reference set R = { zi } nRi=1 ⊂ RN and evaluation set E = { zi } nE i=1 ⊂ RN of data representations with R ̸= E , and approximate the Delaunay graph GD = GD ( R ∪ E ) . By Definition 3.1 , edges in GD correspond to points on the boundary of Voronoi cells which are obtained using Monte Carlo based sampling algorithm presented by Polianskii & Pokorny ( 2019 ) ( see Appendix A for further details ) . The process , visualized in Figure 3 , is based on sampling rays originating from each z ∈ R ∪ E and finding their intersection with the boundary of its Voronoi cell Cell ( z ) . This allows to reconstruct GD via subgraph approximation , and can be additionally exploited for memory optimizations which we present in Section 3.2 . Due to the sampling , the number of found edges directly depends on the number T of rays sampled from each z . However , as we show in ablation studies in Appendix B.1 and B.2 , our evaluation framework is stable with respect to the variations in T . Next , we discuss the distillation of GD into connected components . Phase 2 : Component distillation Since edges in GD are obtained among natural neighbours , GD consists of a single connected component uniting regions of points formed in different densities or shape . These can be distinguished by removing large edges ( depicted with transparent edges in Figure 1 ) , or equivalently , by finding clusters of points having similar natural neighbours ( depicted with opaque edges in Figure 1 ) . We distill GD into connected components by adapting the stateof-the-art hierarchical clustering algorithm HDBSCAN ( McInnes et al. , 2017 ) summarized in Appendix A . We apply the part of HDBSCAN that extracts connected components { Gi } from the minimum spanning tree1 MST ( GD ) . We emphasise that applying HDBSCAN directly on MST ( GD ) efficiently bypasses the calculation of a complete pairwise distance matrix2 of size nR + nE which becomes a computational burden for large R and E sets . Such calculation is an integral part of HDBSCAN performed to reduce the sensitivity of the method to noise or outliers which can be omitted in case of Delaunay graphs due their natural neighbourhood structure ( see Appendix A for further details ) . In this way , our modification of HDBSCAN inherits only one of its original hyperparameters , the minimum cluster size mcs parameter , determining the minimum number of points needed for a set of points to form a cluster . This parameter is intuitive to tune and can be flexibly adjusted depending on the nature of the application . In our ablation studies reported in Appendix B.1 and B.2 , we show that DCA is stable with respect to variations in mcs . At the end of this phase , we obtain the distilled Delaunay graph GDD = ⊔ i Gi of GD which we denote simply by G when no confusion arises . Lastly , we analyze the components of G as summarized below . Phase 3 : Component evaluation We analyze the connected components Gi of G using local and global evaluation scores introduced by Poklukar et al . ( 2021 ) which we recap below . Following their notation , we denote by |G|V and |G|E the cardinalities of the vertex and edge sets of a graph G = ( V , E ) , respectively , and by GQ = ( V|Q , E|Q×Q ) ⊂ G its restriction to a set Q ⊂ V . We start by introducing the two local scores : component consistency and quality . Intuitively , a component Gi attains high consistency if it is equally represented by points from R and E , i.e. , if |GRi |V ≈ |GEi |V , where GRi , GEi denote the restrictions of Gi to R and E. Similarly , Gi obtains a high quality score if points from R and E are also well mixed which is measured in terms of edges connecting R and E , i.e. , the points are geometrically well aligned if the number of homogeneous edges among points in each of the sets , |GRi |E and |GEi |E , is small compared to the number of heterogeneous edges connecting representations from R and E. This is rigorously defined as follows : Definition 3.2 ( Local Evaluation Scores , Poklukar et al . ( 2021 ) ) Consistency c and quality q of a component Gi ⊂ G are defined as the ratios c ( Gi ) = 1− | |GRi |V − |GEi |V | |Gi|V and q ( Gi ) = { 1− ( |G R i |E+|G E i |E ) |Gi|E if |Gi|E ≥ 1 , 0 otherwise , ( 1 ) 1Minimum spanning tree of a graph is a tree minimizing the total edge length and connecting all vertices . 2The matrix represents mutual reachability distance calculated among each pair of points with respect to the minimum samples parameter . respectively . Moreover , Gi is called consistent if c ( Gi ) > ηc and of high-quality if q ( Gi ) > ηq for given thresholds ηc , ηq ∈ [ 0 , 1 ) ⊂ R. A consistent component of high-quality as determined by ηc , ηq is called a fundamental component . The union of fundamental components is denoted by F = F ( G , ηc , ηq ) and indexed by the subscript f , i.e. , we write Gf ∈ F . The thresholds ηc , ηq are designed to enable a flexible definition of fundamental components and are assumed to be set depending on the application and available prior knowledge . By examining the proportion of R and E points contained in F , we obtain the first two global scores : precision and recall , respectively . Two more global scores , network consistency and network quality , used to measure global imbalances and misalignment betweenR andE , can be simply derived by extending Definition 3.2 to the entire graph G. In summary , we consider the following global evaluation scores : Definition 3.3 ( Global Evaluation Scores , Poklukar et al . ( 2021 ) ) We define network consistency c ( G ) and network quality q ( G ) as in Definition 3.2 , as well as precision P and recallR as P = |F E |V |GE |V and R = |F R|V |GR|V , ( 2 ) respectively , where FR , FE denote the restrictions of F = F ( G , ηc , ηq ) to the sets R and E .

Summary: This paper explores a new method (DCA) to evaluate the quality of learned data representations. The goal of this paper is to score representations from sets R (reference) and E (evaluation) using geometric and topological properties (like connected components) of the representations spaces. To put simply, if the local geometry and overall topology of both R and E are similar, then the representations are considered consistent and aligned (hence yielding overall high precision and recall).   The authors primarily compare with 3 methods: GS, IPR and GCA (including a sparse variant) and argue that in scenarios of prevalent outliers and varying component densities of the representation spaces, such methods typically yield non-informative and suboptimal scores to compare representations. To that aid, the main method proposed in this paper proposes to alleviate these drawbacks and comprises of 3 steps: 1.) manifold estimation, 2.) component distillation 3.) component evaluation.   The main contribution of this paper is the use of Delaunay graphs in step (1.) as opposed to eps-neighbor or knn graphs as is done in prior methods. The authors claim that this choice of graph construction along with the distillation procedure leads to identifying consistent and high quality connected components leading to more stable results. In addition the authors also propose a ‘out-of-sample’ variant of their method to handle individual queries.  The authors demonstrate their metric in three different setups: (A.) Contrastive Learning on the synthetic Chamzas et al. (2021) images dataset (B.) Style GAN (C.) Representation space of VGG16. Results, especially on (A.) show that the proposed evaluation metric is better suited with accurate recognitions of mode-collapse and mode discovery.