Title: Topological Parallax: A Geometric Specification for Deep Perception Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: topological data analysis

persistent homology

convexity

AI safety

interpolation
Abstract: For safety and robustness of AI systems, we introduce _topological parallax_ as a 
theoretical and computational tool that compares a trained model to a reference dataset to determine whether they have similar multiscale geometric structure. 

Our proofs and examples show that this geometric similarity between dataset and model is essential 
to trustworthy interpolation and perturbation, and we conjecture that this new concept will add value to the current debate regarding the unclear relationship between "overfitting"' and "generalization'' in applications of deep-learning. 

In typical deep-learning applications, an explicit geometric description of the model is
impossible, but parallax can estimate topological features (components, cycles, voids, etc.)
in the model by examining the effect on the Rips complex of geodesic distortions using the reference dataset.
Thus, parallax indicates whether the model shares similar multiscale geometric features with the dataset.

Parallax presents theoretically via topological data analysis [TDA] as a bi-filtered persistence module,
and the key properties of this module are stable under perturbation of the reference dataset.

Topological Parallax: A Geometric Specification for Deep Perception Models

Abstract

For safety and robustness of AI systems, we introduce topological parallax as a theoretical and computational tool that compares a trained model to a reference dataset to determine whether they have similar multiscale geometric structure.

Our proofs and examples show that this geometric similarity between dataset and model is essential to trustworthy interpolation and perturbation, and we conjecture that this new concept will add value to the current debate regarding the unclear relationship between 'overfitting' and 'generalization' in applications of deeplearning.

In typical DNN applications, an explicit geometric description of the model is impossible, but parallax can estimate topological features (components, cycles, voids, etc.) in the model by examining the effect on the Rips complex of geodesic distortions using the reference dataset. Thus, parallax indicates whether the model shares similar multiscale geometric features with the dataset.

Parallax presents theoretically via topological data analysis [TDA] as a bi-filtered persistence module, and the key properties of this module are stable under perturbation of the reference dataset.

Michael J. Catanzaro

Geometric Data Analytics, Inc. 343 W. Main Street Durham, NC 27701 USA

Nirav Patel

1 Introduction

Suppose X is a finite subset of V = R n with the Euclidean metric. 1 In data science-particularly in applications of DNNs-we often encounter the situation where X is a dataset, and some opaque algorithm has produced a trained model k : V →{ 0 , 1 } , where k ( x ) = 1 for all x ∈ X . This defines the model as a set of accepted inputs K = { x : k ( x ) = (1) } ⊂ V , which has no available description beyond evaluation of the perception function k on samples.

Our main contribution in this paper is a method we call topological parallax to estimate the multiscale geometry of K from the persistent homology ([14, 37]) of X , in a situation where K does not have an explicit description. 2 This method provides meaningful geometric information about K through a simple computational approach that can be applied to any perception model k . We prove that the resulting criterion of homological matching satisfies a stability property. We propose homological matching via parallax as a geometric specification that could be applied to many machine-learning systems. The measurement of homological matching also admits a back-propagation scheme, which could be used to improve the geometric similarity between the model K and the dataset X .

Because of the generality of Definition 1.1, it may be that V represents any layer of a neural network, X represents any dataset mapped into that layer, and K represents an activated region in that layer. For example, the 'neural collapse' concept from Papyan et al. [29] can be seen as a special case of this specification, because the conception from [29] is that the dataset X becomes a tight blob in the penultimate layer, and the penultimate model is simply a Voronoi region K surrounding that blob.

1.1 Assumptions and Motivation

As discussed by Belkin [5], DNNs usually achieve high statistical accuracy, but some resulting models are better than others at capturing patterns in the dataset. Despite having perfect statistical accuracy on the dataset X , fundamental questions arise about the model K : 'Is it safe to deploy? Is it trustworthy? Is it a good model?' To broadly paraphrase [5], it used to be good practice to tell data analysts not to overfit their data, because overfit models were 'bad' due to poor generalization; however, essentially every DNN fits the training data perfectly, so it is not clear what distinguishes 'good' models from 'bad.' We suggest that a model K is 'good' if the geometry of K matches the geometry of X . Consider the two example models in Figure 1. Although both models achieve perfect statistical accuracy, only one of them appears to have learned the geometric structure of the dataset. This suggestion is likely intuitive to many ML engineers, but the subtlety lies in the implicit assumptions often made about the nature of the dataset and the available models. We assess this matching in a way that is independent of the architecture that produced k , and that makes very few assumptions about X and K , as encapsulated by Definition 1.1, which are assumed henceforth.

Definition 1.1 (Datasets and Models) . Suppose that V is a geodesic space. We define a model K ⊂ V to be the closure of an open set, K = K ◦ ⊂ V , colloquially known as a 'solid. ' For any finite dataset X ⊂ V , we consider the collection of all models for which X is contained in the interior of the model, X ⊂ K ◦ ; let M ( X ) := { K ⊂ V : X ⊂ K ◦ , K ◦ = K } . With subset inclusions as morphisms, M ( X ) is a small category. For any K ⊂ V with K = K ◦ , let M ∗ ( K ) := { X ⊂ V : X ⊂ K ◦ , # X < ∞} . With functions of finite sets as morphisms, M ∗ ( K ) is a small category. Of course, X ∈ M ∗ ( K ) ⇔ K ∈ M ( X ) . Note that V ∈ M ( X ) .

Note that we make no assumptions whatsoever about the architecture or training method that yielded K . With such a broad definition of X and K , we need a notion of 'geometry' with very few assumptions. We use standard topological notation [23, 17, 14].

Definition 1.2 (Void) . Given a set K ⊂ V , an interpolative void in K is a bounded open set in the complement of K , Ω ⊂ K c , such that there exists a pair of points x, y ∈ K for which all V -geodesics pass through Ω . If V = R n , this means Ω intersects the convex hull of K .

DatasetX

PersDiag of R

ModelK:Decision Tree

notx

X

K

K

4.000

3.000

2.000

1.000

0.000

1

Q

0

1

2

3

4

Birth

40

60

λ

sup

=

∞

λ

hi

=

∞

λ

sup

= 7

.

1

λ

hi

=

∞

λ

sup

= 7

.

1

λ

hi

=

∞

λ

sup

= 7

.

1

λ

hi

= 30

λ

sup

= 7

.

1

λ

hi

= 20

Why focus on voids? As observed by Balestriero et al. [2], when V is high-dimensional, it is unlikely that any points in X lie in the convex hull of any others. However, it also seems unlikely that a 'good' model K of X will be merely a convex solid. If convex models were sufficient for real-world problems, DNNs would be unnecessary, and the field would have concluded with PCA, Gaussian kernels, and convex polytopes. For example, many models implicitly or explicitly rely on the so-called manifold hypothesis, which is the hope that realistic datasets X will tend to be distributed near a union of lower-dimensional manifolds immersed in V , in which case a 'good' model K would be a slight thickening of those manifolds to allow for measurement error. If there are multiple manifolds or there is nonzero curvature, such a model will have voids.

Any void in K indicates a region where K does not allow interpolation, which is where interesting geometry occurs. Voids occur if and only if some geodesic in K is strictly longer than the corresponding geodesic in V . Hence, for this article, we interpret 'geometry of K ' as the presence or absence of voids. We do not make assumptions about the geometric features of X or about the family from which K is chosen; rather, we ask that K respects the features of X , whatever they are. We propose that a 'good' model K is one whose voids represent the highly persistent features of X , in the sense of Topological Data Analysis [TDA, [14, 27]]

A model K can be 'bad' because it has too many or too few voids at various scales. For example, mismatch of voids of K and features of X would indicate that K is over-sensitive to small error or under-sensitive to large error, either of which could lead to adversarial attack. Also, numerous small-scale voids could make K incompatible with some forms of the manifold hypotheses, by obstructing the coverage of K by an atlas of local convex charts of moderate size.

1.2 Outline

Section 2 introduces our key object, the bi-graded [7] parallax complex, in Definition 2.2, which measures geodesic distortion via the Rips complex. Section 3 provides a notion of dataset perturbation

and shows that the parallax complex and its homology remain stable under those perturbations. Section 4 defines local simplicial matching and shows how parallax detects small-scale changes in the Rips complex of X in K versus V , giving a clearly interpretable scale of locality, λ lo. Section 5 defines homological matching and shows how parallax detects large-scale voids in K , giving a scale λ hi above which homological features in X are respected by voids in K . Together, these results provide an overall interpretation as a specification in Section 6, which largely achieves the goals laid out in Section 1. Section 7 provides computational approaches to computing parallax, and links to our open-source software that has many practical improvements not detailed in this paper. Section 8 illustrates the effectiveness of parallax as a specification, as demonstrated on two models using the cyclo-octane dataset [22]. Additional proofs, details, and examples are provided in the Supplementary Material appendices.

1.3 Related Work

To the best of our knowledge this is the first work to use TDA to express a desired geometric relationship that holds directly between datasets and models trained on them. There has been some work, for example the Manifold Topology Divergence of Barannikov et al. [3] or the Geometric Score of Khrulkov and Oseledets [20], which uses various TDA-based measures to quantify the difference between training data and new data generated by generative models. Quite a few other papers (see Fernández et al. [15] for a very recent example) use TDA-based constructions to infer properties of underlying data manifolds, usually under very strict sampling assumptions.

More broadly, there has been a recent explosion of work (e.g. Hensel et al. [19]) connecting TDA to ML/DL. Several works (e.g. Adams et al. [1], Bendich et al. [6]) use TDA as a feature extraction method, pre-processing more complicated data objects before running standard ML pipelines. Later works (e.g. Chen et al. [10], Demir et al. [12], Solomon et al. [32], Nigmetov and Morozov [25]) use TDA to define novel losses within ML algorithms. Note that we comment below on ways in which our notion of homological matching can be used to define a TDA-based loss. TDA has also been used (e.g. Naitzat et al. [24], Wheeler et al. [36]) to analyze the behavior of data as it passes through the layers of a DNN. Some works (e.g. Guss and Salakhutdinov [16]) assess the capacity of a specific DNN to classify datasets with specific shapes, but do not provide tools to quantify shape mismatch between model and dataset. Finally, several works (e.g. Carriere et al. [8], Papillon et al. [28]) use TDA to define novel DNN architectures, including GNNs and other higher-order combinatorial structures.

There is also a recent stream (e.g. Liu et al. [21], Wang et al. [35]) of work that builds validation and verification/falsification techniques for desired properties of DNN-trained models; these mostly focus on the mechanics of how to verify/falsify such properties, rather than attempting to define them as we do. Perhaps the closest work in this stream to ours is Dola et al. [13], which uses a prior assumption on the underlying data distribution to verify/falsify DNN properties.

2 The Parallax Bi-Complex

Let B α ( x ) denotes the closed geodesic ball of radius α around x ∈ V . For a formal edge e = ( x 0 , x 1 ) between points in X , ρ V ( e ) is the minimum radius for which B ρ V ( e ) ( x 0 ) intersects B ρ V ( e ) ( x 1 ) . Thus, 2 ρ V ( e ) is the geodesic distance between x 0 and x 1 . The Rips complex R ( X,V ) is the simplicial complex generated by these edges, as filtered by ρ V ( e ) . A chain is a formal sum of simplices in a complex [14]. More generally, for any K ∈ M ( X ) , the Rips complex R ( X,K ) and its filtration by 0 ≤ α < ∞ is defined by glyph[negationslash]

( x 0 , . . . , x d ) ∈ R d ( X,K ) α if and only if B α ( x i ) ∩ B α ( x j ) ∩ K = ∅ , ∀ 0 ≤ i < j ≤ d. (2.1)

For any chain Y ∈ R ( X,K ) , let ρ K ( Y ) = min { α : Y ∈ R ( X,K ) α } . When X and V are understood in context, we abbreviate R = R ( X,V ) for the ambient case K = V .

Lemma 2.1. If K 1 , K 2 ∈ M ( X ) with K 1 ⊂ K 2 ⊂ V , then there is a natural inclusion of filtered modules, R ( X,K 1 ) α ⊂ R ( X,K 2 ) α ⊂ R α . That is, ρ V ( Y ) ≤ ρ K 2 ( Y ) ≤ ρ K 1 ( Y ) , with the convention min ∅ = ∞ .

The previous lemma is simply because geodesic lengths in K 1 are never shorter than geodesic lengths in K 2 , and neither is shorter than geodesic lengths in V . Our approach to the question 'does the geometry of K match the geometry of X?' relies on detecting the inequality ρ V ( Y ) < ρ K ( Y ) .

Definition 2.2 (Parallax Complex) . For K ∈ M ( X ) , let P ( X,K,V ) denote the subcomplex of R defined for each real pair ( α, ε ) by

When X,K,V are understood in context, we abbreviate P = P ( X,K,V ) .

The parameter ε measures the distortion of geodesic length in K versus V . The next few lemmas are immediate consequences of the definition.

Lemma 2.3 (Parallax is Bi-Filtered) . If α ≤ α ′ , then P α,ε ⊂ P α ′ ,ε . If ε < ε ′ , then P α,ε ⊂ P α,ε ′ .

Lemma 2.4. For all α, ε , we have P α,ε ⊂ R ( X,K ) α ⊂ R α .

Let ι : P α,ε → R α denote the inclusion of complexes, and let ι ∗ : HP α,ε → HR α denote the induced homomorphism on homology [17].

Corollary 2.5 (Homology Deaths are later in Parallax) . Suppose that [ Y ] is a class in HP α 1 ,ε 1 such that the bi-transition map HP α 1 ,ε 1 → HP α 2 ,ε 2 takes [ Y ] ↦→ [0] . Then there exists t ≤ α 2 such that the transition map HR α 1 → HR t satisfies ι ∗ ([ Y ]) ↦→ [0] .

The proofs of 2.5 and 2.6 are given in the Supplemental Material.

It is sometimes useful to create single-parameter filtrations through P , parameterized by Rips radius, for the purpose of computing barcodes and persistence diagrams.

Definition 2.7 (Rips-like Path) . A Rips-like path is a filtered module L α = P α,ε ( α ) for 0 ≤ α < ∞ such that ε ( α ) is a non-decreasing function satisfying ε (0) = 0 .

A Rips-like path has homology HL and a barcode or persistence diagram. By Lemma 2.6, one Rips-like path is R ( X,K ) α = P α,α . Another is the 'inflexible' path L α = P α, 0 .

3 Perturbation

This section establishes lemmas that ensure Parallax and its consequences (notably Theorem 5.4) are stable under certain types of perturbations, which means that the parallax is reasonable in the presence of noise.

Definition 3.1 (Pointwise Perturbation) . Given X ∈ M ∗ ( V ) , a pointwise κ -perturbation is X ′ ∈ M ∗ ( V ) such that the sets X and X ′ admit a one-to-one correspondence f : X → X ′ satisfying ‖ f ( x ) -x ‖ ≤ κ . We write f : X κ ≈ X ′ .

Definition 3.2 (Pointwise K -perturbation) . Suppose f : X κ ≈ X ′ such that each ( x, f ( x )) pair is connected by a K -geodesic of length ≤ κ . We write f : X κ ≈ K X ′ .

Note that any f : X κ ≈ X ′ is an isomorphism in the category M ∗ ( V ) , and any f : X κ ≈ K X ′ is an isomorphism in the category M ∗ ( K ) . The relation κ ≈ is reflexive and symmetric, but not transitive; hence, it is a way of describing proximity but does not provide an equivalence relation. Of course, X ′ κ ≈ X implies the Hausdorff distance satisfies d H ( X,X ′ ) ≤ κ . A pointwise κ -perturbation can cause length distortions by 2 κ , as said formally in the following lemma.

Lemma 3.3 (Data Perturbation Lemma) . If f : X κ ≈ X ′ , then the Rips complexes identified via f admit a κ -interleaving

Specifically, for any edge e = ( x i , x j ) ∈ R ( X,V ) α defined by the existence of a V -geodesic of length 2 α , the edge f glyph[sharp] ( e ) = ( f ( x i ) , f ( x j )) in R ( X ′ , V ) has length 2 α ′ satisfying 2 α -2 κ ≤ 2 α ′ ≤ 2 α +2 κ .

Moreover, the same holds when replacing V with K , under the assumption X κ ≈ K X ′ .

Proof. Note that a geodesic of length 2 α corresponds with the intersection of two balls of radius α ; hence, the factor of 2. The worst-case perturbation is to move each of x i and x j by κ in opposite directions, away from each-other, along their geodesic.

Lemma 3.4 (Parallax Interleaving Lemma) . Suppose f : X κ ≈ K X ′ . Let P = P ( X,K,V ) and P ′ = P ( X ′ , K, V ) . For any α, ε , these parallax complexes admit a ( κ, 2 κ ) -interleaving

Corollary 3.5 (Parallax Interleaving Lemma, Homology Version) . Suppose f : X κ ≈ K X ′ . Let HP = HP ( X,K,V ) and HP ′ = HP ( X ′ , K, V ) . For any α, ε , these homology groups admit a

The proof of Lemma 3.4 is a worst-case distance estimate given in the Supplemental Material, and Corollary 3.5 follows functorially.

4 Sampling Density and Local Simplicial Matching

The goal 'the geometry of K should match the geometry of X ' requires that X has sufficient sampling density throughout K to express a meaningful comparison. The ideal situation would require there is a (small) scale λ for which: (1) K ◦ is homeomorphic to ⋃ x ∈ X B λ ( x ) , so that these λ balls capture the topology of K ; (2) all 'highly persistent' homological features of X are born before λ ; and (3) ⋃ x ∈ X B λ ( x ) ⊂ K , so that perturbations of size λ in the dataset X are allowed, and so that these balls can be used as local charts in K . These sampling properties may or may not be true for any particular pair ( X,K ) , but Definition 4.1 provides scales for comparison.

Definition 4.1 (Locally Simplicially Matched) . We say that X and K ∈ M ( X ) are λ -locally simplicially matched [LSM] if the subset P t, 0 ⊂ R t is an equality for all t ≤ λ . For any K ∈ M ( X ) , the first non-LSM scale realized by the Rips complex is

The last LSM scale realized by the Rips complex is

When X and K are λ -LSM, we will identify P t, 0 with R t so that H ∗ P t, 0 = H ∗ R t whenever t ≤ λ . Furthermore, locally simplicially matched implies P t,glyph[epsilon1] = R t for any glyph[epsilon1] > 0 and t < λ sup ,X ( K ) , following directly from the definition of P a,glyph[epsilon1] .

However, it may be that λ lo ,X ( K ) = 0 , if the shortest edge e ∈ R has ρ V ( e ) < ρ K ( e ) .

Corollary 4.3. For any K,K ′ ∈ M ( X ) , there is some 0 < λ such that each pair ( X,K ) and ( X,K ′ ) is λ -locally simplicially matched.

Corollary 4.4. For any X,X ′ ∈ M ∗ ( K ) , there is some 0 < λ such that each pair ( X,K ) and ( X ′ , K ) is λ -locally simplicially matched.

The next lemma provides a bound on λ sup under small data perturbations, which is a form of stability.

Lemma 4.6. Assume Euclidean V . Suppose that K ∈ M ( X ) ∩ M ( X ′ ) . If f : X κ ≈ K X ′ for κ < 1 2 λ ball ,X ( K ) then 1 2 λ ball ,X ( K ) -κ ≤ λ sup ,X ′ ( K ) .

The proof of Lemma 4.6 is a triangle-inequality argument in the Supplemental Material. The other results are immediate observations from the definitions.

5 Homological Matching

From Section 1, our purpose is to determine whether the geometry of K matches the geometry of X . In Section 4, we introduced 'local simplicial matching' as a way to compare small-scale geometry. In this section, we introduce 'homological matching' as way to compare large-scale geometry. Generally this is done by asking whether highly persistent features of X in V (as measured by HR ) are also highly persistent as features of X in K (as measured by HP ). This comparison is sensible if X and K are λ -locally simplicially matched, so that cycles can be identified between HR λ = HP λ, 0 . We phrase it algebraically in Definition 5.1, but Lemma 5.3 provides the interpretation that, among cycles born before λ , those of long persistence ( δ -λ ) in HR have even longer persistence ( ω -λ ) in HL , meaning that K has large-scale homological features corresponding to those of X .

Definition 5.1 (Homologically Matched) . For K ∈ M ( X ) , and 0 ≤ λ < λ sup ,X ( K ) < δ ≤ ω , we say that X and K are ( λ, δ, ω ) -homologically matched [HM] if the transition maps of HR and HP satisfy ker HR λ → δ ⊂ ker HP ( λ,ε 1 ) → ( ω,ε 2 ) for some 0 ≤ ε 1 ≤ ε 2 . Equivalently, if ker HR λ → δ ⊂ ker HL λ → ω for some Rips-like path L .

Definition 5.1 is guided by the Void Lemma (5.2) and the Matching Lemma (5.3).

Lemma 5.2 (Void Lemma) . Suppose K ∈ M ( X ) and 0 < λ lo ,X ( K ) . Let L be any Rips-like path. If [ Y ] ∈ HR ( X,V ) with birth b < λ lo ,X ( K ) , then the deaths c , d , e of [ Y ] in HR ( X,V ) , HR ( X,K ) , HL , respectively, satisfy c ≤ d ≤ e . Moreover, c < e implies that K has a void that disrupts the death of [ Y ] , and that void contains a ball of radius r satisfying ( π -2) r ≤ 2( d -c ) ≤ 2( e -c ) .

In particular, if the class [ Y ] has e = ∞ for the 'inflexible' path L t = P t, 0 , then K contains a void. The proof of Lemma 5.2 appears in the Supplementary Material and uses simple distance estimates.

Lemma 5.3 (Matching Lemma) . Suppose that all pairs in X have distinct lengths, and that X and K ∈ M ( X ) are ( λ, δ, ω ) -HM. Then there is a Rips-like path L for which each dot ( b, d ) in the persistence diagram of L (or bar in the barcode) with b ≤ λ < ω < d corresponds via λ -LSM to a dot ( b, c ) in the persistence diagram of R with b ≤ λ < δ < c .

Note: to check whether X and K are ( λ, δ, ω ) -HM, it suffices to check a Rips-like path through P λ, 0 and P ω, ∞ . At the other extreme, we get an overly-strict bound by checking the 'inflexible' Rips-like path L t = P t, 0 .

Because of the filtration stability of HP in Corollary 3.5, ( λ, δ, ω ) -HM is also stable to perturbation in X , as seen in Theorem 5.4.

Theorem 5.4 (Stability of Homological Matching) . Suppose that X and K are ( λ, δ, ω ) -HM. If f : X κ ≈ K X ′ such that X ′ and K are ( λ -κ ) -LSM, then ( X ′ , K ) are ( λ -κ, δ -κ, ω + κ ) -homologically matched.

The proof of Lemma 5.3 is functorial, and Theorem 5.4 is a diagram chase. Both are given in the Supplemental Material.

Definition 5.5. Given X and K ∈ M ( X ) let λ hi ,X ( K ) denote the minimum of those δ for which X,K are ( λ lo ,X ( K ) , δ, ∞ ) -HM.

6 Interpretation and Specification

Therefore, to answer our original purpose, we can assess whether a model K ∈ M ( X ) is a 'good geometric match' for X using the following procedure: (1) Regardless of K , examine the persistence diagram of R ( X,V ) to identify dots with early birth and long persistence, which TDA theory tells us (e.g. the Homology Inference Theorem [11]) should indicate genuine geometric features of X ; (2) for a model K ∈ M ( X ) , compute λ lo ,X ( K ) and λ hi ,X ( K ) ; and (3) check whether those dots are born before λ lo ,X ( K ) and die after λ hi ,X .

If we believe that the multiscale geometric patterns among the points in X is meaningful, then this procedure is essentially a specification for how a 'good' model ought to behave.

This process can give a 'bad geometric match' in various ways, for example: if step (1) does not show a clear collection of well-separated dots, then it is unlikely that X actually has computable geometry that can be captured with the Rips complex; if step (2) yields λ lo ,X ( K ) = 0 , then K has

voids between every pair of points in X , possibly due to under-sampling or over-fitting, and should not be trusted for any interpolative purpose; or if step (3) shows that the quadrant to the upper-left of ( λ lo ,X ( K ) , λ hi ,X ( K )) in the birth-death plane does not capture the desired dots of X , then K is failing to capture specific high-persistence features of X .

7 Computational Methods

In this section, we provide algorithms to estimate P α,ε in the practical case V = R n . These algorithms are implemented in Python (and development continues) in our open-source software at https://gitlab.com/geomdata/topological-parallax

The Rips complex R can be computed efficiently, using [26, 33]. But, the set K is known only through the indicator function k , and the Rips complex R ( X,K ) cannot be computed directly.

Consider x, y ∈ X joined by a V -geodesic (line segment) xy of length 2 ρ V ( e ) , representing a Rips edge e ∈ R ( X,V ) ρ V ( e ) . We would like to estimate ρ K ( e ) , thus giving e ∈ P α,ε for α = ρ K ( e ) and ε = α -ρ V ( e ) . Some simple geometric observations allow us to estimate α and ε .

Lemma 7.1. If there exists p ∈ xy such that k ( p ) = 0 , then e glyph[negationslash]∈ P α, 0 .

This is because in R n , xy is the only path of minimal length.

Algorithm 7.2 (Estimation of e ∈ P α, 0 ) . For each e ∈ R α , sample points p ∼ xy along the corresponding line segment xy . (One method of sampling is simply to check the barycenter.) Return True for e if and only if ' k ( p ) = 1 ∀ p ∼ xy '

Definition 7.3 (Transverse Disk) . Let V = R n . Given an edge e ∈ R and a radius r , let D r ( e ) denote the codimension-1 disk, oriented perpendicular to xy , and centered at e 's barycenter x + y 2 .

Any continuous path from x to y that does not intersect D r ( e ) , must have length exceeding 2 √ ( ρ V ( e )) 2 + r 2 . If k ( B r ( e )) = { 0 } , then all K -paths avoid D r ( e ) , so all K -paths representing e must have length exceeding 2 √ ( ρ V ( e )) 2 + r 2 , giving the following Lemma.

For the second inequality, recall the 2nd order Taylor approximation ρ V ( e ) + 1 2 ρ V ( e ) r 2 ≤ √ ( ρ V ( e )) 2 + r 2 .

Algorithm 7.5 (Bounding e ∈ P α,ε ) . For each e ∈ R ( X,V ) , From r = 0 , loop:

Evaluate k ( p ) for samples p ∼ D r ( e ) .

If k ( p ) = 0 ∀ p , increment r . Else, break.

Return the lower bounds √ ( ρ V ( e )) 2 + r 2 ≤ α and α -ρ V ( e ) = ε .

These algorithms can be extended easily to sample radii along a sequence of points on the edge e , thus providing an estimated K -path for e .

7.1 Back-Propagation

Recent work has shown that various topological properties can be expressed as loss functions that are compatible with back-propagation methods; for example [9, 30, 32]. The method in [32] allows backpropagation for a piecewise-smooth loss function of the form Φ(PDiag( f )) , where f is the filtration function on a simplicial complex, and PDiag( f ) is the persistence diagram for the f -persistent homology of that complex. Lemma 5.3 allows us to interpret homological matching via persistence diagrams PDiag( f ( Y )) , where f ( Y ) = min { t : Y ∈ L t } for some Rips-like path L .

The following function Φ could be used to improve homological matching in this framework. Suppose that X and K are λ lo -LSM. Consider the persistence diagrams PDiag( R ) and PDiag( L ) , where L is some Rips-like path through P ( X,K ) . By λ lo -LSM, we know that PDiag( R ) and PDiag( L ) are identical to the lower-left of ( λ lo , λ lo ) . Choose a desired target value of λ hi . Now, we alter the filtration on L by the following Φ : for dots ( x, y ) to the lower-left of PDiag( L ) ,

PersDiag ofR

Pers Diag of L

+0

×

1.250

1.000

1.000

入sup

Asup

0.750

0.500

0.500

0

0

0.250

1

1

2

2

0.000

0.000

0.000.250.50

0.751.00

1.251.50

0.000.25

0.500.751.001.251.50

Birth

Birth

originaldata

interpolatedbarycenters

20.7%

original data

0.00141%

interpolated pv(e) <1.0

16.6%

interpolated pv(e) >3.0

0.00106%

12.4%

0.00071%

8.3%

4.1%

0.00035%-

0.0%

0.00000%

-0.0001373

-0.0000001

0.0001379

0.8537

1.3844

1.5201

bond length,A

+1.52

bond length,A

we penalize Wasserstein distance to PDiag( R ) . For dots ( x, y ) ∈ PDiag( L ) with x ≤ λ lo < λ hi ≤ y , we penalize by the quantity ‖ x -x 0 ‖ exp( -y ) , where ( x 0 , y 0 ) is the best-match dot from PDiag( R ) . This loss function should force L to express long-lived topological features similar to those of PDiag( L ) , while minimizing the error introduced at scales below λ lo . As of this June 2023 publication, our code does not implement back-propagation to improve homological matching of models, but it is planned as upcoming work.

8 Example: Cyclo-Octane

The conformation space of cyclo-octane [18] is well-known to have novel topological structure. From physical principles, Martin et al. [22] show that real data sampled from the conformation space X can be reduced to lie in V = R 24 , and furthermore X must lie near a 2-dimensional stratified space consisting of a sphere and a Klein bottle. As in Section 1, we suggest that a machine-learning model k : V →{ 0 , 1 } trained to recognize cyclo-octane should not be considered 'good' or trustworthy unless K = { x : k ( x ) = (1) } also takes this geometric form at the appropriate scale. In this section, we demonstrate how topological parallax can support or reject the hypothesis that the geometry of a model K matches the geometry of X .

Figure 3 visualizes the dataset X ⊂ R 24 using a 3D Isomap projection. 3 Following the workflow from Section 5, we compute the 0-, 1-, and 2-dimensional persistence diagrams of X ⊂ R 24 using gudhi [34], and we observe that there are highly persistent features-one 1-cycle and two 2-cycles. 4 The insight from [22] provides the meaning of these cycles, but the precise structure of the 'data manifold' is typically not known a priori in examples. What we know in any case is that we want any potential K to respect these cycles, because the geometry of K should match the geometry of X , whatever it might be.

To understand the validity of a generated conformation, we compute its bond lengths-the distances between adjacent carbon atoms in the conformation. Bond length is an important physical property, together with bond angle, torsion angle, and energy [18]. Given the rigid geometry assumptions of the cyclo-octane data [22], we expect individual conformations generated by trained models to have similar bond lengths, and therefore the distribution of bond lengths from valid conformations should be similar to that of generated conformations. See Figure 4 to compare the bond lengths of conformations from the dataset X versus those at barycenters for short edges and long edges in the Rips complex R of X . Notably, interpolation across longer edges leads to invalid conformer geometries with too short of bond lengths and thus too sharp of bond angles for realistic molecules.

Suppose someone trains a standard neural network k 1 to recognize this data. For this demonstration, we used a 3-layer fully connected network with a ReLU and a SoftMax, implemented in PyTorch. The network was trained to near-perfect accuracy within a few minutes on a two class problem of real data versus a nearby background. (Hyperparameters and training details are provided in the Supplementary Material.) Thus, the model k 1 represents a common starting point that any data analyst might find encouraging. We apply Algorithm 7.2 to estimate which edges in R are accepted by k 1 , and discover 2 λ lo ,X ( K 1 ) = 3 . 45 , which is the longest edge available. So, the Rips complex cannot distinguish K 1 from the convex hull; the model does not reflect the geometry of X . This is particularly unfortunate in this case, because the model k 1 might be used to generate many new conformations (any interpolation between two valid conformations), the vast majority of which will not be valid conformations. An alternative model k 2 is offered, which is built from many local charts (details in Supplementary Material). The new model has 2 λ lo ,X ( K 2 ) = 1 . 0 . Moreover, the most persistent cycles in X have infinite death as measured by the Rips-like path L t = P t, 0 , so X and K 2 are homologically matched with 2 λ hi ≈ 1 . 25 . Therefore, we can claim that the geometry of K matches the geometry of X at these scales.

Limitations

The parallax complex and associated objects are well-defined only for datasets and models that satisfy Definition 1.1. The algorithms in Section 7 assume that V = R n with the Euclidean metric, but could be adapted for other geodesic spaces. Computation of the Rips complex and its persistence diagram scale favorably with the intrinsic dimension of the dataset X ; however, the sampling methods discussed in Section 7 scale with the dimension of V , which might invoke the Curse of Dimensionality. A deeper question is whether real-life datasets X of applied interest actually have enough sample density to exhibit a multiscale metric geometry, to which K can be compared. This question suggests a followup study to verify whether datasets and models with demonstrated real-world efficacy actually have computable and comparable geometries; that is, future work should use parallax to assess the profound epistemological question of whether metric geometry is a valid metaphor for understanding deep learning in real-life applications.

Acknowledgments and Disclosure of Funding

Work by all authors was partially supported by the DARPA AIE Geometries of Learning Program under contract HR00112290076, and by the National Institute of Aerospace (NIA) under sub-award C21-202066-GDA. We are very grateful to Bruce Draper of DARPA and Alywn Goodloe of NASA for their technical guidance during these efforts. We are also very grateful to Matt Dwyer, Rory McDaniel, Tom Fletcher, Yinzhu Jin, Jay Hineman, and Joey Tatro for technical discussions.

References

Henry Adams, Tegan Emerson, Michael Kirby, Rachel Neville, Chris Peterson, Patrick Shipman, Sofya Chepushtanova, Eric Hanson, Francis Motta, and Lori Ziegelmeier. Persistence Images: AStable Vector Representation of Persistent Homology. Journal of Machine Learning Research , 18(8):1-35, 2017. ISSN 1533-7928. URL http://jmlr.org/papers/v18/16-337.html .

Randall Balestriero, Jerome Pesenti, and Yann LeCun. Learning in High Dimension Always Amounts to Extrapolation, October 2021. URL http://arxiv.org/abs/2110.09485 . arXiv:2110.09485 [cs].

Serguei Barannikov, Ilya Trofimov, Grigorii Sotnikov, Ekaterina Trimbach, Alexander Korotin, Alexander Filippov, and Evgeny Burnaev. Manifold Topology Divergence: a Framework for Comparing Data Manifolds. November 2021. URL https://openreview.net/forum?id= Fj6kQJbHwM9 .

Ulrich Bauer, Talha Bin Masood, Barbara Giunti, Guillaume Houry, Michael Kerber, and Abhishek Rathod. Keeping it sparse: Computing Persistent Homology revisited, December 2022. URL http://arxiv.org/abs/2211.09075 . arXiv:2211.09075 [cs, math].

Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation, May 2021. URL http://arxiv.org/abs/2105.14368 . arXiv:2105.14368 [cs, math, stat].

Paul Bendich, J. S. Marron, Ezra Miller, Alex Pieloch, and Sean Skwerer. Persistent homology analysis of brain artery trees. Ann. Appl. Stat. , 10(1), March 2016. ISSN 1932-6157. doi: 10.1214/15-AOAS886. URL https: //projecteuclid.org/journals/annals-of-applied-statistics/volume-10/ issue-1/Persistent-homology-analysis-of-brain-artery-trees/10.1214/ 15-AOAS886.full .

Magnus Bakke Botnan and Michael Lesnick. An Introduction to Multiparameter Persistence, March 2023. URL http://arxiv.org/abs/2203.14289 . arXiv:2203.14289 [cs, math].

Mathieu Carriere, Frederic Chazal, Yuichi Ike, Theo Lacombe, Martin Royer, and Yuhei Umeda. PersLay: A Neural Network Layer for Persistence Diagrams and New Graph Topological Signatures. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics , pages 2786-2796. PMLR, June 2020. URL https: //proceedings.mlr.press/v108/carriere20a.html .

Mathieu Carriere, Frederic Chazal, Marc Glisse, Yuichi Ike, Hariprasad Kannan, and Yuhei Umeda. Optimizing persistent homology based functions. In Proceedings of the 38th International Conference on Machine Learning , pages 1294-1303. PMLR, July 2021. URL https://proceedings.mlr.press/v139/carriere21a.html .

Chao Chen, Xiuyan Ni, Qinxun Bai, and Yusu Wang. A Topological Regularizer for Classifiers via Persistent Homology. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics , pages 2573-2582. PMLR, April 2019. URL https: //proceedings.mlr.press/v89/chen19g.html .

David Cohen-Steiner, Herbert Edelsbrunner, and John Harer. Stability of Persistence Diagrams. Discrete Comput Geom , 37(1):103-120, January 2007. ISSN 1432-0444. doi: 10.1007/ s00454-006-1276-5. URL https://doi.org/10.1007/s00454-006-1276-5 .

Andac Demir, Elie Massaad, and Bulent Kiziltan. Topology-Aware Focal Loss for 3D Image Segmentation, April 2023. URL https://www.biorxiv.org/content/10.1101/2023.04. 21.537860v4 .

Swaroopa Dola, Matthew B. Dwyer, and Mary Lou Soffa. Distribution-Aware Testing of Neural Networks Using Generative Models. In 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE) , pages 226-237, Madrid, ES, May 2021. IEEE. ISBN 9781-66540-296-5. doi: 10.1109/ICSE43902.2021.00032. URL https://ieeexplore.ieee. org/document/9402100/ .

Herbert Edelsbrunner and John L. Harer. Computational Topology: An Introduction . American Mathematical Society, 2010. ISBN 978-1-4704-6769-2. Google-Books-ID: LiljEAAAQBAJ.

Ximena Fernández, Eugenio Borghini, Gabriel Mindlin, and Pablo Groisman. Intrinsic Persistent Homology via Density-based Metric Learning. Journal of Machine Learning Research , 24(75): 1-42, 2023. ISSN 1533-7928. URL http://jmlr.org/papers/v24/21-1044.html .

William H. Guss and Ruslan Salakhutdinov. On Characterizing the Capacity of Neural Networks using Algebraic Topology, February 2018. URL http://arxiv.org/abs/1802.04443 . arXiv:1802.04443 [cs, math, stat].

Allen Hatcher. Algebraic Topology . Cambridge University Press, 2001. ISBN 0-521-79540-0.

James B. Hendrickson. Molecular geometry. V. Evaluation of functions and conformations of medium rings. J. Am. Chem. Soc. , 89(26):7036-7043, December 1967. ISSN 0002-7863. doi: 10.1021/ja01002a036. URL https://doi.org/10.1021/ja01002a036 . Publisher: American Chemical Society.

Felix Hensel, Michael Moor, and Bastian Rieck. A Survey of Topological Machine Learning Methods. Front. Artif. Intell. , 4:681108, May 2021. ISSN 2624-8212. doi: 10.3389/frai.2021.681108. URL https://www.frontiersin.org/articles/10.3389/ frai.2021.681108/full .

Valentin Khrulkov and I. Oseledets. Geometry Score: A Method For Comparing Generative Adversarial Networks. February 2018. URL https://www.semanticscholar.org/paper/ Geometry-Score%3A-A-Method-For-Comparing-Generative-Khrulkov-Oseledets/ 3548baab3e5e6abc278cefdd33f3b58efb924554 .

Changliu Liu, Tomer Arnon, Christopher Lazarus, Christopher Strong, Clark Barrett, and Mykel J. Kochenderfer. Algorithms for Verifying Deep Neural Networks. FNT in Optimization , 4(3-4):244-404, 2021. ISSN 2167-3888, 2167-3918. doi: 10.1561/2400000035. URL http: //www.nowpublishers.com/article/Details/OPT-035 .

Shawn Martin, Aidan Thompson, Evangelos A. Coutsias, and Jean-Paul Watson. Topology of cyclo-octane energy landscape. J. Chem. Phys. , 132(23):234115, June 2010. ISSN 0021-9606. doi: 10.1063/1.3445267. URL https://aip.scitation.org/doi/10.1063/1.3445267 . Publisher: American Institute of Physics.

James R. Munkres. Topology . Prentice Hall, Inc, Upper Saddle River, NJ, 2nd ed edition, 2000. ISBN 978-0-13-181629-9.

Gregory Naitzat, Andrey Zhitnikov, and Lek-Heng Lim. Topology of deep neural networks. J. Mach. Learn. Res. , 21(1):184:7503-184:7542, January 2020. ISSN 1532-4435.

Arnur Nigmetov and Dmitriy Morozov. Topological Optimization with Big Steps, March 2022. URL http://arxiv.org/abs/2203.16748 . arXiv:2203.16748 [cs, math].

Nina Otter, Mason A. Porter, Ulrike Tillmann, Peter Grindrod, and Heather A. Harrington. A roadmap for the computation of persistent homology. EPJ Data Sci. , 6(1):1-38, December 2017. ISSN 2193-1127. doi: 10.1140/epjds/s13688-017-0109-5. URL https://epjdatascience. springeropen.com/articles/10.1140/epjds/s13688-017-0109-5 .

Steve Y. Oudot. Persistence Theory: From Quiver Representations to Data Analysis . American Mathematical Society, Providence, Rhode Island, May 2017. ISBN 978-1-4704-3443-4.

Mathilde Papillon, Sophia Sanborn, Mustafa Hajij, and Nina Miolane. Architectures of Topological Deep Learning: A Survey on Topological Neural Networks, April 2023. URL http://arxiv.org/abs/2304.10031 . arXiv:2304.10031 [cs].

Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences , 117 (40):24652-24663, October 2020. doi: 10.1073/pnas.2015509117. URL https://www.pnas. org/doi/10.1073/pnas.2015509117 . Publisher: Proceedings of the National Academy of Sciences.

Adrien Poulenard, Primoz Skraba, and Maks Ovsjanikov. Topological Function Optimization for Continuous Shape Matching. Computer Graphics Forum , 37(5):13-25, August 2018. ISSN 01677055. doi: 10.1111/cgf.13487. URL https://onlinelibrary.wiley.com/doi/10. 1111/cgf.13487 .

Elchanan Solomon, Alexander Wagner, and Paul Bendich. From Geometry to Topology: Inverse Theorems for Distributed Persistence, February 2022. URL http://arxiv.org/abs/2101. 12288 . arXiv:2101.12288 [cs, math, stat].

Yitzchak Solomon, Alexander Wagner, and Paul Bendich. A Fast and Robust Method for Global Topological Functional Optimization. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics , pages 109-117. PMLR, March 2021. URL https: //proceedings.mlr.press/v130/solomon21a.html .

Guillaume Tauzin, Umberto Lupo, Lewis Tunstall, Julian Burella Pérez, Matteo Caorsi, Anibal M. Medina-Mardones, Alberto Dassatti, and Kathryn Hess. giotto-tda: : A Topological Data Analysis Toolkit for Machine Learning and Data Exploration. Journal of Machine Learning Research , 22(39):1-6, 2021. ISSN 1533-7928. URL http://jmlr.org/papers/v22/ 20-325.html .

The GUDHI Project. GUDHI User and Reference Manual . GUDHI Editorial Board, 3.8.0 edition, 2023. URL https://gudhi.inria.fr/doc/3.8.0/ .

Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang, and Suman Jana. Efficient formal safety analysis of neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , NIPS'18, pages 6369-6379, Red Hook, NY, USA, December 2018. Curran Associates Inc.

Matthew Wheeler, Jose Bouza, and Peter Bubenik. Activation Landscapes as a Topological Summary of Neural Network Performance. In 2021 IEEE International Conference on Big Data (Big Data) , pages 3865-3870, Orlando, FL, USA, December 2021. IEEE. ISBN 978-166543-902-2. doi: 10.1109/BigData52589.2021.9671368. URL https://ieeexplore.ieee. org/document/9671368/ .

Afra Zomorodian and Gunnar Carlsson. Computing Persistent Homology. Discrete Comput Geom , 33(2):249-274, February 2005. ISSN 1432-0444. doi: 10.1007/s00454-004-1146-y. URL https://doi.org/10.1007/s00454-004-1146-y .

A Supplementary Material: Formal Proofs

Proof of Corollary 2.5. The hypothesis [ Y ] ↦→ [0] implies that for some α 0 ≤ α 2 and ε 0 ≤ ε 2 , there exist chains Y ∈ P α 1 ,ε 1 and Z ∈ P α 2 ,ε 2 and W ∈ P α 0 ,ε 0 such that ∂Z = Y -W in P α 2 ,ε 2 . By Lemma 2.4, these chain can be included to their respective levels in R , giving Y ∈ R α 1 , Z ∈ R α 2 , and W ∈ R α 0 , still satisfying ∂Z = Y -W in R α 2 . Hence, applying ι ∗ , either ι ∗ ([ Y ]) is trivial in HR α 1 , or the transition map HR α 1 → HR α 2 takes ι ∗ ([ Y ]) ↦→ [0] .

Proof of Lemma 2.6. If ε ≥ α , then the second condition in Definition 2.2 becomes ρ K ( Y ) -ρ V ( Y ) ≤ α , which is a trivial condition for 0 ≤ ρ V ( Y ) ≤ ρ K ( Y ) ≤ α . Thus, the sets P α,ε are identical for all ε ≥ α .

Proof of Lemma 3.4. Suppose an edge e = ( x i , x j ) lies in P α,ε , meaning that e is represented by a V -geodesic of length 2 ρ V ( e ) satisfying ρ V ( e ) ≤ α and a K -geodesic of length 2 ρ K ( e ) satisfying ρ V ( e ) ≤ ρ K ( e ) ≤ min { α, ρ V ( e ) + ε } . By Lemma 3.3, the corresponding edge e ′ = f glyph[sharp] ( e ) = ( f ( x i ) , f ( x j )) satisfies ρ V ( e ) -κ ≤ ρ V ( e ′ ) ≤ ρ V ( e )+ κ . Moreover, by the data perturbation lemma on K , we know e ′ is represented by a K -geodesic with length 2 ρ K ( e ′ ) satisfying ρ K ( e ′ ) ≤ ρ K ( e )+ κ . Combining these overall, we have the parallax bounds

Proof of Lemma 4.6. Fix λ ′ = 1 2 λ ball ,X ( K ) -κ . We aim to show that X ′ and K are λ ′ -LSM. Fix any edge e ′ = ( x ′ , y ′ ) in R ′ satisfying ρ V ( e ′ ) ≤ λ ′ . (The set of such edges might be empty, which is allowed by Definition 4.1.) For any such e ′ , there is a corresponding e = ( x, y ) ∈ R with f ( x ) = x ′ , f ( y ) = y ′ , and f glyph[sharp] ( e ) = e ′ . The triangle inequality provides ‖ y -x ‖ ≤ ‖ y -y ′ ‖ + ‖ y ′ -x ′ ‖ + ‖ x ′ -x ‖ = 2 λ ′ +2 κ ≤ λ ball ,X ( K ) . We are assuming V is Euclidean, so all balls in V are convex. Since x ′ , y, y ′ are all within the ball of radius λ ball ,X ( K ) around x , all their pairwise geodesics are included in K . Hence, e ′ ∈ P ′ λ ′ , 0 .

Proof of Lemma 5.2. The relationship c ≤ d follows from Lemma 2.5. The relationship d ≤ e follows from Lemma 2.3 and the observations that P α,α = P α, ∞ = R ( X,K ) . Suppose L is parameterized as L α = P α,ε ( α ) . Suppose that c < e , so either c < d or d < e .

If c < d , then we can conclude that the edge that killed [ Y ] in HR ( X,V ) is not present in HR ( X,K ) , so that edge intersected K c , which is an open set. Hence, that killing edge intersects a void in the sense of Defn 1.2. Let B r be an open ball in K c centered on some point on the V -geodesic of length 2 c . If B r = K c , then the shortest K -geodesic replaces the diameter 2 r with the half-circumference πr . Therefore, πr -2 r ≤ 2 d -2 c .

If d < e , then [ Y ] ↦→ [0] via HP b,b → HP d,d and via HP b,b → HP e,ε ( e ) , but not for any HP b,b → HP α,ε ( α ) with α < e . Therefore, there is a K -geodesic of length 2 d and a different K -geodesic of length 2 e , either of which could kill [ Y ] . The former edge is not allowed in L because 0 ≤ ε ( d ) < d -c . Hence, c < d , returning us to the first case.

Proof of Lemma 5.3. Let L be a Rips-like path given by Definition 5.1. Because X and K are λ -locally simplicially matched, the filtrations levels of R and L are identical up to λ . Therefore, there is a bijection between the persistence diagrams of R and L for all dots born by λ .

Suppose that ( b, d ) is a dot in the persistence diagram of L with b ≤ λ < ω < d . This dot represents a class [ Y ] born in HL b that dies in HL d . The class [ Y ] ∈ HR b is born at b must die at some value c , so [ Y ] ∈ ker HR b → c . It cannot be that c ≤ δ , because that would imply [ Y ] ∈ ker HL b → ω , contradicting ω < d . Therefore, δ < c .

Proof of Theorem 5.4. Let f ∗ : HP → HP ′ denote the map induced by f and similarly, let f -1 ∗ : HR ′ → HR denote the map induced by f -1 . Consider the following commutative diagram in which

the unlabeled morphisms are transition maps of the respective persistence modules.

Note that the bi-degrees in the diagram are determined by Lemma 3.3 and Corollary 3.5. Set N ′ = ker HR ′ λ -κ → δ -κ ⊂ HR ′ λ -κ . Commutativity of the top left square implies that f -1 ∗ ( N ′ ) ⊂ HR λ maps vertically to 0 in HR δ and hence, f -1 ∗ ( N ′ ) lies in ker HR λ → δ . The homologically matched assumption on ( X,K ) implies f -1 ∗ ( N ′ ) ⊂ ker HP ( λ,glyph[epsilon1] 1 ) → ( ω,glyph[epsilon1] 2 ) . By commutativity of the bottom right square, f ∗ ( f -1 ∗ ( N ′ )) maps horizontally to 0 ∈ HP ′ ω + κ,glyph[epsilon1] 2 +2 κ . Finally, the locally simplicially matched assumption means we can instead think of N ′ as a subset of HP ′ λ -κ,glyph[epsilon1] 1 +2 κ which maps to 0 in HP ′ ω + κ,glyph[epsilon1] 2 +2 κ by commutativity. Thus,

completing the proof.

B Supplementary Material: Code

The code supporting this article is an initial proof-of-concept. It is available publicly at

It is designed as a simple Python package, following community best-practices for tooling and layout. Documentation is provided there. We recommend that the reader go to the repository for issue tracking, improvements, bug fixes, testing pipelines, etc.

Because this package relies on GUDHI The GUDHI Project [34], the filtration values are by diameter (not radius). This is important to note, because the theory in the paper is written using radius (not diameter) as the filtration value.

Topological parallax has the same computational complexity as the computation of Rips complexes and their persistence diagrams-albeit with a larger constant. This is because parallax merely inserts a model-evaluation step upon the examination of each edge. The constant therefore is tN 2 for N points and a model that takes time t to evaluate. There are very interesting dimension- and structure-dependendent estimates for the real-life/expected timing of Rips computations, such as Bauer et al. [4]. Distributed persistence can parallelize this process, as in Solomon et al. [31]. See https://gitlab.com/geomdata/dispers .

Figure 1 was produced with the Jupyter notebook notebooks/Winky Example.ipynb . Compare and contrast the Python scripts runners/run_winky_nn.py versus runners/run_winky_tree.py

Figure 3 was produced with the Python script runners/run_cyclooctane_balls.py . Compare and contrast that script with runners/run_cyclooctane_nnfar.py and run_cyclooctane_nnhull.py . These examples may take 100-500 GB of RAM to compute the persistence diagrams as currently implemented.

Figure 4 was produced with the Python script runners/bond_lengths.py

C Supplementary Material: Utah Jar Example

This example demonstrates a situation where a classifier appears to be 100% correct on a semantically meaningful dataset, but the resulting model is too strict for interpolation within a data class, Topological parallax detects this phenomenon, as discussed in Section 6.

We applied parallax to an imagery dataset inspired by the 'Utah teapot.' The dataset consists of 14000 images-4000 images of a rotated teapot and 10000 images of a rotated teapot with no spout or handle (referred to as the 'Utah jar'). See Figure 5a. All images were rendered with PoV-Ray.

We constructed a bespoke classifier on this dataset of images which accepts any image within a distance of 1.0 to the set of jar images using the Euclidean metric on flattened images. Our classifier rejects all other inputs. We applied parallax to this model and found very small values for both λ lo and λ sup , with λ lo ∼ λ sup < 2 . See Figure 5b.

Given the hard cut-off distance of 1.0 in the classifier and these λ values, we expect the model to prohibit reasonable interpolations of images. Indeed, we found numerous interpolations rejected by the model which seem valid to the human eye. See Figure 5c.

PersDiagofR

+

5.000

4.000

3.000

2.000

sup

0

1.000

１2

0.000

0

1

2

3

4

5

6

Birth

O

0

D

D Supplementary Material: Understanding the Bi-Complex

The parallax complex P ( X,K,V ) is bi-filtered by the parameters α (the length of a geodesic) and ε (the distortion of geodesic length between K and V ). Figure 6 provides some visualizations that may help the reader interpret traditional barcodes of Rips-like paths, and how these bi-filtration parameters may be estimated. These parameters are related to the size of voids in the model, as seen in Lemma 5.2.

ε

α

ε

HL

α

barcode of

L

birth

death

x

y

ρ

V

r

√

ρ

2

V

+

r

2

α

E Supplementary Material: Table of Symbols

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a framework for analyzing the validity of deep classification models by comparing its multi-scale geometric features with that of the training dataset. The comparison is conducted through tools of topological data analysis, mainly using persistence diagrams to detect and characterize the nontrivial topological features (connected components, 1-cycles, 2-cycles, etc.) as they are revealed by changing the radius of data points to construct Ribs complex. The main challenge is to conduct such analysis on the implicit space of the deep classification model; the paper addresses this challenge by devising distance estimate algorithms for the implicit space, and comparing it with the ambient Euclidean distance to decide the matching of geometric features. The paper shows an application of the analysis on a single dataset and accompanying simple classification models.\n\nSTRENGTHS:\nThe paper asks the important question of how to decide if a deep perception model faithfully captures the true characteristics of a dataset. It proposes to check if the learned implicit space of the perception model has geometric features that closely match the dataset distribution. \n\nThe paper develops a theoretical framework to characterize the geometric matching of model and data. The important notion is to construct a parallel complex, that is obtained by the co-filtration of Rips complex by metrics of both the data set and the learned model. Based on this construction, the comparison of model and data becomes homological computations on the series of complexes, which draws on topological data analysis extensively. \n\nThe paper discusses applications through a concrete example, and of the possibility of using the assessment as objective function for training more accurate networks.\n\nWEAKNESSES:\nThe paper is theoretical in nature. To ease understanding, it is better to provide tables to summarize the notations introduced and use more figures to motivate and illustrate the ideas. I find this can be helpful particularly for the perturbation lemmas and local simplicial matching. Drawing more connections to TDA can be helpful too, to motivate the constructions introduced in this paper.\n\nAfter developing the theoretical framework, it is desirable to immediately position Sec.6 in context, maybe through the example of Sec.8 or other cases. \n\nMore application examples can be used. The cyclo-octane dataset has a very unintuitive structure. If more intuitive datasets and perception tasks can be tested, the usefulness of this analysis can be more convincing. Such intuitive datasets can come from computer vision data, e.g. MNIST, CIFAR. \n\nDeep learning models are witnessing a shift from perception to generative modeling, achieved mostly through minor variations in objective function and output format. How do the motivation and framework proposed by this paper apply to generative models? This paper does not include any such discussion.\n\nQUESTIONS:\nThe major questions have been posed in the above discussions about weaknesses.\n\nMinor questions:\n\nDefinition 1.1, $K^\\circ, \\overline{K^\\circ}$ should be defined clearly. A summary of important notations could be given in the text.\n\nLine54, the order of $M$ and $M^*$ seems to be reversed?\n\nLIMITATIONS:\nThe authors have discussed limitations extensively.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors suggest that a model is good if its geometry matches the geometry of data, and introduce a persistent-homology based method to evaluate this similarity.\n\nSTRENGTHS:\nThere is a number of theoretical results that seem to support the soundness of the proposed approach.\n\nWEAKNESSES:\n(W1) The paper is extremely hard to read, which significantly limits its impact. For example, you mention that you rely on bi-filtered persistence module, but it is still unclear to me what the two filtrations are (what is reflected by alpha and epsilon), and why each of them is useful. As another example, rho_K(Y) is a crucial notion for your work, but you never name or describe it, let alone motivate it or provide intuitive explanation; also, this requires an explicit Definition. As a final example, Figure 2 remains a mystery. \n\nAll definitions and lemmas could be named, but more importantly, they could be first motivated and then interpreted. The latter could be done in the beginning of a section, by summarizing all the results and their relevance in a paragraph or two, or throughout the section in a story-like manner. The same holds for the whole text, e.g., when you describe your procedure, you could describe the concepts in words (even if you did this before, it is good to remind the reader, especially since the paper is extremely notation-heavy):  (2) for a model [nice] K ∈ M(X), compute [what is this] λlo,X(K) and [what is this] λhi,X(K)”, or “In this section, we provide algorithms to estimate [what is this] P_{alpha, epsilon}”. A well-organized notation table might also be helpful.\n\nFor this reason, a lot of my feedback is an educated guess, and I am definitely open to substantially changing my score for this paper if it is presented more clearly, so that I can actually assess the soundness and contributions.\n\n(W2) Experiments consider only one data set.\n\nQUESTIONS:\n(Q1) Can you motivate the name Parallax?\n\n(Q2) You write: “We suggest that a model K is ‘good’ if the geometry of K matches the geometry of X.” How is this related to “If a certain architecture is incapable of expressing a decision region that is equivalent in topology to training data, then there is no hope of it ever generalizing to the true data” in [1]? More generally, articles [1]-[3] also consider persistent homology of the data and model, can you comment if and how they are related to your approach?\n\n(Q3) Can you elaborate on the relationship between your work and the related articles you mention in Section 1.3, in particular with [17] and [28]? \n\n(Q4) Where is the proof of Lemma 2.6?\n\n(Q5) “In Section 4, we introduced ‘local simplicial matching’ as a way to compare small-scale geometry. In this section, we introduce ‘homological matching’ as way to compare large-scale geometry.” Should this not be mentioned earlier in the paper? Is it possible to provide an illustration?\n\n(Q6) “We apply Algorithm 7.2 to estimate which edges in R are accepted by k1, and discover 2λlo,X(K1) = 3.45, which is the longest edge available. So, the Rips complex cannot distinguish K1 from the convex hull; the model does not reflect the geometry of X.” How do you make this conclusion, can you elaborate? This is related to weakness (W1).\n\n\nOther minor comments:\n\n- Line 28: should be applied -> could be applied?\n- Figure 1: It would be better if the axes ranges are also provided in the left plot, to make a connection with its PD.\n- Line 168: Def’n -> Definition\n- Explicitly mention that the code is made publicly available.\n- Is Limitations Section 9?\n\n\n[1] Guss, William H., and Ruslan Salakhutdinov. \"On characterizing the capacity of neural networks using algebraic topology.\" arXiv preprint arXiv:1802.04443 (2018).\n\n[2] Ramamurthy, Karthikeyan Natesan, Kush Varshney, and Krishnan Mody. \"Topological data analysis of decision boundaries with application to model selection.\" International Conference on Machine Learning. PMLR, 2019.\n\n[3] Khrulkov, Valentin, and Ivan Oseledets. \"Geometry score: A method for comparing generative adversarial networks.\" International Conference on Machine Learning. PMLR, 2018.\n\nLIMITATIONS:\nLimitations are discussed in the final section.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 1, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper introduces topological parallax, a theoretical framework for analyzing the similarity of multiscale geometric structures between datasets and models. It estimates the topological features in the model by examining the effect on the Rips complex of geodesic distortions using the reference dataset. It shows the stability of the proposed framework under dataset perturbations. It also provides a practical computational method on top of the theory.

STRENGTHS:
- The paper provides a novel point of view. It claims that it is the first work to use TDA to express a desired geometric relationship between models and datasets, and to the best of my knowledge, I don't see any existing work studying this problem.
- This paper provides interesting insights into the concepts of "overfitting" and "generalization" of neural networks, which are important for the safety and robustness of AI.
- It proposes a theoretical framework with abundant derivations and proofs.
- It also introduces a practical computational method and demonstrates it on concrete data and network examples.

WEAKNESSES:
- Section 8 shows an interesting example of a data space with novel topological structures. But is this method also applicable to other real-world scenarios such as image recognition? What is the complexity of the computation w.r.t. dataset size and dimensionality?

QUESTIONS:
- The theoretical framework is built upon a binary classification problem. Is it potentially generalizable to more complex model outputs, e.g. multi-class classification (especially when different output channels are inter-dependent)? How would the geodesics be defined in such a case?

LIMITATIONS:
Limitations are well-discussed in the paper. As a theory work, I believe it won't have direct social impacts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this paper, topological parallax is introduced to compare a trained model with a dataset and determine if they share similar multiscale geometric structures. The authors argue that the model is \"good\" if the geometries are similar. To determine the similarity between the model and the data, they calculate a homologous matching that can be applied to many ML systems. This method can thus be used to assess whether a model has good generalization or is robust to perturbations. The authors validate topological parallax with a toy data set, and the qualitative and numerical results support the authors' claim.\n\nSTRENGTHS:\nTopological data analysis is emerging as a powerful tool for understanding AI systems. Based on TBA, the proposed topological parallax measures the geometric similarity between model and data, which helps to understand whether the trained model is good or not. Moreover, it has many good properties, e.g., it is stable to perturbation, which is crucial for AI attack detection.\n\nWEAKNESSES:\nI think that further experiments could be helpful to understand the proposed tool. For example, Figure 3 shows only the results of one neural network, and it is expected to show the comparison of multiple AI models using topological parallax. For example, compare the decision tree model and the neural network in Figure 1.\nAlso, try to compare several known neural networks. For example, compare a ReLU MLP with a Sin MLP. Given an image, the ReLU MLP may not fit the data well, but if you replace ReLU with Sin in the MLP (called Sin MLP), the latter method can fit the data with almost zero error.\n\nSin-MLP, also known as SIREN, was described in the article \"Implicit neural representations with periodic activation functions\".\n\nThe reviewer is not familiar with this area and therefore basically respects the other reviews in principle.\n\nQUESTIONS:\nHow can topological parallax be used to derive geometric regularities and improve neural network training?\n\nLIMITATIONS:\nThe limitation is well discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces the concept of parallax as a bi-filtered persistence module that measures the geodesic distortion between the dataset and the model. The paper also proves that parallax is stable under perturbations of the dataset and provides a criterion of homological matching to assess whether the model captures the persistent features of the dataset. The paper demonstrates the effectiveness of parallax on two models using the cyclo-octane dataset and discusses the limitations and future directions of the method.\n\nSTRENGTHS:\nThis paper presents an interesting and novel approach to evaluate the geometric similarity between a dataset and a model using topological data analysis. The paper is well-written, clear, and provides sufficient background and motivation for the problem.\n\nWEAKNESSES:\nThere is little experimental analysis on real-world dataset. Perhaps the authors could consider merging some part into the appendix and include more analysis.\n\nQUESTIONS:\n1. This paper presented a novel theorical way to measure the performance of model by analyzing the geometry of model and data. However, we already have lots of metrics, such as accuracy, which can be measured once the trained and the corresponding dataset are available, it is not clear to me how the proposed method advance the existing method. \n\n2. The paper is more related to math/statistics, while the analysis seems to be very solid. The analysis of overfitting and generalization capability (especially under the setting of covariate shift), which is claimed as the major contribution by the authors, is not easy to follow, more concrete examples are needed for illustration.\n\n3. This paper does not discuss the details of models, i.e., ConvNet, RNN, transformer. I am curious whether the proposed theory could be applied to all models trained by back-propagation?\n\nLIMITATIONS:\nSee above", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose a method to evaluate how well a model learned a data distribution based on topological data analysis. The authors assume that the model is a classifier (the output of the model is binary, or consists of a finite set of classes that can be evaluated separately), and topologically compare the set of positive data samples to the set of data points where the model outputs a positive value. Topological properties are computed using filtrations of the Rips complex on the data points. The quality of a model is evaluated by checking 1) at which scale the complex in the full ambient space starts to diverge from a complex that is restricted to the subspace where the model is positive, and 2) if persistent features of the full complex are also present in the restricted complex. The authors given an example of measuring and comparing the quality of two models applied the a small dataset with known geometry of the data manifold.\n\nSTRENGTHS:\n- The approach for measuring the similarities between the manifold learned by a model, and the manifold implicitly described by a set of data samples seems interesting, and is novel as far as I can tell (although I am not familiar with topological data analysis).\n- It seems like it could be useful for finding adversarial examples for a model, or identifying regions where a model does not perform well.\n- If this approach would work with real-world data (or if it can be extended to do that), it could applicable quite broadly to evaluations of models that learn a distribution in high-dimensional spaces.\n\nWEAKNESSES:\n- The empirical evaluation is not thorough enough. I do not have a good intuition in which situations the proposed measure of model quality would be useful, and I do not think it is obvious from the description of the method (see below for a discussion). Therefore, a more thorough empirical evaluation is needed to show in which situations the measure is useful. Specifically, more datasets should be evaluated, and the metric used to evaluate success could be improved as well: the current metric using bond lengths is specific to the dataset used, and would not be useful for comparing the performance across different datasets. A more useful metric might be to use a dense held-out test set, and compare the model quality predicted by the proposed method to the model quality according to the test set.\n- The method is not compared to any alternatives for measuring the quality of the distribution learned by a model, or for finding adversarial examples. For example, what are advantages/disadvantages compared to the standard approach of using a held-out test set? (I can imagine that the proposed method might have advantages if the test set cannot sample the space densely enough, but disadvantages for regions outside the convex hull of the training samples.) Or compared to other methods for finding adversarial examples (there is a large body of literature, a lot of work can be found, for example, by searching for \"adversarial examples\"  in Google Scholar)? This should be at least discussed in the related work, and the advantages/disadvantages are not clear from a theoretical standpoint, ideally an empirical comparison should be provided.\n- It seems like the measure could be less useful for detecting erros of the model (false positives or false negatives) outside the convex hull of the data samples, since the space outside the convex hull is not explored by the Rips complex. But it seems like real-world data might have true positives outside convex hull (i.e. the true data manifold may extend significantly outside the convex hull of the data samples). See details for a discussion.\n- The computational complexity of the method is not provided. Ideally the authors should provide the number of evaluations of the model that is typically needed, in addition to the overhead from building the Rips complex and computing the algorithm described in Section 7.\n- The exposition is very hard to follow for non-experts in topological data analysis, and additionally some symbols/variables are undefined (if these are common symbols in topological data analysis, I am not familiar with them).\n\nMore details:\n\n* More empirical evaluation is needed to understand in which situations the proposed measure gives good results. From the description of the proposed measure alone, it does not seem clear to me in which situations the measure gives a good estimate of model quality. On one hand, I can see how topological properties could be relatively stable descriptors of a data manifold in high-dimensional space, but on the other hand, I do not have a good intuition how the data manifold typically behaves in high-dimensional space. For example in the manifold of natural images or their latent features, I can imagine that the true data manifold could lie significantly outside the convex hull of the data samples, and that it would be hard to capture enough data samples to cover the full data manifold in their convex hull. In that case, a good model would not just be a \"thickening\" of the manifold given by the data samples (as described by the authors), or even a thickening of their convex hull, but would also include samples significantly outside the convex hull. For example, a dataset like CLEVR may contain images showing a blue sphere at multiple random locations in the image, but some regions of the image may not be covered by the blue sphere in any of the samples; in that case, we would still expect a good classifier of the data manifold to also classify images as positive where the blue sphere is in a location that was not directly observed in the data samples. Thus, it seems to me that a good model needs to extrapolate significantly outside of the convex hull of the data samples. Would the Rips complex constructed on the data samples be able to capture the part of the true data manifold that extends beyond the convex hull of the samples? And would the proposed method therefore be able to handle such cases, which might be quite common in real-world datasets? Arguably the manifold would be better behaved in latent spaces that have a more semantical representation of the data, but this then means that the latent space that the proposed measure is applied to has to be chosen carefully, and it is unclear how to choose the latent space.\n\n* The exposition is missing definitions in several places:\n\n    - In Section 1, there are a few missing definitions:\n        - $K^\\circ$ is not defined.\n\t\t- $\\overline{K}$ is not defined. Does this denote the set complement? If so, would this not mean that $K^\\circ$ is the complement of $K$, and in that case, why have two different notations for the complement?\n\t\t- $K^c$ is not defined. It could also be the set complement, but then there would be three different notations for the complement, so I guess both $K^\\circ$ and $\\overline{K}$ do not actually denote the set complement.\n\t- In Section 2, there are several missing definitions that make it hard to follow the exposition:\n\t\t- A Rips complex is not defined. Rips complex may be well-known in topological data analysis, but I think that only a small fraction of NeurIPS readers will be experts in topological data analysis, so giving a short definition would be good. Also, even if the Rips complex is known to readers, the arguments to the Rips complex that are used here may need to be defined. The first argument X is quite clear, but second argument $K$ is less clear. Does the second argument restrict the metric used to construct the Rips complex to geodesics in the subspace $K$?\n\t\t- A filtration of a Rips complex is not defined.\n\t\t- A chain $Y$ is not defined.\n\t\t- $B$ is not defined.\n\t\t- In Corollary 2.5, $i_*$ is not defined.\n* Eq. 2.2 could probably be simplified. Since $K$ is defined as subspace of $V$, $\\rho_V(Y) \\le \\rho_K(Y)$ is true for all $Y \\in R$ (according to Lemma 2.1) and does not need to be mentioned explicitly in the definition. Therefore a simpler definition would be: $P(X, K, V)_{\\alpha,\\epsilon} =$ {$Y \\in R\\ |\\ \\rho_K(Y) \\le \\rho_V(Y) + \\epsilon, \\rho_K(Y) \\le \\alpha$}\n* in Algorithm 7.5, step 2, should f(p) be k(p) instead?\t\t\n* On Line 21: I would define $K$ as {$x \\in V\\ |\\ k(x) = 1$}, since $k^{-1}$ is not well defined for non-injective functions.\n\nQUESTIONS:\nA preview of the discussion of the advantages/disadvantages compared to related work might be useful, as wall as a discussion of the issue with evaluating regions outside the convex hull of the data samples.\n\nLIMITATIONS:\nA few limitations have been discussed, including that the authors are not certain if the method would work on more complex real-world data.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}