Topological Parallax: A Geometric Specification for Deep Perception Models
Authors: Abraham David Smith
Michael J. Catanzaro
Gabrielle Angeloro
Nirav Patel
Paul Bendich
Venue: NeurIPS 2023 spotlight (2023)
Keywords: topological data analysis
persistent homology
convexity
AI safety
interpolation
TL;DR: Topological Parallax compares a trained model to a reference dataset to determine whether they have similar multiscale geometric structure, which we argue is a desirable property..

Abstract: For safety and robustness of AI systems, we introduce _topological parallax_ as a 
theoretical and computational tool that compares a trained model to a reference dataset to determine whether they have similar multiscale geometric structure. 

Our proofs and examples show that this geometric similarity between dataset and model is essential 
to trustworthy interpolation and perturbation, and we conjecture that this new concept will add value to the current debate regarding the unclear relationship between "overfitting"' and "generalization'' in applications of deep-learning. 

In typical deep-learning applications, an explicit geometric description of the model is
impossible, but parallax can estimate topological features (components, cycles, voids, etc.)
in the model by examining the effect on the Rips complex of geodesic distortions using the reference dataset.
Thus, parallax indicates whether the model shares similar multiscale geometric features with the dataset.

Parallax presents theoretically via topological data analysis [TDA] as a bi-filtered persistence module,
and the key properties of this module are stable under perturbation of the reference dataset.

Topological Parallax: A Geometric Specification for Deep Perception Models
Abraham D. Smith
Michael J. Catanzaro
Geometric Data Analytics, Inc.
Geometric Data Analytics, Inc.
343 W. Main Street Durham, NC 27701 USA abraham smi tw@ggoddaaaa.....oommoomoraaaaat com University of Wisconsin-Stout Math, Stats, and CS Dept Menomonie, WI 54751 USA smithabr@@wsttout, edu
343 W. Main Street Durham, NC 27701 USA
michael wwt.:aaattttaaregmdaaaaaa.oo
Gabrielle Angeloro
Nirav Patel
Geometric Data Analytics, Inc.
Geometric Data Analytics, Inc.
343 W. Main Street Durham, NC 27701 USA
343 W. Main Street Durham, NC 27701 USA nirav, www.eeeeomddaaaccoommm com
Paul Bendich
Geometric Data Analyties, Inc.
343 W. Main Street Durham, NC 27701 USA
paul. bendi bendiihh@eooddat com
Duke University Mathematics Dept
Durham, NC 27708 USA
bendich@matie duke. edu
Abstract
For safety and robustness of Al systems, we introduce topological parallax as a theoretical and computational tool that compares trained model to a reference dataset to determine whether they have similar multiscale geometric structure.
Our proof's and examples show that this geometric similarity between dataset and model is essential to trustworthy interpolation and perturbation, and we conjecture that this new concept will add value to the current debate regarding the unclear relationship between "overfitting" and "generalization" in applications of deep- learning-
In typical DNN applications, explicit geometric description of the model is impossible, but parallax can estimate topological features (components, cycles, voids, etc.) in the model by examining the effect the Rips complex of geodesic distortions using the reference dataset. Thus, parallax indicates whether the model shares similar multiscale geometric features with the dataset.
Parallax presents theoretically via topological data analysis [TDA] as bi-filtered persistence module, and the key properties of this module are stable under pertur- bation of the reference dataset.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Introduction
Suppose X is a finite subset of V = R" with the Euclidean metric.' In data science particularly in applications of DNNs we often encounter the situation where X is a dataset, and some opaque algorithm has produced a trained model k : V + {0. 1), where k(z) = for all x E X. This defines the model as a set of accepted inputs K = {xr k<())= = (1)) c V. which has no available description beyond evaluation of the perception function A on samples.
Our main contribution in this paper is a method we call topological parallax to estimate the multiscale geometry of K from the persistent homology ([14, 37J) of X, in a situation where K does not have an explicit description. 2 This method provides meaningful geometric information about K through simple computational approach that can be applied to any perception model k. We prove that the resulting criterion of homological matching satisfies a stability property. We propose homological matching via parallax as a geometric specification that could be applied to many muchine-learning systems. The measurement of homological matching also admits a back back-propase propaggaiinnnn scheme. which could be used to improve the geometric similarity between the model K and the dataset X.
Because of the generality of Definition 1.1, it may be that V represents any layer of neural network, X represents any dataset mapped into that layer, and K represents an activated region in that layer. For example, the "neural collapse" concept from Papyan et al. [29] can be seen as a special case of this specification, because the conception from [29] is that the dataset X becomes a tight blob in the penultimate layer, and the penultimate model simply a Voronoi region K surrounding that blob.
1.1 Assumptions and Motivation
As discussed by Belkin [5]. DNNs usually achieve high statistical accuracy, but some resulting models are better than others at capturing patterns in the dataset. Despite having perfect statistical accuracy on the dataset X, fundamental questions arise about the model K: "Is it safe to deploy? Is trustworthy? Is it a good model?" To broadly paraphrase [5], it used be good practice to tell data analysts not to overfit their data, because overfit models were "bad" due to poor generalization; however, essentially every DNN fits the training data perfectly, so is not clear what distinguishes "good" models from "bad." We suggest that model K is "good" if the geometry of K matches the geometry of X. Consider the two example models in Figure 1. Although both models achieve perfect statistical accuracy, only one of them appears to have learned the geometric structure of the dataset. This suggestion is likely intuitive to many ML engineers, but the subtlety lies in the implicit assumptions often made about the nature of the dataset and the available models. We assess this matching in a way that is independent of the architecture that produced k, and that makes very few assumptions about X and K, as encapsulated by Definition 1.1, which are assumed henceforth.
Definition 1.1 (Datasets and Models), Suppose that is a geodesic space. We define model K C V to be the closure of an open set, K = K C V, colloquially known as a "solid." For any finite dataset X C V,, we consider the collection of all models for which X is contained in the interior of the model, X C K"; let M(X)= == {K C V X C K2 K K). With subset inclusions as morphisms, M(X) is a small category. For any K C. V with K = =", let M (K) == (X c CV: xx K", #X <o)). With functions of finite sets as morphisms, (K) is a small category. course, X E (M) + K E M(X), Note that V E M(X)..
Note that we make no assumptions whatsoever about the architecture or training method that yielded K. With such a broad definition of X and K, we need a notion of "geometry" with very few assumptions. We use standard topological notation [23, 17, 14].
Definition 1.2 (Void). Given set K C V, an interpolative void in K is a bounded open set in the complement of K, Ω C KS, such that there exists a pair of points x, y E K for which all V V-geddesics pass through Q. V R", this means Ω intersects the convex hull of K.
1 'The theoretical results apply if V is any geadesic space-a metric space in which each distance is realized by path- but our motivation and examples use V = 18" to avoid distraction from our central theme of model assessment.
TTooooggcca parallax was named by analogy to the method in astronomy, where an inaccessible object cannot be measured directly (in this case, the geometry of K), so we must infer its location by comparing multiple observations from available vantage points (in this case, the dataset X).

---

Nodel K. Oecison Tree
Shaisl Kc Next Networ
2000
Figure 1: 1: A dataset X C. R2 2: The persistence diagram of the Rips complex of X, showing three components (two of which die at the dim-O) dots near 2.0) and two cycles (the dim-1 dots, representing the annulus and the overall arrangement) among the persistent features. 3: A tree model K1. 4: A neural network model K2. The tree model has many small voids that would forbid interpolation perturbation in many locations. Without the luxury to visualize models K V for dim V > Parallax can measure geometric similarity between X and K via the Rips complex I of X.
40
60
sup
00
Xpup 7.1
Agup 7.1 Ami 00
Aoup 7.1
Agup
Api 0C
Api 00
Ani 30
An 20
Figure 2: Five models for the same dataset, and the scales detected by parallax. The circle have radii 20 and 30, respectively. See Section 6 for general interpretation. The missing edge causing Asup is highlighted in red at filtration radius 7.1, and the missing edges that cause Ani are highlighted in blue. The dashed blue edge does not affect Am because the larger cycle determines that value.
Why focus on voids? As observed by Balestriero et al. [2], when V is high-dimensional, it is unlikely that any points in X lie in the convex hull of any others. However, it also seems unlikely that "good" model K of X will be merely convex solid. If convex models were sufficient for real- world problems, DNNs would be unnecessary, and the field would have concluded with PCA, Gaussian kernels, and convex polytopes. For example, many models implicitly or explicitly rely on the so-called manifold hypothesis, which is the hope that realistic datasets X will tend be distributed near union of lewer-dimensioal manifolds immersed in V, in which case "good" model K would be slight thickening of those manifolds to allow for measurement error. If there are multiple manifolds or there is nonzero curvature, such model will have voids.
Any void in K indicates region where K does not allow interpolation, which is where interesting geometry occurs. Voids occur if and only if some geodesic in K is strictly longer than the correspond- ing geodesic in V. Hence, for this article, we interpret "geometry of K" the presence or absence of voids. We do not make assumptions about the geometric features of X or about the family from which is chosen; rather, we ask that K respects the features of X, whatever they are. We propose that "good" model K is one whose voids represent the highly persistent features of X, in the sense of Topological Data Analysis [TDA, [14, 27]]
A model K can be "bad" because it has too many or too few voids at various scales. For example, mismatch of voids of K and features of X would indicate that K is over-sensitive to small error or under-sensitive to large error, either of which could lead to adversarial attack. Also, numerous small-scale voids could make K incompatible with some forms of the manifold hypotheses, by obstructing the coverage of K by an atlas of local convex charts of moderate size.
1.2 Outline
Section 2 introduces our key object, the bi-graded [7] parallax complex, in Definition 2.2, which measures geodesic distortion via the Rips complex. Section 3 provides a notion of dataset perturbation

---

and shows that the parallax complex and its homology remain stable under those perturbations. Section 4 defines local simplicial matching and shows how parallax detects small-scale changes in the Rips complex of X in K versus V, giving a clearly interpretable scale of locality, Ato- Section 5 defines homological matching and shows how parallax detects large-scale voids in K, giving scale An above which homological features in X are respected by voids in K. Together, these results provide an overall interpretation as a specification in Section 6, which largely achieves the goals laid out in Section 1. Section 7 provides computational approaches to computing parallax, and links to our open-source software that has many practical improvements not detailed in this paper. Section illustrates the effectiveness of parallax specification, as demonstrated on two models using the cyclo-octane dataset [22]. Additional proofs, details, and examples are provided in the Supplementary Material appendices.
1.3 Related Work
To the best of our knowledge this is the first work use TDA express a desired geometric relationship that holds directly between datasets and models trained on them. There has been some work, for example the Manifold Topology Divergence of Barannikov et al. [3] or the Geometric Score of Khrulkov and Oseledets [20], which uses various
measures to quantify the difference between training data and new data generated by generative models. Quite a few other papers (see Fernández et al. [15] for very recent example) use TDA-based constructions to infer properties of underlying data manifolds, usually under very strict sampling assumptions.
More broadly, there has been a recent explosion of work (e.g. Hensel et al. [19]) connecting TDA to ML/DL. Several works (e.g. Adams et al. [1]. Bendich et al. [6]) use TDA a feature extraction method, pre-processing more complicated data objects before running standard ML pipelines. Later works (e.g. Chen et al. [10], Demir et al. [12], Solomon et al. [32], Nigmetov and Morozov [25]) use TDA to define novel losses within ML algorithms. Note that we comment below on ways in which our notion of homological matching can be used define TDA-based loss. TDA has also been used (e.g. Naitzat et al. [24]. Wheeler et al. [36]) to analyze the behavior of data as it passes through the layers of DNN. Some works (e.g- Guss and Salakhutdinov [16]) assess the capacity of a specific DNN to classify datasets with specific shapes, but do not provide tools to quantify shape mismatch between model and dataset. Finally, several works (e.g. Carriere et al. [8], Papillon et al. [28]) use TDA to define novel DNN architectures, including GNNs and other higher-order combinatorial structures.
There is also a recent stream (e.g. Liu et al. [21], Wang et al. [35]) of work that builds validation and verification//alliiicaaion techniques for desired properties of DNN-trained models; these mostly focus on the mechanics of how to verify/falsify such properties, rather than attempting to define them as we do. Perhaps the closest work in this stream to ours is Dola et al. [13], which uses a prior assumption on the underlying data distribution to verify/falsify DNN properties.
2 The Parallax Bi-Complex
Let B, (x) denotes the closed geodesic ball of radius a around V. For formal edge (I0, x1) between points in X, (v(() is the minimum radius for which B (ev (2n) intersects B (x))). Thus, (c) is the geodesic distance between xo and æ1. The Rips complex R(X, V) is the simplicial complex generated by these edges, as filtered by pv(c). A chain is formal sum of simplices in a complex [14]. More generally, for any K E EM(X), the Rips complex R(X, K) and its filtration by 0 S a co is defined by
(Zo...., II) E Ra(X, K),, if and only if (x.) n (a(z,) For any chain Y E R(X, K), let (Y) minfa : Y E K) When X and V are understood in context, we abbreviate R = RRX, V) for the ambient case K V.
0A SOSsi<jSS (2.1)
Lemma 2.1. If K1, K₂ E M(X) with K₁ C K2 C V, then there is a natural inclusion of filtered modules, R( X, K1)a R(X, K2), -Ra- That is, PY) S PV,(Y) S PKi (Y), with the convention min 0 00.
The previous lemma is simply because geodesic lengths in K₁ are never shorter than geodesic lengths in K2, and neither is shorter than geodesic lengths V. Our approach to the question "does the geometry of K match the geometry of relies
relies on detecting the inequality pv (Y) < pK (Y).

---

Definition 2.2 (Parallax Complex), For K E M(X), let P(X, K, V) denote the subcomplex of R defined for each real pair (a, by
P(X, K, Va. = (Y E R PK(Y) S a, pv S px(Y) & pv + When X, K, V are understood in context, we abbreviate P P(X, V.).
(2.2)
The parameter measures the distortion of geodesic length in K versus The next few lemmas are immediate consequences of the definition.
Lemma 2.3 (Parallax is Bi-Filtered), If S d', then Pa.e C Par.e Ife < e', then Pa.e C Pa.e'
Lemma 2.4. For all we have Pa.e C R) C Ra.
Let L Pa.s + R denote the inclusion of complexes, and let b, H.P. + HR denote the induced homomorphism on homology [17].
Corollary 2.5 (Homology Deaths later in Parallax). Suppose that [Y] is class in H H such that the bi-transition map H Ho + Pa2 takes [Y] + [0]. Then there exists S or such that the transition map H Ba, HR, satisfies to ([[]) [0].
Lemma 2.6. For all O, we have Pa Pa, R(X, K). Ro.
The proofs of 2.5 and 2.6 are given in the Supplemental Material.
is sometimes useful create for the purpose of computing barcodes and persistence diagrams.
filtrations through P. parameterized by Rips radius,
Definition 2.7 (Rips-like like Path). A Rips-like path is a filtered module La = Pa,eta) for 0 S < co such that ( is non-decreasing function satisfying e(0) = O.
Rips-like path has homology HL and a barcode or persistence diagram. By Lemma 2.6, one Rips-like path is ((X,K)) Pa.a. Another is the "inflexible" path La = Pa.0-
Perturbation
This section establishes lemmas that ensure Parallax and its consequences (notably Theorem 5.4) are stable under certain types of perturbations, which means that the parallax is reasonable in the presence of noise.
Definition 3.1 (Pointwise Perturbation). Given E M* a pointwise k-perturbation X' (V) such that the sets X and X' admit a one-to-one correspondence f X - X' satisfying (z)(x) æ|| I k. We write f & X".
Definition 3.2 (Pointwise K K-perturbationn). Suppose f f R. X' such that each (x, f(x)) pair is connected by K- K-geodesic of length S We write f X X'.
Note that any f X ~ X' is an isomorphism in the category M M²((" (V). and any f X = X' is an isomorphism the category M* (K). The relation & reflexive and symmetric, but not transitive; hence, it is a way of describing proximity but does not provide equivalence relation. Of course,
X' 2 X implies the Hausdorff distance satisfies dh XXX') S K. pointwise 6-perturbation can cause length distortions by 2x, as said formally in the following lemma.
Lemma 3.3 (Data Perturbation Lemma). If X 2 X', then the Rips complexes identified via admit a k- x-interleaving
E R(X, ft
E. R(X,V V)
R(X', V)
Specifically, for any edge = (1), E R(X, 20, the edge fa(e) (z.), R(X', V) has length 2 satisfying 2a 2x S. 2a' < 22+2r.
LX.V) V) defined by the existence of V --geddesic of length
Moreover, the same holds when replacing V with K, under the assumption X 2 X'.
5

---

Proof. Note that geodesic of length 2a corresponds with the intersection of two balls of radius a; hence, the factor of 2. The worst-case perturbation move each of z; and 2j by K in opposite directions, away from each-other, along their geodesic.
Lemma 3.4 (Parallax Interleaving Lemma). Suppose f X _ X'. Ler = P(X, and P' = P'X', For any a, E, these parallax complexes admit (R, intteeetreee
Pa.s St Pa Patn,tt2k
f7 P Po+2n,t4k
Corollary 3.5 (Parallax Interleaving Lemma, Homology Version). Suppose f X E X' Let HP = HP(X, K, V) and HP'- = HP((X', K, V). For any a, E, these homology groups admit a
HPa,c 5, HP If " H aatt
(K. 2n)-eeeteg
The proof of Lemma 3.4 is a worst-case distance estimate given in the Supplemental Material, and Corollary 3.5 follows functorially.
4 Sampling Density and Local Simplicial Matching
The goal "the geometry of K should match the geometry of X" requires that X has sufficient sampling density throughout K to express meaningful comparison. The ideal situation would require there is a (small) scale À for which: (1) K is homeomorphic to UPE. Bx (2:). so that these balls capture the topology of K; (2) all "highly persistent" homological features of X are born before X; and (3) Usex Bx (z) C K, so that perturbations of size À in the dataset are allowed, and so that these balls be used as local charts in K. These sampling properties may or may not be true for any particular pair (X, K), but Definition 4.1 provides scales for comparison.
Definition 4.1 (Locally Simplicially Matched), We say that and K E M(X) are A-locally simplicially matched (LSM) if the subset Pr, C Rg is equality for all t S X. For any K E M(X), the first non-LSM scale realized by the Rips complex is
Asup, x { K) == sup(> : X, K are X-LSM} = min(ev(Y) PK(()) > pv BY E R).
The last LSM scale realized by the Rips complex is
X1o (K) = max {} < (k)). pv()) = = E R).
Another LSM scale is Xball, (K) = max{) USEx Bx(x) C Kº}.
When X and K are X-LSM, we will identify Pzo with R2 so that Pr.o = H. RS whenever S X. Furthermore, locally simplicially matched implies Pt,c Rí for any 0 and / < Asup, (K), following directly from the definition of Paçc-
Lemma 4.2. If K E M(X), then 0 < \ball,((K) < Asup,
However, may be that X1o, (K) 0, if the shortest edge e E R has pv (e) < pp (e). Corollary 4.3. For any K, K' MMX)), there is some X such that each pair (X, K) and (X, (X,K)) is X-locally simplicially matched.
Corollary 4.4. For any X' E (K), there is some 0 < X such that each pair (X, K) and (X', K) is X-locally simplicially matched.
Lemma 4.5. If K1, K2 E M(X) and K1 K2, then >oup, (K) s Xsu,,
The next lemma provides bound Asup under small data perturbations, which is form of stability.
Lemma 4.6. Assume Euclidean V. Suppose that K MMX) nMX)') Is X A X' for I K < 3 >ball (K) then >bball (K) K S >aup.x.
The proof of Lemma 4.6 a triangle-inequalie argument the Supplemental Material. The other results are immediate observations from the definitions.

---

5 Homological Matching
From Section 1, our purpose is to determine whether the geometry of K matches the geometry of X. In Section 4, we introduced "local simplicial matching" as a way to compare small-scale geometry. In this section, we introduce "homological matching" as way to compare large-scale geometry. Generally this is done by asking whether highly persistent features of X in V (as measured by H R) are also highly persistent as features of X in K (as measured by P). This comparison sensible X and K are X-locally simplicially matched, so that cycles can be identified between H Rx = Px We phrase algebraically in Definition 5.1, but Lemma 5.3 provides the interpretation that, among cycles born before X, those of long persistence (Ó X) in H R have even longer persistence (w - X) in I, meaning that K has large-scale homological features corresponding to those of X.
Definition 5.1 (Homologically Matched). For K M( and 0 S X < Xxu,. < 6 w. we say that X and K are (X, 6,w) w) -homologically matched [HM] if the transition maps of HR and HP satisfy ker H Rx-+8 C ker ((.es))) ker R C ker H for some Rips-like path T.
for some 0 S e1 S 82. Equivalently, if
Definition 5.1 is guided by the Void Lemma (5.2) and the Matching Lemma (5.3).
Lemma 5.2 (Void Lemma). Suppose K E (M() and 0 < x (K), Let L be any Rips-like path. If [Y] E H V) with birth < X then the deaths c, d, of [Y] in H HR(,, V). H H(X, K), HL, respectively, satisfy S d < e. Moreover, < implies that K has void that disrupts the death of [Y), and that void contains a ball of radius satisfying (IT 2)r S 2(d c) S 2(e c).
In particular, if the class [Y] has == oo for the "inflexible" path L,, = Pr,o- then K contains a void. The proof of Lemma 5.2 appears in the Supplementary Material and uses simple distance estimates.
Lemma 5.3 (Matching Lemma). Suppose that all pairs in X have distinct lengths, and that X and K E M(X) are (X, 8, W)H Then there is a Rips-like path for which each dot (b, d) in the persistence diagram of I. for bar in the barcode) with b S X < w < d corresponds via X-LSM to dot (b, e) in the persistence diagram of R with S X < 8 <.
Note: to check whether X and K are (A, (.W)).,M,MM... it suffices to check a Rips-like path through Px.o and Pu,co- At the other extreme, we get an overly-strict bound by checking the "inflexible" Rips-like path L, Pr,o-
Because of the filtration stability of HP in Corollary 3.5, (A, 6, HHM also stable to perturbation in X, as seen in Theorem 5.4.
Theorem 5.4 (Stability of Homological Matching). Suppose that X and K are (2, -HM If x R X' such that X' and K are (A K)-LSM, then (X', K) are (A (x-Kō k,W rt)- homologically matched.
The proof of Lemma 5.3 is functorial, and Theorem 5,4 is diagram chase. Both are given in the Supplemental Material.
Definition 5.5. Given X and K E M(X) let X (K) denote the minimum of those ö for which X, K are (Ki, HM)-
6 Interpretation and Specification
Therefore, answer our original purpose, geometric match" for X using the following procedure: (1) Regardless of K, examine the persistence diagram of R(X, V) to identify dots with early birth and long persistence, which TDA theory tells us (e.g. the Homology Inference Theorem [LJ] should indicate genuine geometric features of X; (2) for a model K MXX), compute ((K) and Xpi,x (K); and (3) check whether those dots are born before X1,,(()) (K) and die after Ani..
assess whether model K E M(X) is a "good
If we believe that the multiscale geometric patterns among the points in X is meaningful, then this procedure is essentially a specification for how a "good" model ought to behave.
This process can give "bad geometric match" in various ways, for example: if step (1) does not show a clear collection of well-separated dots, then it is unlikely that X actually has computable geometry that can be captured with the Rips complex; if step (2) yields X (K) 0, then K has
7

---

voids between every pair of points in X, possibly due to under-sampling or over- wver-fittnng, and should be trusted for any interpolative purpose; or if step (3) shows that the quadrant the upper-left of (Aio,x(K)) (K), X (K)) in the birth-death plane does not capture the desired dots of X, then K is failing to capture specific high-persistence features of X.
Computational Methods
In this section, we provide algorithms to estimate Pa.s in the practical case V = R These algorithms are implemented in Python (and development continues) in our open-source software at tot.ddtttaaollloicaallll.. -ararallalllppaalaaaa
https://gi gigltb..bbb........ com/
The Rips complex R can computed efficiently, using [26, 33]. But, the set K is known only through the indicator function k, and the Rips complex R(X, K) cannot be computed directly.
Consider 2, y E X joined by V-geodesic (line segment) zy of length (c), representing a Rips edge e E R(X, V) (e)) We would like to estimate px (e), thus giving E Pa.e for a px(e) and E = a pv (e). Some simple geometric observations allow us to estimate a and E.
Lemma 7.1. If there exists p E Ty such that k(p) 0, then e o Pa.0-
This is because in R", "" is the only path of minimal length.
Algorithm 7.2 (Estimation of e E Pa.0). For each e E Ra. sample points ~ Ty along the corresponding line segment zy. (One method of sampling is simply check the barycenter.) Return True for if and only if "k(p) = 1Vp ~ Ty
Definition 7.3 (Transverse Disk), Let V R". Given an edge R and a radius r, let D,(e) denote the codimension-l disk, oriented perpendicular to ty. and centered at e's barycenter 2
Any continuous path from x to that does not intersect De(e), must have length exceeding 2v(Pv(()) +2. If k (c), {0), then all K K-paths avoid Dr(e), so all K K-paths represent- ing e must have length exceeding 2 (e) + + giving the following Lemma.
Lemma 7.4. If K n (e) = 0. then ((v(ee))² + r2 < px (e) and Zpa(e) < <P((e) - -x(())
For the second inequality, recall the 2nd order Taylor approximation pv(c) 2pv(e)"² S / (e) 2 2
Algorithm 7.5 (Bounding Pa.e). For each R(X,V) V), From 0, loop:
1. Evaluate k<p) for samples ~ Dr(c).
2. k(p) = OYp, increment r. Else, break.
Return the lower bounds V(vv (c)) +r) & a and a - pv
These algorithms can be extended easily sample radii along sequence of points on the edge e, thus providing an estimated K -ath for e.
7.1 Back-Prropggttion
Recent work has shown that various topological properties can be expressed as loss functions that are compatible with nack-propagation methods; for example [9, 30, 32]. The method in [32] allows back- propagation for piecewisc-smooke loss function of the form & (PDiagg()) (f)), where f is the filtration function on simplicial complex, and PDiag(f) f) is the persistence diagram for the f-persistent homology of that complex. Lemma 5.3 allows to interpret homological matching via persistence diagrams PDiag(f(Y'), where f(Y)= = minft Y E L2) for some Rips-like path L.
The following function & could be used to improve homological matching in this framework. Suppose that and K are ALSML.M. Consider the persistence diagrams PDiag R) and PDiag(L) L), where L is some Rips-like path through P(X, K). By AIM-LMM, we know that PDiag( R)) and PDiag( T.) are identical to the lower-left of (Aio; \to). Choose desired target value of Ani- Now, alter the filtration on L by the following 4: for dots (x, y) to the lower-left of PDiag(L). L).
8

---

Pers Diag
Pers Diag of
*
1,230
1,000
2.000
Aue
Auz
0.20
0,500
0.3
025
0.5
0.75
1.00
1.25 1.50
5.00
0.55
0.5
6.75
1,00
1.25
1.50
Birth
Birth
Figure 3: (1) Valid cyclo-octane data under 3D Isomap, colored for viewability. (2) Persistence Diagram of valid data, showing two 2-cycles and one 1-cycle. (3) Persistence Diagram from Parallax of model K2, showing Asup in magenta and three homologically matched cycles moved to infinity, These diagrams use diameter, not radius, via gudhi.
original data
interpolated barycenters
0.00141%
liter polated avie) x
ovten
0.00106%
600071%
8.3%
8O00355
00.0003353
-O 0000001
0:0003779
0.8537
3.3044 3.52C1
bond length,
sL.52
bend length
Á
Figure 4: Distributions of bond lengths for conformations cyclo-octane, Left: Bond lengths for the original data set, distributed tightly around 1.52 A, standard deviation of 4.09 10-5 Å. Right: Bond lengths for all barycenters of edges e with 2pv (e) < 1.0, as would be allowed by the model K₂ described in Figure 3. The bond lengths vary from 1.3844 to 1.5201 Á with mean of 1.5059 Á. Also, bond lengths for all barycenters of edges with 2py (e) 3.0, which are all allowed by the model Kj. The bond lengths vary from 0.8537 to 1.5201 Á with a mean of 1.2876 Å. The extremely narrow blue lines in the Right plot overlays the distribution from the original data set on the Left.
we penalize Wasserstein distance to PDiag(R). For dots (x, v) PDiag(L) with æ Ato < Ani S y. penalize by the quantity || - rol exp(-y), where (Zo, Bo) is the best-match dot from PDiag(R). This loss function should force L to express long-lived topological features similar to those of PDiag(LL L), while minimizing the error introduced at scales below Ålo- As of this June 2023 publication, our code does not implement propaaropng to improve homological matching of models, but it is planned as upcoming work.
Example: Cyclo-Octane
The conformation space of cyclo-octane [18] is well-known have novel topological structure. From physical principles, Martin et al. [22] show that real data sampled from the conformation space X can be reduced lie in consisting of sphere and a Klein bottle. As in Section 1, we suggest that machine-learing model V - (0, 1} trained to recognize cyclo-octane should not be considered "good" or trustworthy unless K {x k(x)= (1)} also takes this geometric form at the appropriate scale. In this section, we demonstrate how topological parallax can support or reject the hypothesis that the geometry of model K matches the geometry of X.
24 and furthermore X must lie near 2-dimensional stratified space

---

Figure 3 visualizes the dataset X C I2 using 3D Isomap projection. Following the workflow from Section 5, we compute the O-, 1-, and 2-dimensional persistence diagrams of X C I2 using gudhi [34], and we observe that there are highly persistent features one 1-cycle and two 2-cycles.* The insight from [22] provides the meaning of these cycles, but the precise structure of the "data manifold" is typically not known priori in examples. What we know in any case is that we want any potential K to respect these cycles, because the geometry of K should match the geometry of X, whatever it might be.
To understand the validity of a generated conformation, we compute its bond lengths the distances between adjacent carbon atoms in the conformation. Bond length is an important physical property, together with bond angle, torsion angle, and energy [18]. Given the rigid geometry assumptions of the cyclo-octane data [22], we expect individual conformations generated by trained models to have similar bond lengths, and therefore the distribution of bond lengths from valid conformations should be similar to that of generated conformations. See Figure 4 to compare the bond lengths of conformations from the dataset X versus those at barycenters for short edges and long edges in the Rips complex R of X. Notably, interpolation across longer edges leads to invalid conformer geometries with too short of bond lengths and thus too sharp of bond angles for realistic molecules.
Suppose someone trains a standard neural network K1 to recognize this data. For this demonstration, we used 3-layer fully connected network with ReLU and a SoftMax, implemented in PyTorch. The network was trained to near perfectt accuracy within a few minutes on two class problem of real data versus a nearby background. (Hyperparameters and training details are provided in the Supplementary Material.) Thus, the model hn represents a common starting point that any data analyst might find encouraging. We apply Algorithm 7.2 to estimate which edges in are accepted by k1, and discover (Xto.x(K)) = 3.45, which is the longest edge available. So, the Rips complex cannot distinguish Ki from the convex hull; the model does not reflect the geometry of X. This is particularly unfortunate in this case, because the model ki might be used to generate many new conformations (any interpolation between two valid conformations), the vast majority of which will not be valid conformations. An alternative model hy is offered, which is built from many local charts (details in Supplementary Material). The new model has 2 (K2) = 1.0. Moreover, the most persistent cycles in X have infinite death as measured by the Rips-like path L, Pt.o. X and K₂ are homologically matched with 2 x 1.25. Therefore, we can claim that the geometry of K matches the geometry of X at these scales.
Limitations
The parallax complex and associated objects are well-defined only for datasets and models that satisfy Definition 1.1. The algorithms in Section 7 assume that V R" with the Euclidean metric. but could be adapted for other geodesic spaces. Computation of the Rips complex and its persistence diagram scale favorably with the intrinsic dimension of the dataset X; however, the sampling methods discussed in Section 7 scale with the dimension of V, which might invoke the Curse of Dimensionality. deeper question is whether real-life datasets X of applied interest actually have enough sample density to exhibit a multiscale metric geometry, to which K can be compared. This question suggests a followup study to verify whether datasets and models with demonstrated real-world efficacy actually have computable and comparable geometries; that is, future work should use parallax to assess the profound epistemological question of whether metric geometry is a valid metaphor for understanding deep learning in real-life applications.
Acknowledgments and Disclosure of Funding
Work by all authors was partially supported by the DARPA AIE Geometries of Learning Program under contract HR001 12290076, and by the National Institute of Aerospace (NIA) under sub-award 021-202066-GDD GDA.. We are very grateful to Bruce Draper of DARPA and Alywn Goodloe of NASA for their technical guidance during these efforts. We are also very grateful to Matt Dwyer, Rory McDaniel, Tom Fletcher, Yinzhu Jin, Jay Hineman, and Joey Tatro for technical discussions.
3 Note that the pinch in the middle is an artifact of the projection and does not represent singularity in the actual dataset.
Gudhi uses diameter, not radius, as the filtration parameter.
10

---

References
[1] Henry Adams, Tegan Emerson, Michael Kirby, Rachel Neville, Chris Peterson, Patrick Shipman, Sofya Chepushtanova, Eric Hanson, Francis Motta, and Lori Ziegelmeier. Persistence Images: A Stable Vector Representation of Persistent Homology. Journal of Machine Learning Research, 18(8):1-35, 2017. ISSN 1533-7928. URL http: //tp:://mmmi jmlr mall... orw.org/./............. wwwggpapps//aaee http////11111111 337 .tmll.
[2] Randall Balestriero, Jerome Pesenti, and Yann LeCun. Learning in High Dimension Al- ways Amounts to Extrapolation, October 2021. URL http: htap://aarxx /ttp///axivvv///a///////aaaaiaaa org/abs/ abs/aa10/ 209.//11.994 1xXiv2211000945. [es].
09485.
[3] Serguei Barannikov, Ilya Trofimov, Grigorii Sotnikov, Ekaterina Trimbach, Alexander Korotin, Alexander Filippov, and Evgeny Burnaev. Manifold Topology Divergence: Framework for Comparing Data Manifolds. November 2021. URL httpp:://ppenrrrtiiee wwwp://wweeeffooriieeeo Fj6KQJbHWKS.
[4] UIrich Bauer, Talha Bin Masood, Barbara Giunti, Guillaume Houry, Michael Kerber, and Abhishek Rathod. Keeping sparse: Computing Persistent Homology revisited, December 2022. URL http://arxiv..rr org/ abtp/ass....//e/aaa/aaa/aaaaaaaa 09075. urXiv:2211.0900 [cs, math].
[5] Mikhail Belkin. Fit without fear: remarkable mathematical phenomena of deep learning through the prism of interpolation, May 2021. http:////arxi brrg/abs/2in 14368. arXiv:2105. 14368 [es, math, stat].
[6] Paul Bendich, J. Marron, Ezra Miller, Alex Pieloch, and Sean Skwerer. Persistent homology analysis of brain artery trees. Ann. Appl. Stat., 10(1),
March 2016. ISSN 1932-6157. doi: 10.1214/15-0AAS886 URL https:
//projecteuclid
Saaatuee//Periisttt--hooooooo 15 A.-AADA886..lll
1214/
[7] Magnus Bakke Botman and Michael Lesnick. An Introduction to Multiparameter Persistence, March 2023. URL http:://att /ttp///rxvvvv/..////////////.a org/ 2203. 1 14289. arXiv:2203, 14289 [cs, math].
[8] Mathieu Carriere, Frederic Chazal, Yuichi Ike, Theo Lacombe, Martin Royer, and Yuhei Umeda. PersLay: A Neural Network Layer for Persistence Diagrams and New Graph Topological Signatures. In Proceedings of the Twenty Third International Conference on Artificial Intelligence and Statistics, pages 2786-2796. PMLR, June 2020. URL https: / //proceedings.co mlr, press/ ttressvv08// 2108/carriere/// html.
[9] Mathieu Carriere, Frederic Chazal, Mare Glisse, Yuichi Ike, Hariprasad Kannan, and Yuhei Umeda. Optimizing persistent homology based functions. In Proceedings of the 38th In- ternational Conference Machine Learning, pages 1294-1303. PMLR, July 2021. URL attps://proceediig mlr. ptressvv339) carriereeiaaaaa ppraseeell/// html.
[10] Chao Chen, Xiuyan Ni, Qinxun Bai, and Yusu Wang. A Topological Regularizer for Classifiers via Persistent Homology. In Proceedings of the Twenty-Second International Conference on Artificial Intelligence and Statistics, pages 2573-2582. PMLR, April 2019. URL https:
// ppwppeeiings.... mlr. www.srreesseee..ee chen19g html.
[11] David Cohen-Steiner, Herbert Edelsbrunner, and John Harer. Stability of Persistence Diagrams. Discrete Comput Geom, 37(1):103-120, Jamuary 2007. ISSN 1432-0444. doi: 10.1007/ s00454-006- 1276-5. URL
1ttp//10.0007/0044400000 1276-5.
[12] Andac Demir, Elie Massaad, and Bulent Kiziltan. Topology-Aware Focal Loss for 3D Image Segmentation, April 2023. Utt.https://ww.bbrr www.://www..ww.o.ooooooo wwtp:/iwwwwwrroootoereeeee/ee//e 1101 21100/2022..0 04. 21. 537860v4.
[13] Swaroopa Dola, Matthew B. Dwyer, and Mary Lou Soffa. Distribution-Awar Testing of Neural Networks Using Generative Models. 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), pages 226-237, Madrid, ES, May 2021. IEEF. ISBN 978- 1-66540-296-5. doi: 10. 109/CSFF439022000 1.00332. URL https: http//eeeeeeeeeeeeeeeeeee/eeeeee ieee.
org/document/9402100//

---

[14] Herbert Edelsbrunner and John L. Harer. Computational Topology: An Introduction. American Mathematical Society, 2010. ISBN 978-1-4704-6769-2 Google-Books-ID LIJJAAAAAAAQAII
[15] Ximena Fernández, Bugenio Borghini, Gabriel Mindlin, and Pablo Groisman. Intrinsic Persistent Homology via Density- based Metric Learning. Journal of Machine Learning Research, 24(75):
1 1-42, 2023, ISSN 1533-7928. URL jaaprr////mmrreeeee/eeeeeeeeeeee wwtppetpr..../p./p/peeeaa paapess/22/21111
.tml..
[16] William H. Guss and Ruslan Salakhutdinov. On Characterizing the Capacity of Neural Networks using Algebraic Topology, February 2018. URL http:://tasstt org/ /a////b///// aa////ass//10// 110//1002220 11880.00044,, 04443. arXiv:1802.04444 [cs, math, stat].
[17] Allen Hatcher. Algebraic Topology. Cambridge University Press, 2001. ISBN 0-521-79540-0.
[18] James B. Hendrickson. Molecular geometry. V. Evaluation functions and conformations of medium rings. J. Am. Chem. Soc., 89(66)7036-7043, December 1967. ISSN 0002-7863. doi: 10.1021/aa0002aaa0
URL American Chemical Society.
1021/ja010020033 Publisher:
[19] Felix Hensel, Michael Moor, and Bastian Rieck. A Survey of Topological Machine Learning Methods. Front. Artif. Intell., 4:681108, May 2021. ISSN 2624-8212. doi:
10.3389/frai.2221...0 11221.681000 108. URL https://wwt www.tiirrnntiiiinnnmmmmmm http:/orrttrrriiiooorrraaaaaaiaa wrg//ariicee//....... 3389/
frai.2021. 681108/ful..
[20] Valentin Khrulkov and I. Oseledets. Geometry Score: A Method For Comparing Generative Adversarial Networks. February 2018. URL https://wwa swwnttcschooar org/ org/paper/
For- -oaeeaaannnnnnneeeettieeeeee www.arrative---auuoo------------ 55sballe6sac2776fffffffaaaaaa2.2
[21] Changliu Liu, Tomer Arnon, Christopher Lazarus, Christopher Strong, Clark Barrett, and Mykel J. Kochenderfer. Algorithms for Verifying Deep Neural Networks. FNT Optimization, 4(3-4):244 404, 2021. ISSN 2167-3888, 21008,22216661 2167-3918.
doi: 10.1561/2400000035. http:tp:
/ wwww.nwwwbbll...
com/
OPP-003.
[22] Shawn Martin, Aidan Thompson, Evangelos A. Coutsias, and Jean-Paul Watson. Topology of cyclo-octane energy landscape. Chem. Phys., 132(23):234115, June 2010. ISSN 0021-9606. doi: 10. 10.101 34442667. 1063//..445267. URL https://aipp /ttp:://///...taaatiiaaaeaaaaaaa wwwiattiiiioooo wwt//oo/oo Publisher: American Institute of Physics.
http://aaaa...oooo////////./.1.. 10t.//.....1111111 34452677
[23] James Munkres. Topology. Prentice Hall, Inc, Upper Saddle River, NJ, 2nd ed edition, 2000. ISBN
[24] Gregory Naitzat, Andrey Zhitnikov, and Lek-Heng Lim. Topology of deep neural networks. J.
Mach. Learn, Res., 21(4):1B447503-11447722 January 2020. ISSN 1532-4435.
[25] Arnur Nigmetov and Dmitriy Morozov. Topological Optimization with Big Steps, March 2022.
URL http: http://wwwxx.i..//.//// http/bbb/220033 227.22223..222.2211 16748. arXiv:2203. 16748 [cs. math].
[26] Nina Otter, Mason A. Porter, Ulrike Tillmann, Peter Grindrod, and Heather A. Harrington. A roadmap for the computation of persistent homology. EPJ Data Sci., 6(1):1-38, December 2017. ISSN 2193-1127. doi: 10.1 000114400ppidd111888880000000000
URL https: httpssustss///ppttettttaaeaaa pppringgropa..com/rrriiic....eee 1ttp:/...11111pppasaassse/ 017-0089----7-00--------
[27] Steve Oudot. Persistence Theory: From Quiver Representations to Data Analysis. American Mathematical Society, Providence, Rhode Island, May 2017. ISBN 978-1-4704
[28] Mathilde Papillon, Sophia Sanborn, Mustafa Hajij, and Nina Miolane. Architectures of Topological Deep Learning: A Survey on Topological Neural Networks, April 2023. URL http://arxivs org/ abs//304/ 10031. arXiv:2304. 10031 [cs].
[29] Vardan Papyan, X. Y. Han, and David L. Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117 (40):24652-24663 October 2020. doi: 10. 10.1073/pnas.20 201.p/aaa...15111111 17. URL https: / //www///////////////// .nas. org/doi/10 1073/ pnas. 2015509117. Publisher: Proceedings of the National Academy of
Sciences.
12

---

[30] Adrien Poulenard, Primoz Skraba, and Maks Ovsjanikov. Topological Function Optimization for Continuous Shape Matching. Computer Graphics Forum, 37(5):13-255 August 2018. ISSN 01677055. doi: 10.111 1/ef.. 13487. URL htttp:://solineeeee caterrar.wlllyy... 1111/cgf. 13487.
wiley. com/doi/10.
[31] Elchanan Solomon, Alexander Wagner, and Paul Bendich. From Geometry to Topology: Inverse Theorems for Distributed Persistence, February 2022. URL http://arxiv ottpv///.rrgaa//11111/1/ abs/2101. 12288. arXiv:2101.122 [es, math, stat].
[32] Yitzchak Solomon, Alexander Wagner, and Paul Bendich. A Fast and Robust Method for Global Topological Functional Optimization. In Proceedings of The 24th International Conference on Artificial Intelligence and Statistics, pages 109-117. PMLR, March 2021. URL https: // prroceddings.... mlr. press/ matsss/v13///so v110/ssllooo211a html.
[33] Guillaume Tauzin, Umberto Lupo, Lewis Tunstall, Julian Burella Pérez, Matteo Caorsi, Ani- bal M.
Alberto Dassatti, and Kathryn Hess. giotto-tda: A Topological Data Analysis Toolkit for Machine Learning and Data Exploration. Journal of Machine Learn- ing Research, 22(39):1-6, 2021. ISSN 1533-7928. URL htt.:tttp//mmmlee org/ http:/aap////eeeaaeeeeeeeeeeeesa vv22/ 30--255.0 html.
[34] The GUDHI Project GUDHI User and Reference Manual. GUDHI Editorial Board, 3.8.0 edition, 2023. URL https:// inria.iiii.... ftt.i/a.../////...00000.
[35] Shiqi Wang, Kexin Pei, Justin Whitehouse, Junfeng Yang. and Suman Jana. Efficient formal safety analysis of neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, 18, pages 6369-6379, Red Hook, NY, USA, December 2018. Curran Associates Inc.
[36] Matthew Wheeler, Jose Bouza, and Peter Bubenik. Activation Landscapes as Topological Summary Neural Network Performance. 202/ IEEE International Conference on Big Data (Big Data), pages 3865-38700 Orlando. FL, USA, December 2021. IEEE. ISBN 978-1- 66543-902-2. doi: 10. 100/////DDaaa225522200111.....11 70.9967766688
URL htt.httt///eeeeelrrreeeee ieee.
org/docuueett/967115..
[37] Afra Zomorodian and Gunnar Carlsson. Computing Persistent Homology. Discrete Comput Geom, 33(2):249-22 274,, February 2005. ISSN 1432-0444. doi: 10.10077/500454-004-11 1146-y.
URL https: :/tp:::::/ddii org/10 10. / 000-0444-000-11-5-5 1146-y.
13

---

A Supplementary Material: Formal Proofs
Proof of Corollary 2.5. The hypothesis [Y] + [0] implies that for some dg S 02 and Eo S E2 there exist chains Y E Po1 and Z Poy.52 and W Pao,.o such that 0z Y W in Pan.ry By Lemma 2.4, these chain can be included to their respective levels in R, giving Y E Roys Z E Rog and W Rao still satisfying 0Z Y W in Rog- Hence, applying 6x, either (x trivial in H Ra or the transition map H + H Ray takes 6x (YT + [0].
Proof of Lemma 2.6. If 2 then the second condition in Definition 2.2 becomes PY( Py (Y) I a, which is a trivial condition for S pV S Pk(Y) & a. Thus, the sets Pa.e are identical for all a.
Proof of Lemma 3.4. Suppose an edge e == (I;, lies in Pa.e, meaning that is represented by a V-geodesic of length (c) satisfying (e) I and K K-geodesic of length 2px (e) satisfying (e) I (k(e) I minfa, + t By Lemma 3.3, the corresponding edge e' = ft(e)
(f())) ((z)) satisfies pv(e) S pv(e') S pr(e)+ +.. Moreover, by the data perturbation lemma on K, we know e' is represented by a K -ggedessic with length 2px (e') satisfying px(ce) S px (e) Combining these overall, we have the parallax bounds
pv(e') S px(e') S min(a, PV (e +
which implies both px S pv + E 2n and px (e') aatk.
Proof of Lemma 4.6. Fix X 2 XbalI.X (K) K. We aim to show that X' and K are X-LSM. Fix any edge e' = (x', 3²) in R' satisfying pr(ee) (((''== S X. (The set of such edges might be empty, which is allowed by Definition 4.1.) For any such e'. there is corresponding
(x, y) E R with f(x) I', f(v) = y'. and fe) = e'. The triangle inequality provides ||y =I S 11 s'll + V1 x' + || - 2X + 2x Abail.((K) We are assuming V is Euclidean, so all balls in V are convex. Since x' y, y' are all within the ball of radius Xball, (K) around x, all their pairwise geodesics are included in K. Hence, e' E P
Proof of Lemma 5.2. The relationship S d follows from Lemma 2.5. The relationship d < follows from Lemma 2.3 and the observations that Pa.a = Pa Pa.so = R(X, K). Suppose L is parameterized as La = Pa.sta)- Suppose that e, either < d or d <.
e < d, then we can conclude that the edge that killed [Y] in H V) is not present (X, K). so that edge intersected Ke, which is open set. Hence, that killing edge intersects a void in the sense of Defn 1.2. Let B, be an open ball in K centered on some point on the V V-geodesic of length 2c. If B,- KS, then the shortest K -geodesii replaces the diameter 2r with the half-cireumfferene ar. Therefore, मर 2r < 2d- 2c.
H. but not for any Ho,, + H..rr()) with a < e. Therefore, there is a K K-geodesic of length 2d and a different K K-geodesic of length 2e, either of which could kill [Y]. The former edge is not allowed I, because 0 < (d) < d cc Hence, d, returning us to the first case.
If d c, then [Y] + [0] via H + HPa. and via H..
Proof of Lemma 5.3. Let L be a Rips-like path given by Definition 5.1. Because X and K X-locally simplicially matched, the filtrations levels of R and L are identical up to A. Therefore, there a bijection between the persistence diagrams of R and L for all dots born by X.
Suppose that (b, d) is a dot in the persistence diagram of T. with b < X < < d. This dot represents class [Y] born in HI. that dies in H The class [Y] H Rs is born at b must die at some value c, so [Y] E ker H Ro-se- It cannot be that c S 6, because that would imply [Y] E ker HLb-ww contradicting w d. Therefore, 8 < c.
Proof of Theorem 5.4. Let fa HP + HP' denote the map induced by / and similarly, let f=1 HR' + H. R denote the map induced by f-1 Consider the following commutative diagram in which
14

---

the unlabeled morphisms are transition maps of the respective persistence modules.
HR
H
1.a
N
HR
HR
HP
HP
1.
HP +2,
H x+
H ++
Note that the bi-degrees in the diagram are determined by Lemma 3.3 and Corollary 3.5. Set N' ker H 1 H Commutativity of the top left square implies that (.' C HRx maps vertically to in HR2 and hence, f='('="'))' (N') lies in ker H -2 The homologically matched By commutativity of the bottom right square, (f (N))) maps horizontally to 0 E HH + +2x Finally, the locally simplicially matched assumption means we can instead think of N' as a subset of HP H +2x which maps to 0 in H P 2+ by commutativity. Thus,
assumption on (X, K) implies f. (N') ker H (A.(e)))
X-K,c1+22
N' = ker HR HR ker H ( -o0-
completing the proof.
B
Supplementary Material: Code
The code supporting this article is initial proof-of-concept. It is available publicly at
It is designed simple Python package, following community best-practices for tooling and layout. Documentation is provided there. We recommend that the reader go to the repository for issue tracking, improvements, bug fixes, testing pipelines, etc.
Because this package relies on GUDHI The GUDHI Project [34], the filtration values are by diameter (not radius). This is important to note, because the theory in the paper is written using radius (not diameter) as the filtration value.
Topological parallax has the same computational complexity as the computation of Rips complexes and their persistence diagrams-add albeit with a larger constant. This is because parallax merely inserts a modell-evlluation step upon the examination of each edge. The constant therefore is t.2 for N points and model that takes time / to evaluate. There are very interesting dimension- and prrucure-dependdende
estimates for the real-life/expected timing of Rips computations, such as Bauer al. [4]. Distributed persistence can parallelize this process, as Solomon al. [31]. See https: // g tlab. com/ getodato//aa/////e /dispers.
Figure was produced with the Jupyter notebook notebooks/Winky Example ipynb. Compare and contrast the Python scripts rumners/ru,wwikky.mg versus
meineersrru_winky.. ttankyytree.py
Figure 3 was produced with the Python script rumners/rw__cycloottaee...o .py. Compare and contrast that script with nun_cyclooctane aateee_nnulll_am py. These examples may take 100-500 GB of RAM to com- pute the persistence diagrams as currently implemented.
wwweersrru__cccclleeeeeeeee eereernnnaar_eeeeeeeeeeeeeeeeeee py and
Figure 4 was produced with the Python script runners/bond leigmnhd_ennhhhppppppp
15

---

C
Supplementary Material: Utah Jar Example
This example demonstrates situation where a classifier appears to 100% correct on a semanti- cally meaningful dataset, but the resulting model is too strict for interpolation within data class, Topological parallax detects this phenomenon, as discussed in Section
We applied parallax to an imagery dataset inspired by the "Utah teapot." The dataset consists of 14000 images-4000 images of rotated teapot and 10000 images of a rotated teapot with no spout or handle (referred to as the "Utah jar"). See Figure 5a. All images were rendered with PoV-Ray.
We constructed bespoke classifier this dataset of images which accepts any image within distance of 1.0 to the set of jar images using the Euclidean metric on flattened images. Our classifier rejects all other inputs. We applied parallax to this model and found very small values for both A and Asu, with A
Aoup 2. See Figure 5b.
Given the hard cut-off distance of 1.0 in the classifier and these À values, we expect the model to prohibit reasonable interpolations of images. Indeed, we found numerous interpolations rejected by the model which seem valid the human eye. See Figure 5c.
(a) Examples from the modified Utah Teapot dataset.
Diag of
4.000
3.000
2.000
Aspp
0.000
(c) Interpolative images that are visually similar to true dataset samples but are rejected by the overly sensitive model.
Birth
(b) The persistence diagram dataset marked by parallax.
Figure 5: Using parallax detect dataset-modé geometry mismatch. (a) The images consist of the Utah teapot (rendered with PoV-Ray) with spouts and handles removed for simplification. (b) We construct simple classifier which accepts any images within a Euclidean distance of 1.0 of the (flattened) image and rejects all others, and apply parallax to it. (c) Six interpolated images are shown, all of which look very similar the original dataset but rejected by the model (corresponding to the X marks in (b)).
16

---

D Supplementary Material: Understanding the Bi-Complex
The parallax complex P(X, K, V) is bi-filtered by the parameters a (the length of geodesic) and e (the distortion of geodesic length between K and V). Figure 6 provides some visualizations that may help the reader interpret traditional barcodes of Rips-like paths, and how these bi-filtration parameters may be estimated. These parameters are related to the size of voids in the model, as seen Lemma 5.2.
birth
death
Ha
pV + 2
PV
barcode of L
Figure 6: Left: Births and deaths are bi-filtered in HP,, and are observed by barcodes on Rips-like
paths. Cor 3.5 means this picture is stable within [k, 2x). Center: An edge (z, y) Pa.e can be estimated by sampling tubular neighborhoods. Right: Algorithm 7.5 underestimates o, E.
17

---

E
Supplementary Material: Table of Symbols
Notation
Plain Mcaning
First Appearance
Term
geodesic space, such as I2" finite set in V
p.2
Ambient Space
X
p.2
Dataset
&
A perception function on V Support set k
p.2
Model (as function)
K
p.2
Model (as set)
M(X)
Models compatible with dataset Datasets compatible with model K Interior of set K in topological space V
p.2
(K)
p.2
K
p.2
K
Closure set topological Complement of set in Bounded open set in K
p.2
K
p.2
Ω
p.2
Void
R(X,))
Rips complex of X in geodesic space K a filtration level radius Geodesic ball of radius a about Chain in Rips complex Minimal filtration radius for Y in R(X, K) Gap in filtration radius between V and K Parallax bi-complex for X, K, Homology of P
p.4
p.4
(x)
p.4
p.4
Pk(Y)
p.4
p.4
Parallax
HP
p.4
L
A 1- 1-parameter path through P Homology of T,
p.4
Rips-like Path
HL
P.4
Pointwise perturbation of X in V Pointwise perturbation of X in K Induced map on simplicial complex Push-forward map on homology
p.5
Perturbation
p.5
K T-PPeturbation
ft
p.5
fe
p.5
A.
A meaningful filtration value. See subscript.
p.6
18


Reviews:
[{"review_id": "0mpUxAX9fe", "reviewer": "Reviewer_YL9S", "summary": "The authors propose a method to evaluate how well a model learned a data distribution based on topological data analysis. The authors assume that the model is a classifier (the output of the model is binary, or consists of a finite set of classes that can be evaluated separately), and topologically compare the set of positive data samples to the set of data points where the model outputs a positive value. Topological properties are computed using filtrations of the Rips complex on the data points. The quality of a model is evaluated by checking 1) at which scale the complex in the full ambient space starts to diverge from a complex that is restricted to the subspace where the model is positive, and 2) if persistent features of the full complex are also present in the restricted complex. The authors given an example of measuring and comparing the quality of two models applied the a small dataset with known geometry of the data manifold.", "questions": "A preview of the discussion of the advantages/disadvantages compared to related work might be useful, as wall as a discussion of the issue with evaluating regions outside the convex hull of the data samples.", "limitations": "A few limitations have been discussed, including that the authors are not certain if the method would work on more complex real-world data.", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 3, "strengths": "- The approach for measuring the similarities between the manifold learned by a model, and the manifold implicitly described by a set of data samples seems interesting, and is novel as far as I can tell (although I am not familiar with topological data analysis).\n- It seems like it could be useful for finding adversarial examples for a model, or identifying regions where a model does not perform well.\n- If this approach would work with real-world data (or if it can be extended to do that), it could applicable quite broadly to evaluations of models that learn a distribution in high-dimensional spaces.", "weaknesses": "- The empirical evaluation is not thorough enough. I do not have a good intuition in which situations the proposed measure of model quality would be useful, and I do not think it is obvious from the description of the method (see below for a discussion). Therefore, a more thorough empirical evaluation is needed to show in which situations the measure is useful. Specifically, more datasets should be evaluated, and the metric used to evaluate success could be improved as well: the current metric using bond lengths is specific to the dataset used, and would not be useful for comparing the performance across different datasets. A more useful metric might be to use a dense held-out test set, and compare the model quality predicted by the proposed method to the model quality according to the test set.\n- The method is not compared to any alternatives for measuring the quality of the distribution learned by a model, or for finding adversarial examples. For example, what are advantages/disadvantages compared to the standard approach of using a held-out test set? (I can imagine that the proposed method might have advantages if the test set cannot sample the space densely enough, but disadvantages for regions outside the convex hull of the training samples.) Or compared to other methods for finding adversarial examples (there is a large body of literature, a lot of work can be found, for example, by searching for \"adversarial examples\"  in Google Scholar)? This should be at least discussed in the related work, and the advantages/disadvantages are not clear from a theoretical standpoint, ideally an empirical comparison should be provided.\n- It seems like the measure could be less useful for detecting erros of the model (false positives or false negatives) outside the convex hull of the data samples, since the space outside the convex hull is not explored by the Rips complex. But it seems like real-world data might have true positives outside convex hull (i.e. the true data manifold may extend significantly outside the convex hull of the data samples). See details for a discussion.\n- The computational complexity of the method is not provided. Ideally the authors should provide the number of evaluations of the model that is typically needed, in addition to the overhead from building the Rips complex and computing the algorithm described in Section 7.\n- The exposition is very hard to follow for non-experts in topological data analysis, and additionally some symbols/variables are undefined (if these are common symbols in topological data analysis, I am not familiar with them).\n\nMore details:\n\n* More empirical evaluation is needed to understand in which situations the proposed measure gives good results. From the description of the proposed measure alone, it does not seem clear to me in which situations the measure gives a good estimate of model quality. On one hand, I can see how topological properties could be relatively stable descriptors of a data manifold in high-dimensional space, but on the other hand, I do not have a good intuition how the data manifold typically behaves in high-dimensional space. For example in the manifold of natural images or their latent features, I can imagine that the true data manifold could lie significantly outside the convex hull of the data samples, and that it would be hard to capture enough data samples to cover the full data manifold in their convex hull. In that case, a good model would not just be a \"thickening\" of the manifold given by the data samples (as described by the authors), or even a thickening of their convex hull, but would also include samples significantly outside the convex hull. For example, a dataset like CLEVR may contain images showing a blue sphere at multiple random locations in the image, but some regions of the image may not be covered by the blue sphere in any of the samples; in that case, we would still expect a good classifier of the data manifold to also classify images as positive where the blue sphere is in a location that was not directly observed in the data samples. Thus, it seems to me that a good model needs to extrapolate significantly outside of the convex hull of the data samples. Would the Rips complex constructed on the data samples be able to capture the part of the true data manifold that extends beyond the convex hull of the samples? And would the proposed method therefore be able to handle such cases, which might be quite common in real-world datasets? Arguably the manifold would be better behaved in latent spaces that have a more semantical representation of the data, but this then means that the latent space that the proposed measure is applied to has to be chosen carefully, and it is unclear how to choose the latent space.\n\n* The exposition is missing definitions in several places:\n\n    - In Section 1, there are a few missing definitions:\n        - $K^\\circ$ is not defined.\n\t\t- $\\overline{K}$ is not defined. Does this denote the set complement? If so, would this not mean that $K^\\circ$ is the complement of $K$, and in that case, why have two different notations for the complement?\n\t\t- $K^c$ is not defined. It could also be the set complement, but then there would be three different notations for the complement, so I guess both $K^\\circ$ and $\\overline{K}$ do not actually denote the set complement.\n\t- In Section 2, there are several missing definitions that make it hard to follow the exposition:\n\t\t- A Rips complex is not defined. Rips complex may be well-known in topological data analysis, but I think that only a small fraction of NeurIPS readers will be experts in topological data analysis, so giving a short definition would be good. Also, even if the Rips complex is known to readers, the arguments to the Rips complex that are used here may need to be defined. The first argument X is quite clear, but second argument $K$ is less clear. Does the second argument restrict the metric used to construct the Rips complex to geodesics in the subspace $K$?\n\t\t- A filtration of a Rips complex is not defined.\n\t\t- A chain $Y$ is not defined.\n\t\t- $B$ is not defined.\n\t\t- In Corollary 2.5, $i_*$ is not defined.\n* Eq. 2.2 could probably be simplified. Since $K$ is defined as subspace of $V$, $\\rho_V(Y) \\le \\rho_K(Y)$ is true for all $Y \\in R$ (according to Lemma 2.1) and does not need to be mentioned explicitly in the definition. Therefore a simpler definition would be: $P(X, K, V)_{\\alpha,\\epsilon} =$ {$Y \\in R\\ |\\ \\rho_K(Y) \\le \\rho_V(Y) + \\epsilon, \\rho_K(Y) \\le \\alpha$}\n* in Algorithm 7.5, step 2, should f(p) be k(p) instead?\t\t\n* On Line 21: I would define $K$ as {$x \\in V\\ |\\ k(x) = 1$}, since $k^{-1}$ is not well defined for non-injective functions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for helping identify sections that may be difficult for readers due to the field-specific jargon. We agree that clarification of notation is extremely important to readability to a broad interdisciplinary audience.\nPlease see the discussion of clarifying notation in the general rebuttal.\nHere are some specific changes that may be beneficial to the exposition. In addition we will include\nFigure 1 in the attached PDF to help the reader visually connect our definitions to meaning.\n\n- In Definition 1.1, add prose: We define a model \\\\(K\\\\) to be the closure of an open set, colloquially known as a \"solid,\" \\\\(K= \\\\overline{K^\\\\circ}\\\\). For any finite dataset \\\\(X\\\\), we consider the collection \\\\(\\\\mathcal{M}(X)\\\\) of all models for which \\\\(X\\\\) is contained in the interior of the model, \\\\( X \\\\subset K^\\\\circ \\\\).\n- In Definition 1.2, add prose: A void is a bounded open set \\\\(\\\\Omega\\\\) in the complement of a model, \\\\(\\\\Omega \\\\subset K^c\\\\), such that ...\n- In Section 2, Equation (2.1), we will introduce basic definitions of Rips complexes on subsets and cite standard TDA textbooks.\n- To clarify the notion of a \"chain\" by adding the parenthetical \"(a formal sum of simplices)\" and referencing Hatcher's *Algebraic Topology*.\n- Just before line 105, add a sentence: Let \\\\(B_\\\\alpha(x)\\\\) denote the closed geodesic ball in \\\\(V\\\\) centered at \\\\(x\\\\) of radius \\\\(\\\\alpha\\\\).\n- Note that \\\\(\\\\iota_*\\\\) in Definition 2.5 is defined in the previous sentence; \\\\(\\\\iota_*\\\\) is the homomorphism on homology that is induced by \\\\(\\\\iota\\\\) on complexes.\nTo help orient the reader, we would provide a citation to *Algebraic Topology* by Hatcher.\n- We agree that Eq 2.2 can be simplified as stated.  It was written as shown to emphasize the two-sided bound on \\\\(\\rho_K(Y)\\\\), but either way is acceptable.\n- On line 21, and also line 290, replace \\\\(K=k^{-1}(1)\\\\) with \\\\(K=\\{x \\\\in V : k(x)=1}\\\\) to avoid any confusion regarding the pre-image notation.\n- Yes, in Algorithm 7.5, step 2, \\\\( f( p ) \\\\) should be \\\\( k( p ) \\\\).\n\n### Regarding comparison outside the convex hull\n\nWe agree that models with good generalization will necessarily allow extrapolation outside the convex hull of the training set.\nThis idea is expressed in three ways in the submission.\n\nFirst, in Section 1.1, lines 59--79, we comment on the relationship of reference [2] *Learning in High Dimension Always Amounts to Extrapolation* and the related topology.\n\nSecond, our definitions of \\\\(X\\\\) and \\\\(K\\\\) in Definition 1.1 require that the dataset \\\\(X\\\\) is contained in the *interior* of the model \\\\(K\\\\), so there are necessarily points outside the convex hull of \\\\(X\\\\) that \\\\(K\\\\) would accept.\n\nHowever, the essence of parallax is to ask ``what can I detect about \\\\(K\\\\) using only \\\\(X\\\\)?''\nWe do not assume that \\\\(X\\\\) is an original training set, only that it is an available dataset.\nIf more datapoints were available from an additional source,\nsuch as a generative model or from additional data collection,\nthen those points should also be used for parallax.\n\nThird, recall that the main result is a stability result (Theorem 5.3),\nbased on pointwise perturbation (Section 3).\nThus, it exists to provide confidence that these methods remain consistent, even if the dataset is pushed \"outwards\" (or any other direction) by a modest amount.\n\nWe absolutely agree that semantic interpretation of the latent data manifold\nis an extremely slippery concept,\nand we hope that parallax helps capture one (incomplete) aspect of that relationship.\n\n### Regarding comparison to other methods\n\nWe agree that it would be beneficial to the field to compare parallax to other ways of measuring generalization and robustness.\nThe most comprehensive reference here is\n*Fanstastic Generalization Measures and Where to Find Them* by Jiang et al (arxiv id 1912.02178).\n\nAs discussed in the general rebuttal, we do plan on a broad survey that applies parallax to a wide variety of datasets and network architectures,\nand that survey we will attempt to compute as many of these as is practical.\n\nWe could add a brief conjectural discussion to this submission; however many of the generalization measures available do not have\ntopological foundations (instead, information-theoretic or statistical foundations),\nso meaningful comparisons are difficult without a years-long interdisciplinary research program.\n"}, {"review_id": "VCvv5jcIdP", "reviewer": "Reviewer_kfxD", "summary": "The paper introduces the concept of parallax as a bi-filtered persistence module that measures the geodesic distortion between the dataset and the model. The paper also proves that parallax is stable under perturbations of the dataset and provides a criterion of homological matching to assess whether the model captures the persistent features of the dataset. The paper demonstrates the effectiveness of parallax on two models using the cyclo-octane dataset and discusses the limitations and future directions of the method.", "questions": "1. This paper presented a novel theorical way to measure the performance of model by analyzing the geometry of model and data. However, we already have lots of metrics, such as accuracy, which can be measured once the trained and the corresponding dataset are available, it is not clear to me how the proposed method advance the existing method. \n\n2. The paper is more related to math/statistics, while the analysis seems to be very solid. The analysis of overfitting and generalization capability (especially under the setting of covariate shift), which is claimed as the major contribution by the authors, is not easy to follow, more concrete examples are needed for illustration.\n\n3. This paper does not discuss the details of models, i.e., ConvNet, RNN, transformer. I am curious whether the proposed theory could be applied to all models trained by back-propagation?\n", "limitations": "See above", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "This paper presents an interesting and novel approach to evaluate the geometric similarity between a dataset and a model using topological data analysis. The paper is well-written, clear, and provides sufficient background and motivation for the problem. ", "weaknesses": "There is little experimental analysis on real-world dataset. Perhaps the authors could consider merging some part into the appendix and include more analysis.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Regarding real-world datasets\n\nWe agree that further exploration of real-world datasets is strongly desired in future work.\nSee the comments in the general rebuttal as well as the included PDF for an additional example\nof using parallax to interpret a model of an imagery dataset.\n\n### Regarding comparison to other methods\n\nWe agree that there are a variety of well-studied metrics for understanding the performance of models. We\nbelieve that parallax can be used in conjunction with other metrics to provide additional\nevidence that a model is well-behaved. For example, it is now common to train models to (near) perfect test \naccuracy, and in such scenarios, metrics like accuracy provide no distinguishability. In fact, this is one of the\nmajor points made by the Belkin's survey *Fit Without Fear*, which we discuss briefly in the paper but\nare happy to add more detail about.\nParallax may be used to differentiate and highlight certain models compared to others.\n\nWe agree that the connection between parallax and generalization capability is not fully explored in this work.\nOur intention was to introduce the idea of homological matching provided by parallax, prove statements about its stability, and \nshow via an example it matches what we believe is a widely held intuition about \"the shape\" of manifolds under\nthe manifold hypothesis. The cyclo-octane example and modified Utah-Teapot example do go a long way, we feel towards demonstrating that perception models\nwhich satisfy our criterion accept points which are reasonable and reject points which are not, and that the opposite is true\nof perception models which do not satisfy our criterion. This is intuitively connected to the idea of generalization capability,\nhowever we agree that it is far from a rigorous argument connecting our criterion to precisely quantified statistical\nconcepts like the generalization gap. On the other hand, as the *Fantastic Generalization Measures...* survey shows, such\nrigorous arguments are few and far between in the literature. We are planning a new paper with a large scale experimental suite\nattempting to generate such a rigorous argument, at least empirically.\n\n### Regarding details of models (ConvNet, RNN, transformer, etc)\n\nA fascinating aspect of parallax is that it does not depend whatsoever on the architecture or the training system of the model \\\\(k\\\\).\nAs long as an evaluation oracle is available, parallax will apply.\n\nThus, parallax provides a method to compare how well different models match the topology of a dataset,\nwithout relying on any architectural\nproperties of the models.\n\nAs discussed in the general rebuttal,\nwe do intend to pursue this sort of comparison, particularly for image datasets, in followup work.\n"}, {"review_id": "k7soWoNVNd", "reviewer": "Reviewer_GC8k", "summary": "In this paper, topological parallax is introduced to compare a trained model with a dataset and determine if they share similar multiscale geometric structures. The authors argue that the model is \"good\" if the geometries are similar. To determine the similarity between the model and the data, they calculate a homologous matching that can be applied to many ML systems. This method can thus be used to assess whether a model has good generalization or is robust to perturbations. The authors validate topological parallax with a toy data set, and the qualitative and numerical results support the authors' claim.", "questions": "How can topological parallax be used to derive geometric regularities and improve neural network training?", "limitations": "The limitation is well discussed.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Topological data analysis is emerging as a powerful tool for understanding AI systems. Based on TBA, the proposed topological parallax measures the geometric similarity between model and data, which helps to understand whether the trained model is good or not. Moreover, it has many good properties, e.g., it is stable to perturbation, which is crucial for AI attack detection.", "weaknesses": "I think that further experiments could be helpful to understand the proposed tool. For example, Figure 3 shows only the results of one neural network, and it is expected to show the comparison of multiple AI models using topological parallax. For example, compare the decision tree model and the neural network in Figure 1.\nAlso, try to compare several known neural networks. For example, compare a ReLU MLP with a Sin MLP. Given an image, the ReLU MLP may not fit the data well, but if you replace ReLU with Sin in the MLP (called Sin MLP), the latter method can fit the data with almost zero error.\n\nSin-MLP, also known as SIREN, was described in the article \"Implicit neural representations with periodic activation functions\".\n\nThe reviewer is not familiar with this area and therefore basically respects the other reviews in principle.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Regarding further experiments\n\nWe agree with the reviewer that further exploration of real-world datasets is strongly desired in\nfuture work. Please see the comments in the general rebuttal for this regard as well as the PDF of attached figures for an example of parallax applied to a model on an imagery dataset.\n\nWe thank the reviewer for the suggestion of comparing the decision tree model with the neural network\nof Figure 1, as well as comparison with Sin-MLP and Relu-MLP. The comparison within Figure 1\nwould be easy to perform, and importantly, would provide readers with another\nexample of parallax and its interpretability. We will put this discussion and analysis in the Supplementary Material.\nThe comparisons between Sin-MLP (SIREN) and ReLU-MLP are quite interesting, and we would like to pursue these and additional\ncomparisons in followup work, in order \nto keep this submission from being increasingly complex.\n\nAs the reviewer noted with Sin-MLP, we may assume that all methods will have \"zero\" or \"almost zero\" test error.\nOne of the motivating ideas behind parallax is to distinguish models in the cases when traditional metrics (such as\naccuracy) give (near) perfect accuracy and thus are indistinguishable from the lens of the metric. We believe that\nparallax provides an additional metric by quanitifying geometric consistency between model and data.\n\n\n### Regarding deriving geometric regularities and improving training\n\nRecent advances in topologically-inspired loss functions make including topological\nproperties within the loss function feasible (see refs [7, 22, 24] in original submission).\nThe workflow laid out in Section 7.1 highlights how parallax can be used in neural network\ntraining. We hope to implement this technique in code soon and describe its implementation\nand a suite of experiments in future work.\n"}, {"review_id": "WkhqiWYDKk", "reviewer": "Reviewer_3L38", "summary": "This paper introduces topological parallax, a theoretical framework for analyzing the similarity of multiscale geometric structures between datasets and models. It estimates the topological features in the model by examining the effect on the Rips complex of geodesic distortions using the reference dataset. It shows the stability of the proposed framework under dataset perturbations. It also provides a practical computational method on top of the theory.", "questions": "- The theoretical framework is built upon a binary classification problem. Is it potentially generalizable to more complex model outputs, e.g. multi-class classification (especially when different output channels are inter-dependent)? How would the geodesics be defined in such a case?", "limitations": "Limitations are well-discussed in the paper. As a theory work, I believe it won't have direct social impacts.", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "- The paper provides a novel point of view. It claims that it is the first work to use TDA to express a desired geometric relationship between models and datasets, and to the best of my knowledge, I don't see any existing work studying this problem.\n- This paper provides interesting insights into the concepts of \"overfitting\" and \"generalization\" of neural networks, which are important for the safety and robustness of AI.\n- It proposes a theoretical framework with abundant derivations and proofs.\n- It also introduces a practical computational method and demonstrates it on concrete data and network examples.", "weaknesses": "- Section 8 shows an interesting example of a data space with novel topological structures. But is this method also applicable to other real-world scenarios such as image recognition? What is the complexity of the computation w.r.t. dataset size and dimensionality?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Regarding real-world scenarios such as image recognition\n\nYes!  See the general rebuttal for a discussion of further experimentation with imaging datasets.\n\n### Regarding computational complexity\n\nThe computation of parallax is the same \"big-O\" as the computation of Rips complexes and their persistence diagrams---albeit with a larger constant.\nThis is because parallax merely inserts a model-evaluation step upon the examination of each edge.\nThe constant therefore is \\\\(tN^2\\\\) for \\\\(N\\\\) points and a model that takes time \\\\(t\\\\) to evaluate.\nThere are very interesting dimension- and structure-dependendent estimates for the real-life/expected timing of Rips computations,\nhttps://arxiv.org/abs/2211.09075\nand, we would be happy to include a brief discussion of these considerations in either Section 7 or the Supplemental Material.\n\n### Regarding multi-class classification\n\nThis is a very interesting question.\nThe theory is presented in this submission for single-class perception problems,\nbut as emphasized by the reviewer,\nmany key applications will involve multi-class problems (such as overlapping families in multi-label imaging datasets).\nTo handle these situations, we typically consider each label separately (against the others) or study in semantically meaningful collections of label.\n\nThe overall theme in this case is that the geometry of each class and the combined geometry of relevant mixtures of classes should be respected by the respective models.\nOne can manipulate the definition of the model oracle \\\\(k\\\\) to account for relative likelihoods of various labels, and then study the output of parallax as those relative likelihoods vary.\n\nWe can include a brief discussion of this consideration in the Supplemental Material.\nA more detailed analysis of multiclass problems, especially in the context of imaging datasets, will occur in a forthcoming manuscript focussed on that topic.\n"}, {"review_id": "sEnFDylyOM", "reviewer": "Reviewer_xrbA", "summary": "The authors suggest that a model is good if its geometry matches the geometry of data, and introduce a persistent-homology based method to evaluate this similarity.", "questions": "(Q1) Can you motivate the name Parallax?\n\n(Q2) You write: “We suggest that a model K is ‘good’ if the geometry of K matches the geometry of X.” How is this related to “If a certain architecture is incapable of expressing a decision region that is equivalent in topology to training data, then there is no hope of it ever generalizing to the true data” in [1]? More generally, articles [1]-[3] also consider persistent homology of the data and model, can you comment if and how they are related to your approach?\n\n(Q3) Can you elaborate on the relationship between your work and the related articles you mention in Section 1.3, in particular with [17] and [28]? \n\n(Q4) Where is the proof of Lemma 2.6?\n\n(Q5) “In Section 4, we introduced ‘local simplicial matching’ as a way to compare small-scale geometry. In this section, we introduce ‘homological matching’ as way to compare large-scale geometry.” Should this not be mentioned earlier in the paper? Is it possible to provide an illustration?\n\n(Q6) “We apply Algorithm 7.2 to estimate which edges in R are accepted by k1, and discover 2λlo,X(K1) = 3.45, which is the longest edge available. So, the Rips complex cannot distinguish K1 from the convex hull; the model does not reflect the geometry of X.” How do you make this conclusion, can you elaborate? This is related to weakness (W1).\n\n\nOther minor comments:\n\n- Line 28: should be applied -> could be applied?\n- Figure 1: It would be better if the axes ranges are also provided in the left plot, to make a connection with its PD.\n- Line 168: Def’n -> Definition\n- Explicitly mention that the code is made publicly available.\n- Is Limitations Section 9?\n\n\n[1] Guss, William H., and Ruslan Salakhutdinov. \"On characterizing the capacity of neural networks using algebraic topology.\" arXiv preprint arXiv:1802.04443 (2018).\n\n[2] Ramamurthy, Karthikeyan Natesan, Kush Varshney, and Krishnan Mody. \"Topological data analysis of decision boundaries with application to model selection.\" International Conference on Machine Learning. PMLR, 2019.\n\n[3] Khrulkov, Valentin, and Ivan Oseledets. \"Geometry score: A method for comparing generative adversarial networks.\" International Conference on Machine Learning. PMLR, 2018.\n", "limitations": "Limitations are discussed in the final section. ", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "There is a number of theoretical results that seem to support the soundness of the proposed approach.", "weaknesses": "(W1) The paper is extremely hard to read, which significantly limits its impact. For example, you mention that you rely on bi-filtered persistence module, but it is still unclear to me what the two filtrations are (what is reflected by alpha and epsilon), and why each of them is useful. As another example, rho_K(Y) is a crucial notion for your work, but you never name or describe it, let alone motivate it or provide intuitive explanation; also, this requires an explicit Definition. As a final example, Figure 2 remains a mystery. \n\nAll definitions and lemmas could be named, but more importantly, they could be first motivated and then interpreted. The latter could be done in the beginning of a section, by summarizing all the results and their relevance in a paragraph or two, or throughout the section in a story-like manner. The same holds for the whole text, e.g., when you describe your procedure, you could describe the concepts in words (even if you did this before, it is good to remind the reader, especially since the paper is extremely notation-heavy):  (2) for a model [nice] K ∈ M(X), compute [what is this] λlo,X(K) and [what is this] λhi,X(K)”, or “In this section, we provide algorithms to estimate [what is this] P_{alpha, epsilon}”. A well-organized notation table might also be helpful.\n\nFor this reason, a lot of my feedback is an educated guess, and I am definitely open to substantially changing my score for this paper if it is presented more clearly, so that I can actually assess the soundness and contributions.\n\n(W2) Experiments consider only one data set.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Re: the bi-filtration\n\nThe bifiltration is the most technically challenging concept in this project, but necessary for the stability results (Lemma 3.4 and Theorem 5.4).\n\nThe first parameter \\\\(\\\\alpha\\\\) is the filtration radius (that is, half the length) of \\\\(K\\\\) geodesic.\nThe second parameter \\\\(\\\\varepsilon\\\\) is the difference between the length of that \\\\(K\\\\) geodesic and the corresponding \\\\(V\\\\) geodesic.\nThis meaning is embedded in Definition 2.2.\n\nConsider two antipodal points on a unit circle, where the model is a tight annulus matching that circle.\nIn the ambient \\\\(V\\\\), the Rips edge between those two antipodal points would have filtration radius 1.\nIn the annular model \\\\(K\\\\), the Rips edge between those two points would have filtration radius \\\\(\\\\approx \\\\pi/2\\\\), because the geodesic has to go around the circle, so \\\\(\\\\varepsilon \\\\approx \\\\pi/2-1\\\\\\) in this case.  \\\\(\\\\varepsilon\\\\) keeps track of the local distortion of length between \\\\(V\\\\) and \\\\(K\\\\). This can be used to estimate the size of voids and perhaps be used to make estimates of the curvature of \\\\(\\\\partial K\\\\), for example. We would be happy to elaborate upon the meaning of the \\\\(\\\\varepsilon\\\\) parameter briefly, near Definition 2.2.\nWe could also discuss the use of \\\\(\\\\varepsilon\\\\) to estimate the size of voids in the Supplemental Material.\n\n### Re: \\\\(\\\\rho_K(Y)\\\\)\n\nThe definition of \\\\(\\\\rho_K(Y)\\\\) is in lines 105--107, where the Rips complex is defined.\nFor a particular edge \\\\(e=(x_0, x_1)\\\\),\n\\\\(\\\\rho_K(e)\\\\) is half the geodesic distance between \\\\(x_0\\\\) and \\\\(x_1\\\\) in \\\\(K\\\\).\nWe agree that this jargon can be difficult for readers outside our speciality.  We will help orient the reader by providing this brief clarification along with a reference.\n\n### Re: Figure 2\n\nFigure 2 is intended to show the structure of the bi-filtration, and how births\nand deaths of the bifiltration can be detected using TDA. Figure 2 is not crucial, and we will replace it with\nFigure 1 of the included PDF. \n\n### Re: naming of lemmas\n\nWe agree that this is a good strategy, and we named the definitions, theorems, etc. which we see as the most crucial statements.\nA minority of statements\n(2.1, 2.4, 2.6, 4.2, 4.3, 4.4, 4.5, 4.6, 5.2, 5.4, 7.1) were unnamed because they provide intermediate technical bounds needed elsewhere.\nWe are open to additional suggestions for which statements deserve specific names.\n\n### Re: motivation\n\nSections 1.1 and 1.2 were intended\nto give this sort of story-like motivation and summary, but\nit is clear from reviewer comments that they could be improved. We \nwill rephrase section 1.2 to provide a clear\nroadmap of the major definitions and lemmas, and add a table of notation to\nthe Supplemental Material to help orient a reader. We will replace Figure 2\nwith Figure 1 of the attached PDF to help elucidate the main ideas behind parallax\nand the importance of understanding the geometry of models.\n\n### Re: (W2)\n\nWe agree that the paper would be improved with further exploration of parallax on other datasets,\nin particular on imagery datasets. We are planning a suite of experiments in which\nwe will apply parallax to a broad set of commonly used imaging data, but could\nnot include it here due to page constraints. The present work is meant to \nintroduce the main definitions and provide justification for the theorems and ideas of the paper. \n\n### Re: (Q1)\nYes! Parallax was named by analogy to the method in astronomy. There is an\ninaccessible object that cannot be measured directly (in this case, the\ngeometry of the model \\\\(K\\\\)), so we must infer its location by comparing\nmultiple observations from the available vantage points (in this case, the\npoints in the dataset \\\\(X\\\\)).  We will add a sentence to this effect\nto the introduction.\n\n### Re: (Q2)\n\nWe thank the reviewer for reminding us of this paper and we will\nreference it. The key difference between this work and ours is that our theory quanitifies the\nability of a specific model, independent of parameters or network architecture, to match\nthe shape of a dataset in a specified way. The Guss et al. paper discusses the ability of\nany model produced from a specific network architecture to perform well on datasets of a specific\nshape.\n\n### Re: (Q3)\n\nBoth of these references track and study the topology of a dataset as it evolves\nthrough the layers of a neural network. They make empirical claims about what\ntends to happen as training accuracy increases, but do not provide\ngeometric specifications about what should happen if one is to trust the model,\nand these specifications are sorely needed because, as has been shown in the literature,\nmetrics such as perfect training accuracy are not sufficient. \n\n### Re: (Q4)\n\nThe proof is as follows, and should be added to the Supplemental Material or immediately after Lemma 2.6.\nIf \\\\(\\\\varepsilon \\\\geq \\\\alpha \\\\), then the second conditions in Definition 2.2 becomes \\\\(\\\\rho_K(Y) - \\\\rho_V(Y) \\\\leq \\\\alpha\\\\),\nwhich is a trivial condition for \\\\(0 \\\\leq \\\\rho_V(Y) \\\\leq \\\\rho_K(Y) \\\\leq \\\\alpha\\\\).\nThus, the sets \\\\(P_{\\\\alpha,\\\\varepsilon}\\\\) are identical for all \\\\(\\\\varepsilon \\\\geq \\\\alpha\\\\).\n\n### Re: (Q5)\n\nYes, we should have foreshadowed these terms in Section 1.\nFigure 1 of the attached PDF could be added in Section 1 or in the Supplemental Material to clarify the interpretation.\n\n### Re: (Q6)\n\nYes. Since \\\\(\\\\lambda_{lo}\\\\) equals the filtration level of the longest\nedge, all edges are included in the parallax complex. For every edge in the\nRips complex of the original dataset, the geodesic representing that edge is\ncontained in the model. Thus, the Rips complexes \\\\(R(X,V)\\\\) and\n\\\\(R(X,K)\\\\) are identical.  This does not imply that \\\\(K=V\\\\), but any\ndifferences are undetected by the pairwise geodesics between the points of\n\\\\(X\\\\).\n\n### Re: Minor Comments\n\nWe thank the reviewer for these wording changes and figure formatting comments, and \nwe will be happy to make them.\n"}, {"review_id": "PDioUaFfDe", "reviewer": "Reviewer_Hzkd", "summary": "The paper presents a framework for analyzing the validity of deep classification models by comparing its multi-scale geometric features with that of the training dataset. The comparison is conducted through tools of topological data analysis, mainly using persistence diagrams to detect and characterize the nontrivial topological features (connected components, 1-cycles, 2-cycles, etc.) as they are revealed by changing the radius of data points to construct Ribs complex. The main challenge is to conduct such analysis on the implicit space of the deep classification model; the paper addresses this challenge by devising distance estimate algorithms for the implicit space, and comparing it with the ambient Euclidean distance to decide the matching of geometric features. The paper shows an application of the analysis on a single dataset and accompanying simple classification models.", "questions": "The major questions have been posed in the above discussions about weaknesses.\n\nMinor questions:\n\nDefinition 1.1, $K^\\circ, \\overline{K^\\circ}$ should be defined clearly. A summary of important notations could be given in the text.\n\nLine54, the order of $M$ and $M^*$ seems to be reversed?\n", "limitations": "The authors have discussed limitations extensively. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper asks the important question of how to decide if a deep perception model faithfully captures the true characteristics of a dataset. It proposes to check if the learned implicit space of the perception model has geometric features that closely match the dataset distribution. \n\nThe paper develops a theoretical framework to characterize the geometric matching of model and data. The important notion is to construct a parallel complex, that is obtained by the co-filtration of Rips complex by metrics of both the data set and the learned model. Based on this construction, the comparison of model and data becomes homological computations on the series of complexes, which draws on topological data analysis extensively. \n\nThe paper discusses applications through a concrete example, and of the possibility of using the assessment as objective function for training more accurate networks.", "weaknesses": "The paper is theoretical in nature. To ease understanding, it is better to provide tables to summarize the notations introduced and use more figures to motivate and illustrate the ideas. I find this can be helpful particularly for the perturbation lemmas and local simplicial matching. Drawing more connections to TDA can be helpful too, to motivate the constructions introduced in this paper.\n\nAfter developing the theoretical framework, it is desirable to immediately position Sec.6 in context, maybe through the example of Sec.8 or other cases. \n\nMore application examples can be used. The cyclo-octane dataset has a very unintuitive structure. If more intuitive datasets and perception tasks can be tested, the usefulness of this analysis can be more convincing. Such intuitive datasets can come from computer vision data, e.g. MNIST, CIFAR. \n\nDeep learning models are witnessing a shift from perception to generative modeling, achieved mostly through minor variations in objective function and output format. How do the motivation and framework proposed by this paper apply to generative models? This paper does not include any such discussion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### Regarding readability\n\nWe thank the reviewer for their comments regarding readability and\nunderstandability. We will include a table of notation in the Supplemental Material to help\nthe reader understand our mathematical notation. We are also planning on\nreplacing Figure 2 with Figure 1 in the attached PDF to provide a high-level\nunderstanding of parallax, as well as the importance of matching the geometry\nbetween model and data. We will also expand our references to include standard\nTDA textbooks, such as Edelsbrunner and Harer's computational topology\ntextbook, as well as Hatcher's algebraic topology text.\nWe will also add a preview of section 6 and 8 to the introduction to help orient the reader.\n\n### Regarding additional examples\n\nWe agree that additional applications and examples (particularly in imaging data) are desirable to advance the field.\nPlease see the general rebuttal for the description of another imaging dataset example.\nWe are preparing a separate manuscript that surveys imaging data and popular vision networks.\nThe rich field of convolutional neural networks deserves special attention due to the high extrinsic dimension and counter-intuitive metric geometry.\nIn preparing this submission,\nwe found that discussion of the often counter-intuitive metrics took too much attention away from the important discussion of topological interpretation and stability analysis.\nIn this submission, we intentionally restricted the scope to an introduction to the core definitions and algorithms, and we demonstrated their meaning on examples that did not rely on convolutional layers,\nbut we look forward to presenting followup work on imaging data in the near future.\n\n### Regarding application to generative modeling\n\nWe agree that generative modeling is increasingly important, especially in the context of adversarial robustness and safety.\nWe are preparing a separate manuscript on that topic, but the mathematical formulation of generative models is more subtle than perception models.\nIn this submission, we restricted the scope to perception models to keep the (already heavy)\nmathematical formulation manageable within the space available.\n\n### Regarding minor comments\n\nWe thank the reviewer for catching these mistakes. We will be sure to define all of the notation used in Definition 1.1\nand we will add a table summary of notation used to the Supplemental Material.\nIndeed, the order of \\\\(\\\\mathcal{M}\\\\) and \\\\(\\\\mathcal{M}^*\\\\) in Line 54 is reversed and should be corrected."}]

Author rebuttal:
We thank the editors and reviewers for their high-quality work.
It is clear that the reviewers read the submission carefully and thoughtfully, and
that the overall editorial process at NeurIPS is efficient and productive.

Overall, the weaknesses raised by the reviewers fell into two clear categories.

### 1. Exposition of Definitions and Notation

The reviewers were very helpful in identifying sections that may be difficult for readers
due to field-specific jargon.
We agree that expository clarification of the definitions
and notation is extremely important for readability among a broad interdisciplinary audience.
We believe we can address these concerns by inserting some additional explanatory words
(e.g. ``... the interior \\(K^\\circ\\)''), adding a table of symbols in the Supplementary Material,
and by including citations to the standard definitions in the most common topology textbooks (Hatcher's *Algebraic Topology* and Mukres' *Topology, 2nd edition*) to orient the reader.
Specific alterations are suggested in the individual rebuttals.

### 2. More comprehensive experimentation, particularly for imaging applications

We agree that additional applications and examples (particularly in imaging data) are desirable to demonstrate the utility of parallax and to advance the field of topological analysis of ML models.

The comments by the various reviewers are exactly in-line with our overall research agenda.

We are preparing a separate manuscript that surveys imaging data and popular vision networks.
The rich field of convolutional neural networks deserves special attention due to the high extrinsic dimension and counter-intuitive metric geometry.

In preparing the current submission,
we found that discussion of the often counter-intuitive metrics took too much attention away from the important discussion of topological interpretation and stability analysis.
Therefore, we intentionally restricted the scope to an introduction to the core definitions and algorithms, and we demonstrated their meaning on examples that did not rely on convolutional layers.

We look forward to presenting followup work on imaging data in the near future,
but we feel that the present submission must stands on its own as laying structural groundwork for that experimentation and exploration.

However, we believe we will be able to offer one additional example in the
Supplemental Material without distracting too much from the intended scope of
the manuscript. In Figure 2 of the attached figures PDF, we have included an
additional experimental example based on a simplified version of the Utah
teapot dataset consisting of images of teapots with their spouts and handles
removed, essentially images of jars with lids. We construct a simple model
which accepts all test data points but has very poor interpolation properties.
We show that parallax detects this poor interpolation and we highlight 6
interpolated images that the model rejects. In particular, this example
exemplifies the concern of Section 6: "if step (2) yields \\(\\lambda_{lo,X} (K) = 0\\), then \\(K\\) has
voids between every pair of points in \\(X\\), possibly due to under-sampling or over-fitting, and should
not be trusted for any interpolative purpose". Additional description of this
dataset and CNN could be included in the Supplemental Material.


Meta-review: The paper introduces a new methodology to compare a pre-trained model to a reference dataset in terms of their topological similarities, with the hope of shedding light on the delicate balance between overfitting and generalization. It received consistently positive reviews by all 6 reviewers, who appreciated the conceptual and technical novelty, the underlying idea, the theoretical analysis, and the overall exposition (although with some concerns due to the required mathematical background). Some drawbacks were identified in the limited experimental evaluation, but the reviewers also acknowledged the theoretical nature of the paper. The rebuttal clarified all the pending concerns, and some of the reviewers renovated their appreciation for the paper by upgrading their ratings. This is a solid contribution that brings new understanding in the geometrical properties of neural models, and is likely to have a good impact on the community. Therefore, we recommend acceptance.
Decision: Accept (spotlight)