Year: 2021

Abstract:
Comparison of data representations is a complex multi-aspect problem that has not enjoyed a complete solution yet. We propose a method for comparing two data representations. We introduce the Representation Topology Divergence (RTD) measuring the dissimilarity in multi-scale topology between two point clouds of equal size with a one-to-one correspondence between points. The data point clouds are allowed to lie in different ambient spaces. The RTD is one of the few TDA-based practical methods applicable to real machine learning datasets. Experiments show the proposed RTD agrees with the intuitive assessment of data representation similarity and is sensitive to its topological structure. We apply RTD to gain insights on neural networks representations in computer vision and NLP domains for various problems: training dynamics analysis, data distribution shift, transfer learning, ensemble learning, disentanglement assessment.

heading: 1 INTRODUCTION
text: Representations of objects are the essential component learnt by deep neural networks. In opposite to the distance in the original space, similarity of representations are proved to be semantically meaningful. Despite of the significant practical success of deep neural networks many aspect of their behaviour are poorly understood. Only few methods study learned representations without relying on their quality on a specific downstream task. In this work, we focus on the comparison of representations from neural networks.
Comparison of representations is an ill-posed problem without a “ground truth” answer. Early studies were based on variants of Canonical Correlation Analysis (CCA): SVCCA, (Raghu et al., 2017), PWCCA (Morcos et al., 2018). Hovewer, CCA-like measures define similarity too loosely since they are invariant to any invertible linear transformation. The Centered Kernel Alignment (CKA), (Kornblith et al., 2019) is the statistical test to measure the independence of two sets of variables. (Kornblith et al., 2019) proved it to be more consistent with the intuitive similarity of representations. Particularly, neural networks learn similar representation from different seeds as evaluated by CKA. Another line of work studies alignment between groups of neurons (Li et al., 2015), (Wang et al., 2018). The similarity of representation is also a topic of a study in neuroscience (Edelman, 1998; Kriegeskorte et al., 2008; Connolly et al., 2012).
Representations’ comparison metrics like CKA and CCA were used to gain insights on representations obtained in meta-learning (Raghu et al., 2020), to compare representations from different layers of language models (Voita et al., 2019), study the effect of fine-tuning (Wu et al., 2020). Finally, (Nguyen et al., 2021) used CKA to study the phenomenon of a “block structure” emerging in wide and deep networks in computer vision and compare their representations.
In this paper, we take a topological perspective on representations’ comparison. We propose the Representation Topology Divergence (RTD) score which measures a dissimilarity between two point clouds of equal size with one-to-one correspondence between points. Point clouds are allowed to lie in different ambient spaces. Existing geometrical and topological methods are dedicated to other problems: they are either too general and doesn’t incorporate the requirement of one-to-one correspondence (Khrulkov & Oseledets, 2018), (Tsitsulin et al., 2020), or they restrict point clouds to lie in the same ambient space (Kynkäänniemi et al., 2019), (Barannikov et al., 2021). Such
methods (except for (Tsitsulin et al., 2020)) are mostly applied to the evaluation of GANs, where point clouds of real and generated objects are matched. Recently, (Moor et al., 2020) proposed a loss term to compare a topology of data in original and latent spaces (with natural one-to-one correspondence) and applied it as a part of the Topological Autoencoder.
In this work, we make the following contributions:
1. We propose a topologically-inspired approach for comparison of neural network representations;
2. We introduce the R-Cross-Barcode(P, P̃ ), a tool based on Topological Data Analysis (TDA) which measures differences in multi-scale topology of two point clouds P, P̃ with one-to-one correspondence between points;
3. Based on the R-Cross-Barcode(P, P̃ ), we define the Representation Topology Divergence (RTD), the scalar measuring the multi-scale topological dissimilarity between two representations;
4. By doing computational experiments, we show that RTD agrees with an intuitive notion of neural network representations similarity. In contrast with most existing approaches, RTD score is sensitive to cluster and other topological structures of the representations and enjoys very good correlation with disagreement of models predictions. We apply RTD to compare representations in computer vision and NLP domains and various problems: training dynamics analysis, data distribution shift, transfer learning, ensemble learning, disentanglement. We have also compared RTD with CKA, IMD and SVCCA.

heading: 2 COMPARING NEURAL NETWORK REPRESENTATIONS
text: Our starting point is the geometric perspective on representation learning through the lens of the manifold hypothesis (Goodfellow et al., 2016), according to which real-world data presented in a high dimensional space are expected to concentrate in vicinity of a manifold of much lower dimension. The low-dimensional manifoldMP underlying the given data representationP can be accessed generally only through discrete sets of samples. The standard approach to recover the manifold MP is to take a sample P and to approximateMP by a set of simplices with vertices from P . A common approach to select the simplices approximating MP is to fix a threshold α > 0 and consider the simplices with edge lengths not exceeding α (Niyogi et al., 2008; Belkin & Niyogi, 2001). It is difficult in general to guess the correct value of the threshold, hence a reasonable viewpoint is to study all thresholds at once, see e.g. (Chazal & Michel, 2017). This can be accomplished by means of the mathematical tool, called barcode, that quantifies the evolution of manifold topology features over multiple scales.
Given two representations, we consider two corresponding graphs with distance-like weights and compare the difference in the two graphs’ multiscale topology.

heading: 2.1 R-CROSS-BARCODE
text: Let P(V), P̃(V) be two representations giving two embeddings of the same data. The two embeddings P, P̃ belong in general to different ambient spaces and we have the natural one-to-one correspondence between points in P and P̃ . Given a sample of data V ⊆ V , the representation P = P(V ) defines the weighted graph Gw with the vertex set V . The weight wAB of an edge AB is given by the distance between points P (A) and P (B). Similarly the representation P̃ = P̃(V ) defines the weighted graph Gw̃ on the same vertex set. The simplicial approximation to the manifold MP at threshold α consists of simplices whose edges in Gw have weights not exceeding α. Let Gw≤α denote the graph with the vertex set V and the edges with weights not exceeding α. To compare the simplicial approximations to manifolds MP and MP̃ described by the graphs Gw≤α and Gw̃≤α we embed both graphs into the graph Gmin(w,w̃)≤α. The graph Gmin(w,w̃)≤α contains an edge between vertices A and B exactly when the distance between the points A and B is smaller than α in at least one of the representations P , P̃ .
Recall that Vietoris-Rips complex of a graph G equipped with edge weights’ matrix m is the collection of k−simplices, k ≥ 0, which are (k + 1)−elements subsets of the set of vertices of G, with the filtration threshold of a simplex defined by its edges’ maximal weight:
Rα(Gm) = { {A0, . . . , Ak}, Ai ∈ Vert(G)‖mAjAl ≤ α } Our simplicial approximations to the manifolds MP , MP̃ at threshold α are the unions of all simplices from the simplicial complexes Rα(Gw), Rα(Gw̃). The dissimilarity between the filtered simplicial complexes Rα(Gw) and Rα(Gw̃) can be quantified using the homological methods. The relevant tools here are the homology, the Whitehead theorem and the homology exact sequence. Because of the space limitations we sketch how this leads to our construction, described below, in the Appendix, Section B.
Concretely, to compare the multi-scale topology of the two weighted graphs Gw and Gw̃ we introduce the weighted graph Ĝ(w, w̃) with doubled set of vertices and with the edge weights defined as follows. For each vertex A ∈ Vert(G) we add the extra vertex A′ together with A to Ĝ and define the distance-like edge weights in Ĝ(w, w̃) as:
dAB = min(wAB , w̃AB), dAB′ = dA′B = wAB , dAA′ = 0, dA′B′ = 0 (1)
where B ∈ Vert(G), A 6= B.
In practice G, Ĝ are full graphs and the edge weights on the graph Ĝ(w, w̃) are given by 2N × 2N , N = |V |, symmetric matrix
m = ( 0 w w min(w, w̃) ) (2)
where w and w̃ are the distance-like edge weights matrices of Gw and Gw̃.
Next, we construct the Vietoris-Rips filtered simplicial complex from the graph Ĝ(w, w̃). Intuitively, the i−th barcode of Rα(Ĝ(w, w̃)) records the i−dimensional topological features that are born in Rα(Gw̃) but are not yet born at the same place in Rα(Gw) and the (i+ 1)−dimensional topological features that are dead inRα(Gw̃) but are not yet dead at this place inRα(Gw), see Theorem 1 below.
Definition. The R-Cross-Barcodei(P, P̃ ) is the set of intervals recording the “births” and “deaths” of i-dimensional topological features in the filtered simplicial complex Rα(Ĝ(w, w̃)).
The R-Cross-Barcode∗(P, P̃ ) (for Representations’ Cross-Barcode) records the differences in multiscale topology of the two embeddings. The topological features with longer lifespans indicate in general the essential features.
Theorem 1. Basic properties of R-Cross-Barcode∗(P, P̃ ):
• if P (A) = P̃ (A) for any object A ∈ V , then R-Cross-Barcode∗(P, P̃ ) = ∅; • if all distances within P̃ (V ) are zero i.e. all objects are represented by the same point in P̃ , then
R-Cross-Barcode∗(P, P̃ ) = Barcode∗(P ) the standard barcode of the point cloud P ;
Algorithm 1 R-Cross-Barcodei(P, P̃ ) Input: w, w̃ : matrices of pairwise distances
within point clouds P , P̃ Require: vr(m): function computing filtered
complex from pairwise distances matrix m Require: B(C, i): function computing persis-
tence intervals of filtered complexC in dimension i m← (
0 w w min(w, w̃) ) R-Cross-Barcodei ← B(vr(m), i)
Return: intervals’ list R-Cross-Barcodei(P, P̃ ) representing ”births” and ”deaths” of topological discrepancies between P and P̃ .
Algorithm 2 RTD(P, P̃), see section 2.3 for details, suggested default values: b = 500, n = 10
Input: P ∈ R|V|×D, P̃ ∈ R|V|×D̃ : data representations for j = 1 to n do Vj ← random choice (V, b) Pj , P̃j ←P(Vj), P̃(Vj) Bj ← R-Cross-Barcode1(Pj , P̃j) intervals’ list calculated by Algorithm 1 rtdj ← sum of lengths of all intervals in Bj
end for RTD(P, P̃)← mean(rtd)
Return: number RTD(P, P̃) representing discrepancy between the representations P, P̃
• for any value of threshold α, the following sequence of natural linear maps of homology groups
. . .→ Hi(Rα(Gw))→ Hi(Rα(Gmin(w,w̃)))→ Hi(Rα(Ĝ(w, w̃)))→ → Hi−1(Rα(Gw))→ Hi−1(Rα(Gmin(w,w̃)))→ . . . (3)
is exact; recall that it means that the kernel of any map is the image of the previous map
The proof of the first two properties is immediate and the third property follows from the exactness of the corresponding sequence of simplicial complexes, see Appendix for more details.

heading: 2.2 REPRESENTATION TOPOLOGY DIVERGENCE.
text: The R-Cross-Barcode∗(P, P̃ ) is by itself, to our opinion, a precise and intuitive tool for understanding discrepancies between two representations. There are several numerical characteristics measuring the non-emptyness of R-Cross-Barcode. Based on experiments and on its relation with Earth Moving Distance, see (Barannikov et al., 2021), we have defined the sum of lengths of the bars in R-Cross-Barcodei(P, P̃ ), denoted RTDi(P, P̃ ) as the scalar characterizing the degree of topological discrepancy between the representations P, P̃ . We use most oftenly the average of RTD1(P, P̃ ) and RTD1(P̃ , P ), denoted RTD score, in our computations below.
Proposition 1. If RTDi(P, P̃ ) = RTDi(P̃ , P ) = 0 for all i ≥ 0 then the barcodes of the weighted graphs Gw and Gw̃ are the same in any degree. Moreover in this case the topological features are located in the same places: the inclusions Rα(Gw) ⊆ Rα(Gmin(w,w̃)), Rα(Gw̃) ⊆ Rα(Gmin(w,w̃)) induce for any threshold α the isomorphisms in homology.

heading: 2.3 ALGORITHM
text: First we compute the R-Cross-Barcode1(P, P̃ ) on two representations P, P̃ of a sample V . For this we calculate the matrices of pairwise distances w, w̃ within the point clouds P , P̃ . We assume that the metrics in the ambient spaces of representations are normalized so that the two point clouds are of comparable size, namely their 0.9 quantile of pairwise distances coincide. This ensures that our score has scaling invariance, the reasonable property of a good representation similarity measure, as argued in e.g. (Kornblith et al., 2019). Next the algorithm builds the Vietoris Rips complex from the matrix m defined in Equation (2). Then the 1−dimensional barcode, see (Chazal & Michel, 2017), of the built filtered simplicial complex is calculated. The last two steps can be done using scripts that are optimized for GPU acceleration. Then we sum the lengths of bars in R-Cross-Barcode1(P, P̃ ). To get the symmetric measure we usually take the half-sum with similar sum of bars in R-CrossBarcode1(P̃ , P ). The computation is repeated sufficient number of times to obtain the mean of the chosen characteristics. We have observed experimentally that about 10 times is normally sufficient for common datasets. The main steps of the computation are summarized in the Algorithms 1, 2.
Complexity. Algorithm 1 starts with the computation of the two matrices of pairwise distances w, w̃ for a pair of representations of a sample V : P ∈ Rb×D, P̃ ∈ Rb×D̃ involving O(|V |2(D + D̃)) operations. Next, persistent intervals of the filtered complex must be computed. Given the distance matrix m, the complexity of their computation doesn’t depend on the dimensions D, D̃ of the data representations. Generally, the barcode computation is at worst cubic in the number of simplices involved. In practice, the calculation is quite fast since the boundary matrix is typically sparse for real datasets. For R-Cross-Barcodes’ calculation we used the GPU-optimized software. Thus, the computation of R-Cross-Barcode takes similar time as in the previous step even on datasets of big dimensionality. Since only the dissimilarities in representation topology are calculated, the results are quite robust and a rather low number of iterations is needed to obtain accurate results.

heading: 3 EXPERIMENTS
text: In the experimental section, we study the ability the proposed R-Cross-Barcodes and RTD to detect changes of topological structures using synthetic point clouds; we demonstrate the superiority of RTD over CKA, SVCCA, IMD (Section 3.1). By comparing representations from various architectures (Section 3.3), layers, epochs, ensembles and after data distribution shift (Section 3.4) we show that RTD is in line with natural notion of neural representations’ similarity. High correlation of RTD and disagreement of networks’ predictions is an interesting empirical finding. Additionally, empirical evidence suggests that RTD is useful for evaluating disentanglement of representations (Section 3.2).

heading: 3.1 EXPERIMENTS WITH SYNTHETIC POINT CLOUDS
text: We start with small-scale experiments with synthetic point clouds: “clusters” and “rings”. For the “clusters” experiment (Figure 2, top), the initial point cloud consists of 300 points randomly sampled from 2-dimensional normal distribution having mean (0, 0). Next, we split it into 2,3. . . 12 parts (clusters) and move them to the circle with radius 10. Then, we compare the initial point cloud (having one cluster) with the split ones.
In the “rings” experiment, we compare synthetic point clouds consisting of a variable number of rings, see Figure 2, bottom. Initially, there are 500 points uniformly distributed over the unit circle. Then, the points are moved onto circles with varying radius, from 0.5 to 1.5. Finally, we compare the leftmost point cloud having 5 rings with other ones.
In both of the experimental settings there is a one-to-one correspondence between points in the point clouds. We compared these point clouds by calculating: RTD (proposed), CKA (Kornblith et al., 2019), IMD (Tsitsulin et al., 2020) and SVCCA (Raghu et al., 2017). We calculated linear CKA since (Kornblith et al., 2019) concluded that it provides the same performance as with RBF kernel but doesn’t require to select a kernel width. For SVCCA, we calculated average correlation ρ̄ for the truncation threshold 0.99, as recommended by the authors (Raghu et al., 2017). The IMD score (Tsitsulin et al., 2020) was very noisy and we averaged it over 100 runs.
Figures 4, 5 present the results. RTD measure almost ideally reflects the change of the topological complexity while the alternative measures mostly fail. The Kendall-tau rank correlations of the measures with a number of clusters are: RTD: 1.0, CKA: 0.23, IMD: 0.43, SVCCA: 0.14; for the number of rings: RTD: 0.9, CKA: -0.2, IMD: 0.9, SVCCA: -0.2. We also note than RTD score doesn’t have any tunable parameters as SVCCA and doesn’t require averaging over as many runs as IMD. Figure 3 shows some of the H1 R-Cross-Barcodes calculated while comparing clusters and rings. In accordance with the definition, H0 barcodes are absent. The sum of lengths of segments increases as differences of topology increases. All of the R-Cross-Barcodes are in Appendix D.
3.2 EXPERIMENTS WITH DISENTANGLEMENT
Table 1: Evaluation of the disentanglement for various directions in the latent space of dSprites.
axis RTD axis 1 148.1 axis 2 71.3 axis 3 53.4 axis 4 41.2 axis 5 40.5
random 162.8± 18.6
Learning disentangled representations is a fundamental problem for improving generalization, robustness, and interpretability of generative models. Zhou et al. (2020) proposed to evaluate the disentanglement of generative models by comparing topology of data manifold slices. Let Z be a latent space, X - a space of objects, g : Z → X - a generator. Zhou et al. (2020) compares slices Xv = g(Z |zi=v) for different values of v. If the direction zi corresponds to an interpretable factor, then Xv must be topologically similar for different v.
We use the following experimental design. Zv,n = {z ∈ Z | (z, n) = v} - a slice in a latent space orthogonal to a unit vector n. We take a finite random sample Z1 ⊂ Zv,n and a shifted sample Z2 = {zi + δn}|Z1|i=1 for small δ. By definition, Z1 and Z2 have natural point-wise mapping and we can estimate homological similarity of g(Z1) and g(Z2) by RTD.
In this experiment, we use dSprites1 for the evaluation of disentanglement. dSprites is a dataset of procedurally generated 2D shapes from 5 ground truth independent latent factors: shape, scale, rotation, x-position and y-position of a sprite. Thus, the latent space is disentangled and fully factorized. Particularly, we compare the slices orthogonal to axes-aligned vectors and orthogonal to
1https://github.com/deepmind/dsprites-dataset
random vectors, see Table 1. Except for the first axis, the topological dissimilarity estimated by RTD is significantly less then for a random direction. The first axis corresponds to a categorical factor - shape for which the aforementioned approach is arguably not applicable. The dSprites dataset is quite simple and RTD was calculated for point clouds in the pixel space. However, the same technique can be straightforwardly applied to evaluate disentanglement of image representations for more complex datasets.
3.3 EXPERIMENTS WITH NAS-BENCH-NLP
Recently, neural architecture search attracted a lot of attention in the machine learning community (Liu et al., 2019; Dong & Yang, 2019; Chen et al., 2021). Klyuchnikov et al. (2020) developed a benchmark for neural architecture search which is a collection of 14’322 recurrent architectures; all of them were trained on the PTB dataset. We took 90 random architectures and compared word embeddings: each architecture contains 400-dimensional embeddings of 10’000 words. Then, we evaluated all the pairwise similarities between embeddings2 from the architectures and visualized them via multi-dimensional scaling, see fig. 7, where color depicts a log. perplexity. Accordingly to a common sense, architectures having similar embeddings have similar log. perplexity.

heading: 3.4 EXPERIMENTS WITH CONVOLUTIONAL NEURAL NETWORKS
text: To demonstrate the abilities of RTD to work with image representations, we train ResNet-20 (He et al., 2016) and VGG-11 (Simonyan & Zisserman, 2014) networks on CIFAR (Krizhevsky et al., 2009) datasets. In the experiments, we compare RTD with CKA and disagreement of predictions. For more intuitive comparison we consider 1−CKA instead of CKA. As a measure of the difference in predictions, we use Disagreement (Kuncheva & Whitaker, 2003; Wen et al., 2020), the fraction of mismatched predictions calculated as 1N ∑N n=1 [fθ1(xn) 6= fθ2(xn)] , where fθ(x) denotes the class label predicted by the network for input x. As discussed in (Fort et al., 2019), the lower the accuracy of predictions, the higher its potential mismatch due to the possibility of the wrong answers being random, and then we normalize the Disagreement by (1 − a), where a is the mean accuracy of the predictions. To calculate the final metrics, we averaged values for five random batches of 500 representations from the test dataset.
3.4.1 TRAINING DYNAMICS Table 2: The correlation of metrics with Disagreement in the training dynamics experiment
RTD 1−CKA VGG-11 0.99 0.83 ResNet-20 0.98 0.93 As the first experiment, we analyze the training dynamics of neural networks. On each epoch, we collect the outputs of the convolutional part that extract the representations. To compare dynamics properly, we scaled the metrics by their maximum value. Fig. 1 shows the dynamics of differences with the final representations.
2to speedup the computation, we averaged metrics for 10 random batches of 100 word embedding.
The results coincide with the intuition: the representations on each epoch become more similar to the final one. Moreover, RTD demonstrates the same behavior as disagreement of predictions. RTD better correlates with the Disagreement, see table 2.
3.4.2 LAYERS
As the next experiment, we compare the outputs of layer blocks within the trained network. For VGG-11, the block has the form Conv→BN→Activation→(Pooling), and for ResNet20, we take the output of the first Conv→BN→Activation block and then the outputs of each residual block. In fig. 8, we see that both RTD and 1−CKA show similar results, including the slight difference between adjacent layers. Also, we can see that both metrics reveal the significant changes in the outputs of the ResNet-20 last block.

heading: 3.4.3 DATA DISTRIBUTION SHIFT
text: Here, we apply the data distribution shift to test the RTD. As a shift, we consider different image transformations: noising, blurring, grayscaling, and hue changing. For each transformation, we analyze the metric dynamics as a strength of transformation increases. Fig. 9 confirms our sanity check about the monotony of RTD and other metrics with respect to data
distribution shift. Moreover, Table 3 shows that RTD has a higher correlation with disagreement of predictions.

heading: 3.4.4 ENSEMBLES
text: It is known that an ensemble of neural networks performs better than a single network and can estimate the uncertainty of the predictions. It is showed in (Lee et al., 2015; Opitz et al., 1996) that the diverse ensembles work better. Thus, measuring ensembles’ diversity is important. The disagreement is a good example of such a measure. To show that RTD can measure the diversity as well as disagreement, we learn two types of ensembles: the classical one, when we learn the networks from different random initializations, and the Fast Geometric Ensemble (FGE) (Garipov et al., 2018), which is known to have the lower diversity. We learn four models for each type of ensemble and average the metrics among all pairs. The results in Table 4 confirm that RTD can measure the diversity on the same scale as the disagreement of predictions.

heading: 3.4.5 TRANSFER LEARNING
text: Another possible application is the measure of changes in representations after transferring the pretrained model to a new task. In this experiment, we conduct the transfer learning from CIFAR-100 to CIFAR-10 dataset. We make full fine-tuning with the small learning rate for the convolutional part. In fig. 10, we demonstrate the dynamics for both dataset representations. The results again coincide with the intuition about the difference during the learning steps, and here RTD has also high correletion with Disagreement, see Table 5. Also, we note that RTD can be applied to the continual learning task, where catastrophic forgetting appears, and thus it is crucial to track the changes of network representations.
4 CONCLUSIONS
In this paper, we have proposed a topologically-inspired approach to compare neural network representations. The most widely used methods for this problem are statistical: Canonical Correlation Analysis (CCA) and Centered Kernel Alignment (CKA). But the problem itself is a geometrical one: comparison of two neural representations of the same objects is defacto comparison of two points clouds from different spaces having one-to-one correspondence. The natural way is to com-
pare their geometrical and topological features, taking into account their localization, that is exactly done by the R-Cross-Barcode and RTD. We demonstrated that RTD coincides with the natural assessment of representations similarity. We used the RTD to gain insights on neural networks representations in computer vision and NLP domains for various problems: training dynamics analysis, data distribution shift, transfer learning, ensemble learning, disentanglement assessment.
RTD strikingly well correlates with the disagreement of models’ predictions; this is an intriguing topic for the further research. Finally, R-Cross-Barcode and RTD are general tools and which are not limited only to representations comparison. They could by applied to other problems involving comparison of two point clouds with one-to-one correspondence, for example in 3D computer vision.

heading: A BACKGROUND ON SIMPLICIAL COMPLEXES. BARCODES
text: The simplicial complex is a combinatorial object that can be thought of as a higher-dimensional generalization of a graph.
A simplex is defined via the set of its vertices. Given a finite set V , a k−simplex is a finite (k + 1)−element subset in V . Simplicial complex S is a collection of k−simplices, k ≥ 0, which satisfies the natural condition that for each σ ∈ S, σ′ ⊂ σ implies σ′ ∈ S. A simplicial complex consisting only of 0− and 1−simplices is a graph. Denote via Ck(S) the vector space over the field Z/2Z whose basis elements are k−simplices from S. The boundary linear operator ∂k : Ck(S)→ Ck−1(S) is defined on σ = {A0, . . . , Ak} as
∂kσ = k∑ j=0 {A0, . . . , Aj−1, Aj+1, . . . , Ak}.
The k−th homology group Hk(S) is the factor vector space ker ∂k/ im ∂k+1. The elements c ∈ ker ∂k are called cycles. The elements of Hk(S) represent various k−dimensional topological features in S. A basis in Hk(S) corresponds to a set of basic topological features.
In applications the simplicial complexes are often built via consequential adding of simplices one after another in increasing order of some numerical characteristics. Mathematically this corresponds to the filtration on the simplicial complex. It is defined as a family of simplicial complexes Sα, indexed by a finite set of real numbers, with nested collections of simplices: for α1 < α2 all simplices of Sα1 are also in Sα2 .
The inclusions Sα ⊆ Sβ induce the maps on homology Hk(Sα) → Hk(Sβ). The evolution of cycles across the nested family of simplicial complexes Sαi is described by the principal persistent homology theorem (Chazal & Michel, 2017; Barannikov, 1994), according to which for each dimension there exists a choice of a set of basic topological features across all nested simplicial complexes Sα so that each basic feature c appears in Hk(Sα) at specific time α = bc and disappears at specific time α = dc. The barcode of the filtered complex is the record of the appearence, or “birth” time, and disappearence, or “death” time, of all these basic topological features.
A.1 CONSTRUCTION OF R-CROSS-BARCODE
Here we gather some intuition behind the construction of the graph Ĝ(w, w̃) and the R-CrossBarcode. An inclusion of simple simplicial complexes S ⊂ R is an equivalence in homotopy category, if and only if the induced map on homology is an isomorphism (Whitehead, 1968).
Therefore the maps on homology induced by the inclusions of filtered simplicial complexes
Rα(Gw) ⊆ Rα(Gmin(w,w̃)), Rα(Gw̃) ⊆ Rα(Gmin(w,w̃)) (4) should be as close as possible to isomorphisms, in order that the appoximations to manifolds MP and MP̃ have similar geometrical and topological features located at the same place.
It follows from the exact sequence from Theorem 1 that the R-Cross-Barcode∗(P, P̃ ) is exactly the list of topological features describing the failure of the maps, induced on homology by inclusions (4), to be isomorphisms.

heading: B PROPERTIES OF R-Cross-Barcode∗(P, P̃ ).
text: The proof of the exact homology sequence from Theorem 1 follows from the following proposition.
Proposition 2. The embedding of weighted graphs Gmin(w,w̃) → Ĝ(w, w̃)) gives the exact sequence of chain complexes
0→ Rα(Gmin(w,w̃))→ Rα(Ĝ0(w, w̃))→ R′α(Gw)→ 0. (5)
where Rα(Ĝ0(w, w̃)) has the same homology as Rα(Ĝ(w, w̃)) and R′α(Gw) has the same homology as Rα(Gw) shifted by +1.
The proof follows from the straightforward construction of a homotopy to identity map.
Comparison with Cross-Barcode and Geometry score. The Cross-Barcode from (Barannikov et al., 2021) compares two data manifolds lying in the same ambient space and can not use the information provided by one-to-one correspondence between points. It uses instead the proximity information inferred from the pairwise distances between points from different clouds lying in the same ambient space. Geometry score is based on comparison of standard barcodes for each cloud and for example does not detect any difference when similar topological features are located geometrically in very distant places of the two clouds.

heading: C DISCUSSION OF CKA
text: Given two series of equal size xi ∈ Rnx , yi ∈ Rny , i = 1 . . . n the CKA (Kornblith et al., 2019) is defined as
CKA(K,L) = HSIC(K,L)√
HSIC(K,K)HSIC(L,L)
where HSIC(K,L) is a Hilbert-Schmidt Independence Criterion (Gretton et al., 2005), Ki,j = k(xi, xj), Li,j = l(yi, yj), L = E − n−1 where k(·, ·), l(·, ·) are kernels. HSIC itself an empirical estimate of the Hilbert-Schmidt norm of the cross-covariance operator. HSIC is equivalent to maximum mean discrepancy between the joint distribution P (X,Y ) and the product of the marginal distributions P (X)P (Y ); HSIC = 0 implies independence of X and Y if the associated kernel is universal.
However, CKA is sometimes applied to measure similarity between representations from different layers of neural network. In this case Y = f(X). X and Y are tightly dependent and the joint distribution can always be factorized as P (X,Y ) = P (Y |X)P (X). Thus, the application of CKA to comparison of representation from different layers is questionable.
D DETAILS ON EXPERIMENTS WITH POINT CLOUDS

Review summaries:
This paper proposes a new divergence metric between two point clouds of equal size with a 1-to-1 correspondence, called Representation Topology Divergence (RTD). They compete against other representation comparison metrics like Canonical Correlation Analysis (CCA) and Centered Kernel Alignment (CKA), and other variants. The authors proposed RTD as a good way to compare representations learned by Deep learning networks. They introduce a new barcode called R-Cross-Barcode(P,P^\tilde) which captures the difference in topology of both point clouds in question. RTD is then applied to data from a wide variety of domains spanning CV and NLP, to name a few.  

This paper focuses on measuring the differences in data representation. The authors propose to evaluate it with Representation Topology Divergence, which estimates the dissimilarity on multi-scale topology. The effectiveness of the proposed method is tested on multiple tasks in the experiments. 

In this paper, the authors proposed a method called “representation topology divergence (RTD)” for comparing structured data (e.g., point clouds and sets of embeddings derived by neural networks). The authors applied RTD to various machine learning tasks and compared it with other classic similarity measurements of structured data (e.g., CKA, SVCCA, IMD). Experimental results verify the rationality of the proposed method.