Characterizing and Measuring the Similarity of Neural Networks with Persistent Homology
Authors: David Pérez Fernández
Asier Gutiérrez-Fandiño
Jordi Armengol-Estapé
Marta Villegas
Venue: NeurIPS 2021 Submitted (2021)
Keywords: Neural Networks
Topological Data Analysis
similarity
Persistent Homology

Abstract: Characterizing the structural properties of neural networks is crucial yet poorly understood, and there are no well-established similarity measures between networks. In this work, we observe that neural networks can be represented as abstract simplicial complex and analyzed using their topological 'fingerprints' via Persistent Homology (PH). We then describe a PH-based representation proposed for characterizing and measuring similarity of neural networks. We empirically show the effectiveness of this representation as a descriptor of different architectures in several datasets. This approach based on Topological Data Analysis is a step towards better understanding neural networks and serves as a useful similarity measure.

Machine learning practitioners can train different neural networks for the same task. Even for the 13 same neural architecture, there are many hyperparameters, such as the number of neurons per layer or the number of layers. Moreover, the final weights for the same architecture and hyperparameters Characterizing and Measuring the Similarity of Neural Networks with Persistent Homology
Anonymous Author(s)
Affiliation Address email
Abstract
Characterizing the structural propertics of neural networks crucial yet poorly understood, and there are well-established similarity measures between net- works. this work, we observe that neural networks can be represented as abstract simplicial complex and analyzed using their topological 'fingerprints' via Persis- tent Homology (PH). We then describe a PH-based representation proposed for characterizing and measuring similarity of neural networks. We empirically show the effectiveness of this representation as a descriptor of different architectures in several datasets. This approach based Topological Data Analysis is step towards better understanding neural networks and serves as useful similarity measure.
3
5
7
Introduction
11
15 can vary depending on the initialization and the optimization process itself, which stochastic. Thus, there is no direct way of comparing neural networks accounting for the fact that neural networks
17 solving the same task should be measured as being similar, regardless of the specific weights. This also prevents one from finding and comparing modules inside neural networks (c.g., determining if given sub-net nub-network does the same function as other sub-network another model). Morcover, there no well-known methods for effectively characterizing neural networks.
19
This work aims to characterize neural networks such that they can be measured once trained for the same task, with independence of the particular architecture, initialization, or 23 optimization process. We focus on Multi-Layer Perceptrons (MIPss) for the sake simplicity. We
21
similar
22
24 start by observing that we can represent neural network as a directed weighted graph to which we can associate certain topological concepts!! Considering it as a simplicial complex, we obtain its associated Persistent Diagram. Then, can compute distances between Persistent Diagrams of different neural networks.
25
27
28 The proposed experiments aim to show that the selected structural feature, Persistent Homology,
29 serves to relate neural networks trained for similar problems and that such comparison can be performed by means of predefined measure between the associated Persistent Homology diagrams. 31 To test the hypothesis, we study different classical problems (MNIST, Fashion MNIST, CIFAR- 10, and language identification and text classification datasets), different architectures (number and size
30
33 of layers) as well control experiment (input order).
34 In summary, the main contributions of this work are the following:
See Jonsson 21 for a complete reference on graph topology.
Submitted to 35th Conference on Neural Information Processing Systems (NeurIPS 2021). Do not distribute.

---

35
We propose an effective graph characterization strategy of neural networks based on Persis- tent Homology.
36
37
Based on this characterization, we suggest a similarity measure of neural networks.
38
We provide empirical evidence that this Persistent Homology framework captures valuable information from neural networks and that the proposed similarity measure is meaningful.
39
The remainder of this paper is organized as follows. In Section 2 we go through the related work. we describe our proposal and the experimental framework to validate it. Finally, in sections and we report and discuss the results and arrive to conclusions, respectively.
40
41
Then, in Section 3
42
Related Work
43
44 One of the fundamental papers of Topological Data Analysis (TDA) is presented in Carlsson and suggests the use of Algebraic Topology to obtain qualitative information and deal with metrics for large amounts of data. For an extensive overview of simplicial topology on graphs, see Giblin 18 Jonsson 22 Aktas et al. [2] provide a thorough analysis of PH methods.
45
46
47
48
More recently, number of publications have dealt with the study of the capacity of neural networks using PH. Guss and Salakhutdinov [19] characterize learnability of different neural architectures by computable measures of data complexity. Rieck al. 30 introduce the neural persistence metric, complexity measure based on TDA on weighted stratified graphs. This work suggests a representation of the neural network as multipartite graph and the filtering of the Persistent Homology diagrams are performed for each layer independently. As the filtration contains at most I-simplices (edges), they only capture dimenniiniii topological information, i.e. connectivity information. Donier 14 propose the concept of spatial capacity allocation analysis. Konuk and Smith [22) propose an
49
50
51
52
53
54
55
56
empirical study of how NNs handle changes topological complexity the input data.
57
In terms of pure neural network analysis, there are relevant works, like Hofer et al. 20 that study topological regularization. Clough et al. [1I introduce method for training neural networks for image segmentation with prior topology knowledge, specifically via Betti numbers. Corneanu et al. try to estimate (with limited success) the performance gap between training and testing via neuron activations and linear regression of the Betti numbers.
58
59
80
33
61
On the other hand, topological analysis of decision boundaries has been very prolific area. Ra- mamurthy et al. (28) propose a labeled Vietoris-Rips complex to perform PH inference decision boundaries for quantification of the complexity of neural networks.
69
65
Naitzat et al. 27 experiment on the PH of a wide range point cloud input datasets for binary classification problems to see that NNs transform topologically rich dataset (in terms of Betti numbers) into a topologically simpler one as passes through the layers. They also verify that the reduction in Betti numbers is significantly faster for ReI.U activations than hyperbolic tangent activations.
66
67
68
69
70
25 obtain certain geometrical and topological properties of decision regions for neural models, and provide some principled guidance to designing and regularizing them. Additionally, they use curvatures of decision boundaries in terms of network weights, and the rotation index theorem theorem.
Liu
71
72
73
together with the
74 Regarding neural network representations, one of the most related works to ours, Gebhart et al. [16] focuses on topological representations of neural networks. They introduce method for computing PH over the graphical activation structure of neural networks, which provides access to the task-relevant substructures activated throughout the network for given input.
75
76
77
78
Interestingly, in Watanabe and Yamana 35). authors work on neural network representations through simplicial complexes based on deep Taylor decomposition and they calculate the PH of neural networks in this representation. In Chowdhury et al. (10), they use directed homology to represent 80
81 MLPs. They show that the path homology of these networks is non-trivial in higher dimensions and
82 depends on the number and size of the network layers. They investigate homological differences
83 between distinct neural network architectures.
84 As far as neural network similarity measures are concerned, the literature is not especially prolific. In Kornblith et al. 233 authors examine similarity measures for representations (meaning, outputs of

---

86
different layers) of neural networks based on canonical correlation analysis. However, note that this method compares neural network representations (intermediate outputs), not the neural networks themselves. Remarkably, in Ashmore and Gashler R authors do deal with the intrinsic similarity of neural networks themselves based on Forward Bipartite Alignment. Specifically, they propose an algorithm for aligning the topological structures of two neural networks. Their algorithm finds optimal bipartite matches between the nodes of the two MLPs by solving the well-known graph cutting problem. The alignment enables applications such as visualizations or improving ensembles. However, the methods only works under very restrictive assumptions and this line of work does not appear to have been followed up.
87
88
89
90
91
93
94
95
Finally, we note that there has been a considerable growth of interest in applied topology in the !
96
recent years. This popularity increase and the development of new software libraries along with the growth of computational capabilities, have empowered new works. Some of the most remarkable libraries are Ripser [2 5 and Flagser [26]. They are focused on the efficient computation of PH.
97
98
99
For GPU-Accelerated computation of Vietoris-Rips PH, Ripser++ [77 offers an important speedup. The Python library we are using, Giotto-TDA [31, makes use of both above libraries underneath.
101
We have seen that there a trend towards the use of algebraic topology methods for having better understanding of phenomena of neural networks and having more principled deep learning algorithms. Nevertheless, little to no works have proposed neural network characterizations or similarity measures based on intrinsic properties of the networks, which is what we intend to do.
102
103
104
105
3
Methodology
106
In this section, we propose our method, which is heavily based on concepts from algebraic topology. We refer the reader to the Supplementary Material for the mathematical definitions. In this section, we also describe the conducted experiments.
107
108
109
Intrinsically characterizing and comparing neural networks is difficult, unsolved problem. First, the network should be represented in an object that captures as much information as possible and then it should be compared with measure depending on the latent structure. Due the stochasticity of both the initialization and training procedure, networks are parameterized differently. For the same task, different functions that effectively solve it can be obtained. Being able to compare the trained networks can be helpful to detect similar neural structures.
110
112
114
115 We want to obtain topological characterizations associated to neural networks trained on given task. For doing so, we use the Persistence Homology (from now on, PH) of the graph associated to a neural network. We compute the PH for various neural networks learned on different tasks. We then compare all the diagrams for each one of the task.
116
118
119
More specifically, for each of the studied tasks (image classification on MNIST, Fashion MNIST and CIFAR- 10; language identification, and text classification on the Reuters dataset) we proceed as follows:
120
121
122
We train several neural network models on the particular problem.
123
We create a directed graph from the weights of the trained neural networks (after changing the direction of the negative edges and normalising the weights of the edges).
125
We consider the directed graph as simplicial complex and calculate its PH, using the weight of the edges as the filtering parameter, which range from 0 to 1. This way we obtain the so-called Persistence Diagram.
126
127
128
We compute the distances between the Persistence Diagrams (prior discretization of the Persistence Diagram so that can be computed) of the different networks.
129
130
Finally, we analyze the similarity between different neural networks trained for the same task, for a similar task, and for completely different task, independently of the concrete architecture, to see whether there topological similarity.
131
132
2For example, the two neural networks "must have the same number of units in cach of their corresponding layers", and the match is done layer by layer.
http://hha.aahaaaammmmpppllllloa ihttpa: ptttpa://ww// //www/ . www..maataaa.aeeeeeeoom www.soaae.ee..ee/oooeeaeoo
"For more details, see Section 3.2
3

---

133
As baselines, we set two standard matrix comparison methods that are the 1-Norm and the Frobenius norm. Having adjacency matrix A and B, we compute the difference as norm(A B). However, these methods only work for matrices of similar size and thus, they are not general enough. We could also have used the Fast Approximate Quadratic assignment algorithm suggested in Vogelstein et al. [4]] but for large networks this method becomes unfeasible to compute.
134
135
136
137
138
3.1
Proposal
139
Our method is as follows. We start by associating to a neural network a weighted directed graph that is analyzed as an abstract simplicial complex consisting on the union of points, edges, triangles, tetrahedrons and larger dimension polytopes (those are the elements referred as simplices). Abstract simplicial complexes are used in opposition to geometric simplicial complexes, generated by point cloud embedded in the Euclidean space I..
140
141
142
143
144
Given a trained neural network, we take the collection of neural network parameters as directed and weighted edges that join neurons, represented by graph nodes. Biases are considered as new vertices that join target neurons with an edge having given weight. Note that, in this representation, we lose the information about the activation functions, for simplicity and avoid representing the network as multiplex network. Bias information could also have been ignored because we want large PH groups that characterize the network, while these connections will not change the homology group dimension of any order.
145
148
147
148
149
150
151
For negative edge weights, we reverse edge directions and maintain the absolute value of the weights. We discard the use of weight absolute value since neural networks are not invariant under weight sign transformations. This representation is consistent with the fact that every neuron can be replaced by neuron from which two edges with opposite weights emerge and converge again on another neuron with opposite weights. From the point of view of homology, this would be represented as closed cycle.
152
153
154
155
156
157
We then normalize the weights of all the edges expressed in Equation D where is the weight to normalize, W are all the weights and Č is an smoothing parameter that we set to 0.000001. This smoothing parameter is necessary as want avoid normalized weights of edges to be O. This is
158
159
180
because O implies a lack of connection.
w
max(1
min (W)l) ()
(I)
max( (raax(mmx(( (W)
161 Given this weighted directed graph, we then define a directed flag complex associated to it. Topology of this directed flag complex can be studied using homology groups H,, In this work we calculate homology groups up to degree 3 (Hg-Hfs) due to computational complexity and our neural network representation method's layer connectivity limit.
162
163
164
165
The dimensions of these homology groups are known as Betti numbers. The i-th Betti number is the number of f-dimensional voids in the simplicial complex (Bo gives the number of connected components of the simplicial complex, B1 gives the number of non reducible loops and so on). For a deeper introduction to algebraic topology and computational topology, we refer to Edelsbrunner and
167
168
169
Harer [1]. Ghrist [7].
170
We work with family of simplicial complexes, Ke, for range of values of e E IR so that the complex at step & embedded in the complex at & for & S E41,, i.e. Ke C K our case, E is the minimum weight of included edges of our graph representation of neural networks.
171
172
173
The nested family of simplicial complexes is called a filtration. We calculate sequence of homology groups by varying the E parameter, obtaining a persistence homology diagram. PH calculations are performed on Zz.
174
175
176
This filtration gives a collection of contained directed weighted graph or simplicial complex K S Kg C Kg. c... K where r [0, 1] and Emin 0, Emax 1 (recall that edge weights are normalized).
178
179
Given a filtration, one can look at the birth, when a homology class appears, and death, the time when the homology class disappears. The PH treats the birth and the death of these homological features in Ke for different values. Lifespan of each homological feature can be represented as
180

---

182
an interval (birth, death), of the homological feature. Given a filtration, one can record all these intervals by Persistence Barcode (PB) g or Persistence Diagram (PD), as a collection of multiset of intervals.
183
184
185
As mentioned previously, our interest in this work is compare PDs from two different simplicial complexes. There are two distances traditionally used to compare PDs, Wasserstein distance and Bottleneck distance. Their stability with respect to perturbations on PDs has been object of different
186
187
188
studies [2].
In order to make computations feasible and to obviate noisy intervals, filter the PDs by limiting the minimum PD interval size. We do so by setting a minimum threshold n = 0.01. Intervals with lifespan under this value are not considered. Additionally, for computing distances, we need to remove infinity values. As we are only interested in the deaths until the maximum weight value, we replace all the infinity values by 1.0.
190
192
198
194
Wasserstein distance calculations are computationally hard for large PDs (each PD of our NN models has a million persistence intervals per diagram). Therefore use vectorized version of PDs instead, also called PD discretization. This vectorized version summaries have been proposed and used on recent literature [ 6 7 24 29).
196
198
For the persistence diagram distance calculation, we use the Giotto-TDA library 31 and compute the following supported vectorized persistence summaries: 1. Persistence landscape. 2. Weighted
199
200
silhouette. 3. Heat vectorizations.
201
3.2
Experimental Framework
202
Datasets
To determine the topological structural properties of trained NNs, we select different kinds of datasets. We opt for four well-known benchmarks in the machine learning community and one regarding language identification: (1) the MNIST dataset for classifying handwritten digit images, (2) the Fashion MNIST (36) dataset for classifying clothing images into 10 categories, (3) the
203
204
205
CIFAR-IC (CIFAR) (CIFAR) dataset for classifying 10 different objects. (4) the Reuters dataset for classifying
208
207
news into 46 topics, and (5) the Language Identification Wikipedia datase for identifying 7 different languages.
208
209
We selected these datasets because, apart from being well-known benchmarks, the performances without transfer learning are good enough and they have different data types and sizes. For CIFAR-
210
211
10 and Fashion MNIST datasets we train a Convolutional Neural Network (CNN) first, and the convolutional layers are shared between all the models of the same dataset as a feature extractor.
212
213
Recall that in this work we are focusing on MI.s,, so we do not consider that convolutional weights. For the MNIST, Reuters and Language Identification datasets, we use an MLP. For Reuters and Language identification datasets, we vectorize the sentences with character frequency.
214
215
Experiments Pipeline We study the following variables (hyperparameters): 1. Layer width, 2. Number of layers, 3. Input order 4. Number of labels (number of considered classes).
216
218
We define the base architecture as the one with a layer width of 512, 2 layers, the original features order, and considering all the classes (10 in the case of MNIST, Fashion MNIST and CIFAR, 46 in the case of Reuters and 7 in the case the language identification task). Then, doing one change at a time, keeping the rest of the base architecture hyperparameters, we experiment with architectures with the following configurations:
219
220
222
223
Layer width: 128, 256. 512 (base) and 1024.
224
Number of layers: (base), 4, 6, 8 and 10.
225
Input order: 5 different randomizations (with base structure), the control experiment.
226
Number of labels (MNIST, Fashion MNIST, CIFAR-10): 2, 4, 6, 8 and 10 (base).
http://yann. www.:/aaannneen..mmm http://..eeeeexmmmeeeeeee
htttps://ww.ccr www.ssttoooooooooooooooo walloo..ddd///// waadu//kri//// cifar html
hhttpp://wwwt http:://wwwwlllyhbb..oo wtt.e/uww..ooo/ooonooooooeeaeeee hwt.tt/taaaaaatataaaaaaaaaaaeeee htaaataagaagaaaaaaaaaaaaaiaiaooa
OOder of the input features, the control experiment. This one should definitely not affect the performance in the neural networks, so if our method is correct, it should be uniform per the proposed topological distances.
5

---

9
10
11
11
12
13
13
14
15
15
16
16
17
17
18
18
N28
(a) Reuters
(b) Language Identification
Figure 1: Distance matrices using Silhouette discretization.
Number of labels (Reuters): 2, 6, 12, 23 and 46 (base).
228
Number of labels (Language Identification): 2, 3, 4, and (base).
Note that this is not grid search over all the combinations. We always modify one hyperparameter at a time, and keep the rest of them as in the base architecture. In other words, we experiment with all the combinations such that only of the hyperparameters is set non-base value time.
230
232
For each dataset, we train 5 times (each with different random weight initialization) each of these neural network configurations. Then, we compute the topological distances (persistence landscape, weighted silhouette, heat) among the different architectures. total, we obtain 5 5 3 distance matrices (5 datasets, 5 random initializations, 3 distance measures). Finally, initializations, such that we get 5 3 matrices, one for each distance on each dataset. All the matrices have dimensions 19 19, since 19 is the number of experiments for each dataset (corresponding to the total the number of architectural configurations mentioned above). Note that the base architecture appears 8 times (1, the number of neurons per layer, on the number of layers, I on the number of labels and the 5 randomizations of weight initializations).
234
average the 5 random
236
238
240
All experiments were executed in machine with 2 NVIDIA V100 of 32GB, 2 Intel(R) Xeon(R) Platinum 8176 CPU 2.10GHz, and of 1.5TB RAM, for total of around days.
242
243
The code and results are fully open source under MIT license.
Results & Discussion
244
245
Results from control experiments can be seen in the third group on Figures [ and In these figures, groups separated visually using white dashed lines. Exper- iments groups are specified in Table Control exper- iments all the images appear very dimmed, which means that they are very similar, as expected. Recall that the control experiments consist of 5 (randomiza- (executions) and that 25 different neural networks have been trained; each one of the network has more than 690,000 parameters that have been ran- domly initialized. After the training, results show that these networks have very close topological distance, expected.
246
Number
Experiment
Index
247
Layer size
1-4
249
Number of layers
5-9
250
3 Input order
10-14
251
Number of labels
15-19
tions)
Table 1: Indices of the experiments of the distance matrices.
253
254
255
See Supplementary Material.

---

MNIST
M
The method we present also seems to capture some parts of hyperparameter setup. For in- stance, in Figure 4 we can observe gradual in- crease of distances in the first group regarding layer size meaning that, layer size increases, the topological distance increases too. Similarly, for the number of layers (second group) and number of labels (fourth group) the same situ- ation holds. Note that in Fashion MNIST and CIFAR-10, the distances are dimmer because we dealing with the weights of the CNNs. Recall that the CNN acts as frozen extractor and are pretrained for all runs (with the same weights), such that the MLP layers themselves Fashion MNIST
Fashion MNIST
CIFAR-10
CIFAR-10
Reuters
Renters
Language
Language
MNIST
Fashion MNIST
CIFAR-10 10
Remttrs
Language
MNIST
Faehion MNIST
CIFAR- 10
Renters
Language
(a) 1-norm
Frobenius norm
Figure Control experiments using norms.
Norm
Minimum
Maximum
Mean
Standard deviation
1-Norm
0.6683
4.9159
1.9733
1.5693
Frobenius
0.0670
0.9886
0.4514
0.3074
Table 2: Normalized difference comparison of self-norm against the maximum mean distance of the experiment.
257
For Figure we computed both 1-norm and Frobenius norm (the baselines) for graphs' adjacency matrices of control experiments. Note that for each value of the matrix. In order to show whether the resulting values are positive or negative, we subtract to the maximum difference of each dataset the norm of each cell separately, we take the absolute value and we divide by the maximum difference of each dataset. Therefore, we obtain five values per dataset. Table 2 shows the statistics reflecting that the distance among the experiments large and, thus, they are not characterizing any similarity but rather an important dissimilarity.
the experiment five times, make the mean
259
280
281
282
In contrast, Figure 3 with method (Silhouette), shows perfect diagonal of similarity blocks. In the corresponding numeric results, we obtained show small distances, as shown in Table We can appreciate that each dataset has its hub. This confirms the validity of our proposed similarity measure.
285
267
289
MNIST
271
Faslion MNIST
273
CIFAR-10
275
Reuters
277
Language
278
279
MISSST
Fashion MIVIST
CIF 10
Rantars
Langusge
280
281
282
the only potential source of dissimilarity be- tween runs.
283
Figure 3: Control experiment comparison matrix using Silhouette discretization.

---

9
10
11
11
12
13
13
14
15
15
16
16
17
17
18
18
8
1999 199999
(a) MNIST
(b) Fashion MNIST
7
9
11
12
13
14
16
18
19
N. S
(c) CIFAR-10
0.0
18
Figure 4: Distance matrices using Heat discretization.
Heat distance
Silhouette distance
Dataset
Mean
Deviation
Mean
Deviation
MNIST
0.0291
0,0100
0.1115
0.0364
F. MNIST
0.0308
0.0132
0.0824
0.0353
CIFAR-10
0.0243
0.0068
0.0769
0.0204
Language I.
0.0159
0.0040
0.0699
0.0159
Reuters
0.0166
0.0051
0.0387
0.0112
Table 3: PH distances across input order (control) experiments, normalized by dataset.

---

284
Thus, our characterization is sensitive to the architecture (e.g., if we increase the capacity, distances vary). but the same time, as we saw before, it is not dataset-agnostic, meaning that it also captures whether two neural networks are learning the same problem or not.
285
In Figure 4 Fashion MNIST (Figure 4b) and CIFAR (Figure 4c) dataset results are interestingly different from those of MNIST (Figure 4ap dataset. This is, presumably, because both Fashion MNIST and CIFAR use pretrained CNN for the problem. Thus, must analyze the results taking into account this perspective. The first fully connected layer size is important as it can avoid bottleneck from the previous CNN output. Some works in the literature show that adding multiple
288
289
280
291
292
fully connected layers does not necessarily enhance the prediction capability of CNNs [4, which congruent with our results when adding fully connected layers (experiments 5 9) that result in
293
dimmer matrices than the one from. Concerning the experiments on input order, there is slightly
294
295
more homogeneity than in MNIST, again showing that the order of sample has negligible influence. Moreover, there could have been even more homogeneity taking into account that the fully connected
296
297
network reduced its variance thanks to the frozen weights of the CNN. This also supports the fact
that the CNN is the main feature extractor of the network. As in MNIST results, CIFAR results show that the topologiaal properties are, indeed, a mapping of the practical properties of neural networks.
299
300
We refer to the Supplementary Material for all distance matrices for all datasets and all dis- tances, as well as for the standard deviations matrices and experiment group statisties.
1.0
301
302
303
Conclusions & Future Work
304
10
11
Results from different experiments, in five dif- ferent datasets from computer vision and natural language, lead to similar topological properties and are trivially interpretable, which yields to general applicability.
12
306
13
308
16
17
18
310
The bests discretizations chosen for this work are the Heat and Silhouette. They show better separation of experiment groups, and are effec- tively reflecting changes in a sensitive way. We also explored the Landscape discretization but offers very low interpretability and clearance. In other words, it is not helpful for comparing PH diagrams associated to neural networks.
312
Figure 5: Language Identification dataset PH Land- scape distance matrix.
314
316
317
The most remarkable conclusion comes from the control experiments. The corresponding neural networks, with different input order but the same architecture, are very close to each other. The PH framework does, indeed, abstract away the specific weight values, and captures latent information from the networks, allowing comparisons to based on the function they approximate. The selected neural network representation is reliable and complete, and yields coherent and meaningful results. Instead, the baseline measures, the 1-Norm and the Frobenius norm, implied an important dissimilarity between the experiments in the control experiments, meaning that they did not capture the fact that these neural networks were very similar in terms of the solved problem.
318
319
320
322
324
325
We conclude that our proposed characterization, does, indeed, capture meaningful information from neural network, and the computed distances can serve as effective similarity measure between networks. To the best of our knowledge, this similarity measure between neural networks is the first
326
327
328
of its kind.
329
As future work, we suggest adapting the method to different deep learning libraries and make
330
support popular neural architectures such as CNNs, Recurrent Neural Networks, and Transformers
331
Finally, we suggest performing more analysis regarding the learning of a neural network, and trying to topologically answer the question of how a neural network learns.
33
332

---

Checklist
333
334
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
336
337
(b) Did you describe the limitations of your work? [Yes] See second paragraph of the
338
introduction and last paragraph of the conclusions.
339
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
340
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
343
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
344
(b) Did you include complete proofs of all theoretical results? [N/A]
345
3. If you ran experiments..
346
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] Both code and outputs.
348
349
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] Check the Experimental Framework Section and the code.
350
351
(c) Did you report error bars (e.g., with respect the random seed after running ex- periments multiple times)? [Yes] We include means, standard deviations and raw outputs.
353
354
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] Check Experimental Framework Section.
356
357
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
358
(a) If your work uses existing assets, did you cite the creators? [Yes] In the case of the datasets. We do not use any other additional asset.
359
(b) Did you mention the license of the assets? [No]
(c) Did you include any new assets either in the supplemental material URL? [Yes] Code, results and pictures we have made for explanations.
362
383
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're using/curating? [N/A]
364
365
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
367
5. If you used crowdsourcing conducted research with human subjects...
388
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
389
370
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
373
10

---

References
374
375
[1] H. Adams, T. Emerson, M. Kirby, R. Neville, C. Peterson, P. Shipman, S. Chepushtamova,
376
E. Hanson, F. Motta, and L. Ziegelmeier. Persistence images: A stable vector representation of persistent homology. J. Mach. Learn. Res., 18:8:1-8:35, 2017.
377
378
[2] M. Aktas, E. Akbaş, and A. E. Fatmaoui. Persistence homology of networks: methods and applications. Applied Network Science, 4:1-28, 2019.
380
[3] S. Ashmore and M. Gashler. A method for finding similarity between multi-layer perceptrons
381
by forward bipartite alignment. In 2015 International Joint Conference on Neural Networks (IJCNN), pages 1-7, 2015. doi: 10.1 N0.110///CNN.2215..250
382
383
[4] S. H. S. Basha, S. R. Dubey, V. Pulabaigari, and S. Mukherjee. Impact of fully connected layers on performance of convolutional neural networks for image classification. CoRR, abs/1902.02271,, 2019. htLp://///aarxi org/ 1t////002..0277
384
385
386
[5] U. Bauer. Ripser: efficient computation of vietoris-rips persistence barcodes, 2021.
387 [6] E. Berry, Y.-C. Chen, J. Cisewski-Kehe, and B. T. Fasy. Functional summaries of persistence
diagrams. Journal of Applied and Computational Topology, 4:211-2622 2020.
389
[7] P. Bubenik. Statistical topological data analysis using persistence landscapes. J. Mach. Learn.
390
Res., 16:77-102, 2015.
381
[8] G. Carlsson. Topology and data. Bulletin of the American Mathematical Society, 46:255-308. 2009.
392
393
19] Chazal, V. D. Silva, and S. Oudot. Persistence stability for geometric complexes. Geometriae Dedicata, 173:193-214 2012.
[10] S. Chowdhury. T. Gebhart, S. Huntsman, and M. Yutin. Path homologies of deep feedforward networks. 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pages 1077-1082, 2019.
396
398
[11] J. Clough, I. Öksüz, N. Byrne, V. Zimmer, J. A. Schnabel, and A. P. King. A topological loss function for deep-learning based image segmentation using persistent homology. IEEE transactions on pattern analysis and machine intelligence, PP, 2020.
399
400
401
[12] D. Cohen-Steiner, H. Edelsbrunner, and J. Harer. Stability of persistence diagrams. Proceedings of the twenty-first annual symposium on Computational geomelry, 2005.
402
[13] C. Corneau,, M. Madadi, S. Escalera, and A. Martínez. Computing the testing enror without testing set. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2674-2682, 2020.
404
405
[14] J. Donier. Capacity allocation analysis of neural networks: A tool for principled architecture design. ArXiv, abs/ 902.04485, 2019.
407
408
[15] H. Edelsbrunner and J. Harer. Computational Topology an Introduction. American Mathemat-
ical Society, 2009.
410
[16] T. Gebhart, P. Schrater, and A. Hylton. Characterizing the shape of activation space in deep neu- ral networks. 2019 18th IEEE International Conference On Machine Learning And Applications (ICMLA), pages 1537-1542, 2019.
412
413
[17] R. Ghrist. Elementary Applied Topology. Self-published. 2014.
414
[18] Giblin, Graphs, surfaces, and homology: : an introduction to algebraic topology. Chapman and Hall, 1977.
416
[19] W. H. Guss and R. Salakhutdinov. On characterizing the capacity of neural networks using algebraic topology. ArXiv, abs/1882.00443,, 2018
417
418
[20] C. Hofer, F. Graf, M. Niethammer, and R. Kwitt. Topologically densified distributions. ArXiv, abs/2002.04805,, 2020.
419
420
[21] J. Jonsson. Simplicial complexes of graphs. PhD thesis, KTH Royal Institute of Technology.
2007.
[22] E. Konuk and K. Smith. An empirical study of the relation between network architecture and complexity. 2019 IEFE/CVF International Conference on Computer Vision Workshop (ICCVW), pages 4597-4599, 2019.
423
11

---

425
[23] S. Kornblith, M. Norouzi, H. Lee, and G. E. Hinton. Similarity of neural network represettttsssssssaatttitoino revisited. CoRR, 0bs/1905.00414, 2019.
426
00414
427
[24] Lawson, A. Sholl, J. Brown, B. T. Fasy, and C. Wenk. Persistent homology for the quantitative
428
evaluation of architectural features in prostate cancer histology. Scientific Reports, 9, 2019.
429
[25] B. Liu. Geometry and topology of deep neural networks' decision boundaries. ArXiv, abs/2003.03687, 2020.
[26] D. Litgehetmann, D. Gove, J. Smith, and R. Levi. Computing persistent homology of directed
432
flag complexes. arXiv: Algebruic Topology, 2019.
433
[27] G. Naitzat, A. Zhitnikov, and L. Lim. Topology of deep neural networks. Mach. Learn. Res.,
434
21: 11:184:1-184:40, 2020.
435
[28] K. Ramamurthy, K. R. Varshney, and K. Mody. Topological data analysis of decision boundaries
with application to model selection. ArXiv, abs/ has///1050099999 2019.
437
[29] B. A. Rieck, F. Sadlo, and H. Leitte. Topological machine learning with persistence indicator
438
functions. ArXiv, abs/1907. 13496, 2019.
439
300 B. A. Rieck, M. Togninalli, C. Bock, M. Moor, M. Horn, T. Gumbsch, and K. Borgwardt.
440
Neural persistence: A complexity measure for deep neural networks using algebraic topology.
441
ArXiv, abs/1812.09966. 2019.
[3]] G. Tauzin, U. Lupo, L. Tunstall, J. B. Pérez, M. Caorsi, A.
A. Dassatti, and K. Hess. giotto-tda: A topological data analysis toolkit for machine learning and data exploration, 2020.
443
[32] C. Trallie, N. Saul, and R. Bar-On. Ripser.py: A lean persistent homology library for python. The Journal of Open Source Software, 3(29):925, Sep 2018. doi: 10.21 105/joss.00025. URL. https://doi. org/10. 21105/joss 00925
446
448
[33]
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and
449
I. Polosukhin. Attention is all you need. CoRR, (bs/1776.03762, 2017. URL http://arxiv org/abs/1700 03762)
450
451
[34] T. Vogelstein, J. M. Conroy, V. Lyzinski, I.. J. Podrazik, S. G. Kratzer, E. T. Harley, D. E. Fishkind, R. J. Vogelstein, and C. E. Priebe. Fast approximate quadratic programming for graph matching. PLOS ONE, 10(4):1-17, 04 2015. doi: 10. oon. 0121002. URL
459
454
https://doi.. org/10. 10. 1371/journal, pone. 0121002
[35] S. Watanabe and H. Yamana. Topological measurement of deep neural networks using persistent homology. ISAIM, 2020.
496
457
[36] H. Xiao, K. Rasul, and R. Vollgraf. Fashion-mnist: novel image dataset for benchmarking machine learning algorithms, 2017.
458
459
[37] S. Zhang, M. Xiao, and H. Wang. Gpu-accelerated computation of vietoris-rips persistence barcodes. In Symposium on Computational Geometry, 2020.
460
12


Reviews:
[{"review_id": "N4IFTaS6pUL", "reviewer": "Reviewer_zbft", "summary": "In this work, the authors propose to characterize neural networks with a tool from topological data analysis, called persistence diagram, in order to be able to compare neural networks with different numbers of layers and/or neurons. More specifically, they show that one can filter the neural network computational graph using the edge weights (that are learnt during training of the network) to produce a filtered flag complex, from which persistent homology can be computed. Then, the authors interpreted the distances between the persistence diagrams obtained from networks with varying parameters (number of layers, neurons, labels), and showed that often, the distances have intuitive correlation with the complexity of the neural networks.", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have the following comments:\n\n---It would be good to also mention the topological uncertainty (https://arxiv.org/abs/2105.04404), which is similar to neural persistence, in the related work section.\n\n---How are neuron filtration values computed? Is it the minimum of all filtration values of edges that contain this node?\n\n---As far as I understand, neuron activations are not used in the persistence diagram constructions. This could be added, for instance as potential filtration values for the nodes.\n\n---It seems to me that the relationships between persistence diagrams and network architectures is actually quite intuitive: the more layers or neurons, the bigger the simplicial complexes, and thus the more points you will have in the persistence diagrams. That could explain the distance matrices presented in the article, without any real interpretation or relation with the capacity or generalization error of the architectures.\n\n---I am not sure about how this similarity measure between neural networks can be used in practice. Is there any rule that could be used in order to do model selection, i.e., choose a specific architecture to use for a given data set or data science task, such as classification, regression, transfer learning etc?\n\nEven though I like the proposed idea (and especially the comparison to Frobenius norms, which clearly shows its limitation and the advantages of using persistence), I feel like this work is still in an early shape, and would really benefit from a better interpretation of the distances (in terms of how persistence diagrams are computed), as well as more examples of practical uses.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3"}, "extra_text": {}}, {"review_id": "KRBKHoO3Gq", "reviewer": "Reviewer_g9zG", "summary": "This paper tries to measure the structural similarity of the neural networks by associating them with simplicial complexes. The similarity between neural networks is then measured by the topological feature via persistence homology. The graph is generated by taking the neurons as graph nodes, and the edge weights are defined as the network parameters for the connected neurons.\nThe directed flag complex is attached to the constructed weighted directed graph, then the filtration, a family of simplicial complexes for the neural networks. The persistence diagram and persistence barcode are calculated for the filtration. They are used to show the three persistence summaries: persistence landscape, weighted silhouette and heat vectorizations. \nExperiments were carried out for various benchmarks for typical CNN architectures.", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The topological similarity of neural networks is considered by computing the persistence features for the generated weighted graphs and complexes. It is an interesting idea. The experiments can show the persistence distances (persistence landscape, weighted silhouette and heat vectorizations) for the different architectures of neural networks.\n\n* The proposed method and experiments consider the depth and width of the NNs, and how the neurons are linked, which are important factors for their performance. Curious how the computational blocks are linked can be reflected in the persistence metric for the similarity of the NNs.\n\n* It is not clear whether the similarity reflected by the persistence homology has anything to do with the performance of the NNs for classification tasks.\n\n* Maybe some pictures of the graphs and complexes for a specific CNN network can be shown. \n\n* Line 170-171, there is inconsistent for $\\epsilon$ and $\\epsilon_t$.\n\n* Line 265, remove `show' in ', we obtained show small distances,'", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The architecture of the NNs is determined by the initial design of the network and the specific task it works on. The proposed PH-based similarity measure is developed for the topological architecture of the NNs, where the trained network parameters are incorporated as edge weights. There are two main applications, including persistence inference: to learn the properties of the underlying manifold of data and to use persistence diagram to distinguish different types of data sets where the distances of certain points from the diagonal show the significant features of the data. So the features induced by the persistence homology is not direct to measure for the distance between network architectures. On the other hand, the experiments seem not sufficient to support the expected claim of the paper. More tests are needed for the various architectures of NNs, such as CNN, U-Net, ResNet, to show how the persistence feature can distinguish the architecture of the neural networks.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {}}, {"review_id": "2DqHUQ3mnoJ", "reviewer": "Reviewer_oDNX", "summary": "This paper proposes a method to compare trained neural network based on extracting topological descriptors from their graphs. Two networks are then compared through the lens of these descriptors yielding a similarity score between networks, with the expectation that networks aiming at solving a similar task should be measured as being similar. Authors showcase their approach on qualitative experiments on different datasets of different nature. ", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nQuality:\nThe paper considers an interesting and possibly important question in machine learning, that is proposing similarity measures between neural networks. However, in my opinion, the paper suffers too many weaknesses in its current state to meet NeurIPS requirements. The two main issues I identify is the lack of formalism and the limited numerical experiments, see below for details.\n\nClarity / Motivation:\nThe paper lacks formalism and many (core) notions are not explicitly defined, making impossible a rigorous understanding of what should be expected and judged as a good result. Given that the paper addresses a somewhat novel problem (\"there are no well-known methods for efficiently characterizing neural networks\", line 20), it is crucial to set a clear and rigorous baseline. \n- \"solving a task\" (line 17): when should we consider that two networks solve a given task? For instance, two networks may be trained on a same dataset (say, MNIST) with similar train/test accuracies, but one may be tuned to be robust to outliers or adversarial attacks, be calibrated, etc. (Why) Should these networks be similar?\n- \"practical properties of neural networks\" (line 299).\nFurthermore, there is no intuition given on why should the proposed approach be relevant to solve this problem. The persistence diagram (PD) of a given network (turned into a flag complex) does depend on the network weights (while the paper claims that \"two networks solving the same task should be measured as similar, regardless of the specific weights\"), thus why should we expect the (distance between) PDs (vectorization) to be weight/architecture independent? (*)\n\nExperiments:\nIn my opinion, the experiments face some issues.\n- [Limited benchmark] The experiments are restricted to MLP (line 23), which appears as a major limitation given the goal to design a descriptor \"with independence to the architecture\" (line 22). \n- [Baseline] The paper use matrix norms as a baseline, evaluated on the control experiment. However, the control experiments consists of switching the _input order_. If my understanding is correct, it is thus *obvious* that the graphs' adjacency matrices will be different (while, indeed, the PD is invariant to such transformations).\n- [Scalability] The method requires to store the whole network graph at once, which is likely to be infeasible with more sophisticated network architectures. Even small CNN, when represented as graphs, may be very large (*) and extracting the corresponding PD is likely to be infeasible. To that respect, the approach of Rieck at al., [30], is more reasonnable.\n- [Qualitative Results] The results are mostly qualitative, making harder to assess if they valid - or not - the proposed approach. However, my understanding of these results suggest that the proposed approach does not meet the expectations claimed in the introduction (independence to weights/architecture). For instance, on MNIST (Figure 4, a), a network in situation 1 is closer to networks in situation 15-19 than most networks in its own cluster (1-4). Complementary experiments in the supplementary material (e.g. Figure 6) do not exhibit a clear block structure either. Furthermore, as said in lines 272, 284, the proposed characterization is not architecture/weight independent, one of the claimed goal mentioned in the introduction.\n\nOn a more positive side, I think the results proposed in Figure 3 are interesting. Showing that the PD of the flag complex is invariant with respect to the network training would be an interesting contribution. \n\nOriginality:\nThe approach proposed by the paper is novel and may be interesting, provided sufficient formalism on its goals and more convincing experiments.\n\n\nMinor remarks and suggestions:\n- (Eq 1) I think the denominator is simply $\\max(|W|)$. \n- I think considering _superlevel sets_ filtrations when building the PDs would simplify notations. \n- (line 240) I think \"5 randomizations of weight initializations\" should be \"5 random input orderings\". \n- Lines 259-263: clarity would be greatly improved with some equation here. \n- The results on the Persistence Lanscape, and its discrepency with respect to the persistence silhouette, is somewhat surprising given the similarity of these two objects. Which distance/norm is used on these vectorizations? Also, why not computing the diagram distances directly?\n- (*) For instance, the [PyTorch CNN tutorial for MNIST](https://github.com/pytorch/examples/blob/master/mnist/main.py) connects (among other things) a (26x26x32) layer to a (24x24x64) one.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "NA", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "10"}, "extra_text": {}}, {"review_id": "1zTki4q_Gg", "reviewer": "Reviewer_WKaR", "summary": "This paper presents the similarity measure between neural networks based on the graph characterization strategy of neural networks. This graph characterization strategy is based on Persistent Homology in which neural networks can be represented as abstract simplicial complex. In this work, neural networks are considered as weighted directed graphs which can be further considered as simplicial complexes and we can obtain Persistent Diagrams of these graphs. Then, we can compute distances between Persistent Diagrams of different neural networks. Only MLP layers are considered to compute the PDs. Results are shown on four popular datasets from two different areas of images and text. Experiments with different persistence summaries are provided to compute the distance matrices and results are compared between them on different datasets.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths: \n\n1. The idea of considering weight matrices in MLP as weighted directed graphs and comparing MLPs based on PH is interesting. \n2. In the area of NN+TDA, where there are only a few works to compare NNs, this work can be a notable contribution. \n3. Paper is clearly explained and easy to follow. \n4. Experiments are conducted on different hyperparameters of NN for different sets of values. \n5. Some of the experiments lead to results that are also intuitively correct, especially in the case of results from heat and silhouette discretization.\n\nWeaknesses: \n\n1. There is no example figure showing NN layers and how the layers are converted to the weighted directed graph, combining all the layers into one graph, etc. \n2. Some details are not clearly mentioned. For example, are we considering neurons from layer1 and layer2 as the |V| in the graph for 2 layers NN? And all the neurons from all the layers of NN as |V| for more than 2 layers NN. \n3. Method takes only MLP into account. However, CNNs are also the most used which is adapted by all kinds of networks these days. Even the results from Fig 4 highlight this point. \n4. Due to point 2. and 3., the method seems trivial, and I’m not sure about its significance. \n5. I see that the expected results are captured in some experiments for some datasets due to only MLP weights being taken or suitable discretization are used, etc. This raises the question of the generalizability of this method to compare NNs. \n6. From the results, it has been observed that the heat and silhouette discretization work best. On the other hand, landscape discretization does not work well here. But why is that the case? Why doesn’t landscape work or heat and silhouette work? This is not analyzed or explained in the paper. \n7. From the related work, I understand that there is no work that compares NN using a distance matrix based on TDA or PH. However, I can see the possibility of extending neural persistence work to compare the average neural persistence of two NNs using bottleneck or Wasserstein distance. This result can be added and compared with the proposed method in the paper. ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No. This paper is based on comparing neural network architectures. I believe there is no negative societal impact of this work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a similarity measure for neural networks, which is an interesting topic.  While the proposed measure based on PH is of some significance, a systematic understanding of the resulting distances is still premature, and it is not clear how to use the measure in practice.  From these evaluations, we need to judge that this paper is unfortunately below the acceptance threshold.  
Decision: Reject