Title: Characterizing and Measuring the Similarity of Neural Networks with Persistent Homology
Authors: David Pérez Fernández
Asier Gutiérrez-Fandiño
Jordi Armengol-Estapé
Marta Villegas
Venue: NeurIPS 2021
Keywords: Neural Networks
Topological Data Analysis
similarity
Persistent Homology

Abstract: Characterizing the structural properties of neural networks is crucial yet poorly understood, and there are no well-established similarity measures between networks. In this work, we observe that neural networks can be represented as abstract simplicial complex and analyzed using their topological 'fingerprints' via Persistent Homology (PH). We then describe a PH-based representation proposed for characterizing and measuring similarity of neural networks. We empirically show the effectiveness of this representation as a descriptor of different architectures in several datasets. This approach based on Topological Data Analysis is a step towards better understanding neural networks and serves as a useful similarity measure.

TL;DR: We provide a method to compare and characterize Neural Networks

Review discussion:
Strengths:
1. The idea of considering weight matrices in MLP as weighted directed graphs and comparing MLPs based on PH is interesting.
2. In the area of NN+TDA, where there are only a few works to compare NNs, this work can be a notable contribution.
3. Paper is clearly explained and easy to follow.
4. Experiments are conducted on different hyperparameters of NN for different sets of values.
5. Some of the experiments lead to results that are also intuitively correct, especially in the case of results from heat and silhouette discretization.
Weaknesses:
1. There is no example figure showing NN layers and how the layers are converted to the weighted directed graph, combining all the layers into one graph, etc.
2. Some details are not clearly mentioned.
For example, are we considering neurons from layer1 and layer2 as the |V| in the graph for 2 layers NN? And all the neurons from all the layers of NN as |V| for more than 2 layers NN.
3. Method takes only MLP into account.
However, CNNs are also the most used which is adapted by all kinds of networks these days. Even the results from Fig 4 highlight this point.
4. Due to point 2. and 3., the method seems trivial, and I’m not sure about its significance.
5. I see that the expected results are captured in some experiments for some datasets due to only MLP weights being taken or suitable discretization are used, etc. This raises the question of the generalizability of this method to compare NNs.
6. From the results, it has been observed that the heat and silhouette discretization work best.
On the other hand, landscape discretization does not work well here.
But why is that the case? Why doesn’t landscape work or heat and silhouette work? This is not analyzed or explained in the paper.
7. From the related work, I understand that there is no work that compares NN using a distance matrix based on TDA or PH.
However, I can see the possibility of extending neural persistence work to compare the average neural persistence of two NNs using bottleneck or Wasserstein distance. This result can be added and compared with the proposed method in the paper.
No
Many thanks for your kind feedback regarding the novelty and the write-up.
Regarding your comment on the lack of a figure to show how NN layers are converted to a weighted directed graph, please check Figure 1 from the supplementary material, not included in the main body of the article due to space constraints. If it is not enough, we can improve it to make it more intuitive for the camera-ready version.
Regarding your comment on the lack of certain details, please check section 3.1 and Figure 1 from supplementary material. If you consider that this is not sufficiently explanatory, we can expand the description of our proposal for the camera-ready. As stated in line 144, "Given a trained neural network, we take the collection of neural network parameters as directed and weighted edges that join neurons, represented by graph nodes"; the neurons correspond to the nodes of the associated graph.
With regard to the fact that we only use MLPs, while our article, aiming to design a new method (not adapting an existing one) focuses on MLPs for the sake of simplicity, we note that it is also applicable to CNNs, which have an MLP equivalent. The same holds for Transformer architectures, and we plan to further investigate the applicability of the method to these other architectures.
However, the proposed simplicial representation model does not seem to be directly applicable to RNNs given the existence of direct multiplications of the inputs (LTSM and GRU model).
Finally, we also note that we experimented with a static CNN layer previous to the MLP.
Answering your point 4 in weaknesses, There are other representations in the literature but in our opinion, they have the following problems:
1) When working with activations, the representation of the NN depends on the data it processes (Corneau et al).
2) When consecutive layer analysis is used, the global topology of the network is not taken into account (there are trivial global transformations of an NN that are not captured by analyzing pairs of layers) (Rieck et al).
3) The matrix representation does not allow comparison of NNs of different architectures and the simple permutation of neurons is a major problem in the comparison of NNs.
4) There are trivial modifications of an NN, adding pairs of neurons with reversed weight axes, which create an equivalent NN but the above representations fail.
For these reasons, we believe that our work on TDA is a promising approach.
Regarding your point 5 in weaknesses, we note that this is an experimental paper in which we searched for the best representation and also the best way to evaluate the distance between trained NNs on this representation. Each comparison has been repeated 5 times with different initializations of the NN. See line 232.
In comment #6 on weaknesses, regarding your question on discretization methods, as shown in comparative studies such as https://openreview.net/pdf?id=X1bxKJo5_qL, different distances and different ways of vectorizing persistence diagrams have results with different levels of stability and quality. In the case of our paper, we focus on showing the results with different methods but giving a theoretical explanation was out of the scope of this paper. We just show an empirical result that we believe to be very relevant.
Finally, regarding the related work, we are not sure we understand your comment. We see different possible interpretations of your comment:
1) If it refers to averaging the distances in the experiments. As explained in line 232, 5 different initializations have been performed and the average distance is shown.
2) If it refers to a new form of average persistent homology, we do not know its existence and topological meaning.
3) In the case that it refers to Rieck's paper, our representation applies to the complete network, not to pairs of layers, and does not use absolute values but a directed complex flag.
Quality:
The paper considers an interesting and possibly important question in machine learning, that is proposing similarity measures between neural networks.
However, in my opinion, the paper suffers too many weaknesses in its current state to meet NeurIPS requirements. The two main issues I identify is the lack of formalism and the limited numerical experiments, see below for details.
Clarity / Motivation:
The paper lacks formalism and many (core) notions are not explicitly defined, making impossible a rigorous understanding of what should be expected and judged as a good result. Given that the paper addresses a somewhat novel problem ("there are no well-known methods for efficiently characterizing neural networks", line 20), it is crucial to set a clear and rigorous baseline.
- "solving a task" (line 17): when should we consider that two networks solve a given task?
For instance, two networks may be trained on a same dataset (say, MNIST) with similar train/test accuracies, but one may be tuned to be robust to outliers or adversarial attacks, be calibrated, etc. (Why) Should these networks be similar?
- "practical properties of neural networks" (line 299).
Furthermore, there is no intuition given on why should the proposed approach be relevant to solve this problem. The persistence diagram (PD) of a given network (turned into a flag complex) does depend on the network weights (while the paper claims that "two networks solving the same task should be measured as similar, regardless of the specific weights"), thus why should we expect the (distance between) PDs (vectorization) to be weight/architecture independent? (*)
Experiments:
In my opinion, the experiments face some issues.
- [Limited benchmark] The experiments are restricted to MLP (line 23), which appears as a major limitation given the goal to design a descriptor "with independence to the architecture" (line 22).
- [Baseline] The paper use matrix norms as a baseline, evaluated on the control experiment.
However, the control experiments consists of switching the _input order_. If my understanding is correct, it is thus *obvious* that the graphs' adjacency matrices will be different (while, indeed, the PD is invariant to such transformations).
- [Scalability] The method requires to store the whole network graph at once, which is likely to be infeasible with more sophisticated network architectures. Even small CNN, when represented as graphs, may be very large (*) and extracting the corresponding PD is likely to be infeasible. To that respect, the approach of Rieck at al., [30], is more reasonnable.
- [Qualitative Results] The results are mostly qualitative, making harder to assess if they valid - or not - the proposed approach.
However, my understanding of these results suggest that the proposed approach does not meet the expectations claimed in the introduction (independence to weights/architecture).
For instance, on MNIST (Figure 4, a), a network in situation 1 is closer to networks in situation 15-19 than most networks in its own cluster (1-4). Complementary experiments in the supplementary material (e.g. Figure 6) do not exhibit a clear block structure either.
Furthermore, as said in lines 272, 284, the proposed characterization is not architecture/weight independent, one of the claimed goal mentioned in the introduction.
On a more positive side, I think the results proposed in Figure 3 are interesting. Showing that the PD of the flag complex is invariant with respect to the network training would be an interesting contribution.
Originality:
The approach proposed by the paper is novel and may be interesting, provided sufficient formalism on its goals and more convincing experiments.
Minor remarks and suggestions:
- (Eq 1) I think the denominator is simply $\max(|W|)$.
- I think considering _superlevel sets_ filtrations when building the PDs would simplify notations.
- (line 240) I think "5 randomizations of weight initializations" should be "5 random input orderings".
- Lines 259-263: clarity would be greatly improved with some equation here.
- The results on the Persistence Lanscape, and its discrepency with respect to the persistence silhouette, is somewhat surprising given the similarity of these two objects. Which distance/norm is used on these vectorizations?
Also, why not computing the diagram distances directly?
- (*) For instance, the [PyTorch CNN tutorial for MNIST](https://github.com/pytorch/examples/blob/master/mnist/main.py) connects (among other things) a (26x26x32) layer to a (24x24x64) one.
No
We appreciate the insightful feedback and comments.
Regarding your comment on the lack of formalism, we actually have included abundant formal aspects of the method in the supplementary material, due to space limitations. We can move some of them to the main body of the article for the camera-ready version.
To the best of our knowledge, comparisons of NNs of similar architectures trained on the same problem can be found in the literature (generally, with the same initialization). In our case, we are looking for a representation and comparison method that does not have these limitations (same problem, same initialization). It is therefore difficult to find a baseline.
Regarding your question on when we should consider that two networks are solving the same task, we simply meant having been optimized for the same dataset (MNIST, CIFAR, Language identification, ...) without any further constraint (e.g., robustness). We find very interesting your question on whether 2 networks trained on the same dataset and same accuracies but different robustness should be similar, but we do not experiment on this but believe that could be interesting as future work.
However, we suggest revisiting the experiment on the number of labels. They could be seen as subproblems of the original dataset, and the distances are interesting: a small change on the topology of the network (few edges/nodes more or less) but with a medium-level change on the learning.
Regarding the comments on the experiments:
Limited benchmark: The article focuses on MLPs. Our method is also applicable to CNNs, since they have an MLP equivalent, and also to Transformers. We are planning to further investigate the applicability of our method to these other architectures.
However, we acknowledge that the proposed simplicial representation model does not seem to be directly applicable to RNNs given the existence of direct multiplications of the inputs (LTSM and GRU model).
Finally, we also note that we experimented with a static CNN layer previous to the MLP.
Baseline: The baseline is simple, but as we said, it is difficult to find in the literature methods without the limitations of requiring the same initializations or being compared in the same dataset.
Also, consider that the baseline (as well as the proposed method) has to be time-bound and that the measure must be applied to compare matrices of different sizes or rectangular.
Scalability: Some considerations 1) Regarding Rieck et al, they use the absolute value of weights. Suppose that we reverse the sign of all weights in a trained NN, it is the same NN?
Also, there are trivial global transformations performed by an NN that are captured
Qualitative results: Regarding your comment on the case of MNIST, from our point of view, in the first case there is a strong resizing of the network, which could lead to duplicities or more complex topological entities, and on the other hand, the number of labels of the problem is being reduced, making a minimum topological change, so it seems that most of the network, in charge of reducing the problem, is topologically similar. We must take into account that that experiments in 15-19 are adding a few neurons whilst experiments 1-4 are adding many more neurons.
Regarding your comments on the results in Figure 3, indeed, the PD of the flag complex is invariant to the training of the network.
In fact, we take it as the control experiment for each problem.
With regard to your comment on the originality of our work, we appreciate that you consider its novelty. We note, though, that this is empirical work from which some initial qualitative conclusions can be drawn. In general terms, we do not have a distance between NNs trained for the same dataset, let alone for different datasets. For the latter, we should define a distance between training spaces (input data and labels). In our case, in the design of the experiments, we have used the intuition that MNIST-N-labels and MNIST-M-labels are more similar to each other than to other problems because their training example spaces are a subset of each other.
We do not really have a universal metric to compare any training space.
Thanks for the minor remarks and suggestions, we will fix/apply them in the camera-ready version, but a few comments:
We are not sure we understand your comment on “superlevel sets filtrations”.
The norm used in these vectorizations is documented in Giotto: Check the additional material for more details and/or the Giotto documentation: https://giotto-ai.github.io/gtda-docs/0.5.1/theory/glossary.html#weighted-silhouette
We don’t compute the diagram distances directly: Persistence diagrams typically have 1M intervals. Calculating the distances without vectorisation requires calculating the optimal transport from 1M to 1M intervals, and this calculation is sufficiently complex that it cannot be completed in reasonably computable times with our resources.
For this reason, we have opted for the use of vectorizations of the PDs.
Thank you for your response. I still have some interrogations / remarks regarding this work.
> we actually have included abundant formal aspects of the method in the supplementary material, due to space limitations. We can move some of them to the main body of the article for the camera-ready version.
I checked the supplementary during the reviewing process. The formal content (Appendix I) is mostly related to Persistent homology, diagrams, and their vectorizations. I do not think it would be worth including it in the main document. My comment on the lack of formalism was mostly related to _neural networks_ aspects: what is (formally) considered as a NN in this work, on what does it depend (training data, architecture, (stochastic) training process, etc.), and hence what "similar" means.
> we simply meant having been optimized for the same dataset (MNIST, CIFAR, Language identification, ...) without any further constraint (e.g., robustness)
This should be state more precisely/formally in my opinion. In that context, why do some experiments focus on changing the network architecture for a given dataset (Figure 4, Figure 5)? These figures are hard to interpret: there seems to be a block structure but also variation inside diagonal blocks. Furethermore, this goal is not met according to the claim _our characterization is sensitive to the archutecture_ (line 284). As said, I feel that Figure 3 is interesting (it stresses the dependence on the dataset in a convincing way) but the paper does not serve it properly in my opinion.
Furthermore, this goal seems somewhat surprising; say that I train a really simple network (e.g. a linear regression) on a complex dataset (e.g. CIFAR 10). Why/should this network be considered similar to a sophisticated one (e.g. a wideResNet?)? I guess that one implicitly considers "good" (that achieve good test accuracy?) networks for a given task?
> Our method is also applicable to CNNs, since they have an MLP equivalent
I agree, but as pointed in my review, the "naive equivalent" would lead to an extremely large graph/flag complex if I am correct (due to the typically large number of filters involved).
> Regarding Rieck et al, they use the absolute value of weights. Suppose that we reverse the sign of all weights in a trained NN, it is the same NN?
Isn't it what is done in Eq. (1) as well?
Furthermore, my comment was about working "layer-per-layer" as done by Rieck et al., reducing computational burden (see below). I noticed your answer to reviewer WKaR: one may miss global information in the network, it may be true, but the computational benefit may overcome the information loss (or not, I don't know, that is just a suggestion).
> indeed, the PD of the flag complex is invariant to the training of the network.
Is that an empirical observation, or a theoretical claim?
> Persistence diagrams typically have 1M intervals. (...)  For this reason, we have opted for the use of vectorizations of the PDs.
I agree with the latter point.
But the former stresses the lack of scalability of the proposed approach: the networks considered in this work, though non trivial, are fairly small compared to "state-of-the-art" networks yet they lead to very large diagrams which (probably worse in terms of memory footprint) require first to store extremely large simplicial complex.
> We are not sure we understand your comment on “superlevel sets filtrations”
Superlevel set filtrations are obtained by filtering graphs in decreasing order (usual persistent homology consider sublevel sets, i.e., filtering with increasing filtration values): one starts by adding vertices/edges with highest filtration values and then decrease the threshold progressively. This is implicitly what is done in Eq. (1) when considering $1 - |w|/\max(|W|)$ : edges with higher weight $w$ are added first.
> I checked the supplementary during the reviewing process. The formal content (Appendix I) is mostly related to Persistent homology, diagrams, and their vectorizations. I do not think it would be worth including it in the main document. My comment on the lack of formalism was mostly related to neural networks aspects: what is (formally) considered as a NN in this work, on what does it depend (training data, architecture, (stochastic) training process, etc.), and hence what "similar" means.
Understood. We will include a more formal definition of the considered NNs and the variables on which it depends in the revised version of the article. Let us include some clarifications wrt the notion of similarity in this answer.
We consider about 2 similarities:
a) Similarity between problems (MNIST, CIFAR, language identification).
b) Similarity between NNs, understood as mathematical objects.
Regarding the similarity between problems, we don't have a universal mathematical definition, but an intuitive notion of it. MNIST10 (all 10 classes) should be more similar to MNIST2 (2 classes) than the language identification task. We also believe we can say that MNIST10 is closer to MNIST8 than to MNIST2. In this case, a bigger subset of the instances is included.
Regarding the similarity between NNs, we only work with MLPs but believe that our characterization can compare very different architectures (which would be more complicated with distances based on the adjacency matrices).
The goal of this paper is to show that our NN characterization using PH enables an effective comparison between e.g. different NNs that have been trained to solve the same problem.
> This should be state more precisely/formally in my opinion. In that context, why do some experiments focus on changing the network architecture for a given dataset (Figure 4, Figure 5)?
We will clarify this in the revised version of the article. We modify the architecture for a given dataset to observe which changes this implies to our characterization. What we wanted to show (and actually believe that we have shown) is that different architectures trained for the same problem are closer between them than similar architectures trained for different problems.
> These figures are hard to interpret: there seems to be a block structure but also variation inside diagonal blocks. Furethermore, this goal is not met according to the claim our characterization is sensitive to the archutecture (line 284).
We do our claim according to the following observation.
When we move across similar problems (e.g. decrease of the number of considered classes in the same dataset in MNIST), we see a LESS noticeable change in distances than when we do so with BIG changes in architecture configurations. Our proposed distance is more sensitive to the number of neurons per layer or number of layers than removing a few classes from the dataset. If we adodd to this 1) the observed invariance in the control experiments (change in label order and initialization) and 2) the gradual increase in the distance when we change the number of neurons per layer and number of layers, we believe that the distance behaves as we expected and in a useful way.
> As said, I feel that Figure 3 is interesting (it stresses the dependence on the dataset in a convincing way) but the paper does not serve it properly in my opinion.
This is indeed the main point of the paper, showing that our distance can detect the task for which a NN is trained.
> Furthermore, this goal seems somewhat surprising; say that I train a really simple network (e.g. a linear regression) on a complex dataset (e.g. CIFAR 10). Why/should this network be considered similar to a sophisticated one (e.g. a wideResNet?)?
If they both solve the same problem in the same way (not implying they do), they should indeed be similar. Consider 2 Python functions with identical signatures (parameters, types, and even doc strings specifying the semantics), an equivalent functionality (same outputs for same inputs) but a different implementation, being both interchangeable. Shouldn't they be considered semantically similar? We see it in a similar way for neural networks.
However, regarding your specific example, we don't use networks as simple as linear regressions or as sophisticated as wideresnet.
But in our experiments, NNs with very different accuracies are NOT close as per our proposed distance.
So if the logistic regression performed badly, it would appear to have a high distance with an MLP with good accuracy.
Furthermore, changes in architectures *can* be appreciated with our distance up to a point.
However, we insist that for this work we have restricted ourselves MLPs for the reasons stated in the paper.
>  I guess that one implicitly considers "good" (that achieve good test accuracy?) networks for a given task?
This is another finding in the paper.
When the NN is not good enough for the problem, then distances start increasing.
> I agree, but as pointed in my review, the "naive equivalent" would lead to an extremely large graph/flag complex if I am correct (due to the typically large number of filters involved).
Indeed, the naive equivalent produces big MLPs (https://aul12.me/machinelearning/2019/06/20/cnn-mlp-2.html) but they seem to admit big topological simplifications.
First of all, it is a very sparse simplicial complex.
Also, some of the typical layers in CNNs are topologically simple (e.g., pool layers).
> "Isn't it what is done in Eq. (1) as well?
No. We use a directed flag complex and the axis direction is important.
Also, we make the weights that are more influential (i.e., higher absolute value) to have values that are closer to 0, and we make the weights that are less influential to have values that are closer to 1.
> Furthermore, my comment was about working "layer-per-layer" as done by Rieck et al., reducing computational burden (see below). I noticed your answer to reviewer WKaR: one may miss global information in the network, it may be true, but the computational benefit may overcome the information loss (or not, I don't know, that is just a suggestion).
Regarding Rieck et al., we have made this document available with a summary of our opinion and some ASCII graphs: https://pastebin.com/uWqtC6GE
In general, we think that in order to make a complete homology representation of a NN we should use higher order homology groups (we have found important groups up to degree 3 but there could be cases where the complexity is higher). This is necessary to find non-trivial higher order homology structures that are specific and characterise the NN. We also see it as very important to employ directed graphs, and therefore to use directed flag homology.
> Superlevel set filtrations are obtained by filtering graphs in decreasing order (usual persistent homology consider sublevel sets, i.e., filtering with increasing filtration values): one starts by adding vertices/edges with highest filtration values and then decrease the threshold progressively. This is implicitly what is done in Eq. (1) when considering $1 - |w|/\max(|W|)$: edges with higher weight $w$ are added first.
Indeed, the notation can be simpler but the TDA community is used to using increasing filtration values. If you believe it's necessary, we would be happy to change the notation.
> I agree with the latter point.
But the former stresses the lack of scalability of the proposed approach: the networks considered in this work, though non trivial, are fairly small compared to "state-of-the-art" networks yet they lead to very large diagrams which (probably worse in terms of memory footprint) require first to store extremely large simplicial complex.
There are 2 sources of computational complexity:
- From the graph representation of the NN. We have done some follow-up experiments with Transformers (Roberta Large, 355M) and the computation was feasible (although we focused on the attention layers, omitting the FFNs and residuals). See this paper from 2019 on PH computation: https://arxiv.org/pdf/1906.10458.pdf.
- From the diagram comparison that implies the optimal transport between segments composing the diagram to compare. This one can be solved with vectorization or even with the Flagser approximation (in extreme cases).
Thank you for taking time to answer my comments. This is instructive.
I think the following point is worth discussion.
> Consider 2 Python functions with identical signatures (parameters, types, and even doc strings specifying the semantics), an equivalent functionality (same outputs for same inputs) but a different implementation, being both interchangeable. Shouldn't they be considered semantically similar?
I get the overall idea, but I do not understand why the proposed approach should quantify this.
For instance, let $F, G : \mathbb{R}^d \to \mathbb{R}^k$ be the functions encoded by two trained NN $A,B$;
that is $A,B$ contain the information about the architecture of the network etc., while $F,G$ are just abstract analytic maps. Do you claim (roughly speaking) that $\mathrm{Sim}(A,B) \text{ high } \Rightarrow \mathrm{dist}(F, G) \text{ low }$ (or perhaps the other way around?), where $\mathrm{Sim}$ is the notion of similarity you developed in your paper and $\mathrm{dist}$ a distance between maps---I have no idea in which one, as you may want to compare $F,G$ defined on different spaces (e.g.
when you reduce the number of classes $k$ in the dataset))?
I think this is more or less the idea of the paper but (a) this should be formalized (just to fix the idea;
of course I do not pretend that one can prove such an inequality). (b) Explain why the proposed approach has some chance of solving such a problem. Very roughly speaking, if I take your example of equivalent Python functions, I feel like something similar to your method would be "counting the number of lines in the function and the number of character per line" ; and pretending that it allows one to say if the functions return the same outputs for same inputs would be nonsense.
Of course, I am not saying that your method just count the number of layers and their width! What I mean is that you have a method which depends on the network architecture and I have no clue on why this should allow us to compare the underlying maps $F,G$ (or, in which sense it does compare those maps).
Let us fix some context, assume that two sequential, fully-connected, networks encode the same map (i.e., $F=G$). Could you prove that $\mathrm{Sim}(A,B)=0$? If so, this would be a **huge** addition to the work. I don't know if (and I doubt) this is true, but some discussion (and some formalism) around such questions are, in my opinion, necessary in this work to understand both its goals and its achievements.
Some other shorter comments:
> they seem to admit big topological simplifications.
First of all, it is a very sparse simplicial complex
Agreed, but there is some work to do there.
> Regarding Rieck et al., we have made this document
interesting document, I think some points are worth being discussed (you do not need to precisely compare with Rieck et al., as you are doing different things, but perhaps simply support the validity of your approach saying "our model can handle this/that"). I noticed you say : "it is necessary to work with $1 - w / \max W$ and not $w / \max W$ as done by Rieck et al.
Unless I am missing something, Rieck et al., use _superlevel set_ filtrations, that is, adding edges with higher weights first, which is exactly similar to using $1 - w / \max W$ as done in your work. We all agree that it seems reasonable that edges with higher (absolute) weight should be added first, going for sublevel set filtration on $1 - w / \max W$ (your) or superlevel set on $w / \max W$ (Rieck et al.) is just a change of perspective.
I also agree that "layer-per-layer" approaches (Rieck et al.) can't handle non-planar networks natively, but it is pretty clear the authors were aware of this and designed this technique for efficiency reasons (among other things (*) ). Saying that you can handle non-planner networks by considering the network as a whole, without showcasing it (and, importantly, showcasing that it provides useful additional information), is not valuable as is.
(*) Another interesting aspect of Rieck et al. approach is that you can "normalize" the quantity computed with respect to both the number of layers and the layer width (terms like dividing by $|V_k \times V_{k+1}|$ in their paper).
So that deeper/larger networks do not have "artificially" higher neural persistence (note: your claim that one could add identity layers to disturb their approach is an interesting and valid point!).
> the TDA community is used to using increasing filtration values. If you believe it's necessary, we would be happy to change the notation.
No specific need here, that was just a suggestion to simplify your notations. I am pretty sure that TDA community is also used to decreasing filtration values (or can adapt straightforwardly). It is up to you!
Overall, I think the idea of the paper is praiseworthy but the work seems in a too preliminary state to assess its full potential as many aspects remains unclear or unexplored to me.
Again, thanks for your comments. This discussion is indeed interesting.
> I get the overall idea, but I do not understand why the proposed approach should quantify this.
For instance, let $F, G : \mathbb{R}^d \to \mathbb{R}^k$ be the functions encoded by two trained NN $A,B$;
that is $A,B$ contain the information about the architecture of the network etc., while $F,G$ are just abstract analytic maps. Do you claim (roughly speaking) that $\mathrm{Sim}(A,B) \text{ high } \Rightarrow \mathrm{dist}(F, G) \text{ low }$ (or perhaps the other way around?), where $\mathrm{Sim}$ is the notion of similarity you developed in your paper and $\mathrm{dist}$ a distance between maps---I have no idea in which one, as you may want to compare $F,G$ defined on different spaces (e.g.
when you reduce the number of classes $k$ in the dataset))?
In the paper, we use 2 concepts of distance/similarity that we want to relate to each other: i) The similarity between problems (or tasks/datasets), ii) The distance between NNs.
For the first one, we don't have a precise mathematical definition. In this case, we only have the input data and the corresponding outputs, for any problem, and we would like to compare between different problems. Historically, there have been some attempts at defining the general concept of information. Shannon, Kolmogorov, Fisher, etc. In our case, we don't have established a numerical similarity measure between the problems we have studied, but we have established a hierarchy: sim(MNIST-10, MNIST-8) > sim(MNIST-10, MNIST-6,4,2) o sim(MNIST-X, MNIST-Y) > sim(MNIST-X, LangID). By MNIST-K we mean MNIST restricted to K classes.
For the second one, we do have established a well-defined mathematical representation and an exact similarity function. In the experiments, we have used an approximation (which, as argued in one of the cited works, is stable).
> I think this is more or less the idea of the paper but (a) this should be formalized (just to fix the idea;
of course I do not pretend that one can prove such an inequality). (b) Explain why the proposed approach has some chance of solving such a problem. Very roughly speaking, if I take your example of equivalent Python functions, I feel like something similar to your method would be "counting the number of lines in the function and the number of character per line" ; and pretending that it allows one to say if the functions return the same outputs for same inputs would be nonsense.
We can give more details about this point but we have preferred not to establish a numerical distance between problems. For our comparison, we have believed our hierarchy to be enough. We agree that we should make it explicit and give more details on its foundations.
Perhaps our previous Python metaphor doesn't clarify the difference between these 2 concepts of similarity: between problems themselves posed to NNs and between trained NNs.
> Of course, I am not saying that your method just count the number of layers and their width! What I mean is that you have a method which depends on the network architecture and I have no clue on why this should allow us to compare the underlying maps $F,G$ (or, in which sense it does compare those maps).
To us, this is the central element of our work: we have seen that it is possible to compare NNs using this representation and similarity, and we observe relevant properpties:
1 - Two NNs with the only difference being different initialization and different sample order are very similar.
2 - Some theoretical modifications in a NN that don't modify its outputs don't modify this distance.
3 - When problems are different, distance is increased.
4 - When the NN doesn't learn properly, the distance increases wrt the NN that does learn properly.
We don't have a theoretical model for explaining this but we believe it to be relevant to show these results to the ML community, with the hope that future works will keep iterating and refining our findings.
> Let us fix some context, assume that two sequential, fully-connected, networks encode the same map (i.e., $F=G$). Could you prove that $\mathrm{Sim}(A,B)=0$? If so, this would be a **huge** addition to the work. I don't know if (and I doubt) this is true, but some discussion (and some formalism) around such questions are, in my opinion, necessary in this work to understand both its goals and its achievements.
Yes, this is what we have done with the control experiments.
When we modify the initialization and sample order, the distances between NNs is very small (as expected).
> Interesting document, I think some points are worth being discussed (you do not need to precisely compare with Rieck et al., as you are doing different things, but perhaps simply support the validity of your approach saying "our model can handle this/that"). I noticed you say : "it is necessary to work with $1 - w / \max W$ and not $w / \max W$ as done by Rieck et al.
Unless I am missing something, Rieck et al., use _superlevel set_ filtrations, that is, adding edges with higher weights first, which is exactly similar to using $1 - w / \max W$ as done in your work. We all agree that it seems reasonable that edges with higher (absolute) weight should be added first, going for sublevel set filtration on $1 - w / \max W$ (your) or superlevel set on $w / \max W$ (Rieck et al.) is just a change of perspective.
Agreed. This simplification in the notation is the same one another reviewer has suggested. It is equivalent to the normalization we do to the supervel filtration of Rieck et al.
> Agreed, but there is some work to do there.
We will include more details in the revised version.
> I also agree that "layer-per-layer" approaches (Rieck et al.) can't handle non-planar networks natively, but it is pretty clear the authors were aware of this and designed this technique for efficiency reasons (among other things (*) ). Saying that you can handle non-planner networks by considering the network as a whole, without showcasing it (and, importantly, showcasing that it provides useful additional information), is not valuable as is.
We believe the model also has to be invariant to superfluous cycle insertions and identity layer insertions, apart from supporting "non-flat" NNs. It's not about performance, but of representation capacity. We don't believe Rieck model to be independent of even initialization randomization or sample order.
> (*) Another interesting aspect of Rieck et al. approach is that you can "normalize" the quantity computed with respect to both the number of layers and the layer width (terms like dividing by $|V_k \times V_{k+1}|$ in their paper).
So that deeper/larger networks do not have "artificially" higher neural persistence (note: your claim that one could add identity layers to disturb their approach is an interesting and valid point!).
In our case, since we use homology groups > 0, this normalization is not that natural. We analyze the homological complexity (until the required degree, not only connectivity as in Rieck) associated with the NN. As we have said, Rieck has other problems: superfluous cycle insertions, independence of sample order, non-flat NNs...
> No specific need here, that was just a suggestion to simplify your notations. I am pretty sure that TDA community is also used to decreasing filtration values (or can adapt straightforwardly). It is up to you!
Thanks for the comment.
Another reviewer has made a similar comment so we will consider it.
I have the following comments:
---It would be good to also mention the topological uncertainty (https://arxiv.org/abs/2105.04404), which is similar to neural persistence, in the related work section.
---How are neuron filtration values computed? Is it the minimum of all filtration values of edges that contain this node?
---As far as I understand, neuron activations are not used in the persistence diagram constructions. This could be added, for instance as potential filtration values for the nodes.
---It seems to me that the relationships between persistence diagrams and network architectures is actually quite intuitive: the more layers or neurons, the bigger the simplicial complexes, and thus the more points you will have in the persistence diagrams. That could explain the distance matrices presented in the article, without any real interpretation or relation with the capacity or generalization error of the architectures.
---I am not sure about how this similarity measure between neural networks can be used in practice. Is there any rule that could be used in order to do model selection, i.e., choose a specific architecture to use for a given data set or data science task, such as classification, regression, transfer learning etc?
Even though I like the proposed idea (and especially the comparison to Frobenius norms, which clearly shows its limitation and the advantages of using persistence), I feel like this work is still in an early shape, and would really benefit from a better interpretation of the distances (in terms of how persistence diagrams are computed), as well as more examples of practical uses.
No
Thanks for your kind feedback and insightful comments. Answering your points:
- Thank you for the reference. We were not aware of this work being that recent. The orientation of our paper is different. We are not interested in studying activities because it depends on input data. We are interested in studying trained NNs. Our goal is to compare NNs. The representation proposed in the Topological Uncertainty paper is similar to the work from Corneau et al that we already cited.
- Please, see 173-184. We are not imposing filtration values on the neurons (graph nodes), we only set them on the edges. We think it is an interesting proposal. We could employ such a strategy. Flagser allows you to control the filtering of nodes as well as edges.
- We are interested in characterizing and comparing NNs as functions, independently of the data to which they are applied. In the case of activation analysis, activations are associated with input data.
We have questions regarding the use of activations: a) How much data do you pass through the network? b) It is not the same to pass {training, validation, testing, production} data. Isn't it? c) The subset of data selected may not be representative. f) Information flow of the network (weights) might not be taken into account completely.
- See Figure 3, very similar architectures trained for the different problems are not similar and in Figure 4 (a, b) from 15 to 19.  Adding a few labels makes a LARGE topological difference.
Homology groups do not have a cardinality associated with the number of nodes and edges of the source graph.
For example, the homology groups associated with any triangularization of a torus are the same regardless of the number of triangles used. We want to use homology as a tool precisely because of this ability to abstract from the concrete realization of the metric object (the weighted graph associated with the NN). "
- Our research is empirical and aims to start a research line on that. There are potential applications, for instance for analyzing neural networks (e.g., learning dynamics).
- As we said before this is a baseline paper for NN comparison. This is the scope of the paper.
Although practicality is important, first a solid baseline should be established and this is our aim.
Thanks for the clarification.
Concerning second point: I was asking because there are multiple ways to filter a simplicial complex if only edge filtration values are provided. Which one do the authors use in practice? Do the authors make all the nodes appear in the filtration at the same time and then the edges? Or do the nodes also appear gradually in the filtration?
Concerning fourth point: sure 15-19 definitely shows the influence of topology without changing the complexes. I also agree that homology groups are NOT directly related to the size of the complex in their definition. My comment was more about the fact that if you assign random values to the simplices of a complex, the size of the resulting persistence diagram will be comparable to that of the complex.
So I was wondering if this was the kind of phenomenon that is observed in 1-4 and 5-9.
> Concerning second point: I was asking because there are multiple ways to filter a simplicial complex if only edge filtration values are provided. Which one do the authors use in practice? Do the authors make all the nodes appear in the filtration at the same time and then the edges? Or do the nodes also appear gradually in the filtration?
In this paper, all the nodes appear in the filtration at the same time, when filtration starts, and then the edges appear depending on normalized edge weight.
On the other hand, we think it would be a good idea to experiment with the gradual appearance of the nodes when they are the boundary of edges that appear on the filtration. We believe that Flagser, the software that uses the Giotto-TDA library that we have used, has support for this functionality.
> Concerning fourth point: sure 15-19 definitely shows the influence of topology without changing the complexes. I also agree that homology groups are NOT directly related to the size of the complex in their definition. My comment was more about the fact that if you assign random values to the simplices of a complex, the size of the resulting persistence diagram will be comparable to that of the complex.
So I was wondering if this was the kind of phenomenon that is observed in 1-4 and 5-9.
The topological changes seem more pronounced when modifying the tasks/problems (dataset), less pronounced when modifying the NN configuration, and negligible when modifying the initialization and order of training examples. We find your observations very interesting and look forward to analyzing it.
Thanks for the clarifications.
The topological similarity of neural networks is considered by computing the persistence features for the generated weighted graphs and complexes. It is an interesting idea. The experiments can show the persistence distances (persistence landscape, weighted silhouette and heat vectorizations) for the different architectures of neural networks.
* The proposed method and experiments consider the depth and width of the NNs, and how the neurons are linked, which are important factors for their performance. Curious how the computational blocks are linked can be reflected in the persistence metric for the similarity of the NNs.
* It is not clear whether the similarity reflected by the persistence homology has anything to do with the performance of the NNs for classification tasks.
* Maybe some pictures of the graphs and complexes for a specific CNN network can be shown.
* Line 170-171, there is inconsistent for $\epsilon$ and $\epsilon_t$.
* Line 265, remove `show' in ', we obtained show small distances,'
No
Thanks for your insightful feedback and comments.
Also, thanks for the typos corrections.
Regarding your main review and the potential limitations:
- To the best of our knowledge, comparisons of NNs of similar architectures trained on the same problem can be found in the literature (generally, with the same initialization). In our case, we are looking for a representation and comparison method that does not have these limitations (same problem, same initialization). It is therefore difficult to find a baseline.
- The article focuses on MLPs. Our method is also applicable to CNNs, since they have an MLP equivalent, and also to Transformers. We are planning to further investigate the applicability of our method to these other architectures.
However, we acknowledge that the proposed simplicial representation model does not seem to be directly applicable to RNNs given the existence of direct multiplications of the inputs (LTSM and GRU model).
Finally, we also note that we experimented with a static CNN layer previous to the MLP.
Thanks.
I would prefer to see the comparison with other baselines even with slight different initialisation.
Again, thanks for your comment.
We have experimented with different initializations and sample order, as a control experiment, and the resulting NN similarity is close (meaning that those changes don't have much effect).
Regarding the baselines, we haven't found a way to compare networks with different architectures. For networks with the same architecture, we have executed 2 baselines (Frobenius-norm, 1-norm) and the results have not been positive. We believe we have made our best effort to regarding baselines, and that there is no feasible alternative.

Meta-review:
role: PC
data: comment: The paper proposes a similarity measure for neural networks, which is an interesting topic.  While the proposed measure based on PH is of some significance, a systematic understanding of the resulting distances is still premature, and it is not clear how to use the measure in practice.  From these evaluations, we need to judge that this paper is unfortunately below the acceptance threshold.  

Decision: Reject