Persistent Similarity in Internal Representations of Large Language Models

Authors: Yuri Gardinazzi, Giada Panerai, Karthik Viswanathan, Alessio ansuini, Alberto Cazzaniga, Matteo Biagetti
Primary area: learning on graphs and other geometries & topologies
Keywords: Topological Data Analysis, Persistent Homology, Large Language Models, Internal Representations, Similarity, Layer Pruning

Abstract:
Understanding the decision-making processes of large language models (LLMs) is critical given their widespread applications. Towards this goal, describing the topological and geometrical properties of internal representations has recently provided valuable insights. For a more comprehensive characterization of these inherently complex spaces, we present a novel framework based on zigzag persistence, a method in topological data analysis (TDA) well-suited for describing data undergoing dynamic transformations across layers. Within this framework, we introduce persistence similarity, a new topological descriptor that quantifies the persistence and transformation of topological features such as $p$-cycles throughout the model layers. Unlike traditional similarity measures, our approach captures the entire evolutionary trajectory of these features, providing deeper insights into the internal workings of LLMs. As a practical application, we leverage persistence similarity to identify and prune layers, demonstrating comparable performance to state-of-the-art methods across several benchmark datasets. Additionally, our analysis reveals similar topological behaviors across various models and hyperparameter settings, suggesting a universal structure in LLM internal representations.

Decision: reject
Reviewer scores: 3, 6, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730395285272

thread_type:
official_review

<thread_object>
creator: reviewer_tvbg

<content>
summary:
The paper proposes a new framework based on zigzag persistence to analyze internal
representations in LLMs by characterizing the birth and death of topological features within the
model’s layers.   it provides a fine-grained geometric analysis of the internal representations through TDA.  Distinct from traditional methods that solely compare representations at individual layers, the proposed method  captures their entire evolutionary path, providing a richer understanding of how these features evolve and contribute to the model’s decision-making processes

strengths:
Originality:
The idea of applying topological data analysis to LLM  is novel and interesting
Clarity:
The paper is well-written and easy to follow.
Significance:
The proposed Persistence Similarity and zigzag algorithm have rigorous mathematical formulation and are easy to compute. The experiment in Section 4.3 is promising. Author compares the performance of layer pruning Persistence Similarity and other common llm layer pruning methods and the proposed method performs well in most cases under 10% pruning rate.

weaknesses:
Major concerns:
1. The discussion in Section 4 did not clearly explain the sensitivity of the proposed method when pruning LLM layers.  The result shown in Fig 5 indicates that, across different models, our metric consistently identifies the deep layers as “redundant”. Additionally, in the results from Table 1, the proposed metric does not show an advantage at a 20% sparsity level, which further heightens my concerns. 

2. The authors mention, "Due to the autoregressive nature of these models, the representation of the last token in a sequence captures information about the entire sequence and is the only token used for predicting the next." I disagree with this assumption; predicting the next token is clearly based on all preceding tokens. Therefore, I would like to see an analysis of certain special tokens, such as the EOS and BOS tokens.

3.  Can Persistence Similarity and the zigzag algorithm help explain the varied responses of LLMs across different abilities or areas of knowledge? The pruning experiments presented in the paper are limited to base models. Could you provide an analysis of the instruction-following abilities of chat models or their capabilities in math and code?

questions:
Overall, I believe this paper is likely to present a novel perspective on understanding LLMs. 
However, the experimental section is overly limited. I look forward to discussing these questions further in the weaknesses section. Addressing them may help improve the overall score.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730436692145

thread_type:
official_review

<thread_object>
creator: reviewer_99nq

<content>
summary:
This paper proposes a novel TDA-based method for measuring LLM similarity.  Zigzag-persistence is introduced to take account of the properties of the LLM graph structure. As part of the evaluation, it has been applied to Pruning to check its effectiveness.

strengths:
- This paper introduces a new idea to introduce Zigzag-persistentce by considering the layer structure.
- It has been applied to pruining to check its effectiveness.

weaknesses:
- Methods that regard Neural Networks as graphs and apply TDA have been proposed in [1], for example. This is the usual persistent homology, but the use of zigzag persistence for graph structures has been verified in [2],[3] and elsewhere. The structure focusing on the LAYER structure has some aspects of novelty, but it seems to be a simple conventional combination idea and the degree of novelty is weak.

[1] T. Lacombe et al., Topological Uncertainty: Monitoring trained neural networks through persistence of activation graphs, IJCAI2021

[2] A. Myerset al., Temporal network analysis using zigzag persistence. EPJ Data Science, 12(1):6, 2023.

[3] Y. Chen et al., Time-Conditioned Dances with Simplicial Complexes: Zigzag Filtration Curve based Supra-Hodge Convolution Networks for Time-series Forecasting NeuRIPS2022

- It is difficult to see what the similarity of the models is intended to achieve. The proposed method is similarity of the graph structure and may be similar for different models. The paper evaluates the similarity in application to pruning to demonstrate its validity. However, it is difficult to accept from the experimental results alone that the proposed method is a superior similarity simply because the implications that the similarity indicates are not clear. As a method for measuring the degree of change, such as model modifications, such as pruning, it is intuitively effective. However, as no uses have been indicated, the current situation is that there is only evidence for it as a pruning method. If the claim is that it is a superior pruning method, it should be clearly stated as such. In addition, if the pruning method is slaughtered and claimed, it should be compared with a generally sufficient advanced puruning method, for example [4],[5].

[4] S. Ashkboos et al., SliceGPT: Compress Large Language Models by Deleting Rows and Columns, ICLR2024

[5] X. Ma et al.,LLM-Pruner: On the Structural Pruning of Large Language Models, NeuRIPS2023

questions:
- Clarify the claim point(s) for which the novelty of the contribution in this paper is sufficient.
- Please clearly indicate in what sense the similarity of the proposal is valid, what use scenarios are covered (other than pruning) and whether it is sufficient evidence for this.


Why are the titles on the system different from the titles in the PDF?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655149724

thread_type:
official_review

<thread_object>
creator: reviewer_reus

<content>
summary:
The paper uses zigzag persistence to empirically study LLMs.

strengths:
Section 3 rigorously introduced the zigzag persistence.

weaknesses:
The major problem of this paper is the lack of a problem. In fact, the word "problem" was not found in the whole paper at all. 

Lines 520-521: "Our approach aims to provide a high-level geometrical and topological description of positional and relational changes across layers".

If this phrase in the conclusions should be considered a problem statement, then almost any empirical study using geometry and topology can fit this description. 

Section 3 "Method" describes over 3 pages the known facts about zigzag persistence without even mentioning LLMs from the title of the paper.

The world "layer" appears in the paper 50+ times without a proper definition. Initially, this word is used in the context of neural networks but section 3 seems to assume that a layer is a cloud of unordered points in R^d, which is a standard input for computing persistence in TDA.

The "persistence similarity" introduced in (5) raises many concerns. First, the concept of "persistent similarity" appeared in the literature, e.g. https://www.intlpress.com/site/pub/pages/journals/items/cis/content/vols/0018/0004/a004/, but no past work on such similarity was cited.

More importantly, the denominator in (5) can vanish, which makes the persistence similarity S_p undefined. In this definition, it is still unclear if l_1,l_2 denote layers or point clouds because min/max of l_1,l_2 seem to be real numbers. If a layer is indeed a point cloud, the first metric axiom surely fails for l_1 and its translated image l_2, which have the same persistence.

Even if we consider point clouds under isometry, because persistence is an isometry invariant of a point cloud for standard filtrations of complexes, the first metric axiom also fails for infinitely many non-isometric point clouds, see J Applied Comp Topology 2024. Computational geometry has known much stronger isometry invariants of unordered point clouds for more than 20 years, see Boutin and Kemper, 2004.   
 
Lines 402-403: "Note that the plot is not symmetric by definition (cfr. equation 5)". 

Does it mean that the symmetry axiom fails? 

The triangle axiom also seems unlikely to hold, so a proof is needed. If a distance fails the triangle axiom with any positive error, then results of clustering are not trustworthy as proved in https://ieeexplore.ieee.org/abstract/document/10574843? 

The form on "soundness" asked to evaluate "the technical claims, experimental and research methodology and on whether the central claims of the paper are adequately supported with evidence".

No technical claims were found, no words "claim", "proposition", "theorem" in the paper. Here are the comments on informal claims about contributions.

Lines 74-76: "We propose a new metric to measure which topological features persist across the layers of an LLM."

If persistence similarity is called a metric, proofs of metric axioms are expected. The original persistence already measures "which topological features persist".

Lines 77-79: "By identifying layers with high persistence similarity, we prune redundant layers without significantly degrading performance".

The adjective "redundant" appears in the paper without explanation. If performance refers to percentages in Table 1, they degrade by 10%. Is it not significant?

Lines 80-82: "Our findings indicate that the behavior of persistent topological features and their similarities are consistent across different models, layers, and choices of hyperparameters of the framework".

To make this claim meaningful, the words "consistent" and "consistency" should be properly defined.

questions:
Lines 523-524: "This approach allows for effective model pruning by identifying and removing redundant layers without significantly compromising performance" 

All accuracies in Table 1 are between about 40% and 70%. Are these numbers enough to guarantee safety in real applications?    

How are outputs of LLMs converted into point clouds in the experiments? What is the meaning of persistence in terms of word tokens or other language-related concepts?

Should the review mention the previous work explaining LLMs as stochastic parrots at https://dl.acm.org/doi/abs/10.1145/3442188.3445922 and in stronger terms at https://link.springer.com/article/10.1007/s10676-024-09775-5?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730655261720

thread_type:
official_review

<thread_object>
creator: reviewer_n98n

<content>
summary:
This paper uses zigzag persistence, an approach from TDA, to analyze the hidden representations of LLMs. With these tools, they track the evolution of topological properties of LLM hidden representations across layers. They make numerical observations such as increased persistence of topological features formed in later layers, and high similarity between representations in later layers. As an application, they prune layers based on persistence similarity.

strengths:
1. Generally good and clear explanation of the requisite background and method.
2. Experiments show some insights into several open-weight language models of interest, across datasets that people care about (both for computing the metrics, and for testing downstream accuracy after pruning).

weaknesses:
1. Besides the layer pruning application, the qualitative and quantitative insights from these topological features are not particularly interpretable or grounded. We can measure how the number of $k$-cycles evolves, but there are no interpretable quantities that come out of this, besides similarities between layers. To alleviate this, perhaps further discussion of motivation and related work (why should people use TDA tools to analyze internal representations), or interpretability work (what kind of 1-cycles form, and on what kind of data) could be interesting.
2. Utility in one downstream application (layer pruning) is questionable (the simpler methods from prior work do solidly and are not consistently outperformed).
3. The use of zigzag persistence in layer pruning only depends on the similarity matrix formed by the method. Other simple baselines could be considered for making similarity matrices.

questions:
1. Have you looked at how topological features vary across different data samples / domains? For instance, are there any interesting data features that form between representations of tokens from certain programming languages?
2. Could you provide a bit more information on where the embeddings are extracted from (this is currently described in Section 4.1 as "each prompt is processed so that the last token is extracted at each normalization layer and the final normalization applied to the output layer.") Do you take the embedding before or after the normalization layer? Also, does this mean you take two embeddings from each block (one from the attention normalization, one from the MLP normalization)? Also, some discussion on how this compares to how other works extract LLM hidden representations  would be nice.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730685010505

thread_type:
official_review

<thread_object>
creator: reviewer_bmgz

<content>
summary:
This paper proposes a novel topological data analysis method based on zigzag persistence, which describes data undergoing dynamic transformations across layers within the large language models. The proposed persistence similarity is used to quantify the persistence and transformation of topological features throughout the model layers and provide insights to prune redundant architecture modules without significantly degrading performance. Experimental results indicate that the behavior of persistent topological features and their similarities are consistent across different models, layers, and choices of hyperparameters of the framework, suggesting a certain degree of universality in the topological structure of LLM representations.

strengths:
- The paper is clearly written, making complex concepts accessible to the reader.
- The application of zigzag filtration to the internal representation of LLM is novel and insightful. 
- The approach to zigzag persistence is firmly rooted in theoretical foundations, providing a robust framework that enhances its applicability and relevance in the field.

weaknesses:
- The zigzag filtration, and persistence images are previously well-defined and existing. This raises concerns about the originality and depth of the theoretical and technical contributions presented in the paper.
- The performance gains over existing pruning methods, as shown in Table 1, appear marginal. Including a significance test would enhance the robustness of these findings.
- While the paper provides a geometric interpretation of the topological features, it is not clear how these features can be directly interpreted in the context of language models. The paper could benefit from a more in-depth discussion on the interpretability and implications of the observed topological properties.
- The zigzag algorithm is computationally expensive, especially for large datasets and high-dimensional representations. The paper should discuss strategies to optimize the algorithm's performance and make it more scalable.

questions:
- Figure 5 is somewhat unclear. Are the 10% (orange) segments included within the 20% (yellow) segments in most cases? The authors might consider a more effective visualization method to enhance clarity.
- Since other similarity measures can also characterize layer-wise behavior, what specific advantages does the proposed persistence similarity offer over these existing methods?
- What is the computational complexity of the proposed persistence similarity in comparison to other existing methods? 
- Can the authors provide insights or a high-level intuition regarding the consistent patterns observed in Figure 4?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732616093850

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response to all reviewers

comment:
# General response
We thank the reviewers for carefully reading our draft, recognizing its strengths and providing feedback on its
weaknesses. These suggestions have helped us to greatly improve our work. Here we provide a general response to
the main points raised and reply more in detail in each thread. 

**Strengths**.

The reviewers recognized the novelty and potential impact of applying topological data analysis
(TDA), specifically zigzag persistence, to the study of internal representations in large language models (LLMs).
We appreciate the acknowledgment of our work for bringing a fresh perspective to model analysis and geometric
interpretation through robust theoretical foundations. A strong point raised by more than one review has been the
clarity and presentation of our paper, making complex concepts accessible.

**Weaknesses**

Some of the reviewers consistently highlighted a few key areas for improvement:

- Novelty of zigzag approach, being already developed and employed in other applications.
- The work does not concretely link the topological features and observables to interpretable insights within
the LLM context.
- Layer pruning using persistent similarity does not seem superior to other methods, and its practical utility
has not been investigated.
- Overall, further experiments or applications of the framework would have strengthened the work.

In summary, our work is criticized for not developing/being novel enough either of the two subfields it's talking about, i.e. TDA and layer pruning. This is a fair concern, so we address it in the following response and make modifications to the draft accordingly. To preserve the integrity of the original draft, we have momentarily included results of new experiments in the appendix and not in the main body.

## Why ZigZag on LLMs?
On a high level, using topology to study how representations change in neural networks has been recognized
for a while (e.g. [1]) as a promising approach. As point clouds embedded in a
representation space, most of the existing work has considered TDA as a promising avenue for making progress
in this direction. In the context of internal representations of neural network, there is a sequence (layers) of these point clouds, and previous works analysing these sequences using TDA have tried to gather insights on how these point clouds change with respect to to different models, different tasks, and within the sequence. However, to the best of our knowledge, none of the previous works have built a TDA framework
that models the sequence of point clouds as a unique evolving system, rather than analyzing each step of the sequence separately and comparing results. This means that we can follow topological features across layers tracking the trajectory of these features through each layer. This implies that, for instance, swapping two layers is not a symmetry of our framework, i.e. our descriptors change, differently than previous work.

Overall, while not rigorously defining a new theoretical TDA framework, our paper serves as a crucial conceptual brige to open a range of possibilities for an interpretable set of experiments to be done on internal representations. 

Moreover, on a practical and more technical level, our zig-zag algorithm introduces a few novel points which we believe are important in this context:

- **Choice of filtration**: we introduce a $k_{\rm NN}$ graph-based filtration, which is relatively novel in previous TDA literature [1], and never used in the context of ZigZag. This choice is motivated by the fact that the point clouds are largely transformed from one layer to another (e.g. space rotations, translations, stretching of distances). For this reason, conventional ways to set up a filtration (e.g. Vietoris Rips complexes) are not viable. In addition, we provide a discussion in the appendix on how to combine the $k_{\rm NN}$ graph with a Vietoris Rips complex, which is also novel in the context of zigzag filtrations.

- **Layers as Time snapshots**: while the theoretical framework for zigzag has been developed in depth, its true potential seems still underexploited, as applications of ZigZag are still relatively few. As cited in the paper and one of the reviewers, current applications are in very different contexts/uses. We believe the interpretation of transformers as a dynamical system which evolves points clouds in "time" as layers is novel in the context of TDA. 

- **Persistence Similarity**: persistence similarity is a new topological descriptor which we introduce for the first time. The layer pruning is just a showcase of its use, while we expect other potential applications. We show a couple of explorations in the new paper appendix D (and explained below).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission presents a new method for tracking the evolution of geometrical-topological features through the layers of a large language model. Specifically, using zigzag persistent homology (a method from the nascent field of computational topology), the manuscript develops *persistence similarity*, a score for assessing common topological features (here, to be understood as features like connected components, cycles, ...; the method is capable of calculating such features across multiple spatial scales) between layers. Based on this score, the manuscript outlines a new algorithm for the identification (and subsequent pruning) of (partially) redundant layers, exhibiting promising results in comparison to other methods.

Coming from a background of computational topology myself, I appreciate the two primary conceptual *strengths* of the paper, to wit:

1. The paper is well-written and aims to explain its methods, thus exhibiting a high degree of clarity
2. The use of TDA or TDA-like methods in this context is highly original, and the topic is highly relevant to the community as a whole

At the same time, I also have some concerns and observed several *weaknesses* of the work, namely:

1. A lack of experiments that are compelling to the larger ML community. I realise that this sounds very negative and I hope I can take away some of the sting of these words. My point is that this paper, which makes creative use of TDA techniques to address a compelling problem is interesting _per se_ for people with a background in TDA. However, the goal of ICLR (and of ML conferences in general) is to have works that are compelling by providing new insights for a larger community and ideally considerably advance the state of the art. The authors are honest about the shortcomings of their method—which I appreciate—and mention that they cannot show statistical significance of the results.
2. The derivation of interpretable/explainable results from the method, along with a better justification for the use of topological methods. Again, I realise that this sounds overly negative, so I want it to be clear that I share the enthusiasm about TDA techniques and zigzag persistence in particular, but I need to consider the broader ramifications of using the "topology hammer" in front of a broader ML audience. Given the computational but also the conceptual complexity of the method, it is unfortunately our "onus" to also ensure that we justify the choice of method. Here, potential justifications could be interpretability/explainability or robustness but none of these aspects is considered so far.

It is for these reasons that I have to suggest rejecting the paper, and I hope the authors believe me when I say that this decision was not reached ligthtly. On the contrary: I am biased towards having _more_ "TDA goodness" at ML conferences, but I believe that in its current form, this work is not quite ready yet. Moving forward, I would suggest that the authors invest in expanding the experimental section, both in depth and in breadth. With a few modifications (which, unfortunately, will amount to some additional experiments), this has the potential to be a strong, impactful contribution.

I also want to emphatically state that such work _is_ in-scope with ICLR and related conferences, as well as other A*-conferences with a focus on NLP, and I sincerely hope that the authors can benefit from the discussion phase.