Persistent Topological Features in Large Language Models
Authors: Yuri Gardinazzi
Karthik Viswanathan
Giada Panerai
Alessio ansuini
Alberto Cazzaniga
Matteo Biagetti
Venue: ICML 2025 poster (2025)
Keywords: Topological Data Analysis
Persistent Homology
Large Language Models
Internal Representations
Layer Pruning
Primary area: deep_learning->large_language_models
TL;DR: We introduce zigzag persistence for the interpretation of internal representations of large language models

Abstract: Understanding the decision-making processes of large language models is critical given their widespread applications. To achieve this, we aim to connect a formal mathematical framework—zigzag persistence from topological data analysis —with practical and easily applicable algorithms. Zigzag persistence is particularly effective for characterizing data as it dynamically transforms across model layers. Within this framework, we introduce topological descriptors that measure how topological features, 
$p$-dimensional holes, persist and evolve throughout the layers. Unlike methods that assess each layer individually and then aggregate the results, our approach directly tracks the full evolutionary path of these features. This offers a statistical perspective on how prompts are rearranged and their relative positions changed in the representation space, providing insights into the system’s operation as an integrated whole. To demonstrate the expressivity and applicability of our framework, we highlight how sensitive these descriptors are to different models and a variety of datasets. As a showcase application to a downstream task, we use zigzag persistence to establish a criterion for layer pruning, achieving results comparable to state-of-the-art methods while preserving the system-level perspective.

Persistent Topological Features in Large Language Models
Yuri Gardinazzi 12
Karthik Viswanathan 13 Giada Panerai
Alessio Ansuini!
Alberto Cazzaniga
Matteo Biagetti
Abstract
about interpretability and transparency (Liao & Vaughan, 2023). Moreover, their large scale demands considerable amount of computational resources (Samsi et al., 2023; Bai et al., 2024), making it essential to reduce their size without compromising performance (Ma et al., 2023; Gromov et al., 2024; Men et al., 2024).
Understanding the decision-making processes of large language models is critical given their widespread applications. To achieve this, we aim to connect a formal mathematical frame- work-zigzag persistence from topological data analysis -with practical and easily applicable algorithms. Zigzag persistence is particularly ef- fective for characterizing data as it dynamically transforms across model layers. Within this frame- work, we introduce topological descriptors that measure how topological features, p-dimensional holes, persist and evolve throughout the layers. Unlike methods that assess each layer individu- ally and then aggregate the results, our approach directly tracks the full evolutionary path of these features. This offers statistical perspective on how prompts are rearranged and their relative po- sitions changed in the representation space, pro- viding insights into the system's 's operation as an integrated whole. To demonstrate the expressivity and applicability of our framework, highlight how sensitive these descriptors are to different models and variety of datasets. As a show- case application to a downstream task, we use zigzag persistence to establish a criterion for layer pruning, achieving results comparable to state-of- the-art methods while preserving the system-level perspective.
One strategy for addressing these issues has been study the models' internal representations. Early works (Zeiler & Fergus, 2014) demonstrated that visualization techniques can effectively uncover hierarchical representations within convolutional neural networks, highlighting how lower lay- ers focus on edge detection while higher layers correspond to object parts and semantic concepts. Additionally, (Olah et al., 2018) illustrated that analyzing weight matrices and neuron activations can reveal interpretable features and or- ganizational structures within deep networks, providing in- sights into how complex patterns are encoded and processed.
More recently, geometric studies made progress by intro- ducing concepts like intrinsic dimension to characterize the manifold of internal representations and its evolution across layers (Ansuini et al., 2019; Doimo et al., 2020; Pope et al., 2021). These methods have been successfully applied to transformer models in various works (Valeriani et al., 2023; Tulchinskii et al., 2024; Cheng et al., 2023; 2024; Viswanathan et al., 2025). One notable achievement of this approach has been to show the emergence of semantic knowledge and abstraction phases in the middle layers of models, rather than at the final layers, as might be intuitively expected.
However, these approaches provide only static view of internal representations and suffer limitations in tracking their changes across layers.
1. Introduction
Large Language Models (LLMs) have revolutionized nat- ural language processing by achieving unprecedented per- formance levels across a wide range of tasks (see Raiaan et al. (2024) for review). Despite their success, the black- box nature of these models has raised significant concerns
A natural framework to address these limitations and to of- fer a more comprehensive characterization of the geometry of internal representations of neural networks is Topolog- ical Data Analysis (TDA). TDA is a set of unsupervised techniques that offers robust methods to describe the shape and structure of complex datasets. It has seen exponential growth with applications in computational biology (Man- dal et al., 2020), cosmology (Biagetti et al., 2021; Yip et al., 2024)]. personalized medicine (Skaf & Laubenbacher, 2022), time-dependent data analysis (EI- Ynagoubi al.,
'qqual contribution 'Area Science Park University of Trieste Unnieessity of Amsterdam. Correspondence to: Mattco Biagetti <mattco.biagetti areascicneepark.it..
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Persistent Topological Features in Large Language Models
2023), and machine learning (Hensel al., 2021), just to name few. One prominent tool within TDA is persis- tent homology, which tracks the birth and death of topo- logical features across different scales, thereby capturing the multiscale behavior of a point cloud. Several studies have proposed persistent homology to investigate neural net- works and their internal representations (e.g. (Rieck, Bastian Alexander et al., 2023), (Naitzat et al., 2020; Lacombe et al., 2021; Magai & Ayzenberg. 2022)).
2. Related Work
Topology of Internal Representations. TDA has been ex- tensively used in machine learning (see (Hensel et al., 2021) for recent review). In the context of studying internal representation, studies Convolutional Neural Networks (CNN) used topological descriptors to explore the shape of activation functions (Rathore et al.) or their relations to performance (Naitzat et al., 2020). (Magai & Ayzen- berg, 2022) introduced persistent homology dimension as an estimator of the intrinsic dimension of internal represen- tations CNNs, while (Barannikov al., 2022) proposed measure of similarity based on topological descriptors to compare representations. Betti numbers have been observed to remain stable across different datasets for the same archi- tectures and to decrease as depth increases (Suresh et al., 2023).
However, in the context of TDA applications, it has not yet been recognized that the internal representations of neural networks can essentially be viewed as point clouds dynam- ically evolving in time (layers). In the particular case of LLMss, as pre-trained models process inputs, they transform these point clouds within the representation space layer by layer, capturing essential features and relationships through- out the model's 's depth. Thus, it is natural to interpret these transformations as an evolving discrete dynamical system. To address this problem, we exploit a TDA tool developed to characterize time-varying point clouds and temporal net- works, known as zigzag persistence.
Zigzag Persistence. Zigzag persistence was introduced in (Carlsson & de Silva, 2010; Carlsson et al., 2009; Tausz & Carlsson, 2011) as extension of persistent homology to study the persistence of topological features across se- quences of spaces. This approach is particularly useful when data undergo dynamic changes or transformations over time. Since its introduction, zigzag persistence has been applicd in various fields, including Hopf bifurcations in dynamical sys- tems (Tymochko et al., 2020), commuting patterns in Great Britain's 's transportation network Myers et al. (2023), coral reef ecosystems (McDonald et al., 2023), cell location time series (Yang et al., 2023; Zhang et al., 2023), and honeybee aggregations (Gharooni-Fas et al., 2024). It has also in- spired methodological extensions such as multidimensional persistence (Kim & Mémoli, 2021) and the development of formigrams and crocker stacks (Xian et al., 2022).
Our approach achieves the following results:
We build a fast and scalable pipeline characterize the birth and death of topological features across transformer models' layers. As new contributions in the context of zigzag appli- cations, we introduce the k-Nearest Neighbors-based filtration, and we interpret layers as time snapshots, tracking the trajectory of features across layers.
ZigZag Framework for LLMs:
Identification of Phases of Prompt Processing: Us- ing interpretable topological descriptors, we character- ize the model's 's dynamical processing of prompts in representation space across layers. We use this char- acterization to identify four distinct phases prompts move in representation space across layers: an initial phase with rapid rearrangement of positions, mid- dle phase characterized by stable, long-lived relations among prompts, a transition phase where the model refines these relations, and a final phase of new rear- rangements preparing data for output.
Distinguishing Transformer Stages. Recent works have studied the geometry of internal representations to identify distinct stages' in the way pre-trained transformer mod- els process input across layers. For instance, Valeriani et al. (2023) demonstrated that transformer models exhibit a phase transition in the middle layers, characterized by a spike in intrinsic dimensionality, which correlates with the emer- gence of semantic and syntactic abstractions. Similarly, Cheng et al. (2023) highlighted that middle layers play crucial role in compressing input representations into lower- dimensional manifolds, enabling the model to generalize and handle complex linguistic tasks. Subsequent work has confirmed and expanded these findings in different ways (Lad et al., 2024; Artzy & Schwartz, 2024; Skean al., 2024).
As a showcase downstream task, we use our descriptors to define criterion to prune layers without significantly degrading performance, finding comparable results to state-of-the-ee methods.
Model Pruning:
Our topological descriptors show quantitatively different results but qualitatively similar across models, datasets, and choices of zigzag hyperparameters. This proves their ex- pressivity and simultancously shows a degree of universality in the topological structure of L.LM representations.
Layer Pruning in Large Language models. Among ex-
'While previous work has frequently used the word "stage" in this context, we prefer "phase" to emphasize the continuous. evolving nature of the process.
2

---

Persistent Topological Features in Large Language Models
isting methods reduce the size of neural networks, layer pruning has gained particular relevance in the context of LLMs. The first applications to BERT models (Fan et al.,
Layer 2
Laverr 3
Layer
Inter Iotornal Rapresser wwrsssaattttton tations
2020; Zhang& & He, 2020; Fan et al., 2021; Jha et al., 2024) inspired a long series of experiments employing similar techniques (Sajjad et al., 2023; Siddiqui et al., 2024; He et al., 2024; Zhang et al., 2024a; Kim et al., 2024; Zhang et al., 2024b). Many of these efforts base their method- ology on similarity measures of internal representations, which have conveniently been summarized in recent re-
G
Kr
Zypay Perssaanc
1-oyoloo G-oycles
view (Klabunde et al., 2023). In this work, we consider
(Gromov et al., 2024), which uses angular similarity, and (Men et al., 2024), which uses Block-Influent similarity, as a reference point for comparison.
Figure I. A schematic representation of the zigzag algorithm. shows how zigzag can track the evolution of topological features over time, and the descriptors of this work are built upon them.
3. Method
In this section, we introduce the zigzag persistence frame- work, which we use to analyze the internal representations of LLMs pre-trained with an autoregressive loss. These models typically receive an input sequence of n tokens (often repre- senting a sentence) embedded in a d-dimensional space. The input is transformed across the network layers without al- tering the embedding dimension. Due to the autoregressive nature of these models, the representation of the last token in a sequence captures information about the entire sequence and is used to predict the next. As a result, we choose to focus on the last token representation of each sequence at each layer. Thus, our point cloud is represented by last tokens embeddings, i.c. vectors of the form {xx) e Rd for i 1, .... Nsentences and j 1, ..... Nayerss These last tokens are extracted from a variety of datasets and serve as an observational probe of how the model processes input.
ball drawn around each point of the dataset.
Filtrations can be generalized to a more flexible structure called a zigzag filtration. Unlike a standard filtration, a zigzag filtration allows the sequence of complexes to move both forward and backward, meaning that inclusions be- tween complexes reverse at certain steps. We take this approach in our study to track the evolution of the internal representations across layers, rather than at fixed snapshot, as done in traditional persistent homology implementations. In this sense, our parameter is not a distunce/courseningg scale, but a discrete time scale represented by the layer num- ber. We track topological features as they are formed and de- stroyed along the model layers and statistically characterize these changes to describe a complex series of transforma- tions in tigh-dimensional space. Differently than standard persistent homology, short- and long-lived features represent how the model dynamically evolves. Short-lived features indicate a high rate of rearrangement of the points between adjacent layers, while long-lived features suggest phase of retention of (relative) positions across several layers. This is crucial point in our analysis, as it provides novel tool to interpret how the model processes different inputs by moving them and changing their relative positions in the representation space. Recent computational advances have allowed feasible implementation of these methods for ana- lyzing time-varying point clouds within the broader machine learning community. For instance, efficient algorithms such the fast zigzag persistence method introduced by Dey & Hou (2022) have enabled scalable analysis of evolving topological features, making persistence-bel based approaches much more accessible in large-scale applications. We now outline the main steps of the zigzag algorithm, leaving a rigorous mathematical formulation to Appendix A.
3.1. Topological Data Analysis
Topological data analysis (Edelsbrunner et al., 2002; Zomorodian & Carlsson, 2004) provides tool for geo- metrically characterizing highly complex datasets. Within this framework, persistent homology (Carlsson, 2009) is the key methodology to characterize a point cloud on multiple scales at once. Its goal is to identify the range of scales over which a particular class of topological features (connected components, loops, voids, higher dimensional "holes") re- main relevant, or "persistent", as opposed to "topological noise", i.c. features disappearing roughly at the same scale they formed. The basic ingredients for this technique are i) a criterion to connect points, forming a simplicial complex and ii) a scale parameter (often coarsening scale) such that given P1 S 1/2 then the two corresponding simplicial com- plexes are related by Kr, C Kyy. The ordered sequence of simplicial complexes for varying scale parameters is called filtration. An intuitive example is the Vietoris-Rip -ips filtra- tion, built from complexes parametrized by the radius of the
3

---

Persistent Topological Features in Large Language Models
3.2. Zigzag Persistence for Layer Analysis
to zigzag persistence. A notable effort in describing spatio- temporal networks similarly to this work is (Kim & Mémoli, 2021), where the main summary statistic (the rank invariant) involves calculating 6-dimensional data vector (4 across layers and 2 across scale) and thus combines a variation of both a time and a scale parameter, using the Rips filtration. Varying also the scale parameter is worth investigating in this context, and the techniques in (Kim & Mémoli, 2021) would be starting point for implementing it. The work (Kim & Mémoli, 2023) from the same authors is also related to our work since the maximal group diagram and the persis- tence clustergram (cfr. Figure 2) are "annotated" (with the representative topological features) barcodes. In this work, they fix a scale similar to our case.
We aim to study internal representations by tracking statisti- cal changes in the formation of p-dimensional holes gener- ated by connecting nearby data points within each layer &. As introduced above, the first ingredient for a TDA formula- tion is a criterion for connecting points. In this regard, we construct a k-Nearest Neighbors graph Gg, = (V2,, Et,) at every layer 4,, where the number ANN of neighbors is fixed hyperparameter (see (Le & Taylor, 2024) for a previous use of kxN-bassd filtration). To explore higher-order relations among points, we extend the dimension of the graph by fill- ing ligher-dimensional simplices. More precisely, we fill a simplex when its boundary, composed of lower- dimensional simplices (such as vertices and edges), is complete. In par- ticular, we consider a triangle as filled when it has three vertices with pairwise connections. Similarly, a tetrahedron is filled when four vertices are all interconnected by edges, totaling six edges. This concept extends to higher dimen- sions up to a specified maximum dimension m. Thus, in each layer, we construct the simplicial complex Ke, defined by:
Zigzag Persistence Diagram. The output of the zigzag al- gorithm is then multiset of birth-death pairs [b, d] known as the persistence diagram
Persp ($) = {{ [b,d] d] | b, E {0,..., 2(N - 1)}} f.
(3)
We thus work with a zigzag filtration naturally indexed by {0, 2,........ 2 1)}. Specifically, as shown in the Figure 1, even numbers starting from 0 are assigned to p- dimensional holes that emerge and disappear within the model layers. In contrast, odd numbers are designated for features at the intersection layers. It is important to note that homology classes are defined as equivalence classes, meaning that a connected component (in the case of O- dimensional homology) need not maintain the same form at the level of simplices throughout its lifetime. The orange connected component in the figure exemplifies this: in Layer 1, it corresponds to the three points {x5, To. 27) } connected by edges, forming a triangle. In the intersection layer, it is reduced to the edge (ma, In Layer 2 this edge merges with another connected component (depicted in red), mark- ing the death of the orange component. This feature ensures the robustness of our construction small changes in the KNN graph. A mathematical explanation of this is provided in Appendix A. The algorithm that generates Persp ( is schematically described in Appendix B, and in Appendix C we show a toy example using a calendar month task to visualize how we track zigzag barcodes.
VI E S, (Ie, E Ee, Ke.- = U {s SEV.
and |S| S m + 1}.
(1)
To track changes in the network, we compute intersection layers by identifying simplices present simultaneously in both adjacent layers. This allows to construct a sequence of inclusions between these complexes
Ke.
Ni.
Kec-
Nr, Ki.
Ni.. OKe.s
where define L = Mayers for conciseness. This se- quence represents our zigzag filtration, denoted by 4. This filtration is the second ingredient needed to define persis- tent homology. We thus define a notion of birth and death of p-dimensional holes, with p = 0, ... m7 - 1, being m the maximum dimension at which we expand the graph. Throughout this work, we choose m 4. which implies that the p-dimensional holes are well defined up to dimen- sion p- = 3. We can track the persistence of these objects as they appear in a given layer when a group of points ex- hibits a particular proximity and distribution in the complex and disappear at a subsequent layer when some points have moved apart, causing them to vanish. We illustrate the idea in Figure 1.
Effective Persistence Image. The pairs generated within Persp (b) can be visualized through a persistence im- age, a well-known descriptor within the TDA tools. The persistence image in our case results in a grid of size (2.N 1) x (2Niayers - 1), for each homology dimen- sion p. Each pixel in the grid is associated with an integer
Comparison to similar frameworks. A distinguishing feature of our methodology is the choice of a KNx filtration, whose stability was discussed in (Le & Taylor, 2024) in the context of persistent homology, though it was never applied
The repetition of a pair [b, d] indicates that multiple holes in dimension p have been created and destroyed in correspondence of the same layers.
4

---

Persistent Topological Features in Large Language Models
value corresponding the number of holes appearing with that birth-death pair. Defined this way, the persistence image does not discriminate between the model and intersection layers. Their behavior is generally fairly different, and have an alternating structure between model and intersec- tion layers. Hence, persistence images are not smooth as a function of layers. To achieve a smoother representation, we introduce effective persistence images, obtained by ex- cluding the intersection layers from the construction. This is achieved by defining a map, similar to the approach in (Kim & Mémoli, 2017), that translates the collection of intervals from the zigzag persistence diagram of the filtration in (2) into intervals, where the birth and death occur only across model layers. Formally, for b, d > O, we obtain:
Inter-Layer Persistence. To better track the persistence of features across layers, we can calculate the fraction of p-dimensional holes in one layer, L1, that exist in another layer, E2, as well, and have existed throughout the layers in between. 5 Mathematically it can be expressed as
22((1,(22) EASMI PIp (61, E2) 1o((1)
where Mj = min(e1, E2); M2 = max(e1, f2) and (t) is the Betti number, i.e. the number of alive p-dimensional holes at layer f. We can then further summarize this quan- tity again by power-weighted averaging it,
2 E w(e, (.) (p(
dIp(b/2,d/2) =PIp d) + (1,(6-1.d)
(8)
Ee=1 w(t,
Musyus
+ (P,,,,,- Pl 1,d 1); (4)
where we fix one of the two layers and average over all other layers, and the weight is the same as Eq. (6). Given that the birth or death of a given p-dimensional hole implies the rearrangements of points in space, 2,p tracks the dynamical movement of prompts' relative positions in representation space as a function of the model's depth.
where РT is the effective persistence image for the p- dimensional holes and b, are model layers indexed by even numbers."
3.3. Zigzag Descriptors
The collection of PTps taken over all contains all the information output from our zigzag algorithm, and gives useful overview of the model as whole. On the other hand, they are not easily tractable statistically and are hard to interpret. We extract two descriptors from the effective persistence image, defined below.
4. Experiments
4.1. Models, Datasets and Benchmarks
We work with 4 models: Llama2 (Touvron et al., 2023), Llama3 (AI@Meta, 2024). Mistral (Jiang et al., 2023) and Pythia 6.9B (Biderman et al., 2023). These models are open-source decoder-only transformers, and they achieve high performance in the benchmarks we consider in this work. We analyze Llama2-7B, Llama3-8B, Mistral 7B, and Pythia 6.9B because they have 32 hidden layers and have comparable parameter sizes. In Appendix E.2 we show results for larger models as a consistency check.
Births' Relative Frequency. A useful way to summarize a persistence diagram is by counting features within a spe- eific region of interest. In our context, is informative to measure the rate with which new p-dimensional holes created, this reflects the model's propensity to move prompts toward each other in specific regions of space. We thus define the births' relative frequencies as
The input dataset from which we take internal represen- tations must provide a fair test of how the model pro- cesses and understands language. We consider the follow- ing datasets: 1) The Standford Sentiment Treebank (SST) dataset (Socher et al., 2013). 2) The Pile dataset (Gao et al., 2020) from which we take subset of 10K prompts, accessi- ble on HuggingFace. ?3) A dataset of mathematical problems (Hendrycks et al., 2021b). 4) A dataset of codes retrieved
((e,)) P1,((e)) Bp(() = Ee, w(e l.) Ee, ())
(5)
where
w((,4)= = 1--1-.4 2,10
(6)
is a weight with varying exponent a.* 4 For negative values of a, the average gives weight to counts with low death values, effectively tracing the fraction of short-lived features. On the other hand, positive values of a give more weight to long-persisten features.
'We note that this is related to the generalized rank invariants in the context of multiparameter and zigzag persistence (Clause et al., 2024; Dey et al.. 2024), which measures the rank of the homology maps between consecutive layers in a zigzag filtration. However, it differs in that it is normalized by the Betti number and explicitly enforces the continuity of holes throughout the intermediate layers.
NNote that this operation does not modify the information about the model layers contained the original Pers, (.), as redefines consistently all the births and deaths.
'Note that (7) is well-defined only when (,(()) > If there are no p-dimensional holes at either E or fz. Zp(f1, f2) should he 0 by definition. We omitted this limit case from (7) for readability. "htts://huggingface htwp:/ggnnnnaacccddaataeeeeeeeee woldatasets/Neellanee Nacaaassee///Naa/////eelee1000
"This type of weighting has heen used previously for topologi- cal descriptors, see e.g. (Chazal et al., 2014).
5

---

Persistent Topological Features in Large Language Models
from GitHub.
born earlier on. This aspect is going to be evident when computing also the topological descriptors, below. In the Appendix E. we show a wider range of images, comparing them across models by taking the element-wise difference of effective persistence images.
Each prompt is processed from these datasets to extract the last token at each normalization layer and the final nor- malization is applied to the output layer. To ensure fair comparisons and eliminate potential biases in our descrip- tors caused by varying point cloud sizes, all datasets are reduced to the first 8,000 prompts. Additionally, we di- vide the datasets into incremental subsets of {100, 200,... .... 1000] prompts and compute the mean and standard devia- tion across subsets to systematically evaluate the scalability of our descriptors and to quantify their sensitivity to changes in point cloud size. For our experiments, we consider the 500 prompts subset, amounting to 16 subsets. In Appendix E.4 we perform a more detailed analysis at varying subset sizes. We use the SST dataset as a reference dataset for all the plots and show results for varying datasets in Appendix E.3.
30
Liama
4.0
3.5
Persistence (Maath -aan
25
1-dimemmemm holes
3.0
20
2.5
15-
2.0
1.5
10
1.0
5-
0.5
0
0.0
Birth Layer (famm)
We use 3 benchmarks for layer pruning performance evalua- tion: MMLU (Hendrycks et al., 2021a), HellaSwag (Zellers et al., 2019), and Winogrande (Sakaguchi et al., 2019), which have been widely used for similar purposes in previ- ous analyses. The benchmarks are evaluated for the models with the use of the library aa--eval-harness by (Gao et al., 2024) with a 5-shot setup.
Figure 2. Effective persistence image of 1-dimensional holes for the Llama 3 8B model using the SST dataset, where we fix KNN 4. The density plot shows the number of holes (color bar) for a given birth-persistence pair (x- and y-axis), where values refer the model layer. This plot shows that a large amount of I-dimensional holes are short-lived and that long-lived features appear after the first half of the model.
4.2. Zigzag persistence applied to LLM models
Births' Relative Frequency. In the left panel of Figure we show B1 (Eq. 5) for Llama 3 8B on the SST dataset, for varying a = - 1, 0, 0.5, 1, We can clearly distinguish two behaviors for short- and long-lived 1-dimensional holes: the former peaks at early layers and progressively decreases, while the latter peaks at middle layers. Additionally, strong increase in births number is seen in the last few layers. We highlight a horizontal line corresponding to the uniform distribution for comparison. We complement these results in Appendix E.3 with a comparison across models and datasets, finding qualitatively similar results.
We generate zigzag diagrams for each model and dataset and for each homology dimension up p = 3, for range of val- ues of ANN E [1, 15). We find that 0-, 2 and 3-dimensional holes are relatively lower in number, while 1-dimensional holes reach tens of thousands of elements per layer. This behavior might be expected for a kNN WoN--gaap-baseer construc- tion since connections are dense even for low values of KNN, especially if points are concentrated in low dimensional regions of the representation space. We examine this be- havior in detail to make sure that our construction is stable for different choices on the ANN graph, see Appendix D for details. The choice of the hyperparameter NNN is done so as to maximize the total number of holes. Therefore, in what follows, we show results for our topological descriptors for 1 1-dimensional holes and ANN 4 only.
Inter-Layer Persistence. We show the power-weights inter-layer persistence (Eq. (8) in the middle and right panel of Figure 3. In the middle panel, we use Llama 3 with the SST dataset at varying = - 1,, 0.5, 1. As for B1. we can distinguish two behaviors for livrt-lived and long-lived features, though now 2, traces the probability of features that are alive at a given layer to be still alive in earlier or later layers. We see that for short-lived features, this probability grows steadily until the second half of the model's depth, where it reaches a plateau, and then suddenly drops in the last few layers. On the other hand, for long- lived features, there is a peak in probability at the middle layers. We can qualitatively see the same behavior for the other models in the right panel of Figure 3, though with quantitative differences across models. We cross-check for
Effective Persistence Image. We show an example of an effective persistent image of 1-dimensional holes in Figure 2, where we use the Llama 3 8B model and the SST dataset. The x-axis represents the layer at which 1-dimensional hole is born, and the y-axis represents persistence, i.c. death layer- - birth layer. The color bar measures the amount of 1-dimensional holes on a given grid point. We see that features born after the first half of the model's depth have a higher tendency to be long-lived with respect to features
6

---

Persistent Topological Features in Large Language Models
0.12
0.12
em-000
Liama
d- 0.00
Liama
0.10
0.50
0.10
Births Relative frequency
d=
2,0
Persistence
0=1.00
Niter-Layer Persistence
0.08
Uniform Distribution
0.08
0.3
0.2
Liama BB
Liama 8.88
a= 15 Layer
20
25
30
20
20
Layer
Layer
Figure 3. Left Panel: Births' Relative Frequency, B1, as function of model's layers for Llama 8B on the SST dataset, for varying which traces short-(long-)it (long) features for negative(positive) values. Short-lived features peak early in the model and progressively decrease, while long-lived features peak in middle layers. All features experience sharp growth in the last two layers. The dashed line represents a uniform distribution of births across the layers. Middle Panel: Inter-Layer Persistence as a function of model's 's layers for Llama 8B the SST dataset, for varying The persistence of short-lived features consistently grows and plateaus towards the end of the model, while long-lived lived features are primarily present across middle layers. Right Panel: Inter-Layer Persistence for Llama 3, The models considered exhibit qualitatively similar, but quantitatively different
Llama Mistral and Pythia for the SST dataset behavior, with Pythia experiencing the higher peak, and Mistral the lowest. This seems inversely related to model's performance once pruning layers in this range. In all pancls, curves and shaded regions represent the mean and standard deviation over 16 subsets of 500 prompts, respectively.
other datasets in Appendix E.3, also finding qualitatively similar, but quantitatively different results across datasets.
intrinsic dimension are progressively better distinguish- able and more stable to noise.
Middle Late Layers: Short-lived 1-dimensional holes born after the first layers decrease in number. At the same time, the probability that 1-dimensional hole is short-lived increases until after the middle lay- ers when it reaches a plateau. Concurrently, the proba- bility (and the amount) of long-lived ones drops. This indicates a phase of relatively few short-lived adjust- ments in the relative positions of prompts since many of the features that formed in the middle layers are still there (because they are livelivee). We expect that these short-lived adjustments relate phase of spe- cialization (Lad et al., 2024) and to phase of relatively constant dimensionality (Valeriani et al., 2023).
4.3. Interpretation and implications for the model's performance
In interpreting our results, it is essential to recognize that: 1) models process each token of the prompt, while we use only the last token as a proxy for the entire prompt; and 2) each prompt is processed separately from others, such that each point moves priori independently from the others in the representation space. Within this framework, the zigzag algorithm effectively tracks how the models dynamically ganize prompts across both spatial and temporal dimensions (layers). Our findings, as illustrated in Figure 3, reveal four distinct phases:
Early to Middle Layers: In the first layers, large num- ber of short-lived 1- 1-dimensional holes are formed, in- dicating that most prompts are fastly rearranged within a few layers. This finding relates with previous work identifying local contextualization (Lad al., 2024) and increased dimensionality (Valeriani et al., 2023).
Last Layers: In the last relative frequency grows rapidly while the inter- layer persistence drops. These two behaviors are compatible, given that the fraction of newborn 1-dimensional holes is large, and that there are no layers left to persist. This results suggest another strong rearrangement of points, which can be linked to the model producing the required output (Valeriani al., 2023; Lad et al., 2024) and thus changing abruptly the position of prompts in representation space.
three layers, the births'
Middle Layers: In this phase, 1-dimensional holes born in middle layers have the highest probability of being long-lived than in other phases. This implies that rela- tive positions before, but especially after these layers are kept relatively stable. This would seem related to the decreasing dimensionality found in (Cheng et al., 2024), since relevant degrees of freedom estimated by
Relation to model's performance. As a test of the inter- pretation of the phases, we perform the following exper-

---

Persistent Topological Features in Large Language Models
5-block sliding window
MMLU
HellaSwag
WinoGonde
Liama 2 88
Models
Full
This
Other
Full
This
Other
Full
This
Other
Lama 70
work
works
work
works
work
works
kythis
Lloma
45.74
57.58
43,95
58.54
44.71
42.78
74.43
68.67
67.72
Winourann occutaLy
Mazrol random
Llama
65.07
53.44
53.44
61.37
41.60
41.60
77.10
70,00
70.00
Mistral 7B
62.40
53.17
38.20
62.83
36.67
34.45
77.35
66.55
63.76
Pythia
49.70
31.43
34,96
63.30
55.71
58.09
Table I. Benchmark Table, For each benchmark, we show three columns: (i) Full, represents the accuracy of the model without any layer pruned. (ii) This work, accuracy of the model, where layers are pruned following the algorithm 2). (iii) Other works, accuracy obtained by considering the same amount of layer pruned estimated with our method and then computing the layer to be pruned with two different similarity measures: angular distance from (Gromov et al., 2024) and Bi-score from (Men al., 2024). The chosen layers turn out to be the same for the two methods, so the results are condensed into one column.
12_0)
10)
[0_12]
[20. [10_
(12_16)
114. 181
[16. (16_20)
118 [18_22]
120. 24]
[22 (2_26)
124.2] 22.281
120. 30]
[20. 32]
Te.
Figure 4. Winogrande performances for 4 models obtained with sliding window of blacks of adjacents layers and moved through the models every? 2 layers. This experiment reflects 4 phases: 1) removing early layers brings performances close to random choice, 2) while performance grows to almost maximum after middle layers. and 3) plateaus: 4) removing late layers causes another drop in performance right before the end of the model.
based on the plateau observed in the inter-layer persistence of short-lived features. Specifically, we prune layers that lie within 10% of the maximum value of 2, This is com- puted for each different model, using the Pile dataset as proxy. We show schematic summary of the algorithm in Appendix G.2.
iment: we prune blocks of layers with a sliding window from early to late layers as a way to compare the relative importance of layers in the various phases. We show the results of the experiment in Figure 4, where we show the performance of the 4 models considered for the Winogrande benchmark, as a function of the sliding window of pruned layers. We see that removing layers in the first phase signifi- cantly affects performance. After the second phase, pruning weakly affects performance, being phase of relative adjust- ments. Removing the last few layers causes another drop in performance. Interestingly, the overall performance of models is inversely related to the peak height of the inter- layer persistence of long-lived features. This relation is seen also in the other limit of a < O. We also notice that the two best-performing models, Mistral and Llama, 3 exhibit drop in performance when removing layers at the end of the third phase, right before the fourth. We zoom in on these layers for the MMLU benchmark where the drop is particularly evident in Appendix G. confirming that the drop is caused by removing the last 2-3 layers of the third phase. Importantly, all these results are qualitatively the same across the three benchmarks considered in this work.
We compare our layer pruning methods to recent work (Gro- mov et al., 2024) and (Men et al., 2024) performing pruning using similarity measures. Both approaches are designed to take as input the desired number of layers to prune N Nprune and measure performance as Nprime grows. For fair com- parison, we feed the number of layers cut by method as input to the other two methods, and verify which layers they select to cut given this input, and the corresponding performance. We show which layers are cut for each method in Table 2 in Appendix G.2. Interestingly, both considered methods from (Gromov et al., 2024) and (Men al., 2024) give the same result at fixed Nprune. thus we refer to them simply as "other works". We show performance results in Table 1, 10 where in bold we indicate the layer pruning method that has better or equal performance with respect to the other method. Despite often selecting different layers, our zigzag- based pruning strategy achieves comparable re- sults to methods from (Gromov et al., 2024) and (Men et al., 2024).
5. Conclusions
4.4. Layer Pruning
Recent work has argued in different ways that large language models process inputs across layers through distinct phases,
Recently, measures of layer similarity have been used to identify layers that contribute minimally to the performance of L.LMs. These layers can be pruned, and the performance re-evaluated to validate this assumption. Given our results in Figure 4, we can argue that layers belonging to the third phase might be pruned without affecting for model's per- formance. Consequently, we establish a pruning criterion
"We use Pile since it is characterized by a broad range of topics, representing wider range of prompts. Note that the shape of the curve around the peak of Z, approximately similar across datasets (see Figure 9 in Appendix).
Reesults for Pythia on MMLU tasks are not shown because the model is designed for following the format of the tasks, as shown in (Biderman et al., 2023).
8

---

Persistent Topological Features in Large Language Models
and that understanding these phases is important for the models' interpretation. We exploit topological data analysis tools to build descriptors that allow to statistically character- ize the dynamics of prompts within internal representations of large language models. Based on this characterization, we distinguish four phases and connect them to the model's behavior through experiments based on layer pruning and performance benchmarking. Our method consistently pro- vides qualitatively similar results across different models, datasets, and parameter selections. Simultaneously, our topological descriptors allow for quantitative differentiation across models and datasets, creating opportunities for ex- periments designed to address more specific and practical questions regarding particular models or datasets.
F53C22001770002 A.A. and A. C. were supported by the European Union NextGenerationEUU within the project PNRR "PRP@ CER IR0000028 Mission 4 Component 2 Investment 3.1 Action 3.1.1.
We thank Area Science Park supercomputing platform OR- FEO made available for conducting the research reported in this paper and the technical support of the Laboratory of Data Engincering staff.
References
AI@ Meta. Llama 3 model card. 2024. URL aalllamaamlaaaaa .nd.
blob/maann/MCDE__.CO
There are several limitations in our study that future research could address. First, while our method shows robustness across hyperparameters within the framework, these choices need not be optimal. Defining an appropriate criterion for connecting points in the representation space, and conse- quently, a filtration, is a delicate task in TDA that could require further investigations to detail the impact of the vari- ous choices on the construction of the filtration. Secondly, our study primarily focuses on static, pre-trained models. Extending this framework to track the evolution of inter- nal representations during training might provide important insights into model efficiency and behavior.
Ansuini, A., Laio, A., Macke, J. H., and Zoccolan, D. In- trinsic dimension of data representations in deep neural networks. Advances in Neural Information Processing Systems, 32, 2019.
Artzy, A. B. and Schwartz, R. Attend first, consolidate later: On the importance of attention in different LLM layers. Belinkov, Y., Kim, N., Jumelet, J., Mohebbi, H., Mueller, A., and Chen, H. (eds.), Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Inter- preting Neural Networks for NLP, pp. 177-184, Miami, Florida, US, November 2024. Association for Computa- tional Linguistics. doi: 10. 10118653w//20244beekbxx--- 10. URL https: / wwwaanthhoog..ooooooooooooooeeoe 2024. b lackkooolppp--. 1. 10/.
6. Reproducibility
All the results contained in this work reproducible by means of GitHub repository that be found at this link
Bai, G., Chai, Z., Ling, C., Wang. S., Lu, J., Zhang, N., Shi, T., Yu, Z., Zhu, M., Zhang, Y., Yang, C., Cheng, Y., and Zhao, L. Beyond efficiency: A systematic survey of resource-efficient large language models, 2024.
Impact Statement
This paper presents work whose goal to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Barannikov, S., Trofimov, I., Balabin, N., and Burnaev, E. Representation topology divergence: A method for com- paring neural network representations. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S. (eds.). Proceedings of the 39th International Confer- ence on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 1607-1626. PMLR, 17-23 Jul 2022. httthttts://proceee nigs mlr. press/ https//v12/bbarraae html..
Acknowledgments
We thank Mathicu Carrière and Magnus Botnan for help- ful discussions on the TDA implementation and sugges- tions on the zigzag algorithm. M.B. Y.G. and G.P. are par- tially supported by the Programma Nazionale della Ricerca with title "FAIR- "FAIRR-b-design" K.V. was partially supported by Programma Nazionale della Ricerca (PNR) grant 195F2100283000 tle "FAIR-by -FAlR-by-design" during his visit to Area Science Park while this project was in its development phase. A.A. and A.C. were supported by the project "Supporto alla diag- nosi di malattie rare tramite l'intelligenza artificiale" CUP:
Biagetti, M., Cole, A., and Shiu, G. The persistence of large scale structures. part primordial non -ou-sausiiantty Journal of Cosmology and Astroparticle Physics, 2021 (04):061, April 2021. ISSN 1475-7516.
(PNR) grant J95F2100283000
with the ti-
doi: 10. 1088/1475--5116/22210/0/011 URL org/10 000..01000/1445755550000000
Biderman, S., Schoelkopf, H., Anthony, Q., Bradley,
H., O'Brien, K., Hallahan, E.,, Khan, M., Purohit, S.,
9

---

Persistent Topological Features in Large Language Models
Prashanth, U., Raff, E., al. Pythia: A suite for ana- lyzing large language models across training and scaling, 2023, latent knowledge in language models without su- pervision. In The Eleventh International Conference on Learning Representations, 2023.
Dey, T. K., Kim, W., and Mémoli, F. Computing Generalized Rank Invariant for 2-Parameter Persis- tence Modules via Zigzag Persistence and Its Appli- cations. Discrete & Computational Geometry. 71(1): 67-94, January 2024, ISSN 1432-0444. doi: 10.1007/
500454-023-0058842 URL https:: //tp:///oooo//ooo org/10. 1007 020-00044-023-0000000
Carlsson, G. Topology and data. Bulletin of the American Mathematical Society, 46(2):255-308, 2009.
Doimo, D., Glielmo, A., Ansuini, A., and Laio, A. Hierar- chical nucleation in deep neural networks. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M. F., and Lin, H. (eds.), Advances in Neural Information Processing Sys- Curran Associates, Inc.,
Carlsson, G. and de Silva, Zigzag persistence. Found. Comut. Math., 10(4):367-405, August 2010.
Carlsson, G. E., de Silva, V., and Morozov, D. Zigzag persistent homology and real-valued functions. In SCG '09, 2009. URL https://p/////c /api. http:///aa...aaaaaa.e.ee.eesseee semantccccholar. org org/Corpuse ID: 5501226..
tems, volume 33, pp. 2020.
Edelsbrunner, Letscher, and Zomorodian. Topological per- sistence and simplification. Discrete & computational geometry, 28:51 1-533, 2002.
Chazal, F., Fasy, B. T., Lecci, F., Rinaldo, A., and Wasser- man, L. Stochastic convergence of persistence landscapes and silhouettes. In Proceedings of the Thirlieth An- nual Symposium on Computational Geometry, SOCG'14, pp. 474-483, New York, NY, USA, 2014. Associa- tion for Computing Machinery. ISBN 9781450325943. doi: 10. 1 14511445222122225222211 org/ /1/0110.00
EI- Yaagoubi, B., Chung, M. K., and Ombao, H. Topologi- cal data analysis for multivariate time series data. Entropy, 25(11), 2023. ISSN 1099-4300. doi: 10.3390/e25111509 URL httpt: /www. htww:/wwwpppppoto com/ wwt. 4300/25/ 11/1509.
URL https: //doi. 2582222
Engels, J., Michaud, E. J., Liao, I., Gurnee, W., and Tegmark, M. Not all language model features are one- dimensionally linear. In The Thirteenth International Conference on Learning Representations. URL https: / /ooenreview net (oorummiddd633 4AM4hb.
Cheng. E., Kervadec, C., and Baroni, M. Bridging information theorretcc and geometric compression lan- guage models. In Bouamor, H., Pino, J., and Bali, K. (eds.), Proceedings of the 2023 Conference on Em- pirical Methods Natural Language Processing, pp- 12397-12420, Singapore. December 2023. Association for Computational Linguistics. doi: 10. 10.18653/v1/0233 emnlp-main.762. URL patp:://acaaattholoo.oo org/2023. emn 1p 1-mpp-maa n. 762.
Fan, A., Grave, E., and Joulin, A. Reducing trans- former depth demand with structured dropout. In International Conference on Learning Representations, 2020. URL https: /ttp:////ppeeeiiieeee net forum? i id=Sy102yStDr.
Fan, C., Li, J., Zhang, T., Ao, X., WU, F., Meng, Y., and Sun, X. Layer-wise model pruning based on mutual information. In Moens, M.-F., Huang, X., Specia, L., and Yih, S. W.-t (eds.), Proceedings of the 2021 Con- ference Empirical Methods in Natural Language Pro- cessing, pp. 3079-3090, Online and Punta Cana, Domini- can Republic, November 2021. Association for Computa- tional Linguistics. doi: 00.18653/v1/2021.mmm 246. URL https:////// https///aaaaatthhaceee emn 1p 1alpp-miin mw..maii.... 246.
Cheng, E., Doimo, D., Kervadec, C., Macocco, I., Yu, J. Laio, A., and Baroni, M. Emergence of high- dimensional ubstraction phase in language transformers. arXiv preprint a arXiv:2405. /5471, 2024.
Clause, N., Kim, W., and Mémoli, F. The generalized rank invariant: Möbius invertibility, discriminating power, and connection to other invariants, 2024. URL https: s/arxiv.orr org/ abs/ 11591.
org/20211
Dey, T. K. and Hou, T. Fast Computation of Zigzag Persis- tence. Chechik, S., Navarro, G., Rotenberg, E., and Herman, G. (eds.), 30th Annual European Symposium on Algorithms (ESA 2022), volume 244 of Leibniz International Proceedings in Informatics (LIPles), pp. 43:1-43:15, Dagstuhl, Germany, 2022. Schloss Dagstuhl - Leibniz-Zentuum für Informatik. ISBN 978- URL https: //drops. dagst wwwpuuhh..eeeettiteeeeeee http://rooo..ggguu....eeeeeeeeee 420.04..1IIII.55
Gabriel, P. Unzerlegbare darstellungen manuscripta mathematica, 6(1):71-103, March 1972. ISSN 1432- 1785. doi: 10. 10.107/6611298411 URL http://dk doi. wtt.:o///......//...... 100./8F0119//111
Gao, I.., Biderman, S., Black, S., Golding, I.., Hoppe, T., Foster, C., Phang, J., He, H., Thite, A., Nabeshima, N., Presser, S., and Leahy, C. The Pile: An 800gb dataset diverse text for language modeling. arXiv preprint arXiv:2101.0000 2020.
3-95977-247-1. doi: 10. 1330
document/10. 42300 LIPICS. ESA. 2022.43.
10

---

Persistent Topological Features in Large Language Models
Gao, L.,, Tow, J., Abbasi, B., Biderman, S., Black, S., DiPofi, A., Foster, C., Golding, L., Hsu, J., Le Noac"h, A., Li, H., McDonell, K., Muennighoff, N., Ociepa, C., Phang,
Kim, B.-k., Kim, G., Kim, T.-H., Castells, T., Choi, S., Shin, J., and Song, H.-K. Shortened Ilama: Depth pruning for large language models with comparison of retraining methods, 2024. URL https: hatp:://aaxxv.rrrrr/ abs/ C2834.
J., Reynolds, L., Schoelkopf, H.. Skowron, A., Sutawika, L.,, Tang, F., Thite, A., Wang, B., Wang, K., and Zou, A. A framework for few-shot language model evaluation, 07 2024. URL https: //zp:/d/////.doo////// org/records/ 12608602.
Kim, W. and Mémoli, F. Stable signatures for dynamic graphs and dynamic metric spaces via zigzag persistence. arXiv: Algebraic Topology, 2017. URL htw.::ttpt/t/tttc /api. s semanticscholars org/ Cttp://opppsI:::::ee 44017455.
Gharooni-Fard, G., Byers, M., Deshmukh, V., Bradley, E., Mayo, C., Topaz, C. M., and Peleg, O. A computa- tional topology-based spatiotemporal analysis technique for honeybee aggregation. npj Complexity, 1(1):,, 2024.
Kim, W. and Mémoli, F. Spatiotemporal persistent ho- mology for dynamic metric spaces. Discrete Comput. Geom., 66(3):831 875, October 2021. ISSN 0179-
5376. doi: 10.10077/504554-1190000000 / //doi. http://g//0.1100/000044-0-----30
URL https:
Gromov, A., Tirumala, K., Shapourian, H., Glorioso, P., and Roberts, D. A. The unreasonable ineffectiveness of the deeper layers, 2024. URL https: htttps://axxiv.rrg abs/2403 17887.
68-
Kim, W. and Mémoli, F. Extracting persistent clusters in dynamic data via möbius inversion. Discrete Comput. Geom., 71(4):1276-13)
October 2023. ISSN 0179- 5376. doi: 10.1007/.00455--23--00000-- URL https: //dbi. 1ttp:/10.1000000 1007/000054 00590--0000111
He, S., Sun, G., Shen, Z., and Li, A. What matters in transformers? not all attention is needed. CoRR,
abs/2406.15786, 2024. URL https: httt:p://dci.orgg// 48550/ar 211.xxvv2406.17888
Klabunde, M., Schumacher, T., Strohmaier, M., and Lem- merich. F. Similarity of neural network models: A sur- vey of functional and representational measures. arXiv preprint arXiv:2305.06322 2023.
Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring massive multitask language understanding. Proceedings of the International Conference on Learning Representations (ICLR), 2021a.
Lacombe, T., Ike, Y., Carriere, M., Chazal, F., Glisse, M., and Umeda, Y. Topological uncertainty: Monitoring trained neural networks through persistence of activation graphs. arXiv [stat.MJ,, May 2021.
Hendrycks, D.,, Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring mathematical problem solving with the MATH dataset. In Thitty-fith Conference on Neural Information Process- ing Systems Datasets and Benchmarks Track (Round 2), 2021b. htthttps::/ /tpptttpp///opereeeeeee repireevee..... id- 7yytt
Lad, V., Gurnee, W., and Tegmark, M. The remark- able robustness of LLMs: Stages of inference? In ICML 2024 Workshop on Mechanistic Interpretability, 2024. URL https:
forum?
/tt.tttt////peeeiiiiii /ttp:///pppeeeeeeeeee/epe. https/iwwwwweeeeeeeeeeeeeeeeeeee wttp:/ornttffr forum? id=R5umwb9KPc.
Hensel, F, Moor, M. and Rieck, B. A survey of topological machine learning methods. Fron- fiers in Artificial Intelligence, 4, 2021. ISSN URL
Le, M. Q. and Taylor, D. Persistent homology with k-nearest-neighhor filtrations reveals topological convergence of pagerank. Foundations of Data Science, 2024. doi: 10.3934/fodds.224403.. URL
2624-8212. doi: 10.3389/frai.22221681111
https://ww..frontieesii..cccc
https: /wwp://wwwww///oo/ wwtp:wwww....................t.w (a666:3008be77225666666556 225ea2264676627722
artifi artffciaal-Ineeliggneeeattttnnne 28111000...18...
Liao, Q. V. and Vaughan, J. W. Ai transparency in the age of
Jha, A. H., Sherborne, T., Walsh, E. P., Groeneveld, D., Strubell, E., and Beltagy, Just chop: Embarrassingly simple Ilm compression, 2024. URL htttt:tt////aat org/ abs arg/abs/ 2205 14864.
1mms: A human-centered research roadmap, 2023. URL
wwtp:///g///baaa/aeeo/.oo 01941.
Ma, X., Fang, G., and Wang, X. LLM-pruner: On the struetural pruning of large language models. In Thirty- seventh Conference Neural Information Processing Systems, 2023. URL https: http://ppeeeeeeeeeeeeeeeeeeeeeee nne// forum? i j=
Jiang, A. Q. Sablayrolles, A., Mensch, A., Bamford, C., Chaplot, D. S., de las Casas, D., Bressand, F., Lengyel, G., Lample, G., Saulnier, L., Lavaud, L. R., Lachaux. M.- A., Stock, P., Scao, T. L., Lavril, T., Wang, T., Lacroix, T., and Sayed, W. E. Mistral 7b, 2023. URL https: //a //arxiv.orr www.xivorggbbs///oo abs///s//1//111 06825.
Magai, G. and Ayzenberg, A. Topology and geometry of data manifold in deep learning. ArXiv, abs/2204.08624, April 2022.

---

Persistent Topological Features in Large Language Models
Mandal, S., Guzmán-Sáenz, A., Haiminen, N., Basu, S., and Parida, I.. A topological data analysis approach predicting phenotypes from gene expression data. In Martín- Vide, C., Poga-Rodriguez, M. A., and Wheeler, T. (eds.), Algorithms for Computational Biology, pp- 178-187, Cham, 2020. Springer International Publish- ing. ISBN 978-3-030-42266-0.
Thomas, and Borgwardt, Karsten. Neural persis- tence: A complexity measure for deep neural net- works using algebraic topology. 2023, doi: 10.3929/ ETHZ-B-000327000 URL http:/ hnd..handle net/20. 2et/20.500.. 11180112882201
Sajjad, H., Dalvi, F., Durrani, N., and Nakov, On the ef- fect of dropping layers of pre-trained transformer models. Comput. Speech Lang. 77(C). Jamuary 2023. ISSN 0885- 2308, doi: 10. (0..1016/jccs..222210110
McDonald, R., Neuhausler, R., Robinson, M., Larsen, L.,
Harrington, H., and Bruna, M. Zigzag persistence for coral reef resilience using stochastic spatial model. Jour- nal of the Royal Society, Interface, 20:20230280, 08 2023. doi: 10. 10..000ssi..2222.2222222
URL https: orddi..og/1000 111111111111 202..
Sakaguchi, K., Bras, R. L., Bhagavatula, C., and Choi, Y. Winogrande: An adversarial winograd schema challenge at scale, 2019. URL https: https::/arxiv..rrm http://axivv.... abs/ 10641.
Men, X., Xu, M., Zhang. Q. Wang. B.. Lin, H., Lu, Y., Han, X., and Chen, W. Shortgpt: Layers in large language models are more redundant than you expect, 2024. URL https:/ // //axxvivv..//////// org/ abs /2003 03853.
Samsi, S., Zhao, D., McDonald, J., Li, B., Michaleas, A., Jones, M., Bergeron, W., Kepner, J., Tiwari, D., and Gadepally, V. From words watts: Benchmarking the energy costs of large language model inference. In 2023 IEEE High Performance Extreme Computing Conference (HPEC), pp. 1-9, 2023. doi: 10.1 (0.1109/HPEC58863.22222 10363447.
Morozov, D. Dionysus2. URL https:: /ttp//wwww/w//////////////////tw org/so hrg/softten htg/software///e setaarr/ddinyyeee dioysuss2/.
Myers, A., Muñoz, D., Khasawneh, F. A., and Munch, E. Temporal network analysis using zigzag persistence. EPJ Data Science, 12(1):6, 2023.
Siddiqui, S. A., Dong, X., Heinrich, G., Breuel, T. M., Kautz, J., Krueger. D., and Molchanov, P. A deeper look at depth pruning of llms. CoRR, abs/2407.162866 2024. URL https: http:/dooo.oo////////// 48550/arXiv www.:iiwi............. 15286.
Naitzat, G., Zhitnikov, A., and Lim, L.-H. Topology of deep neural networks. Journal of Machine Learning Research, 21(184):1 2020. URL http:// jmml.. arg/pappess//21200 245220000344000000 html.
Olah, C., Satyanarayan, A., Johnson, I., Carter, S., Schubert, L.. Ye, K., and Mordvintsev, A. The building blocks of interpretability. 2018. doi:
Skaf, Y. and Laubenbacher, R. Topological data
analysis in biomedicine: A review. Journal of Biomedical Informatics, 130:104082, 2022. ISSN 1532- 0464. doi: https///ooaagg///llooojjj2222220 URL attense/artcce///////22222220200
Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., and Gold- stein, T. The intrinsic dimension of images and its impact learning. In International Conference Learning Representations, 2021. URL https: /ppperevee..... net/ / forum depp-id--- 1 9XzGq2J.
Skean, O., Arefin, M. R., and Shwartz-Ziv, R. Does repre- sentation matter? exploring intermediate layers in large language models. In Workshop Machine Learning and Compression, NeurIPS 2024, 2024. URL https: / net// forum?
Raiaan, A. K., Mukta, M. S. H., Fatema, K., Fahad, N. M., Sakib, S., Mim, M. M. J.. Ahmad, J., Ali, M. E., and Azam, S. A review on large language models: Ar- chitectures, applications, taxonomies, open issues and challenges. IEEE Access, 12:26839-26888 226844, 2024. doi: ALL.9//ACCEESS22223336555
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A., and Potts, C. Recursive deep models for semantic compositionality over sentiment treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp- 1631- 1642, Seattle, Washington, USA, October 2013. Asso-
Rathore, A., Chalapathi, N., Palande, S., and Wang, B. Topoact: Visually exploring the shape of ac- tivations in deep learning. Computer Graphics Fo- rum, 40(1):382-397 doi: https://ooi.rr//00.11111
ciation for Computational Linguistics. URL https: / htwww. aclweb. org wntptoothhoooooooo/oooooooooooot /D13 1170.
14195. URL httts::/oolinelllee www.iirrrrrrwwlllll........ wiley. hom/doi/aee abs /10.///0... wtt.aa/...//.eeeeeeeeaaaaaaaeeee wtt.111//cff..1111111 14195.
Suresh, S., Das, B., Abrol, V., and Roy, S. On characteriz- ing the evolution of embedding space of neural networks using algebraic topology. arXiv [cs.LG], November 2023.
Rieck, Bastian Alexander, Togninalli, Mattco, Bock, Christian, Moor, Michael, Horn, Max, Gumbsch,
12

---

Persistent Topological Features in Large Language Models
Tausz, A. and Carlsson, G. E. Applications of zigzag persistence to topological data analysis. CORR, abs/1108.3545, 2011. URL attp://arxiv..om abs/1108. 3545.
classification of tumour-immune interactions and dynam- ics. arXiv preprint arXiv:2308.052224 2023.
Yip. J. H. T., Biagetti, M., Cole, A., Viswanathan, K.,, and Shiu, G. Cosmology with persistent homology: a Fisher forecast. JCAP, 09:034, 2024, doi: 10. (0.1088/1475-756) 2024/09/034.
Touvron, H., Martin, L., Stone, K., Albert, P., Almahairi, A., Babaei, Y., Bashlykov, N., Batra, S., Bhargava, P., Bhosale, S., Bikel, D., Blecher, L., Ferrer, C. C., Chen, M., Cucurull, G., Esiobu, D., Fernandes, J., Fu, J., Fu, W., Fuller, B., Gao, C., Goswami, V., Goyal, N., Hartshom, A., Hosseini, S., Hou, R., Inan, H., Kardas, M., Kerkez, V. Khabsa, M., Kloumann, I., Korenev, A., Koura, P. S., Lachaux, M.-A., Lavril, T., Lee, J., Liskovich, D., Lu, Y., Mao, Y., Martinet. X., Mihaylov, T., Mishra, P., Moly- bog, I., Nie, Y., Poulton, A., Reizenstein, J., Rungta, R., Saladi, K., Schelten, A., Silva, R., Smith, E. M., Subra- manian, R., Tan, X. E., Tang, B., Taylor, R., Williams, A., Kuan, J. X., Xu, P., Yan, Z., Zarov, I., Zhang, Y., Fan, A., Kambadur, M.. Narang. S., Rodriguez, A., Sto- jnic, R., Edunov, S., and Seialom, T. Llama 2: Open foundation and fine-tuned chat models, 2023. URL https: / arxiv. org/ has///b////// 2207 09288.
Zeiler, M. D. and Fergus, R. Visualizing and understanding convolutional networks. In Fleet, D., Pajdla, T., Schiele, B., and Tuytelaars, T. (eds.), Computer Vision ECCV 2014, pp. 818-833, Cham, 2014. Springer International Publishing. ISBN 978-3-319 978-3-319-10590-1.
Zellers, R., Holtzman, A., Bisk, Y., Farhadi, A., and Choi, Y. HellaSwag: Can a machine really finish your sen- tence? In Korhonen, A., Traum, D., and Mårquez. L. (eds.), Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 4791- 4800, Florence, Italy, July 2019. Association for Compu- tational Linguistics. doi: 10. 10.18653/v1//119-4472 URL https: http://ccaaatttoaoooooaoo wwt.:/9gg/pp99/.
Zhang, M. and He, Y. Accelerating training of transformer-basee language models with progres- sive layer dropping. In Larochelle, H., Ranzato, M., Hadsell, R., Balcan, M., and Lin, H. (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 14011-14023, Curran Associates, Inc., 2020. URL https://proceeaiing neurgps... cc / [papper__filessppppe///////// a aa150a8000fccceeeae5444333333333 pdf.
Tulchinskii, E., Kuznetsov, K., Kushnareva, L.., Cherni- avskii, D., Nikolenko, S., Burnaev, E.,, Barannikov, S., and Piontkovskaya, I. Intrinsic dimension estimation for robust detection of ai-generated texts. In Proceedings of the 37th International Conference on Neural Informa- tion Processing Systems, NIPS '23, Red Hook, NY, USA, 2024. Curran Associates Inc.
Tymochko, S.. Munch, E., and Khasawneh, F. A. Using zigzag persistent homology to detect hopf bifurcations in dynamical systems. Algorithmss, 13(11)):78, October 2020. ISSN 1999-4893. doi: 10. 10278. URL http: // dx. doi. org/10 00/./110:000 3390//31199/1137
Zhang, M., Chowdhury, S., and Saggar, M. Temporal map- per: Transition networks in simulated and real neural dynamics. Network Neuroscience, 7(2):331 2023.
Valeriani, L.. Doimo, D., Cuturello, F., Laio, A., ansuini, A., and Cazzaniga, A. The geometry of hidden represen- tations of large transformer models. In Thirty-seventtl Conference on Neural Information Processing Systems, 2023. URL https: http:////peeeeeeeeeeeeeeee id=cCYvakUSER.
Zhang, Y., Dong, Y., and Kawaguchi, K. Investigating layer importance in large language models. In Be- linkov, Y., Kim, N., Jumelet, J., Mohebbi, H., Mueller, A., and Chen, H. (eds.), Proceedings of the 7th Black- boxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP, pp. 469-479, Miami, Florida, US, November 2024a. Association for Computational Linguis- tics. URL https:tpp///om.... / aaa/aalaantt lanthhooogyyy org/ 2024. b blackboxnlp-1.22
Viswanathan, K., Gardinazzi, Y., Panerai, G., Cazzaniga, A., and Biagetti, M. The geometry of tokens in internal representations of large language models, 2025. URL https://arxixz org/ abs/ 10573.
Zhang, Y., Li, Y., Wang, X., Shen, Q., Plank, B., Bischl, B., Rezaei, M., and Kawaguchi, K. Finercut: Finer- grained interpretable layer pruning for large language models, 2024b. URL https: / /aarraivvv.or.//.............. org/abs/////////o/////////// 2405. 18218.
Xian, L., Adams, H., Topaz, C. M., and Ziegelmeier, I.. Capturing dynamics of time- varying data via topology, 2022. URL https:
//www wwwwwaaimsciences... wtt.arrcceeee/iadloooooooooo //tp:aallccceeeees..orr/rrrrldld 2acaee54 6688-66888888 4055-446a8 2a4-235d-447844440
Zomorodian, A. and Carlsson, G. Computing persistent homology. In Proceedings of the twentieth annual sym- posium on Computational geometry, pp. 347-356, 2004.
Yang, J., Fang, H., Dhesi, J., Yoon, I. H., Bull, J. A., Byme, H. M., Harrington, H. A., and Grindstaff, G. Topological
13

---

Persistent Topological Features in Large Language Models
A. Mathematical Formulation of Zig Zag Persistence
In our approach, the use of intersection layers is essential for computing zigzag persistence, as it allows the construction of injective maps between the ANN complexes of model layers (see (2)) Since our primary goal is to analyze the topological changes between model layers, we eliminate the construction of intersection layers while preserving the topological features by shifting each persistence interval such that the birth and death times occur strictly within the layers.
Zigzag persistence is computational topology method that extends classical persistent homology to handle more com- plex data structures and filtration processes. Unlike standard persistence, which analyzes a single sequence of spaces filtered by inclusion, zigzag persistence allows for the ex- ploration of data where sequences of spaces and maps can move both forward and backward.
For an interval [b, d] in the zigzag persistence diagram of dimension p of filtration 2, the mapping that enables a bi- jective transformation to a new interval [b, d) 13 only across model layers is defined as follows:
A zigzag filtration of topological spaces is sequence:
X: X1
X2
X
where each X, is a topological space and each arrow represents a continuous function pointing forwards X, X;+1 or backwards X; - X,+1-
b b + 1
if b is an intersection layer b otherwise
d++1 1 if dl an intersection layer (13)
If we apply a homology functor Hp with coefficients in
d =
field k to such a filtration, we get zigzag filtration of k-vector spaces, called zigzag module:
d otherwise
The relationship between the persistence image and the ef- fective persistence image for p-dimensional holes, denoted respectively by Plp and PI where b, d are the model lay- ers indexed by even numbers, is described by the following system of equations:
Hp(x): H2 + Hx()))
Hp (%.).. )-
(10)
is proven in (Carlsson & de Silva, 2010) that the algebraic classification of zigzag modules resembles Gabriel's s clas- sification of the persistence module described in (Gabriel, 1972). In particular, every inite-dimensions zigzag module, iee for which all the k-vector spaces in the sequence that are liite-dimenssinal, can be decomposed as a direct sum of interval modules, where a (finitely indexed) interval module a module of the form:
PI PI,(0,0)= = PI,(0,0)
P1p(b/2,d/2)= Plz(b,d) + PI,(b-1,c, (, 1, (PI, - 1)+ PI, 1, 1)
01,) ol) + PIz(b- 1,,
B. Zigzag algorithm
Ib,qq: : 11
12
In
(11)
The zigzag algorithm is schematically described below.
where I; = k for b S i < d, and 1; 0 otherwise, and every arrow of the form k <- k or k -> k is the iden- tity map. Moreover, the list of summands is unique up to reordering.
It exploits two existing public codes that were developed for zigzag computations: DIONYSUS2 (Morozov) and FASTZIGZAG (Dey & Hou, 2022). DIONYSUS2 is C++ library for computing persistent homology, with a specific library for zigzag persistence. In case, it has the role of extracting the filtration f and computing the times array, i.e. the list of layer indices to be associated with the birth and death of features. FASTZIGZAG allows to calculate effi- ciently the persistence diagram Persp ( by converting the imput zigzag filtration a non non-zigzag filtration of an equiv- alent complex with the same length, and then converts the
The zigzag persistence diagram of a filtration x in dimen- sion p is the multiset of intervals [6, dI corresponding to the list of interval summands I(a.4) of Hp(x). In other words,
(x)s = dbb, + H₂(x) = A Ip
JEJ
(12)
Fach interval [b, d] is called persistence interval and is thought of as a persistent homological feature of x that appears at time (referred to as the "birth") ") and disappears at time d (referred to as the "death !!").
12 An alternative method constructing these maps and ob- taining the zigzag persistence diagram is to use a filtration where, instead of intersections, the union of the complexes from two con- secutive layers is considered. However, the Diamond Lemma, as discussed (Carlsson et al., 2009), guarantees that both the intersection- and union-hased filtrations encode the same homo- logical information.
In our setting we say a p-dimensional holes "dies", we mean that the corresponding homology class no longer persists in subse- quent layers. In the zigzag filtration, this happens when the hole is no longer represented hy independent equivalence class in the homology group
"By construction, all resulting intervals contain even numbers, as the model layers are indexed with these numbers.
14

---

Persistent Topological Features in Large Language Models
Algorithm 1 Zigzag algorithm
The visualization demonstrates two key capabilities of zigzag barcodes:
Require: model, dataset, ANN, m
reps- + oxtraceeprrseenattonnsmoodd dataset)
K ++1
Layer-wise tracking: Barcodes reveal when topo- logical features emerge and disappear across network depth.
for i 4-1 to model_getNimLaayes// do
graph < -Wweeeeeteeigbbonrrarrrreeeeeeee ANd))
K
Differential patterns: Different token types exhibit distinct persistence signatures.
end for
Kimt= + commputelttersscttooalvveees/ss
S. liiess +- compute K, Kint)
This toy example illustrates the framework's 's ability visu- alize geometric structure evolution in transformer represen- tations.
4 FastZigZZag(F times)
obtained persistence intervals back to zigzag. The compu-
D. Combining the kNY graph with the Vietoris-Rips complex
tational cost of our algorithm is O(n" * Mayers)) + ) where the first part is the NN graph creation cost for the input dataset each layer, and the second part the the- oretical cost of FastZigZag with w < 2.37286. The algo- rithm performs well even for the relatively large datasets we employ for this analysis: with 10K points embedded in a space with dimension d = 4096, a number of neighbors for the ANN graph of kNN = 10, and a maximum homology dimension of m 10 on an AMD EPYC 7H12 takes approximately hours.
The k-Nearest Neighbors (AxN) complex is built by expand- ing the corresponding KNN graph to fixed dimension m. A key limitation of the ANN complex is that it ranks points by proximity without considering their actual distances. As a result, once & is fixed on each layer, each point is connected to its k-Nearest Neighbors, regardless of the absolute dis- tances involved. In our setting, the number of connected components (the Betti 15 number So) of the ANN complexes as a function of the layers tends to be unity, i.e. the whole complex is connected, even for relatively small values of NNN 2 6. This implies that connected components contain no useful topological information on the internal representa- tions.
C. A Toy Example to Visualize Zigzag Barcodes
To demonstrate the zigzag persistence visualization frame- work, analyze a simple calendar arithmetic task the Mistral model. We use prompts from the form
To address this issue, we follow the approach in (Naitzat et al., 2020), which combines the ANN complex with the Vietoris-Rips complex. Starting from the KNN graph, the idea is to introduce a threshold radius R on each layer and use it to filter out edges of the graph whose lengths are less than or equal to R, and then expand, denoting this new complex VRR This filtering step allows us to focus longer-range connections, uncovering significant topo- logical features that may be hidden by shorter, more local connections.
"Let's do some calendar math. Four months from [MONTH] is"
where [MONTH] cycles through all twelve months. This example was used (Engels et al.) to analyze circular representations in language models.
We extract hidden representations from all 33 transformer l and apply zigzag persistent homology with k = 2 neighbors and maximum simplex dimension 3. The point cloud at each layer comprises 12 tokens corresponding to the 12 different month prompts. We analyze two token positions: month tokens (semantic input) and answer tokens (given by the is token). So the point cloud at each layer comprises of 12 tokens for each prompt
To ensure consistency across layers, we select the radius R in each layer such that the number of connected components, 30, of the ANN complex falls in a pre-determined range. We then compute the observables presented in this work and verify the results. For clarity, we refer to KNN complex the construction used in the main body, and kxx-VE VR complexes the one presented in this section. For the sake of conciseness, we present only results for the inter-layer persistence Z.
Figure 5 shows the analysis for both token types. The per- sistence barcode for month tokens (top panel) exhibits a prominent feature from layers 5- 541, while answer tokens (bottom panel) show persistent structure emerging later (lay- ers 21-32).
In Figure we show the inter-layer persistence of 1-
15 Betti numbers have been used in previous works (Naitzat et al., 2020; Suresh et al., 2023) for interpreting internal representations of neural networks. However, they describe each layer indepen- dently from the others, which is not the purpose of this work.
LLayer 0 is the embedding layer.
15

---

Persistent Topological Features in Large Language Models
Layer
Layer
Layer 13
Layer 20
Layer 27
Layer 32
Sept
May
13
Perseters features
13
1
10
15
70
24
10
Layens
20
32
Dec
P
125
10.0
25-
6.0
Layors
Figure 5. Zigzag persistence analysis of calendar arithmetic tokens. Top panel: Month token representations exhibit persistent topological structure emerging layer and persisting until layer The points are plotted with the and second component PCA. respectively X and Y axis. Bottom panel: Answer token representations show late-energing persistent structure from layers 21-32, with the points plotted on
second component of PCA. For each panel, the upper half displays k--earest neighbor graphs (k = 2) for the prompts across selected transformer layers, while the lower half shows the corresponding 1-dimensional persistence barcodes tracking topological feature evolution across all 33 layers. The red dotted lines in the barcode plots indicate the specific layers visualized in the upper half.
dimensional holes of the æNN and the kxx-Ve VR complexes and the O-dimensional holes of the VR complexes
homology) tendency to retain relational connections among partieles in the middle-late layers of the model.
puted by imposing 3g 500 + 100. 16 We observe all three curves are qualitatively similar. This indicates the stability of the results, even when removing considerable amount of short edges. Moreover, we observe the same behavior also on O-dimensional holes, now that we modified the com- plex such that their statistics are large enough to reliably compute persistence. We argue this indicates a universal (in
E. Consistency of Results
E. Effective Persistent Images across Models
Given a fixed dataset, effective persistent images can be calculated across models and subtracted element-wise to highlight differences in how the models process the same information. We show all the possible comparisons for the models considered in this work in Figure 7, which
''e checked that results are stable as long Bo is much lower than the total number of points.
16

---

Persistent Topological Features in Large Language Models
Jane 20
Awe-VR (2s)
0.00
J 2.0
0.30
Line2 179
Wraen Layer Persidence
Ker-VR (2c)
0.20
0.20
0.0
0.00
0.15
0.10
am
0.05
0.05
Liema 3 8B
0.00
0.6 Relidive Depth
0.8
0.0
0.00
0.2
Reldfe Depth
0.4
0.6
0.0
0.2
0.4
Layer
0.6
0.8
1.0
Figure 8. Births' relative frequency (left) and inter-layer persis- tence (right) as function of the models' depth for larger models, namely Llama 2 13B, Llama 2 70B, and Llama 3 70B, compared to Llama 8B, computed for the SST dataset with weight
Figure 6. Inter-Layer Persistence with weight 0 function of model layers computed for Llama3 8B the SST dataset for both ANN and VR complexes. impose the number of connected components, S0 500 + 100 to build the RNN-VR
complexes.
Llama 8B model and choose a = 0 as weight for both the births' relative frequency and inter-layer persistence. We show results averaged over subsets of size 500 points in Figure We note qualitative similarity for both descrip-
Uemma 15 Limno 3.00
Liano 2 TBNs Maiel
Utmm 13 Mohh
10:1
10-1
10-1
0.07
MT
30
37'
0.05
Det Uhane
Larra 2015. Marral
107
Line 2 Pytha
167
Hizra un. Pytha
UP
10-11
10-1
10-
0.00
0.05
Usme 388
10
15
25
15
1ayer
Layer
-17
-37
LaperBun!
Figure 9. Births' Relative Frequency and Inter-Layer Persistence for weight -Oas function of model layers for Llama 8B for a range of datasets, averaged over 16 subsets of size 500.
Figure 7. Element-wist difference of effective persistence image calculated for Llama 2. Llama 3, Mistral and Pythia the SST dataset. The color bar indicates normalized difference between -1 and 1, logarithmic scale.
tors across all datasets, though quantitative differences can clearly seen especially for inter-layer persistence, In- terestingly, we observe that the code dataset has slight divergent behaviour in middle layers. To investigate this fur- ther, we filter the Code dataset for programming languages with different levels of verbosity (C, Java, HTML, Mark- down, Python) and for each one we extract 10K prompts. We then calculate the inter-layer persistence for a = -1 and a 2 for these datasets so as to highlight separately short- and long-lived features. In this case, we average over 8 sub- sets of size 1000 for computational convenience. We show results in Figure 10. From these calculations, gather that most programming languages generally exhibit drop in the amount of short-lived features in the middle layer, the effect being more visible for verbose codes (java, C). This feature is not seen for other types of datasets. These results deserve further investigation, which we leave future work.
calculate for the SST dataset. We can observe clear patterns in differences across models, reflecting what is observed in Figure 3.
E.2. Larger Models
We verify that our topological descriptors exhibit the same qualitative results for larger models, namely Llama 13B, Llama 2 70B and Llama 70B, using the SST dataset. We show the births' relative frequency and inter-layer persis- tence in Figure 8 in the left and right panels, respectively. As a representative value, we choose a weight of a = 0 for both descriptors, which gives equal weight to short- and long-lived features.
E.3. Varying Datasets
E.4. Results as a Function of Varying Subsets
We test our topological descriptors on different datasets, as presented in Section As a reference, we consider the
Here we show that our topological descriptors show consis- tent results across various subset sizes. While the variance
17

---

Persistent Topological Features in Large Language Models
Layer3
kmn Yytton
Ddb
46
0.85
20003
Intectere
2.2
0.22
2.0034
043
0
0.8
30
20
20
10
20
25
30
203
440
Layer
Lagar
Sunsets Site
Suboeet Size
Figure 10. Inter-Laayer Persistence for weights a = -1 (left panel) and 2 (right panel) function of model layers for I. Ilama 3 8B for a range of programming languages, averaged over subsets of size 1000.
Lager Pututores 25 0.0000
8.004
2.004
9.00M
2.032
increases for smaller subsets, descriptors computer dif- ferent subset sizes are within standard deviation. We show these results for both the births' relative frequency and the
L0030
0,3000
3180e55 SUe
3JEsets Size
inter-layer persistence calculated for the Llama 3 8B model on the SST dataset Figure 11.
Figure 12. Variance of the inter-layer persistence with weight for Llama model computed on the SST dataset as function of subset sizes. We show four different layers: 3, 15, 23 and The black dashed lines represent fitting function.
N--10
Am102
2007
2m220
250
No-10
No-
0.25
1000
305-
effeective persistence, Births' Relative Frequency and Inter- Layer Persistence Llama 2, Mistral, and Pythia, in Figure 13.
0.25
303
302
0.15
201
Liama 338
Uama
Figure If. Births' relative frequency (left) and inter-layer persis- tence (right) with weight a for the Llama 8B model com- puted on the SST dataset for different subset sizes function of model layers.
611
Variance of 2, as function of data points. Given all the subsets with {100, 200. 1000} points, can calculate how the variance of these subsets scales as compared to the size of the subset. As test case, we take the Llama 8B model with the SST dataset and compute the inter- layer persistence weight a over all the subsets. We then plot the variance of each subset subset size for different layers. We choose layers so that they roughly representative of the dynamical phases identified in the main text. We show results in Figure 12, where we overlay fitted curve in black. Apart for the first layers, where the variance grows with the number of points, N, in later layers the relation seems to be approximately 2 x N3/2
function of the
Figure 13. Supplementary plots for Llama 2, Mistral, and Pythia the SST dataset. The first row displays Inter-layer persistence, the second row shows the Births' relative frequency, and the third row presents the effective persistent images.
F. A shuffling test
As a test of our topological descriptors and the phases seen in Section 4, we perform a shuffling of tokens within the prompts of the SST dataset, as a way of destroying the structure and semantic coherence of the prompts, without
E.5. Additional result across models
In this section we present supplementary evidence of the consistency the results across models for our descriptors,
18

---

Persistent Topological Features in Large Language Models
modifying their unigram frequency distribution (see e-g-
5-block sliding window
(Cheng al., 2024) for application of shuffling internal
Liama 8B
representations of transformers).
Hellaswag accuracy
70
Llama 7B
Pythia
In Figures 14 and 15, we show the births' relative frequency and the inter-layer persistence for shuffled and structured prompts. For the former, we can see a clear difference in behavior on the birth of the long-lived features between the shuffled and structured cases, the peak middle layers being higher for shuffled prompts. A reversed trend is seen for the inter-layer persistence. Overall, the frequency of births of short-lived features is not significantly affected by the shuffling, while the inter-layer persistence drops on the second half of the model's depth. These findings deserve further investigations, which we postpone to future work.
60
Mistral
random
50
43
30
20
[_6]
[4. [4_8]
10]
[9_12]
14]
167
181
116 201
118 221
[20_24]
24]
26]
30
12
16
18
[20
[12
120
122
124
126
128
Figure 16. Hellaswag 5-shot benchmark run Llama3 8B,
Llama2 7B, Mistral and Pythia. sliding window size applicd to cut blocks every layers.
3.30
Shiffed 527
0.20
bhalludhh
3.08
2.01
3.00
and I for the left, middle and right panels, respectively. We see that performance is at the level of random choice during the increasing phase and it maximizes close to the maximum Inter-Layer Persistence during the plateau phase. Consis- tently with the Winogrande benchmark, performance right in correspondence with the decreasing phase. For both Llama and Mistral, the relevant layers are few layers before the last. This finding deserves a closer investigation, which we leave for future work.
3.04
0.24
3.02
0.92
3.00
0.00
0-
drop in
Layer
Layer
Figure 14. Births' relative frequency at weight a -1 1 (left) and 2 (right) as a function of model layers for the Ilama 3 8B for shuffled and unshuffled prompts from the SST datasett.
G.2. Layer pruning algorithm
Shafted
Staftee SCT
Here we schematically describe the algorithm for layer prun- ing used to produce results presented in Table 1.
0.22
0.24
Algorithm 2 Pruning algorithm
0.92
0.2
Require: 2,, model, threshold, max +- max(21)
2.2
0:2
30
15 1
20
30
0.00
10
25
for 1 <- Lo modell.el.N um do
layersTooReeroee <-
Layer
Laye
if 2: >> maa * thresholdd then layer ToeesoTRRemoe.pppe (ppendd()
Figure 15. Inter-layer persistence at weight a = -1 (left) and 2 (right) function of model layers for the Llama 3 8B for shuffled and unshuffled prompts from the SST dataset.
end if end for madell.emooeeLagrrssttarrr
G. More on Pruning
Models
Nprone
Our method
Other works
G.I. Sliding Window on Other Benchmarks
Llama
8
[20-27]
[21-28]
Llama 9
[20-28]
[20-28]
We can test the sliding window experiments on the other two benchmarks, MMLU and Hellaswag, show in Figures 17 and 16, respectively.
Mistral
10
[20-29]
[19-28]
Pythia
11
[19-29]
[18-28]
For the case of MMLU, we zoom in on the drop in per- formance seen at the end of the third phase: we show the performance of the MMLU benchmark against block sizes of 5, 3, and 2 adjacent layers with sliding windows of 2,
Table 2. Table with the list of models and the layers that we cut for the pruning experiment. With given Nprune we show the layers cut with our method and for the Angular Distance and the Bi Score (Other works).
19

---

Persistent Topological Features in Large Language Models
Seblock siding window
3-block sliding window
2-block sliding wincow
lama 80
Misrral
random
MM/L accuracy
MILLI accuracy
Liarra
Lama 3 88
Mistral
rancoom
random
(2.01
18.01
18.
[0_12]
(10
14)
(12_16)
[I4 10/
(70.20]
(10. 22)
120. 24)
(22_-e)
(24
201
(26 301
128 (2.
(22_24)
122
251
[24
26]
125.27
126 20]
127 (27_29)
120. (0.30)
129. p0.31]
(30
12622
122 22
Da 291
129 30]
130 331
B3 (2)
Figure MMLU 5-shot benchmark run Llama3 8B, Llama2 and Mistral. The different benchmarks shown are done by cutting
blocks of layers with fixed size and by changing the starting point with sliding window, Left Panel: benchmark made with block size of and sliding windows Middle Panel: benchmark made with block size 3 and sliding windows Right Punel: with block size of and sliding window of 1.
Commensense-02 QA
Model
Gull
Lhis
other
Full
this
other
full
this
other
laama
32.17
28.51
30.00
55.94
38.40
52,83
88.63
84,60
75.80
Bama
50.96
34.02
34.02
73.45
65.33
65.93
85.70
80.94
80.94
Pythin
81.67
60.06
71.43
Table Supplementary 5-shots benchmarks done with the methodology as in Table 1.
G.3. Additional benchmarks
In Table 3, present additional benchmarks beyond those presented in the main text, using the pruning method out- lined in 4.4. Results are overall consistent with what find with benchmarks in the main text.
20


Reviews:
[{"review_id": "0eXPhepjpe", "reviewer": "Reviewer_4Kx7", "summary": "This work introduces a framework for applying the topological descriptor zigzag persistence to analyze the internal representations of large language models (LLM). The experiments are conducted to evaluate the LLM models (Llama2-7B, Llama3-8B, Mistral 7B, and Pythia 6.9B), which demonstrates the effectiveness of the proposed framework in understanding the internal dynamics of LLMs and its practical utility in tasks like layer pruning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reply to 4Kx7\n\nWe thank reviewer 4Kx7 for their useful feedback. While recognizing the novelty of our approach in combining zigzag persistence for LLM interpretability, the reviewer correctly points out that our work should have included a broader discussion of similar TDA approaches. We agree with this comment and recognize it would improve the effectiveness of our work. Here we reply to direct points raised by the referee in this regard, and we would be open to including part of this discussion in an edited version of the manuscript\n\n> Dynamic Persistence [1], related to zigzag persistence, should be added in references and compared with.\n\n\nThe reference is already cited in the main text as “Kim and Memoli, 2021.”\n\n> The integration of the proposed method into filtration, and the tracking of the evolution of internal representations both are desired, which deserve to be appended for further validating the proposed zigzag persistence based method.\n\nGiven the large number of points considered in this analysis, we adopt a statistical approach to study the effect of layer transformations on internal representations, where the identity of individual points is not essential. The idea of tracking the zigzag representatives using [3] could certainly be investigated. Testing this pipeline on an example in a more controlled setting would be useful in validating the proposed zigzag framework and it would allow in principle to track more closely the evolution of features across layers. We deserve these studies for future work.\n\n>The comparison to related TDA approach is insufficient, How does the proposed method compare to dynamic persistence?\n\nHere are a few points of comparison:\n\n1. We choose kNN filtration (whereas it is Rips in [1]) since it is more suitable to high dimensional data, especially for LLM reps. The stability of kNN filtrations is discussed in [4] in the context of persistent homology, we recognize that we missed this reference in the submitted manuscript, we would be open to add it in an edited version.\n2. [1] varies both a time and a scale parameter whereas we fix the scale. Their main summary statistic (the rank invariant) involves calculating a 6-dimensional summary statistic (4 across layers and 2 across scale) which can make analysis hard for both computational reasons as well as finding the informative summary statistic. Nevertheless, varying also the scale parameter is definitely worth investigating in this context, and the techniques in [1] would be a starting point for implementing it.\n3. The work [5] from the same authors is more related to our work since the maximal group diagram and the persistence clustergram described in the above paper are \"annotated\" (with the representative topological features) barcodes. In this work, they fix a scale similar to our case.\n\n>The runtime efficiency evaluation for the proposed method is missing, which is quite desirable to know the efficiency performance of the proposed method.\n\n\nA broad assessment of computational efficiency is present in the current version (Appendix B).  We agree this is an important point to address, thus here are more details on it: the theoretical complexity of FastZigZag is O($m^{\\omega}$) with $\\omega < 2.37286$ for a sequence of $m$ deletion and additions that is an advancement from previous algorithms with a cubic complexity, which allowed us to do our experiments with point clouds of 10k points over ~30 layers. More details on the algorithm are described in the paper by Dey and Hou [2]. \nAs for the knn graph, its creation with a greedy algorithm is O($n^2$) with n being the data points. In total, we have a O($n^2 * N_{layers}$) + O($m^{\\omega}$). We would be open to adding this discussion in an edited version.\n\n\n[1] Kim, Woojin, and Facundo Mémoli. \"Spatiotemporal persistent homology for dynamic metric spaces.\" Discrete & Computational Geometry 66 (2021): 831-875.\n\n[2] Dey, T. K. and Hou, T. Fast Computation of Zigzag Persistence. In Chechik, S., Navarro, G., Rotenberg, E., and Herman, G. (eds.), 30th Annual European Symposium on Algorithms (ESA 2022), volume 244 of Leibniz International Proceedings in Informatics\n\n[3] Dey, Tamal K., Tao Hou, and Dmitriy Morozov. \"A Fast Algorithm for Computing Zigzag Representatives.\" Proceedings of the 2025 Annual ACM-SIAM Symposium on Discrete Algorithms (SODA). Society for Industrial and Applied Mathematics, 2025.\n\n[4] Le, Minh Quang, and Dane Taylor. \"Persistent homology with k-nearest-neighbor filtrations reveals topological convergence of PageRank.\" Foundations of Data Science 7.2 (2025): 536-567.\n\n[5] Kim, Woojin, and Facundo Mémoli. \"Extracting persistent clusters in dynamic data via möbius inversion.\" Discrete & Computational Geometry 71.4 (2024): 1276-1342.", "claims_and_evidence": "The authors provide experiments, statistical descriptors, and visualizations to validate the proposed framework.", "methods_and_evaluation_criteria": "The proposed method does make sense.", "theoretical_claims": "Just briefly check the proofs, no significant issues found.", "experimental_designs_or_analyses": "The effectiveness of zigzag PD and the effective persistence image should be evaluated further.", "relation_to_broader_scientific_literature": "Relation to TDA,  neural network interpretability on large language models, with the focus on on neural network interpretability by introducing topological descriptors that capture the stability and evolution of prompt relations across layers.", "essential_references_not_discussed": "Dynamic Persistence [1], related to zigzag persistence, should be added in references and compared with. \n\n[1] Kim, Woojin, and Facundo Mémoli. \"Spatiotemporal persistent homology for dynamic metric spaces.\" Discrete & Computational Geometry 66 (2021): 831-875.", "other_strengths_and_weaknesses": "Strengths:\nThis work provides a novel approach on combining zigzag persistence, a kind of topological descriptor, with LLM analysis, for tasks such as model interpretability, and the result illustrates the efficacy of the proposed method. \n\n\nWeaknesses:\n1. The integration of the proposed method into filtration, and the tracking of the evolution of internal representations both are desired, which deserve to be appended for further validating the proposed zigzag persistence based method.\n2. The comparison to related TDA approach is insufficient.", "other_comments_or_suggestions": "The runtime efficiency evaluation for the proposed method is missing, which is quite desirable to know the efficiency performance of the proposed method.", "questions_for_authors": "How does the proposed method compare to dynamic persistence?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8oYB8FtwR1", "reviewer": "Reviewer_zXJm", "summary": "The authors introduce the concept of Zigzag persistence from topological data analysis to understand how features evolve through layers. The authors aim to offer a statistical perspective on how prompts are rearranged and their relative positions changed in the representation space, providing insights into the system’s operation as an integrated whole.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Reply to zXjm\n\nWe thank the reviewer for their useful feedback. They highlight the novelty of using zigzag persistence from topological data analysis to understand feature evolution in LLMs, appreciating the accessible presentation. They raise a few concerns, which we address below:\n\n> My biggest question is why the idea of Zigzag is important/interesting to the ML community. I'm not an expert in TDA, but this looks like a niche concept to me\n\nWithin the TDA community, there exist several works addressing time-varying point clouds. The reason for the relatively slower progress in this field has been mainly computational: in a similar way as in multi-persistence, the complexity grows rapidly when adding filtration parameters, and real-world applications are harder to implement.  Nevertheless, recent advances in fast algorithms have allowed these methods to be applied more widely. In the specific case of Zigzag persistence, the fast zigzag algorithm of [1] allowed us to run the pipeline on large and high-dimensional datasets (see reply to 4Kx7 for computational complexity).\nConsequently,  these algorithms have become feasible to apply in the framework of interpreting dynamical changes in internal representations of neural networks.\n\nIn short, we would argue that recent computational advances in analysing time-varying point clouds with TDA approaches have greatly increased relevance for the broader ML community.\n\n> The improvement in the accuracy of the authors' method compared to the pruning method looks marginal\n\nAs the review remarks earlier on, as the work’s main objective is to connect a relatively underexploited mathematical algorithm (zigzag) to LLMs interpretability, the scope of the layer pruning analysis is to show that our method can be used on a downstream task with performances on par with state of the art methods.\n\n> Is it possible to report SE so the interpretation of the results can be more robust?\n\nThere is relatively little variability in the benchmarks used for layer pruning, as it comes from boostrapping. Consequently, our computation reveals a standard deviation of ~1e-4. We note that whenever the accuracy score is the same for our work and the cited references, the results are exactly equal, i.e. the methods suggest pruning the same layers.\n\n>Also the computational cost of the authors' method seems expensive.\n\nWe refer to our reply to 4Kx7 for computational costs of the algorithm. As for the specific comparison to the  layer pruning methods cited in our work, our algorithm is relatively slower, but it contains strictly more information (beyond layer pruning), as it analyzes all layers at the same time, rather than considering pairwise comparisons among layers.\n\n\n\n[1] Dey, T. K. and Hou, T. Fast Computation of Zigzag Persistence. In Chechik, S., Navarro, G., Rotenberg, E., and Herman, G. (eds.), 30th Annual European Symposium on Algorithms (ESA 2022), volume 244 of Leibniz International Proceedings in Informatics", "claims_and_evidence": "The idea of using zigzag filtration to understand the internal representation of LLM is novel, and the authors present the concept in an accessible way to readers.", "methods_and_evaluation_criteria": "This paper focuses more on explaining the idea rather than evaluation. The authors evaluate their methods with four smaller, open-sourced models.", "theoretical_claims": "This paper is heavily dependent on the idea of zigzag in TDA. There's no proof.", "experimental_designs_or_analyses": "The experiment results show that persistent topological features and their similarities remain consistent across various models, layers, and hyperparameter settings within the framework, indicating a level of universality in the topological structure of LLM representations.", "relation_to_broader_scientific_literature": "See my questions for authors.", "essential_references_not_discussed": "NA.", "other_strengths_and_weaknesses": "The strength is the novelty of this work; the authors apply this new idea to understand the consistency and features flow between different layers. The weakness is that the paper is proof-of-concept type, so the experiment result is still limited.", "other_comments_or_suggestions": "NA", "questions_for_authors": "My biggest question is why the idea of Zigzag is important/interesting to the ML community. I'm not an expert in TDA, but this looks like a niche concept to me.\nThe improvement in the accuracy of the authors' method compared to the pruning method looks marginal. Is it possible to report SE so the interpretation of the results can be more robust? Also the computational cost of the authors' method seems expensive.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "MfSDR2zlze", "reviewer": "Reviewer_N5pr", "summary": "The paper tackles the problem of understanding how LLMs work by looking at how layers sequentially transform prompts. Unlike current art that only provides static views of internal representations, the paper uses zig-zag persistence across layers obtained from simplicial complexes built using kNN. Based on their empirical analysis, the authors identify 4 phases of LLM processing. Finally, the paper shows how we can leverage insights from the initial analysis to perform layer pruning, obtaining competitive results with SOTA.\n\n--- \n\n**Post-rebuttal update**\n\nI want to thank the authors for their efforts to address my concerns, including the additional plots/experiments. After reading the other reviews/responses, I decided to keep my initial score.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their insightful feedback. They recognize the novelty of our approach to analyzing LLMs and its importance in enhancing understanding of these models. They also acknowledge a proper review of existing literature. \nAs for feedback on points to improve, we reply element-wise below:\n\n> the shape of the plots in Figure 3 changes significantly with alpha\n\nThe variability in alpha is a feature of the method since it allows to explore different regimes (i.e. short-lived vs long-lived features). This is explained in the paragraphs where phases are described. As a side note, we show that this behavior is consistent across models in App. D.\n\n> the paper only reports Birth's relative frequency plots for variations of Llama (do not consider other LLMs). [...] Also, in Appendix D1, I expected to see the same plots for different datasets and models.\n\nWe have computed these plots, but did not add them to the manuscript. They can be accessed here: https://anonymous.4open.science/r/conferenceProject-019A/src/plots/Rebuttal-plots.md\n\n> [...] it is unclear if this is the case --- e.g., the number of tuples of persistence = 5 seems similar for l=5 and l=25 (or persistence = 10 for l=5 and l=20).\n\nWe agree that this statement is hard to confirm by eye in Fig. 2. We are open to rephrase it as: “Figure 2 shows that features born after the first half of the model’s depth have a higher tendency to be long-lived with respect to features born earlier on.”\n\n> [..] This is less than found in other papers on layer pruning like \"ShortGPT: Layers in Large Language Models are More Redundant Than You Expect\".\n\nWe have run additional benchmarks, results included here:  https://anonymous.4open.science/r/conferenceProject-019A/src/plots/Rebuttal-plots.md.\n\n>how does it decide which simplex dies when two simplices get merged in the subsequent layer? Does it matter?\n\nThe topological feature that dies is uniquely determined by the decomposition of the zigzag module provided in Equations 11-12, removing any ambiguity in its identification. \n\n> [..] the specific form of Eq. (4) is not properly motivated. In addition, why is \"smoothness as a function of layers\" important for the application at hand?\n\nPIs are usually linked to a choice of kernel. In our case, the PD is already amenable to a density grid, the value of density being feature counts. Intersection layers are only necessary for defining the filtration (e.g. one could use the union equivalently). This is desirable since we are interested in studying statistically the LLM layers.\nThe smoothness is related to smoothing formigram (Prop. C.2) in https://arxiv.org/abs/1712.04064 where they smooth barcodes defined on X (all the layers including intersection our case) to barcodes on S(X) (just the LLM layers)\n\n>Overall, the results regarding layer pruning are not impressive. Indeed, since the competitive methods seem simpler (angular distance and bi-score), I am not sure if one would prefer this approach.\n\nSee a reply to a similar feedback in response to reviewer zXjm.\n\nReplies to minor issues:\n\n>the choice for (Gromov et al, 2024) and (Men at al, 2024) is not motivated.\n\nThese methods were chosen as it is easy to compare with ours, since the algorithms require as an input the number of blocks to be removed, which is what our criterion based on inter-layer persistence outputs. \n\n>Based on Fig. 1, the range of (b,d) should be {0, \\dots, 2(N - 1)}.\n\nCorrect, this is a typo caused by Python notation. In fact, there are 2N-1 snapshots.\n\n>Is $l_i =i$ ? in all cases? If so, why use $l_i$?\n\nThe notation $\\ell_i$ was used to be suggestive of the fact that the indices refer to layers.\n\nReplies to “Questions for Authors”:\n\n1. As noted at the beginning of Sec. 4.2, experiments are performed for 0,1,2,3-dim holes, however only 1-dim holes have large number counts so that a statistical study is stable.\n\n2. Yes, the method can be used to analyze models with variable embedding dimension (e.g. CNNs), this is mostly because our method is distance-based and we can expect neighborhoods to shift smoothly in CNNs (see eg https://arxiv.org/pdf/2007.03506)\n\n3. Even though our method allows us to cut non-consecutive layers, our results are in line with the compared methods cutting consecutive layers. If tested on more specialized prompts, our algorithm might cut non-consecutive layers for a low enough threshold (see e.g. Java code in Fig. 9).\n\n4. The key point of our work is to build an interpretable framework that allows studying the internal reps of NNs as a whole system, rather than collecting summaries of snapshots and then combining them a posteriori. This is important in for interpreting different phases of prompt processing, as argued in our work.\n\n5. The computational cost of our method is higher than other methods, as detailed in the reply to 4Kx7. While it is an important worry for scaling up our analysis, for this specific task, it is not an issue given pruning is performed only once.", "claims_and_evidence": "The paper claims a \"fast and scalable pipeline to characterize Trasformer's layers\". In particular, the paper establishes that prompts are transformed according to 4 phases:\n\n- Rapid arrangement of positions\n- Stable, long-lived relations among prompts\n- A transition phase where the model refines these relations\n- Another phase of new rearrangements\n\nI would rank the evidence for this claim as moderate. More specifically, the distinction between the phases is a bit fuzzy and hyper-parameter-dependent — for instance, the shape of the plots in Figure 3 changes significantly with alpha. Also, the paper only reports Birth's relative frequency plots for variations of Llama (do not consider other LLMs). \n\nAlso, Figure 2 aims to show \"that a large amount of 1-dimensional holes are short-lived and that long-lived features appear after the first half of the model\". First, it is unclear if this is the case  --- e.g., the number of tuples of persistence = 5 seems similar for l=5 and l=25 (or persistence = 10 for l=5 and l=20). Also, in Appendix D1, I expected to see the same plots for different datasets and models. However, the paper shows persistence differences for different model combinations, and it is unclear if it should lead to the same conclusion.", "methods_and_evaluation_criteria": "In the first set of experiments, the paper mainly considers variants of Llama. Thus, it is hard to check if the findings apply to different models. The proposed evaluation criteria seem sensible, but since they are related to the proposed descriptor, they are not standard in the literature. Also, the paper's analysis only leverages the last token embeddings as a surrogate for the latent representation of the whole prompt.\n\nFor the experiments on layer pruning, the paper considers two other LLMs (Mistral and Pythia) and three benchmarks. This is less than found in other papers on layer pruning like \"ShortGPT: Layers in Large Language Models are More Redundant Than You Expect\".\n\nFinally, it seems the analysis relies on 1-dim topological descriptors despite the paper mentioning $p$-dim holes.", "theoretical_claims": "The paper does not make theoretical claims.", "experimental_designs_or_analyses": "Please see my comments about the experiments in \"Methods and Evaluation Criterion\" and \"Claims and Evidence\".", "relation_to_broader_scientific_literature": "In Section 4.3, the paper briefly mentions how its findings (the identified four phases) can be related to previous literature/findings. \n\nFor instance, phase 1 has been related to local contextualization (Lad et al, 2024) and increased dimensionality (Valeriani et al, 2023). Phase 2 may be related to the decreasing dimensionality found in (Cheng et al., 2024). Overall, I found the discussion in Section 4.3 shallow. \n\nIt would be good if the authors could provide a summary of new insights and how it advances our understanding of LLMs.", "essential_references_not_discussed": "Overall, the paper does a good job covering related literature. I have no additional recommendations.", "other_strengths_and_weaknesses": "**Strengths**\n\n*Novelty*: As far as I know, this is the first paper to propose using zig-zag PH to track the evolution of internal representations of LLMs and NNs in general.\n\n*Relevance*: The paper tackles the very relevant problem of understanding how LLMs work. \n\n\n**Weaknesses**:\n \n*Presentation*: The authors could provide more details on the zig-zag algorithm in the Appendix. For instance, how does it decide which simplex dies when two simplices get merged in the subsequent layer? Does it matter?\n\nIn addition, the paper introduces a variant of the PI vectorization scheme. However, there is little discussion about the variant. In particular, the specific form of Eq. (4) is not properly motivated. In addition, why is \"smoothness as a function of layers\" important for the application at hand?\n \n*Results on layer pruning*. Overall, the results regarding layer pruning are not impressive. Indeed, since the competitive methods seem simpler (angular distance and bi-score), I am not sure if one would prefer this approach.", "other_comments_or_suggestions": "Minor issues:\n\n- Line 117: the choice for (Gromov et al, 2024) and (Men at al, 2024) is not motivated.\n- Based on Fig. 1, the range of $b,d$ should be {0, \\dots, 2(N - 1)}.\n- Is $l_i=i$ in all cases? If so, why use $l_i$?", "questions_for_authors": "1. Do the experiments only consider 1-dim holes? Why?\n2. Could the proposed approach be used to analyze (or prune) other models (e.g., CNNs)? Would it apply to settings where the embedding dimension changes across layers?\n3. Why do the proposed model and baselines only cut consecutive layers (see Table 2)? \n4. What novel insights does this paper bring to the community? In other words, how does this paper contribute to advancing our understanding of LLMs that were not present in prior works? \n5. How does the computational cost of the proposed method compare to those of (Gromov et al, 2024) and (Men at al, 2024)?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OGaLOKtiSD", "reviewer": "Reviewer_yjVt", "summary": "The authors of this paper explore the applicability of Zigzag filtrations in the Persistent Homology framework for feature extraction in LLM analysis. They propose building filtration on top of simplicial complexes defined by kNN-neighborhoods instead of proximity-induced cliques more commonly used in the persistent homology framework.\n\nAuthors use those filtrations to analyze the evolution of internal representations of a set of texts across different layers of LLM (a text is represented by its last token; thus, a set of texts is a point cloud). They introduce a metric of similarity between layers topology and identify layers that contribute little to the model performance. Finally, this paper covers a method of model pruning by removing blocks of the identified layers.\n\n## update after rebuttal\n\nI think that with the proposed changes, this paper will be an interesting contribution to the research area, although some concerns remain regarding the advantages of the proposed method compared to the \"usual\" simplicial filtrations. I have raised my score to 'Accept'.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their useful feedback. They recognize the novel approach of using zigzag persistence in the context of interpretability of NNs and the clarity of presentation. Additionally, they raise a few points which we address element-wise below:\n\n> Claim of “Identification of Phases of Prompt processing” is too broad - prompts can’t be truly represented with only one token (as it is done in the paper).\n\nWhile we agree with the referee that an analysis of whole sentences would allow for finer-grained information about phases, we chose to simplify the analysis computationally by adopting the common practice in studies of the geometry of representations of looking at the last token only.\n\n> Why was used sliding window of 5, why no other sizes were reported?\n\nFigure 15 shows the analysis for different sliding windows, which we use to zoom in on a phase in which accuracy drops significantly. In other ranges, the size of the window would not change results significantly, if kept small enough.\n\n> Is there any difference (in terms of identified 4 phases) for encoder-only models (in the paper only decoder models are covered)?\n\nWe believe that a difference might indeed be seen for encoder-only models. For those models, considering the last token as a proxy for the whole sentence would not be correct. Along these lines, previous studies of protein language models (see e.g. Valeriani et al. 2023) have considered averaging over tokens. We deserve these studies to future work.\n\n> Only one task of layer pruning was explored. Some other tasks, especially those focusing on topology of individual texts, would be much appreciated.\n\nWe thank the referee for suggesting exploring different downstream tasks. Indeed previous work has considered using geometric quantities (e.g. intrinsic dimension) to identify human text from AI-generated text (Tulchinski et al 2024). An analysis of individual sentences would imply using our algorithm on the tokens of a prompt, i.e. without reducing the whole prompt to a single prompt. Such analysis would be interesting, though quite different from the present work since the manifolds in the two cases would represent rather different dynamics.\n\n> Have you performed your main experiments only with 0-, and 1-dimensional topological features?\n\nWe have performed our calculations for up to 3-dimensional holes. However, as mentioned in section 4.2, 0-,2- and 3-dimensional holes have relatively low number counts, making it hard to draw solid conclusions from them. A brief discussion on which homology dimension contains relevant information is included in Appendix C.\n\n>From what dimensionality topology of kNN-complexes becomes trivial?\n\nWe find non-zero counts up to 3-dimensional holes. This is specific to kNN complexes which are typically more connected as compared to other complex such as e.g. Vietoris-Rips complexes. \n\n\n> Could you please provide for Figures 4 plot of average length of intervals in (classic) persistence diagrams for each of the layers for comparison?\n\nWe apologize with the referee, but we did not understand this question. Are they asking to perform persistent homology at each layer for comparison?", "claims_and_evidence": "Claim of “Identification of Phases of Prompt processing” is too broad - prompts can’t be truly represented with only one token (as it is done in the paper).", "methods_and_evaluation_criteria": "Seems reasonable.", "theoretical_claims": "In this paper no theorems or statements requiring mathematical proofs were made.", "experimental_designs_or_analyses": "1)  Why was used sliding window of 5, why no other sizes were reported?\n\n2) Is there any difference (in terms of identified 4 phases) for encoder-only models (in the paper only decoder models are covered)?", "relation_to_broader_scientific_literature": "Application of the already existing topological method to a new problem.", "essential_references_not_discussed": "I didn’t find any critically important references that were not covered in the literature review.", "other_strengths_and_weaknesses": "Strengths:\n\nS.1. Applications of zigzag filtrations had little coverage in the literature as of now.\n\nS.2. The paper is well-written and easy to follow.\n\nWeaknesses:\n\nW.1. The title feels a little too broad. Different methods of extraction of persistent features from the internal representations of LLMs was covered in lots of previous works, while this paper introduces only one type of new filtration (new features) and explores only one downstream task.", "other_comments_or_suggestions": "1) Only one task of layer pruning was explored. Some other tasks, especially those focusing on topology of individual texts, would be much appreciated.", "questions_for_authors": "1) Have you performed your main experiments only with 0-, and 1-dimensional topological features?\n\n2) From what dimensionality topology of $kNN$-complexes becomes trivial?\n\n3) Could you please provide for Figures 4 plot of average length of intervals in (classic) persistence diagrams for each of the layers for comparison?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper uses zigzag persistence to track overlapping homological features in adjacent layers as data passes through an LLM. The authors analyze various statistics derived from the resulting persistence diagrams to characterize the internal dynamics of LLM as well as apply the resulting observations for pruning. Other than questions about exposition of the relatively obscure concept of zigzag persistence, the reviewers raised concerns about the marginal improvement in the authors' pruning methods, compared to existing techniques. The authors offered a rebuttal that pruning is only meant to illustrate that it's possible to use the ideas in the paper for downstream analysis; the main contribution is in offering a way to use a mathematical technique of zigzag persistence. The reviewers were satisfied with the rebuttal and raised some of their scores.

Please note: The discussion of FastZigzag algorithm in your rebuttal is misleading. 
> We agree this is an important point to address, thus here are more details on it: the theoretical complexity of FastZigZag is $O(m^\omega)$ with $\omega < 2.37286$, for a sequence of $m$ deletion and additions **that is an advancement from previous algorithms with a cubic complexity**, which allowed us to do our experiments [...]. More details on the algorithm are described in the paper by Dey and Hou [2]. [...] We would be open to adding this discussion in an edited version.

This statement is imply not true. Not only has computation of zigzag persistence been known to be in matrix multiplication time [Milosavljevic et al., 2011], the argument in FastZigZag paper for this complexity simply appeals to that same paper. Moreover, the implementation of FastZigZag does not use matrix multiplication time algorithms. That is not the explanation for why it is faster in practice.

All that is to say, please don't add these comments to the paper, as you volunteer in the rebuttal.
Decision: Accept (poster)