Estimating class separability of text embeddings with persistent homology.
Authors: Kostis Gourgoulias
Najah Ghalyan
Maxime Labonne
yash satsangi
Sean Moran
Joseph Sabelja
Venue: Accepted by TMLR ()

Abstract: This paper introduces an unsupervised method to estimate the class separability of text datasets from a topological point of view. Using persistent homology, we demonstrate how tracking the evolution of embedding manifolds during training can inform about class sep- arability. More specifically, we show how this technique can be applied to detect when the training process stops improving the separability of the embeddings. Our results, validated across binary and multi-class text classification tasks, show that the proposed method’s estimates of class separability align with those obtained from supervised methods. This approach offers a novel perspective on monitoring and improving the fine-tuning of sentence transformers for classification tasks, particularly in scenarios where labeled data is scarce. We also discuss how tracking these quantities can provide additional insights into the properties of the trained classifier.

Published in Transactions on Machine Learning Research (06/2024)
Estimating class separability of text embeddings with per- sistent homology
Kostis Gourgoulias JPMorgan Chase Co.
kgourgou@ym.me
Najah Ghalyan JPMorgan Chase Co.
Maxime Labonne JPMorgen Chase Co.
Yash Satsangi MoMorg Chase Co.
Sean Moran JPMorgen Chase Co.
Joseph Sabelja JPMorgen Chase Co.
Reviewed on OpenReview: https: /ttps://openneen opw.reveen..ee.... net/ forum? ITWlya
Abstract
This paper introduces an unsupervised method estimate the class separability of text datasets from topological point of view. Using persistent homology, we demonstrate how tracking the evolution of embedding manifolds during training can inform about class sep- arability. More specifically, show how this technique be applied to detect when the training process stops improving the separability of the embeddings. Our results, validated across binary and multi-class text classification tasks, show that the proposed method's estimates of class separability align with those obtained from supervised methods. This approach offers a novel perspective on monitoring and improving the fine-tuning of sen- tence transformers for classification tasks, particularly in scenarios where labeled data is scarce. We also discuss how tracking these quantilies can provide additional insights into the properties of the trained classifier.
1 Introduction
Pre-trained language models, having undergone extensive preliminary training, have demonstrated remark- able versatility. They have shown the capacity to generalize to new tasks from a mere handful of examples by employing techniques such as in-context few-shot lcarning (Brown et al. 2020), parameter-cffiiiett fine- tuning (PEFT) (Liu et. al. 2022), and pattern-exploiting training (Schick and Schütze 2020a).. Commonly, these methodologies necessitate the use of language models that vary in size from few hundred million Lan
ct al. 2019: Schick and Schütze 2020b) to billions of parameters, exemplified by GPT-3 (Brown et al.) 2020) and TO (Sanh et al.) 2021). Furthermore, these techniques often involve the conversion of a task, such as classification, into a language modeling format, akin to a cloze question.
Text embedding a crucial part of any language model that maps text to numerical feature vectors that can be fed downstream machine learning operations aimed at performing specific tasks, e.g., text classification. Texts with similar meanings need to be mapped to feature vectors closely spaced in the embedding space,

---

Published in Transactions on Machine Learning Research (06/2024)
while texts with largely distinct meanings need to be mapped further away from each other. If the pre-trained language model is used for text classification, metrics capturing class separability can be used to assess how good or bad a text embedding is performing. In this case, a good embedding process would generate an embedding manifold with high class separability so that a downstream classification process would perform well.
In classification problems, class separability metrics capture how easily features distinguish their correspond- ing classes Fuukunaga 2013). The intuition for adopting these metrics for feature ranking is that we expect good features to embed objects of the same class close to each other in the feature space, while objects of different classes are embedded far away from each other. This can mcasure the discriminative power of a feature [Rajoub] 2020). Having those metrics can also assist with capturing the minimum complexity of a decision function for a specific problem, e.g., VC-dimension (Vapnik, 1998). Typically, separability metrics range from model-specific, also known as "wrapper" methods (e.g., Gini importance), to more generic "filter" methods that capture intrinsic properties of the data independently of a classifier (e.g., mutual information). A simple such metric is Fisher's discriminant ratio, which quantifies the linear separability of data by us- ing the mean and standard deviation of each class (Li and Wang 2014). However, this metric comes with strong assumptions of normality. Goometrically-insphe
methods [Greene) 2001 Zighed et al. 2002 Guan and Loew 2022 Gilad- Bachrachh et al. 2004] look at the distances and neighborhoods of features to give a model-independent view of how well-separated the classes are. However, high-dimensional settings and/or large datasets can be challenging for most separability metrics, Most nnformation-theoreee
estimators do not scale well with data size or require training separate models (Belghazi et al.] 2018). Geometric meth- ods are attractive as they are not tied to classifier, but can be informative only for specifie separation regimes (Mthembu and Marwala, 2008), and are impacted by the complexity of computing graph neigh- borhoods and distances between the points. Moreover, common requirement for these class-separability metrics is that they require labeled data, making them mainly limited to the supervised learning setting.
In mathematics, homology is a general way of associating a sequence of algebraic objects, such as abclian groups, with other mathematical objects, such as topological spaces, e.g., data manifolds (Hatcher) 2002). The fundamental groups of topological spaces, introduced by Poincaré (Munkres) 2018), are the first and simplest homotopy groups (Hatcher 2002) and are algebraic invariants that are critically important for characterizing and classifying topological spaces (Massey 1991).
Topological data analysis (TDA) is a mathematical framework that uses topological concepts to analyze and understand complex data sets. TDA is increasingly being used in machine learning to extract information from high-dimensional data that is insensitive to the choice of metric, allowing for more robust analysis. TDA also provides dimensionality reduction and robustness to noise (Carlsson 2009). One of the most important techniques in TDA is persistent homology (PH) (Malott and Wilsey| 2019), which analyzes the topological features of data set across different scales. Persistent homology tracks the birth and death of topological features, such as connected components, loops, voids, and identifies those that persist as the filtration threshold increases Lazar and Ryu (2021). This approach provides more muanced understanding of the underlying structure of the data and allows for clustering and data analysis.
In this context, Barannikov et al. (2022) introduced a topologically-inspiee approach for comparison of neural network representations by introducing R-Cross-Barcode tool, which mcasures the differences in the multi-scale topology of two point clouds. Based on this tool, they proposed a Representation Topology Divergence (RTD) that can measure multi-scale topological dissimilarity between two representations. Tn parallel, Garrido et al. (2023) introduced a method coined RankMe that allows for assessing the performance of Joint- Embedding Self Supervised Learning (JE-SSL) methods providing informative data ions, even on different downstream datasets, without requiring any labels. Their experiments demonstrate RankMe's ability to inform about JE-SSL downstream performances across different methods, e.g. VICReg, SimCLR, DINO, and their variants, and across architectures, e.g. using a projector network and/or /or a nonlinear evaluation method. Furthermore, results show that RankMe can enable hyperparameters cross- rross-validatinn for JE-SSL methods, and is able to retrieve and sometimes surpass most of the performance previously found by manual and/or label-guided search while not employing any labels, on both in-domain and out-of-domain datasets. In the context of persistent homology for characterizing text data, Tulchinskii
et al. 2023) developed an algorithm for estimating the intrinsic dimensionality of natural language text
2

---

Published in Transactions on Machine Learning Research (06/2024)
that can be used to distinguish between human-generated and Al-generated text. Their experiments show that the intrinsic dimension can serve good score for artificial text detection with reduced bias against non-native speakers.
1.1 Related Work
Hajij and Ist.van 2021 framed the classification problem in machine learning by expressing it in topological terms,
Using this topological framework, they showed the circunstances under which the classification problem is achievable in the context of neural networks. While we do not make direct use of this formalism, some of our experiments are motivated by the discussion in this work.
Riock et al. (2019) developed a complexity mcasure for deep neural networks, called "ncural persistence", using algebraic topology. This measure was used as a stopping criterion that shortens the training process while achieving comparable accuracies as early stopping based on validation loss by taking into account the layers and weights of the whole model. PérezzFernández et al. (2021) represented neural networks as abstract simplicial complex, analyzing them using their topological fingerprints via persistent homology. They then described a PH-based representation proposed for characterizing and measuring the similarity of neural networks. Experiments demonstrated the effectiveness of this representation as a descriptor of different architectures in several datascts. While there are similaritics with our work, we do not use persistent homology to compare different models. Instead, we explicitly focus on examples from classification and only use information from the first homology group of the text embeddings, Ho, that get from our model to assess separability. This homology group captures information about the connected components in the data.
Setiérrez-Fandiin et al. (2021) suggested studying the training of neural networks with Algebraic Topology, specifically persistent homology. Using simplicial complex representations of neural networks, they studied the persistent homology diagram distance evolution on the neural network learning process with different architectures and several datasets. Results showed that the persistent homology diagram distance between consecutive neural network states correlates with the validation accuracy, implying that the generalization error of neural network could be intrinsically estimated without any holdout set. While we are also inter- ested in getting some signal about validation performance, our approach uses only the simplest topological information from the embeddings, Ho, instead of explicitly taking into account the weights and connections of the whole model. We also track not just statistics of death times, but also their overall density as epochs progress.
Griffin et al. (023)) showed that the topological structure of training data can have dramatic effect on the ability of deep neural network (DNN) classifier to learn classify data. Previously, Naitzat et al. (2020) highlighted that DNNs tend to simplify the topology of input it gets passed through the DNN's layers. Both of those works are connected to ours through the tracking of changes in the topology of embeddings during training and how that affects the performance of the DNN.
1.2 Contributions
Primary contributions of the paper are summarized below:
An unsuperoised method for sepaassppryyllily eslimation: We use information from the O-homology groups of data manifolds to understand the class-separabilityy of embeddings during model training. Unlike standard supervised techniques, which require labels for computing class separability, the proposed method can estimate class separability without them. In addition, in contrast to metrics that capture clusters and the distance between them, like the Harabasz (CH) Index, we present evidence that this topological view of separability aligns more in terms of behavior with the supervised metrics.
Experimental analysis of pintence-transfommss embeddings on Hx(X) density space: The paper in- volves experimental analysis of embedding manifold evolution over training epochs based on the densities of Ho(X) persistence times. By tracking those densities, we see how the embedding space
3

---

Published in Transactions on Machine Learning Research (06/2024)
is organized during training. We discover that normalizing the embedding layer causes the embed- dings to organize in well-defined connected components during fine-tuning.
Comparison of the behavior of different measures of separability of our experiments, we compare the behavior of various metries of separability when applied to high- dimensional embedding spaces. We find that supervised metrics of separability have similar behavior during training, whereas some unsupervised metries can miss important details like the diminishing return of further training. In the experiments, we also show how the proposed (unsupervised) metric does not have this issue.
nigh-dimensiondl spaces: As part
1.3 Organization
The paper is organized into six sections, including the present section. Section 2 provides a brief intro- duction to homology groups and persistent homology of data manifolds. This background is essential for understanding the proposed method for estimating class separability of datasets presented in Section B along with the baseline metries in Section 3.1 Two sets of experiments are presented in Section covering binary and multi-class cases. Section 5 covers computational costs of the study as well as a discussion on the pro- posed statistic and the behavior of the persistence times during model training. Section G summarizes and concludes the paper along with recommendations for future research.
2 Background on Persistent Homology
We now briefly introduce the essential notation and concepts of persistent homology that we will use this work. For more details, please see the supplementary material or Section from Naitzat et al. (2020).
Suppose we have collection of points X {=1...,.0 ...,, zNN C Rd and norm II.I1 R Rd + R. At high level, persistent homology (PH) concerns itself with identifying the shape and topological features of data manifolds in a way that is robust to noise. To be able to identify those, we can start with a standard construction in PH, the Sietoris-Rippp set at scale e:
VR. = c
vll ( c).
(1)
For two elements to belong to the same a, their e-balls need to intersect. Thus VR,(X) generates a filtration, called the Vietoris-Rips filtration, on the normed space (X, (I-I),,),, where VRa(X) = ((x) : x E X) and &R (X) = #N}].
In PH, the Victoris-Rin Rips set is called abstract simplicial complez. If we interpret every o E VR,(X) describing a relationship between the points x E a, we can construct a geometric realization of VX, by building a graph with vertices the points in X and edges described by the relationships in (X.(X). With the techniques of homology and lincar algebra, we can assign to a simplicial complex K, = (X)(X) a set of groups, called the homology groups, H.,.).. These groups describe topological features, the connected components and k-dimensional holes for 1: S d.
One thing that is special about (X) is that e can be chosen such that the Vietoris-Rips is homotopy- equivalent to the manifold from which X comes from; see Proposition 3.1 in Niyogi al. (2008) for more details on how c can be appropriately chosen to ensure the homotops-cquivalenne property. In other words, descriptions of the topology of the right VR, translate to the topology of the original manifold.
Everything described thus far is standard simplicial homology. However, when applied to noisy real data (represented as point clouds), small differences the distance between points could imply a large difference in terms of the topology. To account for this, PH considers the Vietoris-Rips filtration, {K, e > 0}. As grows, the topology of Ke changes as result of increasing the radius of every ball in Equation ①.. For example, connected components that appeared at scale at scale 6s 62 (death time). PH tracks those changes as e increases from 0 to infinity, providing the birth and death time for topological features as well as their total number at each scale.
E2 (birth time) may merge into one component
We borrow the notation from Wheeler et al. 2021 to work with metric d.
The definition does not necessarily need norm, but can he adjusted
4

---

Published in Transactions on Machine Learning Research (06/2024)
We call the difference between the death time and the birth time of a topological feature the persistence time of the feature. Features with large persistence times tend to be the most topologically important Hensel et al. (2021). In this work, we only concern ourselves with the persistence times of the connected components (i.e., corresponding to Ho) which we get from the ripser Python library (Trllc et al. 2018 All such persistence times are positive numbers.
For the remainder of this manuscript and given dataset X C RR ripser will provide an array of persistence times, Pa, i = .,.... M, one for each connected component discovered. Because persistence times grow with the diameter of X, diam(X)= maX,,y ||z- we normalise them to [0, by dividing with the maximum persistence time (after removing the special persistence time Pas cо at oo). As a consequence, the distribution of the persistence times will be invariant to point-cloud scaling.
3
Class Separability
3.1 Baseline measures of separability
In this section, briefly describe the measures we will as proxies for separability.
ROC-AUC: An estimate of the area under the ROC curve for logistic regression models trained on labeled data.
Accuracy: An estimate of the (balanced) accuracy of logistic regression models trained on labeled data. This is calculated with the scikit-learn "balanced accuracy" function.
Thornton Index (Greene 2001 The Thornton index is the probability that random data point's label is the same as that of each of the nearest neighbors. We estimate it by using every point's ten nearest neighbors.
ROC-AUC, accuracy, and the Thornoon index require an additional set of labeled data, e.g., a validation set, to be estimated. For an unsupervised baseline for separability, we use the -arbsskz- Index.
Dalinski-Harabasss Index (CH) (Caliñski and Harabasz/ 1974): The CH is often used to measure clustering performance, e-g., of k-means. With clusters and N data points, the index is proportional SS B 3///S where SSg is the between-cluster variance and SSw is the within-cluster variance. The larger the CH, the more well-defined the clusters are, and this is unbounded metric the clusters can always be more concentrated around their center. Because of this and to aid in comparisons, we normalize CH by its maximum value in every experiment.
We use CH by first applying k-means clustering with & = 10 to the embeddings in order to assign them to clusters. Then we use the scikit-lcarn function "calinski "calinski_harabasz ftarabasz_scorde to estimate CH.
As will discuss in the next section, we interested in using information from Hn to learn about how separability evolves during training. Ho tracks the existence of connected components so comparison against CH is instructive.
3.2 Using persistent homology capture separability
The proposed method for estimating the class separability of dataset X is solely based on the persistence times of the O-homology group of the data manifold, Ho(X). We argue (and will explore experimentally) that tracking the evolution of the distribution of persistence times can provide information how classification model organizes its embedding space as well as whether we are getting diminishing returns by further training. As mentioned in Section 2 for every point cloud X we compute the persistence times p. remove the +oo persistence time, and then normalize the rest by the maximum persistence time. This procedure makes p invariant to point-cloud scaling.
We will often refer to ttopologically-simple point clouds, e.g., embedding vectors. This is a shorthand for "topologically-simple point clouds with respect to the Ho homology". The simplest such point clouds
We found that CH is very sensitive long as large enough.
5

---

Published in Transactions on Machine Learning Research (06/2024)
noise 0.5
15
noise = 0.05
noise 0.01
Density
10
0
5
-2
0
0
0.0
0.5
1.0
Normalised persistence times
Figure 1: Evolution of the densities of persistence times for & moon dataset (sampled with "sklearn. dattaaanndattttts.mattt _oatee"___oss with different noise parameters. As noise decreases, the points in each cluster approach each other, making most (but not all!) persistence times small (see Equation [ and increasing the confidence that small number of connected components exists.
are ones where connected components can be discerned with high confidence, e.g., two well-separated point clusters; this would correspond to only having few large persistence times (corresponding to the largest connected components) and the vast majority of them close to the smallest persistence time. In the most complex cases, it is hard to discern more than one component and we would expect to see larger spread of persistence times. An example of those behaviors can be seen in Figure
To compare the information from the persistence times with the metrics from Section 3.1 we need to use an appropriate statistic. We first make two assumptions:
Topologically-smmpeer embedding spaces result in casier classification problems: This is a reasonable assumption that supported by similar work referenced the end of Section L.] c.g.. the work of Griffin et al. (2023). Heuristically, if well-defined point clusters exist in the embedding space and each cluster contains mostly examples from one class, then the classification problem should be casier.
2. During classification training, models tend to topologically simplify their embedding space.
The inverse of Assumption is not true in general and depends how define topological-simpliityy Consider the point clouds Xi =(1,,) : (1,..., [i,...,,0]] 10}} and Xo {(0, i (l,..., 10}} and assign labels and 0 to them respectively. Then the (finite) persistence times of the Hg homology for the set Xo U_1 are all oqual to 1 as all points have a ncarest neighbor with that distance (refer to the definition of Victoris-Rips, Equation 1 This indicates a single persistent component, yet the classification problem is linearly separable. This special case is not applicable this work as we will study the evolution of embeddings during training.
Assumption 2 does not always hold (see Section 5), but empirically supported for some cases (Naitzat al. 2020). In practice, can check it by tracking the evolution of the densities of the persistence times during training. We noted experimentally that we can also push the models towards this behavior during fine-tuning by normalizing the embeddings so that they have norm equal to one (see Figure 3 We hypot that this effect is due to normalization and the max-margin convergence during optimization (Soudry et al. 2018 Ji and Telgarsky 2019), essentially making the embeddings more concentrated as training progresses.
Persistence score: To summarize the behavior of the normalized persistence times and compare with the rest of the metrics, we need statistic. S(p), where p denotes the set of normalized persistence times

---

Published in Transactions on Machine Learning Research (06/2024)
of the point cloud X. Given Assumption 2 would like S to be bounded and to increase most persistence times concentrate closer to min (P), indicating more confidence that connected components exist (for intuition see Figure D. A simple statistic of concentration is the expected squared deviation from min (p), E[(p- - (in) (Eīp) - [[ip]=mi(pp))- +Vpp,, where Elp] denotes the sample mcan over the elements of p whereas (p)) denotes the sample variance.
We now study the possible behaviors of this statistic as it converges towards zero, first we set Elp] == (E'pl mm) and ajp] Elp]/Vlp) Then, notice that E[p] + V/p] could converge to 0 while either - - or finally (3) + oo (variance term goes to zero faster). While Case
(1)
(both terms decay to zero at the same rate), or (2) + 0 (expectation term goes to zero faster),
(2) suggest concentration (1) and Case
close to min(p); (p), as the expectation term is controlled, Case (3) does not. For a fixed small > O, we would have E[p] + pp < with Elp] R ō, so we could have concentration away from min(p)- To decrease the influence Case (3) has on the statistic, we can increase the weight of the expectation term, finally defining Sp(p) == (p) -minp))) V(p), where E (0, Sg retains the properties we discussed while placing more weight the distance from min (p). As our purpose is summarize the behavior of p,
S = S in this work. Because we are dealing with normalized persistence limes, the statistic is invariant to changes in the diameter of the point cloud X.
By analyzing persistence times and the persistence score S(p), our objective to gain insights into the structure of the embedding space as models are trained. Given Assumption D we aim to assess the sepa- rability of embeddings from topological perspective and better understand how embeddings evolve during the training process.
4 Experiments
This section includes experimental validation of the main scheme of the paper. Experiments involve a binary and a multi-class text example with pre-trained sentence transformers (ST).
Before each experiment, we split a text classification dataset into a training split, a tracking split, and a validation split. We then fine-tune the ST for text classification with cross-entropy loss on the train split, Once before training and then at the end of every epoch, we embed every element of the tracking split with the updated ST.
How do we estimate the supervised metrics of separability? We use cross-validation to estimate the model-dependent supervised metrics, balanced accuracy, and ROC-AUC, by training logistic-regression models on part of the embedded tracking set and computing the metrics on the remainder. We average over multiple splits reduce the noise of model fitting on those estimates. Thornton's index, CH, and S(p) are caleulated once per epoch the whole tracking set.. After this, we can compare the behavior of the metrics per epoch.
To reduce the dependence on a particular train-tracking-validee validation split, we compute the scores over seven randomly-picked train-trrakking-vwlddttoo splits.
4.1 Binary-Class Text Classification
In this experiment, we start with ST and a set of binary classification datasets.
Datascts:
We use the train splits of SetFit/amazon hatpt//ammccooooottfftfttttet and the SetFit/sst2 datascts from Hugging F The amaavo-coountrfaatuud dataset contains text from English, German, and Japanese, well as label for whether the text is describing a counterfactual or not-counterfactual statement. Only 19% of the text describes counterfactuals. The train split of the sst2 dataset is close to being balanced, with 52.8% of the dataset having "positive" label.
We only the SetFit datasets Tunstall al. 2022) because their column-name standardization, we da the SetFit paradigm this work.
7

---

Published in Transactions on Machine Learning Research (06/2024)
Metric behavior over different datasets and models during line-tuning.
1.0
0.6
Metrics
variable
0.4
Thornton (supervised)
Calinski-Harabasz (unsupervised)
0.2
persistence score
roc auc score
accuracy
0.0
2
3
Finetuning Epochs
Figure 2: Scparability metrics for the binary classification text example. Every line tracks the mcan over all models (all-MmiiM--L6-o0),
and sendemce--ruussjorrrrs// carnaaoon_coutteffutttut and SetFit/sst2 datasets albert-small-22) and datasets (the train splits of /model combination. Accuracy, Thornton, and our from Hugging Face). We use seven splits per dataset/mood persistence-score exhibit similar behavior. Additionally, the increase in the persistence score indicates that the embeddings are organized into more well-defined components. Intervals are 95% confidence bands.
We split each set into a training set with 1000 examples and a tracking set with 1000 examples. At the end of every epoch, embed the examples in the tracking set with each sentence transformer. Although we use those embeddings at the end of the run to gather information about the training process by computing how the various metrics of Section 3.1 changed, in practice those metrics can be computed during the training run.
Models: We consider three pre-trained sentence transformers available on Hugging Face and through the sentenae-transformers Python library (Reimers and Gurevych] 2019 The models are www.r/aasoorreeeeeeaaalaaaaaaaa v2 (TinyBert), and sentence-tranee wwwsstttornssss//raaraaaaaaa hatps//sssssppsssssssessssaaaaaa albert-small-v? (Albert). MiniLM is six-layer net- work outputting normalized 384-dimensional sentence embeddings, whereas TinyBert and Albert both output 768-dimensional unnormalized embeddings. All models produce sentence-embeddingg via averaging over the token embeddings that are produced by the last layer. As per Section 3.2 we normalize the embeddings so that they all have a norm equal to one. For each of the above models, first before we start finetuning and then after every epoch, we create sentence embeddings for each element in the tracking set. Then we use the "ripser" Python library to compute the persistence times. This way we can inspect the change in the organization of the embeddings during finetuning.
(MiniLM), sentence-transformeey paraphrase-TinnBEEE
Finetuning Procedure: We attach a randomly-initialized laeer sigmoid head to construct a binary text classifier that outputs the probability of one of the classes. This a classical configuration when fine- tuming a classification model, such as ALBERT (Lan al.) 2211)). We fine-tune each model on the training splits of each dataset. We use the prodigy optimizer (Mishchenko and Defazio 2023) with d coef le-1, cross-entropy loss, and batch size 32 for all models. Each training run lasts 7 epochs. As we fine-tune on relatively small datasets, we only train the last layer of each ST and the classifier head.
Figure 2 shows the evolution of the various metrics of separability on the tracking sets. The models never see the labels of these sets during training. As we train, the supervised metrics capture a small increase in the separability of the embeddings. The persistence metric exhibits similar behavior indicating this
We the "sentence-trrasffrmees" version "2.4.0" to load the models and encode text into sentence embedding vectors.
8

---

Published in Transactions on Machine Learning Research (06/2024)
Unnormalized embeddings
Normalized embeddings
30
Epoch
Epoch =
Density
Density
20
Epoch =
Epoch
10
Epoch
Epoch =
Epoch =
Epoch
0.0
0.2
0.4
0.6
1.0
0.00
0.25
0.50
Densities of He persistence times
Figure 3: A comparison of the evolution of the normalized persistence times the tracking set the dataset SetFit/am /tta///aoonoootttftttttt for the TinyBert model during fine-tuning. On the right plot, TinyBert includes additional normalization step that ensures the embeddings have a norm equal to one. Otherwise, the setup between the two plots is the same (same splits, optimizer configuration, weight initialization for the ST). The point cloud is organized during fine-tuning so that the connected components are well-soparated on the unit ball (compare with Figure D On the left plot, no normalization is taking place and there are more persistence times close to one, indicating different organization of the embedding space that nevertheless also gets an acceptable balanced accuracy (0.79 vs 0.88 for the normalized case in this example). We remind here that the persistence scores are normalized so they are invariant point-cloud scaling.
slow increase as well. However, because of the topological motivation behind the persistence score, this convergence to larger values also implics that we are approaching a topologically simpler solution to the classification problem. Finally, the CH metric keeps on increasing throughout fine-tuning, showing the qualitative difference between the notion of separability it captures versus the one from the persistence score. In Figure 3 we can the distribution of normalized persistence times for the case of normalized and unnormalized embeddings the SetFit amazoan counterfactual data and the TinyBert model. We note that the normalised case shows behavior that is consistent with our previous Assumption 2
4.2 Multi-Class Text Classification
In this experiment, we consider the behavior of pre-trained sentence transformers during fine-tuning for multi-class classification. The sctup is similar to the binary class experiment in Scction 4.1
Datasets: We use the train split of two datasets: SetFil/emotin and financial phrusebank, both of which can be found on Hugging Face.
The dataset SetFit/Emotee Emotion contains six classes (with approximate corresponding appearance in the set): "joy" (3355)), "sadness" (29.1%), "anger" (13.5%), "fear" (12.1%), "love" (8.1%), and "surprise" (3.6%). The dataset ffnancial_phrusebankk contains three classes: "neutral" (62%), "positive" (25.7%), and "negative" (12.2%). Both datasets contain English text. The financial phrasebank dataset variant we use is labeled according to a 75% agreement. between annotators.
phrusebank
Model: We use the models and the training and tracking setup from Section except now the head of each ST leads to softmax with the probability of each class.
Thornton's index and accuracy continue to have very similar behavior in Figure and Figure Our persistence score also mimics those behaviors and, more importantly, shows the diminishing returns after a few epochs of training. In both cases CH keeps on increasing throughout the epochs, indicating that the clusters keep on becoming better defined.
Figure similar the binary classification example we saw Figure Here too we see that when the embeddings are normalized, the embedding space gets more well-defined connected components in

---

Published in Transactions on Machine Learning Research (06/2024)
Dataset name: SetFit/emotion, all splits and models.
1.0
0.6
Metrics
variable
0.4
Thornton (supervised)
Dalinski-Harabas (unsupervised)
0.2
persistence score
score
accuracy
0.0
0
2
10
12
Finetuning Epochs
Figure The behavior of the various separability metrics on the embeddings as epochs progress. Each line includes 95% confidence interval and is composed of the corresponding metrie seven splits and the three models: all MI--MinnLMLL--- albert-small-o2. The persistence score mimics the behavior of the supervised metrics, including the upward are starting at epoch and subsequent slowdown. The CH score continues to increase linearly throughout the epochs after epoch 4.
Tane--TyyyEERTT---- and hottrcetttrunfooomrs//rrre
Dataset name: financial financial phrassbank, over all splits and models.
1.0
0.8
0.6
Metrics
variable
0.4
Thorton (supervised)
Calinski- Haraki-- (unsupervised)
0.2
persistence score
accuracy
2
Finetuning Epochs
Figure 5: Another example of behavior of metrics through training; the setup is the same as for Figure 4 In this example, we notice saturation in terms separability after epoch the supervised metrics and epoch for the persistence score. The CH score never truly saturates indicating that the clusters continue become more separated in the embedding space, yet any further difference does not make such difference to the rest of the metrics.
10

---

Published in Transactions on Machine Learning Research (06/2024)
Unnormalized embeddings
Normalized embeddings
Epoch
8
Epoch =
Density
Density
6
Epoch =
Epoch
2-
4
Epoch
Epoch =
2
Epoch
0
0.0
0.2
0.4
0.8
1.0
0.0
0.4
0.6
0.8
1.0
Densities of He persistence times
Figure Evolution of the densities of the normalised persistence times of Ho for the multi-class Set- Fit/Enn Emmotion datasct and the TinyBert model. The sctup similar Figure 3 We can see that the behavior is similar to the binary case; the ST with normalized embeddings jumps through distinct states and eventually reduces most persistence times, whereas the persistence time distribution in the unnormalized case does not change that much.
Unnormalised embeddings
Normalised embeddings
10a
Epach 0
10a
Epoch 6
Count
Count
10a
10a
10a
2.5
5.0
7.5
10.0
0.0
0.5
1.0
Figure 7: This is the case described in Figure (dataset: SetFii/EEmotin Emotion, model: TinyBert), but we remove the normalization of the persistence scores; now the persistence scores also affected by the diameter of the embedding set. We have switched the y-axis to log scale to make the tails of each distribution more visible. In the left plot, can see that the distances between some of the connected components become larger from epoch 0 to epoch 6, indicating that the model is partially using the magnitude of the embeddings to separate the classes. We expect that, after a certain point, this increase distance will not make a difference terms of separability; see Figure 5 and Figure In the normalized case on the right, we see the expected collapse of most of the persistence times.
parison to the unnormalized case as finetuning progresses. To provide further insight in this behavior, in Figure 7 we look at the distribution of the original persistence scores, ie., without any normalization. this case, the persistence scores depend also on the diameter of the embedding set and we can see that the unnormalized case the embedding model also adjusting the magnitude of the embeddings to separate the classes.
5 Discussion
Computational complexity of Ho computations: The computational bottleneck of our study is the computation of the persistent homology of the point clouds. Methods and algorithms for computing per- sistent homology is rapidly evolving area, where new algorithms and software implementations are being
11

---

Published in Transactions on Machine Learning Research (06/2024)
updated and released at a rapid pace (Otter et al. 2017 In this paper, we use the ripser library for computing the persistent homology of datasets (Bauer, 2021 Vietoris-Rips complex calculations have a runtime complexity of ((2"), where n is the dataset size, but constant with data dimension, which makes this method preferable for high-dimensional manifolds (Somasundaram et al. 2021 Variants exist that make use of data-reduction techniques lo produce approximations of the persistence times for fraction of the computational cost; a comparison of different methods can be found in the work of Malott et al. (2020). Apart from their computation-saviig advantages, we also think they are interesting because such methods can focus "higher-scale" structure in the point cloud, e.g., between connected components that contain a minimum number of points, while decreasing the effect of lower-scale structure between points. We plan to explore the application of those methods in future work.
Selection of the summary statistic: While we hope that we have shown the usefulness of tracking the distribution of the persistence times, we do not claim that the statistic S()) is optimal when it comes to summarizing persistence time behavior. Its usefulness depends on the assumptions we make Section Our focus was in having way to compare the behavior of the persistence times alongside common separability statistics. In future work we would like to study the sensitivity of the results to this statistic on other tasks, such as text gencration, and better formalize its selection.
Behavior of persistence times during training: While the persistence times of the embedding manifold do not use label information explicitly for their calculation, their behavior depends on the training labels, the model architecture, and the training objective. As we have seen, normalizing the embeddings is one way to control the behavior of the persistence times. With the constraint of normalization, the classifier has to use angles to organize the embeddings for the best-possible separability which leads the clustering see. Further experiments are required to understand the dynamics of persistence times in the case of unnormalized embeddings.
6 Conclusion
This paper proposes using information from the topological characteristics of the data manifold to look the training process from different angle. We showed experimental evidence that the information of the Ho homology group of the embeddings can give valuable information on where we are in the training process: is separability still improving or have we reached a plateau? To be able to do this, we need to expect a certain behavior from the persistence times and we showed evidence that most of those collapse during training when the embeddings are normalized, When this happens, our summarising statistic shows similar behavior with the supervised metrics, but also gives information about the shape of the embedding space (in terms of concentration of the connected components).
Potential topics for future work involve:
Formalizing the selection of the summary statistic and how that depends on the architecture of the model we train.
Expanding the analysis to the densities of persistence times of higher homology groups H, of data manifold and their relations to class-separability of the dataset.
Expanding the study to tasks outside classification: regression, text generation, and model probing.
Disclaimer
This paper was prepared for informational purposes by the Applied Innovation of Al team and the Global Technology Applied Research center, both of JPMorgan Chasc & Co. This paper is not product of the Research Department of JPMorgan Chase & Co. or its affiliates. Neither JPMorgan Chase & Co. nor any of its affiliates makes any explicit or implied representation or warranty and none of them accept any liability in connection with this paper, including, without limitation, with respect to the completeness, accuracy, or reliability of the information contained herein and the potential legal, compliance, tax, or accounting effects thereof. This document is not intended as investment research or investment. advice, as recommendation.
12

---

Published in Transactions on Machine Learning Research (06/2024)
offer, solicitation for the purchase sale of any security, financial instrument, financial product service, or to be used in any way for evaluating the merits of participating in any transaction.
References
Barannikov, S., Trofimov, I., Balabin, N., and Burnaev, E. (2022). Representation topology divergence: A method for comparing neural network representations. In Chaudhuri, K., Jegelka, S., Song, L., Szepesvari, C., Niu, G., and Sabato, S., editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Reseurch, pages 1607-1626. PMLR.
Bauer, U. (2021). Ripser: efficient computation of Vietoris-Rips persistence barcodes. J. Appl. Compul. Topol. 5(3)
Belghazi, M. I., Baratin, A., Rajeshwar, S., Ozair, S., Bengio, Y., Courville, A., and Hjelm, D. (2018). Mutual information neural estimation. In International conference on machine learning, pages 531-540. PMLR.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems, 33:1877 1901.
Caliñski, T. and Harabasz, J. (1974). A dendrite method for cluster analysis. Communications in Statistics- theory and Methods, 3(1):1-27.
Carlsson, G. (2009). Topology and data. Bulletin of The American Mathematical Society BULL AMER MATH SOC, 46:255-300
Fukunaga, K. (2013). Introduction to Statistical Pattern Recognition. Computer science and scientific com- puting. Elsevier Science.
Garrido, Q., Balestricro, R., Najman, L., and Lecun, Y. (2023). Rankme: Assessing the downstream performance of pretrained self-supervised representations by their rank.
Gilad-Bachrach, R., Navot, A., and Tishby, N. (2004). Margin based feature selection-theory and algorithms. In Proceedings of the twenty-firs forsty-firstt international conference on Machine learning, page 43.
Greene, J. (2001). Feature subset selection using thornton's separability index and its applicability a number of sparse proximity-based classifiers. In Proceedings of annuual symposium of the pallern recognition association of South Africa.
Griffin, C., Karn, T., and Apple, B. (2023). Topological structure is predictive of deep neural network success in learning.
Guan, S. and Loew, M. (2022). novel intrinsic measure of data separability. Applied Intelligence, pages 1-17.
Matiérrez-Fandiine A., Feréándzz, D., Armengol-Estapé, J., and Villegas, M. (2021). Persistent ho- mology captures the generalization of neural networks without a validation set.
Hajij, M. and Istvan, K. (2021). Topological deep learning: Classification neural networks.
Hatcher, A. (2002). Algebraic Topology. Algebraic Topology. Cambridge University Press.
Hensel, F., Moor, M., and Ricck, B. (2021). A survey of topological machine lcarning methods. Front. Artif. Intell., 46681108).
Ji, Z. and Telgarsky, M. (2019). The implicit bias of gradient descent on nonseparable data. In Conference Learning Theory, pages
PMLR.
Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., and Soricut, R. (2019). Albert: A lite bert for self-supervised learning of language representations. ArXiu, 11941990.11942.
13

---

Published in Transactions on Machine Learning Research (06/2024)
Lazar, N. and Ryu, H. (2021). The shape of things: Topological data analysis. CHANCE, 34(2):59 64.
Li, C. and Wang, B. (2014). Fisher linear discriminant analysis. CCIS Northeastern University, page
Liu, H., Tam, D., Muqeeth, M., Mohta, J., Huang, T., Bansal, M., and Raffel, C. (2022). Few-shot parameter- efficient fine-tuning is better and cheaper than in-context learning. ArXiv, 0bbs/2205.0505
Malott, N. O., Sens, A. M., and Wilsey, P. A. (2020). Topology preserving data reduction for computing persistent homology. In 2020 IEEE International Conference on Big Data (Big Dala), pages 2681 2690. IEEE.
Malott, N. O. and Wilsey, P. A. (2019). Fast computation of persistent homology with data reduction and data partitioning. In 2019 TEFE International Conference on Big Data (Big Data), pages 880 889.
Massey, W. (1991). A Basic Course in Algebraic Topoloyy. Graduate Texts in Mathematics. Springer New York.
Mishchenko, K. and Delazio, A. (2023). Prodigy: An expeditiously adaptive parameter-free learner. arviv preprint arXiv:2306.066 06101.
Mthembu, L. and Marwala, T. (2008). note on the separability index. arXiv preprint arXiv:0812.1107. Munkres, J. (2018). Elements Of Algebraic Topology- CRC Press.
Naitzat, G., Zhitnikov, A., and Lim, L.-H. (2020). Topology of deep neural networks. The Journal of Machine Learning Research, 21(1::7083-752.
Niyogi, P., Smale, S., and Weinberger, S. (2008). Finding the homology of submanifolds with high confidence from random samples. Discrele & Compulational Geomelry, 39:419-444
Otter, N., Porter, M. A., Tillmann, U., Grindrod, P., and Harrington, H. A. (2017). A roadmap for the computation of persistent homology. EPJ Data Science, 6(1).
Féreadeer,,,dez, D., Gutiérrez-Pandiñoo, A., Baveengo--Estppee J., and Villegas, M. (2021). Characterizing and measuring the similarity of neural networks with persistent homology.
Rajoub, B. (2020). Chapter 2 characterization of biomedical signals: Feature engineering and extraction. Zgallai, W., editor, Biomedical Signal Processing and Artificial Intelligence Healthoare, Developments in Biomedical Engineering and Bioelectronics, pages 29 50. Academic Press.
Reimers, N. and Gurevych, I. (2019). Sentence-bert: Sentence embeddings using siamese bert-networks. Inui, K., Jiang, J., Ng, V., and Wan, X., editors, EMNLP/IJCNLP (1), pages 3980 3990. Association for Computational Linguistics.
Ricck, B. A., Togninalli, Mattco, Bock, Christian, Moor, Michael, Horn, Max, Gumbsch, Thomas, and Borgwardt; Karsten (2019). Neural persistence: A complexity measure for deep neural networks using algebraic topology.
Sanh, V., Webson, A., Raffel, C., Bach, S. H., Sutawika, L.,, Alyafeai, 7., Chaffin, A., Stiegler, A., Scao,
T. L., Raja, A., Dey, M., Bari, M. S., Xu, C., Thakker, U., Sharma, S., Szczechla, E., Kim, T., Chhablani, G., Nayak, N. V., Datta, D., Chang, J., Jiang, M. T.-.., Wang. H., Manica, M., Shen, S., Yong, Z. X., Pandey, H., Bawden, R., Wang, T., Neeraj, T., Rozen, J., Sharma, A., Santilli, A., Févry, T., Fries, A.,
Tochan, R., Biderman, S. R., Gao, I.., Bers, T., Wolf, T., and Rush, A. M. (2021). Multitask prompted training enables zero-shot task generalization. ArXiv, abs/2110.08207..
Schick, T. and Schütze, Н. (2020a). Exploiting cloze-questions for few-shot text classification and natural language inference. In Conference of the European Chapter of the Association for Computational Linguis- tics.
Schick, T. and Schütze, H. (2020b). It's not just size that matters: Small language models are also few-shot learners. ar Xiu preprint or arXi::2009.07 0111009997711.88 07118.
14

---

Published in Transactions on Machine Learning Research (06/2024)
Somasundaram, E. V., Brown, S. E., Litzler, A., Scott, J. G., and Wadhwa, R. R. (2021). Benchmarking packages for calculation of persistent homology. The R journal, 13(1):184-198,
Soudry, D., Hoffer, E., Nacson, M. S., Gunasekar, S., and Srebro, N. (2018). The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 19(1):2822-255) 28788.
Tralie, C., Saul, N., and Bar-On, R. (2018). Ripser-py- lean persistent homology library for python. The Journal of Open Source Software, 3(29).925.
Tulchinskii, E., Kuznetsov, K., Kushnareva, L., Cherniavskii, D., Barannikov, S., Piontkovskaya, I.,
Nikolenko, S., and Burnaev, E. (2023). Intrinsic dimension estimation for robust detection of ai-generated texts.
Tunstall, L., Reimers, N., Jo, U. E. S., Bates, L., Korat, D., Wasserblat, M., and Pereg, O. (2022). Efficient few-shot learning without prompts. Xix preprint orXim:2209. 11055.
Vapnik, V. N. (1998). Statistical Learning Theory. Wilee-Interscience.
Wheeler, M., Bouza, J. J., and Bubenik, P. (2021). Activation landscapes as a topological summary of neural network performance. 2021 IEEE International Conference on Big Data (Big Dala), pages 1865-3870)
Zighed, D. A., Lallich, S., and Muhlenbach, F. (2002). Separability index in supervised learning. In PKDD, volume 2, pages 475 487. Springer.
15


Reviews:
[{"review_id": "uVSL84HTIS", "reviewer": "Reviewer_4hwr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper presents a novel method for tracking the training process of text embeddings without requiring additional labels. This innovative approach leverages information from the H0 homology group of embeddings to assess the progression of training, providing valuable insights into whether separability improves or plateaus.\n\nOne notable weakness is the absence of experimental evidence demonstrating how the proposed method contributes to achieving better performance or guiding model training effectively.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a novel approach for analyzing the training process of text embeddings from a topological perspective.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It is critical to conduct experiments to evaluate the proposed method's impact on model performance in downstream tasks such as classification or regression. This evaluation is essential for demonstrating the practical utility and effectiveness of the approach. Additionally, it would strengthen the paper's contribution and relevance to the field.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "0Jp377QtrP", "reviewer": "Reviewer_oc5T", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper proposes a novel unsupervised approach to assessing separability without labeled data. The promising empirical results showing alignment with supervised metrics are encouraging. The analysis of how embedding normalization influences persistence times and separability is insightful.\n\nThe choice of the summary statistic S(p) appears heuristic and may not generalize optimally. Only using 0-homology information limits richer topological insights from higher groups. The computational complexity of persistent homology is not discussed, which could hinder scalability.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes an unsupervised topological method to estimate the class separability of text data during model training. By tracking the evolution of the 0-homology group of the embedding manifold using persistent homology, the authors argue that monitoring changes in the persistence time distribution provides insights into how well the embedding space separates classes. Experiments on text classification tasks demonstrate alignment between the proposed persistence-based score and supervised separability metrics.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Exploring alternative summary statistics or providing a principled way to select S(p) could strengthen the method. Incorporating higher homology group information may enhance the approach's capabilities. Discussing computational complexity would address practical scalability concerns.\n\nExpanding experiments beyond text classification could further validate the method's generality. More insights into interpreting persistence times and their relationship with separability could deepen the theoretical understanding.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "YHKg4LBj2p", "reviewer": "Reviewer_jAxw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\n\nMeasuring class separability without labels is of interest to many ML communities. This is true for self-supervised learning, where ideally no labels are used (leading to methods like RankMe https://arxiv.org/abs/2210.02885 and LiDAR https://arxiv.org/abs/2312.04000). Labels may also be ambiguous, change with time, or be few in number.\n\nThe work is mostly self-contained, providing a useful guide through the concepts of persistent homology needed for understanding the proposed method and results. Additional references are provided for further technical details.\n\nThe work provides a good statistical analysis of the claims, using thorough cross-validation over many splits, as well as reporting confidence intervals where appropriate.\n\n# Weaknesses\n\nThe layout of the results is challenging to follow, e.g.:\n\n* Figure 1 (page 7) is only mentioned in the caption of Figure 3 (page 8).\n* Figure 2 (page 7) is only mentioned on page 9\n* Figure 3 (page 8) is only mentioned on page 6 and in the caption of Figure 6 (page 10) \n\nFigures and tables should be anchored by in-text references, and should occur as close as possible (preferably on the same page) as those references.\n\nThe paper is unclear which exact object is having the persistent homology being tracked. In Section 3.2 the authors discuss tracking $H_0(X)$ where $X$ is the dataset, as well as under Section 1.2. In the context of their experiments in Section 4 this cannot be the case as $X$ does not change with time. I think the authors mean to track some specific embedding layer(s) of the models they are looking at. For example, the CLS token, or the max or mean pool of the last L layers of the model in equation. The paper would be much improved if the authors included a description of which objects their metric is tracking in Section 4.\n\nI find the chosen definition of $S(p)$ unusual. I agree that the statistic obtains the correct values for $S(p)=0$. My specific question regards adding expectation, min and variance operators together. If we scale the persistence times by a constant, then the current formulation of $S(p)$ places more weight on the $\\textrm{Var}(p)$ term, since $\\textrm{Var}(cp)=c^2\\textrm{Var}(p)$ for some constant c, whereas $\\mathbb E(cp)=c\\mathbb E(p)$ and similarly for min. This feels undesirable. Wouldn’t it make more sense to compute instead $S(p)=1-(\\mathbb E(p) - \\textrm{min}(p) + \\textrm{std}(p))$ which would have a well-defined scaling behavior. Alternatively, why not compute directly the motivated $1-\\mathbb E[(p-\\textrm{min}(p))^2]$ (potentially with a sqrt)?\n\nThe presentation of the figures can be improved.  Figures should be vector graphics where possible. Text size on all figures should more closely match document text size. Captions for variable names should not contain underscores unless they are important. x and y axes should be labeled, and be capitalized if appropriate (e.g. epoch on Figure 2 should be Training Epochs or Finetuning Epochs). Other suggested recablings include Figure 3 x axis should be labelled $S(p)$ and y axis should be labelled Density $S(p)$.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work develops a method for estimating class separability without labels using the tools of persistent homology.\n\nSpecifically, the work:\n\n* Argues that a simpler embedding space results in easier classification problems\n* Measures the persistence time of the $H_0$ group\n* Measures $S(p)$, the concentration around the minimum persistence time, as a summary of the simplicity of the embedding space\n* Observe that the measure $S(p)$ behaves consistently with supervised measures like Thornton (supervised) and accuracy as a function of training.\n\nAdditionally, the authors find that in the case of L2-normalized embeddings, models push the distribution of persistence times to be concentrated around 0 during training, whereas the pattern is less clear without L2-normalization.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Required (critical)\n\n* Restructure paper to colocate figures with in-text references.\n* Explanation in-text for the choice of $S(p)$, addressing the scaling behavior of the terms under $p\\rightarrow cp$, explaining why summing the expectation, min and variance is a reasonable thing to do.\n* Explain what space the persistent homology is acting upon in Section 4 for each model.\n* Improve presentation of figures as described above.\n* (Page 3, first line) Typo on quotes, change persistence\" → persistence''\n* (Page 3, paragraph 1) $H_0$ is used without definition, add definition before this point.\n\n\n# Recommended for strengthening but not critical \n\n* (Section 1.2) “Major contributions” → “Primary contributions”\n* (Section 1.2, first bullet) “We leverage” → “We use”\n* (Section 1.2, second bullet) State the conclusion of the analysis here, not just the existence of analysis\n* (Section 1.2, second bullet) State the conclusion of the analysis here, not just the existence of analysis\n* (Figure 2, page 7) Mention which datasets are used in the caption.\n* (Page 4, under equation 1) I think this is a typo but not 100%, if it is, please change. $\\textrm{VR}_{\\infty}=$ {{ $ x_1,\\ldots,x_N $ }}, i.e. the set containing a single sigma which is the set of all points in X.\n* (Page 4, penultimate paragraph) provide citation for “Features with large persistence times tend to be the most topologically important”.\n* (Page 6) It would be useful to have a \\paragraph{Persistence score $S(p)$} to call out the definition of the score you will use in Section 4.\n* (Page 6) Add a simple clustering figure showing that your $S(p)$ gives the correct ordering in the right scenario. I understand this is the purpose of Figure 1. Since most of the text discusses well-formed clusters, it might be useful to include e.g. 1) random box uniform noise (low separability) and 2) two tight clusters (high separability). Compute $S(p)$ for each case and have them as figure subtitles.\n* (page 8) Add a \\paragraph{Finetuning procedure} to call out where you discuss finetuning procedure (distinct from models).\n* (page 10, Figure 6) Change caption end “that dramatically” → “as much”\n* Remove instances of “etc.” throughout the work and complete the sentences. E.g. Conclusion final point “regression, text generation, etc.” → “for example, regression and text generation”.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper presents an unsupervised, topology-motivated method for estimating the class separability of text datasets. Most reviewers believed that the proposed method was solid, novel, and effective. However, the reviewers also noted that the choice of the summary statistic S(p) appeared heuristic and the lack of complexity and scalability analyses. The AE believes that the paper has enough merits to warrant acceptance, and encourages the authors to address these concerns in the revision.
Decision: Accept (poster)