Metric Space Magnitude for Evaluating the Diversity of Latent Representations
Authors: Katharina Limbeck
Rayna Andreeva
Rik Sarkar
Bastian Rieck
Venue: NeurIPS 2024 poster (2024)
Keywords: diversity evaluation
generative model evaluation
metric space magnitude
geometric machine learning
Primary area: evaluation
TL;DR: We develop novel diversity measures for evaluating latent representations based on metric space magnitude, a novel geometric invariant.

Abstract: The *magnitude* of a metric space is a novel
invariant that provides a measure of the 'effective size' of a space across
multiple scales, while also capturing numerous geometrical properties, such as curvature, density, or entropy.
We develop a family of magnitude-based measures of the intrinsic
diversity of latent representations, formalising a novel notion of
dissimilarity between magnitude functions of finite metric spaces.
Our measures are provably stable under perturbations of the data, can be
efficiently calculated, and enable a rigorous multi-scale characterisation and comparison of
latent representations. 
We show their utility and superior performance across different domains and tasks, including
the automated estimation of diversity,
the detection of mode collapse, and
the evaluation of generative models for text, image, and graph data.

Metric Space Magnitude for Evaluating the Diversity of Latent Representations
Katharina Limbeck 1.2
Rayna Andreeva
Rik Sarkar
Bastian Rieck
Helmholtz Munich
Technical University of Munich
"University of Edinburgh
University of Fribourg
Abstract
The magnitude of a metric space is novel invariant that provides measure of the "effective size' of a space across multiple scales, while also capturing numerous geometrical properties. such as curvature, density, entropy. We develop family of magnitude-based measures of the intrinsic diversity of latent representations, formalising novel notion of dissimilarity between magnitude functions of finite metric spaces. Our measures are provably stable under perturbations of the data, can be efficiently calculated, and enable a rigorous multi-scale characterisation and comparison of latent representations. We show their utility and superior perform- ance across different domains and tasks, including (i) the automated estimation of diversity, (ii) the detection of mode collapse, and (iii) the evaluation of generative models for text, image, and graph data.
Introduction
Diversity is key concept in representation learning, referring the relative abundance and distinct- iveness of model outputs. Given the inherent complexity of deep learning models, the evaluation of diversity is thus crucial, enabling (i) the assessment of the intrinsic richness of latent representations, and (ii) the evaluation of the extent to which models are capable of preserving the properties of an input distribution. While the quantitative evaluation of generative models in particular relies on assessing trade-offs between fidelity and diversity with regards to known reference distribution, reference-free diversity measures are becoming increasingly relevant when ground-truth distribution is unknown intractable. However, reference-basee diversity metrics such as recall are notoriously fallible, sensitive to parameter choices and therefore prone to incorrectly approximate the true data manifold, whereas reference-free free diversity measures often rely on simple mean summaries that fail to pass basic consistency checks [13]. Thus, existing methods lack expressivity to fully capture what means for space to be diverse, resulting in critical need for novel measures that are (i) thcoretically motivated, (ii) robust to noise, and (iii) capable of encoding the intrinsic diversity of data across varying levels of similarity rather than single fixed threshold.
Our contributions. Addressing this need, we propose novel family of diversity measures based on metric space magnitude. a mathematical invariant that captures numerous important multi-scale geometric characteristies of metric spaces, including curvature, density, and entropy of an input space. Metric space magnitude merely requires a notion of dissimilarity between data points, permitting to operate on both local and global scales. Hence, magnitude is poised to compare latent spaces, yielding compact holistic summary of diversity that satisfies relevant theoretical requirements. Our work is the first to (i) introduce magnitude as a general tool for evaluating the diversity of latent representations, and (ii) formalise notion of difference between the magnitude of two spaces across
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

multiple scales of similarity. We demonstrate that magnitude is stable and can detect curvature, highlighting its use as a multi-scale summary of the local and global geometry of data. Moreover, we empirically showcase the utility of our magnitude-based diversity measure across different modalities, namely text, image, and graph embeddings, for which we observe that our measure outperforms alternative seeeddding-baedd measures of intrinsic diversity. Finally, when reference distribution is known, our magnitude-base notion of difference reliably detects mode collapse and mode dropping, thus assisting practitioners in model evaluation and selection.
In nutshell: We propose novel multi-scale diversity measures based on the magnitude of latent representations and show their theoretical and empirical advantages for evaluating the diversity of text, image, and graph embeddings arising from generative models.
2
Related Work
Latent representations and embeddings have become indispensable tools for analysing data types such as images, text, and graphs. As evidenced by LLMs, understanding semantic relationships in data requires meaningful embeddings. Our work focuses on improving representation-basee diversity evaluation and we thus consider the role diversity plays in this context.
Diversity measures. Assessing generative model diversity remains a challenge irrespective of the domain [39], as ground truth reference distributions or labelled data are often unavailable, and human evaluation remains costly. Thus, there exists a need for interpretable, automated and unsupervised measures of intrinsic diversity. Reference-free evaluation is of particular importance for assessing generated text given the black-box bot--ntureeeeeeeeee of LLMs [6], but also applicable across modalities. Motivated by this, varied collection of diversity measures has been proposed, many of which are task-, domain- or -roocl-specific [13]; only a fraction of them are applicable to analysing latent representations specifically. The most flexible methods summarise intrinsic diversity using average pairwise dissimilarities like I," distances or BERT-scores [38]. More recently, Friedman and Dieng [13] proposed the Vendi Score, inspired by principles from theoretical ecology. Other diversity measures are computed directly on embedding spaces, using e.g. the geometric mean of the standard deviation across each embedding dimension [20] or cluster-based measures [10]. However, as we explore in Section 3.1, none of these measures satisfy all theoretical guarantees required by an axiomatic approach to diversity, and they are limited in expressivity, providing only snapshots of diversity at a single fixed resolution. Reference-based metrics define diversity as the extent to which generated samples cover the full variability of the real data [28]. Examples include the Fréchet Inception Distance (FID) or the Inception score (IS). However, they do not exclusively measure diversity but are also concerned with evaluating fidelity, i.e. the assessment of similarity between generated data and real data, making it unclear how single-number summaries such as FID and IS account for each aspect in the trade-off between diversity and quality. Thus, precision and recall have been suggested as more informative summary metrics [32] and seen various improvements [19, 28, 35]. Unfortunately, as Naeem et al. [28] show, even the improved versions of precision and recall fail to satisfy the useful conditions for strong evaluation metrics, such as (i) detecting identical reference and generated distributions, (ii) capturing mode dropping, and (iii) simplicity in selecting hyperparameters. To address these concerns, density and coverage have been proposed [28]. Nevertheless, these metrics still rely on fixed-seale manifold approximations to assess diversity making them sensitive to parameter choices. By contrast, our magnitude-based measures have less stringent assumptions and can be defined in a parameter-free fashion.
Magnitude in machine learning. Since its introduction to measure biological diversity [36]. magnitude was formalised by Leinster [21]. Nevertheless, despite strong geometric properties [22]. magnitude has only rarely been applied in machine learning context. Recent publications started to bridge this gap, linking magnitude to boundary detection [5], edge detection images [1], and the generalisation error of neural networks [3], as well as demonstrating its utility for multi-objective optimisation [18]. However, the full potential of magnitude for measuring diversity remains largely unexplored since existing works ignore the nature of magnitude as an intrinsic multi-scale summary, which captures both local and global geometry and diversity of the data manifold. Our work is thus the first to leverage magnitude as a flexible, multi-scale measure of diversity in latent representations.

---

3 Methods
We first discuss the theoretical properties suitable diversity measure should satisfy and then introduce metric space magnitude. Based on this, we outline our proposed method using magnitude for measuring the diversity of latent representation and its practical implementation.
3.1 Desiderata for Diversity Measures
Given the difficulty in defining diversity, diversity metries never measure diversity itself, but rather quantify related ideas. Entropy-based approaches, including magnitude, in particular share close links to diversity, often favoured in ecology for their computational benefits and agreement with fundamental axioms of diversity [8]. Following this axiomatic approach, we highlight the following key requirements [22]:
Effective size: A dataset with a fixed number of points is more diverse when points are separated e.g. distributed uniformly maximally disordered and becomes less diverse as observations cluster together. Diversity is maximised when points are completely distinct and minimised when all observations identical.
Monotonicity in observations: Including new observation does not decrease diversity. Twin property: Including duplicate observation does change diversity. Mulli-scale: Diversity is summarised across multiple scales of (dis)similarity and thus captures both local and global trends in the data manifold.
This list not conclusive; Appendix C.3 provides a more rigorous discussion of desirable propertics and their definitions. We observe that many diversity measures for evaluating representations in ML do not satisfy these requirements as shown via counterexamples in Appendix C.4. For example, average similarity (AvGSIM), the most frequently-used diversity measure in ML, cannot capture nuances in diversity and fails even in simple toy scenarios [13]. Specifically, it does not give a measure of effective size and does not encode the entropy or disorder of space, which is a key aspect of diversity. Consequently, AVGSIM fails to distinguish that more clustered representation is less diverse than more uniformly sampled space illustrated in Appendix C.5.1. Likewise, the geometric mean of the standard deviations across each embedding dimension [20, GMSTDS] does not measure effective size, and even worse it equals zero whenever an embedding feature constant. Even the Vendi Score [13, VSJ. a more purpose-buit built diversity measure, calculated as the exponential of the Shannon entropy of the eigenvalues of normalised similarity matrix, shows undesirable behaviour under the inclusion of observations. Moreover, neither one of the aforementioned diversity measures fulfil the twin property nor monotonicity observations [22], leading to counter-intuitive behaviour when capturing changes in diversity. For example, an exact repetition of the reference data could be wrongly judged to be more diverse than a model that generates more samples with small but relevant deviations from the reference. Further, we argue that diversity is multi-scale trend that should describe the effective size of a space across multiple levels of (dis)similarity rather than rely on fixed-scale snapshots. Indeed, summarising both the coarse and more granular geometry of the data manifold is necessary to get complete picture of both local and global differences in entropy, clusterability and diversity.
This discussion thus points out glaring need for more principled diversity measures. Addressing this, magnitude functions are particularly promising candidates for improved diversity measures that inherently satisfy all desiderata listed above, as shown in Appendix C.3. Many alternative summaries trivially fulfil number of basic properties of diversity. However, it is non-trivial to satisfy all the desired axioms, making magnitude functions unique their formulation. This axiomatic justification as well as our multi-resolution approach to diversity are the distinguishing characteristics and main advantage of our proposed diversity evaluation methods.
3.2 The Magnitude of a Metric Space
Magnitude is an invariant that measures diversity by deseribing the "effective number of points' of a metric space function of its scaled distances [21].
Definition 3.1 (Magnitude of a metric space). Let X = {a...... ,......,,,,.... S IR be a finite metric space with associated distance metric d. For 1 < i,j n, the similarity matrix of X is calculated as
3

---

5x(1.5) = exp(-d(a;;zy)) If Çx is invertible, the magnitude of X is defined as
Mag(X) = -((xx) 2g
(I)
The existence of magnitude is thus contingent the existence of 5x For negative definite metries d like the I and I,² distance, Sx is invertible [12]. Subsequently, we assume that (X, d) permits the calculation of magnitude; in particular, X must not have any duplicate points. While the magnitude of metric space is expressive even at single scale [5, 21, 23], magnitude unleashes its full potential mulli-scule setting, assigning to a metric space not just a scalar but a function. To this end, we scale the distances in X, effectively viewing the same space through different lenses, or at different "zoom factors,' for example by converting distances from centimetres to metres. Computing the magnitude for all such scales then yields the magnitude function.
Definition 3.2 (Magnitude function). Let (X, d) be a metric space and t. := (X, de) its scaled version with di y) ;= t d(x, for scaling factor E R4. The magnitude function of (X, d) is the function Magx: ++ M..(
For E (0, oo), the magnitude function is defined for all but finitely many values of / [21]. The mag- nitude function also continuous [26, Corollary 5.5] for negative definite metrics." 1 For finite metric spaces, we have lim;- Mag((X) limit behaviour exemplifies to what extent the magnitude function describes the diversity of a space as 'the effective number of points at scale t.' Here, we extend magnitude functions the domain [0, oo) by defining Magx (0) ;= 1.² Intuitively, this extension means that any metric space, when viewed from far away, looks like single point. Notice that neither Definition 3.1 nor Definition 3.2 explicitly require specific properties of a metric (like the triangle inequality) and we find magnitude computable for generalised distance functions, including cosine distances, provided the similarity matrix Sx is invertible. Figure illustrates how magnitude functions measure the effective number of distinct points for toy data, thus describing their diversity. Moreover, it provides overview of our diversity evaluation framework, which we will now introduce.
X n. i.e. the cardinality of X [21, Proposition 2.2.6]. This
3.3 Magnitude for Evaluating Diversity
As multi-scale geometric invariant, magnitude can be extended to evaluate the diversity of latent representations. Here, we are studying a set latent representations X' = X2, where each X; E X is finite subset of some latent space sharing the same notion of distance, e.g. X; C RD Given a latent representation X E X, e.g. a text, image, or graph embedding, we can use the L metric semi-metries like the cosine distance. Based the choice of metric,
or L distance we can interpret Magx (t) as the effective number of points at scale t. In practice, this summarises how diverse points in the space are when observed at said scale factor. This multi-scale behaviour motivates us to propose a simple but expressive summary of a representation 's magnitude function.
Definition 3.3 (Area under the magnitude function, MAGAREA). Let X be metric space whose magnitude function Magx (t) has been evaluated across the interval T = [4o, lem])- We define the area under the magnitude function to be MAGAREA := = Magx (t)dt.
plar
Moreover, we extend this proposed summary to measure the difference in diversity between two representations generated by the same (embedding) model. Notice that distances in these spaces directly comparable and the respective magnitude functions can be compared across the same domain.
Definition 3.4 (Magnitude function difference, MAGDIFF), Let X and Y be two metric spaces that share the same notion of distance. Assume the associated magnitude functions Magx (t) and Magy (t) have been evaluated across the same interval T = [to. feut). We define the magnitude function difference to be MAGDIFF := | (Magx (t) - Magy dt.
Definition 3.3 and Definition 3.4 constitute novel multi-scale approaches for summarising and comparing magnitude functions, leading to theoretically well-founded diversity measures. MAGAREA
'Magx is continuous for t leit, where fent is the supremum finitely many singularities. "This assumes the so- so-called one-point property, i.e. lim Mag (0) = 1, which was shown to hold generically for almost all finite metric spaces [31].

---

imput data reference data
b] automated
diversity evaluation
embeidiing model
a) lateut reprosentations
metric
scale
intrinsic diversity
MagAreaa Magx(t)) dt
c) magaitude
X,Y
compatation
c) difference diversity MagDiff Magx (1) Magg(() dt
b] find I It foyne] automatically hated
d]
MagArea
T3
1220
converpence betsa viour
magnitaule MapxxCO
highhrr
magnitude functions
diversiity
XO
63
magntorde Magx(B
aic
[X]
May DifE
higher
granularity
latent represetttnt
scaling factort
scaling
Overview of our diversity evaluation pipeline. (a) We start with an example of four latent spaces with 200 points, varying in diversity. (b) The magniuude function measures the effective number of points at f, a scale of distance between observations. When the scale factor t almost equals zero, magnitude is close 1, and space effectively looks like one point. For large t, the number of effective points is noticeably higher and magnitude converges towards the cardinality. We find the approximate convergence scale, fconv, at which magnitude almost equals the cardinality, and use it to define the evaluation interval T across which diversity changes most notably. (c) The more diverse the space, the higher the value of its magnitude function. By construction, X1 is more diverse than X2, X3, and Xs, respectively, as we can see from the effective size of each space. We leverage this behaviour define novel multi-scale indicators of diversity. (d) Our proposed measure of intrinsic diversity, MAGAREA, summarises the area under each magnitude function for reference-free diversity evaluation. (e) In reference-based setting, we assess the difference in diversity using MAGDIFF, the area between two magnitude functions.
Figure 1:
measures the cumulative value of magnitude summarising a space's intrinsic diversity while MAGDIFF measures the accumulated difference in diversity between two spaces. As we will later demonstrate in our experiments, integrating the changes in magnitude across a range of scale factors retains the desirable properties of single-scale magnitude, but yields more robust multi-scale summaries of diversity (see Appendix A.2 for investigation of stability perturbations). Furthermore, this comparison terms of the effective number of points across scales remains directly interpretable.
3.4 Practical Usage and Implementation
In order to use our magnitude metric for reference-free and reference-based diversity evaluation, obviate the choice of evaluation interval using knowledge about the convergence behaviour of magnitude functions. As a consequence, our magnitude-base bastd diversity measures do not require manual parameter selection. First, we define a magnitude function's 's convergence scale.
Definition 3.5 (Convergence scale, teonv). Given a magnitude function Mag (t), we define its approximate convergence scale as teonv R, with Mag (fcomv) = |X| for some small e 0. We set < 0.05| in this work.
This convergence scale thus indicates the resolution at which at least 95% of observations recognised by magnitude as being distinct. After reaching this convergence seale, we know that magnitude functions and hence diversity can increase by most based the convergence of magnitude towards the cardinality as illustrated in Figure 1. In practice, however, we find that all relevant changes in diversity happen at smaller scales of distance when individual points are not yet clearly separated. We thus choose the convergence scale defined in Definition 3.5 to be the upper bound of the evaluation interval T to determine the most informative range of scales. We then find the convergence scale using numeric root-finde nott-findin procedures as illustrated in Appendix B.2. When comparing the intrinsic diversity of multiple embeddings without a reference dataset, we compute MAGAREA across T [0, feal and choose feut to equal the median of the convergence scales of the embeddings. Taking the median here provides a stable compromise between the convergence behaviour of all functions. For reference-bused comparisons, we simply calculate MAGDIFF, the difference between the magnitude functions, across T [0, her] where her the convergence scale of the reference embedding. In practice, approximate the integrals in Definition 3.3 and
5

---

Table 1: Magnitude estimates curvature. MAGAREA outperforms more complex meth- ods [41] using single feature.
500
Canatars
130
Method
MSE (J)
200
SVR (selected PH features)
0.27 0.07
Valuna
MAGARRA
SVR (PH vectorisation)
0.17
SVR (all PH features)
0.16
Figure 2: Magnitude detects curvature. Left: Magnitude functions for unit disks with varying curvature between [-2, 2]. Right: MAGAREA exhibits a linear relationship with curvature, in- dicating that it serves as a expressive predictor.
SVR (distance matrices)
0.24 = 0.04
MLP (shallow)
1.15 0.52
MLP (deep)
1.56 = 0.68
MAGAREA (quantile)
0.10 0.05
MAGAREA (piecewise linear)
0.05 0.03
Definition 3.4 via numerical integration across evenly-spaced scales sampled from the evaluation interval T. Choosing the number of scales is a trade-off between accuracy and computational performance as computational costs increase linearly with the number of times magnitude evaluated. In terms of implementations, we also improve the efficiency of magnitude computations using Cholesky decomposition (see Appendix A.5 for more details). Together with our automated scale- selection procedure, we thus overcome the main algorithmic hurdles that hitherto prevented the wider use of magnitude functions. Finally, we implement our methods in a Python package.
3.5 Limitations
MAGDIFF is reference-free measure of intrinsic diversity, but does not measure fidelity. It should therefore not be interpreted in isolation, but jointly with coverage- based metries, for instance. Moreover, while we improve the efficiency of magnitude computations (see Appendix A.5) compared to previous implementations [5], thus making magnitude calculations feasible for practical analyses, novel approximation methods would be required to enable scaling to hundreds of thousands of observations. Finally, we focus on evaluating representation-basee diversity and show that, given latent representation, magnitude yields better notion of diversity than current embedding- beasddddee methods. We do not investigate whether uubedding-basee similarities are outperformed by alternative task- or domain-specific similarities. Instead, our evaluation relies on the utility of embedding models and assumes that latent spaces encode useful/realistic relationships between samples.
Experiments
Our experiments demonstrate how magnitude leads to a better understanding of representational diversity. We show the following results: (i) Magnitude functions capture the curvature of a space. (ii) Magnitude functions are interpretable measures of the intrinsic diversity of embeddings, yielding superior results than other diversity measures when predicting the diversity of sentence embeddings across different text-generation tasks. (iii) Magnitude functions characterise and distinguish latent representations of large language models. (iv) Magnitude functions successfully detect mode dropping in distributions image, and graph embeddings, while also reliably detecting mode collapse in graph embeddings. We subsequently use MAG AREA in reference- free settings to characterise intrinsic diversity (i, ii), while using MAGDIFF for reference-bart based comparisons (iii, iv).
4.1 Magnitude Functions Summarise Geometry
Magnitude functions encode the 'shape,' i.e. the geometry that is characteristic of the intrinsic data manifold, by capturing curvature and diversity. Curvature estimation important task in numerous domains like computer vision, computational geometry. and computer-aided aided design. The notion of curvature is inherently linked to diversity: The more positively curved space is, the lower its diversity as points on the more curved surface move closer and closer together, thus decreasing its diversity. For specific examples of manifolds, magnitude can be expressed terms of volume and total scalar curvature [44], theoretical connection that we are the first to investigate empirically for
Thhe code for computing magnitude is available at https: : /ttpt/gtttttttt/////e/ commaiios--- lab/magnipy.

---

prompt
repp
story
diverity
roberta base mean tokers
MAnAREA
all mpnet base 12
AvtaStu
GMSrDS
al-Meil
Figure 3: MAGAREA outperforms alternative diversity measures at predicting the ground truth- diversity of generated sentences, controlled by the softmax- eetmaax-emmpertuure across tasks and embed- ding models. Baseline measures, AVGSIM and GMSTDS, perform worse in terms of the R²² scores. Points show the mean of the Ii2 scores, while lines represent the standard deviations across 5-fold cross-validation (repeated 10 times).
broader class of spaces. Previous works have shown that alternative multi-scale methods, such as persistent homology, are able to detect curvature [4, 41]. Here, we demonstrate that the magnitude function is capable of achieving comparable performance, using simpler methods and only a single feature, namely MAG ARBA. To this end, we generate a balanced dataset of point clouds of different curvature (following Turkes et al. [41] and detailed in Appendix D.I). We first assess what extent the magnitude function can detect whether unit disk has positive or negative curvature. Our main observation from plotting the functions for both groups Figure is that there is a clear separation between spaces of negative and positive curvature. We further test if we can predict curvature as regression task. To this end, we try both piecewise linear and quantile regression." 4 using the area under the magnitude curve, MAGAREA, as single feature. With 5-fold cross validation, we achieve an MSE of 0.05 + 0.03 with the piecewise linear model and 0.10 + 0.05 using quantile regression. Both scores substantially improve on previous methods [41] that made use of highly-sohhisticated topology-basse features and more
deep learning models (see Table 1). These results underscore the expressivity and power of magnitude-based metrics, which enable us solve the same task with a highly-simpliffed model. Moreover, this also demonstrates how magnitude describes the data manifold across multiple resolutions, motivating the use of magnitude functions as flexible, geometry-ave aware descriptors of diversity.
4.2 Magnitude Measures the Intrinsic Diversity of Text Embeddings
Next, we demonstrate the utility of us- ing magnitude for intrinsic diversity evalu- ation and study its correspondence to known ground-truth diversity of text data. We ana- lyse data from Tevet and Berant [38], con- sisting of IK sets of 10 sentences each, gen- erated for unique input prompts for 3 differ- sentence generation tasks, namely story completion (story), dialogue response gen- eration (resp), and 3-word prompt comple- tion (prompt). Per task, 10 response sets have been generated using the same decod- dec,
10.0
(agx(()
7.5
MA4ARE
255
60
p: [0.83, 0.86]
dec
Figure 4: MAGAREA correlates well with dec in- dicating the true diversity. Here, mpnet em- beddings for the resp dataset. p denotes the rank correlation between MAGAREA and dec (95% boot- strap interval, 1000 resamples).
ing parameter, the poftmax-temperatuur which controls the diversity and randomness of the generated text. As dec decreases, models are skewed towards avoiding low-probability tokens. This leads to potentially higher quality and fidelity but lower diversity and creativity in generated text. We embed each set of responses using 5 pre-trained sentence transformer mod- els [30], i.e. (1) peert-aarge-niie -ww-sttsbbb-teeeeeee (3) all 1ell-mpne--be saaneet-bae-vee v2, (4) all- www--iiiisrrretteeeeee
tokens, wea---sss-mmeaa--eeeeee - (2) a11-MiniIM-L12-v2 For each and (5)
dataset and model, we compute the area under the magnitude function MAGAREA, evaluated until the
'Both models were chosen after explanatory analysis to offer multiple proposals on how to interpolate between the MAGAREA scores for surfaces of negative and positive curvature. The piecewise linear model better fits the trend in Figure 2, which why outperforms the quadratic relationship modelled via quantile regression.
7

---

median convergence scale across all embeddings as detailed in Section 3.4 using cosine distances. We compare this to the Vendi Score (VS), AVGSIM, and GMSTDS, calculated using cosine similarities. Moreover, we analyse the performance of each diversity metric at predicting the ground-truth diversity scores, dec, using 5-fold coossvvaliationn repeated 20 times, trained via isotonic regression models; and report their performance in terms of the coefficient of determination, R Figure 4 depicts the positive rank correlation between magnitude and the softmax-temperne while Figure 3 shows results concerning the predictive performance of different diversity measures.
for one example setting,
We observe that MAGAREA consistently outperforms alternative diversity measures computed from the same representations. MAGAREA achieves a median rank of 1 across experiments in terms of R² scores, followed by Vs, AVGSIM and GMSTDS. Indeed, MAGAREA most frequently the best- rest-performing diversity measure for 77% of resamples when predicting decoding parameters, ranking second in the remaining cases. Meanwhile, VS most often achieves second place. This and entropy- based measure
demonstrates the strength of MAGAREA as of intrinsic diversity. By contrast, the baseline measure GMSTDS fails for any embedding that has at least one constant dimension, even reaching negative Rta values for three of the five embedding models. This is followed by AVGSIM, which, while being less fallible than GMSTDS, simply measures average similarity and even ranks last across 27% of resamples. A further comparison of performance scores shows that MAGAREA outperforms AVGSIM by 0.12 higher mean R² scores on story and 0.07 on resp or prompt across embedding models. We find no dataset for which either AVGSIM or GMSTDS can be considered preferable predictors of the ground truth diversity of text. Our results thus show the benefits of replacing simple summaries the current standard for automated diversity evaluation with more sophisticated diversity measures like MAGAREA.
4.3 Magnitude Distinguishes and Characterises Embedding Models
Motivated by the capability of magnitude functions to encode representations, now check whether the embedding spaces of different large language models can be distinguished via their intrinsic structure. To this end, analyse 16384 documents of four different HuggingFace datasets, as embedded by Wayland et al. [43] using six different models (see Appendix D.3 for more details). We then either use PCA and normalisation reduce each embedding space to 384 dimensions (to obtain a comparable dimensionality) or use the original embeddings without preprocessing. Further we subsample 300 documents random from each space, repeating this procedure 200 times. Finally we use a 5-NN classifier to predict the embedding model based on the values of each diversity measure. This task chosen to assess whether a simple classifier can distinguish embedding spaces solely based on their intrinsic diversity estimates. Table 2 reports the results of 5-fold cross-validation with 20 repetitions for both prepossessing choices. We either use Euclidean distances between single number summaries or, in the case of magnitude, use MAGDIFF directly as precomputed input distances for k-NN classification. We first observe that MAGDIFF best predicts the embedding model (with accuracies typically above 90%). Supplementary results in Table S.5 verify that these performance scores are almost identical for varying hyperparameter choices of l neighbours. Surprisingly, the results further remain consistent for both pre-processing choices. This indicates that there are inherent differences in the structure and diversity of embedding spaces, which are preserved throughout dimensionality reduction and captured by magnitude. By using the difference between magnitude functions as a holistic summary, we once again surpass other summary statistics (which we observe to fail in distinguishing the smaller embedding models). Our results thus demonstrate that using MAGDIE DIFF for comparing latent spaces across multiple scales is considerably more expressive than using single-number summaries of diversity.
4.4 Magnitude Evaluates Image Embeddings
Mode dropping is a common issue in generative modelling, referring to the inability of model to capture all parts of an input distribution (for instance, model trained to generate images of animals suffers from mode dropping if it can only generate images of dogs). To simulate this, we randomly sample 100 images from each of the 10 classes in CIFAR 10 and embed them using pre-trained Inception V3 model [37]. Subsequently, we re-sample increasingly more observations from one preferred image class. We either drop modes sequentially, we move the same number of observations simultancously from all other classes. Thus, diversity decreases gradually with the same
'ee use these models capture the non-lineat monotonic relationship between dec and diversity.
8

---

Table Magnitude characterises text embedding models. We show the accuracy (f) of different diversity scores for distinguishing between six embedding models, using 5-NN classifier.
No pre- pre-processing
PCA prepprocssiing
Method
MAGDIIF
AVGSIM
VS
GMSTDS
MAGDIIT
AVGSIM
VS
GMSTDS
Dataset
cnn
0.94 = 0,02 0.87 = 0.01 0.63 0.01 0.66 = 0.02 = 0.88 + 0.02 0.67 0.03 0.66 + 0.03
patents
0.99 = 0.01 0.92 # 0.01 0.63 0.02
0.91 # 0.02 0.64 0.03 0.66 + 0.03 0.66 + 0.02 =.,01
arXiv
0.99 0.01 0.89 a: 0.01 0.01 0.66 e 0.02 0.99 0.01 0.88 4: 0.02 0.78 + 0.02 0.66 a: 0.03
bbc
0.98 0.01 0.74 0.01 0.84 = 0.02 0.66 + 0.02 0.95 0.01 0.73
0.03 0.84 a 0.02 0.66 + 0.03
"speed" across both procedures, but fidelity should not change. We treat each class as the preferred image class twice, leading to 20 re-samples per mode dropping scenario [28]. Our analysis compares the changes in recall and coverage, setting the number of nearest neighbours to k = 10. Further, we calculate the relative change in Mag(0. (.. i.e. magnitude computed at half the convergence scale of the reference using Euclidean distances. Similarly, MAGDIFF is the difference between the magnitude functions relative to the area under the reference magnitude function.
Figure shows the changes in diversity as modes being dropped. Ideally, every diversity measure should show the same decrease in diversity, irre-
0.6
MAGDIFF sim sinuttrreuss
spective of resampling strategy. How- ever, we observe that both recall and coverage wrongly assess that diversity decreases faster during sequential res- ampling. Even worse, coverage only de- tects simultaneous mode dropping after around 70% of all points have shifted to one mode. This undesirable behaviour of both metrics is caused by their reli- ance on a fixed neighbourhood size for approximating the underlying manifold, thus overestimating the extent to which the perturbed samples reflect the diversity of the reference distribution. In comparison, MAGDIFF as well as magnitude evaluated at single scale both successfully measure the gradual decrease in diversity across both mode dropping scenarios.
sequential
D.5
eaquential
simultanozss
simultarauus
sequentis
eaquential
0L6
D.T
0.8
Degine portubation
Degree pertubation
Figure 5: Magnitude correctly detects that diversity decreases in the same manner across simultaneous and sequential mode dropping outperforming recall and coverage. Lines show the mean values of each metric across 20 resamples, shaded areas the standard deviations.
4.5 Magnitude Evaluates Graph Generative Models
Diversity evaluation in graph learning is fraught with difficulties, in particular when aiming to detect common problems like mode collapse or mode dropping [29, 40]. In the following, we will study graph generative models (GGMs), which take set of input graphs and generate new samples that should follow the same distribution. The question that we aim to answer here whether our proposed magnitude-base
metric is more expressive in capturing the diversity of the generated graphs than classical metrics like maximum mean discrepancy (MMD) and measures inspired from evaluating image generative models (precision, recall, coverage, density). To this end, we analyse synthetic (Lobster, Grid, and Community) and real-world (Proteins and Ego) graph datasets, and compute commonly-used evaluation metrics [29, 40] as detailed in Appendix D.5. To test the diversity of generated samples, we replicate the experimental setup of Thompson et al. [40] and add own measure, MAGDIFF computed using L distances from Graph Isomorphism Network [45, GINJ embeddings with varying hyperparameters. For the mode collapse experiments, we substitute each embedded graph with its cluster centre. Thus, the degree of perturbation p equals the proportion of clusters collapsed in this manner. The larger the value of p. the more clusters have been perturbed decreasing the diversity. For the mode dropping experiments, we remove clusters, and keep the size of the generated dataset the same as the reference by randomly resampling from the remaining classes.
Figure 6 shows the results of both mode collupse and mode dropping for the Lobster dataset. We observe similar trends across all datasets, but have chosen this dataset as running example. Ideal measures should exhibit high rank correlation to the degree of perturbation, indicating that they
9

---

Mlde celigta
Coopiigg
10
Preme
Meree
85
20
30
22
Figure 6: MAGDIFF outperforms existing graph diversity metrics at detecting mode collapse and mode dropping. We report the Spearman correlation between each metric and the degree of perturbation for the Lobster dataset (the same pattern holds for Proteins, Community, Ego, Grid, see Appendix D.5). Violin and box plots show the distributions across different hyperparameter choices. Measures that capture the decrease in diversity accurately should increase function of p. Rank correlation of corresponds to an ideal metric. Our metric best captures the changes diversity for both mode dropping and collapse.
are capable of capturing the decrease in diversity properly, i.c. as function of p. We note that in contrast our magnitude-based metric, recall and coverage exhibit worse results, as evidenced by their lower mean correlation coefficient. Despite being specifically designed to measure the diversity of dataset [40], they only catch up our magnitude metric when the degree of perturbation p around 0.9 (see Figure 6, right-hand hand plots). Magnitude dominates in the majority of the values of p best showing the steady decrease diversity. while recall and coverage become more sensitive for exceedingly large values of p. i.e. in unrealistic situations where most of the modes have been dropped. Moreover, their performance highly contingent on the parameter used to construct k-NN graph for computing these neighhourhood- metrics. Magnitude functions meanwhile give more holistic summaries of both local and global patterns diversity. Please refer to Figure for the aggregated results over all datasets, which exhibit a similar pattern (in that our metric outperforms both recall and coverage).
Discussion
We have proposed novel diversity measures for evaluating latent representations. Our measures based on metric space magnitude, multi-scale invariant summarising geometrical characteristics of the input data. We have demonstrated axiomatically and empirically that our magnitude-basee measures are superior to current baseline measures of intrinsic diversity. In a reference-free scenario, observe that magnitude outperforms alternative measures when predicting the ground truth diversity for text embeddings. Given reference dataset, we find that magnitude captures mode collapse and mode dropping better than existing metrics for evaluating generative models for both image and graph modalities. Furthermore, we have shown that magnitude can measure the intrinsic curvature of input data, outperforming previous methods. Magnitude thus gives provably stable, unsupervised diversity metric that can be computed efficiently and allows users to flexibly choose a notion of dissimilarity. For future work, we believe that magnitude exhibits a strong potential for applications to unaligned spaces with varying notions of distances. Moreover, we believe that integrating magnitude into deep learning models would be beneficial for obtaining novel diversity- and seometry-basee regularisation strategies.
10

---

Acknowledgements
The authors are grateful for the stimulating discussions with the anonymous reviewers and the area chair, who believed in the merits of this work. We further want to thank Dr. rer. nat. Dr. iur. Corinna Coupette, Emily Simons, and Jeremy Wayland for their help in proofreading the paper. K.L. is supported by the Helmholtz Association under the joint research school 'Munich School for Data Science (MUDS).
Funding Disclosure
K.L. gratefully acknowledges support from the 2024 NeurIPS Financial Assistance Program. R.A. was supported by (i) the United Kingdom Research and Innovation (grant (P/S02431X/I) UKRI Centre for Doctoral Training in Biomedical AI the University Edinburgh. School of Informatics, (ii) the International Helmholtz--Eiinburgh Research School for Epigenetics (BpiCrossBorders)) and (iii) a Helmholtz Visiting Researcher Grant. B.R. was partially supported by the Bavarian state government with funds from the Hightech Agenda Bavaria. This work has received funding from the Swiss State Secretariat for Education, Research, and Innovation (SERI). The authors declare no competing interests. The funders had no role in the preparation of the manuscript the decision to publish.
Impact Statement
This paper presents work whose goal is advance the evaluation diversity in representation learning. leading to increased fairness and trustworthiness in model evaluation. While representational diversity terms of model outputs may have potential negative impacts, depending on the task hand, we feel there are none that need to be specifically highlighted here. However, we acknowledge the potential for societal harm if our notion of representational diversity is confused with the meaning diversity in the colloquial or societal context, which is admittedly even harder to measure and requires larger discussion involving all affected communities.
References
[1] M. F. Adamer, E. De Brouwer, L. O'Bray, and B. Rieck. The magnitude vector of images. Journal of Applied and Computational Topology, 6()):447447. 2024. doi: 10. 10.1007/s41463 024-00182-9.
[2] G. Alefeld, F. A. Potra, and Y. Shi. Algorithm 748: Enclosing zeros of continuous func- tions. ACM Transactions on Mathematical Software (TOMS), 21(3):327-344, 1995. doi:
10.1145/210089.21011
[3] R. Andreeva, K. Limbeck, B. Rieck, and R. Sarkar. Metric space magnitude and generalisation in neural networks. In Proceedings of 2nd Annual Workshop on Topology, Algebra, and Geometry Machine Learning (TAG-ML), volume 221, pages 242 253,, 2023.
[4] Bubenik, M. Hull, D. Patel, and B. Whittle. Persistent homology detects curvature. Inverse Problems, 36(2):225008, 2020. doi: 10. 10.1088//361-6642/ab44444
[5] E. Bunch, Kline, Dickinson, Bhat, and G. Fung. Weighting vectors for machine learning:
numerical harmonic analysis applied to boundary detection. arXiv preprint 27X.v:2106.00827, 2021. doi: 10.48550/arxiv.2 106.00887..
[6] A. Celikyilmaz, E. Clark, and J. Gao. Evaluation of text generation: A survey, arXiv preprint arXiv:2006.1 14799, 2020. doi: 20.488550arxxi.2000.47777 14799.
[7] H. Dai, A. Nazi, Li, B. Dai, and D. Schuurmans. Scalable deep generative modeling for sparse graphs. In International Conference Machine Learning, pages 2302-2312, 2020.
[8] A. Daly, Baetens, and B. De Baets. Ecological diversity: Measuring the unmeasurable. Mathematics, 6(7):119, 2018, doi: 10.3390/ath60001119

---

[9] P. D. Dobson and A. J. Doig. Distinguishing enzyme structures from non enzymes without alignments. Journal of Molecular Biology. 330(4):771-783, 2003. doi: 10.1016/S0022- 2836(03))0628-4.
[10] W. Du and A. W. Black. Boosting dialog response generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 38-43, 2019, doi:
1118653/1//11910000
[11] P. Erdös, A. Rényi, et al. On the evolution of random graphs Publication of the Mathematical Institute of the Hungarian Academy of Science, 5(1):17-60, 1960.
[12] A. Feragen, F. Lauze, and S. Hauberg. Geodesic exponential kernels: When curvature and linearity conflict. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 1082-3042, 2015. doi: 10.1 10.100//VPR.22115.22989
[13] D. Friedman and A. B. Dieng. The Vendi Score: diversity evaluation metric for machine learning. Transactions on Machine Learning Research, 2023.
[14] D. Gove and R. Hepworth. Persistent magnitude. Journal of Pure and Applied Algebra, 225(3): 106517, 2021. doi: 10. 10.1016//.jpaa.000 2020. 106517.
[15] A. Gretton, K. Borgwardt, M. Rasch, B. Schölkopf, and A. Smola. A kernel method for the two-sample-problem. In Advances in Neural Information Processing Systems, volume 19, 2006.
[16] N. J. Higham. Cholesky factorization. Wiley Interdisciplinary Reviews: Computational Statistics, 1(2):251-25,, 2009, doi: 10. 10.1002/wics.18
[17] S. Huntsman. Parallel black-box optimization of expensive high-dimensional multimodal func- tions via magnitude. arXiv preprint arXiv:2201.11677, 2022. doi: 10.48500arX1..22111167..
[18] Huntsman. Diversity enhancement via magnitude, Evolutionary Multi- Criterion Optimiza- tion, pages 377-390, 2023. doi:
[19] T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models. Advances in Neural Information Processing Systems, volume 32, 2019.
[20] Y. Lai, X. Zhu, Y. Zhang, and M. Diab. Diversity, density, and homogeneity: Quantitative characteristic metrics for text collections. In Proceedings of the 12th Language Resources and Evaluation Conference, pages 1739-1746, 2020.
[21] T. Leinster. The magnitude of metric spaces. Documenta Mathematica, 18:857-905, 2013. 2015. doi: 10.4171/DM/415
[22] T. Leinster. Entropy and Diversity: The Axiomatic Approach. Cambridge University Press, 2021. doi: 10. (0.1017/978100866555
[23] T. Leinster and M. Shulman. Magnitude homology of enriched categories and metric spaces. Algebraic & Geometric Topology, 2165):2175-2221, 2021. doi: 10.21 10.2140/ag..2221.22.222
[24] R. Liao, Y. Li, Y. Song, S. Wang, W. Hamilton, D. K. Duvenaud, R. Urtasun, and R. Zemel. Efficient graph generation with graph recurrent attention networks. In Advances in Neural Information Processing Systems, volume 32, 2019.
[25] M. W. Meckes. Positive definite metric spaces. Positivity, 17(3):33-7777, 2013. doi: 10.1007/111117002200000
[26] M. W. Meckes. Magnitude, diversity, capacities, and dimensions of metric spaces. Potential Analysis, 42(2): 572, 2015. doi: 10.1007/s1 1188011--4444333
[27] H. Minc. Nonnegative Matrices. Wiley, New York, NY, USA, 1988.
[28] M. F. Naeem, S. J. Oh, Y. Uh, Y. Choi, and J. Yoo. Reliable fidelity and diversity metrics for generative models. In International Conference on Machine Learning, volume 119, pages 7176-7185 2020. doi: 10.48550/axxv2220200977
12

---

[29] L. O'Bray, M. Horn, B. Rieck, and K. Borgwardt. Evaluation metrics for graph generative models: Problems, pitfalls, and practical solutions. In International Conference on Learning Representations, 2022.
[30] N. Reimers and I. Gurevych. Sentence-BEar : Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conferencee on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3982-3992, 2019, doi: (0.18655//1/DD111444
[31] E. Roff and M. Yoshinaga. The small-scale limit of magnitude and the one-point property. arXiv preprint arXiv: 144977 2023, doi: 10.485500/rXiw.231 12..22211..447..
[32] M. S. M. Sajjadi. O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems, volume 31, pages 5243,, 2018.
[33] S. Salim. The q-spread dimension and the maximum diversity of square grid metric spaces. PhD thesis, University of Sheffield, 2021.
[34] Sen, G. Namata, M. Bilgic, I.. Getoor, B. Galligher, and TT. Fliassi-Rad. Collective classifica- tion in network data. Al Magazine, 29(3):93-106, 2008. doi: 10. http9//aaa..22933..mm
[35] L. Simon, R. Webster, and J. Rabin. Revisiting precision recall definition for generative modeling. In International Conference on Machine Learning, volume 97, pages 5799-5808, 2019.
[36] A. R. Solow and S. Polasky. Measuring biological diversity. Environmental and Ecological Statistics, 1:95-103, 1994. doi: 10. N0.1007/BFO242265000
[37] C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architec- ture for computer vision. In JEEE Conference on Computer Vision and Pattern Recognition, pages 281 1818-228 2-2626, 2016. doi: 10.1 10.1/9///P.2001 6.308.
[38] G. Tevet and J. Berant. Evaluating the evaluation of diversity in natural language generation. In Proceedings of the 16th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 326-346. Association for Computational Linguistics, 2021. doi: 10.18653/v1/200 leacl-main.225
[39] L. Theis, A. d. Oord, and M. Bethge. A note the evaluation of generative models. In International Conference on Learning Representations, 2016.
[40] R. Thompson, B. Knyazev, E. Ghalebi, Kim, and G. W. Taylor. On evaluation metries for graph generative models. In International Conference on Learning Representations, 2022.
[41] R. Turkes, G. F. Montufar, and N. Otter. On the effectiveness of persistent homology. In Advances in Neural Information Processing Systems, volume 35, pages 35432-355555 35448, 2022.
[42] X. Wang and I. H. Sloan. Low discrepancy sequences in high dimensions: How well are their projections distributed? Journal of Computational and Applied Mathematics, 213(2)366-386, 2008. doi: 10. 10.10166..cmm 2007.0 .01 .005.
[43] J. Wayland, C. Coupette, and B. Rieck. Mapping the multiverse of latent representations. In International Conference on Machine Learning, 2024.
[44] S. Willerton. On the magnitude of spheres, surfaces and other homogeneous spaces. Geometriae Dedicata, 168(1):291-31) 2014. doi: 111100//10011-01--99500
[45] K. Xu, W. Hu, J. Leskovec, and Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
[46] J. You, R. Ying, X. Ren, W. Hamilton, and J. Leskovec. GraphRNN: Generating realistic graphs with deep auto-regressive models. In International Conference Machine Learning, pages 5708-5717, 2018.
[47] Y.DD. Zhou, K.TT. Fang, and J.-H. Ning. Mixture discrepancy for quasi-random point sets. Journal of Complexity, 29(3-4):2833011 2013. doi: 10.00.6/.000.221111..0000
13

---

Appendix (Supplementary Materials)
Extended Theory and Empirical Validation
15
A.1 Illustration of Magnitude and Magnitude Weights
15
A.2
Stability Proof
15
A.3
Empirical Stability
16
A.4 Isometry Invariance
16
A.5
Computing Magnitude
17
A.6
Benchmarking Computational Times
17
Additional Details for Our Methods
18
B.1
Embedding Data
18
B.2
Scale-Finding Procedure
19
B.3 Integration
19
C Extended Discussion on Diversity Measures
19
C.1 Definitions of Intrinsic Diversity Measures
19
C.2 Defining Reference-based Evaluation Metrics
20
C33 An Axiomatic Approach to Defining Intrinsic Diversity
21
C.4 Counterexamples
23
C,5 Simulation Studies
24
C.5.1 Effective Size and Uniformity
24
C.5.2
Investigating the Twin Property for Diversity Measures
24
Additional Details for Our Experiments
25
D.1
Curvature Experiments
25
D.2
Measuring the Intrinsic Diversity of Text Embeddings
26
D.3 Characterising Text Embedding Spaces
D.4 Image Experiments
32
D.5 Graph Embedding Experiments
33
14

---

A Extended Theory and Empirical Validation
A.1 Illustration of Magnitude and Magnitude Weights
While we did not use magnitude weights, which are the individual contribution of each point in a space to its overall magnitude, throughout our experiments, they play more central role in some of the later proof's and the computation of magnitude in practice. Further, magnitude weights give an intuitive explanation on how each individual observation influences magnitude and the magnitude function as illustrated in Figure S.1.
Definition A.1 (Magmitude weights), Let X {II...... .n.......n]].... be a finite metric space with an associated distance metric d. The similarity matrix of X is defined as {x (i.j) = exp(-d(x,, xy)) for 1 < و.j < n. If (x is invertible, the magnitude weighting vector wx is defined as wxx := 1= Denoting the ith element of wx by Wxy, we obtain an equivalent characterisation of magnitude as Mag(X)= = Σs Wxz
(agy(0.41) 33
zooming in
Magx(100) =
Magx(0.01)
Magx(t)
magnitude weights
0.0
0.5
1.0
1.5
0.0
0.5
1.0
Figure S.1: Example of magnitude weights and the magnitude function for a metric space with 5 points. When the scaling factor is very small, e.g. t 0.01, the magnitude weights of all points sum up to approximately 1, so that magnitude is very close to 1, and the space effectively looks like one point. Following this, as we zoom in further, magnitude grows and at / - 0.41, 3 distinct clusters or points are visible. Finally, for = 100, all the points are clearly separated, their magnitude weights converge to one, and the value of magnitude approaches i.e. the cardinality of the space.
A.2 Stability Proof
Next to the theoretical properties linking magnitude to geometrical properties of space, which we previously outlined, we further prove that magnitude, as a metric space invariant, also satisfies properties that are advantageous in the setting of analysing latent representations. Specifically, we prove that magnitude and thus the proposed magnitude differences satisfy certain stability properties in light of perturbations of metric space. By this, we mean that if two metric spaces X, Y are close, we want to obtain bounds on the differences between their magnitude values. The canonical choice to measure closeness would be the Gtromov-Hauddoff distance, but in the absence of strong results concerning the behaviour of magnitude under this distance [21], we resort to a more general- but also meaker-notion of similarity in terms of continuity. More precisely, we will show that the similarity matrices used in the calculation of magnitude are well- bell-behaved in the sense that closeness of metric spaces (under some matrix norm) translates to continuous bound the variation of the similarity matrices. We first prove a general result about matrices and their associated transformations.
Lemma A.2. Let ||Al|a := sup A||Az|| : x E R" with ||æ||| = 1} refer to the induced 2-norm for matrices, and let A, B be two n x n matrices with || A B 2 S c. Moreover, let f(M == M1. Then (A) f [B) I ne.
Proof. Because ||. ||2 is consistent norm, we have [| < |1 M|>1111 n n matrices M. Without loss of generality, assume that [|](2 2 [I(B) 12 and || 2 || B 2-
for all
Thus, [[](2 [B)]l S d([]A]]2 [Bll2) d(IIA - B1(2) ne.
15

---

Treating A, B as inverse similarity matrices, the preceding statement shows that if the two inverse similarity matrices are close with respect to their spectral radius, the difference between their magnitude can be bounded. The following lemma shows that the similarity matrices satisfy general continuity condition."
Lemma A.3. Let (X, dx) and v be two metric spaces with corresponding distance matrices Dx, Dy and cardinality n. For all 6 >> 0, there exists 6 > O such that if Dxx - Dyl < holds elementwise, then Il5x Sr S.
Proof. As a consequence of the continuity of the exponential function, we know that there is 6 such that Kxx - srl <n=te. The row sums of Çx <y are therefore upper-bounded by e. We thus have Ilsx - Sr la S [27. Theorem 1.1, p. 24].
As consequence of Lemma A.3, and the continuity of matrix inversion, we know that magnitude is well-behaved under small perturbations of the respective distance matrices. Given a pre-defined threshold c, we can always find perturbations that preserve the magnitude difference accordingly. Notice that this result does not make any assumptions about the -Haudo-f distance of the metric space and only leverages the distance matrices themselves. Moreover, this result applies in case X, Y are close with respect to the Huusdorff distance. If du Y) < 6, the elementwise condition xxx- Dy| < 5 satisfied a fortiori. This stability of single-scale magnitude then further ensures the stability of the difference between magnitude functions as defined in 3.4 in the same sense. Nevertheless, from a theoretical point of view, this result could be made stronger by showing bounds in terms of distances between the metric spaces. We leave such result for future work, noting in passing that such strong results remain elusive at the moment [14]; it is known, however, that the magnitude function is at least lower semicontinous [25, Theorem 2.6].
A.3 Empirical Stability
We further investigate the empirical stability of the magnitude function difference. Given the difficulty in proving strong theoretical stability results, we verify that, in practice, the magnitude function difference remains stable when adding noise to the input space. We thus sample points from Laplace distribution with mean b E {0.0001, 0.001, 0.005, 0.01, 0.05J. Figure S.2 depicts the errors in magnitude function difference relative the the area under the magnitude function of the unperturbed data across three different datasets (circles, Swiss Roll, Gaussian blobs), using a different number of samples (varying between 100 and 5000 across 50 repetitions). The bound of 5000 points has been chosen given the clear downwards trend across multiple noise levels; we expect the same trend to hold for larger sample sizes. We observe that the magnitude function difference does not increase above the value of 1 x 10-3 with increasing sample size. In fact, the difference fluctuates more for smaller number of points, but this is still within very small range. We therefore conclude that the magnitude function difference between the original space and its noisy version does not change much, which indicates that our measure is reliable and stable across multiple experimental conditions.
O and variance 26 with different levels of noise, i.e.
A.4 Isometry Invariance
A measure of the difference (in diversity) between latent spaces should fulfill certain desirable properties from both practical and theoretical perspectives. In the following will show minimum requirement, namely that the magnitude difference between isomorphic spaces equals zero
Definition A.4 (Isometry). Let (X, dx) and (Y,dy) be two metric spaces. A map f: X Y called an isometry, or distance-preserving if for any a, b E X, we have dx (a, b) = dy(f(a), X and Y are called isometric if there is a bijective isometry from X to Y.
Y, we have Magx = Magy- Lemma A.5 (Isometry invariance). Given two isometric spaces X,Y,
Proof. Let (X, dx) and (Y, dy) be metric spaces with cardinality n and let f: X + Y denote their isometry. Then, the similarity matrix of X is (x(i..) exp(-dx (I x;)). Since f is
'It is clear that the mapping itself is continuous because of the functions involved in its calculation. However, we find it important to remark on the bound obtained with respect to the spectral norm of the two similarity matrices.
16

---

0.004
Magnitute difference
0.004
Level of noise
Magnitude difference
0.003
0.001
0.0001
noise
Magnitude difference
0.003
Level of
0,003
0.035
0.05
0.82
0.002
0,002
6-03
0.05
0.002
0.001
0.001
0.001
0.000
0.000
Number of samples
2000
4000
Number samples
2000
4000
Number samples
Figure S.2: Empirical stability of magnitude. Magnitude difference is stable across different datasets (from left to right: Circles, Swiss Roll, Gaussian blobs) and sample sizes. The lines show the mean magnitude difference relative to the magnitude area of the unperturbed data and the shaded area the standard deviation calculated across 50 repetitions.
an isometry, we have, dx (x,,,,))) (f((fx;;) f(x))). Hence, Sx(i,)) = (x,(-dx(z,,zz)) exp( (dy ((z))) = (i,j, Since X and have the same similarity matrix, we have Magx = Magy-
Corollary A.6. The magnitude functions of two isometric spaces X, Y are equal for all t 2 0.
Notice that the converse of this statement is not true in general, i.e. there are non-isometric spaces whose magnitude functions are the same [21].
Corollary A.7. Let X be a metric space and Y = =x with c R4. Then the magnitude functions of X and 1/2Y are equal. Also, the magnitude functions of \/diamx X and 1/dimmy Y are equal, where diamx max(dx).
Corollary A.S. Magnitude function difference equals zero for isomorphic spaces.
A.5 Computing Magnitude
naïve calculation of magnitude according to Definition 3.1 requires inverting the similarity mat- rix Çx, which has worst-case complexity of (nz²) and is numerically unstable. However, inverting Sx not required in practice; instead, suffices solve certain linear equations as also poin- ted out by Huntsman [17]. First, we notice that the calculation of magnitude can be written as Mag(X) == 1 'I. For finite metric spaces and negative definite metrics, Çx is a symmetric positive definite matrix, thus affording a Cholesky decomposition, which factorises Cx LL with L being lower triangular matrix. This operation is numerically stable and more efficient than matrix inversion [16]. We thus have Mag(X) 1 " = 1 (LL 1 11 (L-1))((-11 (L-11). This is equivalent to calculating x: TT with 11 which since L is lower triangular. Likewise, we can reformulate the calculation of the magnitude weight vector wx = cx as solving (xuex = 1, which also benefits from the Cholesky factorisation.
efficiently obtain by solving Lx 1
A.6 Benchmarking Computational Times
To assess the improvements computational efficiency discussed Appendix A.5, we benchmark the following computational methods in Python:
Numpy inv: Inversion of the whole matrix < using numpy. linalgi inv as suggested by Bunch et al. [5]; see also https://githut: com/ hom/AmFaade Am/ AnFamMLTeam/ for an implementation.
metric-space
Scipy solve: Solving for the magnitude weights using scipy. linalg, solve and assuming { to be positive definite.
Cholesky weights: Cholesky decomposition using scipy linall...................l...... cho factor to com- pute the magnitude weights.
.
Cholesky: Using a Cholesky decomposition suggested above to compute the value of magnitude directly. This the method we implemented to compute magnitude throughout this work.
17

---

Numpy Iy
Scipy salve
30
Cholesky
Cholesky
weights
Krylov weights
serords
20
5-
500
1000
1500
2000
2500
3000
3500 4000
points
Figure S.3: Benchmark of computational times in seconds for magnitude functions evaluated across 10 scales. We observe that Cholesky decomposition performs well, even for larger number of observations ensuring that magnitude functions can be computed in a matter of seconds. Lines show the time in seconds across five repeats, shaded areas the standard deviations.
Cg weights: Conjugate gradient iteration using scipy -aapss.spare.... linalg.nnlgg...... Cg and absolute tolerance of 1e-3 to solve for the magnitude weights.
Krylov weights: Pre-conditioned conjugate gradient iteration using krypy linsys Cg as implemented by Salim [33] to calculate magnitude weights.
All methods are evaluated simulated data of a Swiss Roll with an increasing number of points. For each space, magnitude is evaluated at ten scales evenly spaced between zero and the convergence point. The computational times and their standard deviations Figure S.3. Results clearly show that naive inversion of the whole similarity matrix is by far the most costly method for computing magnitude. This is followed by the conjugate gradient methods described above, where the pre-conditioned version is somewhat faster than the implementation without pre-conditioning for larger numbers of points. However, for evaluating magnitude at only 10 scales these approaches do not necessarily lead to improved performance compared to solving for the weights simply using scipy. .lliallgg........... solve. Finally, note that our proposed implementation using Cholesky decomposition is the fastest computational method achieving less than a third of the computational time of the most naive implementation for larger datasets. Indeed, these results confirm that even for thousands of points magnitude functions are efficiently computable in matter of seconds. Overall, this computational performance is more than sufficient for the relevant diversity evaluation tasks discussed in this study. Statt-of-tth--at the-art graph datasets are typically small, the output of text generation models is often assessed on specific tasks and even image embeddings are frequently evaluated in terms of meaningful subsets, e.g. by studying intra-class diversity. Indeed, we ran all our experiments locally with the following hardware specifications:
recorded across five re-runs in
CPU: 12th Gen Intel(R) Core(TM) 17-1265U,
RAM:: 16 GB DDR4,
SSD: 512 GB NVMe SSD
B Additional Details for Our Methods
B.1 Embedding Data
Creating latent representation or embedding X MI), whose diversity should be evaluated, depends on the complexity of the specific model M and the input data I that should be represented. This step independent of our design choices. Given a latent representation we then choose a suitable notion of distance, for example cosine distances are a natural choice for text embeddings, which (compared to e.g. Fuclidean distances) better represent similarity in meaning rather than lengths; or Euclidean distances understand more general latent spaces.
18

---

B.2 Scale-Finding Procedure
Next, we give a brief illustration of the scale-finding procedure developed to automate mag- nitude computations. We use TOMS 748 root-finding algorithm [2] as implemented via toms748. Our aim find the scale feea at which Magx (feome X - c such that |/uue - /comm| I atol + rol |/cooo| where lame is the true scale at which Magx (fcom) = [X] -, rtol is the relative error and atol the absolute error. We initialise the algorithm at the search interval (a. b] setting
0 and b = 100 as an initial guess. We then check if (Magx (a) |X| + <) and (Magx(b) X + <) have opposite signs. If not we update a = and 100 repeating this at most 100 times and raising an error if they still share the same sign indicating the root-finding failed. Otherwise, we run TOMS 748 algorithm to find feom as specified above using at most 100 iterations. This algorithm requires Magx (fcom) to be continuous to perform reliably, which holds for negative definite metric spaces as proven in Appendix A.2.
After reaching the convergence scale as defined in Definition 3,5 we know that magnitude and hence diversity can change by at most 6, which directly follows from the convergence behaviour of magnitude. Based on this, the default parameter = 0.05] X| sensible compromise for determining the scales of interest across which diversity changes most notably. To support this choice of convergence scale, we empirically investigate the impact it has on the results reported in our work. In particular, Figure S.12 investigates how the choice e influences the correspondence between human evaluation scores and the diversity of generated text, while Figure S.17 outlines the results of the graph generative model evaluation experiment for varying values of We note that choices of S 0.05 X give stable and generally very good results for both reference-free free and reference-baste diversity evaluation as further detailed in Appendix D.
0.01| X is chosen
Figure S.4 gives some further intuition on this scale finding procedure. In particular, it demonstrates how taking the median convergence scale across four example spaces gives a suitable evaluation interval across which their magnitude can be compared as explained in more detail in Section 3.4. This simple example also illustrates why to compare latent spaces in reference-free setting we recommend using the median another suitable quantile of the converge scales rather than the minimum or the maximum, which are less robust and more sensitive to outliers.
Finally, we note that the mane-ffnding approach can not only be used to compute MAGAREA or MAGDIFF, but also find suitable scale at which evaluate magnitude leading to a summary of diversity at single threshold. In practice, we recommend choosing this single scale to be less or equal to the convergence scale of space and that the best resolution to choose depends on the question of interest. Lower scales give a more coarser view summarising the diversity of large clusters while higher scale parameters show a clearer separation between individual observations.
single resolution.
B.3 Integration
In practice, we evaluate the integral from Definition 3.4 and Definition 3.3 using Trapezoidal integration as implemented via scipy.
seciyy.inteee .integrate....... trapezoid across a certain number, Thes of evenly spaced evaluation scales in T = [fo, fuu)). This numerical integration method is chosen due its simplicity and computational efficiency, but more complex approximation methods can also be employed.
C Extended Discussion on Diversity Measures
C.1 Definitions of Intrinsic Diversity Measures
The difficulty in defining diversity in representation learning has led to few varying proposals for evaluating the intrinsic diversity of latent representations. Amongst these we consider the following three methods as bascline measures:
GMSTDS: For a X, a D-dimensional embedding, it is directly computed as
D
GMSTDS = D
0
(2)
19

---

XX
175
magnitude Mag. (t)
130
120
100
Fose
Feorr
100
100
scaling factor
Figure S.4: Illustration of the proposed seale finding procedure for comparing the diversity of four examples from Figure 1. We compute the approximate convergence scale for each space defined in Definition 3.5 at which magnitude has reached a certain value [.X] close to the cardinality. We then take the median of these scales find shared endpoint teur for defining a shared evaluation interval T [0, fear).
2 is the standard deviation across the j-th embedding dimension [20]. Thus, GMSTDS regards embedding cluster and assessing diversity by quantifying its spread.
where oj v (Ei=i,... Fig )²
AVGSIM: Average mean similarity (or variations of it) is the most frequently used diversity measure ML. is simply computed as
AVGSIM
1
Σ
((i,))
KJSnjンi
across all distinct pairs of points in X assuming ( is symmetrie [38]. This approach simply summar- ises that in more diverse space, observations should average be less similar.
Vendi Score (VS): We also consider the Vendi Score, which is the only entropy- based diversity measure proposed in related ML literature. Let Ç be positive semi-definit similarity matrix with ((i, for all I n. Compute A, the eigenvalues of Ç/n. Then the Vendi Score defined as
VS exp(- >> log(AA))))
4
taking 0 (0) by convention. That is, the Vendi Score the exponential the Shannon entropy of the eigenvalues of ç/n [13]. It can thus be interpreted summarising the effective number of modes in a space at a specific scale of similarity.
C.2 Defining Reference-based Evaluation Metrics
Here we define the metrics used in the image and graph embedding experiments.
A
Precision =
y Emanlfold(X.......
(5)
M
==1
N
Recall
1
Σ 2-1
1 M......
(6)
M
where the manifold defined as manifold(X,,,........ XN) = - B(X,, NNDx Here, B(x,, r) the sphere in IRd of radius around x, NND&(X) is the distance to the k-th nearest neighbour, and 16) is the indicator function.
20

---

Similarly, density and coverage are defined as follows:
M
N
Density
Σ
1
YEDD(X..NND(X.))
M =
N L3juLYYEB(X,,NND((..))
Coverage
(8)
N 1=1
Maximum Mean Discrepancy (MMD) [15, 46] is metric based on graph statistics. MMD Linear is computing MMD with a linear kernel [29]. We define S, ..' ..m) ~ P. and S {a?..... 2 ~ Pg. where 2; is a feature vector from a corresponding graph G₁. Therefore, MMD is defined as follows:
MMD(S,.S,))
1 m
Σ (k(2)
Σ (k(z),,)))) n2 (
2
2))))
(9)
where k(--) is a general kernel function. For the case of the metric MMD Linear, used in our graph experiments, we use a linear kernel.
C.3 An Axiomatic Approach to Defining Intrinsic Diversity
The attempt to define, prove and interpret theoretically well-founded notion of diversity has inspired decades of heated debate in theoretical ecology. While in the context of machine learning, diversity is still very seldomly explored axiomatically, measures of biodiversity are often defined in a more well-established and unified framework built on mathematically complex ideas, such as entropy and extended notions of size. However, there lot of benefit be gained by a more theoretical discussion on evaluating diversity in representation learning. Indeed, as pointed out by Leinster [22], if a diversity measure does pass basic logical tests, it is likely to be misleading and potentially useless for practical applications, which can have far-reaching detrimental consequences. By discussing some of these fundamental properties we thus uncover exactly how existing diversity measures for evaluating latent representations fail at essential requirements. We further discuss how magnitude functions improve upon alternative diversity metrics.
Throughout this section let mxX) E R. be a diversity measure of the (metric) space X, let dx be a distance metric this space and Çx matrix of pairwise similarities.
Diversity measures the absolute richness in observations. Metric space magnitude as a diversity measure has been introduced with the following three properties fundamental properties in mind [36]:
Monotonicity in observations. Including a new observation with all positive distances to a metric space with a negative definite metric does not decrease diversity. Formally, for all zo $ X where (X, dx) is metric space define (Z X U zo, dz) via inclusion so that dz is a valid metric. If dz (Tg, x) > 0 for all æ E X, the diversity measure m is monotone in observations m.x S mz. For magnitude, when taking m Mag. this directly follows
from Corollary 2.4. in Leinster [21].
Twin property. Diversity does not change when including a duplicate observation. Formally, for 2o E X define Z = 20 U X. A diversity measure m then respects the twin property if mz = mx. Magnitude fulfils this property, which follows directly from the definition of a metric space because X is set and cannot include duplicate observations. That is, for Zo E X where (X, is metric space, we get that Z = xo U.X = X and Magz = Magx- Monotonicity in distances. For Y |X| 2, when f: (X, + (Y,dy) is bijective mapping so that no distance decreasing and at least distance is increasing and dx and dy are negative definite, diversity does not decrease. Formally, when dx (x,, < dy (f(1), f(x22) for all æ1, D2 E X and dx (a. b) < dy (f))
for some a, b E X, diversity measure is monotone in distances if mx S my. Magnitude fulfils this conjecture for all known examples of negative definite metric spaces [21, 36).
Given that these three essential properties hold for magnitude at every choice of I, they also hold for the area under the magnitude function for a space with a negative definite distance metric, whose magnitude function is necessarily continuous demonstrated in Appendix A.2. However, none of
21

---

the alternative diversity measures defined in Appendix C.1 fulfil the twin property or the condition of being monotonic observations as demonstrated in Appendix C.4. This can lead an undesirable decrease in diversity when including novel observations to a space that adds information and should thus not reduce the total diversity but is similar to existing points. Indeed, given this behaviour we argue that a decrease in diversity as measured by the existing metrics can actually be misleading.
Diversity requires basic invariances. Elaborating on the idea that diversity measures should pass basic sanity checks, the following more basic properties are desirable for measuring the intrinsic diversity of a space as intended [22]:
Diversity does not change under isometric transformations of a space. Magnitude is isometry invariant as defined and proven in Appendix A.4.
Isometry invariance:
Symmetry: Diversity is invariant under permutations of the input observations. Formally, let a X + X be a permutation of X. Then, a diversity measure is symmetrie if m(X) (X)).
Absence invariance: Diversity only depends on the samples and features present in the dataset. Formally, let X' C RR' be dataset with feature space of dimension D', and X S X' be the subset of observed samples and non-zero features X S R.. Then, diversity measure m is absence-invarias change when removing elements or features that have not been observed or have zero probability.
if mXX) m(X). That is, diversity does not
Magnitude and thus MAGAREA fulfil the aforementioned conditions by definition. Specifically, magnitude is absence-invariant symmetric and ssometry-invariane because the distance metric dx itself is absence-invariant, symmetric and ssometry-invarian in Appendix A.4. Overall, invariances are key properties when studying the diversity of latent representations. Indeed, symmetry is so essential that all diversity measures evaluated in this study fulfil it. GMSTDS however, not absence invariant as always equals zero whenever one feature or dimension in the embedding space is constant, which limits its usefulness and makes it sensitive to the absence of information.
These properties can be proven as
Diversity measures the effective number of distinet observations. Originating from using entropy to define suitable notion of diversity for theoretical ecology, the following aspects of measuring diversity in deserve to be highlighted:
Effective size: A dataset with fixed number of points is more diverse when points are separated e.g. distributed uniformly or maximally disordered and becomes less diverse as observations cluster together. Diversity is maximised when points are completely distinct and minimised when all observations are identical. Formally, let e X + XX' be transformation that decreases the entropy of space X.7 We then require that m (X) S mme(X)). Say diversity measure m(X) E R has a minimum myin and maximum value mmmax We further require that lim m(tX) = Mmin and lim m(xX) = mmax-
Effective number: Diversity should be measured the effective size of a space in the range [1, e m(X), so diversity is expressed the effective number of distinct points or clusters in a space.
We note that the latter condition on diversity being measured as effective number is very helpful for measuring biodiversity [8] and to give sensible summary of the clustering structure of a dataset, but being interpretable Nevertheless, we find that relating diversity to assessing entropy and requiring that diversity is aware of clusterability and uniformity in the data, is essential for useful diversity measure as further illustrated in Appendix C.5.1. Note that magnitude fulfils the conditions of measuring both an effective size and an effective number by definition defined in Section 3.2 and discussed in more detail by Leinster [22]. Based on this, MAGAREA as a multi-scale summary of magnitude also fulfils the condition of measuring the effective size of space and can easily be converted into an effective number.
number of points is potentially not necessary in all applications.
"We leave the precise definition of entropy up to discussion [22]. but give examples of what we consider to be -ecoopy-eerraiigg operations. One example is scaling through a negative definite metric space tXX by increasing the similarity between observations i.e. by decreasing t. Removing informative features, mode dropping or mode collapse further phenomena that decrease diversity.
22

---

Table S.1: Counterexamples demonstrating that alternative diversity measures fail to fulfil funda- mental axioms of diversity, whereas magnitude passes these sanity checks.
Space
MAGAREA (1)
VS (t)
AVGSIM ())
GMSTDS (1)
X two point space
4.602
1.867
0,368
0,500
Q absence
4.602
1.867
0.368
0
Z duplicate point
4.602
1.77
0.579
0.471
Y three point space
4.613
1.809
0.577
0.469
Diversity is multi-scale summary of (dis)similarities. Finally, we note that diversity should be seen as a continuous function of the scale of similarity and behave accordingly:
senisiiityseessitiiitty:: Diversity is computed from and determined by the (dis)similarities between observations. That is, diversity measure m E of a dataset is defined as f(Dx) m(x) f((x) for some function f. where Dx is a matrix of distances between observations in X and similarly Çx is a matrix of similarities.
Scale-dependence: Further, we require that diversity measure my E R is continuous function of the scale of (disliimilarity Thus, diversity is not just one-number summary, but function of said scale. Formally, mx f(Dx (t)) m, (X) == (({x(t)),
Multi-scale: A multi-scale measure encodes both local and global trends in the data manifold by considering multiple levels of scale resolution simultaneously. Let m, be scale-dependent diversity measure. A multi-scale measure, m, further summarises diversity across multiple scales i.e. [m (Mit::.. -o> (m,,)) R for 2 and some summary function f.
Rather than giving snapshot of diversity at fixed degree of (dis)similarity, multi-scale methods summarise diversity across varying scales of (dis)similarity. We reason that this property is advant- ageous capture more complete picture on how both coarse and more nuanced dissimilarities in observations affect diversity. Being a multi-scale summary is a distinguishing characteristic of our proposed diversity measure, MAGAREA. Alternative diversity measures, such as average similarity, the Vendi Score or magnitude computed at one scale, do not fulfil this criterion as they are single resolution snapshots computed from fixed similarity matrix.
C.4 Counterexamples
In the following, we now demonstrate how the diversity measures introduced in Appendix C.I fail at some of the fundamental axioms of diversity introduced in Appendix C.3 via simple examples. Consider the following feature matrices:
X =
Q = 0
Z
and Y =
(10)
0
0..11
First, consider the two point space as given by X as a reference space. We then use Manhattan distances dx and the similarity matrix Çx as given in Definition 3.1 to compute each diversity measure. In particular, compute MAGAREA we use the convergence scale of X as reference. The resulting diversity values are summarised in Table S.1.
Absenc invariance. To check for absence invariance, include a constant feature dimension to and get Q. Clearly, Q has the same diversity as X. Indeed, all diversity values are equal for these spaces other than GMSTDs. This counterexample thus shows that GMSTDs is not absence invariant.
Twin property. Next, include duplicate observation to X and get Z to examine the twin property. Note that including a repeated observation does not change diversity as the space still consists of only two unique observations. However, VS, AVGSIM and GMSTDs all assess that Z is less diverse than X and thus do not fulfil the twin property.
Monotonicity in observations. Lastly, include one new observation to X and consider the three point space Y. While Y very similar to X, we can see that overall, in terms of absolute diversity, X is not more diverse than Y. However, MAGAREA is the only measure in Table S.I that indicates
23

---

Table S.2: MAGAREA shows the correct order in diversity when comparing the simulated examples in Figure la) from the main text. In contrast, two baseline diversity measures, AVGSIM and GMSTDS, as well as the discrepancy measure 1.2STAR fail to distinguish that the random point pattern, X,, is more diverse than the clustered point pattern, X2-
MAGAREA (1)
VS (t)
AVGSIM (4)
GMSTDS (1)
L22TAR (1)
X1 Poisson Process
14.6
0.39
0.59
0.004
X2 Hawkes Process
99
13.1
0.39
0,59
0.004
Xs Two Gaussians
69
5.7
0.51
0.53
0.041
X4 One Gaussian
48
3.1 0.79
0.14
0.260
that Y is slightly more diverse than X. Indeed, all of VS, AVGSIM and GMSTDS indicate that Y is less diverse than X and they are thus not monotone in observations.
New or duplicate observations. Including a duplicate observation should lead to lower diversity than including new unseen observation to space. However, GMSTDS actually indicates that Z. the space with duplicate points, is less diverse than the three point space Y, which is clearly counterintuitive.
C,5 Simulation Studies
C.5.1 Effective Size and Uniformity
To gain more insights, we further compare diversity metries on the examples from Figure Ia) and report them in Table S.2. That is, we simply ask the question, which of these simulated examples are more diverse. Note that each space has the same number points, but the examples vary in their effective size and clustering behaviour. We know ground truth ordering, namely that the uniform pattern in X, is the most diverse example as points are most clearly separated and more evenly spread across the entire sampling domain.
Results then show that two of the baseline measures, AVGSIM and GMSTDS, fail to capture notable differences in diversity on simple simulations as they do not detect that the random pattern in X1 is more diverse than clustered pattern in X2- Here, the difference in diversity between X₁ and X₂ is driven by the difference in the disorder or clustering behaviour of their respective observations. Linking back to the axioms of diversity, these examples illustrate how diversity should be aware of an effective number of distinct clusters or points and evaluate diversity via assessing entropy.
Our results thus demonstrate how magnitude gives a superior notion of diversity, that captures variations effective size, clustering behaviour and uniformity. In contrast, both AVGSIM and GMSTDS are unaware of these differences. In practice, this could further lead misleading assessments of diversity for c.g. generative model evaluation. For example, a clustered sentence embedding, corresponding to few distinct groups of sentences that share the same meaning, might be wrongly deemed to have the same diversity as more varied set of generated sentences, which notably differ in meaning and show a more uniform distribution in the embedding space.
To further contrast diversity measures against uniformity criteria, we also compute the L2-star discrepancy of each sample, L2STAR. Discrepancy measures in general assess how much an empirical distribution deviates from a uniform distribution on the unit hypercube [42, 47]. Lower discrepancy implies higher agreement with uniformity, which in our context corresponds to higher diversity. However, we observe that that 1.2STAR does not clearly distinguish a difference between X₁ and X2, but rather assesses that both examples are close to uniformity. Results indicate that this classical discrepancy measure gives global summary of evenness, which assesses X2 appears uniform on large-scale, but does not pick up on local difference in effective size and small-scale clustering behaviour. This illustrates the necessity of evaluating both local and global trends in diversity via a multi-scale summary of diversity as given by MAGAREA.
C.5.2 Investigating the Twin Property for Diversity Measures
To link our investigation to the theoretical axioms of diversity, we examine the twin property. This requirement asserts that diversity should not change when including duplicate observations into
24

---

0.0
duplicated points
3.0
1001
X
25
Figure S.5: Our method, MAGAREA, is the only diversity measure that satisfies the twin property, one the fundamental axioms of diversity. MAGAREA correctly assesses that diversity does not change when including duplicate observations. All baseline diversity scores fail to fulfil this property and show inconsistent changes. VS even switches trends from increasing initially to decreasing after a certain number of replicates has been reached for X6. We investigate two simulations: X5 shows a Gaussian distribution with 100 points, whose centre gets duplicated. X6 shows a Gaussian blob with 100 observations as well as 10 outliers, which get duplicated with the specified multiplicity. Changes in the diversity scores as increasing numbers duplicate copies added are shown below. Dashed horizontal lines show the ground truth value of each diversity measure prior to including duplicate points.
given dataset. When evaluating generative models, diversity measures that satisfy the twin property are advantageous because they penalise models that just repeat existing observations again, as opposed to providing genuinely 'novel' outputs. Results of this case study are reported in Figure S.5, showing how the popular baseline diversity measures, AVGSIM, VS and GMSTDS as well as the discrepancy measure L2STAR, the L2-star discrepancy, all fail to fulfil the twin property, instead exhibiting highly-incone iichlyiinoonsstent behaviour. Our proposed method meanwhile is the only diversity measure that respects the twin property and remains consistent, demonstrating one of its practical advantages.
D Additional Details for Our Experiments
In the following we give further details and elaborate on the experimental setup and datasets used for our experiments well showcase extended results.
D.1 Curvature Experiments
Here we provide more details about the curvature experiments, which builds on the approach by Turkes et al. [41]; see https: // thub. www.cooo//aaaeee htte//enntta-eee ternnttttturks// tiikee//turrie turess/sereeeeeeeoonnmm for an implementation. We generate a unit disks Dx of surfaces of constant curvature K, with 3 cases: the first one is when 0 (we then have the Euclidean plane), disk model of the hyperbolic plane), x > (sphere with radius kK) We vary the curvature k to be in the interval [-2, 2). For each value of K, construct point clouds by sampling 500 points
0 (we have space of negative curvature, the Poincaré
from Dx- We generate 201 surfaces with equally spaced curvature in the interval [-2, 2]. Then, we compute magnitude for each space using Euclidean distance and 30 evenly spaced intervals until the scale frut 73.
From the resulting values of MAGAREA plotted against the curvature values in Figure 2, we can intuitively explain the relationship between diversity i.e. magnitude and curvature. For unit disks of positive curvature, the higher the curvature the lower the value of ARGAREA. This indicates that
25

---

points move closer and closer the more curved the surface is decreasing the diversity in Euclidean space. For surfaces with negative curvature we see the opposite trend. The more negatively curved the Poincaré disk the lower the value of MAGAREA. This is because Fuclidean distances between points and thus diversity are decreasing.
For the results reported in Table we further apply 5-fold cross-validation and aim to predict the curvature values from the diversity score. We first train a quantile regression model on the MAGAREA after applying polynomial feature transformation of degree 2 to the training data suspecting a seadratic-looking relationship between MAGAREA and curvature after exploratory analysis. Further, we compare this to piecwise linear regression with two breakpoints under the assumption that the relationship between MAGAREA and curvature as plotted in Figure rather depicts a piecewise linear relationship clearly separating spaces of positive and negative curvature. Both regression models were compared in the final results in Table to investigate multiple proposals on how to interpolate between the MAGAREA scores for surfaces of negative and positive curvature.
We further report six alternative models from Turkes et al. [41], which are using features from persistent homology (PH) summarising persistence diagrams (PDs). See Bubenik et al. [4] for more detailed explanation on PH and its relationship curvature. Specifically, in Table I we reproduce the following models from Figure 4. and Table 3. of Turkes et al. [41]]
SVR (all PH features) referred to as O-dim PH simple by Turkes et al. [41], which uses the lifespans of the persistence diagram computed on the samples;
SVR (selected PH features) denoted O-dim PH simple 10 by Turkes et al. [41], which uses the 10 longest lifespans; and SVR (PH vectorisation) corresponding O-dim PH by Turkes et al. [41], which selects the best PD vectorisation amongst a number of options, namely persistence images (PI) or persistence landscapes (PL).
All the PH-based methods use support vector regression (SVR) with a RBF kernel. Hyperparameter tuning for these models is conducted as reported by Turkes et al. [41] using grid search with choice of C parameters in {0.001, 1, 100). We further reproduce method based on pairwise distance matrices:
SVR (distance matrices) denoted as MI. by Turkes et al. [41].
Finally, we restate the performance scores of these two methods directly from Turkes et al. [41]:
MLP (shallow) denoted as NN shallow by Turkes et al. [41]; and
MLP (deep) denoted as NN deep by Turkes et al. [41].
We also note that the other models achieve different performance scores on our dataset than reported by Turkes et al. [41] due slight difference in dataset and cross- validaaiio splits. We use a smaller subset of samples than Turkes et al. [41] each having a unique curvature value as described above, and ensure that all models are evaluated on the same splits of data across 5-fold CV for fair comparison. Finally, we summarise the MSE achieved by each model in Table 1. Illustrating this, Figure 2 further shows examples of both magnitude functions for negative and positive curvature as well as the clear linear trend between MAGAREA and curvature.
D.2 Measuring the Intrinsic Diversity of Text Embeddings
We analyse data from Tevet and Berant [38], consisting of 1K sets of 10 sentences each generated for unique input prompts for 3 sentence generation tasks. The code by Tevet and Berant [38] is available at https:/ gi thub. com wwwmm//atteeeeeerrvvrrrrrtteeeee data can be downloaded from /ttt:://ddverritttt... zip. These tasks are story completion (storyGen) and dialogue response generation (respGen), both using MASS model fine-tuned on each dataset, and 3-word prompt completion (promptGen) using GPr-2-large without fine tuning. From the 1K response sets per task, 10 have been generated using the same decoding parameter, the eeftmax-temperatuue
under an MIT licence and -3- amazonaws... cww. data.
dec, sampled evenly across the range [0.2, 1.2], which controls the trade-off between quality and diversity by skewing models towards avoiding cow-probability tokens as dec decreases. This leads to potentially higher quality or fidelity but lower diversity or creativity in generated text. Further, for 200 response sets per task, mean human evaluation scores of text diversity were collected. Human workers rated the level of diversity
26

---

of the response set on scale from 5 (highest) to (lowest). The mean of these scores, absolute HDS (ABSHDS), measures the human perception of text diversity.
We then create embeddings for each set of sentences utilise the centence-transfoone library by Reimers and Gurevych [30]. In particular, we apply the following five embedding models:
all-distiroberra--ll: general-purpon model, embedding dimension 768;
all-MiniLM--L--2:: canerrl-puupose model, embedding dimension 384;
all-mpnet-base-v2. general-purpose model, embedding dimension 768;
bert-large-nli satpre--l---ss--mm-ttoooooe general purposs model, embedding dimension 1024; and roberta-base bastta-bbaee-llle www--mea--toeeeeee
purpose model, embedding dimension 768.
For each text embedding we compute magnitude functions and our diversity measure, MAGARFA, across 20 evenly sampled scales in [0, feu] where feur is the median of the convergence scales across all embeddings setting 0.01 | We then run prediction tasks as described in Section 4.2 repeating the same procedure for predicting both human scores and decoding parameters. Figure 10 details the results for predicting human scores. Figure S.11 then shows summaries of the ranks achieved by each metric across the two experiments. For additional context on the choice of convergence scales, Figure S.12 shows that changing the value of for computing MAGAREA has small effect on its correspondence with human evaluation scores indicating that our measure is relatively robust to this choice. Results from Figure 3 are further expanded is expanded in Table S.3 which shows the mean R2 scores and median ranks across datasets and Table S.4 summarising the difference in performance scores between MAGAREA and other diversity measures highlighting that entropy-hasee diversity metrics give superior notions of diversity. Illustrating the relationship between magnitude and diversity, Figure 4 and Figure S.7 plot both magnitude functions and the values of MAGAREA against both human scores and decoding parameters for one of the embedding models (plots for other embeddings follow similar trends). Indeed, these visualisations then demonstrate that MAGAREA can be used as a descriptor of diversity in generated text achieving high rank correlation with known ground truth values of diversity.
prompt
resp
story
10.0
10.0
10.0
Mapx(()
Moggx()
Magx(()
5.0
5.0
5.0
2.5
2.5
0.0
2.5
5.0
7.5
10.0
0.0
2.5
3.0
7.5
10.0
10
7.5
MACARBA
MAGARRA
MXGAREA
2.5
0.78
(0.76,
0.81]
0.85
[o.
0.86]
p:
0.74
(0.71,
0.77]
0.25
0.50
0.75
0.25
0.50
0.75
1.00
1.25
0.25
0.50
0.75
1.00
1.25
dec
des
dec
Figure S.6: MAGAREA and magnitude functions computed using all-mpnet-base-v2 plotted against the decoding parameters. The value shows the mean rank correlation between MAGAREA and the softmax-temperature well 95% bootstrap intervals computed by resampling 1000 times.
27

---

prompt
resp
story
10.0
10.0
10.0
Magxtt)
7.5
Moge(
7.5
Monpx
7.5
0.0
2.5
5.0
1.5
10.0
0.0
2.5
5.0
7.5
10.0
10
20
MAGAAREA AREA
M.C.ARE
5.0
0.86 (0.0, 0.9
0.79 [0.71, 0.85]
0.84 [0.77.
2
3
ABSLIDS
ABSTIDS
ABSTIDS
Figure S.7: MAGAREA and magnitude functions computed using all- all-mpnet- caapnet-base-ye v2 embeddings plotted against the human scores. The value P shows the mean rank correlation between MAGAREA and ABSHDS as well as 95% bootstrap intervals computed by resampling 1000 times.
prompt
resp
story
bart-krge--l-cssbbmmm----
diverity massumes
roberta base nii mean tokers
MALAREA
alleisthhobbrrwe
all mpnet bose v2--
GM8ms
Avusis
al-MiiLMAL12-22
correlation
correlation
correlation
Figure S.8: MAGAREA shows higher rank correlation with the ground truth softmax temperat- ure than alternative diversity measures across 3 tasks and embedding models. Baseline measures, AVGSIM and GMSTDS, show noticeably worse rank correlation. Points show the mean Spearman correlation and lines represent 95% bootstrap intervals computed across 1000 resamples.
prompt
resp
story
bert large stsb mean tokers wbbrtaabacelllamnnntkee.ss
divenity measines
VS GMSms
al-mpnat-bsse-22-- all L22
correlztion
correlation
correlation
Figure S.9: MAGAREA shows higher rank correlation with human evaluation scores than alternative diversity measures across tasks and 5 embedding models. Baseline measures. AVGSIM and GMSTDS, show noticeably worse rank correlation. Points show the mean Spearman correlation and lines represent 95% bootstrap intervals computed across 1000 resamples.
28

---

prompt
resp
story
wwn--lage-ni--ssbbmmm-. tokers
MAGAREA
bose
AvuSist
GMSIDS
Figure S.10: MAGAREA predicts human evaluation scores, across tasks and 5 embedding models. Baseline measures, AVGSIM and GMSTDS, perform worse terms of the mean R scores. Points show the of the R² scores, while lines represent the standard deviations across 5-fold cross-validation (repeated 10 times).
(a) Predicting decoding parameters
(b) Predicting human scores
Ma
MAGARBA
rank
1.0
Avo8t4
AVG8tM
4.0
GMSTD6
GM8tn4
0.22
0.67
0.0
0.22
0.44
0.67
proportion of resamples
proportion of resamples
Figure S.11: The area under the magnitude function outranks baseline diversity measures at (a) predicting decoding parameters and (b) predicting numan-evaluaan diversity scores across all experiments and cross- volssvaaiiationn resamples.
cark conslation ManAnEa and sbeHDS
1.0
prompt
story
prompt
0.2
reap
story
8.0
0.0
0.7
0.6
0.8 0.9
Figure S.12: Impact of the choice of convergence scales on predicting human evaluation scores. Both the rank correlation between MAGAREA and absHDS well as the R² scores achieved by MAGAREA for predicting absHDS remains consistent across different choices of convergence scales. Here take the median convergence scale across embeddings cutoff scale, but change the parameter from Definition 3.5. In the left plot, points show the mean rank correlation across 1000 repeated bootstrap resampling and lines the corresponding standard deviations. The right plot, points show the mean performance scores with standard deviations from 20 repeated 5-fold CV.
29

---

Table S.3: The mean performance each diversity measure in terms of R scores for predicting the decoding parameter. We also report 95% percentile intervals of these scores well standard deviations.
Model
Median Rank
Mean It
Standard Deviation
Lower 95% Pl
Upper 95% Pl
Task
MAGAREA
0.62
0.05
0.50
0.70
Prompt
VS
2
0.61
0.06
0.49
0.71
Prompt
AVGSIM
3
0.55
0.06
0.42
0.66
Prompt
GMSTDS
4
0.19
0.24
-0.02
0.55
Prompt
MAGAREA
0,70
0.62
Resp
VS
0.69
0.60
0.76
Resp
AVGSIM
3
0.64
0.07
0.50
0.74
Resp
GMSTDS
4
0.23
0.30
-0.02
0.66
Resp
MAGAREA
1
0.53
0.05
0.44
0.62
Story
0.38
0.58
Story
AVGSIM
3
0.41
0.29
0.51
Story
GMSTDS
4
0.17
0.22
-0.03
0.50
Story
Table S.4: The difference between each diversity measure and MAGAREA in terms of the difference in R² scores when predicting the decoding parameter. intervals of these differences and standard deviations.
We also report 95% percentile
Measure
Mean Difference in R² Scores
Stundard Deviation
Lower 95% PI
Upper 95%
Dataset
0.00
0.02
-0.02
0.06
Prompt
AVCSIM
0.07
0.03
0.03
0.15
Prompt
GMSids
0.42
0.26
0.05
0.71
Prompt
VS
0.01
0.01
-0.01
0.04
Resp
AVGSIM
0.07
0.05
0.00
0.17
Resp
GMSIds
0.47
0.30
0.07
0.77
Resp
VS
0.04
0.02
0.01
0.09
Story
AvGSIM
0.12
0.03
0.06
0.18
Story
GMSids
0.36
0.23
0.06
0.62
Story
30

---

D.3 Characterising Text Embedding Spaces
For further experiments, we analyse 16384 embedded documents of from four different HuggingFace
datasets as processed by Wayland et al. [43]. In particular, we use the following datasets:
arxiv: abstracts of all arXiv articles up to the end of 2021;
bbe: summaries of BBC news articles;
cnn: summaries of news articles from CNN and DailyMail; and
patents: abstracts U.S. patent applications.
The first 2 samples from the corresponding training datasets were embedded using the sentence- transformer library by Reimers and Gurevych [30]. The following pre-trained models were used:
all-distilroberta-ll:: general-purpose purpose model, embedding dimension 768:
L----iniLM--6--2:: general purposse model, embedding dimension 384;
cal-mppettbuse-VVe
model, embedding dimension 768; and
QA-specialized model, embedding dimension 768, maximum sequence length 512 word pieces.
multi-gaadditillbet--ee
Further, embeddings from large language models, ada-002 (embedding dimension: 536) and mistral-embed (embedding dimension: 024) were obtained through queries via the corresponding APIS of their providers (OpenAl and MistralAI, respectively).
We then use PCA to reduce each embedding space to 384 dimensions to obtain a comparable dimensionality. This is done to mitigate some of the influence the difference in dimensionalities can have on the results the analysis. Further, we sample 300 points at random from each space, repeating this procedure 200 times, which yields dataset of 1000 embeddings generated by different models for each dataset. We further compute MAGAR REA across 20 scales up to the median convergence point across all embeddings per dataset again using cosine distances and setting we take the pairwise magnitude differences MAGDIFF between all subsample's 's magnitude functions. This is compare it to alternative diversity measures, VS, GMSTDS and AVGSIM computed as defined in Appendix C.I. For each dataset we then use a simple 5-NN classifier to classify the embedding model based on the one number summaries such MAGAREA and report the mean and standard deviation of the accuracy across 5-fold cross- cross-validation with 20 repetitions. We compare this to the more expressive descriptor MAGDIFF, which we similarly use as pre-computed distance matrix for 5-NN classification. Figure S.13 illustrates the pairwise magnitude differences between all subsamples for each dataset. Table 2 then shows the results of this classification task. For further sensitivity analysis, Table S.5 shows analogous results that were computed across varying choices of & neighbours for i--N classification. Results demonstrate that classification accuracy remains consistent across varying parameter choices.
0.05| X Similarly,
Table S.5: Classification performance remains consistent across varying choices of l: for k-NN classification. We show the accuracy (1) of different diversity scores for distinguishing between six embedding models of the bbc dataset, using PCA pre-processing and k-NN classifier across varying values of k. These results are analogous to Table 2 in the main text.
k
MAGDIFF
AVGSIM
VS
GMSTDS
1
0.93 + 0.01
0.842
0.72 0.03
0.66 0.03
3
0.94 0.01
0.02
0.02
0.66 # 0.03
5
0.95 0.01
0.04
0.72
0.66 0.03
0.01
0.84
0.66 0.73
9
0.00
0.84
0.66 0.74
11
0.00
0.04
0.04
31

---

bbc
arxiv
ada-002-
120
mistral-embed
100
all-MiniLL-LG-vv2
all-distirroberta-v1
all-mpnet-base-v2
multi-ga-disilbbet-cos--
patent
cnn
ada-002
mistral-embed
all-MiniLM--6-v22
all-dstilroberta-v1
-0
all-mpnet-base-v2
multi-aadiitiibert-oos--
Figure S.13: Pairwise MAGDIFF for subsamples from six different embedding models and four document datasets. For each dataset, magnitude is computed until the median convergence scale. The plot shows the absolute MAGDIFF normalised by this cut-off scale to show the average absolute difference magnitude across the evaluation interval. We see a clear block-wise structure between the pairwise magnitude differences of each subsample highlighting that the magnitude functions of these samples are clearly distinct between different embedding models.
D.4 Image Experiments
We adapt code by Friedman and Dieng [13] to download and process the CIFARI0 test dataset using torchvision. Specifically, we use utility functions from https: /tpp::///itii thub. com/ vertaix/ Vendi-Score available under an MIT licence. Further, these data are embedded using Incep- tion_v3 [37]. We then simulate sequential and simultaneous mode dropping for this embeddings. We compute magnitude for 10 scales sampled uniformly from to until the 95% convergence scale of the reference embedding where each class still evenly represented. The main results in Figure then report the relative changes in MAGDIFF across the evaluation interval. In Figure S.14 we further report the precision and density for the same simulated mode dropping scenarios as explained image embedding experiments. We also show the relative decrease in magnitude computed at three fixed scales at 33%, 50% and 100% the convergence scale of the reference space chosen to explore variation of resolutions. Finally, Figure S.15 summarises analogous results for different variations of the Vendi Score, which has also been proposed to measure mode dropping. We observe that while VS shows steady trends when computed using linear kernel the Laplacian kernel normalised embeddings. However, VS does not perform well at when simply using the Laplacian kernel without preprocessing, which demonstrates that finding the right scale of similarity is also important for VS and it is possible to misspecify the degree of similarity between observations leading to unreliable performance.
32

---

1.0
1.0
0.9
smultanoous
suquettal
0..8
simullamsous
Precsion
0.8
simultanoos
MaG/0.33%trr
vequential
sequential
Density
0.7
Mac((.ar)
sequential
sqquntaal
0.6
0.2
0.4
0.66
0.8
1.0
0.2
0.4
0.0
Weight the first mode
Weight the first mode
Figure S.14: Line plots of the proportion points on the first mode against recall, coverage, relative difference in magnitude at = 0.5 and magnitude function difference relative to the reference across simultaneous sampling vs. sequential sampling. Lines show the mean values of each normalised metric across 20 resamples, shaded areas the standard deviations.
1.0
1.0000006
0.9
Vendi Linear simultancous
L0000002
0.7
sequential
1.00000000
Vendi Laplacian
Vendi Laplacian Norm simultaneous
simultaneous
0.6
0,9889908
sequential
sequential
0.5
0.8
0.4
Weight on the first mode
Weight the first mode
Figure S.15: Line plots of the proportion of points on the first mode against the relative change in different variations of the Vendi Score, VS, across two different mode dropping scenarios. VS is computed using Laplacian kernel without (left) and with normalisation (right) well linear kernel (right). Lines show the mean values of each normalised metric across 20 resamples, shaded areas the standard deviations.
D.5
Graph Embedding Experiments
To assess our new diversity measure's utility for graph generative model evaluation we reproduce the benchmark by Thompson et al. [40] and include proposed reference-based diversity measure, MAGDIFF, to the diversity evaluation benchmark. The code for reproducing this graph evaluation benchmark is available at https: //tts::/eet //gi//ii gighhu.. com/uoguelpppppeppooooooomee mwllmmllrrrr/ee memrime under an MIT licence. Specifically, we conduct this experiment on five graph datascts:
Lobster: A dataset consisting of 100 stochastic graphs generated so that each node is at most 2 hops removed from backbone path and the number of vertices varies between 10 and 100.[7, 40]
Grid: dataset of 100 two-dimensional graphs consisting of 100 400 vertices [7, 24, 40, 46].
Proteins: dataset 918 protein networks. Each vertex is an amino acid and edges connect amino acids that are less than 6 Angstroms away from each other [9]. Only graphs with between 100 to 500 vertices are selected [7. 24, 40, 46].
Ego: A dataset of 757 graphs that are 3-hop networks with 50 to 399 vertices [40, 46). These graphs were extracted from the CiteSeer citation network where nodes represent documents [34].
Community: A dataset with 500 two-community graphs with between 60 to 160 vertices, where each community has been generated using the Erdös-Rény: Rényi model [11] setting equal to half the number vertices and = 0.3. Additional edges amounting to 5% the number of vertices have been added each graph with uniform probability (40, 46).
Further, this experiment uses a Graph Isomorphism Network [45, GINJ architectures as an embedding model and following the procedure by [40] we vary the following hyperparameters for these models:
33

---

We vary the number of layers between .2.3..... 7] and vary the hidden dimensions in the interval [5, 30) with an increment of 5 resulting total of 36 architectures. We repeat the experiments for 5 different random seeds. The experimental setup used to evaluate the evaluation metrics for both mode collapse and mode dropping then is as follows: First P, Pg. so that Pr, the real distribution, is identical to Pg- the generated distribution. Then the perturbation parameter p E [0, 1] introduced, which transforms the generated graph datasets step-wise and increases the dissimilarity (and hence diversity) between the reference and generated datasets. Therefore, we use it proxy to measure the difference in diversity between P. and Pg. To evaluate this decrease in diversity, we compute magnitude for the corresponding graph embeddings across 40 evenly-spaced scales until the convergence scale the reference choosing e 0.05 X .. For precision, recall, density and coverage take the parameter k 5, proposed previously by [28], to ensure a fair comparison. We then normalise all metrics such that their value O when P. = Pg (which is exactly when the degree of permutation is 0). For this, follow the normalisation strategy by [40] and normalise MAGDIFF by the cardinality of each embedding. Next, we vary the parameter p and compute each evaluation metric. We report the Spearman correlation coefficient between each metric and the degree of the perturbation Hence, the value of metric which captures the decrease in diversity accurately should increase with the increase of p, and rank correlation of corresponds for the whole experiment across all datasets are presented in Figure S.16. The violin and boxplots reported in this figure then summarise the distribution of each evaluation measures rank correlation to the degree of perturbation across the random seeds and the aforementioned hyperparameter choices influencing the embedding models. Finally, Figure S.17 investigates the influence the choice of convergence scale has on the results of these experiments and we observe that low values of lead better agreement with the true degree of perturbation. Further, the trends the value of MAGDIFF chosen throughout this study.
ideal metric. Results
are stable across choices of c S 0.05|
34

---

Mode collapse
Mode dropping
Mode collapse
Mode disppiigg
Metrc Valee
MDD Laver Dealiy
kocel
Noo1
Preceson
Covtrge
vil inwe
Nopmitaa
Frecteo
Boce
Covenge
Degree pertor batioon
D4
100
0.0 0.0
0.2
Degree perturbation
0.8
MNO
(a) Results datasets
Burk parmation
Mode collapse
Mode dropping
Mode collagse
Mode dipppnn
Nelt: value
merato
Dematy
ueD meer
MayD4mt
Precaton
Courrage
MHO hear
of of DR
patturbation
36
0.0
62
Dogres porturbation
0.
06
0.0
1.0
(h) Proteins dataset
collapse
collacse
dioppnng
Runk corataiion
Metk value
Desdyy
Denaty
Cone ayo
HAD Uneer
MayDii
KHO thea
04
partubation
D.
3.0
Prec
(c) Grid dataset
Mode
Mode dropping
Mode collanse
Mode dropcing
SWAr value
1.0
Contee
Magletoul
0.5
Dordit
Neck
Corcrap
Prdda
Denait
Conergg
NHO Unear
NighiMbal
perturbrien
perturtation
(d) Community dataset
Mode collass
Mode dropping
Mode collaroe
Mede drugcing
Bank conriation
1.0
1.0
Deratyy
Megiiboul
Dorskr
Goveag
Nop maal
Dondtr
MAD Uner
Nago litaal
Begren of perturbation
Degres
perturbatiion
(e) Ego dataset
Figure S.16: Results for the mode collapse and mode dropping experiments. The patterns for
each of the datasets is similar to the results on the Lobster graphs, which we show in the paper.
35

---

Mode colapes
Vode
Modk Marhoodd Saaca
Nosk droppl Connd
Matherto Medhhnda
Medhorto
Vulhero
Diggri puraatarion
Deggne parto catice
(a) Results for all datasets
Node colapes
Vodx
Mode
Mode dropp
100
607
Siae Shadl
Sala Clann
Mid NerTher
MeDDuea
Deggns partaturions
Degnn parto Satice
(b) Proteins dataset
Wede ollee
Wel cupplng
Morl callage
Miade Misk--hopp
100
Methura
MedDori
Degant potarbuttoos
Oegrn portersstion
(c) Grid dataset
Made ol
Made dogginginn
Made vellete
Mede Aupping
Siadioett Max
Memi
MoDDria
Macbome
Medorma
Muubava
Melbrric
Degace of pertorbition
Degace of perturbutton
(d) Community dataset
Mode on
Made depind
Made callepe
Mede Aupping
45
M5 Ma-0a
Mechee
Mectrr
Metmi
MeDroN
Machraa
Mecheon
Degace of pertortation
Degint of perturbutoon
(c) Ego dataset
Figure S.17: Rank correlation between MAGDIFF and the degree of perturbation for different
choices of convergence scale. Here we vary the choice of influencing the reference scale that is chosen to compute (Ix) for the mode collapse and mode dropping experiments. We clearly observe that low values of as given by MAGDIFF95 or MAGDIFF99 lead to higher rank correlation and better agreement with the true decrease in diversity.
36

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: Yes. the main claims made throughout the paper are supported by both theoretical discussion (in Section 3.1 and Appendix C.3) and experimental results (Section 4) as motivated throughout the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Yes, discuss the main limitations of our approach discussing both com- putational performance as well as our assumptions on the utility of using embeddings for diversity evaluation in Section 3.5. Properties and theoretical assumptions of our proposed method are further investigated in Appendix C.3.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play import- ant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
37

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
Justification: Yes, all theoretical contributions have been carefully checked and referenced as for example reported in Section 3 and Appendix A.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Yes, all of our contributions are reproducible. Our submission includes the necessary code to compute the algorithms and novel sagnituddebbssee diversity measures introduced and detailed in this work. All our experimental results rely on publicly avail- able datasets and pre-trained embedding models and can be reproduced by following the instructions detailed in our submission e.g. Appendix D.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily the architecture clearly and fully.
model architecture, the paper should describe
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
38

---

(d)
We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: Our submission includes the necessary code compute the al- gorithms and novel magnitude-basee diversity measures introduced and detailed in this work. reproducibility package is available at https: /ttps://gi h///gttuut thub. com/ a aaaaddosslle -abb/ magnitude- -iw.uddddiiersityy and code for computing our algorithms can be found at https: //github. com/ awwaa- las--laabmmgipp We further detail how to access the data for each experiment. Merely the experiment on graph generative model evaluation is omitted, but it can easily be reproduced by including our measure into an existing benchmark (see Appendix D).
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed run reproduce the results. See the NeurIPS code and data submission guidelines (https: //nips. wwtppbccc//ppeeccccccccccccccccc htt///guissssseeeeeee / sminisoicocccc for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: Yes, all necessary experimental settings reported in the main text and expanded on in the supplementary matcrials (Appendix D) and supplementary code, which we also published at https: htt//ggitttt /gggttu..... com/aidos ham//manniiuud -ei--diiiiisyy
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
39

---

Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Yes, all our main claims are supported by uncertainty estimates and it is clearly stated how these have been computed.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confid-
ence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative етгог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: All our experiments have been run locally on a personal laptop highlighting fact that they are readily reproducible. We further benchmark the computational times of our proposed method highlighting can be computed a matter of seconds (see Appendix A.5).
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics attps://neeut wttpuuur..s..ppppieeeeeeeeeeeeee httpppuulc//hhhcsu....ie
Answer: [Yes]
Justification: Yes, our work complies with the ethics guidelines.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
40

---

The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our work has little potential for negative impact and includes impact statement.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation. generating fake profiles, surveillance), faimesss considerations (e.g., deploymeet of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out, For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our contributions pose no such safety risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
41

---

Justification: We clearly cite each existing assets our results build upon. We publish our own code and implemented methods in our Python package MAGNIPY under a BSD 3-Clause licence and further respect all licences for existing assets such datasets or benchmarks used for our experiments. The code available https: //gi::///ttte git..b..tt com/ aidoss- lab/ magnipy.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- se-packaedd, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes)
Justification: We publish our proposed algorithms as a reproducible Python package, named MAGNIPY, under a BSD 3-Clause licence including detailed documentation. The code is available at https: /ttts::/eee //gi githuu...bb.... com/ aidos awwaaaiddooosaaaaaeaaaaaeaaaaaaa saab/nagniy...
Guidelines:
The answer NA means that the paper does release assets.
Researchers should communicate the details the dataset/code/mmodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: Neither crowd- nowd-soureing nor research with human subjects has been employed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
42

---

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Neither crowd-soureing nor research with human subjects has been employed.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
43


Reviews:
[{"review_id": "6BjJGKRh0u", "reviewer": "Reviewer_vep5", "summary": "In this paper, authors introduce magnitude of a metric space to evaluate the diversity of the learned latent representation.\n\nExtensive experimental results are proposed, which to some extent illustrate the effectiveness of the proposed criterion.", "questions": "1. More theoretical comparison between existing methods will make the results more convincing.  Is the proposed method superior over other counterparts theoretically?\n\n2. Can the proposed criterion combine with the existing methods to provide more comprehensive description of the learned representation and come up with better performance. Is it compatible with the existing methods?", "limitations": "See the questions and weakness of the paper.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The expression of the paper is easy to follow.\n\n2. The solving problem is important and could have good impact in the community.\n\n3. The proposed method has shown good theoretical and empirical performance to some extent, illustrating the effectiveness of the proposed measurement.\n\n4. The experimental results are comprehensive.", "weaknesses": "1. The theoretical analysis lacks comparison.\n\n2. More discussion should be conducted over the circumstances when the proposed algorithm performs not as good as the existing algorithms to give more understanding of the pros and cons of the proposed criterion. \n\n3. It is welcomed if a case study can be conducted to illustrate more insight of the proposed criterion.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "> The theoretical analysis lacks comparison.\n\n> More theoretical comparison between existing methods will make the results more convincing. Is the proposed method superior over other counterparts theoretically?\n\nWe appreciate the interest in discussing theoretical analyses in comparison to existing algorithms and briefly summarise our results (we will further improve this in our revisions): Section 3.1 and Appendix C.3 discuss the theoretical properties and fundamental axioms of diversity in comparison to existing baseline measures of intrinsic diversity. Further, Appendix C.4 demonstrates which diversity axioms hold for which of these measures. For revisions, we extend this investigation of theoretic properties to a simulation study reported in Figure 1 of the attached PDF. Our magnitude-based diversity score is the **only approach** that fulfils all desired criteria, showing  that our method is superior to alternative diversity measures from a theoretical perspective. We will highlight this theoretical discussion in revisions and emphasise the importance of fulfilling fundamental axioms of diversity via an extended simulation study. \n\n___\n\n\n> More discussion should be conducted over the circumstances when the proposed algorithm performs not as good as the existing algorithms to give more understanding of the pros and cons of the proposed criterion.\n\n**Throughout our experiments on state of the art diversity evaluation benchmarks, we have not encountered a scenario, where our methods perform worse at measuring diversity than existing methods.**\n\nIn terms of the cons of our method, we noted the main limitations of our approach in Section 3.5:\n* Our method does not assess fidelity.\n* Scalability w.r.t to the size of embeddings.\n\nWe note that practitioners can assess fidelity using specialised scores. Our method can then be interpreted **alongside** fidelity metrics, covering diversity aspects not yet addressed by existing measures. \n\nFurther, scalability was not a limitation in our experiments. Relevant diversity evaluation tasks typically study small graph datasets, evaluate the response of text generation models for specific tasks, or study image embeddings in terms of meaningful subsets (like measuring intra-class diversity). However, in case scalability becomes an issue, we can work with efficiently approximations of our score, based on subsets.\n\nWe believe that the advantages of our method by far outweigh its limitations.\nTo sum up, the main benefits of our methods are:\n* Their agreement with fundamental axioms of diversity.\n* Their expressivity.\n* Their multiscale nature.\n* Their flexibility (w.r.t choosing a dissimilarity).\n* Their connection to geometric properties.\n\nWe thus believe that magnitude provides a hitherto-unaddressed perspective on diversity that complements additional evaluation measures. For revisions, we will include a broader discussion on the pros and cons.\n\n\n___\n\n\n\n> It is welcomed if a case study can be conducted to illustrate more insight of the proposed criterion.\n\nWe agree it is important to build this intuition! For revisions, we provided an **extended case study in the attached PDF**, showing a comparison of our proposed magnitude-based approach and alternative measures.\n\nTo link our investigation to the theoretical axioms of diversity, we examine the so-called twin property. This requirement asserts that diversity should not change when including duplicate observations into a given dataset. When evaluating generative models, diversity measures that satisfy the twin property are advantageous because they penalise models that just repeat existing observations again, as opposed to providing genuinely “novel” outputs.\n\nResults of this case study are reported in Figure 1 of the attached PDF, showing how the popular baseline measures **all fail to fulfil** the twin property, instead exhibiting  highly-inconsistent behaviour. Our proposed method meanwhile is the **only** diversity measure that respects the twin property and remains consistent, demonstrating another one of its practical advantages. \n\nTo gain more insights, we further compare diversity metrics on the examples from Figure 1 and report them in Table 1 of the attached PDF. The results show that two of the baseline measures fail to capture notable differences in diversity on simple simulations as they do not detect that the random pattern in $X_1$ is more diverse than a clustered pattern in $X_2$.\n\n\n___\n\n\n\n> Can the proposed criterion combine with the existing methods to provide more comprehensive description of the learned representation and come up with better performance. Is it compatible with the existing methods?\n\nIn general, our method is compatible with existing scores. Our proposed diversity measure can be used to evaluate and improve the **performance of generative models**. This allows us to choose a generative model that best captures the diversity of a reference (achieving low MagDiff values) while simultaneously retaining high fidelity scores. Section 4.5 explores this scenario for evaluating graph generative models; we will focus on combinations of existing scores with our score in future work, thanks for the suggestion! \n\nOur measure is very versatile and can be extended to other settings to improve model performance via **incorporating it into the model training**, for instance 1) for optimisation purposes, as part of the loss function; 2) as an early stopping criterion for monitoring the diversity of the learnt representations; or 3) by improving how well the learnt representation preserves the ground truth diversity of a known reference. We would be excited to further explore these scenarios in future work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bNo55IoiVf", "reviewer": "Reviewer_cXLQ", "summary": "This paper focuses on evaluating the diversity of latent representations. The authors develop a family of magnitude-based measures of the intrinsic diversity of latent representations, formalizing a novel notion of dissimilarity between magnitude functions of finite metric spaces. Moreover, they demonstrate the practicality and performance of the proposed measures in different domains and tasks.", "questions": "1. In Table 1, what is the reason for the apparent performance difference between MAGAREA with the piecewise linear and MAGAREA using quantile regression?\n2. In section 4.3, what is the basis for the author to use the 5-NN classifier to predict the embedding model?\n3. In section 4.3, what is the purpose of the author's comparison of the results of PCA pre-processing and no pre-processing? How do the authors view that these two methods have both positive and negative effects on different models?\n4. In Section 4.5, the authors discussed magnitude evaluation graph generative models. Can the diversity evaluation proposed in this paper also be used to evaluate the representation learning ability of graph contrastive learning models?", "limitations": "1. MAGDIFF is a reference-free measure of intrinsic diversity, but does not measure fidelity.\n2. The paper does not investigate whether embedding-based similarities are outperformed by alternative task- or domain-specific similarities. Instead, the evaluation relies on the utility of embedding models and assumes that latent spaces encode useful/realistic relationships between samples.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The method proposed in this paper is innovative, and the writing is logical.\n2. This paper conducts an in-depth theoretical analysis and sufficient experimental discussion and analysis.", "weaknesses": "1. In Section 4.3, the reason for choosing the 5-NN classifier needs to be explained. In addition, what is the purpose of designing a comparative experiment between PCA pre-processing and no pre-processing?\n2. This paper discusses several application scenarios of the proposed diversity evaluation, and we can further explore whether there are more scenarios worth exploring, such as measuring the representation ability of graph contrastive learning models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback and are looking forward to discussing the questions raised as well as clarify them in the text during revisions.\n\n___\n\n> In section 4.3, what is the basis for the author to use the 5-NN classifier to predict the embedding model?\n\nThe 5-NN classifier is chosen as a very simple model, which somewhat surprisingly already manages to separate the embedding spaces of different models extremely well based on their difference in intrinsic diversity. Here, 5 neighbours are chosen as a default as implemented in the $\\texttt{sklearn}$ package without any hyperparameter tuning. Motivated by your question, we conducted further sensitivity analysis to assess how much the reported results change under different parameter choices. Table 2 in the attached PDF then shows that classification accuracy hardly varies across different choices of $k$ neighbours. For revisions, we will add an explanation for the choice of model in the text.\n\n___\n\n> In section 4.3, what is the purpose of the author's comparison of the results of PCA pre-processing and no pre-processing? How do the authors view that these two methods have both positive and negative effects on different models?\n\nPCA preprocessing is applied under the assumption that the varying dimensionality between embedding spaces could be the main driver of the observed disparities in diversity. We thus wanted to test if dimensionality reduction to the same number of 384 dimensions could account for the differences in diversity and reduce the classifiers’ performance scores. However, our results show that this does not change the experimental results. There thus remain inherent differences in each models’ signature as distinguished via diversity. We will include further explanations and motivations for these model choices in the revisions. **We also want to clarify that our method does not depend on such a specific pre-processing; we merely used it here to make the task of detecting a specific model harder.**\n\n___\n\n\n> In Table 1, what is the reason for the apparent performance difference between MAGAREA with the piecewise linear and MAGAREA using quantile regression?\n\nThank you for pointing this out! We will clarify this in revisions: The input data to these models, that is the values of MagArea for different curvature values, is plotted in Figure S.5. Explanatory analysis of this relationship then informed our model choice. The piecewise linear model better fits the trend in Figure S.5, which is why it outperforms the quadratic relationship modelled via quantile regression. Both models were included to offer multiple proposals on how to interpolate between the MagArea scores for surfaces of negative and positive curvature. \n\n___\n\n> In Section 4.5, the authors discussed magnitude evaluation graph generative models. Can the diversity evaluation proposed in this paper also be used to evaluate the representation learning ability of graph contrastive learning models?\n\nWe appreciate this suggestion and are excited to explore and discuss more scenarios for which our proposed method is of relevance. Evaluating representation learning models themselves is an exciting question in itself that deserves further exploration. \n\nIn the **context of graph contrastive learning**, we have reasons to believe that our method can be extended to the evaluation of self-supervised representations. As an intrinsic measure of the diversity of a space, magnitude measures the effective size of an embedding space. Conceptually, this can be likened to measuring the effective rank of an representations as computed by [RankMe (Garrido et al. 2023)](https://arxiv.org/pdf/2210.02885) for the evaluation of joint-embedding self supervised learning. That is, we believe that problems such as **dimensional collapse** could also be assessed in an expressive manner by a representation’s multiscale magnitude. \n\nFurther, it is of interest to explore the role of diversity in a self-supervised context and investigate how diversity measures can be effectively used to improve model performance, potentially via **maximising diversity during training**. \n\nWe thus believe that using magnitude for evaluating representation models deserves its **own thorough investigation** as well as further theoretical links to learning theory in SSL, which we look forward to conducting in the future.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9UZkiHTIfq", "reviewer": "Reviewer_WoGi", "summary": "This paper studies the diversity in representation learning. The authors propose a novel family of diversity measures based on metric space magnitude, a mathematical invariant that captures numerous important multi-scale geometric characteristics of metric spaces.\n\nThe main contributions are as follows:\n1. The authors first introduce magnitude as a general tool for evaluating the diversity of latent representations,\n2. The authors formalise a notion of difference between the magnitude of two spaces across multiple scales of similarity.", "questions": "See weakness.\n1. Can authors better explain my confuses? I think the proposed approach is more like to evaluate the better representation instead of measuring the diversity.\n2. I didn't see clearly connections between magnitude and curvature. Can authors explain clearly?", "limitations": "See weaknesses.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The idea of introducing magnitude as a general tool for evaluating the diversity of latent representations is quite attractive and it seems can be generalized.\n2. The authors formalise a notion of difference between the magnitude of two spaces across multiple scales of similarity. \n3. The authors discuss the theoretical properties a suitable diversity measure should satisfy.", "weaknesses": "1. As mentioned in the paper, it is difficult to define the diversity. The paper does not clearly explain how to link their method with diversity. Maybe I did't get their point. But I think their method is more like to evaluate the better representation instead of measuring the diversity?\n2. The authors demonstrate that magnitude is stable and can detect curvature, however such connection makes people a little confused.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their questions and the positive assessment of our work. \n\n**For revisions, we take this feedback as motivation to clarify and further explain the links between magnitude and diversity.**\n\n___\n\n\n> As mentioned in the paper, it is difficult to define the diversity. The paper does not clearly explain how to link their method with diversity. Maybe I didn't get their point. But I think their method is more like to evaluate the better representation instead of measuring the diversity?\n\nThanks for raising this question about our current write-up! We will use this to revise our manuscript. To summarise our main points: \n\nIn our paper, we define diversity by considering fundamental axioms which a sensible diversity measure should satisfy, namely monotonicity in distance (diversity should increase as dissimilarity between observations increases) and the twin property (diversity should not change when including duplicate observations). We state these main theoretical requirements, and demonstrate that our proposed measures fulfil them. In this way, we provide the theoretical link between diversity and our methods.\n\nSpecifically, we use the magnitude of a latent representation to evaluate diversity. Magnitude measures diversity as the “effective number of points” at a distance scale or “zoom factor.” Given a distance between data points, and a zoom factor, magnitude answers the question: How many *distinct* points or clusters can be distinguished at this scale? \n\nIntuitively, diverse spaces have clearly separated observations and high magnitude. In comparison, less diverse spaces, whose observations are more clustered, have lower magnitude. We then define formalised measures of diversity that summarise trends in magnitude across multiple resolutions.\n\nThis is further illustrated in Figure 1 in the paper. Here, the random pattern in $X_1$ is the most diverse example because its points are most clearly separated and more evenly spread across the entire area. Thus, it achieves the highest values of magnitude. In comparison, the clustered pattern in $X_2$ generated via a Hawkes process is less diverse because it shows a more uneven distribution with points being concentrated at specific areas. Overall, the magnitude of $X_2$ is thus lower than for $X_1$. Our diversity evaluation pipeline illustrated in Figure 1 then summarises these differences in diversity across multiple resolutions in a principled manner.\n\nIn the context of representation learning, diversity then is a key concept on what it means for representations to be better. Our goal thus is to evaluate latent representations by measuring their diversity. **In revisions, we will revise the introduction of magnitude and diversity to better explain what diversity means in our context and how our method measures diversity. Further, we will add an extended case study to the appendix in order to illustrate how our method measures diversity via intuitive examples.**\n\n\n___\n\n\n\n> I didn't see clearly connections between magnitude and curvature. Can authors explain clearly?\n\nWe thank the reviewer for this question. The motivation of the curvature experiment was to substantiate our argument that magnitude encodes important geometrical properties with experimental evidence. Our results then support and are motivated by the theoretical results linking magnitude and curvature ([Willerton 2010](https://arxiv.org/pdf/1005.4041)).\n\n**At the same time, this experiment shows that magnitude is more effective at this task than other, more complex geometrical summaries** (like persistent homology).\n\nFor the experiment itself, we can intuitively explain the relationship between diversity i.e. magnitude and curvature as illustrated in Figure S.5. For unit disks of positive curvature, the higher the curvature the lower the value of MagArea. This indicates that points move closer and closer the more curved the surface is decreasing the diversity in Euclidean space. \n\nFor surfaces with negative curvature we see the opposite trend. The more negatively curved the Poincaré disk the lower the value of MagArea. This is because Euclidean distances between points and thus diversity are decreasing. \n\nIn revisions, we will clarify the connection between magnitude, diversity and curvature. To do so, we will reference the relationship to relevant theoretical results in the text. Further, we will add an introductory text earlier in the paper to motivate this experiment.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their unanimous support of our work and contributions, as well as for their interesting questions and suggestions for further clarifications. We are confident that we can implement the required changes mentioned below by making **small amendments** to our manuscript. Moreover, to complement these changes, we address reviewers’ specific questions in our responses to their individual reviews, and we are happy to provide further clarifications in the discussion phase.

Meta-review: The paper successfully proposed a reference-free representation diversity metric family. Reviewers agreed that the paper tackled important problems, the paper's presentation was clear, and the idea was sufficiently supported by the experiments.

The Area Chair suggests that there might be room for improvement in justifying the proposed metric. The Authors listed four axioms for a metric to satisfy:

- **Monotonicity in observations**: Including a new observation does not decrease diversity.
- **Twin property**: Including a duplicate observation already in the set does not change diversity.
- **Absence invariant**: Diversity only depends on the samples and features present in the dataset.
- **Multi-scale**: Diversity encodes both local and global trends in the data manifold.

However, the first three are automatically satisfied if we take the sum $\\sum\_{i,j \in X} f(d(x\_i,x\_j))$ for **arbitrary** non-increasing function $f$. Also, it considers all the pairs, so it considers both *local* and *global* trends in some sense. Hence, it could be better to refine the axioms in the desiderata section. Perhaps the Authors prefer to add an additional axiom. 

Nevertheless, the Area Chair judged that the above presentation issue was minor, which led to the decision this time.
Decision: Accept (poster)