Intrinsic Dimension for Large-Scale Geometric Learning
Authors: Maximilian Stubbemann
Tom Hanika
Friedrich Martin Schneider
Venue: Accepted by TMLR ()

Abstract: The concept of dimension is essential to grasp the complexity of data. A naive approach to determine the dimension of a dataset is based on the number of attributes. More sophisticated methods derive a notion of intrinsic dimension (ID) that employs more complex feature functions, e.g., distances between data points. Yet, many of these approaches are based on empirical observations, cannot cope with the geometric character of contemporary datasets, and do lack an axiomatic foundation. A different approach was proposed by V. Pestov, who links the intrinsic dimension axiomatically to the mathematical concentration of measure phenomenon. First methods to compute this and related notions for ID were computationally intractable for large-scale real-world datasets. In the present work, we derive a computationally feasible method for determining said axiomatic ID functions. Moreover, we demonstrate how the geometric properties of complex data are accounted for in our modeling. In particular, we propose a principle way to incorporate neighborhood information, as in graph data, into the ID. This allows for new insights into common graph learning procedures, which we illustrate by experiments on the Open Graph Benchmark.

Published in Transactions on Machine Learning Research (01/2023)
Intrinsic Dimension for Large-Scale Geometric Learning
Maximilian Stubbemann
Knowledge 8 Data Engineering Group, University of Kassel, Kassel, Germany
Tom Hanika
wwm.hannkaacc.uui-ikkaooo de
Knowledge & Dota Engineering Group, University of Kassel, Kassel, Germony
Friedrich Martin Schncider
Aaaaterrraaahhuuu hat--rerbbrr.... de Institute of Discrete Mathematics and Algebra, TU Bergakademie Freiberg, Freiberg, Germany
Reviewed on OpenReview: https: // openreview. net/ forum? id- B5BBDDDYYMBY
Abstract
The concept of dimension essential grasp the complexity of data. naive approach to determine the dimension of a dataset is based on the number of attributes. More so- phisticated methods derive a notion of intrinsic dimension (ID) that employs more complex feature functions, e.g., distances between data points. Yet. many of these approaches are based on empirical observations, cannot cope with the geometric character of contempo- rary datasets, and do lack an axiomatic foundation. A different approach was proposed by V. Pestov, who links the intrinsic dimension axiomatically to the mathematical concentra- tion of measure phenomenon. First methods to compute this and related notions for TD) were computationally intractable for largo-scale real-world datascts. In the present work, derive computationally feasible method for determining said axiomatic ID functions. Moreover, demonstrate how the geometric properties of complex data are accounted for in our modeling. In particular, we propose a principle way to incorporate neighborhood ion, as in graph data, into the ID. This allows for new insights into common graph learning procedures, which we illustrate by experiments on the Open Graph Benchmark.
1 Introduction
Contemporary real-world datasets employed in artificial intelligence are often large in size and comprised of complex structures, which distinguishes them from Euclidean data. To cousider these properties appropri- ately is a challenging task for procedures that analyze or learn from said data. Moreover, with increasing complexity of real-world data, the necessity arises to quantify which extent this data suffer from the curse of dimensionality. The common approach for estimating the dimension curse of a particular dataset is through the notion of intrinsic dimension (ID) (Bac & Zinovyev, 2020; Granata & Carnevale, 2016; Pestov, 2007). There exists variety of work on how to estimate the ID of datasets (Facco et al., 2017; Levina & Bickel, 2004; Costa et al., 2005; Gomtsyan et al., 2019; Bac & Zinovyev, 2020). Most approaches to quantify the ID are based on distances between data points, assuming the data to be Euclidean. A multitude of works Klock, 2021; Gomtsyan et al., 2019), which
base their modeling on the manifold hypothesis (Cloninger assumes that the observed data is embedded in manifold of low dimension (compared to the number of data attributes). The ID then is approximation of the dimension of this manifold. Pestov (2000) pro- posed different concept of intrinsic dimension by linking it to the mathematical concentration of measure phenomenon. His modcling is based on thorough axiomatic approach (Pestov, 2007; 2008; 2010) which resulted in a novel class of intrinsic dimension functions. In contrast to the manifold hypothesis, Pestov's ID functions measure to which extent a dataset affected by the curse dimensionality, i.., to which extent the complexity of the dataset hinders the discrimination of data points. Yet, to compute said ID functions is an intractable computational endeavor. This limitation was overcome principle by an adaptation to geo- metric dalasets (Hanika et al., 2022). However, two limitations persisted: First, the computational effort

---

Published in Transactions on Machine Learning Research (01/2023)
found to remain quadratic in the number of data points, which is insufficient for datasets of contemporary size; second, it is unclear how to account for complex structure, such as in graph data.
With this in mind, we propose in the present work a default approach for computing the intrinsic dimension of geometric data, such as graph data, as used in graph neural networks. To do this, we revisit the compu- tation of the ID based on distance functions (Hanika et al., 2022) and overcome, in particular, the inherent computational limitations in the works by Pestov (2007) and Hanika et al. (2022). In detail, we derive a novel approximation formula and present an algorithm for its computation. This allows to compute ID bounds for datasets that are magnitudes larger than in earlier works. That equipped with, we establish natural approach to compute the ID of graph data.
We subsequently apply our method to seven real-world datasets and relate the obtained results to the observed performances of classification procedures. Thus, we demonstrate the practical computability of our approach. In addition, we study the extent to which the intrinsic dimension reveals insights into the performance of partieularly classes of Graph Neural Networks. Our code is publicly available on GitHub.
2 Related Work
In numerous works, the intrinsic dimension is estimated using the pairwise distances between data points (Chávez el al., 2001; Grassberger & Procaccia, 2004). More sophisticated approaches use distances lo nearest neighbors (Facco et al., 2017; Levina & Bickel, 2004; Costa et al., 2005; Gomtsyan et al., 2019). All these works have in common, that they assume the data to be Euclidean and that they favor local properties.
Recent work has drawn different connections between intrinsic dimension (ID) and modern learning theory. For example, Cloninger & Klock (2021) show that functions of the form f(z) g()(x)), where ø maps into manifold of lower dimension, can be approximated by neural networks. On the other hand, Wojtowytsch & E (2020) prove that modern artificial neural networks suffer from the curse of dimensionality in the sense that gradient training on high dimensional data may converge insufficiently. Additional to these theoretical results, there is an increasing interest of empirically estimating the ID of contemporary learning architectures. Li et al. (2018) study the ID of neural networks by replacing high dimensional parameter vectors with lower dimensional ones. Their approach results in a non-deterministic ID. More recent works studied ID in the realm of goometric data and their standard architectures. Ansuini et al. (2019) investigate the ID for convolutional neural networks (CNN). In detail, they are interested to which extent the ID changes at different hidden layers and how this is related to the overall classification performance, Another work (Pope et al., 2020) associates an ID to popular benchmark image datasets. These two works on ID estimators do solely rely on the metric information of the data and do not consider any geometric structure of image data.
Our approach allows to incorporate such underlying geometric structures while incorporating the mathe- matical phenomenon of measure concentration (Gromov & Milman, 1983; Milman, 1988; 2010). Linking this phenomenon to the occurrence of the dimension curse was done by Pestov (Pestov, 2000: 2007: 2008; 2010). He based his considerations on thorough axiomatic approach using techniques from metric-measure spaces. The resulting ID functions unfortunately turn out to be practically incomputable. In contrast, Bac & Zinovyev (2020) investigate computationally feasible ID estimators that are related to the concentration phenomenon. Yet, their results elude a comparable axiomatic foundation. Our modeling for the ID of large and geometric data based on Hanika et al. (2022). We build on their axiomatization and derive a computationally feasible method for the intrinsic dimension of large-scale geometric datasets.
3 Intrinsic Dimension
Since our work is based on the formalization from Hanika et al. (2022), we shortly revisit their modeling and recapitulate the most important structures. Based on this, we derive and prove an explicit formula to compute the ID for the special case of finite geometric datasets. This first result is essential for Section 4.
'https://github. http://hhh....cco//tttttebeeee/p
2

---

Published in Transactions on Machine Learning Research (01/2023)
Let D= (., where X is set and F S RX is set of functions from X to R, in the following called Seature functions. We require that y dr(x, v) < oc, where dr(x, == supjeF [f()) - s(y)l- If (X, dd) constitutes a complete and separable metric space such that μ is a Borel probability measure on (X, d we call D a geometric dataset (GD). The aforementioned propertics are satisfied when it holds that [ < oo and that can discriminate all data points, i.e., dr(a, 3) for all x, E X with x # y.
Two geometric datasets D1 (X,, f1), D2 F2, (2) are isomorphic if there exists a bijection o Xi X2 such that F2 o Fi and (,(1) M2, where o. (B)) = (i(o=1(B)) is the push-forward measure and the closures are taken with respect to point-wise convergence. From this point geometric dataset with its isomorphism class. The triple (10), RR v(dy) represents the trivial geometric dalasel.
identify
Pestov (2007) defines the curse of dimensionality as "[...] a name given to the situation where all or some of the important features of datasct sharply concentrate near their median (or mcan) values and thus become non-discriminating, In such cases, X is perceived as intrinsically high-dimensional. Thus, the ID estimator aims to compute to which extent the features allow to discriminate different data points. For specific feature/ f F, Hanika et al. (2022) therefore defines the partial diameter of f with regard to specific a E (0, 1) such that it displays to which extent f can discriminate subsets with minimal mcasure 1 - a, i.c., via
PartDiam (f.(), 1-a) inf (df(dia((B) B S R Borel, vB) 2 a),
where diam (B) == supo,bes la-bl. The observable diameter with respect to a then defines to which extent F can discriminate points with minimum measure 1-a by being defined as the supremum of the partial diameter
of all f F, i.e, ObsDiam(() -a) = se Part.Diamff (It), o... To observe the discriminability for different minimal measures a, the discriminability A of D and the intrinsic dimension a(D) are defined via
A(D) = L. ObsDiam -a) da.
O(D) = A(D)2' where
(1)
In other words, lower values of intrinsic dimensionality correspond to geometric datasets with points that can be better discriminated by the given set of feature functions. This intrinsic dimension function is, in principle, applicable to a broad variety of geometric data, such as metric data, graphs or images. This applicability arises from the possibility to choose suitable feature functions which reflect the underlying data structure. The appropriate choice of feature functions is part of Section 5. Furthermore, the ID O(D) A(D respects the formal axiomatization (Hanika et al., 2022) for ID) functions, informally:
Axiom of concentration: sequence of geometric datasets converges against the constant dataset (meaning having no chance to separate data points!), and only if their IDs diverge against infinity.
Axiom of feature antitonicity: datasel D2 has more feature functions then Do (i.e. having potentially more information to separate data points), it should have a lower intrinsic dimension.
Axiom of continuity: If a sequence of geometric datasets converge against a specific geometric dataset, the sequence of the IDs should converge against the ID of the limit geometric dataset.
Axiom of geometric order of divergence: If sequence of geometric datasets converges against the constant dataset, its IDs should diverge against infinity with the same order as A(DI) does.
3.1 Intrinsic Dimension of Finite Data
We want to apply Equation (1) to real-world data. In the following, let = FX, such that 0 < X < oc and 0 [F]
M and let be the normalized counting measure on X, i.c., v(M) = [X] for M C X. In this case, it is possible to compute the partial diamcter and Equation (1), as we show the following. Let a E (0, 1) and let ca = [] The following arguments were already hinted in previous work (Hanika et al., 2022), yet not formally discussed or proven.
3

---

E Published in Transactions on Machine Learning Research (01/2023)
Lemma 3.1. For F it holds that
PartDiam(fs(()),11 -o)
min max [f(x) - f(y)l. |M|=c, a.ye M
Proof. It holds that
PartDiam(f.(t)) inf(diam(BB) B S IR Borel, v(f 2 1-a)
inf inf(diam (B) B S R [orel. X f(x) E BB)cca)
We have to show that
inf(diam(B) B S E Borel, 1 X f(x) E BY| 2 ca)
(2)
min{ max a.EEM
[f(x) f(y) M C X, |M| 2 cat.
"<:" We show that (diam(B) B C R Borel, ( E X f(x B) ca) 2 (maxz,weM [f()) f(v)) M C x, | ca). Let WM [f()) f()) such that M S X with [M] 2 ca. Without loss of generality we assume that
Vz E X (3m1, m2 E M f(m1) - f(x) - f(m2) = x ZEMI M).
Let b max f(x), a mingeM M fx), then M fo E X f(x) [a, b1}. Hence,
{diam (B) _ R Borel, x1f(x) B) ca).
"2:" Let B C be Borel with [xx E X f(x) E BJ| 2 Ca. Furthermore, let = It holds that (Ba suPa,yEB x yl 2 vE [f(x) f(y) because of the choice of M. As B was chosen arbitrarily, it follows "2"
= (a E f(x) E B).
Finally, we need that
min { max S.EM
[f(z) f(v)l |M 2 ca) minf max 1f(x) f(y) | ca
J.EEM
"<" follows directly from the fact that {max (f) M 2 ca} 2 {max f(z) [M| c. ">" follows from the fact that for every |M| 2 Ca and for every N C M with | = Co the following equation holds: su as (f()) f(y)l suPa.yeN ff()) - f(y))-
This lemma allows for more tractable formula for the computation of the partial diameter of finite GD. That in lurn enables the following theorem.
Theorem 3.2. It holds that
X max min max [f(z)) f(w))-
AD)
(3)
JEF MCX #.E
|M|=
Proof. Let g: (0, + a, maxjer mC max [f(x) f(y)). Because of Lemma 3.1 we know that A(D) Jo g(a) da. The function is step function which can be expressed for each a E (0, 1) via
g(a) Σ
LI
1III v) v)(a) IXT
(a) max min max 1f(x) f(y)l
JEF MCX 2.jEM
Mi=k
almost everywhere. Hence, Equation (3) follows from the definition the Lebesgue Jabbesuu-Intterall with the fact that min mw [f(x) f(w) 0.
In general, the addition of features should lower the ID since we have additional information that helps discriminate the data. However, there are certain features that are not helping to further discriminate data points. These are for example:
4

---

Published in Transactions on Machine Learning Research (01/2023)
1. Constant features. This is due to the fact that for a constant feature f it always holds for all M C X that maXa.eEM (f(z) [(v)] 0.
Permutations of already existing features. Let f: X R have the form f = fo with being permutation X. Then there exist for all M C X with |M] = k set N C X with |N| =
X
and maXa,yEM a [f(x) f()) max [f(x) f()) and vice versa.
Thus, we have the following Lemma.
Lemma 3.3. Let D F.) be a finite geometric dataset. Furthermore, let F be a set of constant functions X + R and let F' be set of funclions X + IR such that there erist for each f E F a f F and Then it holds that @(D) a(E).
permutation X + X with f == for. Let E = (X.
3.2 Computing the Intrinsic Dimension of Finite Data
In this section we will propose an algorithm for computing the ID based on Equation (3). For this, given a finite geometric dataset D, we use the shortened notations , == minMgx,|M]= ya If(x) f(y)
and du (D) == max (D). Then, Equation (3) can be written as
X (D)
XI
A(D)
Σ
max (D).
(4)
X
X
JEF
The straightforward computation of Equation (4) is hindered by the task to iterate through all subsets M S X of size k. This yields exponential complexity with respect to |X] for computing A(D). We can overcome this towards a quadratic computational complexity in |X] using the following concept.
Definition 3.4. (Feature Sequence) For a feature f E F let , RIx| be the increasing sequence of all values f(z) for x E X. We call ,,D (14.D. _(xx) the feature sequence of f.
Using these sequences, the following lemma allows us to efficiently compute (D.. Lemma 3.5. For l E (2,...,|X]] | f.
F and ,,D, it holds that
minn k-j 12 1 (O..... x1-
min
Proof. For all j (0.....|xX there exist M C X with |M = k and IID - -1- = max 1f(x) -v)) Thus, it is sufficient to show dx. (D) e -44--11 1 (O..... [x|- k+j that (D) maxa.ee [f(x) f[xx)f(()) holds. Furthermore, let a = (IM ,I.....,) be the increasing sequence of values (m) for E M and let {o....,/Xx" |X| k} such that 1/M 1441' Since 1 is ordered sequence of which each element is also an element of the ordered sequence I,.,, it holds that 1/ > I.P and thus TM - I 2 14-2 - 1/-2 There is an N X with size k such that max (f(x) [ = 1/ - I.f.D
ktj
Choose M S X with |M| k such IM)
k+1- Since
M X is of size le such that y [f(z) ((x))f(()] is minimal, it follows 1/ -1M max ff <
WEN (f - f(y) = I.D 1: hence Φx.r (D) = ma [f(x) - f()) |= = 1/ - f.D - - k+1
F.D
To sum up, Lemma 3.5 enables the efficient computation (D) via a sliding window, ie., by using 14:2). The algorithm based on this is shown in Algorithm 1. We want to provide a brief description of the most relevant steps. In Line 4 we iterate through the sizes of X by setting k (2..... [X]] in order to compute ox (D) in Lines 6 and 7. For this we also need to iterate over all f (Line 5) to compute the necessary values of (D)) in Line 6. For a given I E F, k E (I...., [X],, Line 6 consumes X - subtraction operations. Assuming that computing feature values can be done [X|
only pairs of elements (4/2).1447
in constant time, the runtime for computing A(D) from the feature sequences is O(JFI Σ k= [X| - k + 1) (IF X k) O(F (xP)) The creation of all feature sequences requires | log((X))) computations which negligible compared the aforementioned complexity. Thus, Algorithm 1 has quadratic complexity with respect to X|. Therefore, Algorithm 1 is straightforward and easy to implement
5

---

Published in Transactions on Machine Learning Research (01/2023)
Algorithm 1: The pscudocode to compute A(D for a finite geometric dataset D = (X, pl, F).
Input -Finite geometric dataset D = (X, 14, F).
Output: A(D)
forall f in F do
Compute feature sequence I,D
A(D)= =0
forall k in , ] do
forall f in do
I1.D 19-3. (D, 10.
A(D)+ f ok (D)
(DD | A(D
return A()
solution for the computation of the ID. However, its quadratic runtime is obstructive for the application in large-scale data problems, which raises the necessity for a modification. We will present such a modification in the following section.
4 Intrinsic Dimension for Large-Scale Data
In order to speed up the computation of the we modify Algorithm with regard to the accuracy of the result. Hence, we settle for an efficiently computable approximation of the ID, To give an overview over the necessary steps, we will
approximate the ID) by replacing ( Line of Algorithm with a smaller subset S C f2...., [X], which we represent by S (si...., st). For all & & S, we will use føs, (D)..... Φ. (D)} to estimate 6u (D). This will eventually lead to two approximations of the ID, an underestimation and overestination.
compare the upper and lower approximation provide an error bound of these approximations with respect to the exact ID. This error bound can be computed without knowing the exact ID.
3. argue how, the computation of the exact ID can be sped up with the help of knowledge about Φ.,, (D) for all 8; E S. For this, we will in particular show that we can replace for all E (2..... [ \ M the set with a subset F, see Line 5-6 of Algorithu 1.
4. derive a formula which estimates the amount of computation cost which is saved by using only subsets of F for the computation of the ID. This information can be used to estimate and decide whether the exact computation of the ID computational feasible for a specific dataset.
The ensuing algorithm is shown in Algorithm 2. The underlying theory that justifies it is presented in the following. This theory will be based on the monotonicity of i (D). Theorem 4.1. For m > n 2 2 and f E F the following statements hold.
1. o 2 (
(D) 2 (D))
Proof. The second inequality follows directly from the first one. Since per definition Pm.y(D)
min minjcxx max,,9MM/ [f() f(y) and also (D) = min max If(x) - -f()),, we nood
to show that for each M S X with [M = m there exist N X with [N] = and max f(z) f 2
max 1f(z) [(y]]- It is sufficient to show that for
1. Choose 31,32 E M such that

---

Published in Transactions on Machine Learning Research (01/2023)
Algorithm 2: The pscudocode to compute A,
for finite GD D (X, F).
Input Finite GD D (X, u, F), support sequence (2 11 31 [x]), ezact (Boolean) Output:
forall in F do
Compute feature sequence J.D.
A(D)=1 =
8o 1
2s0 (D) =0
forall i in (1.....I} do
// Iterate over support sequence indices
forall f in do
L (D) minje (0.,, sitj
I.D
sitj - 444
// Compute (D) with Lemma 3.5
(D) maxjEr Pe., (D)
10
F = {f F @mn, (D) 1
// Compute Fs for Lemma 4.7
11
A = =b,, (D)
12
(D) -
(A(D) EE
E, Sjenn 0n, (D))
// Compute lower ID from Definition 4.2
13
(D) = (A(D + + Encjaan 0 (D))
// Compute upper ID from Definition 4.2
// Approximation finished.
Continue with exact computation, if desired.
14 if enacl then
15
forall i in {1....(} [ do forall 8; < < 8;-1 do
16
// Iterate through all indices between two support elements
forall f in Fs, do
// Only iterate through Fe, because of Lemma 4.7
L (D) = min.eeoo.... ( *ttj
I.D - IfD
18
14j"
19
(D) max(fe;, ff (o.,))
// Use Lemma 4.7
20
(D) = (D)
A(D) X A(D)
22
return A. (D) A(D)
return As, (D)
23
WE (f(x) - [(w) [f(]) f(x2)) As [M| 2 we find 13 E M \ 121, Let N = :M (ra). It holds that M ff(x) - f(y) If(x) - f(22)l max If(x) f(y)].
]
4.1 Computing Intrinsic Dimension via Support Sequences
Equipped with Theorem 4.1, we can bound A(D) and thus the intrinsic dimension through computing 6x. for fow 2 = $1 < 82 < = [X].
Definition 4.2. (Support Sequences and Upper / Lower ID) Let (2 $1, $i...... S |X] be strictly
increasing and finite sequence of natural numbers. We call support sequence of D. We additionally define
Aa.-(D)
D.
Σ
Σ
X
(5)
1
A,,
Σ 2s. (D) + Σ Σ
(s)
X
f=1 s:<j<e111

---

Published in Transactions on Machine Learning Research (01/2023)
and call accordingly (D) == (D)z the lower intrinsic dimension of D and 0z,((D) AD.- the
upper intrinsic dimension of D.
The governing idea is for i E {l....,(} I and with 8, < 8141 to substitute (D) with ds, (D) or Pa4 (D). With Theorem 4.1 this results in lower and upper bounds for A(D) and thus for the intrinsic ID. approximate the ID and estimate the approximation error.
By comparing upper and lower bounds,
Corollary 4.3. For support sequences holds (D) AD) (D, and (D) - a(D) 2 (
Definition 4.4 (Approximation Error). For support sequence the (relative) approximation error of (D)
with respect to 8 is given by
E(s, = O2 0 10--0,,-00 02)
-2,((D)
With the computation of the upper and lower ID it is possible to bound the error with respect to the ID @D).. The following corollary can be deduced from Corollary 4.3 and Definition 4.4. Corollary 4.5. For support sequence s the following statements hold.
max 0.+ (D)- O(D) 0. (D)
J, S E(s; D),
O(D) 0.-(1) (D)
2. (D) (0() - 0 10., (D) (D)
If the error of the approximation of specific support sequence is not sulficient, further elements can be added to the support sequence. Directly from Equation (5) follows the following corollary.
Corollary 4.6. Let (2 : = (X]) be support sequence and let (81,-.-, 86, P. Sitli-rr 81)
with support sequence with an additional element p. Then it holds that
A(D)s,- = A(D),, + Epsiecus ((0r(D)) 0.. (D))
X
A(D),,+ = A(D),,+ + Emcisp (l)p(D) 0ol4. (D))
X
For a given support sequence s, Corollary 4.5 gives us an upper bound for the error when 02,(DD) or -D,((D) are used to approximate O(D) without knowing O(D). Hence, we can compute (a lower bound) for the accuracy when approximating the ID with Definition 4.4. As and Section 6, comparable small support sequences lead to sufficient approximations. Support sequences can also be used to shorten the computation of the exact intrinsic dimension as the following lemma shows.
see in Section 5.1, Section 5.4
X() be support sequence. Furthermore, let i E {l,..., / 1} and let j E N with 8; < j < 8141- Let Fs, = Ef F ( > (D)). Then it holds that
Lemma 4.7. Let s = (2 = 31....:
Φ, (D) = max(( (D) If E F.,) J (o,,
Proof. ">" follows from Theorem 4.1 and the definition of Φ (D). "5" holds because for F Fs, holds that (D) S (D), due to Theorem 4.1, and , S Φr, (D), due the construction of F... ]
Hence, given a specific j, is possible compute (D) using a subset of F. Based on the particular GD D, this fact can considerably speed up the computation of the ID of D, as we will see in Section
An algorithm to approximate and compute the ID through support sequences is depicted in Algorithm 2. This algorithm takes as input GD D and chosen support sequence s. A reasonable choice for support and A(D), if desired (Line 14). In Line 2, all feature sequences are computed. In Line 6 to Line 11, d., (D) and Fe..,; as defined in Lemma 4.7, are computed. From Line to Line 13, the feature sequences and the lower and upper ID are computed. If desired, the exact computation is done in Line 15 Line 21. Here, we iterate for all support elements (Line 15) through all "gaps" between them (Line 16) and compute (D) using Lemma 4.7 (Line 17 to Line 19).
sequences is discussed in Section 5.1. The output is AD),
8

---

Published in Transactions on Machine Learning Research (01/2023)
Table 1: Statistics of all datasets used in this work. Nodes
Edges
Attributes
PubMed
19, 717
88, 648
500
Cora
2, 708
10, 556
1, 433
CiteSeer
3, 3,327
9, 104
3, 703
ogbn-arxiv
169, 343
1, 166, 243
128
ogbn-products
2, 449, 029
859, 140
100
ogbn-mag
1, 1,939, 743
21, 007
128
ogbn-papers100N 100M
111, 956
1, 1,6,5, 685, 872
128
4.2 Estimating Computational Costs
Let (si,...., s:) be a support sequence. After the computation of Fs.....,FFr.. we can estimate how much computation steps
avoid in order to compute ((D) with Algorithm 2 compared to Algorithm 1. Together with the error function E(s), this estimation can help us to decide if it is desirable to compute the exact value O(D) or leave it at (D) and 0z,+ (D). This is done in the following manner. For a specific f E F, Lemma 3.5 shows that the computation of (x. requires | - different subtractions and to keep the minimum value. Honce, the cost for computing A(D) and therefore O(D) via Algorithm 1 can be estimated via use Algorithm 2, solely have the cost to compute 6..- For all values with 8; < j 81+1, cost estimation is |F,, - 1). Hence, for given support sequence (sI...., 8t), can estimate how many computations are saved using the following notions.
- 1)) OOFI> k) O(F1 - However, if we
We address the naive computation costs for computing the ID of a GD with
[F X |X
C(D)
2
In contrast, for a support sequence s s1) of D, the computation costs are
CD(D) = ([F)
Sk 1) +
F
Σ
-
+11.
(6)
Sk 15K313611
Hence, the saved costs of are
C2(D)
SC, ==
C(D)
Once we have computed Φs. (D) and F,, depending on the saved costs, we can decide to discard the support sequence or to contimue further computations with it. Furthermore, using the crror estimation, we can docide to compute the exact or to settle with the approximation.
5 Intrinsic Dimension of Graph Data
Graph data of major interest the realm of geometric lcarning and beyond. In the following, a graph dalasel D GX consists of an undirected, unweighted graph G (V, where V {ui,..., vn) is finite set of vertices, E C (K) and X' E Raxd is d-dimensional attribute matriz. The row-vector X (K....... X..)) is called the aftribute vector of 0;-
Learning from such data is often done via graph nearal nelworks. The idea is to extend common multi-layer perceptrons by a so called neighborhood aggregation, where internal representations graph neighbors are combined at specific layers. In carlier works, neighborhood aggregation is done at multiple layers (Kipf & Welling, 2017: Hamilton et al., 2017; Velickovic et al., 2018). Due to scalability, recent approaches perform multiple iterations of neighborhood aggregation preprocessing step and then the aggregated features

---

Published in Transactions on Machine Learning Research (01/2023)
1 0
V/
X
1
2/
1/2
01
17
14
1
11
32
1.
8
22
0
0
A=
1
V8
Ax=
1
1
0
V8
1
2
0
0
3
0
VR
Figure 1: Example of an k-hop geometric dataset with & = 1. Given are a graph and an attribute matrix X. Then, the normalized adjacency matrix Á and then ÁX are computed. The feature set F consists the coordinate projections of X and Åx. In the figure, every column after "F="" represents one F. Note, that in this example the normalization factor Zmox is
as combined input (Rossi al., 2020; Sun Wu, 2021; Zhang et al., 2021). To be more specific, the employod networks use inputs of the form
Axx Atx),
(7)
where Å a transition matric that is derived from the graph structure. The most common choice for such a matrix is the normalized adjacency matrix, i.e., A = deg(uy) deg(v;))-1 if uy E N(us) and ÅcG = 0 else. Here, (w)) = {uz E V {u, vj} e E} U {ry} is the set of neighbors i and deg(v;) = [N(v:)] is the node degree of Ug. The feature set of the following geometric dataset corresponds to the input in Equation (7).
Definition 5.1. Let & E and À be the normalized adjacency matria of graph dataset D. Furthermore, let dmax maxje(1..... () ma [R Xx,sl. We call the set
FD. = {w: + dmax
XAm)))ss | |m (O.....k).j (I,....dd)
the k-hop feature functions of D. Let v be the normalized counting measure on V. If there exist for each Us, Us V with v, # 0y elements m E [O,....kk].j {l....,d} such that - (ATH X).s + (Am X)R.. then D = FD..
GD. We call the k-hop geometric dataset of D.
Basic statistics of all seven graph datasets considered in the following sections are depicted in Table 1. The statistics for Cora, PubMed and CiteSeer were taken from Pylorrc Geometric 2, The statistics of the OGB datasets were taken from the Open Graph Benchmark. 3 An example of k-hop geometric dataset is depicted in Figure 1. It. is well-known that the normalized adjacency matrix Å E Baxa of graph has a spectral radius of 1. As A is symmetric, this yields | Ayll < ||x yll for x, y E R". The significance of this property is for the respective computations, however, limited, since it primarily leads to insights of the behavior of the columns of under multiplication with powers of A. In contrast, the attribute vectors of the vertices are represented via the rows. Moreover, we may point out that we are not considering the Euclidean distances between attribute vectors, but differences between coordinate values. Thus, the spectral radius of Á does not provide direct insights into FD,k-
hhatps://pttorc--gemsttr....
Planetoid
htttps://gb... http://..b..taafr.d.ee.. wtw.foodd.euu///cce/neaooo edu/doccsnodeepop
10

---

Published in Transactions on Machine Learning Research (01/2023)
0.04
1.000
ogbn-mag ogbn-arxiv ogbn-products
0.03
0.975
Saved Cost
Error
0.02
0.950
0.01
0.925
ogbn-mag
ogbn-arxiv
ogbn-products
0.00.0000
0.0125
0.0250
0.0375
0.0500
0.900.0000
0.0125
0.0250
0.0375
0.0500
Relative Support Size
Relative Support Size
Figure 2: Errors and saved costs for approximating and computing the intrinsic dimensionality for 2 - hop geometric datasets with different lengths of the support sequence.
5.1 Choosing Support Sequences
Algorithm relics on proper choice for support sequence 8. To choose 8, two propertics have to be considered. Namely, the length of the support sequence and the spacing of the elements. Regarding the second point, decided to log-scale spacing. To get such support sequence for a geometric dataset D (X, F. p), we first choose geometric sequence = (s),... st) of length from [X| to 2. We derive the final support sequence s from s' = [1| + sil...., ||X| + stl) by removing duplicated clements.
In the following, we study the error and the saved costs for different lengths of the support sequence. Here, for a geometric dataset, we investigate how E(s, D) and SC(s) vary for chosen with e [[0.001 [ [0.002* [XI...., 10.05 [X]]] Here, if / Ir [X]], we call E IR the relative support size of the resulting support sequence s. We experiment with common benchmark datasets, namely ogbn-arxiv, ogbn-mag and ogbn- products from the Open Graph Benchmark (Hu et al., 2020; 2021). Since for oghn-mag only subset of vertices is equipped with attribute vectors, generate the missing vectors via metapath2vec (Dong et al., 2017). For all datasets, we consider the 2-hop geometric dataset. The results are depicted in Figure 2.
5.1.1 Results
For all datasets, low errors and high saved costs can be reached with a remarkably short support sequence. With relative support sizes of under 0.015 all datasets are approximated with an accuracy of over 99%. Furthermore, the saved costs for sequences with comparable relative support sizes is over 0.98. It stands out, that for the larger datasets ogbn-mag and ogbn-products, shorter sequences (relative to the size of the datasct) Icad to lower errors and higher saved costs then for ogbn-arxiv. Our results further indicate, that relative support size between 0.01 and 0.02 is a reasonable range for maximizing the saved costs. For longer support sequences, the saved cost decrease while the error does not change dramatically, at least for the 2-hop geometric datasets of ogbn-mag and ogbn-products. Note, that longer support sequence do not always lead a higher amount of saved costs. For longer support sequences s the costs of computing de (D) for l e decreases. However, the costs of computing da, (D) for all elements 8; increase.
11

---

Published in Transactions on Machine Learning Research (01/2023)
Table 2: Intrinsic dimension and performances on classification tasks, In the upper table, we display IDs for all k-hop geometric datasets for & (0,...,5), 5]. In the middle table, display the ID estimated by the MLE baseline. In the lower table display mean and standard derivations for test accuracy of standard SIGN model on the classification tasks which belongs to the dataset.
k-hop
1
2
3
4
Dataset
PubMed
2542.3425
2336.6611
2077.5821
2077.0953
2077.0886
2077.0848
Cora
6,2523
3.8324
3,6689
3. 3.6627
3.6624
3.6623
CiteSeer
22.3337
11.3166
10.2347
9.8134
9.5191
9.3795
ogbn-arxiv
83.9160
31.4731
31.4731
31.4730
30.7370
30.3767
ogbn-products
1, 1699 323.2496
1, 119,66944444 044.4736
1, 169, 044.
1. 169, 044.2216
169, 044.2212
1, 169, 044.2216
ogbn-mag
2, 3.1.3109
2, 284.0290
284.0290
2, 284.0290
2, 284.0290
2, 284.0290
k-hop
1
2
3
4
Dataset
PubMed
24.4623
24.7303
23.3924
22.2779
21.3495
20.5642
Cora
30.6049
28.1785
19.9316
10.8186
9.2970
8.6155
CiteSeer
58 18.9593
26.5031
16.5556
12.0495
9.3171
7.9572
ogbn-arxiv
16.29415
19.8571
18,9068
18.2265
17 17.4905
16.9325
ogbn-products
2.8694
4.7542
4.7950
4.7659
4.6943
4.6687
ogbn-mag
30.7024
33.2848
31.5140
30.4844
29.9080
29.5956
k-hop
1
2
3
4
Dataset
PubMed
.6850 + 0144
.7191 = .0123
.7378 +: .0362
.7633 + .U66
.7615 = .0160
.7571 +: .0334
Cora
.5329 + .0120
.7223 + .0117
.7766 + .0045
.7870 + .0076
.7917 + .0084
.7511 + .0047
CiteSeer
.4975 1: .0775
.6165 1 .1660
.6300 1: 01011
.6677 1. .0074
.695 1 .0085
.6734 1: .0080
ogbn-arxiv
.5341 + .0900
.6022 = .0052
.6933 + .0066
.9177 = .0074
.6901 = .0083
.8900 + .0051
ogbn-productts
.5969 .0166
7204 .0017
.7390 1 .0177
7660 .0014
76788 1 .0022
.7877 .0019
ogbn-mag
.2712 .000
3635 00099
.3899 1: 0000
3339 .0029
3983 - .0050
.0122 .0000
5.2 Neighborhood Aggregation and Intrinsic Dimension
We study how the choice of k affects the intrinsic dimension value of the k-hop geometric dataset. For this, we compute the intrinsic dimension for & E {0, 1,.......) .5} for six datasets: the three datasots mentioned above and PubMed, Cora and CiteSeer (Yang et al., 2016), which we retrieved from PyTorch Geometric (Fey & Lenssen, 2019). Furthermore, we train GNNs which use the feature functions of k-hop geometric datasets as information for training and inference. This allows us to discover connections between the ID specific datascts with respect to the considered feature functions and the performance of classifiers, which rely on these feature funetions. For this, train SIGN models (Rossi et al., 2020) for k E [O,...,5) Implementation details and parameter choices can be found in Appendix A.1.
5.2.1 Baseline Estimator
To investigate which extent our ID function surpasses established ID) estimators with respect to estimating the discriminability of dataset, also compute all ID values with the Maximum Likelihood Estimator (MLE) (Levina & Bickel, 2004). This estimator is commonly used in the realm of deep learning (Pope et al., 2020; Ma ct al., 2018a;b). For our experiments, we use the corrected version propased by MacKay & Ghahramani (2005). Note, (hat the MLE is only applicable to datasets X' E R and is thus not able to respect the neighborhood aggregated feature functions. Hence, we incorporate the neighborhood information of k-hop datasct by concatenating feature voctors with the neighborhood aggregated feature vectors. Duc to performance reasons, only subsets of the data points are considered for ogbn-mag and ogbn-products. More details to usage of the ML.E. are discussed in Appendix A.2.
12

---

Published in Transactions on Machine Learning Research (01/2023)
10000
0 0_hop
3000
1 1_hop
1 1_hop
2_ hop
2 2_hop
hop
3_hop
4_hop
8000
2500
4 4_hop
5 5_hop
5_hop
Intrinsic Dimension
Intrinsic Dimension
6000
2000
4000
1500
2000
1000
0
1000
2000
3000
4000
5000
100
200
300
400
500
#Fdges
#Fdges
Figure Intrinsic dimension of one-hot encoded features. The right plot zoom of the left one which hides the 0-hop geometric datasct.
5.2.2 Results
We find that one iteration of neighborhood aggregation always leads to a huge drop of the ID when using our ID function. However, consccutive itcrations only lead to a small decrease. For the datasets from OGB, some iterations lead significantly decreases the ID, for ogbn-products, only the first two iterations are relevant for decreasing the ID. It stands out, that for ogbn-arxiv, the second and third iteration lead to no significant decrease, but the fourth and fifth do. The results for PubMed stand out. Here, the second iteration of neighborhood aggregation leads to a comparable decrease as the first one.
drop of the ID dimension all. For ogbn-mag, only the first iteration
Considering the classification performances, the first iteration is again the key factor, leading to a significant increase in accuracy. As for the ID, the PubMed dataset behaves differently than the other datasets: the second iteration of neighborhood aggregation leads to comparable increase in accuracy as the first.
The MLE ID behaves different. Here, pattern of the first iteration of aggregation being the key for decreasing the data complexity is observed. For some datasets, the first rounds of feature aggregation may even increase the intrinsic dimension. To sum up, our results indicate that our ID is a better indicator for classification performance then the MLE ID.
5.3 Synthetic Data
To get further insights into the behavior of our ID notion, we now consider k-hop geometric dataset for one-hot encoded graph data, ie., D (X, E)), with X' = [| where Пιv] is the |V]ddimensional identity matrix. We consider the case of
{1,.... 100} and determine the ID for E (O....,5) 5) for increasing edge sizes. To do so, we place the 4950 possible edges in random order and add them step by step and compute the ID notions for the hop geometric dataset. The results can be found in Figure 3.
5.3.1 Results
The 0 hop goometric dataset has an ID which does not depend on the amount of edges. This is not surprising since it does not incorporate any graph information. For all other values, the ID first sharply decreases and then increases. This indicates that the addition of neighborhood aggregation is particularly useful for
13

---

Published in Transactions on Machine Learning Research (01/2023)
Table 3: Approximation of intrinsic dimension for ogbn-paperss00N.
k
0s.-(2) (D)
282.2380
171.7385
148.3323
337.7662
128.2751
125.3418
(2,+(D)
282.33 382.3387
171.7997
148.3852
137.8153
128.3208
125.3864
E(s, D)
0.0004
0.0004
0.0004
0.0004
0.0004
0.0004
Table Error on randomly generated data.
n
d
Es,
106
10
2.55 * 10 = 1.13 10
106
50
2.55 10= 5.36 10-9
10s
250
2.55 10 2.31
10-9
107
10
3.08 * 10 4.78 10-10
107
50
3.08 * 4 6.57 10-10
10"
250
3.08 + 10- 6.16 1 10-10
10
10
3.55 10-1 10-11
10a
50
3.55 + 10
10
250
3.55 * 10-4 2.69 10-11
graphs moderate density. Here, the addition of additional rounds aggrogation beyond the first one can further lower the ID. For higher edge sizes, the ID difference between different & values vanishes.
5.4 Approximation of Intrinsic Dimension on Large-Scale Scale Data
To demonstrate the feasibility of our approach, we use it to approximate the ID of the well known, large- scale ogbn-mag- papers100M dataset. For this, we construet the support sequence as in Section 5.1 with I= 100.000. The results are depicted Table On our Xeon Gold System with 16 corcs, approximating the ID of a k-hop geometric dataset build from ogbn-mag-paperss1o000 papers100 is possible within a few hours. While the ID drops for every iteration of neighborhood aggregation, the decrease becomes smaller. The ID of the different k-hops can be differentiated by the approximation, i.c., (D,) stands out, that even for such short support sequence (compared to the size of the dataset), the observed error is remarkably low. In detail, we can approximate the ID) with an accuracy of over 99.95%. Tt is further remarkable, that the error does not change significantly for different k. We observed this effect also for the other datasets. Our results on ogbn-papers1000 indicate, that with short support sequences, we can sufficiently approximate the ID of large-scale graph data.
(D;+)) for i E {0, . 4). It
6 Errors of Random Data
To further understand how our approximation procedure behaves we conducted experiments on random data. We considered different data sizes and different amount of attributes. For this, we experimented with real-valued datasets, ie. datasets represented by an attribute matrix E Raxd Here, the feature functions are given by the data columns. To be more detailed, the considerod geometric dataset is ((X; | e [I,...,n)), (X, + Xi
(l,...,,u dd)...,,))),,) u). Here, v is again the normalized counting measure and X; is he i i- th row vector of X. We iterate through {10%, 10", 107,100} and d through {10, 50, 250}. We repeat all experiments 3 times. For all datasets, we build support sequence as described in Section 5.1 with - 100, 000. The results can be found in Table 4.
For all datasets, the errois are small and the accuracy is over 99.9% for all considered data sizes. The difference the error for different values of negligible. Furthermore, we have small standard deviations. All this indicates that I 100, 000 is a reasonable default choice that leads to sufficient approximations in a large range of data and attribute sizes.
14

---

Published in Transactions on Machine Learning Research (01/2023)
7 Conclusion and Future Work
We presented principle way to efficiently compute the intrinsic dimension (ID) of geometric datascts. Our approach is based on an axiomatic foundation and accounts for underlying structures and is therefore especially tailored to the field of geometric learning. We proposed a novel speed up technique for an algorithm which has quadratic complexity with respect to the amount of data points. This enabled us to compute the ID of several real-world graphs with up to millions of nodes. Equipped with this ability, we shed light on connections of classification performances of graph neural networks and the observed intrinsic dimension for common benchmark datasets. Finally, using novel approximation technique, we were able to show that our method scales to graphs with over 100 million nodes and billions of edges. We illustrated this by using the well-knowe ogbn-papers100M dataset.
Future work includes the identification of suitable feature functions for other domains, such as learning on text or image data. Incorporating the structure of such datasets into the computation of intrinsic dimensionality is an open research problem. Another promising research direction to investigate how the ID of datasets could be manipulated. Since our investigations suggest connections between a low ID and high classification performances, this has the potential to enhance learning procedures.
Acknowledgement
The authors thank the State of Hesse, Germany for funding this work as part of the LOFWE Exploration project "Dimension Curse Detector' under grant
References
Alessio Ansuini, Alessandro Laio, Jakob H. Macke, and Davide Zoccolan. Intrinsic dimension of data repre- sentations in deep neural networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d'Alché Buc, Emily B. Fox, and Roman Garnett (eds.), NeurlPS, pp. 6109-6119 2019.
Jonathan Bac and Andrei Zinovyev. Local intrinsic dimensionality estimators based on concentration of measure. In 2020 International Joint Conference on Neural Networks (IJCNN), pp. 1-8. IEEE, 2020, doi: CAW.NNN46666.000..9990 2020.9207096.
Edgar Chávez, Gonzalo Navarro, Ricardo A. Baeza- Yates, and José L. Marroquín. Searching in metric spaces. ACM Comput. Sure., (3(3):273-321, 2001. doi: 10.1145/502807.50222
Alexander Cloninger and Timo Klock. A deep network construction that adapts to intrinsic dimensionality beyond the domain. Neural Networks, 141:404-419, 2021. ISSN 0893-6080. doi: https://doi.org/10.000 2.eeumet2021.06.000
J.A. Costa, A. Girotra, and Alfred Hero. Estimating local intrinsic dimension with k-nearest neighbor graphs. volume 30, pp. 417 422, 08 2005. ISBN 0-7803-9403-8. doi: N0.1109/SSPP2005. 1SPP.2000..626631.
Yuxiao Dong, Nitesh V. Chawla, and Ananthram Swami. metapath2vec: Scalable representation learning for heterogeneous networks. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, Halifar, NS, Canada, August 13- 17, 2017, pp. 135-144. ACM, 2017. doi: 10.1145/309 00044//339996330900330
Elena Facco, Maria d' d'Errico, Alex Rodriguez, and Alessandro Laio. Estimating the intrinsic dimension of datasets by a minimal neighborhood information. Scientific reports, 7(1):188, 2017.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR
Workshop on Representation Learning on Graphs and Manifolds, 2019.
Marina Gomtsyan, Nikita Mokrov, Maxim Panov, and Yury Yanovich. Geometry-aware maximum likelihood estimation of intrinsic dimension. In Wee Sun Lee and Taiji Suzuki (eds.), Proceedings of The Eleventh Asian Conference on Machine Learning, volume 101 of Proceedings of Machine Learning Research, pp. 1126-1141 PMLR, 17-19 Nov 2019.
15

---

Published in Transactions on Machine Learning Research (01/2023)
Danicle Granata and Vincenzo Carnevale. Accurate estimation of the intrinsic dimension using graph dis- tances: Unraveling the geometric complexity of datasets. Scientific reports, 6(1):1-12, 2016.
Peter Grassberger and Itamar Procaccia. Measuring the strangeness of strange attractors. In The theory of chaotic attractors, pp. 170-189. Springer, 2004.
M. Gromov and V. D. Milman. topological application of the isoperimetric inequality. Americun Journal of Mathematics, 105(4))843-85) 854, 1983. ISSN 00029327, 10806377.
William I.. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Tn Advances in Newral Information Processing Systems 30, pp. 1024-1034, 2017.
Tom Hanika, Friedrich Martin Schneider, and Gerd Stumme. Intrinsic dimension of geometric data sets. Tohoku Mathematical Journal, 74(1):23 52, 2022. doi: 10.2748/un httmtmmm...201144
Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.0066 2020.
Weihua Hu, Matthias Fey, Hongyu Ren, Maho Nakata, Yuxiao Dong, and Jure Leskovec. Ogb-Isc: A large-scale challenge for machine learning on graphs. arXiv preprint: 2103.09490, 2021.
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In 5th Int. Conf. on Learning Representations, 2017.
Elizaveta Levina and Peter Bickel. Maximum likelihood estimation of intrinsic dimension. In NIPS, pp- 777-784, 2004.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In ICLR (Poster). OpenReview.nee net, 2018.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi N. R. Wijewickrema, Grant Schoenebeck, Dawn Song, Michael E. Houle, and James Bailey. Characterizing adversarial subspaces using local intrinsic dimensionality. In 6th International Conference Learning Represenlations, ICLR 2018, Vancouver, BC, Canada, April 30- May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018a. URL https://openreiiee http:://ppearrrreeeeeettttoouaaa
net/forum? Bidiuumddd-BBBB2a1
Xingjun Ma, Yisen Wang, Michael E. Houle, Shuo Zhou, Sarah M. Erfani, Shu-Tao Xia, Sudanthi N. R. Wijewickrema, and James Bailey. Dimensionality-driver learning with noisy labels. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmássan, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 3361 3370. PMLR, 2018b. URL http://proccediing.ntt mlr. hatp:///gg......peee///////aaaa press//800ma11dd html.
David JC MacKay and Zoubin Ghahramani. Comments on m'maxiimmm likelihood estimation of intrinsic di- mension'll by e. levina and p. bickel (2005). The Inference Group Website, Cavendish Laboratory, Combridge University, 2005.
V. Milman. Topics in Asymptotic Geometric Analysis, pp. 978-3-0346-0445--3 doi: 10.1007 1111007997883-0344----2222 8.
Birkhäuser Basel, Basel, 2010. ISBN
V. D. Milman. The heritage of P. Lévy in geometrical functional analysis. In Colloque Paul Lévy sur les processus stochastiques, number 157-158 in Astérisque. Société mathématique de France, 1988.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learm: Machine learning in Python. Journal of Machine Learning Research, 12:2825 2000-88000 2011.
Vladimir Pestov. On the geometry of similarity search: Dimensionality curse and concentration of measure. Tnf. Process. Lett., 73(1-2):47-B5 51, 2000. doi: 10.1016 10.1016/50020-01000 000002000000009900000-06660040
16

---

Published in Transactions on Machine Learning Research (01/2023)
Vladimir Pestov. Intrinsic dimension of dataset: what properties does expect? In Proceedings of the International Joint Conference Neural Nelworks, IJCNN 2007, Celebrating 20 years of neural networks, Orlando, Florida, USA, August 12-17, 2007, pp. 2959 2964. IEEE, 2007. doi: 10.1109/LJCNN. 2007.4371431.
Vladimir Pestov. An axiomatic approach to intrinsic dimension of datasct. Neural Networks, 21(2-3): 204 213, 2008. doi: 2000::...euueee.200.22...00
Vladimir Pestov. Intrinsic dimensionality. ACM SIGSPATIAL Special, 2(2):8-11, 2010. doi: 10.1145/
1862413.1862416.
Phil Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein. The intrinsic dimension images and its impact on learning. In International Conference on Learning Representations, 2020.
Emanuele Rossi, Fabrisio Frasca, Ben Chamberlain, Davide Eynard, Michael M. Bronstein, and Federico Monti. SIGN: scalable inception graph neural networks. CoRR, abs/2004.11195 2020.
Chuxiong Sun and Guoshi Wu. Scalable and adaptive graph neural networks with -eah-eedeleehhaneedd train- ing. CoRR, abs/2104.00 0997400049977,, 2021.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canado, April 30 May 3, 2018, Conference Track Proceedings, 2018.
Stephan Wojtowylsch and Weinan E. Can shallow neural networks beat the curse of dimensionalityy? a mean field training perspective. IEEE Transactions on Artificial Intelligence, 1(2):121-12)) 1299 2020, doi:
Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In Maria-Florina Balcan and Kilian Q Weinberger (eds.), Proceedings of the 33nd Inter- national Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pp. 40-48. JMLR.org, 2016.
Wentao Zhang, Ziqi Yin, Zeang Sheng, Wen Ouyang, Xiaosen Li, Yangyu Tao, Zhi Yang, and Bin Cui. Graph attention multi-layer perceptron. CoRR, abs/2108. 10097, 2021.
17

---

Published in Transactions on Machine Learning Research (01/2023)
A Appendix
A.1 Setup of SIGN classifiers
For PubMed, Cora and CiteSeer, we train on the classification task provided by Pytorch Geometric (Fey & Lenssen, 2019) which was earlier studied by Yang et al. (2016). All Open Graph Benchmark datasels are trained and tested on the official node property prediction task.* 4 Our goal is not to find optimal classifiers but discover connections between the choice of k, the ID and classifier performance. Thus, we omit excessive parameter luning and stick lo reasonable standard parameters. For all tasks, we use a simple SIGN model Rossi et al. (2020) with one hidden inception layer and one classification layer. For PubMed, CiteSeer and Cora, use batch sizes of 256, hidden layer size of 64 and dropout at the input and hidden layer with 0.5. The learning rate is set to 0.01. All these parameters were taken from Kipf & Welling (2017). For ogbn-arxiv, ogbn-mag and ogbn-products, we stick to the parameters from the SIGN implementations on the OGB leaderbord. For ogbn-arxiv, we use a hidden dimension 512, dropout at the input with 0.1 and with 0.5 at the hidden layer. For ogbn-mag, we use a hidden dimension 512, do not dropout at the input and use dropout with 0.5 the hidden layer. For ogbn-products, we use a hidden dimension of 512, input dropout of 0.3 and hidden layer dropout of 0.4. For all ogbn tasks, the learning rate is 0.001 and the batch-size 50000. For all experiments, early stopping on the validation accuracy. Here, we use palience of 15. These are the standard parameters of Pytorch Lightning. $ For all models, we use an Adam optimizer with weight decay of 0.0001. We report mean test accuracics over 10 runs. The intrinsic dimensions and the test accuracy are shown in Table 2.
train for maximum of 1000 epochs with
A.2 Details on Baseline ID Estimator
To use the MLE ID, we have to convert the k-hop geometric dataset (V, FD, v) of graph data D = (X, G), where X' E Raad into real-valued feature matrix X. This done by concatenating the rows of X with the rows of Ax. A i.c., & [k+x(d with
Xhs 8is = (Amx))s
j (L....,,,d)
j = nd + for E (1,...,k),,5
k). E (l....d).
The MLE is given via
12
d(x, Ni M(x)))) di d(X,, N2 (x.)))
MLE($)= n(k 1)
log(
(8)
where d is the euclidean metric and (X.) is the j-th nearest neighbor of X, with respect to the Euclidean metric. Thus, the MLE depends on parameter I, which we set to 5.
We implement the MLE by using the NearestNVeighbors class of seikit-learn (Pedregosa et al., 2011) and then building the mean of all log( d(X.I.N((&..)) ((X)) with i [I....,n] and j e {1,....55] 5}. Here, we skip all elements where d(X;, Nx(,)) 0. This can happen, when X has duplicated rows, representing data points with equal attribute vectors.
For ogbn-mag and Here, we sample 169, 343 6
computing Equation (8) is not possible due to performance reasons. 6 I (1,..., ..) and only compute
d d(x, Ni()))) log
MLE(X)= = n(k - 1) GeT j=l
d( ((,,,N
'nttps://ogb.stan http::///obb.ssaoo...... stanford..ccoo....ooooo sttanford.edu//occ//aeee edu//occsnodeprop/
Shttps://wwv. httt:::///wx..yoocchii..
6Thhi is the amount nodes of ogbn-arxiv, the largest network which the full computation was feasible.
18


Reviews:
[{"review_id": "kSzU5kADuo", "reviewer": "Reviewer_HCNE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n\nThe algorithm presented is simple and easily implementable and has rigorous proofs\n\n**Weaknesses**\n\n1) The motivation for computing this particular ID estimate is not clear to me. In relation to this I don't the goal of the experiment in section 5.2. As we can see the two ID estimates can be very very different (eg. 3 vs over 1 million). \n2) While most of the proofs are clear. I think the proofs could be further simplified (but this is minor complaint)\n\n**Questions**\n\n\n1) Why does the saved time initially increase? Should't the curve be monotonic?\n2) Do the authors have any intuition as to why the dimension should drop as we increase $k$?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper looks at the Intrinsic Dimension (ID) estimator from Hanika, Schneider, and Stumme 2022. The paper presents two algorithms, the first algorithm is a quadratic algorithm for computing this ID estimate for finite data sets and the second algorithm is faster for approximating the same ID estimate. \n\nThe paper has a few proofs to justify their algorithm. They then test the approximation algorithm to determine the error rate and time saved. \n\nThe paper then computes this ID for a variety of different datasets. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I would like the authors to at least address the first weakness and better explain the motivation for this estimate and the experiments in section 5.2", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "fzv7kwDbOZ", "reviewer": "Reviewer_Tgwr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The methods are limited to data where distance is measured with respect to a feature set F, and one computes a distance d_F(a,b) = max_{f in F} |f(a) - f(b)|, and |F| is a finite set.  The algorithm iterates over this set, its runtime is linear in |F|.  I interpret the |F| as the original dimension, but note that this notion is not powerful enough to capture the common Euclidian l_2 distance, where the corresponding \"test\" function is a unit vector u in R^d, and f_u(a) = <u,a>.  Then one can write \n  ||a-b||_2 = sup_{f_u in F} |f_u(a) - f_u(b)| = sum_{u a unit vector} |<u,a> - <u,b>|.  \nNote that in this case, when in greater than 1 dimension, then F is not finite, so this method does not apply.  \n\nThis would apply for instance with the l-infinity distance on vector data.  \nThe paper mostly finds applications in analyzing large \"k-hop\" graphs where the finite set F.  As someone unfamiliar with this definition (of k-hop graphs) before the paper, I was not able to fully understand it or its relevance from the description in the paper (start of Section 5 and Figure 1).  \n\nIn short, the studied notion of intrinsic dimensionality (ID) of a data set X is 1/Delta(X)^2 where Delta(X) is the average \"partial diameter\" of X under d_F, where partial diameter is over the most compact subset of a 1-alpha fraction of the measure of X, averaged over all alpha in (0,1).  \nWhile the paper mentions ways this notion of ID might be useful, it does not provide motivation for it, does not explain how it has a modeling advantage over more standard (Euclidean) distance based notions, and does not demonstrate its use in any application.  \nThis concept was new to me as a reader, and I did not come away with a sense of why it captured dimensionality, or why this notion was useful.  So did not find a strong motivation in machine learning for this study from the writing in the paper.  \n\n\nThe main contribution of the paper is to provide a heuristic for computing a quantity that should be close to the intended ID value on a finite point set X (size n) with a finite set of feature functions F.  The key is to compute the partial diameter for X for each f in F and each alpha in (1/n, 2/n, ..., (n-1)/n).  The central idea in this paper is to find a small S subset X or size l, and for each f in F, compute the partial diameter values on S instead of X.  Through a simple argument, it provides quantities that are under- and over-estimates of the true values.  On experiments, it shows these upper and lower bounds are close to each other on the graph data sets considered.  \n\nThe paper does not prove a relationship between l (the size of the subset S), and the amount of error.  Hence it is not an approximation algorithm, but only a heuristic.  \n\n\nUltimately,\n - I did not find the problem well-motivated\n - I did not find the algorithmic insights that deep\n - I found the writing hard to follow (e.g., the new method, Algorithm 2 was hard to follow)\n - The paper did not convince me the proposed approach was useful in machine learning.  \nSo I do not recommend acceptance at TMLR with significant changes.  ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper considers an obscure notion of dimensionality (proposed by V. Pestov 20 years ago), and proposes a heuristic to speed up its computation with some error.  Experiments show the error of the heuristic is small on some data sets realized by graphs.  \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "This paper would need significant changes for me to recommend its acceptance at TMLR. \n - provide intuition for why this notion of ID is capturing dimension\n - provide motivation for why this method is useful for machine learning, and why it is superior to other notions in the literature.  I expect this would require a completely different experimental section.  \n - improve the writing to make the description of Algorithm 2 more clear (just pseudo code and a few comments its not sufficient\n - improve writing to explain what k-hop graphs are, and why the ID values computed measure a useful notion of dimensionality for those graphs, and why those associated numerical values are related and important for machine learning\n - explain the relationship between l  (the size of the subset) and the approximation that can be achieved (its ok if some guarantee does not match the very small error found on some data sets, but you want to know how large it could be on *any* data set).  ", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "wKraWvThlV", "reviewer": "Reviewer_pQDm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- the paper is quite complete, and the experiments pretty thorough\n- the study of different notions of ID is particularly relevant in the current landscape, especially for geometric data where classical ones such as VC-dimension or Rademacher complexities may be difficult to define\n- the paper is clear and well-written\n\nWeaknesses:\n- the proposed ID could be somewhat better motivated. At the end of the day, it is difficult to interpret and to get an intuition behind the performed computations, unlike some other classical notions of dimension such as the intrinsic dimension of a supporting manifold or the relationship between separability and VC dimension. The intuition behind the \"concentration properties\" and \"axioms\" are somewhat quickly skipped, they could be better detailed, even if appearing in earlier work.\n- while quite thorough, the experiments on real data are not very convincing as to the definitive existence of a link between the proposed ID and learning performance; a singular drop (or increase) in ID (or performance) between k=0 and k=1 is far from being surprising for graph data. The difference between the behavior of ID and MLE is not so striking, especially given that the ID has extremely different range depending on the dataset (eg a drop from 1169323 to 1169044 is small in relative term). Maybe some experiments on synthetic data could help both better motivate the proposed ID and give better intuition ?\n- the choice of support sequence is quite arbitrary. Can computation be done when $S$ is a stream, where the user adds new $s_l$ sequentially until a desired accuracy is reached ?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this work, the authors study a notion of intrinsic dimension (ID) for finite datasets, based on how discriminative some given features are. This is an instanciation of a notion of ID proposed is earlier work related to some notion of concentration, to the particular case of finite sets and counting measure.\n\nThe authors first propose a polynomial algorithm to compute the ID, based on some reformulation that uses sorted sequences of features. Since the resulting algorithm is still quadratic, they propose a faster version with subsampling, as well as a way to bound the resulting error and evaluate savings of computation cost.\n\nThey then apply their methodology to graph data, where the features are taken as a concatenation of node features smoothed by the normalized adjacency matrix at several order. They find that, for different order, the ID is somewhat correlated with the performance of a GNN classifier.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I think synthetic experiments and better motivation could help improving the paper, but they are not critical to my acceptance.\n\nSome minor comments/typos:\n- p3, top: the choice of a measure $\\mu$ on $\\mathbb{R}$ is confusing, it was a measure on $X$ two paragraphs earlier. Also it is confused with $\\nu$ after\n- the notation $-\\alpha$ (instead of just $\\alpha$) is weird, is it motivated in earlier work?\n- the \"without loss of generality\" in the proof of Lemma 3.1 is not so trivial, maybe it is possible to be more rigorous here\n- unless I am mistaken, in Lemma 3.4, $\\phi$ is the minimum of the set, not simply a member of. It is computed that way in the algorithm.\n- page 5 bottom: \"$k \\notin$\", missing $S$\n- $s$ should be $S$ in algorithm 2\n- the notation $\\sum_{s_i<l<s_{i+1}} \\phi_{s_i}$ is confusing, isn't it just $(s_{i+1}-s_i - 1)$ times $\\phi_{s_i}$ ?\n- I think theorem 4.1 is immediate, by definition of $\\phi$ as a minimum over $M$ of size $m$ or $n$, no need for a proof.\n- It is quite counter-intuitive that the saved computation cost is not strictly decreasing with the size of the support sequence, it needs to be written and emphasized\n- Maybe the table 2 of results could be better presented as curves, that will better outline the potential correlation between ID and performance ?\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper considers a non-standard notion of intrinsic dimension based on the concentration-of-measure, and provides computationally efficient algorithms to approximate this notion of dimension. The proposed methods are then tested on large data sets. 

The reviewers all pointed out the lack of motivation of this non-geometrical notion of intrinsic dimension. But the discussion period has been fruitful and has lead to major improvement of the paper, in particular on this aspect. Additional experiment have been added, too. 

Despite the reserve of one reviewer concerning the usefulness of the notion for ML, the claims are well supported and thus the paper deserves publication. Indeed, TMLR guidelines mention that the two main criteria for acceptance are:
"Are the claims made in the submission supported by accurate, convincing and clear evidence?" It the case here.
"Would some individuals in TMLR's audience be interested in the findings of this paper?" I believe so, even if it may a bit niche.

Given all this I recommend acceptance as such.


Decision: Accept (poster)