Differentiable Local Intrinsic Dimension Estimation with Diffusion Models
Hamidreza Kamkari 1 Brendan Leigh Ross 1 Rasa Hosseinzadeh 1 Jesse C. Cresswell 1 Gabriel Loaiza-Ganem 1

Abstract
High-dimensional data commonly lies on lowdimensional submanifolds, and estimating the local intrinsic dimension (LID) of a datum is a longstanding problem. LID can be understood as thenumber of local factors of variation: the morefactors of variation a datum has, the more complex it tends to be. Estimating this quantity has
proven useful in contexts ranging from generalization in neural networks to detection of outof-distribution data, adversarial examples, andAI-generated text. While many model-free andseveral model-based estimation techniques exist,
none are differentiable by design. In this work, weshow that the Fokker-Planck equation associatedwith a diffusion model can provide the first LID
estimator that is differentiable and scales to highdimensional data while outperforming existingbaselines on LID estimation benchmarks.

Figure 1. An illustration showing that LID is a natural measure of
relative complexity. We depict two manifolds of MNIST digits,corresponding to 1s and 8s, as 1d and 2d submanifolds of R3 ,respectively. The relatively simpler manifold of 1s exhibits asingle factor of variation (“tilt”), whereas 8s have an additional
factor of variation (“disproportionality”).

1. Introduction
The manifold hypothesis (Bengio et al., 2013) states thathigh-dimensional data in RD often lies on low-dimensional
submanifolds. For a given datum x ∈ RD , this hypothesis
motivates using its local intrinsic dimension (LID), denoted
LID(x), as a natural measure of its complexity. LID(x) canbe intuitively understood as the minimal number of variables
needed to describe x, as illustrated in Figure 1.

submanifolds, suggesting they can be used to construct LID
estimators. However, existing model-based estimators sufferfrom drawbacks including being inaccurate and computationally expensive (Stanczuk et al., 2022), not leveraging
the best existing generative models (Zheng et al., 2022),
requiring training several models (Tempczyk et al., 2022)
or altering the training procedure rather than relying on apre-trained model (Horvat & Pfister, 2024). In all thesecases the LID estimators are non-differentiable in practice.

LID has been used to detect outliers (Houle et al., 2018;
Kamkari et al., 2024b), AI-generated text (Tulchinskii et al.,2023), and adversarial examples (Ma et al., 2018), among
other applications. Traditional model-free estimators of
intrinsic dimension (Fukunaga & Olsen, 1971; Levina &Bickel, 2004; Johnsson et al., 2014) typically rely on pairwise distances and nearest neighbours, rendering them expensive and non-differentiable in practice. Recent work
uses deep generative models that implicitly learn the data

We address all these issues by showing how to estimate LID
efficiently using only a single pre-trained diffusion model(DM). We leverage the Fokker-Planck equation to propose
FLIPD1 , the first LID estimator to scale to high-resolution
images (∼ 106 dimensions), and the first that is tractably
differentiable. While differentiating through local intrinsic
dimension estimation is not a common sub-task today, thismay simply be because no existing methods enable it. We
present these ideas with the hope that future work will findeven more applications given these new capabilities.

1
Layer 6 AI, Toronto, Canada. Correspondence to: Hamidreza
Kamkari <hamid@layer6.ai>.

Published at the 2 nd Differentiable Almost Everything Workshop
at the 41 st International Conference on Machine Learning, Vienna, Austria. July 2024. Copyright 2024 by the author(s).

1Pronounced as “flipped”, the acronym is a rearrangement of“FP” from Fokker-Planck and “LID”.

1Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

2. Background and Related Work

Indeed, one could trivially train m DMs and replace theflows with them. As opposed to the normalizing flows usedin LIDL which are individually trained on datasets with different levels of noise added, a single diffusion model alreadyworks by convolving data with various noise levels and allows density evaluation of the resulting noisy distributions(Song et al., 2021b). Hence, we show that LIDL can beused with a single DM. Throughout this section, we assumeaccess to a pre-trained DM such that f (x, t) = b(t)x. This
choice implies that the transition kernel pt|0 associated with
Equation 1 is Gaussian (Särkkä & Solin, 2019):

Diffusion Models Score-based diffusion models (Songet al., 2021b) convert data to noise through the forward (Itô)
stochastic differential equation (SDE),
dXt = f (Xt , t)dt + g(t)dWt ,X0 ∼ p(·, 0),(1)where Wt denotes a D-dimensional Brownian motion. We
write the distribution of Xt as p(·, t). The backward process
Yt := X1−t obeys a backward SDE involving the scorefunction s(x, t) := ∇ log p(x, t) which is modeled by aneural network ŝ(x, t) via denoising score matching (Vincent, 2011). DMs admit density evaluation, placing them inthe same category as other deep generative models, such as
normalizing flows. As we will see, generative models withthis capability can be utilized to produce LID estimates.

pt|0 (xt | x0 ) = N (xt ; ψ(t)x0 , σ 2 (t)ID ).

We assume that b and g are such that ψ and σ are differentiable and λ(t) := σ(t)/ψ(t) is injective. This encompasses
all DMs commonly used in practice, including varianceexploding, variance-preserving (of which DDPM (Ho et al.,2020) is a discretized instance), and sub-variance-preserving(Song et al., 2021b). In Appendix A we include explicit
formulas for ψ(t), σ 2 (t), and λ(t) for these particular DMs.

LID The local intrinsic dimension of a point x is thedimension of the manifold it belongs to, hence, LID is not
an intrinsic property of the point x, but rather a property ofx with respect to the manifold that contains it. Tempczyk
et al. (2022) proposed LIDL, a method for LID estimation
relying on normalizing flows. LIDL works thanks to asurprising result linking Gaussian convolutions and LID(Loaiza-Ganem et al., 2022; Tempczyk et al., 2022; Zheng
et al., 2022). We will denote the convolution of p(·, 0) and
Gaussian noise with log standard deviation δ as ϱ(·, δ), i.e.
Zϱ(x, δ) := p(x0 , 0)N (x − x0 ; 0, e2δ ID )dx0 . (2)3.1. FLIPD: A Fokker-Planck-Based LID Estimator
LIDL is based on Equation 3, which justifies usinga regression. Differentiating this equation yields that∂ log ϱ(x, δ0 )/∂δ ≈ LID(x) − D for negative enough δ0 .
By leveraging the Fokker-Planck equation associated with
Equation 1, we show in Appendix C that, for DMs withtransition kernel as in Equation 4,
 

∂
log ϱ(x, δ)=σ 2 t(δ) tr ∇s ψ(t(δ))x, t(δ)
(5)
∂δ
 2

+ s ψ(t(δ))x, t(δ) 2 =: ν t(δ); s, x .

The aforementioned result suggests that, for negative
enough values of δ (i.e. small enough standard deviations):log ϱ(x, δ) ≈ δ(LID(x) − D) + c

(4)

(3)

Equation 5 directly provides the rate of change that the
regression in LIDL aims to estimate, from which we get:LID(x) ≈ D + ν t(δ0 ); ŝ, x =: FLIPD(x, t0 ), (6)for some constant c. If we could evaluate log ϱ(x, δ) for various values of δ, this would provide an avenue for estimating
LID(x): set some values δ1 , . . . , δm , fit a linear regression
using {(δi , log ϱ(x, δi ))}mi=1 (with δ as the covariate andlog ϱ(x, δ) as the response), and let β̂x be the corresponding
slope. It follows that β̂x estimates LID(x) − D, so that
LID(x) ≈ D + β̂x is a sensible estimator of LID.
Since ϱ(x, δ) is unknown, LIDL requires training m normalizing flows. More specifically, for each δi , a normalizing
flow is trained on data to which N (0, e2δi ID ) noise is added.
In LIDL, the log densities of the trained models are thenused instead of the unknown true log densities log ϱ(x, δi )when fitting the regression as described above.

where t(δ) := λ−1 (eδ ) and t0 := t(δ0 ). Computing FLIPD
requires only a single hyperparameter, whereas regressionbased estimators require m. Since ν(t(δ); ŝ, x) depends onδ only through t(δ), we can directly set t0 as the hyperparameter rather than δ0 , which avoids the potentially cumbersome computation of t(δ0 ) = λ−1 (eδ0 ): instead of setting
a suitably negative δ0 , we set t0 > 0 sufficiently
close to 0.
Moreover, the expression for ν t(δ0 ); ŝ, x in Equation 5contains only standard operations which are amenable to automatic differentiation. In Appendix A we include explicit
formulas for FLIPD(x, t0 ) for common DMs.

3. Method

4. Experiments

Although Tempczyk et al. (2022) used normalizing flows in
LIDL, they did point out that these models could be swapped
for any other generative model admitting density evaluation.

Throughout our experiments, we use variance-preserving
DMs, the most popular variant of DMs, and compatible with
DDPMs; see Appendix D.1 for hyperparameters.
2

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Table 1. MAE (lower is better) | concordance indices (higher is better with 1.0 being the gold standard). Rows show synthetic manifolds
and columns represent LID estimation methods. Columns are grouped based on whether they use a generative model, with the best resultsfor each metric within each group being bolded.
Model-based
Model-freeSynthetic Manifold
String within doughnut ⊆ R3
L5 ⊆ R10
N90 ⊆ R100
U10 + U30 + U90 ⊆ R100
N10 + N25 + N50 ⊆ R100
F10 + F25 + F50 ⊆ R100
U10 + U80 + U200 ⊆ R800
U900 ⊆ R1000

FLIPD
0.06
0.17
0.49
1.30
1.81
3.93
14.3
12.8

NB
1.00
1.00
1.00
1.00
1.00
-

1.48
1.00
0.18
61.6
74.2
74.2
715
100

LIDL
0.48
0.34
0.34
0.34
0.34
-

1.10
0.10
0.33
8.46
8.87
18.6
120
24.9

ESS
0.99
0.74
0.74
0.70
0.70
-

0.02
0.07
1.67
21.9
7.71
9.20
2.35
75.2

LPCA
1.00
0.74
0.88
0.90
1.00
-

0.00
0.00
21.9
20.1
5.72
6.77
33.7
801

1.00
0.86
0.91
1.00
1.00
-

dimensional spaces, and then embed the samples into RD .
We denote uniform, Gaussian, and Laplace distributions
as U , N , and L, respectively, with sub-indices indicating
LID, and a plus sign denoting mixtures. To embed samples
into higher dimensions, we apply a random matrix withorthonormal columns and then apply a random translation.
For example, N10 +L20 ⊆ R100 indicates a 10-dimensional
Gaussian and a 20-dimensional Laplace, each of which undergoes a random affine transformation mapping to R100
(one transformation per component). We also generate nonlinear manifolds, denoted with F, by applying a randomly
initialized D-dimensional neural spline flow (Durkan et al.,2019) after the affine transformation (when using flows, theinput noise is always uniform); since the flow is a diffeomorphism, it preserves LID.

Figure 2. FLIPD curves with knees at the true LID.

The effect of t0 FLIPD requires setting t0 close to 0. It
is important to note that DMs fitted to low-dimensional
manifolds are known to exhibit numerically unstable scoress(·, t0 ) as t0 ↘ 0 (Vahdat et al., 2021; Lu et al., 2023;
Loaiza-Ganem et al., 2024). Our first set of experiments
examines the effect of t0 on FLIPD(x, t0 ) by varying t0
within the range (0, 1).

Here, we summarize our synthetic experiments in Table 1
using two metrics of performance: the mean absolute error(MAE) between the predicted and true LID for individual datapoints; and (when the dataset has variability in its
ground truth LIDs) the concordance index, which measuressimilarity in the rankings between then true LIDs and theestimated ones. We compare against the LIDL estimator
described in Section 2, as well another DM-based method(NB, (Stanczuk et al., 2022)) and two of the most performant
model-free baselines (LPCA, (Fukunaga & Olsen, 1971);
ESS (Johnsson et al., 2014)). For the NB baseline, we use
the exact same DM backbone as for FLIPD, and for LIDL
we use 8 neural spline flows. In terms of MAE, FLIPD tends
to be the best model-based estimator particularly as dimension increases. Although model-free baselines perform wellin simplistic scenarios, they produce unreliable results as
LID increases or more non-linearity is introduced in the data
manifold. In terms of concordance index, FLIPD achieves
perfect scores in all scenarios, meaning that even when its
estimates are off, it always provides correct LID rankings.
In Appendix D we include ablations for FLIPD and comparisons on more high-dimensional datasets, and with othermodel-free baselines.

In Figure 2, we train a DM on a mixture of three isotropic
Gaussians with dimensions 2, 4, and 8, embedded in R10(each embedding is carried out by multiplication against arandom matrix with orthonormal columns plus a randomtranslation). While FLIPD(x, t0 ) is inaccurate at t0 = 0
due to the aforementioned instabilities, it quickly stabilizes
around the true LID for all datapoints. We refer to thispattern as a knee in the FLIPD curve. We persistently see
knees in FLIPD curves (in Appendix D.2, we show similar
curves for more complex data manifolds). Not only is thisin line with the observations of LIDL on normalizing flows(see Figure 5 of Tempczyk et al. (2022)), but it also gives
us a fully automated approach to setting t0 . We leveragekneedle (Satopaa et al., 2011), a knee detection algorithm
which aims to find points of maximum curvature. Rather
than fixing t0 , we evaluate Equation 6 for 50 values of t0
and pass the results to kneedle to automatically detect the
t0 where a knee occurs.
Synthetic experiments We create a benchmark for LID
evaluation on complex unions of manifolds where true LID
is known. We sample from simple distributions on low-

Image experiments We first focus on the simple image
datasets MNIST and FMNIST. We flatten the images and use
3

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Table 2. Spearman’s correlation between LID estimates and PNG
compression size.
MethodMNISTFMNISTCIFAR10

SVHN

FLIPD
NB
ESS
LPCA

0.837
0.864
0.444
0.413

0.883
0.480
0.063
0.01

0.819
0.894
0.326
0.302

0.876
0.573
0.019
−0.008

Finally, we consider high-resolution images from LAIONAesthetics (Schuhmann et al., 2022) and, for the first time,
estimate LID for extremely high-dimensional images with
D = 3 × 512 × 512 = 786,432. To achieve this, we use Stable Diffusion (Rombach et al., 2022), a latent DM pretrainedon LAION-5B (Schuhmann et al., 2022). This includes anencoder and a decoder trained to preserve relevant characteristics of the data manifold in latent representations. Since
the encoder and decoder are continuous and effectively invert each other, we argue that the Stable Diffusion encoder
can, for practical purposes, be considered a topological embedding of the LAION-5B dataset into its latent space ofdimension 4 × 64 × 64 = 16,384. Therefore, the dimensionof the LAION-5B submanifold in latent space should beunchanged. This approximation allows us to estimate image
LIDs by carrying out FLIPD in the latent space of Stable
Diffusion. Here, we set the Hutchinson sample count to 1,
meaning we only require a single Jacobian-vector-product.
When we order a random subset of 1600 samples according
to their FLIPD at t0 = 0.3, the more complex images are
clustered at the end, while the least complex are clustered at
the beginning: see Figure 3 for the lowest- and highest-LID
images from this ordering, and Figure 24 in Appendix E.6
to view the entire subset and other values of t0 . In comparison to orderings according to PNG compression size
(Figure 3), FLIPD estimates prioritize semantic complexity
over low-level details like colouration.

Figure 3. Top: Images with small and large FLIPD estimates fromFMNIST, MNIST, SVHN, and CIFAR10. Bottom: LAION images with small and large FLIPD estimates using Stable Diffusion(top, t0 = 0.5) and PNG compression sizes (bottom).

the same MLP architecture as in our synthetic experiments.
Despite using an MLP, our DMs can generate reasonablesamples (Appendix E.1). The average LID estimates are
130 and 170, respectively. Although our LID estimates arehigher than given in prior work (Pope et al., 2021; Brown
et al., 2023), our high-dimensional experiments (Table 7 ofAppendix D.4) and findings by Tempczyk et al. (2022) andStanczuk et al. (2022) show that model-free baselines underestimate LID of high-dimensional data, especially images.
On more complex image datasets like SVHN and CIFAR10,
the simplistic MLP score network backbone fails to generate
high-quality samples. Therefore, we replaced it with stateof-the-art UNets (Ronneberger et al., 2015; von Platen et al.,2022) (see Appendix E.2). We took a random subset of 4096
images from FMNIST, MNIST, SVHN, and CIFAR10, andsorted them according to their FLIPD estimates. We show
the top and bottom 5 images for each dataset in Figure 3, andinclude more samples in Appendix E.3. Our visualizationshows that higher FLIPD estimates indeed correspond to
images with more detail and texture, while lower estimatescorrespond to less complex ones. FLIPD not only visuallycorresponds to image complexity, but also required only 50
Jacobian-vector-products, resulting in a significant speedup
over all prior work.

5. Conclusions, Limitations, and Future WorkIn this work, we have shown that the Fokker-Planck equation can be utilized for efficient LID estimation with any
pre-trained DM. We have provided strong theoretical foundations and extensive benchmarks showing that FLIPD estimates accurately reflect data complexity. Although FLIPD
produces excellent LID estimates on synthetic benchmarks,
the lack of knees in FLIPD curves on image data when using state-of-the-art architectures is surprising, and results inunstable LID estimates which strongly depend on t0 . We
see this behaviour as a limitation, even if FLIPD provides
a meaningful measure of complexity in these cases. Given
that FLIPD is tractable, differentiable, and compatible withany DM, we hope that it will find uses in applications where
LID estimates have already proven helpful, including OOD
detection, AI-generated data analysis, and adversarial example detection.

Further, we quantitatively assess our estimates by computing
Spearman’s rank correlation coefficient between different
LID estimators and PNG compression size, used as a proxyfor complexity in the absence of ground truth. As shown in
Table 2, FLIPD has a high correlation with PNG, whereas
model-free estimators do not. We find that the NB estimator
correlates slightly more with PNG on MNIST and CIFAR10,
but significantly less in FMNIST and SVHN.
4

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

References

Johnsson, K., Soneson, C., and Fontes, M. Low bias local
intrinsic dimension estimation from expected simplex
skewness. IEEE Transactions on Pattern Analysis andMachine Intelligence, 37(1):196–202, 2014.

Albergante, L., Bac, J., and Zinovyev, A. Estimating theeffective dimension of large biological datasets using
fisher separability analysis. In 2019 International Joint
Conference on Neural Networks, pp. 1–8. IEEE, 2019.

Kamkari, H., Balazadeh, V., Zehtab, V., and Krishnan, R. G.
Order-based structure learning with normalizing flows.
arXiv:2308.07480, 2024a.

Bac, J., Mirkes, E. M., Gorban, A. N., Tyukin, I., and Zinovyev, A. Scikit-Dimension: A Python Package forIntrinsic Dimension Estimation. Entropy, 23(10):1368,
2021.

Kamkari, H., Ross, B. L., Cresswell, J. C., Caterini, A. L.,
Krishnan, R. G., and Loaiza-Ganem, G. A geometricexplanation of the likelihood OOD detection paradox.
arXiv:2403.18910, 2024b.

Bengio, Y., Courville, A., and Vincent, P. Representation
learning: A review and new perspectives. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(8):1798–1828, 2013.

Kirichenko, P., Izmailov, P., and Wilson, A. G. Why normalizing flows fail to detect out-of-distribution data. In Advances in Neural Information Processing Systems, 2020.

Brown, B. C., Caterini, A. L., Ross, B. L., Cresswell, J. C.,
and Loaiza-Ganem, G. Verifying the union of manifolds
hypothesis for image data. In International Conference
on Learning Representations, 2023.

Levina, E. and Bickel, P. Maximum likelihood estimation of
intrinsic dimension. In Advances in Neural Information
Processing Systems, 2004.

Cangelosi, R. and Goriely, A. Component retention inprincipal component analysis with application to cdna
microarray data. Biology Direct, 2:1–21, 2007.

Loaiza-Ganem, G., Ross, B. L., Cresswell, J. C., andCaterini, A. L. Diagnosing and fixing manifold overfitting in deep generative models. Transactions on Machine
Learning Research, 2022.

Durkan, C., Bekasov, A., Murray, I., and Papamakarios, G.
Neural spline flows. In Advances in Neural Information
Processing Systems, 2019.

Loaiza-Ganem, G., Ross, B. L., Hosseinzadeh, R., Caterini,
A. L., and Cresswell, J. C. Deep generative models
through the lens of the manifold hypothesis: A survey
and new connections. arXiv:2404.02954, 2024.

Durkan, C., Bekasov, A., Murray, I., and Papamakarios,
G. nflows: normalizing flows in PyTorch, November 2020. URL https://doi.org/10.5281/
zenodo.4296287.

Lu, Y., Wang, Z., and Bal, G. Mathematical analysis of
singularities in the diffusion model under the submanifold
assumption. arXiv:2301.07882, 2023.

Fukunaga, K. and Olsen, D. R. An algorithm for finding
intrinsic dimensionality of data. IEEE Transactions on
Computers, 100(2):176–183, 1971.

Ma, X., Li, B., Wang, Y., Erfani, S. M., Wijewickrema, S.,
Schoenebeck, G., Song, D., Houle, M. E., and Bailey, J.
Characterizing adversarial subspaces using local intrinsic
dimensionality. In International Conference on Learning
Representations, 2018.

Harrell Jr, F. E., Lee, K. L., and Mark, D. B. Multivariable
prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring andreducing errors. Statistics in Medicine, 15(4):361–387,
1996.
Ho, J., Jain, A., and Abbeel, P. Denoising diffusion probabilistic models. In Advances in Neural Information
Processing Systems, 2020.

MacKay, D. J. and Ghahramani, Z. Comments on “Maximum likelihood estimation of intrinsic dimension’ by
E. Levina and P. Bickel (2004). The Inference Group
Website, Cavendish Laboratory, Cambridge University,
2005.

Horvat, C. and Pfister, J.-P. On gauge freedom, conservativity and intrinsic dimensionality estimation in diffusion
models. In International Conference on Learning Representations, 2024.

Mayr, P. and Petras, V. Cross-concordances: terminology
mapping and its effectiveness for information retrieval.
arXiv:0806.3765, 2008.

Houle, M. E., Schubert, E., and Zimek, A. On the correlation
between local intrinsic dimensionality and outlierness. In
Similarity Search and Applications: 11th International
Conference, SISAP 2018, pp. 177–191. Springer, 2018.

Pope, P., Zhu, C., Abdelkader, A., Goldblum, M., and Goldstein, T. The intrinsic dimension of images and its impact
on learning. In International Conference on Learning
Representations, 2021.
5Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Rombach, R., Blattmann, A., Lorenz, D., Esser, P., andOmmer, B. High-resolution image synthesis with latentdiffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
10684–10695, 2022.

Tulchinskii, E., Kuznetsov, K., Kushnareva, L., Cherniavskii, D., Nikolenko, S., Burnaev, E., Barannikov, S.,
and Piontkovskaya, I. Intrinsic dimension estimation forrobust detection of ai-generated texts. In Advances in
Neural Information Processing Systems, 2023.

Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted
intervention–MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings,
part III 18, pp. 234–241. Springer, 2015.

Vahdat, A., Kreis, K., and Kautz, J. Score-based generative modeling in latent space. In Advances in Neural
Information Processing Systems, 2021.
Vincent, P. A connection between score matching and denoising autoencoders. Neural Computation, 23(7):1661–
1674, 2011.

Särkkä, S. and Solin, A. Applied stochastic differential
equations. Cambridge University Press, 2019.

von Platen, P., Patil, S., Lozhkov, A., Cuenca, P., Lambert,
N., Rasul, K., Davaadorj, M., and Wolf, T. Diffusers:
State-of-the-art diffusion models. https://github.
com/huggingface/diffusers, 2022.

Satopaa, V., Albrecht, J., Irwin, D., and Raghavan, B. Finding a "kneedle" in a Haystack: Detecting Knee Points in
System Behavior. In 31st International Conference onDistributed Computing Systems Workshops, pp. 166–171.
IEEE, 2011.

Zheng, Y., He, T., Qiu, Y., and Wipf, D. P. Learning
manifold dimensions with conditional variational autoencoders. In Advances in Neural Information Processing
Systems, 2022.

Schuhmann, C., Beaumont, R., Vencu, R., Gordon, C.,
Wightman, R., Cherti, M., Coombes, T., Katta, A., Mullis,C., Wortsman, M., et al. Laion-5b: An open large-scale
dataset for training next generation image-text models.
Advances in Neural Information Processing Systems, 35:
25278–25294, 2022.
Song, Y., Durkan, C., Murray, I., and Ermon, S. Maximum likelihood training of score-based diffusion models.
In Advances in Neural Information Processing Systems,
2021a.
Song, Y., Sohl-Dickstein, J., Kingma, D. P., Kumar, A., Ermon, S., and Poole, B. Score-based generative modelingthrough stochastic differential equations. In International
Conference on Learning Representations, 2021b.
Stanczuk, J., Batzolis, G., Deveney, T., and Schönlieb, C.-B.
Your diffusion model secretly knows the dimension of thedata manifold. arXiv:2212.12611, 2022.
Steck, H., Krishnapuram, B., Dehing-Oberije, C., Lambin,P., and Raykar, V. C. On ranking in survival analysis:
Bounds on the concordance index. In Advances in Neural
Information Processing Systems, volume 20, 2007.
Teles, J. Concordance coefficients to measure the agreement
among several sets of ranks. Journal of Applied Statistics,
39(8):1749–1764, 2012.
Tempczyk, P., Michaluk, R., Garncarek, L., Spurek, P.,
Tabor, J., and Golinski, A. LIDL: Local intrinsic dimension estimation using approximate likelihood. In International Conference on Machine Learning, pp. 21205–
21231, 2022.
6

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

A. Explicit Formulas
A.1. Variance-Exploding Diffusion Models
Variance-exploding DMs are such that f (x, t) = 0 with g being non-zero. In this case (Song et al., 2021b):
ψ(t) = 1,andσ 2 (t) =Z t

g 2 (u)du.

(7)0

Since g is non-zero, g 2 is positive, so that σ 2 is increasing, and thus injective. It follows that σ is also injective, so thatλ = σ/ψ = σ is injective. Equation 6 then implies that
FLIPD(x, t0 ) = D + σ 2 (t0 ) tr ∇s(x, t0 ) + ∥s(x, t0 )∥22 .
(8)
A.2. Variance-Preserving Diffusion Models (DDPMs)

Variance-preserving DMs are such that1f (x, t) = − β(t)x,2

andg(t) =

where β is a positive scalar function. In this case (Song et al., 2021b):1

ψ(t) = e− 2 B(t) ,and σ 2 (t) = 1 − e−B(t) ,We then have thatsλ(t) =pβ(t),where B(t) :=

(9)

Z t

β(u)du.

(10)0

σ 2 (t) p B(t)− 1.
= eψ 2 (t)(11)

Since β is positive, B is increasing and thus injective, from which it follows that λ is injective as well. Plugging everything
into Equation 6, we obtain:

 
1
1


 2
FLIPD(x, t0 ) = D + 1 − e−B(t0 ) tr ∇s e− 2 B(t0 ) x, t0 + s e− 2 B(t0 ) x, t0 2 .
(12)A.3. Sub-Variance-Preserving Diffusion Models
Sub-variance-preserving DMs are such that1f (x, t) = − β(t)x,2and g(t) =qβ(t) 1 − e−2B(t) ,where B(t) :=Z tβ(u)du,(13)

0

and where β is a positive scalar function. In this case (Song et al., 2021b):1

ψ(t) = e− 2 B(t) ,We then have that2and σ 2 (t) = 1 − e−B(t) .

1
1
σ(t)
λ(t) == e 2 B(t) − e− 2 B(t) = 2 sinh
ψ(t)




1
B(t) .
2

(14)

(15)

Since β is positive, B is increasing and thus injective, from which it follows that λ is injective as well due to the injectivityof sinh. Plugging everything into Equation 6, we obtain:2  
11
 2− 2 B(t0 )− 2 B(t0 )−B(t0 )
x, t0 + s e
x, t0 2 .
FLIPD(x, t0 ) = D + 1 − e(16)tr ∇s e7

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Table 3. Comparing the discretized DDPM notation with score-based DM side-by-side.
Term

DDPM (Ho et al., 2020)

Score-based DM (Song et al., 2021b)

Timestep(Noised out) datapoint
Diffusion process hyperparameter
Mean of transition kernel
Std of transition kernel
Network parameterizationt ∈ {0, 1, . . . , T }
xt
βt
√
√ᾱt
1 − ᾱt √−ϵ(x, t)/ 1 − ᾱtt/T = t ∈ [0, 1]
xt/T = xtβ(t/T ) = β(t)ψ(t/T ) = ψ(t)σ(t/T ) = σ(t)ŝ(x, t/T ) = ŝ(x, t)

B. Adapting FLIPD for DDPMs
Here, we adapt FLIPD for state-of-the-art DDPM architectures and follow the discretized notation from Ho et al. (2020)
where instead of using a continuous time index t from 0 to 1, a timestep t belongs instead to the sequence {0, . . . , T } withT being the largest timescale. We use the colour gold to indicate the notation used by Ho et al. (2020). We highlight that thecontent of this section is a summary of the equivalence between DDPMs and the score-based formulation established bySong et al. (2021b).
As a reminder, DDPMs can be viewed as discretizations of the forward SDE process of a DM, where the process turns into a
Markov noising process:pp(xt | xt−1 ) := N (xt ; 1 − βt · xt−1 , βt ID ).
(17)

We also use sub-indices t instead of functions evaluated at t to keep consistent with Ho et al. (2020)’s notation. This in turnimplies the following transition kernel:p(xt |x0 ) = N (xt ;where αt := 1 − βt and ᾱt :=√ᾱt x0 , (1 − ᾱt )ID )

(18)

Qt

s=1 αt .

DDPMs model the backward diffusion process (or denoising process) by modelling a network ϵ : RD × {1, . . . , T } → RDthat takes in a noised-out point and outputs a residual that can be used to denoise. In particular, for every t:
xt + ϵ(xt , t) ∼ p(xt−1 |xt ).

(19)Song et al. (2021b) show that one can draw an equivalence between the network ϵ(·, t) and the score network (see their
Appendix B). Here, we rephrase the connections in a more explicit manner where we note that:√−ϵ(x, t)/ 1 − ᾱt = s(x, t/T ).

(20)

Consequently, plugging into Equation 12, we get the following formula adapted for DDPMs:
FLIPD(x, t0 ) = D −ppp
1 − ᾱt0 tr ∇ϵ( ᾱt0 x, t0 ) + ∥ϵ( ᾱt0 x, t0 )∥22 ,(21)

where t0 = t0 × T (best viewed in colour). We include Table 3, which summarizes all of the equivalent notation whenmoving from DDPMs to score-based DMs and vice-versa.

C. Derivations
C.1. Diffusion Model Densities are Convolutions
In this section we give a concrete relation between the density of a DM and the convolution of data with Gaussian noise
defined in Equation 2. We show that for any arbitrary DM with transition kernel as in Equation 4, it holds thatlog ϱ(x, δ) = D log γ(δ) + log p γ(δ)x, t(δ) ,8(22)Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

where t(δ) := λ−1 (eδ ) and γ(δ) := ψ(t(δ)). We have that:p γ(δ)x, t(δ) =
=

Z

Z

Z


pt(δ)|0 γ(δ)x | x0 p(x0 , 0)dx0


(23)




N γ(δ)x; ψ t(δ) x0 , σ 2 t(δ) ID p(x0 , 0)dx0 
N γ(δ)x; γ(δ)x0 , σ 2 t(δ) ID p(x0 , 0)dx0
 !
Z
σ 2 t(δ)
1
 ID p(x0 , 0)dx0
N x; x0 , 2
= D
γ (δ)
ψ t(δ)
Z

 
1
= D
N x; x0 , λ2 t(δ) ID p(x0 , 0)dx0
γ (δ)
Z

1
1
= D
N x; x0 , e2δ ID p(x0 , 0)dx0 = D ϱ(x, δ),
γ (δ)
γ (δ)
=

(24)
(25)
(26)
(27)
(28)

where we used that N (ax; ax0 , σ 2 ID ) = a1D N (x; x0 , (σ 2 /a2 )ID ). Taking logarithms yields Equation 22.
C.2. Derivation of Equation 5First, we recall the Fokker-Planck equation associated with the SDE in Equation 1, which states that:1∂p(x, t) = −p(x, t) [∇ · f (x, t)] − ⟨f (x, t), ∇p(x, t)⟩ + g 2 (t) tr ∇2 p(x, t) .
∂t
2

(29)

We begin by using this equation to derive ∂/∂t log p(x, t). Noting that ∇p(x, t) = p(x, t)s(x, t), we have that:

Becausetr ∇2 p(x, t) = tr (∇ [p(x, t)s(x, t)]) = p(x, t) tr (∇s(x, t)) + tr s(x, t)∇p(x, t)⊤= p(x, t) tr (∇s(x, t)) + ∥s(x, t)∥22 .

(31)

∂
∂
p(x, t) = p(x, t) log p(x, t),
∂t
∂t

(32)

(30)

it then follows that
1∂log p(x, t) = − [∇ · f (x, t)] − ⟨f (x, t), s(x, t)⟩ + g 2 (t) tr (∇s(x, t)) + ∥s(x, t)∥22 .
∂t
2

(33)

Then, from Equation 22 and the chain rule, we get:

i


d h
∂
D log ψ t(δ) + log p ψ t(δ) x, t(δ)
log ϱ(x, δ) =
∂δ
dδ

 ⊤




 !
∂
∇ log p ψ t(δ) x, t(δ)

d
ψ
t(δ)
x ∂


=D
log ψ t(δ) +  ∂
t(δ)
∂t

dδ
∂δ
log p ψ t(δ) x, t(δ)
1
∂t

 !⊤
∂
"


 !#
∂
ψ t(δ)s γ(δ)x, t(δ)
∂
ψ
t(δ)
x
∂t

t(δ) D
+ ∂
=
∂t
∂δ
γ(δ)
log p γ(δ)x, t(δ)
1
∂t
#

"


D
E



∂
∂
D
∂
=
t(δ)
ψ t(δ)+ x, s γ(δ)x, t(δ)
log p γ(δ)x, t(δ) .
+
∂δ
∂t
γ(δ)
∂t
9

(34)

(35)

(36)

(37)

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Substituting Equation 33 into Equation 37 yields:



"
D

E
∂
∂
∂
D
log ϱ(x, δ) =t(δ)ψ t(δ)+ x, s γ(δ)x, t(δ)
∂δ
∂δ
∂t
γ(δ)

 D

E
− ∇ · f γ(δ)x, t(δ) − f γ(δ)x, t(δ) , s γ(δ)x, t(δ)

#
 

 2
1 2
+ g t(δ) tr ∇s γ(δ)x, t(δ) + s γ(δ)x, t(δ) 2 .
2(38)

From now on, to simplify notation, when dealing with a scalar function h, we will denote its derivative as h′ . Sincet(δ) = λ−1 (eδ ), the chain rule gives:
λ t(δ)eδ∂.
t(δ) = ′ −1 δ  = ′(39)∂δ
λ λ (e )λ t(δ)

So far, we have not used that f (x, t) = b(t)x, which implies that ∇ · f (x, t) = Db(t) and that ⟨f (x, t), s(x, t)⟩ =b(t)⟨x, s(x, t)⟩. Using these observations and Equation 39, Equation 38 becomes:!
"
λ t(δ)
ψ ′ t(δ)∂
 − b t(δ) D
ϱ(x, δ) = ′∂δλ t(δ)
ψ t(δ)DE+ ψ ′ t(δ) − b t(δ) ψ t(δ) x, s γ(δ)x, t(δ)
#
 

 2
1 2
(40)
+ g t(δ) tr ∇s γ(δ)x, t(δ) + s γ(δ)x, t(δ) 2 .
2
If we showed thatψ ′ (t) − b(t)ψ(t) = 0,

and thatλ(t) 2g (t) = σ 2 (t),2λ′ (t)(41)

then Equation 40 would simplify to Equation 5. From equation 5.50 in (Särkkä & Solin, 2019), we have thatψ ′ (t) = b(t)ψ(t),(42)

which shows that indeed ψ ′ (t) − b(t)ψ(t) = 0. Then, from equation 5.51 in (Särkkä & Solin, 2019), we also have thatand from the chain rule this gives that′σ 2 (t) = 2b(t)σ 2 (t) + g 2 (t),σ ′ (t) =2b(t)σ 2 (t) + g 2 (t)
.
2σ(t)

(43)

(44)

We now finish verifying Equation 41. Since λ(t) = σ(t)/ψ(t), the chain rule implies thatσ(t)σ(t)ψ(t)λ(t) 2g 2 (t)g 2 (t)
ψ(t)
g
(t)
=
=
σ ′ (t)ψ(t) − σ(t)ψ ′ (t) 22λ′ (t)σ ′ (t)ψ(t) − σ(t)b(t)ψ(t) 22
ψ (t)g 2 (t)σ(t)σ(t)
g 2 (t)= ′=22σ (t) − b(t)σ(t) 222b(t)σ (t) + g (t)− b(t)σ(t)2σ(t)
g 2 (t)2σ 2 (t)= σ 2 (t),=2b(t)σ 2 (t) + g 2 (t) − 2b(t)σ 2 (t) 2which, as previously mentioned, shows that Equation 40 simplifies to Equation 5.

10

(45)

(46)

(47)

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Table 4. Essential hyperparameter settings for the Diffusion models with MLP backbone.

Property
Learning rate
Optimizer
Scheduler
EpochsModel Configuration
10−4
AdamW
Cosine scheduling with 500 warmup steps (von Platen et al., 2022)
200, 400, 800, or 1000 based on the ambient dimension

Score-matching loss
SDE drift
SDE diffusionβ(t)Likelihood weighting (Song et al., 2021a)
f (x, t) := p
− 21 β(t)xg(t) := β(t)Linear interpolation: β(t) := 0.1 + 20t

Score network
MLP hidden sizes
Time embedding size

MLP
⟨4096, 2048, 2 × 1024, 3 × 512, 2 × 1024, 2048, 4096⟩
128

D. Experimental Details
Throughout all our experiments, we used an NVIDIA A100 GPU with 40GB of memory.
D.1. DM Hyperparameter Setup
Throughout our experiments, we use an MLP architecture with a bottleneck as our score network: 2 × L + 1 fully connected
layers with dimensions ⟨h1 , h2 , . . . , hL , . . . h2L+1 ⟩ forming a bottleneck, i.e., hL has the smallest size. Notably, for1 ≤ i ≤ L, the ith transform connects layer i − 1 (or the input) to layer i with a linear transform of dimensions “hi−1 × hi ”,
and the (L + i)th layer (or the output) not only contains input from the (L + i − 1)th layer but also, contains skip connections
from layer (L − i) (or the input), thus forming a linear transform of dimension “(hL+i−1 + hL−i ) × hL+i ”. For image
experiments, we scale and shift the pixel intensities to be zero-centered with a standard deviation of 1. In addition, we
embed times t ∈ (0, 1) using the scheme in (von Platen et al., 2022) and concatenate with the input before passing to thescore network. All hyperparameters are summarized in Table 4.
D.2. FLIPD Estimates and Curves for Synthetic Distributions
Figure 4 shows pointwise LID estimates for a lollipop distribution
taken from (Tempczyk et al., 2022). It is a uniform distribution overthree submanifolds: (i) a 2d candy, (ii) a 1d stick, and (iii) an isolated
point of zero dimensions. Note that the FLIPD estimates at t0 = 0.05
for all three submanifolds are coherent.
Figure 5 shows the FLIPD curve as training progresses on the lollipop
example and the Gaussian mixture that was already discussed in
Section 4. We see that gradually, knee patterns emerge at the correct
LID, indicating that the DM is learning the data manifold. Notably,data with higher LID values get assigned higher estimates even aftera few epochs, demonstrating that FLIPD effectively ranks data basedon LID, even when the DM is underfitted.
Finally, Figure 6 presents a summary of complex manifolds obtained
from neural spline flows and high-dimensional mixtures, showing
knees around the true LID.

Figure 4. The FLIPD estimates on a Lollipop dataset
from (Tempczyk et al., 2022).

D.3. A Simple Multiscale ExperimentTempczyk et al. (2022) argue that when setting δ, all the directions of data variation that have a log standard deviation below
δ are ignored. Here, we make this connection more explicit.
11Differentiable Local Intrinsic Dimension Estimation with Diffusion Models(a) After 10 epochs.

(b) After 20 epochs.

(c) After 100 epochs.

(d) After 10 epochs.

(e) After 20 epochs.

(f) After 100 epochs.

Figure 5. Evolution of the FLIPD curve while training the DM to fit a Lollipop (top) and a manifold mixture N2 + N4 + N8 ⊆ R10(bottom).

(a) F400 ⊆ R800 .

(b) U900 ⊆ R1000 .

(c) N10 + N25 + N50 ⊆ R100 .

Figure 6. The FLIPD curve for complex and high-dimensional manifolds.

We define a multivariate Gaussian distribution with a prespecified eigenspectrum for its covariance matrix: having three eigenvalues of 10−4 , three eigenvalues of 1, and four eigenvalues of 103 . This ensures that the distribution
is numerically 7d and that the directions and amount of data variation arecontrolled using the eigenvectors and eigenvalues of the covariance matrix.
For this multivariate Gaussian, the score function in Equation 5 can be written
in closed form; thus, we evaluate FLIPD both with and without training a DM.
We see in Figure 7 the estimates obtained in both scenarios match closely, withsome deviations due to imperfect model fit, which we found matches perfectly
when training for longer.

Figure 7. FLIPD curve for a multivariate

Gaussian with controlled covariance eigenApart from the initial knee at 7, which is expected, we find another at t0 = 0.6
spectrum.
(corresponding to eδ ≈ 6.178 with our hyperparameter setup in Table 4) wherethe value of FLIPD is 4. This indeed confirms that the estimator focuses solely
on the 4d space characterized by the eigenvectors having eigenvalues of 103 , and by ignoring the eigenvalues 10−4 and 1
which are both smaller than 6.178.

12

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

D.4. In-depth Analysis of the Synthetic Benchmark
Generating Manifold Mixtures To create synthetic data, we generate each component of our manifold mixture separately
and then join them to form a distribution. All mixture components are sampled with equal probability in the final distribution.
For a component with the intrinsic dimension of d, we first sample from a base distribution in d dimensions. This basedistribution is isotropic Gaussian and Laplace for Nd and Ld and uniform for the case of Ud and Fd . We then zero-pad thesesamples to match the ambient dimension D and perform a random D × D rotation on RD (each component has one suchtransformation). For Fd , we have an additional step to make the submanifold complex. We first initialize a neural spline
flow (using the nflows library (Durkan et al., 2020)) with 5 coupling transforms, 32 hidden layers, 32 hidden blocks, andtail bounds of 10. The data is then passed through the flow, resulting in a complex manifold embedded in RD with an LIDof d. Finally, we standardize each mixture component individually and set their modes such that the pairwise Euclidean
distance between them is at least 20. We then translate each component so its barycenter matches the corresponding mode,ensuring that data from different components rarely mixes, thus maintaining distinct LIDs.
LIDL Baseline Following the hyperparameter setup in (Tempczyk et al., 2022), we train 8 models with different
noised-out versions of the dataset with standard deviations eδi ∈ {0.01, 0.014, 0.019, 0.027, 0.037, 0.052, 0.072, 0.1}. The
normalizing flow backbone is taken from (Durkan et al., 2020), using 10 piecewise rational quadratic transforms with 32
hidden dimensions, 32 blocks, and a tail bound of 10. While (Tempczyk et al., 2022) uses an autoregressive architecture, we
use coupling transforms for increased training efficiency and to match the training time of a single DM.
Setup We use model-free estimators from the skdim library (Bac et al., 2021) and across our experiments, we sample
106 points from the synthetic distributions for either fitting generative models or fitting the model-free estimators. We thenevaluate LID estimates on a uniformly subsampled set of 212 points from the original set of 106 points. Some methods are
relatively slow, and this allows us to have a fair, yet feasible comparison. We do not see a significant difference even when
we double the size of the subsampled set. We focus on four different model-free baselines for our evaluation: ESS (Johnsson
et al., 2014), LPCA (Fukunaga & Olsen, 1971; Cangelosi & Goriely, 2007), MLE (Levina & Bickel, 2004; MacKay &Ghahramani, 2005), and FIS (Albergante et al., 2019), all with default settings. Note that FIS does not scale beyond 100
dimensions. Computing pairwise distances on high dimensions (D ≥ 800) on all 106 samples takes over 24 hours even with40 CPU cores. Therefore, for D ≥ 800, we use the same 212 subsamples we use for evaluation.
Evaluation We have three tables to summarize our analysis: (i) Table 6 shows the MAE of the LID estimates, comparing
each datapoint’s estimate to the ground truth at a fine-grained level; (ii) Table 7 shows the average LID estimate for synthetic
manifolds with only one component, this average is typically used to estimate global intrinsic dimensionality in baselines;
finally, (iii) looks at the concordance index (Harrell Jr et al., 1996) of estimates for cases with multiple submanifolds of
d N
different dimensionalities. Concordance indices for a sequence of LID estimates {LID}
n=1 are formally defined as follows:
 


X
N
N
d
d
d
(48)
)/
≤
LID
I(
LID
=
C {LIDn }N
,
{
LID
}
n2
n1
n n=1
n=1
2
1≤n1 ̸=n2 ≤N
LIDn1 ≤LIDn2

where I is the indicator function; a perfect estimator will have a C of 1. Instead of emphasizing the actual values of the LID
estimates, this metric assesses how well the ranks of an estimator align with those of ground truth (Steck et al., 2007; Mayr
& Petras, 2008; Teles, 2012; Kamkari et al., 2024a), thus evaluating LID as a “relative” measure of complexity.
Model-free Analysis Among model-free methods, LPCA and ESS show good performance in low dimensions (with
LPCA being exceptionally good) but falter as dimensions increase. As we see in Table 6, while model-free methods
outperform other estimators when D < 100, as D increases the paradigm shifts with model-based estimators outperforming
the model-free ones. In addition, as shown in Table 7, all model-free baselines underestimate intrinsic dimensionality tosome degree, with LPCA and MLE being particularly drastic for D ≥ 800. We note that ESS performs relatively well, evenbeating model-based methods in some 800-dimensional scenarios. However, we note that the C values in Table 8 suggest it
cannot rank data by LID as effectively as even the least performant MLE estimator.
Model-based Analysis Moving to model-based methods, we see in Table 8 that all perform poorly in ranking data basedon LID except our FLIPD estimator. Remarkably, FLIPD achieves perfect C values among all datasets; further justifying it
as a relative measure of complexity. We also note that while LIDL and NB provide better global estimates in Table 7 for
13

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Table 5. MAE (lower is better). Rows show synthetic manifolds and columns represent different variations of our Fokker-Planck-based
estimators.
Synthetic Manifold
FLIPD
FLIPD
FPRegress
FPRegress2

Lollipop in R
String within doughnut R3
Swiss Roll in R3
L5 ⊆ R10
N90 ⊆ R100
U10 + U30 + U90 ⊆ R100
F10 + F25 + F50 ⊆ R100
U10 + U80 + U200 ⊆ R800

t0 = .05

kneedlekneedleδ1 = −1

0.142
0.052
0.053
0.100
0.501
3.140
14.37
39.54

0.419
0.055
0.055
0.169
0.492
1.298
3.925
14.30

0.572
0.398
0.087
1.168
3.142
5.608
16.32
30.06

0.162
0.377
0.161
0.455
0.998
10.61721.01
29.99

high dimensions, they have worse MAE performance in Table 6. This once again suggests that at a local level, our estimatoris superior compared to others, beating all baselines in 2 out of 3 groups of synthetic manifolds with D ≥ 100 in Table 6.
Finally, we see a curious flipped behaviour with NB estimators where they perform better when increasing dimensionality.
D.5. Ablations
We begin by evaluating the impact of using kneedle. Our findings, summarized in Table 5, indicate that while setting asmall fixed t0 is effective in low dimensions, the advantage of kneedle becomes particularly evident as the number ofdimensions increases.
We introduce “FPRegress” which uses a single DM to perform the regression required by LIDL. To obtain LIDL estimates,
one must solve an ODE m times for a set of log standard deviations δ1 < . . . < δm . For a given origin δ1 , we set δi = i × δ1
and use Euler’s method with m − 1 steps to get log ϱ̂(x, δi ) for 1 ≤ i ≤ 8. Finally, a regression similar to LIDL is used to
extract the LID estimate. This estimator is not only slower than FLIPD, but as we see in Table 5, it has worse MAEs.
We also tried combining it with kneedle by sweeping over the origin δ1 and arguing that the estimates obtained from thismethod also exhibit knees. Despite some improvement in high-dimensional settings, Table 5 shows that even coupling itwith kneedle does not help.
D.6. Improving the NB Estimators with kneedle
We recall that the NB estimator requires computing rank S(x), where S(x) is a K × D matrix formed by stacking thescores ŝ(·, t0 ). We set t0 = 0.01 as it provides the most reasonable estimates. To compute rank S(x) numerically, Stanczuk
et al. (2022) perform a singular value decomposition on S(x) and use a cutoff threshold τ below which singular values are
considered zero. Finding the best τ is challenging, so Stanczuk et al. (2022) propose finding the two consecutive singularvalues with the maximum gap. Furthermore, we see that sometimes the top few singular values are disproportionately higher
than the rest, resulting in severe overestimations of the LID. Thus, we introduce an alternative algorithm to determine theoptimal τ . For each τ , we estimate LID by thresholding the singular values. Sweeping 100 different τ values from 0 to1000 at a geometric scale (to further emphasize smaller thresholds) produces estimates ranging from D (keeping all singularvalues) to 0 (ignoring all). As τ varies, we see that the estimates plateau over a certain range of τ . We use kneedle todetect this plateau because the starting point of a plateau is indeed a knee in the curve. This significantly improves thebaseline, especially in high dimensions: see the third column of Tables 6, 7, and 8 compared to the second column.

14Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Table 6. MAE (lower is better). Each row represents a synthetic dataset and each column represents an LID estimation method. Rows are
split into groups based on the ambient dimension: the first group of rows shows toy examples; the second shows low-dimensional datawith D = 10; the third shows moderate-dimensional data with D = 100; and the last two show high-dimensional data with D = 800 and
D = 1000, respectively.
Synthetic Manifold

FLIPD

NB Vanillakneedle

NB

LIDL

ESS

LPCA

MLE

FIS

kneedleLollipop in R2
Swiss Roll in R3
Doughnut Mixture in R3

0.419
0.055
0.055

0.577
0.998
1.475

0.855
0.016
0.414

0.052
0.532
1.104

0.009
0.017
0.017

0.000
0.000
0.000

0.142
0.165
0.128

0.094
0.018
0.041

Summary (Toy Manifolds)

0.176

1.017

0.428

0.563

0.014

0.000

0.145

0.051

N5 ⊆ R10
L5 ⊆ R10U5 ⊆ R10F5 ⊆ R10
N2 + N4 + N8 ⊆ R10
L2 + L4 + L8 ⊆ R10
U2 + U4 + U8 ⊆ R10
F2 + F4 + F8 ⊆ R10

0.084
0.169
0.324
0.666
0.287
0.253
0.586
0.622

5.000
1.000
4.994
1.216
5.706
5.708
5.677
5.709

0.005
0.146
0.933
0.765
1.078
0.772
3.685
2.187

0.071
0.101
0.123
0.487
0.308
0.515
0.363
1.013

0.061
0.068
0.153
0.168
0.156
0.193
0.331
0.428

0.000
0.000
0.000
0.000
0.000
0.001
0.115
0.115

0.441
0.462
0.451
0.497
0.406
0.437
0.540
0.642

0.206
0.203
0.186
0.176
0.206
0.018
0.222
0.269

Summary (10-dimensional)

0.066

0.381

0.161

0.211

0.005

0.000

0.054

0.019

U10 ⊆ RU30 ⊆ R100
U90 ⊆ R100
N30 ⊆ R100
N90 ⊆ R100
F80 ⊆ R100
U10 + U25 + U50 ⊆ R100
U10 + U30 + U90 ⊆ R100
N10 + N25 + N50 ⊆ R100
F10 + F25 + F50 ⊆ R100

0.910
0.505
0.640
0.887
0.492
1.869
0.868
1.298
1.813
3.925

30.11550.521
1.157
52.43
0.184
20.00
57.96
61.58
74.20
74.20

0.000
0.000
1.327
0.000
2.693
3.441
0.890
1.482
0.555
6.205

1.370
0.542
0.332
0.892
0.329
1.871
5.869
8.460
8.873
18.61

0.644
1.465
2.034
0.534
1.673
3.660
4.988
21.89
7.712
9.200

0.000
0.002
21.90
0.000
21.88
16.78
6.749
20.06
5.716
6.769

1.263
7.622
39.65
5.703
39.45
34.41
16.12
41.05
14.37
16.78

−
−
−
−
−
−
−
−
−
−

Summary (100-dimensional)

1.321

42.24

1.659

4.715

5.380

9.986

21.64

−

U200 ⊆ R
F400 ⊆ R800
U10 + U80 + U200 ⊆ R800

11.54
20.46
14.30

600.0
400.0
715.3

7.205
10.15
18.82

55.98
207.2
120.7

5.116
26.18
2.349

104.7
301.0
33.69

139.5
312.5
57.19

−
−
−

Summary (800-dimensional)

15.43

571.8

12.06

128.0

11.22

146.5

169.7

-

N900 ⊆ R
U100 ⊆ R1000
U900 ⊆ R1000
F500 ⊆ R1000

3.913
12.81
12.81
21.77

100.0
900.0
100.0
500.0

24.38
62.68
0.104
52.19

10.45
12.65
24.90
341.3

76.40
2.055
75.17
38.63

801.0
28.09
801.0
401.0

774.4
59.62
762.2
401.6

−
−
−
−

Summary (1000-dimensional)

12.83

400.0

34.84

97.33

48.06

507.8

499.5

−

100

800

1000

15

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Table 7. Average LID for manifolds with a single global intrinsic dimension. Rows represent uni-dimensional datasets and each column
represents an LID estimation method.
Synthetic Manifold

FLIPD

NB VanillakneedleSwiss Roll in R3
N5 ⊆ R10
L5 ⊆ R10U5 ⊆ R10F5 ⊆ R10U10 ⊆ R100
U30 ⊆ R100
U90 ⊆ R100
N30 ⊆ R100
N90 ⊆ R100
F80 ⊆ R100
U200 ⊆ R800
F400 ⊆ R800
N900 ⊆ R1000
U100 ⊆ R1000
U900 ⊆ R1000
F500 ⊆ R1000

2.012
5.017
4.968
4.796
4.722
11.68
31.01
89.54
30.79
89.88
77.97
211.5
454.7
890.3
135.76
864.9
582.7

NB

LIDL

ESS

LPCA

MLE

FIS

2.527
5.067
5.089
5.107
5.482
11.29
30.08
90.24
30.85
90.21
81.46
256.0
607.2
924.4
112.6
911.85
841.3

2.008
5.004
4.985
4.880
4.890
9.361
28.54
88.27
29.67
88.89
76.35
195.3
373.8
823.6
98.41
824.8
461.4

2.000
5.000
5.000
5.000
5.000
10.00
29.99
68.10
30.00
68.12
63.22
95.30
99.00
99.00
71.91
99.00
99.00

2.013
5.108
5.123
4.833
4.829
8.905
22.38
50.35
24.38
50.55
45.59
60.51
87.53
125.6
40.38
137.8
98.38

2.003
5.187
5.182
5.152
5.131
−
−
−
−
−
−
−
−
−
−
−
−

kneedle

2.998
10.00
6.000
9.994
6.216
40.12
80.52
91.16
82.43
90.18
100.0
800.0
800.0
1000.
1000.
1000.
1000.

1.984
4.995
4.854
4.067
4.461
10.00
30.00
91.32
30.00
92.62
83.23
207.2
410.1
924.4
162.7
900.1
552.2

Table 8. Concordance index (higher is better with 1.000 being the gold standard). Each row represents a mixture of multi-dimensional
manifolds, and each column represents an LID estimation method. This table evaluates how accurately different estimators rank datapoints
based on their LID.
LIDL
ESS
LPCA
MLE
FIS
Synthetic Manifold
FLIPD
NB Vanilla NB
kneedlekneedle2

Lollipop in R
Doughnut Mixture in R3
N2 + N4 + N8 ⊆ R10
L2 + L4 + L8 ⊆ R10
U2 + U4 + U8 ⊆ R10
F2 + F4 + F8 ⊆ R10
U10 + U25 + U50 ⊆ R100
U10 + U30 + U90 ⊆ R100
N10 + N25 + N50 ⊆ R100
F10 + F25 + F50 ⊆ R100
U10 + U80 + U200 ⊆ R800

1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000
1.000

0.426
0.483
0.341
0.342
0.334
0.342
0.467
0.342
0.342
0.342
0.342

0.394
0.486
0.725
0.752
0.462
0.578
0.879
0.826
0.866
0.731
0.841

0.999
0.565
0.943
0.884
0.903
0.867
0.759
0.742
0.736
0.695
0.697

16

1.000
1.000
1.000
1.000
1.000
1.000
0.855
0.742
0.878
0.847
1.000

1.000
1.000
1.000
1.000
1.000
1.000
0.897
0.855
0.917
0.897
1.000

1.000
1.000
1.000
1.000
1.000
0.999
1.000
1.000
1.000
1.000
1.000

1.000
1.000
1.000
1.000
1.000
1.000
−
−
−
−
−

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

E. Image Experiments
E.1. FLIPD Curves and DM Samples
Figures 8 and 9 show DM-generated samples and the associated LID curves for 4096 samples from the datasets. In Figure 9,
the FLIPD curves with MLPs, which have clearly discernible knees as predicted by the theory, are more reasonable thanthose generated with UNets. The fact that FLIPD estimates are worse for UNets is surprising given that MLPs produce
worse-looking samples, as shown in Figure 8. However, comparing the first and second rows of Figure 8, it also becomes
clear that MLP-generated images still match some characteristics of the true dataset, suggesting that they may still be
capturing the image manifold in useful ways. To explain the surprisingly poor FLIPD estimates of UNets, we hypothesizethat the convolutional layers in the UNet provide some inductive biases which, while helpful to produce visually pleasing
images, might also encourage the network to over-fixate on high-frequency features which are not visually perceptible.
For example, Kirichenko et al. (2020) showed how normalizing flows over-fixate on these features, and how this cancause them to assign large likelihoods to out-of-distribution data, even when the model produces visually convincing
samples. We hypothesize that a similar underlying phenomenon might be at play here, and that DMs with UNets might beover-emphasizing “high-frequency” directions of variation in their LID estimates, even if they learn the semantic ones andthus produce pleasing images. However, an exploration of this hypothesis is outside the scope of our work, and we highlightonce again that FLIPD remains a useful measure of complexity when using UNets.
E.2. UNet Architecture
We utilize UNet architectures from the diffusers library (von Platen et al., 2022). The DM setup mirrors that in Table 4
except for the score backbone. For greyscale images, we employ a convolutional block followed by two attention-based
downsampling blocks. The channel sizes are 128, 256, and 256, respectively. For colour images, we use two convolutional
downsampling blocks (each with 128 channels), followed by two attention downsampling blocks (each with 256 channels).
In both cases, these blocks are inverted using their mirrored upsampling counterparts.
E.3. Images Sorted by FLIPD
Figures 10, 11, 12, and 13 show 4096 samples of CIFAR10, SVHN, MNIST, and FMNIST sorted according to their
FLIPD estimate, showing a gradient transition from the least complex datapoints (e.g., the digit 1 in MNIST) to the mostcomplex ones (e.g., the digit 8 in MNIST). We use MLPs for greyscales and UNets for colour images but see similar trends
when switching between backbones.
E.4. How Many Hutchinson Samples are Needed?
Figure 14 compares the Spearman’s rank correlation coefficient between FLIPD estimates while we use k ∈ {1, 50}
Hutchinson samples vs. computing the trace deterministically with D Jacobian-vector-products. We see that: (i) Hutchinson
sampling is particularly well-suited for UNet backbones, having generally higher correlations compared to their MLP
counterparts; (ii) as t0 increases, the correlation becomes smaller, suggesting that the Hutchinson sample complexity
increases at larger timescales; (iii) for small t0 , even one Hutchinson sample is enough to estimate LID; (iv) for the UNet
backbone, 50 Hutchinson samples are enough and have a high correlation (larger than 0.8) even for t0 as large as 0.5.
E.5. Multiscale Analysis on Images
Figure 15 shows the correlation of FLIPD estimates with PNG compression for t0 ∈ (0, 1), indicating a consistently highcorrelation at small t0 , and a general decrease while increasing t0 . In addition, we see that UNet backbones correlate betterwith PNG.
Figures 16, 18, 20, and 22 show images with smallest and largest FLIPD estimates at different values of t0 for the UNet
backbone and Figures 17, 19, 21, and 23 show the same for the MLP backbone: (i) we see a clear difference in thecomplexity of top and bottom FLIPD estimates, especially for smaller t0 ; this difference becomes less distinct as t0
increases; (ii) interestingly, even for larger t0 values with smaller PNG correlations, we qualitatively observe a clustering of
the most complex datapoints at the end; however, the characteristic of this clustering changes. For example, see Figure 16
at t0 = 0.3 or Figure 21 and Figure 23 at t0 = 0.8, suggesting that FLIPD focuses on more coarse-grained measures ofcomplexity at these scales; and finally (iii) while MLP backbones underperform in sample generation, their orderings are
more meaningful, even showing coherent visual clustering up to t = 0.8 in all Figures 17, 19, 21, and 23.

17

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

E.6. Stable Diffusion
To test FLIPD with Stable Diffusion v1.5 (Rombach et al., 2022), which is known to have been finetuned on a subset of
LAION-Aesthetics, we sampled 1600 images from LAION-Aesthetics-650k and computed FLIPD scores for each.
We ran FLIPD with t0 ∈ {0.01, 0.1, 0.3, 0.8} and a single Hutchinson trace sample. In all cases, FLIPD ranking clearlycorresponded to complexity, though we decided upon t0 = 0.3 as best capturing the "semantic complexity" of image
contents. All 1600 images for t = 0.3 are depicted in Figure 24. For all timesteps, we show previews of the 8 lowest- andhighest-LID images in Figure 25. Note that LAION is essentially a collection of URLs, and some are outdated. For thecomparisons in Figure 25, we remove placeholder icons or blank images, which likely correspond to images that, at the timeof writing this paper, have been removed from their respective URLs and which are generally given among the lowest LIDs.

(a) CIFAR10 with UNet backbone.

(b) SVHN with UNet backbone.

(c) MNIST with UNet backbone.

(d) FMNIST with UNet backbone.

(e) CIFAR10 with MLP backbone.

(f) SVHN with MLP backbone.

(g) MNIST with MLP backbone.

(h) FMNIST with MLP backbone.

Figure 8. Samples from DMs with different score network backbones, using the same seed for control. Despite the variation in backbones,
images of the same cell in the grid (comparing top and bottom rows) show rough similarities, especially on CIFAR10 and SVHN.

(a) CIFAR10 with UNet backbone.

(b) SVHN with UNet backbone.

(c) MNIST with UNet backbone.

(d) FMNIST with UNet backbone.

(e) CIFAR10 with MLP backbone.

(f) SVHN with MLP backbone.

(g) MNIST with MLP backbone.

(h) FMNIST with MLP backbone.

Figure 9. FLIPD curves from all the different DMs with different score network backbones.

18

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Figure 10. CIFAR10 sorted (left to right and top to bottom) by FLIPD estimate (UNet) at t0 = 0.01.

19

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Figure 11. SVHN sorted (left to right and top to bottom) by FLIPD estimate (UNet) at t0 = 0.01.

20

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Figure 12. MNIST sorted (left to right and top to bottom) by FLIPD estimate (MLP) at t0 = 0.1.

21

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Figure 13. FMNIST sorted (left to right and top to bottom) by FLIPD estimate (MLP) at t0 = 0.1.

22

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models(a) Multiscale Spearman’s correlation on CIFAR10.

(b) Multiscale Spearman’s correlation on SVHN.

(c) Multiscale Spearman’s correlation on MNIST.

(d) Multiscale Spearman’s correlation on FMNIST.

Figure 14. Spearman’s correlation of FLIPD estimates while using different numbers of Hutchinson samples compared to computing
the trace term of FLIPD deterministically with D Jacobian vector product calls. These estimates are evaluated at different values oft0 ∈ (0, 1) on four datasets using the UNet and MLP backbones.

(a) Multiscale correlation with PNG on CIFAR10.

(b) Multiscale correlation with PNG on SVHN.

(c) Multiscale correlation with PNG on MNIST.

(d) Multiscale correlation with PNG on FMNIST.

Figure 15. Spearman’s correlation of FLIPD estimates with PNG for t0 ∈ (0, 1) on different backbones and different image datasets.

23

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models(a) t0 = 0.01.

(b) t0 = 0.1.

(c) t0 = 0.3.

(d) t0 = 0.8.

Figure 16. The 204 smallest (top) and 204 largest (bottom) CIFAR10 FLIPD estimates with UNet evaluated at different t0 .

(a) t0 = 0.01.

(b) t0 = 0.1.

(c) t0 = 0.3.

(d) t0 = 0.8.

Figure 17. The 204 smallest (top) and 204 largest (bottom) CIFAR10 FLIPD estimates with MLP evaluated at different t0 .

24

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models(a) t0 = 0.01.

(b) t0 = 0.1.

(c) t0 = 0.3.

(d) t0 = 0.8.

Figure 18. The 204 smallest (top) and 204 largest (bottom) SVHN FLIPD with UNet estimates evaluated at different t0 .

(a) t0 = 0.01.

(b) t0 = 0.1.

(c) t0 = 0.3.

(d) t0 = 0.8.

Figure 19. The 204 smallest (top) and 204 largest (bottom) SVHN FLIPD with MLP estimates evaluated at different t0 .

25

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models(a) t0 = 0.01.

(b) t0 = 0.1.

(c) t0 = 0.3.

(d) t0 = 0.8.

Figure 20. The 204 smallest (top) and 204 largest (bottom) MNIST FLIPD with UNet estimates evaluated at different t0 .

(a) t0 = 0.01.

(b) t0 = 0.1.

(c) t0 = 0.3.

(d) t0 = 0.8.

Figure 21. The 204 smallest (top) and 204 largest (bottom) MNIST FLIPD with MLP estimates evaluated at different t0 .

26

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models(a) t0 = 0.01.

(b) t0 = 0.1.

(c) t0 = 0.3.

(d) t0 = 0.8.

Figure 22. The 204 smallest (top) and 204 largest (bottom) FMNIST FLIPD estimates with UNet evaluated at different t0 .

(a) t0 = 0.01.

(b) t0 = 0.1.

(c) t0 = 0.3.

(d) t0 = 0.8.

Figure 23. The 204 smallest (top) and 204 largest (bottom) FMNIST FLIPD estimates with MLP evaluated at different t0 .

27

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models

Figure 24. 1600 images from LAION-Aesthetics-625K, sorted by FLIPD estimates with t0 = 0.3.

28

Differentiable Local Intrinsic Dimension Estimation with Diffusion Models(a) t0 = 0.01.

(b) t0 = 0.1.

(c) t0 = 0.8.

Figure 25. The 8 lowest- and highest-FLIPD values out of 1600 from LAION-Aesthetics-625k evaluated at different values of t0 .
Placeholder icons or blank images have been excluded from this comparison.

29