A Geometric View of Data Complexity: Efficient Local Intrinsic Dimension Estimation with Diffusion Models

Authors: hamidreza  kamkari
brendan  leigh
ross  rasa hosseinzadeh
jesse c cresswell
gabriel  loaiza-ganem
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 17:48:48

Abstract: High-dimensional data commonly lies on low-dimensional submanifolds, and estimating the local intrinsic dimension (LID) of a datum -- i.e. the dimension of the submanifold it belongs to -- is a longstanding problem. LID can be understood as the number of local factors of variation: the more factors of variation a datum has, the more complex it tends to be. Estimating this quantity has proven useful in contexts ranging from generalization in neural networks to detection of out-of-distribution data, adversarial examples, and AI-generated text. The recent successes of deep generative models present an opportunity to leverage them for LID estimation, but current methods based on generative models produce inaccurate estimates, require more than a single pre-trained model, are computationally intensive, or do not exploit the best available deep generative models: diffusion models (DMs). In this work, we show that the Fokker-Planck equation associated with a DM can provide an LID estimator which addresses the aforementioned deficiencies. Our estimator, called FLIPD, is easy to implement and compatible with all popular DMs. Applying FLIPD to synthetic LID estimation benchmarks, we find that DMs implemented as fully-connected networks are highly effective LID estimators that outperform existing baselines. We also apply FLIPD to natural images where the true LID is unknown. Despite being sensitive to the choice of network architecture, FLIPD estimates remain a useful measure of relative complexity; compared to competing estimators, FLIPD exhibits a consistently higher correlation with image PNG compression rate and better aligns with qualitative assessments of complexity. Notably, FLIPD is orders of magnitude faster than other LID estimators, and the first to be tractable at the scale of Stable Diffusion.

Main paper: 
 introduction: 
 The manifold hypothesis [7] , which has been empirically verified in contexts ranging from natural images [49, 10] to calorimeter showers in physics [15] , states that high-dimensional data of interest in R D often lies on low-dimensional submanifolds of R D . For a given datum x ∈ R D , this hypothesis motivates using its local intrinsic dimension (LID), denoted LID(x), as a natural measure of its complexity. LID(x) corresponds to the dimension of the data manifold that x belongs to, and can be intuitively understood as the minimal number of variables needed to describe x. More complex data needs more variables to be adequately described, as illustrated in Figure 1 . Data manifolds are typically not known explicitly, meaning that LID must be estimated. Here we tackle the following problem: given a dataset along with a query datum x, how can we tractably estimate LID(x)? This is a longstanding problem, with LID estimates being highly useful due to their innate interpretation as a measure of complexity. For example, these estimates can be used to detect outliers [27, 2, 32] , AI-generated text [64] , and adversarial examples [42] . Connections between the generalization achieved by a neural network and the LID estimates of its internal representations have also Figure 1 : (Left) A cartoon illustration showing that LID is a natural measure of relative complexity. We depict two manifolds of MNIST digits, corresponding to 1s and 8s, as 1-dimensional and 2-dimensional submanifolds of R 3 , respectively. The relatively simpler manifold of 1s exhibits a single factor of variation ("tilt"), whereas 8s have an additional factor of variation ("disproportionality"). (Right) The 4 lowest-and highest-LID datapoints from a subsample of LAION-Aesthetics, as measured by our method, FLIPD, applied to Stable Diffusion v1.5. FLIPD scales efficiently to large models on high-dimensional data, and aligns closely with subjective complexity.been shown [4, 8, 44, 9] . These insights can be leveraged to identify which representations contain maximal semantic content [66] , and help explain why LID estimates can be helpful as regularizers [72] and for pruning large models [70] . LID estimation is thus not only of mathematical and statistical interest, but can also benefit the empirical performance of deep learning models at numerous tasks.Traditional estimators of intrinsic dimension [22, 38, 43, 12, 30, 20, 1, 5] typically rely on pairwise distances and nearest neighbours, so computing them is prohibitively expensive for large datasets. Recent work has thus sought to move away from these model-free estimators and instead take advantage of deep generative models which learn the distribution of observed data. When this distribution is supported on low-dimensional submanifolds of R D , successful generative models must implicitly learn the dimensions of the data submanifolds, suggesting they can be used to construct LID estimators. However, existing model-based estimators suffer from various drawbacks, including being inaccurate and computationally expensive [59] , not leveraging the best existing generative models [62, 71] (i.e. diffusion models [56, 25, 58] ), and requiring training several models [62] or altering the training procedure rather than relying on a pre-trained model [26] . Importantly, none of these methods scale to high-resolution images such as those generated by Stable Diffusion [51] .We address these issues by showing how LID can be efficiently estimated using only a single pretrained diffusion model (DM) by building on LIDL [62] , a model-based estimator. LIDL operates by convolving data with different levels of Gaussian noise, training a normalizing flow [50, 17, 18] for each level, and fitting a linear regression using the log standard deviation of the noise as a covariate and the corresponding log density (of the convolution) evaluated at x as the response; the resulting slope is an estimate of LID(x) -D, thanks to a surprising result linking Gaussian convolutions and LID. We first show how to adapt LIDL to DMs in such a way that only a single DM is required (rather than many normalizing flows). Directly applying this insight leads to LIDL estimates that require one DM but several calls to an ordinary differential equation (ODE) solver; we further show how to circumvent this with an alternative ODE that computes all the required log densities in a single solve. We then argue that the slope of the regression in LIDL aims to capture the rate of change of the marginal log probabilities in the diffusion process, which can be evaluated directly thanks to the Fokker-Planck equation. The resulting estimator, which we call FLIPD, 1 is highly efficient, and circumvents the need for an ODE solver. Notably, FLIPD is differentiable; this property opens up exciting avenues for future research as it enables backpropagating through LID estimates.Our contributions are: (i) showing how DMs can be efficiently combined with LIDL in a way which requires a single call to an ODE solver; (ii) leveraging the Fokker-Planck equation to propose FLIPD, thus improving upon the estimator and circumventing the need for an ODE solver altogether; (iii) motivating FLIPD theoretically; (iv) introducing an expanded suite of LID estimation benchmark tasks that reveals gaps in prior evaluations, and specifically that other estimators do not remain accurate as the complexity of the manifold increases; (v) demonstrating that when using fullyconnected architectures for diffusion models, FLIPD outperforms existing baselines -especially as dimension increases -while being much more computationally efficient; (vi) showing that when applied to natural images, despite varying across network architecture (i.e., fully-connected network or UNet [52, 68] ), FLIPD estimates consistently align with other measures of complexity such as PNG compression length, and with qualitative assessments of complexity, highlighting that the LID estimates provided by FLIPD remain valid measures of relative image complexity; and (vii) demonstrating that when applied to the latent space of Stable Diffusion, FLIPD can estimate LID for extremely high-resolution images (∼ 10 6 pixel dimensions) for the first time.2 Background and Related Work 
diffusion models: 
 Forward and backward processes Diffusion models admit various formulations [56, 25] ; here we follow the score-based one [58] . We denote the true data-generating distribution, which DMs aim to learn, as p(•, 0). DMs define the forward (Itô) stochastic differential equation (SDE), dX t = f (X t , t)dt + g(t)dW t , X 0 ∼ p(•, 0),where f : R D × [0, 1] → R D and g : [0, 1] → R are hyperparameters, and W t denotes a Ddimensional Brownian motion. We write the distribution of X t as p(•, t). The SDE in Equation 1prescribes how to gradually add noise to data, the idea being that p(•, 1) is essentially pure noise. Defining the backward process as Y t := X 1-t , this process obeys the backward SDE [3, 24] ,EQUATIONwhere s(x, t) := ∇ log p(x, t) is the unknown (Stein) score function, 2 and Ŵt is another Ddimensional Brownian motion. DMs leverage this backward SDE for generative modelling by using a neural network ŝ : R D × (0, 1] → R D to learn the score function with denoising score matching [67] . Once trained, ŝ(x, t) ≈ s(x, t). To generate samples Ŷ1 from the model, we solve an approximation of Equation 2:EQUATIONwith ŝ replacing the true score and with p(•, 1), a Gaussian distribution chosen to approximate p(•, 1) (depending on f and g), replacing p(•, 1).Density Evaluation DMs can be interpreted as continuous normalizing flows [13] , and thus admit density evaluation, meaning that if we denote the distribution of Ŷ1-t as p(•, t), then p(x, t 0 ) can be mathematically evaluated for any given x ∈ R D and t 0 ∈ (0, 1]. More specifically, this is achieved thanks to the (forward) ordinary differential equation (ODE) associated with the DM:EQUATIONSolving this ODE from time t 0 to time 1 produces the trajectory (x t ) t∈[t0,1] , which can then be used for density evaluation through the continuous change-of-variables formula:EQUATIONwhere p(•, 1) can be evaluated since it is a Gaussian, and where v(x, t) := f (x, t) -g 2 (t)ŝ(x, t)/2.Trace estimation Note that the cost of computing ∇v(x t , t) for a particular xt amounts to Θ(D) function evaluations of ŝ (since D calls to a Jacobian-vector-product routine are needed [6] ). Although this is not prohibitively expensive for a single xt , in order to compute the integral in Equation 5 in practice, (x t ) t∈[t0,1] must be discretized into a trajectory of length N . If we denote by F the cost of evaluating v(x t , t) -or equivalently, ŝ(x t , t) -deterministic density evaluation is Θ(N DF ), which is computationally prohibitive. The Hutchinson trace estimator [29] -which states that forM ∈ R D×D , tr(M ) = E ε [ε ⊤ M ε],where ε ∈ R D has mean 0 and covariance I D -is thus commonly used for stochastic density estimation; approximating the expectation with k samples from ε results in a cost of Θ(N kF ), which is much faster than deterministic density evaluation when k ≪ D.LID Various definitions of intrinsic dimension exist [28, 21, 37, 11] . Here we follow the standard one from geometry: a d-dimensional manifold is a set which is locally homeomorphic to R d . For a given disjoint union of manifolds and a point x in this union, the local intrinsic dimension of x is the dimension of the submanifold it belongs to. Note that LID is not an intrinsic property of the point x, but rather a property of x with respect to the manifold that contains it. Intuitively, LID(x) corresponds to the number of factors of variation present in the manifold containing x, and it is thus a natural measure of the relative complexity of x, as illustrated in Figure 1 .Estimating LID The natural interpretation of LID as a measure of complexity makes estimating it from observed data a relevant problem. Here, the formal setup is that p(•, 0) is supported on a disjoint union of manifolds [10] , and we assume access to a dataset sampled from it. Then, for a given x in the support of p(•, 0), we want to use the dataset to provide an estimate of LID(x). Traditional estimators [22, 38, 43, 12, 30, 20, 1, 5] rely on the nearest neighbours of x in the dataset, or related quantities, and typically have poor scaling in dataset size. Generative models are an intuitive alternative to these methods; because they are trained to learn p(•, 0), when they succeed they must encode information about the support of p(•, 0), including the corresponding manifold dimensions. However, extracting this information from a trained generative model is not trivial. For example, Zheng et al. [71] showed that the number of active dimensions in the approximate posterior of variational autoencoders [33, 50] estimates LID, but their approach does not generalize to better generative models.LIDL Tempczyk et al. [62] proposed LIDL, a method for LID estimation relying on normalizing flows as tractable density estimators [50, 17, 18] . LIDL works thanks to a surprising result linking Gaussian convolutions and LID [39, 62, 71] . We will denote the convolution of p(•, 0) and Gaussian noise with log standard deviation δ as ϱ(•, δ), i.e.ϱ(x, δ) := p(x 0 , 0)N (x -x 0 ; 0, e 2δ I D )dx 0 .The aforementioned result states that, under mild regularity conditions on p(•, 0), and for a given x in its support, the following holds as δ → -∞:EQUATIONThis result then suggests that, for negative enough values of δ (i.e. small enough standard deviations):EQUATIONfor some constant c. If we could evaluate log ϱ(x, δ) for various values of δ, this would provide an avenue for estimating LID(x): set some values δ 1 , . . . , δ m , fit a linear regression using {(δ i , log ϱ(x, δ i ))} m i=1 with δ as the covariate and log ϱ(x, δ) as the response, and let βx be the corresponding slope. It follows that βx estimates LID(x) -D, so that LID(x) ≈ D + βx is a sensible estimator of local intrinsic dimension.Since ϱ(x, δ) is unknown and cannot be evaluated, LIDL requires training m normalizing flows. More specifically, for each δ i , a normalizing flow is trained on data to which N (0, e 2δi I D ) noise is added. In LIDL, the log densities of the trained models are then used instead of the unknown true log densities log ϱ(x, δ i ) when fitting the regression as described above.Despite using generative models, LIDL has obvious drawbacks. LIDL requires training several models. It also relies on normalizing flows, which are not only empirically outperformed by DMs by a wide margin, but are also known to struggle to learn low-dimensional manifolds [14, 39, 40] . On the other hand, DMs do not struggle to learn p(•, 0) even when it is supported on low-dimensional manifolds [48, 16, 40] , further suggesting that LIDL can be improved by leveraging DMs.Estimating LID with DMs The only works we are aware of that leverage DMs for LID estimation are those of Stanczuk et al. [59] , and Horvat and Pfister [26] . The latter modifies the training procedure of DMs, so we focus on the former since we see compatibility with existing pre-trained models as an important requirement for DM-based LID estimators. Stanczuk et al. [59] consider variance-exploding DMs, where f = 0. They show that, as t ↘ 0, the score function s(x, t) points orthogonally towards the manifold containing x, or more formally, it lies in the normal space of this manifold at x. They thus propose the following LID estimator, which we refer to as the normal bundle (NB) estimator: first run Equation 1 until time t 0 starting from x, and evaluate ŝ(•, t 0 ) at the resulting value; then repeat this process K times and stack the K resulting D-dimensional vectors into a matrix S(x) ∈ R D×K . The idea here is that if t 0 is small enough and K is large enough, the columns of S(x) span the normal space of the manifold at x, suggesting that the rank of this matrix estimates the dimension of this normal space, namely D -LID(x). Finally, they estimate LID(x) as:EQUATIONNumerically, the rank is computed by performing a singular value decomposition (SVD) of S(x), setting a threshold, and counting the number of singular values exceeding the threshold. Computing S(x) requires K function evaluations, and intuitively K should be large enough to ensure the columns of S(x) span the normal space at x; the authors thus propose using K = 4D, and recommend always at least ensuring that K > D. Computing the NB estimator costs Θ(KF + D 2 K), where F again denotes the cost for evaluating ŝ. Thus, although the NB estimator addresses some of the limitations of LIDL, it remains computationally expensive in high dimensions.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We state the linearity assumption in Theorem 3.1 is an idealization, we are straightforward about the lack of knees on the FLIPD curves with UNets, and we highlight this in Section 5.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
method: 
 Although Tempczyk et al. [62] only used normalizing flows in LIDL, they did point out that these models could be swapped for any other generative model admitting density evaluation. Indeed, one could trivially train m DMs and replace the flows with them. Throughout this section we provide a sequence of progressive improvements to this naïve application of LIDL with DMs, culminating with FLIPD. We assume access to a pre-trained DM such that f (x, t) = b(t)x for a function b : [0, 1] → R. This choice implies that the transition kernel p t|0 associated with Equation 1 is Gaussian [53] :EQUATIONwhere ψ, σ : [0, 1] → R. We also assume that b and g are such that ψ and σ are differentiable and such that λ(t) := σ(t)/ψ(t) is injective. This setting encompasses all DMs commonly used in practice, including variance-exploding, variance-preserving (of which the widely used DDPMs [25] are a discretized instance), and sub-variance-preserving [58] . In Appendix A we include explicit formulas for ψ(t), σ 2 (t), and λ(t) for these particular DMs.As opposed to the several normalizing flows used in LIDL which are individually trained on datasets with different levels of noise added, a single DM already works by convolving data with various noise levels and allows density evaluation of the resulting noisy distributions (Equation 5). Hence, we make the observation that LIDL can be used with a single DM. All we need is to relate ϱ(•, δ) to the density of the DM, p(•, t). In the case of variance-exploding DMs, ψ(t) = 1, so we can easily use the defining property of the transition kernel in Equation 10 to getEQUATIONwhich equals ϱ(x, δ) from Equation 6when we choose t = σ -1 (e δ ). 3 In turn, we can use LIDL with a single variance-exploding DM by evaluating each p x, σ -1 (e δi ) through Equation 5. This idea extends beyond variance-exploding DMs; in Appendix B.1 we show that for any arbitrary DM with transition kernel as in Equation 10, it holds that log ϱ(x, δ) = D log ψ t(δ) + log p ψ t(δ) x, t(δ) ,where t(δ) := λ -1 (e δ ). This equation is relevant because LIDL requires log ϱ(•, δ), yet DMs provide log p(•, t): linking these two quantites as above shows that LIDL can be used with a single DM.Using Equation 12with LIDL still involves computing log p ψ(t(δ i ))x, t(δ i ) through Equation 5for each i = 1, . . . , m before running the regression. Since each of the corresponding ODEs in Equation 4 starts at a different time t 0 = t(δ i ) and is evaluated at a different point ψ(t(δ i ))x, this means that a different ODE solver call would have to be used for each i, resulting in a prohibitively expensive procedure. To address this, we aim to find an explicit formula for ∂/∂δ log ϱ(x, δ). We do so by leveraging the Fokker-Planck equation associated with Equation 1, which provides an explicit formula for ∂/∂t p(x, t). Using this equation along with the chain rule and Equation 12, we show in Appendix B.2 that, for DMs with transition kernel as in Equation 10,EQUATION=: ν t(δ); s, x . (Then, assuming without loss of generality that δ 1 < • • • < δ m , we can use the above equation to define log ρ(x, δ) through the ODEEQUATION)Solving this ODE from δ 1 to δ m produces the trajectory (log ρ(x, δ)) δ∈[δ1,δm] . Since ν(t(δ); ŝ, x) does not depend on ρ(x, δ), when ŝ = s the solution to the ODE above will be off by a constant, i.e., log ρ(x, δ) = log ϱ(x, δ) + c init for some c init that depends on the initial condition of the ODE (0 in this case) but not on δ. Furthermore, while setting the initial condition to 0 might at first appear odd, recall that LIDL fits a regression using {(δ i , log ρ(x, δ i ))} m i=1 , and thus c init will be absorbed in the intercept without affecting the slope. In other words, the initial condition is irrelevant, and we can use LIDL with DMs by using a single call to an ODE solver on Equation 14.The LIDL estimator with DMs presented in Section 3.2 provides a massive speedup over the naïve approach of training m DMs, and over the method from Section 3.1 requiring m ODE solves. Yet, solving Equation 14 involves computing the trace of the Jacobian of ŝ multiple times within an ODE solver, which, as mentioned in Section 2.1, remains expensive. In this section we present our LID estimator, FLIPD, which circumvents the need for an ODE solver altogether. Recall that LIDL is based on Equation 8, which justifies the regression. Differentiating this equation yields that ∂/∂δ log ϱ(x, δ 0 ) ≈ LID(x) -D for negative enough δ 0 , meaning that Equation 13 directly provides the rate of change that the regression in LIDL aims to estimate, from which we getEQUATION)where t 0 := t(δ 0 ). Computing FLIPD is very cheap since the trace of the Jacobian of ŝ has to be evaluated only once when calculating ν(t(δ 0 ); ŝ, x). As mentioned in Section 2.1, exact evaluation of this trace has a cost of Θ(DF ), but can be reduced to Θ(kF ) when using the stochastic Hutchinson trace estimator with k ≪ D samples. Notably, FLIPD provides a massive speedup over the NB estimator -Θ(kF ) vs. Θ(KF + D 2 K) -especially in high dimensions where K > D ≫ k. In addition to not requiring an ODE solver, computing FLIPD requires setting only a single hyperparameter, δ 0 . Furthermore, since ν(t(δ); ŝ, x) depends on δ only through t(δ), we can directly set t 0 as the hyperparameter rather than δ 0 , which avoids the potentially cumbersome computation of t(δ 0 ) = λ -1 (e δ0 ): instead of setting a suitably negative δ 0 , we set t 0 > 0 sufficiently close to 0. In Appendix A we include explicit formulas for FLIPD(x, t 0 ) for common DMs.Finally, we present a theoretical result further justifying FLIPD. Note that the O(1) term in Equation 7need not be constant in δ as in Equation 8, even if it is bounded. The more this term deviates from a constant, the more bias we should expect in both LIDL and FLIPD. The following result shows that in an idealized linear setting, FLIPD is unaffected by this problem:Theorem 3.1 (FLIPD Soundness: Linear Case). Let L be an embedded submanifold of R D given by a d-dimensional affine subspace. If p(•, 0) is supported on L, continuous, and with finite second moments, then for any x ∈ L with p(x, 0) > 0, we have:EQUATIONProof. See Appendix B.3.We conjecture that our theorem can be extended to non-linear submanifolds since it is a local result and every manifold can be locally linearly approximated by its tangent plane. More specifically, ϱ(x, δ) becomes "increasingly local as δ → -∞" in the sense that its dependence on the values p(x 0 , 0) becomes negligible as δ → -∞ when x 0 is not in a close enough neighbourhood of x; this is because N (0, e 2δ I D ) concentrates most of its mass around 0 as δ → -∞ (see Equation 6). However, we leave generalizing our result to future work.Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean. • It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments: 
 Throughout this section, we use variance-preserving DMs, the most popular variant of DMs. We provide a "dictionary" to translate between the score-based formulation of FLIPD and DDPMs in Appendix C. We hope this will enable practitioners who are less focused on the theoretical aspects to effortlessly apply FLIPD to their pre-trained DMs. Our code is available at https: //github.com/layer6ai-labs/flipd; see Appendix D for more experimental details. The effect of t 0 FLIPD requires setting t 0 close to 0 since all the theory holds in the δ → -∞ regime. It is important to note that DMs fitted to low-dimensional manifolds are known to exhibit numerically unstable scores s(•, t 0 ) as t 0 ↘ 0 [65, 41, 40] . This fact does not invalidate FLIPD, but it suggests that it might be sensitive to the choice of t 0 . Our first set of experiments examines the effect of t 0 on FLIPD(x, t 0 ) by varying t 0 within the range (0, 1).In Figure 2 , we train DMs on two distributions: (i) a mixture of three isotropic Gaussians with dimensions 2, 4, and 8, embedded in R 10 (each embedding is carried out by multiplication against a random matrix with orthonormal columns plus a random translation); and (ii) a "string within a doughnut", which is a mixture of uniform distributions on a 2d torus (with a major radius of 10 and a minor radius of 1) and a 1d circle (aligning with the major circle of the torus) embedded in R 3 (this union of manifolds is shown in the upper half of Figure 3 ). While FLIPD(x, t 0 ) is inaccurate at t 0 = 0 due to the aforementioned instabilities, it quickly stabilizes around the true LID for all datapoints. We refer to this pattern as a knee in the FLIPD curve. In Appendix D.2, we show similar curves for more complex data manifolds.FLIPD is a multiscale estimator Interestingly, in Figure 2b we see that the blue FLIPD curve (corresponding to "doughnut" points with LID of 2) exhibits a second knee at 1, located at the t 0 shown with a vertical line. This confirms the multiscale nature of convolution-based estimators, first postulated by Tempczyk et al. [62] in the context of normalizing flows; they claim that when selecting a log standard deviation δ, all directions along which a datum can vary having log standard deviation less than δ are ignored. The second knee in Figure 2b can be explained by a similar argument: the torus looks like a 1d circle when viewed from far away, and larger values of t 0 correspond to viewing the manifolds from farther away. This is visualized in Figure 3 with two views of the "string within a doughnut" and corresponding LID estimates: one zoomed-in view where t 0 is small, providing fine-grained LID estimates, and a zoomed-out view where t 0 is large, making both the string and doughnut appear as a 1d circle from this distance. In Appendix D.3 we have an experiment that makes the multiscale argument explicit. Finding knees As mentioned, we persistently see knees in FLIPD curves. This in line with the observations of Tempczyk et al. [62] (see Figure 5 of [62] ), and it gives us a fully automated approach to setting t 0 . We leverage kneedle [54] , a knee detection algorithm which aims to find points of maximum curvature. When computationally sensible, rather than fixing t 0 , we evaluate Equation 15for 50 values of t 0 and pass the results to kneedle to automatically detect the t 0 where a knee occurs.Experimental setup We create a benchmark for LID evaluation on complex unions of manifolds where the true LID is known. We sample from simple distributions on low-dimensional spaces, and then embed the samples into R D . We denote uniform, Gaussian, and Laplace distributions as U, N , and L, respectively, with sub-indices indicating LID, and a plus sign denoting mixtures. To embed samples into higher dimensions, we apply a random matrix with orthonormal columns and then apply a random translation. For example, N 10 + L 20 ⊆ R 100 indicates a 10-dimensional Gaussian and a 20-dimensional Laplace, each of which undergoes a random affine transformation mapping to R 100 (one transformation per component). We also generate non-linear manifolds, denoted with F, by applying a randomly initialized D-dimensional neural spline flow [18] after the affine transformation (when using flows, the input noise is always uniform); since the flow is a diffeomorphism, it preserves LID. To our knowledge, this synthetic LID benchmark is the most extensive to date, revealing surprising deficiencies in some well-known traditional estimators. For an in-depth analysis, see Appendix D.4.Results Here, we summarize our synthetic experiments in Table 1 using two metrics of performance: the mean absolute error (MAE) between the predicted and true LID for individual datapoints; and the concordance index, which measures similarity in the rankings between the true LIDs and the estimated ones (note that this metric only makes sense when the dataset has variability in its ground truth LIDs, so we only report it for the appropriate entries in Table 1 ). We compare against the NB and LIDL estimators described in Section 2.2, as well as two of the most performant model-free baselines: LPCA [22, 12] and ESS [30] . For the NB baseline, we use the exact same DM backbone as for FLIPD (since NB was designed for variance-exploding DMs, we use the adaptation to variance-preserving DMs used in [32] , which produces extremely similar results), and for LIDL we use 8 neural spline flows. In terms of MAE, we find that FLIPD tends to be the best model-based estimator, particularly as dimension increases. Although model-free baselines perform well in simplistic scenarios, they produce unreliable results as LID increases or more non-linearity is introduced in the data manifold.In terms of concordance index, FLIPD achieves perfect scores in all scenarios, meaning that even when its estimates are off, it always provides correct LID rankings. We include additional results in the appendices: in Appendix D.5 we ablate FLIPD, finding that using kneedle indeed helps, and that FLIPD also outperforms the efficient implementation of LIDL with DMs described in Section 3.2 that uses an ODE solver. We notice that NB with the setting proposed in [59] consistently produces estimates that are almost equal to the ambient dimension; thus, in Appendix D.6 we also show how NB can be significantly improved upon by using kneedle, although it is still outperformed by FLIPD in many scenarios. In addition, in Table 7 and Table 8 we compare against other model-free baselines such as MLE [38, 43] and FIS [1] . We also consider datasets with a single (uni-dimensional) manifold where the average LID estimate can be used to approximate the global intrinsic dimension.Our results in Table 9 demonstrate that although model-free baselines indeed accurately estimate global intrinsic dimension, they perform poorly when focusing on (pointwise) LID estimates.We first focus on the simple image datasets MNIST [36] and FMNIST [69] . We flatten the images and use the same MLP architecture as in our synthetic experiments. Despite using an MLP, our DMs can generate reasonable samples (Appendix E.1) and the FLIPD curve for both MNIST and FMNIST is shown in Figure 4a . The knee points are identified at t 0 = 0.1, resulting in average LID estimates of approximately 130 and 170, respectively. Evaluating LID estimates for image data is challenging due to the lack of ground truth. Although our LID estimates are higher than those in [49] and [10] , our experiments (Table 9 of Appendix D.4) and the findings in [62] and [59] show that model-free baselines underestimate LID of high-dimensional data, especially images.When moving to more complex image datasets, the MLP backbone fails to generate high-quality samples. Therefore, we replace it with state-of-the-art UNets [52, 68] (see Appendix E.2). Surprisingly, we find that using kneedle with UNets fails to produce sensible LID estimates with FLIPD (see curves in Figure 10 of Appendix E.1). We discuss why this might be the case in Appendix E.1, and from here on we simply set t 0 as a hyperparameter instead of using kneedle. Although avoiding kneedle when using UNets results in increased sensitivity with respect to t 0 , we argue that FLIPD remains a valuable measure of complexity as it produces sensible image rankings and it is highly correlated with with PNG compression length. We took random subsets of 4096 images from each of FMNIST, MNIST, SVHN [46] , and CIFAR10 [35] , and sorted them according to their FLIPD estimates (obtained using UNet backbones). We show the top and bottom 5 images for each dataset in Figure 4b , and include more samples in Appendix E.3. Our visualization shows that higher FLIPD estimates indeed correspond to images with more detail and texture, while lower estimates correspond to less complex ones. Additionally, we show in Appendix E.4 that using only k = 50 Hutchinson samples to approximate the trace term in FLIPD is sufficient for small values of t 0 .Further, we quantitatively assess our estimates by computing Spearman's rank correlation coefficient between different LID estimators and PNG compression size, used as a proxy for complexity in the absence of ground truth. We highlight that although we expect this coefficient to be high, a perfect LID estimator need not achieve a correlation of 1. As shown in Table 2 , FLIPD has a high correlation with PNG, whereas model-free estimators do not. We find that the NB estimator correlates slightly more with PNG on MNIST and CIFAR10, but significantly less in FMNIST and SVHN. Moreover, in Appendix E.5, we analyze how increasing t 0 affects FLIPD by re-computing the correlation with the PNG size at different values of t 0 ∈ (0, 1). We see that as t 0 increases, the correlation with PNG decreases. Despite this decrease, we observe an interesting phenomenon: while the image orderings change, qualitatively, the smallest FLIPD(•, t 0 ) estimates still represent less complex data compared to the highest FLIPD(•, t 0 ), even for relatively large t 0 . We hypothesize that for larger t 0 , similar to the "string within a doughnut" experiment in Figure 3 , the orderings correspond to coarse-grained and semantic notions of complexity rather than fine-grained ones such as textures, concepts that a metric such as PNG compression size cannot capture.We also consider high-resolution images from LAION-Aesthetics [55] and, for the first time, estimate LID for extremely high-dimensional images with D = 3 × 512 × 512 = 786,432. We use Stable Diffusion [51] , a latent DM pretrained on LAION-5B [55] . This includes an encoder and a decoder trained to preserve relevant characteristics of the data manifold in latent representations. Since the encoder and decoder are continuous and effectively invert each other, we argue that the Stable Diffusion encoder can, for practical purposes, be considered a topological embedding of the LAION-5B dataset into its latent space of dimension 4 × 64 × 64 = 16,384. Therefore, the dimension of the LAION-5B submanifold in latent space should be unchanged. We leave an empirical verification of this hypothesis to future work and thus estimate image LIDs by carrying out FLIPD in the latent space of Stable Diffusion. Here, we set the Hutchinson sample count to k = 1, meaning we only require a single Jacobian-vector-product. When we order a random subset of 1600 samples according to their FLIPD at t 0 = 0.3, the more complex images are clustered at the end, while the least complex are clustered at the beginning: see Figure 1 and Figure 4c for the lowest-and highest-LID images from this ordering, and Figure 25 in Appendix E.6 to view the entire subset and other values of t 0 . In comparison to orderings according to PNG compression size (Figure 4c ), FLIPD prioritizes semantic complexity over low-level details like colouration. Finally, we compare the runtimes for computing FLIPD and NB for all models using UNet backbones. We show results in Table 3 . For 28 ×Variance-exploding DMs are such that f (x, t) = 0 with g being non-zero. In this case [58] :EQUATION)Since g is non-zero, g 2 is positive, so that σ 2 is increasing, and thus injective. It follows that σ is also injective, so that λ = σ/ψ = σ is injective. Equation 15 then implies thatEQUATIONA.2 Variance-Preserving Diffusion Models (DDPMs)Variance-preserving DMs are such thatEQUATIONEQUATION)where β is a positive scalar function. In this case [58] : t) , where B(tψ(t) = e -1 2 B(t) , and σ 2 (t) = 1 -e -B(EQUATIONWe then have thatEQUATIONSince β is positive, B is increasing and thus injective, from which it follows that λ is injective as well. Plugging everything into Equation 15, we obtain: t0) tr ∇s e -1 2 B(t0) x, t 0 + s e -1 2 B(t0) x, t 0 2 2 . (22)FLIPD(x, t 0 ) = D + 1 -e -B(Sub-variance-preserving DMs are such that t) , where B(tEQUATIONg(t) = β(t) 1 -e -2B(EQUATIONand where β is a positive scalar function. In this case [58] :EQUATION)We then have thatEQUATION)Since β is positive, B is increasing and thus injective, from which it follows that λ is injective as well due to the injectivity of sinh. Plugging everything into Equation 15, we obtain:EQUATION)B.1 Derivation of Equation 12We have that:EQUATION= N ψ t(δ) x; ψ t(δ) x 0 , σ 2 t(δ) I D p(x 0 , 0)dx 0 (28)EQUATIONwhere we used that N (ax; ax 0 , σ 2 I D ) = 1 a D N (x; x 0 , (σ 2 /a 2 )I D ), along with the definition λ(t) := σ(t)/ψ(t). It is thus easy to see that taking logarithms yields Equation 12. 13First, we recall the Fokker-Planck equation associated with the SDE in Equation 1, which states that:EQUATIONWe begin by using this equation to derive ∂/∂t log p(x, t). Noting that ∇p(x, t) = p(x, t)s(x, t), we have that: tr ∇ 2 p(x,it then follows thatEQUATION)Then, from Equation 12 and the chain rule, we get:EQUATIONEQUATIONEQUATIONEQUATIONSubstituting Equation 36into Equation 40 yields:EQUATIONFrom now on, to simplify notation, when dealing with a scalar function h, we will denote its derivative as h ′ . Since t(δ) = λ -1 (e δ ), the chain rule gives:EQUATION)So far, we have not used that f (x, t) = b(t)x, which implies that ∇ • f (x, t) = Db(t) and that ⟨f (x, t), s(x, t)⟩ = b(t)⟨x, s(x, t)⟩. Using these observations and Equation 42, Equation 41 becomes:EQUATION.If we showed that ψ ′ (t) -b(t)ψ(t) = 0, and thatEQUATIONfor every t, then Equation 43 would simplify to Equation 13. From equation 5.50 in [53] , we have thatEQUATIONwhich shows that indeed ψ ′ (t) -b(t)ψ(t) = 0. Then, from equation 5.51 in [53] , we also have thatEQUATIONand from the chain rule this gives thatEQUATION)We now finish verifying Equation 44. Since λ(t) = σ(t)/ψ(t), the chain rule implies thatEQUATIONwhich, as previously mentioned, shows that Equation 43 simplifies to Equation 13.We begin by stating and proving a lemma which we will later use.Lemma B.1. For any ϵ > 0 and ξ > 0, there exists ∆ < 0 such that for all δ < ∆ and y ∈ R d with ∥y∥ 2 > ξ, it holds that:EQUATION)Proof. The inequality holds if and only ifEQUATIONwhich in turn is equivalent toEQUATION)The limit of the right hand side as δ → -∞ is 0 (and it approaches from the positive side), while ∥y∥ 2 2 is lower bounded by ξ 2 , thus finishing the proof.We now restate Theorem 3.1 for convenience:Theorem 3.1 (FLIPD Soundness: Linear Case). Let L be an embedded submanifold of R D given by a d-dimensional affine subspace. If p(•, 0) is supported on L, continuous, and with finite second moments, then for any x ∈ L with p(x, 0) > 0, we have:EQUATION)Proof. As the result is invariant to rotations and translations, we assume without loss of generality thatL = {(x ′ , 0) ∈ R D | x ′ ∈ R d , 0 ∈ R D-d }. Since x ∈ L, it has the form x = (x ′ , 0) for some x ′ ∈ R d . Note that formally p(•, 0) is not a density with respect to the D-dimensional Lebesgue measure, however, with a slight abuse of notation, we identify it with p(x ′ ), where p(•) is now the d-dimensional Lebesgue density of X ′ , where (X ′ , 0) = X ∼ p(•, 0). We will denote p(•) ⊛ N (•; 0, e 2δ I d ) as p δ (•). In this simplified notation, our assumptions are that p(•) is continuous at x ′ with finite second moments, and that x ′ is such that p(x ′ ) > 0.For ease of notation we use N δ d to represent the normal distribution with variance e 2δ on a ddimensional space, i.e. N δ d (•) = N (•; 0, e 2δ I d ). For any subspace S we use S c to denote its complement where the ambient space is clear from context. B ξ (0) denotes a ball of radius ξ around the origin.We start by noticing that the derivative of the logarithm of a Gaussian with respect to its log variance can be computed as:EQUATION)We then have that:EQUATIONEQUATIONwhere c 0 is a constant that does not depend on δ.EQUATION2 )N δ d (y)dyEQUATIONEQUATION)where we exchanged the order of derivation and integration in Equation 60 using Leibniz integral rule (because the normal distribution, its derivative, and p are continuous; note that p does not depend on δ so regularity on its derivative is not necessary), and where Equation 61 follows from Equation 54. Thus, proving thatEQUATIONwould finish our proof. Now let ϵ > 0. By continuity of p at x ′ , there exists ξ > 0 such that if ∥y∥ 2 < ξ, then |p(x ′ -y) -p(x ′ )| < ϵ. Let ∆ be the corresponding ∆ from Lemma B.1. Assume δ < ∆ and define c 1 and c 2 as follows: EQUATIONEQUATION)Analoguously to c 1 and c 2 , there exists c 3 ∈ [-1, 1] and c 4 ∈ [-d, d] so that the Equation 66 is equal to:EQUATION)We still need to prove that lim δ→-∞ I = d. Taking lim sup and lim inf as δ → -∞ yields:EQUATIONwhere c ′′ i = lim sup δ→-∞ c i and c ′ i = lim inf δ→-∞ c i . Note that although the values of c ′ i and c ′′ i depend on ϵ and ξ, the bounds on them do not. We can thus take the limit of this inequality as ξ and ϵ approach zero:EQUATIONEQUATIONso that I does not depend on ϵ nor on ξ. In turn, this implies thatEQUATIONwhich finishes the proof. Here, we adapt FLIPD for state-of-the-art DDPM architectures and follow the discretized notation from Ho et al. [25] where instead of using a continuous time index t from 0 to 1, a timestep t belongs instead to the sequence {0, . . . , T } with T being the largest timescale. We use the colour gold to indicate the notation used by Ho et al. [25] . We highlight that the content of this section is a summary of the equivalence between DDPMs and the score-based formulation established by Song et al. [58] .As a reminder, DDPMs can be viewed as discretizations of the forward SDE process of a DM, where the process turns into a Markov noising process:EQUATIONWe also use sub-indices t instead of functions evaluated at t to keep consistent with Ho et al. [25] 's notation. This in turn implies the following transition kernel:EQUATIONwhere α t := 1 -β t and ᾱt := t s=1 α t . DDPMs model the backward diffusion process (or denoising process) by modelling a network ϵ : R D × {1, . . . , T } → R D that takes in a noised-out point and outputs a residual that can be used to denoise. Song et al. [58] show that one can draw an equivalence between the network ϵ(•, t) and the score network (see their Appendix B). Here, we rephrase the connections in a more explicit manner where we note that:-ϵ(x, t)/ √ 1 -ᾱt = s(x, t/T ).(74) Consequently, plugging into Equation 22, we get the following formula adapted for DDPMs:EQUATIONwhere t 0 = t 0 × T (best viewed in colour). We include Table 4 , which summarizes all of the equivalent notation when moving from DDPMs to score-based DMs and vice-versa. Throughout all our experiments, we used an NVIDIA A100 GPU with 40GB of memory.To mimic a UNet, we use an MLP architecture with a bottleneck as our score network for our synthetic experiments. This network contains 2 × L + 1 fully connected layers with dimensions ⟨h 1 , h 2 , . . . , h L , . . . h 2L+1 ⟩ forming a bottleneck, i.e., h L has the smallest size. Notably, for 1 ≤ i ≤ L, the ith transform connects layer i -1 (or the input) to layer i with a linear transform of dimensions "h i-1 × h i ", and the (L + i)th layer (or the output) not only contains input from the (L + i -1)th layer but also, contains skip connections from layer (L -i) (or the input), thus forming a linear transform of dimension "(h L+i-1 + h L-i ) × h L+i ". For image experiments, we scale and shift the pixel intensities to be zero-centered with a standard deviation of 1. In addition, we embed times t ∈ (0, 1) using the scheme in [68] and concatenate with the input before passing to the score network. All hyperparameters are summarized in Table 5 .Figure 5 : The FLIPD estimates on a Lollipop dataset from [62] .Figure 5 shows pointwise LID estimates for a "lollipop" distribution taken from [62] . It is a uniform distribution over three submanifolds: (i) a 2d "candy", (ii) a 1d "stick", and (iii) an isolated point of zero dimensions. Note that the FLIPD estimates at t 0 = 0.05 for all three submanifolds are coherent.Figure 6 shows the FLIPD curve as training progresses on the lollipop example and the Gaussian mixture that was already discussed in Section 4. We see that gradually, knee patterns emerge at the correct LID, indicating that the DM is learning the data manifold. Notably, data with higher LID values get assigned higher estimates even after a few epochs, demonstrating that FLIPD effectively ranks data based on LID, even when the DM is underfitted.Finally, Figure 7 presents a summary of complex manifolds obtained from neural spline flows and highdimensional mixtures, showing knees around the true LID. Tempczyk et al. [62] argue that when setting δ, all the directions of data variation that have a log standard deviation below δ are ignored. Here, we make this connection more explicit. We define a multivariate Gaussian distribution with a prespecified eigenspectrum for its covariance matrix: having three eigenvalues of 10 -4 , three eigenvalues of 1, and four eigenvalues of 10 3 . This ensures that the distribution is numerically 7d and that the directions and amount of data variation are controlled using the eigenvectors and eigenvalues of the covariance matrix.For this multivariate Gaussian, the score function in Equation 13can be written in closed form; thus, we evaluate FLIPD both with and without training a DM. We see in Figure 8 the estimates obtained in both scenarios match closely, with some deviations due to imperfect model fit, which we found matches perfectly when training for longer.Apart from the initial knee at 7, which is expected, we find another at t 0 = 0.6 (corresponding to e δ ≈ 6.178 with our hyperparameter setup in Table 5 ) where the value of FLIPD is 4. This indeed confirms that the estimator focuses solely on the 4d space characterized by the eigenvectors having eigenvalues of 10 3 , and by ignoring the eigenvalues 10 -4 and 1 which are both smaller than 6.178.Generating Manifold Mixtures To create synthetic data, we generate each component of our manifold mixture separately and then join them to form a distribution. All mixture components are sampled with equal probability in the final distribution. For a component with the intrinsic dimension of d, we first sample from a base distribution in d dimensions. This base distribution is isotropic Gaussian and Laplace for N d and L d and uniform for the case of U d and F d . We then zero-pad these samples to match the ambient dimension D and perform a random D × D rotation on R D (each component has one such transformation). For F d , we have an additional step to make the submanifold complex. We first initialize a neural spline flow (using the nflows library [19] ) with 5 coupling transforms, 32 hidden layers, 32 hidden blocks, and tail bounds of 10. The data is then passed through the flow, resulting in a complex manifold embedded in R D with an LID of d. Finally, we standardize each mixture component individually and set their modes such that the pairwise Euclidean distance between them is at least 20. We then translate each component so its barycenter matches the corresponding mode, ensuring that data from different components rarely mixes, thus maintaining distinct LIDs.LIDL Baseline Following the hyperparameter setup in [62] , we train 8 models with different noised-out versions of the dataset with standard deviations e δi ∈ {0.01, 0.014, 0.019, 0.027, 0.037, 0.052, 0.072, 0.1}. The normalizing flow backbone is taken from [19] , using 10 piecewise rational quadratic transforms with 32 hidden dimensions, 32 blocks, and a tail bound of 10. While [62] uses an autoregressive flow architecture, we use coupling transforms for increased training efficiency and to match the training time of a single DM.Setup We use model-free estimators from the skdim library [5] and across our experiments, we sample 10 6 points from the synthetic distributions for either fitting generative models or fitting the model-free estimators. We then evaluate LID estimates on a uniformly subsampled set of 2 12 points from the original set of 10 6 points. Some methods are relatively slow, and this allows us to have a fair, yet feasible comparison. We do not see a significant difference even when we double the size of the subsampled set. We focus on four different model-free baselines for our evaluation: ESS [30] , LPCA [22, 12] , MLE [38, 43] , and FIS [1] . We use default settings throughout, except for high-dimensional cases (D > 100), where we set the number of nearest neighbours k to 1000 instead of the default 100. This adjustment is necessary because we observed a significant performance drop, particularly in LPCA, when k was too small in these scenarios. We note that this adjustment has not been previously implemented in similar benchmarks conducted by Tempczyk et al. [62] , despite their claim that their method outperforms model-free alternatives. We also note that FIS does not scale beyond 100 dimensions, and thus leave it blank in our reports. Computing pairwise distances on high dimensions (D ≥ 800) on all 10 6 samples takes over 24 hours even with 40 CPU cores. Therefore, for D ≥ 800, we use the same 2 12 subsamples we use for evaluation.Evaluation We have three tables to summarize our analysis: (i) Table 7 shows the MAE of the LID estimates, comparing each datapoint's estimate to the ground truth at a fine-grained level; (ii) Table 9 shows the average LID estimate for synthetic manifolds with only one component, this average is typically used to estimate global intrinsic dimensionality in baselines; finally, (iii) Table 8 looks at the concordance index [23] of estimates for cases with multiple submanifolds of different dimensionalities. Concordance indices for a sequence of estimates { LID} N n=1 are defined as follows:EQUATION)where I is the indicator function; a perfect estimator will have a C of 1. Instead of emphasizing the actual values of the LID estimates, this metric assesses how well the ranks of an estimator align with those of ground truth [60, 45, 61, 31] , thus evaluating LID as a "relative" measure of complexity.Model-free Analysis Among model-free methods, LPCA and ESS show good performance in low dimensions, with LPCA being exceptionally good in scenarios where the manifold is affine. As we see in Table 7 , while model-free methods produce reliable estimates when D is small, as D increases the estimates become more unreliable. In addition, we include average LID estimates in Table 9 and see that all model-free baselines underestimate intrinsic dimensionality to some degree, with LPCA and MLE being particularly drastic when D > 100. We note that ESS performs relatively well, even beating model-based methods in some 800-dimensional scenarios. However, we note that both the C values in Table 8 and MAE values in Tables 7 and 9 suggest that none of these estimators effectively estimate LID in a pointwise manner and cannot rank data by LID as effectively as FLIPD. Model-based Analysis Focusing on model-based methods, we see in Table 8 that all except FLIPD perform poorly in ranking data based on LID. Remarkably, FLIPD achieves perfect C values among all datasets; further justifying it as a relative measure of complexity. We also note that while LIDL and NB provide better global estimates in Table 9 for high dimensions, they have worse MAE performance in Table 7 . This once again suggests that at a local level, our estimator is superior compared to others, beating all baselines 4 out of 5 groups of synthetic manifolds in Table 7 .We begin by evaluating the impact of using kneedle. Our findings, summarized in Table 6 , indicate that while setting a small fixed t 0 is effective in low dimensions, the advantage of kneedle becomes particularly evident as the number of dimensions increases.We also tried combining it with kneedle by sweeping over the origin δ 1 and arguing that the estimates obtained from this method also exhibit knees. Despite some improvement in high-dimensional settings, Table 6 shows that even coupling it with kneedle does not help.We recall that the NB estimator requires computing rank S(x), where S(x) is a K × D matrix formed by stacking the scores ŝ(•, t 0 ). We set t 0 = 0.01 as it provides the most reasonable estimates.To compute rank S(x) numerically, Stanczuk et al. [59] perform a singular value decomposition on S(x) and use a cutoff threshold τ below which singular values are considered zero. Finding the best τ is challenging, so Stanczuk et al. [59] propose finding the two consecutive singular values with the maximum gap. Furthermore, we see that sometimes the top few singular values are disproportionately higher than the rest, resulting in severe overestimations of the LID. Thus, we introduce an alternative algorithm to determine the optimal τ . For each τ , we estimate LID by thresholding the singular values. Sweeping 100 different τ values from 0 to 1000 at a geometric scale (to further emphasize smaller thresholds) produces estimates ranging from D (keeping all singular values) to 0 (ignoring all). As τ varies, we see that the estimates plateau over a certain range of τ . We use kneedle to detect this plateau because the starting point of a plateau is indeed a knee in the curve. This significantly improves the baseline, especially in high dimensions: see the third column of Tables 7, 9 , and 8 compared to the second column. 1.000 0.426 0.394 0.999 1.000 1.000 1.000 1.000 String withing doughnut ⊆ R 3 0.483 0.486 0.565 1.000 1.000 1.000 1.000 N2 + N4 + N8 ⊆ R 10 1.000 0.341 0.725 0.943 1.000 1.000 1.000 1.000 L2 + L4 + L8 ⊆ R 10 1.000 0.342 0.752 0.884 1.000 1.000 1.000 1.000 U2 + U4 + U8 ⊆ R 10 1.000 0.334 0.462 0.903 1.000 1.000 1.000 1.000 F2 + F4 + F8 ⊆ R 10 1.000 0.342 0.578 0.867 1.000 1.000 0.999 1.000U10 + U25 + U50 ⊆ R 1001.000 0.467 0.879 0.759 0.855 0.897 1.000 -U10 + U30 + U90 ⊆ R 1001.000 0.342 0.826 0.742 0.742 0.855 1.000 -N10 + N25 + N50 ⊆ R 1001.000 0.342 0.866 0.736 0.878 0.917 1.000 -F10 + F25 + F50 ⊆ R 100 1.000 0.342 0.731 0.695 0.847 0.897 1.000 -U10 + U80 + U200 ⊆ R 8001.000 0.342 0.841 0.697 1.000 1.000 1.000 - 10 , the FLIPD curves with MLPs have clearly discernible knees at which the estimated LID averages around a positive number, as expected. When changing the architecture of the DMs from MLPs to UNets, however, the curves either lack knees or obtain them at negative values of LID. This makes it harder to set t 0 automatically with kneedle when using UNets, and thus makes the estimator non-robust to the choice of architecture. Note that our theory ensures that FLIPD(x, t 0 ) converges to LID(x) as t 0 → 0 when ŝ is an accurate estimator of the true score function; indeed, it does not make any claims about the behaviour of FLIPD when t 0 is large or when the model fit is poor. Yet, the fact that the FLIPD estimates obtained from UNets are less reliable than the ones from MLPs is somewhat surprising, especially given that DMs with MLP backbones clearly produce worse-looking samples (shown in Figure 9 ), suggesting that while the model fit is better in the former case, the LID estimates are worse. Throughout the paper, we have argued that, notwithstanding this, FLIPD estimates derived from both architectures offer useful measurements of relative complexity, and can still effectively rank data based on its complexity; refer to all the images in Figures 17, 18, 19, 20, 21, 22, 23, and 24. However, this discrepancy between FLIPD estimates -in absolute terms -when altering the model architecture is counterintuitive and requires further treatment. In what follows, we propose hypotheses for the significant variation in FLIPD estimates when transitioning from MLP to UNet architectures, and we encourage further studies to investigate this issue and to adjust FLIPD to better suit DMs with state-of-the-art backbones.While MLP architectures do not produce visually pleasing samples, comparing the first and second rows of Figure 9 side-by-side, it also becomes clear that MLP-generated images still match some characteristics of the true dataset, suggesting that they may still be capturing the image manifold in useful ways. To explain the unstable FLIPD estimates of UNets, we hypothesize that the convolutional layers in the UNet provide some inductive biases which, while helpful to produce visually pleasing images, might also encourage the network to over-fixate on high-frequency features. More specifically, score functions can be interpreted as providing a denoiser which removes noise added during the diffusion process; our hypothesis is that UNets are particularly good at removing noise along highfrequency directions but not necessarily along low-frequency ones. Inductive biases or modelling errors which make the learned score function ŝ deviate from s might significantly alter the behaviour of FLIPD estimates, even if they result in visually pleasing images; for a more formal treatment of the effect of this deviation, please refer to the work of Tempczyk et al. [63] , who link modelling errors to LID estimation errors in LIDL.The aforementioned inductive biases have been found to produce unintuitive behaviours in other settings as well. For example, Kirichenko et al. [34] showed how normalizing flows over-fixate on these high-frequency features, and how this can cause them to assign large likelihoods to outof-distribution data, even when the model produces visually convincing samples. We hypothesize that a similar underlying phenomenon might be at play here, and that DMs with UNets might be over-emphasizing high-frequency directions of variation in their LID estimates. On the other hand, MLPs do not inherently distinguish between the high-and low-frequency directions and can therefore more reliably estimate the manifold dimension, even though their generated samples appear less appealing to the human eye. However, an exploration of this hypothesis is outside the scope of our work, and we highlight once again that FLIPD remains a useful measure of relative complexity when using either UNets or MLPs.We use UNet architectures from diffusers [68] , mirroring the DM setup in Table 5 except for the score backbone. For greyscale images, we use a convolutional block and two attention-based downsampling blocks with channel sizes of 128, 256, and 256. For colour images, we use two convolutional downsampling blocks (each with 128 channels) followed by two attention downsampling blocks (each with 256 channels). Both setups are mirrored with their upsampling counterparts.Figures 11, 12, 13, and 14 show 4096 samples of CIFAR10, SVHN, MNIST, and FMNIST sorted according to their FLIPD estimate, showing a gradient transition from the least complex datapoints (e.g., the digit 1 in MNIST) to the most complex ones (e.g., the digit 8 in MNIST). We use MLPs for greyscales and UNets for colour images but see similar trends when switching between backbones.E.4 How Many Hutchinson Samples are Needed?Figure 15 compares the Spearman's rank correlation coefficient between FLIPD while we use k ∈ {1, 50} Hutchinson samples vs. computing the trace deterministically with D Jacobian-vectorproducts: (i) Hutchinson sampling is particularly well-suited for UNet backbones, having higher correlations compared to their MLP counterparts; (ii) as t 0 increases, the correlation becomes smaller, suggesting that the Hutchinson sample complexity increases at larger timescales; (iii) for small t 0 , even one Hutchinson sample is enough to estimate LID; (iv) for the UNet backbone, 50 Hutchinson samples are enough and have a high correlation (larger than 0.8) even for t 0 as large as 0.5.While we use 100 nearest neighbours in Table 2 for ESS and LPCA, we tried both with 1000 nearest neighbours and got similar results. Moreover, Figure 16 shows the correlation of FLIPD with PNG for t 0 ∈ (0, 1), indicating a consistently high correlation at small t 0 , and a general decrease while increasing t 0 . Additionally, UNet backbones correlate better with PNG, likely because their convolutional layers capture local pixel differences and high-frequency features, aligning with PNG's internal workings. However, high correlation is still seen when working with an MLP. To test FLIPD with Stable Diffusion v1.5 [51] , which is finetuned on a subset of LAION-Aesthetics, we sampled 1600 images from LAION-Aesthetics-650k and computed FLIPD scores for each.We ran FLIPD with t 0 ∈ {0.01, 0.1, 0.3, 0.8} and a single Hutchinson trace sample. In all cases, FLIPD ranking clearly corresponded to complexity, though we decided upon t 0 = 0.3 as best capturing the "semantic complexity" of image contents. All 1600 images for t = 0.3 are depicted in Figure 25 . For all timesteps, we show previews of the 8 lowest-and highest-LID images in Figure 26 . Note that LAION is essentially a collection of URLs, and some are outdated. For the comparisons in Figure 26 and Figure 4c , we remove placeholder icons or blank images, which likely correspond to images that, at the time of writing this paper, have been removed from their respective URLs and which are generally given among the lowest LIDs. Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: See Appendix D.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Justification: We read the code of ethics and adhered to it.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: We do not believe that improved estimates of local intrinsic dimension can be directly used in any way that would have societal consequences.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Since we do not believe our work can have negative societal consequences, we also do not believe putting safeguards around it to be necessary.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We cite PyTorch [47] as well as the papers whose code we used.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.38th Conference on Neural Information Processing Systems (NeurIPS 2024).Pronounced as "flipped", the acronym is a rearrangement of "FP" from Fokker-Planck and "LID".Throughout this paper, we use the symbol ∇ to denote the differentiation operator with respect to the vector-valued input, not the scalar time t, i.e. ∇ = ∂/∂x.Note that we treat positive and negative superindices differently: e.g. σ -1 denotes the inverse function of σ, not 1/σ; on the other hand σ 2 denotes the square of σ, not σ • σ.greyscale (MNIST/FMNIST) and 3 × 32 × 32 low-resolution RGB (SVHN/CIFAR10) images, we use k = 50 Hutchinson samples: at these dimensions, FLIPD achieves ∼ 10× and ∼ 100× respective speedups over NB. For 4 × 64 × 64 LAION-Aesthetics images on the latent space of Stable Diffusion, we use k = 1 for FLIPD, which ensures it remains highly tractable. At this resolution NB is completely intractable: constructing S(x) for a single image x takes 2.5 hours, and computing NB would then still require performing a SVD on this 16, 384 × 65, 536 matrix.5 Conclusions, Limitations, and Future WorkIn this work we have shown that the Fokker-Planck equation can be utilized for efficient LID estimation with any pre-trained DM. We have provided strong theoretical foundations and extensive benchmarks showing that FLIPD estimates accurately reflect data complexity. Although FLIPD produces excellent LID estimates on synthetic benchmarks, its instability with respect to the choice of network architecture on images is surprising, and results in LID estimates which strongly depend on this choice. We see this behaviour as a limitation, even if FLIPD nonetheless still provides a meaningful measure of complexity regardless of architecture. Given that FLIPD is tractable, differentiable, and compatible with any DM, we hope that it will find uses in applications where LID estimates have already proven helpful, including OOD detection, AI-generated data analysis, and adversarial example detection. 
 
 Appendix:  


Decision: This paper proposes a Fokker-Planck-based estimator for local intrinsic dimension estimation, which uses diffusion models in implementation and scales to high-resolution images at the level of Stable Diffusion. Both the theoretical and empirical contributions are appraised by the reviewers.

The proposed method is indeed novel and demonstrates promising performance compared with existing baselines. The paper is clearly written in general and I invite the authors to revise according to the discussion with the reviewers. (Special thanks to Reviewer MJca and the authors for the very dedicated efforts in revision and I trust the authors to make necessary polishing in the final version.) An outstanding concern about the computation efficiency in the initial review is further addressed by the authors' general rebuttal. Therefore, I am recommending acceptance of the paper.

Reviews: review: paper_summary: paper_summary: The paper addresses the challenge of estimating the local intrinsic dimension (LID) of high-dimensional data, a measure reflecting the number of local factors of variation and data complexity. Traditional methods for LID estimation have limitations such as inaccuracy, high computational demand, and dependency on multiple pre-trained models. This work introduces FLIPD, an LID estimator based on the Fokker-Planck equation associated with a single pre-trained diffusion model. FLIPD outperforms existing baselines in LID estimation benchmarks and remains tractable even for high-resolution images. The authors adapt the LIDL method to DMs, leveraging the Fokker-Planck equation to enhance efficiency and accuracy. FLIPD's performance is validated through theoretical motivations and empirical results, demonstrating its superior correlation with non-LID complexity measures and qualitative complexity assessments compared to other estimators.
strength_weakness: strength_weakness: **1.** The most critical section of this paper is Section 3.2, which introduces the core innovation of the FLIPD estimator. However, the insights from Equations (12) and (13) are not immediately clear. It is difficult to understand the insights of these derivations. Moreover, many details are deferred to appendix. The authors should provide more detailed explanations of the Fokker-Planck equation and its relevance to LID estimation. 

**2.** One weakness of the paper is the unclear demonstration of the computational advantages of the proposed FLIPD estimator compared to the method in [58]. The authors argue that while the method in [58] addresses some limitations of LIDL, it remains computationally expensive. However, the paper lacks a thorough theoretical analysis or experimental evidence to substantiate the claim that FLIPD is more computationally efficient.
questions: questions: **Q1.** It seems that the proposed estimator over-estimate the LID compared to [10,48]. Can you provide some insights? 

**Q2.** How does the FLIPD estimator maintain accuracy and efficiency in LID estimation across varying levels of data complexity and different types of high-dimensional datasets, and what specific mechanisms within the Fokker-Planck framework contribute to these properties?
limitations: limitations: Not applicable
paper_summary: paper_summary: The authors propose to employ the best available generative models, i.e., diffusion models, to the estimation of local intrinsic dimension. To this end, they build upon the LIDL estimator of Tempczyk et al. [ICML, 2022], but crucially resolve a number of limitations: (1) direct application of LIDL requires training $m$ diffusion models, (2) even by enabling only a single diffusion model, LIDL still requires $m$ ODE solves, and finally (3) even by enabling a single ODE, it requires repeated estimation of the trace of a Jacobian. The result is a set of explicit formulas for the LID estimate for each of the common diffusion models, leveraging the Fokker-Planck equation associated with the Ito stochastic differential equation. The asymptotic correctness of the estimate is established for an idealized linear setting. This algorithmic contribution is further demonstrated through an enhanced intrinsic dimension benchmark, which highlights the limitations of prior techniques.
strength_weakness: strength_weakness: Unfortunately, there seem to be a number of technical issues; see below. It doesn't help that some of those issues appear out of the blue upon reading into the experiments section, and don't match the impression one gets from the high level summary given in the abstract and introduction.

Disregarding those issues for a moment, it's still unsettling that UNets deviate from MLPs in terms of their FLIPD curves lacking easily detectable knees. This calls into question whether knees are the best or only way to select values of $t$, or whether there are other techniques.
questions: questions: Technical Issues
=============
Section 3
- L211: In my experience, such constant factors relate to the curvature of the manifold. So, the result makes sense when the manifold is an affine subspace. In this sense, the reasoning given on L219-222 appears erroneous. That is, concentration on a point mass in the limit still leaves the intrinsic dimension intact, rather than totally detaching from the manifold structure to yield an intrinsic dimension of 0.

Section 4
- L309: It seems that this is the first mention of an MLP, which this line suggests is what was used for the synthetic experiments.
  - Unfortunately, this seems like more than a simple lapse in communication.
  - Rather, upon switching from those previously undisclosed MLPs to UNets, those nice findings about knees in the curve no longer hold.
  - This critical transition comes out of the blue, since it wasn't mentioned either in the abstract or the introduction.
  - From a first reading, it was difficult to keep track of which findings reported for MLPs carry over upon switching to UNets, where the authors claim the resulting estimates remain valuable measures of complexity; see L323.
- FLIPD on high resolution images
  - L359: is it correct that running FLIPD in the latent space inherits existing biases, e.g., prioritizing semantics rather than "true" manifold geometry?
  - It's unfortunate that the actual results as conveyed on L345-360 seem to be misrepresented in the high level summaries given in the abstract and introduction.

Other Technical Comments
=====================
Section 3
- L190: I'm not sure I follow how $\nu$ does not depend on $\hat{\varrho}$. Perhaps what's meant is that $\nu$ is computed from Eq. 12, and the resulting values are fixed to define $\hat{\varrho}$ through the ODE in Eq. 13. That way, if $\nu$ was computed with $\hat{s} = s$, then Eq. 13 can be expressed in terms of $\log \varrho(x, \delta)$ itself.

Section 4
- L284: Does the diffeomorphism of the flow hold for practical numerical evaluations? Did the authors observe any fluctuations in the LID due to such numerical errors? If so, can those errors be bounded?
limitations: limitations: I'm guessing the authors wanted to go all the way, even when UNets failed to deliver the same nice story afforded by MLPs. It's unfortunate that the story didn't go that way - or perhaps it's an opportunity? Either way, the current presentation of this story may initially appear to misrepresent the actual findings. In my view, this is a serious issue that must be addressed before publication.

All that aside, the paper does achieve a lot. Only that the story had to be a bit more complex / interesting.
paper_summary: paper_summary: This paper proposes a novel method for estimating local intrinsic dimension using an existing approximate likelihood estimator with diffusion models. The proposed approach is able to exploit a pre-train diffusion model to evaluate the log density function of the noisy version of the data (i.e., adding Gaussian noise), where solving the ordinary differential equation (ODE) can be avoided by setting $t_0$ as a hyperparameter. Compared to prior works, in which normalizing flow is used for function evaluation and multiple runs of density evaluations are needed for estimation, this framework is much more efficient and shows more promising results.
strength_weakness: strength_weakness: If the claim is that the proposed estimation method is simpler and more computationally efficient, it might be useful to include complexity and/or time comparison with existing baselines.
questions: questions: 1.	What are the causes for the spikes in Fig. 2(b) and Fig. 6(c)?
2.	There seems not much difference between different $t_0$’s in Fig. 26. Does it imply that the estimator is not sensitive to the hyperparameter under the case with data of large dimension?
3.	In line 130, what are the ‘mild regularity conditions’ leading to Eq. (7)?
limitations: limitations: Limitations have been discussed.
paper_summary: paper_summary: This paper is concerned with estimating the local intrinsic dimension (LID) of a given data manifold. Intuitively, high intrinsic dimension indicates more complex data distribution and an accurate representation of the intrinsic dimension has useful applications in ML such as detecting outliers or adversarial examples. 

The approach in the current work suggests incorporating diffusion models as a model-based estimator since these models learn the distribution of the data and implicity the intrinsic dimension. At a high-level the idea is the following: the noise added during diffusion can be seen as a form of Gaussian convolution applied to the data. By analyzing how the data density changes with increasing noise levels, the model is able to implicitly learn the local intrinsic dimension at different points on the manifold.

Furthermore, they uncover and leverage some computational efficiencies using the Fokker-Planck equation which boil down to making a single call to an ODE solver. This computation efficiency also allows for LID estimation for high resolution images, and is the first to be able to do so.

The paper has strong theoretical background and justifications and empirical results that verify their theory. They also provide code with examples to reproduce results. Overall a really excellent and well-executed effort.
strength_weakness: strength_weakness: I have a few minor comments and questions below but overall I don't see any substantial weaknesses of the work that are worth addressing further.
questions: questions: line 107 (suggestion): see also "A Course in Metric Geometry" by Burago, Burago, Ivanov for related discussion about intrinsic dimension and how to compute it for metric spaces. 

(nit) Figure 1: For the sake of intuition, this Figure makes sense. However while it is useful to justify that lower/higher FLIPD scores indicate less/more complex images, it leaves open the question of if less/more complex images necessarily have lower/higher FLIPD scores. For example, who's to say that there aren't some images that look even more complex that the 4 you surface which have the max FLIPD values. It's just a personal note and makes figures like this feel misleading without noting that important distinction. That being said, once getting through the Appendix more comprehensive comparison between entire datasets and FLIPD scores are included. Perhaps just point to those additional plots in the appendix when discussing the Figure 1 as well.
limitations: limitations: N/A

Comments: title: Thank you
Discussion
Making progress
Updated phrasing
Remaining gaps and concerns over presentation & claims
Thank you
Follow up and thanks
Follow up
comment: Thanks for your continued engagement and patience with my detailed inquisitions.

Now that we're on the same page, we can actually think about the remaining questions regarding (2) and (4). I'm noting the following, which I'm only sharing as suggestions:
- (re: 4) L49 and the abstract: the word "address" may give the impression that the noted shortcomings are resolved, which is not the same as pushing SOTA with remaining shortcomings still.
- (re: 2) L60-61: this is a good place to discuss how the theory holds in the limit. I gather this is a common situation for algorithms based on DEs, and can be seen in how diffusion models need to sample time with carefully tuned schedules as well as e.g. Varadhan’s formula for the heat equation. A brief survey of techniques to cope with such difficulties would be appreciated, especially in light of the known numerical instabilities, and the aim to require a single ODE solve.

For the authors' reference:
- Yes, the two remarks aren't necessarily contradictory but the first one can be taken to over promise while only the second acknowledges the intricacies of practical implementations.
- Yes, that characterization was a bit blunt. Doing my best to share feedback in time before the rebuttal window closes.
Thank you once again for the additional feedback.

We will incorporate points (1) and (3) from your TLDR into our final version of the paper. As for point (2), we are not sure how you want us to surface these issues: the discussion about $t_0$ is very technical and we believe that it only makes sense to discuss this once the method has been fully presented. We are nonetheless happy to surface this much more clearly throughout the experiments, and also note that in the updated version of the introduction from our previous post, we do discuss a lack of stability of the FLIPD estimates when using UNets, which we believe gets to the point you are concerned about. Also, as for point (4), we will rephrase, but we do highlight that calling FLIPD more accurate than other LID estimators is perfectly valid, see e.g. Table 1. We will also clarify the relationship between our theory, knees, and architectures when updating the manuscript.

`(Note that those last two remarks seem to contradict each other)`

We do not believe these remarks contradict each other: the theory holds when the true score function $s$ is used, but FLIPD uses a learned score function $\hat{s}$ given by a neural network. If two different architectures managed to recover the true score, they would produce the exact same FLIPD estimates. In practice different architectures do not recover the exact same function, and each has different inductive biases. Thus, some deviation from the theory (which assumes access to the exact score function) is not contradictory, and neither is it to observe different behaviours from different architectures. We will also elaborate on this point for added clarity in the camera-ready version.

`whereas the experiments section mainly focuses on validating the estimates to demonstrate the theory despite the accommodations needed to make it work.`

We respectfully disagree with this characterization of the experiments section, which we believe achieves more than just validating theory. The section has two main goals. First, to validate our estimator on synthetic data where we have access to ground truth LID values; this is indeed about validating theory and about establishing that FLIPD outperforms competing estimators. Second, on images, we aim to show that FLIPD values, regardless of how accurate they are as LID estimates (because of the lack of knees), remain meaningful measures of relative image complexity and produce useful rankings of images based on complexity; this part is not about validating theory, but about showing the practical usefulness of FLIPD as a measure of complexity.
Thanks for the updates. I appreciate the authors reflecting on the core claims in light of the discussion. To be clear, it is not my intention to diminish the contributions, but to present exactly what has been achieved and what's left.

TL;DR: please (1) tune down the **Momentum** adequately positioning the theoretical contributions separately from their practical implementations, (2) Surface and discuss **Issue#1** and **Issue#2** early on in the introduction, all through the experiments section. (3) Consider reducing focus on knees a little bit, and instead paying more attention to how to best use the estimates across multiple settings of the hyperparameters as commonly done in robust estimation, (4) Revisit the wording in "addresses all the aforementioned deficiencies" in the abstract, noting remaining gaps applying the estimators in the absence of knees.

Taking a step back after yet another reading, I think I can identify the main underlying gap here. Please feel free to clarify any of those points:
- The paper builds momentum based off the theoretical derivations [**Momentum**] culminating in the proposed estimator as encoded in Theorem 3.1 (Equation 15). This is a great theoretical result with lots of merit towards publication.
- However, applying this neat result in practice immediately hits a couple issues as recounted across a number of pages:
  - [**Issue#1**] L230-232: The effect of $t_0$ FLIPD requires setting $t_0$ close to 0 (since all the theory holds in the $\delta  \to -\infty$ regime). It is important to note that DMs fitted to low-dimensional manifolds are known to exhibit numerically unstable scores $s(·, t_0)$ as $t_0 \to 0$.
  - [**Issue#2**] L319-320: As long as the DM accurately fits the data manifold, our theory should hold, regardless of the choice of backbone. Yet, we see that UNets do not produce a clear knee in the FLIPD curves
  - [**Issue#2**] L792-795:  To explain the surprisingly poor FLIPD estimates of UNets, we hypothesize that the convolutional layers in the UNet provide some inductive biases which, while helpful to produce visually pleasing images, might also encourage the network to over-fixate on high-frequency features which are not visually perceptible.
  - (Note that those last two remarks seem to contradict each other)
- To cope with the challenges of applying the theory in practice, the authors adopted a number of accommodations
  - First, the notion of knees, which seem to be partly motivated by related work on normalizing flows
    - L269-270: As mentioned, we persistently see knees in FLIPD curves. Not only is this in line with the observations of LIDL on normalizing flows
  - From there, attention shifts heavily to knees overshadowing **Issue#1**].
    - I really did not appreciate conflating the observation of knees with possible implications of Theorem 3.1 (L787-788), which follows the general sense of overshooting the **Momentum**.
    - The multiscale interpretation was appreciated though (L249)
  - Upon transitioning to UNets, the focus is still on knees and their sudden disappearance, which further dilutes **Issue#1** which is now mixed in non-trivial ways with **Issue#2**.
  - There's clearly more work to be done on how to best utilize the resulting estimates for various hyperparameters, whereas the experiments section mainly focuses on validating the estimates to demonstrate the theory despite the accommodations needed to make it work.

Looking at the revised summaries, I note the following elements of the abstract:
- [..] current methods based on generative models *produce inaccurate estimates*, require more than a single pre-trained model, are computationally intensive, or do not exploit the best available deep generative models, i.e. diffusion models (DMs)
  -  In this work, we show that the Fokker-Planck equation associated with a DM can provide a LID estimator which *addresses all the aforementioned deficiencies*
  - I don't think the work addresses all the aforementioned deficiencies. There are still major questions regarding the accuracy of the resulting estimates in practice.
- Other changes were more favorable
  - (before) outperforms existing baselines on LID estimation benchmarks
    - (after) outperforms existing baselines on synthetic LID estimation benchmarks while staying robust to the choice of hyperparameter <<<
  - (after) Despite the resulting LID estimates being less stable over the choice of hyperparameter, FLIPD remains a valid measure of relative image complexity <<<
  - (before) FLIPD exhibits higher correlation with non-LID measures of complexity
    - (after) FLIPD exhibits a consistently higher correlation [..]
  - (before) remain tractable with high-resolution images at the scale of Stable Diffusion
    - (after) first one to be tractable at the scale of Stable Diffusion
Thank you again for the continued engagement. Below we include revised versions of the abstract and of the last paragraph of the introduction. If you still have lingering concerns about phrasing please let us know and we will make further updates.

## Abstract

High-dimensional data commonly lies on low-dimensional submanifolds, and estimating the local intrinsic dimension (LID) of a datum -- i.e. the dimension of the submanifold it belongs to -- is a longstanding problem. LID can be understood as the number of local factors of variation: the more factors of variation a datum has, the more complex it tends to be. Estimating this quantity has proven useful in contexts ranging from generalization in neural networks to detection of out-of-distribution data, adversarial examples, and AI-generated text. The recent successes of deep generative models present an opportunity to leverage them for LID estimation, but current methods based on generative models produce inaccurate estimates, require more than a single pre-trained model, are computationally intensive, or do not exploit the best available deep generative models, i.e. diffusion models (DMs). In this work, we show that the Fokker-Planck equation associated with a DM can provide a LID estimator which addresses all the aforementioned deficiencies. Our estimator, called FLIPD, is compatible with all popular DMs, and relies on a single hyperparameter. FLIPD outperforms existing baselines on synthetic LID estimation benchmarks while staying robust to the choice of hyperparameter. We also apply FLIPD on natural images where the true LID is unknown. Despite the resulting LID estimates being less stable over the choice of hyperparameter, FLIPD remains a valid measure of relative image complexity; compared to competing estimators, FLIPD exhibits a consistently higher correlation with non-LID measures of image complexity, and better aligns with qualitative assessments of complexity. Moreover, FLIPD is orders of magnitude faster than other LID estimators, and the first one to be tractable at the scale of Stable Diffusion.

## Introduction

Our contributions are: $(i)$ showing how DMs can be efficiently combined with LIDL in a way which requires a single call to an ODE solver; $(ii)$ leveraging the Fokker-Planck equation to propose FLIPD, thus improving upon the estimator and circumventing the need for an ODE solver altogether; $(iii)$ motivating FLIPD theoretically; $(iv)$ introducing an expanded suite of LID estimation benchmark tasks that reveals gaps in prior evaluations, especially that other estimators do not remain accurate as the complexity of the manifold increases; $(v)$ demonstrating that when using fully-connected architectures for diffusion models, FLIPD produces estimates which are robust with respect to the choice of its hyperparameter and outperforms existing baselines -- especially as dimension increases -- while being much more computationally efficient; $(vi)$ showing that when applied to natural images along with UNet architectures, despite not exhibiting the same hyperparameter robustness, FLIPD estimates consistently align with other measures of complexity such as PNG compression length, and with qualitative assessments of complexity, highlighting that the LID estimates provided by FLIPD remain valid measures of relative image complexity;  and $(vii)$ demonstrating that when applied to the latent space of Stable Diffusion, FLIPD can estimate LID for extremely high-resolution images ($\sim 10^6$ pixel dimensions) for the first time.
TL;DR please provide (1) the updated abstract, and (2) the contribution summary as typically included near the end of the introduction section.

Will follow up with more discussion.
Thank you for your response. These results are interesting. I don't have any additional concerns. I'll maintain my score
Thank you! We are happy we were able to address your concerns.
Thank you for your reply and for raising your score! Of course it is not possible to make manuscript modifications in the rebuttal phase for NeurIPS this year, but we commit to making the requested updates for the final version of the paper: We will clearly distinguish between the experiments involving MLPs and those involving UNets, highlight in the experiments section that some empirical findings observed in MLPs (such as knees) do not fully extend to UNets. Additionally, we will clarify that identifying knee points is not essential in the theoretical analysis. Despite these modifications, we will highlight the results obtained from UNet experiments still correspond to meaningful measures of image complexity. Furthermore, we will make our conjecture in Section 3.3 more explicit by adding extra context from the rebuttal.


As for further modifications:


## Please clarify explicitly which parts of the theory predict those clearly-discernible knees. (L787-788)

We will clarify phrasing here: the theory does not formally suggest that a knee should occur, it simply predicts the limit as $t \to 0$ being the LID.  Therefore, it is reasonable to expect the FLIPD curve to stabilize around a certain value (i.e., the LID) as $t$ approaches zero. The appearance of knees in the MLP experiments is an empirical observation on our part. In other words, while the theory suggests that observing knees is plausible, their absence in the UNet experiments is also consistent with the theoretical framework.

## Surprisingly poor FLIPD estimates of UNets (L792-793)

We will also clarify phrasing here. Our intention here was to point out that even though the lack of knees observed when using UNets makes it harder to select $t_0$ to recover the true LID, the relative rankings obtained by FLIPD remain meaningful. This is evidenced qualitatively when looking at images sorted by their FLIPD values in Figures 17, 19, 21, 23, and 25, and quantitatively by measuring the correlation against PNG compression length in Table 2. This is why we claim that FLIPD remains a valid measure of relative complexity when using UNets, despite the lack of knees.
Thank you for adding the complexity comparison and the response. I think that would make the work more comprehensive and solid.
Thanks for the reviewers' rebuttal. My main concerns have been addressed. I have raised my score.
Thanks for addressing my comments. I revised the language for my confusion about what carries over from MLPs to UNets. Since this was my main reservation, I have increased my score. It would have been nice to see the revised draft, but I trust the authors to reflect discussions with reviewers in the final version.


Follow up technical remarks
=====================

**L211** - Regarding curvature terms, I didn't find any mention of those in the paper. It would be nice to briefly discuss that highlighting how they show up in the formulation of LIDL. And thanks for resolving the simplistic counter-argument I posed, though it was probably triggered by the comment on approaching point-mass. Speaking of the tangent space seems more helpful.

**L309** - I encourage the authors to reflect on how to best smooth out the abandonment of knees upon switching to UNets. (I can see that knee analysis helped demonstrate how the MLP approach captures the true ID for the synthetic dataset, and better aligns with the theory - see also next block.) More concretely, it may help to break up the experiment section, e.g., into two subsections, in order to allow for a more proper anticipation of the added complications of image experiments. That is, those complications appear to be more than just nuance with the specific experiments and their setup, but rather stem from remaining gaps in the development.

Beyond revising the experiments section, I would also recommend tuning down the claimed contributions, as early as the abstract/introduction/summary, to be based more heavily on MLPs and the synthetic evaluations against true IDs, while suitably positioning contributions related to image datasets and the adaptations needed for them as more of an empirical contribution or an extension of the core contributions. I trust the authors to make the right call there, and to further identify and discuss the remaining gaps in the limitations section.

**L787-788**: FLIPD curves with MLPs, which have clearly-discernible knees as predicted by the theory
- Please clarify explicitly which parts of the theory predict those clearly-discernible knees.

**L792-793**: surprisingly poor FLIPD estimates of UNets
- It would help to reflect on this language in light of the authors' assertion*. It would help to tune down the more subjective language and refer to specific qualitative/quantitative empirical results.
(*) From the rebuttal: *we firmly believe that even without the knees, the FLIPD estimators with UNets are useful measures of complexity, as evidenced by the high correlation with PNG compression size shown in Table 2*

**Overall recommendation:**
Given the remaining gaps in analysis - theoretical (i.e. non-linear manifolds) and empirical (i.e. MLP to UNets), I strongly recommend to tune down further hypotheses and conjectures, and instead utilize the space to better present the known facts and available evidence.