Title: A Geometric View of Data Complexity: Efficient Local Intrinsic Dimension Estimation with Diffusion Models
Venue: NeurIPS 2024
Accepted: yes
Keywords: diffusion models

deep generative modelling

manifold hypothesis

intrinsic dimension
Abstract: High-dimensional data commonly lies on low-dimensional submanifolds, and estimating the local intrinsic dimension (LID) of a datum -- i.e. the dimension of the submanifold it belongs to -- is a longstanding problem. LID can be understood as the number of local factors of variation: the more factors of variation a datum has, the more complex it tends to be. Estimating this quantity has proven useful in contexts ranging from generalization in neural networks to detection of out-of-distribution data, adversarial examples, and AI-generated text. The recent successes of deep generative models present an opportunity to leverage them for LID estimation, but current methods based on generative models produce inaccurate estimates, require more than a single pre-trained model, are computationally intensive, or do not exploit the best available deep generative models: diffusion models (DMs). In this work, we show that the Fokker-Planck equation associated with a DM can provide an LID estimator which addresses the aforementioned deficiencies. Our estimator, called FLIPD, is easy to implement and compatible with all popular DMs. Applying FLIPD to synthetic LID estimation benchmarks, we find that DMs implemented as fully-connected networks are highly effective LID estimators that outperform existing baselines. We also apply FLIPD to natural images where the true LID is unknown. Despite being sensitive to the choice of network architecture, FLIPD estimates remain a useful measure of relative complexity; compared to competing estimators, FLIPD exhibits a consistently higher correlation with image PNG compression rate and better aligns with qualitative assessments of complexity. Notably, FLIPD is orders of magnitude faster than other LID estimators, and the first to be tractable at the scale of Stable Diffusion.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is concerned with estimating the local intrinsic dimension (LID) of a given data manifold. Intuitively, high intrinsic dimension indicates more complex data distribution and an accurate representation of the intrinsic dimension has useful applications in ML such as detecting outliers or adversarial examples. \n\nThe approach in the current work suggests incorporating diffusion models as a model-based estimator since these models learn the distribution of the data and implicity the intrinsic dimension. At a high-level the idea is the following: the noise added during diffusion can be seen as a form of Gaussian convolution applied to the data. By analyzing how the data density changes with increasing noise levels, the model is able to implicitly learn the local intrinsic dimension at different points on the manifold.\n\nFurthermore, they uncover and leverage some computational efficiencies using the Fokker-Planck equation which boil down to making a single call to an ODE solver. This computation efficiency also allows for LID estimation for high resolution images, and is the first to be able to do so.\n\nThe paper has strong theoretical background and justifications and empirical results that verify their theory. They also provide code with examples to reproduce results. Overall a really excellent and well-executed effort.\n\nSTRENGTHS:\nThe ideas behind this paper are very well motivated and compelling. The exposition is very clear and the use of diffusion models to improve LIDL feels like a natural next step. \nAlso, with respect to the previous work of Stanczuk et al, the improvements to decrease the computational complexity is noteworthy.\n\nOverall the paper is well-written and the theoretical background and proofs are carefully written and appear complete and correct to me. There are extensive experiments including both synthetic and real-world datasets.\n\nWEAKNESSES:\nI have a few minor comments and questions below but overall I don't see any substantial weaknesses of the work that are worth addressing further.\n\nQUESTIONS:\nline 107 (suggestion): see also \"A Course in Metric Geometry\" by Burago, Burago, Ivanov for related discussion about intrinsic dimension and how to compute it for metric spaces. \n\n(nit) Figure 1: For the sake of intuition, this Figure makes sense. However while it is useful to justify that lower/higher FLIPD scores indicate less/more complex images, it leaves open the question of if less/more complex images necessarily have lower/higher FLIPD scores. For example, who's to say that there aren't some images that look even more complex that the 4 you surface which have the max FLIPD values. It's just a personal note and makes figures like this feel misleading without noting that important distinction. That being said, once getting through the Appendix more comprehensive comparison between entire datasets and FLIPD scores are included. Perhaps just point to those additional plots in the appendix when discussing the Figure 1 as well.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

comments: SUMMARY:
This paper proposes a novel method for estimating local intrinsic dimension using an existing approximate likelihood estimator with diffusion models. The proposed approach is able to exploit a pre-train diffusion model to evaluate the log density function of the noisy version of the data (i.e., adding Gaussian noise), where solving the ordinary differential equation (ODE) can be avoided by setting $t_0$ as a hyperparameter. Compared to prior works, in which normalizing flow is used for function evaluation and multiple runs of density evaluations are needed for estimation, this framework is much more efficient and shows more promising results.

STRENGTHS:
This is a technically sound paper with some theoretical depth and practical usefulness. Although the core idea of the estimation in Eq. (8) is not new, the authors come up with a novel approach to estimate the derivative of the log density function with variance-preserving diffusion models. For real data experiments, using PNG compression as a proxy for evaluation is informative and illustrative. The overall paper is well organized and relatively clearly written.

WEAKNESSES:
If the claim is that the proposed estimation method is simpler and more computationally efficient, it might be useful to include complexity and/or time comparison with existing baselines.

QUESTIONS:
1.	What are the causes for the spikes in Fig. 2(b) and Fig. 6(c)?
2.	There seems not much difference between different $t_0$’s in Fig. 26. Does it imply that the estimator is not sensitive to the hyperparameter under the case with data of large dimension?
3.	In line 130, what are the ‘mild regularity conditions’ leading to Eq. (7)?

LIMITATIONS:
Limitations have been discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose to employ the best available generative models, i.e., diffusion models, to the estimation of local intrinsic dimension. To this end, they build upon the LIDL estimator of Tempczyk et al. [ICML, 2022], but crucially resolve a number of limitations: (1) direct application of LIDL requires training $m$ diffusion models, (2) even by enabling only a single diffusion model, LIDL still requires $m$ ODE solves, and finally (3) even by enabling a single ODE, it requires repeated estimation of the trace of a Jacobian. The result is a set of explicit formulas for the LID estimate for each of the common diffusion models, leveraging the Fokker-Planck equation associated with the Ito stochastic differential equation. The asymptotic correctness of the estimate is established for an idealized linear setting. This algorithmic contribution is further demonstrated through an enhanced intrinsic dimension benchmark, which highlights the limitations of prior techniques.\n\nSTRENGTHS:\n- Greatly simplifies LID estimation via diffusion models, presenting a significant speed up over prior work.\n- The technique is well-principled, and generalizes to different diffusion model families, enabling explicit formulas to be derived as demonstrated by the authors.\n- The proposed benchmark (with an in-depth analysis) helps to further test against prior work, better demonstrating their shortcomings.\n- Relating LID to PNG compression size is also a valuable idea.\n\nWEAKNESSES:\nUnfortunately, there seem to be a number of technical issues; see below. It doesn't help that some of those issues appear out of the blue upon reading into the experiments section, and don't match the impression one gets from the high level summary given in the abstract and introduction.\n\nDisregarding those issues for a moment, it's still unsettling that UNets deviate from MLPs in terms of their FLIPD curves lacking easily detectable knees. This calls into question whether knees are the best or only way to select values of $t$, or whether there are other techniques.\n\nQUESTIONS:\nTechnical Issues\n=============\nSection 3\n- L211: In my experience, such constant factors relate to the curvature of the manifold. So, the result makes sense when the manifold is an affine subspace. In this sense, the reasoning given on L219-222 appears erroneous. That is, concentration on a point mass in the limit still leaves the intrinsic dimension intact, rather than totally detaching from the manifold structure to yield an intrinsic dimension of 0.\n\nSection 4\n- L309: It seems that this is the first mention of an MLP, which this line suggests is what was used for the synthetic experiments.\n  - Unfortunately, this seems like more than a simple lapse in communication.\n  - Rather, upon switching from those previously undisclosed MLPs to UNets, those nice findings about knees in the curve no longer hold.\n  - This critical transition comes out of the blue, since it wasn't mentioned either in the abstract or the introduction.\n  - From a first reading, it was difficult to keep track of which findings reported for MLPs carry over upon switching to UNets, where the authors claim the resulting estimates remain valuable measures of complexity; see L323.\n- FLIPD on high resolution images\n  - L359: is it correct that running FLIPD in the latent space inherits existing biases, e.g., prioritizing semantics rather than \"true\" manifold geometry?\n  - It's unfortunate that the actual results as conveyed on L345-360 seem to be misrepresented in the high level summaries given in the abstract and introduction.\n\nOther Technical Comments\n=====================\nSection 3\n- L190: I'm not sure I follow how $\\nu$ does not depend on $\\hat{\\varrho}$. Perhaps what's meant is that $\\nu$ is computed from Eq. 12, and the resulting values are fixed to define $\\hat{\\varrho}$ through the ODE in Eq. 13. That way, if $\\nu$ was computed with $\\hat{s} = s$, then Eq. 13 can be expressed in terms of $\\log \\varrho(x, \\delta)$ itself.\n\nSection 4\n- L284: Does the diffeomorphism of the flow hold for practical numerical evaluations? Did the authors observe any fluctuations in the LID due to such numerical errors? If so, can those errors be bounded?\n\nLIMITATIONS:\nI'm guessing the authors wanted to go all the way, even when UNets failed to deliver the same nice story afforded by MLPs. It's unfortunate that the story didn't go that way - or perhaps it's an opportunity? Either way, the current presentation of this story may initially appear to misrepresent the actual findings. In my view, this is a serious issue that must be addressed before publication.\n\nAll that aside, the paper does achieve a lot. Only that the story had to be a bit more complex / interesting.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper addresses the challenge of estimating the local intrinsic dimension (LID) of high-dimensional data, a measure reflecting the number of local factors of variation and data complexity. Traditional methods for LID estimation have limitations such as inaccuracy, high computational demand, and dependency on multiple pre-trained models. This work introduces FLIPD, an LID estimator based on the Fokker-Planck equation associated with a single pre-trained diffusion model. FLIPD outperforms existing baselines in LID estimation benchmarks and remains tractable even for high-resolution images. The authors adapt the LIDL method to DMs, leveraging the Fokker-Planck equation to enhance efficiency and accuracy. FLIPD's performance is validated through theoretical motivations and empirical results, demonstrating its superior correlation with non-LID complexity measures and qualitative complexity assessments compared to other estimators.\n\nSTRENGTHS:\n**1.** This paper introduces a new approach to estimating local intrinsic dimension (LID) by leveraging the Fokker-Planck equation associated with diffusion models (DMs). This new method effectively addresses several limitations of existing LID estimation techniques, such as inaccuracy, high computational demand, and the need for multiple pre-trained models. \n\n**2.** The authors thoroughly validate the performance of the FLIPD estimator through a series of benchmark tasks, demonstrating its superiority over existing baselines, particularly in high-dimensional settings.\n\nWEAKNESSES:\n**1.** The most critical section of this paper is Section 3.2, which introduces the core innovation of the FLIPD estimator. However, the insights from Equations (12) and (13) are not immediately clear. It is difficult to understand the insights of these derivations. Moreover, many details are deferred to appendix. The authors should provide more detailed explanations of the Fokker-Planck equation and its relevance to LID estimation. \n\n**2.** One weakness of the paper is the unclear demonstration of the computational advantages of the proposed FLIPD estimator compared to the method in [58]. The authors argue that while the method in [58] addresses some limitations of LIDL, it remains computationally expensive. However, the paper lacks a thorough theoretical analysis or experimental evidence to substantiate the claim that FLIPD is more computationally efficient.\n\nQUESTIONS:\n**Q1.** It seems that the proposed estimator over-estimate the LID compared to [10,48]. Can you provide some insights? \n\n**Q2.** How does the FLIPD estimator maintain accuracy and efficiency in LID estimation across varying levels of data complexity and different types of high-dimensional datasets, and what specific mechanisms within the Fokker-Planck framework contribute to these properties?\n\nLIMITATIONS:\nNot applicable", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-26"}