Title: The Adaptive Complexity of Minimizing Relative Fisher Information
Venue: NeurIPS 2025
Accepted: yes
Keywords: parallel sampling

non-log-concave

relative fisher information
Abstract: Non-log-concave sampling from an unnormalized density is fundamental in machine learning and statistics. As datasets grow larger, computational
efficiency becomes increasingly important, particularly in reducing adaptive complexity, namely the number of sequential rounds required for sampling algorithms. In this work,  we initiate the study of the adaptive complexity of non-log-concave sampling within the framework of relative Fisher information introduced by Balasubramanian et al. in 2022.  To obtain a relative fisher information of at most $\varepsilon^2$ from the target distribution, we propose a novel algorithm that reduces the adaptive complexity from $\mathcal{O}(d^2/\varepsilon^4)$ to $\mathcal{O}(d/\varepsilon^2)$ by leveraging parallelism. Furthermore, we show our algorithm is optimal for a specific regime of large $\varepsilon$. Our algorithm builds on a diagonally parallelized Picard iteration, while the lower bound is based on a reduction from the problem of finding stationary points.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the adaptive complexity of sampling from a non-log-concave distribution $\\pi\\propto\\exp(-V)$ on $\\mathbb{R}^d$ with $\\nabla^2L\\preceq LI$, such that the output distribution is $\\varepsilon^2$-close to $\\pi$ in Fisher divergence. The paper proposes a parallelized algorithm based on average LMC [BCE+22] and parallel Picard method [ZS24], and provides an improved complexity upper bound; also the paper presents a lower bound based on the adaptive complexity of finding a stationary point of $V$ [ZHTS25].\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths\n\nThis paper studies an interesting and important problem in the intersection of non-log-concave sampling and parallel sampling, and provides a comprehensive analysis. The paper is well-written with a strong motivation and a clear structure. The main results are also clearly stated, and the proofs are provided in an organized manner. I have checked most of the proofs and did not find significant issues, though with some minor typos and notations that could be improved. Also, the discussion on future work is very insightful.\n\n# Weaknesses\n\nThis work is mainly based on the proof techniques developed in [BCE+22], [ZS24], and [ZHTS25]. Although the results (especially the matching upper and lower bounds) are very nice, the paper does not introduce outstandingly novel techniques for the proof.\n\nAlso, I'd like to point out the following issue on readability. The paper [BCE+22] introduced the averaged LMC, which is easy to understand for the community; however, the second paper [ZS24] on the parallel Picard method is relatively new and not well-known in the sampling community. For readers that are unfamiliar with parallel sampling like me, this may pose a challenge. Though I have followed most of the proof, I'm still trying to understand the basic intuition behind this algorithm. I suggest the authors to provide a more detailed explanation of this algorithm, including its motivation and key components. For instance: What's the purpose of $J$ and $N$? Why we need to split the same update rule (lines 11 & 15 in Alg. 1) into two parts with different indices? How does it improve upon [ACV24]? A more interesting question to think about is that, if one applies the averaged LMC into the algorithm in [ACV24], how would the complexity bound be compared with Thm. 3.1?\n\nQUESTIONS:\n- We may not need such a formal definition of the adaptive algorithms like in Sec. 2.1, as the only concept that would appear later in presenting the main results is $\\mathsf{Comp} _ \\mathsf{R}$. They can be moved to the appendix, and you can change the way for stating Thm. 4.1.\n- In line 3 of Alg. 1, this is not the correct way to initialize a trajectory of Brownian motion, as $B _ {t+s_1}$ and $B _ {t+s_2}$ are not independent conditional on $B _ t$. The initialization should be a cumulative sum of independent $\\mathcal{N}(0,(mh/M)I)$ increments.\n- If my understanding of Alg. 1 is correct, $x^\\cdot _ {n,m}$ corresponds to time $nh+\\frac mMh$, and I hope the authors could mention this in the text for easier understanding.\n\n# Minor comments\n\n- Line 104: $f$ should be $V$.\n- Line 127: should be $\\mathsf{FI}(\\rho\\|\\pi):=\\mathbb{E} _ \\rho\\|\\nabla\\log(\\rho/\\pi)\\|^2$.\n- Lines 152 and 428: better to include the lower and upper limits of the integral.\n- Equation between lines 157 and 158: $w _ i$ should be a scalar, so do not use bold.\n- Line 158: the inequality should be on $t$'s, not $x _ t$'s: $t _ n=t _ {n+\\tau _ {n,0}}\\le ...\\le t _ {n+\\tau _ {n,M}}\\le t _ {n+1}$.\n- Line 160: $m'$ should be $m$, should add $p=0,1,...,K-1$, and $\\bf{x} _ 0$ should be $\\bf{x} _ {t _ n}$.\n- Typo in the title of Sec. 3.\n- Alg. 1, last equation: I believe $t-kh$ should be $t-\\frac{kh}{M}$ and $B _ {\\lfloor k/M \\rfloor,k-\\lfloor k/M \\rfloor M}$ should be $B _ {hk/M}$.\n- Lines 398, 403, and the following equation: $\\frac{(m+1)hh}M$ should be $\\frac{(m+1)h}M$.\n- Line 401: $\\beta$ should be $L$.\n- Line 416: no need to mention $p$.\n- Lines 419 and 424: I hope the authors could add more details to the derivation of the equations that follows. There are some missing steps that are not obvious to me.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors study the adaptive complexity of sampling from non log-concave targets and prove some novel results. In particular, when all the factors are constant except for dimension, the parallel averaged Langevin MC algorithm is shown to have an adaptive complexity of $\\mathcal{O}(\\log d)$. This apparently matches the case for strongly log-concave sampling.  In some particular regime, the lower bound of parallel LMC matches the sequential LMC, and is optimal. Unlike in optimisation, parallelism can accelerate sampling even when the target is not log-concave. The authors use the tool of relative Fisher information, introduced in another paper from 2022.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n- The results show not only that average LMC can be accelerated by parallelism, but also show that high-accuracy (in the sense of FI minimiser) via parallelism is not possible in the case of non log-concave. \n\n**Weaknesses:**\n- Notation is at times confusing. \n    - On line 104, is $f$ the same as $V$? $V$ is also displayed without its argument when defining the normalising constant (I believe $V(x)$ might be easier to read). Same confusion about $f$ versus $V$ on line 113.\n    - $L$-log smooth is defined without stating the conditions on $L$. I believe $L$ should be a scalar. Does it need to be negative?\n    - The oracle $\\mathcal{O}$ clashes with the complexity $\\mathcal{O}$ used in the introduction.\n- The initialisation assumption on line 129 implies a condition on the overlap of the support of the initial sampling distribution versus the target distribution. This is okay, but it might be nice to explicitly mention this to give further intuition about the assumption.\n- I am trying to understand why the last line above 425 needs to be used. Why can't we just keep the form with the exponential function and fold that through the rest of the statements?\n- The paper does not contain any experiments whatsoever. Is it possible to somehow plot empirical bounds for non log concave densities?\n\n**Minor:**\n- Formatting of Appendix A is very unusual. It is titled \"Useful facts\", however just lists one single fact, which is just the variational form of the f divergence (in particular, KL divergence).\n- Lemma B.2 is not grammatically correct.\n- \"We denotes\" on line 397 should be \"We denote\"\n- Line 82, typo in \"Appendix\".\n- Line 91. \"studying\" should be \"study\", I think.\n- LMC in equation (1) is not described as Langevin Monte Carlo.\n- Sometimes \"Fisher\" is written \"fisher\"\n- Table 1, \"Quires per iteartion\" has two typos.\n- $\\lesssim$ is not defined\n\nQUESTIONS:\nPlease address my weaknesses above. In particular:\n- Clarify the notation\n- Clarify the implication of the bound on the initial KL diveregence.\n- Why the bound on the exponential function needs to be applied.\n- Comment on why you chose not to ad any experiments.\n- No need to respond to each minor comment, unless you feel it is important or I have misunderstood (I don't).\n\nLIMITATIONS:\nAppendix C lists one sentence of limitation, pointing out the gap between the lower and upper bounds. I would also add that failing to include any experimental investigation is also a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a parallel algorithm for Langevin Monte Carlo for non-log-concave sampling. By leveraging parallelism, we can obtain a sample $\\varepsilon^2$ close to the target distribution under relative Fisher information with the adaptive complexity $O(d/\\varepsilon^2)$, which improves from the complexity $O(d^2/\\epsilon^4)$ without parallelism. The paper also shows a matching lower bound when $\\varepsilon = \\sqrt{Ld}$. It turns out that the sequential algorithm is also adaptively optimal in this regime, which rules out the possibility of a general high-accuracy sampler via parallelism.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The question of non-log-concave sampling with a parallel algorithm of Langevin Monte Carlo is interesting.\n2. The related work is clearly covered.\n3. The exposition of the technical proof of the upper bound is clear, and the obtained bound has a substantial improvement over the existing ones. A matching lower bound is also provided.\n4. The results have interesting implications. For example, a general high-accuracy sampler via parallelism is not possible, and there is a separation between optimization and sampling when using parallelism to accelerate Langevin Monte Carlo in strongly log concave vs non-log concave cases.\n\nWeaknesses:\n1. The algorithm is not clearly presented. By reading the reference [ZS24] I realized that there is parallelization both within time slices and across time slices. I would otherwise only get the part of parallelization within time slices. Currently, Figure 1 + Algorithm 1 do not have enough explanations of the algorithm, and arrows in Figure 1 are quite abstract and confusing.\n2. I am not sure how practical the assumption that poly($d$) queries can be made per iteration is, especially in high dimensions.\n3. The current exposition of the lower bound is not very friendly to a nonexpert in this field.\n\nQUESTIONS:\nQuestions:\n- Theorem 3.1 states the result for the averaged distribution. But this averaging is not mentioned in the statement of the theorem.\n- Algorithm 1: does line 8 initializes all $x_{n,m}^0$ identically? Also, I have difficulty parsing the setup of $n$ and $j$ in line 10 and line 14 of the algorithm. Maybe some explanations will help readers.\n- page 8 line 231-233, why is the probability at least $1/2$? I am also not sure how this is used in the lower bound proof.\n- Lemma 4.5 is not proved/referenced. And in general, the lower bound proof is very confusing to me. I think the outline of the proof is not well-stated at the beginning, which makes it easier for readers to get distracted by the cited lemmas and theorems (especially theorem 4.4 is quite novel for nonexperts). Also, when using the Donsker-Varadhan variational principle, the involved function is required to be bounded, but a quadratic function is used here. Could you clarify why this is legitimate?\n- In the proof of Lemma B.5, the equation after line 424, I am not sure how those two binomials pop out in the second inequality.\n\n\nTypos:\n- page 3 line 82, Appnedix -> Appendix\n- page 3 line 104: the potential function is first given as $f$ and later used as $V$. Notations in the lower bound section are also not unified (such as the Lipschitz constant).\n- page 5 line 160, for $m' = 1, \\dots ,M$ -> maybe $m'$ should be $m$. Also, the equation afterwards: $x_0$ should be $x_{t_n}$?\n- page 7 line 198, the interval upper bound has a typo. Also, in the equation after line 199, $\\rm{KL}_{0,0}$ is not defined in the main text.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper advances the theory of non-log-concave sampling by establishing both upper and lower bounds on the adaptive complexity required to minimize relative Fisher information. Building on the work of [BCE+22] for Langevin Monte Carlo (LMC) in the non-log-concave setting and [ZS24] for the Parallel Picard method, the paper’s central contribution is the innovative integration of the Parallel Picard method with LMC for non-log-concave distributions. This combined approach demonstrably improves adaptive complexity over parallel LMC. Moreover, the paper derives a lower bound on adaptive complexity in the large $\\epsilon$ regime.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\nNovel Application of the Parallel Picard Method: The application of the Parallel Picard method is particularly compelling, as it circumvents challenges such as estimating the complexity of score-matching functions in diffusion models, as encountered in [ZS24]. This enables a more efficient improvement over vanilla LMC.\n\n**Weaknesses:**\nWhile parallel LMC lends itself to straightforward vectorized or matrix-based implementations, the proposed integration with the Parallel Picard method requires explicit implementation details to demonstrate its practical feasibility and validate the claimed theoretical improvements. Furthermore, for high-dimensional settings, a discussion of the space complexity of the parallel algorithm is necessary, especially in light of the absence of empirical validation.\n\nQUESTIONS:\n1. Why is the focus on the specific regime of large FI, with $\\epsilon=\\sqrt{Ld}$, particularly significant? Is it possible to extend the results of Theorem 4.1 to smaller values of $\\epsilon$? If such an extension is not feasible, it would be helpful to clarify the obstacles involved. Without this clarification, the claim of tightness for the lower bound might appear less robust, potentially weakening the perceived quality and significance of the result.\n\n2. Theorem 4.1 states the lower bound in terms of $L$ and $d$, but does not explicitly show the dependence on $\\epsilon$, even though the accuracy is constrained by $\\epsilon\\geqslant \\sqrt{dL}$. Would it be clearer to retain the $\\epsilon$ dependence in the expression on line 248, i.e. \\# iteration$\\gtrsim\\dfrac{\\Delta L}{\\epsilon^2}$, while explicitly noting the limitation $\\epsilon\\geqslant\\sqrt{dL}$.\n\n3. The absence of experimental results limits the validation and practical support of the theoretical findings.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}