Title: Restricted Spectral Gap Decomposition for Simulated  Tempering Targeting Mixture Distributions
Venue: NeurIPS 2025
Accepted: yes
Keywords: Simulated tempering

Multimodal sampling

Mixing time

Markov chain decomposition

Restricted spectral gap
Abstract: Simulated tempering is a widely used strategy for sampling from multimodal distributions. In this paper, we consider simulated tempering combined with an arbitrary local Markov chain Monte Carlo sampler and present a new decomposition theorem that provides a lower bound on the restricted spectral gap of the algorithm for sampling from mixture distributions.  By working with the restricted spectral gap, the applicability of our results is extended to broader settings such as when the usual spectral gap is difficult to bound or becomes degenerate. We demonstrate the application of our theoretical results by analyzing simulated tempering combined with random walk Metropolis--Hastings for sampling from mixtures of Gaussian distributions. Our complexity bound scales polynomially with the separation between modes, logarithmically with $1/\varepsilon$, where $\varepsilon$ denotes the target accuracy in total variation distance, and exponentially with the dimension $d$.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThere has been a lot of work in recent years on establishing the complexity of Langevin, MALA or related MCMC algorithms when sampling from a log-concave density. However, such approaches fail short when the target distribution is severely multi-modal. A related line of work is to study the complexity of tempering-based algorithms (such as parallel tempering, simulated tempering, or tempering SMC) under the weaker assumption that the underlying Markov kernels mixed well at least *locally*, in various subsets forming a partition of the sampling space. This paper continues this line of work, and used the concept of *restricted spectral gaps* to derive new, tighter complexity bounds for simulated tempering. \n\nThe first part derives new result under very general assumptions on the target distribution and the local kernels, whereas the second part specialises these results to the case where the target distribution is a Gaussian mixture, and the local kernels are random walk Metropolis kernels.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n=======\n\n* The paper is nicely written, I enjoyed reading it. \n\n* From a cursory reading, it looks like a significant improvement over previous results of the same type. In particular, the paper is quite close in spirit to Ge et al. (2018), but Ge et al consider the continuous-time settings (which is typically *easier* to deal with, because no Metropolis correction is required), and this paper uses novel tricks such as the the projected Markov chain (see Remark 3). Disclaimer: I have not read Ge et al. (2018). \n\nWeaknesses\n==========\n\n* A journal paper might be a better format for presenting this kind of results.\n\n* The direct relevance to ML is a bit debatable, especially the part on sampling from Gaussian mixture distributions (since it is a toy problem).\n\nQUESTIONS:\n* It would be nice if the authors could discuss a bit more the relevance of their work to multi-modal target distributions arising in ML, e.g., posterior distributions for neural networks.\n\nLIMITATIONS:\n* 42: there are more recent/relevant references for SMC samplers than Liu & Chen, e.g. Jasra et al. (2006, JRSSSB, read paper) the book of Chopin and Papaspiliopoulos (2020), or Syed et al (https://arxiv.org/abs/2408.12057).", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper analyzes the spectral gap of simulated tempering under a model of multimodality. In contrast to many related work, one strength of section 2 of the manuscript is that it does not assume a specific algorithm for the per-chain base MCMC algorithm. The authors then specialize the general framework to a specific algorithm in section 3 (namely, random walk MH) to demonstrate the utility of their approach.\n\nSTRENGTHS AND WEAKNESSES:\nI appreciate the honesty of the authors in stating that \"It is important to note that this approximate STMH sampler cannot be implemented in practice, as it requires knowledge of the component distributions\". I am OK with that leap of faith. Just as statisticians find value in a data model that is wrong but useful, theory can benefit from this kind of model-based approach.\nOf course, it is always good to describe the limitation of the model, and while a good effort has been made already by the author, I will add that it would be good to mention that real world multimodal posterior have an exponential number of components, and that irregular posteriors are not necessarily difficult because of multimodality, concentration to a manifold being another example. \n\nI really enjoyed the modularity of the theory, with the general framework in section 2 being applicable to other types of samplers. This modularity reminds me the work of Biron-Lattes et al (JASA, 2024), but from a different perspective.  \n\nA bit more discussion on whether the analysis has methodological recommendations associated with it would be good, if something can be said. \n\nThe paper briefly mentions in Remark 5 that the new bound has an exponential dependence in the dimensionality. I feel this should be flagged in the introduction, as well as by adding a row in Table 1 to include the proposed bound.\n\nThe paper falls in the usual fallacy of using upper bound comparisons to make statements about comparing algorithms, for example \"while 254 STMH achieves a more favorable logarithmic dependence on ε\" - what is being compared there are upper bounds, so no statement should be made about \"STMH achieving a more favourable dependence\". \n\nThe introduction promises results on the approximation of the target distribution. However, the closest formally stated result in the main text consists in bounds in TV with respect to P, which is ST's stationary distribution. Now, in contrast to the typical scenario where the target distribution is a marginal of the augmented target (such as with PT or HMC), with ST the target appears as a conditional, not a marginal. Hence I was not able to see exactly what result delivers the abstract's promises. Theorem 5 seems a likely candidate but it is only informally stated and the reader is sent to \"Appendix C\" in which again I could not exactly locate a result on TV of conditionals of ST. \n\nThere description of the classical ST algorithm in the supplement makes the notation very confusing. Algorithm 1 is presented as taking as input a deterministic number of steps. Then, in Algorithm 2, the inner loop calls Algorithm 1 to obtain a fixed number of samples. However, I infer from the importance sampling estimator in the line below that these samples are only the random subset at level s. However an algorithm with that kind of random stopping time does not match with how Algo 1 was defined. So I recommend to fix in Algo 1 what is the stopping time, to make it random, and also make explicit how the output samples are collected (to emphasize that it is NOT the list of all samples produced in contrast to standard MCMC algorithms). This makes me a bit worried that there is a gap between the actual algorithm and the analysis, because I did not see analysis of the random stopping time involved to get a prescribed number of samples. (alternatively, if the algorithm meant s to be random, then the algorithm is just ill defined since there could have been zero visit making the procedure undefined). \n\nRelatedly, the manuscript is often unclear on whether the adaptive MCMC algorithm estimating the normalization constants is being analyzed, or the homogeneous MCMC ST from Def 2.\n\nQUESTIONS:\nMy main question is the issue of the marginal/conditional described above.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a new theoretical framework for analyzing simulated tempering algorithms, particularly in the context of sampling from multimodal distributions. The authors focus on bounding the restricted spectral gap, which allows for analyzing mixing times without requiring global spectral gap assumptions—often hard to establish in high dimensions or multimodal settings. Their main contribution is a discrete-time Markov chain decomposition theorem that yields lower bounds on the restricted spectral gap of the simulated tempering chain. This framework is applied to a simulated tempering Metropolis–Hastings (STMH) sampler targeting Gaussian mixture models, showing that the algorithm mixes in time polynomial in the mode separation and logarithmic in the target accuracy.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Quality:  The technical quality of the paper is high. The decomposition theorem is rigorously developed and soundly argued. The authors make a meaningful extension to discrete-time chains, which is valuable for practical MCMC algorithms like Metropolis–Hastings. The proofs, though deferred to the appendix, are sketched well in the main text and the assumptions are clearly stated.\n\n-Clarity: overall, the paper is clearly written, especially for readers familiar with MCMC theory and spectral gap analysis. However, certain key steps (e.g., interpretations of Assumptions 2 and 3, or the intuition behind Equation (2)) could benefit from additional motivation or examples. The paper assumes a fairly expert audience.\n\n-Significance: the contribution is significant for the MCMC and sampling community, especially in contexts where standard spectral gap bounds are either loose or intractable. The approach improves upon prior work (e.g., Ge et al., 2018) by simplifying the projection chain and relaxing some assumptions. Polynomial dependence on d  (mode separation) is significantly better than many standard samplers (e.g., unadjusted Langevin or proximal methods) which suffer exponential dependence on d.\n\n-Originality: the paper offers a novel discrete-time decomposition for restricted spectral gap analysis. While inspired by prior continuous-time analyses, the authors substantially modify the approach, especially in how they define and analyze the projected chain. The use of restricted spectral gap (vs. full spectral gap) as the main analytical tool is also a meaningful shift.\n\n\nWeaknesses:\n- Limited assumptions: (eg Ass 2) The paper does not offer constructive guidance or examples on how to ensure this assumption holds in practice for general mixture models.\n-The dependence on dimension d is exponential in their result (hidden in constants due to the use of random walk Metropolis–Hastings).\n- limited experiments: the empirical study focuses only on a symmetric 2D Gaussian mixture, which is a toy example.\nThere are no comparisons with baseline methods (e.g., Langevin, annealed Langevin, or proximal samplers), and no error bars or statistical significance reported.\n\nQUESTIONS:\nOn Assumption 2 :\nWhile the theoretical condition in Equation (2) is essential to the decomposition, in practical settings (especially for general Metropolis–Hastings chains), verifying or constructing Mij  that satisfy the inequality seems challenging. Can the authors provide further guidance or heuristics on how to check or enforce this assumption in practice?\n\nLIMITATIONS:\nThe conclusion discusses some limitations, in particular on the assumptions 2/3.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe work provides a framework of analysing the convergence of STMH algorithms using a restricted spectral gap. First, in Theorem 2 they provide a general bound on the Spectral Gap. Next, in Theorem 5, the authors provide a time bound on obtaining samples (with a STMH algorithm) from a distribution \\epsilon TV distance from a (multivariate normal mixture) target distribution. They frame the theoretical result they achieve amongst other theoretical results in the literature and give comparisons. Finally they offer an empirical result and observe how close their bound is in practice.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The problem of sampling from mixture distributions is well known and is a fundamental task in MCMC sampling. Work towards this problem is certainly valuable.\n\n2. Concise introduction, which efficiently lays out the context of the problem. It also provides a literature review which appears comprehensive and is easy to follow and understand.\n\n3. The mathematics is easy to follow, and is well structured and concise. The bound achieved in Theorem 5, a time bound on achieving samples within \\epilson TV distance from the target, seems very strong and useful. I also greatly appreciate the comparison to STLMC, it provides details of the relative strengths of their results and provides some intuition.\n\n4. I appreciate the details regarding the proof of Theorem 5, and also like the fact this is included in the main text. It offers a lot of intuition for the various pieces of the proof, and gives readers an understanding from a high level.\n\nWeaknesses:\n1. I would like to see more justification for this work: what examples in the literature are there of people using STMH in practice? What implications will this work have on the wider field, how should it be interpreted etc?\n\n2. I would appreciate some text on the interpretation of a spectral gap. I would also appreciate a more gentle/intuitive introduction to the simulated tempering algorithm.\n\n3. I would appreciate a more intuitive (in words) explanation of many of the assumptions of the paper (namely, Assumptions 1 and 2). Are they standard assumptions? Are they likely true in reality? What are the consequences on performance if they aren’t true? More generally, there is very little intuition given to many of the assumptions, constants, and bounds throughout. Perhaps a large readership will be familiar with such concepts, but to those outside that group this is hard to interpret and thus use.\n\n4. I would appreciate more justification as to why we should care about convergence results on the STMH algorithm - what are the implications of such results? What is the practical advice to be gained from this paper? \n\n5. There is very little discussion on their empirical results: why is it only ‘approximately’ linear (Figure 1) - my understanding is that given the theoretical results it should be exactly linear? Is the discrepancy down to the assumptions not holding in practice?\n\nMinor: \n1. Why is Theorem 2 not Theorem 1 (same with Theorem 5, shouldn’t this be Theorem 2)?\n\n2. What is meant by [Line 178] ‘which can be compared to the continuous-time decomposition theorem of Ge et al. [2018]’ - is it that the bounds align, if so, say that. Or is it that the results can now be directly compared?\n\n3. Section 3 studies STMH on a multivariate Gaussian mixture, I think this should be clear from the Section title.\n\nQUESTIONS:\n1. My understanding is that r_i is a hyperparameter of the STMH algorithm which a practitioner would have to specify beforehand. As such, is it realistic to specify r_i as in Line 220? When one knows the target density (e.g. multivariate Gaussian mixture) perhaps this is possible, but this is not a situation in which practitioners will find themselves?\n\n2. To what extent can we understand whether the Assumptions on which Theorem 5 relies are true for the MVN-mixture case? Are the empirical results merely an inspection of the underlying assumptions and ‘what happens if they are not true’? More discussion on this would be appreciated, and would further justify the existence of the results section.\n\n3. What about those same assumptions applied to more realistic applied settings? Do you have any thoughts regarding what would happen?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-07"}