Title: Parallelizing MCMC Across the Sequence Length
Venue: NeurIPS 2025
Accepted: yes
Keywords: probabilistic methods

Markov chain Monte Carlo

parallel algorithms
Abstract: Markov chain Monte Carlo (MCMC) methods are foundational algorithms for Bayesian inference and probabilistic modeling. However, most MCMC algorithms are inherently sequential and their time complexity scales linearly with the sequence length. Previous work on adapting MCMC to modern hardware has therefore focused on running many independent chains in parallel. Here, we take an alternative approach: we propose algorithms to evaluate MCMC samplers in parallel across the chain length. To do this, we build on recent methods for parallel evaluation of nonlinear recursions that formulate the state sequence as a solution to a fixed-point problem and solve for the fixed-point using a parallel form of Newton's method. We show how this approach can be used to parallelize Gibbs, Metropolis-adjusted Langevin, and Hamiltonian Monte Carlo sampling across the sequence length. In several examples, we demonstrate the simulation of up to hundreds of thousands of MCMC samples with only tens of parallel Newton iterations. Additionally, we develop two new parallel quasi-Newton methods to evaluate nonlinear recursions with lower memory costs and reduced runtime. We find that the proposed parallel algorithms accelerate MCMC sampling across multiple examples, in some cases by more than an order of magnitude compared to sequential evaluation.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of speeding up Markov chain Monte Carlo (MCMC) by parallelizing across the sequential steps of each chain. The authors adapt the DEER algorithm (a parallel Newton-based method originally used for RNN-style recurrences) to solve an entire chain’s updates in parallel. This approach is novel (up to my best knowledge): by performing $O(\\log T)$ parallel Newton iterations instead of $T$ sequential steps, they show it’s possible to generate long trajectories with only a few dozen iterations. The key innovation is recasting common MCMC samplers (Gibbs, MALA, HMC) as fixed-point problems that can be solved using parallel Newton-type methods.\n\nThe authors develop several algorithms within the DEER framework:\n- **DEER**: Uses full Jacobians for quadratic convergence with O(log T) parallel time complexity\n- **Quasi-DEER**: A cheap variant using only Jacobian diagonals \n- **Stochastic Quasi-DEER**:  reduces the memory footprint by estimating Jacobian diagonals\n- **Block Quasi-DEER**: Optimized for HMC's leapfrog integrator structure\n\nThe methods demonstrate  10×–30× speedups over sequential sampling across various applications, including hierarchical Gaussian models, Bayesian logistic regression, and complex targets. \n\nThis breakthrough transforms MCMC from a fundamentally sequential O(T) process into a parallelizable O(log T) problem, opening new possibilities for scaling Bayesian inference on modern accelerators. The work establishes both theoretical foundations and practical algorithms for leveraging massive parallelism in Monte Carlo sampling.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper builds on a existing framework (treating the chain update as a fixed-point problem solved by Newton’s method) and extends it with appropriate modifications. For example, the authors acknowledge that the straightforward DEER algorithm would require storing and multiplying through $T$ Jacobian matrices of size $D\\times D$, leading to cubic time and quadratic memory in state dimension $D$. They address this by using quasi-Newton updates that only use the diagonal of each Jacobian (or Hutchinson’s stochastic diagonal estimate), reducing complexity to $O(D)$ and making the method much more scalable. They also introduce a block-diagonal Newton scheme for HMC’s leapfrog integrator, which accounts for position–momentum coupling and roughly halves the number of iterations needed to converge compared to a naive diagonal approach.\n-  the authors identified potential bottlenecks (memory use, slow convergence) and devised solutions grounded in numerical methods. The approach for handling non-differentiable Metropolis acceptance steps is also reasonable: they perform the exact accept/reject in the forward pass but omit its gradient in the backward pass (stop-gradient), citing a proposition that guarantees Newton convergence despite this discontinuity. While this is a somewhat heuristic workaround, it’s backed by theory and appears to work well in practice (the parallel MALA yields correct results). Overall, the methodology is sound and the authors seem aware of its theoretical conditions and limitations\n- The authors implement their parallel MCMC algorithms in JAX on  GPU, ensuring a comparison with optimized baselines. They test the approach on multiple problems: a hierarchical Bayesian model (the 18-dimensional “eight schools” problem) using a reparameterized Gibbs sampler, a Bayesian logistic regression (BLR) with MALA and HMC on the German Credit dataset, and even a 501-dimensional item-response model (in the appendix) to demonstrate scaling to higher dimensions. For each case, the paper reports  raw speed, sample quality via Maximum Mean Discrepancy (MMD) to a ground-truth or long-run baseline. The numerical results strongly support the paper’s claims. For instance, in the BLR experiment, parallel MALA converges in only tens of Newton iterations even for very long chains. The Gibbs sampler results show a more modest but still solid gain – roughly 2× speedup over traditional sampling on the eight-schools model when chains are long. The benefit there is limited (likely by the low dimension and cheaper updates per step), but it demonstrates that even a relatively simple Gibbs chain can leverage GPU parallelism for moderate gains. \n\nWeaknesses\n- The main weaknesses of the paper lie in the limitations and scope of the proposed approach. First, while the method does achieve large speedups, it comes at the cost of substantially increased computation per iteration and memory usage. The authors explicitly note that their algorithms “trade memory and compute for time”. In scenarios where the target density or gradient is extremely expensive to evaluate, performing many parallel gradient evaluations (as this method does) might not actually be efficient – the paper acknowledges that if the model is very costly per step, those resources might be better spent on more sequential steps or independent chains.\n- Secondly, the memory overhead is non-trivial. Even with the quasi-Newton diagonal approximation, parallelizing very long chains or using large batch sizes can exhaust GPU memory (the authors report out-of-memory errors for their largest tested setting, 32 parallel chains of length 64K, using the memory-hungry version). The paper’s solution – the memory-efficient Jacobian estimator – was critical to make the BLR experiments feasible, but this comes at the price of using stochastic estimates, which might in some cases slow convergence (indeed, they observed the plain quasi-Newton often took the maximum allowed iterations to converge for HMC until the block method was introduced). Thus, while the authors did mitigate this issue, scalability to very high dimensions or extremely long chains remains a concern \n- One minor concern is that the analysis of convergence and conditions for quadratic speed-up are mostly cited from prior work; the paper itself does not deeply analyze when Newton’s method will converge quickly for a given MCMC (aside from empirical observation). However, given space constraints, this is understandable and the empirical results provide evidence that a few dozen Newton iterations usually suffice. \n- The paper focuses on Gibbs (with continuous reparameterizable conditionals), Langevin dynamics, and HMC with fixed hyperparameters. It does not address more complex adaptive methods like NUTS or discrete-variable samplers. The authors explicitly leave extension to adaptive HMC and other MCMC variants for future work. Consequently, the immediate applicability is somewhat limited – many practical Bayesian inference tasks use NUTS or other adaptive schemes that would need non-trivial adaptation of this method. Finally, from an originality perspective, one could argue that the paper is an incremental advance in that it leverages an existing technique (parallel Newton solver from prior work) rather than inventing a wholly new sampling paradigm. The true novelty is in recognizing that this technique can be applied to MCMC and making it work for those algorithms; this is a meaningful innovation, but it’s not a completely new theory of MCMC.\n\nQUESTIONS:\n-  The paper references the DEER algorithm (from Lim et al. [5]) as part of the parallelization approach, but its presentation is quite limited. Even if DEER is not the central theme, a clearer self-contained explanation of what DEER does and how it integrates into your method would greatly improve readability. For instance, please elaborate on the intuition and steps of the DEER algorithm (and quasi-DEER) in the main text or appendix. This helps readers who are unfamiliar with prior work [5,6] understand the foundation of your parallel MCMC approach without extensive external reading. \n- The paper makes several claims (e.g. about convergence guarantees, sublinear scaling, “orders of magnitude” speedups, etc.) that would benefit from stronger theoretical backing or at least references to supporting theory. I encourage the authors to support these claims with background results or analysis. For example, if you claim that the stochastic quasi-DEER “enjoys provable convergence guarantees” or that Newton’s method yields quadratic convergence, please either cite the precise theorems/propositions (from prior work or your appendix) or outline the conditions under which these claims hold. \n- While I find the application of parallel Newton/DEER methods to MCMC to be novel and exciting, the manuscript could do more to explicitly position this contribution relative to prior work. I suggest the authors clearly highlight what is new in this work (e.g. parallelizing specific MCMC samplers across chain length, the new quasi-Newton variants, etc.) versus what is adopted from existing methods. For instance, discuss how your approach differs from or improves upon prior parallel MCMC efforts or parallel diffusion simulations [18–22,28,29] mentioned in related work.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrevious work has shown that nonlinear RNNs can be evaluated in parallel using Newton's\nmethod (DEER) and this has been made stable and faster in subsequent work (ELK).\nThe current paper applies this technique to the task of drawing MCMC samples thus\nenabling withing-chain parallelism.\n\nThe paper also explores the application of various standard approximations to further\nbring down the computation cost to a product of the state space times the length of the\nsequence (as opposed to quadratic in the size of the state space).\n\nFurther the paper demonstrates that the application of DEER (and its approximations) to\nMCMC chains leads to very fast convergence. Thus we don't need a very long sequence of\nstates.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths - novel application of RNN literature to MCMC\n\nThe original DEER and ELK were designred for parallelizing nonlinear RNNs. The authors\nhave taken that work and drawn an interesting parallel to MCMC. This appears to be\nnovel and allows for MCMC to leverage advances in a different area of AI.\n\n## Strengths - fast convergence and high ESS.\n\nThe experiments do seem to suggest that the application of DEER to MCMC allows for \nconvergence with very few samples and these tend to have very high Effective Sample Size\n(ESS). This makes the approach very promising.\n\n## Weakness - unclear MALA algorithm\n\nI couldn't follow the use of the stop_gradient in the MALA algorithm. I do understand\nhow this trick normally works, but this formulation of the MALA algorithm was not very\nclear to me and Appendix A doesn't have any further explanation even though line 101 \npromises that appendix A would have additional details.\n\n## Weakness - very small scale experiments\n\nThe N schools example only had an 18-dim posterior. Given the significant improvements\nclaimed in the paper the authors could have demonstrated on a higher dimenstional problem.\n\n## Weakness - no demonstration of accuracy\n\nThe paper has proposed a number of approximations but none of the experiments demonstrate\nthat the results are actually correct.\n\nFurther none of the models have multimodal posteriors so it is unclear whether all of these\napproximations will allow for efficient exploration of the posterior space.\n\nQUESTIONS:\nOn line 94 what does it mean to say \"g <- \\sigma(g) + ... \"? (This looks like a recurrence?)\n\nHow big is the posterior state space in the German Credit Bayesian Logistic Regression \nexample?\n\nHow does this method work on problems with multimodal posteriors? Does the small number of\nsamples that seem to be needed for DEER also allow for efficient exploration of \nmultimodal posteriors.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nMCMC has a computational cost that is linear to the number of samples that one would like to simulate. This computational cost can be improved via parallelization. Most existing methods does so by simulating several independent Markov chains and aggregate all resulting samples. This paper takes a different approach and simulates a sequence of MCMC states (given a fixed length) by treating the sequence as the solution to a fixed-point problem, which can be solved using a parallelized form of Newton's method. The author specifies the nonlinear sequence model for a number of MCMC methods, which can be used to obtain the MCMC state sequence as the solution of corresponding fixed-point problems. It is shown that this approach can lead to improved wall-clock times compared to the traditional sequential simulation of MCMC states when the number of states generated is the same.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper is for the most part very well organized and clearly written. The problem studied here is very important as improving the computational cost of MCMC is a long-standing topic. The proposed algorithm here is easy to understand and simple to use. Compared to other methods that combine states from multiple independent MCMC instances, there is no extra care needed for aggregating samples from different chains.\n\nQUESTIONS:\nOne suggestion that I have is to expand the background section. While the author does provide all necessary citations to understand the DEER algorithm (which is a core component of the proposed methodology), the text in section 2 itself may not be detailed enough for those that have not heard of the DEER algorithm before (myself included). \n\nMore specifically, it wasn't immediately clear to me that the number (T) of MCMC states to be simulated is fixed, and that the Newton iterations are optimizing the entire sequence of T states as a whole. It may be a good idea to outline the DEER algorithm in section 2. This would help make the subsequent sections of the paper easier to read.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

comments: SUMMARY:
Previous work on parallelising MCMC on multi-core CPUs and GPUs has focused on running multiple chains in parallel. This method takes recent work on parallel evaluation of non-linear recursions using Newton's method (DEER, quasi-DEER) and proposes a framework to run an MCMC sampler in parallel across the chain length. The authors demonstrate their approach by extending Gibbs sampling, Metropolis-adjusted Langevin, and Hamiltonian Monte Carlo. They then extend their framework to use quasi-Newton methods which reduces the wall-clock runtimes by orders of magnitude for a number of test problems.

STRENGTHS AND WEAKNESSES:
Strengths

1) Very explanation of the problem, and demonstration of their solutions effectiveness with a number of widely-used MCMC samplers.
2) Significantly reduces wall-clock time for a number of MCMC samplers by permitting the use of greater computational resources.
3) Good evaluation of the method for a number of test problems, including Bayesian Logistic Regression.

Weaknesses

1) Method cannot clearly be applied directly to adaptive MCMC samplers.
2) Speedup is less than linear relative to hardware, so 2x hardware doesn't lead to 2x performance improvement. Although this is mitigated through the availability of GPUs.

QUESTIONS:
1) Do you see a clear path towards extending your framework to support adaptive MCMC samplers? If so, could you include a sentence or two about what would be required? It is not clear whether this is feasible.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-07