Title: $\Psi$-Sampler: Initial Particle Sampling for SMC-Based Inference-Time Reward Alignment in Score Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Inference-time reward alignment

Score-based generative models
Abstract: We introduce $\Psi$-Sampler, an SMC-based framework incorporating pCNL-based initial particle sampling for effective inference-time reward alignment with a score-based model. Inference-time reward alignment with score-based generative models has recently gained significant traction, following a broader paradigm shift from pre-training to post-training optimization. At the core of this trend is the application of Sequential Monte Carlo (SMC) to the denoising process. However, existing methods typically initialize particles from the Gaussian prior, which inadequately captures reward-relevant regions and results in reduced sampling efficiency. We demonstrate that initializing from the reward-aware posterior significantly improves alignment performance. To enable posterior sampling in high-dimensional latent spaces, we introduce the preconditioned Crank–Nicolson Langevin (pCNL) algorithm, which combines dimension-robust proposals with gradient-informed dynamics. This approach enables efficient and scalable posterior sampling and consistently improves performance across various reward alignment tasks, including layout-to-image generation, quantity-aware generation, and aesthetic-preference generation, as demonstrated in our experiments.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a method to better initialize particle samples for Sequential Monte Carlo samplers used to solve the problem of training-free guidance of diffusion methods. It does so by introducing a MCMC algorithm that targets the tractable version of the posterior distribution at time t=1, obtained through Tweedie's formula. It proposes in particular the pCNL MCMC algorithm to sample from the aforementioned distribution. The numerical experiments show both that there is a real gain of doing so in one toy problem and in several applications using several of the most recent generative models. It also shows that amognst the several candidates samplers that could in theory be used to initialize the distributions, pCNL seems to be the most efficient one.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper proposes an original idea to enhance considerably the performance of Sequential Monte Carlo methods for training-free conditional generation in diffusion models. It does so by proposing a suitable MCMC sampler to target the Tweedie's approximation of the posterior distribution at t=1. It evaluates the proposed method in several relevant and nowadays used problems, namely with the latest generative models used. \n\nWeaknesses:\nI think that while the evaluation of the current method in industry relevant problems, SMC is a fundamentally probabilistic tool. In this sense, I think that the evaluation in \"toy problems\" could be further enhanced and clarified. Namely, by running systematic evaluation for several mixtures and several different conditioning problems with a statistical like metric (such as the sliced Wasserstein), one would be more confident in the performance of the proposed fix (Namely, are any modes not recovered in some situations?). I would recommend reproducing the mixture of Gaussians example presented in [1].\nA second point is that there are some training-free guidance algorithms that could be included in the comparison to better reflect the state of the art, such as [2] and [3].\n\n[1] Cardoso, G. V., El Idrissi, Y. J., Le Corff, S., & Moulines, E. (2024, January). Monte Carlo guided Diffusion for Bayesian linear inverse problems. In ICLR International Conference on Learning Representations.\n\n[2] Janati, Y., Moufad, B., Qassime, M. A. E., Durmus, A., Moulines, E., & Olsson, J. (2025). A Mixture-Based Framework for Guiding Diffusion Models. arXiv preprint arXiv:2502.03332.\n\n[3] Qu, Q., & Shen, L. (2024). Solving Inverse Problems with Latent Diffusion Models via Hard Data Consistency.\n\nQUESTIONS:\n1. What happens to MALA in the mixture of Gaussian case?  Seems like an initialization problem, are those points stuck in local Minima? If so, why the pCNL is not stuck in the same local minima? Here the acceptance rate should not have a big importance, as the dimension is low. \n2. It should be stated somewhere in the text that the target distribution for the initialization is not exactly the posterior, but it's tractable version (obtained through Tweedie's) somewhere close to equation 11. It is stated that the approximations will be used in equation 3 but still can be misleading by the way equation 11 is written. I suggest the authors either define a notation for the Tweedie approximation that is distinct of the real potential or explicitly say in the text that they will use the tractable counterpart.\n3. The part of the toy problem of better mode coverage than MALA does not seem relevant, as both cover all the modes. The problem with MALA seems to be that it adds other modes to the prior due to the initialization.\n4. A detailled description of the setting used to produce the toy problem is lacking. It is not currently given in the appendix.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores Sequential Monte Carlo (SMC) for inference-time reward alignment, which becomes popular since it does not require fine-tuning. However, existing methods assume a Gaussian prior for initialization. This ignores two issues: (1) the reward signal tends to weaken in later steps, reducing the effectiveness of SMC and (2) modern diffusion models often benefit from better initial points due to mode connectivity in early steps and trajectory straightening.\n\nTo address this, the authors propose using pCNL to initialize samples with reward information. Since pCNL keeps the Gaussian prior even in high dimensions, it is suitable for image generation tasks that require high-dimensional noise sampling. The paper compares standard SMC (prior-based), MALA+SMC (posterior-based), and the proposed psi-sampler (pCNL + SMC), and shows that the proposed method performs better.\n\nSTRENGTHS AND WEAKNESSES:\nStrength\n- The motivation is clear and well-explained with literatures of the recent works. (such as SMC-based methods and Uehara et al.) \n- It is practically useful since the method improves performance at inference time without extra training.\n- The paper shows that pCNL is more effective than other sampling methods such as Top-K-of-N or MALA.\n- The method works well across three different practical tasks: layout-to-image, quantity preference, and aesthetic preference.\n\nWeakness\n- The main concern is the trade-off between sampling cost and the quality improvement. Since each sample requires a reward query on the predicted x0, the cost may be high.\n- It is a bit surprising that the simple Top-K-of-N approach achieves comparable results in some cases, which may reduce the overall impact of the proposed method.\n- As the authors mention, the method depends on a reliable reward model.\n\nQUESTIONS:\n- How much does inference time differ between the proposed method and the baselines?\n- Could you add one more simple toy example (like Figure 1)? While the three tasks are convincing, such intuitive toy examples would help.\n- The method seems closely related to Uehara et al. [20]. Is the main difference that their method requires training, while yours works as inference-time initialization? Could you elaborate more on this difference?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the $\\Psi$-sampler, a SMC-based framework for accurately sampling the posterior distribution formed by a pre-trained diffusion model and a reward function. The algorithm contains two components: a pCNL-based MCMC sampling for the initial, non-Gaussian prior distribution, and an SMC-based sampling scheme. $\\Psi$-sampler is numerically evaluated on various image generation tasks and demonstrates convincing performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strength:**\n1. The paper is very readable with a clear motivation and easy-to-follow logic.\n2. The proposed algorithm is novel and aims at an important application: post training of diffusion models.\n3. The numerical experiments are quite extensive and demonstrate the advantage of the proposed algorithms.\n\n**Weakness:**\n \n**I have a major concern regarding the validity of this idea of sampling from \"optimal initial distribution\" for technical reasons**. As is mentioned in [1], as well as eq (9) in the appendix, the optimal initial distribution is indeed proportional to $p_1(x_1)\\exp(V_1(x_1))$, where $v_t$ is the value function. From the expression of value function in eq (11),  the value function is given by\n$V_1(x_1)) = \\alpha \\log \\mathbb{E}[\\exp(r(x_0)/\\alpha) | x_1]$ where this expectation is taken with respect to the coupling between $x_0$ and $x_1$ produced by the path measure given by pretrained diffusion model. This creates an issue, as $x_0$ and $x_1$ are independent when the forward noising process add enough noise (which is usually the case in practice), making the conditional expectation just a regular expectation. Therefore, $V_1(x_1)$ is just a constant function, and the optimal initial distribution is just the Gaussian prior. This is also discussed in [2]. \n\nTherefore, I am concerned about the actual significance of starting from a non-Gaussian prior and whether SMC does all the heavy lifting for the performance of $\\Psi$-sampler. This creates additional computation overhead due to the use of MCMC (and one of the main novelties the paper claims is built upon this). This would certainly need more clarification and ablations. \n\nSome other minor issues are listed below:\n1. Missing some references in related work. I note that continuous time SMC-based diffusion posterior sampling algorithms have been proposed in [3] for image inverse problems, as well as in [4]. While this does not affect my evaluation of the contribution of this paper as these are concurrent, I think it's important to discuss connections to these works for a more comprehensive overview of the field to further improve readability.\n\n2. The introduction of eq (3), (4) might confuse readers. I think it's not emphasized enough that with eq (3), (4), the method **does not converge to the correct posterior distribution** even asymptotically when SMC has large number of particles. This is in contrast to the finetuning-based approach, which indeed recovers the correct distribution. The paragraph in appendix is much better for setting up the problem, and the approximation is more evident. I suggest moving that into the main text.\n\n**References**\n\n[1] Uehara, Masatoshi, et al. \"Fine-tuning of continuous-time diffusion models as entropy-regularized control.\"\n\n[2] Domingo-Enrich, Carles, et al. \"Adjoint matching: Fine-tuning flow and diffusion generative models with memoryless stochastic optimal control.\"\n\n[3] Chen, Haoxuan, et al. \"Solving inverse problems via diffusion-based priors: An approximation-free ensemble sampling approach.\"\n\n[4] Skreta, Marta, et al. \"Feynman-kac correctors in diffusion: Annealing, guidance, and product of experts.\"\n\nQUESTIONS:\nI have the following questions that I would appreciate your answers to. Answers to them are very important for me to maintain a positive support of this paper. Clarifications are certainly welcome if I have some misunderstandings.\n\n1. Related to my major concern: how different is the initial distribution used in $\\Psi$ sampler from pure Gaussian, and can you visualize them in some toy examples? \n\n2. Related to my major concern: how would the algorithm perform if we just start from a Gaussian prior and do the SMC resampling only? Ablations on this part are essential to understanding the core contributions of this work. While I think that doing SMC for reward alignment itself is already a significant contribution to the field, it's even more critical to determine what truly works. \n\n3. Related to my minor concern: how much bias would there be by using Tweedie's formula for approximation instead of learning a real value function? I understand that this is a somewhat standard practice in the literature, but I am curious about how much bias would be mitigated when combining it with SMC. It would be beneficial to have some numerical experiments discussing this, as SMC is known to be an asymptotically exact sampling algorithm.\n\nLIMITATIONS:\nyes, limitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the limitations of the existing SMC framework, i.e. the initial particles have to be coming from the standard Gaussian prior. This limitation will give very poor coverage over the high-probability region. Also, it can significantly impeded the sampling efficiency. Therefore, this paper proposes a new approach to put the initial particles in more likely regions through pCNL. This reward-informed initialization ensures that the particle set is better aligned with the target distribution from the outset, which will make the sampling more efficient and accurate. \nThe paper shows a crude approximation of Top-K-of-N method, which works surprisingly well. Therefore, the paper adopts this crude and simple approach to do SMC sampling on layout-to-image generation tasks. Empirical results show that the model is able to outperform the baseline posterior-based sampling methods like ULA, MALA, etc significantly.\n\nSTRENGTHS AND WEAKNESSES:\nStrength:\n1. The paper proposes a very solid approach, which has a significant technical contribution.\n2. The paper's theorem is highly sound.\n3. The paper's results also reflect the advantages of the proposed method.\n4. The paper is written quite well. The motivation and background has been written in details to help the readers understand the limitations of the existing methods.\n\nWeakness:\n1. It's not particularly clear why the three specific applications are chosen. Why don't the authors use widely adopted benchmarks like COCO-image generation or Flickr-image generation? Is there a reason to not do normal text-to-image generation evaluation and focus on the more specific ones?\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nThe paper is well written. The main limitation is that the bridge between the theorem and the application. The authors have not stated clearly why they chose the three specific evaluation tasks.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-03"}