Title: Kernel Density Steering: Inference-Time Scaling via Mode Seeking for Image Restoration
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion Model

Image Restoration

Image Processing
Abstract: Diffusion models show promise for image restoration, but existing methods often struggle with inconsistent fidelity and undesirable artifacts. To address this, we introduce Kernel Density Steering (KDS), a novel inference-time framework promoting robust, high-fidelity outputs through explicit local mode-seeking. KDS employs an $N$-particle ensemble of diffusion samples, computing patch-wise kernel density estimation gradients from their collective outputs. These gradients steer patches in each particle towards shared, higher-density regions identified within the ensemble. This collective local mode-seeking mechanism, acting as "collective wisdom", steers samples away from spurious modes prone to artifacts, arising from independent sampling or model imperfections, and towards more robust, high-fidelity structures. This allows us to obtain better quality samples at the expense of higher compute by simultaneously sampling multiple particles. As a plug-and-play framework, KDS requires no retraining or external verifiers, seamlessly integrating with various diffusion samplers. Extensive numerical validations demonstrate KDS substantially improves both quantitative and qualitative performance on challenging real-world super-resolution and image inpainting tasks.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors present a method, called Kernel Densitiy Steering, to improve parceptual quality in diffusion models for image restoration. The method employs an ensmeble of particles to estimate a local approximation of the latent density at a given timestep. This alows to steer sampling towards a local density maximum, improving perceptual quality. A patchwise mechanism is used to avoid high-dimensional spaces which would make the method impractical.\n\nSTRENGTHS AND WEAKNESSES:\nStrenghts:\n- the paper is generally clear and well written\n- the method is well justified and appears to be mathematically sound\n- since it works at test-time, it does not require expensive retraining of state-of-the-art models\n- empirically, it clearly improves the baseline methods on standard benchmarks. Experiments present results on a large number of distortion and perceptual metrics.\n\nWeaknesses:\n- Reference [29] is a highly related method which seeks multiple perception-distortion tradeoff points with test-time iterations. Similarly to the authors work, it is a (progressive) mode-seeking strategy. The authors do not discuss similarities and differences of their work with respect to [29]. I think this point should be signficantly expanded. Moreover, since [29] essentially addresses the same problem, the authors should include experimental comparisons with it. At the current stage, it is unclear whether the authors' paper brings significant novelty or experimental improvements with respect to [29]. \n- An expanded discussion on computational complexity (and runtime experiments) would be useful to better understand the practical cost of the method\n\nQUESTIONS:\n- The experimental results seem to suggest that both distortion and perception metrics are improved by KDS. This is somewhat counterintuitive as it goes against the perception-distortion tradeoff principle, and does not happen in other works like [29] where improving one degrades the other. I wonder if the authors have an explanation for this? Could it be that the chosen baselines are rather weak? (e.g. [29] reports significantly better PSNR and FID numbers on DIV2K)\n- How complex is the proposed method in terms of runtime compared to the baseline without KDS?\n- It seems that 1x1 is the best patch size. Is this due to the curse of dimensionality for even moderately larger sizes or is it due to the low number of chosen particles? Would larger patches with a lot more particles work better (without caring for complexity)?\n\nLIMITATIONS:\nSome limitations have been addressed by the authors.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nA method for improving image restoration in diffusion models is proposed. The main idea is to use a Kernel Density Estimator (KDE) for steering the generation process towards higher likelihoods/modes to obtain higher quality restorations. The need for a large number of samples to evaluate the KDE is circumvented by evaluating on image patches. The method is evaluated on a toy dataset of a Gaussian mixture, where it is shown that generations lie close to the modes, and on real-world datasets, where image restorations are measured using standard metrics.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. The general idea is clearly explained\n2. The proposed method is computationally inexpensive\n3. Empirically, the proposed solution seems to bring some improvement over a baseline\n\n**Weaknesses:**\n1. My major complaint is twofold. First, as primary motivation, the authors aim to \"steer particles towards the modes\" (line 42) and \"steer diffusion sampling towards high-density\" (line 58). To me, an obvious solution to this problem is to utilize the already estimated score function. I do not understand why the authors consider the gradient of KDE instead of the learned score. It is not mentioned in the paper why this would be a better choice. Second, these two problems, i.e., \"mode-seeking\" and \"log-density control\", are already solved for diffusion models. Specifically:\n    1. [1] derives a principled ODE that tracks the modes in diffusion model sampling. Even Figure 6 in [1] looks very similar to Figure 2 in this submission.\n    2. [2] provides a principled ODE for sampling in diffusion models and controlling the log-density of the generated samples.\n    3. To make the connection very explicit: if the authors considered the score function already estimated by the model (instead of the KDE, which in my opinion is not necessary nor justified), the main equation in the paper (Equation 2) is equivalent to Equation 23 in [2], and Equation 18 in [1] (High-Density ODE) is its special case.\n    4. Furthermore, the proposed method requires tuning certain parameters to make the KDE work, such as the bandwidth $h$, or the sample size $N$, which are not required in [1] or [2].\n2. Statistical significance. In the submission checklist, the authors claim that they \"reported error bars suitably and correctly defined\". I do not think this is true. I would even claim the opposite. None of the quantitative results seem to have any error bars reported. None of the Tables: 1, 2, 3, 4 5, 6, 7, 8, nor Figure 6 have any error bars reported.\n\n---\n[1] Karczewski et al. \"Diffusion Models as Cartoonists: The Curious Case of High Density Regions\" (ICLR 2025)\n\n[2] Karczewski et al. \"Devil is in the Details: Density Guidance for Detail-Aware Generation with Flow Models\" (ICML 2025, first appeared on arxiv in February 2025 - not considered contemporaneous work)\n\nQUESTIONS:\n1. Why do authors use the KDE instead of the already estimated score function? Is this something that was not considered, or is there a reason that the authors believe this is a better approach?\n2. How does the method compare to using the known algorithms for mode-seeking [1] or log-density control [2]?\n3. Are the quantitative improvements over the baseline statistically significant?\n\nLIMITATIONS:\nThe authors discuss the limitations of their method, i.e., computational complexity, and the need for tuning hyperparameters.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 1, "ORIGINALITY": 1, "DATE": "2025-06-28"}

comments: SUMMARY:
This paper proposes to enhance the performance of image restoration by ensembling N particles of diffusion samples with kernel density steering.

STRENGTHS AND WEAKNESSES:
**Strengths**

- Qualitative results demonstrate the superiority of this method.
- The performance is enhanced by a test-time scaling method that uses KDE gradients to steer diffusion sampling trajectories.

**Weaknesses**

- Some aspects of the proposed methodology are not clearly explained and lack theoretical justification. (a) Why do the authors choose kernel density estimation over other distance metrics, such as a vanilla $\ell_2$ loss? (b) The abstract states that "the patches are steered towards shared, higher-density regions." A clearer explanation is needed as to why a higher-density region is beneficial for sample quality.
- The quantitative improvement appears marginal, e.g., the DISTS metric in Table 1.

QUESTIONS:
- There are several other kernel functions available, such as the Laplacian kernel. Is the Gaussian kernel superior to other kernels in this context? How do different kernels affect the final performance?

- What is the motivation behind using a patch-wise mechanism? Why don't the authors directly use the whole image for Kernel Density Steering (KDS)? I am concerned that a patch-wise mechanism might introduce seaming artifacts into the image.

LIMITATIONS:
See Weaknesses and Questions.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-01

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Kernel Density Steering (KDS), an inference-time framework that improves the performance of diffusion models in image restoration by using an N-particle latent ensemble. At each diffusion step, KDS performs patch-wise kernel density estimation on the predicted clean latents of the ensemble and applies mean-shift gradients to steer samples towards high-density modes. This improves the distortion–perception balance without the need for retraining or external verifiers. Extensive experiments on synthetic and real-world super-resolution and inpainting tasks demonstrate the effectiveness of KDS.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n\n- The manuscript is well-structured, logically coherent, and clearly written.\n\n- Novel ensemble-based local mode-seeking in diffusion sampling. The proposed Kernel Density Steering (KDS) provides a substantial novelty in inference-time guidance for diffusion-based image restoration, introducing an ensemble-based local mode-seeking mechanism without retraining.  KDS demonstrates a broadly applicable, plug-and-play enhancement that consistently improves distortion–perception trade-offs in super-resolution and inpainting, demonstrating clear impact potential.\n\n# Weaknesses:\n\n- The computational cost scales linearly with the ensemble size and there is no adaptive reduction mechanism.\n\n- The kernel bandwidth selection h is manually tuned; the author lacks an automated strategy.\n\n- I am wondering whether the KDS could be used for other inverse problems. For example, its applicability to dynamic tasks such as video super-resolution and medical imaging has not been explored.\n\nQUESTIONS:\n- Is it possible to develop an adaptive strategy for choosing ensemble size $N$ or patch dimensions based on noise level or sample distribution? Is it possible to integrate automatic bandwidth selection methods (e.g., plug-in estimators) to alleviate manual tuning of h?\n\n- Have you considered extending KDS to other inverse problems (e.g., denoising, deblurring, MRI/CT) or temporal tasks such as video restoration?\n\n- Could you compare or combine KDS with Best-of-N selection using non-reference metrics (e.g., NIQE, CLIP scores) to highlight relative advantages?\n\nLIMITATIONS:\nThe authors acknowledge the issues of computational overhead and bandwidth tuning. Further work on adaptive or approximate methods is encouraged.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}