Title: Linearly Constrained Diffusion Implicit Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Diffusion

Inverse Problems

Optimization

Projection
Abstract: We introduce Linearly Constrained Diffusion Implicit Models (CDIM), a fast and accurate approach to solving noisy linear inverse problems using diffusion models. Traditional diffusion-based inverse methods rely on numerous projection steps to enforce measurement consistency in addition to unconditional denoising steps. CDIM achieves a 10–50× reduction in projection steps by dynamically adjusting the number and size of projection steps to align a residual measurement energy with its theoretical distribution under the forward diffusion process. This adaptive alignment preserves measurement consistency while substantially accelerating constrained inference.
For noise-free linear inverse problems, CDIM exactly satisfies the measurement constraints with few projection steps, even when existing methods fail. We demonstrate CDIM’s effectiveness across a range of applications, including super-resolution, denoising, inpainting, deblurring, and 3D point cloud reprojection.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work introduces new techniques for solving linear inverse problems with diffusion model priors. The work extends the sample-then-project approach of diffusion posterior sampling with a multi-step projection approach guided by the distribution of the residual under the forward process. The authors show how to define appropriate stopping criteria for the optimization at each step and employ an efficient line search for speed. In their experiments, the authors show that their approach is competitive with other recent approaches to solving linear inverse problems such as inpainting and deblurring as measured by FID and LPIPS, while being efficient in terms of runtime.\n\nSTRENGTHS AND WEAKNESSES:\nStengths:\n\nThe runtime benefits of the proposed approach seem notable and are a significant improvement over some methods.The quality of the solutions are not necessarily state of the art, but appear to be quite good despite the fast sampling time. \n\nThe proposed heuristics for step-size and stopping criteria appear to be reasonably well-motivated, with some caveats. The experiments are using difficult real-world datasets and tasks with a wide variety of baselines\n\nThe background writing is clear and complete.\n\nWeaknesses:\n\nThe novelty is somewhat limited. The key update in equation 12 is the same as DPS and the idea of using multiple projection steps at each sampling time has been discussed in prior work, not cited here. In particular \"Constrained Diffusion with Trust Sampling\" for multiple step. \"Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models\" optimizes a version of equation 11, while combining with other techniques such as multi-step approximation and time-travel. Given the relationship between this method and those, they should be cited and how this method compares to and builds on them should be discussed.\n\nUnlike other extensions of DPS, such as PSLD (\"Solving Linear Inverse Problems Provably via\nPosterior Sampling with Latent Diffusion Models\") this approach doesn't seem to extend to latent diffusion models or general non-linear inverse problems, limiting its usefulness for many state of the art image generation models. The model used for imagenet experiments is somewhat outdated as model for natural images. \n\nMore qualitative examples of the results would be useful, particularly for more challenging tasks such as in-painting. I would like to be more confident the results are not cherry-picked.\n\nSome of the writing for the methods section is somewhat confusing. The notation for $\\hat{x}$ seems inconsistent, for instance in equations 10 & 11, its not clear if it's meant to approximate $E[x_0 | x_t]$ or $E[x_0 | x_{t-1}]$. It's slightly odd that equation 12 is not directly optimizing eq. 11, though this does end up being the same as DPS.\n\nQUESTIONS:\nI don't fully understand the paragraph from 145-150. Should the equation on line 147 be $\\hat{x}_0$? Could you shed some more light on why is this the best way to define the plausible region?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-21"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Conditional Diffusion Implicit Models (CDIM), a novel framework for solving linear inverse problems using pre-trained diffusion models. CDIM effectively addresses both noisy and noise-free linear inverse problems. By imposing constraints on the prior diffusion objective, it efficiently solves these problems in terms of both computational time and utility. The paper also incorporates techniques such as Early Stopping and an adaptive learning rate to facilitate more efficient convergence. Experimental results demonstrate that CDIM is fast, powerful, and straightforward to implement with pre-trained diffusion models, requiring no additional modules.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. At the heart of this work is a novel integration of the DDIM sampling procedure within an optimization-based framework, ensuring close alignment between the posterior mean and the observed data. \n2. The manuscript is clearly written, logically organized, and supported by comprehensive experiments. \n3. It also provided a analysis how  CDIM projection strategy both enforces measurement consistency and dramatically cuts down the number of diffusion‐model evaluations.\n\n\nWeakness: \n1. While the proposed method shows potential, its performance still lags behind state-of-the-art approaches such as DAPs [6], and it has not been evaluated on nonlinear inverse problems (e.g., nonlinear deblurring or phase retrieval).\n\n2. The experimental section omits comparisons with several recent diffusion-based sampling algorithms for inverse problems [1–7]. At minimum, the authors should benchmark against leading methods like DAPs [6]\n\n[1] Song, Jiaming, et al. \"Pseudoinverse-Guided Diffusion Models for Inverse Problems.\" ICLR 2023.\n\n[2] Song, Jiaming, et al. \"Loss-Guided Diffusion Models for Plug-and-Play Controllable Generation.\" ICML 2023.\n\n[3] Li, Xiang, et al. \"Decoupled Data Consistency with Diffusion Purification for Image Restoration.\" arXiv preprint arXiv:2403.06054 (2024).\n\n[4] Wu, Zihui, et al. \"Principled Probabilistic Imaging using Diffusion Models as Plug-and-Play Priors.\" arXiv preprint arXiv:2405.18782 (2024).\n\n[5] Xu, Xingyu, et al. \"Provably Robust Score-Based Diffusion Posterior Sampling for Plug-and-Play Image Reconstruction.\" arXiv preprint arXiv:2403.17042 (2024).\n\n[6] Zhang, Bingliang, et al. \"Improving Diffusion Inverse Problem Solving with Decoupled Noise Annealing.\" CVPR 2025.\n\n[7] Dou, Zehao, et al. \"Diffusion Posterior Sampling for Linear Inverse Problem Solving: A Filtering Perspective.\" ICLR 2024.\n\nQUESTIONS:\nPlease address the weakness of the previous part\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce CDIM, a method for solving noisy linear inverse problems using pretrained diffusion models. CDIM modifies DDIM sampling by projecting each step to enforce consistency with linear measurements. It handles Gaussian and Poisson noise, achieves exact recovery in the noiseless case, and performs well on tasks like inpainting and super-resolution. The authors conduct experiments across multiple linear inverse problems and 3D point cloud reprojection.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe paper is clearly written, well-motivated, and presents its core ideas in a structured and accessible manner. The proposed projection-based modification of DDIM sampling is easy to implement and allows for consistent reconstruction under linear constraints. \n\nWeaknesses:\nDespite these strengths, the contribution is somewhat limited in novelty. The primary reason for the speedup is the use of DDIM instead of DDPM, which is already well-established. Several strong baselines are missing, including variational approaches [1] and recent flow-based methods that currently define the state of the art. Some of the experiment, such as 4x super-resolution or sparse inpaintin are too simple and could be effectively addressed with classical or non-generative methods, limiting the practical relevance of the comparisons. Lastly, the method is restricted to pixel-space models due to its reliance on linear measurement operators, and cannot be applied in latent diffusion or nonlinear inverse problems.\n\n[1] A Variational Perspective on Solving Inverse Problems with Diffusion Models\n\nQUESTIONS:\nWhy are variational methods, such as RedDiff, and recent flow-based baselines omitted from the comparisons?\n\nHow does the method perform on more challenging settings like 8x SR?\n\nCan the approach be extended to latent diffusion models or nonlinear measurement settings, or is it inherently limited to pixel-space and linear operators?\n\nLIMITATIONS:\nThe paper does not discuss any limitations, despite the fact that the method cannot be applied to nonlinear forward operators.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe goal of this paper is to solve linear inverse problems using a pretrained generative model. While many existing methods in the literature (e.g., guidance-based or Plug-and-Play approaches) can tackle such problems, none of them explicitly enforce the linear constraint $Ax = y$ in the case of noiseless degradation. However, in certain applications—such as inpainting—it is meaningful to impose a hard constraint, since observed (non-masked) pixels should remain unchanged during reconstruction.\n\nThis paper aims to address this limitation by proposing an efficient method for solving inverse problems with hard constraints. The main contributions are:\n\n- An accelerated DDIM-based scheme for the prior denoising step.\n\n- A penalized formulation coupled with gradient descent to enforce constraint satisfaction.\n\n- Practical heuristics for step size selection during optimization.\n\n- Experimental results demonstrating competitive visual quality, faster computation compared to state-of-the-art methods, and better constraint satisfaction.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- The paper is well written and relatively easy to follow overall.\n\n- The numerical results are convincing, particularly in terms of computational speed-up compared to methods like DPS.\n\n- The acceleration of the DDIM scheme is an interesting and relevant contribution.\n\n\n**Weaknesses**\n\n- *Theoretical gaps*: The authors claim that their method satisfies the hard constraints and provide informal arguments in support. However, these arguments appear to be mathematically incorrect or at least imprecise. While the numerical results are more convincing, the theoretical justification remains weak.\n\n- *Lack of focus and clarity on contributions*: The paper seems to pursue two distinct goals:\n(1) enforcing hard constraints in inverse problems, and\n(2) proposing a fast and visually effective diffusion-based method for inverse problem solving.\nThe first objective is less convincing, given the theoretical limitations mentioned above. The second is more promising, but the paper lacks a clear analysis of which components of the method (accelerated DDIM, step-size heuristics, gradient descent updates) are most responsible for the performance gains. A more modular ablation or discussion would improve clarity on this point.\n\nQUESTIONS:\nFirst of all, I would like to thank the authors for this paper. I believe the two main questions addressed—hard constraints and acceleration—are relevant for the community. There are some interesting ideas in the paper, but I think the role of each component deserves further clarification.\n\n**1. On the empirical results and ablations**\n\nI appreciated the ablation study against DPS (Fig. 5). Do you have an explanation for why DPS performs poorly when combined with DDIM? Apart from the use of DDIM and the adaptive step size, is there any fundamental difference between your method and DPS?\n\nIn the same figure, the claim that “DPS doesn’t match the constraint” is not immediately obvious. You might consider reporting the numerical error on the constraint $Ax = y$ for each method.\n\nIt would also be interesting to highlight your method on box inpainting tasks, where other methods may suffer from hallucinations (i.e., inventing content in known regions). This would make the hard constraint enforcement more visible.\n\n**2. On the theoretical justification of constraint satisfaction**\n\nUnfortunately, I remain unconvinced by the theoretical arguments regarding constraint satisfaction:\n\na. Once the constrained optimization problem is replaced with a penalized one, there is little to no guarantee that the constraints will be exactly satisfied.\n\nb. The justification for the penalized formulation (footline on p.4) is questionable. If I understand correctly, the constraint $A \\hat{x_0}(x_{t-1}) = y$ should hold at $t = T$, not $A \\hat{x_0}(x_t) = y$. So the argument seems invalid as stated.\n\nc. Referring to Eq. (11) as a “Lagrangian” is incorrect. The formulation lacks dual variables and does not follow the standard Lagrangian structure.\n\nd. I went through the proof in Appendix C and found several steps questionable. For instance:\nHow do the assumptions in lines 336–337 imply that $x_t$ is bounded?\nI disagree with the claim made in line 348.\nThere seems to be a confusion in the status of $x_t$: sometimes it is defined by the diffusion model, other times it is treated as an optimization variable. In Eq. (37), for instance, $x_t$ is a variable being optimized, so the statement that $|A \\hat{x}_0 - x_t| \\to 0$ is not meaningful in this context.\n\ne. Have you considered using an Augmented Lagrangian approach? While this may come at a computational cost, it would provide a principled way to handle hard constraints.\n\n**3. Additional Baseline**\nIt would be useful to include a numerical comparison with Diff-PIR [1], which is a fast PnP method and may serve as a relevant baseline.\n\n**4. Additional suggestions and clarifications**\n\n- I could not fully understand Proposition 1. It seems the notations it relies on are only introduced later, making it hard to follow.\n\n- The Introduction and Related Work sections could benefit from further elaboration. In particular, it would help to provide more details about DPS, which is your main point of comparison.\n\n- To improve readability, I suggest consistently using the notation $\\hat{x}_0(x_t)$ to emphasize the dependence on $x_t$ in the reconstructed estimate.\n\n[1] Zhu, Y.; Zhang, K.; Liang, J.; Cao, J.; Wen, B.; Timofte, R. & Van Gool, L. Denoising diffusion models for plug-and-play image restoration, IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshop, 2023, 1219-1229\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces CDIM, a method for solving linear inverse problems through adaptive projection sampling (using a pretrained diffusion model). Authors show that under the forward noising process, the measurement residual follows a chi-square distribution, with tractable mean and variance. Using this, CDIM chooses projection step sizes and stopping criteria adaptively and projects $x_t$ just enough to keep residual within one standard deviation of its expected value at each step. \nIn the noiseless setting, this adaptive method guarantees exact constraint satisfaction and bounds the final reconstruction error with high probability. \n\nTheir empirical results show that CDIM achieves image quality on par with SoTA methods like DPS with 50x fewer model evaluations across multiple tasks on FFHQ and ImageNet, and some samples new applications like photo time travel and 3D point-cloud reconstruction.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The paper proposes a novel method for using pretrained diffusion models to solve inverse problems that uses an adaptive projection mechanism grounded in the residual distribution.\n2. The method achieves the same level of measurement fidelity with an order of magnitude fewer denoiser calls.\n3. Empirical results are provided on FFHQ and ImageNet datasets which are standard sets for these tasks.\n\nWeakness:\n1. In a couple of cases, CDIM shows perofrmance that is close, but not exceeding the best slower baseline. For example, DPS outperformed  on random inpainting and FPS-SMC outperformed CDIM on super resolution FID.\n\nQUESTIONS:\n1. Would it be possible to extend this method for latent-based models?\n3. How sensitive are final results to the choice of \"±1 σ\" as the stopping criterion? What if you choose a dynamic tolerance (e.g. varying with t) improve performance?\n2. Section D of appendix looks incomplete. Subsection D.1 has typos and inconsistency in notation (like \"64 ∝ 64\" and \"showcase a a 50% inpainting\"). Subsection D.4 should be referring to table 3?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}