Title: Training-Free Constrained Generation With Stable Diffusion Models
Venue: NeurIPS 2025
Accepted: yes
Keywords: Stable Diffusion

Constrained Generation

Differentiable Optimization
Abstract: Stable diffusion models represent the state-of-the-art in data synthesis across diverse domains and hold transformative potential for applications in science and engineering, e.g., by facilitating the discovery of novel solutions and simulating systems that are computationally intractable to model explicitly. While there is increasing effort to incorporate physics-based constraints into generative models, existing techniques are either limited in their applicability to latent diffusion frameworks or lack the capability to strictly enforce domain-specific constraints. To address this limitation this paper proposes a novel integration of stable diffusion models with constrained optimization frameworks, enabling the generation of outputs satisfying stringent physical and functional requirements. The effectiveness of this approach is demonstrated through material design experiments requiring adherence to precise morphometric properties, challenging inverse design tasks involving the generation of materials inducing specific stress-strain responses, and copyright-constrained content generation tasks. All code has been released at https://github.com/RAISELab-atUVA/Constrained-Stable-Diffusion.


Peer reviews:
comments: SUMMARY:
This paper presents a framework that integrates constrained optimization with stable diffusion models to enable constraint-aware data generation.  The proposed approach introduces a constraint formulation that can be incorporated into the diffusion sampling process, allowing for the generation of outputs that satisfy stringent physical, functional, and legal requirements while maintaining high synthesis quality. Two theorems provides convergence properties in theory. Through experiments, the authors demonstrate the framework's ability to handle complex constraint in stable diffusion models.

STRENGTHS AND WEAKNESSES:
The strengths are introducing a frameworks with constrained optimization in stable diffusion models.   The detailed proof of their convergence represent a non-trivial and significant research endeavor with both theoretical and practical implications.

QUESTIONS:
1. Is the speed of the methods depending on the complexity of the constraint? 
2. How to evluate the complex of the constraint? Is there some numerical value to measure that？

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-07-02

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a novel approach for constraining latent diffusion models, such as Stable Diffusion, based on constraints formulated in the original (e.g., image) space. Then, the constraints are projected via the decoder into the latent space of the diffusion model and subsequently enforced during each diffusion step via an inner optimization loop akin to projected gradient descent. It is important to stress that this is done after training during inference and the gradients with respect to the model parameters are not optimized. The constraints can even be possibly non-convex and not differentiable. To remedy situations where the constraints are not differentiable out of the box (e.g., where the constraint is the outcome of an experiment or simulation), the authors adopt existing approaches from literature, such as differentiable surrogates or smoothed approximated gradients via random local perturbations, to estimate the gradients of the constraints.\nThe method is quantitatively evaluated on three different domains: microstructure generation, metamaterial inverse design, and copyright-safe generation and compared against a text-conditioned diffusion model and a baseline method that enforces the constraints postpriori directly in the output/image space.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n\n- The paper is very well written and clearly structured.\n- The paper tackles a very timely and relevant problem in the field of generative models. The ability to infuse physical priors, constraints, or other guarantees into the generation process without sacrificing the quality of the generated samples (too much) is a very important capability (e.g., with respect to video-generation models) and this paper could have a significant impact on this domain.\n- The method is evaluated on problem settings from three different domains and the quantitative results are strong.\n\n## Weaknesses\n\n- Some of the figures are a bit unclear - also, because the captions are very brief and do not explain the figures in detail. Instead, a reader should be able to understand the figures (at least their gist) without having to read the main text. Examples include Figure 4 and Figure 5.\n- Although the two considered baseline methods are suitable and well-chosen, it could be beneficial to include additional baselines from related work, or, for example, a baseline that enforces the constraints during training. This would allow for a more comprehensive evaluation of the proposed method.\n- The computational complexity, efficiency and runtime of the method is not discuss in detail.\n\n### Detailed Comments on Minor Issues\n\n- Line 26: What is the difference between \"physical laws\" and \"first principles\"? Are physical laws not a subset of first principles? Please clarify.\n- Line 36: Although the authors later define how stable diffusion models operate in a learned latent space, at this point in the text it might not clear to a reader who is not familiar with latent diffusion models what a \"latent diffusion model\" is. One subsentence could be added to clarify this.\n- Figure 4: To the reviewer, it is not clear what is the meaning of the different rows in the \"Structural analysis\" section of the figure. Please define in the caption.\n- \"Differentiating through blackbox simulators\" lines 243-244: It seems a bit strange that there is no division of the equation by the (direction) of perturbation or something similar needed. Why is the case? Usually, you would have in finite differences something of the form $\\frac{f(x + \\epsilon) - f(x)}{\\epsilon}$ with $\\epsilon \\in \\mathbb{R}$, but here it seems like the authors are just using $f(x + \\epsilon) - f(x)$.\n- The heading of Appendix H \"Missing Proofs\" sounds a bit informal or even confusing. A more formal heading could be \"Supplementary Proofs\" or something similar.\n\nQUESTIONS:\n- How would training-time enforcement of the constraints compare to the proposed approach? Would it make sense to adopt a baseline approach that trains a model with the constraints enforced during training, and then compare the performance of the proposed method to that baseline? If not, why not?\n- For example in Equation (6), the authors introduce the trade-off between constraint satisfaction and maintaining similarity to the original learned distribution. Could instead of this soft constraint also a hard constraint be used? If so, how would this affect the performance of the method?\n- In line 237, the authors mention a smoothing function that is used. Where is the smoothing function defined? Is it just the sum operator?\n- Figure 8: Does the performance get even better when you perform more iterations/steps? Why did you stop at four steps?\n\nLIMITATIONS:\nAlthough the paper does discuss limitations in Appendix A, the reviewer would like to see a few-sentence summary of the limitations included in the main text of the paper (e.g., in the conclusion section).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 2, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a method of incorporating proximal projection optimization as a guidance method into latent (stable) diffusion sampling models, such that the samples from the model obey feasibility constraints. This is an important consideration when using stable diffusion models for generating high dimensional, physically plausible samples in a variety of scientific and engineering applications, etc. The contribution is in formulation and analysis of the method, and in a thorough empirical evaluation on real high dimensional problems.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The problem is interesting, relevant an impactful — and will help allow diffusion models to generalise to tackle new, high-dimensional problems in science and engineering that have feasibility constraints.\n- The contribution is clear and significant, and notable the authors supply a convergence analysis of their method, in terms of both the fidelity of the (constrained) generative distribution, and feasibility of generated samples.\n- Strong empirical/experimental evaluation on interesting end relevant problems, which back up the claims of the authors.\n- Clearly and well written for the most part.\n\nWeaknesses:\n- Some of the presentation of the method could be clearer to aid understanding — e.g. there is some confusing potential overloading of notation, and the algorithm could be explained more clearly. See the “questions” section for more detail on these points. \n\nOverall I found this paper a pleasure to read and review. It is interesting, and I think has decent originality and significance. The quality and clarity is generally high, though could be better with some small additional effort on the part of the authors.\n\nWith these clarifications, I think this is a worthy submission to NeurIPS 2025.\n\nQUESTIONS:\nQuestions:\n- Do all intermediate samples, $x_t$, have to obey the constraints, or just the final sample at $t=0$? Is the reason for constraining $x_t$ because it is too hard to propagate the constraints from only  $x_0$ to the latent space, or are the convergence rates worse, or is there another reason? It would seem to me that in some applications, feasibility of all $x_t$ would be a strong requirement that may be hard to satisfy.\n- I’m confused by the dimensionality of $\\mathbf{z}$ and the mapping $\\mathbf{g}$ in Section 4.1, and 4.2. It appears that in section 4.1 we require $\\mathcal{D} : \\mathcal{Z} \\to \\mathcal{X}$ in section 4.1 for the inputs to $\\mathbf{g}$, but in Section 4.2. Eqn. 6 this mapping is not used, and $\\mathbf{g}$ has inputs from the latent space? Are you referring to different distances, $\\mathbf{g}$ in these cases, or a different $\\mathbf{z}$, and the notation has been overloaded?\n- Inner and outer optimizer presentation confusing — I’m not actually 100% sure what these are referring to. Do you mean every diffusion step, $t$, the projection in (7) needs to be run, and then additionally, the equation after line 178 needs to be run for the inner Langevin sampling steps w.r.t. $i$? Or, is equation 7 implicitly being satisfied by the corrected Langevin sampling steps in the eqn after line 178? The latter seems to be implied by Algorithm 1, but it's not totally clear.\n- “Choosing $\\mathbf{g}$ as the indicator of a set, reproduces the familiar projection step introduced earlier” — This seems a little unclear to me, do you mean $g(y) = [[y \\in C]]$? Would this then have to be a complement given the arg min?\n- In Section 6.3 — do you need a projection path in, e.g. latent space like PCA? Or could you directly use the classifier and something like a Brier score as the $\\mathbf{g}$? You mention this as future work in appendix A, what is stopping the implementation of it now?\n\n\nComments:\n- Should there be no commas in the definition of $x_t$ before eqn. (1)?\n- reminds me of “Deep declarative networks” (https://arxiv.org/abs/1909.04866) where notes in a network can be the solution to an optimization problem (implicit nodes).\n- Algorithm 1: would be good to indicate $z^0_t$ comes from omitted sampling steps.\n\nLIMITATIONS:\nYes, the authors supply a nice comment on limitations in the Appendix.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses a critical challenge in diffusion models, that is, their inability to strictly adhere to domain-specific constraints. Existing diffusion models explore  constrained generation where diffusion takes place in original data space, however, that does not immediately translate to latent space. The authors propose a novel approach that incorporates proximal mappings into the reverse steps of stable diffusion models, allowing the generated content to conform to complex constraints. Their method does not require retraining and demonstrates strong performance across tasks such as porous-material synthesis, meta-material inverse design, and copyright-constrained content generation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe paper targets a practical and impactful problem. This method would improve applicability of latent diffusion models to critical domains where generation must adhere to string physical constraints. \nThe integration of proximal mappings with pretrained diffusion models without retraining is novel, practical an efficient, especially as it works with both convex and non-convex constraints. Authors also provie solid theoretical justification for their approach. \nThe approach is validated across diverse and challenging use cases, showing near-zero constraint violations and strong performance, suggesting generality and robustness\n\nWeakness\nFor black-box constraints, the approach estimates subgradients via finite differences, which may be noisy or inefficient in high-dimensional settings.\nHowever, authors compare their work with too few baselines, which limits the clarity on relative advantage.\nWhile the method is empirically shown to work for non-convex and non-differentiable constraints, strong theoretical convergence guarantees are provided only for convex constraints.\n\nQUESTIONS:\nQuestions Summarized in weakness\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-04"}