Training-Free Constrained Generation With Stable Diffusion Models
Authors: Stefano Zampini
Jacob K Christopher
Luca Oneto
Davide Anguita
Ferdinando Fioretto
Venue: NeurIPS 2025 spotlight ()
Keywords: Stable Diffusion
Constrained Generation
Differentiable Optimization
Primary area: deep_learning
TL;DR: By incorporating differentiable constraint evaluation functions directly into stable diffusion’s iterative sampling, this approach enforces physical and functional requirements in generative tasks.

Abstract: Stable diffusion models represent the state-of-the-art in data synthesis across diverse domains and hold transformative potential for applications in science and engineering, e.g., by facilitating the discovery of novel solutions and simulating systems that are computationally intractable to model explicitly. While there is increasing effort to incorporate physics-based constraints into generative models, existing techniques are either limited in their applicability to latent diffusion frameworks or lack the capability to strictly enforce domain-specific constraints. To address this limitation this paper proposes a novel integration of stable diffusion models with constrained optimization frameworks, enabling the generation of outputs satisfying stringent physical and functional requirements. The effectiveness of this approach is demonstrated through material design experiments requiring adherence to precise morphometric properties, challenging inverse design tasks involving the generation of materials inducing specific stress-strain responses, and copyright-constrained content generation tasks. All code has been released at https://github.com/RAISELab-atUVA/Constrained-Stable-Diffusion.

Training-Free Constrained Generation With Stable Diffusion Models
Stefano Zampini" Polytechnic of Turin stefano zampiii @ polito.it
Jacob K. Christopher' University of Virginia csk4sr@ viisrr@iiiiii
Luca Oneto
Davide Anguita
Ferdinando Fioretto"
University of Genoa vica.onet@@unin unige.it
University of Genoa davide.anguita @ unige.it
University of Virginia fioretto virginia.edu
Abstract
Stable diffusion models represent the state-of-the-art in data synthesis across di- verse domains and hold transformative potential for applications in science and engineering, c.g.. by facilitating the discovery of novel solutions and simulating systems that are computationally intractable to model explicitly. While there is increasing effort to incorporate physics-basek constraints into generative models, existing techniques are either limited in their applicability to latent diffusion frame- works lack the capability strictly enforce domain-specific constraints. To address this limitation this paper proposes a novel integration stable diffusion models with constrained optimization frameworks, enabling the generation of out- puts satisfying stringent physical and functional requirements. The effectiveness of this approach is demonstrated through material design experiments requiring adherence to precise morphometric properties, challenging inverse design tasks involving the generation of materials inducing specific stress-strain responses, and copyright-constrained content generation tasks. All code has been released at http://gggttccooooa.oaaaaaaattta
Introduction
Diffusion models have emerged as powerful generative tools, synthesizing structured content from random noise through sequential denoising processes []] These models have driven significant advancements across diverse domains. including engineering 3 4 automation [ o chemistry
g and medical analysis ( 10 The advent of stable diffusion models has further extended these
capabilities, enabling efficient handling of high high-dimensional data and more complex distributions
This scalability makes stable diffusion models particularly promising for applications in science and engineering, where data is highly complex and fidelity is paramount.
However, despite their success in generating coherent content, diffusion models face critical limitation when applied to domains that require outputs to adhere to strict criteria. In scientific and engineering contexts, generated data must go beyond merely resembling real-world examples; must rigorously comply with predefined specifications, such as physical laws, safety standards, or design constraints grounded in first principles. When these criteria are not met, the outputs may become unreliable, unsuitable for practical use, or even hazardous, undermining trust in the model's applicability. Bridging this gap is crucial for realizing the potential of diffusion models in scientific applications.
"qqual contribution.
CContcct author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent research has reported varying success in augmenting model training with (often specialized classes of) constraints and providing adherence to desired properties in selected domains [2 Many of these methods, however, are restricted to simple constraint sets or feasible regions that can be easily approximated, such as a simplex, 1.2-ball, or polytope. These assumptions break down in scientific and engineering tasks where constraints may be non-linear, non-convex, or even non non-differentiabl.. Others are fundamentally limited, as training- enforcement offers only distribution-level adherence to constraints rather than per-sample guarantees, even in convex settings, and cannot generalize to unseen or altered constraints without retraining. While inference-ting enforcement methods address these shortcomings, these approaches modify the reverse processes in the original data space [5] 16]. This makes them incompatible with diffusion models like Stable Diffusion, which operate on learned lower- dimensional representations of the data. Latent-space variants have begun to appear, but they rely on special measurement operators [ or on learned soft penalties (20 21, thus limiting general applicability to the problems of interest in this work.
We address this challenge by integrating a proximal mappings into the reverse steps of pretrained stable diffusion models. The paper makes the following contributions: At each iteration, the generated latent is adjusted with gradient descent step on the score field followed by a proximal update for constraint correction, all without retraining the network. When the constraint set is convex we prove that every iterate remains in the feasible region and that the Markov chain converges almost surely to feasible point. The same algorithm, kept unchanged, extends to strongly non-convex constraints and even to constraints that can be checked only through a black-box simulator by estimating stochastic subgradients with finite differences. Experiments on (i) porous maturiaattrril synthesis with exact porosity and tortuosity, (ii) meta-material inverse design that matches target stress-strain curves through finite-element solver in the loop, and (iii) content generation subject to copyright filters show near-zero violations and state-of-the-ar performance in constrained generation.
2 Preliminaries: Diffusion Models
Score-Based Diffusion Models 22 23
learn data distribution by coupling a noising (forward) Markov chain with a learned denoising (reverse) chain. Let {xx) 1T be the sequence of diffusion states with æp drawn from the original data distribution P In the forward process, Gaussian noise is added according to a fixed variance schedule ū₂ where 0--1 S (x,, so that the learned marginal q(x") approaches (V as & + T. A score network sp(x;, E) is trained learn the score function sy [Z,, t)- = vv, log ((, (z)) z Vz, log p( where the approximation holds under the assumption that q(x,, (20)) is close proxy for the true distribution ((0) The network is trained by applying noise ~ /N I) perturbations to the Zp. such that It = Vā + /1 - āzc, with the training objective minimizing the predicted error in the score estimate of the noisy samples:
min 0 [1,T,, Pestss, ((0,) [|so(zz,, t) - Va, log (xx 20)1
E
In the reverse process, the trained score network so f) is used to iteratively reconstruct data samples from the noise distribution p(ær). At each step t, the model approximates the reverse transition, effectively reversing the diffusion process to sample high-quality data samples. Notably, Denoising Diffusion Probabilistic Models have been shown to be mathematically equivalent 2
the same scheme operates in a compressed latent space. The architecture uses an encoder-decoder pair E and D. where the encoder & maps the high -igh-dmmnssionl image data to a latent space, denoted z,, and the decoder D reconstructs the final image from the latent space after the diffusion model has operated on it. The training objective remains consistent with Equation ( with the exception that noise is applied to Zu = (xxa) as opposed to directly to Ty as in ambient space diffusion models. The latent diffusion model is thus trained to denoise over the latent space as opposed to the image space. Notice, however, that training the denoiser does not directly interact with the decoder, as the denoiser's loss is defined over the latent space and does not connect to the finalized samples. This consideration is relevant to the design choice taken by this paper the proposed solution, discussed in Section 4 After iterative denoising, the final sample can be obtained by decoding zo with D.
Stable Diffusion. In latent-diffusion variants such as Stable Diffussion 24 25),

---

3 Projected Langevin Dynamics
Score-based diffusion models sample by running annealed Langevin dynamics, a form of Langevin Monte Carlo adapted to multiple noise levels. At each level t, one takes M iterations of
(++1) m!! + s0(a (6) + 2 6 6~ c,,
where s0(2; t) R v log ((x) is the fixed, learned score and 7/ is step size. This update performs stochastic gradient ascent the learned log-density log g(x) with added Gaussian noise [26]. This optimization is performed either directly [22] or as the corrector step within a predictor-corrector framework [23]; ; in both cases, the result is approximate gradient ascent the density function, subject to noise. Provided this understanding, Christopher et al. [15] showed that for a constraint set
C, enforcing æt (i+1) E C' turns sampling into series of constrained problems.
Specifically, this framing shifts the traditional diffusion- based sampling procedure into a series of independent, per-timestep optimization subproblems, each responsible for denoising a single step while enforcing constraints:
zt (++1) Pc
= (2c(2,) (at 2i + V a;
log (x0) + V2
where the projection operator Pc(#)) = yec |yy - æ1 returns the nearest feasible sample. Note that Langevin dynamic annealing occurs in an external loop decreasing the noise level across t t= ...... 1. so that noise level e and step size "/e remain fixed inside each subproblem. As A - 0 the noise vanishes and the Langevin dynamics converge to deterministic gradient ascent on the learned density function. Note also that annealed Langevin dynamics is known to reach an e-stationary point of the objective with high probability (27). In diffusion processes, the negative learned density function, log q( (a, (zo). serves the objective, so Equation 2 can be viewed as projected gradient descent on this landscape. The results of [27] extend non-convex settings, giving further theoretical support to the constrained sampler adopted here.
While this approach is applicable when diffusion models operate across the image space, it cannot be directly adapted to the context of stable diffusion as C cannot be concretely represented in the latent space where the reverse process occurs. Other works have attempted to impose select criteria on latent representations, but these methods rely on lcarning-base
approaches that struggle in out-of-distribution settings (20 [20 [1], making them unsuitable for scenarios requiring strict constraint
adherence. This limitation likely explains their inapplicability in the engineering and scientific applications explored by 15.
4 Latent Space Correction
Addressing the challenge of imposing constraints directly in the latent space hinges on key insight: while constraints may not be representable in the latent domain, their satisfaction can be evaluated at any stage of the diffusion process. Indeed, the decoder, DD facilitates provides a differentiable transformation from the latent representation to the image space, where constraint violations can be directly quantified. Therefore, when the constraint function is differentiable, or even if its violations can be measured via a differentiable mechanism, gradient-basee methods can be used to iteratively adjust the latent representation throughout the diffusion process to ensure constraint adherence.
4.1 Imposing Ambient Space Constraints on the Latent
This section begins by describing how constraints defined in the ambient space can be imposed on the latent during the denoising process. First, following the constrained optimization framework discussed in Section B] at each noise level we recast the reverse process for Stable Diffusion as the following constrained optimization problem:
minimize TT.... 21
Σ
- log ((ze(z)
s.t.: g(D(z.)) = 0,
where D maps the latent representation vector-valued function g Idd > [0, oo) measuring the distance to the constraint set C. At each iteration of the diffusion process, our goal is to restore fcasibility with respect g.
into its original dimensions and g is a differentiable
3

---

As the constraint function can only be meaningfully represented in the image space, its gradients with respect the latent variables are computed by evaluating the function on the decoded representation D. This process is facilitated by the computational graph:
2 D(z+) = 21 + g( = inf l zz|l-
V
This enables iterative updates that reduce constraint violations by backpropagating gradients from the constraint function directly to the latent representation as,
V2, (DD/dz)) Va.g.
Thus constraint information, computed where it is meaningful (image space), can be back-propagated through the frozen decoder to steer the latent variables. The method leaves the score network and the decoder unchanged, adds no learnable parameters, and enables feasibility enforcement every step even when C' is non-convex or specified only by a black-box simulator.
4.2 Proximal Langevin Dynamics
The representation of latent diffusion constrained optimization task enables the application of established techniques from constrained optimization. To this end, this section discusses a generalization of the orthogonal projections used in projected Langevin dynamics, formulated through a Proximal Langevin Dynamics scheme. Let a constraint be encoded by a proper, lower- semicontinuous convex penalty Rd - [0, oo] whose zero set coincides with the feasible region. After each noisy ascent step we apply proximal map:
(++1) T
prox Ag
+ 7/ (0 log ((a,, mn) + v 22
(5)
Langevin Dynamics Step
with the proximal operator defined as:
prox (xx) = arg min {g(y) + 2x y xxr13]
(6)
This operator balances maintaining similarity to the updated sample and adhering to the constraint function g as weighted by hyperparameter X. Choosing g as the indicator of a set, reproduces the familiar projection step introduced earlier, but the proposed Proximal Langevin Dynamics also accom- modates non-smooth regularizers, composite penalties, and constraints specified only through inner optimization subroutines (0]. Thus, because proximal maps can be evaluated efficiently even for implicit or geometrically intricate constraints, Proximal Langevin Dynamics extends Langevin-basee sampling settings where explicit projections are impractical while preserving the convergence guarantees of the original scheme. The next results provides a characterization on convergence to (i) the constraint set and (ii) the original data distribution Pdata-
Theorem 4.1 (Convergence to the Constraint Set). Let C be non-empty and (i-pro--rgullrr in the
sense of 137), Def. 13.27, the score network satisfy || a, log g(x,)] S G (a standard consequence of the bounded-dan data domain after normalization), and 'D is E-Lipschitz such that (V z) S É. Then, for positive step sizes Tt S Z02.. B, the the following inequality holds for the distance to C:
(non-asymptotic feasibility) where 2i is the pre-proximal mapping iterate, g is L-smooth, B' = B/((L), and dist(zi, C) is the distance from z the feasible sel C.
dist (ist), < <1 2,3 ve)) (D(tt(D()))))) C)²² + G2,
Theorem 4.1 illustrates that the distance to the feasible set C' decreases at a rate of 1 - 2,8' 7++1 each step (up to an additive G2 noise). Thus, the iterates converge to an e-feasible set in log(3 {)) steps, with Tmnin min, ?t- This is experimentally reflected in Section 6.1 where C is convex and all samples converge to the feasible set.
Tumin
Theorem 4.2 (Training Distribution Fidelity), Suppose the assumptions stated in Theorem 4.2 are satisfied. Then, for positive step sizes "It, S 205 B, the following inequality holds for the Kullback- Leibler (KL) divergence from the data distribution:
(L(g(z(1)) Pdata) S (L(((z)) IPddtta) + G2G
(fidelity)
4

---

The inequality in Theorem
4.2 shows that the divergence from the training data distribution increases
3) at each step and, consequently the cumulative divergence from E 7h) and thus vanishes as - O. Consequently, after at most ) steps the expected constraint violation drops below while their KL divergence from the original data distribution grows at most linearly in Er " (negligible because Yt -00 along the chain). This implies that our approach inherits the same sample quality guarantees as unconstrained diffusion, up to a tunable drift bounded by ""m (with Ymax = max; 7:), and, since 7t2 along the chain, this drift is negligible in practice. This result provides theoretical rationale for our method's comparable FII) scores to unconstrained baselines for the image generation Proofs for both theorems provided in Appendix HH
by at most 2 the training distribution 2012
& (since 7y I
T - 28-mni log(
(iist(zr,.))) (
tasks in Sections 6.1 and 6.3
4.3 Training-Fres Free Correction Algorithm
With the theoretical framework of our approach established, we are now ready to formalize the proposed training-free frree algorithm to impose constraints on zz. The algorithm can be decomposed into an outer minimizer, which corrects 2, throughout the sampling process, and an inner minimizer, which solves the proximal mapping subproblem. The former describes high-level view of the entire constrained sampling process, whereas the latter presents the single-step uub-optimization procedure.
Outer minimizer. First, to impose corrections throughout the latent denoising process Equation (6) is modified to accommodate the 'D mapping:
prox Ag (z.) = arg min (g(D(y)) + 2\ (D( D(z
At each sampling iteration, we first compute the pre-projection update z/ = Ze + yt Zz, log q(z ((zz -
V2nu which incorporates both gradient and stochastic noise terms. We then apply proximal operator to obtain the corrected latent 2, = prox (zt'). ). This follows the Proximal Langevin Dynamics scheme outlined in the previous section.
Inner minimizer. Each proximal mapping is composed of a series of gradient updates to solve the outer minimizer. In practice, gradient descent is applied on the proximal operator's objective:
i+1
i Vz; (=((E()()) + 1 (D())) - D(2
b
When a convergence criterion met (e.g., (z)) < 8), the algorithm proceeds to the next denoising step. [esorithm[[prviiiie proximal operator within the stable diffusion sampling process.
pseudo-code for this gradient- approach to applying the
In the case that is an indicator function,
Algorithm 1: Sampler with Constraint Correction
0,
VEC,
Input: 8 (violation tolerance). (learning rate)
g(y)
DO, y C
Define prox_objective(rz))
the proximal mapping reduces to a projection onto the constraint set. If Pe can be formu- lated in the ambient space, the minimization be fully imposed the projection objec- tive, ((Pc(D().) - (2z,) )12. Notably, the gradients of this objective capture both the constraint violation term, g( g(D(z,)). and the
violation + g(x;);
distance 2 | x 12:
return violation + distance;
for / < to do
distance term, 2/ ( D(z,)) Il within the prescribed tolerance, resulting in the so- lution to Eq. 77 the end of the minimiza- tion. When C is convex, the projection can be constructed closed-form (Section 66.1) but otherwise Lagrangian relaxation can be employed (32] For our experiments with non-convex constraint sets (Sections and and
// Sampling steps (omitted).
i ++ 0; while g(D(z)) 2 8 do
g V v proprox_ooo pprox_objecivee(((e)))
2 1-1 t 2 - (g) i+i+l:
6.3) we leverage an Augmented Lagrangian
Output: D(z))
re relaxation described in Appendix D 33
5

---

Importantly, in these cases the corrective step can be considered a projection of the latent under appropriate smoothness assumptions. We justify this claim with the following rationale: under the assumption of latent space smoothness [4]. the nearest feasible point in the image space closely corresponds to the nearest feasible point in the latent space. Therefore, while the distance component of the proximal operator, 2X (x()) D(2() is evaluated in the image space, minimizing this distance in the image space also minimizes the corresponding distance in the latent space, validating the use of this corrective step as a latent projector.
Finally, when g cannot be represented by differentiable function, such as when the constraints are evaluated by an external simulator (as in Section 6.22) or when the constraints are too general to represent in dosed-fform (as in Section 6.3), it is necessary to approximate this objective to Equation (7 using other approaches. We discuss this further in the next section and empirically validate such approaches in Section 6
5 Complex Constraint Evaluation
While in the previous section we discuss how to endow mathematical properties within stable diffusion, many desirable properties cannot be directly expressed as explicit mathematical expressions. Particularly when dealing with physical simulators, heuristic-based analytics, and partial differential equations, it becomes often necessary to either (i) estimate these constraints with surrogate models or (ii) approximate the gradients of black-box models. To this end, we propose two proxy constraint correction methods that leverage differentiable optimization to enforce constraints.
Differentiable surrogates. Surrogate models introduce the ability to impose soft constraints that would otherwise be intractable. Specifically, we replace g(x;), the constraint evaluation function used in the optimization process, with constraint violation function dependent on the surrogate model (e.g., distance function between the target properties and the surrogate model's predictions for these properties in æ, as shown in Section 6.3p. This allows the surrogate to directly evaluate and guide the samples to adhere to the desired constraints at each step. Apart from this substitution, the overall algorithm remains identical to Algorithm 1. Through iterative corrections, the model converges to corrected sample 2, that satisfies the target constraints to the extent permitted by the surrogate's 's predictive accuracy. Appendix provides an in-depth view of the differences of this approach with respect to classifier guidance.
Differentiating through black-box simulators. some cases (including the setting of our meta- material design experiments in Section 6.2p. strong surrogate model cannot be leveraged to derive accurate violation functions. In such settings, the only viable option is to use a simulator to evaluate the constraints. However, these simulators are often non-differentiable, making it impossible to directly compute gradients for the proximal operator. To incorporate the non-differentiable simulator into the inner minimization process, this paper exploits sensitivity analysis method inspired by the differentiable perturbed optimizer (DPO) adopted the context of differentiable optimization [35] 36
DPO introduces controlled perturbations the optimization variables and smooths the resulting objective, yielding differentiable surrogate gradients. Following this idca, random local perturbations injected into the simulator inputs and define smoothed function Φv (x,) E, [Ø(x,, + ve)], where & is the external simulator, e ~ NoO,, 7) is a random perturbation, and v is a temperature parameter controlling the smoothing scale. An unbiased Monte Carlo estimate of J. is obtained as
M
(x,))
Σ ds ((., + (m))
M
m==1
where M is the number of perturbed samples. The gradient of this smoothed objective can be written
1 V N., (xx) E[o(x, + ve)
which corresponds to a finite-difference estimator whose scaling term 1/v is absorbed into the proximal step size during optimization. Finally, this smoothed estimator enables a differentiable loss formulation:
Vz, ((o(x.) = - (x. - target)
which is used update the latent variable z, (via H, D(z))) through proximal Langevin dynamics.

---

Generative Methods
Voids diameter distribution
Ground
P(%)
Cond
PDM
(Ours)
P 30%
P P= 50%
Ground
Ground
Lavent
Lanent
30
(umi((um)
50
MSE w.r.t. Ground
FID scores:
10,8++0.9
30.7=68
13.5433.1
Cond: 1.58
Cond: 0,31
error 10%: 68.4% 124
0%00
0%
Ours: 0.47
Ours: 0.12
Figure 1: Comparison of model performance in terms of FID score and constraint satisfaction (percentage of samples that does not satisfy the target porosity with a margin of 10%).
Figure 2: Distribution of void diameters in the training set (Ground) and in data generated by Conditional diffusion model and Latent Con- strained Diffusion models.
Experiments
The performance of our method is evaluated in three domains, highlighting its applicability to diverse settings. Supplementary results and baselines are discussed in Appendices and
Baselines. In each setting, performance is benchmarked against a series of baselines.
Conditional Diffusion Model (Cond): To assess the contribution of latent diffusion itself, we include a reference baseline consisting of an identical Stable Diffusion with text-conditioned constraints guidance [37].
2. Projected Diffusion Models (PDM): Following 1. this approach enforces feasibility by
projecting the constraint set each denoising step in the image space. PDM represents the current state-of-the-ar for constrained generation in Sections 6.1 and 6.3
3. Bastek and Kochmann (2023): For the task in Section 6.2 where PDM is inapplicable, we compare with this specialized method, which constitutes the state-of-the-ee in that domain [38].
Collectively, these baselines capture the strongest existing constrained-gencee -eneration methods, enabling a clear assessment of the improvements introduced by our latent constrained diffusion framework.
6.1 Microstructure Generation
Microstructure imaging data is critical in material science domains for discovering structure- structure-property linkages. However, the availability of this data is limited on account of prohibitive costs to obtain high- resolution images of these microstructures. In this experiment, samples subject to a constraint on the porosity levels of the output microstructures. Specifically, the goal is generate new microstructures with specified, and often previously unobserved, porosity levels from a limited dataset of microstructure materials.
task the model with generating
For this experiment obtain the dataset used by 15 39]. Notably, there are two significant obstacles to using this dataset: data sparsity and absence of feasible samples. To address the former limitation, we subsample the original microstructure images to generate the dataset using 64 x 64 images patches that have been upscaled to 1024 x 1024. To the latter point, while the dataset contains many samples that fall within lower porosity ranges, it is much constraining the porosity these cases, often no feasible samples exist at given porosity level.
sparse at higher porosities. Hence, when
Inner minimizer. To model the proximal operator for our proposed method, we use projection operator in the image space and optimize with respect to this objective. Let an be the pixel value for row and column j. where f E [11,1]] for all values of and j. The porosity is then,
porosity
Σ
1( < 0),

---

where 1() is the indicator function, which evaluates to 1 if the condition inside holds and 0 otherwise. We can then construct projection using a top-k algorithm to return,
Pa = argmin yn
Σ
11
S.t. Vy [-1,1
1Σ K
where K is the number of pixels that should be "porous". Importantly. since the above program convex, our model provides certificate on the satisfaction of such constraints in the generated materials. We refer the interested reader to Appendix G for additional discussion.
Results. A sample of the results of experiments is presented in Figure [ Compared to the Projected Diffusion Model (PDM), latent diffusion approaches show a sigmificant improvement. Latent diffusion models enable higher-quality and higher-resolution images. The previous state- of-the-art PDM, which operates without latent diffusion, had an FID more than twice as high as the models incorporating latent diffusion. Alternatively, the Conditional Diffusion Model, utilizing text-to-image conditioning, reports the best average FID of 10.8 but performs poorly with regard to other evaluation metrics. Conditioning via text prompts proved unsuitable for enforcing the porosity constraints, and on average, only 31.6% of the samples had porosity error less than 10%, indicating that this method lacks reliability in constraint satisfaction despite its ability match the training distribution. Furthermore, as shown in Figure 2 the conditional model performs significantly worse than our method on producing realistic microstructures.
Fraction of physically invalid shapes [L.
Model
MSE [1]
Cond
7.1 4:45
55%
Bastek and Kochmann
6.4
20%
Latent (Ours)
1.4:1
5%
Figure 3: Compare MSE w.r.t. target stress-strain response and rejection rate of physically inconsis- tent shapes.
In contrast, our Latent Constrained Model ex- hibits the most optimal characteristics. The proposed method satisfies the porosity con- straints exactly, achieves an excellent FID scores, and provides the highest level of microstructure realism as assessed by the cuuristic-based analysis. This indicates that our approach effectively balances constraint satisfaction with high-quality image genera- tion. This is a significant advantage over ex- isting baselines, as the method ensures both high-quality image generation and precise ad- herence the physical constraints.
Original
Step O
Step 2
Step
i
Structural analysis
increasing stress
6.2 Metamaterial Inverse Design
Now, we demonstrate the efficacy of our method for inverse-design of mechanical meta- materials with specific nonlinear stress-strain behaviors. Achieving desired mechanical re- sponses necessitates precise control over fac- tors such buckling, contact interactions, and large-strain deformations, which are inher- ently nonlinear and sensitive to small paramet- ric variations. Traditional design approaches often rely trial-and-ens -error methods, which are time-consuming and may not guarantee optimal solutions.
Stress-strain curves
Torget
Original
-Latent (Oum)
MSE
Specifically, our task is to generate mechani- cal metamaterials that closely match a target stress-strain response. From 388 we obtain a dataset of periodic stochastic cellular struc- tures subjected to large-strain compression. This dataset includes full-field data capturing
179.5
175.6
12.5
1.2
Figure 4: Successive steps of DPO. The sample is iteratively improved and the stress-strain curve aligns with the target. Structural analysis shows progressive deformation under controlled compression.
8

---

complex phenomena such as buckling and contact interactions. Because the problem is invariant with respect to length scale, the geometric variables can be treated as dimensionless. The stress is expressed in megapascals (MPa).
Inner minimizer. Exact constraint evaluation requires the use of an external, non-differentiable simulator Φ. Hence, we employ DPO as described in Scction[Sto facilitate the proximal mapping with respect the signals provided from this module. To compute the ground truth results for the stress-strain response, we employ Abaqus A0, using this simulator both for our correction steps and for validation of the accuracy of the generations. For this implementation, we set the number of perturbed samples M = 10, finding this provides strong enough approximations of the gradients to converge to feasible solutions.
Results. We illustrate the DPO process for our Latent Constrained Model in Figure 4 Firstly, note that our method facilitates the reduction of error tolerance in our projection arbitrarily low levels. By reducing the tolerance and performing additional DPO iterations, we can tighten the bound on the projection operator, thereby enhancing its accuracy. Moreover, the integration of the simulator enables the model to generalize beyond the confines of the existing dataset (see Figure 7
Due to the complexity of the stress-strain response constraints in this problem, other constraint-aware methods (i.e. Projected Diffusion Models) are inapplicable, and, hence, our analysis focuses on the performance of Conditional Diffusion Model baselines. We compare to (1) unconstrained stable diffusion model identical to the one used for our method and (2) ofate-of-the-art method proposed by Bastek and Kochmann (38), which operates in the ambient space. While our approach optimizes samples to arbitrary levels of precision, we observe that these baselines exhibit high error bounds relative to the target stress-strain curves that unable be further optimized. As shown in Figure 3 with five DPO steps our method provides a 4.6x improvement over the state-of-the-art model by [38] and a 5.1x improvement over the conditional stable diffusion model MSE between the predicted structure stress-strain response and target response. These results demonstrate the efficacy of our approach for inverse-design problems and generating samples that adhere to the target properties.
6.3 Copyright-Safe Generation
Next, we explore the applicability of the proposed method for satisfying surrogate constraints. An important challenge for safe deployment of generative models is mitigating the risk of generating outputs which closely resemble copyrighted material. For this setting, pretrained proxy model is fine-tuned to determine whether the generation infringes upon existing copyrighted material. This model has been calibrated so that the output logits can be directly used to evaluate the likelihood that the samples resemble existing protected material. Hence, by minimizing this surrogate constraint function, we directly minimize the likelihood that the output image includes copyrighted material.
To implement this, we define a permissible threshold for the likelihood function captured by the classifier. A balanced dataset of 8,000 images is constructed to fine-tune the classifier and diffusion models. Here, we use cartoon mouse characters 'Jerry' from Tom and Jerry, and character 'Mickey Mouse'. When fine-tuning the diffussin model, we do not discriminate between these two characters, but the classifier is tuned to identify 'Mickey Mouse' as a copyrighted example.
Inner minimizer. Our correction step begins by performing Principal Component Analysis (PCA) on the 512 features input to the last layer and selecting the two principal components. This analysis yields two well-defined clusters corresponding to the class labels. Provided this, the inner minimizer differentiates with respect to a projection of the noisy sample onto the centroid of the target cluster, as illustrated in relaxation of the projection, terminating the iterative optimization based on proximity to the 'Jerry' cluster. Empirically, we found it was only necessary to trigger this proximal update if the classifier assigns a high probability to the sample being 'Mickey Mouse" at a given step.
Given the complexity of the constraints, we define Lagrangian
Results. Figure 5 ((botoo-rriht) reports the FID and constraint satisfaction, and Figure 5 (top-left) compares the evolution of the original sample and corrected sample. We implement a Conditional Diffussion Model baselines using and unconstrained stable diffusion model identical to the one used for our method. The conditional baseline generates the protected cartoon character (Mickey Mouse) 33% of the time, despite conditioning it against these generations. The Projected Disfussio Model also struggles in this domain, particularly due to the maage-space architecture's 's inability to handle high dimensional data as effectively as the latent models; it failed to generate reasonable samples at

---

Denoising process
Projection Path
25%
50%
75%
100%
Cond
Model
Constraint [1]
FID [v]
Latent (Ours)
Cond
%
61.2
PDM
%
75.3
Latent (Ours)
%
65.1
Figure 5: Left: Denoising process of Cond vs. Latent (Ours). Out method drives the denoising toward a copyright-safe image. Top-right: Showing projection from original (O) to projected (P) in the PCA-2 space. Bottom-right: Constraint satisfaction and FID scores.
a resolution of x024x 1024, unlike the latent models which maintained image quality at this scale. As result, we were constrained to operate at much lower resolution of 64x64 during sampling, subsequently relying on post-hoc upscaling techniques to reach the target resolution. This led to higher FID scores and only marginal improvement over the conditional model's constraint satisfaction. Conversely, our Latent Constrained Model only generates the protected cartoon character 10% of the time, aligning with the expected bounds of the classifier's predictive accuracy. Our method has proven to be highly effective because it preserves the generative capabilities the model while imposing the defined constraints. The FID scores of the generated images remain largely unaltered by the gradient- based correction. This demonstrates that our approach can selectively modify generated content to avoid copyrighted material without compromising overall image quality.
7 Conclusion
This paper provides the first work integrating constrained optimization into the sampling process of stable diffusion models. This intersection enables the generation of outputs that both resemble the training data and adhere to task-specific constraints. By leveraging differentiable constraint evaluation functions within a constrained optimization framework, the proposed method ensures the feasibility of generated samples while maintaining high-qualit synthesis. Experimental results in material science and safety-critical domains highlight the model's ability to meet strict property requirements and mitigate risks, such as copyright infringement. This approach paves the way for broader and more responsible applications of diffusion domains where strict adherence to constraints is paramount.
Contributions
FF and JC conceived the idea and developed the initial methods. SZ and JC implemented the algo- rithms, contributed to discussions, and refined the research direction. SZ conducted the experiments and analysis, while JC contributed theoretical development and additional experiments. FF pro- vided overall guidance and coordination. JC, FF, and SZ co-wrote the paper. L.O and DA funded SZ's visit to FF's lab.
Acknowledgments
The material is based upon work supported by National Science Foundations (NSF) awards 2533631, 2401285, 2334448, and 2334936, and Defense Advanced Research Projects Ageney (DARPA) under Contract No. #HR001 1252E005. The authors acknowledge Research Computing at the University of Virginia for providing computational resources that have contributed to the results reported within this paper. The views and conclusions of this work are those of the authors only.
10

---

References
[1] Jascha Bohl-Diccsteen, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics, In International conference machine learning, pages 2256-2265. PMLR, 2015.
[2] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-668 -6544, 2020.
[3] Tsun- Ssun-Hsuan Wang, Juntian Zheng, Pingchuan Ma. Yilun Du, Byungchul Kim, Andrew Spiel- berg, Joshua Tenenbaum, Chuang Gan, and Daniela Rus. Diffusebot: Breeding soft robots with physies-augmented generative diffusion models. arXiv preprint arXiv:2311. 2023.
[4] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer, Tong Che, Baishakhi Ray, and Mareo Pavone. Guided conditional diffusion for controllable traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3560-3566. IEEE, 2023.
[5] Joao Carvalho, An T Le, Mark Baierl, Dorothea Koert, and Jan Peters. Motion planning diffusion: Learning and planning of robot motions with diffusion models. In 2023 IEEE/RS/ International Conference on Intelligent Robots and Systems (IROS), pages 1916-19233 IEEE, 2023.
[6] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205. 09991, 2022.
[7] Namrata Anand and Tudor Achim. Protein structure and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint TrXiv:2205.15011 2022.
[8] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference on machine learning, pages 1867-8887. PMLR, 2022.
[9] Chentao Cao, Zhuo-Xu Cui, Yue Wang, Shaonan Liu, Taijin Chen, Hairong Zheng, Dong Liang, and Yanjie Zhu. High-freque frequecyy space diffusion model for accelerated mri. IEEE Transactions on Medical Imaging, 2024.
[10] Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical image analysis, 50247,, 2022.
[11] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2022. URL hattps://arx: org/
abs abs/ 10752
[12] Thomas Frerix, Matthias Nießner, and Daniel Cremers. Homogeneous linear inequality con- straints for neural network activations. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition Workshops, pages 748 749,749,17 2020.
[13] Nic Fishman, Leo Klarner, Valentin De Bortoli, Emile Mathieu, and Michael Hutchinson. Diffusion models for constrained domains. arXiv preprint wrXiv:2304.05364. 2023.
[14] Nic Fishman, Leo Klarner, Emile Mathieu, Michael Hutchinson, and Valentin De Bortoli. Metropolis sampling for constrained diffusion models. Advances in Neural Information Pro- cessing Systems, 36, 2024.
[15] Jacob K Christopher, Stephen Back, and Ferdinando Fioretto. Constrained synthesis with projected diffusion models, 2024.
[16] Guan-Horng Liu, Tianrong Chen, Evangelos Theodorou, and Molei Tao. Mirror diffusion models for constrained and watermarked generation. Advances in Neural Information Processing Systems, 36, 2024.
[17] Bowen Song. Soo Min Kwon, Zecheng Zhang, Xinyu Hu, Qing Qu, and Liyue Shen. Solv- ing inverse problems with latent diffusion models via hard data consistency. arXiv preprint 207.v2230..08123. 2023.

---

[18] Rayhan Zirvi, Bahareh Tolooshams, and Anima Anandkumar. Diffusion state-guided projected gradient for inverse problems. arXiv preprint arXiv:2410.03463, 2024.
[19] Litu Rout, Negin Raoof, Giannis Daras, Constantine Caramanis, Alex Dimakis, and Sanjay Shakkottai. Solving linear inverse problems provably via posterior sampling with latent diffusion models. Advances in Neural Information Processing Systems, 36:49960-49990, 2023.
[20] Jesse Engel, Matthew Hoffman, and Adam Roberts. Latent constraints: Learning to generate conditionally from unconditional generative models, 2017.
[21] Lei Shi and Andreas Bulling, Clad: Constrained latent action diffusion for vision-language procedure planning. arXiv preprint (2Xiv:2503.06633 2025.
[22] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[23] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011. 13456, 2020.
[24] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[25] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müllrr, Joc Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01955 01952,, 2023.
[26] Max Welling and Yee W Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference on machine learning (ICML-/I), pages 681-688. Citeseer, 2011.
[27] Pan Xu, Jinghui Chen, Difan Zou, and Quanquan Gu. Global convergence of langevin dynamics based algorithms for nonconvex optimization. Advances in Neural Information Processing Systems, 31, 2018.
[28] Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in Optimization, 13))1277239,, 2014.
[29] Patrick L Combettes and Jean-Chrisse -hhisthisteephe Pesquet. Proximal splitting methods in signal processing. Fixed-point algorithms for inverse problems in science and engineering, pages 185-212, 2011.
[30] Nicolas Brosse, Alain Durmus, Éric Moulines, and Marcelo Pereyra. Sampling from a log- concave distribution with compact support with proximal langevin monte carlo. In Conference on learning theory, pages 319-342. PMLR, 2017.
[31] Tyrrell Rockafellar and Roger J-B Wets. Variational analysis, volume 317. Springer Science & Business Media, 2009.
[32] Magnus R Hestenes. Multiplier and gradient methods. Journal of optimization theory and applications, 4(5):303-320, 1969.
[33] Ferdinando Fioretto, Pascal Van Hentenryck, Terrence W. K. Mak. Cuong Tran, Federico Baldo, and Michele Lombardi. Lagrangian duality for constrained deep learning. In European Conference Machine Learning, volume 12461 of Lecture Notes in Computer Science, pages 118 135. Springer, 2020. doi: 1007 000079978-3-000-67777-00
URL httpss/:s///oi org/10
[34] Jiayi Guo, Xingqian Xu, Yifan Pu, Zanlin Ni, Chaofei Wang, Manushree Vasu, Shiji Song, Gao Huang, and Humphrey Shi. Smooth diffusion: Crafting smooth latent spaces in diffusion models. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7548-7555 2024.
12

---

[35] Quentin Berthet, Mathieu Blondel, Olivier Teboul, Marco Cuturi, Jean- Philippe Vert, and Francis Bach. Learning with differentiable pertubed optimizers. Advances in neural information processing systems, 33:9508 9119, 2020.
[36] Jayanta Mandi, James Kotary, Senne Berden, Maxime Mulamba, Victor Bucarey, Tias Guns, learning: Foundations, state of the art, benchmark and Ferdinando Fioretto. Journal of Artificial Intelligence Research, 80: 1623-1777,, 2024. and future opportunities.
[37] Michail Dontas, Yutong He, Naoki Murata, Yuki Mitsufuji. J Zico Kolter, and Ruslan Salakhut- dinov. Blind inverse problem solving made easy by text-to-image latent diffusion. arXiv preprint arXiv:2412.00557. 2024.
[38] Jan-Hendrik Bastek and Dennis M Kochmann. Inverse design of nonlinear mechanical metama- terials via video denoising diffusion models. Nature Machine Intelligence, 5(12):1466-144 1475, 2023.
[39] Schyun Chun, Sidhartha Roy, Yen Thi Nguyen, Joseph B Choi, HS Udaykumar, and Stephen Back. Deep learning for synthetic microstructure generation in materiial--y-ddeee framework for heterogeneous energetic materials. Scientific reports, 10(1):13307, 2020.
[40] Börgesson. Abaqus. Developments in geotechnical engineering, volume 79, pages 565-570.
Elsevier, 1996.
[41] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
[42] Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics- guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 16010-16021, 2023.
[43] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8799 1894,, 2021.
[44] Hyungjin Chung, Jeongsol Kim, Michael TT' Mccann, Mare I. Klasky, and Jong Chul Ye. Diffu- sion posterior sampling for general noisy inverse problems. arXiv preprint 22X9::2009.1688 2022.
[45] Joao Carvalho, Le. Piotr Kicki, Dorothea Koert, and Jan Peters. Motion planning diffu- sion: Learning and adapting robot motion planning with diffusion models. arXiv preprint arXiv:2412. /9948, 2024.
[46] Jiwen Yu, Yinhuai Wang, Chen Zhao, Bernard Ghanem, and Jian Zhang. Freedom: Training- free energy-guided conditional diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13174-23184 2023.
[47] Sicheng Mo, Fangzhou Mu, Kuan Heng Lin, Yanli Liu, Bochen Guan, Yin Li, and Bolei Zhou. Freecontrol: Training-free spatial control of any text-to-image diffusion model with any condition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages
2024.
[48] Yutong He, Naoki Murata, Chieh-Hsin Lai, Yuhta Takida, Toshimitsu Uesaka, Dongjun Kim, Wei-Hsiang Liao, Yuki Mitsufuji, Zico Kolter, Ruslan Salakhutdinov, et al. Manifold preserv- ing guided diffusion. arXiv preprini arXiv:2311.16444 2023.
[49] Arpit Bansal, Hong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 843-852, 2023.
[50] Haotian Ye, Haowei Lin, Jiaqi Han, Minkai Xu, Sheng Liu, Yitao Liang, Jianzhu Ma, James Zou, and Stefano Ermon. Tfg: Unified training-free guidance for diffusion models. arXiv preprint arXiv:2409.
2024.
[51] William Huang, Yifeng Jiang, Tom Van Wouwe, and C Karen Liu. Constrained diffusion with trust sampling, arXiv preprint 11X9922411.10932, 2024.
13

---

[52] Thomas Power, Rana Doltani-Zarriin Soshi lba, and Dmitry Berenson. Sampling constrained trajectories using composable diffusion models. In IROS 2023 Workshop Differentiable Probabilistic Robotics: Emerging Perspectives on Robot Learning, 2023.
[53] Giorgio Giannone, Akash Srivastava, Ole Winther, and Faez Ahmed. Aligning optimiza- tion trajectories with diffusion models for constrained design generation. arXiv preprint 2755 2023.
[54] François Mazé and Faez Ahmed. Diffusion models beat gans topology optimization. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Washington, DC, 2023.
[55] Andrea Coletta, Sriram Gopalakrishnan, Daniel Borrajo, and Svitlana Vyetrenko. On the constrained time-series generation problem. Advances in Neural Information Processing Systems, 00666001999999..000 2023.
[56) Kartik Sharma, Srijan Kumar, and Rakshit Trivedi. Diffuse, sample, project: plug-and-play controllable graph generation. Forty-first International Conference on Machine Learning. 2024.
[57] Stephen P Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract are theoretically and empirically supported by the content of the paper. We include proofs for all theoretical claims and three distinct, real-world settings to evaluate the proposed methodology.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations are discussed in a standalone section (Appendix A
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[Yes]
Justification: Proofs for all theoretical results are included in Appendix These proof' are complete and correct to the best of our knowledge.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Our experimental section and Appendix E provide all necessary details to reporduce our work (c.g., datasets, implementation details, etc.).
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
16

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We provide code with our submission and intend to release public repository following the reviewing process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips... cc/
Policy) for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
//nips.cc/pblle wwwppppgpuuiiieeeep hetpc///uuudess/eeeuuuuiiiii SooodeSubiissi.e ssiiiiolcccy for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We include these details when applicable.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We have included error bars our experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi-
dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for
example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
17

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma enror bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We describe the used resources Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: All ethical standards have been observed in conducting this research.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The societal impacts of this work are common other advancements in machine learning. As these are general and well-known by those in the field, we do not expressly highlighted them.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairesss considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not deal with data models that pose high risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets include used in the capacity allowed by their respective licenses, and the original owners have been properly cited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets are provided by this paper.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No human subjects were involved in this research.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs were not involved in the core method development.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy petttp://neuriaee http://rpppscccoooffferrreee//// cc/Coonferencces/222/LLLL for what should or should be described.
21

---

A Limitation
Classifier constraints. Section 6.3 motivates the use of classifier-based constraints for stable diffusion models. While we illustrate one potential use case of this approach, we hold that this can generalize to arbitrary properties that can be captured using classifier. We defer more rigorous comparison to classifier guidance [2] and classifier-free guidance [1] for future work.
Stable video generation. While there many exciting applications for using this approach for scientific and safety-critical domains when generating data the image space, many more applications will be enabled by extending this work to video diffusion models. While we compare to video diffusion baselines in Section 6.2 our training-free free correction algorithm only applied to stable image diffusion models. The introduction of temporal constraints over video frames holds significant potential that we plan to investigate in subsequent studies.
Integration of external simulators. This paper motivates future study of embedding non- differentiable simulators within generative process. Yuan et al. 42 previously proposed the inclusion of physics-based simulators to augment diffusion generations, but in their case, differentiaaiiity was not considered as their simulation used a reinforcemeentllarnigg environment to directly return a modified version of the noisy sample Zt. More often, external simulators are used to provide a measure of constraint satisfaction rather than to transform a sample. The techniques explored in this paper provide vastly more general framework for incorporating these black- box simulators as differentiable components of the sampling process and, hence, opens the door for the integration of increasingly complex constraints. Further use cases will be explored in consecutive works.
B
Related Work
Conditional diffusion guidance. Conditional diffusion models have emerged as a powerful tool to guide generative models toward specific tasks. Classifier-based [43] and classifier-free [4]] conditioning methods have been employed to frame higher-level constraints for inverse design
problems [3] 10 38 44 and physically grounding generations [5 42 45) Rombach et al., extended conditional guidance to stable diffusion models via class- claasscooniiiionnng allowing similar guidance schemes to be applied for latent generation. However, while conditioning based approaches can effectively capture class-level specifications, they are largely ineffective when lower-level properties need to be satisfied (as demonstrated in Section 6.1.
Training-free diffusion guidance. Similar to classifier based conditioning, training-free guidance ap- proaches leverage an external classifier to guide generations to satisfy specific constraints. Juxtaposed to classífier-based conditioning, and the method proposed in this paper, training-free guidance lever- ages off-the-shelf classifiers which have been trained exclusively on clean data. Several approaches have been proposed which incorporate slight variations of training-frve guidance to improve constraint adherence 49). Ye et al. compose unified view of these methods, detailing search strategies to optimize the implementation of this paradigm. [Huang et al. iiprove constraint adherence by introduc- ing a "trust schedule" " that increases the strength of the guidance as the reverse process progresses but remain unable to exactly satisfy the constraint set, even within the statistical bounds of the employed classifier. Importantly. training- free guidance approaches suffer from two significant shortcomings First, this paradigm exhibits worse performance than classifier- classifiers provide inaccurate gradients at higher noise levels. Second, like classifier-based guidance, these guidance schemes are ineffective in satisfying lower-level constraints
guidance as the off-th tee-tee-shh shelf
Post-processing optimization. When strict constraints are required, diffusion outputs are frequently used as initial guesses for a subsequent constrained optimization procedure. This approach has been shown to be particularly advantageous in non-convex scenarios where the initial starting point strongly influences convergence objectives directly into the diffusion training process, essentially framing the post-pr post-processing opti- mization steps as an extension of the generative model [53] [54. However, these methods rely on a succinctly formulated objective and therefore often remain effective only for niche problems- such as constrained trajectory optimization-limiting their applicability to a wider set of generative tasks. Furthermore, post-processing steps are agnostic to the original data distribution, and, hence, the constraint correction steps often results in divergence from this distribution altogether. This has been empirically demonstrated in previous studies constrained diffusion model generation [15)
feasible solution [52]. Other methods incorporate optimization
22

---

Hard constraints for generative models. Frerix et al. 12 proposed an approach to impose hard constraints on autoencoder outputs by scaling the generated data so that feasibility is enforeed, but this solution is limited to simple linear constraints. Liu et al. [66] introduced "mirror mappings" to handle constraints, though their method applies solely to familiar convex constraint sets. Given the complex constraints examined in this paper, neither of these strategies was suitable for our experiments. Alternatvely, Fishman et al. [13 [4] extended the classes of constraints that can be handled, but their approach is demonstrated only for trivial predictive tasks with MLPs where constraints can be represented as convex polytopes. This confines their method to constraints approximated by simple geometric shapes, such L2-balls, simplices, or polytopes. Coletta et al. [55] provide generalization of constrained diffusion approaches for time-series problems, and, while these settings not explored in our work, we take inspiration from their framing of diffusion sampling
constrained optimization framework. Most similar to our work are Sharma et al. [56) and Christopher et al. [15]. concurrent studies which propose projected Langevin dynamics sampling processes for constrained generation tasks. Sharma et al. [56) develop this approach for constrained graph generation, making this work not directly applicable to the tasks explored in this work. [Chrittoherr et al. generalize this sampling process for arbitrary constraint sets, but, like the other methods for hard constraint imposition discussed, their work is not extended stable diffusion models.
Stable diffusion for constrained settings. Although inverse design has been extensively studied in diffusion models operating directly in image space, fewer studies have investigated the potential of stable diffusion models for inverse design, as these models have primarily been applied in commercial rather than scientific contexts. Song al. [17] propose ReSample, which repurposed latent diffusion models as unverse-problen solvers by alternating a hard data-consistency projection and stochastic resampling steps returning the sample the learned manifold. While applicable to image reconstruction with a fixed measurement operator, this approach cannot be adapted to handle open-ended constraint sets as explored in this work as it assumes knowledge of this operator for the data-consistency projection. More recently, Zirvi et al. [18] follow-up on this line of research, demonstrating improvement over (17 and [9)], a method proposed concurrently ReSample, but all three methods are limited in the requirement of a forward measurement operator which does not appear in our explored settings. This dependency narrows the scope of these inverse design methods, and this rigid definition of inverse design problems similarly restricts the applicability similarly presented methodology for diffusion operating the image space.
Notably, a handful of other works explore constrained settings more generally without requiring dependency the measurement operator. Engel et al. (20) proposes learning the constraints in the latent space via a neural operator, applying this methodology to auto-encoder architectures. Although effective for soft constraint conditioning, our attempts to adapt such approaches latent diffusion were unsuccessful and such approaches can be viewed as a weaker alternative to classifier- conditional guidance. Rather, conditioning approaches such as proposed by (44 have proven much stronger alternatives, leading to our selection of constraint-conee conditionn stable diffusion as bascline throughout the explored settings. Finally, concurrent to our study, Shi and Bulling (21) proposes CLAD for imposing learned constraints on latent diffusion model generations, but, importantly, their study has very different objectives than our work. As opposed to our work, which enables imposition of hard constraints at inference time, CLAD learns latent vectors to statistically optimize feasibility for visual planning tasks. Importantly, this approach outputs symbolic action tokens as opposed to high-resolution pixel images, distinctly disconnected modality from those explored in this paper.
C Comparison to Classifier Guidance
At first glance, this approach might appear similar to classifier-guided diffusion [3), as both rely on an external predictive model to direct the generation process. However, the two methods fundamentally differ in how the methods apply the model's gradient. Classifier- guider diffusion encourages similarity to feasible training samples, offering implicit guidance. In contrast, our approach provides statistical guarantees to constraint satisfaction within the confidence levels of the classifier, providing more direct and targeted mechanism for integrating constraints into the generative process.
Classifier- guidance. Applies Bayesian principles to direct generation toward a target class y. based on the decomposition:
Va, aa logp(æ: ()) Va, log + Va, log xx))
(9)
23

---

This conditional generation incorporates a classifier ply zt) into the sampling process. During generation, the model updates the noisy sample æ, by combining the standard denoising step with the classifier's gradient:
21+11 = z4t + VV. log p(r:) /2e + Vv loge(w (2a))
(10)
Here, the classifier's gradient V.=, z, log p(y | ax)) guides the denoising toward samples likely belonging to class y. with controlling the guidance strength.
Training-fre frree guidance. Extends the principles of classifier-based guidance by leveraging pretrained, "off-the- shelf" classifiers to steer the generation process without requiring additional training. As with classifier-hased guidance, the conditional generation incorporates a classifier p(y | æ.) into the sampling process. However, rather than training custom classifier tailored to the diffusion model, this approach directly uses existing models compute the guidance term. By decoupling the classifier from the diffusion model training. training free guidance achieves flexibility and reusability, making practical choice for tasks where suitable pretrained classifiers are available.
Surrogate constraint corrections. Introduce structured method to enforce class-specific constraints by adjusting samples at specific diffusion steps. In this approach, a surrogate model modifics the sample zı to 2, to meet the target constraints. These corrections can be introduced either at the beginning of the diffusion process, setting strong initial alignment to the target class and then allowing the model to evolve naturally, designated points within the denoising sequence to enforce the constraints more explicitly al each selected step. In contrast, while mussificr-bbsed guidance and training free guidance continuously integrate classifier gradients to steer generation toward the target class, surrogate constraint corrections offer discrete, targeted adjustments throughout the reverse diffusion process. This makes surrogate constraints particularly effective when striet adherence to certain class-specific conditions necessary at particular stages of the generation process.
D Augmented Lagrangian Method
In experimental settings and 6.3 a closed-form projection operator cannot be derived in the image space, and it becomes necessary to solve this subproblem through gradient- based relaxations. In these settings, adopt the Augmented Lagrangian Method to facilitate this relaxation [57]. This method operated by restructuring constrained optimization problem, in our case
arg min ly D(z+)]l y
minimization objective which captures feasibility constraints through penalty terms. Expressly, the
of Lagrangian multipliers A = (Ai,.... ....... An) and quadratic penalty terms μ (W form the dual variables used to solve the relaxation. Hence, the objective becomes:
arg min y D(z.)ll + X Ag(3) + g(y))
This problem provides a lower-hound approximation to the original projection. Its Lagrangian dual solves:
arg max arg min ly D(2:) + Ag(v) + +5 ((v)))
A,
Following the dual updates described by 333, we perform the following updates:
A Algorithm 2: Augmented Lagrangian
Input: xt, A, 14, Y, a,
y f y - VVLLL A,,
(1
y + D(z:)
while ø(y) 5 do for j + to max_aaxinna inner_iter do
< X + (v(z).
(11b)
1 f- min(ap, (mmax)
(11c)
LALM f-
ly D(z,) || + A Ag(z) + + B(8) y
where y is the gradient step size, is scalar which increases y over iterations, and jmaa upper bound μ.. This drives y to satisfy g(y) ss 0 while staying close to D(z.).
V CALM
A + x x Ō(y); # min (ap, Hlmex)
return
E Extended Results
In this section, we include additional results and figures from our experimental evaluation.
E.1 Mierostructure Generation
Additional baselines. To supplement the evaluation presented in paper. we also implemented the following baselines:
24

---

Generative Methods PDM
Ground
P(%)
Cond
Latent (Ours)
50
FID scores:
10, 68.4% 12.4
6.. 0%10
13.5
Perror 10%:
0%.
Figure 6: Extended version Figure
1. Image Space Correction: We implement naive approach which converts the latent representation to the image space, projects the image, and then passes the feasible image through the encoder layer return to the latent space.
2. Learned Latent Corrector: Adapting the implementation by restore feasibility prior to the decoding step.
20 for diffusion models, we train network
The Image Space Correction method, which involves re-eneoding the image into the latent space after correcting during various denoising steps, and the Learned Latent Corrector method, where a network is trained project latent vector toward new stale ensuring constraint satisfaction, both failed to produce viable samples. Both basclines deviated significantly from the training set distribution, resulting in high FID scores and generated images that lacked quality, failing to capture essential features of the dataset. Due to the inability of these methods to produce viable samples, we do not include them in Figure T
E.2 Metamaterial Inverse Design
Figure iillsstrates the performance of different models interpolation (i.c., when the target curve falls within the stress range covered by the training set) and extrapolation (i.c., when the target is outside this range). In addition to the proposed model, a Conditional Stable Diffusion model and Conditional Video Diffusion model [8 are shown. The proposed model allows for arbitrarily small tolerance settings and outperforms the baselines both tests.
tolerance and compute budget for tailored for the specific application. Each iteration DPO necessitates approximately 30 seconds of computational time. Given our prescribed emor tolerance, convergence is achieved within five iterations, culminating a total computational duration of approximately 2.5 minutes per optimization run. Additionally, note that has not been optimized for runtime, operating exclusively on CPU cores.
Practically,
select
25

---

Interpolation
Extrapolation
Model
Shape
Stress curve
MSE
Shape
Stress curve
MSB
Cond (Stable Image)
7.0
127.3
Cond (Video Diffussoo))
99.6
Bastek and Kochmann
Latent (Ours)
78.3
Figure 7: Performance of different models in interpolation and in extrapolation.
E.3 Copyright-Sar Generation
As opposed other approaches, the projection schedule tuned differently for this setting. Particularly, it only necessary to project during the initial stages of generation. After this correction, the denoising process allowed to evolve naturally without further intervention. This method ensures that the generated images are guided away from resembling copyrighted material while still allowing the model to produce high-qualit, effectively
outputs. By selectively modifying the generated content during the initial stages of denoising. prevent the model from producing images that infringe on copyrights without significantly affecting the overall image quality.
Surrogate implementation. We begin by fine-tuning classifier capable of predicting membership to one two classes: 'Mickey Mouse' or 'Jerry". The architecture of the classifier consists of a ResNet50 backbone, which is followed by two fully connected layers. These layers serve to progressively reduce the dimensionality of the feature map, first from 2048 to 512 and then from 512 to single scalar feature, which represents the oulput of classifier. Sigmoid activation function then applied to this final feature to estimate the probability that the input sample belongs to either the 'Mickey Mouse' or 'Jerry' class. This process ensures that the model outputs a value between 0 and 1, indicating the likelihood of each class membership. The classifier was evaluated held-out test and demonstrated strong performance, achieving accuracy greater than 87%, which showcases its effectiveness distinguishing between the classes.
Runtime
Note that the simulator employed in Section has been specifically optimized for CPU execution; consequently, runtimes observed for these experiments are inherently longer compared GPU-optimized procedures (refer Table Importantly, optimizing computational runtime was not the primary focus of this study. Rather, the emphasis was placed on scientific discovery, prioritizing the fidelity and accuracy of the generated content, In the scientific domains explored, achieving exact adherence to physical constraints and established principles is crucial, deviations would render the models unreliable and impractical for
significant bottleneck frequently encountered in these experimental domains is the iterative cycle from experimental design validation, which typically necessitates multiple trial-and-erro iterations. Consequently, the capability to generate realistic synthetic materials exhibiting precisely controlled morphological or structural characteristics, demonstrated at the fidelity levels reported herein, holds substantial potential for accelerating scientific workflows from initial experimentation to full-scale production.
26

---

Original
Step
Step
Step 4
Structural analysis
increasing stress
Stress-strain curves
Target
Original
Latent (Ours)
MSE [4]
179.5
175.6
12.5
1.2
Figure Enlarged version of Figure 4
Successive steps of DPO shown. At each stage, M perturbed shapes are generated, each undergoing structural analysis with ø. which provides the corresponding stress-strain. The perturbation that produces the curve closest the target is then selected, and a new perturbation-structur. analysis-selection cycle begins, continuing until convergence is achieved. The convergence tolerance can tightened as desired, provided it compatible with the available computational cost.
27

---

Experiment
Sampling Type
Time
Hardware
Microstructure
Conditional Sampling
10s
Nvidia A -100-SXM4-80GB
Microstructure
Constrained Sampling
50s
Nvidia A100-SXM4-8000
Metamaterials
Conditional Sampling
5s
Nvidia A100-SXM--80GB
Metamaterials
Single Simulation
30
Intel Core 17-8550U CPU
Copyright
Conditional Sampling
10s
Nvidia A 1000-XMM-80GGB
Copyright
Constrained Sampling
65
Nvidia 1000-XXM-800GB
Table Experiment runtimes by sampling type and hardware.
G
Feasibility Guarantees
First, that when a projection approximation thereof) he constructed in the image space, striet guarantees can provided
feasihility of the final output the stahle diffusion model. final projection can he applied after decoding 2o, and, as this operator is applicd directly in the image space, constraint satisfaction is ensured if the projection is onto a convex set. As detailed in (S7. (). projection of any vector a E I2" onto non-empty closed convex set C exists and is unique.
Theorem G.1. Convex Constraint Guarantees: The proposed method provides feasibility guarantees for convex constraint.
As we derive closed-form projection for convex sets in the ambient space after decoding sample (e.g.. Section 6.1 where our proximal operator used a projection), the proposed method thus provides guarantees for convex constraints.
H Supplementary Proofs
H.1 Proof of Theorem 4.1 (Part 1): Non-Asymptotic Feasibility in the Image Space
Proof. We begin by proving this bound holds for projected diffusion methods operating in the image space:
dist (at,, C)2 < (1 2.7+4) dist (i't(,'-1, G2,
(12)
For clarity and simplicity of this proof, we omit superscripts in subsequent iterations, using only subscripts to denote both inner and outer iterations of the Langevin sampler. For instance, the sequence 2; ...., 2; x! now represented as .t:--.; -...----ttaa -t-M zh Rt-(M11))- Consequently, as annealing occurs the outer loop level (every M iterations), the step size " is no longer strictly decreasing with each iteration f, and instead satisfics 2 7
Consider that at each iteration of the denoising process, projected diffusion methods can be split into two steps:
1. Gradient Step: æ! + Va, (xx)
2. Projection Step: Pa(a))
These steps sequentially applied in the reverse process sample from constrained subdistribution.
x: 7t8: + Pe(x)) = #c-1 + Li 18 + Pe( = -t-2
By construction, æ;-1 Po(x!) E C. Next, let us define the projection distance to C as:
f f(x) dis C)² = || P=(x))1
Since C 8-prox regular, by definition the following hold:
f is differentiable outside C (in a neighborhood)
(=) 2(x - -Pe(=)
V is L-Lipshitz with L
28

---

The "descent lemma" for L-smooth functions applies:
Lemma H.1. Vo, y in the neighborhond of C:
f(v) - f()) + (Vf(x).y 2)
x² f(x) + 2(x Po(a), a) + E lly x0
Applying this lemma, let us use a = 2/ and y Vy Noting that Pc(x() 11- we get:
dist(a',, C)²
dist(æ/ C)² + (z/-11 32 x 1 ||ar x'-1l1
(+)
Tenm
Torm R
Term
Decomposing Term B. First, consider that since the step size decreasing ? 2 7e-11
# -4-2 ( 2z-2) + 78-188-1
I 11-11 x2-2) + 7680-1
By the same rationale,
xi 1/-1 S (æe, - xe-1) + (e(st 81-1).
(Definition B.1)
Proof of non-expansivennss of the projection operator. Next, we prove the non-expansiveness of the projection operator:
| - 22
(C+)
Given o; Po((x-1) and æc-1 Pe(x)).
> 1 - (| Pe(x/))l s | æ:l
since projections onto closed prox-regular sets are L-Lipshitz
Now:
x/41 = dc+1 + 71+186+13
x/ wi 7884
#/41 x/ = (#+41 - 22) + (74418441 7.88).
(Definition B.2)
Making the projection residual,
2.41 3; 0:+1 Po(x(41)
orthogonal to the target space at æa (and any vector of the form Sc41 - 8t). Thus, since ||se|| S G Vt:
|| w!ll 1|144184411 ||ussell (bet1 + z?)G
Taking the square root:
| - xill S Vh+ + G2G
Since Yt+1 2: Ty:
|| - 2'll S V G
2
Finally, by applying Definition (B.1), || - xzll < | 0/i1, and thus:
| xe 2 G
Now, prox-regularity gives:
(xl 24 21 a/-1) $ Bla x1-1
$ 2=
(Bound B.I)
where the Bound B.I is derived by applying (C*).
29

---

Since C in 8-prox regular, for any point near C and E C:
(u Po(u), PP(())) s B1le Po(")
Above, substitute:
xt + 7886
3t
Pa(u) = 11-1
Now, expanding the inner product:
(a6-11 20-2, v/ a/-1) (16-11 21-2, (2z mse) 7e-18t11))
< (x/-1 Zt-2, (
2c-1) ((1) 71(:: 8t-1))
(x/-1 xt-2, (x: - x. + (0/ 0i-2s 7: - 81-1)
and since ||s||| < G
vt: (81+1, st) s ||st1||||se|| s G² so (st-1, 82) ||st+1|12 S G², and:
(al 2 Mi(sc s(-1)) S G22²
(Bound B.2)
By applying Definition (B.2):
(w'-1 21-2, x/-1) = (a0i-1 - 21-2, (xr w.) + (7/88t (0-88-1))
S (x)-1 - 2t-2; (at xz-1))
Therefore, by applying Bound (B.1) the previous inequality and Bound (B.2) directly, Term B upper bounded by:
2 (w/-11 21-2, x² 1/-1) < 83 G² + +222
(Bound B.3)
Decomposing Term C. Next, we derive bound on Term C in Eq. (x). As already shown,
|| #/-1l1 4y.C,
given:
x/ x'-1 S (x: 8z Yc(s St
S21111
S2G
< 44++1G
Thus,
16 b | a/ 1/111
G21 62
(Bound C.1)
B
Combining bounds (B.3) and (C.1) into (x), and recalling that 7 2 7x:
16
dist(æ',, C)² < dist(z/-1,,)
(80 +88 2
B 1
Now, rewrite Term A, which for ease of notation will refer d:
d (1 28 23
Thus:
dist(w', C)2 < d8 - 23 dt 28 & + K2 = = 23 + 120 + K K2)
Next, through Young's 's incquality, we simplify this expression further. Theorem H.2. (Young's Incquality) Vu, 2 0,e > 0:
2
2
Wv
<
2c
2
30

---

If we choose -23744id, == VK 14 and 23d
then
km++1 G2
wv
128741d G4GG
V2K Gd
Applying Young's Inequality:
u²
2F
2
EV²
2( KTt-1
2012
2
K1oti G²
+ Ev²
2
K2
(1
23d
K K1
2
KT G
+ 3/
2
Thus,
Krtt
V2K Gd
+ gd
Finally, taken altogether:
Gr
2 + KT s 1 +
+ Snttdd)
2
K
23
2 14 G2
Since 7641 <
8 20 then
K 2
G
S 2
8
16
g²
0(33)
2
3
1G
which is bounded by 2 G2 for B 2
Thus,
20 + Kr G S 28 + G.
Finally, by substitution:
dist(ar', C)² S (1 2 C)² G2
0(08)
H.2
Proof Theorem (Pa 2): Non-Asymptotic Feasibility in the Latent Space
Proof. The previous proof can be naturally extended to latent space models under smoothness assumptions:
Assumption H.3.
(1) (Decoder regularity) D R -> dR is é-Lipschitz and [/V(())) S & for all z.
(2) (Constraint regularity) g R -> Rm is L-smooth and (g(x) has full row rank on {x g(x) = 0). hence C is prox-regular.
Let x = D(z)) be the decoded pre-projection iterate. Prox-regularity implics
|xx41
Po((x)] < (1 3) ||x Pe(x))] = (1 - 3) dist(x C),
where X.41 = (x(()) is the data-space projection produced by the algorithm.
Because D is é-Lipschitz we have
dist(x), C) dist(D(z(), C) < dist((,' D (C)),
31

---

where the inverse image is D
D (C) [=: 9::09(D0 (D())) 0). Using the same prox-regularity argument (now
applied to the level set D) Thus, by the chain-rule and prox-regularity
[ o (z) (gg o (z'Il & fLil - z'Il.
which shows that for projection in latent space, the contraction factor (1 (3) is replaced with
1 3' == 3/(EL).
Therefore, denoting by z the latent iterate after one full update but before decoding, we obtain the fundamental recursion
dist D(z)), C)* v (1 -1-22 28+ dis(( C)² + G2t
inon-asymptotic feasibility)
where G is the same (finite) uniform bound used in Part 1.
H.3 Proof of Theorem 4.2: Fidelity
Proof. Finally, we prove the fidelity bound:
KL (z || Paata) I (L((( ([ + G
We begin by assuming feasibility of the training set Pdata inequality,
C. Provided this holds, but the data datc-sson
(L(q(Z:-1) Paata) (L(Pe(g(z))) (P((pata))
(L((P(((22) Paataa
(L(((z() || paata)
This result shows that since the projection operator is deterministic, measurable mapping, the KL will not be increased by this operation iff the training set is subset of the constraint set (all samples in the distribution fall in C). Henee, subsequently we will ignore the presence of this operator in obtaining our bound.
Next, consider that while in continuous time the application of the truc score results in strictly monotonically decreasing KL, Euler discretizations introduce bounded error.
Consider that z xe + 7e8t. Each denoising step shifts the vector z, by vector of at most 7y G. provided that || S
Lemma H.4. Suppose || S everywhere, Then,
KL. KI. (g((z) Paasa) (L((((z)) Peiass S 7..
Proof of Lemma H.4
KL (g(x:) paasa) = g(z)) log g(Zx) dx, Pdata
and
= ((zi) log q( ((z))
KL (g(x)) paata)
da,
pdata
Therefore,
(L(((z)) ll Paata) KL(g(z)) pdata) = f ((x)) [log g(z:) - log g(z, + M(s) dr,
Since ||sell < G, we assume that the true score bounded by a comparable norm Voogp(zz) S G. one-step Taylor bound gives:
logp(z:) - log p(z: + 7888) S Iesell logr(()) s G. for some E on the line segment between 2, and z!. Integrating a(z)) gives
(L(((2)) || paaa)) KL(g(2() pdata) S G2
Lemma H.4 can then be directly applied to derive our bound:
KL KL (Z(=))) paata) < (L(q(Ze) IPpaa)) + G²
32


Reviews:
[{"review_id": "5KOrISmWw3", "reviewer": "Reviewer_2mmt", "summary": "This paper addresses a critical challenge in diffusion models, that is, their inability to strictly adhere to domain-specific constraints. Existing diffusion models explore  constrained generation where diffusion takes place in original data space, however, that does not immediately translate to latent space. The authors propose a novel approach that incorporates proximal mappings into the reverse steps of stable diffusion models, allowing the generated content to conform to complex constraints. Their method does not require retraining and demonstrates strong performance across tasks such as porous-material synthesis, meta-material inverse design, and copyright-constrained content generation.", "questions": "Questions Summarized in weakness", "limitations": "NA", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\nThe paper targets a practical and impactful problem. This method would improve applicability of latent diffusion models to critical domains where generation must adhere to string physical constraints. \nThe integration of proximal mappings with pretrained diffusion models without retraining is novel, practical an efficient, especially as it works with both convex and non-convex constraints. Authors also provie solid theoretical justification for their approach. \nThe approach is validated across diverse and challenging use cases, showing near-zero constraint violations and strong performance, suggesting generality and robustness\n\nWeakness\nFor black-box constraints, the approach estimates subgradients via finite differences, which may be noisy or inefficient in high-dimensional settings.\nHowever, authors compare their work with too few baselines, which limits the clarity on relative advantage.\nWhile the method is empirically shown to work for non-convex and non-differentiable constraints, strong theoretical convergence guarantees are provided only for convex constraints.", "was_revised": true, "final_justification": "I am in favor of accepting the paper.", "rebuttal": "Thank you for your review and for highlighting the _novelty points in our work_!. We found the aspect you pointed out to be very constructive, and we address them below. Please let us know if any pending aspect remains to be clarified; we are very happy to engage further. \n\n> For black-box constraints, the approach estimates subgradients via finite differences, which may be noisy or inefficient in high-dimensional settings.\n\nThe point is well taken: numerical differentiation can be costly and unstable as the problem size or simulation cost grows. Our goal, however, is **to give a general framework that can be applied to any black‑box model, even when classical differentiability fails**. Crucially, the differentiable sensitivity analysis tool provided makes it possible to insert previously incompatible simulators into the inner loop of diffusion models! We believe this is an important and exciting contribution, as illustrated by our results. \nLet us also remark that, as the reviewer may be aware, in this generative‑model context an approximate gradient is often sufficient; the stochastic nature of diffusion already tolerates noise, and even coarse gradient information still drives the sampler toward high‑quality outputs. \n\nIn terms of efficiency, this can be improved on a case‑by‑case basis, e.g., batching model calls, parallel execution, GPU acceleration, but we consider these as engineering steps that are orthogonal to the main message and contribution of the paper and were not the focus of this initial study. While we have currently addressed this point in the limitation section, we will be happy to further emphasize it, should the reviewer deem it essential. \n \n\n> Authors compare their work with too few baselines.\n\nThroughout our work, we took special care in identifying what we found to be the **the strongest available state-of-the-art baselines** for each task: Christopher et al., for microstructure generation, and Bastek & Kochmann for metamaterial design. In the case of copyright-safe generation, since it has not been addressed in prior work, the best available state-of-the-art approach we found was conditional diffusion (Ho et al.). During development, we also considered other baselines (which are discussed in Appendix E), but we omitted them from the main text to keep the exposition focused on meaningful targets for improvement.\n\n\n> Strong theoretical convergence guarantees are provided only for convex constraints.\n\nThis is not entirely accurate. Our convergence guarantees are provided under a $\\beta$-prox-regularity assumption (in the sense of [A] Definition 13.27). This is *a relaxation of typical convexity assumptions*, which implies that for each viable point and normal direction, small perturbations still project uniquely and smoothly back to $\\mathbf{C}$. Hence, the theoretical convergence guarantees extend beyond solely convex constraint sets.\n\nHowever, as the reviewer will surely know, guarantees for truly general non‑convex constraints remain out of reach for the optimization field at large, so proving results under prox‑regularity *should be considered a strength rather than a limitation*.\n\n---\n\nThank you again for your positive assessment of our work! We hope these clarifications address your concerns, and would appreciate your championing our work further. We are happy to follow-up if other questions arise. Many thanks!\n\n\n[A] Rockafellar, R. Tyrrell, and Roger JB Wets. Variational analysis. Berlin, Heidelberg: Springer Berlin Heidelberg, 1998."}, {"review_id": "YByaqLgcDz", "reviewer": "Reviewer_8ptq", "summary": "This paper presents a method of incorporating proximal projection optimization as a guidance method into latent (stable) diffusion sampling models, such that the samples from the model obey feasibility constraints. This is an important consideration when using stable diffusion models for generating high dimensional, physically plausible samples in a variety of scientific and engineering applications, etc. The contribution is in formulation and analysis of the method, and in a thorough empirical evaluation on real high dimensional problems.", "questions": "Questions:\n- Do all intermediate samples, $x_t$, have to obey the constraints, or just the final sample at $t=0$? Is the reason for constraining $x_t$ because it is too hard to propagate the constraints from only  $x_0$ to the latent space, or are the convergence rates worse, or is there another reason? It would seem to me that in some applications, feasibility of all $x_t$ would be a strong requirement that may be hard to satisfy.\n- I’m confused by the dimensionality of $\\mathbf{z}$ and the mapping $\\mathbf{g}$ in Section 4.1, and 4.2. It appears that in section 4.1 we require $\\mathcal{D} : \\mathcal{Z} \\to \\mathcal{X}$ in section 4.1 for the inputs to $\\mathbf{g}$, but in Section 4.2. Eqn. 6 this mapping is not used, and $\\mathbf{g}$ has inputs from the latent space? Are you referring to different distances, $\\mathbf{g}$ in these cases, or a different $\\mathbf{z}$, and the notation has been overloaded?\n- Inner and outer optimizer presentation confusing — I’m not actually 100% sure what these are referring to. Do you mean every diffusion step, $t$, the projection in (7) needs to be run, and then additionally, the equation after line 178 needs to be run for the inner Langevin sampling steps w.r.t. $i$? Or, is equation 7 implicitly being satisfied by the corrected Langevin sampling steps in the eqn after line 178? The latter seems to be implied by Algorithm 1, but it's not totally clear.\n- “Choosing $\\mathbf{g}$ as the indicator of a set, reproduces the familiar projection step introduced earlier” — This seems a little unclear to me, do you mean $g(y) = [[y \\in C]]$? Would this then have to be a complement given the arg min?\n- In Section 6.3 — do you need a projection path in, e.g. latent space like PCA? Or could you directly use the classifier and something like a Brier score as the $\\mathbf{g}$? You mention this as future work in appendix A, what is stopping the implementation of it now?\n\n\nComments:\n- Should there be no commas in the definition of $x_t$ before eqn. (1)?\n- reminds me of “Deep declarative networks” (https://arxiv.org/abs/1909.04866) where notes in a network can be the solution to an optimization problem (implicit nodes).\n- Algorithm 1: would be good to indicate $z^0_t$ comes from omitted sampling steps.", "limitations": "Yes, the authors supply a nice comment on limitations in the Appendix.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The problem is interesting, relevant an impactful — and will help allow diffusion models to generalise to tackle new, high-dimensional problems in science and engineering that have feasibility constraints.\n- The contribution is clear and significant, and notable the authors supply a convergence analysis of their method, in terms of both the fidelity of the (constrained) generative distribution, and feasibility of generated samples.\n- Strong empirical/experimental evaluation on interesting end relevant problems, which back up the claims of the authors.\n- Clearly and well written for the most part.\n\nWeaknesses:\n- Some of the presentation of the method could be clearer to aid understanding — e.g. there is some confusing potential overloading of notation, and the algorithm could be explained more clearly. See the “questions” section for more detail on these points. \n\nOverall I found this paper a pleasure to read and review. It is interesting, and I think has decent originality and significance. The quality and clarity is generally high, though could be better with some small additional effort on the part of the authors.\n\nWith these clarifications, I think this is a worthy submission to NeurIPS 2025.", "was_revised": true, "final_justification": "The authors have suitably responded to my queries, and by adding these clarifications, corrections and discussion points to the final manuscript will make for an interesting addition to NeurIPS.", "rebuttal": "Thank you for reviewing our work carefully and for highlighting its strengths and novel contributions. We answer each question below. Please let us know if we can provide any further clarification. \n\n\n> **Q1:** Do all intermediate samples, $x_t$, have to obey the constraints, or just the final sample at $t = 0$? \n\nThis is also the question we asked ourselves at the beginning of this investigation. In practice, only the final sample *must* satisfy the constraints, but constraining the intermediates matters.\nIndeed, already earlier work showed that enforcing post-sampling correction at $x_0$ results in substantial quality degradation because a late projection can *push away the point from the target probability density region* [1,2]. \nOur Theorem 4.1 formalizes the gain: keeping $x_t$ within the constrained manifold steers the chain toward the constraint set while still converging to $p_\\text{data}$ (Theorem 4.2). \n\nAs you noted, early samples could be too noisy in some domains to be evaluated. We follow this general principle: apply projections as early as possible, while ensuring that the constraint is being properly evaluated. This is when the *highest sample quality is achieved* according to our observations.\n\n\n> **Q2:** I’m confused by the dimensionality of $z$ and the mapping $g$ in Section 4.1, and 4.2.\n\nThank you for noting this! The notation was overloaded. In Equation (6), we meant a sample and penalty function operating in the same space (latent or ambient). We will replace $z_t$ with $x_t$ there and keep Equation (7) as the latent‑space generalization. This makes the ambient‑space case clear while preserving generality.\nAgain, we appreciate this observation and this has been addressed on our end already.\n\n\n> **Q3:**  Do you mean every diffusion step, $t$, the projection in (7) needs to be run, and then additionally, the equation after line 178 needs to be run for the inner Langevin sampling steps w.r.t. $i$? Or, is equation 7 implicitly being satisfied by the corrected Langevin sampling steps in the eqn after line 178?\n\nAt each diffusion step (the outer loop in Algorithm 1), Equation (7) is satisfied by the correction (the inner loop in Algorithm 1). The proximal mapping defined in this equation (and implemented in the inner loop) is facilitated by the \"inner minimizer\" -- essentially, the equation presented for the inner minimizer can be viewed as a *single unrolled step* of the iterative process used to solve the optimization defined by Equation (7). We will add verbiage to better clarify this.\n\n\n> **Q4:** “Choosing $g$ as the indicator of a set, reproduces the familiar projection step introduced earlier”\n\nTo clarify, here we are referring to the case where $g$ is defined as an *indicator function*. In proximal operator, an indicator function has a very specific definition:\n$$\ng(y) =\n\\begin{cases}\n0, & \\text{if } y \\in C \\\\\\\\\n+\\infty, & \\text{if } y \\notin C.\n\\end{cases}\n$$\n\nHence, the constraint violation term completely dominates the optimization whenever $y \\notin C$, and the proximal map becomes:\n\n$$\n\\text{prox}_{\\lambda g}(x) = \\underset{y \\in C}{\\text{argmin}} \\\\; \\frac{1}{2 \\lambda} \\|y - x\\|^2\n$$\n\n\n\n\nexactly the projection onto $C$. Additionally, provided this definition of an indicator function, there is no need for a complement. We hope this clarifies your question. \n\n\n> **Q5:** In Section 6.3 — do you need a projection path in, e.g. latent space like PCA?\n\nNot strictly, but it can significantly enhance the efficacy of the method. For instance, a scalar classifier output could potentially be used as a score to guide the correction, but, as we had observed in our early experiment, it can be too lossy (the information captured by the model is compressed into a single value in the final layer). We thus project the penultimate activations to PCA‑2 so the guidance uses richer features without running into high‑dimensional clustering issues. Higher‑dimensional PCA might also help, but exploring that trade‑off is left as a possible future direction.\n\n\n---\n\nThank you agin for your feedback! We hope these clarifications strengthen the paper and are happy to discuss any remaining concerns.\n\n\n[1] Christopher, Jacob K., Stephen Baek, and Nando Fioretto. \"Constrained synthesis with projected diffusion models.\" Advances in Neural Information Processing Systems 37 (2024): 89307-89333.\n\n[2] Yuan, Ye, et al. \"Physdiff: Physics-guided human motion diffusion model.\" Proceedings of the IEEE/CVF international conference on computer vision. 2023."}, {"review_id": "bDB3gHi6Ur", "reviewer": "Reviewer_2eXJ", "summary": "The paper presents a novel approach for constraining latent diffusion models, such as Stable Diffusion, based on constraints formulated in the original (e.g., image) space. Then, the constraints are projected via the decoder into the latent space of the diffusion model and subsequently enforced during each diffusion step via an inner optimization loop akin to projected gradient descent. It is important to stress that this is done after training during inference and the gradients with respect to the model parameters are not optimized. The constraints can even be possibly non-convex and not differentiable. To remedy situations where the constraints are not differentiable out of the box (e.g., where the constraint is the outcome of an experiment or simulation), the authors adopt existing approaches from literature, such as differentiable surrogates or smoothed approximated gradients via random local perturbations, to estimate the gradients of the constraints.\nThe method is quantitatively evaluated on three different domains: microstructure generation, metamaterial inverse design, and copyright-safe generation and compared against a text-conditioned diffusion model and a baseline method that enforces the constraints postpriori directly in the output/image space.", "questions": "- How would training-time enforcement of the constraints compare to the proposed approach? Would it make sense to adopt a baseline approach that trains a model with the constraints enforced during training, and then compare the performance of the proposed method to that baseline? If not, why not?\n- For example in Equation (6), the authors introduce the trade-off between constraint satisfaction and maintaining similarity to the original learned distribution. Could instead of this soft constraint also a hard constraint be used? If so, how would this affect the performance of the method?\n- In line 237, the authors mention a smoothing function that is used. Where is the smoothing function defined? Is it just the sum operator?\n- Figure 8: Does the performance get even better when you perform more iterations/steps? Why did you stop at four steps?", "limitations": "Although the paper does discuss limitations in Appendix A, the reviewer would like to see a few-sentence summary of the limitations included in the main text of the paper (e.g., in the conclusion section).", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n\n- The paper is very well written and clearly structured.\n- The paper tackles a very timely and relevant problem in the field of generative models. The ability to infuse physical priors, constraints, or other guarantees into the generation process without sacrificing the quality of the generated samples (too much) is a very important capability (e.g., with respect to video-generation models) and this paper could have a significant impact on this domain.\n- The method is evaluated on problem settings from three different domains and the quantitative results are strong.\n\n## Weaknesses\n\n- Some of the figures are a bit unclear - also, because the captions are very brief and do not explain the figures in detail. Instead, a reader should be able to understand the figures (at least their gist) without having to read the main text. Examples include Figure 4 and Figure 5.\n- Although the two considered baseline methods are suitable and well-chosen, it could be beneficial to include additional baselines from related work, or, for example, a baseline that enforces the constraints during training. This would allow for a more comprehensive evaluation of the proposed method.\n- The computational complexity, efficiency and runtime of the method is not discuss in detail.\n\n### Detailed Comments on Minor Issues\n\n- Line 26: What is the difference between \"physical laws\" and \"first principles\"? Are physical laws not a subset of first principles? Please clarify.\n- Line 36: Although the authors later define how stable diffusion models operate in a learned latent space, at this point in the text it might not clear to a reader who is not familiar with latent diffusion models what a \"latent diffusion model\" is. One subsentence could be added to clarify this.\n- Figure 4: To the reviewer, it is not clear what is the meaning of the different rows in the \"Structural analysis\" section of the figure. Please define in the caption.\n- \"Differentiating through blackbox simulators\" lines 243-244: It seems a bit strange that there is no division of the equation by the (direction) of perturbation or something similar needed. Why is the case? Usually, you would have in finite differences something of the form $\\frac{f(x + \\epsilon) - f(x)}{\\epsilon}$ with $\\epsilon \\in \\mathbb{R}$, but here it seems like the authors are just using $f(x + \\epsilon) - f(x)$.\n- The heading of Appendix H \"Missing Proofs\" sounds a bit informal or even confusing. A more formal heading could be \"Supplementary Proofs\" or something similar.", "was_revised": true, "final_justification": "The reviewer appreciates the author's rebuttal and finds that it addresses the reviewer's questions and concerns to satisfaction. The reviewer keeps the score of \"6: Strong Accept\" after studying the rebuttal and the reviews and rebuttal to other reviews.\n\n- Indeed, the small number of baselines is also highlighted by other reviewers, but the reviewer is not sufficiently familiar with the literature in this domain to assess with confidence if there would be another suitable baseline that would have needed to be included or not.\n- The paper is already well written and promises to be even more improved in quality after the authors have implemented the promised changes and clarifications.", "rebuttal": "Thank you for the time you dedicated to this detailed and constructive review! We appreciate that the strengths of our work were identified and highlighted, and we find all the comments helpful for further improving it. We address each of the points in detail below.\n\n> **W1 & C3:** Some of the figures are a bit unclear - also, because the captions are very brief and do not explain the figures in detail.\n\nThank you for your comment. We agree this as an area for improvement and will revise the captions accordingly:\n\n- **Figure 4:** Successive steps of DPO. The sample is iteratively improved and the stress-strain curve aligns with the target.\nStructural analysis shows the progressive deformation under controlled compression.\n\n- **Figure 5:** Left: Denoising process of Cond vs. Latent (Ours). Out method drives the denoising toward a copyright-safe image. Top-right: showing projection from original (O) to projected (P) in the PCA-2 space. Bottom-right: Constraint satisfaction and FID scores.\n\n\n> **W2:** It could be beneficial to include additional baselines from related work, or, for example, a baseline that enforces the constraints during training.\n\nDuring development we indeed evaluated several additional baselines (which are discussed in Appendix E) but, due to space constraints, *we included those that either represent the state-of-the-art in specific domains* (Christopher et al. (NeurIPS 2024) and  Bastek & Kochmann (Nature 2023) for microstructure generation and metamaterial design, respectively) or that we considered to be *the best performing among existing alternatives* (text-conditional model for copyright-safe generation). \n\nWe also highlight that in the cases of metamaterials and copyright, *the selected baselines indeed incorporate constraints during the training phase*. For example, Bastek & Kochmann impose periodic structure constraints during training. Similarly, constraints are encoded in Section 6.3 using a text-conditional model, which closely resembles existing work for training-time latent diffusion methods for constrained inverse-design settings [1]. Additionally, given the complexity of the constraints modeled in this paper (e.g., in Sections 6.2 we use a finite element analysis solver is used in the loop, and in Section 6.3 the constraints are highly nonconvex) very few existing methods are applicable. To the best of our knowledge, *we compare to the strongest methods that currently exist for the settings explored*. \nWe will, however, empahsize in the paper, the additional baselines discussed in Appendix E. \n\n\n> **W3:** The computational complexity, efficiency and runtime of the method is not discuss in detail.\n\nWe are happy to expand on this point. Appendix F provides the requested runtimes, but we will make sure to give them greater visibility in the final version of the paper. In terms of constraint complexity: \n\n- **Section 6.1:** This is a knapsack constraint. While in general knapsack problems are NP-complete, the version adopted (which uses integers as weight) is known to be weakly NP-complete and admits a fully polynomial approximation. It is solved efficiently in $O(nm)$ where $n$ is the number of pixels and $m$ is the number of values to compute for the dynamic program. \n\n- **Sections 6.2 and 6.3:** The inclusion of a blackbox simulator and a nonconvex neural surrogate characterizes these problems as NP-complete, but also quite compelling applications.\n\nWe will report these details in the final version of the paper. Thanks for the suggestion.\n\n\n> **C1:**  What is the difference between \"physical laws\" and \"first principles\"?\n\nFirst principles are fundamental concepts; physical laws are their mathematical consequences or empirical summaries. We favor the law form when writing constraints. In practice, our usage partially overlaps; we will clarify as needed. \n\n\n> **C2:** Although the authors later define how stable diffusion models operate in a learned latent space, at this point in the text it might not clear to a reader who is not familiar with latent diffusion models what a \"latent diffusion model\" is.\n\nThis is a good observation, thank you. We will clarify. \n\n\n> **C4:** It seems a bit strange that there is no division of the equation by the (direction) of perturbation or something similar needed. Why is the case? \n\nWe will add a short derivation to show that, for the chosen step size, the direction scaling is absorbed into the proximal update, eliminating an explicit division.\n\n\n> **C5:** A more formal heading could be \"Supplementary Proofs\" or something similar.\n\nAgreed. Thank you!\n\n\n> **Q1:** How would training-time enforcement of the constraints compare to the proposed approach\n\nThank you for the question, since this is one of our contributions. Besides requiring heavier training, we see two key limitations of training-time enforcement methods: (1) they cannot guarantee constraint satisfaction even for the case of convex constraints, and (2) It does not support generalization to unseen constraints, since the model only learns to ``satisfy'' (we quote the word satisfy here because this is only true at a distribution-level, not at the level of each sample) those present during training. These are two key strengths of our proposed approach. \nInterestingly, we also find that this allows us to fine-tune models using very sparse datasets, a property we do not observe with training-time methods. This raises a broader and open question: _why does sampling-time enforcement enable generalization under limited supervision?_ We are actively investigating this.\n\nAdditionally, let us point out that some of the baselines we compare against do use training-time constraint enforcement, so the empirical comparisons reflect this distinction.\n\n\n> **Q2:** For example in Equation (6), the authors introduce the trade-off between constraint satisfaction and maintaining similarity to the original learned distribution. Could instead of this soft constraint also a hard constraint be used? \n\nYes, in principle, and we do so when the constraint is treated as an indicator function. However, the hard version requires solving a projection exactly at every step and requires a closed form solution. This is impractical for our black‑box constraints settings (Sections 6.2 and 6.3). The soft penalty gives a practical balance.\n\n\n> **Q3:** Where is the smoothing function defined? Is it just the sum operator?\n\nThanks for this question, since this is a key contribution of our work but, we agree its description would deserve additional space to make full justice to it. \n\nThe *smoothing function* is the map\n\n$$\n\\Phi_\\nu(\\mathbf{x}) = \\mathbb{E}_{\\varepsilon} \\left[ \\phi(\\mathbf{x} + \\nu\\varepsilon)\\right],\n$$\n\nwhere $\\phi$ is the non‑differentiable simulator, $\\varepsilon \\sim \\mathcal{N}(0,I)$ (or any full‑support noise, really) and $\\nu>0$ is a temperature that scales the perturbation.  Averaging the simulator's output over these random local perturbations _smooths away the simulator’s discontinuities_: for any fixed $\\nu$, $\\Phi_\\nu$ is continuously differentiable even though $\\phi$ may not be. \nAs $\\nu \\to 0$ the bias vanishes and $\\Phi_\\nu$ approaches the raw simulator; as $\\nu$ grows, the function becomes smoother but drifts toward the mean behavior of the simulator in a neighborhood of $\\mathbf{x}$.\n\nTo make this practical, we estimate the expectation with $M$ Monte‑Carlo samples,\n$$\n\\bar{\\phi}_\\epsilon(\\mathrm{x}) = \\frac{1}{M} \\left[ \\phi(\\mathrm{x} + \\nu \\epsilon^{(1)}) + \\cdots + \\phi(\\mathrm{x} + \\nu \\epsilon^{(M)}) \\right]\n$$\nand treat the sample average as a differentiable proxy. \n\nNow, because the noise is injected *outside* the simulator, gradients flow through the re‑parameterization $\\nabla_{\\mathbf{x}} \\Phi_\\nu(\\mathbf{x}) = \\frac{1}{\\nu} \n\\mathbb{E}\\left[ \\phi(\\mathbf{x} + \\nu\\varepsilon) \\varepsilon\\right]$. \nThe same formula with the finite‑sample average gives an unbiased gradient estimator that we use into our proximal Langevin update.  \n\nWe will make sure to extend this discussion. \n\n\n> **Q4:** Figure 8: Does the performance get even better when you perform more iterations/steps?\n\nYes. By proceeding with more steps, it is in principle possible to achieve even smaller errors, as we have empirically observed. Since the computation time increases, however, it is also important to strike a good trade-off between these two desiderata. \n\n\n---\n\nThank you again for your review and for championing our work. We believe our response has clarified all the outstanding questions, but we are happy to provide additional details as needed.\n\n\n\n[1] Dontas, Michail, et al. \"Blind Inverse Problem Solving Made Easy by Text-to-Image Latent Diffusion.\" arXiv preprint arXiv:2412.00557 (2024)."}, {"review_id": "3xr0zMXIgM", "reviewer": "Reviewer_1Dzz", "summary": "This paper presents a framework that integrates constrained optimization with stable diffusion models to enable constraint-aware data generation.  The proposed approach introduces a constraint formulation that can be incorporated into the diffusion sampling process, allowing for the generation of outputs that satisfy stringent physical, functional, and legal requirements while maintaining high synthesis quality. Two theorems provides convergence properties in theory. Through experiments, the authors demonstrate the framework's ability to handle complex constraint in stable diffusion models.", "questions": "1. Is the speed of the methods depending on the complexity of the constraint? \n2. How to evluate the complex of the constraint? Is there some numerical value to measure that？", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "The strengths are introducing a frameworks with constrained optimization in stable diffusion models.   The detailed proof of their convergence represent a non-trivial and significant research endeavor with both theoretical and practical implications.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the time invested in reviewing our work and for praising the innovative aspects of the proposed method. We will address the point raised below and will integrate the suggestions in our final version.\n\n> **Q1:** Is the speed of the methods depending on the complexity of the constraint?\n\nYes. The performance of the proposed method depends largely on the complexity of the constraints enforced and the ease with which their violations can be measured. In the paper, *we presented three cases with varying levels of complexity (and execution times)* and reported runtimes in Appendix F. \n\n\n> **Q2:** How to evaluate the complexity of the constraint? Is there some numerical value to measure that？\n\nGenerally, one may want to categorize them in convex and non-convex, but in this paper, we go well beyond non-convexity as we include non-differentiable black box simulators that are used to constrain our model. To provide some details:\n\n- **Section 6.1:** This is a knapsack constraint. While in general knapsack problems are NP-complete, the version adopted (which uses integers as weight) is known to be weakly NP-complete and admits a fully polynomial approximation. It is solved efficiently in $O(nm)$ where $n$ is the number of pixels and $m$ is the number of values to compute for the dynamic program. \n\n- **Sections 6.2 and 6.3:** The inclusion of a blackbox simulator and a nonconvex neural surrogate characterizes these problems as NP-complete. This makes these settings complex but also very compelling.\n\nOverall, it is evident that evaluating a constraint through a simple differentiable operation on the sample (e.g., microstructure porosity) is straightforward and leads to a fast and efficient correction algorithm. In contrast, evaluating a constraint using external numerical simulators or, more generally, black-box models (e.g., as in the case of our metamaterials experiment) involves higher computational costs and greater implementation complexity. \n\n---\n\nThank you again for your review and your praise for our work. We believe that our responses have addressed each concern presented in your assessment but we are more than happy to provide additional details as needed. Many thanks!"}]

Author rebuttal:

Meta-review: This paper suggests a training-free algorithm for sampling latent diffusion models with differentiable constraints by composing a proximal projection operator to the standard diffusion sampling steps. The projection entails differentiation through the differentiable constraints and the latent decoder. 

Reviewers found the problem tackled by this paper important and the solution devised is novel, practical and rather efficient. The method is both theoretically justified as well as validated in a solid experimental setup. 
The criticism was focused on the relatively low number of baselines compared to, and some exposition issues that the authors have indicated that they will improve in camera ready version of the paper.
Decision: Accept (spotlight)