Constrained Synthesis with Projected Diffusion Models
Authors: Jacob K Christopher
Stephen Baek
Ferdinando Fioretto
Venue: NeurIPS 2024 poster (2024)
Keywords: Constraint satisfaction
Generative diffusion models
physics-informed models
Primary area: diffusion_based_models
TL;DR: We propose an alteration of the sampling step in diffusion models to generate outputs that satisfy desired constraints and physical principles

Abstract: This paper introduces an approach to endow generative diffusion processes the ability to satisfy and certify compliance with constraints and physical principles. The proposed method recast the traditional sampling process of generative diffusion models as a constrained optimization problem, steering the generated data distribution to remain within a specified region to ensure adherence to the given constraints.
These capabilities are validated on applications featuring both convex and challenging, non-convex, constraints as well as ordinary differential equations, in domains spanning from synthesizing new materials with precise morphometric properties, generating physics-informed motion, optimizing paths in planning scenarios, and human motion synthesis.

Constrained Synthesis with Projected Diffusion Models
Jacob K. Christopher University of Virginia cak4er@virginia.
Stephen Baek University of Virginia baek@virginia, edu
Ferdinando Fioretto University of Virginia fi fiorettolvirginis edu
Abstract
This paper introduces an approach to endow generative diffusion processes the ability to satisfy and certify compliance with constraints and physical principles. The proposed method recast the traditional sampling process of generative diffu- sion models as constrained optimization problem, steering the generated data distribution to remain within specified region to ensure adherence to the given constraints. These capabilities are validated on applications featuring both convex and challenging. non-convex, constraints as well as ordinary differential equations, in domains spanning from synthesizing new materials with precise morphomet- ric properties, generating physics-informed motion, optimizing paths planning scenarios, and human motion synthesis.
Introduction
Generative diffusion models excel at robustly synthesizing content from raw noise through a sequential denoising process [14 24] They have revolutionized high-fidelity creation of complex data, and
their applications have rapidly expanded beyond mere image synthesis, finding relevance areas
34 automation [3] [6], chemistry [i [5], and medical analysis [2] 6). However, although diffusion models excel at generating content that is coherent and aligns closely with the original data distribution, their direct application in scenarios requiring stringent adherence to predefined criteria poses significant challenges. Particularly the use of diffusion models in scientific and engineering domains where the generated data needs not only resemble real-world examples but also rigorously comply with established specifications and physical laws remains open challenge.
such as engineering [29
Given these limitations, adheres to specific constraints. However, even with "feasible" training data, this approach does not guarantee adherence desired criteria due to the stochastic nature of the diffusion process. Furthermore, there are frequent scenarios where the training data must be altered to generate outputs that align with specific properties, potentially present in the original data. This issue often leads to distribution shift further exacerbating the inability of generative models to produce "valid" data. As will show in a real-world experiment ($6.1). this challenge is particularly acute in scientific and engineering domains, where training data is often sparse and confined to specific distributions, yet the synthesized outputs are required meet stringent properties or precise standards (29).
might consider training a diffusion model dataset that already
This paper addresses these challenges and introduces Projected Diffusion Models (PDM), novel ap- proach that recast the traditional sampling strategy in diffusion processes constrained-optimian problem. This perspective allows us to apply traditional techniques from constraint optimization to the sampling process. In this work, the problem is solved by iteratively projecting the diffusion sampling process onto arbitrary constraint sets, ensuring that the generated data adheres strictly to imposed constraints physical principles. We provide theoretical support for PDM's capability not only certify adherence to the constraints but also to optimize the generative model's original objective of replicating the true data distribution. This alignment a significant advantage of PDM. yielding state-of-the-ee FID scores while maintaining strict compliance with the imposed constraints.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Contributions. In summary, this paper makes the following key contributions: (1) It introduces PDM, a new framework that augments diffusion-based synthesis with arbitrary constraints order to generate content with high fidelity that also adheres to the imposed specifications. The paper elucidates the theoretical foundation that connects the reverse diffusion process to an optimization problem, facilitating the direct incorporation of constraints into the reverse process of score-based diffusion models. (2) Extensive experiments across various domains demonstrate PDM's effectiveness. These include adherence to morphometric properties in real-world material science experiments, physics-informed motion governed by ordinary differential equations, trajectory optimization in motion planning, and constrained human motion synthesis, showeasing PDM's ability to produce content that adheres to both complex constraints and physical principles.
(3) We further show that PDM is able generate out-of-distribution samples that meet stringent constraints, even in scenarios with extremely sparse training data and when the training data does not satisfy the required constraints. (4) Finally, we provide theoretical basis elucidating the ability of PDM to generate highly accurate content while ensuring constraint compliance, underpinning the practical implications of this approach.
2 Preliminaries: Diffusion models
Diffusion- based generative models [14 24] expand a data distribution, whose samples are denoted 2o- through a Markov chain parameterization {ær)T= defining a Gaussian diffusion process p(ær) =
((ær)) 1 p( la,))dr
In the forward process, the data is incrementally perturbed towards Gaussian distribution. This process is represented by the transition kernel ((=))<(-1) = (xx;; - 3,1) for some 0 < Bg < 1, where the 8-schedule {8:)-= is chosen that the final distribution p(ær) is nearly Gaussian. The diffusion time allows an analytical expression for variable 2, represented by
(=o <) = va + vI- where e ~ (O, I) is a noise term, and a, = -II'- (1 3.). This process is used train neural network en (xt l), called the denoiser, which implicitly approximates the underlying data distribution by learning to remove noise added throughout the forward process. The training objective minimizes the error between the actual noise and the predicted noise eo (xx e), t) via the loss function:
[||c co(xe(x0, t)
min L~IL1 M(c00.)
E
The reverse process uses the trained denoiser, cØ t), to convert random noise p(æT) iteratively into realistic data from distribution p(æo). Practically, ey predicts a single step in the denoising process that can be used during sampling to reverse the diffusion process by approximating the transition p(x;-1/a0 1 at each step t.
while also operating on the principle of gradually adding and removing noise, focus on directly modeling the gradient (score) of the log probability of the data distribution at various noise levels. The score function Va, log p( p(x:)) identifies the direction and magnitude of the greatest increase in data density each noise level. The training aims to optimize neural network So (a t) to approximate this score function, minimizing the difference between the estimated and true scores the perturbed data:
Score-based models 25 26
min 0
E t~ (=(TT.)pp(())
(1 - (1) [|sso(Ze,) () Var, log (()))))())))))
(mo)
where q( (b))= = Nax; Varza, (1 I)) defines a distribution of perturbed data Zt. generated from the training data, which becomes increasingly noisy as l approach T. This paper considers score-based models.
3 Related work and limitations
While diffusion models are highly effective in producing content that closely mirrors the original data distribution, the stochastic nature of their outputs act as an impediment when specifications or constraints need to be imposed on the generated outputs. In an attempt to address this issue, two main approaches could be adopted: (1) model conditioning and (2) post-processing corrections.

---

Model conditioning [33 aims to control generation by augmenting the diffusion process via a conditioning variable to transform the denoising process via classifier-fres free guidance:
Eo def x co (T t, c) + (1 - X) x tt,
where À E (0, 1) is the guidance scale and 1 is null vector representing non-conditioning. These methods have been shown effective in capturing properties of physical design (29), positional awareness [3], and motion dynamics 32) However, while conditioning may be effective to influence the generation process, it lacks the rigor to ensure adherence to specific constraints. This results in generated outputs that, despite being plausible, may not be accurate reliable. Figure [ (red colors) illustrates this issue on a physics-informee motion experiment (detailed in $6.4). The figure reports the distance of the model outputs to feasible solutions, showcasing the constraint violations identified in a conditional model's outputs. Notably, the model, conditioned on labels corresponding positional constraints, fails to generate outputs that adhere these constraints, resulting in outputs that lack meaningful physical interpretation.
Convergence to Feasible Subdistribution
Additionally, conditioning in diffusion models often quires training supplementary classification and regression models, process fraught with its own set of challenges. This approach demands the acquisition of extra labeled data, which can be impractical unfeasible in specific sce- narios. For instance, our experimental analysis will demon- strate a situation in material science discovery where the target property is well-defined, but the original data dis- tribution fails embody this property. This scenario is common in scientific applications. where data may not naturally align with desired outcomes or properties [19].
PDM
Constraint Violation
Conditional Model
30
20
10
2
Time Step
Figure I: Sampling steps failing to converge to feasible solutions in conditional models (red) while minimizing the constraint diver- gence to under PDM (bluc).
Post-procetd post-processing correction. An alternative approach in- volves applying post-processing steps to correct deviations from desired constraints in the generated samples. This correction is typically implemented in the last noise removal stage, (2 1). Some approaches have augmented this process to use optimization solvers to impose constraints on synthesized samples [TO 19 21. However these approaches present two main limitations. First, their objective does not align with optimizing the score function. This inherently positions the diffusion model's role as ancillary. with the final synthesized data often resulting significant divergence from the learned (and original) data distributions, as we will demonstrate in $6 Second, these methods are reliant on limited and problem specific class of objectives and constraints. such as specific trajectory "constraints" or shortest path objectives which can be integrated as post-processing step [10 21
Other methods. Some methods explored modifying either diffusion training inference to adhere to desired properties. For instance, the methods in [ and [17], support simple linear or convex sets, respectively. Similarly, Fishman et al. g focus on predictive tasks within convex polytope, which are however confined to approximations by simple geometries like L.2-balls. While important contributions, these approaches prove insufficient for the complex constraints present in many real- world tasks. Conversely, in the domain of image sampling, Lou and Ermon (18) and Saharia et al. 23 introduce methods like reflections and clipping to control numerical errors and maintain pixel values within the standard [0,255] range during the reverse diffusion process. These techniques, while enhancing sampling accuracy, do not address broader constraint satisfaction challenges.
To overcome these gaps and handle arbitrary constraints, our approach casts the reverse diffusion process to a constraint optimization problem that is then solved throught repeated projection steps.
4 Constrained generative diffusion
This section establishes a theoretical framework that connects the reverse diffusion process as an optimization problem. This perspective facilitates the incorporation of constraints directly into the process, resulting in the constrained optimization formulation presented in Equation (6).
The application of the reverse diffusion process of score-based models is characterized by iteratively transforming the initial noisy samples 2t back to data sample 2n following the learned data distribution g(x)). This transformation is achieved by iteratively updating the sample using the
3

---

estimated score function Va, log (o)), where ( (D)) is the data distribution at time t. At each time step f, starting from zy. the process performs M iterations of Stochastic Gradient Langevin Dynamics (SGLD) 30
z 2z + T/ Vaz log q (2))) + 22
(3)
where e. is standard normal, Tt > is the step size, and Va; log (a2)/aa0) is approximated by the learned score function (x(x,, t).
4.1 Casting the reverse process as an optimization problem
First note that SGLD is derived from discretizing the continuous-time Langevin dynamics, which are governed by the stochastic differential equation:
d' = V log (()) dt dB)(,,
where (t) standard Brownian motion. Under appropriate conditions, the stationary distribution of this process is ((x;) (22), implying that samples generated by Langevin dynamics will, over time, be distributed according to (xx,). In practice, these dynamics are simulated using a discrete-time approximation, leading the SGLD update in Equation B Therein the noise term /2n c₁ allows
the algorithm to explore the probability landscape and avoid becoming trapped in local maxima.
Next notice that, detailed in 30 under some regularity conditions this iterative SGLD
algorithm converges toward stationary point, bounded by -ー²ーXー log(L/e), where, o² represents the variance schedule, A denotes the uniform spectral gap of the Langevin diffusion, and d is the dimensionality of the problem. Thus, the reverse diffusion process progresses towards 7 - 0, and the variance schedule decreases, the stochastic component becomes negligible, and SGLD transitions toward deterministic gradient ascent on log q(x)). In the limit of vanishing noise, the update rule simplifies to:
x
x + Y Va log g(:;
which is standard gradient ascent aiming to maximize log ((x)). This allow us to view the reverse diffusion process an optimization problem minimizing the negative log- ligelikel of the data distribution q(v, (en)) at each time step l.
In traditional score-based models, at any point throughout the reverse process, C; is unconstrained. When these samples required to satisfy some constraints, the objective remains unchanged, but the solution to this optimization must fall within a feasible region C, and thus the optimization problem formulation becomes:
minimize 2. 2..... 20
Σ
log (xx
(6a)
S.L.:
CT
20 E C.
(6b)
Operationally, the negative log likelihood is minimized at each step of the reverse Markov chain, as the process transitions from ØT to wo In this regard, and importantly, the objective of the PDM's reverse sampling process is aligned with that of traditional score-based diffusion models.
4.2 Constrained guidance through iterative projections
The score network sy (z,, f) directly estimates the first-order derivatives of Equation (6a), providing the necessary gradients for iterative gradient- based updates defined in Equation (3) In the presence of constraints (6b), however, an altermative iterative method is necessary to guarantee feasibility. PDM models projected guidance approach to provide this constraint-aware optimization process.
First, we define the projection operator, Pc, as a constrained optimization problem,
Pe(œ) = argmin |y - x0122
y
that finds the nearest feasible point to the input æc The cost of the projection || - =012 represents the distance between the closest feasible point and the original input.

---

To retain feasibility through an application of the projection operator after each update step, the paper defines projected diffusion model sampling step as
x Pe
(x) + Tt V log (( (z0) + 22
(8)
where C is the set of constraints and Pc is projection onto C. Hence, iteratively throughout the Markov chain, a gradient step is taken to minimize the objective defined by Equation (6a) while ensur- ing feasibility. Convergence is guaranteed for convex constraints sets [20] and empirical evidence in showccssss the applicabiliit of this methods to arbitrary constraint sets. Importantly, the projection
operators can be stam-started during the repeated sampling step providing piratical solution even for hard non-convex constrained regions. The full sampling process is detailed in Algorithm] |
Algorithm 1: PDM
209 ~ (0, oTI)
1
2 for f = T to do
By incorporating constraints throughout the sam- pling process, the interim learned distributions are steered to comply with these specifications. This is empirically evident from the pattern in Figure I (blue curves): remarkably, the constraint viola- tions decrease with each addition of estimated gradients and noise and approaches O-violation as I nears zero. This trend not only minimizes the impact but also reduces the optimality cost of pro- jections applied in the later stages of the reverse process. We provide thcoretical rationale for the effectiveness of this approach in Ind conclude this section by noting that this approach can be clearly distinguished from other methods which use diffusion model's ss sampling process to generate starting points for constrained optimization algorithm [10 21 Instead, PDM leverages minimization of negative log likelihood as the primary objective of the sampling algorithm akin to standard unconstrained sampling procedures. This strategy offers key advantage: the probability of generating a sample that conforms to the data distribution optimized directly, rather than an external objective, while simultaneously imposing verifiable construints. In contrast, existing baselines often neglect the conformity to the data distribution, which, as we will show in the next section, can lead to deviation from the learned distribution and an overemphasis external objectives for solution generation, resulting in significant divergence from the data distribution, reflected by high FID scores.
3
It f- of/207
for = 1 to M do
5
E ~ AO,
g t s8 (2) 1 (t)
o Pa + kg + /27c))
7 1-1 t xt
8 return =i
Effectiveness of PDM: A theoretical justification
Next, we theoretically justify the use of iterative projections guide the sample the constrained distribution. The analysis assumes that the feasible region C' is a convex set. All proofs are reported in the Appendix. We start by defining the update step.
Definition 5.1. The operator U defines single update step for the sampling process as,
u(=:)) a². + 7080 1) v /2
The next result establishes a convergence criteria on the proximity to the optimum, where for each time step there exists a minimum value of i such that,
aî s.t.
((ac)
+ % V V log qa [zo)
||pell2
(10)
where Pt is the closest point to the global optimum that can be reached via single gradient step from any point in C.
Theorem 5.2. Let Pc be a projection onto C, æ be the sample at time step and iteration and 'Error' be the cost of the projection 7 Assume V log p(x,) is convex. For any i 2 I,
E (r)))( C) > E [ C)]
(11)
The proof for Theorem 5.2 is reported in [ This result suggests that PDM's projection steps ensure the resulting samples adhere more closely to the constraints compared to samples generated
5

---

through traditional, unprojected methods. Together with the next results, it will allow us to show that PDM samples converge the point of maximum likelihood that also satisfy the imposed constraints.
The theoretical insight provided by Theorem 3..vvdovee an explanation for the observed discrepancy between the constraint violations induced by the conditional model and PDM, as in Figure]] T
Corollary 5.3. For arbitrary small there exist and i 2 I such that:
Error((((c(x))))) S.
The above result uses the fact that the step size "i is strictly decreasing and converges to zero, given sufficiently large T, and that the size of each update step U decreases with "/- As the step size shrinks, the gradients and noise reduce in size. Hence, (Pror(((Pc(a:) approaches zero with t, as illustrated in Figure T (right). This diminishing error implies that the projections gradually steer the sample into the feasible subdistribution of p(aro), effectively aligning with the specified constraints.
Feasibility guarantees. PDM provides feasibility guarantees when solving convex constraints. This assurance is integral in sensitive setrings, such material analysis (Section [61), plausible motion synthesis (Section 6.2). and physics-basee simulations (Section 6.4), where strict adherence the constraint set necessary.
Corollary 5.4. PDM arassi guarantees for convex constraint sets, for arbitrary density functions V æ, log p(z;).
Experiments
We compare PDM against three methodologies, each employing state-of-the-art specialized methods tailored to the various applications tested:: (1) Conditional diffusion models (Cond) [3] are the state- of-the-art methods for generative sampling subject to a series of specifications. While conditional diffusion models offer a way to guide the generation process towards satisfying certain constraints, they do not provide compliance guarantees. (2) To encourage constraints satisfaction, we additionally compare to conditional models with post-processing projection step (Condt), emulating the post- processing approaches of [10 various domains presented next. Finally, (3) we use a score-based model identical to implementation but with single post-processing projection operation (Post*) performed at the last sampling step. Additional details are provided in $C
The performance of these models are evaluated by the feasibility and accuracy of the generated samples. Feasibility assessed by the degree and rate at which constraints are satisfied, expressly, the percentage of samples which satisfy the constraints with given error tolerance. Accuracy is measured by the FID score, standard metric in synthetic sample evaluation. To demonstrate the broad applicability of our approach, our experimental settings have been selected to exhibit:
1. Behavior in low data regimes and with original distribution violating constraints ($6.1). as part of
a real-world material science experiment.
2. Behavior on 3-dimensional sequence generation with physical constraints ($..2)
3. Behavior on complex non-com con-convex constraints ($6.3).
4. Behavior on ODEs and under constraints outside the training distribution. (6..4)
6.1 Constrained materials (low data regimes and constraint-violatii distributions)
The first setting focuses on a real-world application in material science, conducted as part of an experiment expedite the discovery of cructure-propette linkages (please see $C for extensive additional details). From a sparse, uniform collection of microstructure materials, we aim to generate new structures with desired, previously unobserved porosity levels.
There are two key challenges in this setting: (1) Data sparsity: critical factor in this setting the cost of producing training data. Our dataset, obtained from the authors of [5], consists of 64 x 64 image patches subsampled from a 3, 000 3, 000 pixel microscopic image, with pixel values scaled to 1]. These patches are upscaled to 256 256 for model training. (2) Out-of-distribation
constraints: Constraints on the generated material's porosity, defined by pixels below a threshold representing damage, are far from those observed in the original dataset.

---

Porosity Constraint Satefaction
Generative Methods
100
Ground P(%)
PDM
Cond
Post
Cond
80
Satisfaction
60
40
30
Error Tolerance (%)
50
Figure Conditional diffusson model (Cond): Frequency of porosity con- straint satisfaction (y-axis) within error tolerance (x-axis) over 100 runs.
FID scores: 30.7
31,7415.6
41.7
46.4+10.7
Figure 3: Porosity constrained microstructure visualization at varying of the imposed porosity constraint amounts (P) and FID scores.
Previous work demonstrated the use of conditional GANs [ 11 to material generation, but these studies failed impose verifiable constraints on desired properties. To establish conditional baseline (denoted as Cond), we implement a conditional diffusion model, following the state-of- the-art approach by Chun al. [5). conditioning the sampling on porosity measurements. Figure 2 reports the constraint violations achieved by this model. The plot depicts the frequency of constraint satisfaction (y-axis) as a function of the error tolerance (x-axis), percentage. Observe that this state-of-the-art model struggles to adhere to the imposed constraints.
In contrast, PDM ensures both exact constraint satisfaction and identical image quality the conditional model, which is sigmificant given the complexity of the original data distribution. Figure 3 visualizes the outputs and FID scores obtained by our proposed model compared various baselines. The constraint correction step applied by Post* and Cond* leads to noticeable decrease in image quality, evident both visually and in the FID scores, rendering the generated images unsuitable for this application context. Additionally, we find that PDM outperforms Cond in generating microstructures that resemble those in the ground truth data (see ŞE These results are significant: the ability to precisely control morphological parameters in synthetic microstructures has broad impact in material synthesis, addressing critical challenges in data collection and property specification.
6.2
3D human motion (dynamic motion and physical principles)
Next we focus on dynamic motion generation adhering to strict physical principles using the challeng- ing HumanML3D dataset [12]. This benchmark employs three-dimenniooal figures across fourth dimension time to simulate motion. Thus, the main challenges here are generating 3D figures (1) including temporal component, while (2) ensuring they neither penetrate the floor nor float in the air, as proposed by Yuan et al. [32]. Previous studies have seen physical law violations in motion diffusion models, with none achieving col-ratee results [2] PDM's capability to overcome these limitations.
We next demonstrate
First, we remark that previous approaches, such as 132 relies on computationally demanding actions,
physics simulators to transform diffusion model predictions into using motion imitation policy trained via proximal policy optimization. This simulator is crucial to produce realistic results and dramatically alters the diffusion model's outputs. In contrast, and remarkably, PDM does not require such simulator. and inherently satisfies the non-penetration and non-floating constraints without external assistance, showing zero violations. For comparison, the best outcomes reported in 332 ranged from 0.918 to 0.998 for penetration violations and 2.601 to 3.173 for floating violations (see E.2 for more details).
Additionally, to more accurately assess the abilities of a diffusion model in the absences of physics simulator, we evaluate a conditional model with the same architecture as the PDM model, adapted from MotionDiffuse [33]. Results visualized in Figure demonstrate that PDM achieves outputs

---

Figure 4: PDM (left, FID: 0.71) and conditional (Cond) (tight, FID: 0.63) generation.
par with state-of-the-art FID scores. Additionally, the use of projections here is guaranteed to provide
exact constraint satisfaction, we will elaborate further in $
6.3 Constrained trajectories (nonconvex constraints)
The next experiment showcase the ability of PDM to handle nonconvex constraints. Path planning is classic optimization problem which is integral to finding smooth, collision-free paths in autonomous systems. This setting consists of minimizing the path length while avoiding path intersection with various obstacles in given topography. Recent research has demonstrated the use of diffusion models for these motion planning objectives [3]. In this task, the diffusion model predicts series of points, Po.P1.... Pi.....P2 PN. where each pair of consecutive points represents a line segment. The start and end points for this path are determined pscudo- ranudoll for each problem instance, with the topography remaining constant across different instances. Additionally, the problem presents obstacles at inference time (shown in red on Figure that were present during training, rendering a portion of the training data infeasible, and thus testing the generalization of these methods. The performance is evaluated on two sets of maps adapted from Carvalho al., shown in Figure The main challenge in this setting is the non-convex nature of the contraints.
To circumvent the challenge of guaranteeing collision-free free paths, previous methods have relied on sampling large batches of trajectories, selecting a feasible solution if available B.. We use the state-of- the-art Motion Planning Diffussion [3] conditional model baseline for this experiment and the associated datasets to train each of the models. For the Cond* model, we emulate the approach proposed by Power et al. (21], using the conditional diffusion model to generate initial points for an optimization solver. In this setting, the projection operator used by PDM is non-convex, and the implementation uses interior point method (28). While the feasible region is non-convex, approach never report unfeasible solutions as the distance from the learned distribution decreases, unlike other methods. In contrast, using the same method for single post-processing projection (Posrt) does not enhance the feasibility of solutions compared to the unconstrained conditional model (Cond), highlighting the limitations of these single corrections in managing local infeasibilities. These observations are consistent with the analysis of Figure [
Figure 5 Visually demonstrates that PDM can identify feasible path with just single sample. This capability marks a significant advance over existing state-of-the-an motion planning methods with diffusion models, as it eliminates the necessity of multiple inference attempts, which greatly affects the efficiency in generating feasible solutions. The experimental results (Table demonstrate the effectiveness of PDM in handling complex, non-convex constraints in terms of success percentage for single trajectories generated (top) and path length (bottom). Notices how, both the Cond and Cond
PDM
Cond (MPD) S
Cond
Tp
100.0
77.1 29.2
77.1 29.2
Tp
100.0 0.0
53.3 + 35.7
53.3 + 35.7
Tp
2.21
0.26
2.08 1. 0.51
2.08 .. 0.51
PL
Tp
2.05
0.17
2.09
0.31
2.09 11 0.31
Figure 6: Constrained trajectories evaluation on suc- cess percentage (S) for single run (higher the better, top) and path length, PL, (lower the better, bottom).
Figure 5: Constrained trajecotires synthetized by PDM on two topographies (Tpl, left and Tp2, right).
8

---

Positional Constraint Satisfaction
Earth (in distribution) PDM
Moon (out of distribution)
100
Ground
Post
Cond
Ground
PDM
Post
Cond
jout of dbstrittion)
Satisfaction
60
40
Error Tolerance (Pixets)
PDM
Post"
Cond
Cond"
Figure 8: Conditional diffusion model (Cond): Frequency of constraint satisfaction (y-axis) given an error tolerance (x-axis) over 100 runs.
FID
26.5
52.5
22.5 + 0.1
53.0
Figure 7: Sequential stages of the physics-informed models for in-distribution (Farth) and out-of-distribution (Moon) constraint imposition.
models fall short in finding feasible trajectories, taking shortcuts resulting in collisions. At the same time, notice how all methods find comparable path length within the reported tolerances. These results are significant: they show that PDM can not only handle complex, non-convex constraints, but also produce results that are on par with state-of-the-ad models in solution optimality.
6.4 Physics-informed motion (ODEs and -uttr--distributtin constraints)
Finally, we show the applicability of PDM in generating video frames adhering to physical principles. In this task, the goal generate frames depicting an object accelerating due gravity. The object's position in given frame is governed by
P: = P vve (v:+ (0.
au: Ōt
0p:
(12a)
Vi+1
+
(12b)
Ot
Ot
where p the object position, the velocity, and the frame number. This positional information can be directly integrated into the constraint set of PDM, with constraint violations quantified by the pixel distance from their true position. In our experiment, the training data based solely on earth's gravity and we test the model to simulate gravitational forces from the moon and other planets, in addition to earth. Thus there are two challenges this setting (1) satifying ODEs describing our physical principle and (2) generalize to out-of- aut-of-distribution constraints.
Figure (left) shows randomly selected generated samples, with ground images provided for reference. The subsequent rows display outputs from PDM, post- post-processing projection (Post), and conditional post-processing (Cond*), For this setting, we used state-of-the-art masked conditional video diffusion model, following Voleti et al. (27). Samples generated by conditional diffusion models are directly shown the figure, as the white object outline in the Cond+ frames shows where the Cond model originally positioned the object. Notice that, without constraint projections, the score-based generative model produce samples that align with the original data arbitrarily place the object within the frame (white ball outlines in the 3rd column). Post-processing repositions the object accurately but significantly reduces image quality. Similarly, Cond shows inaccuracies conditional model's object positioning, as indicated by the white outline in the 4th column. These deviations from the desired constraints are quantitatively shown in Figure S (light red bars), which depicts the proportion of samples adhering to the object's 's behavior constraints across varying error tolerance levels. Notably, this approach fails to produce any viable sample within a zero-tolerance error margin. In contrast, PDM generates frames that exactly satisfy the positional constraints, with FID scores comparable to those of Cond. Using the model proposed by Song et al. 226 further narrows this gap (see SD)
Next, Figure 7 (right) shows the behavior of the models in settings where the training data does not include any teasible data points. Here we adjust the governing equation (12) to reflect the moon's gravitational pull. Remarkably, PDM not only synthesizes high-quality images but also ensures no constraint violations (0-tolerance). This stands in contrasts to other methods, that show increased

---

constraint violations in out out-of- di-striibutton contexts, as shown by the dark red bars in Figure PDM can be adapted handle complex governing equations using ODEs and can be guarantee satisfaction of out-of cul-of-distribution constraints with no decrease in sample quality.
7 Discussion and limitations
In many scientific and engineering domains and safety-critical applications, constraint satisfaction guarantees a critical requirement. It is however important to acknowledge the existence of an inherent trade-off, particularly in computational overhead. applications where inference time is a critical factor, it may be practical to adjust the time step t at which iterative projections begin, which guides a trade-off between the FID score associated with the starting point of iterative projections and the computational cost of projecting throughout the remaining iterations ($F). Other avenues to improve efficiency also exists, from the adoption of specialized solvers within the application domain
of interest to the adoption of warm-start strategies for iterative solvers. The latter, in particular, relies exploiting solutions computed in previous iterations of the sampling step and was found to be practical strategy substantially decrease the projections overhead.
We also note the absence of constraints in the forward process. As illustrated empirically, it unnecessary for the training data to contain any feasible points. We hold that this not only applies to the final distribution but to the interim distributions as well. Furthermore. by projecting perturbed samples, the cost of the projection results in divergence from the distribution that is being learned. Hence, we conjecture that incorporating constraints into the forward process will not only increase computational cost of model training but also decrease the FID scores of the generated samples.
Finally, while this study provides a framework for imposing constraints on diffusion models, the representation of complex constraints for multi-task large scale models remains an open challenge. This paper motivates future work for adapting optimization techniques to such settings, where constraints ensuring accuracy in task completion and safety in model outputs bear transformative potential to broaden the application of generative models in many scientific and engineering fields.
8 Conclusions
This paper was motivated by significant challenge in the application of diffusion models in contexts requiring strict adherence to constraints and physical principles. It presented Projected Diffusion Models (PDM), an approach that recasts the score-based diffusion sampling process constrained optimization process that can be solved via the application of repeated projections. Experiments in domains ranging from physical-informer motion for video generation governed by ordinary differentiable equations, trajectory optimization in motion planning, and adherence to morphometric properties in generative material science processes illustrate the ability of PDM to generate content of high-fidelity that also adheres to complex non-convex constraints as well as physical principles.
Acknowledgments
This research is partially supported by NSF grants 2334936, 2334448, and NSF CAREER Award 2401285 185. Fioretto is also supported by an Amazon Research Award and Google Research Scholar Award. The authors acknowledge Research Computing at the University of Virginia for providing computational resources that have contributed to the results reported within this paper. The views and conclusions of this work are those the authors only.
Authors Contributions
JC and FF formulated the research question, designed the methodology, developed the theoretical analysis, and wrote the manuscript. Moreover, JC contributed to developing the code and performed the experimental analysis. SB acquired the data for the micro-structure experiment, formulated the desired properties for such experiment, and participated in the interpretation of the results.
10

---

References
[1] Namrata Anand and Tudor Achim. Protein strueture and sequence generation with equivariant denoising diffusion probabilistic models. arXiv preprint arXiv:2205.15019 2022.
[2] Chentao Cao. Chuo-Xu Cui, Yue Wang, Shaonan Liu, Taijin Chen, Hairong Zheng, Dong Liang, and Yanjie Zhu. High-freag frequecy space diffusion model for accelerated mri. IEEE Transactions on Medical Imaging, 2024.
[3] Joao Carvalho, An T Le, Mark Baierl, Dorothea Koert, and Jan Peters. Motion planning diffusion: Learning and planning of robot motions with diffusion models. In 2023 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 1916-1923. IEEE, 2023.
[4] Joseph B. Choi, Phong C. H. Nguyen, Oishik Sen, H. S. Udaykumar, and Stephen Baek. Artifi- cial intelligence approaches for energetic materials by design: State of the art, challenges, and future directions. Propellants, Explosives, Pynotechnicx, 2023. doi: 10. 20.1002/prep.20220000 httpttttss//onnnnelllel
wi wiley. http//aaww.aaal.oooo///ooooo/oo. 1002/ ptt.:/prep.20200077....
[5] Sehyun Chun, Sidhartha Roy. Yen Thi Nguyen, Joseph B Choi, HS Udaykumar, and Stephen S Baek. Deep learning for synthetic microstructure generation in materials-by-dest framework for heterogeneous energetic materials. Scientific reports, 10(1):13307, 2020.
[6] Hyungjin Chung and Jong Chul Ye. Score-based diffusion models for accelerated mri. Medical image analysis, 80: 102479, 2022.
[7] Nic Fishman, Leo Klamer, Valentin De Bortoli, Emile Mathieu, and Michael Hutchinson. Diffusion models for constrained domains. arXiv preprint arXiv:2304.05334, 2023.
[8] Nic Fishman, Leo Klarner, Emile Mathieu, Michael Hutchinson, and Valentin De Bortoli. Metropolis sampling for constrained diffusion models. Advances in Neural Information Pro- cessing Systems, 36, 2024.
[9] Thomas Frerix, Matthias Nießner, and Daniel Cremers. Homogeneous linear inequality con- straints for neural network activations. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition Workshops, pages 748-749, 2020.
[10] Giorgio Giannone, Akash Srivastava, Ole Winther, and Faez Ahmed. Aligning optimiza- tion trajectories with diffusion models for constrained design generation. arXiv preprint arXiv:2305.18440, 2023.
[11] Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, pages 2672-2680, 2014.
[12] Chuan Guo, Shihao Zou, Xinxin Zuo Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152-5161, 2022.
[13] Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207. 12598, 2022.
[14] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
[15] Emiel Hoogeboom, Victor Garcia Satorras, Clément Vignac, and Max Welling. Equivariant diffusion for molecule generation in 3d. In International conference machine learning, pages 8867-8887. PMLR, 2022.
[16] Michael Janner, Yilun Du, Joshua B Tenenbaum, and Sergey Levine. Planning with diffusion for flexible behavior synthesis. arXiv preprint arXiv:2205.0999) 2022.
[17] Guan-Horng Liu, Tianrong Chen, Evangelos Theodorou, and Molei Tao. Mirror diffusion models for constrained and watermarked generation. Advances Neural Information Processing Systems, 36, 2024.
11

---

[18] Aaron Lou and Stefano Ermon. Reflected diffusion models. In International Conference on Machine Learning, pages 22771-22701. PMLR, 2023.
[19] François Mazé and Faez Ahmed. Diffusion models beat gans on topology optimization. In Proceedings of the AAAI Conference on Arlificial Intelligence (AAAI), Washinglon, DC, 2023.
[20] Neal Parikh and Stephen Boyd. Proximal algorithms. Foundations and Trends in Optimization, 13):1127-239, 2014.
[21] Thomas Power, Rana Soltani-Zarrin, Soshi lba, and Dmitry Berenson. Sampling constrained trajectories using composable diffusion models. In IROS 2023 Workshop on Differentiable Probabilistic Robotics: Emerging Perspectives on Robot Learning, 2023.
[22] Gareth O. Roberts and Richard L. Tweedie. Exponential convergence of langevin distributions and their discrete approximations. Bernoulli, 2(4):341-363, 1996.
[23] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-imase image diffusion models with deep language understanding. Advances in neural information processing systems, 35:36479-36644 36494, 2022.
[24] Jascha Bohl-DDccsttein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[25] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
[26] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13455 2020.
[27] Vikram Voleti, Alexia Jolicoeur-Martin Martineau, and Chris Pal. Mevd-masked conditional video dif- fusion for prediction, generation, and interpolation. Advances in Neural Information Processing Systems, 35:23371-22533 2022.
[28) Andreas Wächter and Lorenz T Biegler. On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear programming. Mathematical programming, 106: 25-57, 2006.
[29] Tsun-Hsuan Wang, Juntian Zheng, Pingchuan Ma. Yilun Du, Byungchul Kim, Andrew Spiel- berg, Joshua Tenenbaum, Chuang Gan, and Daniela Rus. Diffusebot: Breeding soft robots with physics-augmented generative diffusion models. arXiv preprint arXiv:2311. 1. 1703,, 2023.
[30] Max Welling and Yee W Teh. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th international conference machine learning (ICML-11), pages 681-688, Citeseer, 2011.
[31] Pan Xu, Jinghui Chen, Difan Zou, and Quanquan Gu. Global convergence of langevin dynamics based algorithms for nonconvex optimization. Advances in Neural Information Processing Systems, 31, 2018
[32] Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics- gusise-gudddddd human motion diffusion model. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 16010-16021, 2023.
[33] Mingyuan Zhang, Zhongang Cai, Liang Pan, Fangzhou Hong, Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model, 2022.
[34] Ziyuan Zhong, Davis Rempe, Danfei Xu, Yuxiao Chen, Sushant Veer. Tong Che, Baishakhi Ray, and Marco Pavone. Guided conditional diffusion for controllable traffic simulation. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 3560-3566. IEEE, 2023.
12

---

A Broader impacts
The development of Projected Diffusion Models (PDM) may significantly enhance the application of diffusion models in fields requiring strict adherence to specific constraints and physical principles. The proposed method enables the generation of high-fidelity content that not only resembles real- world data but also complies with complex constraints, including non-convex and thysical-baase specifications. PDM's ability to handle diverse and challenging constraints in scientific and engineer- ing domains, particularly in low data environments, may potentially lead to accelerating innovation and discovery in various fields.
B Expanded related work
Diffusion models with soft constraint conditioning. Variations of conditional diffusion models L3 serve as useful tools for controlling task specific outputs from generative models. These methods have demonstrated the capacity capture properties of physical design (2)), positional awareness (3), and motion dynamics (32) through augmentation of these models. The properties imposed in these architectures can be viewed as soft constraints, with stochastic model outputs violating these loosely imposed boundaries.
Post- Prot-procsssing optimization. In settings where hard constraints needed to provide meaningful samples, diffusion model outputs have been used as starting points for a constrained optimization algorithm. This has been explored in non-convex settings, where the starting point plays an important role in whether the optimization solver will converge to a feasible solution (21]. Other approaches have augmented the diffusion model training objective to encourage the sampling process emulate an optimization algorithm, framing the post-processing steps as an extension of the model [10 19]. However, an existing challenge in these approaches is the reliance on an easily expressible objective. making these approaches effective in a limited set of problems (such as the constrained trajectory experiment) while not applicable for the majority of generative tasks.
Hard constraints for generative models. Frerix et al. [9] proposed an approach for implementing hard constraints the outputs of autoencoders. This was achieved through scaling the generated outputs in such a way that fcasibility was enforced, but the approach is to limited simple linear constraints. (17) proposed approach imposing constraints using "mirror mappings" with applicability exclusively to common, convex constraint sets. Due to the complexity of the constraints imposed in this paper, neither of these methods were applicable to the constraint sets explored in any of the experiments. A Alternatively, work by Fishman et al. 2023 2024) broadens the classes of constraints that can be represented but fails to demonstrate the applicability of their approach to a empirical settings similar to ours, utilizing an MLP architecture for trivial predictive tasks with be represented by convex polytopes. We contrast such approaches to our
constraints sets that work, noting that this prior work is limited to constraint sets that can be approximated by simple neighborhoods, such as an b2lb simplex, or polytope, whereas PDM can handle constraint of arbitrary complexity.
Sampling process augmentation. Motivated by the compounding of numerical error throughout the reverse diffusion process, prior work has proposed inference time operations to bound the pixel values of an image dynamically while sampling [18 23). Proposed methodologies have either applied reflections or simple clipping operations during the sampling process, preventing the generated image from significantly deviating from the [0,255] pixel space. Such approaches augment the sampling process in a way that mirrors our work, but these methods are solely applicable to mitigating sample drift and do not intersect our work in general constraint satisfaction.
C Experimental settings
In the following section, further details are provided as to the implementations of the experimental settings used in this paper.
13

---

Generative Methods
Ground
P(
PDM
Cond
Post
Cond
10
20
40
Figure 9: Porosity constrained microstructure visualization at varying of the imposed porosity constraint amounts (expanded from Figure 3).
C.1 Constrained materials
Microstructures are pivotal in determining material properties. Current practice relies physics- based simulations conducted upon imaged microstructures to quantify intricate drructure-propert linkages [4]. However, acquiring real material microstructure images is both costly and time- consuming, lacking control over attributes like porosity, crystal sizes, and volume fraction, thus necessitating "cut-and-try" experiments. Hence, the capability generate realistic synthetic material microstructures with controlled morphological parameters can significantly expedite the discovery of properyy-property linkages.
Previous work has shown that conditional generative adversarial networks (GAN) [T]] can be used for this end (5), but these studies have been unable to impose verifiable constraints on the satisfaction of these desired properties. To provide a conditional bascline, we implement a conditional DDPM modeled after the conditional GAN used by Chun al. 5 with porosity measurements used to condition the sampling.
Projections. The porosity of image represented by the number of pixels in the image which are classified as damaged regions of the microstructure. Provided that the image pixel intensities are scaled [-1, 1]. a threshold is set zero, with pixel intensities below this threshold being classified as damage regions. To project, we implement top-k algorithm that leaves the lowest and highest intensity pixels unchanged, while adjusting the pixels nearest to the threshold such that the total number of pixels below the threshold precisely satisfies the constraint.
The conditional baseline conditioned on the porosity values of the training samples. The implementation of this model is described by Ho and Salimans
Conditioning.
14

---

Earth (in distribution)
Moon (out of distribution)
Ground
PDM
Post
Cond
Ground
PDM
Post
Cond
Figure 10: Sequential stages of the physics-inl -nfforee models for in-distribution (Earth) and out-of-distribuics (Moon) constraint imposition (expanded from Figure 7
Original training data. We include samples from the original training data visually illustrate how closely our results perform compared to the real images. As the specific porosities we tested on are not adhered to the dataset, we illustrate this here as opposed to in the body the text.
We observe that only the Conditional model and PDM synthesize images that visually adhere to the distribution, while post-procst processoc methods do not provide adequate results for this complex setting.
C.2 3D human motion
The penetration and floatation constraints can be handled by ensuring that the lowest point on the z-axis is equal the floor height. Additionally, to control the realism of the generated figures, we impose equality constraints on the size of various body parts, including the lengths of the torso and appendages. These constraints can be implemented directly through projection operators.
Projections.
The model is directly conditioned on text captions from the HumanML3D dataset. The implementation is described in 333
Conditioning.
C.3 Constrained trajectories
Projections. For this experiment, we represent constraints such that the predicted path avoids intersecting the obstacles present in the topography. These are parameterized to non-convex interior point method solver. For circular obstacles, this can be represented by minimum distance requirement, the circle radius, imposed on the nearest point to the center falling on a line between Pa and Pn+1. These constraints are imposed for all line segments. We adapt similar approach for non-circular obstacles by composing these of multiple circular constraints, hence, avoiding over- constraining the problem. More customized constraints could be implement to better represent the feasible region, likely resulting in shorter path lengths, but these were not explored for this paper.
The positioning of the obstacles in the topography are passed into the model as vector when conditioning the model for sampling. Further details can be found the work presented by Carvalho et al.) from which this bascline was directly adapted.
Conditioning.
15

---

C.4 Physics-informed motion
The dataset is generated with object starting points sampled uniformly in the interval [0. 63). For each data point, six frames are included with the position changing defined in Equation and the initial velocity Vo = 0. Pixel values are scaled to [-1, lj. The diffusion models are trained on 1000 points with 90/10 train/test split.
Projections. Projecting onto positional constraints requires a two-step process. First, the current position of the object is identified and all the pixels that make up the object set the highest pixel intensity (white), removing the object from the original position. The set of pixel indices representing the original object structure correct position, as computed by the constraints, as each pixel from the original structure is placed onto the center point of the true position. Hence, when the frame is feasible prior to the projection, the image is returned unchanged, which is consistent with the definition of a projection.
stored for the subsequent step. Next, the object is moved to the
Conditioning. For this setting, the conditional video diffusion model takes two ground truth frames as imputs, from which it infers the trajectory of the object and the starting position. The model architecture is otherwise as specified by Voleti et al.
D PDM for score-based generative modeling through stochastic differential equations
D.1 Algorithms
While the majority of our analysis focused on the developing these techniques to the sampling architecture proposed for Noise Conditioned Score Networks [25], this approach can directly be adapted to the diffusion model variant Score-Based Generative Modeling with Stochastic Differential Equations proposed by (Song al. Although our observations suggested that optimizing across continuum of distributions resulted in less stability in diverse experimental settings, we find that this method is still effective in producing high-quality constrained samples in others.
We included an updated version of Algorithm adapted these architectures.
Algorithm 2: PDM Corrector Algorithm
X NO, o2 o I) 2 for f T to do
3
for i t 1 1o M do
6 ~ 10,
g - (x ot)
7 - 2(r||c|12/ g )2
x - Pa(x + ng + 2
8
1-1 - x
return x
We note that primary discrepancy between this algorithm and the one presented in Section 4.2 the difference in y,. As the step size is not strictly decreasing, the guidance effect provided by PDM is impacted as Corollary (03]larr5333deee not hold for this approach. Hence, we do not focus on this architecture for our primary analysis, instead providing supplementary results in the subsequent section.
D.2 Results
We provide additional results using the Score-Based Generative Modeling with Stochastic Differential Equations. This model produced highly performative results for the Physics-informed Motion experiment, with visualisations included Figures TT and [2] This model averages an impressive inception score of 24.2 this experiment, slightly outperforming the PDM implementation for Noise
16

---

Frame
Frame
Frame 3
Frame 4
Frame
Ground
PDM (SDE)
Figure 11: In distribution sampling for physics-infomned model via Score-Based Generative Modeling with
SDEs.
Frame
Frame
Frame 3
Frame
Frame
Ground
PDM (SDE)
Figure distribution sampling for phyysis--nfformed model via Score-Based Generative Modeling with SDEs.
Conditioned Score Networks. Furthermore, it is equally capable in generalizing to constraints that not present in the training distribution.
E
Additional results
E.1
Constrained materials morphometric parameter distributions
0.20
50
Cond
0.35
Ground
40
0.6
0.10
20
0.4
10
1.00
30
Dveie(ju))
o(")
Figure 13: Distributions of the morphometric parameters, comparing the ground truth to PDM and Cond models using heuristic-bast analysis.
17

---

When analyzing both real and synthetic materials, heuristic- guided metrics are often employed to extract information about microstrucutres present in the material. When analyzing the quality of synthetic samples, the extracted data can then be used to assess how well the crystals and voids in the microstructure adhere to the training data, providing an additional qualitative metries for analysis. To augment the metrics displayed within the body of the paper, we include here the distribution of three metries describing these microstructures, mirroring those used by Chun et al.
We observe that the constraint imposition present in PDM improves the general adherence of the results to the ground truth microstructures. This suggests that the Cond model tends to generate to certain microstructures at frequency that is not reflected in the training data. By imposing various porosity constraints, PDM able generate more representative set of microstructures in the sampling process.
E.2 3D human motion
We highlight that unlike the approach proposed by (32, our approach guarantees the generated motion does not violate the penetrate and float constraints. The results are tabulated in Table (Ieft) and report the violations in terms of measured distance the figure is either below (penetrate) or above (float) the floor. For comparison, we include the projection schedules utilized by PhysDiff which report the best results to show that even in these cases the model exhibits error.
Method
FID
Penetrate
Float
PhysDiff 33 (Start 3, End I)
0.51
0.918
3.173
PhysDiff 32
(End 4, Space I)
0.43
0.998
2.601
PDM
0.71
0.00
0.00
Table 1: PDM performance compared to (best) Phys results on HumanML3D.
The implementation described by 32 applies physics simulator at scheduled intervals during the sampling process map the diffusion model's prediction "physically-plausibbe action that imitates data points of the training distribution. This simulator dramatically alters the diffusion model's outputs utilizing a learned motion imitation policy, which has been trained to match the ground truth samples using proximal policy optimization. In this setting the diffusion model provides starting point for the physics simulator and is not directly responsible for the final results of these predictions. Direct parallels can be drawn between this approach and other methods which solely task the diffusion model with initializing an external model 10 21 Additionally, while the authors characterize this mapping as projection, it is critical to note that this is a projection onto the learned distribution of the simulator and not projection onto a feasible set, explaining the remaining constraint violations in the outputs.
Projection Cost Upper Bound
E.3 Convergence of PDM
FBM Upper
Error (UbK)) C) (log)
As shown in Figure I the PDM sampling process con- verges to feasible subdistribution, a behavior that is gen- erally not present in standard conditional models. Corol- lary [.3 provides insight into this behavior as it outlines the decreasing upper bound on 'Error' that can be intro- duced in a single sampling step. To further illustrate this behavior, the decreasing upper bound can be illustrated in Figure
Exrror (UXx))
ΣError (ULPEDE)))
10
2
Time Stop
Figure 14: Visualization of the decreasing upper bound on error introduced a single sampling step for PDM, as opposed to the strictly increasing upper bound of conditional (Cond) models.
F Computational costs
To compare the computational costs of sampling with PDM to our baselines, we record the execution times for the reverse process of a single sample. The implementations of PDM have not been optimized for runtime, and represent an upper bound. All sampling is run two NVIDIA A100 GPUs. All computations conducted on these GPUs with
18

---

the exception of the interior point method projection used in the 3D Human motion experiment and the Constrained Trajectories experiment which runs on two CPU cores.
Constrained Materials
3D Human Motion
Constrained Trajectories
Physics-informed Motion
PDM
26.89
682.40*
383,40"
48.85
Posr
26.01
27.58
Cond
18.51
13.79
0.56
35.30
Cond
18.54
106.41
36.63
Table 2: Average sampling run-time in seconds.
We implement projections at all time steps in this analysis, although practically this can optimized to reduce the total number of projections described in the subsequent section. Additionally, set M = 100 and = 10 for each experiment. The increase in computational cost present in PDM is directly dependant the tractability of the projections and the size of M.
The computational cost of the projections is largely problem dependant, and we conjecture that these times could be improved by implementing more efficient projections. For example, the projection for Constrained Trajectories could be dramatically improved by implementing this method on the GPUs instead of CPUs (*). However, these improvements are beyond the scope of this paper. Our projection implementations are further described in $C
Additionally. the number of iterations for each f can often be decreased below M = 100 or the projection frequency can be adjusted (as has been done for in this section for the CPU implemented projections), offering additional speed-up.
G Variational lower bound training objective
As defined in Equation 2, PDM uses coore-matching objective learn the gradients of the log probability of the data distribution. This understanding allows the sampling process to be framed drawn between the
light that is consistent to optimization theory, allowing equivalences proposed sampling procedure and projected gradient descent. This aspect is also integral to the theory presented in Section 4.2.
Other DDPM and DDIM implementation utilize a variation lower bound objective, which is a tractable approach minimizing the negative log likelihood the network's noise predictions. While this approach was inspired by the score-matching objective, we empirically demonstrate that iterative projections perform much worse in our tested settings than models optimized using this training
Frame 1
Frame
Frame 3
Frame 4
Frame
Material generation Physics-inf inf. motion
Figure 15: Iterative projections using model trained with variational lower bound objective.
19

---

objective, producing clearly inferior solutions in the Physics- Physics-informen informed experiments and failing to produce viable solutions in the material science domain explored.
This approach (visualized in Figure 15 resulted in an FID score of 113.8 + 4.9 on the Physics- informed Motion experiment and 388.2 13.0 the Constrained Materials experiment, much higher than those produced using the score acore-matching objective, adopted in our paper. We hold that this is because the approach proposed in our paper more theoretically sound when framed in terms of gradient-based sampling process.
H Missing proofs
Proof of Theorem 5.2
Proof. By optimization theory of convergence in a convex setting, provided an arbitrarily large number of update steps M, at will reach the global minimum. Hence, this justifies the existence of I as some iteration as co, thereafter.
xt + "/ Vaj log [(0] s l|eell2 will hold for every iteration
Consider that gradient step is taken without the addition of noise, and i > T. Provided this, there are two possible cases.
Case 1:
Assume æi 7 at log ((x)) mn) is closer to the optimum than Pt- Then, ot is infeasible.
This claim is true by the definition of pt. æ; + Yi V xe log q(x) (zo) is closer j than is achievable from the nearest feasible point to μ.. Hence, x; must be infeasible. Furthermore, the additional gradient step produces a point that is closer to the optimum than possible by a single update step from the feasible region. Hence it holds that
Error(æ) + V =x log q(
((2) (o)) > (rror(Pa(a)) + T V Pe(a:) log (PP((t)))
(13)
as the distance from the feasible region to the projected point will be most the distance to pr. As this point is closer to the global optimum than (t. the cost of projecting of + 7 Vay log (o)) is greater than that of any point that begins in the feasible region.
Assume कt + 7 Var log g(rr (z)) is equally close the optimum as pe. In this case, there are two possibilities; either (1) xf is the closest point C to or (2) x is infeasible.
Case 2:
If the former is true, Ø;- (x]), implying
Error(x) + 7/ Va, =! log g ( (oo) = Error(P((x)) + 7h V Pe(a)) log (PC((()))))))
(14)
Next, consider that the latter is true. If x; is not the closest point in to the global minimum, then must be equally close point to y that falls outside the feasible region. Now, subsequent gradient step of æ; will be the same length as a gradient step from the closest feasible point to μa, by our assumption.
Since the feasible region and the objective function are convex, this forms triangle inequality, such that the cost of this projection is greater than the size of the gradient step. Thus, by this inequality, Equation applies.
Finally, for both cases we must consider the addition of stochastic noise. As this noise is sampled from the Gaussian with a mean of zero, we synthesize this update step as the expectation over.
Error(a + 7 Vay log logg(o;(zo) + /271t)) > E [rroo((P((x)) + 7: Po log (Po((()))) + v 22
(15)
or equivalently as represented in Equation I1
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The contributions of this paper defined succinetly in the abstract, with clear references to the experimental settings explored in the manuscript
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Discussion of limitations and open questions left in this work are described in
Section 7
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiiatinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
21

---

Answer: [Yes]
Justification: Yes, all assumptions are clearly defined the body of the paper, and the missing proofs are included in Appendix H
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: Code and datasets have been provided, containing the necessary components reproduce the experiments conducted.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
22

---

reproducible, they Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The code and datasets for all experiments are provided in the supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
Cooee//ddeeeSbmimaia ssi onPolicy) for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: All necessary settings and details provided in Appendix Cor or the provided code.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars are provided in the reported metrics.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
23

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful figures symmetric error bars that would yield results that are out of range (e.g. negative rates).
show in tables or
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
Details on the used compute resources are described in Appendix F
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't I make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: NeurIPS ethical guidelines have been reviewed by the authors, and this research meets the described standards.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive socictal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The societal impacts of this work are discussed in Appendix A
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This work and the associated resources do not pose risks of misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Existing assets have been used with consideration of licensing, and the original owners have been credited in the manuscript.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
25

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
batasets.papersstt
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Details on assets (models/code//atta) have been documented in the manuscript as well as the code base provided.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: Human subjects were not involved in the research conducted.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: Human subjects were not involved in the research conducted.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
26

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
27


Reviews:
[{"review_id": "qehKNe69bH", "reviewer": "Reviewer_Jdpw", "summary": "This paper proposes a diffusion model that imposes constraints on the generated output. However, the constraints here are not abstract verbal instructions but rather formalizable constraints. The authors propose projected diffusion model sampling to perform constraint conditional log-likelihood maximization at each time step. They demonstrate the effectiveness of the proposed method on a variety of application problems.", "questions": "1. Please provide a more detailed explanation of the similarity between the reverse diffusion process of score-based models and maximizing the conditional density function.\n2. As mentioned in Reference [20], it is common to gradually decrease the learning rate and $\\gamma$ to improve the convergence rate of the proximal algorithm. Have you tried decreasing $\\gamma$ as $1/i$ or $1 / \\sqrt i$ within the inner loop of projected diffusion model sampling?\n3. Please provide the computational order of the projection algorithm used in each experiment.", "limitations": "Section 7 adequately describes the limitations.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The paper demonstrates the effectiveness of the proposed method under various constraint conditions.\n* Section F reports on the impact of projection on computational cost.\n* The paper compares the proposed method to appropriate existing methods, such as post-processing correction.", "weaknesses": "1. Equation (3) is introduced without a clear explanation of the relationship between the reverse diffusion process of score-based models and maximizing the conditional density function.\n2. The details of the projection algorithm used in each experiment are only described in words. Pseudocode would be helpful.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your time and efforts in providing feedback on our paper. First, let us emphasize the significant contribution provided by our work: Our proposed method provides constraint imposition with _formal guarantees_ for several important classes and for the first time in the general and complex form posed! All prior work has either been unable to cope with the complex constraints that are crucial for the scientific and engineering application of interest (which include ODEs, nonconvex constraint sets, and real-world, scientific applications), relied on costly post-processing methods that we have shown _fail to produce meaningful samples in the diverse and real-world set_ of studied domains, and have failed to supply formal guarantees of constraint satisfaction. We appreciate your acknowledgment of our use of \"formalizable constraints,\" as we view this as a key contribution that distinguishes our work from existing literature. Let us reiterate that the method we propose is able to handle _arbitrary constraint sets_, meaning that this can generalize to *any formal constraints that are posed.* We believe these to be compelling reasons as to why our paper should be considered!\n\n>**Weakness 1 and Question 1: Equation (3) is introduced without a clear explanation of the relationship between the reverse diffusion process of score-based models and maximizing the conditional density function.**\n\nThe objective of the reverse diffusion process is to maximize the density function, and Equation 4 shows the _actual_ update steps that are learned by the score-based model: a noisy (SGLD) gradient ascent on the density function. The formalization of the objective provided in Equation (3a) is equivalent by construction to that of the reverse diffusion process as _this is the optimization procedure that is learned during the forward diffusion process_.\nThe model has learned to estimate the gradients necessary to solve this optimization problem, and the update step in Equation 4, which is taken directly from Song et al. [24,25], converges to a solution (or sample) that maximizes the density function. **In short, Equation (3a) is how the reverse diffusion process would be directly formalized as an optimization problem, and Equation (3b) is our method's extension of this objective to a constrained optimization problem.** We hope this clarifies your doubt. \n\n>**Weakness 2: The details of the projection algorithm used in each experiment are only described in words. Pseudocode would be helpful.**\n\nWe appreciate the interest in the lower-level implementation details. First, notice that the actual implementations for each of these projections **are indeed included in our submission**, and if you are interested we would encourage you to look at the provided code. Additionally, we would encourage you to refer to our response to _Reviewer p9zr Weakness 1_ where we provide the mathematical formalization of the projections.\n\n>**Question 2: As mentioned in Reference [20], it is common to gradually decrease the learning rate and $\\gamma$ to improve the convergence rate of the proximal algorithm. Have you tried decreasing $\\gamma$ as $1/i$ or $1/\\sqrt{i}$ within the inner loop of projected diffusion model sampling?**\n\nIndeed, our implementation in Section D (Algorithm 2) does provide a dynamic adjustment of $\\gamma$ which uses stochastic differential equations to create smoother transition kernels between timesteps [25]. We find that for our physics-informed motion experiments this produces better FID scores, but in most of our experiments, $\\gamma$ has been adjusted at the outer loop level, thus while decreasing with time it does not provide as a smooth of a change as that explored in Section D, without producing meaningful alteration to the algorithm performance.\n\n>**Question 3: Please provide the computational order of the projection algorithm used in each experiment.**\n\nWe note that additional details for each projection are also reported in Section C, as we mentioned in the main paper. We provide here additional details on the problems' complexity:\n- **Constrained Materials:** This is a knapsack constraint. While in general knapsack problems are NP-complete, the version adopted (which uses integers as weight) is known to be weakly NP-complete and admits a fully polynomial approximation. It is solved efficiently in $O(nm)$ where $n$ is the number of pixels and $m$ is the number of values to compute for the dynamic program. \n- **3D Human Motion:** This is a scaling constraint that runs in $O(n)$ time, where $n$ is the length of the internal representation.\n- **Constrained Trajectories:** The interior point method used by the nonconvex solver in our implementation [27] has a time complexity of $O(n^{3.5})$ where $n$ is the number of variables in the Quadratic Program. In this case, there are 128.\n- **Physics-informed Motion:** This projection runs in $O(n+m)$ where the size of each image is $n$ by $m$.\n\nWe will be happy to add this additional information to the appendix in our final version. \n\n---\nWe appreciate the constructive feedback and have diligently addressed all the concerns raised. We believe that our detailed responses provide the necessary clarifications for the arguments presented in our paper. In light of our clarifications, we kindly request that you re-evaluate your score, particularly given that the only weaknesses identified were minor points that have now been fully addressed.\n\nWe also noticed your current score for soundness = 1. This appears to be unjustified, especially considering the rigorous methodological approach and theoretical contributions presented. If the score was intended to be revised following our responses, we hope that the comprehensive explanations provided will justify a favorable assessment. If there are any lingering doubts or additional questions, we are more than willing to discuss them further! Thank you.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sA6ySdBJiW", "reviewer": "Reviewer_e4oh", "summary": "This paper proposed Projected Diffusion Models (PDM) inspired by stochastic gradient Langevin dynamics, to generate samples that satisfy given arbitrary constraints and remain within the specified regions. The authors claimed that the proposed algorithm is compatible across various applications, including satisfying morphometric properties when synthesizing materials, physics-informed motion generation, constrained path planning and human motion generation, and provided theoretical analysis to guarantee the generated samples reside in the constrained regions.", "questions": "Following the weaknesses above,\n\n- For Equation (3b), it requires the whole trajectory in the reverse process satisfying the constraints, while $x_t$s' while $t$ is large are basically noise. Is this optimization setup reasonable?  \n- What's the time complexity to find the nearest feasible point while doing the projection? Will take it too long time? How do you choose $M$ in the algorithm 1 to control the number of Langevin Dynamics being applied?\n- While doing projections, from my understanding, the projected feasible point should be on the boundary of the constraints. Does that mean the final samples will be all projected on the exact boundary then? \n\nTo summarize this question, even though PDM has better metrics than others, I still doubt the sampling distribution is not aligned with the data distribution since the samples might be all on the constraint boundaries.\n\n- When evaluating out-of-distribution samples, is the constraint satisfaction rate the only metric?", "limitations": "Limitations and positive societal impact are discusses in the paper, but negative societal is not mentioned.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The approach this proposed achieves zero-violation to constraints during sampling process.\n2. Compared against several existing approaches in the various experiments.", "weaknesses": "1. In limitation discussion, computational overhead is mentioned. How's the time complexity of PDM compared to ``Cond``, ``Cond+`` and ``Post+``?\n2. Not quite sure if I really understand how ``Cond+`` is different from PDM in general except the Langevin dynamics part from the descriptions but perceive the differences between them and if I am correct, ``Cond`` is in classifier-free guidance regime, ``Cond+`` is in classifier guidance regime and ``Post+`` only applies projection once at the end of sampling process. It would be better to describe them using mathematical expressions.\n\nNot a serious problem, but for the writing style, I found it a bit hard to follow when cited papers stays in the middle of the sentence, such as,  ``, emulating the post-processing approaches of [10, 21] in various domains presented next.`` from line 190-191, but I do not really know what 10 and 21 are about and what authors point to unless I check the reference list. It also looks a bit weird to me: ``The implementation of this model is as described by Ho and Salimans.``, where no number follows after the citation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We will include abridged versions of our responses in this rebuttal window, but we ask that the reviewer refers to our complete answers in the comments.*\n\nThank you for your valuable feedback. Before addressing your specific questions, let us emphasize the significant contribution provided by our work: Our proposed method provides constraint imposition with _formal guarantees_ for several important classes and for the first time in the general and complex form posed! All prior work has either been unable to cope with the complex constraints that are crucial for the scientific and engineering application of interest (which include ODEs, nonconvex constraint sets, and real-world, scientific applications), relied on costly post-processing methods that we have shown _fail to produce meaningful samples in the diverse and real-world set_ of studied domains, and have failed to supply formal guarantees of constraint satisfaction as provided by our work. \n\n>**W1: How's the time complexity of PDM compared to Cond, Cond+ and Post+?**\n\nWe report the runtime difference between PDM and the other baselines in Section F. Additional details on time complexity of our projections are included in the comments below. Also, we will refer you to our response to Q2 by Reviewer yvsu, as this is closely related to your question of additional overhead (*as our PDM is much faster than conditional models when constraint adherence is necessary to generate viable samples!*).\n\n>**W2: Unclear how Cond+ is different from PDM....**\n\nTo clarify, *Cond* is a classifier-free guidance. The conditional model uses the guidance scheme provided at the end of page 2 (line 83). *Cond+* and *Post+* are equivalent to current post-processing methods, and we would like to emphasize the significant performance gap between these methods and PDM. We appreciate your suggestion of including a more clear formalization of these methods in the paper and will do so in our final version of the manuscript. *More details provided in the comments.*\n\n>**Q1: For Equation (3b), it requires the whole trajectory in the reverse process satisfying the constraints, while $x_t$s' while $t$ is large are basically noise. Is this optimization setup reasonable?**\n\nTheoretically, as discussed in Section 6, this approach is based on the insight that the cost of projection increases with the number of unconstrained steps. Theorem 6.2 supports this. Importantly, this theorem shows that the projection cost is lower when the sample starts from the feasible set, leading to better convergence properties. This is further demonstrated empirically, as methods imposing constraints only at the final step perform significantly worse than PDM, especially in non-convex settings. Importantly, \"performing worse\" here refers *not* to constraint adherence, but to the ability to generate images from the original data distribution (i.e., high-fidelity outputs) while satisfying the imposed constraints. Indeed, we showed how projecting only in the last step results in a significant divergence from the distribution in all settings we explored. Additionally, paralleling the theory behind projected gradient descent, as we aim to converge to the constrained minimum, there is a clear benefit in using projections to guide the optimization process toward a region where the minimum satisfying the constraints can be found. Practically, we've observed that unconstrained optimization steps can deviate significantly from the feasible domain, making it difficult to converge to a feasible sample. This is effectively illustrated in Figure 1 in the paper, where our proposed guidance scheme, PDM, navigates the constraint-defined landscape to ensure convergence to a feasible sub-distribution.\n\n\n>**Question 3: ... I still doubt the sampling distribution is not aligned with the data distribution since the samples might be all on the constraint boundaries.**\n\nThe projection will provide a point on the boundary of the constraints *if the original sample was infeasible.* In effect, this is why *Post+* and *Cond+* report such high FID scores, as more likely than not (for example, see Figures 2 and 8) prior to post-processing these samples are infeasible. In contrast, PDM samples converge to feasible subdistributions. For instance, notice that in Figure 1 later timesteps result in no violations of the constraints. This implies that unless subsequent gradient steps were consistently along the constraint boundaries (for all the samples visualized), *the samples output did not fall on the constraint boundaries and instead were from unique points within the feasible subdistribution.* PDM samples are \"moved\" to a subset of the distribution which is still optimal (maximizes the density function) but is also feasible. The projections solely enforce that the generated samples are taken from this region of the distribution.\n\nFurthermore, FID scores are widely used in image generation tasks because of how well they benchmark the similarity between distributions. This metric is particularly useful because **it reliably asesses both sample quality and diversity;** models which lack sample diversity perform very poorly when reporting FID scores. It is _widely accepted that this is the most appropriate method for assessing how well the sampling distributions and data distributions align_ and we show how well our proposed method performs in FID scores, while also satisfying the imposed constraints.\n\n>**Q4: When evaluating out-of-distribution samples, is the constraint satisfaction rate the only metric?**\n\nIn this setting, the FID scores of the out-of-distribution generations did not change from the scores reported for the in-distribution generations. Hence, we do not report these separately.\n\n---\nIf there are any additional reasons for the current assessment, we would appreciate further justification to address any remaining concerns. Thank you!", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IfFJWHTLiN", "reviewer": "Reviewer_yvsu", "summary": "This paper proposes Projected Diffusion Models (PDM) for constrained generative modeling. The key idea is to reframe the denoising process of diffusion models as a constrained optimization problem, iteratively projecting the generated samples onto a constraint set at every denoising step. The method is validated on several applications involving both convex and non-convex constraints, generally outperforming both conditional models and models that only project onto the constraint set after the last denoising step. The authors provide feasibility guarantees for convex constraints and optimality guarantees for convex constraints and likelihoods.", "questions": "* The implementational details in Appendix F state that all experiments were carried out with $T=10$ diffusion time steps. This is a very small value, compared to the hundreds or thousands of time steps that are used in standard image diffusion papers. Is there a reason for this?\n* How many samples were used to compute the quantitative performance metrics reported in Section 5? Would you expect a change in relative performance when using the same inference-time compute budget, i.e., generating 50% more conditional samples in the Constrained Materials application, since the conditional model is ~50% faster?\n* Figure 2 visualizes the constraint satisfaction rate of a conditional model as a function of the relative error tolerance, given in percentage points. What does an error tolerance of 100% correspond to?", "limitations": "The main methodological limitation of the proposed technique is its increased computational cost. The authors adequately address this limitation and suggest different approaches to overcome it. As outlined above, I believe that the main limitation of the experimental evaluation presented in the manuscript is its reliance on the FID metric.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* The paper is well-written and structured. The authors clearly describe their method, the motivation behind it, and the experimental settings it is evaluated in.\n* The method is straightforward to implement and compatible with pre-trained diffusion models, only requiring access to a projection operator at inference time.\n* The experiments cover a wide range of applications and constraint types, including both convex and non-convex constraints, as well as in- and out-of-distribution scenarios.", "weaknesses": "* The paper heavily relies on Fréchet Inception Distance (FID) scores for the empirical comparison of models in Sections 5.1, 5.2, and 5.4. While FID is a standard metric for evaluating generative models on ImageNet-like images, it is unclear how meaningful it is in the specialized domains explored in this work (e.g., material microstructures, human poses, physics-informed simulations).\n* The paper presents PDM as a novel optimization technique that \"recasts traditional denoising strategies as a constrained optimization problem\". However, if I understand it correctly, it is an application of commonly employed post-processing projections (e.g. the references provided in the manuscript) to the Langevin MCMC sampling scheme of Song and Earmon, NeurIPS 2019. It would be helpful if the authors could clarify this and outline any additional novelty/contributions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "*We will include abridged versions of our responses in this rebuttal window, but we ask that the reviewer refers to our complete answers in the comments.*\n\nThank you for your time and efforts in providing feedback on our paper. We appreciate your acknowledgment of the diversity of our applications and constraint sets, as we consider this to be a significant component of our empirical validation of the proposed method. Let us emphasize that PDM reports state-of-the-art results in these diverse domains, outperforming baselines [5,11] in real-world microstructure synthesis when evaluated using FID, heuristic-based analysis, and constraint adherence, providing the first method we are aware of to _report zero-violations in human motion synthesis_, surpassing the current state-of-the-art in diffusion-based trajectory optimization reporting perfect feasibility and identical path length, and demonstrating applicability to video generation with complex, ODE-based constraints. Not only the range of domains we evaluate our method on exhibits its robustness across various constraint sets and data settings, but we also provide solid theoretical arguments for these behaviors. We hope you would agree that this demonstrates the generalizability of our framework and provides a compelling reason for why our paper should be positively considered.\n\n>**W1: The paper heavily relies on Fréchet Inception Distance (FID) scores for the empirical comparison...**\n\nFirst, we would like to point out, as the reviewer has acknowledged, that FID is a standard metric for evaluating generative models, and evaluation using this metric is _more than reasonable_. However, we believe the reviewer may have missed that **the paper indeed already reports several additional, domain specific metrics to supplement these results**. For Sections 5.1, 5.2, and 5.3 we include additional metrics (please see Section E.1, Section E.2, and Figure 6). Additionally, we will highlight that the baselines in Sections 5.2 and 5.4 use FID as their primary metric for evaluation ([29,30] and [26]).\n\n>**W2: The paper presents PDM as a novel optimization technique that \"recasts traditional denoising strategies as a constrained optimization problem\"... it is an application of commonly employed post-processing projections...**\n\nNote that, we have indeed acknoledged that some post-processing steps have been proposed previously, and indeed we compare aginst such methods. However, these methods post-process **after** the sampling process, which is a key difference, in light of our theoretical analysis. As discussed in Section 6, our approach is based on the insight that the cost of projection increases with the number of unconstrained steps (see also the illustration in Section E.3). Crucially, we have shown that post-processing approaches produce samples of much lower quality than those produced by PDM, improving material synthesis FID scores by over 30%, feasibility rates of tarjectory optimization by 90%, and increasing the quality of physics-based video generaiton by a factor of two. We argue theoretically, and empirically demonstrate, that a single post-processing projection leads to a significant divergence from the distribution in all the settings we examined. **This is a key novel contribution of this paper**.\n\n>**Q2: Would you expect a change in relative performance when using the same inference-time compute budget, i.e., generating 50% more conditional samples in the Constrained Materials application, since the conditional model is ~50% faster?**\n\nFor the example that was referenced, as one may extrapolate from Figure 2, while the conditional model may generate 3000 samples within the time span it took for PDM to generate 2000 samples, if the error tolerance is less than ~35% (which is an absurdly high margin in this setting!!) then PDM generates many more feasible samples than the conditional model within the same compute budget. This discrepency is further emphasized when the tolerance is within a more reasonable margin, such as 5% where **PDM generates nearly seven times as many feasible samples within the same compute budget.** Hence, when constraints are integral to the outputs, PDM does, in fact, outperform conditional models in relative speed as well!\n\n>**Q3: Figure 2 visualizes the constraint satisfaction rate... what does an error tolerance of 100% correspond to?**\n\nWhen representing the porosity levels, we provide a percentage of pixels that should be below a provided threshold, as dark regions of the image represent damaged regions of the microstructure. For example, if an image had 50% porosity and 40% porosity was specified, the error tolerance that would make this feasible is 10%. Notice that our proposed method **guarantees** constraint adherence here, which is key for the scientific application tested. \n\n>**L1: The main methodological limitation... is its increased computational cost... the main limitation of the experimental evaluation presented in the manuscript is its reliance on the FID metric.**\n\nFirst, we would agree with the reviewer point about increased computational cost. This is an inherent byproduct of constraining any optimization problem, especially when providing guarantees upon the adherence to the constraint set. However, we would point you to our response to Question 2 for more context on this overhead. Additionally, we will note that in many settings, including those studied in this paper, *constraint-agnostic model cannot be used due to the necessity of adhering to the constraint set.* \n\nSecond, we will point the reviewer to our response to Weakness 1. We hope that the reviewer will take the opportunity to revisit the additional metrics that we bring attention to here and also to examine the evaluation criteria used by the referenced baselines. As this is the primary justification provided for the score that the reviewer provided, we would ask you to consider raising your score with this provided context.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IDXjOenrTS", "reviewer": "Reviewer_p9zr", "summary": "This paper proposes an approach to sample generation using diffusion models which adheres to a set of constraints. The approach is based on the score matching formulation of diffusion models, and applies a projection step which finds the nearest feasible sample to each iteration of SGLD. A theoretical justification for the method is proposed, which provides intuition around why the approach yields superior results to conditioning and post-processing methods. The method is evaluated on a wide variety of experiments covering multiple classes of constraints (including non-convex) and data modalities, and is demonstrated to yield high-quality samples which strictly adhere to non-trivial constraints.", "questions": "line 158: Does \"convergence of convex constraint sets\" simply mean that the projection step will always return a feasible solution? Or are you referring more broadly to the DDPM iterates?\nline 186: Double colon typo \"::\"\nFig. 1: Given that the approach projects to the constraint set at each iteration, why are we still seeing constraint violation early on in the diffusion process? Is this plotting the pre-projected samples? It is unclear what is actually being plotted.\nline 272: typo, \"unfeasible\"\nline 331: some comment around the reasonability of the assumption of the score function being convex (locally for small step size say?) would be appreciated", "limitations": "The authors have clearly described the limitations of the approach, however I believe a clearer statement of compute overhead would improve the clarity around the limitations.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "This is a very strong submission which is well-written and easy to read. The method is very sensible, and described clearly.\n\nThe experiments are diverse and interesting, spanning multiple application domains, data modalities (e.g., images, configuration space trajectories) and constraint types. In particular, the motion planning experiments with non-convex constraints are particularly interesting to me since synthesis with non-convex constraints has not really been approached in prior works. \n\nFurthermore, the demonstration of synthesis with constraints applied only at test time, with the constraints not being applied during training and having the training data violating the constraints is also very appealing. \n\nFinally, the authors also provide some theoretical justification and intuition of the proposed approach for convex constraints, which is a nice addition.", "weaknesses": "I think a mathematical description of the projection operator and constraints for the experiments 5.1-5.3 would be helpful to understand the degree of non-convexity and difficulty for these experiments. While I have a rough idea, being concrete would help a lot. \n\nAlso, I believe it is important to be more forward in the main document around the computational overhead when it comes to applying the projection operator at each SGLD iteration. While I believe that high overhead does not particularly detract from the contribution (this can be addressed in future works), I think it detracts from the paper to hide this information in the appendix and not refer to it obviously in the main text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for their time reviewing our work and their praise of our submission. We appreciate your consideration of our work and would like to address your outstanding questions and concerns.\n\n>**Weakness 1: I think a mathematical description of the projection operator and constraints for the experiments 5.1-5.3 would be helpful to understand the degree of non-convexity and difficulty for these experiments. While I have a rough idea, being concrete would help a lot.**\n\n- **Constrained Materials:** For this experiment we use a classical knapsack constraint. Note that while in general knapsack problems are NP-complete, the version adopted (which uses integers as weight) is known to be weakly NP-complete and admits a fully polynomial approximation. It is solved efficiently in $O(nm)$ where $n$ is the number of pixels and $m$ is the number of values to compute for the dynamic program. \n- **3D Human Motion:** This is a scaling constraint that adjusts all positions of the figure by the relative distance between the lowest joint and the floor, adjusting the relative position to prevent floating and penetration i.e., \n$ argmin ||y-x||$ s.t. $\\min_i (y_i) = 0 \\quad \\text{and} \\quad ||y_j - y_k|| = ||x_j - x_k|| \\forall j, k$.\nAdditionally, we impose a realism constraint on appendages, keeping them consistent with one another i.e.\n$||elbow_\\textrm{right} - wrist_\\textrm{right}|| = ||elbow_\\textrm{left} - wrist_\\textrm{left}||$. The implementation runs in $O(n)$ time, where $n$ is the length of the internal representation.\n- **Constrained Trajectories:** This problem primarily represents constraints as a minimum distance, $d$, between the center point of an obstacle, $q$, and the closest point falling between each set of consecutive points. More formally, this is expressed by evaluating the distance $d_{\\text{min}}$ from $q$ to the line segment $\\overline{p_i p_{i+1}}$, where $d_{\\text{min}}$ is determined as follows: if the projection of $q$ onto the line defined by $p_i$ and $p_{i+1}$ falls within the segment, $d_{\\text{min}}$ is the perpendicular distance from $q$ to the line; otherwise, $d_{\\text{min}}$ is the distance from $q$ to the nearest endpoint, $p_i$ or $p_{i+1}$. The constraint is thus $d_{\\text{min}} > d$, ensuring that the nearest point on the segment to $q$ is at least a distance $d$ away. \nThe interior point method used by the nonconvex solver in our implementation [27] has a time complexity of $O(n^{3.5})$ where $n$ is the number of variables in the Quadratic Program. In this case, there are 128.\n- **Physics-informed Motion:** This is similar to Section 5.2 and runs in $O(n+m)$ where the size of each image is $n$ by $m$.\n\n\nWe appreciate this suggestion and will add the formalization of these operators to our next draft.\n\n>**Weakness 2: Also, I believe it is important to be more forward in the main document around the computational overhead when it comes to applying the projection operator at each SGLD iteration. While I believe that high overhead does not particularly detract from the contribution (this can be addressed in future works), I think it detracts from the paper to hide this information in the appendix and not refer to it obviously in the main text.**\n\nWe are open to additional suggestions on how to more obviously showcase this component in our paper. While we do explicitly reference Section F in our Discussion and limitations section (Section 7), we were unable to make space to display this table in the nine pages of the main paper allowed for submission. This is definetively something we will address in the extra page allowed for the accepted version. \n\n>**Question 1: line 158: Does \"convergence of convex constraint sets\" simply mean that the projection step will always return a feasible solution? Or are you referring more broadly to the DDPM iterates?**\n\nThis line specifically refers to the latter, as **by proximal gradient descent theory we can guarantee convergence to both a feasible and optimal solution when the problem is convex.** You are also correct in that PDM _guarantees_ feasiblity for convex constraints! \n\n>**Question 2: Fig. 1: Given that the approach projects to the constraint set at each iteration, why are we still seeing constraint violation early on in the diffusion process? Is this plotting the pre-projected samples? It is unclear what is actually being plotted.**\n\nFigure 1 shows the samples after the gradient step is applied (line 6 of Algorithm 1) but **prior to the projection.** The point we would like to illustrate here is that later, in the reverse process, the projections have minimal impact on the sample as additional diffusion steps do not result in constraint violations. This also means that the samples do not directly fall on the constraint boundaries as they converge within the feasible region. Conversely, when post-processing a conditional model's outputs (*Cond+*), the sample is dramatically altered by this projection, resulting in _much higher FID scores_ and _deviation from the real data distribution_.\n\n---\nWe appreciate your review and feedback and are happy to address any other questions you may have. It is our hope that these responses have addressed any concerns that you may have and provided confidence in supporting even further our work! Thank you for your time and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper was an interesting application of ideas from diffusion and constrained optimization to have more controllable generative models that can satisfy some known physical principles/constraints. 
The reviews overall are a bit borderline. Some of it stems from the paper subject itself which lies at an intersection of modern ML methods and applications to practical problems. The method being being more expensive than an unconstrained setup is likely irresolvable. The paper does report a fair variety of metrics to understand how well the method works. The method introduced is quite general purpose and should find wider applicability and open-sourcing the implementation would be valuable to the community.
Decision: Accept (poster)