Constrained Synthesis with Projected Diffusion Models

Authors: jacob k christopher
stephen  baek
ferdinando  fioretto
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-15 16:46:19

Abstract: This paper introduces an approach to endow generative diffusion processes the ability to satisfy and certify compliance with constraints and physical principles. The proposed method recast the traditional sampling process of generative diffusion models as a constrained optimization problem, steering the generated data distribution to remain within a specified region to ensure adherence to the given constraints.
These capabilities are validated on applications featuring both convex and challenging, non-convex, constraints as well as ordinary differential equations, in domains spanning from synthesizing new materials with precise morphometric properties, generating physics-informed motion, optimizing paths in planning scenarios, and human motion synthesis.

Main paper: 
 introduction: 
 Generative diffusion models excel at robustly synthesizing content from raw noise through a sequential denoising process [14, 24] . They have revolutionized high-fidelity creation of complex data, and their applications have rapidly expanded beyond mere image synthesis, finding relevance in areas such as engineering [29, 34] , automation [3, 16] , chemistry [1, 15] , and medical analysis [2, 6] . However, although diffusion models excel at generating content that is coherent and aligns closely with the original data distribution, their direct application in scenarios requiring stringent adherence to predefined criteria poses significant challenges. Particularly the use of diffusion models in scientific and engineering domains where the generated data needs to not only resemble real-world examples but also rigorously comply with established specifications and physical laws remains an open challenge.Given these limitations, one might consider training a diffusion model on a dataset that already adheres to specific constraints. However, even with "feasible" training data, this approach does not guarantee adherence to desired criteria due to the stochastic nature of the diffusion process. Furthermore, there are frequent scenarios where the training data must be altered to generate outputs that align with specific properties, potentially not present in the original data. This issue often leads to a distribution shift further exacerbating the inability of generative models to produce "valid" data. As we will show in a real-world experiment ( §6.1), this challenge is particularly acute in scientific and engineering domains, where training data is often sparse and confined to specific distributions, yet the synthesized outputs are required to meet stringent properties or precise standards [29] . This paper addresses these challenges and introduces Projected Diffusion Models (PDM), a novel approach that recast the traditional sampling strategy in diffusion processes as a constrained-optimization problem. This perspective allows us to apply traditional techniques from constraint optimization to the sampling process. In this work, the problem is solved by iteratively projecting the diffusion sampling process onto arbitrary constraint sets, ensuring that the generated data adheres strictly to imposed constraints or physical principles. We provide theoretical support for PDM's capability to not only certify adherence to the constraints but also to optimize the generative model's original objective of replicating the true data distribution. This alignment is a significant advantage of PDM, yielding state-of-the-art FID scores while maintaining strict compliance with the imposed constraints.Contributions. In summary, this paper makes the following key contributions: (1) It introduces PDM, a new framework that augments diffusion-based synthesis with arbitrary constraints in order to generate content with high fidelity that also adheres to the imposed specifications. The paper elucidates the theoretical foundation that connects the reverse diffusion process to an optimization problem, facilitating the direct incorporation of constraints into the reverse process of score-based diffusion models. (2) Extensive experiments across various domains demonstrate PDM's effectiveness. These include adherence to morphometric properties in real-world material science experiments, physics-informed motion governed by ordinary differential equations, trajectory optimization in motion planning, and constrained human motion synthesis, showcasing PDM's ability to produce content that adheres to both complex constraints and physical principles. (3) We further show that PDM is able to generate out-of-distribution samples that meet stringent constraints, even in scenarios with extremely sparse training data and when the training data does not satisfy the required constraints. (4) Finally, we provide a theoretical basis elucidating the ability of PDM to generate highly accurate content while ensuring constraint compliance, underpinning the practical implications of this approach.Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?Answer: [Yes] Justification: The contributions of this paper are defined succinctly in the abstract, with clear references to the experimental settings explored in the manuscript Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper. • The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers. • The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. • It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper. 
preliminaries: diffusion models: 
 Diffusion-based generative models [14, 24] expand a data distribution, whose samples are denoted x 0 , through a Markov chain parameterization {x t } T t=1 , defining a Gaussian diffusion process p(x 0 ) = p(x T ) T t=1 p(x t-1 |x t )dx 1:T . In the forward process, the data is incrementally perturbed towards a Gaussian distribution. This process is represented by the transition kernel q(x t |x t-1 ) = N (x t ; √ 1 -β t x t-1 , β t I) for some 0 < β t < 1, where the β-schedule {β t } T t=1 is chosen so that the final distribution p(x T ) is nearly Gaussian. The diffusion time t allows an analytical expression for variable x t represented by χ t (x 0 , ϵ) = √ α t x 0 + √ 1 -α t ϵ, where ϵ ∼ N (0, I) is a noise term, and α t = t i=1 (1 -β i ). This process is used to train a neural network ϵ θ (x t , t), called the denoiser, which implicitly approximates the underlying data distribution by learning to remove noise added throughout the forward process. The training objective minimizes the error between the actual noise ϵ and the predicted noise ϵ θ (χ t (x 0 , ϵ), t) via the loss function: ∥ϵ -ϵ θ (χ t (x 0 , ϵ), t)∥2 .(EQUATIONThe reverse process uses the trained denoiser, ϵ θ (x t , t), to convert random noise p(x T ) iteratively into realistic data from distribution p(x 0 ). Practically, ϵ θ predicts a single step in the denoising process that can be used during sampling to reverse the diffusion process by approximating the transition p(x t-1 |x t ) at each step t.Score-based models [25, 26] , while also operating on the principle of gradually adding and removing noise, focus on directly modeling the gradient (score) of the log probability of the data distribution at various noise levels. The score function ∇ xt log p(x t ) identifies the direction and magnitude of the greatest increase in data density at each noise level. The training aims to optimize a neural network s θ (x t , t) to approximate this score function, minimizing the difference between the estimated and true scores of the perturbed data:EQUATIONwhere q(x t |x 0 ) = N (x t ; √ α t x 0 , (1 -α t )I) defines a distribution of perturbed data x t , generated from the training data, which becomes increasingly noisy as t approach T . This paper considers score-based models.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes] Justification: Discussion of limitations and open questions left in this work are described in Section 7.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
related work and limitations: 
 While diffusion models are highly effective in producing content that closely mirrors the original data distribution, the stochastic nature of their outputs act as an impediment when specifications or constraints need to be imposed on the generated outputs. In an attempt to address this issue, two main approaches could be adopted: (1) model conditioning and (2) post-processing corrections.Model conditioning [13] aims to control generation by augmenting the diffusion process via a conditioning variable c to transform the denoising process via classifier-free guidance:εθ def = λ × ϵ θ (x t , t, c) + (1 -λ) × ϵ θ (x t , t, ⊥),where λ ∈ (0, 1) is the guidance scale and ⊥ is a null vector representing non-conditioning. These methods have been shown effective in capturing properties of physical design [29] , positional awareness [3] , and motion dynamics [32] . However, while conditioning may be effective to influence the generation process, it lacks the rigor to ensure adherence to specific constraints. This results in generated outputs that, despite being plausible, may not be accurate or reliable. Figure 1 (red colors) illustrates this issue on a physics-informed motion experiment (detailed in §6.4). The figure reports the distance of the model outputs to feasible solutions, showcasing the constraint violations identified in a conditional model's outputs. Notably, the model, conditioned on labels corresponding to positional constraints, fails to generate outputs that adhere to these constraints, resulting in outputs that lack meaningful physical interpretation. Additionally, conditioning in diffusion models often requires training supplementary classification and regression models, a process fraught with its own set of challenges. This approach demands the acquisition of extra labeled data, which can be impractical or unfeasible in specific scenarios. For instance, our experimental analysis will demonstrate a situation in material science discovery where the target property is well-defined, but the original data distribution fails to embody this property. This scenario is common in scientific applications, where data may not naturally align with desired outcomes or properties [19] .Post-processing correction. An alternative approach involves applying post-processing steps to correct deviations from desired constraints in the generated samples. This correction is typically implemented in the last noise removal stage, s θ (x 1 , 1). Some approaches have augmented this process to use optimization solvers to impose constraints on synthesized samples [10, 19, 21] . However these approaches present two main limitations. First, their objective does not align with optimizing the score function. This inherently positions the diffusion model's role as ancillary, with the final synthesized data often resulting in a significant divergence from the learned (and original) data distributions, as we will demonstrate in §6. Second, these methods are reliant on a limited and problem specific class of objectives and constraints, such as specific trajectory "constraints" or shortest path objectives which can be integrated as a post-processing step [10, 21] .Other methods. Some methods explored modifying either diffusion training or inference to adhere to desired properties. For instance, the methods in [9] and [17] , support simple linear or convex sets, respectively. Similarly, Fishman et al. [7, 8] focus on predictive tasks within convex polytope, which are however confined to approximations by simple geometries like L2-balls. While important contributions, these approaches prove insufficient for the complex constraints present in many realworld tasks. Conversely, in the domain of image sampling, Lou and Ermon [18] and Saharia et al. [23] introduce methods like reflections and clipping to control numerical errors and maintain pixel values within the standard [0,255] range during the reverse diffusion process. These techniques, while enhancing sampling accuracy, do not address broader constraint satisfaction challenges.To overcome these gaps and handle arbitrary constraints, our approach casts the reverse diffusion process to a constraint optimization problem that is then solved throught repeated projection steps.4. Behavior on ODEs and under constraints outside the training distribution. ( §6.4).Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Justification: Details on the used compute resources are described in Appendix F.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
constrained generative diffusion: 
 This section establishes a theoretical framework that connects the reverse diffusion process as an optimization problem. This perspective facilitates the incorporation of constraints directly into the process, resulting in the constrained optimization formulation presented in Equation (6) .The application of the reverse diffusion process of score-based models is characterized by iteratively transforming the initial noisy samples x T back to a data sample x 0 following the learned data distribution q(x 0 ). This transformation is achieved by iteratively updating the sample using the estimated score function ∇ xt log q(x t |x 0 ), where q(x t |x 0 ) is the data distribution at time t. At each time step t, starting from x 0 t , the process performs M iterations of Stochastic Gradient Langevin Dynamics (SGLD) [30] :EQUATIONwhere ϵ is standard normal, γ t > 0 is the step size, and ∇ x i t log q(x i t |x 0 ) is approximated by the learned score function s θ (x t , t).First note that SGLD is derived from discretizing the continuous-time Langevin dynamics, which are governed by the stochastic differential equation:EQUATIONwhere B(t) is standard Brownian motion. Under appropriate conditions, the stationary distribution of this process is q(x t ) [22] , implying that samples generated by Langevin dynamics will, over time, be distributed according to q(x t ). In practice, these dynamics are simulated using a discrete-time approximation, leading to the SGLD update in Equation (3). Therein the noise term √ 2γ t ϵ i t allows the algorithm to explore the probability landscape and avoid becoming trapped in local maxima.Next notice that, as detailed in [30, 31] , under some regularity conditions this iterative SGLD algorithm converges toward a stationary point, bounded by d 2 σ 1/4 λ * log(1/ϵ), where, σ 2 represents the variance schedule, λ * denotes the uniform spectral gap of the Langevin diffusion, and d is the dimensionality of the problem. Thus, as the reverse diffusion process progresses towards T → 0, and the variance schedule decreases, the stochastic component becomes negligible, and SGLD transitions toward deterministic gradient ascent on log q(x t ). In the limit of vanishing noise, the update rule simplifies to:EQUATIONwhich is standard gradient ascent aiming to maximize log q(x t ). This allow us to view the reverse diffusion process as an optimization problem minimizing the negative log-likelihood of the data distribution q(x t |x 0 ) at each time step t.In traditional score-based models, at any point throughout the reverse process, x t is unconstrained. When these samples are required to satisfy some constraints, the objective remains unchanged, but the solution to this optimization must fall within a feasible region C, and thus the optimization problem formulation becomes: minimize x T ,...,x1 t=T,...,1EQUATIONOperationally, the negative log likelihood is minimized at each step of the reverse Markov chain, as the process transitions from x T to x 0 . In this regard, and importantly, the objective of the PDM's reverse sampling process is aligned with that of traditional score-based diffusion models.The score network s θ (x t , t) directly estimates the first-order derivatives of Equation (6a), providing the necessary gradients for iterative gradient-based updates defined in Equation (3). In the presence of constraints (6b), however, an alternative iterative method is necessary to guarantee feasibility. PDM models a projected guidance approach to provide this constraint-aware optimization process.First, we define the projection operator, P C , as a constrained optimization problem,EQUATIONthat finds the nearest feasible point to the input x. The cost of the projection ||y -x|| 2 2 represents the distance between the closest feasible point and the original input.To retain feasibility through an application of the projection operator after each update step, the paper defines projected diffusion model sampling step asEQUATION)where C is the set of constraints and P C is a projection onto C. Hence, iteratively throughout the Markov chain, a gradient step is taken to minimize the objective defined by Equation (6a) while ensuring feasibility. Convergence is guaranteed for convex constraints sets [20] and empirical evidence in §6 showcases the applicability of this methods to arbitrary constraint sets. Importantly, the projection Algorithm 1: PDM1 x 0 T ∼ N (0, σ T I) 2 for t = T to 1 do 3 γ t ← σ 2 t/2σ 2 T 4 for i = 1 to M do 5 ϵ ∼ N (0, I); g ← s θ * (x i-1 t , t) 6 x i t = P C (x i-1 t + γ t g + √ 2γ t ϵ) 7 x 0 t-1 ← x M t 8 return x 0 0operators can be warm-started during the repeated sampling step providing a piratical solution even for hard non-convex constrained regions. The full sampling process is detailed in Algorithm 1.By incorporating constraints throughout the sampling process, the interim learned distributions are steered to comply with these specifications. This is empirically evident from the pattern in Figure 1 (blue curves): remarkably, the constraint violations decrease with each addition of estimated gradients and noise and approaches 0-violation as t nears zero. This trend not only minimizes the impact but also reduces the optimality cost of projections applied in the later stages of the reverse process. We provide theoretical rationale for the effectiveness of this approach in §5 and conclude this section by noting that this approach can be clearly distinguished from other methods which use a diffusion model's sampling process to generate starting points for a constrained optimization algorithm [10, 21] . Instead, PDM leverages minimization of negative log likelihood as the primary objective of the sampling algorithm akin to standard unconstrained sampling procedures. This strategy offers a key advantage: the probability of generating a sample that conforms to the data distribution is optimized directly, rather than an external objective, while simultaneously imposing verifiable constraints. In contrast, existing baselines often neglect the conformity to the data distribution, which, as we will show in the next section, can lead to a deviation from the learned distribution and an overemphasis on external objectives for solution generation, resulting in significant divergence from the data distribution, reflected by high FID scores. 
effectiveness of pdm: a theoretical justification: 
 Next, we theoretically justify the use of iterative projections to guide the sample to the constrained distribution. The analysis assumes that the feasible region C is a convex set. All proofs are reported in the Appendix. We start by defining the update step. Definition 5.1. The operator U defines a single update step for the sampling process as,EQUATIONThe next result establishes a convergence criteria on the proximity to the optimum, where for each time step t there exists a minimum value of i = Ī such that,EQUATIONwhere ρ t is the closest point to the global optimum that can be reached via a single gradient step from any point in C. Theorem 5.2. Let P C be a projection onto C, x i t be the sample at time step t and iteration i, and 'Error' be the cost of the projection (7) . Assume ∇ xt log p(x t ) is convex. For any i ≥ Ī,EQUATIONThe proof for Theorem 5.2 is reported in §H. This result suggests that PDM's projection steps ensure the resulting samples adhere more closely to the constraints as compared to samples generated through traditional, unprojected methods. Together with the next results, it will allow us to show that PDM samples converge to the point of maximum likelihood that also satisfy the imposed constraints.The theoretical insight provided by Theorem 5.2 provides an explanation for the observed discrepancy between the constraint violations induced by the conditional model and PDM, as in Figure 1 .Corollary 5.3. For arbitrary small ξ > 0, there exist t and i ≥ Ī such that:Error(U(P C (x i t )), C) ≤ ξ.The above result uses the fact that the step size γ t is strictly decreasing and converges to zero, given sufficiently large T , and that the size of each update step U decreases with γ t . As the step size shrinks, the gradients and noise reduce in size. Hence, Error(U(P C (x i t )) approaches zero with t, as illustrated in Figure 1 (right). This diminishing error implies that the projections gradually steer the sample into the feasible subdistribution of p(x 0 ), effectively aligning with the specified constraints.Feasibility guarantees. PDM provides feasibility guarantees when solving convex constraints. This assurance is integral in sensitive settings, such as material analysis (Section 6.1), plausible motion synthesis (Section 6.2), and physics-based simulations (Section 6.4), where strict adherence to the constraint set is necessary.Corollary 5.4. PDM provides feasibility guarantees for convex constraint sets, for arbitrary density functions ∇ xt log p(x t ). 
experiments: 
 We compare PDM against three methodologies, each employing state-of-the-art specialized methods tailored to the various applications tested:: (1) Conditional diffusion models (Cond ) [13] are the stateof-the-art methods for generative sampling subject to a series of specifications. While conditional diffusion models offer a way to guide the generation process towards satisfying certain constraints, they do not provide compliance guarantees. (2) To encourage constraints satisfaction, we additionally compare to conditional models with a post-processing projection step (Cond + ), emulating the postprocessing approaches of [10, 21] in various domains presented next. Finally, (3) we use a score-based model identical to our implementation but with a single post-processing projection operation (Post + ) performed at the last sampling step. Additional details are provided in §C.The performance of these models are evaluated by the feasibility and accuracy of the generated samples. Feasibility is assessed by the degree and rate at which constraints are satisfied, expressly, the percentage of samples which satisfy the constraints with a given error tolerance. Accuracy is measured by the FID score, a standard metric in synthetic sample evaluation. To demonstrate the broad applicability of our approach, our experimental settings have been selected to exhibit:1. Behavior in low data regimes and with original distribution violating constraints ( §6.1), as part of a real-world material science experiment. 2. Behavior on 3-dimensional sequence generation with physical constraints ( §6.2).The first setting focuses on a real-world application in material science, conducted as part of an experiment to expedite the discovery of structure-property linkages (please see §C for extensive additional details). From a sparse, uniform collection of microstructure materials, we aim to generate new structures with desired, previously unobserved porosity levels.There are two key challenges in this setting: (1) Data sparsity: A critical factor in this setting is the cost of producing training data. Our dataset, obtained from the authors of [5] , consists of 64 × 64 image patches subsampled from a 3, 000 × 3, 000 pixel microscopic image, with pixel values scaled to [-1, 1]. These patches are upscaled to 256 × 256 for model training. (2) Out-of-distribution constraints: Constraints on the generated material's porosity, defined by pixels below a threshold representing damage, are far from those observed in the original dataset. Previous work demonstrated the use of conditional GANs [5, 11] to material generation, but these studies failed to impose verifiable constraints on desired properties. To establish a conditional baseline (denoted as Cond), we implement a conditional diffusion model, following the state-ofthe-art approach by Chun et al. [5] , conditioning the sampling on porosity measurements. Figure 2 reports the constraint violations achieved by this model. The plot depicts the frequency of constraint satisfaction (y-axis) as a function of the error tolerance (x-axis), in percentage. Observe that this state-of-the-art model struggles to adhere to the imposed constraints.In contrast, PDM ensures both exact constraint satisfaction and identical image quality to the conditional model, which is significant given the complexity of the original data distribution. Figure 3 visualizes the outputs and FID scores obtained by our proposed model compared to various baselines. The constraint correction step applied by Post + and Cond + leads to a noticeable decrease in image quality, evident both visually and in the FID scores, rendering the generated images unsuitable for this application context. Additionally, we find that PDM outperforms Cond in generating microstructures that resemble those in the ground truth data (see §E.1). These results are significant: the ability to precisely control morphological parameters in synthetic microstructures has broad impact in material synthesis, addressing critical challenges in data collection and property specification.Next we focus on dynamic motion generation adhering to strict physical principles using the challenging HumanML3D dataset [12] . This benchmark employs three-dimensional figures across a fourth dimension of time to simulate motion. Thus, the main challenges here are generating 3D figures (1) including a temporal component, while (2) ensuring they neither penetrate the floor nor float in the air, as proposed by Yuan et al. [32] . Previous studies have seen physical law violations in motion diffusion models, with none achieving zero-tolerance results [32] . We next demonstrate PDM's capability to overcome these limitations.First, we remark that previous approaches, such as [32] , relies on a computationally demanding physics simulators to transform diffusion model predictions into "physically-plausible" actions, using a motion imitation policy trained via proximal policy optimization. This simulator is crucial to produce realistic results and dramatically alters the diffusion model's outputs. In contrast, and remarkably, PDM does not require such a simulator. and inherently satisfies the non-penetration and non-floating constraints without external assistance, showing zero violations. For comparison, the best outcomes reported in [32] ranged from 0.918 to 0.998 for penetration violations and 2.601 to 3.173 for floating violations (see §E.2 for more details).Additionally, to more accurately assess the abilities of a diffusion model in the absences of physics simulator, we evaluate a conditional model with the same architecture as the PDM model, adapted from MotionDiffuse [33] . Results visualized in Figure 4 demonstrate that PDM achieves outputs on par with state-of-the-art FID scores. Additionally, the use of projections here is guaranteed to provide exact constraint satisfaction, as we will elaborate further in §5.The next experiment showcase the ability of PDM to handle nonconvex constraints. Path planning is a classic optimization problem which is integral to finding smooth, collision-free paths in autonomous systems. This setting consists of minimizing the path length while avoiding path intersection with various obstacles in a given topography. Recent research has demonstrated the use of diffusion models for these motion planning objectives [3] . In this task, the diffusion model predicts a series of points, p 0 , p 1 , . . . , p N , where each pair of consecutive points represents a line segment. The start and end points for this path are determined pseudo-randomly for each problem instance, with the topography remaining constant across different instances. Additionally, the problem presents obstacles at inference time (shown in red on Figure 5 ), that were not present during training, rendering a portion of the training data infeasible, and thus testing the generalization of these methods. The performance is evaluated on two sets of maps adapted from Carvalho et al., shown in Figure 5 . The main challenge in this setting is the non-convex nature of the contraints.To circumvent the challenge of guaranteeing collision-free paths, previous methods have relied on sampling large batches of trajectories, selecting a feasible solution if available [3] . We use the state-of-the-art Motion Planning Diffusion [3] as a conditional model baseline for this experiment and the associated datasets to train each of the models. For the Cond + model, we emulate the approach proposed by Power et al. [21] , using the conditional diffusion model to generate initial points for an optimization solver. In this setting, the projection operator used by PDM is non-convex, and the implementation uses an interior point method [28] . While the feasible region is non-convex, our approach never report unfeasible solutions as the distance from the learned distribution decreases, unlike other methods. In contrast, using the same method for a single post-processing projection (Post + ) does not enhance the feasibility of solutions compared to the unconstrained conditional model (Cond), highlighting the limitations of these single corrections in managing local infeasibilities. These observations are consistent with the analysis of Figure 1 .Figure 5 visually demonstrates that PDM can identify a feasible path with just a single sample. This capability marks a significant advance over existing state-of-the-art motion planning methods with diffusion models, as it eliminates the necessity of multiple inference attempts, which greatly affects the efficiency in generating feasible solutions. The experimental results (Table 6 ) demonstrate the effectiveness of PDM in handling complex, non-convex constraints in terms of success percentage for single trajectories generated (top) and path length (bottom). Notices how, both the Cond and Cond + models fall short in finding feasible trajectories, taking shortcuts resulting in collisions. At the same time, notice how all methods find comparable path length within the reported error tolerances. These results are significant: they show that PDM can not only handle complex, non-convex constraints, but also produce results that are on par with state-of-the-art models in solution optimality.Finally, we show the applicability of PDM in generating video frames adhering to physical principles. In this task, the goal is to generate frames depicting an object accelerating due to gravity. The object's position in a given frame is governed byEQUATION)where p is the object position, v is the velocity, and t is the frame number. This positional information can be directly integrated into the constraint set of PDM, with constraint violations quantified by the pixel distance from their true position. In our experiment, the training data is based solely on earth's gravity and we test the model to simulate gravitational forces from the moon and other planets, in addition to earth. Thus there are two challenges in this setting (1) satifying ODEs describing our physical principle and (2) generalize to out-of-distribution constraints.Figure 7 (left) shows randomly selected generated samples, with ground-truth images provided for reference. The subsequent rows display outputs from PDM, post-processing projection (Post), and conditional post-processing (Cond + ). For this setting, we used a state-of-the-art masked conditional video diffusion model, following Voleti et al. [27] . Samples generated by conditional diffusion models are not directly shown in the figure, as the white object outline in the Cond + frames shows where the Cond model originally positioned the object. Notice that, without constraint projections, the score-based generative model produce samples that align with the original data arbitrarily place the object within the frame (white ball outlines in the 3rd column). Post-processing repositions the object accurately but significantly reduces image quality. Similarly, Cond + shows inaccuracies in the conditional model's object positioning, as indicated by the white outline in the 4th column. These deviations from the desired constraints are quantitatively shown in Figure 8 (light red bars), which depicts the proportion of samples adhering to the object's behavior constraints across varying error tolerance levels. Notably, this approach fails to produce any viable sample within a zero-tolerance error margin. In contrast, PDM generates frames that exactly satisfy the positional constraints, with FID scores comparable to those of Cond. Using the model proposed by Song et al. [26] further narrows this gap (see §D).Next, Figure 7 (right) shows the behavior of the models in settings where the training data does not include any feasible data points. Here we adjust the governing equation (12) to reflect the moon's gravitational pull. Remarkably, PDM not only synthesizes high-quality images but also ensures no constraint violations (0-tolerance). This stands in contrasts to other methods, that show increased constraint violations in out-of-distribution contexts, as shown by the dark red bars in Figure 8 . PDM can be adapted to handle complex governing equations using ODEs and can be guarantee satisfaction of out-of-distribution constraints with no decrease in sample quality. 
discussion and limitations: 
 In many scientific and engineering domains and safety-critical applications, constraint satisfaction guarantees are a critical requirement. It is however important to acknowledge the existence of an inherent trade-off, particularly in computational overhead. In applications where inference time is a critical factor, it may be practical to adjust the time step t at which iterative projections begin, which guides a trade-off between the FID score associated with the starting point of iterative projections and the computational cost of projecting throughout the remaining iterations ( §F). Other avenues to improve efficiency also exists, from the adoption of specialized solvers within the application domain of interest to the adoption of warm-start strategies for iterative solvers. The latter, in particular, relies exploiting solutions computed in previous iterations of the sampling step and was found to be a practical strategy to substantially decrease the projections overhead.We also note the absence of constraints in the forward process. As illustrated empirically, it is unnecessary for the training data to contain any feasible points. We hold that this not only applies to the final distribution but to the interim distributions as well. Furthermore, by projecting perturbed samples, the cost of the projection results in divergence from the distribution that is being learned. Hence, we conjecture that incorporating constraints into the forward process will not only increase computational cost of model training but also decrease the FID scores of the generated samples.Finally, while this study provides a framework for imposing constraints on diffusion models, the representation of complex constraints for multi-task large scale models remains an open challenge. This paper motivates future work for adapting optimization techniques to such settings, where constraints ensuring accuracy in task completion and safety in model outputs bear transformative potential to broaden the application of generative models in many scientific and engineering fields. 
conclusions: 
 This paper was motivated by a significant challenge in the application of diffusion models in contexts requiring strict adherence to constraints and physical principles. It presented Projected Diffusion Models (PDM), an approach that recasts the score-based diffusion sampling process as a constrained optimization process that can be solved via the application of repeated projections. Experiments in domains ranging from physical-informed motion for video generation governed by ordinary differentiable equations, trajectory optimization in motion planning, and adherence to morphometric properties in generative material science processes illustrate the ability of PDM to generate content of high-fidelity that also adheres to complex non-convex constraints as well as physical principles. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?Answer: [Yes] Justification: NeurIPS ethical guidelines have been reviewed by the authors, and this research meets the described standards.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The societal impacts of this work are discussed in Appendix A.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: This work and the associated resources do not pose risks of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: Existing assets have been used with consideration of licensing, and the original owners have been credited in the manuscript.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.• If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: Details on new assets (models/code/data) have been documented in the manuscript as well as the code base provided. Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used. • At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file. 
crowdsourcing and research with human subjects: 
 Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? Answer: [NA] Justification: Human subjects were not involved in the research conducted. Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects. • Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: The development of Projected Diffusion Models (PDM) may significantly enhance the application of diffusion models in fields requiring strict adherence to specific constraints and physical principles. The proposed method enables the generation of high-fidelity content that not only resembles realworld data but also complies with complex constraints, including non-convex and physical-based specifications. PDM's ability to handle diverse and challenging constraints in scientific and engineering domains, particularly in low data environments, may potentially lead to accelerating innovation and discovery in various fields.Diffusion models with soft constraint conditioning. Variations of conditional diffusion models [13] serve as useful tools for controlling task specific outputs from generative models. These methods have demonstrated the capacity capture properties of physical design [29] , positional awareness [3] , and motion dynamics [32] through augmentation of these models. The properties imposed in these architectures can be viewed as soft constraints, with stochastic model outputs violating these loosely imposed boundaries.Post-processing optimization. In settings where hard constraints are needed to provide meaningful samples, diffusion model outputs have been used as starting points for a constrained optimization algorithm. This has been explored in non-convex settings, where the starting point plays an important role in whether the optimization solver will converge to a feasible solution [21] . Other approaches have augmented the diffusion model training objective to encourage the sampling process to emulate an optimization algorithm, framing the post-processing steps as an extension of the model [10, 19] . However, an existing challenge in these approaches is the reliance on an easily expressible objective, making these approaches effective in a limited set of problems (such as the constrained trajectory experiment) while not applicable for the majority of generative tasks.Hard constraints for generative models. Frerix et al. [9] proposed an approach for implementing hard constraints on the outputs of autoencoders. This was achieved through scaling the generated outputs in such a way that feasibility was enforced, but the approach is to limited simple linear constraints. [17] proposed an approach to imposing constraints using "mirror mappings" with applicability exclusively to common, convex constraint sets. Due to the complexity of the constraints imposed in this paper, neither of these methods were applicable to the constraint sets explored in any of the experiments. Alternatively, work by Fishman et al. [2023 Fishman et al. [ , 2024] ] broadens the classes of constraints that can be represented but fails to demonstrate the applicability of their approach to a empirical settings similar to ours, utilizing an MLP architecture for trivial predictive tasks with constraints sets that can be represented by convex polytopes. We contrast such approaches to our work, noting that this prior work is limited to constraint sets that can be approximated by simple neighborhoods, such as an L2-ball, simplex, or polytope, whereas PDM can handle constraint sets of arbitrary complexity.Sampling process augmentation. Motivated by the compounding of numerical error throughout the reverse diffusion process, prior work has proposed inference time operations to bound the pixel values of an image dynamically while sampling [18, 23] . Proposed methodologies have either applied reflections or simple clipping operations during the sampling process, preventing the generated image from significantly deviating from the [0,255] pixel space. Such approaches augment the sampling process in a way that mirrors our work, but these methods are solely applicable to mitigating sample drift and do not intersect our work in general constraint satisfaction.In the following section, further details are provided as to the implementations of the experimental settings used in this paper. Microstructures are pivotal in determining material properties. Current practice relies on physicsbased simulations conducted upon imaged microstructures to quantify intricate structure-property linkages [4] . However, acquiring real material microstructure images is both costly and timeconsuming, lacking control over attributes like porosity, crystal sizes, and volume fraction, thus necessitating "cut-and-try" experiments. Hence, the capability to generate realistic synthetic material microstructures with controlled morphological parameters can significantly expedite the discovery of structure-property linkages.Previous work has shown that conditional generative adversarial networks (GAN) [11] can be used for this end [5] , but these studies have been unable to impose verifiable constraints on the satisfaction of these desired properties. To provide a conditional baseline, we implement a conditional DDPM modeled after the conditional GAN used by Chun et al. [5] with porosity measurements used to condition the sampling.Projections. The porosity of an image is represented by the number of pixels in the image which are classified as damaged regions of the microstructure. Provided that the image pixel intensities are scaled to [-1, 1], a threshold is set at zero, with pixel intensities below this threshold being classified as damage regions. To project, we implement a top-k algorithm that leaves the lowest and highest intensity pixels unchanged, while adjusting the pixels nearest to the threshold such that the total number of pixels below the threshold precisely satisfies the constraint.Conditioning. The conditional baseline is conditioned on the porosity values of the training samples. The implementation of this model is as described by Ho and Salimans. Original training data. We include samples from the original training data to visually illustrate how closely our results perform compared to the real images. As the specific porosities we tested on are not adhered to in the dataset, we illustrate this here as opposed to in the body of the text.We observe that only the Conditional model and PDM synthesize images that visually adhere to the distribution, while post-processing methods do not provide adequate results for this complex setting.Projections. The penetration and floatation constraints can be handled by ensuring that the lowest point on the z-axis is equal to the floor height. Additionally, to control the realism of the generated figures, we impose equality constraints on the size of various body parts, including the lengths of the torso and appendages. These constraints can be implemented directly through projection operators.Conditioning. The model is directly conditioned on text captions from the HumanML3D dataset. The implementation is as described in [33] .Projections. For this experiment, we represent constraints such that the predicted path avoids intersecting the obstacles present in the topography. These are parameterized to a non-convex interior point method solver. For circular obstacles, this can be represented by a minimum distance requirement, the circle radius, imposed on the nearest point to the center falling on a line between p n and p n+1 . These constraints are imposed for all line segments. We adapt a similar approach for non-circular obstacles by composing these of multiple circular constraints, hence, avoiding overconstraining the problem. More customized constraints could be implement to better represent the feasible region, likely resulting in shorter path lengths, but these were not explored for this paper.Conditioning. The positioning of the obstacles in the topography are passed into the model as a vector when conditioning the model for sampling. Further details can be found the work presented by Carvalho et al., from which this baseline was directly adapted.The dataset is generated with object starting points sampled uniformly in the interval [0, 63]. For each data point, six frames are included with the position changing as defined in Equation 12and the initial velocity v 0 = 0. Pixel values are scaled to [-1, 1]. The diffusion models are trained on 1000 points with a 90/10 train/test split.Projections. Projecting onto positional constraints requires a two-step process. First, the current position of the object is identified and all the pixels that make up the object are set to the highest pixel intensity (white), removing the object from the original position. The set of pixel indices representing the original object structure are stored for the subsequent step. Next, the object is moved to the correct position, as computed by the constraints, as each pixel from the original structure is placed onto the center point of the true position. Hence, when the frame is feasible prior to the projection, the image is returned unchanged, which is consistent with the definition of a projection.Conditioning. For this setting, the conditional video diffusion model takes two ground truth frames as inputs, from which it infers the trajectory of the object and the starting position. The model architecture is otherwise as specified by Voleti et al..While the majority of our analysis focused on the developing these techniques to the sampling architecture proposed for Noise Conditioned Score Networks [25] , this approach can directly be adapted to the diffusion model variant Score-Based Generative Modeling with Stochastic Differential Equations proposed by Song et al. Although our observations suggested that optimizing across a continuum of distributions resulted in less stability in diverse experimental settings, we find that this method is still effective in producing high-quality constrained samples in others.We included an updated version of Algorithm 1 adapted to these architectures.Algorithm 2: PDM Corrector Algorithm1 x 0 N ∼ N (0, σ 2 max I) 2 for t ← -T to 1 do 3 for i ← -1 to M do 4 ϵ ∼ N (0, I) 5 g ← -s θ* (x i-1 t , σ t ) 6 γ ← -2(r||ϵ|| 2 /||g|| 2 ) 2 7 x i t ← -P C (x i-1 t + γg + √ 2γϵ) x 0 t-1 ← -x M t 9 return x 0 0We note that a primary discrepancy between this algorithm and the one presented in Section 4.2 is the difference in γ. As the step size is not strictly decreasing, the guidance effect provided by PDM is impacted as Corollary 5.3 does not hold for this approach. Hence, we do not focus on this architecture for our primary analysis, instead providing supplementary results in the subsequent section.We provide additional results using the Score-Based Generative Modeling with Stochastic Differential Equations. This model produced highly performative results for the Physics-informed Motion experiment, with visualisations included in Figures 11 and 12 . This model averages an impressive inception score of 24.2 on this experiment, slightly outperforming the PDM implementation for Noise Conditioned Score Networks. Furthermore, it is equally capable in generalizing to constraints that were not present in the training distribution. When analyzing both real and synthetic materials, heuristic-guided metrics are often employed to extract information about microstrucutres present in the material. When analyzing the quality of synthetic samples, the extracted data can then be used to assess how well the crystals and voids in the microstructure adhere to the training data, providing an additional qualitative metrics for analysis. To augment the metrics displayed within the body of the paper, we include here the distribution of three metrics describing these microstructures, mirroring those used by Chun et al..We observe that the constraint imposition present in PDM improves the general adherence of the results to the ground truth microstructures. This suggests that the Cond model tends to generate to certain microstructures at a frequency that is not reflected in the training data. By imposing various porosity constraints, PDM is able to generate a more representative set of microstructures in the sampling process.We highlight that unlike the approach proposed by [32] , our approach guarantees the generated motion does not violate the penetrate and float constraints. The results are tabulated in Table 1 (left) and report the violations in terms of measured distance the figure is either below (penetrate) or above (float) the floor. For comparison, we include the projection schedules utilized by PhysDiff which report the best results to show that even in these cases the model exhibits error.PhysDiff [32] The implementation described by [32] applies a physics simulator at scheduled intervals during the sampling process to map the diffusion model's prediction to a "physically-plausible" action that imitates data points of the training distribution. This simulator dramatically alters the diffusion model's outputs utilizing a learned motion imitation policy, which has been trained to match the ground truth samples using proximal policy optimization. In this setting the diffusion model provides a starting point for the physics simulator and is not directly responsible for the final results of these predictions. Direct parallels can be drawn between this approach and other methods which solely task the diffusion model with initializing an external model [10, 21] . Additionally, while the authors characterize this mapping as a projection, it is critical to note that this is a projection onto the learned distribution of the simulator and not a projection onto a feasible set, explaining the remaining constraint violations in the outputs. Step As shown in Figure 1 , the PDM sampling process converges to a feasible subdistribution, a behavior that is generally not present in standard conditional models. Corollary 5.3 provides insight into this behavior as it outlines the decreasing upper bound on 'Error' that can be introduced in a single sampling step. To further illustrate this behavior, the decreasing upper bound can be illustrated in Figure 14 .Error (U(x i t ), C) (log) Error (U(x i 4 ), C) Error (U( C(x i 8 )), C)To compare the computational costs of sampling with PDM to our baselines, we record the execution times for the reverse process of a single sample. The implementations of PDM have not been optimized for runtime, and represent an upper bound. All sampling is run on two NVIDIA A100 GPUs. All computations are conducted on these GPUs with the exception of the interior point method projection used in the 3D Human motion experiment and the Constrained Trajectories experiment which runs on two CPU cores. We implement projections at all time steps in this analysis, although practically this is can be optimized to reduce the total number of projections as described in the subsequent section. Additionally, we set M = 100 and T = 10 for each experiment. The increase in computational cost present in PDM is directly dependant on the tractability of the projections and the size of M .The computational cost of the projections is largely problem dependant, and we conjecture that these times could be improved by implementing more efficient projections. For example, the projection for Constrained Trajectories could be dramatically improved by implementing this method on the GPUs instead of CPUs ( * ). However, these improvements are beyond the scope of this paper. Our projection implementations are further described in §C.Additionally, the number of iterations for each t can often be decreased below M = 100 or the projection frequency can be adjusted (as has been done for in this section for the CPU implemented projections), offering additional speed-up.As defined in Equation 2, PDM uses a score-matching objective to learn to the gradients of the log probability of the data distribution. This understanding allows the sampling process to be framed in a light that is consistent to optimization theory, allowing equivalences to be drawn between the proposed sampling procedure and projected gradient descent. This aspect is also integral to the theory presented in Section 4.2.Other DDPM and DDIM implementation utilize a variation lower bound objective, which is a tractable approach to minimizing the negative log likelihood on the network's noise predictions. While this approach was inspired by the score-matching objective, we empirically demonstrate that iterative projections perform much worse in our tested settings than models optimized using this training objective, producing clearly inferior solutions in the Physics-informed experiments and failing to produce viable solutions in the material science domain explored.This approach (visualized in Figure 15 ) resulted in an FID score of 113.8 ± 4.9 on the Physicsinformed Motion experiment and 388.2 ± 13.0 on the Constrained Materials experiment, much higher than those produced using the score-matching objective, adopted in our paper. We hold that this is because the approach proposed in our paper is more theoretically sound when framed in terms of a gradient-based sampling process.Proof of Theorem 5.2Proof. By optimization theory of convergence in a convex setting, provided an arbitrarily large number of update steps M , x M t will reach the global minimum. Hence, this justifies the existence of Ī as at some iteration as i -→ ∞, x i t + γ t ∇ x i t log q(x i t |x 0 ) 2 ≤ ∥ρ t ∥ 2 will hold for every iteration thereafter.Consider that a gradient step is taken without the addition of noise, and i ≥ Ī. Provided this, there are two possible cases.Case 1: Assume x i t + γ t ∇ x i t log q(x i t |x 0 ) is closer to the optimum than ρ t . Then, x i t is infeasible. This claim is true by the definition of ρ t , as x i t + γ t ∇ x i t log q(x i t |x 0 ) is closer to µ than is achievable from the nearest feasible point to µ. Hence, x i t must be infeasible. Furthermore, the additional gradient step produces a point that is closer to the optimum than possible by a single update step from the feasible region. Hence it holds thatEQUATIONas the distance from the feasible region to the projected point will be at most the distance to ρ t . As this point is closer to the global optimum than ρ t , the cost of projecting x i t + γ t ∇ x i t log q(x i t |x 0 ) is greater than that of any point that begins in the feasible region. Case 2: Assume x i t + γ t ∇ x i t log q(x i t |x 0 ) is equally close to the optimum as ρ t . In this case, there are two possibilities; either (1) x i t is the closest point in C to µ or (2) x i t is infeasible. If the former is true,EQUATIONNext, consider that the latter is true. If x i t is not the closest point in C to the global minimum, then it must be an equally close point to µ that falls outside the feasible region. Now, a subsequent gradient step of x i t will be the same length as a gradient step from the closest feasible point to µ, by our assumption.Since the feasible region and the objective function are convex, this forms a triangle inequality, such that the cost of this projection is greater than the size of the gradient step. Thus, by this inequality, Equation 13 applies.Finally, for both cases we must consider the addition of stochastic noise. As this noise is sampled from the Gaussian with a mean of zero, we synthesize this update step as the expectation over,EQUATIONor equivalently as represented in Equation 11.NeurIPS Paper Checklist 


Decision: The paper was an interesting application of ideas from diffusion and constrained optimization to have more controllable generative models that can satisfy some known physical principles/constraints. 
The reviews overall are a bit borderline. Some of it stems from the paper subject itself which lies at an intersection of modern ML methods and applications to practical problems. The method being being more expensive than an unconstrained setup is likely irresolvable. The paper does report a fair variety of metrics to understand how well the method works. The method introduced is quite general purpose and should find wider applicability and open-sourcing the implementation would be valuable to the community.

Reviews: review: paper_summary: paper_summary: This paper proposes a diffusion model that imposes constraints on the generated output. However, the constraints here are not abstract verbal instructions but rather formalizable constraints. The authors propose projected diffusion model sampling to perform constraint conditional log-likelihood maximization at each time step. They demonstrate the effectiveness of the proposed method on a variety of application problems.
strength_weakness: strength_weakness: 1. Equation (3) is introduced without a clear explanation of the relationship between the reverse diffusion process of score-based models and maximizing the conditional density function.
2. The details of the projection algorithm used in each experiment are only described in words. Pseudocode would be helpful.
questions: questions: 1. Please provide a more detailed explanation of the similarity between the reverse diffusion process of score-based models and maximizing the conditional density function.
2. As mentioned in Reference [20], it is common to gradually decrease the learning rate and $\gamma$ to improve the convergence rate of the proximal algorithm. Have you tried decreasing $\gamma$ as $1/i$ or $1 / \sqrt i$ within the inner loop of projected diffusion model sampling?
3. Please provide the computational order of the projection algorithm used in each experiment.
limitations: limitations: Section 7 adequately describes the limitations.
paper_summary: paper_summary: This paper proposed Projected Diffusion Models (PDM) inspired by stochastic gradient Langevin dynamics, to generate samples that satisfy given arbitrary constraints and remain within the specified regions. The authors claimed that the proposed algorithm is compatible across various applications, including satisfying morphometric properties when synthesizing materials, physics-informed motion generation, constrained path planning and human motion generation, and provided theoretical analysis to guarantee the generated samples reside in the constrained regions.
strength_weakness: strength_weakness: 1. In limitation discussion, computational overhead is mentioned. How's the time complexity of PDM compared to ``Cond``, ``Cond+`` and ``Post+``?
2. Not quite sure if I really understand how ``Cond+`` is different from PDM in general except the Langevin dynamics part from the descriptions but perceive the differences between them and if I am correct, ``Cond`` is in classifier-free guidance regime, ``Cond+`` is in classifier guidance regime and ``Post+`` only applies projection once at the end of sampling process. It would be better to describe them using mathematical expressions.

Not a serious problem, but for the writing style, I found it a bit hard to follow when cited papers stays in the middle of the sentence, such as,  ``, emulating the post-processing approaches of [10, 21] in various domains presented next.`` from line 190-191, but I do not really know what 10 and 21 are about and what authors point to unless I check the reference list. It also looks a bit weird to me: ``The implementation of this model is as described by Ho and Salimans.``, where no number follows after the citation.
questions: questions: Following the weaknesses above,

- For Equation (3b), it requires the whole trajectory in the reverse process satisfying the constraints, while $x_t$s' while $t$ is large are basically noise. Is this optimization setup reasonable?  
- What's the time complexity to find the nearest feasible point while doing the projection? Will take it too long time? How do you choose $M$ in the algorithm 1 to control the number of Langevin Dynamics being applied?
- While doing projections, from my understanding, the projected feasible point should be on the boundary of the constraints. Does that mean the final samples will be all projected on the exact boundary then? 

To summarize this question, even though PDM has better metrics than others, I still doubt the sampling distribution is not aligned with the data distribution since the samples might be all on the constraint boundaries.

- When evaluating out-of-distribution samples, is the constraint satisfaction rate the only metric?
limitations: limitations: Limitations and positive societal impact are discusses in the paper, but negative societal is not mentioned.
paper_summary: paper_summary: This paper proposes Projected Diffusion Models (PDM) for constrained generative modeling. The key idea is to reframe the denoising process of diffusion models as a constrained optimization problem, iteratively projecting the generated samples onto a constraint set at every denoising step. The method is validated on several applications involving both convex and non-convex constraints, generally outperforming both conditional models and models that only project onto the constraint set after the last denoising step. The authors provide feasibility guarantees for convex constraints and optimality guarantees for convex constraints and likelihoods.
strength_weakness: strength_weakness: * The paper heavily relies on Fréchet Inception Distance (FID) scores for the empirical comparison of models in Sections 5.1, 5.2, and 5.4. While FID is a standard metric for evaluating generative models on ImageNet-like images, it is unclear how meaningful it is in the specialized domains explored in this work (e.g., material microstructures, human poses, physics-informed simulations).
* The paper presents PDM as a novel optimization technique that "recasts traditional denoising strategies as a constrained optimization problem". However, if I understand it correctly, it is an application of commonly employed post-processing projections (e.g. the references provided in the manuscript) to the Langevin MCMC sampling scheme of Song and Earmon, NeurIPS 2019. It would be helpful if the authors could clarify this and outline any additional novelty/contributions.
questions: questions: * The implementational details in Appendix F state that all experiments were carried out with $T=10$ diffusion time steps. This is a very small value, compared to the hundreds or thousands of time steps that are used in standard image diffusion papers. Is there a reason for this?
* How many samples were used to compute the quantitative performance metrics reported in Section 5? Would you expect a change in relative performance when using the same inference-time compute budget, i.e., generating 50% more conditional samples in the Constrained Materials application, since the conditional model is ~50% faster?
* Figure 2 visualizes the constraint satisfaction rate of a conditional model as a function of the relative error tolerance, given in percentage points. What does an error tolerance of 100% correspond to?
limitations: limitations: The main methodological limitation of the proposed technique is its increased computational cost. The authors adequately address this limitation and suggest different approaches to overcome it. As outlined above, I believe that the main limitation of the experimental evaluation presented in the manuscript is its reliance on the FID metric.
paper_summary: paper_summary: This paper proposes an approach to sample generation using diffusion models which adheres to a set of constraints. The approach is based on the score matching formulation of diffusion models, and applies a projection step which finds the nearest feasible sample to each iteration of SGLD. A theoretical justification for the method is proposed, which provides intuition around why the approach yields superior results to conditioning and post-processing methods. The method is evaluated on a wide variety of experiments covering multiple classes of constraints (including non-convex) and data modalities, and is demonstrated to yield high-quality samples which strictly adhere to non-trivial constraints.
strength_weakness: strength_weakness: I think a mathematical description of the projection operator and constraints for the experiments 5.1-5.3 would be helpful to understand the degree of non-convexity and difficulty for these experiments. While I have a rough idea, being concrete would help a lot. 

Also, I believe it is important to be more forward in the main document around the computational overhead when it comes to applying the projection operator at each SGLD iteration. While I believe that high overhead does not particularly detract from the contribution (this can be addressed in future works), I think it detracts from the paper to hide this information in the appendix and not refer to it obviously in the main text.
questions: questions: line 158: Does "convergence of convex constraint sets" simply mean that the projection step will always return a feasible solution? Or are you referring more broadly to the DDPM iterates?
line 186: Double colon typo "::"
Fig. 1: Given that the approach projects to the constraint set at each iteration, why are we still seeing constraint violation early on in the diffusion process? Is this plotting the pre-projected samples? It is unclear what is actually being plotted.
line 272: typo, "unfeasible"
line 331: some comment around the reasonability of the assumption of the score function being convex (locally for small step size say?) would be appreciated
limitations: limitations: The authors have clearly described the limitations of the approach, however I believe a clearer statement of compute overhead would improve the clarity around the limitations.

Comments: title: Thanks again
Response to the authors
comment: Thank you very much for your recognition of our work. Once again, we sincerely appreciate the time and effort you spent reviewing our paper.
Thank you for your continued discussion with us.

Firstly, please note that we report very strong empirical results in our paper, demonstrating state-of-the-art in Section 5.1 and 5.3 *(we believe you referencing “high FID score from experiment 5.1” may be a typo here)* and state-of-the-art for constraint satisfaction methods on Sections 5.2 and 5.4, dramatically out-performing post-processing techniques which have found an important place in the literature.

Your concerns seem now to be more theoretically-based. First, we would encourage you to look at our discussion with Reviewer Jdpw, where we provide additional discussion for our formulation of the reverse process as an optimization problem. We have **proved convergence properties of our method in Section 6, under standard convexity assumptions**. However, let us go further - **[A] proves convergence for non-convex optimization using Langevin dynamics**, making our method broadly applicable, and again our strong results are empirical evidence of this.

As we have shown that the reverse process is the minimization of $- \log q(x)$, our method adapts the reverse process from a simple variation of gradient descent (SGLD) to a variation of projected gradient descent. **Projected gradient descent is known to converge (even in non-convex settings)** [B].

Our samples are not “meaningless”. We have provided rigorous theoretical justification (and an excess of empirical results), and provided evidence that this work could be a significant resource to adopt generative models in many engineering and scientific domains where physical principles and user-imposed constraint must be satisfied for the outputs to be considered as “valid”. We do hope that you agree with us on the significance and potential broader impact of these results, given the evidence provided.

---
[A] Raginsky, Maxim, Alexander Rakhlin, and Matus Telgarsky. "Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis." Conference on Learning Theory. PMLR, 2017.

[B] Vu, Trung, Raviv Raich, and Xiao Fu. "On convergence of projected gradient descent for minimizing a large-scale quadratic over the unit sphere." 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing (MLSP). IEEE, 2019.
Thank you for clarifying where you stand with respect to our rebuttal. As demonstrated by the heuristic-based metrics in Figure 13, we have been intentional in selecting domain specific metrics for the experiments we have conducted. It appress that, as you have acknowledged the inclusion of these metrics for Section 5.1, your only remaining concern is the use of FID score in Sections 5.2 and 5.4.

As we pointed out in our original response, the use of FID scores for these experiments *comes directly from the literature/baselines with which we compare*. FID score remains the predominant metric used for evaluation of the HumanML3D dataset used in Section 5.2. This is not a metric we just decided to use, but a metric that is necessary to compare to existing literature. While you may make the claim that this is not an appropriate metric, we would argue that this is the most appropriate metric as it is the only one that allows us to compare to existing work. **When working on these problems that the community is adopting as benchmarks it seems most appropriate to use the metrics that have already been selected.**

A similar case can is made for Section 5.4. As this evaluation metric is used throughout the diffusion model literature, **and particularly in the literature we are comparing to,** we’d ask to please not punish us for something as accepted and recognized in the community.
My apologies, there was an issue with the visibility settings of my original response. Here it is, including a discussion of my remaining concerns:

I would like to thank the authors for the detailed response.

---

**Re W1:** The paper heavily relies on Fréchet Inception Distance (FID) scores for the empirical comparison...

> First, we would like to point out, as the reviewer has acknowledged, that FID is a standard metric for evaluating generative models, and evaluation using this metric is more than reasonable.

Perhaps there is a misunderstanding. My concern was that FID is a standard metric for evaluating generative models **trained on natural image datasets**, since it relies on the last layer of an Inception v3 model that was trained on ImageNet. It is still unclear to me why it would be "more than reasonable" to expect these representations to facilitate meaningful performance comparisons when applied to material microstructures, 3D human poses, etc. since these application domains are strongly out-of-distribution with respect to the ImageNet data the Inception model was trained on.

That being said, I appreciate the detailed clarifications and the pointers to alternative performance metrics. As far as I could tell, most of them are constraint satisfaction metrics (i.e. the *success percentage* in Figure 6 and the *penetrate* and *float* distances in Table 1), rather than sample quality metrics. However, the heuristics in Figure 13 do suggest that PDM matches the properties of the ground truth data better than conditional models on two of the three presented metrics. In light of this, I will raise my score by 1, but as outlined above my main concern regarding the applicability of FID to the settings in Sections 5.1, 5.2 and 5.4 remains.
Thank you for your response forwards my concerns. I still believe the algorithm that directly projecting sample to the feasible region at every Langevin dynamics sampling step is the **flaw** it possesses, which can be reflected onto the **high FID score** from experiment 5.1. As what I mentioned from the last round of response, ``Theoretically, the step size in the Langevin dynamics should be chosen very carefully such that the injected noise brings the randomness while not messing up the effect from the gradient step. in this case, there is no guarantee, and I also kindly disagree that the projection can still maintain the sample at its' noisy level and this will definitely mess up the sampling scheme, or the noise level should be correspondingly adjusted according to the projection but that seems infeasible`` is the reason why **the direct projection will make the algorithm deficient**, but not all projection ideas would not work. For example, [1] also has the idea that bounces the samples back to the feasible regions through the whole reverse sampling process but it is with reasonable assumptions and justifies why this would not mess up the sampling scheme as well, which makes more sense. [1] also pointed out, in its introduction section, ``Although thresholding avoids failure, it is theoretically unprincipled because it leads to a mismatch between the training and generative processes.``, which is also the information that I tried to convey for the whole time.

I firmly believe that the samples satisfying constraints are necessary, but if they are not generated in the way that follows the principle of how diffusion models train and sample, and how the theory works behind, the samples are meaningless from my point of view.

[1] Lou, Aaron, and Stefano Ermon. "Reflected diffusion models." International Conference on Machine Learning. PMLR, 2023.
Thank you for your continued efforts to review our work. We are glad the additional clarifications have assisted in your understanding of our method. 

Do we understand that there are no more limitations pending and that as you mentioned our work is sound, novel, and significant? As you have stated that the only remaining hesitation for you to increase your score is that you would like to see our revisions, let us provide the changes we intend to make:

- We will expand the beginning of Section 4 explaining our derivation of Equation 3a. We will begin by introducing the update step Equation 4 and explaining briefly explaining SGLD. This will be verbatim:

>The application of the reverse diffusion process of score-based models is characterized by iteratively fitting the initial noisy samples $x_T$ to the learned approximation of $p(x_0)$. 
This optimization is formulated such that a variation of the traditional gradient descent algorithm, *Stochastic Gradient Langevin Dynamics* (SGLD), is used to iteratively transform a sample from the Gaussian distribution $q(x_T | x_0)$ to a sample from the learned distribution $q(x_1 | x_0)$. The update step is provided by:
\begin{equation}
    x_{t}^{i+1} = x_{t}^{i} + \gamma_t \nabla_{x_{t}^{i}} \log q(x_{t}^i|x_0) + \sqrt{2\gamma_t}\epsilon,
\end{equation}
where $\epsilon$ is standard normal and  $\gamma_t > 0$ is the step size. This step is repeated $M$ times for each $x_T$ to $x_0$. To prevent a deterministic behavior, an additional term is added to the gradient descent algorithm, $\sqrt{2\gamma_t}\epsilon$, drawing from *Langevin Dynamics* \cite{song2020score}.

>SGLD can be viewed as an extension of traditional gradient descent algorithms, where the primary goal is to minimize a specified objective function. However, SGLD incorporates a stochastic component, introducing noise into this process. Formally, this procedure converges to a region characterized as an "almost-minimizer" of the objective function, with proximity to the minimizer bounded by $\frac{d^2}{(\sigma^{1/4}\lambda^*)}\log(1/\epsilon)$, where $\sigma^2$ represents the variance schedule, $\lambda^*$ denotes the uniform spectral gap of the Langevin diffusion, and $d$ is the dimensionality of the problem, as outlined in reference [C].

>In this framework, the SGLD algorithm yields samples that are statistically concentrated around the global maximum of the underlying density function, as noted in [B].  Thus, the reverse diffusion process can effectively be approximated by minimizing the negative log-likelihood $-\log q(x_t|x_0)$, or equivalently as maximizing the density function $\log q(x_t|x_0)$ at each given noise level, the gradients of which are estimated by $s_\theta (x_t^i, t)$.

>In traditional score-based models, at any point throughout the reverse process, $x_t$ is _unconstrained_. 
When these samples are required to satisfy some constraints, the objective remains unchanged, but the solution to this optimization must fall within a feasible region $C$,
\begin{equation}
\min_{x_{T}, \ldots, x_1} \sum_{t = T}^{1} -\log q(x_{t}|x_0)
\end{equation}
\begin{equation}
\text{s.t.} \quad x_{T}, \ldots, x_0 \in C
\end{equation}

> Operationally, the negative log likelihood is minimized at each step of the reverse Markov chain, as the process transitions from $x_T$ to $x_0$. In this regard, and importantly, the objective of the PDM's sampling process is aligned with that of traditional score-based diffusion models. 

> To avoid low density data regions, the sample is optimized to conform to the previous distribution in the Markov chain before proceeding to the consecutive distribution, the transitions being  ensured by setting $x_{t-1}^{0} = x_{t}^{M}$, where as $x_t^M$ is the final iterate of the previous time step.

- Following this, we will continue with Section 4.1.

As we believe this is a fairly simple change, requiring rearranging some text and adding a few short paragraphs, we do not mind providing this. We hope that this summary of our intended revisions, provided at your request, addresses the remaining doubts. We also notice that adding this paragraph several to only strengthen our contribution, in addition to the strong empirical and theoretical support provided for what we believe is an important contribution for the adoption of diffusion models in scientific and engineering domains. As you have stated that this is the only remaining concern, we hope that you will consider advocating for strong acceptance.
Again, we would like to thank you for your assessment and are grateful for the recognition of our paper’s *methodological clarity* and comprehensive *experimental scope*. It has come to our attention that you have updated your score to a 5. As we have not had the opportunity to engage with you during this discussion period, we would like inquire as to what remaining concerns you have preventing you from advocating for strong acceptance? We believe we have addressed all the questions that were in your original review and would welcome the opportunity to discuss any outstanding doubts.
Thanks to the references you provided, I now understand how the reverse diffusion process converges to its Gibbs distribution, which concentrates on the maximum likelihood solution. I have increased the soundness score from 1 to 2. I also agree that considering the reverse diffusion process as an optimization problem is a significant contribution of this paper. However, it is a pity that the introduction of this contribution is very vague, described as "akin to maximizing the density function." Although the author mentioned a plan to revise this, I hesitate to strongly accept the paper without seeing how the revision will be made, so I would like to keep my score.
Thank you for your response and allowing us the opportunity to clarify further:
> The objective of the reverse diffusion process is to maximize the density function

First, let us clarify that the papers we cited [24,25] are to point out that the update step taken in our paper (before projecting) is identical to those in existing literature (for instance it is also Equation 4 in [24]). You will also notice that our Algorithm 1 directly employs this update step (line 6) as in [24]. Next, notice that immediately following the introduction of this update step in [24] it is explained that the sample converges to $p(x)$ by the repeated application of this update step under the regularity conditions in [A]. These conditions are introduced in Equation 2 of [A] and are expressly for the purpose of “convergence to a local maximum” by ensuring that the solution “will reach the high probability regions”.

This can also be directly derived from SGLD. For instance, refer to the formalization of SGLD in [B] Equation 1.2: 

$$
d\mathbf{X}(t) = -\nabla F_n (\mathbf{X}(t))dt + \sqrt{2\beta^{-1}}d\mathbf{B}(t)
$$

First, notice that this matches the update step in Equation 4 (of our paper and [24]). As explained in [B], this optimization procedure “concentrates around the global minimum of $F_n(x)$” and parallels the task of directly minimizing $F_n$. Adapted to our update step (as shown below), this becomes the minimization of $-\log q(x)$ or, in other words, the maximization of the density function.

$$
d\mathbf{X}(t) = \nabla \log q (\mathbf{X}(t))dt + \sqrt{2\gamma(t)}d\epsilon(t)
$$

Additionally, we will note that [C] provides a complete proof showing that SGLD converges to an "almost-minimizer" of the function, as the noise maintains some degree of stochasticity. Hence, from these works we can support our claim that this process is "akin to maximizing the density function".

Importantly, we would like to remind the reviewer that formalizing the reverse process as an optimization problem is part of the novelty of our work. Hence, existing papers have not explicitly presented the reverse process as in our work. But *the presentation of this objective in our paper is posed to show how can one incorporate constraints in the reverse process, and how these can be attained using a gradient-based projection method.*

Finally, we obviously agree with your point about using the score function to increase tractability, but this does not change the overall objective from an optimization standpoint. Indeed again our method (Algorithm 1) directly employs the update step (line 6) used in [24] (with the variation of using a projected gradient version).

We thank the reviewer for the opportunity to clarify these points. It is our intent to update the manuscript to better explain our derivation of Equation 3a; thank you for the suggestion.

> Indeed, our implementation in Section D (Algorithm 2) ...

We apologize if our response was unclear here. We reference Algorithm 2 because it dynamically adjusts the gradients to improve convergence; however, we do not explicitly attempt the learning rate schedules referenced in the question.

---
Again, thank you for your willingness to engage with us during the discussion phase. We hope our responses clarify your points. Are there any additional concerns that we could address? We are ready to provide further insights as needed.


---
[A] Welling, Max, and Yee W. Teh. “Bayesian learning via stochastic gradient Langevin dynamics.” Proceedings of the 28th international conference on machine learning (ICML-11). 2011.

[B] Xu, Pan, et al. “Global convergence of Langevin dynamics based algorithms for nonconvex optimization.” Advances in Neural Information Processing Systems 31 (2018).

[C] Raginsky, Maxim, Alexander Rakhlin, and Matus Telgarsky. "Non-convex learning via stochastic gradient langevin dynamics: a nonasymptotic analysis." Conference on Learning Theory. PMLR, 2017.
> [T]he proof provided in the section 6 assumes the feasible region is convex, which is very strict and doesn’t apply in many cases.

From an optimization perspective, assuming convexity of the constraint set is more than reasonable. In fact, when providing convergence guarantees, this is almost universally assumed (as guaranteeing convergence of non-convex constraint sets is an unsolved problem).
While the theoretical guarantees hold for convex settings, our experiments extend to non-convex constraints as well (see Section 5.3). In constraint optimization, this is reality that is widely accepted as also supported empirically by our results.
Please notice that *inclusion of theoretical results is meant to strengthen the understanding of our work, and should not be considered as a weakness.*

---
We hope our responses clarify your points. Are there any additional concerns that we could address? We are ready to provide further insights as needed.

(2/2)
Thank you for continuing to engage with us during this discussion phase. We would like to take the opportunity to respond to the points you have made.
> [W]e shouldn’t sacrifice the sample quality to [satisfy the constraints.]

While this may be the case for some applications - such as recreational image generation, the opposite is true in scientific and engineering applications that motivate this study. When considering the practicality of diffusion models to these domains, the ability to satisfy constraints is integral to the viability of the outputs! In such settings, high quality samples are desirable, making the FID score still relevant, **but feasible samples are necessary**. Thus, it is not sufficient to merely provides samples which report well on this single metric, making our method stand out. *Furthermore, if FID was the only metric that mattered in these settings, post-processing methods would never have been proposed in previous work, as empirically we have demonstrated that these provide much worse FID scores.* Yet these methods have still found an important place in existing discourse!

Note that this paper deals with real scientific and engineering settings presenting unique challenges such as data scarcity and the need to satisfy out-of-distribution constraints. In many scientific applications, such as the material science application we study, data collection is extremely expensive (it means synthetizing new materials) and the data collected may not provide feasible samples. This is exactly the in our material science application (see Section 5.1). There, it is essential that exact morphomoteric properties are satisfied, but these settings were never observed nor measured before!
We remind the reviewer here that these constraints are imposed by expert material scientist with whom we collaborate and whose work particularly motivated this paper. The settings explored require stringent constraint adherence for the laboratory testing (which our work is currently undergoing attesting for the broader impact of our method).

Similarly, constraints are necessary to generate accurate simulations (Sections 5.2 and 5.4) and clearly integral to trajectory optimization. Samples which are not feasible provide no merit when exact constraint satisfaction must be considered. The ODE constrains enforced to capture the motion of free-falling object also allow us to generate objects falling on other planets, when such data is unlikely to be collected (see our simulations in Section 5.4).

_Next, let us note a simple but important fact pertaining to constrained optimization in general_: when imposing constraints on an optimization procedure, inherently the feasibility space shrinks, and thus, of course, the cost objective typically is higher (in case of a minimization procedure) than that associated to an unconstrained one. However, when constraints are part of the optimization problem, solutions (or samples) which do not satisfy the constraints **cannot be considered “optimal”, or better solutions than feasible ones**. In our work, infeasible samples are not viable.

We would also like to highlight that the “much lower and better FID score” that is referred to in the response cannot be directly compared as Yuan et al. uses an **external simulator to post-process their outputs**; here, the simulator takes as input a vector of points produced by the generative model and is crucial to produce realistic results. This  dramatically alters the diffusion model’s outputs! The baseline we provide is a more apt point of comparison with regard to the FID scores because it does not rely on external simulators, thus effectively compares the ability of generative models.

Also note that in all experiments our FID scores are *very close or outperform the conditional model* (for instance Section 5.1 outperforms in terms of FID and heuristic-based metrics and Section 5.3 outperforms in the domain specific metrics). Furthermore, given that in many of these settings the training data is not necessarily feasible, sampling from a subdistribution of this data set will inherently result in FID score increases (as the diversity of the samples is limited by some margin for this conditional distribution).
While it may make sense to still use a conditional model in settings where constraint adherence is not valued, we will reiterate that in many settings this necessary. Hence, we disagree with the “blanket statement” provided.

(1/2)
I thank the authors' responses; however, the explanations confirms my doubt and concerns that the extra projection of the sample into the constrained set after every Langevin dynamics is not rational from my perspective. Theoretically, the step size in the Langevin dynamics should be chosen very carefully such that the injected noise brings the randomness while not messing up the effect from the gradient step. in this case, there is no guarantee, and I also kindly disagree that the projection can still maintain the sample at its' noisy level and this will definitely mess up the sampling scheme, or the noise level should be correspondingly adjusted according to the projection but that seems infeasible. This can be shown from the some metrics that evaluates the sample quality and how well they aligned with the data distribution . For example, the 3D human motion experiment from section 5.2 has worse FID score compared to the baseline that the paper provided, which is 0.71 (PDM) versus 0.63 (baseline). This paper also mentioned [1], which couldn't guarantee the samples always falling into the constrained region, but has much lower and better FID score, 0.551 and 0.433 as what they reported in [1].

I understand that every sample generated by PDM that this paper proposed satisfy the constraints, but in my point of view, we shouldn't sacrifice the sample quality to achieve the goal. More or less, the proof provided in the section 6 assumes the feasible region is convex, which is very strict and doesn't apply in many cases.


[1] Ye Yuan, Jiaming Song, Umar Iqbal, Arash Vahdat, and Jan Kautz. Physdiff: Physics-guided
460 human motion diffusion model. In Proceedings of the IEEE/CVF International Conference on
461 Computer Vision, pages 16010–16021, 2023.
Thank you for your detailed response to my comments. This has addressed my comments, and I am glad the authors' will make the necessary changes to the manuscript. My opinion of the paper is still positive and remains unchanged by the discussion period.
Thank you for your detailed response.

> The objective of the reverse diffusion process is to maximize the density function, and Equation 4 shows the actual update steps that are learned by the score-based model: a noisy (SGLD) gradient ascent on the density function. 

Please cite papers that state that the objective of the reverse diffusion process is density function maximization.
I cannot find that the score-based methods maximize the likelihood of the density function in [24,25].
I believe Song et al. [24,25] are not likely-based. By modeling the score function instead of the density function, we can sidestep the difficulty of intractable normalizing constants of the density function.  

My current score for soundness = 1 reflects the above concern about Equation (3).

> Indeed, our implementation in Section D (Algorithm 2) ...

Since the $\gamma$ in Algorithm 2 is not always decreasing, your answer to Question 2 is no, right?
I don't think it's a problem that the answer to this question doesn't significantly detract from the main contributions.

>  Additionally, we would encourage you to refer to our response to Reviewer p9zr Weakness 1 where we provide the mathematical formalization of the projections.

Thank you for providing the answer to Question 3 and the mathematical formalization of the projections to p9zr. It is very helpful to understand the overview of the projection algorithms, and I recommend including them in the main text.
As the discussion period is nearing its end, we wanted to ask if there are any follow-up points we can clarify. Please also note our summary in the previous comment. Many thanks!
As the discussion period is nearing its end, we wanted to ask if there are any follow-up points we can clarify. Please also note our summary in the previous comment. Many thanks!
As the discussion period is nearing its end, we wanted to ask if there are any follow-up points we can clarify. Please also note our summary in the previous comment. Many thanks!
As the discussion period is nearing its end, we wanted to ask if there are any follow-up points we can clarify. Please also note our summary in the previous comment. Many thanks!
We thank Reviewer p9zr for their thoughtful feedback and for recognizing the strengths of our submission and praising our experiments as _diverse and interesting_. We appreciate the opportunity to clarify the aspects of our work that you highlighted in your review.
In our main rebuttal, we addressed the following key points:
1. **Detailed Mathematical Descriptions**: We have provided a more detailed mathematical description of the projection operators used in experiments 5.1-5.3. This includes the specific constraints applied and their computational complexities, which help illustrate the degree of non-convexity and the challenges associated with these experiments.
2. **Computational Overhead**: We acknowledge your point and this information is now more prominently discussed.
3. **Other Clarifications**: For Figure 1: it shows the samples post-gradient step but pre-projection. This helps in understanding the iterative convergence of the method within the constraint set through the diffusion process. Additionally, we provided a clearer explanation of what we mean by “convergence of convex constraint sets” in line 158, linking it to both feasibility and optimality in the context of proximal gradient descent.

Are there any additional concerns that we could address? We are ready to provide further insights to assist in your evaluation and to enhance the understanding of our findings.
We appreciate Reviewer yvsu’s detailed assessment and are grateful for the recognition of our _paper’s methodological clarity_ and _comprehensive experimental scope_.
In our main rebuttal, we addressed the following key points:
1. **Use of FID Scores**: Our evaluation does not only consider FID scores (which is the standard metric for evaluating diffusion) but also included results on constraint satisfiability at various fidelity levels, and provided by additional domain-specific metrics reported in our experiments (see full response for details).
2. **Optimization Technique**: The distinctiveness of our method from traditional post-processing projections relies on the integration of projections directly into the Langevin dynamics, which is a novel contribution to the field. This approach not only improves sample quality but also ensures constraint adherence throughout the diffusion process. The theoretical analysis is also a important novel contribution of this work which justifies our modeling choice and provides guarantees for constraint adherence for important constraint classes.
3. **Computational Efficiency**: We addressed concerns about the use of a reduced number of diffusion steps (T=10), explaining that this setting is also what is adopted by previous score-based diffusion models and aligns with our model’s efficiency and sufficiency for achieving high-quality results. We also discussed how our method maintains efficiency compared to baselines, particularly when generating feasible samples within a given compute budget, showing that our model is, in effect, much faster in such settings.

Are there any additional concerns that we could address? We are ready to provide further insights to assist in your evaluation and to enhance the understanding of our findings.
We appreciate the feedback provided by Reviewer e4oh and are grateful that _our proposed method’s effectiveness and the theoretical underpinnings_ were recognized as strengths of our work.
In our main rebuttal, we addressed the following key points:
1. **Methodological Differences and Formalizations**: We provided a detailed explanation to distinguish our method from the baseline proposed: Cond+, Cond, and Post+ techniques. These, together with additional mathematical expressions will also be reflected in the final version of our paper.
2. **Computational Overhead**: We compared the time complexity of our Projected Diffusion Models with other models and highlighted the efficiency of our approach, especially when constraint adherence is critical, which is the case in all setting studied. For instance, in Section 5.1, with the error tolerance is within 5%, our method generates nearly 7 times as many feasible samples within the same compute budget.
3. **Optimization Setup**: As also pointed out in the paper, we remarked why projections needs to be applied to the entire trajectory. Our response included a theoretical backing and empirical evidence demonstrating the effectiveness of our approach, particularly in avoiding divergences from the target distribution.
4. **Sample Diversity**: We clarified how our projections guide the sampling to a feasible subdistribution (Figure 1), addressing concerns as to samples falling on the constraint boundary. Furthermore, we have explained how this is captured in the FID score - as it is widely accepted that this is the most appropriate method for assessing how well the sampling distributions and data distributions align.

Are there any additional concerns that we could address? We are ready to provide further insights to assist in your evaluation and to enhance the understanding of our findings.
We sincerely appreciate Reviewer Jdpw’s thorough evaluation and are grateful that _our method’s effectiveness under various constraints_ and the _comprehensive coverage of computational cost impacts_ were recognized.
In our main rebuttal, we addressed the following key points:
1. **Clarification on Equation (3)**: We provided a detailed explanation of how Equation (3) relates to maximizing the conditional density function through the reverse diffusion process. The subsequent equations and the update steps, explained in our rebuttal, demonstrate the rigorous theoretical underpinnings of our model.
2. **Details of Projection Algorithm**: We have provided references to the detailed mathematical formalization and the actual code in our submission, ensuring that the implementation details are accessible and transparent. We will also provide additional formalization in the paper, based on the response above.
3. **Dynamic Adjustment of Parameters**: We clarified our method’s approach to parameter adjustment (which indeed explores a dynamic adjustment of $\gamma$ in Appendix D). A discussion on how these adjustments affect the model’s performance across different experimental setups is also included.

Are there any additional concerns that we could address? We are eager to provide further information to assist in your evaluation and to enhance the understanding of our findings.
>**Question 3: While doing projections, from my understanding, the projected feasible point should be on the boundary of the constraints. Does that mean the final samples will be all projected on the exact boundary then? To summarize this question, even though PDM has better metrics than others, I still doubt the sampling distribution is not aligned with the data distribution since the samples might be all on the constraint boundaries.**

The projection will provide a point on the boundary of the constraints *if the original sample was infeasible.* In effect, this is why *Post+* and *Cond+* report such high FID scores, as more likely than not (for example, see Figures 2 and 8) prior to post-processing these samples are infeasible. In contrast, PDM samples converge to feasible subdistributions. For instance, notice that in Figure 1 later timesteps result in no violations of the constraints. This implies that unless subsequent gradient steps were consistently along the constraint boundaries (for all the samples visualized), *the samples output did not fall on the constraint boundaries and instead were from unique points within the feasible subdistribution.* The effect of guiding the sampling with projections is that PDM samples are "moved" to a subset of the distribution which is still optimal (maximizes the density function) but is also feasible. The projections solely enforce that the generated samples are taken from this region of the distribution.

Furthermore, FID scores are widely used in image generation tasks because of how well they benchmark the similarity between distributions. This metric is particularly useful because *it reliably asesses both sample quality and diversity;* models which lack sample diversity perform very poorly when reporting FID scores. It is _widely accepted that this is the most appropriate method for assessing how well the sampling distributions and data distributions align_ and we show how well our proposed method performs in FID scores, while also satisfying the imposed constraints. We believe this is a huge strength of the proposed approach and hope you see its significance given the many scientific and engineering domains in which this can be adopted, as we demonstrate in our experiments. 

>**Question 4: When evaluating out-of-distribution samples, is the constraint satisfaction rate the only metric?**

When answering this, we are presuming you are referring specifically to Section 5.4, although please correct us if that is not the case. We also report FID scores here; in this setting, the FID scores of the out-of-distribution generations did not change from the scores reported for the in-distribution generations. Hence, we do not report these separately, but they are applicable to both settings.

Additionally, we'll note that similarly various out-of-distribution constraints are added in Section 5.3 (the red obstacles in Figure 5 which were not present in the training data). Our method is equally robust in these settings, with consistent results across the studied topographies. 

---
We believe that our detailed responses provide the necessary clarifications to all your questions and reinforce the robustness and significance of our work. In light of these clarifications, we kindly request that you re-evaluate your score to reflect the merit and significance of our contributions. Our work makes substantial contribution for the application of generative processes for various engineering and scientific application settings requiring satisfaction of constraints and physical rules, as we demonstrate in the paper. 

If there are any additional, specific reasons for the current assessment, we would appreciate further justification to understand and address any remaining concerns. Thank you for your consideration!

(3/3)
--- 
Once again, we would like to express our thanks for your review. We have addressed each question and concern presented in your review. We believe our thorough responses provide the necessary clarifications to justify the significance of our work.
With these points in mind, we would greatly appreciate if you would re-evaluate your score to reflect the merit and significance of our contributions. If there are any additional specific reasons for the current assessment, we would appreciate further clarification and would be happy to discuss further. Thank you for your consideration!

(3/3)
Dear Reviewers and AC,

It has come to our attention that at this point in time some comments may be visible to the reviewers while others will remain hidden until the end of the author rebuttal phase (on Tuesday). This clarification was shared today by the NeurIPS PCs as this differs from the rebuttal process for many other venues. We apologize for any confusion this may have caused. The complete responses will be available when the rebuttals are made visible.

Thank you all for your understanding!
>**Q1: The implementational details in Appendix F state that all experiments were carried out with $T = 10$ diffusion time steps...**

This is actually *standard for the score-based models we used*. Specifically, we would like to point the reviewer to Section 5 of [24] where they explain in the "Setup" paragraph that they set this value to 10 (note that these authors use $L = 10$, which is equivalent in their notation).

>**Question 2: How many samples were used...? Would you expect a change in relative performance when using the same inference-time compute budget, i.e., generating 50% more conditional samples in the Constrained Materials application, since the conditional model is ~50% faster?**

For the computed metrics, we used 2000 samples from each model for FID metrics and additional metrics provided in Section E. You bring up an interesting point as to the relative performance, which we assume to refer to the number of generated feasible samples, within a given given compute budget. For the example that was referenced, as one may extrapolate from Figure 2, while the conditional model may generate 3000 samples within the time span it took for PDM to generate 2000 samples, if the error tolerance is less than ~35% (which is an absurdly high margin in this setting!!) then PDM generates many more feasible samples than the conditional model within the same compute budget. This discrepency is further emphasized when the tolerance is within a more reasonable margin, such as 5% where **PDM generates nearly seven times as many feasible samples within the same compute budget.** Hence, when constraints are integral to the outputs, PDM does, in fact, outperform conditional models in relative speed as well!

>**Q3: Figure 2 visualizes the constraint satisfaction rate... what does an error tolerance of 100% correspond to?**

When representing the porosity levels, we provide a percentage of pixels that should be below a provided threshold, as dark regions of the image represent damaged regions of the microstructure. Note that in our generated data higher porosity levels have more dark regions than lower porosities. In Figure 2, the x-scale represents the deviation between the generated data's porosity and the specified porosity. For example, if an image had 50% porosity and 40% porosity was specified, the error tolerance that would make this feasible is 10%. A reasonable error tolerance would likely be below 10%, although for precision applications this would still be far be too high. Notice that our proposed method **guarantees** constraint adherance here, which is key for the scientific application tested. 

*A note on error tolerance in Figure 2:* We have been working directly with a material scientist collaborator in this  domain, and for their applications it is _necessary_ to generate results which report zero-violations, making wide error tolerances inviable for their work. This was a key motivation for our development of PDM. We would like to emphasize that generation of microstructures for energetic materials is a critical real-world problem in material science, presenting unique challenges such as data scarcity and the need to satisfy out-of-distribution constraints. This experiment has significant practical implications for the creation of new material structures, and our results are undergoing testing in laboratory settings. This speaks about the significance of our method. 

>**L1: The main methodological limitation... is its increased computational cost... the main limitation of the experimental evaluation presented in the manuscript is its reliance on the FID metric.**

First, we would agree with the reviewer point about increased computational cost. This is an inherent byproduct of constraining any optimization problem, especially when providing guarantees upon the adherence to the constraint set. However, we would point you to our response to Question 2 for more context on this overhead. Additionally, we will note that in many settings, including those studied in this paper, *constraint-agnostic model cannot be used due to the necessity of adhering to the constraint set.* Our method is specifically tailored to scientific and engineering applications and the associated challenges including settings with low data, the absence of meaningful conditioning values, out-of-distribution generation, and particularly when exact constraint satisfaction in integral to the data quality. In such settings, conditional models often cannot be used.

Second, we will point the reviewer to our response to Weakness 1. We hope that the reviewer will take the opportunity to revisit the additional metrics that we bring attention to here and also to examine the evaluation criteria used by the referenced baselines. As this is the primary justification provided for the score that the reviewer provided, we would ask you to consider raising your score with this provided context.

(2/3)
>**Weakness 1: The paper heavily relies on Fréchet Inception Distance (FID) scores for the empirical comparison...**

First, we would like to point out, as the reviewer has acknowledged, that FID is a standard metric for evaluating generative models, and evaluation using this metric is _more than reasonable_. However, we believe the reviewer may have missed the additional metrics we use which are tailored to specific domains. 
- For Section 5.1, we use **the same heuristic-based metrics used by Choi et al. [5]**; we report these in Section E.1, finding the realism of PDM's generations surpass the baselines using this evaluation as well.
- Additional metrics for Section 5.2 are reported in Section E.2, although we will highlight that the compared baselines report FID as their primary metric for generation quality [29,30].
- Section 5.3 uses **domain specific metrics from the state-of-the-art baseline [3]**.
- Finally, we highlight that the primary metrics used by the baseline in Section 5.4 [26] is a variation of FID score, making this an appropriate point of comparison.

Thus, we believe our use of FID score is indeed appropriate for the settings explored, especially as **the paper indeed already reports several additional metrics to supplement these results**.

>**Weakness 2: The paper presents PDM as a novel optimization technique that "recasts traditional denoising strategies as a constrained optimization problem". However, if I understand it correctly, it is an application of commonly employed post-processing projections (e.g. the references provided in the manuscript) to the Langevin MCMC sampling scheme of Song and Earmon, NeurIPS 2019. It would be helpful if the authors could clarify this and outline any additional novelty/contributions.**

Thank you for this question. As already discussed in our related work section, the novelty of this paper arises from (1) framing the reverse diffusion process as a constrained optimization problem, (2) formulating and implementing **general** projections for constraints and physical principle with important relevance for scientific and engineering applications, and (3) proposing a novel theoretical analysis to show that constraint adherance is not only feasbile, but guarantees can also be attained in many important classes of constraints with significant application relevance.

Note that, we have indeed acknoledged that some post-processing steps have been proposed previously, and indeed we compare aginst such methods. However, these methods post-process **after** the sampling process, which is a key difference, in light of our theoretical analysis. As discussed in Section 6, our approach is based on the insight that the cost of projection increases with the number of unconstrained steps (see also the illustration in Section E.3). Theorem 6.2 supports this by showing that projection cost is lower when the sample starts from the feasible set, leading to better convergence properties. We explain that this is because high projection cost results in significant divergence from the distribution in all settings we explored. To our knowledge, this is the first theoretical result on constraint adherence in diffusion models, marking a key contribution of our work.

Crucially, we have shown that post-processing approaches produce samples of much lower quality than those produced by PDM, improving material synthesis FID scores by over 30%, feasibility rates of tarjectory optimization by 90%, and increasing the quality of physics-based video generaiton by a factor of two. We argue theoretically, and empirically demonstrate, that a single post-processing projection leads to a significant divergence from the distribution in all the settings we examined. **This is a key novel contribution of this paper**. 

Additionally, similar to the theory behind projected gradient descent, using projections throughout the reverse process is advantageous as we strive to reach the constrained minimum. Once again, this is a key novel contribution of our work. 
Projections help guide the optimization process toward a region of the distribution that satisfies the constraints. Without regular projections, the optimization path may venture far outside the feasible space, potentially resulting in poor convergence properties, as we demonstrate in several experiments in the paper. In many ways projections guide the diffusion process of PDM in a similar spirit to the effect of conditioning *Cond* models, with the exception that this guidance imposes hard constraints on the generation process which we have shown are vastly more reliable than state-of-the-art conditioning techniques.

(1/3)
>**Question 1: For Equation (3b), it requires the whole trajectory in the reverse process satisfying the constraints, while $x_t$s' while $t$ is large are basically noise. Is this optimization setup reasonable?**

Our decision to enforce constraints at every step of the denoising process is driven by theoretical insights as well as practical observations. 

Theoretically, as discussed in Section 6, this approach is based on the insight that the cost of projection increases with the number of unconstrained steps (also see illustration in Section E.3). Theorem 6.2 supports this. importantly, this theorem shows that the projection cost is lower when the sample starts from the feasible set, leading to better convergence properties. This is further demonstrated empirically, as methods imposing constraints only at the final step perform significantly worse than PDM, especially in non-convex settings. Importantly, "performing worse" here refers *not* to constraint adherence, but to the ability to generate images from the original data distribution (i.e., high-fidelity outputs) while satisfying the imposed constraints. Indeed, we showed how projecting only in the last step results in a significant divergence from the distribution in all settings we explored. Additionally, paralleling the theory behind projected gradient descent, as we aim to converge to the constrained minimum, there is a clear benefit in using projections to guide the optimization process toward a region where the minimum satisfying the constraints can be found. Without regular projections, the optimization path may explore regions far outside the feasible space, potentially leading to poor convergence properties.
We also notice that this is the first theoretical result on constraint adherence in diffusion models, to the best of our knowledge, and is indeed a key contribution of our work. 

Practically, we've observed that unconstrained optimization steps can deviate significantly from the feasible domain, making it difficult to converge to a feasible sample. This is effectively illustrated in Figure 1 in the paper, where our proposed guidance scheme, PDM, navigates the constraint-defined landscape to ensure convergence to a feasible sub-distribution.

Notice also that continuous projection throughout the sampling process is crucial for converging to feasible solutions in non-convex settings, as shown in Section 5.3. In these experiments, projecting throughout the sampling process allows our method to converge to feasible solutions consistently (i.e., **we never produced an unsatisfiable trajectory**!) with a single sample. In contrast, as shown in Figure 6, the *Cond+* method, a state of the art method introduced to solve this specific problem in [21] which imposes "constraints" (a post-processing step) only at the final step, was never able to correct the infeasible samples, with the solver repeatedly reporting local infeasibility. This alone should be considered as a substantial improvement over the state-of-the-art.

>**Question 2: What's the time complexity to find the nearest feasible point while doing the projection? Will take it too long time? How do you choose $M$ in the algorithm 1 to control the number of Langevin Dynamics being applied?**

For the first part of this question, please refer to our response for Weakness 1.

To answer the latter part, we use the value of $M$ used by Song et al. [24] when this architecture was proposed, finding that the samples effectively converge using this value in our experiments. Analysis of an optimal value for $M$ could be conducted but is out of the scope of the question investigated in this work, which is on demonstrating a new constraint and physical principle adherence in generative models.

(2/3)
>**Weakness 1: In limitation discussion, computational overhead is mentioned. How's the time complexity of PDM compared to Cond, Cond+ and Post+?**

We report the runtime difference between PDM and the other baselines in Section F. We would be happy to provide details on the time complexity of our projection methods: 
- **Constrained Materials:** This is a knapsack constraint. While in general knapsack problems are NP-complete, the version adopted (which uses integers as weight) is known to be weakly NP-complete and admits a fully polynomial approximation. It is solved efficiently in $O(nm)$ where $n$ is the number of pixels and $m$ is the number of values to compute for the dynamic program. 
- **3D Human Motion:** This is a scaling constraint that runs in $O(n)$ time, where $n$ is the length of the internal representation.
- **Constrained Trajectories:** The interior point method used by the nonconvex solver in our implementation [27] has a time complexity of $O(n^{3.5})$ where $n$ is the number of variables in the Quadratic Program. In this case, there are 128.
- **Physics-informed Motion:** This projection runs in $O(n+m)$ where the size of each image is $n$ by $m$.

As we note in Section F, these projection operations have not been optimized for runtime, and these time complexities represent an upper bound. Additionally, we will refer you to our response to Question 2 by Reviewer yvsu, as this is closely related to your question of additional overhead (*as our PDM is much faster than conditional models when constraint adherence is necessary to generate viable samples!*).

>**Weakness 2: Unclear how Cond+ is different from PDM in general except the Langevin dynamics part from the descriptions but perceive the differences between them and if I am correct, Cond is in classifier-free guidance regime, Cond+ is in classifier guidance regime and Post+ only applies projection once at the end of sampling process. It would be better to describe them using mathematical expressions.**

To clarify,
- *Cond:* is a classifier-free guidance. The conditional model uses the guidance scheme provided at the end of page 2 (line 83).
- *Cond+:* This model is identical to *Cond*, but the final output $x_1$ is projected using our projection operator $\mathcal{P}_C$. Thus, this is a baseline introduced by this work and inspired by generative models using post-processing steps. 
- *Post+:* This is an unconditioned score-based model of identical architecture to PDM. Instead of constraining the entire generation as with PDM, we project only on the final output $x_1$.

*Cond+* and *Post+* are equivalent to current post-processing methods, and we would like to emphasize the significant performance gap between these methods and PDM. We appreciate your suggestion of including a more clear formalization of these methods in the paper and will do so in our final version of the manuscript.

(1/3)