Constraining Generative Models for Engineering Design with Negative Data
Authors: Lyle Regenwetter
Giorgio Giannone
Akash Srivastava
Dan Gutfreund
Faez Ahmed
Venue: Accepted by TMLR ()

Abstract: Generative models have recently achieved remarkable success and widespread adoption in society, yet they still often struggle to generate realistic and accurate outputs. This challenge extends beyond language and vision into fields like engineering design, where safety-critical engineering standards and non-negotiable physical laws tightly constrain what outputs are considered acceptable. 
In this work, we introduce two approaches to guide models toward constraint-satisfying outputs using `negative data' -- examples of what to avoid. Our negative data generative models (NDGMs) outperform state-of-the-art NDGMs by 4x in constraint satisfaction and easily outperform classic generative models using 8x less data in certain problems. To demonstrate this, we rigorously benchmark our NDGMs against 14 baseline models across numerous synthetic and real engineering problems, such as ship hulls with hydrodynamic constraints and vehicle design with impact safety constraints. Our benchmarks showcase both the best-in-class performance of our new NDGM models and the widespread dominance of NDGMs over classic generative models in general. In doing so, we advocate for the more widespread use of NDGMs in engineering design tasks.

Published in Transactions on Machine Learning Research (12/2024)
Constraining Generative Models for Engineering Design with Negative Data
Lyle Regenwetter
regenwet@@mit.cd edu
Massachusetts Institute of Technology
Giorgio Giannone
ggiorgioShmit.com edu
Amazon
Massachusetts Institute Technology
Akash Srivastava
akash. wtt..rrrrissttaa@@im...om
MIT-IBM Watson Al Lab
Dan Gutfreund MIT-IBM Watson AI Lab
dgutfrrehus..o wtt.bussssbm..........
Faez Ahmed
faea@mitt.co edu
Massachuselts Institute of Technology
Reviewed on OpenReview: oppenreveee. net/ forum? id= FNBu2
Abstract
Generative models have recently achieved remarkable success and widespread adoption in socicty, yet they often struggle to generate realistic and accurate outputs. This challenge extends beyond language and vision into fields like engineering design, where safety-criticn engineering standards and non-negotiable physical laws tightly constrain what outputs considered acceptable. In this work, we introduce novel training method to guide generative model toward monstranttsaatsfyiiii outputs using 'negative data' - examples of what to avoid. Our negative-data generative model (NDGM) formulation easily outperforms classic models, generating 1/6 as many constraint-violating samples using 1/8 as much data in certain problems. It also consistently outperforms other baselines, achieving a balance between constraint satisfaction and distributional similarity that is unsurpassed by any other model in 12 of the 14 problems tested. This widespread superiority is rigorously demonstrated across numerous synthetic tests and real enginecring problems, such as ship hull synthesis with hydrodynamic constraints and vehicle design with impact safety constraints. Our benchmarks showcase both the best-in-class performance of our new NDGM formulation and the overall dominance of NDGMs versus classic generative models. We publicly release http.//////.eeeeeeeeeteee////oaa
the code and benchmarks at
1 Introduction
Generative models have demonstrated impressive results in vision, language, and speech. However, even with massive datasets, they struggle with precision, often generating physically impossible images or factually incorrect text responses. These mistakes are examples of constraint violation; ideally, generative models would be constrained to only generate valid 'correct' samples. While constraint violation is nuisance in image or text synthesis, it is paramount concern in domains like engineering design with high-stakes (including safety-critical) constraints. Gonerative models synthesizing designs for car or airplane components, for example, may be subject to geometric restrictions (such as colliding components)) functional requirements (such as load-bearing capacity), industry standards, and manufacturing limitations.
Work completed Massachusetts Institute of Technology

---

X
(a) Positive datapoints are constraint-satislying exam- ples that generative models should broadly replicate.
(b) Negative datapoints are coustraint-voolatinn exam- ples that generative models should avoid.
(c) Generative models are typically trained using only positive data, causing them to ignore constraints.
(d) Negative data allows generative models to better satisfy constraints through tighter density estimates.
Negalive data helps generative models learn real-world data distributions, which often have gaps in their support caused by constraints. For example, by examining bike frames with disconnected components, a model can better learn to generate geometrically valid framos.
Figure 1:
As generative models are increasingly applied to engineering problems, their blatant violation of objective, ubiquitous, and non-negotiable constraints becomes increasingly problematic. We assert that this constraint- violation issue is largely attributable to the fact that generative models are classically shown only 'positive' coonstrain--satisfying) datapoints during training, and are never exposed to 'negative" (constraint-violatingg datapoints to avoid. Completely satisfying constraints using this training approach is equivalent to learning a binary classification problem with only one class present in the data, a challenging task. Instead, by studying negative data in addition to positive data, generative models can better avoid constraint constraint-violatinng samples during generation (Figure 1). This aligns with their distribution-matchin objective since negative datapoints should have ncar-zero density in the original real-world distribution that the model is trying to mimic. We will refer to models that train using negalive data as negalive-data generalive models, or NDGMs.
Although many existing generative modeling formulations, such as binary class-condiiional models, can be simply adapted into NDGMs, specialized NDGMs have also been proposed. The prior ofato-of-tho-art (SOTA) method suffers from two major shortcomings: It never specifically learns the densily ratio between the positive and negative data and it often suffers from mode collapse. We conceptualize and test a new NDGM that overcomes these issues through the use of a multi-class discriminative model that learns individual density ratios and a Determinantal- Point-Processs (DPP)-based loss that encourages diverse sample sets. Through extensive benchmarking, we demonstrate that new formulation outperforms both simple baselines and the current state-of-the-art NDGMs on highly non-convex test problems, a variety of real-world engineering tasks, and high-dimensional image-based tasks. Our key contributions are summarized below.
(i) We introduce a new NDGM which significantly outperforms the constraint satisfaction of vanilla generative models while addrcssing mode collapse issues in current NDGMs. These advancements are enabled by estimating individual density ratios and introducing a diversity-based training loss.
(ii) We evaluate our model on an expansive set of benchmarks including specially-conssruueed test problems, authentic engineering tasks featuring real-world constraints from engineering standards, and final high-dimensional topology optimization study. We compare our model to 10 baseline training formulations spanning adversarial models, variational autoencoders, and diffusion models.
(iii) We demonstrate that our model frequently achieves an unmatched tradeoff between constraint satisfaction and distribution learning, some cases attaining 95-98% lower coist violation than classic generative models, while achieving top-three distributional similarity scores.
(iv) We show that our NDGM model can significantly outperform vanilla models, generating 1/6 as many constraint-violating samples using only 1/8 as much data. This makes our model an excellent choice in data-constrained problems involving constraints.
2

---

2 Background
In this section, we introduce key considerations in constraint satisfaction for engineering design, discuss divergence minimization in generative models, then introduce negative-data generative models. For more background and related work, see Appendix C.
2.1 Constraints in Engineering and Design
Constraints are ubiquitous in design. A designer creating ship hulls, for example, must adhere to a medley of geometric constraints, performance requirements, and safety regulations from authoritative bodies. Generating constraint-satisfyye designs can be exceedingly difficult. As many practitioners turn to data-driven generative modcls to tackle enginecring problems (Regenwetter et al., 2022a), this difficulty remains (Woldseth et al., 2022; Regenwelter et al., 2023). For example, even a generative model that sees 30K examples of valid ship hulls can only generate valid hulls with a 6% success rate in our experiments.
The overwhelming majority of deep generative models in design do not actively consider constraints (Woldseth et al., 2022; Regenwetter et al., 2022a), despite constraint satisfaction being explicit goal in many of the design problems they address (Oh et al., 2019; Nie et al., 2021; Bilodeau et al., 2022; Chen et al., 2022; Chen & Fuge, 2019; Cheng et al., 2021). Several engineering design datasets feature constraint-violatinn designs (Regenwetter et al., 2022b; Bagazinski & Ahmed, 2023; Giannone & Ahmed, 2023; Mazé & Ahmed, 2023), and many others have checks for validity (Whalen et al., 2021; Wollstadt et al., 2022), allowing datasets of constraint-violating (negative) designs be curated. In some cases, datasots of positive examples are even created through search by rejecting and discarding negative samples (Bagazinski & Ahmed, 2023; Regenwetter et al., 2022b), making negative data essentially free. In any problem where negative data is available or be generated, NDGMs can be applicd.
2.2 Divergence Minimization in Generative Models
Before discussing divergence minimization in NDGMs, we first discuss divergence minimization in conventional generative models. Let P2(2) be the (positive) data distribution and (1) the distribution sampled by the generalive model. Given N samples from (*)),, the objective of generative modeling is to find setting 0* of 0, such that, for an appropriate choice of discrepancy measure, P1 ~ pp. A common choice for this discrepancy mcasure is the Kullback-Leibler or KI. divergence:
KL KLlpollep] 11plppp111 - po(x)) f
[log px(x)
dx.
(1)
Ppp (x)
To minimize the discrepancy, we find 0r as the solution to the following optimization problem:
0- = arg min (lolpplllpp)
(2)
In practice, direct optimization of Eq. (2) is often intractable. As such, it is common in deep generative modeling to learn 9 by using either tractable lower bound to a slightly different variant of Eq. (2) (Kingma & Welling, 2013; Burda al., 2015; Нo et al., 2020; Sonderby et al., 2016) or by using plug-in or direct estimators of the divergence measure (Casella & Berger, 2002; Sugiyama et al., 2012a; Gutmann & Hyvärinen, 2010; Srivastava et al., 2017; Goodfellow et al., 2014; Srivastava et al., 2023; Poole et al., 2019). In both of these cases, under certain conditions, as N + oo, theoretically, it holds that, 0 + 0r. However, since N is limitod, there remains a finite discrepancy between the model and data distributions. This mismatch often manifests in pe allocating high probability mass in regions where Pp may not have significant empirical support. In domains such as engincering design, where invalid (negative) designs tend to be very close to the valid (positive) designs, this leads to the generation of invalid designs with high probability. This lack of precision underpins the relatively limited success of deep generative models in the engineering design domain (Regenwetter et al., 2023).
Divergence minimization in GANs. Generative Adversarial Networks (GANs) (Goodfellow el al., 2014; Arjovsky et al., 2017; Mohamed & Lakshminarayanan, 2016; Srivastava et al., 2017; Nowozin et al., 2016) are a powerful framework for generating realistic and diverse data samples. GANs have two main components: a generator fo, which generates samples according to the density pa, and a discriminator fox which is a
3

---

binary classifier. The generator learns to generate synthetic data samples by transforming random noise into meaningful outputs, while the discriminator aims to distinguish between real and generated samples. The standard GAN loss can be written as:
[(,, Φ) - (x)) [x)] + (x)( log(fo(x))))-
(3)
Training a GAN involves iteraling over ming max; L(0, Φ). GANs can also be interpreted in terms of estimating the density ratio (Gutmann & Hyvärinen, 2010; Srivastava et al., 2017) between the data and the generated distribution r(x) = Pp(x)/po(xx) This ratio can be estimated by a discriminative model as To = fx)) - fx)x) and TΦ = 1 gives us Pa = Pp- The optimal discriminator prediction and generator
distribution are:
(x fo(x)= (po(x) + (x)) P0(x) pp(x).
(4)
Divergence minimization in other generative models. Many other types of generative models similarly minimize divergence between Po and Pp- These models include popular likelihood-bassd models like Variational Autoencoders (VAEs) (Kingma & Welling, 2013) and Denoising Diffussin Probabilistic Models (DDPMs) (Ho et al., 2020). We will not discuss the mathematics behind divergence minimization for these likelihood-based models, but we do benchmark several variants in our results. In general, we refer to unaugmented GANs, VAEs, and DDPMs as 'vanilla' models throughout the paper.
2.3 Negative- Data Generative Models (NDGMs)
In this section, we discuss the NDGM framework. We explain how generative models can be adjusted exploit negative data to improve constraint satisfaction. Let Ps denote the negative distribution i.c., the distribution of constrainl-violating datapoints. A key assumption in the negative data formulation is that Pp and Pn have nearly mutually exclusive support. Instead of training using only the positive distribution Pp- we now seek to train a generative model using both Pp and Pn- In this section, we discuss several existing methods to do so. These methods range from simple basclines like auxiliary classifiers to statc-of-the-art formulations like discriminator overloading.
2.3.1 Class Conditioning
Class conditional modeling is simple approach to incorporate constraints into generative models, which is popular in many design generation problems (Nie et al., 2021; Behzadi & Ilies, 2021; Mazé & Ahmed, 2023; Shin et al., 2023; Heyrani Nobari et al., 2021). In conditional modeling, a generative model typically conditions on the constraints denoted as c and learns a conditional distribution, p(x(c), where x represents the generated output. 'Off-tho-shelf" class-conditional models can be simple NDGMs, where the positive and negative data each constitute one class. During inference, the model attempts to satisfy constraints by generating conditionally positive samples. Broadly speaking, the negative data formulation for generative models can be seen as a specific case of class-conditional generation. However, we demonstrated in Appendix A, generic class-conditional training formulations for generative models are not effective specialized NDGMs.
2.3.2 Classifier- Clasiffer-Augmenneed Generative Models
Another common approach to actively satisfy constraints using generative models involves training a supervisod model to predict constraint satisfaction. After training, this supervised model is subsequently queried during generative model training inference. Often, this model predicts constraint violation likelihood, though it also predict intermediates that are combined in more complex constraint check (Wang et al., 2022). Typically, this classifier fy learns:
(x) (x())
(5)
(x(x) (x)
This frozen classifier can be incorporated into the training of generative model by adding an auxiliary loss, LPC to the generative model's loss, Las to calculate total loss, LT LaM A LPC, in (Regenwetter & Ahmed, 2022). Here, X is some weighting parameter and LPC is expressed as:
Lrc Ex) (og (x)).
(6)
4

---

Pre-trained classifiers for constraints can also be applied during inference in certain models, such in diffusion model guidance (Mazé & Ahmed, 2023; Giannone & Ahmed, 2023). They can also work alongside a trained generative model as rejection-sampling postprocessing step, which is discussed in Appendix D.
2.3.3 Discriminator Overloading (DO)
Discriminator overloading technique to directly incorporate negative data into GAN model training. This formulation was proposed in two of the first papers to train a generative model using both positive and negative data (though we have made slight modifications for generality): Rumi-GAN (Asokan & Seelamantula, 2020) and Negative Data Augmentation GAN (NDA-GAN) (Sinha et al., 2021). We refer to these formulations as *discriminator overloading' since the discriminator is 'overloaded' by learning to discriminale between (1) positives and (2) fakes or negatives. As such, the discriminator estimates:
(x( fx(x)= (po(x) + (1 ()) + Pp())
with X being a weighting parameter. As usual, the generator attempts to generate samples that are classified as real, in this case indicating that they look similar to positive data and dissimilar to negative data. The
loss is expressed as:
L(,, (x, (o) [o(x)] (x) (og(f((xx))) co..co (ogc/ lx)(fx(x))]=
(8)
In our benchmarks, we refer to GAN models trained using discriminator overloading GAN-DO models. Discriminator overloading is effective and can be considered the existing state-of- the-trt in NDGMs. However, as we will show, discriminator overloading has significant shortcomings which we address in our new training formulation.
3 Methodology
NDGMs suffer from a widespread tendeney toward mode collapse seen across a variely of baselines. Critically, the state-of-the-art GAN-DO training formulation is particularly prone to this issue due to its conflation of fakes and negatives during training. To address these shortcomings, we propose new NDGM training formulation that roduces two new innovations: First, and central to our approach, we propose to learn the ratios between the positive, negative, and fake distributions individually, rather than conflating the negatives and fakes. Second, we add an auxiliary diversity-based training objective to NDGM training which directly mitigates mode collapse. We describe these innovations in detail below.
3.1 Learning Individual Density Ratios Using Multi Class Discriminator
GAN-DO learns density ratio between Pp and amalgamation of Pr. and pe- Rather than conflating the negalive data and fake samples, instead advocale to learn pairwise density ralios between the three distributions. Noting that multi-class classifiers are strong density ratio estimators (Srivastava et al., 2023), we propose to learn these ratios using a multi-class discriminator. This multi-class discriminator model Icarns to discriminate three classes: positive, negative, and fake, thereby learning their pairwise density ratios:
(x) So,(()) (x(x) + (x) + (x) VCEp,n...
Note that fe,p a reweighted version of Eq. 7. Though this multi-class formulation is similar to discriminator overloading, instcad of showing the discriminator a weighted amalgamation of fakes and negatives (as in DO), the multi-class discriminator instead treats fakes and negatives separate classes, and can potentially refine its knowledge by distinguishing them. Complemented by generator model which tries to maximize fo.( this classifier fulfills the role of the discriminator in an adversarial training formulation. Notably, dap/Jan estimates Pp/Pn, which is never directly learned in the discriminator overloading formulation.
We also note that there are numerous other solutions to learn density ratios between positive and negative data distributions. For example, a direct estimator of Pa/Pn can operate alongside the classic discriminator in a two-discriminator NDGM variant. We discuss the general motivation for density ratio learning in NDGMs in Appendix C and the mathematical formulation behind the double discriminator variant in Appendix F.
5

---

3.2 Addressing Mode Collapse Using a Diversity-Based Loss
When augmented with validity-based training objectives, NDGMs tend to collapse in valid regions of the sample space. This effectively allows them to excel in validity and precision, but struggle with recall and diversity. This tendency arises because conservative NDGM will avoid regions of the distribution near the constraint boundary, resulting in incomplete coverage. One approach to improve recall is to explicitly encourage diversity of generated samples. Diversity is often a desired goal in generative modeling for engineering design applications (Regenwetter et al., 2023). As Chen & Ahmed (2021a) note, incorporating diversity can also help models generalize and avoid mode collapse. Determinantal Point Process (DPP)-based diversity measures (Kulesza et al., 2012) have been used in variety of generative applications in design (Chen & Ahmed, 2021b; Nobari et al., 2021; Regenwetter & Ahmed, 2022) and elsewhere (Elfeki et al., 2019; Mothilal et al., 2020).
The DPP loss is calculated using a positive semi-definite DPP kernel S. Entries of this matrix are calculated using some modality- and problem-dependent similarity kernel, such as the Euclidean distance kernel. The (i, element of can be expressed in terms of the similarity kernel k and samples I2 and x, Si.j = 2g); and the loss as:
1 (x) = logdet(S(x))) = B
1
Σ log A
(10)
where X; is the i-th eigenvalue of and B is the number of samples in the batch. The loss is incorporated by appending to the overall loss term of the generative model LaM:
Cr (x) LGM (xx + (D).
(11)
where y is a weighting parameter modulating the diversity loss contribution. Adding this loss to NDGMs can help the generative model achieve better coverage, an observation demonstrated in our experiments below,
3.3 GAN with Multi-Class Discriminator and Diversity Loss
We combine the above innovations into a new training formulation called GAN-MDD (Multiclass Discrimi- nator + Diversity). The generator trains to generate samples which minimize:
Le (x(x, の) = fo.p (xo) + 7 (io
(12)
To illustrate the effectiveness of this specific formulation, we inchude ablation studies on generation in Section 4.3. Training pseudocode is included in Appendix B. In the upcoming Section 4, we extensively benchmark GAN-MDD against baseline models and GAN-DO on variety of problems.
4 Experiments and Results
We now present experiments on (i) 2D densities, where we benchmark 11 different models including ours, the SOTA, bascline NDGMs, and vanilla models; (ii) A dozen diverse engincering tasks featuring real-world constraints from regulatory authorities and engineering standards, where we demonstrate the potency of our approach (iii) a high-dimensional free-form topology optimization problem where we explore the impact of negative data quality. We also perform ablation studies and data-efficiency studies. We include significant additional visualization and statistical testing for experimental results in Appendix A. Details on datasets and model training are included in Appendix F.
4.1 Extensive Benchmarking on 2D Densities with Constraints
We first construct a pair of 2D densities easy-to-visualize tests for NDGM models to visually showcase their characteristics. Despite being low-dinensional and relatively structured, these problems are very challenging for vanilla models and NDGMs alike. Notably, constraint-violating regions appear in close proximity to high-density regions of the valid data distribution. For a model to succeed, precise estimation of constraint boundaries is essential.
6

---

(a) Positive Data
(b) Negative Data
(c) DDPM
(d) DDPM + Guidance
(e) VAE
(f) GAN
(g) GAN-DO (SOTA)
(h) GAN-MDD (Ours)
Figure 2: Generated distributions from select generative models on Problem 1, a mixture of Gaussians with invalid region in the center of each mode. Positive data points and samples are shown blue and negative ones in black. Our proposed NDGM model, GAN-MDD learns the distribution most faithfully.
(a) Positive Data
(b) Negative Data
(c) DDPM
(d) DDPM + Guidance
(e) VAE
(f) GAN
(g) GAN-DO (SOTA)
(h) GAN-MDD (Ours)
Figure 3: Generated distributions from select generative models on Problem 2, a uniform distribution with many circular invalid regions. Positive data points and samples are shown in blue and negative ones in black. Our proposed NDGM model, GAN-MDD learns the distribution most faithfully.

---

Problem 1
Problem 2
10
DDPM
GAN (CC)
GAN
VAE VAE (CC)
DDPM
VAE (CC)
VAE
8
DDPM
12
Invalidity Rate (%)
(CC)
Invalidity Rate (%)
DDPM
10
(CC)
6
DDPM (Guided)
GAN
VAE
6
(CC)
(CLF)
GAN
GAN (CLF)
GAN GAN-MDD (CLF)
2
(Ours)
GAN-DO DDPM (SOTA) (Guided)
2
GAN-MDD
(Ours)
GAN-DO (SOTA)
VAE (CLF)
0
0.8
0.7
0.6 F1 Score
0.5
0.4
0.85
0.80
0.75
0.70
F1 Score
Figure 4: Comparison of Fl scores (t) and invalidity rates (4) for benchmarked models Problem (left) and Problem (right). Mcan scores over six instantiations are plotted. Scores closer to the bottom left are more optimal. Triangular markers indicate that the score lies off the plot. in the indicated direction. Class conditioning, classifier loss, and guidance are denoted with (CC), (CLF), and (Guided), respectively.
Models. We test 11 variants of GAN, VAE, and DDPM models. Among these are: three vanilla models (GAN, VAE, DDPM), trained only on positive data; three class conditional models (GAN, VAE, DDPM), trained both positive and negative data in binary class conditional setting as in Section 2.3.1; three models augmented with frozen pre-trained validity classifier to steer models during training (GAN, VAE) during inference using guidance (DDPM), in Section 2.3.2; GAN with discrininator overloading as in NDA-GANs (Sinha et al., 2021) and Mami-GANs (Asokan & Seelamantula, 2020) (GAN-DO) (Section 2.3.3); Our multi-clas--discimmnnet
GAN with diversity (GAN-MDD). Each model tested six times.
Metrics. We seek to measure each model's reliability constraint- salislactniissni distribution learning ability, and ability to avoid mode collapse. We therefore score each model on three metrics: 1) Invalidity the fraction of generated samples that violate the constraints (negative samples). 2) F1 score for generative models, a common distributional similarity metric proposed in Sajjadi et al. (2018). 3) DPP Diversity score, metric which highlights mode collapse and is measured as described in Section 3.2. The ideal model maximizes F1 and minimizes invalidity and DPP diversity.
Figures 2 and 3 plot the datasets and the generated distributions of a subset of the tested models. All models' distributions are plotted Figures 13 and in Appendix A. Compared baseline models, our GAN-MDD model learns the best estimate of the data distribution while avoiding constraint violation.
Results.
Figure 4 plots mean F1 scores and invalidity rates on both problems for all models. The scores confirm that our GAN-MDD achieves optimal tradeoff between statistical similarity and constraint satisfaction. Table contains the numerical scores, with mean and standard deviations over the six training runs, as well as statistical significance testing using 2-sample t-tests. Our GAN-MDD model significantly outperforms the ofate-of-thhear GAN-DO distributional similarity and diversity while achieving similar constraint- satisfaction performance.
4.2
Is Negative Data More Valuable than Positive Data?
Given an infinite amount of data, model capacity, and computational resources, generative models can theorotically approach an exact recovery of the underlying data distribution, Pp- In practical scenarios, however, data and computational throughput are limited, particularly in fields like engineering design and scientific research. Thus, simply increasing the volume of data is not viable strategy improve constraint satisfaction. Fortunately, we find that NDGMs can be significantly more data-cfficient. than vanilla generative models, giving them significant advantage in data-consttrined domains.
8

---

Table Study of invalidity rates for GAN-MDD trained with different numbers of positive datapoints (Np) and negative datapoints (Nv,). Note that GAN- MDD without negative data (N,, = 0) trains as vanilla. GAN. Diversity loss is turned off. Scores are averaged over four instantiations. Lower is better. NDGMs can generate significantly fewer constraint-violating samples, even when trained on orders of magnitude less data.
Figure 5: Comparison of invalidity rate (4) and number of datapoints (4) under various mixtures of positive and negative data on Problem 2. Points labeled using their proportion of negative data. Any amount of negative data tested improves per- formance. Interestingly, small proportion of nega- tive data (20%) yields better constraint satisfaction than higher proportions in this problem.
Problem
0%
Np = 1K
Np =
=1
Np = 16K
Nn = 0
10.7%
11.6%
11.9%
Invalidity Rate (%)
Nn = 1K
2.0%
0.7%
1.5%
Nn = 4K
0.5%
0.5%
0.5%
0%
0%
Nr = -6K
0.5%
0.7%
0.6%
Problem 2
50%
N₂ 1K
Np =K
Np 16K
80%
N₂ 0
6.0%
3.2%
3.3%
50%
80%
Nn 1K
2.3%
1.6%
1.9%
50%
Nn 1K
0.6%
0.7%
0.7%
20%
20%
N₂ 16K
0.5%
0.2%
0.2%
10
20
30
Datapoints (thousands)
In Table and Figure 5, we explore how the number of positive and negative training datapoints (N, and Nn, respectively) affect invalidity rates on the 2D test problems. Full scores with standard deviations and statistical tests included in Table 8. To remove confounding factors, benchmark fixed-size GAN-MDD model without diversity loss (note: GAN-MDD is equivalent to vanilla GAN when trained with no negative data). We test four model instantiations per datasct mixture.
We find that for pure positive datasets, adding more data yields diminishing returns, while mixing in negative data drastically improves constraint satisfaction. For example, Problem 1, with N, = 1K, N₂ 1K, GAN-MDD generates 1/6 as many invalid samples with 1/8 as much data compared N, 16K, N, = 0. On Problem 2 with N, 4K, N₁, 1 1K, GAN-MDD generates 1/5 as many invalid samples with 1/3 much data compared to Np 16K, N, vanilla models, practitioners seeking to improve their generative models may attain much more value by collecting even small amount of negative data, rather than additional positive data. This study also prompts an interesting research question: What is the optimal ratio of positive to negative data? rigorous answer, though certainly problem dependent, could lead to more officient datasct generation and curation.
Since NDGMs can be significantly more data-cfficient than
4.3 Ablation Study: Examining the Effect of Diversity Loss
In many negative data settings, diversity-based training objective can serve to modulate the tradeoff between constraint satisfaction and distributional similarity. To showcase this, we examine the performance of GAN-MDD over a sweep of diversity loss weights (y from Eq. 11). Simultaneously, we examine the performance of GAN-DO augmented with our DPP-based diversity loss. These experiments illustrate the effect of the diversity loss in modulating the tradcoff between constraint satisfaction and distributional similarity. They also serve as experimental ablation study to confirm the effect. of our two NDGM innovations: the multi-class discriminator (versus discriminator overloading) and the diversity-based loss.
We visually showcase distributions generated by GAN-MDD and GAN-DO under different weights of in Figures 6 and 7. As expected, higher diversity yields better distributional similarity but poorer constraint.

---

satisfaction in both models across both problems. GAN-MDD generates very neat distributions with higher diversity, creating the most visually similar sampled distribution compared to the ground-truth distribution. In contrast, GAN-DO never truly captures the ground truth distribution, despite overcoming its more egregious distribution collapse issues.
(a) 7 = 0.0
(b) 7 0.1
(c) 7 = 0.2
(d) 7 =
(e) = 0.1
(f) 7 = 0.2
Figure Generated distributions from by our GAN-MDD for Problem 1 (a-c) and Problem (d-f), demonstrating effect of diversity loss weight, A. Diversity weight elegantly modulates the tradeoff between distributional similarity and constraint satisfaction.
(a) 7 = 0.0
(b) 7 = 0.1
(c) 7 = 0.2
(d) 7 = 0.0
(e) 7 = 0.1
(f) 7 = 0.2
Figure 7: Generated distributions from the bascline GAN-DO for Problem (a-c) and Problem (d-f), demonstrating effect of diversity loss weight, A. Although adding diversity loss eases mode collapse issues, GAN-DO with large diversity weight struggles with high invalidity.
Problem 1
Problem
2.00
M
4.0
AA
1.75
3.5
DO-0.2
1.50
3.0
Invalidity Rate (%)
Invalidity Rate (%)
1.25
MDD-0.2
1.00
DO-0.2
0.75
DO-0.1
1.5
DO-0.1
0.50
MDD-0.2
1.0
MDD-0.0
DO-0. 0.0
0.25
MDD-0.1
MDD-0.0
0.5
MDD-0.1
DO-0.0
0.00
0.0
0.8
0.7
0.6
0.5
0.4
0.850 0.825 0.800 0.775 0.750 0.725 0.700 Score
F1 Score
Figure 8: Comparison of GAN-MDD and GAN-DO invalidity rates (4) and F1 scores (t) for a variety of diversity weights on Problem (left) and Problem 2 (right). Gray markers indicate scores for other benchmarked models. Mean scores over six instantiations are plotted. Scores closer the bottom left are more optimal. Triangular markers indicate that the score lies off the plot in the indicated direction. Exact scores inchuded in Table 9.
10

---

We present summary of numerical scores in Figure 8, with full scores in Table 9. These scores illustrate GAN-MDD's widespread dominance across variety of diversity weights. Across all weights tested, GAN- MDD achieves significantly better distributional similarity scores than GAN-DO. Furthermore, for every nonzero diversity weight, GAN-MDD achieves significantly better constraint satisfaction.
4.4 Negative-Datt Data Generative Models Excel in Engineering Tasks
Generative models are commonly used to tackle engineering problems with constraints (Oh et al., 2019; Nie et al., 2021), but are often criticized for their inability to satisfy them (Woldseth et al., 2022; Regenwetter et al., 2023). To assess how NDGMs fare in real engineering problems, we have curated a benchmark of dozen diverse engineering tasks, which are discussed in detail Appendix F. These problems span numerous engincering disciplines including assorted industrial design tasks (compression spring, gearbox, heat exchanger, pressure vessel), structural and material design tasks (Ashby chart, cantilever beam, reinforced concrete, truss, welded beam), and several complex high-level design problems: Ship hulls with hydrodynamic constraints; bike frames with loading requirements; automobile chassis with performance requirements in impact testing. A variety of constraints are applied, including engineering standards from authoritative bodies like the American Concrete Institute (ACT), the American Society of Mechanical Engineers (ASME), and the European Enhanced Vehicle-Safety Committee (EEVC). As a select example, we visualize several positive and negative dalapoints from the FRAMED bike frame datasel (Regenwetter al., 2022b) in Figure 9.
N
(a) Constraint-satisllinng (positive) bike frames
(b) Constraint-violating (negative) bike frames
Figure Example visualization for the bike frame engincering problem. The FRAMED bike frame dataset describes 37-dimensional 3D parametric CAD problem that defines dozens of geometric constraints (disconnected components, negative tube thickness, etc.). Vanilla generative models generate variety of constraint-violating bikes, while NDGMs like GAN-MDD reliably generate valid bikes.
We evaluate vanilla GAN, the state-of-the-art GAN-DO, and our GAN-MDD. We measure invalidity rate, F1 score, and DPP diversity scores over seven training runs for each problem. For each problem and metric, we evaluate whether models are significantly (p < 0.05) superior to one another using one-sided 2-sample t-tests. A summary of benchmarking results is presented in Table 2, with detailed results in Tables 3, 4, and 5. We summarize key results as follows:
Invalidity Score: GAN-DO and GAN-MDD are the highest performers in invalidity score, sig- nificantly outperforming vanilla GAN in almost all problems. GAN-DO is more often the top performer, but sometimes falls short of GAN-MDD.
F1 Score: GAN-MDD is the winner in distributional similarity by a small margin over the GAN. However, the stato-of-tho-art GAN-DO falls significantly short of both the GAN and GAN-MDD in many problems.
Diversity Score: GAN-MDD the overwhelming winner in diversity score, significantly outper- forming both the GAN and GAN-DO in the majority of problems, and achieving the highest mean score in all but one problem.
In summary, our GAN-MDD is ablc to achieve significantly better sample validity and diversity than vanilla models while maintaining generally comparable distributional similarity scores. In contrast, the state-of-the art GAN-DO is able to achieve marginally higher sample validity, but does so at the expense of distributional similarity and sample diversity. Unless constraint satisfaction is favored above all clse, we foresoe that GAN-MDD will offer more optimal tradeoff of performance considerations for most in most problems.
11

---

Table 2: Invalidity rates, Fl scores, and diversity scores for 12 engineering datasets. We benchmark vanilla GAN, GAN with discriminator overloading (DO), and GAN with multi-class discriminator and diversity loss (MDD). Mean scores over seven training instances are presented, with best scores for each metric on cach problem underlined. Lower invalidity rates and diversity scores are better, while higher F1 scores are better. For each dataset and metric, we compare models in a pairwise fashion and identify when a model outperforms a competitor to a statistically significant degree (details in Tables 3-5). These 'pairwise wins' are tallied across all problems and presented in the last row.
Invalidity (%)
(+)
F1 Score (1)
Diversity Score (+)
GAN
DO
MDD (ours)
GAN
DO (sota)
MDD (ours)
DO
MDD
Dataset
GAN
(base)
(sota)
(base)
(sota)
(ours)
Ashby Chart
1.54
1.12
0.63
0.959
0.960
0.922
10.30
10.26
10.19
Bike Frame
4.77
2.85
5.46
0.681
0.684
0.731
3.17
3.06
1.28
Cantilever Beam
4.16
2.51
3.00
0.845
0.818
0.875
1.25
1.94
0.94
Car Impact
4.78
1.92
3.84
0.883
0.844
0.893
0.54
1.12
0.36
Comp. Spring
1.49
0.77
1.06
0.960
0.956
0.962
10.84
10.83
10.84
Concrete Beam
1.03
0.16
1.14
0.957
0.954
0.956
10.33
10.31
10.29
Gearbox
0.33
0.02
0.07
0.899
0.872
0.891
5.32
5.89
4.56
Heat Exchanger
5.35
3.77
3.68
0.876
0.869
0.867
4.82
5.12
4.64
Pressure Vessel
1.30
0.11
0.95
0.947
0.944
0.932
9.42
9.39
9.35
Ship Hull
93.97
93.54
92.05
0.769
0.708
0.713
11.06
11.09
11.05
Three-Bas Bar Truss
0.32
0.00
0.34
0.938
0.948
0.957
14.32
14.28
14.19
Welded Beam
1.74
0.67
0.53
0.955
0.936
0.850
9.44
9.37
9.30
Pairwise Wins*
14
2
17
*Number of models statistically significantly outperformed in pairwise comparisons. Details in Tables 3-5.
Table 3: Means and standard deviations of invalidity rate for engineering datasets over seven tests. Lower
scores are better. Problems are sorted by GAN's invalidity rate. A model's symbol (V/x/t) is shown if statistically significant in a pairwise comparison. Best mean scores are bolded.
GAN (baseline)
DO
MDD (ours)
DO +
MDD / GAN X
MDD /
(sota)
GAN X
DO +
Three-Bae Bar Truss
0.32.10.11%
0.00+10.00%
0.34.10.9%
Gearbox
0.39%
0.02:10.2%
0.07:10.05%
Concrete Beam
1.03++0.97%
0.16:10.14%
1.1440.46%
Pressure Vessel
1.30+0.32%%
0.11+0.07%
0.95+0.29%
Comp. Spring
1.49+1.00%
0.77+0.67%
1.06+0.58%
Ashby Chart
1.5410.97%
1.12=0.39%
0.63-10.17%
Welded Beam
1.741..89%%
0.67-0.39%
0.53+0..14%
Cantilever Beam
4.16++0.8%
2.51+0.79%
3.00+0.62%
Bike Frame
4.77+11211%
2.85+0.63%
5.46+3.39%
Car Impact
4.78%%
1.92+0.48%
0.748%
Heat Exchanger
5.35+1.00%
3.77=0.70%
3.68+0.82%
Ship Hull
93.97+0.64%
93.5440.97%
92.05+2.31%%
12

---

Table 4:
Means and standard deviations of F1 score for engineering datasets over seven tests. Higher scores are better. Problems are sorted by GAN's F1 score. A model's symbol (V/x/t) is shown statistically significant pairwise comparison. Best mcan scorcs are bolded.
GAN
DO
MDD
DO
MDD / GAN
MDD /
(baseline)
(sota)
(ours)
GAN X
DO t
Comp. Spring
0.960+0.003
0.956+0.004
0.962:10.005
X
Ashby Chart
0.959=10.007
0.960=10.005
0.922:10.014
X
Concrete Beam
0.957=0.002
0.954
0.956:10.005
Welded Beam
0.955=1:0006
0.936+0.013
0.850+0.025
X
Three-Bat Truss
0.938110.022
0.941:10.012
0.957
Pressure Vessel
0.917
0.944110.013
0.902
Gearbox
0.899=100033
0.872#0.021
0.891
X
Car Impact
0.88310.0017
0.84440.041
0.000
X
Heat Exchanger
0.036
0.869
0.867
Cantilever Beam
0.845+0.038
0.818
0.875
Ship Hull
0.76910.082
0.708
0.713
Bike Frame
0.681+
0.684
0.701
Table 5: Means and standard deviations of diversity score for engineering datasets tests. Lower scores are better. Problems are sorted by GAN's diversity score. A model's symbol (V/x/t) is shown if statistically significant in a pairwise comparison. Best mean scores are bolded.
GAN
DO
MDD (ours)
DO
MDD / GAN
MDD /
(baseline)
(sota)
GAN X
DO
Car Impact
0.54
1.12+:0.39
0.36+10.06
X
Cantilever Beam
1.25+0038
1.94+0.55
0.94
X
Bike Frame
3.17+:0.40
3.06++0.2
1.28+0.24
Heat Exchanger
4.82+:041
5.1240.21
4.64+00.55
Gearbox
5.32++0.0
5.89+0.26
4.56+0.33
X
Pressure Vessel
9.42++0.0
9.39+0.02
9.35+10.02
T
Welded Beam
9.44+10.03
9.37+00.2
9.30+0.08
Concrete Beam
10.33+0.00
10.31+0.03
10.29+0.05
Ashby Chart
10.30+0.03
10.26+0.02
10.19:0.03
V
Comp. Spring
10.84+0.07
10.83
10.84+0.08
Ship Hull
11.06
11.09:10.85
11.05++0.2
Three- Bar Truss
14.32+0.07
14.28+0.088
14.1940.03
13

---

4.5 Examining Negative Data Quality in High- Dimensional Constrained Engineering Problems
Having tested variety of tabular engineering problems, next consider whether proposed methods can translate to higher-dimensional domains such images. We examine common engineering design problem known as topology optimization (TO), which seeks to optimally distribute material in spatial domain to achieve a certain objective (often minimizing mechanical compliance) (Sigmund & Maute, 2013). Simply put, TO often used to create structures with high rigidity and low weight. The use of generative models for TO is very popular (Shin et al., 2023), but existing methods have been criticized for significant shortcomings (Woldseth et al., 2022) related to constraint satisfaction, such generated topologies not being fully connected. Disconnected topologies tend to be highly sub-optimal and are impractical to fabricate.
(a) Positive (connected) topologies (b) Procedurally-geneee generated negatives
(c) Enjection-samplee negatives
Figure 10: Topology optimization is a challenging structural engineering problem that searches for optimal placements material. Valid (positive) structural topologies are completely continuous (left). Invalid (negative) topologies have floating material and can be procedurally--eeneaate by artificially adding small floating patches of material (middle). They can also be collected through rejection-samphee of topologies created by generative model (right). Constraint violation is annotated with red circles.
We train NDGMs using disconnected topologies as negative data, using the classification guidance dataset from Mazé Ahmed (2023). The positive data is comprised of optimized, spatially continuous structures, while the negative data is largely comprised of procedurallly-genrratee negatives with artificially-addee floating components. For comparison, we create an alternative negative dataset by replacing procedurally-generaas negalives in the dalaset with rejection-sampled topologies generated by vanilla. GAN trained on the positive data. A simple continuity check flags any discontinuous topologies to add to the rejection-sampler negative datasct. A few topologics from cach dataset are visualized in Figure 10. We hypothesize that these rejeclion-sampled negalives are 'harder" negatives (closer to the positive distribution) and are hence more informative than the proceduralll-geneeaa negatives.
Table Means and standard deviations of performance metrics for the topology optimization problem over six lests with pairwise statistical significance comparisons. Lower scores are better. A model's symbol (V/X/t) is shown if statistically significant in a pairwise comparison. Best mean scores are bolded.
Negative Dataset
GAN
DO (sota)
MDD (ours)
DO MDD
MDD /
Metric
(baseline)
GAN X
GAN X
DO f
Invalidity Rate (%)
Procedural
32.8+2.5
28.248.7
17.6:15.7
Rejection
32.8+2..5
15.5+1.6
14.8:10.8
Violation
Procedural
2.01+0.29
3.03+-0.67
2.78++0.45
X
X
Mag. (10
Rejection
2.0110.29
2.23
1.144:0.12
/
Diversity
Procodural
14.49:1.06
14.44+0.14
14.36+0.11
/
Score
Rejection
14.49:1.006
14.5240.13
14.38+0..10
/
14

---

(a) GAN using negatives
(b) GAN-DO using
(c) GAN-DO using rejected negalives
(d) GAN-MDD using procedural negatives
(e) GAN-MDD using rejected negatives
procedural negalives
Figure 11: Visualization of topologies generated by GAN trained only positive topologies and CAN-DO/GAN-MDD models additionally trained on procedurally-generaaee negatives or rejection-sampled negatives. Valid samples are marked with a blue check mark while invalid samples are marked with a red X. Additional samples are visualized in Figures 15 through 19.
We run five types experiments: GAN models trained on only positive data, GAN-DO and GAN-MDD trained on procodurally-generan data, and GAN-DO and GAN-MDD trained on rejection-sampled data. Some generated samples visualized in Figure 11. We evaluate six instantiations of each experiment and evaluate pairwise comparisons using 2-sample t-tests with p 0.05 significance. In evaluating models, we mcasure the proportion of generated topologies with disconnected components (invalidity rate), as well as the average fraction of image pixels disconnected from the largest continuous structure in each generated topology (violation magnitude). To identify mode collapse, we also measure DPP the pixel space (diversity score). Numerical scores are presented in Table 6. These results suggest several noteworthy conclusions:
For both types of negative data, GAN-MDD outperforms GAN-DO in mean scores in every metric. proceduralll--geerrate negatives
This difference is statistically significant for invalidity rate and for violation magnitude score and diversity score rejection-saamled negatives.
2. GAN-DO significantly outperforms the vanilla GAN in invalidity rate only for rejection-sampled negatives, suggesting that it more sensitive to negative data quality than our GAN-MDD. As seen in Figure 12, GAN-DO can suffer from severe mode collapse. GAN-DO scores are also the most variable, indicating unpredictable training, stability, and convergence.
All NDGMs trained on rejection-samplet negatives achieve better mean scores in every metric than any NDGM trained can be even more impact/ul than the choice of NDGM lype. It also highlights the potency of rejection sampling as a negative data sampling strategy when a black-box constraint check is available.
senddurrally-enerrate negatives. This indicates that negative data quality
In summary, our benchmarking on the topology optimization problem illustrates the potency GAN-MDD on high-dimensional image-based problem but also illustrates the importance of negative data quality in NDGM performance. Best practices generate high-quality negative data remain an open research area. Regardless, GAN-MDD the highest overall performer either lype of negative data, particularly in constraint satisfaction rate.
15

---

Figure 12: A batch of 32 random topologies generated by select GAN-DO instantiation trained rejection-sampled negalives. Samples fall under just a handful of data modes, indicating egregious mode collapse. A similar grouping is shown for GAN-DO model trained on sereedurally-generatee negatives in Figure 20.
5 Discussion & Conclusion
Adding pairwise density ratio estimation and diversity to NDGMs. We presented a NDGM formulation, GAN-MDD, which estimates individual density ratios, rather than conflating fakes and negatives as done in the current SOTA. Our model also incorporates a Determinantal Point Process-based diversity loss. These innovations empower GAN-MDD to achieve an optimal tradeoff between constraint satisfaction and distributional similarity, allowing it to outperform baseline models and existing NDGMs alike, across a variety of problems. In specific benchmarks, GAN-MDD manages generate 1/6 as many constraint-volaating samples vanilla. model using only 1/8 as much data.
GANs versus diffusion models using negative data. Despite the growing popularity of diffusion models, GANs remain state of the art in many engineering design problems. On the challenging 2D problems, we find that: 1) vanilla DDPMs struggle to learn constraints and 2) DDPMs using guidance based on negative data can only achieve good constraint satisfaction at the expense of distributional similarity. This preliminary study indicates that DDPMs struggle to achieve as optimal of a tradeoff between constraint satisfaction and distributional similarity as our specialized adversarial NDGM model, least in some problems. We look forward to future rescarch which advances the capabilities of negative data diffusion models.
NDGMs are underutilized. We believe NDGMs are underutilized in engineering design. This assertion is substantiated by several observations: 1) The widespread use of vanilla models in engineering design despite their limitations (Regenwetter et al., 2022a). 2) The relatively low cost of collecting negative data versus positive data in many engincering contexts. 3) The overwhelming dominance NDGMs over vanilla models in our engineering benchmarks. 4) The data-efficiency improvements we demonstrated using negative data.
Generating high-quality negative data. Selecting strategies to generate negative data is an important research question. In the final case study topology optimization, rejection-samplnng resulted in "stronger" negative data than the procedural generation method. Tt also required access to an oracle (constraint. evaluator), which may be unavailable or prohibitively expensive in some applications. However, there are not always cheap, viable procedural generation approaches for negative data either. Effective negative data generation remains largely -roblem--ependent and the relative quality of negative data generation approaches is not necessarily straightforward. We anticipate that domain-agnostic methods to generate high-quality negative data could pair well with NDGMs and expand their impact.
16

---

Leveraging negative data for classifier-based rejection sampling. Negative data can be used to train supervised binary classification models to predict constraint satisfaction. These classifiers can then be applied during generative model inference to iteratively reject and regenerate any samples projected to violate constraints until a sufficiently large sample set is attained. If the generative model has a very low validity rate, this can increase the inference cost by orders of magnitude and add significant stochasticity to the generation process. To generate single ship hull, for example, we estimate that a vamilla generative model would take projected 33 times longer, 222 times longer in a reasonable bad-case scenario, compared to NDGM. Although expensive, this rejection sampling approach yields extremely strong validity rates, surpassing even the most accurate NDGMs. Though in many senses apples-to-oranger juxtaposition, comparing NDGMs with rejection sampling offers hints about the theoretical limits of NDGM performance. Classifier-based rejection sampling using negalive data is discussed in more detail in Appendix D.
Limitations. As we demonstrate, NDGMs are sensitive to the quality of their negative training data. Although negative data often cheaper than positive data in engineering design problems, generating high-quality negative data may be challenging in some domains. In other domains, sourcing any kind of negative data may be impossible. In domains where high-quality negative data unavailable, NDGMs will naturally be impractical.
6 Conclusion
In this paper, we presented new negative-data generative model (NDGM), GAN-MDD, that innovates over previous methods by learning individual pairwise density ratios and avoiding mode-collapse using diversity-basee loss. In extensive benchmarks across constructed tests and dozen real engineering problems, we demonstrated that GAN-MDD outperforms 10 other formulations. Morcover, it is data-efficient, generating 1/6 as many invalid samples as a vanilla model using 1/8 as much data. Finally, we showed that GAN-MDD can excel in challenging high-dimensional problems. GAN-MDD> data efficiency and optimal balance between distributional similarity and constraint satisfaction make significantly more practical than existing generative models for engineering and design tasks. Thus, we anticipate and advocate for the more widespread use of NDGMs in data-driven engineering design and other constrained generative modeling domains.
17

---

References
Martin Arjovsky, Soumith Chintala, and Léon Bottou. Wasserstein generative adversarial networks. In International conference on machine learning, ICML'17, pp. 214 223. PMLR, JMLR.org, 2017.
Siddarth Asokan and Chandra Seelamantula. Teaching gan what not to learn. Advances in Neural Information Processing Systems, 33:3964 3975, 2020.
Noah J Bagazinski and Facz Ahmod. Ship-d: Ship hull datasct for design optimization using machine learning. arXiv preprint arXiv:230500827,, 2023.
Mohammad Mahdi Behzadi and Horea T. llis. GANTL: Toward Practical and Real-Time Topology Optimization With Conditional Generative Adversarial Networks and Transfer Learning. Journal of Mechanical Design, 144(2), 12 2021. ISSN 1050-0472. doi: 10.1115/1.4052755 URL https://doi.onn 10. 09.1115/1.40575555 021711.
Martin Philip Bendsøe and Noboru Kikuchi. Generating optimal topologies in structural design using a homogenization method. Computer Methods Applied Mechanics and Engineering, 71(2):197 224, 11 1988. ISSN 00457825. doi: 10.1016/0045- (55000005577255889900666 URL https://linkinghub. elsevier.com rettr/aree//110044577599995
Camille Bilodean, Wengong Jin, Tommi Jaakkola, Regina Barzilay, and Klavs Jensen. Generative models for molecular discovery: Recent advances and challenges. Wiley Interdisciplinary Reviews: Computational Molecular Science, 12(5)xe1608, 2022.
Ramin Bostanabad, Yichi Zhang, Xiaolin Li, Tucker Kearney, L Catherine Brinson, Daniel W Apley, Wing Kam Liu, and Wei Chen. Computational microstructure characterization and reconstruction: Review of the state-offthe-art tochniques. Progress in Materials Science, 95:1-41, 2018.
Andrew Brock, Theodore Lim, James Millar Ritchie, and Nick Weston. Context-aware content generation for virtual environments. In International Design Engineering Technical Conferences and Computers and Information in Engincering Conference, volume 50084, pp. VO1BT02A045. American Society Mechanical Engineers, 2016.
Yuri Burda, Roger Grosse, and Ruslan Salakhutdinov. Importance weighted autoencoders. arXiv preprint 1XXiv:1509.00511 2015.
Ruijin Cang, Hechao Li, Hope Yao, Yang Jiao, and Yi Ren. Improving direct physical properties prediction of heterogeneous materials from imaging data via convolutional neural network and morphology-awes generative model. Computational Materials Science, 150:212-221, 2018.
George Casella and Roger L. Berger. Statistical Inference. Duxbury, Pacific Grove, CA, 2002.
Michael Chang, Alyssa I. Dayan, Franziska Meier, Thomas L Griffiths, Sergey Levine, and Amy Zhang. Neural constraint satisfaction: Hierarchical abstraction for combinatorial generalization in object rearrangement. arXiv preprint arXiv:2303.1 11373, 2023.
Hongrui Chen and Xingchen Liu. Geometry enhanced generative adversarial networks for random heteroge- neous material representation. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC-21, Virtual, Online, Aug 2021. ASME.
Qiuyi Chen, Jun Wang, Phillip Pope, Wei Chen, and Mark Fuge. Inverse design of two-dimensional airfoils using conditional generative models and surrogate log-likelihoods, Journal of Mechanical Design, 144(2): 021712, 2022.
Wei Chen and Faez Ahmed. Mo-padgan: Reparameterizing engineering designs for augmented multi-objective oplimization. Applied Soft. Computing, 113:107909, 2021a.
Wei Chen and Faez Ahmed, Padgan: Learning to generate high-quality novel designs. Journal of Mechanical Design, 143(3):031703, 2021b.
18

---

Wei Chen and Mark Fuge. Béziergan: Automatic generation of smooth curves from interpretable low- dimensional parameters. arXiv preprint arXiv:1808.0888 808.08871 2018.
Wei Chen and Mark Fuge. Syuthesizing designs with interpart dependencies using hierarchical generative adversarial networks. Journal of Mechanical Design, 141(11):11140; 2019.
Wei Chen, Kevin Chiu, and Mark Fuge. Aerodynamic design optimization and shape exploration using generative adversarial networks. In AIAA Scitech 2019 Forum, pp. 2351, 2019.
Yu Cheng, Yongshun Gong, Yuansheng Liu, Bosheng Song, and Quan Zou. Molecular design in drug discovery: a comprehensive review of deep generative models. Briefings in bioinformatics, 22(6))bbab344, 2021.
Kristy Choi, Chenlin Meng, Yang Song, and Stefano Ermon. Density ratio estimation via infinitesimal classification. In International Conference Artificial Intelligence and Statistics, pp. 2552-2573 PMLR, 2022.
Matthew Dering, James Cunningham, Raj Desai, Michael A Yukish, Timothy W Simpson, and Conrad S Tucker. A physics- physics-based virtual environment for enhancing the quality of deep generative designs. In International Design Engineering Technical Conferences and Computers and Information in Engincering Conference, volume 51753, pp. V02AT03A015. American Society of Mechanical Engineers, 2018.
Shrinath Deshpande and Anurag Purwar. Computational creativity via assisted variational synthesis of mechanisms using deep generative models. Journal of Mechanical Design, 141(12), 2019.
Shrinath Deshpande and Anurag Purwar. An Image-Based Approach to Variational Path Synthesis of Linkages. Journal Computing and Information Science in Engineering, 21(2), 10 2020. ISSN 1530-9827. doi:
URL https://dol httpe::/ddooooog/1////
021005.
Mohamed Elfeki, Camille Couprie, Morgane Riviere, and Mohamed Elhoseiny. Gdpp: Learning diverse
generations using determinantal point processes. In International conference on machine learning, pp. 1774-1783. PMLR, 2019.
Amir Hossein Gandomi and Xin-She Yang. Benchmark problems in structural optimization. In Computational optimization, methods and algorithms, pp. 259-281. Springer, 2011.
Amir Hossein Gandomi, Xin-She Yang, and Amir Hossein Alavi. Mixed variable structural optimization using firefly algorithm. Computers & Structures, 89(23-24):2325 2336, 2011.
Giorgio Giannone and Faez Ahmed. Diffusing the optimal topology: A generative optimization approach. International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume 87301, pp. V03AT03A012. American Society of Mechanical Engineers, 2023.
Giorgio Giannone, Akash Srivastava, Ole Winther, and Facz Ahmed. Aligning optimization trajectorics with diffusion models for constrained design generation. Advances in Neural Information Processing Systems, 36, 2024.
Tan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Advances in neural information processing systems, NIPS'14, pp. 2672 2680, Cambridge, MA, USA, 2014. MIT Press.
Michacl Gutmann and Aapo Hyvärinen. Noiso-contrastive estimation: A new estimation principle for unnormalized statistical models. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 297-304. JMLR Workshop and Conference Proceedings, 2010.
Jessica B Hamrick, Kelsey R Allen, Victor Bapst, Tina Zhu, Kevin R McKee, Joshua B Tenenbaum, and Peter W Battaglia. Relational inductive bias for physical construction in humans and machines. arXiv preprint 1XXi:1806600103 2018
19

---

Amin Heyrani Nobari, Wei (Wayne) Chen, and Facz Ahmed. RANGE-GAN:
Design Synthesis Under Constraints Using Conditional Generative Adversarial Networks. Journal of Mechanical Design, pp. 1-16, 09 2021. ISSN 1050-0472. doi: 10.1115/L4052442. URL https://doi.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Tn H. Larochelle, M. Ranzato, R. Hadscll, M.F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6840 6851. Curran Associales, Inc., 2020. URL https::/proceeding.. meeriiss..nnn.......... Paderr.d........................
neurips.
Fergus Imrie, Anthony R Bradley, Mihaela van der Schaar, and Charlotte M Deane. Deep generative models for 3d linker design. Journal of chemical information and modeling, 60(4):1983-199)) 2020.
Cole Jetton, Matthew Campbell, and Christopher Hoyle. Constraining the Feasible Design Space in Bayesian Optimization With User Foodback. Journal of Mechanical Design, 146(4))041703, 11 2023. ISSN 1050-0472. doi: 10.1115/1.4063008 URL /ttps://doi..oo 11691........9.6.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint
arXiv:1412.696 6980, 2014.
Diederik P Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint 13X1v1311..614, 2013.
Alex Kulesza, Ben Taskar, et al. Determinantal point processes for machine learning. Foundations and Trends in Machine Learning, ((2-3):123 286, 2012.
Jin- HoonWoong Lee, Nam Hoon Goo, Woon Bae Park, Myungho Pyo, and Kee-Sun Sohn. Virtual microstructure design for steels using generative adversarial networks. Engineering Reports, 311)e12227, 2021.
Baotong Li, Congjia Huang, Xin Li, Shuai Zheng, and Jun Hong. Non-iterative structural topology optimization using deep learning, Computer-Aided Design, 115:172 180, 2019. ISSN 0010-4485. doi:
URL
pis//80010448559300888.
Runze Li, Yufei Zhang, and Haixin Chen. Learning the aerodynamic design of supercritical airfoils through deep reinforcement learning. AIAA Journal, pp. 1 14, 2021.
Xiang Li, Shaowu Ning, Zhanli Liu, Ziming Yan, Chengcheng Luo, and Zhuo Zhuang. Designing phononic crystal with anticipated band gap through deep learning based data-driven method. Computer Methods in Applied Mechanics and Engineering, 361:112737, 2020.
Siyan Liu, Zhi Zhong, Ali Takbiri- Borbbri--ooujen,, Mohammad Kazemi, Qinwen Fu, and Yuhao Yang. case study homogeneous and heterogeneous reservoir porous media reconstruction by using generative adversarial networks. Energy Procedia, 668:6164-616, 2019.
Zhaocheng Liu, Lakshmi Raju, Dayu Zhu, and Wenshan Cai. hybrid strategy for the discovery and design of photonic structures. IEEE Journal on Emerging and Selected Topics in Circuits and Systems, 10(1): 126 135, 2020.
F. Mazé and F. Ahmed. Diffusion models beat gans on topology optimization. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), Washington, DC, 2023. URL 09591.
ortp:/...r/bbbbsbb arrg//bbs/208... 2208.
Shakir Mohamed and Balaji Lakshminarayanan. Learning in implicit generative models. arXiv preprint arXiv:1610.03483, 2016.
Lukas Mosser, Olivier Dubrule, and Martin J Blunt. Reconstruction of three-dimensional porous media using generative adversarial neural networks. Physical Review E, 9644)04433,, 2017.
20

---

Ramaravind K Mothilal, Amit Sharma, and Chenhao Tan. Explaining machine learning classifiers through diverse counterfactual explanations. In Proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 607 617, 2020.
Zhenguo Nie, Tong Lin, Haoliang Jiang, and Levent Burak Kara. Topologygan: Topology optimization using generative adversarial networks based on physical fields over the initial domain. Journal of Mechanical Design, 143(3):031715, 2021.
Amin Heyrani Nobari, Wei Chen, and Faez Ahmed. Podgan: A continuous conditional diverse generative adversarial network for inverse design. arXiv preprint arXiv:2106.03620, 2021.
Amin Heyrani Nobari, Wei Chen, and Faez Ahmed. Range-constrained generative adversarial network: Design synthesis under constraints using conditional generative adversarial networks. Journal of Mechanical Design, 144(2), 2022.
Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 29, 2016.
Sangeun Oh, Yongsu Jung, Ikjin Lec, and Namwoo Kang. Design automation by integrating generative adversarial networks and lopology optimization. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume 51753, pp. V02AT03A008, American Society of Mechanical Engineers, 2018.
Sangeun Oh, Yongsu Jung, Scongsin Kim, Tkjin Lee, and Namwoo Kang. Deep generative design: Integration of topology optimization and generative models. Journal of Mechanical Design, 141(11), 2019.
Wamiq Para, Shariq Bhat, Paul Guerrero, Tom Kelly, Niloy Mitra, Leonidas Guibas, and Peter Wonka. Sketchgen: Generating constrained cad sketches. Advances in Neural Information Processing Systems, 34: 5077-5088, 2021.
Ben Poole, Sherjil Ozair, Aaron Van Den Oord, Alex Alemi, and George Tucker. On variational bounds of mutual information. In International Conference on Machine Learning, pp. 5171-5180. PMLRR 2019.
Sharad Rawat and MH Herman Shen. Application of adversarial networks for 3d structural topology optimization. Technical report, SAE Technical Paper, 2019.
Lyle Regenwetter and Faez Ahmed. Design target achievement index: A differentiable metric to enhance deep generative models in multi-objective inverse design. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume 86236, pp. V03BT03A046.
American Society of Mechanical Engineers, 2022.
Lyle Regenwetter, Brent Curry, and Faez Ahmed. BIKED: A dataset and machine learning benchmarks for data-driven bicycle design. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC-21, Virtual, Online, Aug 2021. ASME
Lyle Regenwetter, Amin Heyrani Nobari, and Facz Ahmod. Deep generative models in engincoring design: A review. Journal of Mechanical Design, 144(7):071704, 2022a.
Lyle Regenwetter, Colin Weaver, and Faez Ahmed. Framed: Data-driven structural performance analysis of coummnity-designee bicycle frames, 2022b.
Lyle Regenwetter, Akash Srivastava, Dan Gulfreund, and Faez Ahmed. Beyond statistical similarity: Rethinking metrics for deep generative models in engineering design. arXiv preprint arXiv:2302.02913, 2023.
Benjamin Rhodes, Kai Xu, and Michael U Gutmann. Telescoping density-ratio estimation. Advances in neural information processing systems, 33:4905 4916, 2020.
Stuart Russell. Artificial intelligence a modern approach. Pearson Education, Inc., 2010.
21

---

Mehdi SM Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. Assessing generative models via precision and recall. Advances in neural information processing systems, 31, 2018.
Ari Seff, Wenda Zhou, Nick Richardson, and Ryan P Adains Vitruvion: A generative model of parametric cad sketches. arXiv preprint 1ttii::2009.111244 2021.
Shashank Sharma and Amurag Purwar. Path synthesis of defect-free spatial 5-ss mechanisms using machine learning. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume 83990, pp. V010T10A034. American Society of Mechanical Engineers, 2020.
Conner Sharpe and Carolyn Conner Seepersad. Topology design with conditional gencrative adversarial networks. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, volume 59186, pp. V02AT03A062. American Society of Mechanical Engineers, 2019.
Seungyeon Shin, Dongju Shin, and Namwoo Kang. Topology optimization via machine learning and deep lcarning: A review. Journal of Computational Design and Engineering, 10(4):1736-1766, 2023.
Dule Shu, James Cunningham, Gary Stump, Simon W Miller, Michael A Yukish, Timothy W Simpson, and Conrad S Tucker. 3d design using generative adversarial networks and physics-based validation. Journal of Mechanical Design, 147(7):07 2020.
Ole Sigmund and Kurt Maute. Topology optimization approaches: A comparative review. Structural and Multidisciplinary Optimization, 18(6):1031-1055. 2013. ISSN 1615-147X. doi: 10.1007/s00158. 00007//000588001-00777 6.
Abhishek Sinha, Kumar Ayush, Jiaming Song, Burak Uzkent, Hongxia Jin, and Stefano Ermon. Negative data augmentation. arXiv preprint arXiv:2102.05113, 2021.
Kevin Smith, Lingjie Mei, Shunyu Yao, Jiajun Wu, Elizabeth Spelke, Josh Tenenbaum, and Tomer Ullman. Modeling expectation violation in intuitive physics with coarse probabilistic object representations. Advances in neural information processing systems, 32, 2019.
Casper Kaae Sonderby, Tapani Raiko, Lars Maalee, Soren Kaae Sonderby, and Ole Winther. Ladder variational autoencoders. In Advances in neural information processing systems, pp. 3738 3746, 2016.
Akash Srivastava, Lazar Valkov, Chris Russell, Michael U. Gutmann, and Charles Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 30, pp. 3308 3318. Curran Associates, Inc., 2017. URL https: wtwp:ecedddnngnneuuss........ pdf.
Akash Srivastava, Seungwook Han, Kai Xu, Benjamin Rhodes, and Michael U. Gutmann. Estimating the density ratio between distributions with high discrepancy using multinomial logistic regression. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL https:// http://oppneeee..eeeee/ net/
forum??dd=jMBn22ZBBB
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning.
Cambridge University Press, 2012a.
Masashi Sugiyama, Taiji Suzuki, and Takafumi Kanamori. Density ratio estimation in machine learning. Cambridge University Press, 2012b.
Ren Kai Tan, Nevin T. Zhang, and Wenjing Ye. A deep learning based method for the design of microstructural materials. Structural and Multidisciplinary Optimization, 61(4):1417 1438, 2020.
Yingheng Tang, Keisuke Kojima, Toshiaki Skike,Alino, Ye Wang, Pengxiang Wu, Mohammad Tahersima, Devesh Jha, Kieran Parsons, and Minghao Qi. Generative deep learning model for a multi-level nano-optic broadband power splitter. In 2020 Optical Fiber Communications Conference and Exhibition (OFC), pp. 1-3. IEFE, 2020.
22

---

Sofia Valdez, Carolyn Seepersad, and Sandilya Kambampati. A framework for interactive structural design exploration. In International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, IDETC-21, Virtual, Online, Aug 2021. ASME.
Jun Wang, Wei Wayne Chen, Daicong Da, Mark Fuge, and Rahul Rai. Th-gan: A conditional generative model for implicit surfaco- based inverse design of cellular structures. Computer Methods in Applied Mechanics and Engineering, 396: 116:00 2022.
Liwei Wang, Yu-Chin Chan, Facz Ahmod, Zhao Liu, Ping Zhu, and Wei Chen. Doep generative modeling for mechanistic-based learning and design of metamaterial systems. Computer Methods in Applied Mechanics and Engineering, 372:113377, 2020.
Eamon Whalen, Azariah Beyene, and Caitlin Mueller, Simjeb: simulated jet engine bracket dataset. Computer Graphics Forum, 40(5):9-17, 2021.
Rebekka V Woldseth, Niels Aage, J Andreas Bærentzen, and Ole Sigmund. On the use of artificial neural networks in topology optimisation. Structural and Multidisciplinayy Optimization, 65(1))2944 2022.
Patricia Wollstadt, Mariusz Bujny, Satchit Ramnath, Jami J Shah, Duane Detwiler, and Stefan Menzel. Carhoods10k: An industry-grade data set for representation learning and design optimization in engineering applications. IEEE Transactions on Evolutionary Computation, 26(6):1221-122) 2022.
Tianju Xue, Thomas J Wallin, Yigit Mengue, Sigrid Adriaenssens, and Maurizio Chiaramonte. Machine learning generative models for automatic design of multi-material 3d printed composite solids. Extreme Mechanics Letters, 41:100992, 2020.
Xin-She Yang and Amir Hossein Gandomi. Bat algorithm: a novel approach for global engineering optimization. Engineering computations, 2965):464-483, 2012.
Zijiang Yang, Xiaolin Li, L Catherine Brinson, Alok N Choudhary, Wei Chen, and Ankit Agrawal. Microstruc- tural materials design via deep adversarial learning methodology. Journal Mechanical Design, 140(11), 2018.
Furre Yilmaz and Brian German. Conditional generative adversarial network framework for airfoil inverse design. In AIAA aviation 2020 forum, pp. 3185, 2020.
Rosen Yu, Cyril Picard, and Faez Ahmed. Fast and accurate bayesian optimization with pre-trained transformers for constrained engincering problems. arXiv preprint arXiv:2404.04995 2024.
Yonggyun Yu, Taeil Hur, Jaeho Jung, and In Gwun Jang. Deep learning for determining a near-optimal topological design without any iteration. Structural and Multidisciplinary Optimization, 59(3)7787-799 2019.
Hui Zhang, Lei Yang, Changjian Li, Bojian Wu, and Wenping Wang. ScafToldgan: Synthesis of scaffold materials based on generative adversarial networks. Computer-Aine Aided Design, 138:103041, 2021. ISSN 0010-4485. doi: aciecce/artcce///////00011111010
202.2001..1031111. URL
Wentai Zhang, Zhangsihao Yang, Haoliang Jiang, Suyash Nigam, Soji Yamakawa, Tomotake Furuhata, Kenji Shimada, and Levent Burak Kara. 3d shape synthesis for conceptual design and optimization using variational autoencoders. Tn International Design Engincering Technical Conferences and Computers and Information in Engineering Conference, volume 59186, pp. V02AT03A017. American Society of Mechanical Engineers, 2019.
23

---

A Additional Visualization and Statistical Testing for Main Experiments A.1 2D Experiments
Generated distributions for all 10 baselines are included in Figures 13 and 14, alongside tabulated numerical scores in Table 7.
aa) Positive Data
(b) Negalive Data (c) CAN-MDD (Ours)
(d) GAN-DO
(e) GAN
ff) Cond. GAN
(g) GAN + CIf.
(h) DDPM
(1) Cond. DDPM
(i) DDPM + Guid.
(k) VAE
(1) Cond. VAE
(m) VAE + Clf.
Figure 13: Visual comparison of our GAN-MDD against all 10 baseline models on Problem 1. Positive data points and samples are shown in blue and negative ones in black.
Problem 1
Problem 2
Inval. (%) (L)
F1 (t)
Diversity (1)
Inval. (%) (J)
F1 (t)
Diversity (L)
VAE
20.51:0..57
0.397:10.017
15.0110.02
14.1910.558
0.751:100007
14.36=0.02
VAE-Cone Re-Cond
26.64:11.05
0.389-10.009
15.05:10.01
14.23:10.44
0.755:10.003
14.36=0.01
VAE-CIE
0.05110.0
0.40810.012
15.0510.04
0.06:10.13
0.345:0.0722
15.42-0.19
DDPM
8.6010065
0.820:10.021
14.86:10.02
14.97:10.68
0.84810.005
13.92-0.04
DDPM-Coud
8.3010.55
0.81410.00
14.94:0.01
11.4410.81
0.858110005
13.98=0.02
DDPM-Guud
0.2611000
0.40410.014
14.07:10.03
5.92-1.54
0.82110.017
14.10=0.02
GAN
8.3243323
0.596 109
15.1740.08
4.30
0.859+0.007
14.09=0.02
GAN-Cond
8.1311102
0.725
15.06.10.03
5.2411.18
0.85010.008
14.05=0.07
GAN-CIF
1.06+1.14
0.609-10.139
15.1440.11
2.22-0.94
0.749 101
14.30=0.20
GAN-DO
0.22510.24
0.493-10.054
15.1140.111
0.52-1.06
0.75410.018
14.34=0.03
GAN-MDD (ours)
0.40++0.13
0.700++0.02
14.99+0.05
0.73+t0.19
0.830+0.000
14.10-0.02
GAN-MDD Rank
3
6
Table 7: Mean and standard deviations for 2D problem scores over six instantiations, Model scores that beat or are beaten by GAN-MDD's scores to p 0.05 significance are colored light blue dark red, respectively. GAN-MDD's mean ranking is included in the last row.
24

---

(a) Positive Data
(b) Negative Data
(c) GAN-MDD (Ours)
(d) GAN-DO
(e) GAN
(f) Cond. GAN
(g) GAN + Clf.
(h) DDPM
(i) Cond. DDPM
(j) DDPM + Guid.
(k) VAE
(I) Cond. VAE
(m) VAE + Clf.
Figure 14: Visual comparison of our GAN-MDD against all 10 baseline models on Problem 2. Positive data points and samples are shown in blue and negative ones in black.
A.2 Dataset Size
Full scores with standard deviations and signifienace testing are included in Table
Table 8: Full scores from the dataset size study presented the main paper, showing both mean scores and standard deviations over the four runs. Scores significantly (p 0.05 using a 2-sample t-test) better than the corresponding GAN trained with the same amount of positive data bolded. Lower is better. The relatively large deviation seen in vanilla models may explain the slight counterintuitive increase in invalidity rate with more data in Problem 1.
(a) Models
(b) Problem 1
(c) Problem
Negative Samples
Positive Samples
Positive Samples 4K
1K
4K
16K
1K
16K
GAN
10.7%
5.3%
11.6% +: 2.9%
11.9% +: 1.6%
6.0%
1.5%
3.2%
1.3%
3.3% = 0.7%
GAN- MDD
2.0% + 2.3%
0.7%
0.2%
1.5%
0.8%
2.3%
0.7%
1.6% = 0.7%
1.9% = 0.7%
BAN-MDD
0.5% + 0.3%
0.5%
0.3%
0.5%
0.2%
0.6%
0.2%
0.7% = 0.3%
0.7% = 0.2%
GAN-MDD
16K
0.5% +: 0.3%
0.7% = 0.5%
0.6%
0.3%
0.5% + 0.1%
0.2% = 0.1%
0.2% 1: 0.1%
25

---

A.3 Diversity Loss Ablation Studies
Full scores for diversity loss studies are included in Table 9.
Problem 1
Problem 2
Inval. (%) (4)
F1(()
Diversity (L)
Inval. (%) (4)
(t)
Diversity (4)
GAN-DO (7 0.0)
0.2510.24
0.49311.004
15.111::0.10
0.5210.06
0.73410.0017
14.343.10.02
GAN-DO (7 0.1)
0.6320.26
0.629:10.013
14.85410.011
1.36:10.15
0.817=0.010
14.201=0.01
GAN-DO (7 0.2)
0.8010.14
0.645:10.014
14.766:10.02)
2.9410.29
0.83620.004
14.097:0.002
GAN-MDD (7 0.0)
0.33+0.27
0.62740.159
15.119::0.78
0.8420.38
0.829=0.010
14.165=0.022
GAN-MDD 0.1)
0.4020.13
0.70040.026
14.98840.045
0.7310.119
0.83020.008
14. 14.15940.001
GAN-MDD 0.2)
0.35=0.15
0.79440.013
14.7164:0.010
2.0110.28
0.85820.008
14.043=200011
Table 9: Scores for GAN-DO and GAN-MDD on Problem 1 and Problem 2 over variety of diversity loss weights. Mean scores and standard deviations over six instantiations are shown.
A.4 Topology Optimization
We visualize more samples generated by GAN, GAN-DO and GAN-MDD, annotating validity in Figures through 19 and visualize mode collapse GAN-DO on the synthotic negative data Figure 20.
Figure 15: mandomly-selected topologies generated by GAN with constraint violations annotated.
26

---

ZABZO
Figure 16: Randomly-selected topologies generated by GAN-DO trained on proceuuaallyggeeeraae negative data with constraint violations annotated.
DLO
Figure 17: Randomly-selected topologies generated by GAN-DO trained on rejection-sampled negative data with constraint violations annotated.
27

---

Figure 18: Randomly-selected topologies generated by GAN-MDD trained on procedurally-generateed negative data with constraint violations annotated.
I
Figure 19: Randoomyy-selected topologies generated by GAN-MDD trained on rejection-sampled negative data with constraint violations annotated.
28

---

Figure 20: A batch of 32 random topologies generated by GAN-DO trained on procedurally-generae negatives. Topologies are manually grouped into just handful of data modes, indicating severe mode collapse.
29

---

B
Pseudocode
Pseudocode for GAN-MDD training formulation is shown below:
Algorithm GAN-MC Training Procedure
while step I Nsteps do
Sample Pootch Paataset and Nbatch c Natoset Sample ~ N(0, 1)
Gbatch- = Generator(e)
D
preds Discriminator (Pbttch)
preds Discriminator (Nbatoh)
DG
preds Discriminator battcc)))
loss fn CatteoricalCCrossnntroo
D_ loss = loss (n preds 0) + loss fn preds: 1) + loss fn(DN Diversily loss = DPP (hihchht))))
preds 2)
G_ flloss= loss fn
preds 1) + Diversity_ loss D loss)
CaaimizzoDiscrmmnnaa
Optimize( Generator, G_loss)
step step end while
C Extended Background and Related Work
Constraints in Engineering Problems. Generally, we can categorize the constraint information of engineering problems into four types (Regenwetter et al., 2023).
(i) No Constraint Information: No information about constraints is given or can be collected, and learning constraints typically infeasible or extremely challenging a finite data regime.
(ii) 'Negative' Dataset of Invalid Designs: A collection of constraint-violating negative designs is available. Our method leverages such negative data to learn constraint-satisfyin generative model.
(iii)
Constraint Check: A black-box 'oracle' is available to determine whether a design satisfies constraints. This check may be computationally expensive, limiting its use.
(iv)
Closed-form Constraints: An inexpensive closed-form constraint is available. In such scenarios, direct optimization often favored over generative models design problems. In other cases, constraint-enforcing rules can be built into the model structure, an approach used in some generative models for molecular design (Cheng et, al., 2021; Imrie et al., 2020).
We note that each level of constraint information is strictly more informative than the previous. In this paper, we focus on the scenario in which a limited dataset of negative samples is available (fi) or can be generated using an oracle (iii), but closed-form constraints are not available. This scenario is common in applications such as structural design, mobility design (e.g., cars, bikes, ships, airplanes), and material synthesis.
Density Ratio Estimation. Density Ratio Estimation (DRE) (Sugiyama el al., 2012b) a critical technique machine learning, particularly when evaluating distributions is infeasible or is computationally expensive (Mohamed & Lakshminarayanan, 2016). DRE techniques are heavily employed for generative modeling and score matching estimation (Goodfellow et al., 2014; Gutmann & Hyvärinen, 2010; Srivastava et al., 2023; Choi et al., 2022). In the context of GANs (Goodfellow et al., 2014; Arjovsky et al., 2017), the DRE methodology forms the underlying basis for their operation. A well-known technique for DRE is probabilistic classification Sugiyama et al. (2012b), where a binary classifier is used to learn the ratio. However, accurately performing DRE with finite samples is particularly challenging in sigh-dimensional spaces. To overcome this challenge, prior works have employed divide-and-conquer approach. An example of this is the Telescoping Density Ratio Estimation (TRE) method (Gutmann & Hyvärinen, 2010; Rhodes et al., 2020), which divides the problem into sequence of easier DRE mub-problems. Despite its success, there are limitations to this approach, especially when the number of intermediate bridge distributions is increased.
30

---

Noise contrastive estimator (NCE (Gutmann & Hyvärinen, 2010)) and hybrid generative models (Srivastava et al., 2023; 2017; Rhodes et al., 2020) are also based on the density ratio as underlying methodology, providing a flexible paradigı for large scale generative modeling.
Density Ratio Estimation in the Negative Data Context. Let Ph denote the negative distribution ie., the distribution of austraint-violatinn datapoints. Instead of training using only the positive distribution Pp: NDGM formulations soek to train a generative model ps using both Pp and Pn- Assuming mutual absolute continuity of Pp. Pe and Pa, and starting from first principles, we can now re-write Eq. 2 as:
/ po po - log 1og)p(x)] dx
arg min
/ (o((x)
log (x) logpp(x) + (ogg Pn((x)
Pr Pr( ) dx
arg min
/ (x())
log (x
log (x(
arg min
Pn Pn(x)) dx.
(13)
Pa(x))
While the solution for Eq. 13 is the same as the solution for Eq. 2 i.c. po- = Pp, the model is now directly incentivized to allocate the same amount of probability mass to the samples from Pa as does the data distribution Pp. This ensures that when trained using finite N, the model avoids allocating high probability mass to invalid samples. In other words, training under Eq. 13 encourages the model to minimize its discrepancy with respect to Pp such that its discrepancy with respect to pn matches exactly that of Pp and Pn-
Generative Models for Engineering Design. Generative models have recently seen extensive use in design generation tasks (Regenwetter et al., 2022a). Generative Adversarial Nets, for example, have seen extensive use in many applications. In topology optimization, GANs (Li et al., 2019; Rawat & Shen, 2019; Oh et al., 2018; 2019; Sharpe & Seepersad, 2019; Nie et al., 2021; Yu et al., 2019; Valdez et al., 2021; Mazé & Ahmed, 2023) are often used to create optimal topologies, potentially bypassing time-consuming iterative solvers. In computational materials design, GANs (Tan et al., 2020; Yang et al., 2018; Zhang et al., 2021; Mosser et al., 2017; Lee et al., 2021; Liu et al., 2019), VAFs (Cang et al., 2018; Li et al., 2020; Liu et al., 2020; Wang et al., 2020; Xue et al., 2020; Tang et al., 2020; Chen & Liu, 2021), and other models are used to generale synthetic data to better learn process-structure-rreee
relations (Bostanabad et al., 2018). A variety of generative models have been applied to 2D shape synthesis problems (Yilmaz & German, 2020; Chen & Fuge, 2018; Chen et al., 2019; Chen & Fuge, 2019; Nobari et al., 2022; Li et al., 2021; Dering et al., 2018), such as airfoil design, and 3D shape synthesis problems (Shu et al., 2020; Nobari et al., 2022; Brock et al., 2016; Zhang et al., 2019) such as mechanical component synthesis in engineering design. Finally, generative models have been proposed as a method to tackle various miscellaneous product and machine design tasks (Deshpande & Purwar, 2019; Sharma & Purwar, 2020; Regenwetter et al., 2021; Deshpande & Purwar, 2020).
Constraint Satisfaction in Machine Learning. From general point of view, Constraint Satisfaction Problems (CSPs) have been long studied in computer science and optimization about optimal allocation, graph search, games, and path planning (Russell, 2010). However, such constraints are mostly related to algorithmmic complexity and memory allocation. In generative design, the goal of constraint satisfaction differs as it aims to achiove both regh-perforancc designs and diverse distribution coverage through probabilistic modeling (Regenwelter el al., 2022a). Recently, Neural Constraint Satisfaction (Chang el al., 2023) has been proposed to deal with objects in scene to solve intuitive physics problems (Smith et al., 2019; Hamrick et al., 2018). In the CAD domain, structured models to handle constraints have been proposed (Seff et al., 2021; Para et al, 2021). Conditional generative models have been proposed for structural topology optimization (Nie et al., 2021), leveraging physical fields (Nie et al., 2021; Mazé & Ahmed, 2023), dense approximations (Giannone& & Ahmed, 2023), and trajectory aligument (Giannone et al., 2024) for high-quality candidate generation. These approaches rely on explicit constraint satisfaction. Instead, we focus on implicit constraint satisfaction, leveraging a dataset of invalid configurations to enhance the model capacity to generate valid designs.
31

---

D Comparison to Rejection Sampling
In constrained generation problems, rejection sampling is simple, yet powerful strategy to ensure constraint satisfaction. Unfortunately, a black-box constraint check is not always available and may be prohibitively costly. In the negative data domain, negative data provides an opportunity to train a supervised constraint evaluation surrogate which can be used during rejection sampling, generally at lower cost than a query to a ground-truth constraint oracle. To generate batch of samples, the generative model and predictive model will be queried in loop, discarding any predicted invalid samples and collecting predicted valid samples until sufficient number of predicted valid samples is accumulated.
D.1 Benchmarking
We benchmark a vanilla GAN, VAE, and DDPM this manner, training a supervised classifier using the negative data, then applying the classifier during inference. Figure 21 and Table present the scores, while Figures 22 and 23 show the distribution plots. Rejection sampling outperforms GAN-MDD and GAN-DO (and most baselines) in terms of invalidity rate. Noting that supervised classifier has much simpler task than constrained generative model, we would not expect NDGMs to learn constraints boundaries more precisely than classifiers. Thus, it is unrealistic to expect NDGMs to beat this rejection sampling baseline in constraint satisfaction scores. As expected, vanilla models augmented by rejection sampling still often underperform NDGMs in Fl score.
Problem 1
Problem 2
2.00
4.0
AA
1.75
3.5
DO-0.2
1.50
3.0
Invalidity Rate (%)
Invalidity Rate (%)
1.25
MDD-0.2
1.00
2.0
DO-0.2
0.75
DO-0.1
1.5
DO-0.1
DDPM+RS
0.50
MDD-0.2
MDD-0.0
1.0
MDD-0.0
DO-0.0
0.25
MDD-0.1
0.5
MDD-0.1
DO-0.0
DDPM+RS
GAN+RS
VAE+RS
GAN+RS
VAE+RS
0.00
0.0
0.8
0.7
0.6
0.5
0.4
0.85
0.80
0.75
0.70
F1 Score
F1 Score
Figure 21:
Comparison of F1 scores (f) and invalidity rates (L) for GAN-MDD and GAN-DO against
rejection sampling basclines (*+RS") on Problem 1 (left) and Problem 2 (right). Gray markers indicate scores for other models benchmarked in the main paper. Mean scores over six instantiations are plotted. Scores closer to the bottom left are more optimal. Triangular markers indicate that the score lies off the plot in the indicated direction.
Problem 1
Problem 2
Inval. (%) (L)
F1 Score (t)
Diversity (-)
Inval. (%) (L)
Score (t)
Diversity (L)
VAE+RS
0.037++0.022
0.300
14.976=0.010
0.067+00.02
0.789+0.002
14.393:+0.004
GAN+RS
0.047:40.055
0.63441.0000
15.071+0.060
0.037-10.018
0.853-10.015
14.157-10.031
DDPM+RS
0.01740.001
0.797=4.000
14.82940.013
0.080++0.059
0.870=0.005
14.086+0.001
Table 10: Detailed scores for vanilla models with rejection sampling on Problem 1 and Problem 2. Best scores are bolded. Mean scores and standard deviations over six instantiations are shown.
32

---

(a) Positive Data
(b) VAE+RS
(c) GAN+RS
(d) DDPM+RS
Generated distributions for vanilla models on Problem 1 after rejection sampling using a classifier trained distinguish negative and positive data.
Figure 22:
(a) Positive Data
(b) VAE+RS
(c) GAN+RS
(d) DDPM+RS
Figure 23: Generated distributions for vanilla models on Problem 2 after rejection sampling using classifier trained to classify negative vs. positive data.
D.2 Rejection Sampling Carries Significant Inference- Time Costs
Rejection sampling using a predictive model carries significant. downsides when deployed in practice. Most notably, the number of model calls is expected to at least double, and it may increase by orders of magnitude if the generative model's validity rate is very low. Furthermore, the inference time of the model will be stochastic. For model with validity rate v, the expected mean and variance of the number of function calls (n.) is:
2 0. Varine] = 4(1 -v v)
E n
(14)
x2
illustrate the huge inference cost scaling and unpredictability for models with low validity rale, Table compares the rejection sampling cost of GAN trained on Problem and a GAN trained on the ship hull engineering problem. The validity rate of these Rejection sampling increases the expected number of model calls from to 2.18 for Problem and to 33.17 for the ship hull problem. We also estimate the expected worst case scenario out of 1000 inference runs. As shown, in one of a thousand inferences, we expect to require at least 222 calls to the model to generate a valid ship hull.
models is hugely different 91.7% and 6.1%, respectively.
Inference memory costs also increase, further complicating real-time deployment of generative models. This combination of factors can make rejection sampling less practical for real-time deployment of generative models. These significant downsides motivate the further exploration and development of NDGMs.
Naturally, NDGMs can also be used with rejection sampling to accelerate inference due to their higher validity rates and sometimes superior distribution learning. Nonetheless, rejection sampling serves as useful reference point for projecting the frontier of NDGM modeling capabilities, assuming perfect generation based on the quality of constraint boundary estimates learned by supervised models.
33

---

Problem 1
Ship Hull
GAN model validity rate (%) Expected number of model calls Variance in number of model calls 99.9th percentile number of model calls
91.68
6.07
2.182
33.17
0.396
4081
222
Table 11: Inference statistics for Problem and ship hull. As seen for the ship hull design problem, low model validity rates cause significantly higher unpredictabiiity and expected number of model calls.
E
Double Discriminator Training Formulations
E.1 Formulation
Though using GAN-MDD''s multi-class discriminator to learn pairwise density ratios between Pps Pas and pa is simpler, can also accomplish the task using multiple discriminative models. For example, fo can estimate the ratio Pp/Po (a standard GAN discriminator), while f estimates Pa/po- The loss is then expressed as:
C(0, ) = pxp log fo [x)] + (x) (xg(f((Xo)) (x) log f (x)) (xpo(>) log(s (x+))].
(15)
Here, À E [0 1] a tuning parameter adjusting the weight of the negative data's contribution to the loss and avoiding instability. Optimal discriminators learn:
fo(x) (x( (xo (x)) fø(x) (po(x) + (x))
Px(x)
(16)
The rationale behind the double discriminator algorithm is intuitive when viewed expansion of vanilla GAN. The generator aims for its samples to be classified as positive by the original discriminator and not as negative by the extra discriminator.
We benchmark this double discriminator variant below, titled GAN-DDD (double discriminator + diversity). In practice, we also find that an alternate formulation that combines this simple wwo-discriminator concept with discriminator overloading (DO) also works well in many cases. The alternative formulation consists of the classic discriminator, fe estimating Pp/Po and an overloaded discrininator, fg estimating (Pp + po)/Pn. The total loss function is then expressed as:
[( 6, )+ = (x) [log fo (x)] + Epu(x]] - log(fo(Xx)))
(17)
+ AB (x) [log (x)) + X f (xo)] + p.. log(s(
Once again, À is weighting parameter modulating the contribution of the negative data.
E.2 Testing
We benchmark GAN-DDD on the 2D test problems using a negative data weight of A = 0.4. Scores are plotted against other models in Figure 24 and tabulated in Table 12, while generated distributions plotted in Figure 25. Although it performs very well in Problem GAN-DDD generates many invalid samples in Problem 2. In general, we find that the careful tuning of the negative data weighting parameter and diversity loss weighting parameter (A and 7, respectively) have significant impacts on the tradeoff between distributional similarity and validity. Therefore, we generally recommend GAN-MDD as a simpler off-tho-shelf method, even though GAN-DDD may perform better specific scenarios.
34

---

Problem
Problem
Inval. (%) (4)
F1
Diversity (4)
Inval. (%) (4)
(t)
Diversity (4)
GAN-DDD
0.60+10.16
0.793-10.015
14.703+0.006
5.21+:0.6
0.858++0.001
14.028-:0.036
Table 12: Scores for GAN-DDD Problem and Problem 2. Mean scores and standard deviations over six instantiations are shown.
Problem 1
Problem
10
DDPM
GAN
VAE
DDPM
VAE
8
12
Invalidity Rate (%)
Invalidity Rate (%)
10
6
GAN-DDD (Ours)
DDPM (Guided)
4
6
GAN
2
GAN-DDD
GAN-MDD (Ours)
(Ours)
GAN-DO DDPM (SOTA) (Guided)
2
GAN-MDD
(Ours)
GAN-DO (SOTA)
0
0.8
0.7
0.6
0.5
0.4
0.85
0.80
0.75
0.70
F1 Score
Score
Figure 24: Comparison of GAN-DDD's Fl scores (1) and invalidity rates (1) to baselines on Problem 1 (left) and Problem 2 (right). Gray markers indicate scores for other models benchmarked in this paper, with a few selectively annotated. Mean scores over six instantiations are plotted. Scores closer to the bottom left are more optimal. Triangular markers indicate that the score lies the plot the indicated direction.
(a) Positive Data
(b) GAN-DO (SOTA)
(c) GAN-MDD (Ours)
(d) GAN-DDD (Ours)
(e) Positive Data
(I) GAN-DO (SOTA)
(g) GAN-MDD (Ours)
(h) GAN-DDD (Ours)
Figure 25: Comparison of GAN-DDD's generated distributions with those of GAN-DO and GAN-MDD
Problem 1 and Problem 2.
35

---

F Details on Datasets, Models, and Training
F.1 2D Experiments
F.1.1 Dataset Details
Problem 1: Datapoints are randomly sampled from one of six modes, each of which is a 2D Gaussian. Distribution centers are spaced at an equal radius. Points in close proximity to the center of any distribution are labeled as negatives, while others are labeled as positives. Sampling performed until 5K positive samples and 5K negative samples are acquired, and any excess in the oversampled class is discarded.
Problem 2: Datapoints are uniformly sampled. A square grid of "centerpoints' is overlaid over the distribution. Datapoints within a specified proximity to a 'centerpoint" are considered negative, while all others are labeled as positive. Sampling is performed until 5K positive samples and 5K negative samples are acquired, and any excess in the oversampled class is discarded.
F.1.2 Training Details
All tested networks (encoder, decoder, generator, discriminator, DDPM noise model, auxiliary discriminator) are deep networks with one hidden layer of 400 neurons and ReLU activations. A batch size of 256 is used throughout. Models are trained using the Adam optimizer (Kingma & Ba, 2014) with a learning rate 3 10-1. Models are trained for 10000 epochs. The noise dimension for the GAN and latent dimension for the VAE are set at 8. Diversily weights are set at 0.1.
F.2 Engineering Experiments
F.2.1 Dataset Details
Several of the engineering datasets were compiled and described in Yu et al. (2024). For all datasets in this section, optimization objectives are not utilized.
Ashby Chart: Taken from (Jetton et al., 2023), this problem explores physically feasible combinations of material properties, according to known physical matcrials from an Ashby chart. The constraint function combines an analytical constraint and a lookup from an Ashby chart. Material properties considered are density, yield strength, and Young's modulus. Material classes included are foams, natural materials, polymers, composites, ceramics, and metals. 1K positive samples and 1K negative samples are selected using uniform random sampling.
The FRAMED dataset (Regenwetter et al., 2022b) is comprised of 4292 in-distribution (positive) human-designed bicycle frame models. FRAMED also contains 3242 constraint-violating (negative) designs, some of which were human-designed and some of which were synthesized by generative models. FRAMED also contains 10095 generative model-syythesized valid designs that are not assumed to be in-distribution and are thus unused in this benchmark. Constraints consist of a set of empirical geometric checks and a black-box 3D reconstruction chock. Constraints are unified using an all-or-nothing approach. Validity scores this dataset are only evaluated using empirical checks.
Bike Frame:
This problem considers the design of a five-component stepped cantilever beam. The thickness and height of each of the five components are the design variables, while the lengths of each component are given (fixed). Taken from (Gandomi & Yang, 2011), this problem has numerous geometric constraints and an overall constraint limiting the total deflection allowed by the design under a simple concentrated load at the tip of the beam. 1K positive samples and 1K negative samples are selected using uniform random sampling.
Cantilever Beam:
This problem quantifics the performance of car design under side impact scenario based European Enhanced Vehicle-Safety Committee (EEVC) procedures (Gandomi el al., 2011). The car chassis is represented by 11 design parameters. Critical deflection, load, and velocity thresholds are specified for various components of a crash dummy, constituting 10 constraints. 1K positive samples and 1K negative samples are selected using uniform random sampling.
Car Impact:
Compression Spring: This problem, taken from (Gandomi Yang, 2011), centers around the design of helical compression spring parameterized over coil diameter, wire diameter, and number of spring coils.
36

---

A constraint free length and constraint displacement under compressive load are specified. 1K positive samples and 1K negative samples are selected using uniform random sampling.
Taken from (Gandomi & Yang, 2011), this problem centers around the design of a simply supported concrete beam under a distributed load case. The beam is parameterized using a cross sectional area, base length, and height and is subject to a safety requirement indicated in the American Concrete Institute (ACI) 319-77 code. 1K positive samples and 1K negative samples are selected using uniform random sampling.
Concrete Beam:
Gearbox: This gearbox (speed-reducer) design problem, taken from (Gandomi & Yang, 2011) features 7 parameters describing key geometric components like shaft diameters, number of teeth on gears, and face width of gears. Nine constraints are given, spanning consideration like bending st.ress on gear teeth, transverse stress and deflection on shafts, and surface stresses. 1K positive samples and 1K negative samples are sclected using uniform random sampling.
This problem, sourced from (Yang & Gandomi, 2012) considers the design of a heat. exchanger involving eight design parameters and six constraints focused geometric validity. 1K positive samples and 1K negative samples are selected using uniform random sampling.
Heat Exchanger:
This cylindrical pressure vessel design problem is taken from (Gandomi & Yang, 2011). The pressure vessel is parametrized according to four parameters, namely the cylinder thickness, spherical head thickness, inner radius, and cylinder length. Four geometric and structural constraints are specified in accordance with American Society of Mechanical Engineers (ASME) design codes. 1K positive samples and 1K negative samples are selected using uniform random sampling.
Pressure Vessel:
The SHIPD Dataset (Bagazinski & Ahmed, 2023) is comprised of 30k valid (positive) ship hull designs and 20k invalid (negative) ship hull designs. The SHIPD dataset includes numerous constraints spanning geometric rules and functional performance targets, focusing on varions types of hydrodynamic performance.
Ship Hull:
Taken from (Yang & Gandomi, 2012), this truss design problem considers the design of three-beam truss parameterized by the length of two of the beams (symmetry specifies the length of the third). The system is subject to one geometric constraint and two maximum stress constraints. 1K positive samples and 1K negative samples are selected using uniform random sampling.
Three-Bar Truss:
Welded Beam:
Taken from (Gandomi & Yang, 2011), this problem concerns a cantilever beam welded to a flat surface under a simple concentrated load at the tip of the beam. The beam is parametrized using a weld thickness, welded joint length, beam thickness, and beam width. Five structural constraints are given, specifying a maximum shear stress, bending stress, buckling load, and deflection, as well as a geometric constraint on the beam. 1K positive samples and 1K negative samples are selected using uniform random sampling.
F.2.2 Training Details
The generator and discriminator are deep networks with one hidden layer of 400 neurons and ReLU activations. A batch size of 256 is used throughout. Models are trained using the Adam optimizer with a learning rate 3 10-1 Models are trained for 10000 epochs. The noise dimension for the GAN and latent dimension for the VAE are set at 8. Diversity weights are tuned by selecting the highest score from 10 increments tested.
F.3
Topology Optimization Experiments
F.3.1 Dataset Details
The GAN was trained exclusively on 32436 valid (connected) topologies generated through iteralive oplimiza- tion (SIMP) (Bendsøe & Kikuchi, 1988). The GAN-MDD and GAN-DO models are trained on a medley of disconnected topologies generated by iterative optimization (2564), and either procedurally-geenraaee synthetic topologies (35000) or GAN-generated disconnected topologies (92307). Synthetic topologies were sourced directly from the classification dataset of (Mazé & Ahmed, 2023). The GAN used to generate disconnected topologies for rejection was the exact GAN benchmarked in the paper (the first instantiation of six). Topologies were chocked for continuity and rejected samples were added to the negative dataset. All
37

---

positive and negative data augmented sevenfold using horizontal and vertical flips, well quarter-turn rotations, prior to training.
F.3.2 Training Details
The model architectures of the GAN, GAN-DO and GAN-MDD are identical except for the final output dimension of the discriminator. Both the generator and discriminator simple 5-layer convolutional neural networks. The generator has 3.6M parameters, while the discriminator has 2.8M parameters. For more architectural details, refer the reader to the codebase. The latent dimension is 100, batch size is 128, and learning rate for both models is 3 10-,, using the Adam optimizer.
38


Reviews:
[{"review_id": "abHQxQwmIk", "reviewer": "Reviewer_RJhg", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The proposed NDGMs significantly enhance constraint satisfaction across a variety of synthetic and challenging engineering problems. \n2. NDGMs  achieve superior performance using much less data compared to other models, highlighting their data efficiency.\n3. The paper includes a comprehensive benchmark, providing a robust evaluation framework for NDGMs.\n\nWeaknesses:\n1. The use of dual and multi-class discriminators may result in higher computational costs and longer training times, particularly for large-scale datasets.\n2. Although the methods are effective, the proposed multi-category discriminator and dual-discriminator methods are not technically significant innovations. However, the results are good.\n3. A simple application of GAN in the field of Engineering Design. Whether it is inspiring for other fields is uncertain.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work introduces two new Negative-Data Generative Models (NDGMs) that match or surpass state-of-the-art methods. The proposed NDGMs can significantly outperform vanilla models using less data. Meanwhile, to evaluation it, the paper also includes an extensive benchmark set across many synthetic problems and engineering tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "No change request.", "broader_impact_concerns": "I'm not sure how much of an impact this work will have on the field of Engineering Design.I am not an expert in the field of Engineering Design.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "CS1Y6h5paS", "reviewer": "Reviewer_W5Nm", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strength**\n1) The NDGM Formulations are quite easy to understand, and the authors provided extensive benchmarks.\n2) The experiments showed improved constraint satisfaction rates and improved data efficiency.\n\n**Weakness**\n1) This paper mainly focuses on GAN models instead of diffusion models. It's interesting to compare with other diffusion models.\n2) In the main paper, there is no discussion about the sampling diversity.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "For generative modeling with constrains, the paper proposed two approaches to guide models toward constraint-satisfying outputs using negative data. With extensive benchmarks, the NDGM models outperform against baseline models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "It will be great to add the sample diversity discussion in the main paper. So that readers can understand the tradeoff of new approaches easier.", "broader_impact_concerns": "n/a", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "loQkNF5Jq2", "reviewer": "Reviewer_SoqL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# strengths\n- Provides strong evidence for the importance of negative data to a breadth of engineering tasks. The breadth of evidence is compelling and there are a healthy variety of alternative models tested.\n- Contributes two conceptually straightforward GAN extensions with reasonably good empirical performance.\n\n\n# weaknesses\n- Lack of reporting statistical variability in the main text (supplement includes some)\n\t- There is no reporting of statistical tests of differences among models when claims are made about which model or models are better (worse) for a given task. The appendix tables include measures of statistical variation, so the text should include empirical estimates of effect sizes (and statistical significance) when comparing models. This will strengthen the claims about differences among models.\n- The two contributed models are not included in all of the evaluations. \n\t- GAN-MC and GAN-DD-a (and possibly GAN-DD-b) should be included in all tables in the main text as they are the main technical contributions of the paper.\n\t- It would help to include the DDPM model variants in all results as well. Or a table clearly summarizing their performance. The claim that diffusion models are inferior on these tasks is interesting and provocative, so providing strong evidence of the consistency of this trend (or specificity to particular evaluations) will be valuable. The text also lacks a single easy reference summary for readers to assess the evidence in favor the the inferiority of diffusion models across the many evaluations.\n- (minor) The benchmarks do not include any sense of what \"good\" performance is. \n\t- Is there a human baseline or other baseline to include to compare these generative methods to?\n\t- A \"gold standard\" accepted by engineers in the respective tasks?\n\t- This would strengthen the ecological validity of these tasks.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- contributions\n\t- Develops two approaches to incorporating negative data into generative models (specifically GANs):\n\t\t- multi-class model: Trains a multi-class classifier to distinguish positive, negative, and generated samples as an auxiliary GAN loss.\n\t\t- double discriminator: Train two discriminators, one for positive samples and one for negative samples. A weighted combination of these is used in the GAN training loss.\n\t- Evaluates several negative data generative model variants (and baseline models that do not use negative samples) across toy examples, an engineering task suite, a block stacking task, and topology optimization for engineering design.\n\t- Evaluates some of the scaling properties obtained by adding negative samples.\n- new knowledge\n\t- GANs outperform DDPM (representing the class of diffusion models) on engineering tasks with hard constraints.\n\t- Modest amounts of negative samples can substantially improve model performance in terms of satisfying hard constraints (at least for some tasks).\n\t- Negative samples can vary in how much they improve performance based on the form of the negative samples (being \"easy\" or \"hard\"). Specifically contrasts samples generated from heuristically generated synthetic data and samples generated by rejection sampling from a baseline generative model.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# critical\n- In the main text tables please include the following:\n\t- Estimates of statistical variability for all results reported in tables.\n\t- Estimates of effect sizes and statistical significance for any differences claimed. For example, if the claim is GAN-MC outperforms GAN, include an estimate of the magnitude of improvement.\n- Include GAN-DO, GAN-DD, GAN-MC in all mainline reported results.\n\t- These are the contributed methods (and strong baseline in the case of GAN-DO) that are claimed contributions, and thus merit comprehensive evaluation.\n\n# strengthen\n- Ideally also include DDPM in all the evaluations. The claim that DDPM is inferior is very interesting and merits more comprehensive evaluation and summarization.\n- (less critical) The potential of negative samples to improve generative models is fascinating. I wonder about two scaling studies:\n\t- 1) Extending the results reported in Table 2 to include other methods (particularly DDPM given it's popularity as a model elsewhere)\n\t- 2) Extending the scaling range to cases where models train on more negative samples than positive. In at least some engineering domains it is easy to generate negative samples but positive samples are scarce. How well do the sample efficiency results extend to the regime of far more negative than positive samples?\n\t\t- The GAN-DD results from Table 2 are quite promising in this regard. 4x as many positive samples (from 1K) only yields a ~0.1 percentage point improvement (if any) for models with 4K or 16K negative samples. If possible this same scaling would be interesting to apply to the topology optimization problem, given the ability to generate negative samples there.\n\n\nOther notes to help strengthen the submission:\n- Some experiments would benefit from matching the number of training samples used in total (positive and negative). some comparisons are somewhat \"unfair\" in providing the model access to additional data (as negative samples) that are not used by vanilla baselines\n- (reiterating above) A general remark on the tables: Please report the number of trials for each table (ideally in the caption). The tables should include reports of statistical variability (often included in the appendix) and some statistical test of differences for any differences among models being claimed. This remark applies to all the tables provided and claims about differences in the main text.\n- Table 1\n\t- It would be interesting to see these results but training the vanilla models (GAN, VAE, DDPM) with additional positive samples to match the total number of samples given to the negative models. For example, providing 20k positive samples to the vanilla models to make the 10k positive and 10k negative. The idea is to be somewhat more \"fair\" in granting access to the same amount of data to learn from, where the only difference is whether the samples come from the negative distribution.\n- Table 2\n\t- I find it interesting that 4K positive samples often outperforms 16K for both problems and most model (except GAN-DD 1K). Any idea why this might be?\n\t- This table should include GAN-MC results for the same scaling test.\n\t- It would be interesting to see DDPM performance on this task: do they scale more poorly than GANs? Could the problem be that DDPMs require more data to start being effective (hence the negative main result in the experiments so far)?\n\t- This table should at least include GAN-MC results.\n- Table 3\n\t- This table should include GAN-MC results.\n- Table 4\n\t- This table should include GAN-DD results.\n- Table 5\n\t- This table should include GAN-DO and GAN-DD for comparison.\n\t- How many pixels are in the output images? It is hard to interpret a score of 0.29 pixels without that information.\n- Tables 14-19\n\t- These tables include \"GAN-D2\" and \"GAN-RM\", but I could not find either defined in the text.\n\t- These tables lack reporting of variability over multiple trials (in addition to lacking statistical tests of differences for the claims being made in the text).\n- Section 5.1\n\t- \"Figure 5 plots the datasets and the generated distributions of several select models (vanilla GAN and the two NDGMs we propose)\"\n\t\t- \"Figure 5\" should be \"Figure 2\"", "broader_impact_concerns": "Broader impact is not directly addressed in an explicit Broader Impact Statement section. It does not seem necessary as the goal of the work is enabling better control over the outputs of generative models.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All reviewers agree on the simplicity and superior performance of the proposed method, as evidenced by extensive experiments. The concerns raised regarding computational cost, sample diversity, and statistical variability have been thoroughly addressed by the authors. Considering the strong performance in the engineering task domain, I support the acceptance of this paper.

Please prepare the final version of the manuscript, ensuring that you adequately incorporate the reviewers' comments and your responses to them.
Decision: Accept (poster)