SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Giorgio Giannone
Technical University of Denmark
ggiorgio@mit.edu
Ole Winther
olwi@dtu.dk
Akash Srivastava
akashsri@mit.edu
Faez Ahmed
faez@mit.edu
Generative models have significantly influenced both vision and language domains, ushering in innovative multimodal applications. Although these achievements have motivated exploration in scientific and engineering fields, challenges emerge, particularly in constrained settings with limited data where precision is crucial. Traditional engineering optimization methods rooted in physics often surpass generative models in these contexts. To address these challenges, we introduce Diffusion Optimization Models (DOM) and Trajectory Alignment (TA), a learning framework that demonstrates the efficacy of aligning the sampling trajectory of diffusion models with the trajectory derived from physics-based iterative optimization methods. This alignment ensures that the sampling process remains grounded in the underlying physical principles. This alignment eliminates the need for costly preprocessing, external surrogate models, or extra labeled data, generating feasible and high-performance designs efficiently. We apply our framework to structural topology optimization, a fundamental problem in mechanical design, evaluating its performance on in- and out-of-distribution configurations. Our results demonstrate that TA outperforms state-of-the-art deep generative models on in-distribution configurations and halves the inference computational cost. When coupled with a few steps of optimization, it also improves manufacturability for out-of-distribution conditions. DOM's efficiency and performance improvements significantly expedite design processes and steer them toward optimal and manufacturable outcomes, highlighting the potential of generative models in data-driven design.
The remarkable progress in large vision [27, 20, 43, 85] and language models [30, 21, 78] has ushered in unparalleled capabilities for processing unstructured data, leading to innovations in multimodal and semantic generation [79, 67, 18, 80]. This momentum in model development has influenced the rise of Deep Generative Models (DGMs) in the realms of science [51, 61] and engineering [84, 101], especially in constraint-bound problems like structural Topology Optimization (TO [95]), offering potential to make the design process faster.
Many engineering design applications predominantly rely on iterative optimization algorithms. These algorithms break down physical and chemical phenomena into discrete components and incrementally enhance design performance while ensuring all constraint requirements are met. As an example, topology optimization aims to determine the optimal material distribution within a given design space,
(a) Sampling and Optimization Trajectory
(b) Trajectory Alignment
Figure 1: Trajectory Alignment. Intermediate sampling steps in a Diffusion Optimization Model are matched with intermediate optimization steps. In doing so, the sampling path is biased toward the optimization path, guiding the data-driven path toward physical trajectories. This leads to significantly more precise samples.
under specified loads and boundary conditions, to achieve the best performance according to a set of defined criteria, such as minimum weight or maximum stiffness. Traditional iterative methods like SIMP [15] are invaluable but grapple with practical challenges, especially for large-scale problems, owing to their computational complexity.
Recent advancements have sought to address these challenges by venturing into learning-based methods for topology optimization, specifically DGMs. These models, trained on datasets comprising optimal solutions across varying constraints, can either speed up or even substitute traditional optimization processes. They also introduce a diverse set of structural topologies by tapping into large datasets of prior designs. The dual advantage of generating a variety of solutions and accommodating numerous design variables, constraints, and goals renders these learning-based methodologies especially enticing for engineering design scenarios.
However, purely data-driven approaches in generative design often fall short when benchmarked against optimization-based methods. While data-driven techniques might prioritize metrics such as reconstruction quality, these metrics might not adequately gauge adherence to engineering performance and constraint requirements. The core of this challenge lies in the data-driven methods' omission of vital physical information and their inability to incorporate iterative optimization details during inference. This oversight can compromise the quality of the solutions, particularly when navigating complex constraints and stringent performance requirements. Consequently, there's an emerging need for methodologies that amalgamate the strengths of both data-centric and physics-informed techniques to more adeptly navigate engineering applications. Paving the way forward, structured generative models have made notable strides in this domain, exemplified by innovative techniques like TopologyGAN [69] and TopoDiff [62].
Limitations. Structured generative models, despite their recent advancements in engineering designs, grapple with some pressing challenges. For one, these models often require additional supervised training data to learn guidance mechanisms that can improve performance and manufacturability [84]. In the context of Diffusion Models [43], executing forward simulations is not a one-off task; it requires repetition, often in the order of tens to hundreds of times, to derive an appropriate topology [62]. Moreover, integrating physical data into these models isn't straightforward. It demands a time-consuming FEA (Finite Element Analysis) preprocessing step during both the training and inference phases, which is pivotal for computing critical parameters like stress and energy fields[62, 69]. As a result, the sampling process is slow, and the inference process is computationally expensive. Such overheads challenge the scalability and adaptability of these models. Consequently, the touted benefits of data-driven techniques, notably fast sampling and quick design candidate generation, are somewhat diminished.
Proposed Solution. We introduce a conditional diffusion model that synergistically combines data-driven and optimization-based techniques. This model is tailored to learn constrained problems and generate candidates in the engineering design domains (refer to Fig. 3). Instead of relying on computationally heavy physics-based exact solutions using FEM, our method employs cost-effective physics-informed approximations to manage sparsity in conditioning constraints. We introduce a
Trajectory Alignment (TA) mechanism in the training phase that allows the model to leverage the information in the trajectory that was used by an iterative optimization-based method in the training data, drastically cutting down the sampling steps required for a solution generation using diffusion models. Moreover, our framework can amplify performance and manufacturability in complex problems by integrating a few steps of direct optimization. This method strikes a balance between computational efficiency and precision, offering adaptability to novel design challenges. By bridging the gap between generative modeling and engineering design optimization, our solution emerges as a powerful tool for tackling complex engineering problems. At its core, our findings highlight the value of diffusion models benefitting from optimization methods' intermediate solutions, emphasizing the journey and not just the final outcome.
Contribution. Our contributions are the following:
(i) We introduce the Diffusion Optimization Models (DOM), a versatile and efficient approach to incorporate performance awareness in generative models of engineering design problems while respecting constraints. The primary objective of DOM is to generate high-quality candidates rapidly and inexpensively, with a focus on topology optimization (TO) problems. DOM consists of
- Trajectory Alignment (TA) leverages iterative optimization and hierarchical sampling to match diffusion and optimization paths, distilling the optimizer knowledge in the sampling process. As a result, DOM achieves high performance without depending on FEM solvers or guidance and can sample high-quality configurations in as few as two steps.
- Dense Kernel Relaxation, an efficient mechanism to relieve inference from expensive FEM pre-processing and
- Few-Steps Direct Optimization that improves manufacturability using a few optimization steps.
(ii) We perform extensive quantitative and qualitative evaluation in- and out-of-distribution, showing how kernel relaxation and trajectory alignment are both necessary for good performance and fast, cheap sampling. We also release a large, multi-fidelity dataset of sub-optimal and optimal topologies obtained by solving minimum compliance optimization problems. This dataset contains low-resolution (64x64), high-resolution (256x256), optimal (120k), and suboptimal (600K) topologies. To our knowledge, this is the first large-scale dataset of optimized designs that also provides intermediate suboptimal iterations.
Here we briefly introduce the Topology Optimization problem [14], diffusion models [105, 44, 98], a class of deep generative models, conditioning and guidance mechanisms for diffusion models, and deep generative models for topology optimization [69, 62]. For more related work see Appendix A.
The Topology Optimization Problem. Topology optimization is a computational design approach that aims to determine the optimal arrangement of a structure, taking into account a set of constraints. Its objective is to identify the most efficient utilization of material while ensuring the structure meets specific performance requirements. One widely used method in topology optimization is the Solid Isotropic Material with Penalization (SIMP) method [13]. The SIMP method employs a density field to model the material properties, where the density indicates the proportion of material present in a
Figure 2: In topology optimization, the objective is to find the design with minimum compliance under given loads, boundary conditions, and volume fractions.
particular region. The optimization process involves iteratively adjusting the density field, considering constraints such as stress or deformation. In the context of a mechanical system, a common objective is to solve a generic minimum compliance problem. This problem aims to find the distribution of material density, represented as  $\mathbf{x} \in \mathbb{R}^n$ , that minimizes the deformation of the structure under prescribed boundary conditions and loads [59]. Given a set of design variables  $\mathbf{x} = \{\mathbf{x}_i\}_{i=0}^n$ , where
$n$  is the domain dimensionality, the minimum compliance problems can be written as:
$$
\min  _ {\mathbf {x}} c (\mathbf {x}) = F ^ {T} U (\mathbf {x})
$$
$$
\text {s . t .} \quad v (\mathbf {x}) = v ^ {T} \mathbf {x} <   \bar {v} \tag {1}
$$
$$
0 \leq \mathbf {x} \leq 1
$$
The goal is to find the design variables that minimize compliance  $c(\mathbf{x})$  given the constraints.  $F$  is the tensor of applied loads and  $U(\mathbf{x})$  is the node displacement, solution of the equilibrium equation  $K(\mathbf{x})U(\mathbf{x}) = F$  where  $K(\mathbf{x})$  is the stiffness matrix and is a function of the considered material.  $v(\mathbf{x})$  is the required volume fraction. The problem is a relaxation of the topology optimization task, where the design variables are continuous between 0 and 1. One significant advantage of topology optimization is its ability to create optimized structures that meet specific performance requirements. However, a major drawback of topology optimization is that it can be computationally intensive and may require significant computational resources. Additionally, some approaches to topology optimization may be limited in their ability to generate highly complex geometries and get stuck in local minima.
Diffusion Models. Let  $\mathbf{x}_0$  denote the observed data  $\mathbf{x}_0\in \mathbb{R}^D$ . Let  $\mathbf{x}_1,\dots,\mathbf{x}_T$  denote  $T$  latent variables in  $\mathbb{R}^D$ . We now introduce, the forward or diffusion process  $q$ , the reverse or generative process  $p_{\theta}$ , and the objective  $L$ . The forward or diffusion process  $q$  is defined as [44]:  $q(\mathbf{x}_{1:T}|\mathbf{x}_0) = q(\mathbf{x}_1|\mathbf{x}_0)\prod_{t = 2}^{T}q(\mathbf{x}_t|\mathbf{x}_{t - 1})$ . The beta schedule  $\beta_{1},\beta_{2},\ldots ,\beta_{T}$  is chosen such that the final latent image  $\mathbf{x}_T$  is nearly Gaussian noise. The generative or inverse process  $p_{\theta}$  is defined as:  $p_{\theta}(\mathbf{x}_0,\mathbf{x}_{1:T}) = p_{\theta}(\mathbf{x}_0|\mathbf{x}_1)p(\mathbf{x}_T)\prod_{t = 2}^{T}p_{\theta}(\mathbf{x}_{t - 1}|\mathbf{x}_t)$ . The neural network  $\mu_{\theta}(\mathbf{x}_t,t)$  is shared among all time steps and is conditioned on  $t$ . The model is trained with a re-weighted version of the ELBO that relates to denoising score matching [105]. The negative ELBO  $L$  can be written as:
$$
\mathbb {E} _ {q} \left[ - \log \frac {p _ {\theta} \left(\mathbf {x} _ {0} , \mathbf {x} _ {1 : T}\right)}{q \left(\mathbf {x} _ {1 : T} \mid \mathbf {x} _ {0}\right)} \right] = L _ {0} + \sum_ {t = 2} ^ {T} L _ {t - 1} + L _ {T}, \tag {2}
$$
where  $L_0 = \mathbb{E}_{q(\mathbf{x}_1|\mathbf{x}_0)}\left[-\log p(\mathbf{x}_0|\mathbf{x}_1)\right]$  is the likelihood term (parameterized by a discretized Gaussian distribution) and, if  $\beta_{1},\ldots ,\beta_{T}$  are fixed,  $L_{T} = \mathbb{KL}[q(\mathbf{x}_{T}|\mathbf{x}_{0}),p(\mathbf{x}_{T})]$  is a constant. The terms  $L_{t - 1}$  for  $t = 2,\dots,T$  can be written as:  $L_{t - 1} = \mathbb{E}_{q(\mathbf{x}_t|\mathbf{x}_0)}\Big[\mathbb{KL}[q(\mathbf{x}_{t - 1}|\mathbf{x}_t,\mathbf{x}_0)\mid p(\mathbf{x}_{t - 1}|\mathbf{x}_t)]\Big]$ . The terms  $L_{1:T - 1}$  can be rewritten as a prediction of the noise  $\epsilon$  added to  $\mathbf{x}$  in  $q(\mathbf{x}_t|\mathbf{x}_0)$ . Parameterizing  $\mu_{\theta}$  using the noise prediction  $\epsilon_{\theta}$ , we can write
$$
L _ {t - 1, \epsilon} (\mathbf {x}) = \mathbb {E} _ {q (\epsilon)} \left[ w _ {t} \| \epsilon_ {\theta} \left(\mathbf {x} _ {t} \left(\mathbf {x} _ {0}, \epsilon\right)\right) - \epsilon \| _ {2} ^ {2} \right], \tag {3}
$$
where  $w_{t} = \frac{\beta_{t}^{2}}{2\sigma_{t}^{2}\alpha_{t}(1 - \bar{\alpha}_{t})}$ , which corresponds to the ELBO objective [50, 56].
Conditioning and Guidance. Conditional diffusion models have been adapted for constrained engineering problems with performance requirements. TopoDiff [62] proposes to condition on loads, volume fraction, and physical fields similarly to [69] to learn a constrained generative model. In particular, the generative model can be written as:
$$
p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \mathbf {c}, \mathbf {g}\right) = \mathcal {N} \left(\mathbf {x} _ {t - 1}; \mu_ {\theta} \left(\mathbf {x} _ {t}, \mathbf {c}\right) + \sum_ {p = 1} ^ {P} \mathbf {g} _ {p}, \gamma\right), \tag {4}
$$
where  $\mathbf{c}$  is a conditioning term and is a function of the loads  $l$ , volume fraction  $v$ , and fields  $f$ , i.e.  $\mathbf{c} = h(l, v, f)$ . The fields considered are the Von Mises stress  $\sigma_{vm} = \sqrt{\sigma_{11}^2 - \sigma_{11}\sigma_{22} + \sigma_{22}^2 + 3\sigma_{12}^2}$  and the strain energy density field  $W = (\sigma_{11}\epsilon_{11} + \sigma_{22}\epsilon_{22} + 2\sigma_{12}\epsilon_{12}) / 2$ . Here  $\sigma_{ij}$  and  $\epsilon_{ij}$  are the stress and energy components over the domain.  $\mathbf{g}$  is a guidance term, containing information to guide the sampling process toward regions with low floating material (using a classifier and  $\mathbf{g}_{fm}$ ) and regions with low compliance error, where the generated topologies are close to optimized one (using a regression model and  $\mathbf{g}_c$ ). Where conditioning  $\mathbf{c}$  is always present and applied during training, the guidance mechanism  $\mathbf{g}$  is optional and applied only at inference time.
TopoDiff Limitations. TopoDiff is effective at generating topologies that fulfill the constraints and have low compliance errors. However, the generative model is expensive in terms of sampling time, because we need to sample tens or hundreds of layers for each sample. Additionally, given the model conditions on the Von Mises stress and the strain energy density, for each configuration of loads and
boundary conditions, we have to preprocess the given configurations running a FEM solver. This, other than being computationally expensive and time-consuming, relies on fine-grained knowledge of the problem at hand in terms of material property, domain, and input to the solver and performance metrics, limiting the applicability of such modeling techniques for different constrained problems in engineering or even more challenging topology problems. The guidance requires the training of two additional models (a classification and a regression model) and is particularly useful with out-of-distribution configurations. However such guidance requires additional topologies, optimal and suboptimal, to train the regression model, assuming that we have access to the desired performance metric on the train set. Similarly for the classifier, where additional labeled data has to be gathered.
Figure 3: The DOM pipeline with conditioning and kernel relaxation (top left) and trajectory alignment (top right). The Diffusion Optimization Model generates design candidates, which are further refined using optimization tools. After the generation step (left side), we can improve the generated topology using a few steps of SIMP (5/10) to remove floating material and improve performance (right side). See Appendix 10 for a comparison of the inference process for DOM and TopoDiff [62].
To tackle such limitations, we propose Diffusion Optimization Models (DOM), a conditional diffusion model to improve constrained design generation. One of our main goals is to improve inference time without loss in performance and constraint satisfaction. DOM is based on three main components: (i) Trajectory Alignment (Fig. 1 and Fig. 4) to ground sampling trajectory in the underlying physical process; (ii) Dense Kernel Relaxation (Fig. 5) to make preprocessing efficient; and (iii) Few-Steps Direct Optimization to improve out-of-distribution performance (Fig. 3 for an overview). See appendix C for algorithms with and without trajectory alignment.
Empirical Methodology. We would like to highlight that our primary contribution, trajectory alignment, is predominantly empirical. While we do make assumptions about the optimization and sampling trajectory and utilize TA, we have not established a comprehensive theoretical framework ensuring convergence of the regularized loss to the score optimized by a diffusion model. Nonetheless, our empirical findings sampling process to the desired solutions.
Figure 4: Distance between intermediate sampling steps in DOM and optimization steps with and without Trajectory Alignment. Given a random sampling step  $t$  and the corresponding optimization step  $s(t) = \mod (t,n)$  where  $n\in [2,10]$ . We compute the matching in clean space, using the approximate posterior  $q$  to obtain an estimate for  $\mathbf{x}^{\theta}$  given  $\mathbf{x}_t$  and the noise prediction  $\epsilon_{\theta}$ . Then we compute the distance  $\| \bar{\mathbf{x}}^{\theta}(\mathbf{x}_t,\epsilon_\theta) - \mathbf{x}_{s(t)}^{opt}\| _2$ .
Trajectory Alignment (TA). Our goal is to align the sampling trajectory with the optimization trajectory, incorporating optimization in data-driven generative models by leveraging the hierarchical
sampling structure of diffusion models. This aligns trajectories with physics-based information, as illustrated in Fig. 1a. Unlike previous approaches, which use optimization as pre-processing or post-processing steps, trajectory alignment is performed during training and relies upon the marginalization property of diffusion models, i.e.,  $q(\mathbf{x}_t|\mathbf{x}_0) = \int q(\mathbf{x}_{1:t}|\mathbf{x}_0)d\mathbf{x}_{1:t - 1}$ , where  $\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + (1 - \bar{\alpha}_t)\epsilon$  with  $\epsilon \sim N(0,I)$ . The trajectory alignment process can match in clean space (matching step 0), noisy space (matching step  $t$ ), performance space, and leverage multi-fidelity mechanisms. At a high level, TA is a regularization mechanism that injects an optimization-informed prior at each sampling step, forcing it to be close to the corresponding optimization step in terms of distance. This process provides a consistency mechanism [108, 104, 97] over trajectories and significantly reduces the computational cost of generating candidates without sacrificing accuracy.
Alignment Challenges. The alignment of sampling and optimization trajectories is challenging due to their differing lengths and structures. For example, the optimization trajectory starts with an image of all zeros, while the sampling path starts with random noise. Furthermore, Diffusion Models define a Stochastic Differential Equation (SDE, [106]) in the continuous limit, which represents a collection of trajectories, and the optimization trajectory cannot be directly represented within this set. To address these issues, trajectory alignment comprises two phases (see Figure 1b): a search phase and a matching phase. In the search phase, we aim to find the closest trajectory, among those that can be represented by the reverse process, to the optimization trajectory. This involves identifying a suitable representation over a trajectory that aligns with the optimization process.
In the matching phase, we minimize the distance between points on the sampling and optimization trajectories to ensure proximity between points and enable alignment between trajectories.
Trajectory Search. We leverage the approximate posterior and marginalization properties of diffusion models to perform a trajectory search, using the generative model as a parametric guide to search for a suitable representation for alignment. Given an initial point  $\mathbf{x}_0$ , we obtain an approximate point  $\mathbf{x}_t$  by sampling from the posterior distribution  $q(\mathbf{x}_t|\mathbf{x}_0)$ . We then predict  $\epsilon_{\theta}(\mathbf{x}_t)$  with the model and use it to obtain
Figure 5: Comparison of iterative (left), sparse (center), and dense single-step (right) conditioning fields for a Constrained Diffusion Model. Unlike the expensive iterative FEA method, the physics-inspired fields offer a cost-effective, single-step approximation that's domain-agnostic and scalable.
$\tilde{\mathbf{x}}^{\theta}(\mathbf{x}_t,\epsilon_\theta (\mathbf{x}_t))$  . In a DDPM,  $\tilde{\mathbf{x}}^{\theta}$  is an approximation of  $\mathbf{x}_0$  and is used as an intermediate step to sample  $\mathbf{x}_{t - 1}^{\theta}$  using the posterior functional form  $q(\mathbf{x}_{t - 1}^{\theta}|\mathbf{x}_t,\tilde{\mathbf{x}}^\theta)$  . In DOM, we additionally leverage  $\tilde{\mathbf{x}}^{\theta}$  to transport the sampling step towards a suitable representation for matching an intermediate optimization step  $\mathbf{x}_{step(t)}^{opt}$  corresponding to  $t$  using some mapping. Trajectory alignment involves matching the optimization trajectory, which is an iterative exact solution for physics-based problems, with the sampling trajectory, which is the hierarchical sampling mechanism leveraged in Diffusion Models [43] and Hierarchical VAEs [100]. In practice, in DOM we sample  $\mathbf{x}_t = \sqrt{\bar{\alpha}_t}\mathbf{x}_0 + (1 - \bar{\alpha}_t)\epsilon$  from  $q(\mathbf{x}_t|\mathbf{x}_0)$  and run a forward step with the inverse process  $\epsilon_{\theta}(\mathbf{x}_t,\mathbf{c})$  conditioning on the constraints  $\mathbf{c}$  to obtain the matching representation  $\tilde{\mathbf{x}}^{\theta}$  for step  $t$  ..
$$
\tilde {\mathbf {x}} ^ {\theta} \sim q (\tilde {\mathbf {x}} ^ {\theta} | \boldsymbol {\mu} ^ {\theta} (\mathbf {x} _ {t}, \epsilon_ {\theta}), \gamma)
$$