SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\tilde {\mu} ^ {\theta} \left(\mathbf {x} _ {t}, \epsilon_ {\theta}\right) = \left(\mathbf {x} _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \epsilon_ {\theta} \left(\mathbf {x} _ {t}, \mathbf {c}\right)\right) / \sqrt {\bar {\alpha} _ {t}}. \tag {5}
$$
Trajectory Matching. Then we match the distribution of matching representation  $q(\tilde{\mathbf{x}}^{\theta}|\mathbf{x}_t,\epsilon_\theta)$  for sampling step  $t$  with the distribution of optimized representations  $q(\mathbf{x}_{s(t - 1)}^{opt}|\mathsf{opt})$  at iteration  $s$  (corresponding to step  $t - 1$ ) conditioning on the optimizer  $S$ . In general, given that the sampling steps will be different than the optimization steps, we use  $s(t) - 1 = n_s\times (1 - t / T)$ , where  $n_{s}$  is the number of optimized iterations stored, and then select the integer part. We then can train the model as a weighted sum of the conditional DDPM objective and the trajectory alignment regularization:
$$
\mathcal {L} _ {\mathrm {D O M}} = \mathbb {E} _ {q (\mathbf {x} _ {t} | \mathbf {x} _ {0})} \left[ \mathbb {K L} \left[ q \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}, \mathbf {x} _ {0}\right) \mid p _ {\theta} \left(\mathbf {x} _ {t - 1} ^ {\theta} \mid \mathbf {x} _ {t}, \mathbf {c}\right) \right] + \mathbb {K L} \left[ q \left(\tilde {\mathbf {x}} ^ {\theta} \mid \mathbf {x} _ {t}, \epsilon_ {\theta}\right) \mid q \left(\mathbf {x} _ {s (t - 1)} \mid \operatorname {o p t}\right) \right] \right]. \tag {6}
$$
This mechanism effectively pushes the sampling trajectory at each step to match the optimization trajectory, distilling the optimizer during the reverse process training. In practice, following practice
1for example, if  $n_{s} = 5$  and  $T = 1000$ ,  $s = 1$  for  $(T:T - 200)$ , and  $s = 5$  for steps  $(200:1)$ .
in DDPM literature, the distribution variances are not learned from data. For the trajectory alignment distributions, we set the dispersion to the same values used in the model. By doing so we can rewrite the per-step negated lower-bound as a weighted sum of squared errors:
$$
\mathcal {L} _ {\mathrm {D O M}} = \underbrace {\mathbb {E} _ {q (\epsilon)} \left[ w _ {t} \| \epsilon_ {\theta} \left(\mathbf {x} _ {t} \left(\mathbf {x} _ {0} , \epsilon\right) , \mathbf {c}\right) - \epsilon \| _ {2} ^ {2} \right]} _ {L _ {t - 1, \epsilon} (\mathbf {x}, \mathbf {c})} + \underbrace {\alpha_ {c} \| \tilde {\mathbf {x}} ^ {\theta} \left(\mathbf {x} _ {t} , \epsilon_ {\theta}\right) - \mathbf {x} _ {s (t - 1)} ^ {o p t} \| _ {2} ^ {2}} _ {\mathcal {L} _ {\text {c l e a n}} ^ {\mathrm {T A}}} \tag {7}
$$
where  $\mathcal{L}_{clean}^{\mathrm{TA}}$  is the trajectory alignment loss for step  $t$ , and  $L_{t-1,\epsilon}(\mathbf{x},\mathbf{c})$  is a conditional DDPM loss for step  $t$ . This is the formulation employed for our model, where we optimize this loss for the mean values, freeze the mean representations, and optimize the variances in a separate step [68]. Alignment can also be performed in alternative ways. We can perform matching in noisy spaces, using the marginal posterior to obtain a noisy optimized representation for step  $t-1$ ,  $q(\mathbf{x}_{t-1}^{opt}|\mathbf{x}_{s(0)}^{opt})$  and then optimize  $\mathcal{L}_{noisy}^{\mathrm{TA}} = \alpha_n||\mathbf{x}_{t-1}^\theta - \mathbf{x}_{t-1}^{opt}||_2^2$ . Finally, we can match in performance space: this approach leverages an auxiliary model  $f_\phi$  similar to (consistency models) and performs trajectory alignment in functional space,  $\mathcal{L}_{perf}^{\mathrm{TA}} = \alpha_p||f_\phi(\mathbf{x}_{t-1}^\theta) - P_{s(t-1)}||_2$ , where we match the performance for the generated intermediate design with the ground truth intermediate performance  $P_{s(t-1)}$  for the optimized  $\mathbf{x}_{s(t-1)}^{opt}$ . We compare these and other variants in Table 6.
Dense Conditioning over Sparse Constraints. All models are subject to conditioning based on loads, boundary conditions, and volume fractions. In addition, TopoDiff and TopoDiff-GUIDED undergo conditioning based on force field and energy strain, while TopoDiff-FF and DOM are conditioned based on a dense kernel relaxation, inspired by Green's method [36, 34], which defines integral functions that are solutions to the time-invariant Poisson's Equation [33, 41]. More details are in Appendix D. The idea is to use the kernels as approximations to represent the effects of the boundary conditions and loads as smooth functions across the domain (Fig. 5). This approach avoids the need for computationally expensive and time-consuming Finite Element Analysis (FEA) to provide conditioning information. For a load or source  $l$ , a sink or boundary  $b$  and  $r = ||\mathbf{x} - \mathbf{x}_l||_2 = \sqrt{(\mathbf{x}_i - \mathbf{x}_i^l)^2 + (\mathbf{x}_j - \mathbf{x}_j^l)^2}$ , we have:
$$
\begin{array}{l} K _ {l} (\mathbf {x}, \mathbf {x} _ {l}; \alpha) = \sum_ {l = 1} ^ {L} \left(1 - e ^ {- \alpha / | | \mathbf {x} - \mathbf {x} _ {l} | | _ {2} ^ {2}}\right) \bar {p} (\mathbf {x} _ {l}) \\ K _ {b} (\mathbf {x}, \mathbf {x} _ {b}; \alpha) = \sum_ {b = 1} ^ {B} e ^ {- \alpha / | | \mathbf {x} - \mathbf {x} _ {b} | | _ {2} ^ {2}} / \max  _ {\mathbf {x}} \left(\sum_ {b = 1} ^ {B} e ^ {- \alpha / | | \mathbf {x} - \mathbf {x} _ {b} | | _ {2} ^ {2}}\right). \tag {8} \\ \end{array}
$$
where  $\bar{p}$  is the module of a generic force in 2D. Notice how, for  $r\to 0$ ,  $K_{l}(\mathbf{x},\mathbf{x}_{l})\rightarrow p$ , and  $r\to \infty$ ,  $K_{l}(\mathbf{x},\mathbf{x}_{l})\rightarrow 0$ . We notice how closer to the boundary the kernel is null, and farther from the boundary the kernel tends to 1. Note that the choice of  $\alpha$  parameters in the kernels affects the smoothness and range of the kernel functions. Furthermore, these kernels are isotropic, meaning that they do not depend on the direction in which they are applied. Overall, the kernel relaxation method offers a computationally inexpensive way to condition generative models on boundary conditions and loads, making them more applicable in practical engineering and design contexts.
Few-Steps Direct Optimization. Finally, we leverage direct optimization to improve the data-driven candidate generated by DOM. In particular, by running a few steps of optimization (5/10) we can inject physics information into the generated design directly, greatly increasing not only performance but greatly increasing manufacturability. Given a sample from the model  $\tilde{\mathbf{x}}_0\sim p_\theta (\mathbf{x}_0|\mathbf{x}_1)p_\theta (\mathbf{x}_{1:T})$ , we can post-process it and obtain  $\mathbf{x}_0 = \mathrm{opt}(\tilde{\mathbf{x}}_0^\theta ,n)$  an improved design leveraging  $n$  steps of optimization, where  $n\in [5,10]$ . In Fig. 3 we show a full pipeline for DOM.
Our three main objectives are: (1) Improving inference efficiency, and reducing the sampling time for diffusion-based topology generation while still satisfying the design requirements with a minimum decrease in performance. (2) Minimizing reliance on force and strain fields as conditioning information, reducing the computation burden at inference time and the need for ad-hoc conditioning mechanisms for each problem. (3) Merging together learning-based and optimization-based methods, refining the topology generated using a conditional diffusion model, and improving the final solution in terms of manufacturability and performance.
Setup. We train all the models for 200k steps on 30k optimized topologies on a 64x64 domain. For each optimized topology, we have access to a small subset (5 steps) of intermediate optimization
Figure 6: Few-Step sampling for Topology generation. Top row: Diffusion Optimization Model (DOM) with Trajectory Alignment. Middle row: TopoDiff-GUIDED. Bottom row: The optimization result. DOM produces high-quality designs in as few as two steps, greatly enhancing inference efficiency compared to previous models requiring 10-100 steps. Trajectory Alignment helps DOM generate near-optimal geometries swiftly, improving few-step sampling in conditional diffusion models for topology optimization. See appendix Fig. 13 for more examples.
steps. We set the hyperparameters, conditioning structure, and training routine as proposed in [62]. Appendix G for more details. For all the models (Table 1) we condition on volume fraction and loads. For TopoDiff, we condition additional stress and energy fields. For TopoDiff-FF [37], a variant of TopoDiff conditioning on a kernel relaxation, we condition on boundary conditions and kernels. TopoDiff-GUIDED leverages a compliance regressor and floating material classifier guidance. We use a reduced number of sampling steps for all the experiments.
Dataset. We use a dataset of optimized topologies gathered using SIMP as proposed in [66, 62]. Together with the topologies, the dataset contains information about optimal performance. For each topology, we have information about the loading condition, boundary condition, volume fraction, and optimal compliance. Additionally, for each constraint configuration, a pre-processing step computes the force and strain energy fields (see Fig. 5) when needed. Appendix F for more details on the dataset and visualizations.
We evaluate the model using engineering and generative metrics. In particular, we consider metrics that evaluate how well our model fulfills: physical constraints using error wrt prescribed Volume Fraction (VFE); engineering constraints, as manufacturability as measured by Floating Material (FM); performance constraints, as measured by compliance error (CE) wrt the optimized SIMP solution; sampling time constraints (inference constraints) as measure by sampling time (inference and pre-processing). We consider two scenarios of increasing complexity: (i) Indistribution Constraints. The constraints in this test set are the same as those of the training set. When measuring performance on this set, we filter generated configurations with high compliance. (ii) Out-of-distribution Constraints. The constraints in this test set are different from those of
Table 1: Comparative study of generative models in topology optimization considering factors like conditional input (COND), finite element method (FEM), and guidance (GUID). Unlike others, the DOM model operates without FEM preprocessing or GUIDANCE. More visualizations and optimization trajectories are in the Appendix.
w/ COND	w/o FEM	w/o GUI
TopologyGAN [69]	✓	X	X
TopoDiff [62]	✓	X	✓
TopoDiff-G [62]	✓	X	X
DOM (ours)	✓	✓	✓
the training set. When measuring performance on this set, we filter generated configurations with high compliance. The purpose of these tasks is to evaluate the generalization capability of the machine learning models in- and out-of-distribution. By testing the models on different test sets with varying levels of difficulty, we can assess how well the models can perform on new, unseen data. More importantly, we want to understand how important the role of the force field and energy strain is with unknown constraints.
In-Distribution Constraints. Table 3 reports the evaluation results in terms of constraints satisfaction and performance for the task of topology generation. In Table 2 we report metrics commonly employed to evaluate the quality of generative models in terms of fidelity (IS, sFID, P) and diversity (R). We see how such metrics are all close and it is challenging to gain any understanding just by relying on classic generative metrics when evaluating constrained design generation. These results justify the need for an evaluation that considers the performance and feasibility of the generated design.
In Table 3 DOM achieves high performance and is at least  $50\%$  less computationally expensive at inference time, not requiring FEM preprocessing or additional guidance through surrogate models like TopologyGAN and TopoDiff. We also compare with Consistency Models [104], a Diffusion Model that tries to predict its input at each step. DOM can be seen as a generalization of such a method when a trajectory is available as a ground truth. Overall, DOM with Trajectory Alignment is competitive or better than the previous proposal in terms of performance on in-distribution constraints, providing strong evidence that TA is an effective mechanism to guide the sampling path toward regions of high performance.
Table 2: Generative metrics on indistribution metrics. Precision denotes the fraction of generated topologies that are realistic, and recall measures the fraction of the training data manifold covered by the model.
IS ↑	sFID ↓	P ↑	R ↑
cDDPM	3.41	40.30	0.73	0.85
TopoDiff	3.57	36.20	0.80	0.86
TopoDiff-G	3.63	35.96	0.79	0.86
DOM w/o TA	3.48	37.54	0.76	0.86
DOM w/ TA	3.68	36.73	0.77	0.85
Table 3: Evaluation of different model variants on in-distribution constraints. CE: Compliance Error. VFE: Volume Fraction Error. FM: Floating Material. We use 100 sampling steps for all diffusion models. We can see that DOM w/ TA is competitive with the SOTA on topology generation, being computationally  $50\%$  less expensive at inference time compared to TopoDiff. Trajectory Alignment greatly improves performance without any additional inference cost. See appendix Fig. 11 for confidence intervals.
STEPS	CONSTRAINTS	AVG % CE ↓	MDN % CE ↓	% VFE ↓	% FM ↓	INFERENCE (s) ↓
TopologyGAN [69]	1	FIELD	48.51	2.06	11.87	46.78	3.37
Conditional DDPM [68]	100	RAW	60.79	3.15	1.72	8.72	2.23
Consistency Model [104]	100/1	KERNEL	10.30	2.20	1.64	8.72	2.35
TopoDiff-FF [37]	100	KERNEL	24.90	1.92	2.05	8.15	2.35
TopoDiff [62]	100	FIELD	5.46	0.80	1.47	5.79	5.54
TopoDiff-GUIDED [62]	100	FIELD	5.93	0.83	1.49	5.82	5.77
DOM w/o TA (ours)	100	KERNEL	13.61	1.79	1.86	7.44	2.35
DOM w/ TA (ours)	100	KERNEL	4.44	0.74	1.52	6.72	2.35
Table 4: Evaluating sampling topologies with few steps (2-10) for TopoDiff and DOM. G: Guided using regression and classifier guidance. AVG CE: average compliance error. MDN CE: median compliance error. VFE: volume fraction error. FM: floating material. INF: inference time. UNS: unsolvable configurations. LD: load disrespect. DOM largely outperforms TopoDiff in the few sampling step regimes, showing Trajectory Alignment's effectiveness as a grounding mechanism. DOM can generate reasonable topologies in just two sampling steps, where TopoDiff and DOM w/ TA fail completely, even presenting cases of load disrespect.
in-distro	STEPS	SIZE	AVG % CE ↓	MDN % CE ↓	% VFE ↓	% FM ↓	INF (s) ↓	% UNS ↓	% LD ↓
TopoDiff-G	2	239M	681.53	436.83	80.98	98.72	3.36	2.00	15.92
DOM (ours)	2	121M	22.66	1.46	3.34	33.25	0.17 (-94.94 %)	2.11	0.00
TopoDiff-G	5	239M	43.27	15.48	2.76	77.65	3.43	1.44	0.00
DOM (ours)	5	121M	11.99	0.72	2.27	20.08	0.24 (-93.00 %)	2.77	0.00
TopoDiff-G	10	239M	6.43	1.61	1.95	20.55	3.56	0.00	0.00
DOM (ours)	10	121M	4.44	0.57	1.67	11.94	0.35 (-90.17 %)	0.00	0.00
out-distro
TopoDiff-G	2	239M	751.17	548.26	81.46	100.00	3.36	1.90	16.48
DOM (ours)	2	121M	79.66	10.37	3.69	44.20	0.17 (-94.94 %)	2.80	0.00
TopoDiff-G	5	239M	43.50	19.24	2.58	79.57	3.43	2.20	0.00
DOM (ours)	5	121M	38.97	5.49	2.56	26.70	0.24 (-93.00 %)	1.40	0.00
TopoDiff-G	10	239M	10.78	2.55	1.87	21.36	3.56	2.10	0.00
DOM (ours)	10	121M	32.19	3.69	1.78	14.20	0.35 (-90.17 %)	0.40	0.00
Generation with Few-Steps of Sampling. Table 4 compares two different algorithms, TopoDiff-GUIDED and DOM, in terms of their performance when using only a few steps for sampling. The table shows the results of the in and out-of-distribution comparison, with TopoDiff-G and DOM both having STEPS values of 2, 5, and 10, and SIZE of 239M and 121M. We can see that DOM outperforms by a large margin TopoDiff-G when tasked with generating a new topology given a few steps, corroborating our hypothesis that aligning the sampling and optimization trajectory is an effective mechanism to obtain efficient generative models that satisfy constraints. DOM outperforms
TopoDiff-GUIDED even being  $50\%$  smaller, without leveraging an expensive FEM solver for conditioning but relying on cheap dense relaxations, making it 20/10 times faster at sampling, and greatly enhancing the quality of the generated designs, providing evidence that Trajectory Alignment is an effective mechanism to distill information from the optimization path. In Fig. 6, we provide qualitative results to show how DOM (top row) is able to generate reasonable topologies, resembling the fully optimized structure running SIMP for 100 steps (bottom row), with just two steps at inference time, where the same model without TA or a TopoDiff are not able to perform such task. Overall these results corroborate our thesis regarding the usefulness of trajectory alignment for high-quality constrained generation. For more experiments on inference see Appendix B.
Merging Generative Models and Optimization for Out-of-Distribution Constraints.
Table 5 shows the results of experiments on out-of-distribution constraints. In this scenario, employing FEM and guidance significantly enhances the performance of TopoDiff. Conditioning on the FEM output during inference can be seen as a form of test-time conditioning that can be adapted to the sample at hand. However, merging DOM and a few iterations of optimization is extremely effective in solving this problem, in particular in terms of improving volume fraction and floating material. Using the combination of DOM and SIMP is a way to impose the performance constraints in the model without the need for surrogate models or guidance.
Trajectory Alignment Ablation. The core contribution of DOM is trajectory alignment, a method to match sampling and optimization trajectories of arbitrary length and structure mapping intermediate steps to appropriate CLEAN (noise free or with reduced noise using the model and the marginalization properties of DDPM) representations. However, alignment can be performed in multiple ways, leveraging NOISY representation, matching performance (PERF), and using data at a higher resolution to impose consistency (MULTI). In Table 6 we perform an ablation study, considering DOM with and without kernel relaxation, and leveraging different kinds of trajectory matching. From the
Table 5: Out-of-Distribution Scenario Comparison: TopoDiff-G outperforms DOM due to its adaptive conditioning mechanism, which leverages expensive FEM-computed fields. However, DOM coupled with a few steps of direct optimization (5/10) greatly surpasses TopoDiff in performance and manufacturability. This underscores the effectiveness of integrating data-driven and optimization methods in constrained design creation.
STEPS	MDN % CE ↓	% VFE ↓	% FM ↓
TopoDiff-FF	100	16.06	1.97	8.38
TopoDiff-G	100	1.82	1.80	6.21
DOM	100	3.47	1.59	8.02
DOM + SIMP	100+5	1.89	1.77	10.19
DOM + SIMP	100+10	1.15	1.10	2.61
Table 6: Ablation study with and without kernel and trajectory alignment. We explore different ways to match the sampling and optimization trajectory and we measure the Median Compliance Error. TA: trajectory alignment. CM: Consistency Models [104].
KERNEL	TA	MODE	IN-DISTRO	OUT-DISTRO
DOM	X	X	-	3.29	8.05
DOM	X	✓	CLEAN	1.11	9.01
CM	✓	X	-	2.20	5.25
DOM	✓	X	-	1.80	5.62
DOM	✓	✓	MULTI	34.95	54.73
DOM	✓	✓	NOISY	2.08	6.23
DOM	✓	✓	PERF	2.41	6.82
DOM	✓	✓	CLEAN	0.74	3.47
table, we see that using dense conditioning is extremely important for out-of-distribution performance, and that matching using CLEAN is the most effective method in and out-of-distribution. In Fig. 4 we report a visualization of the distance between sampling and optimization trajectory during training. From this plot, we can see how the kernel together with TA helps the model to find trajectories that are closer to the optimal one, again corroborating the need for dense conditioning and consistency regularization.
We presented Diffusion Optimization Models, a generative framework to align the sampling trajectory with the underlying physical process and learn an efficient and expressive generative model for constrained engineering design. Our work opens new avenues for improving generative design in engineering and related fields. However, our method is limited by the capacity to store and retrieve intermediate optimization steps, and, without a few steps of optimization, it underperforms out-of-distribution compared to FEM-conditional and guided methods.
We would like to thank Noah Joseph Bagazinski, Kristen Marie Edwards, Amin Heyrani Nobari, Cyril Picard, Lyle Regenwetter and Binyang Song for insightful comments and useful discussions. We extend our gratitude to the anonymous reviewers for their constructive feedback and discussions that greatly enhanced the clarity and quality of the paper. OW's work was funded in part by the Novo Nordisk Foundation through the Center for Basic Machine Learning Research in Life Science (NNF20OC0062606) and by the Pioneer Centre for AI, DNRF grant number P1.
[107] I. Sosnovik and I. Oseledets. Neural networks for topology optimization. Preprint, 9 2017.
[108] A. Srivastava, L. Valkov, C. Russell, M. U. Gutmann, and C. Sutton. Veegan: Reducing mode collapse in gans using implicit variational learning. Advances in neural information processing systems, 30, 2017.
[109] H. Sun and L. Ma. Generative design by using exploration approaches of reinforcement learning in density-based structural topology optimization. *Designs*, 4(2):10, 5 2020.
[110] E. Ulu, R. Zhang, and L. B. Kara. A data-driven investigation and estimation of optimal topologies under variable loading configurations. Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, 4(2):61-72, 3 2016.
[111] M. Y. Wang, X. Wang, and D. Guo. A level set method for structural topology optimization. Computer Methods in Applied Mechanics and Engineering, 192(1-2):227-246, 1 2003.
[112] D. Watson, J. Ho, M. Norouzi, and W. Chan. Learning to efficiently sample from diffusion probabilistic models. arXiv preprint arXiv:2106.03802, 2021.
[113] R. V. Woldseth, N. Aage, J. A. Barentzen, and O. Sigmund. On the use of artificial neural networks in topology optimisation. Structural and Multidisciplinary Optimization, 65(10):294, 2022.
[114] Q. Wu, C. Yang, W. Zhao, Y. He, D. Wipf, and J. Yan. Differmer: Scalable (graph) transformers induced by energy constrained diffusion. arXiv preprint arXiv:2301.09474, 2023.
[115] Y. M. Xie, G. P. Steven, Y. Xie, and G. Steven. Basic evolutionary structural optimization. Springer, 1997.
[116] M. Xu, L. Yu, Y. Song, C. Shi, S. Ermon, and J. Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923, 2022.
[117] S. Xu, Y. Cai, and G. Cheng. Volume preserving nonlinear density filter based on heaviside functions. Structural and Multidisciplinary Optimization, 41:495-505, 2010.
[118] L. Xue, J. Liu, G. Wen, and H. Wang. Efficient, high-resolution topology optimization method based on convolutional neural networks. Frontiers of Mechanical Engineering, 16(1):80-96, 3 2021.
[119] A. Yildiz, N. Öztürk, N. Kaya, and F. Öztürk. Integrated optimal topology design and shape optimization using neural networks. Structural and Multidisciplinary Optimization, 25(4):251-260, 10 2003.
[120] S. Yoo, S. Lee, S. Kim, K. H. Hwang, J. H. Park, and N. Kang. Integrating deep learning into cad/cae system: generative design and evaluation of 3d conceptual wheel. Structural and Multidisciplinary Optimization, 64(4):2725-2747, 10 2021.
[121] Y. Yu, T. Hur, J. Jung, and I. G. Jang. Deep learning for determining a near-optimal topological design without any iteration. Structural and Multidisciplinary Optimization, 59(3):787-799, 2019.
[122] J.-Y. Zhu, T. Park, P. Isola, and A. A. Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pages 2223-2232, 2017.