SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Topology Optimization. Engineering design is the process of creating solutions to technical problems under engineering requirements [22, 91, 23]. Often, the goal is to create highly performative designs given the required constraints. Topology Optimization (TO [14]) is a branch of engineering design and is a critical component of the design process in many industries, including aerospace, automotive, manufacturing, and software development. From the inception of the homogenization method for TO, a number of different approaches have been proposed, including density-based [13, 86, 64], level-set [3, 111], derivative-based [99], evolutionary [115], and others [19]. The density-based methods are widely used and use a nodal-based representation where the level-set leverages shape derivative to obtain the optimal topology. To improve the final design, filtering mechanisms have been proposed [117, 39, 94]. Hybrid methods are also widely used. Topology Optimization has evolved as a more and more intensive computational discipline, with the availability of efficient open-source implementations [92, 93, 46, 59, 4]. See [59] for more on this topic. See [95, 96] for a comprehensive review of the Topology Optimization field.
Generative Models for Topology Optimization. Following the success of Deep Learning (DL) in vision, a surging interest arose recently for transferring these methods to the engineering field. In particular, DL methods have been employed for direct-design [1, 5, 11, 60, 110], accelerating the optimization process [7, 49, 52, 107, 118], improving the shape optimization post-processing [42, 119], super-resolution [32, 65, 120], sensitivity analysis [6, 75, 10, 89], 3d topologies [54, 87, 11], and more [26, 24, 25, 29]. Among these methods, Generative Models are especially appealing to improve design diversity in engineering design [2, 72, 70, 71]. In TO, the work of [47, 82, 83, 55, 109] focus on increasing diversity leveraging data-driven approaches. Additionally, Generative Models have been used for Topology Optimization problems conditioning on constraints (loads, boundary conditions, volume fraction for the structural case), directly generating topologies [81, 90, 40] training dataset of optimized topologies, leveraging superresolution methods to improve fidelity [121, 58], using filtering and iterative design approaches [74, 73, 17, 35] to improve quality and diversity. Methods for 3D topologies have also been proposed [12, 55]. Recently, GAN [69] and DDPM-based [62] approaches, conditioning on constraints and physical information, have had success in modeling the TO problem. For a comprehensive review and critique of the field, see [113].
Conditional Diffusion Models. Methods to condition DDPM have been proposed, conditioning at sampling time [28], learning a class-conditional score [106], explicitly conditioning on class information [68], features [38, 9], and physical properties [116, 45]. Recently, TopoDiff [62] has shown that conditional diffusion models with guidance [31] are effective for generating topologies that fulfill the constraints and have high manufacturability and high performance. TopoDiff relies on physics information and surrogate models to guide the sampling of novel topologies with good performance. Alternatives to speed up sampling in TopoDiff have been recently proposed [37], trading performance for fast candidate generation. Improving efficiency and sampling speed for diffusion models is an active research topic, both reducing the number of sampling steps [68, 103, 102, 8], improving the ODE solver [48, 112, 57], leveraging distillation [88, 63], and exploiting autoencoders for dimensionality reduction [85, 77, 76]. Reducing the number of sampling steps can also be achieved by improving the property of the hierarchical latent space, exploiting a form of consistency regularization [108, 97, 122] during training. Consistency Models [104] proposes reconstructing its input from any step in the diffusion chain, effectively forcing the model to reduce the sampling steps needed for high-quality sampling. We similarly want to improve the latent space properties but leverage trajectory alignment with a physical process. Recently, energy-constrained diffusion models [114] have been proposed to regularize graph learning a learn expressive representations for structured data.
B.1 Sampling Topologies in Few-Steps
Table 7: Evaluating sampling topologies with few steps (2-10) for TopoDiff and DOM. G: Guided using regression and classifier guidance. AVG CE: average compliance error. MDN CE: median compliance error. VFE: volume fraction error. FM: floating material. INF: inference time. UNS: unsolvable configurations. LD: load disrespect.
in-distro	STEPS	SIZE	AVG % CE ↓	MDN % CE ↓	% VFE ↓	% FM ↓	INF (s) ↓	% UNS ↓	% LD ↓
TopoDiff-G	2	239M	681.53	436.83	80.98	98.72	3.36	2.00	15.92
DOM (ours)	2	121M	22.66	1.46	3.34	33.25	0.17 (-94.94 %)	2.11	0.00
TopoDiff-G	5	239M	43.27	15.48	2.76	77.65	3.43	1.44	0.00
DOM (ours)	5	121M	11.99	0.72	2.27	20.08	0.24 (-93.00 %)	2.77	0.00
TopoDiff-G	10	239M	6.43	1.61	1.95	20.55	3.56	0.00	0.00
DOM (ours)	10	121M	4.44	0.57	1.67	11.94	0.35 (-90.17 %)	0.00	0.00
out-distro
TopoDiff-G	2	239M	751.17	548.26	81.46	100.00	3.36	1.90	16.48
DOM (ours)	2	121M	79.66	10.37	3.69	44.20	0.17 (-94.94 %)	2.80	0.00
TopoDiff-G	5	239M	43.50	19.24	2.58	79.57	3.43	2.20	0.00
DOM (ours)	5	121M	38.97	5.49	2.56	26.70	0.24 (-93.00 %)	1.40	0.00
TopoDiff-G	10	239M	10.78	2.55	1.87	21.36	3.56	2.10	0.00
DOM (ours)	10	121M	32.19	3.69	1.78	14.20	0.35 (-90.17 %)	0.40	0.00
B.2 Inference Time
With the previous experiments, we proved that a data-driven approach, biased towards the physical process, can distill the optimization process and sample a novel topology in a few steps. We then provide experiments for in and out-of-distribution and ablate our choice of kernel relaxation and trajectory alignment mechanism. However, the final goal of data-driven design is to learn a general tool for fast candidate generation. In Table 8 we compare inference time for different models and, more importantly, for low- (64) and high-resolution (256). Given the computational burden of training such models at high resolution, we train all the models for only 10k steps (around  $5\%$  of the full training), and then we use them for sampling. With this experiment, we want to emphasize how fast DOM can perform inference compared to a SOTA model. We choose to run all the generative models and optimized for 100 steps: this setting is not suited for DOM, because as we have seen the model excels in the few-step sampling task. Table 8 presents a comparison of various models based on different factors such as resolution (RES), size, preprocess time, postprocess time, sampling, and inference time.
Table 8: Inference time for different models at low and high resolution. For all the diffusion models we sample 100 steps and for SIMP we iterate for 100 steps. *We report optimization time for a full comparison, but it is important to emphasize that SIMP runs on CPU and the DDPM-based models on GPU.  $\Delta T_{\text{inference}} = (T_{\text{model}} - T_{\text{topodiff}}) / T_{\text{topodiff}}$
RES	SIZE	PREPROCESS	POSTPROCESS	SAMPLING	INFERENCE	ΔTinference
TopoDiff	64	121M	3.31	0.00	2.23	5.54	+0.00 %
TopoDiff-GUIDED	64	239M	3.31	0.00	2.46	5.77	+4.15 %
DOM (ours)	64	121M	0.12	0.00	2.23	2.35	-57.58 %
SIMP	64	-	0.00	18.12	0.00	18.12	+227.07 %*
TopoDiff	256	553M	31.84	0.00	7.78	38.62	+0.00 %
TopoDiff-GUIDED	256	1092M	31.84	0.00	8.46	40.30	+4.35 %
DOM (ours)	256	553M	0.31	0.00	7.78	8.09	-79.05 %
SIMP	256	-	0.00	316.02	0.00	316.02	+718.28 %*
Code: https://github.com/georgosgeorgos/trajectory-alignment-diffusion
C Algorithms
Algorithm 1 DOM with Trajectory Alignment
Require: Optimized Topologies  $\mathbf{X}_0$    
Require: Constraints  $C = (BC,L,VF)$    
Require: Intermediate Optimization Steps  $\mathbf{X}^{opt}$  while Training do Sample batch  $(\mathbf{x}_0,c,\mathbf{x}^{opt})$  Compute Dense Relaxation  $k = K(bc,l)$  Compute Conditioning  $\mathbf{c} = (k,c)$  Sample  $t,\epsilon ,\mathbf{x}_{s(t)}^{opt}$  Compute  $\mathbf{x}_t\sim q(\mathbf{x}_t|\mathbf{x}_0)$  Forward Model  $\epsilon_{\theta}(\mathbf{x}_t,\mathbf{c})$  Compute Loss  $L_{t - 1}(\mathbf{x},\mathbf{c}) = ||\epsilon_{\theta}(\mathbf{x}_t,\mathbf{c}) - \epsilon ||_2^2$  Trajectory Search  $\tilde{\mathbf{x}}^{\theta}(\mathbf{x}_t,\epsilon_\theta) = (\mathbf{x}_t - \sqrt{1 - \bar{\alpha}_t}\epsilon_\theta (\mathbf{x}_t,\mathbf{c})) / \sqrt{\bar{\alpha}_t}$  Trajectory Matching  $\mathcal{L}_{\mathrm{TA}} = ||\tilde{\mathbf{x}}^{\theta}(\mathbf{x}_t,\epsilon_\theta) - \mathbf{x}_{s(t)}^{opt}||_2^2$  Compute Loss  $\mathcal{L}_{\mathtt{DOM}}(\theta) = L_{t - 1}(\mathbf{x},\mathbf{c}) + \mathcal{L}_{\mathtt{TA}}$  Backpropagate  $\theta \gets \nabla_{\theta}\mathcal{L}_{\mathtt{DOM}}(\theta)$    
end while
Algorithm 2 DOM without Trajectory Alignment
Require: Optimized Topologies  $\mathbf{X}_0$    
Require: Constraints  $C = (BC,L,VF)$    
while Training do Sample batch  $(\mathbf{x}_0,c)$  Compute Dense Relaxation  $k = K(bc,l)$  Compute Conditioning  $\mathbf{c} = (k,c)$  Sample  $t,\epsilon$  Compute  $\mathbf{x}_t\sim q(\mathbf{x}_t|\mathbf{x}_0)$  Forward Model  $\epsilon_{\theta}(\mathbf{x}_t,\mathbf{c})$  Compute Loss  $L_{t - 1}(\mathbf{x},\mathbf{c}) = ||\epsilon_{\theta}(\mathbf{x}_t,\mathbf{c}) - \epsilon ||_2^2$  Compute Loss  $\mathcal{L}_{\mathtt{DOM}}(\theta) = L_{t - 1}(\mathbf{x},\mathbf{c})$  Backpropagate  $\theta \gets \nabla_{\theta}\mathcal{L}_{\mathtt{DOM}}(\theta)$    
end while
import numpy as np   
import torch as th   
def compute_kernel_load(batch_load_sample, axis):   
size  $=$  batch_load_sample.size(-1) if axis  $= =$  "x": ix  $= 0$  xx  $=$  th.argmax(batch_load_sample[0] != 0) coord  $=$  xx elif axis  $= =$  "y": ix  $= 1$  yy  $=$  th.argmax(batch_load_sample[1] != 0) coord  $=$  yy if len(coord)  $= = 0$  return batch_load_sample[ix], []   
x_grid  $=$  th:tensor([i for i in range(size)]) y_grid  $=$  th:tensor([j for j in range(size)])   
kernel_load  $= 0$  for l in range(len(coord)): x_grid  $=$  th:tensor([i for i in range(size)]) y_grid  $=$  th:tensor([j for j in range(size)]) #distance x_grid  $=$  x_grid - coord[1][0] y_grid  $=$  y_grid - coord[1][1] grid  $=$  thmeshgrid(x_grid,y_grid)   
r_load  $=$  th.sqrt(grid[0]**2+grid[1]**2) if axis  $= =$  "x": p  $=$  batch_load_sample[0][coord[1][0],coord[1][1]] elif axis  $= =$  "y": p  $=$  batch_load_sample[1][coord[1][0],coord[1][1]] kernel  $= 1$  -th.exp(-1/r_load**2) kernel_load  $+ =$  kernel \*p   
return kernel_load,coord
Listing 1: Dense Kernel Relaxation for Sparse Loads.
D Physics-based Conditioning on Constraints
Seeking a more efficient diffusion-based topology generation while reducing dependency on force and strain fields, our approach is to approximate boundary conditions and loads using kernels. These kernels approximate the impact of constraints on the domain. The kernel architecture we opt for draws inspiration from Green's method [36, 34, 16, 41]. This method establishes integral functions as solutions to the time-invariant Poisson's Equation [33, 41], an extended form of Laplace's Equation addressing point source excitations. Expressed mathematically, Poisson's Equation is given as  $\nabla_x^2 f(x) = h$ , with  $h$  as the force term and  $f$  representing a general function over domain  $\mathcal{X}$ . This equation underpins numerous natural phenomena, with a notable case being a force part  $h = 0$ . This particular case brings forth Laplace's Equation, which is frequently used in heat transfer scenarios.
Green's method, part of a larger family of methods, offers a structured approach to address partial differential equations, even when domain specifics are unknown. The solutions derived using this method are termed Green's functions [53]. Even though such solutions can be generally complex, for a broad range of physical issues where constraints and forces are point-approximated, a straightforward functional framework can be established, hinging on the source and sink concept.
Consider, for instance, a laminar domain like a beam or a plate, restrained in a feasible manner. If a point force is applied (such as a downward pressure on a beam's edge or a plate's center) at  $x_{f}$ , this can be captured using the Dirac delta function,  $\delta(x - x_{f})$ . The delta function is highly discontinuous but has powerful integration properties, specifically  $\int f(x) \delta(x - x_{f}) dx = f(x_{f})$  within domain  $\mathcal{X}$ . The solution for the time-invariant Poisson's Equation, when focusing on pinpoint forces, can be represented as a Green's function solution. This solution is primarily influenced by the distance from where the force is applied. Specifically:
$$
\mathcal {G} \left(x, x ^ {\prime}\right) = - \frac {1}{4 \pi} \frac {1}{| x - x ^ {\prime} |}, \tag {9}
$$
where  $r = |x - x'| = \sqrt{|x_i - x_i'|^2 + |x_j - x_j'|^2}$ . To approximate the forces and loads impacting our topologies, we utilize a kernel approximation grounded in Green's functions. While this model may not precisely map to all loads and boundary conditions, it does furnish us with computationally efficient conditioning data that aligns with core physical and engineering constraints. Harnessing these concepts, we aspire to enrich the conditioning data for the model, aiming to elevate generative models constrained by such conditions.
Table 9: Design and Modelling requirements for a constrained generative model for topology optimization. Our goal is to improve the requirements that are challenging to fulfill. In this work, we focus on improving Floating Material, reducing Compliance Error, and reducing Sampling Time.
CLASS	METRICS	GOAL
Hard-constraint	Loads Disrespect	Feasibility
Hard-constraint	Floating Material	Manufacturability
Soft-constraint	Volume Fraction	Min Cost
Functional Performance	Compliance Error	Max Performance
Modeling Requirements	Sampling Time	Fast Inference
E Tables and Visualizations
(a) Ratio of generated samples with  $\mathrm{CE} < t$
(b) Ratio of the training set with compliance  $< t$ .
(c) Manufacturability vs Performance.
(d) Distance sampling and optimization path.
Table 10: Conditioning or guiding variables for different optimization methods and model configurations.
Load	BC	Kernel Load	Kernel BC	Force Field	Energy Field	VF	Performance
SIMP [14]	✓	✓	X	X	X	X	✓	✓
TopologyGAN [69]	✓	✓	X	X	✓	✓	✓	X
TopoDiff [62]	✓	✓	X	X	✓	✓	✓	✓
TopoDiff-FF [37]	✓	✓	✓	✓	X	X	✓	X
DOM (ours)	✓	✓	✓	✓	X	X	✓	X
DOM + Trajectory Alignment (ours)	✓	✓	✓	✓	X	X	✓	✓
DOM + Optimizer (ours)	✓	✓	✓	✓	X	X	✓	✓
Figure 8: Histogram empirical distribution compliance for generated and optimized topologies.
Figure 9: DOM Sampling. Conditional Samples from a Diffusion Optimization Model trained with Trajectory Alignment. For each row, we select a constraint configuration (boundary condition, loads, volume fraction) and sample the model 10 times. We use 100 sampling steps. We repeat this process 12 times. During sampling, DOM is a standard conditional DDPM and does not have access to the optimization trajectory.
Figure 10: DOM Sampling Process at Inference Time. DOM (left) and TopoDiff-Guided (right) sampling process during inference. DOM is a conditional diffusion model. DOM is conditioned on the boundary conditions, loads, volume fraction, and kernels. After conditioning, we sample DOM with standard ancestral sampling like any DDPM. Contrary to TopoDiff (right), DOM does not rely on expensive preprocessing to compute energy and strain fields and auxiliary models for guidance. DOM can generate samples in a few steps (2/5) contrary to TopoDiff-G which requires hundreds of steps to generate reasonable topologies.
Figure 11: Confidence interval for design requirements on in-distribution constraint configurations.
Figure 12: Examples of generated topologies with good performance.
Figure 13: Comparison DOM w/ TA and DOM w/o TA generation with 2 steps.
Figure 14: Comparison DOM w/ TA and TopoDiff-GUIDED generation with 2 steps.
Figure 15: Comparison DOM and TopoDiff-GUIDed generation with 100 steps.
Figure 16: Left:  $\mathbf{x}_t^\theta$  from 10 intermediate generation steps for DOM w/ TA (top block), DOM w/o TA (middle block), SIMP iterations (bottom block). Right: Prediction of  $\tilde{\mathbf{x}}^{\theta}$  from 10 intermediate generation steps for DOM w/ TA (top block), DOM w/o TA (middle block), SIMP iterations (bottom block).
F Dataset
We build a dataset of optimized topologies and intermediate optimization steps at low-resolution (64x64) and high-resolution (256x256). In particular:
- 50K low-resolution optimized topologies w/ constraints.
- 60K high-resolution optimizer topologies w/ constraints.
- 250K low-resolution intermediate steps [10, 20, 30, 50, 70] w/ constraints.
- 300K high-resolution intermediate steps [10, 20, 30, 50, 70] w/ constraints.
In Figure 17, 18, 19, 20 we show examples of intermediate steps at 10, 20, 30, and optimized topologies at high-resolution.
Training Topologies
Figure 17: Intermediate optimization output after 10 iterations. Resolution: 256x256.
Training Topologies
Figure 18: Intermediate optimization output after 20 iterations. Resolution: 256x256.
Training Topologies
Figure 19: Intermediate optimization output after 30 iterations. Resolution: 256x256.
Training Topologies
Figure 20: Optimized output. Resolution:  $256 \times 256$ .
Table 11: Relevant Hyperparameters for baselines and DOM on 64x64 datasets.
TopoDiff	TopoDiff-G	DOM w/o TA	DOM w/ TA
Dimension	1x64x64	1x64x64	1x64x64	1x64x64
Model Set	30k	30K	30K	30K
Guidance Set	-	150K	-	-
Intermediate Set	-	-	-	150K
Test Configurations	1800	1800	1800	1800
Batch size	64	64	64	64
Architecture	Unet	Unet	Unet	Unet
Iterations	200K	200K	200K	200K
Learning rate	\( 2e^{-4} \)	\( 2e^{-4} \)	\( 2e^{-4} \)	\( 2e^{-4} \)
Loss	\( L_{\epsilon} \)	\( L_{\epsilon} \)	\( L_{\epsilon} \)	\( L_{\epsilon} + L_{TA} \)
Optimizer	Adam	Adam	Adam	Adam