SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Tsun-Hsuan Wang $^{1,*}$ , Juntian Zheng $^{2,3}$ , Pingchuan Ma $^{1}$ , Yilun Du $^{1}$ , Byungchul Kim $^{1}$ , Andrew Spielberg $^{1,4}$ , Joshua B. Tenenbaum $^{1}$ , Chuang Gan $^{1,3,5,\dagger}$ , Daniela Rus $^{1,\dagger}$ ,  $^{1}$ MIT,  $^{2}$ Tsinghua University,  $^{3}$ MIT-IBM Watson AI Lab,  $^{4}$ Harvard,  $^{5}$ UMass Amherst https://diffusebot.github.io/
Nature evolves creatures with a high complexity of morphological and behavioral intelligence, meanwhile computational methods lag in approaching that diversity and efficacy. Co-optimization of artificial creatures' morphology and control in silico shows promise for applications in physical soft robotics and virtual character creation; such approaches, however, require developing new learning algorithms that can reason about function atop pure structure. In this paper, we present DiffuseBot, a physics-augmented diffusion model that generates soft robot morphologies capable of excelling in a wide spectrum of tasks. DiffuseBot bridges the gap between virtually generated content and physical utility by  $(i)$  augmenting the diffusion process with a physical dynamical simulation which provides a certificate of performance, and  $ii$  introducing a co-design procedure that jointly optimizes physical design and control by leveraging information about physical sensitivities from differentiable simulation. We showcase a range of simulated and fabricated robots along with their capabilities.
Designing dynamical virtual creatures or real-world cyberphysical systems requires reasoning about complex trade-offs in system geometry, components, and behavior. But, what if designing such systems could be made simpler, or even automated wholesale from high-level functional specifications? Freed to focus on higher-level tasks, engineers could explore, prototype, and iterate more quickly, focusing more on understanding the problem, and find novel, more performant designs. We present DiffuseBot , a first step toward efficient automatic robotic and virtual creature content creation, as an attempt at closing the stubborn gap between the wide diversity and capability of Nature vis-a-vis evolution, and the reiterative quality of modern soft robotics.
Specifically, we leverage diffusion-based algorithms as a means of efficiently and generatively co-designing soft robot morphology and control for target tasks. Compared with with previous approaches, DiffuseBot's learning-based approach maintains evolutionary algorithms' ability to search over diverse forms while exploiting the efficient nature of gradient-based optimization. DiffuseBot is made possible by the revolutionary progress of AI-driven content generation, which is now able to synthesize convincing media such as images, audio, and animations, conditioned on human input. However, other than raw statistical modeling, these methods are typically task- and physics-oblivious, and tend to provide no fundamental reasoning about the performance of generated outputs. We provide the first method for bridging the gap between diffusion processes and the morphological design of cyberphysical systems, guided by physical simulation, enabling the computational creative design of virtual and physical creatures.
*This work was supported by the NSF EFRI Program (Grant No. 1830901), DSO grant DSOCO2107, DARPA Fellowship Grant HR00112110007, and MIT-IBM Watson AI Lab. † indicates equal advising.
Figure 1: DiffuseBot aims to augment diffusion models with physical utility and designs for high-level functional specifications including robot geometry, material stiffness, and actuator placement.
While diffusion methods can robustly sample objects with coherent spatial structure from raw noise in a step-by-step fashion, several roadblocks preventing existing generative algorithms from being directly applied to physical soft robot co-design. First, while existing diffusion methods can generate 2D or 3D shapes, useful for, say, sampling robot geometry, they do not consider physics, nor are they directly aligned with the robotic task performance. As an alternative approach, one might consider learning a diffusion model supervised directly on a dataset of highly-performant robot designs mapped to their task performance. This leads us to the second roadblock, that is, that no such dataset exists, and, more crucially, that curating such a dataset would require a prohibitive amount of human effort and would fail to transfer to novel tasks outside that dataset.
To tackle these challenges, we propose using physical simulation to guide the generative process of pretrained large-scale 3D diffusion models. Diffusion models pretrained for 3D shapes provide an expressive base distribution that can effectively propose reasonable candidate geometries for soft robots. Next, we develop an automatic procedure to convert raw 3D geometry to a representation compatible with soft body simulation, i.e. one that parameterizes actuator placement and specifies material stiffness. Finally, in order to sample robots in a physics-aware and performance-driven manner, we apply two methods that leverage physically based simulation. First, we optimize the embeddings that condition the diffusion model, skewing the sampling distribution toward better-performing robots as evaluated by our simulator. Second, we reformulate the sampling process that incorporates co-optimization over structure and control. We showcase the proposed approach of DiffuseBot by demonstrating automatically synthesized, novel robot designs for a wide spectrum of tasks, including balancing, landing, crawling, hurdle, gripping, and moving objects, and demonstrate its superiority to comparable approaches. We further demonstrate DiffuseBot's amenability to incorporating human semantic input as part of the robot generation process. Finally, we demonstrate the physical realizability of the robots generated by DiffuseBot with a proof-of-concept 3D printed real-world robot, introducing the possibility of AI-powered end-to-end CAD-CAM pipelines.
In summary, we contribute:
- A new framework that augments the diffusion-based synthesis with physical dynamical simulation in order to generatively co-design task-driven soft robots in morphology and control.
- Methods for driving robot generation in a task-driven way toward improved physical utility by optimizing input embeddings and incorporating differentiable physics into the diffusion process.
- Extensive experiments in simulation to verify the effectiveness of DiffuseBot, extensions to text-conditioned functional robot design, and a proof-of-concept physical robot as a real-world result.
In this section, we first formulate the problem (Section 2.1) and then describe the proposed Diffuse-Bot framework, which consists of diffusion-based 3D shape generation (Section 2.2), a differentiable procedure that converts samples from the diffusion models into soft robots (Section 2.3), a technique to optimize embeddings conditioned by the diffusion model for improved physical utility (Section 2.4), and a reformulation of diffusion process into co-design optimization (Section 2.4).
2.1 Problem Formulation: Soft Robot Co-design
Soft robot co-design refers to a joint optimization of the morphology and control of soft robots. The morphology commonly involves robot geometry, body stiffness, and actuator placement. Control is the signal to the actuators prescribed by a given robot morphology. It can be formally defined as,
Figure 2: The DiffuseBot framework consists of three modules: (i) robotizing, which converts diffusion samples into physically simulatable soft robot designs (ii) embedding optimization, which iteratively generate new robots to be evaluated for training the conditional embedding (iii) diffusion as co-design, which guides the sampling process with co-design gradients from differentiable simulation. Arrow (A): evaluation of robots to guide data distribution. (B): differentiable physics as feedback.
$$
\min  _ {\Psi , \phi} \mathcal {L} (\Psi , \phi) = \min  _ {\Psi , \phi} \mathcal {L} \left(\left\{\mathbf {u} _ {h} \left(\mathbf {s} _ {h}; \phi , \Psi\right), \mathbf {s} _ {h} \right\} _ {h \in [ 1, H ]}\right), \quad \text {w h e r e} \mathbf {s} _ {h + 1} = f \left(\mathbf {s} _ {h}, \mathbf {u} _ {h}\right) \tag {1}
$$
where  $\Psi$  is robot morphology that includes geometry  $\Psi_{\mathrm{geo}}$ , stiffness  $\Psi_{\mathrm{st}}$ , and actuator placement  $\Psi_{\mathrm{act}}$ ;  $\mathbf{u}_h$  is actuation with the controller's parameters  $\phi$  and dependency on the robot morphology  $\Psi$ ,  $\mathbf{s}_h$  is the simulator state,  $f$  is the environmental dynamics (namely the continuum mechanics of soft robots), and  $H$  is robot time horizon (not to be confused with the later-on mentioned diffusion time). Co-design poses challenges in optimization including complex interdependencies between body and control variables, ambiguity between competing morphology modifications, trade-offs between flexibility and efficacy in design representations, etc. [61]. In this work, we aim at leveraging the generative power of diffusion models in searching for optimal robot designs with (1) the potential to synthesize highly-diverse robots and (2) inherent structural biases in the pre-trained 3D generative models learned from large-scale 3D datasets to achieve efficient optimization.
2.2 3D Shape Generation with Diffusion-based Models
Diffusion-based generative models [24, 52] aim to model a data distribution by augmenting it with auxiliary variables  $\{\mathbf{x}_t\}_{t=1}^T$  defining a Gaussian diffusion process  $p(\mathbf{x}_0) = \int p(\mathbf{x}_T) \prod_{t=1}^T p(\mathbf{x}_{t-1} | \mathbf{x}_t) d\mathbf{x}_{1:T}$  with the transition kernel in the forward process  $q(\mathbf{x}_t | \mathbf{x}_{t-1}) = \mathcal{N}(\mathbf{x}_t; \sqrt{1 - \beta_t} \mathbf{x}_{t-1}, \beta_t \mathbf{I})$  for some  $0 < \beta_t < 1$ . For sufficiently large  $T$ , we have  $p(\mathbf{x}_T) \approx \mathcal{N}(\mathbf{0}, \mathbf{I})$ . This formulation enables an analytical marginal at any diffusion time  $\mathbf{x}_t = \sqrt{\bar{\alpha}_t} \mathbf{x}_0 + \sqrt{1 - \bar{\alpha}_t} \epsilon$  based on clean data  $\mathbf{x}_0$ , where  $\epsilon \sim \mathcal{N}(\mathbf{0}, \mathbf{I})$  and  $\bar{\alpha}_t = \prod_{i=1}^t 1 - \beta_i$ . The goal of the diffusion model (or more precisely the denoiser  $\epsilon_\theta$ ) is to learn the reverse diffusion process  $p(\mathbf{x}_{t-1} | \mathbf{x}_t)$  with the loss,
$$
\min  _ {\theta} \mathbb {E} _ {t \sim [ 1, T ], p (\mathbf {x} _ {0}), \mathcal {N} (\epsilon ; \mathbf {0}, \mathbf {I})} \left[ \| \epsilon - \epsilon_ {\theta} \left(\mathbf {x} _ {t} \left(\mathbf {x} _ {0}, \epsilon , t\right), t\right) \| ^ {2} \right] \tag {2}
$$
Intuitively,  $\epsilon_{\theta}$  learns a one-step denoising process that can be used iteratively during sampling to convert random noise  $p(\mathbf{x}_T)$  gradually into realistic data  $p(\mathbf{x}_0)$ . To achieve controllable generation with conditioning  $\mathbf{c}$ , the denoising process can be slightly altered via classifier-free guidance [25, 12],
$$
\hat {\epsilon} _ {\theta , \text {c l a s s i f i e r - f r e e}} := \epsilon_ {\theta} \left(\mathbf {x} _ {t}, t, \varnothing\right) + s \cdot \left(\epsilon_ {\theta} \left(\mathbf {x} _ {t}, t, \mathbf {c}\right) - \epsilon_ {\theta} \left(\mathbf {x} _ {t}, t, \varnothing\right)\right) \tag {3}
$$
where  $s$  is the guidance scale,  $\varnothing$  is a null vector that represents non-conditioning.
2.3 Robotizing 3D Shapes from Diffusion Samples
We adopt Point-E [39] as a pre-trained diffusion model that is capable of generating diverse and complex 3D shapes, providing a good prior of soft robot geometries. However, the samples from the diffusion model  $\mathbf{x}_t$  are in the form of surface point cloud and are not readily usable as robots to be evaluated in the physics-based simulation. Here, we describe how to robotize the diffusion samples  $\mathbf{x}_t \mapsto \Psi$  and its gradient computation of the objective  $\frac{d\mathcal{L}}{dx_t} = \frac{\partial\mathcal{L}}{\partial\Psi_{\mathrm{geo}}}\frac{\partial\Psi_{\mathrm{geo}}}{\partial\mathbf{x}_t} +\frac{\partial\mathcal{L}}{\partial\Psi_{\mathrm{st}}}\frac{\partial\Psi_{\mathrm{st}}}{\partial\mathbf{x}_t} +\frac{\partial\mathcal{L}}{\partial\Psi_{\mathrm{act}}}\frac{\partial\Psi_{\mathrm{act}}}{\partial\mathbf{x}_t}$ .
Algorithm 1 Training: Embedding Optimization
Initialize:  $\mathcal{D}\gets \emptyset$  ,c←0
while within maximal number of epochs do
Generate data with the diffusion model:  $\mathbf{x}_0\sim p_\theta (\mathbf{x}_0|\mathbf{c})$
Evaluate samples with physics-based simulation:  $l(\mathbf{x}_0) = \mathcal{L}(\Psi (\mathbf{x}_0),\phi)$
Aggregate and update datasets:  $\mathcal{D}\gets$  Filter  $(\mathcal{D}\cup \{\mathbf{x}_0,l\})$
Optimize the embedding  $\mathbf{c}$  on  $\mathcal{D}$  using the objective (5).
Solid Geometry. We use a Material Point Method (MPM)-based simulation [61], which takes solid geometries as inputs. This poses two obstacles: (1) conversion from surface point clouds into solid geometries, and (2) the unstructuredness of data in the intermediate samples  $\mathbf{x}_t$ ,  $t \neq 0$ . The second issue arises from the fact that the diffusion process at intermediate steps may produce 3D points that do not form a tight surface. First, we leverage the predicted clean sample at each diffusion time  $t$ ,
$$
\hat {\mathbf {x}} _ {0} = \frac {\mathbf {x} _ {t} - \sqrt {1 - \bar {\alpha} _ {t}} \cdot \epsilon_ {\theta} (\mathbf {x} _ {t} , t)}{\sqrt {\bar {\alpha} _ {t}}} \tag {4}
$$
This approach is used in denoising diffusion implicit model (DDIM) [53] to approximate the unknown clean sample  $\mathbf{x}_0$  in the reverse process  $p(\mathbf{x}_{t - 1}|\mathbf{x}_t,\hat{\mathbf{x}}_0)$ . Here, we use it to construct a better-structured data for simulation. Hence, we break down the robotizing process into  $\mathbf{x}_t\mapsto \hat{\mathbf{x}}_0\mapsto \Psi$  with gradient components as  $\frac{\partial\Psi}{\partial\hat{\mathbf{x}}_0}\frac{\partial\hat{\mathbf{x}}_0}{\partial\mathbf{x}_t}$ , where  $\frac{\partial\hat{\mathbf{x}}_0}{\partial\mathbf{x}_t}$  can be trivially derived from (4). To convert the predicted surface points  $\hat{\mathbf{x}}_0$  into solid geometry, we first reconstruct a surface mesh from  $\hat{\mathbf{x}}_0$ , and then evenly sample a solid point cloud  $\Psi_{\mathrm{geo}}$  within its interior. For mesh reconstruction, we modify the optimization approach from Shape As Points [40], which provides a differentiable Poisson surface reconstruction that maps a control point set  $\mathbf{V}_{\mathrm{ctrl}}$  to a reconstructed surface mesh with vertices  $\mathbf{V}_{\mathrm{mesh}}$ . We calculate a modified Chamfer Distance loss indicating similarity between  $\mathbf{V}_{\mathrm{mesh}}$  and  $\hat{\mathbf{x}}_0$ :
$$
\mathcal {L} _ {\mathrm {r e c o n}} = \frac {\lambda_ {\mathrm {m e s h}}}{| \mathbf {V} _ {\mathrm {m e s h}} |} \sum_ {\mathbf {v} \in \mathbf {V} _ {\mathrm {m e s h}}} d (\mathbf {v}, \hat {\mathbf {x}} _ {0}) + \frac {\lambda_ {\mathrm {t a g r e t}}}{| \hat {\mathbf {x}} _ {0} |} \sum_ {\mathbf {v} \in \hat {\mathbf {x}} _ {0}} w (\mathbf {v}) d (\mathbf {v}, \mathbf {V} _ {\mathrm {m e s h}}),
$$
in which  $d(\cdot, \cdot)$  denotes minimal Euclidean distance between a point and a point set, and  $w(\mathbf{v})$  denotes a soft interior mask, with  $w(\mathbf{v}) = 1$  for  $\mathbf{v}$  outside the mesh, and  $w(\mathbf{v}) = 0.1$  for  $\mathbf{v}$  inside. The introduced mask term  $w(v)$  aims to lower the influence of noisy points inside the mesh, which is caused by imperfect prediction of  $\hat{\mathbf{x}}_0$  from noisy intermediate diffusion samples. The weight parameters are set to  $\lambda_{\mathrm{mesh}} = 1$  and  $\lambda_{\mathrm{target}} = 10$ . We back-propagate  $\frac{\partial\mathcal{L}_{\mathrm{recon}}}{\partial V_{\mathrm{mesh}}}$  to  $\frac{\partial\mathcal{L}_{\mathrm{recon}}}{\partial V_{\mathrm{ctrl}}}$  through the differentiable Poisson solver, and then apply an Adam optimizer on  $V_{\mathrm{ctrl}}$  to optimize the loss  $\mathcal{L}_{\mathrm{recon}}$ . After mesh reconstruction, the solid geometry  $\Psi_{\mathrm{geo}}$  represented by a solid interior point cloud is then sampled evenly within the mesh, with sufficient density to support the MPM-based simulation. Finally, to integrate the solidification process into diffusion samplers, we still need its gradient  $\frac{\partial\Psi_{\mathrm{geo}}}{\partial\hat{\mathbf{x}}_0}$ . We adopt Gaussian kernels on point-wise Euclidean distances as gradients between two point clouds:
$$
\frac {\partial \mathbf {u}}{\partial \mathbf {v}} = \frac {\exp (- \alpha \| \mathbf {u} - \mathbf {v} \| ^ {2})}{\sum_ {v ^ {\prime} \in \hat {\mathbf {x}} _ {0}} \exp (- \alpha \| \mathbf {u} - \mathbf {v} ^ {\prime} \| ^ {2})}, \mathbf {u} \in \Psi_ {\mathrm {g e o}}, \mathbf {v} \in \hat {\mathbf {x}} _ {0}.
$$
Intuitively, under such Gaussian kernels gradients, each solid point is linearly controlled by predicted surface points near it. In practice, this backward scheme works well for kernel parameter  $\alpha = 20$ .
Actuators and Stiffness. A solid geometry does not make a robot; in order for the robot to behave, its dynamics must be defined. After sampling a solid geometry, we thus need to define material properties and actuator placement. Specifically, we embed actuators in the robot body in the form of muscle fibers that can contract or expand to create deformation; further, we define a stiffness parameterization in order to determine the relationship between deformation and restorative elastic force. We adopt constant stiffness for simplicity since it has been shown to trade off with actuation strength [61]; thus we have  $\Psi_{\mathrm{st}}(\hat{\mathbf{x}}_0) = \mathrm{const}$  and  $\frac{\partial\Psi_{\mathrm{st}}}{\partial\hat{\mathbf{x}}_0} = 0$ . Then, we propose to construct actuators based on the robot solid geometry  $\Psi_{\mathrm{act}}(\Psi_{\mathrm{geo}}(\hat{\mathbf{x}}_0))$  via clustering; namely, we perform k-means with pre-defined number of clusters on the coordinates of 3D points from the solid geometry  $\Psi_{\mathrm{geo}}$ . The gradient then becomes  $\frac{\partial\Psi_{\mathrm{act}}}{\partial\Psi_{\mathrm{geo}}}\frac{\partial\Psi_{\mathrm{geo}}}{\partial\hat{\mathbf{x}}_0}\frac{\partial\hat{\mathbf{x}}_0}{\partial\mathbf{x}_t}$ , where  $\frac{\partial\Psi_{\mathrm{act}}}{\partial\Psi_{\mathrm{geo}}}\approx 0$  as empirically we found the clustering is quite stable in terms of label assignment, i.e., with  $\Delta \Psi_{\mathrm{geo}}$  being small,  $\Delta \Psi_{\mathrm{act}}\rightarrow 0$ . Overall, we keep only the gradient for the robot geometry as empirically it suffices.
Algorithm 2 Sampling: Diffusion As Co-design
Initialize: initial sample  $\mathbf{x}_T\sim \mathcal{N}(\mathbf{0},\mathbf{I})$    
while within maximal number of diffusion steps  $t\geq 0$  do Perform regular per-step diffusion update:  $\mathbf{x}_t\Leftarrow \mathbf{x}_{t + 1}$  if perform co-design then while within  $K$  steps do Run update in (6) and (7) to overwrite  $\mathbf{x}_t$  end while end if   
end while
2.4 Physics Augmented Diffusion Model
**Embedding Optimization.** To best leverage the diversity of the generation from a large-scale pre-trained diffusion models, we propose to (1) actively generate new data from model and maintain them in a buffer, (2) use physics-based simulation as a certificate of performance, (3) optimize the embeddings conditioned by the diffusion model under a skewed data distribution to improve robotic performance in simulation. Curating a training dataset on its own alleviates the burden of manual effort to propose performant robot designs. Optimizing the conditional embeddings instead of finetuning the diffusion model eliminates the risk of deteriorating the overall generation and saves the cost of storing model weights for each new task (especially with large models). We follow,
$$
\min  _ {\mathbf {c}} \mathbb {E} _ {t \sim [ 1, T ], p _ {\theta} (\mathbf {x} _ {0} | \mathbf {c}), \mathcal {N} (\epsilon ; \mathbf {0}, \mathbf {I})} \left[ \| \epsilon - \epsilon_ {\theta} \left(\mathbf {x} _ {t} \left(\mathbf {x} _ {0}, \epsilon , t\right), t, \mathbf {c}\right) \| ^ {2} \right] \tag {5}
$$
Note the three major distinctions from (2): (i) the optimization variable is the embedding  $\mathbf{c}$  not  $\theta$  (ii) the denoiser is conditioned on the embeddings  $\epsilon_{\theta}(\dots, \mathbf{c})$  and (iii) the data distribution is based on the diffusion model  $p_{\theta}$  not the inaccessible real data distribution  $p$  and is conditioned on the embeddings  $\mathbf{c}$ . This adopts an online learning scheme as the sampling distribution is dependent on the changing  $\mathbf{c}$ . The procedure is briefly summarized in Algorithm 1, where Filter is an operation to drop the oldest data when exceeding the buffer limit. In addition, during this stage, we use fixed prescribed controllers since we found empirically that a randomly initialized controller may not be sufficiently informative to drive the convergence toward reasonably good solutions; also, enabling the controller to be trainable makes the optimization prohibitively slow and extremely unstable, potentially due to the difficulty of the controller required to be universal to a diverse set of robot designs. After the embedding optimization, we perform conditional generation that synthesizes samples corresponding to robot designs with improved physical utility via classifier-free guidance as in (3).
Diffusion as Co-design. While the optimized embedding already allows us to generate performant robots for some target tasks, we further improve the performance of individual samples by reformulating the diffusion sampling process into a co-design optimization. As described in Section 2.3, we can convert the intermediate sample at any diffusion time  $\mathbf{x}_t$  to a robot design  $\Psi$ , rendering an evolving robot design throughout the diffusion process. However, regular diffusion update [24] much less resembles any gradient-based optimization techniques, which are shown to be effective in soft robot design and control with differentiable simulation [26, 2]. Fortunately, there is a synergy between diffusion models and energy-based models [54, 15, 14], which allows a more gradient-descent-like update with Markov Chain Monte Carlo (MCMC) sampling [14]. Incorporating the soft robot co-design optimization with differentiable physics [61] into the diffusion sampling process, we have
Design Optim.:  $\mathbf{x}_t^{(k)} = \mathbf{x}_t^{(k - 1)} + \frac{\sigma^2}{2}\left(\epsilon_\theta (\mathbf{x}_t^{(k - 1)},t) - \kappa \nabla_{\mathbf{x}_t^{(k - 1)}}\mathcal{L}(\Psi (\mathbf{x}_t^{(k - 1)}),\phi_t^{k - 1})\right) + \sigma^2\epsilon$  (6)
Control Optim.:  $\phi_t^{(k)} = \phi_t^{(k - 1)} + \gamma \nabla_{\phi_t^{(k - 1)}}\mathcal{L}(\Psi (\mathbf{x}_t^{(k - 1)}),\phi_t^{k - 1})$  (7)
where  $\epsilon \sim \mathcal{N}(\mathbf{0},\mathbf{I})$ ,  $\mathbf{x}_t^{(0)} = \mathbf{x}_{t - 1}^{(K)}$ ,  $\kappa$  is the ratio between two types of design gradients,  $K$  is the number of MCMC sampling steps at the current diffusion time,  $\gamma$  is the weight for trading off design and control optimization, and  $\phi_t^{(0)}$  can be either inherited from the previous diffusion time  $\phi_{t - 1}^{(K)}$  or reset to the initialization  $\phi_T^{(0)}$ . We highlight the high resemblance to gradient-based co-optimization with  $\mathbf{x}_t$  as the design variable and  $\phi_t$  as the control variable. This procedure is performed once every  $M$  diffusion steps (Algorithm 2), where  $M$  is a hyperparameter that trade-offs "guidance" strength from physical utility and sampling efficiency. Intuitively, the entire diffusion-as-co-design process is
Figure 3: We consider passive dynamics tasks (balancing, landing), locomotion tasks (crawling, hurdle), and manipulation tasks (gripping, moving a box).