SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 1: Improved physical utility by augmenting physical simulation with diffusion models.
Embed. Optim.	Diffusion as Co-design	Passive Dynamics	Locomotion	Manipulation
Balancing	Landing	Crawling	Hurdling	Gripping	Moving a Box
0.081·164	0.832·217	0.011·012	0.014·020	0.014·008	0.019·020
✓	0.556·127	0.955·032	0.048·007	0.019·014	0.025·006	0.040·018
✓	✓	0.653·107	0.964·029	0.081·018	0.035·030	0.027·004	0.044·021
guided by three types of gradients: (i)  $\epsilon_{\theta}(\mathbf{x}_t^{(k - 1)},\cdot)$  provides a direction for the design toward feasible 3D shapes based on the knowledge of pre-training with large-scale datasets (and toward enhanced physical utility with the optimized embeddings via classifier-free guidance using  $\hat{\epsilon}_{\theta}(\mathbf{x}_t^{(k - 1)},\cdot ,\mathbf{c}))$  (ii)  $\nabla_{\mathbf{x}_t}\mathcal{L}(\Psi (\mathbf{x}_t),\cdot)$  provides a direction for the design toward improving co-design objective  $\mathcal{L}$  via differentiable simulation, and (iii)  $\nabla_{\phi_t}\mathcal{L}(\cdot ,\phi_t)$  provides a direction for the controller toward a better adaption to the current design  $\mathbf{x}_t$  that allows more accurate evaluation of the robot performance.
3.1 Task Setup
We cover three types of robotics tasks: passive dynamics, locomotion, and manipulation (Figure 3).
- Passive Dynamics tasks include balancing and landing. Balancing initializes the robot on a stick-like platform with small contact area with an upward velocity that introduces instability; the robot's goal is to passively balance itself after dropping on the platform. Landing applies an initial force to the robot toward a target; the robot's goal is to passively land as close to the target as possible.
- Locomotion tasks include crawling and hurdles. Crawling sets the robot at a rest state on the ground; the robot must actuate its body to move as far away as possible from the starting position. Hurdling places an obstacle in front of the robot; the robot must jump over the obstacle.
- Manipulation tasks include gripping and moving objects. Gripping places an object underneath the robot; the goal of the robot is to vertically lift the object. Box Moving places a box on the right end of the robot; the robot must move the box to the left.
Please refer to the appendix Section D for more detailed task descriptions and performance metrics.
3.2 Toward Physical Utility In Diffusion Models
Physics-augmented diffusion. In Table 1, we examine the effectiveness of embedding optimization and diffusion as co-design for improving physical utility. For each entry, we draw 100 samples with preset random seeds to provide valid sample-level comparison (i.e., setting the step size of co-design optimization to zero in the third row will produce almost identical samples as the second row). We report the average performance with standard deviation in the superscript. First, we observe increasing performance across all tasks while incorporating the two proposed techniques, demonstrating the efficacy of DiffuseBot. Besides, the sample-level performance does not always monotonically improve, possibly due to the stochasticity within the diffusion process and the low quality of gradient from differentiable simulation in some scenarios. For example, in gripping, when the robot fails to pick up the object in the first place, the gradient may be informative and fails to bring proper guidance toward better task performance; similarly in moving a box. In addition, we found it necessary to include control optimization during the diffusion sampling process, since, at diffusion steps further from zero, the predicted clean sample  $\hat{\mathbf{x}}_0$  (derived from the intermediate sample  $\mathbf{x}_t$ ) may differ significantly from the clean sample  $\mathbf{x}_0$ , leaving the prescribed controller largely unaligned.
Comparison with baselines. In Table 2, we compare with extensive baselines of soft robot design representation: particle-based method has each particle possessing its own distinct parameterization of design (geometry, stiffness, actuator); similarly, voxel-based method specifies design in voxel level; implicit function [37] uses use a shared multi-layer perceptron to map coordinates to design;
Table 2: Comparison with baselines.
Methods	Passive Dynamics	Locomotion	Manipulation
Balancing	Landing	Crawling	Hurdling	Gripping	Moving a Box
Particle-based	0.040·000	0.863·005	0.019·001	0.006·001	-0.010·001	0.043·027
Voxel-based	0.040·000	0.853·002	0.024·000	0.027·000	-0.009·000	0.025·022
Implicit Function [37]	0.106·147	0.893·033	0.043·024	0.044·063	0.006·012	0.033·030
Diff-CPPN [18]	0.091·088	0.577·425	0.055·023	0.019·029	0.007·008	0.022·017
DiffAqua [35]	0.014·023	0.293·459	0.027·015	0.022·011	0.010·001	0.007·008
DiffuseBot	0.706·078	0.965·026	0.092·016	0.031·011	0.026·002	0.047·019
Figure 4: Examples of DiffuseBot evolving robots to solve different tasks.
DiffCPPN [18] uses a graphical model composed of a set of activation function that takes in coordinates and outputs design specification; DiffAqua [35] computes the Wasserstein barycenter of a set of aquatic creatures' meshes and we adapt to use more reasonable primitives that include bunny, car, cat, cow, avocado, dog, horse, and sofa. These baselines are commonly used in gradient-based soft robot co-design [26, 56, 61]. For each baseline method, we run the co-optimization routine for the same number of steps as in the diffusion-as-co-design stage in DiffuseBot. To avoid being trapped in the local optimum, we run each baseline with 20 different random initializations and choose the best one. Since DiffuseBot is a generative method, we draw 20 samples and report the best; this is sensible an applications-driven perspective since we only need to retrieve one performant robot, within a reasonable sample budget. We observe that our method outperforms all baselines. DiffuseBot leverages the knowledge of large-scale pre-trained models that capture the "common sense" of geometry, providing a more well-structured yet flexible prior for soft robot design.
Soft robots bred by DiffuseBot. In Figure 5, we demonstrate the generated soft robots that excel in locomotion and manipulation tasks. We highlight the flexibility of DiffuseBot to generate highly diverse soft robot designs that accommodate various purposes in different robotics tasks. Furthermore, in Figure 4, we show how robots evolve from a feasible yet non-necessarily functional design to an improved one that intuitively matches the task objective. By manually inspecting the evolving designs, we found that the role of the embedding optimization is to drive the diverse generations toward a converged, smaller set with elements having higher chance to succeed the task; on the other hand, the role of diffusion as co-design brings relatively minor tweaks along with alignment between the control and design. Due to space limit, we refer the reader to our project page for more results.
Figure 5: Examples of robots bred by DiffuseBot to achieve the desired tasks.
Performance
MT	0.016 .014
FT	0.031 .024
Ours	0.048 .007
Table 3: Ablation on embedding optimization. MT means manually-designed text. FT means finetuning models.
Figure 6: Varying starting point and strength of diffusion as co-design.
3.3 Ablation Analysis
In this section, we conduct a series of ablation studies to provide a deeper understanding of the proposed method. For simplicity, all experiments in this section are done with the crawling task.
**Embedding optimization.** In Table 3, we compare the optimization of the embedding conditioned by the diffusion models with other alternatives. The pre-trained diffusion model [39] that DiffuseBot is built upon uses CLIP embeddings [43], which allows for textual inputs. Hence, a naive approach is to manually design text for the conditional embedding of the diffusion model. The result reported in Table 3 uses "a legged animal or object that can crawl or run fast". We investigated the use of text prompts; in our experience, text was difficult to optimize for functional robot design purposes. This is expected since most existing diffusion models perform content generation only in terms of appearance instead of physical utility, which further strengthens the purpose of this work. In addition, with exactly the same training objective as in (5), we can instead finetune the diffusion model itself. However, this does not yield better performance, as shown in the second entry in Table 3. Empirically, we found there is a higher chance of the generated samples being non-well-structured with fractured parts. This suggests that finetuning for physical utility may deteriorate the modeling of sensible 3D shapes and lead to more unstable generations.
Diffusion as co-design. Recall that the co-design optimization can be seamlessly incorporated into any diffusion step. In Figure 6, we examine how the strength of the injected co-design optimization affects the task performance in terms of where to apply throughout the diffusion sampling process and how many times to apply. In the left figure of Figure 6, we sweep through the maximal diffusion time of applying diffusion as co-design, i.e., for the data point at  $t = 400$ , we only perform co-design from  $t = 400$  to  $t = 0$ . We found that there is a sweet spot of when to start applying co-design (at  $t \approx 200$ ). This is because the intermediate samples at larger diffusion time  $\mathbf{x}_t, t \gg 0$  are extremely under-developed, lacking sufficient connection to the final clean sample  $\mathbf{x}_0$ , hence failing to provide informative guidance by examining its physical utility. Furthermore, we compare against post-diffusion co-design optimization, i.e., run co-design based on the final output of the diffusion (0.064 vs ours 0.081). We allow the same computational budget by running the same number of times of differentiable simulation as in DiffuseBot. Our method performs slightly better, potentially due to the flexibility to alter the still-developing diffusion samples. Also note that while our method is interleaved into diffusion process, it is still compatible with any post-hoc computation for finetuning.
3.4 Flexibility To Incorporate Human Feedback
Beyond the generative power, diffusion models also provide the flexibility to composite different data distributions. This is especially useful for computational design since it empowers to easily incorporate external knowledge, e.g. from human. We follow the compositionality techniques introduced in [34, 14], which can be directly integrated into our diffusion as co-design
Figure 7: Incorporating human textual feedback.
framework. In Figure 7, we demonstrate incorporating human feedback in textual form as "a unicorn" into a crawling robot generated by DiffuseBot. We can see the emergence of the horn-like body part.
3.5 From Virtual Generation To Physical Robot
We further fabricate a physical robot for the gripping task as a proof-of-concept to demonstrate the possibility of real-world extension. We use a 3D Carbon printer to reconstruct the exact geometry of a generated design and fill the robot body with a Voronoi lattice structure to achieve softness.
For actuators, we employ tendon transmission to realize the contraction force utilized in the soft robot gripper. In our project page, we demonstrate the robots generated by DiffuseBot are capable of picking up an object. Note that physical robot fabrication and real-world transfer have countless non-trivial challenges including stiffness and actuator design, sim-to-real gap, etc. Hence, this experiment is only meant to demonstrate the potential instead of a general, robust pipeline toward physical robots, which is left to future work. We refer the reader to the appendix for more details.
Heuristic Search For Soft Robot Co-Design. Heuristic searches are simple but useful tools for code-signing soft robots. A long line of work has focused on evolutionary algorithms [7, 8, 10], with some including physical demonstrations [22, 31, 32] and recent benchmarks incorporating neural control [5]. These methods are often powered by parameterized by compositional pattern-producing networks [58], which parameterize highly expressive search spaces akin to neural networks [50, 51]. Similar to [5], [47] combines a heuristic approach with reinforcement learning, and demonstrates resulting designs on physical hardware. Other notable methods include particle-filter-based approaches [11] and simulated annealing [60]. Heuristic search methods tend to be less efficient than gradient-based or learning-based algorithms, but can reasoning about large search spaces; our approach employs the highly expressive diffusion processes, while leveraging the differentiable nature of neural networks and physical simulation for more efficient and gradient-directed search.
Gradient-Based Soft Robot Co-Optimization. A differentiable simulator is one in which useful analytical derivatives of any system variable with respect to any other system variable is efficiently queryable; the recent advent of soft differentiable simulation environments [26, 56, 13, 33, 41, 42, 61] has accelerated the exploration of gradient-based co-optimization methods. [26, 56] demonstrated how differentiable simulators can be used to co-optimize very high-dimensional spatially varying material and open-loop/neural controller parameters. [38] presented gradient-based search of shape parameters for soft manipulators. Meanwhile, [61] showed how actuation and geometry could be co-optimized, while analyzing the trade-offs of design space complexity and exploration in the search procedure. DiffuseBot borrows ideas from gradient-based optimization in guiding the design search in a physics-aware way, especially in the context of control.
Learning-Based Soft Robot Co-Design Methods. Though relatively nascent, learning-based approaches (including DiffuseBot) can re-use design samples to build knowledge about a problem. Further, dataset-based minibatch optimization algorithms are more robust to local minima than single-iterate pure optimization approaches. [57] demonstrated how gradient-based search could be combined with learned-models; a soft robot proprioceptive model was continually updated by simulation data from interleaved control/material co-optimization. Other work employed learning-based methods in the context of leveraging available datasets. [35] learned a parameterized representation of geometry and actuators from basis shape geometries tractable interpolation over high-dimensional search spaces. [55] leveraged motion data and sparsifying neurons to simultaneously learn sensor placement and neural soft robotic tasks such as proprioception and grasp classification.
Diffusion Models for Content Generation. Diffusion models [24, 52] have emerged as the de-facto standard for generating content in continuous domains such as images [44, 46], 3D content [66, 65], controls [28, 9, 1], videos [23, 49, 16], and materials [63, 62, 48]. In this paper, we explore how diffusion models in combination with differentiable physics may be used to design new robots. Most similar to our work, [64] uses differentiable physics to help guide human motion synthesis. However, while [64] uses differentiable physics to refine motions in the last few timesteps of diffusion sampling, we tightly integrate differentiable physics with sampling throughout the diffusion sampling procedure through MCMC. We further uses differentiable simulation to define a reward objective through which we may optimize generative embeddings that represent our desirable robot structure.
We presented DiffuseBot, a framework that augments physics-based simulation with a diffusion process capable of generating performant soft robots for a diverse set of tasks including passive dynamics, locomotion, and manipulation. We demonstrated the efficacy of diffusion-based generation with extensive experiments, presented a method for incorporating human feedback, and prototyped a physical robot counterpart. DiffuseBot is a first step toward generative invention of soft machines, with the potential to accelerate design cycles, discover novel devices, and provide building blocks for downstream applications in automated computational creativity and computer-assisted design.
A Background On Point-E
Point-E [39] is a diffusion-based generative model that produces 3D point clouds from text or images. The Point-E pipeline consists of three stages: first, it generates a single synthetic view using a text-to-image diffusion model; second, it produces a coarse, low-resolution 3D point cloud (1024 points) using a second diffusion model which is conditioned on the generated image; third, it upsamples/“densifies” the coarse point cloud to a high-resolution one (4096 points) with a third diffusion model. The two diffusion models operating on point clouds use a permutation invariant transformer architecture with different model sizes. The entire model is trained on Point-E's curated dataset of several million 3D models and associated metadata which captures a generic distribution of common 3D shapes, providing a suitable and sufficiently diverse prior for robot geometry. The diffused data is a set of points, each point possessing 6 feature dimensions: 3 for spatial coordinates and 3 for colors. We ignore the color channels in this work. The conditioning for the synthesized image in the first stage relies on embeddings computed from a pre-trained ViT-L/14 CLIP model; in the embedding optimization of DiffuseBot, the variables to be optimized is exactly the same embedding. Diffusion as co-design is only performed in the second stage (coarse point cloud generation) since the third stage is merely an upsampling which produces only minor modifications to robot designs. We refer the reader to the original paper [39] for more details.
B Theoretical Motivation
Online learning in embedding optimization. In Section 2.4, we discuss how to online collect a dataset to optimize the embedding toward improved physical utility. Given a simplified version of (5)
$$
\min  _ {\mathbf {c}} \mathbb {E} _ {p _ {\theta} \left(\mathbf {x} _ {0} \mid \mathbf {c}\right)} [ g (\mathbf {x} _ {0}, \mathbf {c}) ] \tag {8}
$$
where, for notation simplicity, we drop  $t \sim [1,T]$ ,  $\mathcal{N}(\epsilon; \mathbf{0}, \mathbf{I})$  in the sampling distribution, and summarize  $\left[||\epsilon - \epsilon_{\theta}(\mathbf{x}_{t}(\mathbf{x}_{0}, \epsilon, t), t, \mathbf{c})||^{2}\right]$  as  $g(\mathbf{x}, \mathbf{c})$ . We can rewrite the expectation term as,
$$
\int p _ {\theta} \left(\mathbf {x} _ {0}\right) \frac {p _ {\theta} \left(\mathbf {c} \mid \mathbf {x} _ {0}\right)}{p _ {\theta} (\mathbf {c})} g \left(\mathbf {x} _ {0}, \mathbf {c}\right) d x \tag {9}
$$
which allows to sample from  $p_{\theta}(\mathbf{x}_0)$  (i.e., generating samples from the diffusion model) and reweight the loss with  $\frac{p_{\theta}(\mathbf{c}|\mathbf{x}_0)}{p_{\theta}(\mathbf{c})}$ ; the latter scaling term is essentially proportional to a normalized task performance. Empirically, we can maintain a buffer for the online dataset and train the embedding with the sampling distribution biased toward higher task performance; we use a list to store samples with top-k performance in our implementation (we also tried reshaping the sampling distribution like prioritized experience replay in reinforcement learning but we found less stability and more hyperparameters required in training compared to our simpler top-k approach).
Connection to MCMC. In diffusion sampling, the simplest way to perform reverse denoising process as in Section 2.2 follows [24],
$$
p _ {\theta} \left(\mathbf {x} _ {t - 1} \mid \mathbf {x} _ {t}\right) = \mathcal {N} \left(\frac {1}{\sqrt {\alpha_ {t}}} \left(\mathbf {x} _ {t} - \frac {\beta_ {t}}{\sqrt {1 - \bar {\alpha} _ {t}}} \epsilon_ {\theta} \left(\mathbf {x} _ {t}, t\right)\right), \frac {1 - \bar {\alpha} _ {t - 1}}{1 - \bar {\alpha} _ {t}} \beta_ {t} \mathbf {I}\right) \tag {10}
$$
Here, the denoising term can either be unconditional  $\epsilon_{\theta}(\mathbf{x}_t,t)$  or conditional via classifier-free guidance  $\hat{\epsilon}_{\theta ,\mathrm{classifier - free}}(\mathbf{x}_t,t,\mathbf{c})$  as in (3). We use the latter to incorporate the optimized embedding. To further leverage physics-based simulation, we aim to introduce physical utility during diffusion sampling process. One possibility is to utilize classifier-based guidance [12],
$$
\hat {\epsilon} _ {\theta , \text {c l a s s i f i e r - b a s e d}} := \epsilon_ {\theta} (\mathbf {x} _ {t}, t) - s \cdot \sqrt {1 - \bar {\alpha} _ {t}} \nabla_ {\mathbf {x} _ {t}} \log p (\mathbf {c} | \mathbf {x} _ {t}) \tag {11}
$$
where  $p(\mathbf{c}|\mathbf{x}_t)$  can be conceptually viewed as improving physical utility and  $\nabla_{\mathbf{x}_t}\log p(\mathbf{c}|\mathbf{x}_t)$  can be obtained using differentiable physics and the unconditional score. Note that we slightly abuse the notation here by overloading  $\mathbf{c}$  with conditioning from differentiable simulation during sampling other than the classifier-free guidance using the optimized embedding. However, combining (10) and (11) much less resembles any gradient-based optimization techniques, which are shown to be effective in soft robot co-design with differentiable simulation [26, 2]. Fortunately, drawing a connection to energy-based models [54, 15, 14], yet another alternative to incorporate conditioning in diffusion
Table 4: Configuration of embedding optimization.
| Balancing | Landing | Crawling | Hurdling | Gripping | Moving a Box
Buffer Size	600
Min. Buffer Size	60
Num. Samples / Epoch	60
Train Iter. / Epoch	1
Buffer Top-K	12
Batch Size	6
Table 5: Configuration of diffusion as co-design.
Balancing	Landing	Crawling	Hurdling	Gripping	Moving a Box
tmax	400	150	400	400	400	400
tmin	0	0	0	0	0	0
Δt	50	25	50	50	50	50
K	3	3	5	5	5	5
σ	10-4·β	10-4·β	10-4·β	10-4·β	10-4·β	10-4·β
κ	104	104	104	104	104	104
γ	-	-	0.01	0.001	0.001	0.001
Renorm Scale	10	10	10	10	10	10
models is Markov Chain Monte Carlo (MCMC) sampling [14], where we use Unadjusted Langevin Dynamics,
$$
\mathbf {x} _ {t} = \mathbf {x} _ {t} ^ {(K)}, \text {w h e r e} \mathbf {x} _ {t} ^ {(k)} \sim \mathcal {N} \left(\mathbf {x} _ {t} ^ {(k)}; \mathbf {x} _ {t} ^ {(k - 1)} + \frac {\sigma^ {2}}{2} \nabla_ {\mathbf {x}} \log p (\mathbf {c} | \mathbf {x} _ {t} ^ {(k - 1)}), \sigma^ {2} \mathbf {I}\right) \tag {12}
$$
where  $K$  is the number of samples in the current MCMC with  $k$  as indexing,  $\sigma^2$  is a pre-defined variance, and  $\mathbf{x}_t^{(0)} = \mathbf{x}_{t - 1}$ . In the context of diffusion models, this procedure is commonly performed within a single diffusion step to drive the sample toward higher-density regime under the intermediate distribution  $p(\mathbf{x}_t) = \int q(\mathbf{x}_t|\mathbf{x}_0)p(\mathbf{x}_0)d\mathbf{x}_0$  at diffusion time  $t$ . Inspired by its resemblance to gradient ascent with stochasticity from the added Gaussian noise of variance  $\sigma^2$ , we establish a connection to design optimization, reformulating diffusion process as co-design optimization as in (6) and (7). Specifically, we can apply Bayes rule to decompose the score of  $p(\mathbf{c}|\mathbf{x}_t^{(k - 1)})$ ,
$$
\nabla_ {\mathbf {x}} \log p (\mathbf {c} | \mathbf {x} _ {t}) = \nabla_ {\mathbf {x}} \log p (\mathbf {x} _ {t} | \mathbf {c}) - \nabla_ {\mathbf {x}} \log p (\mathbf {x} _ {t}) \tag {13}
$$
where  $\nabla_{\mathbf{x}}\log p(\mathbf{x}_t)$  is simply the denoiser output  $\epsilon_{\theta}(\mathbf{x}_t,t)$  and  $\nabla_{\mathbf{x}}\log p(\mathbf{x}_t|\mathbf{c})$  is the gradient of task performance with respect to the intermediate sample of the diffusion model from differentiable physical simulation and robotizing process. Overall, this leads to (6).
C Implementation Details In Algorithm
In this section, we provide more implementation details and experimental configurations of Diffuse-Bot and other baselines. In Table 4, we list the configurations of the embedding optimization. With respect to Algorithm 1, "buffer" refers to the online dataset  $\mathcal{D}$ .