SoftZoo: A Soft Robot Co-design Benchmark For Locomotion In Diverse Environments
Authors: Tsun-Hsuan Wang
Pingchuan Ma
Andrew Everett Spielberg
Zhou Xian
Hao Zhang
Joshua B. Tenenbaum
Daniela Rus
Chuang Gan
Venue: ICLR 2023 poster (2023)
Keywords: soft robot
co-design
differentiable physics

Abstract: While significant research progress has been made in robot learning for control, unique challenges arise when simultaneously co-optimizing morphology. Existing work has typically been tailored for particular environments or representations. In order to more fully understand inherent design and performance tradeoffs and accelerate the development of new breeds of soft robots, a comprehensive virtual platform — with well-established tasks, environments, and evaluation metrics — is needed. In this work, we introduce SoftZoo, a soft robot co-design platform for locomotion in diverse environments. SoftZoo supports an extensive, naturally-inspired material set, including the ability to simulate environments such as flat ground, desert, wetland, clay, ice, snow, shallow water, and ocean. Further, it provides a variety of tasks relevant for soft robotics, including fast locomotion, agile turning, and path following, as well as differentiable design representations for morphology and control. Combined, these elements form a feature-rich platform for analysis and development of soft robot co-design algorithms. We benchmark prevalent representations and co-design algorithms, and shed light on 1) the interplay between environment, morphology, and behavior (2) the importance of design space representations 3) the ambiguity in muscle formation and controller synthesis and 4) the value of differentiable physics. We envision that SoftZoo will serve as a standard platform and template an approach toward the development of novel representations and algorithms for co-designing soft robots’ behavioral and morphological intelligence. Demos are available on our project page.

Published as a conference paper at ICLR 2023
SoFTZoo: A SOFT ROBOT CO-DESIGN BENCHMARK FOR LOCOMOTION IN DIVERSE ENVIRONMENTS
Tsun-Hsuan Wang' 1* Pinggchuan Andrew Spielberg' 1,, Zhou Xian Hao Zhang', Joshua B. Tenenbaum', Daniela Rus', Chuang Gan
MIT CSAIL,, MIT-IBM Watson Al Lab, UMass Amherst, 'aavvard, $CMU
tsunwêmi t. edu, pcma @mi t edd,, aespie www.mmit.ee.ee xi (andrew. cmu cmu.edu, hao. zhange whangeumass........ umass. edu, jbt@mit..die rus8csail mit.eduu chuangg umass. edu
ABSTRACT
While significant research progress has been made robot learning for control, unique challenges arise when simultaneously co-optimizing morphology. Existing work has typically been tailored for particular environments or representations. order to more fully understand inherent design and performance tradeoffss and ac- celerate the development of new breeds of soft robots, a comprehensive virtual platform - with well-established tasks, environments, and evaluation metrics is needed. In this work, we introduce SoftZoo, soft robot co-design platform for locomotion in diverse environments. SoftZoo supports an extensive, naturally- inspired material set, including the ability to simulate environments such as flat ground, desert, wetland, clay, ice, snow, shallow water, and ocean. Further, pro- vides variety of tasks relevant for soft robotics, including fast locomotion, agile well as differentiable design representations for
turning, and path following, morphology and control. Combined, these elements form a feature-rich platform for analysis and development of soft robot co-design algorithms. We benchmark prevalent representations and co-design algorithms, and shed light on 1) the inter- play between environment, morphology, and behavior 2) the importance of design space representations 3) the ambiguity in muscle formation and controller syn- thesis and 4) the value of differentiable physics. We envision that SoftZoo will serve as a standard platform and template an approach toward the development of novel representations and algorithms for co-designing soft robots' behavioral and morphological intelligence. Demos available on project page'.
INTRODUCTION
The natural world demonstrates morphological and hehavioral complexity to a degree unexplored in soft robotics. A jellyfish's gently undulating geometry allows it to efficiently travel across large bodies of water; an ostrich's s spring-like feet allow for fast, agile motion over widely varying topog- raphy; a chameleon's 's feet allows for dexterous climbing up trees and across branches. Beyond their comparative lack of diversity, soft robots' designs are rarely computationally optimized in silico for the environments in which they are to be deployed. The degree of morphological intelligence observed in the natural world would be similarly advantageous in artificial life.
In this paper, we present SoftZoo, a framework for exploring and benchmarking algorithms for co- designing soft robots in behavior and morphology, with emphasis on locomotion tasks. Unlike pure control or physical design optimization, co-design algorithms co-optimize over a robot's brain and body simultaneously, finding more efficacious solutions that exploit their rich interplay (Ma et al., 2021: Spielberg et al., 2021; Bhatia et al., 2021). We have seen examples of integrated morphology and behavior in soft manipulation (Puhlmann et al., 2022), swimming (Katzschmann et al., 2018), flying (Ramezani et al., 2016), and dynamic locomotion (Tang al., 2020). Each of these robots was designed manually: algorithms that design such robots, and tools for designing the algorithms that design such robots have the potential to accelerate the invention of diverse and capable robots.
"This work was done during an internship at the MIT-IBM Watson Lab. Support for this work was also provided in part by the NSF EFRI Program (Grant No. 1830901). DARPA MCS Program, MIT-IBM Watson AI Lab, and gift funding from MERL, Cisco, and Amazon. Project Page: https: // a tes googl google.com. con/vi ew wttp:///wwww/aaatooooooooo//

---

Published as a conference paper at ICLR 2023
SoftZoo decomposes computational soft robot co-design into four elements: design representations (of morphology and control), tasks for which robots are to be optimized (mathematically, reward or objective functions), environments (including the physical models needed to simulate them), and co- design algorithms. Representationally, SoftZoo provides an unified and flexible interface of robot geometry, body stiffness, and muscle placement that can take operate on common 3D geometric primitives such point clouds, voxel grids, and meshes. For benchmarking. SoftZoo includes variety of dynamic tasks important robotics, such as fast locomotion, agile turning, and path fol- lowing. To study environmentally-drinn naturally-ing inspired material set that allows to not only simulate hyperelastic soft robots, but also emulate ground, desert, wetland, clay, ice, and snow terrains, as well as shallow and deep bodies of water. SoftZoo provides a differentiable multiphysics engine built atop the material point method (MPM) for simulating these diverse biomes. Differentiability provides a crucial ingredient for the development of co-design algorithms, which increasingly commonly exploit model-based gradients for efficient design search. This focus on differentiable multiphysical environments in contrast to to previous work (Bhatia et al., 2021; Graule et al., 2022) which relied on simplified physical models with limited phenomena and no differentiability; this limited the types of co-design problems and gorithms to which they could be applied. The combination of differentiable multiphysics simulation with the decomposition of environmentally-drivee (representation, algorithm, environment physics, task) makes SoftZoo particularly well suited to systematically understanding the influence design representations, physical modeling, and task objectives in the development of soft robot co-design algorithms. In summary, we contribute:
robot design and motion, SoftZoo supports extensive,
co-design into its distinet constituent elements
soft robot co-design platform for locomotion diverse environments with the support of an extensive, naturally-inspired material set, variety of locomotion tasks, unified interface for robot design, and a differentiable physics engine.
Algorithmic benchmarks for various representations and learning algorithms, laying a foundation for studying behavioral and morphological intelligence.
Analysis of /) the interplay between environment, morphology, and behavior 2) the importance of design space representations 3) the ambiguity muscle formation and controller synthesis 4) the value of differentiable physics, with numerical comparisons of gradient-based and tradient-free design algorithms and intelligible examples of where gradient-based co-design fails. This analysis provides insight into the efficacy different aspects state-of-the-an methods and steer future co-design algorithm development.
Ground
Wetland
Ice
Shallow Water
Desert
Clay
Snow
Ocean
Figure 1: An overview of SoftZoo with demonstration of various biologically-insprred designs.
2 SoFTZoo
2.1
OVERVIEW
SoftZoo is soft robot co-design platform for locomotion in diverse environments. supports var- ied, naturally-inspieed materials that can construct environments including ground, desert, wetland, clay, ice, shallow water, and ocean. The task consists of fast locomotion, agile turning, and path
2

---

Published as a conference paper at ICLR 2023
following. The suite provides a seamless and flexible interface of robot design that specifies robot geometry, body stiffness, and muscle placement, and accepts common 3D representations such as point clouds, voxel grids, or meshes. The robot can then be controlled according to muscle groups. The underlying physics engine also supports differentiability that provides model-based gradients, which gains increasing attention in soft robot control and design. In the following, we walk through high-level components of SoftZoo: simulation engine, environment setup, and locomotion tasks.
2.2 SIMULATION ENGINE
In this section, we briefly introduce the underlying simulation technique to facilitate later-on dis- cussion. SoftZoo implemented using the Taichi library (Hu et al., 2019b), which compiles phys- ical simulation code (and its reverse-mode autodifferentiated gradients) for efficient parallelization on GPU hardware. Continuum mechanics in SoftZoo follows the discretization of Moving Least Squares Material Point Method (ML.S-MMM) (Hu et al., 2018), a variant of B-spline MPM (Stom- akhin al., 2013) with improved efficiency. Multimaterial simulation and dignedddistance-funttion based boundary conditions for terrain are implemented to construct diverse environments.
Material Point Method (MPM). The material point method is hybrid Eulerian-Lagrangio method, where both Lagrangian particles and Eulerian grid nodes are used to transfer simulation state information back-and-forth. At a high level, MPM is comprised of three major steps: particle- to-grid transfer (P2G), grid operation, and grid-to-particle transfer (G2P). Material properties, in- cluding position Xp. velocity Vp. mass mp, deformation gradients Fp. and affine velocity field Cp, stored in Lagrangian particles that move through space and time. MPM allows large defor- mation, automatic treatment of self-collision and fracture, and simple realization of multi-material interaction, and is hence well-suited for soft robots in diverse environments. We show the governing equations used by MPM and more implementation details in Appendix D.
Contact Model. The terrain in SoftZoo is represented as meshes that interact with either the robot body or other ground cover materials such snow sand. We employ grid-based contact treat- ment (Stomakhin et al., 2013) to handle particle perrain-pattccle collision, using a Coulomb friction model. We compute the surface normal of the terrain to measure the signed distance function (SDF) and construct boundary conditions for velocity in grid space.
Multiphysical Materials. We present set of environments with multiphysical material support. The materials cover diverse set of physical phenomena, including hyperelasticity, plasticity, fluid- ity, and inter-particle friction. These phenomena combine to model common real-world materials such sand, snow, rubber, mud, water, and more. We list all environments with their correspond- ing multiphysical materials in Appendix E. In addition to categorical choices of environments, we also provide a set of elastic constitutive models (e.g., St. Venant-Kirchhoff corotated linear, neo- Hookean, etc.) and expose their parameters so that the practitioners can easily fine-tune the material behaviors. In our experiments, we use neo-Hookeen material since it a) is known to be accurate materials common in real-world soft robots, b) skips the costly calculation of singular value decomposition (SVD) and e) results in an improvement in simulation speed and numerical stability and reduces gradient instability caused by singular value degeneracy.
for modeling
Actuation Model. We define our actuation model as an anisotropic muscle installed along specified particles. Each muscular element generates directional force along unit fiber vector f. We realize this actuation model using the anisotropic elastic energy from Ma et al. (2021) & s s|]l all?, where = |f]l F is the deformation gradient, 8 is a muscular stiffness parameter, and is the actuation signal provided by controller. This energy is added to the constitutive material energy of particles along which the muscle fiber is installed.
2.3 ENVIRONMENT SETUP
Initialization. Environment construction requires three steps: terrain generation, robot placement, and material covering. First, we procedurally generate terrain height-maps using Perlin noise with user-defined height range and other parameters for roughness. The generated height map is con- verted to mesh for ground surface. We then instantiate a robot using specified design parameters. A user may choose to have the robot placed randomly or at a fixed pre-defined position on the ter- rain. Then, we compute an occupancy map in the Eulerian grid in MPM based on particles of robot body and terrain's SDF, Finally, we use the terrain's 's surface normal to layer particles of specified terrain materials atop it, using the occupancy map avoid particle placement in non-free space.
3

---

Published as a conference paper at ICLR 2023
Table 1: Large-scale benchmark of coologiaally-inspiee designs in SoftZoo. Each entry shows results from differentiable physics (left) and RL (right). The higher the better.
Task
Animal
Ground
Ice
Wetland
Environment
Clay
Destert
Snow
Occan
Baby Seal
0.122/ 0.154
0,048/ 0.000
0.032/0 0.000
0.01270.005
0,059 10,034
0.039/0,015
0.033/0.029
Movement Speed
Caterpillar
11080/D005
0.023/ 0.006
0.02//0.06
0.005
D.047/ /0.07
0.134/ / 00.18
Fish
0.053/D.00 /D003
0.029/ /D.11
0.026/ / 0.00
0.037/ 0.001
0.115/0.02
0.087/ 0.042
0.084/0 00.4//.115
Panda
0.046 /0.019
0.038/ / 00000
0.016 / 0.008
0.019: 1 00000
0.023/0.000
0.031 / 00.004
0.024 / /0.007
Baby Seal
0,088 0.024
0.014/0.008
0,021 10,011
0,081 10,026
0.047 70,028
0.059/0.000
Turning
Caterpillar
Hish
0.053/0.021
0.032/D.047
0.021
/..02
0,040/ 0.000
0.023/0.000
0015/00.007
/
0,021 70.019
0.034/0.015
0.028 / 0.029
0.069
0.195 0.358
0..41/0.013
Panda
0.064 /0.04/
0.023 / 000003
0.009 / 0.004
0.008 / 0.000
0.014/0.003
0.013/ 0.002
0.035 / 0.031
Baby Seal
0.343/0.410
0.257 1914
0.249 0.222
0.231 10,205
0.290 10,265
0.215/ 70,19
0.379 / 0.066
Velocity
Caterpillar
0.502/0.366
0.101/ 156
0,426/ 0.199
0,282 /.005
10.376
0.379 0.133
0.555 0.714
Tracking
Hish
0.216/D.256
0.416/0.0 /.316
0.221 / /0.236
0.234/ / 0.303
0.457 10.311
0u638 / 0.330
0.657/ 0.554
Panda
0.575/0.424
0.555 / 0.383
0.536 / 0.533
0.153 / 0.199
0.450 00,359
0.472/ / 0.288
0.395 / 0.220
Baby Seal
4.012/
-0.018/ / -0.027
-0.01. / -0.027
-0,0200 / -0002
-0.014/ -0.026
-0.010/ -0.026
Waypoint
Caterpillar
-0.016
-0.019
-0.01 /-0.22
-0,020 00.027
-0.13/ 0.027
-0.018/ -00026
-0,002 /0.019
Fallowing
Hish
-0.004 /.004/-0.006
-0.916/-0.00
-0.022./-411 -4.029
-0.024/ / -0.027
-0.007 /-00224
-0.003 /0.024
Panda
-0,003 / -0.014
4.004/ -0.025
.0,021/ / /0.0227
.0.023/ / -0.028
.0.010 -0.024
-0.008 / -0.025
-0.013/
2.013/-0.026 D00026
A co-design algorithm consists of 1) design optimizer that proposes robot design at the start of a simulation trial (an "episode" in reinforcement learning terminology) and 2) a control optimizer that specifies a controller that determines robot actuation based on its observed state. Accordingly, each task interfaces with the algorithm through the robot design interface, observation, action, and reward. We introduce each element as follows.
Robot Design Interface. Robot design involves specification of geometry, stiffness, and muscle placement. We integrate these design specifications into Lagrangian particles in MPM simulation. Geometry is modeled by mass mp R (clamping regions of sufficiently low mass to 0); stiffness is modeled by a the Young's modulus in elastic material s, E R. We remove non-existing (zero-mass) particles to eliminate their effect. (Low and zero mass particles cause numerical instabilities during simulation.) For muscle placement r E E2 on robot with at most K actuators, we augment each actuated particle with a K -Kddimensional unit-vector where the magnitude of component i specifies the contribution of actuator to that particle well as 3-dimensional unit- to specify muscle direction f, E IRS. While a robot is represented as a particle set in the simulator, SoftZoo allows common 3D representations other than point clouds such as voxel grids or meshes. We instantiate set of particles in bounding base shape (e.g., box or ellipsoid), which defines the workspace of a robot design with position {xp]. velocity [vp], and other attributes required in MPM. For voxel grids, we utilize a voxelizer to transfer voxel occupancy particle mass that resides in that voxel. For meshes, we compute SDF for particles in the bounding base shape and assign non-zero mass to sub-zero level set (those with negative signed distance).
Observation. Robot state observations, which are fed to controllers, are computed at every step in an episode, consisting of robot proprioceptive information, environmental information, and task- related information. For robot proprioceptive information, considering states of all particles in robot body is unrealistic from both the perspective of sensor placement and computational tractability. Instead, we compute the position and velocity centroids of the robot body, or of pre-defined body parts. The environment is summarized as a semantic occupancy map; from the MPM Eulerian grid, 3D voxel grid is constructed in with each voxel indicates occupancy of terrain (below the SDF boundary), terrain material particles, or the robot. Finally, task-related information provides sufficient specifications in order to solve certain task, e.g., target waypoints to be followed.
Action. At each time step, the simulator queries the robot controller for an action vector E R (Note that the time step is at the time scale of a robot controller, different from simulation steps often referred as substeps.) We the same action vector for all simulation steps within an environment time step. The returned action vector is of length as K, the maximal number of actuators. The action space bound is set to achieve reasonable robot motion without easily causing numerical fracture of robot body. We use (-3.3, 0.3) as coordinate- wise bound in this paper.
Reward. A robot's task performance is quantitatively measured by a scalar- reward function. The reward definition is task-specific and computed at every time step to provide feedback signal to the robot. Please refer to Section 2.4 for detailed descriptions of reward for every tasks.
2.4 LOCOMOTION TASKS
We only mention the high-level goal of each task. Please refer Section B for more details.
4

---

Published as a conference paper at ICLR 2023
Forward Desert
Caterpillar Forward Wetland
Panda Turning on Ground
Baby Seal Turning on Ice
Figure Unique motions arising from morphology and environment to achieve locomotion.
Movement Speed. In this task, the robot aims to move as fast as possible along target direction.
Turning, this task, the robot is encouraged to turn as fast as possible counter-clockwise about the upward direction of robot's canonical pose.
Velocity Tracking. In this task, the robot is required to track series of timestamped velocities.
Waypoint Following, In this task, the robot needs to follow a sequence of waypoints.
EXPERIMENTS
3.1 BIOL.GICALLL-NNSPIRED MORPHOLOGIES
To study the interplay between environment, morphology. and behavior, we first use set of ani- mal meshes (selected from http://wwwwaaidddiill....oo locations. Through optimizing their controllers, observe that different designs exhibit better- performing behavior in different environments.
as robot bodies and label muscle
Muscle Annotation. We implement a semi-automatic muscle annotator that first converts a mesh into a point cloud, secondly performs K-means clustering with user-defined number of body groups, and finally applies principal component analysis on indepdendent clusters to extract muscle direc- tion. Users may then fine-tune the resulting muscle placement.
Animals. We test four animals: Baby Seal, Caterpillar, Fish, and Panda. These animals are chosen since they exhibit distinct strategies in nature. We refer the readers to Section K for visualization of each animal-inspired design, including their muscle layouts, and detailed description.
Large-scale Benchmarking. Table shows the performance of the four animals when optimized for each locomotion task, each environment. We parameterize the controller based on set of sine function bases with different frequencies, phases, and biases (see Section H). We perform con- trol optimization with differentiable physics and RI. (Schulman al., 2017), corresponding the left/right values in every entry respectively. We would like to emphasize that this experiment is not superiority of differentiable physics RL. Rather, highlight in-
meant to draw conclusion teresting emergent strategies of different morphologies find that the Baby Seal morphology is particularly well-suited for Ground and Ice, the Caterpillar morphology is well-suited for Wetland and Clay, the Fish morphology well-suited for Desert and Snow, and both the Caterpillar and Fish morphologies are well-suited for the Ocean.
response to diverse environments. A We
Environmentally- and in Figure 2, we showcase several interesting emergent motion of different animals. The Fish surpris- ingly demonstrates efficient movement on granular materials (Desert), by lifting its torso to reduce friction and pushing itself forward with its tail. The Caterpillar produces a left-to-right undulation motion to locomote in Werland; the strategy allows it to disperse mud as it moves forward. The Panda's 's legs act in a spring-like fashion to perform galloping motion on flat terrain. Finally, the Baby Seal uses its tail muscle to create a flapping motion to move on the low-friction Ice terrain, where push-off would otherwise be difficult.
saahooooicalll-driveee Motion. Based on previous quantitative analysis,
5

---

Published as a conference paper at ICLR 2023
Table 2: Quantitative comparison of design space representations.
imfich Function
DIT CPPN
Representation
Performance
Particle
0.027
Voxel
0.023
Barycenter
Implicit Function
0.113
Diff-CPPN
0,091
SDF-Lerp
0.152
Wass BasseBaryeentrr
0.158
Figure 3; Visualization of optimized designs.
1.00
Matching Relative Eiror
0.59
Rolative Matching
0.25
-0.2
0.50
0.75
Stffiness Multiplier
Number Actuators
(a) Under different stiffness.
(b) Under different muscle placement.
Figure 4: Ambiguity of muscle formation and controller synthesis.
3.2 THE IMPORTANCE OF DESIGN SPACE REPRESENTATION
In this section, demonstrate the importance of the design space representation for morphology optimization. We apply gradient-based optimization methods across a wide variety of morphological design representations fixing the controller. We focus on achieving fast movement speed in the Ocean environment since aquatic creatures tend to manifest a more canonical muscle distribution, typically antagonistic muscle pairs along each side of the body. The low failure rate of this task means that we achieve wide spread of performances, determined by the design representation, which allows us to compare the effect of each on performance.
Baselines. We implement variety of design space representations. We briefly mention the high- level concept each method as follows. Please refer to Section for more in-depth description.
Particle Representation. The geometry, stiffness, and muscle placement is directly specified at the particle level, with each particle possessing its own distinct parameterization.
Voxel Representation. This is similar to particle representation but specified at the voxel level.
Implicit Function (Mescheder et al., 2019). We use shared multi-layer perceptron that takes in particle coordinates and outputs the design specification for the corresponding particle.
CPF-CCPPN (Fernando et al., 2016). We adapt the differentiable CPPN, which provides global mapping that converts particle coordinates to a design specification.
SDF-Lerp. Given a set of design primitives (with design specification obtained by using the tech- nique in Section 3.1), compute SDF based on each design primitives for particles in robot design workspace. For each particle, we then linearly interpolate the signed distances and set occupancy for those with negative values to obtain robot geometry. We directly perform linear interpolation on stiffness and muscle placement of the primitive set. For muscle direction, we use weighted rotation averaging with special orthogonal Procrustes orthonormalization (Brégier, 2021).
Wass-Barycenter (Ma al., 2021). We compute Wasserstein barycenter coordinates based on of coefficients to obtain robot geometry. We follow SDF-Lerp for stiffness and muscle placement.
Design Optimization. In Table 2, show the results of design optimization. We can roughly categorize these design space representations into (1) no structural prior (Purticle and Voxel) (2) preference for spatial smoothness (Implicit Function and Diff- CPPN) and (3) highly-structured de- sign basis (SDF-Lerp and Wass-Bard Baryceeare) We observe superior performance as increasing ductive prior is injected into the representation. The use of design primitives casts the problem to composition of existing functional building blocks and greatly reduces the search space, leading

---

Published as a conference paper at ICLR 2023
2.634
0.455
-0.400
-3.990
-0.45
-2388
0450
-22002
-445
-23683
0.6
Varying
5. Shape Icentered at 0.51
(a) Multiple local optimum.
(b) Discontinuity.
Figure 5: Examples of pathological loss landscape for design optimization.
effective optimization. Further, structural priors, lead to designs with fewer thin sharply changing features; such extreme designs have unwanted artifacts such as numerical fracture during motion. Figure 3 presents optimized designs: meshes indicate geometry and colored point clouds indicate active muscle groups. These results suggest the development of novel representation for robot design that facilitates improved optimization and learning.
3.3 AMBIGUITY BETWEEN MUSCLE STIFFNESS AND ACTUATION SIGNAL STRENGTH
In this section, we investigate the ambiguity between musele formation and actuation from the troller. We aim to discover if there cxists unidentifiability between design and control optimization. We adopt trajectory optimization for control in this set of experiments since it has the best flexibility.
Stiffness as "Static" Actuation. Here, we explore whether optimizing active actuation can repro- duce motion of robots with different static muscle stiffness. With a fixed robot geometry and muscle placement, we randomly sample 100 sets of controllers and muscle stiffness, and record trajectories of all particles for 100 frames. We then try to fit a controller for robots with a different set of stiff- ness and measure how well they match the last frame of the pre-collected dataset. We the Earth Mover's s Distance (EMD) loss (Achlioptas et al., 2018) and differentiable physics for training. Specifically, we use EMD to compare the difference between the motions of each particle set at the end of each trajectory. In Figure 4a, we show matching error distribution across different stiffieess multipliers (with respect to the original stiffness of the robot). This experiment indicates we can roughly replicate the motion of robot with different stiffness by control optimization. This aligns with the muscle model, where we can interpret stiffness as a static component of actuation.
Approximating Motion of Random Muscle Placement. Here, investigate if we reproduce the motions of robots with different muscle placements by optimizing their controllers. With fixed robot geometry and stiffness, we randomly sample 6 sets of controllers and muscle placements with 3 actuators. We follow the same training procedure mentioned previously to controllers for robots with different set of muscle placement. In Figure 4b, we show relative matching error at different number of actuators with shaded area confidence interval from the 6 random seeds. Note that we adopt a fixed muscle direction and soft muscle placement for simplicity. The plot suggests that with redundant (> 3 in this case) actuators, robot motion can be roughly reproduced with control optimization even under different muscle placement.
Overall, we demonstrate the ambiguity between muscle construction and actuation that may induce challenges co-design optimization. However, these results also unveil the potential of casting muscle optimization to control optimization in soft robot co-design.
3.4
THE GOOD AND BAD OF DIFFERENTIABLE PHYSICS FOR SOFT ROBOT CO-DESIGN
The Value of Differentiable Physics. Gradient-based control optimization methods powered by dif- ferentiable simulation, are increasingly popular in computational soft robotics. Such optimizers can efficiently search for optimal solutions and decrease the number of computationally-ing intensive sim- ulation episodes needed to achieve optimal results in various computational soft robotics problems compared to model-free approaches such evolutionary strategies or RL. Despite its effectiveness, the local nature of gradient-based methods poses issues for optimization. While recent research has rigorously studied differentiable physics in control (Suh et al., 2022), similar investigation is lacking in the context of design. Here, we take a preliminary step analyze gradient for design optimiza- tion. In Figure 5a, we show the loss landscape of a single parameter of actuator placement. We consider voxel grid and smoothly (with soft actuator placement) transition the membership of the
7

---

Published as a conference paper at ICLR 2023
Table 3: The efficacy of co-optimization.
Table 4: Co-design with gradient-frec/based methods.
Optimization Target
Performance
Method
Performance
Controller
0.107
Evolution Strategy
0.247
Design
0,152
Diff-physics
0.332
Design + Control
0.332
Evolution
Strateat
Dift. Physics
Frame 5
Frame 30
Frame 60
Frame 90
Figure 6: Swimming motion sequences of co-design baselines.
actuator in voxel. We can observe multiple local optima, which may trap gradient-based based optimiza- tion. In Figure 5b, we show the discontinuity of loss resulted from shape changes. A box with a missing corner (red) ends up very different location from that of complete box (gray), as it topples when placed in metastable configurations. These results present unique yet understandable challenges for differentiable physics in robot design, which suggests more future research to better leverage gradient information.
3.5 CO-OPTIMIZING SOFT ROBOTIC SWIMMERS
In this section, we demonstrate co-design results for a swimmer in the Ocean environment. First, in Table 3, we compare between (I) optimizing control only (2) optimizing design only (3) co- optimizing design and control. We use SDF-Lerp as design space representation and differentiable physics for optimization. To generate meaningful results, for the control-only case, we use a fish- like design; and for the design-only case, we use controller trained for fish included in the design primitive of SDF-Lerp. The outcome verifies the effectiveness of generating superior soft robotic swimmer with co-design. In Table 4, we compare co-design results from evolution strategy (ES) and differentiable physics. We follow the above-mentioned setup for differentiable physics. For evolution strategy, we adopt common baseline: CPPN representation (Stanley, 2007) with HyperNEAT (Stanley et al., 2009); to make control optimization consistent with evolution strategy, we use CMA-ES (Hansen et al., 2003). Furthermore, we show the optimized design and the corresponding motion sequence of both methods in Figure Interestingly, the better- eetter-performing co- design result much resembles the results of performing design optimization only, shown in Figure 3.
4 RELATED WORK
Co-Design of Soft Bodied Machines. Evolutionary algorithms (EAs) have been used design virtual agents since the pioneering work of Sims (1994); as these algorithms improved the prob- lems they could be applied to grew in complexity. Cheney et al. (2014b) demonstrated EAs for co- designing soft robots over shape, materials, and actuation for open-loop cyclic controllers; follow-on work explored soft robots with circuitry (Cheney al., 2014a) and those grown from virtual cells (Joachimezak et al., 2016). Evolutionarily designed soft robots were demonstrated to be physically manufacturable using soft foams (Hiller & Lipson, 201 I), and biological cells (Kriegman et al., 2020). Virtually all such results have been powered by the neuroevolution of augmenting topologies (NEAT) algorithm (Stanley & Miikkulainen, 2002), and, typically, compositional pattern produc- ing networks (CPPN) (Stanley, 2007) for morphological design representation. Recent work has explored other effectual heuristic search algorithms, such as simulated annealing, for optimizing locomoting soft robots represented by grammars (Van Diepen & Shea, 2019; van Diepen & Shea, 2022). Recently, Bhatia et al. (2021) provided an expansive suite of benchmark tasks for evaluating algorithms for co-designing soft robots with closed-loop controllers. That work provides baseline co-design method that employs CPPN-NEAT for morphological search and reinforcement learning for control optimization (Schulman et al., 2017). Similar to (Bhatia et al., 2021), our work bench- marks EAS alongside competing methods. We provide suite of ceriirommentallyddiver tasks in
8

---

Published as a conference paper at ICLR 2023
differentiable simulation environment, which allows the use of efficient gradient- based search algo- rithms. Further, we focus on a systematic decoupling of design space representation, control, task and environment, and search algorithms in order to distill the influence of each.
Differentiable Simulation for Soft Robot Optimization. A differentiable simulator is one in which the gradient of any measurement of the system can be analytically computed with respect to any variable of the system, which can include behavioral (control) and physical (morphological and en- vironmental) parameters. Differentiability provides particular value for soft robotics; gradient- based optimization algorithms can reduce the number of (typically expensive) simulations needed to solve computational control and design problems. Differentiable material point method simulation (MPM) has been used in co-optimizing soft robots over closed-loop controllers and spatially varying mate- rial parameters (Hu al., 2019c; Spielberg et al., 2021). as well as proprioceptive models Spielberg et al. (2019). Co-optimization procedures rooted in differentiable simulation have also been ap- plied to more traditional finite element representations of soft robots. Notably, (Ma et al., 2021) demonstrated efficient co-optimization of swimming soft robots' geometry. actuators, and control, incorporating learned (differentiable) Wasserstein basis for tractable search over ligh-dimensional morphological design spaces, Follow-on work further showed that differentiable simulators can be combined with learned dynamics for co-design (Nava et al., 2022). Such finie-eleement-bas representations, however, have struggled with smoothly differentiating through rich contact.
Our work presents a rich comparison of differentiable and non-differentiable approaches for soft- bodied co-design, well as set of baseline methods. We present differentiable soft-bodied simulation environment capable of modeling multiphysical materials. Our environment is based in MPM, because of its ability to naturally and differentiably handle contact and multiphysical cou- pling- Similar to other recent work, such as Suh et al. (2022) and Huang et al. (2021), we perform a thorough analysis (in our case, empirical) of the value and limitations of differentiable simulation environments, with an emphasis on cyberphysical co-design.
Envirommenntaly--Diieee Computational Agent Design. Similar to their biological counterparts, virtual creatures have different optimized forms depending on their environment. Auerbach & Bon- gard (2014) and Corucci et al. (2017) analyzed real-world biological morphological features from the perspective of EAs; Cheney et al. (20155) studied the interplay between of virtual terrain and robot geometry for soft-bodied locomotion locomotion. In rigid robotics where simulation is less com- putationally expensive, online data-driver neural generative design methods are emerging. These include methods that reason over parameterized geometries Ha (2019). topological structure Zhao et al. (2020); Xu et al. (2021); I); Hu et al. (2022), and shapeshifting behavior Pathak et al. (2019). These methods currently require too much simulation-based data generation for soft robotics appli- cations, but are useful as inspiration for future research directions.
CONCLUSION
In this work, we introduced SoftZoo, soft robot co-design platform for locomotion in diverse environments. By using SoftZoo to investigate the interplay between design representations, en- vironments, tasks, and co-design algorithms, by environmental and morphological factors in a way that often mirrors nature 2) injecting design priors in design space representation can improve optimization effectiveness 3) muscle stiffness and controllers introduce redudnancy in design spaces, and 4) trapping local minima are common even in very simple morphological design problems. Impact and Future Direction. SoftZoo provides a eell-established platform that allows for systematic training and evaluation of soft robot co-design algorithms; we hope it will accelerate algorithmic co-design development. It also lays a cornerstone for studying morphological and behavioral intelligence under diverse environments. The exten- sive experiments conducted using SoftZoo not only sheds light on the significance of co-design but also identifies concrete challenges from various perspectives. Based
have found that /) emergent motions are driven
findings, we suggest several future directions: 1) 3D representation learning to construct more effective and flexible de- alternative formulation that
sign space representations 2) morphology-aware policy learning more elegantly handles the inter-dependency between design and control optimization 3) principled approaches to combine differentiable physics and gradient-fir free methods like RL or EAs that mar- ries priors from physics to optimization techniques less susceptible to pathological loss landscapes. Overall, we believe SoftZoo paves a road to study morphological and behavioral intelligence, and bridges soft robot co-design with variety of related research topics.

---

Published as a conference paper at ICLR 2023
REFERENCES
Panos Achlioptas, Olga Diamanti, Ioannis Mitliagkas, and Leonidas Guibas. Learning representa- tions and generative models for 3d point clouds. In International conference on machine learning, pp. 40-49. PMLR, 2018. 7
Joshua E. Auerbach and Josh C Bongard. Environmental influence on the evolution of morphological
complexity in machines. PLoS computational biology, 10(1)ee 2014. 9
Jagdeep Bhatia, Holly Jackson, Yunsheng Tian, Jie Xu, and Wojciech Matusik. Evolution gym:
A large-scale benchmark for evolving soft robots. Advances in Neural Information Processing Systems, 34:2201-2214, 2021. 1, 2, 8, 15
Romain Brégier. Deep regression on manifolds: 3d rotation case study, In 2021 International Conference on 3D Vision (3DV), pp. 166-174. IEEE, 2021. 6, 18, 19
Nicholas Cheney, Jeff Clune, and Hod Lipson. Evolved electrophysiological soft robots. ALIFE 14: The Fourteenth International Conference the Synthesis and Simulation of Living Systems, pp. 222-229. MIT Press, 2014a. 8
Nick Cheney, Robert MacCurdy, Jeff Clune, and Hod Lipson. Unshackling evolution: evolving soft robots with multiple materials and powerful generative encoding. ACM SIGEVOlution, 7(1): 11-23, 2014b. 8
Nick Cheney, Josh Bongard, and Hod Lipson. Evolving soft robots in tight spaces. In Proceedings of the 2015 annual conference on Genetic and Evolutionary Computation, pp. 935-942 2015.
Francesco Corucei, Nick Cheney, Sam Kriegman, Josh Bongard, and Cecilia Laschi. Evolutionary developmental soft robotics as framework to study intelligence and adaptive behavior in animals and plants. Frontiers in Robotics and AI, 4:34, 2017.
Chrisantha Fernando. Dylan Banarse, Malcolm Reynolds, Frederic Besse, David Pfau, Max Jader- berg, Mare Lanctot, and Daan Wierstra. Convolution by evolution: Differentiable pattern produc- ing networks. In Proceedings of the Genetic and Evolutionary Computation Conference 2016,
pp. 109-116, 2016. 6, 18
Oscar Gonzalez and Andrew Stuart. A first course in continuum mechanics, volume 42, Cam- bridge University Press, 2008. 15
Moritz A Graule, Thomas P McCarthy, Clark B Teeple. Justin Werfel, and Robert J Wood. Somo- gym: A toolkit for developing and evaluating controllers and reinforcement learning algorithms for soft robots. IEEE Robotics and Automation Letters, 7(2): 40788, 2022. 2, 15
David Ha. Reinforcement learning for improving agent design. Artificial life, 25(4):352 2019.
Nikolaus Hansen, Sibylle D Müller, and Petros Koumoutsakos. Reducing the time complexity of the derandomized evolution strategy with covariance matrix adaptation (cma-es). Evolutionary computation, 11(1):1-18, 2003. 8, 16
Jonathan Hiller and Hod Lipson. Automatic design and manufacture of soft robots. IEEE Transac- tions on Robotics, 28(2) 466, 2011.
Jiaheng Hu, Julian Whitman, Matthew Travers, and Howie Choset. Modular robot design optimiza- tion with generative adversarial networks. 2022 International Conference on Roboties and Automation (ICRA), pp. 4282-4288. IEEE, 2022. 9
Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions Graphics (TOG), 37(4):1-14, 2018. 3, 14
Yuanming Hu, Luke Anderson, Tzu-Mao Li. Qi Sun, Nathan Carr, Jonathan Ragan Beean-Kelley, and Frédo Durand. Difftaichi: Differentiable programming for physical simulation. arXiv preprint arXiv:/910.000 00935, 2019a. 15
10

---

Published as a conference paper at ICLR 2023
Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Bagan-Kelley and Frédo Durand. Taichi: language for high-performance computation on spatially sparse data structures. ACM Transac- lions Graphics (TOG), 38(6):1-16, 2019b. 3
Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua B Tenenbaum, William T Freeman, Jiajun Wu, Daniela Rus, and Wojeiech Matusik. Chainqueen: A real-time differentiable physical simu- lator for soft robotics. In 2019 International conference on robotics and automation (ICRA), pp.
6265-6271. IEEE, 2019c. 9
Zhiao Huang, Yuanming Hu, Tao Du, Siyuan Zhou, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Plasticinelab: soft-body manipulation benchmark with differentiable physics. arXiv preprint arXiv:2/04.03311 2021.
Michat Joachimezak, Reiji Suzuki, and Takaya Arita. Artificial metamorphosis: Evolutionary design of transforming, soft-bodied robots. Artificial life. 22(3):271-299 2016. 8
Robert K Katzschmann, Joseph DelPreto, Robert MacCurdy, and Daniela Rus. Exploration of un- derwater life with an acoustically controlled soft robotic fish. Science Robotics, 3(16):eaa33449, 2018.
Sam Kriegman, Douglas Blackiston, Michael Levin, and Josh Bongard. A scalable pipeline for designing reconfigurable organisms. Proceedings of the National Academy of Sciences, 117(4): 1853-1859, 2020. 8
Pingchuan Ma, Tao Du, John Z Zhang, Kui Wu, Andrew Spielberg, Robert K Katzschmann, and Wojciech Matusik. Diffaqua: A differentiable computational design pipeline for soft underwater swimmers with shape interpolation. ACM Transactions on Graphics (TOG), 40(4):1 2021. 1, 3, 3,6,9. 15, 19
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Oc- cupancy networks: Learning 3d reconstruction in function space. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp.
2019. 6, 17
Elvis Nava, John Zhang, Mike Yan Michelis, Tao Du, Pingchuan Ma, Benjamin F Grewe, Woj- ciech Matusik, and Robert Kevin Katzsehmann. Fast aquatic swimmer optimization with differ- entiable projective dynamics and neural network hydrodynamic models. International Confer- ence on Machine Learning, pp. 16413-1642) PMLR, 2022. 9
Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexei A Efros. Learning to control self-assembling morphologies: study of generalization via modularity. Advances in Neural Information Processing Systems, 32, 2019. 9
Steffen Puhlmann, Jason Harris, and Oliver Brock. Rbo hand 3: A platform for soft dexterous manipulation. IEEE Transactions on Robotics, 2022. 1
Alireza Ramezani, Xichen Shi, Soon-Jo Chung, and Seth Hutchinson. Bat bot (b2), biologically inspired flying machine. In 2016 IEEE International Conference on Robotics and Automation (ICRA), pp. 2219-3226 IEEE, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.063477 2017. 5, 8, 16
Karl Sims. Evolving virtual creatures. In Proceedings of the 21st annual conference on Computer graphics and interactive techniques, pp. 15-22, 1994. 8
Andrew Spielberg, Allan Zhao, Yuanming Hu, Tao Du Wojciech Matusik, and Daniela Rus. bearning-in-the-loon optimization: End-to-end control and co-design of soft robots through learned deep latent representations. Advances in Neural Information Processing Systems, 32, 2019. 9
Andrew Spielberg, Tao Du, Yuanming Hu, Daniela Rus, and Wojeiech Matusik. Advanced soft robot modeling in chainqueen. Robotica, pp. 1-31, 2021. 1,9
Kenneth O Stanley. Compositional pattern producing networks: novel abstraction of development.
Genetic programming and evolvable machines, 8(2):131-162, 2007. 8, 18
11

---

Published as a conference paper at ICLR 2023
Kenneth O Stanley and Risto Mikkkulainen. Evolving neural networks through augmenting topolo- gies. Evolutionary computation, 10(2):99-127, 2002.
Kenneth O Stanley, David D' A'mrroso, and Jason Gauci. A hypercube-based encoding for evolv- ing large-scale neural networks. Artificial life, 15(2):185-22) 2009. 8, 16
Alexey Stomakhin, Craig Schroeder, Lawrence Chai, Joseph Teran, and Andrew Selle. A material point method for snow simulation. ACM Transactions on Graphics (TOG), 32(4):1-10, 2013. 3
Hyung Suh, Max Simchowitz, Kaiqing Zhang, and Russ Tedrake. Do differentiable simulators give better policy gradients? In International Conference on Machine Learning, pp. 20668- 20696. PMLR, 2022. 7,9
Yichao Tang, Yinding Chi, Jiefeng Sun, Tzu-Hao Huang, Omid H Maghsoudi, Andrew Spence, Jianguo Zhao, Hao Su, and Jie Yin. Leveraging elastic instabilities for amplified performance: Spine-inspired high-speed and high-force soft robots. Science advances, 6(19):eaaz6912, 2020.
Merel Van Diepen and Kristina Shea. A spatial grammar method for the computational design synthesis of virtual soft locomotion robots. Journal of Mechanical Design, 141(10), 2019. 8
Merel van Diepen and Kristina Shea. Co-design of the morphology and actuation of soft robots for locomotion. Journal of Mechanical Design, 144(8)0083305, 2022. 8
Jie Xu, Andrew Spielberg, Allan Zhao, Daniela Rus, and Wojciech Matusik. Multi-objective graph heuristic search for terrestrial robot design. In 202/ IEEE International Conference on Robotics and Automation (ICRA), pp. 9863-986680 IEEE, 2021.
Allan Zhao, Jie Xu, Mina Konaković-Lukovis Josephine Hughes, Andrew Spielberg, Daniela Rus, and Wojciech Matusik. Robogrammar: graph grammar for terrain terrain-optimized robot design. ACM Transactions Graphics (TOG), 39(6):1111 1-66, 2020. 9
12

---

Published as a conference paper at ICLR 2023
A NOTATIONS
Table A shows a list of notations used in the description of SoftZoo, where the subscript indicates association with particles. Note that actuation defined per particle and action vector defined as the output of controllers (which can be very low-dimensional such as R! i.c., Op = u fp.
Symbol
Type
Description
mp
sealar
mass
Sp
scalar
stiffness
Op
scalar
actuation
vector
action vector
Xp
vector
position
Vp
vector
velocity
Fp
vector
muscle placement membership
Ep
vector
muscle direction
P
matrix
deformation gradient
Op
matrix
Cauchy stress
C
matrix
affine velocity field
B LOCOMOTION TASKS
B.I MOVEMENT SPEED
In this task, the goal of the robot is to move fast possible along a pre-defined direction. We compute average velocity of existing (non-zero-mass) particles on robot body and project to the target direction to obtain scalar estimate. We thus write the metric per timestep as,
Σ E
Tspeed
where my is mass, vp is velocity, and is pre-defined target direction.
B.2 TURNING
In this task, the robot is encouraged to turn as fast as possible counter-clockwin about the upward direction of robot's canonical pose. We first compute the relative position of all particles on robot body with respect to its center mass. We then compute the cross product the upward direction followed by unit-vector normalization to obtain tangential directions for all particle while turning. Finally, we compute velocity projection every particle along the tangent and return the average as the measure of turning performance. We thus write the metric per timestep as,
Yturning = -mmp
Vp vpll
||d
where mp is mass, vp is velocity, and d is the up direction (defined manually based robot canon- ical forward direction).
B.3 VELOCITY TRACKING
In this task, the robot is required track series of timestamped velocities. We use quintic poly- nomials formulated as a function of time for path parameterization since it generates smooth tra- jectories with easy access to different orders of derivative. A path be fully specified with start and target position X velocity v.. Ve, and acceleration a,, a,. can then obtain quintic polynomial as,
Xyut((t) Ca cxt cxt cgt + cat cat
13

---

Published as a conference paper at ICLR 2023
where t is time and Cu, C2, Ca. Ca. C5 R are coefficients. After setting target state, query first-order derivative of the polynomial at each environment time step as the target velocity,
V Vpan (t) = C1 + 2cpt + 3czt² + 4c,t + 5cgt
We write target velocity time instance as Vurget for the following. Furthermore, due to the deforming nature of soft robot, need estimate the heading of the robot in order compare with the target velocity. We manually label particles corresponding to a head and a tail of the robot priori Phead Pail based on its canonical heading direction.
Pheal
X
d
P Xpaal
During robot motion, extract rotation from deformation gradient of these particles and inversely transform them back to material space to compute heading direction.
Fp U.
proj = (R vR-
where F2 is deformation gradient, Up. R are the results of polar decomposition. Velocities of all particles on robot body are then projected to the heading direction and averaged in order to compare to the target velocity. We separate the measurement of magnitude and direction as this allows different weighting the two terms. We can then write the metric per timestep as,
Tvll-rnck mmagm'ma..p mag.p + AdirPd
where Tmag.F"
- V
||
|/rro 1])
PPojj
ddir
Vuaget
|
Proj
Σ
mp v ?
where O'mag Oldir are coefficients of magnitude and direction term respectively and we Omag 0.1 and Oddir = 0.9. We put more emphasis on the alignment direction since better indicates maneuverability.
B.4 WAYPOINT FOLLOWING
In this task, the robot needs to follow sequence of waypoints. The waypoints generated by the abbve-mentioned quintic polynomial method Xpath (t). We compute root mean squared error between robot center of mass and the target waypoint for each time step. We denote target position at time instance as Xsarget. can then write the metric per timestep as,
Σ m
Twaypoint
(Xcorget
Remark that well-performing velocity tracking can induce large waypoint following error but trace out similar-shaped yet different-scaled trajectories. Both serve role as distinct aspects of evaluating path following.
C PLATFORM COMPARISON
comprehensive comparison to existing soft robot platform is shown in Table
D CONTINUUM MECHANICS SIMULATION
We formulate the continuum mechanics simulation in the framework of the moving least squares material point method (MLP-MPMI) (Hu et al., 2018), whose governing equations characterized by:
Dv V P DE
pfct
Dp
P DL.
-p V,
(2)
14

---

Published as a conference paper at ICLR 2023
Table 5: Comparison existing soft robot platforms.
Platform
Sinalatiio Method
Tasks
Design
Control
Differentihility
Mulüiphysccll Materials
Mostl
SoMoGym (Graule al., 2022)
Rigid-link System
Manipulation
Dit (Ma et al,, 2021)
FEM
Swimmer
v
2D Mass-spring System
Locomotion
EvoGym (Bhatia all, 2021)
Manipulation
SoftZoo (Ours)
MPM
Locomotion
Table The physical phenomenons that each environment covered in our multiphysical simulation.
Environment
Elasticity
Plasticity
Fluid
Friction
Ground
High
Desert
V
Wetland
Mixed
Clay
V
Ice
Low
Snow
Shallow Water
Shallow
Ocean
Deep
where p is the density of the material, v the velocity, or is the Cauchy stress of the energy, and fox is the external forces applied, which is the gravity in cases. We solve these equations for an equilibrium between different materials coupled our environments. We will not dive further into continuum mechanics and point the interested reader to Gonzalez & Stuart (2008) for more details. In terms of the implementation the differentiable physics-based simulation, we massively use DifiTaichi (Hu et al., 2019a) as the backbone.
E MULTIPHYSICAL MATERIALS
For result validation and visual entertainment, we present a diverse set of environments spanned by different material setups and tasks. Here we illustrate the materials that each environment covered in Table Note that even though Desert, Clay, and Snow share the same composition of mate- rial types, distinguish their elastoplasticity by imposing different parameters and models (e.g., friction cone). We will release the code-level implementation all materials for reproducibility.
F GRADIENT CHECKPOINTING
Simulating environments like ocean, desert, etc requires a significant amount of particles. This poses a challenge in differentiation the computation graph needs to be cached for backward pass, leading to considerably high memory usage. Accordingly, we implement gradient checkpointing that allows very large-scale simulation with gradient computation. Instead of caching simulation state every single step, we only store data every N steps. When doing backward pass at N steps, we perform recomputation of forward pass from (i 1) x N to x N to reconstruct the computation graph in-between for reverse-mooe mode automatic differentiation.
G OPTIMIZATION
In this section, describe the implementation details of each optimization method.
15

---

Published as a conference paper at ICLR 2023
G.1 DIFFERENTIABLE PHYSICS
Model-based gradient provides much accurate searching direction and thus considerably more effi- cient optimization. However, gradient information is susceptible to local optimum and often leads to bad convergence without proper initialization. Hence, we adopt simple yet effective approach that samples 8 random seeds, performs optimization with differentiable physics for all runs, and picks the best result. More formally we write as,
0" F(C; arg min F(C, Oo),
80
where 9 are the model parameters, Og is the initialization of model, L is cost function, F summa- rizes the optimization that follows the gradient update 0x+1 = 8g 1hh( L). 77 is the learning rate, and h is function that specifies different gradient descent variants. For the large-scale benchmark with biologically-inpiree design (Table 1), we use learning rate 0.1 and training iterations 30. For all control-only and design-only optimization, we use learning rate 0.01 and training iterations 100. For co-design, we use learning rate 0.01 for both control and design with training iterations 250. We use Adam as the optimizer.
G.2 REINFORCEMENT LEARNING
RL is only used in control optimization. We use Proximal Policy Optimization (PPO) (Schulman et al., 2017) with the following hyperparameters: number of timesteps 10", buffer size 2048, batch size 32, GAE coefficient 0.95, discounting factor 0.98, number of epochs 20, entropy coefficient 0.001, learning rate 0.0001, clip range 0.2. use same controller parameterization as all other experiments throughout the paper.
G.3 EVOLUTION STRATEGY
We implement fully-E3 ES-based method a co-design baseline. The genome fitness function is set as the episode reward of the environment. We pose a constraint on connected component of robot body. For CPPN, selu, abs, log, exp. The inputs of CPPN include x, z coordinates along with distance along xy, yz planes and radius from the body center. We use HyperNEAT (Stanley et al., 2009) for design optimization and CMA-ES (Hansen et al., 2003) for control optimization with initial standard scheme for co-design. Instead, HyperNEAT and
of activation functions including sigmoid, tanh, sin, gaussian,
deviation as 0.1. We don't i use an inner-outer-lon CMA-ES share the same of population with population size 10. We ES for 100 generations for the co-design baseline.
H CONTROLLER PARAMETERIZATION
Locomotion often exhibits eyclic motion and thus control optimization can significantly benefit from considering periodic functions in controller parameterization. Specifically, we use a set of sine functions with different frequency and phase (offset) bases. The controller is hence parameterized with a set of weights on these bases along with bias terms.
ú(x, 1) = Σ aijois Bij, di((t) = (z)
where di are the bases, Cijy and 3ey are learnable weights and biases, x is controller inputs, (x),, 40, (z) can be either learnable (as neural network) or pre-defined. The control signal is then modulated by a tanh function multiplied by a pre-defined constant to ensure satisfication of control bound. We use 4 different phases with frequency 20 and 80 rad/s throughout the paper. While all experiments presented are not confined using this sinewave basis controller, we empirically found it extremely efficient to generate reasonable results in comparison to other controller parameteriza- tions like generic neural network trajectory optimization.
I DESIGN SPACE REPRESENTATION
In this section, we provide more implementation details of design space representations. We first recall the notation of robot design interface. mp E IR. is geometry modeled by mass. 8p RR is
16

---

Published as a conference paper at ICLR 2023
stiffness. Tp E RK is muscle placement (muscle group assignment) with K the maximal number of actuators/muscles. fp R is muscle direction represented in Euler angle. Tm is the cutoff threshold of excluding low mass regions for numerical stability. Note that the subscript p indicates attributes associated with particle. M= denotes a multi-layer perceptron with inputs, b parameters (we omit hidden layers for clean notation and the output layer is linear).
outputs, and mo. &0 E R are pre-defined and constant reference mass and stiffness respectively. Geometry of all methods are processed by the following formula for numerical stability.
mp 1 2 Mp
where I. is indicator function. Xp E R$ is centered and standardized position Xp E RR We denote the position of a base particle set {&p} as a point cloud representation that span the robot design workspace, which is natural representation of continuum material in MPM. Next, describe each design space representation individually.
1.1 PARTICLE-BASED REPRESENTATION
Given base particle set { we instantiate two trainable scalars followed by sigmoid for geometry and stiffness, and K --dimensionnl vector followed by softmax for muscle placement with fixed muscle direction along the canonical heading direction.
m mg Sigmoid
PBR so Sigmoid(3p)
pp
PBR Softmax(F,)
p
PPBR (())
where {mp 5p F, R* are learnable parameters associated with &p, and f( is the fixed muscle direction along the canonical heading direction.
1.2 VOXEL. BASED REPRESENTATION.
We voxelize the given base particle set to obtain voxel grid and follows similar modelling technique to particle-baser representation in voxel level.
mp
VBR mg
Sigmoid(V2P((m.))
VBR
8p Sigmoid(V2P($..)
VBR
Sofimax(V2P(F_)
Pp
where V2P is mapping from voxel space to particle space (e.g.. suppose
2 x 2
voxel grid to represent 10 particles, we may associate the voxel coordinate (0, 0, 0) first 53 particles), {my R, E F. [ are learnable parameters, and the muscle direction of the same definition particle-based representation PYBR f,BR, In comparison to particle-batt based representations, in practice the voxel-based representations have much fewer parameters that need to be learned.
1.3 IMPLICIT FUNCTION
We extend the idea of OccupancyNet (Mescheder al., 2019) predicting robot geometry, stiff- ness, and muscle placement. is modeled by a multi-layer perceptron (MIP) with layers and 32 dimensions for each. We use tanh activation. The MLP takes in y. coordinates, distance along xy, XZ, yz planes and radius from the body center. The network outputs occupancy as geometry using sigmoid, stiffness multiplier using sigmoid, and K Kiddimnnsional vector using softmax for muscle placement. We describe in formula as,
mi mg Sigmoid(MLPP. 3+1 (&p: 0m))
,I So Sigmoid(MLP (xp; 0.))))
"
IIF Softmax(MLP3--> (xp; 0r))
P
where (0m, 0s, 0r) are learnable parameters formulated as neural networks, and the muscle direction of the same definition as particle- representation flf pPAR This representation has inherent spatial continuity due to the use of smooth functions MI.P with spatial coordinate x, as inputs.
17

---

Published as a conference paper at ICLR 2023
1.4 DIFF-CPPN
Diff- Diff-CPPN is a differentiable version of Compositional Pattern Producing Networks (CPPN) (Stan- ley, 2007), following similar concept in (Fernando et al., 2016). CPPN is a graphical model G = E composed of set of activation functions Σ with interesting geometrie properties (e.g., sine, tanh) that takes in particle or voxel coordinates and output occupancy or other proper- ties. Each node n, E N has set of imput edges e, E & that can be changed by evolution, and an activation function o; E - The input-output relationship of a layer can be then written as,
oze
E wyn,))
0yEE
originally designed to be optimized with varying graph topologies. We use meta graph to allow gradient flow and mimic the augmenting topolgies process in NEAT yet in a differentiable manner, i.., the varying topology wy {0,1) is softened to W/ R. Remark the similarity of the above construction with a layer in regular MLP except for varying activation function across neurons. We then can define,
Dift-CPPN mh MiL PPMMMPPPPPPPC ()(()))0m)) ( mp
CCPN Din-CPPN = Sigmoid 3+1 (((x));0.)) p
Din-CPPN Softmax(MLP 3>R (p(x)))0)) 8x))
p
where each layer of MLPCPPN follows the aforementioned input-output relationship. (6m 0.. 9r) are learnable parameters, and g(-) is function of expanding position to additional spatial coordinates such distance the center (which is normally used in CPPN). We fixed muscle direction. The model takes in x, y. z coordinates, distance along xy, xz, yz planes and radius from the body center, and outputs occupancy a K K--dimensional vector using softmax for muscle placement. We use sin and sigmoid activation functions with 3 hidden layers and 20 graph nodes in each layer.
geometry using sigmoid, stiffness multiplier using sigmoid, and
I.5 SDR-LERP
Given a base particle set { we compute the linear interpolation among a set of pre-defined design primitives [v, where N is number of design primitives. The shape of the robot design my is then determined by a set of coefficients weighting the SDFs from design primitives. In other words, the trainable parameters for robot geometry only construct N-dimensional vector. We then compute weighted sum of the SDF bases and extract robot body with the final SDF smaller or equal to zero. We use low-temperature sigmoid in implementation to keep gradient flow. For stiffness Sp. we can directly perform linear interpolation. For muscle group membership rp, we use linear interpolation upon the one-hot vectors from design primitives and effectively realize soft muscle group assignment. For muscle direction fp. we adopt interpolation designed for rotation matrices (Brégier, 2021). Formally,
LDr-Lar m
mo
1 TEN & (Xp))
à;
Σ, 0j
N
3, 8 Σ9 Bj
SDF Leep
(x,,),
p
1=1
SDF-1
Σ
(x))))))
9/
%
pp
Σ3 To
1=1
N
P
SDP-Lep
M2E aarg min| R
R,E2M V (.))))))).
R RF Σ,
RESO(3)
where {a E R. 3 E RN, RM,K RR are learnable coefficients of the interpolation, SDD 1 V. V.. are the SDF, stiffness, muscle placement, and muscle direction of design primi- tive respectively, T is temperature (where we set to - 1000 to mimic SDF < 0), E2M conversion from an Euler angle to a rotation matrix, M2E is conversion from rotation matrix to Euler
SDD-Lerp follows rotation angle, || || is Frobenius norm The computation of muscle direction r
18

---

Published as a conference paper at ICLR 2023
Table 7: Full RL results of Table 1, Each entry is the mean and standard deviation of random seeds. The higher the better.
Task
Animal
Ground
Ice
Wetland
Rnvinonment Clay
Desert
Snow
Ocean
Baby Seal
0.154 +: 0.096
0.010 = 0.010
0.020
0.004
0.008
0.002
0,034 + 0.010
0.016 + 0.011
0.029 + 0.003
Movement
Caterpillar
0,032 0.020
0,006
0.005
0,016
0.005
0.015
-0.004
0.032 00.004
0.017 0.003
0.181 0.014
Speed
0433
0.011 0.009
0,013
0.014
0.003
0.022 0.005
0.042 0.017
0.151 D011
0.19 0.008
0.006
0.005
0.008
0.005
0.004
Rohy Seal
0.077 + 0.020
0.024
0.009
0.008
0.003
0.011
0.001
0.026 + 0.011
0.028 + 0.009
0.020 + 0.026
Tuming
Caterpillar
Fish
0,047 0.010
0.021 +: 0.007
0,012
0.009
0.009
0.005
0,010
0.006 0.002
0.006
0.005
0.007
0.002
0.003
0.019 0.008
0.015 : 0.008
0.006 + 0.005
0.029 1: 0.019
0,013 0.005
0.358 4 0.024
0.007
Roby
0.208
0.265
Velocity
Caterpillar
0.368 +: 0.081
0.156 0.388
0.192 0.034
0.058
0.028
0.376 + 0.047
0.133 + 0.221
0.714 + 0.072
Tracking
Fisb
0.256 0.113
0.319 = 0.123
0.236 0.026
0.303 0.070
0,311 + 0.157
0.307 + 0.127
0.574 4 0.141
Panda
0.124 0,049
0.363
0.139
0.082 0.534
0,195
0.063
0.359 0.034
0,258 0.073
0,220 1 0.268
-0.027
-0.027
Waypoint
Caterpillar
0.016 :: 0.001
-0.028 + 0.003
-0.028
0.000
-0.027
0.000
-0027
0.001
-0026
0.001
-0019 0.007
Following
Fish
4.016 = 0.005
-0.026
+ 0.002
.0.029 + 0.003
.0.02
+ 0.003
0.024
0.001
0.024
0.001
0.023 0.000
Panda
0.0144 =: 0.004
-0.025 4: 0.004
.0.027 1: 0.001
-0.028
1: 0.000
00.04 1 0.001
00.05 2: 0.001
0.006 0.000
matrix interpolation using special Procrustes method, detailed in (Brégier, 2021). In comparison to the above methods, SDF-Lerp leverages prior knowledge of design primitives Vi that can provide more structure to the optimization from design space representation.
1.6 WASSERSTEIN BARYCENTER
This method also uses design primitives { [V))
First, it adopts the same approach for stiffness Wass SDD-Lerp We use a fixed muscle SDI-Lerp , rp
and muscle placement as SDF-Lerp * Wass xp direction along the canonical heading direction. The major difference is the way to represent robot geometry. Following (Ma al.. 2021), we define a probability simplex (i.e., set of coefficients with length as the number of design primitives) that serves as a weighting in the sense of Wasserstein distance among different shapes. It can be written as,
Wass mp
mg (xp)
where Pa is the Wasserstein barycenter with coefficients (intuitively, with some abuse of notation, we can view it as probability density function that determines the particle existence), defined as,
Pa Σ W
// 1/xx-xx,)ddxx dd (xp: x,)
W2(P.Q)
inf [
where W/ 2- Wasserstein distance and I is shape primitive. This representation better preserves the volume from the shape of design primitives. We refer the reader to the original paper for more details.
J FULL RL RESULTS
In Table we show large-scale benchmark of peglooically-inspiini designs in SoftZoo. The com- parison includes optimization using differentiable physics and RI. We use 5 random seeds for RI. results yet only report average performance due space limit. In Table 7, we report the full RL. results with both mean and standard deviation among all seeds.
K VISUALIZATION OF BIOLOGICALLYYINSPIEER DESIGN
In this section, we demonstrate the biologically-inspiee designs used in this paper. In Figure we show the four animals used in the main paper, including Baby Seal, Caterpillar, Fish, and Panda, In Figure 8, we show the set of design primitives used in SDF-Lerp and Wasserstein Barycenter.
19

---

Published as a conference paper at ICLR 2023
Baby Seal
Caterpillar
Fish
Panda
Figure 7: Visualization of animals for biologically -innsprred design.
Fish 4
Orca
Shark
Figure 8: Visualization of fish-like design primitives.
20


Reviews:
[{"review_id": "xSgYez3vcP", "reviewer": "Reviewer_wyCA", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a platform for soft robot codesign research. This platform provides a wide range of environments and tasks. Large-scale experiments are performed on design and control optimization.", "summary_of_the_review": "This paper provides a useful platform to conduct research in soft robot design and control. Experiments are shown to showcase the various features of the platform and provide insights and suggestions for future research direction. Although it is unclear whether the proposed platform can be useful for real robot design, I believe this is still a nice contribution to the learning design/control community. ", "strength_and_weaknesses": "Pro:\n\n1. Introduction of a platform for design/control co-optimization for soft robots.\n\n2. Large-scale experiments with insight that can guide future research direction on this topic. \n\nCon:\n\nIt is unclear whether the simulated platform is accurate enough such that the design and control coming out of the optimization can actually be used for real robots.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "prKLNRpDzOI", "reviewer": "Reviewer_yNAY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors introduced a framework (SoftZoo) that can co-design the morphology and controller for soft robots in diverse environments. SoftZoo explores a differentiable material point method (MPM) to simulate soft bodies and their interaction with different terrain types, and the soft robots are actuated through anisotropic elastic energy.  The authors constructed a set of biologically inspired soft robots and benchmarked their performances in various environments, and discussed the importance of design space representations and the ambiguity between design and control optimization.", "summary_of_the_review": "Overall the paper provides interesting (and large scale) results with plenty of baselines; The paper is not well written and needs improvement in its clarity. ", "strength_and_weaknesses": "Strengths of the paper:\n1) A thorough paper with plenty of results (i.e. large scale benchmarks for various robot morphologies and environments.\n2) Many baselines are compared for the design space choices, which makes the conclusion more convincing. \n3) The accompanying video provides\n\nWeaknesses of the paper:\n1) The paper is overall not clearly written with many missing details and leaves the readers to guess. For example, section 3.2 discussed various baselines used to perform gradient based design optimization. However, even combining the appendix it is still difficult to understand how the design optimization is actually performed. Another example is the colored muscle groups. The word “muscle group” has appeared twice in the paper without a proper definition. I assume this has something to do with the design optimization. \n2) The website is not well constructed with many illustrations missing. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "YB9iBjZVKDV", "reviewer": "Reviewer_zLWw", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a framework called SoftZoo for co-learning design and control of soft robots in a variety of locomotion environments. It includes the design parameterizations for designs and control policies, task definitions, environments created in the TaiChi differentiable simulator, and prior co-design algorithms. The paper also introduces experiments to analyze the problem of soft robot co-design from multiple perspectives: how biologically inspired morphologies perform in different environments, choice of design spaces, muscle stiffness, and the use of differentiable simulation. Finally they demonstrate co-optimization for a robotic swimmer.\n", "summary_of_the_review": "In summary, this paper presents a benchmark built on a differentiable simulator for the co-optimization of design and control of soft robots. It allows testing locomotion tasks in a variety of terrains. The paper is well-executed and provides valuable analysis, and I think this benchmark will be a nice contribution to the field. ", "strength_and_weaknesses": "Strengths:\n- The paper is well motivated, and the proposed platform seems like it will help to drive research in the direction of developing improved co-design algorithms. It supports a variety of design parameterizations and diverse terrain types.\n- Compared to prior platforms, SoftZoo is differentiable (unlike EvoGym) and supports more tasks (unlike DiffAqua) but only supports locomotion tasks (a weakness).\n- The experimental analysis is very insightful. I think that they highlight several of the most critical issues currently in the co-optimization of design and control. \n- The presentation of the paper is clear and it is well written. The visualizations of the demos are really nice!\n\nWeaknesses:\n- As mentioned earlier, the tasks are all locomotion tasks. Therefore their complexity is somewhat limited, especially because the different terrains may not require as much specialization in design compared to, say, manipulation tasks.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster