DittoGym: Learning to Control Soft Shape-Shifting Robots
Authors: Suning Huang
Boyuan Chen
Huazhe Xu
Vincent Sitzmann
Venue: ICLR 2024 poster (2024)
Keywords: Deep Reinforcement Learning
Soft Robot
Primary area: reinforcement learning
TL;DR: We develop algorithm and benchmark for reconfigurable robots, who can dynamically change their fine-grained morphology to accomplish a task.

Abstract: Robot co-design, where the morphology of a robot is optimized jointly with a learned policy to solve a specific task, is an emerging area of research. It holds particular promise for soft robots, which are amenable to novel manufacturing techniques that can realize learned morphologies and actuators. Inspired by nature and recent novel robot designs, we propose to go a step further and explore the novel reconfigurable robots, defined as robots that can change their morphology within their lifetime. We formalize control of reconfigurable soft robots as a high-dimensional reinforcement learning (RL) problem. We unify morphology change, locomotion, and environment interaction in the same action space, and introduce an appropriate, coarse-to-fine curriculum that enables us to discover policies that accomplish fine-grained control of the resulting robots. We also introduce DittoGym, a comprehensive RL benchmark for reconfigurable soft robots that require fine-grained morphology changes to accomplish the tasks. Finally, we evaluate our proposed coarse-to-fine algorithm on DittoGym,  and demonstrate robots that learn to change their morphology several times within a sequence, uniquely enabled by our RL algorithm. More results are available at https://dittogym.github.io.

Published as a conference paper at ICLR 2024
DITTOGYM: LEARNING TO CONTROL SOFT SHAPE- SHIFTING ROBOTS
Suning Huang
Boyuan Chen
Department of Automation
CSAIL
Tsinghua University
Massachusetts Institute of Technology
hsnl 90mai il 1s. ts iiigia...eeee.... on
boyuancêmit.edd
Huazhe Xu IIIS Tsinghua University
Vincent Sitzmann
CSAIL
Massachusetts Institute of Technology s sitzzannmiic
muazhe_xuêmal ts nggua edu.cn
ABSTRACT
Robot co-design, where the morphology robot optimized jointly with a learned policy to solve specific task, is an emerging area of research. It holds particular promise for soft robots, which are amenable to novel manufacturing techniques that can realize learned morphologies and actuators. Inspired by nature and recent novel robot designs, we propose to go a step further and explore the novel reconfigurable robots, defined robots that can change their morphology within their lifetime. We formalize control of reconfigurable soft robots as a high- dimensional reinforcement learning (RL) problem. We unify morphology change, locomotion, and environment interaction in the same action space, and introduce an appropriate, coarse-to-fine curriculum that enables to discover policies that fine-grained accomplish fine grained control of the resulting robots. We also introduce Ditto-
Gym, a comprehensive RL benchmark for reconfigurable soft robots that require fine-grained morphology changes accomplish the tasks. Finally, we evaluate our proposed coarse-to-Be algorithm on DittoGym and demonstrate robots that learn to change their morphology several times within sequence, uniquely enabled by our RL algorithm. More results are available https://itoogym.gttbbb.. github.
INTRODUCTION
Over millions of years, morphologies of species change function of evolutionary pres- sures (Minelli 2003 Rail 2012). robotics, this process of evolution has inspired the task robot co-design: the joint optimization of a robot's morphology and control policy that best enable the robot to accomplish given task (Gupta et al., 2022 Wang et al. Ha 2019 Yuan et al.] (002T)
Yet, in nature, creatures do not only change their morphology over millions of years function of evolution. Almost all living beings go through process of morphology changes even in their lifetime. These changes can be dramatic in magnitude, like when mighty tree grows from tiny sapling, but they can also be dramatic in form, like across the many examples of metamorphosis. where frogs, for instance, go through a water-dwelling stage with tail for propulsion, then lose their tail and grow legs to live on land (Rose 2005) Holmann et al. 2003).
Inspired by this process, we propose the study of reconfigurable soft robots. Reconfigurable soft robots, as the name suggests, are robots that are capable of dynamically changing their shape and morphology during their lifetime to accomplish varying tasks. Resembling organisms nature that change their morphology in response to evolving conditions (Ning et al.) 2023). reconfigurable soft robots offer an exciting avenue for addressing complex real-world challenges. While the actual manufacture of reconfigurable soft robots will remain challenging problem for some time, recent
Work done visiting researcher MTT. Code available at DittoGym and CFP

---

Published as a conference paper at ICLR 2024
advances have seen the first implementations, such as recent design manufactored from ferro- magnetic slime (Sun et al. 2022) that has exciting applications in healthcare.
In this paper, we identify three key challenges in the algorithmic study of reconfigurable robots and propose means to overcome them.
Firstly, while it is easy to describe what reconfigurable robot is, there is consensus about how to simulate these robots, and how one may parameterize their actions. In this paper, we propose to simulate soft reconfigurable robots via the Material Point Method (MPM) while modifying the Cauchy stress update to incorporate actuation specified via a continuous muscle field. This leads to simulation that models the recently proposed reconfigurable magnetic slime robot (Sun al.. 2022). As result, we are able to formalize reconfigurable robot control into a reinforcement learning problem with a 2d continuous muscle field its action space.
Secondly, the unstructured nature of soft robots renders traditional control algorithms unsuitable (Lee) al., 2017). While learning based methods like reinforcement learning excel in variety of unstructured control problems (Li) 2017), reconfigurable robots pose a unique challenge due to the extremely high-dimensional action space necessary for fine-grained morphology change. For instance, for RL to succeed, we need to ensure that random exploration will lead to meaningful morphology changes (Tang et al.] 2017). practice, this means that actions have to move large chunks of the robot time. On the flip side, to execute fine-gree grained actions such as walking, the action space needs to support fine-gree une-grained actuation. As we will see, trying to learn a fine-grained policy via existing reinforcement learning fails. We hence design multi-scale musele field, where policy with coarse discretization for actions is trained first to actuate large chunks of the robot to effect meaningful morphology changes while a fine-grained policy is learned on top. We benchmark our algorithm with strong baselines and find that our coarse-to-fine framework is the only algorithm that can control the reconfigurable robot accomplish mulli-stage tasks in our benchmark.
Finally, there is no standard benchmark for fine-gres aineggrinedd reconfigurable robots. We introduce Ditto- Gym, the first RL benchmark for reconfigurable robots. DittoGym is a suite of eight long-horizon tasks that require varying degrees of morphological change. Four environments require the robot to change its morphology more than once during the execution of the task.
In summary, our contributions are three-fold. Firstly, we formalize the joint control of morphology and policy of deformable soft robots into a unified reinforcement learning problem under a continuous 2D muscle field. Secondly, we propose Coarse-to- Fine Policy (CFP), a novel RI. algorithm with hierarchical action space that enables such robots to explore meaningful actions to control a deformable soft robot. Thirdly, we build comprchensive benchmark, DittoGym, that evaluates control algorithms for deformable soft robots in complex tasks requiring dynamical morphological changes.
+
Figure I: Reconfigurable soft robots. In this paper, we address challenges in controlling reconfig-
urable- or shape-shifting robots, who can change their morphology accomplish desired tasks. The figure above illustrates a task where a circular robot needs to alter its body shape to fit within a confined chamber to manipulate the square cargo to the target point (right). We introduce bench- mark with 8 tasks for shape-shifting robots, which we dub "DittoGym" inspired by a shape-shifting Pokémon (left).
RELATED WORK
Robot design simulators. The field of automated robot morphology design has garnered significant attention, resulting in variety of simulators suitable for testing and simulation. Broadly, these simulators can be classified into two categories: those designed for rigid robots and those for soft robots. For simulating rigid robots, representative simulators include Mujoco (aocoooTTo (Todorov al. 2012
2

---

Published as a conference paper at ICLR 2024
and RoboGrammar (Zhao et al., 2020). In these simulators, the basic components of the robot are predefined, and the robot's ' morphology is parameterized using a graph-like structure. While these simulators offer straightforward representation of morphology, they lack flexibility and generalizability in design and are limited to simulations of rigid robots. In contrast, for simulating soft-body robots, representative simulators include EvoGym (Bhatia et al.] (02T) and Softzoo (Wang) iet al. 2023a). These simulators focus on the co-design of soft-body robot morphology and control. However, prior work using these simulators exclusively focuses on co-design, with no prior work investigating reconfigurable soft robots. Consequently, these simulators primarily focus on fully elastic soft-body robots, not accounting for robots that can undergo plastic deformation to accomplish task. A notable gap exists in the lack of a simulator capable of accurately simulating highly reconfigurable soft robots. Therefore, we have developed a novel simulation platform, DittoGym, designed to accurately simulate reconfigurable robots for the study of their motion and control. The parameters of our simulation platform are adjustable, enabling it to simulate a range of robot types, from near-rigid robots to completely clastic soft-body robots.
Robot design algorithms. Many co-design algorithms adopt a common strategy of exploring the morphology design space. For instance, RoboGrammar and SoftZoo (Zhao et al. [2020; Wang ct al., a employ a heuristic search algorithm based on morphology grammar, while previous studies have utilized evolutionary algorithms to optimize robot morphology by maintaining numerous morphology candidates (Cheney et al., 2014] Gupta et al.] (021). However, these search- based zeroth-order optimization methods are computationally demanding and inefficient (Golovin et al.) 2019). An alternative research direction aims to seamlessly integrate robot morphology into robot states and learn end-to-end control policies (Yuan et al.. (02T). Recent efforts have also tried to learn latent space of morphologies using vatialee models such as Variational Autoencoders (Hu al. 2023 Ha et al. 2021 Wu et al., 2020) and Diffusion model (Wang et al. 2023b), though they either rely on predefined graph grammars or need pre- tr-iniiiinn on a large dataset. Despite the various branches that have emerged from co-design methods, they all result in static morphologies that cannot be modified during the robot lifetime, making them unsuitable for reconfigurable robot motion and control.
PRELIMINARY
Material Point Method. The Material Point Method (MPM) is a versatile computational particle- based simulation method widely employed in the fields of solid mechanics and computational physics (Jiang et al., 2016)] Hu et al. 2018 Han et al. 2019: Ram et al.] 20155: Stomakhin et al., 2013 Russell & Celia| 2002). As hybrid of Fulerian and Lagrangian methods, MPM involves both grid-based and barticle-based operations. The grid stage primarily serves to handle boundary conditions, while the particle stage is focused on addressing interactions among thousands of particles. In this latter stage, critical task is to compute the material strain, also known as Cauchy stress, as it profoundly influences the subsequent motion of particles. The Cauchy stress is determined by the following equation: Cauchy Stress (F p r + diag(A(J, 1)J,) Here, j and À represent material parameters, rp is an orthogonal matrix obtained from the Polar Decomposition in MPM, and Jp is the determinant of F--
Von Mises yield criterion. The von Mises yield criterion is employed to predict when a material will undergo plastic deformation (Madenci & Oterkus (016). The deformation gradient F, can be decomposed using Singular Value Decomposition (SVD) to obtain a diagonal matrix Ep- which represents the deformation scale for each particle. Plastic deformation initiates when the norm of this matrix, denoted as Eplla, exceeds the yield criterion Yield,. At this stage, the material "forgets" its initial state, necessitating deformation gradient projection to account for this deviation from the original configuration.
Reinforcement learning. We employ Reinforcement Learning (RL) to train the control policy for our highly reconfigurable robot. In the realm of RL, we model the problem as a Markov Decision Process (MDP), characterized by the quintuple (S, A, 7,r, Here, S signifies the state space, A denotes the action space, P S x A + S encapsulates the environment's 's transition dynamics, r(s,a) a): S x A - IR quantifies rewards, and 7 E (0, 1) governs the temporal discounting of rewards.
3

---

Published as a conference paper at ICLR 2024
Our RL agent's objective is to derive policy Te(ae)))). parameterized by deep neural network, aimed at maximizing the expected cumulative reward 7 a;)l.
4 METHOD
In this section, we aim address the three aforementioned challenges in the algorithmic research of reconfigurable soft robots, namely lack of formalization, lack of appropriate algorithm and the lack of a benchmark.
Act(O.0) 0)
Material Point.
Act(i.))
Act(l + 1,1)
Caudty Stress
Action
Yield Criterion
Node
nct(n, Act(I,1 1)
Act(I + 1)
(a) Initial Robot
(b) Action Grid
(c) Signal Distribution
(d) Different State
Figure 2: (a) The soft reconfigurable robot, initialized with specific shape, ready to venture
into the task. (b) Instead of directly controlling material points in the robot, the policy first applies actions to the action grid, which corresponds to the grid in the material point method (MPM). (c) Each particle aggregates actuation signals from its adjacent grid points during the grid-to-pan pii-tto-prrticle distribution stage. (d) Under the principles of Cauchy stress and the von Mises yield criterion, we can actuate the particles in physically plausible manner, thereby changing the robot's 's morphology and state.
4.1 FORMALIZATION
Controlling reconfigurable robots is an unstructured control problem that is naturally suited for trial-and-error methods such as reinforcement learning. We thus formalize the control problem into building blocks of an MDP as defined in Section
In order define the transition dynamics in a MDP, we need formalize reconfigurable soft robots into a simulatable system while staying close to real hardware.
To do so, we adopt the MPM as the simulation backbone for soft robots. As shown in Figure to simulate realistic deformations, we draw inspiration from prior work (Huang al. 202 Gao ett al., 2017). introducing the von Mises yield criterion into the MPM algorithm. This criterion models the robot elasto-plastic materials, exhibiting clastic deformation at low strains and plastic deformation at higher strains. This physical property is exhibited by many real-world materials, including the material most recently used build real-world reconfigurable robot (Sun al. 2022). We also introduce a modification to the Cauchy stress equation (Rothenburg & Selvadurai 1981. Hu et al. 2019b), allowing particles to stretch compress their shape response action signal, analogous to natural muscles. Specifically, this additional term can be expressed as multiplication of the gradient deformation matrix and action signals: F F. where ΣG is second- order diagonal matrix designed to receive 2D action signal vector. This operation directly alters the strain of the particle along its two principal axes. Consequently, the final Cauchy stress in an MPM simulation can be expressed shown in Equation [ This modification enables us to explicitly assign a 2-dimensional action signal vector Ap each particle's Cauchy diagonal matrix (Eş),, governed by deformation equations known in material science. As result, the robot exhibits macroscopic changes in morphology when an action forces significant portion of the particles to undergo plastic deformation, while stress below the von Mises yield stress leads to elastic deformations, serving spring-lit like mechanism to provide, for instance, means for locomotion.
Final Cauchy Stress (Fp(F, r) diag(A(Ip 1)1p) F,E
(1)
4

---

Published as a conference paper at ICLR 2024
Obs: [v.@v, (VOYAEEAPP
Resichal Policy
Coarse Policy
(1
M) acoarse +
aresidual
Faeie
interpolated coarse)
Action: interpolated(a)
Stage 1 Stage
Figure 3: Illustration of CFP. In stage one, we train coarse policy to efficiently explore the action space and discover meaningful action patterns. Subsequently, in stage two, employ coarse-to fine approach to train a high-resolution residual policy that delves deeper into optimizing the actions for improved performance.
The above MDP transition dynamics require to specify actuation vector for every single particle. However, we note that MPM a particle- based simulation method that approximates continuous dynamics in continuous space. Hence, action space should similarly be continuous field where each coordinate within the robot has an action vector. This also mirrors the control of the magnetic slime robot via external magnetic field (Sun et al. 2022). We illustrate this in Figure 2
With the action space defined, it is easy to define the remaining components of our MDP: The state simply the state of all particles in the MPM simulation, whereas reward functions in a case-by-case manner for different environments.
choose observation and
We have thus formalized the abstract problem of controlling reconfigurable robot into MDP problem that can be solved via reinforcement learning.
4.2 ALGORITIM
The model is designed with collyyconvolutional architecture, enabling it to analyze pixel-Ievel observations of the robot's shape and its velocity. Based on this analysis, it generates discrete grid action signals, which are instrumental in approximating the ideal force field, as shown in Figure Please refer to Appendix D for more detailed illustration of CFP.
Discretizing the action parameterization. Our MDP requires an infinite-dimensional action space in the form of continuous 2D force field acting the robot. To parameterize this continuous force field, we note that bicubic interpolation can interpolate a regular n-D grid of features into a mapping from any n-D coordinate outputs discrete n-D grid of actions and interpolate it onto our continuous action space.
feature. We thus parameterize the policy via neural network that
Fully convolutionl policy network. While the discretized parameterization provides differentiable way to specify a spatially continuous action, we require a discretization resolution that is high enough control fine-grained morphology changes. High resolutions lead nigh-dimensional action space, which has proven to be challenging for traditional reinforcement learning algorithms (Duan al. 2016 Metz al. 2017). However, we note that in our two-dimensional simulator, observations are 2D images that contain both the robot as well as its immediate environment (see Figure 3) Further, actions act on the same 2D neighborhoods that we observe in the image. We thus propose to utilize a fully convolutional autoencoder policy network to parameterize the policy, which can share parameters across spatial locations, effectively reducing the size of the search space. This network takes images of the reconfigurable robot and its immediate environment input, and generates a 2D action output that approximates the continuous muscle field via bicubic interpolation. Additionally, drawing inspiration from (Jiang al. 2015), adopt trid-to-particle approach to smoothly distribute the grid output to individual particles.
Coarse-to-fine policy learning. As highlighted in Section I random actions high resolution will tug and pull at single particles without leading to a significant macroscopic morphology change, leading to meaningless parameter updates the policy. To address this issue, observe that action
5

---

Published as a conference paper at ICLR 2024
output coarser resolution, when interpolated, actuates large muscle chunks to together due to the increased grid size. Such coarse discretization often deforms the soft robot substantially even under randomly sampled actions. Therefore, first train coarse policy, denoted as Ma within our Markov Decision Process (MDP) framework, leveraging low-resolution action discretization for expedited exploration of significant morphology changes.
Nevertheless, relying solely on the coarse policy proves insufficient for achieving success complex tasks, as it does not harness the complete control granularity available within the action space. To overcome this limitation, we introduce refinement stage: we freeze the parameters of the trained coarse policy Te while training new residual policy with increased resolution atop the predicted coarse actions. Specifically, we concatenate the observation with the coarse policy's output as input the fine policy neural network. The fine policy then produces a residual action prediction aresiuaal and gating mask M with values between (0, 1) via sigmoid activation. Utilizing this mask, we blend the coarse policy and residual policy via weighted sum:
aiiee
M aresidual + (1 -M) a.gurre
It is important to note that in this equation, the output from the residual policy is of higher resolution compared to the coarse action, necessitating upsampling of coarse actions before applying the mask.
The utilization of gated mask, while freezing the fixed Th..ne. renders Th less susceptible to local optima during exploration of the high-dimensional action space. Simultancously, the gated mask selectively overrides coarse policy predictions when finer actions are deemed more appropriate, allowing for much finer control granularity to maximize expected returns across various tasks.
This coarse-to-fine training methodology can be iteratively applied to incrementally scale up the resolution of the predicted action. our implementation, we set the output resolution the residual policy to be twice that of the coarse policy output.
Initial Morphology
Adaptive Morphology
SHA 2EE
NATTO
GROW
01ST
SLOT
Figure 4: Illustration of reconfigurable robots in diverse tasks. This figure showcases selected visualization results from DittoGym. Notably, policies trained under the guidance of CFP exhibit precise control over highly reconfigurable robots, enabling them successfully accomplish their respective tasks. More visualization results can be found in Appendix C
4.3 BENCHMARK
Digital benchmark environments are widely utilized by control algorithms researchers to expedite their research and facilitate fair comparisons Yuu et al. 2020 Bhatia al.) (2021). Our benchmark. DittoGym, addresses the absence of standardized benchmarks for reconfigurable soft robots with fine-grained morphology changes. We incorporate formalized simulation (Section 4.1 into set

---

Published as a conference paper at ICLR 2024
of OpenAI Gym environments. To model deformable soft robots and external objects for interaction, we implement the simulation of relevant materials in the Material Point Method (MPM) using the Taichi (Hu et al. 2019a) framework. Consequently, DittoGym not only provides user-friendly Python interface but also fully leverages hardware accelerators, such as GPUs, to simulate MPM particles. We choose the observation space to be a rasterized 2D image of the square around the geometric center of the robot, with both occupancy and velocity as channels. The action space implemented as 2D actuation grid for the same square with the highest possible resolution for simulation (MPM grid resolution). As mentioned in Section 3 each 2D vector in the 2D action grid controls the stress applied the corresponding point, where the magnitude determines whether plastic or elastic deformation occurs.
Under this framework, we implemented a variety of tasks covering shape matching, locomotion, reaching, and manipulation. Each task necessitates morphological adaptation to meet specific fine- grained requirements, sometimes requiring multiple morphological changes to achieve the desired goals. We provide brief overview of these eight tasks in Appendix A
5 EXPERIMENTS
In this section, we hope to answer the following questions via empirical evaluation. 1) Can our formalization, when combined with appropriate algorithms, enable simulated soft robots to perform interesting tasks that require fine-grained morphology changes? 2) Can our benchmark DittoGym sufficiently evaluate algorithms designed to control fine-grained morphology changes for soft robots? 3) Compared to relevant baselines, how sample efficient is CFP for reconfigurable soft robot control?
5.1 INTERESTING TASKS WE ENABLE
Sample Efficient
One of our main contributions the formaliza- tion of reconfigurable robot control into a rein- forcement learning problem. In this section, we demonstrate series of tasks a simulated recon- figurable robot can uniquely perform under this formalization when combined with appropriate algorithms. Figure [
Coarse Policy
Coarse- to fine!
illustrates an interesting multi-stage task made possible with our frame- work. In Figure 4 we show four additional tasks from DittoGym that require the robot to alter its morphology during execution.
Coarse Action
Fine Action
Fine-from- Stro--ccracc Policy
In the SHAPE MATCH task, the robot is able to successfully mimic the target alphabet shape, highlighting the robot's ability to reconfigure to arbitrary target shapes. In the GROW task, the robot learns to elongate and curve its body. This novel shape enables the robot weave its way around obstacles to reach the target point, similar to a seed germinating. In the OBSTA- CLE task, the robot elongates its body and leans forward in order to leverage the force of grav- ity to swiftly stride over the obstacle. Subse- quently, the robot grows two leg-like stumps and uses them to walk, enabling efficient for- ward progress. Lastly, for the SLOT task, the robot strategically reduces its height while growing two very short legs maneuver through an exceptionally narrow space, resembling a mouse running through a pipe. To open the wooden lid at the end of the pipe, the robot ungrows its legs and grows a tall torso to manipulate the lid effectively.
Figure 5: Qualitative analysis of different meth- ods. A policy learns to shape the robot into star. Fine-from-scraid fiee-froo-srrathh scratch policy suffers from exploration random actions at high resolution will tug and pull at single particles, preventing meaningful mor- phology changes. A coarse policy have higher sample efficiency as it can make meaningful explo- rations, but cannot take advantage of fine-grained action space. CFP takes advantage of both.
7

---

Published as a conference paper at ICLR 2024
5.2 WHETHER DITTOGYM NECESSITATES FINE- GRAINED ACTION
Before we can quantitatively evaluate CFP against baselines in DittoGym, it's necessary to prove that DittoGym actually requires requiresfine-rraanee fine-grained morphology changes for the soft robot to achieve high rewards.
Quantitatively, we take the expert policies trained under each action space granularity and visualize the maximum reward they can achieve. We study this metric for three different levels of granularity, coarse (4 4 x 2), medium action resolution almost always leads to higher possible reward across all 8 environments. This trend confirms that DittoGym requires the agent to learn fine- mne-griieed morphology changes.
2) and fine (16 16 2). As illustrated in Figure higher
Qualitatively, we can visualize the expert policies' behaviors under different levels of granularity. In Figure 5 we visualize the best shape the rohot can achieve in the SHAPE MATCH task under different action resolutions. Under the coarser resolution, the robot can make meaningful changes towards the target shape but fails to generate high- high-frequency detail of the target shape, such as the tips of each arm of the star. On the other hand, the high-resolution action parameterization allows the robot to match the star shape well.
5.3 SAMPLE EFFICIENCY OF CFP
Finally, we hope to understand whether CFP has higher performance compared to baselines when controlling reconfigurable robots with fine-grained morphology changes. We therefore benchmark CFP and aforementioned baselines in 8 different environments in DittoGym under 3 seeds.
Baselines. We benchmark our coarse-to-fine toneeo,,-eeeoene fully convolutional action parameterization with prominent alternatives:
GNN Policy: Mased control emerges as a potential solution for our challenge, as enables the treatment of robot distinct modules and offers the flexibility to adapt to various robot topologies. However, these methods face limitations in direct application to our context, as highly-reconfigurabl robots often lack articulated joints and links, and therefore do not possess the explicit topological structure required by these baseline methods. spite of these challenges, we integrate the well-established Modular Policy baseline (Pathak al. 2019) into our problem framework. To effectively modularize the robot, we utilize k-means clustering to assign topology, thereby allowing the application of the Graph Neural Network (GNN) from the original method. At each step, the robot is initially segmented using the unsupervised K K-manns method. Following this, we employ the state information of each mass center point, treated as joints, along with the adjacency relationships of the segments, to construct the graph. This approach enables to harness a standard RL framework for training the control policy.
Transformer Policy: The GNN-based modular policy requires explicit assignment of adjacency relationships among all segments, potentially limiting information propagation during training. To address this, introduce additional baseline wherein an attention mechanism autonomously determines the topology for information processing. A more comprehensive explanation and visualization of modular-based approaches available in Appendix
Coarse Policy (ablation): This baseline only trains a coarse policy (8 x 8 x 2) but does not implement the coarse-to-fine technique.
Fine-from-Scratee Policy (ablation): This baseline directly trains a fine policy (16 16 x 2) without implementing the proposed coarse-to-fe to-fine curriculum. The intention of this baseline is to investigate whether, under the current framework, an agent can learn efficient high-dimennsonal policy without the coarse-to-fine process.
In Figure 6 we plot the episode reward curves for all the runs. Our proposed algorithm, CFP, outperforms all basclines consistently across all tasks in terms of overall sample efficiency. In addition, CFP consistently attain higher episode reward upon convergence. When we visualize the policies, only CFP is able to succeed in Long-horizon multi-stage environments, presenting it as the only viable option to solve these complex tasks.
Although in CATCH task, the best Transformer-based baseline could achieve same-level performance CFP, it is highly oscillated and unstable. In most tasks, the baselines does perform well. For
8

---

Published as a conference paper at ICLR 2024
each GNN and Transformer-basee method, we implement pparse-segment version (with 16 seg- ments) and a dense-segment segment version (with 64 segments). It's not unexpected that the modular-based control policy, originally developed for addressing control issues in robots with clearly defined topo- logical structures, falls short when applied to this innovative reconfigurable rohot. More comparison results can be found in Appendix
MATCH
RUN
GROW
KICK
22002
300
9002
Eploade Roward
255
10002
150
3002
5000
52-
2002
100
203
300
400
202
605
200
4.00
605
201
403
OBSTACLE
CATCH
30002
70000
Episode Reward
4000
12500
8003
60000
2000
12000
0003
50000
7500
4003
2030
5000
2002
1030
2500
200
400
800
202
Troin Stops (X 10"))
Train Steps (x 10*)
Train Stops (x 10")
Train Stops (x 10")
coarse
coarse-to to-fie-to-fine (Ours)
fineffrmm-cceer scratch
GNN-16
Transformer-16
Figure 6: Comparison of fine-resolution policies trained with and without coarse-to-fine tech- nique. The plots showcase the underperformance of a fine-resolution policy trained from scratch compared with a coarse-to-fine result. It further reflects that parametrizing the action space via modular policies (GNN-based and Transformer-basedd still fall short of the coarse-to- to-fee technique. The green dot represent the checkpoint where we start to implement coarse-to-fine technique.
Why coarse-to- to-fiie so good? In Figure qualitative visualization of different methods shows that fine-from-scratte fails to match the shape of the star despite having sufficient control resolution to do so. This can be explained by the exploration problem brought by high-dimensional action spaces we detailed in Section 4.2 On the other hand, while the coarse policy isn't isn't fine-gr aen-igained enough to get a perfect star shape, it has a lower- dimee--iieniionall action space which actuates larger chunks of muscles together, leading meaningful morphology explorations. However, the performance of coarse policy is ultimately limited by its granularity and would produce a star shape that's far worse than the one produced by our coarse-to-fine method.
Both the quantitative comparison and qualitative analysis demonstrates the effectiveness of CFP in DittoGym, illustrating its unique ability to control reconfigurable soft robots to accomplish long- horizon tasks that requires fine-grained shape changes.
6 CONCLUSION
Many creatures in nature change their morphology throughout their lifetime. Finding ways in which we may simulate and train reconfigurable robots that mimic this impactful across robotic appli- cations, with first practical examples already proposed [Snn et al. (2022). This paper establishes formal reinforcement learning framework for investigating and controlling reconfigurable soft robots. We first introduce an innovative parameterization of reconfigurable robots soft material actuated via a muscle field. We then present a framework, CFP, which leverages a fully-convoluttionll state estimation and action parameterization. enhanced by toarse-to-tine curriculum that guarantees meaningful exploration while preserving fine-grained control capabilities. We introduce the first benchmark, DittoGym, with eight environments that require drastic morphology changes to accom- plish the embedded tasks. We demonstrate that our robots, trained by CFP, learn to dynamically alter their morphology several times throughout their lifetime accomplish these tasks. We significantly outperform baselines that do not employ a coarse-to-fine parameterization of the action space.

---

Published as a conference paper at ICLR 2024
REFERENCES
Jagdeep Bhatia, Holly Jackson, Yunsheng Tian, Jie Xu, and Wojciech Matusik. Evolution gym: large-scale benchmark for evolving soft robots. Advances in Neural Information Processing Systems, 34:2201-220 2021.
Nick Cheney, Robert MacCurdy, Jeff Clune, and Hod Lipson. Unshackling evolution: evolving soft robots with multiple materials and a powerful generative encoding. ACM SIGEVOlution, 7(1): 11-23, 2014.
Yan Duan. John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Picter Abbeel. RI:: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv: 161 1.02779, 2016.
Ming Gao, Andre Pradhana Tampubolon, Chenfanfu Jiang, and Eftychios Sifakis. An adaptive generalized interpolation material point method for simulating elastoplastic materials. ACM Transactions on Graphics (TOG), 36(6) 2017. 2017.
Daniel Golovin, John Karro, Greg Kochanski, Chansoo Lee, Xingyou Song, and Qiuyi Zhang, Gradi- entless descent: High-dimensional zeroth-order optimization. arXiv preprint arXiv: 1eXiv:99111 1911.6631, 2019.
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications, 12(1):5721, 2021.
Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers. arXiv preprint arXiv:2203. 1931, 2022.
David Ha. Reinforcement learning for improving agent design. Artificial life, 25(4):352-365, 2019.
Huy Ha, Shubham Agrawal, and Shuran Song. Fit2form: 3d generative model for robot gripper form design. In Conference Robot Learning, pp. 176-187. PMLR, 2021.
Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning. pp. 1861-1877 PMLR, 2018.
Xuchen Han, Theodore F Gast, Qi Guo, Stephanie Wang, Chenfanfu Jiang, and Joseph Teran. hybrid material point method for frictional contact with diverse materials. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 2(2):1224, 2019.
DIETRICH K Hofmann, WK Fitt, and Fleck. Checkpoints in the life-cycle of cassiopea spp.: control of metagenesis and metamorphosis in tropical jellyfish. International Journal of Developmental Biology, 40(1):0())))))) (1)):331-338, 2003.
Jiaheng Hu, Julian Whitman, and Howie Choset. Glso: Frammar-guidee latent space optimization for sample-efficient robot design automation. In Conference on Robot Learning, pp. 1321-1331. PMLR, 2023.
Yuanming Hu, Yu Fang, Ziheng Ge, Ziyin Qu, Yixin Zhu, Andre Pradhana, and Chenfanfu Jiang. A moving least squares material point method with displacement discontinuity and two-way rigid body coupling. ACM Transactions Graphics (TOG), 37(4):1-14, 2018.
Yuanming Hu, Tzu-Mao Li, Luke Anderson, Jonathan Ragan-Kelley, and Frédo Durand. Taichi: language for high-performance computation on spatially sparse data structures. ACM Transactions on Graphics (TOG), 38(6):1-16, 2019a.
Yuanming Hu, Jiancheng Liu, Andrew Spielberg, Joshua Tenenbaum, William T Freeman, Jiajun Wu, Daniela Rus, and Wojciech Matusik. Chainqueen: A real-time differentiable physical simulator for soft robotics. 2019 International conference on robotics and outomation (ICRA), pp. 6265- 6271. IEEF, 2019b.
Zhiao Huang, Yuanming Hu, Tao Du. Siyuan Zhou, Hao Su, Joshua B Tenenbaum, and Chuang Gan. Plasticinelab: A soft-body manipulation benchmark with differentiable physics. arXiv preprint arXiv:2104 033 2021.
10

---

Published as a conference paper at ICLR 2024
Chenfanfu Jiang, Craig Schroeder, Andrew Selle, Joseph Teran, and Alexey Stomakhin. The affine particle-in-cell method. ACM Transactions on Graphies (TOG), 34(4):1-10, 2015. 2015.
Chenfanfu Jiang, Craig Schroeder, Joseph Teran, Alexey Stomakhin, and Andrew Selle. The material point method for simulating continuum materials. Acm siggruph 2016 courses, pp. 1-52. 2016.
Chiwon Lee, Myungjoon Kim, Yoon Jae Kim, Nhayoung Hong, Seungwan Ryu, H Jin Kim, and Sungwan Kim. Soft robot review. International Journal of Control, Automation and Systems, 15: 3-15, 2017.
Yuxi Li. Deep reinforcement learning: An overview. arXiv preprint arXiv: 2017.
Erdogan Madenci and Selda Oterkus. Ordinary state-based peridynamics for plastic deformation
according to mises yield criteria with isotropic hardening. Journal of the Mechanics and Physics of Solids, 86: 16:192-219, 2016.
Luke Metz, Julian Ibarz, Navdeep Jaitly, and James Davidson. Discrete sequential prediction of continuous actions for deep rl. arXiv preprint #rXiv:1705.05000 2017.
Alessandro Minelli. The development of animal form: ontogeny, morphology, and evolution. Cam- bridge University Press, 2003.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602, 2013.
Linxiaohai Ning, Chayabhan Limpabandhu, and Zion Tsz Ho Tse. Engineering magnetic soft and reconfigurable robots. Soft Robotics, 2023.
Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexci A Efros. Learning control self-assembling morphologies: study of generalization via modularity. Advances in Neural Information Processing Systems, 32, 2019.
Rudolf Raff. The shape of life: genes, development, and the evolution of animal University of Chicago Press, 2012.
Antonin Raffin, Ashley Hill, Maximilian Ernestus, Adam Gleave Anssi Kanervisto, and Noah Dormann. Stable baselines3, 2019.
Daniel Ram, Theodore Gast, Chenfanfu Jiang, Craig Schroeder, Alexey Stomakhin, Joseph Teran, and Pirouz Kavehpour. A material point method for viscoelastic fluids, foams and sponges. In Proceedings of the 14th ACM SIGGRAPH//Euograppen Symposium on Computer Animation, pp. 157-163, 2015.
Christopher S Rose. Integrating ecology and developmental biology explain the timing of frog metamorphosis. Trends in Ecology & Evolution, 20(3):129-1355 2005.
L Rothenburg and APS Selvadurai. A micromechanical definition of the cauchy stress tensor for particulate media. Mechanics of structured media, pp. 469-486, 1981.
Thomas F Russell and Michael A Celia. An overview of research eulerian lagranggan localized adjoint methods (ellam). Advances in Water resources, 25(8-12) 2002.
Alexey Stomakhin, Craig Schroeder, Lawrence Chai, Joseph Teran, and Andrew Selle. A material point method for snow simulation. ACM Transactions on Graphics (TOG), 32(4):1-10. 2013.
Mengmeng Sun, Chenyao Tian, Liyang Mao, Xianghe Meng, Xingjian Shen, Bo Hao, Xin Wang, Hui Xie, and Li Zhang. Reconfigurable magnetic slime robot: deformation, adaptability, and multifunction. Advanced Functional Materials, 32(26):21 12508, 2022.
Haoran Tang, Rein Houthooft, Davis Foote, Adam Stooke, OpenAl Xi Chen, Yan Duan, John Schulman, Filip DeTurck, and Pieter Abbeel. # exploration: A study of count-based exploration for deep reinforcement learning. Advances in neural information processing systems, 30, 2017.
11

---

Published as a conference paper at ICLR 2024
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ international conference on intelligent robots and systems, pp. 5026-5033 IEEE, 2012.
Tsun-Hsuan Wang, Pingchuan Ma, Andrew Everett Spielberg, Zhou Xian, Hao Zhang, Joshua B Tenenbaum, Daniela Rus, and Chuang Gan. Softzoo: A soft robot co-design benchmark for locomotion in diverse environments. arXiv preprint arXiv:2303.0955 2023a.
Tsun-Hsuan Wang, Juntian Zheng, Pingchuan Ma, Yilun Du, Byungchul Kim, Andrew Spielberg. Joshua Tenenbaum, Chuang Gan, and Daniela Rus. Diffusebot: Breeding soft robots with physics- augmented generative diffusion models. arXiv preprint arXiv:2311. 17055,, 2023b.
Yuxing Wang, Shuang Wu, Haobo Fu, Qiang Fu, Tiantian Zhang, Yongzhe Chang, and Xueqian Wang. Gurriculum-based co-design of morphology and control of voxel-based soft robots. In The Eleventh International Conference on Learning Representations.
Yizhe Wu, Sudhanshu Kasewa, Oliver Groth, Sasha Salter, Li Sun, O Parker Jones, and H Posner. Learning affordances in object-centric generative models. International Conference on Machine Learning, 2020.
Tianhe Yu, Deirdre Quillen, Zhanpeng He, Ryan Julian, Karol Hausman, Chelsea Finn, and Sergey Levine. Meta-world: benchmark and evaluation for multi-tak and meta reinforcement learning. In Conference robot learning, pp. 1094 184-11000 1100. PMLR, 2020.
Ye Yuan, Yuda Song, Zhengyi Luo, Wen Sun, and Kris Kitani. Transform2act: Learning transform- and-control policy for efficient agent design. arXiv preprint arXiv:2110.03659 2021.
Allan Zhao, Jie Xu, Mina Konaković Lukuvvvv Josephine Hughes, Andrew Spielberg, Daniela Rus, and Wojciech Matusik, Robogrammar: graph grammar for terrain-optimized robot design, ACM Transactions Graphics (TOG), 39(6):1-16 2020.
12

---

Published as a conference paper at ICLR 2024
APPENDIX
A BENCHMARK DETAILS
A.1 TASKS
The DittoGym includes eight distinct types of tasks encompassing shape matching, locomotion, reaching. and manipulation. The environment settings and task objectives for each of these tasks are detailed as follows:
The robot is initialized as a circle a zero-gravity environment. has to alter its shape to mirror predefined geometric or alphabetical form. The score is calculated on the basis of the congruence between the robot's current shape and the target shape.
SHAPE MATCH:
The robot is initialized on a plain and the task requires robot to move forward to the greatest possible distance within a stipulated period. The score is governed by the distance traversed and the speed maintained.
Locomotion RUN:
The robot is initialized as circle and is designed to kick square target to the maximum possible distance. Ground friction prevents simple pushing, therefore the robot is required flipping and rolling technique.
Locomotion KICK:
Locomotion DIG: The robot is initially in the shape of a circle placed on the top of soil-filled container, aiming to reach a target located the bottom-right corner.
Locomotion OBSTACLE: The robot, in the shape of square, faces obstacle in its path while the task is to move forward. The score is based solely on how far the obstacle is bypassed.
square on the ground and is required to extend its superior segment reach target represented by purple dot. Multiple obstacles hinder the direct route to the target. The reward function measures the distance between the target point and the robot.
Reaching GROW: The robot is initialized
The circular robot, placed outside cube, is tasked with manipulating small square target within the cube to a specifie point. The score computed considering the distance between the robot and the cube, as well as the cube and the final point.
Manipulation CATCH:
Manipulation SLOT:: The circular robot, initially outside a box with only a narrow slot to get inside, must squeeze its body through the slot and manipulate cap target on top of the box. The reward function measures the distance between the robot and the cap, and whether the cap was successfully removed from the box.
Our task selection follows both generic RI benchmark design principles and the specific needs of reconfigurable robots. We highlight the factors we hoped to cover in our benchmark and the corresponding factor vector of each environment as shown in Table
Tasks have to cover a range of difficulties (A=easy, B-hard).
Some tasks evaluate robot itself, others require the robot's ability to interact with external ob- jects (A=not require, B=require).
Some tasks require just one morphology change, while others need changes of morphology multiple times (A=one change, B=multiple changes).
Some have non-convex reward landscape that requires the algorithm to be long-horizon (A=short- horizon, B=long-horizon)
Some features softer material and some features less soft material (A=softer, B=less softer).
A.2 MPM PARAMETERS
Since this algorithm involves grid operations and requires a fixed grid defined in space the computational domain for MPM, it is essential to define a relatively small grid space to enhance the computational efficiency of the MPM algorithm. However, in some locomotion tasks, highly reconfigurable robots may have movement distances that exceed the predefined maximum length of the grid area. To address this issue, have adjusted the MPM algorithm by modifying the grid to
13

---

Published as a conference paper at ICLR 2024
Table Benchmark Factors for Reconfigurable Robots.
Task
Factor Vector
SHAPE MATCH
[,, A, A, A, A
RUN
A, A, A. A. B
GROW
[A, A, A, A, A
KICK
[B, B, A, A, B]
DIG
[B, B, A, A, BI
OBSTACLE
[B, A, B, B
CATCH
[B. B. B. B, A]
SLOT
[B. B. B. B, B
move along with the rohot's center of mass during motion. This ensures that the robot always remains within the scope of the MPM algorithm, effectively resolving the challenge of efficiently simulating large-scale motions of the robot. The hyperparameters of MPM are shown in Table? 2 Parameters in reference to the particle numbers, material characteristic and the maximum action depend on the tasks, which can be checked from our codes.
Table 2: MPM Hyperparameters.
Name
Value
P_vol
1
p. mass
2
n_ grid
128
le-4
Repeat Times
100
Coeff
0.5
Bound
1
Seed
0
B TRAINING DETAILS
B. MODEL STRUCTURE
In this project, we have adopted the SAC (Haarnoja et al. 2018) algorithm as the core framework for reinforcement learning. For the policy network architecture, we have implemented fully- convolutional structure similar to the Nature CNN (Mnih et al.) 2013). In addition to this, prior to each input, we have introduced a concatenation step where image is combined with a position embedding channel. This augmentation significantly enhances the network's sensitivity to spatial positions. Regarding the critic network structure, we have employed an identical CNN encoder architecture used in the policy network. Subsequently, dimensional MLP network to compute and output the critic value. The code for this project will be publicly accessible upon acceptance. The network details are shown follows:
have integrated two layers of 256-
Critic Network. The 3-layer CNN based encoder can embed the input of 64 64 (3 channels for state, 2 channels for the upsampled action) images into a vector with 512 dims following a CNN architecture widely used in RL papers and frameworks (Raffin et al.) 2019 Huang et al. 2021). Then it will get through a 3-layer MLP with 256 dims latent space to yield the Q value.
Policy Network. The similar 3-layer CNN based encoder can embed the input of 64 X 64 images into embedding shaped 4 x 4 x 32 (still 512 dims). Then will get through a 3-layer Deconvolutional network to yield grid-like like action (8 x 8 2 or 16 16 2). In the residual training stage, the policy network's 's input channel will also be (3 channels for state, 2 channels for the upsampled coarse action).
14

---

Published as a conference paper at ICLR 2024
B.2 REINFORCEMENT LEARNING
The specific parameters of the reinforcement learning algorithm we employed are presented in Table If simulation alone, the environment can run at average speed of 70.1 FPS on the RTX4090 GPU, with only 20-30% volatile utility usage and 15% memory usage. Training a single agent runs at approximately 20 FPS when executed the same GPU. Parameters like alpha and max episode length depend on the tasks, which can be checked from our codes.
Table 3: SAC Training Hyperparameters.
Name
Value
Batch Size
256
Auto Entropy-tuning
False
Gamma
0.99
Critic Network Hidden Size
256
Learning Rate
3e-4
Momentum
0.99
Optimizer
Adam
Replay Buffer Size
200000
Seed
0,1 and
C EXPERIMENT VISUALIZATION
In Figure 7 we illustrate the eight tasks within the DittoGym: SHAPE MATCH, RUN, KICK, DIG, GROW, OBSTACLE, CATCH, SLOT.
In the SHAPE MATCH task, every robot is able to successfully mimic the target alphabet shape, highlighting the robot's ability to reconfigure arbitrary target shapes.
In the RUN task, notable improvement is observed when the robot transforms into feet-like structure. This adaptation aligns with the biological adjustments observed in many terrestrial organisms, underscoring the robot's 's ability to replicate natural processes effectively.
In the KICK task, the robot required exert force on object to roll it on the ground, leading to reshape into configuration resembling an 'earth-mover', demonstrating an example of functional adaptation.
In the DIG task, the robot assumes a drill-like structure to meticulously navigate toward the target, showeasing a distinctive problem-solving approach and flexible adaptability.
In the GROW task, the robot learns to elongate and curve its body. This novel shape enables the robot to weave its way around obstacles to reach the target point, similar to a seed germinating.
In the OBSTACLE task, the robot elongates its body and leans forward in order to leverage the force of gravity to swiftly stride over the obstacle. Subsequently, the robot grows two leg-like stumps and uses them to walk, enabling efficient forward progress.
In the CATCH task, the robot initially alters its configuration to a slimmer form to access a confined space, followed by a transformation into "L" shape to manipulate square object to the target location. This demonstrates remarkable versatility when confronted with complex environmental conditions.
Lastly, for the SLOT task, the robot strategically reduces its size to maneuver through an exceptionally narrow space, resembling mouse fitting into small hole. To push the target cap atop box, it then showcases its reconfigurability by regrowing its stature, providing the necessary height to manipulate the cap effectively.
15

---

Published as a conference paper at ICLR 2024
Initial Marphology
MlaptiveMMrphologg
SEATP MATOR M
RUN
KICK
DIG
GROW
OBST.
CATCH
SLOT
Figure 7: DittoGym. The visualization of robots successfully completing all eight tasks defined within the DittoGym while being controlled by the policy generated from CFP.
D METHOD ILLUSTRATION
The policy model's structure is a fully-convolutional framework. The input of the model is multi- channel state image, which contains the information of the robot's shape and velocity (from x and direction). The output of the model in the action field. The illustration is shown in Figure g (in Figure 3 we use animation to illustrate the real input and output data). In our project, we employ grid with resolution of x 64 to store the action information. At every time step, the process involves two key stages:
same-size, same-place action image, representing the strength
Upsampling Coarse Action. The coarse action, represented at resolution of 8 x 2, is first upscaled to match the 64 x 64 grid. This upsampling is vital step in preparing the action for distribution.
16

---

Published as a conference paper at ICLR 2024
Distributing Signals to Particles. Once upscaled, these action signals then distributed to the individual particles in MPM's grid operation stage. This distribution is critical component of how the robot receives and reacts action commands.
The same methodology applied when dealing with fine policy. The coarse action is firstly upsampled to 16 x 16 x 2, then calculated the fine action using equation in Section 4.2 and finally to output the signals we upsample the result to x 64
Vg= relative velocity x-axis
64 dims
Vy: relative velocity in y-axis
SHAPE: relative partieles' density
Input robot's mass centric state images Obs: @.HAPE
agi action signals in x axis
Observation Yellow shadone)
Conv
ay: action signals in y-axis
Coarse action: 8x8x2
Deconv
Residual action: 16x16x2
action: 16x16x2
(All actions will be unsampled to 64 x 64 x2 before fed into the action field)
dims
upsample(a.)) to bicubically upsample any resolution of to the same resolution as state images (64x64)
Output robot's mass centric control images
Action
(Green sbadow)
Action: (ipaappl(a./@uppspplc.) (upsampl((aa)
Decorv
... [..] [
Con
Coarse- fine
Ingut observation state and unsampled coarse action images Obs @ upsample(Conrse Action)
Output residual action weighted mask [upsample(Res (.)@@psammlleRerr aR MMSSK]
Figure 8: Further illustration of CFP.
MATCH
RUN
GROW
KICK
10000
20020
0032
Eplcodo RDWr22
280
0320
4002
K59
3002
10020
5020
50
2000
1.03
Coarzo
003r50
meddiu CATCH
coarse
medium
DIG
OBSTACLE
SLOT
3000
37500
70020
Episode Reward
4032
15000
10005
00030
3002
12500
8032
10030
10000
40070
2000
7500
20070
1000
5050
2500
2002
20020
10000
0oarse
m.diium
fine
coarse
medium
fine
coarse
modiuu
Tine
coarse
medium
fine
coarse
medium
fine
Figure 9: Performance of expert policies under different action resolutions. Expert policies using the highest resolution (fine) achieve much higher episode rewards compared to those with medium or coarse resolutions, illustrating the tasks in DittoGym require fine-grained morphology changes.
17

---

Published as a conference paper at ICLR 2024
E MODULAR-BASED BASELINES
The modular robot control methods are not directly applicable for our problem because highly- reconfigurable robots lack articulated joints, links and thus lack of explicit topological structure these baselines need. Despite that, we adopt the Modular Policy baseline (Pathak et al. 2019) by assigning topology with k-means clustering so GNN can be applied. We provide an additional baseline by letting an attention mechanism to figure out topology itself. Figure (egurreOOillustaaall the cluster while Figure TI shows CFP's performance is still much higher than the dense-segmented modular baselines in most tasks (except for CATCH task).
GNN-based (16 segments)
GNN-based (64 segments)
Figure 10: Illustration of GNN-based modular control policy. The robot's particles are segmented into several pieces (16 64) and then being treated joint inputs the modular policy. For GNN-based method, we need to artificially establish the graph among the segments according to their adjacent relationships, as shown on the left.
MATCH
RUN
GROW
KICK
20000
6003
Splsade Rewarrs
250
8029
5003
10030
150-
4010
3003
2003
383
200
300
400
203
AB2
203
450
500
203
403
OBSTACLE
CATCH
SLOT
70030
10002
Epicoda Reward
125:0
8003
00030
2032
10020
6003
50030
7520
4003
2002
2003
-203
200
Train Steps (x 10²)
Train Steps (x 10")
Train Steps (x 10";
Troin Stops (x 10")
coarse
coarse-to-fine (Ours)
fine-from-scratch
GNN-64
Transformer-64
Figure 11: Comparison of fine-resoution policies trained with and without coarse-to-fine technique. In Figure 6 we compare the performance CFP with sparse-segmented (16 segments) modular policies. In this section, we further compare its performance with dense-segmented (64 segments) modular policies.
F OTHER EVALUATION METRICS
We compare the running speed and the usage of memory of the proposed method with modular based baselines in the RUN task. The results in Table indicate that CFP is a faster and more lightweight method.
18

---

Published as a conference paper at ICLR 2024
Table 4: Running Speed and RAM Consumption Comparison.
Method
Average Time Per Episode
Used RAM (MB)
CFP
54
4053
Transformer- based (dense-segment) GNN-based (dense-segment
104
6895
83
4199
We report the success rate of the SLOT task whose success be clearly defined. The results in Table indicate that CFP outperforms the additional baselines.
Table 5: SLOT Task Success Rate Comparison.
Method
SLOT (%)
CFP
15.4
Transformer- based (dense-segment GNN-based (dense-segment)
5.9
We also conduct experiments on CFP to test if it is robust to the observation or actuation failure cases. already added Gaussian noise to action output when getting result in the main paper. emulate noisy control. The SAC algorithm naturally optimizes expected return under high entropy (noisy) actions, which explains why result is already good in the paper.
Our two added experiments randomly zeros out 20% of the action output or observation input respectively when evaluating our expert policy. This emulates actuator or sensor failures respectively, Figure [2] shows their performance as a ratio to original performance. Our performance is negligibly impacted (<10%) by action failure except out of the 8 tasks, DIG, which shows performance drop of only 30%. The policy also seems to show some robustness to sensor failure from the plot, a less common setting.
3.0
Perfomance Ratoo
0.0
044
0.4
0.2
0.2
None
Oiservation Action
None Observation
Action
None Observation
Action
None Observation
Action
DIG
OBSTACLE
CATCH
SLOT
Performance Ratio
0.8
3.8
0.8
0.5
0.0
02-
Aciinn
None
Actian
Action
Actian
None Fallure
Observation Fallure
Action Fallure
Figure 12: Robustness justification for CFP.
19


Reviews:
[{"review_id": "Dr2RpqjZcm", "reviewer": "Reviewer_5Sv3", "summary": "This paper proposes a new task of **reconfigurable** soft robot design. The goal of this task is to **continuously** optimize the morphology of soft robots **during** the episodes of accomplishing a long-horizon task. \n\nThis paper first develops a simulation platform that implements the action space and transition dynamics of soft robots and environments. It also implements diverse tasks for benchmarking. Specifically, they define long-horizon tasks that require multiple times of morphology changes.\n\nTo address the reconfigurable robot design task, this paper proposes a novel RL algorithm that enables morphology changes from coarse to fine. This algorithm leads to efficient morphology optimization.\n\nThis paper conducts experiments on the proposed simulator with 4 types of tasks.", "questions": "1. Is the action space or environment 2D or 3D? Why use 2D images as raw states for policy instead of using ground truth robot states(i.e., exact morphology parameterizations)?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The task of reconfigurable robot design is important. This paper formulates this task as an MDP and provides a simulator that implements the action space & transition dynamics of the MDP.\n2. This paper proposes a novel residual RL algorithm that shows impressive results in the 4 types of tasks.", "weaknesses": "1. There is no adequate comparison with classical robot design baselines, e.g., Bayesian optimization, genetic algorithms, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "z9THMqVU5B", "reviewer": "Reviewer_NgHH", "summary": "This paper proposes an approach to learning morphology for soft robot control. The paper formulates reconfigurable soft robot control as a high-dimensional reinforcement learning problem in a continuous 2D muscle field and designs a coarse-to-fine hierarchical policy (CFP) to expedite the exploration of the action space. The paper also implements a benchmark that allows simulating the plastic deformation of robots for various tasks.", "questions": "Please see the weaknesses section.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "+ Existing methods are well-reviewed, and the classification of existing challenges in reconfigurable soft robots is a strength.\n\n+ The proposed benchmark is non-trivial, and the demo video supports the proposed method.\n\n+ If applicable to real robots, the problem of controlling reconfigurable soft robots is important.", "weaknesses": "- The explanation of the full approach is unclear. For example, multiple variables and terms are not defined or explained in the figures, including vx, vy, and SHAPE in Figure 3, the dimension of actions, architectures of the encoder, Coarse, and Residual policy. In the appendix, the paper briefly explains that the core framework is based on existing works SAC and Nature CNN with minor modifications, but it is unclear how to reproduce the full approach.\n\n- Given the concern above, the theoretical novelty seems not high, as the main theoretical novelty of CFP is the introduction of adding residual action to coarse action.\n\n- The paper lacks comparisons with existing state-of-the-art methods. The experiments only evaluate the performance of two ablation baseline methods and one NFP method. More comparisons will be appreciated. Besides the evaluation metric on reward, if more metrics can be used to evaluate real-robot applications, it will be appreciated, such as the successful rate or time efficiency.\n\n- As one of the main challenges proposed to address in this paper, the justification of lifetime adaptation is insufficient in the experiments. For example, if the approach can adapt to noise or actuator failures during the lifetime operation?\n\n- Even though the simulation experiments are impressive, there is still a gap between applying the proposed work to real-world robots, as shown in the demo video. An explanation of how to extend the work from simulations to real-world robots will strengthen the paper significantly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2f2Mphae7b", "reviewer": "Reviewer_mQFN", "summary": "This paper presents a reinforcement learning approach able to control a reconfigurable and simulated soft robot that has to change its shape to perform different tasks. In addition to the novel RL algorithm, which first modifies the coarse structure of the robot and then more fine-grained details, the authors also introduce a new benchmark environment for reconfigurable soft robots. The presented control approach is compared to various baselines, which do not employ a coarse-to-fine control but instead use fine-grained control directly.", "questions": "- Is the code available? \n- How computationally expensive is the simulation environment?\n- How large is the action space?\n- how is the upsampling of coarse actions done?\n- What are the network details (e.g. number of layers etc.?)", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Exciting research with a path to being employed on real robots such as ones based on ferromagnetic slime\n- Interesting multi-scale muscle field control, where course and fine grain actions can affect the robots morphology\n- Introduced a new interesting benchmark “Morphological Maze” that tests algorithms for their ability to perform morphological change", "weaknesses": "- A potential weakness of the approach, especially when it should ultimately be transferred to real robots, is that the control of the robot's shape do not happen based on local interactions, i.e. the employed controller has to take into account the complete shape of the robot and its environment\n- Missing relevant literature on evolving soft robots and simulators (e.g.VoxCad). \"Cheney, Nick, et al. \"Unshackling evolution: evolving soft robots with multiple materials and a powerful generative encoding.\" ACM SIGEVOlution 7.1 (2014): 11-23.\" In fact, in this work, the authors use a CPPN-encoding which is the foundation for neural field models and should be mentioned.\n- \"However, these search-based zeroth-order optimization methods are computationally demanding and inefficient” - is this shown in the paper? It's a common argument but it would be good to have some references here backing it up.\n- Importantly, what are the lessons for the larger machine learning community? e.g. for which other tasks could the course-to-fine (CFP) algorithm be useful? What about other hierarchical RL methods? The research is exciting from an alife/robot perspective but it could be better motivated for an ICLR audience \n- Since this conference is about representation learning, it would be interesting to investigate further what type of representations the RL algorithms learn to control these soft robots. How do the coarse and fine-grained control interact to solve the tasks at hand?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jiJLCenTH9", "reviewer": "Reviewer_UEFW", "summary": "This paper studies the problem of reconfigurable robots, which can change morphology to accomplish a task. The paper makes the following contributions: (a) a new simulator to model both elastic and plastic deformation; (b) a benchmark based on the aforementioned simulator, and (c) a coarse to fine reinforcement learning approach for controlling such robots.", "questions": "It would be great if the authors could clarify why the proposed simulator is a good model of real robots, why the proposed tasks were selected, and clarify the experimental setup. In addition, it would be great to include more relevant benchmarks for this problem.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The proposed simulator and benchmark fill a hole in the current literature on morphology-changing robots\n2. The proposed hierarchical control approach outperforms other ablations and another baseline in a set of experiments.", "weaknesses": "Overall, I found this paper quite interesting to read. The main limitations, I think, are:\n1. It is unclear whether the proposed simulator and benchmark is a good model of reality. Is it modeling a specific robot or class of robots? Could there be some experiments to show the validity of the simulator? I'm not necessarily referring to simulation to reality transfer, but rather to show that the simulator dynamics are correlated with the dynamics of a real robot.\n2. The proposed hierarchical reinforcement learning approach is relatively standard. Its application to the problem of morphology-changing robots might be new, though. \n3. I don't understand the reasoning behind the proposed baselines. There seem to be other approaches that are more related to the problem (e.g. Phatak et al., Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity; Whitman et al., Learning Modular Robot Control Policies). Having a more detailed comparison could give a better idea of how difficult is the proposed benchmark for current algorithms.\n\nMinor points:\n1. Why are these tasks selected? What exactly is it that they evaluate? Why is it important to have them instead of others?\n2. I am not sure I understand how the baseline of Neural Field Policy is implemented and whether this is a novel contribution of the paper or some other work proposed before for this problem.\n3. I don't think Fig. 6 is helpful, the concept it represents is trivial and easily explained in a sentence", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)