Leveraging Hyperbolic Embeddings for Coarse-to-Fine Robot Design
Authors: Heng Dong
Junyu Zhang
Chongjie Zhang
Venue: ICLR 2024 poster (2024)
Keywords: Robot Design
Hyperbolic Space
Coarse-to-Fine
Multi-Cellular
Primary area: reinforcement learning
TL;DR: We propose to design multi-cellular robots in a coarse-to-fine manner and leverage hyperbolic embeddings for realization.

Abstract: Multi-cellular robot design aims to create robots comprised of numerous cells that can be efficiently controlled to perform diverse tasks. Previous research has demonstrated the ability to generate robots for various tasks, but these approaches often optimize robots directly in the vast design space, resulting in robots with complicated morphologies that are hard to control. In response, this paper presents a novel coarse-to-fine method for designing multi-cellular robots. Initially, this strategy seeks optimal coarse-grained robots and progressively refines them. To mitigate the challenge of determining the precise refinement juncture during the coarse-to-fine transition, we introduce the Hyperbolic Embeddings for Robot Design (HERD) framework. HERD unifies robots of various granularity within a shared hyperbolic space and leverages a refined Cross-Entropy Method for optimization. This framework enables our method to autonomously identify areas of exploration in hyperbolic space and concentrate on regions demonstrating promise.  Finally, the extensive empirical studies on various challenging tasks sourced from EvoGym show our approach's superior efficiency and generalization capability.

Published as a conference paper at ICLR 2024
LEVERAGING HYPERBOLIC COARSE-TO-FINE ROBOT DESIGN
EMBEDDINGS
FOR
Heng Dong Junyu Zhang Chongjie Zhang
1 IIs, Tsinghua University 2 Huazhong University of Science and Technology
3 Washington University in St. Louis
(drdhxi, jyzhang1 208) gmail. com, chongjiee@wutl.edd
ABSTRACT
Multi-cellular robot design aims create robots comprised of numerous cells that can be efficiently controlled to perform diverse tasks. Previous research has demonstrated the ability to generate robots for various tasks, but these approaches often optimize robots directly in the vast design space, resulting in robots with complicated morphologies that are hard to control. In response, this paper presents novel coarse-to- fine method for designing multi-cellular robots. Initially, this strategy seeks optimal coarse- gannsegrraiedd robots and progressively refines them. To mitigate the challenge of determining the precise refinement juncture during the coarse-to-fine transition, we introduce the Hyperbolic Embeddings for Robot De- sign (HERD) framework. HERD unifies robots of various granularity within a shared hyperbolic space and leverages a refined Cross-Entropy Method for opti- mization. This framework enables our method autonomously identify areas of exploration in hyperbolic space and concentrate on regions demonstrating promise. Finally, the extensive empirical studies on various challenging tasks sourced from EvoGym show our approach's 's superior efficiency and generalization capability.
INTRODUCTION
For decades, humans have envisioned the creation of artificial creatures with morphological intelli- gence (Lipson & Pollack 2000 Howard et al. 2019 Gupta et al.) 2021b). To achieve this goal, highly promising solution is to learn optimal robot morphologies for variety of tasks in simulated environments (Sims) 1994a Wang et al. 2019, Yuan et al. 2021 Dong et al., 2023). Inspired by the wide existence of multi-cellular organisms, such as animals, land plants, and most fungi, one natural way of representing robots is to formulate them as multi-cellular systems. The creation of multi-cellular robots is challenging because of the twin difficulties: 1) the design space robots, including cell types, positions, and parameters, etc., is immensely large, and 2) the evaluation of each design requires learning its optimal control policy, which is often computationally expensive.
For melti--elllllar robot design. previous methods typically adopt Genetic Algorithms (GA) (Sims) 1994b; Medvet et al. 2021 Cheney et al., 2014b)), which sample robots from a large population to accomplish the given tasks and then only keep the top-performing robots and their offspring. However, these methods are inclined to learn from scratch in the vast design space and could be highly sample- inefficient (Wang al. 2022). Intuitive examples are provided Figure 1 where use time-lapse images to show the control process of different designs performing task
(Bhatia al.) 202T). This task requires 2D robots to cross the terrain that is full of obstacles and getting increasingly bumpy. GA (Figure I (a-b)) fails to learn any effective robot structures to solve this task because of the vast design space. To overcome this issue, Wang et al. (0022) uses a curriculum-bant design (CuCo) method by gradually expanding the robot design from a small size to the target size through a predefined curriculum. However, as shown in Figure I (c,d) and empirical evidence in Section 5 robots of small size can hardly solve the original task and thus, unfortunately, cannot provide helpful information for the next stage of the curriculum.
In view of these challenges, propose to design multi-cellular robots in coarse-to-fine manner by first searching for coarse -rriined robots with satisfactory performance and subsequently refining
Equal contributions.

---

Published as a conference paper at ICLR 2024
Iteration
Iteration B
Veriial Actuator
Honiznnaa Actuator
Rigid Cwl
Target Direction
Soft Cell
Obstacle
Stuck
Empoy Call
(a)
CuCo
Stuck
HERD (Ours) (c)
(d)
(e)
S
Figure 1: Visualization of the control process of method HERD compared with baseline GA and (Bhatia et al. 2021). Iteration one of the iterations later than Iteration A, such that corresponding algorithms have updated the robot. (a-b) GA directly searches in the vast design space and fails to learn effective structures to cross the obstacles; (c-d) CuCo adopts predefined curriculum from smaller robots larger robots, but the smaller robot typically faces more challenges when solving the original tasks, e.g., the same obstacles could be more difficult for Thus, smaller robots cannot offer useful guidance for the remaining stage in the curriculum. (e-f) Our method designs robots in a coarse-to-fine manner and can focus on promising regions with the helpful guidance of coarse-grained design. Finally, our method successfully finds simple but effective design to solve this challenging task.
CuCo (Wang et al. 2022) during training in task
them. The key insight our approach is that the coarse-grained robot design stage can provide computationally inexpensive guidance for subsequent fine-grained design stages and help focus on refining promising robot designs. There are two reasons for this insight. First, coarse-grained robots are less complicated to design and control due to their low degree of freedom. Second, -earse-grained robots can usually solve part of the tasks (Figure [ (e)). and serve as the starting points for the subsequent fine-grained design stage (Figure 1 (f)) In this way. coarse-to-fine robot design can potentially learn simple but effective robots for the given tasks. In literature, the idea of coarse-to-fine is common in computer vision (Zhu et al. 2015) Pavlakos et al., 2017) and molecule generation QQang et al. 2023), but has not been well studied in the field of robot design to our best knowledge.
The realization of coarse-to-fine robot design involves two major challenges. The first challenge is how to define the granularity of robots. To consider a wide range of granularities for robots of any shape, propose to coarsen robots by aggregating adjacent cells based their initial positions. The higher the degree of aggregation, the coarser the robots are. The coarse- seee---raiddd robots under this definition are less complicated and informative for refinement because of their similarity to fine-grained robots. The second challenge is how to determine the coarse-to- interval, i.e., when refine current coarse-grained robots. Specifically, the design space of any-grained robots can be formulated as a tree structure, in which the parent nodes correspond to coarse-grained robots, and the child nodes represent fine-gra fine-grained robots similar to their parent robots. Optimizing in this discrete tree is troublesome, as we have to determine whether to search child nodes sibling nodes. To tackle this challenge, we introduce a novel Hyperbolic Embeddings for Robot Design (HERD) framework. where we first embed the above tree into hyperbolic space, which is naturally suitable for hierarchical representations, and then use modified Cross-Fntropp Method to optimize the robot design this hyperbolic space. As result, HERD unifies and optimizes any- any-grained robots in shared space, and it demonstrates significant improvements over other methods.
We evaluate our HERD framework on 15 tasks in the benchmark EvoGym (Bhatia al. 2021). HERD sigmificantly outperforms previous state-of-the- art algorithms in terms of both sample efficiency and final performance on most tasks. Training performance comparison and visualization of optimizing in hyperbolic space strongly support the effectiveness of our method. Our experimental results highlight the importance of coarse-to-fine robot design and the representation learning of robots.
2

---

Published as a conference paper at ICLR 2024
2
RELATED WORKS
Multi-Cellular Soft Robot Design. Automatic robot design focuses on generating robots that possess easy controllability for a wide range of tasks. It requires concurrently optimizing the robot morphology and control policy. A line of works in this field aims for evolving morphologies
composed of rigid elements, including skeletal structures and attributes of limbs and joints (Sims)
1994a] Lipson & Pollack 2000 [Lehman & Stanley 201T] Yuan et al. 202T). However, this setting will result in limited design space due to the compositions of predefined elements and restricted capacity of robots to interact with environments (Cheney et al. 20) To design robots with stronger generalization and better flexibility (Pfeifer et al. 2007), multi-cellular soft robots that are constructed by combining small cells might be a promising alternative. Previous works in multi-cellular robot design directly explore the vast design space through evolutionary search or reinforcement learning (Cheney et al.) (2088: Jelisavcic et al. [019 Walker & Hauser (202]). Yet, evolutionary search is sample inefficient, and remforcement learning can be easily trapped in local optimal due to exploration issues (Spielberg et al. 2019). To alleviate the burden of the thorough search, recent work CuCo incorporated curriculum learning design processes by gradually expanding robot size (Wang et al., 2022). However, CuCo required prior knowledge to predefine the curriculum, and smaller robots might not be helpful in complex tasks. In comparison, our work enables automatic determination of coarse-to-fine transition during training, and coarse- coarse-grained grained robots are informative for refinement.
Coarse-to-Fine. Coarse- toase-ttofiine strategy refers to progressively refining system from high-level representations by incorporating detailed information. It has been applied for some intelligent systems (Pedersoli et al. 2015) mainly including computer vision, robot manipulation, and molecule generation. Previous works on computer vision primarily mimic natural coarse-to-fine process to capture features through hierarchical localization paradigm (Sarlin et al.) 2019 Yu et al. 202T), or propose a unified model architecture that enables efficient parameter reuse between different training stages (Qian et al. 2020: Dou et al. 2022). Other works in robot manipulation use this mechanism to address exploration difficulty in sparsely-rewareee rewarddd and nonghooizzon tasks (Johns) 2021 James et al., 2022). In the field of drug discovery, allows for 3D non-autorose annouorrrreresivee molecule generations with a hierarchical diffusion-based model (Qiang et al.) (023). The coarse-to-fine strategy has shown its strengths in other fields. However, to our best knowledge, this strategy has not been well studied in robot design problems and our method is the first work to apply it in multi-cellular robot design.
Reinforcement Learning for Incompatible State-Action Space. Robot design typically requires learning generalizable control policies that could be shared among robots of diverse morphologies where the state and action spaces are incompatible. To address this challenge, prior works mainly utilize modular reinforcement learning to control each actuator separately. In the case of multiple different morphologies, one approach is to leverage GNNs to learn joint controllers (Khalil et al. 2017 Huang et al. 2020). While GNN-based methods employ graph representation relevant to the robot's s morphology to ensure message passing in shared policy (Battaglia et al.) 2018 Yuan et al.) 202I). Transformer-bart based approaches with attention mechanisms can further improve generalization, dealing with the limitation of aggregating multi-hop information (Kurin et al. 2020). In this paper, we build our control policy based on Transformer VVaswani et al. 2017) ) to d dynamically track the
relationships between components of the robots (Gupta al. 202 Dong et al. (022).
3 PRELIMINARIES
In this section, we present essential background knowledge and notations for our method.
3.1 HYPERBOLIC SPACE AND POINCARÉ BALL
Riemannian manifold. A Riemannian manifold is defined as a tuple (M, g) (Petersen 2006), where M and g are the manifold and the metric tensor, respectively, as defined below. A manifold M is set of points z that are locally similar to the linear space. Every point 2 of the manifold M is attached to a tangent space T.M. which is a real vector space of the same dimensionality as M and contains all the possible directions that can tangentially pass through z. Each point z of the manifold also associates metric tensor g that defines an inner product of the tangent space: g (,))2 : T&M x T&M- + R.
Then the inner product can induce norm T-M= ||2 = / )z
3

---

Published as a conference paper at ICLR 2024
Hyperbolic space. A d-dimensional hyperbolic space H6 is d-dimensional Riemannian manifold with constant negative curvature cc IH can be represented using various isomorphic models, such the hyperboloid model, the Beltrami- Seltrami-Klein model, the Poincaré half-plane model, and the Poincaré ball (Beltrami 1868). In this paper, we use Poincaré ball for hyperbolic embeddings.
Poincaré ball. The Poincaré ball can be formally defined
Riemannian manifold By (Bd Ge),
where 13dh {z E Rd c| | is an open ball, and g. is the metric tensor defined as g.=(2²) ( (
where 2 |- and g = is the Euclidean metric tensor, i.e., the usual dot product.
Using this metric tensor, we can induce the distance in Poincaré ball:
Jc cosh -1 (1 1
|| 22113
Dz(21, 22)
2c (1 c||=| (1 cl 22
To connect hyperbolic space and Euclidean space, we can use an exponential map and a logarithm map map from Euclidean space to the hyperbolic space and vice versa with anchor On the Poincaré ball, both maps have the closed-fore expressions:
expí(v) = te taanh VVC x
X
(2)
2
Veluull
2 Vexg
-2 +e
(o)
tanh -1 (vell the wll) ||
(3)
2 +. wll
Here the operator C. the Möbius addition (Ungar 2022 defined as
(1 + 20(z, v) + c cllee 1 ) = + c
B v
1 2c(2, v) + c²
Note that we can recover the Euclidean addition
O.
3.2 SARKAR'S CONSTRUCTION FOR HYPERBOLIC EMBEDDING
Embedding a tree into Poincaré ball B has many solutions. We will introduce training- free com- binatorial construction method: Sarkar's Construction (Sarkar 201I), whose resulting embeddings can exhibit arbitrarily low distortion (Sala et al. 2018). For simplicity, we discuss the setting where dimensionality d = and curvature e = 1. The basic idea to embed the tree's root at the origin and recursively embed the children of each node by spacing them evenly around sphere centered the parent node as in Algorithm 2 Specifically, consider node q of degree deg(g) with parent node p and suppose 9 and p have already been embedded into 2g and zp in B² (omitting curvature 1). We now place the children a, .2, Adeg(g)-1 of q into I3..
Several steps are needed. First, 29 and un reflected across geodesic of the Poincaré ball B² by using circle inversion so that zy is mapped to the origin and zp is mapped to some point z/p. The reflection map is:
|| = u
Fz,po(Zp))
(zp u)
(5)
where u = =z/g/=/|1² Next, place the children nodes to embeddings /1 32. "* - that are equally spaced around circle with radius ei-1 where is scaling factor, and maximally separated from the reflected parent node embedding z/p- One embedding method is:
e" yn
cos (0+
2mm deg(g)
sin 0 2nn
))
(6)
e +
deg(g)
where 0 = arg(2/)) is the angle of 2/. from x-axis in the plane. Lastly, reflect all the points back using the same reflection map: Fo)) To embed the entire tree, first position the root node at the origin and its children a circle around it, then recursively position children nodes as discussed above until all nodes are placed. Figure (b) provides intuitive example for this construction.
4

---

Published as a conference paper at ICLR 2024
Higher
Vertica Artustor
Rigid Cell
Soit Cell
Empty
Coarse- Grained
Sampled Robos
Hyperbolic Embedding
Coerne- Eo Fline
optimization proceeds
Dj
talial State
Fine- Grained
C(P,)
C(Ds)
(a)
(b)
(c)
Figure 2: Hyperbolic embeddings for coarse-to-hie robot design framework. (a) An example of coarse- coarse-grained and fine-grained robots of EvoGym (Bhatia et al. 2021), where the designs C(D;) are D,'s child robots, i.e., refined robots and the robots A and boxes are all fine- ane-grained but have different cell segmentation styles; (b) Hyperbolic embeddings in the Poincaré ball for any-grained robot designs; (c) Our method initialized at the center the Poincaré ball, and automatically learn to move promising regions that may close to the edge. An important property is that sampling robots from the center to the border is exactly the process coarse-to- robot design.
METHOD
In this section, we present our novel learning framework that leverages Hyperbolic Embeddings for coarse-to-fine Robot Design (HERD). As in most previous works robot design, our learning framework mainly consists of robot design optimization and control policy learning. The novelty of our framework design robots in a coarse-to- embeddings. To this end, our method is characterized by two components: (1) embedding any any-grained robot designs into hyperbolic space (Section 4.1) and (2) optimizing robot designs in this hyperbolic space (Section 4.2). The algorithm is outlined in Algorithm
manner whose realization relies on hyperbolic
4.1 EMBEDDING ANY-GRAINED ROBOT DESIGNS INTO HYPERBOLIC SPACE
In this subsection, we first discuss the definition of granularity of robots in multi-cellular robot designs, then reveal the nature of the hierarchical structure of design space when learning in coarse-to-fine manner, and finally, discuss why and how to embed any- -ry-grained robot designs in hyperbolic space. In this paper, we will focus on multi-cellular robot design and use EvoGym (Bhatia al. 202 as our benchmark, where each robot design D E D consists of N= 25) inter-connected robot cells. Each robot cell has fixed initial position (a,, y, 2) and will be assigned cell type chosen from (Empty, Rigid, Soft, Horizontal Actuator, Vertical Actuator) during the design process.
Granularity refers to the extent to which groups small components robots. e.g., robot cells, have joined together to become large components. Coarse-grained robot designs have fewer, larger discrete components than fine-grained robot designs. The benefit of coarse-graining that it can remove certain redundant degrees of freedom, such as the designing types and controlling actions between several neighboring robot cells.
To determine which robot cells can be aggregated together,
simple but effective clustering algorithm, K-Means (Hartigan & Wong 1979). purely based on the position features of robot cells, to cluster neighboring robot cells into groups. The larger the number of clusters, the finer the granularity of the robots. In practice, we cluster the robot cells into & components iteratively, where k (16,8.... ... Importantly, the clustering results in the last iteration will be used as the input to
5

---

Published as a conference paper at ICLR 2024
K-Means in the next iteration. Figure (a) shows several results of this clustering, where robots in A and B boxes are fine- ane-ggained robots but with different clustering results due to the randomness of K-Means. It is worth noting that our method can also use other clustering methods.
Hierarchical structure in coarse-to-fine robot design. Given coarse- gaasseegrineddd robots (D; in Figure 2 (a)), we refine them by dividing large components of these robots into smaller ones (C(D;) Figure 2 (a)) according to the clustering results for fine-grained robots mentioned above. All child robots originating from the same parent robot use the same clustering results. By recursive refinement, the space of robot designs with varying granularity can be organized into hierarchy in which the parent nodes are coarse grrsinedg robots, and the child nodes are fine- one-grained robots that are similar to their parent robots. In practice, similarity here means that fine- grained robots only need to change one component to be the same as their parent robots. In the following. we use Do to represent the root node of the hierarchy and use C(D) to denote robot design D's child robots. Denote the robot hierarchy with ((D)) where N is parameter controlling the size of hierarchy.
Why utilize hyperbolic space. Directly searching for the optimal robot in the above robot hierar- chy is cumbersome because we have to determine the coarse-to-fine interval, i.e., when to refine current robots. Short intervals may lead to wrong branches of the robot hierarchy due to inaccurate performance evaluation, and long intervals may result in inefficiency. This is verified in Section 5.3
To solve this problem, we propose embed the robot hierarchy into shared hyperbolic space. There are two main advantages to this solution. First, embedding any-grained robot designs in a shared space allows us search for coarse- -grained robots and refine promising ones in unified way. This can lead to simpler and more efficient optimization, as shown in Section Second, hyperbolic space quite suitable for representing hierarchies, and actually, it can be viewed continuous analog of trees (Cetin et al. 2022 Hsu et al. 2021). In particular, the typical geometric property of hyperbolic space that its volume increases exponentially in proportion to its radius, whereas the Euclidean space grows polynomically (Yang et al. 2022). Because of this nature, it is possible to embed robot hierarchy into Poincaré ball BE one of the representations for hyperbolic space. with arbitrarily low distortion (Sarkar 2011 Sala et al.) 2018). Remarkably, Euclidean space cannot embed hierarchies with arbitrarily low distortion for any number of dimensions. Section 3.1
provides formal discussion.
Algorithm 1: HERD: Hyperbolic Embeddings for Coarse-to-Fine Robot Design
Input: robot design space D, hierarchy size N, Poincaré ball Bé. population size of CEM N, (D;,C(D)) +- build the robot hierarchy for designs (D.) N D using K-Means;
= (D, z, + embed the robot hierarchy {D,, C(D))) in Poincaré ball B& by applying Sarkar's 's Construction in Algorithm recursively;
initialize control policy æ, CEM mean fa and variance o:;
while not reuching max iterations do replay buffer H < 0; for i E {1, 2, ..... N₂) do
0i ~ N (H, diag (6) (o)): // sample an embedding from Euclidean space
expo (v)) // map to Poincaré ball, Equation (2)
D,, 2, + arg min (D,2) 2e(z,)); find the nearest valid embedding and its corresponding design, Equation use ग to control current robot design D, and store trajectories H:: update with PPO using samples in 7;;
10
11
update μ by averaging the elite v,s based on the performance in H, and linearly decrease o; 12
13 D x² + arg min (D.) (exp) (H), [4),2) 2): // optimal robot design
14 Output: optimal robot design D", control policy π
How to embed in hyperbolic space. Embedding the robot hierarchy into hyperbolic space can be realized with two main strategies. The first is to use a loss function and gradient descent to learn the embedding, such as Poincaré Variational Auto-Encoder (VAE) (Mathieu et al. 2019). However, Poincaré VAE cannot embed too many randomly generated robots as we will do during the training. Thereby, we resort to the second, training- free strategy, which is termed Sarkar's 's Construction (Sarkar 2011). The idea of Sarkar's Construction is simple but effective. First, it places the root node Do of the given hierarchy at the origin of Poincaré ball and places all its children nodes C( equally

---

Published as a conference paper at ICLR 2024
HERD (Ours)
Vanilla RL
80
Handcrafted Robot Best Performance
CuCo ObstarieTrwwersarr
CPPN-NEAT Piotfomjumaarvo
Goolumparvo
Epliadi Rewsrdd
Episocle Rewerd
10
15
Imill
(mil)
(mil) Hurdler-00
Traverser-v
Carriervv1
Epoode Reward
Epicade Reward
Eplande Reward
10
15
100
3.5
20
10
15
20
Imill
T (mil)
T (mil)
Figure 3: Training performance of HERD compared against baselines in hard tasks.
spaced in a circle around it. It then moves one of the children to the origin using hyperbolic reflection defined in Equation (5) and repeats. the following, use S { J 1.N to represent the robots and their resulting embeddings.
We show the embedding results of simplified robot hierarchy in Figure 2 where each node is the embedding for its corresponding robot design. For more details. please refer to Section 3.2) A very important property of this representation that as the norm of the embedding increases, the robots become more fine-grained. This property allows us to realize coarse-to-fine in an elegant way.
4.2 OPTIMIZING ROBOT DESIGNS IN HYPERBOLIC SPACE
To optimize robot design in hyperbolic space B. our method HERD adapts the idea of the Cross- Entropy Method (CEM) (Rubinstein & Kroese 2004). It works by repeating the following four phases: (1) sampling embeddings from normal distribution, (2) finding the corresponding robots of the embeddings, (3) evaluating these robots, and (4) updating the probability distribution based on the top-performing samples. Figure 2
(c) shows an example of this optimization in Poincaré ball.
Recalling that sampling robots from the center to the border of Poincaré ball is exactly the process of refining robots, we only need to initialize the mean μ E IRd of CEM to zero to enable coarse-to-fine robot design. To control the exploration rate, we use time-decayed variance E Rdd, which decreases from 0.2 to 0.01 linearly during training.
At each iteration, HERD samples N₂ embeddings (u,mu NN,, diag(o)) Vi 1, .N). To find the corresponding robots, HERD maps each v; from Euclidean space to the Poincaré hall by using the exp map defined in Equation (2), i.e., Σ, exp)( As not all points in Poincaré ball are valid under Sarkar's Construction. HERD chooses the nearest z; in the robot hierarchy with the distance function D. defined Equation 4 and uses z, robot for 2;
Next, the generated robots are evaluated in the given task with a shared control policy learned in CuCo (Wang et al. 2022). which adopts self-attention mechanism (Vaswani al. 2017) to handle incompatible state-action space and capture the internal dependencies between robot cells. The shared control policy learned using PPO (Schulman al. 2017), an actor-critic reinforcement learning algorithm.
Finally, we update the mean μ by averaging the elite v:s based on their fitness defined by the discounted sum of rewards E, 7'r, where 7 is the discount factor and ra is the reward at time step t.
5 EXPERIMENTS
In this section, we benchmark our method HERD various tasks from EvoGym (Bhatia al. 2021). We evaluate the effectiveness HERD by answering the following questions: (I) Can robot design help improve performance in comparison with handerafted design? (Section 5.2) (2) Can
Can
HERD outperform other robot design baselines various tasks? (Section (3) How does each
component of HERD contributes its performance? (Section 5.3) (4) How does HERD design

---

Published as a conference paper at ICLR 2024
HERD (Qurs)
NGE
HERD w/o HE
HERD wio C2F
Plalfomyjupeero0
Gaujumper--0
Epiiode Reward
Episodde Rewerrd
Egiondl
[mill]
(mil)
(mil)
Traverser
Carrien11
Hurdlervo
Egisode Rewward
Epicode Reward
Episode Resward
10 T [ml]]
10 T (mil)
10
15
T (mil)
Figure 4: Ablation studies of HERD in hard tasks.
robots in a coarse-to to-fine manner? (Section (544). For qualitative results, please refer to the videos on our anonymous project website" And our code is available on GitHub 1P
5.1 EXPERIMENT SETUP
We conduct experiments on 15 tasks from EvoGym and only show the results of hard tasks in this section. Please refer to Appendix C for the full results. All runs are conducted with four random seeds, and the mean value as well as 95% confidence intervals are shown.
Environments. We follow the original setting of EvoGym, where each robot is made up of 5 x 5 inter-connected 2D voxel cells. Each cell has unique type chosen from five options: empty, soft, rigid, horizontal actuator, and vertical actuator. The design space is huge with roughly 2.98 x 10 designs. The designed robot is controlled by assigning actions to all the actuator cells. The action value is within the range [0, 6, 1.6] and corresponds to a gradual expansion/contractio of that actuator times its rest length. The designed robots are required to perform multiple tasks, varying in objectives (locomotion, manipulation), terrains (flat, bumpy), and difficulties (easy, medium, hard). Please refer to Appendix [.T for detailed description of these tasks.
Implementations and Baselines. Our method HERD is implemented on the top of CuCo (Wang jet al.) 2022) by replacing its design policy with our novel optimization method in hyperbolic space. We use the same control policy and hyperparameters the strengths HERD, we consider the following baselines: (1) CuCo: a curriculum-based robot design method by expanding the design space from small size (3 3) to the target size (5 x 5) gradually through a predefined curriculum; (2) GA (Michalewicz & Michalewicz 1996: Sims 1994a Wang al., 2019): widely used black-box optimization method by relying biologically inspired operators such as mutation, crossover and selection. It directly encodes the robot as a
CuCo for fair comparison. To show
vector, where each element is the cell type. (3) CPPN-NEAT (Cheney et al.) 2014a b Corucci
et al. 2018): the predominant method for evolving soft robot design. It indirectly encodes robots by Compositional Pattern Producing Network (CPPN) (Stanley 2007 Cheney al.) 2014a) and trained by NeuroEvolution of Augmenting Topologies (NEAT) (Stanley & Mikkkulainen (2002) (4)
BO (Kushner 1964. Močkus 1975): a commonly used global optimization method of black-box functions by training surrogate model based on Gaussian Process (GP) to alleviate the burden of computationally expensive fitness evaluation. (5) Vanilla RI.: strong baseline that uses reinforcement learning to update the design policy and control policy. (6) HandCrafted Robot: human-designed robot using expert knowledge that can efficiently solve many tasks. Please refer to Appendix B.2 and
Appendix B.3 for more details of HERD and other baselines.
5.2 TRAINING PERFORMANCE COMPARISON
We summarize the training performance in Figure where the x-axis and y-axis are timesteps and episode reward, respectively. We also show one representative robot designed by HERD at the end of training in the upper left corner of each sub-figure offer more intuition, and the time-lapsed images of the control stage for these robots are provided in Appendix B.I Here, we only show the hard
https://gi http:/hh..ccmmm/////aaaaaaa
8

---

Published as a conference paper at ICLR 2024
Iteration
Iteration
lleration 10
Noration 30
Iteration 80
Hyporbale
Embediingg
x
x
x
Selected
Designe
Figure 5: Visualization of the learning process of HERD in PlatformJumperx0..
tasks, and please refer to Figure 22 for full results. HERD significantly outperforms the baselines and handcrafted robots in most tasks in terms of both design efficiency and effectiveness. This validates that (I) robot design can improve final performance compared with handerafted robots and (2) leveraging hyperbolic embedding for coarse-to-fine learning can effectively help design robots in both efficiency and final performance.
5.3 ABLATIONS
To investigate the contributions of each component of HERD, we conduct elaborate ablation studies here. There are two main contributions that characterize our method: (1) the idea of coarse-to-fine robot design and (2) realization of this idea in hyperbolic space. To validate the effectiveness of our method, we designed the following ablation studies. (1) HERD w/o HE: Using other method, e.g., evolutionary algorithm NGE (Wang al. 2019), to implement the idea coarse-to-fine robot design. This ablation will show the effectiveness of hyperbolic embeddings. (2) HERD w/o C2F: Do not adopt coarse-to- coarse-to-fine robot design, and use CEM to directly search the optimal design. This ablation can verify the effectiveness of coarse-to- to-fine robot design. (3) NGE: Directly use NGE for multi-cellular-rooot design. More details are available in Appendix B.4
We show the results hard tasks in Figure 4 and please refer to Figure for other tasks. HERD is better than these variants in most tasks. Specifiially, HERD outperforms HERD w/o C2F and NGE in all tasks, which shows the effectiveness of the idea of coarse-to-fine. Further, HERD is better than HERD w/o HE in most tasks, which indicates the usefulness of hyperbolic embeddings in coarse-to-Bie fine robot design. As for our HERD being inferior to HERD w/o HE in Carrier vO and Carriervvl, we speculate that these two tasks may only need find- find-grate grained robots, whose embeddings are close to the border of Poincaré ball. There may be problem with bits of precision in hyperbolic embeddings (Sala et al. 2018), which is further discussed in Section G
5.4 ROBOT DESIGN PROCESS ANALYSIS
To explore how HERD can learn these robot designs, in this subsection, we visualize the evolution process of the designed robots by HERD in Platform/umper-vO in Figure We show the hyperbolic embeddings and the selected robots from five iterations the training. Figure shows that as the sampled embeddings approach the border, the designed robots change from coarse-graing to fine-grained. An interesting phenomenon is that the final robot is not the finest-grained. This is partly because coarscr-grained designs are already sufficient to solve some tasks in EvoGym.
DISCUSSION
In this paper, we leverage hyperbolic embeddings for coarse-to-fine robot design. The idea of coarse- to-fine avoids directly searching in the vast design space, and hyperbolic embeddings further bring us better optimization properties. Finally, the empirical evaluations and visualizations of HERD show its superior effectiveness. However, our approach still has two limitations. First, the idea of coarse-to-fin might not be helpful in the tasks that only the finest-grained robots can solve. In these tasks, coarse-grained robot designs will not provide useful guidance for refinement if the coarse-gae coarse-grained robots cannot address even part of the tasks. Second, embedding robots in hyperbolic space may face the problem of bits of precision (Sala et al. 2018). To embed tner-grained robots better, our method requires more bits. Specifically, given the height of the robot hierarchy Hmux- and the maximum number of child robots degmax= embedding will need Q Hmxx log (gg(dgmma)) bits (Sala et al., 20188). Solving these two limitations could be promising for future work and may further improve performance in other tasks.

---

Published as a conference paper at ICLR 2024
REPRODUCIBILITY STATEMENT
Our research lindings are fully reproducible. The source code is included in the Supplementary Material and will be made public once accepted. We have also thoroughly documented the installation instructions and experimental procedures in a README file attached to the code.
REFERENCES
Peter Battaglia, Jessica B Hamrick, Victor Bapst. Alvaro Sanchez-Gonzalez Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint 1rXiv:1806.000 01261, 2018.
Eugenio Beltrami. Teoria fondamentale degli spazii di curvatura costante memoria. F. Zanetti, 1868.
Jagdeep Bhatia, Holly Jackson, Yunsheng Tian, Jie Xu, and Wojeiech Matusik. Evolution gym: large-scale benchmark for evolving soft robots. Advances Neural Information Processing Systems, 34:2201-2214, 2021.
Joan Bruna, Wojeiech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks graphs. arXiv preprint arXiv:1312.6203: 2013.
Edoardo Cctin, Benjamin Chamberlain, Michael Bronstein, and Jonathan Hunt. Hyperbolic deep reinforcement learning. arXiv preprint arXiv:2210.01 2022.
Ines Chami, Albert Gu, Dat P Nguyen, and Christopher Ré. Horopca: Hyperbolic dimensionality reduction via horospherical projections. In International Conference Machine Learning, pp. 1419-1429. PMLR, 2021.
Nicholas Cheney, Jeff Clune, and Hod Lipson. Evolved electrophysiological soft robots. In Artificial life conference proceedings, pp. 222-229. MIT Press One Rogers Street, Cambridge, MA 02142- 1209, USA journals-info .... 2014a.
Nick Cheney, Robert MacCurdy, Jeff Clune, and Hod Lipson. Unshackling evolution: evolving soft robots with multiple materials and powerful generative encoding. ACM SIGEVOlution, 7(1): 11-23, 2014b.
Nick Cheney, Josh Bongard, Vytas SunSpiral, and Hod Lipson. Scalable co-optimization of morphol- ogy and control in embodied machines. Journal of The Royal Society Interface, 251147) 2018.
Francesco Corucci, Nick Cheney, Francesco Giorgio- -erchi, Josh Bongard, and Cecilia Laschi. Evolving soft locomotion in aquatic and terrestrial environments: effects of material properties and environmental transitions. Soft robotics, 5(4):475 2018.
Heng Dong, Tonghan Wang, Jiayuan Liu, and Chongjie Zhang. Low-rank modular reinforcement learning via muscle synergy. arXiv preprint arXiv:2210. 150.9, 2022.
Heng Dong, Junyu Zhang, Tonghan Wang, and Chongjie Zhang. Symmetry-aware robot design with structured subgroups. In Proceedings of the 40th International Conference on Machine Learning, 2023. /tlaattts///peeeeelleee wwwp:/wwwwwwwwwweeeeeeeeeeeeeeee wwwp:nneeetttonmmmeeee
jeHP 6aBCBu
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, et al. Coarse-to-fine vision-language pre-training with fusion in the backbone. Advances in neural information processing systems, 35:32942-3299 2022.
Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers. In International Conference Learning Representations, 2021a.
Agrim Gupta, Silvio Savarese, Surya Ganguli, and Li Fei-Fei. Embodied intelligence via learning and evolution. Nature communications, 12(1):1-12, 2021b.
10

---

Published as a conference paper at ICLR 2024
John A Hartigan and Manchek A Wong. Algorithm as 136: A k-means clustering algorithm. Journal of the royal statistical society. series e (applied statistics), 28(1):100-100) 1979.
David Howard, Agoston E Eiben, Danielle Frances Kennedy, lean-Baption Mouret, Philip Valencia, and Dave Winkler. Evolving embodied intelligence from materials to machines. Nature Machine Intelligence, 1(1):12-19, 2019.
Joy Hsu, Jeffrey Gu, Gong Wu, Wah Chiu, and Serena Yeung. Capturing implicit hierarchical structure in 3d biomedical images with self-supervised hyperbolic representations. Advances in Neural Information Processing Systems, 34:5112-2 12-31 2021.
Jiaheng Hu, Julian Whitman, and Howie Choset. Glso: Grammar-guie latent space optimization for sample-efficient robot design automation. In Conference on Robot Learning, pp. 1321-1331. PMLR, 2023.
Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pp. PMLR, 2020.
Stephen James, Kentaro Wada, Tristan Laidlow, and Andrew Davison. Coarse-to- soase-to-fing fine q-attention: Ef- ficient learning for visual robotic manipulation via discretisation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 13739-13748, 2022.
Milan Jelisavcic, Kyrre Glette, Evert Haasdijk, and AE Eiben. Lamarckian evolution of simulated modular robots. Frontiers in Roboties and AI, 6:9, 2019.
Edward Johns. Coarse-to-fine imitation learning: Robot manipulation from a single demonstration. In 2021 IEEE international conference robotics and automation (ICRA), pp. 4613 4619. IEEE, 2021.
Elias Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning combinatorial optimization algorithms over graphs. Advances in neural information processing systems, 30, 2017.
Thomas Kipf and Max Welling. Semi- sumi-super classification with graph convolutional networks. arXiv preprint 1o0vv:6609009900 2016.
Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. My body is a cage: the role of morphology in graph-based incompatible control. In International Conference on Learning Representations, 2020.
Harold Kushner. A new method of locating the maximum point an arbitrary multipeak curve in the presence of noise. 1964.
Joel Lehman and Kenneth Stanley. Evolving diversity of virtual creatures through novelty search and local competition. In Proceedings of the 13th annual conference on Genetic and evolutionary computation, pp. 21 1-2188 2011.
Hod Lipson and Jordan B Pollack. Automatic design and manufacture of robotic lifeforms. Nature, 406(6799):974-9788 2000.
Emile Mathicu, Charline Le Lan, Chris Maddison, Ryota Tomioka, and Yee Whye Tch. Contin- uous hierarchical representations with poincaré variational auto-encoders. Advances in neural information processing systems, 32, 2019.
Eric Medvet, Alberto Bartoli, Federico Pigozzi, and Marco Rochelli. Biodiversity in evolved voxel- based soft robots. In Proceedings of the Genetic and Evolutionary Computation Conference, pp. 129-137, 2021.
Zbigniew Michalewicz and Zbigniew Michalewiez. GAs: Why Do They Work? Springer, 1996.
Jonas Močkus. On bayesian methods for seeking the extremum. In Optimization Techniques IFIP Technical Conference: Novosibirsk, July 1-7, 1974, pp. 400-404. Springer, 1975.
11

---

Published as a conference paper at ICLR 2024
Georgios Pavlakos, Xiaowei Zhou, Konstantinos G Derpanis, and Kostas Daniilidis. Coarse-to tonseettofinne fine volumetric prediction for single-image 3d human pose. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7025-7034, 2017.
Marco Pedersoli, Andrea Vedaldi, Jordi Gonzalez, and Xavier Roca. A coarse-to-fine approach for fast deformable object detection. Pattern Recognition, 48(5):1844-1855 2015.
Peter Petersen. Riemannian geometry, volume 171. Springer, 2006.
Rolf Pfeifer, Max Lungarella, and Fumiya lida. Self-organization, embodiment, and biologically inspired robotics. science, 31 188883 2007.
Rui Qian, Di Hu, Heinrich Dinkel, Mengyue Wu, Ning Xu, and Weiyao Lin. Multiple sound sources localization from coarse to fine. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings. Part XX 16, pp. 292-308. Springer, 2020.
Bo Qiang, Yuxuan Song, Minkai Xu, Jingjing Gong, Bowen Gao, Hao Zhou, Wei-Ying Ma, and Yanyan Lan. Coarse-to-fine hierarchical diffusion model for molecule generation in 3d. In International Conference on Machine Learning, pp. 28277-287888 28299. PMLR, 2023.
Reuven Rubinstein and Dirk Kroese. The cross-entropy method: a unified approach to com- binatorial optimization, Monte- Carte simulation, and machine learning, volume 133. Springer, 2004.
Frederic Sala, Chris De Sa, Albert Gu, and Christopher Ré. Representation tradeoff's for hyperbolic embeddings. In International conference machine learning, pp. 4460-4469. PMLR, 2018.
Rik Sarkar. Low distortion delaunay embedding of trees in hyperbolic plane. In International symposium on graph drawing, pp. 355-366. Springer. 2011.
Paul-Edouard Sarlin, Cesar Cadena, Roland Siegwart, and Marcin Dymczyk. From coarse to fine: Robust hierarchical localization at large scale. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12716-12725, 2019.
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE transactions on neural networks, 20(1):61-80, 2008.
Charles Schaff, David Yunis, Ayan Chakrabarti, and Matthew R Walter. Jointly learning construct and control agents using deep reinforcement learning. In 20/9 International Conference on Robotics and Automation (ICRA), pp. 9798-9805, IEEE, 2019.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06344 2017.
Karl Sims. Evolving virtual creatures. In Proceedings of the 2 annual conference on Computer graphies and interactive techniques, pp. 15 15-22, 1994a.
Karl Sims. Evolving 3d morphology and behavior by competition. Artificial life, 1(4):353-372, 1994b.
Andrew Spielberg, Allan Zhao, Yuanming Hu, Tao Du, Wojciech Matusik, and Daniela Rus. Learning- in-the-loop optimization: End-to-end control and co-design of soft robots through learned deep latent representations. Advances in Neural Information Processing Systems, 32, 2019.
Kenneth O Stanley. Compositional pattern producing networks: A novel abstraction of development. Genetic programming and evolvable machines, 8:131-162, 2007.
Kenneth O Stanley and Risto Miikkulainen. Evolving neural networks through augmenting topologies. Evolutionarr computation, 10(2):99-13) 127, 2002.
Abraham Ungar. A gyrovector space approach to hyperbolic geometry. Springer Nature, 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
12

---

Published as a conference paper at ICLR 2024
Kathryn Walker and Helmut Hauser. Evolution of morphology through sculpting in voxel based robot. In ALIFE 2021: The 2021 Conference on Artificial Life. MIT Press, 2021.
Tingwu Wang, Yuhao Zhou, Sanja Fidler, and Jimmy Ba. Neural graph evolution: Towards efficient automatic robot design. arXiv preprint arXiv: MeKiv:9906 1906.05370, 2019.
Yuxing Wang, Shuang Wu, Haobo Fu, Qiang Fu, Tiantian Zhang, Yongzhe Chang, and Xueqian Wang. Curriculum-based co-design of morphology and control of voxel-based soft robots. In The Eleventh International Conference on Learning Representations, 2022.
Menglin Yang, Min Zhou, Zhihao Li, Jiahong Liu, Lujia Pan, Hui Xiong, and Irwin King. Hyperbolic graph neural networks: review of methods and applications. arXiv preprint 238.2222..8852. 2022.
Hao Yu. Fu Li. Mahdi Saleh, Benjamin Busam, and Slobodan Ilie. Cofinet: Reliable coarse-to-fine correspondences for robust pointcloud registration. Advances in Neural Information Processing Systems, 34:23872-23884, 2021.
Ye Yuan, Yuda Song, Zhengyi Luo, Wen Sun, and Kris Kitani. Transform2act: Learning a transform- and-control policy for efficient agent design. arXiv preprint urXiv22110003653 2021.
Allan Zhao, Jie Xu, Mina Konaković-Lukovic Josephine Hughes, Andrew Spielberg, Daniela Rus, and Wojeiech Matusik. Robogrammar: graph grammar for terrain-optimized robot design. ACM Transactions on Graphies (TOG), 39(6):1-16, 2020.
Shizhan Zhu, Cheng Li, Chen Change Loy, and Xiaoou Tang. Face alignment by coarse-to-fine shape searching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4998-5006, 2015.
13

---

Published as a conference paper at ICLR 2024
A
METHOD DETAILS
We outline the concrete algorithm for Sarkar's Construction in Algorithm 2
Algorithm Sarkar's Construction adapted from Sala et al. 2018)
Input: Node y with parent p. children to place a1, 02 (0, 2/)) t Fzz 2p): 0 s ;;
(1 scaling factor r.
// defined in Equation
// angle of z/p from x-axis in the plane
for n E (1.... deg(() } do
Un
cos + deg(a)
2an
siin ( 2nn
deg(a) ))
(Zg: 2pt 21 (a) -1) + Fz,, 0 (0, 2/p1 11.-. : (dog(() -d
Output: Embeddings in 2 ø1, 22,
(11
B EXPERIMENT DETAILS
B. DETAILS THE TASKS
We provide details for 15 various tasks ranging from simple to complex simulated in the EvoGym platform the following section. More elaborate explanations can be referred to Bhatia et al. (2021). In each task, the initial design space is 5 x 5.
p" represents a vio-dimensional position vector for the centroid of an object o at time t, which is computed by averaging all the positions of point masses used to compose object pe and Pg are x and y components this vector, respectively.
Position.
Velocity, Likewise, v2 denotes two-dimensional velocity vector for the center of mass of an object o at time f, which calculated by averaging all the velocities of point point-masses masses used to make up object of and og are and y components of this vector, respectively.
Similarly, 8 refers to a mie-imensionnl orientation vector for an object at time t. Given ps. which represents the position of point mass i of object and po, averaging over the angle between the vector p; p" at time t and time for all the point masses. Note that this average a weighted average determined by ||p; pa---t p time 0.
Orientation.
calculate do by
Special observations. c² describes 2n-dim position vector for all n point masses of an object relative p", which can be computed by subtracting pe from each column of the 2 n position matrix.
hg(d) and (d) are vectors of length 2d 1 that represent elevation information around the robot below or above its centroid. More specifically, for some integer S d, the corresponding entry in vector hy (d) will be the highest point of the terrain, which is less than p& between range of [x, x voxels from pe in the x-direction.
B.1.1 WALKER-VO
Figure Walker-vO.
In this task, the robot is required to walk as far as possible on flat terrain. This task is easy. Given the observation space S E TR that is formed by concatenating vectors of robot robot where is the
14

---

Published as a conference paper at ICLR 2024
number of point masses, the reward R. can be represented below:
R Ap
which provides reward to the robot for making forward progress in the positive x-direction. It also receives an additional reward of 1 if the robot reaches the end of the terrain.
B.1.2 DOWNSTEPPER-VO
Figure 7: DownStepper-vO.
In this task, the robot achieves rewards by descending stairs of various lengths. This task is easy. Given the observation space E TR=+14 that constituted by vectors robut grobot crobot hg (5). where n is the number point masses, the reward R can be formulated follow:
R Aprobot
which rewards the robot for moving in the positive x x-direction.
The robot also receives additional reward of 2 upon successfully reaching the end of the terrain but will penalized with a one-time deduction of for rotating more than 75 degrees in either direction from its initial orientation, similar to the Up Stepper task.
B.1.3 JUMPER-VO
Figure 8: Jumper-vO.
In this task, the robot is required to jump high possible in place flat terrain. This task is easy. Given the observation space S E Ru+7 that is formed by concatenating vectors o robot rabot hg (2), where is the number of point masses, the reward R is
R = 10 Ap, 5 Ip tooobtt
which implies that the robot will receive reward for its upward motion in the positive y-direction and penalty for any movement in the x-direction.
1.1.4 FLIPPER-VO
In this task, the robot aims to perform as many counter-clockwise flips as possible on flat terrain. This task is easy. Given the observation space S Rn+1 that formed by concatenating vectors grobot rooot the reward R is represented as below: R roobtt
which rewards the robot for executing counter-clockwise rotations.
15

---

Published as a conference paper at ICLR 2024
Figure 9: Flipper-vO.
B.1.5 PUSHER-VO
A A
Figure 10: Pusher-vO.
In this task, the objective of the robot is to push a box that has been placed in front of it. This task
easy. Given the observation space S E gn+6 that is constituted by vectors boo pboz prbout y robot robot where n is the number of point masses, the reward Rt can be represented as the sum of R1 and Rg.
R1 = 0.5 Apr robot + 0.75 Apz box
which rewards the robot for its movement in the positive x-direction.
Rs A p robot
which penalizes the robot for moving apart in the x-direction.
It also receives an additional reward of 1 if the robot reaches the end of the terrain.
B. CARRIER-VO
Figure 11: Carrier-vO.
In this task, the robot is required catch a box initialized above it and carry as far as possible on flat terrain. This task is easy. Given the observation space S E R that is constituted by vectors boz boo p roboot probot crobot where the number of point masses, the reward R, R is the sum of several components.
R1 0.5 Apt robot 0.5 Ap how
which rewards both the robot and box for movements in the positive x-direction.
0
how 2 fy otherwise
Ra
10
Apper box
which implies that the robot will receive penalty for dropping the box below threshold height by
16

---

Published as a conference paper at ICLR 2024
Figure 12: UpStepper-vO.
B.1.7 UPSTEPPER-VE
In this task, the robot is rewarded by climbing up stairs of diverse lengths. This task medium. Given the observation space 1-++1 that is constituted by vectors robot grobot cobbtt arobot(5), where n is the number point masses, the reward formulated as follow:
R = Ap
which incentivizes the robot to move in the positive x-direction.
The robot is also granted an additional reward upon reaching the end of the terrain but will be penalized with a one-time deduction of 3 for rotating more than 75 degrees in either direction from its original orientation.
B.1.8
Figure 13:
In this task, the robot is rewarded by walking across terrain that gets increasingly bumpy. This task is medium. Given the observation space S ga++1 that constituted by vectors rabof grabot rabot
hrobot (5), where refers to the number of point masses, the reward R is formulated as follow:
Ap, robot
which incentivizes the robot move in the positive x-direction.
The robot also receives an additional reward of 2 upon successfully reaching the end of the terrain but will be penalized with a one-time deduction of 3 for rotating more than 90 degrees in either direction from its initial orientation.
B. THROWER-VE
In this task, the robot is rewarded by throwing a box that is initially positioned on top of This task is medium. Given the observation space S + that is constituted by vectors yoo pooxr p robot rooot robot where n is the number point masses, the reward R can be represented as the sum of
R, and R2-
Ri
Ap boz
17

---

Published as a conference paper at ICLR 2024
Figure 14: Thrower-vO.
which rewards the box for moving in the positive x-direction.
Approbot Aprobot
probot 2 0
R2
otherwise
which imposes penalty on the robot if it moves too far away from the 0 while throwing the box.
1.1.10 PLATFFRMIIUMPER-VO
Figure 15: PlatformJumper-vO.
In this task, the robot navigates through sequence of floating platforms positioned at varying heights. This task is hard. Given the observation space S R=+11 that is constituted by vectors robot grobot robot robot (5), where n is the number of point masses, the reward R is formulated as follow:
Ap
which provides a reward to the robot for making forward progress in the positive x-direction.
If the robot rotates more than 90 degrees from its initial orientation in either direction falls off the platforms, it incurs a one-time penalty of -3.
B.1.11 OBSTACLETTAVERREER.
In this task, the robot is required to traverse a rougher terrain compared the previous Obstacle Traverser task. This task is hard, Given the observation space S E 1R*+14 that is constituted by vectors robot grobot robot hr hrooot (5), where refers the number of point masses, the reward R can be formulated as follow:
R Aprooot
which rewards the robot for its movement in the positive x-direction
The robot also receives an additional reward 2 upon successfully reaching the end of the terrain but won't t be penalized.
18

---

Published as a conference paper at ICLR 2024
Figure 16:
B.1.12 HURDLER-VO
Figure 17: Hurdler-vO.
In this task, the robot achieves rewards by walking across terrain that features tall obstacles. This task is hard. Given the observation space E IR * that is similar to ObstacleTrrverserr-yy the task-specific reward R can be formulated as follow:
R = Ap robut
which rewards the robot for moving in the positive x-direction.
incurs one-time penalty of if the robot deviates more than 90 degrees from its original orientation in either direction.
B.1.13 TRAVERSER-VO
Figure 18: Traverser-vO.
In this task, the robot navigates across pit filled with rigid blocks without sinking into the pit. This task is hard. Given the observation space 11 that is similar to @bstacleTraverser. the task-specific reward R can be represented below:
R Apt robot
which provides reward to the robot for making forward progress in the positive x-direction.
The robot is also granted additional reward of upon reaching the end of the terrain.
19

---

Published as a conference paper at ICLR 2024
Figure 19: Carriervvl.
B.1.14 CARRIER-V1
In this task, the objective of the robot is to carry a box to table and place it on the table. This task hard. Given the observation space S R=+6 that is similar to Thrower-vO, the reward R can be formulated as the sum of R1 R2 and Rz.
R1 2 A Algg
box boz
where g refers to the goal x-position for the box. It rewards the box for moving to its goal in the x-direction.
R2
A
robot
Pr.
robob
where grahot describes the goal x- x-position for the robot. The robot is rewarded for making forward progress its target the positive x-direction.
0
if p hour 2 ty otherwise
Rs
10
Apher
which imposes a penalty on the robot if it drops the box below a threshold height by that varies with the elevation of the terrain.
B.1.15 GAPJUMPER-VO
Figure 20: GapJumper-vO.
In this task, the robot moves across sequence of spaced-out floating platforms, all positioned at the same height. This task is hard. Given the observation space Ro+14 that is constituted by vectors robot grabot robot brobot (5), where refers the number of point masses, the reward R
formulated as follow:
R = Apr robot
which incentivizes the robot to move in the positive x-direction.
If the robot falls off the platforms, it incurs one-time penalty of -3.
20

---

Published as a conference paper at ICLR 2024
12
Figure 21: Use the nearest valid embedding and its corresponding design.
B.2 IMPLEMENTATION OF OUR METHOD HERD
We implement HERD based on the public code code CuCo (Wang et al. 2022). which uses Transformer- based Kurin et al. (2020) control policies. This architecture can deal with variable input sizes across different robot designs and can capture internal; dependencies between cells. Note that our method is general and can also use any other networks that are able to handle incompatible state-action space, such as GNN (Scarselli et al. 2008 Bruna et al., 20133 Kipf & Welling (016)) and message passing network (Huang et al. 2020). For control policy learning, we use Proximal Policy Optimization
(PPO) (Schulman et al.
2017 in CuCo.
For the designing component, the implementations of Sarkar's Construction and Poincaré ball are based the released code of HoroPCA (Chami et al., 2021). In Figure 21 we simplified Poincaré disk, where the depth of the tree and the number of branches is reduced for better visualiza- tion, show how select the nearest valid embedding and its corresponding design. We simply calculate the distance between the sampled point 2 and other points (such as 21, 22, 23, ...) that are embedded in the hyperbolic manifold. Then we choose the point and its corresponding robot design with minimum distance, i.e., 22 in Figure 21
Here we provide the hyperparameters needed to replicate our experiments in Table and also include our codes in the supplementary.
Experiments are all conducted on NVIDIA GTX 2080TI GPUs with 80 CPUs. Taking TratteceeTaavvrrr-vvv as an example, HERD requires approximately 4G of RAM and IG of video memory and takes about 28 hours to finish 25M timesteps of training.
B.3 DETAILS OF BASELINES
CuCo. CuCo (Wang et al. 2022 incorporates curriculum learning for co-optimization of design and control by establishing a simple-to to-pleex process. It enables gradually expanding the design space from initial size to target size through predefined curriculum. By utilizing an indirect- encoding Neural Cellular Automata (NCA) and an inheritance mechanism, CuCo achieves a balanced exploration and exploitation. At each stage of the curriculum, it learss transferable control patterns via Reinforcement Learning (RL) and adopts the self-attention mechanism to capture the internal dependencies between voxels effectively.
GA. We have implemented a basic Genetic Algorithm (GA) that employs clitism selection and simple mutation strategy to evolve the population of robot designs. The elitism selection mechanism preserves robots with high fitness from the current population as survivors while discarding the rest. The mutation strategy introduces random modifications individual voxels of the robot with specified probability. Although we have not incorporated crossover for the sake of simplicity, well-designed crossover operator could potentially enhance final performance.
CPPN-NEAT. In this approach, the morphology of the soft robot is represented and parameterized using a Compositional Pattern Producing Network (CPPN) (Cheney et al. 20 4a Stanley 2007).
httpar//attaaaocco//////eeeeeegu arrooymm
21

---

Published as a conference paper at ICLR 2024
Table 1: Hyperparameters of HERD.
Hyperparameter
Value
Dimension of Poincaré ball d
Curvature of Poincaré ball c
1.0
Design Optimization
Hierarchy size N
2
10
Population size of CEM No
10
Use GAE
True
GAE parameter
0.95
Learning rate
2.5
10-1
Use linear learning rate decay
True
Clip parameter
0.1
Value loss coefficient
0.5
Entropy coefficient
0.01
Time steps per rollout
2048
Num processes Optimizer
PPO
Adam
Evaluation interval
10
Discount factor y
0.99
Clipped value function
True
Observation normalization
True
Observation clipping
10, 10)
Reward normalization
True
Reward clipping
| 10, 10)
Policy epochs
Number of layers
Number of attention heads
Embedding dimension Feedforward dimension
64
Transformer
128
Non linearity function Dropout
ReIUU
0.0
As CPPN receives all the spatial coordinates of voxels as input, we can acquire the type of each corresponding voxel. The NeuroEyolution of Augmenting Topologies (NEAT) algorithm (Stanley & Müikkulainen 2002) is leveraged to evolve the structure of CPPNs, which operates as a generic algorithm tailored for network structure.
BO. Bayesian Optimization (BO) (Kushner) 1964 Schaff et al. 2019) a widely adopted technique for global optimization, especially for optimizing functions that are computationally expensive evaluate. Specifically, the surrogate model employed in this approach is based on Gaussian processes, while batch Thompson sampling is extract the acquisition function. We rely on the L-BFGS algorithm during the optimization process.
Vanilla RL. Vanilla RL method shares the same network architectures as CuCo (Wang et al. 2022 but removes the curriculum component from the framework. It learns to design and control from scratch within the design space.
HandCrafted Robot. To evaluate the effectiveness of automatic robot design, construct a robot in the design phase using human knowledge, and solely requires learning control policies for specific morphologies.
The baselines CuCo, Vanilla RL, and HandCrafted Robot adopt the same control policy architecture (transformers) and learning method (PPO) as method HERD. For the other baselines GA, BO, and CPPN-NEAT, we use their official implementations from CuCo and EvoGym. Specifically, they also transformers control policy architecture and use PPO optimization method. The main difference that the control policy not shared across robot designs and needs to learned from scratch.
22

---

Published as a conference paper at ICLR 2024
B.4 DETAILS ABLATIONS
HERD w/o HE. The idea of coarse-to-fine can be implemented by other algorithms. HERD w/o HE uses NGE (Wang al. 2019). an evolutionary algorithm originally used for rigid robot design. We re-implemented the algorithm in EvoGym based on their released code The core modification is the mutation procedure. To enable multi-cellular robot design, we randomly change the type of each robot cell with probability 0.1, where the types are chosen from {Empty, Rigid, Soft, Horizontal Actuator, Vertical Actuator). To enable coarse-to-fine robot design, we refine each robot with probability 0.2 if it is coarse-graine -ggained..... Then the mutated robots are viewed as offsprings and included in the robot population NGE.
NGE. For full comparison, also use the original NGE without coarse-to-fine. The modification of NGE is similar to HERD w/o HE and the only difference that NGE does not have coarse-to-fine component.
HERD w/o C2F. The optimization the robot designs HERD is based on CEM, which works in Poincaré ball. HERD w/o C2F does not adopt the idea of coarse-to-fine, but uses CEM to directly search the optimal design. This ablation can verify the effectiveness of coarse-to-fing robot design. Specifically, the major difficulty of using CEM for optimizing robots in EvoGym is that the design space of EvoGym discrete. We choose to let CEM optimize the probabilities of robot cells types (P E 22555), and take the types with maximum probabilities ("Pargmax(dim=1)),
For a fair comparison, the control policies of all the variants in ablation studies adopt the same network architecture and same optimization algorithm (PPO) method HERD.
Algorithm 3: HERD w/o HE
Input: robot design space D. population size of NGE NNge== 64,
initialize control policy æ, population of boarse-graines robots {D:): N while not reaching max iterations do
replay buffer H <- 0;
for i {1,
Nnge] do
use л to control current robot design D, and store trajectories to Hi; Eliminate 30% of the robots with poor performance:
while not reaching the population size Nnge do sample one robot from the remaining robots and draw μ from [0, uniformly;
if μz < 0.8 then
10
alter the type of each component with the probability of 0.1: else
12
alter the granularity of one component; add the offssring into the candidate pool;
// coarse-to-fine mutation
update æ with PPO using samples in 74; 14
15 D +- robot design with the highest performance:
// optimal robot design
16
Output: optimal robot design D control policy
C EXTRA EXPERIMENTS
In this section, provide the full experiment results of 15 tasks. Figure 22 show the training performance of HERD compared with baselines. HERD outperforms other baselines in most tasks, including easy, medium and hard, which further showcase the effectiveness of our method.
presents the performance of several variants of HERD. It again supports our observation that the idea of marse-to-fine helpful for robot design problems and hyperbolic embeddings can further bring better optimization properties.
Figure 23
The performance of HandCrafted Robot is obtained by averaging the final performance of four random runs. The reason why we did not show its learning curve in the main text is that it is not a robot design method. HandCrafted Robot has fixed human-designed robot morphology and only
htttps:///gt www.. wi'h BowNangg
23

---

Published as a conference paper at ICLR 2024
Algorithm 4: HERD w/o C2F
Input: robot design space D, population size of CEM N.
initialize control policy CEM mean and variance o;
while not reaching max iterations
replay buffer H <- 0:
for E {1, (M, d // sample embedding from Euclidean space D; f arg maxdim=1 Di control current robot design D, and store trajectories H; update with PPO using samples in H;
No) do
update ş by averaging the clite 0;s based on the performance in H. and linearly decrease or; // optimal robot design
10 D < arg max μ
Output: optimal robot design D", control policy
(Ours)
Vanilla RI.
Handcrafted Robot Best Performance
CuCo
GA
CRN--EAT Down sespesul
easy
Jump
Epicode Rescaat
7.5-
Epiizote
5.0-
Spicoda
(mi0
T (mil)
[mi)
50
Rispeeud
easy
15
Pushenul
easy
Carrieuo
easy
Sprode Retond
40-
Eptonde floaadd
Eptodd Rouard
18
kg
30
10
15
20
10
20
10
T
UnStepperso
medium
DestacteTrvverprro0
medium
Throwerv
medium
Epioda Roward
Couadde Recvrdd
Roward
T (mill
(ml)
vo
hard
Gestadeefrverrsrv
hard
Hurdiervo
hard
Extend Ronag
Eppande Roward
20
10
Timill
T
T (mi)
Travesseuo
hard
Camrntt
hard
Geglumpeson
hard
Ramard
Edtrade Reward
a
Figure 22: Training performance of HERD compared against baselines in 15 tasks.
needs to learn control policy. unfair to directly compare this baseline with other robot design methods, whose robot morphologies are learned online. In this paper, the final performance
24

---

Published as a conference paper at ICLR 2024
Velkerv0
6asy
(Ouns)
DoenStaspar
easy
w/o
Jumpervd pasy
10.0
Edtood Roward
Estrndk Roward
Edicdd
2-5
10
15
20
10
15
20
30
15
20
T(mll)
(mll
Rippervo
easy
Pushers0
easy
Carreevv
easy
Enisdde Sewwnd
Epiands Rarward
Epraeds Reverd
15
(mml))
T(rill
Emil
medium
medium
medium
Spicade hewand
Epteate Rowordd
Eptzode Roward
10
T (mill
T [mi)
Patformjumpec hard
Obvene le Tracersenel
hard
Hunllesw
hard
Epioodi Roovord
Epiocde Roward
Epioctte
1.4
10
10
15
Carreorv1
Gastumpervv
hard
Epkunce thoordd
Estroc Revard
T (m/l)
T 0m/l
mm0
Figure 23: Ablation studies of HERD in 15 tasks.
improvement of our method over this baseline to show the necessity of creating robot design methods. However, for complete comparison, we also include the learning curve Figure 24
For full comparison, we also test our method as well as baselines in other tasks of EvoGym in Figure 25 The tasks that have not been solved by our method and other baselines can be roughly classified into two categories: (1) the tasks require complicated control policies such as Pusher-vl and (2) the tasks require delicate robots such as Lifter-vO, These unsolved tasks might require further investigation. An do not include the task Walkerrr, because it contains bugs in its task implementation.
D QUANTITATIVE RESULTS
To better understand the evaluation of different methods, we also show the quantitative results in Table T
E DISCUSSION OF OTHER RELATED WORK
Our work adopts an embedding approach in hyperbolic space and optimizes robot designs this space. The most similar previous work to our paper GI Gl (003)). which uses variational
25

---

Published as a conference paper at ICLR 2024
(Ouns)
BO
CuCo
GA
CPPN NEAT
DownStespperd
10.0
Eplioou Rlwward aaard
Sptrooe Roward
7.5
Eptoode
T [mil)
T [mill
(mill
50
15
Pushetuu
Sphodde Rexeen
60
Epecde Reviar
10
Spicogle Novvun
10
10
20
10
10
15
(mill
(miill
UpStepperv0
Obstadlerrveeraruo
Trrowerve
Eskcadl Requrrd
Edkodde Roward
M
10
10
15
10
15
20
[mmll
(mill
Patromjunpe-v0
ObstacteTawerreevu
Esiianda Nurarr
Espoodo Noward
Espaante
T [mi)
T [mil]
T (mill
Traversewv
Camanee
Capfommnnc
Spieose Neward
Spio0e Reward
10
10
15
10
35
20
(mill
Figure 24: Training performance HERD compared against baselines 15 tasks.
Table 2: Quantitative results of our method HERD compared with various baselines.
(Ours)
CuCo
Vanilla
BO
CPPN-NEAT
Handerafted
Walker-vO
9.74 0.81
6.68 2.73
9.49 0.62
10.57 0.00
2.22 0.67
10.50 0.10
DownStepper-v0
8.04 = 0.76
3.00 2.33
6.87 1.78
4.39 0.66
2.20 0.58
2.01 = 0.60
8.06 0.34
Jumper-v0
2.71 1.74
-2.45 0.30
-2.49 0.29
0.04
-0.04 0.07
0.02=00.0
3.15 1.29
Flipper-v0
35,55 + 18.37
15.01 13:22
13.00 + 13.96
0.52
0.71 0.43
0.37 00.16
26.96 2.00 2.00
Pusher-vO
1.30
0.16
0.73
0.94
1.77
Carrier vO
4.33 0.62
0.06 0.04
0.14 0.18
1
0.11
0.21 0.13
5.84 0.60
U
3.48
Obstacle ObstacleThaeeseer Traverser w0)
6.02 0.63
0.89 0.41
0.62
1.87
1.48 à 0.03
1.35 0.10
2.90 2 1.03
Thrower-v0
2.96 1.78
0.91
PlatformJumper-v0
3.34 1.97
1,51 1.51
0.06
0.66
1,36
1.36
1.41
PestncleTrvverser-//
0.33
Hurdler-vO
3.21 0.45
-0.02
0.08
0.71
0.50
0.46
3.18 1,90
TraverservvO
5.48 0.80
3.10
0.96
2.77
1.10
1.550
2.81 0.16
Carrie- v1
1.93
0.125
0.01 0.0.1
2.50
0.27
0.03 1.03=0.00
1.81 0.47
Gaplumper-vO
4.05 1.00
1.03 + 2.79
2.58 1.15
1.95
1.36
1.40
2.34 0.60
auto-encoder to embed rigid robots. We did not compare our method with GLSO because requires some prior knowledge about design space that is not available in EvoGym and it cannot be directly used multi-cellular robot design problems. In summary, our method has several advantages over GLSO.. (1) Training cost. HERD uses training-free embedding method while GLSO requires
26

---

Published as a conference paper at ICLR 2024
(Durs)
Handcraftd Performance
Bridge/Woheeoo
Epiadd Roward
Eahoode Barmannd
15
20
15
20
-10
10
15
20
(mm])
T ((ml)
Mingseemaavmmeeeeo
Balancerv
Pusher
Edronde Reward
Ediadd Reward
3
10
15
20
10
15
20
-1.9
10
15
20
Imlll
(mi)
(mi)
ClimbervO
Climberv1
Beam Tooplerv
Eplrad Rewend
20
20
Imill ArwoMinimizeseal
(m/l)
T (ml)
10
1.0
Balantwoer
Noward
Epcooe Reword
2
15
-1.0g
20
0ri0
[mi]
T [ml) Lltterd
10
Cotcher
10
10.
Epsend Reward
0:
15
mmill
T (ml)
(ml)
Climber
2
mmll
Figure 25: Performance of HERD in Other Tasks of EvoGym.
training VAE, which could be computationally expensive and unstable. (2) Prior knowledge. HERD does not require as much domain- specific knowledge GLSO does. Firstly. for data collection, GLSO adopts the set of grammar rules proposed in RoboGrammar Zhao et al. (2020). which requires prior knowledge of design space to produce tractable and meaningful subspace of designs (Section 3.1 of the GL.SO paper). Secondly, GL.SO also co-train a proper prediction network (PPN) with VAE, and the training objective of PPN is also built prior knowledge of the locomotion tasks they test (Section 3.4 the GLSO paper). (3) Inductive bias. HERD explicitly encodes the hierarchical structure of design space, which could benefit the optimization process as we show in our experiment section, while GLSO does not consider any similar inductive bias.
27

---

Published as a conference paper at ICLR 2024
Table 3: Quantitative results of our method HERD compared with various ablations.
HERD (Ours)
NGE
HERD w/o HE
HERD w/o C2F
Walker-vO
9.74 + 0.81
7.16 + 0.41
8.07 + 0.32
2.76 2.49
DownStepper-vO
7.91 = 0.76
2.49 = 0.09
6.22 + 0.61
3.21 0.69
Jumper-v0
2.71 # 1.74
0.35
0,03 + 0.67
-0.06 0.22
Flipper-v0
35.55 18.37
2..08
19.27 + 8.98
1.06
Pusher-v0
7.40 1.46
1.80
5.45 0.39
3.28
Carrier-v0
0.79
5,85 1.88
1.81
UpStepper-v0
4.14
0.22
2.55 1.31
0.77
ObstacleTrrvvereer-v00
0.63
0.07
2.26 0.26
1.15
Thrower-v0
3.01
0.01
0.80 0,42
0.52
PlafformJumper-v0
1.97
1.14
1.50 0.08
1.14
ObsteeleTraavrserrvvll
4.92
0.27
2.00 0.43
0.17
Hurdler-vO
2.69
0.06
0
0.46
Traverser-v0
5.48 # 0.80
2.28 0.40
2.70
2.57
Carrier-vl
1.93 = 0.33
2.06
2.43 0.40
1.68 0.166
GapJumper-v0
4.05 1.00
1.10
1.95 0.41
1.61 0.38
28


Reviews:
[{"review_id": "TF5Yv0EEI9", "reviewer": "Reviewer_WvQM", "summary": "This paper addresses the task of multi-cellular soft robot design given a specific robotic task in simulation. \n\nIt proposes a novel robot design optimization method to enable efficient optimal design search in a vast robot design space. The proposed method searches the design space in a coarse-to-fine manner. The design space is first transformed into a hyperbolic space, where each robot design is embedded into the hyperbolic space via a train-free strategy. The finer design has a larger embedding norm, forming a coarse-to-fine robot hierarchy in the hyperbolic space. Then, the Cross-Entropy Method is adopted to search in the hyperbolic space.\n\nThis paper uses 15 tasks from EvoGym for experiments. It compares with manual design and several robot design methods. It also conducts ablations on the coarse-to-fine and hyperbolic embeddings.", "questions": "1. What are the policy architecture and learning methods adopted for the baselines? Are they the same as those of HERD?\n2. HEARD w/o HE:  how to integrate NGE with CEM? Could you provide more details on this ablation?\n3. How is the performance of a HandCrafted robot determined？ Does it use a transformer-PPO policy? If yes, it should also be reported with variance, e.g., in Figure 3. \n4. How can we guarantee that sequential refinements on a coarse design are better than sequential refinements on another coarse design, given that the former coarse design outperforms the latter coarse design?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. This paper is well-written.\n2. The experiment section provides enough evaluations, i.e., 15 diverse tasks in the simulation.\n2. Searching robot design space in a coarse-to-fine manner via hyperbolic space is novel.", "weaknesses": "1. The issue of local optimal (not very good performance compared to other baselines) and large variance: Does the parent node of an optimal robot design node consistently outperform the parent node of a non-optimal robot design node? If not, coarse-to-fine seems greedy and vulnerable to local optimal. The proposal method, HERD, appears to have no mechanism for exploration in the global design space, potentially leading to the high variance in Figure 3. And HERD is not significantly better than baselines.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Mvrr3YoZji", "reviewer": "Reviewer_9c5Z", "summary": "This paper proposes a coarse-to-fine method for designing multi-cellular robots with hyperbolic geometry.", "questions": "please see the weaknesses", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The paper is well written, easy to follow, the visualizations are nicely designed and the accompanying website shows impressive results.\n2. Incorporating hyperbolic geometry into robot planning is innovative and interesting.\n2. The experiment shows promising results over a variety of tasks.", "weaknesses": "1. As one of the concerns of multi-cellar system, the design space is very large, so how hyperbolic solution solved this problem is not well examined.\n2. The quantitative analysis of the proposed model is somewhat absent, some tables will help the understanding of the evaluation of different methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KPdmE7VUO2", "reviewer": "Reviewer_S1iZ", "summary": "The paper proposes a novel co-design algorithm for multi-cellular robots. The authors noticed two critical insights 1. \"coarse-grained robots are less complicated to design and control due to their low degree of freedom\" and 2. \"coarse-grained robots can usually solve part of the task\". Built on these two insights, the authors proposed a coarse-to-fine design algorithm where a CEM optimizer optimize robot design in Poincare ball hyperbolic embedding space. The hyperbolic embedding embeds a coarse-to-fine tree structure so the optimization effectively. The authors benchmark the proposed algorithm on a variety of environments in evo-gym and shows the effectiveness of the algorithm.", "questions": "- The authors chose Poincare ball as the hyperbolic embedding space for the design tree. I am wondering how general is this? In particular, hyperbolic manifold seems very limiting. For example, let's say the robot can be comprised of material A and material B. From the center, I extend two different trees branching towards material A and material B separately. However, in each branch, fine-grained branches start to contain configurations that blends A & B which means the two branches should merge. In this case, you will have to have a finite cylinder manifold, correct?\n\n- How exactly does the manifold correspond to designs or robot parameters? How many branches are there? I understand how do you map between the euclidean space and the hyperbolic space but this seems unclear. Giving more visualizations across the disk manifold will be helpful.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The writing quality of this paper is very good. All design choices are introduced with clear motivation. Ideas are explained clearly with text and high-quality figures.  \n- It's quite novel for the authors to map the robot design to a space that has a favorable optimization landscape for methods as simple as CEM. \n- The two insights the authors point out, 1. \"coarse-grained robots are less complicated to design and control due to their low degree of freedom\" and 2. \"coarse-grained robots can usually solve part of the task\", has a potential to motivate new research\n- The proposed method is a sound algorithm  to the cellular robot design problem and the experiments thoroughly justified the point.\n\nOverall this is a technically solid paper with novel contributions that's clearly above acceptance threshold.", "weaknesses": "- It's unclear how general is the hyperbolic space is for robotics design. See my questions in the next section\n- It's unclear to me to me how exactly any point on the manifold correspond to a robot configuration. Some concrete example may help understanding. If I randomly sample a point using a coordinate, what's the robot configuration it correspond to? It seems that human has to handcraft such rules.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3KNlyoEJ1u", "reviewer": "Reviewer_Wtet", "summary": "This paper investigates the soft cellular robot design problem. Instead of traditional approaches that apply genetic algorithms directly in the design space, this paper proposes to optimize the robot design in the robot embedding space, specifically a 2D hyperbolic space defined by Poincare ball. It constructs the embedding space by first sampling a set of robots in a coarse-to-fine procedure and applying Sarkar’s method to construct the Poincare ball. It then finds the optimal design from the sampled set by applying CEM in the hyperbolic space. During design optimization, it follows previous work [1] to gradually adapt and optimize a transformer-based policy via PPO to control the robot designs in the current CEM population.", "questions": "1. The coarse-to-fine process seems specialized for the cellular robot representation in EvoGym. Is it generalizable to other robot design representations, such as articulate rigid robots (e.g. Wang et al. 2019 [1], Gupta 2021 [2])?\n\n2. Since coarse-to-fine clustering is based on running K-Means with different random seeds, it seems to be possible that there are two robot designs located in different subtrees. Will such two designs be embedded in similar locations in the hyperbolic space?\n\n3. Being relevant to the last question, it seems that the cell type is not used during the embedding. If I understand correctly, the embedding only captures the structural/subdivision similarity but not the cell functionalities. How can we interpret the distance in the embedding space into the similarity between designs? \n\n4. Some notations are unnecessarily complicated. For example, it would be easier to understand if just saying the algorithm clusters the robot cells into 2, 4, 8, …, Nc/2, Nc components instead of using logarithm and exponential expressions.\n\n5. Page 6 “In practice, similarity here means that fine-grained robots only need to change one component to be the same as their parent robots”. What does it mean? The fine-grained robots have more clusters than the parent robot, does it mean the fine-grained robots only change the cell type of one “new” cell cluster?\n\n6. During the algorithm, is the robot design set fixed after the initial sampling? In other words, when you do $argmin$ in line 9 (Algorithm 1), do you always search for a design in the pre-sampled robot design pool?\n\n7. After reading the paper, the phrase “coarse-to-fine” is indeed the way of building the candidate robot design pool instead of a coarse-to-fine optimization process. If this understanding is correct, the paper (especially the abstract and introduction) needs to be more clear about this point to prevent confusion.\n\n8. EvoGym has 32 tasks however the algorithm is only tested on 15 of them. It would be great if the authors could evaluate the proposed algorithm on other tasks as well.\n\n9. It would be great to provide the pseudo-code for the ablation variations to help understand the design of those ablation methods. For example, does HERD w/o C2F differ from full NERD mainly on the pre-sampled robot design set? So how does HERD w/o C2F sample the robot design set?\n10. From the visual results, most of the optimized designs only have one single cell type. Does this result from the limitation of the embedding approach since the embedding only captures structural information but is not informative about the cell types?\n\n11. How does the proposed embedding approach compare to other embedding approaches such as neural network embedding (e.g. GLSO [3])\n\n[1] Tingwu Wang, Yuhao Zhou, Sanja Fidler, and Jimmy Ba. Neural graph evolution: Towards efficient automatic robot design\n\n[2] Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Metamorph: Learning universal controllers with transformers\n\n[3] Jiaheng Hu, Julian Whitman, and Howie Choset. GLSO: Grammar-guided Latent Space Optimization for Sample-efficient Robot Design Automation", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. Parameterizing the discrete robot design space into continuous embedding space and optimizing the robot in this continuous space is novel and interesting.\n\n2. The experiments clearly show the advantages of the proposed NERD over baselines.", "weaknesses": "1. The generalizability of the proposed approach to other robot design representations is uncertain. Testing the approach on one common robot design representation would make it more convincing.\n\n2. The choice of Sarkar’s construction for hyperbolic embedding needs more justification.\n\n3. The experiments need improvements.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)