Curriculum Reinforcement Learning via Morphology-Environment Co-Evolution
Authors: Shuang Ao
Tianyi Zhou
Guodong Long
Jing Jiang
Venue: Submitted to ICLR 2023 (2023)
Keywords: Curriculum RL
Morphology Evolution

Abstract: Throughout long history, natural species have learned to survive by evolving their physical structures adaptive to the environment changes. In contrast, current reinforcement learning (RL) studies mainly focus on training an agent with a fixed morphology (e.g., skeletal structure and joint attributes) in a fixed environment, which can hardly generalize to changing environments or new tasks. In this paper, we optimize an RL agent and its morphology through ''morphology-environment co-evolution (MECE)'', in which the morphology keeps being updated to adapt to the changing environment, while the environment is modified progressively to bring new challenges and stimulate the improvement of the morphology. This leads to a curriculum to train generalizable RL, whose morphology and policy are optimized for different environments. Instead of hand-crafting the curriculum, we train two policies to automatically change the morphology and the environment. To this end, (1) we develop two novel and effective rewards for the two policies, which are solely based on the learning dynamics of the RL agent; (2) we design a scheduler to automatically determine when to change the environment and the morphology. We find these two designs are critical to the success of MECE, as  verified by extensive ablation studies. In experiments on two classes of tasks, the morphology and RL policies trained via MECE exhibit significantly better generalization performance in unseen test environments than SOTA morphology optimization methods. Our ablation studies on the two MECE policies further show that the co-evolution between the morphology and environment is the key to the success.

Under review as a conference paper at ICLR 2023
CURRICULUM REINFORCEMENT LEARNING VIA MORPHOLOGY-EIVIROOMENY CO-EVOLUTION
ABSTRACT
Throughout long history, natural species have learned to survive by evolving their physical structures adaptive to the environment changes. In contrast, current reinforcement learning (RL. studies mainly focus on training an agent with a fixed morphology (e.g.. skeletal structure and joint attributes) in a fixed environment, which can hardly generalize to changing environments new tasks. In this paper, we optimize an RL agent and its morphology through mmorphology-enviionnee co-evolution (MECE)", in which the morphology keeps being updated adapt the changing environment, while the environment is modified progressively to bring new challenges and stimulate the improvement of the morphology. This leads curriculum to train generalizable RL, whose morphology and policy are optimized for different environments. Instead of hand-crafting the curriculum, train two policies to automatically change the morphology and the environment. To this end, (1) we develop two novel and effective rewards for the two policies, which are solely based on the learning dynamics of the RL agent; (2) we design a scheduler to automatically determine when to change the environment and the morphology. We find these two designs are critical to the success of MECE, verified by extensive ablation studies. experiments on tWO classes tasks, the morphology and RL policies trained via MECE exhibit significantly better generalization performance in unseen test environments than SOTA morphology optimization methods. Our ablation studies the two MECE policies further show that the co-evolution between the morphology and environment is the key to the success.
INTRODUCTION
Deep Reinforcement learning (RL) has achieved unprecedented success in some challenging tasks (Lil- licrap et al., 2016; Mnih et al., 2015). Although current RL can excel on specified task fixed enviromment through massing training, it usually struggles to generalize to unseen tasks and/or adapt to new environments. promising strategy to overcome this problem is to train the agent on multiple tasks in different environments (Wang et al., 2019a; Portelas et al., 2019; Gur et al., 2021; Jaderberg et al., 2017) via multi-task learning or meta-learning (Salimans et al., 2017; Finn et al., 2017). How- ever, it increases the training cost and the space for possible environments/tasks can be too large to be fully explored by RL. So how to select the most informative and representative environments/tasks to train an RL agent to evolve generalizable skills becomes critical open challenge. Curriculum learning (Narvekar et al., 2020) for RL aims at developing a sequence of tasks for RL to progressively improve its generalization performance through multiple training stages. However, the curriculum usually relies more on human heuristics, e.g. moving from easy environments to hard ones, chasing the ones with the greatest progress, while lacking sufficient innate incentives from the agent itself to drive the changes towards improving the learning process.
Unlike RL agents that do actively seek new environments to improve their learning capability, natural species have full motivations to do so to survive in the competitive world, and one underlying mechanism drive them is evolution. Evolution a race with the changing environment for every species, and a primary goal is to accelcrate its adaptation to new environments. Besides merely optimizing its control policy, evolution more notably changes the morphology of species, i.e., the skeletal structure and the attributes for each part, in order to make them adapt to the environment. In fact, the improvement on morphology can be more critical because there could exist a variety of actions or skills an agent cannot do (no matter how the control policy is optimized) without certain structures, e.g., more than one leg, a long-enough limb, or a 360-degree rotation joint. For RL, we claim that A good morphology should improve the agent's 's adaptiveness and versatility, i.e,, learning faster and making more progress in different environments. From prehistoric person to modern Homo sapiens, there is definite association between the rise of civilization and the Homo sapiens'

---

Under review as a conference paper at ICLR 2023
optimization of their physical form for improved tool use. Unfortunately, the morphology in many RL researches are pre-defined and fixed so it could be sub-optimal for the targeted environments/tasks and may restrain the potential of RL. Although morphology Wang et al., 2019b; Kurin et al., 2021; Yuan et al., 2022) to improve the final performance for a specifie task or environment, it not optimized for the adaptiveness to varying environments. Moreover, instead of re-initializing the control/RL policy after every morphology modification or environment change, the millions of years of evolution demonstrate the effectiveness of continual and progressive learning of the control policy over generations of morphology evolution along with sequence of varying environments. How to optimize the morphology and control policy of an RL agent for the above goal is still an underexplored problem.
be optimized using RL (Sims, 1994;
Fixed Enviranment
Fixed Environment
Siwestent
Sush ajc Saee schoer axpice
Inplectto
Syploraio
rorenss
Agents set
Agents set
TIZZA
(b) Main structure of Transform2Act
Avensie robiian
palns
Environment
Environment Eyg
Sylorste
(a) Main structure of ES-based methods
(c) Co-evolution in MECE
Figure I: MECE vs. previous methods. (a) Evolution strategy (ES)-based method for morphology optimiza- tion. It starts from agents with different morphology, eliminates those with poor performance, and then applies random mutation to the survivors. (b) Transform2Act, which trains an RL policy to modify the morphology of an agent in fixed environment. (c) MECE (ours). We train two policies optimize the morphology and change the environment to form curriculum over the course of training control policy.
Given that the agent has the freedom to improve its morphology and control policy for faster adaptation to different environments/tasks. the remaining question is: what environment can improve this learning process and incentivize the agent to keep finding better morphology (rather than staying with the same one)? The learning environment plays an essential role in RI. since it determines the data and feedback the agent can collect for its training. In this paper, we adopt simple criterion: A good environment should accelerate the evolution of the agent and help it find better morphology sooner, which will enhance the agent's 's adaptiveness and versatility over new environments/tasks.
The interplay between morphology and environment has lasted for billions of years. In the endless game, species which find the appropriate environments to evolve its morphology and policy may survive and keep being improved via adaptation to new environmentsstasks. Inspired by this, propose an htwphhhoogg--eviooonneeee curriculum of varying environments optimize the agent's morphology and its RL policy they can be generalized to unseen environments and adapted to new morphology, respectively. In MECE, we train two RL policies to modify the morphology and change the environment, which create sequence of learning scenarios to train the control policy continuously. This differs from previous work that select the morphology or the environment from a pre-defined set of candidates. Specifically, unlike zeroth- -odeerrrrdrrrrooo optimization approaches (Wang et al., 2019b; Hejna et al., 2021; Wang et al., 2018), morphology policy network can generate modifications based on all the morphology evaluated in diverse environments from history. On the other hand, an environment policy can produce sequence of progressively modified environments instead of randomly selected or sampled ones (Portelas et al., 2019; Wang et al., 201 2019a). In Fig. 1, illustrate the co-evolution process of MECE and compare it with evolution-strategy (ES) based method (Wang et al., 2019b) and Transform2Act (Yuan et al., 2022), which uses RL to optimize the morphology in fixed environment.
co-evolution (MECE) scheme that automatically generates a
Inspired by the former discussion of "good morphology" and "good environment", we optimize the morphology policy to improve the agent's 's adaptiiennesstvessallll
and optimize the environment policy to improve the morphology's ' evolution. To this end, we design two rewards for the two policies based on the learning progress of the control policy, which is estimated by comparing its concurrent
2

---

Under review as a conference paper at ICLR 2023
performance validation set and its historical trajectories. Therefore, the three policies complementary and mutually benefit each other: (1) the morphology and environment policies create curriculum to improve generalizability of the control policy; (2) the environment policy creates curriculum to optimize the morphology policy; (3) the control policy provides rewards to train the other two policies. In MECE, we train the control policy and automatically determine when apply the other two policies to modify the morphology or change the environment: they are triggered by the slow progress on the current morphology and environment, respectively. In experiments over different tasks, the morphology and control policy learned through MECE significantly outperforms the existing method in new environments. Moreover, MECE exhibits much faster-learning speed than baselines. We further provide an extensive ablation study to isolate each main component of MECE's contribution and compare it with other options.
RELATED WORK
Continuous
Design Optimization, A line of research in the community has studied optimizing an agent's continuous design parameters without modifying its skeletal structure, and they commonly optimize on specific kind of robots. Baykal & Alterovitz (2017) studies an annealing-based framework for cylindrical robots optimization. Ha et al. (2017); Desai et al. (2017) optimize the design of legged robots by trajectory optimization and implicit function. Applying deep RL into design optimization becomes more popular recently. Chen et al. (2020) uses computational graphs to model robot hardware as part of the policy. CMA-ES (Luck et al., 2019) optimizes robot design via learned value function. Ha et al. (2017) proposes population-based policy gradient method. Another line of work (Yu et al., 2019; Exarchos et al., 2021; Jiang et al., 2021) search for the optimal robot parameters to fit an incoming domain by RL. Different from the prior works, our approach learns to find the most general skeleton of an agent while maintaining its continuous design parameters optimization to diverse environments tasks.
Combinatorial morphology Optimization. One closely related line of work is the design of modular robot morphology spaces and developing algorithms for co-optimizing morphology and control (Sims, 1994) within a design space to find task-optimized combinations of controller and robot morphology. When the control complexity is low, evolutionary strategies have been successfully applied to find diverse morphologies in expressive soft robot design space (Cheney et al., 2013; 2018). For more expressive design spaces, GNNs have been leveraged to share controller parameters (Wang et al., 2019b) across generations or develop novel heuristic search methods for efficient exploration of the design space (Zhao et al., 2020). In contrast to task specific morphology optimization, Hejna et al. (2021) propose evolving morphologies without any task or reward specification. Hejna et al. (2021) employ an information thhoreeti objective to evolve task-agnostic agent designs.
Generalization to environments. Several recent works (Wang et al., 2019a; Portelas et al., 2019; Gur et al., 2021) show that the efficiency and generalization of RL can be improved by training the policy in different environments. By sampling the parameters of environmental features, they Portelas ct al. (2019); Wang ct al. (2019a) teach the RL agent using a curriculum of various enviromments. To estimate the distribution of environments, however, requires evaluating the RL policy on numerous sampled environments, which might be expensive and inaccurate for intricate ones. The modified environments may also be unfeasible too difficult for RL exploration. In MECE, train an environment policy to change the environments adaptive to the evolved agent's learning progress, and it is possible to regulate the level of difficulty of the modified environments.
3 FORMULATIONS OF THE THREE POLICIES IN MECE
In MECE, we have three RL policies, control policy æ that learns to control the agents of evolved morphology, a morphology policy T. that learns to modify the agent's morphology for better robust- ness in diverse environments, and an environment policy ne that learns to change the environment to boost the morphology evolution. During the training phase, T. and He are alternately applied to evolve the agent's morphology and the training environment, respectively. Taking into account that might require various environment steps for training in distinct morphologies or environments, we propose dynamic time-scaling for u. and n. that is adaptive to m learning process.
3

---

Under review as a conference paper at ICLR 2023
Agent control. The problem of controlling an agent can be modeled Markov decision process (MDP), denoted by the tuple A, p.r.p) where and A represent the set of state space and action space respectively, p means the transition model between states, and r is the reward function. An agent of morphology m, E M in state & E S at time / takes action a, E A and the environment returns the agent's new state 81+1 according to the unknown transition function p(*++1 |st 0t, m,) along with associated reward Tt = r(st, Qt). The goal is to learn the control policy π' : + A mapping states actions that maximizes the expected return R[R. which takes into account reward
at future time-steps [x) [[R,] E
with discount factor ?y [0, 1], where H is E 7'r
the variable time horizon.
Morphology evolution. We model the morphology evolution as an MDP problem, denoted by the tuple {Sm, Pm, Tm: Tm. p}. Sn represents the set of state space, and a state sat E Sm at time- step ad is defined as Sat Goz. where Got is the skeleton structure of agent. The action b sampled from the set of action space B can modify the topology of the agent, that has three choices in { [AddJ oint, Del.J NoChange). The transition dynamics Pm (s ( bat) reflects the changes in state and action. We define the reward function for n. as the average improvement of training ग with the current evolved morphology in different environments. The reward I' at time-step at is denoted as
(R(H, Gat) - R(H, 0E, ,att-)) - X| OEEE
Σ
(1)
at
E
note that the first term in Eq. indicates the average performance of the current morphology in diverse environments, and the second term is a cost for each action taken by nm that modifies the morphology of the agent. E is the set of environments used for evaluation.
Environment evolution. We model environment evolution as an MDP problem, denoted by the tuple Pe, Tes Pes p}. The state si. BG 0 in the set of state space Se includes the agent's topology G (Vze. Est) and the parameter of environment 0g E θg. The transition dynamics Pe(# | Gst) reflects the changes in the state by taking an action at time-step Bt. We define the reward for n, as the learning progress of multiple morphologies in the current environment. Thus, training Te will encourage Tm to optimize the morphology of better generalization sooner. The reward Te at time-step St is denoted as
Bt = Par - P where
Pss = RRH, 0 - O,,, 0t-1: GBt)-
(2)
in Eq. 2, PA is the learning progress of the RL agent in an environment defined on R(H, G3t). which denotes the expected return of the RL agent Gs: of H time-steps evaluated in environment OF.
Short evaluation window. Since I'm and re are respectively based on the training process of on different morphologies or environments, it is cost but necessary to rollout " periodically to collect data. However, Hejna al. (2021) demonstrates that evaluating with short horizon is enough to provide informative feedback to reflect the training process of the current policy. In light of this, we run short evaluation window for after every period of environment steps taken by the agent. In each evaluation window, we rollout multiple morphologies of the best performance by π in several most recent envirouments, and then we can calculate I'm and re based on the evaluation results, More details refer to Appendix. C.
In this paper, we use a standard policy gradient method, PPO (Schulman al., 2017), to optimize these three policies.
4 ALGORITHM OF MECE
In MECE, we need jointly train three policies: control policy и learns to complete tasks, morphology policy Tm learns to evolve the agent's morphology to be more adaptive to diverse environments, and environment policy тe learns to modify the training environments more challenging. At first glance, training a single RL policies may appear more difficult than training a single RL policy, as it needs the collection of additional data via interaction. However, MECE enables three policies to help in each other's 's training through the co-evolving mechanism, where each policy is trained on a curriculum of easy-to-hane tasks utilizing dense feedback from the training experience of other policies. By
4

---

Under review as a conference paper at ICLR 2023
iterating this co-evolution process, MECE considerably improves the training efficiency of each policy, resulting in an RL agent of morphology that more generalizable across environments.
In each episode, the control policy starts from training on an initial agent in a randomly sampled environment (refer Line.3-9 in Alg. 1). This initial agent's s skeleton structure relatively simple and is easy to learn according to its naive mor- phology. Even though the naive agent is not generalized due to its constrained morphol-
Algorithm 1 MECE
Initialize: control policy x, morphology policy Tms envi-
romment policy Te, dataset 'D t- 0, initial agent morphol- ogy initial environment parameter 0g:
2: while Not reaching max iterations do
for = 0, 1, Taaax do
& RL exploration
4:
Sample control action ae ~ w;
5:
Sc41 + - az); environment reward:
ogy, can initially provide dense feedback
6:
for training rm and Te- On the other hand,
7:
m4+1 mc, 0 0F;
beginning from an agent with a random 8:
Store (st, S141, me, 0f into D;
morphology is theoretically feasible but in- efficient when initializing a complicated morphology that is difficult to learn to con- trol and yields uninformative feedback for training the other policies.
9:
end for
10:
Update with PPO using samples in D;
1I:
(me+1, 0f+1) = CO-EVO(m, n. 0,
12:
end while
We assume that the control policy is now mature the current morphology after training. Then MECE proceeds to se- ries of co-evolution phases, where a phase is associated with achieving robustness across the evolved morphologies and en- vironments (refer to Line. 11 in Alg. 1). In each phase, as shown Line.3-4 Alg. 2, we first evaluate the control policy to col- lect rewards for training Tm and Te. Note that this step is not cost, the RL agent ex- ecutes environment steps with a short hori-
Algorithm 2 CO-EVO
Input: Current morphology m.t. control policy train-
ing environment &5. dataset D.
2: Procedure CO-EVO(maa: n, O.00..:
3: Evaluate with morphology Mat in 8
4: Calculate reward 7' and r. following Eq. and Eq. p Modify the agent's morphology 6: Apply Tm to modify mat to mo++1;
if rm S 6, then
7:
Update with PPO using samples in D;
8:
else
if &e then & Modify the training environment
10:
Apply ne to modity 05 to 8 Update Te with PPO using samples in D;
zon. After that, alternately apply 7m and Te to co-evolve the agent's 's morphol-
11:
12:
end if
ogy and the training environments based 13:
end if
14: Return (mat+1, 0g(+1)
two criteria of I'm and re (refer to Line.5
and Alg. 2).
Reward criteria. As mentioned in section 3, I' and indicate the learning progress of the current morphology in the training environment respectively. In particular, a nominal value of r' corresponds to the adaptability of the current morphology to different environments and indicates that training with the current morphology cannot significantly increase л performance. In this instance, we should apply Tm to optimize the morphology to increase its generalization to environments. Similarly, a minor re indicates that modifying the agent's 's morphology will have a negligible effect on its performance in the current environment, and the environment should be modified to be more challenging to boost the morphology evolution. As a result, we propose two independent criteria for Tm and Te, which allows MECE to adapt to the learning progress of morphology and environment and apply Tm or Te to produce corresponding evolution and alterations.
We now have an agent of newly evolved morphology forward to the next iteration of training the control policy æ on them. Using T'm and re, MECE has achieved the alternating evolution of morphology and environment. MECE not only improves the robustness of the morphology to various environment, but also improves the learning efficiency of the policies through the use of dense feedback.
modified training environment, and then
5 EXPERIMENTS
We design experiments answer the following questions: (1) Given dynamic environments, does our method outperform previous methods in terms of convergence speed and final performance? (2) Does our method create agents of better generalization to diverse environments?
5

---

Under review as a conference paper at ICLR 2023
5.1 ENVIRONMENTS SETUP
Diverse environments have vary features that require the agent to evolve out different morphologies, e.g., bipedal agent has too short legs to go across high obstacle, agent of four legs navigate on rough road more smoothly than one of one/two legs. In our experiments, we conduct three environments that requires various morphologies environment parameters to denote the possible physical features of environments.
complete the tasks. We use a tuple 6 of
In experiments, we have three types of environments: (1) 2d locomotion: In the 2D locomotion environment, agents are limited to movement in the X-Z plane. The state Sy is given by the final (x. positions of the morphology joints. We evaluate morphologies on three tasks: running forwards, running backwards, and jumping. Environment policy takes actions to change the roughness of terrains, that is controlled by 0². (2) 3d locomotion: where a 3D agent's goal to move as fast as possible along x-axis and is rewarded by its forward speed along x-axis. Environments have fixed number of obstacles and terrains of different roughness which are controlled by 8 Tee can not only change the roughness of terrains in the environments, also learns to modify the average distance between obstacles. (3) Gap crosser: Gap Crosser, where a 2D agent living in an =2 -plaee needs to cross periodic gaps and is rewarded by its forward speed. The width of the gap is controlled by Te. In order to avoid unlearnable environments, the upper and lower limits of the gap width are limited. More details refer Appendix. A.
Baselines. We compare our method with the following baselines that also optimize both the skeletal structure and joint attributes of an agent. (1)Neural Graph Evolution (NGE) et al., 2019b), which is an ES-based method that uses GNNs to enable weight sharing between an agent and its offspring. (2) Transform2Act (Yuan et al., 2022), that optimizes an RL policy to control the evolution of agents. Note that both baselines do not have diverse environments in the original code. For fair comparison, we apply an environment set randomly sampled from a uniform distribution in the training and test phase for them.
5.2 MAIN RESULTS
In Fig. (ca)-(c), we report the MECE and all baseline method performance on test environments for three environments. In terms of the learning efficiency and final performance, MECE outperforms baseline methods in all environments. The results show that while Transform2Act's final performance can be improved by periodically changing the training environments, MECE's learning efficiency still notably higher than that of Transform2Act. The best morphologies found by each approach are listed in Fig. 2(d)-(f) for easier comparison. For 2d locomotion MECE develops a morphology that resembles a crawling animal that can move fast over terrain. In 3D locomotion, MECE has developed spider-like framework. MECE evolves a more streamlined structure in comparison to Transform2Act so that it can more possibly avoid obstacles. Finally, the Hopper- like agents developed by Transform2Act and MECE are comparable in the Gap crosser that can jump across gaps. Overall, MECE-optimized morphologies have superior structural symmetry and are more consistent with the characteristics of biological evolution in various environments.
5.3 ABLATION STUDY AND ANALYSIS
We product out series of ablation studies prove the effectiveness of each part in MECE, We list the introduction information of each ablation study in Tab. 1, and report the results in Fig. 3. Note that in each ablation study, we only change one part of MECE and maintain the other parts the same as the original MECE, and all results are the test performance evaluated on the same environment set in Fig. 2 (b).
Ablation Study I. In this ablation study, we focus on the effectiveness of Te for the training and report the results Fig. 2(a). Note that the initial environment is relatively simple, while the final environment that evolved by nm is more challenging. The results show that the evolved morphology and control policy trained in the diverse environments are more general than trained in the fixed environment, matter the environment is simple not. On the other hand, compare the performance of MECE with MECE (periodic envs), we can find that n, helps the training in terms the efficiency and the final performance. This is because training in the environment modified by ne, compared to

---

Under review as a conference paper at ICLR 2023
10020
www www.
4200
4000
w/ Mateemm
4000
2000
2000
2000
20
50
10
20
3D
50
0
10
20
30
50
Simdlel on Stop (Niilio))
Step (hillion)
Simular on (Millon))
(a) 2d-locomation
(b) 3d-locomation
Gap-crosser
Initial Design
Initial Design
Initial Design
Transforn2Act
Transform2Act
Transfonn2Act
MECE
MECE
(d) Evolved Morphology (2d)
(c) Evolved Morphology
Evolved Morphology (Gap)
Figure 2: Baselines comparison and optimized morphology. In Fig.(a)-(c), we report the evaluation results of MECE and baseline methods in three environments, and we plot the accumulated rewards (mean std averaged over 6 random seeds) against the number of simulation steps for all methods. For fair comparison, periodically changing environments that randomly sampled from a fixed distribution in training of baseline methods. MECE has better learning efficiency and final performance than all baseline methods. In Fig.(d)-(0 we list the optimal morphology that evolved by each method. Intuitively, the structural symmetry and environmental adaptability of the MBCCF-ppiimize morphology is better. Especially in 3d locomotion, the agent developed by MECE is better navigating terrain and avoiding obstacles.
add
Ablation Study
Defniiton of Legends
original: Periodic environments generated by T.-
Ablation study-I for w..
periodic envs-random: Periodic environments of randomly sampled.
Fig. 3(a)
fixed envs-initial: Initial and easy eavironment.
fixed envs-final: Fixed environment generated by Te-
original: Dynamic morphology evolved by mm- random morph: Randomly mutate the morphology.
Ablation study-Il for mn.
Fig. 3(b)
fixed morph-initial: Fixed agent of the initial morphology.
fixed morph-final: Fixed agent of the final morphology evolved by Am-
Ablation study-Ill for dynamic update window. Fig. 3(c)
Original: Dynamic evaluation window.
fixed update window:
Update ne n. at fixed frequency.
original: Follow the reward setting of nm (T₂) in Eq. I (Eq. 2).
Ablation study-IV for T.m and re. Fig. 3(d)
reward-i: Remove the reward for mm.
reward-ii: change Twn to Eq. reward-iil: change r. to Eq. 2.
Table List of ablation studies. This includes definitions for each set controls as well objectives of ablation study I-IV. Since we only validated one design of MECE in each ablation study, only one design from each control group was different from MECE (original).

---

Under review as a conference paper at ICLR 2023
3993
5000
5220
5000
2000
2000
2230
2000
1930
1930
00
20 (1 roorph
20 Clan Ablision
indeen Stop
Figure 3: Ablation study. We design four ablation studies to illustrate the advantages of each part in MECE. The results of ablation study and II show the effectiveness of Tee and Tm on the learning efficiency and generalization. Ablation study III proves that applying ne and 'mm adaptive to the training process of the control policy can improve its robustness. In ablation study IV, we try different setting of the reward function of лe and Tm, and the results prove that the original reward setting optimal. More details of each ablation study can be found in Tab. 1.
the randomly generated one, avoids the learning gap between two extremely different environments, and is smoother for and fm learn. On the other hand, Te learns to generate environment to accelerate the learning progress of т and Tm. which is shown clearly in the learning curves of 10 20 million simulation steps in Fig. (a)).
Ablation Study II. The purpose of this ablation study is to demonstrate how Tm can produce morphology that more adaptable to diverse environments and increase learning effectiveness. The results are shown in Fig. 3(b). When comparing the learning curves for MECE (random morph) and MECE (original), the former has a far higher early learning efficiency than the latter. This is due to ne remaining constant and the environment not changing significantly, which prevents the morphology from being adequately adapted to a variety of habitats, even some comparable environments. Theoretically, the "fixed morph-final" morphology should be able to be more adaptive to environments and, with training, reach competitive performance. Although it performs better than "random morph" and "fixed morph-initial. the final morphology is more complicated and therefore directly teaching its control policy to adapt to various environments would be more challenging.
(orignna)))))l))" is still far behind. This is because that
Ablation Study III. Through a co-evolutionary process of adaptive criterion on T' and Fe. Ti and Tee learn to evolve the morphology and environment. To determine whether this design is effective, we perform an ablation experiment. The results are shown in Fig 3(c). For "MECE (fixed evaluation window)", every fixed number of environment steps, Ti and me take actions to change the agent's morphology and environment. The results indicate that by removing the dynamic window, MECE's learning effectiveness has drastically decreased. It is challenging to coordinate the training developments of the three policies, particularly the control policy, without a dynamic evaluation window. Only a somewhat steady control policy, as was already mentioned, can give the other two reliable feedback for training. Moreover, the performance of Transform2Act is taken into account for easy comparing. Even though "MECE((ixed evaluation window)" is less effective, a competitive final performance to Transform2Act is still feasible.
Ablation Study IV. We designed this ablation investigation to verify the reward of Th and m. in MECE, and the findings are displayed in Fig. 3(d). Note that just one incentive is altered in each scenario in this experiment; all other rewards remain unchanged. We explore two scenarios for Tm: the first involves removing the reward ("MECE (reward-i)")), same to the setting of transform2Act), and the second involves substituting the formal of re ("MECE (reward-ii)"), i.e., accelerating learning progress. We take into account employing learning progress (in the form of rm, "MECE (reward-iil)") for Te. The results show that the original design has obvious advantages in both learning efficiency and final performance. For the issues addressed by MECE, 7. can be thought of as the meta learner of ग, whereas Te is the meta learner of the latte. In order to help perform better on various tasks, Tm learns to modify the agent's 's morphology during training, and Te speeds up both of their learning processes. Therefore, in both theoretical and practical tests, the original design is the most logical.
5.4
CASE STUDY: CO-EVOLUTION BETWEEN MORPHIOLOGY AND ENVIRONMENT
To further comprehend the co-evolution of morphology and environment in MECE we perform a case study in 2d locomotion and present results in Fig. 4. The y-axis of the point plot indicates the ratio of the training environment's 's roughness to the current morphology' 's height. Fig. 4(a)-(f) are schematic descriptions of the current morphology and the training environment corresponding it.
8

---

Under review as a conference paper at ICLR 2023
MECE
MECE (random envs)
1,2
[b)
0.4
Simulation Step (Millon))
Figure 4: Case study results. This figure more vividly illustrates a schematic diagram of the co co-evolution of morphology and environment during training. The ordinate in the illustration shows the proportion of the environment's ' roughness to the agent's height. Since the deviation is minimal and the roughness of the training environment produced by MECE is about similar the agent's height (the ratio from pointplot that w, can not only guarantee challenging environments but also more stable than randomly sampled. Fig.(a)-(1 compare the effectiveness of т. that correspond to changes in the training environments (blue indicates original MECE, and orange indicates random environments). Contrarily. it is evident that no is likely to result in exceptionally difficult environments (d), while the environment produced by challenging leamablee (c)..
to
be observed
Generally, an effective training environment should meet two conditions simultaneously. The first is learnable, and the RL agent collect training data via exploration. The second factor is the level of difficulty; environments that are either too easy or too challenging may reduce the learning efficiency of the RI. agent. Therefore, in 2D) locomotion, reasonable environment ment's roughness is quite large (challenging), but not so large as to render the agent unconquerable (learnable) during exploration. Using the results of the pointplot, the environment policy can ensure that the ratio of environmental unevenness during the training process to the height of the current agent is approximately 1, which satisfies the training environment criteria. contrast, despite the fact that the randomly generated environment can ensure the ideal in some phases, it extremely unstable (the standard deviation visibly high), which would undoubtedly lower the learning effectiveness.
in which the environ-
Specifically, comparing Fig. 4(a) and (c), the environment generated by we in the early stage of training unlearnable for the current agent, whereas the middle stage of training, the training environment is relatively flat, but challenging, especially in terms of the number of occurrences of particular feature. The roughness of the environment on the right fluctuates frequently. Note again compared to Fig. 4(d) that the right side of the randomly produced environment is unlearnable for the current agent in the middle of training. This phenomena does not occur MECE because Te modifies the environment to be morphology-adaptiv formed by n. 's slope exhibits noticeable but relatively subtle alterations. This due to the fact that the existing morphology has high degree of adaptation varied contexts, and environments with more extreme alterations have less effect on the morphology's optimization. To better train the control policy, an environment of moderate difficulty should be adopted.
Comparing Figs. 4(e) and (f), the environment
CONCLUSION
In this paper, we offer framework for morphology-envie eevirommmennii co-evolution that evolves the agent's morphology and the training environment in alternation. Compared to previous morphology optimization approaches, ours is more sample-efficient and learns to develop morphologies that are more robust across diverse enviromments. Experiments demonstrate that our approach outperforms baseline methods in terms of convergence speed and overall performance. In the future, interested using curriculum RL approaches to further improve the learning efficiency our approach that it adapt
challenging environments and tasks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Cenk Baykal and Ron Alterovitz. Asymptotically optimal design of piecewise cylindrical robots using motion planning. In Robotics: Science and Systems, 2017.
Tianjian Chen, Zhanpeng He, and Matei Ciocarlie, Hardware as policy: Mechanical and computa- tional co-optimization using deep reinforcement learning. In CoRI., 2020.
Nick Cheney, Robert MacCurdy, Jeff Clune, and Hod Lipson. Unshackling evolution: evolving soft robots with multiple materials and a powerful generative encoding. In GECCO '13, 2013.
Nick Cheney, Josh C. Bongard, Vytas SunSpiral, and Hod Lipson. Scalable co-optimization of morphology and control embodied machines. Journal of The Royal Society Interface, 15, 2018.
Ruta Desai, Ye Yuan, and Stelian Coros. Computational abstractions for interactive design of robotic devices. 2017 IEEE International Conference on Robotics and Automation (ICRA), pp. 1 1196-1203, 2017.
Ioannis Exarchos, Yifeng Jiang, Wenhao Yu, and C. Karen Liu. Policy transfer via kinematic domain randomization and adaptation. 2021 IEEE International Conference Robotics and Automation (ICRA), pp. 45-51, 2021.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. ArXiv, 0bs/1903.02422 2019.
Chelsea Finn, Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. ArXiv, abs/1703.0340M 2017.
Izzeddin Gur, Natasha Jaques, Kevin Malta, Manoj Tiwari, Honglak Lee, and Aleksandra Faust. Adversarial environment generation for learning to navigate the web. ArXiv, abs/2103.01991, 2021.
Schoon Ha, Stelian Coros, Alexander Alspach, Joohyung Kim, and Katsu Yamane. Joint optimization of robot design and motion parameters using the implicit function theorem. Robotics: Science and Systems XIII, 2017.
Donald Hejna, Abbeel, and Lerrel Pinto. Task-agnostic morphology evolution. ArXiv, abs/2102.13100, 2021.
Max Jaderberg. Valentin Dalibard, Simon Osindero, Wojeiech M. Czamecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukcuoglu. Population based training of neural networks. ArXiv, 101///111.9966, 2017.
Yifeng Jiang, Tingnan Zhang, Daniel Ho, Yunfei Bai, C. Karen Liu, Sergey Levine, and Jie Tan. Simgan: Hybrid simulator identification for domain adaptation via adversarial reinforcement learning. 2021 IEEE International Conference on Robotics and Automation (ICRA), pp. 2884- 2890, 2021.
Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. My body is a cage: the role of morphology in graph-based incompatible control. ArXiv, abs/2010.01856.
2021.
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. CoRR, abs/1509.02971. 2016.
Kevin Sebastian Luck, Heni Ben Amor, and Roberto Calandra. Data-efficient co-adaptation of morphology and behaviour with deep reinforcement learning. In CoRL,, 2019.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A. Rusu, Joel Veness, Marc G. Belle- mare, Alex Graves, Martin A. Riedmiller, Andreas Fidjeland, Georg Ostrovski, Stig Petersen, Charlie Beattie, Amir Sadik, Ioannis Antonoglou, Helen King, Dharshan Kumaran, Daan Wierstra, Shane Legg, and Demis Hassabis. Human-level control through deep reinforcement learning. Nature, 518:529-533, 2015.
10

---

Under review as a conference paper at ICLR 2023
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe. Weisfeiler and leman go neural: Higher-order graph neural networks. ArXiv, abs/1810.022444 2019.
Sanmit Narvekar, Bei Peng, Matteo Leonetti, Jivko Sinapov, Matthew E. Taylor, and Peter Stone. Curriculum learning for reinforcement learning domains: A framework and survey. J. Mach. Learn. Res., 21:181:1-181:500 2020.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach Vetito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, tegh-performanc deep learning library. In NeurIPS, 2019.
Rémy Portelas, Cédric Colas, Katja Hofmann, and Pierre- Yves Oudeyer. Teacher algorithms for curriculum learning of deep rl in continuously parameterized environments. CoRL, 2019.
Tim Salimans, Jonathan Ho, Xi Chen, and Ilya Sutskever. Evolution strategies as a scalable alternative to reinforcement learning. ArXiv, abs/1703.03864, 2017.
John Schulman, Philipp Moritz, Sergey Levine, Michael I. Jordan, and P. Abbeel. High-dimensional continuous control using generalized advantage estimation. CoRR, abs/ 0tbs/1506.02/ 006602248, 2016.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. ArXiv, abs/1 00011777006444 211.00637,,,111
Karl Sims. Evolving 3d morphology and behavior by competition. Artificial Life, 1:353-372, 1994.
Rui Wang. Joel Lehman, Jeff Clune, and Kenneth O. Stanley, Paired open-ended trailblazer (poet): Endlessly generating increasingly complex and diverse learning environments and their solutions. ArXiv, abs/1901.01755. 2019a.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In ICLR, 2018.
Tingwu Wang. Yuhao Zhou, Sanja Fidler, and Jimmy Ba. Neural graph evolution: Towards efficient automatic robot design. ArXiv,
2019b.
Wenhao Yu, C. Karen Liu, and Greg Turk. Policy transfer with strategy optimization. ArXiv, abs/1810.05751, 2019.
Ye Yuan, Yuda Song, Zhengyi Luo, Wen Sun, and Kris M. Kitani. Transform2act: Learning a cunsform-and-commen policy for efficient agent design. ArXiv, abs/2110.03659, 2022.
Allan Zhao, Jie Xu, Mina Konakovic-Lukovic, Josie Hughes, Andrew Spielberg, Daniela Rus, and Wojeiech Matusik. Robogrammar: graph grammar for terrain cerrain-pptimized robot design. ACM Trans. Graph., 39: 18:81881::16,,,61111111111111111 2020.

---

Under review as a conference paper at ICLR 2023
A ENVIRONMENT DETAILS
In this part, share further information about the three experiment environments. A simple skeleton structure is used to initialize the agent. Each joint of the agent is connected to its parent joint via a hinge. The environment state of each joint contains its joint angle and velocity. In addition to the height, phase, and global velocity, we also include additional information for the root joint, such as the phase and height. Zero padding is employed ensure that the length of each joint state same. The attribute characteristic of each joint includes the bone vector, bone size, and motor gear value. Using loosely-define attribute range, each attribute normalized within the interval (-1, 1]
2d locomotion The agent in this environment lives inside an 2:2-plane with a terrain ground. Each joint of the agent allowed to have maximum of three child joints. For the root joint, we add its height and 2D world velocity to the environment state. The enviornment reward function is defined as:
= 1p - P /ôt + 1,
(3)
where p denotes the x x-position of the agent and ōt = 0.008 is the time step. An alive bonus of is also used inside the reward. The episode is terminated when the root height is below 1.4.
3d locomotion The agent in this environment lives freely in a 3D world with an uneven ground. There are fixed number of obstacles randomly scattered on the surface. Each joint of the agent is allowed to have a maximum of three child joints except for the root. For the root joint, we add its height and 3D world velocity to the environment state. The reward function is defined
1 J
Ft Pi+1 1/1/5t w
Σ WEV.
||a,,el?
(4)
where w 0.0001 is weighting factor for the control penalty term. denotes the each node of the skeleton structure V22 of the agent at time-step t. J is the total number of agent's ' joints, and the time step St 0.04.
Gap crosser The agent in this environment lives inside an zzzplaaee The terrain of this environment includes periodic gap. The height of the initial terrain is at 0.2. The agent must cross these gaps to move forward. Each joint of the agent is allowed to have a maximum of three child joints. For the root joint, add its height, 2D world velocity, and a phase variable encoding the periodic z-position of the agent to the environment state. The reward function is defined as:
F p 1/1/5t + 0.1,
(5)
where the time step öt = 0.08. An alive bonus of 0.1 used inside the reward. The episode is terminated with the root height is below 1.5.
Implementation of morphology poicy 7. MuloCo agents are specified using XML strings, during the morphology modify phase, we represent each agent's 'ss skeleton structure as an XML string and modify its content based on the morphology actions. At the start of the RL exploration stage, the modified XML string is used to reset the MuJoCo simulator and load the agent of modii-ed morphology.
Implementation of environment policy Environment policy modify the training environment by taking actions to change the environment parameters. For locomotion environments, we sample terrain height maps using random Gaussian mixtures. There is environment tuple of two parameters to control the generation of terrains. The first environment parameter is the maximum height of the terrain, which is limited 2.4(2.7) in 2d (3d) locomotion. The second environment parameter is to control the variance of the sampled environments, which is restricted in [2.4, 7.2] for 2d, and [2.7,5.4 for 3d. In 3d locomotion, we have more environment parameter in the tuple to control the averaging spacing between the obstacles, which is restricted in [1.6, 4.4].
12

---

Under review as a conference paper at ICLR 2023
B ADDITIONAL EXPERIMENTS RESULTS
To respond the request of reviewer QUPT, we report the changing environments and agent's morphol- ogy during the training Fig. 5. The results show that when the evolved morphology is relatively complex, the training environment focuses more on complex cases than the randomly sampled training environments with environment generator. On the other hand, the prevalence of easy environ- ments significantly decreases as morphology evolves (indicating a higher ability to move in different environments).
2d-locomotion
3d-locomotion
Gap-crosser
Figure 5: This figure shows the changing of the agent's morphology and environments in the training phase. The results illustrate that with the environment generator (#.), the training environment focuses complex cases (higher roughness). The presence agents with more complex morpholegies more noticeable among them. Thus, the environment generator creates the training environment based on how the morphology changes during training.
C SHORT EVALUATION WINDOW (SEW) IN MECE
In Alg. 2, we modify the agent's morphology and the training environment based on the control policy's 's learning progress. In light of the previous method (Hejna et al., 2021), we construct short evaluation window (SEW) on collect data for computing Tm and re- In accordance with Sec. 3, we should evaluate the present morphology many settings in the current training environment for in Eq. 2. In this short evaluation phase, therefore maintain some of the most recent morphologies and training environments. We attempted to save several combinations of morphology and number of contexts for this purpose. All experimental and control data are provided under the assumption that the optimal combination exists. We list all tried combination in Tab. 2.
Tm in Eq. and several morphologies
D HYPERPARAMETERS AND TRAINING PROCEDURES
In this section, present the hyperparameters searched and used for MECE in Tab. and the hyperparameters for baseline in Tab. and Tab. 4. All models are implements by PyTorch (Paszke al., 2019). For control policy and morphology policy, & Lenssen, 2019) and GraphConv (Moris et al., 2019) as the GNN layer. When training the policy with PPO, we adopt generalized advantage estimation (GAE) (Schulman et al., 2016). For the baselines, we employ the released code of NGE and Transform2Act to build our version.
the PyTorch Geometric package (Fey
For fair comparison, we employ the same GNN architecture for MECE and baselines. In addition, we ensure that the number of simulation steps used for optimization is the same for both the baseline methods and ours. MECE and Transform2Act optimizes the policy with batch size 50000 for 1000 epochs, totaling 50 million simulation steps. NGE employs population of 20 agents, and each agent trained with a batch size of 20,000 for 125 generations, totaling 50 million simulation steps.
13

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Values Searched Selected
Policy GNN Layer Type
GraphConv
Number of morphologies in SEW
2, 2.3,4
Number of environments in SEW
2,3, 4,5
GNN Size (Tm)
(32, 32), (64, 64, 64), (128, 128, 128), (256, 256, 256) (32,32, (64, 64, 64), (128, 128, 128), (256, 256, 256)
GNN Size (T)
Hidden layers for тe network
2,3, 4
Hidden units for ne network
200, 300, 400
Policy Learning Rate (x)
5e 5, le 4,30 - -4
Policy Learning Rate (7m)
5, lc
4, 3e
Policy Learning Rate (T₂)
3c 4, 5c
4,
1c 4
Value GNN Layer Type
GraphConv Tanh
Value Activation Function
Value GNN Size
(64, 64, 64), (128, 128, 128), (256, 256, 256) (256, 256), (512, 256), (256, 256, 256)
Value MLP Size
Value Learning Rate
le 4,3e 0.2
PPO clip Pize
PPO Batch Size
10000, 20000, 50000
PPO Minibatch Size
512, 2048
Num. of PPO Iterations Per Batch
1,5, 10
Num. of Training Epochs
1000
Discount factor ? GAE
0.95
0.99, 0.995, 0.997, 0.999
Table Hyperparameters searched and used by MECE, The bold numbers among multiple values are the final selected ones.
Hyperparameter
Values Searched & Selected
Num. of Skeleton Transforms N₂
3, 5, 10 1,3,5 5
Num. of Attribute Transforms N,
Policy GNN Layer Type
GraphConv Tanh
JSMLP Activation Function
GNN Size (Skeleton Transform)
(64, 64, 64). (128, 128, 128), (256, 256, 256)
ISMLP Size (Skeleton Transform)
(256, 256), (512, 256), (256, 256, 256)
GNN Size (Attribute Transform)
(64, 64, 64), (128, 128, 128), (256, 256, 256)
JSMLP Size (Attribute Transform)
(256, (512, 256), (256, 256, 256) (64, 64, 64), (128, 128, 128), (256, 256, 256)
GNN Size (Execution)
JSMLP Size (Execution)
(128, 128), (256, 256), (512, 256), (256, 256, 256)
Diagonal Values of Σ
1.0, 0.04, 0.01
Diagonal Values of E
1.0. 0.04,.44. 0.01
Policy Learning Rate
5e -
Value GNN Layer Type
GraphConv Tanh
Value Activation Function
Value GNN Size
(64, 64, 64), (128, 128, 128). (256, 256, 256) (256, 256), (512, 256), (256, 256, 256)
Value MLP Size
Value Learning Rate
1c 4. 3c 4
PPO clip e Pize
0.2
PPO Batch Size
50000
PPO Minibatch Size
2048
Num. of PPO Iterations Per Batch
1, 10
Num. of Training Epochs Discount factor ?
1000
0.95
GAE A
0.995
Table 3: Hyperparameters searched and used by Transform2Act in our version. The bold numbers among multiple values the final selected ones.
14

---

Under review as a conference paper at ICLR 2023
Hyperparameter
Values Searched & Selected
Num. of Generations
125 10, 20, 50, 100 0.15, 0.2, 0.3, 0.4 GraphConv Tanh
Agent Population Size
Elimination Rate
GNN Layer Type
MLP Activation
Policy GNN Size
(32, 32, 32), (64, 64, 64), (128, 128, 128) (128, 128), (256, 256), (512, 256)
Policy MLP Size
Policy Log Standard Deviation
0.0.-1.6
Policy Learning Rate
5e 5
Value GNN Size
(32, 32), (64, 64, 64), (128, 128, 128) (128, 128), (256, 256), (512, 256)
Value MI.P Size
Value Learning Rate
3e 4 4
PPO clip e
0.2
PPO Batch Size
20000, 50000
PPO Minibatch Size
2048
Num. of PPO Iterations Per Batch
10
Discount factor 7
0.99, 0.995
GAE À
0.95
Table 4: Hyperparameters searched and used by NGE in our version. The bold numbers among multiple values are the final selected ones.
15


Reviews:
[{"review_id": "mggcjPNg79", "reviewer": "Reviewer_ugJV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose to use three different policies to modify simultaneously the actions, morphology, and environment in a Reinforcement Learning problem. \nWhile the control policy is very standard in RL applications, this paper also considers morphology and environment modifications as Markov Decision Processes. The state of the morphology considers some aspects of the robot's skeleton, while the state of the environment is some aspects such as roughness or dip length. The action spaces correspond to modifications of the state (of either the morphology or the environment). Finally, the reward functions are designed to encourage the agent to improve its learning progress and adaptability to varying environments. \nThe proposed algorithm is evaluated in 3 different tasks (2d-locomotion, 3d-locomotion, and gap-crosser). It is compared to two baselines (T2A and NGE. Finally, a series of ablation studies are presented to explain the inner workings of the proposed algorithms. In all the presented results, the algorithm outperforms the baselines.", "summary_of_the_review": "Overall, the paper presents an interesting concept. Unfortunately, the selected baselines do not provide a fair comparison to evaluate the benefits of using MDP and policies for morphology and environment co-evolution.  ", "strength_and_weaknesses": "I find the concept of considering morphology and environment optimisation as MDP problems a very very interesting concept. I really like it and find this very interesting. \nThis is for me the main strength of the paper. \n\nUnfortunately, the propsoed experiments are not enough to gauge the benefits of the proposed method. All the consdered baselines use fixed environment (as depicted in Fig1) without any possible curriculum learning. The introduction of the paper stresses clearly that co-evolving the morphology and the environment is essential. Yet, none of the considered baselines does this, while the paper cites several papers that co-evolve morphology and environment to create such curriculum learning. Typically, the proposed methods should be compared to POET[1] and Enhanced POET[2], which are both designed with the same objective and motivation than the proposed algorithms. Only with such comparison, we will be able to know if training policies to modify environments and morphology is a promising alternative. \n\n[1]Wang, R., Lehman, J., Clune, J., & Stanley, K. O. (2019, July). Poet: open-ended coevolution of environments and their optimized solutions. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 142-151).\n[2]Wang, R., Lehman, J., Rawal, A., Zhi, J., Li, Y., Clune, J., & Stanley, K. (2020, November). Enhanced POET: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning (pp. 9940-9951). PMLR.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "bQg1BqXGVF1", "reviewer": "Reviewer_9QtA", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper porposes an interesting method to learn the agent's morphology that are robust to the change of environments. To achive this, this paper proposes a co-evolution method to learn  two policies that automatically change the morphology and the environment, respectively. The experimental results demonstrate the supriority of the proposed method over baselines.", "summary_of_the_review": "Overall, the proposed method is interesting and novel in the morphology design area. The experimental results also show the effectiveness of the proposed method.  The authors can provide more evidence in resolving my concerns provided in the \"Weaknesses\" section.", "strength_and_weaknesses": "Strength:\n\n1. The proposed method is interesting and new, and may inspire more subsequent works in this area.\n\n2. The presentation is clear and easy-to-follow.\n\n3. The experimental results clearly demonstrate the supriority of the proposed method over baselines.\n\nWeaknesses:\n\n1. Although the proposed method only provides a quantitative evaluation of the learned morphology, a qualitative evaluation is also required for this task. Can the authors include some videos of how the agents perform in various environments?\n\n2. The paper only demonstrates the performance of the proposed method when trained in a changing environment. Can the authors conduct some experiments on baselines trained in varying environments (even if they are random)? It is, in my opinion, compelling evidence for the efficacy of the proposed co-evolution method.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "FpDCtNtAEmY", "reviewer": "Reviewer_QUPT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a curriculum learning approach for reinforcement learning (RL) that changes the morphology of the agent as well as the environment for training agents that are robust to test-time environments. The authors provide thorough experimental results and ablation studies in three domains and demonstrate the effectiveness of their method.", "summary_of_the_review": "The idea behind the paper is interesting, and the authors provided thorough experiments in support of their method. That said, I still have some concerns regarding the applicability/generality of the method, lack of analysis on the exact curriculum induced by MECE, etc. I have added several questions which I hope the authors will respond to, as well as fix minor issues that I have identified.", "strength_and_weaknesses": "## Strengths\n\n- The paper takes on an interesting problem in deep RL, jointing learning an environment and morphology of the agent for zero-shot generalization on previously unseen tasks.\n- This paper provides a comprehensive set of experiments and, particularly, ablation studies to make sure that all aspects of the MECE are important and necessary for superior performance.\n\n## Weaknesses\n\n- I have some concerns and questions regarding the morphology-environment co-evolution in MECE.\n    - It is unclear to me why the authors update $\\pi_m$ and $\\pi_e$ with samples from D before using those to modify either the morphology or the environment. I think the opposite makes more sense.\n    - It is unclear to me why it is not possible to modify both (they are in the different blocks of the if-else statement). Have the authors tried ablating this part of the algorithm?\n    - Most importantly, I’m concerned about the new hyperparameters $\\delta_m$ and $\\delta_e$. It seems that these change the nature of the CO-EVO significantly, but fine-tuning these would be difficult because there are no intuitive values that would work for any environment. I’d like to hear what the authors have to say about this.\n- Rewards for each environment used in these papers are different (Appendix A). I suspect this limits the generality of the method and makes the “off-the-shelf” usage of it impossible. Instead, it seems, that people would need to spend a lot of time and compute to find a good reward signal for the environment.\n- I would expect this paper to include graphs showing how the morphology and environment change for all the domains used in experimentation. Specifically, plotting morphology metrics (number of joints, legs) and environment (e.g. roughness) as a y-axis throughout the training. I would expect this to be in the paper for all three domains.\n- It is unclear to me if the AddJoint action has any arguments or just adds a joint at a random leg of the agent. Have the authors considered a more parametrized morphology-changing interface?\n\n### Minor issues.\n\n- Please increase the font size of the legend, etc in all figures. It is very difficult to read these.\n- Please bring Table 1 forward, near the beginning of Section 5.3.\n- I think in the second paragraph of Section 5.3 (first sentence) you should refer to Figure 3, not 2.\n- It is common to have the pseudo-code within a Procedure indented. (Algorithm 2)\n- “Unlike” is capitalized unnecessarily on Page 2, in the paragraph that starts with “The interplay …”\n- E is undefined in Eq (1)\n- In the first sentence of Section 4, please use $\\pi$ and $\\pi_m$ rather than ***pi*** and ***pim.***\n- The second sentence in Section 4 suggests that “training three policies may be trivial than a single RL policy”. Is this a mistake (you meant the opposite)?\n- Figure 2 doesn’t have (d) (e) and (f) labels as subfigures.\n- The word “unconqueerable” needs to be fixed on page 9.\n\n### Questions\n\n- Is the test set used for evaluation in-distribution or out-of-distribution compared to training environment variations?\n- What do the “Number of environments in SEW” and “Number of morphologies in SEW” specifically mean in Table 2?\n- Were the morphologies of the agents and environment parameters constrained in any way?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "0-oaGBZJo-b", "reviewer": "Reviewer_JRxx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new framework named MECE for the co-evolution of both agent morphology and environment. The core ingredient of the framework is three separate policies: one for controlling the agent, one for changing the agent morphology and one for changing the environment. The three policies are trained by exploration in the simulation environment. The experiments are done on three types of environments in simulation and the proposed method shows gain in terms of learning efficiency in terms of rewards compared to baseline methods.", "summary_of_the_review": "Though the newly introduced problem is interesting, it is questionable whether the formulation is meaningful given the fact that the method essentially changes the task. Also, it is questionable whether the resulting task distribution can be controlled.", "strength_and_weaknesses": "Strength:\n\n- The paper investigates a new problem of agent-environment co-evolution, which is interesting and valuable to the community.\n- The method of using three separate policies for control, agent evolution and environment evolution is interesting and can be viewed as an extension of Transform2Act.\n- The reward design for agent evolution policy and environment evolution policy is interesting, but it would be better to provide more explanation on these two.\n- The experiment results are strong compared to previous methods.\n\nWeakness:\n\n- The biggest weakness is the problem definition itself. Though the problem formulation is interesting, I'm not sure if it is the right way to deal with the problem of robot agent learning. In practice, we would like to fix the distribution of environments but change the agent only:\n  - Conceptually, \"environment\" represents the task itself and is the goal of agent design and learning.The method proposed in this paper essentially changes the task and makes the learning objective meaningless. Note that it is OK to evolve the agent because having better agents to work in the same task/environment is still meaningful. But it is not OK to evolve the environment because evolving/changing environment to make the task easier is cheating.\n  - The authors may argue that the environment distribution with environment evolution policy is more diverse therefore better, as they argued in the first paragraph of Section 5.1. However, the final distribution of the environment has become uncontrollable, unless the author can show that the final distribution of the environments after environment evolution using policy is guaranteed to be exactly the wanted distribution. However, I cannot see that from the paper.\n- Following up on the previous point, the algorithm may also have a problem. What replay buffer $\\mathcal{D}$ contains is essentially an evolution trajectory of morphology $m_t$ and environments $\\theta_t^E$ as suggested by Algorithm 1. This means the trajectory of morphology and environments in $\\mathcal{D}$ cannot possibly have certain desired distribution, e.g. uniform distribution over a range. Moreover, the trajectory of morphology and environments is uncontrollable.\n- The formal math notation definition of several variables are missing and makes the paper confusing. For example, what is the difference between $\\mathcal{R}$ in Eq (1) and $\\mathcal{R}^H_{\\theta_{\\beta t}^E}$ in Eq (2)?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject