Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper porposes an interesting method to learn the agent's morphology that are robust to the change of environments. To achive this, this paper proposes a co-evolution method to learn  two policies that automatically change the morphology and the environment, respectively. The experimental results demonstrate the supriority of the proposed method over baselines.
strength_and_weaknesses: Strength:

1. The proposed method is interesting and new, and may inspire more subsequent works in this area.

2. The presentation is clear and easy-to-follow.

3. The experimental results clearly demonstrate the supriority of the proposed method over baselines.

Weaknesses:

1. Although the proposed method only provides a quantitative evaluation of the learned morphology, a qualitative evaluation is also required for this task. Can the authors include some videos of how the agents perform in various environments?

2. The paper only demonstrates the performance of the proposed method when trained in a changing environment. Can the authors conduct some experiments on baselines trained in varying environments (even if they are random)? It is, in my opinion, compelling evidence for the efficacy of the proposed co-evolution method.
clarity,_quality,_novelty_and_reproducibility: Quality:
The paper is good and clear.

Novelty:

The main idea is novel in this task.

Reproducibility:

The paper provides enough details to reproduce this paper.

summary_of_the_review: Overall, the proposed method is interesting and novel in the morphology design area. The experimental results also show the effectiveness of the proposed method.  The authors can provide more evidence in resolving my concerns provided in the "Weaknesses" section.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a new framework named MECE for the co-evolution of both agent morphology and environment. The core ingredient of the framework is three separate policies: one for controlling the agent, one for changing the agent morphology and one for changing the environment. The three policies are trained by exploration in the simulation environment. The experiments are done on three types of environments in simulation and the proposed method shows gain in terms of learning efficiency in terms of rewards compared to baseline methods.
strength_and_weaknesses: Strength:

- The paper investigates a new problem of agent-environment co-evolution, which is interesting and valuable to the community.
- The method of using three separate policies for control, agent evolution and environment evolution is interesting and can be viewed as an extension of Transform2Act.
- The reward design for agent evolution policy and environment evolution policy is interesting, but it would be better to provide more explanation on these two.
- The experiment results are strong compared to previous methods.

Weakness:

- The biggest weakness is the problem definition itself. Though the problem formulation is interesting, I'm not sure if it is the right way to deal with the problem of robot agent learning. In practice, we would like to fix the distribution of environments but change the agent only:
  - Conceptually, "environment" represents the task itself and is the goal of agent design and learning.The method proposed in this paper essentially changes the task and makes the learning objective meaningless. Note that it is OK to evolve the agent because having better agents to work in the same task/environment is still meaningful. But it is not OK to evolve the environment because evolving/changing environment to make the task easier is cheating.
  - The authors may argue that the environment distribution with environment evolution policy is more diverse therefore better, as they argued in the first paragraph of Section 5.1. However, the final distribution of the environment has become uncontrollable, unless the author can show that the final distribution of the environments after environment evolution using policy is guaranteed to be exactly the wanted distribution. However, I cannot see that from the paper.
- Following up on the previous point, the algorithm may also have a problem. What replay buffer $\mathcal{D}$ contains is essentially an evolution trajectory of morphology $m_t$ and environments $\theta_t^E$ as suggested by Algorithm 1. This means the trajectory of morphology and environments in $\mathcal{D}$ cannot possibly have certain desired distribution, e.g. uniform distribution over a range. Moreover, the trajectory of morphology and environments is uncontrollable.
- The formal math notation definition of several variables are missing and makes the paper confusing. For example, what is the difference between $\mathcal{R}$ in Eq (1) and $\mathcal{R}^H_{\theta_{\beta t}^E}$ in Eq (2)?

clarity,_quality,_novelty_and_reproducibility: Clarity needs to be improved by providing more formal math definitions.

Quality is good.

Novelty is also good since this paper deals with a new problem.

Reproducibility is unclear since it's not possible to re-produce the results for now because of no release of code.
summary_of_the_review: Though the newly introduced problem is interesting, it is questionable whether the formulation is meaningful given the fact that the method essentially changes the task. Also, it is questionable whether the resulting task distribution can be controlled.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this paper, the authors propose to use three different policies to modify simultaneously the actions, morphology, and environment in a Reinforcement Learning problem. 
While the control policy is very standard in RL applications, this paper also considers morphology and environment modifications as Markov Decision Processes. The state of the morphology considers some aspects of the robot's skeleton, while the state of the environment is some aspects such as roughness or dip length. The action spaces correspond to modifications of the state (of either the morphology or the environment). Finally, the reward functions are designed to encourage the agent to improve its learning progress and adaptability to varying environments. 
The proposed algorithm is evaluated in 3 different tasks (2d-locomotion, 3d-locomotion, and gap-crosser). It is compared to two baselines (T2A and NGE. Finally, a series of ablation studies are presented to explain the inner workings of the proposed algorithms. In all the presented results, the algorithm outperforms the baselines.
strength_and_weaknesses: I find the concept of considering morphology and environment optimisation as MDP problems a very very interesting concept. I really like it and find this very interesting. 
This is for me the main strength of the paper. 

Unfortunately, the propsoed experiments are not enough to gauge the benefits of the proposed method. All the consdered baselines use fixed environment (as depicted in Fig1) without any possible curriculum learning. The introduction of the paper stresses clearly that co-evolving the morphology and the environment is essential. Yet, none of the considered baselines does this, while the paper cites several papers that co-evolve morphology and environment to create such curriculum learning. Typically, the proposed methods should be compared to POET[1] and Enhanced POET[2], which are both designed with the same objective and motivation than the proposed algorithms. Only with such comparison, we will be able to know if training policies to modify environments and morphology is a promising alternative. 

[1]Wang, R., Lehman, J., Clune, J., & Stanley, K. O. (2019, July). Poet: open-ended coevolution of environments and their optimized solutions. In Proceedings of the Genetic and Evolutionary Computation Conference (pp. 142-151).
[2]Wang, R., Lehman, J., Rawal, A., Zhi, J., Li, Y., Clune, J., & Stanley, K. (2020, November). Enhanced POET: Open-ended reinforcement learning through unbounded invention of learning challenges and their solutions. In International Conference on Machine Learning (pp. 9940-9951). PMLR.
clarity,_quality,_novelty_and_reproducibility: The paper is relatively clear and well-structured. Most, if not all, the hyper-parameters are reported in the appendix and it is easy to follow. 


summary_of_the_review: Overall, the paper presents an interesting concept. Unfortunately, the selected baselines do not provide a fair comparison to evaluate the benefits of using MDP and policies for morphology and environment co-evolution.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a curriculum learning approach for reinforcement learning (RL) that changes the morphology of the agent as well as the environment for training agents that are robust to test-time environments. The authors provide thorough experimental results and ablation studies in three domains and demonstrate the effectiveness of their method.
strength_and_weaknesses: ## Strengths

- The paper takes on an interesting problem in deep RL, jointing learning an environment and morphology of the agent for zero-shot generalization on previously unseen tasks.
- This paper provides a comprehensive set of experiments and, particularly, ablation studies to make sure that all aspects of the MECE are important and necessary for superior performance.

## Weaknesses

- I have some concerns and questions regarding the morphology-environment co-evolution in MECE.
    - It is unclear to me why the authors update $\pi_m$ and $\pi_e$ with samples from D before using those to modify either the morphology or the environment. I think the opposite makes more sense.
    - It is unclear to me why it is not possible to modify both (they are in the different blocks of the if-else statement). Have the authors tried ablating this part of the algorithm?
    - Most importantly, I’m concerned about the new hyperparameters $\delta_m$ and $\delta_e$. It seems that these change the nature of the CO-EVO significantly, but fine-tuning these would be difficult because there are no intuitive values that would work for any environment. I’d like to hear what the authors have to say about this.
- Rewards for each environment used in these papers are different (Appendix A). I suspect this limits the generality of the method and makes the “off-the-shelf” usage of it impossible. Instead, it seems, that people would need to spend a lot of time and compute to find a good reward signal for the environment.
- I would expect this paper to include graphs showing how the morphology and environment change for all the domains used in experimentation. Specifically, plotting morphology metrics (number of joints, legs) and environment (e.g. roughness) as a y-axis throughout the training. I would expect this to be in the paper for all three domains.
- It is unclear to me if the AddJoint action has any arguments or just adds a joint at a random leg of the agent. Have the authors considered a more parametrized morphology-changing interface?

### Minor issues.

- Please increase the font size of the legend, etc in all figures. It is very difficult to read these.
- Please bring Table 1 forward, near the beginning of Section 5.3.
- I think in the second paragraph of Section 5.3 (first sentence) you should refer to Figure 3, not 2.
- It is common to have the pseudo-code within a Procedure indented. (Algorithm 2)
- “Unlike” is capitalized unnecessarily on Page 2, in the paragraph that starts with “The interplay …”
- E is undefined in Eq (1)
- In the first sentence of Section 4, please use $\pi$ and $\pi_m$ rather than ***pi*** and ***pim.***
- The second sentence in Section 4 suggests that “training three policies may be trivial than a single RL policy”. Is this a mistake (you meant the opposite)?
- Figure 2 doesn’t have (d) (e) and (f) labels as subfigures.
- The word “unconqueerable” needs to be fixed on page 9.

### Questions

- Is the test set used for evaluation in-distribution or out-of-distribution compared to training environment variations?
- What do the “Number of environments in SEW” and “Number of morphologies in SEW” specifically mean in Table 2?
- Were the morphologies of the agents and environment parameters constrained in any way?
clarity,_quality,_novelty_and_reproducibility: The paper is clear and generally easy to follow, although the clarity can be improved with minor changes (indicated above). The technical quality of the work is high. The idea of the joint curriculum learning over the environment and agent morphology is novel, as far as I’m concerned. The authors provided the hyperparameter values for their experiments (but not the code itself).
summary_of_the_review: The idea behind the paper is interesting, and the authors provided thorough experiments in support of their method. That said, I still have some concerns regarding the applicability/generality of the method, lack of analysis on the exact curriculum induced by MECE, etc. I have added several questions which I hope the authors will respond to, as well as fix minor issues that I have identified.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Summary:
This paper presents an algorithm for co-evolving the control, morphology, and environment of an agent. The rewards are set up so that the environment provides a curriculum that trains the agent's morphology and control. The result is agents that behave robustly in held-out test environments. 

Strengths:
* Co-evolving control, morphology, and environment is an interesting and novel idea
* Method convincingly outperforms baselines that do not co-evolve the environment

Weaknesses:
* No comparisons to baselines that do co-evolve the environment (like POET)
* The method is hard to follow and there was some disagreement about whether or not it is sound
* Hyperparameters may need to be tuned per environment
* More qualitative results and analysis of the learned behaviors, environments, and morphologies would be appreciated
justification_for_why_not_higher_score: All reviewers found this to be a very interesting paper, but expressed a variety of concerns about the method and experiments. JRxx engaged in a long discussion on the soundness of the method, which the authors rebutted. The AC sides with the authors here and agrees that the general idea, in principle, could provide an effective teaching curriculum. However, this discussion does raise the issue of clarity: in the paper itself, it's hard to follow exactly how the rewards work and why they lead to a curriculum; in particular, Eqn. 2 appears to reward steps toward ever "easier" environments (perhaps there is a typo?) and does not clearly relate to morphology progress.

Another concern was with the baselines: ugJV suggested comparing to methods, such as POET, that evolve a curriculum of environements. The authors rebutted that prior such methods do not evolve morphologies. Nonetheless, the AC agrees that, while not entirely critical, it would improve the paper to at least attempt to compare to such methods. For example, POET could be adapted to also update the morphology of agents (e.g., using the morphology update algorithm from the current paper).

Finally there were lingering concerns that the paper is not as thorough as it could be, in terms of qualitative results and analysis of the method. Taking all these issues in sum, the AC feels that the paper is very promising but could use another pass on clarity, comparisons, and analysis before it meets the ICLR bar.
Decision: Reject