Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose control graphs as a way to train agents with different morphologies and across different tasks. They develop MxT-Bench to train with different agent morphologies and across different tasks. They show a set of offline experiments that demonstrate the usefulness of their control graphs over MLPs and GNNs.
strength_and_weaknesses: Strengths:
- The paper is well presented and the motivation is very clear.
- The research question of designing architectures that generalize to different agent embodiments and tasks is very interesting, practically useful, and well-discussed.
- The idea for control graphs and setting them up to be used in a transformer is pretty neat!
- The analysis of the attention weights is well written and done.

Weaknesses:
- W1. The agent morphology and task setups are really limiting (agents being basic variants of ant, centipede, and claw agents, tasks being very limited to reach, touch, twister, and push tasks). It would be hard for me to accept the usefulness of control graphs without much more experimentation with different robotic setups and tasks. I would have liked to see something along the lines of trying control graphs with respect to parameterizable simulated robots and with respect to more real-world tasks. It doesn't seem like this work 
- W2. The work only deals with optimizing offline datasets.
- W3. It appears that the Transformer CGv1 and CGv2 do not encode any temporal information, that is, there is no information flow from the previous time steps to the current time step, which would make the current formulation not work well in any tasks that require memory.
- W4. In Table 1, performance for in-distribution seems lower than I would have expected. I would have expected a transformer to practically be able to overfit with near perfect training performance. Why isn't this happening?
clarity,_quality,_novelty_and_reproducibility: The tasks and setup were fairly easy to follow. There's been some recent work in morphology generalization, but not too much with respect to both different morphologies and to different tasks, which makes this work novel in that respect. I don't expect much of an issue with reproducibility as the tasks were relatively simple.
summary_of_the_review: The work is well-presented and easy to follow, and the direction of the work is quite fascinating. However, I'm concerned about the lack of demonstrated practical usefulness of control graphs. The experiments were only on relatively simple tasks with basic mujoco-styled agents. I would have liked to see more experiments where the agent requires temporal memory and with more real-world robotic embodiments. As the work stands now, it's hard for me to see the usefulness of using control graphs beyond pretty simple setups.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors created an benchmark called "MxT-Bench", which study the cross product of morphology and task generalization. Moreover, the authors proposed a unified graph-based representation for multi-morphology and multi-task. The authors provide a wide range of experiments and showcase the benchmark and models.
strength_and_weaknesses: I want to preface this by saying that I'm not really familiar with the field of controls across morphologies. 

Strength:
- MxT bench is well designed and seems to be technically sound. This can be of interest to many different researchers.
- The experiment study is quite thorough and offers some interesting insights. 

Weakness:
- I feel like the analysis in 5.3 is not so strong. Especially figure 6, where I don't see much to be honest. 
- I wonder if a comparison of models learned across benchmarks can be useful. For example, does the cross morphology models learned in other benchmarks perform well in MxT?
clarity,_quality,_novelty_and_reproducibility: Great on all front.
summary_of_the_review: I like this paper and think this is above acceptance threshold. However, I'm waiting to see other reviewers's feedback (who probably are more familiar with the field than I am), before giving it a very strong endorsement.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper focuses on multi-morphology and multi-task generalization in robotics. There are three main contributions to this paper. First, a comprehensive benchmark called MxT-Bench is developed for training and evaluating morphology-task generalization. This benchmark also supports the scalable procedural generation of both agents and tasks. Second, a new concept Control Graph is proposed as a universal interface representing observations, actions, and goals/tasks. Last, the authors study several offline supervised behavior distillation methods with different design choices and obtain conclusions about generalizable model design.

Compared to existing works of universal controller and morphology/controller co-design, this paper provides a more comprehensive and systematic design, which has the potential to unify the tools and standards used in this area. 

strength_and_weaknesses: ### **Strength**

* **A systematic platform.** Universal controllers for different robots and co-design of morphology and controller are prevalent topics in robotics. Different papers propose many different simulators and protocols, making the development and comparison between algorithms difficult. The benchmark proposed by this paper looks promising to me. I think it will greatly contribute to this area by unifying the tools and standards used in it.

* **Thorough analysis of experiments.** Beside the benchmark, this paper also explores multiple generalization settings, including multi-morphology, multi-task, and out-of-distribution downstream tasks. Some interesting results will definitely provide guidance for future work.


### **Weaknesses**

One weakness in my opinion is that an overview of the structure of this paper is missing. Since this paper proposes many new concepts, understanding the connections between them could be difficult.
Figure 1 tries to summarize all contents but still needs to be improved. For example, section numbers can be added to different components and main contributions can be highlighted.
The title can also be improved. I think the Control Graph is not the main contribution. A more precise title helps understand the content.

clarity,_quality,_novelty_and_reproducibility: ### **Clarity**

The writing is clear and easy to understand. An overview of the paper structure could help readers quickly understand the connections between all concepts.

### **Quality**

This paper has very high quality. All three contributions are important to this area.

### **Novelty**

The novelty of this paper comes from the new benchmark and unified representation. No novel algorithm is proposed.


### **Reproducibility**
I think the results are reproducible.

summary_of_the_review: I suggest accepting this paper. The morphology-task generalization topic becomes a prevalent trend recently. I believe the benchmark and unified interface proposed in this paper contribute a lot to this area.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper proposes a continuous control benchmark for multitask reinforcement learning supporting three axes of change: creature morphology (supporting state and action spaces of different dimensions), task (e.g. reach some place or touch some point), and goal.
The paper further develops the graph-based representation pioneered by NerveNet, adding the support of the goal-directed setting. Finally, the authors investigate policy distillation behaviour on their own benchmark
strength_and_weaknesses: ## Strengths

* The proposed benchmark is a large benchmark supporting a variety of degrees of freedom: morphology, different mass/size for limbs, multiple tasks (using the authors' terminology), goal distribution.
* The paper investigates policy distillation in the incompatible setting (mismatching state and action space dimensions across tasks) which, I don't think, was done before.
* The benchmark seems to be super fast with single-agent PPO taking 5-30 minutes per environment

## Weaknesses

I think the positioning of the paper paper is its main weakness. I am not entirely sure what this paper is about. There are three potential storylines:

* The paper is about the benchmark 
* The paper is a policy distillation study in a novel setting (mismatching state/action spaces across agents)
* The paper is about the input representation

I think, all three stories are equally legit for this paper, but each of the story will define the requirements for experiments, and jumping around the stories will confuse the reader.

For instance, the title of the paper says that the paper is about control graph, i.e. input representation. However, the intro's motivational section speaks about the generalist agent and morphology generalisation. If the paper is about morphology generalisation, then Representation Selection, which is Appendix F now, should be in the main body of the paper etc. I think, the main body can accommodate more information if the transformer basics are moved to the Appendix.

I'm very confused about the terminology used throughout the paper: 'multi-task' is used in a very narrow meaning implying that the agents of the same morphology doing semantically different tasks (e.g. reaching some goal or moving limbs to specific positions). Usually, In RL, a task is a synonym of an MDP, and different morphology (even same morphology with different limb sizes/masses) constitute different tasks. As you mentioned, previous literature (e.g. Gupta et al., 2022, Metamorph) consider morphology/your definition of task separately and that 'broad generalisation over task and morphology jointly remains a long-standing problem'. Do you claim that there is a qualitative difference between generalising across morphologies and generalising across your definition of tasks? In other words, what is the motivation behind separating multitask into two separate groups?

Finally, I find it concerning, that the paper that does policy distillation does not cite policy distillation papers, including Parisotto's "Actor-Mimic" and Rusu's "Policy Distillation". The "Behaviour Distillation" section includes only 2020/21 papers with a single exception of Levine et al 2016. Task generalisation with GNN/Transformers work should include papers on combinatorial optimisation (see Cappart's 'Combinatorial optimization and reasoning with graph neural networks' for references). GNNs for continuous control work misses Blake's "Snowflake: Scaling GNNs to High-Dimensional Continuous Control via Parameter Freezing".

Other questions:
* I am not sure I fully understand how the goals are encoded to either control graph v1 or v2. In v1, when you add the goal observation to the node features, are other nodes' features padded? Do you provide the task identifier somehow (e.g. if that's reach or touch or twisters). In v2, figure three shows the dashed edges from the limb nodes to goal nodes, how do you encode this information in the transformer case? Also, judging by Figure 3, the dimensionalities of the goal nodes for different tasks are different. How do you ensure that they are all the same before going to the GNN/transformer?
* When you generate tasks, are they all solvable (whatever metric you choose for this). Can you provide plots with the return ranges for single-agent policies you use to collect the data? How did you select thresholds for goal-reaching?
* Could you elaborate a bit more on the learnable position embeddings, how exactly is this implemented. When zero-shot transferred, how are position embeddings calculated?
* How consistent is the behaviour in Figure 6 across the seeds?
* What do you think the wider implications of the proposed input representation are? Can we draw any conclusions that can be used outside the continuous control setting?

clarity,_quality,_novelty_and_reproducibility: I found the paper often confusing and list my clarity concerns below:

* I find the definition of 'multitask' to be narrow and counterintuitive (more details in the 'weaknesses' section above).
* It is hard to understand the title without reading the paper, i.e. what is control graph? "Generalizable input representation for graph-based policy distillation" sounds more applicable from my perspective.
* "policy distillation" is an established term, why do you use "behaviour distillation" instead?
* I do not understand the following sentence "which is significantly more efficient than multi-morphology training done in prior work" (Section 4.2) 
* "incorporates goal information while preserving the geometric structure of a task", what is "geometric structure of a task"?
* "because the task structure may change over time", what is "task structure"?
* "we examine how control graph works well..." is unclear to me.
* Nit: In Section 3, when the Deep RL is introduced, the 'policy' term has not been introduced yet.


summary_of_the_review: I think the paper has interesting findings and has the potential, but issues with the positioning and clarity I pointed out made me choose the score of 5. I am ready to increase the score after the discussion period if my concerns are addressed. 

UPD after rebuttal: The authors were actively engaged in the discussion and updated the paper to address my concerns. I would give the paper 7, but the scoring allows only 6 or 8, and I will update the score to 8.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper aims to learn a single policy for different forms of agents to perform multiple different tasks. To this end, a benchmark, called MxT-Bench is proposed, that procedurally generates different morphology-task combinations. The paper also proposes control graphs as a generic representation of observations, actions, and tasks. 

Strengths:
- The paper motivates the approach very well and it nicely presents the ideas.
- The paper includes very interesting ideas such as policy distillation when there is a mismatch between state and action spaces across tasks and also representing the control graphs in a transformer architecture.
- The benchmark enables a systematic study of these approaches. The efforts in this domain are quite scattered. 
 - The experiments and the analysis are thorough and provide interesting insights.

Weakness:
- The embodiments and the tasks are quite simplistic. 

The reviewers are generally positive about the paper. The AC also finds the ideas very neat and the experiments solid. The only drawback is the simplicity of the embodiments and tasks, but that is fine since the paper proposes interesting ideas. In the future, more complex scenarios should be considered. The AC recommends acceptance.

justification_for_why_not_higher_score: The embodiments and tasks are somewhat simplistic. I would vote for Oral if the paper considered more realistic tasks and robot embodiments.
justification_for_why_not_lower_score: The paper presents interesting ideas, and the experiments are thorough. So, I believe choosing the paper as Spotlight will highlight the work and encourage the community to continue research in this domain.
Decision: Accept: notable-top-25%