Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper presents MetaMorph, a Transformer based universal controller to learn behaviors across different robot morphologies. Their Transformer module takes a sequence of tokens as input, corresponding to the number of modules in the robot. Each input token comprises the proprioceptive and morphology information for each constituent robot’s module. MetaMorph is trained using a standard model-free Proximal Policy Optimization (PPO) method. However, to ensure all robots get to explore given environments and gather a similar amount of experience, the paper introduces a dynamic replay buffer which prioritizes robots’ selection for experience collection based on their previous iteration's episode lengths. The results are compared against MLP based experts trained for all individual robots, Graph Neural Network-based universal controller, and ablated modules of MetaMorph to highlight design choices. Moreover, results also demonstrate the generalization capacity of MetaMorph to different robots through zero-shot transfer and fine-tuning. 

main_review: Strengths:

Overall the paper is well written. It proposes an interesting way to use Transformers for learning a universal controller for different robot morphologies. The idea of a dynamic replay buffer is simple but exhibits improvement in learned behaviors. The results are thorough and cover most aspects of their approach and exhibit improved performance over prior methods. The ablation studies are also well-suited to describe the design choices. 

Weakness:
The contributions on algorithmic and network designs are marginal. Their approach is a direct application of Transformers and Reinforcement Learning approaches. 

Clarification: The description of section 4.3 can be improved, it is slightly confusing to follow. It seems Eq 5 will give a higher probability to robots with longer episode lengths which is in contrast to their description. Please elaborate or clarify if I missed something.  

summary_of_the_review: It proposes an interesting way to use Transformers for learning a universal controller for different robot morphologies and presents an extensive set of experiments. However, the overall contributions to algorithms and network designs are limited.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper uses transformers for incompatible multitask continuous control and shows impressive empirical results on a wide range of tasks in multitask training, zero-shot generalisation and transfer. The authors propose morphology-based conditioning and dynamic replay buffer balancing to increase the performance.
main_review: ### Pros

- Strong empirical results for multitask training/zero-shot generalisation and transfer.
- Dynamic replay balancing is an idea that might be useful for multitask RL outside this particular setting.
- The paper is mostly clearly written and has nice graphics.
- Well-written related work section.

### Cons

- I find it confusing that the method name has 'meta' in it?
- Lack of clear comparison with Amorpheus (Kurin et al, ICLR 2021) that also uses transformers for incompatible multitask control.
- Dynamic replay balancing will only work if the environments are somewhat similar given the used metric (e.g. similar number of episode steps).
- Section 5.5 is confusingly written. The relationship to muscle synergies sounds cool and sells well, but I don't think it makes a lot of sense to me in this setting or it requires more explanation.
- I don't think that MetaMorph-NM is a fair ablation that tells how morphological information is useful to training. A fairer ablation would be providing a one-hot encoding of a task instead of some meaningful morphological information. Can you run this ablation during the rebuttal period?
- Some of the very important details are omitted, i.e. what does $s^k_m$ contain? How do you learn positional embeddings? See more in Questions/Comments section.

### Questions/Comments

- Does NerveNet benefit from balancing? When you compare with GNN in Figure 4, does it include the balancing?
- Amorpheus (Kurin et al, ICLR 2021) shows that zero-shot generalisation exhibits quite a large variance across the seeds, however, your approach does not suffer from this issue, why do you think this is the case?
- When you study zero-shot/transfer behaviour, what is the experimental protocol? You run multiple seeds → select a model → run zero-shot evaluation/finetune for transfer. How do you select a model? Do you select a single model or one per each of the original training runs?
- You don't compare to Amorpheus, why? What ablation in Figure 4 is the closest to Amorpheus?
- W_{pos} should take a k-th row for the dimensions to make sense, not adding the full matrix.
- How do you obtain a positional embedding matrix for your zero-shot transfer experiments?
- DFS order depends on the internal representation of your tree, i.e. A→B←C might go to A or C after B depending on the order you store them in. What happens to your method if you do zero-shot generalisation transfer on the mirrored agent (so that DFS returns a different ordering of the same nodes)? Can you provide the results of this experiment?
- How do you learn the positional encodings? How do they look like if you plot them as a matrix?
- What does the morphological information contain, i.e. what is $s_m^k$?
- typo on page 2 'planer' → 'planar'
- Are Figure 9 results consistent across different seeds?
- The related work section is missing a recent paper: "*Snowflake: Scaling GNNs to high-dimensional continuous control via parameter freezing", Blake et al, NeurIPS 2021*
summary_of_the_review: This is a good paper with impressive empirical results. While transformers have been applied for incompatible multitask continuous control before, none of the works scaled the setting that much. Dynamic replay balancing proposed in the paper might be useful in multitask RL in general. I would give this paper a 7, but the system does not allow it. I am putting 6 for now and am willing to increase the score if the authors address my questions/comments during the discussion period.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper presents MetaMorph (MM), a transformer-based architecture for generalization over robot morphologies. Extensive experimentation demonstrates that policies learned by MM transfer well to new robot morphologies.
main_review: This is beautiful work! The approach is well-motivated, and the experiments are thorough, covering a wide diversity of robots and environments, and demonstrating remarkable generalization. On the whole, the paper is exceptionally well-written and informative.

The main weakness I see is a failure to adequately distinguish the proposed approach (MM) from the very similar AMORPHEUS approach (AM) by Kurin et al., 2021. The current work claims that MM conditions the transformer on morphological information, while AM does not. However, it seems to me that MM and AM condition the transformer on essentially the same information. The confusion seems to arise because the two papers use the term *morphology* in two separate ways. This requires careful explanation.

Each robot consists of one body and some number of limbs branching from it. (This paper calls the body and limbs *modules*, but I’ll call them all limbs for ease of visualization.) The information about the robot’s limbs can be divided into 3 categories:  limb-to-limb connectivity, static limb state (like limb type and its joint’s range of motion), and dynamic limb state (like instantaneous angular position and velocity). For processing by the network, standard practice is to combine the state of each joint (both static and dynamic) with the state of the limb that it swings.

AM (Kurin et al., 2021) is a modification of SMP (Huang et al., 2020). In all three architectures (MM, AM & SMP), observations include both static and dynamic state information of joints and limbs, but not explicit limb-to-limb connectivity information. SMP uses limb connectivity to define the computational graph over which messages are passed, from limb-to-limb. By contrast, MM and AM use a transformer encoder which passes messages between every pair of limbs, whether directly connected to each other in the robot or not. Kurin et al., 2021, demonstrate that AM’s all-to-all message passing performs better than SMP’s restricted message passing.

Now to the terminology. Kurin et al., 2021, uses the term *morphology* to refer to the *connectivity* of the limbs. But *this* paper uses the term *morphology* to refer to the *static state* of limbs (or their joints). This seems to explain why the paper states that MM differs from AM in “conditioning the Transformer on morphological information”, as if AM didn’t. This is incorrect, because both AM and MM condition the transformer on both static and dynamic state, and neither one conditions the transformer on connectivity information. If AM leaves out some important piece of static information, or if there is some other essential difference between AM and MM, the paper in its current form does not make this difference clear.

I agree that the current work differs from that of Kurin et al., 2021, “in the diversity and scale of training robots, complexity of the environments, … and showcasing strong generalization to unseen morphologies and tasks”. But I don’t see an important architectural difference between the models. (As the paper states, “the position embeddings carry no information about the graph structure”, so limb-to-limb connectivity is not carried by MM’s position embeddings.) AM is clearly the most meaningful baseline model to compare MM against, but this was not done, despite the availability of the AM code.

The experimental results show that MM’s position embeddings boost performance, but I find no explanation for why they do. Could it just be that they provide unique IDs to help the transformer differentiate the modules from each other, similar to a scheme employed by SMP?

summary_of_the_review: The paper presents compelling experimental results, but does not adequately differentiate MetaMorph from AMORPHEUS by Kurin et al., 2021, and does not include AMORPHEUS as a baseline in the experiments.

--- POST-DISCUSSION UPDATE ---

The authors have clarified the distinction between their work and that of Kurin et al., 2021, and have performed new experiments comparing MetaMorph to an ablated version (MetaMorph-AO) that approximates the more restricted morphological observations used by Amorpheus. 

Although MetaMorph and Amorpheus use differing sets of input features, they are still fundamentally the same architecture. (The "morphology aware transformer" used by MetaMorph is just a standard transformer encoder, like that of Amorpheus.) So in my view, there is little novelty in the Amorpheus architecture, and the work's value lies in the scope of its experiments. I have raised my evaluation accordingly.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: In robotics, researchers primarily train a single robot for a single task, while modular robot systems suggest the flexible combination of general-purpose building blocks into task optimized morphologies. The authors proposed a Transformer approach to learn a universal controller over a modular design space leveraging the “large-scale pre-training and fine-tuning” scheme.
main_review: Strengths
- The problem formulation (especially 4.2 Morphology Aware Transformer) is novel and quite persuasive.
- The proposed claim is appropriate in that robot morphology can be understood as a modality on which we can condition the output of a Transformer.
- Implementation details are shared so as easy to reproduce.

Weaknesses
- In the 5 Experiments, the model sizes of the other design choices for comparison are smaller than the proposed model, which may lead to unfair comparison.
summary_of_the_review: I generally agree with the argument of the paper. As the authors mentioned in the Abstract, Deep learning research fields like vision, natural language, and audio actively leverage Transformers for large-scale pre-training followed by task-specific fine-tuning. The idea makes sense that it deals with robot morphology as a modality on which we can condition the output of a Transformer in the modular robot design setting.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a Transformer-based universal policy to control modular robots. Being the space of modular robots combinatorial, it is (almost) impossible to train specific policy for each sample of the "robot distribution". Therefore, the paper proposes a transformer-based architecture design with explicit information about the robot morphology that, trained on a large number of possible morphologies. This universal controller generalizes zero-shot to unseen robot designs belonging to the training distribution and can be efficiently fine-tuned on new robots and tasks. The approach is thoroughly evaluated on different tasks.
main_review: Overall, I think this paper is very interesting. In my opinion, its main strengths are:
1. The design changes to the standard transformer architecture are sound and interesting for the problem setup. The main idea, i.e. conditioning the architecture to the robot morphology, is sound and interesting. Also, the dynamic replay buffer (basically storing samples in function of performance) is a simple but very nice idea. None of those are really breakthroughs but they are sound technical contributions that could influence future robotics research.
2. The experimental results are solid (even though some things could be improved, more on this later).
3. Writing is good and easy to follow, with figures helping to understand the main concepts.

The following is a set of weaknesses that I think could improve the contribution. None are major weaknesses or flaws, but just recommendations to improve the current submission:

1. The motivation to study the problem of a universal controller for robot morphologies could be improved. It looks like the main motivation is to "bring the advantages of pre-training into robotics", which sounds more like a methodology than a research problem. I think a more solid connection to real-world problems could be the situation when some parts of the robot fail (a propeller for a drone, a joint for a manipulator, etc.), and the control policy should be robust to these changes. The robustness to morphologies in the same training distribution could be very valuable there I think.
2. Several prior works on legged locomotion ( like R1 or R2) condition the control policy on some environment and robot characteristics. This is in a way similar to the proposed approach of conditioning the policy on morphology, so I think it deserves mention.
3. I did not fully get Eq. 5, which explains the dynamic replay buffering. Assuming $\beta>0$ (which seems okay according to the text),  the longer the episode length, the higher the probability of being sampled. This is in contrast with the motivation of dynamic replay buffering balancing, as explained in the example in section 4.3, where one wants to sample robots with short episode length (thus difficult to train).
4. I find it interesting that in the presence of obstacles, the meta-policy works much better than the per-robot MLP. Why is this the case? Why don't we observe the same in the flat and variable terrain?
5. As baseline, it would be nice to have a naive approach that substitutes the transformer architecture with a simple MLP. This MLP would have the same observations of the transformer and the same number of parameters to make the comparison as fair as possible. This naive baseline could give more intuitions on the advantages of the transformers for the task.
6. In Section 5.5, it would be nice to analysize a specific robot morphology and try to decode the resulting motor synergies. For example, would be nice to answer the question: why is limb N highly correlated to all other parts of the robot?
7. Finally, it would be nice to have a section discussion limitations of the approach. The biggest now seems that the generalization to robot out of distribution (in particular with different kinematics) is limited, and 2x10^7 iterations or more might be required to fine-tune on a new robot (which is not really few-shot). Overall, it would be nice to be open about the limitations to facilitate future work.
 
[R1] RMA: Rapid Motor Adaptation for Legged Robots, Kumar et al.

[R2] Learning Quadrupedal Locomotion over Challenging Terrain, Lee et al.
summary_of_the_review: The paper tackles an interesting problem in a sound way introducing some key technical contributions. Addressing some weaknesses in the presentation and in the experimental part, as well as being more open about limitations, could improve the quality of the submission.
review_title: null

Meta-review: The paper aims to improve generalization and sample efficiency in robotic control. In this regard authors note that modular robot systems, which provide building blocks for a task-specific morphology, can be considered just another domain where transformers can be used. The authors propose to learn a universal controller over this modular design space and leverage successful “large-scale pre-training and fine-tuning” scheme for transformer. We thank the reviewers and authors for engaging in an active discussion. Reviewers found that methodological novelty is limited ("The contributions on algorithmic and network designs are marginal. Their approach is a direct application of Transformers and Reinforcement Learning approaches.") and there are very similar approaches in the literature (e.g., AMORPHEUS approach (AM) by Kurin et al., 2021.) Despite these shortcomings, overall the reviewers found the experimental results to be strong and analysis to be thorough, which will be valuable to the community.
Decision: Accept (Poster)