Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors tackle the problem of learning to locomote with morphological perturbations, e.g. when the length and the thickness of different body parts vary. They propose a biologically-inspired, attention-based policy network architecture. It combines a distributed policy, with individual controllers for each joint, and an attention mechanism, to dynamically gate sensory information from different body parts.

They ablate against two baselines: 1) A policy based on the proprioceptive state, that performs poorly with highly variable body configurations, and 2) an (oracle) agent with access to a learned encoding of the perturbation that performs significantly better. (And RMA and TCN)

strengths_and_weaknesses: Strengths
- The paper is written well and is easy to follow
- The challenge of dealing with unknown morphological perturbations is interesting and relevant.
- The application of RMA as a form of system identification is very helpful to compare the approach to inferred information (sysid) in addition to a baseline that uses ground truth information (Oracle)
- DMAP convincingly outperforms all baselines
- The visuals are great. Especially Figures 6 and 7 give an interesting intuition

Weaknesses
- The presented architecture seems similar to common approaches in hierarchical RL. A comparison seems appropriate. E.g. [R1] but I can imagine that there are quite a few approaches out there
- Literature on hierarchical RL would be helpful
- All methods were trained with SAC. It is hard to estimate the impact of entropy regularization on the proposed method. It would be helpful to also include DDPG/TD3
- RMA was able to identify the morphological context in 30 transitions (0.5s). It seems like the morphological changes in the presented tasks are easy to infer. One could say that the task itself is not very hard and so the benefit of the presented approach is in question. It might be helpful to develop even harder environments up to the point where RMA fails or takes significant amount of time. How well would the presented algorithm fare?


[R1] Rao, D., Sadeghi, F., Hasenclever, L., Wulfmeier, M., Zambelli, M., Vezzani, G., ... & Hadsell, R. (2021). Learning transferable motor skills with hierarchical latent mixture policies. arXiv preprint arXiv:2112.05062.


questions: -
limitations: Yes
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: Generalization in RL is a difficult problem. In this paper the authors investigate generalization to agents with changing morphology. This is accomplished be learning a policy that is conditioned on a history of observations and can use this information to learn a better joint attention over the parts of the state space the actuators on the robot should focus on when deciding actions. They evaluate this model against others on a task where the size and length of limbs change across a set of 4 agents. The results appear to be strong across prior methods evaluated in the paper.


---------

 I have updated my score after the initial responses.

I have updated my score further to reflect details covering other reviewers.
strengths_and_weaknesses: Pros:
- Learning a more end-2-end approach for estimating and generalizing to these types of changes in morphology is easier to scale.
- The network design is sensible and should results in some interesting learned attention.
- The focus on the 3 main indicators of a robust sensory-motor system is interesting.

Cons:
- The particular motivation in the paper is somewhat confusing. Are there many scenarios where the length and thickness of agent limbs change frequently? More discussion of this motivation would help readers understand what larger problem this work is connected to. 
- I also find the experiment section somewhat unclear. The design experiments are difficult to connect back to the claims in the paper. It would be very helpful to include some additional details on the particular types of experiments and why those particular experiments are being performed. For example, it's still somewhat unclear that the type of adaptation that is being claimed in the paper is mid-episode adaptation across a changing morphology. Or if the overall idea is to be able to train a policy that can adapt to other more apologies without additional episode data given. These details are important to understand what are the best methods or most related methods to compare in the experiment section.
questions: Questions/Comments:

- It does not appear to be many experimental comparisons to prior work. For example, the experimental conditions described in the paper are very similar to the ones in Pearl, a meta-learning method that infers agent morphology to be able to do some tasks quickly. and that paper, they even perform this type of morphological modification to two different agent types as one of their motivational environments for being able to adjust for quicker adaptation.
  - Rakelly, Kate, Aurick Zhou, Deirdre Quiilen, Chelsea Finn, and Sergey Levine. 2019. “Efficient off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables.” 36th International Conference on Machine Learning, ICML 2019 2019-June: 9291–9301.
- There is related work that fits this type of experimental analysis. If the claims in the paper and analysis where more clear it is likely the problem statement in the paper would align well with these prior works. It would also be important to compare to these prior works.
  - Nagabandi, Anusha, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. 2018. “Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning.” 7th International Conference on Learning Representations, ICLR 2019. arXiv. http://arxiv.org/abs/1803.11347v6.
  - Nagabandi, Anusha, Chelsea Finn, and Sergey Levine. 2019. “Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL.” 7th International Conference on Learning Representations, ICLR 2019, 1–15.
- It would be interesting to better understand the limitations of the attention module. Often these model require a large amount of data to train a useful representation. Some analysis to understand how many morphologies or tasks are needed to learn a generalizable representation would increase the strength of the work.
- Additional related work
  - Kurin, Vitaly, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. 2020. “My Body Is a Cage: The Role of Morphology in Graph-Based Incompatible Control.” arXiv [cs.LG]. arXiv. http://arxiv.org/abs/2010.01856.
  - Trabucco B, Phielipp M, Berseth G. AnyMorph: Learning Transferable Polices By Inferring Agent Morphology.  ICML 2022.
limitations: Some are discussed in the paper.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - This paper studies the problem of zero shot adaptation in the context of changing bodies doing a continuous control task. 

- Contrary to typical RL benchmarks that train to maximize expected rewards and test on the same metric, the metric here is to evaluate the adaptation capability of the learn policies under morphological changes. 

- The key idea is to factor out each proprioceptive channel and learn local embeddings. An attention based architecture is then used to implement a form of dynamic gating to solve the control task. 

- During training, the bodies are perturbed in simulation with a fixed distribution and then evaluated on test environment with unseen values. 

- The key result is that this architecture is at par or even surpasses the performance of an oracle given unseen body configurations during test time. 

strengths_and_weaknesses: Strengths:

- I really like the setting of this paper -- humans and animals have to learn under similar settings because the body changes everyday while the sensory motor loop still has to keep improving as the organism finds itself in new situations. 

- This setup will also be increasingly relevant in the robotics community as we develop robots with more dexterity and human-like hands -- because as the embodiment becomes more complicated, the robot will needs lots of iterations and fixes due to wear-and-tear, and new methods will be needed to handle to adapt to these morphological changes. 

- The experiments and writing are also clear. The approach is demonstrated on a set of continuous control tasks where the embodiment is structurally changed. 

- Figure 6 shows emergent symmetries in the body space, which has natural biological analogs. 

Weakness: 

I wonder if the authors have a tried a sequence to sequence based approach (e.g Decision Transformer) to model this problem. This would be an "inductive-bias" free approach. This is less of a limitations in my opinion but more of a contrast to the bias that this paper proposes. I suspect this approach would be more data efficient or could adapt quicker with more drastic body amputations.

questions: - It seems like Figure 2 varies the body w.r.t various parameters like size and shape. However, how does the approach fare under more radical transformations like broken legs? In order words, where does this approach fail? 

- The qualitative analysis using UMAP and the observation stated "We speculate that this untangling mechanism might contribute
232 to the robustness of the policy to morphological perturbations." is quite interesting. 
limitations: Discuss and perform evaluation of where the approach fails. Currently the morphological variations seems to be constrained to shape and size, but not breakage.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper introduces DMAP, an algorithm that allows reinforcement learning to adapt the policy to locomotion environments where the character doesn’t have a fixed body. DMAP is based on an attention-based feature encoder.
strengths_and_weaknesses: - The idea behind this work is very interesting and the method shows good results.
- The paper motivates clearly the reasons of the architectural choices for DMAP pipeline.
- It ablates the different components to understand the effects of the choices.

questions: I think most of the questions are caused by the fact that too much information is squeezed into the main paper and details are skipped. Which makes me believe that it’s quite hard to reproduce from reading the paper alone. I would perhaps suggest to put some ablations in the appendix and expand on specifying the model itself.

- The fact that the character changes is equivalent with saying that the environment changes and as you mentioned this resembles the problem setting of zero-shot adaptation, or meta-learning. It would be interesting to see how methods and algorithms from the zero-shot adaptation literature trained in your environment settings compare to learning morphological changes with attention with respect to results or learning efficiency.

- In the distributed joint controllers, the proprioceptive state included in the state, is the local of just that specific joint or of the whole character?

- How are you producing the matrices K and V from the stack of independent representations? You call K “joint-channel attention”, is this produced via attention/transformer layers? I can’t find it specified in the description of the method.

- What do cells in V represent? Is the size V a hyperparameter?

- It’s hard to read figure 3 and 4 separately, I would suggest put the 4 lines together.

- Regarding the above, it seems, but I might be wrong, that the oracle is worse than DMAP. From my understanding the oracle should have full observability of the environment and should perform better. Do you have explanations for this?

- In section 5.4, I don’t understand how this ablation is performed. Are you removing $e_i$ from the input of each $\omega$ ? If so, doesn’t this become equivalent to the TCN architecture?

- Figure 6. Is there an explanation why the hip controller just looks at the front knee velocity and the knee controllers only the back knee velocity?

- Having $N_a$ separate policies will definitely increase computation time. How does learning speed compare to a single policy method? A way to ablate the importance of distributed joint controllers could be to give the full $E^{(t)}$ to a single policy that controls the full action.

- Regarding the point above, are separate policies feasible with a more complex character like humanoid? 

limitations: Overall the method and the idea is really interesting. Most of my concerns are pointed out in the questions section.

I am willing to update my score after my concerns and questions have been clarified.
review_title: null

Meta-review: metareview: This paper examines the problem of learning locomotion for a simulated robot body, when the length and thickness of the body parts are varied.  The proposed method generates a distributed policy with controllers for each joint, and an attention mechanism that dynamically gates sensory information from different body parts.  The method performed well when compared to a policy based on proprioceptive state, and performs comparably to an oracle method given an embedding of the perturbed parameters.

Strengths of the paper noted by the reviewers included the clear writing, the relevance of the problem, and the convincing results.  The author response addressed potential limitations of the work raised by each of the reviewers. The reviewers were satisfied with the clarifications provided by the authors, and raised no additional concerns.

Four reviewers indicate to accept the paper for its clear contributions on how to learn locomotion policies when the body parameters are changed.  The paper is therefore accepted.
confidence: Certain
Decision: Accept