DMAP: a Distributed Morphological Attention Policy for learning to locomote with a changing body
Authors: Alberto Chiappa
Alessandro Marin Vargas
Alexander Mathis
Venue: NeurIPS 2022 Accept (2022)
Keywords: Computational Neuroscience
Reinforcement learning
Sensorimotor learning
Motor adaptation
Motor learning
Morphology Perturbations

Abstract: Biological and artificial agents need to deal with constant changes in the real world. We study this problem in four classical continuous control environments, augmented with morphological perturbations. Learning to locomote when the length and the thickness of different body parts vary is challenging, as the control policy is required to adapt to the morphology to successfully balance and advance the agent. We show that a control policy based on the proprioceptive state performs poorly with highly variable body configurations, while an (oracle) agent with access to a learned encoding of the perturbation performs significantly better. We introduce DMAP, a biologically-inspired, attention-based policy network architecture. DMAP combines independent proprioceptive processing, a distributed policy with individual controllers for each joint, and an attention mechanism, to dynamically gate sensory information from different body parts to different controllers. Despite not having access to the (hidden) morphology information, DMAP can be trained end-to-end in all the considered environments, overall matching or surpassing the performance of an oracle agent. Thus DMAP, implementing principles from biological motor control, provides a strong inductive bias for learning challenging sensorimotor tasks. Overall, our work corroborates the power of these principles in challenging locomotion tasks. The code is available at the following link: https://github.com/amathislab/dmap

DMAP: a Distributed Morphological Attention Policy for Learning to Locomote with a Changing Body
Alberto Silvio Chiappa
Alessandro Marin Vargas
Alexander Mathis
École Polytechnique Fédérale de Lausanne (EPFL), Switzerland [first name] [surname] @epf1.ch
Abstract
Biological and artificial agents need to deal with constant changes in the real world. We study this problem in four classical continuous control environments, aug- mented with morphological perturbations. Learning to locomote when the length and the thickness of different body parts vary challenging, as the control policy is required to adapt to the morphology to successfully balance and advance the agent. We show that a control policy based on the proprioceptive state performs poorly with highly variable body configurations, while an (oracle) agent with access to learned encoding of the perturbation performs significantly better. We intro- duce DMAP, a pooggically-inspiree attention based policy network architecture. DMAP combines independent proprioceptive processing, distributed policy with individual controllers for each joint, and an attention mechanism, to dynamically gate sensory information from different body parts to different controllers. Despite not having access to the (hidden) morphology information, DMAP can be trained end-to-end all the considered environments, overall matching or surpassing the performance of an oracle agent. Thus DMAP, implementing principles from biological motor control, provides a strong inductive bias for learning challenging sensorimotor tasks. Overall, work corroborates the power of these principles in challenging locomotion tasks. The code is available at the following link: [DMAP
1
Introduction
Animals and humans are highly adaptive to changes in the external environment and in their own body. For instance, animals can, after an early onset, locomote throughout their development despite dramatic changes in their body size and weight. Animals can also robustly deal with irregular surfaces, and load changes
5].
Here, we focus on the problem of learning to walk when the body is subject to morphological perturbations, i.e.. changes in the length and thickness of body parts. Remarkably, animals can rapidly adapt to these kind of perturbations. For example, desert ants with elongated or shortened legs can immediately locomote, although misjudging the traveled distance [6]. To challenge artificial agents on such skills, we adapted four different locomotion environments (Ant, Half Cheetah, Walker, Hopper) from the OpenAl Gym 77 in which robots have to learn to walk as fast as possible. In our adaptive setting. the agent's body parameters are sampled from morphological perturbation space at the beginning of each episode and we sought to find reinforcement learning policies that could deal with these varying bodies.
We hypothesize that principles of the sensorimotor system provide a strong inductive bias to robustly learn such a policy. Specifically, we built on three well-known principles:
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

DMAP Distributed Morphological Attention Policy
Proprioceptive state history
Independent representations
Distributed Joint controllers
K
H
?
TCN
Joint-channel attontion
Q2
Time
TCN
Action
V
TCN
Q
Channel features
Epicodes
DMAP
HID Test
Time
OOD Test
Figure 1: A. Overview of the architecture of DMAP. Each component of the state history is processed independently by a temporal convolutional network (with shared weights). The extracted features transformed into the value encoding matrix and the attention weights of each joint sensor and action component. The product of these matrices results joint-specific morphology encoding, which is the convex combination of the value encoding vectors, weighted by their attention score. Independent fully-connected networks control a single joint, based on the current proprioceptive and the morphology encoding B. DMAP is trained for several episodes with randomly sampled perturbed bodies within the training distribution. At test time, the zero-shot performance of the model assessed on fixed set of body configurations, sampled either from within (IID) or outside (OOD) of the training distribution.
Independent low-level processing. Somatosensory inputs are first processed locally (e.g.. by temporal filtering) before they are integrated across body parts and represented an topographically ordered fashion 8 IT]
Gated connectivity. Sensory information is dynamically gated in task dependent manner and integrated with action and value signals in the brain [13 10 [4].
Distributed control. Different body parts are controlled in a distributed fashion
10
Based on these principles, propose the Distributed Morphological Attention Policy (DMAP) architecture (Figure D. We show that DMAP can robustly learn to locomote with changing bodies without access to the morphological parameters. To assess the performance of DMAP, we compare to several alternative agents: Simple, Oracle and rapid motor adaptation (RMA) [5] Experiments reveal that an agent with access only the proprioceptive state, consisting of its joint angles and velocities (Simple), fails to learn an effective locomotion strategy for a large variety of body configurations. In contrast, augmenting the observation with the current body parameters leads to strong ceiling performance (Oracle). Remarkably, for all different environments, DMAP is as good as Oracle. We also compare DMAP to powerful, recently proposed two-step training technique called rapid motor adaptation (RMA) 15 This algorithm trains temporal convolutional network to infer a context encoding from a short history of proprioceptive state, imitating the Oracle's s morphology encoder. While RMA can also reach Oracle performance, show that end-to-end training of such

---

network does not succeed for more complicated environments (Ant and Walker). In contrast, DMAP's policy can be trained end-to-end without access to the morphological parameters and reaches or surpasses Oracle performance for all four environments (Ant, Half Cheetah, Walker, Hopper).
Overall, our work suggests that independent low-level processing. distributed control and dynamic gating mechanisms provide a strong inductive bias for sensorimotor learning. Finally, we analyze the information gating mechanisms of DMAP. The strength of gated connections reveals morphology- specific patterns, such as bilateral and back-front symmetry. The low dimensional embedding of the gating mechanism exhibits rotational dynamics, robustly across different morphologies.
2
Related work
Benchmarks for zero-shot adaptation. The setup of the experiments described in this paper can be framed as zero-shot adaptation problem, as the agents are evaluated starting from the first moment in which they interact with an unseen context [166]. While classic RL benchmarks, such as Arcade 17 and rllab [88] assess the ability of an agent to maximize the cumulative reward the training environment, more recent ones introduce perturbations in the dynamics [19 20, in the observation function 21 22) or the reward function (23 at test time. For a more complete list, we refer to the survey by Kirk et al. about generalization in Deep Reinforcement Learning [66. Mann et al. showed that the performance of RL agents under morphological distributional shifts correlates with the ability of internal models to generalize 244 We took their design of environments an inspiration for the design of morphological changes in our work.
Context encoding and adaptation. Algorithms which enable learning in changing environments are important for the sim-to-real transfer problem, where domain randomization represents a successful strategy to bridge the reality gap [25 26
27 A notable approach to solving the zero-shot transfer problem consists in encoding the context into a hidden state, e.g., by online system identification (28) or with recurrent neural network (RNN) [8
29]. However, while off-policy continuous control algorithms such as TD3 B0] and SAC BT are the current state of the art among model-free RL algorithms, both in sample efficiency and final performance
22 training recurrent neural network
off-policy presents relevant technical challenges 133
The problem of adaptation to unseen environmental conditions is central in the Meta-RL literature 34 Both model-based (ReBAL, GrBal (35), MOLe (36)), and model-free (PEARL (37)) algorithms prove capable to accelerate adaptation to unseen contexts. However, they require an adaptation phase in the test environment, which can last few episodes K or some hours (35), seemingly suboptimal in our experimental setup (Appendix A.4). As the focus of our research is on the development of a network architecture to enable robust locomotion in a zero-shot manner, we chose to compare it to RMA, a powerful system identification approach by Kumar et al. [5 They showed that
context encoding can be successfully inferred from a short sequence of past states and actions, and this context encoding can guide robust policy [15].
Distributed control. An agent can learn to generalize to multiple body shapes by using modular policies to control each joint independently, e.g., by treating each joint (38) or group of joints B3
as a node of graph. For example, Nervener 400 achieves state-of-the-ar performance on standard
RL benchmarks by leveraging different policies for each joint type. However, it cannot generalize to different designs a zero-shot manner (without fine-tuning). On the other hand, Shared Modular Policies (SMP) [1]] employs the same policy for each joint, enabling an agent to deal with variable observation and action space sizes. Further solutions to handle bodics with variable connectivity graphs include transformer- based solutions, such Amorpheus 42 and the recently proposed AnyMorph 43 While these works study the problem of controlling bodies with a variable number of segments and connectivity graph, we focus perturbations which affect morphological parameters, but without changing the body structure.
Hierarchical RL A common approach to solve complex RL tasks is to decompose them in a hierarchy of subproblems 44 Walking with different body shapes could be interpreted as the union of long-horizen task (locomotion) and short-horizon one (body shape identification). Algorithms to tackle multi-horizon control problems by separating high-level behavior from motor primitives include Hierarchical Actor-Critic 45). HcLMS [46] and HiPPO 47
3

---

3 A biology-inspired architecture to handle morphological perturbations
Adaptation to morphological perturbations requires an agent to rapidly identify which joints are affected and what compensatory torques are required. We propose Distributed Morphological Attention Policy (DMAP) to address this problem (Figure [ With DMAP, we seek to facilitate this process by promoting the formation of communication pathways from individual body parts to the specific joint controllers. The architecture is designed according to three principles of (biological) sensorimotor control (also mentioned in the introduction):
Independent low-level processing, Each proprioceptive channel is first processed independently, in order to obtain channel-specioe representation. This is inspired by the processing in the proprioceptive pathway 9 T0
Gated connectivity. An attention mechanism assigns the connection weight between the control policy of joint and the features of each component of the proprioceptive state. This is inspired by dynamic gating mechanisms in the brain 1|I
Distributed control. Independent networks control each of the agent's joints, by outputting one single action scalar (corresponding to joint's torque). This is inspired by the architecture of the spinal cord and the motor homunculus g [0
We implemented those principles in the following way (Figure 1 and for more details Appendix A.1
3.1 Independent processing and attention-based feature encoder
A feature encoder receives in input a history of T proprioceptive states and actions ((s(t-T), (t(-T)) (s(t-1), a ((- which we denote as H E (NNs Ng and NA being the state and action space size, respectively. Each row h IRT of H is processed independently by the same Temporal Convolutional Network (TCN). A learned linear transformation maps each temporally-fitten filtered input channel into the vectors k!") E R-*A, representing the attention of each joint towards that input channel i, and v! E NR, value encoding vector of size Nv (a hyperparameter of the network). The vectors k(t) and ((() from different channels are stacked form the matrices (1()) NN)NN+ and ((()) (Ns-
The matrix (c) is obtained by applying soft- max operation to each column of K The context encoding vectors for each joint controller are the rows (!) of the matrix ( ((t)) which are convex combinations of the features extracted from each sensory modality, weighted by their attention score (See diagram in Appendix A.T.. This attention mechanism enables each controller to focus on the relevant morphological information carried by specific sensory channels, while ignoring the irrelevant ones. Furthermore, the attention matrix is conditioned to the recent transition history, making adaptable both across episodes (and thus morphological states) and within single episode.
3.2 Distributed joint controllers
Differently from standard policy networks, we adopt independent controllers Q, i E 1,.., for each joint, implemented as distinct fully-connected networks. Each network outputs the element a; of the action vector a based on the current state (()) the previous action a (t-1) and the context encoding e Distributed action selection is natural way to enable the emergence of sensorimotor pathways from the sensory data of different body parts to the controller of a joint. As one controller needs to predict one single action value, we show in our experiments that smaller fully-connected networks sufficient to successfully solve the considered tasks, limiting the complexity overhead stemming from distributed control (See hyperparameters in Appendix A.2 Qualitatively, independent policy networks can more easily adapt their behavior when morphological perturbation affects localized area of the body, leaving the control policy of the other networks unaltered.
(t)
4 Morphological perturbation environments
We consider four locomotion environments of the PyBullet physics simulator 44 Ant and Half-Cheeee Chef-Chhetah, which are standard benchmarks for continuous control reinforeement learning
Hopper, Walker,

---

Ant
Half Cheetah
Walker
Hopper
Figure 2: Example of perturbed body configurations for the four agents, sampled the 0.5 perturbation level. The segments composing the body vary in thickness and length.
To test the ability of an agent to learn locomotion with changing body, we introduce set of morphological perturbations for each of them. The perturbations consist in modifying the size and the length of one or more body parts (Appendix A.3 Such perturbations strongly deteriorate the performance of an agent trained in the unperturbed environment [24]]. The parameter controls the size of the perturbation space C., limiting the intensity of the perturbation morphological parameter
algorithms 40 32
between -a and a. The perturbation is defined as the relative deviation from its original value. For example, perturbation of 0.5 applied to a leg thickness will make it up to 1.5 times thicker. This construction results in widely varying body shapes for each of the four environments (Figure 2
As in the unperturbed environments 32. the training is organized in episodes of most 1000 time steps. At the beginning of each episode, bodily parameters are sampled uniformly at random in the space of admissible perturbations (and fixed throughout this episode). This setup defines a Contextual Markov Decision Process [ M (S. O, T, R. 7, Ca) .M is determined by the state space S, observation function O, the action space A, the transition dynamics the reward function R, discount factor and the context C,, The observation only includes proprioceptive information, in form angular position and velocity the joints, besides other non-visual variables (orientation of the body, contact forces with the floor; contain explicit information about the current context, as identical observations can come from bodics with different parameters. On the other hand, the context
Appendix A.3). The observation space S does not
C. influences the transition dynamics p(s | St, a,, Ca), modifying the effect of an action depending on the body configuration. This condition of partial observability makes the locomotion problem considerably harder. However, confirming the intuition that contextual parameters influence the transition dynamics, we show that agent can adapt by extracting information from the movement history.
Table 1: The five policy networks analyzed the experiments. MLP: multi-layer perceptron. TCN: remporal-connolutionn network. 2-step: imitation of the Oracle morphology encoder. P. proprioceptive state. M: morphology information. H: transition history.
Simple
Oracle
RMA
TCN
DMAP
Encoder
MLP
TCN
TCN
Attention Distributed MLP
Policy
MLP
MI.P
MLP
MLP
Input
P
P+ M
P H
P+H
P H
Training
End-to-end
End-to-end
2-step
End-to-end
End-to-end
5
Experiments
We trained all the agents with Soft Actor Critic (SAC) The experiments have been run on local CPU cluster, for total of approx. 100000 cpu-hours. We used similar hyperparameters to Raffin et al. (2) which provide state-of-the-arr results in the base versions of the four PyBullet environments considered in our work (Appendix A.3). We ran every experiment for random seeds. We found that the training of the critic network benefits slightly from augmenting the input observation with the current morphological perturbation. Therefore,
B1 using the open source library RLLib (50]
5

---

included it for all policy architectures, except for the Simple agent. This addition does not pose any limitation to the deployment of the proposed algorithms, as the policy network alone is necessary at that stage.
The experiments involve five different policy networks (Simple, Oracle, RMA, TCN, DMAP), which differ in architecture, required input data and training procedure (Table DI. During training, the morphology parameters are randomly sampled from perturbation space, defined by the parameter a, at the beginning of each episode, An episode lasts 1000 time steps, unless an early termination condition is met (in case the agent falls). In this time, the agent has to learn to adapt to its unseen body configuration and run as fast as possible.
We evaluate the performance of trained agent on its ability to adapt in zero-shot manner to an unseen body morphology. To do so, we use a test set of 100 body configurations for each perturbation intensity, which were part of the body configurations used during the training phase. The agent is tested on a single episode per test morphology, in which it only observes the proprioceptive states (without any information about the body state except for the Oracle). During the testing phase, the agent does not observe the rewards it collects. We apply this testing protocol to the performance evaluation of all the trained policies and all the environments and perturbation levels. To evaluate the agent's 's ability to adapt to an unseen morphology, we consider the reward it accumulates in the complete test episode (without changing the policy; zero-shot setting). We do so for all the body configurations in the test set. Table 2 and the tables in Appendix li4 the zero-shot performance of all the trained agents in different test environments. The rewards in bold are significantly higher according to paired t-test (p- (alve 0.001). We denote the tests as IID if the 100 body configurations were extracted from the morphological space used during the training (Atrain 2 Otest), otherwise we denote them as OOD (Atrain < Otest).
5.1 A morphology encoding boosts learning in strongly perturbed environments
Simple
Oracle
Ant
Half Cheetai
Walker
Happer
2.0
2.0
= 0.1
1.0
1.0
1.0
1.0
a
0.0
0.0
0.0
0.0
Increasing complexity
Episode reward [x1K]
10
10
2
1.0
a= 0.3.3
1.0
1.0
1.0
0.0
0.0
0.0
0.0
10
5
1.0
wi WH Nile
a= 0.5
0.5
0.5
0.5
0.5
0.2
0.0
0.0
0.0
0.0
5
10
2
0
5
10
0
2
Number of episodes (x2K)
Figure 3: Learning curves of Simple and Oracle (mean + SEM across 5 random seeds). While the Simple performs competitively in low perturbation environments (a 0.1), Oracle learns better at higher perturbation levels.
To assess whether an agent can benefit from an encoding of its morphology when learning lo- comotion policy, we compare the performance achieved when providing only the proprioceptive state (Simple) and when augmenting it with the perturbation parameters (Oracle). Specifically, this Oracle follows the architecture of Kumar et al. [15]. It extracts morphology encoding from the raw perturbation vector and concatenates it to the observation vector, thus augmenting the input of the policy network and breaking the partial observability of the environment. We found that, while in mildly perturbed environments Simple performs competitively with Oracle, learning control policy solely based on the proprioceptive state becomes difficult with increasing perturbation strength (Figure and Appendix A.4 In all our experiments with {0.3, 0.5}, learning a morphology

---

representation improved performance (Table 14. This demonstrates, as expected, that learning an environment encoding is viable option to deal with strongly perturbed environments.
5.2 The morphology encoding can be regressed from experience
The morphology parameters c, E C influence the transition dynamics p(s St ar, Co) by chang- ing the physical properties of the body. Therefore, they implicitly represented in sequence of transitions, from which they might be inferred. To test this possibility, we consider a recently proposed algorithm for adaptive locomotion called rapid motor adaptation (RMA) [1]] It leverages
the latent representation of the body generated by a trained Oracle, which has access to the contextual information, to train TCN network to regress the context encoding from short history of T propri-
oceptive states and actions H(t) ((s(t-7) T), (t-T)),, (s ((t- with T 30 transitions (see Appendix A.2 for hyperparameters).
Table 2: IID performance of RMA and Oracle, mean episode reward + SEM.
Eav Algo
Ant RMA
Half Cheetah RMA
Walker RMA
Hopper
Oracle
Oracle
Oracle
RMA
Oracle
0.1
2138 + 16
2148
2197
2203
1750 28
1780 26
1859
1859 + 18
0.3
1700 + 17
1723
1402
1469 24
836
908 + 23
1267
1296 +: 26
0.5
966
16
974 1 18
595 1: 26
668 1: 24
579 17
625 1: 17
730 1 21
919 18
RMA achieves similar average reward to the Oracle (Table 2 Thus, a sequence of 30 transitions of previous proprioceptive states and actions (approx. 0.5 an embedding of the context that yields high reward. This demonstrates that RMA's two-phase adaptation procedure also viable for morphological perturbations.
experiments) is sufficient extract
DMAP (ours)
Ant
Half Cheetah
Walker
Hopper
0=0.1
1.0
1.0
0.0
0.0
0.0
S Conppeiiy
Epizode reward lx33)
100
0
0
10
0
1.0
==
1.0
1.0
0.5
1.0
0.0
1.0
1.0
0=0.3
0.5
0.5
//
0.5
0.0
0.0
0.0
0.0
5 10
0
Number of episodes (X]K)
Figure 4: Learning curves of Oracle, DMAP and TCN, the network architecture used in RMA, but trained end-to-end (mean + SEM across 5 random seeds). The performance gap between TCN and DMAP particularly clear for Ant and Walker, in which the training of TCN characterized by high variance and overall low reward. Overall, DMAP, without access to the perturbation state, matches the learning performance of Oracle in most environments, with the exception of Walker a 0.1, where Oracle is better, and Walker = 0.5, where DMAP is better.
5.3
DMAP enables end-to-end training of a morphology encoding network
Our RMA experiments demonstrated that a temporal convolutional network can regress the mor- phology encoding from sequence of proprioceptive states and actions representing 0.5 seconds of transition history. However, is unclear whether RMA's TCN encoder requires RMA's two-step training through imitation to successfully extract morphology encoding from sequence of transi-
7

---

tions. In fact, if such an encoding helps maximizing the reward, the TCN might successfully learn to extract the same representation by end-to-end reinforcement learning.
Remarkably, end-to-end training of a TCN encoder proves successful for the Hopper and the Half Cheetah. However, we encountered strong instability for the Ant and Walker across the random seeds (Figure 4. Surprisingly, in these two latter environments the addition of the TCN encoder is even harmful for the overall performance, as Simple achieves stronger results in the same environments (Appendix A.4.. Further analysis reveals that for certain random seeds, the training process is disrupted before the agent can learn how to walk (Appendix A.4
We hypothesize that this inability to learn is due to the difficulty of extracting a meaningful encoding to directly optimize the reward without suitable inductive bias. For complex morphologies, the TCN encoder might even act as a confounder for the policy network. Indeed, replacing the TCN with DMAP leads to better stability during the training (Figure 4 and general performance improvement, particularly for the Ant and Walker (Appendix A.4.. Differently from TCN, DMAP performs on par with Oracle in all the considered environments, with the exception of Walker with = 0.1. Oracle and DMAP also perform similarly when tested OOD, with a slight advantage for DMAP (Figure and Appendix A.4
A.5) Finally, we compared the adaptation speed of RMA and DMAP to unseen morphological perturbations. We found that the two algorithms perform motor adaptation equally fast (Appendix A.6 Thus, DMAP learns to robustly locomote in an end-to-end fashion, without access to the morphology parameters.
Oracie
DMAP (ours)
TCN
Hopper
Episode reward (xk
2.0
2.0
1.0
1.0
1.0
1.
0.0
0.5
0.0
0.0
-1.0
0.0
0.5
1.5
0.5
1.0
1.5 Perturbation intensity
0.5
1.0
1.0 Perturbation intensity
0.5
Perturbation intensity
Perturbation intensity
Figure 5: Reward obtained by Oracle, TCN and DMAP, trained the a 0.3 environments, on a wide range of perturbations (both IID) and OOD). Each point represents the average reward across random seeds for one body configuration, while the perturbation intensity is the Eucledian norm of the perturbation vector. The curves are obtained by fitting sigmoid function. Despite the larger perturbations falling outside of the training distribution, DMAP slightly outperforms Oracle, which receives the exact perturbation information. TCN performs worse for more challenging morphologies (Ant, Walker).
5.4 Ablation: The morphology encoding is key in statically unstable environments
To isolate the contribution of the morphology encoding on the performance of DMAP, we performed test-time ablation by evaluating the trained, distributed DMAP policy, while removing the morphology encoding. We found a performance drop for all the environments and perturbation levels (Table 3). This performance decay is especially severe in Hopper and Walker (approx. 50%), where the controller with ablated encoding fails to stabilize the agents, causing the early termination condition be met sooner and thus yielding lower episode reward. This suggests that the attention mechanism fundamental for the dynamic stability of these agents.
Table 3: IID performance of DMAP and non-encoding/ablatee (ne) DMAP, mean episode reward + SEM.
Fnv Algo
Ant
Half Cheetah
Walker
Hopper
DMAP
DMAP-ne
DMAP
DMAP-ne
DMAP
DMAP-ne
DMAP
DMAP-ne
0.1
2240
1966
2261 11
1493
1229
337
18
984
0.3
1623 19
1542 + 19
1577 + 22
1132 29
893 11
470 16
1316 + 24
748 23
0.5
960 1 14
881 1 12
669 1 23
507 1 29
743 15
341 17
953 1 15
482 20
8

---

5.5 How does attention gate sensorimotor information?
We hypothesized that DMAP enables the agent to learn which components of the input are relevant for each joint's controller. Next, we verify that such pathways emerge by analyzing the values of the attentional matrix K(*), both in the steady and dynamic condition.
The steady attentional matrix K represents the strength of each input output (i.e., sensorimotor) connection. We computed morphology-specing patterns by averaging elements across time and episodes (Figure ()) As in the spinal cord, the controllers typically attend to the sensory input of mechanically linked parts (e.g., foot controller to foot velocity, etc.). Yet, this topography is not universal: for instance the hopper; controllers focus on the leg angle and the foot velocity. Furthermore, the attention patterns seem reflect the bilateral symmetry of the Ant and Walker and front-back asymmetry due to the directionality of the locomotion task. However, the attention maps developed by DMAP vary for different seeds, indeed even gait patterns vary across seeds. Furthermore, many state and action parameters are correlated during simple locomotion. This makes interpreting why specific connections emerge in the attention maps challenging. Despite this, across seeds, we often observe bilateral and back-front symmetry. the future, we want to study this with more challenging locomotion tasks that decorrelate the states (e.g. non-flat terrains).
Ant
Walker
mut B35k Tort ba0
000
0.06
1911
GD4
LOIURIT
lift
19711
0.00
0.00
back
Dack Fam
Angs
Angg
Viledly
Angil FFOOT
Vilecty
Angel
Valecty
Angle
THION
LEG
HNEE
Half Cheetah
Hopper
+7001 +SHN THEGH
Dack
0.06
FOOT LE0 TH404
015
0.4
G10
Hiont
0.05
(bock
Velpoity
Anyie
Veluety
Viecety
THIGH
+FOOT
LEGI
RFOIC
Figure 6: Attentional matrices K, averaged across time and IID test body perturbations, represented as a heatmap for each agent, trained with a = 0.1. We can observe several interpretable patterns, e.g., the bilateral symmetry in the Ant, well front-back asymmetry (due the fixed running direction). Also in the Ant, the hip controllers mainly look at the angular velocity of the front knees and of the back hips, while the ankle controllers do the opposite. In the Half Cheetah, too, the attention map develops front-bact back asymmetry, with preference for the input angular position, and each controller mainly attends the state of the joint controls.
To analyze the temporal evolution of attention, we projected the attentional matrix K onto 3-dimensional space learned with UMAP [51] Depending on the phase of the gait, the attentional matrix (() strengthens or weakens specific sensorimotor pathways (Figure A). The graphs reveal a rotational dynamics of the gain modulation typical of biological, legged locomotion 52 53
Interestingly, in DMAP the attentional dynamics seem significantly less tangled than that of the proprioceptive and action input (Figure B). We quantified this visual impression across different environments and found robust effect (Appendix(A)) [A.7]. We speculate that this untangling mechanism might contribute to the robustness of the policy to morphological perturbations.
Discussion
We proposed DMAP, novel architecture that integrates computational principles of the sensorimotor system reinforcement learning architectures. We find that DMAP enables the end-to-end training of a RL policies in environments with changing body parameters, without having access to the

---

A
B
Observation space
La 25.47
5.
2.5
10
-2.
0.3
Attention space
-2.0
Rue 2.98
12
fo.9
Proj 2
Proj
5.0
Proj.
Proj.
Action space
-51
Quer 8.51
Proj.
Proj. 2
0.0
Proj. 12
Proj.7
Proj
7.5 5.0
Proj
Proj.
Frot.2
Figure 7: UMAP embedding of the attention matrix K, evolving during single episode for each agent. The attentional dynamics of K2 are cyclic, consistently with the periodic gait of the agents. Thus, the connectivity patterns between input channels and joint controllers change throughout the different phases of the gait. B. UMAP embeddings of the input observations O E R N input actions A(t) E R.VaxT' and attentional matrix Ke for different morphologies of the Ant. The change of body configuration causes a shift of the attentional dynamics in the low-dimensional space. The trajectories of the attention embedding seem less tangled than the corresponding input. We confirmed this impression by quantifying trajectories with a tangling metric Q (averaged over time and morphologies). Larger values correspond more tangled trajectories and the resulting Q is shown in the plots. We refer to Appendix A.7 for more technical details.
perturbation information. Thereby, DMAP matches even surpasses the performance of Oracle when tested in zero-shot manner on unseen body configurations.
Interestingly, unlike DMAP, TCN cannot be trained end-to-end to achieve robust locomotion for more complex morphologies (Section 5.3). Our work provides an example for biological principles providing excellent inductive biases for learning, consistent with the idea that innate, structured brain connectivity nurtures biological learning [54]] Furthermore, the attention mechanism of DMAP solves this challenging task by establishing morphologies. Notably, synaptic gains in the nervous system also change in a phase-dependent
gain modulation, robust across
way 52 53
Limitations and broader impact. One characteristic our current architecture design is that each joint controller observes the full proprioceptive state. A future direction will be to limit the input to the local sensory data. Such design, separating low-latency sensorimotor connection (reflexes) and slower connections distal body parts, would be closer to the locomotor system of the animals [2]. Furthermore, in our experiments, we have used an oracle critic to learn an estimate the cumulative reward and guide the training of the policy. This gave slightly better results, however, suitable inductive bias (like DMAP) or possibly inspired by the neural circuits responsible for reward prediction error, could remove this (training) limitation. Indeed using purely local connectivity, the critic does not need access to the body parameters; we will describe this in a future study.
Our research aims to facilitate the training of control policies by transferring insights from neuro- science to artificial intelligence. Powerful reinforcement learning algorithms might find application in robotics, potentially enabling, in the long term, large-scale deployment of autonomous agents. Automation comes with great opportunities, and ethical concerns 55
Acknowledgments and Disclosure of Funding
This research was supported by EPFL and the Swiss Government Excellence Scholarships to AMV. We thank Çağlar Gülçehre, Mackenzie Mathis, Ann Huang, Viva Berlenghi, Lucas Stoffl, Bartlomiej Borzyszkowski, Mu Zhou and Haozhe Qi for their feedback earlier versions of this manuscript. We also thank Viva Berlenghi for contributing to the single leg perturhation experiments.
10

---

References
[1] Andrew Biewener and Sheila Patek. Animal locomotion. Oxford University Press, 2018.
[2] Michael H Dickinson, Claire T Farley, Robert Full, MAR Koehl, Rodger Kram, and Steven Lehman. How animals move: an integrative view. science, 288(546))100-1066 2000.
[3] Françoise Leurs, Yuri P Ivanenko, Ana Bengoetxea, Ana-Maria Cebolla, Bernard Dan, Francesco Lacquaniti, and Guy A Cheron. Optimal walking speed following changes in limb geometry. Journal of Experimental Biology, 214(13):2276-22)) 2282, 2011.
[4] Ana Gonçalves, Jacob A Zavatone- Veth, Megan R Carey, and Damon A Clark. Parallel locomotor control strategies in mice and flies. Current Opinion Neurobiology, 73:102516, 2022.
[5] Roberto Leiras, Jared M Cregg, and Ole Kiehn. Brainstem circuits for locomotion. Annual review of neuroscience, 45, 2022.
[6] Matthias Wittlinger, Rüdiger Wehner, and Harald Wolf. The ant odometer: stepping on stilts and stumps. science, 312(57822 2006.
[7] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint /60.00//60000/54400 2016.
[8] Wilder Penfield and Edwin Boldrey. Somatic motor and sensory representation in the cerebral cortex man as studied by electrical stimulation. Brain, 60(4):389 443, 1937.
[9] Peter BC Matthews. Muscle spindles: their messages and their fusimotor supply. Handbook of physiology: I. The nervous system. American Physiological Society 1981.
[10] Reza Shadmehr and John W Krakauer. A computational neuroanatomy for motor control. Experimental brain research, 185(3):359-381, 2008.
[11] Eiman Azim and Kazuhiko Seki. Gain control in the sensorimotor system. Current opinion in physiology, 8: 171777887, 2019.
[12] Grace W Lindsay. Attention in psychology, neuroscience, and machine learning. Frontiers in computational neuroscience, page 29, 2020.
[13] Angelo Maravita and Atsushi Iriki. Tools for the body (schema). Trends cognitive sciences, (2):79-86, 2004.
[14] Daniel M Wolpert and Randall Flanagan. Computations underlying sensorimotor learning. Current opinion in neurobiology, 37:7-11, 2016.
[15] Ashish Kumar, Zipeng Fu, Deepak Pathak, and Jitendra Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.04034 2021.
[16] Robert Kirk, Amy Zhang, Edward Grefenstette, and Tim Rocktäschel. A survey of generalisation in deep reinforcement learning. arXiv preprint arXiv:2 1,09794, 2021.
[17] Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of Artificial Intelligence Research, 47:253-2799 2013.
[18] Yan Duan, John Schulman, Xi Chen, Peter L Bartlett, Ilya Sutskever, and Pieter Abbeel. R12: Fast reinforcement learning via slow reinforcement learning. arXiv preprint arXiv:1611.0277 2016.
[19] Marlos C Machado, Mare G Bellemare, Erik Talvitie, Joel Veness, Matthew Hausknecht, and Michael Bowling. Revisiting the arcade learning enviromment: Evaluation protocols and open problems for general agents. Journal of Artificial Intelligence Research, 61:523-561 2018.
[20] Chenyang Zhao, Olivier Sigaud, Freek Stulp, and Timothy M Hospedales. Investigating generalisation in continuous deep reinforcement learning. arXiv preprint 1902:19000000111 2019.
11

---

[21] Matthew Crosby, Benjamin Beyret, Murray Shanahan, José Hernández-Oraad Lucy Cheke, and Marta Halina. The animal-ai testbed and competition. NeurIPS 2019 competition and demonstration track, pages 164-176, 2020.
[22] Karl Cobbe, Chris Hesse, Jacob Hilton, and John Schulman. Leveraging procedural generation benchmark reinforcement learning. International conference on machine learning, pages 2048 2056, 2020.
[23] Meire Fortunato, Melissa Tan, Ryan Faulkner, Steven Hansen, Adria Puigdomênech Badia, Gavin Buttimore, Charles Deck, Joel Z. Leibo, and Charles Blundell. Generalization of re- inforcement learners with working and episodic memory. Advances in Neural Information Processing Systems, 32, 2019.
[24] Khushdeep Singh Mann, Steffen Schneider, Alberto Chiappa, Jin Hwa Lee, Matthias Bethge, Alexander Mathis, and Mackenzic W Mathis. duitof-diittibuttin generalization of internal models is correlated with reward. SelffSupervision for Reinforcement Learning Workshop-ICLR 2021, 2021.
[25] Josh Tobin, Rachel Fong, Alex Ray, Jonas Schneider, Wojciech Zaremba, and Pieter Abbeel. Domain randomization for transferring deep neural networks from simulation to the real world. 2017 IEEE/RSJ international conference on intelligent robots and systems (IROS), pages 23-30, 2017.
[26] Fereshteh Sadeghi and Sergey Levine. Cad2rl: Real single-image flight without single real image. arXiv preprint arXiv: 1.0201, 2016.
(27] Xue Bin Peng. Marcin Andrychowicz, Wojeiech Zaremba, and Pieter Abbeel. Sim-to-real transfer of robotic control with dynamics randomization. 2018 IEEE international conference on robotics and automation (ICRA), pages 3803-3810, 2018.
[28] Wenhao Yu, Jie Tan, C Karen Liu, and Greg Turk. Preparing for the unknown: Learning universal policy with online system identification. arXiv preprint arXiv:1702.02453 2017.
[29] Jane X Wang, Zeb Kurth-Nelson, Dhruva Tirumala, Hubert Soyer, Joel Leibo, Remi Munos, Charles Blundell, Dharshan Kumaran, and Matt Botvinick. Learning to reinforcement learn. arXiv preprint arXiv: |61 1.05763, 2016.
[30] Scott Fujimoto, Herke Hoof, and David Meger. Addressing function approximation error in actor-critic methods. International conference on machine learning, pages 1587-1596, 2018.
[31] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with stochastic actor. International conference on machine learning, pages 1861-1870, 2018.
[32] Antonin Raffin, Jens Kober, and Freek Stulp. Smooth exploration for robotic reinforcement 2022.
learning. Conference on Robot Learning, pages
[33] Steven Kapturowski, Georg Ostrovski, John Quan, Remi Munos, and Will Dabney. Recurrent experience replay in distributed reinforcement learning. International conference on learning representations, 2018
[34] Timothy Hospedales, Antreas Antoniou, Paul Micaelli, and Amos Storkey. Meta-learning in neural networks: A survey. IEEE transactions on pattern analysis and machine intelligence, 44 (9):5149-5169, 2021.
[35] Anusha Nagabandi, Ignasi Clavera, Simin Liu, Ronald S Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. Learning adapt in dynamic, real-world environments through meta- reinforcement learning. arXiv preprint arXiv:1803: 1347, 2018.
[36] Anusha Nagabandi, Chelsea Finn, and Sergey Levine. Deep online learning via meta- meta-learning: 2018.
Continual adaptation for model-based rl. arXiv preprint
[37] Kate Rakelly, Aurick Zhou, Chelsea Fino, Sergey Levine, and Deirdre Quillen. Efficient off-policy meta-reiinforeement learning via probabilistic context variables. pages 5331-5340. PMLR, 2019.
12

---

[38] Deepak Pathak, Christopher Lu, Trevor Darrell, Phillip Isola, and Alexei A Efros. Learning to control self celf-assemlling morphologies: study of generalization via modularity. Advances in Neural Information Processing Systems, 32, 2019.
[39] Julian Whitman, Matthew Travers, and Howie Choset. Learning modular robot control policies. arXiv preprint arXiv:2105. 2021.
[40] Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. International conference on learning representations, 2018.
[41] Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy control them all: Shared modular policies for -antt-ggootic control. International Conference on Machine Learning. pages 4455 4464, 2020.
[42] Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. My body is a cage: the role of morphology in graph-based incompatible control. arXiv preprint arXiv:2010.018366 2020.
[43] Brandon Trabucco, Mariano Phielipp, and Glen Berseth. Anymorph: Learning transferable polices by inferring agent morphology. pages 21677-21691. PMIR, 2022.
[44] Shubham Pateria, Budhitama Subagdja, Ah-hwee Tan, and Chai Quek. Hierarchical rein- forcement learning: A comprehensive survey. ACM Computing Surveys (CSUR), 54(5):1-35, 2021.
[45] Andrew Levy, Robert Platt, and Kate Saenko. Hierarchical actor-critic. arXiv preprint 171.00094,, 12, 2017.
[46] Dushyant Rao, Fereshteh Sadeghi, Leonard Hasenclever, Markus Wulfmeier, Martina Zambelli, Giulia Vezzani, Dhruva Tirumala, Yusuf Aytar, Josh Merel, Nicolas Heess, et al. Learning trans- ferable motor skills with hierarchical latent mixture policies. arXiv preprint arXiv:21 12.05062, 2021.
[47] Alexander C Li, Carlos Florensa, Ignasi Clavera, and Pieter Abbeel. Sub-policy adaptation for hierarchical reinforcement learning, arXiv preprint 19Xiv:1906.0500 2019.
[48] Erwin Coumans and Yunfei Bai. Pybullet, python module for physics simulation for games, robotics and machine learning. htttp://pybulett.tr 2016-2019.
[49] Antonin Raffin. RI baselines zoo. httpe///ggtaa..ccm//aafffi//atbt
2018.
[50] Eric Liang, Richard Liaw, Robert Nishihara, Philipp Moritz, Roy Fox, Ken Goldberg, Joseph Gonzalez, Michael Jordan, and Ion Stoica. Rllib: Abstractions for distributed reinforcement learning. International Conference on Machine Learning, pages 3053-3062, 2018.
[51] Leland MeInnes, John Healy, and James Melville. Umap: uniform manifold approximation and projection for dimension reduction. 2020.
[52] Philip Holmes, Robert J Full, Dan Koditschek, and John Guckenheimer. The dynamics of legged locomotion: Models, analyses, and challenges. SIAM review, 48(2):207-304, 2006.
[53] Wouter Hoogkamer, Frank Van Calenbergh, Stephan P Swinnen, and Jacques Duysens. Cuta- neous reflex modulation and self-induced reflex attenuation in cerebellar patients. Journal of Neurophysiology, 1 113(3)915599)) 2015.
[54] Anthony M Zador. A critique of pure learning and what artificial neural networks can learn from animal brains. Nature communications, 10(1):1-7, 2019.
[55] Lambèr Royakkers and Rinie van Est. A literature review love to war. International journal of social robotics, 7(5):549-570, 2015.
roboties: automation from
[56] Abigail A Russo, Sean R Bittner, Sean Perkins, Jeffrey S Seely, Brian M London, Antonio H Lara, Andrew Miri, Najja J Marshall, Adam Kohn, Thomas M Jessell, et al. Motor cortex embeds muscle-like commands in an untangled population response. Neuron, 9744)9553-96 2018.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section g for a discussion on
the scope and limitations of this work.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] In Section
we discuss the borader impact of our research.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] In Section we introduce the experimental setup, specifying which information can be accessed by the agents and what is not available.
(b) Did you include complete proofs of all theoretical results? [N/A] The results of this work are computational.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material as URL)? [Yes] The code is available at: https:: / /tt.tttttt....... com/ amathislabe
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section A for a description of the training setup, Section A.3 for the details about the environments and Section A.2 for the hyperparameters of SAC and of the architecture of the policies.
(c) Did you report error bars (c.g., with respect to the random seed after running exper- iments multiple times)? [Ys]] All the tables and the graphs of the learning curves report the mean value and the standard error of the mean, obtained by running each experiment with 5 random seeds.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section S
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] In Section we cite
the two main libraries used in our work: PyBullet for the physics simulation and RLLiib for the reinforcement learning experiments.
(b) Did you mention the license the assets? [Yes] In Section 4 and Section 5 we specify that the two libraries are open source.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] The code is available at: https: thgpub. com/ amat hiaathisn his wwwislab/aee /dmmp
(d) Did you discuss whether and how consent using/curating? [N/A] No data from external sources was used for this study
obtained from people whose data you's 're
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] No data from external sources used for this study
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A] No human subject was involved in this study.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] No human subject was involved this study.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] No human subject was involved in this study.
14


Reviews:
[{"review_id": "w5EsP4ouYQV", "reviewer": "Reviewer_1n2P", "summary": "- This paper studies the problem of zero shot adaptation in the context of changing bodies doing a continuous control task. \n\n- Contrary to typical RL benchmarks that train to maximize expected rewards and test on the same metric, the metric here is to evaluate the adaptation capability of the learn policies under morphological changes. \n\n- The key idea is to factor out each proprioceptive channel and learn local embeddings. An attention based architecture is then used to implement a form of dynamic gating to solve the control task. \n\n- During training, the bodies are perturbed in simulation with a fixed distribution and then evaluated on test environment with unseen values. \n\n- The key result is that this architecture is at par or even surpasses the performance of an oracle given unseen body configurations during test time. \n", "questions": "- It seems like Figure 2 varies the body w.r.t various parameters like size and shape. However, how does the approach fare under more radical transformations like broken legs? In order words, where does this approach fail? \n\n- The qualitative analysis using UMAP and the observation stated \"We speculate that this untangling mechanism might contribute\n232 to the robustness of the policy to morphological perturbations.\" is quite interesting. ", "limitations": "Discuss and perform evaluation of where the approach fails. Currently the morphological variations seems to be constrained to shape and size, but not breakage.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n- I really like the setting of this paper -- humans and animals have to learn under similar settings because the body changes everyday while the sensory motor loop still has to keep improving as the organism finds itself in new situations. \n\n- This setup will also be increasingly relevant in the robotics community as we develop robots with more dexterity and human-like hands -- because as the embodiment becomes more complicated, the robot will needs lots of iterations and fixes due to wear-and-tear, and new methods will be needed to handle to adapt to these morphological changes. \n\n- The experiments and writing are also clear. The approach is demonstrated on a set of continuous control tasks where the embodiment is structurally changed. \n\n- Figure 6 shows emergent symmetries in the body space, which has natural biological analogs. \n\nWeakness: \n\nI wonder if the authors have a tried a sequence to sequence based approach (e.g Decision Transformer) to model this problem. This would be an \"inductive-bias\" free approach. This is less of a limitations in my opinion but more of a contrast to the bias that this paper proposes. I suspect this approach would be more data efficient or could adapt quicker with more drastic body amputations.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "m6HcoLRE_yo", "reviewer": "Reviewer_t2fY", "summary": "Generalization in RL is a difficult problem. In this paper the authors investigate generalization to agents with changing morphology. This is accomplished be learning a policy that is conditioned on a history of observations and can use this information to learn a better joint attention over the parts of the state space the actuators on the robot should focus on when deciding actions. They evaluate this model against others on a task where the size and length of limbs change across a set of 4 agents. The results appear to be strong across prior methods evaluated in the paper.\n\n\n---------\n\n I have updated my score after the initial responses.\n\nI have updated my score further to reflect details covering other reviewers.", "questions": "Questions/Comments:\n\n- It does not appear to be many experimental comparisons to prior work. For example, the experimental conditions described in the paper are very similar to the ones in Pearl, a meta-learning method that infers agent morphology to be able to do some tasks quickly. and that paper, they even perform this type of morphological modification to two different agent types as one of their motivational environments for being able to adjust for quicker adaptation.\n  - Rakelly, Kate, Aurick Zhou, Deirdre Quiilen, Chelsea Finn, and Sergey Levine. 2019. “Efficient off-Policy Meta-Reinforcement Learning via Probabilistic Context Variables.” 36th International Conference on Machine Learning, ICML 2019 2019-June: 9291–9301.\n- There is related work that fits this type of experimental analysis. If the claims in the paper and analysis where more clear it is likely the problem statement in the paper would align well with these prior works. It would also be important to compare to these prior works.\n  - Nagabandi, Anusha, Ignasi Clavera, Simin Liu, Ronald S. Fearing, Pieter Abbeel, Sergey Levine, and Chelsea Finn. 2018. “Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning.” 7th International Conference on Learning Representations, ICLR 2019. arXiv. http://arxiv.org/abs/1803.11347v6.\n  - Nagabandi, Anusha, Chelsea Finn, and Sergey Levine. 2019. “Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL.” 7th International Conference on Learning Representations, ICLR 2019, 1–15.\n- It would be interesting to better understand the limitations of the attention module. Often these model require a large amount of data to train a useful representation. Some analysis to understand how many morphologies or tasks are needed to learn a generalizable representation would increase the strength of the work.\n- Additional related work\n  - Kurin, Vitaly, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. 2020. “My Body Is a Cage: The Role of Morphology in Graph-Based Incompatible Control.” arXiv [cs.LG]. arXiv. http://arxiv.org/abs/2010.01856.\n  - Trabucco B, Phielipp M, Berseth G. AnyMorph: Learning Transferable Polices By Inferring Agent Morphology.  ICML 2022.", "limitations": "Some are discussed in the paper.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Pros:\n- Learning a more end-2-end approach for estimating and generalizing to these types of changes in morphology is easier to scale.\n- The network design is sensible and should results in some interesting learned attention.\n- The focus on the 3 main indicators of a robust sensory-motor system is interesting.\n\nCons:\n- The particular motivation in the paper is somewhat confusing. Are there many scenarios where the length and thickness of agent limbs change frequently? More discussion of this motivation would help readers understand what larger problem this work is connected to. \n- I also find the experiment section somewhat unclear. The design experiments are difficult to connect back to the claims in the paper. It would be very helpful to include some additional details on the particular types of experiments and why those particular experiments are being performed. For example, it's still somewhat unclear that the type of adaptation that is being claimed in the paper is mid-episode adaptation across a changing morphology. Or if the overall idea is to be able to train a policy that can adapt to other more apologies without additional episode data given. These details are important to understand what are the best methods or most related methods to compare in the experiment section.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Bxbnj_yVBKg", "reviewer": "Reviewer_WjH3", "summary": "The authors tackle the problem of learning to locomote with morphological perturbations, e.g. when the length and the thickness of different body parts vary. They propose a biologically-inspired, attention-based policy network architecture. It combines a distributed policy, with individual controllers for each joint, and an attention mechanism, to dynamically gate sensory information from different body parts.\n\nThey ablate against two baselines: 1) A policy based on the proprioceptive state, that performs poorly with highly variable body configurations, and 2) an (oracle) agent with access to a learned encoding of the perturbation that performs significantly better. (And RMA and TCN)\n", "questions": "-", "limitations": "Yes", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n- The paper is written well and is easy to follow\n- The challenge of dealing with unknown morphological perturbations is interesting and relevant.\n- The application of RMA as a form of system identification is very helpful to compare the approach to inferred information (sysid) in addition to a baseline that uses ground truth information (Oracle)\n- DMAP convincingly outperforms all baselines\n- The visuals are great. Especially Figures 6 and 7 give an interesting intuition\n\nWeaknesses\n- The presented architecture seems similar to common approaches in hierarchical RL. A comparison seems appropriate. E.g. [R1] but I can imagine that there are quite a few approaches out there\n- Literature on hierarchical RL would be helpful\n- All methods were trained with SAC. It is hard to estimate the impact of entropy regularization on the proposed method. It would be helpful to also include DDPG/TD3\n- RMA was able to identify the morphological context in 30 transitions (0.5s). It seems like the morphological changes in the presented tasks are easy to infer. One could say that the task itself is not very hard and so the benefit of the presented approach is in question. It might be helpful to develop even harder environments up to the point where RMA fails or takes significant amount of time. How well would the presented algorithm fare?\n\n\n[R1] Rao, D., Sadeghi, F., Hasenclever, L., Wulfmeier, M., Zambelli, M., Vezzani, G., ... & Hadsell, R. (2021). Learning transferable motor skills with hierarchical latent mixture policies. arXiv preprint arXiv:2112.05062.\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "50cGMs5orCt", "reviewer": "Reviewer_ZnNr", "summary": "This paper introduces DMAP, an algorithm that allows reinforcement learning to adapt the policy to locomotion environments where the character doesn’t have a fixed body. DMAP is based on an attention-based feature encoder.", "questions": "I think most of the questions are caused by the fact that too much information is squeezed into the main paper and details are skipped. Which makes me believe that it’s quite hard to reproduce from reading the paper alone. I would perhaps suggest to put some ablations in the appendix and expand on specifying the model itself.\n\n- The fact that the character changes is equivalent with saying that the environment changes and as you mentioned this resembles the problem setting of zero-shot adaptation, or meta-learning. It would be interesting to see how methods and algorithms from the zero-shot adaptation literature trained in your environment settings compare to learning morphological changes with attention with respect to results or learning efficiency.\n\n- In the distributed joint controllers, the proprioceptive state included in the state, is the local of just that specific joint or of the whole character?\n\n- How are you producing the matrices K and V from the stack of independent representations? You call K “joint-channel attention”, is this produced via attention/transformer layers? I can’t find it specified in the description of the method.\n\n- What do cells in V represent? Is the size V a hyperparameter?\n\n- It’s hard to read figure 3 and 4 separately, I would suggest put the 4 lines together.\n\n- Regarding the above, it seems, but I might be wrong, that the oracle is worse than DMAP. From my understanding the oracle should have full observability of the environment and should perform better. Do you have explanations for this?\n\n- In section 5.4, I don’t understand how this ablation is performed. Are you removing $e_i$ from the input of each $\\omega$ ? If so, doesn’t this become equivalent to the TCN architecture?\n\n- Figure 6. Is there an explanation why the hip controller just looks at the front knee velocity and the knee controllers only the back knee velocity?\n\n- Having $N_a$ separate policies will definitely increase computation time. How does learning speed compare to a single policy method? A way to ablate the importance of distributed joint controllers could be to give the full $E^{(t)}$ to a single policy that controls the full action.\n\n- Regarding the point above, are separate policies feasible with a more complex character like humanoid? \n", "limitations": "Overall the method and the idea is really interesting. Most of my concerns are pointed out in the questions section.\n\nI am willing to update my score after my concerns and questions have been clarified. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The idea behind this work is very interesting and the method shows good results.\n- The paper motivates clearly the reasons of the architectural choices for DMAP pipeline.\n- It ablates the different components to understand the effects of the choices.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept