My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Control
Authors: Vitaly Kurin
Maximilian Igl
Tim Rocktäschel
Wendelin Boehmer
Shimon Whiteson
Venue: ICLR 2021 Poster (2021)
Keywords: Deep Reinforcement Learning
Multitask Reinforcement Learning
Graph Neural Networks
Continuous Control
Incompatible Environments

Abstract: Multitask Reinforcement Learning is a promising way to obtain models with better performance, generalisation, data efficiency, and robustness. Most existing work is limited to compatible settings, where the state and action space dimensions are the same across tasks. Graph Neural Networks (GNN) are one way to address incompatible environments, because they can process graphs of arbitrary size. They also allow practitioners to inject biases encoded in the structure of the input graph. Existing work in graph-based continuous control uses the physical morphology of the agent to construct the input graph, i.e., encoding limb features as node labels and using edges to connect the nodes if their corresponded limbs are physically connected.
In this work, we present a series of ablations on existing methods that show that morphological information encoded in the graph does not improve their performance. Motivated by the hypothesis that any benefits GNNs extract from the graph structure are outweighed by difficulties they create for message passing, we also propose Amorpheus, a transformer-based approach. Further results show that, while Amorpheus ignores the morphological information that GNNs encode, it nonetheless substantially outperforms GNN-based methods.

Published as a conference paper at ICLR 2021
MY BODY IS A CAGE: THE ROLE OF MORPHOLOGY IN GRAPH-BASED INCOMPATIBLE CONTROL
Vitaly Kurin
Maximilian Igl
Department of Computer Science University of Oxford Oxford, United Kingdom
Department of Computer Science University of Oxford Oxford, United Kingdom
vitaly. kurinkrssss...... ox. ac. uk
maximi wwm.miiiaa....m wew.eeggeegnn.... ox. ac
Tim Rocktäschel
Wendelin Böhmer
Department of Computer Science
Department of Software Technology
University College London
Delft University of Technology
London, United Kingdom
Delft, Netherlands
t. E.rocktascher
uel.ac ac
j. w. bohme
Shimon Whiteson
Department of Computer Science
University of Oxford
Oxford, United Kingdom shimon. whiteson aac.acc............o............
ABSTRACT
Multitask Reinforcement Learning is promising way obtain models with better performance, generalisation, data efficiency, and robustness. Most existing work is limited compatible settings, where the state and action space dimensions are the same across tasks. Graph Neural Networks (GNN) are one way to address in- compatible environments, because they can process graphs of arbitrary size. They also allow practitioners to inject biases encoded in the structure of the input graph. Existing work in graph-based based continuous control uses the physical morphology the agent to construct the input graph, i.c., encoding limb features node labels and using edges to connect the nodes if their corresponded limbs are physically connected. In this work, we present series of ablations existing methods that show that morphological information encoded in the graph does not improve their performance. Motivated the hypothesis that any benefits GNNs extract from the graph structure are outweighed by difficulties they create for message pass- ing, we also propose AMORPHEUS, transformer-bart based approach. Further results show that, while AMORPHEUS ignores the morphological information that GNNs encode, nonetheless substantially outperforms GNN-based methods that use the morphological information to define the message-passing scheme.
INTRODUCTION
Multitask Reinforcement Learning (MTRL) (Vithayathil Varghese & Mahmoud, 2020) leverages commonalities between multiple tasks to obtain policies with better returns, generalisation, data efficiency, or robustness. Most MTRL. work assumes compatible state-action spaces, where the dimensionality of the states and actions is the same across tasks. However, many practically impor- tant domains, such as robotics, combinatorial optimization, and object-oriented environments, have incompatible state-action spaces and cannot be solved by common MTRL approaches.
Incompatbble environments are avoided largely because they are inconvenient for function approxi- mation: conventional architectures expect fixed-size inputs and outputs. One way to overcome this limitation to use Graph Neural Networks (GNNs) (Gon et al., 2005) ( /Scarselli et al. (005) Battaglia al. 2018.. A key feature of GNNs is that they can process graphs of arbitrary size and thus, in

---

Published as a conference paper at ICLR 2021
principle, allow MTRL in incompatible environments. However, GNNs also have second key fea- ture: they allow models to condition on structural information about how state features are related, robot's limbs connected. In effect, this enables practitioners to incorporate additional domain knowledge where states are described as labelled graphs. Here, a graph is a collection of labelled nodes, indicating the features of corresponding objects, and edges, indicating the relations between them. In many cases, c.g., with the robot mentioned above, such domain knowledge is readily available. This results in a structural inductive bias that restricts the model's computation graph, determining how errors backpropagate through the network.
e.g., how how
GNNs have been applied to MTRL in continuous control environments, staple benchmark of mod- em Reinforcement Learning (RL), by leveraging both of the key features mentioned above (Wang et al. 2018: Huang et al. 2020). In these two works, the labelled graphs are based on the agent's physical morphology, with nodes labelled with the observable features of their corresponding limbs, e.g., coordinates, angular velocities and limb type. If two limbs are physically connected, there is
an edge between their corresponding nodes. However, the assumption that it is beneficial to restrict the model's computation graph in this way has to our knowledge not been validated.
To investigate this issue, we conduct series of ablations on existing GNN-based continuous control methods. The results show that removing morphological information does not harm the performance of these models. In addition, we propose AMORPHEUS,
continuous control MTRL method based on transformers (Vaswani et al. 2017) instead of GNNs that use morphological information to define the message-passing scheme. AMORPHEUS is motivated by the hypothesis that any benefit GNNs can extract from the morphological domain knowledge encoded in the graph is outweighed by the difficulty that the graph creates for message passing. state information must be communicated across multiple hops, which we hypothesise is difficult in practice to learn. AMORPHEUS uses transformers instead, which can be thought of fully connected GNNs with attentional aggregation (Battaglia et al. 2018). Hence, AMORPHEUS ignores the mor- phological domaín knowledge but in exchange obviates the need to learn multi-hop communication. Similarly, in Natural Language Processing, transformers were shown to perform better without an explicit structural bias and even learn such structures from data (Vig & Belinkov 2019
sparsely connected graph, crucial
2019 Goldberg.
Tenney et al. 2019 Peters al. 2018).
Our results on incompatible MTRL continious control benchmarks (Huang et al. 2020 Wang et al.) 2018) strongly support our hypothesis: AMORPHEUS substantially outperforms GNN based alter- natives with fixed mmessage-passing schemes in terms of sample efficiency and final performance. In addition, Amorpheus exhibits nontrivial behaviour such eyclic attention patterns coordinated with gaits.
2 BACKGROUND
We now describe the necessary background for the rest of the paper.
2.1 REINFORCEMENT LEARNING
Markov Decision Process (MDP) is a tuple (S, R. T. Po). The first two elements define the set of states S and the set of actions A. The next element defines the reward function R(s, a,,') s') with s, E S and a E A. T(s'ls, |s, a) is the probability distribution function over states s' after taking action in state The last element of the tuple Po is the distribution over initial states. Task and environment are synonyms for MDPs in this work.
A policy π (a]s) is mapping from states to distributions over actions. The goal of an RL agent is to find a policy that maximises the expected discounted cumulative return J = E E rrr,). where E [0, 1) is discount factor, Ê is the discrete environment step and r, is the reward at step t. In the MTRL setting, the agent aims to maximise the average performance across N tasks: N EN J.. We use MTRL return to denote the average performance across the tasks.
In this paper, assume that states and actions are multivariate, but dimensionality remains constant for MDP: R Vss E S, and R² Ya A. We use dim(S) lc and dim(AA) = k' to denote this dimensionality, which can differ amongst MDPs. We consider two tasks MDP, and MDP, as incompatible if the dimensionality of their state or action spaces disagree, i.e., dim(Si) *
2

---

Published as a conference paper at ICLR 2021
dim(S2) dim(AA) & dim(A2) with the subscript denoting task index. In this case MTRL policies or value functions can not be represented by a Multi-layer Perceptron (MLP), which requires fixed input dimensions. We do not have additional assumptions the semantics behind the state and action set elements and focus on the dimensions mismatch only.
Our approach, well as the baselines in this work (Wang et al. 2018 Huang et al. 2020), Policy Gradient (PG) methods (Peters & Schaal 2006). PG methods optimise a policy using gradient ascent on the objective: 01+1 = 0z + a V 6=0,, where 0 parameterises a policy. Often, to reduce variance in the gradient estimates, one learns a critic so that the policy gradient becomes VoJ
E E[ AT Vo log To [o(aa]s))]] where Aø is an estimate of the advantage function (c.g.. TD residual it + 7V - (s))).. The state-value function V (s) is the expected discounted return a policy receives starting at state |Wang et al. (2018) use PPO (Schulman et al. (017). which restricts a policy update to avoid instabilities from drastic changes in the policy behaviour. Huang et al. (2200) use TD3 (Fujimoto et al. 2018), PG method based DDPG (Lillicrap et al. 2016).
2.2
GRAPH NEURAL NETWORKS FOR INCOMPATIBLE MULTITASK RL
GNNs can address incompatible environments because they can process graphs of arbitrary sizes and topologies. A GNN function that takes labelled graph as input and outputs a graph g' with different labels but the same topology. Here, labelled graph G := E) consists of set of vertices v' E V, labelled with vectors v' E Rm and a of directed edges ed E & from vertex of to v², labelled with vectors e E mR. The output graph G' has the same topology but the labels can be of different dimensionality than the input, that is, 8t E RTd and e'M E Rmd. By graph topology we mean the connectivity of the graph, which can be represented by an adjacency matrix, binary matrix {atij whose elements aij equal to one iff there is an edge Eij & connecting vertices Us U/ E V.
GNN computes the output labels for entities of type k by parameterised update functions oi represented by neural networks that can be learnt end-to-end via backpropagation. These updates can depend on a varying number of edges or vertices, which have to be summarised first using aggregation functions that we denote p. Apart from their ability to operate on sets of elements, aggregation functions should be permutation invariant. Examples of such aggregation functions include summation, averaging and max or min operations.
Incompatible MTRL for continuous control implies learning a common policy for a set of agents with different number of limbs and connectivity of those limbs, i.c. morphology. To be more precise, set of incompatible continuous control environment is a set of MDPs described in Section 2. When state is represented as graph, each node label contains features of its corresponding limb, e.g., limb type, coordinates, and angular velocity. Similarly, each factor of an action set element corresponds to a node with the label meaning the torque for a joint to emit. The typical reward function of a MuloCo (Todorvv al. 2012) environment includes reward for staying alive, distance covered, and penalty for action magnitudes.
We now describe two existing approaches to incompatible control: NERVENET (Wagg et 2018 and Shared Modular Policies (SMP) (Huang et al. 2020).
2.2.1 NERVENET
In NERVENET, the input observations are first encoded via a MLP processing each node labels as a batch element: of + (x(x",, Ve' V. After that, the message-passing part of the model block performs the following computations (in order):
ei
& (+)) of ple je EE)
Vet E Vot V.
The edge updater of in NERVENET is an MLP which does not take the receiver's state into account. Using only one message pass restricts the learned function to local computations on the graph. The node updater of is Gated Recurrent Unit (GRU) (Cho al. 2014) which maintains the internal state when doing multiple pressage-passing iterations, and takes the aggregated outputs of the edge updater for all incoming edges as inputs. After the message- passeggggging stage, the MLP decoder takes the states of the nodes and, like the encoder, independently processes them, emitting scalars used as
3

---

Published as a conference paper at ICLR 2021
the mean for the normal distribution from which actions sampled: vaen - (,))))) Ve E V. The standard deviation of this distribution is a separate state-independent vector with one scalar per action.
2.2.2 SHARED MODULAR POLICIES
SMP is variant of a GNN that operates only on trees. Computation is performed in two stages: top-down and bottom-up. In the first stage, information propagates level by level from leaves to the root with parents aggregating information from their children. In the second stage, information propagates from parents to the leaves with parents emitting multiple messages, one per child. The policy emits actions at the second stage of the computation together with the downstream messages.
Instead of permutation invariant aggregation, the messages are concatenated. This, as well as separate messages for the children, also injects structural bias to the model, e.g.. separating the messages for the left and right parts of robots with bilateral symmetry. In addition, its message- passing schema depends the morphology and the choice of the root node. In fact, Huang et al. (2200) show that the root node choice can affect performance by 15%.
SMP trains a separate model for the actor and critic. An actor outputs one action per non-root node. The critic outputs a scalar per node as well. When updating a critic, a value loss is computed independently per each node with targets using the same scalar reward from the environment.
2.3 TRANSFORMERS
Transformers can be seen as GNNs applied to fully connected graphs with the attention as an edge- to-vertex aggregation operation (Battaglia al.] 2018). Self-attention used in transformers is an associative memory-like mechanism that first projects the feature vector of each node of E R into three vectors: query qi := Ov' E R key k == Ov' E Rd and value é, == θv' E R"" Parameter matrices O θ and ê are learnt. The query of the receiver is compared to the key value of senders using a dot product. The resulting values w, are used as weights the weighted sum of all the value vectors in the graph. The computation proceeds as follows:
softmax (ki.... kn. .a.TTa)) a.) va
w:
Ve, EV.
(I)
[0, ,
with [x1, 22, Fn. being a heads, i.e., θ θ and matrices, are used to learn different interactions between the nodes and mitigate the consequences of unlucky initialisation. The output of multiple heads is concatenated and later projected to respect the dimensions.
&xx matrix of concatenated vectors 24 E R Often, multiple attention
A transformer block is combination of an attention block and feedforward layer with possible normalisation between them. In addition, there are residual connections from the input to the atten- tion output and from the output of the attention to the feedforward layer output. Transformer blocks can be stacked together to take higher order dependencies into account, i.c., reacting not only to the features of the nodes, but how the features of the nodes change after applying transformer block.
THE ROLE OF MORPHOLOGGY IN EXISTING WORK
In this section, we provide evidence against the assumption that GNNs improve performance by exploiting information about physical morphology (Huang et al. 2020 Wang et al., 2018). Here and in all of the following sections, we run experiments for three random seeds and report the average undiscounted MTRL return and the standard error across the seeds.
To determine if information about the agent's 'ss morphology encoded in the relational graph structure essential to the success of SMP, we compare its performance given full information about the structure (morphology), given information about the structure (star), and given structural bias unrelated to the agent's 'ss morphology (line). Ideally, we would test a fully connected architecture as well, but SMP only works with trees. Figure in Appendix illustrates the tested topologies.
The results in Figure la and Tb demonstrate that, surprisingly, performance is not contingent on having information about the physical morphology. A star agent performs par with the
4

---

Published as a conference paper at ICLR 2021
5000
meppelogg
Arorags 1t.m enstrnment
5000
2500
3000
5000
1500
Acsrage retum
2000
1000
1000
1000
nEVETT:
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Tota cnvrommmnnt steps
Set
Total cnvromment stops
staps
(a) SMP, Walker++
(b) SMP, Humanoid+
(c) NERVENET, wa wales
Figure I: Neither SMP nor NERVENET leverage the agent's morphological information, or the posi- tive effects are outweighted by their negative effect message passing.
morpho 1 ogy agent, thus refuting the assumption that the method learns because exploits infor- mation about the agent's physical morphology. The 1 line agent performs worse, perhaps because the network must propagate messages even further away, and information is lost with each hop due to the finite size of the MLPs causing information bottlenecks (Alon & Yahav 2020).
We also present similar results for NERVENET. Figure Ic shows that all of the variants tried perform similarly well on Walkers from (Wang et al. 2018), with star being marginally better. Since NERVENET can process non-tree graphs, we also tested fully connected variant. This ver- sion learns more slowly at the beginning, probably because of difficulties with differentiating nodes at the aggregation step. Interestingly, in contrast to SMP, in NERVENET 1 line performs on par with morphology, This might be symptomatic of problems with the message- message-passing mechanism of SMP, e.g., bottlenecks leading to information loss.
AMORPHEUS
Inspired by the results above, we propose AMORPHEUS, transformer-based method for incompati- ble MTRL in continuous control. AMORPHEUS is motivated by the hypothesis that any benefit GNNs can extract from the morphological domain knowledge encoded in the graph is outweighed by the difficulty that the graph creates for message passing. In sparse graph, crucial state information must be communicated across multiple hops, which we hypothesise difficult to learn in practice.
AMORPHEUS belongs to the peooode-rocess--eee process-decode decode family of architectures (Battaglia et al. 2018) with a transformer at its core. Since transformers can be seen as GNNs operating on fully connected graphs, this approach allows us learn message passing schema for each state and each pass sep- arately, and limits the number of message passes needed to propagate sufficient information through the graph. Multi-hop message propagation the presence of aggregation, which could cause prob- lems with gradient propagation and information loss, is no longer required. We implement both actor and critic in the SMP codebase (Huang et al. 2020) and made our implementation available online at https: /ttp//gitthaa//a com. haan/yobbbytte. amor Like in SMP, there is no weight
encoder
transformer x
decoder
limb 2
limb
torso
Figure 2: AMORPHEUS architecture. Lines with squares at the end denote concatenation. Arrows going separately through encoder and decoder denote that rows of the input matrix are processed independently as batch elements. Dashed arrows denote message-passing in transformer block. The diagram depicts the policy network, the critic has an identical architecture, with the decoder outputs interpreted as value function values.
5

---

Published as a conference paper at ICLR 2021
sharing between the actor and the critic. Both of them consist of three parts: linear encoder, transformer in the middle, and the output decoder MLP.
Figure 2 illustrates the AMORPHEUS architecture (policy). The encoder and decoder process each node independently, as if they are different elements of a mini-batch. Like SMP, the policy network has one output per graph node. The critic has the same architecture on Figure and, in Huang et al. 2020), each critic node outputs a scalar with the value loss independently computed per node.
Similarly to NERVENET and SMP, AMORPHEUS is modular and can be used in incompatible environ- ments, including those not seen in training. In contrast to SMP which is constrained by the maximum number of children per node seen the model initialisation in training, AMORpHEuS can be applied to any other morphology with no constraints on the physical connectivity.
Instead of one-hot encoding used in natural language processing, we apply linear layer on node observations. Each node observation uses the same state representation as SMP and includes a limb type (e.g. hip or shoulder), position with a relative z coordinate of the limb with respect to the torso, positional and rotational velocities. rotations, angle and possible range of the values for the angle normalised to [o, 1]. We add residual connections from the input features to the decoder output to avoid the nodes forgetting their own features by the time the decoder independently computes the actions. Both actor and critic use two attention heads for each of the three transformer layers. Layer Normalisation (Ba et al.) 2016) is a crucial component of transformers which we also use in AMORPHEUS, See Appendix A for more details on the implementation.
4.1 EXPERIMENTAL RESULTS
We first test AMORPHEUS the set of MTRL environments proposed by Huang et al. (2020). For Wa 1 ker++, we omit flipped environments, since Huang et al. (2020) implement flipping on the model level. For AMORPHEUS, the flipped environments look identical to the original ones. Our experiments in this Section are built on top of the TD3 implementation used in Huang et al. (2020).
Figure D supports our hypothesis that explicit morphological information encoded in graph topology is not needed to yield a single policy achieving high average returns across set of incompatible con- tinuous control environments. Free from the need to learn multi-hop communication and equipped with the attention mechanism, AMORPHEUS clearly outperforms SMP, the tfate-of-the-art algorithm for incompatible continuous control. Huang et al. (2020) report that training SMP on Cheetah+t together with other environments makes SMP unstable. By contrast, AMORPHEUS has trouble learning in this regime (Figure 3g and 3h).
Our experiments demonstrate that node features have enough information for AMORPHEUS to per- form the task and limb discrimination needed for successful MTRL continuous control policies. For example, model can distinguish left from right, not from structural biases as in SMP, but from the relative position of the limb w.r.t. the root node provided in the node features.
While the total number of tasks in the SMP benchmarks is high, they all share one key characteristic. All tasks in a benchmark are built using subsets of the limbs from an archetype (e.g.. Wa ker++ or Cheetah++) +)). To verify that results hold more broadly, we adapted the Walkers benchmark (Wang et al. 2018) and compared AMORPHEUS with SMP and NERVENET on it. This benchmark includes five agents with different morphologies: Hopper, a HalfCheetah, FullCheetah, a Walker, and an Ostrich. The results in Figure 4 are consisten(] with our previous experiments, demonstrating the benefits of AMORPHEUS' fully-connected graph with attentional aggregation.
Note that the performance of NERVENET not directly comparable, as the observational features and the Icarning algorithm differ from AMORPHEUS and SMP. We do not test NERVENET on SMP benchmarks because the codebases are not compatible and comparing NERVENET and SMP is not the focus of the paper. Even if we implemented NERVENET in the SMP training loop, it is unclear how the critic of NERVENET would perform new setting. The original paper considers two options for the critie: one GNN-based and one MLP- based. We use the latter in Figure A as the former takes only the root node output labels as an input and is thus The MLP critic should perform better inefficient when the number of tasks is large. agent would need to learn 12 different critics. Finally, NERVENET leams
most likely to face difficulty in learning multi-hop pesssage-passigg because training an MLP is easier, though it might be For example, in Cheetah++ separate ML. encoder per task, partially defeating the purpose of using GNN for incompatible environments.

---

Published as a conference paper at ICLR 2021
8000
Arsaze toum aoo aionmmmen
7000
4000
3010
5000
6000
3000
3000
Arooze roum
Assrage rotu
2000
2000
1000
1500
1000
1000
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Tota env rotment steps
bet
Tats emu momment steps
bet
Tota env rotment steps
(a) Walker++
(b) Cheetah++
(c) Humanoid++
Ketragg retum acrovo enateo wnwtoooeeeeee voarte
4000
4000
2000
2000
2000
1000
1000
1000
(d) Hopper++
Walker- Humanoid++
Walker- Humanood-ooorrrr---
3000
2500
1rg
Arcroje roum acoos cnétoreneno
2500
2000
1500
1500
1000
1000
500
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0,8
1.0
Total invfrommennttteee steps
Totsl menleenttromennnoo stepe
(g) Shectah-Walker--Huannide
(h) Cheetah Walker:Humannee Hoppen-
Figure AMORPHEUS consistently outperforms SMP on MTRL benchmarks from Huang et al. (220)), supporting our hypothesis that no explicit structural information is needed to learn suc- cessful MTRL policy and that facilitated message- passing procedure results in faster learning.
While we focused on MTRL in this work, we also evaluated AMORPHEUS in zero-shot generalisa- tion setting. Table in Appendix DopendixDDrrovidee proviides initial results demonstrating Amorpheus's potential.
4.2 ATTENTION MASK ANALYSIS
GNN-based policies, especially those that use attention, are more interpretable than monolithic MLP policies. We now analyse the attention masks that AMORPHEUS learns. Having an implicit structure that is state dependent is one of the benefits of AMORPHEUS (every node has access to other nodes' annotations, and the aggregation weights depend the input as shown in Equation I By contrast, NERVENET and SMP have rigid message-passing structure that does not change throughout training or throughout a rollout. Indeed, Figure shows variety of masks a Wa ker++ model exhibits within a Wa k rollout, confirming that AMORpHeuS attends to different parts of the state space based on the input.
Both Wang al. (2018) and Huang et al. (020) notice periodic patterns arising in their models. Smilarly, AMORPHEUS demonstrates cycles in attention masks, usually arising for the first layer of the transformer. Figure 6 shows the column-wiss upper-leg limb of Wa Iker-7 agent. Intuitively, the column-wise sum shows how much other nodes interested in the node corresponding that column.
of the attention masks coordinated with an
Interestingly, attention masks in earlier layers change more slowly within a rollout than those of the downstream layers. Figure 13 in Appendix C.2 demonstrates this phenomenon for three different

---

Published as a conference paper at ICLR 2021
1.
5000
left
4000
lel2
0.6
3000
IwlS
2000
right
1000
rgte2
0.
right
0.4
0.6
right
right rignc
right
right
torso
Nit
night
righ
Figure 4: MTRL performance on Walkers (Wang et al. 2018).
Figure 5: State-dependent masks of AMORPHEUS (3rd attention
layer) within Wa Walker-7 rollout.
normallsed angle
made
Figure 6: In the first attention layer of Walker-7 rollout, nodes attend upper leg (column- wise mask sum ~ 3) when the leg is closer to the ground (normalized angle ~ 0).
Walker++ models tested on Walker-7. This shows that AMORPHEUS might, in principle, learn rigid structure (as in GNNs) if needed.
Finally, we investigate how attention masks evolve over time. Early in training, the masks are spread across the whole graph. Later on, the mask weights distributions become less uniform. Figures TO and 12 in Appendix C.1 demonstrate this phenomenon on Wa l ker-7.
RELATED WORK
Most MTRL research considers the compatible case (Rusu et al. 2016 Parisotto et al. 2016 Tch
al.) 2017 Vithayathil Varghese & Mahmoud] (2020). MTRL for continuous control often done
from pixels with CNNs solving part of the compatibility issue. DMLLab (Beattie et al., 2016 is popular choice when learning from pixels with a compatible action space shared across the environ-
ments (Hessel et al. 2019
Song et al, 2020).
GNNs started stretch the possibilities of RL allowing MTRL in incompatible environments. Khalil al. (2017) learn combinatorial optimisation algorithms over graphs. Kurin et al. (2020) learn a branching heuristic of SAT solver. Applying approximations schemes typically used RL to these settings is impossible, because they expect input and output to be of fixed size. Another form of (potentially incompatible) RL using message passing are coordination graphs (c.g. DCG, Bochmer al. 2020), that the max-plus algorithm (Pearl 1989) coordinate action selection between multiple agents. One can apply DCG in single-agent RL using ideas of Tavakoli et al.) (2021).
Several methods for incompatible continuous control have also been proposed. Chen al. (2018)
pad the state vector with zeros to have the same dimensionality for robots with different num ber of
joints, and condition the policy on the hardware information of the agent. D Bramo al. (2020)
demonstrate a positive effect of learning a common network for multiple tasks, learning a specific encoder and a decoder one per task. We expect this method suffer from sample-inefficinncy be- cause it has to learn separate input and output heads per each task. Moreover, Wang et al.) (2018) have a similar implementation of their MTRL. baseline showing that GNNs have benefits over MLPs for incompatible control. [Huang et al.) (2020), whose work is the main baseline in this paper, ap- ply a GNN-like approach and study its MTRL and generalisation properties. The method can be used only with trees, its aggregation function is not permutation invariant, and the message-passing schema stays fixed throughout the training procedure. |Wann et al. (218)) and Huang et al. (2020) attribute the effectiveness of their methods to the ability of the GNNs to exploit information about agent morphology. In this work, we present evidence against this hypothesis, showing that existing approaches do not exploit morphological information was previously believed.
8

---

Published as a conference paper at ICLR 2021
Attention mechanisms have also been used in the RL setting. Zambaldi al. (2018) consider self- attention to deal with an object-oriented state space. They further generalize this to variable action spaces and test generalisation
Starcraft-II mini- gaini that have varying number of units and other environmental entities. Duan et al. (017) apply attention for both temporal dependency and a factorised state space (different objects in the scene) keeping the action space compatible. PParisoto et al.) (2020) use transformers as a replacement for a recurrent policy. Loynd et al. (2020) use transformers to add history dependence node per game object. The authors do not consider a factored action space, with the policy receiving the aggregated information of the graph after the message passing ends. [akker et al. (2020) use self-attention to account for factored state-space to attend over objects or other agents in the scene. Amorpheus does not use a transformer for recurrency but for the factored state and action spaces, with each non-torso node having an action output. Iqbal & Sha (2019) apply attention to generalise MTRL mulli--aeem policies over varying environmental objects and Iqbal et al. (2020) extend this to factored action space by summarising the values of all agents with mixing network (Rashid et al.
POMDP as well as for factored observations, having
2020 Li et al. (2020) learn embeddings for a multi aagent actor-critic architecture by generating the weights of graph convolutional network (GCN, Kipf & Welling 2017) with attention. This allows a different topology in every state, similar to AMORPHEUS, which goes one step further and allows
to change the topology in every round of message passing.
Another line of work aims to infer graph topology instead of hardcoding one. Differentiable Graph Module (Kazi et al.] 2020) predicts edge probabilities doing a continuous relaxation of k-nearest neighbours to diffcrentiate the output with respect to the edges in the graph. [Johnson et al. (2020) learn to augment a given graph with additional edges to improve the performance of a downstream task. Kipf et al. (2018) use variational autoencoders (Kingma & Welling 2014) using GNN for reconstruction. Notably, the authors notice that message passing on a fully connected graph might work better than when restricted by skeleton when evaluated on human motion capture data.
6 CONCLUSIONS AND FUTURE WORK
In this paper, we investigated the role of explicit morphological information in graph-based con- tinous control. We ablated existing methods SMP and NERVENET, providing evidence against the belief that these methods improve performance by exploiting explicit morphological structure en- coded in graph edges. Motivated by our findings, we presented AMORPHEUS, transformer-base method for MTRL in incompatible environments. AMORPHEUS obviates the need to propagate mes- sages far away in the graph and can attend to different regions of the observations depending on the input and the particular point in training. As result, AMORpIEUS clearly outperforms existing work in incompatible continuous control. In addition, AMORpHEUS exhibits non-trivial behaviour such as periodic cycles of attention masks coordinated with the gait. The results show that information in the node features alone is enough to learn a successful MTRL policy. We believe our results further push the boundaries of incompatible MTRL and provide valuable insights for further progress.
One possible drawback of AMORPHEUS is its computational complexity. Transformers suffer from quadratic complexity in the number of nodes with growing body of work addressing this issue (Tay et al. 2020). However, the number of the nodes in continuous control problems is relatively low compared much longer sequences used in NLP (Devlin et al. 2019). Moreover, Transformers are higly parallelisable, compared to SMP with the data dependency across tree levels (the tree is processed level by level with each level taking the output of the previous level as input).
We focused on investigating the effect of injecting explicit morphological information into the model. However, there also opportunities to improve the learning algorithm itself. Potential directions of improvement include averaging gradients instead of performing sequential task up- dates, or balancing tasks updates with multi-armed bandits or PopArt (Hessel et al. 2019).
ACKNOWLEDGMENTS
VK is doctoral student at the University of Oxford funded by Samsung R&D Institute UK through the AIMS program. SW has received funding from the European Research Council under the Euro- pean Union's 's Horizon 2020 research and innovation programme (grant agreement number 637713). The experiments were made possible by a generous equipment grant from NVIDIA. The authors would like to thank Henry Kenlay and Mare Brockschmidt for useful discussions GNNs.

---

Published as a conference paper at ICLR 2021
REFERENCES
Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. CoRR, abs/2006.05205, 2020. https:::s///a/xi http://.rrrgabb/////////
2006. 05205
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization.
CoRR,
abs/1607.06444 2016. http://axxxi org/ abs / /100//160.... 110110607000000000
Bowen Baker, Ingmar Kanitscheider, Todor M. Markov, Yi Wu, Glenn Powell, Bob McGrew, and Igor Mordatch. Emergent tool use from multi-agent autocurricula. In 8th International Con- ference on Learning Represeniations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-Gonzales Vinícius Flores Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, Çaglar Gülçehre, H. Francis Song, Andrew J. Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey R. Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matthew Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases, deep learning, and graph networks. CoRR, 0tss/100600120 2018. URL httptttpp///xii/e org/ www.aa/s../..////..
Charles Beattic, Joel Z. Leibo, Denis Teplyashin, Tom Ward, Marcus Wainwright, Heinrich Küttler, Andrew Lefrancq. Simon Green, Víctor Valdés, Amir Sadik, Julian Schrittwieser, Keith Ander- son, Sarah York, Max Cant, Adam Cain, Adrian Bolton, Stephen Gaffney, Helen King, Demis Hassabis, Shane Legg, and Stig Petersen. Deepmind lab. CoRR, abs/1612.03801, 2016. URL www./oggggbbbboo.e abs/ / .08001
Wendelin Boehmer, Vitaly Kurin, and Shimon Whiteson. Deep coordination graphs. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 980-991, PMLR, 2020. URL
ngs www.rppsssss.. vl. 9/wbeh/eeeeeeaeeeeeeeeee html
Tao Chen, Adithyavairavan Murali, and Abhinav Gupta. Hardware conditioned policies for multi-robot transfer learning. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 31: Annual Conference on Neural Information Pro- cessing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, pp. 9355 9366, 2018 URL
neuriisps CC/ wtt.ccccpppppee hc/ppaerr/201/haas/ Abstrrrttt.ssttttt.t.ea....aa... html
Kyunghyun Cho, Bart van Merrienboer, Çaglar Gülçehre. Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Alessandro Moschitti, Bo Pang, and Walter Daelemans (eds.), Proceedings of the 2014 Conference on Empirical Methods in Natural Language Pro- cessing, EMNLP 2014, October 25-29, 2014, Doha, Qatar, meeting of SIGDAT, a Special Interest Group of the ACL, pp. 1724-1734. ACL, 2014. doi: 10.3115/11/114-1179, URL ptttps:/ddi..rgg/110 3et..1111wwv1cc http://.....01.11111/111111//11
1179
Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. Sharing knowl- edge in multi-task deep reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL httpt:://tppeeeeeeeeeee
net forum? id rkgpv2VVvr
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Jill Burstein, Christy Doran, and Thamar Solorio (eds.), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2019, Minneapolis, MN, USA. June 2-7, 2019, Volume 1 (Long and Short Papers), pp. 4171- 4186. Association for Computational Linguistics, 2019. doi: 10.18653/ 1/n19 1423. URL
10

---

Published as a conference paper at ICLR 2021
Yan Duan, Marcin Andrychowicz, Bradly C. Stadie, Jonathan Ho, Jonas Schneider, llya Sutskever, Pieter Abbeel, and Wojeiech Zaremba. One-shot imitation learning. In Isabelle Guyon, Ulrike von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- 1087-1098, 2017. URL
neurips. wwtpppper//1//aaaaasseee Abstract. html
Scott Fujimoto, Herke van Hoof, and David Meger. Addressing function approximation enror in actor-critic methods. In Jennifer G. Dy and Andreas Krause (eds.), Proceedings of the 35th Inter- national Conference Machine Learning, ICML 2018, Stockholmsmddssann Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pp. 1582-1591. htmll
PMLR, 2018. URLhttp::/ httpet::ppproeeeinngge http:/pooceeeggeeee ngs m0....r..... www.presss/eee Bateass/vO//uuuuuuau
Yoav Goldberg. Assessing bert's syntactic abilities. CoRR, abs/1901 2019. URL http: //arxiv. 1901 05287
Marco Gori, Gabriele Monfardini, and Franco Scarselli. new model for learning in graph domains. In Proceedings. 2005 IEEE International Joint Conference on Neural Networks, 2005., volume 2, pp. 729-734. IEEE, 2005.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojeiech Czarnecki, Simon Schmitt, and Hado van Hasselt. Multi-task deep reinforcement learning with popart. In The Thirty-Third AAAI Con- ference on Artificial Intelligence, AAAI 20/9, The Thirty-First Innovative Applications of Ar- rificial Intelligence Conference, IAAI 2019, The Ninth AAAI Symposium on Educational Ad- vances in Artificial Intelligence, EAAI 2019, Honolulu, Hawaii, USA, January 27- Febru- doi: 10. (609//aai.v33000 33110.3.37 URL
ary 1, 2019, pp. 3796-3803, AAAI Press, 2019,
org/10 1609 awwaaaai....// ......v33111111 .33.....30111.33.......33333333. 33013796
Wenlong Huang. Igor Mordatch, and Deepak Pathak. One policy to control them all: Shared mod- ular policies for agent-agnostic control. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 4455 4464. PMLR, 2020. URL http: / http:///coceeenng..... mlr. ALr.ppess/vlls httpl/////hnnn....ee h
Shariq Iqbal and Fei Sha. Actor-attentiioncciiicc for multi-agent reinforcement learning. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pp. 2961-2970. PMLR, 2019. URL http:// proccedingss.o .ww..m...........m..mm.........m www.s//vvv... vttps//a/////pb/a///aa 9a. html
Shariq Iqbal, Christian A. Schröder de Witt, Bei Peng. Wendelin Böhmer, Shimon Whiteson, and Fei Sha. AI-QMIX: attention and imagination for dynamic multi-agent reinforcement learning- 2020. URL htt.thttss::/aiiii org abs 2006. 04222
CoRR,
Daniel D. Johnson, Hugo Larochelle, and Daniel Tarlow. Learning graph structure with A finite- state automaton layer. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: An- mual Conference on Neural Information Processing Systems 2020, NewrIPS 2020, December 6-12, 2020, virtual, 2020. URL
httpppp..c.cpppee/////asass html
Anees Kazi, Luca Cosmo, Nassir Navab, and Michael M. Bronstein. Differentiable graph module (DGM) graph convolutional networks. CoRR, abs/2002.04999, 2020. URL UtRpsstss://axxi org/ ab////ba///a 2002. 04999
Elias B. Khalil, Hanjun Dai, Yuyu Zhang, Bistra Dilkina, and Le Song. Learning com- binatorial optimization algorithms over graphs. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.). Advances in Neural Information Processing Systems 30: Annual Conference on Neu- ral Information Processing Systems 2017, December 4-9, 2017. Long Beach, CA, USA, pp. 6348 6358, 2017 URL / neurips wttperrps..c..pppp/////////eeee. d9896 a6 www. ot html
Abst

---

Published as a conference paper at ICLR 2021
Diederik P. Kingma and Max Welling. Auto- Auto-encoding variational bayes. In Yoshua Bengio and Yann LeCun (eds.), 2nd International Conference on Learning Representations, ICLR 2014, Banff. AB, Canada, April 14-16, 2014, Conference Track Proceedings, 2014. U http:/ pttttp:/aaxxiv.rr.
abs/1311 1312.6666 6114
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In 5th International Conference Learning Representations, ICLR 2017, Toulon, France, 24-26 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. http:::::
//openzeview.ro net/
Thomas N. Kipf, Ethan Fetaya, Kuan-Chieh Wang, Max Welling, and Richard S. Zemel. Neural re- lational inference for interacting systems. In Jennifer G. Dy and Andreas Krause (eds.), Proceed- ings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmissun, Stockholm. Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Re- search, pp. 2693-2702 PMLR, 2018. URL http::/proccedii/ http:///w...i...pe/eeeeee/./ wwwps/ve.///.e./p kipfl8a. html
mlr
Vitaly Kurin, Saad Godil, Shimon Whiteson, and Bryan Catanzaro. Can q-learning with graph networks learn generalizable branching heuristic for a SAT solver? In Hugo Larochelle, Marc' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL 6d70cb65d 521 (226dccceec0e7771221c-----2000 html
cc/ httpp/ppppp/ppaaaaaae
Sheng Li, Jayesh Gupta, Peter Morales, Ross E. Allen, and Mykel J. Kochenderfer. Deep implicit coordination graphs for multi-agent reinforcement learning. CoRR, abs/2006. 11388, 2020. URL
https::/arxix: /ttp:p///aaxii.rr////////a hwtp:////oo.bb//..oooooo 2016 11438
Timothy Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan Puerto Rico, May 2-4, 2016, Conference Truck Proceedings, 2016. URLhttp: //arxiva org/ abs/ 1509. 02971
Ricky Loynd, Roland Fernandez, Asli Çelikyilmaz, Adith Swaminathan, and Matthew J. Hausknecht. Working memory graphs. In Proceedings of the 37th International Conference on Machine Learning, ICMI. 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 1404-6414 PMLR, 2020. URL / /ttp:://rrocceeee pttpp/ddeee.n.....
mlr. wwt.press/vlll /al/s//v11.. 1ttpss//111//ooood2aaaff
Emilio Parisotto, Lei Jimmy Ba, and Ruslan Salakhutdinov. Actor-mimic: Deep multitask and transfer reinforcement learning. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico May 2-4, 2016, Conference Track Proceedings, 2016. URL http:://aaxvvvv..oo www.//////bb//eo/ee///./ 1511. 06342
Emilio Parisotto, H. Francis Song, Jack W. Rae, Razvan Pascanu, Çaglar Gülçehre, Siddhant M. Jayakumar, Max Jaderberg, Raphaël Lopez Kaufman, Aidan Clark, Seb Noury, Matthew Botvinick, Nicolas Heess, and Raia Hadsell. Stabilizing transformers for reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning Research, pp. 7487- 7498. PMLR, 2020. html
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang. Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Judea Pearl. Probabilistic reasoning in intelligent systems -esenns-neettn networks of plausible inference. Morgan Kaufmann series in representation and reasoning. Morgan Kaufmann, 1989.
12

---

Published as a conference paper at ICLR 2021
Jan Peters and Stefan Schaal. Policy gradient methods for roboties. In 2006 JEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2006, October 9-15, 2006, Beijing, China, pp.
IEEE, 2006. doi: 10.1109/1ROS 200.0.02...00022225566 URL https://aoi.oo
10. 18.1106 1001119//110 TROS. 2006. 282564)
Matthew E. Peters, Mark Neumann, Luke Zettlemoyer, and Wen-tau Yih. Dissecting contextual word embeddings: Architecture and representation. In Ellen Riloff, David Chiang, Julia Hock- enmaier, and Jun'ichi Tsujii (eds.), Proceedings of the 2018 Conference ods Natural Language Processing, Brussels, Belgium, October 31 November 4, 2018, pp- 1499-1509. Association for Computational Linguisties, 2018. doi: 10. 10.18653/v1/018- 1179. URL
Empirical Meth-
wttp://....o..1.....////////... 1179
Tabish Rashid, Mikayel Samvelyan, Christian Schröder de Witt, Gregory Farquhar, Jakob N. Fo- erster, and Shimon Whiteson. Monotonic value function factorisation for deep multi-agent rein- forcement learning. J. Mach. Learn. Res., 21:178:1-178:51, 2020. URL WoLnttt::// / org/
papers//21/22-00B html
Andrei A. Rusu, Sergio Gomez Colmenarejo, Çaglar Gülçchre, Guillaume Desjardins, James Kirk- patrick, Razvan Pascanu, Volodymyr Mnih, Koray Kavukcuoglu, and Raia Hadsell. Policy dis- tillation. In Yoshua Bengio and Yann LeCun (eds.), 4th International Conference on Learning Representations, ICLR 20/6, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceed- ings, 2016. URL
abs 1511. 06295
Franco Scarselli, Sweah Liang Yong, Marco Gori, Markus Hagenbuchner, Ah Chung Tsoi, and Marco Maggini. Graph neural networks for ranking web pages. Andrzej Skowron, Rakesh Agrawal, Michael Luck, Takahira Yamaguchi, Pierre Morizet-Mahoudeaut Jiming Liu, and Ning TEEE/WIC/ / ACM International Conference on Web Intelligence (WI 2005), 19-22 September 2005, Compiegne, France, pp. 666 666-672. IEEE Computer Society, 2005. doi: 10.1109/W1.2005.67 URL
Zhong (eds.), 2005 IEEE /
WI. 2005.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. CoRR, abs/1 7077066777 2017. URL attp://arxiv. org/ abs/
17C7.06347
Sequence-to-Sequence Modeling, Pytorch Tutorial Sequence-lo--eqquence modeling with nn.transformer and torchtext. URL https:///pyoorcc.oo/tttttttttttt
html
[Online; accessed 8-August-2020].
H. Francis Song, Abbas Abdolmaleki, Jost Tobias Springenberg, Aidan Clark, Hubert Soyer, Jack W. Rae, Seb Noury, Arun Ahuja, Siqi Liu, Dhruva Tirumala, Nicolas Heess, Dan Belov, Martin A. Riedmiller, and Matthew M. Botvinick. V-MPO: on-policy maximum a posteriori policy opti- mization for discrete and continuous control. In 8th International Conference on Learning Repre- sentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL net/ https://cpenrevie/ http://wwwweeeeeeeiiieeeeeaa.e wow.mmiddSyl01lpaeeaee
Arash Tavakoli, Mehdi Fatemi, and Petar Kormushev. Learning to represent action values hypergraph on the action vertices. In International Conference on Learning Representations, 2021. URLLattps://ppeeeeeeeee
www.:eneeefottmteeeeee wtrf//oommmmiii/// s64FiXTv.
Yi Tay, Mostafa Dehghani, Dara Bahri, and Donald Metzler. Efficient transformers: A survey. CoRR, abs/2009.06633. 2020. URL httphttts://aaaee
org abs///////// /etss//29...0 06732
Yee Whye Teh, Victor Bapst, Wojciech M. Czarnecki, John Quan, James Kirkpatrick, Raia Hadsell, Nicolas Heess, and Razvan Pascanu. Distral: Robust multitask reinforcement learning. In Is- abelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus. S. V. N. Vish- wanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. 4496-4506, 2017. URL htwps:tpp..............tt..... http::/ppp//po.eeeeieppp ngs neurips www.rrss.cccc/ccco...oo .tmll
paper/2011 httpr//211/hahh/aaaaaaa (3redddd633a00e 105.e httpa///aaa.aaaeoa..oeeeeeeeeee.
Ian Tenney, Patrick Xia, Berlin Chen, Alex Wang. Adam Poliak, R. Thomas McCoy, Najoung Kim, Benjamin Van Durme, Samuel R. Bowman, Dipanjan Das, and Ellie Pavlick. What do you
13

---

Published as a conference paper at ICLR 2021
learn from context? probing for sentence structure in contextualized word representations, In 7th International Conference on Learning Representations, ICLR 2019, New Orleans. LA, USA, May 6-9, 2019. OpenReview.net, 2019. URL
hwtp:/wwwwwwwweeeeeeeeeeeeeeeeee www.w/rrttttttettteeemeam forum?id
SJZSgnRcKX
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model-based con- trol. 2012 JEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2012, Vilamoura, Algarve, Portugal, October 7-12, 2012, pp. 5026-5033 IEEE, 2012. doi: 10.1 109/ RKOS.2012.6386009 httt:::::://// /doi: org 1 1109/ TROS. 2012. 6386109
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp- 2017. URL https:: neurips......................... wwtpaper//01111. 20tper/201//// / 3 Abstrccccceebaaaaa....tatt....c html
Jesse Vig and Yonatan Belinkov. Analyzing the structure of attention in transformer language model. CoRR, 004444 2019. URL httthhttp///arxt./ wwwpwrwwwww.ooo..aaaoaaraaaaaaee 906 04284
Nelson Vithayathil Varghese and Qusay H. Mahmoud. A survey of multi-task deep reinforce-
ment learning. Electronics, 9()):136--1844 1384, Aug 2020. ISSN 2079-9292. doi: 10.3390/
electronics9091366 URL
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL httpp:://ppezeeiiieeeeee
net wtt.:eoorrmimmeeeegegge......
Viníciu Flores Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David P. Reichert, Timothy P. Lillicrap. Edward Lockhart, Murray Shanahan, Victoria Langston, Razvan Pascanu, Matthew Botvinick, Oriol Vinyals, and Peter W. Battaglia. Relational deep reinforcement learning. CoRR, abs/1806.01 000000083000 2018. URL http:////// //arxavavvvv///a////////aaaaaaav erg/abs/
T8C6. 01830
14

---

Published as a conference paper at ICLR 2021
A REPRODUCIBILITY
We initially took the transformer implementation from the Official Pytorch Tutorial (Sequence- lo-Sequence Modeling, Pytorch Tutorial) which uses Transformerincoderiiee torch (Paszke et al. 2017). We modified it for the regression task instead of classification, and removed masking and the positional encoding. Table provides all the hyperparameters needed to replicate our experiments.
from Py-
Table I: Hyperparameters of our experiments
Hyperparameter
Value
Comment
AMORPHEUS
Learning rate
0.0001
Gradient clipping
0.1
Normalisation
LayerNorm
As an argument to Transformerâncon in torch. nn
Attention layers
3
Attention heads
2
Attention hidden size
256
Encoder output size
128
Training
runs
3
per benchmark
AMORPHEUS makes use of gradient clipping and a smaller learning rate. We found, that SMP also performs better with the decreased learning rate (0.0001) well and we use it throughout the work. Figure demonstrates the effect of smaller learning rate on Walker++. All other SMP hyperpa- rameters are reported in the original paper with the two-directional message passing.
smp. srreiiee
vardila
4000
2000
3000
Episnde
1500
return
2000
1000
1000
500
nonronce
nervanet, ginal
0.0
0.2
0.4
0.8
1.0
0.4 Environment steps
1.0
Totall environment steps
1+7
Figure Smaller learning rate make SMP yield better results on Walker++,
Figure 8: Removing the return limit slightly de- teriorates the performance of NerveNet on Walk- ers.
Wang et al. 2018 add artificial return limit of 3800 for their Walkers environment. We remove this limit and compare the methods without it. For NerveNet, we plot the results with the option best for it. Figure compares the options.
15

---

Published as a conference paper at ICLR 2021
Table 2: Full list of environments used in this work.
Environment
Training
Zero-shot testing
Walker++
walker_3.main walker.6.main
walker 1ocer__4mme mainn walker.5.main
walker 7 ma
humanoid** +
humamo...22.7.eeft....m
hunaroid.2d7.eef._eee
bumanoid.2d.7.1owrr.....
humanoidd2d...rightt..
humanoid.28.7.rightt...
humanold.22...riihhtt...
humamoid22..@eef...keeee
humanoid 2d 9 full
Cheetah++
cheetah_2.back
cheetah.3.ba baalcee
cheetah_2.front
cheetah.5.back
cheetah.3.back
cheetah.6.front
Cheetah.3.front
chestah.4.allbacc
cheetah.4.allfront
cheetah.4.back
cheetah.4.front
theetah.5.balanced
cheetah.5.front cheetah 6 back
cheetah.7.full
Sreetah-Walkee
-Humanoid
All in the column above
All in the column above
Hopper++
hopper_3 hopper 4
hopper 5
Cheetah-Walker
-Humanoid Hooperr
All in the column above
All in the column above
Walkers from Wang et al. 20018
Ostrich
HalfCheetah
FullCheetah
Hopper
Hal HalfHumanoid
16

---

Published as a conference paper at ICLR 2021
B MORPHOLOGY ABLATIONS
Figure 9
shows examples of graph topologies we used in structure ablation experiments.
shin
upper
upper arm
thigh
upper
lower arm
upper arm
upper arm
lower arm
lower arm
shin
lower
torso
lower torso
shin
thigh
thigh
thigh
upper arm
shin
shin
thigh
thigh
shin
torso
lower arm
(a) Morphology
(b) Star
(c) Line
Figure Examples of graph topologies used in the structure ablation experiments.
17

---

Published as a conference paper at ICLR 2021
C
ATTENTION MASK ANALYSIS
C.1 EVOLUTION OF MASKS THROUGHOUT THE TRAINING PROCESS
Figures and
demonstrate the evolution of AMORPHEU'S attention masks during training.
1:12-
1oh2
k0h2
1:13
Ieh3
knt
1g142
righ2
13143
right
bro0
1M3
1913
Iro
lnn2
1942
troo
Figure 10: Walker+ masks for the attention layers on Walker- at the beginning of "training.
kh1
42141
figh1
16122
ngh2
ngn
rith3
Jet2
MR3
rgh33
MR2
Mn2
AH2
MR3
rukh
Figure 11: Walker+ masks for the 3 attention layers on Walker-7 after 2.5 mil frames.
tono
1o:00
tor32
loft
ldtt
InR3
nght
righ2
43713
righ33
4122
1000
Figure 12: Walkert+ masks for the attention layers Walker-7 at the end of training.
18

---

Published as a conference paper at ICLR 2021
C.2 ATTENTION MASKS CUMULATIVE CHANGE
2500
2500
2000
2000
Canustvv clarge
2000
1500
Camusttue
1500
1500
1000
1000
500
500
500
200
4.0
1002
200
400
830
803
1000
203
400
600
830
1200
Figure 13: Absolutive cumulative change in the attention masks for three different models on Walker-7.
D GENERALISATION RESULTS
Table 3: Initial results on generalisation. The numbers show the average performance of three seeds evaluated on 100 rollouts and standard error the mean. While the average values are higher for AMORPHEUS on 5 out of 7 benchmarks, high variance of both methods might be indicative of instabilities generalisation behaviour due large differences between the training and testing tasks.
AMORPHEUS
SMP
wa walker-3-ma in walker-6-main
666.24 (133.66)
175.65 (157.38)
1171.35 (832.91)
729.26 (135.60)
humanooidt207--eeff--egg
2821.22 (1340.29)
2158.29 (785,33)
humano1d@2d4--rrgght--ee
2717.21 (624.80)
327.93 (125.75)
cheetaah-3-balanceed
474.82 (74.05)
156.16 (33.00)
cheetah-5-back
3417.72 (306.84)
3820.77 301,95)
cheetah-6-front
5081.71 (391.08)
6019.07 (506.55)
E
RESIDUAL CONNECTION ABLATION
We use the residual connection in AMORPHEUS safety mechanim prevent nodes from for- getting their own observations. To check that AMorpheus's improvements do not come from the residual connection alone, performed the ablation. As can Figure 14] cannot tribute the success of our method to this improvement alone. High variance on Humanoide is related the fact that one seed started improve much later, and the average performance suffered as the result.
19

---

Published as a conference paper at ICLR 2021
8000
6000
7000
5000
6000
2500
5000
3000
4000
3000
2000
fureage
1000
Seaaphous condtiiiing
1000
0.0
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Toaal unvirommentt stops
Total unviromment steps
Total enviromment steps
(a) Walker++
(b) Cheetah++
(c) Humanoid++
Aaerage retum acroe 4mt nummmnns
arercheua cPp
4000
4000
6mp
3000
3000
3300
2000
2000
2000
1000
1000
ema condtiiing
0.0
0.2
0.4
0.6
0.8
1.0
0.0
0.2
0.4
0.6
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Tota endirnmmett steps
Toral enviromme steps
Total row.romennntttee
(d) Hopper++
(e) Walker Humanoid++
(l) Hww.--Huumannngoeeeeeeee
Average retum across envirorments
3000
Average emuirorments
2500
amorpheus
amorpheus, conditaning smp
amorpheus, conditioning simp
2500
2000
2000
1500
1500
1000
1000
500
500
0.2
0.4
0.6
0.8
1.0
0.2
0.4
0.6
0.8
1.0
Total environment steps
Total environment stops
(g) Cheetah- Seeethh-Wallkr--Hummnikke
(h) Cheetah Walker Humanoid Hlumanoid-Hopperr--
Figure 14: Residual connection ablation experiment.
20


Reviews:
[{"review_id": "nFtIICKMaWE", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work considers continuous control environments in which each agent limb (actuator) is associated with one action and a set of observation factors. As in prior work, the proposed policy class is modular, where each module is mapped to one limb (or the root), and the modules share information through some GNN message-passing schedule. The experimental results indicate that fully connected, transformer-style message passing is more effective in this setting than message passing restricted to directly connected pairs of limbs.\n\nPros\n- Good framing of the problem and choice of experiments.\n- Insightful discussion of related work.\n\nCons\n- There is no discussion of the additional computational requirements of transformers over SMP.\n- The results would be stronger if hyperparameters had been systematically tuned.\n\nQuestions\n- Why were no results provided for the Walker-Hopper or Walker-Hopper-Humanoid combinations tested by Huang et al?\n\nSuggestions\n- The paper mentions in passing that this work involves agents “with each non-torso node having an action output”. This limitation probably deserves to be highlighted more prominently. \n\n- As the paper explains, “transformers can be seen as GNNs operating on fully connected graphs”. In other places, the paper contrasts transformers with GNN-based methods (“substantially outperforms GNN-based methods”), as if transformers were not GNNs. To avoid confusing readers, it would help to explain that GNNs are a broad class that includes both transformers and SMP, which differ in their message passing schedules, etc. \n\n- It would help to have a more detailed description of the limb torques and observation factors, so that readers don’t have to look in Huang et al (section 4) for these details.\n\n- The following phrases are important but unclear for readers who are not very familiar with GNNs or transformers:  “Having an implicit structure that is state dependent is one of the benefits of AMORPHEUS.”  and  “the implicit state-dependent message-passing schema learnt by AMORPHEUS can be better”\n\n- The paper says “We use entity to denote both vertices and edges.” But the term “entity” appears nowhere else in the paper.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "iYlL6s5z1jV", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes that recent methods that used graphical neural networks to help solve the multitask reinforcement learning problem and assume that there's an advantage from being able to encode the agent's morphology using a graphical neural network do not provide additional generalization and benefits for learning. Instead, they claim that the benefits from being able to encode this morphology are counteracted by the difficulty in having to train the graphical neural network using the message passing system. This paper instead proposes to use Transformers as a simpler mechanism to be able to train and discover the helpful morphological distinctions between agents in order to better solve multitask reinforcement learning problems.\n\nThe motivation that graphical neural networks are bogged down by their message passing framework is not necessarily a motivation for using transformers. There needs to be a separate motivation for why you want to use transformers and why they should perform better than GNNs or normal networks.\n\nThe author claims that the SMP paper does not work better due to the morphology encoding and then they point out that it instead works because of the encoding of the subtrees and some specific detail related to message passing. This could be correct the explanation but the paper so far hasn't gone into enough detail for the reader to understand the importance of this message passing and how it works and how it is not improving training for GNNs.\n\nFigure one does provide some information related to how the transformer is used with respect to some morphology it would be far more helpful if this figure method was described well enough so that anyone can understand how to apply this to a different morphology. One of the challenges with reading and understanding this paper is the lack of information on how graphical neural networks are used and designed to understand later comments in the paper.\n\nThere needs to be an ablation with respect to the residual connections added to the Transformer based network to make sure the improvement for amorphous is not working well just because of these residual connections.\n\nWhile I do agree that training a graphical neural network to be able to produce a quality policy for a number of control tasks from the opening item environment is difficult the author of the paper might be missing at least one of the key points from the previous work in that you can learn a stronger modularization of policy. And that a goal of the SMP work was to understand how more modular policies or policies with modular components could be learned.\n\nIt is stated in the paper that amorphous does better for state of the art incompatible continuous control? What is meant by incompatible continuous control? This term has not been defined anywhere in the paper and without this definition, it's difficult to understand the contribution this paper is making.\n\n----- Post Discussion ----\nI have updated my rating for the paper after the authors have provided additional discussion and experiments to address my concerns.", "extra_scores": {}, "extra_text": {}}, {"review_id": "E053xHFPd0b", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper focuses on the problem of multi-task control with a shared policy in the continuous action setting. Unlike current assumption of compatible state-action spaces, the proposed architecture is transferable across different morphologies. The paper includes ablation experiments that clearly show that current works that use the body morphology structure to constrain the graph structure of graph neural network based approaches do not actually improve the performance. The paper instead forgoes trying to input the body structure and uses a transformer based architecture that is capable of learning the appropriate (even dynamic) graph structure actually useful for control.\n\nI enjoyed the relatively simple experiments showing how the specific graph struture based on body morphology wasn't important at all. Although would be useful to know how many runs were performed given the noisy nature of RL.\nSimilarly the cyclical structure noticed in Fig. 6 definitely points towards the powerful nature of transformer architectures at learning this relations. The strong performances (none of which seem to have converged yet) compared to baselines speak for themselves. Although, again not clear about the number of seeds the experiments were repeated.\n\nThe architecture description is somewhat unclear. Both actor and critic seem to have three parts to their architecture. But for a critic you need to output value information which might be scalar unlike decoder MLPs for independent node action. If there is some sort of aggregation going on, it needs to be clarified as to specifically how.\nAlthough Fig. 5 shows changing attention patterns, it doesn't warrant the confirmation that the proposed architecture benefits from \"state-dependent message passing of transformers\" which itself consists of two things. One can do state-dependent message passing in such architectures without the transformers (see DICG [1] for an example with attention and graph convolutions). Second, there could be a static graph structure which is better than the dynamic masks: the paper didn't actually perform the experiments to rule that out. Maybe the obvious morphology is the wrong graph structure but there is something else which would work better.\nAgain, the paper's claim is probably true, but the causal language is not justified from Fig 5.\n\n[1] https://arxiv.org/abs/2006.11438\n\nEdit: Updated score to reflect the changes from the revision.", "extra_scores": {}, "extra_text": {}}, {"review_id": "7tzEXs-rfc", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The manuscript studies the usefulness of Graph Neural Networks (GNN) in incomplete environments for Multitask Reinforcement Learning (MTRL). First, authors explore to what extend morphology information improves performance in GNN. By use of the Shared Modular Policies and the NerveNet methods, authors find that restricting morphology information does not improve performances. Based on this finding, authors apply GNN to fully connected graphs with memory/attention, i.e. the use transformers. Simulation results in different environments show that the proposed approach outperforms conventional methods.\n\nThe paper is well written, and methods and analysis approach used are clear. The selected approach and the authors findings are meaningful in a neuroscientific context, as the transformers approach better resembles the function of a human brain. The contribution is original and relevant for the community.\n\nThe introduction does not clearly explain why the assumption that restricting the model and encoding morphological information may be beneficial. A weakness of the manuscript is the assessment of the results. No statistical tests were calculated to support the authors claims, nor is it clear how the performance of the different methods was compared. For example, the comparison f the curves depicted in Figure 3. It is also not clear to the reviewer what specifically the highlighted areas in the figures represents? Confidence intervals or standard deviation? Moreover, a few statements are vague. For example, what is a “good MTRL policy” (page 6).\n\nOverall an interesting approach that is expected to improve performance in MTRL and needs further exploring.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper shows that using graph neural networks to address multi-task control problems with incompatible environments does not provide benefits to the learning process. The authors instead propose to use Transformers as a simpler mechanism to be able to train and discover the helpful morphological distinctions between agents in order to better solve multitask reinforcement learning problems.

The paper is well written and the analysis of the literature has been appreciated.  The contribution is original and relevant to the community.

All the reviewers agree that this paper deserves acceptance. We invite the authors to modify the paper by following the suggestions provided by the reviewers. In particular:
- improve the analysis of the empirical results
- update the plots
- add the suggested references
Decision: Accept (Poster)