Feudal Graph Reinforcement Learning
Authors: Tommaso Marzi
Arshjot Singh Khehra
Andrea Cini
Cesare Alippi
Venue: Accepted by TMLR ()

Abstract: Graph-based representations and message-passing modular policies constitute prominent approaches to tackling composable control problems in reinforcement learning (RL). However, as shown by recent graph deep learning literature, such local message-passing operators can create information bottlenecks and hinder global coordination. The issue becomes more serious in tasks requiring high-level planning. In this work, we propose a novel methodology, named Feudal Graph Reinforcement Learning (FGRL), that addresses such challenges by relying on hierarchical RL and a pyramidal message-passing architecture. In particular, FGRL defines a hierarchy of policies where high-level commands are propagated from the top of the hierarchy down through a layered graph structure. The bottom layers mimic the morphology of the physical system, while the upper layers correspond to higher-order sub-modules. The resulting agents are then characterized by a committee of policies where actions at a certain level set goals for the level below, thus implementing a hierarchical decision-making structure that can naturally implement task decomposition. We evaluate the proposed framework on a graph clustering problem and MuJoCo locomotion tasks; simulation results show that FGRL compares favorably against relevant baselines. Furthermore, an in-depth analysis of the command propagation mechanism provides evidence that the introduced message-passing scheme favors learning hierarchical decision-making policies.

Published in Transactions on Machine Learning Research (12/2024)
Feudal Graph Reinforcement Learning
Tommaso Marzi
Coramaso.mariiuusi..d
Università della Suizzero italiana, IDSIA
Arshjot Khehra
arshjjot.hhclomouss.dd
Universitd della Svizzero italiana
Andrca Cini
Universitd della Svizzero italiana, IDSIA
Cesare Alippi
cesare.alippi@usi.cm ch
Universitd della Svizzero italiana, IDSIA Politecnico di Milano
Reviewed on OpenReview: https: openreview. net/ forum? id=
Abstract
Graph-based based representations and message- saasggggasiinggggg modular policies constitute prominent ap- proaches to tackling composable control problems in reinforcement learning (RT.). However, as shown by recent graph deep learning literature, such local message-passing operators can create information bottlenecks and hinder global coordination. The issue becomes more serious in tasks requiring high-level planning. Tn this work, we propose a novel methodol- ogy, named Feudal Graph Reinforcement Learning (FGRL), that addresses such challenges by relying on hierarchical RL and a pyramidal message passsggg-assinggggg architecture. In particular, FGRL defines a hierarchy of policies where high-level commands are propagated from the top of the hicrarchy down through layered graph structure. The bottom layers mimic the morphology of the physical system, while the upper layers correspond to higher-order sub-modules. The resulting agents are then characterized by committee of policies where actions at a certain level set goals for the level below, thus implementing a hierarchical decision-making structure that can naturally implement task docomposition. We evaluate The proposed framework on a graph clustering problem and MuJoCo locomotion lasks; simu- lation results show that FGRL compares favorably against relevant baselines. Furthermore, an in-depth analysis of the command propagation mechanism provides evidence that the introduced message- message-passing scheme favors learning hierarchical decision-making policies.
1 Introduction
Although reinforcement learning (RL) methods paired with deep learning models have recently led to out-
standing results, e.g., see (Silver et al. 2017 Degrave et al. 2022 Wurman et al. 2022 Ouyang et al, 2022 achievements have come with a severe cost terms sample complexity. A possible way out foresees em- bedding inductive biases into the learning system, for instance, by leveraging the relational compositionality of the tasks and physical objects involved (Battaglia et al. 2018 Hamrick et al. 2018 Zambaldi et al. 2018 Funk al. 2022a). Within this context, physical systems discrete entities interconnected by binary relationships. In such cases, graphs emerge as suitable repre- sentation to capture the system's underlying structure. When processed by message-passingg graph neural networks (GNNs) [Gilmer et al. 2017 Bacciu et al.) 2020 Bronstein et al., 2021), these graph representa- tions enable the reuse of experience and the transfer models across agents (Wang al.) (0188:: node-level modules (policies) can easily be applied graphs (systems) with different topologies (structures). However, while conceptually appealing, the of modular message-passing policies also bears some concerns in terms
be often decomposed into collection of

---

Published in Transactions on Machine Learning Research (12/2024)
of the constraints that such representations impose the learning system. As example, consider robotic agent: natural relational representation can be obtained by considering a graph capturing its morphology, with links represented as edges, and different types of joints (or limbs) as nodes. In this framework, policies can be learned at the level of the single decision-making unit (actuator, in the robotic example), in a dis- tributed fashion (Wang et al. 2018: Huang et al., 2020). Nevertheless, recent evidence suggests that existing approaches are not successful in learning composable policies able to fully exploit agents' morphology (Kurin) et al., 2020). Notably, in such modular architectures, the same replicated processing module must act as a low-level controller (e.g., by applying a torque to a joint of a robotic leg), but, at the same time, attend to complex, temporally extended tasks at the global level, such as running or reaching far-away goal.
In light of this, we propose to tackle the problem of high-level coordination in modular architect intro- ducing a novel hierarchical approach to designing graph-based message- message-passing policies. Our approach is inspired by the guiding principles of hierarchical reinforcement learning (HRL) (Barto & Mahadevan, 2003) and, in particular, feudal reinforcement learning (FRL) (Dayan & Hinton 1992): we argue that GNNs are natural candidates to implement HRL algorithms, their properties implicitly account for HRL desiderata such as transferability and task decomposition. Indeed, one of the core ideas of HRL is structuring the learning systems to facilitate reasoning at different levels of spatiotemporal abstraction. The options frame- work (Sutton et al., 1999). as an example, pursues temporal abstraction by expanding the set of available actions with temporally extended behavioral routines, implemented as policies named options. Conversely, FRL implements spatiotemporal abstractions by relying on a hierarchy of controllers where policies at the top levels set the goals for those at lower levels. In the FRL framework, each maaisnn unit is seen as manager that controls its sub-managers and, in turn, is controlled by its super-manager. In its original formulation, FRL is limited to tabular RL scttings and relics on domain knowlodge for the dcfinition of subtasks and intermediate goals.
Although extensions to the deep RL settings exist (Vezhnevets et al. 2017), the idea of learning a hierarchy of communicating policies has not been fully exploited yet. In this paper, we rely on these ideas to propose a novel hierarchical graph-based methodology, named Feudul Graph Reinforcement Learning (FGRL), to build hierarchical committees of composable control policies. In FGRL, policies are organized within a feudal, i.e., pyramidal, structure, and each layer corresponds to graph. Messago-passing GNNs provide the proper compoaccmtatttionall framework to implement the architecture. More in detail, the hierarchy of controllers is represented as multilayered graph where nodes at the bottom (workers) can be seen as actuator-level controllers, while upper-level nodes (sub-managers and manager) can focus on high-level planning, possibly by exploiting higher-order relationships. In the robotic example, workers would correspond to the agent's 's joints and the corresponding actuators, i.e., to those entities directly applying a control action to the system. Conversely (sub-) mmangerns would correspond to modules responsible for coordinating subordinate nodes by sending appropriate (possibly high-level) commands. Depending on their role in the hierarchy, such commands might specify, for instance, target position for a specific joint control the orientation of a body part. In FGRL, each discrete decision-making unit is responsible for the control of the system at a certain level. In particular, each node participates in lcarning policy by exchanging messages with controllers of the same levels and setting local goals for the level below. While exchanged messages enable coordination, the hierarchical pyramidal structure constrains the information flow, implementing information hiding. As a result, the system is biased towards learning a hicrarchical decomposition of the problem into sub-tasks.
To summarize, our main novel contributions are as follows.
1. We introduce the FGRL paradigm, a new methodological deep learning framework for graph-based HRL in composable environments (Scc. 4h.
2. We evaluate possible implementation of the proposed method on graph clustering problem and on composable continuous control tasks from the MuJoCo locomotion benchmarks (Todorov et al., 2012), where the proposed approach obtains competitive performance w.r.t. relevant baselines for composable control Sec.. 5.2 and 5.3
3. We provide empirical evidence that supports the adoption of hicrarchical pressage-passine schemes and graph-based representations to implement hierarchical control policies (Sec. 5.2 and 5.3
2

---

Published in Transactions on Machine Learning Research (12/2024)
Our work paves the way for novel take on hierarchical and graph-based reinforcement learning, marking significant step toward designing deep RT architectures incorporating biases aligned with the structure of HRL agents.
The paper is structured as follows. Sec. 2 discusses related works. The FGRL framework is introduced in Sec. and empirically evaluated Sec. 5
Sec. reports final considerations and discusses directions for
future works.
2 Related Works
Several RL methods rely on relational representations. Zambaldi al. (2018) embed relational inductive biases into a model-free deep RL architecture by exploiting the atlention mechanism. /Sanchez-Gonzalee/ et al. (0188) use GNNs to predict the dynamics of simulated physical systems and show applications of such models in the context of model-based RL. Other works adopt GNNs in place of standard, fully-connected, feed-forward networks to learn policies and value functions for specific structured tasks, such as physical construction (Hamrick et al.] 2018 Bapst et al., 2019). Moreover, GNNs have been exploited also in the context of multi-agent systems (Jiang et al. 2020) and robotics (Funk et al.) 2022a b). Ha & Tang (2022)
provide an overview of deep learning systems based on the idea of collective intelligence, i.c., systems where the desired behavior emerges from the interactions of many simple (often identical) units.
More related to our approach, NerveNet (Wang et al. 2018) relies message passing to propagate infor- mation across nodes and learn actuator-level policy. Similarly to NerveNet, the shared modular policies (SMP) method (Huang et al. 2020) learns a global policy that is shared across the limbs of a target agent and controls simultancously different morphologics. The agents' structure encoded by a tree where an arbitrary limb acts as a root node. Information propagated through the tree in two stages, from root to leaves and then backward, Kurin al. (2020), however, show that constraining the exchange of information to the structure of the system being controlled can hinder performance. This issue is a well-known problem in graph machine learning: the best structure to perform message passing does not necessarily correspond
to the input topology (Toppngg et al. 2022 Rusch et al.] 2023 Graph pooling (Ying et al. 2018 Bianchi
et al. 2020 Grattarola et al.) 2022
Bianchi & Lachi 2024 tackles this problem by clustering nodes and
rewiring the graph to learn hicrarchical representations. Our work can be scen as introducing a similar idea in graph-bases based RL. Moreover, in FGRL, the hierarchical structure corresponds to the structure of the decision- making process. A different research line on composable RL leverages Transformers (Vaswani 2017) to learn universal policies designed for handling multiple morphologies simultaneously (Trabucco et al.] 2022
Gupta
et al. 2022 Xiong et al. 2023 possibly encoding geometric symmetries in the model (Chen al. 2023
In this context, Xiong et al. (2024) propose approach to improve the efficiency of existing multi-task R.
methods by generating policies using morphology-condttooned hypernetwork (Ha et al. 2017).
FRL Dayan & Hinton 1992) has been extended to the deep RL. setting with the introduction of FeUdal Networks (FUN) (Vezhnevets et al.] 2017) and explored in the context of multi-agent systems (Ahilan & Dayan 2019). However, none of the previous works match FRL with hicrarchical graph-based architecture to learn modular policies, as we do here instead.
3 Preliminaries
Markov decision processes A Markov decision process (MDP) (Sutton & Barto 2018) is tuple AS,A,P, where S R is state space, A C R is action space, P S x A > S is Marko- vian transition function and R S x - R is a payoff (reward) function. We focus the episodic RL setting where the agent acts in the environment for a fixed number of time steps or until it receives a ter- mination signal from the environment. The objective is to learn a parameterized stochastic policy To that maximizes the total expected reward received in an episode. We focus on environments where the state representation can be broken down into sub-parts, each part mapped to a node of a graph structure.
Graphs and passaggggpsssinggg.oo neural networks A graph is a tuple g (V,E), where V and denote the set of vertices (nodes) and links (edges), respectively. In attributed graphs, each node i is equipped
3

---

Published in Transactions on Machine Learning Research (12/2024)
with an attribute (or feature) vector æ; E R Similarly, edges can be associated with attribute vectors Eij E IR where (i,j) indicates the edge connecting the i-th and the j-th nodes. Message-passing neural networks (MPNNs) (Gilmer et al,, 2017) encompass a large variety of GNN architectures under the same general framework. In particular, in MPNNs, representations associated with each node are updated at cach round by aggregating messages from its neighbors. More precisely, represenlation at of node at round with neighbors (ii) updated as:
x 1+ -+1 =
-c- (dd AGOR (t) (EN JEN(()
(1)
where AGGR is permutation-invarine aggregation function, while ç' and ø¹ are, respectively, differentiable update and message functions, e.g., implemented by multilayer perceptrons (MLPs).
4 Feudal Graph Reinforcement Learning
A physical system can often be described a set of entities and relationships among these entities. As a case study, we consider structured agents (simplified robots), as those typically used in continuous control RL benchmarks (Tassa et al. 2018). Such agents are modeled as made of several joints and limbs that correspond to actuators of a system to control. The following section provides a method for extracting and exploiting hierarchical graph-based representations RT.
4.1 Graph-based Agent Representation
A structured agent with K limbs can be repre- sented as an undirected graph G1. The subscript denotes the level in the hierarchy and will be con- textualized in the next subsection. In this setup, each i-th limb with E (l...., .....,K K} is mapped to node whose feature vector st S contains in- formation regarding its state at time (e.g., the position, orientation, and velocity of the node); an edge between two nodes indicates that the cor- responding limbs are connected. Each limb can be paired with an actuator and outputs are asso- ciated with control actions at E A. Limbs with associated action act auxiliary hub nodes and simply account for the morphology of the agent; in practice, representations of such nodes are simply not mapped to actions and are discarded in the final processing steps. Fig. provides an example of the graph extraction process for the 'Humanoid" environment from MuJoCo, in which the torso node acts as simple hub for message passing.
Morphology extraction
Graph extraction
Lew emn ID
Low
Thightt mi
Original agent
Agent morphology
Agent graph 9t
Figure 1: Constructing the agent graph 91 for 'Humanoid' environment. Blue squares in the agent's 's morphology rep- resent the joints of the agent and are not mapped to nodes, differently from the green labels which, instcad, refer to the limbs and constitute the nodes of G1-
4.2 Building the Feudal Hierarchy
The core idea of FGRL consists of exploiting a multi-level hierarchical graph structure 9" to model and control a target system by leveraging a pyramidal decision-making architecture. this framework, each i-th node in the hicrarchy reads from the representation of subordinate (child) nodes C(i) and assigns them goals; in turn, it subject goals imposed by its supervisor (parent) nodes P(i) through the same goal- assignment mechanism. Each node has no access to state representations associated with the levels above, as the structure is layered. We identify three types of nodes:
Manager: It single node the highest level of the hicrarchy, i.c., has no supervisor. It receives the reward directly from the environment and is responsible for coordinating the entire hierarchy.
4

---

Published in Transactions on Machine Learning Research (12/2024)
2. Sub- These nodes constitute the intermediate levels of the hierarchy and, for each level I,, they can interact among each other through message passing the associated graph 9t,- Each sub-manager receives an intrinsic reward based on the goals set by its direct supervisors.
3. Workers:
These nodes are at the lowest level of the hierarchy. Workers correspond to nodes in the previously introduced base graph G1 and are in charge of directly interacting with the environment. At each time step t, the i-th worker has access to state vector s and receives an intrinsic reward that a function of its assigned goals.
This feudal setup allows for decomposing the original task into simpler sub-tasks by providing goals different scales. In particular, increasing the hierarchy depth L, increases the resolution at which goals are assigned: upper levels can focus high-level planning, while lower levels can set more immediate goals. The resulting hicrarchical committee facilitates temporal abstraction, which can be further promoted by letting each level of the hierarchy act al different. temporal scales (Vezhnevets et al., 2017). In practice, we can keep the commands set by the higher levels fixed for a certain (tunable) number of steps. Depending on the sctup, sub-managers and workers can rely on intrinsic rewards alone (reward hiding principle: Dayan & Hinton (1992) or maximize both intrinsic and extrinsic reward the same time (Vezhnevets et al., 2017].
The hierarchical graph G* can be built by clustering nodes of the base graph G1 (with nodes possibly belonging to more than one cluster) and assigning each group to sub-manager. The same clustering process can be repeated to obtain a single manager at the top of the hierarchy; see Fig. 2 (left) and Fig. for reference. We remark that our study focuses on composable control problems in structured environments, i.e, where agents can be decomposed into interconnected functional units. In this setting, the base graph and, possibly, the hierarchy can be casily inferred. Indeed, clustering can be performed in several ways, e.g., by grouping nodes according to physical proximity or functionality (e.g., by aggregating actuators belonging to cortain subcomponent). Nevertheless, when hcuristies do not allow for extracting the hicrarchy, graph pooling (Grattarola et al., 2022 Bianchi & Lachi 2024) provides set of robust and well-understood operators for constructing the hierarchical graph. We indicate with Gi, the pooled graph representation at the 44-th level of the hicrarchy and remark that the state of each node is hidden from entities at lower levels. The number of hicrarchy levels is a hyperparameter that depends on the problem at hand.
4.3 Learning Architecture
Given the introduced hicrarchical setup, this subsection illustrates the lcarning architecture and the proce- dure to generate actions starting from the environment state. In particular, we generate the initial repre- sentations of nodes in g* starting from raw observations, in bottom-up fashion. Subsequently, to take full advantage of both the feudal structure and the agent's morphology, information is propagated across nodes at the same level well as through the hierarchical structure. Finally, Buab-)managers set goals top-down through g", while workers act according to the received commands. We now break this process down into a step-by-step procedure, visual representation of which is reported in Fig.
State representation The environment state is partitioned and mapped to a set {st) of K local node states, cach corresponding to an actuator. We omit the temporal index t as there no ambiguity. Addi- tional (positional) node features {fi) can be included in the representation, thus generating observation vectors (a:) where, for each i-th limb, as, is obtained by concatenating 8; and f:. Starting from the observation vectors, the initial representation hy of each i-th node in G* is obtained recursively as:
WiD AGGR (dt (h)), i Gi,, ln E 1 JEC(s)
h
where Wi E Rd,xdd is a learnable weight matrix, p a (trainable) differentiable function and AGGR a generic aggregation function. The superscript of hς indicates the stage of the information propagation. In practice, the initial state representation of cach worker obtained through linear transformation of the corresponding observation vector, while, for (sub-)managen) representations are initialized bottom-up with many-to-one mapping obtained by aggregating recursively representations at the lower levels of g*.
5

---

Published in Transactions on Machine Learning Research (12/2024)
Hierarchical graph 6*
State representation (Eq. 2)
Propagation layer(Eq, 3)
Goallaction generation (Eqs. 4, 5)
Manager 9s
S/
Sub--managen 93
91
91
9
(AQ
(M
02
M
91
Workers 9h
3
Wi
W
A3
X
A
52 f
A
Envtronment
Figure 2: Learning architecture given the hierarchical graph G* and the graphs 9r, for the 'Walker' environ- ment. Trainable functions are reported in red and hierarchical operations are represented with dashed lines: in 9*, information flows bottom-up, while goals are assigned top-down.
Propagation layer Information across nodes is propagated through message passing. Nodes at the 4-th level can read from representations of neighbors in Gi, and subordinate nodes in -1, ie., those correspond- ing the lower level. We combine these information flows in a single message- passing step. In particular, starting from the initial representations h., each round {I,..., 1,} of message passing updales the corresponding state representation h
h clecd h AGGR JEN(i)
Jonnn (o (h/ ... h/y e.s) AGGR SEC(E)
Lodn (hl ( 22
where the message function &i regulates the exchange of information among nodes at the same hierarchy level, ohroth conveys, conversely, information from the subordinate nodes, and clidh updates the represen- tation. We comment that workers (nodes at the lowest level in the hierarchy) only receive messages from neighbors in G1, while the top-level manager simply reads from its direct subordinates. We remark that, in general, for each round 4, and hierarchy level L we can have different message and update functions. How- ever, multiple steps of message passing can lead to over-smoothing in certain graphs (Rusch et al., 2023 In such cases, the operator in Eq. can be constrained to only receive messages from neighbors at the same level. Note that. information nonetheless flows bottom-up al the initial encoding step [Eq. 2 and top-down through goals, discussed below.
Goal generation State representations used to generate goals (or commands) in recursive top-down fashion through g*. In particular, each supervisor i Gi,, with 4n E (2...., Lh), sends local goal gi-z to each subordinate node j E C(1) as:
wotn hh!-r n,tr .n9).
i E GLn otherwise
gi- =
(4)
with
AGGR (gk-re} h nn
kEP(1)
where the superscript L, denoles the last round of message passing (see Eq. 3 level manager has no supervisor: goals here are directly generated from its state representation, which encompasses the global state of the agent. All goal (or command) functions with used by (sub-)managers can be implemented, for example, as MLPs, while worker nodes do have an associated goal-generation mechanism, but, instead, have dedicated action-generation network.
We remark that the top-

---

Published in Transactions on Machine Learning Research (12/2024)
Action generation Lowest-level modules map representations to raw actions, which are then used to interact with the environment. For each i-th node in 91, the corresponding action d, is computed function of the aggregation of the goals set by the corresponding supervisors and, possibly, its state representation:
a: = === AGGR (AGGR(OJ-sl h
(5)
JEP())
The action-gente generation function μ can be again implemented by, e.g., an MLP shared among workers, We remark that actions associated with nodes that do not correspond to any actuator are discarded.
Rewards Each node of the hierarchy receives a different reward according to the associated goals and its role in the hierarchy. As already mentioned, the top-level manager coordinates the entire hierarchy and collects rewards directly from the environment. On the other hand, sub-managers and workers receive intrinsic rewards that can be used either as their sole reward signal or added to the external one. As an example, a dense reward signal for the i-th worker can be generated as function of the received goals and the state transition:
ri = fR ACGR (OcGR(g(-+) (g)-it, 8,, (EP(L))
(6)
where fR is score function and s; denotes the subsequent state. We remark that since goals are learned and are (at least partially) a function of 8,, designing a proper score function JR is critical to avoid de- generate solutions. Nevertheless, the reward scheme outlined in Appendix D.2 and empirically validated in Appendix C.3 can be easily applied to many tasks without any prior knowledge. At each step, rewards of nodes belonging to the same level 4 are combined and then aggregated over time generate a cumulative reward (or return) R which is subsequently used as learning signal for that level.
Scalability MPNNs are inductive, ic., not restricted to process input graphs of fixed size. As result, the number of learning paramelers mainly depends the depth of the hierarchy, which is a hyperparameter that depends the problem at hand. Nevertheless, we remark that, in graph deep learning literature, hicrarchics with a small number of levels (c.g., 3 levels) have proved to be effective even when dealing with large graphs (Wu et al. 2022), so there is usually no need for using very deep hierarchies.
5 Experiments
This section introduces the experimental setup and compares FGRL against relevant basclines; furthermore, we provide an in-depth study of the goal-generation mechanism. Note that the main objective here is to compare FGRL against other composable architectures such agents based on flat, i.e., not hierarchical, GNNs.
5.1
Experimental Setup
We validate our framework on two scenarios, namely a synthetic graph clustering problem inspired by Bianchi
et al. (2200) and continuous control environments from the standard MuJoCo locomotion tasks (Todorov
et al. 2012), where we follow Huang et al. (2020). More details on the graph clustering environment are
reported in Appendix
Compared approaches
The proposed approach is compared against multiple variants thereof:
Feudal graph neural network (FGNN): This is a complete implementation of the proposed framework.
Feudal deep set (FDS): This is an implementation of our framework that does not exploit any message passing. FDS can be seen as a modular version of hierarchical agent akin to Vezhnevets et al. (017)
7

---

Published in Transactions on Machine Learning Research (12/2024)
3. Graph neural network (GNN): Similarly to NerveNet (Wang et al. 2018 this model performs message passing on the agent graph 91 (see Eq. 1,, but does not exploit the hierarchical structure. An MLP then maps node-level representations to actions.
4. Deep set (DS): This baseline models the agent as a set of entities each corresponding to a limb, but does not rely on any structure to propagate representations. The policy network is implemented as a Deep Sets architecture (Zaheer et al. 2017) that maps node features to actions with an MLP. Learnable weights are shared among nodes.
Additional results including comparisons with non-modular baseline are reported in Appendix C.2 Tab. I provides a summary of the salient properties of each architecture. Additional details are reported in Appendix D For FGNN, messago-passing lay- ers simply propagate representations among neigh- bors and not across different levels; messages flow bottom-up at the encoding step only. In models ex- ploiting the hierarchical structure, the intrinsic re- wards of lower levels incentivize behaviors that align nodes with the assigned goals. More precisely, at each time step, the intrinsic signal of node is defined as a function of the cosine similarity between its states and the received commands (refer to Appendix D.2 for a detailed explanation).
Table 1: Summary of baselines and variants used.
FGNN
FDS
GNN
DS
Hierarchy
4
/
x
x
Message passing
1
x
/
X
Modularity
/
v
V
Optimization algorithm In FGRL, each graph 9i, represents a structured agent in which each node acts according to shared policy that is specific for that level. Each level is trained independently from the others as it maximizes its own reward signal. In principle, the model could also be trained end-to-end, i.e., as a single network, but this would most likely result in the collapse of goals' representations, and would not allow for having different reward signals for each level. We provide a sensitivity analysis of this multi-level optimization approach in Appendix C.3
In principle, cach level-specific policy can be paired with a standard RL optimization algorithm, but the hierarchical paradigm introduces additional challenges in the training procedure. As an example, instabilities at single level can hinder global performance and make credit assignment more difficult: good commands propagated by the upper layers are not rewarded properly if workers fail to select the correct actions. We use Covariance Matriz Adaptation Evolution Strategy (CMA-ES) (Hansen & Ostermeier 2001 as the optimization method since evolutionary algorithms have proven to be competitive with policy gradient methods and less likely get stuck in such sub-optimal solutions (Salimans et al.) 2017); we provide a comparison with a standard gradient-based algorithm in Appendix C.2
Finally, policies of intermediate levels can commit to sub-optimal behaviors if they receive the intrinsic reward as the only learning signal. Therefore, similarly to Vezhnevets et al. (2017), we add the external reward to the intrinsic one at each level. Further details regarding both the policy optimization and reward scheme can be found in Appendix D.1 and D.2 respectively.
5.2 Graph Clustering Problem
Given a graph with 3 communities and Ng nodes per community with random labels, the objective is to cluster nodes so that only nodes belonging the same community have the same label, i.c., are assigned to the same cluster. The agents receive a sparse reward based on the MinCut loss (Bianchi et al. 2020) at the end of each episode, with an additional reward bonus for eventually solving the problem. Further details regarding observation space, action space, and reward function hicrarchical graph g* of both FGNN and FDS is a 3-level graph that is built by assigning a sub-manager to each communily; all the 33 sub-managers are then connected to a single top-level manager. As a result, goals sent to workers and sub-managers are conditioned to be a 3-dimensional vector representing a target assignment at the node level and community level, respectively. Furthermore, we let the top-level manager and managesss act on different time scales by keeping the selected goals fixed for 10 and time steps, respectively. All baselines have access to static coordinates f E R22 of each node.
be found in Appendix B The
8

---

Published in Transactions on Machine Learning Research (12/2024)
FGNN
FDS
GNN
DS
Success Rate (%)
100
40
1.00
1.00
0.86
0.01
0.75
0.54
0.01
0.02
0.03
0.21
0.02
0.03
75
1.00
1.00
1.00
1.00
1.00
0.11
0.14
0.61
0.52
-5
0
2
6
2
6
4
8
B
B
Figure 3: Percentage of correct clustering (color) and median of NMT (value) over 4 independent runs. We remark that given a configuration Ng,, all the models are trained on the same topology to ensure fairness.
In Fig. we show the success rate of each agent clustering the graph and the median of the Normalized Mutual Information (NMT) score computed across different runs. Further details regarding the computation of the NMI are reported in Appendix C.1 FGNN outperforms all the baselines in settings with large communities, ic., with Ng- = 40: it fails to perfectly solve the task only in the environment with the highest complexity, ie., with 8 = 6 and NB 40, where the corresponding NMI score indicates good clustering nonetheless. The performance of the FGRL model without message passing (FDS) deteriorates when the number of nodes per community increases, while it is able to perfectly solve the task for N₂ 10. Indeed, for such small commmitiies, static coordinates alone appear sufficient for the feudal paradigm to reach one of the possible target configurations, while messnge-passing features result in being redundant. The comparison of the clustering scores achieved by the modular basclines without hicrarchy for N3- = 10 further supports this observation: the DS model, which not graph-based, achieves better NMI scores than GNN. In general, modular baselines solve simple scenarios, but performance degrades rapidly the complexity increases. When compared with GNN, results obtained by FGNN support our claims: the hicrarchical approach effectively enables long-range coordination. However, hierarchical architecture alone (without any local message passing) fails in solving the task for large communities where message-passiggg helps in coordinating the eeriiiionmmaiin process locally.
Effect of temporally extended goals As already mentioned, the top-level manager and sub-managers send a new command every and steps, respectively. By doing so, in the 10-step interval sub-managers set 2 commands, i..., one every steps, to steer subordinate workers towards a specific configuration; in turn, each worker has 5 time step to reach its assigned label configuration and obtain positive intrinsic reward. We investigate the effectiveness of temporally extended goals by comparing the results of FGNN and FDS against versions of these models where (sub-) goals). Results in Fig. 4 show that this mechanism is pivotal for solving the task. In particular, while the performance FGNN with N3- = 10 nodes is only slightly worse with one-step goals, the upper levels in the hierarchy fail to properly coordinate levels below for N₂ = 40, resulting in a drastic performance decrease. FDS with long-term goals outperforms its one-step counterpart when Ng- 10; however, none of the FDS variants is able to achieve good results for N3 40.
(sub-)managers managers send different goal at each time step (one-step
FGNN (long term)
FGNN (one step)
FDS (long term)
FDS (one step)
Success Rate (%)
100
40
1.00
1.00
0.86
0.01
0.02
0.26
0.01
0.75
0.54
0.01
0.20
0.75
75
Ne
50
1.00
0.97
1.00
1.00
1.00
0.78
1.00
1.00
1.00
0.52
0.88
0.88
2
2
4
B
B
Figure 4: Percentage of correct clustering (color) and median of NMI score (value) over 4 independent runs with long-term and one-step goals. We remark that given a configuration (8, Ng), all the models are trained on the same topology to ensure fairness.
9

---

Published in Transactions on Machine Learning Research (12/2024)
Humanoid
Walker
Half Cheetah
Hopper
1.00
Return
0.75
0.50
0.25
0.00
0.0
0..5
1.0
1.5
2.0 X10
0.0
0.5
1.0
1.5
2.0
2.5
0.0
0.5
1.0
1.5
2.0 X10*
0.0
0.5
1.0
1.5
2.0
2.5
Generation
Generation
x10*
Generation
Generation
X10
FGNN
FDS
GNN
DS
Figure 5: Average return and standard deviation of the considered agents on the MuJoCo benchmarks (av- eraged over 4 runs). Each generation refers to a population of 64 episodes. To ease the visualization, the plots show a running average with returns normalized w.r.b. the maximum oblained values, that are: 4025 (Humanoid), 2817 (Walker), 1918 (Half Cheetah), and 3175 (Hopper).
5.3 MuJoCo Benchmarks
We consider the modular variant of environments adapted from the standard MuJoCo locomotion tasks namely 'Humanoid', 'Walker2D', 'Hopper', and 'Half Cheetah' For these variants, use the same setup introduced by Huang et al. 2200 differently from the original environments where states (actions) are global, here the state (action) space is collection of local limb states (actions), one for each actuator of the agent. The composable nature of such variants makes GNNs suitable candidates for tackling these problems [Wang et al. 2018 Huang et al. 2020). In particular, the rationale for selecting these environments as benchmarks that empirical evidence (Kurin et al., 2020) shows the limitations of flat GNNs in achieving good coordination and motivates the adoption of our hierarchical approach in these settings. agents do not crash, episodes last for 1000 time steps; environment rewards are defined according to the distance covered: the faster the agent, the higher the reward. Nodes in the base graph the actuators of the physical agent: hence, in hierarchical models (FGNN and FDS), workers are clustered using simple heuristic, i.e., grouping together nodes belonging to the same body part (see Appendix D.3 for details).
We report the results for the 4 agents in Fig. 5 Returns obtained by the FGNN architecture in 'Humanoid' and 'Walker' support the adoption of message passing within the feudal paradigm in structured environ- ments, where the agent requires a higher degree of coordination. Indeed, hierarchical graph-based policies outperform the baselines by larger margin in the more complex tasks. In 'Half Cheetah', FGNN and GNN achieve similar results. A possible explanation for this behavior can be found by considering the morphology of the agent itself: this agent cannot fall, i.c., there is no need for high-level planning accounting to maintain balance. Similarly, in 'Hopper' (i.e., the agent with the simplest morphology), good locomotion policies can be learned without a high degree coordination, making the architecture of the more sophisticated models redundant. As one would expect, then, the performance of FGNN and GNN for this agent comparable with that of DS, i.c., the simpler modular variant. In general, FDS obtains subpar performance across envi- ronments, suggesting that, in modular architecture, the feudal paradigm might not be as effective without any other coordination mechanism.
Analysis of generated goals The feudal paradigm relies on the goal-generation mechanism, that es- tablishes communication among different hierarchical levels by propagating goals. This key feature enables policy improvement on both a global and local scale: the top-level manager has to generate goals aligned with the external return, while sub-managers must learn to break down the received goals into subtasks.
To investigate whether commands generated coherently, we run t-SNE Van der Maaten & Hinton 2008) goal vectors received by pairs of nodes with symmetric roles in the morphology of trained 'Walker' agent and analyze their time evolution during an episode. More precisely, for each pair of nodes (e.g., left and right knee) we collect the goals signals received in an episode, compute the one-dimensional t-SNE embeddings from such sample, and then plot the embeddings corresponding to the goals sent to the left and right nodes.
10

---

Published in Transactions on Machine Learning Research (12/2024)
Empirical observations emerging from the results in Fig. provide experimental evidence to support the effective- ness of the proposed goal-propagation mechanism. First, we highlight that commands sent by the top-level man- ager to the intermediate sub-managers responsible for legs (purple and blue nodes in Fig. 10p show a tempo- ral structure that is coherent with the oscillating patterns of worker nodes. Indeed, goals generated using top- down approach (refer to [o 4 where commands assigned to lower levels are a function of those received by up-
Sub- mannagggee
Idh
right
I-dim goal embedding
Heels
right
per levels; as such, the temporal structure is preserved. Second, while curves associated with workers intersect at time stops corresponding to the actual steps taken by the
Feet
physical agent, goal representations received by left and right nub-managers appear less predictable, and the cor- responding t-SNE embeddings are more diverse. Indeed, the abstraction of commands increases with the depth of the hierarchy: goals at the upper levels are not directly mapped into physical actions and should provide super- vision to the levels below. Conversely, goals assigned by sub-managers to workers become more specific and, e-g. depend on the specific joints being controlled. This analysis shows that propagated goals are meaningful and capture salient aspects of the learned gait. Results show coordination emerging from the interaction of nodes at different levels and support the adoption of such an ar- chitecture to implement hierarchical control policies.
ASSS
left
right
Knees
0
200
400
6i00 800
1000
Episode time steps
Figure 6: Analysis of goals received by sub- managers (orange and blue lines) and workers (green and red lines) in an episode of a 'Walker' environment (refer to Fig. 10 for the hierarchical graph). Plots show a 50-step running average.
6 Conclusions and Future Works
We proposed a novel hierarchical graph- based reinforcement learning framework, named Feudal Graph Re- inforcement Learning. Our approach exploits the feudal RL paradigm to learn modular hicrarchical policies within message-passing architecture. We argue that hierarchical graph neural networks provide the proper computational and learning framework to achieve spatiotemporal abstraction. In FGRL, nodes are orga- nized in multilayered graph structure and act according to committee of composable policies, each with a specific role within the hicrarchy. Nodes at the lowest level (workers) take actions in the environment, while (sub-)managee implement higher-level functionalities and provide commands al levels below, given a coarser state representation. Experiments on graph clustering and MuJoCo locomotion benchmarks together with the in-depth analysis of the learned behaviors highlight the effectiveness of the approach.
There are several possible directions for future research. In the first place, the main current. limitation of FGRL, shared among several HRL algorithms (Vezhnevets et al., 2017 , is in the inherent issues in jointly and efficiently learning the different components of the hierarchy in an end-to-end fashion. Solving this limitation would facilitate learning hierarchies of policies less reliant on the external reward, allowing state-of maate-of-the-at optimization methods to be integrated into the framework while ensuring stability in the training process. In this regard, different implerennaatios of the intrinsic reward mechanism could be explored. Lastly, while in this paper we addressed limitations of flat graph- approaches, future directions might investigate the effectiveness of the proposed framework in scenarios where credit assignment is more challenging. Preliminary experiments conducted in Sec. 5.2 show promising results in this sense.
Acknowledgements
This research was funded by the Swiss National Science Foundation under grant 204061: High-Order Rela- tions and Dynamics in Graph Neural Networks.
11

---

Published in Transactions on Machine Learning Research (12/2024)
References
Sanjeevan Ahilan and Peter Dayan. Feudal Multi-Agent Hierarchies for Cooperative Reinforcement Learning. arXiv:1901.00 1900:9990000.99 [es], January 2019. URL
08492 arXiv: 1901.08492.
Davide Bacciu, Federico Errica, Alessio Micheli, and Marco Podda. A gentle introduction to deep learning for graphs. Newral Networks, 129:203-221 221, 2020.
Victor Bapst, Alvaro Gonzaee,, Carl Doersch, Kimberly Stachenfeld, Pushmeet Kohli, Peter Battaglia, and Jessica Hamrick. Structured agents for physical construction. In International confer- ence on machine learning, pp. 464 474. PMLR, 2019.
Nikhil Barhate. Minimal pytorch implementation of proximal policy optimization. https://github. com/ m 2021.
Andrew G Barto and Sridhar Mahadevan. Recent advances in hierarchical reinforcement learning. Discrete event dynamic systems, 13(1):11-77, 2003.
Peter W Battaglia, Jessica B Hamrick, Victor Bapst, Alvaro Sancaez- Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacchetti, David Raposo, Adam Santoro, Ryan Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint orXiv:1806.012: 2018.
Filippo Maria Bianchi and Veronica Lachi. The expressive power of pooling in graph neural networks. Aduances in neural information processing systems, 36, 2024.
Filippo Maria Bianchi, Daniele Grattarola, and Cesare Alippi. Spectral clustering with graph neural networks for graph pooling. In International conference on machine learning, pp. 874 883. PMLR, 2020.
Michael M Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learning: Grids, groups, graphs, geodesies, and gauges. arXiv preprint urXic:2104.13470 2021.
Runfa Chen, Jiaqi Han, Fuchun Sun, and Wenbing Huang. Subequivariant graph reinforcement learning in 3d environments. In International Conference on Machine Learning, pp. 4545 4565. PMLR, 2023.
Peter Dayan and Geoffrey E Hinton. Feudal reinforcement learning. Advances in neural information pro- cessing systems, 5, 1992.
Jonas Degrave, Federico Felici, Jonas Buchli, Michael Neunert, Brendan Traccy, Francesco Carpanese, Timo Ewalds, Roland Hafner, Abbas Abdolmaleki, Diego de Las Casas, et al. Magnetic control of tokamak plasmas through deep reinforcement learning. Nature, 602((897):414 419, 2022.
Niklas Funk, Georgia Chalvatzaki, Boris Belousov, and Jan Peters. Learn2assemble with structured represen- tations and search for robotic architoctural construction. In Conference on Robot Learning, pp. 1401-1411.
PMLR, 2022a.
Niklas Funk, Svenja Menzenbach, Georgia Chalvatzaki, and Jan Peters. Graph-based reinforcement learn- ing meets mixed integer programs: An application to 3d robot assembly discovery. In 2022 IEEE/RS] International Conference on Intelligent Robots and Systems (IROS), pp. 10215 10222. IEEE, 2022b.
Justin Gilmer, Samuel S Schocnholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Ncural message passing for quantum chemistry. In International conference on machine leorning, pp. 1263-12221 1272. PMLR, 2017.
Daniele Grattarola, Daniele Zambon, Filippo Maria Bianchi, and Cesare Alippi. Understanding pooling in graph neural networks. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Agrim Gupta, Linxi Fan, Surya Ganguli, and Li Fei-Fei. Motamorph: Learning universal controllers with transformers. In International Conference on Learning Representations, 2022. URL https:://ppenreriir..
het/forum?idd=ppqtta. GvYL
12

---

Published in Transactions on Machine Learning Research (12/2024)
David Ha and Yujin Tang. Collective intelligence for deep learning: A survey of recent developments. Collective Intelligence, (1):263391322111 11122211144444 2022.
David Ha, Andrew M. Dai, and Quoc V. Le. Hypernetworks. In International Conference on Learning Representations, 2017. URL http:s///ppaaaaeeee..eeee////a/a
Jessica B Hamrick, Kelsey R Allen, Victor Bapst, Tina Zhu, Kevin McKee, Joshua B Tenenbaurn, and Peter W Battaglia. Relational inductive bias for physical construction humans and machines. arXiv preprint Xiv: 01203, 2018.
Nikolaus Hansen and Andreas Ostermeier. Completely derandomized self-adaptation in evolution strategics. Evolutionary compulation, 9(2):159 195, 2001.
Wenlong Huang, Igor Mordatch, and Deepak Pathak. One policy to control them all: Shared modular policies for agent-agnostic control. In International Conference on Machine Learning, pp. 4455 4464. PMLR, 2020.
Jiechuan Jiang, Chen Dun, Tiejun Huang, and Zongging Lu. Graph convolutional reinforcement learning. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.ng not, 2020. URL
Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arv preprint arXiv:142.6900 2014.
Vitaly Kurin, Maximilian Igl, Tim Rocktäschel, Wendelin Boehmer, and Shimon Whiteson. My body is a cage: the role of morphology in graph-based incompatible control. In International Conference Learning Representations, 2020.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in Newral Information Processing Systems, 35:27730 27744, 2022.
T Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing graph neural networks. arXie preprint 2rXi::2303.19990 2023.
Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Πlya Sutskever. Evolution Strategies as a Scal- able Alternative to Reinforcement Learning, September 2017. URL http:///arxiv.orggbb////0000 03864 Number: S.Siv:1703.03S66 arXiv:1703.0m 038664 [cs, stat].
Alvaro Gonzale,,,,,a,ez, Nicolas Heess, Jost. Tobias Springenberg, Josh Merel, Martin Riedmiller, Raia Hadsell, and Peter Battaglia. Graph networks as learnable physics engines for inference and control. International Conference on Machine Learning, pp. 4470 4479. PMLR, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimiza- tion algorithms. arXiv preprint Xrxi:: 06347, 2017.
David Silver, Julian Schrittwieser, Karen Simonyan, loannis Antonoglou, Aja Huang, Arthur Guez, Thomas Hubert, Lucas Baker, Matthew Lai, Adrian Bolton, et al. Mastering the game of go without human knowledge. nature, 550(7676):354-33)) 359, 2017.
Richard S. Sutton and Andrew G. Barto. Reinforcement learning: introduction. Adaptive computation and machine learning series. The MIT Press, Cambridge, Massachusetts, second edition edition, 2018, ISBN 978-0-266-0392446.
Richard S Sutton, Doina Precup, and Satinder Singh. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artificial intelligence, 112(1-2):181-211 1999.
Yuval Tassa, Yotam Doron, Alistair Muldal, Tom Erez, Yazhe Li, Diego de Las Casas, David Budden, Abbas Abdolmaleki, Josh Merel, Andrew Lefrancq. et al. Deepmind control suite. arXiv preprint arXiv:1801.00660 2018.
13

---

Published in Transactions on Machine Learning Research (12/2024)
Emanuel Todorov, Tom Erez, and Yuval Tassa. Mujoco: A physics engine for model- control. In 2012 IEFE/RSJ international conference on intelligend robols and systems, pp. 5026 5033. IEFE, 2012.
Jake Topping, Francesco Di Giovanni, Benjamin Paul Chamberlain, Xiaowen Dong, and Michael M. Bron- stein. Understanding over-squashing and bottlenecks on graphs via curvature. In International Conference on Learning Representations, 2022. URL htttss://ppenrevior htte:://ppeaaoooooa../////ooo/op wtt..n/e//ooorrr/ddddnnnnrae
Brandon Trabucco, Mariano Phielipp, and Glen Berseth. Auymorph: Learning transferable polices by inferring agent morphology. In International Conference on Machine Learning, pp. 21677-21691. PMLR, 2022.
Laurens Van der Maaten and GeofTrey Hinton. Visualizing data using (-sne. Journal of machine learning research, ((1),, 2008.
A Vaswani. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Alexander Sasha Vezhnevets, Simon Osindero, Tom Schaul, Nicolas Heess, Max Jaderberg, David Silver, and Koray Kavukcuoglu. Feudal networks for hierarchical reinforcement learning. In International Conference on Machine Learning, pp. -549. PMLR, 2017.
Tingwu Wang, Renjie Liao, Jimmy Ba, and Sanja Fidler. Nervenet: Learning structured policy with graph neural networks. In International conference on learning representations, 2018.
Junran Wu, Xucyuan Chen, Ke Xu, and Shangzhe Li. Structural entropy guided graph hicrarchical pooling, In International conference on machine learning, pp. 24017 24030. PMLR, 2022.
Peter R Wurman, Samuel Barrett, Kenta Kawamoto, James MacGlashan, Kaushik Subramanian, Thomas J Walsh, Roberto Capobianco, Alisa Devlic, Franziska Eckert, Florian Fuchs, et al. Outracing champion gran turismo drivers with deep reinforcement learning. Nature, (1276666):223-12 2022.
Zheng Xiong, Jacob Beck, and Shimon Whiteson. Universal morphology control via contextual modulation. In International Conference on Machine Learning, pp.
PMLR, 2023.
Zheng Xiong, Risto Vuorio, Jacob Beck, Matthieu Zimmer, Kun Shao, and Shimon Whiteson. Distilling morphologg--onditionee hypernetworks for efficient universal morphology control. Forty-first Interna- tional Conference on Machine Learning, 2024. URL httpp///ppppooooeeeeeeeerrrrrr/o
Zhitao Ying, Jiaxuan You, Christopher Morris, Xiang Ren, Will Hamilton, and Jure Leskovec. Hierarchi- cal graph representation learning with differentiable pooling. Advances in neural information processing systems, 31, 2018.
Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems. 35:24611 15:24461-24624, 2022.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexan- der J Smola. Deep sets. Advances in neural information processing systems, 30, 2017.
Vinicius Zambaldi, David Raposo, Adam Santoro, Victor Bapst, Yujia Li, Igor Babuschkin, Karl Tuyls, David Reichert, Timothy Lillicrap, Edward Lockhart, al. Deep reinforcement learning with relational inductive biases. In International conference on learning representations, 2018.
14

---

Published in Transactions on Machine Learning Research (12/2024)
Appendix
A Extraction of the Hierarchical Graph
In Fig.1 7 we report example of extraction of the hierarchical graph g* for the 'Humanoid' environment: the basc graph G1 is obtained directly from the agent's 's morphology (sec Fig. for reference) and the number of layers of g is a hyperparameter.
Agent graphs G
Hierarchical graph g
Workers
Sub-managers
Manager
lo
Clustaring
Clustering
Clustaring
Muttilevel structure
1/2
8. m1
Figure 7: Extraction of the hierarchical graph 9* starting from the workers' graph 91 the 'Humanoid' environment. Hierarchical edges of 9 are represented as dashed lines and denote parent-child relationships. As shown in the second level, each node can have more supervisors.
B Graph Clustering Environment
The environment of the synthetic graph clustering problem is defined by 3-community graph with Nj nodes for each community, that is, community graph with 8Ng nodes. The objective is to cluster the graph so that each community has a unique label for all the nodes, different from other communities. For each simulation, we generate a (B, Na) graph with binary adjacency matrix A E {0, 10.1888,8,N, and degree matrix D = diag(Alx), undor the assumption that thore are no isolated nodes. Each node i has a static feature vector given by the fi E IR² coordinates: nodes belonging to the same community are characterized by similar features. At each time step t, the observation vector of of each i-th node the concatenation of the one-hot encoding of the current cluster assignment 8 its coordinates, and the normalized remaining time steps before the end of the episode:
2 = s s [|fl|l--t//
(7)
We remark that assignments are initialized at random at the beginning each episode. The action is a dimensional vector (left/no-on /nt//nooooo/t//eee that allows nodes change cluster by possibly changing its label of most one index; periodic boundary conditions are applied the label vector. We consider an episodic setting where episodes last for T = 50 time steps, unless the graph reaches one of the possible target configurations where nodes belonging to the same community have the same label. The sparse reward at the termination time step < I' is given by:
R TT(CAC)
CCC
I VK
+ (T
(8)
Tr(CTDC)
|C
where C is the cluster matrix, À = D AD R8N5XAND is the normalized adjacency matrix, and D is the degree matrix of A. The first two terms represent the negative of the MinCut loss (Bianchi et al. 2020): the first one promotes clustering among strongly connected nodes, while the sccond one prevents degenerate solutions by ensuring similarity in size and orthogonality between different clusters. We remark that, for this maximization problem, those terms are bounded in [0, 1] and |-2, O], respectively: therefore, the bonus factor (T - t) strongly encourages solving the task with as few iterations as possible. We consider the task as solved if the running average of the success rate percentages of the last 20 evaluation samples is greater than 95%.
15

---

Published in Transactions on Machine Learning Research (12/2024)
C Additional Results
C.1 NMI Score in the Graph Clustering Problem
The NMI score measures the similarity between two independent label assignments X, Y. It is computed as:
2T(X;Y) NMI(X,Y) H(X) + H(Y
(9)
where H-) and I(X;)) denote the entropy of the labels and mutual information, respectively. This metric is bounded in [0, 1], where the boundaries represent perfect dissimilarity (NMI = 0) and similarity (NMI = 1), and is invariant under labels permutation. Hence, in our setting, we measure the NMI score of the predicted clustering w.r.t. generic target configuration where nodes belonging to the same community have the same cluster-specific label. In Tab. 2 we report the maximum (blue) and minimum (red) values achieved by the models in each configuration (Fgg. 3 and for reference).
Table 2: Minimum and maximum values of the NMT score achieved by the models in the graph clustering problem.
N₂ 10
N3 =
3 2
B 4
8 6
B =
3 4
3 6
FGNN
(1.00, 1.00)
(0.10, 1.00)
(0.15, 1.00)
(0.01, 1.00)
(0.02, 1.00)
(0.84, 0.94)
FDS
(1.00, 1.00)
(1.00, 1.00)
(1.00, 1.00)
(0.01, 1.00)
(0.02, 1.00)
(0.39, 1.00)
GNN
(1.00, 1.00)
(0.11, 0.24)
(0.14, 0.15)
(0.01, 0.03)
(0.02, 0.02)
(0.03, 0.03)
DS
(1.00, 1.00)
(0.21, 0.76)
(0.42, 0.88)
(0.01, 1.00)
(0.02, 0.03)
(0.03, 0.03)
FGNN (one-step goals)
(0.03, 1.00)
(0.09, 1.00)
(0.16, 1.00)
(0.01, 0.02)
(0.02, 0.88)
(0.03, 0.79)
FDS (one-step goals)
(0.04, 1.00)
(0.76, 1.00)
(0.76, 0.88)
(0.004, 0.013)
(0.02, 0.44)
(0.60, 0.87)
C.2 Comparison with Non-modular Baselines
The FGRL framework is specifically designed for learning modular policies. Therefore, the experiments con- ducted in Sec. 5.2 and 5.3 validate the effoctivencss of the full proposed mothodology (FGNN) in comparison to other modular approaches, such as flat GNNs. Nevertheless, to better contextualize the results, here we provide a comparison with an MLP (a non-modular policy) where each state and action are the concatena- tion of node-level observations and actions, respectively. Being non-modular, it cannot be directly applied to different morphologies and the model size increases with the number of nodes. We remark that, in Sec. 5.2 and 5.3 all the models were trained using the same learning algorithm, i.e., CMA-FS (Hansen & Ostermeier 2001h, to highlight the performance differences brought by each architectural variant rather than those com- ing from the learning algorithm. Specifically, we chose evolution strategics because, as mentioned in Sec. 5.1 learning a hierarchy of level-specific policies using independent reward signals is challenging. In this regard, this section provides a comparison against two different MLP basclines that use different learning algorithms: one is trained with CMA-ES to assess the difference in performance w.r.t. the composable FGNN, while the other uses Prozimal Policy Optimization (PPO) (Schulman et al., 2017) and allows to analyze the results in the broader context of gradient-based methods. For simplicity, we denote the two variants as MI. and PPO, respectively. Hyperparameters are reported in Appendix D.4 and the code for PPO was adapted from a public repository (Barhate, 2021). We chose PPO as a gradient-based algorithm because of its popularity and wide applicability in both single and multi-agent reinforcement learning problems (Yu et al. 2022
In general, evolution strategies rely on selection process that evaluates sample (population) of parameters for each update step, resulting in worse sample efficiency compared to gradient- based methods that
16

---

Published in Transactions on Machine Learning Research (12/2024)
Humanoid
Walker
Half Chectah
Hopper
1.00
Return
0.50
0.25
0.00
0.0
10.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
2.5 x10
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
2.5
Generation
X10
Generation
Generation
X10
Generation
X10
FGNN
MLP
PPO (max)
PPO (meun)
Figure 8: Comparison of the average return achieved with FGNN (4 runs) and MLP (4 runs) with average and maximum value of PPO (5 runs) at the last training step in the 4 MuJoCo benchmarks. To ease the visualization, the plots show a running average with returns normalized w.r.t. the maximum obtained values, that are: 4025 (Humanoid), 3125 (Walker), 2402 (Half Cheetah), and 3950 (Hopper).
instead perform multiple updates with a smaller sample. Therefore, in Fig. report the average returns obtained with FGNN and MLP, logether with the mean and maximum values among 5 independent. seeds of PPO at the last training step, L.e., 2. 107; we applied running average to account for fluctuations. PPO hyperparamnters were tuned on the 'Walker' agent (as we similarly did for CMA-ES). In this scenario, the performance of FGNN and MLP is comparable with PPO, highlighting that evolution strategies allow to achieve absolute returns that are competitive w.r.t. a widely used gradient-based method. Conversely, PPO appears be less robust when the same hyperparameters are used to learn locomotion policies for other morphologies. FGNN performs similarly better than the MLP baseline in all the environments, except for 'Hopper': already mentioned, this agent has the simplest morphology and, expected, simple MLP is enough to learn good policy. Furthermore, we qualitatively observed that FGNN learns more natural gaits in 'Humanoid' and 'Walker'.
C.3 Analysis of the Multi-Level Optimization
The proposed feudal framework relies on a multi-level hierarchy where nodes at each level Gi, act according to a level-specific policy trained to maximize its own reward. Notably, the policy trained at level l, completely ignores rewards received at different levels: coordination completely relies on the message-passingg and goal- generation mechanisms. This peculiar aspect implies that each policy can have its own policy optimization routinc.
To analyze the impact of such multi-level structure and optimization routine, we compare results obtained by a 3-level FGNN model in the 'Walker' environment against 1) a 3-level FGNN model without intrinsic reward, 2) a 2-level FGNN model, and 3) 3-level FGNN model where all the policies are jointly trained to maximize the external reward only. We remark that the number of learnable parameters is the same for all the 3-level variants. As shown in Fig. 9 our method achieves the highest average return, while baselines are more likely to get stuck in sub-optimal policies (e.g., by learning jump rather than to walk). Comparing the full model (blue curve) with the variant without intrinsic reward (left, red curve) highlights the advantage of the hierarchical intrinsic reward mechanism. In the full 3-level FGRL model, the reward at each level incentivizes workers and sub-managers to act accordingly to the received commands. This auxiliary task turns out to be instrumental in avoiding sub-optimal gaits. The 2-level model (middle, green curve) performs akin to the variant without intrinsic reward, hence hinting at the benefits using a deeper hierarchy for agents with complex morphologies. Lastly, for the variant where all levels are jointly optimized (right, black curve), empirical results show that the resulting agent does not only achieve lower return but, surprisingly, it is also less sample efficient.
17

---

Published in Transactions on Machine Learning Research (12/2024)
Return
0.75
0.25
3 levels intrinsic reward
1 level
3 levels single optimizer
2 levels
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
0.0
0.5
1.0
1.5
2.0
Generation
x10
Generation
x x10*
Generation
x10
Figure 9: Analysis of the 3-1evel FGNN model with variants that use a different reward choice (left), depth of the hicrarchy (middle), and optimization scheme (right) in 'Walker' environment (4 seeds for each variant). Returns are reported with standard deviation and each generation refers to population of 64 episodes. To case the visualization, the plot shows a running average with returns normalized w.r.t. the maximum obtained value, i.e., 2725.
D Implementation Details
D.1 Optimization Algorithm
As already mentioned, to learn the parameters for our setup we use evolution strategies, namely CMA- ES (Hansen & Ostermeier 2001). The number of instances of CMA-ES we initialize corresponds to the depth of the hierarchy, as each level has independent policy. Indeed, each CMA-ES optimizes only modules corresponding to its level:
Workers CMA-ES (lovel 1): weight matrix W1, functions μ and Φ1-
Sub-managers CMA-ES (levels la ( La 1}): functions p²,, wht and o..
Manager CMA-FS (level Tx): functions p and Lh
Experiments were run on workstation equipped with AMD EPYC 7513 CPUs. Depending on the model and environment, each seed can take from 30 minutes to day for the graph clustering experiment and from 12 hours to 3 days for the MuJoCo benchmarks. Code to reproduce experiments available online at https://github. com//oommasmmarr//frr
D.2 Intrinsic Reward
In variants that take advantage the hierarchical structure (FGNN and FDS), we define the intrinsic reward of sub-managers and workers as a signal that measures their alignment with the assigned goals. At cach time step I, for each hierarchical level we add the environment reward to the average intrinsic signal. For both environments, the reward scheme was inspired by Veeznnvets et al. (017)
Graph clustering problem In this enviromment, each i-th node has single propagated goals represent target label. At each time step, the reward of each i-th worker is defined as:
P(i) and
ri T de (IP()))))))) kt) 0.5]
(10)
where de is the cosine similarity function, while JP()) and s denote the one-hot encoding of the received command and subsequent cluster, respectively. Similarly, intermediate nodes are responsible of the ag- gregated cluster of subordinated nodes, and the intrinsic reward of each i-th sub-manager can be defined as:
Ti T de gP(0-si
0.5
(11)
|(()
kEC(s)
18

---

Published in Transactions on Machine Learning Research (12/2024)
Humanoid
Walker
Half Cheetah
Hopper
Figure 10: Extraction of the hierarchical graphs G* used in the FGNN model. The filled colored ellipses with dashed lines highlight the hierarchical connections among different levels, and overlapping circles imply that the node is subordinate of both the sub-managers. Graphs with green nodes represent the original agent graphs G1.
Since & is a one-hot encoding vector and goals represent a larget label, we set the one-step intrinsic rewards in the range r3 [-0.01, 0.01] to penalize subordinate nodes for not following assigned commands. In order to be comparable with the values of the MinCut loss, intrinsic signals are normalized w.r.t. the length of the episode.
MuJoCo benchmarks In this control problem, goals are instead unconstrained latent vectors. Nodes at the lowest level have the most fine-grained representations and learn to follow goals at the simulation scale. Thus, each i-th worker receives an intrinsic signal:
= de AGGR (g, (EP(()
8, +
(12)
where s/ denotes the subsequent state. Conversely, intermediate nodes operate on coarser scale, and the intrinsic reward of each i-th sub-manager can be similarly defined as:
T; = de AGGR (gy-i), nn.²) +
(AGGR (EP(())
(13)
where h." denotes the subsequent initial state representation, computed using Eq. 2 Since goals are latent, we prevent lower levels from getting negative reward for not following the assigned commands in the first stages of learning by providing positive intrinsic rewards in the range E [0,
D.3 Hierarchical Graphs
The graphs under analysis for the MuJoCo benchmarks have low number of nodes, and each actuator has proper physical meaning in the morphology of the agent. Thus, decide to create the hicrarchical graphs using heuristic. As an example, in the 'Walker' agent we expect nodes of the same leg to be clustered together, and the associated sub-manager to be at the same hierarchical level as that of the other leg: in this way, the topology of the intermediate graph reflects the symmotry of the agent graph G1-
The hierarchical graphs for the FGNN model are reported in Fig. 10 simple morphology with no immediate hierarchical abstraction and where each actuator has different role: consoquence, meaningful hicrarchy cannot be trivially extracted, and results revealed benefit in implementing 3-level hierarchy for this agent. We remark that hierarchical graphs used in the FDS variant are not reported because in all the enviromments empirical evidence did not show improvements as the depth of the hierarchy increased, leading to 2-level hierarchical graphs where all the workers are connected to a single top-level manager (see 'Hopper' in Fig. 10 for an example).
Notice that the 'Hopper' agent has a
D.4 Reproducibility
The code for the experiments was developed by relying on open-source libraries and on the publicly available code of previous works All the learnable components in message passsagg-pssing blocks are implemented as
ESoool: https: https://githup ggbbbbubhhuutt.g com/harddarr//estooll
2SMP Huang al. 2020 http:://gghbb..co//uuuup/tt/lttt
3 NerveNet Wang al. 2018: https://github.. http:://gghhhttooooooo/c/aaaaaaa
19

---

Published in Transactions on Machine Learning Research (12/2024)
MLPs. In [ab. are reported the values of the hyperparameters used in experiments. For each baseline, we tuned the number of hidden units and CMA-FI step size by performing a grid search on the average return. Note that for the MuJoCo benchmarks the best configuration of the MLP baseline and that of FGNN results a similar number of learnable parameters. For the graph chustering experiment we used the same set of hyperparamelers, except for the aggregation function of subordinate nodes in the feudal models where we used the average instead of the sum.
The hyperparameters used for the comparison with PPO (Appendix C.2 are the following: used Adam optimizer [Kingma & Baj 2014) with learning rate of 3 10 and hidden layers [64, 64] with tanh non- lincarities for both actor and critic; discount factor and clipping value are fixed to 0.99 and 0.2, respec- tively; policy is updated for 10 epochs with batch size of 64 and the updating horizon is 2048 time steps; the initial action standard deviation is 0.6 and decays every 10 episodes 0.05, up to a minimum of 0.2. We performed a grid search on such hyperparameters, focusing mainly on learning rate, hidden layers, updating epochs, and updating horizon.
Table 3: Hyppepaarameters used in each model, where the X marker indicates those that not part of the architecture. For the total number of parameters, in the non-modular baseline (MLP) reported the maximum among the four environments of the MuJoCo benchmark. We remark that in the modular models, the number of parameters does not depend on the environment, but since in the feudal architectures it depends on the hierarchical height, in FGNN we reported the number corresponding to the maximum one, i.e., levels.
Context
Hyperparameter
MLP
DS
GNN
FDS
FGNN
Population size
64
64
64
64
64
CMA-ES
Initial step size
0.25
0.25
0.25
0.5
0.25
Dimension of state representation
X
X
32
32
20
Dimension of hidden layer
64
32
32
32
30
Activation function
tanh
tanh
tanh
tanh
tanh
AGGR (subordinate nodes)
X
sum
sum
AGGR (message passing)
X
X
sum
x
sum
Policy
AGGR (goals aggregation)
X
X
mean
mean
Maximal hierarchy height
X
X
X
2
Message-passing rounds
X
X
X
Shared weights (message passing)
x
X
/
# of parameters (MuJoCo benchmark)
11593
673
4865
7124
12700
20


Reviews:
[{"review_id": "kj0SPRbrc4", "reviewer": "Reviewer_ktDR", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n(1) The integration of feudal (hierarchical) reinforcement learning principles with graph-based modularity introduces an effective structure for decomposing and coordinating tasks across various levels.\n\n(2) The paper demonstrates that hierarchical goals enable long-range coordination and facilitate composable task learning.\n\nWeaknesses:\n\n(1) Constructing a graph-based agent representation may limit the algorithm’s applicability, as it could be restrictive for tasks beyond robotic control.\n\n(2) The baseline comparisons rely on older models (proposed in 2017 and 2018), and the empirical evaluation resembles an ablation study. It omits comparisons with more recent hierarchical RL algorithms, which would provide a more comprehensive evaluation.\n\n(3) The performance results in Figure 5 are notably below state-of-the-art levels, suggesting potential areas for improvement in the algorithm's design. For reference, see the state-of-the-art results reported in https://tianshou.org/en/v0.4.8/tutorials/benchmark.html.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "FGRL introduces a layered graph hierarchy, aligning hierarchical reinforcement learning (HRL) principles with graph neural network (GNN) capabilities, and improves task decomposition and coordination for structured environments.\n\nThe paper’s primary contributions include:\n\n(1) The FGRL framework, which builds a pyramid-like, hierarchical control structure using GNNs. This structure helps the model break complex tasks into manageable sub-tasks by passing commands through hierarchical levels.\n\n(2) Empirical evaluations on graph clustering and MuJoCo locomotion tasks, showing FGRL’s strong performance against standard baselines.\n\n(3) Analysis supporting hierarchical message-passing as an effective method for learning task-decomposing, decision-making policies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please see the weaknesses.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "gZKPjrwwro", "reviewer": "Reviewer_ELFu", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n\n- The paper is well-written, easy to follow, and the figures are nice.\n- Integrating hierarchical structure with graph-based reinforcement learning is a reasonable approach when the task exhibits a specific structure.\n- Compared to simpler message-passing methods, the hierarchical message-passing approach improves global coordination and alleviates information bottlenecks common in complex tasks.\n- Analysis of the generated goals indicates that the hierarchical structure results in understandable and interpretable command propagation, effectively representing key features of the learned behaviors.\n\n## Weaknesses\n\n- While the proposed method introduces considerable complexity compared to a flat GNN, the empirical evaluation doesn't demonstrate a corresponding significant improvement.  For example, Figure 8 shows the proposed method failing to outperform even a basic MLP agent, which requires neither specialized design nor prior knowledge of the graph structure.  Furthermore, the feasibility of relying on heuristics to create hierarchical graph remains unclear on a borader range of applications.\n\n- The application of the proposed approach to continuous control tasks like MuJoCo is nice.  However, the weak performance is a signifcant issue.  Mainstream reinforcement learning algorithms like SAC achieve substantially better results on Humanoid, Walker, and Half Cheetah environments than the proposed method.  While not every method needs to achieve state-of-the-art performance, results significantly below the community standard raise concerns about the proposed method's practical utility.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a methodology called Feudal Graph Reinforcement Learning. It addresses the limitations of standard graph-based RL methods, which can struggle with global coordination and high-level planning due to information bottlenecks created by local message-passing operators. \n\nFGRL introduces a hierarchical RL approach with a pyramidal message-passing architecture to improve composability in control problems.  The key idea is to establish a hierarchy of policies, mimicking a feudal structure, where high-level commands are propagated down through a layered graph structure. Lower layers in the hierarchy represent the physical system's morphology (like the limbs of a robot), while upper layers represent higher-order sub-modules and overall control.  This hierarchical structure allows for task decomposition, with actions at each level setting goals for the level below.\n\nFGRL was evaluated on a graph clustering problem and MuJoCo locomotion tasks. The experiments showed that the hierarchical message-passing scheme facilitates learning hierarchical decision-making policies and improves coordination, especially in complex tasks requiring higher-level planning.\n\nThe main contributions of this paper include:\n\n- They introduce the FGRL paradigm, a new methodological deep learning framework for graph-based HRL in composable environments, and apply it to a graph clustering problem and composable continuous control tasks.\n- They provide empirical evidence that supports the adoption of hierarchical message-passing schemes and graph-based representations to implement hierarchical control policies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "N/A", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "C5RjrrTJ6I", "reviewer": "Reviewer_14WH", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Graph neural networks and feudal RL seem like a reasonably good match. It is a valuable contribution to the community to explore these together.\n- The paper is well written and has clear figures illustrating the concepts.\n- Some details are missing or are not clearly presented in the current manuscript. Please see requested changes for more.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "- Proposes a graph neural network -based hierarchical RL method.\n- Evaluates on MuJoCo and graph clustering tasks.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Since Kurin et al. [2020], a number of other works targeting similar graph-based morphology control setting have been published. These seem like missing related work from the current paper. Discussion of similarities should be included.\n\t- Trabucco et al. 2022., \"AnyMorph: Learning Transferable Polices By Inferring Agent Morphology\".\n\t- Gupta et al., 2022., \"Metamorph: Learning universal controllers with transformers\".\n\t- Xiong et al., 2023., \"Universal Morphology Control via Contextual Modulation\".\n\t- Chen et al., 2023., \"Subequivariant Graph Reinforcement Learning in 3D Environments\".\n\t- Xiong et al., 2024., \"Distilling Morphology-Conditioned Hypernetworks for Efficient Universal Morphology Control\".\n- Since the whole architecture depends on the submanagers and workers optimizing their rewards in a useful way, relegating the discussion of the reward design fully to the appendix seems odd. Please include a high-level description of the reward you ended up using in the main paper.\n- Apparently the MuJoCo setup introduced by Huang et al. [2020] is different from the vanilla MuJoCo setting considered elsewhere in RL. Please include a brief description on how the environments differ and why standard \"flat\" (as opposed to hierarchical) RL is not appropriate.\n- The discussion of the temporal structure in Figure 6 is not clear. I don't think you can say that the top panel (sub-managers) has less structure in time than the third panel (feet). Furthermore, whatever temporal structure there is in the top panel seems highly correlated with the structure in the other panels. That said, I like this analysis, please make it clearer.\n- While this is a well presented and interesting approach to HRL, the paper does not really explore the limitations of HRL in general and this method specifically. There is a sentence referring to the limitations listed in Vezhnevets et al. [2017] but one would imagine that research in HRL would have a different set of concerns more than seven years later. It does not become clear from the manuscript how applicable the proposed method is, e.g., in standard RL environments with challenging temporal credit assignment problems, for which HRL has been proposed as a solution. Discussion of the limitations would help place this work in broader context.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Reviewers are mostly satisfied with responses and adjustments made in response to their reviews, and in favour of acceptance. 

There is one remaining concern revolving around choice of baselines to compare to (e.g., no SAC) and the strength of empirical results. While stronger results and more baselines would of course never be bad, claims and conclusions are already carefully phrased, and the experiments as they are now adequately support them. Therefore, I do not view this as a reason for rejection.

For the camera-ready version, please try to include source code (which, at this point, need not be anonymous anymore), as promised in Appendix D.1.
Decision: Accept (poster)