Learning a Distributed Hierarchical Locomotion Controller for Embodied Cooperation
Authors: Chuye Hong
Kangyao Huang
Huaping Liu
Venue: CoRL 2024 (2024)
Keywords: Cooperation
Locomotion
Hierarchical reinforcement learning

Abstract: In this work, we propose a distributed hierarchical locomotion control strategy for whole-body cooperation and demonstrate the potential for migration into large numbers of agents. Our method utilizes a hierarchical structure to break down complex tasks into smaller, manageable sub-tasks. By incorporating spatiotemporal continuity features, we establish the sequential logic necessary for causal inference and cooperative behaviour in sequential tasks, thereby facilitating efficient and coordinated control strategies. Through training within this framework, we demonstrate enhanced adaptability and cooperation, leading to superior performance in task completion compared to the original methods. Moreover, we construct a set of environments as the benchmark for embodied cooperation.

Learning a Distributed Hierarchical Locomotion Controller for Embodied Cooperation
Chuye Hong Kangyao Huang Huaping Liu Department Computer Science and Technology, Tsinghua University "Equal contributions, *Corresponding author.
Abstract: In this work, we propose a distributed hierarchical locomotion con- trol strategy for whole-body cooperation and demonstrate the potential for scal- ing to large numbers of agents. Our method utilizes a hierarchical structure to break down complex tasks into smaller, manageable sub-tasks. By incorporating spatiotemporal continuity features, establish the sequential logic necessary for causal inference and cooperative behaviour in sequential tasks, thereby facilitating efficient and coordinated control strategies. Through training within this frame- work, we demonstrate enhanced adaptability and cooperation, leading to superior performance in task completion compared to the original methods. Moreover, we construct set of environments as the benchmark for embodied cooperation. visual depiction of highlights of the learned cooperative behaviours can be found on this website'.
Keywords: cooperation, locomotion, hierarchical reinforcement learning
Introduction
Cooperatively accomplishing embodied tasks with multiple robots has consistently been a highly challenging area of research. Recent studies mainly focus on embodied manipulation cooperation among robotic arms or formation control over the upper level within a group of mobile robots [1, 2]. Nevertheless, multi-agent cooperation via whole-body and end-to-end locomotion control is rarely studied.
Some previous works showcase the manipulation via locomotion [3] but are only tested on two agent systems, and the scalability of this method is still agnostic for migration to any number of agent populations. In this work, we aim to realize more complex embodied multi-agent cooper- ation by learning distributed hierarchical locomotion control system, decomposing the complex and coupled behaviours while maintaining the potential for unlimited expansion on the swarm. As the foundation for implementation and validation, construct three scenarios in IsaacSim [4] as benchmarks for embodied cooperation study.
(a) Cooperative Transport
(b) Corridor Crossing
Ravine Bridging
Figure 1: Embodied Cooperation Environments.
Videos of our results are available at our project webpage: https//dd-rrl.iihubb.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Concurrently, training robot for a specific function can be effectively achieved through reinforce- ment learning (RI.), such as learning movement patterns [5], interactive behaviours [6], as well as logical inference in games [7]. Although RL provides a recognized powerful exploration capa- bility and tremendous progress has been made sampling efficiency [4], finding and mastering sequence of sophisticated tasks through searching remains challenging problem. Hicrarchical reinforcement learning (HRL) alleviates this to certain extent, aiming to understand the logical relationships among "control, action, behaviour, dynamic outcomes, and feedback" in a segmented manner. An increasing number of studies are employing hierarchical approaches to decompose complex tasks [8, 9, 10, 11, 12, 13, 14, 15]. Inspired by these, we propose a fully-decenttaaizzed HRL framework to naturally decompose perception feature extraction, cooperative behaviours, and fundamental actions.
In the previous work [3), embodied cooperation is completed via locomotion control by small number of robots. However, as the number of agents increases, previous methods fail to effectively address the issue, while the scalability of the swarm is crucial in multi-agent reinforcement learning (MARL) field [16, 17, 18, 19]. To tackle this challenge, we propose a fully-decentralizedd framework that can be applied to a flock of agents to enhance the method's ' scalability and promote impressive cooperative behavior among larger-scale intelligent agents. aim to apply this framework to an ex- pandable MAS. Moreover, spatiotemporal features between sequence of events are not considered in the previous work, which causes failure in understanding interactive and cooperative behaviours. To enhance this, we deploy sequential behaviour learning with spatiotemporal memory capabilities in the middle layer of the hierarchical network by constructing Recurrent Neural Network.
The key contributions of this work can described as follows:
A distributed HRL locomotion control framework is constructed for scale-expandable em- bodied cooperation. This architecture can be applied to both homogeneous and heteroge- neous groups.
We build set of scenarios for whole-body embodied cooperation in IsaacSim shown in Fig. 1, benchmark for embodied cooperation.
We showcase the remarkable emergence of cooperative behaviours among agents and demonstrate the advantages compared with previous approaches.
2 Related Works
2.1 Embodied Cooperation
Research cooperation be divided according with without body interaction. Some col- laborative tasks are based on particle games, focusing solely on coordination and alignment at position/velacity levels [20, 21]. In contrast, studies related to embodied cooperation lie on the foundations of the robotic platform that owns the real body, paying more attention to interaction, collision, and whole-body control. Optimization methods and computational intelligence methods can solve embodied cooperation problems. The model predicted control (MPC) based methods are widely used in formation control on quadruped robots [22, 23, 24, 25] and acrial robots [26, 27], and nature-inspired algorithms can obtain good performance in a self-organized swarm coopera- tive task [2]. These approaches provide highly accurate control but lack flexibility and generaliza- tion. Learning-based methods can compensate for this [28, 3]. Moreover, the large language model (LLM) can provide robots with powerful cognitive reasoning capabilities for collaboration, bringing a new way to solve the problem [29, 30, 31, 32].
2.2 HRL in locomotion Control
HRL is well suited for complex and spatiotemporal tasks [33]. Its effectiveness has been proven and widely acknowledged [34]. Multiple steps can decouple an intricate task to reduce the training difficulty, corresponding to various layers of policies [35]. Decisions between layers can be made

---

Upper Layer
UL
Distributed HRL for agent
Middle Layer
UL
Feature
UL
Feature
Lower I Layer Module
Module:
LL
ML
ML
Recurrent
Command
Command
LL
Propriocaptive Observvatoo::
Exterooetive
Observation:
Target
Action:
Self-agent
agents:
Object Trajectory.
Info flow:
Figure 2: Distributed HRL pipeline: demonstrate the information flow within the distributed hi- erarchical reinforcement learning, using scenario Cooperative Transport as an example. Here we maintain centralized training but distributed control hierarchical reinforcement learning frame- work, where we decompose the complex behaviours into three levels: Upper Layer (UL), Middle Layer (ML), and Lower Layer (LL.). UL module processes the external information e, including environmental perception and relative position to colleagues, extracting features and sending to ML; ML module is recurrent neural network layer that maintains a recurrent state considering tem- poral and spatial correlation, and outputs locomotion command into LL module; LL module is a pre-trained locomotion control layer that generates action a and applies it to agent according to the proprioceptive observation Pt, where have two modes: position and velocity. The goal of Cooperative Transport is to move a cylinder object to the red target zone collaboratively by a group of Ant robots.
by deciding whether the previous action has been completed [36]. Essentially, hierarchical learning provides artificial direction and reduces the need for model to have a thorough understanding of the scenario and the tasks, preventing RL from wandering in a sparse search space.
Furthermore, HRL inherently well-suited for tasks that involve progressive logical relation- ship [37, 38]. A typical work fuses simplified environmental observation and proprioceptive state then generates lower-level control commands for actuators[39]. Some recent attractive benchmarks like HumanoidBench [40], also divide tasks into planning and reaching control and learning hierar- chically.
3 Problem Statement
We first formulate the problem of learning embodied cooperative behaviour in swarm of agents, then try to solve this problem with MARL. Since the number of agents is fixed within an episode, the synchronous framework and policy gradient descent algorithms are selected. To realize real-time execution, we further proposed fully-decentralized training framework to support our implementa- tion. is crucial to mention that agents the group might be homogeneous or heterogeneous, and agents in the same species share the same policy.
The embodied cooperation problem can be formulated as Markov Decision Process (MDP) math- ematically expressed as A, R) including N agents M heterogeneous species. S.A are state and action tuples, respectively, where S = (S1, S2. ... SN)), = (A1, .......................... ... ANy) ).). T the state transition function, and T (s))),a) represents the transition probability distribu- tion of the system when transiting into the next states s' (s( 8/2 sxx) S from states (S1, 82, 82, SN) E S after taking set of actions R (R,.... RM) denotes the tuple of reward functions in which different types of agents corre-
(a1, 02, .NN) E A. Besides,

---

spond to distinct reward functions, R3 (st, as) is the reward given to agent after taking actions a, under states 8.
For the homogenous multiple agents group, the reward function R determined and shared for each agent. The aim find the best shared policy n², maximize the expected average value of cumulative discount reward:
N N
π argmaax/ =
7 E ([)
(1)
[m 20
where oj E S; and of A, are the state and action of the i-th agent. There are some subtle differences in the representation of the heterogeneous population. The aim find the best policy tuple s (M (T) to maximize the joint multiplication of costs J:
M
K
Ω = argmiax
II Jm, where Jm m=1
Σ 1=1 1=0
7
E Rm (8), aj) TTm.
1
K
where Jm denotes the cost function of type species, and K' is the agent number of this species. The final optimization cost for the heterogeneous population is a form of co-multiplication because task assignment exists among the entire team. Each subtask must be completed otherwise the overall task will fail. Even some subtasks obtain high rewards, any failure in a subtask will prevent the achievement of the overall goal.
4 Distributed Hierarchical Reinforcement Learning
As aforementioned, locomotion control for embodied cooperation is an challenging task. There three main difficulties: (i) How to decompose and decouple the complicated tasks into step-by-step mode? (ii) Whether the trained models can expand to a large scale of multiagent collaboration? (iii) How to establish the temporal and spatial correlation among collective behaviors? In this part, we will address each of these three questions in detail.
4.1 Task Decomposition and Hierarchical Learning
Network Structure: Traditionally, hierarchical neural network structure divides tasks into trol and decision layers. Unlike this, in order to address collaborative tasks, propose deeper hierarchical structure consisting of three layers. We designate perception-related tasks as the upper layer of the three-layer network that extracts exteroceptive observation including other agents' in- formation into feature vector; then, we define the middle layer of the network as the planning and decision layer, which provides spatiotemporally correlated commands to the lower-level controller; finally, the lower-layer network focuses on the execution commands, which can be pre-trained in various modes. The overall information flow in the proposed hierarchical network is illustrated in Fig. 2.
Hierarchical Perception Processing: Each layer of the network is responsible for processing dis- tinct types of raw data, which relates to how we acquire and classify perceptual information. Propri- oceptive perceptual states including local position, velocity, and joint torque are closely associated with personal locomotion control. Therefore, the proprioceptive state p should only processed by the lower-layer policy. Furthermore, the exteroceptive state formed by concatenating environmental perception which provides terrain information in the form of height measurement array [41] with relative observations of other agents [5], depicted in Fig.3.
4.2 Distributed Scalable HRL
In contrast to previous studies that employed HRI. or end-to-end approaches for robotic coopera- tive manipulation locomotion, our distributed HRL method considers the capability of scaling the population. To achieve this, need to strictly define the format of input and output informa- tion for each agent, particularly the observation. Herein, propose to partial observation of

---

Others
Neighber Agent
Obstacle
relative obs state
envirommental obs state
Figure 3: Exteroceptive state configuration in distributed HRL.
agents related their colleagues, according their visibility [42]. Each agent can only obtain the nearest neighbour's relative position. As illustrated in Fig.2, the dimension of exteroceptive state limited. This actually changes the difficulty of collaboration learning. Since partially observed agents can only see nearby peers and environmental information, they need to estimate the states of other invisible agents from the dynamics of the environment and predict future changes in the environment. However, such observation settings theoretically enable our framework to scale to a significantly large amount of agents system, which unattainable by CTDE frameworks.
4.3 Spatiotemporal Memory Recurrence
Capturing and leveraging spatiotemporal features is crucial for long-term decision makinn tasks [43] particularly the complex embodied cooperation tasks we designed. We anticipate that fully- decentralized agents should possess the capability comprehend their own and other agents' con- tinuous behaviors, thus forming collaborative relationships in space across continuous time inter- vals. Furthermore, this capability holds particular significance in dynamic environments, espe- cially within fully-decentrallized nonstationary multi-agent environments. Therefore, we propose spatiotemporal memory recurrence utilizing RNNs to enhance the agent's 's ability to make informed decisions over extended periods and complex environments. Consequently, we maintain a hidden recurrent state to pass the feature of scenarios continuously, as shown Fig. 2.
4.4 Training Curriculum
Acquiring collaborative behaviours requires the agents' random search to span a sufficiently large space, which complex tasks can often lead to an inability to learn usable knowledge due to the vastness of the search space. In this case, we design some appropriate training curricula that rea- sonably sparse rewards and dense rewards in the early stages to intentionally guide the agent to approach success. The curriculum designs are described in detail in the appendix.
Experiments
The experiments are conducted on a workstation with Intel Xeon W-2150b and RTX2080Ti. Inde- pendent Proximal Policy Optimization (IPPO) multi-agent setting is employed as the algorithm. Adam optimizer is used with learning rate of 0.0005. PPO clipping 0.2, the discount factor is 0.995. We could complete the training within several hours thanks to IsaacSim GPU parallel sampling via multiple environments [4].
5.1 Environments Construction
Three scenarios established that represent different task types: Cooperative Transport, Corridor Crossing and Ravine Bridging.
Transporting an object collaboratively is a classical task in the multiagent system field. Herein, we build an embodied cooperative transport task via locomotion control. A group of Ant attempt move cylinder object to the red target area, shown in Fig. (a)) We consider fully decentralised and distributed cooperation in homogeneous and self-organized flock. Thus, the scale of flock can have theoretically unlimited expansion.
Cooperative Transport:

---

Corridor Crossing: A group of agents navigate through narrow corridor sequentially, aiming to complete the passage as quickly as possible. Only one person can pass through the slit at any one time. Agents in this scenario should adjust their position and speed reasonably according to the surrounding observations and pass through narrow passage without collision. Cooperation in formation is required, as shown Fig. 1(b). The number of agents can also be expanded in this task.
As illustrated in Fig. I(c), two agents push a movable bridge, enabling another group of agents to cross the chasm smoothly via the bridge. There are two categories of agents in this scenario, which is different from the other two environments. Agents in Ravine Bridging are heterogeneous, which means we should maintain more than one policy for different missions.
Ravine Bridging:
5.2 Results of Distributed HRL
5.2.1 Training Basic Locomotion Control
In the training of lower-level motion control, two control paradigms are available: velocity con- trol and position control. We first train the two control paradigms in the single-agent environment to obtain accurate control of the fundamental actions. We give reciprocal L2 distance between the current position and the target position, or the dot product of the agent velocity and target ve- locity, as the main reward in each training curriculum. Fig 4(a) shows the training results of two paradigms. After training, We find that the velocity control offers greater flexibilityy than position control. However, position control presents higher accuracy. Moreover, velocity control makes it difficult to maintain stationary position at specific point.
Layer
Cooperative Transport
Drossing
Revine Bridging
3000
Anc Incrgg
1000
Tin:Pap
TimoSlap
TimeBkoo
TimeStpp
(a) Basic Locomotion
(b) Cooperative Transport
(c) Corrider Crossing
(d) Ravine Bridging
Figure 4: Distributed HRL results for all scenarios.
5.2.2 Training the Interactive Collaboration
After the lower layer is elaborately trained,we we adopt the position control paradigm and freeze this network. Then, the upper and middle layers are trained on different scenes via distinct reward mechanisms. The training results are shown in Fig. 4(b,c,d).
This task reveals clear collaborative behaviors among agents. They demon- strate a strong understanding of the situation, actively choosing tasks based on their circumstances. Some agents first converge towards the object and slightly adjust their position for collaboration; meanwhile, other agents move directly towards the target and prepare decelerate the object, en- suring falls within the target area accurately and stably. The success rate of this task reaches approximately 75%.
Cooperative Transport:
Corridor Crossing: Agents in this task naturally develop characteristics of: "sacrifice" or "conces- sion". If the model pre-trained in a single-agent scenario is directly applied to a multi-agent scenario, all agents would get stuck in the corridor, unable to move. However, after training, agents learn to observe the relative positions of others and adjust their speed accordingly, choosing the appropriate timing for moving or waiting. Typically, agents in the middle rush through the corridor, while others approach the entrance more slowly and accelerate after observing the passage of agents ahead. The success rate of this task reaches approximately 88%.

---

Ravine Bridging: The group in this task is a heterogeneous team. Although all agents are Ant, there are two sets of policies: one for agents on the plane and one for agents in the ravine. Agents on the plane will arrive at the edge of ravine and stop, waiting for the agents in the ravine to transport the bridge to the front of The success rate of this task reaches approximately 50%. The main reason that Ants the ravine might get stuck or fall.
Table I: Success Rate Comparison: We present success rates with standard error on the three tasks, comparing our method to ablations. Each pereentage results from 500 trials- - 100 attempts per five independently trained models.
Success rate
HRL
No Hierarchy
No Spatiotemporal Memory
Cooperative Transport
76.8 5.3%
0.0 0.0%
9.6 3.2%
Corridor Crossing
88.4 :: 4.8%
0.0 + 0.0%
12.0 = 4.1%
Ravine Bridging
50.4 + 4.4%
0.0 + 0.0%
5.4 + 4.2%
5.3 Ablation
5.3.1
The role of Hierarchical Learning
In this experiment, we replace the hicrarchical networks with single-layer MLP. Given no hicrar- chical structure, there is no form of target command and velocity command, which precludes the possibility of training identical lower-level controller in advance. Therefore, when training with- out hierarchy, provide the same training curriculum tailored to each task, as comprchensively detailed in the Appendix. Furthermore, we provide proprioceptive states and exteroceptive states as observations and the same task reward and locomotion reward as the total rewards.
We find that agents can only learn simple strategy and get the basic reward which are easy to reach. These results show that hierarchy is crucial to success in more complex tasks that require coordina- tion. Without hierarchy, agents cannot complete the mission (in Table. 1). The reason for failure is obvious. As with other HRL objectives, single network facing high-dimennional observations and complex multi-item reward functions finds it difficult to distinguish features between channels and learn behaviour guided by a single main reward. Especially, we need frozen lower-level locomo- tion controller, which is the foundation for success.
5.3.2 The effectiveness of Spatiotemporal Memory
To validate the effectiveness of spatiotemporal features, we re-product the method in previous work [3] as the baseline, which does not consider the event correlation. We provide the same low- level locomotion controller. The approach proposed in [3] which does not have RNNs, converges to a slightly higher reward than in the case of no hierarchy but still far from completing the mission. As shown Table. the success rate of this method maintains a relatively low level which rarely exceeding 15%. We find that agents cannot learn set of systematic behaviours in specific tasks, and there are occasional successes. Actually, if the agents only react to the current observation, they could never form strategic behaviours. In other words, he would never think for himself in the next second without spatiotemporal memory. This is even more deadly in collaborative tasks, as we can obviously find that the conciliatory and sacrificial behaviour emerging in crossing corridor tasks is a strategy of thinking for long-term consequences.
5.4 Scalability Experiments
We emphasize the advantages our proposed fully- cully-decentralized approach in terms of scalability, which includes the advantage of training when the number of agents is large, as we do not need to provide the global observations of all agents like CTDE framework during training to avoid the huge increase in training complexity. On the other hand, our framework also has advantages the ability of directly transferring the trained model to more agents tasks.
7

---

First, we demonstrate the capability of our framework to effectively train agents with nice per- formance as the numher of agents increases, even when these agents only have access to partial observations. We validate this in the task cooperative transport as illustrated in Fig. 5(a). The final reward increments as the number of agents rises, which is also in line with our expectations for this task. The participation of more agents should indeed lead better transportation results.
Second, we conducted zero-shot transfer tests on our trained model (trained with four agents) across diverse populations of Ant in each task(Agent number indicates the agent on the plane in the task Ravine Bridging), and the success rates of each task with different populations are shown in the Tab. 5.4. The outcomes demonstrate the potential of our framework's 's transfer capability.
Success rate
Two Ants
Three Ants
Four Ants
Five Ants
Cooperative Transport
12.8 5.6%
39.8 i 4.5%
76.8 + 5.3%
82.2 - 4.5%
Corridor Crossing
97.1 = 1.8%
93.6 + 2.2%
88.4 + 4.8%
73.5 + 2.5%
Ravine Bridging
50.4 + 4.4%
36.2 + 7.8%
22.3 5.1%
10.9 3.3%
Third, we conduct the experiment to show the advantages of our fully decentralized method training advantage over the CTDE methods when the number of agents increases. The most appropriate task to show is Corridor Crossing because as the number of agents increases, the difficulty of the task Cooperative Transport decreases, while Ravine Bridging too complex and the completion rate is too low when the number of agents is large. So add the number of agents to 6 and 7 in corridor crossing. And the Fig. 5(b)cc) shows that the reward of centralized training grows slower and ultimately not converge.
Scalability
Comparision with CTDE
Comparision with
2008
Avarage Epiands Roward
Average Epiarde Reward
Avannge Epleads Reward
1000
2000
-200
Two
Diriibeedd Hi (PPO)
200
Taak
Centmibeed TriningMA/PP))
Conmrized TONP
TimeStep
TimeStep
TimeStep
(a) Training results for different Ant populations
(b) Training with six agent num- ber
(c) Training with seven agent number
Figure 5: (a) shows the training results of different Ant populations used in Cooperative Transport (b)(c)shows the result training with large agent numbers.
6 Conclusion and Limitations
In this work, we propose distributed HRL architecture for embodied cooperation, and the results showcase the effectiveness of our method in embodied and whole-body control collaborative tasks.
Despite the encouraging findings, is essential to recognize the limitations our research. Firstly, although the hierarchical network method endows our framework with certain ability to address lorg-horizon problems, its capacity to solve complex issues remains limited as all layers are still fully trained based reinforcement learning. We primarily relied continuous reward shaping and increasing the total training steps obtain better training outcomes. We contend that if we aim to solve more complex long-horizon cooperative tasks with the participation of more intelligent agents, additional planning-based guidance might be necessary at the upper level.
Moreover, our work indeed more algorithmically and simulation Senchmark-orientte and we have not explored robots with more complex morphology. Despite having already demonstrated the promising results of our method some tasks and witnessed the emergence of cooperative behavior, there is still a considerable engineering workload required in terms of control and sim2real transfer.

---

Acknowledgments
This work was supported by the National Natural Science Foundation of China under Grants 62025304.
References
[1] K. Huang, J. Chen, J. Oyekan, and X. Zhang. Bio-inspired Multi-agent Model and Optimiza- tion Strategy for Collaborative Aerial Transport. Lecture Notes Electrical Engineering, 801 LNEE:591-598 2022. ISSN 18761119, doi:
[2] K. Huang. J. Chen, and Oyekan. Decentralised aerial swarm for adaptive and energy efficient transport of unknown loads. Swarm and Evolutionary Compulation, 67, 2021. ISSN 22106502. doi: wwwp::000006.sswo22222......
[3] O. Nachum, M. Ahn, H. Ponte, Gu, and V. Kumar. Multi-Agent Manipulation via Locomo- tion using Hierarchical Sim2Real. Proceedings of Machine Learning Research, 100:110-124 2019. ISSN 26403498.
[4] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning. 2021. URL http:://x///xxi wwtpv/.rgggbbbb/..ooooo 10470.
[5] K. Huang, D. Guo, X. Zhang. X. Ji, and H. Liu. Stimulate the Potential of Robots via Compe- tition. 2024. URL wtt.::ttp///xiiiii
Frrg/abs/2490 10487.
[6] K. Huang, D. Guo, X. Zhang, X. Ji, and H. Liu. CompetEvo: Towards Morphological Evolu- tion from Competition. 2024. URL http:://// arxiv org/ abs//405. 18300.
[7] C. Yu, A. Velu, E. Vinitsky, J. Gao, Y. Wang, A. Bayen, and Y. Wu. The Surprising Effective- ness of PPO in Cooperative Multi-Agent Games. Advances in Neural Information Processing Systems, 35, 2022. ISSN 10495258.
[8] R. Wang, D. Zhao, A. Gupte, and B. C. Min. Initial Task Allocation in Multi-Human Multi- Robot Teams: An Enbenceon-Enhanned Hierarchical Reinforcement Learning Approach. IEEE Robotics and Automation Letters, (4):3451-33) 2024. ISSN 23773766. doi: 109/LRA. 2024.3366414.
[9] X. Liu, G. Wang, 7. Liu, Y. Liu, Z. Liu, and P. Huang. Hierarchical Reinforcement Learning Integrating With Human Knowledge for Practical Robot Skill Learning in Complex Multi- Stage Manipulation. IEEE Transactions on Automation Science and Engineering, 2023. ISSN 15583783. 1oi: 109/TASSF.22233388333
[10] L. Chang, L. Shan, W. Zhang, and Y. Dai. Hierarchical unknown environments via deep reinforcement learning and distributed optimization. Robotics and Computer- Manufacturing, 83, 2023. ISSN 07365845. doi: 10. 10.1016/).rcim.2022 102570.
navigation and formation in
[11] C. Hao, C. Weaver, C. Tang, K. Kawamoto, M. Tomizuka, and W. Zhan. Skill-Critie: Refin- ing Learned Skills for Hierarchical Reinforcement Learning. IEEE Robotics and Automation Letters, 9(4):3625-3622 2024, ISSN 23773766. 10. 100..00000R
[12] S. Wang, Z. Wu, X. Hu, Y. Lin, and K. Lv. Skill-Based Hierarchical Reinforcement Learning for Target Visual Navigation. IEEE Transactions Multimedia, 25:8920-8932. 2023, ISSN 19410077. doi: 10. 2009TMMM.2023332436118
[13] E. Triantafyllidiss F. Acero, Liu, and Z. Li. Hybrid hierarchical learning for solving com- plex sequential tasks using the robotic manipulation network ROMAN, Nature Machine Intel- ligence, 109)9991-1055 2023. ISSN 25225839. 110:10..108//422--00-000000

---

[14] X. Huang, Z. Li. Y. Xiang, Ni, Y. Chi, L., Yang. X. B. Peng, and K. Sreenath. Creating Dynamic Quadrupedal Robotic Goalkeeper with Reinforcement Learning. IEEE International Conference on Intelligent Robots and Systems, pages 2715-2722, 2023. ISSN 21530866. doi:
10. N..1009/R055552..2222 2023. 00........119995 10341936.
[15] J. Cao, L. Dong, and C. Sun. Hierarchical reinforcement learning for kinematic control tasks with parameterized action spaces. Neural Computing and Applications, (6(1):323-334) 2024. ISSN 14333058. 1111:1..1077/0051-000------000
[16] S. Mohanty, E. Nygren, F. Laurent, M. Schneider, Scheller, N. Bhattacharya, J. Watson, A. Egli, C. Eichenberger, C. Baumberger, G. Vienken, I. Sturm, G. Sartoretti, and G. Spigler. Flatland-RL: Multi-Agent Reinforcement Learning on Trains. 2020. URL http://arxxi.. erg/ arg/abs/2014 20822
[17] M. Parvini, M. R. Javan, N. Mokari, Abbasi, and A. Jorswieck. Aol-Aware Re- source Allocation for Platoon-Based C-V2X Networks via Multi- Multi-Task Reinforce- ment Learning. IEEE Transactions on Vehicular Technology, 72(8):8880-988 2023. ISSN
19399359. doi: 11111TTTT.2022333556668
[18] Z. Xiong, B. Chen, S. Huang, W.-W. Tu, Z. He, and Y. Gao. Mqe: Unleashing the power of interaction with multi-agent quadruped environment. arXiv preprint arXiv:2403. /6015, 2024. URL https: /ttpe/////xiveeeeee wwreiii.orgg/aee btrrg/bbs/2400 16015.
[19] M. Bettini, A. Prorok, and V. Moens. Benchmarl: Benchmarking multi-agent reinforcement
learning. arXiv preprint arXiv:2312.01472, 2024. URL 01472.
[20] Y. Yang, R. Luo, M. Li, M. Zhou, W. Zhang, and Wang. Mean field multi-agent rein- forcement learning. 35th International Conference Machine Learning, ICML 2018, 12: 8869-8886, 2018.
[21] X. Ma, Yang, Li, Y. Lu, Q. Zhao, and Yang. Modeling the interaction between agents in cooperative multi-agent reinforcement learning. Proceedings of the International Joint Con- ference Autonomous Agents and Multiagent Systems, AAMAS, 2:853-861 2021. ISSN 15582914.
[22] Xu, W. Zhang, L. Zhu, and C. P. Ho. Distributed Model Predictive Formation Control with Gait Synchronization for Multiple Quadruped Robots. Proceedings IEEE International Conference on Robotics and Automation, 2023-May:9955-1002, 2023, ISSN 10504729. doi: N0.1109/ICRA4889 .2023. 10161260.
[23] J. Kim, R. Fawcett, V. R. Kamidi, A. D. Ames, and K. A. Hamed. Layered Control for Cooperative Locomotion of Two Quadrupedal Robots: Centralized and Distributed Ap- proaches. IEEE Transactions on Robotics, 39(6):4728-4744 4748, 2023. ISSN 19410468. doi: 10.1109/TR0.2023333319
[24] C. Yang, G. N. Sue, Z. Li, L. Yang, H. Shen, Y. Chi, A. Rai, J. Zeng, and K. Sreenath. Collaborative Navigation and Manipulation of Cable-Towed Load by Multiple Quadrupedal Robots. IEEE Robotics and Automation Letters, 7(4):10041-1044, 2022. ISSN 23773766.
doi:10.1109/1RA.2222..5 170.
[25] W. L. Ma, N. Csomay-Shanklin S. Kolathaya, K. A. Hamed, and A. D. Ames. Coupled Control Lyapunov Functions for Interconnected Systems, with Application to Quadrupedal Locomotion. IEEE Robotics and Automation Letters, 602):3761-3768, 2021. ISSN 23773766.
[26] L. Quan, L. Yin, T. Zhang, M. Wang, R. Wang, S. Zhong, X. Zhou, Y. Cao, C. Xu, and F. Gao. Robust and Efficient Trajectory Planning for Formation Flight in Dense Environments. IEEE
10

---

Transactions on Robotics, 39(6):4785-48888 2023. ISSN 19410468. doi: 109.0...99///0.2023 3301295: 295.
[27] S. Papaioannou, P. Kolios, C. G. Panayiotou, and M. M. Polycarpou. Cooperative simul- taneous tracking and jamming for disabling a rogue drone. IEEE International Confer- ence on Intelligent Robots and Systems, pages 7919-7926, 2020. ISSN 21530866. doi:
10.1 10.10000R04444432000005220
[28] S. De Witte, T. Van Hauwermeiren, T. Lefebvre, and G. Crevecoeur. Learning to Cooperate: A Hierarchical Cooperative Dual Robot Arm Approach for Underactuated Pick-and- Pick-and-Placing. Placing. IEEF/ASME Transactions on Mechatronics, 10.1007TMEECH.222223144444444
2022. ISSN 1941014X. doi:
[29] X. Guo, K. Huang, J. Liu, W. Fan, N. Vélez, Q. Wu, H. Wang, T. L. Griffiths, and M. Wang. Embodied LLM Agents Learn to Cooperate in Organized Teams. 2024. URL http://arxiv. org/abs/24000 12482.
[30] H. Zhang, W. Du, J. Shan, Q. Zhou, Du, B. Tenenbaum, Shu, and Gan. Building Cooperative Embodied Agents Modularly with Large Language Models. 2023. URL http: // //arxiv.ore orgg/vv.rr/aao abs /3077 02485.
[31] C. Sun, S. Huang, and D. Pompili. LLM-based Multi-Agent Reinforcement Learning: Current and Future Directions. 2024. URL httt:://axxiii wtt.:oog//////o.o 11106.
[32] X. Liu, D. Guo, X. Zhang, and H. Liu. Heterogeneous Embodied Multi-Agent Collaboration. IEEE Robotics and Automation Letters, 96):53777-538,, 2024. ISSN 23773766. doi:10. 109/ LRA.2024.3390588
[33] O. Nachum, Lee, S. Gu, and Levine. Data-cfficient hierarchical reinforcement learning. Advances in Neural Information Processing Systems, 2018- Deceemm333-331111111113333333331 2018. ISSN 10495258.
[34] C. Sun, W. Liu, and I.. Dong. Reinforcement Learning with Task Decomposition for Cooper- ative Multiagent Systems. IEEE Transactions Neural Networks and Learning Systems, 32 (5):2054-2065, 2021. ISSN 21622388. doi: 1111111NNSS..20022220.00...
[35] M. Eppe, C. Gumbsch, M. Kerzel, P. D. Nguyen, M. V. Butz, and S. Wermter. Intelligent problem-solving as integrated hierarchical reinforcement learning. Nature Machine Intelli- gence, 4(1):11-20) 2022. ISSN 25225839. 000:10..0000/422560011-00000
[36] X. Yang. Z. Ji, Wu, K. Lai, C. Wei, G. Liu, and R. Setchi. Hierarchical Reinforcement Learning With Universal Policies for Multistep Robotic Manipulation. JEEE Transactions on Neural Networks and Learning Systems, 2021. ISSN 21622388. doi: 119:0..111//NNNSS.2000 3059912.
[37] J. Wöhlke, F. Schmitt, and H. Hoof. Hierarchies of Planning and Reinforcement Learning for Robot Navigation. Proceedings IEEE International Conference Robotics and Au- tomation, 2021-May:10682-1000 10688, 2021. ISSN 10504729. 461:1001100/CRR4485666..0 9561151.
[38] Li, N. Lambert, R. Calandra, Meier, and A. Rai. Learning Generalizable Locomo- tion Skills with Hierarchical Reinforcement Learning. Proceedings IEEE International Conference on Robotics and Automation, pages 413-419, 2020. ISSN 10504729. doi: D0.1009/CRA44945.20000.
[39] N. Heess, G. Wayne, Y. Tassa, T. Lillicrap, M. Riedmiller, and D. Silver. Learning and Transfer of Modulated Locomotor Controllers. 2016. URL http: /atriv///ivvv/v/a org/ wwt//bs////aaeee abs/1610. 05182.

---

[40] C. Sferrazza, D.-M. Huang,
X. Lin, Y. Lee. and P. Abbeel. HumanoidBench: Simu- lated Humanoid Benchmark for Whole-Body Locomotion and Manipulation. 2024. URL.
http://arxiv.. www.xiii.org/bbs..... wttp./orgg/bbb//.....as 105066.
[41] N. Heess, D. TB, S. Sriram, J. Lemmon, J. Merel, G. Wayne, Y. Tassa, T. Erez, Z. Wang, S. M. A. Eslami, M. Riedmiller, and D. Silver. Emergence of Locomotion Behaviours in Rich Environments. 2017. URL Tttt::://arr..o.orr//aa//TTOT.. 02286.
[42] Amorim, T. Nascimento, P. Petracek, G. De Masi, E. Ferrante, and M. Saska. Self-Organized UAV Flocking Based Proximal Control. 2021 International Conference Unmanned Aircraft Systems, ICUAS 2021, pages 1374-1382, 2021. 9476847.
1 10/1010019//CUSSS1888880000
[43] Zhao, F. Huang, J. Lv, Y. Duan, Z. Qin, G. Li, and G. Tian. Do RNN and LSTM have long memory? 37th International Conference Machine Learning, ICML 2020, PartF166 168147115 11302-11312, 2020.
12

---

A Appendix
A.1 Environment Description
A.1.1 Cooperative Transport
The agents should cooperate to push a cylinder to a target location. The target position are initialized randomly. The cylinder is set in the middle of the map, and agents are initialized in the four quadrants of the cylinder respectively.
Observation The agents receive two sets of observations: (i) a set of "proprioceptive" states which would be given to the lower layer controller. This part of observation exactly the same in all agents in different tasks because all tasks use the same pre-trained lower layer policy, including their self- observed information, such as its local position, local velocity, angular velocity, orientation, body upright angle, joint force and its limitations, etc. (ii) set of "exteroceptive" features containing task- relevant information which would be given to upper layer controller. In this task, the observation includes the locations of the nearest agent, the current position of the cylinder, and the destination.
Rewards We set three types of rewards: (i) Cylinder distance reward: a term which is the reciprocal L2 distance of the agent to the target. This is the main reward of this task. It will increase when the cylinder becomes closer to the target. The increasing trend the same as the inverse proportional function. (ii) Agent cylinder distance reward: this term is to encourage agents firstly come closer to the cylinder so that they can learn how to push the object. (iii) Agent distance reward: a term used to guide the agent to walk towards the target. thereby obtaining a large cylinder distance reward.
A.1.2 Corridor Crossing
The agents expected to reach designated line through narrow slit. Two walls block most of the way forward, leaving only narrow corridor that only allows one agent to pass at a time. Yet all agents are at the same distance to the entrance of the slit, which means that agents would get stuck at the entrance of the slit together if nobody is slow or stop to wait for others to pass first. We expected agents can acquire a strategy so that all the agents could go through the corridor as fast as possible.
The lower layer controllers' observation is the same the last task. In this task, agents' upper layer controllers' observation includes the locations of itself and the nearest neighbour, the position, translation, and orientation of the walls.
Observation
Rewards Two rewards are set: (i) Destination reward: an agent has passed the finish line, all agents will receive a positive reward. It stimulates the strategy allowing as many agents as possible to pass through the corridor to the finish line. (ii) Distance reward to enter: term to encourage agents first come close to the corridor.
A.1.3 Ravine Bridging
The agents are expected to reach designated line where they must cross over ravine first. There is a ravine blocking the way of the agents. Agents cannot jump over climb out of the ravine. Agents need colleagues' help to come over the ravine. At this point, we place a stone of the right size and depth in the ravine to serve as a "bridge" to help the agents cross it. We pre-set two agents on either side of the stone to push the stone to help their colleagues cross the ravine and reach the destination. So this is a complex task that requires teamwork among heterogeneous agents.
The observation of the lower layer policy is similar to the previous tasks. Agents' up- per layer controllers' observation includes the locations of itself and the nearest agents, the position of the stone, and the position and depth of the ravine.
Observation
Rewards Two rewards are set: (i) Destination reward: if any agent has passed the finish line, all agents will receive a positive reward. It stimulates the strategy allowing as many agents possible to go through the ravine and get to the finish line. (ii) Bridging reward: term to encourage agents in the ravine carry the stone to the agent above which is the closest to the ravine.
13

---

A.2 Typical Result
A.2.1 Cooperative Transport
We analyse the typical result of task cooperative transport. To test the robustness of our method, we conduct more than one hundred trials with randomized reset positions and count the success rate then plot the scatter diagram, as illustrated in Fig. 6(b). Each point represents an initialized position of an agent, and the object is located at (0,0) point. Red points denote agents generated at these positions fail to complete the task, while each blue point denotes a success. We can observe that when the initial position is closer the object, the success rate tends be higher, and there only one failure case within 5-meter range of the object; conversely, the success rate decreases the distance from the object increases. This is because the probability of accidents is greatly reduced by short- distance tasks. It is worth mentioning that most of the failure cases result from the agent tipping over or pushing the object obviously to the side with uneven force during the pushing process. (ig. showcase the evaluation results under no hierarchy and no-spatiotemporal memory. Only few successes can be found the evaluation of no-spatiotemporal memory, and all cases fail in training without hierarchy.
HRL Distribution
No Hierarchy Distribution
No Memary Distribution
Sucress
Fature
(a) Distributed HRL
(b) No hicrarchy
(c) No memory
Figure 6: (a)(b)c) shows the success and failure distribution of task Cooperative Transport.
A.2.2 Corridor Crossing
We record the success rates of each Ant at its initial position separately. as illustrated in Fig. A.3. We have a higher success rate when the initial position is closer to the center, and a higher failure rate when the initial position closer to the edge. This is also explainable, as the Ant at the centre closer to the slit and does not need to adjust velocity.
Three
Rve
Succos Rotc (96)
Success Rate
X_Postion
X_PosRion
X
X_Postton
A.3 Comparison to Other MARL Alogithms
We conduct comparison between the proposed distributed HRL method to other state-of-the-an art MARL approaches based on the article [19]. Specifically, we compare our fully-decentralizeed method with MAPPO and MASAC in task Cooperative Transport(4 agents) since the Isaac Sim algorithm is nested within the RLGame and the CTDE algorithms supported by RLGame only MAPPO and MASAC. Fig. A.3 shows the results. We find that MAPPO converges a little faster than
14

---

distributed method because it is centrally trained with global observations. However, the CTDE al- gorithm has worse performance and take much more training time when facing a larger number of agents, which has been demonstrated in Fig. 5 of the scalability experiments part.
Algorithms Comparison
Average Episode Reward
5000
4000
3000
IPPO
2000
MAPPO
ISAC
1000
MASAC
Task Done
0.0
0.5
1.0
1.5
2.0
TimeStep
1e8
A.4 Training pipeline of Ravine Bridging
As we mentioned in Appendix A.2, the Ravine Bridging task more intricate, which means we have to provide more guidance. And the agent's behavior is divided into stages, so the training process must also be divided accordingly. Therefore, there series of pre-train courses to complete this task. For the agent in the ravine, its upper layer receives observations that include the position of the bridging stone, its own position, and the position of the Ant the plane closest to the ravine. The Ant on the plane receives observations that includes the position of the stone, its own position, and the x coordinate of the ravine and the finish line. Before training together, we train them separately first to obtain their featured skills. For the agents in the ravine, we randomize the position of the Ant closest to the ravine in the input and give the agent y-axis direction distance reward of the distance between the stone and the randomize Ant's position, so that the agent the ravine has the ability to move the stone to the closest Ant the plane. For the agents on the plane, we add reward for the x-axis direction distance the finish line and punishment for falling (death), so that it learns the ability to move straight towards the finish line and stop before falling into the ravine. Finally, we train all of them together, and once an Ant crosses the finish line, we give all of them large sparse reward. For all the reward precise equations mentioned above, we will list them in the following section.
A.5
Reward Details
A.5.1
Lower layer
Target paradigm
Reward Type
Formula
Distance to target reward
x
1+ 1oo.:ppowo.00mmmmaa.oooo00010110 -trpppoo///.l--rrrrooppp/////pp
Alive reward
0.5
Up reward
0.1
Action cost
0.005
action cost
Energy cost
0.05 x energy cost
Dof limit cost
0.1 dof limit cost
15

---

Velocity paradigm
Reward Type
Formula
Vel reward
5 clamp ::2 cmd direction, max 1)
Alive reward
0.5
Up reward
0.1
Action cost
0.005 x action cost
Energy cost
0.05 x energy cost
Dof limit cost
0.1 x dof limit cost
A.5.2 Upper layer
Cooperative Transport
Reward Type Cylinder-target distance reward
Formula
10 x
httppp/pww0.ppppooooooooooooooaa targettpogsslppoooooo 1|)
Agent-cylinder distance reward
5
1+ (agent.pos]:
Agent-target distance reward
5 x
1+
(agent.pos/:.0) (arg)]]aarggppoo::oooooo (agentppss/::111 [argppppp-oret.
Reach destination reward
50 (all agents)
Cross Corridor
Reward Type Reach destination reward
Formula
50 (all agents)
Agent-target distance reward
5
(agent,por) [.00] Parget.pos|:::e [+ (agent ppos:...s111111 [.1] (unget.pos[:.1]]
Ravine Bridging
Noward/Punishmeen Type
Formula
Ravine pretrain: Distance of stone and nearest agent
5 x
I+ Istonc.y -agentyy)
reward
Plane pretrain: Distance destination line reward
5 x
I+ |esstination taatitionnx--peeee agentxx
Plane pretrain: Falling
-20
down punishment
Integrated training: Pass destination line reward
50 (all agents)
16


Reviews:
[{"review_id": "CLoLUtZIbo", "reviewer": "Reviewer_pYHV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a distributed hierarchical reinforcement learning (HRL) approach for multi-agent cooperation tasks. The policy network is a three-layer hierarchical structure: feature extraction from exteroceptive observations from environment and neighboring agents, spatiotemporal decision-making using recurrent neural networks (RNNs), and low-level action control. The method is evaluated on three custom-built environments within IsaacSim/Gym. The experimental results demonstrate the emergence of cooperative behaviors.", "summary_of_recommendation": "The paper presents a sound approach to MARL systems and some additional experimental results will strengthen the claims in the paper.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "1. The authors claim the method is scalable and can be applied to both homogeneous and heterogeneous groups. However, the experiments only demonstrate scalability within the homogeneous setting (Cooperative Transport). Can the authors provide additional results or analysis to support the claim of generalizability to heterogeneous groups? The scalability result considers up to 5 agents. Can it be demonstrated that baseline methods cannot scale to 5 agents?\n2. While the paper provides a baseline comparison to a previous work [3], it would be beneficial to compare the proposed distributed HRL method to other state-of-the-art MARL approaches to better contextualize the contribution and performance of the proposed method. Specifically, consider running this method on “BenchMARL: Benchmarking Multi-Agent Reinforcement Learning” (https://arxiv.org/pdf/2312.01472) to get a comparison against many MARL algorithms.\n3. The current evaluation is conducted in simulation. Could the authors discuss the potential challenges and steps involved in evaluating their method on real-world robots?\n4. In section 5.3.1, when discussing the role of hierarchy, the baseline network should be matched with the hierarchical network in the number of parameters for fair comparison. Please mention the network sizes. There should also be a comparison with a baseline RNN model without a hierarchy of similar size.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HSBPwI9bsu", "reviewer": "Reviewer_Mqnz", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper proposes a decentralized hierarchical structure for multi-robot whole-body locomotion and manipulation. The high-level layer is an observation encoder, the medium-layer is a hidden state encoder, i.e., RNN, and the low-level hierarchy is a pre-trained locomotion controller. The proposed method is validated on three multi-agent cooperative tasks built in IssacSim compared to one baseline.", "summary_of_recommendation": "I believe the presentation of this paper still needs to be improved and its contributions and novelties have not been sufficiently elaborated.", "technical_quality": 2, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "* How many seeds are used to demonstrate the results?\n* Why can the method with noisy hidden states sometimes outperform the baseline without using hidden states? What are the type and level of noises applied?\n* The authors mention that the proposed method has great potential to migrate a trained policy to a larger number of agents. Does any results support this argument?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "egi3eue7Lj", "reviewer": "Reviewer_gDjg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This work presents a hierarchical reinforcement learning architecture for multi-agent cooperative locomotion tasks. The hierarchical architecture consists of three layers. The first layer is responsible for perception, receiving exteroceptive information, and feature extraction. The second layer handles decision-making and planning, sending commands to the low-level controller. The third layer focuses on executing these commands. This controller is capable of performing cooperative tasks such as Ravine Bridging, Cooperative Transport, and Corridor Crossing. It has been demonstrated that these tasks cannot be successfully completed with a single layer; the hierarchical scheme is essential for success. However, a new policy must be trained for each new task.", "summary_of_recommendation": "The methodology of the paper and the simulation results are nice. However, the presentation could be improved, and validation using a real robot or a realistic robot morphology in the simulation is necessary. Therefore, I recommend a weak acceptance.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 3, "quality_of_limitations": 1, "questions_for_rebuttal": "Please check the above review comments for the questions.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The reviewers are relatively positive but not enthusiastic about the work. The reviewers highlight that the proposed approach and experiments are well motivated. The authors proposed design decisions for the network architecture / composition and observations that make sense for the task. The main concern of the reviewers is the thoroughness of the evaluation  the limited to non-existent baselines and the evaluation in simulation using unrealistic embodiment.   I personally want to add that the exact details of the ablation studies are not clear. For example what does no-hierarchy mean? Does this mean no pre-trained locomotion model in which case the comparison would be unfair. Furthermore  it is unclear whether the comparison is really clear as I would expect the baselines to solve at least the corridor task.  We encourage the authors to to better clarify the algorithm and experiments. Further, evaluation in real-world can strengthen the paper and is highly encouraged.
Decision: Accept