MBC: Multi-Brain Collaborative Control for Quadruped Robots
Authors: Hang Liu
Yi Cheng
Rankun Li
Xiaowen Hu
Linqi Ye
Houde Liu
Venue: CoRL 2024 (2024)
Keywords: Quadruped Robots
Perception Fails
Multi-Brain Collaborative
TL;DR: We propose the concept of Multi-Brain Collaborative Control based on Multi-Agent systems, establishing a quadruped robots training framework that achieves both perceptive motion and robust obstacle traversal in the event of perception failure .

Abstract: In the field of locomotion task of quadruped robots, Blind Policy and Perceptive Policy each have their own advantages and limitations. The Blind Policy relies on preset sensor information and algorithms, suitable for known and structured environments, but it lacks adaptability in complex or unknown environments. The Perceptive Policy uses visual sensors to obtain detailed environmental information, allowing it to adapt to complex terrains, but its effectiveness is limited under occluded conditions, especially when perception fails. Unlike the Blind Policy, the Perceptive Policy is not as robust under these conditions. To address these challenges, we propose a MBC:Multi-Brain collaborative system that incorporates the concepts of Multi-Agent Reinforcement Learning and introduces collaboration between the Blind Policy and the Perceptive Policy. By applying this multi-policy collaborative model to a quadruped robot, the robot can maintain stable locomotion even when the perceptual system is impaired or observational data is incomplete. Our simulations and real-world experiments demonstrate that this system significantly improves the robot's passability and robustness against perception failures in complex environments, validating the effectiveness of multi-policy collaboration in enhancing robotic motion performance.

MBC: Multi-Brain Collaborative Control for Quadruped Robots
Hang Liu"?,
Yi Cheng" Linqi Ye Houde Liu
Rankun Lii
Xiaowen Hu
TTsnnguua University, #University Michigan, SShanghai University
Equal Contributions Corresponding Authors
Page: https://quad mbc githubio
Destination (250m)
Origin
L (0m)
Lidar damage
We conducted long-distance test on our controller. At the beginning the map, the robot relied on height-map and proprioception to traverse through terrain. During the test, we simulated scenario where the lidar suddenly malfunctioned (by covering with orange bag). The robot did not experience any mode crashes and was still able to handle complex terrains effectively.
Figure I:
Abstract: In the field of locomotion task of quadruped robots, Blind Policy and Perceptive Policy each have their own advantages and limitations. The Blind Pol- icy relies on preset sensor information and algorithms, suitable for known and structured environments, but it lacks adaptability in complex or unknown environ- ments. The Perceptive Policy uses visual sensors obtain detailed environmental information, allowing it to adapt complex terrains, but its effectiveness lim- ited under occluded conditions, especially when perception fails. Unlike the Blind Policy, the Perceptive Policy is not as robust under these conditions. To address these challenges, we propose MBC:Mulli-Brain collaborative system that in- corporates the concepts of Multi-Agent Reinforcement Learning and introduces collaboration between the Blind Policy and the Perceptive Policy. By applying this multi-policy collaborative model a quadruped robot, the robot can maintain stable locomotion even when the perceptual system is impaired or observational data is incomplete. Our simulations and real-world experiments demonstrate that this system significantly improves the robot's passability and robustness against perception failures in complex environments, validating the effectiveness of multi- policy collaboration in enhancing robotic motion performance.
Keywords: Quadruped Robots, Perception Fails, Multi-Brain Collaborative
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
What happens if a robot suddenly loses its perception? Can it maintain its previous stable motion? In natural environments, there are instances of sensory failure in humans, such as "dark adaptation" phenomenon, but humans and animals can rely on past experiences to immediately switch to a state of motion without sensory input, ensuring safe movement. For humans, this ability stems from two main sources. First, the human brain has a strong adaptive and memory capacity. Secondly, the human control system has high degree of redundancy and multisensory integration. For exam- ple, when vision fails, the proprioceptive and vestibular systems enhance their role in maintaining balance movement.
In the motion tasks bipedal and quadrupedal robots, sensory systems may fail due to incomplete information hardware malfunctions. These robots rely on various sensors to gather environmental data, such LiDAR, cameras, and ultrasonic sensors. However, the effectiveness of these sensors can be limited in some circumstances. Therefore, researching how to maintain stable robot motion under these unfavorable conditions is a challenge in current studies.
In locomotion tasks, blind policies and perceptive policies each have their advantages and limita- tions [1]. Blind policies rely on sensors and preset algorithms for movement, requiring no visual input [2, 3, 4, 5]- Although they are fast and consume fewer resources, their adaptability in com- plex unknown environments is limited. Perceptive policies enable robots to adapt to complex terrains [6, 7. 8]. However, in less than ideal visual conditions, perceptive policies may not be as efficient as blind policies. Researching how to effectively merge these two policies to cope with complex environments an equally challenging research issue.
Addressing the challenges mentioned, this study integrates MARL [9, 10] to propose the concept of MBC: Multi-Brain Game Collaboration controller. We envision quadruped robot system in- tegrating multiple policies to form a collective "brain" with each policy tailored to different input policies. Specifically, we explore the interaction between Blind Policy, and a Perceptive Policy that utilizes external information. This model excels in scenarios with incomplete observational data or impaired sensory capabilities, accurately simulating and analyzing the robot's interactions with its environment. This approach enhances decision-making and adaptability complex envi- ronments. The primary contributions of this research are as follows:
A Novel Multi-Brain Game Collaboration System(MBC): This study introduces and successfully implements a multi-brain game collaboration system using MARL. In this system, each policy or "brain" independently and collaboratively optimizes decisions for different tasks.
Perception "Hot Swap": The research realizes "Hot Swaping" of external perception in quadruped robots control. Experiments in both simulation and real world have proven that this method can keep robust locomotion when sudden failure of external perception.
Enhanced Mobility in Complex Environments: Through the non-zero-sum game [11] between blind and perceptive policies, this policy allows the robot to make accurate and effective motion decisions.
2 Related work
Multi-Agent Reinforcement Learning In the field of Multi-Agent Reinforcement Learning (MARL), there are generally three learning paradigms: centralized learning, independent learn- ing, and Centralized Training with Decentralized Execution (CTDE) [12]. Among these, CTDE effectively combines the advantages of centralized learning with the flexibility of decentralized ex- ccution.MADDI [13] a typical representative of the CTDE paradigm, employing an actor-critic framework. However, as an off-policy algorithm, MADDPG requires extensive memory storage to previous experiences and may not perform stably in dynamic environments on-policy algorithms. MATD3 [14], multi-agent version of TD3, enhances the stability of multi-agent coop-

---

eration through double Q-learning and delayed policy updates, but this also increases computational complexity, especially in large-scale multi-agent environments, and is extremely sensitive to hy- perparameters, which may require extensive tuning and experimentation practical applications to achieve optimal performance.
MAPPO [15], for the first time, effecttivly extends the single-agent PPO algorithm to multi- agent environment, becoming an on-policy strategy that can handle complex multi-agent collabo- rations while maintaining the stability and efficiency of policy updates. MAPPO not only retains the advantages of PPO but also successfully addresses the collaboration problems in multi-agent environments. Its application on the SMAC platform demonstrates its high sample efficiency and consistency of policies [16].
Blind Policy & Perceptive Motion Policy In locomotion task of quadruped robots in complex environments, current research explores three primary policies. The first policy, termed the blind policy, relies on the robot's historical proprioception, primarily utilizing forelimb probing, to esti- mate terrain [3, 5, 2]. This policy faces limitations in complex environments like cliff due to percep- tion limitation. The second policy employs external sensory inputs to gather environmental details, helping the robot plan movements and navigate complex terrains [17, 18, 19, 20, 21]. However, this often involves isolated end-to-end network architectures without testing for sensor reliability. The third, composite policy [22, 23] integrates blind and visual policies into synergistic mechanism, quickly adapting to sudden failures in external perception systems.
In sim2real applications for vision-based motion controllers using reinforcement learning, two main approaches are prevalent: end-to-end training with depth or RGB images [21, 24J, effective in quadrupedal robots, and using elevation maps [25, 26] or height scans from Global Reference Frame. However, the former always faces the trouble of the long training time in simulation, camera noise, camera distortion, which introduce sim2real gap.
We introduce a novel framework that combining MAPPO in multi-brain system for quadruped con- troll. Our approach can accurately simulate and analyze the complex interactions between the robot and the environment, even under conditions of incomplete observation data or sensory loss, thereby enhancing the robot's motion performance in various environments.
3 Method
Blind Pretrain
All Tasks Train
Tipp
Hightaed
h
single Agent
Agents
af
Nind
Regularzzation
Blind Policy
Blind Policy
Perception Policy
a
of
pero
Figure 2: Two-stage multi-brain game collaborative training overview.
3.1 Task Formulation
In the locomotion task of a quadruped robot, we define a process that combines blind policy and an external perception-based policy to handle complex environments. Specifically, the quadruped robot flexibly navigate various obstacles such highlands, gaps, obstacles, and stairs when external perception (c.g., LiDAR elevation maps) is functioning properly. However, when external

---

perception suddenly fails, the quadruped robot, although unable to navigate terrains such as gaps, should still retain the capability to traverse complex terrains like stairs and ramps.
We have designed two-stage training approach, as illustrated in Figure In the first stage, a training mode without external perception is used, involving only blind policy. In the second stage, multi-agent approach is employed, incorporating external perception and simultancously training both the blind policy and perceptive policy with perception capabilities. The collaboration between these two policies is guided by a terrain reconstruction error regularization term. This ensures that our robot can effectively traverse terrains both with and without perception.
3.2 Base Set
Theorem We describe the locomotion problem of quadruped robots using Partially Observable Markov Decision Process (POMDP) [27, 28e.The POMDP framework effectively models decision- making scenarios where information is incomplete, defining key elements such as states, actions, observations, and rewards. In this model, the environment at time step f is represented by a complete state 24. Based on the agent's policy, an action a is performed, resulting in a state transition to 2(+1 with a probability P(z++1 Ie, de). The agent then receives reward ys and partial observation 0++1- The aim of reinforcement learning here is to identify policy that maximizes the expected discounted sum of future rewards:
o0
J(T) = Ex
Action Space & State Space The action spaces for the blind policy and perceptive policy are respectively of blind E I2 and af percep 12, representing the offset from the default posi- tion for each joint. The critic networks for both policies observe the global state sp spritic (ot, Rt; et, af percep di bbiind which includes proprioceptive observations ot, linear velocities et, height map he and latent variables e, such as body mass, center of mass position, friction coeffi- cients, and motor strength. These global observations are crucial for the second phase of training, helping the critic network make balanced decisions during the interactions between the two poli- cies and preventing training collapse due to excessive competition. For the actor networks, the state space for the blind policy includes proprioceptive observations ot, estimated linear velocity út and latent variables e,. Additionally, aligning with the multi-agent game theory approach, the state space for the blind policy also incorporates the output from the Perceptive Policy at pereepp expressed Iblndd = [og, tt, Ct, ot peroeepT. Similarly, the state space for the Perceptive Policy is s peroop [0r, lr, at blind'T where h; represents the local elevation map centered around the robot.
During the training for the Blind policy, we employed the Regularized Online Adaptation (ROA) method [29] to estimate the explicit observations é, and the latent variables e,. In this phase, at percep was set to zero. the second phase of training, the final action a at percep + a blind
3.3 Blind Pretrain
In the first stage of training, we primarily developed a proprioceptive motion system for the quadruped robot, aimed at enabling the robot to traverse various complex terrains such as uneven slopes, stairs, and discrete terrains without direct visual or elevation map input to the policy. During this phase, the output action of perceptive policy af aerrep was to zero, ensuring that the blind policy operates without interference from perceptive policy' outputs. Our blind policy, inspired by the ROA [29], uses history proprioceptive inputs to estimate the robot's explicit privileged infor- mation and implicit environment and dynamic information. Additionally, the training utilized an asymmetric Actor-Critic structure to better evaluate the quality of the actions output by the Actor
For the robot's elevation map, we trained a Variational Autoencoder (VAB) model primarily to mem- orize the terrains encountered by the blind policy and to compute regularization terms for action constraints in the subsequent training phase.

---

3.4
All Tasks train
In the second stage of learning, we introduced a multi-agent learning approach, utilizing a Non-zero- sum game thinking and MAPPO[15] to optimize the external perception controllers for quadruped robots. Unlike traditional single-policy approaches such as parkour[21], this method allows for adaptation when one controller fails, other controllers detect and adjust their actions, enhanc- ing the system's robustness. Additionally, this model supports "hot-swapping" of the perception system, enabling the robot to move based on sensory data when available and to continue proprio- ceptive movement without malfunction when perception is unexpectedly lost.
The primary implementation policy is as follows: As shown in Figure 2, we first load the pre-trained model of the single-agent blind policy as blind agent, and we initialize the perceptive policy as perceptive agent. Inputs the perceptive policy include proprioceptive data, outputs from the blind policy, and elevation map information, primarily adjusted for terrain. The robot's final actions are a combination of perceptive and blind actions. In the second stage, challenging terrains such as gaps, pillars, and highlands were introduced, which are difficult for the quadruped to traverse without external observation. The robots were encouraged to traverse these difficult terrain without collision. Specifically, although the terrains vary. We established a general reward structure as shown in 4. The robots will not be rewarded if they fail to follow the desired heading, ensuring they traverse the terrain instead of avoiding it. Additionally, the weights of the collision penalty were set high to encourage the robots to rely external observations in these challenging terrains rather than solely on proprioception.
This framework ensures that during training, the perceptive and blind policies interact and collabo- rate to optimize movement. All networks use the CTDE approach with MAPPO [15] updates, where each agent's Critic network shares all environmental information, including the inputs and outputs of other agents, during training, while each operates independently during execution. The loss cal- culations and updates for the blind policy remain as in the first phase, while the perceptive policy's loss includes surrogate loss, value loss, entropy loss, and a Reconstruction Error Regularizer, which will be explained in detail in 3.5.
3.5
VAE & Perception Cooperation Constraint Regularization
In the first stage, we trained the quadruped robot to nav- igate rough terrains without relying on external percep- tion. These terrains were chosen because they enable the robot to learn fundamental locomotion skills and to lift its legs and react to tripping, thereby enhancing overall mobility. We believe these terrains exemplify the types of environments a robot can navigate without perception in real-world scenarios. We employed a Variational Autoen- coder (VAB) encode and decode these features. We evaluate the VAE reconstruction error the first stage terrain after training and obtain the maximum reconstruc- tion error 7 during testing. The VAE was trained once in first stage and frozen in second stage.
Stage
Second Stage
Famillar Stair
Unfamilier Highland
Regularization
Perceptive Action
Perceptive Action
Blind
Blind Action
Regularization determines whether the terrain is familiar by recon- structing the elevation map, and guides and balances Blind Policy and Perceptive Policy
Figure 3:
In the second stage, we introduced more challenging ter- rains, such as highlands, gaps, and dense pillars, shown
in Figure 2, which difficult for the robot to navigate using only the Blind Policy trained in the first stage. In particular, the highlands require the robots to jump up and down, the gaps necessitate learning to jump over them, and the pillars demand that the robots avoid them before returning to their original direction. Therefore, it must rely on the Perceptive policy with external perception input for compensation.
However, the complexity introduced by Multi-Agent Learning can lead to policies converging to local optima, with the blind policy and Perceptive policy potentially competing against each other,
5

---

hindering coordinated control. To address this, we introduced a perception cooperation constraint regularization term based on elevation maps. This term helps ensure that the current elevation map reconstruction error, as produced by the VAE, is below a threshold т, indicating familiarity with the terrain, the regularization term increases with Percep policy's output, limiting its action. reconstruction error exceeds the threshold 7, indicating unfamiliar terrain, the regularization set to zero, encouraging the Perceptive policy to compensate.
Specifically, in the second stage, the robot's current elevation map his is input into the VAE, which reconstructs the elevation map hug. The reconstruction error is then calculated as E, 1 (he) huy)² where represents the i-th sample in the batch, j represents the index of the dimensions the elevation map and action, and represents the dimension of the elevation map. Based the reconstruction error and the threshold, define the penalty factor:
I 0 E S
This means that when the reconstruction exceeds the threshold, the regularization term is set to 1, otherwise it is O. The perception cooperation constraint regularization term is then introduced as:
m k P.
m
1=1
j=1
where & represents the dimension of the action, and represents the batch size. Finally, the tal loss function consists of the surrogate loss, value function loss, policy entropy, and the action regularization term:
C = Surrogate + Cvalue - AeH((T) + AaP
4 Experimental Results
4.1 Experiment Setup
We used the Unitree Go2 robot as our experimental subject. The training terrain comprised six types: ramps, stairs, discrete obstacles, highlands, gaps, and pillar terrain, as shown in Figure 2. Additional environment setup information and training specifics are detailed in the appendix. The purpose of our experiment is to prove whether our method can achieve the following behaviors:
Can the robot successfully navigate highlands, gaps, pillar with perceptual input?
Can the robot adapt and successfully navigate the first three types of terrain without any mode collapse when perceptual input suddenly fails?
4.2 Simulation Experiment
Terrain Passability Experiment: We first tested the survival rate our policy across last three tough terrains with varying levels of difficulty. For each terrain and difficulty level, we conducted four trials, with each trial consisting of 100 environment samples. We calculated the success rate for each trial and averaged these four success rates obtain the final experimental result. The success rate for the Gap and Pit terrains was defined as the robot successfully crossing climbing over the obstacle, while for the Pillar terrain, it was defined as the proportion of environments the robot navigated without collisions. As shown in Table 1, our policy achieved high success rates across various tough terrains. The highest difficulty level for each terrain was beyond the scope of our curriculum settings, demonstrating the robustness our algorithm.
Comparison Experiment: We evaluated the robot's 's ability to traverse complex terrains under per- ception failure conditions and compared methods with several basclines and ablations as follows:

---

Gap
Success Rate
Pit
Success Rate
Pillar
Success Rate
0.35m
99.3%
0.30m
97.6%
obstacle size=0,4: distance=1.6
86.7 %
0.45m
98.3%
0.40m
97.6%
obstacle size=0.5 distance=1.5
80.4 %
0.55m
91.3%
0.50m
85.0%
obstacle size=0.6 ; distance=1.4
65.0 %
0.65m
44.3%
0.55m
49.3%
obstacle sizc=0.7 ; distance=1.3
60.7 %
Table 1: Success Rates in Tough Terrains
Baseline: Training directly with proprioception and height map.
RMA[30]: Employing an Adaptation Module to estimate all privileged observations, but
directly inputting the elevation map into proprioception.
MLith|20]]: Utilizing GRU neural network as the Actor, with proprioceptive and extero-
ceptive inputs fed directly into the GRU.
Dust-listory(31)] Utilizing dual-history structure, where short-term and long-term his- tory observation are processed separately.
Ours w/o Regularizer: Training without Perception Cooperation Constraint Regulariza- tion.
As shown in Table 2, comparing with other methods, our method demonstrates the most robust performance under external perception failure, especially when climbing stairs. Other strategies failed to handle obstacles without perception, resulting in tripping over obstacles. In contrast, our method can easily climb steps, and the MXD indicates that our method can also follow the high desired speed (I m/s). Figure 3. shows the effect of our run in simulation. We believe that the difference in the success rate of going up and down stairs comes from the fact that going down stairs only requires considering the policy's ability to maintain balance, as gravity will guide the robot down, while going up stairs requires the robot to sense obstacles and determine the implicit type of obstacles in order to present a regular foot lift height.
We noticed that with the regularizer, the robots performed better when going upstairs and overall in MXD. This improvement is due to the regularizer guiding the collaboration between the blind policy and the perceptive policy across different terrains during training.
Method
Up Stair Success
Stuir Success
Discrete Success
Stuir MXD
Discrete
Ours
97 %
100%
90%
19.97
17.04
Ours w/o Regularizer
87%
100%
90%
16.42
14.99
RMA
0%
100%
81%
8.2
12.38
0%
100%
84%
14.61
Dual History
0%
100%
82%
10.9
13:77
Baseline
100%
76%
Table 2: we primarily compared the success rates of different methods on stairs and discrete terrains, as well as the Mean X-Displacement (MXD) for each environment. For this experiment, elevation map inputs were set to zero, desired velocity was set to lm/s, and we tested 1048 environments over 1000 steps. The stairs had a width of 0.31 and a height 0.13, while the maximum height of the discrete terrain was 0.15. Failure conditions were defined as either the roll or pitch exceeding 1.3, or the robot's foot getting stuck and unable to move forward. The optimal value for MXD is expected to be 20 meters.
4.3 Physical Experiments
Navigating Complex Terrains with Sensory Input Our policy substantially enhanced the quadruped robot's capability navigate vertical challenges, such wooden boxes and low walls. In our experiments, the robot tasked with climbing 32 cm high wooden box. It adeptly lifted its front legs preemptively and elevated its body to surmount the box, as shown in Figure 4. This sequence of movements, successfully culminating in the robot climbing over the box, exemplifies the efficacy of our integrated elevation map and perceptual policies in enabling the robot to tackle climbing obstacles.
In the obstacle avoidance trials, corresponding to the Pillar terrain used during training, the robot encountered various obstacles including trees, pillar, or human figures. Leveraging policy, it
7

---

Pre-lifting leg
Litting body
Contacting
Climbing
Figure 4: Climbing a wooden box with Lidar
quickly recognized a human-shaped obstacle through its elevation map, then adeptly adjusted its trajectory, sidestepping bypass the obstacle efficiently and safely, depicted in Figure 5. This performance underscores method's effectivenesss, particularly noting that despite the absence of y-direction velocity training, the robot adeptly maneuvered in the y-direction, showcasing the robustness and adaptability of our approach.
Identifying the obstacle
Preemptive turning.
Moving the y direstion
Returning original path
Figure 5: Avoiding person with Lidar
Long-Distance Test with Outdoor Terrain Perception Failure Initially, with effective LiDAR elevation map inputs, the robot used a comprehensive policy for movement, efficiently climbing 16 stairs and handling slopes. Subsequently, deliberately covered the LiDAR, disabling the elevation map input, and conducted long-distance test unstructured terrains. We tested the robot over a 250m path that included dense grass, irregular terrain, soft and slippery grasslands, gentle slopes, and stair terrains, where the robot successfully navigated through all (see Figure 1).
Quantitative Experiments We conducted quantitative ex- periments with the physical robot in both perceptive and non- perceptive scenarios. With Lidar input case, tested the performance of climbing and descent on highland, avoid- ing pillar-like terrain. For the scenario Lidar fails, The exper- iment includes stairs, slopes and discrete obstacle. Compared to the perceptive scenario, we defined more stringent metric, the No-Snagged Success Rate(NS-Success Rate), in which the quadruped robot successfully traverses the terrain without no- ticeable tripping stalling. Each terrain was tested times. During the experiments, in Table 3, the robot can not only traverse challenging terrain using LiDAR input, but can also maintain smooth movement in difficult terrain in the event of a sudden LiDAR failure. Please refer to supplementary for experiment videos and detailed experiments setup.
Perception Work
Success Rate
Highland
0.40
Pillar
0.60
Perception Fail
NS-Success Ratef
Upstairs
0.90
Downstairs
1.00
Slope
1.00
Discrete
0.80
Table 3: Real-World Quantitative Ex-
periments:
used hood to cover the Lidar simulate perception failure. As shown
Conclusion, Limitations and Future Directions
We propose the concept Multi-Bran Collaborative ConIrol(MBC) based on MARL, establishing training framework that achieves both perceptive motion and robust locomotion when percep- tion fail. We tested system in both simulations and real-world experiments, demonstrating the effectiveness and robustness of our algorithm. However, our robot's elevation maps are derived from LiDAR, which heavily depends on the frequency and stability of the odometry, and involves significant computational overhead. The laser radar odometer has limited accuracy in high-speed, high-frreqeene and vibration-pront scenarios, which results in a lower success rate in our physi- cal experiments than in simulation experiments passing through challenging terrain, as shown in Table In the future, we aim use end-to-end method construct local elevation maps without relying odometr[[22[[1]]. We will also explore how to apply algorithm control various legged robots.

---

Acknowledgments
The research is supported by the National Natural Science Foundation of China under grants No.92248304 and Shenzhen Science Fund for Distinguished Young Scholars under Grant RCJC20210000001196600
References
[1] M. T. Shahria, M. S. H. Sunny, M. I. Zarif, J. Ghommam, S. Ahamed, and M. Rah- man. comprehensive review of vision-based robotic applications: Current state, components, approaches, barriers, and potential solutions. Robotics, 11(6):139, 2022.
[2] I. M. A. Nahrendra, B. Yu, and H. Myung. Dreamwaq: Learning robust quadrupedal lo- comotion with implicit terrain imagination via deep reinforcement learning. In 2023 IEEE International Conference on Robotics and Automation (ICRA), pages 5078-5084. IEEE, 2023.
[3] J. Lee, J. Hwangbo, I.. Wellhausen, Koltun, and M. Hutter. Learning quadrupedal locomo- tion over challenging terrain. Science robotics, 5(47):eabc5986, 2020.
[4] Y. Cheng, H. Liu, G. Pan, L. Ye, H. Liu, and B. Liang. Quadruped robot traversing 3d complex environments with limited perception, 2024.
[5] J. Siekmann, K. Green, J. Warila, A. Fern, and J. Hurst. Blind bipedal stair traversal via sim- to-real reinforcement learning. arXiv preprint arXiv:2105.083226 2021.
[6] Y. D. Yasuda, L. E. G. Martins, and A. Cappabianco. Autonomous visual navigation for mobile robots: systematic literature review, ACM Computing Surveys (CSUR), 53(1):1-34, 2020.
[7] Fankhauser, M. Bjelonic, C. D. Bellicoso, T. Miki, and M. Hutter. Robust rough-terrain locomotion with quadrupedal robot. In 2018 IEEE International Conference Robotics and Automation (ICRA), pages 5761-5768. IEEE, 2018.
[8] P. Fankhauser. Perceptive locomotion for legged robots in rough terrain. PhD thesis, ETH Zurich, 2018.
[9] D. Huh and P. Mohapatra. Multi-agent reinforcement learning: comprehensive survey. arXiv preprint arXiv:2312.10256. 2023.
[10] S. Gronauer and K. Diepold. Multi-agent deep reinforcement learning: a survey. Artificial Intelligence Review, 55(2):895 0005,993,1111 2022.
[11] D. B. Gillies. Solutions to general non-zero-sum games. Contributions to the Theory of Games, 4(40):47-85; 1959.
[12] C. Zhu, M. Dastani, and S. Wang. A survey of multi-agent reinforcement learning with com- munication. arXiv preprint arXiv:2203.om 0.90000.779..... 2022.
[13] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor- critic for mixed cooperatiie--ompetiive environments. Advances in neural information pro- cessing systems, 30, 2017.
[14] Ackermann, Gabler, T. Osa, and M. Sugiyama. Reducing overestimation bias in multi- agent domains using double centralized critics. arXiv preprint arXiv:1910.01665, 2019.
[15] C. Yu, Velu, Vinitsky, Gao, Wang, A. Bayen, and Wu. The surprising effectiveness of ppo in cooperative multi-agent games. Advances in Neural Information Processing Systems, 35:2461 1-24624, 2022.

---

[16] J. Hu, S. Hu, and S.-w. Liao. Policy regularization via noisy advantage values for cooperative multi-agent actor-critic methods. arXiv preprint arXiv:2 14334, 2021.
[17] I. Miki, J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning robust per- ceptive locomotion for quadrupedal robots in the wild. Science Robotics, 7(62))eab22822, 2022.
[18] W. Yu, D. Jain, A. Escontrela, A. Iscen, P. Xu, E. Coumans, S. Ha, J. Tan, and T. Zhang. Visual- locomotion: Learning to walk on complex terrains with vision. In 5th Annual Conference Robot Learning, 2021.
[19] C. Mastalli, I. Havoutis, M. Focchi, D. G. Caldwell, and C. Semini. Motion planning for quadrupedal locomotion: Coupled planning, terrain mapping, and whole-body control. IEEE Transactions Robotics, 3666):1635-1648, 2020.
[20] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. Conference on robot learning, pages 403-415. PMLR, 2023.
[21] Z. Zhuang, Z. Fu, J. Wang. C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning. arXiv preprint trXXiv2309.05666 2023.
[22] H. Duan, B. Pandit, M. S. Gadde, B. J. van Marum, J. Dao, C. Kim, and A. Fern. Learning vision-based bipedal locomotion for challenging terrain. arXiv preprint arXiv:2309. A0Mv:2239.11594. 2023.
[23] Z. Fu, A. Kumar, A. Agarwal, H. Qi. J. Malik, and D. Pathak. Coupling vision and propriocep- tion for navigation of legged robots. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17283-17283, 2022.
[24] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots. arXiv preprint arXiv: 14341, 2023.
[25] Fankhauser, M. Bloesch, C. Gehring, M. Hutter, and R. Siegwart. Robot-centric elevation mapping with uncertainty estimates. In Mobile Service Robotics, pages 433 433-440. World Sci- entific, 2014.
[26] T. Miki. L. Wellhausen, R. Grandia, Jenelten, T. Homberger, and M. Hutter. Elevation map- ping for locomotion and navigation using gpu. In 2022 IEEE/RSJ International Conference Intelligent Robots and Systems (IROS), pages 2273-2280. IEEE, 2022.
[27] G. Shani, J. Pineau, and R. Kaplow. survey of point-based pomdp solvers. Autonomous Agents and Multi-Agent Systems, 27:1-51, 2013.
[28] M. T. Spaan and N. Spaan. A baset pomdp algorithm for robot planning. In IEEE International Conference on Robotics and Automation, 2004. Proceedings. ICRA'O (CRA'04. 2004, volume 3, pages 2399-2404. IEEE, 2004.
[29] Z. Fu, X. Cheng, and D. Pathak. Deep whole-body control: Learning unified policy for manipulation and locomotion. In Conference on Robot Learning, pages 138-149. PMLR, 2023.
[30] A. Kumar, Z. Fu, D. Pathak, and J. Malik. Rma: Rapid motor adaptation for legged robots. arXiv preprint arXiv:2107.0400 2021.
[31] Z. Li, X. B. Peng, P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control. arXiv preprint 2XXv:2401.16889, 2024.
[32] J. Long, Z. Wang, Q. Li, J. Gao, L. Cao, and J. Pang. Hybrid internal model: A simple and efficient learner for agile legged locomotion. arXiv preprint arXiv:2312.11460 2023.
10

---

[33] J. Wu, G. Xin, C. Qi, and Y. Xue. Learning robust and agile legged locomotion using adver- sarial motion priors. IEEE Robotics and Automation Letters, 2023.
[34] W. Xu and F. Zhang. Fast-lio: A fast, robust lidar-inertial odometry package by tightly-coupled iterated kalman filter. IEEE Robotics and Automation Letters, 6(2):3317-3324, 2021.
[35] Fankhauser and M. Hutter. A Universal Grid Map Library: Implementation and Use Case for Rough Terrain Navigation. In A. Koubaa, editor, Robot Operating System (ROS) The Com- plete Reference (Volume 1), chapter Springer, 2016. ISBN 978-3-319-26005-5.. doi:10.10077 978- 1-0-----99-22--99461-1-1 26055-9-55-4- URL http://wwww.ppppgaoo.ooooooooo/a

---

Appendix
A
Comparison Experiments
pespejercep
prop-percep
prop History
proptiercap
Longi(O peop thatory
Figure 6: Comparison methods
Robustness
Lidar damage
Ours
RMA
Figure 7:
Robustness testing In simulation, the perception-based RMA mode collapses when the height map is corrupted while our policy works well.
B Ablation Studies
We conducted ablation experiments from multiple angles to examine the effectiveness of our policy in various aspects. The main ablation experiments we performed were:
Without VAE and cooperation regularization.
Without pre-training the blind policy in the first stage.
Our method with KL adaptive learning rate.
The experimental results are shown in Figure We found that both the VAE and our regularization term contribute to improving the final performance. Additionally, without the pre- pre-trained model, training often fails, likely due to the difficulty in converging when training multi-agent systems. Moreover, this multi-agent training approach is very sensitive to the learning rate; an excessively high learning rate or adaptive adjustment of the learning rate can easily cause gradient explosion. As shown in Table 2, the performance ours w/o VAE is worse than that of ours on stairs and discrete obstacle terrains. Although the reward difference between ours w/o VAE and ours is not very pronounced, since these two terrains constitute only a portion of the total terrain, the results still demonstrate that the regularization effectively guides and balances the collaboration between the Blind Policy and the Perceptive Policy.
12

---

Rewards of Different Strategies Over Training Steps
7-
ours
ours w/o VAE pre training
ours KL learning rate
3-
1-
1000
2000
3000
4000
5000
Figure 8: Rewards of Different Strategies Over Training Steps
C Outdoors Experiments
We tested our controller across various outdoor terrains, which included actions such as climbing and dodging in complex terrains using perception, as well as navigating through grass, slopes, soft soil, and steps cases where perception suddenly failed, as illustrated Figure 9 and based on methodologies described by Li et al. [31].
Figure 9: Performance of Robot in Various Terrains.
D Reward Functions
We used the reward function as shown in Table 4, where the Task reward guides the robot to track the desired speed and complete motions on various terrains. Our setting for the regularization reward refers to Long al. [32]; Kumar al. A3a]:Agarwal al. [20]; Cheng et al. [4]. Through extensive training trials, we optimized our reward weight settings to ensure that the robot moves in a relatively ideal manner.
13

---

Table 4: Reward Functions
Reward Type
Equation
Weight
Task Reward
and
1
Linear Velocity Tracking
exp
1.5
2a
Angular Velocity Tracking
exp
)
0.5
Linear Velocity Z
-1.0
Angular Velocity XY
War
-0.1
Regularization Reward
Z Velocity
-1.0
X & Y Velocity
Way
-0.1
Orientation
1
-0.7
Dof Acceleration
1= 9
12
1.5 10
Collision
Fbase
F head
-20.0
Action Rate
lc 0 1 11
-0.11
Delta Torques
im (Tr 1)
1.0 10
Torques
+== Tt
-0.00001
Hip Position
Troti
-0.8
Dof Error
(q Adefault )
-0.04
1 1=1
Feet Stumble
4 x Jiver
-2
Feet
Termination
Dof Position Limits
E12
(ggut,
qi > mmax V q < 8min)
-13.0
E
Training Details
Environment detail: We used the Unitree Go2 robot as our experimental subject, which features 12 degrees of freedom in its legs. Utilizing single NVIDIA RTX 4090 GPU, we simultaneously trained 4096 domaii-randomized Go2 robot environments in Isaac Gym. During training, em- ployed PD position controllers for each joint, with both the Blind Policy and Perceptive Policy running frequency of 50 Hz. The elevation map update rate was set to 10 Hz, and the robot's control signal delay was 20 ms.
During the training process, we utilized the following domain randomization parameters to enhance the robustness of our policy. The range of randomization was referenced from Long al. [32]; Wu al. [33]. In actual robots, factors such as communication delays can lead to action execution delays of approximately 20ms. Therefore, domain randomiza- tion of action delays during robot training significantly improved the real-world performance of the robots.
Robot Domain Randomizations:
Heightmaps Domain Randomizations: We utilize the 'Fast lio' odometer[34] and the method from P. Fankhauser and M. Hutter's[25] construct the elevation map. Due inherent random errors typically associated with laser odometry in practical deployments, have implemented do- main randomization for both the elevation map and the z-axis height of the robot's base.
14

---

Table Robot Domain Randomizations
Parameter
Range [Min, Max]
Base Mass
[0.3] x default kg
CoM
[-0.2,0.2] x default m
Ground Friction
[0.6, 2.0]
Motor Strength
[0.8, 1.2] x default Nm
Joint Kp
[0.8, 1.2] x default
Joint Kd
[0.8, 1.2] x default
Initial Joint Positions
[0.5,1 1. JJ
System Delay
[0,20] ms
Robot Pushing Interval
8s [0. 0.5Jm/s
Push Velocity XY
Table 6: Heightmap Domain Randomizations
Parameter
Range [Min, Max]
Height map updates delay
100ms [-0.05,0.05] m [-0.02, 0.02] m
Robot base Z Noisy
Height Gaussian Noisy
Height Spike Noisy Proportion
5% [0.1, 0.5]
Height Spike Noisy
Terrains Setting: We have designed a training environment containing six different types of terrains: slopes, stairs, discrete obstacles, pits, gaps, and pillars. The first three terrains are relatively easier for robot navigation, while the latter three require more reliance on external perception for anticipation.
Phase One: Blind Policy Training
Table 7: Terrain Parameters and Proportion in Blind Policy Training
Terrain
Proportion
Parameters
Slope
30%
Inclination: [0, 40)
Stairs
60%
Step Height: [2cm, 15cm]
Discrete Obstacles
10%
Obstacle Height: [3cm, 18cm]
Phase Two: Advanced Perceptual Policy Training
Table 8: Terrain Parameters and Proportion in Advanced Perceptual Policy Training
Terrain
Proportion
Parameters
Slope
10%
Inclination: [0, 40]
Stairs
60%
Step Height: [2em, 15cm] Pit: [m, 0,45m];
Complex Terrain
30%
Gap: [0.15m, 0.45m]
Pillar: size [0.4m, 0.6m], center distance [1.6m, 1.4m]
Hyperparameters: Tables 9 and 10 list the hyperparameters used during our two-stage training process. It is important note that multi-agent training, especially with MAPPO, is quite sensitive
hyperparameter settings, for which we referred to the settings recommended in Yu et al. [15]. We observed that the learning rate particularly impacts multi-agent training, where excessively high learning rate can lead to issues such as gradient explosion.
15

---

Phase One: Blind Policy Training
Table PPO Parameters Blind Policy Training
Parameter
Value
Discount factor
0.99
GAE discount factor
0.95
Timesteps per rollout
21
Epochs per Rollout
5
Minibatches per Epoch
4
Entropy Bonus
0.01
Value Loss Coefficient
1.0
Clip range
0.2
Learning rate
KI. Adaptive Learning Rate
Desired KI. Divergence
0.01
Environments
4096
Policy control frequency
50hz
PD controller frequency
200hz
Using history encoder frequency
20
Action Penalty Coefficient
0.1
Height Map VAE Learning rate
1 x 10-4
Phase Two: Advanced Perceptual Policy Training
Table 10: PPO Parameters in Advanced Perceptual Policy Training
Training Parameter
Blind Policy
Perceptive Policy
Discount factor
0,99
0.99
GAE discount factor
0.95
0.95
Timesteps per rollout
21
21
Epochs per Rollout
5
5
Minibatches per Epoch
4
4
Entropy Bonus
0.01
0.01
Value Loss Coefficient
1.0
1.0
Clip range
0.2
0.2
Learning rate
1 10-5
1 1 10
Environments
4096
4096
Using history encoder frequency
20
None
Action Penalty Coefficient
None None
0.01
Reconstruction threshold
0.08
Network Architecture: Tables 11 list the network size used during our two-stage training process.
Table 11: Network Architecture
Parameter
Blind Policy
Perceptive Policy
Actor Hidden Layer
[512, 256, 128] [512, 256, 128] [256, 128]
[512, 256, 128] [512, 256, 128]
Critic Hidden Layer
Priv Encoder Layer
None
VAE Hidden Dims
512
513
VAE Latent Dims
36
36
16

---

Real-World Experiments Setup
In real-world experiments, we utilize Lidar for external perception, employing FAST LIO[3] as odometry system and point cloud map estimator. The final policy input is derived from 2.5D heightmap constructed using Grid Map[35]. Besides, L.M was conducted to transfering heightmap from ROS policy. When Lidar suddenly fails, such covered by hood or hardware damage, the policy will not receive the heightmap updates and set its value to zero.
Table 12: Experiments Setup
Term
Value
Highland
35cm high Pillar Diameter 0.65m, Distance 1.5m height 13em, width 25cm
Upstairs
Downstairs
height 13cm, width 25cm angle 15 degree Maximum height difference 20cm
Slop
Discrete
G
Sim2Real Details
In sim2real deployment, our lidar and rohot parameters, as shown in Table13, are based on configu- rations recommended by Agarwal et [20].
Table 13: Sim2real Parameters
Parameter
Value
Radar relative to base coordinates (xyz rpy)
[-0.33, 0, -0.35, -0.1, -0.55, 0]
Point cloud clipping height
[-0.5m, +0.5m]
Elevation map update frequency
50Hz
Other coefficients for elevation maps
size: 3m 3m, resolution: 0.05m 10Hz
Odometer update frequency
Blind Policy frequency
50Hz (synchronized with Perceptive Policy)
Perceptive Policy frequency
50Hz (synchronized with Blind Policy)
PD controller frequency
1kHz
Joint Kp
40
Joint Kd
40
17


Reviews:
[{"review_id": "WYpEM420MR", "reviewer": "Reviewer_yXBP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a novel approach to robot locomotion learning that leverages Multi-Agent Reinforcement Learning to combine Blind and Perceptive policies. To this end, the authors develop a methodology to train a Multi-Brain collaborative system in two phases, where the first is dedicated to training the Blind policy, and the second trains both the Blind and Perceptive policies in a Multi-Agent collaborative game. The authors report experiments both in simulated and real-world experiments, demonstrating that this methodology allows to correctly switch to the correct type of policy depending on the scenario, surpassing the performance of other baseline methods.", "summary_of_recommendation": "I recommend acceptance, since the paper presents an original, empirically validated and relevant approach to locomotion in robot learning, and contains relevant real-world experiments.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Aside from fixing the very few typos, the only significant improvement that could be made is a slightly more detailed discussion of the limitations (possible solutions, how impactful these limitations are for the end performance).", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aybRXsYpdH", "reviewer": "Reviewer_mQDL", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a novel multi-brain collaborative control system for quadruped robots, combining blind and perceptive policies using multi-agent reinforcement learning (MARL). This approach aims to enhance the robot's locomotion stability and adaptability in complex environments, even under conditions of perception failure. The paper presents both simulation and real-world experiments to validate the effectiveness of this system.", "summary_of_recommendation": "This paper presents a solid contribution to the field, with high relevance and interesting, exhaustive results and experiments. Provided my somewhat minor comments are addressed, I recommend this paper be accepted.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. In the comparison experiments, the RMA is trained with real elevation map input; however, in the experiment, the input of the elevation map is all zeros. Why not try to avoid using the elevation map as input for the RMA policy (blind) or add a large domain randomization (large noise) to the elevation map input for a fair comparison? And why does training with and without Perception Cooperation Constraint Regularization have almost the same performance in the experiments?\n\n2. How do you define the threshold for the reconstruction error?\n\n3. In the obstacle avoidance experiment,was the robot trained to do this or was it a generalized behavior?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mWP5oXqEVu", "reviewer": "Reviewer_xrV5", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper explores getting the best of both perceptive and blind locomotion and trying to obtain a controller that is robust. It uses a Multi-agent approach (MAPPO - multi-agent PPO) to achieve this by treating the two policies (blind/perceptive) as two different agents.", "summary_of_recommendation": "The description of the methods can be significantly improved and clarified and should be done. Greater detail on experimental methods is also required to justify results.", "technical_quality": 3, "clarity_of_presentation": 1, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Questions (Unclear things): \n- The description of the actor networks are confusing. During inference especially, how is it done? The blind policy need the action from the perceptive policy while the perceptive policy needs the action from the blind policy? This isn’t very clear.\n- How is the threshold value for the reconstruction computed? Manually set? How does one set this value and how is it decided? \n- Line 246: what does directly inputting the elevation map into proprioception mean? This is confusing.\n- Experiments: how did you control the policy (what commands were given) in the obstacle settings? How are these standardises across runs and baselines?\n- Computing the success rate 4 times is very confusing. I think what is meant here is not 4 independent runs  of 100 samples are done. Runs/replications would be more suitable here.\n- What do the pillar and gap environments look like? Images of this would be good.\n- Why is there no blind policy baseline? Would be nice to have a blind policy baseline as well (as another sort of ablation)", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Summary: This paper proposes a Multi-Brain collaborative system combining Blind and Perceptive Policies using Multi-Agent Reinforcement Learning for quadruped robots, enhancing stability and robustness in complex environments and perception failure conditions, as demonstrated by simulations and real-world experiments.

Strengths:
* Introduces a novel Multi-Brain collaborative system combining Blind and Perceptive policies using Multi-Agent Reinforcement Learning (MARL).
* Demonstrates the effectiveness of the proposed system through both simulation and real-world experiments.
* Shows significant improvements in stability and adaptability of quadruped robots in complex environments, even under perception failures.

Weakness:
* Some aspects of the methodology, particularly the training and inference processes, need better explanation.
* The real-world deployment lacks quantitative results, with some experiments only described visually without detailed findings.
* Some variables and experimental setups are not clearly defined, and the comparison experiments could be more comprehensive.
* While limitations are discussed, the paper does not explore their impact on results or propose potential solutions in detail.

---- Post rebuttal:

Post rebuttal: The paper received two accept and one reject. The reviewer who recommended reject did not further engage with the rebuttal. The AC carefully went throughs the rebuttal and revised paper and discussion and agreed with the majority of the reviewers to accept the paper.
Decision: Accept