Multi-agent Reinforcement Learning with Hybrid Action Space for Free Gait Motion Planning of Hexapod Robots
Authors: Huiqiao Fu
Kaiqiang Tang
Peng Li
Guizhou Deng
Chunlin Chen
Venue: CoRL 2024 (2024)
Keywords: Free Gait
Hexapod Robot
Hybrid Action Space
Multi-agent Reinforcement Learning
TL;DR: We tackle the problem of free gait motion planning for hexapod robots walking in randomly generated plum blossom pile environments.

Abstract: Legged robots are able to overcome challenging terrains through diverse gaits formed by contact sequences. However, environments characterized by discrete footholds present significant challenges. In this paper, we tackle the problem of free gait motion planning for hexapod robots walking in randomly generated plum blossom pile environments. Specifically, we first address the complexity of multi-leg coordination in discrete environments by treating each leg of the hexapod robot as an individual agent. Then, we propose the Hybrid action space Multi-Agent Soft Actor Critic (Hybrid-MASAC) algorithm capable of handling both discrete and continuous actions. Finally, we present an integrated free gait motion planning method based on Hybrid-MASAC, streamlining gait, Center of Mass (COM), and foothold sequences planning into a single model. Comparative and ablation experiments in both of the simulated and real plum blossom pile environments demonstrate the feasibility and efficiency of our method.

Multi-agent Reinforcement Learning with Hybrid Action Space for Free Gait Motion Planning of Hexapod Robots
Huiqiao Fu', Kaiqiang Tang" Peng Li2, Guizhou Deng', Chunlin Chen
'Department of Control Science and Intelligent Engineering, Nanjing University Institute of Software, Chinese Academy of Sciences 'Corresponding author {hqfu, kqtang, gzdeng) @smail.nju.ede lipeng ascas.ac.on, clchen @ nju.edu.cn
Abstract: Legged robots are able to overcome challenging terrains through di- verse gaits formed by contact sequences. However, environments characterized by discrete footholds present significant challenges. In this paper, tackle the problem of free gait motion planning for hexapod robots walking in randomly generated plum blossom pile environments. Specifically, we first address the com- plexity of multi-leg coordination discrete environments by treating each leg the hexapod robot as an individual agent. Then, we propose the Hybrid action space Multi-Agent Soft Actor Critic (Hybrid-MASACH algorithm capable of han- dling both discrete and continuous actions. Finally, we present integrated free gait motion planning method based MASSC,-MASAC, streamlining gait, Center of Mass (COM), and foothold sequences planning into a single model. Compara- tive and ablation experiments in both of the simulated and real plum blossom pile environments demonstrate the feasibility and efficiency of our method.
Keywords: Free Gait, Hexapod Robot, Hybrid Action Space, Multi-agent Rein- forcement Learning
Introduction
Legged robots can navigate challenging environments using a variety of gaits defined by contact se- quences [1, 2, 3]. Compared to bipedal and quadrupedal robots, hexapod robots possess advantages such as more flexible gaits and higher stability, making them applicable to a variety of fields, such as disaster rescue, material transportation, planet exploration, and more [4. 5, 6]. However, in extreme conditions like discrete environments, these robots encounter major challenge due to the limited foothold choices, and need meticulous planning for selecting reliable gait, Center of Mass (COM), and foothold sequences. In this paper, we address the challenge of automatically generating reliable gait, COM, and foothold sequences for hexapod robots in uneven plum blossom pile environments with a single model.
During locomotion, each leg of the hexapod robot undergoes alternating support and swing phases, generating diverse gait patterns through phase combinations. These gaits categorized into pe- riodic and aperiodic types based on their movement rhythm [7]. Gait planning for hexapod robots often involves formulating sequences based on rules or heuristics [8]. Additionally. biologically inspired methods like the Central Pattern Generator (CPG) can control gait rhythmically without sensory or descending inputs [9, 10]. However, these approaches typically demand significant man- ual design effort. In COM trajectory planning, methods usually fall into two categories: discrete search and continuous optimization. Discrete search methods involve a successor set and a list of possible poses for one state relative to another, forming tree of trajectories explored to find path from start to goal [11, 12]. While, in high-dimensional spaces, these methods encounter significant
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

challenges, termed the "curse of dimensionality". On the contrary, continuous optimization aims to find optimal motion trajectories by solving a set of optimization objectives and constraints, typically assuming convexity [13, 14]. However, is difficult to establish an accurate mathematical model of the problem or find a feasible solution based on these methods. [15].
In recent years, Deep Reinforcement Learning (DRL) algorithms, along with Multi-Agent Rein- forcement Learning (MARL), have shown great potential for intelligent decision-makinn in high- dimensional state-action spaces, and have been widely used in the field of robotics [16, 17, 18, 19, 20, 21]. For instance, Tsounis et al. [22] combined DRL with model-based motion planning, formulating the MDP by evaluating dynamic feasibility criteria instead of relying on physical sim- ulation for motion planning in challenging terrains. Fu al. [23] achieved multi-contact motion planning for a hexapod robot in uneven plum blossom pile environments, demonstrating feasibility in both simulation and reality. However, these methods, relying on fixed gait patterns, may result in poor passibility in complex environments, Recently, some researchers addressed the free gait motion planning problem for hexapod robots in the plum blossom pile environment using hierar- chical framework, utilizing the DRL method to generate the COM sequence and pre-defined rules to select the foothold sequence [24, 25]. Yet, most existing methods plan gait, COM, and foothold sequences independently, presenting two primary challenges in planning the overall motion of hexa- pod robots in complex environments. Firstly, decision variables encompass both discrete actions (e.g., leg phases) and continuous actions (e.g., COM position, foothold), making most RL algo- rithms unsuitable. Secondly, the numerous combinations of foothold points in the environment lead to an exponential growth of the action space with the number of robot legs.
In this paper, we address the challenge of free gait motion planning for hexapod robots in uneven plum blossom pile environments using a unified model. To simplify the complex action space, each leg is treated an independent agent, resulting in action space independent of the number of legs. These agents aim collaborate guide the robot's COM to designated target area. We model the motion of all legs as a Markov game with state transition function determined by the free gait Transition Feasibility Model (free gait TFM). Based on the Soft Actor Critic (SAC) algorithm [26), we propose the Hybrid action space Multi-Agent Soft Actor Critic (Hybrid-MASAC) algorithm, capable of handling both discrete and continuous actions in the free gait motion planning process. Finally, we present an integrated free gait motion planning method based on Hybrid-MASAC for hexapod robots moving in uneven plum blossom pile environments. Experimental results demon- strate that method outperforms tfate-of-the-ar fixed gait and free gait motion planning algo- rithms. Videos are shown at http:/ /ww. hwxapod. cn/mar 1 hexa. html. The contributions of this paper are threefold:
1. We formulate the free gait motion planning of the hexapod robot as a Markov game, and solve the optimization problem to generate optimal gait, COM, and foothold sequences.
2. We propose the Hybrid-MASAC algorithm to solve the multi-contact motion planning problem in the hybrid action space and the multi-agent framework.
3. We test the trained policies in different settings of plum blossom pile environments and compare the proposed Hybrid-MAn MA-AAC-aaeed motion planning method with the baseline methods. Both simulation and real-world experimental results demonstrate the feasibility and efficiency of the proposed method.
2 Preliminaries and Problem Description
2.1 Markoy Game in Hybrid Action Space
The motion of the hexapod robot in the plum blossom pile environment can be formulated as a Markov game of 6-tupee (N,S, A, R. O), where represents the state space containing the local observation of N agents {OI,.... ON} and other environmental information. A {Ai,". A is the action space of N agents. The state transition function T S x A1 x AN + P(S) defines the distribution of the next state given the current state and the

---

tion. R ::Sxx A1 x... AN + R is the reward function. Each agent needs to learn policy
Ty : O - PAA) to map the local observation O to the action distribution P ((A)). The goal is to learn policy that maximize the cumulative discounted returns Ga: -> 7 Ra where E [0, 1] is the discount factor and T is the time horizon.
When both the discrete action a and the continuous action af are included in the action space, the hybrid action space of the agent i defined as A, {a. af). where a fay. ... a consists of D discrete components and af { afc] consists of C' continuous components. We assume that actions of the same type (discrete continuous) are independent of each other, and the hybrid policy of the agent i is decomposed as
Taa af o.) = T (a oz)
o a,,aa)
= II j=1
x (a o.)
x (a II ==1
og, a*).
(I)
In this paper, we represent each component of the discrete policy as a categorical distribution of K possible categories and represent each component of the continuous policy M-dimensional diagonal Gaussian distribution.
2.2 Problem Description
The robot consists of an unactuated floating base and 6 legs, each with 3 actuated rotational joints, and the goal is to move from a random initial point pinitial to a random target area ptarger with the shortest possible trajectory, while fulfilling all the kinematic and dynamic constraints. At each time step I, the state of the hexapod robot is defined as s; := (sb,. p?), where vb == (r Oz. Př, C.) is the proprioceptive information of the hexapod robot, rB R is the COM coordinates of the body, 0Bt IR2 represents the roll and pitch Euler angles of the body, P (Pf.. foot position of the legs time step 6, is the binary historical phase of each leg in the last three timesteps, where 0 indicates the swing phase and indicates the support phase, and P. = P - Pf represents the relative position of the current foot position and the target foot position.
por) represents the
The hybrid action a consists of the discrete action at, and the continuous action af, := (a af) of the leg i, i E (1..... 63. When a 0, the i-th leg is in swing phase, and the corresponding continuous action a determines the next foothold position. When a 1, the i-th leg is in support phase, and the corresponding continuous action asr determines the next position and the next Euler angles of the body. Distinct phase combinations give rise to various gait types. The objective is to find an optimal policy that outputs hybrid actions for the hexapod robot, so as to generate optimal gait, COM, and foothold sequences.
To ensure the traceability of the output trajectory sequences, we expand upon the fixed gait Tran- sition Feasibility Model (fixed gait TFM) proposed by Fu et al. [23] to create free gait version. This free gait TFM, denoted as T(s,, a,, 1) evaluates whether two adjacent states can be transi- tioned. T(s a,, 8(+1) = 1 indicates that the state transition is feasible, and the robot successfully progresses to the next state 8t+1. Conversely, 7(s, a,, $(41) = 0 indicates that the state transition unfeasible, and the robot returns to the last state St (please refer to the Appendix C for further details). The agents collectively aim to guide the robot's COM designated target area within the Markov game framework, while satisfying all kinematic and dynamic constraints.
3 Methodology
3.1 Overall Control Structure
The hexapod robot aims to navigate from an initial point to a target area in the uneven plum blossom pile environment, utilizing a free gait while minimizing the traveled distance. Each leg of the hexa- pod robot undergoes alternating support and swing phases, where swing phase legs select the next foothold position, while support phase legs adjust the body's direction and step size. This iterative process generates sequences for gait, COM, and footholds. The overall control structure is illustrated
3

---

Enviromment-
Free gait TFM
A gent
KNN
Figure I: Overview of the proposed control structure. Each leg of the hexapod robot is regarded as an independent agent with policy symbolized as ,,, i E (1,".,, 1.... 63.
in Fig. 1. Each agent, denoted as T; {1,.... 6), receives local observations o, and generates corresponding actions a, To determine the phase of each agent within a step cycle, π; initially produces the discrete action ay, parameterized as a categorical distribution with two possible cate- gories. If a = O, agent enters the swing phase, and π, generates the corresponding continuous action a governing the selection of the next foothold. Conversely, if a agent enters the support phase, and и, produces the relevant continuous action a& controlling the movement of the COM. These discrete and continuous actions collectively form the hybrid action a,. Based on a, the environment calculates the position of each foot in the swing phase, then employs the K-Nearest Neighbor (KNN) algorithm to identify the nearest plum blossom pile as the target foothold. Subse- quently, the next COM position and Euler angles are determined based on the actions generated by all support phases. The resulting next pending state 5/+1 for the hexapod robot then generated.
Given the current state S, and the next pending state 4/+1 free gait Transition Feasibility Model (free gait TFM), denoted as 7 ['s a, $/41) assesses the transition feasibility between the two states. More details about the free gait TFM can be found in Appendix C. A feasible transition allows the hexapod robot to advance to the next state, 8 8/41, while an unfeasible transition prompts the robot to revert to the previous state, 5;+1 = St. Finally, the environment outputs the next state $,+1 and computes the reward Tt. When reaching the target area or surpassing the maximum number of steps, the current episode is terminated, and the hexapod robot is reset to a new initial point, with
target area generated to start trained by the proposed Hybrid-MASAC algorithm, detailed in Subsection 3.2, and the optimal gait, COM, and foothold sequences of the hexapod robot are finally output by the optimal policy set n = (Mi, ( TT).
episode. The optimal policy nt of each agent is
3.2 Hybrid-MASAC for Free Gait Motion Planning
In multi-agent environment, each agent updates its policy independently during the learning process, resulting in a non-stationary environment that violates the MDP hypothesis conditions. Thus, we use the framework of Centralized Training with Decentralized Execution (CTDE) [27] to solve this problem. Furthermore, to address the decision making problem in the hybrid action space that includes both discrete and continuous actions, we draw inspiration from the work of Delalleau et al.[28], and further perform the hybrid policy iteration within the multi-agent framework, yielding the Hybrid action space Multi-agent Soft Actor Critic (Hybrid-MASAC) algorithm. The network architecture illustrated in Figure and the pseudocode of the integrated Hybrid-MASAC--aan free gait motion planning method is detailed in Algorithm 1.
In the Hybrid-MASAC algorithm, the hybrid policy is defined as
rTa s) (Taa,aaaa ad s) = x a
s. s. aT s),

---

where a represents the discrete actions, a" repre- sents the continuous actions, n is the discrete pol- icy distribution, and лe is the continuous policy dis- tribution. The joint entropy of the hybrid policy is calculated
R6
x" (-1 (1ss) H. (T (-1s, + H (T s)) (3)
Sel-attention
Sel--attertion
Based on the hybrid policy, the joint entropy, and the multi-agent framework, we redefine the soft state- value function of the SAC for the i-th agent
MLP
MLP
Vo, (o.) = -a Th. (- (=)) (Qo, (0)),a))
af log m² (-1 Oit))
Q a(...a
Q (S, a.))))
-a log Tे (a oz, a4t
(4) where Toe is the discrete policy, The is the con- tinuous policy, 0; and Φ; are Q-network param- eters and policy network parameters respectively, o = for ... .N}} is the local observation set, ad {a
Figure 2: Hybrid-MASAC architecture. The policy т. initially outputs the discrete action ag and the parameters of the continuous ac- tion distribution fe of, based on the input observation o, Next, all the observation and continuous action pairs are fed into the self- attention mechanism. and the resulting output features are added to the observation and con- tinuous action pairs as the input of the MI.. Finally, the Q-value Q; (s, af. "......a0 ag) is output by the MLP.
a&x } is the discrete action set, a" [af.... .ay) } the continuous action set, ad is the temperature for the discrete policy, and of is the temperature for the continuous policy.
We incorporate an attention mechanism [29] to allo- cate attention within the Q-network to different agent behaviors. The Q-function, enhanced by this attention mechanism, is computed follows:
Qe, (o. a²) fs (g. (o.,,, af) v:),
(5)
where is a single-layer fully connected network, which outputs statc-action feature vectors ac- cording to the state-action pair of the agent i, fi is two-layer fully connected network that outputs Q-values according to the feature vectors concatenating by v and 9, (o,,,, a(), e is the weighted sum of the state-action feature vectors of all agents except agent which is calculated as
..
V
Σ ##
Oiiiht (ojs ag)
(6)
The attention weight oij is calculated as exp (Br
0i Σ jzz exp (Bi,) Brg = of a.) W WTW Wn9i (oj, ag),
where Big represents the correlation between the state-action feature vectors agent and other agents, Wà and Wg are the weight matrices be learned, a,, is the result of softmax on $r,j.
The policy evaluation of the Hybrid-MASAC is performed by minimizing the redefined Behrman residuals:
Jo(O.) =E =E (or (Qa, (o,, a5) - (ra + 7iE,,, 1i~p [Vo, (o==1))))
(8)
where Vo, (0.+1) is calculated by the target value network parameters 0 and target policy network
parameters Φ,. The target function in the policy improvement of the Hybrid-MASAC redefined
as
(x( =EOrND Eaş=ro m (-1ox)²² " (a log nd (.)ox)
(9)
+ af log T (af Ozr add - Qo. (o,, : a,
5

---

The choice of the temperatures a and a" directly impact the resulting policy. To enable an auto- matic adjustment of these temperatures, we draw inspiration from the temperature update method introduced by Haarnoja et al. [26] and extend it to accommodate the multi-agent framework and hybrid action space. For agent i, the temperature target functions for continuous and discrete policies are defined follows:
J(a)) = Tip (z)))))) (-a (log (log (. o,²) + H)),,
(10)
J(a = Eng I-a (log 3 (af o a + H.),
(11)
where Hd and H represent two hyperparameters for discrete and continuous policy, respectively. In our experiments, set 17 = 0.25 and He 0..25.
Experiments
4.1 Training Setup
Three different types of simulated plum blossom pile environments, denoted as E1, Ez, Es and real plum blossom pile environment, E4, are constructed to evaluate the proposed Hybrid-MASAC: based free gait motion planning method, as illustrated in Fig. 3. Each environment has dimensions of 1.2 in length and width. The number of plum blossom piles in each environment is denoted as Np. For each simulated environment, we set Np to be 300, 200, and 150, respectively, resulting in total of 9 different simulated environments.
We use 3 metrics to evaluate the performance of the proposed motion planning method: Average Episode Reward (AER), Average Episode Step (AES) and Average Success Rate (ASR). AER is calculated the ratio of the total reward to the total number of steps in episode. AES is deter- mined the ratio of the total steps to the distance between the initial point and the target area. ASR represents the average success rate observed in the last 200 episodes.
(b)
Figure 3: Different types of plum blossom pile environment with N2 = 150. (a) Randomly dis- tributed plum blossom piles with the same height Ej. (b) Randomly distributed plum blossom piles with random height Eg. (c) Stair shape plum blossom pile environment Ez. (d) simplified version of Es in the real-world E,. The transparent blue polyhedrons in the simulated environments are the simplified foot workspace.
4.2 Experimental Results
We use the proposed Hybrid-MASAC algorithm to train the optimal hybrid policies different simulated environments. Fig. 4 shows the top view of the plum blossom pile environment used in the training process. Where, the green block is the initial point, the red circle is the target area, yellow dots represent plum blossom piles, red dots represent footholds and CoM sequences, and blue wireframes represent the body of the hexapod robot. As training progresses, all agents gradually learn to cooperate in guiding the robot to the target area, resulting in higher average rewards, fewer steps taken on average, and an increasing average success rate, as shown in Fig. 5. At the end of the training, the optimal policy can leads the hexapod robot the target area with the shortest trajectory as shown in Fig. 4(b).

---

To evaluate the trained policies, we randomly set 100 groups of initial points and target areas in each environment, and compute the corresponding AER, AES and ASR metrics. The results are shown in Table 1. The optimal hybrid policies trained with Hybrid-MASAC can generate effective sequences of gait, COM, and foothold, and guide the hexapod robot to the target area with ASR exceeding 80%. When the number of plum blossom piles exceeds 200, the ASR approaches 100%. Conversely, as the number of plum blossom piles decreases, the ASR decreases due to the lack of suitable footholds for certain legs.
(b)
anitial point
target
plum blossom piles
footholds
COM
body
Figure 4: Top view of the plum blossom pile environment used in the training process. (a) The beginning of the training. (b) The end of the training.
To comprehensively evaluate the performance of the proposed Hybrid-MASAC. algorithm, we conduct comparative analysis against several baseline meth- ods. These include traditional graph-based methods: (a) A and (b) A *-FFree as well as DRL-based methods: (c) Multi-agent Soft Actor Critic (MASAC) [30], a simplified counterpart to Hybrid-MASAC in the continuous action space, (d) Single-agent Soft Actor Critic with free gait (SAC-Free) [26], (e) Single-agent Soft Actor Critic with tripod gait (SAC-Tripod), and (f) HFG- DRI. [24], a hierarchical DRL-based algorithm for free gait motion planning. More details about the above methods can be found in Appendix D.
1I
12
0.2
5206
10000
15000
20000
10e00
11000
20000
5206
10000
15620
2000
Ipsendex
Ipsendes
Ipsendex
(a)
(b)
(c)
Figure 5: Learning curves of E3 with N, 150 random seeds). (a) The learning curve of AER.
(b) The learning curve of AES. (c) The learning curve of ASR.
Table 1: Metries of the trained policies different types of environments using Hybrid-MASAC.
N₂
Metrics
E2
Es
AER
1.196 2.575
4.13120.469
300
AES
0.07 10.022
0.057:0.012
0.042=0.017
ASR
98%
100%++0%
100%++0
AER
1. ..798
20.61
2.628=0.290
200
AES
10.025
0.063
0.057=0.016
ASR
97%
99%++1%
99%++%%
AER
0.504
1.805 412
1.867=0.122
150
AES
0.133
0.09110.020
0.085_0.014
ASR
83%49%
92%%++%
96%+2%
For a fair comparison with the motion planning method proposed by Fu et al. [23], we substitute the PPO algorithm with the SAC algorithm the tripod gait case. The evaluation takes place within identical environmental conditions, maintaining consistent hyperparameters, such as learning rate, temperature, discount factor, et Table presents comparison of the metrics between baseline methods. For each experiments, we perform training with different random seeds and use 100 groups of initial points and target areas for evaluation. The * algorithm benefits from free gait and achieves higher success rate compared to *rTriio:;; however, due the COM being
7

---

Table 2: Comparison of Hybrid MASAC and bascline methods in E3 with Np. = 150.
Algorithms
AER
AES
ASR
A -
0.257
21 11%
A*-Frec
0.189
42%
MASAC [30]
-0.69620.225
0.21520.013
46%625%
SAC-Free [26]
00,911
0.23 1.0.09
32%
SAC-Tripod [23]
1.652
0.122
90%.1%%
HFG-DRL [24]
0.
0.202++0.015
58%++%
Hybrid-MASSC (Ours)
1.867=0.422
0.085+0.014
96 % +2%%
represented as discrete grids, the success rates of such algorithms are generally low. The HFG-DRL. algorithm also has a relatively low success rate when the number of piles is only 150, as it selects gaits based on rules. The proposed Hybrid-MASAC algorithm outperforms all the baseline methods across all metrics, showcasing the feasibility and efficiency of our method in the planning of free gait motion for the hexapod robot walking in complex plum blossom pile environments.
Finally, we conducted the experiment in the real-world environment E.. The video can be found in http://wwa. http:://wwwhhxxpo.doom www.poddddd..oooo cn/marlhexe html. We assume access to environmental information and global COM coordinates. The trained hybrid policy enables the generation of optimal gait, COM, and foothold sequences for the hexapod robot. The target joint rotation angles for each time step are calculated by inverse kinematics. Subsequently, PD) controller guides each joint of the hexapod robot to follow the target trajectory. An inherent challenge stems from error accumulation during prolonged movements in the real environment. Nevertheless, in our experiment, the size of the environment is relatively small, where cumulative errors remain within acceptable thresholds. Worthy of note, the random plum blossom pile environments can characterize any non-structured environment to a certain extent. Our method holds promise for deployment in more challenging real-world environments, offering safer and more reliable motion reference trajectories.
5 Limitations
In the real world experiment, assume the environmental information is known; however, for dynamic real-world scenarios, real-time environmental perception and localization are necessary. To tackle this issue, advanced techniques such as Simultaneous Localization and Mapping (SLAM) can be introduced for precise mapping and positioning, although this falls beyond the scope of our paper. It's important to highlight that the motion planning method outlined in this paper holds applicability to quadruped and biped robots as well. We can adapt the approach by adjusting the state space, action space, and transition feasibility model accordingly. However, advanced and robust control methodologies such as Model Predictive Control (MPC), Linear Quadratic Regulator (LQR), and various trajectory tracking techniques is needed to accurately track the output trajectory sequences.
6 Conclusion
A free gait motion planning method is proposed in this paper, where each leg of the hexapod robot is regarded as an independent agent, with the common objective of moving the COM from a initial point to designated target area in the uneven plum blossom pile environment. The motion all legs is modeled as a Markov game with a specific state transition function determined by the proposed free gait TFM. Based on the multi-agent framework and the hybrid action space, we propose the Hybrid-MASAC algorithm to train and generate the optimal hybrid policy for each agent. These trained policies enable the generation of optimal gait, COM, and foothold sequences for the hexapod robot, while fulfilling all the kinematic and dynamic constraints. All the experimental results in both simulation and real-world demonstrate the feasibility and efficiency of the proposed method. Our future work will focus on training more robust policies and combining the proposed method with localization and map-building methods.
8

---

Acknowledgments
This work was supported in part by the National Natural Science Foundation of China (Nos. 62073160 & 72394363), the Nanjing Science and Technology Plan under Grant 202309018, and the Nanjing University Integrated Research Platform of the Ministry of Education--Top Talents Pro- gram.
References
[1] M. Tranzatto, T. Miki, M. Dharmadhikari, L. Bernreiter, M. Kulkarni, F. Mascarich, O. Ander- sson, S. Khattak, M. Hutter, R. Siegwart, et al. Cerberus in the darpa subterranean challenge. Science Robotics, 7(66):eabp9742, 2022.
[2] A. Agarwal, A. Kumar, J. Malik, and D. Pathak. Legged locomotion in challenging terrains using egocentric vision. Conference on Robot Learning, pages 403-415, 2023.
[3] P. Arm, G. Waibel, J. Preisig, T. Tuna, R. Zhou, V. Bickel, G. Ligeza, T. Miki, F. Kehl, H. Kol- venbach, and M. Hutter. Scientific exploration of challenging planetary analog environments with a team of legged robots. Science Robotics, 8(80))cade9548, 2023.
[4] J. Coelho, F. Ribeiro, B. Dias, G. Lopes, and P. Flores. Trends in the control of hexapod robots: survey, Robotics, 10(3): 100, 2021.
[5] W. Ouyang, H. Chi, Pang, W. Liang, and Q. Ren. Adaptive locomotion control of a hexapod robot via bio-inspired learning. Frontiers Neurorobotics, 15:627157, 2021.
[6] S. Wang. Z. Chen, J. Li, J. Wang, J. Li, and J. Zhao. Flexible motion framework of the six wheel-legged robot: Experimental results. JEEE/ASME Transactions on Mechatronics, 27(4): 2246-2257, 2022.
[7] F. Tedeschi and G. Carbone. Design issues for hexapod walking robots. Robotics, 3(2):181- 206, 2014.
[8] D. Wettergreen and C. Thorpe. Gait generation for legged robots. IEEE International Conference on Intelligent Robots and Systems, pages 1413-1420, 1992.
[9] B. Zhong, S. Zhang, M. Xu, Y. Zhou, T. Fang, and W. Li. On opg-based hexapod robot: Amphihex-ii with variable stiffness legs. IEEE/ASME Transactions Mechatronics, 23(2): 542-551, 2018.
[10] C. Akkawutvanich and Manoonpong. Personalized symmetrical and asymmetrical gait gen- eration of lower-limb exoskeleton. IEEE Transactions Industrial Informatics, 2023.
[11] D. Belter, P. Labecki, and P. Skrzypezynski. Adaptive motion planning for autonomous rough terrain traversal with a walking robot. Journal of Field Robotics, 33(3):337-33 -70, 2016.
[12] P. Xu, L. Ding, Z. Wang, H. Gao, R. Zhou, Z. Gong, and G. Liu. Contact sequence planning for hexapod robots in sparse foothold environment based monte-carlo tree. IEEE Robotics and Automation Letters, 7(2):826-833 2021.
[13] A. Winkler, C. D. Bellicoso, M. Hutter, and Buchli. Gait and trajectory optimization for legged systems through phase-based end-effector parameterization. IEEE Roboties and Automation Letters, (3)--550-1567. 2018.
[14] S. Kuindersma, R. Deits, M. Fallon, A. Valenzuela, H. Dai, F. Permenter, T. Koolen, P. Marion, and R. Tedrake. Optimization bassea locomotion planning, estimation, and control design for the atlas humanoid robot. Autonomous Robots, 40(3)429-453 2016.
[15] S. P. Boyd and L. Vandenberghe. Convex optimization. Cambridge University Press, 2004.

---

[16] S. Li. Deep reinforcement learning. In Reinforcement Learning for Sequential Decision and Optimal Control, pages 365-402. 2023.
[17) W. Zhu, X. Guo, D. Owaki, K. Kutsuzawa, and M. Hayashibe. A survey of sim-to-real transfer techniques applied to reinforcement learning for bioinspired robots. IEEE Transactions Neural Networks and Learning Systems, 2021.
[18] S. De Witte, Van Hauwermeiren, T. Lefebvre, and G. Crevecoeur. Learning to cooper- ate: A hierarchical cooperative dual robot arm approach for underactuated pick-and-placing. IEEE/ASME Transactions Mechatronics, 27(4):194-1172. 2022.
[19] K. Zhang, Z. Yang, and T. Başar. Multi-agent reinforcement learning: selective overview of thcories and algorithms. Handbook of reinforcement learning and control, pages 321-384, 2021.
[20] A. Perrusquía, W. Yu, and X. Li. Multi-agent reinforcement learning for redundant robot control in task-space. International Journal of Machine Learning and Cybernetics, 12(1): 231-241, 2021.
[21] J. Orr and A. Dutta. Multi-agent deep reinforcement learning for multi-robot applications: survey. Sensors, 23(7):3625, 2023.
[22] V. Tsounis, M. Alge, J. Lee, F. Farshidian, and M. Hutter. Deepgait: Planning and control of quadrupedal gaits using deep reinforcement learning, IEEE Robotics and Automation Letters, (99:3699-37000 2020.
[23] H. Fu, K. Tang, P. Li, W. Zhang, X. Wang, G. Deng, T. Wang, and Chen. Deep reinforcement learning for
motion planning of hexapod robots. In Proceedings of the Thirtitth International Joint Conference Artificial Intelligence (IJCAI), pages 2381-2388, 2021.
[24] X. Wang, H. Fu, G. Deng, C. Liu, K. Tang, and C. Chen. Hierarchical free gait motion plan- ning for hexapod robots using deep reinforcement learning. IEEE Transactions on Industrial Informatics, 19(1 (ep(11):1990) 10100011-99911 2023.
[25] K. Tang, H. Fu, G. Deng, X. Wang, and C. Chen. Hierarchical multicontact motion planning of hexapod robots with incremental reinforcement learning. IEEE Trunsuctions on Cognitive and Developmental Systems, 16(4):1327-1341 2024.
[26] T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, Abbeel, and S. Levine. Soft actor-critic algorithms and applications. arXiv preprint 11X1v:1812.05505, 2018.
[27] R. Lowe, Y. I. Wu, A. Tamar, J. Harb, O. Pieter Abbeel, and I. Mordatch. Multi-agent actor- critic for mixed cooperative-compettive environments. Advances in Neural Information Pro- cessing Systems, 30, 2017.
[28] O. Delalleau, M. Peter, E. Alonso, and A. Logut. Discrete and continuous action representation for practical rl in video games. In AAAI Workshop on Reinforcement Learning in Games, 2019.
[29] Iqbal and F. Sha. Actor-attention-critic for multi-agent reinforcement learning. Interna- tional Conference on Machine Learning (ICML), volume 97, pages 2961-2970, 2019.
[30] D. Cao, J. Zhao, W. Hu, F. Ding, Q. Huang, 7. Chen, and F. Blaabjerg. Data-driven multi- agent deep reinforcement learning for distribution system decentralized voltage control with high penetration of pvs. IEEE Transactions on Smart Grid, 12(5):4137-413 2021.
[31] A. W. Winkler. Optimization- based motion planning for legged robots. PhD thesis, ETH Zurich, 2018.
10

---

A Pseudocode of Hybrid-MASAC for Free Gait Motion Planning
The integrated Hybrid-MASAA based free gait multi-contact motion planning algorithm is shown as in Algorithm
Algorithm 1: Hybrid-MASAC. for Free Gait Motion Planning
1: Initialize network parameters Φ, 0, replay buffer 'D. 2: Initialize target network parameters Φ < Φ, <- 0.
3: for episode k 1, 2, ...., M do
4: Randomly initialize pinitial and Plarge
5: Reset the hexapod robot and obtain the state So and observation Oo.
6: for step = 1, 2, ..., do
7: for agent 1, do
8:
Sample discrete actions ay. Th (a oa). Sample continuous actions a ~ x (a a end for
9:
11:
Perform the hybrid action and obtain the pending state 1/+1- if transition feasibility T'ss, a, $(41) 1 then
12:
13:
St+1 = 5 obtain 0,+1 and Tt-
14:
else
15:
St-1 = St, obtain o++1 and rt-
16:
end if
17:
Store experience (o,, a, rt. 0+41) in Dj-
18:
end for
19:
for agent = 1, .... do
20:
0, + 0: Λo Vo (0i),
21:
o - Φ; da V., (n
22:
a + a A a (a af + af - -V Jaa))
23:
0, + + (1 - 7)0, 0, Td; + (1 7)
24:
end for
25: end for
Markov Game Formulation
B.1 Observation Space
The observation of the hexapod robot is composed of 6 local observations {or, ,0], which is subset of the state space. The local observation o, E is defined
O; = (v/.p.) p!),
(12)
where vb : (TB, Og, DS, c.) represents the proprioceptive information of the hexapod robot. p P p is the relative position between the current position pÊ of the i-th foot and its corresponding target position p in the world frame. p is generated according to the coordinate of the target area.
B.2 Action Space
Under the multi-agent framework, each leg of the hexapod robot is defined as independent agent. At each time step the agent needs to choose the current discrete phase and the corresponding continuous action that should be performed under that phase. The action a; of agent i, generated by its corresponding policy ,G, is defined as
a, (a. a a(
(13)

---

where the discrete action a {0, 1 represents the next phase of the i-th leg. When a 0, leg i is in swing phase, otherwise leg i is in support phase. a == Arp; represents the continuous action de- noting the coordinate change of the swing phase, and a == (Argi, A B) is the continuous action of the support phase. The coordinate change of the COM is calculated as Årg = N. No Ar N. A Bi, where N. is the
and the Euler angle change of the body is calculated as A0g = N. number of the support phase, and the yaw angle is assumed to remain unchanged in the motion process.
B.3 Reward Function
For all agents, the objective is to work together move the COM from an initial point target area via the shortest trajectory, while fulfilling all the kinematic and dynamic constraints. To achieve this task, the reward r is defined as
Ti.t Tw,t Fket + Tstt Ts,t + Thi, Tdiit + Tggts
(14)
where T,, penalizes the improper combination of support phases, including two cases: the number of total support phases Ne. 2, and there is no support phase in the adjacent three legs, The penalizes the hexapod robot for any leg that does not meet the kinematics constraint in the next state. fft penalizes the unfeasible state transition of hexapod robot according to the free gait TFM. *s.1 penalizes the hexapod robot for every step it takes. This penalizes the leg i for more than three consecutive time steps in the same phase to prevent the hexapod robot from standing still. rai, rewards the agent for moving towards the target area. rg.r rewards the hexapod robot for reaching the target area.
C Free gait Transition Feasibility Model
Referring to the fixed gait Transition Feasibility Model (TFM) of hexapod robots [23], we propose free gait TFM Ts, a,, s(4)) determine whether the transition between two adjacent states can be completed under all the kinematic and dynamic constraints, which includes three parts:
(pf)): Check for illegal phase combinations. For example, the number of total support phases is less than 3, or there is no support phase in the three adjacent feet.
Teetkes 4b(4)): Check for kinematic constraints.
Taaabe, :b4)): Check for dynamic constraints.
where s'42 represents the next pending state. We formulate the transition feasibility nonlinear constrained optimization problem using the direct multiple shooting method and solve it by open- source tool for nonlinear optimization, CasADi'. In CasADi, it uses interior point method to find the solution, and when the problem has no solution the number of optimization steps reaches the preset maximum, we consider the problem to be infeasible.
C.1 Phase Combination Constraints
The hexapod robot can use three or more support phases to form a support polygon to maintain good static stability. Thus, for the current state St, we set phase combination constraints to filter out illegal phase combinations, including the number of total support phases N₂ 2 and the support phase number for the three adjacent feet should not be zero.
C.2 Kinematic Constraints
For the initial state 8p and each pending state 8/41 the kinematic constraints are set up avoid the robot's behavior violating the physical constraints of its own structure. The dimension parameters
'https://veb. casadi. oag/
12

---

Table 3: Dimensional parameters and joint rotation ranges of the hexapod robot.
Parameter
Body
Coxa
Femur
Tibia
Lengt//mm 240
120
145
Range/
[-45, 45]
[-45, 45]
[-135, -45]
10C
R
Pa
100
216
200
31
100)
(a)
(b)
Figure 6: The foot workspace of the hexapod robot. (a) The point cloud of original foot workspace. (b) Simplified foot workspace which is represented polyhedron.
and joint rotation ranges of the hexapod robot shown in Table 3. By sampling within the joint rotation range, we can obtain the point cloud of the original foot workspace, as shown in Fig. 6(a). Since the original foot workspace is highly nonlinear and inconvenient for optimization, we con- servatively simplify the kinematics constraint for leg i as keeping its foot p within the polyhedron workspace R2 as shown in Fig. 6(b). For each plane in the polyhedron, we can find its normal vector given the coordinates of three points. Here, we take Pl, P2 and P3 Fig. 6(b) as an example, the normal vector of this plane n (D2 P1) (P3 P1). To keep the foot p inside the polyhedron, D needs to be the right side of the plane, yielding the following constraint:
Rz (ax)) [ Rp pi).n] >0 0,
(15)
where 1 R. is the rotation matrix from the world frame to the body frame. Note that the polyhedron described in the body frame and P is described in the world frame. Rg(as) is the z-axis rotation matrix and a; is the z-axis deflection angle of the coxa frame to the body frame.
Each simplified foot workspace polyhedron consists of 12 planes, For each foot, there are total of 12 kinematic constraints.
C.3 Dynamic Constraints
In order to determine whether the hexapod robot can transfer from the current state S, to the next pending state 41 we set up the corresponding dynamic constraints and solve the problem as mathematical feasibility problem using existing optimization tools. Specifically, we need to find set of decision variables such the COM position rg R3, the body Euler angles O R33 the foot contact force f E R6x3 et al., which fulfill all the constraints given and 8'41- The dynamic constraints include three parts: the dynamic model of the hexapod robot, the pushing force constraint and the friction cone constraint.
The dynamic model represents the dime-dependent aspects of the system, and we approximate by Single Rigid Body Dynamics (SRBD) [31]. Then, we get the Newton-Fules equations of the hexapod robot, which is defined as the SRBD:
N
mřg Σ f, + mg.
(16a)
iml
L w Iw
Σ
(rB -P;),
(16b)
where m is the mass of the hexapod robot, f, E R is the contact force of foot E Rd is the acceleration of gravity, E R² is the angular velocity of the body, I E 323x3 is the inertia tensor
13

---

of the hexapod robot in the world frame, which can be calculated by the inertia tensor in the body frame B and the rotation matrix W B R: = W R R is the number of the support foot. With free gait, N = 6 and we constrain the contact force the swing phase to zero.
B
Using the SRBD, the state of the hexapod robot be expressed by an ordinary differential equation x Fxx, f), where x [rg. f' B w]" is the state of the body which is only affected by the contact force. The rates of the Euler angles 0 B can be calculated by the optimized Euler angles 0p and the angular velocities w:
sin By sin Oz cos fy w.
DB C(OB)/
0
cos 0r sin 0z
(17)
0
cos 0₂ cos 8g
Using numerical integration methods, such as trapezoidal quadrature, we obtain the discrete dynamic constraint formula of the ordinary differential equation:
Xk++ Xk (1x+1 (F fR+1) + F(xx, fx))-
(18)
The pushing force constraints restrict the force provided by the environment to the hexapod robot can only be thrust:
n (pf) 0,
(19)
where n (p.) the normal vector of the environmental surface coordinate p..
The friction cone constraint follows from Coulomb's law that pushing stronger into surface allows exerting larger side-way ways forces without slipping. Therefore, the resultant force on each support foot always in the interior of the friction cone. The linear approximation is as follows:
[ (1.2) (pf) S f; (Pf),
(20)
where fy1,2} (pf) is the tangential vector of the environment at coordinate pf and μ is the friction coefficient.
D Additional Experimental Details
At the beginning of a training episode, an initial point and a target area with a radius of 150 mm are randomly generated, and the distance between them is greater than 300 mm. All agents collaborate to guide the hexapod robot's COM from the initial point to the target area, and the trained policies yield the optimal gait, COM, and foothold sequences.
The policy network trained on a computer with an i7-7700 CPU and Nvidia GTX 1060ti GPU, The networks are implemented using Pytorch², and the transition feasibility model used in the train- ing process is solved using CasADi. All the simulated environments are built by PyBullet'. The hyperparameters of the Hybrid-MASAC algorithm detailed in Table 4.
We conduct a comparative analysis against existing baseline methods: (a) Ar-Trip an A al- gorithm with the fixed tripod gait, using a 20x20 2-dim grid map to represent the robot's center of mass position in the plum blossom pile environment, and employing a fixed gait transition feasibility model determine whether two nodes are transferable.
(b) A+eFee an A algorithm with the free gait, utilizing free gait transition feasibility model assess the transferability of two nodes, with other settings identical to A Aripoo. (c) Multi-agent Soft Actor Critic (MASAC) [30], simpli- fied counterpart to Hlyriid-MASCE in the continuous action space, involves each agent generating a 10-dimensional continuous action vector, with each agent generating a 10-dimensional contin- uous action vector including 2 phase actions, 3 swing foot actions, 3 support foot actions, and body Euler angle actions. (d) Single-agent Soft Actor Critic with free gait (SAC-Free) [26] utilizes a di-dinensional continuous action vector for free gait motion planning, encompassing groups
"natpe://pytorcch.og htttps///pybullet.orgg
14

---

Table 4: Hybrid-MASAC hyperparanttss during training.
Parameter
Value
Optimizer
Adam
Learning rate
10-2
Discount factor
0.95
Replay buffer size
106
Minibatch size
1024
Target smoothing coefficient
10 2
Maximum number of episode steps
100
Discrete policy target temperature
0.25
Continuous policy target temperature
-0.25
of 2-dimensional phase actions, 6 groups of 3-dimensional swing foot actions, alongside 3 COM actions and 2 body Euler angle actions. (e) Single-agent Soft Actor Critic with tripod gait (SAC- Tripod) uses 14-dimensional continuous action vector for tripod gait, involving COM actions, body Euler angle actions, and 3 groups of 3-dimensional swing foot actions. (f) HFG-DRL [24]: A hierarchical DRL-based algorithm with the free gait, where the upper layer implements center of mass path planning and the lower layer implements motion planning, selecting gait types based on some rules.
E Additional Experimental Results
With the free gait generated by the policies, the hexapod robot can adapt to various gait patterns such as tripod gait, quadruped gait, and wave gait according to the surrounding environment. When the policy converges the optimum, the proportion of the above three gaits in each environment illustrated in Fig. Notably, in the same environment, the number of plum blossom piles decreases, to get better passibility, the proportion of tripod gait decreases, while the proportion of wave gait increases. The proportion of the quadruped gait is relatively stable across all environments. Additionally, the distribution of gait types varies among different environments.
0.6
0.5
Proportion
0.4
0.3
0.2
0.1
0
E1-300
El-200
El-150
E2-300
E2-200
E2-150
E3-300
E3-200
E3-150
Tripod Gait
Quadrppd Gait
Wave Gait
Figure 7: Statistics of gait types in 9 environments with free gait.
Fig. shows the coxa angle curves of the hexapod robot obtained by the trained policy in environ- ment Eg with Np 150. Since each state transition the hexapod robot is checked by the proposed free gait transition feasibility model, the coxa angle is always within the angle range shown in Table 3.
Fig. shows the heat map of the passable area of the optimal free gait policy trained by the Hybrid- MASAC algorithm and the optimal tripod gait policy trained by the SAC-Tripod algorithm in the environment E3 with N, 150. The heat map covers the passable area of the COM in the en- vironment, and the darker the color, the greater the probability of passability. Fig. 9(a) shows the heat map of the passable area using free gait, and Fig. 9(b) shows the heat map of the passable area using tripod gait. It can be observed that the distribution of the passable area with free gait is rela-
15

---

20
Analee/
Dasa
Dana
Coan
40
Coss
Tixe/s
Figure The coxa angles of the robot E3 with N, = 150.
Table The Average Success Rate (ASR) of the trained policy in Eg when transferred to Ei, E2. and another random Eg.
Direct transfer
Fine tune for 100 episodes
Fine tune for 500 episodes
F1
6%x11%
26%css%
58%o14%%
Fg
1%
33%26%
65%
Ea
62%:13%
84%+2%
90%;22%
tively uniform, and there are more impassable with fixed tripod gait, which further proves that the Hybrid-MASAC algorithm is more adaptable to the complex environment than the SAC-Tripod algorithm.
0.8
3.4
3.2
(a)
(b)
Figure Passable areas COM in (a) free gait and (b) tripod gait.
Table 5 shows the performance comparison of the algorithm trained in Es when transferred Et, Eg, and another random E,, with performance evaluated based whether fine-tuning conducted. The trained policy can still achieve success rate of around 62% in completely new F. environ- ment. After only 100 episodes of fine-tuning, the success rate can reach 84%, further proves the robustness of our method in the same type of environment. However, when transferred to E₁ and Eg, the accuracy of the policy significantly decreases, which may be alleviated by more advanced transfer learning algorithms. We will leave this as future work.
16


Reviews:
[{"review_id": "IPnRmYJi3S", "reviewer": "Reviewer_oUGJ", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper considers free gait motion planning for hexapod robots in uneven plum blossom pile environments, by treating each leg of the hexapod robot as an individual agent and proposing the Hybrid action Multi-Agent Soft Actor Critic (Hybrid-MASAC) algorithm to solve it.", "summary_of_recommendation": "This paper uses multi-agent reinforcement learning to solve free gait motion planning for hexapod robots. However, the motivation of using reinforcement learning is not clear and there is a lack of comparison with other search or optimization based methods. Some parts of the paper need more clarifications.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "Please see the above concerns.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "awSESpZAQu", "reviewer": "Reviewer_z2Hc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper presents a Multi-agent combined with hybrid action space reinforcement learning approach to address the challenge of navigating through a complex environment using a hexapod robot. The algorithm is able to automatically generate COM motion, gait and foothold sequences.", "summary_of_recommendation": "Weak accept, additional information on real world experiments could strengthen the paper", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "Questions \n1) The approach is compared to three baselines, but none of them include a case where its a single policy with hybrid action space. Is this too challenging to train ? Any rationale for that I am missing? \n2) For different environments in Table 1. The average success rate for E3 is higher than E1, although intuitively E3 is a more challenging environment. The reason stated for that is better policy exploration leading to more robust behavior, did the authors try deploying a policy trained in E3 in E1? \n3) Im also curious to see how this approach would perform when various uncertainties, for example from sensor noise, transition probabilities, etc.. are considered.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "QpJgAHciWm", "reviewer": "Reviewer_XYuJ", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper introduces a novel free gait motion planning method for hexapod robots, treating each leg as an independent agent with the collective goal of moving the robot's center of mass (COM) through an uneven plum blossom pile environment. The method models leg movements as a Markov game, defined by a free gait Time-Free Markov (TFM) model, and utilizes the Hybrid Multi-Agent Soft Actor-Critic (Hybrid-MASAC) algorithm to train and generate optimal hybrid policies for each leg. These policies enable the creation of optimal gait patterns, COM trajectories, and foothold sequences while meeting all kinematic and dynamic constraints. The feasibility and efficiency of the approach are validated through experiments in both simulated and real-world settings. Future research aims to enhance policy robustness and integrate the method with localization and map-building techniques.", "summary_of_recommendation": "The paper proposes a novel free gait motion planning method for hexapod robots, treating each leg as an independent agent with the collective goal of moving the robot's center of mass (COM) through an uneven plum blossom pile environment. Despite the innovative approach, the paper lacks clarity on several significant points that are crucial to understanding the contributions of the proposed method. To enhance the paper's contribution and impact, the author should address and revise these issues comprehensively.", "technical_quality": 3, "clarity_of_presentation": 2, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "The author should consider revising the following issues\n•\tClarify how to define the Transition Feasibility Model concretely.\n•\tExplain and derive each term of the reward function.\n•\tClarify whether the action of the foothold during the swing phase is continuous or discrete.\n•\tVerify the polyhedron is bounded inside the point cloud.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper received 3 fairly confident reviews.  This paper considers free gait motion planning for hexapod robots in uneven plum blossom pile environments  by treating each leg of the hexapod robot as an individual agent and proposing the Hybrid action Multi-Agent Soft Actor Critic (Hybrid-MASAC) algorithm to solve it.  To address in rebuttal (in addition to reviewer comments): - Address the justification of the use or RL - Expand experimental trials; consider ablations proposed by reviewers - Address introduction of uncertainty. - Address clarity issues pointed out by reviewers.  Post-rebuttal: This paper uses multi-agent reinforcement learning to solve free gait motion planning for hexapod robots.  The authors have addressed many issues raised and have added comparative experiments with the traditional graph-based A algorithm* and the latest free-gait based HFG-DRL algorithm; they also included two sets of ablation experiments. The motivation of using reinforcement learning could be better demonstrated and we encourage to do so in the camera ready. A reviewer updated to weak accept, but didn't change the score -- so finally there are two weak accepts and a weak reject. 
Decision: Accept