HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulation
Authors: Jin Wang
Rui Dai
Weijie Wang
Luca Rossini
Francesco Ruscelli
Nikos Tsagarakis
Venue: CoRL 2024 (2024)
Keywords: Loco-manipulation
Large Language Models
Humanoid Robot Learning

Abstract: Enabling robots to autonomously perform hybrid motions in diverse environments can be beneficial for long-horizon tasks such as material handling, household chores, and work assistance. This requires extensive exploitation of intrinsic motion capabilities, extraction of affordances from rich environmental information, and planning of physical interaction behaviors. Despite recent progress has demonstrated impressive humanoid whole-body control abilities, they struggle to achieve versatility and adaptability for new tasks. In this work, we propose HYPERmotion, a framework that learns, selects and plans behaviors based on tasks in different scenarios. We combine reinforcement learning with whole-body optimization to generate motion for 38 actuated joints and create a motion library to store the learned skills. We apply the planning and reasoning features of the large language models (LLMs) to complex loco-manipulation tasks, constructing a hierarchical task graph that comprises a series of primitive behaviors to bridge lower-level execution with higher-level planning. By leveraging the interaction of distilled spatial geometry and 2D observation with a visual language model (VLM) to ground knowledge into a robotic morphology selector to choose appropriate actions in single- or dual-arm, legged or wheeled locomotion.  Experiments in simulation and real-world show that learned motions can efficiently adapt to new tasks, demonstrating high autonomy from free-text commands in unstructured scenes. Videos and website: hy-motion.github.io//

HYPERmotion: Learning Hybrid Behavior Planning for Autonomous Loco-manipulation
Jin Wang!?,
12t Rui 22 Weijie Wang 22 Luca Rossini', Francesco Ruscelli', Nikos Tsagarakis'
Istituto Italiano di Tecnologia
Universita Genova Eqqua Contribution
Abstract: Enabling robots autonomously perform hybrid motions in diverse environments can be beneficial for long-horizon tasks such as material handling, household chores, and work assistance. This requires extensive exploitation of in- trinsic motion capabilities, extraction of affordances from rich environmental in- formation, and planning of physical interaction behaviors. Despite recent progress has demonstrated impressive humanoid whole-body control abilities, they strug- gle to achieve versatility and adaptability for new tasks. In this work, we propose HYPERmotion, framework that learns, selects and plans behaviors based tasks in different scenarios. We combine reinforcement learning with whole-body optimization to generate motion for 38 actuated joints and create motion library to store the learned skills. We apply the planning and reasoning features of the large language models (LLMs) to complex aco-mmnipuulatio tasks, constructing a hierarchical task graph that comprises series of primitive behaviors to bridge lower-level execution with higher-level planning. By leveraging the interaction of distilled spatial geometry and 2D observation with visual language model (VLM) to ground knowledge into robotic morphology selector choose appro- priate actions in single- dual-arm, legged wheeled locomotion. Experiments in simulation and real-world show that learned motions can efficiently adapt to new tasks, demonstrating high autonomy from free-text commands in unstruc- tured scenes. Videos and website: motion.githuubiio
Open the drawer under the desk, find the drill and put it into the tool box, then bring me the box.
1
noee sening
Features
LLM
Mode
Taak Graph
5
0
Motion Learning
flat floor
</Leg Mode>
</wheel Mode>
Figure 1: HYPERmotion enables the humanoid robot to learn, plan, and select behaviors to complete bonghhorizon tasks. Steps 1-5 illustrate how the robot, guided by foundation models, au- tonomously performs locomotion and manipulation after interpreting verbal instruction and chooses motion modes for different scenarios independently.
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

Introduction
Humanoid robots with behavioral autonomy have consistently been regarded as ideal collaborators in our daily lives and promising representations of embodied intelligence. Compared to fixed-base robotic arms, humanoid robots, due to their configuration characteristics, offer a larger operational space while significantly increasing the difficulty of control and planning. Despite the rapid progress towards purposs humanoid robots [1, 2], most studies remain focused on locomotion with few investigations into learning whole-body coordination, which results in simplistic control designs that struggle to adapt to new tasks and environments, thus limiting the potential to demonstrate long- horizon tasks under open-ended instructions. One major challenge is how to explore actionability of humanoid robots and diversify their behaviors, while learning to infer affordances and spatial geometric constraints, thus making plans for various tasks with learned skills like humans.
Recently, vibrant advances in robotics learning have made a promising avenue for manipulation and locomotion [3, 4, 5, 6]. Learning- bearning-based methods, such as reinforcement learning (RL), have become effective tools for task-oriented action generation [7, 8, 9] while facilitating generalization across diverse scenarios. However, extending learning algorithms to humanoid robots remains a challenge stemming from the exponential increase in training costs induced by high degrees of free- dom (DoF) and the difficulty of deployment on real robots under dynamic constraints. Mcanwhile, the rise of large language models (LLMs) and their remarkable capabilities in robotic planning [10, 11, 12] have made it possible to perform logical reasoning and construct hierarchical action sequences for complex tasks. By integrating observations from different modalities, those models can be used for extracting features of objects and environments for robot perception and decision- making. Nevertheless, limitations to the utilization of LLM in humanoid robots exist, particularly in complex whole-body motion control and precise coordination between body parts.
To address these issues, we first recognized that directly outputting whole-body trajectories for a real-world multi-joint system through simulation training is inefficient and impractical. Therefore, adopt decomposed training strategy that modularly selects the actuation components related to given tasks, and project lower-dimensional space trajectory on the whole-body space with a unified motion generator. The trained actions are stored as skill units in the motion library. We utilize the LLM's ability to decompose complex semantic instructions consisting of multiple sub-tasks and design a modular user interface as model's input. The LLM selects skills from the motion library and arranges a sequence of actions, referred as task graphs. Furthermore, the 3D features extracted from captured 2D images and depth data can be integrated with the visual language model (VLM) and robotic intrinsic characteristics, acting a robotic motion morphology selector.
We refer to this study as HYPERmotion, framework that tackles behavior planning for humanoid robot autonomous loco- rico-manipulation using language models. By leveraging the interaction of dis- tilled spatial geometry and 2D observation with VLM, it grounds knowledge to guide morphology selection combining robotic affordance. And bridge the gap between semantic space, robotic per- ception and action. We demonstrate learning-based whole-body control to generate humanoid motion that adapts to new tasks and performs hong-horizon tasks with primitive skills. We further illustrate through experiments how HYPERmotion can be learned and deployed on high-DoF, hybrid wheeled-leg robot, performing zero-shot online planning under human instructions.
2 Related Work
Planning and Reasoning via Language Model Grounding pre- pre-trained language models has be- come a promising avenue for robotic studies. Extensive prior works [13, 14, 15, 16) focus on plan- ning and reasoning of robotic tasks with LLM, using it as a tool for code generation [10, 17], reward design [18, 19, 20], and interactive robotic learning [21, 22, 11, 23]. Several wransformer-base architectural planners [24, 25, 26] showcase the potential for embodied usage of generating robot action. And with the integration of multi modalities such as visual and auditory [27, 28, 29, 30, 31], perception and behavior can be directly bridged with semantic commands. Further research involves creating customized skill library[32, 33, 12] to link robot execution and high-level planning. A related line of works [34, 35, 36, 37] has also explored grounding affordances with foundation mod-

---

els to enable spatial reasoning and guide manipulation. However, most efforts focus on employment of fixed robotic arms, with few attempts made to extend language modes to humanoid robots, due to their complex dynamics and precise coordination between different components. [2] presents an end-to-end humanoid manipulation towards speech reasoning but lacks the cooperation of mobil- ity. [38][39] use LLMs for decision making and learning. while demonstrating solely in simulation scenes. In contrast, realize language model based online planning and humanoid motion boot- strapping, distilling spatial knowledge for robotic morphology selection using only onboard sensing.
Task-orient Humanoid Control As the practical value of general-purpose humanoid robots be- comes evident, a substantial amount of research has focused on the hardware of humanoid robots [40, 41, 42], as well as gait generation and balance control [43, 44, 45, 46]. Methods based on learn- ing and model predictive control (MPC) have significantly enhanced the mobility of such robots [47, 48, 49, 50, 51, 52, 53]. Some demonstrate motion generation through teleportation [54] and im- itation learning [55], while these often lack autonomy and struggle to organize learned short-horizon skills. Recent works on long-hove onn-horizon tasks [56, 57] and bimanual coordination [58, 59] show dex- terity and stabilization, but these are often limited to specific tasks to utilize their characteristics. Our approach enables the robot to perform composite tasks involving locomotion and manipulation unstructured environments, engaging in rich physical interactions with various objects. Our work can also decompose tasks into sub-modules based on verbal instructions and autonomously perform tasks using pre-trained motions.
Learning-based Whole-body Motion Recent research has demonstrated significant advancements in robust walking [60, 61], trotting [62, 63, 64), and parkour [7, 8, 9] for legged robots using end-to- locomotion policy with model-based manipulation of attached arm shows feasible whole-body motion on rough terrain [5]. However, most learning- -baed controllers are implemented quadruped robots with few DoFs, while highly redundant humanoids rarely addressed. For the latter, optimization- based control is still necessary ensure the safety and adherence to constraints due to limited reactive frequencies. Nowadays, learning-based MPC demonstrates capabilities in system dynamics identification [65, 66), closed loop performance [67, 68, 69] and safety assurance [70, 71, 72]. [73] shows a whole-body MPC on legged manipulators, but the tasks are limited by manually defined trajectories. this work, we leverage RL to enable a wide range of motion skills without relying on predefined trajectories and employ low-level optimization-bant based controller ensure the feasibility of whole-body motion.
end RL, The combination of learning-base
3 Methodology
In this section, we illustrate how the HYPERmotion framework enables the humanoid robot to au- tonomously perform loco-manipulation guided by semantic instructions (Sec. 3.1). We then provide the method for task-orient whole-body motion learning policy and how we build the humanoid mo- tion skills library (Sec. 3.2). We further describe how to achieve robotic morphology selection based on spatial reasoning by integrating multi-modality language models, and map the long-horizon task to hierarchical behavior structure using learned motions (Sec. 3.3).
3.1 Autonomous Loco-manipulation via HYPERmotion
To address the autonomous loco-manipulation challenge for complex robotic platforms such as hu- manoid robots. This work proposes a method to perform language-guie guiiedd behavior planning, motion generation and selection towards different scenarios. As shown in Fig 2, we divide the pipeline into four interrelated sectors that are learned and deployed sim-to-real manner. The motion generation sector selects RL training configurations for specific tasks and conducts training in parallel. The trajectory obtained from the training is provided as a reference to the optimizer, which ultimately generates whole-body motion skills and the skills will be stored in the motion library. The user input sector contains a user interface as well as pre-defined basic prompts, function options, and motion library, all of which together constitute the textual material fed to the L.M. After receiving a command, the task planning sector first generates a hierarchical task graph that includes task logic, condition determinations, and actions using the LLM. Once the task graph is loaded, it is interpreted a Behavior Tree to guide the robot and to pass actions to lower-level execution. When task
3

---

Motion Generation
Morpholigyy Selecior
Wholekodly
Traiing
Tregeborry Fin vennict
2D M
Tabk
Loconatoon
Proner
topur
Planning
BT
Xbot
Large
Task
Open the xpp drawer. find the diil..
Language Model
stoeetstee unkoo coiet
Basic Prompts
Funsson Options
Molion Library
Training Contig
BT Behavior Tree
Figure 2: Overview of HYPERmotion. We decompose the framework into four sectors: Motion generation is assigned for learning and training whole-body motion skills for new tasks and storing them in the motion library. User input includes received task instructions and initialization prompt sets. Task planning generates a task graph that guides the robot's behavior through reasoning and planning features of LLM and passes action commands to the real robot. Morphology Selector is used for action determination in specific sub-tasks, selecting the appropriate morphology for loco- motion and manipulation based grounded spatial knowledge and robot intrinsic features.
requires selecting the robot's morphology, depth-sensing information is invoked and distilled into 2D images and geometric features. These data, along with the task state and prompts, are fed to the VLM, which then selects the morphology capable of achieving the goal. Through the coordi- nation of these sectors, HYPERmotion facilitates semantic command understanding and zero-shot behavioral planning and action execution for humanoid robots, shown in Fig 1.
3.2 Learning Whole-body Motion Generation
Tasks Learning with RL In this section, we show the details of learning whole-body motion gen- eration based on different tasks, as shown in Fig. 3. To reduce the action space, we separate the robot's upper body from its legs. To guarantee feasibility, the floating base motion heuristically limited to avoid generating unfeasible motions for the legs once projecting the trajectory the whole-body space of the robot. For single-arm tasks, the action space A C RR1 consists of the 6-DoF right arm joint angles, 6-DoF floating base translation distances and Euler angles, one torso yaw, and one gripper joint angle. The left arm is fixed during these tasks. For the dual-arm picking task, the action space A2 S R 19 appends the 6-DOF left joint angles, with the gripper joint closed in this case. The observations include the states of the corresponding targets and the joint states of the robot's upper body, detailed in the Appendix along with the policy settings. All tasks utilize proximal policy optimization (PPO) [2] because of its efficiency. The output in the RL layer is joint position trajectory q' E R220 of the upper body. We train all skill policies separately using general reward formulation:
01 01 Q22Trrech + acTToot + a4T fänger + asTtask + acTpenalty
(1+37) )² and Trreach (T+T) )² with and d, representing the distance of the operational target to the left and right end effectors, respectively. The term Frot sign(dz) dh + sign(dz) d² is the reward for aligning the gripper's orientation with the task's object (e.g. drawer handle, door handle, drill). Here d, and de are the dot products of the gripper's forward and axes with the object's 's inward and up axes, respectively. The term finger = B - (d, + dz,) encourages the gripper to grasp the objects, where 3 fine-tuning parameter related to the size of the operational object, and d, and de are the distances from the top and bottom links of the gripper to the task's object, respectively. Tpenolty
where Tlroach
||al2 penalizes excessive actions a ensure smooth operation. Finally, Ftask denotes the specific reward for task completion, which will be detailed in the Appendix along with the specific settings of the parameters 0₁ to ag and the axes for different tasks.

---

Draweer ogening
Door opening
Dual picking
Single picking
Learning
Physical
Action
Figure Whole-body tasks learning illustration in training, simulation and real-world settings.
Whole-body Motion Generation The reduced robot space from RL should be mapped into the whole body joint space to generated feasible trajectory. To fit the gap between the upper body trajectory and the whole body action, we solve an optimal control problem to merge them enforcing the whole-body dynamics the robot to guarantee the feasibility of the resulting motion. In this case, we choose a unified whole-body trajectory generator[75), specifically, we use the framework presented in [75] solve the following non-linear problem:
minx() 2 L(x(t), u(t), t)dt (() ((((), u(),
(2)
91 (x(t) (tt),tt)))))..))) =
92 (x(t) ((t, 5
where x(!) = [q. v] E RM,, n, = 93 is the state number,u(() - [v, fe] R"=,, nu. 58 are vectors of state and input variables, q, are the generalized coordinates and generalized velocities, f. is foot is intermediate cost, q" is the upper body joint variables. (x((), u(t),t) is the whole body dynamics, 91 represents the set of equality constraints, 92 the set of inequality constraints consisting of joint limits, velocity limits, and unilaterality of the contact forces. Finally, we could get the whole body motion combining upper body reference trajectory and whole body dynamic feasibility to complete the given tasks.
force. L(x, u, = ||q" - q" + ||l|| 2 As for constraints, x(1)
Motion Library After learning task-orient motion skills, we constructed a motion library to host these primitives, which consists of attribute and functional descriptions of these actions, and the corresponding learning-based whole-body policies. Then, the LLM can reason the attribute-function descriptions to create sequences of actions be executed based on different tasks and generate task graph to invoke the execution of each node without additional training or demonstration.
3.3 Humanoid Robot Task Planning with grounded language models
Migrating foundation models from a fixed robotic arm to a humanoid robot with a floating base presents numerous issues and challenges. The addition of robotic components not only imposes complex dynamic constraints, making difficult to coordinate and control various parts. It also requires addressing the potential for different manipulation modes inherent in human-like structures, as well as the increased DoF for spatial mobility by the addition of wheels and legs. Due to the construction of our motion library, the usage of the LLM for planning no longer requires additional considerations for constraints such as self-collision or self-posture balance maintenance. This allows more focus on the decomposition given tasks, and the selection of the robot's morphology.
Humanoid Motion Morphology Selection Humans utilize common sense and learned experiences to extract the affordances of objects they manipulate and select appropriate movement based on the estimation of geometric constraints of the environment. Inspired by this, we leverage VLM to
5

---

implement similar functionalities in robots. First, we include descriptions of the robot's structure and functions, and the robot's achievable range of motion in the prompts Pv- While determining the morphology for manipulation task Tm. the robot utilizes 2D and depth images from its head camera. Object detection and pose estimation algorithms [76, 77] are invoked to acquire the position and orientation of the target object Ve RR which then transformed into the robot's coordinate system Vn RR The VLM V, based on the current task state s, the scene's 2D images I seeee and the target object's ' 6D pose Vis generates the robot's manipulation morphology Xm for the task scenario. For locomotion tasks, the robot uses the depth information from its pelvis depth camera to generate the point cloud Pc. which is down-sampled to create a voxel grid Vo- This spatial information containing the current moving path together with the 2D image Le and the task state are finally fused to select the robot's locomotion morphology x; using the VLM V.
Xm ((s, Remee VR. pv)
(3)
X = Vss, Secnce, Vo Pv)
(4)
Head Camera
Depth image
Va
O Veniouinton Node
20
M
</s5tngle-tre</>
S
Task
VLM
Pelvis Camera
20 path image
Locerision Made
Down
Sample
cooud
Figure 4: Robotic Morphology Selector extracts spatial geometric data and 2D observations from the physical environment upon receiving the language-conditioned task state and interacts with the VLM incorporating the grounded robot's 's affordances, so as to provide the optimal motion morphol- ogy that meets the requirements of given task scenario during manipulation and locomotion process.
Zero-Shot Robot Behavior Planning To obtain the desired response from L.IM, necessary impose constraints on the input. In the user interface, we define three types of constraints. The basic prompt provides a description of task background and characteristics of the robot, well as interpretation of the user command and the output format. The motion library offers catalog of learned skills and their description. Function option module offers specifications of the added func- tions developed for humanoid robot and determines whether these predefined functions are invoked during planning. Such as, if the morphology selection chosen, the LLM will incorporate the mor- phology selector based on the task scenario; otherwise, this function will not be considered (See the Appendix). This approach allows for systematic construction of prompts and modular addition of constraints, thereby enhancing the flexibility of planning. We utilize BehaviorTree (BT) [78] as an intermediate bridge to convert high-level instructions into executable low-level skill sequences. BT provides hierarchical structure for guiding actions and making decisions for the robot, which is composed of nodes with different effects. With a pre-defined motion library, LLM can generate task graph consisting of learned motions and BT nodes, build it in an XML file, which constructs the complete BT. Thus realizing the robot's behavior planning with LLM by giving verbal instruction.
4 Experiment
We demonstrate HYPERmotion's ability to learn, plan, and select behaviors for different tasks in both simulation and real-world experiments using objects that can be commonly found in daily life. Our robot is a centaur-like humanoid robot, supported by four legs with wheels. The robot has two with one claw gripper on its right arm. There are two depth cameras, one on the head and another is in the pelvis position. We use Xbot [79] to achieve real-time communication between the underlying actuators and the control commands. We use Isaac Gym [80] training environment

---

0.8
B
0.8
x 04
g/m
FC3
0.2
(urrea)
-0.6
10
20
30
40
50
70
10
30
(m))
10 20
50
70
10 waypoints
30
40
waypoints lndex
(a) opening the drawer
(b) opening the door
Figure 5: End effector position trajectory when executing different tasks in various environments
and validate the planned whole-body motion using Gazebo [81]. We access the gpt-4o model as the LLM planner and the gpt-4v model the VLM from OpenAl API [82].
We trained individual motion primitives based on everyday tasks, deploying these skills on the robot to verify their feasibility (Sec. 4.1). We tested the adaptability and versatility of morphology selector various task scenarios (Sec. 4.2) and conducted further study to validate the behavioral planning capability and performance of our framework in response open-ended instructions (Sec. 4.3).
4.1 Whole-body Trajectory Learning
We pick two representative motions and compare the trajectories in Isaac Gym, Gazebo and the real world, respectively, as illustrated in Fig. 5, using the end effector position trajectories as an example. In the drawer opening task, the end effector first approaches the drawer handle and then pulls it out, demonstrating an initial increase followed by decrease in the x-axis distance. In the door opening task, the end effector reaches the door handle and pushes it down. Self-collision is considered during the training period because we still need to account for collisions between the upper body and legs, causing the arm to reach the target point at faster speed. Instead, the whole-body planner projects the trajectories within constraints to avoid self-collision of the entire body. The trajectories from training are effectively tracked, with smoother results achieved after applying the whole-body controller as a filter. This indicates the successful deployment of our methods on the real robot.
4.2 Morphology Selection Towards Different Scenarios
We investigated whether a VLM can zero-shot determine robot's s morphology based on task scenarios. We picked ten scenarios each for manipulation and locomotion in both simulation and real-world environments (See the Appendix). We compared the success rate of the VLM's morphology selection using 2D image input only versus image combined with spatial geometric input. Each scenario tested 10 times under both inputs, shown in Fig. 6. We found the morphology selector effectively chooses the optimal mode for ev- eryday object manipulation and mobile environment with a high av- erage success rate. Compared to solcly image input, adding spatial information improves the selector's accuracy, particularly in deter- mining locomotion modes and adapting to complex scenarios (paths with obstacles of varying types and heights), thus leveraging the robot's is affordances and leading to robust execution.
Mob Scen-
com)
Scane
(10)
Manti
Figure 6: Success rate of the morphology selector for dif- ferent scenarios. "2D" and "SD" are image and Spatial Data inputs.
4.3 Loco-manipulation Tasks with Language Model Planner
Tasks with human instructions We validate the ability of LLM to plan motion primitives for differ- ent loco- arcommanipulation tasks, as well as the effect of the modular user input designed for humanoid robots regarding reasoning and planning. Experiments were conducted on tasks requiring a combi- nation of perceptions and actions. We recorded the success rate and the impact of different emros of 4 representative tasks and provided quantitative evaluations in Fig.8. The results show the LLM based planner effectively plan semantic instructions based on learned skills and guide the robot to complete variety of tasks according to the action sequences, achieving desired success rate

---

maco-aanipllation Task
Louk
Open the drawer
Pick up the drill
Place drill Into box and pick box
Give box to the instructor
Figure 7: Overall look of the long-horizon task Images above show the timelapse of roll outs to robot motion trajectory. A. semantic navigation using AprilTag. B. object detection and pose estimation. manipulation morphology selection. D. locomotion morphology selection. The BehaviorTree below shows the details of LLM task planning.
(2 60%) real-world robots. Whereas adding task complexity and selecting multiple functional modules as input increases the difficulty of planning, and execution errors mainly stem from intricate dynamical constraints on the actions and misalignment of the floating sensing with robot execution.
Long-horizon Task We further explored whether HY- PERmotion can enable behavior planning for humanoid robot towards long-horizen tasks. We orchestrated col-
Approach and
laborating task scenario and input verbal instruction shown in 1ig.1. Qualitative results including time-lapse shots of robot motion execution and a Behavior Tree
Open drower
mapped out by LLM are shown in Fig.7. We demon- strate that our framework can synthesize sequences of
pick
Duak-mm
32%
motion primitives based on designed user input and ac- curately infer the logic of semantic knowledge while se- lecting robotic morphology of locomotion and manipula- tion according to the environment and state of the task. We found that language-based behavior planner exhibits greater versatility and adaptability to more complex tasks compared to existing methods.
Plannina Enor
Exacution trerr
Figure 8: Average rate of the humanoid robot successfully performing various LLM planning tasks, and failure caused by different type of errors during tasks.
Conclusion
We present HYPERmotion, framework that enables humanoid robots to learn, select, and plan behaviors, integrating knowledge and robotic affordance to perform embodied tasks. We evaluate the framework's 's efficiency and versatility through real-world experiments and long-horizon tasks. Despite achieving expected results, there are limitations: the motion library's ' size restricts the range of task commands, and learning of new skills requires separate training optimization, hindering generalization from existing actions. Moreover, the system struggles to handle external disturbances and collisions, lacks real-time linguistic interaction during the task and has limited capability for re-planning in response to unexpected tasks. Future work will focus enriching the robot's 's action skills, enhancing LLM dynamic planning ability, and improving robot navigation and perception to achieve close-loop humanoid motions and safe human-robot collaboration.

---

Acknowledgments
This work was supported by the European Union's Horizon 2020 research and innovation pro-
gramme, euROBIN EPUE034001.
References
[1] Tesla. Optimus gen 2. Dec. 2024. URL http:::/wwt..ouuuue.c.oo////////
epraXaw7dyc.
[2] Al. Figure status update- openai speech-to sp-secch reasoning, Mar. 2024. URL https:
//www. www.coo///oaao oatp:e/ee/eoomtctctto///g1/1///
[3] O. Kroemer, S. Nickum, and G. Konidaris. A review of robot learning for manipulation: Challenges, representations, and algorithms. Journal of Machine Learning Research, 22(30): 1-82, 2021. URL http: /ttp:///mmmreee//e/ org/ pttpp/appppp111--aaaaa http:ppp//.....ooooppppsss/////
htmm.
[4] H. Zhang, G. Solak, G. G. Lahr, and A. Ajoudani. Srl-vic: A variable stiffness-based safe reinforcement learning for contact-rich robotic tasks. IEEE Robotics and Automation Letters, 9()):6631-6638 2024. doi: 10i:10.14 101.///R..224.3366660000
[5] Y. Ma, Farshidian, T. Miki, Lee, and M. Hutter. Combining Cearning-based locomotion policy with model-based manipulation for legged mobile manipulators. IEEE Robotics and Automation Letters, 7(2):2377-235 2022. 100.1100////RR222214555550
[6] Q. Zhang, P. Cui, D. Yan, J. Sun, Y. Duan, A. Zhang, and R. Xu. Whole-body humanoid robot locomotion with human reference, 2024.
[7] 7. Zhuang, 7. Fu, J. Wang, C. Atkeson, S. Schwertfeger, C. Finn, and H. Zhao. Robot parkour learning, 2023.
[8] D. Hoeller, N. Rudin, Sako, and M. Hutter. Anymal parkour: Learning agile navigation for quadrupedal robots. Science Roboties, :(8))e,,,,7566, 2024.
ad17556..
URL httph:tt//:/// http://wwww.iicnccc wrg//oi//baae www.////cioootics.ccoomm
[9] X. Cheng, K. Shi, A. Agarwal, and D. Pathak. Extreme parkour with legged robots, 2023.
[10] J. Liang, W. Huang, F. Xia, P. Xu, K. Hausman, B. Ichter, P. Florence, and A. Zeng. Code as policies: Language model programs for embodied control, 2023.
[11] A. Z. Ren, A. Dixit, A. Bodrova, S. Singh, S. Tu, N. Brown, P. Xu, L. Takayama, Xia, J. Varley, Z. Xu, D. Sadigh, A. Zeng, and A. Majumdar. Robots that ask for help: Uncertainty alignment for large language model planners. In 7th Annual Conference Robot Learning, 2023. URL https::///// /ttpss://ppente /tpp//evi///eeeeeee net/forum comummic fedd:?dd==zsas 4Zks0=0KKK.....
[12] brian ichter, A. Brohan, Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Ser- manet, A. T. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sievers, C. Tan, S. Xu, D. Reyes, J. Rettinghouse, J. Quiambao, P. Pastor, L. Luu, K.-H. Lee, Y. Kuang, S. Jesmonth, K. Jeffrey, R. J. Ruano, J. Hsu, K. Gopalakr- ishnan, B. David, A. Zeng. and C. K. Fu. Do as can, not as say: Grounding lan- guage robotic affordances. In 6th Annual Conference on Robot Learning, 2022. URL https://openrevier: www.forumffmmmiiee DiddHH
[13] K. Kawaharazuka, T. Matsushima, A. Gambardella, J. Guo, C. Paxton, and A. Zeng. Real- world robot applications of foundation models: A review, 2024.
[14] W. Huang, P. Abbeel, D. Pathak, and Mordatch. Language models as zero-shot planners:
Extracting actionable knowledge for embodied agents, 2022.

---

[15] M. G. Arenas, T. Xiao, S. Singh, V. Jain, A. Z. Ren, Q. Vuong, J. Varley, A. Herzog, I. Leal, S. Kirmani, D. Sadigh, V. Sindhwani, K. Rao, J. Liang, and A. Zeng. How to prompt your robot: A promptbook for manipulation skills with code as policies. In 2nd Workshop on Language and Robot Leurning: Language as Grounding, 2023. URL https: //openreview net/forrmmiddTTAA77]11
[16] S. Zhang, P. Wicke, L. K. Şenel, L. Figueredo, A. Naceri, S. Haddadin, B. Plank, and H. Schütze. Lohoravens: A long-horizon language-condittoned benchmark for robotic tabletop manipulation, 2023.
[17] I. Singh, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models. In Workshop on Language and Robotics at CoRL 2022, 2022. URL https://openreviiee
[18] S. Han, I. Shenfeld, A. Srivastava, Y. Kim, and P. Agrawal. Value augmented sampling for language model alignment and personalization, 2024.
[19] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang, T. Erez, L. Hasenclever, Humplik, brian ichter, T. Xiao, Xu, A. Zeng, T. Zhang, N. Heess, D. Sadigh, J. Tan, Tassa, and F. Xia. Language to rewards for robotic skill synthesis. In 7th Annual Conference on Robot Learning, 2023. URL https://openreviee..ee//// SgTPdyehXMA.
[20] Y. Tang, W. J. Tan, H. Zen, A. Faust, and T. Harada. Saytap: Language to quadrupedal lo- comotion. In 7th Annual Conference on Robot Learning, 2023. URL https://pperreviion net/forum 1d=7TYe02XVQQ..
[21] I.. Zha, Y. Cui, I..H. Lin, M. Kwon, M. G. Arenas, A. Zeng. Xia, and D. Sadigh. Distilling and retrieving generalizable knowledge for robot manipulation via language corrections, 2024.
[22] S. Belkhale, T. Ding, T. Xiao, P. Sermanet, Q. Vuong, J. Tompson, Y. Chebotar, D. Dwibedi, and D. Sadigh. Rt-h: Action hierarchies using language, 2024.
[23] Y.J. Wang, B. Zhang. J. Chen, and K. Sreenath. Prompt a robot to walk with large language models. arXiv preprint UrXiv:2309.09969 2023.
[24] L. X. Shi, Z. Hu, T. Z. Zhao, A. Sharma, K. Pertsch, Luo, S. Levine, and C. Finn. Yell at your robot: Improving on fl- from language corrections, 2024.
[25] A. Brohan, N. Brown, J. Carbajal, Y. Chebotar, J. Dabis, C. Finn, K. Gopalakrishnan, K. Haus- man, A. Herzog, J. Hsu, J. Ibarz, B. Ichter, A. Irpan, T. Jackson, S. Jesmonth, N. J. Joshi, R. Julian, D. Kalashnikov, Y. Kuang, I. Leal, K.-H. Lee, S. Levine, Y. Lu, U. Malla, D. Manju- nath, I. Mordatch, O. Nachum, C. Parada, J. Peralta, E. Perez, K. Pertsch, J. Quiambao, K. Rao, M. Ryoo, G. Salazar, P. Sanketi, K. Sayed, J. Singh, S. Sontakke, A. Stone, C. Tan, H. Tran, V. Vanhoucke, S. Vega, Q. Vuong. F. Xia, Xiao, Xu, S. Xu, T. Yu, and B. Zitkovich. Rt-1: Robotics transformer for real-world control scale, 2023.
[26] B. Zitkovich, T. Yu, S. Xu, P. Xu, T. Xiao, F. Xia, J. Wu, P. Wohlhart, S. Welker, A. Wahid, Q. Vuong, V. Vanhoucke, H. Tran, R. Soricut, A. Singh, J. Singh, P. Sermanet, P. R. Sanketi, G. Salazar, M. S. Ryoo, K. Reymann, K. Rao, K. Pertsch, I. Mordatch, H. Michalewski, Y. Lu,
S. Levine, L. Lee, T.-W. E. Lee, I. Leal, Y. Kuang, D. Kalashnikov, R. Julian, N. J. Joshi, A. Irpan, brian ichter, J. Hsu, A. Herzog, K. Hausman, K. Gopalakrishnan, C. Fu, P. Florence, Dubey, Driess, T. Ding, K. M. Choromanski, X. Chen, Y. Chebotar, Car- bajal, N. Brown, A. Brohan, M. G. Arenas, and K. Han. RT-2: Vision-language-action models transfer web knowledge to robotic control. In 7th Annual Conference on Robot Learning, 2023.
C. Cinn,,,
URL https: httpss:///peeeeeieeeeeeee wwwp:eewwwwieeeetnuuneeuueeeeeee
10

---

[27] S. Saxena, M. Sharma, and O. Kroemer. Multi-resolution sensing for real-time control with vision-language models. In 7th Annual Conference on Robot Learning, 2023. URL https: //openreview net/forum' -itrun//ddiWBBBIIOOU
[28] L.-H. Lin, Y. Cui, Hao, F. Xia, D. Sadigh. Gesture-informed robot assistance via foundation models. In 7th Anmual Conference on Robot Learning, 2023. URL https:// openreview.
[29] Z. Liu, A. Bahety, and S. Song. REFLECT: Summarizing robot experiences for failure explanation and correction. In 7th Annual Conference on Robot Learning, 2023. URL https://openreviee
phet/forumiid-8jp) __Ilxx..
[30] D. Driess, F. Xia, M. S. M. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu, W. Huang, Y. Chebotar, P. Sermanet, D. Duckworth, S. Levine, V. Vanhoucke, K. Hausman, M. Toussaint, K. Greff, A. Zeng, I. Mordatch, and P. Florence. Palm-e: An embodied multimodal language model, 2023.
[31] A. Stone, T. Xiao, Lu, Gopalakrishnan, K.-H. Lee, Q. Vuong, Wohlhart, Kirmani, B. Zitkovich, F. Xia, C. Finn, and K. Hausman. Open-world object manipulation using pre- trained vision-language models. In 7th Annual Conference Robot Learning, 2023. URL httppp////ppeeeeeeeeeeeee//e http://oormmmdd91116660 wat.iidddlllaaaaaaa...
[32] J. Zhang, J. Zhang, K. Pertsch, Z. Liu, X. Ren, M. Chang, S.-H. Sun, and J. J. Lim. Bootstrap your own skills: Learning to solve new tasks with large language model guidance. In 7th Annual Conference on Robot Learning, 2023. URL https:t///// http:///ppeeeeeeeeeeeeeeeeeeeeee wwwprrvviee.nee/fouumiii.. net/forummid aOmFRgadGO.
[33] W. Huang, F. Xia, T. Xiao, Chan, Liang, Florence, A. Zeng. Tompson, I. Mordatch, Y. Chebotar, P. Sermanet, Jackson, N. Brown, L. Luu, Levine, Hausman, and brian ichter. Inner monologue: Embodied reasoning through planning with language models. In 6th Annual Conference Robot Learning, 2022. URL https: http:///e////eeeeeeee/mmm 3R3PZ510tye.
[34] W. Shen, G. Yang, A. Yu, J. Wong. L. P. Kaelbling, and P. Isola. Distilled feature fields enable few-shot language-guinn guided manipulation. In 7th Annual Conference on Robot Learning, 2023. URL https://openreviii
net:/forumidd-bonnI..
[35] W. Huang, C. Wang. R. Zhang, Y. Li, Wu, and L. Fei-Fei. Voxposer: Composable 3d value maps for robotic manipulation with language models. In 7th Anmual Conference on Robot Learning, 2023. URL https: http::///ppeeeeeeeeeee wwt ceetfoorumiide 7id=9_8LF3000C
[36] A. Rashid, S. Sharma, C. M. Kim, J. Kerr, L. Y. Chen, A. Kanazawa, and K. Goldberg. Lan- guage embedded radiance fields for zero-shot task-oriented grasping. In 7th Annual Conference on Robot Learning, 2023. URL https::/openreviiii wwt.:eettoooooeeee Fid=kFFg8JD@mmm
[37] B. Chen, Z. Xu, S. Kirmani, B. Ichter, D. Driess, P. Florence, D. Sadigh, L. Guibas, and F. Xia. Spatialvim: Endowing vision-language models with spatial reasoning capabilities, 2024.
[38] Y. Guo, Y.J. Wang, L. Zha, Z. Jiang, and Chen. Doremi: Grounding language model by detecting and recovering from plan-execution misalignment, 2023.
[39] Y. Wang, Zhou, Chen, T.-H. Wang, Wang, Fragkiadaki, Z. Frickson, D. Held, and
C. Gan. Robogen: Towards unleashing infinite data for automated robot learning via generative simulation. In arXiv (2311.01455) 2023.
[40] B. Dynamics. Atlas@ and beyond: the world's most dynamic robots, 2024. URL https: / //bostoonynamics.. cwm/ atlas/.

---

[41] U. Robotics. Unitree introducing unitree gl humanoid agent ai avatar price from 16k, May
2024. URL httths:///wwwt wwtp///.www..uub...ccoo
[42] A. Roboties. Meet digit, mobile manipulation robot. URL com/ppodduct/ddigit.
[43] I. Radosavovic, T. Xiao, B. Zhang, T. Darrell, J. Malik, and K. Sreenath. Real-world humanoid locomotion with reinforcement learning, Science Robotics, 989)ceadi9579, 2024.
[44] B. van Marum, A. Shrestha, H. Duan, P. Dugar, J. Dao, and A. Fern. Revisiting reward design and evaluation for robust humanoid standing and walking, 2024.
[45] Z. Wang, W. Wei, R. Yu, J. Wu, and Q. Zhu. Toward understanding key estimation in learning robust humanoid locomotion, 2024.
[46] G. Zambella, R. Schuller, G. Mesesan, A. Bicchi, C. Ott, and J. Lec. Agile and dynamic standing-up control for humanoids using 3d divergent component of motion in multi-contact scenario. IEEE Robolics and Aulomation Letfers, 80))5622-5631, 2023. doi:10.1 109/LRA. 2023.3297060.
[47] Z. Li, X. B. Peng. P. Abbeel, S. Levine, G. Berseth, and K. Sreenath. Reinforcement learning for versatile, dynamic, and robust bipedal locomotion control, 2024.
[48] X. Gu, Y.-J. Wang, and J. Chen. Humanoid gym: Reinforcement learning for humanoid robot with zero-shot sim2real transfer, 2024.
[49] C. Sferrazza, D.-M. Huang, X. Lin, Y. Lee, and P. Abbeel. Humanoidbench: Simulated hu- manoid benchmark for whole-body locomotion and manipulation, 2024.
[50] Dadiotis, A. Laurenzi, and N. Tsagarakis. Whole-body mpe for highly redundant legged
manipulators: Experimental evaluation with 37 dof dual-arm quadruped. In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), pages 1-8, 2023. doi: 10. 2t0p:mammnnid571002222....
[51] Q. Zhang, P. Cui, D. Yan, J. Sun, Y. Duan, A. Zhang, and R. Xu. Whole-body humanoid robot locomotion with human reference, 2024.
[52] J. Dao, H. Duan, and A. Fern. Sim-to-real learning for humanoid box loco-manipulation, 2023.
[53] M. H. Yeganegi, M. Khadiv, A. D. Prete, S. A. A. Moosavian, and L. Righetti. Robust walking based on mpc with viability guarantees. IEEE Transactions on Robotics, 38(4):2399-2404,
2022. doi: 10.
[54] He, Z. Luo, W. Xiao, Zhang, K. Kitani, C. Liu, and G. Shi. Learning real-time whole-body teleoperation, 2024.
[55] X. Cheng, Y. Ji, J. Chen, R. Yang, G. Yang, and X. Wang. Expressive whole-body control for humanoid robots, 2024.
[56] C. Wang. L. Fan, Sun, R. Zhang, L. Fei-Fei, D. Xu, Y. Zhu, and A. Anandkumar. Mimicplay: Long-horizon imitation learning by watching human play. In 7th Annual Conference on Robot Learning, 2023. URL
[57] D. Driess, J.-S. Ha, R. Tedrake, and M. Toussaint. Learning geometric reasoning and control for long-horizon horizon tasks from visual input. 2021 JEEE international conference on robotics and automation (ICRA), pages 1428--1435.. IEEE, 2021.
[58] Z. Fu, T. Z. Zhao, and C. Finn. Mobile aloha: Learning bimanual mobile manipulation with low-cost whole-body telcoperation, 2024. 2024.
12

---

[59] J. Grannen, Y. Wu, B. Vu, and D. Sadigh. Stabilize to act: Learning to coordinate for bi- manual manipulation. In 7th Annual Conference on Robot Learning, 2023. URL https: //op //openreview. net/forumm 2 Jn6hX9F.
[60] N. Rudin, D. Hoeller, P. Reist, and M. Hutter. Learning to walk in minutes using massively parallel deep reinforcement learning. In Proceedings of the 5th Conference on Robot Learning, volume 164 of Proceedings of Machine Learning Research, pages 91-100. PMLR, 08-11 11 Nov 2022. URL https pttppeeeeeeeens.e/// mlr, wa...prss. /1w////uiii/2....//// htmml.liiiia...........mmm..
[61] S. Chen, B. Zhang, M. W. Mueller, A. Rai, and K. Sreenath. Learning torque control for quadrupedal locomotion. In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), pages 1-8, 2023. doi:10. (tttuummmmids5711002223..0000.
[62] T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine. Learning to walk via deep reinforcement learning, 2019.
[63] J. Lee, J. Hwangbo, L. Wellhausen, V. Koltun, and M. Hutter. Learning quadrupedal locomotion over challenging terrain. Science Robolics, 5(47):eabc3986, 2020. doi: 10. 1112//cciroooticsaacc5555 scirobotics.abctt
URL https://wwi www.nceeence..om .raeecceerrr//eeoeee wtrg/ddi/abb//tt 1126/
[64] J. Hwangbo, J. Lee, A. Dosovitskiy, D. Bellicoso, V. Tsoumis, V. Koltun, and M. Hutter. Learn- ing agile and dynamic motor skills for legged robots. Science Robotics, 4(26):eaa5877, 2019. doi: 122//ccrroootics.au58833 1126/scirobotice. aau5872.
URL
[65] V. R. Desaraju, A. Spitzer, and N. Michael. Experience-driven predictive control with robust constraint satisfaction under time-varying state uncertainty. In Robotics: Science and Systems, 2017.
[66] C. D. McKinnon and A. P. Schoellig. Learn fast, forget slow: Safe predictive learning control for systems with unknown and changing dynamics performing repetitive tasks. IEEE Robotics and Automation Letters, 4(2):2180-22 180-2187, 2019. doi: wtt.//.........91......
[67] A. Marco, Hennig, Bohg, S. Schaal, and S. Trimpe. Automatic lqr tuning based on gaussian process global optimization. In 2016 IEEE International Conference Robotics and Automation (ICRA), pages 270-277, 2016. 101.1.01...11111991191000 110.1110///RAA2211677777777
[68] R. Dai, W. Liu, Z. Bing, and Knoll. Variable weight model predictive contour control for autonomous tracking based reinforcement learning and nonlinear disturbance observer. In 2023 IEEE 26th International Conference Intelligent Transportation Systems (ITSC), pages 1563-1569, 2023. doi: 10. 10..1000/SS37777.2222.... 2022444777...22..00444.... 10422496.
[69] C. Finn, S. Levine, and P. Abbeel. Guided cost learning: Deep inverse optimal control via policy optimization. In M. F. Balcan and K. Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 49-58, New York, New York, USA, 20-22 Jun 2016. PMLR. URL attps://proceeddnggs mlr. wat..prees/v//.. www48/finnn6.... html.
[70] K. P. Wabersich and M. N. Zeilinger. Safe exploration of nonlinear dynamical systems: A predictive safety filter for reinforcement learning. arXiv preprint arXiv:1812.05506, 2018.
[71] K. P. Wabersich, L. Hewing, A. Carron, and M. N. Zeilinger. Probabilistic model predictive safety certification for bamiing-based control. IEEE Transactions Automatic Control, 67 (1):76-188,, 2022. doi: 10.
304993333...
[72] L. Hewing, K. P. Wabersich, M. Menner, and M. N. Zeilinger. Learning-based model pre- dictive control: Toward safe learning in control. Annual Review of Control, Robotics, and Autonomous Systems, 3(Volume 3, 2020):269-296, 2020. ISSN 2573-5144. doi:https:
13

---

wol.loorroo..l//mmuuooooooooooo0 URL http:://www...aaaaaaa annualreviews. http:///www......///m/oooooooooa
[73] I. Dadiotis, A. Laurenzi, and N. Tsagarakis. Whole-body mpe for highly redundant legged
manipulators: Experimental evaluation with 37 dof dual-arm quadruped. In 2023 IEEE-RAS
22nd International Conference on Humanoid Robots (Humanoids), pages 1-8, 2023. doi:
10. (100/Hummnndd5711002333317777770
[2] Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, Proximal policy optimization algorithms, 2017.
[75] Ruscelli, A. Laurenzi, G. Tsagarakis, and E. Mingo Hoffman. Horizon: trajectory optimization framework for robotic systems. Frontiers in Robotics and AI, 9:899025, 2022.
[76] B. Wen, W. Yang. J. Kautz, and S. Birchfield. Foundationpose: Unified 6d pose estimation and tracking of novel objects, 2024.
[77] J. Tremblay, T. To, B. Sundaralingam, Y. Xiang, D. Fox, and S. Birchfield. Deep object pose estimation for semantic robotic grasping of household objects. In Conference Robot Learn- ing (CoRL). 2018. URL https: / www.i..oggg/a.....e..o.o.oooolla att////s///100 wttp://w.rr//b////.....oo 10790.
[78] M. Colledanchise and Ögren. Behavior trees in robotics and Al: An introduction. CRC Press, 2018.
[79] A. Laurenzi, D. Antonucci, N. G. Tsagarakis, and L. Muratore. The xbot2 real-time middle- ware for robotics. Robotics and Autonomous Systems, 163:104379, 2023. ISSN 0921-8890. dothhugs//oooooo//////10l/////o0
URL hRLp::/:://wwp http://wwww.ciinnndrreee.c... com/
[80] V. Makoviychuk, L. Wawrzyniak, Guo, M. Lu, K. Storey, M. Macklin, D. Hoeller, N. Rudin, A. Allshire, A. Handa, and G. State. Isaac gym: High performance gpu-based physies simula- tion for robot learning, 2021.
[81] N. Koenig and A. Howard. Design and use paradigms for gazeho, an open-source multi-robot simulator. In 2004 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (IEEE Cat. No.04C137566), volume 3, pages 2149-2154 vol.3, 2004. doi:10.1109/ 100..000411
[82] J. Achiam, S. Adler, S. Agarwal, L. Ahmad, Akkaya, F. L. Aleman, D. Almeida, Altenschmidt, S. Altman, S. Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
14

---

Appendix
Contents
A1 Robot System Setup
16
A1.1 Robot hardware
16
A1.2 Robot software
16
A2 Additional Experiment and Evaluation
17
A2.1 Experiment Setup
17
A2.2 LLM based Task Planner
17
A2.3 Failure Cases Study
19
A3 Details of Robot Learning
19
A3.1 task design details
20
A3.2 task evaluation
21
A4 Details of Whole-body Optimization
21
A4.1 Formulation
22
A4.2 Dynamic Equation
22
A4.3 Feasible Constrain
22
A4.4 Cost
23
A5 Motion Library
23
A6 Motion Morphology Selection
25
A6.1 Manipulation Scenarios
25
A6.2 Locomotion Scenarios
25
A6.3 VLM Prompts
28
A7 User Input
29
A7.1 Basic Prompts
29
A7.2 Function Options
29
A7.3 User Interface
30
A8 Task Planning with LLM
30
15

---

A1
Robot System Setup
A1.1 Robot hardware
Our robot is a centaur-like robot platform. The upper body the robot is humanoid in design and similar in size the average human to adapt both dual-arm and single-arm manipulation. The robot's mobility relies on its quadrupedal lower body and maintains whole-body balance to cope with variety of terrain conditions and perform loco loco-manipulation tasks. Moreover, to improve the robot's mobility on flat ground, wheel modules are integrated underneath each leg and can control the direction and steering of the wheels.
The robot's whole body consists 38 actuatable joints. The robot's torso is mounted on the pelvis of the lower body via yaw joints, allowing the upper body to rotate in the transverse plane. Each arm of the robot includes DoF, where the right hand gripper contains opening and closing. The robot's legs are designed to provide an omni-directional wheeled motion and articulated legged locomotion, with each leg containing six degrees of freedom, allowing for positioning, orientation, and rotation of the wheeled-leg module.
extra DoF that controls its
The perception system of the robot consists two on-board RealSense Depth Camera D435i, one located in the robot's head and the other in the robot's pelvis, which are used to provide 2D images and depth information of the surrounding environment and objects. The complete computing system consists of two on-board computing units COTAC-ENI170K PC, COM Express conga-TSI700 170) for system communication and real-time robot control and an external pilot PC (Inter Core 19-03X0 CPU @ 3.3900 3.90GHz NVIDIA GeForce RTX 4090) for task planning and sensory data processing as well as a user interface.
Gripper
D435i camera,
wheeledHegg
Figure Al: Robot hardware setup
A1.2 Robot software
We use XBotCore, cross-platform, real-time, open-source software designed for interfacing with low-level hardware components of robots [1]. This innovative tool enables effortless programming and management of various robotic systems by offering a standardized interface that conceals the intricacies of the hardware. Additionally, a proprictary Cartesl/O motion controller [2] handles higher-order motion instructions. capable of managing multiple responsibilities and restrictions, prioritized according to the demands of specific situations. Through solving series of quadratic programming (QP) challenges, each linked to unique priority tier, the controller ensures optimal performance across all preceding priority stages.
16

---

A2 Additional Experiment and Evaluation
A2.1 Experiment Setup
We experimentally validate the efficacy of HYPERmotion in behavior learning, planning, and Decision-making by implementing the framework and evaluating its performance on wheel-legged humanoid robot executing long-horizon tasks in response to semantic commands.
We conducted the experiment using objects that are commonly found in an office kitchen. The test environment was open area inside the lab, with objects randomly placed. The AprilTag system [3], which incorporates vision-driven algorithm, was used during the long-horizon task to identify the relative objects' location and direction of recognized tags. Within the actual environment, we employ AprilTags to gather task-specific observations. A single visual marker on the door allows for the determination of the door handle's ' relative position. The robot searches for the tag doesn't ' exit the camera's field of view (FOV). Additionally, AprilTags enable the identification of the drawer's relative positions.
A2.2 LLM based Task Planner
Long-horizon Task
A long-horizon task refers to task that needs to be completed extended time period, typ- ically involving the execution of multiple subtasks and decision-making across several steps. Such tasks usually require the system to possess strong planning capabilities and long-term strategies to effectively achieve objectives in complex environments, In this study, we define long-horizon tasks based on the number of robotic actions included in the task. When the task described by the given instructions comprises four or more robotic actions, we classify it as long-horizon task. Here, robotic actions refer to those included in the action library. For example, the task 'Pick the box' consists of three actions: Defect >, MoveTTarret >, and < DualPick >, and is simple motion task. In contrast, the long- long-horizon task described in Fig.
thus categorized requires the robot to perform multiple actions, including < FindObject >, < MoveT arget >, < Object >, < SinglePick >, and < OpenDrawer > et., with a longer time span required for task completion.
Behavior Planner
We initially conducted simulation experiments to compare the construction of behavior trees using traditional method [4] with LLM based planner. The experiments included qualitative analysis of the capabilities of both approaches, as well as the planning results for various robotic tasks. Exe- cutable rate refers to whether the generated BT is logically consistent and executable by the robot. Compared to normal BT-planner, the integration of LLMS'' reasoning and decision becision-making capabil- ities enables the direct generation of BTs from tasks described in free-text instructions. Moreover, BT can be restructured by simply altering the input instructions, significantly enhancing the auton- omy of task planning. Quantitative analysis in Table Al indicates that BTs generated by LLMs exhibit higher executability and task success rates compared to those generated by BT-planner. Ad- ditionally, the time required for robots to complete tasks in the simulation environment was reduced. These results underscore the advantage of LLMs' reasoning and text generation capabilities con- structing behavior trees, which rely on a deep understanding of task logic and adherence to fixed construction rules.
Loco-manipnlation Tasks Simulation Results
After verified that the LLM-based planner is capable of effectively planning robotic tasks, we se- lected six mooppuaaptlation tasks to validate the performance and adaptability of the HYPERmotion framework performing different tasks. We compared its performance with traditional whole-body MPC control methods [5] and conducted 25 experiments for each tasks separately, the simulation results are shown in Table A2.
17

---

Table Al: Comparison of different methods for behavior planning
Abilities
Loco-manipulation Task
Long-horizon Task
Method
Autonomy Replan
Text Input
Exec
Succ 1 Avg. Time 4
Exec 1 Succ f Avv. Time +
BT-Planner
low
X
92%
80%
39,07s 12.4
74%
68%
145.37s 32.6
LLM-BT
high
98%
94%
32.7% + 8.6
86%
82%
121.20s 20.4
Table A2: Comparison of different methods towards various robotic tasks. TLE(time limit exceeded)
WB-MPC
LLM-Planner
LLM-Planner+-MS
Task
Suce
Avg. Time 4
Succ f
Avg. Time
Suce
Avg. Time +
Move target
84%
25.7s =
96%
10.6
92%
40.1s 29.3
Approach Object
72%
24.3s =
88%
34.9s + 8.6
96%
39.8s 22.1
Open door
64%
84%
12.6
88%
42.5x
Pick object
68%
38.9s 1:
80%
1:
84%
44.3s 22.3
Pick and place object
60%
50.2s 47.3
72%
49.26 23.6
84%
54.7s 32.1
Open drawer and pick object
0%
TLE
64%
72%
126.3s + 33.6
The experimental results indicate that using an LLM to plan pre-trained motion primitives for exe- cuting voco-manipulation tasks yields higher success rate compared to WB-MPC, particularly for complex tasks such 'pick and place object' and tasks requiring long-term planning such as 'open drawer and pick object.' However, for some simpler tasks, the time required by the LLM-Planner is slightly longer than that of WB-MPC. attributable to the time taken for online API requests and BT construction. Adding Morphology Selector (MS) into the task planning process results in modest increase in the average planning time, but it concurrently enhances the success rate of tasks, especially those involving interaction with the surrounding environment and objects.
Success Rate
70
Complexity Tasks
Figure A2: Success rate of loco-manipulation tasks in relation to task complexity.
We further investigated the relationship between the success rate of executing loco-manipulation tasks using HYPERmotion and the complexity the tasks. We conducted ten separate simulation experiments for tasks involving 3, 4. 5, and actions. The results show that as task complexity in- creases, the success rate of task completion gradually decreases, though the overall average success rate remains above 60%. Additionally, task success rates vary depending on the specific content of the tasks and the difficulty of the required robotic actions, with this variability being more pro- nounced in long- long-horizon tasks (those involving four more actions).
18

---

A2.3 Failure Cases Study
During the real-world experiment, failure of different cases happened in the process of the robotic lasks performing In Fig. 8, we recorded the average rate of failure caused by different type of errors. Planning errors primarily occur during the BT construction process by the LLM-Planner. When tasks are highly complex and require multiple estimations of the environment or object states, the task graph generated by the LLM may exhibit logical errors or missing actions, leading to task failure. To address this issue, optimized the motion primitives by integrating the robot's self- state detection environmental perception into the primitives to reduce the logical complexity of constructing BT.
Perception errors stem from inaccuracies in the robot's sensing system, particularly in tasks involv- ing navigation and target or object detection. The perception system consists of AprilTag localization and deep object pose estimation. While using an RGB camera for AprilTag localization, there is margin of error of 2-4 cm, especially when the robot is in motion. The accuracy of the object pose detection algorithm is also influenced by the object's orientation and potential occlusions, which can affect the robot's 's manipulation posture.
When performing robotic tasks in the real world, execution errors are the most common and can arise from a variety of factors. During task execution, since the actions are pre-trained and op- timized for full-body motion, unexpected disturbances can sigmificantly impact performance. For instance, while opening door, if the robot's leg collides with the doorframe, it may struggle to adapt its trajectory in real-time. In the case of long-horizon tasks, errors tend to accumulate with the increasing number of actions performed. For example, after pulling open drawer and retrieving an object, any deviation in the grasping pose can lead to failure in placing the object, causing it to collide with the environment and drop. Additionally, tasks with extended durations can result in overheating of the robot's actuators, further compromising task performance.
Applying LLMs to a high-DOF, floating-based humanoid robot, where manipulation must be co- ordinated with locomotion, is complex and challenging endeavor. Due to the low real-time per- formance of LLMs, they are unable to update plans in real-time during task execution and cannot directly generate control trajectories for multiple joints simultaneously. To address this, we consider the integration task graph and motion library, where the LLM is responsible for reasoning and docision-making constructing task graph composed of motion primitives. This task graph then issues control commands to enable the robot complete long-horizon tasks.
During the acquisition of motion primitives, the prioritization in the whole-body motion optimiza- tion phase significantly influences action generation. For instance, in actions like 'opening door' and 'open drawers', the optimization must not only account for the accuracy of the motion trajec- tory but also maintain robot balance and avoid self-collision. Additionally, to accommodate drawers of varying heights, we incorporate floating base adjustments in the optimization process.
When dealing with horg-horizon tasks, differences in pose solving can lead to issues with action continuity. For example, in the "Pick and place object" task, we introduced a homing action between the pick and place phases to ensure seamless transitions between actions to reduce the error rate.
A3 Details of Robot Learning
We utilize Proximal Policy Optimization (PPO) [6] for training our tasks, employing multi-layer perceptron within an actor-critic framework. The network architecture for the drawer opening, door opening, and dual-arm picking tasks consists of layers with [256, 128, 64] units while the picking task uses layers with [256. 128, 64] units. The activation function applied across all tasks is ELU. Below, we detail the observations, task-specific rewards (Ftask), and reward parameters for each task in Sec. A3.1 1 and the task evaluation in Sec. A3.2
19

---

A3.1
task design details
A3.1.1 drawer opening
First, we define the frame of the drawer handle. The x-axis of the handle points towards the robot, while the z-axis points upwards. The handle's inward direction is aligned negatively along the x- axis, and the upward direction is consistent with the z-axis. The task reward is defined as
Ttosk = arToroand + (drawer Toround + Ldrawer
(A1)
where Taround 0.5 when the gripper's top link is above the handle's 's position and the bottom link below the handle's s position, otherwise Taround = Idrawer represents the length by which the drawer has been pulled.
The observations and reward parameters for this task are listed in Tab, Al and A2.
2.0
0r2
0.0
normalized upper body joints position upper body joints velocity 0.1 drawer pulled length
0.5
044
7.5
7.5
vector from gripper to drawer handle
0kg
0.01
0.7
0.04
Table A3: observations of drawer opening task
Table A4: reward parameters of drawer opening task
A3.1.2 door opening
The door handle has the same frame as the drawer handle. The task reward is defined as
Ttask U'7 aro'aro + anglehandle Taround anglehandle + anglledor
(A2)
where Taround is the same setting as the drawer opening task and anglehandte represents the angle by which the door handle has been pushed. angle is the angle of the opened door.
The observations and reward parameters for this task are listed in Tab. A3 and A4.
01
2.0
base pose
012
0.0
right arm joints position
0fg
1.5
door handle pose
04
7.5
gripper pose
0
2.0
door handle angle
0k
0.01
door opened angle
077
0.125
0
0.02
Table A5: observations
Table A6; parameters
A3.1.3 single arm picking
We define the object's upward direction as aligning negatively along the x-axis, and the inward direction as aligning negatively along the z-axis. This orientation encourages the gripper to adopt top-to-bottom pose, facilitating proper grasp of the object. The task reward is defined
Ftask arTaround
(A3)
where Taround the same setting as the previous tasks with the corresponding object frame and if the object is been picked up, otherwise 0.
The observations and reward parameters for this task are listed in Tab. A5 and A6.
20

---

Q1
7.5
0.0
base pose
0
5.0
right arm joints position
2,5
object pose
05
7.5
gmpper pose
0.01
O07
0.7
3
0.1
Table A7: observations
Table A8: parameters
A3.1.4 dual arm picking
In the dual arm picking task, the distance di and d, represents the left end-effector and right end- effector to the left and right side of the object, respectively. The task reward is defined as
Itas h
(A4)
where h 1 if the object is been picked up, otherwise h
The observations and reward parameters for this task are listed in Tab. A7 and A8.
2.0
base pose
two arms joints position
0g2
2.0
0.0
object pose
left end-effector pose
0k4
0.0
7.5
right end-effector pose
0g
0.01
vector from object left side to left end-effector vector from object right side to right end-effector
0.0
0.0
Table A9: observations
Table A10: parameters
A3.2
task evaluation
We evaluate the learning results by applying the trained policy with 2000 rollouts for each task across four different position randomization scopes. We train and evaluate under the same randomization scope. Except for single picking task, which randomizes the position of the object, other tasks randomize the position of the robot. We summarize the success rate of each task in Tab. A9.
mean relative position of
domain of position randomization (m)
success rate
robot's base to goal's 's origin
(m)
drawer opening
-0.4, 0.65
[0.01, 0.10]
77.8% + 11.0%
door opening
1.0. 0.1,
[0.02, 1.0]
85.3% + 19.6%
single arm picking
(0.45, -.33 0.8
[0.2, 0.5]
88.5% + 6.9%
dual arm picking
[0.9, 0.0,
[0.02, 1.0]
73.8% + 15.7%
Table A11: success rate of each task under different position randomization scope
A4 Details of Whole-body Optimization
The trajectory optimization problem essentially constitutes Nonlinear Programming (NLP) chal- lenge characterized by predetermined quantity of nodes and intervals. Its canonical formulation
21

---

typically adheres to (q5)
(.n)))).(.))) _-' L()( s.t. x(t) f((()) 91 (x(t), t(t) 92 (x(x(t) ((), 00
(A5)
A4.1 Formulation
The standard formulation(AS) necessitates conversion into discrete programming format Subse- quently, we discrete the state and input variable as the follow sets, is the node number
X1
X'
(A6)
XN
uN
then the general optimization form (q,(A becomes (q.
J Σ Tx(x;; us)
(A7)
x; f (x,, u,) i 0, ...
Cmin < C(x;, u:) < Cmax 0, N
where, C(x;, u:) the discrete form of equality and inequality constrain, Cunin is the lower limit, Cmax is the upper limit.
A4.2 Dynamic Equation
In order to keep the motion critical dynamic feasible, construct robotic's motion equation as following the whole body dynamic:
Vi h(q. v.) ST.) ]
x M(q.)- (q.) (1)
where 1. v] R is the state number, variables, q. are the generalized coordinates and generalized velocities, f. E TR the vector of contact forces, E Rn the vector of actuated joint torques, M E gn, is the inertia matrix, the ny is the dimension of general velocity, h E Rnv the non-linear bias terms accounting for gravity, Coriolis and centrifugal torques, Je E I x is the contacts Jacobian, matrix Xn,, used to map actuated torques to the full vector of efforts.
fv Rh vectors of state and input
A4.3 Feasible Constrain
Specifically, in order to keep the trajectory physical feasible, we should shape the constrains as the following:
1) Initial state: In order to safely execute the trajectory, the robotic should start acting from a initial state which has stability and safety. The initial state constrains are constructed as the flowing:
qinit initial position
(A8)
initial velocity
(A9)
22

---

2) State limitation: Before we have got the joints trajectory from RL, which has a rough feasible, so it is neccessary to consider the physical limit state of robot. The state limitation constrains are constructed as the flowing:
Amia S
q
Amax position bounds Ak E [1,N 1]
(A10)
v
V velocity bounds
Ehe -1]
(A11)
S
vill mux
acceleration upper bounds
AK 00 - -1]
(A12)
min
V
v-
acceleration lower bounds
AH N0
(A13)
min
max
3) Contact Condition: As mentioned before, our robot has two strategies legged and wheel. So if just in the wheel motion, we should just keep the following constrain which keep the force reasonable:
> 0
(A14)
if the robotic motion model is legged, we should keep the robotic not slip just as the con- strain of friction cone (A15), so the contact condition should be (A14) and (A15):
(Eat (2)) 112 S pi
(ee) nz)
leg contact force bounds
vk E [0, -]
1] (A15)
where fe. = [f j"j fegl. j is the j-th leg contact force, j = 1,2,
4) Torque limitation:
The underlying foundation of robot motion action is depend on the ability of joints motor, so it is important to keep the torque in the capability limitation of motor, just as the following (A16) and (A17):
ST. = M + h(q;, v.) (
(A16)
Jo 0 Ak [o, T min S ST 7 max
Vk V [0
(A17)
where To is the virtual float joints keep zero, J the actuated joints torque, min the joints lower limitation, T max is the joints upper limitation.
A4.4 Cost
At the end of programming, its function the whole body trajectory is to realize the motion learned from RL framework, we implement the cost as
L, u.) = |ل 11 + ||u/||
(A18)
the term || 1 is for merging the gap between RI. trajectory and actually feasible trajectroy, q is the upper body trajectory from RL, q is the upper body trajectory from whole body optimiza- tion, ||ul| for reduce the energy of the whole motion.
A5 Motion Library
We constructed a motion library to house the learned whole-body skills as well as the action and condition nodes used construct the task graph. The motion library includes information about the skills fed the LLM, well as the control code corresponding to each skill. The following Fig. A2, A3 shows the action skills and nodes inside the motion library that LLM choose to invoke to construct the task graph.
23

---

ABB Action Node Bв
cHomingPose>: "nane'- hereil-hhoinnggonn ; 'type' "general, 'label'wstarn the robot initial position; description'= =control the robot to power up and back to the initial robot pose.
<Findbjjects: "nane" find object'; 'type "general; 'label'-look around for object; descrrptionn scontrol the robot to turn on the head camera, and rotates itself to find "object" and acquire its 30 position.
<MoveTarget>: "name"=' navve="nove_it ; 'type' wheel; 'label' -approach to target with wheels; "deseription' :cootoon the robot to approach to the target location using wheel motion (require knowing 30 position of 'target").
<WalkTarget>: 'name'= "ame''"walk_targee ''type'= 'label' happroach to target with legs; "description' scontrol the rabot approach to the target location using motion (require knowing pesition 'target').
<ObjectDetect::
"nane'='object"" jeejecttdeeett;; sggeraagee;:;; "label==bbjf" detection and est estination; estinate the position pose
"description' -using 'target_on ceeett'bect'.
canera
<ObjectPlace>: "'abject_paace' "type" ageneral; 'label' 'label'splace object target position; description' control the robot to put the object to target position. (require knowing 'target" 3D position).
<OpenDoors: 'name'= "ooee'"'doooooo door_open'; 'type" -general; 'label' -ppen the door; "description" -control the robot to open the door, (require knowing 30 position of "door').
<SinglePicho: "nane" mamee"=ssings "sinnsse__nna_ppiii 'description -control the robot to grasp the target object with right arm, and pick it up (require knowing ttaret_obffcct position pose).
"type'=single. 'label' sgrasp object and pick it up;
<DualPicko: 'name" "-' chullarrn_icch 'type' aape'-dual_rge _arm; "label'=holl object with dual ares and pick it "deccription' scontrol robot hold the target object with dual arns, pick it (require knowing "target_objec"" position and pose).
<openDrawers: "name" "amee'"'pen_ddee open_drawer'; 'type "=eneral;; 'label'-open drawer. 'description insscription=<contree -contral the robot open the drawer. (require knowing 30 position 'drawer').
Figure A3: Action nodes in the motion library, where the blue nodes are based on learned whole- body motion skills.
### Condition Node #A#
<Distance>: 'name" ' nnbjett' rabjettinn_reach ji__r_errrr;; 'type' 'label'sis object in reach; ddescription' -measure the distance from the ooject the robot. 1f it is larger than 80cm then return fail. (require knowing 'object' position)
<uhettherSingeeAm:: 'name" "hame'='wheree" hiteesshhhheesssnee__rrr based nanipulation task; "decription'
'sblect robotic morphology reason whether single to manipulate object, can be used to make decisions before picking actions.
WWhettherkheelHooe::: 'nwhether wheel move'; "genea=g: based on locemotion task; 'description' VLN to reason whether to use wheel or leg to move, can be used to make decisions before locanation actions.
robotic morphology
<IsActionSuccess:: "name'='1s - asttinnttoon___ cocioo__cmmietedd:
"reeson about the success ot the action; 'description'-ape -apply VLM to reason whether the previous manipulation action is successful, if repeat the action time in hehavior
Figure A4: Condition nodes with different functions in the motion library.

---

A6
Motion Morphology Selection
In this section, we show the task scenarios used for the motion morphology selection experiments.
A6.1 Manipulation Scenarios
For the robot manipulation morphology selection experiments included six simulated and four real- world scenarios. We conducted ten morphology selections for each scenario, and before each trial, the positions and poses of the objects in the scenarios reset. We applied the same prompts for all manipulation morphology selections, with the instructions for each scenario shown in Fig. A5.
A6.2 Locomotion Scenarios
The robot locomotion morphology selection experiments included six simulated and four real-world scenarios, as shown in Fig.A6. We conducted ten morphology selections for each scenario, and before each trial, the positions of the robot and obstacles in the scenarios were reset. We applied the same prompts for all locomotion morphology selections.
25

---

Task: Pick up the </drill>
Task: Pick up the </containe>>
Task: Pick up the </bag>
Task: Pick up the </drill>
Task: Pick up the </bin>
Task: Pick up the </ball>
Task: Pick the </cracker>
Task: Pick the </soup>
Task: Pick up </tool box>
Task: Pick up </sugar>
Figure A5: Task scenarios for manipulation morphology selection experiments.
26

---

1. Construuctio Site
Road
Parkinn
BrBridge
5.Slope
6.Space Station
7. Laboratorry
8. Wooden Board
9. Foam
10. Litter
Figure A6: Task scenarios for locomotion morphology selection experiments.
27

---

A6.3 VLM Prompts
The prompt words used for the motion morphology selector shown in the figures, where the prompt words for manipulation morphology selector will be fed into the VLM along with the ceived textual task instructions from the Behavior Tree.
The motion morphology selector are packaged as one of the functions in "User Input' module and must be enable in 'Function
turned "off" by default. When it needs to be invoked in task planning, it Options' or specified be to 'on' when inputting the task instructions.
"Suppose you are a humanoid robot and you have two arms, the right arm has a claw gripper as the end-effector."
"You have two ways to manipulate object: single arm manipulation dual arm manipulation."
camera on your head that see object "And you can choose the manipulation method based the object and task instruction reasoning."
environment." "
"When you are doing single arm manipulation, your claw jaw gripper can open up to 10cm." "You only have ability grasping methods"
the jaws to pick up object, regardless of other skillful of the jaw' opening and closing,"
"meaning that you cannot single grasping if the size the object exceeds the size
"or 1f the object current pose is capable for grasping with hand."
"For example: if the given task is 'pick up the drill', and the image shows that a drill is on the desk."
"Then you will choose to use single arm manipulation, because the size the drill in the image can be manipulated with single arm"
"Now you receive the image from the camera the task below, please answer whether use 'single arm' or 'dual arm' to do the manipulation." "(Please answer with only 'single' or 'dual')"
Figure A7: Prompts used for Manipulation Morphology Selection.
"Suppase you are robot and you have two ways to move: legs and wheels. "You have depth camera that can obtained the 2D image and point cloud of the road in front you."
"Now you have to pass the road in front you and here is the 2D image of the road, and the down sampled point cloud
"Please choose whether the road should be passed with legs or wheels."
(Note that wheels are used when the road ahead is flat, or a slope, or the maximum height of the obstacles is lower than 5 cm."
"And legs should be used when there are obstacles or wooden planks 'maximum height higher than 5 cm² "Determine which type of movement the robot should use to pass through the roadway.
(Please answer with only 'leg' or 'wheel', and the data below is the point cloud)"
Figure A8: Prompts used for Locomotion Morphology Selection.
28

---

A7
User Input
The "User Input' is the module that links the instructor the language model and contains prede- fined prompts for initializing the language system environment and limiting the model output, as well as an interface for accepting task commands sent from the user side.
A7.1 Basic Prompts
Basic prompts provide description of the task context and robot characteristics, as well as an explanation of user commands and output formatting requirements. As shown below:
###Basic Prompts#s#
"You are now a robot controller, please output a XML file for constructing behavior tree to control the robot under the requirements and given task."
"The robot you control is centaur like robot, with humanoid upper body and four legs, each leg has wheel at the bottom." "The robot has two arms, with a claw gripper on the right are. It can manipulate objects with two ways: single-are manipulation and dual manipulation. "The robot has two modes of movement: wheel motion and leg motion. The robot default manipulation and locomotion modes are 'single arm' and 'wheel'."
"The robot has two depth cameras: one located the head to view objects, and
the vaist view the road and terrain ahead."
A7.2 Function Options
We designed number of functions for the robot and packaged them into condition nodes for selec- tive invocation by the LLM during the planning of the task. These functions include: 'Manipulation Morphology Selector', "Locomotion Morphology Selector', 'Failure Detection and Recovery'. We add the descriptions these functions acting as 'Function Options' inside the 'User Input', and set all functions to 'off" state by default. When the instructor expects a function be manually set to 'on' include declaration use the function in the
added during
task planning, instruction.
### Function Options ###
"The robot has the following functions, all of which are offf' by default."
"When function 'on', need to be involved planning the given task, when it 'off', should not be used. "Functions:
"1. P mmaipulattion selector':: this function allows the robot to add the condition node <WhetherSinglotm to the planning BehaviorTree, which is used to determine whether the current manipulation task should use the 'single_arm' arm or 'dual_arm" type of action."
"2. 'Locomotion_mode mode_selector"' this function allows the robot to the condition node <WhetherWheelMove' to the planning of the behavior tree, which is used to deterzine whether the current locomotion task should the 'wheel'
'leg' type action."
"3. "detection_ this allows the robot to add the condition node <IsActionSuccess>,, which is used to determine whether the previous action has been successfully completed and, if not, to employ recovery zechaniam that repeat the action."

---

A7.3
User Interface
The user interface is responsible for accepting task commands from the instructor and combining them with pre-defined prompt for imput to the LLM. The complete user input is as follows.
User Interface: http:///ww.ootooooooooo..oooa/rp
ini
Motion hilpsyyw.rraattu........../////r ini
Basic Prompts: Function Options: byt.:otww..gggtt....//ooo//o/oot ini
A8 Task Planning with LLM
After receiving the prompts from "User Input", the LLM output hierarchical task graph that con- tains a series of nodes and actions for accomplishing the task. The task graph is saved in xml file and serves as framework for constructing the Behavior Tree that guides the robot's actions. Below we show the detail experiments in 'Tasks with human instructions' part of Sec. 4.3. For each task, we present the task graph generated by LLM,, and the Behavior Tree constructed from it.
Input: Open the drawer and pick up the drill.
Forrat="""
<lehaviorTree I0="MuinTree's
<Sequence nont="root">
<FindObject find_object' goale" (Ormweroosition)
<Fallbacks
<bistance "object
(DrawerPosition)"
<hoveTarget namez' noveetarget"" ressages" (DroerFossitonn) />
</fallbacko
<OpenDrawer oeoooopendaar spen_drawer"/>
<objectDetect names object,
object_detectt "detect" (orl="(=-ilPosee
<SinglPPick single_are
(Dri1Posee'
</Sequence>
</BehaviorTree>
Figure A9: Task planning of "Open drawer and pick object'.
Input:
Find the door and open it.
<root STCP9_ formate'4">
"AainTree"">
<Sequence n
<FindDbject object" goal=" (DoorPositionn)
<Fallbacko
"aneettbject_iii in_reech" nessage= (DoorPosition)
<toveTarget nanc- "nove_ target" messago= (DoorFosition)"
<dpenDoor nance' susor_.peen//
</Sequence>
</BehaviorTree>
Figure A10: Task planning of Approaah and open door".
30

---

Input: Pick up the cracker and put it into the box.
kroot SтCcP formate "
céchaviorTree I0= "oainTre""
ront"roott">
<Findokject fine_object" (enctteerrrettttttiin
cfallbacts
cBistance nasset reach""" sissage" (CrackerPesittin)) />
cNoveTarget nasee weoee target" asssages (crackerPesition) />
</Fallback
<SinglePick taweleeng baangge__mmppiss aessage* "(rackerkosttion)"
<findübject
sbjectt (alaa(taaasitioon)) >>
cFallbacks
<Bistance nasee "objectt_.iin.co Increach aessagee (Borositioss">>
(Boxhosition)"
</Falllacko
<lrjectPlace nares pobject_pacee"" aessages (Bonkssttoos
</Sequence>
</reot>
Figure A11: Task planning of PPick and place".
Input: Pick up the box and put it the table.
"'amipulation_ndd_eeeee selector =on)
<reat BrOW fowate"2's
dieaavtorrree E0 "inTaatnree''s
<sequence name" rect">
(rinsteecct narce "fineteeeet goale (tesPoottoon) ft
trallback
<distance namas saject. Jrunead" eussages "BooPosttinn"" tb
cipveTarget
target" massagen" (BosPusition)
</rlllscck
cijjectreect wbject detecct" gool* (souros))
chingletick
(howhose)
</Sequances
claiPPckk
pick" eassage"" (BesPese)"A
</hellback
<Findte/jct natce Fllrd objectt goalle (talerrtioon)))) /
<Fa3Ibacko
<odattnce namen oaject In, meach" massagen" Cewuagge""Tabbeeeee (Tablaaoosttion)">>
<PoveTanget nance Theve Sorget" eessagee" (tibggee"(tllloeeeeeeetoo (1abbcooosettrrrsstttsee
</ralliscke
ciedectPlace fobject, place" aeseage, "tanoetttaaaloottttisaa""
</Sequnces
1/Behedorrree>
Figure A12: Task planning of Dual-arm arm pick place'.
31

---

References
[1] A. Laurenzi, D. Antonucci, N. G. Tsagarakis, and L. Muratore. The xbot2 real-time middleware for robotics, Robotics and Autonomous Systems, 163:104379, 2023.
A. Laurenzi, E. M. Hoffman, L. Muratore, and N. G. Tsagarakis. Cartesi/o: A based real- time capable cartesian control framework. In 2019 International Conference Robotics and Automation (ICRA), pages 591-596. IEFE, 2019.
[3] E. Olson, Apriltag: A robust and flexible visual fiducial system. In 2011 IEEE international conference on robotics and automation, pages 3400-3407. IEEE, 2011.
[4] A. De Luca, L. Muratore, and N. G. Tsagarakis. Autonomous navigation with online replanning and recovery behaviors for wheeled-legged robots using behavior trees. IEEE Robotics and Automation Letters, 8(10):6803 6810, 2023. 16011011001LRA223333115555
[5] I. Dadiotis, A. Laurenzi, and N. Tsagarakis. Whole-body mpe for highly redundant legged manipulators: Experimental evaluation with 37 dof dual-arm quadruped. In 2023 IEEE-RAS 22nd International Conference on Humanoid Robots (Humanoids), pages 1-8, 2023. doi: 009//ummanidd51102223...33333
[6] J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. Proximal policy optimization algorithms, 2017.
32


Reviews:
[{"review_id": "G0XI0tlgh5", "reviewer": "Reviewer_Ec5Y", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper tackles the loco-manipulation problem for a dual-arm quadruped robot. Specifically, the paper focuses on an LLM-based pipeline with various components to complete loco-manipulation tasks such as articulated object manipulation, grasping, and traversing different terrains, where low-level skills are trained with RL and provided in a skill library. Demonstrations are shown on real-robot hardware.", "summary_of_recommendation": "While the paper provides impressive real-world demonstrations, it appears that it lacks a central focus to provide insightful contributions to the community. Additionally, given the large scope of the claimed framework, the evaluations and their details appear to be lacking as well. Therefore, I’m withholding a higher recommendation.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See weakness section above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Wne73RKGTa", "reviewer": "Reviewer_7D92", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "In this paper the authors present an fully integrated system demonstration that autonomous execution of long horizon tasks from human language commands on a platform capable of locomotion and manipulation.", "summary_of_recommendation": "I would recommend a weak accept since this work is valuable for the research community but its contribution lies in system integration rather than the development of novel techniques.", "technical_quality": 4, "clarity_of_presentation": 3, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "The authors could further improve the clarity of the paper by concisely stating and pinpointing the contribution of the manuscript in the introduction section.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "57GK5qhiFq", "reviewer": "Reviewer_3ACE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 2, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents a framework for enabling humanoid robots to learn, select, and plan behaviors for loco-manipulation tasks. The approach combines RL for generating a motion primitive library, LLM for high-level task planning, and VLM for selecting appropriate robot morphologies. The authors demonstrate the system on a centaur-like humanoid robot in both simulated and real-world environments.", "summary_of_recommendation": "The proposed approach shows promise in zero-shot generalization to new tasks. While it represents a step forward in autonomous loco-manipulation, it lacks comprehensive quantitative comparisons and detailed technical explanations. The system's complexity raises concerns about usability and reproducibility. Despite these weaknesses, the integration of language models with robotic control is timely and potentially impactful. I recommend a weak accept, contingent on addressing these issues, particularly by providing more rigorous evaluation and critical analysis of limitations.", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 2, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "1. Can you provide more quantitative comparisons to baseline methods, e.g. planning without language models using a custom decision tree? It is hard to grasp the complexity of the domain for the limited text description of the tasks and how much role do LLMs and VLMs actually play.\n\n2. Can you elaborate on failure cases or limitations observed in your experiments? Are there particular types of tasks or scenarios where the system struggles?\n\n3. Can you provide more details on the whole-body motion optimization process? How do you ensure dynamic feasibility?\n\n4. How did you approach the question of what primitives to include in the learning library and how easy it is to acquire a specific primitive?\n\n5. Can you give a comment on what component of the system could be: combined, remove, added? and why?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper has addressed my concerns. I encourage the authors to follow Ec5Y's advice and frame it as a systems paper. Overall, I found the paper interesting, and showcases an interesting application of VLM and LLM for humanoid control in the real world.


==============================

Strengths:
- Impressive system that showcases what integration of VLM, RL, and VLM can achieve
- Real-world experiment is appreciated.

Weaknesses:
- It lacks a scientific contribution and the paper reads as a tech report that tells the reader how the system can be integrated.

The authors are encouraged to pinpoint the technical challenges they faced and how they solved those challenges.
Decision: Accept