Harmon: Whole-Body Motion Generation of Humanoid Robots from Language Descriptions
Authors: Zhenyu Jiang
Yuqi Xie
Jinhan Li
Ye Yuan
Yifeng Zhu
Yuke Zhu
Venue: CoRL 2024 (2024)
Keywords: Humanoid Robot
Whole-Body Motion Generation
TL;DR: Humanoid motion generation with human motion prior and VLM-based motion refinement.

Abstract: Humanoid robots, with their human-like embodiment, have the potential to integrate seamlessly into human environments. Critical to their coexistence and cooperation with humans is the ability to understand natural language communications and exhibit human-like behaviors. This work focuses on generating diverse whole-body motions for humanoid robots from language descriptions. We leverage human motion priors from extensive human motion datasets to initialize humanoid motions and employ the commonsense reasoning capabilities of Vision Language Models (VLMs) to edit and refine these motions. Our approach demonstrates the capability to produce natural, expressive, and text-aligned humanoid motions, validated through both simulated and real-world experiments. More videos can be found on our website https://ut-austin-rpl.github.io/Harmon/.

HARMON: Whole-Body Motion Generation of
Humanoid Robots from Language Descriptions
Zhenyu Jiang!
Yuqi Xie
Jinhan Lift
Ye Yuan²
Yifeng Zhu
Yuke Zhu
'The University of Texas at Austin
NVIDIA Research
{zhenyu, yukez] utexas edu
Play Rock Paper
Sign language: am
Point top right
Shake right hand
danbing
Give thumb ap in from thes!
forward, stop and give high-five, than continue walk
rock-n-rolll sign
wave to greet
Figure 1:
We generate diverse whole-body humanoid motions from free-form language descriptions exe- cute these motions on the real humanoid robot.
Humanoid robots, with their human embodiment, have the poten- tial to integrate seamlessly into human environments. Critical to their coexistence and cooperation with humans is the ability to understand natural language com- munications and exhibit human-like behaviors. This work focuses on generating diverse whole-body motions for humanoid robots from language descriptions. We leverage human motion priors from extensive human motion datasets to initialize humanoid motions and employ the commonsense reasoning capabilities of Vi- sion Language Models (VLMs) to edit and refine these motions. Our approach demonstrates the capability to produce natural, expressive, and text-aligned hu- manoid motions, validated through both simulated and real-world experiments. More videos be found on our website https /tt///uutuutiinn gi thub. io/Harmon/.
Abstract:
Keywords: Humanoid Robot, Whole-Body Motion Generation
Introduction
Humanoid robots have great potential for seamlessly integrating into the human world due to their human-like physique. We envision these robots operating centn-eentered environments and coexisting with people in shared physical spaces. To deploy humanoid robots work and live with
This work was done while Jinhan Li was visiting rescarcher at UT Austin.
Equal contribution
8th Conference Robot Learning (CoRL. 2024), Munich, Germany.

---

us, the robots should possess the capability to understand natural language instructions akin to our everyday conversations. Furthermore, the ability to exhibit human-like behaviors will render these robots more social and predictable, thus enhancing effective and safe collaborations between humans and robots. To this end, this work focuses on developing a method for generating diverse behaviors of humanoid robots from text descriptions. Our approach takes natural language description of the desired motions as input and produces corresponding whole-body motions.
Training a model to generate natural humanoid motions from language descriptions is challenging due to the lack of large-scale datasets of language-motion pairs. Fortunately, the narrow embodiment gaps between humans and humanoid robots unlock vast amounts of human-centered data sources for model training. Particularly, there an abundance of large-scale human motion data [1-5]. These data, paired with language descriptions, capture wide range of human motions. These human mo- tions can be mapped to humanoid robots informed priors for motion generation. Nonetheless, discrepancies between the human model and the humanoid hinder the effectiveness of direct mo- tion retargeting, First, head and finger motions are typically absent from the human motion dataset, which consequently cannot produce humanoid motions for these body parts. Yet, these motions are critical for the expressiveness of the robot's 's whole-body motion. Further, retargeting cannot pre- cisely replicate human motion on humanoids due to their kinematic constraints, potentially changing the semantic meaning and legibility of the motion.
In this work, we introduce HARMON (Humanoid Robot Motion Generation), which generates whole-body humanoid motions from free-form language descriptions by incorporating human mo- tion priors. We employ PhysDiff [6]. a diffusion-based generative model trained on large-scale human motion datasets to generate human motions from language descriptions. PhysDiff' incor- porates physics constraints during the diffusion process and generates physically plausible human motions. It generates human motion sequence of SMPL [7] parameters. We then retarget the human motion simulated humanoid robot using inverse kinematics. As mentioned earlier, the generated motion from PhysDiff does not involve hand and finger motions, and retargeting errors can lead to the misalignment between the motion and the language description. To address these issues, we leverage the commonsense reasoning capability of Vision Language Models ( (VLMs) to edit the humanoid motion. Given a rendering of the humanoid motion and its language description, the VLM generates head and finger motions and refines the arm motion. To realize the whole-body motion on the real robot, we separate the upper- and lower-body motions and control locomotion and upper-body motion independently. Benefiting from the human motion prior and VLM-based motion editing, we can generate natural humanoid motions that align with the language description and execute the motions on simulated and real humanoid robots.
We use a Fourier GRI humanoid robot for the simulation and real-world experiments. We curate test set with texts from the HumanML3D test set and LLM-ggnerated motion descriptions and conduct human study evaluate the quality of the generated motion. HARMON demonstrates natural and text-aligned humanoid motions and is favored by human evaluators on 86.7% of test cases. Furthermore, we execute the generated motions on the physical robot and illustrate diverse and expressive humanoid motions in the real world.
2 Related Work
Humanoid Motion Control. Humanoid motion control has been widely studied in the computer graphics community, where human avatars are simulated to demonstrate diverse and physically realistic behaviors [8-24]. These methods can demonstrate physically realistic motions on the humanoid avatar by imitating human motions with reinforcement learning. However, deploying them on the real robot is difficult since they use an unrealistic humanoid model (SMPL robot with 23 ball joints and no torque limit). Recent studies [25-27] have explored the imitation of human motions on real and simulated humanoid robots. These approaches involve retargeting human motion humanoid robots and using reinforcement learning (RL) to train the robots in simulation. Some of these studies [26. 27] have successfully deployed the trained policies to real- wealdworld robots. Our work focuses on text-conditional humanoid motion generation. Yoshida et al. [28] also address

---

this problem by directly generating humanoid motion using a Large Language Model. However, their generated motions often appear unnatural and rigid. In comparison, we incorporate human motion priors to produce more natural humanoid motions.
Human Motion Generation. Recently, significant developments have occurred in human mo- tion generation. Early studies [29-36] generated human motion deterministically. Subsequently, stochastic generative models were applied to human motion generation, employing GANs [37] or VAEs [38-41] to create human motions from various conditions, such as action labels and texts. More recently, diffusion models [42, 6, 43] have also been utilized for human motion generation. These models can accept various conditions such as text or keyframes and produce diverse and nat- ural human motions. Additionally, some recent works [44, 45] tokenize human motions and use transformer-based autoregressive models to generate human motion through next-token prediction.
In our work, we use human motion generated from text as an initialization for humanoid motion. While these text-conditioned human motion generation models provide good prior for humanoid motion, the structural differences between the human models in these frameworks and the real hu- manoid necessitate further refinement of the humanoid motion.
Foundation Models for Robot Control. Foundation models, such Large Language Models (LLMs) and Vision-Language Models (VLMs), have shown exceptional performance as high-level semantic planners for tasks involving embodied agents and robotics [46-55]. Recent studies [56- 58] have started to explore their potential for learning low-level robot behaviors. For example, L2R [58] uses few-shot examples to prompt LLMs to generate reward functions for robot training, while Eureka [56] eliminates the need for few-shot examples by employing an evolutionary search system iteratively propose improved reward functions. RL-VLM-F [59] generates reward functions for agents to learn new tasks with feedback from VLMs. We leverage the zero-shot commonsense reasoning capabilities of VLMs to evaluate and edit humanoid motions, improving the alignment between humanoid motions and corresponding language descriptions.
3 Method
We study the problem of generating humanoid motions from language descriptions. From a text description X, we aim to generate a sequence of robot joint configurations Q = .q ... qr q E RRC Here, T is the sequence length, and e the number of joints of the humanoid robot.
Fig. 2 depicts our proposed method, HARMON. Firstly, generate human motion based on the language description and retarget this human motion to create the initial humanoid motion (Sec. 3. To improve the alignment between the humanoid motion and the language description, we employ VLM to generate finger and head motions and iteratively adjust the body motion (Sec. 3.2). Finally, we execute the generated motions on the real humanoid robot (Sec. 3.3).
3.1 Retargeting Text-Conditioned Human Motion
Training a model to generate humanoid motions from language descriptions directly challenging due the absence of paired dataset. We utilize human motion generation model to generate human motions from language descriptions, which then retarget to the humanoid robot.
Given the text description X, we use PhysDiff [6], a physics-guided motion diffusion model, to generate the corresponding human motion. The output is a sequence of SMPL [7] parameters P = {(0,, tt),,t), (0r, t,)), where f, is the joint rotations and t; is the root translation time step 10 encoding attributes such as height and size. Given 0,, t,, the positions of each human joint J; S(O,, L, B) computed using the SMPL model S. Each Jj 24x3 contains the positions of 24 human joints time step i.
The SMPL model also includes body shape parameter
Before retargeting human motion to humanoids, we first minimize the disparity between human body shape and humanoids to ensure the robots can reach human joint positions. Inspired by He et al. [27], we set the SMPL model and humanoid model the same T pose, select 17 corresponding joint pairs on both models, and minimize the joint position differences. We employ the Adam optimizer [60] to minimize the joint position difference by optimizing B. The optimized 5 is subsequently used to compute joint positions from the SMPL parameters.
3

---

Human Motion Generation
IK-Based Motion Retargeting
IK Target
Motion
Humanoid Motion
Description
"Seratch the head
Human Motion Generation
IK-based Mation Retargating
VLM-Based Editing
with the right
VLM-Based Ecitiing
Finger
Description
Rendering
Rendering
VLM Judge
Rendering
Finger Motion Generation
"Hand not on head"
Head Motion Generation
VLM Motion Adjust
"Move hand to head"
Head
Motion
Description
Description
Iterstive Motion Adjustment
Figure 2: Overview HARMON. Given the language description of motion, first generate corresponding human motion and retarget the humanoid using inverse kinematics. Next, we utilize VLM refine the humanoid motion. This process involves extracting finger and head motion descriptions from the initial language description and generating the corresponding motions using the VLM. Given the rendered humanoid motion, the VLM iteratively evaluates and adjusts the motion to ensure alignment with the language description. Finally, HARMON generates whole-hody humanoid motion that accurately aligns with the language description.
Next, we utilize the inverse kinematies (IK) solver from pink [61] to align the key joints of the hu- manoid with those of the human model by optimizing the humanoid's joint configuration q. The key joints include the wrists, elbows, shoulders, knees, and ankles. Given the current joint configuration of the humanoid and the target positions of the key joints, the solver calculates the joint velocities to drive the key joints to their target positions. We sequentially set the target as the spatial position of the SMPL model and update the joint configuration with the results from the IK solver at each time step. After iterating through the SMPL parameters for each time step, we obtain a sequence robot joint configurations, Qr, from the retargeting process. Controlling the robot to follow this sequence results in humanoid motion that closely mimics the generated human motion.
3.2 VLM-Based Humanoid Motion Editing
The retargeting process initializes humanoid motion based on the generated human motion. How- due to the differences in the kinematic structures between the SMPL human model and the hu- manoid, the retargeted motion might not fully align with the intended language description. Specifi- cally, the humanoid can actuate the neck and dexterous hands, whereas the generated human motion lacks head and finger movements. Consequently, the retargeted motion does not fully exploit the humanoid's 's potential for expressive motion. Additionally, the retargeting process cannot precisely replicate human motion on the humanoid, potentially resulting humanoid motion with a dif- ferent semantic meaning from the original human motion. These factors can cause misalignment between the humanoid motion and the language description. To address this issue, render the retargeted motion into videos and utilize GPT-4 to edit the humanoid motion for better alignment.

---

Finger Motion Generation
Head Motion Generation
Rendered Humanoid Motion
Finger Motion Description
GPT-4 Ceneration
Finger Motion
Head Motion Description
GPT-4
Generation
Motion
"Left indes finger struight while other fingers are bent."
"frmme: [1.2]
"Look left downsaadd Yotal
Index":
"yew": ":
"L "Lother": 80
frames.
"pitch":
VLM-Based Editing
Iterative Adjustment
Humancid Motion with Head&Finger
Motion Description
GPT-4 Judgement
GPT-4 Adjustment
IK- Kk-Based Editing
"reasoning": "asssoniin"""""he "The robot points
straight aleed."
"trome": 1
downoorrd a 45-degree angle loft index finger.
"sucooss" faise
"eft_wrist":
"suggestion": "The robot should
"move down;
move arm downward 4B dogiee."
mave forward"
wrist":
Refined
"no_ change"
Humanoid Motion
Figure 3: VLM-based motion editing. Top left: GPT-4 generates finger motions at keyframes based the rendered humanoid motions and the finger motion description. Top right: GPT-4 identifies keyframes and generates head motions from the head motion description. Bottom: GPT-4 itcratively adjusts arm motion by evaluating and refining the rendered humanoid frames based on the motion description.
Finger and Head Motion Generation. To maximize the expressiveness of the humanoid motion, essential generate finger and head movements and integrate them with the whole-body motion from the retargeting process. As illustrated in Fig. 3, the finger and head motions are generated separately, each with its own motion description. We employ GPT-4 to extract these specific de- scriptions from the original motion description. This approach allows the VLM/LLM to concentrate on generating precise finger and head movements while avoiding unnecessary motion generation when these parts are not involved.
GPT-4 The finger motion needs to be coordinated with the motion. To achieve this, to observe the rendered video of the retargeted whole-body motion and generate the corresponding finger movements. Since GPT-4 only accepts images as input, we sample four frames at equal inter- vals from the video. Although more frames might capture additional details, we found empirically that the existing VLMs exhibit reduced reasoning ability when input sequences are too long. For our task, four frames are sufficient for the VLM to generate the finger motion accurately. We generate
put these four frames along with the finger motion description into GPT-4, prompting hand joint configurations q. E R for each interval, where nz 12 represents the total number of finger joints. The results for different intervals are then concatenated into sequence of finger joint configurations, Q2-
The head motion is more independent and has lower dimensionality than the finger motion. There- fore, we use an LLM, specifically GPT-4, to directly generate head movements from the head motion description. We provide GPT-4 with the head motion description, the total number of frames, and the frames per second (FPS) input. GPT-4 then determines the joint configurations q E R for the three neck joints keyframes. By interpolating between these keyframes, we obtain smooth head movements. represented by the sequence of joint configurations Qn. The key frame indices are determined autonomously by GPT-4, allowing for the generation of high-frequency head motions.
Iterative Motion Adjustment. The retargeted motion may not align with the language description for primary reasons: 1) the generated human motion may accurately reflect the language description, and 2) the retargeting process may alter the human motion, resulting in humanoid mo- tion with different semantic meaning. To address these discrepancies, we implement an iterative motion adjustment schema to align the humanoid motion with the language description.
As shown in the bottom row of Fig. 3, we employ judgment agent and an adjustment agent to ensure alignment. Similar the process for generating finger motion, select four frames from the generated motion video at equal intervals. These frames, along with the human motion description, provided to the judgment agent. GPT-4V first generates caption describing the actions of the
5

---

humanoid in the video. Then, it assesses whether the actions match the motion description and provides suggestions for improvement. In the next step, the same four screenshots and the generated suggestions are input into the adjustment agent. The adjustment agent then predicts the necessary adjustments to align the motion with the provided suggestions.
The key effective motion edits is providing intuitive interface for the adjustment agent. Di- rectly allowing the VLM to edit the joint configurations of retargeted motion Q, is challenging due to the non non-intuitive mapping between joint configurations and motion. Additionally, editing lower body motion is not particularly meaningful for this project, as the lower body is controlled by a separate controller the real robot experiment. Therefore, our focus is on the upper body motion, which demonstrates richer semanties and is easily controllable on the real robot. We design a set of control primitives that move the left and right wrists in specific directions using inverse kinematics, such moving up/down towards the head/chest. Details about these primitives the appendix. The adjustment agent can combine these primitives to create motion adjustments. We then apply these adjustments, render a new video, and return to the judgment agent to start new evaluation round. This process is repeated until the judgment agent confirms that the motion aligns with the language description or the adjustment process exceeds two rounds. Since our control prim- itives are limited to spatial wrist movements, we include an additional step in the judgment agent determine if the current motion can be improved by the editing process based on the language description. If improvement unlikely, the process is skipped.
provided in
If any adjustments are made, we use the body joint configuration sequence Qs resulting from the final round of editing. no edits are necessary, we directly use the retargeted motion Q, as the final body joint configuration sequence Qb. Finally, we combine Qo. Q,, and Qn. to form the complete body joint configuration sequence Q"
3.3 Motion Execution on the Real Robot
Directly executing the whole body joint configuration sequence Q* the real humanoid is infeasi- ble because the kinematic motion does not account for the robot's dynamics and balance. Therefore, following Cheng et al. [26], we separate the lower- and upper-body motion in the real robot exper- iment. We simplify the lower body motion into locomotion commands and utilize a Zero Moment Point (ZMP)-based [62] controller for locomotion. These locomotion commands are extracted from the trajectory of the humanoid robot's pelvis projected onto the ground plane. Simultaneously, we execute the upper-body motion from the generated joint configuration sequence Q" on the real robot using joint position control. By separating locomotion and upper-body motion, we successfully execute our generated motion on the real robot.
4 Experiments
4.1
Evaluation Setup and Baselines
Evaluating whole-body humanoid motion generation is challenging due to the absence of datasets containing paired language and humanoid motion data. We conduct a human study to assess our generated motions. We curate test set of approximately 50 language descriptions of motions. The first part of this test set comprises texts randomly sampled from the HumanML3D [1] test set, focusing primarily on body motions without involving head and finger movements. To evaluate whole-hody humanoid motions, we use GPT-4 to generate descriptions that include head and finger motions, forming the second part of the test set.
We generate humanoid motions from these descriptions using HARMON and compare them against three baselines:
VLM-based Motion Generation. Inspired by Yoshida et al. [28], this baseline generates humanoid motions directly using Large Language Model (LLM). In this approach, we exclude the human motion prior from HARMON and initiate VI.M-bssed motion editing from a static SMPI. T-pose. Since VLM-based editing only modifies upper-body joints, we incorporate the lower-body motion from HARMON for fair comparison. This allows us to evaluate the significance of the human motion priors.

---

Overall Text-Motion Aligmment
Body
Text-Motion Alignment
Normalized Score %
70
80
68.5
64.4
81.2
72.4
80.7
76.5
79.4
78.5
83.3
51.9
54.8
48.9
50
38.3
36.1
40
20
14.3
0
Total
Finger& Head Motion
Arm Motion
Body Coordination
VLM-based Motion Generation
Human Motion Retargeting
Harmon w/o Finger or Head
Harmon (Ours)
Quantitative results of human study. A higher normalized score indicates better alignment between the humanoid motion and the language description.
Figure 4:
Human Motion Retargeting. This baseline uses the retargeted human motion directly as the body motion. To ensure a fair comparison with HARMON, we add the finger and head motions from HAR- MON to create the complete whole-body motion. This comparison helps us assess the effectiveness of the iterative motion adjustment.
HARMON w/o Head Finger. This is ablated version of HARMON where the head and finger joints defaulted to zero position rather than generated.
By comparing HARMON with these baselines, we aim to evaluate the importance of human motion priors, iterative motion adjustment, and the head and finger motion generation.
The resulting humanoid motion from HARMON and the baseline models are rendered into simulation videos and shown to participants in human study. For each video, participants evaluateevaluate whether the humanoid's movements align with the text, focusing on three specific aspects: 1) finger and head movements, 2) arm movements, and 3) overall body coordination. The movement of these body parts reflects the expressiveness, accuracy, and naturalness of the generated motion. total, we collect assessments for 1728 results from different methods provided by 12 participants.
4.2 Evaluation of Humanoid Motion Generation
We first evaluate the overall alignment between the generated humanoid motion and the correspond- ing language description for each method. Each aspect (finger and head movements, arm move- ments, and overall body coordination) is scored by participants, contributing one point if marked as correct. We aggregate the scores across all aspects, results, and participants to obtain total score for each method. The total scores are then divided by the maximum possible score to calculate the normalized scores. The overall normalized scores are displayed on the left part of Fig. 4. HARMON achieves a high normalized score of 81.2%, significantly outperforming the baselines. The VLM- based motion generation baseline receives the lowest normalized score, underscoring the importance of incorporating human motion priors in humanoid motion generation.
We conduct more detailed analysis of the text-motion alignment of different body parts for each method. We compute the scores for each body part separately and visualize the results on the right part of Fig. 4. The VLM-based motion generation bascline consistently scores the lowest across almost all body parts. This result highlights the difficulty for VLM in generating accurate motions directly from language descriptions without reasonable initialization from human motion priors, especially for complex motions. The human motion retargeting baseline shows lower normalized score for arm motions, demonstrating that the VLM-based iterative motion adjustment improves alignment between arm motions and the language description. The baseline without finger or head motions naturally scores lower in these aspects. However, the normalized score is non-zero because some test motion descriptions do not involve finger motions.
Additionally, we visualized some of HARMON's results in Fig. 5. The retargeted human motion pro- vides solid initialization for humanoid motion, especially for fighfrreuencc actions like clapping (first row). These visualizations also highlight various instances where the retargeted human motion
7

---

Generated Human Motion
Retargeted Humanoid Motion
Refined Humanoid Motion
Text: "Clap the hands above the head."
Text: "Raise the right thumb-up front the chest the head."
Text: 'Tilt the head to the right and scratches the head."
Text: "Point diagonally upwards to the left with index finger and places the left hand on chest."
Figure 5: Qualitative results HARMON. We highlight the generated circles and the motion adjustment with red arrows.
finger motions with
misaligns with the language description and how VLM-based motion editing corrects these issues. For example, in the second row, VLM-generated finger and head motions significantly enhance the expressiveness of the humanoid motion. In the third row, the retargeting error causes the humanoid motion to deviate from the original human motion, but the iterative motion adjustment successfully realigns it with the text description. Additionally, there are cases where the generated human motion itself did not align well with the language description, as seen the fourth row. Here, iterative motion editing also improved the alignment, demonstrating its effectiveness in refining the motion match the language description better.
4.3 Real-Robot Deployment
When executing the generated motions on the real robot, we decouple locomotion from upper-body movements to maintain the robot's balance while demonstrating the generated motions. We tested both standing motions and motions requiring locomotion on real GR1 humanoids. We illustrate some example results in Fig. 1. Videos showcasing the real robot executing the generated motions are on the project website.
Conclusion
This work addresses the problem of generating humanoid motion from free-form language descrip- tions. We introduce HARMON, which utilizes human motion priors and the commonsense reasoning capabilities of VLMs for humanoid motion generation. Our results demonstrate that HARMON produce natural, expressive, and text-aligned humanoid motions, which are executable real hu- manoid robots. Our iterative motion adjustment process relies on fixed set of intuitive control primitives. These primitives struggle when the generated human motion significantly deviates from the language description, especially for high- high-frequency motions. A potential future direction is to enable the VLM to generate free-form control primitives. In this work, we separate lower- and upper-body control for the real robot. As result, some generated motions may cause the robot to lose balance collide with itself. We aim to incorporate RL-based whole-body control for more robust execution on the real robot in future work.

---

References
[1] C. Guo, S. Zou, X. Zuo, S. Wang, Ji, X. Li, and L. Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5152-5161, 2022.
[2] N. Mahmood, N. Ghorbani, N. F. Troje, G. Pons-Moll, and M. J. Black. AMASS: Archive of motion capture as surface shapes. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 5441-5450, Oct. 2019.
[3] Advanced Computing Center for the Arts and Design. ACCAD MoCap Dataset. URL https::
// aaccadd osu.
-adaaa
[4] C. Mandery, O. Terlemez, M. Do, N. Vahrenkamp, and Asfour. Unifying representations and large-scale whole-body motion databases for studying human motion. IEEE Transactions on Robotics, 32(4):796-805 2016.
[5] Carnegie Mellon University. CMU MoCap Dataset. URL http: hatt:::///oaap:: cs cmu edu.
[6] Y. Yuan, J. Song, U. Iqbal, A. Vahdat, and J. Kautz. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 16010-16021, 2023.
[7] M. Loper, N. Mahmood, J. Romero, G. Pons-Moll, and M. J. Black. SMPL.: A skinned multi- person linear model. ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6))244:1-244 16, Oct. 2015.
[8] X. B. Peng, P. Abbeel, S. Levine, and M. van de Panne. Deepmimic: Stample-guideds deep inforcement learning of physics-basee character skills. ACM Trunsuctions on Graphics (TOG), 37(4):1-14, 2018.
[9] X. B. Peng, A. Kanazawa, J. Malik, P. Abbeel, and S. Levine. Sfv: Reinforcement learning of physical skills from videos. ACM Transactions Graphics 3766 2011,
[10] Y. Yuan, S.-E. Wei, Simon, Kitani, and J. Saragih. Simpoe: Simulated character control for 3d human pose estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[11] Z. Luo, J. Cao, K. Kitani, W. Xu, et al. Perpetual humanoid control for real-time simulated avatars. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10895-10904, 2023.
[12] Z. Luo, J. Cao, J. Merel, A. Winkler, Huang, K. Kitani, and W. Xu. Universal humanoid motion representations for physics-based control. arXiv preprint arXiv:2310.04582 2023.
[13] X. B. Peng, G. Berseth, K. Yin, and M. Van De Panne. Deeploco: Dynamic locomotion skills using hierarchical deep reinforcement learning. Acm transactions on graphics (tog), 36(4): 1-13, 2017.
[14] X. B. Peng, M. Chang, G. Zhang, P. Abbeel, and S. Levine. Mep:
Learning composable hi- erarchical control with multiplicative compositional policies. Advances in Neural Information Processing Systems, 32, 2019.
[15] X. B. Peng, Y. Guo, I.. Halper, S. Levine, and S. Fidler. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), 41(4):1-17, 2022.
[16] X. B. Peng, Z. Ma, P. Abbeel, Levine, and A. Kanazawa. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions Graphics (ToG), 40(4): 1-20, 2021.

---

[17] N. Chentanez, M. Müller, M. Macklin, V. Makoviychuk, and S. Jeschke. Physics- thysics-based mo- tion capture imitation with deep reinforcement learning. In Proceedings of the 11th ACM SIGGRAPH Conference on Motion, Interaction and Games, pages 1-10, 2018.
[18] T. Wang, Y. Guo, M. Shugrina, and S. Fidler. Unicon: Universal neural controller for physics- based character motion. arXiv preprint arXiv:2011.15119 2020.
[19] Merell, S. Tunyasuvunakool, A. Ahuja, Y. Tassa, L. Hasenclever, V. Pham, T. Erez, G. Wayne, and N. Heess. Catch & carry: reusable neural controllers for vision-guided whole-body tasks. ACM Transactions on Graphics (TOG), 39(4):39-1, 2020.
[20] L. Hasenclever, F. Pardo, R. Hadsell, N. Heess, and J. Merel. Comic: Complementary task learning & mimicry for reusable skills. In International Conference on Machine Learning, pages 4105-4115. PMLR, 2020.
[21] K. Bergamin, S. Clavet, D. Holden, and J. Forbes. Drecon: data-driven responsive control of physics-based characters. ACM Transactions Graphics (TOG), 38(6):1-11, 2019.
[22] L. Fussell, K. Bergamin,, and D. Holden. Supertrack: Motion tracking for physically simulated characters using supervised learning. ACM Transactions on Graphics (TOG), 40(6):1-13, 2021.
[23] A. Winkler, J. Won, and Y. Ye. Questsim: Human motion tracking from sparse sensors with simulated avatars. In SIGGRAPH Asia 2022 Conference Pupers, pages 1-8, 2022.
[24] K. Gong, B. Li, J. Zhang, T. Wang. J. Huang, M. B. Mi, J. Feng, and X. Wang. Posetriplet: Co-evolving 3d human pose estimation, imitation, and hallucination under self-supervision. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11017-11027, 2022.
[25] K. N. Kumar, I. Essa, and S. Ha. Words into action: Learning diverse humanoid robot behaviors using language guided iterative motion refinement. arXiv preprint lrXiv:2310.06226, 2023.
[26] X. Cheng, Y. Ji. J. Chen, Yang, G. Yang, and Wang. Expressive whole-body control for humanoid robots. arXiv preprint arXiv:2402.16796 2024.
[27] T. He, Z. Luo, W. Xiao, C. Zhang, K. Kitani, C. Liu, and G. Shi. Learning human-to-huue ho-mma-t- real-time whole-body teleoperation. arXiv preprint arXiv:2403.044333 2024.
[28] T. Yoshida, A. Masumori, and T. Ikegami. From text motion: Grounding gpt-4 a humanoid robot" alter3". arXiv preprint arXi::2312.06577 2023.
[29] K. Fragkiadaki, S. Levine, P. Felsen, and J. Malik. Recurrent network models for human dynamics. In Proceedings of the IEEE International Conference on Computer Vision, pages 4346 4354, 2015.
extended complex human motion synthesis. arXiv preprint arXiv: 17X7v:1707.053666 2017. [30] Z. Li, Y. Zhou, S. Xiao, C. He, Z. Huang, and H. Li. Auto-conditioned recurrent networks for
[31] Martinez, M. J. Black, and J. Romero. On human motion prediction using recurrent neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recogni- fion, pages 2891-2900, 2017.
[32] D. Pavllo, D. Grangier, and M. Auli. Quaternet: A quaternion-based recurrent model for human motion. arXiv preprint
2018.
[33] A. Gopalakrishnan, A. Mali, D. Kifer, L. Giles, and A. G. Ororbia. A neural temporal model for human motion prediction. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 12116-12125, 2019.
10

---

[34] E. Aksan, M. Kaufmann, and O. Hilliges. Structured prediction helps 3d human motion mod- elling. In Proceedings of the IEEF International Conference on Computer Vision, pages 7144- 1153,2015
[35] W. Mao, M. Liu, M. Salzmann, and H. Li. Learning trajectory dependencies for human motion prediction. In Proceedings of the IEEE International Conference on Computer Vision, pages 9489-9497, 2019.
[36] Y. Yuan and K. Kitani. Ego-pose estimation and forecasting as real-time pd control. In Proceedings of the IEEE International Conference on Computer Vision, pages 10082-10092, 2019.
[37] E. Barsoum, J. Kender, and Liu. Hp-gan: Probabilistic human motion prediction via gan. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1418 1427, 2018.
[38] X. Yan, A. Rastogi, R. Villegas, K. Sunkavalli, E. Shechtman, S. Hadap, E. Yumer, and H. Lee. Mt-vae: Learning motion transformations to generate multimodal human dynamics. In Pro- ceedings of the European Conference on Computer Vision (ECCV), pages 265-281, 2018.
[39] Aliakbarian, F.S. Saleh, M. Salzmann, L. Petersson, and S. Gould. A stochastic conditioning scheme for diverse human motion prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5223-5232, 2020.
[40] Y. Yuan and K. Kitani. Dlow: Diversifying latent flows for diverse human motion prediction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 346-364. Springer, 2020.
[41] M. Petrovich, M. J. Black, and G. Varol. Action-conditioned 3d human motion synthesis with transformer vae. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10985-10995, 2021.
[42] G. Tevet, S. Raab, B. Gordon, Y. Shafir, D. Cohen-Or, and A. H. Bermano. Human motion diffusion model. arXiv preprint arXiv: 2291::62009.99166 2022.
[43] M. Zhang. Z. Cai, L. Pan, F. Hong, X. Guo, L. Yang, and Z. Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208. 15001, 2022.
[44] B. Jiang, X. Chen, W. Liu, Yu, G. Yu, and Chen. Motiongpt: Human motion foreign language. Advances in Neural Information Processing Systems, 36, 2024.
[45] Y. Zhang, D. Huang, B. Liu, S. Tang, Y. Lu, L. Chen, L. Bai, Q. Chu, N. Yu, and W. Ouyang. Motiongpt: Finetuned iims are general-purpost motion generators. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 7368-7376, 2024.
[46] Liang, W. Huang, Xia, P. Xu, K. Hausman, B. Ichter, Florence, and Zeng. Code as policies: Language model programs for embodied control, 2023.
[47] K. Lin, C. Agia, T. Migimatsu, M. Pavone, and J. Bohg. Text2motion: from natural
language instructions to feasible plans. Autonomous Robots, 47(5):13451365, Nov. 2023. ISSN 1573-7527. 10011000000//0111002----000 110514-023-10111--
URL http://áx.dom http:///dt.ddi..o/1......0.0 ort./o..0/00......1
[48] I. Singg, V. Blukis, A. Mousavian, A. Goyal, D. Xu, J. Tremblay, D. Fox, J. Thomason, and A. Garg. Progprompt: Generating situated robot task plans using large language models, 2022.
[49] G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu. L. Fan, and A. Anandkumar. Voyager: An open-ended embodied agent with large language models, 2023.
11

---

[50] Y. Ding, X. Zhang, C. Paxton, and S. Zhang. Task and motion planning with large language models for object rearrangement, 2023.
[51] Y. Xie, C. Yu, T. Zhu, J. Bai, Z. Gong, and H. Soh, Translating natural language to planning goals with large-language models, 2023.
[52] Silver, Dan, K. Srinivas, J. Tenenbaum, I.. P. Kaelbling, and M. Katz. Generalized planning in pddl domains with pretrained large language models, 2023.
[53] B. Liu, Y. Jiang, X. Zhang, Q. Liu, S. Zhang, J. Biswas, and P. Stone. Lim+p: Empowering large language models with optimal planning proficiency, 2023.
[54] S. Huang, Z. Jiang, H. Dong. Y. Qiao, P. Gao, and H. Li. Instruct2act: Mapping multi-modality instructions to robotic actions with large language model, 2023.
[55] H. Wang, G. Gonzalez-Pumariers Y. Sharma, and Choudhury. Demo2code: From summa- rizing demonstrations synthesizing code via extended chain-of-thought, 2023.
[56] Y. J. Ma, W. Liang, G. Wang, D.-A. Huang, O. Bastani, D. Jayaraman, Y. Zhu, L. Fan, and
A. Anandkumar. Eureka: Human-level reward design via coding large language models, 2024.
[57] H. Liu, A. Chen, Y. Zha, A. Swaminathan, A. Kolobov, and C.-A. Cheng. Interactive robot learning from verbal correction. arXiv preprint arXiv:2310.17555, 2023.
[58] W. Yu, N. Gileadi, C. Fu, S. Kirmani, K.-H. Lee, M. G. Arenas, H.-T. L. Chiang. T. Erez, L. Hasenelever, J. Humplik, B. Ichter, T. Xiao, P. Xu, A. Zeng, T. Zhang, N. Heess, D. Sadigh,
J. Tan, Y. Tassa, and F. Xia. Language to rewards for robotic skill synthesis, 2023.
[59] Y. Wang. Z. Sun, J. Zhang, Z. Xian, E. Biyik, Held, and Z. Erickson. Rl-vIm-f: Reinforce- ment learning from vision language foundation model feedback, 2024.
[60] D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[61] S. Caron, Y. De Moni-Marin, R. Budhiraja, and S. H. Bang. Pink: Python inverse kinematics based on Pinocchio, 2024. URL httts:ttt//tthhie cwm/ hetp/stephanee-actee carion/eee-arr//pppe sarronn/pnn..
[62] M. Vukobratovic and D. Juricie. Contribution the synthesis of biped gait. IEEE Transactions on Biomedical Engineering, (1):1 1969.
12


Reviews:
[{"review_id": "1UW44rgIb6", "reviewer": "Reviewer_4CM1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "The paper proposes a method called HARMON. The whole system starts with a language description, uses that to generate a human mesh which is retargetted using an IK based solver, and then uses a VLM to generate and iterate head and finger motions. The proposed method is shown on hardware and compares against various ablations of the proposed method.", "summary_of_recommendation": "The paper takes an interesting approach to handling head and finger refinement of generated motions, presents compelling real world results, but is lacking some comparisons which help understand the importance of the proposed method.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "- For \"VLM based motion generation\" baseline, it might be a fairer comparison to compare upper body only since attaching a lower body which is incoherent might make it a lower quality output overall.  \n- If I understand correctly, HARMON w/o head and finger uses the default head and finger values from the previous retargetted prediction?\n- It looks like another baseline would be \"VLM-based Motion Generation\" + starting from a motion prior model + iterative refinement (without explicit hand and head module) but still using the additional text description from GPT4. Essentially, this will highlight the importance of the hand and head motion generation module before iterative refinement.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UAv4C18Eaa", "reviewer": "Reviewer_HfRF", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 3, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "This paper presents an approach to create humanoid motion trajectories from text instruction. The approach retargets upper-body human motion trajectories to a humanoid robot using IK and then uses a fine-tuning step that leverages the common-sense reasoning capabilities of VLMs. The paper evaluates the approach using a human study and showcases hardware results by executing the trajectory on a humanoid robot.", "summary_of_recommendation": "The paper has an interesting idea but additional analysis needed to make it useful for the robotics community", "technical_quality": 3, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 2, "questions_for_rebuttal": "1. What is the difference between the “human motion Retargeting” and Harmon w/o Finger or Head” baselines?\n2. Can the framework support more fine-grained modifications that are not supported by the current control primitives? \n3. Follow up to the above question, can the VLM reason about parameterized control primitives? eg: move target by x, where x has to be generated by VLM. \n4. How are self-collisions handled by the approach?", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5BsdDpjfkn", "reviewer": "Reviewer_Ynst", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": 4, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "An IK-based humanoid motion generation approach with motion prior extracted from existing human motion datasets following a VLM-based motion refinement step.", "summary_of_recommendation": "The paper proposes a pipeline to generate language conditioned motions for humanoid robots. The method extracts human motions prior from existing datasets and refines the motion via VLM for better retargeting. However, the adopted IK solver may cause improper motions and can be improved. Therefore, my recommendation is weak accept.", "technical_quality": 4, "clarity_of_presentation": 4, "potential_impact": 3, "robotics_focus": 4, "quality_of_limitations": 3, "questions_for_rebuttal": "See the weaknesses above.", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: Evaluating this paper is very challenging as there is very little to no prior art on the topic. The reviewers are overall mildly positive but not enthusiastic about the paper. The reviewers raise the concerns that the approach is very limited due to the required primitives which are very vague, e.g. move hand in direction x. Furthermore, the approach relies on very elaborate prompts and it is unclear how sensitive the system is to small changes in the prompts. It is unclear how much the VLM really understands the task or just understands how to match words in the prompt. 

As it is the first CoRL where such work is appearing it is hard to evaluate. On the positive side, the authors provide the full prompts in the appendix, apply their method to a humanoid and present their approach clearly. On the negative side, the performed experiments are very simple and the robot movement only vaguely resembles the described simple movement aka wave hand. 

I would recommend accepting the paper as VLM + robotics is a hot topic and the authors did one approach and clearly described their approach. However, I would not defend my recommendation.
Decision: Accept