From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidance
Authors: Zhe Li
Yangyang Wei
Boan Zhu
Yibo Peng
Tao Huang
Pengwei Wang
Zhongyuan Wang
Cheng Chi
Chang Xu
Shanghang Zhang
Venue: ICLR 2026 Poster (2026)
Keywords: Humanoid locomotion; Reinforcement Learning; Motion Generation
Primary area: applications to robotics, autonomy, planning

Abstract: Natural language offers a natural interface for humanoid robots, but existing text-to-motion pipelines remain cumbersome and unreliable. They typically decode human motion, retarget it to robot morphology, and then track it with a physics-based controller. However, this multi-stage process is prone to cumulative errors, introduces high latency, and yields weak coupling between semantics and control. These limitations call for a more direct pathway from language to action, one that eliminates fragile intermediate stages. Therefore, we present RoboGhost, a retargeting-free framework that directly conditions humanoid policies on language-grounded motion latents. By bypassing explicit motion decoding and retargeting, RoboGhost enables a diffusion-based policy to denoise executable actions directly from noise, preserving semantic intent and supporting fast, reactive control. A hybrid causal transformer–diffusion design further ensures long-horizon consistency while maintaining stability and diversity, yielding rich latent representations for precise humanoid behavior. Extensive experiments demonstrate that RoboGhost substantially reduces deployment latency, improves success rates and tracking accuracy, and produces smooth, semantically aligned locomotion on real humanoids. Beyond text, the framework naturally extends to other modalities such as images, audio, and music, providing a general foundation for vision–language–action humanoid systems.

Published as a conference paper at ICLR 2026
FROM LANGUAGE TO LOCOMOTION: RETARGETING- HUMANOID CONTROL VIA MOTION LATENT GUIDANCE
Zhe Li Yangyang Wei Boan Zhu Yibo Peng', Tao Huang", Pengwei Wang* Zhongyuan Wang', Cheng Chi Chang X Shanghang Zhang BAAL, Harbin Institute of Technology. 3 Hong Kong University of Science and Technology Shanghai Jiao Tong University, Peking University, University of Sydney
Previous
Ours
Time Cost and Succ Rate
Prompt
Policy
Prompt
Diffusion Policy
Motion Generator
Decoder
Retarget
Motion Generator
D 0000
Text Input: The person performing backflip.
Text Input: A dances with fluir then leaps forward.
RoboGhost is retargeting- free latent driven policy for language-guided humanoid locomotion. By removing dependency motion retargeting. language commands. The figure showcases (a) the previous pipeline with motion retargeting, (b) our proposed retargeting free latent-driven pipeline. (c) quantitative comparisons of success rate and time cost between baseline and RoboGhost, (d) performing the backflip, and (e) dancing and leaping forward.
Figure 1:
allows robots controlled directly open-ender
ABSTRACT
Natural language offers natural interface for humanoid robots, but existing language-guided humanoid locomotion pipelines remain cumbersome and un- reliable. They typically decode human motion, retarget it to robot morphology, and then track it with physics-based controller. However, this multi-stage process is prone cumulative errors, introduces high latency, and yiclds weak coupling between semantics and control. These limitations call for more direct pathway from language action, one that eliminates fragile intermediate stages. There- fore, we present RoboGhost, retargeting free framework that directly conditions humanoid policies on language-grount grounded motion latents. By bypassing explicit motion decoding and retargeting, RoboGhost enables diffusion-based policy denoise executable actions directly from noise, preserving semantic intent and supporting fast, reactive control. A hybrid causal transformer diffussio motion generator further ensures long-horizon consistency while maintaining stability and diversity, yielding rich latent representations for precise humanoid behavior. Extensive experiments demonstrate that RoboGhost substantially reduces deploy- ment latency, improves success rates and tracking accuracy, and produces smooth,
"Equal Contribution
Project Leader
Corresponding Author

---

Published as a conference paper at ICLR 2026
semantically aligned locomotion real humanoids. Beyond text, the framework naturally extends to other modalities such as images, audio, and music, providing a general foundation for vision- hanga-llggggeeeee action humanoid systems. Project Page
INTRODUCTION
Natural language provides an intuitive interface for humanoid robots, enabling users to translate free- form instructions into physically feasible humanoid motion. Recent text-to-motion (T2M) models can generate diverse and semantically meaningful human motions (Li et al. 2024d] Chen et al. 20233
Guo et al. 2024| Li et al., 2024b Tevet et al. 2022). However, deploying these models on real robots typically requires hicrarchical pipeline: decoding human motion from language, retargeting it to robot morphology, and tracking the trajectory with a physics- based controller (Peng et al. 2018 Ji
al. 2024 Chen et al., 2025 He et al. 2025a Zhou et al. 2025b).
This pipeline, while functional in constrained settings, suffers from systemic drawbacks. (1) Errors accumulate across decoding. retargeting, and tracking, degrading both semantic fidelity and physical feasibillity. (2) High latency from multiple sequential stages, making real-time interaction difficult. (3) Loose coupling between language and control, since each stage is optimized in isolation rather than end-to-end. Recent refinements (Yue et al. 2025 Serifi et al. 2024 Shao et al. (025) attempt to mitigate these issues, but improvements are applied locally to decoders or controllers, leaving the overall pipeline fragile and inefficient.
Our key insight is simple: treat motion latents as first-class conditioning signals, directly use them as conditions to generate humanoid actions without decoding and retargeting altogether. We therefore propose RoboGhost, retargeting- free framework named highlight the latent representa- tions that are invisible like ghos! yel strongly drive humanoid behavior. As shown in Fig [ rather than producing explicit human motion, RoboGhost leverages language-conditinn
motion latents as semantic anchors to guide a diffusion-based humanoid policy. The policy denoises executable actions directly from noise, climinating error-prone intermediate stages while preserving fine-gred fine-grained intent and enabling fast, reactive control. To our knowledge, this is the first diffusion-based humanoid policy driven by motion latents, achieving smooth and natural locomotion with DDMM-accelerated sampling (Song et al. 2020) for real-time deployment.
To enhance temporal coherence and motion diversity, RoboGhost employs a hybrid causal trans- former-diffusion architecture. The transformer backbone captures long-horizon dependencies and ensures global consistency (Zhou et al.] 2024 2025a Han et al.) (202). 3. The diffusion component contributes stability and stochasticity for fine- mne-grained motion synthesis. Together, this design miti-
gates drift and information loss typical of autoregressive models, while producing expressive motion latents that provide downstream policies with rich semantic conditioning for precise control.
Extensive experiments validate the effectiveness and practicality of RoboGhost. We dramatically accelerate the full pipeline from motion generation to humanoid deployment, cutting the time from 17.85 s to 5.84 s. Beyond sheer speed, our approach yields higher-quality control by circumventing retargeting losses and improving generalization, which reflected in 5% higher success rate and reduced tracking error compared to baseline methods. Our framework achieves robust, semantically aligned locomotion on real humanoids, substantially reducing latency compared to retargeting- based pipelines. We can further extend this framework to support other input modalities such images, audio, and music, thereby offering a reference architecture for humanoid vision- aasion-language-attinn systems. In short, RoboGhost moves text-driven humanoid control from fragile pose imitation to robust, real-time interaction.
In summary, key contributions can be summarized follows:
We propose RoboGhost, retargeting-Fe ffee framework that directly leverages language- generated motion latents for end-to-end humanoid policy learning, eliminating error-provd pron-p decoding and retargeting stages.
We introduce the first tiffusion-based humanoid policy conditioned on motion latents, which denoises executable actions directly from noise and achieves smooth, diverse, and physically plausible locomotion with DDMM-accelerated sampling.
2

---

Published as a conference paper at ICLR 2026
We design hybrid wansformer-diftsioo architecture that unifies Long-horizon tempo- ral coherence with stochastic stability, yielding expressive motion latents and strong lan- guage-motion alignment.
We validate RohoGhost through extensive experiments, demonstrating its effectiveness and generality in enabling robust and real-time toguage-guider humanoid locomotion.
RELATED WORK
2.1
HUMAN MOTION SYNTHESIS
Sanguage-guidess humanoid locomotion leverages advances in text-to-motion generation, which
primarily transformer-basee discrete modeling diffusion-based continuous modeling. Discrete methods model motion as tokens, evolving from early vector quantization (Guo et al. (202cc) to
GPT-style autoregression (Zhang et al. 2023a), scaling with LLMs (Jiang al.) [023 Zhang
ct al.
2024) or improved attention (Zhong et al. (023), and recently, bidirectional masking (Guo
al.
2023) and enhanced text alignment (Li et al. 2024d). In parallel, diffusion models excel in
synthesis (Kim et al., [023) Hu et al., [023] Tevet et al. 2022 Li et al.) 2024c Bai et al.) 2025
al. 2023b), with progress in efficiency (Chen et al. 2023), sstreval-augmentaale
(Zhang et al.
2023b), controllability (Li et al., 2202bb), and architectures (Meng et al., 2024 Li et al. (202cc). Our
work builds on the continuous autoregressive framework (Li et al. 2024a), combining the benefits of both paradigms.
2.2 HUMANOID WHOLE-BODY CONTROL.
Whole-body control (WBC) Li et al. 2025alb b) is crucial for humanoids, yet learning a general- purpose policy remains challenging. Existing approaches exhibit inherent trade-offs: methods like OmniH20 (He et al. 2024) and HumanPlus (Fu et al. 2024) prioritize specific robustness at the cost of generality long-ter term accuracy, while others like Hover (He et al.] 2025b). ExBody2 (Ji et al.) 2024), and GMT (Chen et al., 2025) employ strategies (c.g.. masking, curriculum learning, MoE) to enhance adaptability, though generalization not guaranteed. Recent language-guided works also face limitations: Lang WBC (Shao et al., 2025) scales poorly with no generalization guarantee unseen instructions, RLPF (Yue et 2025) risks catastrophic forgetting and limited diversity, large model relying on motion retargeting and discrete action tokenization, and LeVERB (Xue et al., 2025) uses hierarchical CVAE-RL architecture for vision-language WBC but lacks support for high-dynamic motions. To overcome these issues, we propose a MoE-based oracle paired with a latent-driven diffusion student, enhancing generalization while reducing deployment cost via retargeting-free design.
UH-I (Mao et al.) 2025) is a transformer-
METHOD
This section presents the core components of our framework, which is depicted in Fig 2 We begin with an overview of our method in Section 3.1, providing a high-level description the architecture and motivation. Section 3.2 details the construction of our motion generator, which leverages continuous autoregression and causal autoencoder. Furthermore, Section 3.3 elaborates on our novel retargeting-fres latent-driven reinforcement learning architecture based on diffusion models, including its training and inference procedures. Finally, we present our causal adaptive sampling strategy in Section 3.4. Other details can be found in appendix 9.2
3.1 OVERVIEW
Our work introduces a novel retargeting-free, latent-driven reinforcement learning architecture for language-guided humanoid control, which fundamentally diverges from conventional motion-tracking pipelines. As depicted in Fig. our approach comprises three core components: a continuous autoregressive motion generator, a MoE-based teacher policy, and a latent-driven riffuusion-based student policy. We focus on the challenge of generating diverse, physically plausible motions from high-level instructions without the need for complex kinematic retargeting.
3

---

Published as a conference paper at ICLR 2026
Stagel-Motion Generation
"A person walks circle."
Task Instruction
Encoder
Text
Motion Latent
Frozen
Tra Inable
Add
Motion
Causal Transforser
AdaLN
Diffusion Model
Encoder
Stage2-Policy Training
Inference
Privilaged
p
Teacher Policy
Information
"A person doing jump jacks."
Text
Causal Transformer
Isturgtted Oatgueet
Proprieception
P
MOE
d+
Encoder
gt
Diffusion Model
Gaussian Noise
Diffusion
Task Instruction
History observations
notion latent
PD
Motion
Ot
Generator
DifFusion
at
Latent
Encoder
Student Policy
Figure 2: Overview of RoboGhost. We propose a two-stage approach: motion latent is first generated, then MoB- based teacher policy trained with RL and diffusion-bart based student policy is trained to denoisc actions conditioned motion latent. This latent-driven scheme bypasses the need for motion retargeting.
The process begins by feeding a textual prompt T into the continuous autoregressive motion generator. Unlike prior works that decode the generated motion into explicit kinematic sequence requiring tedious retargeting to the robot, our generator produces a compact latent motion representation f This design is pivotal, degradation caused by limited generator capability. The generation process can be formulated as:
it bypasses the error-prone retargeting step and mitigates performance
lre/ G(T), where G' denotes our motion generator. This latent representation Lref, alongside proprioceptive states and historical observations, then conditions a diffusion baasedd student policy TT The policy operates denoising process to output actions directly executable on the physical humanoid. This latent-driven paradigm eliminates the dependency on privileged information and explicit reference motions during deployment, significantly streamlining the sim-to-real pipeline.
To efficiently train the high-level oracle teacher policy, introduce causal adaptive sampling strategy. It dynamically prioritizes challenging motion segments by attributing failures to their causal antecedents. thereby maximizing sample efficiency and enabling the learning of motor skills. Finally, a meticulously designed reward function ensures accurate and expressive whole-body motion tracking. Collectively, our framework achieves robust, direct-drive control from language commands, setting new paradigm for retargeting- free humanoid locomotion.
agile
3.2 CONTINUOUS AUTOREGRESSIVE MOTION GENERATOR
Given that discretized models are prone to information loss and to leverage the advantages of both masked modeling and autoregressive approaches, adopt causal autoencoder and continuous masked autoregressive architecture with causal attention mask, which effectively captures temporal dependencies between tokens and produces rich contextual conditions for the subsequent diffusion process. Specifically, we first randomly mask motion tokens following the practice in language models (Devlin et al. 2019). obtaining set of masked tokens. The temporal masking strategy follows the same mask ratio scheduling as (Chang et al. 2022), defined by the function:
((T) = cos
(1)
where 7 E [o, 1]. During training, is uniformly sampled from U(0, 1). yielding a mask ratio (()). Accordingly, 7()) x N tokens randomly selected for masking.
4

---

Published as a conference paper at ICLR 2026
Unlike previous masked autoregressive methods (Li et al. 2024a Meng et al. 2024 Li et al. 2023a our approach does not involve random token shuffling or batch-token prediction. Moreover, to mitigate the limitation in model expressiveness caused by low-rank matrix approximations during training, we replace bidirectional attention masks with causal attention masking. And for the input text prompt T, we use the LaMP (Li et al., (0244d text transformer extract textual features. This model captures linguistic nuances and semantic structures effectively, resulting in high-dimensional feature representations that provide essential guidance for motion generation.
After the transformer completes the masked token prediction task, the predicted latent representations are used to condition the diffusion model. guiding the denoising process produce more accurate and semantically rich latent representations. This enables downstream latent- driven action generation.
3.3
LATENT-DRIVEN DIFFUSION POLICY FOR RETARGETING- FREE DEPLOYMENT
3.3.1 MOE-BASED TEACHER POLICY
The core challenge in locctooloocooooon is the inherent open-endedness of language. Therefore, generalization capability is the key enabler for policies to respond successfully to novel prompts and achieve true deployment flexibility.. First, we train an oracle teacher policy using PPO (Schulman al. 2017) with privileged simulator-state information. To learn policy я that generalizes across diverse motion inputs, we first train an initial policy To on a curated motion dataset Do of high diversity. We then evaluate the tracking accuracy of mo for each motion sequence s Do. focusing on the lower body through an error metric e(s) the mean key-body position error and Edor(s) the mean joint angle tracking error of the lower body. Motion sequences with e(s) > 0.6 are filtered out, and the remaining data D are used to train a general teacher policy.
Ekey (s) + 3 Edar(s), where Exey(s) denotes
The teacher policy Te is trained as oracle in simulation using PPO, leveraging privileged infor- mation unavailable the real ineludinl ground-truth root velocity, global joint positions, physical properties (e.g., friction coefficients, motor strength), proprioceptive state, and reference motion. The policy outputs target joint positions â, E 2R for proportional derivative (PD) control, maximizing cumulative rewards to achieve accurate motion tracking and robust behavior, resulting in a gegh-perffrmance expert policy trained solely on simulated data. Finally, we seek:
arg max E.ED [Performance(T,ee s]]
(2)
To enhance the expressive power and generalization capability of the model, we incorporate a Mixture of Experts (MoE) module into the training of the teacher policy. The policy network consists of a set of expert networks and gating network. The expert networks take as input both the robot state observations and reference motion, and output the final action a, The gating network receives the same input observations and produces probability distribution over all experts. The final action is computed as a weighted combination of actions sampled from each expert's 's output distribution:
EE= P: ag where p; denotes the probability assigned to the i-th expert by the gating network, and a, represents the output of the i-th expert policy. This architecture enhances the policy's ss generalization capacity and obtains actions that accurately track reference motions, thereby providing more precise supervised signals for the student policy.
3.3.2 DIFFUSION-BASE BASED STUDENT POLICY
Unlike prior approaches where the student policy म, distills knowledge from the teacher using refer- ence motion, we propose a novel latent-driven student policy that takes motion latent representations as input. In addition the observation history, it incorporates a latent representation (ref from the motion generator as input. This implicit design enables the policy to operate during inference without retargeted explicit reference motion, thereby streamlining deployment, reducing performance degradation caused by limitations of motion generator capability and retargeting.
Following a DAgger-like (Ross et al., 2011) approach, we roll out the student policy in simulation and query the teacher for optimal actions O at visited states. During training, we progressively inject Gaussian noise e, into the teacher's 's actions and use the first-stage pretrained motion generator to obtain latent representation with textual descriptions. The forward process can be modeled
5

---

Published as a conference paper at ICLR 2026
Markov noising process using:
((m((==1)) M(r a 21-1, 0z.T
where the constant 0g E (0. 1) is hyper-parameter for sampling. Here, we denote the denoiser as cØ. use { to denote the noising sequence, and Ft-1 = c0(2z, t) for the t-step denoising.
While the motion generator can produce motions lacking physical realism, our method does not blindly follow its output. Instead, a trainable latent encoder intelligently conditions the policy, translating raw proposals into actionable and stable commands. This enables the policy to generate diverse and robust actions even from imperfect guidance. This representation, along with proprioceptive states and historical observations, serves as conditioning to guide the denoising process. For tractability, we adopt an zy Wo-prediction strategy and supervise the student policy by minimizing the mean squared error loss L ||a dellă. where -=-VI-a The process iterates until convergence, voc yielding a policy that requires neither privileged knowledge nor explicit reference motion, and is suitable for real-world deployment.
3.3.3 INFERENCE PIPELINE
To ensure motion fluency and smoothness, it is essential to minimize the time required for the denoising process. We therefore adopt DDIM sampling (Song et al., (020) and an MLP-based diffusion model to generate actions for deployment. The reverse process can be formulated as:
Ft
col(s,))) + /1 01-1--11- (x( ap-1
2--1
04-1
(4)
a
Our framework is entirely retargeting- free and latent-driven During inference, the textual description is first imput into a motion generator and obtains a latent motion representation Lee.- Crucially, we bypass the decoding of this latent into explicit motion sequences, thus eliminating the need for retargeting to the robot. We sample a random noise as input to the student policy and condition the diffusion model via AdaLN (Huang Belongie 2017) on the motion latent, proprioceptive states P. and historical observations O.-Hit. producing a clean action a = eœ(e]fref PP. Ot- (e-H)),. that is directly executable on the physical robot. This streamlined pipeline not only reduces complexity but also mitigates issues such as low-quality motion generation due to limited generator capability, Entargeting-nduueee errors, and insufficient action diversity.
3.4 CAUSAL ADAPTIVE SAMPLING
RoboGhost aims tracking reference motion more expressively in the whole body. To this end, we employ novel sampling method to facilitate the teacher policy in mastering more challenging and longer motion sequences. Details about various reward functions, curriculum learning, and domain randomizations can be found in appendix 9.2
Training long-horizon motor skills faces a fundamental challenge: motion segments exhibit het- erogeneous difficulty levels. Conventional approaches typically employ uniform sampling across trajectories (He et al. 2025a Truong et al. 2024), which leads to oversampling of trivial segments and under-sampling of challenging ones, resulting in high reward variance and suboptimal sample efficiency. To address this, we propose a causality-aware adaptive sampling mechanism. The motion sequence is divided into K equal-length intervals, and the sampling probability for each interval is dynamically adjusted based on empirical failure statistics. Let k denote the interval in which a rollout terminates due to failure. We hypothesize that the root cause often arises steps prior to ku-such as a misstep or collision- propagates into eventual failure at ky To enable corrective learning, we increase the sampling likelihood of these antecedent intervals.
Motivated by the temporal structure of catluuurss-tppiaall preceded by suboptimal actions-wa apply an exponential decay kernel alu) 7 (7 E (0, 1)) to assign higher weights time steps leading up to termination. The base sampling probability for interval s is defined as:
Ap: a(t i) p. Ap: 0,
i [ 8,
(5)
i 4 [t s,

---

Published as a conference paper at ICLR 2026
where K controls the backward horizon for causal attribution. Subsequently, we update and normalize the sampling probabilities across all intervals. Let p; denote the base probability for interval i. The updated probability is given by: P/ P; + Apt, where Ap; denotes the increment derived from failure attribution. The distribution is then renormalized to ensure P1 = 1. Finally, the initial interval is sampled from the multinomial distribution Multinomial (p) 1.-....... ........ P'x ). enabling selective focus on high-difficulty segments. Since each interval consists of multiple frames, after sampling the restart interval, we further select the exact restart frame uniformly within that interval.
EXPERIMENT
We present rigorous empirical evaluation of proposed method in this section, structured to systematically address three core research questions central to its design and efficacy. We organize our analysis around the following three research questions:
Q1: Advantages of Latent-driven Pipeline. To what extent does a retargeting-fres latent-
driven approach improve efficiency, robustness, and generalization in motion generation, and how do these advantages manifest in real-world deployment scenarios?
Q2: Generalization Gains via Diffusion Policy. Does replacing MLP-basee policies with diffusion-based policies enhance generalization to unseen instructions or environments?
Q3: Better Motion Generator. Does the continuous autoregressive architecture yield more semantically and kinematically precise latent embeddings than standard alternatives? Further, how do architectural variants the diffusion model (e.g., MLP and DiT (Peebles &
Xie 2023)) affect motion generation quality?
Q4: Greater Robustness Policy. What advantages does the diffusion policy offer over the MLP policy the presence of noise interference, and will exhibit greater robustness?
4.1 EXPERIMENTS SETTINGS
Datasets and Metries We evaluate two MotionMillion (Fan et al. 2025 subsets: HumanML and Kungfu. For policy training, we use only index-O sequences per motion name (c.g., "00000 _nnpy") excluding mirrored variants to reduce redundancy. Motions with non-planar terrains infeasible contacts are filtered to ensure flat-ground consistency. The curated set split 8:2 (train:test) with stratified category balance. For motion generator training, we leverage the full subsets. Evaluation metrics include motion generation and motion tracking. Generation follows Guo et al. (2022b) with retrieval precision R@ 1, 3, MM Dist, FID, and Diversity. Tracking is assessed physics simulators, consistent with OmniH20 (He et al. (2024) and PHC (Luo et al.) (003). using success rate (primary), mean per-joint error (Emppjee), and mean per-keypoint error (Empkpe). Success rate reflects both accuracy and stability. Further details are in appendix 9.3
4.2 EVALUATION OF MOTION GENERATION
To validate the effectiveness of our continuous autoregressive motion generation model, we train and evaluate it two distinct skeleton representations: the 263-dim HumanML3D (Guo et al. 2022a and the 272-dim Humanml and Kungfu subsets of MotionMillion. We benchmark against text-to motion methods, including diffusion- and measformer-basel approaches. Motivated by SiT (Ma al. 2024 and MARDM (Meng et al. (2024), we reformulate the training objective of the motion generator from noise prediction velocity prediction, leading improved motion quality and enhanced dynamic consistency in generated sequences. As summarized in Table I our model achieves competitive performance across both formats, demonstrating robustness to representation variation.
4.3 EVALUATION OF MOTION TRACKING POLICY
To further validate the efficacy of our motion tracking policy, we conduct evaluations on the Hu- manML and Kungfu subsets of MotionMillion, measuring Empipe and Empkpe under physics-basee based simulation in both IsaacGym and MuJoCo. The pipeline operates follows: textual descriptions first input into the motion generator to produce a latent motion representation, which is subsequently input by the student policy for action execution. As summarized in Table our method achieves high
7

---

Published as a conference paper at ICLR 2026
Precision
Methods
FID,
MM-Dist
Diversity +
Top
Top
Top
HumanML3D
Ground Truth
0.702
0.864
0.914
0.002
15.151
27.492
MDM Tevet et al.
2023
0.523
0.692
0.764
23,454
17.423
26.325
ML.D Chen al. 2025
0.546
0.730
0.792
18,236
16.638
26.352
T2M-GPT
Zhang ct al. 202.3a
0.606
0.774
0.838
12.475
16.812
27.275
MotionGPT
[Jang
al 2023)
0,456
0.598
0.628
14.375
16.892
27.114
MoMask
(uo et al. 2023)
0,621
0.784
0,846
12.232
16.138
27.127
AttT2M Zhong et al.
2023
0,592
0.765
0.834
15.428
15.726
26.674
MotionStreamer (Xtao et al.
2025
0,631
0.802
0.859
11.790
16.081
27.284
Ours-DDPM Ours-SiT
0.639
0.808
0.867
11.706
15.772
27.230
0.641
0.812
0.870
11.743
15.663
27.307
HumanML (MotionMillion)
Ground Truth
0.714
0.876
0,920
0.002
14.984
26.571
Ours-DDPM Ours-SiT
0.644
0.819
0.873
11.724
15.870
26.395
0.646
0.818
0.872
11.716
15.603
26.471
Table 1: Quantitative results of text-to-motion generation on the HumanML3D dataset and HumanML subset. -+ denotes that if the value is closer to the ground truth, the metric is better.
IsaacCimm
MuJoCo
Method
Suce
Pappipe +
Eapppee
Suce t
Empipe +
Emphpe
HumanML (MotionMillion)
Baseline
0.92
0.23
0.19
0.64
0.34
0.31
Ours-DDPM
0.97
0.12
0.09
0.74
0.24
Ours-SiT
0.98
0.14
0.08
0.72
0.26
0.23
Kungfu (MotionMillion)
Baseline
0.66
0.43
0.37
0.51
0.58
0.52
Ours-DDPM
0.72
0.34
0.31
0.57
0.54
0.50
Ours-SiT
0.71
0.36
0.32
0.55
0.53
0.48
Table 2: Motion tracking performance comparison in simulation on the HumanML. and Kungfu test sets.
success rates on HumanMI, alongside low joint and keypoint errors, indicating robust alignment between generated motion semantics and physically executable trajectories. Here, Baseline refers to the conventional, explicit framework where both the teacher and student policies use MLP backbones.
4.4 QUALITATIVE EVALUATION
We present a qualitative evaluation of the motion tracking policy across three deployment stages: simulation (IsaacGym), cross-simulator transfer (MuloCo), and real-world execution the Unitree GI humanoid. Fig 3 visualizes representative tracking sequences, highlighting the policy's asicy's ability to preserve motion semantics, maintain balance under dynamic transitions, and generalize across physics engines and hardware. In particular, real-robot deployments demonstrate that our latent- driven, retargeting-free free framework enables smooth, temporally coherent motion execution without manual tuning, validating its readiness for practical embodied applications.
4.5 ABLATION STUDIES
To systematically answer the three research questions posed at the outset this section, we conduct various ablation studies.
8

---

Published as a conference paper at ICLR 2026
*TH 7N
V4PS
A G GAEAAIEAA grHaL
1" (IStOnmAE SOP V. SE7 $2.4
FURDYY INVERNN SNIVERSST SUSTERIIS dIR BUe
F (FAPON MTV. 100 NRARTF WRONE COROOO
Figure 3: Qualitative results in the IsaacGym and MuJoCo.
Method
Time (s)
Suce f
Empsse
Empkpe
PHC-100
1.63
0.81
0.45
0.40
PHC-500
0.88
0.31
0.25
PHC-800
9.87
0.91
0.25
0.21
PHC-1000
11.89
0.93
0.21
0.17
Table Average inference time and tracking performance different retargeting methods.
To answer Q1 (Advantages of Retargeting-free Pipeline), we evaluate the conventional pipeline: text prompts are fed to the motion generator, the output latent decoded into explicit motion, which is then retargeted to the GI robot and executed by the policy. As shown in Table 4 this approach underperforms due to its multi-stage complexity and error accumulation, and incurs higher real-world deployment time cost. This confirms the efficiency and performance advantage of retargeting free, latent-driven framework.
Furthermore, investigate the time consumption and corresponding tracking performance of different PHC retargeting iteration steps under the explicitly-driven framework. As shown in Table 3 reducing the number of PHC iteration steps accelerates the retargeting process, but significantly degrades the tracking performance.
To answer Q2 (Generalization Gains via Diffusion Policy), we conduct ablation by replacing the diffusion policy with an MLP-based policy that concatenates latent and other observations to predict actions. Diffusion policies, by design, better capture various action distributions, enabling more robust adaptation to diverse or imperfect latents. As shown in Table 5 the diffusion policy significantly outperforms its MLP counterpart.
To further evaluate generalization, we test both policies on 10 randomly sampled motions from unseen MotionMillion subsets (fitness, perform, 100style, haa). Although the motion generator was not trained on these subsets, resulting in suboptimal latents, the diffusion-basee policy still achieves substantially better tracking and robustness than MLP, as evidenced Table This highlights humanoid diffusion policy's superiority.
To address Q3 (Better Motion Generator), we conduct an ablation study on the diffusion backbone in framework, comparing 16-layer MLP and 4-layer DiT under identical settings. As shown in 6 DiT offers slight gains in generation metrics but no measurable improvement in tracking success joint accuracy, while incurring higher latency due to larger model size. Balancing effectiveness and efficiency, we therefore adopt the 16-layer MLP as our default backbone.
To address Q4 (Greater Robustness Policy), we present the tracking performance of different methods on the same motion in simulation the upper part of Figure 4 When tracking highly dynamic motions, the MIP policy performs significantly worse than the diffusion policy. Furthermore,

---

Published as a conference paper at ICLR 2026
IsaacGym
MuloCo
Method
Time Cost (8)
Suce
Empipe
Empkpe |
Suce
Empipe
Empkpe
Ours-Explicit
0.93
0.21
0.17
0.66
0.32
0.27
17.85
Ours-Implicit
0.97
0.12
0.09
0.74
0.24
0.20
5.84
Table 4: Motion tracking performance comparison across different simulators on the HumanML. and Kungfu test sets, Explicit version including PHC retargeting (1000 interations) and latent decode processes.
IsaacGym
IsaacGym
Method
Method
Suce
Empige 4 Empkpe
Suce
Empipe
Empkpe
MLP Policy
0.96
0.17
MLP Policy
0.54
0.48
0.45
Diffusion Policy
0.97
0.12
0.09
Diffusion Policy
0.68
0.42
0.39
Table 5:
Comparison of MLP-based and diffusion- basod policy. The left table shows the tracking performance on HumanML subset, and the right table presents the generalization ability of two different policy architectures.
evaluate the robustness of different policies. As shown in the lower part of Figure 4 in the MuJoCo simulator, when we randomly add noise with a noise scale of 0.2 to the observations both policies, the MLP policy maps this noise to noisy actions, ultimately causing the robot to fall. In contrast, the diffusion policy exhibits strong robustness against such noise and still achieves favorable tracking performance. We also investigate the maximum tolerable noise scale for both policies: the MLP policy can only complete the task when the noise scale is 0.12, while the diffusion policy maintains task completion at noise scale of up 0.33.
IsaacGrym
HumanML33
Method
Time Cost (s) |
Succ
Empspe
Empkye
Top
FID
DIT
0.96
0.11
0.11
0.870
11.697
14.28
MLP
0.12
0.09
0.867
11.706
5.84
Table Comparison tracking performance across different diffuusio backbones motion generation.
MLP Policy
Our Ditfusion Policy
No Noise
Tringinz Vorma
Tracksh
Noise Scale 0.2
Figure 4: Comparison between our diffusion policy and MLP policy regarding their tracking performance under noise-free conditions and robustness after the introduction of noise.
5 CONCLUSION
In this paper, we introduce RoboGhost, a vetargeting-fiee latent-driven framework that bridges natural language instructions with physically feasible humanoid motion control. Our method harnesses rich semantic representations from pretrained text-to-motion model and integrates them into diffusion-basee humanoid policy, thereby bypassing motion decoding and retargeting stages that not only error-prone but also time-consuming. This design enables real-time, language-guidee humanoid locomotion with improved adaptability. Extensive experiments show that RoboGhost re- duces cumulative distortions and exccution latency while preserving semantic alignment and physical realism. Beyond advancing efficiency and robustness, RoboGhost offers a new and practical pathway environments.
toward more intuitive, responsive, and deployable humanoid robots in
10

---

Published as a conference paper at ICLR 2026
6 ETHICS STATEMENT
This research was conducted exclusively in simulation environments (IsaacGym, MuJoCo) and on humanoid robot hardware. It does not involve the use of any human subjects personal identifiable data. All motion data utilized are sourced from publicly available, authorized human motion capture databases. The primary goal of our work is to advance the field of humanoid robot control, with po- tential applications (e.g., assistive robotics) aimed at benefiting society. We have carefully considered the fairness and safety of our algorithms and are committed developing robust control systems to minimize the risk of unintended physical harm in real-world deployment. There are no known conflicts of interest associated with this study.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of this work, have provided comprehensive details in the main text and appendix, including the network architecture, hyperparameter settings, and curriculum learning strategies. All motion sequence data used for training are sourced from publicly available datasets. Demo videos have been included in the supplementary material for review. Detailed descriptions of the dynamic model, reward function design, and termination conditions can be found in Section and Appendix A. We encourage interested readers to consult these resources to replicate our results. Upon acceptance of the paper, the relevant code will be made publicly available.
THE USE OF LARGE LANGUAGE MODELS (LLMs)
LLMs were used in the preparation of this work solely for the purpose of grammar correction and proofreading. The LLM did not contribute to the idcation, scientific content, writing, or analysis of the research. The authors take full responsibility for the entire intellectual content of this paper.
REFERENCES
Joao Pedro Araujo, Yanjie Ze, Pei Xu, Jiajun Wu, and C Karen Liu. Retargeting matters: General motion retargeting for humanoid motion tracking. arXiv preprint arXiv:2510.02252, 2025.
Shuanghao Bai, Wenxuan Song, Jiayi Chen, Yuheng Ji, Zhide Zhong, Jin Yang. Han Zhao, Wanqi Zhou, Wei Zhao, Zhe Li, Pengxiang Ding, Cheng Chi, Haoang Li, Chang Xu, Xiaolong Zheng, Donglin Wang, Shanghang Zhang, and Badong Chen. Towards unified understanding of robot manipulation: A comprehensive survey. arXiv preprint arXiv:2510 10903, 2025.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1131--11333 13355, 2022.
Xin Chen, Biao Jiang, Wen Liu, Zilong Huang, Bin Fu, Tao Chen, and Gang Yu. Executing your commands via motion diffusion in latent space. In Proceedings the IEEE/CVF conference computer vision and pattern recognition, pp. 18000-18010, 2023.
Zixuan Chen, Mazeyu Ji, Xuxin Cheng, Xuanbin Peng. Xue Bin Peng, and Xiaolong Wang. Gmt: General motion tracking for humanoid whole-body control. arXiv preprini 25Xi:2550611777 2025.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pp. 4171-4186, 2019.
Ke Fan, Shunlin Lu, Minyue Dai, Runyi Yu, Lixing Xiao, Zhiyang Dou, Junting Dong, Lizhuang Ma, and Jingbo Wang. Go zero: Towards zero-shot motion generation with million-scale data. arXiv preprint arXiv:2507.07095 2025.
Zipeng Fu, Qingqing Zhao, Qi WU, Gordon Wetzstein, and Chelsea Finn. Humanplus: Humanoid shadowing and imitation from humans. arXiv preprint arXiv:2406. 10454, 2024.
11

---

Published as a conference paper at ICLR 2026
Xinyang Gu, Yen-Jen Wang, and Jianyu Chen. Humanoid gym: Reinforcement learning for humanoid robot with zero-shot sim2real transfer. arXiv preprint arXiv:2404.05695 2024.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5152-5161 June 2022a.
Chuan Guo, Shihao Zou, Xinxin Zuo, Sen Wang, Wei Ji, Xingyu Li, and Li Cheng. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF conference computer vision and pattrrn recognition, pp. 5152-5161, 2022b.
Chuan Guo, Xinxin Zuo, Sen Wang, and Li Cheng. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In European Conference on Computer Vision, pp. 580-597. Springer, 2022c.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. 2023.
Chuan Guo, Yuxuan Mu, Muhammad Gohar Javed, Sen Wang, and Li Cheng. Momask: Generative masked modeling of 3d human motions. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 1900-1910, 2024.
Yi Han, Cheng Chi, Enshen Zhou, Shanyu Rong. Jingkun An, Pengwei Wang, Zhongyuan Wang, Lu Sheng, and Shanghang Zhang. Tiger: Tool-integrated geometric reasoning in vision-language models for robotics. arXiv preprint arXhv:2510.07188/ 2025.
Tairan Hc, Zhengyi Luo, Xialin He, Wenli Xiao, Chong Zhang, Weinan Zhang, Kris Kitani, Changliu Liu, and Guanya Shi. Omnih2o: Universal and dexterous suman-to-humann whole-body telcoperation and learning. arXiv preprint arXiv:2406. 08858, 2024.
Tairan He, Jiawei Gao, Wenli Xiao, Yuanhang Zhang, Zi Wang. Jiashun Wang. Zhengyi Luo. Guanqi He, Nikhil Sobanbab, Chaoyi Pan, et al. Asap: Aligning simulation and real-world physics for learning agile humanoid whole-body skills. arXiv preprint ar arXiv:2502.01143 143, 2025a.
Tairan He, Wenli Xiao, Toru Lin, Zhengyi Luo, Zhenjia Xu, Zhenyu Jiang, Jan Kautz, Changliu Liu, Guanya Shi, Xiaolong Wang, et al. Hover: Versatile neural whole-body controller for humanoid robots. In 2025 IEEE International Conference on Robotics and Automation (ICRA), pp. IEEE, 2025b.
Vincent Tao Hu, Wenzhe Yin, Pingchuan Ma, Yunlu Chen, Basura Fernando, Yuki M Asano, Efstratios Gavves, Pascal Mettes, Bjorn Ommer, and Cees GM Snoek. Motion flow matching for human motion synthesis and editing. arXiv preprint arXiv:2312.08895
2023.
Albert S Huang, Edwin Olson, and David C Moore. Lem: Lightweight communications and marshalling. In 2010 JEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 4057-4062. IEEE, 2010.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance normal- ization. In Proceedings of the IEEE international conference computer vision, pp. 1501-1510, 2017.
Mazeyu Ji, Xuanbin Peng, Fangchen Liu, Jialong Li, Ge Yang. Xuxin Cheng, and Xiaolong Wang. Exbody2: Advanced expressive humanoid whole-body control. arXiv preprint 13X1v:2412.13196, 2024.
Yuheng Ji. Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, et al. Robobrain: A unified brain model for robotic manipulation from abstract to concrete. Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1724-1734, 2025.
Biao Jiang, Xin Chen, Wen Liu, Jingyi Yu. Gang Yu, and Tao Chen. Motiongpt: Human motion as a foreign language. Advances in Neural Information Processing Systems, 36:20067-20079, 2023.
12

---

Published as a conference paper at ICLR 2026
Jihoon Kim, Jiseob Kim, and Sungjoon Choi. Flame: Free-form language based motion synthesis & editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 8255-8263, 2023.
Tanhong Li. Yonglong Tian, He Li, Mingyang Deng, and Kaiming He. Autoregressive image generation without vector quantization. Advances in Neural Information Processing Systems, 37: 56424-56445, 2024a.
Zhe Li, Zhangyang Gao, Cheng Tan, Stan Li, and Laurence T Yang. General point model with autoencoding and autoregressive. arXiv preprint arXiv:2310. 2023a.
Zhe Li, Laurence T Yang, Xin Nie, BoCheng Ren, and Xianjun Deng. Enhancing sentence rep- resentation with visually-supervised multimodal pre-training In Proceedings of the 31st ACM International Conference Multimedia, pp. 5686-5695, 2023b.
Zhe Li, Yisheng He, Lei Zhong, Weichao Shen, Qi Zuo, Lingteng Qiu. Zilong Dong, Laurence Tianruo Yang, and Weihao Yuan. Mulsmo: Multimodal stylized motion generation by bidirectional control flow. arXiv preprint urXiv:2412.09901 2024b.
Zhe Li, Laurence T Yang, Bocheng Ren, Xin Nie, Zhangyang Gao, Cheng Tan, and Stan Z Li. Mlip: Enhancing medical visual representation with divergence encoder and knowledge-gunde contrastive learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11704-11714 2024c.
Zhe Li, Weihao Yuan, Yisheng He, Lingteng Qiu, Shenhao Zhu, Xiaodong Gu, Weichao Shen, Yuan Dong, Zilong Dong, and Laurence T Yang. Lamp: Language-motion pretraining for motion generation, retrieval, and captioning. arXiv preprint arXiv:2410.07093, 2024d.
Zhe Li, Cheng Chi, Yangyang Wei, Boan Zhu, Tao Huang, Zhenguo Sun, Yibo Peng, Pengwei Wang, Zhongyuan Wang, Fangzhou Liu, al. Robomirror: Understand before you imitate for video to humanoid locomotion. arXiv preprint arXiv:2512.23649, 2025a.
Zhe Li, Cheng Chi, Yangyang Wei, Boan Zhu, Tao Huang, Zhenguo Sun, Yibo Peng, Pengwei Wang, Zhongyuan Wang, Fangzhou Liu, et al. Do you have freestyle? expressive humanoid locomotion via audio control. arXiv preprint arXiv:25 22..22302.22. 2025b.
Zhe Li, Weihao Yuan, Weichao Shen, Siyu Zhu, Zilong Dong, and Chang Xu. Omnimotion: Multimodal motion generation with continuous masked autoregression, 2025c. URL https
//arxiv. warxiv.orr//abs//5511 14954
Zhengyi Luo, Jinkun Cao, Kris Kitani, Weipeng Xu, al. Perpetual humanoid control for real-time simulated avatars. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10895-10904, 2023.
Nanye Ma, Mark Goldstein, Michael S Albergo, Nicholas M Boffi, Eric Vanden-Eijnden, and Saining Xie. Sit: Exploring flow and diffusion-ber based generative models with scalable interpolant transformers. In European Conference on Computer Vision, pp. 23-40. Springer, 2024.
Jiageng Mao, Siheng Zhao, Siqi Song, Chuye Hong, Tianheng Shi, Junjie Ye, Mingtong Zhang, Haoran Geng, Jitendra Malik, Vitor Guizilini, et al. Universal humanoid robot pose learning from internet human videos. In 2025 IEEE-RAS 24th International Conference Humanoid Robots (Humanoids), pp. 1-8. IERE, 2025.
Zichong Meng, Yiming Xie, Xiaogang Peng. Zeyu Han, and Huaizu Jiang. Rethinking diffusion for text-driven human motion generation. arXiv preprint arXiv:241 122411116775 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEFF/CVF international conference on computer vision, pp. 4195-4205, 2023.
Xue Bin Peng, Pieter Abbeel, Sergey Levine, and Michiel Van de Panne. Deepmimic: Example- guided deep reinforcement learning of physics-basee based character skills. ACM Transactions On Graphics (TOG), -144):1-14, 2018.
13

---

Published as a conference paper at ICLR 2026
Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 627-635. JMLR Workshop and Conference Proceedings, 2011.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov, Proximal policy optimization algorithms. arXiv preprint arXiv: (707:06777.06477,, 2017.
Agon Scrifi, Ruben Grandia, Espen Knoop, Markus Gross, and Moritz Bächer. Robot motion diffusion model: Motion generation for robotic characters, In SIGGRAPH asia 2024 conference papers, pp. 1-9, 2024.
Yiyang Shao, Xiaoyu Huang, Bike Zhang, Qiayuan Liao, Yuman Gao, Yufeng Chi, Zhongyu Li, Sophia Shao, and Koushil Sreenath. Langwbe: Language-directed humanoid whole-body control via end-to-end learning. arXiv preprint arXiv:2504. 2173004..11.... 2025.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
BAAI RoboBrain Team, Mingyu Cao, Huajie Tan, Yuheng Ji, Minglan Lin, Zhiyu Li, Zhou Cao, Pengwei Wang, Enshen Zhou, Yi Han, et al. Robobrain 2.0 technical report. arXiv preprint D7Xi0:2507.02029, 2025.
Guy Tever, Sigal Raab, Brian Gordon, Yonatan Shafir, Daniel Cohen-Or, and Amit H Bermano. Human motion diffusion model. arXiv preprint arXiv:2209. 14916, 2022.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Kleventh International Conference on Learning Represen- tations, 2023.
Shashank Tripathi, Lea Müller, Chun-Hao P Huang, Omid Taheri, Michael J Black, and Dimitrios Tzionas. 3d human pose estimation via intuitive physics. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4713-4725, 2023.
Takara Everest Truong, Michael Piseno, Zhaoming Xie, and Karen Liu. Pdp: Physics- based character animation via diffusion policy, In SIGGRAPH Asia 2024 Conference Papers, pp. 1-10, 2024.
Lixing Xiao, Shunlin Lu, Huaijin Pi, Ke Fan, Liang Pan, Yueer Zhou, Ziyong Feng, Xiaowei Zhou, Sida Peng, and Jingbo Wang. Motionstreamer. Streaming motion generation via diffusiin-based autoregressive model in causal latent space. arXiv preprint arXiv:2503. 15451, 2025.
Weiji Xie, Jinrui Han, Jiakun Zheng, Huanyu Li, Xinzhe Liu, Jiyuan Shi, Weinan Zhang, Chenjia Bai, and Xuelong Li. Kungfubot: Physics-based humanoid whole-body control for learning highly-dynamic skills. arXiv preprint arXiv:2506. 2tt.::5506..885,,, 2025.
Haoru Xue, Xiaoyu Huang, Dantong Niu, Qiayuan Liao, Thomas Kragerud, Jan Tommy Gravdahl, Xue Bin Peng, Guanya Shi, Trevor Darrell, Koushil Sreenath, et al. Leverb: Humanoid whole-body control with latent vision-language instruction. arXiv preprint arXiv:2506. 13751, 2025.
Junpeng Yue, Zepeng Wang, Yuxuan Wang, Weishuai Zeng. Jiangxing Wang. Xinrun Xu, Yu Zhang, Sipeng Zheng. Ziluo Ding, and Zongqing Lu. RI from physical feedback: Aligning large motion models with humanoid control. arXiv preprint arXiv:2506. 12769, 2025.
Jianrong Zhang, Yangsong Zhang, Xiaodong Cun, Yong Zhang, Hongwei Zhao, Hongtao Lu, Xi Shen, and Ying Shan. Generating human motion from textual descriptions with discrete representations. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14730-14740, 2023a.
Mingyuan Zhang, Xinying Guo, Liang Pan, Zhongang Cai, Fangzhou Hong, Huirong Li, Lei Yang, and Ziwei Liu. Remodiffuse: eetrievallaugmeene motion diffusion model. In Proceedings of the IEFF/CVF International Conference Computer Vision, pp. 364-373, 2023b.
14

---

Published as a conference paper at ICLR 2026
Yaqi Zhang, Di Huang, Bin Liu, Shixiang Tang, Yan Lu, Lu Chen, Lei Bai, Qi Chu, Nenghai Yu, and Wanli Ouyang. Motiongpt: Finetuned Ilms are general purpose motion generators. Proceedings of the AAAI Conference Artificial Intelligence, volume 38, pp. 7368-7376 2024.
Chongyang Zhong, Lei Hu, Zihao Zhang, and Shihong Xia. Att22:: Text-driven human motion gen- eration with multi-perspective attention mechanism. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 509-519, 2023.
Enshen Zhou, Qi Su, Cheng Chi, Zhizheng Zhang, Zhongyuan Wang. Tiejun Huang, Sheng, and He Wang. Code-as-monitor: Constraint-aware visual programming for reactive and proactive robotic failure detection. arXiv preprint arXiv:2412.04455 2.445.540555 2024.
Enshen Zhou, Jingkun An, Cheng Chi, Yi Han, Shanyu Rong. Chi Zhang, Pengwei Wang, Zhongyuan Wang, Tiejum Huang, Lu Sheng, et al. Roborefer: Towards spatial referring with reasoning in vision-language models for robotics. arXiv preprint 2rXiv:2506.0444 2025a.
Enshen Zhou, Cheng Chi, Yibo Li, Jingkun An, Jiayuan Zhang, Shanyu Rong, Yi Han, Yuheng Ji, Mengzhen Liu, Pengwei Wang, et al. Robotracer: Mastering spatial trace with reasoning in vision-language models for robotics. arXiv preprint arXiv:2512. 2025b.
15

---

Published as a conference paper at ICLR 2026
9
APPENDIX
APPENDIX OVERVIEW
This appendix provides additional details and results, organized follows:
Section 9.1
Detailed information of implementation, including detailed proprioceptive states, privileged information and hyper-parameters of network structure.
Elaboration on some details during training, including dataset details, motion filter and retargeting, simulator, domain randomization, regularization, reward functions, penalty and curriculm learning.
Section 9.2
Section
Details about evaluation, including metrics about motion generation and motion
tracking.
Section 9.4
Deployment details, including sim-to-sim and sim-to-real.
Section
9.5
Extra qualitative experiment results and visualizations, including in the simula-
tion and the real-world.
Section 9.6 Extra ablation experiment results and visualizations, including the effect of causal adaptive sampling (CAS), hyper-parameter X, and the number of experts in MoE of teacher policy.
Section 9.7 policy.
Pseudo codes of the algorithmic workflow and details of the diffusion student
9.1 IMPLEMENTATION DETAILS
In this section, we describe the state representation used for policy training, covering proprioceptive states, privileged information, and
of network. The proprioceptive state components for both the teacher and student policies are summarized Table 7 A key distinction is that the student policy utilizes a longer history of observations, compensating for its lack of access to privileged information by relying on extended temporal context.
For privileged information, the teacher policy incorporates privileged information to achieve precise motion tracking. The full set of privileged state features is provided in Table 7 Previous methods receive motion target information as part of the observation in both teacher and student policy, which includes keypoint positions, desired joint (DoF) positions, and root velocity. But in our method, only the teacher policy receives these information, and the student policy only receives the latent from motion generator. A detailed target state composition is presented in Table [8] The action output corresponds to target joint positions for mropottioalldderiaatii (PD) control, with a dimensionality of 23 for the Unitree G1 humanoid platform.
RoboGhost employs teacher-student training framework. The teacher policy, trained via standard PPO (Schulman et al. 2017). utilizes privileged information, tracking targets, and proprioceptive states. contrast, the student policy is trained using Dagger without access to privileged information, but instead relies an extended history of observations. Furthermore, while the teacher policy uses explicit reference motion information, the student policy replaces this with a latent representation from motion generator, resulting in a retargeting- -free and latent-driven approach. For the teacher policy, inputs are concatenated and processed by a learning. The student policy feeds its concatenated inputs conditions diffusion model with an MLP backbone. Furthermore, we employ AdaLN to inject conditional information throughout the denoising process within the diffusion model. final MLP layer is appended to the backbone network, which projects the output to a 23-dimensional action space while additionally incorporating the conditional signals. Detailed hyperparameters for both policies are provided in Table 9
(MoE) network for policy
9.2 TRAINING DETAILS
Dataset Details Our motion generator is pretrained the MotionMillion dataset (Fan et al. 2025). Given the substantial scale this dataset, we restrict pretraining to the humanml and kungfu subsets from the MotionUnio branch, comprising total of 50,378 motion sequences. During policy training,
16

---

Published as a conference paper at ICLR 2026
Proprioceptive States
Teacher Policy
State Component
Dim.
State Component
Dim.
DoF position
23
DoF position
23
DoF velocity
23
Keypoint position
36
Last Action
23
Root Velocity
3
Root Angular Velocity Roll
3
Root Angular Velocity
3
1
Roll
Pitch
1
Pitch
Yaw
Yaw
Total dim
75 10
Height
Privileged Information
Total dim
69
DoF Difference
23
Student Policy
Keybody Difference
36
Latent
64
Root velocity
3
Total dim
64
Total dim
62
Table 8: Reference information in the teacher and student policies.
Table 7: Proprioceptive states and privileged informa- tion.
Hyperparameter
Value
Optimizer
Adam W 0.9, 0.999 1 x 10-4
B1, .2
Learning Rate
Batch Size
4096
Teacher Policy
Discount factor (7)
0.99
Clip Parameter
Entropy Coefficient
0.005
Max Gradient Norm
Learning Epochs
Mini Batches
Value Loss Coefficient
Value MLP Size
[512, 256, 128]
Actor MLP Size
[512, 256, 128]
Experts
Student Policy
MLP Layers
MIPP Size
[256, 256, 256]
Table 9: Hyperparameters for teacher and student policy training.
we again draw data from the humanml and kungfu subsets. However, due to the presence of extensive duplicate and mirrored sequences, we perform a data cleaning process to remove redundancies and con-flat-ground motions. Using this filtered dataset, we first train an initial policy, denoted mo- This policy is then used to further filter the dataset by excluding sequences with a tracking error exceeding 0.6. After this refinement, the humanml subset contains 3,261 motion sequences, and the kungfu subset contains 200. And due to the significant domain gap and difference in complexity between these subsets, we opt train two separate policies, each specialized one subset.
Motion Filter and Retargeting Following Xie et al. 2025) and Tripathi et al. (202)). we compute
the ground- projected distance between the center of mass (CoM) and center of pressure (CoP) for each frame and apply stability threshold. Let p.OM = CoC Pr. and p.C (pfoP pCoP) denote
the 2D ground projections of CoM and CoP frame f, respectively. Let Ad, ||pCoM
pCoP ||
17

---

Published as a conference paper at ICLR 2026
We define the stability criterion of a frame Ad, < Csab A motion sequence will not be filtered if its first and last frames are stable, and the longest consecutive unstable segment shorter than 100.
Simulator Following established protocols in motion tracking policy research (Ji al. 2024 He
et al. 2025a Ji et al. 2025 Team et al. 2025), we also adopt Stree-stage evaluation pipeline: (1)
large-scale reinforcement learning training in IsaacGym; (2) zero-shot transfer to MuJoCo assess
cross-simulator generalization; and (3) physical deployment on the Unitree GI humanoid platform to validate the performance real-world.
Reference State Initialization Task initialization is critical factor in reinforcement learning (RL) training. We observe that naïvely initializing episodes from the beginning of the reference motion frequently leads to policy failure, especially for difficult motions. This can cause the environment overfit to easier frames and fail to learn the most challenging segments of the motion.
To mitigate this issue, we employ the Reference State Initialization (RSI) framework (Peng et al. 2018). Concretely, we sample time-phase variables uniformly over [0, 1]. randomizing the starting point within the reference motion that the policy must track. The robot's state-including root position, orientation, linear and angular velocities, well joint positions and velocities is then initialized to the values from the reference motion at the sampled phase. This approach substantially enhances motion tracking performance, particularly for highly-dynamic whole-body motions, by enabling the policy learn various segments of the movement in parallel rather than being limited to strictly sequential learning process.
Domain Randomization and Regularization To improve the robustness and generalization of the pretrained policy, we utilize the domain randomization techniques and regularization items, which are listed in Table IO
Domain Randomization
Term
Value
Friction
((..5.2.2) u0..75, 1.25) x default U(200, 40) ms
P Gain
Control delay
External Perturbation
Push robot
interval 8s, Ury 0.5 m/s
Regularization
Term Expression
Weight
DoF position limits (d, & [qmin 9max]
10
DoF acceleration ||d, 12
3 10-7
DoF error 1d. do13
0.1
Action rate ||a, a-11l Feet air time Tai - 0.5 Feet contact force || Stumble 1 1(F
0.5
10
-0.03
x F
-2
Waist roll pitch error [|p] ff
Pb 1
-0.5
Ankle Action ||atakle
0.3
Table 10: Domain randomization and regularization parameters.
Motion Tracking Rewards We define the reward function as the sum of penalty, regularization, and task rewards, which is meticulously designed to improve both the performance and motion realism of the humanoid robot. It consists of terms that incentivize accurate tracking of root velocity, direction, and orientation, well precise keypoints and joints position tracking. Inspired by ASAP (He ctt al. (2025)), we additionally incorporate regularization terms to enhance stability and sim-to-real transferability, including torques, action rate, feet orientation, feet heading, and slippage. In terms of
18

---

Published as a conference paper at ICLR 2026
penalty, adopt penalty terms for DoF position limits, torque limits, and termination. The detailed more terms can be seen in the supplementary material.
reward functions are summarized in Table
Task Reward
Root velocity
10.0
Root velocity direction
6,0
Root angular velocity
1.0
Keypoint position
10.0
Feet position
12.0
DoF position
6.0
DoF velocity
6.0
Penalty
DoF position limits
-10.0
Torque limits
55.0
Termination
-200.0
Table 11: Reward terms for pretraining
Curriculum Learning Training policies to track agile motions in simulation is challenging, as certain dynamic behaviors are too difficult for the policy to learn effectively from the outset. To mitigate this issue, we utilize termination curriculum (He et al.) (025a) that progressively tightens the motion tracking error tolerance during training, thereby guiding the policy to gradually improve its tracking fidelity. Initially, we set lenient termination threshold of 1.5m- episodess are terminated the robot deviates from the reference motion by more than this margin. As training progresses, the threshold is annealed down gradually, incrementally increasing the precision required for successful tracking. This curriculum enables the policy to first acquire basic balancing skills before refining them under stricter tracking constraints, ultimately facilitating the successful execution of highly dynamic behaviors.
9.3 EVALUATION DETAILS
Motion Generation Metrics Following (Guo et al., 2022c et al. 2024d b), we evaluate our approach using established text-to-motion metrics: retrieval accuracy (R@ @ R @3), Multimodal Distance (MMDist), and Fréchet Inception Distance (FID).
Retrieval Accuracy (R- C--Precision: These metrics measure the relevance generated motions to text descriptions in a retrieval setup. R@ denotes the fraction of text queries for which the correct motion is retrieved as the top match, reflecting the model's precision in identifying the most relevant motion. @@2 and R @3 extend this notion, indicating recall within the top two and three retrieved motions, respectively.
Multimodal Distance (MMDist): This quantifies the average feature-space distance between generated motions and their corresponding text embeddings, typically extracted via pretrained retrieval model. Smaller MMDist values indicate stronger semantic alignment between text descriptions and motion outputs.
Fr échet Inception Distance (FID): FID assesses the quality and realism of generated motions by comparing their feature distribution that of real motion data using pretrained feature encoder (c.g., an Inception-style network). It computes the Fréchet distance between multivariate Gaussian distributions fitted to real and generated motion features. Lower FID) scores reflect better distributional similarity and higher perceptual quality.
Diversity: Diversity quantifies the variance of motion sequences within the dataset. It is computed by randomly sampling Ndiver 300 pairs of motions, denoted (fr, f(.2) for each pair i. The metric is calculated as Nawer En || fx,2
Motion Tracking Metrics For motion tracking evaluation, we adopt metrics commonly used in prior work (Ji et al. 2024): Success Rate (Suce), Mean Per Joint Position Error (Empspe), and Mean Per Keybody Position Error (Empkpe).
Success Rate (Succ): This metric evaluates whether the humanoid robot successfully follows the reference motion without falling. A trial is considered failure if the average trajectory
19

---

Published as a conference paper at ICLR 2026
deviation exceeds 0.5 meters any point, if the root pitch angle passes predefined threshold.
Mean Per Joint Position Error (Empipe in rad): Empipe measures joint-level tracking accuracy by computing the average error in degrees of freedom (DoF) rotations between the reference and generated motion.
Empkpe assesses keypoint tracking performance based on the average positional discrepancy between reference and generated keypoint trajectories.
Mean Per Keybody Position Error (Empkpe in m):
9.4 DEPLOYMENT DETAILS
Sim-to-Sim As demonstrated in Humanoid-Gym (Gu al. 2024), MuJoCo exhibits more real- istic dynamics compared to Isaac Gym. Following established practices in motion tracking policy research (Ji et al. 2024), we perform reinforcement learning training in Isaac Gym to leverage its computational efficiency. To assess policy robustness and generalization, transfer to the MuloCo simulator. Finally, we deploy the policy on physical humanoid robot to evaluate the effectiveness of our RoboGhost framework for real-world motion tracking.
then conduct zero-shot
Sim-to-Real Our real-world experiments utilize a Unitree G1 humanoid robot, equipped with an onboard Jetson Orin NX module for computation and communication. The control policy takes motion-tracking targets as input, computes target joint positions, and issues commands to the robot's low-level controller at 50Hz. Command transmission introduces a latency of 18-30ms. The low-level controller operates at 500Hz to ensure stable real-time val-time actuation. Communication between the policy and the low-level interface is implemented using the Lightweight Communications and Marshalling (LCM) (Huang et al. 2010).
9.5 ADDITIONAL QUALITATIVE RESULTS
Qualitave Results in simluation and real-world further validate the effectiveness of our method, we provide additional qualitative results in this section, including motion tracking visualiza- tions in IsaacGym and MuJoCo (Fig and real-world locomotion demonstrations on the physical robot (Fig 6). Moreover, provide additional motion generation qualtative results in Fig 7 A supplementary video showcasing real-robot deployments is provided in the supplementary material.
9.6 ADDITIONAL. ABLATION STUDIES
Hy perparameeeers of Causal Adaptive Sampling To rigorously evaluate the effectiveness of causal adaptive sampling (CAS), we conduct various ablation studies examining both its performance gains and sensitivity to hyperparameters. As shown in Table [2] causal adaptive sampling increases the sampling probability of kinematically challenging frames within a motion sequence, thereby improving training efficiency and enhancing model generalization. Furthermore, as illustrated in Fig 8a and 8b optimal performance is achieved when the adaptation parameters are set to = 0.8 and p 0.005.
Number of Experts in MoE Furthermore, we conduct an ablation study on the number of experts in the MoE architecture, As illustrated in Fig the variation in the number of experts has measurable yet limited impact on the policy's performance, with the optimal result achieved when the number of experts set to 5.
Tracking Performance Against Tracking Policy To further validate the effectiveness of our policy,
we conduct comparisons with other tracking policies. Since the Baseline in Table 2 of the main paper is trained and evaluated on our dataset using the model architecture and settings of Exbody2 (Ti ctt al., 2024), we supplement here the test results the policy trained with the model architecture and settings of GMT (Chen et al. 2025). as shown in Table [3 The specific process is as follows: we train these policies on our dataset and evaluate their tracking performance on our test set. Since both are explicitly driven policies, the student policy takes reference motion input, first generate explicit motion via our motion generator, which then retargeted and fed to the corresponding student policy to obtain the final results.
20

---

Published as a conference paper at ICLR 2026
* ZN
V4P$
1" (EST (FW FRRIHO JSV JOB DRVEN N/PORO
1:1 PEFIOOO DIXA MEAROEARR WF (87.
1APNIRCE INNMORU CYEZ
311 GESITOIEI EES AVISE SPULF CN HILLFT TEEST
INPEST SXX BONZENN CEL
REDROSK 3 GRAS (RBCT
RE (FTIF PS AE TE (FT GRVFTT
Figure 5: Qualitative results in the IsaacGym and MuloCo.
IsaacGym
MuJoCo
Method
Suec f
Empipe
Empkpe
Suce t
Empipe
Empkpe
HumanML (MotionMillion)
w/o CAS
0.92
0.18
0.14
0.68
0.32
0.28
Ours
0.97
0.12
0.09
0.74
0.24
0.20
Kungfu (MotionMillion)
w/o CAS
0.62
0.42
0.41
0.48
0.67 0.54
0.63
Ours
0.72
0.34
0.31
0.57
0.50
Table 12: Ablation study on causal adaptive sampling.
Denoising Steps in Student Policy we test the inference time and tracking performance of DDIM sampling with 2, 4, 6, 8, and 10 steps. Notably, inference time is largely independent of motion sequences, so we only report the average time per step for deploying an action. From Table 14 it can be observed that increasing the number of sampling steps yields almost improvement in tracking performance, but significantly increases the time per step. This introduces inference latency on the real humanoid robot, thereby affecting deployment outcomes.
21

---

Published as a conference paper at ICLR 2026
Imput: person au upright position, performs squats, moving
Text Input: While boxing, the defends, dudges, then delivers few jubs..jubs.
Text Input: The man performing jumping Jacks.
Figure Qualitative results on Unitree GI.
IsaacGym
MuloCo
Method
Succ f Empipe +
Empkpe +
Suce Empipe
Empkpe
HumanML (MotionMillion)
Exbody2 (Ji
2024
0.92
0.23
0.19
0.64
0.34
0.31
GMT Chen et al. Ours
2025
0.95
0.15
0.12
0.73
0.25
0.22
0.97
0.12
0.09
0.74
0.24
0.20
Kungfu (MotionMillion)
Exbody2 (Ji et al. 2024
0.66
0.43
0.37
0.51
0.58
0.52
GMT Chen ct al. Ours
2025
0.70
0.36
0.33
0.54
0.56
0.52
0.72
0.34
0.31
0.57
0.54
0.50
Table 13: Motion tracking performance comparison against other tracking policies in simulation the Hu- manML Kungfu test sets.
Noise Scale in Student Policy We the inference time and tracking performance of DDIM sampling with 2, 4, 6, 8, and 10 steps. Notably, inference time is largely independent of motion sequences,
only report the average time per step for deploying action. From Table [5] can be observed that increasing the number of sampling steps yields almost no improvement in tracking performance, but significantly increases the time per step. This introduces inference latency on the real humanoid robot, thereby affecting deployment outcomes.
Noise Schedule Strategies in Student Policy We test the inference time and tracking performance of DDIM sampling with 2, 6, 8, and 10 steps. Notably, inference time is largely independent of motion sequences, so we only report the average time per step for deploying an action. From Table 16 it can be observed that increasing the number of sampling steps yields almost no improvement in tracking performance, but significantly increases the time per step. This introduces inference latency on the real humanoid robot, thereby affecting deployment outcomes. The success rate and latency curves presented in Figure 10
We conduct ablation experiments on the diffusion policy with respect to different supervision targets, including z0-prediction and e-prediction. As shown in Table adopting z0-prediction the optimization target for the diffusion policy yields significantly better tracking performance. Furthermore, we observe that higher feature dimensions lead to worse results when using e-prediction.
Optimization Objective in Student Policy
22

---

Published as a conference paper at ICLR 2026
DISPORN OUTT CARTLOON
ENNOOOMETOSLER BSE FECCRLL
8 CESPOONN BNT FORLY LARSL 8SS
BO 000 19S0
RERRIRO LRO WO. WO. DIVES
OESPOOLA FORN EPADIN GUHBLLE
BOSEAAA L2 BE XPCLMI
AUG L. GAOERE POL BEV UNLLSSE
183 1P13 HREE ICE ROMINX BIDLE
UFFESPOON
BNBOXDE USUEE BEE URO
DISTIOO GESUSE
UT (ESPOO 87 17 CPN (F
OLL PRCHEMMMTT
KINNIITI OF NO WOON DOFRR
Figure Qualitative results of motion generator.
KsaeCyym
HPJPT MOXFE
0.18
NKKZE
0.16
0.25
Ereer (m)
8.10
0.29
0.5
0.6
0.7
0.003
0.004
0.005
3,006
0.007
(a) Isaac Gym
(b) MuJoCo
Figure 8: The impact of different values on tracking performance in IsaacGym and MuJoCo.
Retargeting Methods and Steps We have conduct the time consumption and tracking performance for different PHC step counts, as presented in Table can observed that reducing the number of PHC steps indeed lowers the time cost, but simultancously leads to decline in tracking performance. During the retargeting of raw data, we find that the data quality is extremely poor when the number of PHC retargeting steps insufficient. In addition, evaluate various metries of the GMR (Araujo
23

---

Published as a conference paper at ICLR 2026
Tacking Performance in IssacGym
Trccing Parformance MujoCo
NAPE N
NHPE NEKPE
0.14
0.25
0.13
0.24
Error (m)
0.32
0.25
0.11
0.09
(a) IsaacGym
(b) MuloCo
Figure 9: The impact of different number of experts in MoE on tracking performance in IsaacGym and MuloCo.
Method
Avg Time (s) x 10
Succ 1
Empipe J
Empkye 1
DDIM-2 sampling
4.8
0.97
0.12
0.09
DDIM-4 sampling
10.7
0.97
0.12
0.09
DDIM-6 sampling
13.2
0.97
0.12
0,09
DDIM-8 sampling
17.8
0.97
0.11
0,09
DDIM-1 10 sampling
18.1
0.97
0.11
0.08
Table 14: Average inference time and tracking performance different DDIM sampling steps.
et al. 2025) during the testing phase, as shown in Table 18
But we observe frequent joint mutations and distortions for certain martial arts motions when we use GMR to retarget the reference motions.
Latent from Different Motion Generators We conduct ablation studies on two aspects: the impact of latent representations from different frozen motion generators policy performance, and the effects of finetuning these motion generators on their motion generation metrics as well as policy performance. The settings are as follows: we used three pretrained motion generators: MLD (Chen) et al., 2023). MoMask (Guo et al., 2023), and T2M-GPT (Zhang et al., 2023a) as generators of motion latents, and tested two scenarios (finetune and frozen) respectively, evaluating both tracking performance and motion quality metrics on HumanML subsets. As shown in Table we observe that our motion generator achieves the best tracking performance. Furthermore, when we finetune the motion generator, it tends to produce more physically plausible actions, which in turn may lead to decline generation metrics, such as MoMask and T2M-GPT. We analyze that the reason may lie in discrete transformer-basee motion generators, which
the fact that both MoMask and T2M-GPT need to extract embeddings from the codebook as latents. Specifically, the latents in the codebook a predefined set of discrete vectors, and during finetuning, the motion generator or combine from the existing discrete candidates. To adapt to the student policy, it may excessively but incomplete in motion expression; secondly,
only select
tend choose latents that distillation-friendd the number of latents in the codebook is limited, and during finetuning, the motion generator may memorize small number of latent combinations preferred by the student policy instead of learning
Time vs. Denosing Steps
Success Rate vs. Time vs. Sampling Strategles
Suce Rate vs. Noise Scale
Time
mm))
Surcesse
Denotsing Steps
Sampling Strategy
Noise Scale
69
Figure 10: Success rate latency curves on different settings diffusion policy.
24

---

Published as a conference paper at ICLR 2026
Noise Scale (amax)
Denoising Steps
Success Rate (%)
Latency (s x x10-3
0.10
2
97.0
4.7
0.15
97.0
4.7
0.20
97.0
4.8
0.25
2
96.0
4.9
0.30
2
94.0
5.0
seltings: denoising sleps, sampling (1y=0), Anx maximum Ar training timesteps.
Table 15; Fine-graines ablation on noise scale.
Sampling Strategy
Denoising Steps
Success Rate (%)
Latency (s x10" 3)
DDIM (rz=)
2
97.0
4.8
DDIM (170.55
2
90.0
4.8
DDPM (Stochastic)
2
72.0
7.7
Note: Fixed settings: cosine noise schedule (0,,0x)=0220),, controls the stochasticity of DDIM sampling.
Table Fine-grained ablation on different sampling strategies
generalized motion laws. This may result in only few embeddings in the codebook being adjusted, leading to the degradation of generation performance.
9.7 PSEUDO CODE
To more clearly present the algorithmic workflow and details of the diffusion student policy, we have provided pseudo code for both the student policy training and the internal operations of the diffusion process, which have been added Algorithms I and in the appendix. Here, elaborate the parameters of each component:
The action output frequency of the teacher policy is consistent with that of the student policy. Specifically, during training, the teacher policy needs to be forward-propagated synchronously to generate a supervised action, which is used to supervise the action output by the student policy.
The supervision target of the diffusion process is x0-prediction. Initially, we also attempted to use e-prediction as the optimization target, but its performance was inferior to that of x0-prediction.
The noise scheduling adopts a cosine schedule, with 50 timesteps during training. The noise scale for each timestep is as follows: For the cosine noise schedule with num diffusion timesteps 50, the noise scale (B,) for each timestep is computed as follows:
The cumulative product of (1 3) up normalized timestep & defined by:
t
0.008
π
2
= cos
1.008
2
where e i] the normalized timestep, calculated as For each timestep (0 Si < 49), the noise scale is derived as:
3 for timestep index E 10..,...,,4
3 min
0.999
where t 50 denotes normalized timestep for current step i), t2 = {+1 denotes normalized timestep for next step 1, 07, and a; are the cumulative retention factors and fz. The upper bound 0.999 avoids numerical singularities.
50
The conditions consist of proprioceptive information, historical observations, and motion latent representations. Since we use x0- 40-prediction as the optimization target the diffusion process, the xO--rredittonn loss be directly employed as the distillation loss.
25

---

Published as a conference paper at ICLR 2026
Supervision Target
Suce
Empipe +
Emphpe +
e-prediction
0.79
0.50
0.47
=00prediccion
0.97
0.12
0.09
Table Tracking performance on different optimization ohjectives.
Method
Time (s)
Suce f
Empipe
Empkeee
GMR
1.41
0.92
0.23
0.18
PHC-100
1.63
0.81
0.45
0.40
PHC-500
6.09
0.88
0.31
0.25
PHC-800
0.91
0.25
0.21
PHC-1000
11.89
0.93
0.21
0.17
Table 18: Average inference time and tracking performance different retargeting methods.
Method
Top 31
FID
MM-Dist
Suce
Empssee
Emppkee
Frozen
MLD
0.792
18.236
16.638
0.88
0.28
0.25
T2M-GPT
0.838
12.475
16.812
0.92
0.20
0.17
MoMask
0.846
12.232
16.138
0.95
0.16
0.14
Ours
0.867
11.706
15.978
0.97
0.12
0.09
Finetune
ML.D
0.815
17.652
16.414
0.90
0.25
T2M GPT
0.829
12,576
16.927
0.94
0.18
0.15
MoMask
0.849
12.376
16.233
0.96
0.14
0.12
Ours
0.878
11.529
15.884
0.98
0.09
0.07
Table 19: Comparison of motion generation metries and tracking performance under frozen vs. finetuned different motion generators.
26

---

Published as a conference paper at ICLR 2026
Algorithm 1: DAgger Data Collection and Aggregation for Diffusion Student Policy
Input: Teacher policy Teacherr Student policy sstudents Environment &, Distillation config $2, Observation dimensions
Output: Rollout buffer B
Initialization;
Initialize replay buffer
obs (including motion latent) oeeeet_osservationn:::
obspist - _gget_extee extra hist hist_obss))
Initialize reward and episode length trackers; for äer to Q.total iter do
Training Phase (Teacher Rollout); for step = to S2.num_ster sit__sttespp______
cacpssper___ee per_env(24) do
Aggregate observations with history for buffer B;
Sample Student Action:
actiongauueat Ttuudent - sample([obs,, obs hist]); Environment step with student action actionaudent
Update episode statistics and historical observations; end
DAgger Data Collection Phase:
for step 0 to Q.num mini batches (4) do
Prepare observations in the Rollout Buffer: obsgudent B. B.pop(;;
Prepare observations for teacher:
=bSieachr obs -
Query teacher for expert actions:
actiongaacher Ttaacer (ObSreacherr;
Student generates actions using diffusion policy (xp predicitiio):
action studeat (obsstudemt):
Compute Distillation Loss:
LDAgger MSE MSE aantinnnonnnt actionneacher)
end end
return 8
27

---

Published as a conference paper at ICLR 2026
Algorithm 2: Diffusion Policy with Cosine Schedule and x0- Po-Pridict
Input:
Training dataset D {obs, obShist; 2, a")), Precomputed cosine noise schedule {ax, )
Training diffusion steps 7 = 50, Inference DDIM steps S 2, Condition encoder Econd. Distillation config
Output: Trained diffusion policy Taiff
Initialization:
Initialize diffusion model M;
Optimizer 0 AdamW( Addmm(,.ppaammtteess) Ir=10-4, weight decay=10-
Training Phase;
for epoch
for batch in Sunmm_ppochs do {(obs,, obs 24, }})} < batch data;
sizes_sieeeeee do
// Step 1: Condition fusion and AdaLN injection
cond, + cobsat(obs,, obSnis,;; 2;) hond_emb; +- (cond))
// Step 2: Sample random diffusion timestep and Gaussian noise
8 + U N(0,,)
// Step 3: Forward diffusion (corrupt teacher action with
noise)
at -aya aa VI-ai
1/ step 4: Model predicts original action ap
(x0 prediction)
âg - M(ar, 4, cond_emb;)
// Step
Compute training loss
L + MSE(@g.a;) at)
// Step
Backpropagation and parameter update
VoL - Cl max_mmapooo norm = 1.0); 0 - 0 - Ir. VoL:
end
end
Inference Phase (2-Step DDIM Sampling);
// DDIM sampling pipelin happliin inne obs, obs +- Input sensory and latent signals;
// Step 1: Encode conditions
cond +- concat(obs, obsmaa:
cond_en + (cond)
// Step 2: Initialize with random noise
as N
// Step 3: 2-step deterministic denoising
for s S do
t + TimestepMap(s, S, åo f- MMaa, cond_emb)
// DDIM update rule (π) for deterministic sampling)
+ GedPrecomputedAlppaatt 1) Eapprox &-
aa Ra- Ra-yarin ao
a--1 f- v åg + vI at-1 Eapproxi
end
return
28


Reviews:
[{"review_id": "y5LS4ZZuY4", "reviewer": "Reviewer_yDs5", "summary": "This paper presents RoboGhost, a novel framework for language-guided humanoid locomotion that eliminates the explicit motion retargeting step. The authors identify that conventional multi-stage pipelines (text-to-motion, decoding, retargeting, tracking) suffer from high latency and cumulative errors. RoboGhost addresses this by proposing a latent-driven approach. A motion generator, based on a hybrid transformer-diffusion architecture , produces a motion latent from a text prompt. This latent directly conditions a diffusion-based student policy , which is trained to denoise executable actions using supervision from an MoE-based teacher policy. This retargeting-free pipeline is shown to significantly reduce deployment latency and improve tracking success rates in both simulation (IsaacGym, MuJoCo) and on a real Unitree G1 humanoid.", "questions": "Could the authors provide a more detailed analysis, or ideally, a direct comparison, of the real-world deployment performance between the Diffusion Policy and a conventional RL-Distilled Student Policy (e.g., a simple, optimized MLP policy trained to imitate $\\pi_t$ using non-privileged observations)?\n\nSpecifically, I am interested in the practical trade-offs introduced by the Diffusion architecture upon deployment:\n\n1. Inference Latency/Throughput: Does the iterative nature of the Diffusion sampling (e.g., DDIM steps) impose a practical latency penalty (higher per-step inference time) on the real robot compared to a single forward pass of a distilled MLP policy? If so, is this increased latency a limiting factor for high-frequency control?\n2. Real-World Benefits: Beyond the simulation-based generalization shown in Table 4, what specific additional benefits (e.g., smoother motion, greater robustness against sensor noise, better handling of physical disturbances due to the mode-covering nature of diffusion) does the Diffusion Policy provide in the real-world deployment that justifies its computational complexity over a simpler, conventionally distilled policy?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. Novel and Elegant Framework: The primary strength is the \"retargeting-free\" pipeline. Instead of the conventional \"generate-retarget-track\" approach, RoboGhost proposes a \"generate-control\" paradigm. Using motion latents as a direct conditioning signal for a diffusion-based action policy  is a effective idea. It tightly couples semantic intent from language with low-level control, addressing the \"weak coupling\" problem of prior work.\n2. Technically Sound and Thorough: The work is technically solid. The two-stage teacher-student architecture is well-designed, featuring a strong MoE-based teacher for robust supervision and a novel diffusion-based student for generalization. The inclusion of Causal Adaptive Sampling (CAS) to focus on difficult motion segments  further strengthens the training methodology.\n3. Real-World Relevance and Efficiency: The reported reduction in deployment latency (from 17.85s to 5.84s) and improved success rates (+5%) are compelling for real-time robotic applications. The sim-to-real transfer without manual tuning is particularly impressive.", "weaknesses": "1. Limited Scope of Baseline Comparisons: The paper's primary evidence for the superiority of its latent-driven framework stems from the comparison in Table 3, which contrasts \"Ours-Implicit\" against \"Ours-Explicit\" (an author-implemented traditional pipeline with retargeting). While this serves as an effective internal ablation study that validates the advantages of the latent-driven paradigm over the explicit one, it is insufficient for fully contextualizing the work. The paper lacks a direct, end-to-end performance comparison against other external, published, state-of-the-art pipelines for language-guided humanoid control. It is unclear how RoboGhost would perform against a system with a highly optimized SOTA tracking policy, such as GMT (Chen et al., 2025) or ExBody2 (Ji et al., 2024).\n2. Strong Dependency on the Initial Motion Dataset and Limited Extrapolation: The effectiveness of the entire RoboGhost framework, from the semantic content of the motion latent ($l_{ref}$) to the capability of the MoE Teacher Policy ($\\pi_t$), is fundamentally bound by the diversity and quality of the initial human motion dataset used for training. While the paper demonstrates success on provided motions, its ability to genuinely handle open-ended language is questionable when commands necessitate novel combinations or extrapolation far outside the training distribution (e.g., highly stylized, physically novel, or long-horizon composite actions). A failure in the dataset translates directly into a failure of the Stage 1 latent generator to encode valid semantics, thereby constraining the overall potential of the open-ended language interface.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "neUj08H8W4", "reviewer": "Reviewer_eaYv", "summary": "\"From Language to Locomotion: Retargeting-free Humanoid Control via Motion Latent Guidance\" simplifies the deployment stack of text to humanoid robot locomotion. Whereas prior approaches generate human motion then retarget onto a robot, this method learns motion representations that can be decoded to robot action directly. This improves inference latency and removes error-prone components of existing pipelines. The authors demonstrate real-world deployments on a Unitree G1 which show physically stable and semantically plausible robot actions corresponding to various text prompts.", "questions": "I see that prior works call their motion FD metric FID, but I find this misleading since an Inception network is not used. I think it's good to clarify this and potentially define a new term for this metric.\n\nNote: typo at line 259", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This method tackles a real, error-prone piece of prior works' deployment pipelines. The proposed solution is simple and practical, reducing inference latency and compounding errors.\n- Most aspects of the method are well-justified and follow conventions from appropriate subfields. For example, AdaLN is a well-tested conditioning scheme for image diffusion models.\n- Real world robot deployment clearly works and generates locomotion that resembles the input prompt.\n- Paper is well-written and the components of the method are clear.", "weaknesses": "- Mixture of Experts is simply a compute efficiency optimization for transformers to maintain inference cost while scaling parameter count. I appreciate the expert count ablation in the appendix, but the obvious comparison is simply making the model larger. I find this part of the method poorly justified.\n- There is minimal analysis of the diffusion model sampling latency and tradeoffs with increasing/decreasing the number of sampling steps. Many existing diffusion distillation methods could reduce the number of NFEs (neural function evaluations) necessary at test-time as well.\n- Since the authors propose a new motion representation learner, it would be helpful to probe those representations to understand them better.\n- It's unclear to me why the text should be encoded through LaMP instead of just using a generic text embedding model like T5. For example, Stable Diffusion 3 moved from CLIP encoding to T5 encoding for text.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "zfMH3062SW", "reviewer": "Reviewer_vQfN", "summary": "This paper proposes RoboGhost, a retargeting-free framework for language-guided humanoid locomotion. The key idea is to bypass motion retargeting by using language-conditioned motion latents to guide a diffusion-based humanoid policy. The system follows a teacher–student distillation scheme where the teacher policy is trained with retargeted motion data and privileged simulation information, and the student policy learns to act directly from latent representations, eliminating explicit retargeting during deployment. Experiments show improved inference speed and comparable or better motion-tracking accuracy compared to retargeting-based baselines.", "questions": "- Since the student policy is distilled from a teacher trained with retargeted data, how can it outperform the retargeting-based pipeline? What factors (e.g., noise injection, diffusion regularization) contribute to this paradoxical result?\n- What exactly constitutes the ground truth for evaluating the motion tracking policy?\n- Can you provide a stage-wise breakdown of inference time (e.g., motion generation, diffusion denoising, policy forward) to show where the latency reduction originates?\n- How does RoboGhost compare to LeVERB and UH-1 in terms of architecture, goal, and performance?\n- Could the authors include additional ablations on PHC iterations and different retargeting methods to substantiate the efficiency advantage?", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- **System design clarity:** The two-stage training (teacher-student) and latent-driven diffusion policy are clearly described, supported by architectural and algorithmic details.\n- **Efficiency gains:** The results show a substantial improvement in inference time.\n- **Broad applicability:** The framework generalizes to different modalities (text, image, audio), providing a foundation for multimodal humanoid control.", "weaknesses": "- **Motivation and logic gap:**\nThe core motivation is to bypass retargeting, yet the teacher policy still relies on retargeted motion data. Hence, the student’s performance remains bounded by retargeting quality. Logically, distilling from a retargeted teacher should not outperform the original retargeted paradigm. However, the results show the opposite. The paper lacks a clear explanation of how or why the student surpasses its teacher, which raises questions about evaluation fairness or metric alignment. The main advantage of retargeting-free I acknowledge here is it can improve inference speed. But there is only very limited experiments on this. No break down on which module cost the most times, as well as lack comprehensive study on different retargeting methods (e.g. GMR [1]), and different iteration times of PHC.\n\n- **Experiments:** \nThe paper evaluates motion generation metrics (e.g., FID, Diversity) even though the final task is humanoid control, making these metrics less meaningful. Most performance improvements (e.g., success rate, tracking error) are minor; the main advantage comes from inference speed. And no real-world or simulation qualitative comparison which is more intuitive to see whether this method is useful.\n\n- **Evaluation inconsistency:**\nThe “Evaluation of Motion Tracking Policy” section uses “ground truth” motions that seem derived from retargeted data. If so, evaluating a non-retargeted policy against a retargeted reference introduces bias and makes interpretation difficult. The authors should clarify where the ground truth comes from and justify why this comparison is meaningful.\n\n- **Limited comparative discussion:**\nNo discussion and comparison with very related works, LeVERB [2] and UH-1 [3], which also pursue latent-based control or retargeting-free. A direct comparison or discussion of conceptual differences is needed.\n\n- **Presentation issues:** \nTable 1 (MM-Dist) highlights the wrong value.\n\n[1] https://github.com/YanjieZe/GMR\n\n[2] LeVERB: Humanoid Whole-Body Control with Latent Vision-Language Instruction\n\n[3] Learning from Massive Human Videos for Universal Humanoid Pose Control", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "HT6xjQIIGy", "reviewer": "Reviewer_3P5o", "summary": "This paper introduces RoboGhost, a framework for language-guided humanoid locomotion that eliminates the need for conventional motion retargeting. Instead of decoding explicit motion sequences and manually adapting them to specific robot morphologies, RoboGhost leverages language-conditioned motion latents as direct control signals to a diffusion-based policy. The method combines a causal transformer-based motion latent generator with a teacher-student policy setup (using a Mixture-of-Experts RL teacher and a diffusion-based student policy) to enable retargeting-free, semantically guided robot control. Extensive experiments across simulation and a real humanoid platform (Unitree G1) demonstrate improved efficiency (lower latency), tracking, and robustness compared to traditional pipelines.", "questions": "1. The paper shows that implicit retargeting with distillation outperforms a PHC-style teacher, but it does not compare against **state-of-the-art explicit retargeting approaches** (e.g., ProtoMotions, GMR) that report substantial tracking gains from carefully engineered retargeting; please add these as strong baselines under matched settings.\n2. Please justify the need to **learn your own human motion latent space** rather than using a pretrained motion latent (e.g., MLD, MotionGPT, PULSE, SMAP) by providing swap-in experiments where your latent is replaced with these pretrained alternatives (both frozen and fine-tuned) and comparing not only generation metrics but also control-side metrics.\n3. The **diffusion student policy** section is ambiguous (e.g., “noise scale and the MDP share a time step $t$”); please provide a clear **algorithm box or pseudocode** detailing the DAgger data-collection and aggregation procedure (rollout scheme, teacher query frequency, buffer management), the supervision target ($\\epsilon$ v.s. $x_0$ prediction), the noise schedule and time-step sampling, conditioning injection (history/ proprioception/ latent), and the online distillation loss composition and stabilization tricks, and add fine-grained ablations on denoising steps, noise scale, sampling strategy, and teacher query ratio with success-rate and latency curves.\n4. Current baselines are predominantly **human motion generation methods**; to substantiate advantages for humanoid motion generation/control, please include or align representative **humanoid-specific baselines** on the same tasks and metrics and discuss where your method excels.\n5. Please specify exactly how $E_{mpjpe}$ is computed in Table 3 Implicit Tracking, and clarify the conditioning: is the student driven solely by the text-derived latent from the HumanML/Kungfu test captions while errors are measured against the paired dataset reference motion? If not, please detail the source of the latent and the reference used for scoring.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The paper convincingly argues for eliminating the retargeting step in language-to-motion pipelines, highlighting both latency and cumulative error benefits.\n- The paper proposes a thoughtful shift in control pipeline design by the hybrid of a causal transformer for latent generation and a diffusion policy for action denoising .\n- Multiple quantitative tables provide a comprehensive performance landscape. Results show clear improvements in motion generation quality (precision, FID), tracking accuracy, and especially latency (17.85s → 5.84s).", "weaknesses": "- While the student policy avoids decoding/retargeting at deployment, the teacher still tracks explicit reference motions with privileged info, then the student distills from that oracle. This leaves open whether gains come mainly from inference-time design or from the strong teacher and filtering/curation choices.\n- The central claim is a retarget-free latent-to-policy pipeline for humanoid control, but the current experimental evidence only partially targets this claim; stronger head-to-head comparisons with explicit-retargeting SOTAs, swap-in pretrained latents on control metrics, and detailed diffusion-student ablations are needed to firmly establish that the retarget-free design—not ancillary choices—drives the gains.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)