cagataydev/protomotions-gtp-unitree-g1

ProtoMotions GTP — Unitree G1 Universal Motion Tracker

A generalized text-conditioned motion tracker (GTP) ONNX model that converts kinematic reference motion trajectories into PD-actuator control targets for the Unitree G1 humanoid (29 DoF, 33 bodies). Trained with the BeyondMimic-style RL pipeline from NVIDIA GEAR's ProtoMotions framework.

This is a tracker, not a text-to-motion generator. Pair it with a kinematic motion source like nvidia/Kimodo-G1-RP-v1 (text → motion diffusion) to get a full text → physics pipeline.

---

🎬 Verified end-to-end (Kimodo → GTP → MuJoCo, 2026-08-14 on NVIDIA Thor)

Prompt: "a person walking forward with confident strides"

Pipeline:

Kimodo diffusion (100 steps, 11 it/s)         → 8.0s
  → 120 frames @ 30Hz kinematic ref
  → SLERP retime → 199 frames @ 50Hz
  → ProtoMotions GTP ONNX (this model) @ 50Hz  → 1.2s
  → G1 physics @ 1kHz (MuJoCo)                 → 3.2× realtime
  → 100 rendered frames @ 25fps                → ~2s

Result: G1 walked forward, physics-stable, zero-shot on unseen prompt.

---

Files

File	Purpose
unified_pipeline.onnx	The tracker (22 MB) — 8 inputs, 4 outputs (see below)
unified_pipeline.yaml	Model+robot+control config (joint order, PD gains, timing)

---

Interface

Control rate: 50 Hz (control_dt=0.02s) Physics rate: 1000 Hz (physics_dt=0.001s, decimation=20) Anchor body: torso_link (index 16 of 33) Root body: pelvis (index 0)

Inputs (8)

Name	Shape	Kind
current_anchor_rot	(1, 4)	quaternion xyzw of torso_link
current_dof_pos	(1, 29)	joint positions (rad)
current_dof_vel	(1, 29)	joint velocities (rad/s)
current_root_local_ang_vel	(1, 3)	pelvis ang-vel in local frame
historical_processed_actions	(1, 1, 29)	previous action (PD target rescaled)
mimic_future_anchor_rot	(1, 4, 4)	reference torso rot @ steps [1,2,4,8]
mimic_future_dof_pos	(1, 4, 29)	reference joints @ steps [1,2,4,8]
mimic_future_dof_vel	(1, 4, 29)	reference joint vels @ steps [1,2,4,8]

Future step lookahead: [1, 2, 4, 8] control steps = [20, 40, 80, 160] ms.

Outputs (4)

Name	Shape	Meaning
actions	(1, 29)	Raw policy output (before PD conversion)
joint_pos_targets	(1, 29)	PD position targets — feed to G1 actuators
stiffness_targets	(1, 29)	Per-joint PD stiffness (kp)
damping_targets	(1, 29)	Per-joint PD damping (kd)

Control mode: BUILT_IN_PD — the tracker outputs per-step Kp/Kd/target, then the PD controller runs at 1 kHz between control steps.

---

Joint order (29 DoF)

0-5   left_hip_[pitch,roll,yaw]_joint, left_knee, left_ankle_[pitch,roll]
6-11  right_hip_[pitch,roll,yaw]_joint, right_knee, right_ankle_[pitch,roll]
12-14 waist_[yaw,roll,pitch]_joint
15-21 left_shoulder_[pitch,roll,yaw], left_elbow, left_wrist_[roll,pitch,yaw]
22-28 right_shoulder_[pitch,roll,yaw], right_elbow, right_wrist_[roll,pitch,yaw]

Compatible with the g1_29dof_rev_1_0 URDF/MJCF variant (torso waist_yaw+roll+pitch, wrists roll+pitch+yaw, no hand fingers — rubber hand end-effectors).

MJCF reference: mjcf/g1_holo_compat.xml

---

Body order (33 bodies)

0  pelvis
1  head
2-7  left leg: [hip_pitch, hip_roll, hip_yaw, knee, ankle_pitch, ankle_roll]_link
8-13 right leg (mirror)
14   waist_yaw_link
15   waist_roll_link
16   torso_link       ← anchor
17-23 left arm: [shoulder_pitch, shoulder_roll, shoulder_yaw, elbow, wrist_roll, wrist_pitch, wrist_yaw]_link
24   left_rubber_hand
25-31 right arm (mirror)
32   right_rubber_hand

---

Quickstart (Python)

import numpy as np, onnxruntime as ort, yaml

sess = ort.InferenceSession("unified_pipeline.onnx", providers=["CPUExecutionProvider"])
cfg  = yaml.safe_load(open("unified_pipeline.yaml"))

# Build observation from your robot state + kinematic reference motion
obs = {
    "current_anchor_rot":           np.zeros((1, 4),      dtype=np.float32),   # torso quat
    "current_dof_pos":              np.zeros((1, 29),     dtype=np.float32),   # rad
    "current_dof_vel":              np.zeros((1, 29),     dtype=np.float32),   # rad/s
    "current_root_local_ang_vel":   np.zeros((1, 3),      dtype=np.float32),
    "historical_processed_actions": np.zeros((1, 1, 29),  dtype=np.float32),
    "mimic_future_anchor_rot":      np.zeros((1, 4, 4),   dtype=np.float32),   # 4 lookahead steps
    "mimic_future_dof_pos":         np.zeros((1, 4, 29),  dtype=np.float32),
    "mimic_future_dof_vel":         np.zeros((1, 4, 29),  dtype=np.float32),
}

actions, joint_targets, stiff, damp = sess.run(
    ["actions", "joint_pos_targets", "stiffness_targets", "damping_targets"],
    obs,
)
# → feed joint_targets/stiff/damp to G1 PD controller at 50 Hz

---

Text → Motion → Physics pipeline (with Kimodo)

# 1) Generate kinematic reference from text (nvidia/Kimodo-G1-RP-v1)
from diffusers import DiffusionPipeline  # or Kimodo's native API
motion = kimodo.generate("a person walking forward with confident strides")
# motion: (120, 3+29) root_pos + joint_pos at 30 Hz

# 2) Retime to 50 Hz (SLERP for rotations, linear for positions)
motion_50hz = slerp_retime(motion, from_fps=30, to_fps=50)   # → (199, 32)

# 3) Roll out through GTP tracker in MuJoCo (see full script link below)
for step in range(len(motion_50hz)):
    obs = build_obs(mj_data, motion_50hz, step, lookahead=[1,2,4,8])
    _, joint_targets, kp, kd = sess.run(None, obs)
    apply_pd(mj_data, joint_targets, kp, kd)
    for _ in range(20):  # 20× physics substeps → 1 kHz
        mujoco.mj_step(mj_model, mj_data)

---

Training

• Framework: ProtoMotions (NVIDIA GEAR)
• Method: BeyondMimic-style RL tracking (imitation + PD residual)
• Simulator: IsaacLab / MuJoCo Warp
• Checkpoint: exps/exp-20260306_110148/last.ckpt
• Reference paper: BeyondMimic — arXiv:2408.07295

---

Integration status

• ✅ MuJoCo — verified via kimodo-g1-e2e end-to-end script (walk, boxing, squats, jumping-jacks generated on Thor)
• 🚧 strands_robots — first-class KimodoPolicy provider prototype: cagataycali/robots @ feat/kimodo-mujoco-integration
• 🚧 Real hardware (Unitree G1 via cyclonedds LowCmd) — design filed at robots-harness#271

---

Citation

If you use this tracker, please cite ProtoMotions + BeyondMimic:

@article{beyondmimic,
  title={BeyondMimic: From Motion Tracking to Versatile Humanoid Control via Guided Diffusion},
  year={2024},
  eprint={2408.07295},
  archivePrefix={arXiv},
}
@misc{protomotions,
  author={NVIDIA GEAR},
  title={ProtoMotions},
  howpublished={\url{https://github.com/NVlabs/ProtoMotions}},
}

---

License

Apache-2.0 (matches ProtoMotions upstream). Robot assets (Unitree G1 MJCF) subject to Unitree's own license.

---

Contact

• Maintainer: @cagataycali
• Uploaded from NVIDIA Thor (Jetson Thor Dev Kit) as part of the strands-labs/robots research stack.