groot_dex_warehouse_ae

GR00T N1.7 action-expert fine-tune on Kasra99/dex-warehouse, a teleoperated warehouse pick-and-place dataset recorded on a Dexmate Vega 1 Pro mobile manipulator.

ae = action expert: the Cosmos-Reason2 / Qwen3-VL backbone (LLM + vision tower) is frozen; the flow-matching action head is trained.

What was trained

Base weights	nvidia/GR00T-N1.7-3B
Embodiment tag	new_embodiment (projector slot 10)
Trainable	projector 327M + DiT 1.09B + vlln/vl-attn 201M = 1.62B / 3.14B (51.5%)
Frozen	LLM 1.12B + vision tower 407M
Steps / batch	40,000 / 32 (5.8 epochs over 219,260 frames)
Optimizer	AdamW, lr 1e-4, cosine, warmup
Chunk / action steps	40 / 40 (N1.7 native horizon)
EMA	constant decay 0.99 (weights published are the EMA weights)
Augmentation	photometric jitter — brightness, contrast, saturation, hue, sharpness; ≤3 per frame
Validation split	none — all 177 episodes used for training

new_embodiment maps to embodiment id 10, which is unused in NVIDIA's pretraining (absent from both embodiment_id.json and statistics.json). Its category-specific projector is therefore randomly initialised and trained from scratch on this robot; normalization statistics come from the dataset.

Observation and action space

Three cameras, renamed to GR00T/π-style keys:

observation.images.base_0_rgb          (head camera)
observation.images.left_wrist_0_rgb
observation.images.right_wrist_0_rgb

20-dimensional state and action, in this order:

 0      arm_center_z                      torso lift
 1-7    L_arm_j1 .. L_arm_j7              left arm joints
 8-14   R_arm_j1 .. R_arm_j7              right arm joints
15      right_hand.open_close_ratio
16      right_hand.thumb_opposition_ratio
17-19   base_vx, base_vy, base_wz         mobile base velocity command

The two left-hand DoF present in the raw dataset were dropped: the left hand is commanded in only 1,414 of 219,260 frames, and its stored quantile range spans ~0.01, which maps the rare 1.0 to a normalized +199. Removing those two dimensions drops the maximum normalized action magnitude from 199 to 10.

Checkpoints

EMA weights at five points in training:

step_008000/  step_016000/  step_024000/  step_032000/  step_040000/

There is no model at the repository root — download a step folder and load it by local path (from_pretrained has no subfolder argument):

from huggingface_hub import snapshot_download
from lerobot.policies.groot.modeling_groot import GrootPolicy

STEP = "step_016000"
root = snapshot_download("Kasra99/groot_dex_warehouse_ae", allow_patterns=f"{STEP}/*")
policy = GrootPolicy.from_pretrained(f"{root}/{STEP}")

Requires lerobot >= 0.6.2 with the groot extra (pip install 'lerobot[groot]').

Tasks

22 language instructions, all warehouse pick-and-place, factorising into 5 objects (banana, batman toy, bear toy, blue bird, box) × 5 destinations (box, gaylord, table, conveyor belt, pick-only). Coverage is very uneven — box and gaylord destinations hold 89,043 and 86,050 frames, while conveyor belt has 1,796 frames across 3 episodes. Instructions in the thin tail should not be expected to work.

Known limitations

• Trained on 177 episodes. This is a small-data fine-tune, not a generalist policy.
• Object identity correlates strongly with destination in the training data (the bear toy goes in the box 72% of the time and is never placed on a table), so the policy may key on the object noun rather than the destination phrase.
• The left arm moves in the data but is very likely incidental teleoperation drift rather than purposeful bimanual manipulation: its per-episode path length is 1.75× the right arm's while its cross-episode reproducibility (R² 0.087) is far lower than the right arm's (0.226). Treat this as a right-arm manipulation policy on a mobile base.
• No held-out validation set, by design — validation loss is a weak proxy for task success in behavior cloning. Select a checkpoint by evaluating on the robot.