groot_dex_warehouse_ae_2

GR00T N1.7 action-expert fine-tune on Kasra99/dex-warehouse, a teleoperated warehouse pick-and-place dataset recorded on a Dexmate Vega 1 Pro mobile manipulator.

ae = action expert: the Cosmos-Reason2 / Qwen3-VL backbone (LLM + vision tower) is frozen; the flow-matching action head is trained.

What was trained

Base weights	nvidia/GR00T-N1.7-3B
Embodiment tag	new_embodiment (projector slot 10)
Trainable	projector 327M + DiT 1.09B + vlln/vl-attn 201M = 1.62B / 3.14B (51.5%)
Frozen	LLM 1.12B + vision tower 407M
Steps / batch	45,000 / 32 (5.9 epochs over 245,541 frames)
Optimizer	AdamW, lr 1e-4, cosine, warmup
Chunk / action steps	40 / 40 (N1.7 native horizon)
EMA	constant decay 0.99 (weights published are the EMA weights)
Augmentation	photometric jitter — brightness, contrast, saturation, hue, sharpness; ≤3 per frame
Validation split	none — all 213 episodes used for training

new_embodiment maps to embodiment id 10, which is unused in NVIDIA's pretraining (absent from both embodiment_id.json and statistics.json). Its category-specific projector is therefore randomly initialised and trained from scratch on this robot; normalization statistics come from the dataset.

Observation and action space

Three cameras, renamed to GR00T/π-style keys:

observation.images.base_0_rgb          (head camera)
observation.images.left_wrist_0_rgb
observation.images.right_wrist_0_rgb

20-dimensional state and action, in this order:

 0      arm_center_z                      torso lift
 1-7    L_arm_j1 .. L_arm_j7              left arm joints
 8-14   R_arm_j1 .. R_arm_j7              right arm joints
15      right_hand.open_close_ratio
16      right_hand.thumb_opposition_ratio
17-19   base_vx, base_vy, base_wz         mobile base velocity command

The two left-hand DoF present in the raw dataset were dropped: the left hand is commanded in only 1,414 of 219,260 frames, and its stored quantile range spans ~0.01, which maps the rare 1.0 to a normalized +199. Removing those two dimensions drops the maximum normalized action magnitude from 199 to 10.

Checkpoints

EMA weights at five points in training:

step_009000/  step_018000/  step_027000/  step_036000/  step_045000/

There is no model at the repository root — download a step folder and load it by local path (from_pretrained has no subfolder argument):

from huggingface_hub import snapshot_download
from lerobot.policies.groot.modeling_groot import GrootPolicy

STEP = "step_027000"
root = snapshot_download("Kasra99/groot_dex_warehouse_ae_2", allow_patterns=f"{STEP}/*")
policy = GrootPolicy.from_pretrained(f"{root}/{STEP}")

Requires lerobot >= 0.6.2 with the groot extra (pip install 'lerobot[groot]').

Tasks

22 language instructions, all warehouse pick-and-place, factorising into 5 objects (banana, batman toy, bear toy, blue bird, box) × 5 destinations (box, gaylord, table, conveyor belt, pick-only). Coverage is uneven — box and gaylord destinations dominate, while conveyor belt has 1,796 frames across 3 episodes. Instructions in the thin tail should not be expected to work.

Relation to groot_dex_warehouse_ae

Run 2 of the same recipe. Identical base weights, hyperparameters and seed; only the dataset differs (+36 episodes: +17 bear toy, +18 banana, +1 batman toy — the objects the first model handled least reliably). 13 corrupted task strings present in the raw dataset (typos such as 'pcik up the batman toy...', 'ick up the blue bird', and phrasing variants such as 'drop the box on conveyor belt') were merged into their canonical form before training, so the language conditioning is not fragmented across near-duplicate instructions.

Step counts were chosen so the five checkpoints land on the same epochs as run 1's (1.17 / 2.35 / 3.5 / 4.7 / 5.85), making the two runs directly comparable. step_027000 here is the analogue of run 1's step_024000.

Known limitations

• Trained on 213 episodes. This is a small-data fine-tune, not a generalist policy.
• Object identity correlates strongly with destination in the training data (the bear toy goes in the box 72% of the time and is never placed on a table), so the policy may key on the object noun rather than the destination phrase.
• The left arm moves in the data but is very likely incidental teleoperation drift rather than purposeful bimanual manipulation: its per-episode path length is 1.75× the right arm's while its cross-episode reproducibility (R² 0.087) is far lower than the right arm's (0.226). Treat this as a right-arm manipulation policy on a mobile base.
• No held-out validation set, by design — validation loss is a weak proxy for task success in behavior cloning. Select a checkpoint by evaluating on the robot.