RyanL22/openarm-idm-depthanything-v2-large

OpenArm IDM — DepthAnything-V2-Large vision tower

An inverse dynamics model (IDM) for a bimanual OpenArm robot with Inspire RH56F1 hands. It is video-only: given frames at t and t+20 from a stereo egocentric camera pair, it predicts the 20-step, 28-D action chunk that connects them (neck 2 + left arm 7 + right arm 7 + left hand 6 + right hand 6).

Architecture is GR00T-Dreams' flow-matching IDM with the vision encoder replaced: DepthAnything-V2-Large (a DINOv2-Large backbone) instead of SigLIP2-large. The rest — DiT, action encoder/decoder, VL self-attention — is unchanged.

Motivation: 1X's world-model report describes an IDM built from a Depth Anything backbone feeding a separate flow-matching head. This checkpoint tests whether a depth-pretrained representation beats a contrastive-pretrained one on a small (3.75 h) dataset.

Results

Single-variable A/B against the SigLIP2-large baseline — same data, action_horizon=20, global batch 128, 30 000 steps, mirror augmentation, stereo pair. Only the tower differs. Scored on the same 34 test episodes (1013 query chunks, 20 260 step samples) by one script in one job.

	SigLIP2-large	DepthAnything-V2-L (this model)
overall MAE (rad)	0.0359	0.0336	−6.4 %
overall RMSE (rad)	0.0754	0.0718	−4.8 %

Per joint group (MAE, radians) — all five improve:

group	SigLIP	this model
left_arm	0.0230	0.0215
right_arm	0.0330	0.0303
left_hand	0.0369	0.0354
right_hand	0.0651	0.0608
neck	0.0000	0.0000

Error decomposition — most of the gain is in the pose term, the part that reads absolute joint angles out of pixels, which is where depth pretraining should help:

	SigLIP	this model
pose (absolute)	0.0360	0.0338
motion (within chunk)	0.0169	0.0163

Uniform across the horizon: t+1 −6.0 %, t+5 −6.9 %, t+10 −6.4 %, t+15 −5.9 %, t+20 −5.5 %.

Limitation, stated plainly

Both arms are still worse than the static baseline. Repeating action[t] for 20 steps gives MAE 0.0281, against 0.0336 here and 0.0359 for SigLIP; only right_arm beats it (0.0303 vs 0.0389). The backbone comparison is unaffected — the static baseline is identical for both arms — but this model does not yet beat "hold still" overall. Treat the 6.4 % as an improvement on top of that open problem, not as a solved IDM.

Trained on 3.75 h of data. Do not expect the absolute numbers to transfer to another robot, camera rig, or hand.

Inputs

• Two views, in this order: video.camera_ego_left, video.camera_ego_right. The view embedding is indexed by position, so the order is part of the interface.
• 224×224 RGB, ImageNet normalisation (mean [0.485, 0.456, 0.406], std [0.229, 0.224, 0.225]). This differs from the SigLIP baseline (256×256, mean/std 0.5) and feeding SigLIP-range input raises rather than silently degrading.
• No state and no language input — the model is video-only.

Usage

Code, the pre-flight gate, and the training/eval commands are in the GR00T-Dreams fork under IDM_dump/docs/DEPTH_ANYTHING_BACKBONE.md. Evaluation needs no flag for the preprocessing — it is read from config.json:

python IDM_dump/scripts/eval_idm_openarm.py \
  --checkpoint  <this checkpoint> \
  --test-dir    <dataset>/test \
  --train-dir   <dataset>/train \
  --max-episodes 34 \
  --video-keys video.camera_ego_left video.camera_ego_right

Loading the model fetches depth-anything/Depth-Anything-V2-Large-hf for the tower definition, so that repo must be reachable or already cached.

Training

30 000 steps, global batch 128 (32 × 4), 4×H200, 7 h 11 min, 14.2 epochs, final train loss 0.0401. The vision tower is fine-tuned (291.8 M trainable parameters in the tower); encoder.layers[11] and vision_model.head are frozen to match the baseline's freezing exactly, so the freeze pattern is not part of the measured difference.