srlwam — stacking policies

LiLa-WAM policies for the two-cube Franka stacking task, trained in Isaac Lab simulation and packaged for deployment on the real robot. Training data lives in the companion dataset repo aabyaneh/srlwam.

Checkpoint	Weights	Trained on	Status
policy_stack_v0	epoch 40 of sft_2026-09-01_19-41-16	data_stack_v0 — 2000 Mimic-generated demos	deploy this
archive/policy_stack_v0_ep6.pt	epoch 6, best validation loss	same	superseded; kept for comparison

Epoch 40 was chosen on evidence rather than on validation loss — see Which checkpoint.

policy_stack_v0

Flow-matching action model over a frozen DINOv3 ViT-L/16 encoder. Two camera views plus a 9-D proprioceptive state in, 32-step chunks of 7-D actions out. Weights are epoch 40 of sft_2026-09-01_19-41-16; sim validation at that epoch was action L1 0.0172, position 0.0126 m, gripper accuracy 0.975.

policy_stack_v0/
├── policy_stack_v0.pt        the action model, epoch 40 (2.4 GB)
├── model/
│   ├── config.yaml           camera order, state keys, chunk size, image size
│   ├── dinov3_pretrain/      frozen vision encoder (1.2 GB), loaded at construction
│   ├── models/, utils/       inference code and the normalisation statistics
│   └── robotwin_infer.py
├── lilawam_policy.py         observation -> action chunk
├── lilawam_deploy.py         the sim-to-real adapter; every convention lives here
├── check_deployment.py       five-check self-test, run it before enabling the arm
├── fixture.npz               16 real observations + reference predictions (19 MB)
├── fixture_meta.json
└── ZEROSHOT_PLAN.md          packaging, transfer, calibration, control loop, eval protocol

Quick start

hf download aabyaneh/srlwam --repo-type model --include "policy_stack_v0/*" \
    --local-dir ./srlwam --max-workers 8
cd ./srlwam/policy_stack_v0
python check_deployment.py --bundle .          # nothing moves a robot

Needs Python 3.10+, torch with CUDA (bf16, so Ampere or newer), torchvision, transformers, omegaconf, numpy, opencv-python, h5py. About 1.1 GB of GPU memory; inference is 36 ms median at batch 1 on an RTX 4080 SUPER, which fits a 10 Hz control loop with room to re-plan every step.

Interface

Observations — the policy speaks the simulator's dialect, and each row below is a conversion the robot side must apply. All were measured against 20 real captures, not assumed; lilawam_deploy.policy_observation implements them and check_deployment.py proves it.

Policy input	Shape	From the robot	Conversion
table_cam	(B, 256, 256, 3) uint8 RGB	agentview 640×480	crop raw[:, 85:565], resize 256² INTER_AREA
wrist_cam	(B, 256, 256, 3) uint8 RGB	wrist 640×480	crop raw[:, 125:605], resize 256² INTER_AREA
eef_pos	(B, 3) float32 [m]	O_T_EE translation (panda_hand origin)	+0.1034 m along the hand's own z
eef_quat	(B, 4) float32	O_T_EE rotation (panda_link8 flange)	right-multiply by Rz(−45°), keep (x, y, z, w)
gripper_pos	(B, 2) float32 [m]	gripper width	(w/2, −w/2)

Three of these fail silently if you get them wrong — the policy still emits actions, they are just meaningless:

• The two camera crops are different windows. The wrist is 40 px right of centre. A naive centre crop disagrees with the reference observations by 23.6/255 against a correct 1.5/255.
• Do not reorder the quaternion. Both sides use (x, y, z, w) despite Isaac Lab's nominal (w, x, y, z). Reordering puts 100% of frames outside the training normalisation range on three of four components.
• The flange→hand 45° roll is required, same failure mode.

Actions — 7 numbers per step, in the simulator's convention:

Index	Channel	Meaning	On the robot
0:3	dx, dy, dz	IK-relative command, realised at 0.192× by the sim controller	tool displacement 0.192 × a[0:3] per 10 Hz period
3:6	d roll, d pitch, d yaw	a Mimic data-generation artefact; the task needs none	drop, hold orientation
6	gripper	+1 = OPEN, −1 = CLOSE	> 0 → open, ≤ 0 → close

The gripper sign is inverted relative to the real recordings, where +1 means close. Consume the policy's convention on the robot; apply the flip only when scoring against recorded real labels. Getting this backwards makes the arm open on every grasp, and nothing errors.

Control rate is 10 Hz. The tool point is the ee_frame, 0.1034 m ahead of panda_hand.

Which checkpoint

Validation loss and validation L1 disagree across this training run: loss is best at epoch 6 (0.356, rising to 0.669 by epoch 40) while action L1 keeps improving (0.0251 → 0.0172). Rather than pick on either, both checkpoints were scored open-loop against the 20 real episodes — the metric that actually bears on deployment. Epoch 40 wins on nearly everything, at the executed step and across the chunk:

on real data	epoch 6	epoch 40
direction cosine, executed step	+0.380	+0.490
cosine > 0, executed step	0.661	0.716
translation L1, executed step [m]	0.01049	0.01020
gripper accuracy, executed step	0.907	0.880
direction cosine, full chunk	0.263	0.308
gripper accuracy, full chunk	0.721	0.759
action L1, full chunk	0.0879	0.0752
spurious rotation L1 [rad]	0.00675	0.00539

Epoch 40 is better at where to move; epoch 6 holds a 2.7-point edge on the immediate gripper sign. Since reaching dominates and the adapter re-plans every step, epoch 40 is the deployment choice — but the grasp-timing caveat below applies to both, so comparing them on the robot is worthwhile.

Honest performance expectations

Scored open-loop against 20 real teleoperated episodes (predict at every frame, compare to what the operator did next), this checkpoint is not expected to succeed zero-shot:

	held-out sim	real
action L1	0.0181	0.0752
direction cosine, executed step	+0.626	+0.490
gripper accuracy, executed step	0.989	0.880
gripper accuracy by chunk step 31	0.972	0.579

It responds to real images — direction and current gripper state carry real signal — but open-loop error grows with horizon on real data while staying flat in sim, and gripper accuracy collapses below chance partway through the episode: it mistimes the grasp. The likeliest cause is object placement. In those recordings every cube sat 3–7 cm beyond the training spawn box, putting eef x outside the training normalisation range for 26.9% of frames — the only state dimension that leaves it. Place cubes inside x ∈ [0.40, 0.50] m, |y| ≤ 0.10 m before drawing conclusions.

ZEROSHOT_PLAN.md has the calibration procedure, the safety envelope, and the evaluation protocol.

Contents note

fixture.npz contains 16 frames captured in the lab (both camera streams, raw 640×480) with the matching robot state, used to verify a deployment reproduces the reference pipeline.

Training data

data_stack_v0: 2000 Mimic-generated trajectories from 19 teleoperated source demonstrations, domain-randomised over lighting, table material, cube appearance and camera mount jitter.

The rotation in this policy's action space is an artefact of data generation, not of the task: the teleoperated demonstrations contain exactly zero rotation, and Mimic manufactures it from isotropic action noise plus an orientation feedback term. A successor corpus regenerated without it (data_stack_v1_norot, 2004 demos, rotation identically zero, slightly better generation yield) exists for retraining.