chomeed/robometer-4b-lora-hammer-cleanup-seg

Robometer-4B LoRA — MimicGen hammer_cleanup

LoRA fine-tune of robometer/Robometer-4B on MimicGen hammer_cleanup, a three-stage task: open the drawer, place the hammer inside, close the drawer. Emits per-frame progress in [0,1] and a success probability.

Task: Open the drawer, place the hammer inside, and close the drawer.

Results — against the zero-shot base

Both models on chomeed/mimicgen_hammer_cleanup_test_seg_rfm (n=40: 20 successful, 20 failure), cutoff-independent metrics.

metric	base Robometer-4B	this LoRA
ranking_acc	0.927	0.945
kendall — progress monotonicity	0.246	0.673	2.7x
success_auprc	0.846	0.901
succ_fail_gap	+0.262	+0.316
pos_acc (> 0.5)	1.00	1.00
neg_acc (≤ 0.5)	0.40	0.35

The headline gain is monotonicity: kendall 0.246 → 0.673. The base's progress barely increases along a trajectory; this model's does, which is what makes it usable as a dense reward rather than only a final-frame classifier.

Use threshold 0.70, not 0.50

pos_acc/neg_acc above are measured at a fixed 0.5, and that is the wrong operating point here. Final progress is median 0.913 on successes and 0.563 on failures — well separated, but the whole scale sits high:

threshold	pos_acc	neg_acc	balanced
0.50	1.00	0.35	0.675
0.60	1.00	0.60	0.800
0.70	1.00	0.85	0.925
0.80	0.90	0.90	0.900

At 0.70 the model gets every success right and 85% of failures. Reading the 0.5 numbers alone would badly understate it. A per-task threshold sweep is worth running before deploying any of these as a reward signal.

Training

• LoRA r=32, α=64, dropout=0.05 on attention + MLP projections; 1000 steps, batch 8, lr 2e-5, cosine
• ~41 min on one B200; loss 0.42 → 0.18; success cutoff 0.95

Data

chomeed/mimicgen_hammer_cleanup_train_seg_rfm — 160 trajectories, 80 successful / 80 failure.

Failures from ..._d1_..._35k_failure_seg (tail cut at the outcome, approach kept). Successes half from ..._d1_..._35k_success, half from the d0 expert demos.

Variant caveat: failures are all d1 while half the successes are d0. The two variants randomise the scene differently, so part of the class separation may come from scene statistics rather than task outcome. Retraining with d1-only positives would settle it; that has not been run.

Usage

out = compute_batch_outputs(model, tokenizer, inputs, sample_type="progress",
                            is_discrete_mode=True, num_bins=10)   # MUST be True
progress = np.array(out["progress_pred"][0])   # (T,) in [0,1]

Two things that fail silently if you get them wrong:

• is_discrete_mode=True is required. With False the call returns raw per-bin logits (T×bins values, range ~[-3,5]) instead of progress. No exception — just meaningless numbers.
• The task string conditions the model. Passing a different task's string produces confidently wrong scores; an earlier version of this evaluation did exactly that and had to be redone.

Caveat

save_best could not rank checkpoints (its metric names do not match the ones logged for this dataset), so these are the final step-1000 weights, not a selected best.