BrutalCaesar/act_so101_cubcyl_poshold_chunk50_bc

ACT without CVAE (behavioural cloning) - SO-101 - wrist+front - chunk 50

Action Chunking Transformer with the CVAE objective disabled (use_vae=false) — no latent, no KL term, plain L1 regression onto action chunks. 34.20M parameters against 51.57M for the CVAE variant.

This is the deliberate baseline for act_so101_cubcyl_poshold_chunk50_cvae, which is identical in every other respect.

Final eval_loss (held-out 30 episodes): 0.1988 — the lowest of four variants trained on this split. Read the evaluation caveat below before drawing any conclusion from that.

Why this baseline exists

The ACT paper's ablation reports that removing the CVAE objective costs almost nothing on scripted data but drops human-demonstration performance from 35.3% to 2%. Our demonstrations are human and our latent measured as collapsed, so this run tests whether the collapse matters.

On loss, the answer so far is that it does not — this model is at least as good as every CVAE variant. Whether that survives contact with the robot is the open question, and the metric is biased in this model's favour (see below).

✅ Camera keys are correct — wire by name

Physical camera	Observation key
wrist (gripper module)	observation.images.wrist
front (desk-level Brio 101)	observation.images.front

Verified against pixels at episodes 8, 39 and 67 of the source dataset. Do not apply the transposed mapping from the 8bin models — that swap belongs to a different dataset.

The top camera was recorded but is not an input to this policy.

Held-out episodes — evaluate on these

30 episodes were excluded from training: 5 from each of the 6 object x container blocks.

[0,1,2,3,4, 20,21,22,23,24, 45,46,47,48,49, 65,66,67,68,69, 90,91,92,93,94, 110,111,112,113,114]

Objects and containers, by episode_index: red cube 0-39, yellow cylinder 40-79, white cube 80-119; cardboard box in the first 20 of each block, white 3D-printed bin in the second 20. Only one container was on the table at a time, so the visible bin is always the target.

Inputs / outputs

	Shape
observation.images.wrist	(3, 480, 640)
observation.images.front	(3, 480, 640)
observation.state	(6,)
action	(50, 6) — 50-step chunk, 1.67 s at 30 fps, all executed

Normalization ships as LeRobot 0.6.0 processor files, not baked into the weights. Load through ACTPolicy.from_pretrained or actions come out in the wrong units.

🚨 Evaluation caveat that applies to both models

These two policies are indistinguishable on held-out loss. Final eval_loss: 0.2041 (CVAE) vs 0.1988 (BC). But a single model's eval bounces by ~0.008 between consecutive checkpoints, so the 0.005 spread between models is smaller than the noise within either one.

Worse, the metric is biased for this data. The demonstrations contain roughly three distinct grasp approaches per object, and held-out L1 systematically rewards a policy that averages them — a policy committing crisply to approach 2 is penalised on every held-out episode that used approach 1 or 3. BC's marginally lower number is exactly what an averaging model would produce.

Only scored rollouts can rank these. Do not conclude from the loss.

Training

Dataset	phi_so101_cubes_cylinder_v1, 90 of 120 episodes, 49,969 frames
Steps / batch / seed	100,000 / 8 / 1000 (16.0 epochs)
Optimizer	AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4
Backbone	ResNet-18, ImageNet-pretrained, not frozen
n_obs_steps	1 — single frame, no history
Hardware	1x H200, Northeastern Explorer

No overfitting was observed: eval_loss is flat from step 60,000 to 100,000, so any checkpoint in that range is equivalent. This is the 100,000-step checkpoint.

Known limitations

• n_obs_steps=1, so no velocity information.
• Scene-dependent: the cardboard box and white bin as used during recording. A different container is a domain shift.
• One room, one lighting setup, one operator.
• ⚠️ The operator observed a camera frame-rate warning on battery power during roughly the first 50 episodes of collection. Recorded as reported; a pixel-level check found 4-9% stalled frames uniformly across the dataset with no early-vs-late pattern, so this is unconfirmed.
• ⚠️ The white cube is low contrast against the light table in the overhead view. Not an input here (top camera unused), but it is the object most likely to underperform.