BrutalCaesar/act_so101_cubcyl_poshold_chunk50_cvae

ACT (CVAE) - SO-101 - 3 objects, 2 containers - wrist+front - chunk 50

Action Chunking Transformer, standard CVAE objective (use_vae=true, kl_weight=10.0), trained with LeRobot 0.6.0.

Task: pick up one of three objects (red 25 mm cube, yellow cylinder, white 45 mm cube) and place it in whichever container is on the table (cardboard box or white 3D-printed bin).

Paired baseline: act_so101_cubcyl_poshold_chunk50_bc — identical in every respect except use_vae=false. Trained to answer whether the CVAE objective matters on this data.

Final eval_loss (held-out 30 episodes): 0.2041. Note this includes the kl_weight x KL term and the BC model's does not, so the two numbers are not directly comparable.

The latent is collapsed, and lowering the penalty did not change that

Measured across a β sweep at kl_weight = 10, 1 and 0.1: all three finished with train loss 0.056-0.057, identical to the BC run's 0.059, which has no KL term at all. At β=0.1 a KL of 1 nat would add 0.1 to the loss; nothing of the sort appears, putting the KL below roughly 0.02 nats regardless of β.

So the collapse is not caused by the penalty. The likely explanation is that observation.state already discloses which grasp approach is underway for most frames — the demonstrations are multimodal at the task level but close to deterministic per-observation, which is what the encoder actually conditions on.

Practical consequence: this model behaves close to a deterministic chunk regressor. Do not expect behavioural diversity from the latent.

✅ Camera keys are correct — wire by name

Physical camera	Observation key
wrist (gripper module)	observation.images.wrist
front (desk-level Brio 101)	observation.images.front

Verified against pixels at episodes 8, 39 and 67 of the source dataset. Do not apply the transposed mapping from the 8bin models — that swap belongs to a different dataset.

The top camera was recorded but is not an input to this policy.

Held-out episodes — evaluate on these

30 episodes were excluded from training: 5 from each of the 6 object x container blocks.

[0,1,2,3,4, 20,21,22,23,24, 45,46,47,48,49, 65,66,67,68,69, 90,91,92,93,94, 110,111,112,113,114]

Objects and containers, by episode_index: red cube 0-39, yellow cylinder 40-79, white cube 80-119; cardboard box in the first 20 of each block, white 3D-printed bin in the second 20. Only one container was on the table at a time, so the visible bin is always the target.

Inputs / outputs

	Shape
observation.images.wrist	(3, 480, 640)
observation.images.front	(3, 480, 640)
observation.state	(6,)
action	(50, 6) — 50-step chunk, 1.67 s at 30 fps, all executed

Normalization ships as LeRobot 0.6.0 processor files, not baked into the weights. Load through ACTPolicy.from_pretrained or actions come out in the wrong units.

🚨 Evaluation caveat that applies to both models

These two policies are indistinguishable on held-out loss. Final eval_loss: 0.2041 (CVAE) vs 0.1988 (BC). But a single model's eval bounces by ~0.008 between consecutive checkpoints, so the 0.005 spread between models is smaller than the noise within either one.

Worse, the metric is biased for this data. The demonstrations contain roughly three distinct grasp approaches per object, and held-out L1 systematically rewards a policy that averages them — a policy committing crisply to approach 2 is penalised on every held-out episode that used approach 1 or 3. BC's marginally lower number is exactly what an averaging model would produce.

Only scored rollouts can rank these. Do not conclude from the loss.

Training

Dataset	phi_so101_cubes_cylinder_v1, 90 of 120 episodes, 49,969 frames
Steps / batch / seed	100,000 / 8 / 1000 (16.0 epochs)
Optimizer	AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4
Backbone	ResNet-18, ImageNet-pretrained, not frozen
n_obs_steps	1 — single frame, no history
Hardware	1x H200, Northeastern Explorer

No overfitting was observed: eval_loss is flat from step 60,000 to 100,000, so any checkpoint in that range is equivalent. This is the 100,000-step checkpoint.

Known limitations

• n_obs_steps=1, so no velocity information.
• Scene-dependent: the cardboard box and white bin as used during recording. A different container is a domain shift.
• One room, one lighting setup, one operator.
• ⚠️ The operator observed a camera frame-rate warning on battery power during roughly the first 50 episodes of collection. Recorded as reported; a pixel-level check found 4-9% stalled frames uniformly across the dataset with no early-vs-late pattern, so this is unconfirmed.
• ⚠️ The white cube is low contrast against the light table in the overhead view. Not an input here (top camera unused), but it is the object most likely to underperform.