BrutalCaesar/act_so101_8bin_wrist_top_chunk50

ACT · SO-101 · 8-bin pick-and-place · wrist + top · chunk 50

Action Chunking Transformer trained with LeRobot 0.6.0 on BrutalCaesar/phi_so101_8bin_v1. Two cameras: wrist (gripper module) + top (EMEET C960, overhead on a boom arm).

One of six runs in a chunk-size × camera-pair sweep ({50, 75, 100} × {wrist+top, wrist+front}). Sibling with the same chunk size, different second camera: act_so101_8bin_wrist_front_chunk50.

🚨 Both camera keys are transposed — read this before inference

The dataset was recorded with the wrist and top camera keys swapped, and this model uses both of them. So the key names are exactly backwards from what you want:

Feed this physical camera	Under this observation key
wrist (gripper module)	observation.images.top
top (overhead, boom arm)	observation.images.wrist

Wire these by name and you get a fully inverted setup: the policy receives an overhead shot where it expects a gripper close-up and vice versa. It will not error. It will load, run, and produce plausible-looking garbage.

Verified against pixels (episode 0, frame 200): observation.images.top holds a gripper close-up of the target object; observation.images.wrist holds the overhead view of the whole arm and table. The key names are wrong in the data; the physical rig is labelled correctly.

Camera framing matters too — the mounts must sit where they sat during recording. A camera that drifted between recording and evaluation silently invalidates the policy.

Inputs / outputs

	Shape	Notes
observation.images.top	(3, 480, 640)	physical wrist camera, no resize anywhere in the pipeline
observation.images.wrist	(3, 480, 640)	physical top / overhead camera
observation.state	(6,)	SO-101 joint positions
action	(50, 6)	chunk of 50 future actions, all executed (n_action_steps=50)

Normalization ships with the checkpoint as LeRobot 0.6.0 processor files (policy_preprocessor*, policy_postprocessor*) — not baked into the policy. Load through ACTPolicy.from_pretrained so they are picked up; hand-rolling a forward pass without them produces actions in the wrong units.

Training

Steps	100,000 (= 800K samples, ≈14.6 epochs over 54,800 frames)
Batch size	8
Final train loss	0.056
Optimizer	AdamW, lr 1e-5, backbone lr 1e-5, weight decay 1e-4
Backbone	ResNet-18, ImageNet-pretrained, not frozen
dim_model / chunk_size / kl_weight	512 / 50 / 10.0
n_obs_steps	1 (single frame, no history)
Seed	1000
Hardware	1× Tesla V100-32GB, Northeastern Explorer cluster, 4 h 41 m

Everything except chunk_size and the camera pair is LeRobot default, deliberately — the sweep is meant to be comparable across people training the same task on the same defaults.

Train / held-out split

Trained on 89 episodes covering 6 of the 8 bins. Left bin 3 and right bin 2 are held out entirely — no episode touching them appears in training. Those two bins are the generalization test; the other six measure fit.

Evaluation

Not yet evaluated on hardware. No success rate is reported here because none has been measured. The planned protocol is 20 scored rollouts per bin, reported per bin and never averaged — an average over 8 bins hides exactly the failure this split is designed to expose.

This model's train loss (0.056) is within 0.001 of its wrist+front sibling (0.055). Do not read that as the two camera setups being equivalent — it means training loss cannot distinguish them, which is why the rollouts exist.

Usage

from lerobot.policies.act.modeling_act import ACTPolicy

policy = ACTPolicy.from_pretrained("BrutalCaesar/act_so101_8bin_wrist_top_chunk50")

Or point a LeRobot eval script at it with --policy.path=BrutalCaesar/act_so101_8bin_wrist_top_chunk50, remembering the transposed camera keys above when you write --robot.cameras.

Known limitations

• Single observation frame, so no velocity information.
• The CVAE latent is effectively collapsed at kl_weight=10.0 on this dataset (KL ≈ 0 throughout training), so this behaves close to a deterministic chunk regressor. Expected for ACT, not a defect — but do not read the latent as a source of behavioral diversity.
• Trained on one lighting setup in one room. No domain randomization beyond LeRobot's default image augmentation.
• The overhead camera sits ~105 cm above the workspace at 90° dFOV; a boom arm that sags or gets bumped changes the observation more than the wrist camera does.