cyanYuki/act_box_101eps_chunk100_exec25

ACT on SO-101: pick a ring and place it in a box

An Action Chunking Transformer (ACT) fine-tuned from scratch on 101 teleoperated demonstrations of a 6-DOF SO-ARM101 picking a ring off a table and placing it in a shallow box.

Measured on the real robot: roughly 70-80% success.

Robot and task

• SO-101 follower, 6 joints: shoulder_pan, shoulder_lift, elbow_flex, wrist_flex, wrist_roll, gripper
• 2 cameras: front/overhead 640x480, wrist 640x480, both 30 fps
• Instruction: Pick up the blue ring and place it in the box
• Dataset: cyanYuki/so101_box_101eps

Policy

parameter	value
type	ACT
chunk_size	100
n_action_steps	25
kl_weight	10.0
use_vae	true
dropout	0.1
vision_backbone	resnet18 (ImageNet)
optimizer	AdamW, lr 1e-5, weight decay 1e-4
batch_size	64
steps	15000
checkpoint used	14000 (best eval_loss 0.2698)
parameters	51,597,190

Two settings that mattered more than any hyperparameter

1. dataset.image_transforms.enable must be false for manipulation.

LeRobot's image transform pipeline includes RandomAffine(degrees=(-5,5), translate=(0.05,0.05)), and it is applied to the images only -- observation.state and the action labels are not transformed. On a 640 px frame a 5% translate is 32 px, so the model is taught that the object's image position carries no information about the correct action. That is exactly the failure this project kept hitting: a policy that reaches but cannot aim. Turning it off was the single largest improvement in the whole debugging effort.

2. n_action_steps greater than 1 needs action smoothing at rollout time.

With n_action_steps=25 the policy re-plans every 0.83 s, and each chunk boundary can jump. Without a low-pass filter the arm is pushed off the intended trajectory, the next chunk's input state leaves the training distribution, and the loop diverges into self-amplifying oscillation. A simple new = prev + alpha * (action - prev) with alpha = 0.6 cut the command reversal rate from ~15/s to ~7/s and the arm's from ~6/s to ~1.8/s, and the policy went from failing outright to 70-80%.

Deployment notes

• Start pose must come from the same dataset the policy was trained on: shoulder_pan 1.36, shoulder_lift -100.31, elbow_flex 90.26, wrist_flex 73.96, wrist_roll 1.15, gripper 0.67
• Task string must match the training data exactly, or a language-conditioned policy receives an out-of-distribution instruction
• The reference implementation and the full diagnostic tooling live in a LeRobot-based project; the measurements above come from flight-recorder logs of commanded vs observed joint streams.