Pipe1213/act_obj_t3_n20_s2

ACT on LIBERO-Object, chocolate pudding, 20 demonstrations, seed 2000

One of the 60 policies trained for vla-vs-scratch, a comparison of a fine-tuned pretrained vision-language-action model (SmolVLA) with a policy trained from scratch (ACT) on the same few demonstrations, in simulation. This model is ACT trained from scratch, on one task only: "pick up the chocolate pudding and place it in the basket". It is published to make the study reproducible, not as a general-purpose policy.

Results of this model

Each checkpoint was tested on the 50 fixed starting scenes of the task (LIBERO initial states 0 to 49), the same scenes for every model in the study.

Checkpoint (training step)	Successes	Success rate
6,000	10 of 50	20%
8,000	36 of 50	72%
10,000	31 of 50	62%
Mean of the 3 checkpoints (the study's score)	77 of 150	51.3% (95% Wilson interval 43 to 59%)

The repository root holds the last checkpoint (10,000 steps, 62%). The Wilson interval covers evaluation sampling only; differences between training seeds are larger, especially for ACT.

For context, pooled over the study's 3 tasks and all training seeds, ACT with 20 demonstrations per task scored 33.1%, and SmolVLA with the same demonstrations scored 74.2%. Full results: README and analysis.

Training

ACT (Zhao et al., 2023) with LeRobot's default settings: 52M parameters, chunks of 100 actions, and a ResNet-18 image encoder initialised from ImageNet weights. Everything else is trained from scratch on the demonstrations below. The only change from the defaults is at evaluation: the policy re-plans every 10 actions instead of running a whole chunk.

• Task: LIBERO-Object, simulator task id 8, "pick up the chocolate pudding and place it in the basket".
• Demonstrations: 20 episodes of lerobot/libero at revision a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4: 840, 841, 847, 878, 910, 918, 953, 966, 972, 975, 977, 1048, 1081, 1106, 1138, 1149, 1168, 1210, 1250, 1259. The smaller sets of the study are nested inside the larger ones, and both methods use the same episodes.
• Settings: batch size 8, 10,000 steps, a checkpoint every 2,000 steps, seed 2000. The step budget is the same for every number of demonstrations.
• Software: LeRobot 0.6.2 at commit b9cb121cb7d4e3e26ec5c906d08dda68d151ba52.
• Compute: 0.42 h on a laptop RTX 4050 (6 GB).

Exact command, generated from the repository's configs/runs.yaml by scripts/train_grid.py:

lerobot-train \
  --dataset.repo_id=lerobot/libero \
  --dataset.revision=a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4 \
  '--dataset.episodes=[840, 841, 847, 878, 910, 918, 953, 966, 972, 975, 977, 1048, 1081, 1106, 1138, 1149, 1168, 1210, 1250, 1259]' \
  --seed=2000 \
  --log_freq=200 \
  --output_dir=outputs/train/act_obj_t3_n20_s2 \
  --job_name=act_obj_t3_n20_s2 \
  --wandb.enable=true \
  --policy.type=act \
  --policy.device=cuda \
  --policy.push_to_hub=false \
  --batch_size=8 \
  --steps=10000 \
  --save_freq=2000 \
  --num_workers=4 \
  --wandb.disable_artifact=true

Evaluation

50 episodes per checkpoint, starting from LIBERO's fixed initial states, 256 by 256 pixel images, re-planning every 10 actions, evaluation seed 1000, MuJoCo rendering on the CPU (WSL2). To evaluate the last checkpoint:

lerobot-eval --policy.path=Pipe1213/act_obj_t3_n20_s2 \
  --env.type=libero --env.task=libero_object --env.task_ids="[8]" \
  --env.init_states=true --env.observation_height=256 --env.observation_width=256 \
  --eval.n_episodes=50 --eval.batch_size=1 --seed=1000 --policy.n_action_steps=10

The other evaluated checkpoints are in checkpoints/<step>/pretrained_model (006000, 008000, 010000): download that folder and point --policy.path at it.

Limitations

• Simulation only, one task from one benchmark. Not tested on a real robot.
• The study compares two methods as they are used in practice. SmolVLA and ACT differ in pretraining but also in size, architecture and language input, so the gap between them cannot be put on pretraining alone.
• The 3 tasks were chosen among LIBERO-Object tasks that a published SmolVLA LIBERO checkpoint already solved, so results describe these tasks.
• States and actions are normalised with statistics from the whole lerobot/libero dataset (LeRobot's behaviour), not only the selected demonstrations.
• ACT used LeRobot's defaults and was not tuned for this task.
• ACT's checkpoints from the same run often perform very differently (in 30 of 36 ACT runs in the study). Evaluate a checkpoint before using it.

Code and full results: https://github.com/Pipe1213/vla-vs-scratch.