Pipe1213/act_obj_t2_nall

ACT on LIBERO-Object, butter, all 45 demonstrations, seed 1000

One of the 60 policies trained for vla-vs-scratch, a comparison of a fine-tuned pretrained vision-language-action model (SmolVLA) with a policy trained from scratch (ACT) on the same few demonstrations, in simulation. This model is ACT trained from scratch, on one task only: "pick up the butter and place it in the basket". It is published to make the study reproducible, not as a general-purpose policy.

Results of this model

Each checkpoint was tested on the 50 fixed starting scenes of the task (LIBERO initial states 0 to 49), the same scenes for every model in the study.

Checkpoint (training step)	Successes	Success rate
6,000	17 of 50	34%
8,000	15 of 50	30%
10,000	18 of 50	36%
Mean of the 3 checkpoints (the study's score)	50 of 150	33.3% (95% Wilson interval 26 to 41%)

The repository root holds the last checkpoint (10,000 steps, 36%). The Wilson interval covers evaluation sampling only; differences between training seeds are larger, especially for ACT.

For context, pooled over the study's 3 tasks and all training seeds, ACT with all demonstrations (45 to 50 per task) scored 55.4%, and SmolVLA with the same demonstrations scored 87.3%. Full results: README and analysis.

Training

ACT (Zhao et al., 2023) with LeRobot's default settings: 52M parameters, chunks of 100 actions, and a ResNet-18 image encoder initialised from ImageNet weights. Everything else is trained from scratch on the demonstrations below. The only change from the defaults is at evaluation: the policy re-plans every 10 actions instead of running a whole chunk.

• Task: LIBERO-Object, simulator task id 6, "pick up the butter and place it in the basket".
• Demonstrations: 45 episodes of lerobot/libero at revision a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4: 819, 827, 828, 844, 855, 861, 872, 873, 881, 884, 907, 914, 917, 921, 927, 938, 939, 958, 960, 961, 994, 999, 1012, 1014, 1026, 1052, 1059, 1063, 1076, 1088, 1091, 1113, 1115, 1118, 1124, 1145, 1158, 1178, 1183, 1185, 1186, 1212, 1215, 1220, 1238. The smaller sets of the study are nested inside the larger ones, and both methods use the same episodes.
• Settings: batch size 8, 10,000 steps, a checkpoint every 2,000 steps, seed 1000. The step budget is the same for every number of demonstrations.
• Software: LeRobot 0.6.2 at commit b9cb121cb7d4e3e26ec5c906d08dda68d151ba52.
• Compute: 0.44 h on a laptop RTX 4050 (6 GB).

Exact command, generated from the repository's configs/runs.yaml by scripts/train_grid.py:

lerobot-train \
  --dataset.repo_id=lerobot/libero \
  --dataset.revision=a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4 \
  '--dataset.episodes=[819, 827, 828, 844, 855, 861, 872, 873, 881, 884, 907, 914, 917, 921, 927, 938, 939, 958, 960, 961, 994, 999, 1012, 1014, 1026, 1052, 1059, 1063, 1076, 1088, 1091, 1113, 1115, 1118, 1124, 1145, 1158, 1178, 1183, 1185, 1186, 1212, 1215, 1220, 1238]' \
  --seed=1000 \
  --log_freq=200 \
  --output_dir=outputs/train/act_obj_t2_nall \
  --job_name=act_obj_t2_nall \
  --wandb.enable=true \
  --policy.type=act \
  --policy.device=cuda \
  --policy.push_to_hub=false \
  --batch_size=8 \
  --steps=10000 \
  --save_freq=2000 \
  --num_workers=4

Trained before --wandb.disable_artifact=true was added to the ACT settings (it only stops checkpoint uploads to Weights & Biases).

Evaluation

50 episodes per checkpoint, starting from LIBERO's fixed initial states, 256 by 256 pixel images, re-planning every 10 actions, evaluation seed 1000, MuJoCo rendering on the CPU (WSL2). To evaluate the last checkpoint:

lerobot-eval --policy.path=Pipe1213/act_obj_t2_nall \
  --env.type=libero --env.task=libero_object --env.task_ids="[6]" \
  --env.init_states=true --env.observation_height=256 --env.observation_width=256 \
  --eval.n_episodes=50 --eval.batch_size=1 --seed=1000 --policy.n_action_steps=10

The other evaluated checkpoints are in checkpoints/<step>/pretrained_model (006000, 008000, 010000): download that folder and point --policy.path at it.

Limitations

• Simulation only, one task from one benchmark. Not tested on a real robot.
• The study compares two methods as they are used in practice. SmolVLA and ACT differ in pretraining but also in size, architecture and language input, so the gap between them cannot be put on pretraining alone.
• The 3 tasks were chosen among LIBERO-Object tasks that a published SmolVLA LIBERO checkpoint already solved, so results describe these tasks.
• States and actions are normalised with statistics from the whole lerobot/libero dataset (LeRobot's behaviour), not only the selected demonstrations.
• ACT used LeRobot's defaults and was not tuned for this task.
• ACT's checkpoints from the same run often perform very differently (in 30 of 36 ACT runs in the study). Evaluate a checkpoint before using it.

Code and full results: https://github.com/Pipe1213/vla-vs-scratch.