Pipe1213/smolvla_obj_t1_n10

SmolVLA on LIBERO-Object, BBQ sauce, 10 demonstrations, seed 1000

One of the 60 policies trained for vla-vs-scratch, a comparison of a fine-tuned pretrained vision-language-action model (SmolVLA) with a policy trained from scratch (ACT) on the same few demonstrations, in simulation. This model is SmolVLA fine-tuned from lerobot/smolvla_base, on one task only: "pick up the bbq sauce and place it in the basket". It is published to make the study reproducible, not as a general-purpose policy.

Results of this model

Each checkpoint was tested on the 50 fixed starting scenes of the task (LIBERO initial states 0 to 49), the same scenes for every model in the study.

Checkpoint (training step)	Successes	Success rate
3,000	41 of 50	82%
4,000	40 of 50	80%
5,000	44 of 50	88%
Mean of the 3 checkpoints (the study's score)	125 of 150	83.3% (95% Wilson interval 77 to 88%)

The repository root holds the last checkpoint (5,000 steps, 88%). The Wilson interval covers evaluation sampling only; differences between training seeds are larger, especially for ACT.

For context, pooled over the study's 3 tasks and all training seeds, SmolVLA with 10 demonstrations per task scored 82.4%, and ACT with the same demonstrations scored 34.6%. Full results: README and analysis.

Training

SmolVLA (Shukor et al., 2025), fine-tuned from lerobot/smolvla_base at revision 5e8d12a6e2975b0e5e5fce7c8caf47c371d257b6. Only the action expert is trained (about 100M of 450M parameters); the vision-language model stays frozen. The policy also reads the task instruction.

• Task: LIBERO-Object, simulator task id 3, "pick up the bbq sauce and place it in the basket".
• Demonstrations: 10 episodes of lerobot/libero at revision a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4: 843, 853, 871, 889, 893, 932, 1005, 1067, 1077, 1235. The smaller sets of the study are nested inside the larger ones, and both methods use the same episodes.
• Settings: batch size 32, 5,000 steps, a checkpoint every 1,000 steps, seed 1000; camera names mapped to the base model's with --rename_map. The step budget is the same for every number of demonstrations.
• Software: LeRobot 0.6.2 at commit b9cb121cb7d4e3e26ec5c906d08dda68d151ba52.
• Compute: 1.49 h on Hugging Face Jobs, NVIDIA L4 (24 GB), $1.19.

Exact command, generated from the repository's configs/runs.yaml by scripts/train_grid.py (submitted to Hugging Face Jobs):

lerobot-train \
  --dataset.repo_id=lerobot/libero \
  --dataset.revision=a1aaacb7f6cd6ee5fb43120f673cebb0cfea7dd4 \
  '--dataset.episodes=[843, 853, 871, 889, 893, 932, 1005, 1067, 1077, 1235]' \
  --seed=1000 \
  --log_freq=200 \
  --output_dir=outputs/train/smolvla_obj_t1_n10 \
  --job_name=smolvla_obj_t1_n10 \
  --wandb.enable=true \
  --policy.path=lerobot/smolvla_base \
  --policy.train_expert_only=true \
  --policy.private=true \
  --batch_size=32 \
  --steps=5000 \
  --save_freq=1000 \
  --save_checkpoint_to_hub=true \
  --num_workers=6 \
  --job.target=l4x1 \
  --job.timeout=3h \
  --job.detach=true \
  '--rename_map={"observation.images.image": "observation.images.camera1", "observation.images.image2": "observation.images.camera2"}' \
  --policy.repo_id=Pipe1213/smolvla_obj_t1_n10

LeRobot ignored --policy.repo_id when fine-tuning with --policy.path and created the repo with a timestamp in its name; it was renamed to this name and made public after training.

Evaluation

50 episodes per checkpoint, starting from LIBERO's fixed initial states, 256 by 256 pixel images, re-planning every 10 actions, evaluation seed 1000, MuJoCo rendering on the CPU (WSL2). To evaluate the last checkpoint:

lerobot-eval --policy.path=Pipe1213/smolvla_obj_t1_n10 \
  --env.type=libero --env.task=libero_object --env.task_ids="[3]" \
  --env.init_states=true --env.observation_height=256 --env.observation_width=256 \
  --eval.n_episodes=50 --eval.batch_size=1 --seed=1000 --policy.n_action_steps=10 \
  --rename_map='{"observation.images.image": "observation.images.camera1", "observation.images.image2": "observation.images.camera2"}'

The other evaluated checkpoints are in checkpoints/<step>/pretrained_model (003000, 004000, 005000): download that folder and point --policy.path at it.

Limitations

• Simulation only, one task from one benchmark. Not tested on a real robot.
• The study compares two methods as they are used in practice. SmolVLA and ACT differ in pretraining but also in size, architecture and language input, so the gap between them cannot be put on pretraining alone.
• The 3 tasks were chosen among LIBERO-Object tasks that a published SmolVLA LIBERO checkpoint already solved, so results describe these tasks.
• States and actions are normalised with statistics from the whole lerobot/libero dataset (LeRobot's behaviour), not only the selected demonstrations.

Code and full results: https://github.com/Pipe1213/vla-vs-scratch.