Project-IRA/TPSoSe2026_SmolVLA_LeRobot_SO101_Finetuning_V3_Desk_Cleanup

SmolVLA SO-101 — V3 Desk Cleanup (does not work)

SmolVLA fine-tuned on the Desk Cleanup task only. This model does not work and is published for completeness only.

Part of Project-IRA — Interactive Robotic Arm. Code: https://github.com/Project-IRA/interactive-robotic-arm

Base model	lerobot/smolvla_base
Robot	SO-101 follower (6-DOF)
Training data	Project-IRA/TPSoSe2026_Dataset_Desk_Cleanup_LeRobot_SO101
Recommended checkpoint	n/a
Inputs	camera1 (wrist) + camera2 (desk) images — renamed keys, see Usage — 6-dim joint state, English instruction
Outputs	6-dim continuous action chunks

Quality

Does not work. No checkpoint from this run produced usable behaviour on the physical arm.

Desk Cleanup is the hardest of the four tasks: it is open-ended (a variable number of arbitrary objects, in arbitrary positions), unlike the Lego task's constrained brick-to-plate mapping. It is also a strong instance of the multimodal action distribution problem — with several objects on the desk, any of them is a valid next target, and a policy that averages across those modes reaches for the space between them.

For desk cleanup that actually works, use Pi05 V7 Full V2.

Training

Single L40S GPU.

Setting	Value
Base	lerobot/smolvla_base
Dataset	TPSoSe2026_Dataset_Desk_Cleanup_LeRobot_SO101
Steps	target 500000 (--save_freq=5000), stopped at 150000
Batch size	64
Vision encoder	frozen (SmolVLA default)
Parameters	99,880,992 trainable of 450,046,176 total
Camera keys	renamed — wrist_left->camera1, desk_view->camera2
SLURM	--gres=gpu:L40S:1 --cpus-per-task=16 --mem=92G --time=35:00:00

The 500000-step target — five times the Lego run — reflects how much was thrown at this task before concluding it would not work. Training was stopped at 150000 once it was clear no checkpoint produced usable behaviour on the robot.

Retained as a negative result. Together with V1, it established that single-task SmolVLA fine-tuning did not transfer to the open-ended tasks, which is what pushed the project toward the merged multi-task dataset and Pi0.5.

Usage

This model expects renamed camera keys. Training used --rename_map to remap the dataset's camera features:

Dataset feature	What the policy expects	Physical camera
observation.images.wrist_left	observation.images.camera1	wrist
observation.images.desk_view	observation.images.camera2	desk

If you feed this policy wrist_left / desk_view it will fail or silently misbehave. Name your cameras camera1 (wrist) and camera2 (desk) at inference time, or apply the same --rename_map when re-training. The Pi0.5 models do not do this — they use the native wrist_left / desk_view names.

This model does not produce usable behaviour. Do not deploy it.

For completeness, the layout is:

outputs_V3/train/smolvla_desk/checkpoints/<step>/pretrained_model/

Checkpoints present: every 5000 steps from 005000 to 150000, plus last (090000 was deleted). Repo total ~26.4 GB.

import torch
from lerobot.policies.smolvla.modeling_smolvla import SmolVLAPolicy

policy = SmolVLAPolicy.from_pretrained("Project-IRA/TPSoSe2026_SmolVLA_LeRobot_SO101_Finetuning_V3_Desk_Cleanup")
policy = policy.to("cuda").eval()

On-robot rollout:

lerobot-record \
  --robot.type=so101_follower \
  --robot.port=/dev/ttyACM0 \
  --robot.id=$ROBOT_ID \
  --robot.cameras='{
      camera1: {type: opencv, index_or_path: /dev/v4l/by-path/$WRIST_PATH, width: 640, height: 480, fps: 30},
      camera2: {type: opencv, index_or_path: /dev/v4l/by-path/$DESK_PATH,  width: 640, height: 480, fps: 30}
  }' \
  --policy.path=Project-IRA/TPSoSe2026_SmolVLA_LeRobot_SO101_Finetuning_V3_Desk_Cleanup \
  --dataset.repo_id=$HF_USER/eval_run \
  --dataset.single_task="Sort the lego by color" \
  --episodes=10

Use one of the exact training prompts (see the dataset card) as the task string. Both cameras run at 640x480 at inference time even though desk_view was recorded at 800x600.

Robot setup

Robot	SO-101 follower arm (6-DOF), robot_type: so_follower
Teleoperation	SO-101 leader arm
Control frequency	30 fps
State / action space	6-dim: shoulder_pan.pos, shoulder_lift.pos, elbow_flex.pos, wrist_flex.pos, wrist_roll.pos, gripper.pos
Camera observation.images.desk_view	800x600, h264 (recording)
Camera observation.images.wrist_left	640x480, h264 (recording)

Inference note: both cameras are run at 640x480 during inference, not at their recording resolutions, to reduce the payload sent to the inference server.

Environment notes

All training ran on a SLURM cluster with L40S GPUs. Two environment details were required and are easy to miss when reproducing:

• ffmpeg libraries for torchcodec. A minimal conda env supplies the shared libraries that torchcodec discovers at runtime: export LD_LIBRARY_PATH=$CONDA_PREFIX/envs/ffmpeg_libs_v8/lib:<venv>/lib/python3.12/site-packages/nvidia/npp/lib:$LD_LIBRARY_PATH
• --tolerance_s=0.01 on every run, to accommodate timestamp jitter in the recorded episodes.

Multi-GPU runs additionally set PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True. Datasets and the virtualenv were copied to node-local /scratch before training rather than read from shared storage.

No Weights & Biases logging was enabled for any run (--wandb.enable=false), so there are no public training curves — the job.*.err SLURM logs are the record.

Tasks and prompts

The model is conditioned on English natural-language instructions. Prompt phrasing was varied roughly every 10 episodes during recording, giving 93 distinct prompts in the merged dataset. Use one of the training prompts verbatim for best results — the full lists are on the dataset card.

Limitations

• Behaviour cloning. The policy imitates teleoperated demonstrations and has no notion of recovery beyond what was demonstrated. It is susceptible to covariate shift and can fail to recover from states outside the demonstration distribution.
• Recovery data is incidental, not systematic. Recovery behaviour appears in the data only where the operator happened to make and correct a mistake during recording; no recovery episodes were scripted deliberately.
• Single environment. All data comes from one lab desk with one lighting setup, one camera geometry, and one set of physical objects. Expect degradation elsewhere.
• Prompt sensitivity. Language conditioning was trained on a fixed set of phrasings (listed in the dataset card). Prompts far from those phrasings may behave unpredictably.
• No formal evaluation. Quality assessments below are qualitative, from operators observing rollouts on the physical arm. There are no success-rate numbers.
• Not safety-rated. Supervise all physical execution and keep the workspace clear.

Upstream licensing & attribution

This model is a derivative work of Apache-2.0 licensed components:

Component	Upstream	License
LeRobot framework	https://github.com/huggingface/lerobot	Apache-2.0
lerobot/smolvla_base	https://huggingface.co/lerobot/smolvla_base	Apache-2.0

Apache-2.0 permits relicensing derivative works. We retain the upstream copyright notices, license text, and NOTICE files for the incorporated material, as Apache-2.0 Section 4 requires. The upstream components remain under Apache-2.0 — only this project's own contributions (the fine-tuned weights and training configuration) are offered under CC BY-SA 4.0.

CC BY-SA 4.0 was chosen because it is share-alike: derivatives must be released under the same licence, so this work cannot be taken closed-source. The project's source code lives in a separate repository under its own licence — see https://github.com/Project-IRA/interactive-robotic-arm.

Citation

@misc{project_ira_2026,
  title        = {Project-IRA: Interactive Robotic Arm},
  author       = {Baten, Cleo and Keppler, Bela and Sapper, Jonas},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/Project-IRA}},
  note         = {Code: \url{https://github.com/Project-IRA/interactive-robotic-arm}}
}