smolvla_tactile_charger_20ep

SmolVLA vision-language-action policy — vision + tactile — fine-tuned from lerobot/smolvla_base on 20 episodes.

Task / language instruction: grab and remove the charger from the socket and put it in the black box

Trained with LeFlexiTac. Docs: https://tna001-ai.github.io/LeFlexiTac/docs.html

Training data

source dataset	aryankakad/tactile_charger_inserting
episodes	20 (indices 0-19)
frames	10,717 @ 30 fps
cameras	top -> camera1, gripper -> camera2 (see rename_map below)
tactile	observation.tactile.primary (12x32)

Task string was retagged. The source dataset is labelled stack cup (carried over from an earlier recording session). SmolVLA is language-conditioned, so it was retrained on a copy retagged to the real instruction above. Use that instruction at inference.

Camera rename is required. smolvla_base expects camera1/2/3; this dataset uses top/gripper. Training passed:

--rename_map='{"observation.images.top": "observation.images.camera1", "observation.images.gripper": "observation.images.camera2"}'

You must pass this again at inference. The mapping is saved in policy_preprocessor.json, but lerobot-record overwrites it with its own --dataset.rename_map (which defaults to empty) at src/lerobot/scripts/lerobot_record.py:522. Without the flag the policy reports camera1/camera2 as missing features:

--dataset.rename_map='{"observation.images.top": "observation.images.camera1", "observation.images.gripper": "observation.images.camera2"}'

Adjust the left-hand keys if your cameras are named differently.

Configuration

base	lerobot/smolvla_base
steps	19,200
batch size	64
epochs	114.7
trainable params	~101M of ~451M (VLM backbone frozen)
mixed precision	bf16
chunk_size	50
n_action_steps	50
vlm_model_name	HuggingFaceTB/SmolVLM2-500M-Video-Instruct
optimizer_lr	0.0001
freeze_vision_encoder	True
n_tactile_tokens	4
tactile_feature_dim	256

Epoch-matched at ~114.4 epochs with every other model in this series.

Training command actually used

python -u -m lerobot.scripts.lerobot_train \
  --dataset.repo_id=Dimios45/tactile_charger_retagged --dataset.root=/mnt/data/mritunjoyh/data/datasets/tactile_charger_retagged --dataset.episodes=[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19] \
  --policy.path=lerobot/smolvla_base --policy.use_tactile=true --policy.tactile_features='["observation.tactile.primary"]' --policy.n_tactile_tokens=4 \
  --policy.repo_id=Dimios45/smolvla_tactile_charger_20ep \
  --rename_map='{"observation.images.top": "observation.images.camera1", "observation.images.gripper": "observation.images.camera2"}' \
  --policy.private=true --policy.device=cuda \
  --output_dir=outputs/train/G_smolvla_tactile_20 --job_name=G_smolvla_tactile_20 \
  --batch_size=64 --num_workers=8 --steps=19200 --save_freq=5000 --wandb.enable=true

Evaluation / rollout

Not run here (no robot on the training machine). Load with --policy.path=Dimios45/smolvla_tactile_charger_20ep and prompt with the instruction above.

Reference: the lerobot-record eval invocations in tactile_cmd.txt and the project docs.

Notes

• Trained on AMD MI300X (ROCm 6.2.4). cudnn.benchmark must stay off on ROCm or MIOpen runs an exhaustive search before step 1; persistent_workers=True avoids ~410 s stalls at epoch boundaries.
• Training loss is not a proxy for task success - compare by rollout success rate.