SleepMastger/dish-utensil-lingbot

Dish-utensil LingBot-VA

Public archival bundle for the dish-utensil LingBot-VA model trained by Amin (Slurm job 1713, dish_utensil_base: fine-tuned from the robot-co-pretrained lingbot-va-base checkpoint on 100 Franka teleop episodes — 50 fork + 50 spoon, 10 Hz, two 256x256 cameras). This bundle contains the completed step 3,000 checkpoint. It has not yet been evaluated on a robot.

Checkpoints mirror the existing WAM repository layout:

• checkpoints/weights/step_000500.safetensors — SHA-256 74bedd53dc70337920c1b051d810f5c597b1cb874c3e9dabafc795931dd9fa91
• checkpoints/weights/step_001000.safetensors — SHA-256 f793e9b412b3251d05f5bfd6bc7a203f47454d47a87ec65294cedfc215af1ec4
• checkpoints/weights/step_001500.safetensors — SHA-256 3a8dd3b653497ed35febade7ab42c23da8faa76ea4a83d237edca92de7c0fadb
• checkpoints/weights/step_002000.safetensors — SHA-256 bbba5a0ee3ca84552d83ca4c3c90706b3d2894dbbff36ddc11a340c0d7f3f8c8
• checkpoints/weights/step_002500.safetensors — SHA-256 e88a75dea2c29c0827afb9580ad85f0c34ba40373936913e6f05b84788049650
• checkpoints/weights/step_003000.safetensors — SHA-256 203b50ca9dfadb55990349964963d786e87cafce855918154bfb9df9de33ded6

Conditioning

Exact task text (must byte-match at deployment):

pick the dishes out of the rack and place them on the table, then pick and place the spoon to the gray dish and fork to the pink dish

⚠️ This differs from the SleepMastger/dish-utensil-*-v2 fastwam/flashwam repos, whose task string says "spoon in the gray dish and fork in the pink dish". This LingBot run was trained with "to".

• conditioning/text_embedding.pt: the exact UMT5 embedding used by every training segment, shape [512, 4096], bfloat16.
• conditioning/empty_text_embedding.pt: the empty-string UMT5 embedding used for classifier-free guidance (cfg_prob: 0.1 during training).

Both were produced with the Wan2.2 Diffusers tokenizer/text encoder; the generation code is included in training_code/extract_latents.py.

Training recipe

Phase-2 fine-tune recipe shared with lift_new_base / place_cube_bowl_base / fruit_pick_base: 3,000 steps, lr 1e-5, global batch 32, warmup 10, weight decay 0.1, checkpoint every 500 steps. Action is 7-D (dx dy dz drx dry drz gripper); the gripper channel is absolute with real units {0.0=closed, 1.0=open}, thresholded at 0.5 (per the training config — verify polarity on the robot before trusting it). Actions occupy channels 0-6 of a 30-channel tensor, normalized from task q01/q99 to [-1, 1] and clipped to [-1.5, 1.5].

The exact config (training_code/dish_utensil_config.py), dataset processor, latent extractor, model mask implementation, and the task-string relabeling script are included under training_code/. No license is asserted here for the bundled upstream code; its original terms continue to apply.

Base components

This is a fine-tuned transformer checkpoint, not a standalone pipeline. It requires the matching LingBot-VA/Wan2.2 VAE and scheduler code. The transformer configuration is included as config.json.