Fichtl00/Unifolm-VLA_mix

UnifoLM-VLA — G1 Dex3 Block-Stacking (Real + Synthetic Mix)

Full finetune (VLM backbone not frozen) of UnifoLM-VLA on the Unitree G1 + Dex3-hand block-stacking task, trained on a mix of real and simulation-generated (synthetic) demonstrations.

Companion model trained on real data only: Fichtl00/Unifolm-VLA_real.

Training data

Mixed via data_mix with a 3:1 relative sampling weight (~75% real / ~25% synthetic per epoch):

• Real: unitreerobotics/G1_Dex3_BlockStacking_Dataset
• Synthetic: Fichtl00/Cube_Stacking_synth_jointspace (Isaac Lab simulation, joint-space state/action matching the real dataset's 28-dim layout: 14 arm joints + 14 Dex3 hand joints per side, no waist)

Training setup

• Base VLM: unitreerobotics/UnifoLM-VLM-Base (Qwen2.5-VL)
• Full finetuning (backbone unfrozen) — the reason this ran on KISSKI HPC (4×A100 80GB) rather than locally
• DeepSpeed ZeRO Stage 2, bf16
• Global batch size 24 (per-device 6 × 4 GPUs)
• 30,000 steps, learning rate 4e-5
• Action/state: 28-dim joint space, action horizon 16

Final loss: action_dit_loss = 0.0076, mse_score = 0.0010 (step 30000).

Checkpoints

final_model/ plus intermediate checkpoints every 2000 steps (checkpoints/steps_2000 … steps_30000) for checkpoint selection / comparison against the real-only run.