StarVLA/VLAct-Qwen3VL4B-OFT-RoboDojo

VLAct QwenOFT Qwen3-VL-4B for RoboDojo (100K Fine-tuned)

This repository contains a RoboDojo-fine-tuned StarVLA QwenOFT policy. It was fine-tuned for 100,000 steps on the 35-task RoboDojo LeRobot v2.1 mixture using 2 nodes x 8 GPUs. The uploaded checkpoint is the exact steps_100000_pytorch_model.pt used for the evaluation reported below. It is a complete framework state dict, not a LoRA or adapter-only release.

Initialization

VLAct pretrained describes only the initialization source. RoboDojo fine-tuning started from the 100K-step 0706 multi-embodiment, three-head VLAct pretraining run. The checkpoint released here is the resulting RoboDojo fine-tune; it is not the raw VLAct pretraining checkpoint.

Model details

Item	Value
Framework	StarVLA QwenOFT
VLM backbone	StarVLA/Qwen3-VL-4B-Instruct-Action
Source initializer	0706_agilex_franka_5data_manualvel_balance33_66_50_vlm3_qwen3OFT_GR00T_PI_xrobot_3head_agilex_franka, step 100K
Initialization setting	reload_modules: qwen_vl_interface; training starts from step 0 (is_resume: false)
Fine-tuning dataset	robodojo_v21_all_h50_q99, 35 training tasks
Observation	Three RGB cameras at 224 x 224 plus 14D proprioceptive state
Action	14D absolute joint position (abs_qpos)
Action horizon	50
Action model	QwenOFT diffusion model, 16 layers
Repeated diffusion steps	8
Inference diffusion steps	4
Evaluation execution horizon	16
Fine-tuning checkpoint	100,000 steps
Fine-tuning hardware	2 nodes x 8 GPUs
Batch size	16 per GPU, global batch 256

The original run used the action-extended Qwen3-VL snapshot at revision 8c99a9a3db659019e6e785add928fcea8e14a18a. The packaged config.yaml uses the public Hub model ID so that the framework can reconstruct the model. training_config.original.yaml preserves the exact training configuration and local provenance paths used by the run.

Training details

Setting	Value
Training seed	42
Frozen modules	None
VLA / VLM loss scale	1.0 / 0.0
Gradient accumulation	1
Gradient checkpointing	Enabled
Optimizer	AdamW, betas (0.9, 0.95), epsilon 1e-8
VLM learning rate	1e-5
Qwen-VL interface learning rate	1e-5
Action-model learning rate	1e-4
Schedule	Cosine, 5,000 warmup steps, minimum LR 5e-7

RoboDojo results

Values are shown as success rate (SR, %) and score; higher is better. Group values are unweighted means over their tasks, and Average is the unweighted mean over all 42 tasks.

Evaluation group	SR (%)	Score
Average	8.17	11.28
Generalization	4.72	8.18
Precision	18.33	23.44
Long-Horizon	15.42	21.29
Memory	0.56	0.67
Open	1.67	1.71

Evaluation protocol

• 42 tasks, seeds 0/1/2, 10 episodes per task and seed: 1,260 episodes total.
• Each task therefore has 30 evaluated episodes.
• Each Generalization task uses 5 standard and 5 randomized episodes per seed, preserving the official standard/random split at one-fifth scale.
• All other tasks use 10 episodes per seed.
• Evaluation uses state (include_state=true), arx_x5 normalization, three camera inputs, and executes 16 actions before requesting the next chunk.
• Task SR is successful episodes divided by 30. Task score is the mean episode score multiplied by 100.

This is a scaled local reproduction protocol. It is not the official 50-episode-per-task / 2,100-episode protocol, and the values should not be presented as a verified official leaderboard submission.

Per-task results

Group / task	SR (%)	Score
Generalization	4.72	8.18
stack_bowls	26.67	31.67
push_T	0.00	0.00
pack_objects_into_box	0.00	5.33
fold_clothes	0.00	4.67
hang_mugs	3.33	7.83
sweep_blocks	0.00	0.00
pour_liquid_into_cup	6.67	6.67
make_toast	0.00	5.83
arrange_largest_number	0.00	2.17
sort_nesting_dolls_by_size	3.33	3.33
store_laptop_and_headphones	3.33	11.33
stack_blocks	13.33	19.33
Precision	18.33	23.44
fasten_screws	0.00	5.67
plug_in_charger	10.00	10.00
insert_tubes	56.67	66.00
pour_balls_into_vase	16.67	16.67
play_Xylophone	0.00	0.00
deposit_coin	10.00	14.67
insert_key	0.00	14.50
build_tower	53.33	60.00
Long-Horizon	15.42	21.29
put_bottles_into_dustbin	63.33	71.33
fill_pen_holder	0.00	12.00
classify_objects	0.00	4.83
play_tic_tac_toe	0.00	4.33
fill_egg_holder	0.00	2.00
organize_table	3.33	19.17
make_kong	56.67	56.67
play_stacking_toy	0.00	0.00
Memory	0.56	0.67
cover_blocks	0.00	0.00
match_and_pick_from_conveyor	3.33	3.33
swap_blocks	0.00	0.00
swap_T	0.00	0.00
press_by_number	0.00	0.00
imitate_sorting_sequence	0.00	0.67
Open	1.67	1.71
align_blocks	0.00	0.00
general_pickup	13.33	13.33
stack_blocks_by_language	0.00	0.00
solve_equation	0.00	0.00
classify_objects_by_language	0.00	0.33
pick_from_conveyor_by_image	0.00	0.00
store_tools_in_toolbox	0.00	0.00
pour_by_language	0.00	0.00

Machine-readable values and protocol metadata are in evaluation_results.json.

Files

README.md
config.yaml
dataset_statistics.json
evaluation_results.json
summary.jsonl
training_config.original.yaml
checkpoints/
└── steps_100000_pytorch_model.pt

config.yaml and dataset_statistics.json must remain two directory levels above the checkpoint file; StarVLA resolves both relative to the checkpoint.

Inference

Download the repository, then start the StarVLA model server:

huggingface-cli download StarVLA/VLAct-Qwen3VL4B-OFT-RoboDojo \
  --local-dir VLAct-Qwen3vl4b-OFT-RoboDojo

CKPT=VLAct-Qwen3vl4b-OFT-RoboDojo/checkpoints/steps_100000_pytorch_model.pt
python deployment/model_server/server_policy.py \
  --ckpt_path "$CKPT" \
  --port 57700 \
  --use_bf16

RoboDojo evaluation must match the training-time input contract:

export STARVLA_CKPT_PATH="$CKPT"
export STARVLA_INCLUDE_STATE=True
export STARVLA_UNNORM_KEY=arx_x5
export STARVLA_EXECUTE_HORIZON=16

Changing state usage, camera ordering, normalization statistics, action order, or execution horizon can materially change the result.

Intended use

This checkpoint is intended for research on RoboDojo with the ARX X5 dual-arm embodiment. Performance under different camera calibration, state/action ordering, normalization, robot embodiments, or real hardware has not been established.