StarVLA/VLAct_Qwen3OFT_Robotwin_Finetune

VLAct · Qwen3-VL-4B OFT · RoboTwin 2.0 (Base / Clean)

This repository contains the 50K-step VLAct downstream fine-tuning checkpoint for RoboTwin 2.0 Base (Clean), introduced in Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models. It starts from StarVLA/VLAct_Qwen3_Pretrain and adapts the shared VLAct backbone with a randomly initialized OFT action head for the target benchmark.

This is a StarVLA training / evaluation checkpoint, not a standard transformers.AutoModel package. Load it with the matching StarVLA framework (QwenOFT) and the packaged config.yaml / dataset_statistics.json. Safe robot deployment still requires embodiment-specific action mapping, normalization, camera calibration, control-rate handling, workspace constraints, and independent safety systems.

What is VLAct?

VLAct is a representation-centric continued-pretraining recipe for vision-language-action models. It preserves the VLM prior, co-trains multiple continuous action heads on a shared latent, and shares action semantics across embodiments with a partially unified padded layout and wrap-aware joint loss. Downstream policies discard the pretraining heads, randomly initialize a target action head, and fine-tune from the VLAct backbone.

The complete method, ablations, and evaluation protocols are documented in the paper and code repository.

Checkpoint details

Item	Value
Framework	StarVLA QwenOFT
Base VLM	StarVLA/Qwen3-VL-4B-Instruct-Action
Pretrained backbone	StarVLA/VLAct_Qwen3_Pretrain @ 100K
Action head	OFT (DiT-B), randomly initialized at fine-tuning start
Action representation	Continuous absolute joint (action_mode: abs)
Action dimensions	14-D
Action horizon	32 steps (future_action_window_size: 31)
Dataset mix	robotwin_wrap_32
Data root	./playground/Datasets/RoboTwin-Clean
Image size	224×224, buckets [[320,180],[280,210]]
Training step	50,000
Per-GPU VLA batch size	16
Optimizer	AdamW, 5K-step warmup, cosine schedule (min_lr: 5e-7)
Learning rates	Qwen-VL interface 1e-5; action/base modules 1e-4
Seed	42

Fine-tuning data

Fine-tuning uses the RoboTwin 2.0 Clean / Base mixture (robotwin_wrap_32) on AgileX absolute-joint trajectories. The CoT prompt asks the policy to ground key objects with bounding boxes before acting. Wrap-aware angular-joint loss is enabled.

Recommended use: download and evaluate

1. Install StarVLA / VLAct

git clone https://github.com/starVLA/VLAct.git
cd VLAct

conda create -n vlact python=3.10 -y
conda activate vlact

# Install a CUDA-compatible PyTorch build first.
python -m pip install -r requirements.txt
python -m pip install flash-attn==2.7.4.post1 --no-build-isolation
python -m pip install -e .

2. Download the checkpoint

Run from the VLAct repository root:

hf download StarVLA/VLAct_Qwen3OFT_Robotwin_Finetune \
  --local-dir playground/Pretrained_models/VLAct-Qwen3VL4B-OFT-Robotwin

The checkpoint path is then:

playground/Pretrained_models/VLAct-Qwen3VL4B-OFT-Robotwin/checkpoints/steps_50000_pytorch_model.pt

Keep the downloaded directory structure unchanged. StarVLA resolves config.yaml and dataset_statistics.json from the run directory two levels above the checkpoint file.

Action un-normalization

The policy predicts actions normalized to roughly [-1, 1]. StarVLA maps them back to physical units using the q01 / q99 / mask statistics stored in dataset_statistics.json, and the unnorm_key selects which statistics block to use.

This run packages a single key, new_embodiment. StarVLA resolves it automatically when only one key is present, so you normally do not need to set anything. If your evaluation config exposes an unnorm_key field (for example DOMINO's examples/DOMINO/eval_files/deploy_policy.yml or the eval config generated by the RoboTwin launcher), set it to new_embodiment.

Changing the normalization statistics, camera ordering, state usage, action ordering, or execution horizon can materially change results.

3. Evaluate with the StarVLA / VLAct scripts

Follow the RoboTwin evaluation guide:

RoboTwin evaluation runs as two processes, a StarVLA policy server and the RoboTwin simulator. See examples/Robotwin/README.md for the environment setup and the exact launch order.

To reproduce training, start from the RoboTwin OFT launcher and switch it to the Base (Clean) setting, since the shipped script defaults to the larger All mixture:

bash scripts/run_scripts/RoboTwin/train_robotwin_qwen3oft.sh

Override these variables at the top of the script:

data_mix=robotwin_wrap_32
data_root_dir=./playground/Datasets/RoboTwin-Clean
run_id=VLAct_Qwen3OFT_Robotwin_Finetune

and set --trainer.max_train_steps 50000 with --trainer.save_interval 10000.

Loading the policy

Reconstruct the policy with the matching StarVLA framework (QwenOFT) and the packaged configuration. The .pt file contains model parameters only; it does not package optimizer or scheduler state.

This checkpoint is intended for evaluation or further fine-tuning on the same embodiment / action contract. Transferring it to a different robot, camera setup, or action space usually requires additional adaptation.

Files

VLAct-Qwen3VL4B-OFT-Robotwin/
├── README.md
├── config.yaml
├── training_config.original.yaml
├── dataset_statistics.json
├── summary.jsonl
└── checkpoints/
    └── steps_50000_pytorch_model.pt

File	Purpose
checkpoints/steps_50000_pytorch_model.pt	Fine-tuned PyTorch state dict for the downstream policy
config.yaml	Portable resolved configuration using the public base-model ID
training_config.original.yaml	Original resolved run configuration as produced by training; it records the internal base-model ID used at training time and is kept for provenance only
dataset_statistics.json	Dataset statistics used by StarVLA normalization utilities
summary.jsonl	Saved-checkpoint step history

Checkpoint SHA-256:

e3823ca683057371ab94d4d66434538e2e2a53b861a87ee77a1981f941209873

Benchmark results

This checkpoint matches the head (OFT) and setting of the VLAct RoboTwin 2.0 Base, Clean result reported in the paper:

Setting	VLAct (OFT head)	Matched Qwen3-VL-OFT baseline
RoboTwin 2.0 Base, Clean	80.5%	61.7%

For the larger Data-Scaling setting (92.5% / 90.8%), use VLAct_Qwen3OFT_Robotwin_all_Finetune.

Use the paper and RoboTwin evaluation scripts for the exact protocol.

Intended use and limitations

This checkpoint is intended for research on VLA representation transfer and benchmark evaluation on RoboTwin 2.0 Base (Clean).

• It has not been validated as a universal zero-shot policy across arbitrary robots.
• Safe deployment requires embodiment-specific action mapping, normalization, camera calibration, control-rate handling, workspace constraints, and independent safety systems.
• Performance depends on evaluation protocol, simulator / real-robot setup, observation configuration, and action execution settings.
• The model may inherit limitations and biases from its base VLM, the VLAct pretraining mixture, and the downstream fine-tuning data.

Citation

@misc{yang2026vlact,
  title   = {Beyond Data Scaling: Representation-Centric Continued Pre-training
             for Vision-Language-Action Models},
  author  = {Yang, Senqiao and Wang, Chengyao and Chen, Yuxin and Wang, Zixuan and
             Tang, Longxiang and Gui, Haokun and Ye, Jinhui and Lu, Changsheng and
             Wu, Xiaoyang and Zhu, Mingkang and Chen, Pengguang and Liu, Shu and
             Tian, Zhuotao and Zhao, Hengshuang and Yu, Bei and Jia, Jiaya},
  year    = {2026},
  month   = aug,
  note    = {Preprint},
  url     = {https://starvla.github.io/VLAct/}
}

License and acknowledgements

The checkpoint is released under the Apache License 2.0. The VLAct code repository is released separately under the MIT License. Users must also comply with the licenses and terms of the base model, the VLAct pretraining checkpoint, and the training / evaluation datasets.

VLAct builds on StarVLA, LeRobot, GR00T, and Qwen3-VL.

For questions, email yangsenqiao.ai@gmail.com or open an issue in the VLAct repository.