PointmapVLA-Pretrained

The pretrained PointmapVLA model: the initialisation for every fine-tuned PointmapVLA checkpoint. Code, fine-tuning recipes and the evaluation harness are in the PointmapVLA repository.

What it is

PointmapVLA is a vision-language-action model built on the Qwen3.5-4B vision-language backbone with two additions:

• a pointmap encoder, a copy of the vision tower that consumes raw metric XYZ pointmaps in the robot-base frame (built from depth and camera parameters) and whose features are added patch-wise to the RGB tokens, and
• a 32-layer layer-wise flow-matching DiT action expert that reads the backbone's hidden states at every layer through per-layer projections and predicts a chunk of normalised actions (4 Euler steps at inference).

6.08 B parameters, bf16. This checkpoint has a generic 32-D action head (17 real dimensions: end-effector position 3, rotation as 6-D (first two rows of the rotation matrix) 6, gripper 1, joints 7, then padding) and a 17-D state head. It is not meant to be run as is: fine-tune it on your target robot and action layout.

Provenance

Two pretraining stages on 76.3 M pointmap-enriched robot frames and 10.7 M pointmap-paired 3D spatial VQA samples, global batch 368:

1. Stage 1, 50,000 steps: next-token prediction on the VQA data and discrete (tokenised) action prediction on the robot data.
2. Stage 2, 50,000 steps: flow-matching action prediction with the DiT expert, co-trained with the VQA data.

The pretraining data and code are not part of this release. The configuration stored in config.yaml records the architecture and the data recipe (prompt template, pointmap options, 256 x 320 pointmaps); the benchmark fine-tuning configs override the head dimensions and pointmap size.

Files

config.yaml                        architecture and data recipe (release schema)
dataset_statistics.json            pretraining action/state statistics (informational)
base_vlm/                          Qwen3.5-4B config, tokenizer and processor only; no weights
checkpoints/
  model-00001-of-00003.safetensors
  model-00002-of-00003.safetensors
  model-00003-of-00003.safetensors   ~11 GB in total, bf16
  model.safetensors.index.json

State-dict key groups: qwen_vl_interface.model.* (backbone), pointmap_encoder.*, action_model.* (DiT expert), project_layers.* (per-layer projections). The language model head is tied to the token embedding and stored once; the index records it under metadata.tied_weights and the loader re-ties it.

How to load

from starVLA.model.framework.VLM4A.PointmapVLA import PointmapVLA

model = PointmapVLA.from_pretrained("DAVIAN-Robotics/PointmapVLA-Pretrained")

from_pretrained also accepts a local directory or a weight file inside its checkpoints/. For fine-tuning, set trainer.pretrained_checkpoint in a training config to this repo id (the shipped configs already do) and run

scripts/train.sh examples/simBenchmarks/Robocasa_kitchen/train_files/pointmapvla_robocasa_kitchen.yaml
scripts/train.sh examples/simBenchmarks/LIBERO/train_files/pointmapvla_libero.yaml

On load into a fine-tuning config, the four action-head tensors whose shape depends on the action/state layout (state encoder input, action encoder input, action decoder output weight and bias) are remapped: the slots shared by both layouts (end-effector position, r6 rotation, gripper) are copied, the rest are freshly initialised. Everything else transfers unchanged. Four GPUs is the minimum for fine-tuning; see docs/training.md in the repository.

Evaluation

This checkpoint is not evaluated directly. The fine-tuned checkpoints derived from it:

Fine-tune	Benchmark	Paper	Our reproduction
PointmapVLA-RoboCasa	RoboCasa kitchen-24	78.2	77.8 ± 0.6
PointmapVLA-LIBERO	LIBERO (4 suites)	99.0	99.1 ± 0.1
PointmapVLA-LIBERO	LIBERO-Plus	92.4	92.4 ± 0.3

Intended use and limitations

• Research on 3D-aware vision-language-action models; initialisation for fine-tuning.
• Inputs: RGB views, a language instruction, a proprioceptive state and metric pointmaps in the robot-base frame (depth + intrinsics + camera-to-world extrinsics + base pose). Without depth there is no pointmap; the model was trained with pointmaps present.
• The 32-D / 17-D head layout is a pretraining convenience. Fine-tune before use on any robot; the released fine-tunes target simulation benchmarks only.
• This checkpoint is not a servable policy: the evaluation server needs the fine-tuning data registry to un-normalise actions, and the pretraining embodiments (interndata_m1, droid in dataset_statistics.json) are not part of this release. Use it only as trainer.pretrained_checkpoint.
• Not validated on physical robots in this release. Actions are predicted in normalised space and must be un-normalised with statistics of the fine-tuning data.

License

MIT (code and weights of this release). Weights are derived from Qwen3.5-4B, distributed under the Apache License 2.0. See NOTICE in the repository.

Citation

TBD. The paper is under double-blind review; the BibTeX entry and author list will be added at camera-ready.