Wukong (悟空): An Open Spatial-Reasoning Vision-Language-Action Model

Wukong is an open-source Vision-Language-Action (VLA) model focused on spatial reasoning for embodied agents. It is designed for robots that must understand, remember, and reason about 3D space from egocentric visual streams — and then act in it.

Named after the Monkey King (Sun Wukong) of Chinese mythology — famed for his piercing "fiery eyes" (火眼金睛) that see through the world — Wukong aims to give embodied agents a comparably sharp sense of space.

What's in This Release (v0.1, T2 stage)

This repository currently hosts Wukong-T2-Adapter: a QLoRA adapter (rank 16, 51.5M trainable params, 206MB) on top of unsloth/Qwen2.5-VL-7B-Instruct, fine-tuned for visual-spatial question answering from egocentric images.

• adapter_config.json + adapter_model.safetensors — the trained adapter (final checkpoint, step 49730)
• Tokenizer / preprocessor / chat template files — everything needed to load and chat
• eval_results.jsonl — per-item evaluation records (question, reference, prediction, score) for full reproducibility

Training setup: 198,919 samples (195,919 VSI-style spatial QA derived from Hypersim + 3,000 programmatic egocentric↔allocentric coordinate-transform QA), 2 epochs, total batch size 8, ~18.5 hours on a single RTX 5090 with Unsloth. Final training loss ≈ 0.19.

Roadmap stages beyond T2 (semantic waypoint prediction, embodiment-specific action adapters, world-action predictive pathway) are under active development and will be released as separate artifacts.

Evaluation

Held-out spatial QA split from our own data pipeline (never seen in training, n=500), scored in VSI-Bench style: exact match for multiple-choice items; MRA (max(0, 1 − |pred−ref|/ref)) for numeric items.

Question type	n	Score
relative_direction_camera	57	0.877
relative_distance_object	130	0.846
relative_size_object	23	0.826
relative_direction_object	108	0.815
absolute_count	38	0.805
absolute_size_object	27	0.758
relative_distance_camera	4	0.750
relative_count	23	0.739
absolute_distance_object	56	0.610
absolute_direction_object	34	0.551
TOTAL	500	0.782

Breakdown by answer format: multiple-choice (n=345) accuracy 0.832; numeric (n=155) MRA 0.671.

Benchmark	What it measures	Status
VSI-style spatial QA (own held-out split)	Visual-spatial intelligence from egocentric images	0.782 (see above)
VSI-Bench (official)	Visual-spatial intelligence from egocentric video	Planned
SPAR-Bench	3D spatial perception and reasoning	Planned
R2R-CE / RxR-CE (VLN-CE, Habitat)	Instruction-following navigation (SR / SPL)	Planned
Real-robot trials (Unitree)	Sim-to-real transfer	Planned

Note: the reported 0.782 is measured on our own held-out split, not the official VSI-Bench; official-benchmark evaluation is on the roadmap.

Usage

from unsloth import FastVisionModel

model, tokenizer = FastVisionModel.from_pretrained(
    "unsloth/Qwen2.5-VL-7B-Instruct",
    load_in_4bit=True,
    max_seq_length=4096,
)
model.load_adapter("FINWHALE/Wukong")   # adapter files at repo root
FastVisionModel.for_inference(model)

A FastAPI serving wrapper and an evaluation script are available in the companion training pipeline (to be linked from this card).

Intended Use

• Spatial question answering and measurement estimation from egocentric images
• Research on spatial intelligence, world models, and embodied reasoning
• A foundation for downstream fine-tuning on specific robot embodiments (navigation adapters on the roadmap)

Limitations

• Metric-precision spatial estimation (exact distances, absolute cardinal directions) remains the weakest capability (0.55–0.61) — an open challenge shared by all current VLM/VLA systems.
• This release is a spatial-reasoning VLM stage: it does not output motor commands. Closed-loop navigation requires the upcoming waypoint/action adapters.
• Long-horizon imagination may drift; closed-loop correction with real observations is required in deployment.
• Not intended for safety-critical deployment without additional validation.

Citation

@misc{wukong2026,
  title  = {Wukong: An Open Spatial-Reasoning Vision-Language-Action Model},
  author = {FINWHALE},
  year   = {2026},
  url    = {https://huggingface.co/FINWHALE/Wukong}
}

License

Apache 2.0. Please also comply with the licenses of the underlying base models and datasets.