Masar

Masar takes a first-person outdoor photo and an instruction such as "Go to the bench on the left" and returns the path a person would walk, as a list of points in the image.

It is a LoRA adapter for Qwen3-VL-8B-Instruct, trained only on synthetic images rendered in Blender (VLN Synthetic Scenes).

Results on NaviTrace (Human embodiment)

Scores on the 500 Human scenarios of the NaviTrace validation split, using NaviTrace's scoring code. 0 is walking straight ahead and 100 is the expert path.

Category Qwen3-VL-8B before training Masar
Overall -28.3 36.9
Dynamic Obstacle -25.0 50.5
Stationary Obstacle -23.7 41.1
Semantic Terrain -24.0 39.7
Geometric Terrain -27.8 36.1
Accessibility -34.1 32.8
Visibility -27.6 30.3
Social Norms -59.8 24.1

For reference, Human-only means on the NaviTrace leaderboard are 75.5 for a human expert, 35.4 for Gemini 2.5 Pro, 21.1 for GPT-5 and 19.3 for Qwen3-VL-235B. Those are measured on the hidden test split. Masar is measured on the validation split and has not been submitted.

How to use

import importlib.util
import torch
from huggingface_hub import hf_hub_download
from PIL import Image
from peft import PeftModel
from transformers import AutoProcessor, Qwen3VLForConditionalGeneration

base = "Qwen/Qwen3-VL-8B-Instruct"
processor = AutoProcessor.from_pretrained(base)
model = Qwen3VLForConditionalGeneration.from_pretrained(base, torch_dtype=torch.bfloat16, device_map="cuda")
model = PeftModel.from_pretrained(model, "ahd03/masar").eval()

# NaviTrace's prompts, the same ones used in training
spec = importlib.util.spec_from_file_location("prompts", hf_hub_download("ahd03/masar", "prompts.py"))
prompts = importlib.util.module_from_spec(spec)
spec.loader.exec_module(prompts)

image = Image.open("photo.jpg").convert("RGB")
image.thumbnail((896, 896))
messages = prompts.messages(image, "Go to the bench on the left")

text = processor.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = processor(text=[text], images=[image], return_tensors="pt").to("cuda")
out = model.generate(**inputs, max_new_tokens=256, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0])
# [[0.500, 0.950], [0.470, 0.880], ..., [0.310, 0.560]]   x, y from 0 to 1, origin top-left

Training

Setting Value
Base model Qwen3-VL-8B-Instruct, vision encoder frozen
Adapter LoRA rank 16, alpha 32, dropout 0.05, on the language model's attention and MLP projections
Data VLN Synthetic Scenes, train split (8,392 samples)
Optimisation 2 epochs, learning rate 1e-4 with cosine decay, batch 16, bf16, images at 896 px
Prompts NaviTrace's system and user prompts

Limitations

  • Human embodiment only.
  • Trained on rendered scenes with simple people and static traffic. Social Norms is the weakest category.
  • Single-image input, no memory of earlier frames.
Downloads last month
70
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for ahd03/masar

Adapter
(259)
this model

Datasets used to train ahd03/masar