PanoVLM1.5-500M

PanoVLM1.5 is a vision-language model: a FastViT-HD vision encoder (timm) feeding the PanoLM1.5-400M causal LM through a linear projector. The LM is a hybrid: 20 of its 24 layers use KDA linear attention, the other 4 use multi-head latent attention (MLA).

• Type: Vision-Language (image-text-to-text) Model
• LM: PanoLM1.5-400M
• Vision encoder: FastViT-HD (timm), NCHW input
• Projector: linear (encoder dim -> LM dim)
• Default image resolution: 768x768, the resolution it was trained at (pad-resized)

Parameters

Component	Parameters
PanoLM1.5 LM	388.5 M
FastViT-HD vision encoder	122.7 M
Projector	3.1 M
Total	514.3 M

Requirements

torch==2.12.0
transformers==5.8.1
flash-linear-attention==0.5.0
timm==1.0.25

Usage

import requests
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

repo_id = "PanocularAI/PanoVLM1.5-500M"
model = AutoModelForImageTextToText.from_pretrained(
    repo_id,
    trust_remote_code=True,
).cuda()  # fla's RMSNorm uses Triton kernels that only run on CUDA tensors.
processor = AutoProcessor.from_pretrained(repo_id, trust_remote_code=True)

url = "https://llava-vl.github.io/static/images/view.jpg"
image = Image.open(requests.get(url, stream=True).raw)

# Put the <|image|> placeholder inline in the message text (the processor expands
# it into the image tokens). processor.apply_chat_template strips the whitespace
# around the placeholder, as the training data pipeline did.
messages = [{"role": "user", "content": "<|image|>\nAre there mountains in the image?"}]
prompt = processor.apply_chat_template(
    messages,
    add_generation_prompt=True,
    tokenize=False,
)
inputs = processor(text=prompt, images=image, return_tensors="pt").to(model.device)

out = model.generate(**inputs, max_new_tokens=512)
print(processor.decode(out[0], skip_special_tokens=True))

Evaluation

Evaluated with lmms-eval (commit 7439e2bf): greedy decoding, each task's default prompt and answer length, batch size 16. Answer parsing is lenient about format only: one trailing period is stripped, and the option letter is read out of answers like "Answer: B".

Benchmark	Score
POPE (F1)	83.3
MME (/2800)	1109.5
HallusionBench	25.8
COCO captions (CIDEr)	93.9
NoCaps (CIDEr)	84.2
GQA (testdev)	42.6
TextVQA (val)	42.5
RealWorldQA	36.7

• Metrics. POPE is F1 over all 9,000 questions (accuracy is 82.8). MME is the full score, perception (880.9 of 2,000) plus cognition (228.6 of 800), with lmms-eval's default instruction. HallusionBench is the mean of question, figure and question-pair accuracy, with the instruction "Answer yes or no." and rule-based yes/no extraction (no GPT judge). COCO (2017 val) and NoCaps (val) are CIDEr with lmms-eval's one-sentence caption prompt. GQA, TextVQA and RealWorldQA are accuracy, with lmms-eval's default prompts.