VisionLang

A research on LLaVA-style VLM training, evaluation, hallucination diagnosis, and GRPO alignment.

Models

Model	Description
vlm-3b-okvqa-sft	3B OK-VQA two-stage SFT (3 seeds = 0.179 ± 0.027)
vlm-3b-okvqa-grpo	GRPO-aligned open-ended OK-VQA (full 0.2299 / held-out 0.207)
vlm-3b-okvqa-qabal	Balanced yes/no model (fixes policy collapse; POPE)
vlm-0.5b-clip-captioning	0.5B COCO captioning (CIDEr 1.020)
vlm-0.5b-qa-clip	0.5B recognition / POPE evaluation (random 0.83)
mae-imagenet100-tiny	MAE self-supervised pretrained encoder
retrieval-coco2017-v2	Image-text retrieval dual encoders (3 inits × 3 seeds)
visionlang-checkpoints/	Full archive: all seeds, ablations and datasets

Architecture

ViT-B/16 (frozen; CLIP or MAE initialization) + 2-layer MLP projector + Qwen2.5-3B / Qwen2-0.5B + LoRA (r=64, α=128). Base LLMs are loaded from Qwen/Qwen2-0.5B and Qwen/Qwen2.5-3B.

Training

• Two-stage SFT: stage-1 projector, stage-2 LoRA + projector.
• GRPO: rule reward + within-group advantage normalization + KL constraint (β=0.05), 150 steps on 300 disjoint prompts.

Evaluation

OK-VQA (5,046 questions), GQA testdev_balanced (12,578 questions), POPE (random / popular / adversarial), COCO2017 captioning (CIDEr / BLEU / METEOR / SPICE).