mini-LLaVA

GitHub Repo - arnav-tyagii/mini-llava

A from-scratch reproduction of the LLaVA recipe at small scale, trained and evaluated end-to-end on a single 4 GB laptop GPU (RTX 3050 Ti).

• Frozen openai/clip-vit-base-patch32, penultimate layer, 49 visual tokens
• Trainable 2-layer MLP projector (768 → 896)
• Qwen/Qwen2.5-0.5B-Instruct, frozen, with LoRA (r=16, α=32) on attention + MLP

Three checkpoints are published, because the differences between them are the result:

File	What it is
stage2_mixed/	Recommended. LoRA + projector, stage-2 trained on mixed data
stage1_projection.pt	Projector after stage-1 caption alignment only
stage2_lora/	The first stage-2 run, on LLaVA-Instruct alone — kept because the regression it shows is the finding

Results

Measured on COCO val2017, which has zero overlap with training data. This matters: the Karpathy test split that captioning work normally uses is 88.1% contaminated here, because COCO train2017 absorbed 35,504 of the 40,504 val2014 images. Evaluating there would have been evaluating on training data.

Captioning — 5,000 held-out images

Config	CIDEr	BLEU-4	METEOR	ROUGE-L	mean words
Stage 1 only	94.28	16.15	25.39	52.53	9.75
Stage 1+2, LLaVA-Instruct only	6.86	7.24	24.28	33.73	33.62
Stage 1+2, mixed data	102.00	19.27	26.76	53.97	9.63

References average 10.44 words. On the 593 clean Karpathy-test images: 100.61.

POPE — object hallucination, 3,000 questions per split

With "Answer the question using a single word or phrase.":

Config	Split	Accuracy	Precision	Recall	Yes-ratio
Mixed	random	0.7703	0.9645	0.5613	0.2910
Mixed	popular	0.7520	0.9065	0.5620	0.3100
Mixed	adversarial	0.7363	0.8621	0.5627	0.3263
LLaVA-Instruct only	adversarial	0.5000	0.5000	1.0000	1.0000

Without that instruction the mixed model is still at chance (0.500, yes-ratio 0.999). The discrimination is real but format-conditioned.

VQAv2 — 2,000 questions, official soft accuracy

Config	Prompt	Official
Mixed	brief	57.79
Mixed	plain	0.00
LLaVA-Instruct only	either	0.00

By answer type: yes/no 77.00, number 44.11, other 45.81.

Not zero-shot. Training includes VQAv2 questions from train-pool images. Training on VQAv2 train and scoring on val is the standard protocol behind published VQAv2 numbers, but it should not be read as a zero-shot result. POPE's images, templates and object sampling were never trained on, though its question type was — the same status as LLaVA-1.5's POPE numbers.

Intended use and limitations

Use it for: short English captions of natural images, and single-word visual question answering when you append "Answer the question using a single word or phrase."

Measured limitations — these are numbers, not caveats:

• Needs the brevity instruction. Without it, POPE accuracy is 0.500 and official VQAv2 is 0.00.
• Over-predicts "no": yes-ratio 0.31 against a true 0.50, and recall 0.56, so it misses ~44% of objects that are actually present.
• Counting is weak: 44.11 on number questions.
• No OCR. ViT-B/32 at 224px cannot read text and the LLM invents it.
• No calibrated uncertainty — wrong details are asserted as confidently as correct ones.

Reproducing

Code, evaluation harness and every results JSON (each with its git SHA and generation config): https://github.com/Arnav-Tyagii/mini-LLaVA

python -m eval.run --benchmark contamination
python -m eval.run --benchmark caption -- --variant mixed
python -m eval.run --benchmark pope -- --variant mixed \
    --question-suffix " Answer the question using a single word or phrase."
python -m eval.ablation

Benchmarks use greedy decoding. Hardware, driver and CUDA versions are in HARDWARE.md.