Aspaklarya-122-cap-step1500

Tested and rejected. Published as a record of a refuted hypothesis, not as a recommended model. For captioning the untuned base model is still better than this adapter; for joint use the recommendation remains Aspaklarya-101-step4500.

The captioning half of exp121, a task-routed two-expert study. Its partner is ExperimentField/Aspaklarya-121-vqa-step3000. Numbered 122 to keep the two experts as separate lines on the hub.

What it was testing

Every prior Aspaklarya run trained VQA and captioning in a single adapter. If those objectives compete for adapter capacity, splitting them into specialists should let each exceed the mixture. This arm removes VQA entirely: 11,993 caption items, 1 caption per image, 0 QA.

Scoring is routed MEAN4 - VQA slices read off the VQA expert, caption slices off this one, as if a router dispatched each query to its specialist.

Result: null, and the hypothesis is refuted

checkpoint	CAP_DM	CAP_RSICD	sum
step750 (2 ep)	81.55	79.29	160.84
step1125 (3 ep)	81.75	79.66	161.41
step1500 (4 ep)	81.51	80.04	161.55
exp101-step4500 (joint adapter)	82.53	79.64	162.17
exp020 (untuned base)	82.79	79.99	162.78

Caption-only training made captioning worse, not better: ~1.3 below the joint adapter and ~1.2 below the untuned base, with CAP_DM about 5 sigma under the base (per-slice sigma 0.25).

Combined with the VQA expert (74.22 / 69.47), routed MEAN4 is 76.31, against a pre-registered win gate of 77.34 and exp101's single-adapter 76.72. Two specialists are worse than one generalist, while costing 2x adapter memory plus a routing decision per request.

So the mixture was not holding captioning back - it was helping it. Task interference is not what caps this model.

The interference is asymmetric

trained on	own task	off-task collapse
captions only (this model)	CAP_RSICD 80.04	VQA_DM 63.52 (degraded)
VQA only (121-vqa)	VQA_DM 74.22	CAP_RSICD 34.11 (destroyed)

Caption-only training leaves VQA degraded but functional; VQA-only training destroys captioning outright. Caption data appears to be the more general signal of the two.

Training

base	Qwen/Qwen3-VL-8B-Instruct
data	VLR_Dataset ImageTable, train+other splits
items	11,993 (1 caption per image, 0 QA)
caption prompts	sampled from a varied query pool, so the model learns the task rather than one prompt string
epochs	4 (1,500 steps, save every 375)
LoRA	r=16, alpha=16, dropout 0, vision tower trained
optim	lr 2e-4 linear, warmup 0.03, wd 0.001, grad clip 0.3
batch	4 x 8 grad accum, seq 2048, max image side 1024
seed	3407
final train loss	1.121
hardware	1x A100 40GB, 206 min

Exposure is matched to exp101, which saw every caption item 4 times across 4 epochs over the full mixture. This arm also runs 4 epochs over its own subset, so batch composition is the only difference.

Evaluation

500 held-out validation items per slice, fixed seed across all experiments, generated on both harnesses and graded by an LLM judge (gemma-4-31b-it-imiai) for agreement with the reference caption. The judge is text-only: it measures agreement with references, not pixel truth.

Usage

from peft import PeftModel
from transformers import AutoModelForImageTextToText, AutoProcessor

base = AutoModelForImageTextToText.from_pretrained(
    "Qwen/Qwen3-VL-8B-Instruct", dtype="auto", device_map="auto")
model = PeftModel.from_pretrained(base, "ExperimentField/Aspaklarya-122-cap-step1500")
proc = AutoProcessor.from_pretrained("Qwen/Qwen3-VL-8B-Instruct")