Potestates/qwen3-vl-8b-frame-lora

Qwen3-VL-8B — FRAME track LoRA adapter (checkpoint 1600)

LoRA adapter fine-tuned on top of Qwen3-VL-8B-Instruct for the ORena SAVE FOCUS Challenge, FRAME track (single-frame surgical foreign-object VQA).

This is our current best checkpoint (step 1600 of a 2000-step run). Local official-evaluator score (our own test split, NOT the official leaderboard split — treat as optimistic): pre_evaluation_score = 65.98%.

Files

• adapter_config.json / adapter_model.safetensors — PEFT LoRA weights (r=32, alpha=64, dropout=0.05), targeting the language-tower attention/MLP projections and the vision-tower attention/MLP/patch-embed layers (target_modules: q_proj k_proj v_proj o_proj gate_proj up_proj down_proj qkv proj linear_fc1 linear_fc2).
• chat_template.jinja, processor_config.json, tokenizer_config.json, tokenizer.json — copied from the base model, bundled here only so the whole thing is self-contained. They are not modified by training.

How to use

You do not need to separately clone/download the base model by hand — from_pretrained("Qwen/Qwen3-VL-8B-Instruct") will pull it from the Hub automatically. Just load the base model, then attach this adapter with PEFT.

Requires transformers new enough to support Qwen3-VL (AutoModelForImageTextToText, roughly transformers>=4.57 / the "v5" line) and peft.

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
from PIL import Image

BASE = "Qwen/Qwen3-VL-8B-Instruct"
ADAPTER = "<this-repo-id>"          # e.g. "Potestates/qwen3-vl-8b-frame-lora"

model = AutoModelForImageTextToText.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="cuda:0"
).eval()
model = PeftModel.from_pretrained(model, ADAPTER).eval()
processor = AutoProcessor.from_pretrained(ADAPTER)  # bundled tokenizer/chat template

image = Image.open("frame.png").convert("RGB")      # NATIVE resolution — do not resize
question = "How many foreign objects are visible?"   # raw question text, nothing prepended

messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text", "text": question},
]}]
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False
)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
answer = processor.batch_decode(
    out[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True
)[0].strip()
print(answer)

Things that matter for reproducing the score (learned the hard way)

• No system prompt, no FO-class definitions text — the model was fine-tuned on the raw question only ("question_only" style). Prepending extra instructions or the FO_definitions.json content hurts accuracy.
• Do not resize the image. Qwen3-VL has no fixed input resolution; resizing changes the number of vision tokens and moves you off the training distribution.
• enable_thinking=False — Qwen3-VL/3.6 are reasoning-capable; leaving thinking on burns the token budget on a reasoning trace instead of an answer.
• Greedy decoding, max_new_tokens=32 is enough (answers are short: numbers, class names, yes/no, short phrases).
• Strip trailing punctuation from the generated answer before scoring — the model sometimes ends with a period and the official parser is strict about exact-match formats (number, binary).

Known weak point

Aggregation/counting questions (answer_format=number, primary_capability=3a) are the weakest category, especially the out-of-distribution split. Don't be surprised if a from-scratch eval shows a big accuracy gap between counting/aggregation and object-recognition questions — that gap is real and consistent across every checkpoint and base model we tried.