Potestates/qwen3.6-27b-frame-lora

Qwen3.6-27B — FRAME track LoRA adapter (clip-corrected)

LoRA adapter for Qwen3.6-27B, fine-tuned for the ORena SAVE FOCUS Challenge, FRAME track: single-frame visual question answering about surgical foreign objects (clips, sponges, needles, specimen bags, drains) left in the operative field.

Official leaderboard result: pre_evaluation_score = 0.5911 (rank 24). That is the challenge's own held-out evaluation, not a local split.

bucket	accuracy
aggregation — in-distribution	55.52%
aggregation — out-of-distribution	60.64%
object_recognition — in-distribution	68.54%
object_recognition — out-of-distribution	51.76%
pre_evaluation_score (unweighted mean)	59.11%

Latency on the challenge hardware (single L40S, FP8 base): 1.38 s/question, 0 forfeited, 0 unanswered.

What is different about this adapter

It is a two-stage adapter. The first stage is ordinary supervised fine-tuning on the challenge data. The second stage corrects a specific, measurable failure.

The base fine-tune had learned an unconditional habit of predicting "clip" — about 40% of the class-list answers in the training data contain one, so the model said clip almost regardless of the image. On the in-distribution procedure this is a good bet (answers containing clip were 78.5% correct). On the out-of-distribution procedure it was fatal: 99 out of 99 answers containing clip were wrong.

A dedicated probe showed the habit was not a perception error. The model said clip on 92.2% of frames with no clamps present and 94.2% of frames with clamps present — essentially independent of what was in the picture. It was a language prior, not a misidentification, and it could not be fixed by telling the model the procedure type (every procedure in the training data is clip-heavy, so that field carries no signal).

The correction stage therefore adds out-of-distribution frames where clips are genuinely absent, forcing the prediction to be grounded in pixels. Frames come from SAR-RARP50 (EndoVis 2022, CC BY-NC-SA 4.0), with foreign-object labels derived from its hand-drawn segmentation masks — suturing needle → Needle; instruments ignored; ambiguous frames dropped.

Measured on 180 held-out prostatectomy frames, the rate of answers wrongly containing a clip fell from 89.4% to 0.6%, and the official out-of-distribution object-recognition bucket rose +6.58 points.

Known trade-off. This adapter buys that gain partly by collapsing its answer diversity on unfamiliar footage (it answers "needle" on 172 of those 180 frames), which cost −2.62 points on the in-distribution object-recognition bucket. Net gain over the uncorrected adapter is +1.14 points. A revision that keeps the correction without the collapse is in progress.

Configuration

LoRA r=32, alpha=64, dropout=0.05, 16 target module names resolving to 614 modules: the full vision tower (attention, MLP, patch_embed, and the vision→language merger), and the full language tower — including the 48 Gated DeltaNet layers of this hybrid-attention model (in_proj_qkv, in_proj_z, in_proj_a, in_proj_b, out_proj), which a naive target list silently misses.

Usage

import torch
from transformers import AutoModelForImageTextToText, AutoProcessor
from peft import PeftModel
from PIL import Image

BASE = "Qwen/Qwen3.6-27B"
model = AutoModelForImageTextToText.from_pretrained(
    BASE, dtype=torch.bfloat16, device_map="cuda:0").eval()
model = PeftModel.from_pretrained(model, "Potestates/qwen3.6-27b-frame-lora").eval()
processor = AutoProcessor.from_pretrained(BASE)

image = Image.open("frame.png").convert("RGB")   # native resolution — do not resize
question = "List all foreign objects that are visible in this video frame. " \
           "Please provide the class names or answer with none."

messages = [{"role": "user", "content": [
    {"type": "image", "image": image},
    {"type": "text",  "text": question},
]}]
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True, enable_thinking=False)
inputs = processor(text=[text], images=[image], return_tensors="pt").to(model.device)

with torch.inference_mode():
    out = model.generate(**inputs, max_new_tokens=32, do_sample=False)
print(processor.batch_decode(out[:, inputs["input_ids"].shape[1]:],
                             skip_special_tokens=True)[0].strip().rstrip(" ."))

Things that matter for reproducing the score

• Raw question only. No system prompt, no foreign-object definitions, no procedure type. The model was fine-tuned this way; prepending anything moves you off distribution.
• Do not resize the image. Qwen3.6 has no fixed input resolution, so resizing changes the number of vision tokens.
• enable_thinking=False. Leaving the reasoning trace on burns the token budget.
• Greedy decoding. Self-consistency sampling was measured and it loses 1.7 points on counting questions.
• Strip trailing punctuation before scoring; the official parser is strict about exact-match formats.

Known weak point

Counting (answer_format=number) is the weakest category and has resisted every intervention we tried: self-consistency (−1.7), high-resolution upsampling (−0.4), count-weighted loss (−2.1), a weakly-supervised density head (0.000 exact match), enumeration prompting (flat), and an external detector (−13). Linear probes across layers read counts worse than the model's own generation, which suggests the count is not linearly present in the hidden states at all.

Citation

Base model: Qwen3.6-27B (Apache-2.0). Correction data: SAR-RARP50, EndoVis 2022 (CC BY-NC-SA 4.0) — non-commercial.