vision-gpt-oss-120B (autotrust/vision-gpt-oss-120b)

Vision-language model that grafts a vision stack onto gpt-oss-120B so the text-only gpt-oss can see.

• LLM: gpt-oss-120B
• Vision: visiont tower (1152→5376, frozen, transplanted) → trained connector (5376→8192→2880) → injected at <|vis_pad|> (LLaVA-style).
• Vision token budget: 1120 (dense — good for charts/documents/figures).

Results (zero-shot, no MMMU in training)

benchmark	score	notes
MMMU (val, 900, zero-shot)	43.7%	full validation set, no few-shot
HumanEval pass@1 (subset)	97.5%	text ability preserved (LLM frozen)

Training data contains no MMMU (DocVQA/AI2D/ChartQA/OCR/VQAv2, teacher-distilled) — the MMMU number is uncontaminated. Companion dataset: autotrust/vision-gpt-oss-data.

Quick start

Runnable script in the repo — example_inference.py:

python example_inference.py --image test_images/photo.jpg

Minimal code:

import torch, re
from transformers import AutoProcessor, AutoModelForImageTextToText
from PIL import Image

repo = "autotrust/vision-gpt-oss-120b"
proc = AutoProcessor.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForImageTextToText.from_pretrained(
    repo, trust_remote_code=True, dtype=torch.bfloat16, device_map="cuda").eval()

img = Image.open("photo.jpg").convert("RGB")
if max(img.size) > 1536:                 # gotcha #3
    img.thumbnail((1536, 1536))

batch = proc(images=img, text="Describe this image in detail.", reasoning_effort="low")
batch = {k: (v.cuda() if torch.is_tensor(v) else v) for k, v in batch.items()}

out = model.generate(**batch, max_new_tokens=512, do_sample=False)   # gotcha #2
text = proc.decode(out[0], skip_special_tokens=False)                # gotcha #1
ans = re.search(r"final<\|message\|>(.*?)(?=<\|return\||<\|end\||$)", text, re.DOTALL)
print(ans.group(1).strip() if ans else text)

proc(...) also accepts messages=[{"role":"user","content":"..."}] and reasoning_effort="low"|"medium"|"high" (gpt-oss harmony format).

Bundled test images (test_images/)

python example_inference.py --image test_images/chart.png --prompt "What % is the top bar?"