Rainier-VL-2B-MTP

Paper: Rainier-VL: A Compact Vision-Language Model for Industrial Inspection Across Edge GPUs
Myan Sudharsanan, Hemmanth Puppala, Gavin Daher, and Jack Rosenbloom
Denali Advanced Integration Physical AI Team
Accepted at the 2026 IEEE EdgeComm conference.

Abstract

Efficiently running defect detection via visual inspection on a production line or on returns benches is a problem that is the intersection of three concerns, speed, accuracy, and cost. In the domain of defect detection, no other Vision Language Model (VLM) can satisfy the constraints for the areas of concern out of the box. Hence, we present Rainier-VL, a 2.1 B-parameter VLM that meets that operating point across the edge GPU range, from a 50 W, 8 GB workstation card to a Blackwell-class part. There are two novel architectural advancements with Rainier-VL. The OrthoBernstein projector treats the vision→language connector as Householder-orthogonal layers composed with a learnable Bernstein activation, and an SVD warm-started to a dense teacher. This results in giving 24.2× fewer parameters and 12.2× fewer forward FLOPs than the dense equivalent. A hybrid state-space backbone replaces the growing KV cache with a fixed recurrent state, making per-frame cost constant on a continuous camera feed. Rainier-VL reaches 0.949 image-AUROC on VisA defect detection, ahead of every open-source 2–10 B model we evaluate in the same harness, including a 10.3 B model at 0.852, at 4.6× less inference memory than that model and no quantization anywhere.

This repository

This is the Rainier-VL-2B checkpoint with multi-token prediction (MTP), packaged with its own inference engine and an OpenAI-compatible server. One image plus a short instruction returns a defect verdict with a calibrated probability, per-defect segmentation masks, bounding boxes, a dense defect-probability map, or a one-sentence description. It runs on an 8 GB GPU such as the RTX A1000.

The repository also ships two CPU text-reading add-ons that work with these weights unchanged:

• OCR sidecar (ocr/, rainier_engine.ocr_sidecar): a small PP-OCRv6 model reads the image and its text is put in front of the question, with a question router that runs OCR only when a question needs it.
• Label reader (ocr/label/, rainier_engine.label_reader): brand and printed size from a close-up of a garment label.

Model details

Parameters	2.094 B unique (0.878 B vision + 1.215 B language + 1.08 M connector); 1.644 B active at inference
Vision encoder	SigLIP-SO400M, 384 px, 729 visual tokens per frame
Projector	OrthoBernstein (Householder-orthogonal + Bernstein activation), 1.08 M parameters
Language model	Zamba2-1.2B (hybrid Mamba2 / attention)
Extra heads	3 dense mask heads, 1 box head (10.07 M total); MTP, 4 heads (16.78 M, 33.6 MB)
Weights on disk	3.99 GB, bf16, no quantization
Input	one RGB image, longest side 384 px, JPEG q90
Output	JSON (verdicts, RLE masks, boxes, scores) or free text
OCR add-ons	PP-OCRv6 small detector + recognizer (scene text) and a label-tuned pair, 44 MB ONNX in total, CPU

One vision forward per image produces a shared projected grid that the language model, the box head and the mask heads all read, so asking for a verdict, a box and a mask on the same frame pays the vision cost once.

Tasks

The task is selected by a tag at the start of the prompt.

task	prompt	returns
detect	[detect]	answer (yes/no), p_yes (calibrated probability), raw generated JSON
seg	[seg] Segment the defect.	per-defect instances (mask, box, label, severity), union mask_rle, object silhouette
od	[od] Localize the defect.	one box per defect plus a primary bbox
anomaly	[anomaly] Localize the defect.	defect-probability map and anomaly_score
caption	[caption] Describe this <noun> in one short sentence, noting any visible damage or defects.	one sentence of text

For detect the engine fills in the canonical question itself; send only the tag. od and anomaly reuse the seg pass on the same frame, so asking for all three costs about one seg call.

Benchmarks

Defect detection vs open-source VLMs

Image-AUROC across four domains. Every row is scored in the same harness on the same images (per-category cap 40). Parameters are unique parameters.

model	params (B)	VisA	BSData	BTAD	MVTec (leather)
Rainier-VL-2B	2.1	0.949	0.974	0.975	0.995
GLM-4.6V-Flash	10.3	0.852	0.943	0.898	0.987
Qwen3-VL-4B	4.4	0.840	0.915	0.571	0.997
Qwen3-VL-2B	2.1	0.802	0.916	0.531	0.996
Cosmos-Reason2-2B	2.4	0.801	0.923	0.666	0.989
Holo-3.1-4B	4.5	0.728	0.855	0.637	0.999
NuExtract3-4B	4.5	0.709	0.911	0.270	0.991
Qwen2.5-VL-3B	3.8	0.702	0.619	0.785	0.988
Qwen3.5-4B	4.5	0.683	0.794	0.766	0.998
Gemma-4-E2B	5.1	0.625	0.933	0.321	0.996

Logical anomalies (MVTec-LOCO) are excluded: every model evaluated, Rainier-VL included, scores between 0.47 and 0.56 there. The strongest peer on VisA (GLM-4.6V-Flash) needs 20.9 GB of GPU memory; Rainier-VL peaks at 6.4 GB.

Defect detection vs vision specialists

Industrial automation largely runs on specialist vision models. The YOLO family is the most widely deployed detector in industrial inspection, and SegFormer and OneFormer are common choices for pixel-level defect maps. Every model below is scored with the same image-AUROC on the same images, on one RTX PRO 6000 (Blackwell). YOLO11, SegFormer and OneFormer were fine-tuned in-domain for this comparison; YOLO-World and SAM 3.1 are open-vocabulary models prompted with defect words and used zero-shot. No model was trained on MVTec-AD, so the MVTec leather column is zero-shot for every entry.

model	params	in-domain (VisA, 720 images)	zero-shot (MVTec-AD leather, 124 images)	latency (batch 1, p50)	throughput
Rainier-VL-2B	2.1 B	0.970	0.995	61.4 ms	46 img/s
YOLO11n / YOLO11s (fine-tuned)	2.6 / 9.5 M	0.969	0.38 / 0.35	11.3 / 10.8 ms	140 img/s
SegFormer-B2 (fine-tuned)	27 M	0.977	0.956	15.2 ms	69 img/s
OneFormer Swin-T (fine-tuned)	51 M	0.971	0.903	26.7 ms	37 img/s
YOLO-World S / M (zero-shot)	13 / 28 M	0.55 / 0.51	0.58 / 0.68	10.4 / 13.9 ms	140 / 129 img/s
SAM 3.1 (zero-shot)	0.85 B	0.51–0.56	0.81–0.94	56–226 ms	4–18 img/s

The fine-tuned specialists match Rainier-VL in their own domain, but they overfit it: during fine-tuning, OneFormer's validation loss rose 14% after its best epoch and SegFormer's 8% while training loss kept falling, and on a surface they never saw they lose 2 to 63 AUROC points. Rainier-VL holds 0.995 there, and the zero-shot specialists reach neither level. The specialists are faster per image, as expected of single-task models of 3 to 51 M parameters; Rainier-VL's verdict stays well inside an inline decision window while the same model also returns typed records, masks, descriptions and a calibrated confidence. Throughput is at batch 8 where the model accepts a batch, otherwise batch 1; Rainier-VL is timed on its verdict path. Box-level localization is not compared, because Rainier-VL's localization output is tuned per surface and environment through request parameters before deployment.

Full held-out reserves

On the held-out reserves (VisA 60 images per category, n = 720; BSData n = 485; BTAD n = 567) this checkpoint reaches:

benchmark	image-AUROC
VisA	0.970
BSData	0.974
BTAD	0.978

Segmentation heads

Validation IoU of the mask heads shipped in weights/heads/:

head	role	IoU
defect_cisba	defect masks and anomaly map	0.858
typed_text	per-instance defect typing	0.735
generic_coco	object silhouette	0.626

Text reading with the OCR add-ons

The model is the same; only the prompt changes.

task	without add-on	with add-on
questions about text in photos (2,000, public scene-text QA validation set), OCR sidecar	0.078	0.243
garment label brand (345 held-out label photos), label reader	0.009	0.594
garment label printed size (317 held-out label photos), label reader	n/a	0.394

With the general sidecar instead of the label reader, label brand accuracy is 0.159. When the label reader finds the brand in its vocabulary (61% of labels) the brand is correct 91.4% of the time, so brand_source == "ocr_vocab" is the field to auto-fill from. OCR adds about 120 to 140 ms per image on 4 CPU threads, and only when the router or the label task asks for it.

Decode cost vs open-source VLMs

Per-token decode cost on an RTX PRO 6000, best of 3 after three warm-ups. Rainier-VL is measured with the bundled rainier_engine (static hybrid cache, fused multi-token verify, MTP speculative decoding); the other models are served through vLLM with CUDA graphs.

model	params (B)	ms / token	relative
Rainier-VL-2B	2.1	3.40	1.0×
Qwen3-VL-2B	2.1	8.99	2.6×
Qwen2.5-VL-3B	3.8	15.74	4.6×
Qwen3-VL-4B	4.4	20.57	6.0×

Speed and latency

RTX A1000 8 GB (50 W, deployment target)

Measured by the bundled engine per request, single stream, one 384 px frame, fresh compute (caches excluded), card under sustained load.

task	p50	p95	notes
detect	671 ms	800 ms	time to first token 558 ms; speculative decode, 82 tok/s
seg	746 ms	1,486 ms	mask head 323 ms of the total; includes silhouette and defect typing
od	+2 ms	+3 ms	reuses the seg pass on the same frame
anomaly	+2 ms	+3 ms	reuses the seg pass on the same frame
caption	882 ms	2,334 ms	greedy decode, 30 tok/s, 64-token cap

Prefill of a 739-token sequence takes about 390 ms at the default chunk_size=256. Peak GPU memory is about 6.4 GB with MTP and all heads loaded.

In a deployed inspection station (capture, inference, overlay and write, 2,747 frames over 65 minutes, two camera streams) the full application runs at 908 ms per frame end to end, 1.101 frames/s sustained, with per-camera revisit at 1,591 ms (p10 1,504 ms, p90 1,669 ms). Adding mask and box output to a verdict-only frame costs 39 ms.

RTX PRO 6000 (Blackwell)

The engine's cost per request is a fixed part (preprocess, vision encode, prefill) plus a per-token part, so answer length is the biggest lever you control.

request	generated tokens	latency	frames / s
verdict only (P(yes), no generation)	0	~59 ms	~17
{"defect": "no"}	7	~79 ms	~12.6
one sentence	48	~219 ms	~4.6

Fixed cost ≈ 55.6 ms plus ≈ 3.40 ms per generated token. Batching two frames in one call amortises about 57% of the per-request overhead.

Vision encoder alone, batch 1 / 4 / 8: 19.9 / 36.5 / 70.3 ms.

Throughput

setting	figure
detect, A1000, fresh frames	~89 requests / min
seg, A1000, fresh frames	~80 requests / min
deployed station, A1000, end to end	1.101 frames / s over 2 streams
full five-task sweep on a static scene, A1000, warm cache	p50 27 ms, ~67 frames / min
verdict-only stream, RTX PRO 6000	~17 frames / s
vision encode, RTX PRO 6000, batch 1 / 4 / 8	50.2 / 109.6 / 113.8 img / s
sustained decode, RTX PRO 6000	198 tok / s

On a static scene the server answers repeats from its scene cache, so a monitoring loop runs far above the fresh-compute rate and pays the fresh cost once when the scene changes.

Multi-token prediction (MTP)

MTP adds four small heads to the language model, each predicting one of the next four tokens from the model's last hidden state. The base model is trained with the MTP objective, and the shipped heads are then fitted on the frozen base by self-distillation; they share its embedding table, so they add only 16.8 M parameters. At inference the heads propose a block of up to k tokens and the base model checks the whole block in one forward pass, keeping only the tokens that match its own greedy choice. The recurrent state is rewound to the last accepted token from per-position checkpoints, so no extra forward is needed.

Measured with the bundled weights and drafter on an RTX PRO 6000 (Blackwell), 40 held-out prompts, bf16, engine default k = 8:

Decode cost, greedy → MTP	12.52 → 4.88 ms per token
Decode speedup	2.56× (2.91× on answers of 8 tokens or more)
End-to-end speedup, including prefill	1.93×
Accepted tokens per verify round	3.72
Decode speedup with fp32 verify	2.68×, output identical to greedy on 40 / 40 prompts
Decode speedup at k = 4	2.02×

In bf16, batched verification rounds slightly differently from single-token decoding, and 2 of 40 prompts produced a different token sequence from greedy decoding. Run the verify in fp32 when records must match greedy decoding exactly. On lower-bandwidth GPUs such as the RTX A1000 the verify step costs relatively more; the server's adaptive policy (RAINIER_SPEC=adaptive) turns speculation off whenever acceptance drops below break-even, so decoding is never slower than greedy.

Determinism

Outputs are bit-identical within a process and across server restarts when CUBLAS_WORKSPACE_CONFIG=:4096:8 and a persistent TRITON_CACHE_DIR are set before launch.

Requirements

• NVIDIA GPU with 8 GB VRAM or more, Ampere or newer. Peak use is about 6.4 GB. The 4 GB RTX A1000 variant cannot run this model, and there is no quantized version.
• CUDA 12.8-capable driver, Python 3.12
• torch==2.10.0, transformers==5.5.4 (pinned; other versions change the SigLIP module layout), mamba-ssm and causal-conv1d CUDA wheels. See requirements.txt.
• For the HTTP server additionally: fastapi, uvicorn, scipy.
• Access to google/siglip-so400m-patch14-384 and Zyphra/Zamba2-1.2B on the Hugging Face Hub. The loader builds both towers from these configs and then loads Rainier's weights over them. After the first download they are read from the local HF cache.

Quick start

1. Download and install

hf download Denali-AI/Rainier-VL-2B-MTP --local-dir rainier-vl-2b-mtp
cd rainier-vl-2b-mtp

python3.12 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
pip install fastapi uvicorn scipy

2. Check the GPU and run the smoke test

PYTHONPATH=code python -m ortho_vlm.kernel_probe   # reports which decode kernel tier this GPU gets
python smoke_test.py                               # loads the weights and checks for a valid JSON answer

smoke_test.py must end with PASS. It verifies that the weights and MTP heads load and that the model returns a parseable {"defect": ...} answer.

3. Python API

import sys; sys.path.insert(0, "code")
from PIL import Image
from rainier_engine import Rainier

m = Rainier.from_pretrained("weights/vlm.pt", drafter="weights/drafter/head.pt")
img = Image.open("part.jpg").convert("RGB")
out = m.generate(img, "Describe any defect in this image.", max_new_tokens=64)
print(out["texts"][0])

The Python API covers generation (detect and caption). The mask heads, object profiles, confidence gate and scene cache live in the server.

Reading text and labels (CPU add-ons)

pip install rapidocr onnxruntime opencv-python-headless rapidfuzz

from rainier_engine.ocr_sidecar import OCRSidecar
from rainier_engine.label_reader import LabelReader

ocr = OCRSidecar("ocr")
q = "What time does the exhibition start?"
print(m.generate(img, ocr.prompt(img, q), max_new_tokens=16)["texts"][0])

lr = LabelReader("ocr")
label = lr.read_label(Image.open("label.jpg").convert("RGB"), m)
print(label["brand"], label["brand_source"], label["size_final"])

OCRSidecar.prompt adds a Text in image: ... line in front of the question only when the question router decides the question needs reading and the image has text; otherwise the question is passed through unchanged.

4. Start the server

export CUBLAS_WORKSPACE_CONFIG=:4096:8          # deterministic outputs across restarts
export TRITON_CACHE_DIR=$PWD/.triton-cache      # persistent compiled kernels
python serve/engine_shim.py --model-dir . --port 8150

The server warms up before it starts listening; wait for ready on :8150. Then:

curl localhost:8150/health       # lists loaded modes and spec-decode status

For an offline host, copy a populated HF cache alongside the model and use serve/serve_standalone.sh (it sets HF_HUB_OFFLINE=1); set MODEL_PATH to this directory and PYTHON to your venv's interpreter.

5. Send a request

import base64, io, json, requests
from PIL import Image

img = Image.open("part.jpg").convert("RGB")
img.thumbnail((384, 384), Image.Resampling.BOX)       # longest side 384, never upscale
buf = io.BytesIO(); img.save(buf, "JPEG", quality=90)
uri = "data:image/jpeg;base64," + base64.b64encode(buf.getvalue()).decode()

def ask(text, max_tokens=24):
    r = requests.post("http://localhost:8150/v1/chat/completions", json={
        "model": "rainier-vl-2b-jointmtp-v2-a1000",
        "temperature": 0, "max_tokens": max_tokens,
        "messages": [{"role": "user", "content": [
            {"type": "text", "text": text},
            {"type": "image_url", "image_url": {"url": uri}}]}]})
    return r.json()["choices"][0]["message"]["content"]

verdict = json.loads(ask("[detect] [pf:metal]"))
print(verdict["answer"], verdict["p_yes"])

seg = json.loads(ask("[seg] [pf:metal] Segment the defect."))
for inst in seg.get("instances", []):
    print(inst.get("label"), inst.get("severity"), inst["bbox"])

print(ask("[caption] Describe this metal part in one short sentence, "
          "noting any visible damage or defects.", max_tokens=64))

detect, seg, od and anomaly return a JSON string in message.content; caption returns plain text. Boxes are normalized [x0, y0, x1, y1]. Masks are run-length encoded as {"size": [h, w], "counts": [...]}, row-major, alternating 0-runs and 1-runs starting with a 0-run. "stream": true is supported and returns the whole answer as one SSE chunk.

Prompting rules

• Send the bare prompt. Do not wrap it in USER: / ASSISTANT: or any chat-role template. The model was trained on bare prompts and a role wrapper breaks the answer format.
• Put tags first, then the task text. Tags are stripped before the text reaches the model.
• Use temperature: 0.
• Questions about text in the image: build the prompt with OCRSidecar.prompt rather than asking unaided.

Control tags

tag	effect
[pf:<profile>]	object profile: sets defect vocabulary and object noun (see below). Default generic
[gate:on|off|<p>]	confidence gate: per-camera calibration, off, or a fixed P(yes) threshold such as 0.90
[cam:<id>]	camera id, used to look up per-camera gate thresholds
[tg:<type>]	limit seg/od/anomaly output to one defect type (defect = all)
[dw:x]	mask tightness, 0 to 1
[ds:x]	defect sensitivity, 0 to 1
[es:x]	edge sensitivity, 0 to 1
[as:x]	anomaly sensitivity, 0 to 1
[bs:x]	box size, 0 to 1
[tp:type=dw,ds,es,as,bs;...]	per-defect-type values for the five sliders above
[fresh]	bypass all caches for this request

Slider values of 0.5 are neutral.

Object profiles

A profile tells the model what kind of object is in frame and which defect types to look for.

profile	object	defect types
generic	product	scratch, crack, dent, stain, dirt, tear, contamination, missing_part
shoes	shoe	stain, dirt, scratch, tear, glue_residue, discoloration, crease
packaging	box	tear, dent, open_seam, stain, hole, contamination
pcb	circuit board	missing_part, contamination, scratch, stain
electronics	electronic device	scratch, crack, dent, missing_part, burn_mark, contamination
clothing	garment	stain, tear, missing_part, contamination
metal	metal part	scratch, dent, corrosion, crack, contamination
weld	weld joint	crack, porosity, undercut, spatter, incomplete_fusion
painted_frame	painted metal tube	paint_run, orange_peel, blister, chip, contamination

The full registry, including the probe questions used per defect type, is in hyperparameters/object_profiles.engine.json. New profiles are added to PROFILES in serve/engine_shim.py.

Calibrating for a new line

The generated yes/no verdict should be calibrated per camera before it is trusted on a new production line. Collect about 20 clean and 20 defective frames per camera, measure how well p_yes separates them, and write the per-camera thresholds to serve/detect_calib.json (Denali's calibrate_detect.py tool produces this file). With [gate:on] the server then applies the threshold with a 3-frame rolling median per camera, and masks follow the gated decision. Enable the gate only on cameras where calibration AUROC is 0.9 or higher.

No calibration file ships with this repository. Without one, [gate:on] has no effect and the server returns the model's generated verdict.

Configuration

Server behaviour can be tuned through environment variables on the serve process. Defaults and descriptions are in hyperparameters/engine_env_defaults.json. The most commonly changed:

variable	default	effect
RAINIER_MAX_NEW_CAP	64	maximum tokens for captions and free-text answers
RAINIER_SPEC	adaptive	speculative decoding policy
RAINIER_DETECT_CALIB	serve/detect_calib.json	path to per-camera gate thresholds (reloaded on change)
RAINIER_GATE_MEDIAN_N	3	rolling-median window for the gate

Limitations

• Defects smaller than about 10 px at 384 px are usually detected but not localized. Use a closer crop or a zoomed camera for them.
• The scene cache matches frames with a perceptual hash. On a static view, a small new defect can be served the previous clean result for up to five requests; send [fresh] when every frame must be recomputed.
• The yes/no verdict is not calibrated for new lines out of the box (see above).
• Speculative decoding speeds up the short JSON answers of detect; free-form captions fall back to greedy decoding.
• The server processes one request at a time on one GPU.

Files

path	contents
weights/vlm.pt	base model weights
weights/drafter/	MTP heads
weights/heads/	mask heads (defect_cisba, generic_coco, typed_text), each with its paired projector
weights/box_head.pt	box head, used only when no mask head is loaded
weights/tokenizer/	tokenizer
ocr/	OCR sidecar models, question router and configuration; ocr/label/ label reader models and brand vocabulary; licence notices in ocr/NOTICE.md
code/	rainier_engine (inference API, ocr_sidecar, label_reader) and ortho_vlm (model code and kernels)
serve/	engine_shim.py server, serve_standalone.sh offline launcher, helper modules
hyperparameters/	object-profile registry and server environment defaults
smoke_test.py	install check
config.json	model configuration summary
MANIFEST.json	sha256 of every weight file
paper/	the Rainier-VL paper (PDF)

Citation

@inproceedings{sudharsanan2026rainiervl,
  title     = {Rainier-VL: A Compact Vision-Language Model for Industrial Inspection Across Edge GPUs},
  author    = {Sudharsanan, Myan and Puppala, Hemmanth and Daher, Gavin and Rosenbloom, Jack},
  booktitle = {IEEE EdgeComm},
  year      = {2026},
  note      = {To appear. Model: https://huggingface.co/Denali-AI/Rainier-VL-2B-MTP}
}

License

Released under the Apache License 2.0. Copyright 2026 Denali AI. The OCR add-on models are Apache-2.0; see ocr/NOTICE.md.