Instructions to use aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoProcessor, AutoModelForMultimodalLM

processor = AutoProcessor.from_pretrained("aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO")
model = AutoModelForMultimodalLM.from_pretrained("aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO", device_map="auto")
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
inputs = processor.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(processor.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO

SGLang

How to use aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO with Docker Model Runner:
```
docker model run hf.co/aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO
```

AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO

Supervised fine-tuning followed by GRPO. On this backbone the reinforcement stage really does help, which is the interesting part.

A LLaVA-OneVision-7B-SI model fine-tuned for explainable industrial anomaly detection (IAD). Given one product image it writes a reasoning trace in <think>, then a defect <location> and <type> when it sees a defect, and finally a binary <answer>. Research artefact from the MSc thesis Reasoning-Enhanced Vision-Language Models for Explainable Industrial Anomaly Detection (TU Delft, 2026).

The backbone is the same one IAD-R1 uses. Under one shared harness this checkpoint beats the released IAD-R1 model on both benchmarks, on IAD-R1's own backbone.

Results

Metric is balanced accuracy, (TPR + TNR) / 2, in percent.

Benchmark	Balanced accuracy
MMAD DS-MVTec (1,670 images)	87.66
MMAD VisA (2,141 images)	72.58

Reference rows measured on the exact same harness:

Model	DS-MVTec	VisA
LLaVA-OneVision-7B-SI base	75.66	53.80
IAD-R1 released checkpoint	81.92	71.34
AnomalyThink LLaVA SFT (the init for this run)	85.91	68.26
This model (SFT then GRPO)	87.66	72.58
AnomalyThink LLaVA KCR (SFT only, best of family)	88.45	74.25

GRPO adds +1.75 on DS-MVTec and +4.32 on VisA over its own SFT initialisation. Most of that comes from recall. The SFT init misses a lot of defects and GRPO trades some precision for those misses.

Evaluation protocol. One shared harness for every row above. The DS-MVTec and VisA subsets of MMAD, single image per prompt, the same instruction the model was trained on, greedy decoding at temperature 0, at most 1024 new tokens, images capped at 262,144 pixels. Answers are parsed from the <answer> tag. Generation ran through vLLM 0.10.2, which agreed with the plain transformers generate path on 99 percent of a probe set. Nothing here is a re-scored or best-of-N number.

Contamination note, please read this before you compare DS-MVTec numbers

The public LLaVA-OneVision training mixture (lmms-lab/LLaVA-OneVision-Data, config vision_flan(filtered)) contains 426 rows whose id matches %MVTecAD%. The base model has therefore seen MVTec-AD material during its own instruction tuning. Every DS-MVTec number for any LLaVA-OneVision derived model carries that caveat, including the 87.66 above, and including the IAD-R1 row. We do not know how much of the gap is real capability and how much is recall.

VisA is not affected. The same query over the mixture returns 0 rows for VisA. So the 72.58 on VisA is the clean number and it is the one to trust for a cross-model comparison. It is also where the win over IAD-R1 is smallest, at +1.24.

Training

Two stages.

Stage 1, SFT. Base llava-hf/llava-onevision-qwen2-7b-si-hf fine-tuned on anomalythink_6k/combined_6k_train.json, 6,000 AnomalyThink traces distilled from Gemini 2.5-Flash on Real-IAD images. Frozen SigLIP vision tower, trainable projector and language model, learning rate 1e-5, cosine, effective batch 32, bf16. The epoch-1 checkpoint of that run is published separately as AnomalyThink-LLaVA-OneVision-7B-SFT and is the exact initialisation for stage 2.

Stage 2, GRPO. Group Relative Policy Optimisation on anomalythink_15k/c1_only_fixed/grpo_train.json, 4,236 Real-IAD prompts.

Reward is accuracy plus format, no separate type or location terms.
Group size G = 4 rollouts per prompt, 32 completions per optimiser step.
Learning rate 1e-6, sampling temperature 1.0, KL coefficient beta = 0.
DeepSpeed ZeRO-3 with CPU offload, gradient checkpointing, bf16.
Max prompt 8,192 tokens, max completion 512 tokens, one image per prompt.
The vision tower is trained during GRPO, unlike stage 1 where it is frozen. This follows IAD-R1.
No system prompt. The instruction sits in the user turn.

Epoch: this is epoch 1 of 2 (step 530). Epoch 2 scored 87.86 / 72.10, which is a wash, so epoch 1 is the released checkpoint. Note also that epoch 2 was restarted from this checkpoint after a server maintenance kill, so it has fresh optimiser state and a re-seeded dataloader. It is not a clean continuation.

One honest caveat on the objective. Our GRPO runs use beta = 0, so there is no KL term against the reference policy. IAD-R1's own code used beta = 0.04. Their paper shows a KL term in the objective but never states the value. This model therefore reproduces IAD-R1's recipe in structure, not in the exact objective, and the comparison should be read that way.

The KCR corpus in this family is LLaVA native, not borrowed from Qwen

This model is not trained on a KCR corpus. The note still belongs here, because this checkpoint is the rollout source for that corpus.

KCR stands for keep, correct, rewrite. The corpus used by the sibling KCR model was built from LLaVA-OneVision's own rollouts, not from the Qwen rollouts used elsewhere in the thesis. This exact checkpoint generated 10,236 rollouts on Real-IAD training images at k = 8 and temperature 0.7. Each trace was bucketed into keep, needs correction, or needs rewrite, then scored by a Gemini 2.5-Flash faithfulness judge, then repaired where it failed, and finally sampled to a balanced 6,000.

All three arms of that corpus are published at llava_kcr/: sft_llava_A_kept.json, sft_llava_B_kept_corrected.json, and sft_llava_C_train.json.

Because the rollouts come from this backbone, the corpus is on-policy for LLaVA-OneVision, and that matters. The control, which is the Qwen derived KCR corpus trained on this same backbone, peaked at 87.70 on DS-MVTec and 73.52 on VisA in different epochs. The native corpus reaches 88.45 and 74.25 in one checkpoint.

Tested transformers versions

The weights and configs in this repo were written by transformers 5.0.0. Loading, the processor, and the full evaluation were verified under transformers 4.57.1, both through the plain HF generate path and through vLLM 0.10.2.

This checkpoint is the one the rope trap actually bit, so read this if you fork it. Transformers 5.0 writes the rope settings under text_config.rope_parameters. Transformers 4.x does not read that key and silently falls back to rope_theta = 10000, which is 100 times too small. The model stays perfectly fluent but goes blind. It answered "no" to almost everything and scored 27.5 on DS-MVTec, which looks like a collapsed training run rather than a loading bug. It cost us a day.

The config in this repo carries both forms, so it loads correctly on 4.x and on 5.x:

"text_config": {
  "rope_parameters": { "rope_theta": 1000000.0, "rope_type": "default" },
  "rope_theta": 1000000.0
}

If you re-save this model from transformers 5.0, check that the plain rope_theta key survives before you evaluate on 4.x.

Usage

import torch
from PIL import Image
from transformers import AutoProcessor, LlavaOnevisionForConditionalGeneration

repo = "aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO"
model = LlavaOnevisionForConditionalGeneration.from_pretrained(
    repo, torch_dtype=torch.bfloat16, device_map="auto")
processor = AutoProcessor.from_pretrained(repo)

image = Image.open("part.png").convert("RGB")
product = "tile"          # the product category, it goes into the prompt

question = (
    f"Analyze the provided image of the {product}. "
    "Determine if there are any anomalies present. "
    "If an anomaly is detected, specify its type and location, "
    "and provide a detailed reasoning for your conclusion."
)
messages = [{"role": "user", "content": [
    {"type": "image"},
    {"type": "text", "text": question},
]}]
prompt = processor.apply_chat_template(messages, add_generation_prompt=True)
inputs = processor(images=image, text=prompt, return_tensors="pt").to(
    model.device, torch.bfloat16)

out = model.generate(**inputs, max_new_tokens=1024, do_sample=False)
print(processor.decode(out[0][inputs["input_ids"].shape[1]:],
                       skip_special_tokens=True))

Use this exact instruction. The model was trained on it and it degrades on a different phrasing.

Expected output on a defective part:

<think>
I am inspecting a tile with a speckled, grayish-white surface. ... In the center of the
tile, I detect a triangular, translucent plastic fragment. ...
</think>
<location>center</location>
<type>Contamination</type>
<answer>Yes</answer>

On a normal part the model emits <think> and then <answer>No</answer>, with no <location> or <type> tag.

Intended use and limitations

Research on explainable industrial anomaly detection. This is a thesis artefact, not a production inspection system.

Known limitations:

The DS-MVTec contamination caveat above.
GRPO buys recall with precision. On VisA this checkpoint raises false alarms from 246 to 330, out of roughly 940 normal parts, compared to its SFT init. If false alarms are expensive in your setting, the SFT init or the KCR model may suit you better.
GRPO lineage models over-predict the "missing parts" defect type.
The reward pays type and location credit on defective items regardless of the verdict, and the format check is conditioned on the gold label. No exploitation of this was observed on-policy, but it is a design flaw and it is disclosed rather than hidden.
The model can write a confident and well argued trace for a defect that is not there.
It was trained on Real-IAD style single-object images on plain backgrounds. Cluttered scenes, multiple parts per image, and very different lighting are out of distribution.
Reasoning traces were distilled from a teacher model. A fluent trace is not proof that the model looked at the right pixels.

Citation

@mastersthesis{acudad2026anomalythink,
  title  = {Reasoning-Enhanced Vision-Language Models for Explainable Industrial Anomaly Detection},
  author = {Acudad, Adnane},
  school = {Delft University of Technology},
  year   = {2026}
}

The thesis is deposited in the TU Delft education repository. The training data is at aacudad/AnomalyThink.

Related models

AnomalyThink-LLaVA-OneVision-7B-SFT, the initialisation for this run
AnomalyThink-LLaVA-OneVision-7B-KCR, the best model of the family
AnomalyThink-Qwen2.5-VL-7B-SFT-GRPO, the same two-stage recipe on the Qwen2.5-VL backbone

License

Apache-2.0, inherited from the LLaVA-OneVision-7B-SI base. Trained on Real-IAD images, which are not redistributed here, so cite Real-IAD separately. Reasoning traces were distilled from Gemini 2.5-Flash.

Downloads last month: -

Safetensors

Model size

8B params

Tensor type

BF16

Model tree for aacudad/AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO

Base model

llava-hf/llava-onevision-qwen2-7b-si-hf

Finetuned

(4)

this model

aacudad
/

AnomalyThink-LLaVA-OneVision-7B-SFT-GRPO