Herb-VISOR

Visual Inspector for Specimen Observation & Recognition

A 4B vision-language model that reads herbarium specimen images and emits structured, controlled-vocabulary JSON describing visible attributes (foliage, stem type, reproductive presence, and reference markers such as labels, barcodes, and scale bars). It reports what is visible on the sheet; it does not perform taxonomic identification.

Given a specimen image and its taxon name, the model returns schema-valid JSON with no prompt engineering.

• Base model: Qwen/Qwen3-VL-4B-Instruct (Apache 2.0)
• Method: full-weight fine-tune, teacher-student distillation
• Format: GGUF (llama.cpp-native), runs offline on an 8 GB-class GPU
• Code, validation, and documentation: GitHub repository

Quickstart (recommended)

One command — downloads Q8 to llama.cpp's cache and auto-fetches the projector:

llama-server -hf CapPow/herb-visor:Q8 --temp 0 -c 8192

Serves an OpenAI-compatible endpoint at 127.0.0.1:8080.

Manual download (alternative)

Only needed for offline/air-gapped use or to pin a specific file. The pull above already handles downloads, so don't do both. Download the projector (required) plus one weight file:

File	Purpose
herb-visor-4b-mmproj-f16.gguf	vision projector — required for image input
herb-visor-4b-q8.gguf	model weights, q8 (recommended; ~8 GB VRAM)
herb-visor-4b-f16.gguf	model weights, f16

Pair the mmproj with either weight file, then run against the local files:

llama-server \
  --model herb-visor-4b-q8.gguf \
  --mmproj herb-visor-4b-mmproj-f16.gguf \
  --temp 0 \
  -c 8192 \
  --host 127.0.0.1 --port 8080

The inference contract is deliberately minimal: no system prompt, no schema instructions. The only text input is the taxon binomial (standard casing, e.g. Acer pseudoplatanus), with the specimen image attached. Use temperature 0 for deterministic output. The model also returns valid JSON without a taxon name; the name is included to aid reproductive-trait alignment.

A minimal client (infer.py, pure Python standard library):

python infer.py path/to/specimen.jpg "Acer pseudoplatanus"

Or via the OpenAI-compatible endpoint. Build the request payload in Python (a base64 image is too large to pass as a shell argument), then send it:

python3 <<'PY'
import json, base64
img = base64.b64encode(open("path/to/specimen.jpg", "rb").read()).decode()
payload = {
    "messages": [{
        "role": "user",
        "content": [
            {"type": "text", "text": "Acer pseudoplatanus"},
            {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{img}"}}
        ]
    }],
    "temperature": 0
}
open("/tmp/req.json", "w").write(json.dumps(payload))
PY

curl -s http://localhost:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  --data-binary @/tmp/req.json | python -m json.tool

Example output

For a pressed Acer pseudoplatanus sheet:

{
  "type": "PH",
  "attached_photo": false,
  "structures": {
    "foliage": "present",
    "foliage_type": "leaf",
    "stem": "woody",
    "phenology": {
      "flower": false, "fruit": false, "pollen_cone": false,
      "seed_cone": false, "sporulating": false, "reproductive_unknown": false
    }
  },
  "refs": {
    "label": true, "barcode": false, "stamp": false,
    "crc": true, "scale_bar": true
  }
}

The full output schema is in the repository.

Training

The model was trained by distilling a larger teacher (Qwen3.6-27B, Qwen3.6-27B-UD-Q5_K_XL), whose structured-JSON captions were the training ground truth. Training used two phases: phase 1 with full schema instructions in the prompt, and phase 2 with only the image and taxon name. Phase 2 bakes the schema into the weights, so end users need no prompt beyond the binomial. On the held-out test set, output was schema-valid, strict-parsed, controlled-vocabulary JSON in all 643 of 643 cases.

Evaluation

Accuracy was measured against human-validated labels on a 100-specimen blind sample (a single non-specialist annotator scored each field cold from the image, with no access to model predictions). Per-field accuracy is strong on reference markers and foliage; the weaker fields are stem type and stamp detection.

Field	Accuracy
structures.foliage	0.97
structures.stem	0.79
attached_photo	0.95
refs.label	0.99
refs.barcode	1.00
refs.stamp	0.70
refs.crc	1.00
refs.scale_bar	1.00
repro_visible (category-level)	0.88

Whole-specimen strict exact match (all 10 fields correct at once) was 0.438, against 0.484 for the 27B teacher. Distillation preserved teacher behavior closely, including its errors; the student did not exceed the teacher.

Speed: roughly 5.0 s/img for this model versus 68.6 s/img for the 27B teacher on the same hardware (single stream).

Full methodology, the label-free taxonomic-consistency check, and reproduction instructions are in the GitHub repository.

Limitations

• repro_visible is validated at the category level only (a reproductive structure is present). Fine-grained phenology (flower vs fruit vs cone type) was not human-validated.
• Ground truth is a single non-specialist annotator (n=100); some apparent errors are annotator-limited. Treat reported accuracies as a conservative floor.
• Output is a curator-assist candidate, not authoritative write-back.
• type is always PH on herbarium input and is not a discriminative result.

License and attribution

This model is a full-weight fine-tune of Qwen/Qwen3-VL-4B-Instruct, which is licensed under Apache License 2.0. Herb-VISOR is released under the same Apache 2.0 license. The weights were modified by fine-tuning on distilled teacher captions over herbarium specimen images.

Repository code is released under the MIT license. Training images are GBIF-derived and follow their source-institution terms; they are not redistributed here.

Citation

@software{powell2026herbvisor,
  author  = {Powell, Caleb and Sterner, Beckett},
  title   = {Herb-VISOR: a compact vision-language model for
             structured captioning of herbarium specimens},
  year    = {2026},
  url     = {https://github.com/CapPow/herb-visor},
  note    = {Software and model weights; manuscript in preparation}
}