GG-QandV/gliner_small-v2.5-onnx

gliner_small-v2.5 — ONNX / INT8

ONNX conversion of gliner-community/gliner_small-v2.5, dynamically quantised to INT8, packaged as a self-contained bundle for offline NER.

This is a re-serialisation, not a fine-tune: the weights are the upstream ones. Only the format (PyTorch → ONNX) and the precision (fp32 → INT8) are ours.

---

Model summary

Source	gliner-community/gliner_small-v2.5 (revision f227d3cd), converted to ONNX/INT8
GLiNER project	https://github.com/urchade/GLiNER · paper: https://arxiv.org/abs/2311.08526
Base model	gliner-community/gliner_small-v2.5
Encoder	microsoft/deberta-v3-small (GLiNER uni-encoder span model)
Span mode	markerV0, has_rnn: true
Hidden size	768
Vocabulary	128 003
max_width	12 (words per span)
max_len	768
Entity types	open vocabulary — supplied in the prompt, not baked into the weights
Language	multilingual
Upstream revision	f227d3cd637bd4e6757ae143935316d062393341
License	Apache-2.0 (see LICENSE)

---

Files

File	Size (bytes)	sha256
model_quantized.onnx	196 786 385	60f2f4da1ccad2230626ecc00cbbb18474b5415d3a9fddfca8078f52c2ab2930
tokenizer.json	8 332 739	08bb5853718f4a829fa9ce773d7984f7f3f6a7073fdc82a07a382675c5061ba6
tokenizer_config.json	531	54121ac6feec6b4d5bf85245a54c3c5b1ff05ef2318cf83e45340849b8981566
gliner_config.json	2 274	b327b6b5fe3cbefc4583e8cc50ecce3442f5c42855d0f5362dd51d8fa620d84f
LICENSE	11 358	Canonical Apache-2.0 text
NOTICE	1 110	Provenance record

The fp32 reference graph (model.onnx, sha256 5245733ccb2b75072cce0b4bbb14424988f92f9daf775d97bdf0de74be28df63) is not shipped — it is only needed to reproduce the INT8 graph. Its hash is recorded in NOTICE.

---

Quick start

from huggingface_hub import hf_hub_download

REPO = "GG-QandV/gliner_small-v2.5-onnx"
for f in ("model_quantized.onnx", "tokenizer.json", "tokenizer_config.json", "gliner_config.json"):
    hf_hub_download(REPO, filename=f, local_dir="models/gliner_small-v2.5-onnx")

The graph has six inputs, fed per span-encoded prompt:

input_ids, attention_mask, words_mask, text_lengths, span_idx, span_mask

span_mask is bool (not int64) — the graph declares tensor(bool). The prompt follows the GLiNER label format, using the model's own special tokens from gliner_config.json: <<ENT>> <type> … <<SEP>> <text words>. Word splitting is whitespace-based (words_splitter_type: whitespace), subtoken_pooling: first. Inference in ONNX Runtime on CPU; result is a logits tensor of shape (batch, words, max_width, num_types); entities are sigmoid(logits) ≥ threshold (the reference runtime uses ner_score_threshold = 0.5).

---

Export & quantisation

# dev-only, requires torch; the ONNX bundle is what ships, torch is never a runtime dependency
python scripts/export_gliner_onnx.py \
    --src      gliner_small_v25_hf \
    --out      gliner_small_v25_bundle \
    --revision f227d3cd637bd4e6757ae143935316d062393341

Exporter	gliner 0.2.28 (export_to_onnx, opset 19)
Quantisation	onnxruntime.quantization.quantize_dynamic, QuantType.QUInt8
Environment	onnxruntime 1.25.0 · torch 2.13.0 · python 3.12.3
Determinism	re-running reproduces both sha256 above

---

Evaluation

Measured on the source bundle during acceptance (ONNX Runtime CPU, 12 vCPU).

Parity with upstream

20 texts (10 ru + 10 uk), 6 entity types, identical session options on both paths, same model_quantized.onnx:

Path	max |Δscore|	Set differences
Upstream GLiNER vs this graph	1.192e-07	0 / 20

The port is numerically exact against the upstream implementation, at both ORT_ENABLE_ALL and ORT_DISABLE_ALL.

INT8 vs fp32 reference

32 texts, organisation/person/location/date/technology/product, threshold 0.5:

Configuration	Found	Matched fp32	Missing vs fp32	Extra vs fp32
INT8 ORT_ENABLE_ALL	74	68	7	6
INT8 ORT_DISABLE_ALL	74	68	7	6

The two optimisation levels produce identical entity sets (0 differences). All 7 vs-fp32 "misses" resolve to type reassignment (ORG → LOC, technology → product), span splits (Наталія Бондаренко → two tokens; 15 марта 2027 → date + year) or dropped low-score noise — no entity is genuinely lost. Large score deltas (up to 0.40) occur only on sub-threshold noise spans.

Latency & memory (INT8, 6 types, p50, 12 vCPU)

ORT level	~30 tok	~128 tok	~512 tok	RSS (footprint)
ORT_DISABLE_ALL	107 ms	281 ms	1169 ms	240.8 MB
ORT_ENABLE_ALL	59 ms	134 ms	845 ms	419.8 MB

Both levels are numerically equivalent; the choice is purely memory vs speed.

Languages

10 PER/ORG/LOC sentences per language, threshold 0.5:

Language	Found	FP		Language	Found	FP
fr	30/30	0		el	27/30	3
de	30/30	0		fi	26/30	0
es	28/30	2		tr	26/30	6
nl	28/30	2		it	25/30	2
pl	28/30	2		lt	25/30	0
ro	27/30	2		lv	25/30	0
sv	27/30	2		et	24/30	0
cs	27/30	2		pt	23/30	3
bg	27/30	2
sr	27/30	2
hu	27/30	1

All 19 languages score ≥ 0.77. Degradation tracks sub-token fragmentation (up to 2.9 sub-tokens per word in Turkish) rather than language distance.

Entity types (measured, not asserted)

Of the eleven candidate types tested, six were kept for production: person, organization, location, date, technology, product.

Type	Behaviour
person, location	reliable on the evaluated languages
technology, product, date	reliable on domain text
organization	low recall; organisation names frequently surface as location
decision, prohibition	never fired on explicit decision/prohibition sentences — not offered as model types
artifact, concept, question	removed — fired only on noise at low scores

Changing the type list changes the input prompt and therefore shifts scores; a type set must be validated as a whole.

---

Intended use & limitations

• Intended: offline, open-vocabulary NER on European languages inside a local runtime.
• Not intended: as a legal, medical or safety-critical extractor; as a source of ground truth.
• Organisation recall is weak and organization counters should not be treated as authoritative.
• This is an INT8 graph: expect small, threshold-adjacent differences from the fp32 reference.
• Dates may be split into components; downstream code should normalise before storing them.

Licensing

Weights: Apache-2.0, inherited from gliner-community/gliner_small-v2.5 (the upstream model card declares Apache-2.0; the GLiNER project is Apache-2.0, verified against its LICENSE file). The full license text is in LICENSE; provenance and all hashes are in NOTICE. No modification of the model itself was made — only ONNX serialisation and INT8 quantisation.

Citation

@inproceedings{zaratiana-etal-2024-gliner,
  title     = {{GLiNER}: Generalist Model for Named Entity Recognition using Bidirectional Transformer},
  author    = {Zaratiana, Urchade and Tomeh, Nadi and Holat, Pierre and Charnois, Thierry},
  booktitle = {Proceedings of NAACL 2024},
  year      = {2024}
}

• Source model: https://huggingface.co/gliner-community/gliner_small-v2.5
• GLiNER repository: https://github.com/urchade/GLiNER
• Original GLiNER weights: https://huggingface.co/urchade