goby-1-ie-vi

Joint entity + relation extraction for Vietnamese, fine-tuned from knowledgator/gliner-relex-large-v1.0.

The goal was to raise Vietnamese performance without degrading the base model. On clean text that held. On uppercase text it did not — see Robustness below before you deploy this. Both halves were measured on a hand-annotated set, not assumed.

Try it: live demo

Headline results

Evaluation set: 267 Vietnamese sentences / 283 triples sampled from vi.wikipedia and annotated by hand, triple by triple. Decoding: threshold=0.3, relation_threshold=0.7, flat_ner=True.

	P	R	F1
Vietnamese, partial match
base model	23.8	52.7	32.7
goby-1-ie-vi	55.5	57.2	56.3
Vietnamese, strict match
base model	9.6	21.2	13.2
goby-1-ie-vi	36.3	37.5	36.9

strict requires an exact (head, relation, tail) match. partial allows substring-level span overlap.

Vietnamese F1 rises from 32.7 to 56.3 (partial) and 13.2 to 36.9 (strict).

Paired bootstrap over sentences, 1000 resamples, this model minus base:

	difference	95% CI	resamples positive
Vietnamese strict	+23.8	[+18.2, +29.1]	100%
Vietnamese partial	+23.7	[+18.3, +29.1]	100%

English was not sacrificed

Measured on a 30-sentence English probe covering all 18 relation types (relation_threshold=0.5). English improved:

	English strict F1	English partial F1
base model	70.6	82.4
goby-1-ie-vi	78.8	90.9

Relative L2 weight drift from the base model is small: 0.24% in the DeBERTa encoder, 1.69% in the task heads — the Vietnamese gain did not come from overwriting the base model.

Per-relation results

Vietnamese, partial match, on the same 267-sentence set. n is the number of gold triples.

relation (label used)	Vietnamese label	n	base F1	goby-1-ie-vi F1
date of birth	sinh ngày	24	41.7	82.4
award received	đoạt giải	21	66.7	74.4
headquartered in	đặt trụ sở tại	12	52.9	72.7
inception date	thành lập ngày	24	37.3	69.6
author	tác giả	14	6.2	66.7
date of death	qua đời ngày	20	34.3	65.1
educated at	học tại	15	33.3	62.9
member of	thuộc tổ chức	5	12.5	62.5
event location	diễn ra tại	14	22.2	55.2
place of birth	quê quán	21	20.7	52.4
developed by	phát triển bởi	16	52.4	50.0
position held	giữ chức	13	36.4	48.3
family relation	quan hệ gia đình	17	29.5	46.7
owned by	thuộc sở hữu	7	16.7	44.4
founder of	sáng lập	19	39.3	35.3
located in	nằm ở	13	12.2	34.8
signed document	ký	18	40.0	34.3
successor of	kế nhiệm	10	16.7	9.1

The model improves on 14 of 18 relation types.

Usage

from gliner import GLiNER

model = GLiNER.from_pretrained("belumind/goby-1-ie-vi")

ents = ["người", "chức vụ", "tổ chức", "địa điểm", "ngày", "tác phẩm"]
rels = ["sinh ngày", "quê quán", "giữ chức", "sáng lập", "đặt trụ sở tại"]

text = "Phạm Văn Đồng sinh ngày 1 tháng 3 năm 1906 tại Quảng Ngãi."

entities, relations = model.inference(
    texts=[text], labels=ents, relations=rels,
    threshold=0.3, relation_threshold=0.7,
    return_relations=True, flat_ner=True)

for r in relations[0]:
    print(r["head"]["text"], "|", r["relation"], "|", r["tail"]["text"], "|", round(r["score"], 3))

Expected output

Phạm Văn Đồng | sinh ngày | ngày 1 tháng 3 năm 1906 | 0.957
Phạm Văn Đồng | quê quán | Quảng Ngãi             | 0.934

entities[0] from the same call:

Phạm Văn Đồng          | người      | 0.916
ngày 1 tháng 3 năm 1906 | ngày       | 0.897
Quảng Ngãi             | địa điểm   | 0.878

Labels are free text — the model is zero-shot over label names and works with English labels too. relation_threshold=0.7 is the recommended operating point; 0.5 raises recall but costs precision sharply.

Training

• 5,824 Vietnamese sentences, 18 relation types, distantly supervised from vi.wikipedia.
• Labels are rotated through synonym sets during training (18 relations presented as 72 surface forms, 10 entity types as 40) so the model does not memorise one phrasing of a label.
• Final stage: 700 steps, lr 1.5e-6 (encoder) / 4e-6 (heads), cosine schedule, warmup 0.1, batch size 8, focal loss (alpha 0.75, gamma 2.0), grad clip 1.0.

Deployment

All numbers below were measured on this model, not estimated. Hardware: NVIDIA A100-SXM4-80GB, torch 2.11.0+cu128 / CUDA 12.8, and for CPU rows an Intel Xeon @ 2.20GHz (12 vCPU). Your hardware will differ; treat these as shape, not promise.

Size

Parameters	466,576,896
Checkpoint on disk (fp32)	1.87 GB
Weights in memory, fp32 / fp16	1.87 GB / 0.93 GB
Peak activations, batch 8	0.32 GB (fp32), 0.17 GB (fp16)
Total VRAM needed	~2.3 GB fp32, ~1.2 GB fp16

It fits on a 4 GB GPU.

Latency, single sentence (GPU, 6 entity + 5 relation labels)

sentence length	p50	p95
13 words	57.2 ms	57.2 ms
22 words	57.5 ms	58.8 ms
62 words	59.0 ms	64.2 ms

Latency is essentially flat in sentence length — it is dominated by fixed overhead, not sequence length.

Throughput and batch behaviour (GPU, fp32)

batch	p50	sentences/sec	peak activations
1	57.9 ms	17.8	0.03 GB
4	72.2 ms	55.5	0.17 GB
8	117.9 ms	68.1	0.32 GB
16	245.2 ms	65.2	0.39 GB
32	496.3 ms	64.7	0.39 GB
64	966.7 ms	66.2	0.39 GB

Throughput saturates at batch 8. Larger batches cost latency and buy nothing — memory stops growing past batch 16 because the library chunks internally. Use batch 8.

fp16 is close to free

	fp32	fp16
batch 8 latency	117.9 ms	70.3 ms
batch 8 throughput	68.1/s	113.8/s
batch 1 latency	57.9 ms	56.0 ms
Vietnamese strict F1	36.9	36.8
Vietnamese partial F1	56.3	56.6

1.7x faster at batch 8, half the activation memory, and accuracy is unchanged (the F1 differences are noise on a 267-sentence set). Call .half() for batch work. It buys nothing at batch 1, which is overhead-bound.

Label count is the biggest knob you control

relation labels passed	sentences/sec (batch 8)
1	78.9
5	67.7
10	57.7
18	47.2
30	39.3

entity labels passed	sentences/sec (batch 8, 5 relations)
2	72.7
6	68.2
10	61.1

Going from 1 to 30 relation labels doubles the cost per sentence. Pass only the labels you actually need for the job — this matters more than batch size tuning.

CPU inference

	latency	throughput
batch 1	379 ms	2.6/s
batch 8	3331 ms	2.4/s

CPU is ~6.5x slower per sentence at batch 1 and ~28x slower at batch 8 — batching does not help on CPU. Fine for interactive, one-document-at-a-time use. Not viable for bulk.

Serving

No inference provider hosts this model. Options that do work:

• PyTorch + .half(), batch 8 — the measured configuration above, ~114 sentences/sec on one A100
• The live demo Space runs on ZeroGPU and is the quickest way to try it

Using it well

Two measured facts change how you should call this model.

Feed it sentences, not pages. Recall falls off a cliff as input grows, while precision stays flat:

input length	P	R	F1
1 sentence (~32 words)	55.9	58.3	57.1
2 sentences (~64 words)	57.8	55.1	56.4
4 sentences (~127 words)	59.0	42.8	49.6
8 sentences (~250 words)	57.1	25.4	35.2

The model returns a roughly fixed number of relations regardless of how much text you give it, so a whole page loses more than half of them. Split on sentence boundaries and batch the pieces.

The score ranks well but is not a probability. Measured precision by score band:

score	predictions	actually correct
0.9 - 1.0	50	84%
0.8 - 0.9	137	66%
0.7 - 0.8	108	32%
0.6 - 0.7	131	20%

A score of 0.8 does not mean 80% confidence. Note the cliff between the 0.7-0.8 and 0.8-0.9 bands: use relation_threshold=0.8 if you care about precision, 0.7 for balanced F1.

Per-relation thresholds were tried and do not help: tuned on half the evaluation set they scored worse on the held-out half (54.2 vs 56.7 partial F1). One global threshold is the right call.

Robustness

Same 267-sentence set, same decoding, with the input text distorted. Partial-match F1:

input	base model	this model	this model keeps
as written	32.9	57.1	100%
all lowercase	29.8	50.1	88%
ALL UPPERCASE	16.5	14.1	25%
diacritics stripped	16.8	19.0	33%

Three things to take from this table.

Uppercase text is a regression. On ALL-CAPS input this model scores below the base model (14.1 vs 16.5). It does not produce wrong answers — precision actually rises to 75.9 — it goes quiet, dropping from 295 predictions to 29. Headings, form fields and shouty administrative documents will return almost nothing. Lowercase such text before passing it in.

Diacritic-stripped Vietnamese loses most of its recall (58.3 to 11.7). Vietnamese is often typed without diacritics, so this matters in practice. If you must process unaccented text, keep your relation labels accented — stripping the labels too makes it worse (F1 10.3 vs 19.0).

Casing carries span boundaries. Lowercasing costs only 7 points of partial F1 but nearly half of strict F1 (36.7 to 22.6): the model still finds the right pair, but stops getting the exact extent right.

The base model is fragile here too — it keeps 50-51% on the same distortions — so this is largely inherited rather than introduced. But fine-tuning made it relatively worse, and on uppercase input absolutely worse. Fixing this is the next release.

Known limitations

Measured, and worth reading before you deploy this.

Training labels are noisy. A hand audit of 450 training sentences found 40.7% with a wrong relation label and a further 32.0% needing span correction. This is inherent to distant supervision. Weak relation types track label noise rather than example count: the six weakest carry a 54.7% bad-label rate versus 33.7% for the rest.

Weakest relation types (see the table above): successor of 9.1, located in 34.8, signed document 34.3, founder of 35.3, owned by 44.4.

Long spans are unreliable. Span exactness against 566 gold endpoints, by length: 1 word 96.3%, 2 words 94.5%, 3-4 words 67.2%, 5+ words 40.0%. Vietnamese proper names often contain lowercase syllables internally, which defeats capitalisation-based span heuristics.

About 22% of gold triples are not recovered at any threshold. Dropping the relation threshold from 0.7 to 0.5 recovers only 41.5% of the misses.

Relation direction on an 8-case probe: base 6/8, this model 5/8. Too small to be meaningful, but direction was not verified at scale and should not be assumed correct.

License and attribution

Apache 2.0, inherited from knowledgator/gliner-relex-large-v1.0. Training data derives from Vietnamese Wikipedia (CC BY-SA 4.0).

Released by Belumind.