NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested

NAMAA Community — AdaptNER (MARBERTv2, nested)

KnowledgeGraphEval 2026 · Subtask 1 (AdaptNER) · ArabicNLP 2026

Multi-label nested Arabic Named Entity Recognition. This checkpoint fine-tunes UBC-NLP/MARBERTv2 on the nested variant of the Wojood corpus, predicting 21 entity types simultaneously so that overlapping mentions are preserved.

It is released as the dialect-oriented backbone comparison point from the team's development-phase encoder sweep. It is not the team's best system.

How this checkpoint relates to the paper. This is a development-phase checkpoint from the NAMAA submission to KnowledgeGraphEval 2026. It is not the system described in the camera-ready paper. The submitted AdaptNER system, which ranked 1st with a test micro-F1 of 75.96, is an exact-span majority vote over three models: ARBERTv2 + CRF, XLM-R-large + CRF, and ARBERTv2 + GlobalPointer. See §6. We release this checkpoint because the team's in-domain Wojood experiments were built on it.

⚠️ Read this before using the model

This checkpoint never predicts four of the 21 entity types. PERCENT, PRODUCT, UNIT and QUANTITY all score exactly F1 = 0.0000 on the Wojood test set, and CURR is effectively dead at 0.0465 (1 of 41 mentions recovered). Its headline micro-F1 of 0.9092 looks close to the AraBERTv02 sibling's 0.9284, but macro-F1 over the 21 types is 0.5877 vs 0.8411 — a 25-point gap.

For general-purpose nested Arabic NER, use KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested instead: it beats this model on all 21 types. Full breakdown in §5.3.

---

1. Model details

Developed by	NAMAA Community (see Citation)
Team / submission	NAMAA at KnowledgeGraphEval 2026 (development-phase checkpoint)
Experiment ID	EXP-002 (nested)
Model type	BERT encoder + multi-label token classification head
Base encoder	UBC-NLP/MARBERTv2
Language	Arabic (dialect-heavy pre-training; MSA + Levantine fine-tuning)
Parameters	162,283,050 (F32)
Architecture	12 layers, hidden 768, 12 heads, vocab 100,000, max positions 512
Label space	42 outputs = 21 entity types x {B, I}; O is implicit
Decision rule	Independent sigmoid per label, threshold 0.5
Max sequence length	256 sub-words
License	Apache-2.0
Finetuned from	MARBERTv2 (Abdul-Mageed et al., 2021)

The larger parameter count relative to the AraBERTv02 sibling (162M vs 135M) is entirely the 100,000-entry vocabulary versus 64,000; the transformer body is the same size.

Task head

The nested formulation treats each token as carrying a set of tags rather than one tag. The head emits 42 independent logits (B-<TYPE> and I-<TYPE> for each of the 21 types); every logit is squashed with a sigmoid and thresholded at 0.5. A token with no label above threshold is O. Entity spans are then BIO-decoded per type channel.

---

2. Intended use

In scope

• Backbone-comparison and ablation studies for Arabic nested NER.
• Reproducing the team's development-phase encoder comparison.
• Ensemble membership, where its errors are decorrelated from AraBERT-family members.

Out of scope

• Any application needing PERCENT, PRODUCT, UNIT, QUANTITY or CURR. The model does not predict them. Numeric/measurement extraction will silently return nothing.
• General-purpose nested Arabic NER — the AraBERTv02 sibling dominates it on every type.
• Same-type nested mentions (an ORG inside another ORG) — see Limitations.
• Languages other than Arabic.

---

3. Task and data

KnowledgeGraphEval 2026 (Aljabari et al., 2026) is the first ArabicNLP shared task on Arabic knowledge-graph construction. It splits the problem into Subtask 1 (AdaptNER) — cross-domain nested NER — and Subtask 2 (RE) — relation extraction. Both are scored with strict entity-level micro-F1.

Role	Dataset	Size	Notes
Training / model selection	Wojood 1.1, nested variant (Jarrar et al., 2022)	23,125 train / 3,304 dev / 6,606 test sentences	~448K tokens, ~72K mentions, 21 entity types, 22.5% of mentions nested
Official cross-domain evaluation	Konooz (Hamad et al., 2025)	~50K tokens, 10 unseen MSA domains	Dev release: 100 sentences (10 per domain), 3,166 tokens, unlabelled

The model never sees Konooz during training — that domain shift is the challenge the task targets.

Konooz domains: Agriculture, Art, Economics, Finance, Health, History, Law, Politics, Science, Sport.

Label space

21 entity types, in the official submission column order (a mis-ordered submission is scored against the wrong type channels and collapses the score):

CARDINAL, CURR, DATE, EVENT, FAC, GPE, LANGUAGE, LAW, LOC, MONEY, NORP,
OCC, ORDINAL, ORG, PERCENT, PERS, PRODUCT, QUANTITY, TIME, UNIT, WEBSITE

---

4. Training configuration

Recorded verbatim in repro_bundle.json shipped with this repo. Identical to the AraBERTv02 sibling — the two differ only in base encoder, so the comparison in §7 is controlled.

Hyperparameter	Value
Epochs	8
Learning rate	3e-5
Batch size (train / eval)	32 / 64
Gradient accumulation	1
Optimizer	AdamW
Weight decay	0.01
Warmup ratio	0.1
Max length	256
Prediction threshold	0.5
Precision	bf16 (+ TF32)
Early stopping	patience 3, monitor dev micro-F1
Seed	42
Sub-word alignment	label on first sub-word; continuations masked (-100)

Environment — Python 3.12.13, PyTorch 2.11.0+cu128, Transformers 5.10.2, Datasets 4.0.0, CUDA 12.8, Tesla T4. Arabic normalisation via CAMeL Tools (Obeid et al., 2020).

The AraBERTv02 sibling trained on an A100-40GB under Transformers 5.13.1; this run used a T4 under 5.10.2. Hyperparameters match, but the environments are not bit-identical.

---

5. Evaluation

5.1 Methodology

Entity-level strict micro-F1: a prediction counts as correct only if the span boundaries and the entity type match the gold mention exactly.

micro-F1 = 2TP / (2TP + FP + FN)

Spans are decoded independently per type channel, then compared as a set of (sentence, type, start, end) tuples.

5.2 Overall results

Split	Precision	Recall	micro-F1	Macro-F1 (21 types)
Wojood dev (in-domain)	0.9042	0.9100	0.9071	—
Wojood test (in-domain)	0.9058	0.9127	0.9092	0.5877
Konooz dev (cross-domain, official)	—	—	0.67	—

Test-set totals: 18,210 gold entities, 18,316 predicted, TP 16,590 / FP 1,726 / FN 1,620.

Two independent gaps are visible here and both matter:

• Micro vs macro (0.9092 vs 0.5877). A class-coverage failure, detailed in §5.3.
• In-domain vs cross-domain (0.9092 vs 0.67). The −0.24 F1 domain-shift drop the task targets.

5.3 Per-entity-type breakdown (Wojood test)

Support = gold mention count. Sorted by frequency. The rightmost column is the AraBERTv02 sibling trained on identical data and hyperparameters.

Type	Support	Precision	Recall	F1	AraBERTv02 F1
GPE	4,344	0.9609	0.9620	0.9615	0.9657
ORG	3,928	0.9098	0.9374	0.9234	0.9341
DATE	3,206	0.9509	0.9604	0.9556	0.9576
PERS	1,571	0.9195	0.9313	0.9254	0.9369
OCC	1,126	0.8798	0.9032	0.8913	0.9074
NORP	1,095	0.7736	0.8237	0.7979	0.8127
ORDINAL	1,083	0.9704	0.9391	0.9545	0.9562
EVENT	570	0.7141	0.7667	0.7394	0.8258
CARDINAL	342	0.8475	0.8450	0.8463	0.8882
FAC	246	0.6883	0.6911	0.6897	0.8200
LOC	219	0.8010	0.7352	0.7667	0.8578
WEBSITE	116	0.5644	0.4914	0.5253	0.7016
LAW	90	0.7400	0.8222	0.7789	0.8342
TIME	84	0.6341	0.3095	0.4160	0.7241
LANGUAGE	44	1.0000	0.2273	0.3704	0.9048
CURR	41	0.5000	0.0244	0.0465	0.9136
MONEY	33	0.7222	0.7879	0.7536	0.8493
PERCENT	33	0.0000	0.0000	0.0000	0.9697
PRODUCT	19	0.0000	0.0000	0.0000	0.7368
UNIT	11	0.0000	0.0000	0.0000	0.6667
QUANTITY	9	0.0000	0.0000	0.0000	0.5000
Macro average				0.5877	0.8411

This is the most important table in the card. Four types are never predicted at all (PERCENT, PRODUCT, UNIT, QUANTITY — zero precision and zero recall, so no predictions were emitted on those channels). Three more are severely degraded:

Type	Symptom
CURR	recall 0.0244 — 1 of 41 mentions found
LANGUAGE	precision 1.0000, recall 0.2273 — extremely conservative, 10 of 44 found
TIME	recall 0.3095 — under half the AraBERTv02 recall

The pattern is consistent: the sigmoid head at a fixed 0.5 threshold collapses on low-support numeric and measurement classes. Those five types (PERCENT, PRODUCT, UNIT, QUANTITY, CURR) total only 113 of 18,210 mentions (0.6%), so micro-F1 barely registers their loss — which is exactly why the macro average, not the headline number, is the honest summary for this checkpoint.

The AraBERTv02 sibling, trained identically, keeps all 21 channels alive. The difference is attributable to the encoder, not the recipe.

5.4 Per-domain breakdown (Wojood test)

F1 is the published figure from test_metrics.json. Precision and recall are recomputed from the shipped test_predictions_conll.txt against gold; they reproduce the published F1 to within ±0.02 (small residual from duplicate sentences during domain re-alignment).

Sub-corpus	Sentences	Gold entities	Precision	Recall	F1
Awraq_G3	1,430	4,503	0.9684	0.9740	0.9706
Awraq_G2	979	3,493	0.9409	0.9530	0.9463
Awraq_G1	931	3,264	0.9369	0.9513	0.9426
PCMA	209	1,181	0.8788	0.8899	0.8841
Curras	1,045	916	0.7898	0.7631	0.7898
T2: Foreign policy	288	716	0.7858	0.7891	0.7836
T1: Terrorism	276	628	0.8775	0.9124	0.8865
T4: Elections	322	620	0.9172	0.9290	0.9231
T6: Health policy	216	516	0.8484	0.8353	0.8411
T3: Security	166	382	0.8016	0.7827	0.7953
QUORA	530	376	0.6580	0.6090	0.6267
T5: Migration	129	268	0.8346	0.7910	0.8054
Lebanese	85	60	0.5660	0.5000	0.5426

Notably, MARBERTv2's dialect-heavy pre-training does not translate into an advantage on the dialect sub-corpora: it trails AraBERTv02 on Curras (0.7898 vs 0.8439), Lebanese (0.5426 vs 0.6074) and QUORA (0.6267 vs 0.6877). The expected dialect benefit does not materialise under this task formulation.

5.5 Per-domain breakdown (Wojood dev)

Sub-corpus	Sentences	F1
Awraq_G3	714	0.9772
Awraq_G2	490	0.9539
Awraq_G1	466	0.9410
T4: Elections	161	0.9214
T1: Terrorism	138	0.8729
PCMA	105	0.8645
T5: Migration	65	0.8547
T3: Security	83	0.8367
T6: Health policy	108	0.8013
Curras	523	0.7801
T2: Foreign policy	144	0.7714
Lebanese	42	0.5970
QUORA	265	0.5567

---

6. Official shared-task results

These are the official test results for team NAMAA, as reported in the camera-ready paper. All scores are entity-level micro-averages under exact span-and-type matching, in percent. This checkpoint is not one of the submitted systems.

Subtask 1 — AdaptNER (Konooz test, 10 unseen domains). Ranked 1st.

System	Recall	Precision	Micro-F1
Shared-task baseline	63.87	62.30	63.00
ARBERTv2 + CRF	70.08	80.69	75.01
XLM-R-large + CRF	71.12	79.41	75.04
ARBERTv2 + GlobalPointer	66.17	84.02	74.03
Exact-span ensemble (≥2 of 3 votes)	69.44	83.83	75.96

Subtask 2 — Relation Extraction (WojoodRelations, 41 labels). Ranked 2nd.

System	Recall	Precision	Micro-F1
Shared-task baseline	–	–	93.09
Qwen3-8B (LoRA)	89.35	95.59	92.36
Qwen3.5-4B (LoRA)	89.58	96.20	92.77
Llama-3.1-8B-Instruct (LoRA)	89.03	96.21	92.48
Majority-vote ensemble	90.38	96.80	93.48

Development-phase Konooz results

During development the team scored its candidate encoders on the Konooz development set (100 sentences, entity-level micro-F1). These scores are not in the camera-ready paper. We include them here because they are the only cross-domain numbers available for this checkpoint.

System	Konooz dev F1	System	Konooz dev F1
ARBERTv2 (unweighted)	0.74	MARBERT	0.68
Top-3 weighted-vote ensemble	0.74	mDeBERTa-v3 base	0.67
AraBERTv02, nested	0.73	MARBERTv2 ‡	0.67
ARBERTv2 (class-weighted)	0.72	ARBERTv2 + Konooz-MLM (last-3)	0.65
XLM-R large	0.71	AraBERTv02 organiser baseline	0.61
AraBERTv02, flat	0.69	Qwen3-235B (zero-shot)	0.26

‡ This MARBERTv2 entry may be a different run from the released MARBERTv2-Nested checkpoint. That entry's in-domain validation F1 was 0.873, while the released checkpoint's is 0.9071.

---

7. Sibling checkpoints

Three checkpoints from this submission are released. All share the same data, seed, hyperparameters and label space.

Checkpoint	Formulation	Wojood dev F1	Wojood test F1	Macro-F1 (test)	Konooz dev F1
AraBERTv02 nested	nested multi-label	0.9245	0.9284	0.8411	0.73
MARBERTv2 nested (this model)	nested multi-label	0.9071	0.9092	0.5877	0.67
AraBERTv02 flat	flat single-label	0.9003	n/a	n/a	0.69

The controlled AraBERTv02-vs-MARBERTv2 comparison is the reason this checkpoint is published: same data, same seed, same recipe, different encoder. AraBERTv02 wins on all 21 entity types and on all 13 sub-corpora.

---

8. Limitations and known issues

• Four entity types are never predicted. PERCENT, PRODUCT, UNIT, QUANTITY return nothing. This is a silent failure: the model emits no error, just no entities on those channels. Do not deploy it where those types matter.
• CURR, LANGUAGE and TIME are severely under-recalled (0.0244 / 0.2273 / 0.3095).
• Threshold is untuned. 0.5 was fixed a priori. Because the dead classes fail on recall with zero predictions, lowering the per-type threshold is the first thing to try before concluding the capability is absent from the weights — the signal may be present below 0.5.
• Same-type nesting is unrecoverable. One BIO channel per type cannot encode an ORG nested inside another ORG. A biaffine span head (Yu et al., 2020) would remove this limitation.
• Cross-domain degradation. −0.24 F1 from Wojood to Konooz, worse than the AraBERTv02 sibling's −0.20.
• No dialect advantage observed, despite MARBERTv2's dialect-heavy pre-training (§5.4).
• Rare-type estimates are noisy. Types with fewer than ~40 gold mentions cannot support a stable F1 estimate; treat those rows as indicative.
• Column order matters. Submissions are token + 21 space-separated B/I/O columns in the exact order listed in §3.

Head architecture

The released head is a multi-label sigmoid + BCE head. config.json has exactly 42 labels (21 × {B, I}, no O class), and repro_bundle.json records pred_threshold: 0.5, so each B/I channel is an independent sigmoid. This differs from the BIO-CRF and GlobalPointer heads of the submitted system described in the camera-ready paper.

---

9. Usage

from transformers import AutoTokenizer, AutoModelForTokenClassification
import torch

REPO = "NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested"
tok = AutoTokenizer.from_pretrained(REPO)
model = AutoModelForTokenClassification.from_pretrained(REPO).eval()

def predict(words, threshold=0.5):
    """Nested NER: returns one tag-set per input word."""
    enc = tok(words, is_split_into_words=True, truncation=True,
              max_length=256, return_tensors="pt")
    with torch.no_grad():
        probs = torch.sigmoid(model(**enc).logits)[0]

    out, seen = [], set()
    for pos, wid in enumerate(enc.word_ids(0)):   # first sub-word of each word only
        if wid is None or wid in seen:
            continue
        seen.add(wid)
        tags = {model.config.id2label[i]
                for i in (probs[pos] > threshold).nonzero().flatten().tolist()}
        out.append((words[wid], tags or {"O"}))
    return out

sentence = "حرب 1948 ( النكبة ) هجرت الفلسطينيون من فلسطين".split()
for word, tags in predict(sentence):
    print(f"{word}\t{'|'.join(sorted(tags))}")

Lower threshold if you need any recall at all on PERCENT, PRODUCT, UNIT, QUANTITY or CURR; at the default 0.5 those channels are empty.

---

10. Reproducibility

Files shipped with this repo:

File	Contents
repro_bundle.json	Full config, resolved data paths, label set, seed, library versions
final_metrics.json	Wojood dev overall + per-domain F1
test_metrics.json	Wojood test overall + per-domain F1
test_predictions_conll.txt	Test-set predictions, CoNLL format (token, tab, pipe-joined tags)

The per-type table in §5.3 was computed by BIO-decoding test_predictions_conll.txt per type channel and comparing against gold as a set of (sentence, type, start, end) tuples; it reproduces the published precision exactly and micro-F1 to within 0.001.

Training notebooks, per-run configs, development-phase Konooz predictions and the evaluation scripts are in KnowledgeGraphEval-2026-AdaptNER-Experiments.

---

11. Citation

This system

@inproceedings{namaa-knowledgegrapheval-2026,
  title     = {{NAMAA} at {KnowledgeGraphEval} 2026: Entity-Span Voting and Constrained
               {LLM} Ensembles for {A}rabic Knowledge Graph Construction},
  author    = {Fathallah, Mahmoud and Zaytoon, Mohamed and Emad Eldin, Fatimah and
               A.azim, Mona and Nacar, Omer},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference
               (ArabicNLP 2026)},
  year      = {2026},
  address   = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics}
}

Affiliations: Alexandria University; Trouve Labs; Ain Shams University; Tuwaiq Academy; NAMAA Community.

Shared task

@inproceedings{aljabari-etal-2026-knowledgegrapheval,
  title     = {{KnowledgeGraphEval} 2026: The First Shared Task on {A}rabic Knowledge Graphs},
  author    = {Aljabari, Alaa and Hamad, Nagham and El Mekki, Abdellah and
               Abdul-Mageed, Muhammad and Zitouni, Imed and Chawla, Sanjay and Jarrar, Mustafa},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference
               (ArabicNLP 2026)},
  year      = {2026},
  publisher = {Association for Computational Linguistics}
}

Training corpus — Wojood

@inproceedings{jarrar-etal-2022-wojood,
  title     = {{W}ojood: Nested {A}rabic Named Entity Corpus and Recognition using {BERT}},
  author    = {Jarrar, Mustafa and Khalilia, Mohammed and Ghanem, Sana},
  booktitle = {Proceedings of the Thirteenth Language Resources and Evaluation Conference (LREC)},
  year      = {2022},
  address   = {Marseille, France},
  pages     = {3626--3636},
  publisher = {European Language Resources Association},
  url       = {https://aclanthology.org/2022.lrec-1.387/}
}

Evaluation corpus — Konooz

@inproceedings{hamad-etal-2025-konooz,
  title     = {{K}onooz: Multi-domain Multi-dialect Corpus for Named Entity Recognition},
  author    = {Hamad, Nagham and Khalilia, Mohammed and Jarrar, Mustafa},
  booktitle = {Findings of the Association for Computational Linguistics: ACL 2025},
  year      = {2025},
  pages     = {7316--7331},
  publisher = {Association for Computational Linguistics},
  url       = {https://aclanthology.org/2025.findings-acl.382/}
}

Base encoder — MARBERTv2

@inproceedings{abdul-mageed-etal-2021-arbert,
  title     = {{ARBERT} \& {MARBERT}: Deep Bidirectional Transformers for {A}rabic},
  author    = {Abdul-Mageed, Muhammad and Elmadany, AbdelRahim and
               Nagoudi, El Moatez Billah},
  booktitle = {Proceedings of the 59th Annual Meeting of the Association for Computational
               Linguistics and the 11th International Joint Conference on Natural Language
               Processing (Volume 1: Long Papers)},
  year      = {2021},
  pages     = {7088--7105},
  publisher = {Association for Computational Linguistics},
  url       = {https://aclanthology.org/2021.acl-long.551/}
}

Also referenced

@inproceedings{obeid-etal-2020-camel,
  title     = {{CAM}e{L} Tools: An Open Source Python Toolkit for {A}rabic
               Natural Language Processing},
  author    = {Obeid, Ossama and Zalmout, Nasser and Khalifa, Salam and Taji, Dima and
               Oudah, Mai and Alhafni, Bashar and Inoue, Go and Eryani, Fadhl and
               Erdmann, Alexander and Habash, Nizar},
  booktitle = {Proceedings of the Twelfth Language Resources and Evaluation Conference (LREC)},
  year      = {2020},
  pages     = {7022--7032},
  url       = {https://aclanthology.org/2020.lrec-1.868/}
}

@inproceedings{yu-etal-2020-named,
  title     = {Named Entity Recognition as Dependency Parsing},
  author    = {Yu, Juntao and Bohnet, Bernd and Poesio, Massimo},
  booktitle = {Proceedings of the 58th Annual Meeting of the Association for
               Computational Linguistics (ACL)},
  year      = {2020},
  pages     = {6470--6476},
  url       = {https://aclanthology.org/2020.acl-main.577/}
}

---

12. Acknowledgements

We thank Tuwaiq Academy for its continued support of this work, and the organisers of KnowledgeGraphEval 2026 for providing the Wojood, Konooz and WojoodRelations resources and the evaluation framework.

Wojood and Konooz are distributed by SinaLab, Birzeit University under their own terms. The Apache-2.0 license here covers the fine-tuned weights and code, not the underlying corpora — users must obtain those under the original licenses.