NAMAA-Space/KnowledgeGraphEval-2026-AdaptNER-Experiments

NAMAA — KnowledgeGraphEval 2026 AdaptNER: Development-Phase Experiments

KnowledgeGraphEval 2026 · Subtask 1 (AdaptNER) · ArabicNLP 2026

This is the development-phase experiment record of team NAMAA. It holds the training and inference notebooks, the per-run metrics and configurations, the Konooz development-set predictions and submissions for every candidate system, the evaluation scripts, and the software environment.

How this relates to the paper. These experiments come before the submitted systems described in the camera-ready paper. The submitted AdaptNER system ranked 1st (test micro-F1 75.96). It is an exact-span majority vote over ARBERTv2 + CRF, XLM-R-large + CRF and ARBERTv2 + GlobalPointer. The submitted RE system ranked 2nd (93.48). It is a majority vote over LoRA-fine-tuned Qwen3-8B, Qwen3.5-4B and Llama-3.1-8B-Instruct. Full test tables are in §2.

The fine-tuned weights are in their own model repositories:

Checkpoint	Formulation	Wojood val F1	Wojood test F1	Konooz dev F1
KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Nested	nested, multi-label	0.9245	0.9284	0.73
KnowledgeGraphEval-2026-AdaptNER-MARBERTv2-Nested	nested, multi-label	0.9071	0.9092	0.67
KnowledgeGraphEval-2026-AdaptNER-AraBERTv02-Flat	flat, single-label	0.9003	not recorded	0.69

All scores are entity-level strict micro-F1.

---

Repository layout

├── experiments/                      per-run metrics + full config (repro_bundle.json)
│   ├── EXP-001_AraBERTv02_Flat/
│   ├── EXP-001_AraBERTv02_Nested/
│   ├── EXP-002_MARBERTv2_Flat/       weights not released; test predictions kept here
│   └── EXP-002_MARBERTv2_Nested/
├── notebooks/
│   ├── training/                     Wojood fine-tuning, one notebook per run
│   ├── konooz_dev_inference/         load weights from the Hub, predict Konooz dev, write submission
│   ├── blind_test_inference/         blind-test inference for the ARBERTv2 (unweighted) system
│   └── eda/                          Wojood + Konooz exploratory analysis
├── konooz_dev_predictions/
│   ├── namaa_wojood_runs/            EXP-001/002 predictions + submission zips
│   ├── backbone_sweep/               the nested backbone comparison (paper Table 4)
│   └── ensembles_and_llm/            top-3 / top-4 voting ensembles, LLM run
├── evaluation/                       per-entity-type and per-domain scorers + results
├── scripts/                          ensemble voting, Konooz EDA, hosted-LLM baseline
└── environment/pip_freeze.txt

---

1. Wojood experiments (in-domain)

Same recipe for all four runs, taken from repro_bundle.json: 8 epochs, learning rate 3e-5, batch 32, warmup 0.1, weight decay 0.01, AdamW, bf16, max length 256, seed 42, early-stopping patience 3. The nested runs use a 42-way sigmoid head (21 types × {B, I}) with a 0.5 threshold. The flat runs use a softmax BIO head.

Run	Encoder	Formulation	Val P	Val R	Val F1	Test P	Test R	Test F1
EXP-001	AraBERTv02	flat	0.8908	0.9101	0.9003	—	—	—
EXP-001	AraBERTv02	nested	0.9178	0.9313	0.9245	0.9208	0.9362	0.9284
EXP-002	MARBERTv2	flat	0.8876	0.9087	0.8980	0.8979	0.9171	0.9074
EXP-002	MARBERTv2	nested	0.9042	0.9100	0.9071	0.9058	0.9127	0.9092

The EXP-001 flat run did not log test-split metrics. Flat and nested scores are measured on different variants of Wojood, so compare them as indicative only.

Per-entity-type macro-F1 on the Wojood test split (evaluation/pertype_results.json):

	AraBERTv02 nested	MARBERTv2 nested
micro-F1	0.9284	0.9092
macro-F1 (21 types)	0.8411	0.5877

MARBERTv2 nested makes no predictions at all for PERCENT, PRODUCT, UNIT and QUANTITY, so their F1 is 0.0000. These types are 0.6% of mentions, which is why a 2-point micro-F1 gap becomes a 25-point macro-F1 gap. AraBERTv02 scores higher on all 21 types and on all 13 Wojood sub-corpora. evaluation/compute_pertype.py and compute_perdomain.py regenerate these numbers from each model's test_predictions_conll.txt. The recomputed precision matches the original scorer exactly, and micro-F1 matches it to within 0.001.

---

2. Official test results (camera-ready paper)

Entity-level micro-averages under exact span-and-type matching, in percent.

Subtask 1 — AdaptNER (Konooz test). Ranked 1st.

System	Recall	Precision	Micro-F1
Shared-task baseline	63.87	62.30	63.00
ARBERTv2 + CRF	70.08	80.69	75.01
XLM-R-large + CRF	71.12	79.41	75.04
ARBERTv2 + GlobalPointer	66.17	84.02	74.03
Exact-span ensemble	69.44	83.83	75.96

Subtask 2 — Relation Extraction (WojoodRelations). Ranked 2nd.

System	Recall	Precision	Micro-F1
Shared-task baseline	–	–	93.09
Qwen3-8B (LoRA)	89.35	95.59	92.36
Qwen3.5-4B (LoRA)	89.58	96.20	92.77
Llama-3.1-8B-Instruct (LoRA)	89.03	96.21	92.48
Majority-vote ensemble	90.38	96.80	93.48

The checkpoints for these submitted systems are not in this repository.

3. Development-phase Konooz leaderboard

These are the team's scores on the Konooz development set (100 sentences) during development. They are not reported in the camera-ready paper.

System	Konooz dev F1	Predictions in this repo
Ensemble (top-3, weighted vote)	0.74	ensembles_and_llm/submission_ensemble_top3.zip
ARBERTv2 (unweighted)	0.74	backbone_sweep/arbertv2-unweighted_*
AraBERTv02, nested (EXP-001)	0.73	namaa_wojood_runs/exp001_nested_konooz/
ARBERTv2 (class-weighted)	0.72	backbone_sweep/arbertv2-weighted_*
XLM-R large	0.71	backbone_sweep/xlm-roberta-large_*
AraBERTv02, flat (EXP-001)	0.69	namaa_wojood_runs/exp001_flat_konooz/
MARBERT	0.68	backbone_sweep/marbert_*
mDeBERTa-v3 base	0.67	backbone_sweep/mdeberta-v3-base_*
MARBERTv2 (sweep run, not the released EXP-002 checkpoint)	0.67	backbone_sweep/marbertv2_*
ARBERTv2 + Konooz-MLM (last-3)	0.65	backbone_sweep/arbertv2-konooz-adapted-last3_*
AraBERTv02 baseline †	0.61	—
Qwen3-235B (zero-shot)	0.26	—

† Organiser reference baseline. On the development set, the top-3 ensemble matches the best single system on micro-F1 but raises macro-F1 from 0.49 to 0.55 (P = 0.75, R = 0.74).

backbone_sweep/ also contains predictions for CAMeLBERT-mix and ARBERTv2 + Konooz-MLM (full). These two runs were scored only on Wojood validation (paper Table 4) and were not submitted to the Konooz leaderboard.

Submission format: one token per line, followed by 21 space-separated IOB2 columns in this order:

CARDINAL CURR DATE EVENT FAC GPE LANGUAGE LAW LOC MONEY NORP
OCC ORDINAL ORG PERCENT PERS PRODUCT QUANTITY TIME UNIT WEBSITE

---

4. Reproducing

1. Get Wojood 1.1 and the Konooz development and blind-test data from SinaLab, Birzeit University. They are not redistributed here.
2. Training: run notebooks/training/<run>.ipynb. They were written for Google Colab (A100 or T4). Each run's exact configuration and library versions are in experiments/<run>/repro_bundle.json.
3. Konooz inference: notebooks/konooz_dev_inference/ loads the released checkpoints from the Hub by name.
4. Ensemble: scripts/ensemble_predictions.py does a weighted vote over per-system prediction files.

The notebooks read the Hub token from a Colab Secret or the HF_TOKEN environment variable. No credentials are stored in this repository.

---

Citation

@inproceedings{namaa-knowledgegrapheval-2026,
  title     = {{NAMAA} at {KnowledgeGraphEval} 2026: Entity-Span Voting and Constrained
               {LLM} Ensembles for {A}rabic Knowledge Graph Construction},
  author    = {Fathallah, Mahmoud and Zaytoon, Mohamed and Emad Eldin, Fatimah and
               A.azim, Mona and Nacar, Omer},
  booktitle = {Proceedings of the Fourth Arabic Natural Language Processing Conference
               (ArabicNLP 2026)},
  year      = {2026},
  address   = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics}
}

The corpus citations (Wojood, Konooz) and backbone citations are in the model cards linked above.

License

Apache-2.0 covers the code, notebooks and predictions in this repository. Wojood and Konooz remain under SinaLab's original terms. Prediction files contain corpus tokens, so follow those terms if you redistribute them.