HassanB4/aragenre-mdeberta-devgold-finetune

AraGenre mDeBERTa — Dev-Gold Entailment Fine-Tune

MoritzLaurer/mDeBERTa-v3-base-mnli-xnli fine-tuned as a cross-encoder entailment scorer for hierarchical Arabic genre classification, deliberately trained on the AraGenre dev_gold.json labels.

Authors: Hassan Barmandah (NAMAA Community; Umm Al-Qura University), Israa Elhosiny (NAMAA Community), Yousra El-Ghawi (NAMAA Community), Omer Nacar (NAMAA Community)

⚠️ Generalization Note

This model's 0.9917 hierarchical F1 is a 5-fold cross-validation estimate on the AraGenre dev set, not a real-world generalization estimate — the model is trained directly on dev labels by design (see Approach below for why), unlike the rest of the project's pipeline, which trains only on the 7 AraGenre TRAIN genres. Per the project's system-description paper, this model belongs to the broader fine-tuned-encoder lineage that collapsed to 0.22–0.44 hierarchical F1 on the actual 27,972-item hidden test set (74 specific genres under 6 broad genres) — training on dev labels does not close that gap, because the test set's official definitions and much larger taxonomy diverge from what dev training exposes the model to.

The system that actually won for this team — 0.7013 hierarchical F1, 3rd of 18 teams on the official CodaBench leaderboard — was a separate, zero-shot DeepSeek-LLM pipeline with no dev-label training at all (stage2_llm_zeroshot_pipeline/ in the project repo). This model is not that system. It is released here for transparency and reproducibility of the project's full experimental record, not as a recommended production classifier.

Approach

Cross-encoder entailment fine-tuning: MoritzLaurer/mDeBERTa-v3-base-mnli-xnli fine-tuned on (Arabic text, augmented English genre definition) pairs with a binary entailment label. Deliberately trains on dev_gold.json (unlike the rest of the pipeline, which trains only on the 7 TRAIN genres) — since the dev genres differ from the train genres, the 110 extra labeled dev examples materially help, and dev genres are not the true held-out test set for this task. The 5-fold cross-validation protocol exists specifically to get an honest, unbiased performance estimate despite training on dev.

Base model

MoritzLaurer/mDeBERTa-v3-base-mnli-xnli

Training data

dev_gold.json (110 examples) plus augmented definitions and synthetic hard-negative examples.

Hyperparameters

• Epochs: 20 (early stopping, patience 5)
• Batch size: 16
• Learning rate: 2e-5

Usage

Load with AutoModelForSequenceClassification.from_pretrained(...), score each (text, definition) pair, and take the definition with the highest P(entailment).

python mdeberta_devgold_finetune.py

Citation

If you use this work, please cite our system-description paper:

@inproceedings{barmandah-etal-2026-namaa,
  title = {NAMAA at AraGenre 2026: From Encoder Baselines to Self-Consistent LLM Ensembling for Hierarchical Arabic Genre Classification},
  author = {Barmandah, Hassan and Elhosiny, Israa and El-Ghawi, Yousra and Nacar, Omer},
  booktitle = {Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026)},
  address = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics},
  year = {2026},
}

Please also cite the AraGenre 2026 shared task overview paper:

@inproceedings{elhaj-etal-2026-aragenre,
  title = {AraGenre 2026: A Hierarchical Definition-Guided Arabic Genre Classification Shared Task},
  author = {El-Haj, Mo and Ezzini, Saad and Abudalfa, Shadi and Lamsiyah, Salima and Jarrar, Mustafa},
  booktitle = {Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026)},
  address = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics},
  year = {2026},
}

License

Apache 2.0