HassanB4/aragenre-13model-combined-devtuned

AraGenre 13-Model Combined Ensemble — Dev-Tuned Weights

The largest ensemble in the project, combining the trained encoder pipeline with the additional pipeline via a 13-member dev-validated weight search, for hierarchical Arabic genre classification. This is not a single fine-tuned checkpoint — the repo contains a weights JSON and a combination script, not trained weights of its own.

Authors: Hassan Barmandah (NAMAA Community; Umm Al-Qura University), Israa Elhosiny (NAMAA Community), Yousra El-Ghawi (NAMAA Community), Omer Nacar (NAMAA Community)

⚠️ Generalization Note

This ensemble's development-set score (0.9917 hierarchical F1) — the highest dev number reported anywhere in this project — is not representative of real-world performance. Per the project's system-description paper, this entire lineage of fine-tuned/ensembled sentence encoders — which scored well on the 110-item, 6-genre AraGenre dev set — collapsed to 0.22–0.44 hierarchical F1 on the actual 27,972-item hidden test set (74 specific genres under 6 broad genres). Its 13 component weights were fit via dev-validated random search, so the 0.9917 number reflects fitting to dev, not just evaluation on it.

The system that actually won for this team — 0.7013 hierarchical F1, 3rd of 18 teams on the official CodaBench leaderboard — was a separate, zero-shot DeepSeek-LLM pipeline with no fine-tuning at all (stage2_llm_zeroshot_pipeline/ in the project repo). This artifact is not that system. It is released here for transparency and reproducibility of the project's full experimental record, not as a recommended production classifier.

Approach

Extends the 9-model ensemble's dev-validated random weight search from 9 members to 13: the original BGE/E5 trained-pipeline models plus mdeberta-nli-zeroshot (score-based), e5-large-paraphrase-augmented (score-based), e5-large-multicorpus-xgenre (score-based), and llm-zeroshot-reasoning folded in as a one-hot pseudo-score vector (1.0 on its predicted genre, 0 elsewhere) so a hard-voting member can sit in the same weighted-softmax-average framework as the rest. Same dev-validated random-search methodology as the 8/9-model ensembles, applied over a more architecturally diverse member set.

For a no-dev-tuning counterpart, see aragenre-combined-generalist-equalweight.

Components

The 9-model ensemble's members plus mdeberta-nli-zeroshot, llm-zeroshot-reasoning (one-hot), e5-large-paraphrase-augmented, e5-large-multicorpus-xgenre — 13 members total.

Training data

None directly — this is a weight recipe over pre-scored component models. The component weights were selected via a dev-validated random search against dev_gold.json labels.

Usage

Requires cached dev scores/predictions from running all 13 component scripts first, then:

python ensemble_13model_combined_devtuned.py

See the project repository for the full script and component-model requirements.

Citation

If you use this work, please cite our system-description paper:

@inproceedings{barmandah-etal-2026-namaa,
  title = {NAMAA at AraGenre 2026: From Encoder Baselines to Self-Consistent LLM Ensembling for Hierarchical Arabic Genre Classification},
  author = {Barmandah, Hassan and Elhosiny, Israa and El-Ghawi, Yousra and Nacar, Omer},
  booktitle = {Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026)},
  address = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics},
  year = {2026},
}

Please also cite the AraGenre 2026 shared task overview paper:

@inproceedings{elhaj-etal-2026-aragenre,
  title = {AraGenre 2026: A Hierarchical Definition-Guided Arabic Genre Classification Shared Task},
  author = {El-Haj, Mo and Ezzini, Saad and Abudalfa, Shadi and Lamsiyah, Salima and Jarrar, Mustafa},
  booktitle = {Proceedings of the 4th Arabic Natural Language Processing Conference (ArabicNLP 2026)},
  address = {Budapest, Hungary},
  publisher = {Association for Computational Linguistics},
  year = {2026},
}

License

Apache 2.0