Swan and ArabicMTEB: Dialect-Aware, Arabic-Centric, Cross-Lingual, and
  Cross-Cultural Embedding Models and Benchmarks
Authors: name: Gagan Bhatia; name: El Moatez Billah Nagoudi; name: Abdellah El Mekki; name: Fakhraddin Alwajih; name: Muhammad Abdul-Mageed
Published: 2024-11-02 09:39:49

Abstract: We introduce Swan, a family of embedding models centred around the Arabic
language, addressing both small-scale and large-scale use cases. Swan includes
two variants: Swan-Small, based on ARBERTv2, and Swan-Large, built on
ArMistral, a pretrained Arabic large language model. To evaluate these models,
we propose ArabicMTEB, a comprehensive benchmark suite that assesses
cross-lingual, multi-dialectal, multi-domain, and multi-cultural Arabic text
embedding performance, covering eight diverse tasks and spanning 94 datasets.
Swan-Large achieves state-of-the-art results, outperforming
Multilingual-E5-large in most Arabic tasks, while the Swan-Small consistently
surpasses Multilingual-E5 base. Our extensive evaluations demonstrate that Swan
models are both dialectally and culturally aware, excelling across various
Arabic domains while offering significant monetary efficiency. This work
significantly advances the field of Arabic language modelling and provides
valuable resources for future research and applications in Arabic natural
language processing. Our models and benchmark will be made publicly accessible
for research.