Atlas-Chat: Adapting Large Language Models for Low-Resource Moroccan
  Arabic Dialect
Authors: name: Guokan Shang; name: Hadi Abdine; name: Yousef Khoubrane; name: Amr Mohamed; name: Yassine Abbahaddou; name: Sofiane Ennadir; name: Imane Momayiz; name: Xuguang Ren; name: Eric Moulines; name: Preslav Nakov; name: Michalis Vazirgiannis; name: Eric Xing
Published: 2024-09-26 14:56:38

Abstract: We introduce Atlas-Chat, the first-ever collection of large language models
specifically developed for dialectal Arabic. Focusing on Moroccan Arabic, also
known as Darija, we construct our instruction dataset by consolidating existing
Darija language resources, creating novel datasets both manually and
synthetically, and translating English instructions with stringent quality
control. Atlas-Chat-9B and 2B models, fine-tuned on the dataset, exhibit
superior ability in following Darija instructions and performing standard NLP
tasks. Notably, our models outperform both state-of-the-art and
Arabic-specialized LLMs like LLaMa, Jais, and AceGPT, e.g., achieving a 13%
performance boost over a larger 13B model on DarijaMMLU, in our newly
introduced evaluation suite for Darija covering both discriminative and
generative tasks. Furthermore, we perform an experimental analysis of various
fine-tuning strategies and base model choices to determine optimal
configurations. All our resources are publicly accessible, and we believe our
work offers comprehensive design methodologies of instruction-tuning for
low-resource language variants, which are often neglected in favor of data-rich
languages by contemporary LLMs.