# GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data

Abderrahman Skiredj, Ferdaous Azhari, Houdaifa Atou, Nouamane Tazi, Ismail Berrada

► To cite this version:

Abderrahman Skiredj, Ferdaous Azhari, Houdaifa Atou, Nouamane Tazi, Ismail Berrada. GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data. 2025. hal-05083676

HAL Id: hal-05083676

https://hal.science/hal-05083676v1

Preprint submitted on 25 May 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# GemMaroc: Unlocking Darija Proficiency in LLMs with Minimal Data

Abderrahman Skiredj¹ Ferdaous Azhari³ Houdaifa Atou¹

Nouamane Tazi² Ismail Berrada¹

¹College of Computing, Mohammed VI Polytechnic University, Benguerir, Morocco

²Hugging Face, Paris, France

³National Institute of Posts and Telecoms, Rabat, Morocco

## Abstract

Open-source large language models (LLMs) still marginalise Moroccan Arabic (Darija), forcing practitioners either to bolt on heavyweight Arabic adapters or to sacrifice the very reasoning skills that make LLMs useful. We show that a rigorously *quality-over-quantity* alignment strategy can surface fluent Darija while safeguarding the backbone's cross-lingual reasoning—at a sliver of the usual compute. We translate three compact instruction suites—LIMA 1 K, DEITA 6 K and TULU 50 K—into Darija, preserve 20 % of the English originals, and add mathematics, coding and scientific prompts. A LoRA-tuned GEMMA 3-4B trained on ~5 K mixed instructions lifts *DarijaMMLU* from 32.8 % to 42.7 %; adding the reasoning-dense TULU portion pushes it to 47.5 % with no English regression. Scaling the identical recipe to GEMMA 3-27B produces **GemMaroc-27B**, which matches Atlas-Chat on *DarijaMMLU* (61.6 %) and *leaps* ahead on Darija commonsense, scoring 60.5 % on *HellaSwag* versus Atlas-Chat's 48.4 %. Crucially, GemMaroc retains Gemma-27B's strong maths and general-reasoning ability, showing only minimal movement on GSM8K and English benchmarks. The entire model is trained in just 48 GPU·h, underscoring a *Green AI* pathway to inclusive, sustainable language technology. We release code, data and checkpoints to spur Darija-centric applications in education, public services and everyday digital interaction.

## 1 Introduction

Open-Source Large Language Models (LLMs) have become the backbone of modern natural-language applications, yet most open models still overlook Moroccan Arabic (Darija), the everyday language of more than **36 million** speakers. This omission narrows digital inclusion and slows e-government initiatives that the *Morocco Digital*

2030 roadmap seeks to accelerate (Ministry of Digital Transition and Administration Reform, 2024). Because public-sector deployments must also safeguard data sovereignty and cost efficiency, an *open, compact*, and *Darija-fluent* model is an urgent prerequisite for truly accessible AI in the Maghreb.

Several ambitious Arabic-centric projects demonstrate what is possible—but at a price few communities can pay. **ALLaM** consumes trillions of tokens and multi-week GPU clusters to reach state-of-the-art Modern-Standard-Arabic scores (Bari et al., 2025); **Fanar** follows a similarly heavy pipeline. For Darija specifically, **Atlas-Chat** applies 458 k supervised examples and dozens of GPU-days to retrofit Gemma checkpoints (Shang et al., 2025). These efforts confirm the value of dialect alignment but leave open the core question: *can we surface Darija competence without massive supervision or carbon cost while preserving the backbone's cross-lingual reasoning power?*

We answer **yes**. Building on the minimal-data alignment insight of **MIG** (Maximising Information Gain) sampler (Chen et al., 2025) and the automatic selection heuristics of **DEITA** (6 k high-complexity pairs) (Liu et al., 2024), we translate three complementary instruction suites into Darija and retain 20 % of the original English prompts. A LoRA adapter on GEMMA 3-4B trained with only ≈5 k mixed instructions lifts *DarijaMMLU* from 32.8 % to **42.7 %**. Expanding the same recipe with 44 k reasoning-dense prompts (*TULU*) drives the score to 47.5 % while improving English GSM8K maths by +2.8 pp. Scaling to GEMMA 3-27B yields **GemMaroc-27B**, which matches Atlas-Chat on *DarijaMMLU* (61.6 %) yet beats it by **+12.1 pp** on the harder *DarijaHellaSwag*. Crucially, the full run finishes in just **48 GPU·h**, consuming ≈26 kWh—less than **2 %** of Atlas-Chat's reported footprint. These results show that *reasoning-centric, quality-over-quantity alignment unlocks Darija at*
Green-AI scale.

## Contributions

* **Lean Darija instruction suite.** We openly release translated versions of *LIMA 1 K*, *DEITA 6 K*, and a 44 k reasoning slice of *TULU 50 K*, each preserving 20 % English for cross-lingual robustness.

* **Minimal-data alignment recipe.** ≤6 k curated Darija instructions raise *DarijaMMLU* by +9.9 pp on a 4 B backbone with *no* English regression; adding reasoning prompts yields a further +4.9 pp and boosts maths and commonsense.

* **GemMaroc-27B.** The first open Darija LLM to reach 61.6 % *DarijaMMLU* and 60.5 % *DarijaHellaSwag*—using 1/10 the supervision and 1/50 the energy of prior work—while retaining strong bilingual reasoning (84.2 % GSM8K).

* **Open assets for the community.** We release all models, code, and datasets, to catalyse inclusive, low-carbon language technology across North Africa.

## 2 Related Work

The pursuit of better language models for low-resource dialects like Moroccan Arabic (Darija) spans several active research areas. This section reviews four key directions that inform our work. First, we examine prior efforts to develop Arabic and Darija-capable LLMs, highlighting their challenges and limitations. Second, we explore research on minimal data regimes for SFT, crucial in multilingual and dialectal contexts with limited data. Third, we analyze methods to enhance multilingual competence and dialect mastery by leveraging latent linguistic knowledge from pretraining. Finally, we review the growing emphasis on integrating multilingual Chain-of-Thought (CoT) and reasoning samples to improve both reasoning and language proficiency in low-resource settings.

### 2.1 Darija and Arabic LLMs

Arabic LLMs have advanced rapidly, driven by digital sovereignty and cultural goals. Yet, Moroccan Arabic (Darija) remains underrepresented. Most work targets Modern Standard Arabic (MSA) or pan-Arabic models, demanding extensive compute and ignoring dialectal diversity.

Saudi Arabia's **ALLaM** (Bari et al., 2025) introduced a scalable Arabic-English pipeline, expanding the tokenizer (32k→61k) and continuing pre-

training on 540B tokens (270B Arabic, 270B translated). It reached state-of-the-art scores on MMLU-ar, ACVA, araMath, and improved MMLU-en by +13 over LLaMA-2-Chat-7B. With 6M high-quality SFT samples, it showed that carefully curated data and a 45/55 Arabic-English ratio can stabilize multilingual performance. However, its reliance on trillion-token corpora and heavy compute limits replicability.

Qatar's **Fanar** (Team et al., 2025) followed a sovereign AI roadmap. **Fanar Star** (7B) was trained from scratch on 1T tokens (40% Arabic, 50% English, 10% code), while **Fanar Prime** (9B) focused on reasoning and STEM via continual pre-training. It also introduced RAG systems (Islamic, Biography RAG) and a morphology-aware tokenizer (MorphBPE). Despite innovations, Fanar's scale and compute demands hinder accessibility for lightweight, sustainable models.

**Atlas-Chat** (Shang et al., 2025) directly addressed the Darija gap, releasing the first open Darija LLMs (2B, 9B, 27B) fine-tuned with LoRA on Gemma-2 using 458k instructions from native, translated, and synthetic sources. The 9B model outperformed Jais-13B and AceGPT-13B by +13 points on DarijaMMLU, achieving state-of-the-art across Darija benchmarks. Yet, its strict monolingual tuning limits generalization.

Another recent contribution is **Al-Atlas** (Bounhar and Majjodi, 2025), which released Atlaset—a large, diverse Darija corpus from curated web, blogs, and forums—and trained both an XLM-RoBERTa-large MLM and a 0.5B Qwen2.5 causal LLM. Despite its small size, the causal model surpassed Atlas-Chat 2B by nearly 10% in human evaluation, underscoring the impact of dialect-specific pretraining.

**Command R+ / Command A** (Cohere et al., 2025) by Cohere proposes modular, efficient multilingual adaptation. The 111B model, combining six expert modules via hybrid transformers and SRPO, rivals GPT-4o on MMLU (85.5%), MATH (80%), and RepoQA (92.6%) while running on just two A100/H100 GPUs. Although not Arabic-centric, it scores 68.8% on NTREX and showcases effective multilingual performance without full retraining.
## 2.2 Improving Multilingual and Dialectal Capabilities in LLMs

Enhancing LLMs' multilingual and dialectal competence has spurred strategies such as synthetic data generation, pivot-based reasoning, and culturally faithful instruction design. This section reviews key works showing how thoughtful fine-tuning can unlock latent capabilities without costly pretraining.

**Parallel Instruction Tuning and Superficial Alignment.** Weber et al. (2024) found that *parallel* instruction tuning (English + target language) yields a **+9.9** accuracy gain over monolingual tuning, especially in low-resource settings. Results challenge the *Superficial Alignment Hypothesis*: smaller models (7B) require parallel data, whereas larger MoE models (50B) generalize better with less.

**Culturally Faithful and Contrastive Tuning.** Yakhni and Chehab (2025) showed that a small native Lebanese corpus outperformed larger auto-translated datasets. Using *contrastive tuning* (good vs. bad responses) improved xCOMET by **3.5 points**, confirming that authentic, culturally relevant data outperforms sheer volume.

**Pivot-Language Reasoning. PLUG** (Zhang et al., 2024) trained models to reason in a high-resource pivot (e.g., English) before answering in the target language. Just **2k** samples surpassed models trained on **96k** conventional samples, improving truthfulness and reasoning (TruthfulQA, SVAMP).

**Synthetic Instruction Generation. Bactrian-X** (Li et al., 2023) created **3.4M** synthetic instruction pairs across 52 languages. With LoRA tuning, **7B models** outperformed larger fully fine-tuned ones like BLOOMZ. Gains saturated beyond **70k pairs/language**, and mixing English improved both multilingual and English performance.

**Reverse Instruction Generation. MURI** (Köksal et al., 2024) reversed instruction-output generation using English LLMs on native texts, back-translating into 200+ languages. With ≤15k pairs/language, models gained **+14%** over mT0 on multilingual MMLU, with minimal human annotation.

**Insights from MT and Synthetic Pipelines.** Holmström and Doostmohammadi (2023) showed that even models with **0% Swedish** pretraining performed well after fine-tuning on translated data. Despite minor perplexity increases, downstream results held, validating synthetic translation with

light curation.

## 2.3 Minimal Data for SFT in Low-Resource Multilingual Settings

Zhou et al. (2023) introduced the *Superficial Alignment Hypothesis*, arguing that pretrained LLMs encode most knowledge and that SFT mainly adjusts style. With just 1k diverse, high-quality instruction-response pairs, LIMA aligned a 65B LLaMA model to rival GPT-4 in 43% of human preferences. Doubling data without increasing diversity gave no benefit. For Darija, a similarly small, style-aligned set (500–1,000 examples) may suffice if the base model has latent exposure.

Liu et al. (2024) proposed DEITA, which selects data based on complexity (EVOL COMPLEXITY), quality (EVOL QUALITY), and diversity (REPR FILTER). With only 6k pairs, DEITA-tuned LLaMA-13B and Mistral-7B models matched or exceeded systems trained on 30× more data. Chen et al. (2025) extended this with MIG, modeling instructions as semantic-label graphs and selecting the most informative 5%, outperforming full-data SFT by 4.6%.

Das and Khetan (2024) introduced DEFT-UCS, clustering embeddings to sample both easy (centroid-near) and hard (outlier) examples. It achieved full-data performance on CoEDIT using just 32.5% of the data and was preferred in 84% of human evaluations. Similarly, Han et al. (2025) proposed UniMax, combining influence and uncertainty for selection; 10% of data matched full-data performance, with strong cross-lingual results on TyDiQA.

Yu et al. (2024) showed that prompt-loss masking and sequence packing significantly improve SFT and DPO efficiency, achieving strong results with only 10k examples. Wu et al. (2024) introduced MOS, which dynamically samples based on transferability and reward signals. With just 10% of data, MOS trained models 2.2x faster while improving performance—highlighting benefits of treating Darija as a distinct "skill" early in training. In contrast, Xia et al. (2024) found that in large datasets, random sampling with length filters matched DEITA/MIG performance, suggesting diversity—not sophistication—drives gains. Finally, Li et al. (2024) showed that cross-lingual alignment with English prompts and native responses (3k pairs/language) yields minimal performance drop (5.7%) and outperforms ChatGPT in low-resource
settings—directly applicable to Darija.

Overall, these works support that compact, high-quality datasets (LIMA, X-Instruction), efficient selection (DEITA, MIG, DEFT, UniMax), and streamlined pipelines (LIONS, MOS) can unlock robust dialectal capabilities like Darija with minimal compute.

## 2.4 The Importance of Multilingual Chain-of-Thought and Reasoning Samples

Initial instruction tuning prioritized conversational fluency, but recent efforts emphasize reasoning-centric data—especially Chain-of-Thought (CoT) annotations—to boost both reasoning and language proficiency. This is vital for low-resource languages like Moroccan Darija, where curated data is scarce but multilingual pretraining may provide latent knowledge.

The **Breaking Language Barriers** study (Chen et al., 2024) introduced *MathOctopus*, trained on 73k translated, formula-validated CoT samples across ten languages. Fine-tuning Llama-2-7B on this data increased multilingual reasoning accuracy from 22.6% to 41.9%, and improved English performance by 8 points. Notably, cross-lingual QA (English questions, foreign-language answers) enhanced generalization—even for English.

Building on this, **XCOT** (Chai et al., 2024) extended GSM8K and SVAMP into ten languages and proposed *code-switched demonstrations* and *Random Online CoT* (translation before reasoning in English). These yielded a +25 point gain (to 47.7%) in MGSM without large native CoT datasets. Ablation studies confirmed code-switching and translation-based prompting as key drivers.

**mCoT** (Lai and Nissim, 2024) scaled this further by translating MetaMathQA and MathInstruct into ten languages, producing the 6.3M-example *mCoT-MATH* corpus. A 7B Mistral model fine-tuned on this data outperformed much larger models like GPT-3 (175B) and PaLM (540B) on MGSM and SVAMP. mCoT also introduced *Correct Consistency* (*CC*) and *Incorrect Consistency* (*IC*), with CC exceeding 50% across all pairs, confirming language-agnostic reasoning. Machine-translated prompts performed nearly on par with human ones, validating automated translation for scalable multilingual alignment.

These findings confirm that reasoning-centric

CoT datasets—whether human- or machine-translated—are essential for enhancing both fluency and reasoning. For Darija, integrating translated high-quality reasoning data into minimal SFT pipelines holds strong promise.

## 3 GemMaroc: Our Fine-Tuning Approach to Unlock Darija Proficiency in Gemma LLMs with Minimal Resources

Building upon recent advances in low-resource language alignment, we propose a highly efficient and ecologically sustainable fine-tuning methodology to unlock latent Moroccan Darija capabilities in Gemma LLMs. Rather than relying on large, compute-intensive datasets, our approach leverages the strategic use of minimal, high-quality instruction datasets, reasoning-focused prompts, and translation samples carefully selected to avoid catastrophic forgetting. This section details our SFT data preparation process, model configurations, evaluation protocols, and the experiments conducted to answer our core research questions.

### 3.1 SFT Data Preparation

In this subsection, we detail the process of preparing the Supervised Fine-Tuning (SFT) datasets used in our experiments. We describe the criteria followed for dataset selection, the filtering strategies applied to ensure data quality and relevance, and the translation procedures adopted to produce high-quality Darija instruction samples.

#### SFT Data Selection

We assemble three minimal-to-mid-scale instruction corpora—translated into Moroccan Darija—to surface latent capabilities with limited compute (Zhou et al., 2023; Liu et al., 2024; Chen et al., 2025). Crucially, **any sample whose system + dialogue serialization exceeds 2 048 tokens is dropped**, guaranteeing every training instance fits our GemMaroc's context window.

* **LIMA 1K** (Zhou et al., 2023): 1 000 single-turn tasks that emphasise stylistic quality and input diversity. Despite its size, LIMA often rivals much larger instruction sets, showing that small, well-curated data can unlock pretrained knowledge.

* **DEITA 6K** (Liu et al., 2024): 6 000 multi-turn prompts selected with EVOL-COMPLEXITY and
EVOL-QUALITY scores. It bolsters dialogue coherence and contextual nuance in Darija.

* **TULU 50K** (Chen et al., 2025): 50 000 information-gain-maximised instructions rich in mathematics, coding, and science—domains under-represented in earlier Darija work such as Shang et al. (2025).

Together these sets let us probe *extreme minimalism* (LIMA 1K), *mid-scale conversational alignment* (DEITA 6K), and *reasoning enrichment* (TULU 50K).

## Translating these datasets:

Prior to translation, we systematically filter out all multilingual data and retain only English instructions. This step is critical to avoid incorrect translations of meta-instructions or prompts explicitly mentioning target languages (e.g., "Translate from language A to B" or "Answer in language X"). Translating such content into Darija introduces inconsistencies and compromises data quality. To ensure precise filtering, we employ an XLM-Roberta-based language detector (ProtectAI, 2025) to accurately identify and keep only English instructions. To preserve the integrity of technical content, we retain code snippets, LaTeX formulas, and specialized terminologies in their original form. Technical terms are only translated into Arabic when widely accepted equivalents exist (e.g., "معادلة for equation, دالة for function"). This ensures that the model remains grounded in standard scientific concepts without introducing translation artifacts.

Translations are performed automatically using the Gemini 2.0 Flash API with a carefully engineered prompt to produce Darija in Arabic script. When uncertain, the model is instructed to prefer Modern Standard Arabic forms, ensuring clarity and naturalness. The full translation prompt is included in **Appendix A.1.1**.

In line with best practices to avoid catastrophic forgetting (Mao et al., 2022; Wu et al., 2025), we preserve approximately 20% of the data in its original English form across all datasets. This approach maintains cross-lingual transfer capabilities and general reasoning competence (Wu et al., 2024; Li et al., 2024; Zhang et al., 2024; Li et al., 2023).

## Dataset Statistics:

<table><caption>Table 1: The table shows the final dataset sizes after filtering and translation. “Darija Translated” indicates samples translated into Moroccan Darija using our automated pipeline. “Original English” refers to retained English samples to maintain cross-lingual capabilities and mitigate catastrophic forgetting.</caption><thead><tr><td>Dataset</td><td>Total Samples</td><td>Darija Translated</td><td>Original English</td></tr></thead><tbody><tr><td>LIMA 1K</td><td>1,000</td><td>700 (70%)</td><td>300 (30%)</td></tr><tr><td>DEITA 6K</td><td>5,000</td><td>3,700 (74%)</td><td>1,300 (26%)</td></tr><tr><td>TULU 50K</td><td>46,000</td><td>33,000 (72%)</td><td>13,000 (28%)</td></tr></tbody></table>

See **Appendix A.2** for concrete translation examples.

## Manual Verification:

Throughout the translation process, we regularly sampled and reviewed outputs to assess quality. Rather than applying manual corrections, we iteratively refined the translation prompts, leveraging our native fluency in Darija to adapt them whenever recurring issues were identified. Prompts requiring English responses were left untranslated to preserve their intended semantics. While the dataset is not perfect, its quality is sufficient for fine-tuning and aligned with the study's exploratory, low-resource goals.

## 3.2 Model and Fine-Tuning Details

We employed two versions of the Gemma LLM family—**Gemma 3-4B** and **Gemma 3-27B**—selected for their strong reasoning abilities and multilingual capabilities. Unlike Atlas-Chat or ALLaM models (Shang et al., 2025; Bari et al., 2025), Gemma models remain untuned on Arabic dialects, providing a clean baseline for evaluating emergent Darija competence without prior dialectal bias.

Fine-tuning was conducted using **LoRA** adapters (Hu et al., 2022), enabling low-resource, efficient training while mitigating catastrophic forgetting. This choice aligns with our *Green AI* objectives by minimizing GPU hours and energy consumption. For the integration of multi-turn conversations, we use the Vicuna-style template

### Hyperparameters:

* **Precision:** `bf16` throughout.

* **LoRA Config:** Rank $r = 32$, Alpha $= 64$ for our 4B models and Rank $r = 16$, Alpha $= 32$ for our GemMaroc-27B.

* **Learning Rates:** 1e-4 (TULU) and 4e-4 (DEITA, and LIMA).
* Epochs: 3 (TULU), 6 (DEITA), 15 (LIMA).

* Max Sequence Length: 2048 tokens.

Hyperparameters were inspired by the settings recommended in Zhou et al. (2023); Liu et al. (2024); Chen et al. (2025) for optimal convergence on these minimal datasets.

## 3.3 Evaluation Benchmarks

We measure progress across five competency axes:

### 1. Darija / Moroccan-Arabic competence

* DarijaMMLU – 22 027 multiple-choice questions spanning 57 academic subjects, re-filtered for Moroccan relevance (Shang et al., 2025).

* DarijaHellaSwag – the 10 k HellaSwag adversarial commonsense continuations rendered into Darija (Shang et al., 2025).

* DarijaBench (Shang et al., 2025) – covering Sentiment Analysis, Summarization, Translation, and Transliteration. We retained only the Sentiment Analysis and Summarization tasks, as the remaining benchmarks were of insufficient quality and could lead to unreliable evaluations.

### 2. General reasoning & world knowledge

* MMLU – The Massive Multitask Language Understanding (MMLU) benchmark comprises 15,908 multiple-choice questions spanning 57 diverse subjects, including mathematics, law, medicine, and history. It evaluates a model's general knowledge and problem-solving abilities across these domains. The final MMLU score is calculated as the average accuracy across all subjects, providing a comprehensive measure of a model's capabilities (Hendrycks et al., 2021).

* HellaSwag (EN) – 10 000 adversarial commonsense continuations that correlate strongly with everyday reasoning (Zellers et al., 2019).

### 3. Mathematical reasoning

* GSM8K – 1 000 grade-school word problems requiring 2–8 arithmetic steps (Cobbe and et al., 2021).

### 4. Truthfulness & safety

* TruthfulQA – 817 short-answer questions across 38 domains designed to expose models' susceptibility to common misconceptions (Lin et al., 2022).

These benchmarks ensure that we accurately assess the model's Darija performance while verifying that reasoning and multilingual capabilities remain intact.

## 4 Experiments, Results, and Discussion

This section reports the empirical findings of the experimental plan outlined in Section 1. We first describe the evaluation protocol (§4.1), then present the main results in a single consolidated leaderboard (§4.2). Finally, we analyse the outcomes with respect to the three research questions and highlight broader implications (§4.3).

### 4.1 Experimental Set-up

**Hardware and cost.** All fine-tuning experiments were conducted on 2xA100-80GB GPUs, except for our largest model, the 27-billion-parameter GEMMAROC, which was fine-tuned on 8xH100-80GB GPUs. The longest run—the GEMMAROC fine-tuning—completed in just 6 hours, wall-clock, consuming approximately 48 GPU-hours and keeping the total cloud cost well under $100.<fnref n="1" />

**Metrics.** All benchmarks were evaluated in a strict zero-shot setting. For classification tasks (DarijaMMLU, DarijaHellaSwag, Sentiment Analysis, MMLU, HellaSwag), we report accuracy. For GSM8K @5, we measure whether the correct answer appears in the model's top-5 outputs. TruthfulQA is evaluated with BLEU accuracy (bleu_acc). Summarization is assessed using chrF, ROUGE-1, ROUGE-L, and BERTScore, covering both surface overlap and semantic similarity.

**Baselines.** We compare against (i) the untuned Gemma checkpoints (4 B & 27 B), (ii) Atlas-Chat (9 B & 27 B) (Shang et al., 2025), and (iii) AL-LaM (7 B & 13 B) (Bari et al., 2025). The latter two represent the previous state of the art for Darija and for large-scale Arabic instruction tuning, respectively.

<footnote n="1">¹Based on the public on-demand rate of $2 per A100-hour in the selected Runpod Cloud region.</footnote>
## 4.2 Main Results

Table 2 presents our unified leaderboard, merging two complementary evaluation tracks: a Darija-centric benchmark suite (DarijaMMLU, DarijaHellaSwag, Sentiment Analysis, Summarization), and an English-centric suite spanning general reasoning, mathematics, code, and truthfulness (MMMLU, TruthfulQA, HellaSwag, GSM8K). This table compares our models—marked as (ours)—with several competitive baselines across both tracks.

## 4.3 Discussion

### Emergent Darija competence with minimal data

Our three “tiny-but-mighty” instruction sets uncover a clear progression in emergent Darija skills:

* **DEITA-6K** (~700 KB text) already lifts the 4 B backbone by +9.9 pp on *DarijaMMLU* and +8.0 pp on *DarijaHellaSwag*, while fully preserving English-world knowledge (MMLU 51.3 % vs 51.1 %). This confirms that a few thousand, well-curated conversational Darija examples are enough for solid coverage.

* **TULU-50K** (~5 MB) pushes the same model to 47.5 % on *DarijaMMLU* and 47.1 % on *DarijaHellaSwag*, surpassing all Arabic-centric models of similar size (e.g. ALLaM-7B) despite using <10 % of their data footprint. The larger gains come without sacrificing cross-lingual performance and even *improve* maths (+2.8 pp GSM8K).

* **LIMA-1K** shows the lower bound of the spectrum: with only 1 000 single-turn prompts, Darija knowledge surfaces but remains fragile (+2.1 pp DarijaMMLU, -16.8 pp sentiment). This extreme minimal regime illustrates that, although some Darija was latent in pre-training, the dialect is too under-represented for 1 K instructions to achieve reliable competence.

**Take-away.** Darija proficiency *emerges* as soon as the model sees a few thousand high-quality, dialect-specific instructions. However, the 1 K “LIMA” budget is below the tipping point: it validates the Superficial Alignment hypothesis in principle, but confirms that Darija’s sparse presence in pre-training demands at least a DEITA-scale corpus for robust, practical performance.

### Advantage of reasoning-focused instructions

Moving from 6 K conversational prompts (DEITA) to 50 K reasoning-dense prompts (TULU) brings a consistent uplift without harming cross-lingual performance. On the same 4 B backbone, GEMMAROC-TULU adds a further +4.9 pp on *DarijaMMLU*, +2.9 pp on *DarijaHellaSwag*, and improves English reasoning as well (+2.8 pp GSM8K, +5.0 pp HellaSwag) while keeping the MMLU drop below 3 pp. The only noticeable trade-off is a small decline in sentiment accuracy, suggesting that the few sentiment items present in Tulu are not yet enough to balance the domain shift. These results support our second hypothesis: **reasoning-centric instructions are the most data-efficient lever for unlocking both Darija fluency and multilingual reasoning coherence**. In practice, adding mathematical & coding problems proved worth every extra sample.

### Our GemMaroc-27B

Applying the same 50 K reasoning-oriented recipe to GEMMA3-27B yields **GemMaroc-27B**, which advances the Pareto frontier on three fronts: Darija coverage, cross-lingual reasoning, and footprint.

**Darija strengths.** GemMaroc reaches **61.6 %** on *DarijaMMLU*, effectively tying Atlas-Chat despite using ≈1/10 the supervision budget, and it surpasses all models on the harder commonsense test with **60.5 %** on *DarijaHellaSwag* (+12.1 pp over Atlas). ROUGE-L in summarisation climbs to **11.2**, a new best for Darija. The only noteworthy regression is sentiment (-13.8 pp vs Atlas), highlighting a domain that still lacks targeted data.

**Reasoning retention.** Crucially, the extra Darija skill does *not* come at the price of English reasoning. GemMaroc holds **84.2 %** on GSM8K—well above all Arabic-centric baselines and within 12 pp of the untuned backbone despite inserting low-rank adapters, confirming that mathematical competence remains largely intact. It also improves commonsense *HellaSwag* to **79.4 %**, edging past Atlas (+1.5 pp) while retaining solid truthfulness (55.5 %).

**Cost Efficiency and Carbon Footprint** Fine-tuning the 27-billion-parameter GEMMAROC completed in just **48 GPU·h** (8 × H100-80 GB, 6 h wall-clock). Using the commonly adopted 0.54 kW/GPU data-centre average power draw (Lacoste
<table><caption>Table 2: Unified leaderboard comprising: (1) Darija-centric benchmarks, and (2) English-centric benchmarks across reasoning, mathematics, and truthfulness.</caption><thead><tr><th rowspan="3">Language<br>Model/Benchmark</th><th colspan="5">Darija</th><th colspan="5">English</th></tr><tr><th colspan="3">Size (B)</th><th colspan="2">Sentiment Analysis</th><th colspan="4">Summarization</th><th colspan="2">GSM8K</th></tr><tr><th>Metrics</th><th>DarijaMMLU</th><th>DarijaHellaSwag</th><th>Acc</th><th>chrF</th><th>ROUGE-1</th><th>ROUGE-L</th><th>BERTScore</th><th>MMLU</th><th>TruthfulQA</th><th>HellaSwag</th><th>@5</th></tr></thead><tbody><tr><td>Gemma3-4b</td><td>4</td><td>32.8</td><td>36.3</td><td>58.94</td><td>27.22</td><td>8.38</td><td>8.19</td><td>37.23</td><td>51.1</td><td>40.88</td><td>47.65</td><td>74.75</td></tr><tr><td>GemMaroc-4b-LIMA (ours)</td><td>4</td><td>34.95</td><td>39.27</td><td>42.1</td><td>26.14</td><td>6.95</td><td>7.04</td><td>34.32</td><td>29.28</td><td>40.15</td><td>44.21</td><td>51.24</td></tr><tr><td>GemMaroc-4b-Deita (ours)</td><td>4</td><td>42.67</td><td>44.26</td><td>60.8</td><td>27.16</td><td>7.4</td><td>7.34</td><td>38.48</td><td>51.35</td><td>44.55</td><td>68.97</td><td>53.15</td></tr><tr><td>GemMaroc-4b-Tulu (ours)</td><td>4</td><td>47.53</td><td>47.13</td><td>53.29</td><td>28.46</td><td>8.89</td><td>8.76</td><td>39.27</td><td>54.14</td><td>43.33</td><td>73.95</td><td>55.95</td></tr><tr><td>ALLaM-Instruct-7b</td><td>7</td><td>59.49</td><td>50.09</td><td>47.33</td><td>10.27</td><td>1.68</td><td>1.68</td><td>12.28</td><td>58.31</td><td>42.11</td><td>75.2</td><td>49.28</td></tr><tr><td>Atlas-Chat-9B</td><td>9</td><td>58.32</td><td>43.65</td><td>81.85</td><td>32.07</td><td>9.5</td><td>9.45</td><td>47</td><td>69.09</td><td><b>67.56</b></td><td>73.35</td><td>73.01</td></tr><tr><td>Atlas-Chat-27B</td><td>27</td><td><b>61.95</b></td><td>48.37</td><td><b>73</b></td><td><b>32.75</b></td><td><b>10.53</b></td><td>10.42</td><td><b>47.82</b></td><td>72.06</td><td>43.82</td><td>77.84</td><td>82.03</td></tr><tr><td>GemMaroc-27b-Tulu (ours)</td><td>27</td><td><b>61.61</b></td><td><b>60.5</b></td><td>59.25</td><td>28.34</td><td>9</td><td><b>11.2</b></td><td><b>39.5</b></td><td><b>73.6</b></td><td>55.45</td><td><b>79.35</b></td><td><b>84.23</b></td></tr></tbody></table>

et al., 2019), this equates to ≈ **26 kWh** of energy and ≈ **10 kg CO₂e** at the 0.38 kg CO₂e/kWh global carbon-intensity average. A comparable full-parameter Atlas-Chat-27B run would consume ~ 1.4 MWh and ~ 610 kg CO₂e—over **48×** more energy and > 98% higher emissions. Thus, GemMaroc delivers state-of-the-art Darija performance and strong bilingual reasoning at under **2%** of the carbon cost of prior work, demonstrating that high-quality, reasoning-dense instructions—not dataset volume—drive sustainable dalect mastery. As shown in **Appendix 8** and **Appendix 9**, training and evaluation losses steadily decrease, with the best validation loss reached at step 900—corresponding to approximately two epochs—confirming that the model achieves optimal performance early in training with minimal overfitting. Full formulas, intermediate values, and the detailed energy-breakdown table are provided verbatim in **Appendix A.3** for complete transparency.

## Conclusion

This work set out to determine whether Morocan Darija competence can be surfaced in modern LLMs with only a handful of carefully chosen instructions, and whether adding reasoning-dense prompts is the most efficient lever for doing so. Our experiments answer both questions affirmatively.

First, we showed that a few thousand high-quality, dialect-specific instructions (DEITA 6 K) already lift a 4 B Gemma model by almost 10 pp on DarijaMMLU while leaving its English-world knowledge intact, confirming that Darija knowledge acquired during pre-training merely needed the right “key” to emerge. One thousand instructions (LIMA 1 K) were not enough, highlighting the sparse representation of the dialect in web corpora, but the tipping point was still two orders of magnitude smaller than in prior Arabic-centric

projects.

Second, enriching the minimal corpus with 50 K reasoning-oriented prompts (TULU) produced consistent gains across dialectal and cross-lingual tasks. The resulting **GemMaroc-27B** matches the best published Darija score on DarijaMMLU (61.6 %) and sets a new high-water mark on the harder DarijaHellaSwag (+12.1 pp over Atlas-Chat) while retaining strong English mathematics (84.2 % GSM8K). Crucially, the entire run consumed only 48 GPU·h—roughly 2 % of the energy and carbon budget reported for comparable baselines—demonstrating that high-impact language inclusion need not come with high ecological cost.

Beyond benchmarking, GemMaroc’s balanced bilingual reasoning, open licensing, and lean LoRA adapters make it immediately deployable for inclusive civic services, education, and research throughout the Maghreb. Remaining gaps in sentiment, summarisation, and tokenisation point to clear next steps: larger-scale human evaluation, script-aware tokenisers, richer Darija domain data, and full alignment pipelines spanning RLAIF and dialogue safety. We will also explore speech, retrieval-augmented generation, and other North-African dialects to extend the benefits of low-carbon language technology across the region.

In sum, GemMaroc shows that with precise data curation and a focus on reasoning, sovereign AI for low-resource dialects can be both state-of-the-art and sustainable—a blueprint for scaling equitable language technologies worldwide.

## Limitations

GemMaroc excels in reasoning—matching or surpassing Atlas—and achieves strong, competitive results across most tasks. Some performance gaps remain in summarization, sentiment analysis and reading comprehension, where richer Darija-specific data would be beneficial. The model was trained solely with parameter-efficient SFT, relies
on machine-translated Darija samples with limited
verification, and retains the original SentencePiece
tokenizer, which is not optimized for Darija's script
and linguistic nuances. Evaluations focused on
academic QA, commonsense reasoning and math-
ematics, without exploring long-context tasks or
open-ended dialogue. Large-scale human evalua-
tions in Darija are also pending, and experiments
were capped at 27 B parameters. Future work
will explore full alignment pipelines, script- and
language-aware tokenization, broader evaluations,
and comprehensive human studies.

References

International Energy Agency. 2024. Electricity 2025
– Emissions Annex. Average global grid intensity
2024: 0.40 kg CO2/kWh.

M. Saiful Bari, Yazeed Alnumay, Norah A. Alzahrani, Nouf M. Alotaibi, Hisham A. Alyahya, Sultan AlRashed, Faisal A. Mirza, Shaykhah Z. Alsubaie, Hassan A. Alahmed, Ghadah Alabduljabbar, Raghad Alkhathran, Yousef Almushayqih, Raneem Alnajim, Salman Alsubaihi, Maryam Al Mansour, Saad Amin Hassan, Majed Alrubaian, Ali Alammari, Zaki Alawami, Abdulmohsen Al-Thubaity, and et al. 2025. Allam: Large language models for arabic and english. In International Conference on Learning Representations.

Abdelaziz Bounhar and Abdeljalil El Majjodi. 2025. Al-atlas: A causal language model for moroccan darija. https://huggingface.co/atlasia/Al-Atlas-0.5B/.

Linzheng Chai, Jian Yang, Tao Sun, Hongcheng Guo, Jiaheng Liu, Bing Wang, Xiannian Liang, Jiaqi Bai, Tongliang Li, Qiyao Peng, and Zhoujun Li. 2024. xcot: Cross-lingual instruction tuning for cross-lingual chain-of-thought reasoning.

Nuo Chen, Zinan Zheng, Ning Wu, Ming Gong, Dongmei Zhang, and Jia Li. 2024. Breaking language barriers in multilingual mathematical reasoning: Insights and observations. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 7001–7016, Miami, Florida, USA. Association for Computational Linguistics.

Yicheng Chen, Yining Li, Kai Hu, Zerun Ma, Haochen Ye, and Kai Chen. 2025. Mig: Automatic data selection for instruction tuning by maximizing information gain in semantic space. arXiv preprint arXiv:2504.13835.

Karl Cobbe and et al. 2021. Training verifiers to solve math word problems. In arXiv preprint arXiv:2110.14168.

Team Cohere, ;, Aakanksha, Arash Ahmadian, Marwan Ahmed, Jay Alammar, Milad Alizadeh, Yazeed Alnumay, Sophia Althammer, Arkady Arkhangorodsky, Viraat Aryabumi, Dennis Aumiller, Raphaël Avalos,

Zahara Aviv, Sammie Bae, Saurabh Baji, Alexandre Barbet, Max Bartolo, Björn Bebensee, Neeral Beladia, Walter Beller-Morales, Alexandre Bérard, Andrew Berneshawi, Anna Bialas, Phil Blunsom, Matt Bobkin, Adi Bongale, Sam Braun, Maxime Brunet, Samuel Cahyawijaya, David Cairuz, Jon Ander Campos, Cassie Cao, Kris Cao, Roman Castagné, Julián Cendrero, Leila Chan Currie, Yash Chandak, Diane Chang, Giannis Chatziveroglou, Hongyu Chen, Claire Cheng, Alexis Chevalier, Justin T. Chiu, Eugene Cho, Eugene Choi, Eujeong Choi, Tim Chung, Volkan Cirik, Ana Cismaru, Pierre Clavier, Henry Conklin, Lucas Crawhall-Stein, Devon Crouse, Andres Felipe Cruz-Salinas, Ben Cyrus, Daniel D'souza, Hugo Dalla-Torre, John Dang, William Darling, Omar Darwiche Domingues, Saurabh Dash, Antoine Debugne, Théo Dehaze, Shaan Desai, Joan Devassy, Rishit Dholakia, Kyle Duffy, Ali Edalati, Ace Eldeib, Abdullah Elkady, Sarah Elsharkawy, Irem Ergün, Beyza Ermis, Marzieh Fadaee, Boyu Fan, Lucas Fayoux, Yannis Flet-Berliac, Nick Frosst, Matthias Gallé, Wojciech Galuba, Utsav Garg, Matthieu Geist, Mohammad Gheshlaghi Azar, Ellen Gilsenan-McMahon, Seraphina Goldfarb-Tarrant, Tomas Goldsack, Aidan Gomez, Victor Machado Gonzaga, Nithya Govindarajan, Manoj Govindassamy, Nathan Grinsztajn, Nikolas Gritsch, Patrick Gu, Shangmin Guo, Kilian Haefeli, Rod Hajjar, Tim Hawes, Jingyi He, Sebastian Hofstätter, Sungjin Hong, Sara Hooker, Tom Hosking, Stephanie Howe, Eric Hu, Renjie Huang, Hemant Jain, Ritika Jain, Nick Jakobi, Madeline Jenkins, JJ Jordan, Dhruti Joshi, Jason Jung, Trushant Kalyanpur, Siddhartha Rao Kamalakara, Julia Kedrzycki, Gokce Keskin, Edward Kim, Joon Kim, Wei-Yin Ko, Tom Kocmi, Michael Kozakov, Wojciech Kryściński, Arnav Kumar Jain, Kopal Kumar Teru, Sander Land, Michael Lasby, Olivia Lasche, Justin Lee, Patrick Lewis, Jeffrey Li, Jonathan Li, Hangyu Lin, Acyr Locatelli, Kevin Long, Raymond Ma, Lukáš Mach, Marina Machado, Joanne Magbitang, Brenda Malacara Lopez, Aryan Mann, Kelly Marchisio, Olivia Markham, Alexandre Matton, Alex McKinney, Dominic McLoughlin, Jozef Mokry, Adrien Morisot, Autumn Moulder, Harry Moynehan, Maximilian Mozes, Vivek Muppalla, Lidiya Murakhovska, Hemangani Nagarajan, Alekhya Nandula, Hisham Nasir, Shauna Nehra, Josh Netto-Rosen, Daniel Ohashi, James Owers-Bardsley, Jason Ozuzu, Dennis Padilla, Gloria Park, Sam Pasaglia, Jeremy Pekmez, Laura Penstone, Aleksandra Piktus, Case Ploeg, Andrew Poulton, Youran Qi, Shubha Raghvendra, Miguel Ramos, Ekagra Ranjan, Pierre Richemond, Cécile Robert-Michon, Aurélien Rodriguez, Sudip Roy, Sebastian Ruder, Laura Ruis, Louise Rust, Anubhav Sachan, Alejandro Salamanca, Kailash Karthik Saravanakumar, Isha Satyakam, Alice Schoenauer Sebag, Priyanka Sen, Sholeh Sepehri, Preethi Seshadri, Ye Shen, Tom Sherborne, Sylvie Shang Shi, Sanal Shivaprasad, Vladyslav Shmyhlo, Anirudh Shrinivason, Inna Shteinbuk, Amir Shukayev, Mathieu Simard, Ella Snyder, Ava Spataru, Victoria Spooner, Trisha Starostina, Florian Strub, Yixuan Su, Jimin Sun, Dwarak Talupuru,
Eugene Tarassov, Elena Tommasone, Jennifer Tracey, Billy Trend, Evren Tumer, Ahmet Üstün, Bharat Venkitesh, David Venuto, Pat Verga, Maxime Voisin, Alex Wang, Donglu Wang, Shijian Wang, Edmond Wen, Naomi White, Jesse Willman, Marysia Winkels, Chen Xia, Jessica Xie, Minjie Xu, Bowen Yang, Tan Yi-Chern, Ivan Zhang, Zhenyu Zhao, and Zhoujie Zhao. 2025. Command a: An enterprise-ready large language model.

NVIDIA Corporation. 2024. NVIDIA A100 Tensor Core GPU Datasheet. URL: https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/a100/pdf/nvidia-a100-datasheet-us-nvidia-1758950-r4-web.pdf.

Devleena Das and Vivek Khetan. 2024. DEFT-UCS: Data efficient fine-tuning for pre-trained language models via unsupervised core-set selection for text-editing. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 20296–20312, Miami, Florida, USA. Association for Computational Linguistics.

Databasemart. 2025. Optimizing vllm performance on A100 80 gb: Benchmark insights. https://www.databasemart.com/blog/vllm-gpu-benchmark-a100-80gb. Accessed May 19, 2025.

Ember and Energy Institute. 2024. Carbon intensity of electricity generation. https://ourworldindata.org/grapher/carbon-intensity-electricity. Accessed May 19, 2025.

Jindong Han, Hao Liu, Jun Fang, Naiqiang Tan, and Hui Xiong. 2025. Automatic Instruction Data Selection for Large Language Models via Uncertainty-Aware Influence Maximization. In *Proceedings of the ACM Web Conference 2025*, pages 4969–4979, Sydney, Australia. ACM.

Peter Henderson, Jieru Hu, Joshua Romoff, Emma Brunskill, Dan Jurafsky, and Joelle Pineau. 2020. Towards the systematic reporting of the energy and carbon footprints of machine learning. CoRR, abs/2002.05651. Revised version published 2022.

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. Measuring massive multitask language understanding. In *Proceedings of the International Conference on Learning Representations (ICLR)*.

Oskar Holmström and Ehsan Doostmohammadi. 2023. Making instruction finetuning accessible to non-English languages: A case study on Swedish models. In *Proceedings of the 24th Nordic Conference on Computational Linguistics (NoDaLiDa)*, pages 634–642, Tórshavn, Faroe Islands. University of Tartu Library.

Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. LoRA: Low-rank adaptation of large language models. In *International Conference on Learning Representations*.

Abdullatif Köksal, Marion Thaler, Ayyoob Imani, Ahmet Üstün, Anna Korhonen, and Hinrich Schütze. 2024. Muri: High-quality instruction tuning datasets for low-resource languages via reverse instructions.

Alexandre Lacoste, Alexandra Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. *arXiv preprint arXiv:1910.09700*.

Huiyuan Lai and Malvina Nissim. 2024. mCoT: Multilingual instruction tuning for reasoning consistency in language models. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 12012–12026, Bangkok, Thailand. Association for Computational Linguistics.

Chong Li, Wen Yang, Jiajun Zhang, Jinliang Lu, Shaonan Wang, and Chengqing Zong. 2024. X-instruction: Aligning language model in low-resource languages with self-curated cross-lingual instructions. In *Findings of the Association for Computational Linguistics: ACL 2024*, pages 546–566, Bangkok, Thailand. Association for Computational Linguistics.

Haonan Li, Fajri Koto, Minghao Wu, Alham Fikri Aji, and Timothy Baldwin. 2023. Bactrian-x: Multilingual replicable instruction-following models with low-rank adaptation.

Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. Truthfulqa: Measuring how models mimic human falsehoods. In *Proceedings of ACL*.

Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, and Junxian He. 2024. What makes good data for alignment? a comprehensive study of automatic data selection in instruction tuning. In *The Twelfth International Conference on Learning Representations*.

Yuren Mao, Yaobo Liang, Nan Duan, Haobo Wang, Kai Wang, Lu Chen, and Yunjun Gao. 2022. Less-forgetting multi-lingual fine-tuning. In *Advances in Neural Information Processing Systems 35: NeurIPS 2022*, New Orleans, USA.

Ministry of Digital Transition and Administration Reform. 2024. Digital Morocco 2030: National Strategy. https://www.mmsp.gov.ma/sites/default/files/2024-09/PlaquetteInstitutionnel_18092024_Ang.pdf. Accessed: 2025-05-13.

NVIDIA. 2024. NVIDIA A100 Tensor Core GPU datasheet. https://www.nvidia.com/content/dam/.../a100-80gb-datasheet.pdf. Accessed May 19, 2025.

ProtectAI. 2025. protectai/xlm-roberta-base-language-detection-onnx. https://huggingface.co/protectai/xlm-roberta-base-language-detection-onnx. Accessed May 25. ONNX version of papluca/xlm-roberta-base-language-detection using the Optimum library. Fine-tuned on the Language Identification dataset.

Guokan Shang, Hadi Abdine, Yousef Khoubrane, Amr Mohamed, Yassine Abbahaddou, Sofiane Ennadir,
Imane Momayiz, Xuguang Ren, Eric Moulines, Preslav Nakov, Michalis Vazirgiannis, and Eric Xing. 2025. Atlas-chat: Adapting large language models for low-resource moroccan arabic dialect. In *Proceedings of the First Workshop on Language Models for Low-Resource Languages*, pages 9–30, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. Energy and policy considerations for deep learning in NLP. In *Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics*, pages 3645–3650, Florence, Italy. Association for Computational Linguistics.

Fanar Team, Ummar Abbas, Mohammad Shahmeer Ahmad, Firoj Alam, Enes Altinisik, Ehsannedin Asgari, Yazan Boshmaf, Sabri Boughorbel, Sanjay Chawla, Shammur Chowdhury, Fahim Dalvi, Kareem Darwish, Nadir Durrani, Mohamed Elfeky, Ahmed Elmagarmid, Mohamed Eltabakh, Masoomali Fatehkia, Anastasios Fragkopoulos, Maram Hasanain, Majd Hawasly, Mus’ab Husaini, Soon-Gyo Jung, Ji Kim Lucas, Walid Magdy, Safa Messaoud, Abubakr Mohamed, Tasnim Mohiuddin, Basel Mousi, Hamdy Mubarak, Ahmad Musleh, Zan Naeem, Mourad Ouzzani, Dorde Popovic, Amin Sadeghi, Husrev Taha Sencar, Mohammed Shinoy, Omar Sinan, Yifan Zhang, Ahmed Ali, Yassine El Kheir, Xiaosong Ma, and Chaoyi Ruan. 2025. Fanar: An arabic-centric multimodal generative ai platform.

Alexander Arno Weber, Klaudia Thellmann, Jan Ebert, Nicolas Flores-Herr, Jens Lehmann, Michael Fromm, and Mehdi Ali. 2024. Investigating multilingual instruction-tuning: Do polyglot models demand for multilingual instructions? In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 20829–20855, Miami, Florida, USA. Association for Computational Linguistics.

Junhong Wu, Yang Zhao, Yangyifan Xu, Bing Liu, and Chengqing Zong. 2025. Boosting LLM translation skills without general ability loss via rationale distillation. In *Submitted to ACL Rolling Review - February 2025*. Under review.

Minghao Wu, Thuy-Trang Vu, Lizhen Qu, and Gholamreza Haffari. 2024. MIXTURE-OF-SKILLS: Learning to optimize data usage for fine-tuning large language models. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 14226–14240. Association for Computational Linguistics.

Tingyu Xia, Bowen Yu, Kai Dang, An Yang, Yuan Wu, Yuan Tian, Yi Chang, and Junyang Lin. 2024. Rethinking Data Selection at Scale: Random Selection is Almost All You Need. *arXiv preprint arXiv:2410.09335*.

Silvana Yakhni and Ali Chehab. 2025. Fine-tuning llms for low-resource dialect translation: The case of lebanese.

Xiao Yu, Qingyang Wu, Yu Li, and Zhou Yu. 2024. LIONs: An empirically optimized approach to align

language models. In *Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing*, pages 8732–8753, Miami, Florida, USA. Association for Computational Linguistics.

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can a machine really finish your sentence? In *Proceedings of ACL*.

Zhihan Zhang, Dong-Ho Lee, Yuwei Fang, Wenhao Yu, Mengzhao Jia, Meng Jiang, and Francesco Barbieri. 2024. PLUG: Leveraging pivot language in cross-lingual instruction tuning. In *Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)*, pages 7025–7046, Bangkok, Thailand. Association for Computational Linguistics.

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. Lima: Less is more for alignment. In *Advances in Neural Information Processing Systems (NeurIPS-2023)*, volume 36, pages 55006–55021.

# A Appendix

## A.1 SFT Dataset Preparation

### A.1.1 Translation Prompt Template

To ensure consistency and high quality in automatic translation from English to Moroccan Darija, we employed a strict prompt with clear translation and preservation guidelines. The full prompt, as used for dataset generation, is presented below.

**TRANSLATION_PROMPT =**

Translate the following text into Moroccan Darija, written in Arabic script, following these strict instructions:

DO NOT TRANSLATE the following elements (keep them exactly as they are):

1. **Code blocks:**
   Keep all programming code (Python, JS, etc.) unchanged.

Translate comments inside code blocks to Darija (Arabic script).

2. **Math formulas and LaTeX:**
   Preserve all mathematical notation and LaTeX equations. Do not alter them in any way.

3. **Technical Terms:**
   Keep technical/scientific terms in English unless they have a known Arabic equivalent.
   For example:
* Use "صيغة" for "formula"

* Use "معادلة" for "equation"

* Use "دالة" for "function"

* But do not translate terms like "API", "eigenvalue", "html", etc.

### 4. Proper Names:

Do not translate names of people, tools, or software (e.g., Python, Newton, GitHub, etc.).

**TRANSLATE into Moroccan Darija (Arabic script):**

* Regular sentences, explanations, and natural narrative text.

* Comments inside code blocks (e.g., # comment here).

* Use a natural mix of Moroccan Darija and Modern Standard Arabic where appropriate:

- Use Darija for informal or conversational parts.

- Use Standard Arabic for formal or widely recognized scientific terms. Do your best to find most suitable translations.

- If translating certain phrases would sound awkward, unclear, or artificial in Darija, then translate them in Standard Arabic.

**GOAL:**

Make the content clear and naturally understandable to Moroccan Arabic speakers while respecting the technical integrity of the original text. Return the translated text without saying anything else.

**Examples:**

**Example 1 (Code + Explanation)**

**Input:**

The 'Function' calculate_mean(data) is used to compute the average of a list of numbers.

// This function iterates through the list
// and calculates the sum. Python

def calculate_mean(data_list):
    # Calculate sum of elements
    total_sum = sum(data_list)
    # Calculate number of elements
    count = len(data_list)
    # Return the mean
    return total_sum / count

Its 'Complexity' is O(n).

**Output:**

ال 'Function' calculate_mean(data)
كستعمل باش نحسبو المعدل دیال شي
ليسته دیال الارقام.

// هاد الدالة كدور على العناصر
Python
وكتحسب المجموع.

def calculate_mean(data_list):
    # Calculate sum of elements
    total_sum = sum(data_list)
    # Calculate number of elements
    count = len(data_list)
    # Return the mean
    return total_sum / count

ال'Complexity' هي O(n). ديالها

**Example 2 (Math Formula)**

**Input:**

Consider the quadratic equation $ax^2 + bx + c = 0$. The solutions can be found using the formula:

$$x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}$$
This is known as the quadratic formula.

**Output:**

ناخدو المعادلة من الدرجة الثانية
الحلول ديالها كتلقاو. $ax^2 + bx + c = 0$
بهاد الصيغة:

$$x = \frac{-b \pm \sqrt{b^2 - 4ac}}{2a}$$
هادي معروفة بصيغة المعادلة من الدرجة الثانية.
### Example 3 (Fully Regular Text)

**Input:**

This chapter introduces the basic concepts. Please read it carefully.

**Output:**

هاد الجزء كيعرف بالمفاهم الأساسية.
عافاك قراه مزيان.

Now, translate the following text:

#### A.1.2 Comprehensive Description of Source Datasets

**Comprehensive Description of LIMA-1k Dataset:**

The LIMA-1k dataset, introduced in (Zhou et al., 2023), comprises 1,000 meticulously curated examples specifically designed to efficiently align large language models (LLMs) with minimal computational resources. This dataset originates from an initiative exploring the hypothesis that effective alignment and instruction-tuning do not necessarily require extensive datasets or reinforcement learning from human feedback (RLHF). Instead, the primary goal of LIMA-1k is to demonstrate that alignment can predominantly be achieved by leveraging the knowledge already embedded within pretrained models, with fine-tuning merely instructing the model in stylistic and structural nuances for interacting effectively with users.

LIMA-1k is curated from several high-quality online sources, notably Stack Exchange, wikiHow, and selected Reddit communities (r/WritingPrompts), augmented with manually authored examples to enhance stylistic uniformity and thematic diversity. Table 3 summarizes the dataset's key statistics.

<table><caption>Table 3: Summary statistics of the LIMA-1k dataset (Zhou et al., 2023).</caption><thead><tr><th>Data Source</th><th>Examples</th><th>Avg. Input Length</th><th>Avg. Output Length</th></tr></thead><tbody><tr><td>Stack Exchange (STEM)</td><td>200</td><td>117</td><td>523</td></tr><tr><td>Stack Exchange (Other)</td><td>200</td><td>119</td><td>530</td></tr><tr><td>wikiHow</td><td>200</td><td>12</td><td>1,811</td></tr><tr><td>Pushshift Reddit r/WritingPrompts</td><td>150</td><td>34</td><td>274</td></tr><tr><td>Natural Instructions</td><td>50</td><td>236</td><td>92</td></tr><tr><td>Manually Authored (Group A)</td><td>200</td><td>40</td><td>334</td></tr><tr><td>Total/Average</td><td>1,000</td><td>93</td><td>727</td></tr></tbody></table>

Below are representative multi-turn dialogue examples illustrating the conversational coherence and stylistic uniformity achieved by LIMA-1k.

**Example 1:**

**Human:** Tell me an interesting fact about geography.

**Assistant:** Here are some interesting facts about geography:

* **Bhutan** is one of the most intriguing places in the world. The Buddhist kingdom is the only carbon-negative country on Earth, with strict environmental laws and cultural norms that preserve its stunning natural landscape. For similar reasons, Bhutan was one of the last countries to introduce television in 1999, and has no traffic lights to this day.

* There are currently few territories on Earth that are unclaimed by any sovereign state (terra nullius). One example is **Bir Tawil**, along the border between Egypt and Sudan, which was created by a discrepancy between borders drawn in 1899 and 1902. The territory remains unclaimed by either country because it would entail forfeiting the much larger Halaib Triangle, which is claimed by both countries. Bir Tawil has no settled population, but the land is used by Bedouins who roam the area.

* The **antipode** of any spot on Earth is the point directly opposite to it on the globe. Interestingly, most land areas' antipodes fall in the ocean. For instance, the antipode of Spain is in New Zealand, but the antipode of most locations in North America lies in the Pacific Ocean.

**Example 2:**

**Human:** I am planning to start a book club with some friends. Can you write an email invitation for the same?

**Assistant:** Here is an example email invitation for a book club:

**Subject:** Join me for a book club!

Hi [Friend's Name], how are you?

I'm starting a book club with some friends, and I'd love to have you join us!
The book club is a casual and fun way to read and discuss books that we love or want to learn more about, and also make new friends. We plan to meet more-or-less once a month.

We're thinking of having our first meeting on [Date & Time]. Could that work for you?

Would love to see you there!

Thanks,
[Your Name]

The significance of LIMA-1k lies in its exceptional efficiency and effectiveness. Remarkably, despite its modest size of only 750,000 tokens, models fine-tuned on LIMA-1k exhibit robust generalization capabilities and compete closely with extensively trained RLHF models. This dataset convincingly demonstrates the practicality of minimal yet highly curated datasets for achieving high-quality alignment, substantially reducing computational costs and democratizing the capability for effective model alignment.

## Comprehensive Description of DEITA-6k Dataset

DEITA-6k is a meticulously curated dataset of **6,000 instruction-response pairs** selected from a pool of **300,000 high-quality public instruction datasets**, including ShareGPT (58k), UltraChat (105k), and WizardLM (143k). It is designed to be the *smallest effective collection* that still achieves state-of-the-art alignment quality after SFT on 7B- and 13B-parameter models.

**Selection Methodology** The DEITA pipeline employs three criteria for selection:

* **Complexity (c):** Measured using an EVOL-COMPLEXITY scorer trained to evaluate the reasoning depth of instructions.

* **Quality (q):** Assessed using an EVOL-QUALITY scorer focusing on the helpfulness and informativeness of responses.

* **Diversity:** Ensured by applying a cosine similarity threshold ($\tau = 0.9$) over sentence embeddings to avoid redundancy.

The final score $s = c \times q$ ranks the samples, and the highest-ranked examples satisfying diversity constraints form the DEITA-6k dataset.

**Key Statistics:**

<table><thead><tr><td>Statistic</td><td>Value</td></tr></thead><tbody><tr><td>Total Examples</td><td>6,000</td></tr><tr><td>Source Pools</td><td>ShareGPT (58k), UltraChat (105k), WizardLM (143k)</td></tr><tr><td>Average Input Length Ceiling</td><td>2,048 tokens</td></tr><tr><td>Domain Coverage</td><td>Balanced across 8 MT-Bench categories</td></tr><tr><td>Complexity Gain vs Random</td><td>+0.43 MT-Bench (6.46 → 6.03 on LLaMA-13B)</td></tr><tr><td>Quality Gain vs Random</td><td>+3.17% AlpacaEval (77.08% → 73.91%)</td></tr><tr><td>Scaling Insight</td><td>3K DEITA items match the performance of 300K examples</td></tr></tbody></table>

## Representative Examples:

### Example 1: Entain's Competitors Discussion

**Human:** list me some of Entain's competitors in their industry

**Assistant:** Entain, formerly known as GVC Holdings, is a multinational sports betting and gaming company. Competitors include:

1. Flutter Entertainment

2. William Hill

3. Bet365

**Human:** competitors who are also expanding into esports, game development

**Assistant:** Companies include:

1. DraftKings

2. FanDuel

3. Betway

4. Unikrn

**Human:** what studios has Unikrn acquired? Furthermore, elaborate on acquired.

**Assistant:** As of my knowledge cutoff, Unikrn has not acquired any gaming studios. Explanation of acquisitions:

1. Tencent/Riot Games: Full acquisition over time.

2. Tencent/Supercell: Majority stake through a consortium.

3. Activision Blizzard: Complete acquisition of Vivendi Games.

### Example 2: IP Ownership and Business Model

**Human:** I have ownership of the IP of video games created by S2 Games excluding Heroes of Newerth... What is the appeal of this, to any
investor?

**Assistant:** Key appeals include:

1. Nostalgia and a loyal player base.

2. Unique IP and lore for expansion.

3. Use of AI-generated art and modern features.

4. Platform potential for indie developers.

5. Revenue-sharing model incentives.

6. Strong growth potential with community engagement.

## Example 3: Data Warehouse Simplifications

**Human:** explain data warehouse architecture in easy language

**Assistant:** A data warehouse collects data from various sources, transforms it, and stores it centrally. Main components:

1. Data Sources

2. ETL Process

3. Data Warehouse

4. OLAP System

5. Front-End Tools

## Example 4: Contracted Chief Brand Officer's Role

**Human:** Generate a description of a Contracted Chief Brand Officer's role

**Assistant:** A Contracted Chief Brand Officer becomes an integral part of your executive team, leading brand strategy and development, working with internal teams and external creatives to build a competitive brand presence, and providing ongoing strategic advisement.

### Summary:

DEITA-6k demonstrates that **quality, complexity, and diversity can be engineered simultaneously** to produce a highly effective dataset for alignment. Despite its compact size, it enables language models to achieve near-SOTA performance with significantly reduced computational costs.

### Comprehensive Description of TULU-50k Dataset

#### Comprehensive Description of TULU-50K Dataset

##### Origin and Purpose:

TULU-50K is a 50 000-example SFT subset dis-

tilled from the public TÜLU 3 SFT pool (≈939 k dialogues) released by the Allen Institute for AI. The subset was produced with the MIG (Maximising Information Gain) sampler (Chen et al., 2025), which automatically balances *quality*—measured by DEITA scores—and *diversity* in a semantic label-graph. Its primary goal is to offer an open, compact yet high-coverage instruction-following corpus for research on post-training large language models, especially where full-scale SFT mixes are computationally prohibitive. The data are licensed ODC-BY and are entirely in English.

### Selection Method:

MIG first tags every dialogue in the 939 k-sample pool with semantic topic labels, then scores each instance with DEITA quality metrics. It models the tag space as a weighted graph, propagates information across semantically-related tags, and greedily selects examples that yield the highest marginal information gain—provably achieving a $(1 - 1/e)$ approximation of the optimal subset. Compared with random or purely quality-based filtering, the resulting 50 k set delivers state-of-the-art downstream scores while cutting sampling time by two orders of magnitude.

### Some Examples:

#### Example 1:

**Human:** Captain Jake, a somewhat short-tempered commercial airline pilot, has to navigate a complex flight path to avoid airspace controlled by airlines embroiled in corporate squabbles. The flight path consists of a series of vectors in 3-dimensional space. Each vector represents a leg of the journey from one waypoint to another.

1. Given the following vectors representing legs of the journey:

- Vector A: $\mathbf{a} = (3, 2, -1)$

- Vector B: $\mathbf{b} = (-2, 4, 5)$

- Vector C: $\mathbf{c} = (1, -3, 2)$

Calculate the total displacement vector $\mathbf{d}$ from the start to the end of the journey. Express $\mathbf{d}$ in its simplest form.

2. Captain Jake needs to maintain a steady altitude change to avoid turbulence caused by corporate squabbles in certain airspace layers. If the journey described by vector $\mathbf{d}$ must ensure that the vertical displacement
(z-component of **d**) is zero, determine the required adjustment vector **e** = (x, y, z) such that the modified displacement vector **d'** = **d** + **e** satisfies this condition. Additionally, compute the magnitude of the adjustment vector **e**.

**Assistant:** Let's break down the problem into the given tasks and solve each part step by step.

1. Calculate the total displacement vector **d**:

Given vectors:

$$ 
\begin{aligned}
\mathbf{a} &= (3, 2, -1), & \mathbf{b} &= (-2, 4, 5), \\
\mathbf{c} &= (1, -3, 2)
\end{aligned}
 $$

The total displacement vector is:

$$ \mathbf{d} = \mathbf{a} + \mathbf{b} + \mathbf{c} $$

Compute each component:

$$ 
\begin{aligned}
\mathbf{d}_x &= 3 + (-2) + 1 = 2, \\
\mathbf{d}_y &= 2 + 4 + (-3) = 3, \\
\mathbf{d}_z &= -1 + 5 + 2 = 6
\end{aligned}
 $$

So, the total displacement vector is:

$$ \mathbf{d} = (2, 3, 6) $$

2. Determine the adjustment vector **e**:

We want **d'** = **d** + **e** such that the z-component of **d'** is zero:

$$ 6 + z = 0 \implies z = -6 $$

Thus, **e** = (x, y, -6). Since only the vertical displacement needs correction, we can choose x = 0 and y = 0 for simplicity. Therefore:

$$ \mathbf{e} = (0, 0, -6) $$

The magnitude of **e** is:

$$ |\mathbf{e}| = \sqrt{0^2 + 0^2 + (-6)^2} = \sqrt{36} = 6 $$

**Final Answer:** The displacement vector is **d** = (2, 3, 6), and the magnitude of the adjustment vector **e** is 6.

**Example 2:**

**Human:** A dedicated spectator named Alex spends their weekends analyzing basketball player statistics to provide data-driven recommendations. Alex

has been tracking the performance of a team throughout the season and has compiled a dataset consisting of the following metrics for each player: points per game (PPG), assists per game (APG), rebounds per game (RPG), and a newly devised metric called *Efficiency Rating* (ER), which is calculated as:

$$ \text{ER} = (\text{PPG} \times 0.4 + \text{APG} \times 0.3 + \text{RPG} \times 0.3) $$

• 1. Alex wants to determine the optimal combination of players in a 5-person lineup to maximize the team's average Efficiency Rating. Given that there are 10 players on the team, develop an algorithm or mathematical model that can identify the best combination. What is the maximum possible average Efficiency Rating for the team?

• 2. After selecting the optimal lineup, Alex decides to project the team's overall performance in the upcoming playoffs. Alex uses a regression model where the team's winning probability in a game is given by:

$$ P(\text{win}) = 0.5+0.1 \times (\text{average ER of the lineup}) $$

Assuming the optimal lineup maximizes the team's average ER to 25, calculate the probability of the team winning a best-of-7 series, where the team must win 4 out of 7 games to advance.

**Assistant:** Let's solve the problem in two parts.

1. Optimal Lineup to Maximize Average ER:

• Sort the 10 players by their Efficiency Rating (ER).

• Select the top 5 players with the highest ER values:

$$ ER_1 \geq ER_2 \geq ER_3 \geq ER_4 \geq ER_5 $$

• Calculate the maximum average ER:

$$ \frac{ER_1 + ER_2 + ER_3 + ER_4 + ER_5}{5} $$

• According to the problem, the maximum average ER is given as 25.

2. Projecting the Team's Winning Probability:
Using the regression model:

$$P(\text{win}) = 0.5 + 0.1 \times 25 = 0.5 + 2.5 = 3.0$$

Since probabilities cannot exceed 1, the model saturates at:

$$P(\text{win}) = 1$$

**Probability of Winning a Best-of-7 Series:**

- The team must win 4 out of 7 games. - If $p = 1$, the team is certain to win each game.

- Therefore, the probability of winning the series is effectively:

$$P(\text{series win}) = 1$$

**Final Answer:** The maximum possible average Efficiency Rating is 25, and the team has a theoretical 100% chance of winning the best-of-7 series under the given model.

**Concluding Remarks:**

TULU-50K strikes a pragmatic balance between dataset footprint and skill coverage. Empirically, models fine-tuned on this 50 k slice match or exceed full-pool baselines on both knowledge-centric (ARC, MMLU) and preference-centric (AlpacaE-val v2, MT-Bench) benchmarks while slashing compute and curation time. Its transparent provenance, permissive license, and rich per-sample metadata make it an efficient, community-friendly test-bed for exploring data-centric alignment questions and rapid prototyping of instruction-tuned LLMs.

## A.2 Illustrative Examples from Final Darija Datasets

### A.2.1 Illustrative Examples from Darija Lima

See Figures 1 and 2.

### A.2.2 Illustrative Examples from Darija Deita

See Figures 3, 4, and 5.

### A.2.3 Illustrative Examples from Darija TULU

See Figures 6 and 7.

## A.3 Efficiency and carbon footprint

**Protocol.** We follow the GREEN 500 methodology and the recommendations of Strubell et al. (2019); Henderson et al. (2020). The total electrical energy $E$ for a training run is

$$E = \sum_{g \in \{\text{A100,H100}\}} N_g t_g P_g \text{ PUE } \eta,$$

where $N_g$ and $t_g$ are the number of GPUs and wall-clock hours, $P_g$ is the average power draw (350 W for A100-80 GB (Corporation, 2024), 500 W for H100-80 GB), PUE = 1.3 is a modern datacentre power-usage effectiveness, and $\eta = 0.9$ down-scales for the fact that accelerators seldom sustain their full TDP for an entire job.

**Our training budget.** Table 4 lists every experiment reported in Section 4. All ablations on the 4 B backbone consume a combined 10 GPU·h on A100-80 GB, while the GEMMAROC run uses 48 GPU·h on H100-80 GB. Plugging these figures into the formula above gives:

$$ 
\begin{aligned}
E_{\text{A100}} &= 10 \text{ h} \times 0.35 \text{ kW} \times 1.3 \times 0.9 = 4.1 \text{ kWh}, \\
E_{\text{H100}} &= 48 \text{ h} \times 0.50 \text{ kW} \times 1.3 \times 0.9 = 28.1 \text{ kWh}, \\
E_{\text{total}} &\approx 32 \text{ kWh}.
\end{aligned}
$$

With a 2024 global mean grid-carbon intensity of 0.40 kg CO₂/kWh (Agency, 2024) this implies

$$C_{\text{ours}} \approx 13 \text{ kg CO}_2\text{e.}$$

**Baselines: Atlas-Chat.** Atlas-Chat 27 B used LoRA rank 256, 3 epochs over 500 k sequences of length 2 048 (≈ 675 M training tokens)—and used the GEMMA-2-27B as its base model. A recent vLLM benchmark on a single A100-80 GB reports an inference throughput of ≈ 5 × 10² tok s⁻¹ for Gemma-2-27B (Databasemart, 2025). Training carries out a forward and a backward pass; empirical FLOP analyses therefore divide inference speed by ~3. We conservatively adopt $r = 55$ tok s⁻¹ GPU⁻¹.

$$t_{\text{wall}} = \frac{675 \times 10^6}{8 r 3600} \approx 4.26 \times 10^2 \text{ h},$$

$$t_{\text{GPU}} = 8 t_{\text{wall}} \approx 3.41 \times 10^3 \text{ GPU-h.}$$

Using the NVIDIA A100-80 GB SXM4 average training draw $P = 0.35$ kW (NVIDIA, 2024) and an industry-average PUE of 1.3 with 90 % utilisation, the energy and carbon footprints are

$$ 
\begin{aligned}
E &= t_{\text{GPU}} P \text{ PUE } 0.9 \approx 1.4 \text{ MWh}, \\
C &= E \times 0.436 \text{ kg CO}_2\text{e kWh}^{-1} \approx 610 \text{ kg CO}_2\text{e.}
\end{aligned}
$$
<table><caption>Table 4: Revised energy-accounting input reflecting the updated 10 A100 GPU·h for all 4 B experiments.</caption><thead><tr><th>Run</th><th>GPUs</th><th>Hours</th><th>GPU·h</th><th>Purpose</th></tr></thead><tbody><tr><td>Gemma-3-4B-LIMA</td><td>2xA100</td><td>0.4</td><td>0.8</td><td>1 K single-turn</td></tr><tr><td>Gemma-3-4B-DEITA</td><td>2xA100</td><td>0.8</td><td>1.6</td><td>6 K dialogue</td></tr><tr><td>Gemma-3-4B-DEITA<sup>+</sup></td><td>2xA100</td><td>0.8</td><td>1.6</td><td>+30</td></tr><tr><td>Gemma-3-4B-TULU</td><td>2xA100</td><td>3</td><td>6</td><td>50 K reasoning</td></tr><tr><td><em>Subtotal 4 B</em></td><td></td><td></td><td><strong>10</strong></td><td></td></tr><tr><td>Gemma-3-27B-TULU</td><td>8xH100</td><td>6</td><td>48</td><td>final GEMMAROC</td></tr><tr><td><strong>Total</strong></td><td>—</td><td>—</td><td><strong>58</strong></td><td></td></tr></tbody></table>

(Ember and Institute, 2024)

Finally, at a public on-demand rate of $2 A100-h⁻¹ the run would cost ~$6.8k, underscoring the budget advantage of our own (far cheaper) fine-tune.

## A.4 Training and Eval Loss of GemMaroc-27b
كيفاش تكون سعيد فزواج ماشي سعيد؟

الناس كيبقاو فزواج ماشي سعيد لأسباب بزاف. إذا كنت واحد من هاد الناس، تقدر تحس براسك ما يمكنش تكون فرحان. على الرغم من هادشي، تقدر تلقى طريقتك الخاصة للسعادة، حتى فشي وضعية خايبة، بممارسة عادات اللي كتوصل للسعادة، وحتى تقدر تخدم على الزواج ديالك باش توصل للسعادة كزوجين.

### تعلم تكون سعيد

1. قلب على طرق باش تكون ممتن. ماشي ديما ساهل تكون ممتن، خاصة وسط علاقة خايبة. على الرغم من هادشي، الامتنان يقدر يعاونك باش تواجه العلاقة الخايبة، ويقدر يوصلك باش تكون سعيد. خود الوقت كل نهار باش تعرف شنو ممتن ليه. حاول تكتب شي حاجات ممتن ليهم كل نهار فواحد الجورنال. تقدر تستعمل حتى المنشورات اللي فيال social media لهاد الغرض. غالبا، واخا تكون الوضعية خايبة، تقدر تلقى شي حاجة تكون ممتن ليها. مثلا، تقدر مايعجبكش كيتعاملو معك فالعلاقة اللي فيها، ولكن تقدر تكون ممتن لأن عندك استقرار مالي فحياتك ف هاد الوقت. أولا، تقدر تكون ممتن لأن الطرف الآخر فالعلاقة كيبقى أب أو أم مزيانة لولادك.

2. شارك ف. 'flow experiences' لحظات ال Flow هي فاش كتنسي راسك فواحد التجربة بكونك مندمج كليا فشنو ما كنت كدير. إلا كنت فنان، كاتب، ولا حتى عداء، تقدر تكون ديجا فهمتي هاد النوع ديلان التجارب. هاديك اللحظة اللي كيطيح فيها العالم، و أنت غير كتجرب و لا كتستمتع باللي كدير. الدراسات بينات باللي كلما كان عندك لحظات Flow بزاف، كلما كنت سعيد بصفة عامه. اختار شي نشاط اللي كيتحدالك شوية، ولكن اللي يكون مالوف باش تقدر تنسى راسك فيه. مثلا، إلا كنت كتستمتع برسم المناظر الطبيعية، تقدر تجرب ترسم موضوع جديد بحال شي بورتريه ولا شي طبسيل ديل الفواكه.

3. حبس من نفس المشاكل. يعني، إلا كنت ديما كتلقى راسك كتجادل على نفس الحوايج، يمكن وصل الوقت باش تخط هاد الموضوع على جنب. خاصك تقرر باللي ماغاديش تناقش فيه حيث مايمكنش تتفقو و لا تحاولو تلقاو شي حل وسط اللي يخدم ليكم بجوج. مثلا، إلا كنتو كتجادلو بزاف على السياسة، تقدر تبغيو تخليو السياسة موضوع ممنوع. و لا، إلا كنتو كتجادلو بزاف على شنو من فيلم غادي تشوفو فالليلة ديل الجمعة، تقدر تبغيو تناوبو فالإختيار ديل الفيلم.

4. طور الاهتمامات الخاصة ديالك. إلا الزواج ديالك ماشي هو اللي بغيتي، يقدر وصل الوقت باش تلقى شي إشباع خارج الزواج ديالك وماشي على شكل خيانة. أن تكون عندك هوايات و اهتمامات خاصة غادي يعاونك باش تبقى مستقل وتبقى سعيد و مندمج مع العالم. فالحقيقة، تطوير الاهتمامات الخاصة فكرة مزيانة حتى إلا كان عندك زواج زوين. فكر أنك تستكشف اهتماماتك فالمكتبة، و لا تنضم لأنديه الهوايات المحلية أو دروس الطبخ، و لا تأخذ دروس مهنية فشي كلية مجتمعية قريبة. [...]

### تخدم باش يكون عندك زواج أحسن

1. خصصو وقت باش تكون مجموعين. تخصيص الوقت لبعضكم البعض طريقة باش تقولو أنكم ملتزمين ببعضكم البعض. قولو بأنكم غادي تقضيو وقت معين كل أسبوع مع بعضكم البعض، و بقاو عليه. غادي يعطيم حتى فرصة باش تعرفو بعضكم البعض من جديد.

2. تفكر و شنو كنتو كتقدروا. فاش يالاه تلاقينتو، غالباً كنتو منجذبين جزئيا للاختلافات اللي كان كيعرضها الطرف الآخر. مثلا، تقدر تكون كتفرحك حقيقة أنه كان ولا كانت مندفعين و كيعبهم يكونو عغويبين. دابا، تقدر تلقى راسك كتكره هاد الصفة. المهم هو تحاول تتفكر علاش كنتي كتحب هاد الصفة فالأول و تولي تستمتع بيها من جديد. مثلا، تقدر تطلع ليك فال راس فاش الزوج ديالك يبغي يخلي كلشي و يسافر للجبال. من جهة أخرى، كتخلي حياتك ما توليش مملة بزاف. حاولو تلقاو توازن، و استمتعو بالي قدرتوا عليه. [...]

<caption>Figure 1: Illustrative example 1 from Darija Lima dataset.</caption>
كاينين جوح ديال المفاهيم لي شوية مستقلين على بعضياتهم فencryption algorithms: ديال backdooring

* ال backdoor يقدر يكون explicit ولا implicit. ال explicit backdoor هو واحد لي كلشي عارف بالي راه كاين. أما ال implicit backdoor يكياول يبقى مخفي على مالين ال. explicit backdoor بطبيعة الحال، فاش كيكون شي algorithm الناس كيميلو يتجنبو ال explicit backdoors بصفة عامة، يعني ممكن يكون عندهم أمل باش يخدمو غير فوجود شي إطار قانوني لي كيفرض على implementers يستعملو ال system لي فيه ال. explicit backdoor مثال على شي Clipper chip ال (لي تخلى عليه ف الأخير). ال backdoor ماشي بالضرورة يكون ف ال algorithm، كثر ف Automatic key escrowing method. و نقنيا كان عبارة على التجميع ديال algorithms ف شي protocol أما بالنسبة ل Dual EC DRBG implicit backdoor، شوف ال EC DRBG Dual EC DRBG كمثال مشهور: كان كيخدم غير إلى مكان تا واحد عارف بالي فيه backdoor.

* ال security ديال ال backdoor يقدر يكون quantifiable، ولا لا. ف حالة ديال Dual EC DRBG، ال mechanism كيستعمل طرق رياضية معروفة: ال NSA كانت عارفة بالي استغلال ال internal secret backdoor كيتطلب معرفة ديال شي key، مبني على (elliptic curves) على discrete logarithm

شي security ماشي quantifiable هو لي كيوقع فاش كتحاول تدفع شي algorithm لي فيه عيب متعمد، ولا شي واحد لي عارف عليه شي cryptanalytic method لي ما نشرتيهش. هادي لعبة خطيرة بزاف بالنسبة ل شي وكالة تجسس، حيث مايمكنش تعرف واش أطراف آخرين يقدروا يلقاو العيب ولا لا. بحال هاد ال backdooring كيقلب عليك ف المدى الطويل.

لي مثير للاهتمام، ال NSA ما كتميلش تستعمل non-quantifiable backdoors. مثال مزيان هو DES. ف الوقت لي تصمم فيه، ال NSA كانت معقده باللي تقدر تواجه شي upfront 256 exhaustive search ، و بالي حتى واحد آخر (خصوصا السوفيات) ماعندوش ال technology و ال budget باش يدير هادشي. ال NSA كانت عارفة حتى على شي cryptanalytic method جديدة. (differential cryptanalysis) يعني فاش ال NSA تدخلات ف التصميم ديال DES، أصرت باش تنقص ال key من 64 ل 56 bits هادي إضافة ديال شي differential backdoor، و كانت واضحة بزاف، يعني شبه explicit، و أيضا باش تقوي التصميم ضد non-quantifiable differential cryptanalysis. هادا مثال مزيان على كيفاش ال NSA تجنبات بوعي شي backdoor. كاين حاجة وحده لي وكالات التجسس كيخافو منها كثر من أنهم ما قدروش يتجسسو: هي الفكرة بالي وكالات تجسس منافسة أخرى ممكن حتى هي تتجسس.

إذن شي backdoor حقيقي و مزيان هو لي كيستعمل quantifiable security باش يعطى شي backdoor. مصرح به ديال explicit. backdoor صعيب بزاف يكون عندك quantifiable security بلا ما تجعل ال Dual EC DRBG، و حتى قبل Dual EC DRBG الأفضل ف فنتها" ف هاد المجال هي Snowden business، ال cryptographers كانو كيشوفوها غريبة بزاف و شكاو بقوة ف شي تلاعب) شوف ال analyses ف 2007، 6 سنين قبل. (Snowden).

<caption>Figure 2: Illustrative example 2 from Darija Lima dataset.</caption>
**Human:**

خطوه بخطوه بمنظور عام certificate pinning process أهلا، بعيتك تشرح ليا ال

**Assistant**

أكيد! ها هي خطوه بخطوه شرح لعملية certificate pinning بمستوى عالي:

1. server بحال (web browser أو mobile app) كيبدأ اتصال ب API أو website بحال.

2. SSL/TLS certificate public key ديلو، اللي كيتضمن ومعلومات تعريفية أخرى.

3. trusted root authorities (محوسة ديل) مع مجموعة certificate كدير فحص لل client مخزنة عند الـ Certificate Authorities.

4. server وكدير اتصال آمن مع certificate موثوق، ال، لا يمكن عملية SSL/TLS handshake.

5. pins معرفة مسبقاً، واللي متخزنة مع مجموعة ديل server خلال هاد العملية، ال، كيفيرفي client عند الـ localement.

6. pins يقدروا يكونوا الـ هاد.

a) Public key pins: بحال (SHA-256 hash) ديل public key pin server عن عبارة hash.

b) Certificate pins: بحال (SHA-256 hash) ديل certificate pinter كامل.

7. clients من الـ pin كيطابق مع شيء server ديل إلا كان SSL/TLS handshake. كيرفض الإتصال وكيسالي الـ client.

8. pin validation ، يقدر يعطى للمستخدم رسالة تحذيرية أو يأخذ إجراءات أخرى باش ينبه الـ client. الـ pin واجه فشل في المستخدم للخطر الأمني المحتمل.

بإختصار، certificate pinning هي آلية كتسمح للclients يتأكدو باللي كيتوصلو مع الـ server المقصود ماشي محتال. عن طريق مقارنة SSL/TLS certificate pins معرفة مسبقاً، ال، clients يقدروا يخفقو من الخطر certificate-based attacks. الـ man-in-the-middle (MITM) attacks ديل.

<caption>Figure 3: Round 1 of an Illustrative example from Darija Deita dataset.</caption>

**Human:**

ماش هادشي كيعني باللي أي axios API call كنستعملو فيه secure؟ مثلا غادي يكون

**Assistant**

شهادة التثبيت بوحدها ما كتضمنش بأن كل API call كنستعمل مكتبة بحال axios غادي تكون آمنة، حيث كابنين ثغرات محتملة أخرى في التطبيق و الشبكة اللي ممكن تستغل. ولكن، certificate pinning تقدر تنقص بشكل كبير من خطر هجمات الوسيط (man-in-the-middle attacks)، اللي هي واحد من طرق الهجوم الشائعة في بزاف ديل الاختراقات الأمنية.

ملي كطبيقي axios certificate pinning مع HTTP client أخرى، مهم تبع أحسن الممارسات وتأكد بأن الـ client مكونفيگوري باش يغيريفي ويثبت الشهادة ديل السيرفر بشكل صحيح. بالزيادة على هادشي، تدابير أمنية أخرى خاصها تطبق، بحال output encoding و input validation ، باش تنقص بشكل أكبر من خطر الثغرات الأمنية.

باختصار، فاش certificate pinning إجراء أمني مهم، خاصها كنستعمل بالاقتران مع ممارسات أمنية أخرى باش نتأكدو بأن axios calls اللي كنستعمل.

<caption>Figure 4: Round 2 of an Illustrative example from Darija Deita dataset.</caption>
؟ certificate pinning Human اوكي، كيفاش نقادو axios باش نستعملو

Assistant

باش تصاوب `https` module ف Node.js ، ممكن تستعمل ال certificate pinning باش تخدم axios باش تكو نفيگوري معرفين مسبقاً. ها هو pins واحد المجموعة ديال against ديل السيرفر certificate لي كيقاليدي ال HTTPS agent واحد ال واحد المثال كيشاش دير هادشي:

```javascript
1 ````
2 const https = require('https');
3 const axios = require('axios');
4
5 // Define the list of pins as an array of SHA-256 hashes
6 const pins = ['sha256/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx',
7 'sha256/yyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyyy';
8
9 // Create a custom HTTPS agent with pin validation
10 const agent = new https AGent({
11 pinning: true,
12 checkServerIdentity: function(host, cert) {
13 const certPins = cert.getPublicKey().getFingerprints();
14 for (let pin of pins) {
15 if (certPins.has.pin)) {
16 return undefined; // Certificate is valid
17 }
18 }
19 // Create a custom HTTPS agent with pin validation
20
21 });
22
23 // Use the custom HTTPS agent with Axios
24 const instance = Axios.create({
25 httpsAgent: agent
26 });
27
28 // Make a secure API call with Axios
29 instance.get('https://example.com/api/data')
30 .then(response => {
31 console.log(response.data);
32 })
33 .catcherror => {
34 console.errorerror);
35 });
```

فهاد المثال، أول حاجة كنديفينيو واحد ال array ديال SHA-256 hashes لي كيمتل اللائحة ديل pins لي بغينا نقاليديو against certificate ديل السيرفر `https Agents` constructor ها. من بعد كنصاوبو واحد ال HTTPS agent باستعمال pin validation.

من بعد، كنديفينيو واحد ال `checkServerIdentity` function لي كتيطع ليها فاش كيوصل ال certificate ديل السيرفر. هاد الدالة كتجيد ال SHA-256 fingerprints ديال public key ديال certificate وكتقارنهم مع اللائحة لي معرفة مسبقا ديل pins. إلا تلاقى `match`، الدالة كترجع `undefined`، وكاتبين باللي ال certificate صحيح. إلا مافاتش شئ `match`، الدالة كترجع واحد ال `Error` object `واحد ال` message لي كيين باللي ال pin validation فثل.

أخيرا، كنصاوبو واحد ال `Axios` باستعمال ال `instance` ديال `Axios.create()` method ال `instance.get()` `option` ال HTTPS agent ديلنا. ديك الساعة ممكن نصاوبو واحد ال `secure` API call باستعمال ال `agent` against `pins` ديالنا لي معرفين مسبقا.

لاحظ باللي هادا غير واحد المثال كيشاش تكو نفيگوري `Axios` باش تخدم pinning لاحظ باللي هادا غير واحد المثال كيشاش تكو نفيگوري ` certificate pin validation` `specific use case` ال environment `feature` على حساب ال pins ديالنا باش يقاليدي ال HTTPS agent لي غيستعمل ال method ديلنا ديال السيرفر `Axios` باش تخدم pinning لاحظ باللي هادا غير واحد المثال كيشاش تكو نفيگوري ` certificate pin validation` `specific use case` ال environment `feature` على حساب ال pins ديالنا باش يقاليدي ال HTTPS agent لي غيستعمل ال method.

<caption>Figure 5: Round 3 of an Illustrative example from Darija Deita dataset.</caption>
Assistant

باش تصاوب شي تصويرة فعالة وعندو معنى، خاص تكون عندك فهمة واضحة للمعطيات والقصة اللي كتعادد. ها بعض الطرق

COVID-19 ديل التلقيح ضد dataset المبتكرة لتصور المعطيات اللي تقدر تفكر فيها بالنسبة لهاد

1. **Time-Series Analysis**:

هادي تقدر تكون الخطوة الأولى فين كتحلل التقدم ديال التلقيح مع مرور الوقت. هادشي ممكن تمثلو بLine Plots ، فين محور السينات (x-axis) كيمثل الوقت، ومحور العينات (y-axis) كيمثل عدد التلقيحات. خطوط مختلفة تقدر تمثل بلدان مختلفة.

2. **Geo-mapping**:

ممكن تستعمل Geographical Heatmaps باش تبين التغطية ديال التلقيحات ضد COVID-19 فالمعالم. المناطق اللي فيها الظلال غامقة كتمثل أعداد كبيرة ديال التلقيحات. هادشي غايعطي نظرة واضحة كيفاش المناطق المختلفة كتمشي فالتقدم ديال التلقيحات.

3. **Comparison Dashboard**:

دحل pie charts اللي كتقارن نسب التلقيح بين بلدان مختلفة، وكيبينو الدول اللي قريبة من مناعة القطيع (herd immunity). وزيادة على هادشي، stacked bar line charts ممكن تستعمل باش تبين النسبة ديال الجرعة الأولى إلى الجرعة الثانية ديال التلقيحات لكل بلد.

4. **Trend Analysis**:

بين المعدل باش كل دولة كتلقح السكان ديالها. لهادشي، استعمل خط بياني فين محور السينات (x-axis) كيشير للوقت ومحور العينات (y-axis) كيبين التلقيحات اليومية. خطوط مختلفة تقدر تدل على بلدان مختلفة.

5. **Correlation Heatmap**:

هادا غادي يساعد في فهم كيفاش شي parameter (بحال مجموع التلقيحات ولا الناس اللي تلحو) مرتبط بparameters أخرى (بحال مجموع السكان ولا النسبة المئوية ديال السكان).

6. **Aging Analysis**:

عرض الأعمار ديال الناس اللي كياخدو اللقاح بدمج معطيات التلقيح مع معطيات ديموغرافية.

7. **Vaccine Type Analysis**:

في حالة إلى ال dataset يتضمن النوع ديال اللقاح اللي تعطى، ممكن تصاوب dashboard كيمثل اللقاحات الأكثر استخداماً فالعالم، ولا شي robust chart كيبين استخدام اللقاحات فبلدان مختلفة.

تذكر، التصورات خاص ديمما تكون تفاعلية باش تعاون المستخدم. أدوات بحال Tableau, Power BI, Python libraries تذكر، التصورات خاص ديمما تكون تفاعلية باش تعاون المستخدم. أدوات بحال JavaScript libraries D3.js ممكن تستعمل باش تصاوب هاد التصورات القوية.

<caption>Figure 6: Illustrative example 1 from Darija TULU dataset.</caption>
**Human** كتب دالة لي ماشي عي تبدل كاع ليسباس فالسلسلة النصية ب'&'39#&20%;39#&';'، ولكن حتى تحيد أي ليسباس كابن فالأول و الآخر قبل ما تبدل. إضافة على هادشي، الدالة خاصها تخدم مع ليسباسات المتتابعين، يعني تبدلهم ب'&'39#&20%;39#&';' وحده. الدالة خاصها تكون حساسة لحالة الأحرف (case-sensitive)، يعني تطرق بين الحروف الكبيره و الصغيره.

باش نزيهو الصعوبة، الدالة خاصها تبدل أي تواجد للحروف 'a'، 'c' بالقيم ASCII ديالهم بصيغة hexadecimal التبديل خاصو يكون case-sensitive، يعني 'A'، 'B'، 'C' خاصهم يتبدلو بقيم مختلفة على 'a'، 'b'، 'c'. الدالة خاصها تخدم مع التكرارات المتتالية ديال هاد الحروف وتبدلهم بقيمة hexadecimal وحده مناسبة.

علاوة على ذلك، الدالة خاصها تقدر تخدم مع سلاسل نصية الطول ديالها كيوصل لـ 1 مليون حرف. الدالة خاصها تكون مهيئة للسرعة و ما تستعملش شي دوال مذمجة للWNABT بالسلسل النصية.

**Assistant** تنفيذ بحال هاد ال'functionality' بكفاءة، خصوصا بلا ما نعمدو على ال built-in string manipulation، كيتجاوز القدرات ديال شي لغة interpreted بحال Python ولا JavaScript لي كستعمل عاده في 'interactive' mode.

ولكن، ها واحد الطريقة بPython مع شوية ديال ال built-in functions باش نفهمو مزيان:

1- **def replace_chars(string):**
2     string = string.strip()
3     nspaces = 0
4     result = ''
5     ascii_hex = {'a': '%61', 'b': '%62', 'c': '%63',
6                 'A': '%41', 'B': '%42', 'C': '%43'}
7
8- **for char in string:**
9     if char.isspace():
10         nspaces += 1
11     if nspaces == 1:
12         result += '%20'
13     else:
14         nspaces = 0
15     if char.lower() in ascii_hex:
16         if not resultendswith(ascii_hex[i]):
17             result += ascii_hex[i]
18     else:
19         result += char
20
21  return result

هادشي مجاوبش على السؤال ديالك نيشان حيث كيستعمل ال built-in string manipulation functions ، بسبب التعقيد اللي عايكون ضروري باش نتفادو كاع هاد الدوال.

لغة بمستوى أدنى بحال C++ أو Rust عادي تكون مناسبة كثر لهاد المستوى ديال ال string manipulation و optimization بشكل عام، بحال هاد ال) constraint بحال عدم استخدام ال built-in functions والتعامل مع سلاسل نصية فيها حتى لمليون حرف مع تبديالت معينة (ماشي بالضروره يكون مطلوب في تطوير التطبيقات، و كيشبه لشي مهمة متخصصة و بمستوى منخفض ديال التعامل مع النصوص.

أيضا، مهمة التعامل مع سلسلة نصية فيها حتى لمليون حرف كتطلب تفكير عميق في إدارة الذاكرة(memory management)، شي حاجة لي اللغات ذات المستوى الأدنى كتكون مؤهلة ليها أحسن. إضافة على ذلك، تقنيات ال optimization بحال string استخدام هياكل بيانات محددة (eg. ropes or gap buffers) كيتدارو بشكل أحسن بلغات ذات مستوى منخفض. إذن، هادا موضوع متقدم كيتطلب معرفة بلغة صديقة للcompiler، فهم الASCII character sets، و ال memory ننصح باستشارة مهندس برمجيات (software engineer) أو مطور متخصص في تحسين الأداء و ال manipulation.

<caption>Figure 7: Illustrative example 2 from Darija TULU dataset.</caption>
<figure><img src="image_2.png" /><figcaption>Figure 8: Training loss (per step) for GemMaroc-27B over 1 300 global steps, showing a rapid decline from 3.5 down to 1.3 as the model learns.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>Figure 9: Evaluation loss at each validation checkpoint (every 50 steps), reaching its minimum (0.915) at global step 900 (≈2 epochs), indicating that after just two passes over the data we obtain our most powerful fine-tuned model.</figcaption></figure>