MoroLMai V1.2 (135M): a small English + Moroccan Darija chat model, trained from scratch

MoroLMai is a 136-million-parameter language model trained from randomly initialised weights, with no pre-made model as a starting point. It reads and writes English and Moroccan Darija (Arabic script and Latin script) and is small enough to run on a laptop CPU or a 2 GB GPU through llama.cpp.

It was built by one person on one consumer GPU (RTX 3050 6 GB). It is an honest small-scale research and education project, not a competitor to large models. Read the Limitations before using it.

Developer	Hamza Daoud
Version	V1.2 (early release; V1.3 is in training)
Parameters	135,961,088
Languages	Moroccan Darija (ary), English
Format	GGUF (f16, Q8_0, Q4_K_M)
Website	https://morolmai.com
Code	https://github.com/hamzadaoud/

Files

File	Size	Use
MoroLM-135m-v1.2-Q8_0.gguf	145 MB	Recommended. Closest to the original quality.
MoroLMai-135m-v1.2-Q4_K_M.gguf	85.9 MB	Smallest and fastest. At this model size, 4-bit visibly loses quality compared with Q8_0.
MoroLM-135m-v1.2-f16.gguf	273 MB	Full 16-bit weights, for further conversion or research.

Quick start

Download a GGUF file and a recent llama.cpp build.

Chat in the terminal (settings that give stable, repeatable answers):

llama-cli -m MoroLM-135m-v1.2-Q8_0.gguf -cnv -c 2048 --temp 0.4 --repeat-penalty 1.15 --top-k 40 --top-p 0.9

Start a local chat web page:

llama-server -m MoroLM-135m-v1.2-Q8_0.gguf --port 8080 -c 4096 --temp 0.4 --repeat-penalty 1.15

Then open http://localhost:8080. On Windows use llama-cli.exe and llama-server.exe.

From Python with llama-cpp-python:

from llama_cpp import Llama

llm = Llama(model_path="MoroLM-135m-v1.2-Q8_0.gguf", n_ctx=2048)
out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Who are you?"}],
    temperature=0.4, repeat_penalty=1.15, max_tokens=200,
)
print(out["choices"][0]["message"]["content"])

Tips for a model this small:

• Use a low temperature (0 to 0.4) for facts and tests, and a repeat penalty around 1.1 to 1.15. Above about 1.3 the answers get strange.
• Keep answers short. The model drifts the longer it writes.
• Clear the conversation between unrelated test prompts, because earlier answers stay in the context and change the next one.
• The chat template is stored inside the GGUF files, so llama-cli -cnv and llama-server apply it automatically.

Chat format

The model was fine-tuned on this template, with special tokens for the roles:

<|system|>
{system prompt}
<|user|>
{user message}
<|assistant|>
{answer}</s>

Model architecture

A LLaMA-style decoder-only Transformer (LlamaForCausalLM).

Setting	Value
Layers	38
Hidden size	512
Feed-forward	SwiGLU, 512 to 1,536 to 512
Attention	8 query heads, 4 key-value heads (grouped-query attention)
Position encoding	RoPE (theta 10,000)
Normalisation	RMSNorm (eps 1e-5)
Embeddings	Tied input and output embeddings
Vocabulary	32,000 (SentencePiece BPE, trained on the project's own text)
Training window	1,024 tokens
Parameters	135,961,088 (about 119.6M in the blocks, 16.4M in the embedding table)

Special tokens: <unk> 0, <s> 1, </s> 2, <pad> 3, <|system|> 4, <|user|> 5, <|assistant|> 6.

Training

1. Pre-training (learning the language)

Setting	Value
Objective	Next-token prediction on raw text
Steps	8,000
Tokens per step	131,072 (micro-batch 8 x gradient accumulation 16 x 1,024 tokens)
Total tokens seen	about 1.05 billion
Optimiser	AdamW, betas (0.9, 0.95), eps 1e-8, weight decay 0.1
Learning rate	Peak 6e-4, 500 warm-up steps, cosine decay to one tenth of the peak
Gradient clipping	1.0
Precision	bfloat16
Speed	about 4,100 tokens per second (about 2.9 days for 1B tokens)
Final validation loss	3.4177 (perplexity about 30.5)

Data: about 1,000,000 documents of Moroccan Arabic web text from FineWeb-2 (ary_Arab) and about 400,000 documents of English educational text from FineWeb-Edu (sample-10BT). About 0.5% of the documents (every 200th) were held out for validation.

For scale: the model saw roughly 8 training tokens per parameter. The Chinchilla scaling result suggests about 20 per parameter as compute-optimal, so the model is under-trained for its size. More data would likely help more than more parameters.

2. Supervised fine-tuning (learning to chat)

The pre-trained model was fine-tuned on question-and-answer conversations in English and Darija. The loss is computed only on the assistant's answer tokens (plus the end-of-sequence token), not on the question.

Version	Data	Validation loss
V1	669,765 conversations	2.4436 (perplexity 11.5)
V1.1	11,536 curated rows, 3 epochs	2.1432 (perplexity 8.5)
V1.2	V1.1 data plus extra rows, 2,000 Moroccan-topic rows, identity rows (x3) and 9,000 replay rows	1.5823 (perplexity 4.9)

Read the V1.2 loss with care. It looks much lower mainly because the Moroccan-topic rows contain many near-duplicates, so the validation set is partly repeated training text. It does not mean the model is three times better. The real answers matter more than this number.

Replay rows (samples from earlier training data) are mixed in to reduce catastrophic forgetting, which is when new fine-tuning erases earlier skills.

Data sources and licences

• Pre-training: FineWeb-2 and FineWeb-Edu (both ODC-By).
• Fine-tuning: a mixture of public English and Darija instruction datasets plus hand-written identity and Moroccan-topic rows. Some of the public sources are licensed non-commercial (for example alpaca-cleaned and darija_english).
• Because of those non-commercial sources, the weights are released under CC BY-NC 4.0 for now. This will be revisited when a clean, commercially usable fine-tuning set exists. The training data itself is not redistributed here.
• Nothing was scraped from private social-media content.

The full source list with licence per dataset will be added once it is finalised.

Hardware

A single desktop PC: NVIDIA RTX 3050 (6 GB), AMD Ryzen 5 3600, 24 GB RAM, Windows. The micro-batch was small with gradient accumulation to fit in memory, and training used automatic checkpoint-and-resume to survive crashes and power cuts. Energy use and carbon emissions were not measured.

Evaluation

There is no formal benchmark for this release, and none is claimed. Quantitative results are limited to the validation losses above.

Development testing used a fixed set of held-out prompts (identity, greetings, simple facts about Morocco, short translations between English and Darija), run with deterministic settings (--temp 0 --repeat-penalty 1.1) and a cleared context between prompts. Test prompts are removed from the training data of later versions so that they stay a fair test.

Sample conversations

Real, unedited outputs from MoroLM-135m-v1.2-Q8_0.gguf, run with llama-cli --temp 0 --repeat-penalty 1.1 (llama.cpp build b11476) on the author's PC. Generation speed in this session was about 70 to 96 tokens per second. The "Verdict" column is the author's own judgement.

All of these prompts were typed one after another in one session without clearing the context, so each answer is partly influenced by the earlier ones. Results are shown as they came out, including the failures.

Prompt	Answer	Verdict
who are you ?	I'm MoroLMai, a small assistant that speaks Moroccan Darija and English.	Good
tell me your name	I'm MoroLMai. I can chat and translate short sentences between Halima and Moroccan Darija.	Name right, but "Halima" is an invented word
شنو سميتك؟	MoroLMai هو سميتي، ومرحبا بيك!	Good
صباح الخير	صباح النور! كيفاش نقدر نعاونك؟	Good
Translate to Moroccan Darija: Where is the train station?	فين كاينة محطة القطار؟	Good
Translate to English: فين كاينة محطة القطار؟	Where is the bus station?	Mostly right ("bus" instead of "train")
wach nta labas?	MoroLMai هو سميتي، ومرحبا بيك!	Wrong: repeats its name instead of answering
salam, kidayr?	MoroLMai هو سميتي، ومرحبا بيك!	Wrong: same identity loop
fin kayn lmarsa f casa?	MoroLMai هو سميتي، ومزيان ليا!	Wrong: same identity loop
كيفاش نمشي من الرباط لمراكش؟	تقدر تبدا بباب بوجلود إلا كان عندك الوقت والطاقة.	Wrong: Bab Boujloud is a gate in Fes, not a route to Marrakech
what is the national dish of Morocco?	الصحرا كاينة فالرباط.	Wrong ("The Sahara is in Rabat")
where is Chefchaouen?	المدينة القديمة للرباط.	Wrong ("The old city of Rabat")
Translate to Moroccan Darija: I want a cup of tea.	بغيت شي حاجة كتاكلني.	Wrong ("I want something that eats me")
Give me three tips to learn English.	1. Put on the train:** In this way, you can use the train to travel to a different place. ...	Wrong: off-topic and repetitive
what is 2+2?	1. Put on the train station: In this case, you are going to the train station. ...	Wrong: no arithmetic, repeats earlier text

What this shows: V1.2 handles its name, a few greetings and simple sentence translation, but it often falls into an identity loop on casual questions, gets Moroccan facts wrong, and cannot do arithmetic or give structured advice. These are the main targets for V1.3.

Limitations

This is a 136M-parameter model trained on about one billion tokens. Expect the following:

• Made-up facts. It frequently gives wrong dates, names and numbers about Morocco and everything else, and sounds confident when it does. Never use it as a source.
• Identity confusion. V1.2 knows its name (MoroLMai) but sometimes drifts back to another identity in short chats. V1.3 targets this.
• Weak translation. English to Darija translation is inconsistent and can change the meaning of a sentence. Back-translating can reveal this.
• No reasoning mode. It does not "think step by step". It predicts the next token from the text so far, and long or multi-step questions usually fail.
• Darija inconsistency. Spelling, grammar and dialect vary, and quality differs between Arabic script and Latin-script (Arabizi) Darija. Arabizi was not specifically represented in the tokenizer.
• Quantisation cost. Q4_K_M answers are noticeably worse than Q8_0. Use Q8_0 when you can.
• Short context. Trained on 1,024-token windows. Long conversations degrade.
• Bias and safety. The model inherits biases from web text, has had little safety tuning, and can produce offensive, false or nonsensical output.

Out-of-scope use

Do not use it for medical, legal, financial or other high-stakes decisions, or as a verified knowledge base. It is not suitable for production use.

Intended use

• Experimenting with small language models trained from scratch.
• Learning how a language model is built end to end (tokenizer, pre-training, fine-tuning, GGUF export).
• Offline and low-resource experiments with Moroccan Darija.
• A starting point for community Darija data and evaluation work.

Roadmap

• V1.3: targeted fixes for identity, greetings and short multi-turn chats; all Moroccan-topic rows; three epochs from V1.1.
• Next: a verified Moroccan facts set with English and Darija question-and-answer variants, human-translated rather than machine-translated.
• Later: a tokenizer that includes Arabizi (this requires pre-training again), a clean commercially usable fine-tuning set, and a real benchmark.

Citation

@misc{morolmai2026,
  title  = {MoroLMai: a 136M English and Moroccan Darija language model trained from scratch},
  author = {Daoud, Hamza},
  year   = {2026},
  url    = {https://huggingface.co/HamzaDavi1/MoroLMai}
}

Glossary

• Darija: Moroccan Arabic, as spoken every day.
• Arabizi: Darija written in Latin letters and numbers (for example "3lash" or "wach").
• Pre-training: teaching a model the language by predicting the next token on a large amount of text.
• SFT (supervised fine-tuning): teaching a pre-trained model to follow the chat format using example conversations.
• GGUF: the single-file model format used by llama.cpp.
• Quantisation: storing weights with fewer bits to save memory, at some cost in quality.
• Perplexity: the exponential of the loss. Lower means the model is less surprised by the text.

Contact

Project website: https://morolmai.com. Source code: https://github.com/hamzadaoud/. Questions and corrections are welcome through the Community tab of this repository.