smallm-wolof

A 16.5M-parameter sparse Mixture-of-Experts language model trained from scratch on Wolof, a Senegalese language with very limited digital resources. This is a research/proof-of-concept project, not a production model — see Limitations before using it for anything beyond experimentation.

Why this exists

Wolof has no dedicated open reasoning/generative LLM despite being spoken by ~10M+ people across Senegal, Mauritania, and Gambia. Existing multilingual LLMs (Gemini, GPT-4o, Gemma, Qwen) are all weak in Wolof because it represents a tiny fraction of their training data. This project explores what a from-scratch, Wolof-first small model can achieve when the whole pipeline — tokenizer, architecture, data, training recipe — is designed around the constraints of a genuinely low-resource language, instead of Wolof being an afterthought in a 100+-language mix.

Architecture

Parameters	16.5M
Layers	8
Hidden size	320
Attention	Grouped-Query Attention (8 heads, 2 KV heads) + RoPE
Feed-forward	Sparse MoE, 4 experts, top-2 routing, SwiGLU
Vocab size	8,192 (dedicated BPE tokenizer, not reused from another model)
Context length	512 tokens
Embeddings	Tied input/output

Design choices, and why:

• Dedicated tokenizer. Reusing Qwen's tokenizer on this corpus measured a fertility of only 2.46 chars/token (vs ~4.3 for French) — Wolof text was being fragmented far more than necessary. A tokenizer trained on Wolof itself improves this to 2.92 chars/token with a 6x smaller vocab (8,192 vs ~150K), which matters a lot when the whole model is 16.5M parameters.
• Sparse MoE, not dense. Data-constrained scaling research (Muennighoff et al., 2023; "When Data Is Scarce") shows sparse architectures tolerate multi-epoch training on repeated data better than dense ones — directly relevant here since this model saw its ~9.7M-token corpus 18 times, far below the Chinchilla-optimal token budget for this parameter count.
• Explicit load-balancing loss on the router. Recent work on cross-lingual MoE routing (Bandarkar et al., 2026) found that low-resource language tokens tend to collapse onto a narrow subset of experts in naively-trained MoE models. The auxiliary load-balancing loss here forces the router to spread Wolof tokens across all 4 experts instead of letting a narrow pathway emerge.

Training data

~9.7M tokens, unified from multiple sources and deduplicated (exact + near-duplicate via SimHash): Wikipedia (Wolof), FineWeb2, MADLAD-400, SMOL-doc, plus smaller lexicon/instruction sources (Jolof, Aya, Gatitos). GSM8K (7,473 grade-school math problems), machine-translated to Wolof and quality-checked, is folded directly into the pretraining mix rather than kept as a separate SFT stage — at this parameter budget, a strict CPT-then-SFT split wastes capacity; mixing lets the model see the reasoning pattern throughout training.

Recipe: 18 epochs (chosen per data-constrained scaling literature: repeating data up to ~4 epochs is nearly free, meaningful gains extend to ~16, returns vanish around 40), AdamW, cosine LR schedule with warmup, bf16 mixed precision. Final validation loss: 3.48 (perplexity ≈32.5). Train/val loss divergence after epoch ~13 indicates mild overfitting, consistent with the token budget being well below what this parameter count would ideally need.

Usage

from transformers import AutoModelForCausalLM, AutoTokenizer

model = AutoModelForCausalLM.from_pretrained("Fallovski/smallm-wolof", trust_remote_code=True)
tokenizer = AutoTokenizer.from_pretrained("Fallovski/smallm-wolof")

inputs = tokenizer("Sama xarit", return_tensors="pt")
output = model.generate(
    **inputs,
    max_new_tokens=100,
    do_sample=True,
    temperature=0.85,
    top_k=40,
    repetition_penalty=1.3,  # important: without this the model loops on short/common prompts
)
print(tokenizer.decode(output[0], skip_special_tokens=True))

trust_remote_code=True is required — this is a custom architecture, not a standard transformers model class.

Limitations

Read this before drawing conclusions from the model's output.

• ~9.7M training tokens is very small. For comparison, Chinchilla-optimal scaling would want ~330M tokens for a 16.5M-parameter model. The model has learned Wolof surface grammar and the style of the corpus (including GSM8K-style "step-by-step, then #### answer" formatting) convincingly, but arithmetic in generated reasoning chains is frequently wrong — it has learned the form of reasoning, not reliable calculation.
• Repetition loops. Short or generic prompts (e.g. single common nouns) can cause the model to loop on a phrase. Use repetition_penalty >= 1.3; even then, some prompts still degrade after a few dozen tokens.
• No independent native-speaker validation of the training corpus. Part of the pretraining data (FineWeb2-derived) mixes standard Wolof orthography with phonetic/social-media spelling, and a portion of the GSM8K translation was produced by Gemini 2.5 Flash rather than human translators — spot checks looked reasonable but no systematic quality audit has been done.
• 512-token context, no long-context capability.
• Not intended for factual reliance, translation accuracy, or any production use case.

Files

• modeling_smallm_wolof.py / configuration_smallm_wolof.py — custom architecture code (loaded via trust_remote_code=True)
• model.safetensors — trained weights
• tokenizer.json — dedicated BPE tokenizer

Citation

If you build on this, please note it is a research/educational project, not a peer-reviewed publication. No formal citation is required, but a link back to this repository is appreciated.