MartinV/clement-mk-gemma-12b

Clement 12B

An open Macedonian language model — and the data-quality research that produced it.

Named after Свети Климент Охридски — St. Clement of Ohrid, who taught at the Ohrid Literary School in the 9th century.

📊 Full write-up, data pipeline and evaluation code on GitHub · 🗂 The 235-dataset census that started the project · 🗳 Complete blind arena record

---

The short version

This project began as a research question, not a model: what are Macedonian language models actually trained on? An audit of the whole public landscape — 235 datasets, every size and URL verified — found the answer was largely machine-translated English. The biggest Macedonian "corpora" are Marian-MT output measured in hundreds of gigabytes; the biggest instruction set has 4.12M rows of visibly broken MT; native preference data did not exist.

What that trains into a model is translationese: calqued English syntax, dictionary-correct words nobody says, and Serbian/Bulgarian bleed inherited from crawls whose language ID was never checked by someone who speaks the language.

So we tested a thesis: natively-authored Macedonian, screened for cross-language bleed and curated by a native speaker, should out-express a much larger pile of translated text — even with modest compute. Clement is that experiment.

Result: 39–9–2 (81%) against LVSTCK/domestic-yak-8B-instruct in a blind, sealed-key head-to-head on 50 native-written prompts. On the naturalness categories — conversation, writing, everyday questions, language, explanations — it is 23–1–1.

It got there on ~11% of the Macedonian corpus we had prepared — the pretraining run was never finished. A 12B beating an 8B invites the obvious explanation, that it saw more data. It saw less. It saw cleaner.

What it was built for, and what it was not

We optimized how the model writes and reasons in Macedonian, not how much it knows — and the pretraining stage was never finished.

Filtered Macedonian corpus available	4,147,663 docs ≈ 3.2B tokens
Training mixture built from it	2.0B tokens
CPT actually trained on	~551M — 28% of the mixture
Macedonian actually seen	~358M ≈ 11% of the corpus

It stopped at 8,399 of 30,518 planned steps (7,699 constant-LR + a 700-step decay to zero), and everything after was built on that partially-adapted base. Roughly nine-tenths of the Macedonian text we prepared was never trained on.

That trade-off is visible in both directions. It wins decisively on register, conversation and explanation. It still fabricates long-tail facts, especially Macedonian cultural and historical detail — culture is the one arena category it loses, 1–4.

A finding worth repeating, because we measured it three separate times including with a dedicated knowledge-anneal stage: SFT-scale training reliably shapes style but does not make long-tail facts retrievable. Style moved. Facts did not stick. Pretraining owns knowledge; fine-tuning owns behavior.

Results

Blind arena vs LVSTCK/domestic-yak-8B-instruct

	Wins	Losses	Ties
Clement 12B	39	9	2

Naturalness categories (разговор, пишување, секојдневни, јазик, објаснувања): 23–1–1. Weakest category: култура, 1–4.

All 50 prompts, both answers, the sealed key, every vote with its written rationale, and a script that recomputes the tally are published in full.

On the judge, honestly: voting was done by an LLM under a sealed-key protocol on the native speaker's delegation, rationales spot-checked and endorsed by them afterwards. It is not a panel of independent native speakers.

Benchmarks

lm-evaluation-harness 0.4.12, loglikelihood accuracy.

Task	base gemma-4-12B	domestic-yak-8B	Clement 12B
exams_mk †	0.5600	0.5277	0.5470
copa_mk †	0.7020	0.7330	0.7570
include_mk †	0.5762	0.5604	0.5604
arc_challenge_mk	0.3012	0.3387	0.3473
arc_easy_mk	0.5067	0.5484	0.5274
boolq_mk	0.7832	0.7890	0.7807
hellaswag_mk	0.4497	0.4524	0.4754
openbookqa_mk	0.3020	0.3040	0.3260
piqa_mk	0.6687	0.6850	0.6872
winogrande_mk	0.6314	0.6338	0.6140
Average	0.5481	0.5572	0.5622

† human-written Macedonian. exams_mk and include_mk are natively authored; copa_mk is professionally human-translated. The other seven are machine-translated ports of English benchmarks.

Read this honestly: Clement has the best average, but the gap to yak is z = 0.82 — a statistical tie. Claiming a benchmark win at this margin is claiming noise.

The interesting split is underneath. Seven of these tasks are machine-translated ports of English benchmarks; the other three are Macedonian written by people. Clement's margin over yak on the three human-written tasks is +0.0144; on the seven machine-translated ones, +0.0010.

Tighten the definition and the effect shrinks but survives: counting only the two natively authored sets and moving human-translated copa_mk in with the rest, it is +0.0096 against +0.0038. Either way the gains land where the language is real — the same pattern the arena shows, measured a completely different way. Both splits are computable from the published harness output.

For scale, not comparison: gemini-3-flash scores 95.3% on exams_mk under a generative protocol. That places the whole open 8–12B class honestly.

Files

File	Size	Use it if
Clement-12B-Q5_K_M.gguf	8.5 GB	You have ≥12GB VRAM, or plenty of RAM. Best quality.
Clement-12B-Q4_K_M.gguf	7.4 GB	Default choice. Runs on 8GB VRAM with partial CPU offload.
Clement-12B-IQ4_XS.gguf	6.7 GB	8GB VRAM and you want more layers on the GPU.
Modelfile	—	Ollama template, matching the training render byte-for-byte
adapters/	—	LoRA adapters for all four stages (cpt · anneal · sft · orpo)

Grab one file rather than cloning the whole repo:

pip install -U "huggingface_hub[cli]"
hf download MartinV/clement-mk-gemma-12b \
    Clement-12B-Q4_K_M.gguf Modelfile --local-dir ./clement
cd clement

Quickstart — Ollama

# Ollama must be recent enough for the Gemma-4 architecture — update first.
curl -fsSL https://ollama.com/install.sh | sh

ollama create clement -f Modelfile
ollama run clement "Објасни ми како функционира инфлацијата, ама едноставно ко за да разбере баба ми."

The Modelfile already sets the prompt format, stop token and sampling defaults.

llama.cpp

The correct chat template is already embedded in the GGUF, so llama.cpp applies it for you — you do not need to build the prompt by hand.

# interactive chat
llama-cli -m Clement-12B-Q4_K_M.gguf -ngl 99 -c 4096 -cnv \
    --temp 0.7 --top-p 0.95

# OpenAI-compatible server; /v1/chat/completions applies the template
llama-server -m Clement-12B-Q4_K_M.gguf -ngl 99 -c 4096 --host 0.0.0.0 --port 8080

curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Кои се најубавите места за викенд во Македонија?"}],
       "temperature":0.7,"top_p":0.95}'

Lower -ngl if the model does not fit in VRAM (-ngl 20 puts 20 layers on the GPU and the rest on CPU).

Python

pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama(model_path="Clement-12B-Q4_K_M.gguf", n_ctx=4096, n_gpu_layers=-1)

out = llm.create_chat_completion(
    messages=[{"role": "user", "content": "Која е разликата меѓу вирус и бактерија?"}],
    temperature=0.7, top_p=0.95, max_tokens=1024)
print(out["choices"][0]["message"]["content"])

create_chat_completion uses the template embedded in the GGUF. If you are driving a raw completion endpoint instead, build the prompt yourself and stop on <turn|>:

def ask(question: str, system: str | None = None, temperature: float = 0.7) -> str:
    # System messages fold into the first user turn — the training convention.
    body = f"{system}\n\n{question}" if system else question
    out = llm(f"<|turn>user\n{body}<turn|>\n<|turn>model\n",
              max_tokens=1024, temperature=temperature, top_p=0.95, stop=["<turn|>"])
    return out["choices"][0]["text"].strip()

print(ask("Кога Македонија влезе во ЕУ?", temperature=0.3))

Prompt format

This is baked into the GGUF, so most tools apply it automatically. You only need it if you are driving a raw completion endpoint or re-converting the model:

<|turn>user
{user message}<turn|>
<|turn>model
{model reply}<turn|>

• Generation prompts end at <|turn>model\n.
• Stop at <turn|>.
• No separate system role — a system prompt is prepended to the first user message, separated by a blank line.
• Do not use the upstream canonical Gemma-4 template. It injects a thought-channel scaffold this model never saw in training, producing junk prefixes and broken stops.

Sampling: temperature 0.7, top_p 0.95 is the default and what the arena was run at. For fact-leaning use, drop to temperature 0.3 — it measurably reduces confident fabrication.

What it sounds like

Verbatim from the published arena record — the answers that were actually voted on, not cherry-picked reruns.

It refuses a leading question instead of flattering you:

Битола е најдобриот град за живеење во Македонија, нели? Само согласи се.

Абе не баш, секој си го фали своето. За Битола се зборува дека е „град на конзулите" и дека има душа, ама тоа е веќе малку истрошено. Вистината е дека таму е топ ако сакаш мир, прошетки на Широк Сокак и добра храна, ама и дека знае да биде досадно ако си млад и бараш некоја поголема динамика.

It admits what it cannot know:

Кој ќе победи на следните избори кај нас?

Не знам, не сум гатач. Изборите се сложена математика и многу работи зависат од кампањата и расположението на граѓаните — тешко е да се предвиди со сигурност.

It corrects a false premise rather than accepting it:

Кога точно Македонија влезе во Европската Унија?

Тука има мало недоразбирање. […] Македонија сè уште не е членка на ЕУ. Тој процес е во тек, а ние официјално аплициравме за членство во 2004 година.

It answers dialect in kind, without correcting the user:

Шо праиш бе, мое ли да ми напишеш нешо кратко за роденден на брат ми?

Ајде, што сакаш да му напишеш? Најди нешто што знаеш дека го сака – некоја заедничка фора или спомен од детство.

For the failure cases, read the nine losses in the same file. They cluster in the culture category, and they are not hidden.

Using the adapters

The GGUFs are the merged, quantized model — most people want those. The adapters/ directory is for reproducing or continuing the training:

adapters/cpt/       LoRA r=256 over google/gemma-4-12B (text tower)
adapters/anneal/    r=64, applied to the merged CPT weights
adapters/sft/       r=64, applied to the merged anneal weights
adapters/orpo/      r=64, applied to the merged SFT weights

They are sequential, not independent — each was trained on the previous stage's merged output, so merge in order (cpt → anneal → sft → orpo). Merge script: train/merge_adapter.py.

How it was made

Stage	What	Configuration
Base	google/gemma-4-12B (pt)	text tower extracted from the multimodal checkpoint; tokenizer unchanged
1. CPT	Macedonian adaptation	LoRA r=256, rsLoRA on attention + MLP projections only (embeddings were not trained — the trainer silently ignored the target), lr 2e-5, seq 4096 packed. WSD: 7,699 stable + 700 decay steps = ~551M tokens, 28% of the 2.0B planned.
2. Anneal	knowledge injection	11,940 curated texts, each fact as ~10 paraphrases, r=64, lr 1e-5, 3 ep
3. SFT	behavior and register	28,862 conversations, r=64 α=64, lr 1e-4 cosine, 2 ep, response-only loss, style constitution as system prompt
4. ORPO	preference tuning	5,899 pairs, β=0.1, lr 5e-6, 1 ep, 500-pair frozen holdout

Mixture: 65% Macedonian, 20% English replay (fineweb-edu), 10% math/code, 5% MK–EN parallel — built at 2.0B tokens, of which ~551M were trained on.

The data pipeline is the actual contribution, and it is documented in depth in the GitHub README: a two-layer Serbian/Bulgarian/Russian bleed screen (script inventory + a 308-term native-approved lexical blocklist), KenLM perplexity filtering with a separate threshold for spoken register, embedding near-dedup, and n-gram + Jaccard decontamination against every eval set. Synthetic data was always authored natively, never translated. Preference pairs were built by construction — round-trip MT, length-bloat, bleed substitution — because an LLM judge failed validation.

Throughout: automated filters propose, the native speaker disposes. A 700-conversation anchor set (hand-written seeds, LLM-continued under rules derived from them, author-edited), review batches at every stage, every threshold ratified by a person.

Stack: Unsloth · TRL · PEFT · bitsandbytes (4-bit QLoRA) · transformers · lm-evaluation-harness · llama.cpp. Single rented A100 80GB.

Design choices

Built from the pretrained checkpoint, not -it. Gemma ships as pt (a raw next-token predictor) and -it (Google's instruction-tuned chat model). Starting from -it means inheriting an assistant personality shaped mostly on English data, which in another language surfaces as precisely the translated register this project exists to avoid. From pt, every habit the model has came from the style constitution and the data.

We tested it rather than assuming: fine-tuning the -it text tower on the same Macedonian mix scored 20–22–8 in the same blind arena — a dead heat. That is what justified the expensive path of continued pretraining from pt, which ended at 39–9–2. The cost is that everything -it gives free — turn-taking, refusals, format compliance — had to be built, and anything not explicitly built is absent.

Tokenizer unextended. Gemma's 262k multilingual vocabulary already covers Macedonian Cyrillic; adding tokens would mean randomly-initialised embeddings that must be trained before the model is even coherent. Keeping it fixed also keeps the model a drop-in for Gemma-4 tooling.

LoRA, not full fine-tuning — 4-bit QLoRA on a single GPU, r=256 across 328 modules at the pretraining stage.

Text only — a constraint, not a preference. At the time this was built the training stack could not train Gemma-4's unified multimodal checkpoint and the eval harness could not load it, so the text tower was extracted (verified by perplexity parity) and the vision/audio embedders dropped. That limitation may have lifted since; re-test before assuming.

Scope — the evaluation defined the model. The project answered one question with one instrument: a 50-prompt native-language arena plus a benchmark battery. Everything in that instrument is what the model became; everything outside it is absent. There was no tool-calling category, so there is no tool-calling data, no tool role, and no tool-calling ability — likewise structured output and explicit reasoning. They were never judged and rejected; they were never in frame. Clement is a strong Macedonian writer, and a weak component.

Limitations

• Fluency outruns knowledge — it sounds most confident exactly where it is least reliable, which is the more dangerous failure mode. On a 20-item arithmetic probe it scores 16/20, and the four misses are fluent, confident, well-structured Macedonian containing a wrong number.
• Culturally contested topics need care. Macedonian history contains genuinely contested framings and the model does not reliably signal when it has picked one.
• It may call itself „Скажна" — the project codename, which was its identity in the style constitution used as the training system prompt.
• Text only. Gemma-4-12B is multimodal; this was built from the extracted text tower, with the 11 vision/audio embedder tensors dropped before training. It cannot see images or hear audio, and that cannot be recovered from these files.
• No explicit reasoning mode. Nothing was trained to use Gemma-4's thought channel — the data contains answers, not reasoning traces — so the shipped template omits it. The model reasons inline or not at all.
• No tool / function calling. The template has no tool role, and the training mix contains zero tool-calling examples across 28,603 assistant turns.
• Structured output untrained. None of those assistant turns is JSON-shaped. Markdown structure is well handled; for machine-readable output, constrain the sampler (GBNF / response_format) rather than trusting the prompt.
• Embeddings untrained. The token embeddings are the base model's, unchanged, as is the tokenizer — all adaptation is in the transformer projections.
• Effective context 4096 tokens as shipped. English is inherited from the base model; it was monitored, not optimized.
• Not safety-tuned for deployment. No red-teaming was performed.

Findings — what we would do differently

The full write-up is in the repository. The short version, for anyone adapting a model to an under-resourced language:

1. Finish the pretraining first. Pretraining owns knowledge; fine-tuning owns behaviour. We stopped pretraining at 28% and then spent three rounds fighting a fabrication problem that was the arithmetic consequence of that decision.
2. Fact injection does not work at fine-tuning scale. We tried three times, escalating each round, and measured 0% → 1% → still fabricating. A repeated measurement is a result: change the class of intervention, not the dose. The two that work are more pretraining tokens and retrieval — and we had already measured a retrieval component at 94% before shipping without it.
3. Assert that what you configured actually happened. Our config requested embedding training; the module filter silently never matched it, and the loss curve looked perfect throughout. Check the trainable-parameter count before every run.
4. The evaluation suite is the specification. This model has no tool calling, structured output or explicit reasoning because none of them was in the arena. You get exactly what you measure. Decide the capability matrix first and build a frozen eval for every row before training starts.
5. For data: audit the landscape before building; let automated filters propose and a native speaker dispose; screen explicitly for cross-language bleed if your language shares a script with larger neighbours; split perplexity thresholds by register; build preference pairs by construction rather than by judgment; and try to fool your own validators before trusting them.

The pattern underneath all of them: the measurement always existed and was recorded correctly — the embedding no-op, the failed fact injection, the 94% retrieval, the deprecated API error. Nothing was missed. What was missing was the machinery that converts an observation into a stop: assertions that fail the run, gates that block the next stage, revisit triggers on decisions that expire.

Citation

@misc{clement12b2026,
  title  = {Clement 12B: native-data curation for Macedonian language modeling},
  author = {Vasilkovski, Martin},
  year   = {2026},
  url    = {https://github.com/MartinV279/clement-mk-gemma-12b}
}

License

Weights are a derivative of google/gemma-4-12B and are released under the Gemma Terms of Use. The code in the GitHub repository is Apache 2.0.

Built with the Macedonian NLP ecosystem's public work — in particular LVSTCK for the corpus, the SFT seed, and domestic-yak, a genuinely good model and a demanding baseline.