MeerDevelopment/gemma-4-31B-it-nl-lora

A Dutch LoRA adapter for google/gemma-4-31B-it. It was trained on about 14,400 synthetic Dutch task examples and makes the model better at Dutch reading comprehension, named entity recognition, grammaticality judgement, plain-language rewriting (B1, "Duidelijke Taal") and local knowledge, measured with EuroEval, the European language model benchmark.

It is worse than the base model at one thing: summarisation as scored by WikiLingua-nl. The details are below.

No benchmark data was used for training. See Data and decontamination.

EuroEval results

Benchmark: EuroEval (Dutch leaderboard, method, Dutch datasets, code), version 17.7.0, few-shot, 10 iterations per dataset, the 9 Dutch datasets of the leaderboard, each scored with the metric the leaderboard reports.

Both columns come from our own runs on the same hardware and serving setup: the base model and the base model with this adapter, both served with vLLM on the int4 quantisation Intel/gemma-4-31B-it-int4-AutoRound. The change is paired over the 10 EuroEval iterations (same few-shot examples and test items for both models); t is the change divided by its paired standard error, and |t| above about 2 is a real difference rather than noise.

Dataset	Task	Metric	Base	This adapter	Change	t	Verdict
DBRD	Sentiment	MCC	92.54	92.68	+0.14	+0.2	tie
CoNLL-nl	Named entities	micro F1 (no MISC)	79.43	84.72	+5.29	+5.0	better
Dutch CoLA	Grammaticality	MCC	52.23	54.23	+2.00	+3.2	better
SQuAD-nl	Reading comprehension	F1	79.70	86.08	+6.38	+16.6	better
WikiLingua-nl	Summarisation	chrF3++	33.86	27.57	-6.30	-3.4	worse
INCLUDE-nl	Knowledge (exams)	MCC	83.55	80.88	-2.67	-1.9	tie, leaning worse
MultiLoKo-nl	Local knowledge	MCC	26.57	29.65	+3.08	+2.3	better
Winogrande-nl	Common sense	MCC	56.43	60.70	+4.27	+1.6	tie, leaning better
Duidelijke Taal	Simplification	METEOR	54.54	58.35	+3.81	+5.5	better
Mean			62.09	63.87	+1.78

Better on 5, tied on 3, worse on 1. The raw EuroEval output for both models is in euroeval_results/.

How these numbers relate to the EuroEval leaderboard

• These are our own runs, not official leaderboard entries. EuroEval's official numbers are computed by the EuroEval team.
• Validation split. We ran EuroEval's default, which evaluates 8 of the 9 datasets on their validation split (MultiLoKo has none and uses test). The official leaderboard reports the test split, so compare our absolute numbers only with leaderboard rows marked (val). The comparison between base and adapter is unaffected, because both ran on identical items.
• Quantisation. We evaluated on an int4 quantisation of the base model. Its validation-split scores average about 0.7 points below the published bf16 google/gemma-4-31B-it test-split row; because the splits differ, that gap is only indicative.

The known regression: summarisation (WikiLingua-nl)

WikiLingua-nl scores summaries with chrF3++, which weights recall three times as much as precision, so longer summaries score higher. The base model writes summaries about 2.3 times as long as the references (often with markdown bullets). This adapter learned to write short, plain summaries of a few sentences, close to the reference length, and loses about 6 points on this metric. When its summaries are as long as the base model's, they score the same. We consider the shorter style better for a product, but if you need long summaries, use the base model or prompt for length explicitly.

Usage

The adapter applies to google/gemma-4-31B-it. We trained it on the Unsloth 4-bit checkpoint and evaluated it with vLLM on the Intel int4 AutoRound quantisation; we have not tested it on the bf16 weights.

vLLM:

vllm serve Intel/gemma-4-31B-it-int4-AutoRound \
  --enable-lora --max-lora-rank 32 \
  --lora-modules nl=MeerDevelopment/gemma-4-31B-it-nl-lora

Transformers and PEFT:

from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained("google/gemma-4-31B-it", device_map="auto", torch_dtype="auto")
model = PeftModel.from_pretrained(base, "MeerDevelopment/gemma-4-31B-it-nl-lora")
tok = AutoTokenizer.from_pretrained("google/gemma-4-31B-it")

Training

• Method: supervised fine-tuning with a LoRA on all linear layers of the language model (vision tower untouched), loss on the final assistant turn only. Rank 32, alpha 32, dropout 0.
• Optimiser: 8-bit AdamW, learning rate 1e-4 with 40 warmup steps then cosine decay, gradient clipping 1.0, effective batch 16 sequences, max length 2,048 tokens, 1 epoch (902 steps).
• Base checkpoint for training: unsloth/gemma-4-31B-it-unsloth-bnb-4bit (QLoRA, Unsloth 2026.10.7).
• Hardware: one RTX 3090 (24 GB), 9.2 hours.

Data and decontamination

The training set has 14,442 conversations, about 6.4M tokens, half of them rendered as few-shot chats. Instructions were paraphrased per item from pools of Dutch templates; EuroEval's own prompt wording was not used.

Family	Rows	Source text	Target
Extractive QA (max 3 words, some unanswerable)	1,200	Dutch Wikipedia, Dutch government websites	generated, answer must be a literal span of the passage
Named entities as JSON	1,441	Wikipedia and government sentences	generated, every entity must occur in the sentence
Grammaticality (ja/nee)	3,000	native sentences, rule-based corruptions, generated minimal pairs, hard negatives	label checked by a second, independent model
Sentiment	1,416	generated reviews in many domains (no book reviews)	label checked by a second model
Knowledge multiple choice	1,438	Dutch and Belgian Wikipedia, government text	generated
Common sense (two options)	1,426	generated scenes	label checked by a second model
Simplification (B1)	1,422	complex Wikipedia and government paragraphs	generated
Summaries	1,200	Wikipedia, government news and how-to pages	generated
Plain-label classification	300	from the families above	bare label
Replay	1,599	varied Dutch and English chat prompts	answers of the base model itself

Teachers: GLM-5.3-Flash (MIT) generated the items and DeepSeek-V4.1-Flash (MIT) independently re-checked labels, both through the Mistral API; items where the two disagreed were dropped.

Decontamination. Nothing from the 9 EuroEval Dutch datasets was used, in any split, nor from their sources (DBRD/Hebban, CoNLL-2002, Dutch CoLA, SQuAD, WikiLingua/wikiHow, INCLUDE, MultiLoKo, Winogrande, Duidelijke Taal). Every source passage and every generated item was checked against all items of all splits of those 9 datasets (18,628 items, plus the MultiLoKo source repository) with three filters: any shared normalised 13-gram, exact match on short strings, and bge-m3 cosine similarity of 0.90 or higher to the nearest benchmark item. Wikipedia pages that are the gold source of a MultiLoKo or SQuAD item were excluded as source text, and no wikiHow text was used. A final scan of the training and held-out sets found no matches.

Limitations

• Evaluated only on EuroEval Dutch, in our own runs, on an int4 quantisation; see the notes above on validation split and leaderboard comparability.
• Short summaries by default (see the regression above).
• The adapter mainly teaches task formats and Dutch writing conventions. It adds little new world knowledge; for factual questions, retrieval (for example over Dutch Wikipedia) helps far more than this adapter.
• The training data is largely machine-generated and may carry the teachers' biases and errors.
• General chat behaviour was anchored with replay data but was not separately benchmarked.

Licence

Apache 2.0, the same licence as Gemma 4. This adapter is a modification of google/gemma-4-31B-it.