EryriLabs/gemma-4-e4b-cymraeg-v4

Gemma 4 E4B Cymraeg v4

Welsh on the edge

A conversational Welsh fine-tune of Google's Gemma 4 E4B, the small "effective 4B" model built for phones, laptops, Raspberry Pi class boards and anything else that has to work with no internet. The GGUF builds run in llama.cpp and LM Studio in about 5 to 9 GB of memory. Welsh should not need a data centre, and with this model it doesn't.

It speaks natural, everyday Welsh (correct mutations, consistent ti/chi register, idiomatic vocabulary) and keeps its English, so it answers in whichever language you use. This repository holds the full merged bf16 weights, ready for from_pretrained, plus the two LoRA adapters that training produced. Quantised GGUF files for edge deployment are in EryriLabs/gemma-4-e4b-cymraeg-v4-GGUF.

Trained on open Welsh data, most of it released by techiaith at Bangor University and by the Welsh public sector. Diolch o galon to everyone who put that data out under open licences; this model exists because they did.

Intended use

General-purpose Welsh and English conversation, drafting and question answering.

Commercial use is permitted for the base model and the named sources below. google/gemma-4-E4B-it is Apache-2.0. The continued-pretraining sources are all commercially usable under their own licences (ODC-By 1.0, CC-BY-4.0, OGL v3, CC0), and the fraud-awareness component of the instruction data is Crown copyright under the Open Government Licence v3.0, which also permits commercial use. Reproduce the attributions below where a licence requires it; see DATA-CARD-v4.md for the full source list.

Not intended for: legal, medical or financial advice; or any setting where an incorrect Welsh translation, or an incorrect fraud-related fact, carries real-world risk without human review. Fraud-related facts (contact numbers, organisation names, reporting routes) should be supplied through the system prompt or retrieved context rather than relied on from the model's own memory , see README-stopthinkfraud.md.

How to run

from transformers import AutoTokenizer, Gemma4ForConditionalGeneration

REPO = "EryriLabs/gemma-4-e4b-cymraeg-v4"
tok = AutoTokenizer.from_pretrained(REPO)
model = Gemma4ForConditionalGeneration.from_pretrained(
    REPO, dtype="bfloat16", device_map="auto")
msgs = [{"role": "user", "content": "Beth yw prifddinas Cymru?"}]
ids = tok.apply_chat_template(msgs, add_generation_prompt=True,
                              enable_thinking=False, return_tensors="pt")
print(tok.decode(model.generate(ids.to(model.device), max_new_tokens=400)[0]))

Load with the class this checkpoint's config.architectures names, which is Gemma4ForConditionalGeneration (model_type: gemma4): not the unified Gemma4Unified... class used by the 12B. Loading E4B through the unified class does not raise: transformers builds the wrong architecture, silently discards the checkpoint tensors that do not fit it, and returns a partly-random model that still produces confident, fluent-looking text. If you load it yourself, check the loading info for missing or unexpected keys rather than trusting that it worked. transformers >= 5.15 is required; earlier releases have no gemma4 model type.

Keep thinking off. Gemma 4 has a thought channel; leaving it enabled lets some clients spend the whole token budget reasoning and return an empty answer. This model was trained and evaluated with thinking off.

Recommended sampling (LM Studio / llama.cpp):

setting	value
temperature	0.3
top_p	0.9
min_p	0.05
repeat_penalty	1.05

Training summary

Two-stage LoRA training: continued pretraining on raw Welsh, then supervised fine-tuning on Welsh instructions.

Stage 1: continued pretraining (CPT)

corpus built	85,269 documents / 78,002,622 tokens (see DATA-CARD-v4.md for per-source counts and corpus_report.json for drop counts)
corpus trained on	43,835,392 tokens: a uniform random subset of the above, drawn after a global shuffle so every source is represented in proportion
method	LoRA, rank-stabilised scaling (rsLoRA)
LoRA	r=128, alpha=32, dropout 0
targets	q, k, v, o, gate, up, down projections
embeddings	embed_tokens trained in full via modules_to_save (lm_head is tied to it in this architecture, so the two are one matrix). 950M of 6.9B parameters trainable: 279M LoRA + 671M embedding
learning rate	1e-4 (LoRA weights), 1e-5 (embedding matrix), separate optimiser groups
schedule	cosine, 2% warmup, 668 optimiser steps
sequence length	2048, raw-only blocks, labels = input_ids
batch	32 x 1 (no gradient accumulation)
precision	bf16, 4-bit base (QLoRA)
epochs	1

The trained subset is smaller than the built corpus because the run was capped by a fixed compute budget, not because the remainder was unusable. Sizing was done from a measured 20-step benchmark on the training GPU (6,100 tokens/s sustained, 48.2 GB peak VRAM) rather than an estimate, and the corpus was shuffled before the cap was applied so the subset is representative rather than a truncation of the source order. The full 78M-token corpus and its build report ship with this repository.

Stage 2: supervised fine-tuning (SFT)

data	19,443 Welsh instruction rows (~4.8M tokens) mixed with a halved raw replay slice: 3,586 documents / 2,500,523 tokens sampled from 7,202 / 5,000,299: against a 370-row validation split. See DATA-CARD-v4.md and DATA-CARD-v4-sft.md
mix ratio	~2:1 instruction to raw. The raw share was reduced after the sibling 12B v3 run, whose stage 2 was raw-heavy (14.3M raw against 4.2M SFT), was judged to have regressed on Welsh with repetition collapses and content-free brevity: the signature of a mix that teaches continuation rather than answering
base	the stage-1 CPT-merged checkpoint, not the stock model
method	LoRA (embedding matrices not trained in this stage)
LoRA	r=64, alpha=64
learning rate	8e-5
schedule	cosine, 3% warmup
sequence length	2048
epochs	1
batching	homogeneous batches (raw blocks and chat rows never share a batch); final 20% of batches are instruction-only, so the low-learning-rate end of the schedule is spent entirely on answering

Both adapters are shipped in adapters/cpt/ and adapters/sft/. The merged bf16 weights at the root of this repository incorporate both, applied in training order (CPT, then SFT).

Evaluation

Blind LLM-as-judge scoring (Opus), identical prompts and settings for every arm (nf4, thinking off, greedy, 400 max tokens, no system prompt). The judge is not a native speaker; structural findings (language choice, loops, checkable facts) are reliable, fine idiom judgements less so: native review welcome.

Single-turn, 40 Welsh prompts

model	replies in the requested language	fluency /5	grammar /5	task /5
stock gemma-4-E4B-it	47.5 %	2.46	2.38	1.45
v3 (SFT only)	97.5 %	3.87	3.59	2.33
v4 (CPT 43.8M tokens + SFT)	100 %	3.68	3.76	2.70

Pairwise v4 vs v3 (blind): 23 wins, 13 ties, 4 losses. English is fully retained (fluency 5.0, task 4.95: best in the family). Invented Welsh words roughly halved versus v3; the corpus-installed mutations (i Gaerdydd, i Fangor, dy fore di) are v3's exact failures now correct.

Multi-turn (14 fixed turns, history kept, sampling as recommended above)

model	conversation score /5
stock	1.97
v3	3.35
v4	3.99

Quantisation: Q6_K scored highest of the three quants under llama.cpp (fluency 4.42, grammar 4.36); Q4_K_M showed no measurable damage; all three quants: 0 corrupt tokens on the corruption-prone subset.

Known limitations: occasional repetition loops on long open-ended answers (mitigated by repeat_penalty); factual recall (dates, statistics) is weak, supply facts via the system prompt or retrieval; when given long retrieved context it can over-quote it. Fraud-safety facts must not be relied on from memory. Full judgement report and per-response scores: eval/judgement_e4b_v4.md, eval/scores_e4b_v4.jsonl.

A head-to-head against BangorAI/Mistral-7B-Cymraeg-Welsh-v2 on the same protocol is in the section below.

Raw outputs and training logs are in eval/. GGUF corruption counts per quantisation are in the GGUF repo's MANIFEST.json.

Head-to-head with BangorAI Mistral-7B-Cymraeg-Welsh-v2

BangorAI/Mistral-7B-Cymraeg-Welsh-v2 is the strongest open Welsh model we know of and the only open comparison in the 7B class, so it is the bar this model is measured against. It is a genuinely good Welsh model, and everything below should be read as a measurement against a high bar rather than as a criticism of it. Diolch to BangorAI for publishing it.

It is not a like-for-like comparison, and it is not meant to be:

	this model (v4)	BangorAI 7B v2
parameters	6.9B total, ~4B effective per token	7B dense
Welsh continued pretraining	43.8M tokens, 1 pass	MADLAD Welsh, 2 epochs, on the order of 412M token-passes
ratio	1x	~9.4x the Welsh tokens, ~1.75x the active parameters

BangorAI was run with its own documented Welsh system prompt, which is its intended interface, in the single-turn arm and in the multi-turn sys arm. The multi-turn nosys arm is the properly matched comparison. Sampling, quantisation, prompts and token limits were identical throughout.

Single-turn, 40 Welsh prompts plus 20 English prompts

arm	replies in the requested language	Welsh fluency /5	Welsh grammar /5	task /5	collapses	English task /5
v4	40/40	3.68	3.76	2.70	5	4.95
BangorAI 7B v2	37/40	4.85	4.65	3.35	0	4.20

Pairwise on the same 40 prompts: BangorAI 23 wins, 13 ties, v4 4 wins. That is the mirror image of v4 against v3. BangorAI's Welsh essentially never breaks: zero degeneration loops and zero corrupt word-forms anywhere. v4's four wins are the three prompts that ask for English output, which BangorAI answers in Welsh, plus one conversational prompt. v4 leads on task in translation (3.50 against 2.00) and assistant chat (3.43 against 2.86); BangorAI leads everywhere else.

Multi-turn, 14 fixed turns with history kept

arm	fluency	grammar	relevance	conversational	memory	mean
v4 / sys	4.07	4.14	3.21	3.36	3.69	3.70
v4 / nosys	4.29	4.14	3.79	3.86	3.85	3.99
BangorAI / sys	4.79	4.93	3.36	3.00	2.92	3.80
BangorAI / nosys	4.64	4.93	3.21	2.93	3.00	3.74

Per-turn wins, ties and losses:

condition	v4	tie	BangorAI
sys vs sys (each with its own intended prompt)	5	2	7
nosys vs nosys (matched)	5	4	5
combined, 28 turns	10	6	12

v4 wins the turns that hold a conversation together: the opening greeting, the summary, the close (both v4 conditions produce an eleven-turn callback, "Mwynha dy daith i Aberystwyth", which no BangorAI condition does), and above all turn 11, the request to switch to English. v4 switches cleanly and translates faithfully in both conditions; BangorAI stays in Welsh in both, and under its own system prompt it states in Welsh that it can only help in English. BangorAI wins the turns that reward content and polish: the tips, the facts, the joke, the fraud advice and the poem.

Verdict in three lines. BangorAI wins overall and wins Welsh polish decisively: it is more fluent, more grammatical and never degenerates, and if you want the best open Welsh output and can run a 7B, use it. The conversation is far closer, and in the matched condition v4 takes it on the arm mean (3.99 against 3.74) and draws on turns, so roughly 4B effective parameters trained on a tenth of the Welsh fights a 7B to a draw over fourteen turns. What the extra scale and data buy is surface quality and degeneration resistance; what they do not buy is memory, register mirroring or switching language on request, which is why a small bilingual model still has a place on edge devices.

Caveats. The judge is an LLM (Opus) and not a native Welsh speaker; structural findings are reliable, fine idiom judgements are not, and a non-native judge is least reliable exactly where BangorAI wins most, so the fluency gap is more likely understated than overstated. Sizes and Welsh token counts are not matched. BangorAI ran through its own system-prompt interface, so only the multi-turn nosys comparison is matched on prompt text. The rubric was written for a friendly conversational model, which favours v4 on register and warmth. Sample sizes are small. Full report and per-item scores: eval/judgement_v4_vs_bangorai.md and eval/scores_v4_vs_bangorai.jsonl in the base repository.

Limitations

• Welsh costs more tokens per word than English for the Gemma 4 tokeniser, which reduces the effective context length available for Welsh text relative to English.
• Some of the instruction data was generated or adapted by an automated pipeline rather than written by a native speaker; treat Welsh fluency as unverified for anything that will be published verbatim.
• Fraud-related facts (numbers, organisation names, reporting routes) can be misremembered or go stale. Supply them through the system prompt or retrieved context rather than relying on the model's memory: see README-stopthinkfraud.md.
• GGUF quantisation introduces measurable degradation at lower bit-widths; read the corruption counts in the GGUF repo's MANIFEST.json before choosing a quantisation below Q6_K.
• Stage 1 was a single epoch over 43.8M tokens, not the full 78M-token corpus that was built (the run had a fixed compute budget). That is enough to move the embedding table and the adapters, but it is a modest amount of continued pretraining and its effect should not be overstated: compare against the v3 arm in eval/ rather than assuming an improvement.
• No tool-calling, no vision use, and no guarantee of factual accuracy.

Repository contents

path	contents
repo root	merged bf16 weights, ready for from_pretrained
adapters/cpt/	stage 1 continued-pretraining LoRA adapter
adapters/sft/	stage 2 supervised fine-tuning LoRA adapter
eval/	raw evaluation outputs and training logs
DATA-CARD-v4.md	full corpus description, per-source counts, filter chain, contamination control
DATA-CARD-v4-sft.md	stage-2 instruction-set composition
corpus_report.json	machine-readable corpus statistics and drop counts
eval/licence_exclusion_report.json	the 447 instruction rows excluded pending licence clarification
README-stopthinkfraud.md	provenance and licence of the Stop! Think Fraud Welsh fraud-awareness data used in training

Attributions

Base model google/gemma-4-E4B-it (trained from the unsloth/gemma-4-e4b-it mirror), Apache-2.0.

Continued-pretraining (raw) sources

source	licence	commercial use	tokens
HuggingFaceFW/fineweb-2 (cym_Latn subset)	ODC-By 1.0	yes	30,729,757
techiaith/cardiff-university-tm-en-cy (Welsh side)	CC-BY-4.0	yes	20,000,703
V3 raw corpus (fineweb-2 + banc-trawsgrifiadau-bangor documents carried forward)	ODC-By 1.0 / CC0	yes	14,271,086
techiaith/llyw-cymru-cy-ogl	OGL v3	yes	8,000,044
techiaith/cofnodycynulliad_en-cy (Welsh side)	OGL v3	yes	5,001,032

techiaith/banc-trawsgrifiadau-bangor (CC0) was not pulled directly: it is gated on the Hub and the build was refused access. It was targeted at ~1.3M tokens (1.7% of the corpus) and was not re-attempted, because passing the gate means accepting the dataset's terms. Bangor transcription material is still present: 3,321 of the 20,560 documents carried forward from the V3 corpus carry source: banc: so the register is represented but was not topped up. corpus_report.json records the failure.

Per-source row counts, drop counts and the exact filter chain are in DATA-CARD-v4.md.

Supervised fine-tuning (instruction) data

A Welsh instruction set mixed with a raw replay slice; full composition and per-source counts are in DATA-CARD-v4.md. The set includes fraud-awareness content adapted from the Welsh (/cy/) pages of the UK government's Stop! Think Fraud campaign site, © Crown copyright, published under the Open Government Licence v3.0: see README-stopthinkfraud.md for the crawl method, licence text and required attribution line.

Excluded pending licence clarification

447 rows were built and then removed before tokenisation: 427 from sft_train.jsonl and 20 from sft_val.jsonl, all of them from the macsen_intent_json and macsen_intent_conv sources. The underlying techiaith dataset states no licence, and this release claims Apache-2.0 with commercial use permitted: a row whose licence cannot be evidenced cannot sit inside that claim. They are excluded pending licence clarification rather than shipped with a caveat. Counts are in eval/licence_exclusion_report.json; the SFT set actually trained on is 19,443 train / 370 validation rows.

Licence pass-through

Where a source licence requires attribution, that requirement passes through to users of this model:

• OGL v3 sources: reproduce "Contains public sector information licensed under the Open Government Licence v3.0."
• CC-BY-4.0 / ODC-By 1.0 sources: attribute per the techiaith and HuggingFaceFW/fineweb-2 dataset cards respectively.

---

Built 2026-08-18. Merged weights, both adapters and evaluation outputs are all in this repository.