Dariachup/dormouse-mt-en-uk

dormouse MT — English → Ukrainian (fine-tuned, v0.7)

Fine-tune of Helsinki-NLP/opus-mt-en-uk (76M params, runs on CPU). Closes the loop for dormouse-ua: the English answer from a cloud or local LLM comes back as fluent Ukrainian, without the LLM generating a single Cyrillic token itself.

Training corpus (v0.7) — 282 945 pairs

source	pairs	what it is
inv	158 521	real Ukrainian chat (Telegram, Threads), inverted
native	97 912	native English chat (hh-rlhf, oasst1), UA side MT
cs	24 702	customer-support bitext, UA side MT
surzhyk	1 810	synthetic surzhyk injected into real sentences

The native block is what v0.6 was missing: earlier versions were trained almost entirely on English that had itself been produced from Ukrainian originals ("translationese"), which is a poor model of what an LLM actually writes.

Metrics (v0.7, 845 held-out pairs, sacrebleu)

slice	n	base BLEU	ft BLEU	base chrF	ft chrF
overall	845	21.12	40.28	44.25	62.90
cs *	250	27.49	68.15	49.35	82.49
native *	250	26.93	49.43	49.75	68.97
inv	250	9.70	19.97	33.97	45.26
surzhyk	95	9.47	14.16	36.55	42.43

Read the starred slices with suspicion

* — on the cs and native slices the Ukrainian reference side was itself produced by gemini-2.5-flash, not by a human. A 68 BLEU there means "the fine-tune reproduces Gemini's Ukrainian well", which is what it was trained to do. It is not evidence of human-level quality.

The honest slices are inv (real Ukrainian chat with human references) and surzhyk. Read those first: 19.97 BLEU / 45.26 chrF, roughly double the base model's 9.70 / 33.97.

That also corrects a tempting misreading. The overall en→uk number (40.28) is now higher than the uk→en sibling model (38.46), which looks like the reverse direction caught up — it did not. Strip the synthetic-reference slices and the picture is unchanged: 31.32 vs 19.97. Generating Ukrainian morphology is still the harder half of the round trip.

Known failure modes

• domain-shift nouns from outside chat: brass → мідь, Necklaces → краватки, ring → обручка
• ти/ви mixing inside one reply — the training chat data is inconsistent about register, and the model inherited that

Usage

from transformers import MarianMTModel, MarianTokenizer
m = MarianMTModel.from_pretrained("Dariachup/dormouse-mt-en-uk")
t = MarianTokenizer.from_pretrained("Dariachup/dormouse-mt-en-uk")
ids = t("So what's up with the order, three days silent", return_tensors="pt")
print(t.decode(m.generate(**ids, num_beams=4)[0], skip_special_tokens=True))

Translate one sentence at a time — MarianMT degrades noticeably on whole multi-line blocks.