Alhite/overlay-translator-nllb-int8

NLLB-200-distilled-600M, game-chat abbreviations (CTranslate2 int8)

An int8-quantized CTranslate2 build of facebook/nllb-200-distilled-600M fine-tuned to handle English game-chat abbreviations and jargon. It is the local, offline translation backend of Overlay-Translator, a Windows app that translates text you select on screen.

Revision	What it is
v0-base	Plain int8 conversion of the base model, no fine-tuning
v1-abbr	LoRA fine-tune on 102 chat abbreviations, merged and converted to int8
v2-game	This version. v1-abbr data plus a game-chat glossary (~1,280 abbreviations and terms), retrained from the base model, merged and converted to int8

Always pin a revision (tag); main follows the newest upload.

What changed, English → Russian (lines not seen in training):

Input	v0-base	v1-abbr	v2-game
idk where the vendor is, lmk if you find it	Я знаю, где продавец, я знаю, если ты найдешь его.	Я не знаю, где продавец, я точно знаю, если найдешь его.	Я не знаю, где продавец, дай мне знать, если найдешь его.
omw to the boss, wait for me	Омв к боссу, подождите меня.	Передай мне босс, подожди меня.	Отправляюсь к боссу, подожди меня.
wts epic sword, 500g, pm me	Эпический меч, 500 граммов, пожалуйста.	wts эпический меч, 500 г, отправьте мне сообщение	wts эпический меч, 500g, посылайте мне сообщение
ty all, gg wp, see you tomorrow	Всем, ГГ ВП, увидимся завтра.	Спасибо всем, gg wp, увидимся завтра.	Спасибо всем, gg wp, увидимся завтра
the weather is nice today, let's go for a walk	Сегодня хорошая погода, пойдем на прогулку.	Погода сегодня хорошая, давайте пойдем на прогулку.	Сегодня хорошая погода, пойдем на прогулку.

What it was trained to do

English game-chat terms come in two kinds:

• translate (177 abbreviations): the meaning is carried over: brb, idk, ty, tbh, lmk, omw, afaik, nvm, lfm, wtt, oom, rez ...
• keep (1,101 terms): gaming terms that players write the same way in every language, so the translation must keep them verbatim: gg, wp, glhf, afk, xd, pvp, dps, aoe, hp, cd, cc, ult, lfg, wts, wtb, op, nerf, gank, adc, rng, ilvl, bis, number suffixes like 500g, 2kk ...

In addition, 768 ordinary English words from the glossary (armor, sword, kill) were trained as plain negatives that must be translated normally, so the model does not learn to leave every short word untranslated.

Only English → de, fr, es, ru, uk, be, pl, cs, hu was trained. Other directions are unchanged from the base model.

Training

Data (23,575 training pairs):

Part	Pairs	Source
v1 abbreviation pairs	2,165	Same as v1-abbr: 102 abbreviations (74 from willwade/txt-sms-abbreviations after filtering, with gaming meanings fixed by hand, plus 28 gaming abbreviations), 398 hand-written game-chat phrases.
Game glossary pairs	15,516	A game-chat glossary written for this project (3,035 rows in 13 sections), curated into 1,944 new entries: rows whose "abbreviation" is a common English word (wordfreq Zipf ≥ 3) became plain negatives or were dropped, ambiguous and conflicting rows were dropped, and v1's validated entries won over the glossary. Per entry: the glossary example, the term alone and two short chat wrappers.
General pairs	5,894 (25 %)	FLORES-200 dev sentences with their human references, so the model keeps its general translation quality. The same sentences as in v1 are held out for evaluation.

As in v1, each pair maps the phrase with the abbreviation to a translation of the expanded phrase (lfm +10, need tank and dps → translation of looking for more +10, need tank and dps, keeping +10 and dps). The targets were translated by the base NLLB model itself (fp32, beam 4); keep terms that NLLB lost were restored by masking them as [n] and translating again. A phrase that consists only of keep terms and punctuation (sks!) is its own target.

Method: LoRA (r=32, α=64, dropout 0.05) on the q/k/v/out projections of every self- and cross-attention block (9.4 M trainable parameters, 1.5 %), trained from the base model (not on top of v1-abbr), using 🤗 transformers 5.17 + peft 0.21. 5 epochs, effective batch 16, lr 2e-4 with linear decay, bf16, group-by-length, seed 42, about 2 hours on one RTX 3060 Ti. The checkpoint with the lowest validation loss (step 6,600 of 7,370) was merged into the base weights (merge_and_unload) and converted the same way as v0-base:

ct2-transformers-converter --model nllb-finetuned-v2-merged \
    --output_dir nllb-200-ct2-int8 --quantization int8 \
    --copy_files tokenizer.json tokenizer_config.json special_tokens_map.json \
                 sentencepiece.bpe.model generation_config.json

All files except model.bin are byte-for-byte identical to v0-base and v1-abbr, so the revisions are drop-in replacements for each other.

Evaluation

All models were run as int8 CTranslate2 on the CPU, beam 4. The metric is chrF (sacrebleu 2.6, β=2); the confidence intervals come from a paired bootstrap (95 %, 1000 resamples).

v1 test set (1,301 pairs, unchanged from v1-abbr): one unseen phrase per v1 abbreviation (911 pairs) and held-out FLORES-200 dev sentences (390 pairs).

Metric	v0-base	v1-abbr	v2-game	v2 − v0 (95 % CI)
Keep terms preserved verbatim	46.9 %	92.8 %	96.4 %	+49.5 pts
chrF, translate-mode abbreviations	41.74	52.31	53.65	+11.9
chrF, all abbreviation phrases	51.21	58.46	59.54	+8.33 (+6.62 … +10.15)
chrF, general sentences (FLORES-200)	55.65	57.41	57.78	+2.14 (+1.18 … +3.12)

Glossary test set (1,539 pairs: 171 held-out phrases × 9 languages: template sentences, held-out glossary examples and plain words).

Metric	v0-base	v1-abbr	v2-game
Keep terms preserved verbatim	39.9 %	76.4 %	94.4 %
chrF, translate-mode abbreviations	35.63	49.38	53.31
chrF, all abbreviation phrases	49.34	54.54	60.22 (+10.87 vs v0, CI +9.17 … +12.59)
chrF, plain words (normal translation)	76.95	48.97	59.67

Per target language, v1 test set:

chrF	be	cs	de	es	fr	hu	pl	ru	uk
Abbreviations, v0-base	50.6	50.4	53.8	55.1	57.9	51.5	50.0	46.5	43.7
Abbreviations, v2-game	53.9	62.9	58.7	66.4	65.4	59.1	58.1	57.3	53.9
General, v0-base	38.1	57.8	58.6	57.2	68.4	57.3	49.8	55.4	53.7
General, v2-game	40.0	58.6	62.8	57.7	72.1	58.0	50.7	59.0	56.3

The general per-language scores rest on about 43 sentences each and are noisy; the pooled number is reliable.

How to read the scores. The references are NLLB translations of the expanded phrases (and of the isolated plain words), not human translations. The abbreviation chrF shows whether the abbreviation was understood, not how good the translation is. The plain-word references penalise valid alternatives (kill → «убить» vs «убивать») and a few are broken; without the single worst reference the plain-word chrF is 86.0 for v0-base, 54.4 for v1-abbr and 66.6 for v2-game.

Usage

pip install ctranslate2 sentencepiece huggingface_hub

import ctranslate2
import sentencepiece as spm
from huggingface_hub import snapshot_download

model_dir = snapshot_download("Alhite/overlay-translator-nllb-int8", revision="v2-game")

translator = ctranslate2.Translator(model_dir, device="cpu", compute_type="int8")
# GPU: ctranslate2.Translator(model_dir, device="cuda", compute_type="int8_float16")
sp = spm.SentencePieceProcessor(model_file=f"{model_dir}/sentencepiece.bpe.model")

def translate(text: str, src: str, tgt: str, beam_size: int = 4) -> str:
    # NLLB input format: <src_lang> tokens... </s>; the target language is forced as a prefix.
    source = [src] + sp.encode(text, out_type=str) + ["</s>"]
    result = translator.translate_batch([source], target_prefix=[[tgt]], beam_size=beam_size)
    tokens = result[0].hypotheses[0][1:]  # drop the target-language token
    return sp.decode(tokens)

print(translate("omw to the boss, wait for me", "eng_Latn", "rus_Cyrl"))

NLLB uses FLORES-200 language codes: eng_Latn, deu_Latn, fra_Latn, spa_Latn, rus_Cyrl, ukr_Cyrl, bel_Cyrl, pol_Latn, ces_Latn, hun_Latn.

Size and speed are the same as v0-base: the architecture and the quantization did not change (model.bin 594 MiB; about 0.4 s per short chat phrase on 4 CPU threads, beam 4).

Limitations

• Ordinary words are sometimes left untranslated. On isolated plain words v2-game scores below v0-base (96 of 513 left in English vs 53 for v0-base); it recovers a large part of what v1-abbr lost, but not all of it.
• Unseen variants are expanded wrongly, e.g. lf2m for mythic dungeon → «Я хочу вернуться в мифический подземелье» (only lf1m is in the training data).
• Keep terms can still be lost in some contexts: rng finally blessed you → «Господь благословил тебя»; overall about 4–6 % of keep terms are lost.
• The model may expand unknown short tokens (names, other slang) as if they were abbreviations.
• The training targets were produced by NLLB itself, so its mistakes were learned too: tank → «резервуар», clutch → «сцепление», healer → «врач».
• Only English as the source language was trained.
• Inherited from NLLB-200: a research model, not intended for production deployment, domain-specific (medical, legal) or document translation, or certified translations.

License

Model weights: CC-BY-NC-4.0. Non-commercial use only.

This model is an adaptation of facebook/nllb-200-distilled-600M by Meta AI (NLLB Team), licensed CC-BY-NC-4.0, and is distributed under the same license. Changes made to the original weights:

1. fine-tuned with LoRA on the data described above, then merged into the weights;
2. converted to the CTranslate2 format and quantized to int8.

The tokenizer files (sentencepiece.bpe.model, tokenizer*.json, special_tokens_map.json) and generation_config.json are copied unchanged from the original model. This model is not affiliated with or endorsed by Meta. It is provided "as is", without warranties of any kind (see section 5 of the license).

Training data

None of the training data is redistributed in this repository; it was only used for training and evaluation.

Data	Author	License	How it was used
willwade/txt-sms-abbreviations	Will Wade	Apache-2.0	74 abbreviations and their expansions (filtered, gaming meanings corrected)
FLORES-200 dev split, via facebook/flores	NLLB Team, Meta AI	CC-BY-SA-4.0	5,894 sentence pairs for training, 390 held out for evaluation
Game-chat phrases and their expansions	Overlay-Translator project	not distributed	398 short phrases written for the v1 fine-tune
Game-chat glossary	Overlay-Translator project	not distributed	3,035 rows (term, meaning, policy, example), curated as described above
wordfreq word frequencies	Robyn Speer	code Apache-2.0, data CC-BY-SA-4.0	only to decide which glossary entries are common English words; no wordfreq data is in the training pairs
Target translations of those phrases	generated by the base model (CC-BY-NC-4.0)	—	training targets

Citation

@article{nllb2022,
  title   = {No Language Left Behind: Scaling Human-Centered Machine Translation},
  author  = {{NLLB Team} and Costa-jussà, Marta R. and Cross, James and others},
  journal = {arXiv preprint arXiv:2207.04672},
  year    = {2022}
}

@article{goyal2022flores,
  title   = {The {FLORES}-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation},
  author  = {Goyal, Naman and Gao, Cynthia and Chaudhary, Vishrav and Chen, Peng-Jen and
             Wenzek, Guillaume and Ju, Da and Krishnan, Sanjana and Ranzato, Marc'Aurelio and
             Guzm{\'a}n, Francisco and Fan, Angela},
  journal = {Transactions of the Association for Computational Linguistics},
  volume  = {10},
  pages   = {522--538},
  year    = {2022}
}

@inproceedings{hu2022lora,
  title     = {{LoRA}: Low-Rank Adaptation of Large Language Models},
  author    = {Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and
               Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu},
  booktitle = {International Conference on Learning Representations},
  year      = {2022}
}

@software{speer2022wordfreq,
  author    = {Speer, Robyn},
  title     = {rspeer/wordfreq: v3.0},
  publisher = {Zenodo},
  year      = {2022},
  doi       = {10.5281/zenodo.7199437}
}