marian-uk-verbalizer-ct2-int8

An int8 CTranslate2 export of aloudreader/marian-uk-verbalizer, a Ukrainian text-verbalization (TTS front-end) model. Same model, ~52 MB instead of ~200 MB, for fast CPU or GPU inference.

Потрібно прочитати XX розділ до понеділка.
  -> Потрібно прочитати двадцятий розділ до понеділка.

Full training/evaluation code and the router this model sits behind: GitHub repository.

Accuracy vs. the full-precision model

Compared against the PyTorch checkpoint on the same benchmarks:

Set	Match rate (CT2 int8 vs. PyTorch fp32 output)
Frozen 3,597-row benchmark	99.1–99.2% output-identical

The int8 export's own scores on the per-domain evaluation sets are within 0.4 percentage points of the PyTorch checkpoint on every domain (usually identical), so it can be used as a drop-in low-latency replacement.

How to use

pip install ctranslate2 transformers sentencepiece huggingface_hub

import ctranslate2
from huggingface_hub import snapshot_download
from transformers import MarianTokenizer

path = snapshot_download("aloudreader/marian-uk-verbalizer-ct2-int8")
tokenizer = MarianTokenizer.from_pretrained(path)
translator = ctranslate2.Translator(path, device="cpu", compute_type="int8")

text = "Потрібно прочитати XX розділ до понеділка."
ids = tokenizer(text, truncation=True, max_length=384, add_special_tokens=True)["input_ids"]
tokens = tokenizer.convert_ids_to_tokens(ids)

# suppress_sequences is required: this export's decoder start token is "<s>"
# (id 0, not the usual pad token), which must never itself be generated.
result = translator.translate_batch([tokens], beam_size=1, suppress_sequences=[["<s>"]])
out_ids = tokenizer.convert_tokens_to_ids(result[0].hypotheses[0])
print(tokenizer.decode(out_ids, skip_special_tokens=True))

For GPU inference, pass device="cuda" to ctranslate2.Translator (requires a CUDA build of CTranslate2 and libcublas).

Why the decoder-start handling matters

A naive Marian → CTranslate2 conversion assumes the decoder starts from a zero embedding and uses the model's pad token as the decoder-start token. This model's training used <s> (token id 0) as the decoder start instead; converting it the naive way silently drops the first generated word from every output. This export was produced with scripts/convert_marian_ct2.py in the GitHub repository, which sets start_from_zero_embedding=False and the correct decoder-start token, and pairs it with suppress_sequences=[["<s>"]] at inference time so <s> is never itself generated as output.

License

MIT. See the GitHub repository.