ukr-static-m3-64k

A static embedding model for Ukrainian: 64,003 tokens × 256 dimensions, 32.8 MB of FP16 weights (about 35 MB with the tokenizer). It was distilled from BAAI/bge-m3 with Model2Vec and then trained with Tokenlearn. Encoding a text is a table lookup and an average, so it runs on a CPU in about 0.1 ms per query and needs no GPU.

On three Ukrainian retrieval sets it scores 0.07–0.12 nDCG@10 higher than potion-multilingual-128M, with weights one-eighth the size at the same precision. It does not replace BM25 for keyword-style questions; see Results.

Usage

from model2vec import StaticModel

model = StaticModel.from_pretrained("SyntaxSpirits/ukr-static-m3-64k")
embeddings = model.encode(["Київ — столиця України.", "Як оформити закордонний паспорт?"])

With sentence-transformers:

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("SyntaxSpirits/ukr-static-m3-64k").float()
embeddings = model.encode(["Київ — столиця України."])

Keep the .float(): the weights are stored in FP16, and without it sentence-transformers returns NaN for inputs that contain no known tokens, such as an empty string.

Embeddings are L2-normalised, so a dot product is the cosine similarity. Inputs that leave no known tokens, such as an empty string or "!!!", get a zero vector.

Long texts are truncated. Both loaders cut at 512 tokens (set in tokenizer.json). Model2Vec by default also cuts at 3,584 characters (512 × the median token length of 7) before tokenizing; in the evaluations below this changed the tokens of no document, for this model or POTION. To embed whole texts with Model2Vec, pass max_length=None to model.encode; otherwise split long documents into chunks.

Results

nDCG@10 for question → passage retrieval.

Model	Weights	Belebele uk	WebFAQ uk	UA-SQuAD val
ukr-static-m3-64k	32.8 MB (FP16)	0.776	0.434	0.684
potion-multilingual-128M	512 MB (FP32)	0.652	0.344	0.613
static-similarity-mrl-multilingual-v1	434 MB (FP32)	0.617	0.352	–
BM25	–	0.693	0.490	0.761
ukr-static-m3-64k + BM25	–	0.770	0.504	0.765
potion-multilingual-128M + BM25	–	0.716	0.462	0.734
BAAI/bge-m3 (teacher)	2.27 GB (FP32)	0.937	–	–

• Belebele ukr_Cyrl: 900 questions, 488 passages. WebFAQ ukr test: 6,851 questions, 68,501 answers. UA-SQuAD val: 5,726 answerable questions, 1,160 paragraphs.
• Differences from the strongest other static model, with 95% bootstrap intervals: Belebele +0.124 [+0.100, +0.147] (POTION), WebFAQ +0.082 [+0.074, +0.091] (static-similarity-mrl), UA-SQuAD +0.071 [+0.060, +0.081] (POTION).
• "+ BM25" is reciprocal-rank fusion (k = 60, top 100 from each system, no tuning). BM25 is Okapi with default parameters on lowercased word tokens. Tied scores are ordered by document index.
• Belebele and WebFAQ were used during development (choice of vocabulary and the decision to train). UA-SQuAD was run with a protocol fixed beforehand. 2 of its 1,160 paragraphs share an 8-word sequence with the training text; leaving out their 10 questions changes no score by more than 0.001.
• These are the scores of this release. An earlier build had a tokenizer bug: the unknown-token id was shared with the subword s. It was fixed and the model retrained before release. All three sets, UA-SQuAD included, were evaluated with both builds, and no score differs by more than 0.0015 nDCG@10.
• Size: at FP16, POTION's weights would be 256 MB, 7.8 times this model's.
• Speed and memory on an Apple M1 Pro CPU, encoding the 900 Belebele questions one at a time after warm-up: p95 0.11 ms per query (POTION 0.12–0.14 ms). The model loads in 0.3 s (POTION 3 s). Peak memory of the Python process is about 360 MiB (POTION about 2.0 GiB), of which about 150 MiB is Python and the datasets library.

Training

1. Vocabulary. The 32,000 most frequent lowercased Ukrainian words in 60 M characters of Ukrainian Wikipedia (dump 20231101) and FineWeb-2 (ukr_Cyrl); only Cyrillic letter sequences were counted. Added to them are the 32,000 BGE-M3 subword tokens most frequent in the same text, so unseen word forms still get vectors, plus a dedicated unknown token with a zero vector. Whole words get the highest Unigram score, so the tokenizer prefers them over subword splits. Of the 9,711 documents used for counting, 86 share an 8-word sequence with WebFAQ, 1 with UA-SQuAD and none with Belebele; they were not removed from the counts.
2. Distillation. Each token was embedded by BGE-M3 (mean pooling), reduced to 256 dimensions with PCA and weighted by corpus frequency (SIF, 1e-4).
3. Tokenlearn. 119,430 chunks of up to 120 words from the same sources; chunks that share an 8-word sequence with Belebele or WebFAQ were removed. Target: the mean BGE-M3 token embedding of each chunk, reduced to 256 dimensions by a PCA fitted on the training chunks. Unknown tokens were removed from the inputs, as Model2Vec does at inference. Tokenlearn defaults: AdamW, learning rate 1e-3, batch 256, MSE loss, early stopping on the last 10,000 chunks.

Limitations

• Input is lowercased, word order is ignored, and long texts are truncated (see Usage).
• On WebFAQ and UA-SQuAD, BM25 alone scores higher than this model, and fusing the two adds only +0.014 and +0.005 nDCG@10. For keyword-style questions, use it alongside BM25, not instead of it.
• It is far behind its teacher (0.776 vs 0.937 on Belebele).
• The vocabulary was built for Ukrainian. Other languages, including Russian and English, are covered only by subwords seen in Ukrainian text and were not evaluated.
• Only retrieval was evaluated; similarity, clustering and classification were not.
• UA-SQuAD is a Ukrainian translation of the SQuAD 2.0 dev set, so its questions share many words with their passages, which favours BM25.

License

MIT (see LICENSE). The model is derived from BAAI/bge-m3 (MIT) and uses its XLM-RoBERTa tokenizer (MIT); their notices are in NOTICE. Training text came from Ukrainian Wikipedia (CC BY-SA 4.0) and FineWeb-2 (ODC-By 1.0, also subject to the Common Crawl Terms of Use). Contains information from FineWeb-2, which is made available under the ODC Attribution License. No training passages are included: the release contains token vectors and a vocabulary of single words and subword pieces.

Citation

Oleksandr Kholodniak (ORCID 0009-0000-1012-7361).

@misc{kholodniak2026ukrstatic,
  author       = {Kholodniak, Oleksandr},
  title        = {ukr-static-m3-64k: a static embedding model for Ukrainian retrieval},
  year         = {2026},
  howpublished = {\url{https://huggingface.co/SyntaxSpirits/ukr-static-m3-64k}}
}