kenga-embed-z2

44M-parameter Russian/English sentence encoder from the Kenga project. Bidirectional Z-factored transformer, SentencePiece-16k tokenizer, mean pooling, 768-d L2-normalised output, 512-token context. Prefix protocol is the FRIDA / BERTA one, so it drops into any pipeline that already uses them.

kenga-embed-z2 is the distillation stage: 30k steps of cosine + relational-KL distillation from sergeyzh/BERTA (128M, itself distilled from FRIDA) on 1.23M Russian/English segments (Wikipedia, dialogues, reviews, headlines, intents; FRIDA-style prefixes). No labels were used. The Z-factored layers follow a rank curriculum: 25% -> 50% -> 100% of the final rank during training.

Official MTEB(rus, v1.1) numbers

Run with mteb==2.20.5, all splits and subsets as defined by the benchmark, no task was skipped or re-weighted. Reference columns are the models' own leaderboard submissions (embeddings-benchmark/results). 23/23 tasks done.

task	kenga-embed-z2	Giga-Embeddings-instruct-480M	BERTA-128M	USER2-small-34M	rubert-tiny-turbo-29M
GeoreviewClassification	44.4	55.4	54.8	41.1	41.4
HeadlineClassification	81.0	89.0	89.0	74.3	68.9
InappropriatenessClassification	59.8	86.1	74.8	60.7	59.1
KinopoiskClassification	60.6	73.0	67.8	52.2	50.5
MassiveIntentClassification	55.1	85.3	74.0	66.1	58.0
MassiveScenarioClassification	65.3	90.9	84.5	70.3	62.9
RuReviewsClassification	67.2	76.3	72.3	60.8	60.7
RuSciBenchGRNTIClassification	60.0	74.0	69.0	63.1	52.9
RuSciBenchOECDClassification	46.3	59.9	54.8	49.2	40.8
CEDRClassification	47.6	69.8	73.0	39.4	39.0
SensitiveTopicsClassification	23.7	44.3	39.9	27.5	25.2
GeoreviewClusteringP2P	42.7	73.8	73.8	66.2	59.7
RuSciBenchGRNTIClusteringP2P	58.1	70.5	65.0	56.4	48.1
RuSciBenchOECDClusteringP2P	49.5	58.1	55.6	48.6	41.1
TERRa	55.0	79.6	65.7	54.0	56.3
RuBQReranking	60.7	80.5	75.2	66.0	62.2
MIRACLReranking	41.5	67.5	64.3	50.5	47.7
RiaNewsRetrievalHardNegatives.v2	34.6	88.9	84.5	74.5	52.3
RuBQRetrieval	38.1	80.6	71.0	61.1	51.7
MIRACLRetrievalHardNegatives.v2	31.8	74.7	65.9	46.1	42.4
RUParaPhraserSTS	57.8	78.3	77.8	69.6	72.1
RuSTSBenchmarkSTS	65.1	83.6	82.2	81.0	78.5
STS22	55.3	65.3	61.1	66.1	64.6
---	---	---	---	---	---
Classification (mean)	60.0	76.7	71.2	59.8	55.0
MultilabelClassification (mean)	35.6	57.1	56.5	33.5	32.1
Clustering (mean)	50.1	67.5	64.8	57.1	49.6
PairClassification (mean)	55.0	79.6	65.7	54.0	56.3
Reranking (mean)	51.1	74.0	69.7	58.3	54.9
Retrieval (mean)	34.8	81.4	73.8	60.6	48.8
STS (mean)	59.4	75.7	73.7	72.2	71.7
mean over tasks	52.2	74.2	69.4	58.5	53.7
mean over task types (leaderboard)	49.4	73.1	67.9	56.5	52.6
tasks done	23	23	23	23	23

Leaderboard-style mean (average of task-type means): 49.4.

What this is and is not: a 44M model, roughly 10x smaller than Giga-Embeddings-instruct-480M, meant to be compared with the 30-40M Russian encoders (USER2-small, rubert-tiny-turbo). On the 23 finished tasks (plain mean) it scores 52.2 vs 74.2 for Giga-Embeddings-instruct-480M (+21.9 gap) and 58.5 for USER2-small-34M (-6.3). It does not beat Giga. The numbers above are the whole story; the raw result files are in mteb_results/ of the training tree.

Usage

import sys; sys.path.insert(0, "<this folder>")     # or trust_remote_code-style import after download
from modeling_kenga_embed_v2 import KengaEmbedV2HF

m = KengaEmbedV2HF.from_pretrained("<this folder>", device="cuda")   # cpu works too

q = m.encode(["??? ??????? ????? ? ????"], prefix="search_query")
d = m.encode(["????? ???? ???????? ...", "?????? ?????"], prefix="search_document")
print(q @ d.T)                       # cosine, embeddings are L2-normalised

a = m.encode(["??? ???? ?? ??????."], prefix="paraphrase")
b = m.encode(["?? ?????? ???? ?????."], prefix="paraphrase")

Prefixes ("<prefix>: <text>" is prepended for you):

use	prefix
retrieval query	search_query
retrieval document	search_document
STS / paraphrase (both sides)	paraphrase
classification / clustering	categorize, categorize_sentiment, categorize_topic
NLI / entailment (TERRa)	categorize_entailment

Download with huggingface_hub.snapshot_download("GermannM/kenga-embed-z2"); the folder contains pytorch_model.bin, config.json, kenga_spm.model and the self-contained modeling_kenga_embed_v2.py (torch + sentencepiece only).

Architecture

d=768, layers=8, heads=12, dff=3072, factorised token embedding (16385 x 128 -> 768), Z-factored attention/FF projections with rank 192/512, learned positions up to 512. 44.2M parameters, fp32 checkpoint 177 MB. Checkpoint step 29000.

Training code

PyTorch trainers live in the z-system lab tree (embed_v2/: build_segments.py, teacher.py, distill.py, build_ft_data.py, mine_hard.py, finetune_prophet.py, run_mteb.py), not in the public kenga-lang repo; the recipe and the Prophet contract are documented in docs/PROPHETS.md. Trained on one GTX 1660 (6 GB).

License

MIT.