kenga-embed-prophet3

44M-parameter Russian/English sentence encoder from the Kenga project. Bidirectional Z-factored transformer, SentencePiece-16k tokenizer, mean pooling, 768-d L2-normalised output, 512-token context. Prefix protocol is the FRIDA / BERTA one, so it drops into any pipeline that already uses them.

kenga-embed-prophet2 starts from kenga-embed-z2 and adds a supervised stage: InfoNCE with mined hard negatives on 80k retrieval pairs (RuBQ / MIRACL-style QA, wiki title-passage), CoSENT on 20k STS pairs, plus an anchor loss that keeps the embedding close to the teacher so the distilled knowledge is not overwritten. This is the Kenga Prophet stage: learn from verified pairs, penalise confident mistakes (hard negatives).

Official MTEB(rus, v1.1) numbers

Run with mteb==2.20.5, all splits and subsets as defined by the benchmark, no task was skipped or re-weighted. Reference columns are the models' own leaderboard submissions (embeddings-benchmark/results). 23/23 tasks done.

task	kenga-embed-prophet3	Giga-Embeddings-instruct-480M	BERTA-128M	USER2-small-34M	rubert-tiny-turbo-29M
GeoreviewClassification	45.2	55.4	54.8	41.1	41.4
HeadlineClassification	81.6	89.0	89.0	74.3	68.9
InappropriatenessClassification	58.6	86.1	74.8	60.7	59.1
KinopoiskClassification	61.0	73.0	67.8	52.2	50.5
MassiveIntentClassification	56.3	85.3	74.0	66.1	58.0
MassiveScenarioClassification	66.6	90.9	84.5	70.3	62.9
RuReviewsClassification	68.4	76.3	72.3	60.8	60.7
RuSciBenchGRNTIClassification	60.8	74.0	69.0	63.1	52.9
RuSciBenchOECDClassification	46.2	59.9	54.8	49.2	40.8
CEDRClassification	51.7	69.8	73.0	39.4	39.0
SensitiveTopicsClassification	23.5	44.3	39.9	27.5	25.2
GeoreviewClusteringP2P	43.9	73.8	73.8	66.2	59.7
RuSciBenchGRNTIClusteringP2P	58.3	70.5	65.0	56.4	48.1
RuSciBenchOECDClusteringP2P	49.6	58.1	55.6	48.6	41.1
TERRa	61.1	79.6	65.7	54.0	56.3
RuBQReranking	60.2	80.5	75.2	66.0	62.2
MIRACLReranking	38.4	67.5	64.3	50.5	47.7
RiaNewsRetrievalHardNegatives.v2	38.2	88.9	84.5	74.5	52.3
RuBQRetrieval	41.5	80.6	71.0	61.1	51.7
MIRACLRetrievalHardNegatives.v2	30.5	74.7	65.9	46.1	42.4
RUParaPhraserSTS	65.9	78.3	77.8	69.6	72.1
RuSTSBenchmarkSTS	70.2	83.6	82.2	81.0	78.5
STS22	51.4	65.3	61.1	66.1	64.6
---	---	---	---	---	---
Classification (mean)	60.5	76.7	71.2	59.8	55.0
MultilabelClassification (mean)	37.6	57.1	56.5	33.5	32.1
Clustering (mean)	50.6	67.5	64.8	57.1	49.6
PairClassification (mean)	61.1	79.6	65.7	54.0	56.3
Reranking (mean)	49.3	74.0	69.7	58.3	54.9
Retrieval (mean)	36.7	81.4	73.8	60.6	48.8
STS (mean)	62.5	75.7	73.7	72.2	71.7
mean over tasks	53.4	74.2	69.4	58.5	53.7
mean over task types (leaderboard)	51.2	73.1	67.9	56.5	52.6
tasks done	23	23	23	23	23

Leaderboard-style mean (average of task-type means): 51.2.

What this is and is not: a 44M model, roughly 10x smaller than Giga-Embeddings-instruct-480M, meant to be compared with the 30-40M Russian encoders (USER2-small, rubert-tiny-turbo). On the 23 finished tasks (plain mean) it scores 53.4 vs 74.2 for Giga-Embeddings-instruct-480M (+20.7 gap) and 58.5 for USER2-small-34M (-5.0). It does not beat Giga. The numbers above are the whole story; the raw result files are in mteb_results/ of the training tree.

Usage

import sys; sys.path.insert(0, "<this folder>")     # or trust_remote_code-style import after download
from modeling_kenga_embed_v2 import KengaEmbedV2HF

m = KengaEmbedV2HF.from_pretrained("<this folder>", device="cuda")   # cpu works too

q = m.encode(["??? ??????? ????? ? ????"], prefix="search_query")
d = m.encode(["????? ???? ???????? ...", "?????? ?????"], prefix="search_document")
print(q @ d.T)                       # cosine, embeddings are L2-normalised

a = m.encode(["??? ???? ?? ??????."], prefix="paraphrase")
b = m.encode(["?? ?????? ???? ?????."], prefix="paraphrase")

Prefixes ("<prefix>: <text>" is prepended for you):

use	prefix
retrieval query	search_query
retrieval document	search_document
STS / paraphrase (both sides)	paraphrase
classification / clustering	categorize, categorize_sentiment, categorize_topic
NLI / entailment (TERRa)	categorize_entailment

Download with huggingface_hub.snapshot_download("GermannM/kenga-embed-prophet3"); the folder contains pytorch_model.bin, config.json, kenga_spm.model and the self-contained modeling_kenga_embed_v2.py (torch + sentencepiece only).

Architecture

d=768, layers=8, heads=12, dff=3072, factorised token embedding (16385 x 128 -> 768), Z-factored attention/FF projections with rank 192/512, learned positions up to 512. 44.2M parameters, fp32 checkpoint 177 MB. Checkpoint step 1000.

Training code

PyTorch trainers live in the z-system lab tree (embed_v2/: build_segments.py, teacher.py, distill.py, build_ft_data.py, mine_hard.py, finetune_prophet.py, run_mteb.py), not in the public kenga-lang repo; the recipe and the Prophet contract are documented in docs/PROPHETS.md. Trained on one GTX 1660 (6 GB).

License

MIT.