PolDense-1B

PolDense is a family of Polish embedding models optimized for dense retrieval. The models are built on top of the ettin-encoders, which use the ModernBERT architecture, and are designed to provide high-quality text representations for Polish search, retrieval, and RAG applications. The family consists of six models, ranging from 17M to 1B parameters. This makes it possible to choose between lightweight, low-latency deployments and larger models aimed at maximum retrieval quality.

Training

PolDense models were trained in a three-stage pipeline with two knowledge distillation stages and one fine-tuning stage. The main teacher embedding model was BGE-Multilingual-Gemma2, while the final retrieval fine-tuning stage used relevance scores produced by BGE-Reranker-v2.5-Gemma2-Lightweight.

1. The first stage adapted the original English ettin-encoder models to Polish using multilingual knowledge distillation. This stage follows the approach introduced in Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillation. The core idea is to use a strong teacher model to define a shared semantic embedding space. For each parallel sentence pair, the teacher encodes the source-language sentence, and the student is trained to produce a similar embedding for the translated sentence. In practice, this teaches the student to map Polish and English texts with the same meaning to nearby locations in the vector space, while preserving the retrieval-oriented structure learned by the teacher. In this stage, BGE-Multilingual-Gemma2 was used as the teacher model. Training was performed on a Polish-English parallel corpus containing approximately 20 million text pairs. Each model was trained for 5 epochs with a batch size of 64.

2. The second stage used a more advanced distillation objective inspired by Jasper and Stella: distillation of SOTA embedding models. The goal of this stage was to align the student models more closely with the teacher and extract as much retrieval quality as possible from the distillation process. Unlike the first stage, this stage was performed only on Polish texts. The training objective combined three complementary losses: a) a cosine alignment loss; b) a pairwise similarity loss; c) a triplet-style ranking loss. Together, these losses train the student not only to imitate individual teacher vectors, but also to reproduce the teacher's local geometry and ranking behavior. This is particularly important for retrieval, where relative similarity between queries and passages matters more than isolated embedding values. The teacher model in this stage was again BGE-Multilingual-Gemma2. Models from 17M to 150M parameters were trained on approximately 70 million Polish texts, while the 400M and 1B models were trained on approximately 33 million Polish texts. The corpus included the 20 million Polish texts used in first stage, with additional texts taken from the Polish portion of the FineTranslations collection. Each model was trained for 5 epochs with a batch size of 128.

3. The final stage fine-tuned the models directly for retrieval using contrastive learning. This stage used 14 retrieval training datasets containing over 4.5 million queries and more than 15 million passages. No ground-truth labels were used during this stage. Instead, the training data was constructed entirely from positive and negative examples selected with the BGE-Reranker-v2.5-Gemma2-Lightweight reranker. The models were trained for 10 epochs with a batch size of 1024.

The figure below presents an evaluation of PolDense models on the PIRB benchmark, which consists of 41 Polish retrieval tasks. As shown, each training stage contributed to the model’s final performance:

Evaluation

The PolDense models were evaluated on the PIRB benchmark. The results show that they outperform previously available retrievers for Polish, with each model achieving the best performance in its size class. The smaller 32M and 68M variants reach results comparable to much larger multilingual models with several hundred million parameters, such as Jina-Embeddings-V5, BGE-M3, and Snowflake-Arctic-Embed-2.0. The 150M model performs close to Qwen3-Embedding-8B, while the 400M model reaches a level similar to BGE-Multilingual-Gemma2 and Llama-Embed-Nemotron-8b. The largest 1B variant surpasses both of these models. The results are shown in the figure below.

Usage

The model utilizes prefixes for retrieval and semantic similarity tasks. For retrieval, queries should be prefixed with "[query]: ". For symmetric tasks such as semantic similarity, both texts should be prefixed with "[sts]: ".

We recommend using the model with sentence-transformers version 5.4.0 or newer and initializing it with the arguments dtype="float16" or "bfloat16", and attn_implementation="flash_attention_2", which enables optimal performance. You can use the model with sentence-transformers:

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

model = SentenceTransformer(
    "OPI-PIB/PolDense-1B",
    device="cuda",
    model_kwargs={
        "dtype": "bfloat16",
        "attn_implementation": "flash_attention_2"
    }
)

# Retrieval example
query_prefix = "[query]: "
queries = [query_prefix + "Jak dożyć 100 lat?"]
answers = [
    "Trzeba zdrowo się odżywiać i uprawiać sport.",
    "Trzeba pić alkohol, imprezować i jeździć szybkimi autami.",
    "Gdy trwała kampania politycy zapewniali, że rozprawią się z zakazem niedzielnego handlu."
]
queries_emb = model.encode(queries, convert_to_tensor=True, show_progress_bar=False)
answers_emb = model.encode(answers, convert_to_tensor=True, show_progress_bar=False)
best_answer = cos_sim(queries_emb, answers_emb).argmax().item()
print(answers[best_answer])

# Semantic similarity example
sim_prefix = "[sts]: "
sentences = [
    sim_prefix + "Trzeba zdrowo się odżywiać i uprawiać sport.",
    sim_prefix + "Warto jest prowadzić zdrowy tryb życia, uwzględniający aktywność fizyczną i dietę.",
    sim_prefix + "One should eat healthy and engage in sports.",
    sim_prefix + "Zakupy potwierdzasz PINem, który bezpiecznie ustalisz podczas aktywacji."
]
emb = model.encode(sentences, convert_to_tensor=True, show_progress_bar=False)
print(cos_sim(emb, emb))

Running with vLLM:

vllm serve OPI-PIB/PolDense-1B --dtype bfloat16 --runner pooling --convert embed

Acknowledgements

The research was supported by the project Large Language Models for the European Union (LLMs4EU). This project is co-funded by the Digital Europe Programme under Grant Agreement 101198470.

The research was supported [in part] by project “Cloud Artificial Intelligence Service Engineering (CAISE) platform to create universal and smart services for various application areas”, No. KPOD.05.10-IW.10-0005/24, as part of the European IPCEI-CIS program, financed by NRRP (National Recovery and Resilience Plan) funds. Computations were carried out using the computers of Centre of Informatics Tricity Academic Supercomputer & Network at Gdansk University of Technology.