Jocana/minicoil-en-es

miniCOIL EN-ES

Bilingual sparse retrieval for English and Spanish: BM25 with a learned semantic overlay. Words in one of the model's 2398 bilingual concepts map to a block of 8 values from that concept's trained head; every other token stays a plain BM25 term. Translations share a concept, so dog and perro match across languages with no translation step. Output is an ordinary sparse vector.

Usage

pip install torch transformers fastembed safetensors mmh3 simplemma huggingface-hub

import sys

from huggingface_hub import snapshot_download

model_dir = snapshot_download("Jocana/minicoil-en-es")  # ships its own encoder code
sys.path.insert(0, model_dir)

from minicoil_v2.encoder import MiniCoilEncoder

encoder = MiniCoilEncoder.from_pretrained(model_dir)

doc = encoder.encode_sparse("El perro corrió por el parque", lang="es")
query = encoder.encode_sparse("dog running", lang="en", is_query=True)

encode_sparse returns {sparse_index: value}. Encode documents in the corpus language and queries in the query language, and pass is_query=True for queries.

The index must apply IDF — in Qdrant, declare the sparse vector with Modifier.IDF. These are BM25 vectors and carry none of their own; without it, ranking quality collapses.

Results — mMARCO test

pair	metric	value	queries	vs baseline
en → en	MRR@10	0.8889	3491	+0.0084 vs bm25, -0.0032 vs minicoil-v1
es → es	MRR@10	0.8398	3341	+0.0446 vs bm25
en → es	MRR@10	0.7109	3145	+0.0081 vs translate-bm25
es → en	MRR@10	0.7154	3248	-0.0588 vs translate-bm25

Measured on the concept-covered slice of mMARCO (queries sharing a concept with their gold passage). translate-bm25 is NLLB translation followed by BM25.

Limitations

• Cross-lingual matching happens only through shared concepts. A query whose content words are all out of vocabulary can return no results against a corpus in the other language. Same-language retrieval always keeps full BM25 behavior.
• Documents are truncated at 256 tokens by the base encoder.

License

cc-by-nc-4.0 — the concept vocabulary derives from the MUSE bilingual dictionaries, which are non-commercial.

Base encoder: intfloat/multilingual-e5-small (MIT). Trained on Wikipedia (EN + ES).