cat-minilm-l12 — MiniLM cross-encoder reranker (no score injection)

A MiniLM-L12 cross-encoder passage reranker trained on MS MARCO by knowledge distillation.

This is the matched baseline for Amdestya/bm25cat-minilm-l12, built to reproduce the CE_CAT / CE_BM25CAT comparison in Askari et al., "Injecting the BM25 Score as Text Improves BERT-Based Re-rankers", ECIR 2023.

The two models share identical training data (the same 5,000,000 pairs), steps, optimizer, learning rate, batch size and sequence length. The only difference is the input construction: this one receives no injected BM25 score.

Usage

from sentence_transformers import CrossEncoder

model = CrossEncoder("Amdestya/cat-minilm-l12", max_length=256)

scores = model.predict([
    ["what is a cat", "A cat is a small domesticated carnivorous mammal."],
    ["what is a cat", "The 1998 federal budget deficit was revised upward."],
])

Input format

text_a = query
text_b = passage
# tokenises to: [CLS] query [SEP] passage [SEP]

Do not inject a BM25 score. This model was trained without one and has never seen a numeric token in that position. For the injected variant use Amdestya/bm25cat-minilm-l12, whose input format differs — see its card.

Outputs are raw logits (identity activation), unbounded, higher = more relevant. They are comparable within a query, not across models.

Training

base model	microsoft/MiniLM-L12-H384-uncased
objective	MSE against a BERTCAT teacher ensemble (Hofstätter et al.)
training pairs	5,000,000 of 79,518,568 available
steps	156,250 @ batch 32, 1 epoch
optimizer	Adam, lr 7e-6, 5,000 warmup steps
max length	256
hardware	1 × RTX A5000

Note this is a knowledge-distillation setup, following the authors' released training notebook, rather than the cross-entropy objective described in the paper's §4.

Intended use

Built as a reference artifact for a dissertation study on reproducing IR papers with LLMs. Usable as a general MS MARCO passage reranker, but the community cross-encoder/ms-marco-MiniLM-L-12-v2 is trained more thoroughly and is the better choice for production use.