artefactory/wepr-mistral-small

Model description

Target model: mistralai/Mistral-Small-3.1-24B-Instruct-2503 -- this detector scores responses produced by that model. It is not a fine-tune of it and contains none of its weights.

A calibrated WEPR hallucination detector for responses generated by mistralai/Mistral-Small-3.1-24B-Instruct-2503.

WEPR (Weighted EPR) keeps the ranks separate, giving the calibration one coefficient per rank (mean and max over the token axis, so 2k features). It reads strictly more of the distribution than EPR at the same calibration cost.

The artifact is the fitted LogisticRegression alone. The feature extraction that feeds it -- parsing top-15 log-probabilities out of a completion response and reducing them to entropy features -- lives in the artefactual library, so this file contains no custom classes and loads with an empty trusted list.

Introduced in Learned Hallucination Detection in Black-Box LLMs Using Token-Level Entropy Production Rate (ECIR 2026); the preprint is arXiv:2509.04492.

Intended uses & limitations

Scores a response on [0, 1], where 1 is the hallucination class.

• Tied to mistralai/Mistral-Small-3.1-24B-Instruct-2503. The coefficients are fit against that model's output distribution. Scoring another model's responses with them is not meaningful, even though nothing in the file prevents it.
• Fixed at k=15. Responses must be generated with logprobs=True and top_logprobs=15. Fewer ranks are rejected rather than zero-filled, because the missing ranks are unfetched rather than absent and padding them would score the response as more confident than it was.
• No published operating point. The paper reports ROC-AUC and PR-AUC, both threshold-free, so no decision threshold is published. Choose one on your own labelled data.

Evaluation Results

See the paper. It reports ROC-AUC and PR-AUC across the evaluated models; no figures are restated here so that this card cannot drift from the published results.

How to Get Started with the Model

from artefactual.scoring import WEPR

detector = WEPR.from_pretrained("artefactory/wepr-mistral-small")
scores = detector.predict_proba(response)[:, 1]

response is an OpenAI-compatible chat completion or responses payload carrying top_logprobs=15.

Requires artefactual>=2026.9, where the detector is the WEPR class. Up to 2026.08.1 the same weights were loaded with the lowercase wepr() factory.

Model Card Authors

Artefact Research Center

Model Card Contact

https://github.com/artefactory/artefactual/issues

Citation

@inproceedings{moslonka2026learned,
  title     = {Learned Hallucination Detection in Black-Box LLMs Using Token-Level Entropy Production Rate},
  author    = {Moslonka, Charles and Randrianarivo, Hicham and Garnier, Arthur and Malherbe, Emmanuel},
  booktitle = {Advances in Information Retrieval},
  series    = {Lecture Notes in Computer Science},
  volume    = {16483},
  pages     = {115--130},
  publisher = {Springer, Cham},
  year      = {2026},
  doi       = {10.1007/978-3-032-21289-4_8},
}