metadata
license: mit
language:
- en
base_model:
- mistralai/Mistral-7B-Instruct-v0.3
Model Card for first_mistral
first_mistral is a language model trained to act as a listwise reranker, decoding from the first-token logits only to improve efficiency while maintaining effectiveness.
first_mistral is built on Mistral-7B-Instruct-v0.3, following FIRST's strategy, trained using 40K GPT-4 labeled rerank instances from RankZephyr.
More details can be found in the paper.
Model description
- Model type: A 7B parameter listwise reranker fine-tuned from Mistral-7B-Instruct-v0.3
- Language(s) (NLP): Primarily English
- License: MIT
- Fine-tuned from model: mistralai/Mistral-7B-Instruct-v0.3
Model Sources
- Repository: https://github.com/castorini/rank_llm
- Paper: https://arxiv.org/abs/2411.05508
Evaluation
At the time of release, first_mistral outperforms the original FIRST implementation on most subsets of the BEIR benchmark.
More details that compare other LLM rerankers on more datasets can be found in the paper.
| Dataset | FIRST (original) | first_mistral |
|---|---|---|
| climate-fever | 0.2672 | 0.2417 |
| dbpedia-entity | 0.5092 | 0.5033 |
| fever | 0.8164 | 0.8413 |
| fiqa | 0.4223 | 0.4778 |
| hotpotqa | 0.7424 | 0.7705 |
| msmarco | 0.4425 | 0.4512 |
| nfcorpus | 0.3725 | 0.3816 |
| nq | 0.6638 | 0.6985 |
| scidocs | 0.2047 | 0.2110 |
| scifact | 0.7459 | 0.7769 |
| trec-covid | 0.7913 | 0.7666 |
| Average | 0.5435 | 0.5564 |
Citation
If you find first_mistral useful for your work, please consider citing:
@ARTICLE{chen2024firstrepro,
title = title={An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking},
author = {Zijian Chen and Ronak Pradeep and Jimmy Lin},
year = {2024},
journal = {arXiv:2411.05508}
}
If you would like to cite the FIRST methodology, please consider citing:
@ARTICLE{reddy2024first,
title = {FIRST: Faster Improved Listwise Reranking with Single Token Decoding},
author = {Reddy, Revanth Gangi and Doo, JaeHyeok and Xu, Yifei and Sultan, Md Arafat and Swain, Deevya and Sil, Avirup and Ji, Heng},
year = {2024}
journal = {arXiv:2406.15657},
}