Model Card for first_mistral

first_mistral is a language model trained to act as a listwise reranker, decoding from the first-token logits only to improve efficiency while maintaining effectiveness. first_mistral is built on Mistral-7B-Instruct-v0.3, following FIRST's strategy, trained using 40K GPT-4 labeled rerank instances from RankZephyr. More details can be found in the paper.

Model description

• Model type: A 7B parameter listwise reranker fine-tuned from Mistral-7B-Instruct-v0.3
• Language(s) (NLP): Primarily English
• License: MIT
• Fine-tuned from model: mistralai/Mistral-7B-Instruct-v0.3

Model Sources

• Repository: https://github.com/castorini/rank_llm
• Paper: https://arxiv.org/abs/2411.05508

Evaluation

At the time of release, first_mistral outperforms the original FIRST implementation on most subsets of the BEIR benchmark. More details that compare other LLM rerankers on more datasets can be found in the paper.

Dataset	FIRST (original)	first_mistral
climate-fever	0.2672	0.2417
dbpedia-entity	0.5092	0.5033
fever	0.8164	0.8413
fiqa	0.4223	0.4778
hotpotqa	0.7424	0.7705
msmarco	0.4425	0.4512
nfcorpus	0.3725	0.3816
nq	0.6638	0.6985
scidocs	0.2047	0.2110
scifact	0.7459	0.7769
trec-covid	0.7913	0.7666
Average	0.5435	0.5564

Citation

If you find first_mistral useful for your work, please consider citing:

@ARTICLE{chen2024firstrepro,
  title   = title={An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking},
  author  = {Zijian Chen and Ronak Pradeep and Jimmy Lin},
  year    = {2024},
  journal = {arXiv:2411.05508}
}

If you would like to cite the FIRST methodology, please consider citing:

@ARTICLE{reddy2024first,
  title   = {FIRST: Faster Improved Listwise Reranking with Single Token Decoding},
  author  = {Reddy, Revanth Gangi and Doo, JaeHyeok and Xu, Yifei and Sultan, Md Arafat and Swain, Deevya and Sil, Avirup and Ji, Heng},
  year    = {2024}
  journal = {arXiv:2406.15657},
}