castorini/rank_zephyr_7b_v1_full

Model Card for RankZephyr 7B V1 - Full

RankZephyr is a series of language models trained to act as helpful reranking assistants built on the Zephyr-7B-β model. RankZephyr Base is the model that follows single-stage fine-tuning on the RankGPT-3.5 model, while RankZephyr Full is the model that is further fine-tuned on RankGPT-4 reorderings of OpenAI's Ada2 orderings for 5K queries.

Model description

• Model type: A 7B parameter GPT-like model initially fine-tuned on a mix of publicly available, synthetic datasets, followed by task-specific listwise reranking data.
• Language(s) (NLP): Primarily English
• License: MIT
• Fine-tuned from model: HuggingFaceH4/zephyr-7b-beta

Model Sources

• Repository: https://github.com/castorini/rank_llm
• Paper: https://arxiv.org/abs/2312.02724

Effectiveness

At the time of release, RankZephyr-7B-Full is the state-of-the-art open-source reranking model on various datasets like DL19/20/21/22 and TREC-COVID and TREC-News.

With the MS MARCO v1 collection:

Model	Size	First Stage	DL19	DL20
RankZephyr-7b-v1-full-rho 🪁	7B	SPLADE++ ED	0.7855	0.8255
RankZephyr-7b-v1-full 🪁	7B	SPLADE++ ED	0.7803	0.8211
RankGPT-4 (PSC)	-	SPLADE++ ED	0.7601	0.7514
RankGPT-4	-	SPLADE++ ED	0.7464	0.7076
RankZephyr-7b-v1-base 🪁	7B	SPLADE++ ED	0.7341	0.7213
RankGPT-3.5	-	SPLADE++ ED	0.7504	0.7120

More details can be found in the paper.

Intended uses & limitations

The model is to be used in conjunction with the RankLLM repository. While rank-llm exists as a PyPI package, we are currently in the early stages of development and encourage users to directly check install from source.

The original Zephyr model is trained for chat. In our case, RankZephyr is fine-tuned to act as a listwise reranking agent. You provide it with a query and documents and get back a reordered list of document identifiers.

Bias, Risks, and Limitations

The following is an excerpt from the Zephyr-7B-β model card:

Zephyr-7B-β has not been aligned to human preferences for safety within the RLHF phase or deployed with in-the-loop filtering of responses like ChatGPT, so the model can produce problematic outputs (especially when prompted to do so). It is also unknown what the size and composition of the corpus was used to train the base model (mistralai/Mistral-7B-v0.1), however it is likely to have included a mix of Web data and technical sources like books and code. See the Falcon 180B model card for an example of this.

Our model is trained specifically on monolingual English data, effectiveness on multilingual sets is not guaranteed.

Citation

If you find RankZephyr is useful in your work, please cite the following paper:

@ARTICLE{pradeep2023rankzephyr,
  title   = {{RankZephyr}: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!},
  author  = {Ronak Pradeep and Sahel Sharifymoghaddam and Jimmy Lin},
  year    = {2023},
  journal = {arXiv:2312.02724}
}