Package: rank-eval
🔥 Newsrank_evalwill be featured inECIR 2022, the 44th European Conference on Information Retrieval!🤖 Dev BulletinWe are awarerank_evalis currently not working onGoogle Colab.That's becauseGoogle Colabruns Python 3.6 whilerank_evalcurrently requires a newer Python version.I will try to downgrade the required Python version soon.If you experienced thenumba.typedissue, it should now be solved. Please, re-installrank_eval.⚡️ Introductionrank_evalis a library of fast ranking evaluation metrics implemented inPython, leveragingNumbafor high-speed vector operations and automatic parallelization.It allows you to compare different runs, perform statistical tests, and export a LaTeX table for your scientific publications.We strongly incourage you to check the example folder to learn how to userank_evalin just a few minutes.✨ Available MetricsHitsPrecisionRecallrPrecisionMean Reciprocal Rank (MRR)Mean Average Precision (MAP)Normalized Discounted Cumulative Gain (NDCG)The metrics have been tested againstTREC Evalfor correctness.🔌 Installationpipinstallrank_eval💡 UsageCreate Qrels and Runfromrank_evalimportQrels,Run,evaluateqrels=Qrels()qrels.add_multi(q_ids=["q_1","q_2"],doc_ids=[["doc_12","doc_25"],# q_1 relevant documents["doc_11","doc_2"],# q_2 relevant documents],scores=[[5,3],# q_1 relevance judgements[6,1],# q_2 relevance judgements],)run=Run()run.add_multi(q_ids=["q_1","q_2"],doc_ids=[["doc_12","doc_23","doc_25","doc_36","doc_32","doc_35"],["doc_12","doc_11","doc_25","doc_36","doc_2","doc_35"],],scores=[[0.9,0.8,0.7,0.6,0.5,0.4],[0.9,0.8,0.7,0.6,0.5,0.4],],)Evaluate# Compute score for a single metricevaluate(qrels,run,"ndcg@5")>>>0.7861# Compute scores for multiple metrics at onceevaluate(qrels,run,["map@5","mrr"])>>>{"map@5":0.6416,"mrr":0.75}# Computed metric scores are saved in the Run objectrun.mean_scores>>>{"ndcg@5":0.7861,"map@5":0.6416,"mrr":0.75}# Access scores for each querydict(run.scores)>>>{"ndcg@5":{"q_1":0.9430,"q_2":0.6292},"map@5":{"q_1":0.8333,"q_2":0.4500},"mrr":{"q_1":1.0000,"q_2":0.5000}}Compare# Compare different runs and perform statistical testsreport=compare(qrels=qrels,runs=[run_1,run_2,run_3,run_4,run_5],metrics=["map@100","mrr@100","ndcg@10"],max_p=0.01# P-value threshold)print(report)Output:#    Model    MAP@100     MRR@100     NDCG@10
---  -------  ----------  ----------  ----------
a    model_1  0.3202ᵇ     0.3207ᵇ     0.3684ᵇᶜ
b    model_2  0.2332      0.2339      0.239
c    model_3  0.3082ᵇ     0.3089ᵇ     0.3295ᵇ
d    model_4  0.3664ᵃᵇᶜ   0.3668ᵃᵇᶜ   0.4078ᵃᵇᶜ
e    model_5  0.4053ᵃᵇᶜᵈ  0.4061ᵃᵇᶜᵈ  0.4512ᵃᵇᶜᵈ📖 ExamplesOverview: This notebook shows the main features ofrank_eval.Create Qrels and Run: This notebook shows different ways of creatingQrelsandRun.📚 DocumentationTo be updated! Please, refer to the examples in the meantime.Search thedocumentationfor more details and examples.🎓 CitationIf you userank_evalto evaluate results for your scientific publication, please consider citing it:@misc{rankEval2021,
  title = {Rank\_eval: Blazing Fast Ranking Evaluation Metrics in Python},
  author = {Bassani, Elias},
  year = {2021},
  publisher = {GitHub},
  howpublished = {\url{https://github.com/AmenRa/rank_eval}},
}🎁 Feature RequestsWould you like to see a new metric implemented? Please, open anew issue.🤘 Want to contribute?Would you like to contribute? Please, drop me ane-mail.📄 Licenserank_evalis an open-sourced software licensed under theMIT license.