Package: ranx
🔥 News[August 3 2023]ranx0.3.16is out!This release adds support for importing Qrels and Runs fromparquetfiles, exporting them aspandas.DataFrameand save them asparquetfiles.
Any dependence ontrec_evalhave been removed to makeranxtruly MIT-compliant.⚡️ Introductionranx([raŋks]) is a library of fast ranking evaluation metrics implemented inPython, leveragingNumbafor high-speedvector operationsandautomatic parallelization.
It offers a user-friendly interface to evaluate and compareInformation RetrievalandRecommender Systems.ranxallows you to perform statistical tests and exportLaTeXtables for your scientific publications.
Moreover,ranxprovides severalfusion algorithmsandnormalization strategies, and an automaticfusion optimizationfunctionality.ranxalso have a companion repository of pre-computed runs to facilitated model comparisons calledranxhub.
Onranxhub, you can download and share pre-computed runs for Information Retrieval datasets, such asMSMARCO Passage Ranking.ranxwas featured inECIR 2022,CIKM 2022, andSIGIR 2023.If you useranxto evaluate results or conducting experiments involving fusion for your scientific publication, please consider citing it:evaluation bibtex,fusion bibtex,ranxhub bibtex.NB:ranxis not suited for evaluating classifiers. Please, refer to theFAQfor further details.For a quick overview, follow theUsagesection.For a in-depth overview, follow theExamplessection.✨ FeaturesMetricsHitsHit RatePrecisionRecallF1r-PrecisionBprefRank-biased Precision (RBP)Mean Reciprocal Rank (MRR)Mean Average Precision (MAP)Discounted Cumulative Gain (DCG)Normalized Discounted Cumulative Gain (NDCG)The metrics have been tested againstTREC Evalfor correctness.Statistical TestsPaired Student's t-Test(default)Fisher's Randomization TestTukey's HSD TestPlease, refer toSmucker et al.,Carterette, andFuhrfor additional information on statistical tests for Information Retrieval.Off-the-shelf QrelsYou can load qrels fromir-datasetsas simply as:qrels=Qrels.from_ir_datasets("msmarco-document/dev")A full list of the available qrels is providedhere.Off-the-shelf RunsYou can load runs fromranxhubas simply as:run=Run.from_ranxhub("run-id")A full list of the available runs is providedhere.Fusion AlgorithmsNameNameNameNameNameCombMINCombMNZRRFMAPFuseBordaFuseCombMEDCombGMNZRBCPosFuseWeighted BordaFuseCombANZISRWMNZProbFuseCondorcetCombMAXLog_ISRMixedSegFuseWeighted CondorcetCombSUMLogN_ISRBayesFuseSlideFuseWeighted SumPlease, refer to thedocumentationfor further details.Normalization StrategiesMin-Max NormMax NormSum NormZMUV NormRank NormBorda NormPlease, refer to thedocumentationfor further details.🔌 Requirementspython>=3.8As ofv.0.3.5,ranxrequirespython>=3.8.💾 Installationpipinstallranx💡 UsageCreate Qrels and RunfromranximportQrels,Runqrels_dict={"q_1":{"d_12":5,"d_25":3},"q_2":{"d_11":6,"d_22":1}}run_dict={"q_1":{"d_12":0.9,"d_23":0.8,"d_25":0.7,"d_36":0.6,"d_32":0.5,"d_35":0.4},"q_2":{"d_12":0.9,"d_11":0.8,"d_25":0.7,"d_36":0.6,"d_22":0.5,"d_35":0.4}}qrels=Qrels(qrels_dict)run=Run(run_dict)Evaluatefromranximportevaluate# Compute score for a single metricevaluate(qrels,run,"ndcg@5")>>>0.7861# Compute scores for multiple metrics at onceevaluate(qrels,run,["map@5","mrr"])>>>{"map@5":0.6416,"mrr":0.75}Comparefromranximportcompare# Compare different runs and perform Two-sided Paired Student's t-Testreport=compare(qrels=qrels,runs=[run_1,run_2,run_3,run_4,run_5],metrics=["map@100","mrr@100","ndcg@10"],max_p=0.01# P-value threshold)Output:print(report)#    Model    MAP@100    MRR@100    NDCG@10
---  -------  --------   --------   ---------
a    model_1  0.320ᵇ     0.320ᵇ     0.368ᵇᶜ
b    model_2  0.233      0.234      0.239
c    model_3  0.308ᵇ     0.309ᵇ     0.330ᵇ
d    model_4  0.366ᵃᵇᶜ   0.367ᵃᵇᶜ   0.408ᵃᵇᶜ
e    model_5  0.405ᵃᵇᶜᵈ  0.406ᵃᵇᶜᵈ  0.451ᵃᵇᶜᵈFusionfromranximportfuse,optimize_fusionbest_params=optimize_fusion(qrels=train_qrels,runs=[train_run_1,train_run_2,train_run_3],norm="min-max",# The norm. to apply before fusionmethod="wsum",# The fusion algorithm to use (Weighted Sum)metric="ndcg@100",# The metric to maximize)combined_test_run=fuse(runs=[test_run_1,test_run_2,test_run_3],norm="min-max",method="wsum",params=best_params,)📖 ExamplesNameLinkOverviewQrels and RunEvaluationComparison and ReportFusionPlotShare your runs withranxhub📚 DocumentationBrowse thedocumentationfor more details and examples.🎓 CitationIf you useranxto evaluate results for your scientific publication, please consider citing ourECIR 2022paper:BibTeX@inproceedings{ranx,author={Elias Bassani},title={ranx: {A} Blazing-Fast Python Library for Ranking Evaluation and Comparison},booktitle={{ECIR} {(2)}},series={Lecture Notes in Computer Science},volume={13186},pages={259--264},publisher={Springer},year={2022},doi={10.1007/978-3-030-99739-7\_30}}If you use the fusion functionalities provided byranxfor conducting the experiments of your scientific publication, please consider citing ourCIKM 2022paper:BibTeX@inproceedings{ranx.fuse,author={Elias Bassani andLuca Romelli},title={ranx.fuse: {A} Python Library for Metasearch},booktitle={{CIKM}},pages={4808--4812},publisher={{ACM}},year={2022},doi={10.1145/3511808.3557207}}If you use pre-computed runs fromranxhubto make comparison for your scientific publication, please consider citing ourSIGIR 2023paper:BibTeX@inproceedings{ranxhub,author={Elias Bassani},title={ranxhub: An Online Repository for Information Retrieval Runs},booktitle={{SIGIR}},pages={3210--3214},publisher={{ACM}},year={2023},doi={10.1145/3539618.3591823}}🎁 Feature RequestsWould you like to see other features implemented? Please, open afeature request.🤘 Want to contribute?Would you like to contribute? Please, drop me ane-mail.📄 Licenseranxis an open-sourced software licensed under theMIT license.