Package: ir-measures
ir_measuresNew: Explore IR measures using our demo atdemo.ir-measur.es!Check out our documentation website:ir-measur.esProvides a common interface to many IR measure tools.Provided by theTerrier Team @ Glasgow. Find us atterrierteam/ir_measures.Getting StartedInstall via pippipinstallir-measuresPython APIimportir_measuresfromir_measuresimport*# imports all supported measures, e.g., AP, nDCG, RR, Pqrels={'Q0':{"D0":0,"D1":1},"Q1":{"D0":0,"D3":2}}run={'Q0':{"D0":1.2,"D1":1.0},"Q1":{"D0":2.4,"D3":3.6}}# aggregated resultsir_measures.calc_aggregate([AP,nDCG,RR,nDCG@10,P(rel=2)@10],qrels,run)# {AP: 0.75, nDCG: 0.8154648767857288, RR: 0.75, nDCG@10: 0.8154648767857288, P(rel=2)@10: 0.05}# by queryforminir_measures.iter_calc([AP,nDCG,RR,nDCG@10,P(rel=2)@10],qrels,run):print(m)# Metric(query_id='Q0', measure=AP, value=0.5)# Metric(query_id='Q0', measure=RR, value=0.5)# Metric(query_id='Q0', measure=nDCG, value=0.6309297535714575)# Metric(query_id='Q0', measure=nDCG@10, value=0.6309297535714575)# Metric(query_id='Q1', measure=AP, value=1.0)# Metric(query_id='Q1', measure=RR, value=1.0)# Metric(query_id='Q1', measure=nDCG, value=1.0)# Metric(query_id='Q1', measure=nDCG@10, value=1.0)# Metric(query_id='Q0', measure=P(rel=2)@10, value=0.0)# Metric(query_id='Q1', measure=P(rel=2)@10, value=0.1)Qrels can be provided in the following formats:# dict of dictqrels={'Q0':{"D0":0,"D1":1,},"Q1":{"D0":0,"D3":2}}# dataframeimportpandasaspdqrels=pd.DataFrame([{'query_id':"Q0",'doc_id':"D0",'relevance':0},{'query_id':"Q0",'doc_id':"D1",'relevance':1},{'query_id':"Q1",'doc_id':"D0",'relevance':0},{'query_id':"Q1",'doc_id':"D3",'relevance':2},])# any iterable of namedtuples (e.g., list, generator, etc)qrels=[ir_measures.Qrel("Q0","D0",0),ir_measures.Qrel("Q0","D1",1),ir_measures.Qrel("Q1","D0",0),ir_measures.Qrel("Q1","D3",2),]# TREC-formatted qrels fileqrels=ir_measures.read_trec_qrels('qrels.txt')# qrels from the ir_datasets package (https://ir-datasets.com/)importir_datasetsqrels=ir_datasets.load('trec-robust04').qrels_iter()Runs can be provided in the following formats:# dict of dictrun={'Q0':{"D0":1.2,"D1":1.0,},"Q1":{"D0":2.4,"D3":3.6}}# dataframeimportpandasaspdrun=pd.DataFrame([{'query_id':"Q0",'doc_id':"D0",'score':1.2},{'query_id':"Q0",'doc_id':"D1",'score':1.0},{'query_id':"Q1",'doc_id':"D0",'score':2.4},{'query_id':"Q1",'doc_id':"D3",'score':3.6},])# any iterable of namedtuples (e.g., list, generator, etc)run=[ir_measures.ScoredDoc("Q0","D0",1.2),ir_measures.ScoredDoc("Q0","D1",1.0),ir_measures.ScoredDoc("Q1","D0",2.4),ir_measures.ScoredDoc("Q1","D3",3.6),]Command Line Interfaceir_measures also functions as a command line interface, with syntax similar to
trec_eval.Example:ir_measures/path/to/qrels/path/to/runP@10'P(rel=2)@5 nDCG@15 Judged@10'NumQNumRelNumRetNumRelRet
P@100.4382
P(rel=2)@50.0827
nDCG@150.4357
Judged@100.9812
NumQ249.0000
NumRel17412.0000
NumRet241339.0000
NumRet(rel=1)10272.0000Syntax:ir_measures qrels run measures... [-q] [-n] [-p 4]qrels: a TREC-formatted QRELS filerun: a TREC-formatted results filemeasures: one or more measure name strings. Note that in bash,()must be in single quotes. For simplicity, you can provide multiple meaures in a single quotation, which are split on whitespace.-q: provide results for each query individually-n: when used with-q, skips summary statistics-p: number of decimal places to report results (default: 4)DocumentationMeasuresProvidersCreditsSean MacAvaney, University of GlasgowCraig Macdonald, University of GlasgowCharlie Clarke, University of WaterlooBenjamin Piwowarski, CNRSHarry Scells, Leipzig University