Package: compare-qrels
compare-qrelsQualitatively compare the qrels results of two IR systems.Why use this?Determining whether one has progressed in IR is often done based on improvement on mean metrics - the mean of certain
metrics across queries - such as MRR, micro-recall or micro-precision. If a new model slightly significantly
outperforms an old model on these mean metrics, it is often concluded that the new model is better.
Statistical significance is insufficient to prove that one system outperforms another in offline
evaluation, because the characteristics of your query set may not perfectly reflect queries from "real" users.
If the new model fails at queries that the old model could handle very well and if those types of queries
are overrepresented in queries  of "real" users, your system may have significantly outperform in an
offline setting, but fails to transfer those results into an online setting.
Researchers should take this possibility into account byqualitatively assessing the strengths and weaknessesbetween different methods and models.Mean metrics can obscure the following characteristics on systems:Specialisation: in the figure the models perform well on different types of queries, i.e.  specialisation. Comparing these models
is difficult or nonsensical, because they could serve different goals/domains/queries. Specialisation can be quickly
observed when plotting, for instance,
the ∆recall in a histogram.  If most queries have a high positive or negative ∆recall, then
the models are specialised on different queries.Distributed performance: the figure shows a situation in which model A performs well on all queries whereas
model  B  performs  perfect  on  some  queries,  but  fails  at  others.   Concluding  that  model  B  is
better than model A could be premature.  When Conversational Passage Ranking (CPR) is being used
in  a  ConvQA  system  to  fulfil  a  user’s  information  need,  a  few  documents  might  be  enough  to  provide
an  answer  for  each  query.   Then  it  is  preferred  that  a  Conversational  Passage  Ranking  (CPR)  system
performs well on a wide range of queries instead of specialising on a few.  An example of such a system is asearch engine like Google.  As a user, it is preferred to have some relevant documents for all your queries.Distributed performance can be quickly observed when plotting, for instance, the recall in a histogramlike in figure 4c.  If most queries have a medium recall score instead of either a high or a low score, themodel performs distributively overall.Search space overlap:  the recall scores of two systems hide the information about which documents
are retrieved. This information can be gained by computing the overlap of retrieved documents of two
models. The figure illustrates the case in which the recall scores as the same, but the overlap is different.  In
the left case, the models find roughly the same documents, but model A finds more relevant documents
than model B, i.e.  the results of A are a super set of the results of B. Then it is fair to conclude that
model A is better than model B. However, if the overlap is low as illustrated in the right case, both models
find different relevant documents, i.e.  the models are complementary.  When the goal of a system is total
recall, for instance in legal IR, not one model but both models should be used.How to useSeesample_compare_qrels.py.Example outputThe example output is the output of comparing two Conversational Passage Retrieval systems.FiguresTerminal outputMeanscoresQuReTec(trained){'map':0.15798155213698378,'recip_rank':0.4623152325205058,'recall_1000':0.5586410184663372}MeanscoresQueryRewriting{'map':0.18083664899945048,'recip_rank':0.47347866813888656,'recall_1000':0.611948585919043}52easyqueries(recall_1000>0.8)forbothmethods:8withlowoverlap(<40.0%),38withhighoverlap(>=60.0%)and6withmediumoverlap(>=40.0%and<60.0%)qidpercentagetopicrecall_1000_1recall_1000_2031_1100.000Whatisthroatcancer?0.9330.933431_587.300Canitspreadtothethroat?0.9310.9481132_3100.000Howdotheycomparewithtigersforbeingdang...0.9900.9902033_1100.000TellmeabouttheNeverendingStoryfilm.0.8700.8702133_228.300Whatisitabout?1.0001.0002233_328.100Howwasitreceived?0.8121.0002433_534.200Wasitabookfirst?1.0001.0002834_1100.000TellmeabouttheBronzeAgecollapse.0.8450.8452934_2100.000Whatistheevidenceforit?0.9380.9383034_3100.000Whataresomeofthepossiblecauses?0.9000.9003534_856.000Whatempiressurvived?0.9521.0003637_1100.000WhatwastheStanfordExperiment?0.9660.9663737_2100.000Whatdiditshow?0.9550.9553937_4100.000Wasitethical?1.0001.0004137_680.100WhathappenedintheMilgramexperiment?0.9861.0004237_7100.000Whywasitimportant?1.0001.0004337_861.900Whatwerethesimilaritiesanddifferencesbet...1.0000.9664640_371.200Whattechnologicaldevelopmentsenabledit?0.8890.8065040_7100.000Whatmakesasongpoppunk?0.8480.8485140_837.400Whatisthedifferencebetweenitandemo?1.0001.0005449_355.900WhatisitsrelationshipwithBlockbuster?0.9050.9525549_4100.000WhendidNetflixshiftfromDVDstoastreamin...0.9130.9135749_6100.000HowdoesitcomparetoAmazonPrimeVideo?0.9830.9835949_8100.000HowhasitchangedthewayTViswatched?0.8020.8026050_1100.000Whatwasthefirstartificialsatellite?0.9740.9746650_7100.000WhatareCubesats?1.0001.0007054_3100.000WhyistheNationalAirandSpaceMuseumimpor...0.9200.9207154_4100.000IstheSpyMuseumfree?1.0001.0008356_745.900Compareandcontrastmicroevolutionandmacroe...0.9810.9439258_885.700Howisitusedinmobileapps?1.0000.9299359_1100.000Whichweekendsportshavethemostinjuries?0.9140.9149559_347.900WhatistheACL?0.8720.8729659_4100.000Whatisaninjuryforit?0.9860.9869959_734.300Whatisarnicausedfor?1.0001.00010461_4100.000WhatistherelationshipofSpider-Mantothe...0.9070.90710967_1100.000Whyisbloodred?0.9300.93013169_1100.000Howdoyousleepafterjetlag?1.0001.00014175_1100.000WhydoturkeyandTurkeysharethesamename?0.9410.94114375_333.800Whatwastheirimportanceinnativecultures?0.8950.84214475_463.600Whenandhowweretheydomesticated?0.8610.91714575_564.500Cantheyfly?0.9600.96014675_622.800WhydidBenFranklinwantittobethenationa...1.0001.00014777_1100.000What's the difference between soup and stew?          0.974          0.974151  77_5     100.000                             How is cassoulet made?          1.000          1.000154  77_8      17.200                             Tell about Bigos stew.          1.000          0.906155  78_1     100.000                             What is the keto diet?          0.860          0.860157  78_3      54.500                                   What is ketosis?          0.870          0.981161  78_7     100.000                      What is intermittent fasting?          1.000          1.000164  79_2      42.700    What is the main contribution of Auguste Comte?          0.897          0.888165  79_3     100.000              What is the role of positivism in it?          0.830          0.830166  79_4     100.000                 What is Herbert Spencer known for?          0.949          0.949171  79_9     100.000       What are modern examples of conflict theory?          0.900          0.90027 hard queries (recall_1000 < 0.2) for both methods:10 with low overlap (<40.0%), 14 with high overlap (>=60.0%) and 3 with medium overlap (>=40.0% and <60.0%)qid  percentage                                              topic  recall_1000_1  recall_1000_21     31_2       0.000                                   Is it treatable?          0.039          0.00010    32_2     100.000                                  What do they eat?          0.000          0.00012    32_4     100.000      Are sharks endangered?  If so, which species?          0.000          0.00013    32_5     100.000                   Tell me more about tiger sharks.          0.093          0.09315    32_7     100.000                    What'sthebiggestevercaught?0.1250.1251832_1030.500Whataretheiradaptations?0.1880.1254740_4100.000Whenandwhydidpeoplestarttakingpopserio...0.0610.0615349_20.000Howdiditoriginallywork?0.0000.0007756_1100.000WhatisDarwin’stheoryinanutshell?0.0090.0097856_27.900Howwasitdeveloped?0.0140.0729759_586.600TellmeabouttheRICEmethod.0.1250.16710261_20.000Tellmeabouttheirfirstappearance.0.0000.00011067_2100.000Whatfoodscontainhighlevelsofiron?0.0000.00011167_317.300Whatimprovesabsorption?0.0000.00011367_561.000Howaretheycreated?0.1190.14811467_640.900Howisoxygentransported?0.0380.01311767_9100.000Canitgoaway?0.1140.11411867_1092.000Whatareitspossiblecauses?0.0000.00011967_1150.700Howisittreated?0.0940.07512168_27.500Whatisthehistoryoftagliatellealragubol...0.0000.11812368_40.900Tellmeaboutcookingschoolsandclasses.0.0000.06812468_566.100Whatarefamousfoodsfromtheregion?0.0430.02212568_6100.000Describethetraditionalprocessformakingba...0.0000.00012668_71.600Whatismortadellaandwhereisitfrom?0.0000.11612768_891.500What’sthedifferencewithBologna?0.0000.00012968_105.700Whatisdonewiththewheyafterproduction?0.0000.04315077_449.900WhatarepopularonesinFrance?0.0000.04223querieswhereQueryRewritingoutperformsQuReTec(trained)(Δrecall_1000>0.3):17withlowoverlap(<40.0%),3withhighoverlap(>=60.0%)and3withmediumoverlap(>=40.0%and<60.0%)qidpercentagetopicrecall_1000_1recall_1000_2531_622.600Whatcausesthroatcancer?0.4920.889631_721.300Whatisthefirstsignofit?0.5860.948831_934.600What'sthedifferenceintheirsymptoms?0.6361.0002633_732.700Whatarethemainthemes?0.5561.0003434_763.600Whataboutenvironmentalfactors?0.2810.9694540_222.500Whatareitscharacteristics?0.2220.7225649_554.800Whatareitsothercompetitors?0.5360.8577354_615.600Whatisthebesttimetovisitthereflecting...0.2501.0007454_70.600Arethereanyfamousfoods?0.0000.3337654_947.300Tellmeaboutitshistory.0.0000.44410059_821.200Whataresomewaystoavoidinjury?0.4110.82111667_81.000Whatarethesymptoms?0.0000.31913269_212.700Howaboutforanxiety?0.0000.89713469_41.700Howwasitdiscovered?0.0000.54213769_717.400Whatarethesideeffects?0.0000.80013869_852.500WhydoesitrequireaprescriptionintheUK?0.0000.63614069_1060.600Isiteffectivefortreatinginsomnia?0.0620.75015377_792.900Howisitsimilarordifferentfromcassoulet?0.3430.88615978_510.200Whatdotheyhaveincommon?0.0000.62516078_60.000Howaretheydifferent?0.0000.75016278_837.300Howisitrelatedtoketo?0.0001.00016779_513.500HowishisworkrelatedtoComte?0.2681.00016979_75.900Whatisitsmaincriticism?0.0430.56511querieswhereQuReTec(trained)outperformsQueryRewriting(Δrecall_1000>0.3):9withlowoverlap(<40.0%),0withhighoverlap(>=60.0%)and2withmediumoverlap(>=40.0%and<60.0%)qidpercentagetopicrecall_1000_1recall_1000_2331_426.300Whatareitssymptoms?0.6730.2041732_93.400Tellmeaboutmakos.1.0000.0003334_623.700Whatotherfactorsledtoabreakdownoftrade?0.5330.1005849_711.100Describeit’ssubscribergrowthovertime.0.7450.3146750_80.300Whataretheiradvantages?1.0000.0009859_651.100Istheredisagreementaboutit?1.0000.00010861_80.400Whoaretheimportantmembers?0.6790.05412268_31.400Whatareitscommonvariations?0.7500.11114275_257.800Whereareturkeysfrom?0.8460.46214977_32.500Howaboutgoulash?1.0000.53815678_20.000Whywasitoriginaldeveloped?0.4380.0002complementaryqueries:1withlowoverlap(<40.0%),0withhighoverlap(>=60.0%)and1withmediumoverlap(>=40.0%and<60.0%)qidpercentagetopicrecall_1000_1recall_1000_2recall_1000_combined4840_522.000Howhasitbeenintegratedintomusiceducation?0.2780.3330.3894940_658.500Describesomeoftheinfluentialpopbands.0.7160.6790.741