Package: ir-axioms
↕️ ir_axiomsIntuitive axiomatic retrieval experimentation.ir_axiomsis a Python framework for experimenting with axioms in information retrieval in a declarative way.
It includes reference implementations of many commonly used retrieval axioms and is well integrated with thePyTerrierframework and thePyserinitoolkit.
Re-rank your search results today withir_axiomsand understand your retrieval systems better by analyzing
axiomatic preferences!Presentation video on YouTubePosterUsageTheir_axiomsframework is easy to use. Below, we've prepared some notebooks showcasing the main features.
If you have questions or need assistance, pleasecontatct us.Example NotebooksWe include several example notebooks to demonstrate re-ranking and preference evaluation inPyTerrierusingir_axioms.
You can find all examples in theexamples/directory.Re-ranking top-20 results using KwikSortRe-ranking top-20 results using KwikSort learned from ORACLE preferencesRe-ranking top-20 results using LambdaMART with axiomatic preference featuresPost-Hoc analysis of rankings and relevance judgmentsComputing axiom preferences for top-20 results of TREC 2022 Deep Learning (passage) runs in parallelSIGIR 2022 showcasefor step-by-step explanations with ourpresentation videoBackendsYou can experiment withir_axiomsin PyTerrier and Pyserini.
However, we recommend PyTerrier as not all features are implemented for the Pyserini backend.PyTerrier (Terrier index)To useir_axiomswith a Terrier index, please use our PyTerrier transformers (modules):Transformer ClassTypeDescriptionAggregatedPreferences𝑅 → 𝑅𝑓Aggregate axiom preferences for each documentEstimatorKwikSortReranker𝑅 → 𝑅′Train estimator for ORACLE, use it to re-rank with KwikSort.KwikSortReranker𝑅 → 𝑅′Re-rank using axiom preferences aggregated by KwikSort.PreferenceMatrix𝑅 → (𝑅×𝑅)𝑓Compute an axiom’s preference matrix.You can also directly instantiate a index context object from a Terrier index if you want to build custom axiomatic modules:fromir_axioms.backend.pyterrierimportTerrierIndexContextcontext=TerrierIndexContext("/path/to/index/dir")axiom.preference(context,query,doc1,doc2)Pyserini (Anserini index)We don't have modules for Pyserini to re-rank or analyze results out of the box.
However, you can still comute axiom preferences to integrate retrieval axioms into your search pipeline:fromir_axioms.backend.pyseriniimportAnseriniIndexContextcontext=AnseriniIndexContext("/path/to/index/dir")axiom.preference(context,query,doc1,doc2)TIRAHere's an example howir_axiomscan be used to get axiomatic preferences for a run in TIRA:tira-run\--input-directory${PWD}/data/tira/input-of-re-ranker\--input-run${PWD}/data/tira/output-of-indexer\--output-directory${PWD}/data/tira/output\--imagewebis/ir_axioms:0.2.13\--command'/venv/bin/python -m ir_axioms --offline --terrier-version 5.7 --terrier-helper-version 0.0.7 preferences --run-file $inputDataset/run.jsonl --run-format jsonl --index-dir $inputRun/index --output-dir $outputDir AND ANTI-REG ASPECT-REG DIV LB1 LNC1 LEN-AND LEN-DIV LEN-M-AND LEN-M-TDC LNC1 M-AND M-TDC PROX1 PROX2 PROX3 PROX4 PROX5 REG STMC1 STMC2 TF-LNC TFC1 TFC3'CitationIf you use this package or its components in your research, please cite the following paper describing their_axiomsframework and its use-cases:Alexander Bondarenko, Maik Fröbe, Jan Heinrich Reimer, Benno Stein, Michael Völske, and Matthias Hagen.Axiomatic Retrieval Experimentation withir_axioms. In45th International ACM Conference on Research and Development in Information Retrieval (SIGIR 2022), July 2022. ACM.You can use the following BibTeX entry for citation:@InProceedings{bondarenko:2022d,author={Alexander Bondarenko andMaik Fr{\"o}be and{Jan Heinrich} Reimer andBenno Stein andMichael V{\"o}lske andMatthias Hagen},booktitle={45th International ACM Conference on Research and Developmentin Information Retrieval (SIGIR 2022)},month=jul,publisher={ACM},site={Madrid, Spain},title={{Axiomatic Retrieval Experimentation with ir_axioms}},year=2022}DevelopmentTo buildir_axiomsand contribute to its development you need to install thebuild, andsetuptoolsandwheelpackages:pipinstallbuildsetuptoolswheel(On most systems, these packages are already pre-installed.)InstallationInstall dependencies for developing their_axiomspackage:pipinstall-e.If you want to develop thePyserinibackend, install dependencies like this:pipinstall-e.[pyserini]If you want to develop thePyTerrierbackend, install dependencies like
this:pipinstall-e.[pyterrier]TestingInstall test dependencies:pipinstall-e.[test]Verify your changes against our test suite to verify.flake8ir_axiomstests
pylint-Eir_axiomstests.unit--ignore-paths=^ir_axioms.backend
pytestir_axioms/tests/unit/--ignore=ir_axioms/backend/Please also add tests for the axioms or integrations you've added.Testing backend integrationsInstall test dependencies (replace<BACKEND>with eitherpyseriniorpyterrier):pipinstall-e.[<BACKEND>]Verify your changes against our test suite to verify.pylint-Eir_axioms.backend.<BACKEND>tests.integration.<BACKEND>
pytesttests/integration/<BACKEND>/Build wheelA wheel for this package can be built by running:python-mbuildSupportIf you hit any problems usingir_axiomsor reproducing our experiments, please write us an email or file anissue:jan.reimer@student.uni-halle.demaik.froebe@informatik.uni-halle.dealexander.bondarenko@informatik.uni-halle.deWe're happy to help!LicenseThis repository is released under theMIT license. If you useir_axiomsin your research, we'd be glad if
you'dcite us.AbstractAxiomatic approaches to information retrieval have played a key role in determining basic constraints that characterize good retrieval models. Beyond their importance in retrieval theory, axioms have been operationalized to improve an initial ranking, to “guide” retrieval, or to explain some model’s rankings. However, recent open-source retrieval frameworks like PyTerrier and Pyserini, which made it easy to experiment with sparse and dense retrieval models, have not included any retrieval axiom support so far. To fill this gap, we proposeir_axioms, an open-source Python framework that integrates retrieval axioms with common retrieval frameworks. We include reference implementations for 25 retrieval axioms, as well as components for preference aggregation, re-ranking, and evaluation. New axioms can easily be defined by implementing an abstract data type or by intuitively combining existing axioms with Python operators or regression. Integration with PyTerrier andir_datasetsmakes standard retrieval models, corpora, topics, and relevance judgments—including those used at TREC—immediately accessible for axiomatic experimentation. Our experiments on the TREC Deep Learning tracks showcase some potential research questions that ir_axioms can help to address.