Package: ir-datasets
ir_datasetsir_datasetsis a python package that provides a common interface to many IR ad-hoc ranking
benchmarks, training datasets, etc.The package takes care of downloading datasets (including documents, queries, relevance judgments,
etc.) when available from public sources. Instructions on how to obtain datasets are provided when
they are not publicly available.ir_datasetsprovides a common iterator format to allow them to be easily used in python. It
attempts to provide the data in an unaltered form (i.e., keeping all fields and markup), while
handling differences in file formats, encoding, etc. Adapters provide extra functionality, e.g., to
allow quick lookups of documents by ID.A command line interface is also available.You can find a list of datasets and their featureshere.
Want a new dataset, added functionality, or a bug fixed? Feel free to post an issue or make a pull request!Getting StartedFor a quick start with the Python API, check out our Colab tutorials:PythonCommand LineInstall via pip:pip install ir_datasetsIf you want the main branch, you install as such:pip install git+https://github.com/allenai/ir_datasets.gitIf you want to build from source, use:$ git clone https://github.com/allenai/ir_datasets
$ cd ir_datasets
$ python setup.py bdist_wheel
$ pip install dist/ir_datasets-*.whlTested with python versions 3.7, 3.8, 3.9, and 3.10. (Mininum python version is 3.7.)FeaturesPython and Command Line Interfaces. Access datasts both through a simple Python API and
via the command line.importir_datasetsdataset=ir_datasets.load('msmarco-passage/train')# Documentsfordocindataset.docs_iter():print(doc)# GenericDoc(doc_id='0', text='The presence of communication amid scientific minds was equa...# GenericDoc(doc_id='1', text='The Manhattan Project and its atomic bomb helped bring an en...# ...ir_datasetsexportmsmarco-passage/traindocs|head-n20ThepresenceofcommunicationamidscientificmindswasequallyimportanttothesuccessoftheManh...1TheManhattanProjectanditsatomicbombhelpedbringanendtoWorldWarII.Itslegacyofpeacefu...Automatically downloads source files(when available). Will download and verify the source
files for queries, documents, qrels, etc. when they are publicly available, as they are needed.
A CI build checks weekly to ensure that all the downloadable content is available and correct:.
We mirror some troublesome files onmirror.ir-datasets.com, and
automatically switch to the mirror when the original source is not available.importir_datasetsdataset=ir_datasets.load('msmarco-passage/train')fordocindataset.docs_iter():# Will download and extract MS-MARCO's collection.tar.gz the first time...forqueryindataset.queries_iter():# Will download and extract MS-MARCO's queries.tar.gz the first time...Instructions for dataset access(when not publicly available). Provides instructions on how
to get a copy of the data when it is not publicly available online (e.g., when it requires a
data usage agreement).importir_datasetsdataset=ir_datasets.load('trec-arabic')fordocindataset.docs_iter():...# Provides the following instructions:# The dataset is based on the Arabic Newswire corpus. It is available from the LDC via: <https://catalog.ldc.upenn.edu/LDC2001T55># To proceed, symlink the source file here: [gives path]Support for datasets big and small. By using iterators, supports large datasets that may
not fit into system memory, such as ClueWeb.importir_datasetsdataset=ir_datasets.load('clueweb09')fordocindataset.docs_iter():...# will iterate through all ~1B documentsFixes known dataset issues. For instance, automatically corrects the document UTF-8 encoding
problem in the MS-MARCO passage collection.importir_datasetsdataset=ir_datasets.load('msmarco-passage')docstore=dataset.docs_store()docstore.get('243').text# "John Maynard Keynes, 1st Baron Keynes, CB, FBA (/ˈkeɪnz/ KAYNZ; 5 June 1883 – 21 April [SNIP]"# Naïve UTF-8 decoding yields double-encoding artifacts like:# "John Maynard Keynes, 1st Baron Keynes, CB, FBA (/Ë\x88keÉªnz/ KAYNZ; 5 June 1883 â\x80\x93 21 April [SNIP]"#                                                  ~~~~~~  ~~                       ~~~~~~~~~Fast Random Document Access.Builds data structures that allow fast and efficient lookup of
document content. For large datasets, such as ClueWeb, usescheckpoint filesto load documents from
source 40x faster than normal. Results are cached for even faster subsequent accesses.importir_datasetsdataset=ir_datasets.load('clueweb12')docstore=dataset.docs_store()docstore.get_many(['clueweb12-0000tw-05-00014','clueweb12-0000tw-05-12119','clueweb12-0106wb-18-19516'])# {'clueweb12-0000tw-05-00014': ..., 'clueweb12-0000tw-05-12119': ..., 'clueweb12-0106wb-18-19516': ...}Fancy Iter Slicing.Sometimes it's helpful to be able to select ranges of data (e.g., for processing
document collections in parallel on multiple devices). Efficient implementations of slicing operations
allow for much faster dataset partitioning than usingitertools.slice.importir_datasetsdataset=ir_datasets.load('clueweb12')dataset.docs_iter()[500:1000]# normal slicing behavior# WarcDoc(doc_id='clueweb12-0000tw-00-00502', ...), WarcDoc(doc_id='clueweb12-0000tw-00-00503', ...), ...dataset.docs_iter()[-10:-8]# includes negative indexing# WarcDoc(doc_id='clueweb12-1914wb-28-24245', ...), WarcDoc(doc_id='clueweb12-1914wb-28-24246', ...)dataset.docs_iter()[::100]# includes support for skip (only positive values)# WarcDoc(doc_id='clueweb12-0000tw-00-00000', ...), WarcDoc(doc_id='clueweb12-0000tw-00-00100', ...), ...dataset.docs_iter()[1/3:2/3]# supports proportional slicing (this takes the middle third of the collection)# WarcDoc(doc_id='clueweb12-0605wb-28-12714', ...), WarcDoc(doc_id='clueweb12-0605wb-28-12715', ...), ...DatasetsAvailable datasets include:ANTIQUEAQUAINTBEIR (benchmark suite)TREC CARC4ClueWeb09ClueWeb12CLIRMatrixCodeSearchNetCORD-19DPR Wiki100GOVGOV2HC4Highwire (TREC Genomics 2006-07)MedlineMSMARCO (document)MSMARCO (passage)MSMARCO (QnA)Natural QuestionsNFCorpus (NutritionFacts)NYTPubMed Central (TREC CDS)TREC ArabicTREC Fair Ranking 2021TREC MandarinTREC Robust 2004TREC SpanishTripClickTweets 2013 (Internet Archive)VaswaniWashington PostWikIRThere are "subsets" under each dataset. For instance,clueweb12/b13/trec-misinfo-2019provides the
queries and judgments from the2019 TREC misinformation track,
andmsmarco-document/orcasprovides theORCAS dataset. They
tend to be organized with the document collection at the top level.See the ir_dataets docs (ir_datasets.com) for details about each
dataset, its available subsets, and what data they provide.Environment variablesIR_DATASETS_HOME: Home directory for ir_datasets data (default~/.ir_datasets/). Contains directories
for each top-level dataset.IR_DATASETS_TMP: Temporary working directory (default/tmp/ir_datasets/).IR_DATASETS_DL_TIMEOUT: Download stream read timeout, in seconds (default15). If no data is received
within this duration, the connection will be assumed to be dead, and another download may be attempted.IR_DATASETS_DL_TRIES: Default number of download attempts before exception is thrown (default3).
When the server accepts Range requests, uses them. Otherwise, will download the entire file againIR_DATASETS_DL_DISABLE_PBAR: Set totrueto disable the progress bar for downloads. Useful in settings
where an interactive console is not available.IR_DATASETS_DL_SKIP_SSL: Set totrueto disable checking SSL certificates when downloading files.
Useful as a short-term solution when SSL certificates expire or are otherwise invalid. Note that this
does not disable hash verification of the downloaded content.IR_DATASETS_SKIP_DISK_FREE: Set totrueto disable checks for enough free space on disk before
downloading content or otherwise creating large files.IR_DATASETS_SMALL_FILE_SIZE: The size of files that are considered "small", in bytes. Instructions for
linking small files rather then downloading them are not shown. Defaults to 5000000 (5MB).CitingWhen using datasets provided by this package, be sure to properly cite them. Bibtex for each dataset
can be found on thedatasets documentation page.If you use this tool, please citeour SIGIR resource paper:@inproceedings{macavaney:sigir2021-irds,
  author = {MacAvaney, Sean and Yates, Andrew and Feldman, Sergey and Downey, Doug and Cohan, Arman and Goharian, Nazli},
  title = {Simplified Data Wrangling with ir_datasets},
  year = {2021},
  booktitle = {SIGIR}
}CreditsContributors to this repository:Sean MacAvaney (University of Glasgow)Shuo Sun (Johns Hopkins University)Thomas Jänich (University of Glasgow)Jan Heinrich Reimer (Martin Luther University Halle-Wittenberg)Maik Fröbe (Martin Luther University Halle-Wittenberg)Eugene Yang (Johns Hopkins University)Augustin Godinot (NAVERLABS Europe, ENS Paris-Saclay)