Package: beir
Paper|Installation|Quick Example|Datasets|Wiki|Hugging Face:beers: What is it?BEIRis aheterogeneous benchmarkcontaining diverse IR tasks. It also provides acommon and easy frameworkfor evaluation of your NLP-based retrieval models within the benchmark.Foran overview, checkout ournew wikipage:https://github.com/beir-cellar/beir/wiki.Formodels and datasets, checkout outHugging Face (HF)page:https://huggingface.co/BeIR.ForLeaderboard, checkout outEval AIpage:https://eval.ai/web/challenges/challenge-page/1897.For more information, checkout out our publications:BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models(NeurIPS 2021, Datasets and Benchmarks Track)Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard(Arxiv 2023):beers: InstallationInstall via pip:pipinstallbeirIf you want to build from source, use:$gitclonehttps://github.com/beir-cellar/beir.git$cdbeir$pipinstall-e.Tested with python versions 3.6 and 3.7:beers: FeaturesPreprocess your own IR dataset or use one of the already-preprocessed 17 benchmark datasetsWide settings included, covers diverse benchmarks useful for both academia and industryIncludes well-known retrieval architectures (lexical, dense, sparse and reranking-based)Add and evaluate your own model in a easy framework using different state-of-the-art evaluation metrics:beers: Quick ExampleFor other example codes, please refer to ourExamples and TutorialsWiki page.frombeirimportutil,LoggingHandlerfrombeir.retrievalimportmodelsfrombeir.datasets.data_loaderimportGenericDataLoaderfrombeir.retrieval.evaluationimportEvaluateRetrievalfrombeir.retrieval.search.denseimportDenseRetrievalExactSearchasDRESimportloggingimportpathlib,os#### Just some code to print debug information to stdoutlogging.basicConfig(format='%(asctime)s-%(message)s',datefmt='%Y-%m-%d%H:%M:%S',level=logging.INFO,handlers=[LoggingHandler()])#### /print debug information to stdout#### Download scifact.zip dataset and unzip the datasetdataset="scifact"url="https://public.ukp.informatik.tu-darmstadt.de/thakur/BEIR/datasets/{}.zip".format(dataset)out_dir=os.path.join(pathlib.Path(__file__).parent.absolute(),"datasets")data_path=util.download_and_unzip(url,out_dir)#### Provide the data_path where scifact has been downloaded and unzippedcorpus,queries,qrels=GenericDataLoader(data_folder=data_path).load(split="test")#### Load the SBERT model and retrieve using cosine-similaritymodel=DRES(models.SentenceBERT("msmarco-distilbert-base-tas-b"),batch_size=16)retriever=EvaluateRetrieval(model,score_function="dot")# or "cos_sim" for cosine similarityresults=retriever.retrieve(corpus,queries)#### Evaluate your model with NDCG@k, MAP@K, Recall@K and Precision@K  where k = [1,3,5,10,100,1000]ndcg,_map,recall,precision=retriever.evaluate(qrels,results,retriever.k_values):beers: Available DatasetsCommand to generate md5hash using Terminal:md5sum filename.zip.You can view all datasets availablehereor onHugging Face.DatasetWebsiteBEIR-NamePublic?TypeQueriesCorpusRel D/QDown-loadmd5MSMARCOHomepagemsmarco✅traindevtest6,9808.84M1.1Link444067daf65d982533ea17ebd59501e4TREC-COVIDHomepagetrec-covid✅test50171K493.5Linkce62140cb23feb9becf6270d0d1fe6d1NFCorpusHomepagenfcorpus✅traindevtest3233.6K38.2Linka89dba18a62ef92f7d323ec890a0d38dBioASQHomepagebioasq❌traintest50014.91M4.7NoHow to Reproduce?NQHomepagenq✅traintest3,4522.68M1.2Linkd4d3d2e48787a744b6f6e691ff534307HotpotQAHomepagehotpotqa✅traindevtest7,4055.23M2.0Linkf412724f78b0d91183a0e86805e16114FiQA-2018Homepagefiqa✅traindevtest64857K2.6Link17918ed23cd04fb15047f73e6c3bd9d9Signal-1M(RT)Homepagesignal1m❌test972.86M19.6NoHow to Reproduce?TREC-NEWSHomepagetrec-news❌test57595K19.6NoHow to Reproduce?Robust04Homepagerobust04❌test249528K69.9NoHow to Reproduce?ArguAnaHomepagearguana✅test1,4068.67K1.0Link8ad3e3c2a5867cdced806d6503f29b99Touche-2020Homepagewebis-touche2020✅test49382K19.0Link46f650ba5a527fc69e0a6521c5a23563CQADupstackHomepagecqadupstack✅test13,145457K1.4Link4e41456d7df8ee7760a7f866133bda78QuoraHomepagequora✅devtest10,000523K1.6Link18fb154900ba42a600f84b839c173167DBPediaHomepagedbpedia-entity✅devtest4004.63M38.2Linkc2a39eb420a3164af735795df012ac2cSCIDOCSHomepagescidocs✅test1,00025K4.9Link38121350fc3a4d2f48850f6aff52e4a9FEVERHomepagefever✅traindevtest6,6665.42M1.2Link5a818580227bfb4b35bb6fa46d9b6c03Climate-FEVERHomepageclimate-fever✅test1,5355.42M3.0Link8b66f0a9126c521bae2bde127b4dc99dSciFactHomepagescifact✅traintest3005K1.1Link5f7d1de60b170fc8027bb7898e2efca1:beers: Additional InformationWe also provide a variety of additional information in ourWikipage.
Please refer to these pages for the following:Quick StartInstalling BEIRExamples and TutorialsDatasetsDatasets AvailableMultilingual DatasetsLoad your Custom DatasetModelsModels AvailableEvaluate your Custom ModelMetricsMetrics AvailableMiscellaneousBEIR LeaderboardCouse Material on IR:beers: DisclaimerSimilar to Tensorflowdatasetsor Hugging Face'sdatasetslibrary, we just downloaded and prepared public datasets. We only distribute these datasets in a specific format, but we do not vouch for their quality or fairness, or claim that you have license to use the dataset. It remains the user's responsibility to determine whether you as a user have permission to use the dataset under the dataset's license and to cite the right owner of the dataset.If you're a dataset owner and wish to update any part of it, or do not want your dataset to be included in this library, feel free to post an issue here or make a pull request!If you're a dataset owner and wish to include your dataset or model in this library, feel free to post an issue here or make a pull request!:beers: Citing & AuthorsIf you find this repository helpful, feel free to cite our publicationBEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models:@inproceedings{
    thakur2021beir,
    title={{BEIR}: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models},
    author={Nandan Thakur and Nils Reimers and Andreas R{\"u}ckl{\'e} and Abhishek Srivastava and Iryna Gurevych},
    booktitle={Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)},
    year={2021},
    url={https://openreview.net/forum?id=wCu6T5xFjeJ}
}If you use any baseline score from the BEIR leaderboard, feel free to cite our publicationResources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard@misc{kamalloo2023resources,
      title={Resources for Brewing BEIR: Reproducible Reference Models and an Official Leaderboard}, 
      author={Ehsan Kamalloo and Nandan Thakur and Carlos Lassance and Xueguang Ma and Jheng-Hong Yang and Jimmy Lin},
      year={2023},
      eprint={2306.07471},
      archivePrefix={arXiv},
      primaryClass={cs.IR}
}The main contributors of this repository are:Nandan Thakur, Personal Website:nandan-thakur.comContact person: Nandan Thakur,nandant@gmail.comDon't hesitate to send us an e-mail or report an issue, if something is broken (and it shouldn't be) or if you have further questions.This repository contains experimental software and is published for the sole purpose of giving additional background details on the respective publication.:beers: CollaborationThe BEIR Benchmark has been made possible due to a collaborative effort of the following universities and organizations:UKP Lab, Technical University of DarmstadtUniversity of WaterlooHugging Face:beers: ContributorsThanks go to all these wonderful collaborations for their contribution towards the BEIR benchmark:Nandan ThakurNils ReimersIryna GurevychJimmy LinAndreas RückléAbhishek Srivastava