Package: retriv
🔥 News[August 23, 2023]retriv0.2.2 is out!This release addsexperimentalsupport for multi-field documents and filters.
Please, refer toAdvanced Retrieverdocumentation.[February 18, 2023]retriv0.2.0 is out!This release adds support for Dense and Hybrid Retrieval.
Dense Retrieval leverages the semantic similarity of the queries' and documents' vector representations, which can be computed directly byretrivor imported from other sources.
Hybrid Retrieval mix traditional retrieval, informally called Sparse Retrieval,  and Dense Retrieval results to further improve retrieval effectiveness.
As the library was almost completely redone, indices built with previous versions are no longer supported.⚡️ Introductionretrivis a user-friendly and efficientsearch engineimplemented inPythonsupporting Sparse (traditional search withBM25,TF-IDF), Dense (semantic search) and Hybrid retrieval (a mix of Sparse and Dense Retrieval).
It allows you to build a search engine in asingle line of code.retrivis built uponNumbafor high-speedvector operationsandautomatic parallelization,PyTorchandTransformersfor easy access and usage ofTransformer-based Language Models, andFaissfor approximatenearest neighbor search.
In addition, it provides automatic tuning functionalities to allow you to tune its internal components with minimal intervention.✨ Main FeaturesRetrieversSparse Retriever: standard searcher based on lexical matching.retrivimplementsBM25as its main retrieval model.TF-IDFis also supported for educational purposes.
The sparse retriever comes armed with multiplestemmers,tokenizers, andstop-wordlists, for multiple languages.
Clickhereto learn more.Dense Retriever: a dense retriever is a retrieval model that performssemantic search.
Clickhereto learn more.Hybrid Retriever: an hybrid retriever is a retrieval model built on top of a sparse and a dense retriever.
Clickhereto learn more.Advanced Retriever: an advanced sparse retriever supporting filters. This is and experimental feature.
Clickhereto learn more.Unified Search InterfaceAll the supported retrievers share the same search interface:search: standard search functionality, what you expect by a search engine.msearch: computes the results for multiple queries at once.
It leveragesautomatic parallelizationwhenever possible.bsearch: similar tomsearchbut automatically generates batches of queries to evaluate and allows dynamic writing of the search results to disk inJSONlformat.bsearchis handy for computing results for hundreds of thousands or even millions of queries without hogging your RAM. Pre-computed results can be leveraged for negative sampling during the training ofNeural ModelsforInformation Retrieval.AutoTuneretrivautomatically tunesFaissconfiguration for approximate nearest neighbors search by leveragingAutoFaissto guarantee 10ms response time based on your available hardware.
Moreover, it offers an automatic tuning functionality forBM25's parameters, which require minimal user intervention.
Under the hood,retrivleveragesOptuna, ahyperparameter optimizationframework, andranx, anInformation Retrievalevaluation library, to test several parameter configurations forBM25and choose the best one.
Finally, it can automatically balance the importance of lexical and semantic relevance scores computed by theHybrid Retrieverto maximize retrieval effectiveness.📚 DocumentationSparse RetrieverDense RetrieverHybrid RetrieverText Pre-ProcessingFAQ🔌 Requirementspython>=3.8💾 Installationpipinstallretriv💡 Minimal Working Example# Note: SearchEngine is an alias for the SparseRetrieverfromretrivimportSearchEnginecollection=[{"id":"doc_1","text":"Generals gathered in their masses"},{"id":"doc_2","text":"Just like witches at black masses"},{"id":"doc_3","text":"Evil minds that plot destruction"},{"id":"doc_4","text":"Sorcerer of death's construction"},]se=SearchEngine("new-index").index(collection)se.search("witches masses")Output:[{"id":"doc_2","text":"Just like witches at black masses","score":1.7536403},{"id":"doc_1","text":"Generals gathered in their masses","score":0.6931472}]🎁 Feature RequestsWould you like to see other features implemented? Please, open afeature request.🤘 Want to contribute?Would you like to contribute? Please, drop me ane-mail.📄 Licenseretrivis an open-sourced software licensed under theMIT license.