Package: cherche
ChercheNeural searchCherche enables the development of a neural search pipeline that employs retrievers and pre-trained language models both as retrievers and rankers. The primary advantage of Cherche lies in its capacity to construct end-to-end pipelines. Additionally, Cherche is well-suited for offline semantic search due to its compatibility with batch computation.Live demo of a NLP search engine powered by ChercheInstallation 🤖To install Cherche for use with a simple retriever on CPU, such as TfIdf, Flash, Lunr, Fuzz, use the following command:pipinstallchercheTo install Cherche for use with any semantic retriever or ranker on CPU, use the following command:pipinstall"cherche[cpu]"Finally, if you plan to use any semantic retriever or ranker on GPU, use the following command:pipinstall"cherche[gpu]"By following these installation instructions, you will be able to use Cherche with the appropriate requirements for your needs.DocumentationDocumentation is availablehere. It provides details
about retrievers, rankers, pipelines and examples.QuickStart 📑DocumentsCherche allows findings the right document within a list of objects. Here is an example of a corpus.fromchercheimportdatadocuments=data.load_towns()documents[:3][{'id':0,'title':'Paris','url':'https://en.wikipedia.org/wiki/Paris','article':'Paris is the capital and most populous city of France.'},{'id':1,'title':'Paris','url':'https://en.wikipedia.org/wiki/Paris','article':"Since the 17th century, Paris has been one of Europe's major centres of science, and arts."},{'id':2,'title':'Paris','url':'https://en.wikipedia.org/wiki/Paris','article':'The City of Paris is the centre and seat of government of the region and province of Île-de-France.'}]Retriever rankerHere is an example of a neural search pipeline composed of a TF-IDF that quickly retrieves documents, followed by a ranking model. The ranking model sorts the documents produced by the retriever based on the semantic similarity between the query and the documents. We can call the pipeline using a list of queries and get relevant documents for each query.fromchercheimportdata,retrieve,rankfromsentence_transformersimportSentenceTransformer# List of dictsdocuments=data.load_towns()# Retrieve on fields title and articleretriever=retrieve.TfIdf(key="id",on=["title","article"],documents=documents,k=30)# Rank on fields title and articleranker=rank.Encoder(key="id",on=["title","article"],encoder=SentenceTransformer("sentence-transformers/all-mpnet-base-v2").encode,k=3,)# Pipeline creationsearch=retriever+rankersearch.add(documents=documents)# Search documents for 3 queries.search(["Bordeaux","Paris","Toulouse"])[[{'id':57,'similarity':0.69513524},{'id':63,'similarity':0.6214994},{'id':65,'similarity':0.61809087}],[{'id':16,'similarity':0.59158516},{'id':0,'similarity':0.58217555},{'id':1,'similarity':0.57944715}],[{'id':26,'similarity':0.6925601},{'id':37,'similarity':0.63977146},{'id':28,'similarity':0.62772334}]]We can map the index to the documents to access their contents using pipelines:search+=documentssearch(["Bordeaux","Paris","Toulouse"])[[{'id':57,'title':'Bordeaux','url':'https://en.wikipedia.org/wiki/Bordeaux','similarity':0.69513524},{'id':63,'title':'Bordeaux','similarity':0.6214994},{'id':65,'title':'Bordeaux','url':'https://en.wikipedia.org/wiki/Bordeaux','similarity':0.61809087}],[{'id':16,'title':'Paris','url':'https://en.wikipedia.org/wiki/Paris','article':'Paris received 12.','similarity':0.59158516},{'id':0,'title':'Paris','url':'https://en.wikipedia.org/wiki/Paris','similarity':0.58217555},{'id':1,'title':'Paris','url':'https://en.wikipedia.org/wiki/Paris','similarity':0.57944715}],[{'id':26,'title':'Toulouse','url':'https://en.wikipedia.org/wiki/Toulouse','similarity':0.6925601},{'id':37,'title':'Toulouse','url':'https://en.wikipedia.org/wiki/Toulouse','similarity':0.63977146},{'id':28,'title':'Toulouse','url':'https://en.wikipedia.org/wiki/Toulouse','similarity':0.62772334}]]RetrieveCherche providesretrieversthat filter input documents based on a query.retrieve.TfIdfretrieve.Lunrretrieve.Flashretrieve.Encoderretrieve.DPRretrieve.Fuzzretrieve.EmbeddingRankCherche providesrankersthat filter documents in output of retrievers.Cherche rankers are compatible withSentenceTransformersmodels which are available onHugging Face hub.rank.Encoderrank.DPRrank.CrossEncoderrank.EmbeddingQuestion answeringCherche provides modules dedicated to question answering. These modules are compatible with Hugging Face's pre-trained models and fully integrated into neural search pipelines.Contributors 🤝Cherche was created for/by Renault and is now available to all.
We welcome all contributions.Acknowledgements 👏TfIdf retriever is a wrapper aroundscikit-learn's TfidfVectorizer. Lunr retriever is a wrapper aroundLunr.py. Flash retriever is a wrapper aroundFlashText. DPR, Encode and CrossEncoder rankers are wrappers dedicated to the use of the pre-trained models ofSentenceTransformersin a neural search pipeline.CitationsIf you use cherche to produce results for your scientific publication, please refer to our SIGIR paper:@inproceedings{Sourty2022sigir,author={Raphael Sourty and Jose G. Moreno and Lynda Tamine and Francois-Paul Servant},title={CHERCHE: A new tool to rapidly implement pipelines in information retrieval},booktitle={Proceedings of SIGIR 2022},year={2022}}Dev Team 💾The Cherche dev team is made up ofRaphaël Sourty,François-Paul Servant,Nicolas Bizzozzero,Jose G Moreno. 🥳