Package: chatnoir-api
🔍 chatnoir-apiSimple, type-safe access to theChatNoirsearch API.Working with PyTerrier? Check out thechatnoir-pyterrierpackage.InstallationInstall the package from PyPI:pipinstallchatnoir-apiUsageThe ChatNoir API offers two main features:searchwith BM25F andretrieving document contents.SearchTo search with the ChatNoir API you need to request anAPI key.
Then you can use our Python client to search for documents.
Theresultsobject is an iterable wrapper of the search results which handles pagination for you.
List-style indexing is supported to access individual results or sub-lists of results:fromchatnoir_api.v1importsearchapi_key:str="<API_KEY>"results=search(api_key,"python library")top10_results=results[:10]print(top10_results)result_1234=results[1234]print(result_1234)Search the new ChatNoirThere's anewChatNoir version with the same API interface. To run your search requests against the new API (e.g., if you want to search the ClueWeb22), setstaging=Truelike this:fromchatnoir_apiimportIndexfromchatnoir_api.v1importsearchapi_key:str="<API_KEY>"results=search(api_key,"python library",staging=True,index=Index.ClueWeb22)Note for Touché 2023 participants: Setindex=Index.ClueWeb22to search the ClueWeb22 index. (Otherwise, results from the ClueWeb09 and ClueWeb12 indices will be included.)Phrase SearchTo search for phrases, use thesearch_phrasesmethod in the same way as normalsearch:fromchatnoir_api.v1importsearch_phrasesapi_key:str="<API_KEY>"results=search_phrases(api_key,"python library",staging=True)ChatTo generate text with the ChatNoir Chat API you need to request an API key from theadmins.
With your API key, you can chat with the cat, like this:fromchatnoir_api.chatimportChatNoirChatClientchat_client=ChatNoirChatClient(api_key="<API_KEY>")response=chat_client.chat("how are you?")Retrieve Document ContentsOften the title and ID of a document is not enough to effectively re-rank a list of search results.
To retrieve the full content or plain text for a given document you can use thehtml_contentshelper function.
Thehtml_contentsfunction expects a ChatNoir-internal UUID, shorthand UUID, or a TREC ID
and the index from which to retrieve the document.Retrieve by TREC IDYou can retrieve a document by its TREC ID like this:fromchatnoir_apiimportcache_contents,Indexcontents=cache_contents("clueweb09-en0051-90-00849",Index.ClueWeb09,)print(contents)plain_contents=cache_contents("clueweb09-en0051-90-00849",Index.ClueWeb09,plain=True,)print(plain_contents)Retrieve by ChatNoir-internal UUIDYou can also retrieve a document by its ChatNoir-internal UUID like this:fromuuidimportUUIDfromchatnoir_apiimportcache_contents,Indexcontents=cache_contents(UUID("e635baa8-7341-596a-b3cf-b33c05954361"),Index.CommonCrawl1511,)print(contents)plain_contents=cache_contents(UUID("e635baa8-7341-596a-b3cf-b33c05954361"),Index.CommonCrawl1511,plain=True,)print(plain_contents)Retrieve by ChatNoir-internal short UUIDFor newer ChatNoir versions, you can also retrieve a document by its ChatNoir-internalshortUUID like this:fromchatnoir_apiimportcache_contents,Index,ShortUUIDcontents=cache_contents(ShortUUID("6svePe3PXteDeGPk1XqTLA"),Index.ClueWeb22,staging=True,)print(contents)plain_contents=cache_contents(ShortUUID("6svePe3PXteDeGPk1XqTLA"),Index.ClueWeb22,plain=True,staging=True,)print(plain_contents)CitationIf you use this package, please cite thepaperfrom theChatNoirauthors.
You can use the following BibTeX information for citation:@InProceedings{bevendorff:2018,address={Berlin Heidelberg New York},author={Janek Bevendorff and Benno Stein and Matthias Hagen and Martin Potthast},booktitle={Advances in Information Retrieval. 40th European Conference on IR Research (ECIR 2018)},editor={Leif Azzopardi and Allan Hanbury and Gabriella Pasi and Benjamin Piwowarski},month=mar,publisher={Springer},series={Lecture Notes in Computer Science},site={Grenoble, France},title={{Elastic ChatNoir: Search Engine for the ClueWeb and the Common Crawl}},year=2018}DevelopmentTo build this package and contribute to its development you need to install thebuild, andsetuptoolsandwheelpackages:pipinstallbuildsetuptoolswheel(On most systems, these packages are already pre-installed.)InstallationInstall package and test dependencies:pipinstall-e.[test]TestingConfigure the API keys for testing:exportCHATNOIR_API_KEY="<API_KEY>"exportCHATNOIR_API_KEY_STAGING="<API_KEY>"exportCHATNOIR_API_KEY_CHAT="<API_KEY>"Verify your changes against the test suite to verify.flake8chatnoir_apitestsexamples
pylint-Echatnoir_apitestsexamples
pytestchatnoir_apitestsexamplesPlease also add tests for your newly developed code.Build wheelsWheels for this package can be built with:python-mbuildSupportIf you hit any problems using this package, please file anissue.
We're happy to help!LicenseThis repository is released under theMIT license.