Package: marqo-haystack
Marqo Document Store for HaystackTable of ContentsMarqo Document Store for HaystackInstallationAboutExamplesUsageUsing LocallyUsing with Marqo CloudLicenseInstallationpip install marqo-haystackAboutThis is a document store integration forMarqowithhaystack.Marqo is an end-to-end vector search engine which includes preprocessing and inference to generate vectors from your data. You can use pre-trained models or bring finetuned ones.Haystack is an end-to-end NLP framework that enables you to build applications powered by LLMs, with haystack you can build end-to-end NLP applications solving your use case using state-of-the-art models.Examplesfrommarqo_haystackimportMarqoDocumentStoredocument_store=MarqoDocumentStore()You can find a code example showing how to use the Document Store and the Retriever under theexample/folder of this repo.UsageFor documentation on Marqo itself, pleaserefer to the documentation.You can use theMarqoDocumentStorein your haystack pipelines for single queries like so:frommarqo_haystackimportMarqoDocumentStorefrommarqo_haystack.retrieverimportMarqoSingleRetrieverdocument_store=MarqoDocumentStore()querying=Pipeline()querying.add_component("retriever",MarqoSingleRetriever(document_store))results=querying.run({"retriever":{"query":"Is black and white text boring?","top_k":3}})Or for a list of queries:frommarqo_haystackimportMarqoDocumentStorefrommarqo_haystack.retrieverimportMarqoRetrieverdocument_store=MarqoDocumentStore()querying=Pipeline()querying.add_component("retriever",MarqoRetriever(document_store))results=querying.run({"retriever":{"queries":["Is black and white text boring?"],"top_k":3}})Using LocallyIf you specify acollection_namethat doesn't exist as a Marqo index then one will be created for you.frommarqo_haystackimportMarqoDocumentStore# Use an existing index (if my-index does exist)document_store=MarqoDocumentStore(collection_name="my-index")# Use an existing index (if my-new-index doesn't exist)document_store=MarqoDocumentStore(collection_name="my-new-index")# Use the default index name, 'documents'. One will be created if it doesn't exist.document_store=MarqoDocumentStore()You can also pass in settings for the index created by the API by passing a dictionary to thesettings_dictparameter. For details on the settings object pleaserefer to the Marqo docs.In this example we specify that the index should use thee5-large-v2model and increase theef_constructionparameter to 512 for the HNSW graph construction.frommarqo_haystackimportMarqoDocumentStoreindex_settings={"index_defaults":{"model":"hf/e5-large-v2","ann_parameters":{"parameters":{"ef_construction":512}}}}document_store=MarqoDocumentStore(settings_dict=index_settings)Using with Marqo CloudThis integration can also be used with Marqo Cloud. You can sign up or access you Marqo Cloud accounthere.To use Marqo Cloud with this integration you will need to pass thecollection_name(index name),url(https://api.marqo.ai), andapi_keyinto the constructor.Note that when using this integration with Marqo Cloud you will need to have already created an index in your Marqo Cloud account.frommarqo_haystackimportMarqoDocumentStoredocument_store=MarqoDocumentStore(url="https://api.marqo.ai",api_key="XXXXXXXXXXXXX",collection_name="my-cloud-index")Licensemarqo-haystackis distributed under the terms of theApache-2.0license.