Package: voyage-embedders-haystack
Voyage Embedders - HaystackCustom component forHaystack(2.x) for creating embeddings using theVoyageAI Embedding Models.Voyage’s embedding models,voyage-2andvoyage-2-code, are state-of-the-art in retrieval accuracy. These models outperform top performing embedding models likeintfloat/e5-mistral-7b-instructandOpenAI/text-embedding-3-largeon theMTEB Benchmark.voyage-2is current ranked second on theMTEB Leaderboard.What's New[v1.2.0 - 02/02/24]:Breaking Change:VoyageDocumentEmbedderandVoyageTextEmbeddernow accept themodelparameter instead ofmodel_name.The embedders have been use the newvoyageai.Client.embed()method instead of the deprecatedget_embeddingandget_embeddingsmethods of the global namespace.Support for the newtruncateparameter has been added.Default embedding model has been changed to "voyage-2" from the deprecated "voyage-01".The embedders now return the total number of tokens used as part of the"total_tokens"in the metadata.[v1.1.0 - 13/12/23]:Added support forinput_typeparameter inVoyageTextEmbedderandVoyageDocument Embedder.[v1.0.0 - 21/11/23]:AddedVoyageTextEmbedderandVoyageDocument Embedderto embed strings and documents.Installationpipinstallvoyage-embedders-haystackUsageYou can use Voyage Embedding models with two components:VoyageTextEmbedderandVoyageDocumentEmbedder.To create semantic embeddings for documents, useVoyageDocumentEmbedderin your indexing pipeline. For generating embeddings for queries, useVoyageTextEmbedder. Once you've selected the suitable component for your specific use case, initialize the component with the model name and VoyageAI API key. You can also
set the environment variable "VOYAGE_API_KEY" instead of passing the api key as an argument.Information about the supported models, can be found on theEmbeddings Documentation.To get an API key, please see theVoyage AI website.ExampleBelow is the example Semantic Search pipeline that uses theSimple WikipediaDataset from HuggingFace. You can find more examples in theexamplesfolder.Load the dataset:# Install HuggingFace Datasets using "pip install datasets"fromdatasetsimportload_datasetfromhaystackimportPipelinefromhaystack.components.retrievers.in_memoryimportInMemoryEmbeddingRetrieverfromhaystack.components.writersimportDocumentWriterfromhaystack.dataclassesimportDocumentfromhaystack.document_stores.in_memoryimportInMemoryDocumentStore# Import Voyage Embeddersfromvoyage_embedders.voyage_document_embedderimportVoyageDocumentEmbedderfromvoyage_embedders.voyage_text_embedderimportVoyageTextEmbedder# Load first 100 rows of the Simple Wikipedia Dataset from HuggingFacedataset=load_dataset("pszemraj/simple_wikipedia",split="validation[:100]")docs=[Document(content=doc["text"],meta={"title":doc["title"],"url":doc["url"],},)fordocindataset]Index the documents to theInMemoryDocumentStoreusing theVoyageDocumentEmbedderandDocumentWriter:doc_store=InMemoryDocumentStore(embedding_similarity_function="cosine")doc_embedder=VoyageDocumentEmbedder(model="voyage-2",input_type="document",)# Indexing Pipelineindexing_pipeline=Pipeline()indexing_pipeline.add_component(instance=doc_embedder,name="DocEmbedder")indexing_pipeline.add_component(instance=DocumentWriter(document_store=doc_store),name="DocWriter")indexing_pipeline.connect(connect_from="DocEmbedder",connect_to="DocWriter")indexing_pipeline.run({"DocEmbedder":{"documents":docs}})print(f"Number of documents in Document Store:{len(doc_store.filter_documents())}")print(f"First Document:{doc_store.filter_documents()[0]}")print(f"Embedding of first Document:{doc_store.filter_documents()[0].embedding}")Query the Semantic Search Pipeline using theInMemoryEmbeddingRetrieverandVoyageTextEmbedder:text_embedder=VoyageTextEmbedder(model="voyage-2",input_type="query")# Query Pipelinequery_pipeline=Pipeline()query_pipeline.add_component("TextEmbedder",text_embedder)query_pipeline.add_component("Retriever",InMemoryEmbeddingRetriever(document_store=doc_store))query_pipeline.connect("TextEmbedder","Retriever")# Searchresults=query_pipeline.run({"TextEmbedder":{"text":"Which year did the Joker movie release?"}})# Print text from top resulttop_result=results["Retriever"]["documents"][0].contentprint("The top search result is:")print(top_result)ContributingPull requests are welcome. For major changes, please open an issue first
to discuss what you would like to change.AuthorAshwin MathurLicensevoyage-embedders-haystackis distributed under the terms of theApache-2.0 license.