Package: pinecone-text
Pinecone Text ClientThe Pinecone Text Client is a Python package that provides text utilities designed for seamless integration with Pinecone'ssparse-dense(hybrid) semantic search.⚠️ WarningThis is apublic preview("Beta") version.For any issues or requests, please reach out to oursupportteam.InstallationTo install the Pinecone Text Client, use the following command:pipinstallpinecone-textIf you wish to useSpladeEncoder, you will need to install thespladeextra:pipinstallpinecone-text[splade]If you wish to useSentenceTransformerEncoderdense encoder, you will need to install thedenseextra:pipinstallpinecone-text[dense]If you wish to useOpenAIEncoderdense encoder, you will need to install theopenaiextra:pipinstallpinecone-text[openai]Sparse EncodingTo convert your own text corpus to sparse vectors, you can either useBM25orSPLADE.BM25To encode your documents and queries using BM25 as vector for dot product search, you can use theBM25Encoderclass.📝 NOTE:Our current implementation of BM25 supports only static document frequency (meaning that the document frequency values are precomputed and fixed, and do not change dynamically based on new documents added to the collection).When conducting a search, you may come across queries that contain terms not found in the training corpus but are present in the database. To address this scenario, BM25Encoder uses a default document frequency value of 1 when encoding such terms.UsageFor an end-to-end example, you can refer to our Quora dataset generation with BM25notebook.frompinecone_text.sparseimportBM25Encodercorpus=["The quick brown fox jumps over the lazy dog","The lazy dog is brown","The fox is brown"]# Initialize BM25 and fit the corpusbm25=BM25Encoder()bm25.fit(corpus)# Encode a new document (for upsert to Pinecone index)doc_sparse_vector=bm25.encode_documents("The brown fox is quick")# {"indices": [102, 18, 12, ...], "values": [0.22, 0.38, 0.15, ...]}# Encode a query (for search in Pinecone index)query_sparse_vector=bm25.encode_queries("Which fox is brown?")# {"indices": [102, 16, 18, ...], "values": [0.22, 0.11, 0.15, ...]}# store BM25 params as jsonbm25.dump("bm25_params.json")# load BM25 params from jsonbm25.load("bm25_params.json")Load Default ParametersIf you want to use the default parameters forBM25Encoder, you can call thedefaultmethod.
The default parameters were fitted on theMS MARCOpassage ranking dataset.frompinecone_text.sparseimportBM25Encoderbm25=BM25Encoder.default()BM25 ParametersTheBM25Encoderclass offers configurable parameters to customize the encoding:b: Controls document length normalization (default: 0.75).k1: Controls term frequency saturation (default: 1.2).Tokenization Options: Allows customization of the tokenization process, including options for handling case, punctuation, stopwords, stemming, and language selection.These parameters can be specified when initializing the BM25Encoder class. Please read the BM25Encoder documentation for more details.SPLADECurrently theSpladeEncoderclass supprts only thenaver/splade-cocondenser-ensembledistilmodel, and followsSPLADE V2implementation.📝 NOTE:Currently pinecone text not supoorts SPLADE with python 3.12 due to compatibility issues with pytorchUsageFor an end-to-end example, you can refer to our Quora dataset generation with SPLADEnotebook.Note: If cuda is available, the model will automatically run on GPU. You can explicitly override the device using thedeviceparameter in the constructor.frompinecone_text.sparseimportSpladeEncoder# Initialize Spladesplade=SpladeEncoder()# encode a batch of documentsdocuments=["The quick brown fox jumps over the lazy dog","The lazy dog is brown","The fox is brown"]document_vectors=splade.encode_documents(documents)# [{"indices": [102, 18, 12, ...], "values": [0.21, 0.38, 0.15, ...]}, ...]# encode a queryquery="Which fox is brown?"query_vectors=splade.encode_queries(query)# {"indices": [102, 18, 12, ...], "values": [0.21, 0.38, 0.15, ...]}Dense EncodingFor dense embedding we also provide a thin wrapper for the following models:All Sentence Transformers models hosted on huggingfaceSee full list of modelsAll OpenAI API supported embedding modelsSee full list of modelsSentence Transformers modelsWhen usingSentenceTransformerEncoder, the models are downloaded from huggingface and run locally. Also, if cuda is available, the model will automatically run on GPU. You can explicitly override the device using thedeviceparameter in the constructor.📝 NOTE:Currently pinecone text not supoorts sentence transformers with python 3.12 due to compatibility issues with pytorchUsagefrompinecone_text.denseimportSentenceTransformerEncoderencoder=SentenceTransformerEncoder("sentence-transformers/all-MiniLM-L6-v2")encoder.encode_documents(["The quick brown fox jumps over the lazy dog"])# [[0.21, 0.38, 0.15, ...]]encoder.encode_queries(["Who jumped over the lazy dog?"])# [[0.11, 0.43, 0.67, ...]]OpenAI modelsWhen using theOpenAIEncoder, you need to provide an API key for the OpenAI API, and store it in theOPENAI_API_KEYenvironment variable before you import the encoder.By default the encoder will usetext-embedding-3-smallas recommended by OpenAI. You can also specify a different model name using themodel_nameparameter.Usagefrompinecone_text.denseimportOpenAIEncoderencoder=OpenAIEncoder()# defaults to the recommended model - "text-embedding-3-small"encoder.encode_documents(["The quick brown fox jumps over the lazy dog"])# [[0.21, 0.38, 0.15, ...]]encoder.encode_queries(["Who jumped over the lazy dog?"])# [[0.11, 0.43, 0.67, ...]]Pinecone text also supports Azure OpenAI API. To use it, you need to import theAzureOpenAIEncoderclass instead ofOpenAIEncoder. You also need to pass Azure specific environment variables to the constructor, along with your specific embeddings  deployment as the model name. For more information please follow theAzureOpenAIEncoderdocumentation.Jina AI modelsWhen using theJinaEncoder, you need to provide an API key for the Jina Embeddings API, and store it in theJINA_API_KEYenvironment variable before you import the encoder.By default the encoder will usejina-embeddings-v2-base-en. You can also specify a different model name using themodel_nameparameter.Usagefrompinecone_text.denseimportJinaEncoderencoder=JinaEncoder()encoder.encode_documents(["The quick brown fox jumps over the lazy dog"])# [[-0.62586284, -0.54578537, 0.5570845, ...]]encoder.encode_queries(["Who jumped over the lazy dog?"])# [[-0.43374294, -0.42069837, 0.773763, ...]]Combining Sparse and Dense Encodings for Hybrid SearchTo combine sparse and dense encodings for hybrid search, you can use thehybrid_convex_scalemethod on your query.This method receives both a dense vector and a sparse vector, along with a convex scaling parameteralpha. It returns a tuple consisting of the scaled dense and sparse vectors according to the following formula:(alpha * dense_vector, (1 - alpha) * sparse_vector).frompinecone_text.hybridimporthybrid_convex_scalefrompinecone_text.sparseimportSpladeEncoderfrompinecone_text.denseimportSentenceTransformerEncoder# Initialize Spladesplade=SpladeEncoder()# Initialize Sentence Transformersentence_transformer=SentenceTransformerEncoder("sentence-transformers/all-MiniLM-L6-v2")# encode a querysparse_vector=splade.encode_queries("Which fox is brown?")dense_vector=sentence_transformer.encode_queries("Which fox is brown?")# combine sparse and dense vectorshybrid_dense,hybrid_sparse=hybrid_convex_scale(dense_vector,sparse_vector,alpha=0.8)# ([-0.21, 0.38, 0.15, ...], {"indices": [102, 16, 18, ...], "values": [0.21, 0.11, 0.18, ...]})