Pinecone Vector Store - Hybrid Search
Source: llama_index

<a href="https://colab.research.google.com/github/run-llama/llama_index/blob/main/docs/docs/examples/vector_stores/PineconeIndexDemo-Hybrid.ipynb" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a># Pinecone Vector Store - Hybrid SearchIf you're opening this Notebook on colab, you will probably need to install LlamaIndex 🦙.```python%pip install llama-index-vector-stores-pinecone``````python!pip install llama-index>=0.9.31 pinecone-client>=3.0.0 "transformers[torch]"```#### Creating a Pinecone Index```pythonimport loggingimport syslogging.basicConfig(stream=sys.stdout, level=logging.INFO)logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))``````pythonfrom pinecone import Pinecone, ServerlessSpec``````pythonimport osos.environ[    "PINECONE_API_KEY"] = #"<Your Pinecone API key, from app.pinecone.io>"os.environ[    "OPENAI_API_KEY"] = "sk-..."api_key = os.environ["PINECONE_API_KEY"]pc = Pinecone(api_key=api_key)``````python# delete if needed# pc.delete_index("quickstart")``````python# dimensions are for text-embedding-ada-002# NOTE: needs dotproduct for hybrid searchpc.create_index(    name="quickstart",    dimension=1536,    metric="dotproduct",    spec=ServerlessSpec(cloud="aws", region="us-west-2"),)# If you need to create a PodBased Pinecone index, you could alternatively do this:## from pinecone import Pinecone, PodSpec## pc = Pinecone(api_key='xxx')## pc.create_index(#  name='my-index',#  dimension=1536,#  metric='cosine',#  spec=PodSpec(#  environment='us-east1-gcp',#  pod_type='p1.x1',#  pods=1#  )# )#``````pythonpinecone_index = pc.Index("quickstart")```Download Data```python!mkdir -p 'data/paul_graham/'!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/paul_graham/paul_graham_essay.txt' -O 'data/paul_graham/paul_graham_essay.txt'```#### Load documents, build the PineconeVectorStore```pythonfrom llama_index.core import VectorStoreIndex, SimpleDirectoryReaderfrom llama_index.vector_stores.pinecone import PineconeVectorStorefrom IPython.display import Markdown, display``````python# load documentsdocuments = SimpleDirectoryReader("./data/paul_graham/").load_data()``````python# set add_sparse_vector=True to compute sparse vectors during upsertfrom llama_index.core import StorageContextif "OPENAI_API_KEY" not in os.environ:    raise EnvironmentError(f"Environment variable OPENAI_API_KEY is not set")vector_store = PineconeVectorStore(    pinecone_index=pinecone_index,    add_sparse_vector=True,)storage_context = StorageContext.from_defaults(vector_store=vector_store)index = VectorStoreIndex.from_documents(    documents, storage_context=storage_context)```    INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"    HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"    Upserted vectors:   0%|          | 0/22 [00:00<?, ?it/s]#### Query IndexMay need to wait a minute or two for the index to be ready```python# set Logging to DEBUG for more detailed outputsquery_engine = index.as_query_engine(vector_store_query_mode="hybrid")response = query_engine.query("What happened at Viaweb?")```    INFO:httpx:HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"    HTTP Request: POST https://api.openai.com/v1/embeddings "HTTP/1.1 200 OK"    INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"    HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"```pythondisplay(Markdown(f"<b>{response}</b>"))```<b>At Viaweb, Lisp was used as a programming language. The speaker gave a talk at a Lisp conference about how Lisp was used at Viaweb, and afterward, the talk gained a lot of attention when it was posted online. This led to a realization that publishing essays online could reach a wider audience than traditional print media. The speaker also wrote a collection of essays, which was later published as a book called "Hackers & Painters."</b>