Response Modes

Right now, we support the following options:
* "refine": ***create and refine*** an answer by sequentially going
  through each retrieved text chunk. This makes a separate LLM call
  per Node/retrieved chunk.
  **Details:** the first chunk is used in a query using the
  "text_qa_template" prompt. Then the answer and the next chunk (as
  well as the original question) are used in another query with the
  "refine_template" prompt. And so on until all chunks have been
  parsed.
  If a chunk is too large to fit within the window (considering the
  prompt size), it is split using a "TokenTextSplitter" (allowing some
  text overlap between chunks) and the (new) additional chunks are
  considered as chunks of the original chunks collection (and thus
  queried with the "refine_template" as well).
  Good for more detailed answers.
* "compact" (default): similar to "refine" but ***compact***
  (concatenate) the chunks beforehand, resulting in less LLM calls.
  **Details:** stuff as many text (concatenated/packed from the
  retrieved chunks) that can fit within the context window
  (considering the maximum prompt size between "text_qa_template" and
  "refine_template"). If the text is too long to fit in one prompt, it
  is split in as many parts as needed (using a "TokenTextSplitter" and
  thus allowing some overlap between text chunks).
  Each text part is considered a "chunk" and is sent to the "refine"
  synthesizer.
  In short, it is like "refine", but with less LLM calls.
* "tree_summarize": Query the LLM using the "summary_template" prompt
  as many times as needed so that all concatenated chunks have been
  queried, resulting in as many answers that are themselves
  recursively used as chunks in a "tree_summarize" LLM call and so on,
  until there's only one chunk left, and thus only one final answer.
  **Details:** concatenate the chunks as much as possible to fit
  within the context window using the "summary_template" prompt, and
  split them if needed (again with a "TokenTextSplitter" and some text
  overlap). Then, query each resulting chunk/split against
  "summary_template" (there is no ***refine*** query !) and get as
  many answers.
  If there is only one answer (because there was only one chunk), then
  it's the final answer.
  If there are more than one answer, these themselves are considered
  as chunks and sent recursively to the "tree_summarize" process
  (concatenated/splitted-to-fit/queried).
  Good for summarization purposes.
* "simple_summarize": Truncates all text chunks to fit into a single
  LLM prompt. Good for quick summarization purposes, but may lose
  detail due to truncation.
* "no_text": Only runs the retriever to fetch the nodes that would
  have been sent to the LLM, without actually sending them. Then can
  be inspected by checking "response.source_nodes".
* "accumulate": Given a set of text chunks and the query, apply the
  query to each text chunk while accumulating the responses into an
  array. Returns a concatenated string of all responses. Good for when
  you need to run the same query separately against each text chunk.
* "compact_accumulate": The same as accumulate, but will "compact"
  each LLM prompt similar to "compact", and run the same query against
  each text chunk.
See Response Synthesizer to learn more.


Module Guides

Basic
First, check out our module guide on Indexes for in-depth guides for
each index (vector index, summary index, knowledge graph index). Each
index corresponds to a default query engine for that index.
Then check out the rest of the sections below.
* Custom Query Engine
* Retriever Query Engine
Structured & Semi-Structured Data
* JSON Query Engine
* Pandas Query Engine
* Knowledge Graph Query Engine
* Knowledge Graph RAG Query Engine
Advanced
* Router Query Engine
* Retriever Router Query Engine
* Joint QA Summary Query Engine
* Sub Question Query Engine
* Multi-Step Query Engine
* SQL Router Query Engine
* SQL Auto Vector Query Engine
* SQL Join Query Engine
* [Beta] Text-to-SQL with PGVector
* SQL Query Engine with LlamaIndex + DuckDB
* Retry Query Engine
* Retry Source Query Engine
* Retry Guideline Query Engine
* CitationQueryEngine
* Recursive Retriever + Query Engine Demo
* Joint Tabular/Semantic QA over Tesla 10K
* Recursive Retriever + Document Agents
* Ensemble Query Engine Guide
Advanced: Towards Multi-Document Querying/Analysis
This specific subsection showcases modules that help with querying
multiple documents.
* Sub Question Query Engine
* Recursive Retriever + Document Agents
* Multi-Document Agents
* Multi-Document Agents (V1)
Experimental
* FLARE Query Engine


Query Engine

Concept
Query engine is a generic interface that allows you to ask question
over your data.
A query engine takes in a natural language query, and returns a rich
response. It is most often (but not always) built on one or many
Indices via Retrievers. You can compose multiple query engines to
achieve more advanced capability.
Tip:
  If you want to have a conversation with your data (multiple back-
  and-forth instead of a single question & answer), take a look at
  Chat Engine
Usage Pattern
Get started with:
   query_engine = index.as_query_engine()
   response = query_engine.query("Who is Paul Graham.")
To stream response:
   query_engine = index.as_query_engine(streaming=True)
   streaming_response = query_engine.query("Who is Paul Graham.")
   streaming_response.print_response_stream()
* Usage Pattern
  * Get Started
  * Configuring a Query Engine
  * Defining a Custom Query Engine
Modules
* Module Guides
  * Basic
    * Custom Query Engine
    * Retriever Query Engine
  * Structured & Semi-Structured Data
    * JSON Query Engine
    * Pandas Query Engine
    * Knowledge Graph Query Engine
    * Knowledge Graph RAG Query Engine
  * Advanced
    * Router Query Engine
    * Retriever Router Query Engine
    * Joint QA Summary Query Engine
    * Sub Question Query Engine
    * Multi-Step Query Engine
    * SQL Router Query Engine
    * SQL Auto Vector Query Engine
    * SQL Join Query Engine
    * [Beta] Text-to-SQL with PGVector
    * SQL Query Engine with LlamaIndex + DuckDB
    * Retry Query Engine
    * Retry Source Query Engine
    * Retry Guideline Query Engine
    * CitationQueryEngine
    * Recursive Retriever + Query Engine Demo
    * Joint Tabular/Semantic QA over Tesla 10K
    * Recursive Retriever + Document Agents
    * Ensemble Query Engine Guide
    * Advanced: Towards Multi-Document Querying/Analysis
  * Experimental
    * FLARE Query Engine
Supporting Modules
* Supporting Modules
  * Query Transformations


Query Transformations

LlamaIndex allows you to perform *query transformations* over your
index structures. Query transformations are modules that will convert
a query into another query. They can be **single-step**, as in the
transformation is run once before the query is executed against an
index.
They can also be **multi-step**, as in:
1. The query is transformed, executed against an index,
2. The response is retrieved.
3. Subsequent queries are transformed/executed in a sequential
   fashion.
We list some of our query transformations in more detail below.
Use Cases
Query transformations have multiple use cases:
* Transforming an initial query into a form that can be more easily
  embedded (e.g. HyDE)
* Transforming an initial query into a subquestion that can be more
  easily answered from the data (single-step query decomposition)
* Breaking an initial query into multiple subquestions that can be
  more easily answered on their own. (multi-step query decomposition)
HyDE (Hypothetical Document Embeddings)
HyDE is a technique where given a natural language query, a
hypothetical document/answer is generated first. This hypothetical
document is then used for embedding lookup rather than the raw query.
To use HyDE, an example code snippet is shown below.
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   from llama_index.indices.query.query_transform.base import HyDEQueryTransform
   from llama_index.query_engine.transform_query_engine import TransformQueryEngine
   # load documents, build index
   documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()
   index = VectorStoreIndex(documents)
   # run query with HyDE query transform
   query_str = "what did paul graham do after going to RISD"
   hyde = HyDEQueryTransform(include_original=True)
   query_engine = index.as_query_engine()
   query_engine = TransformQueryEngine(query_engine, query_transform=hyde)
   response = query_engine.query(query_str)
   print(response)
Check out our example notebook for a full walkthrough.
Single-Step Query Decomposition
Some recent approaches (e.g. self-ask, ReAct) have suggested that
LLM's perform better at answering complex questions when they break
the question into smaller steps. We have found that this is true for
queries that require knowledge augmentation as well.
If your query is complex, different parts of your knowledge base may
answer different "subqueries" around the overall query.
Our single-step query decomposition feature transforms a
**complicated** question into a simpler one over the data collection
to help provide a sub-answer to the original question.
This is especially helpful over a *composed graph*. Within a composed
graph, a query can be routed to multiple subindexes, each representing
a subset of the overall knowledge corpus. Query decomposition allows
us to transform the query into a more suitable question over any given
index.
An example image is shown below.
[image: ][image]
Here's a corresponding example code snippet over a composed graph.
   # Setting: a summary index composed over multiple vector indices
   # llm_predictor_chatgpt corresponds to the ChatGPT LLM interface
   from llama_index.indices.query.query_transform.base import DecomposeQueryTransform
   decompose_transform = DecomposeQueryTransform(
       llm_predictor_chatgpt, verbose=True
   )
   # initialize indexes and graph
   ...
   # configure retrievers
   vector_query_engine = vector_index.as_query_engine()
   vector_query_engine = TransformQueryEngine(
       vector_query_engine,
       query_transform=decompose_transform
       transform_extra_info={'index_summary': vector_index.index_struct.summary}
   )
   custom_query_engines = {
       vector_index.index_id: vector_query_engine
   }
   # query
   query_str = (
       "Compare and contrast the airports in Seattle, Houston, and Toronto. "

Query Transformations

   )
   query_engine = graph.as_query_engine(custom_query_engines=custom_query_engines)
   response = query_engine.query(query_str)
Check out our example notebook for a full walkthrough.
Multi-Step Query Transformations
Multi-step query transformations are a generalization on top of
existing single-step query transformation approaches.
Given an initial, complex query, the query is transformed and executed
against an index. The response is retrieved from the query. Given the
response (along with prior responses) and the query, followup
questions may be asked against the index as well. This technique
allows a query to be run against a single knowledge source until that
query has satisfied all questions.
An example image is shown below.
[image: ][image]
Here's a corresponding example code snippet.
   from llama_index.indices.query.query_transform.base import StepDecomposeQueryTransform
   # gpt-4
   step_decompose_transform = StepDecomposeQueryTransform(
       llm_predictor, verbose=True
   )
   query_engine = index.as_query_engine()
   query_engine = MultiStepQueryEngine(query_engine, query_transform=step_decompose_transform)
   response = query_engine.query(
       "Who was in the first batch of the accelerator program the author started?",
   )
   print(str(response))
Check out our example notebook for a full walkthrough.
Examples
^^^^^^^^
* HyDE Query Transform
* Multi-Step Query Engine


Usage Pattern

Defining a "selector" is at the core of defining a router.
You can easily use our routers as a query engine or a retriever. In
these cases, the router will be responsible for "selecting" query
engine(s) or retriever(s) to route the user query to.
We also highlight our "ToolRetrieverRouterQueryEngine" for retrieval-
augmented routing - this is the case where the set of choices
themselves may be very big and may need to be indexed. **NOTE**: this
is a beta feature.
We also highlight using our router as a standalone module.
Defining a selector
Some examples are given below with LLM and Pydantic based single/multi
selectors:
   from llama_index.selectors.llm_selectors import LLMSingleSelector, LLMMultiSelector
   from llama_index.selectors.pydantic_selectors import (
       PydanticMultiSelector,
       PydanticSingleSelector,
   )
   # pydantic selectors feed in pydantic objects to a function calling API
   # single selector (pydantic)
   selector = PydanticSingleSelector.from_defaults()
   # multi selector (pydantic)
   selector = PydanticMultiSelector.from_defaults()
   # LLM selectors use text completion endpoints
   # single selector (LLM)
   selector = LLMSingleSelector.from_defaults()
   # multi selector (LLM)
   selector = LLMMultiSelector.from_defaults()
Using as a Query Engine
A "RouterQueryEngine" is composed on top of other query engines as
tools.
   from llama_index.query_engine.router_query_engine import RouterQueryEngine
   from llama_index.selectors.pydantic_selectors import PydanticSingleSelector, Pydantic
   from llama_index.tools.query_engine import QueryEngineTool
   from llama_index import (
       VectorStoreIndex,
       SummaryIndex,
   )
   # define query engines
   ...
   # initialize tools
   list_tool = QueryEngineTool.from_defaults(
       query_engine=list_query_engine,
       description="Useful for summarization questions related to the data source",
   )
   vector_tool = QueryEngineTool.from_defaults(
       query_engine=vector_query_engine,
       description="Useful for retrieving specific context related to the data source",
   )
   # initialize router query engine (single selection, pydantic)
   query_engine = RouterQueryEngine(
       selector=PydanticSingleSelector.from_defaults(),
       query_engine_tools=[
           list_tool,
           vector_tool,
       ],
   )
   query_engine.query("<query>")
Using as a Retriever
Similarly, a "RouterRetriever" is composed on top of other retrievers
as tools. An example is given below:
   from llama_index.query_engine.router_query_engine import RouterQueryEngine
   from llama_index.selectors.pydantic_selectors import PydanticSingleSelector
   from llama_index.tools import RetrieverTool
   # define indices
   ...
   # define retrievers
   vector_retriever = vector_index.as_retriever()
   keyword_retriever = keyword_index.as_retriever()
   # initialize tools
   vector_tool = RetrieverTool.from_defaults(
       retriever=vector_retriever,
       description="Useful for retrieving specific context from Paul Graham essay on What I Worked On.",
   )
   keyword_tool = RetrieverTool.from_defaults(
       retriever=keyword_retriever,
       description="Useful for retrieving specific context from Paul Graham essay on What I Worked On (using entities mentioned in query)",
   )
   # define retriever
   retriever = RouterRetriever(
       selector=PydanticSingleSelector.from_defaults(llm=llm),
       retriever_tools=[
           list_tool,
           vector_tool,
       ],
   )
Using selector as a standalone module
You can use the selectors as standalone modules. Define choices as

Usage Pattern

either a list of "ToolMetadata" or as a list of strings.
   from llama_index.tools import ToolMetadata
   from llama_index.selectors.llm_selectors import LLMSingleSelector
   # choices as a list of tool metadata
   choices = [
       ToolMetadata(description="description for choice 1", name="choice_1"),
       ToolMetadata(description="description for choice 2", name="choice_2"),
   ]
   # choices as a list of strings
   choices = ["choice 1 - description for choice 1", "choice 2: description for choice 2"]
   selector = LLMSingleSelector.from_defaults()
   selector_result = selector.select(choices, query="What's revenue growth for IBM in 2007?")
   print(selector_result.selections)


Modules

* Router Query Engine
* Retriever Router Query Engine
* SQL Router Query Engine
* Router Retriever


Routers

Concept
Routers are modules that take in a user query and a set of "choices"
(defined by metadata), and returns one or more selected choices.
They can be used on their own (as "selector modules"), or used as a
query engine or retriever (e.g. on top of other query
engines/retrievers).
They are simple but powerful modules that use LLMs for decision making
capabilities. They can be used for the following use cases and more:
* Selecting the right data source among a diverse range of data
  sources
* Deciding whether to do summarization (e.g. using summary index query
  engine) or semantic search (e.g. using vector index query engine)
* Deciding whether to "try" out a bunch of choices at once and combine
  the results (using multi-routing capabilities).
The core router modules exist in the following forms:
* LLM selectors put the choices as a text dump into a prompt and use
  LLM text completion endpoint to make decisions
* Pydantic selectors pass choices as Pydantic schemas into a function
  calling endpoint, and return Pydantic objects
Usage Pattern
A simple example of using our router module as part of a query engine
is given below.
   from llama_index.query_engine.router_query_engine import RouterQueryEngine
   from llama_index.selectors.pydantic_selectors import PydanticSingleSelector
   from llama_index.tools.query_engine import QueryEngineTool
   list_tool = QueryEngineTool.from_defaults(
       query_engine=list_query_engine,
       description="Useful for summarization questions related to the data source",
   )
   vector_tool = QueryEngineTool.from_defaults(
       query_engine=vector_query_engine,
       description="Useful for retrieving specific context related to the data source",
   )
   query_engine = RouterQueryEngine(
       selector=PydanticSingleSelector.from_defaults(),
       query_engine_tools=[
           list_tool,
           vector_tool,
       ],
   )
   query_engine.query("<query>")
You can find more details using routers as standalone modules, as part
of a query engine, and as part of a retriever below in the usage
pattern guide.
* Usage Pattern
  * Defining a selector
  * Using as a Query Engine
  * Using as a Retriever
  * Using selector as a standalone module
Modules
Below you can find extensive guides using routers in different
settings.
* Modules
  * Router Query Engine
  * Retriever Router Query Engine
  * SQL Router Query Engine
  * Router Retriever


Prompts

Concept
Prompting is the fundamental input that gives LLMs their expressive
power. LlamaIndex uses prompts to build the index, do insertion,
perform traversal during querying, and to synthesize the final answer.
LlamaIndex uses a set of default prompt templates that work well out
of the box.
In addition, there are some prompts written and used specifically for
chat models like "gpt-3.5-turbo" here.
Users may also provide their own prompt templates to further customize
the behavior of the framework. The best method for customizing is
copying the default prompt from the link above, and using that as the
base for any modifications.
Usage Pattern
Defining a custom prompt
Defining a custom prompt is as simple as creating a format string
   from llama_index.prompts import PromptTemplate
   template = (
       "We have provided context information below. \n"
       "---------------------\n"
       "{context_str}"
       "\n---------------------\n"
       "Given this information, please answer the question: {query_str}\n"
   )
   qa_template = PromptTemplate(template)
   # you can create text prompt (for completion API)
   prompt = qa_template.format(context_str=..., query_str=...)
   # or easily convert to message prompts (for chat API)
   messages = qa_template.format_messages(context_str=..., query_str=...)
   Note: you may see references to legacy prompt subclasses such as
   "QuestionAnswerPrompt", "RefinePrompt". These have been deprecated
   (and now are type aliases of "PromptTemplate"). Now you can
   directly specify "PromptTemplate(template)" to construct custom
   prompts. But you still have to make sure the template string
   contains the expected parameters (e.g. "{context_str}" and
   "{query_str}") when replacing a default question answer prompt.
You can also define a template from chat messages
   from llama_index.prompts import ChatPromptTemplate, ChatMessage, MessageRole
   message_templates = [
       ChatMessage(content="You are an expert system.", role=MessageRole.SYSTEM),
       ChatMessage(
           content="Generate a short story about {topic}",
           role=MessageRole.USER,
       ),
   ]
   chat_template = ChatPromptTemplate(message_templates=message_templates)
   # you can create message prompts (for chat API)
   messages = chat_template.format_messages(topic=...)
   # or easily convert to text prompt (for completion API)
   prompt = chat_template.format(topic=...)
Passing custom prompts into the pipeline
Since LlamaIndex is a multi-step pipeline, it's important to identify
the operation that you want to modify and pass in the custom prompt at
the right place.
At a high-level, prompts are used in 1) index construction, and 2)
query engine execution
The most commonly used prompts will be the "text_qa_template" and the
"refine_template".
* "text_qa_template" - used to get an initial answer to a query using
  retrieved nodes
* "refine_tempalate" - used when the retrieved text does not fit into
  a single LLM call with "response_mode="compact"" (the default), or
  when more than one node is retrieved using "response_mode="refine"".
  The answer from the first query is inserted as an "existing_answer",
  and the LLM must update or repeat the existing answer based on the
  new context.
Modify prompts used in index construction
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Different indices use different types of prompts during construction
(some don't use prompts at all). For instance, "TreeIndex" uses a
summary prompt to hierarchically summarize the nodes, and
"KeywordTableIndex" uses a keyword extract prompt to extract keywords.
There are two equivalent ways to override the prompts:
1. via the default nodes constructor

Prompts

   index = TreeIndex(nodes, summary_template=<custom_prompt>)
2. via the documents constructor.
   index = TreeIndex.from_documents(docs, summary_template=<custom_prompt>)
For more details on which index uses which prompts, please visit Index
class references.
Modify prompts used in query engine
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
More commonly, prompts are used at query-time (i.e. for executing a
query against an index and synthesizing the final response).
There are also two equivalent ways to override the prompts:
1. via the high-level API
   query_engine = index.as_query_engine(
       text_qa_template=<custom_qa_prompt>,
       refine_template=<custom_refine_prompt>
   )
2. via the low-level composition API
   retriever = index.as_retriever()
   synth = get_response_synthesizer(
       text_qa_template=<custom_qa_prompt>,
       refine_template=<custom_refine_prompt>
   )
   query_engine = RetrieverQueryEngine(retriever, response_synthesizer)
The two approaches above are equivalent, where 1 is essentially
syntactic sugar for 2 and hides away the underlying complexity. You
might want to use 1 to quickly modify some common parameters, and use
2 to have more granular control.
For more details on which classes use which prompts, please visit
Query class references.
Check out the reference documentation for a full set of all prompts.
Modules
* Completion Prompts Customization
* Chat Prompts Customization


Usage Pattern

Getting Started
The most common usage for an embedding model will be setting it in the
service context object, and then using it to construct an index and
query. The input documents will be broken into nodes, and the emedding
model will generate an embedding for each node.
By default, LlamaIndex will use "text-embedding-ada-002", which is
what the example below manually sets up for you.
   from llama_index import ServiceContext, VectorStoreIndex, SimpleDirectoryReader
   from llama_index.embeddings import OpenAIEmbedding
   embed_model = OpenAIEmbedding()
   service_context = ServiceContext.from_defaults(embed_model=embed_model)
   # optionally set a global service context to avoid passing it into other objects every time
   from llama_index import set_global_service_context
   set_global_service_context(service_context)
   documents = SimpleDirectoryReader("./data").load_data()
   index = VectorStoreIndex.from_documents(documents)
Then, at query time, the embedding model will be used again to embed
the query text.
   query_engine = index.as_query_engine()
   response = query_engine.query("query string")
Customization
Batch Size
By default, embeddings requests are sent to OpenAI in batches of 10.
For some users, this may (rarely) incur a rate limit. For other users
embedding many documents, this batch size may be too small.
   # set the batch size to 42
   embed_model = OpenAIEmbedding(embed_batch_size=42)
Local Embedding Models
The easiest way to use a local model is:
   from llama_index import ServiceContext
   service_context = ServiceContext.from_defaults(embed_model="local")
To configure the model used (from Hugging Face hub), add the model
name separated by a colon:
   from llama_index import ServiceContext
   service_context = ServiceContext.from_defaults(
     embed_model="local:BAAI/bge-large-en"
   )
HuggingFace Optimum ONNX Embeddings
LlamaIndex also supports creating and using ONNX embeddings using the
Optimum library from HuggingFace. Simple create and save the ONNX
embeddings, and use them.
Some prerequisites:
   pip install transformers optimum[exporters]
Creation with specifying the model and output path:
   from llama_index.embeddings import OptimumEmbedding
   OptimumEmbedding.create_and_save_optimum_model("BAAI/bge-small-en-v1.5", "./bge_onnx")
And then usage:
   embed_model = OptimumEmbedding(folder_name="./bge_onnx")
   service_context = ServiceContext.from_defaults(
     embed_model=embed_model
   )
LangChain Integrations
We also support any embeddings offered by Langchain here.
The example below loads a model from Hugging Face, using Langchain's
embedding class.
   from langchain.embeddings.huggingface import HuggingFaceBgeEmbeddings
   from llama_index import ServiceContext
   embed_model = HuggingFaceBgeEmbeddings(model_name="BAAI/bge-base-en")
   service_context = ServiceContext.from_defaults(embed_model=embed_model)
Custom Embedding Model
If you wanted to use embeddings not offered by LlamaIndex or
Langchain, you can also extend our base embeddings class and implement
your own!
The example below uses Instructor Embeddings (install/setup details
here), and implements a custom embeddings class. Instructor embeddings
work by providing text, as well as "instructions" on the domain of the
text to embed. This is helpful when embedding text from a very
specific and specialized topic.
   from typing import Any, List
   from InstructorEmbedding import INSTRUCTOR
   from llama_index.embeddings.base import BaseEmbedding
   class InstructorEmbeddings(BaseEmbedding):
     def __init__(
       self,
       instructor_model_name: str = "hkunlp/instructor-large",

Usage Pattern

       instruction: str = "Represent the Computer Science documentation or question:",
       **kwargs: Any,
     ) -> None:
       self._model = INSTRUCTOR(instructor_model_name)
       self._instruction = instruction
       super().__init__(**kwargs)
       def _get_query_embedding(self, query: str) -> List[float]:
         embeddings = self._model.encode([[self._instruction, query]])
         return embeddings[0]
       def _get_text_embedding(self, text: str) -> List[float]:
         embeddings = self._model.encode([[self._instruction, text]])
         return embeddings[0]
       def _get_text_embeddings(self, texts: List[str]) -> List[List[float]]:
         embeddings = self._model.encode([[self._instruction, text] for text in texts])
         return embeddings
Standalone Usage
You can also use embeddings as a standalone module for your project,
existing application, or general testing and exploration.
   embeddings = embed_model.get_text_embedding("It is raining cats and dogs here!")


Modules

We support integrations with OpenAI, Azure, HuggingFace, Instructor,
Optimum, and anything LangChain offers.
* OpenAI Embeddings
* Langchain Embeddings
* Gradient Embeddings
* Azure OpenAI
* Custom Embeddings
* Local Embeddings with HuggingFace
* Elasticsearch Embeddings
* Embeddings with Clarifai
* Text Embedding Inference


Embeddings

Concept
Embeddings are used in LlamaIndex to represent your documents using a
sophisticated numerical representation. Embedding models take text as
input, and return a long list of numbers used to capture the semantics
of the text. These embedding models have been trained to represent
text this way, and help enable many applications, including search!
At a high level, if a user asks a question about dogs, then the
embedding for that question will be highly similar to text that talks
about dogs.
When calculating the similarity between embeddings, there are many
methods to use (dot product, cosine similarity, etc.). By default,
LlamaIndex uses cosine similarity when comparing embeddings.
There are many embedding models to pick from. By default, LlamaIndex
uses "text-embedding-ada-002" from OpenAI. We also support any
embedding model offered by Langchain here, as well as providing an
easy to extend base class for implementing your own embeddings.
Usage Pattern
Most commonly in LlamaIndex, embedding models will be specified in the
"ServiceContext" object, and then used in a vector index. The
embedding model will be used to embed the documents used during index
construction, as well as embedding any queries you make using the
query engine later on.
   from llama_index import ServiceContext
   from llama_index.embeddings import OpenAIEmbedding
   embed_model = OpenAIEmbedding()
   service_context = ServiceContext.from_defaults(embed_model=embed_model)
To save costs, you may want to use a local model.
   from llama_index import ServiceContext
   service_context = ServiceContext.from_defaults(embed_model="local")
This will use a well-performing and fast default from Hugging Face.
You can find more usage details and available customization options
below.
* Usage Pattern
Modules
We support integrations with OpenAI, Azure, and anything LangChain
offers. Details below.
* Modules
  * OpenAI Embeddings
  * Langchain Embeddings
  * Gradient Embeddings
  * Azure OpenAI
  * Custom Embeddings
  * Local Embeddings with HuggingFace
  * Elasticsearch Embeddings
  * Embeddings with Clarifai
  * Text Embedding Inference


Using LLMs as standalone modules

You can use our LLM modules on their own.
Text Completion Example
   from llama_index.llms import OpenAI
   # non-streaming
   resp = OpenAI().complete('Paul Graham is ')
   print(resp)
   # using streaming endpoint
   from llama_index.llms import OpenAI
   llm = OpenAI()
   resp = llm.stream_complete('Paul Graham is ')
   for delta in resp:
       print(delta, end='')
Chat Example
   from llama_index.llms import ChatMessage, OpenAI
   messages = [
       ChatMessage(role="system", content="You are a pirate with a colorful personality"),
       ChatMessage(role="user", content="What is your name"),
   ]
   resp = OpenAI().chat(messages)
   print(resp)
Check out our modules section for usage guides for each LLM.


Customizing LLMs within LlamaIndex Abstractions

You can plugin these LLM abstractions within our other modules in
LlamaIndex (indexes, retrievers, query engines, agents) which allow
you to build advanced workflows over your data.
By default, we use OpenAI's "gpt-3.5-turbo" model. But you may choose
to customize the underlying LLM being used.
Below we show a few examples of LLM customization. This includes
* changing the underlying LLM
* changing the number of output tokens (for OpenAI, Cohere, or AI21)
* having more fine-grained control over all parameters for any LLM,
  from context window to chunk overlap
Example: Changing the underlying LLM
An example snippet of customizing the LLM being used is shown below.
In this example, we use "gpt-4" instead of "gpt-3.5-turbo". Available
models include "gpt-3.5-turbo", "gpt-3.5-turbo-instruct", "gpt-3.5
-turbo-16k", "gpt-4", "gpt-4-32k", "text-davinci-003", and "text-
davinci-002".
Note that you may also plug in any LLM shown on Langchain's LLM page.
   from llama_index import (
       KeywordTableIndex,
       SimpleDirectoryReader,
       LLMPredictor,
       ServiceContext
   )
   from llama_index.llms import OpenAI
   # alternatively
   # from langchain.llms import ...
   documents = SimpleDirectoryReader('data').load_data()
   # define LLM
   llm = OpenAI(temperature=0.1, model="gpt-4")
   service_context = ServiceContext.from_defaults(llm=llm)
   # build index
   index = KeywordTableIndex.from_documents(documents, service_context=service_context)
   # get response from query
   query_engine = index.as_query_engine()
   response = query_engine.query("What did the author do after his time at Y Combinator?")
Example: Changing the number of output tokens (for OpenAI, Cohere, AI21)
The number of output tokens is usually set to some low number by
default (for instance, with OpenAI the default is 256).
For OpenAI, Cohere, AI21, you just need to set the "max_tokens"
parameter (or maxTokens for AI21). We will handle text
chunking/calculations under the hood.
   from llama_index import (
       KeywordTableIndex,
       SimpleDirectoryReader,
       ServiceContext
   )
   from llama_index.llms import OpenAI
   documents = SimpleDirectoryReader('data').load_data()
   # define LLM
   llm = OpenAI(temperature=0, model="text-davinci-002", max_tokens=512)
   service_context = ServiceContext.from_defaults(llm=llm)
Example: Explicitly configure "context_window" and "num_output"
If you are using other LLM classes from langchain, you may need to
explicitly configure the "context_window" and "num_output" via the
"ServiceContext" since the information is not available by default.
   from llama_index import (
       KeywordTableIndex,
       SimpleDirectoryReader,
       ServiceContext
   )
   from llama_index.llms import OpenAI
   # alternatively
   # from langchain.llms import ...
   documents = SimpleDirectoryReader('data').load_data()
   # set context window
   context_window = 4096
   # set number of output tokens
   num_output = 256
   # define LLM
   llm = OpenAI(
       temperature=0,
       model="text-davinci-002",
       max_tokens=num_output,

   )
   service_context = ServiceContext.from_defaults(
       llm=llm,
       context_window=context_window,
       num_output=num_output,
   )
Example: Using a HuggingFace LLM
LlamaIndex supports using LLMs from HuggingFace directly. Note that
for a completely private experience, also setup a local embedding
model as in *this example*.
Many open-source models from HuggingFace require either some preamble
before each prompt, which is a "system_prompt". Additionally, queries
themselves may need an additional wrapper around the "query_str"
itself. All this information is usually available from the HuggingFace
model card for the model you are using.
Below, this example uses both the "system_prompt" and
"query_wrapper_prompt", using specific prompts from the model card
found here.
   from llama_index.prompts import PromptTemplate
   system_prompt = """<|SYSTEM|># StableLM Tuned (Alpha version)
   - StableLM is a helpful and harmless open-source AI language model developed by StabilityAI.
   - StableLM is excited to be able to help the user, but will refuse to do anything that could be considered harmful to the user.
   - StableLM is more than just an information source, StableLM is also able to write poetry, short stories, and make jokes.
   - StableLM will refuse to participate in anything that could harm a human.
   """
   # This will wrap the default prompts that are internal to llama-index
   query_wrapper_prompt = PromptTemplate("<|USER|>{query_str}<|ASSISTANT|>")
   import torch
   from llama_index.llms import HuggingFaceLLM
   llm = HuggingFaceLLM(
       context_window=4096,
       max_new_tokens=256,
       generate_kwargs={"temperature": 0.7, "do_sample": False},
       system_prompt=system_prompt,
       query_wrapper_prompt=query_wrapper_prompt,
       tokenizer_name="StabilityAI/stablelm-tuned-alpha-3b",
       model_name="StabilityAI/stablelm-tuned-alpha-3b",
       device_map="auto",
       stopping_ids=[50278, 50279, 50277, 1, 0],
       tokenizer_kwargs={"max_length": 4096},
       # uncomment this if using CUDA to reduce memory usage
       # model_kwargs={"torch_dtype": torch.float16}
   )
   service_context = ServiceContext.from_defaults(
       chunk_size=1024,
       llm=llm,
   )
Some models will raise errors if all the keys from the tokenizer are
passed to the model. A common tokenizer output that causes issues is
"token_type_ids". Below is an example of configuring the predictor to
remove this before passing the inputs to the model:
   HuggingFaceLLM(
       ...
       tokenizer_outputs_to_remove=["token_type_ids"]
   )
A full API reference can be found here.
Several example notebooks are also listed below:
* StableLM
* Camel
Example: Using a Custom LLM Model - Advanced
To use a custom LLM model, you only need to implement the "LLM" class
(or "CustomLLM" for a simpler interface) You will be responsible for
passing the text to the model and returning the newly generated
tokens.
Note that for a completely private experience, also setup a local
embedding model (example *here*).
Here is a small example using locally running facebook/OPT model and
Huggingface's pipeline abstraction:
   import torch
   from transformers import pipeline
   from typing import Optional, List, Mapping, Any
   from llama_index import (
       ServiceContext,
       SimpleDirectoryReader,
       SummaryIndex
   )
   from llama_index.callbacks import CallbackManager
   from llama_index.llms import (
       CustomLLM,
       CompletionResponse,
       CompletionResponseGen,

       LLMMetadata,
   )
   from llama_index.llms.base import llm_completion_callback
   # set context window size
   context_window = 2048
   # set number of output tokens
   num_output = 256
   # store the pipeline/model outside of the LLM class to avoid memory issues
   model_name = "facebook/opt-iml-max-30b"
   pipeline = pipeline("text-generation", model=model_name, device="cuda:0", model_kwargs={"torch_dtype":torch.bfloat16})
   class OurLLM(CustomLLM):
       @property
       def metadata(self) -> LLMMetadata:
           """Get LLM metadata."""
           return LLMMetadata(
               context_window=context_window,
               num_output=num_output,
               model_name=model_name
           )
       @llm_completion_callback()
       def complete(self, prompt: str, **kwargs: Any) -> CompletionResponse:
           prompt_length = len(prompt)
           response = pipeline(prompt, max_new_tokens=num_output)[0]["generated_text"]
           # only return newly generated tokens
           text = response[prompt_length:]
           return CompletionResponse(text=text)
       @llm_completion_callback()
       def stream_complete(self, prompt: str, **kwargs: Any) -> CompletionResponseGen:
           raise NotImplementedError()
   # define our LLM
   llm = OurLLM()
   service_context = ServiceContext.from_defaults(
       llm=llm,
       embed_model="local:BAAI/bge-base-en-v1.5",
       context_window=context_window,
       num_output=num_output
   )
   # Load the your data
   documents = SimpleDirectoryReader('./data').load_data()
   index = SummaryIndex.from_documents(documents, service_context=service_context)
   # Query and print response
   query_engine = index.as_query_engine()
   response = query_engine.query("<query_text>")
   print(response)
Using this method, you can use any LLM. Maybe you have one running
locally, or running on your own server. As long as the class is
implemented and the generated tokens are returned, it should work out.
Note that we need to use the prompt helper to customize the prompt
sizes, since every model has a slightly different context length.
The decorator is optional, but provides observability via callbacks on
the LLM calls.
Note that you may have to adjust the internal prompts to get good
performance. Even then, you should be using a sufficiently large LLM
to ensure it's capable of handling the complex queries that LlamaIndex
uses internally, so your mileage may vary.
A list of all default internal prompts is available here, and chat-
specific prompts are listed here. You can also implement your own
custom prompts, as described *here*.


Modules

We support integrations with OpenAI, Anthropic, Hugging Face, PaLM,
and more.
OpenAI
* OpenAI
* Azure OpenAI
Anthropic
* Anthropic
Gradient
* Gradient Base Model
* Gradient Model Adapter
Hugging Face
* HuggingFace LLM - Camel-5b
* HuggingFace LLM - StableLM
* Local Llama2 + VectorStoreIndex
EverlyAI
* EverlyAI
LiteLLM
* LiteLLM
PaLM
* PaLM
Predibase
* Predibase
Replicate
* Replicate - Llama 2 13B
* Replicate - Vicuna 13B
* Llama2 + VectorStoreIndex
LangChain
* LangChain LLM
Llama API
* Llama API
Llama CPP
* LlamaCPP
Xorbits Inference
* Xorbits Inference
MonsterAPI
* Set Monster API Key env variable
* Basic Usage Pattern
RunGPT
* RunGPT
* Setup
Portkey
* Portkey
AnyScale
* Anyscale
Ollama
* Ollama - Llama 2 7B
Konko
Clarifai
* Clarifai LLM


LLM

Concept
Picking the proper Large Language Model (LLM) is one of the first
steps you need to consider when building any LLM application over your
data.
LLMs are a core component of LlamaIndex. They can be used as
standalone modules or plugged into other core LlamaIndex modules
(indices, retrievers, query engines). They are always used during the
response synthesis step (e.g. after retrieval). Depending on the type
of index being used, LLMs may also be used during index construction,
insertion, and query traversal.
LlamaIndex provides a unified interface for defining LLM modules,
whether it's from OpenAI, Hugging Face, or LangChain, so that you
don't have to write the boilerplate code of defining the LLM interface
yourself. This interface consists of the following (more details
below):
* Support for **text completion** and **chat** endpoints (details
  below)
* Support for **streaming** and **non-streaming** endpoints
* Support for **synchronous** and **asynchronous** endpoints
Usage Pattern
The following code snippet shows how you can get started using LLMs.
   from llama_index.llms import OpenAI
   # non-streaming
   resp = OpenAI().complete('Paul Graham is ')
   print(resp)
* Using LLMs as standalone modules
* Customizing LLMs within LlamaIndex Abstractions
LLM Compatibility Tracking
While LLMs are powerful, not every LLM is easy to set up. Furthermore,
even with proper setup, some LLMs have trouble performning tasks that
require strict instruction following.
LlamaIndex offers integrations with nearly every LLM, but it can be
often unclear if the LLM will work well out of the box, or if further
customization is needed.
The tables below attempt to validate the **initial** experience with
various LlamaIndex features for various LLMs. These notebooks serve as
a best attempt to gauge performance, as well as how much effort and
tweaking is needed to get things to function properly.
Generally, paid APIs such as OpenAI or Anthropic are viewed as more
reliable. However, local open-source models have been gaining
popularity due to their customizability and approach to transparency.
**Contributing:** Anyone is welcome to contribute new LLMs to the
documentation. Simply copy an existing notebook, setup and test your
LLM, and open a PR with your resutls.
If you have ways to improve the setup for existing notebooks,
contributions to change this are welcome!
**Legend**
* ✅ = should work fine
* ⚠ = sometimes unreliable, may need prompt engineering to improve
* 🛑 = usually unreliable, would need prompt engineering/fine-tuning
  to improve
Paid LLM APIs
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| Model Name   | Basic Query  | Router Query | Sub Question | Text2SQL     | Pydantic     | Data Agents  | Notes        |
|              | Engines      | Engine       | Query Engine |              | Programs     |              |              |
|==============|==============|==============|==============|==============|==============|==============|==============|
| gpt-3.5-tur  | ✅           | ✅           | ✅           | ✅           | ✅           | ✅           |              |
| bo (openai)  |              |              |              |              |              |              |              |
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| gpt-3.5      | ✅           | ✅           | ✅           | ✅           | ✅           | ⚠           | Tool usage   |

LLM

| -turbo-      |              |              |              |              |              |              | in data-     |
| instruct     |              |              |              |              |              |              | agents seems |
| (openai)     |              |              |              |              |              |              | flakey.      |
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| gpt-4        | ✅           | ✅           | ✅           | ✅           | ✅           | ✅           |              |
| (openai)     |              |              |              |              |              |              |              |
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| claude-2     | ✅           | ✅           | ✅           | ✅           | ✅           | ⚠           | Prone to ha  |
| (anthropic)  |              |              |              |              |              |              | llucinating  |
|              |              |              |              |              |              |              | tool inputs. |
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| claude-      | ✅           | ✅           | ✅           | ✅           | ✅           | ⚠           | Prone to ha  |
| instant-1.2  |              |              |              |              |              |              | llucinating  |
| (anthropic)  |              |              |              |              |              |              | tool inputs. |
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
Open Source LLMs
Since open source LLMs require large amounts of resources, the
quantization is reported. Quantization is just a method for reducing
the size of an LLM by shrinking the accuracy of calculations within
the model. Research has shown that up to 4Bit quantization can be
achieved for large LLMs without impacting performance too severely.
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| Model Name   | Basic Query  | Router Query | SubQuestion  | Text2SQL     | Pydantic     | Data Agents  | Notes        |
|              | Engines      | Engine       | Query Engine |              | Programs     |              |              |
|==============|==============|==============|==============|==============|==============|==============|==============|
| llama2-chat- | ✅           | 🛑           | 🛑           | 🛑           | 🛑           | ⚠           | Llama2 seems |
| 7b 4bit (hu  |              |              |              |              |              |              | to be quite  |
| ggingface)   |              |              |              |              |              |              | chatty,      |
|              |              |              |              |              |              |              | which makes  |
|              |              |              |              |              |              |              | parsing      |
|              |              |              |              |              |              |              | structured   |
|              |              |              |              |              |              |              | outputs      |
|              |              |              |              |              |              |              | difficult.   |
|              |              |              |              |              |              |              | Fine-tuning  |
|              |              |              |              |              |              |              | and prompt   |
|              |              |              |              |              |              |              | engineering  |
|              |              |              |              |              |              |              | likely       |
|              |              |              |              |              |              |              | required for |

LLM

|              |              |              |              |              |              |              | better       |
|              |              |              |              |              |              |              | performance  |
|              |              |              |              |              |              |              | on           |
|              |              |              |              |              |              |              | structured   |
|              |              |              |              |              |              |              | outputs.     |
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| Mistral-7B-  | ✅           | 🛑           | 🛑           | ⚠           | ⚠           | ⚠           | Mistral      |
| instruct-v0  |              |              |              |              |              |              | seems        |
| .1 4bit (hu  |              |              |              |              |              |              | slightly     |
| ggingface)   |              |              |              |              |              |              | more         |
|              |              |              |              |              |              |              | reliable for |
|              |              |              |              |              |              |              | structured   |
|              |              |              |              |              |              |              | outputs      |
|              |              |              |              |              |              |              | compared to  |
|              |              |              |              |              |              |              | Llama2.      |
|              |              |              |              |              |              |              | Likely with  |
|              |              |              |              |              |              |              | some prompt  |
|              |              |              |              |              |              |              | engineering, |
|              |              |              |              |              |              |              | it may do    |
|              |              |              |              |              |              |              | better.      |
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
| zephyr-7b-   | ✅           | ✅           | ✅           | ✅           | ✅           | ⚠           | Overall,     |
| alpha (hugg  |              |              |              |              |              |              | "zyphyr-7b"  |
| ingface)     |              |              |              |              |              |              | is appears   |
|              |              |              |              |              |              |              | to be more   |
|              |              |              |              |              |              |              | reliable     |
|              |              |              |              |              |              |              | than other   |
|              |              |              |              |              |              |              | open- source |
|              |              |              |              |              |              |              | models of    |
|              |              |              |              |              |              |              | this size.   |
|              |              |              |              |              |              |              | Although it  |
|              |              |              |              |              |              |              | still        |
|              |              |              |              |              |              |              | hallucinates |
|              |              |              |              |              |              |              | a bit,       |
|              |              |              |              |              |              |              | especially   |
|              |              |              |              |              |              |              | as an agent. |
+--------------+--------------+--------------+--------------+--------------+--------------+--------------+--------------+
Modules
We support integrations with OpenAI, Hugging Face, PaLM, and more.
* Modules
  * OpenAI
  * Anthropic
  * Gradient

LLM

  * Hugging Face
  * EverlyAI
  * LiteLLM
  * PaLM
  * Predibase
  * Replicate
  * LangChain
  * Llama API
  * Llama CPP
  * Xorbits Inference
  * MonsterAPI
  * RunGPT
  * Portkey
  * AnyScale
  * Ollama
  * Konko
  * Clarifai


Automated Metadata Extraction for Nodes

You can use LLMs to automate metadata extraction with our
"MetadataExtractor" modules.
Our metadata extractor modules include the following "feature
extractors":
* "SummaryExtractor" - automatically extracts a summary over a set of
  Nodes
* "QuestionsAnsweredExtractor" - extracts a set of questions that each
  Node can answer
* "TitleExtractor" - extracts a title over the context of each Node
* "EntityExtractor" - extracts entities (i.e. names of places, people,
  things) mentioned in the content of each Node
You can use these feature extractors within our overall
"MetadataExtractor" class. Then you can plug in the
"MetadataExtractor" into our node parser:
   from llama_index.node_parser.extractors import (
       MetadataExtractor,
       TitleExtractor,
       QuestionsAnsweredExtractor
   )
   from llama_index.text_splitter import TokenTextSplitter
   text_splitter = TokenTextSplitter(separator=" ", chunk_size=512, chunk_overlap=128)
   metadata_extractor = MetadataExtractor(
       extractors=[
           TitleExtractor(nodes=5),
           QuestionsAnsweredExtractor(questions=3),
       ],
   )
   node_parser = SimpleNodeParser.from_defaults(
       text_splitter=text_splitter,
       metadata_extractor=metadata_extractor,
   )
   # assume documents are defined -> extract nodes
   nodes = node_parser.get_nodes_from_documents(documents)
Metadata Extraction Guides
^^^^^^^^^^^^^^^^^^^^^^^^^^
* Extracting Metadata for Better Document Indexing and Understanding
* Automated Metadata Extraction for Better Retrieval + Synthesis
* Entity Metadata Extraction
* Metadata Extraction and Augmentation w/ Marvin
* Pydantic Extractor


Defining and Customizing Nodes

Nodes represent "chunks" of source Documents, whether that is a text
chunk, an image, or more. They also contain metadata and relationship
information with other nodes and index structures.
Nodes are a first-class citizen in LlamaIndex. You can choose to
define Nodes and all its attributes directly. You may also choose to
"parse" source Documents into Nodes through our "NodeParser" classes.
For instance, you can do
   from llama_index.node_parser import SimpleNodeParser
   parser = SimpleNodeParser.from_defaults()
   nodes = parser.get_nodes_from_documents(documents)
You can also choose to construct Node objects manually and skip the
first section. For instance,
   from llama_index.schema import TextNode, NodeRelationship, RelatedNodeInfo
   node1 = TextNode(text="<text_chunk>", id_="<node_id>")
   node2 = TextNode(text="<text_chunk>", id_="<node_id>")
   # set relationships
   node1.relationships[NodeRelationship.NEXT] = RelatedNodeInfo(node_id=node2.node_id)
   node2.relationships[NodeRelationship.PREVIOUS] = RelatedNodeInfo(node_id=node1.node_id)
   nodes = [node1, node2]
The "RelatedNodeInfo" class can also store additional "metadata" if
needed:
   node2.relationships[NodeRelationship.PARENT] = RelatedNodeInfo(node_id=node1.node_id, metadata={"key": "val"})
Customizing the ID
Each node has an "node_id" property that is automatically generated if
not manually specified. This ID can be used for a variety of purposes;
this includes being able to update nodes in storage, being able to
define relationships between nodes (through "IndexNode"), and more.
You can also get and set the "node_id" of any "TextNode" directly.
   print(node.node_id)
   node.node_id = "My new node_id!"


Defining and Customizing Documents

Defining Documents
Documents can either be created automatically via data loaders, or
constructed manually.
By default, all of our data loaders (including those offered on
LlamaHub) return "Document" objects through the "load_data" function.
   from llama_index import SimpleDirectoryReader
   documents = SimpleDirectoryReader('./data').load_data()
You can also choose to construct documents manually. LlamaIndex
exposes the "Document" struct.
   from llama_index import Document
   text_list = [text1, text2, ...]
   documents = [Document(text=t) for t in text_list]
To speed up prototyping and development, you can also quickly create a
document using some default text:
   document = Document.example()
Customizing Documents
This section covers various ways to customize "Document" objects.
Since the "Document" object is a subclass of our "TextNode" object,
all these settings and details apply to the "TextNode" object class as
well.
Metadata
Documents also offer the chance to include useful metadata. Using the
"metadata" dictionary on each document, additional information can be
included to help inform responses and track down sources for query
responses. This information can be anything, such as filenames or
categories. If you are integrating with a vector database, keep in
mind that some vector databases require that the keys must be strings,
and the values must be flat (either "str", "float", or "int").
Any information set in the "metadata" dictionary of each document will
show up in the "metadata" of each source node created from the
document. Additionally, this information is included in the nodes,
enabling the index to utilize it on queries and responses. By default,
the metadata is injected into the text for both embedding and LLM
model calls.
There are a few ways to set up this dictionary:
1. In the document constructor:
   document = Document(
       text='text',
       metadata={
           'filename': '<doc_file_name>',
           'category': '<category>'
       }
   )
2. After the document is created:
   document.metadata = {'filename': '<doc_file_name>'}
3. Set the filename automatically using the "SimpleDirectoryReader"
   and "file_metadata" hook. This will automatically run the hook on
   each document to set the "metadata" field:
   from llama_index import SimpleDirectoryReader
   filename_fn = lambda filename: {'file_name': filename}
   # automatically sets the metadata of each document according to filename_fn
   documents = SimpleDirectoryReader('./data', file_metadata=filename_fn).load_data()
Customizing the id
As detailed in the section Document Management, the "doc_id" is used
to enable efficient refreshing of documents in the index. When using
the "SimpleDirectoryReader", you can automatically set the doc
"doc_id" to be the full path to each document:
   from llama_index import SimpleDirectoryReader
   documents = SimpleDirectoryReader("./data", filename_as_id=True).load_data()
   print([x.doc_id for x in documents])
You can also set the "doc_id" of any "Document" directly!
   document.doc_id = "My new document id!"
Note: the ID can also be set through the "node_id" or "id_" property
on a Document object, similar to a "TextNode" object.
Advanced - Metadata Customization
A key detail mentioned above is that by default, any metadata you set
is included in the embeddings generation and LLM.
Customizing LLM Metadata Text
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Typically, a document might have many metadata keys, but you might not
want all of them visible to the LLM during response synthesis. In the
above examples, we may not want the LLM to read the "file_name" of our
document. However, the "file_name" might include information that will

Defining and Customizing Documents

help generate better embeddings. A key advantage of doing this is to
bias the embeddings for retrieval without changing what the LLM ends
up reading.
We can exclude it like so:
   document.excluded_llm_metadata_keys = ['file_name']
Then, we can test what the LLM will actually end up reading using the
"get_content()" function and specifying "MetadataMode.LLM":
   from llama_index.schema import MetadataMode
   print(document.get_content(metadata_mode=MetadataMode.LLM))
Customizing Embedding Metadata Text
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Similar to customing the metadata visible to the LLM, we can also
customize the metadata visible to embeddings. In this case, you can
specifically exclude metadata visible to the embedding model, in case
you DON'T want particular text to bias the embeddings.
   document.excluded_embed_metadata_keys = ['file_name']
Then, we can test what the embedding model will actually end up
reading using the "get_content()" function and specifying
"MetadataMode.EMBED":
   from llama_index.schema import MetadataMode
   print(document.get_content(metadata_mode=MetadataMode.EMBED))
Customizing Metadata Format
~~~~~~~~~~~~~~~~~~~~~~~~~~~
As you know by now, metadata is injected into the actual text of each
document/node when sent to the LLM or embedding model. By default, the
format of this metadata is controlled by three attributes:
1. "Document.metadata_seperator" -> default = ""\n""
When concatenating all key/value fields of your metadata, this field
controls the separator bewtween each key/value pair.
2. "Document.metadata_template" -> default = ""{key}: {value}""
This attribute controls how each key/value pair in your metadata is
formatted. The two variables "key" and "value" string keys are
required.
3. "Document.text_template" -> default = "{metadata_str}\n\n{content}"
Once your metadata is converted into a string using
"metadata_seperator" and "metadata_template", this templates controls
what that metadata looks like when joined with the text content of
your document/node. The "metadata" and "content" string keys are
required.
Summary
Knowing all this, let's create a short example using all this power:
   from llama_index import Document
   from llama_index.schema import MetadataMode
   document = Document(
       text="This is a super-customized document",
       metadata={
           "file_name": "super_secret_document.txt",
           "category": "finance",
           "author": "LlamaIndex"
       },
       excluded_llm_metadata_keys=['file_name'],
       metadata_seperator="::",
       metadata_template="{key}=>{value}",
       text_template="Metadata: {metadata_str}\n-----\nContent: {content}",
   )
   print("The LLM sees this: \n", document.get_content(metadata_mode=MetadataMode.LLM))
   print("The Embedding model sees this: \n", document.get_content(metadata_mode=MetadataMode.EMBED))
Advanced - Automatic Metadata Extraction
We have initial examples of using LLMs themselves to perform metadata
extraction.
Take a look here!
* Extracting Metadata for Better Document Indexing and Understanding


Documents / Nodes

Concept
Document and Node objects are core abstractions within LlamaIndex.
A **Document** is a generic container around any data source - for
instance, a PDF, an API output, or retrieved data from a database.
They can be constructed manually, or created automatically via our
data loaders. By default, a Document stores text along with some other
attributes. Some of these are listed below.
* "metadata" - a dictionary of annotations that can be appended to the
  text.
* "relationships" - a dictionary containing relationships to other
  Documents/Nodes.
*Note*: We have beta support for allowing Documents to store images,
and are actively working on improving its multimodal capabilities.
A **Node** represents a "chunk" of a source Document, whether that is
a text chunk, an image, or other. Similar to Documents, they contain
metadata and relationship information with other nodes.
Nodes are a first-class citizen in LlamaIndex. You can choose to
define Nodes and all its attributes directly. You may also choose to
"parse" source Documents into Nodes through our "NodeParser" classes.
By default every Node derived from a Document will inherit the same
metadata from that Document (e.g. a "file_name" filed in the Document
is propagated to every Node).
Usage Pattern
Here are some simple snippets to get started with Documents and Nodes.
Documents
   from llama_index import Document, VectorStoreIndex
   text_list = [text1, text2, ...]
   documents = [Document(text=t) for t in text_list]
   # build index
   index = VectorStoreIndex.from_documents(documents)
Nodes
   from llama_index.node_parser import SimpleNodeParser
   # load documents
   ...
   # parse nodes
   parser = SimpleNodeParser.from_defaults()
   nodes = parser.get_nodes_from_documents(documents)
   # build index
   index = VectorStoreIndex(nodes)
Document/Node Usage
Take a look at our in-depth guides for more details on how to use
Documents/Nodes.
* Defining and Customizing Documents
* Defining and Customizing Nodes
* Automated Metadata Extraction for Nodes


Customizing Storage

By default, LlamaIndex hides away the complexities and let you query
your data in under 5 lines of code:
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   documents = SimpleDirectoryReader('data').load_data()
   index = VectorStoreIndex.from_documents(documents)
   query_engine = index.as_query_engine()
   response = query_engine.query("Summarize the documents.")
Under the hood, LlamaIndex also supports a swappable **storage layer**
that allows you to customize where ingested documents (i.e., "Node"
objects), embedding vectors, and index metadata are stored.
[image: ][image]
Low-Level API
To do this, instead of the high-level API,
   index = VectorStoreIndex.from_documents(documents)
we use a lower-level API that gives more granular control:
   from llama_index.storage.docstore import SimpleDocumentStore
   from llama_index.storage.index_store import SimpleIndexStore
   from llama_index.vector_stores import SimpleVectorStore
   from llama_index.node_parser import SimpleNodeParser
   # create parser and parse document into nodes
   parser = SimpleNodeParser.from_defaults()
   nodes = parser.get_nodes_from_documents(documents)
   # create storage context using default stores
   storage_context = StorageContext.from_defaults(
       docstore=SimpleDocumentStore(),
       vector_store=SimpleVectorStore(),
       index_store=SimpleIndexStore(),
   )
   # create (or load) docstore and add nodes
   storage_context.docstore.add_documents(nodes)
   # build index
   index = VectorStoreIndex(nodes, storage_context=storage_context)
   # save index
   index.storage_context.persist(persist_dir="<persist_dir>")
   # can also set index_id to save multiple indexes to the same folder
   index.set_index_id("<index_id>")
   index.storage_context.persist(persist_dir="<persist_dir>")
   # to load index later, make sure you setup the storage context
   # this will loaded the persisted stores from persist_dir
   storage_context = StorageContext.from_defaults(
       persist_dir="<persist_dir>"
   )
   # then load the index object
   from llama_index import load_index_from_storage
   loaded_index = load_index_from_storage(storage_context)
   # if loading an index from a persist_dir containing multiple indexes
   loaded_index = load_index_from_storage(storage_context, index_id="<index_id>")
   # if loading multiple indexes from a persist dir
   loaded_indicies = load_index_from_storage(storage_context, index_ids=["<index_id>", ...])
You can customize the underlying storage with a one-line change to
instantiate different document stores, index stores, and vector
stores. See Document Stores, Vector Stores, Index Stores guides for
more details.
For saving and loading a graph/composable index, see the full guide
here.
Vector Store Integrations and Storage
Most of our vector store integrations store the entire index (vectors
+ text) in the vector store itself. This comes with the major benefit
of not having to explicitly persist the index as shown above, since
the vector store is already hosted and persisting the data in our
index.
The vector stores that support this practice are:
* CognitiveSearchVectorStore
* ChatGPTRetrievalPluginClient
* CassandraVectorStore
* ChromaVectorStore
* EpsillaVectorStore
* DocArrayHnswVectorStore
* DocArrayInMemoryVectorStore
* LanceDBVectorStore
* MetalVectorStore
* MilvusVectorStore
* MyScaleVectorStore
* OpensearchVectorStore
* PineconeVectorStore
* QdrantVectorStore
* RedisVectorStore
* WeaviateVectorStore
A small example using Pinecone is below:
   import pinecone
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   from llama_index.vector_stores import PineconeVectorStore

Customizing Storage

   # Creating a Pinecone index
   api_key = "api_key"
   pinecone.init(api_key=api_key, environment="us-west1-gcp")
   pinecone.create_index(
       "quickstart",
       dimension=1536,
       metric="euclidean",
       pod_type="p1"
   )
   index = pinecone.Index("quickstart")
   # construct vector store
   vector_store = PineconeVectorStore(pinecone_index=index)
   # create storage context
   storage_context = StorageContext.from_defaults(vector_store=vector_store)
   # load documents
   documents = SimpleDirectoryReader("./data").load_data()
   # create index, which will insert documents/vectors to pinecone
   index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)
If you have an existing vector store with data already loaded in, you
can connect to it and directly create a "VectorStoreIndex" as follows:
   index = pinecone.Index("quickstart")
   vector_store = PineconeVectorStore(pinecone_index=index)
   loaded_index = VectorStoreIndex.from_vector_store(vector_store=vector_store)


Index Stores

Index stores contains lightweight index metadata (i.e. additional
state information created when building an index).
See the API Reference for more details.
Simple Index Store
By default, LlamaIndex uses a simple index store backed by an in-
memory key-value store. They can be persisted to (and loaded from)
disk by calling "index_store.persist()" (and
"SimpleIndexStore.from_persist_path(...)" respectively).
MongoDB Index Store
Similarly to document stores, we can also use "MongoDB" as the storage
backend of the index store.
   from llama_index.storage.index_store import MongoIndexStore
   from llama_index import VectorStoreIndex
   # create (or load) index store
   index_store = MongoIndexStore.from_uri(uri="<mongodb+srv://...>")
   # create storage context
   storage_context = StorageContext.from_defaults(index_store=index_store)
   # build index
   index = VectorStoreIndex(nodes, storage_context=storage_context)
   # or alternatively, load index
   from llama_index import load_index_from_storage
   index = load_index_from_storage(storage_context)
Under the hood, "MongoIndexStore" connects to a fixed MongoDB database
and initializes new collections (or loads existing collections) for
your index metadata.
   Note: You can configure the "db_name" and "namespace" when
   instantiating "MongoIndexStore", otherwise they default to
   "db_name="db_docstore"" and "namespace="docstore"".
Note that it's not necessary to call "storage_context.persist()" (or
"index_store.persist()") when using an "MongoIndexStore" since data is
persisted by default.
You can easily reconnect to your MongoDB collection and reload the
index by re-initializing a "MongoIndexStore" with an existing
"db_name" and "collection_name".
A more complete example can be found *here*
Redis Index Store
We support Redis as an alternative document store backend that
persists data as "Node" objects are ingested.
   from llama_index.storage.index_store import RedisIndexStore
   from llama_index import VectorStoreIndex
   # create (or load) docstore and add nodes
   index_store = RedisIndexStore.from_host_and_port(
     host="127.0.0.1",
     port="6379",
     namespace='llama_index'
   )
   # create storage context
   storage_context = StorageContext.from_defaults(index_store=index_store)
   # build index
   index = VectorStoreIndex(nodes, storage_context=storage_context)
   # or alternatively, load index
   from llama_index import load_index_from_storage
   index = load_index_from_storage(storage_context)
Under the hood, "RedisIndexStore" connects to a redis database and
adds your nodes to a namespace stored under "{namespace}/index".
   Note: You can configure the "namespace" when instantiating
   "RedisIndexStore", otherwise it defaults "namespace="index_store"".
You can easily reconnect to your Redis client and reload the index by
re-initializing a "RedisIndexStore" with an existing "host", "port",
and "namespace".
A more complete example can be found *here*


Vector Stores

Vector stores contain embedding vectors of ingested document chunks
(and sometimes the document chunks as well).
Simple Vector Store
By default, LlamaIndex uses a simple in-memory vector store that's
great for quick experimentation. They can be persisted to (and loaded
from) disk by calling "vector_store.persist()" (and
"SimpleVectorStore.from_persist_path(...)" respectively).
Vector Store Options & Feature Support
LlamaIndex supports over 20 different vector store options. We are
actively adding more integrations and improving feature coverage for
each.
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Vector Store   | Type           | Metadata       | Hybrid Search  | Delete         | Store          | Async          |
|                |                | Filtering      |                |                | Documents      |                |
|================|================|================|================|================|================|================|
| Elasticsearch  | self-hosted /  | ✓              | ✓              | ✓              | ✓              | ✓              |
|                | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Pinecone       | cloud          | ✓              | ✓              | ✓              | ✓              |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Timescale      |                | ✓              |                | ✓              | ✓              | ✓              |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Weaviate       | self-hosted /  | ✓              | ✓              | ✓              | ✓              |                |
|                | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Postgres       | self-hosted /  | ✓              | ✓              | ✓              | ✓              | ✓              |
|                | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Cassandra      | self-hosted /  | ✓              |                | ✓              | ✓              |                |
|                | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Qdrant         | self-hosted /  | ✓              |                | ✓              | ✓              | ✓              |
|                | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Chroma         | self-hosted    | ✓              |                | ✓              | ✓              |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Milvus /       | self-hosted /  | ✓              |                | ✓              | ✓              |                |
| Zilliz         | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Typesense      | self-hosted /  | ✓              |                | ✓              | ✓              |                |
|                | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Supabase       | self-hosted /  | ✓              |                | ✓              | ✓              |                |
|                | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| MongoDB Atlas  | self-hosted /  | ✓              |                | ✓              | ✓              |                |
|                | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+

Vector Stores

| Redis          | self-hosted /  | ✓              |                | ✓              | ✓              |                |
|                | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Deeplake       | self-hosted /  | ✓              |                | ✓              | ✓              |                |
|                | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| OpenSearch     | self-hosted /  | ✓              |                | ✓              | ✓              |                |
|                | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Neo4jVector    | self-hosted /  |                |                | ✓              | ✓              |                |
|                | cloud          |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Azure          | cloud          |                | ✓              | ✓              | ✓              |                |
| Cognitive      |                |                |                |                |                |                |
| Search         |                |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| DynamoDB       | cloud          |                |                | ✓              |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| LanceDB        | cloud          | ✓              |                | ✓              | ✓              |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Metal          | cloud          | ✓              |                | ✓              | ✓              |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| MyScale        | cloud          | ✓              | ✓              |                | ✓              |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Tair           | cloud          | ✓              |                | ✓              | ✓              |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| Simple         | in-memory      | ✓              |                | ✓              |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| FAISS          | in-memory      |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| ChatGPT        | aggregator     |                |                | ✓              | ✓              |                |
| Retrieval      |                |                |                |                |                |                |
| Plugin         |                |                |                |                |                |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
| DocArray       | aggregator     | ✓              |                | ✓              | ✓              |                |
+----------------+----------------+----------------+----------------+----------------+----------------+----------------+
For more details, see Vector Store Integrations.
Examples
^^^^^^^^
* Elasticsearch Vector Store
* Simple Vector Store
* Rockset Vector Store
* Qdrant Vector Store
* Faiss Vector Store
* DeepLake Vector Store
* MyScale Vector Store
* Metal Vector Store
* Weaviate Vector Store
* Opensearch Vector Store
* Pinecone Vector Store
* Chroma
* LanceDB Vector Store
* Milvus Vector Store
* Redis Vector Store
* Query the data
* Working with Metadata
* Weaviate Vector Store - Hybrid Search
* Zep Vector Store
* Create a Zep Vector Store and Index
* Querying with Metadata filters
* Pinecone Vector Store - Hybrid Search
* Simple Vector Store - Async Index Creation

Vector Stores

* Tair Vector Store
* Supabase Vector Store
* DocArray Hnsw Vector Store
* DocArray InMemory Vector Store
* MongoDB Atlas
* Cassandra Vector Store
* Neo4j vector store
* Azure Cognitive Search
* Basic Example
* Create Index (if it does not exist)
* Use Existing Index
* Adding a document to existing index
* Filtering
* Epsilla Vector Store
* Timescale Vector Store (PostgreSQL)


Persisting & Loading Data

Persisting Data
By default, LlamaIndex stores data in-memory, and this data can be
explicitly persisted if desired:
   storage_context.persist(persist_dir="<persist_dir>")
This will persist data to disk, under the specified "persist_dir" (or
"./storage" by default).
Multiple indexes can be persisted and loaded from the same directory,
assuming you keep track of index ID's for loading.
User can also configure alternative storage backends (e.g. "MongoDB")
that persist data by default. In this case, calling
"storage_context.persist()" will do nothing.
Loading Data
To load data, user simply needs to re-create the storage context using
the same configuration (e.g. pass in the same "persist_dir" or vector
store client).
   storage_context = StorageContext.from_defaults(
       docstore=SimpleDocumentStore.from_persist_dir(persist_dir="<persist_dir>"),
       vector_store=SimpleVectorStore.from_persist_dir(persist_dir="<persist_dir>"),
       index_store=SimpleIndexStore.from_persist_dir(persist_dir="<persist_dir>"),
   )
We can then load specific indices from the "StorageContext" through
some convenience functions below.
   from llama_index import load_index_from_storage, load_indices_from_storage, load_graph_from_storage
   # load a single index
   # need to specify index_id if multiple indexes are persisted to the same directory
   index = load_index_from_storage(storage_context, index_id="<index_id>")
   # don't need to specify index_id if there's only one index in storage context
   index = load_index_from_storage(storage_context)
   # load multiple indices
   indices = load_indices_from_storage(storage_context) # loads all indices
   indices = load_indices_from_storage(storage_context, index_ids=[index_id1, ...]) # loads specific indices
   # load composable graph
   graph = load_graph_from_storage(storage_context, root_id="<root_id>") # loads graph with the specified root_id
Here's the full API Reference on saving and loading.
Using a remote backend
By default, LlamaIndex uses a local filesystem to load and save files.
However, you can override this by passing a
"fsspec.AbstractFileSystem" object.
Here's a simple example, instantiating a vector store:
   import dotenv
   import s3fs
   import os
   dotenv.load_dotenv("../../../.env")
   # load documents
   documents = SimpleDirectoryReader('../../../examples/paul_graham_essay/data/').load_data()
   print(len(documents))
   index = VectorStoreIndex.from_documents(documents)
At this point, everything has been the same. Now - let's instantiate a
S3 filesystem and save / load from there.
   # set up s3fs
   AWS_KEY = os.environ['AWS_ACCESS_KEY_ID']
   AWS_SECRET = os.environ['AWS_SECRET_ACCESS_KEY']
   R2_ACCOUNT_ID = os.environ['R2_ACCOUNT_ID']
   assert AWS_KEY is not None and AWS_KEY != ""
   s3 = s3fs.S3FileSystem(
      key=AWS_KEY,
      secret=AWS_SECRET,
      endpoint_url=f'https://{R2_ACCOUNT_ID}.r2.cloudflarestorage.com',
      s3_additional_kwargs={'ACL': 'public-read'}
   )
   # save index to remote blob storage
   index.set_index_id("vector_index")
   # this is {bucket_name}/{index_name}
   index.storage_context.persist('llama-index/storage_demo', fs=s3)
   # load index from s3
   sc = StorageContext.from_defaults(persist_dir='llama-index/storage_demo', fs=s3)
   index2 = load_index_from_storage(sc, 'vector_index')
By default, if you do not pass a filesystem, we will assume a local
filesystem.


Key-Value Stores

Key-Value stores are the underlying storage abstractions that power
our Document Stores and Index Stores.
We provide the following key-value stores:
* **Simple Key-Value Store**: An in-memory KV store. The user can
  choose to call "persist" on this kv store to persist data to disk.
* **MongoDB Key-Value Store**: A MongoDB KV store.
See the API Reference for more details.
Note: At the moment, these storage abstractions are not externally
facing.


Document Stores

Document stores contain ingested document chunks, which we call "Node"
objects.
See the API Reference for more details.
Simple Document Store
By default, the "SimpleDocumentStore" stores "Node" objects in-memory.
They can be persisted to (and loaded from) disk by calling
"docstore.persist()" (and "SimpleDocumentStore.from_persist_path(...)"
respectively).
A more complete example can be found *here*
MongoDB Document Store
We support MongoDB as an alternative document store backend that
persists data as "Node" objects are ingested.
   from llama_index.storage.docstore import MongoDocumentStore
   from llama_index.node_parser import SimpleNodeParser
   # create parser and parse document into nodes
   parser = SimpleNodeParser.from_defaults()
   nodes = parser.get_nodes_from_documents(documents)
   # create (or load) docstore and add nodes
   docstore = MongoDocumentStore.from_uri(uri="<mongodb+srv://...>")
   docstore.add_documents(nodes)
   # create storage context
   storage_context = StorageContext.from_defaults(docstore=docstore)
   # build index
   index = VectorStoreIndex(nodes, storage_context=storage_context)
Under the hood, "MongoDocumentStore" connects to a fixed MongoDB
database and initializes new collections (or loads existing
collections) for your nodes.
   Note: You can configure the "db_name" and "namespace" when
   instantiating "MongoDocumentStore", otherwise they default to
   "db_name="db_docstore"" and "namespace="docstore"".
Note that it's not necessary to call "storage_context.persist()" (or
"docstore.persist()") when using an "MongoDocumentStore" since data is
persisted by default.
You can easily reconnect to your MongoDB collection and reload the
index by re-initializing a "MongoDocumentStore" with an existing
"db_name" and "collection_name".
A more complete example can be found *here*
Redis Document Store
We support Redis as an alternative document store backend that
persists data as "Node" objects are ingested.
   from llama_index.storage.docstore import RedisDocumentStore
   from llama_index.node_parser import SimpleNodeParser
   # create parser and parse document into nodes
   parser = SimpleNodeParser.from_defaults()
   nodes = parser.get_nodes_from_documents(documents)
   # create (or load) docstore and add nodes
   docstore = RedisDocumentStore.from_host_and_port(
     host="127.0.0.1",
     port="6379",
     namespace='llama_index'
   )
   docstore.add_documents(nodes)
   # create storage context
   storage_context = StorageContext.from_defaults(docstore=docstore)
   # build index
   index = VectorStoreIndex(nodes, storage_context=storage_context)
Under the hood, "RedisDocumentStore" connects to a redis database and
adds your nodes to a namespace stored under "{namespace}/docs".
   Note: You can configure the "namespace" when instantiating
   "RedisDocumentStore", otherwise it defaults "namespace="docstore"".
You can easily reconnect to your Redis client and reload the index by
re-initializing a "RedisDocumentStore" with an existing "host",
"port", and "namespace".
A more complete example can be found *here*
Firestore Document Store
We support Firestore as an alternative document store backend that
persists data as "Node" objects are ingested.
   from llama_index.storage.docstore import FirestoreDocumentStore
   from llama_index.node_parser import SimpleNodeParser
   # create parser and parse document into nodes
   parser = SimpleNodeParser.from_defaults()
   nodes = parser.get_nodes_from_documents(documents)
   # create (or load) docstore and add nodes
   docstore = FirestoreDocumentStore.from_dataabse(
     project="project-id",

Document Stores

     database="(default)",
   )
   docstore.add_documents(nodes)
   # create storage context
   storage_context = StorageContext.from_defaults(docstore=docstore)
   # build index
   index = VectorStoreIndex(nodes, storage_context=storage_context)
Under the hood, "FirestoreDocumentStore" connects to a firestore
database in Google Cloud and adds your nodes to a namespace stored
under "{namespace}/docs".
   Note: You can configure the "namespace" when instantiating
   "FirestoreDocumentStore", otherwise it defaults
   "namespace="docstore"".
You can easily reconnect to your Firestore database and reload the
index by re-initializing a "FirestoreDocumentStore" with an existing
"project", "database", and "namespace".
A more complete example can be found *here*


Storage

Concept
LlamaIndex provides a high-level interface for ingesting, indexing,
and querying your external data.
Under the hood, LlamaIndex also supports swappable **storage
components** that allows you to customize:
* **Document stores**: where ingested documents (i.e., "Node" objects)
  are stored,
* **Index stores**: where index metadata are stored,
* **Vector stores**: where embedding vectors are stored.
* **Graph stores**: where knowledge graphs are stored (i.e. for
  "KnowledgeGraphIndex").
The Document/Index stores rely on a common Key-Value store
abstraction, which is also detailed below.
LlamaIndex supports persisting data to any storage backend supported
by fsspec. We have confirmed support for the following storage
backends:
* Local filesystem
* AWS S3
* Cloudflare R2
[image: ][image]
Usage Pattern
Many vector stores (except FAISS) will store both the data as well as
the index (embeddings). This means that you will not need to use a
separate document store or index store. This *also* means that you
will not need to explicitly persist this data - this happens
automatically. Usage would look something like the following to build
a new index / reload an existing one.
   ## build a new index
   from llama_index import VectorStoreIndex, StorageContext
   from llama_index.vector_stores import DeepLakeVectorStore
   # construct vector store and customize storage context
   vector_store = DeepLakeVectorStore(dataset_path="<dataset_path>")
   storage_context = StorageContext.from_defaults(
       vector_store = vector_store
   )
   # Load documents and build index
   index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)
   ## reload an existing one
   index = VectorStoreIndex.from_vector_store(vector_store=vector_store)
See our Vector Store Module Guide below for more details.
Note that in general to use storage abstractions, you need to define a
"StorageContext" object:
   from llama_index.storage.docstore import SimpleDocumentStore
   from llama_index.storage.index_store import SimpleIndexStore
   from llama_index.vector_stores import SimpleVectorStore
   from llama_index.storage import StorageContext
   # create storage context using default stores
   storage_context = StorageContext.from_defaults(
       docstore=SimpleDocumentStore(),
       vector_store=SimpleVectorStore(),
       index_store=SimpleIndexStore(),
   )
More details on customization/persistence can be found in the guides
below.
* Customizing Storage
* Persisting & Loading Data
Modules
We offer in-depth guides on the different storage components.
* Vector Stores
* Document Stores
* Index Stores
* Key-Value Stores
* Using Graph Stores