The Development Pathway

In your journey to developing an LLM application, it helps to start
with a discovery phase of understanding your data and doing some
identification of issues and corner cases as you interact with the
system.
Over time, you would try to formalize processes and evaluation
methodology, setting up tools for observability, debugging and
experiment tracking, and eventually production monitoring.
Below, we provide some additional guidance on considerations and
hurdles you may face when developing your application.
The Challenges of Building a Production-Ready LLM Application
Many who are interested in the LLM application space are not machine
learning engineers but rather software developers or even non-
technical folk.
One of the biggest strides forward that LLMs and foundation models
have made to the AI/ML application landscape is that it makes it
really easy to go from idea to prototype without facing all of the
hurdles and uncertainty of a traditional machine learning project.
This would have involved collecting, exploring and cleaning data,
keeping up with latest research and exploring different methods,
training models, adjusting hyperparameters, and dealing with
unexpected issues in model quality.
The huge infrastructure burden, long development cycle, and high risk
to reward ratio have been blockers to successful applications.
At the same time, despite the fact that getting a prototype working
quickly through frameworks like LlamaIndex has become a lot more
accessible, deploying a machine learning product in the real world is
still rife with uncertainty and challenges.
Quality and User Interaction
On the tamer side, one may face quality issues, and in the worse case,
one may be liable to losing user trust if the application proves
itself to be unreliable.
We've already seen a bit of this with ChatGPT - despite its life-
likeness and seeming ability to understand our conversations and
requests, it often makes things up ("hallucinates"). It's not
connected to the real world, data, or other digital applications.
It is important to be able to monitor, track and improve against
quality issues.
Tradeoffs in LLM Application Development
There are a few tradeoffs in LLM application development:
1. **Cost** - more powerful models may be more expensive
2. **Latency** - more powerful models may be slower
3. **Simplicity** (one size fits all) - how powerful and flexible is
   the model / pipeline?
4. **Reliability / Usability** - is my application working at least in
   the general case? Is it ready for unstructured user interaction?
   Have I covered the major usage patterns?
LLM infra improvements are progressing quickly and we expect cost and
latency to go down over time.
Here are some additional concerns:
1. **Evaluation** - Once I start diving deeper into improving quality,
   how can I evaluate individual components? How can I keep track of
   issues and track whether / how they are being improved over time as
   I change my application?
2. **Data-Driven** - How can I automate more of my evaluation and
   iteration process? How do I start small and add useful data points
   over time? How can I organize different datasets and metrics which
   serving different purposes? How can I manage the complexity while
   keeping track of my guiding light of providing the best user
   experience?
3. **Customization / Complexity Tradeoff** - How do I improve each
   stage of the pipeline - preprocessing and feature extraction,
   retrieval, generation? Does this involve adding additional
   structure or processing? How can I break down this goal into more
   measurable and trackable sub-goals?
Differences between **Evaluation** and being **Data-Driven**:
1. **Evaluation** does not necessarily have to be rigorous or fully
   data-driven process - especially at the initial stages. It is more
   concerned with the initial *development* phase of the application -

The Development Pathway

   validating that the overall pipeline works in the general case and
   starting to define possible signals and metrics which may be
   carried forward into production.
2. Being **Data-Driven** is closely tied to *automation*. After we've
   chosen our basic application structure, how can we improve the
   system over time? How can we ensure quality in a systematic way?
   How can we reduce the cost of monitoring, and what are the pathways
   to adding and curating data points? How can we leverage ML systems
   (including but not limited to LLMs) to make this process easier?
Additional considerations:
1. **Privacy** - how can I ensure that my data is not leaked if I am
   feeding it into these models? What infrastructure am I using and
   what is the security guarantee / how is the access control
   structured?
Development Hurdles
Here are some potential problems you may encounter when developing
your LLM application which may lead to unsatisfactory results.
Retrieval
1. **Out of Domain:** If your data is extremely specific (medical,
   legal, scientific, financial, or other documents with technical
   lingo), it may be worth: - trying out alternate embeddings - Check
   the MTEB Leaderboard - You may configure a local embedding model
   *with the steps here* - testing out fine-tuning of embeddings -
   Tools: setfit - Anecdotally, we have seen retrieval accuracy
   improve by ~12% by curating a small annotated dataset from
   production data - Even synthetic data generation without human
   labels has been shown to improve retrieval metrics across similar
   documents in train / val sets. - More detailed guides and case
   studies will come soon. - testing out sparse retrieval methods (see
   ColBERT, SPLADE) - these methods have been shown to generalize well
   to out of domain data - that are starting to be available in some
   enterprise systems (e.g. Elastic Search's ELSeR) - checking out our
   evaluation principles guide on how you might evaluate the above
   changes


End-to-End Evaluation

End-to-End evaluation should be the guiding signal for your RAG
application - will my pipeline generate the right responses given the
data sources and a set of queries?
While it helps initially to individually inspect queries and
responses, as you deal with more failure and corner cases, it may stop
being feasible to look at each query individually, and rather it may
help instead to define a set of summary metrics or automated
evaluation, and gain an intuition for what they might be telling you
and where you might dive deeper.
Setting up an Evaluation Set
It is helpful to start off with a small but diverse set of queries,
and build up more examples as one discovers problematic queries or
interactions.
We've created some tools that automatically generate a dataset for you
given a set of documents to query. (See example below).
* QuestionGeneration
In the future, we will also be able to create datasets automatically
against tools.
The Spectrum of Evaluation Options
Quantitative eval is more useful when evaluating applications where
there is a correct answer - for instance, validating that the choice
of tools and their inputs are correct given the plan, or retrieving
specific pieces of information, or attempting to produce intermediate
output of a certain schema (e.g. JSON fields).
Qualitative eval is more useful when generating long-form responses
that are meant to be *helpful* but not necessarily completely
accurate.
There is a spectrum of evaluation options ranging from metrics,
cheaper models, more expensive models (GPT4), and human evaluation.
Below is some example usage of the evaluation modules:
Discovery - Sensitivity Testing
With a complex pipeline, it may be unclear which parts of the pipeline
are affecting your results.
Sensitivity testing can be a good inroad into choosing which
components to individually test or tweak more thoroughly, or which
parts of your dataset (e.g. queries) may be producing problematic
results.
More details on how to discover issues automatically with methods such
as sensitivity testing will come soon.
Examples of this in the more traditional ML domain include Giskard.
Metrics Ensembling
It may be expensive to use GPT-4 to carry out evaluation especially as
your dev set grows large.
Metrics ensembling uses an ensemble of weaker signals (exact match,
F1, ROUGE, BLEU, BERT-NLI and BERT-similarity) to predict the output
of a more expensive evaluation methods that are closer to the gold
labels (human-labelled/GPT-4).
It is intenteded for two purposes:
1. Evaluating changes cheaply and quickly across a large dataset
   during the development stage.
2. Flagging outliers for further evaluation (GPT-4 / human alerting)
   during the production monitoring stage.
We also want the metrics ensembling to be interpretable - the
correlation and weighting scores should give an indication of which
metrics best capture the evaluation criteria.
We will discuss more about the methodology in future updates.


LlamaHub Tools Guide

We offer a rich set of Tool Specs that are offered through LlamaHub 🦙
. [image: ][image]
These tool specs represent an initial curated list of services that an
agent can interact with and enrich its capability to perform different
actions.
We also provide a list of **utility tools** that help to abstract away
pain points when designing agents to interact with different API
services that return large amounts of data.
Tool Specs
Coming soon!
Utility Tools
Oftentimes, directly querying an API can return a massive volume of
data, which on its own may overflow the context window of the LLM (or
at the very least unnecessarily increase the number of tokens that you
are using).
To tackle this, we’ve provided an initial set of “utility tools” in
LlamaHub Tools - utility tools are not conceptually tied to a given
service (e.g. Gmail, Notion), but rather can augment the capabilities
of existing Tools. In this particular case, utility tools help to
abstract away common patterns of needing to cache/index and query data
that’s returned from any API request.
Let’s walk through our two main utility tools below.
OnDemandLoaderTool
This tool turns any existing LlamaIndex data loader ( "BaseReader"
class) into a tool that an agent can use. The tool can be called with
all the parameters needed to trigger "load_data" from the data loader,
along with a natural language query string. During execution, we first
load data from the data loader, index it (for instance with a vector
store), and then query it “on-demand”. All three of these steps happen
in a single tool call.
Oftentimes this can be preferable to figuring out how to load and
index API data yourself. While this may allow for data reusability,
oftentimes users just need an ad-hoc index to abstract away prompt
window limitations for any API call.
A usage example is given below:
   from llama_hub.wikipedia.base import WikipediaReader
   from llama_index.tools.ondemand_loader_tool import OnDemandLoaderTool
   tool = OnDemandLoaderTool.from_defaults(
   	reader,
   	name="Wikipedia Tool",
   	description="A tool for loading data and querying articles from Wikipedia"
   )
LoadAndSearchToolSpec
The LoadAndSearchToolSpec takes in any existing Tool as input. As a
tool spec, it implements "to_tool_list" , and when that function is
called, two tools are returned: a "load" tool and then a "search"
tool.
The "load" Tool execution would call the underlying Tool, and the
index the output (by default with a vector index). The "search" Tool
execution would take in a query string as input and call the
underlying index.
This is helpful for any API endpoint that will by default return large
volumes of data - for instance our WikipediaToolSpec will by default
return entire Wikipedia pages, which will easily overflow most LLM
context windows.
Example usage is shown below:
   from llama_hub.tools.wikipedia.base import WikipediaToolSpec
   from llama_index.tools.tool_spec.load_and_search import LoadAndSearchToolSpec
   wiki_spec = WikipediaToolSpec()
   # Get the search wikipedia tool
   tool = wiki_spec.to_tool_list()[1]
   # Create the Agent with load/search tools
   agent = OpenAIAgent.from_tools(
    LoadAndSearchToolSpec.from_defaults(
       tool
    ).to_tool_list(), verbose=True
   )


Usage Pattern

You can create custom LlamaHub Tool Specs and Tools or they can be
imported from the "llama-hub" package. They can be plugged into our
native agents, or LangChain agents.
Using with our Agents
To use with our OpenAIAgent,
   from llama_index.agent import OpenAIAgent
   from llama_hub.tools.gmail.base import GmailToolSpec
   from llama_index.tools.function_tool import FunctionTool
   # Use a tool spec from Llama-Hub
   tool_spec = GmailToolSpec()
   # Create a custom tool. Type annotations and docstring are used for the
   # tool definition sent to the Function calling API.
   def add_numbers(x: int, y: int) -> int:
       """
       Adds the two numbers together and returns the result.
       """
       return x + y
   function_tool = FunctionTool.from_defaults(fn=add_numbers)
   tools = tool_spec.to_tool_list() + [function_tool]
   agent = OpenAIAgent.from_tools(tools, verbose=True)
   # use agent
   agent.chat("Can you create a new email to helpdesk and support @example.com about a service outage")
Full Tool details can be found on our *LlamaHub* page. Each tool
contains a "Usage" section showing how that tool can be used.
Using with LangChain
To use with a LangChain agent, simply convert tools to LangChain tools
with "to_langchain_tool()".
   tools = tool_spec.to_tool_list()
   langchain_tools = [t.to_langchain_tool() for t in tools]
   # plug into LangChain agent
   from langchain.agents import initialize_agent
   agent_executor = initialize_agent(
       langchain_tools, llm, agent="conversational-react-description", memory=memory
   )


Tools

Concept
Having proper tool abstractions is at the core of building data
agents. Defining a set of Tools is similar to defining any API
interface, with the exception that these Tools are meant for agent
rather than human use. We allow users to define both a **Tool** as
well as a **ToolSpec** containing a series of functions under the
hood.
A Tool implements a very generic interface - simply define "__call__"
and also return some basic metadata (name, description, function
schema).
A Tool Spec defines a full API specification of any service that can
be converted into a list of Tools.
We offer a few different types of Tools:
* "FunctionTool": A function tool allows users to easily convert any
  user-defined function into a Tool. It can also auto-infer the
  function schema.
* "QueryEngineTool": A tool that wraps an existing *query engine*.
  Note: since our agent abstractions inherit from "BaseQueryEngine",
  these tools can also wrap other agents.
We offer a rich set of Tools and Tool Specs through LlamaHub 🦙.
Blog Post
For full details, please check out our detailed *blog post*.
Usage Pattern
Our Tool Specs and Tools can be imported from the "llama-hub" package.
To use with our agent,
   from llama_index.agent import OpenAIAgent
   from llama_hub.tools.gmail.base import GmailToolSpec
   tool_spec = GmailToolSpec()
   agent = OpenAIAgent.from_tools(tool_spec.to_tool_list(), verbose=True)
See our Usage Pattern Guide for more details.
* Usage Pattern
LlamaHub Tools Guide 🛠
Check out our guide for a full overview of the Tools/Tool Specs in
LlamaHub!
* LlamaHub Tools Guide


Usage Pattern

Get Started
An agent is initialized from a set of Tools. Here's an example of
instantiating a ReAct agent from a set of Tools.
   from llama_index.tools import FunctionTool
   from llama_index.llms import OpenAI
   from llama_index.agent import ReActAgent
   # define sample Tool
   def multiply(a: int, b: int) -> int:
       """Multiple two integers and returns the result integer"""
       return a * b
   multiply_tool = FunctionTool.from_defaults(fn=multiply)
   # initialize llm
   llm = OpenAI(model="gpt-3.5-turbo-0613")
   # initialize ReAct agent
   agent = ReActAgent.from_tools([multiply_tool], llm=llm, verbose=True)
An agent supports both "chat" and "query" endpoints, inheriting from
our "ChatEngine" and "QueryEngine" respectively.
Example usage:
   agent.chat("What is 2123 * 215123")
Query Engine Tools
It is easy to wrap query engines as tools for an agent as well. Simply
do the following:
   from llama_index.agent import ReActAgent
   from llama_index.tools import QueryEngineTool
   # NOTE: lyft_index and uber_index are both SimpleVectorIndex instances
   lyft_engine = lyft_index.as_query_engine(similarity_top_k=3)
   uber_engine = uber_index.as_query_engine(similarity_top_k=3)
   query_engine_tools = [
       QueryEngineTool(
           query_engine=lyft_engine,
           metadata=ToolMetadata(
               name="lyft_10k",
               description="Provides information about Lyft financials for year 2021. "
               "Use a detailed plain text question as input to the tool.",
           ),
       ),
       QueryEngineTool(
           query_engine=uber_engine,
           metadata=ToolMetadata(
               name="uber_10k",
               description="Provides information about Uber financials for year 2021. "
               "Use a detailed plain text question as input to the tool.",
           ),
       ),
   ]
   # initialize ReAct agent
   agent = ReActAgent.from_tools(query_engine_tools, llm=llm, verbose=True)
Use other agents as Tools
A nifty feature of our agents is that since they inherit from
"BaseQueryEngine", you can easily define other agents as tools through
our "QueryEngineTool".
   from llama_index.tools import QueryEngineTool
   query_engine_tools = [
       QueryEngineTool(
           query_engine=sql_agent,
           metadata=ToolMetadata(
               name="sql_agent",
               description="Agent that can execute SQL queries."
           ),
       ),
       QueryEngineTool(
           query_engine=gmail_agent,
           metadata=ToolMetadata(
               name="gmail_agent",
               description="Tool that can send emails on Gmail."
           ),
       ),
   ]
   outer_agent = ReActAgent.from_tools(query_engine_tools, llm=llm, verbose=True)
Advanced Concepts (for "OpenAIAgent", in beta)
You can also use agents in more advanced settings. For instance, being
able to retrieve tools from an index during query-time, and being able
to perform query planning over an existing set of Tools.
These are largely implemented with our "OpenAIAgent" classes (which
depend on the OpenAI Function API). Support for our more general
"ReActAgent" is something we're actively investigating.
NOTE: these are largely still in beta. The abstractions may change and
become more general over time.
Function Retrieval Agents
If the set of Tools is very large, you can create an "ObjectIndex" to
index the tools, and then pass in an "ObjectRetriever" to the agent
during query-time, to first dynamically retrieve the relevant tools
before having the agent pick from the candidate tools.

Usage Pattern

We first build an "ObjectIndex" over an existing set of Tools.
   # define an "object" index over these tools
   from llama_index import VectorStoreIndex
   from llama_index.objects import ObjectIndex, SimpleToolNodeMapping
   tool_mapping = SimpleToolNodeMapping.from_objects(all_tools)
   obj_index = ObjectIndex.from_objects(
       all_tools,
       tool_mapping,
       VectorStoreIndex,
   )
We then define our "FnRetrieverOpenAIAgent":
   from llama_index.agent import FnRetrieverOpenAIAgent
   agent = FnRetrieverOpenAIAgent.from_retriever(obj_index.as_retriever(), verbose=True)
Context Retrieval Agents
Our context-augmented OpenAI Agent will always perform retrieval
before calling any tools.
This helps to provide additional context that can help the agent
better pick Tools, versus just trying to make a decision without any
context.
   from llama_index.schema import Document
   from llama_index.agent import ContextRetrieverOpenAIAgent
   # toy index - stores a list of Abbreviations
   texts = [
       "Abbreviation: X = Revenue",
       "Abbreviation: YZ = Risk Factors",
       "Abbreviation: Z = Costs",
   ]
   docs = [Document(text=t) for t in texts]
   context_index = VectorStoreIndex.from_documents(docs)
   # add context agent
   context_agent = ContextRetrieverOpenAIAgent.from_tools_and_retriever(
       query_engine_tools, context_index.as_retriever(similarity_top_k=1), verbose=True
   )
   response = context_agent.chat("What is the YZ of March 2022?")
Query Planning
OpenAI Function Agents can be capable of advanced query planning. The
trick is to provide the agent with a "QueryPlanTool" - if the agent
calls the QueryPlanTool, it is forced to infer a full Pydantic schema
representing a query plan over a set of subtools.
   # define query plan tool
   from llama_index.tools import QueryPlanTool
   from llama_index import get_response_synthesizer
   response_synthesizer = get_response_synthesizer(service_context=service_context)
   query_plan_tool = QueryPlanTool.from_defaults(
       query_engine_tools=[query_tool_sept, query_tool_june, query_tool_march],
       response_synthesizer=response_synthesizer,
   )
   # initialize agent
   agent = OpenAIAgent.from_tools(
       [query_plan_tool],
       max_function_calls=10,
       llm=OpenAI(temperature=0, model="gpt-4-0613"),
       verbose=True,
   )
   # should output a query plan to call march, june, and september tools
   response = agent.query("Analyze Uber revenue growth in March, June, and September")


Module Guides

These guide provide an overview of how to use our agent classes.
For more detailed guides on how to use specific tools, check out our
*tools module guides*.
OpenAI Agent
* Build your own OpenAI Agent
* OpenAI Agent with Query Engine Tools
* Retrieval-Augmented OpenAI Agent
* OpenAI Agent + Query Engine Experimental Cookbook
* OpenAI Agent Query Planning
* Context-Augmented OpenAI Agent
* Recursive Retriever + Document Agents
* Multi-Document Agents
ReAct Agent
* ReAct Agent with Query Engine Tools


Data Agents

Concept
Data Agents are LLM-powered knowledge workers in LlamaIndex that can
intelligently perform various tasks over your data, in both a “read”
and “write” function. They are capable of the following:
* Perform automated search and retrieval over different types of data
  - unstructured, semi-structured, and structured.
* Calling any external service API in a structured fashion, and
  processing the response + storing it for later.
In that sense, agents are a step beyond our query engines in that they
can not only "read" from a static source of data, but can dynamically
ingest and modify data from a variety of different tools.
Building a data agent requires the following core components:
* A reasoning loop
* Tool abstractions
A data agent is initialized with set of APIs, or Tools, to interact
with; these APIs can be called by the agent to return information or
modify state. Given an input task, the data agent uses a reasoning
loop to decide which tools to use, in which sequence, and the
parameters to call each tool.
Reasoning Loop
The reasoning loop depends on the type of agent. We have support for
the following agents:
* OpenAI Function agent (built on top of the OpenAI Function API)
* a ReAct agent (which works across any chat/text completion
  endpoint).
Tool Abstractions
You can learn more about our Tool abstractions in our Tools section.
Blog Post
For full details, please check out our detailed blog post.
Usage Pattern
Data agents can be used in the following manner (the example uses the
OpenAI Function API)
   from llama_index.agent import OpenAIAgent
   from llama_index.llms import OpenAI
   # import and define tools
   ...
   # initialize llm
   llm = OpenAI(model="gpt-3.5-turbo-0613")
   # initialize openai agent
   agent = OpenAIAgent.from_tools(tools, llm=llm, verbose=True)
See our usage pattern guide for more details.
* Usage Pattern
Modules
Learn more about our different agent types in our module guides below.
Also take a look at our tools section!
* Module Guides
  * OpenAI Agent
  * ReAct Agent


ServiceContext

Concept
The "ServiceContext" is a bundle of commonly used resources used
during the indexing and querying stage in a LlamaIndex
pipeline/application. You can use it to set the global configuration,
as well as local configurations at specific parts of the pipeline.
Usage Pattern
Configuring the service context
The "ServiceContext" is a simple python dataclass that you can
directly construct by passing in the desired components.
   @dataclass
   class ServiceContext:
       # The LLM used to generate natural language responses to queries.
       # If not provided, defaults to gpt-3.5-turbo from OpenAI
       # If your OpenAI key is not set, defaults to llama2-chat-13B from Llama.cpp
       llm: LLM
       # The PromptHelper object that helps with truncating and repacking text chunks to fit in the LLM's context window.
       prompt_helper: PromptHelper
       # The embedding model used to generate vector representations of text.
       # If not provided, defaults to text-embedding-ada-002
       # If your OpenAI key is not set, defaults to BAAI/bge-small-en
       embed_model: BaseEmbedding
       # The parser that converts documents into nodes.
       node_parser: NodeParser
       # The callback manager object that calls it's handlers on events. Provides basic logging and tracing capabilities.
       callback_manager: CallbackManager
       @classmethod
       def from_defaults(cls, ...) -> "ServiceContext":
         ...
Tip:
  Learn how to configure specific modules:
  * LLM
  * Embedding Model
  * Node Parser
We also expose some common kwargs (of the above components) via the
"ServiceContext.from_defaults" method for convenience (so you don't
have to manually construct them).
**Kwargs for node parser**:
* "chunk_size": The size of the text chunk for a node . Is used for
  the node parser when they aren't provided.
* "chunk_overlap": The amount of overlap between nodes (i.e. text
  chunks).
**Kwargs for prompt helper**:
* "context_window": The size of the context window of the LLM.
  Typically we set this automatically with the model metadata. But we
  also allow explicit override via this parameter for additional
  control (or in case the default is not available for certain latest
  models)
* "num_output": The number of maximum output from the LLM. Typically
  we set this automatically given the model metadata. This parameter
  does not actually limit the model output, it affects the amount of
  "space" we save for the output, when computing available context
  window size for packing text from retrieved Nodes.
Here's a complete example that sets up all objects using their default
settings:
   from llama_index import ServiceContext, LLMPredictor, OpenAIEmbedding, PromptHelper
   from llama_index.llms import OpenAI
   from llama_index.text_splitter import TokenTextSplitter
   from llama_index.node_parser import SimpleNodeParser
   llm = OpenAI(model='text-davinci-003', temperature=0, max_tokens=256)
   embed_model = OpenAIEmbedding()
   node_parser = SimpleNodeParser.from_defaults(
     text_splitter=TokenTextSplitter(chunk_size=1024, chunk_overlap=20)
   )
   prompt_helper = PromptHelper(
     context_window=4096,
     num_output=256,
     chunk_overlap_ratio=0.1,
     chunk_size_limit=None
   )
   service_context = ServiceContext.from_defaults(
     llm=llm,
     embed_model=embed_model,
     node_parser=node_parser,
     prompt_helper=prompt_helper
   )
Setting global configuration
You can set a service context as the global default that applies to

ServiceContext

the entire LlamaIndex pipeline:
   from llama_index import set_global_service_context
   set_global_service_context(service_context)
Setting local configuration
You can pass in a service context to specific part of the pipeline to
override the default configuration:
   query_engine = index.as_query_engine(service_context=service_context)
   response = query_engine.query("What did the author do growing up?")
   print(response)


Playground

Concept
The Playground module in LlamaIndex is a way to automatically test
your data (i.e. documents) across a diverse combination of indices,
models, embeddings, modes, etc. to decide which ones are best for your
purposes. More options will continue to be added.
For each combination, you'll be able to compare the results for any
query and compare the answers, latency, tokens used, and so on.
You may initialize a Playground with a list of pre-built indices, or
initialize one from a list of Documents using the preset indices.
Usage Pattern
A sample usage is given below.
   from llama_index import download_loader
   from llama_index.indices.vector_store import VectorStoreIndex
   from llama_index.indices.tree.base import TreeIndex
   from llama_index.playground import Playground
   # load data
   WikipediaReader = download_loader("WikipediaReader")
   loader = WikipediaReader()
   documents = loader.load_data(pages=['Berlin'])
   # define multiple index data structures (vector index, summary index)
   indices = [VectorStoreIndex(documents), TreeIndex(documents)]
   # initialize playground
   playground = Playground(indices=indices)
   # playground compare
   playground.compare("What is the population of Berlin?")
Modules
* Playground


Token Counting - Migration Guide

The existing token counting implementation has been **deprecated**.
We know token counting is important to many users, so this guide was
created to walkthrough a (hopefully painless) transition.
Previously, token counting was kept track of on the "llm_predictor"
and "embed_model" objects directly, and optionally printed to the
console. This implementation used a static tokenizer for token
counting (gpt-2), and the "last_token_usage" and "total_token_usage"
attributes were not always kept track of properly.
Going forward, token counting as moved into a callback. Using the
"TokenCountingHandler" callback, you now have more options for how
tokens are counted, the lifetime of the token counts, and even
creating separate token counters for different indexes.
Here is a minimum example of using the new "TokenCountingHandler" with
an OpenAI model:
   import tiktoken
   from llama_index.callbacks import CallbackManager, TokenCountingHandler
   from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
   # you can set a tokenizer directly, or optionally let it default
   # to the same tokenizer that was used previously for token counting
   # NOTE: The tokenizer should be a function that takes in text and returns a list of tokens
   token_counter = TokenCountingHandler(
       tokenizer=tiktoken.encoding_for_model("text-davinci-003").encode
       verbose=False  # set to true to see usage printed to the console
   )
   callback_manager = CallbackManager([token_counter])
   service_context = ServiceContext.from_defaults(callback_manager=callback_manager)
   document = SimpleDirectoryReader("./data").load_data()
   # if verbose is turned on, you will see embedding token usage printed
   index = VectorStoreIndex.from_documents(documents, service_context=service_context)
   # otherwise, you can access the count directly
   print(token_counter.total_embedding_token_count)
   # reset the counts at your discretion!
   token_counter.reset_counts()
   # also track prompt, completion, and total LLM tokens, in addition to embeddings
   response = index.as_query_engine().query("What did the author do growing up?")
   print('Embedding Tokens: ', token_counter.total_embedding_token_count, '\n',
         'LLM Prompt Tokens: ', token_counter.prompt_llm_token_count, '\n',
         'LLM Completion Tokens: ', token_counter.completion_llm_token_count, '\n',
         'Total LLM Token Count: ', token_counter.total_llm_token_count)


Callbacks

Concept
LlamaIndex provides callbacks to help debug, track, and trace the
inner workings of the library. Using the callback manager, as many
callbacks as needed can be added.
In addition to logging data related to events, you can also track the
duration and number of occurrences of each event.
Furthermore, a trace map of events is also recorded, and callbacks can
use this data however they want. For example, the "LlamaDebugHandler"
will, by default, print the trace of events after most operations.
**Callback Event Types** While each callback may not leverage each
event type, the following events are available to be tracked:
* "CHUNKING" -> Logs for the before and after of text splitting.
* "NODE_PARSING" -> Logs for the documents and the nodes that they are
  parsed into.
* "EMBEDDING" -> Logs for the number of texts embedded.
* "LLM" -> Logs for the template and response of LLM calls.
* "QUERY" -> Keeps track of the start and end of each query.
* "RETRIEVE" -> Logs for the nodes retrieved for a query.
* "SYNTHESIZE" -> Logs for the result for synthesize calls.
* "TREE" -> Logs for the summary and level of summaries generated.
* "SUB_QUESTION" -> Log for a generated sub question and answer.
You can implement your own callback to track and trace these events,
or use an existing callback.
Modules
Currently supported callbacks are as follows:
* TokenCountingHandler -> Flexible token counting for prompt,
  completion, and embedding token usage. See the migration details
  *here*
* LlamaDebugHanlder -> Basic tracking and tracing for events. Example
  usage can be found in the notebook below.
* WandbCallbackHandler -> Tracking of events and traces using the
  Wandb Prompts frontend. More details are in the notebook below or at
  Wandb
* AimCallback -> Tracking of LLM inputs and outputs. Example usage can
  be found in the notebook below.
* OpenInferenceCallbackHandler -> Tracking of AI model inferences.
  Example usage can be found in the notebook below.
* OpenAIFineTuningHandler -> Records all LLM inputs and outputs. Then,
  provides a function "save_finetuning_events()" to save inputs and
  outputs in a format suitable for fine-tuning with OpenAI.


Usage Pattern

Estimating LLM and Embedding Token Counts
In order to measure LLM and Embedding token counts, you'll need to
1. Setup "MockLLM" and "MockEmbedding" objects
   from llama_index.llms import MockLLM
   from llama_index import MockEmbedding
   llm = MockLLM(max_tokens=256)
   embed_model = MockEmbedding(embed_dim=1536)
2. Setup the "TokenCountingCallback" handler
   import tiktoken
   from llama_index.callbacks import CallbackManager, TokenCountingHandler
   token_counter = TokenCountingHandler(
       tokenizer=tiktoken.encoding_for_model("gpt-3.5-turbo").encode
   )
   callback_manager = CallbackManager([token_counter])
3. Add them to the global "ServiceContext"
   from llama_index import ServiceContext, set_global_service_context
   set_global_service_context(
       ServiceContext.from_defaults(
           llm=llm,
           embed_model=embed_model,
           callback_manager=callback_manager
       )
   )
4. Construct an Index
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   documents = SimpleDirectoryReader("./docs/examples/data/paul_graham").load_data()
   index = VectorStoreIndex.from_documents(documents)
5. Measure the counts!
   print(
       "Embedding Tokens: ",
       token_counter.total_embedding_token_count,
       "\n",
       "LLM Prompt Tokens: ",
       token_counter.prompt_llm_token_count,
       "\n",
       "LLM Completion Tokens: ",
       token_counter.completion_llm_token_count,
       "\n",
       "Total LLM Token Count: ",
       token_counter.total_llm_token_count,
       "\n",
   )
   # reset counts
   token_counter.reset_counts()
6. Run a query, mesaure again
   query_engine = index.as_query_engine()
   response = query_engine.query("query")
   print(
       "Embedding Tokens: ",
       token_counter.total_embedding_token_count,
       "\n",
       "LLM Prompt Tokens: ",
       token_counter.prompt_llm_token_count,
       "\n",
       "LLM Completion Tokens: ",
       token_counter.completion_llm_token_count,
       "\n",
       "Total LLM Token Count: ",
       token_counter.total_llm_token_count,
       "\n",
   )


Cost Analysis

Concept
Each call to an LLM will cost some amount of money - for instance,
OpenAI's gpt-3.5-turbo costs $0.002 / 1k tokens. The cost of building
an index and querying depends on
* the type of LLM used
* the type of data structure used
* parameters used during building
* parameters used during querying
The cost of building and querying each index is a TODO in the
reference documentation. In the meantime, we provide the following
information:
1. A high-level overview of the cost structure of the indices.
2. A token predictor that you can use directly within LlamaIndex!
Overview of Cost Structure
Indices with no LLM calls
~~~~~~~~~~~~~~~~~~~~~~~~~
The following indices don't require LLM calls at all during building
(0 cost):
* "SummaryIndex"
* "SimpleKeywordTableIndex" - uses a regex keyword extractor to
  extract keywords from each document
* "RAKEKeywordTableIndex" - uses a RAKE keyword extractor to extract
  keywords from each document
Indices with LLM calls
~~~~~~~~~~~~~~~~~~~~~~
The following indices do require LLM calls during build time:
* "TreeIndex" - use LLM to hierarchically summarize the text to build
  the tree
* "KeywordTableIndex" - use LLM to extract keywords from each document
Query Time
There will always be >= 1 LLM call during query time, in order to
synthesize the final answer. Some indices contain cost tradeoffs
between index building and querying. "SummaryIndex", for instance, is
free to build, but running a query over a summary index (without
filtering or embedding lookups), will call the LLM N times.
Here are some notes regarding each of the indices:
* "SummaryIndex": by default requires N LLM calls, where N is the
  number of nodes.
* "TreeIndex": by default requires \log (N) LLM calls, where N is the
  number of leaf nodes.
  * Setting "child_branch_factor=2" will be more expensive than the
    default "child_branch_factor=1" (polynomial vs logarithmic),
    because we traverse 2 children instead of just 1 for each parent
    node.
* "KeywordTableIndex": by default requires an LLM call to extract
  query keywords.
  * Can do "index.as_retriever(retriever_mode="simple")" or
    "index.as_retriever(retriever_mode="rake")" to also use regex/RAKE
    keyword extractors on your query text.
* "VectorStoreIndex": by default, requires one LLM call per query. If
  you increase the "similarity_top_k" or "chunk_size", or change the
  "response_mode", then this number will increase.
Usage Pattern
LlamaIndex offers token **predictors** to predict token usage of LLM
and embedding calls. This allows you to estimate your costs during 1)
index construction, and 2) index querying, before any respective LLM
calls are made.
Tokens are counted using the "TokenCountingHandler" callback. See the
example notebook for details on the setup.
Using MockLLM
To predict token usage of LLM calls, import and instantiate the
MockLLM as shown below. The "max_tokens" parameter is used as a "worst
case" prediction, where each LLM response will contain exactly that
number of tokens. If "max_tokens" is not specified, then it will
simply predict back the prompt.
   from llama_index import ServiceContext, set_global_service_context
   from llama_index.llms import MockLLM
   llm = MockLLM(max_tokens=256)
   service_context = ServiceContext.from_defaults(llm=llm)

Cost Analysis

   # optionally set a global service context
   set_global_service_context(service_context)
You can then use this predictor during both index construction and
querying.
Using MockEmbedding
You may also predict the token usage of embedding calls with
"MockEmbedding".
   from llama_index import ServiceContext, set_global_service_context
   from llama_index import MockEmbedding
   # specify a MockLLMPredictor
   embed_model = MockEmbedding(embed_dim=1536)
   service_context = ServiceContext.from_defaults(embed_model=embed_model)
   # optionally set a global service context
   set_global_service_context(service_context)
Usage Pattern
Read about the full usage pattern below!
Examples
^^^^^^^^
* Usage Pattern


Usage Pattern (Retrieval)

Using "RetrieverEvaluator"
This runs evaluation over a single query + ground-truth document set
given a retriever.
The standard practice is to specify a set of valid metrics with
"from_metrics".
   from llama_index.evaluation import RetrieverEvaluator
   # define retriever somewhere (e.g. from index)
   # retriever = index.as_retriever(similarity_top_k=2)
   retriever = ...
   retriever_evaluator = RetrieverEvaluator.from_metric_names(
       ["mrr", "hit_rate"], retriever=retriever
   )
   retriever_evaluator.evaluate(
       query="query",
       expected_ids=["node_id1", "node_id2"]
   )
Building an Evaluation Dataset
You can manually curate a retrieval evaluation dataset of questions +
node id's. We also offer synthetic dataset generation over an existing
text corpus with our "generate_question_context_pairs" function:
   from llama_index.evaluation import generate_question_context_pairs
   qa_dataset = generate_question_context_pairs(
       nodes,
       llm=llm,
       num_questions_per_chunk=2
   )
The returned result is a "EmbeddingQAFinetuneDataset" object
(containing "queries", "relevant_docs", and "corpus").
Plugging it into "RetrieverEvaluator"
We offer a convenience function to run a "RetrieverEvaluator" over a
dataset in batch mode.
   eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)
This should run much faster than you trying to call ".evaluate" on
each query separately.


Usage Pattern (Response Evaluation)

Using "BaseEvaluator"
All of the evaluation modules in LlamaIndex implement the
"BaseEvaluator" class, with two main methods:
1. The "evaluate" method takes in "query", "contexts", "response", and
   additional keyword arguments.
       def evaluate(
           self,
           query: Optional[str] = None,
           contexts: Optional[Sequence[str]] = None,
           response: Optional[str] = None,
           **kwargs: Any,
       ) -> EvaluationResult:
2. The "evaluate_response" method provide an alternative interface
   that takes in a llamaindex "Response" object (which contains
   response string and source nodes) instead of separate "contexts"
   and "response".
   def evaluate_response(
       self,
       query: Optional[str] = None,
       response: Optional[Response] = None,
       **kwargs: Any,
   ) -> EvaluationResult:
It's functionally the same as "evaluate", just simpler to use when
working with llamaindex objects directly.
Using "EvaluationResult"
Each evaluator outputs a "EvaluationResult" when executed:
   eval_result = evaluator.evaluate(query=..., contexts=..., response=...)
   eval_result.passing  # binary pass/fail
   eval_result.score  # numerical score
   eval_result.feedback  # string feedback
Different evaluators may populate a subset of the result fields.
Evaluating Response Faithfulness (i.e. Hallucination)
The "FaithfulnessEvaluator" evaluates if the answer is faithful to the
retrieved contexts (in other words, whether if there's hallucination).
   from llama_index import VectorStoreIndex, ServiceContext
   from llama_index.llms import OpenAI
   from llama_index.evaluation import FaithfulnessEvaluator
   # build service context
   llm = OpenAI(model="gpt-4", temperature=0.0)
   service_context = ServiceContext.from_defaults(llm=llm)
   # build index
   ...
   # define evaluator
   evaluator = FaithfulnessEvaluator(service_context=service_context)
   # query index
   query_engine = vector_index.as_query_engine()
   response = query_engine.query("What battles took place in New York City in the American Revolution?")
   eval_result = evaluator.evaluate_response(response=response)
   print(str(eval_result.passing))
[image: ][image]
You can also choose to evaluate each source context individually:
   from llama_index import VectorStoreIndex, ServiceContext
   from llama_index.llms import OpenAI
   from llama_index.evaluation import FaithfulnessEvaluator
   # build service context
   llm = OpenAI(model="gpt-4", temperature=0.0)
   service_context = ServiceContext.from_defaults(llm=llm)
   # build index
   ...
   # define evaluator
   evaluator = FaithfulnessEvaluator(service_context=service_context)
   # query index
   query_engine = vector_index.as_query_engine()
   response = query_engine.query("What battles took place in New York City in the American Revolution?")
   response_str = response.response
   for source_node in response.source_nodes:
       eval_result = evaluator.evaluate(response=response_str, contexts=[source_node.get_content()])
       print(str(eval_result.passing))
You'll get back a list of results, corresponding to each source node
in "response.source_nodes".
Evaluating Query + Response Relevancy
The "RelevancyEvaluator" evaluates if the retrieved context and the
answer is relevant and consistent for the given query.
Note that this evaluator requires the "query" to be passed in, in
addition to the "Response" object.
   from llama_index import VectorStoreIndex, ServiceContext
   from llama_index.llms import OpenAI
   from llama_index.evaluation import RelevancyEvaluator
   # build service context
   llm = OpenAI(model="gpt-4", temperature=0.0)

Usage Pattern (Response Evaluation)

   service_context = ServiceContext.from_defaults(llm=llm)
   # build index
   ...
   # define evaluator
   evaluator = RelevancyEvaluator(service_context=service_context)
   # query index
   query_engine = vector_index.as_query_engine()
   query = "What battles took place in New York City in the American Revolution?"
   response = query_engine.query(query)
   eval_result = evaluator.evaluate_response(query=query, response=response)
   print(str(eval_result))
[image: ][image]
Similarly, you can also evaluate on a specific source node.
   from llama_index import VectorStoreIndex, ServiceContext
   from llama_index.llms import OpenAI
   from llama_index.evaluation import RelevancyEvaluator
   # build service context
   llm = OpenAI(model="gpt-4", temperature=0.0)
   service_context = ServiceContext.from_defaults(llm=llm)
   # build index
   ...
   # define evaluator
   evaluator = RelevancyEvaluator(service_context=service_context)
   # query index
   query_engine = vector_index.as_query_engine()
   query = "What battles took place in New York City in the American Revolution?"
   response = query_engine.query(query)
   response_str = response.response
   for source_node in response.source_nodes:
       eval_result = evaluator.evaluate(query=query, response=response_str, contexts=[source_node.get_content()])
       print(str(eval_result.passing))
[image: ][image]
Question Generation
LlamaIndex can also generate questions to answer using your data.
Using in combination with the above evaluators, you can create a fully
automated evaluation pipeline over your data.
   from llama_index import SimpleDirectoryReader, ServiceContext
   from llama_index.llms import OpenAI
   from llama_index.evaluation import DatasetGenerator
   # build service context
   llm = OpenAI(model="gpt-4", temperature=0.0)
   service_context = ServiceContext.from_defaults(llm=llm)
   # build documents
   documents = SimpleDirectoryReader("./data").load_data()
   # define generator, generate questions
   data_generator = DatasetGenerator.from_documents(documents)
   eval_questions = data_generator.generate_questions_from_nodes()
Batch Evaluation
We also provide a batch evaluation runner for running a set of
evaluators across many questions.
   from llama_index.evaluation import BatchEvalRunner
   runner = BatchEvalRunner(
       {
           "faithfulness": faithfulness_evaluator, "
           "relevancy": relevancy_evaluator
       },
       workers=8,
   )
   eval_results = await runner.aevaluate_queries(
       vector_index.as_query_engine(), queries=questions
   )
Integrations
We also integrate with community evaluation tools.
* DeepEval
* Ragas


Modules

Notebooks with usage of these components can be found below.
Response Evaluation
* Faithfulness Evaluator
* Relevancy Evaluator
* Guideline Evaluator
* Correctness Evaluator
* Embedding Similarity Evaluator
* LlamaIndex + DeepEval Integration
* QuestionGeneration
* BatchEvalRunner - Running Multiple Evaluations
Retrieval Evaluation
* Retrieval Evaluation


Evaluation

Concept
Evaluation and benchmarking are crucial concepts in LLM development.
To improve the performance of an LLM app (RAG, agents), you must have
a way to measure it.
LlamaIndex offers key modules to measure the quality of generated
results. We also offer key modules to measure retrieval quality.
* **Response Evaluation**: Does the response match the retrieved
  context? Does it also match the query? Does it match the reference
  answer or guidelnes?
* **Retrieval Evaluation**: Are the retrieved sources relevant to the
  query?
This section describes how the evaluation components within LlamaIndex
work.
Response Evaluation
Evaluation of generated results can be difficult, since unlike
traditional machine learning the predicted result isn't a single
number, and it can be hard to define quantitative metrics for this
problem.
LlamaIndex offers **LLM-based** evaluation modules to measure the
quality of results. This uses a "gold" LLM (e.g. GPT-4) to decide
whether the predicted answer is correct in a variety of ways.
Note that many of these current evaluation modules do *not* require
ground-truth labels. Evaluation can be done with some combination of
the query, context, response, and combine these with LLM calls.
These evaluation modules are in the following forms:
* **Correctness**: Whether the generated answer matches that of the
  reference answer given the query (requires labels).
* **Semantic Similarity** Whether the predicted answer is semantically
  similar to the reference answer (requires labels).
* **Faithfulness**: Evaluates if the answer is faithful to the
  retrieved contexts (in other words, whether if there's
  hallucination).
* **Context Relevancy**: Whether retrieved context and answer are
  relevant to the query.
* **Guideline Adherence**: Whether the predicted answer adheres to
  specific guidelines.
Question Generation
~~~~~~~~~~~~~~~~~~~
In addition to evaluating queries, LlamaIndex can also use your data
to generate questions to evaluate on. This means that you can
automatically generate questions, and then run an evaluation pipeline
to test if the LLM can actually answer questions accurately using your
data.
Retrieval Evaluation
We also provide modules to help evaluate retrieval independently.
The concept of retrieval evaluation is not new; given a dataset of
questions and ground-truth rankings, we can evaluate retrievers using
ranking metrics like mean-reciprocal rank (MRR), hit-rate, precision,
and more.
The core retrieval evaluation steps revolve around the following:
* **Dataset generation**: Given an unstructured text corpus,
  synthetically generate (question, context) pairs.
* **Retrieval Evaluation**: Given a retriever and a set of questions,
  evaluate retrieved results using ranking metrics.
Integrations
We also integrate with community evaluation tools.
* DeepEval
* Ragas
Usage Pattern
For full usage details, see the usage pattern below.
* Usage Pattern (Response Evaluation)
* Usage Pattern (Retrieval)
Modules
Notebooks with usage of these components can be found below.
* Modules


Usage Pattern

Most commonly, node-postprocessors will be used in a query engine,
where they are applied to the nodes returned from a retriever, and
before the response synthesis step.
Using with a Query Engine
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   from llama_index.indices.postprocessor import TimeWeightedPostprocessor
   documents = SimpleDirectoryReader("./data").load_data()
   index = VectorStoreIndex.from_documents(documents)
   query_engine = index.as_query_engine(
     node_postprocessors=[
       TimeWeightedPostprocessor(
           time_decay=0.5, time_access_refresh=False, top_k=1
       )
     ]
   )
   # all node post-processors will be applied during each query
   response = query_engine.query("query string")
Using with Retrieved Nodes
Or used as a standalone object for filtering retrieved nodes:
   from llama_index.indices.postprocessor import SimilarityPostprocessor
   nodes = index.as_retriever().retrieve("test query str")
   # filter nodes below 0.75 similarity score
   processor = SimilarityPostprocessor(similarity_cutoff=0.75)
   filtered_nodes = processor.postprocess_nodes(nodes)
Using with your own nodes
As you may have noticed, the postprocessors take "NodeWithScore"
objects as inputs, which is just a wrapper class with a "Node" and a
"score" value.
   from llama_index.indices.postprocessor import SimilarityPostprocessor
   from llama_index.schema import Node, NodeWithScore
   nodes = [
     NodeWithScore(node=Node(text="text"), score=0.7),
     NodeWithScore(node=Node(text="text"), score=0.8)
   ]
   # filter nodes below 0.75 similarity score
   processor = SimilarityPostprocessor(similarity_cutoff=0.75)
   filtered_nodes = processor.postprocess_nodes(nodes)
Custom Node PostProcessor
The base class is "BaseNodePostprocessor", and the API interface is
very simple:
   class BaseNodePostprocessor:
       """Node postprocessor."""
       @abstractmethod
       def postprocess_nodes(
           self, nodes: List[NodeWithScore], query_bundle: Optional[QueryBundle]
       ) -> List[NodeWithScore]:
           """Postprocess nodes."""
A dummy node-postprocessor can be implemented in just a few lines of
code:
   from llama_index import QueryBundle
   from llama_index.indices.postprocessor.base import BaseNodePostprocessor
   from llama_index.schema import NodeWithScore
   class DummyNodePostprocessor:
       def postprocess_nodes(
           self, nodes: List[NodeWithScore], query_bundle: Optional[QueryBundle]
       ) -> List[NodeWithScore]:
           # subtracts 1 from the score
           for n in nodes:
               n.score -= 1
           return nodes


Modules

SimilarityPostprocessor
Used to remove nodes that are below a similarity score threshold.
   from llama_index.indices.postprocessor import SimilarityPostprocessor
   postprocessor = SimilarityPostprocessor(similarity_cutoff=0.7)
   postprocessor.postprocess_nodes(nodes)
KeywordNodePostprocessor
Used to ensure certain keywords are either excluded or included.
   from llama_index.indices.postprocessor import KeywordNodePostprocessor
   postprocessor = KeywordNodePostprocessor(
     required_keywords=["word1", "word2"],
     exclude_keywords=["word3", "word4"]
   )
   postprocessor.postprocess_nodes(nodes)
MetadataReplacementPostProcessor
Used to replace the node content with a field from the node metadata.
If the field is not present in the metadata, then the node text
remains unchanged. Most useful when used in combination with the
"SentenceWindowNodeParser".
   from llama_index.indices.postprocessor import MetadataReplacementPostProcessor
   postprocessor = MetadataReplacementPostProcessor(
     target_metadata_key="window",
   )
   postprocessor.postprocess_nodes(nodes)
LongContextReorder
Models struggle to access significant details found in the center of
extended contexts. A study observed that the best performance
typically arises when crucial data is positioned at the start or
conclusion of the input context. Additionally, as the input context
lengthens, performance drops notably, even in models designed for long
contexts.
This module will re-order the retrieved nodes, which can be helpful in
cases where a large top-k is needed.
   from llama_index.indices.postprocessor import LongContextReorder
   postprocessor = LongContextReorder()
   postprocessor.postprocess_nodes(nodes)
SentenceEmbeddingOptimizer
This postprocessor optimizes token usage by removing sentences that
are not relevant to the query (this is done using embeddings).
The percentile cutoff is a measure for using the top percentage of
relevant sentences.
The threshold cutoff can be specified instead, which uses a raw
similarity cutoff for picking which sentences to keep.
   from llama_index.indices.postprocessor import SentenceEmbeddingOptimizer
   postprocessor = SentenceEmbeddingOptimizer(
     embed_model=service_context.embed_model,
     percentile_cutoff=0.5,
     # threshold_cutoff=0.7
   )
   postprocessor.postprocess_nodes(nodes)
A full notebook guide can be found here
CohereRerank
Uses the "Cohere ReRank" functionality to re-order nodes, and returns
the top N nodes.
   from llama_index.indices import CohereRerank
   postprocessor = CohereRerank(
     top_n=2
     model="rerank-english-v2.0",
     api_key="YOUR COHERE API KEY"
   )
   postprocessor.postprocess_nodes(nodes)
Full notebook guide is available here.
SentenceTransformerRerank
Uses the cross-encoders from the "sentence-transformer" package to re-
order nodes, and returns the top N nodes.
   from llama_index.indices.postprocessor import SentenceTransformerRerank
   # We choose a model with relatively high speed and decent accuracy.
   postprocessor = SentenceTransformerRerank(
     model="cross-encoder/ms-marco-MiniLM-L-2-v2",
     top_n=3
   )
   postprocessor.postprocess_nodes(nodes)
Full notebook guide is available here.
Please also refer to the "sentence-transformer" docs for a more
complete list of models (and also shows tradeoffs in speed/accuracy).
The default model is "cross-encoder/ms-marco-TinyBERT-L-2-v2", which
provides the most speed.
LLM Rerank
Uses a LLM to re-order nodes by asking the LLM to return the relevant
documents and a score of how relevant they are. Returns the top N
ranked nodes.
   from llama_index.indices.postprocessor import LLMRerank
   postprocessor = LLMRerank(

Modules

     top_n=2
     service_context=service_context,
   )
   postprocessor.postprocess_nodes(nodes)
Full notebook guide is available her for Gatsby and here for Lyft 10K
documents.
FixedRecencyPostprocessor
This postproccesor returns the top K nodes sorted by date. This
assumes there is a "date" field to parse in the metadata of each node.
   from llama_index.indices.postprocessor import FixedRecencyPostprocessor
   postprocessor = FixedRecencyPostprocessor(
     tok_k=1,
     date_key="date"  # the key in the metadata to find the date
   )
   postprocessor.postprocess_nodes(nodes)
[image: ][image]
A full notebook guide is available here.
EmbeddingRecencyPostprocessor
This postproccesor returns the top K nodes after sorting by date and
removing older nodes that are too similar after measuring embedding
similarity.
   from llama_index.indices.postprocessor import EmbeddingRecencyPostprocessor
   postprocessor = EmbeddingRecencyPostprocessor(
     service_context=service_context,
     date_key="date",
     similarity_cutoff=0.7
   )
   postprocessor.postprocess_nodes(nodes)
A full notebook guide is available here.
TimeWeightedPostprocessor
This postproccesor returns the top K nodes applying a time-weighted
rerank to each node. Each time a node is retrieved, the time it was
retrieved is recorded. This biases search to favor information that
has not be returned in a query yet.
   from llama_index.indices.postprocessor import TimeWeightedPostprocessor
   postprocessor = TimeWeightedPostprocessor(
     time_decay=0.99,
     top_k=1
   )
   postprocessor.postprocess_nodes(nodes)
A full notebook guide is available here.
(Beta) PIINodePostprocessor
The PII (Personal Identifiable Information) postprocssor removes
information that might be a security risk. It does this by using NER
(either with a dedicated NER model, or with a local LLM model).
LLM Version
   from llama_index.indices.postprocessor import PIINodePostprocessor
   postprocessor = PIINodePostprocessor(
     service_context=service_context,  # this should be setup with an LLM you trust
   )
   postprocessor.postprocess_nodes(nodes)
NER Version
This version uses the default local model from Hugging Face that is
loaded when you run "pipeline("ner")".
   from llama_index.indices.postprocessor import NERPIINodePostprocessor
   postprocessor = NERPIINodePostprocessor()
   postprocessor.postprocess_nodes(nodes)
A full notebook guide for both can be found here.
(Beta) PrevNextNodePostprocessor
Uses pre-defined settings to read the "Node" relationships and fetch
either all nodes that come previously, next, or both.
This is useful when you know the relationships point to important data
(either before, after, or both) that should be sent to the LLM if that
node is retrieved.
   from llama_index.indices.postprocessor import PrevNextNodePostprocessor
   postprocessor = PrevNextNodePostprocessor(
     docstore=index.docstore,
     num_nodes=1,  # number of nodes to fetch when looking forawrds or backwards
     mode="next"   # can be either 'next', 'previous', or 'both'
   )
   postprocessor.postprocess_nodes(nodes)
[image: ][image]
(Beta) AutoPrevNextNodePostprocessor
The same as PrevNextNodePostprocessor, but lets the LLM decide the
mode (next, previous, or both).
   from llama_index.indices.postprocessor import AutoPrevNextNodePostprocessor
   postprocessor = AutoPrevNextNodePostprocessor(
     docstore=index.docstore,
     service_context=service_context
     num_nodes=1,  # number of nodes to fetch when looking forawrds or backwards)

Modules

   postprocessor.postprocess_nodes(nodes)
A full example notebook is available here.
All Notebooks
* Sentence Embedding Optimizer
* Cohere Rerank
* LLM Reranker Demonstration (2021 Lyft 10-k)
* LLM Reranker Demonstration (Great Gatsby)
* Recency Filtering
* Time-Weighted Rerank
* PII Masking
* Forward/Backward Augmentation
* Metadata Replacement + Node Sentence Window
* LongContextReorder


Node Postprocessor

Concept
Node postprocessors are a set of modules that take a set of nodes, and
apply some kind of transformation or filtering before returning them.
In LlamaIndex, node postprocessors are most commonly applied within a
query engine, after the node retrieval step and before the response
synthesis step.
LlamaIndex offers several node postprocessors for immediate use, while
also providing a simple API for adding your own custom postprocessors.
Tip:
  Confused about where node postprocessor fits in the pipeline? Read
  about high-level concepts
Usage Pattern
An example of using a node postprocessors is below:
   from llama_index.indices.postprocessor import SimilarityPostprocessor
   from llama_index.schema import Node, NodeWithScore
   nodes = [
     NodeWithScore(node=Node(text="text"), score=0.7),
     NodeWithScore(node=Node(text="text"), score=0.8)
   ]
   # filter nodes below 0.75 similarity score
   processor = SimilarityPostprocessor(similarity_cutoff=0.75)
   filtered_nodes = processor.postprocess_nodes(nodes)
You can find more details using post processors and how to build your
own below.
* Usage Pattern
  * Using with a Query Engine
  * Using with Retrieved Nodes
  * Using with your own nodes
  * Custom Node PostProcessor
Modules
Below you can find guides for each node postprocessor.
* Modules
  * SimilarityPostprocessor
  * KeywordNodePostprocessor
  * MetadataReplacementPostProcessor
  * LongContextReorder
  * SentenceEmbeddingOptimizer
  * CohereRerank
  * SentenceTransformerRerank
  * LLM Rerank
  * FixedRecencyPostprocessor
  * EmbeddingRecencyPostprocessor
  * TimeWeightedPostprocessor
  * (Beta) PIINodePostprocessor
  * (Beta) PrevNextNodePostprocessor
  * (Beta) AutoPrevNextNodePostprocessor
  * All Notebooks


Usage Pattern

Get Started
Build a chat engine from index:
   chat_engine = index.as_chat_engine()
Tip:
  To learn how to build an index, see Index
Have a conversation with your data:
   response = chat_engine.chat("Tell me a joke.")
Reset chat history to start a new conversation:
   chat_engine.reset()
Enter an interactive chat REPL:
   chat_engine.chat_repl()
Configuring a Chat Engine
Configuring a chat engine is very similar to configuring a query
engine.
High-Level API
You can directly build and configure a chat engine from an index in 1
line of code:
   chat_engine = index.as_chat_engine(
       chat_mode='condense_question',
       verbose=True
   )
   Note: you can access different chat engines by specifying the
   "chat_mode" as a kwarg. "condense_question" corresponds to
   "CondenseQuestionChatEngine", "react" corresponds to
   "ReActChatEngine", "context" corresponds to a "ContextChatEngine".
   Note: While the high-level API optimizes for ease-of-use, it does
   *NOT* expose full range of configurability.
Available Chat Modes
~~~~~~~~~~~~~~~~~~~~
* "best" - Turn the query engine into a tool, for use with a "ReAct"
  data agent or an "OpenAI" data agent, depending on what your LLM
  supports. "OpenAI" data agents require "gpt-3.5-turbo" or "gpt-4" as
  they use the function calling API from OpenAI.
* "context" - Retrieve nodes from the index using every user message.
  The retrieved text is inserted into the system prompt, so that the
  chat engine can either respond naturally or use the context from the
  query engine.
* "condense_question" - Look at the chat history and re-write the user
  message to be a query for the index. Return the response after
  reading the response from the query engine.
* "simple" - A simple chat with the LLM directly, no query engine
  involved.
* "react" - Same as "best", but forces a "ReAct" data agent.
* "openai" - Same as "best", but forces an "OpenAI" data agent.
Low-Level Composition API
You can use the low-level composition API if you need more granular
control. Concretely speaking, you would explicitly construct
"ChatEngine" object instead of calling "index.as_chat_engine(...)".
   Note: You may need to look at API references or example notebooks.
Here's an example where we configure the following:
* configure the condense question prompt,
* initialize the conversation with some existing history,
* print verbose debug message.
   from llama_index.prompts  import PromptTemplate
   from llama_index.llms import ChatMessage, MessageRole
   custom_prompt = PromptTemplate("""\
   Given a conversation (between Human and Assistant) and a follow up message from Human, \
   rewrite the message to be a standalone question that captures all relevant context \
   from the conversation.
   <Chat History>
   {chat_history}
   <Follow Up Message>
   {question}
   <Standalone question>
   """)
   # list of `ChatMessage` objects
   custom_chat_history = [
       ChatMessage(
           role=MessageRole.USER,
           content='Hello assistant, we are having a insightful discussion about Paul Graham today.'
       ),
       ChatMessage(
           role=MessageRole.ASSISTANT,
           content='Okay, sounds good.'
       )
   ]
   query_engine = index.as_query_engine()
   chat_engine = CondenseQuestionChatEngine.from_defaults(
       query_engine=query_engine,
       condense_question_prompt=custom_prompt,
       chat_history=custom_chat_history,
       verbose=True
   )
Streaming
To enable streaming, you simply need to call the "stream_chat"

Usage Pattern

endpoint instead of the "chat" endpoint.
Warning:
  This somewhat inconsistent with query engine (where you pass in a
  "streaming=True" flag). We are working on making the behavior more
  consistent!
   chat_engine = index.as_chat_engine()
   streaming_response = chat_engine.stream_chat("Tell me a joke.")
   for token in streaming_response.response_gen:
       print(token, end="")
See an end-to-end tutorial


Module Guides

We provide a few simple implementations to start, with more
sophisticated modes coming soon!
More specifically, the "SimpleChatEngine" does not make use of a
knowledge base, whereas all others make use of a query engine over
knowledge base.
* ReAct Chat Engine
* OpenAI Chat Engine
* Context Chat Engine
* Condense Question Chat Engine
* Simple Chat Engine


Chat Engine

Concept
Chat engine is a high-level interface for having a conversation with
your data (multiple back-and-forth instead of a single question &
answer). Think ChatGPT, but augmented with your knowledge base.
Conceptually, it is a **stateful** analogy of a Query Engine. By
keeping track of the conversation history, it can answer questions
with past context in mind.
Tip:
  If you want to ask standalone question over your data (i.e. without
  keeping track of conversation history), use Query Engine instead.
Usage Pattern
Get started with:
   chat_engine = index.as_chat_engine()
   response = chat_engine.chat("Tell me a joke.")
To stream response:
   chat_engine = index.as_chat_engine()
   streaming_response = chat_engine.stream_chat("Tell me a joke.")
   for token in streaming_response.response_gen:
       print(token, end="")
* Usage Pattern
  * Get Started
  * Configuring a Chat Engine
Modules
Below you can find corresponding tutorials to see the available chat
engines in action.
* Module Guides
  * ReAct Chat Engine
  * OpenAI Chat Engine
  * Context Chat Engine
  * Condense Question Chat Engine
  * Simple Chat Engine


Retriever Modes

Here we show the mapping from "retriever_mode" configuration to the
selected retriever class.
   Note that "retriever_mode" can mean different thing for different
   index classes.
Vector Index
Specifying "retriever_mode" has no effect (silently ignored).
"vector_index.as_retriever(...)" always returns a
VectorIndexRetriever.
Summary Index
* "default": SummaryIndexRetriever
* "embedding": SummaryIndexEmbeddingRetriever
* "llm": SummaryIndexLLMRetriever
Tree Index
* "select_leaf": TreeSelectLeafRetriever
* "select_leaf_embedding": TreeSelectLeafEmbeddingRetriever
* "all_leaf": TreeAllLeafRetriever
* "root": TreeRootRetriever
Keyword Table Index
* "default": KeywordTableGPTRetriever
* "simple": KeywordTableSimpleRetriever
* "rake": KeywordTableRAKERetriever
Knowledge Graph Index
* "keyword": KGTableRetriever
* "embedding": KGTableRetriever
* "hybrid": KGTableRetriever
Document Summary Index
* "llm": DocumentSummaryIndexLLMRetriever
* "embedding": DocumentSummaryIndexEmbeddingRetrievers


Usage Pattern

Get Started
Get a retriever from index:
   retriever = index.as_retriever()
Retrieve relevant context for a question:
   nodes = retriever.retrieve('Who is Paul Graham?')
   Note: To learn how to build an index, see Index
High-Level API
Selecting a Retriever
You can select the index-specific retriever class via
"retriever_mode". For example, with a "SummaryIndex":
   retriever = summary_index.as_retriever(
       retriever_mode='llm',
   )
This creates a SummaryIndexLLMRetriever on top of the summary index.
See **Retriever Modes** for a full list of (index-specific) retriever
modes and the retriever classes they map to.
Configuring a Retriever
In the same way, you can pass kwargs to configure the selected
retriever.
   Note: take a look at the API reference for the selected retriever
   class' constructor parameters for a list of valid kwargs.
For example, if we selected the "llm" retriever mode, we might do the
following:
   retriever = summary_index.as_retriever(
       retriever_mode='llm',
       choice_batch_size=5,
   )
Low-Level Composition API
You can use the low-level composition API if you need more granular
control.
To achieve the same outcome as above, you can directly import and
construct the desired retriever class:
   from llama_index.indices.list import SummaryIndexLLMRetriever
   retriever = SummaryIndexLLMRetriever(
       index=summary_index,
       choice_batch_size=5,
   )
Advanced
* Define Custom Retriever
* BM25 Hybrid Retriever
* Simple Fusion Retriever
* Reciprocal Rerank Fusion Retriever


Module Guides

We are actively adding more tailored retrieval guides. In the
meanwhile, please take a look at the API References.
Index Retrievers
Please see the retriever modes for more details on how to get a
retriever from any given index.
If you want to import the corresponding retrievers directly, please
check out our API reference.
Advanced Retriever Guides
Check out our comprehensive guides on various retriever modules, many
of which cover advanced concepts (auto-retrieval, routing, ensembling,
and more).
External Retrievers
* BM25 Retriever
Knowledge Graph Retrievers
* Custom Retriever (KG Index and Vector Store Index)
* Knowledge Graph RAG Retriever
Composed Retrievers
* Simple Fusion Retriever
* Reciprocal Rerank Fusion Retriever
* Auto-Retrieval (with Chroma)
* Auto-Retrieval (with BagelDB)
* Recursive Retriever + Query Engine Demo
* Router Retriever
* Ensemble Query Engine Guide
* Auto Merging Retriever
* Recursive Retriever + Node References
* Recursive Retriever + Node References + Braintrust
* Metadata Replacement + Node Sentence Window


Retriever

Concept
Retrievers are responsible for fetching the most relevant context
given a user query (or chat message).
It can be built on top of Indices, but can also be defined
independently. It is used as a key building block in Query Engines
(and Chat Engines) for retrieving relevant context.
Tip:
  Confused about where retriever fits in the pipeline? Read about
  high-level concepts
Usage Pattern
Get started with:
   retriever = index.as_retriever()
   nodes = retriever.retrieve("Who is Paul Graham?")
* Usage Pattern
  * Get Started
  * High-Level API
  * Low-Level Composition API
  * Advanced
Modules
* Module Guides
  * Index Retrievers
  * Advanced Retriever Guides
  * External Retrievers
  * Knowledge Graph Retrievers
  * Composed Retrievers


Pydantic Program

A pydantic program is a generic abstraction that takes in an input
string and converts it to a structured Pydantic object type.
Because this abstraction is so generic, it encompasses a broad range
of LLM workflows. The programs are composable and be for more generic
or specific use cases.
There's a few general types of Pydantic Programs:
* **LLM Text Completion Pydantic Programs**: These convert input text
  into a user-specified structured object through a text completion
  API + output parsing.
* **LLM Function Calling Pydantic Program**: These convert input text
  into a user-specified structured object through an LLM function
  calling API.
* **Prepackaged Pydantic Programs**: These convert input text into
  prespecified structured objects.
LLM Text Completion Pydantic Programs
TODO: Coming soon!
LLM Function Calling Pydantic Programs
* OpenAI Pydantic Program
* Guidance Pydantic Program
* Guidance for Sub-Question Query Engine
Prepackaged Pydantic Programs
* DataFrame Structured Data Extraction
* Evaporate Demo


Output Parsing

LlamaIndex supports integrations with output parsing modules offered
by other frameworks. These output parsing modules can be used in the
following ways:
* To provide formatting instructions for any prompt / query (through
  "output_parser.format")
* To provide "parsing" for LLM outputs (through "output_parser.parse")
Guardrails
Guardrails is an open-source Python package for
specification/validation/correction of output schemas. See below for a
code example.
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   from llama_index.output_parsers import GuardrailsOutputParser
   from llama_index.llm_predictor import StructuredLLMPredictor
   from llama_index.prompts import PromptTemplate
   from llama_index.prompts.default_prompts import DEFAULT_TEXT_QA_PROMPT_TMPL, DEFAULT_REFINE_PROMPT_TMPL
   # load documents, build index
   documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()
   index = VectorStoreIndex(documents, chunk_size=512)
   llm_predictor = StructuredLLMPredictor()
   # specify StructuredLLMPredictor
   # this is a special LLMPredictor that allows for structured outputs
   # define query / output spec
   rail_spec = ("""
   <rail version="0.1">
   <output>
       <list name="points" description="Bullet points regarding events in the author's life.">
           <object>
               <string name="explanation" format="one-line" on-fail-one-line="noop" />
               <string name="explanation2" format="one-line" on-fail-one-line="noop" />
               <string name="explanation3" format="one-line" on-fail-one-line="noop" />
           </object>
       </list>
   </output>
   <prompt>
   Query string here.
   @xml_prefix_prompt
   {output_schema}
   @json_suffix_prompt_v2_wo_none
   </prompt>
   </rail>
   """)
   # define output parser
   output_parser = GuardrailsOutputParser.from_rail_string(rail_spec, llm=llm_predictor.llm)
   # format each prompt with output parser instructions
   fmt_qa_tmpl = output_parser.format(DEFAULT_TEXT_QA_PROMPT_TMPL)
   fmt_refine_tmpl = output_parser.format(DEFAULT_REFINE_PROMPT_TMPL)
   qa_prompt = PromptTemplate(fmt_qa_tmpl, output_parser=output_parser)
   refine_prompt = PromptTemplate(fmt_refine_tmpl, output_parser=output_parser)
   # obtain a structured response
   query_engine = index.as_query_engine(
       service_context=ServiceContext.from_defaults(
           llm_predictor=llm_predictor
       ),
       text_qa_template=qa_prompt,
       refine_template=refine_prompt,
   )
   response = query_engine.query(
       "What are the three items the author did growing up?",
   )
   print(response)
Output:
   {'points': [{'explanation': 'Writing short stories', 'explanation2': 'Programming on an IBM 1401', 'explanation3': 'Using microcomputers'}]}
Langchain
Langchain also offers output parsing modules that you can use within
LlamaIndex.
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   from llama_index.output_parsers import LangchainOutputParser
   from llama_index.llm_predictor import StructuredLLMPredictor
   from llama_index.prompts import PromptTemplate
   from llama_index.prompts.default_prompts import DEFAULT_TEXT_QA_PROMPT_TMPL, DEFAULT_REFINE_PROMPT_TMPL
   from langchain.output_parsers import StructuredOutputParser, ResponseSchema
   # load documents, build index
   documents = SimpleDirectoryReader('../paul_graham_essay/data').load_data()

Output Parsing

   index = VectorStoreIndex.from_documents(documents)
   llm_predictor = StructuredLLMPredictor()
   # define output schema
   response_schemas = [
       ResponseSchema(name="Education", description="Describes the author's educational experience/background."),
       ResponseSchema(name="Work", description="Describes the author's work experience/background.")
   ]
   # define output parser
   lc_output_parser = StructuredOutputParser.from_response_schemas(response_schemas)
   output_parser = LangchainOutputParser(lc_output_parser)
   # format each prompt with output parser instructions
   fmt_qa_tmpl = output_parser.format(DEFAULT_TEXT_QA_PROMPT_TMPL)
   fmt_refine_tmpl = output_parser.format(DEFAULT_REFINE_PROMPT_TMPL)
   qa_prompt = PromptTemplate(fmt_qa_tmpl, output_parser=output_parser)
   refine_prompt = PromptTemplate(fmt_refine_tmpl, output_parser=output_parser)
   # query index
   query_engine = index.as_query_engine(
       service_context=ServiceContext.from_defaults(
           llm_predictor=llm_predictor
       ),
       text_qa_template=qa_prompt,
       refine_template=refine_prompt,
   )
   response = query_engine.query(
       "What are a few things the author did growing up?",
   )
   print(str(response))
Output:
   {'Education': 'Before college, the author wrote short stories and experimented with programming on an IBM 1401.', 'Work': 'The author worked on writing and programming outside of school.'}
Guides
Examples
^^^^^^^^
* Guardrails Output Parsing
* Langchain Output Parsing
* Guidance Pydantic Program
* Guidance for Sub-Question Query Engine
* OpenAI Pydantic Program


Query Engines + Pydantic Outputs

Using "index.as_query_engine()" and it's underlying
"RetrieverQueryEngine", we can support structured pydantic outputs
without an additional LLM calls (in contrast to a typical output
parser.)
Every query engine has support for integrated structured responses
using the following "response_mode"s in "RetrieverQueryEngine":
* "refine"
* "compact"
* "tree_summarize"
* "accumulate" (beta, requires extra parsing to convert to objects)
* "compact_accumulate" (beta, requires extra parsing to convert to
  objects)
Under the hood, this uses "OpenAIPydanitcProgam" or
"LLMTextCompletionProgram" depending on which LLM you've setup. If
there are intermediate LLM responses (i.e. during "refine" or
"tree_summarize" with multiple LLM calls), the pydantic object is
injected into the next LLM prompt as a JSON object.
Usage Pattern
First, you need to define the object you want to extract.
   from typing import List
   from pydantic import BaseModel
   class Biography(BaseModel):
       """Data model for a biography."""
       name: str
       best_known_for: List[str]
       extra_info: str
Then, you create your query engine.
   query_engine = index.as_query_engine(response_mode="tree_summarize", output_cls=Biography)
Lastly, you can get a response and inspect the output.
   response = query_engine.query("Who is Paul Graham?")
   print(response.name)
   > 'Paul Graham'
   print(response.best_known_for)
   > ['working on Bel', 'co-founding Viaweb', 'creating the programming language Arc']
   print(response.extra_info)
   > "Paul Graham is a computer scientist, entrepreneur, and writer. He is best known for ..."
Modules
Detailed usage is available in the notebooks below:
* Query Engine with Pydantic Outputs
  * Setup
  * Create the Index + Query Engine (OpenAI)
  * Create the Index + Query Engine (Non-OpenAI, Beta)
  * Accumulate Examples (Beta)
* Pydantic Tree Summarize
  * Load Data
  * Summarize


Structured Outputs

The ability of LLMs to produce structured outputs are important for
downstream applications that rely on reliably parsing output values.
LlamaIndex itself also relies on structured output in the following
ways.
* **Document retrieval**: Many data structures within LlamaIndex rely
  on LLM calls with a specific schema for Document retrieval. For
  instance, the tree index expects LLM calls to be in the format
  "ANSWER: (number)".
* **Response synthesis**: Users may expect that the final response
  contains some degree of structure (e.g. a JSON output, a formatted
  SQL query, etc.)
LlamaIndex provides a variety of modules enabling LLMs to produce
outputs in a structured format. We provide modules at different levels
of abstraction:
* **Output Parsers**: These are modules that operate before and after
  an LLM text completion endpoint. They are not used with LLM function
  calling endpoints (since those contain structured outputs out of the
  box).
* **Pydantic Programs**: These are generic modules that map an input
  prompt to a structured output, represented by a Pydantic object.
  They may use function calling APIs or text completion APIs + output
  parsers. These can also be integrated with query engines.
* **Pre-defined Pydantic Program**: We have pre-defined Pydantic
  programs that map inputs to specific output types (like dataframes).
See the sections below for an overview of output parsers and Pydantic
programs.
🔬 Anatomy of a Structured Output Function
Here we describe the different components of an LLM-powered structured
output function. The pipeline depends on whether you're using a
**generic LLM text completion API** or an **LLM function calling
API**.
[image: ][image]
With generic completion APIs, the inputs and outputs are handled by
text prompts. The output parser plays a role before and after the LLM
call in ensuring structured outputs. Before the LLM call, the output
parser can append format instructions to the prompt. After the LLM
call, the output parser can parse the output to the specified
instructions.
With function calling APIs, the output is inherently in a structured
format, and the input can take in the signature of the desired object.
The structured output just needs to be cast in the right object format
(e.g. Pydantic).
Query Engine Modules
* Query Engines + Pydantic Outputs
  * Usage Pattern
  * Modules
Output Parser Modules
* Output Parsing
  * Guardrails
  * Langchain
  * Guides
Pydantic Program Modules
* Pydantic Program
  * LLM Text Completion Pydantic Programs
  * LLM Function Calling Pydantic Programs
  * Prepackaged Pydantic Programs


Usage Pattern

Get Started
Configuring the response synthesizer for a query engine using
"response_mode":
   from llama_index.schema import Node, NodeWithScore
   from llama_index.response_synthesizers import get_response_synthesizer
   response_synthesizer = get_response_synthesizer(response_mode='compact')
   response = response_synthesizer.synthesize(
     "query text",
     nodes=[NodeWithScore(node=Node(text="text"), score=1.0), ..]
   )
Or, more commonly, in a query engine after you've created an index:
   query_engine = index.as_query_engine(response_synthesizer=response_synthesizer)
   response = query_engine.query("query_text")
Tip:
  To learn how to build an index, see Index
Configuring the Response Mode
Response synthesizers are typically specified through a
"response_mode" kwarg setting.
Several response synthesizers are implemented already in LlamaIndex:
* "refine": ***create and refine*** an answer by sequentially going
  through each retrieved text chunk. This makes a separate LLM call
  per Node/retrieved chunk.
  **Details:** the first chunk is used in a query using the
  "text_qa_template" prompt. Then the answer and the next chunk (as
  well as the original question) are used in another query with the
  "refine_template" prompt. And so on until all chunks have been
  parsed.
  If a chunk is too large to fit within the window (considering the
  prompt size), it is split using a "TokenTextSplitter" (allowing some
  text overlap between chunks) and the (new) additional chunks are
  considered as chunks of the original chunks collection (and thus
  queried with the "refine_template" as well).
  Good for more detailed answers.
* "compact" (default): similar to "refine" but ***compact***
  (concatenate) the chunks beforehand, resulting in less LLM calls.
  **Details:** stuff as many text (concatenated/packed from the
  retrieved chunks) that can fit within the context window
  (considering the maximum prompt size between "text_qa_template" and
  "refine_template"). If the text is too long to fit in one prompt, it
  is split in as many parts as needed (using a "TokenTextSplitter" and
  thus allowing some overlap between text chunks).
  Each text part is considered a "chunk" and is sent to the "refine"
  synthesizer.
  In short, it is like "refine", but with less LLM calls.
* "tree_summarize": Query the LLM using the "summary_template" prompt
  as many times as needed so that all concatenated chunks have been
  queried, resulting in as many answers that are themselves
  recursively used as chunks in a "tree_summarize" LLM call and so on,
  until there's only one chunk left, and thus only one final answer.
  **Details:** concatenate the chunks as much as possible to fit
  within the context window using the "summary_template" prompt, and
  split them if needed (again with a "TokenTextSplitter" and some text
  overlap). Then, query each resulting chunk/split against
  "summary_template" (there is no ***refine*** query !) and get as
  many answers.
  If there is only one answer (because there was only one chunk), then
  it's the final answer.
  If there are more than one answer, these themselves are considered
  as chunks and sent recursively to the "tree_summarize" process
  (concatenated/splitted-to-fit/queried).
  Good for summarization purposes.
* "simple_summarize": Truncates all text chunks to fit into a single

Usage Pattern

  LLM prompt. Good for quick summarization purposes, but may lose
  detail due to truncation.
* "no_text": Only runs the retriever to fetch the nodes that would
  have been sent to the LLM, without actually sending them. Then can
  be inspected by checking "response.source_nodes".
* "accumulate": Given a set of text chunks and the query, apply the
  query to each text chunk while accumulating the responses into an
  array. Returns a concatenated string of all responses. Good for when
  you need to run the same query separately against each text chunk.
* "compact_accumulate": The same as accumulate, but will "compact"
  each LLM prompt similar to "compact", and run the same query against
  each text chunk.
Custom Response Synthesizers
Each response synthesizer inherits from
"llama_index.response_synthesizers.base.BaseSynthesizer". The base API
is extremely simple, which makes it easy to create your own response
synthesizer.
Maybe you want to customize which template is used at each step in
"tree_summarize", or maybe a new research paper came out detailing a
new way to generate a response to a query, you can create your own
response synthesizer and plug it into any query engine or use it on
it's own.
Below we show the "__init__()" function, as well as the two abstract
methods that every response synthesizer must implement. The basic
requirements are to process a query and text chunks, and return a
string (or string generator) response.
   class BaseSynthesizer(ABC):
       """Response builder class."""
       def __init__(
           self,
           service_context: Optional[ServiceContext] = None,
           streaming: bool = False,
       ) -> None:
           """Init params."""
           self._service_context = service_context or ServiceContext.from_defaults()
           self._callback_manager = self._service_context.callback_manager
           self._streaming = streaming
       @abstractmethod
       def get_response(
           self,
           query_str: str,
           text_chunks: Sequence[str],
           **response_kwargs: Any,
       ) -> RESPONSE_TEXT_TYPE:
           """Get response."""
           ...
       @abstractmethod
       async def aget_response(
           self,
           query_str: str,
           text_chunks: Sequence[str],
           **response_kwargs: Any,
       ) -> RESPONSE_TEXT_TYPE:
           """Get response."""
           ...
Using Structured Answer Filtering
When using either the ""refine"" or ""compact"" response synthesis
modules, you may find it beneficial to experiment with the
"structured_answer_filtering" option.
   from llama_index.response_synthesizers import get_response_synthesizer
   response_synthesizer = get_response_synthesizer(structured_answer_filtering=True)
With "structured_answer_filtering" set to "True", our refine module is
able to filter out any input nodes that are not relevant to the
question being asked. This is particularly useful for RAG-based Q&A
systems that involve retrieving chunks of text from external vector
store for a given user query.
This option is particularly useful if you're using an OpenAI model
that supports function calling. Other LLM providers or models that
don't have native function calling support may be less reliable in
producing the structured response this feature relies on.


Module Guide

Detailed inputs/outputs for each response synthesizer are found below.
API Example
The following shows the setup for utilizing all kwargs.
* "response_mode" specifies which response synthesizer to use
* "service_context" defines the LLM and related settings for synthesis
* "text_qa_template" and "refine_template" are the prompts used at
  various stages
* "use_async" is used for only the "tree_summarize" response mode
  right now, to asynchronously build the summary tree
* "streaming" configures whether to return a streaming response object
  or not
* "structured_answer_filtering" enables the active filtering of text
  chunks that are not relevant to a given question
In the "synthesize"/"asyntheszie" functions, you can optionally
provide additional source nodes, which will be added to the
"response.source_nodes" list.
   from llama_index.schema import Node, NodeWithScore
   from llama_index import get_response_synthesizer
   response_synthesizer = get_response_synthesizer(
     response_mode="refine",
     service_context=service_context,
     text_qa_template=text_qa_template,
     refine_template=refine_template,
     use_async=False,
     streaming=False
   )
   # synchronous
   response = response_synthesizer.synthesize(
     "query string",
     nodes=[NodeWithScore(node=Node(text="text"), score=1.0), ..],
     additional_source_nodes=[NodeWithScore(node=Node(text="text"), score=1.0), ..],
   )
   # asynchronous
   response = await response_synthesizer.asynthesize(
     "query string",
     nodes=[NodeWithScore(node=Node(text="text"), score=1.0), ..],
     additional_source_nodes=[NodeWithScore(node=Node(text="text"), score=1.0), ..],
   )
You can also directly return a string, using the lower-level
"get_response" and "aget_response" functions
   response_str = response_synthesizer.get_response(
     "query string",
     text_chunks=["text1", "text2", ...]
   )
Example Notebooks
* Refine
* Refine with Structured Answer Filtering
* Tree Summarize


Response Synthesizer

Concept
A "Response Synthesizer" is what generates a response from an LLM,
using a user query and a given set of text chunks. The output of a
response synthesizer is a "Response" object.
The method for doing this can take many forms, from as simple as
iterating over text chunks, to as complex as building a tree. The main
idea here is to simplify the process of generating a response using an
LLM across your data.
When used in a query engine, the response synthesizer is used after
nodes are retrieved from a retriever, and after any node-
postprocessors are ran.
Tip:
  Confused about where response synthesizer fits in the pipeline? Read
  the high-level concepts
Usage Pattern
Use a response synthesizer on it's own:
   from llama_index.schema import Node
   from llama_index.response_synthesizers import ResponseMode, get_response_synthesizer
   response_synthesizer = get_response_synthesizer(response_mode=ResponseMode.COMPACT)
   response = response_synthesizer.synthesize("query text", nodes=[Node(text="text"), ...])
Or in a query engine after you've created an index:
   query_engine = index.as_query_engine(response_synthesizer=response_synthesizer)
   response = query_engine.query("query_text")
You can find more details on all available response synthesizers,
modes, and how to build your own below.
* Usage Pattern
  * Get Started
  * Configuring the Response Mode
  * Custom Response Synthesizers
  * Using Structured Answer Filtering
Modules
Below you can find detailed API information for each response
synthesis module.
* Module Guide


Supporting Modules

* Query Transformations


Streaming

LlamaIndex supports streaming the response as it's being generated.
This allows you to start printing or processing the beginning of the
response before the full response is finished. This can drastically
reduce the perceived latency of queries.
Setup
To enable streaming, you need to use an LLM that supports streaming.
Right now, streaming is supported by "OpenAI", "HuggingFaceLLM", and
most LangChain LLMs (via "LangChainLLM").
Configure query engine to use streaming:
If you are using the high-level API, set "streaming=True" when
building a query engine.
   query_engine = index.as_query_engine(
       streaming=True,
       similarity_top_k=1
   )
If you are using the low-level API to compose the query engine, pass
"streaming=True" when constructing the "Response Synthesizer":
   from llama_index import get_response_synthesizer
   synth = get_response_synthesizer(streaming=True, ...)
   query_engine = RetrieverQueryEngine(response_synthesizer=synth, ...)
Streaming Response
After properly configuring both the LLM and the query engine, calling
"query" now returns a "StreamingResponse" object.
   streaming_response = query_engine.query(
       "What did the author do growing up?",
   )
The response is returned immediately when the LLM call *starts*,
without having to wait for the full completion.
   Note: In the case where the query engine makes multiple LLM calls,
   only the last LLM call will be streamed and the response is
   returned when the last LLM call starts.
You can obtain a "Generator" from the streaming response and iterate
over the tokens as they arrive:
   for text in streaming_response.response_gen:
       # do something with text as they arrive.
Alternatively, if you just want to print the text as they arrive:
   streaming_response.print_response_stream()
See an end-to-end example


Usage Pattern

Get Started
Build a query engine from index:
   query_engine = index.as_query_engine()
Tip:
  To learn how to build an index, see Index
Ask a question over your data
   response = query_engine.query('Who is Paul Graham?')
Configuring a Query Engine
High-Level API
You can directly build and configure a query engine from an index in 1
line of code:
   query_engine = index.as_query_engine(
       response_mode='tree_summarize',
       verbose=True,
   )
   Note: While the high-level API optimizes for ease-of-use, it does
   *NOT* expose full range of configurability.
See **Response Modes** for a full list of response modes and what they
do.
Low-Level Composition API
You can use the low-level composition API if you need more granular
control. Concretely speaking, you would explicitly construct a
"QueryEngine" object instead of calling "index.as_query_engine(...)".
   Note: You may need to look at API references or example notebooks.
   from llama_index import (
       VectorStoreIndex,
       get_response_synthesizer,
   )
   from llama_index.retrievers import VectorIndexRetriever
   from llama_index.query_engine import RetrieverQueryEngine
   # build index
   index = VectorStoreIndex.from_documents(documents)
   # configure retriever
   retriever = VectorIndexRetriever(
       index=index,
       similarity_top_k=2,
   )
   # configure response synthesizer
   response_synthesizer = get_response_synthesizer(
       response_mode="tree_summarize",
   )
   # assemble query engine
   query_engine = RetrieverQueryEngine(
       retriever=retriever,
       response_synthesizer=response_synthesizer,
   )
   # query
   response = query_engine.query("What did the author do growing up?")
   print(response)
Streaming
To enable streaming, you simply need to pass in a "streaming=True"
flag
   query_engine = index.as_query_engine(
       streaming=True,
   )
   streaming_response = query_engine.query(
       "What did the author do growing up?",
   )
   streaming_response.print_response_stream()
* Read the full streaming guide
* See an end-to-end example
Defining a Custom Query Engine
You can also define a custom query engine. Simply subclass the
"CustomQueryEngine" class, define any attributes you'd want to have
(similar to defining a Pydantic class), and implement a "custom_query"
function that returns either a "Response" object or a string.
   from llama_index.query_engine import CustomQueryEngine
   from llama_index.retrievers import BaseRetriever
   from llama_index.response_synthesizers import get_response_synthesizer, BaseSynthesizer
   class RAGQueryEngine(CustomQueryEngine):
       """RAG Query Engine."""
       retriever: BaseRetriever
       response_synthesizer: BaseSynthesizer
       def custom_query(self, query_str: str):
           nodes = self.retriever.retrieve(query_str)
           response_obj = self.response_synthesizer.synthesize(query_str, nodes)
           return response_obj
See the Custom Query Engine guide for more details.