10K Analysis

In this demo, we explore answering complex queries by decomposing them
into simpler sub-queries.
   import nest_asyncio
   nest_asyncio.apply()
   from llama_index import SimpleDirectoryReader, ServiceContext, VectorStoreIndex
   from llama_index.llms import OpenAI
   from llama_index.tools import QueryEngineTool, ToolMetadata
   from llama_index.query_engine import SubQuestionQueryEngine
   /Users/suo/miniconda3/envs/llama/lib/python3.9/site-packages/deeplake/util/check_latest_version.py:32: UserWarning: A newer version of deeplake (3.6.7) is available. It's recommended that you update to the latest version using `pip install -U deeplake`.
     warnings.warn(
Configure LLM service
   llm = OpenAI(temperature=0, model="text-davinci-003", max_tokens=-1)
   service_context = ServiceContext.from_defaults(llm=llm)
Load data
   lyft_docs = SimpleDirectoryReader(input_files=["../data/10k/lyft_2021.pdf"]).load_data()
   uber_docs = SimpleDirectoryReader(input_files=["../data/10k/uber_2021.pdf"]).load_data()
Build indices
   lyft_index = VectorStoreIndex.from_documents(lyft_docs)
   uber_index = VectorStoreIndex.from_documents(uber_docs)
Build query engines
   lyft_engine = lyft_index.as_query_engine(similarity_top_k=3)
   uber_engine = uber_index.as_query_engine(similarity_top_k=3)
   query_engine_tools = [
       QueryEngineTool(
           query_engine=lyft_engine,
           metadata=ToolMetadata(
               name="lyft_10k",
               description="Provides information about Lyft financials for year 2021",
           ),
       ),
       QueryEngineTool(
           query_engine=uber_engine,
           metadata=ToolMetadata(
               name="uber_10k",
               description="Provides information about Uber financials for year 2021",
           ),
       ),
   ]
   s_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=query_engine_tools)
Run queries
   response = s_engine.query(
       "Compare and contrast the customer segments and geographies that grew the fastest"
   )
   Generated 4 sub questions.
   [uber_10k] Q: What customer segments grew the fastest for Uber
   [uber_10k] Q: What geographies grew the fastest for Uber
   [lyft_10k] Q: What customer segments grew the fastest for Lyft
   [lyft_10k] Q: What geographies grew the fastest for Lyft
   [uber_10k] A:
   Uber experienced the fastest growth in five metropolitan areas—Chicago, Miami, and New York City in the United States, Sao Paulo in Brazil, and London in the United Kingdom. Additionally, Uber experienced growth in suburban and rural areas, though the network is smaller and less liquid in these areas.
   [lyft_10k] A:
   Lyft has seen the fastest growth in its ridesharing marketplace, Express Drive, Lyft Rentals, Light Vehicles, Public Transit, and Lyft Autonomous customer segments. These customer segments have seen increased demand due to the convenience and high-quality experience they offer drivers and riders, as well as the investments Lyft has made in its proprietary technology, M&A and strategic partnerships, and brand and marketing efforts.
   [lyft_10k] A:

10K Analysis

   Lyft has grown rapidly in cities across the United States and in select cities in Canada. The ridesharing market grew rapidly prior to the COVID-19 pandemic, and it is uncertain to what extent market acceptance will continue to grow after the pandemic. The market for Lyft's other offerings, such as its network of Light Vehicles, is also new and unproven, and it is uncertain whether demand for bike and scooter sharing will continue to grow.
   [uber_10k] A: in 2021?
   The customer segments that grew the fastest for Uber in 2021 were Riders and Eaters, who use the platform for ridesharing services and meal preparation, grocery, and other delivery services, respectively. Additionally, Uber One, Uber Pass, Eats Pass, and Rides Pass membership programs grew significantly in 2021, with over 6 million members.

   print(response)
   Uber and Lyft both experienced the fastest growth in their respective customer segments and geographies in 2021.
   For Uber, the fastest growing customer segments were Riders and Eaters, who use the platform for ridesharing services and meal preparation, grocery, and other delivery services, respectively. Additionally, Uber One, Uber Pass, Eats Pass, and Rides Pass membership programs grew significantly in 2021, with over 6 million members. Uber experienced the fastest growth in five metropolitan areas—Chicago, Miami, and New York City in the United States, Sao Paulo in Brazil, and London in the United Kingdom. Additionally, Uber experienced growth in suburban and rural areas, though the network is smaller and less liquid in these areas.
   For Lyft, the fastest growing customer segments were ridesharing, Express Drive, Lyft Rentals, Light Vehicles, Public Transit, and Lyft Autonomous. Lyft has grown rapidly in cities across the United States and in select cities in Canada. The ridesharing market grew rapidly prior to the COVID-19 pandemic, and it is uncertain to what extent market acceptance will continue to grow after the pandemic. The market for Lyft's other offerings, such as its network of Light Vehicles, is also new and unproven, and it is uncertain whether demand for bike and scooter sharing will continue to grow.
   Overall, Uber and Lyft experienced the fastest growth in different customer segments and geographies. Uber experienced the fastest growth in Riders and Eaters, as well as in five metropolitan areas, while Lyft experienced the fastest growth in ridesharing, Express Drive, Lyft Rentals, Light Vehicles, Public Transit, and Lyft Autonomous, as well as in cities across the United States and in select cities in Canada.
   response = s_engine.query("Compare revenue growth of Uber and Lyft from 2020 to 2021")
   Generated 2 sub questions.
   [uber_10k] Q: What is the revenue growth of Uber from 2020 to 2021
   [lyft_10k] Q: What is the revenue growth of Lyft from 2020 to 2021
   [lyft_10k] A:
   The revenue of Lyft grew by 36% from 2020 to 2021.
   [uber_10k] A:
   The revenue growth of Uber from 2020 to 2021 was 57%, or 54% on a constant currency basis.

   print(response)
   The revenue growth of Uber from 2020 to 2021 was 57%, or 54% on a constant currency basis, while the revenue of Lyft grew by 36% from 2020 to 2021. Therefore, Uber had a higher revenue growth than Lyft from 2020 to 2021.


Agents

Context
An "agent" is an automated reasoning and decision engine. It takes in
a user input/query and can make internal decisions for executing that
query in order to return the correct result. The key agent components
can include, but are not limited to:
* Breaking down a complex question into smaller ones
* Choosing an external Tool to use + coming up with parameters for
  calling the Tool
* Planning out a set of tasks
* Storing previously completed tasks in a memory module
Research developments in LLMs (e.g. ChatGPT Plugins), LLM research
(ReAct, Toolformer) and LLM tooling (LangChain, Semantic Kernel) have
popularized the concept of agents.
Agents + LlamaIndex
LlamaIndex provides some amazing tools to manage and interact with
your data within your LLM application. And it can be a core tool that
you use while building an agent-based app.
* On one hand, some components within LlamaIndex are "agent-like" -
  these make automated decisions to help a particular use case over
  your data.
* On the other hand, LlamaIndex can be used as a core Tool within
  another agent framework.
In general, LlamaIndex components offer more explicit, constrained
behavior for more specific use cases. Agent frameworks such as ReAct
(implemented in LangChain) offer agents that are more unconstrained +
capable of general reasoning.
There are tradeoffs for using both - less-capable LLMs typically do
better with more constraints. Take a look at our blog post on this for
a more information + a detailed analysis.
"Agent-like" Components within LlamaIndex
LlamaIndex provides core modules capable of automated reasoning for
different use cases over your data. Please check out our use cases doc
for more details on high-level use cases that LlamaIndex can help
fulfill.
Some of these core modules are shown below along with example
tutorials (not comprehensive, please click into the guides/how-tos for
more details).
**SubQuestionQueryEngine for Multi-Document Analysis**
* *Usage*
* Sub Question Query Engine (Intro)
* 10Q Analysis (Uber)
* 10K Analysis (Uber and Lyft)
**Query Transformations**
* How-To
* Multi-Step Query Decomposition (Notebook)
**Routing**
* *Usage*
* Router Query Engine Guide (Notebook)
**LLM Reranking**
* Second Stage Processing How-To
* LLM Reranking Guide (Great Gatsby)
**Chat Engines**
* Chat Engines How-To
Using LlamaIndex as as Tool within an Agent Framework
LlamaIndex can be used as as Tool within an agent framework -
including LangChain, ChatGPT. These integrations are described below.
LangChain
~~~~~~~~~
We have deep integrations with LangChain. LlamaIndex query engines can
be easily packaged as Tools to be used within a LangChain agent, and
LlamaIndex can also be used as a memory module / retriever. Check out
our guides/tutorials below!
**Resources**
* LangChain integration guide
* *Building a Chatbot Tutorial (LangChain + LlamaIndex)*
* OnDemandLoaderTool Tutorial
ChatGPT
~~~~~~~
LlamaIndex can be used as a ChatGPT retrieval plugin (we have a TODO
to develop a more general plugin as well).
**Resources**
* LlamaIndex ChatGPT Retrieval Plugin
Native OpenAIAgent
With the new OpenAI API that supports function calling, it’s never
been easier to build your own agent!
Learn how to write your own OpenAI agent in **under 50 lines of
code**, or directly use our super simple "OpenAIAgent" implementation.
* Build your own OpenAI Agent
* OpenAI Agent with Query Engine Tools
* Retrieval-Augmented OpenAI Agent

Agents

* OpenAI Agent + Query Engine Experimental Cookbook
* OpenAI Agent Query Planning
* Context-Augmented OpenAI Agent


Use Cases

* Q&A over Documents
* Chatbots
* Agents
* Knowledge Graphs
* Structured Data
* Full-Stack Web Application
* Private Setup
* Finetuning Llama 2 for Text-to-SQL
* Finetuning GPT-3.5 to Distill GPT-4


Chatbots

Chatbots are an incredibly popular use case for LLM's. LlamaIndex
gives you the tools to build Knowledge-augmented chatbots and agents.
Relevant Resources:
* Building a Chatbot
* Using with a LangChain Agent


Structured Data

Relevant Resources:
* A Guide to LlamaIndex + Structured Data
* Airbyte SQL Index Guide


Full-Stack Web Application

LlamaIndex can be integrated into a downstream full-stack web
application. It can be used in a backend server (such as Flask),
packaged into a Docker container, and/or directly used in a framework
such as Streamlit.
We provide tutorials and resources to help you get started in this
area.
Relevant Resources:
* Fullstack Application Guide
* Fullstack Application with Delphic
* A Guide to Extracting Terms and Definitions
* LlamaIndex Starter Pack


Discover LlamaIndex Video Series

This page contains links to videos + associated notebooks for our
ongoing video tutorial series "Discover LlamaIndex".
Bottoms-Up Development (Llama Docs Bot)
This is a sub-series within Discover LlamaIndex that shows you how to
build a document chatbot from scratch.
We show you how to do this in a "bottoms-up" fashion - start by using
the LLMs, data objects as independent modules. Then gradually add
higher-level abstractions like indexing, and advanced
retrievers/rerankers.
Full Repo [Part 1] LLMs and Prompts [Part 2] Documents and Metadata
[Part 3] Evaluation [Part 4] Embeddings [Part 5] Retrievers and
Postprocessors
SubQuestionQueryEngine + 10K Analysis
This video covers the "SubQuestionQueryEngine" and how it can be
applied to financial documents to help decompose complex queries into
multiple sub-questions.
Youtube
Notebook
Discord Document Management
This video covers managing documents from a source that is consantly
updating (i.e Discord) and how you can avoid document duplication and
save embedding tokens.
Youtube
Notebook + Supplementary Material
Reference Docs
Joint Text to SQL and Semantic Search
This video covers the tools built into LlamaIndex for combining SQL
and semantic search into a single unified query interface.
Youtube
Notebook


Q&A over Documents

At a high-level, LlamaIndex gives you the ability to query your data
for any downstream LLM use case, whether it's question-answering,
summarization, or a component in a chatbot.
This section describes the different ways you can query your data with
LlamaIndex, roughly in order of simplest (top-k semantic search), to
more advanced capabilities.
Semantic Search
The most basic example usage of LlamaIndex is through semantic search.
We provide a simple in-memory vector store for you to get started, but
you can also choose to use any one of our vector store integrations:
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   documents = SimpleDirectoryReader('data').load_data()
   index = VectorStoreIndex.from_documents(documents)
   query_engine = index.as_query_engine()
   response = query_engine.query("What did the author do growing up?")
   print(response)
**Tutorials**
* Starter Tutorial
* Basic Usage Pattern
**Guides**
* Example (Notebook)
Summarization
A summarization query requires the LLM to iterate through many if not
most documents in order to synthesize an answer. For instance, a
summarization query could look like one of the following:
* "What is a summary of this collection of text?"
* "Give me a summary of person X's experience with the company."
In general, a summary index would be suited for this use case. A
summary index by default goes through all the data.
Empirically, setting "response_mode="tree_summarize"" also leads to
better summarization results.
   index = SummaryIndex.from_documents(documents)
   query_engine = index.as_query_engine(
       response_mode="tree_summarize"
   )
   response = query_engine.query("<summarization_query>")
Queries over Structured Data
LlamaIndex supports queries over structured data, whether that's a
Pandas DataFrame or a SQL Database.
Here are some relevant resources:
**Tutorials**
* *Guide on Text-to-SQL*
**Guides**
* SQL Guide (Core) (Notebook)
* Pandas Demo (Notebook)
Synthesis over Heterogeneous Data
LlamaIndex supports synthesizing across heterogeneous data sources.
This can be done by composing a graph over your existing data.
Specifically, compose a summary index over your subindices. A summary
index inherently combines information for each node; therefore it can
synthesize information across your heterogeneous data sources.
   from llama_index import VectorStoreIndex, SummaryIndex
   from llama_index.indices.composability import ComposableGraph
   index1 = VectorStoreIndex.from_documents(notion_docs)
   index2 = VectorStoreIndex.from_documents(slack_docs)
   graph = ComposableGraph.from_indices(SummaryIndex, [index1, index2], index_summaries=["summary1", "summary2"])
   query_engine = graph.as_query_engine()
   response = query_engine.query("<query_str>")
**Guides**
* Composability
* City Analysis (Notebook)
Routing over Heterogeneous Data
LlamaIndex also supports routing over heterogeneous data sources with
"RouterQueryEngine" - for instance, if you want to "route" a query to
an underlying Document or a sub-index.
To do this, first build the sub-indices over different data sources.
Then construct the corresponding query engines, and give each query
engine a description to obtain a "QueryEngineTool".
   from llama_index import TreeIndex, VectorStoreIndex
   from llama_index.tools import QueryEngineTool
   ...
   # define sub-indices
   index1 = VectorStoreIndex.from_documents(notion_docs)
   index2 = VectorStoreIndex.from_documents(slack_docs)
   # define query engines and tools
   tool1 = QueryEngineTool.from_defaults(
       query_engine=index1.as_query_engine(),

Q&A over Documents

       description="Use this query engine to do...",
   )
   tool2 = QueryEngineTool.from_defaults(
       query_engine=index2.as_query_engine(),
       description="Use this query engine for something else...",
   )
Then, we define a "RouterQueryEngine" over them. By default, this uses
a "LLMSingleSelector" as the router, which uses the LLM to choose the
best sub-index to router the query to, given the descriptions.
   from llama_index.query_engine import RouterQueryEngine
   query_engine = RouterQueryEngine.from_defaults(
       query_engine_tools=[tool1, tool2]
   )
   response = query_engine.query(
       "In Notion, give me a summary of the product roadmap."
   )
**Guides**
* Router Query Engine Guide (Notebook)
* City Analysis Unified Query Interface (Notebook)
Compare/Contrast Queries
You can explicitly perform compare/contrast queries with a **query
transformation** module within a ComposableGraph.
   from llama_index.indices.query.query_transform.base import DecomposeQueryTransform
   decompose_transform = DecomposeQueryTransform(
       service_context.llm_predictor, verbose=True
   )
This module will help break down a complex query into a simpler one
over your existing index structure.
**Guides**
* Query Transformations
* City Analysis Compare/Contrast Example (Notebook)
You can also rely on the LLM to *infer* whether to perform
compare/contrast queries (see Multi-Document Queries below).
Multi-Document Queries
Besides the explicit synthesis/routing flows described above,
LlamaIndex can support more general multi-document queries as well. It
can do this through our "SubQuestionQueryEngine" class. Given a query,
this query engine will generate a "query plan" containing sub-queries
against sub-documents before synthesizing the final answer.
To do this, first define an index for each document/data source, and
wrap it with a "QueryEngineTool" (similar to above):
   from llama_index.tools import QueryEngineTool, ToolMetadata
   query_engine_tools = [
       QueryEngineTool(
           query_engine=sept_engine,
           metadata=ToolMetadata(name='sept_22', description='Provides information about Uber quarterly financials ending September 2022')
       ),
       QueryEngineTool(
           query_engine=june_engine,
           metadata=ToolMetadata(name='june_22', description='Provides information about Uber quarterly financials ending June 2022')
       ),
       QueryEngineTool(
           query_engine=march_engine,
           metadata=ToolMetadata(name='march_22', description='Provides information about Uber quarterly financials ending March 2022')
       ),
   ]
Then, we define a "SubQuestionQueryEngine" over these tools:
   from llama_index.query_engine import SubQuestionQueryEngine
   query_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=query_engine_tools)
This query engine can execute any number of sub-queries against any
subset of query engine tools before synthesizing the final answer.
This makes it especially well-suited for compare/contrast queries
across documents as well as queries pertaining to a specific document.
**Guides**
* Sub Question Query Engine (Intro)
* 10Q Analysis (Uber)
* 10K Analysis (Uber and Lyft)
Multi-Step Queries
LlamaIndex can also support iterative multi-step queries. Given a
complex query, break it down into an initial subquestions, and
sequentially generate subquestions based on returned answers until the
final answer is returned.
For instance, given a question "Who was in the first batch of the
accelerator program the author started?", the module will first
decompose the query into a simpler initial question "What was the
accelerator program the author started?", query the index, and then
ask followup questions.
**Guides**

Q&A over Documents

* Query Transformations
* Multi-Step Query Decomposition (Notebook)
Temporal Queries
LlamaIndex can support queries that require an understanding of time.
It can do this in two ways:
* Decide whether the query requires utilizing temporal relationships
  between nodes (prev/next relationships) in order to retrieve
  additional context to answer the question.
* Sort by recency and filter outdated context.
**Guides**
* Second-Stage Postprocessing Guide
* Prev/Next Postprocessing
* Recency Postprocessing
Additional Resources
* A Guide to Creating a Unified Query Framework over your ndexes
* A Guide to Extracting Terms and Definitions
* SEC 10k Analysis


Basic Usage Pattern

The general usage pattern of LlamaIndex is as follows:
1. Load in documents (either manually, or through a data loader)
2. Parse the Documents into Nodes
3. Construct Index (from Nodes or Documents)
4. [Optional, Advanced] Building indices on top of other indices
5. Query the index
6. Parsing the response
1. Load in Documents
The first step is to load in data. This data is represented in the
form of "Document" objects. We provide a variety of data loaders which
will load in Documents through the "load_data" function, e.g.:
   from llama_index import SimpleDirectoryReader
   documents = SimpleDirectoryReader('./data').load_data()
You can also choose to construct documents manually. LlamaIndex
exposes the "Document" struct.
   from llama_index import Document
   text_list = [text1, text2, ...]
   documents = [Document(text=t) for t in text_list]
A Document represents a lightweight container around the data source.
You can now choose to proceed with one of the following steps:
1. Feed the Document object directly into the index (see section 3).
2. First convert the Document into Node objects (see section 2).
2. Parse the Documents into Nodes
The next step is to parse these Document objects into Node objects.
Nodes represent "chunks" of source Documents, whether that is a text
chunk, an image, or more. They also contain metadata and relationship
information with other nodes and index structures.
Nodes are a first-class citizen in LlamaIndex. You can choose to
define Nodes and all its attributes directly. You may also choose to
"parse" source Documents into Nodes through our "NodeParser" classes.
For instance, you can do
   from llama_index.node_parser import SimpleNodeParser
   parser = SimpleNodeParser.from_defaults()
   nodes = parser.get_nodes_from_documents(documents)
You can also choose to construct Node objects manually and skip the
first section. For instance,
   from llama_index.schema import TextNode, NodeRelationship, RelatedNodeInfo
   node1 = TextNode(text="<text_chunk>", id_="<node_id>")
   node2 = TextNode(text="<text_chunk>", id_="<node_id>")
   # set relationships
   node1.relationships[NodeRelationship.NEXT] = RelatedNodeInfo(node_id=node2.node_id)
   node2.relationships[NodeRelationship.PREVIOUS] = RelatedNodeInfo(node_id=node1.node_id)
   nodes = [node1, node2]
The "RelatedNodeInfo" class can also store additional "metadata" if
needed:
   node2.relationships[NodeRelationship.PARENT] = RelatedNodeInfo(node_id=node1.node_id, metadata={"key": "val"})
3. Index Construction
We can now build an index over these Document objects. The simplest
high-level abstraction is to load-in the Document objects during index
initialization (this is relevant if you came directly from step 1 and
skipped step 2).
"from_documents" also takes an optional argument "show_progress". Set
it to "True" to display a progress bar during index construction.
   from llama_index import VectorStoreIndex
   index = VectorStoreIndex.from_documents(documents)
You can also choose to build an index over a set of Node objects
directly (this is a continuation of step 2).
   from llama_index import VectorStoreIndex
   index = VectorStoreIndex(nodes)
Depending on which index you use, LlamaIndex may make LLM calls in
order to build the index.
Reusing Nodes across Index Structures
If you have multiple Node objects defined, and wish to share these
Node objects across multiple index structures, you can do that. Simply
instantiate a StorageContext object, add the Node objects to the
underlying DocumentStore, and pass the StorageContext around.

Basic Usage Pattern

   from llama_index import StorageContext
   storage_context = StorageContext.from_defaults()
   storage_context.docstore.add_documents(nodes)
   index1 = VectorStoreIndex(nodes, storage_context=storage_context)
   index2 = SummaryIndex(nodes, storage_context=storage_context)
**NOTE**: If the "storage_context" argument isn't specified, then it
is implicitly created for each index during index construction. You
can access the docstore associated with a given index through
"index.storage_context".
Inserting Documents or Nodes
You can also take advantage of the "insert" capability of indices to
insert Document objects one at a time instead of during index
construction.
   from llama_index import VectorStoreIndex
   index = VectorStoreIndex([])
   for doc in documents:
       index.insert(doc)
If you want to insert nodes on directly you can use "insert_nodes"
function instead.
   from llama_index import VectorStoreIndex
   # nodes: Sequence[Node]
   index = VectorStoreIndex([])
   index.insert_nodes(nodes)
See the Document Management How-To for more details on managing
documents and an example notebook.
Customizing Documents
When creating documents, you can also attach useful metadata. Any
metadata added to a document will be copied to the nodes that get
created from their respective source document.
   document = Document(
       text='text',
       metadata={
           'filename': '<doc_file_name>',
           'category': '<category>'
       }
   )
More information and approaches to this are discussed in the section
Customizing Documents.
Customizing LLM's
By default, we use OpenAI's "text-davinci-003" model. You may choose
to use another LLM when constructing an index.
   from llama_index import VectorStoreIndex, ServiceContext, set_global_service_context
   from llama_index.llms import OpenAI
   ...
   # define LLM
   llm = OpenAI(model="gpt-4", temperature=0, max_tokens=256)
   # configure service context
   service_context = ServiceContext.from_defaults(llm=llm)
   set_global_service_context(service_context)
   # build index
   index = VectorStoreIndex.from_documents(
       documents
   )
To save costs, you may want to use a local model.
   from llama_index import ServiceContext
   service_context = ServiceContext.from_defaults(llm="local")
This will use llama2-chat-13B from with LlamaCPP, and assumes you have
"llama-cpp-python" installed. Full LlamaCPP usage guide is available
in a notebook here.
See the Custom LLM's How-To for more details.
Global ServiceContext
If you wanted the service context from the last section to always be
the default, you can configure one like so:
   from llama_index import set_global_service_context
   set_global_service_context(service_context)
This service context will always be used as the default if not
specified as a keyword argument in LlamaIndex functions.
For more details on the service context, including how to create a
global service context, see the page Customizing the ServiceContext.
Customizing Prompts
Depending on the index used, we used default prompt templates for
constructing the index (and also insertion/querying). See Custom
Prompts How-To for more details on how to customize your prompt.
Customizing embeddings
For embedding-based indices, you can choose to pass in a custom
embedding model. See Custom Embeddings How-To for more details.
Cost Analysis
Creating an index, inserting to an index, and querying an index may
use tokens. We can track token usage through the outputs of these
operations. When running operations, the token usage will be printed.
You can also fetch the token usage through "TokenCountingCallback"
handler. See Cost Analysis How-To for more details.
[Optional] Save the index for future use

Basic Usage Pattern

By default, data is stored in-memory. To persist to disk:
   index.storage_context.persist(persist_dir="<persist_dir>")
You may omit persist_dir to persist to "./storage" by default.
To reload from disk:
   from llama_index import StorageContext, load_index_from_storage
   # rebuild storage context
   storage_context = StorageContext.from_defaults(persist_dir="<persist_dir>")
   # load index
   index = load_index_from_storage(storage_context)
**NOTE**: If you had initialized the index with a custom
"ServiceContext" object, you will also need to pass in the same
ServiceContext during "load_index_from_storage" or ensure you have a
global service context.
   service_context = ServiceContext.from_defaults(llm=llm)
   set_global_service_context(service_context)
   # when first building the index
   index = VectorStoreIndex.from_documents(
       documents, # service_context=service_context -> optional if not using global
   )
   ...
   # when loading the index from disk
   index = load_index_from_storage(
       StorageContext.from_defaults(persist_dir="<persist_dir>")
       # service_context=service_context -> optional if not using global
   )
4. [Optional, Advanced] Building indices on top of other indices
You can build indices on top of other indices! Composability gives you
greater power in indexing your heterogeneous sources of data. For a
discussion on relevant use cases, see our Query Use Cases. For
technical details and examples, see our Composability How-To.
5. Query the index.
After building the index, you can now query it with a "QueryEngine".
Note that a "query" is simply an input to an LLM - this means that you
can use the index for question-answering, but you can also do more
than that!
High-level API
To start, you can query an index with the default "QueryEngine" (i.e.,
using default configs), as follows:
   query_engine = index.as_query_engine()
   response = query_engine.query("What did the author do growing up?")
   print(response)
   response = query_engine.query("Write an email to the user given their background information.")
   print(response)
Low-level API
We also support a low-level composition API that gives you more
granular control over the query logic. Below we highlight a few of the
possible customizations.
   from llama_index import (
       VectorStoreIndex,
       get_response_synthesizer,
   )
   from llama_index.retrievers import VectorIndexRetriever
   from llama_index.query_engine import RetrieverQueryEngine
   from llama_index.indices.postprocessor import SimilarityPostprocessor
   # build index
   index = VectorStoreIndex.from_documents(documents)
   # configure retriever
   retriever = VectorIndexRetriever(
       index=index,
       similarity_top_k=2,
   )
   # configure response synthesizer
   response_synthesizer = get_response_synthesizer()
   # assemble query engine
   query_engine = RetrieverQueryEngine(
       retriever=retriever,
       response_synthesizer=response_synthesizer,
       node_postprocessors=[
           SimilarityPostprocessor(similarity_cutoff=0.7)
       ]
   )
   # query
   response = query_engine.query("What did the author do growing up?")
   print(response)
You may also add your own retrieval, response synthesis, and overall
query logic, by implementing the corresponding interfaces.
For a full list of implemented components and the supported
configurations, please see the detailed reference docs.
In the following, we discuss some commonly used configurations in
detail.
Configuring retriever
An index can have a variety of index-specific retrieval modes. For
instance, a summary index supports the default "SummaryIndexRetriever"
that retrieves all nodes, and "SummaryIndexEmbeddingRetriever" that

Basic Usage Pattern

retrieves the top-k nodes by embedding similarity.
For convenience, you can also use the following shorthand:
       # SummaryIndexRetriever
       retriever = index.as_retriever(retriever_mode='default')
       # SummaryIndexEmbeddingRetriever
       retriever = index.as_retriever(retriever_mode='embedding')
After choosing your desired retriever, you can construct your query
engine:
   query_engine = RetrieverQueryEngine(retriever)
   response = query_engine.query("What did the author do growing up?")
The full list of retrievers for each index (and their shorthand) is
documented in the Query Reference.
Configuring response synthesis
After a retriever fetches relevant nodes, a "BaseSynthesizer"
synthesizes the final response by combining the information.
You can configure it via
   query_engine = RetrieverQueryEngine.from_args(retriever, response_mode=<response_mode>)
Right now, we support the following options:
* "default": "create and refine" an answer by sequentially going
  through each retrieved "Node"; This makes a separate LLM call per
  Node. Good for more detailed answers.
* "compact": "compact" the prompt during each LLM call by stuffing as
  many "Node" text chunks that can fit within the maximum prompt size.
  If there are too many chunks to stuff in one prompt, "create and
  refine" an answer by going through multiple prompts.
* "tree_summarize": Given a set of "Node" objects and the query,
  recursively construct a tree and return the root node as the
  response. Good for summarization purposes.
* "no_text": Only runs the retriever to fetch the nodes that would
  have been sent to the LLM, without actually sending them. Then can
  be inspected by checking "response.source_nodes". The response
  object is covered in more detail in Section 5.
* "accumulate": Given a set of "Node" objects and the query, apply the
  query to each "Node" text chunk while accumulating the responses
  into an array. Returns a concatenated string of all responses. Good
  for when you need to run the same query separately against each text
  chunk.
   index = SummaryIndex.from_documents(documents)
   retriever = index.as_retriever()
   # default
   query_engine = RetrieverQueryEngine.from_args(retriever, response_mode='default')
   response = query_engine.query("What did the author do growing up?")
   # compact
   query_engine = RetrieverQueryEngine.from_args(retriever, response_mode='compact')
   response = query_engine.query("What did the author do growing up?")
   # tree summarize
   query_engine = RetrieverQueryEngine.from_args(retriever, response_mode='tree_summarize')
   response = query_engine.query("What did the author do growing up?")
   # no text
   query_engine = RetrieverQueryEngine.from_args(retriever, response_mode='no_text')
   response = query_engine.query("What did the author do growing up?")
Configuring node postprocessors (i.e. filtering and augmentation)
We also support advanced "Node" filtering and augmentation that can
further improve the relevancy of the retrieved "Node" objects. This
can help reduce the time/number of LLM calls/cost or improve response
quality.
For example:
* "KeywordNodePostprocessor": filters nodes by "required_keywords" and
  "exclude_keywords".
* "SimilarityPostprocessor": filters nodes by setting a threshold on
  the similarity score (thus only supported by embedding-based
  retrievers)
* "PrevNextNodePostprocessor": augments retrieved "Node" objects with
  additional relevant context based on "Node" relationships.
The full list of node postprocessors is documented in the Node

Basic Usage Pattern

Postprocessor Reference.
To configure the desired node postprocessors:
   node_postprocessors = [
       KeywordNodePostprocessor(
           required_keywords=["Combinator"],
           exclude_keywords=["Italy"]
       )
   ]
   query_engine = RetrieverQueryEngine.from_args(
       retriever, node_postprocessors=node_postprocessors
   )
   response = query_engine.query("What did the author do growing up?")
6. Parsing the response
The object returned is a "Response" object. The object contains both
the response text as well as the "sources" of the response:
   response = query_engine.query("<query_str>")
   # get response
   # response.response
   str(response)
   # get sources
   response.source_nodes
   # formatted sources
   response.get_formatted_sources()
An example is shown below. [image: ][image]


Finetuning

Overview
Finetuning a model means updating the model itself over a set of data
to improve the model in a variety of ways. This can include improving
the quality of outputs, reducing hallucinations, memorizing more data
holistically, and reducing latency/cost.
The core of our toolkit revolves around in-context learning /
retrieval augmentation, which involves using the models in inference
mode and not training the models themselves.
While finetuning can be also used to "augment" a model with external
data, finetuning can complement retrieval augmentation in a variety of
ways:
Embedding Finetuning Benefits
* Finetuning the embedding model can allow for more meaningful
  embedding representations over a training distribution of data -->
  leads to better retrieval performance.
LLM Finetuning Benefits
* Allow it to learn a style over a given dataset
* Allow it to learn a DSL that might be less represented in the
  training data (e.g. SQL)
* Allow it to correct hallucinations/errors that might be hard to fix
  through prompt engineering
* Allow it to distill a better model (e.g. GPT-4) into a
  simpler/cheaper model (e.g. gpt-3.5, Llama 2)
Integrations with LlamaIndex
This is an evolving guide, and there are currently three key
integrations with LlamaIndex. Please check out the sections below for
more details!
* Finetuning embeddings for better retrieval performance
* Finetuning Llama 2 for better text-to-SQL
* Finetuning gpt-3.5-turbo to distill gpt-4
Finetuning Embeddings
We've created comprehensive guides showing you how to finetune
embeddings in different ways, whether that's the model itself (in this
case, "bge") over an unstructured text corpus, or an adapter over any
black-box embedding. It consists of the following steps:
1. Generating a synthetic question/answer dataset using LlamaIndex
   over any unstructed context.
2. Finetuning the model
3. Evaluating the model.
Finetuning gives you a 5-10% increase in retrieval evaluation metrics.
You can then plug this fine-tuned model into your RAG application with
LlamaIndex.
* Fine-tuning an Adapter
* Embedding Fine-tuning Guide
**Old**
* Embedding Fine-tuning Repo
* Embedding Fine-tuning Blog
Fine-tuning LLMs
Finetuning GPT-3.5 to distill GPT-4
We have multiple guides showing how to use OpenAI's finetuning
endpoints to fine-tune gpt-3.5-turbo to output GPT-4 responses for
RAG/agents.
We use GPT-4 to automatically generate questions from any unstructured
context, and use a GPT-4 query engine pipeline to generate "ground-
truth" answers. Our "OpenAIFineTuningHandler" callback automatically
logs questions/answers to a dataset.
We then launch a finetuning job, and get back a distilled model. We
can evaluate this model with Ragas to benchmark against a naive
GPT-3.5 pipeline.
* GPT-3.5 Fine-tuning Notebook (Colab)
* GPT-3.5 Fine-tuning Notebook (Notebook link)
* Fine-tuning a gpt-3.5 ReAct Agent on Better Chain of Thought
* [WIP] Function Calling Fine-tuning
**Old**
* GPT-3.5 Fine-tuning Notebook (Colab)
* GPT-3.5 Fine-tuning Notebook (in Repo)
Fine-tuning with Retrieval Augmentation
Here we try fine-tuning an LLM with retrieval-augmented inputs, as

Finetuning

referenced from the RA-DIT paper: https://arxiv.org/abs/2310.01352.
The core idea is to allow the LLM to better use the context from a
given retriever or ignore it entirely.
* Fine-tuning with Retrieval Augmentation
[WIP] Finetuning GPT-3.5 to Memorize Knowledge
We have a guide experimenting with showing how to use OpenAI fine-
tuning to memorize a body of text. Still WIP! Not quite as good as RAG
yet.
* Fine-tuning to Memorize Knowledge
Finetuning Llama 2 for Better Text-to-SQL
In this tutorial, we show you how you can finetune Llama 2 on a text-
to-SQL dataset, and then use it for structured analytics against any
SQL database using LlamaIndex abstractions.
The stack includes "sql-create-context" as the training dataset,
OpenLLaMa as the base model, PEFT for finetuning, Modal for cloud
compute, LlamaIndex for inference abstractions.
* Llama 2 Text-to-SQL Fine-tuning (Repo)
* Llama 2 Text-to-SQL Fine-tuning (Notebook)
Finetuning Cross-Encoders for Re-Ranking
By finetuning a cross encoder, we can attempt to improve re-ranking
performance on our own private data.
Re-ranking is key step in advanced retrieval, where retrieved nodes
from many sources are re-ranked using a separate model, so that the
most relevant nodes are first.
In this example, we use the "sentence-transformers" package to help
finetune a crossencoder model, using a dataset that is generated based
on the "QASPER" dataset.
* Cross-Encoder Finetuning


Private Setup

Relevant Resources:
* Using LlamaIndex with Local Models


Principled Development Practices

In order to develop your application, it can help to implement some
principled development practices.
Here we provide some general guidance to help you better anticipate
the challenges and concerns you may encounter as you develop your LLM
application.
* The Development Pathway
We've also accumulated some techniques for creating more performant
RAG applications.
* Building Performant RAG Applications for Production
With that said, we want to establish some general pillars of
principled development for LLM and RAG applications.
* The first pillar is **observability**: setting up initial tools to
  observe, debug your system and evaluate it on ad-hoc examples.
* The next pillar is **evaluation**: being able to evaluate different
  components of your system so that you can experiment and improve it
  in a more systematic fashion.
* The last pillar is **monitoring**: after the application is
  deployed, we want to continuously monitor and test that it is
  performing well in production.
* Observability
* Evaluation
* Monitoring
Contribute Your Insights!
If you have thoughts on sections to add or how to improve, please make
a contribution (link)


One-Click Observability

LlamaIndex provides **one-click observability** 🔭 to allow you to
build principled LLM applications in a production setting.
A key requirement for principled development of LLM applications over
your data (RAG systems, agents) is being able to observe, debug, and
evaluate your system - both as a whole and for each component.
This feature allows you to seamlessly integrate the LlamaIndex library
with powerful observability/evaluation tools offered by our partners.
Configure a variable once, and you'll be able to do things like the
following:
* View LLM/prompt inputs/outputs
* Ensure that the outputs of any component (LLMs, embeddings) are
  performing as expected
* View call traces for both indexing and querying
Each provider has similarities and differences. Take a look below for
the full set of guides for each one!
Usage Pattern
To toggle, you will generally just need to do the following:
   from llama_index import set_global_handler
   # general usage
   set_global_handler("<handler_name>", **kwargs)
   # W&B example
   # set_global_handler("wandb", run_args={"project": "llamaindex"})
Note that all "kwargs" to "set_global_handler" are passed to the
underlying callback handler.
And that's it! Executions will get seamlessly piped to downstream
service (e.g. W&B Prompts) and you'll be able to access features such
as viewing execution traces of your application.
**NOTE**: TruLens (by TruEra) uses a different "one-click" experience.
See below for details.
Simple (LLM Inputs/Outputs)
This simpe obvservability tool prints every LLM input/output pair to
the terminal. Most useful for when you need to quickly enable debug
logging on your LLM application.
Usage Pattern
   import llama_index
   llama_index.set_global_handler("simple")
Partner "One-Click" Integrations
We offer a rich set of integrations with our partners. A short
description + usage pattern, and guide is provided for each partner.
Weights and Biases Prompts
Prompts allows users to log/trace/inspect the execution flow of
LlamaIndex during index construction and querying. It also allows
users to version-control their indices.
Usage Pattern
~~~~~~~~~~~~~
   from llama_index import set_global_handler
   set_global_handler("wandb", run_args={"project": "llamaindex"})
   # NOTE: No need to do the following
   # from llama_index.callbacks import WandbCallbackHandler, CallbackManager
   # wandb_callback = WandbCallbackHandler(run_args={"project": "llamaindex"})
   # callback_manager = CallbackManager([wandb_callback])
   # service_context = ServiceContext.from_defaults(
   #     callback_manager=callback_manager
   # )
   # access additional methods on handler to persist index + load index
   import llama_index
   # persist index
   llama_index.global_handler.persist_index(graph, index_name="composable_graph")
   # load storage context
   storage_context = llama_index.global_handler.load_storage_context(
       artifact_url="ayut/llamaindex/composable_graph:v0"
   )
[image: ][image]
Guides
~~~~~~
* Wandb Callback Handler
Arize Phoenix
Arize Phoenix: LLMOps insights at lightning speed with zero-config
observability. Phoenix provides a notebook-first experience for
monitoring your models and LLM Applications by providing:
* LLM Traces - Trace through the execution of your LLM Application to
  understand the internals of your LLM Application and to troubleshoot
  problems related to things like retrieval and tool execution.
* LLM Evals - Leverage the power of large language models to evaluate
  your generative model or application's relevance, toxicity, and

One-Click Observability

  more.
Usage Pattern
~~~~~~~~~~~~~
   # Phoenix can display in real time the traces automatically
   # collected from your LlamaIndex application.
   import phoenix as px
   # Look for a URL in the output to open the App in a browser.
   px.launch_app()
   # The App is initially empty, but as you proceed with the steps below,
   # traces will appear automatically as your LlamaIndex application runs.
   import llama_index
   llama_index.set_global_handler("arize_phoenix")
   # Run all of your LlamaIndex applications as usual and traces
   # will be collected and displayed in Phoenix.
   ...
[image: ][image]
Guides
~~~~~~
* Arize Phoenix Tracing Tutorial
OpenInference
OpenInference is an open standard for capturing and storing AI model
inferences. It enables experimentation, visualization, and evaluation
of LLM applications using LLM observability solutions such as Phoenix.
Usage Pattern
~~~~~~~~~~~~~
   import llama_index
   llama_index.set_global_handler("openinference")
   # NOTE: No need to do the following
   # from llama_index.callbacks import OpenInferenceCallbackHandler, CallbackManager
   # callback_handler = OpenInferenceCallbackHandler()
   # callback_manager = CallbackManager([callback_handler])
   # service_context = ServiceContext.from_defaults(
   #     callback_manager=callback_manager
   # )
   # Run your LlamaIndex application here...
   for query in queries:
       query_engine.query(query)
   # View your LLM app data as a dataframe in OpenInference format.
   from llama_index.callbacks.open_inference_callback import as_dataframe
   query_data_buffer = llama_index.global_handler.flush_query_data_buffer()
   query_dataframe = as_dataframe(query_data_buffer)
**NOTE**: To unlock capabilities of Phoenix, you will need to define
additional steps to feed in query/ context dataframes. See below!
Guides
~~~~~~
* OpenInference Callback Handler + Arize Phoenix
* Evaluating Search and Retrieval with Arize Phoenix
TruEra TruLens
TruLens allows users to instrument/evaluate LlamaIndex applications,
through features such as feedback functions and tracing.
Usage Pattern + Guides
~~~~~~~~~~~~~~~~~~~~~~
   # use trulens
   from trulens_eval import TruLlama
   tru_query_engine = TruLlama(query_engine)
   # query
   tru_query_engine.query("What did the author do growing up?")
[image: ][image]
Guides
~~~~~~
* Evaluating and Tracking with TruLens
* Quickstart Guide with LlamaIndex + TruLens
* Colab
HoneyHive
HoneyHive allows users to trace the execution flow of any LLM
pipeline. Users can then debug and analyze their traces, or customize
feedback on specific trace events to create evaluation or fine-tuning
datasets from production.
Usage Pattern
~~~~~~~~~~~~~
   from llama_index import set_global_handler
   set_global_handler(
       "honeyhive",
       project="My HoneyHive Project",
       name="My LLM Pipeline Name",
       api_key="MY HONEYHIVE API KEY",
   )
   # NOTE: No need to do the following
   # from llama_index import ServiceContext
   # from llama_index.callbacks import CallbackManager
   # from honeyhive.sdk.llamaindex_tracer import HoneyHiveLlamaIndexTracer
   # hh_tracer = HoneyHiveLlamaIndexTracer(
   #     project="My HoneyHive Project",
   #     name="My LLM Pipeline Name",
   #     api_key="MY HONEYHIVE API KEY",
   # )
   # callback_manager = CallbackManager([hh_tracer])
   # service_context = ServiceContext.from_defaults(
   #     callback_manager=callback_manager

One-Click Observability

   # )
[image: ][image] [image: ][image] *Use Perfetto to debug and analyze
your HoneyHive traces*
Guides
~~~~~~
* HoneyHive LlamaIndex Tracer


Knowledge Graphs

LlamaIndex contains some fantastic guides for building with knowledge
graphs.
Check out the end-to-end tutorials/workshops below. Also check out our
knowledge graph query engine guides here.
* LlamaIndex Workshop: Building RAG with Knowledge Graphs
* REBEL + Knowledge Graph Index


A Guide to Extracting Terms and Definitions

Llama Index has many use cases (semantic search, summarization, etc.)
that are well documented. However, this doesn't mean we can't apply
Llama Index to very specific use cases!
In this tutorial, we will go through the design process of using Llama
Index to extract terms and definitions from text, while allowing users
to query those terms later. Using Streamlit, we can provide an easy
way to build frontend for running and testing all of this, and quickly
iterate with our design.
This tutorial assumes you have Python3.9+ and the following packages
installed:
* llama-index
* streamlit
At the base level, our objective is to take text from a document,
extract terms and definitions, and then provide a way for users to
query that knowledge base of terms and definitions. The tutorial will
go over features from both Llama Index and Streamlit, and hopefully
provide some interesting solutions for common problems that come up.
The final version of this tutorial can be found here and a live hosted
demo is available on Huggingface Spaces.
Uploading Text
Step one is giving users a way to upload documents. Let’s write some
code using Streamlit to provide the interface for this! Use the
following code and launch the app with "streamlit run app.py".
   import streamlit as st
   st.title("🦙 Llama Index Term Extractor 🦙")
   document_text = st.text_area("Or enter raw text")
   if st.button("Extract Terms and Definitions") and document_text:
       with st.spinner("Extracting..."):
           extracted_terms = document text  # this is a placeholder!
       st.write(extracted_terms)
Super simple right! But you'll notice that the app doesn't do anything
useful yet. To use llama_index, we also need to setup our OpenAI LLM.
There are a bunch of possible settings for the LLM, so we can let the
user figure out what's best. We should also let the user set the
prompt that will extract the terms (which will also help us debug what
works best).
LLM Settings
This next step introduces some tabs to our app, to separate it into
different panes that provide different features. Let's create a tab
for LLM settings and for uploading text:
   import os
   import streamlit as st
   DEFAULT_TERM_STR = (
       "Make a list of terms and definitions that are defined in the context, "
       "with one pair on each line. "
       "If a term is missing it's definition, use your best judgment. "
       "Write each line as as follows:\nTerm: <term> Definition: <definition>"
   )
   st.title("🦙 Llama Index Term Extractor 🦙")
   setup_tab, upload_tab = st.tabs(["Setup", "Upload/Extract Terms"])
   with setup_tab:
       st.subheader("LLM Setup")
       api_key = st.text_input("Enter your OpenAI API key here", type="password")
       llm_name = st.selectbox('Which LLM?', ["text-davinci-003", "gpt-3.5-turbo", "gpt-4"])
       model_temperature = st.slider("LLM Temperature", min_value=0.0, max_value=1.0, step=0.1)
       term_extract_str = st.text_area("The query to extract terms and definitions with.", value=DEFAULT_TERM_STR)
   with upload_tab:
       st.subheader("Extract and Query Definitions")
       document_text = st.text_area("Or enter raw text")
       if st.button("Extract Terms and Definitions") and document_text:
           with st.spinner("Extracting..."):
               extracted_terms = document text  # this is a placeholder!
           st.write(extracted_terms)
Now our app has two tabs, which really helps with the organization.

You'll also noticed I added a default prompt to extract terms -- you
can change this later once you try extracting some terms, it's just
the prompt I arrived at after experimenting a bit.
Speaking of extracting terms, it's time to add some functions to do
just that!
Extracting and Storing Terms
Now that we are able to define LLM settings and upload text, we can
try using Llama Index to extract the terms from text for us!
We can add the following functions to both initialize our LLM, as well
as use it to extract terms from the input text.
   from llama_index import Document, SummaryIndex, LLMPredictor, ServiceContext, load_index_from_storage
   from llama_index.llms import OpenAI
   def get_llm(llm_name, model_temperature, api_key, max_tokens=256):
       os.environ['OPENAI_API_KEY'] = api_key
       return OpenAI(temperature=model_temperature, model=llm_name, max_tokens=max_tokens)
   def extract_terms(documents, term_extract_str, llm_name, model_temperature, api_key):
       llm = get_llm(llm_name, model_temperature, api_key, max_tokens=1024)
       service_context = ServiceContext.from_defaults(llm=llm,
                                                      chunk_size=1024)
       temp_index = SummaryIndex.from_documents(documents, service_context=service_context)
       query_engine = temp_index.as_query_engine(response_mode="tree_summarize")
       terms_definitions = str(query_engine.query(term_extract_str))
       terms_definitions = [x for x in terms_definitions.split("\n") if x and 'Term:' in x and 'Definition:' in x]
       # parse the text into a dict
       terms_to_definition = {x.split("Definition:")[0].split("Term:")[-1].strip(): x.split("Definition:")[-1].strip() for x in terms_definitions}
       return terms_to_definition
Now, using the new functions, we can finally extract our terms!
   ...
   with upload_tab:
       st.subheader("Extract and Query Definitions")
       document_text = st.text_area("Or enter raw text")
       if st.button("Extract Terms and Definitions") and document_text:
           with st.spinner("Extracting..."):
               extracted_terms = extract_terms([Document(text=document_text)],
                                               term_extract_str, llm_name,
                                               model_temperature, api_key)
           st.write(extracted_terms)
There's a lot going on now, let's take a moment to go over what is
happening.
"get_llm()" is instantiating the LLM based on the user configuration
from the setup tab. Based on the model name, we need to use the
appropriate class ("OpenAI" vs. "ChatOpenAI").
"extract_terms()" is where all the good stuff happens. First, we call
"get_llm()" with "max_tokens=1024", since we don't want to limit the
model too much when it is extracting our terms and definitions (the
default is 256 if not set). Then, we define our "ServiceContext"
object, aligning "num_output" with our "max_tokens" value, as well as
setting the chunk size to be no larger than the output. When documents
are indexed by Llama Index, they are broken into chunks (also called
nodes) if they are large, and "chunk_size" sets the size for these
chunks.
Next, we create a temporary summary index and pass in our service
context. A summary index will read every single piece of text in our
index, which is perfect for extracting terms. Finally, we use our pre-
defined query text to extract terms, using
"response_mode="tree_summarize". This response mode will generate a
tree of summaries from the bottom up, where each parent summarizes its

children. Finally, the top of the tree is returned, which will contain
all our extracted terms and definitions.
Lastly, we do some minor post processing. We assume the model followed
instructions and put a term/definition pair on each line. If a line is
missing the "Term:" or "Definition:" labels, we skip it. Then, we
convert this to a dictionary for easy storage!
Saving Extracted Terms
Now that we can extract terms, we need to put them somewhere so that
we can query for them later. A "VectorStoreIndex" should be a perfect
choice for now! But in addition, our app should also keep track of
which terms are inserted into the index so that we can inspect them
later. Using "st.session_state", we can store the current list of
terms in a session dict, unique to each user!
First things first though, let's add a feature to initialize a global
vector index and another function to insert the extracted terms.
   ...
   if 'all_terms' not in st.session_state:
       st.session_state['all_terms'] = DEFAULT_TERMS
   ...
   def insert_terms(terms_to_definition):
       for term, definition in terms_to_definition.items():
           doc = Document(text=f"Term: {term}\nDefinition: {definition}")
           st.session_state['llama_index'].insert(doc)
   @st.cache_resource
   def initialize_index(llm_name, model_temperature, api_key):
       """Create the VectorStoreIndex object."""
       llm = get_llm(llm_name, model_temperature, api_key)
       service_context = ServiceContext.from_defaults(llm=llm)
       index = VectorStoreIndex([], service_context=service_context)
       return index
   ...
   with upload_tab:
       st.subheader("Extract and Query Definitions")
       if st.button("Initialize Index and Reset Terms"):
           st.session_state['llama_index'] = initialize_index(llm_name, model_temperature, api_key)
           st.session_state['all_terms'] = {}
       if "llama_index" in st.session_state:
           st.markdown("Either upload an image/screenshot of a document, or enter the text manually.")
           document_text = st.text_area("Or enter raw text")
           if st.button("Extract Terms and Definitions") and (uploaded_file or document_text):
               st.session_state['terms'] = {}
               terms_docs = {}
               with st.spinner("Extracting..."):
                   terms_docs.update(extract_terms([Document(text=document_text)], term_extract_str, llm_name, model_temperature, api_key))
               st.session_state['terms'].update(terms_docs)
           if "terms" in st.session_state and st.session_state["terms"]::
               st.markdown("Extracted terms")
               st.json(st.session_state['terms'])
               if st.button("Insert terms?"):
                   with st.spinner("Inserting terms"):
                       insert_terms(st.session_state['terms'])
                   st.session_state['all_terms'].update(st.session_state['terms'])
                   st.session_state['terms'] = {}
                   st.experimental_rerun()
Now you are really starting to leverage the power of streamlit! Let's
start with the code under the upload tab. We added a button to
initialize the vector index, and we store it in the global streamlit
state dictionary, as well as resetting the currently extracted terms.
Then, after extracting terms from the input text, we store it the
extracted terms in the global state again and give the user a chance
to review them before inserting. If the insert button is pressed, then
we call our insert terms function, update our global tracking of
inserted terms, and remove the most recently extracted terms from the
session state.
Querying for Extracted Terms/Definitions
With the terms and definitions extracted and saved, how can we use
them? And how will the user even remember what's previously been

saved?? We can simply add some more tabs to the app to handle these
features.
   ...
   setup_tab, terms_tab, upload_tab, query_tab = st.tabs(
       ["Setup", "All Terms", "Upload/Extract Terms", "Query Terms"]
   )
   ...
   with terms_tab:
       with terms_tab:
       st.subheader("Current Extracted Terms and Definitions")
       st.json(st.session_state["all_terms"])
   ...
   with query_tab:
       st.subheader("Query for Terms/Definitions!")
       st.markdown(
           (
               "The LLM will attempt to answer your query, and augment it's answers using the terms/definitions you've inserted. "
               "If a term is not in the index, it will answer using it's internal knowledge."
           )
       )
       if st.button("Initialize Index and Reset Terms", key="init_index_2"):
           st.session_state["llama_index"] = initialize_index(
               llm_name, model_temperature, api_key
           )
           st.session_state["all_terms"] = {}
       if "llama_index" in st.session_state:
           query_text = st.text_input("Ask about a term or definition:")
           if query_text:
               query_text = query_text + "\nIf you can't find the answer, answer the query with the best of your knowledge."
               with st.spinner("Generating answer..."):
                   response = st.session_state["llama_index"].query(
                       query_text, similarity_top_k=5, response_mode="compact"
                   )
               st.markdown(str(response))
While this is mostly basic, some important things to note:
* Our initialize button has the same text as our other button.
  Streamlit will complain about this, so we provide a unique key
  instead.
* Some additional text has been added to the query! This is to try and
  compensate for times when the index does not have the answer.
* In our index query, we've specified two options:
  * "similarity_top_k=5" means the index will fetch the top 5 closest
    matching terms/definitions to the query.
  * "response_mode="compact"" means as much text as possible from the
    5 matching terms/definitions will be used in each LLM call.
    Without this, the index would make at least 5 calls to the LLM,
    which can slow things down for the user.
Dry Run Test
Well, actually I hope you've been testing as we went. But now, let's
try one complete test.
1. Refresh the app
2. Enter your LLM settings
3. Head over to the query tab
4. Ask the following: "What is a bunnyhug?"
5. The app should give some nonsense response. If you didn't know, a
   bunnyhug is another word for a hoodie, used by people from the
   Canadian Prairies!
6. Let's add this definition to the app. Open the upload tab and enter
   the following text: "A bunnyhug is a common term used to describe a
   hoodie. This term is used by people from the Canadian Prairies."
7. Click the extract button. After a few moments, the app should
   display the correctly extracted term/definition. Click the insert
   term button to save it!
8. If we open the terms tab, the term and definition we just extracted
   should be displayed
9. Go back to the query tab and try asking what a bunnyhug is. Now,
   the answer should be correct!
Improvement #1 - Create a Starting Index
With our base app working, it might feel like a lot of work to build
up a useful index. What if we gave the user some kind of starting
point to show off the app's query capabilities? We can do just that!

First, let's make a small change to our app so that we save the index
to disk after every upload:
   def insert_terms(terms_to_definition):
       for term, definition in terms_to_definition.items():
           doc = Document(text=f"Term: {term}\nDefinition: {definition}")
           st.session_state['llama_index'].insert(doc)
       # TEMPORARY - save to disk
       st.session_state['llama_index'].storage_context.persist()
Now, we need some document to extract from! The repository for this
project used the wikipedia page on New York City, and you can find the
text here.
If you paste the text into the upload tab and run it (it may take some
time), we can insert the extracted terms. Make sure to also copy the
text for the extracted terms into a notepad or similar before
inserting into the index! We will need them in a second.
After inserting, remove the line of code we used to save the index to
disk. With a starting index now saved, we can modify our
"initialize_index" function to look like this:
   @st.cache_resource
   def initialize_index(llm_name, model_temperature, api_key):
       """Load the Index object."""
       llm = get_llm(llm_name, model_temperature, api_key)
       service_context = ServiceContext.from_defaults(llm=llm)
       index = load_index_from_storage(service_context=service_context)
       return index
Did you remember to save that giant list of extracted terms in a
notepad? Now when our app initializes, we want to pass in the default
terms that are in the index to our global terms state:
   ...
   if "all_terms" not in st.session_state:
       st.session_state["all_terms"] = DEFAULT_TERMS
   ...
Repeat the above anywhere where we were previously resetting the
"all_terms" values.
Improvement #2 - (Refining) Better Prompts
If you play around with the app a bit now, you might notice that it
stopped following our prompt! Remember, we added to our "query_str"
variable that if the term/definition could not be found, answer to the
best of its knowledge. But now if you try asking about random terms
(like bunnyhug!), it may or may not follow those instructions.
This is due to the concept of "refining" answers in Llama Index. Since
we are querying across the top 5 matching results, sometimes all the
results do not fit in a single prompt! OpenAI models typically have a
max input size of 4097 tokens. So, Llama Index accounts for this by
breaking up the matching results into chunks that will fit into the
prompt. After Llama Index gets an initial answer from the first API
call, it sends the next chunk to the API, along with the previous
answer, and asks the model to refine that answer.
So, the refine process seems to be messing with our results! Rather
than appending extra instructions to the "query_str", remove that, and
Llama Index will let us provide our own custom prompts! Let's create
those now, using the default prompts and chat specific prompts as a
guide. Using a new file "constants.py", let's create some new query
templates:
   from llama_index.prompts import PromptTemplate, SelectorPromptTemplate, ChatPromptTemplate
   from llama_index.prompts.utils import is_chat_model
   from llama_index.llms.base import ChatMessage, MessageRole
   # Text QA templates
   DEFAULT_TEXT_QA_PROMPT_TMPL = (
       "Context information is below. \n"
       "---------------------\n"
       "{context_str}"
       "\n---------------------\n"
       "Given the context information answer the following question "
       "(if you don't know the answer, use the best of your knowledge): {query_str}\n"

   )
   TEXT_QA_TEMPLATE = PromptTemplate(DEFAULT_TEXT_QA_PROMPT_TMPL)
   # Refine templates
   DEFAULT_REFINE_PROMPT_TMPL = (
       "The original question is as follows: {query_str}\n"
       "We have provided an existing answer: {existing_answer}\n"
       "We have the opportunity to refine the existing answer "
       "(only if needed) with some more context below.\n"
       "------------\n"
       "{context_msg}\n"
       "------------\n"
       "Given the new context and using the best of your knowledge, improve the existing answer. "
       "If you can't improve the existing answer, just repeat it again."
   )
   DEFAULT_REFINE_PROMPT = PromptTemplate(DEFAULT_REFINE_PROMPT_TMPL)
   CHAT_REFINE_PROMPT_TMPL_MSGS = [
       ChatMessage(content="{query_str}", role=MessageRole.USER),
       ChatMessage(content="{existing_answer}", role=MessageRole.ASSISTANT),
       ChatMessage(
           content="We have the opportunity to refine the above answer "
           "(only if needed) with some more context below.\n"
           "------------\n"
           "{context_msg}\n"
           "------------\n"
           "Given the new context and using the best of your knowledge, improve the existing answer. "
           "If you can't improve the existing answer, just repeat it again.",
           role=MessageRole.USER,
       ),
   ]
   CHAT_REFINE_PROMPT = ChatPromptTemplate(CHAT_REFINE_PROMPT_TMPL_MSGS)
   # refine prompt selector
   REFINE_TEMPLATE = SelectorPromptTemplate(
       default_template=DEFAULT_REFINE_PROMPT,
       conditionals=[(is_chat_model, CHAT_REFINE_PROMPT)],
   )
That seems like a lot of code, but it's not too bad! If you looked at
the default prompts, you might have noticed that there are default
prompts, and prompts specific to chat models. Continuing that trend,
we do the same for our custom prompts. Then, using a prompt selector,
we can combine both prompts into a single object. If the LLM being
used is a chat model (ChatGPT, GPT-4), then the chat prompts are used.
Otherwise, use the normal prompt templates.
Another thing to note is that we only defined one QA template. In a
chat model, this will be converted to a single "human" message.
So, now we can import these prompts into our app and use them during
the query.
   from constants import REFINE_TEMPLATE, TEXT_QA_TEMPLATE
   ...
       if "llama_index" in st.session_state:
           query_text = st.text_input("Ask about a term or definition:")
           if query_text:
               query_text = query_text  # Notice we removed the old instructions
               with st.spinner("Generating answer..."):
                   response = st.session_state["llama_index"].query(
                       query_text, similarity_top_k=5, response_mode="compact",
                       text_qa_template=TEXT_QA_TEMPLATE, refine_template=REFINE_TEMPLATE
                   )
               st.markdown(str(response))
   ...
If you experiment a bit more with queries, hopefully you notice that
the responses follow our instructions a little better now!
Improvement #3 - Image Support
Llama index also supports images! Using Llama Index, we can upload
images of documents (papers, letters, etc.), and Llama Index handles
extracting the text. We can leverage this to also allow users to
upload images of their documents and extract terms and definitions
from them.
If you get an import error about PIL, install it using "pip install
Pillow" first.
   from PIL import Image
   from llama_index.readers.file.base import DEFAULT_FILE_EXTRACTOR, ImageParser
   @st.cache_resource

   def get_file_extractor():
       image_parser = ImageParser(keep_image=True, parse_text=True)
       file_extractor = DEFAULT_FILE_EXTRACTOR
       file_extractor.update(
           {
               ".jpg": image_parser,
               ".png": image_parser,
               ".jpeg": image_parser,
           }
       )
       return file_extractor
   file_extractor = get_file_extractor()
   ...
   with upload_tab:
       st.subheader("Extract and Query Definitions")
       if st.button("Initialize Index and Reset Terms", key="init_index_1"):
           st.session_state["llama_index"] = initialize_index(
               llm_name, model_temperature, api_key
           )
           st.session_state["all_terms"] = DEFAULT_TERMS
       if "llama_index" in st.session_state:
           st.markdown(
               "Either upload an image/screenshot of a document, or enter the text manually."
           )
           uploaded_file = st.file_uploader(
               "Upload an image/screenshot of a document:", type=["png", "jpg", "jpeg"]
           )
           document_text = st.text_area("Or enter raw text")
           if st.button("Extract Terms and Definitions") and (
               uploaded_file or document_text
           ):
               st.session_state["terms"] = {}
               terms_docs = {}
               with st.spinner("Extracting (images may be slow)..."):
                   if document_text:
                       terms_docs.update(
                           extract_terms(
                               [Document(text=document_text)],
                               term_extract_str,
                               llm_name,
                               model_temperature,
                               api_key,
                           )
                       )
                   if uploaded_file:
                       Image.open(uploaded_file).convert("RGB").save("temp.png")
                       img_reader = SimpleDirectoryReader(
                           input_files=["temp.png"], file_extractor=file_extractor
                       )
                       img_docs = img_reader.load_data()
                       os.remove("temp.png")
                       terms_docs.update(
                           extract_terms(
                               img_docs,
                               term_extract_str,
                               llm_name,
                               model_temperature,
                               api_key,
                           )
                       )
               st.session_state["terms"].update(terms_docs)
           if "terms" in st.session_state and st.session_state["terms"]:
               st.markdown("Extracted terms")
               st.json(st.session_state["terms"])
               if st.button("Insert terms?"):
                   with st.spinner("Inserting terms"):
                       insert_terms(st.session_state["terms"])
                   st.session_state["all_terms"].update(st.session_state["terms"])
                   st.session_state["terms"] = {}
                   st.experimental_rerun()
Here, we added the option to upload a file using Streamlit. Then the
image is opened and saved to disk (this seems hacky but it keeps
things simple). Then we pass the image path to the reader, extract the
documents/text, and remove our temp image file.
Now that we have the documents, we can call "extract_terms()" the same
as before.
Conclusion/TLDR
In this tutorial, we covered a ton of information, while solving some
common issues and problems along the way:
* Using different indexes for different use cases (List vs. Vector
  index)
* Storing global state values with Streamlit's "session_state" concept
* Customizing internal prompts with Llama Index
* Reading text from images with Llama Index
The final version of this tutorial can be found here and a live hosted
demo is available on Huggingface Spaces.


A Guide to Creating a Unified Query Framework over your Indexes

LlamaIndex offers a variety of different use cases.
For simple queries, we may want to use a single index data structure,
such as a "VectorStoreIndex" for semantic search, or "SummaryIndex"
for summarization.
For more complex queries, we may want to use a composable graph.
But how do we integrate indexes and graphs into our LLM application?
Different indexes and graphs may be better suited for different types
of queries that you may want to run.
In this guide, we show how you can unify the diverse use cases of
different index/graph structures under a **single** query framework.
Setup
In this example, we will analyze Wikipedia articles of different
cities: Boston, Seattle, San Francisco, and more.
The below code snippet downloads the relevant data into files.
   from pathlib import Path
   import requests
   wiki_titles = ["Toronto", "Seattle", "Chicago", "Boston", "Houston"]
   for title in wiki_titles:
       response = requests.get(
           'https://en.wikipedia.org/w/api.php',
           params={
               'action': 'query',
               'format': 'json',
               'titles': title,
               'prop': 'extracts',
               # 'exintro': True,
               'explaintext': True,
           }
       ).json()
       page = next(iter(response['query']['pages'].values()))
       wiki_text = page['extract']
       data_path = Path('data')
       if not data_path.exists():
           Path.mkdir(data_path)
       with open(data_path / f"{title}.txt", 'w') as fp:
           fp.write(wiki_text)
The next snippet loads all files into Document objects.
   # Load all wiki documents
   city_docs = {}
   for wiki_title in wiki_titles:
       city_docs[wiki_title] = SimpleDirectoryReader(input_files=[f"data/{wiki_title}.txt"]).load_data()
Defining the Set of Indexes
We will now define a set of indexes and graphs over our data. You can
think of each index/graph as a lightweight structure that solves a
distinct use case.
We will first define a vector index over the documents of each city.
   from llama_index import VectorStoreIndex, ServiceContext, StorageContext
   from llama_index.llms import OpenAI
   # set service context
   llm_gpt4 = OpenAI(temperature=0, model="gpt-4")
   service_context = ServiceContext.from_defaults(
       llm=llm_gpt4, chunk_size=1024
   )
   # Build city document index
   vector_indices = {}
   for wiki_title in wiki_titles:
       storage_context = StorageContext.from_defaults()
       # build vector index
       vector_indices[wiki_title] = VectorStoreIndex.from_documents(
           city_docs[wiki_title],
           service_context=service_context,
           storage_context=storage_context,
       )
       # set id for vector index
       vector_indices[wiki_title].index_struct.index_id = wiki_title
       # persist to disk
       storage_context.persist(persist_dir=f'./storage/{wiki_title}')
Querying a vector index lets us easily perform semantic search over a
given city's documents.
   response = vector_indices["Toronto"].as_query_engine().query("What are the sports teams in Toronto?")
   print(str(response))
Example response:
   The sports teams in Toronto are the Toronto Maple Leafs (NHL), Toronto Blue Jays (MLB), Toronto Raptors (NBA), Toronto Argonauts (CFL), Toronto FC (MLS), Toronto Rock (NLL), Toronto Wolfpack (RFL), and Toronto Rush (NARL).
Defining a Graph for Compare/Contrast Queries
We will now define a composed graph in order to run
**compare/contrast** queries (see *use cases doc*). This graph
contains a keyword table composed on top of existing vector indexes.

To do this, we first want to set the "summary text" for each vector
index.
   index_summaries = {}
   for wiki_title in wiki_titles:
       # set summary text for city
       index_summaries[wiki_title] = (
           f"This content contains Wikipedia articles about {wiki_title}. "
           f"Use this index if you need to lookup specific facts about {wiki_title}.\n"
           "Do not use this index if you want to analyze multiple cities."
       )
Next, we compose a keyword table on top of these vector indexes, with
these indexes and summaries, in order to build the graph.
   from llama_index.indices.composability import ComposableGraph
   graph = ComposableGraph.from_indices(
       SimpleKeywordTableIndex,
       [index for _, index in vector_indices.items()],
       [summary for _, summary in index_summaries.items()],
       max_keywords_per_chunk=50
   )
   # get root index
   root_index = graph.get_index(graph.index_struct.root_id, SimpleKeywordTableIndex)
   # set id of root index
   root_index.set_index_id("compare_contrast")
   root_summary = (
       "This index contains Wikipedia articles about multiple cities. "
       "Use this index if you want to compare multiple cities. "
   )
Querying this graph (with a query transform module), allows us to
easily compare/contrast between different cities. An example is shown
below.
   # define decompose_transform
   from llama_index import LLMPredictor
   from llama_index.indices.query.query_transform.base import DecomposeQueryTransform
   decompose_transform = DecomposeQueryTransform(
       LLMPredictor(llm=llm_gpt4), verbose=True
   )
   # define custom query engines
   from llama_index.query_engine.transform_query_engine import TransformQueryEngine
   custom_query_engines = {}
   for index in vector_indices.values():
       query_engine = index.as_query_engine(service_context=service_context)
       query_engine = TransformQueryEngine(
           query_engine,
           query_transform=decompose_transform,
           transform_extra_info={'index_summary': index.index_struct.summary},
       )
       custom_query_engines[index.index_id] = query_engine
   custom_query_engines[graph.root_id] = graph.root_index.as_query_engine(
       retriever_mode='simple',
       response_mode='tree_summarize',
       service_context=service_context,
   )
   # define query engine
   query_engine = graph.as_query_engine(custom_query_engines=custom_query_engines)
   # query the graph
   query_str = (
       "Compare and contrast the arts and culture of Houston and Boston. "
   )
   response_chatgpt = query_engine.query(query_str)
Defining the Unified Query Interface
Now that we've defined the set of indexes/graphs, we want to build an
**outer abstraction** layer that provides a unified query interface to
our data structures. This means that during query-time, we can query
this outer abstraction layer and trust that the right index/graph will
be used for the job.
There are a few ways to do this, both within our framework as well as
outside of it!
* Build a **router query engine** on top of your existing
  indexes/graphs
* Define each index/graph as a Tool within an agent framework (e.g.
  LangChain).
For the purposes of this tutorial, we follow the former approach. If
you want to take a look at how the latter approach works, take a look
at *our example tutorial here*.
Let's take a look at an example of building a router query engine to
automatically "route" any query to the set of indexes/graphs that you
have define under the hood.
First, we define the query engines for the set of indexes/graph that
we want to route our query to. We also give each a description (about

what data it holds and what it's useful for) to help the router choose
between them depending on the specific query.
   from llama_index.tools.query_engine import QueryEngineTool
   query_engine_tools = []
   # add vector index tools
   for wiki_title in wiki_titles:
       index = vector_indices[wiki_title]
       summary = index_summaries[wiki_title]
       query_engine = index.as_query_engine(service_context=service_context)
       vector_tool = QueryEngineTool.from_defaults(query_engine, description=summary)
       query_engine_tools.append(vector_tool)
   # add graph tool
   graph_description = (
       "This tool contains Wikipedia articles about multiple cities. "
       "Use this tool if you want to compare multiple cities. "
   )
   graph_tool = QueryEngineTool.from_defaults(graph_query_engine, description=graph_description)
   query_engine_tools.append(graph_tool)
Now, we can define the routing logic and overall router query engine.
Here, we use the "LLMSingleSelector", which uses LLM to choose a
underlying query engine to route the query to.
   from llama_index.query_engine.router_query_engine import RouterQueryEngine
   from llama_index.selectors.llm_selectors import LLMSingleSelector
   router_query_engine = RouterQueryEngine(
       selector=LLMSingleSelector.from_defaults(service_context=service_context),
       query_engine_tools=query_engine_tools
   )
Querying our Unified Interface
The advantage of a unified query interface is that it can now handle
different types of queries.
It can now handle queries about specific cities (by routing to the
specific city vector index), and also compare/contrast different
cities.
Let's take a look at a few examples!
**Asking a Compare/Contrast Question**
   # ask a compare/contrast question
   response = router_query_engine.query(
       "Compare and contrast the arts and culture of Houston and Boston.",
   )
   print(str(response)
**Asking Questions about specific Cities**
   response = router_query_engine.query("What are the sports teams in Toronto?")
   print(str(response))
This "outer" abstraction is able to handle different queries by
routing to the right underlying abstractions.


Building RAG from Scratch (Lower-Level)

This doc is a hub for showing how you can build RAG and agent-based
apps using only lower-level abstractions (e.g. LLMs, prompts,
embedding models), and without using more "packaged" out of the box
abstractions.
Out of the box abstractions include:
* High-level ingestion code e.g. "VectorStoreIndex.from_documents"
* High-level query and retriever code e.g.
  "VectorStoreIndex.as_retriever()" and
  "VectorStoreIndex.as_query_engine()"
* High-level agent abstractions e.g. "OpenAIAgent"
Instead of using these, the goal here is to educate users on what's
going on under the hood. By showing you the underlying algorithms for
constructing RAG and agent pipelines, you can then be empowered to
create your own custom LLM workflows (while still using LlamaIndex
abstractions at any level of granularity that makes sense).
We show how to build an app from scratch, component by component. For
the sake of focus, each tutorial will show how to build a specific
component from scratch while using out-of-the-box abstractions for
other components. **NOTE**: This is a WIP document, we're in the
process of fleshing this out!
Building Ingestion from Scratch
This tutorial shows how you can define an ingestion pipeline into a
vector store.
* Building Data Ingestion from Scratch
* Pinecone
* OpenAI
Building Vector Retrieval from Scratch
This tutorial shows you how to build a retriever to query an vector
store.
* Building Retrieval from Scratch
Building Ingestion/Retrieval from Scratch (Open-Source/Local Components)
This tutoral shows you how to build an ingestion/retrieval pipeline
using only open-source components.
* Building RAG from Scratch (Open-source only!)
Building a (Very Simple) Vector Store from Scratch
If you want to learn more about how vector stores work, here's a
tutorial showing you how to build a very simple vector store capable
of dense search + metadata filtering.
Obviously not a replacement for production databases.
* Building a (Very Simple) Vector Store from Scratch
Building Response Synthesis from Scratch
This tutorial shows you how to use the LLM to synthesize results given
a set of retrieved context. Deals with context overflows, async calls,
and source citations!
* Building Response Synthesis from Scratch
Building a Router from Scratch
Beyond the standard RAG pipeline, this takes you one step towards
automated decision making with LLMs by showing you how to build a
router module from scratch.
* Building a Router from Scratch
Building Evaluation from Scratch
Learn how to build common LLM-based eval modules (correctness,
faithfulness) using LLMs and prompt modules; this will help you define
your own custom evals!
* Building Evaluation from Scratch


A Guide to LlamaIndex + Structured Data

A lot of modern data systems depend on structured data, such as a
Postgres DB or a Snowflake data warehouse. LlamaIndex provides a lot
of advanced features, powered by LLM's, to both create structured data
from unstructured data, as well as analyze this structured data
through augmented text-to-SQL capabilities.
This guide helps walk through each of these capabilities.
Specifically, we cover the following topics:
* **Setup**: Defining up our example SQL Table.
* **Building our Table Index**: How to go from sql database to a Table
  Schema Index
* **Using natural language SQL queries**: How to query our SQL
  database using natural language.
We will walk through a toy example table which contains
city/population/country information. A notebook for this tutorial is
available here.
Setup
First, we use SQLAlchemy to setup a simple sqlite db:
   from sqlalchemy import create_engine, MetaData, Table, Column, String, Integer, select, column
   engine = create_engine("sqlite:///:memory:")
   metadata_obj = MetaData()
We then create a toy "city_stats" table:
   # create city SQL table
   table_name = "city_stats"
   city_stats_table = Table(
       table_name,
       metadata_obj,
       Column("city_name", String(16), primary_key=True),
       Column("population", Integer),
       Column("country", String(16), nullable=False),
   )
   metadata_obj.create_all(engine)
Now it's time to insert some datapoints!
If you want to look into filling into this table by inferring
structured datapoints from unstructured data, take a look at the below
section. Otherwise, you can choose to directly populate this table:
   from sqlalchemy import insert
   rows = [
       {"city_name": "Toronto", "population": 2731571, "country": "Canada"},
       {"city_name": "Tokyo", "population": 13929286, "country": "Japan"},
       {"city_name": "Berlin", "population": 600000, "country": "Germany"},
   ]
   for row in rows:
       stmt = insert(city_stats_table).values(**row)
       with engine.connect() as connection:
           cursor = connection.execute(stmt)
Finally, we can wrap the SQLAlchemy engine with our SQLDatabase
wrapper; this allows the db to be used within LlamaIndex:
   from llama_index import SQLDatabase
   sql_database = SQLDatabase(engine, include_tables=["city_stats"])
Natural language SQL
Once we have constructed our SQL database, we can use the
NLSQLTableQueryEngine to construct natural language queries that are
synthesized into SQL queries.
Note that we need to specify the tables we want to use with this query
engine. If we don't the query engine will pull all the schema context,
which could overflow the context window of the LLM.
   from llama_index.indices.struct_store import NLSQLTableQueryEngine
   query_engine = NLSQLTableQueryEngine(
       sql_database=sql_database,
       tables=["city_stats"],
   )
   query_str = (
       "Which city has the highest population?"
   )
   response = query_engine.query(query_str)
This query engine should used in any case where you can specify the
tables you want to query over beforehand, or the total size of all the
table schema plus the rest of the prompt fits your context window.
Building our Table Index
If we don't know ahead of time which table we would like to use, and
the total size of the table schema overflows your context window size,
we should store the table schema in an index so that during query time
we can retrieve the right schema.
The way we can do this is using the SQLTableNodeMapping object, which
takes in a SQLDatabase and produces a Node object for each
SQLTableSchema object passed into the ObjectIndex constructor.

A Guide to LlamaIndex + Structured Data

   from llama_index.objects import SQLTableNodeMapping, ObjectIndex, SQLTableSchema
   table_node_mapping = SQLTableNodeMapping(sql_database)
   table_schema_objs = [(SQLTableSchema(table_name="city_stats")), ...] # one SQLTableSchema for each table
   obj_index = ObjectIndex.from_objects(
       table_schema_objs,
       table_node_mapping,
       VectorStoreIndex,
   )
Here you can see we define our table_node_mapping, and a single
SQLTableSchema with the "city_stats" table name. We pass these into
the ObjectIndex constructor, along with the VectorStoreIndex class
definition we want to use. This will give us a VectorStoreIndex where
each Node contains table schema and other context information. You can
also add any additional context information you'd like.
   # manually set extra context text
   city_stats_text = (
       "This table gives information regarding the population and country of a given city.\n"
       "The user will query with codewords, where 'foo' corresponds to population and 'bar'"
       "corresponds to city."
   )
   table_node_mapping = SQLTableNodeMapping(sql_database)
   table_schema_objs = [(SQLTableSchema(table_name="city_stats", context_str=city_stats_text))]
Using natural language SQL queries
Once we have defined our table schema index obj_index, we can
construct a SQLTableRetrieverQueryEngine by passing in our
SQLDatabase, and a retriever constructed from our object index.
   from llama_index.indices.struct_store import SQLTableRetrieverQueryEngine
   query_engine = SQLTableRetrieverQueryEngine(
       sql_database, obj_index.as_retriever(similarity_top_k=1)
   )
   response = query_engine.query("Which city has the highest population?")
   print(response)
Now when we query the retriever query engine, it will retrieve the
relevant table schema and synthesize a SQL query and a response from
the results of that query.
Concluding Thoughts
This is it for now! We're constantly looking for ways to improve our
structured data support. If you have any questions let us know in our
Discord.