Airbyte SQL Index Guide

We will show how to generate SQL queries on a Snowflake db generated
by Airbyte.
   # Uncomment to enable debugging.
   # import logging
   # import sys
   # logging.basicConfig(stream=sys.stdout, level=logging.DEBUG)
   # logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
Airbyte ingestion
Here we show how to ingest data from Github into a Snowflake db using
Airbyte.
   from IPython.display import Image
   Image(filename="img/airbyte_1.png")
   <IPython.core.display.Image object>
Let's create a new connection. Here we will be dumping our Zendesk
tickets into a Snowflake db.
   Image(filename="img/github_1.png")
   <IPython.core.display.Image object>
   Image(filename="img/github_2.png")
   <IPython.core.display.Image object>
   Image(filename="img/snowflake_1.png")
   <IPython.core.display.Image object>
   Image(filename="img/snowflake_2.png")
   <IPython.core.display.Image object>
Choose the streams you want to sync.
   Image(filename="img/airbyte_7.png")
   <IPython.core.display.Image object>
   Image(filename="img/github_3.png")
   <IPython.core.display.Image object>
Sync your data.
   Image(filename="img/airbyte_9.png")
   <IPython.core.display.Image object>
   Image(filename="img/airbyte_8.png")
   <IPython.core.display.Image object>
Snowflake-SQLAlchemy version fix
Hack to make snowflake-sqlalchemy work despite incompatible sqlalchemy
versions
Taken from https://github.com/snowflakedb/snowflake-
sqlalchemy/issues/380#issuecomment-1470762025
   # Hack to make snowflake-sqlalchemy work until they patch it
   def snowflake_sqlalchemy_20_monkey_patches():
       import sqlalchemy.util.compat
       # make strings always return unicode strings
       sqlalchemy.util.compat.string_types = (str,)
       sqlalchemy.types.String.RETURNS_UNICODE = True
       import snowflake.sqlalchemy.snowdialect
       snowflake.sqlalchemy.snowdialect.SnowflakeDialect.returns_unicode_strings = True
       # make has_table() support the `info_cache` kwarg
       import snowflake.sqlalchemy.snowdialect
       def has_table(self, connection, table_name, schema=None, info_cache=None):
           """
           Checks if the table exists
           """
           return self._has_object(connection, "TABLE", table_name, schema)
       snowflake.sqlalchemy.snowdialect.SnowflakeDialect.has_table = has_table
   # usage: call this function before creating an engine:
   try:
       snowflake_sqlalchemy_20_monkey_patches()
   except Exception as e:
       raise ValueError("Please run `pip install snowflake-sqlalchemy`")
Define database
We pass the Snowflake uri to the SQL db constructor
   snowflake_uri = "snowflake://<user_login_name>:<password>@<account_identifier>/<database_name>/<schema_name>?warehouse=<warehouse_name>&role=<role_name>"
First we try connecting with sqlalchemy to check the db works.
   from sqlalchemy import select, create_engine, MetaData, Table
   # view current table
   engine = create_engine(snowflake_uri)
   metadata = MetaData(bind=None)
   table = Table("ZENDESK_TICKETS", metadata, autoload=True, autoload_with=engine)
   stmt = select(table.columns)
   with engine.connect() as connection:
       results = connection.execute(stmt).fetchone()
       print(results)
       print(results.keys())
   /var/folders/dx/n9yhm8p9039b5bgmgjqy46y40000gn/T/ipykernel_57673/3609487787.py:6: RemovedIn20Warning: Deprecated API features detected! These feature(s) are not compatible with SQLAlchemy 2.0. To prevent incompatible upgrades prior to updating applications, ensure requirements files are pinned to "sqlalchemy<2.0". Set environment variable SQLALCHEMY_WARN_20=1 to show all deprecation warnings.  Set environment variable SQLALCHEMY_SILENCE_UBER_WARNING=1 to silence this message. (Background on SQLAlchemy 2.0 at: https://sqlalche.me/e/b8d9)

Airbyte SQL Index Guide

     table = Table(
   (False, 'test case', '[]', datetime.datetime(2022, 7, 18, 16, 59, 13, tzinfo=<UTC>), 'test to', None, None, 'question', '{\n  "channel": "web",\n  "source": {\n    "from": {},\n    "rel": null,\n    "to": {}\n  }\n}', True, datetime.datetime(2022, 7, 18, 18, 1, 37, tzinfo=<UTC>), None, '[]', None, 134, None, 1658167297, 'test case', None, '[]', False, '{\n  "score": "offered"\n}', 360786799676, 'low', '[]', 'https://d3v-airbyte.zendesk.com/api/v2/tickets/134.json', '[]', 360000358316, 360000084116, '[]', None, '[]', 360033549136, True, None, False, 'new', 360786799676, 'abd39a87-b1f9-4390-bf8b-cf3c288b1f74', datetime.datetime(2023, 6, 9, 0, 25, 23, 501000, tzinfo=pytz.FixedOffset(-420)), datetime.datetime(2023, 6, 9, 0, 38, 20, 440000, tzinfo=<UTC>), '6577ef036668746df889983970579a55', '02522a2b2726fb0a03bb19f2d8d9524d')
   RMKeyView(['from_messaging_channel', 'subject', 'email_cc_ids', 'created_at', 'description', 'custom_status_id', 'external_id', 'type', 'via', 'allow_attachments', 'updated_at', 'problem_id', 'follower_ids', 'due_at', 'id', 'assignee_id', 'generated_timestamp', 'raw_subject', 'forum_topic_id', 'custom_fields', 'allow_channelback', 'satisfaction_rating', 'submitter_id', 'priority', 'collaborator_ids', 'url', 'tags', 'brand_id', 'ticket_form_id', 'sharing_agreement_ids', 'group_id', 'followup_ids', 'organization_id', 'is_public', 'recipient', 'has_incidents', 'status', 'requester_id', '_airbyte_ab_id', '_airbyte_emitted_at', '_airbyte_normalized_at', '_airbyte_zendesk_tickets_hashid', '_airbyte_unique_key'])
Define SQL DB
Once we have defined the SQLDatabase, we can wrap it in a query engine
to query it. If we know what tables we want to use we can use
"NLSQLTableQueryEngine". This will generate a SQL query on the
specified tables.
   from llama_index import SQLDatabase
   # You can specify table filters during engine creation.
   # sql_database = SQLDatabase(engine, include_tables=["github_issues","github_comments", "github_users"])
   sql_database = SQLDatabase(engine)
Synthesize Query
We then show a natural language query, which is translated to a SQL
query under the hood with our text-to-SQL prompt.
   from llama_index.indices.struct_store.sql_query import NLSQLTableQueryEngine
   from IPython.display import Markdown, display
   query_engine = NLSQLTableQueryEngine(
       sql_database=sql_database,
       tables=["github_issues", "github_comments", "github_users"],
   )
   query_str = (
       "Which issues have the most comments? Give the top 10 and use a join on url."
   )
   response = query_engine.query(query_str)

Airbyte SQL Index Guide

display(Markdown(f"{response}"))
 The top 10 issues with the most comments, based on a join on url, are
'Proof of concept parallel source stream reading implementation for
MySQL', 'Remove noisy logging for "LegacyStateManager"', 'Track stream
status in source', 'Source Google Analytics v4: - add pk and lookback
window', 'Connector Health: Fixed SAT for marketo, close, chargebee,
facebook marketing, paystack, hubspot, pipedrive and marketo', '📝
Update outdated docs urls in metadata files', 'Fix emitted
intermediate state for initial incremental non-CDC syncs', 'source-
postgres : Add logic to handle xmin wraparound', ':bug: Source
HubSpot: fix cast string as boolean using string comparison', and 'Fix
db-lib JdbcUtils.java to accept JDBC parameters with = sign.'.
   # You can also get only the SQL query result.
   query_engine = NLSQLTableQueryEngine(
       sql_database=sql_database,
       synthesize_response=False,
       tables=["github_issues", "github_comments", "github_users"],
   )
   response = query_engine.query(query_str)
   display(Markdown(f"{response}"))
[('Proof of concept parallel source stream reading implementation for
MySQL', 'https://api.github.com/repos/airbytehq/airbyte/issues/26580',
'https://api.github.com/repos/airbytehq/airbyte/issues/26580', 104),
('Remove noisy logging for "LegacyStateManager"',
'https://api.github.com/repos/airbytehq/airbyte/issues/27335',
'https://api.github.com/repos/airbytehq/airbyte/issues/27335', 39),
('Track stream status in source',
'https://api.github.com/repos/airbytehq/airbyte/issues/24971',
'https://api.github.com/repos/airbytehq/airbyte/issues/24971', 35),
('Source Google Analytics v4: - add pk and lookback window',
'https://api.github.com/repos/airbytehq/airbyte/issues/26283',
'https://api.github.com/repos/airbytehq/airbyte/issues/26283', 29),
('Connector Health: Fixed SAT for marketo, close, chargebee, facebook
marketing, paystack, hubspot, pipedrive and marketo',
'https://api.github.com/repos/airbytehq/airbyte/issues/24802',
'https://api.github.com/repos/airbytehq/airbyte/issues/24802', 28), ('
📝 Update outdated docs urls in metadata files',
'https://api.github.com/repos/airbytehq/airbyte/issues/27420',
'https://api.github.com/repos/airbytehq/airbyte/issues/27420', 26),
('Fix emitted intermediate state for initial incremental non-CDC
syncs', 'https://api.github.com/repos/airbytehq/airbyte/issues/24820',
'https://api.github.com/repos/airbytehq/airbyte/issues/24820', 25),
('source-postgres : Add logic to handle xmin wraparound',
'https://api.github.com/repos/airbytehq/airbyte/issues/27384',
'https://api.github.com/repos/airbytehq/airbyte/issues/27384', 24),
(':bug: Source HubSpot: fix cast string as boolean using string
comparison',
'https://api.github.com/repos/airbytehq/airbyte/issues/26082',
'https://api.github.com/repos/airbytehq/airbyte/issues/26082', 24),
('Fix db-lib JdbcUtils.java to accept JDBC parameters with = sign.',
'https://api.github.com/repos/airbytehq/airbyte/issues/25386',
'https://api.github.com/repos/airbytehq/airbyte/issues/25386', 22)]

Airbyte SQL Index Guide

# You can also get the original SQL query
   sql_query = response.metadata["sql_query"]
   display(Markdown(f"{sql_query}"))
SELECT gi.title, gi.url, gc.issue_url, COUNT(*) AS comment_count FROM
github_issues gi JOIN github_comments gc ON gi.url = gc.issue_url
GROUP BY gi.title, gi.url, gc.issue_url ORDER BY comment_count DESC
LIMIT 10;
We can also use LLM prediction to figure out what tables to use.
We first need to create an ObjectIndex of SQLTableSchema. In this case
we only pass in the table names. The query engine will fetch the
relevant table schema at query time.
   from llama_index.indices.struct_store.sql_query import SQLTableRetrieverQueryEngine
   from llama_index.objects import SQLTableNodeMapping, ObjectIndex, SQLTableSchema
   from llama_index import VectorStoreIndex
   table_node_mapping = SQLTableNodeMapping(sql_database)
   all_table_names = sql_database.get_usable_table_names()
   table_schema_objs = []
   for table_name in all_table_names:
       table_schema_objs.append(SQLTableSchema(table_name=table_name))
   obj_index = ObjectIndex.from_objects(
       table_schema_objs,
       table_node_mapping,
       VectorStoreIndex,
   )
   table_retriever_query_engine = SQLTableRetrieverQueryEngine(
       sql_database, obj_index.as_retriever(similarity_top_k=1)
   )
   response = query_engine.query(query_str)
   display(Markdown(f"{response}"))
   sql_query = response.metadata["sql_query"]
   display(Markdown(f"{sql_query}"))
   /Users/hongyishi/Documents/GitHub/gpt_index/.venv/lib/python3.11/site-packages/langchain/sql_database.py:279: UserWarning: This method is deprecated - please use `get_usable_table_names`.
     warnings.warn(
[('Proof of concept parallel source stream reading implementation for
MySQL', 'https://api.github.com/repos/airbytehq/airbyte/issues/26580',
'https://api.github.com/repos/airbytehq/airbyte/issues/26580', 104),
('Remove noisy logging for "LegacyStateManager"',
'https://api.github.com/repos/airbytehq/airbyte/issues/27335',
'https://api.github.com/repos/airbytehq/airbyte/issues/27335', 39),
('Track stream status in source',
'https://api.github.com/repos/airbytehq/airbyte/issues/24971',
'https://api.github.com/repos/airbytehq/airbyte/issues/24971', 35),
('Source Google Analytics v4: - add pk and lookback window',
'https://api.github.com/repos/airbytehq/airbyte/issues/26283',
'https://api.github.com/repos/airbytehq/airbyte/issues/26283', 29),
('Connector Health: Fixed SAT for marketo, close, chargebee, facebook
marketing, paystack, hubspot, pipedrive and marketo',
'https://api.github.com/repos/airbytehq/airbyte/issues/24802',
'https://api.github.com/repos/airbytehq/airbyte/issues/24802', 28), ('
📝 Update outdated docs urls in metadata files',
'https://api.github.com/repos/airbytehq/airbyte/issues/27420',
'https://api.github.com/repos/airbytehq/airbyte/issues/27420', 26),
('Fix emitted intermediate state for initial incremental non-CDC
syncs', 'https://api.github.com/repos/airbytehq/airbyte/issues/24820',
'https://api.github.com/repos/airbytehq/airbyte/issues/24820', 25),
('source-postgres : Add logic to handle xmin wraparound',
'https://api.github.com/repos/airbytehq/airbyte/issues/27384',

Airbyte SQL Index Guide

'https://api.github.com/repos/airbytehq/airbyte/issues/27384', 24),
(':bug: Source HubSpot: fix cast string as boolean using string
comparison',
'https://api.github.com/repos/airbytehq/airbyte/issues/26082',
'https://api.github.com/repos/airbytehq/airbyte/issues/26082', 24),
('Fix db-lib JdbcUtils.java to accept JDBC parameters with = sign.',
'https://api.github.com/repos/airbytehq/airbyte/issues/25386',
'https://api.github.com/repos/airbytehq/airbyte/issues/25386', 22)]
SELECT gi.title, gi.url, gc.issue_url, COUNT(*) AS comment_count FROM
github_issues gi JOIN github_comments gc ON gi.url = gc.issue_url
GROUP BY gi.title, gi.url, gc.issue_url ORDER BY comment_count DESC
LIMIT 10;


💬🤖 How to Build a Chatbot

LlamaIndex serves as a bridge between your data and Language Learning
Models (LLMs), providing a toolkit that enables you to establish a
query interface around your data for a variety of tasks, such as
question-answering and summarization.
In this tutorial, we'll walk you through building a context-augmented
chatbot using a Data Agent. This agent, powered by LLMs, is capable of
intelligently executing tasks over your data. The end result is a
chatbot agent equipped with a robust set of data interface tools
provided by LlamaIndex to answer queries about your data.
**Note**: This tutorial builds upon initial work on creating a query
interface over SEC 10-K filings - check it out here.
Context
In this guide, we’ll build a "10-K Chatbot" that uses raw UBER 10-K
HTML filings from Dropbox. Users can interact with the chatbot to ask
questions related to the 10-K filings.
Preparation
   import os
   import openai
   os.environ["OPENAI_API_KEY"] = "sk-..."
   openai.api_key = os.environ["OPENAI_API_KEY"]
   import nest_asyncio
   nest_asyncio.apply()
Ingest Data
Let's first download the raw 10-k files, from 2019-2022.
   # NOTE: the code examples assume you're operating within a Jupyter notebook.
   # download files
   !mkdir data
   !wget "https://www.dropbox.com/s/948jr9cfs7fgj99/UBER.zip?dl=1" -O data/UBER.zip
   !unzip data/UBER.zip -d data
To parse the HTML files into formatted text, we use the Unstructured
library. Thanks to LlamaHub, we can directly integrate with
Unstructured, allowing conversion of any text into a Document format
that LlamaIndex can ingest.
First we install the necessary packages:
   !pip install llama-hub unstructured
Then we can use the "UnstructuredReader" to parse the HTML files into
a list of "Document" objects.
   from llama_hub.file.unstructured.base import UnstructuredReader
   from pathlib import Path
   years = [2022, 2021, 2020, 2019]
   loader = UnstructuredReader()
   doc_set = {}
   all_docs = []
   for year in years:
       year_docs = loader.load_data(
           file=Path(f"./data/UBER/UBER_{year}.html"), split_documents=False
       )
       # insert year metadata into each year
       for d in year_docs:
           d.metadata = {"year": year}
       doc_set[year] = year_docs
       all_docs.extend(year_docs)
Setting up Vector Indices for each year
We first setup a vector index for each year. Each vector index allows
us to ask questions about the 10-K filing of a given year.
We build each index and save it to disk.
   # initialize simple vector indices
   from llama_index import VectorStoreIndex, ServiceContext, StorageContext
   index_set = {}
   service_context = ServiceContext.from_defaults(chunk_size=512)
   for year in years:
       storage_context = StorageContext.from_defaults()
       cur_index = VectorStoreIndex.from_documents(
           doc_set[year],
           service_context=service_context,
           storage_context=storage_context,
       )
       index_set[year] = cur_index
       storage_context.persist(persist_dir=f"./storage/{year}")
To load an index from disk, do the following
   # Load indices from disk
   from llama_index import load_index_from_storage
   index_set = {}
   for year in years:
       storage_context = StorageContext.from_defaults(persist_dir=f"./storage/{year}")
       cur_index = load_index_from_storage(

💬🤖 How to Build a Chatbot

           storage_context, service_context=service_context
       )
       index_set[year] = cur_index
Setting up a Sub Question Query Engine to Synthesize Answers Across 10-K Filings
Since we have access to documents of 4 years, we may not only want to
ask questions regarding the 10-K document of a given year, but ask
questions that require analysis over all 10-K filings.
To address this, we can use a Sub Question Query Engine. It decomposes
a query into subqueries, each answered by an individual vector index,
and synthesizes the results to answer the overall query.
LlamaIndex provides some wrappers around indices (and query engines)
so that they can be used by query engines and agents. First we define
a "QueryEngineTool" for each vector index. Each tool has a name and a
description; these are what the LLM agent sees to decide which tool to
choose.
   from llama_index.tools import QueryEngineTool, ToolMetadata
   individual_query_engine_tools = [
       QueryEngineTool(
           query_engine=index_set[year].as_query_engine(),
           metadata=ToolMetadata(
               name=f"vector_index_{year}",
               description=f"useful for when you want to answer queries about the {year} SEC 10-K for Uber",
           ),
       )
       for year in years
   ]
Now we can create the Sub Question Query Engine, which will allow us
to synthesize answers across the 10-K filings. We pass in the
"individual_query_engine_tools" we defined above, as well as a
"service_context" that will be used to run the subqueries.
   from llama_index.query_engine import SubQuestionQueryEngine
   query_engine = SubQuestionQueryEngine.from_defaults(
       query_engine_tools=individual_query_engine_tools,
       service_context=service_context,
   )
Setting up the Chatbot Agent
We use a LlamaIndex Data Agent to setup the outer chatbot agent, which
has access to a set of Tools. Specifically, we will use an
OpenAIAgent, that takes advantage of OpenAI API function calling. We
want to use the separate Tools we defined previously for each index
(corresponding to a given year), as well as a tool for the sub
question query engine we defined above.
First we define a "QueryEngineTool" for the sub question query engine:
   query_engine_tool = QueryEngineTool(
       query_engine=query_engine,
       metadata=ToolMetadata(
           name="sub_question_query_engine",
           description="useful for when you want to answer queries that require analyzing multiple SEC 10-K documents for Uber",
       ),
   )
Then, we combine the Tools we defined above into a single list of
tools for the agent:
   tools = individual_query_engine_tools + [query_engine_tool]
Finally, we call "OpenAIAgent.from_tools" to create the agent, passing
in the list of tools we defined above.
   from llama_index.agent import OpenAIAgent
   agent = OpenAIAgent.from_tools(tools, verbose=True)
Testing the Agent
We can now test the agent with various queries.
If we test it with a simple "hello" query, the agent does not use any
Tools.
   response = agent.chat("hi, i am bob")
   print(str(response))
   Hello Bob! How can I assist you today?
If we test it with a query regarding the 10-k of a given year, the
agent will use the relevant vector index Tool.
   response = agent.chat("What were some of the biggest risk factors in 2020 for Uber?")
   print(str(response))
   === Calling Function ===
   Calling function: vector_index_2020 with args: {
     "input": "biggest risk factors"
   }
   Got output: The biggest risk factors mentioned in the context are:
   1. The adverse impact of the COVID-19 pandemic and actions taken to mitigate it on the business.

💬🤖 How to Build a Chatbot

   2. The potential reclassification of drivers as employees, workers, or quasi-employees instead of independent contractors.
   3. Intense competition in the mobility, delivery, and logistics industries, with low-cost alternatives and well-capitalized competitors.
   4. The need to lower fares or service fees and offer driver incentives and consumer discounts to remain competitive.
   5. Significant losses incurred and the uncertainty of achieving profitability.
   6. The risk of not attracting or maintaining a critical mass of platform users.
   7. Operational, compliance, and cultural challenges related to the workplace culture and forward-leaning approach.
   8. The potential negative impact of international investments and the challenges of conducting business in foreign countries.
   9. Risks associated with operational and compliance challenges, localization, laws and regulations, competition, social acceptance, technological compatibility, improper business practices, liability uncertainty, managing international operations, currency fluctuations, cash transactions, tax consequences, and payment fraud.
   ========================
   Some of the biggest risk factors for Uber in 2020 were:
   1. The adverse impact of the COVID-19 pandemic and actions taken to mitigate it on the business.
   2. The potential reclassification of drivers as employees, workers, or quasi-employees instead of independent contractors.
   3. Intense competition in the mobility, delivery, and logistics industries, with low-cost alternatives and well-capitalized competitors.
   4. The need to lower fares or service fees and offer driver incentives and consumer discounts to remain competitive.
   5. Significant losses incurred and the uncertainty of achieving profitability.
   6. The risk of not attracting or maintaining a critical mass of platform users.
   7. Operational, compliance, and cultural challenges related to the workplace culture and forward-leaning approach.
   8. The potential negative impact of international investments and the challenges of conducting business in foreign countries.
   9. Risks associated with operational and compliance challenges, localization, laws and regulations, competition, social acceptance, technological compatibility, improper business practices, liability uncertainty, managing international operations, currency fluctuations, cash transactions, tax consequences, and payment fraud.
   These risk factors highlight the challenges and uncertainties that Uber faced in 2020.
Finally, if we test it with a query to compare/contrast risk factors
across years, the agent will use the Sub Question Query Engine Tool.
   cross_query_str = "Compare/contrast the risk factors described in the Uber 10-K across years. Give answer in bullet points."
   response = agent.chat(cross_query_str)
   print(str(response))
   === Calling Function ===
   Calling function: sub_question_query_engine with args: {
     "input": "Compare/contrast the risk factors described in the Uber 10-K across years"
   }
   Generated 4 sub questions.
   [vector_index_2022] Q: What are the risk factors described in the 2022 SEC 10-K for Uber?
   [vector_index_2021] Q: What are the risk factors described in the 2021 SEC 10-K for Uber?
   [vector_index_2020] Q: What are the risk factors described in the 2020 SEC 10-K for Uber?
   [vector_index_2019] Q: What are the risk factors described in the 2019 SEC 10-K for Uber?
   [vector_index_2021] A: The risk factors described in the 2021 SEC 10-K for Uber include the adverse impact of the COVID-19 pandemic on their business, the potential reclassification of drivers as employees instead of independent contractors, intense competition in the mobility, delivery, and logistics industries, the need to lower fares and offer incentives to remain competitive, significant losses incurred by the company, the importance of attracting and maintaining a critical mass of platform users, and the ongoing legal challenges regarding driver classification.
   [vector_index_2020] A: The risk factors described in the 2020 SEC 10-K for Uber include the adverse impact of the COVID-19 pandemic on their business, the potential reclassification of drivers as employees instead of independent contractors, intense competition in the mobility, delivery, and logistics industries, the need to lower fares and offer incentives to remain competitive, significant losses and the uncertainty of achieving profitability, the importance of attracting and retaining a critical mass of drivers and users, and the challenges associated with their workplace culture and operational compliance.

💬🤖 How to Build a Chatbot

   [vector_index_2022] A: The risk factors described in the 2022 SEC 10-K for Uber include the potential adverse effect on their business if drivers were classified as employees instead of independent contractors, the highly competitive nature of the mobility, delivery, and logistics industries, the need to lower fares or service fees to remain competitive in certain markets, the company's history of significant losses and the expectation of increased operating expenses in the future, and the potential impact on their platform if they are unable to attract or maintain a critical mass of drivers, consumers, merchants, shippers, and carriers.
   [vector_index_2019] A: The risk factors described in the 2019 SEC 10-K for Uber include the loss of their license to operate in London, the complexity of their business and operating model due to regulatory uncertainties, the potential for additional regulations for their other products in the Other Bets segment, the evolving laws and regulations regarding the development and deployment of autonomous vehicles, and the increasing number of data protection and privacy laws around the world. Additionally, there are legal proceedings, litigation, claims, and government investigations that Uber is involved in, which could impose a burden on management and employees and come with defense costs or unfavorable rulings.
   Got output: The risk factors described in the Uber 10-K reports across the years include the potential reclassification of drivers as employees instead of independent contractors, intense competition in the mobility, delivery, and logistics industries, the need to lower fares and offer incentives to remain competitive, significant losses incurred by the company, the importance of attracting and maintaining a critical mass of platform users, and the ongoing legal challenges regarding driver classification. Additionally, there are specific risk factors mentioned in each year's report, such as the adverse impact of the COVID-19 pandemic in 2020 and 2021, the loss of their license to operate in London in 2019, and the evolving laws and regulations regarding autonomous vehicles in 2019. Overall, while there are some similarities in the risk factors mentioned, there are also specific factors that vary across the years.
   ========================
   === Calling Function ===
   Calling function: vector_index_2022 with args: {
     "input": "risk factors"
   }
   Got output: Some of the risk factors mentioned in the context include the potential adverse effect on the business if drivers were classified as employees instead of independent contractors, the highly competitive nature of the mobility, delivery, and logistics industries, the need to lower fares or service fees to remain competitive, the company's history of significant losses and the expectation of increased operating expenses, the impact of future pandemics or disease outbreaks on the business and financial results, and the potential harm to the business due to economic conditions and their effect on discretionary consumer spending.
   ========================
   === Calling Function ===
   Calling function: vector_index_2021 with args: {
     "input": "risk factors"
   }
   Got output: The COVID-19 pandemic and the impact of actions to mitigate the pandemic have adversely affected and may continue to adversely affect parts of our business. Our business would be adversely affected if Drivers were classified as employees, workers or quasi-employees instead of independent contractors. The mobility, delivery, and logistics industries are highly competitive, with well-established and low-cost alternatives that have been available for decades, low barriers to entry, low switching costs, and well-capitalized competitors in nearly every major geographic region. To remain competitive in certain markets, we have in the past lowered, and may continue to lower, fares or service fees, and we have in the past offered, and may continue to offer, significant Driver incentives and consumer discounts and promotions. We have incurred significant losses since inception, including in the United States and other major markets. We expect our operating expenses to increase significantly in the foreseeable future, and we may not achieve or maintain profitability. If we are unable to attract or maintain a critical mass of Drivers, consumers, merchants, shippers, and carriers, whether as a result of competition or other factors, our platform will become less appealing to platform users.

💬🤖 How to Build a Chatbot

   ========================
   === Calling Function ===
   Calling function: vector_index_2020 with args: {
     "input": "risk factors"
   }
   Got output: The risk factors mentioned in the context include the adverse impact of the COVID-19 pandemic on the business, the potential reclassification of drivers as employees, the highly competitive nature of the mobility, delivery, and logistics industries, the need to lower fares or service fees to remain competitive, the company's history of significant losses and potential future expenses, the importance of attracting and maintaining a critical mass of platform users, and the operational and cultural challenges faced by the company.
   ========================
   === Calling Function ===
   Calling function: vector_index_2019 with args: {
     "input": "risk factors"
   }
   Got output: The risk factors mentioned in the context include competition with local companies, differing levels of social acceptance, technological compatibility issues, exposure to improper business practices, legal uncertainty, difficulties in managing international operations, fluctuations in currency exchange rates, regulations governing local currencies, tax consequences, financial accounting burdens, difficulties in implementing financial systems, import and export restrictions, political and economic instability, public health concerns, reduced protection for intellectual property rights, limited influence over minority-owned affiliates, and regulatory complexities. These risk factors could adversely affect the international operations, business, financial condition, and operating results of the company.
   ========================
   Here is a comparison of the risk factors described in the Uber 10-K reports across years:
   2022 Risk Factors:
   - Potential adverse effect if drivers were classified as employees instead of independent contractors.
   - Highly competitive nature of the mobility, delivery, and logistics industries.
   - Need to lower fares or service fees to remain competitive.
   - History of significant losses and expectation of increased operating expenses.
   - Impact of future pandemics or disease outbreaks on the business and financial results.
   - Potential harm to the business due to economic conditions and their effect on discretionary consumer spending.
   2021 Risk Factors:
   - Adverse impact of the COVID-19 pandemic and actions to mitigate it on the business.
   - Potential reclassification of drivers as employees instead of independent contractors.
   - Highly competitive nature of the mobility, delivery, and logistics industries.
   - Need to lower fares or service fees and offer incentives to remain competitive.
   - History of significant losses and uncertainty of achieving profitability.
   - Importance of attracting and maintaining a critical mass of platform users.
   2020 Risk Factors:
   - Adverse impact of the COVID-19 pandemic on the business.
   - Potential reclassification of drivers as employees.
   - Highly competitive nature of the mobility, delivery, and logistics industries.
   - Need to lower fares or service fees to remain competitive.
   - History of significant losses and potential future expenses.
   - Importance of attracting and maintaining a critical mass of platform users.
   - Operational and cultural challenges faced by the company.
   2019 Risk Factors:
   - Competition with local companies.
   - Differing levels of social acceptance.
   - Technological compatibility issues.
   - Exposure to improper business practices.
   - Legal uncertainty.
   - Difficulties in managing international operations.
   - Fluctuations in currency exchange rates.
   - Regulations governing local currencies.
   - Tax consequences.
   - Financial accounting burdens.
   - Difficulties in implementing financial systems.
   - Import and export restrictions.
   - Political and economic instability.
   - Public health concerns.
   - Reduced protection for intellectual property rights.
   - Limited influence over minority-owned affiliates.
   - Regulatory complexities.
   These comparisons highlight both common and unique risk factors that Uber faced in different years.
Setting up the Chatbot Loop
Now that we have the chatbot setup, it only takes a few more steps to
setup a basic interactive loop to chat with our SEC-augmented chatbot!
   agent = OpenAIAgent.from_tools(tools)  # verbose=False by default

💬🤖 How to Build a Chatbot

   while True:
       text_input = input("User: ")
       if text_input == "exit":
           break
       response = agent.chat(text_input)
       print(f"Agent: {response}")
Here's an example of the loop in action:
   User:  What were some of the legal proceedings against Uber in 2022?
   Agent: In 2022, Uber faced several legal proceedings. Some of the notable ones include:
   1. Petition against Proposition 22: A petition was filed in California alleging that Proposition 22, which classifies app-based drivers as independent contractors, is unconstitutional.
   2. Lawsuit by Massachusetts Attorney General: The Massachusetts Attorney General filed a lawsuit against Uber, claiming that drivers should be classified as employees and entitled to protections under wage and labor laws.
   3. Allegations by New York Attorney General: The New York Attorney General made allegations against Uber regarding the misclassification of drivers and related employment violations.
   4. Swiss social security rulings: Swiss social security rulings classified Uber drivers as employees, which could have implications for Uber's operations in Switzerland.
   5. Class action lawsuits in Australia: Uber faced class action lawsuits in Australia, with allegations that the company conspired to harm participants in the taxi, hire-car, and limousine industries.
   It's important to note that the outcomes of these legal proceedings are uncertain and may vary.
   User:
Notebook
Take a look at our corresponding notebook.


A Guide to Building a Full-Stack Web App with LLamaIndex

LlamaIndex is a python library, which means that integrating it with a
full-stack web application will be a little different than what you
might be used to.
This guide seeks to walk through the steps needed to create a basic
API service written in python, and how this interacts with a
TypeScript+React frontend.
All code examples here are available from the llama_index_starter_pack
in the flask_react folder.
The main technologies used in this guide are as follows:
* python3.11
* llama_index
* flask
* typescript
* react
Flask Backend
For this guide, our backend will use a Flask API server to communicate
with our frontend code. If you prefer, you can also easily translate
this to a FastAPI server, or any other python server library of your
choice.
Setting up a server using Flask is easy. You import the package,
create the app object, and then create your endpoints. Let's create a
basic skeleton for the server first:
   from flask import Flask
   app = Flask(__name__)
   @app.route("/")
   def home():
       return "Hello World!"
   if __name__ == "__main__":
       app.run(host="0.0.0.0", port=5601)
*flask_demo.py*
If you run this file ("python flask_demo.py"), it will launch a server
on port 5601. If you visit "http://localhost:5601/", you will see the
"Hello World!" text rendered in your browser. Nice!
The next step is deciding what functions we want to include in our
server, and to start using LlamaIndex.
To keep things simple, the most basic operation we can provide is
querying an existing index. Using the paul graham essay from
LlamaIndex, create a documents folder and download+place the essay
text file inside of it.
Basic Flask - Handling User Index Queries
Now, let's write some code to initialize our index:
   import os
   from llama_index import SimpleDirectoryReader, VectorStoreIndex, StorageContext
   # NOTE: for local testing only, do NOT deploy with your key hardcoded
   os.environ['OPENAI_API_KEY'] = "your key here"
   index = None
   def initialize_index():
       global index
       storage_context = StorageContext.from_defaults()
       if os.path.exists(index_dir):
           index = load_index_from_storage(storage_context)
       else:
           documents = SimpleDirectoryReader("./documents").load_data()
           index = VectorStoreIndex.from_documents(documents, storage_context=storage_context)
           storage_context.persist(index_dir)
This function will initialize our index. If we call this just before
starting the flask server in the "main" function, then our index will
be ready for user queries!
Our query endpoint will accept "GET" requests with the query text as a
parameter. Here's what the full endpoint function will look like:
   from flask import request
   @app.route("/query", methods=["GET"])
   def query_index():
     global index
     query_text = request.args.get("text", None)
     if query_text is None:
       return "No text found, please include a ?text=blah parameter in the URL", 400
     query_engine = index.as_query_engine()
     response = query_engine.query(query_text)
     return str(response), 200
Now, we've introduced a few new concepts to our server:
* a new "/query" endpoint, defined by the function decorator
* a new import from flask, "request", which is used to get parameters
  from the request
* if the "text" parameter is missing, then we return an error message
  and an appropriate HTML response code
* otherwise, we query the index, and return the response as a string
A full query example that you can test in your browser might look

something like this: "http://localhost:5601/query?text=what did the
author do growing up" (once you press enter, the browser will convert
the spaces into "%20" characters).
Things are looking pretty good! We now have a functional API. Using
your own documents, you can easily provide an interface for any
application to call the flask API and get answers to queries.
Advanced Flask - Handling User Document Uploads
Things are looking pretty cool, but how can we take this a step
further? What if we want to allow users to build their own indexes by
uploading their own documents? Have no fear, Flask can handle it all
:muscle:.
To let users upload documents, we have to take some extra precautions.
Instead of querying an existing index, the index will become
**mutable**. If you have many users adding to the same index, we need
to think about how to handle concurrency. Our Flask server is
threaded, which means multiple users can ping the server with requests
which will be handled at the same time.
One option might be to create an index for each user or group, and
store and fetch things from S3. But for this example, we will assume
there is one locally stored index that users are interacting with.
To handle concurrent uploads and ensure sequential inserts into the
index, we can use the "BaseManager" python package to provide
sequential access to the index using a separate server and locks. This
sounds scary, but it's not so bad! We will just move all our index
operations (initializing, querying, inserting) into the "BaseManager"
"index_server", which will be called from our Flask server.
Here's a basic example of what our "index_server.py" will look like
after we've moved our code:
   import os
   from multiprocessing import Lock
   from multiprocessing.managers import BaseManager
   from llama_index import SimpleDirectoryReader, VectorStoreIndex, Document
   # NOTE: for local testing only, do NOT deploy with your key hardcoded
   os.environ['OPENAI_API_KEY'] = "your key here"
   index = None
   lock = Lock()
   def initialize_index():
     global index
     with lock:
       # same as before ...
     ...
   def query_index(query_text):
     global index
     query_engine = index.as_query_engine()
     response = query_engine.query(query_text)
     return str(response)
   if __name__ == "__main__":
       # init the global index
       print("initializing index...")
       initialize_index()
       # setup server
       # NOTE: you might want to handle the password in a less hardcoded way
       manager = BaseManager(('', 5602), b'password')
       manager.register('query_index', query_index)
       server = manager.get_server()
       print("starting server...")
       server.serve_forever()
*index_server.py*
So, we've moved our functions, introduced the "Lock" object which
ensures sequential access to the global index, registered our single
function in the server, and started the server on port 5602 with the
password "password".
Then, we can adjust our flask code as follows:
   from multiprocessing.managers import BaseManager
   from flask import Flask, request
   # initialize manager connection
   # NOTE: you might want to handle the password in a less hardcoded way
   manager = BaseManager(('', 5602), b'password')
   manager.register('query_index')
   manager.connect()
   @app.route("/query", methods=["GET"])
   def query_index():
     global index
     query_text = request.args.get("text", None)
     if query_text is None:
       return "No text found, please include a ?text=blah parameter in the URL", 400
     response = manager.query_index(query_text)._getvalue()

     return str(response), 200
   @app.route("/")
   def home():
       return "Hello World!"
   if __name__ == "__main__":
       app.run(host="0.0.0.0", port=5601)
*flask_demo.py*
The two main changes are connecting to our existing "BaseManager"
server and registering the functions, as well as calling the function
through the manager in the "/query" endpoint.
One special thing to note is that "BaseManager" servers don't return
objects quite as we expect. To resolve the return value into it's
original object, we call the "_getvalue()" function.
If we allow users to upload their own documents, we should probably
remove the Paul Graham essay from the documents folder, so let's do
that first. Then, let's add an endpoint to upload files! First, let's
define our Flask endpoint function:
   ...
   manager.register('insert_into_index')
   ...
   @app.route("/uploadFile", methods=["POST"])
   def upload_file():
       global manager
       if 'file' not in request.files:
           return "Please send a POST request with a file", 400
       filepath = None
       try:
           uploaded_file = request.files["file"]
           filename = secure_filename(uploaded_file.filename)
           filepath = os.path.join('documents', os.path.basename(filename))
           uploaded_file.save(filepath)
           if request.form.get("filename_as_doc_id", None) is not None:
               manager.insert_into_index(filepath, doc_id=filename)
           else:
               manager.insert_into_index(filepath)
       except Exception as e:
           # cleanup temp file
           if filepath is not None and os.path.exists(filepath):
               os.remove(filepath)
           return "Error: {}".format(str(e)), 500
       # cleanup temp file
       if filepath is not None and os.path.exists(filepath):
           os.remove(filepath)
       return "File inserted!", 200
Not too bad! You will notice that we write the file to disk. We could
skip this if we only accept basic file formats like "txt" files, but
written to disk we can take advantage of LlamaIndex's
"SimpleDirectoryReader" to take care of a bunch of more complex file
formats. Optionally, we also use a second "POST" argument to either
use the filename as a doc_id or let LlamaIndex generate one for us.
This will make more sense once we implement the frontend.
With these more complicated requests, I also suggest using a tool like
Postman. Examples of using postman to test our endpoints are in the
repository for this project.
Lastly, you'll notice we added a new function to the manager. Let's
implement that inside "index_server.py":
   def insert_into_index(doc_text, doc_id=None):
       global index
       document = SimpleDirectoryReader(input_files=[doc_text]).load_data()[0]
       if doc_id is not None:
           document.doc_id = doc_id
       with lock:
           index.insert(document)
           index.storage_context.persist()
   ...
   manager.register('insert_into_index', insert_into_index)
   ...
Easy! If we launch both the "index_server.py" and then the
"flask_demo.py" python files, we have a Flask API server that can
handle multiple requests to insert documents into a vector index and
respond to user queries!
To support some functionality in the frontend, I've adjusted what some
responses look like from the Flask API, as well as added some
functionality to keep track of which documents are stored in the index
(LlamaIndex doesn't currently support this in a user-friendly way, but
we can augment it ourselves!). Lastly, I had to add CORS support to
the server using the "Flask-cors" python package.
Check out the complete "flask_demo.py" and "index_server.py" scripts

in the repository for the final minor changes, the"requirements.txt"
file, and a sample "Dockerfile" to help with deployment.
React Frontend
Generally, React and Typescript are one of the most popular libraries
and languages for writing webapps today. This guide will assume you
are familiar with how these tools work, because otherwise this guide
will triple in length :smile:.
In the repository, the frontend code is organized inside of the
"react_frontend" folder.
The most relevant part of the frontend will be the "src/apis" folder.
This is where we make calls to the Flask server, supporting the
following queries:
* "/query" -- make a query to the existing index
* "/uploadFile" -- upload a file to the flask server for insertion
  into the index
* "/getDocuments" -- list the current document titles and a portion of
  their texts
Using these three queries, we can build a robust frontend that allows
users to upload and keep track of their files, query the index, and
view the query response and information about which text nodes were
used to form the response.
fetchDocuments.tsx
This file contains the function to, you guessed it, fetch the list of
current documents in the index. The code is as follows:
   export type Document = {
     id: string;
     text: string;
   };
   const fetchDocuments = async (): Promise<Document[]> => {
     const response = await fetch("http://localhost:5601/getDocuments", {
       mode: "cors",
     });
     if (!response.ok) {
       return [];
     }
     const documentList = (await response.json()) as Document[];
     return documentList;
   };
As you can see, we make a query to the Flask server (here, it assumes
running on localhost). Notice that we need to include the "mode:
'cors'" option, as we are making an external request.
Then, we check if the response was ok, and if so, get the response
json and return it. Here, the response json is a list of "Document"
objects that are defined in the same file.
queryIndex.tsx
This file sends the user query to the flask server, and gets the
response back, as well as details about which nodes in our index
provided the response.
   export type ResponseSources = {
     text: string;
     doc_id: string;
     start: number;
     end: number;
     similarity: number;
   };
   export type QueryResponse = {
     text: string;
     sources: ResponseSources[];
   };
   const queryIndex = async (query: string): Promise<QueryResponse> => {
     const queryURL = new URL("http://localhost:5601/query?text=1");
     queryURL.searchParams.append("text", query);
     const response = await fetch(queryURL, { mode: "cors" });
     if (!response.ok) {
       return { text: "Error in query", sources: [] };
     }
     const queryResponse = (await response.json()) as QueryResponse;
     return queryResponse;
   };
   export default queryIndex;
This is similar to the "fetchDocuments.tsx" file, with the main
difference being we include the query text as a parameter in the URL.
Then, we check if the response is ok and return it with the
appropriate typescript type.
insertDocument.tsx
Probably the most complex API call is uploading a document. The
function here accepts a file object and constructs a "POST" request
using "FormData".
The actual response text is not used in the app but could be utilized
to provide some user feedback on if the file failed to upload or not.
   const insertDocument = async (file: File) => {
     const formData = new FormData();
     formData.append("file", file);
     formData.append("filename_as_doc_id", "true");

     const response = await fetch("http://localhost:5601/uploadFile", {
       mode: "cors",
       method: "POST",
       body: formData,
     });
     const responseText = response.text();
     return responseText;
   };
   export default insertDocument;
All the Other Frontend Good-ness
And that pretty much wraps up the frontend portion! The rest of the
react frontend code is some pretty basic react components, and my best
attempt to make it look at least a little nice :smile:.
I encourage to read the rest of the codebase and submit any PRs for
improvements!
Conclusion
This guide has covered a ton of information. We went from a basic
"Hello World" Flask server written in python, to a fully functioning
LlamaIndex powered backend and how to connect that to a frontend
application.
As you can see, we can easily augment and wrap the services provided
by LlamaIndex (like the little external document tracker) to help
provide a good user experience on the frontend.
You could take this and add many features (multi-index/user support,
saving objects into S3, adding a Pinecone vector server, etc.). And
when you build an app after reading this, be sure to share the final
result in the Discord! Good Luck! :muscle:


A Guide to Building a Full-Stack LlamaIndex Web App with Delphic

This guide seeks to walk you through using LlamaIndex with a
production-ready web app starter template called Delphic. All code
examples here are available from the Delphic repo
What We're Building
Here's a quick demo of the out-of-the-box functionality of Delphic:
https://user-
images.githubusercontent.com/5049984/233236432-aa4980b6-a510-42f3
-887a-81485c9644e6.mp4
Architectural Overview
Delphic leverages the LlamaIndex python library to let users to create
their own document collections they can then query in a responsive
frontend.
We chose a stack that provides a responsive, robust mix of
technologies that can (1) orchestrate complex python processing tasks
while providing (2) a modern, responsive frontend and (3) a secure
backend to build additional functionality upon.
The core libraries are:
1. Django
2. Django Channels
3. Django Ninja
4. Redis
5. Celery
6. LlamaIndex
7. Langchain
8. React
9. Docker & Docker Compose
Thanks to this modern stack built on the super stable Django web
framework, the starter Delphic app boasts a streamlined developer
experience, built-in authentication and user management, asynchronous
vector store processing, and web-socket-based query connections for a
responsive UI. In addition, our frontend is built with TypeScript and
is based on MUI React for a responsive and modern user interface.
System Requirements
Celery doesn't work on Windows. It may be deployable with Windows
Subsystem for Linux, but configuring that is beyond the scope of this
tutorial. For this reason, we recommend you only follow this tutorial
if you're running Linux or OSX. You will need Docker and Docker
Compose installed to deploy the application. Local development will
require node version manager (nvm).
Django Backend
Project Directory Overview
The Delphic application has a structured backend directory
organization that follows common Django project conventions. From the
repo root, in the "./delphic" subfolder, the main folders are:
1. "contrib": This directory contains custom modifications or
   additions to Django's built-in "contrib" apps.
2. "indexes": This directory contains the core functionality related
   to document indexing and LLM integration. It includes:
* "admin.py": Django admin configuration for the app
* "apps.py": Application configuration
* "models.py": Contains the app's database models
* "migrations": Directory containing database schema migrations for
  the app
* "signals.py": Defines any signals for the app
* "tests.py": Unit tests for the app
3. "tasks": This directory contains tasks for asynchronous processing
   using Celery. The "index_tasks.py" file includes the tasks for
   creating vector indexes.
4. "users": This directory is dedicated to user management, including:
5. "utils": This directory contains utility modules and functions that
   are used across the application, such as custom storage backends,
   path helpers, and collection-related utilities.
Database Models
The Delphic application has two core models: "Document" and
"Collection". These models represent the central entities the
application deals with when indexing and querying documents using
LLMs. They're defined in "./delphic/indexes/models.py".
1. "Collection":
* "api_key": A foreign key that links a collection to an API key. This
  helps associate jobs with the source API key.
* "title": A character field that provides a title for the collection.
* "description": A text field that provides a description of the
  collection.
* "status": A character field that stores the processing status of the
  collection, utilizing the "CollectionStatus" enumeration.

* "created": A datetime field that records when the collection was
  created.
* "modified": A datetime field that records the last modification time
  of the collection.
* "model": A file field that stores the model associated with the
  collection.
* "processing": A boolean field that indicates if the collection is
  currently being processed.
2. "Document":
* "collection": A foreign key that links a document to a collection.
  This represents the relationship between documents and collections.
* "file": A file field that stores the uploaded document file.
* "description": A text field that provides a description of the
  document.
* "created": A datetime field that records when the document was
  created.
* "modified": A datetime field that records the last modification time
  of the document.
These models provide a solid foundation for collections of documents
and the indexes created from them with LlamaIndex.
Django Ninja API
Django Ninja is a web framework for building APIs with Django and
Python 3.7+ type hints. It provides a simple, intuitive, and
expressive way of defining API endpoints, leveraging Python’s type
hints to automatically generate input validation, serialization, and
documentation.
In the Delphic repo, the "./config/api/endpoints.py" file contains the
API routes and logic for the API endpoints. Now, let’s briefly address
the purpose of each endpoint in the "endpoints.py" file:
1. "/heartbeat": A simple GET endpoint to check if the API is up and
   running. Returns "True" if the API is accessible. This is helpful
   for Kubernetes setups that expect to be able to query your
   container to ensure it's up and running.
2. "/collections/create": A POST endpoint to create a new
   "Collection". Accepts form parameters such as "title",
   "description", and a list of "files". Creates a new "Collection"
   and "Document" instances for each file, and schedules a Celery task
   to create an index.
   @collections_router.post("/create")
   async def create_collection(request,
                               title: str = Form(...),
                               description: str = Form(...),
                               files: list[UploadedFile] = File(...), ):
       key = None if getattr(request, "auth", None) is None else request.auth
       if key is not None:
           key = await key
       collection_instance = Collection(
           api_key=key,
           title=title,
           description=description,
           status=CollectionStatusEnum.QUEUED,
       )
       await sync_to_async(collection_instance.save)()
       for uploaded_file in files:
           doc_data = uploaded_file.file.read()
           doc_file = ContentFile(doc_data, uploaded_file.name)
           document = Document(collection=collection_instance, file=doc_file)
           await sync_to_async(document.save)()
       create_index.si(collection_instance.id).apply_async()
       return await sync_to_async(CollectionModelSchema)(
           ...
       )
3. "/collections/query" — a POST endpoint to query a document
   collection using the LLM. Accepts a JSON payload containing
   "collection_id" and "query_str", and returns a response generated
   by querying the collection. We don't actually use this endpoint in
   our chat GUI (We use a websocket - see below), but you could build
   an app to integrate to this REST endpoint to query a specific
   collection.
   @collections_router.post("/query",
                            response=CollectionQueryOutput,
                            summary="Ask a question of a document collection", )
   def query_collection_view(request: HttpRequest, query_input: CollectionQueryInput):
       collection_id = query_input.collection_id
       query_str = query_input.query_str
       response = query_collection(collection_id, query_str)
       return {"response": response}
4. "/collections/available": A GET endpoint that returns a list of all

   collections created with the user's API key. The output is
   serialized using the "CollectionModelSchema".
   @collections_router.get("/available",
                           response=list[CollectionModelSchema],
                           summary="Get a list of all of the collections created with my api_key", )
   async def get_my_collections_view(request: HttpRequest):
       key = None if getattr(request, "auth", None) is None else request.auth
       if key is not None:
           key = await key
       collections = Collection.objects.filter(api_key=key)
       return [
           {
               ...
           }
           async for collection in collections
       ]
5. "/collections/{collection_id}/add_file": A POST endpoint to add a
   file to an existing collection. Accepts a "collection_id" path
   parameter, and form parameters such as "file" and "description".
   Adds the file as a "Document" instance associated with the
   specified collection.
   @collections_router.post("/{collection_id}/add_file", summary="Add a file to a collection")
   async def add_file_to_collection(request,
                                    collection_id: int,
                                    file: UploadedFile = File(...),
                                    description: str = Form(...), ):
       collection = await sync_to_async(Collection.objects.get)(id=collection_id
Intro to Websockets
WebSockets are a communication protocol that enables bidirectional and
full-duplex communication between a client and a server over a single,
long-lived connection. The WebSocket protocol is designed to work over
the same ports as HTTP and HTTPS (ports 80 and 443, respectively) and
uses a similar handshake process to establish a connection. Once the
connection is established, data can be sent in both directions as
“frames” without the need to reestablish the connection each time,
unlike traditional HTTP requests.
There are several reasons to use WebSockets, particularly when working
with code that takes a long time to load into memory but is quick to
run once loaded:
1. **Performance**: WebSockets eliminate the overhead associated with
   opening and closing multiple connections for each request, reducing
   latency.
2. **Efficiency**: WebSockets allow for real-time communication
   without the need for polling, resulting in more efficient use of
   resources and better responsiveness.
3. **Scalability**: WebSockets can handle a large number of
   simultaneous connections, making it ideal for applications that
   require high concurrency.
In the case of the Delphic application, using WebSockets makes sense
as the LLMs can be expensive to load into memory. By establishing a
WebSocket connection, the LLM can remain loaded in memory, allowing
subsequent requests to be processed quickly without the need to reload
the model each time.
The ASGI configuration file "./config/asgi.py" defines how the
application should handle incoming connections, using the Django
Channels "ProtocolTypeRouter" to route connections based on their
protocol type. In this case, we have two protocol types: "http" and
"websocket".
The “http” protocol type uses the standard Django ASGI application to
handle HTTP requests, while the “websocket” protocol type uses a
custom "TokenAuthMiddleware" to authenticate WebSocket connections.
The "URLRouter" within the "TokenAuthMiddleware" defines a URL pattern
for the "CollectionQueryConsumer", which is responsible for handling
WebSocket connections related to querying document collections.
   application = ProtocolTypeRouter(
       {
           "http": get_asgi_application(),
           "websocket": TokenAuthMiddleware(
               URLRouter(
                   [
                       re_path(
                           r"ws/collections/(?P<collection_id>\w+)/query/$",
                           CollectionQueryConsumer.as_asgi(),
                       ),
                   ]
               )
           ),
       }
   )
This configuration allows clients to establish WebSocket connections
with the Delphic application to efficiently query document collections

using the LLMs, without the need to reload the models for each
request.
Websocket Handler
The "CollectionQueryConsumer" class in
"config/api/websockets/queries.py" is responsible for handling
WebSocket connections related to querying document collections. It
inherits from the "AsyncWebsocketConsumer" class provided by Django
Channels.
The "CollectionQueryConsumer" class has three main methods:
1. "connect": Called when a WebSocket is handshaking as part of the
   connection process.
2. "disconnect": Called when a WebSocket closes for any reason.
3. "receive": Called when the server receives a message from the
   WebSocket.
Websocket connect listener
~~~~~~~~~~~~~~~~~~~~~~~~~~
The "connect" method is responsible for establishing the connection,
extracting the collection ID from the connection path, loading the
collection model, and accepting the connection.
   async def connect(self):
       try:
           self.collection_id = extract_connection_id(self.scope["path"])
           self.index = await load_collection_model(self.collection_id)
           await self.accept()
   except ValueError as e:
   await self.accept()
   await self.close(code=4000)
   except Exception as e:
   pass
Websocket disconnect listener
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The "disconnect" method is empty in this case, as there are no
additional actions to be taken when the WebSocket is closed.
Websocket receive listener
~~~~~~~~~~~~~~~~~~~~~~~~~~
The "receive" method is responsible for processing incoming messages
from the WebSocket. It takes the incoming message, decodes it, and
then queries the loaded collection model using the provided query. The
response is then formatted as a markdown string and sent back to the
client over the WebSocket connection.
   async def receive(self, text_data):
       text_data_json = json.loads(text_data)
       if self.index is not None:
           query_str = text_data_json["query"]
           modified_query_str = f"Please return a nicely formatted markdown string to this request:\n\n{query_str}"
           query_engine = self.index.as_query_engine()
           response = query_engine.query(modified_query_str)
           markdown_response = f"## Response\n\n{response}\n\n"
           if response.source_nodes:
               markdown_sources = f"## Sources\n\n{response.get_formatted_sources()}"
           else:
               markdown_sources = ""
           formatted_response = f"{markdown_response}{markdown_sources}"
           await self.send(json.dumps({"response": formatted_response}, indent=4))
       else:
           await self.send(json.dumps({"error": "No index loaded for this connection."}, indent=4))
To load the collection model, the "load_collection_model" function is
used, which can be found in "delphic/utils/collections.py". This
function retrieves the collection object with the given collection ID,
checks if a JSON file for the collection model exists, and if not,
creates one. Then, it sets up the "LLMPredictor" and "ServiceContext"
before loading the "VectorStoreIndex" using the cache file.
   async def load_collection_model(collection_id: str | int) -> VectorStoreIndex:
       """
       Load the Collection model from cache or the database, and return the index.
       Args:
           collection_id (Union[str, int]): The ID of the Collection model instance.
       Returns:
           VectorStoreIndex: The loaded index.
       This function performs the following steps:
       1. Retrieve the Collection object with the given collection_id.
       2. Check if a JSON file with the name '/cache/model_{collection_id}.json' exists.
       3. If the JSON file doesn't exist, load the JSON from the Collection.model FileField and save it to
          '/cache/model_{collection_id}.json'.
       4. Call VectorStoreIndex.load_from_disk with the cache_file_path.
       """
       # Retrieve the Collection object

       collection = await Collection.objects.aget(id=collection_id)
       logger.info(f"load_collection_model() - loaded collection {collection_id}")
       # Make sure there's a model
       if collection.model.name:
           logger.info("load_collection_model() - Setup local json index file")
           # Check if the JSON file exists
           cache_dir = Path(settings.BASE_DIR) / "cache"
           cache_file_path = cache_dir / f"model_{collection_id}.json"
           if not cache_file_path.exists():
               cache_dir.mkdir(parents=True, exist_ok=True)
               with collection.model.open("rb") as model_file:
                   with cache_file_path.open("w+", encoding="utf-8") as cache_file:
                       cache_file.write(model_file.read().decode("utf-8"))
           # define LLM
           logger.info(
               f"load_collection_model() - Setup service context with tokens {settings.MAX_TOKENS} and "
               f"model {settings.MODEL_NAME}"
           )
           llm = OpenAI(temperature=0, model="text-davinci-003", max_tokens=512)
           service_context = ServiceContext.from_defaults(llm=llm)
           # Call VectorStoreIndex.load_from_disk
           logger.info("load_collection_model() - Load llama index")
           index = VectorStoreIndex.load_from_disk(
               cache_file_path, service_context=service_context
           )
           logger.info(
               "load_collection_model() - Llamaindex loaded and ready for query..."
           )
       else:
           logger.error(
               f"load_collection_model() - collection {collection_id} has no model!"
           )
           raise ValueError("No model exists for this collection!")
       return index
React Frontend
Overview
We chose to use TypeScript, React and Material-UI (MUI) for the
Delphic project’s frontend for a couple reasons. First, as the most
popular component library (MUI) for the most popular frontend
framework (React), this choice makes this project accessible to a huge
community of developers. Second, React is, at this point, a stable and
generally well-liked framework that delivers valuable abstractions in
the form of its virtual DOM while still being relatively stable and,
in our opinion, pretty easy to learn, again making it accessible.
Frontend Project Structure
The frontend can be found in the "/frontend" directory of the repo,
with the React-related components being in "/frontend/src" . You’ll
notice there is a DockerFile in the "frontend" directory and several
folders and files related to configuring our frontend web server —
nginx.
The "/frontend/src/App.tsx" file serves as the entry point of the
application. It defines the main components, such as the login form,
the drawer layout, and the collection create modal. The main
components are conditionally rendered based on whether the user is
logged in and has an authentication token.
The DrawerLayout2 component is defined in the"DrawerLayour2.tsx" file.
This component manages the layout of the application and provides the
navigation and main content areas.
Since the application is relatively simple, we can get away with not
using a complex state management solution like Redux and just use
React’s useState hooks.
Grabbing Collections from the Backend
The collections available to the logged-in user are retrieved and
displayed in the DrawerLayout2 component. The process can be broken
down into the following steps:
1. Initializing state variables:
   const [collections, setCollections] = useState<CollectionModelSchema[]>([]);
   const [loading, setLoading] = useState(true);
Here, we initialize two state variables: "collections" to store the
list of collections and "loading" to track whether the collections are
being fetched.
2. Collections are fetched for the logged-in user with the
   "fetchCollections()" function:

   const
   fetchCollections = async () = > {
   try {
   const accessToken = localStorage.getItem("accessToken");
   if (accessToken) {
   const response = await getMyCollections(accessToken);
   setCollections(response.data);
   }
   } catch (error) {
   console.error(error);
   } finally {
   setLoading(false);
   }
   };
The "fetchCollections" function retrieves the collections for the
logged-in user by calling the "getMyCollections" API function with the
user's access token. It then updates the "collections" state with the
retrieved data and sets the "loading" state to "false" to indicate
that fetching is complete.
Displaying Collections
The latest collectios are displayed in the drawer like this:
   < List >
   {collections.map((collection) = > (
       < div key={collection.id} >
       < ListItem disablePadding >
       < ListItemButton
       disabled={
       collection.status != = CollectionStatus.COMPLETE | |
       !collection.has_model
       }
       onClick={() = > handleCollectionClick(collection)}
   selected = {
       selectedCollection & &
       selectedCollection.id == = collection.id
   }
   >
   < ListItemText
   primary = {collection.title} / >
             {collection.status == = CollectionStatus.RUNNING ? (
       < CircularProgress
       size={24}
       style={{position: "absolute", right: 16}}
       / >
   ): null}
   < / ListItemButton >
       < / ListItem >
           < / div >
   ))}
   < / List >
You’ll notice that the "disabled" property of a collection’s
"ListItemButton" is set based on whether the collection's status is
not "CollectionStatus.COMPLETE" or the collection does not have a
model ("!collection.has_model"). If either of these conditions is
true, the button is disabled, preventing users from selecting an
incomplete or model-less collection. Where the CollectionStatus is
RUNNING, we also show a loading wheel over the button.
In a separate "useEffect" hook, we check if any collection in the
"collections" state has a status of "CollectionStatus.RUNNING" or
"CollectionStatus.QUEUED". If so, we set up an interval to repeatedly
call the "fetchCollections" function every 15 seconds (15,000
milliseconds) to update the collection statuses. This way, the
application periodically checks for completed collections, and the UI
is updated accordingly when the processing is done.
   useEffect(() = > {
       let
   interval: NodeJS.Timeout;
   if (
       collections.some(
           (collection) = >
   collection.status == = CollectionStatus.RUNNING | |
   collection.status == = CollectionStatus.QUEUED
   )
   ) {
       interval = setInterval(() = > {
       fetchCollections();
   }, 15000);
   }
   return () = > clearInterval(interval);
   }, [collections]);
Chat View Component
The "ChatView" component in "frontend/src/chat/ChatView.tsx" is
responsible for handling and displaying a chat interface for a user to
interact with a collection. The component establishes a WebSocket
connection to communicate in real-time with the server, sending and
receiving messages.
Key features of the "ChatView" component include:
1. Establishing and managing the WebSocket connection with the server.
2. Displaying messages from the user and the server in a chat-like
   format.
3. Handling user input to send messages to the server.
4. Updating the messages state and UI based on received messages from
   the server.
5. Displaying connection status and errors, such as loading messages,
   connecting to the server, or encountering errors while loading a
   collection.
Together, all of this allows users to interact with their selected
collection with a very smooth, low-latency experience.

Chat Websocket Client
~~~~~~~~~~~~~~~~~~~~~
The WebSocket connection in the "ChatView" component is used to
establish real-time communication between the client and the server.
The WebSocket connection is set up and managed in the "ChatView"
component as follows:
First, we want to initialize the the WebSocket reference:
const websocket = useRef<WebSocket | null>(null);
A "websocket" reference is created using "useRef", which holds the
WebSocket object that will be used for communication. "useRef" is a
hook in React that allows you to create a mutable reference object
that persists across renders. It is particularly useful when you need
to hold a reference to a mutable object, such as a WebSocket
connection, without causing unnecessary re-renders.
In the "ChatView" component, the WebSocket connection needs to be
established and maintained throughout the lifetime of the component,
and it should not trigger a re-render when the connection state
changes. By using "useRef", you ensure that the WebSocket connection
is kept as a reference, and the component only re-renders when there
are actual state changes, such as updating messages or displaying
errors.
The "setupWebsocket" function is responsible for establishing the
WebSocket connection and setting up event handlers to handle different
WebSocket events.
Overall, the setupWebsocket function looks like this:
   const setupWebsocket = () => {
     setConnecting(true);
     // Here, a new WebSocket object is created using the specified URL, which includes the
     // selected collection's ID and the user's authentication token.
     websocket.current = new WebSocket(
       `ws://localhost:8000/ws/collections/${selectedCollection.id}/query/?token=${authToken}`,
     );
     websocket.current.onopen = (event) => {
       //...
     };
     websocket.current.onmessage = (event) => {
       //...
     };
     websocket.current.onclose = (event) => {
       //...
     };
     websocket.current.onerror = (event) => {
       //...
     };
     return () => {
       websocket.current?.close();
     };
   };
Notice in a bunch of places we trigger updates to the GUI based on the
information from the web socket client.
When the component first opens and we try to establish a connection,
the "onopen" listener is triggered. In the callback, the component
updates the states to reflect that the connection is established, any
previous errors are cleared, and no messages are awaiting responses:
   websocket.current.onopen = (event) => {
     setError(false);
     setConnecting(false);
     setAwaitingMessage(false);
     console.log("WebSocket connected:", event);
   };
"onmessage"is triggered when a new message is received from the server
through the WebSocket connection. In the callback, the received data
is parsed and the "messages" state is updated with the new message
from the server:
   websocket.current.onmessage = (event) => {
     const data = JSON.parse(event.data);
     console.log("WebSocket message received:", data);
     setAwaitingMessage(false);
     if (data.response) {
       // Update the messages state with the new message from the server
       setMessages((prevMessages) => [
         ...prevMessages,
         {
           sender_id: "server",
           message: data.response,
           timestamp: new Date().toLocaleTimeString(),
         },
       ]);
     }
   };
"onclose"is triggered when the WebSocket connection is closed. In the
callback, the component checks for a specific close code ("4000") to
display a warning toast and update the component states accordingly.
It also logs the close event:
   websocket.current.onclose = (event) => {
     if (event.code === 4000) {
       toast.warning(
         "Selected collection's model is unavailable. Was it created properly?",
       );
       setError(true);
       setConnecting(false);
       setAwaitingMessage(false);

     }
     console.log("WebSocket closed:", event);
   };
Finally, "onerror" is triggered when an error occurs with the
WebSocket connection. In the callback, the component updates the
states to reflect the error and logs the error event:
   websocket.current.onerror = (event) => {
     setError(true);
     setConnecting(false);
     setAwaitingMessage(false);
     console.error("WebSocket error:", event);
   };
Rendering our Chat Messages
~~~~~~~~~~~~~~~~~~~~~~~~~~~
In the "ChatView" component, the layout is determined using CSS
styling and Material-UI components. The main layout consists of a
container with a "flex" display and a column-oriented "flexDirection".
This ensures that the content within the container is arranged
vertically.
There are three primary sections within the layout:
1. The chat messages area: This section takes up most of the available
   space and displays a list of messages exchanged between the user
   and the server. It has an overflow-y set to ‘auto’, which allows
   scrolling when the content overflows the available space. The
   messages are rendered using the "ChatMessage" component for each
   message and a "ChatMessageLoading" component to show the loading
   state while waiting for a server response.
2. The divider: A Material-UI "Divider" component is used to separate
   the chat messages area from the input area, creating a clear visual
   distinction between the two sections.
3. The input area: This section is located at the bottom and allows
   the user to type and send messages. It contains a "TextField"
   component from Material-UI, which is set to accept multiline input
   with a maximum of 2 rows. The input area also includes a "Button"
   component to send the message. The user can either click the "Send"
   button or press " Enter" on their keyboard to send the message.
The user inputs accepted in the "ChatView" component are text messages
that the user types in the "TextField". The component processes these
text inputs and sends them to the server through the WebSocket
connection.
Deployment
Prerequisites
To deploy the app, you're going to need Docker and Docker Compose
installed. If you're on Ubuntu or another, common Linux distribution,
DigitalOcean has a great Docker tutorial and another great tutorial
for Docker Compose you can follow. If those don't work for you, try
the official docker documentation.
Build and Deploy
The project is based on django-cookiecutter, and it’s pretty easy to
get it deployed on a VM and configured to serve HTTPs traffic for a
specific domain. The configuration is somewhat involved, however — not
because of this project, but it’s just a fairly involved topic to
configure your certificates, DNS, etc.
For the purposes of this guide, let’s just get running locally.
Perhaps we’ll release a guide on production deployment. In the
meantime, check out the Django Cookiecutter project docs for starters.
This guide assumes your goal is to get the application up and running
for use. If you want to develop, most likely you won’t want to launch
the compose stack with the — profiles fullstack flag and will instead
want to launch the react frontend using the node development server.
To deploy, first clone the repo:
   git clone https://github.com/yourusername/delphic.git
Change into the project directory:
   cd delphic
Copy the sample environment files:
   mkdir -p ./.envs/.local/
   cp -a ./docs/sample_envs/local/.frontend ./frontend
   cp -a ./docs/sample_envs/local/.django ./.envs/.local
   cp -a ./docs/sample_envs/local/.postgres ./.envs/.local
Edit the ".django" and ".postgres" configuration files to include your

OpenAI API key and set a unique password for your database user. You
can also set the response token limit in the .django file or switch
which OpenAI model you want to use. GPT4 is supported, assuming you’re
authorized to access it.
Build the docker compose stack with the "--profiles fullstack" flag:
   sudo docker-compose --profiles fullstack -f local.yml build
The fullstack flag instructs compose to build a docker container from
the frontend folder and this will be launched along with all of the
needed, backend containers. It takes a long time to build a production
React container, however, so we don’t recommend you develop this way.
Follow the instructions in the project readme.md for development
environment setup instructions.
Finally, bring up the application:
   sudo docker-compose -f local.yml up
Now, visit "localhost:3000" in your browser to see the frontend, and
use the Delphic application locally.
Using the Application
Setup Users
In order to actually use the application (at the moment, we intend to
make it possible to share certain models with unauthenticated users),
you need a login. You can use either a superuser or non-superuser. In
either case, someone needs to first create a superuser using the
console:
**Why set up a Django superuser?** A Django superuser has all the
permissions in the application and can manage all aspects of the
system, including creating, modifying, and deleting users,
collections, and other data. Setting up a superuser allows you to
fully control and manage the application.
**How to create a Django superuser:**
1 Run the following command to create a superuser:
sudo docker-compose -f local.yml run django python manage.py
createsuperuser
2 You will be prompted to provide a username, email address, and
password for the superuser. Enter the required information.
**How to create additional users using Django admin:**
1. Start your Delphic application locally following the deployment
   instructions.
2. Visit the Django admin interface by navigating to
   "http://localhost:8000/admin" in your browser.
3. Log in with the superuser credentials you created earlier.
4. Click on “Users” under the “Authentication and Authorization”
   section.
5. Click on the “Add user +” button in the top right corner.
6. Enter the required information for the new user, such as username
   and password. Click “Save” to create the user.
7. To grant the new user additional permissions or make them a
   superuser, click on their username in the user list, scroll down to
   the “Permissions” section, and configure their permissions
   accordingly. Save your changes.


Component Wise Evaluation

To do more in-depth evaluation of your pipeline, it helps to break it
down into an evaluation of individual components.
For instance, a particular failure case may be due to a combination of
not retrieving the right documents and also the LLM misunderstanding
the context and hallucinating an incorrect result. Being able to
isolate and deal with these issues separately can help reduce
complexity and guide you in a step-by-step manner to a more
satisfactory overall result.
Utilizing public benchmarks
When doing initial model selection, it helps to look at how well the
model is performing on a standardized, diverse set of domains or
tasks.
A useful benchmark for embeddings is the MTEB Leaderboard.
Evaluating Retrieval
BEIR dataset
BEIR is useful for benchmarking if a particular retrieval model
generalize well to niche domains in a zero-shot setting.
Since most publically-available embedding and retrieval models are
already benchmarked against BEIR (e.g. through the MTEB benchmark),
utilizing BEIR is more helpful when you have a unique model that you
want to evaluate.
For instance, after fine-tuning an embedding model on your dataset, it
may be helpful to view whether and by how much its performance
degrades on a diverse set of domains. This can be an indication of how
much data drift may affect your retrieval accuracy, such as if you add
documents to your RAG system outside of your fine-tuning training
distribution.
Here is a notebook showing how the BEIR dataset can be used with your
retrieval pipeline.
* BEIR Out of Domain Benchmark
We will be adding more methods to evaluate retrieval soon. This
includes evaluating retrieval on your own dataset.
Evaluating the Query Engine Components (e.g. Without Retrieval)
In this case, we may want to evaluate how specific components of a
query engine (one which may generate sub-questions or follow-up
questions) may perform on a standard benchmark. It can help give an
indication of how far behind or ahead your retrieval pipeline is
compared to alternate pipelines or models.
HotpotQA Dataset
The HotpotQA dataset is useful for evaluating queries that require
multiple retrieval steps.
Example:
* HotpotQADistractor Demo
Limitations:
1. HotpotQA is evaluated on a Wikipedia corpus. LLMs, especially GPT4,
   tend to have memorized information from Wikipedia relatively well.
   Hence, the benchmark is not particularly good for evaluating
   retrieval + rerank systems with knowledgeable models like GPT4.


Observability

Why Observability?
In a complex LLM application with many moving parts, as with
traditional software engineering, it helps to be able to inspect the
artifacts and execution traces of the application.
How to Set Up Observability
You may refer to our "One-Click Observability" guide to set up
observability with your preferred observability provider.


Building Performant RAG Applications for Production

Prototyping a RAG application is easy, but making it performant,
robust, and scalable to a large knowledge corpus is hard.
This guide contains a variety of tips and tricks to improve the
performance of your RAG pipeline. We first outline some general
techniques - they are loosely ordered in terms of most straightforward
to most challenging. We then dive a bit more deeply into each
technique, the use cases that it solves, and how to implement it with
LlamaIndex!
The end goal is to optimize your retrieval and generation performance
to answer more queries over more complex datasets accurately and
without hallucinations.
General Techniques for Building Production-Grade RAG
Here are some top Considerations for Building Production-Grade RAG
* Decoupling chunks used for retrieval vs. chunks used for synthesis
* Structured Retrieval for Larger Document Sets
* Dynamically Retrieve Chunks Depending on your Task
* Optimize context embeddings
We discussed this and more during our Production RAG Webinar. Check
out this Tweet thread for more synthesized details.
Decoupling Chunks Used for Retrieval vs. Chunks Used for Synthesis
A key technique for better retrieval is to decouple chunks used for
retrieval with those that are used for synthesis.
[image: ][image]
Motivation
The optimal chunk representation for retrieval might be different than
the optimal consideration used for synthesis. For instance, a raw text
chunk may contain needed details for the LLM to synthesize a more
detailed answer given a query. However, it may contain filler
words/info that may bias the embedding representation, or it may lack
global context and not be retrieved at all when a relevant query comes
in.
Key Techniques
There’s two main ways to take advantage of this idea:
**1. Embed a document summary, which links to chunks associated with
the document.**
This can help retrieve relevant documents at a high-level before
retrieving chunks vs. retrieving chunks directly (that might be in
irrelevant documents).
Resources:
* Recursive Retriever + Query Engine Demo
* Document Summary Index
**2. Embed a sentence, which then links to a window around the
sentence.**
This allows for finer-grained retrieval of relevant context (embedding
giant chunks leads to “lost in the middle” problems), but also ensures
enough context for LLM synthesis.
Resources:
* Metadata Replacement + Node Sentence Window
Structured Retrieval for Larger Document Sets
[image: ][image]
Motivation
A big issue with the standard RAG stack (top-k retrieval + basic text
splitting) is that it doesn’t do well as the number of documents
scales up - e.g. if you have 100 different PDFs. In this setting,
given a query you may want to use structured information to help with
more precise retrieval; for instance, if you ask a question that's
only relevant to two PDFs, using structured information to ensure
those two PDFs get returned beyond raw embedding similarity with
chunks.
Key Techniques
There’s a few ways of performing more structured tagging/retrieval for
production-quality RAG systems, each with their own pros/cons.
**1. Metadata Filters + Auto Retrieval** Tag each document with
metadata and then store in a vector database. During inference time,
use the LLM to infer the right metadata filters to query the vector db
in addition to the semantic query string.
* Pros ✅: Supported in major vector dbs. Can filter document via
  multiple dimensions.
* Cons 🚫: Can be hard to define the right tags. Tags may not contain
  enough relevant information for more precise retrieval. Also tags
  represent keyword search at the document-level, doesn’t allow for
  semantic lookups.
Resources: **2. Store Document Hierarchies (summaries -> raw chunks) +

Recursive Retrieval** Embed document summaries and map to chunks per
document. Fetch at the document-level first before chunk level.
* Pros ✅: Allows for semantic lookups at the document level.
* Cons 🚫: Doesn’t allow for keyword lookups by structured tags (can
  be more precise than semantic search). Also autogenerating summaries
  can be expensive.
**Resources**
* Auto-Retrieval from a Vector Database
* Document Summary Index
* Recursive Retriever + Document Agents
* Comparing Methods for Structured Retrieval (Auto-Retrieval vs.
  Recursive Retrieval)
Dynamically Retrieve Chunks Depending on your Task
[image: ][image]
Motivation
RAG isn't just about question-answering about specific facts, which
top-k similarity is optimized for. There can be a broad range of
queries that a user might ask. Queries that are handled by naive RAG
stacks include ones that ask about specific facts e.g. "Tell me about
the D&I initiatives for this company in 2023" or "What did the
narrator do during his time at Google". But queries can also include
summarization e.g. "Can you give me a high-level overview of this
document", or comparisons "Can you compare/contrast X and Y". All of
these use cases may require different retrieval techniques.
Key Techniques
LlamaIndex provides some core abstractions to help you do task-
specific retrieval. This includes our router module. This also
includes our data agent module. This also includes some advanced query
engine modules. This also include other modules that join structured
and unstructured data.
You can use these modules to do joint question-answering and
summarization, or even combine structured queries with unstructured
queries.
**Core Module Resources**
* Query Engine
* Routers
* Data Agents
**Detailed Guide Resources**
* Sub Question Query Engine
* Joint QA Summary Query Engine
* Recursive Retriever + Document Agents
* Router Query Engine
* OpenAI Agent + Query Engine Experimental Cookbook
* OpenAI Agent Query Planning
Optimize Context Embeddings
Motivation
This is related to the motivation described above in "decoupling
chunks used for retrieval vs. synthesis". We want to make sure that
the embeddings are optimized for better retrieval over your specific
data corpus. Pre-trained models may not capture the salient properties
of the data relevant to your use case.
Key Techniques
Beyond some of the techniques listed above, we can also try finetuning
the embedding model. We can actually do this over an unstructured text
corpus, in a label-free way.
Check out our guides here:
* Embedding Fine-tuning Guide


Monitoring

Why Monitoring?
When developing your LLM application, it can be helpful to keep track
of production data such as:
* Pipeline performance (latency/token count/throughput of various
  stages)
* Resource usage (LLM/Embedding Inference Cost, CPU/GPU utilization)
* Evaluation metrics (accuracy, precision, recall, qualitative eval
  and drift)
* Pipeline versioning (which versions of sub-components e.g.
  LLM/embedding & artifacts e.g. prompts were used in the pipeline at
  a given time)
We will share more on how to set up monitoring in the future.


Evaluation

Setting the Stage
LlamaIndex is meant to connect your data to your LLM applications.
Sometimes, even after diagnosing and fixing bugs by looking at traces,
more fine-grained evaluation is required to systematically diagnose
issues.
LlamaIndex aims to provide those tools to make identifying issues and
receiving useful diagnostic signals easy.
Closely tied to evaluation are the concepts of experimentation and
experiment tracking.
General Strategy
When developing your LLM application, it could help to first define an
end-to-end evaluation workflow, and then once you've started
collecting failure or corner cases and getting an intuition for what
is or isn't going well, you may dive deeper into evaluating and
improving specific components.
The analogy with software testing is integration tests and unit tests.
You should probably start writing unit tests once you start fiddling
with individual components. Equally, your gold standard on whether
things are working will together are integration tests. Both are
equally important.
* End-to-End Evaluation
* Component Wise Evaluation
Here is an overview of the existing modules for evaluation. We will be
adding more modules and support over time.
* Evaluation
E2E or Component-Wise - Which Do I Start With?
If you want to get an overall idea of how your system is doing as you
iterate upon it, it makes sense to start with centering your core
development loop around the e2e eval - as an overall sanity/vibe
check.
If you have an idea of what you're doing and want to iterate step by
step on each component, building it up as things go - you may want to
start with a component-wise eval. However this may run the risk of
premature optimization - making model selection or parameter choices
without assessing the overall application needs. You may have to
revisit these choices when creating your final application.
Diving Deeper into Evaluation
Evaluation is a controversial topic, and as the field of NLP has
evolved, so have the methods of evaluation.
In a world where powerful foundation models are now performing
annotation tasks better than human annotators, the best practices
around evaluation are constantly changing. Previous methods of
evaluation which were used to bootstrap and evaluate today's models
such as BLEU or F1 have been shown to have poor correlation with human
judgements, and need to be applied prudently.
Typically, generation-heavy, open-ended tasks and requiring judgement
or opinion and harder to evaluate automatically than factual questions
due to their subjective nature. We will aim to provide more guides and
case-studies for which methods are appropriate in a given scenario.
Standard Metrics
Against annotated datasets, whether your own data or an academic
benchmark, there are a number of standard metrics that it helps to be
aware of:
1. **Exact Match (EM):** The percentage of queries that are answered
   exactly correctly.
2. **F1:** The percentage of queries that are answered exactly
   correctly or with a small edit distance (e.g. 1-2 words).
3. **Recall:** The percentage of queries that are answered correctly,
   regardless of the number of answers returned.
4. **Precision:** The percentage of queries that are answered
   correctly, divided by the number of answers returned.
This towardsdatascience article covers more technical metrics like
NDCG, MAP and MRR in greater depth.
Case Studies and Resources
1. (Course) Data-Centric AI (MIT), 2023
2. Scale's Approach to LLM Testing and Evaluation
3. LLM Patterns by Eugene Yan