HoneyHive LlamaIndex Tracer

HoneyHive is a platform that helps developers monitor, evaluate and
continuously improve their LLM-powered applications.
The "HoneyHiveLlamaIndexTracer" is integrated with HoneyHive to help
developers debug and analyze the execution flow of your LLM pipeline,
or to let developers customize feedback on specific trace events to
create evaluation or fine-tuning datasets from production.
   import os
   from getpass import getpass
   if os.getenv("OPENAI_API_KEY") is None:
       os.environ["OPENAI_API_KEY"] = getpass(
           "Paste your OpenAI key from: https://platform.openai.com/account/api-keys\n"
       )
   assert os.getenv("OPENAI_API_KEY", "").startswith(
       "sk-"
   ), "This doesn't look like a valid OpenAI API key"
   print("OpenAI API key configured")
   OpenAI API key configured
   import os
   from getpass import getpass
   if os.getenv("HONEYHIVE_API_KEY") is None:
       os.environ["HONEYHIVE_API_KEY"] = getpass(
           "Paste your HoneyHive key from: https://app.honeyhive.ai/settings/account\n"
       )
   print("HoneyHive API key configured")
   HoneyHive API key configured
   from llama_index.callbacks import CallbackManager, CBEventType
   from llama_index.callbacks import LlamaDebugHandler, WandbCallbackHandler
   from llama_index import (
       SummaryIndex,
       GPTTreeIndex,
       GPTVectorStoreIndex,
       ServiceContext,
       SimpleDirectoryReader,
       LLMPredictor,
       GPTSimpleKeywordTableIndex,
       StorageContext,
   )
   from llama_index.indices.composability import ComposableGraph
   from llama_index import load_index_from_storage, load_graph_from_storage
   from llama_index.llms import OpenAI
   from honeyhive.sdk.llamaindex_tracer import HoneyHiveLlamaIndexTracer
Setup LLM
   llm = OpenAI(model="gpt-4", temperature=0)
HoneyHive Callback Manager Setup
**Option 1**: Set Global Evaluation Handler
   from llama_index import set_global_handler
   set_global_handler(
       "honeyhive",
       project="My LlamaIndex Project",
       name="My LlamaIndex Pipeline",
       api_key=os.environ["HONEYHIVE_API_KEY"],
   )
   hh_tracer = llama_index.global_handler
   service_context = ServiceContext.from_defaults(llm=llm)
**Option 2**: Manually Configure Callback Handler
Also configure a debugger handler for extra notebook visibility.
   llama_debug = LlamaDebugHandler(print_trace_on_end=True)
   hh_tracer = HoneyHiveLlamaIndexTracer(
       project="My LlamaIndex Project",
       name="My LlamaIndex Pipeline",
       api_key=os.environ["HONEYHIVE_API_KEY"],
   )
   callback_manager = CallbackManager([llama_debug, hh_tracer])
   service_context = ServiceContext.from_defaults(
       callback_manager=callback_manager, llm=llm
   )
1. Indexing
   docs = SimpleDirectoryReader("../data/paul_graham/").load_data()
   index = GPTVectorStoreIndex.from_documents(docs, service_context=service_context)
   **********
   Trace: index_construction
       |_node_parsing ->  0.080298 seconds
         |_chunking ->  0.078948 seconds
       |_embedding ->  1.117244 seconds
       |_embedding ->  0.382624 seconds
   **********
2. Query Over Index
   query_engine = index.as_query_engine()
   response = query_engine.query("What did the author do growing up?")
   print(response, sep="\n")
   **********
   Trace: query
       |_query ->  11.334982 seconds

HoneyHive LlamaIndex Tracer

         |_retrieve ->  0.255016 seconds
           |_embedding ->  0.247083 seconds
         |_synthesize ->  11.079581 seconds
           |_templating ->  5.7e-05 seconds
           |_llm ->  11.065533 seconds
   **********
   Growing up, the author was involved in writing and programming. They wrote short stories and tried their hand at programming on an IBM 1401, using an early version of Fortran. Later, they started programming on a TRS-80 microcomputer that their father bought, creating simple games, a program to predict the flight of their model rockets, and a word processor. Despite their interest in programming, they initially planned to study philosophy in college, but eventually switched to AI.
3. Build Complex Indices
   # fetch "New York City" page from Wikipedia
   from pathlib import Path
   import requests
   response = requests.get(
       "https://en.wikipedia.org/w/api.php",
       params={
           "action": "query",
           "format": "json",
           "titles": "New York City",
           "prop": "extracts",
           "explaintext": True,
       },
   ).json()
   page = next(iter(response["query"]["pages"].values()))
   nyc_text = page["extract"]
   data_path = Path("data")
   if not data_path.exists():
       Path.mkdir(data_path)
   with open("data/nyc_text.txt", "w") as fp:
       fp.write(nyc_text)
   # load NYC dataset
   nyc_documents = SimpleDirectoryReader("data/").load_data()
   # load PG's essay
   essay_documents = SimpleDirectoryReader("../data/paul_graham").load_data()
   # While building a composable index, to correctly save the index,
   # the same `storage_context` needs to be passed to every index.
   storage_context = StorageContext.from_defaults()
   # build NYC index
   nyc_index = GPTVectorStoreIndex.from_documents(
       nyc_documents, service_context=service_context, storage_context=storage_context
   )
   **********
   Trace: index_construction
       |_node_parsing ->  0.069026 seconds
         |_chunking ->  0.066652 seconds
       |_embedding ->  1.216197 seconds
       |_embedding ->  0.413493 seconds
       |_embedding ->  0.405327 seconds
       |_embedding ->  0.191452 seconds
   **********
   # build essay index
   essay_index = GPTVectorStoreIndex.from_documents(
       essay_documents, service_context=service_context, storage_context=storage_context
   )
   **********
   Trace: index_construction
       |_node_parsing ->  0.09018 seconds
         |_chunking ->  0.088916 seconds
       |_embedding ->  0.403542 seconds
       |_embedding ->  0.378775 seconds
   **********
3.1. Query Over Graph Index
   nyc_index_summary = """
       New York, often called New York City or NYC,
       is the most populous city in the United States.
       With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2),
       New York City is also the most densely populated major city in the United States,
       and is more than twice as populous as second-place Los Angeles.
       New York City lies at the southern tip of New York State, and
       constitutes the geographical and demographic center of both the
       Northeast megalopolis and the New York metropolitan area, the
       largest metropolitan area in the world by urban landmass.[8] With over
       20.1 million people in its metropolitan statistical area and 23.5 million

HoneyHive LlamaIndex Tracer

       in its combined statistical area as of 2020, New York is one of the world's
       most populous megacities, and over 58 million people live within 250 mi (400 km) of
       the city. New York City is a global cultural, financial, and media center with
       a significant influence on commerce, health care and life sciences, entertainment,
       research, technology, education, politics, tourism, dining, art, fashion, and sports.
       Home to the headquarters of the United Nations,
       New York is an important center for international diplomacy,
       an established safe haven for global investors, and is sometimes described as the capital of the world.
   """
   essay_index_summary = """
       Author: Paul Graham.
       The author grew up painting and writing essays.
       He wrote a book on Lisp and did freelance Lisp hacking work to support himself.
       He also became the de facto studio assistant for Idelle Weber, an early photorealist painter.
       He eventually had the idea to start a company to put art galleries online, but the idea was unsuccessful.
       He then had the idea to write software to build online stores, which became the basis for his successful company, Viaweb.
       After Viaweb was acquired by Yahoo!, the author returned to painting and started writing essays online.
       He wrote a book of essays, Hackers & Painters, and worked on spam filters.
       He also bought a building in Cambridge to use as an office.
       He then had the idea to start Y Combinator, an investment firm that would
       make a larger number of smaller investments and help founders remain as CEO.
       He and his partner Jessica Livingston ran Y Combinator and funded a batch of startups twice a year.
       He also continued to write essays, cook for groups of friends, and explore the concept of invented vs discovered in software.
   """
   from llama_index import StorageContext, load_graph_from_storage
   graph = ComposableGraph.from_indices(
       GPTSimpleKeywordTableIndex,
       [nyc_index, essay_index],
       index_summaries=[nyc_index_summary, essay_index_summary],
       max_keywords_per_chunk=50,
       service_context=service_context,
       storage_context=storage_context,
   )
   **********
   Trace: graph_construction
   **********
3.2 Query
   query_engine = graph.as_query_engine()
   response = query_engine.query(
       "What is the climate of New York City like? How cold is it during the winter?",
   )
   print(response, sep="\n")
   **********
   Trace: query
       |_query ->  28.480834 seconds
         |_retrieve ->  0.002333 seconds
         |_query ->  15.367174 seconds
           |_retrieve ->  0.171675 seconds
             |_embedding ->  0.162042 seconds
           |_synthesize ->  15.194969 seconds
             |_templating ->  4.8e-05 seconds
             |_llm ->  15.179017 seconds
         |_synthesize ->  13.110327 seconds
           |_templating ->  8.2e-05 seconds
           |_llm ->  13.103851 seconds
   **********
   New York City has a humid subtropical climate, which makes it unique as the northernmost major city in North America with this type of climate. The city enjoys an average of 234 days of sunshine each year. During winter, the city is chilly and damp, with influences from the Atlantic Ocean and the Appalachian Mountains helping to keep it warmer than other inland cities at similar latitudes. The average daily temperature in January, which is the coldest month, is 33.3 °F (0.7 °C). However, temperatures can fluctuate significantly, dropping to 10 °F (−12 °C) on some days, and reaching up to 60 °F (16 °C) on others, even in the coldest winter month.

HoneyHive LlamaIndex Tracer

View HoneyHive Traces
When we are done tracing our events we can view them via the HoneyHive
platform. Simply login to HoneyHive, go to your "My LlamaIndex
Project" project, click the "Data Store" tab and view your "Sessions".


Aim Callback

Aim is an easy-to-use & supercharged open-source AI metadata tracker
it logs all your AI metadata (experiments, prompts, etc) enables a UI
to compare & observe them and SDK to query them programmatically. For
more please see the Github page.
In this demo, we show the capabilities of Aim for logging events while
running queries within LlamaIndex. We use the AimCallback to store the
outputs and showing how to explore them using Aim Text Explorer.
**NOTE**: This is a beta feature. The usage within different classes
and the API interface for the CallbackManager and AimCallback may
change!
Setup
   from llama_index.callbacks import CallbackManager, AimCallback
   from llama_index import SummaryIndex, ServiceContext, SimpleDirectoryReader
Let's read the documents using "SimpleDirectoryReader" from
'examples/data/paul_graham'.
   docs = SimpleDirectoryReader("../../data/paul_graham").load_data()
Now lets initialize an AimCallback instance, and add it to the list of
callback managers.
   aim_callback = AimCallback(repo="./")
   callback_manager = CallbackManager([aim_callback])
In this snippet, we initialize a service context by providing the
callback manager. Next, we create an instance of "SummaryIndex" class,
by passing in the document reader and the service context. After which
we create a query engine which we will use to run queries on the index
and retrieve relevant results.
   service_context = ServiceContext.from_defaults(callback_manager=callback_manager)
   index = SummaryIndex.from_documents(docs, service_context=service_context)
   query_engine = index.as_query_engine()
Finally let's ask a question to the LM based on our provided document
   response = query_engine.query("What did the author do growing up?")
The callback manager will log the "CBEventType.LLM" type of events as
an Aim.Text, and we can explore the LM given prompt and the output in
the Text Explorer. By first doing "aim up" and navigating by the given
url.


Wandb Callback Handler

Weights & Biases Prompts is a suite of LLMOps tools built for the
development of LLM-powered applications.
The "WandbCallbackHandler" is integrated with W&B Prompts to visualize
and inspect the execution flow of your index construction, or querying
over your index and more. You can use this handler to persist your
created indices as W&B Artifacts allowing you to version control your
indices.
   import os
   from getpass import getpass
   if os.getenv("OPENAI_API_KEY") is None:
       os.environ["OPENAI_API_KEY"] = getpass(
           "Paste your OpenAI key from: https://platform.openai.com/account/api-keys\n"
       )
   assert os.getenv("OPENAI_API_KEY", "").startswith(
       "sk-"
   ), "This doesn't look like a valid OpenAI API key"
   print("OpenAI API key configured")
   OpenAI API key configured
   from llama_index.callbacks import CallbackManager, CBEventType
   from llama_index.callbacks import LlamaDebugHandler, WandbCallbackHandler
   from llama_index import (
       SummaryIndex,
       GPTTreeIndex,
       GPTVectorStoreIndex,
       ServiceContext,
       SimpleDirectoryReader,
       LLMPredictor,
       GPTSimpleKeywordTableIndex,
       StorageContext,
   )
   from llama_index.indices.composability import ComposableGraph
   from llama_index import load_index_from_storage, load_graph_from_storage
   from llama_index.llms import OpenAI
Setup LLM
   llm = OpenAI(model="gpt-4", temperature=0)
W&B Callback Manager Setup
**Option 1**: Set Global Evaluation Handler
   from llama_index import set_global_handler
   set_global_handler("wandb", run_args={"project": "llamaindex"})
   wandb_callback = llama_index.global_handler
   service_context = ServiceContext.from_defaults(llm=llm)
**Option 2**: Manually Configure Callback Handler
Also configure a debugger handler for extra notebook visibility.
   llama_debug = LlamaDebugHandler(print_trace_on_end=True)
   # wandb.init args
   run_args = dict(
       project="llamaindex",
   )
   wandb_callback = WandbCallbackHandler(run_args=run_args)
   callback_manager = CallbackManager([llama_debug, wandb_callback])
   service_context = ServiceContext.from_defaults(
       callback_manager=callback_manager, llm=llm
   )
   After running the above cell, you will get the W&B run page URL.
   Here you will find a trace table with all the events tracked using
   Weights and Biases' Prompts feature.
1. Indexing
   docs = SimpleDirectoryReader("../data/paul_graham/").load_data()
   index = GPTVectorStoreIndex.from_documents(docs, service_context=service_context)
   **********
   Trace: index_construction
       |_node_parsing ->  0.295179 seconds
         |_chunking ->  0.293976 seconds
       |_embedding ->  0.494492 seconds
       |_embedding ->  0.346162 seconds
   **********
   wandb: Logged trace tree to W&B.
1.1 Persist Index as W&B Artifacts
   wandb_callback.persist_index(index, index_name="simple_vector_store")
   wandb: Adding directory to artifact (/Users/loganmarkewich/llama_index/docs/examples/callbacks/wandb/run-20230801_152955-ds93prxa/files/storage)... Done. 0.0s
1.2 Download Index from W&B Artifacts
   storage_context = wandb_callback.load_storage_context(
       artifact_url="ayut/llamaindex/simple_vector_store:v0"

Wandb Callback Handler

   )
   # Load the index and initialize a query engine
   index = load_index_from_storage(storage_context, service_context=service_context)
   wandb:   3 of 3 files downloaded.
   **********
   Trace: index_construction
   **********
2. Query Over Index
   query_engine = index.as_query_engine()
   response = query_engine.query("What did the author do growing up?")
   print(response, sep="\n")
   **********
   Trace: query
       |_query ->  2.695958 seconds
         |_retrieve ->  0.806379 seconds
           |_embedding ->  0.802871 seconds
         |_synthesize ->  1.8893 seconds
           |_llm ->  1.842434 seconds
   **********
   wandb: Logged trace tree to W&B.
   The text does not provide information on what the author did growing up.
3. Build Complex Indices
   # fetch "New York City" page from Wikipedia
   from pathlib import Path
   import requests
   response = requests.get(
       "https://en.wikipedia.org/w/api.php",
       params={
           "action": "query",
           "format": "json",
           "titles": "New York City",
           "prop": "extracts",
           "explaintext": True,
       },
   ).json()
   page = next(iter(response["query"]["pages"].values()))
   nyc_text = page["extract"]
   data_path = Path("data")
   if not data_path.exists():
       Path.mkdir(data_path)
   with open("data/nyc_text.txt", "w") as fp:
       fp.write(nyc_text)
   # load NYC dataset
   nyc_documents = SimpleDirectoryReader("data/").load_data()
   # load PG's essay
   essay_documents = SimpleDirectoryReader("../data/paul_graham").load_data()
   # While building a composable index, to correctly save the index,
   # the same `storage_context` needs to be passed to every index.
   storage_context = StorageContext.from_defaults()
   # build NYC index
   nyc_index = GPTVectorStoreIndex.from_documents(
       nyc_documents, service_context=service_context, storage_context=storage_context
   )
   **********
   Trace: index_construction
       |_node_parsing ->  0.491078 seconds
         |_chunking ->  0.48921 seconds
       |_embedding ->  0.314621 seconds
       |_embedding ->  0.65393 seconds
       |_embedding ->  0.452587 seconds
       |_embedding ->  0.510454 seconds
   **********
   wandb: Logged trace tree to W&B.
   # build essay index
   essay_index = GPTVectorStoreIndex.from_documents(
       essay_documents, service_context=service_context, storage_context=storage_context
   )
   **********
   Trace: index_construction
       |_node_parsing ->  0.340749 seconds
         |_chunking ->  0.339598 seconds
       |_embedding ->  0.280761 seconds
       |_embedding ->  0.315542 seconds
   **********
   wandb: Logged trace tree to W&B.
3.1. Query Over Graph Index
   nyc_index_summary = """
       New York, often called New York City or NYC,
       is the most populous city in the United States.
       With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2),
       New York City is also the most densely populated major city in the United States,

Wandb Callback Handler

       and is more than twice as populous as second-place Los Angeles.
       New York City lies at the southern tip of New York State, and
       constitutes the geographical and demographic center of both the
       Northeast megalopolis and the New York metropolitan area, the
       largest metropolitan area in the world by urban landmass.[8] With over
       20.1 million people in its metropolitan statistical area and 23.5 million
       in its combined statistical area as of 2020, New York is one of the world's
       most populous megacities, and over 58 million people live within 250 mi (400 km) of
       the city. New York City is a global cultural, financial, and media center with
       a significant influence on commerce, health care and life sciences, entertainment,
       research, technology, education, politics, tourism, dining, art, fashion, and sports.
       Home to the headquarters of the United Nations,
       New York is an important center for international diplomacy,
       an established safe haven for global investors, and is sometimes described as the capital of the world.
   """
   essay_index_summary = """
       Author: Paul Graham.
       The author grew up painting and writing essays.
       He wrote a book on Lisp and did freelance Lisp hacking work to support himself.
       He also became the de facto studio assistant for Idelle Weber, an early photorealist painter.
       He eventually had the idea to start a company to put art galleries online, but the idea was unsuccessful.
       He then had the idea to write software to build online stores, which became the basis for his successful company, Viaweb.
       After Viaweb was acquired by Yahoo!, the author returned to painting and started writing essays online.
       He wrote a book of essays, Hackers & Painters, and worked on spam filters.
       He also bought a building in Cambridge to use as an office.
       He then had the idea to start Y Combinator, an investment firm that would
       make a larger number of smaller investments and help founders remain as CEO.
       He and his partner Jessica Livingston ran Y Combinator and funded a batch of startups twice a year.
       He also continued to write essays, cook for groups of friends, and explore the concept of invented vs discovered in software.
   """
   from llama_index import StorageContext, load_graph_from_storage
   graph = ComposableGraph.from_indices(
       GPTSimpleKeywordTableIndex,
       [nyc_index, essay_index],
       index_summaries=[nyc_index_summary, essay_index_summary],
       max_keywords_per_chunk=50,
       service_context=service_context,
       storage_context=storage_context,
   )
   **********
   Trace: graph_construction
   **********
3.1.1 Persist Composable Index as W&B Artifacts
   wandb_callback.persist_index(graph, index_name="composable_graph")
   wandb: Adding directory to artifact (/Users/ayushthakur/integrations/llamaindex/llama_index/docs/examples/callbacks/wandb/run-20230607_012558-js7j48l9/files/storage)... Done. 0.0s
3.1.2 Download Index from W&B Artifacts
   storage_context = wandb_callback.load_storage_context(
       artifact_url="ayut/llamaindex/composable_graph:v0"
   )
   # Load the graph and initialize a query engine
   graph = load_graph_from_storage(
       storage_context, root_id=graph.root_id, service_context=service_context
   )
   query_engine = index.as_query_engine()
   wandb:   3 of 3 files downloaded.
   **********

Wandb Callback Handler

   Trace: index_construction
   **********
   **********
   Trace: index_construction
   **********
   **********
   Trace: index_construction
   **********
3.1.3 Query
   query_engine = graph.as_query_engine()
   response = query_engine.query(
       "What is the climate of New York City like? How cold is it during the winter?",
   )
   print(response, sep="\n")
   **********
   Trace: query
       |_query ->  58.207419 seconds
         |_retrieve ->  2.672269 seconds
           |_llm ->  2.671922 seconds
         |_query ->  39.630366 seconds
           |_retrieve ->  0.165883 seconds
             |_embedding ->  0.158699 seconds
           |_synthesize ->  39.46435 seconds
             |_llm ->  39.410054 seconds
         |_synthesize ->  15.904373 seconds
           |_llm ->  15.900012 seconds
   **********
   wandb: Logged trace tree to W&B.
   New York City has a humid subtropical climate, making it the northernmost major city in North America with this type of climate. During the winter, the city is chilly and damp. The average daily temperature in January, the coldest month, is 33.3 °F (0.7 °C). Temperatures can drop to 10 °F (−12 °C) several times each winter, but can also reach 60 °F (16 °C) for several days even in the coldest winter month. The city also experiences the urban heat island effect, which can increase nighttime temperatures. The most extreme temperatures have ranged from −15 °F (−26 °C) to 106 °F (41 °C).
Close W&B Callback Handler
When we are done tracking our events we can close the wandb run.
   wandb_callback.finish()


Guardrails Output Parsing

Load documents, build the VectorStoreIndex
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   from IPython.display import Markdown, display
   # load documents
   documents = SimpleDirectoryReader("../paul_graham_essay/data").load_data()
   index = VectorStoreIndex.from_documents(documents, chunk_size=512)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_documents] Total LLM token usage: 0 tokens
   > [build_index_from_documents] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_documents] Total embedding token usage: 18579 tokens
   > [build_index_from_documents] Total embedding token usage: 18579 tokens
Define Query + Guardrails Spec
   from llama_index.output_parsers import GuardrailsOutputParser
   from llama_index.llm_predictor import StructuredLLMPredictor
   llm_predictor = StructuredLLMPredictor()
**Define custom QA and Refine Prompts**
   from llama_index.prompts import PromptTemplate
   from llama_index.prompts.default_prompts import (
       DEFAULT_TEXT_QA_PROMPT_TMPL,
       DEFAULT_REFINE_PROMPT_TMPL,
   )
   # NOTE: we don't need to define the query_str in the rail spec, we can define during query-time.
   rail_spec = """
   <rail version="0.1">
   <output>
       <list name="points" description="Bullet points regarding events in the author's life.">
           <object>
               <string name="explanation" format="one-line" on-fail-one-line="noop" />
               <string name="explanation2" format="one-line" on-fail-one-line="noop" />
               <string name="explanation3" format="one-line" on-fail-one-line="noop" />
           </object>
       </list>
   </output>
   <prompt>
   Query string here.
   @xml_prefix_prompt
   {output_schema}
   @json_suffix_prompt_v2_wo_none
   </prompt>
   </rail>
   """
   output_parser = GuardrailsOutputParser.from_rail_string(
       rail_spec, llm=llm_predictor.llm
   )
   # NOTE: we use the same output parser for both prompts, though you can choose to use different parsers
   # NOTE: here we add formatting instructions to the prompts.
   fmt_qa_tmpl = output_parser.format(DEFAULT_TEXT_QA_PROMPT_TMPL)
   fmt_refine_tmpl = output_parser.format(DEFAULT_REFINE_PROMPT_TMPL)
   qa_prompt = PromptTemplate(fmt_qa_tmpl, output_parser=output_parser)
   refine_prompt = PromptTemplate(fmt_refine_tmpl, output_parser=output_parser)
   # take a look at the new QA template!
   print(fmt_qa_tmpl)
   Context information is below.
   ---------------------
   {context_str}
   ---------------------
   Given the context information and not prior knowledge, answer the question: {query_str}
   Given below is XML that describes the information to extract from this document and the tags to extract it into.
   <output>
       <list name="points" description="Bullet points regarding events in the author's life.">
           <object>
               <string name="explanation" format="one-line"/>
               <string name="explanation2" format="one-line"/>
               <string name="explanation3" format="one-line"/>
           </object>
       </list>
   </output>
   ONLY return a valid JSON object (no other text is necessary). The JSON MUST conform to the XML format, including any types and format requests e.g. requests for lists, objects and specific types. Be correct and concise.

Guardrails Output Parsing

   JSON Output:
Query Index
   query_engine = index.as_query_engine(
       text_qa_template=qa_prompt,
       refine_template=refine_prompt,
       llm_predictor=llm_predictor,
   )
   response = query_engine.query(
       "What are the three items the author did growing up?",
   )
   INFO:llama_index.token_counter.token_counter:> [query] Total LLM token usage: 754 tokens
   > [query] Total LLM token usage: 754 tokens
   INFO:llama_index.token_counter.token_counter:> [query] Total embedding token usage: 11 tokens
   > [query] Total embedding token usage: 11 tokens
   print(response)
   {'points': [{'explanation': 'Writing short stories', 'explanation2': 'Programming on an IBM 1401', 'explanation3': 'Using microcomputers'}]}


DataFrame Structured Data Extraction

This demo shows how you can extract tabular DataFrames from raw text.
This was directly inspired by jxnl's dataframe example here: https://
github.com/jxnl/openai_function_call/blob/main/auto_dataframe.py.
We show this with different levels of complexity, all backed by the
OpenAI Function API:
* (more code) How to build an extractor yourself using our
  OpenAIPydanticProgram
* (less code) Using our out-of-the-box "DFFullProgram" and
  "DFRowsProgram" objects
Build a DF Extractor Yourself (Using OpenAIPydanticProgram)
Our OpenAIPydanticProgram is a wrapper around an OpenAI LLM that
supports function calling - it will return structured outputs in the
form of a Pydantic object.
We import our "DataFrame" and "DataFrameRowsOnly" objects.
To create an output extractor, you just need to 1) specify the
relevant Pydantic object, and 2) Add the right prompt
   from llama_index.program import (
       OpenAIPydanticProgram,
       DFFullProgram,
       DataFrame,
       DataFrameRowsOnly,
   )
   from llama_index.llms import OpenAI
   program = OpenAIPydanticProgram.from_defaults(
       output_cls=DataFrame,
       llm=OpenAI(temperature=0, model="gpt-4-0613"),
       prompt_template_str=(
           "Please extract the following query into a structured data according to: {input_str}."
           "Please extract both the set of column names and a set of rows."
       ),
       verbose=True,
   )
   # NOTE: the test example is taken from jxnl's repo
   response_obj = program(
       input_str="""My name is John and I am 25 years old. I live in
           New York and I like to play basketball. His name is
           Mike and he is 30 years old. He lives in San Francisco
           and he likes to play baseball. Sarah is 20 years old
           and she lives in Los Angeles. She likes to play tennis.
           Her name is Mary and she is 35 years old.
           She lives in Chicago."""
   )
   response_obj
   Function call: DataFrame with args: {
     "columns": [
       {
         "column_name": "Name",
         "column_desc": "Name of the person"
       },
       {
         "column_name": "Age",
         "column_desc": "Age of the person"
       },
       {
         "column_name": "City",
         "column_desc": "City where the person lives"
       },
       {
         "column_name": "Hobby",
         "column_desc": "What the person likes to do"
       }
     ],
     "rows": [
       {
         "row_values": ["John", 25, "New York", "play basketball"]
       },
       {
         "row_values": ["Mike", 30, "San Francisco", "play baseball"]
       },
       {
         "row_values": ["Sarah", 20, "Los Angeles", "play tennis"]
       },
       {
         "row_values": ["Mary", 35, "Chicago", "play tennis"]
       }
     ]
   }
   DataFrame(description=None, columns=[DataFrameColumn(column_name='Name', column_desc='Name of the person'), DataFrameColumn(column_name='Age', column_desc='Age of the person'), DataFrameColumn(column_name='City', column_desc='City where the person lives'), DataFrameColumn(column_name='Hobby', column_desc='What the person likes to do')], rows=[DataFrameRow(row_values=['John', 25, 'New York', 'play basketball']), DataFrameRow(row_values=['Mike', 30, 'San Francisco', 'play baseball']), DataFrameRow(row_values=['Sarah', 20, 'Los Angeles', 'play tennis']), DataFrameRow(row_values=['Mary', 35, 'Chicago', 'play tennis'])])

DataFrame Structured Data Extraction

   program = OpenAIPydanticProgram.from_defaults(
       output_cls=DataFrameRowsOnly,
       llm=OpenAI(temperature=0, model="gpt-4-0613"),
       prompt_template_str=(
           "Please extract the following text into a structured data: {input_str}. "
           "The column names are the following: ['Name', 'Age', 'City', 'Favorite Sport']. "
           "Do not specify additional parameters that are not in the function schema. "
       ),
       verbose=True,
   )
   program(
       input_str="""My name is John and I am 25 years old. I live in
           New York and I like to play basketball. His name is
           Mike and he is 30 years old. He lives in San Francisco
           and he likes to play baseball. Sarah is 20 years old
           and she lives in Los Angeles. She likes to play tennis.
           Her name is Mary and she is 35 years old.
           She lives in Chicago."""
   )
   Function call: DataFrameRowsOnly with args: {
     "rows": [
       {
         "row_values": ["John", 25, "New York", "basketball"]
       },
       {
         "row_values": ["Mike", 30, "San Francisco", "baseball"]
       },
       {
         "row_values": ["Sarah", 20, "Los Angeles", "tennis"]
       },
       {
         "row_values": ["Mary", 35, "Chicago", ""]
       }
     ]
   }
   DataFrameRowsOnly(rows=[DataFrameRow(row_values=['John', 25, 'New York', 'basketball']), DataFrameRow(row_values=['Mike', 30, 'San Francisco', 'baseball']), DataFrameRow(row_values=['Sarah', 20, 'Los Angeles', 'tennis']), DataFrameRow(row_values=['Mary', 35, 'Chicago', ''])])
Use our DataFrame Programs
We provide convenience wrappers for "DFFullProgram" and
"DFRowsProgram". This allows a simpler object creation interface than
specifying all details through the "OpenAIPydanticProgram".
   from llama_index.program import OpenAIPydanticProgram, DFFullProgram, DFRowsProgram
   import pandas as pd
   # initialize empty df
   df = pd.DataFrame(
       {
           "Name": pd.Series(dtype="str"),
           "Age": pd.Series(dtype="int"),
           "City": pd.Series(dtype="str"),
           "Favorite Sport": pd.Series(dtype="str"),
       }
   )
   # initialize program, using existing df as schema
   df_rows_program = DFRowsProgram.from_defaults(
       pydantic_program_cls=OpenAIPydanticProgram, df=df
   )
   # parse text, using existing df as schema
   result_obj = df_rows_program(
       input_str="""My name is John and I am 25 years old. I live in
           New York and I like to play basketball. His name is
           Mike and he is 30 years old. He lives in San Francisco
           and he likes to play baseball. Sarah is 20 years old
           and she lives in Los Angeles. She likes to play tennis.
           Her name is Mary and she is 35 years old.
           She lives in Chicago."""
   )
   result_obj.to_df(existing_df=df)
   /Users/jerryliu/Programming/gpt_index/llama_index/program/predefined/df.py:65: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
     return existing_df.append(new_df, ignore_index=True)
       Name  Age           City Favorite Sport
   0   John   25       New York     Basketball
   1   Mike   30  San Francisco       Baseball

DataFrame Structured Data Extraction

   2  Sarah   20    Los Angeles         Tennis
   3   Mary   35        Chicago
   # initialize program that can do joint schema extraction and structured data extraction
   df_full_program = DFFullProgram.from_defaults(
       pydantic_program_cls=OpenAIPydanticProgram,
   )
   result_obj = df_full_program(
       input_str="""My name is John and I am 25 years old. I live in
           New York and I like to play basketball. His name is
           Mike and he is 30 years old. He lives in San Francisco
           and he likes to play baseball. Sarah is 20 years old
           and she lives in Los Angeles. She likes to play tennis.
           Her name is Mary and she is 35 years old.
           She lives in Chicago."""
   )
   result_obj.to_df()
       Name  Age       Location       Hobby
   0   John   25       New York  Basketball
   1   Mike   30  San Francisco    Baseball
   2  Sarah   20    Los Angeles      Tennis
   3   Mary   35        Chicago
   # initialize empty df
   df = pd.DataFrame(
       {
           "City": pd.Series(dtype="str"),
           "State": pd.Series(dtype="str"),
           "Population": pd.Series(dtype="int"),
       }
   )
   # initialize program, using existing df as schema
   df_rows_program = DFRowsProgram.from_defaults(
       pydantic_program_cls=OpenAIPydanticProgram, df=df
   )
   input_text = """San Francisco is in California, has a population of 800,000.
   New York City is the most populous city in the United States. \
   With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2), \
   New York City is the most densely populated major city in the United States.
   New York City is in New York State.
   Boston (US: /ˈbɔːstən/),[8] officially the City of Boston, is the capital and largest city of the Commonwealth of Massachusetts \
   and the cultural and financial center of the New England region of the Northeastern United States. \
   The city boundaries encompass an area of about 48.4 sq mi (125 km2)[9] and a population of 675,647 as of 2020.[4]
   """
   # parse text, using existing df as schema
   result_obj = df_rows_program(input_str=input_text)
   new_df = result_obj.to_df(existing_df=df)
   new_df
   /Users/jerryliu/Programming/gpt_index/llama_index/program/predefined/df.py:65: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
     return existing_df.append(new_df, ignore_index=True)
               City          State  Population
   0  San Francisco     California      800000
   1  New York City       New York     8804190
   2         Boston  Massachusetts      675647


Langchain Output Parsing

Load documents, build the VectorStoreIndex
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   from IPython.display import Markdown, display
   # load documents
   documents = SimpleDirectoryReader("../paul_graham_essay/data").load_data()
   index = VectorStoreIndex.from_documents(documents, chunk_size=512)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_documents] Total LLM token usage: 0 tokens
   > [build_index_from_documents] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_documents] Total embedding token usage: 18579 tokens
   > [build_index_from_documents] Total embedding token usage: 18579 tokens
Define Query + Langchain Output Parser
   from llama_index.output_parsers import LangchainOutputParser
   from llama_index.llm_predictor import StructuredLLMPredictor
   from langchain.output_parsers import StructuredOutputParser, ResponseSchema
   llm_predictor = StructuredLLMPredictor()
**Define custom QA and Refine Prompts**
   from llama_index.prompts import PromptTemplate
   from llama_index.prompts.default_prompts import (
       DEFAULT_TEXT_QA_PROMPT_TMPL,
       DEFAULT_REFINE_PROMPT_TMPL,
   )
   response_schemas = [
       ResponseSchema(
           name="Education",
           description="Describes the author's educational experience/background.",
       ),
       ResponseSchema(
           name="Work", description="Describes the author's work experience/background."
       ),
   ]
   lc_output_parser = StructuredOutputParser.from_response_schemas(response_schemas)
   output_parser = LangchainOutputParser(lc_output_parser)
   # NOTE: we use the same output parser for both prompts, though you can choose to use different parsers
   # NOTE: here we add formatting instructions to the prompts.
   fmt_qa_tmpl = output_parser.format(DEFAULT_TEXT_QA_PROMPT_TMPL)
   fmt_refine_tmpl = output_parser.format(DEFAULT_REFINE_PROMPT_TMPL)
   qa_prompt = PromptTemplate(fmt_qa_tmpl, output_parser=output_parser)
   refine_prompt = PromptTemplate(fmt_refine_tmpl, output_parser=output_parser)
   # take a look at the new QA template!
   print(fmt_qa_tmpl)
   Context information is below.
   ---------------------
   {context_str}
   ---------------------
   Given the context information and not prior knowledge, answer the question: {query_str}
   The output should be a markdown code snippet formatted in the following schema:
   ```json
   {{
   	"Education": string  // Describes the author's educational experience/background.
   	"Work": string  // Describes the author's work experience/background.
   }}
   ```
Query Index
   query_engine = index.as_query_engine(
       text_qa_template=qa_prompt,
       refine_template=refine_prompt,
       llm_predictor=llm_predictor,
   )
   response = query_engine.query(
       "What are a few things the author did growing up?",
   )
   INFO:llama_index.token_counter.token_counter:> [query] Total LLM token usage: 609 tokens
   > [query] Total LLM token usage: 609 tokens
   INFO:llama_index.token_counter.token_counter:> [query] Total embedding token usage: 11 tokens
   > [query] Total embedding token usage: 11 tokens
   print(response)
   {'Education': 'Before college, the author wrote short stories and experimented with programming on an IBM 1401.', 'Work': 'The author worked on writing and programming outside of school.'}


Guidance Pydantic Program

Generate structured data with **guidance** via LlamaIndex.
With guidance, you can guarantee the output structure is correct by
*forcing* the LLM to output desired tokens.This is especialy helpful
when you are using lower-capacity model (e.g. the current open source
models), which otherwise would struggle to generate valid output that
fits the desired output schema.
   from pydantic import BaseModel
   from typing import List
   from guidance.llms import OpenAI
   from llama_index.program import GuidancePydanticProgram
Define output schema
   class Song(BaseModel):
       title: str
       length_seconds: int
   class Album(BaseModel):
       name: str
       artist: str
       songs: List[Song]
Define guidance pydantic program
   program = GuidancePydanticProgram(
       output_cls=Album,
       prompt_template_str="Generate an example album, with an artist and a list of songs. Using the movie {{movie_name}} as inspiration",
       guidance_llm=OpenAI("text-davinci-003"),
       verbose=True,
   )
Run program to get structured output.Text highlighted in blue is
variables specified by us, text highlighted in green is generated by
the LLM.
   output = program(movie_name="The Shining")
The output is a valid Pydantic object that we can then use to call
functions/APIs.
   output
   Album(name='The Shining', artist='Jack Torrance', songs=[Song(title='All Work and No Play', length_seconds=180), Song(title='The Overlook Hotel', length_seconds=240), Song(title='The Shining', length_seconds=210)])


OpenAI function calling for Sub-Question Query Engine

In this notebook, we showcase how to use OpenAI function calling to
improve the robustness of our sub-question query engine.
The sub-question query engine is designed to accept swappable question
generators that implement the "BaseQuestionGenerator" interface.To
leverage the power of openai function calling API, we implemented a
new "OpenAIQuestionGenerator" (powered by our "OpenAIPydanticProgram")
OpenAI Question Generator
Unlike the default "LLMQuestionGenerator" that supports generic LLMs
via the completion API, "OpenAIQuestionGenerator" only works with the
latest OpenAI models that supports the function calling API.
The benefit is that these models are fine-tuned to output JSON
objects, so we can worry less about output parsing issues.
   from llama_index.question_gen.openai_generator import OpenAIQuestionGenerator
   question_gen = OpenAIQuestionGenerator.from_defaults()
Let's test it out!
   from llama_index.tools import ToolMetadata
   from llama_index import QueryBundle
   tools = [
       ToolMetadata(
           name="march_22",
           description="Provides information about Uber quarterly financials ending March 2022",
       ),
       ToolMetadata(
           name="june_22",
           description="Provides information about Uber quarterly financials ending June 2022",
       ),
       ToolMetadata(
           name="sept_22",
           description="Provides information about Uber quarterly financials ending September 2022",
       ),
       ToolMetadata(
           name="sept_21",
           description="Provides information about Uber quarterly financials ending September 2022",
       ),
       ToolMetadata(
           name="june_21",
           description="Provides information about Uber quarterly financials ending June 2022",
       ),
       ToolMetadata(
           name="march_21",
           description="Provides information about Uber quarterly financials ending March 2022",
       ),
   ]
   sub_questions = question_gen.generate(
       tools=tools,
       query=QueryBundle(
           "Compare the fastest growing sectors for Uber in the first two quarters of 2022"
       ),
   )
   sub_questions
   [SubQuestion(sub_question='What were the fastest growing sectors for Uber in March 2022?', tool_name='march_22'),
    SubQuestion(sub_question='What were the fastest growing sectors for Uber in June 2022?', tool_name='june_22')]


Guidance for Sub-Question Query Engine

In this notebook, we showcase how to use **guidance** to improve the
robustness of our sub-question query engine.
The sub-question query engine is designed to accept swappable question
generators that implement the "BaseQuestionGenerator" interface.To
leverage the power of **guidance**, we implemented a new
"GuidanceQuestionGenerator" (powered by our "GuidancePydanticProgram")
Guidance Question Generator
Unlike the default "LLMQuestionGenerator", guidance guarantees that we
will get the desired structured output, and eliminate output parsing
errors.
   from llama_index.question_gen.guidance_generator import GuidanceQuestionGenerator
   from guidance.llms import OpenAI as GuidanceOpenAI
   question_gen = GuidanceQuestionGenerator.from_defaults(
       guidance_llm=GuidanceOpenAI("text-davinci-003"), verbose=False
   )
Let's test it out!
   from llama_index.tools import ToolMetadata
   from llama_index import QueryBundle
   tools = [
       ToolMetadata(
           name="lyft_10k",
           description="Provides information about Lyft financials for year 2021",
       ),
       ToolMetadata(
           name="uber_10k",
           description="Provides information about Uber financials for year 2021",
       ),
   ]
   sub_questions = question_gen.generate(
       tools=tools,
       query=QueryBundle("Compare and contrast Uber and Lyft financial in 2021"),
   )
   sub_questions
   [SubQuestion(sub_question='What is the revenue of Uber', tool_name='uber_10k'),
    SubQuestion(sub_question='What is the EBITDA of Uber', tool_name='uber_10k'),
    SubQuestion(sub_question='What is the net income of Uber', tool_name='uber_10k'),
    SubQuestion(sub_question='What is the revenue of Lyft', tool_name='lyft_10k'),
    SubQuestion(sub_question='What is the EBITDA of Lyft', tool_name='lyft_10k'),
    SubQuestion(sub_question='What is the net income of Lyft', tool_name='lyft_10k')]
Using Guidance Question Generator with Sub-Question Query Engine
Prepare data and base query engines
   from llama_index import (
       SimpleDirectoryReader,
       LLMPredictor,
       ServiceContext,
       VectorStoreIndex,
   )
   from llama_index.response.pprint_utils import pprint_response
   from llama_index.tools import QueryEngineTool, ToolMetadata
   from llama_index.query_engine import SubQuestionQueryEngine
   lyft_docs = SimpleDirectoryReader(input_files=["../data/10k/lyft_2021.pdf"]).load_data()
   uber_docs = SimpleDirectoryReader(input_files=["../data/10k/uber_2021.pdf"]).load_data()
   lyft_index = VectorStoreIndex.from_documents(lyft_docs)
   uber_index = VectorStoreIndex.from_documents(uber_docs)
   lyft_engine = lyft_index.as_query_engine(similarity_top_k=3)
   uber_engine = uber_index.as_query_engine(similarity_top_k=3)
Construct sub-question query engine and run some queries!
   query_engine_tools = [
       QueryEngineTool(
           query_engine=lyft_engine,
           metadata=ToolMetadata(
               name="lyft_10k",
               description="Provides information about Lyft financials for year 2021",
           ),
       ),
       QueryEngineTool(
           query_engine=uber_engine,
           metadata=ToolMetadata(
               name="uber_10k",
               description="Provides information about Uber financials for year 2021",
           ),
       ),
   ]
   s_engine = SubQuestionQueryEngine.from_defaults(
       question_gen=question_gen,  # use guidance based question_gen defined above
       query_engine_tools=query_engine_tools,
   )
   response = s_engine.query(
       "Compare and contrast the customer segments and geographies that grew the fastest"

Guidance for Sub-Question Query Engine

   )
   Generated 4 sub questions.
   [uber_10k] Q: What customer segments grew the fastest for Uber
   [uber_10k] A: in 2021?
   The customer segments that grew the fastest for Uber in 2021 were its Mobility Drivers, Couriers, Riders, and Eaters. These segments experienced growth due to the continued stay-at-home order demand related to COVID-19, as well as Uber's membership programs, such as Uber One, Uber Pass, Eats Pass, and Rides Pass. Additionally, Uber's marketplace-centric advertising helped to connect merchants and brands with its platform network, further driving growth.
   [uber_10k] Q: What geographies grew the fastest for Uber
   [uber_10k] A:
   Based on the context information, it appears that Uber experienced the most growth in large metropolitan areas, such as Chicago, Miami, New York City, Sao Paulo, and London. Additionally, Uber experienced growth in suburban and rural areas, as well as in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain.
   [lyft_10k] Q: What customer segments grew the fastest for Lyft
   [lyft_10k] A:
   The customer segments that grew the fastest for Lyft were ridesharing, light vehicles, and public transit. Ridesharing grew as Lyft was able to predict demand and proactively incentivize drivers to be available for rides in the right place at the right time. Light vehicles grew as users were looking for options that were more active, usually lower-priced, and often more efficient for short trips during heavy traffic. Public transit grew as Lyft integrated third-party public transit data into the Lyft App to offer users a robust view of transportation options around them.
   [lyft_10k] Q: What geographies grew the fastest for Lyft
   [lyft_10k] A:
   It is not possible to answer this question with the given context information.

   print(response)
   The customer segments that grew the fastest for Uber in 2021 were its Mobility Drivers, Couriers, Riders, and Eaters. These segments experienced growth due to the continued stay-at-home order demand related to COVID-19, as well as Uber's membership programs, such as Uber One, Uber Pass, Eats Pass, and Rides Pass. Additionally, Uber's marketplace-centric advertising helped to connect merchants and brands with its platform network, further driving growth. Uber experienced the most growth in large metropolitan areas, such as Chicago, Miami, New York City, Sao Paulo, and London. Additionally, Uber experienced growth in suburban and rural areas, as well as in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain.
   The customer segments that grew the fastest for Lyft were ridesharing, light vehicles, and public transit. Ridesharing grew as Lyft was able to predict demand and proactively incentivize drivers to be available for rides in the right place at the right time. Light vehicles grew as users were looking for options that were more active, usually lower-priced, and often more efficient for short trips during heavy traffic. Public transit grew as Lyft integrated third-party public transit data into the Lyft App to offer users a robust view of transportation options around them. It is not possible to answer the question of which geographies grew the fastest for Lyft with the given context information.

Guidance for Sub-Question Query Engine

In summary, Uber and Lyft both experienced growth in customer segments related to their respective services, such as Mobility Drivers, Couriers, Riders, and Eaters for Uber, and ridesharing, light vehicles, and public transit for Lyft. Uber experienced the most growth in large metropolitan areas, as well as in suburban and rural areas, and in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain. It is not possible to answer the question of which geographies grew the fastest for Lyft with the given context information.


OpenAI Pydantic Program

This guide shows you how to generate structured data with new OpenAI
API via LlamaIndex. The user just needs to specify a Pydantic object.
We demonstrate two settings:
* Extraction into an "Album" object (which can contain a list of Song
  objects)
* Extraction into a "DirectoryTree" object (which can contain
  recursive Node objects)
Extraction into "Album"
This is a simple example of parsing an output into an "Album" schema,
which can contain multiple songs.
   from pydantic import BaseModel
   from typing import List
   from llama_index.program import OpenAIPydanticProgram
Define output schema
   class Song(BaseModel):
       """Data model for a song."""
       title: str
       length_seconds: int
   class Album(BaseModel):
       """Data model for an album."""
       name: str
       artist: str
       songs: List[Song]
Define openai pydantic program
   prompt_template_str = """\
   Generate an example album, with an artist and a list of songs. \
   Using the movie {movie_name} as inspiration.\
   """
   program = OpenAIPydanticProgram.from_defaults(
       output_cls=Album,
       prompt_template_str=prompt_template_str,
       verbose=True,
   )
Run program to get structured output.
   output = program(movie_name="The Shining")
   Function call: Album with args: {
     "name": "The Shining",
     "artist": "Various Artists",
     "songs": [
       {
         "title": "Main Title",
         "length_seconds": 180
       },
       {
         "title": "Opening Credits",
         "length_seconds": 120
       },
       {
         "title": "The Overlook Hotel",
         "length_seconds": 240
       },
       {
         "title": "Redrum",
         "length_seconds": 150
       },
       {
         "title": "Here's Johnny",
         "length_seconds": 200
       }
     ]
   }
The output is a valid Pydantic object that we can then use to call
functions/APIs.
   output
   Album(name='The Shining', artist='Various Artists', songs=[Song(title='Main Title', length_seconds=180), Song(title='Opening Credits', length_seconds=120), Song(title='The Overlook Hotel', length_seconds=240), Song(title='Redrum', length_seconds=150), Song(title="Here's Johnny", length_seconds=200)])
Extraction into "Album" (Streaming)
We also support streaming a list of objects through our "stream_list"
function.
Full credits to this idea go to "openai_function_call" repo: https://
github.com/jxnl/openai_function_call/tree/main/examples/streaming_mul
titask
   prompt_template_str = "{input_str}"
   program = OpenAIPydanticProgram.from_defaults(
       output_cls=Album,
       prompt_template_str=prompt_template_str,
       verbose=False,
   )
   output = program.stream_list(input_str="make up 5 random albums")
   for obj in output:
       print(obj.json(indent=2))
   {
     "name": "The Journey",
     "artist": "Unknown",
     "songs": [
       {
         "title": "Lost in the Woods",
         "length_seconds": 240
       },
       {
         "title": "Endless Horizon",
         "length_seconds": 320
       },
       {
         "title": "Mystic Dreams",
         "length_seconds": 280
       }
     ]
   }
   {
     "name": "Electric Pulse",
     "artist": "Synthwave Master",
     "songs": [
       {
         "title": "Neon Nights",
         "length_seconds": 300
       },
       {
         "title": "Cyber City",

OpenAI Pydantic Program

         "length_seconds": 280
       },
       {
         "title": "Digital Dreams",
         "length_seconds": 320
       }
     ]
   }
   {
     "name": "Soulful Serenade",
     "artist": "Smooth Jazz Trio",
     "songs": [
       {
         "title": "Midnight Groove",
         "length_seconds": 280
       },
       {
         "title": "Saxophone Serenade",
         "length_seconds": 320
       },
       {
         "title": "Chill Vibes",
         "length_seconds": 240
       }
     ]
   }
   {
     "name": "Rock Revolution",
     "artist": "The Thunderbolts",
     "songs": [
       {
         "title": "High Voltage",
         "length_seconds": 320
       },
       {
         "title": "Guitar Shredder",
         "length_seconds": 280
       },
       {
         "title": "Rock Anthem",
         "length_seconds": 300
       }
     ]
   }
   {
     "name": "Pop Sensation",
     "artist": "The Starlets",
     "songs": [
       {
         "title": "Catchy Melody",
         "length_seconds": 240
       },
       {
         "title": "Dancefloor Hit",
         "length_seconds": 300
       },
       {
         "title": "Pop Princess",
         "length_seconds": 280
       }
     ]
   }
Extraction into "DirectoryTree" object
This is directly inspired by jxnl's awesome repo here:
https://github.com/jxnl/openai_function_call.
That repository shows how you can use OpenAI's function API to parse
recursive Pydantic objects. The main requirement is that you want to
"wrap" a recursive Pydantic object with a non-recursive one.
Here we show an example in a "directory" setting, where a
"DirectoryTree" object wraps recursive "Node" objects, to parse a file
structure.
   # NOTE: defining recursive objects in a notebook causes errors
   from directory import DirectoryTree, Node
   DirectoryTree.schema()
   {'title': 'DirectoryTree',
    'description': 'Container class representing a directory tree.\n\nArgs:\n    root (Node): The root node of the tree.',
    'type': 'object',
    'properties': {'root': {'title': 'Root',
      'description': 'Root folder of the directory tree',
      'allOf': [{'$ref': '#/definitions/Node'}]}},
    'required': ['root'],
    'definitions': {'NodeType': {'title': 'NodeType',
      'description': 'Enumeration representing the types of nodes in a filesystem.',
      'enum': ['file', 'folder'],
      'type': 'string'},
     'Node': {'title': 'Node',
      'description': 'Class representing a single node in a filesystem. Can be either a file or a folder.\nNote that a file cannot have children, but a folder can.\n\nArgs:\n    name (str): The name of the node.\n    children (List[Node]): The list of child nodes (if any).\n    node_type (NodeType): The type of the node, either a file or a folder.',
      'type': 'object',
      'properties': {'name': {'title': 'Name',
        'description': 'Name of the folder',
        'type': 'string'},
       'children': {'title': 'Children',
        'description': 'List of children nodes, only applicable for folders, files cannot have children',
        'type': 'array',
        'items': {'$ref': '#/definitions/Node'}},
       'node_type': {'description': 'Either a file or folder, use the name to determine which it could be',

OpenAI Pydantic Program

        'default': 'file',
        'allOf': [{'$ref': '#/definitions/NodeType'}]}},
      'required': ['name']}}}
   program = OpenAIPydanticProgram.from_defaults(
       output_cls=DirectoryTree,
       prompt_template_str="{input_str}",
       verbose=True,
   )
   input_str = """
   root
   ├── folder1
   │   ├── file1.txt
   │   └── file2.txt
   └── folder2
       ├── file3.txt
       └── subfolder1
           └── file4.txt
   """
   output = program(input_str=input_str)
   Function call: DirectoryTree with args: {
     "root": {
       "name": "root",
       "children": [
         {
           "name": "folder1",
           "children": [
             {
               "name": "file1.txt",
               "children": [],
               "node_type": "file"
             },
             {
               "name": "file2.txt",
               "children": [],
               "node_type": "file"
             }
           ],
           "node_type": "folder"
         },
         {
           "name": "folder2",
           "children": [
             {
               "name": "file3.txt",
               "children": [],
               "node_type": "file"
             },
             {
               "name": "subfolder1",
               "children": [
                 {
                   "name": "file4.txt",
                   "children": [],
                   "node_type": "file"
                 }
               ],
               "node_type": "folder"
             }
           ],
           "node_type": "folder"
         }
       ],
       "node_type": "folder"
     }
   }
The output is a full DirectoryTree structure with recursive "Node"
objects.
   output
   DirectoryTree(root=Node(name='root', children=[Node(name='folder1', children=[Node(name='file1.txt', children=[], node_type=<NodeType.FILE: 'file'>), Node(name='file2.txt', children=[], node_type=<NodeType.FILE: 'file'>)], node_type=<NodeType.FOLDER: 'folder'>), Node(name='folder2', children=[Node(name='file3.txt', children=[], node_type=<NodeType.FILE: 'file'>), Node(name='subfolder1', children=[Node(name='file4.txt', children=[], node_type=<NodeType.FILE: 'file'>)], node_type=<NodeType.FOLDER: 'folder'>)], node_type=<NodeType.FOLDER: 'folder'>)], node_type=<NodeType.FOLDER: 'folder'>))


Evaporate Demo

This demo shows how you can extract DataFrame from raw text using the
Evaporate paper (Arora et al.): https://arxiv.org/abs/2304.09433.
The inspiration is to first "fit" on a set of training text. The
fitting process uses the LLM to generate a set of parsing functions
from the text. These fitted functions are then applied to text during
inference time.
   %load_ext autoreload
   %autoreload 2
   from llama_index import SimpleDirectoryReader, ServiceContext, LLMPredictor
   from llama_index.program.predefined import (
       DFEvaporateProgram,
       EvaporateExtractor,
       MultiValueEvaporateProgram,
   )
   from llama_index.llms import OpenAI
   import requests
Use "DFEvaporateProgram"
The "DFEvaporateProgram" will extract a 2D dataframe from a set of
datapoints given a set of fields, and some training data to "fit" some
functions on.
Load data
Here we load a set of cities from Wikipedia.
   wiki_titles = ["Toronto", "Seattle", "Chicago", "Boston", "Houston"]
   from pathlib import Path
   import requests
   for title in wiki_titles:
       response = requests.get(
           "https://en.wikipedia.org/w/api.php",
           params={
               "action": "query",
               "format": "json",
               "titles": title,
               "prop": "extracts",
               # 'exintro': True,
               "explaintext": True,
           },
       ).json()
       page = next(iter(response["query"]["pages"].values()))
       wiki_text = page["extract"]
       data_path = Path("data")
       if not data_path.exists():
           Path.mkdir(data_path)
       with open(data_path / f"{title}.txt", "w") as fp:
           fp.write(wiki_text)
   # Load all wiki documents
   city_docs = {}
   for wiki_title in wiki_titles:
       city_docs[wiki_title] = SimpleDirectoryReader(
           input_files=[f"data/{wiki_title}.txt"]
       ).load_data()
Parse Data
   # setup service context
   llm = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context = ServiceContext.from_defaults(llm=llm, chunk_size=512)
   # get nodes for each document
   city_nodes = {}
   for wiki_title in wiki_titles:
       docs = city_docs[wiki_title]
       nodes = service_context.node_parser.get_nodes_from_documents(docs)
       city_nodes[wiki_title] = nodes
Running the DFEvaporateProgram
Here we demonstrate how to extract datapoints with our
"DFEvaporateProgram". Given a set of fields, the "DFEvaporateProgram"
can first fit functions on a set of training data, and then run
extraction over inference data.
   # define program
   program = DFEvaporateProgram.from_defaults(
       fields_to_extract=["population"], service_context=service_context
   )
Fitting Functions
   program.fit_fields(city_nodes["Toronto"][:1])
   {'population': 'def get_population_field(text: str):\n    """\n    Function to extract population. \n    """\n    \n    # Use regex to find the population field\n    pattern = r\'(?<=population of )(\\d+,?\\d*)\'\n    population_field = re.search(pattern, text).group(1)\n    \n    # Return the population field as a single value\n    return int(population_field.replace(\',\', \'\'))'}
   # view extracted function
   print(program.get_function_str("population"))
   def get_population_field(text: str):
       """
       Function to extract population.

Evaporate Demo

"""
       # Use regex to find the population field
       pattern = r'(?<=population of )(\d+,?\d*)'
       population_field = re.search(pattern, text).group(1)
       # Return the population field as a single value
       return int(population_field.replace(',', ''))
Run Inference
   seattle_df = program(nodes=city_nodes["Seattle"][:1])
   seattle_df
   DataFrameRowsOnly(rows=[DataFrameRow(row_values=[749256])])
Use "MultiValueEvaporateProgram"
In contrast to the "DFEvaporateProgram", which assumes the output
obeys a 2D tabular format (one row per node), the
"MultiValueEvaporateProgram" returns a list of "DataFrameRow" objects
- each object corresponds to a column, and can contain a variable
length of values. This can help if we want to extract multiple values
for one field from a given piece of text.
In this example, we use this program to parse gold medal counts.
   llm = OpenAI(temperature=0, model="gpt-4")
   service_context = ServiceContext.from_defaults(
       llm=llm, chunk_size=1024, chunk_overlap=0
   )
   # Olympic total medal counts: https://en.wikipedia.org/wiki/All-time_Olympic_Games_medal_table
   train_text = """

   Team (IOC code)

   No. Summer

   No. Winter

   No. Games

    Albania (ALB)

   9
   5
   14

    American Samoa (ASA)

Evaporate Demo

   9
   2
   11

    Andorra (AND)

   12
   13
   25

    Angola (ANG)

   10
   0
   10

   <img alt="" src="//upload.wikimedia.org/wikipedia/commons/thumb/8/89/Flag_of_Antigua_and_Barbuda.svg/22px-Flag_of_Antigua_and_Barbuda.svg.png" decoding="async" width="22" height="15" class="thumbborder" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/8/89/Flag_of_Antigua_and_Barbuda.svg/33px-Flag_of_Antigua_and_Barbuda.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/8/89/Flag_of_Antigua_and_Barbuda.svg/44px-Flag_of_Antigua_and_Barbuda.svg.png 2x" data-file-width="900" data-file-height="600" /> Antigua and Barbuda (ANT)

   11

Evaporate Demo

0
   11

    Aruba (ARU)

   9
   0
   9

   """
   train_nodes = [Node(text=train_text)]
   infer_text = """
    Bangladesh (BAN)

   10
   0
   10

    Belize (BIZ) [BIZ]

   13
   0

Evaporate Demo

13

    Benin (BEN) [BEN]

   12
   0
   12

    Bhutan (BHU)

   10
   0
   10

    Bolivia (BOL)

   15
   7

Evaporate Demo

22

   <img alt="" src="//upload.wikimedia.org/wikipedia/commons/thumb/b/bf/Flag_of_Bosnia_and_Herzegovina.svg/22px-Flag_of_Bosnia_and_Herzegovina.svg.png" decoding="async" width="22" height="11" class="thumbborder" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/b/bf/Flag_of_Bosnia_and_Herzegovina.svg/33px-Flag_of_Bosnia_and_Herzegovina.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/b/bf/Flag_of_Bosnia_and_Herzegovina.svg/44px-Flag_of_Bosnia_and_Herzegovina.svg.png 2x" data-file-width="800" data-file-height="400" /> Bosnia and Herzegovina (BIH)

   8
   8
   16

   <img alt="" src="//upload.wikimedia.org/wikipedia/commons/thumb/4/42/Flag_of_the_British_Virgin_Islands.svg/22px-Flag_of_the_British_Virgin_Islands.svg.png" decoding="async" width="22" height="11" class="thumbborder" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/4/42/Flag_of_the_British_Virgin_Islands.svg/33px-Flag_of_the_British_Virgin_Islands.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/4/42/Flag_of_the_British_Virgin_Islands.svg/44px-Flag_of_the_British_Virgin_Islands.svg.png 2x" data-file-width="1200" data-file-height="600" /> British Virgin Islands (IVB)

   10
   2
   12

    Brunei (BRU) [A]

Evaporate Demo

   6
   0
   6

    Cambodia (CAM)

   10
   0
   10

    Cape Verde (CPV)

   7
   0
   7

   <img alt="" src="//upload.wikimedia.org/wikipedia/commons/thumb/0/0f/Flag_of_the_Cayman_Islands.svg/22px-Flag_of_the_Cayman_Islands.svg.png" decoding="async" width="22" height="11" class="thumbborder" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/0/0f/Flag_of_the_Cayman_Islands.svg/33px-Flag_of_the_Cayman_Islands.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/0/0f/Flag_of_the_Cayman_Islands.svg/44px-Flag_of_the_Cayman_Islands.svg.png 2x" data-file-width="1200" data-file-height="600" /> Cayman Islands (CAY)

Evaporate Demo

11
   2
   13

   <img alt="" src="//upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Flag_of_the_Central_African_Republic.svg/22px-Flag_of_the_Central_African_Republic.svg.png" decoding="async" width="22" height="15" class="thumbborder" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Flag_of_the_Central_African_Republic.svg/33px-Flag_of_the_Central_African_Republic.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Flag_of_the_Central_African_Republic.svg/44px-Flag_of_the_Central_African_Republic.svg.png 2x" data-file-width="900" data-file-height="600" /> Central African Republic (CAF)

   11
   0
   11

    Chad (CHA)

   13
   0
   13

    Comoros (COM)

   7

Evaporate Demo

0
   7

   <img alt="" src="//upload.wikimedia.org/wikipedia/commons/thumb/9/92/Flag_of_the_Republic_of_the_Congo.svg/22px-Flag_of_the_Republic_of_the_Congo.svg.png" decoding="async" width="22" height="15" class="thumbborder" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/9/92/Flag_of_the_Republic_of_the_Congo.svg/33px-Flag_of_the_Republic_of_the_Congo.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/9/92/Flag_of_the_Republic_of_the_Congo.svg/44px-Flag_of_the_Republic_of_the_Congo.svg.png 2x" data-file-width="900" data-file-height="600" /> Republic of the Congo (CGO)

   13
   0
   13

   <img alt="" src="//upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Flag_of_the_Democratic_Republic_of_the_Congo.svg/22px-Flag_of_the_Democratic_Republic_of_the_Congo.svg.png" decoding="async" width="22" height="17" class="thumbborder" srcset="//upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Flag_of_the_Democratic_Republic_of_the_Congo.svg/33px-Flag_of_the_Democratic_Republic_of_the_Congo.svg.png 1.5x, //upload.wikimedia.org/wikipedia/commons/thumb/6/6f/Flag_of_the_Democratic_Republic_of_the_Congo.svg/44px-Flag_of_the_Democratic_Republic_of_the_Congo.svg.png 2x" data-file-width="800" data-file-height="600" /> Democratic Republic of the Congo (COD) [COD]

   11
   0
   11

   """
   infer_nodes = [Node(text=infer_text)]
   from llama_index.program.predefined import MultiValueEvaporateProgram
   program = MultiValueEvaporateProgram.from_defaults(
       fields_to_extract=["countries", "medal_count"], service_context=service_context
   )
   program.fit_fields(train_nodes[:1])
   {'countries': 'def get_countries_field(text: str):\n    """\n    Function to extract countries. \n    """\n    \n    # Use regex to extract the countries field\n    countries_field = re.findall(r\'(.*)\', text)\n    \n    # Return the result as a list\n    return countries_field',
    'medal_count': 'def get_medal_count_field(text: str):\n    """\n    Function to extract medal_count. \n    """\n    \n    # Use regex to extract the medal count field\n    medal_count_field = re.findall(r\'(.*?)	\', text)\n    \n    # Return the result as a list\n    return medal_count_field'}

Evaporate Demo

print(program.get_function_str("countries"))
   def get_countries_field(text: str):
       """
       Function to extract countries.
       """
       # Use regex to extract the countries field
       countries_field = re.findall(r'(.*)', text)
       # Return the result as a list
       return countries_field
   print(program.get_function_str("medal_count"))
   def get_medal_count_field(text: str):
       """
       Function to extract medal_count.
       """
       # Use regex to extract the medal count field
       medal_count_field = re.findall(r'(.*?)	', text)
       # Return the result as a list
       return medal_count_field
   result = program(nodes=infer_nodes[:1])
   # output countries
   print(f"Countries: {result.columns[0].row_values}\n")
   # output medal counts
   print(f"Medal Counts: {result.columns[0].row_values}\n")
   Countries: ['Bangladesh', '[BIZ]', '[BEN]', 'Bhutan', 'Bolivia', 'Bosnia and Herzegovina', 'British Virgin Islands', '[A]', 'Cambodia', 'Cape Verde', 'Cayman Islands', 'Central African Republic', 'Chad', 'Comoros', 'Republic of the Congo', '[COD]']
   Medal Counts: ['Bangladesh', '[BIZ]', '[BEN]', 'Bhutan', 'Bolivia', 'Bosnia and Herzegovina', 'British Virgin Islands', '[A]', 'Cambodia', 'Cape Verde', 'Cayman Islands', 'Central African Republic', 'Chad', 'Comoros', 'Republic of the Congo', '[COD]']
Bonus: Use the underlying "EvaporateExtractor"
The underlying "EvaporateExtractor" offers some additional
functionality, e.g. actually helping to identify fields over a set of
text.
Here we show how you can use "identify_fields" to determine relevant
fields around a general "topic" field.
   # a list of nodes, one node per city, corresponding to intro paragraph
   # city_pop_nodes = []
   city_pop_nodes = [city_nodes["Toronto"][0], city_nodes["Seattle"][0]]
   extractor = program.extractor
   # Try with Toronto and Seattle (should extract "population")
   existing_fields = extractor.identify_fields(
       city_pop_nodes, topic="population", fields_top_k=4
   )
   existing_fields
   ["seattle metropolitan area's population"]


Retrieval-Augmented OpenAI Agent

In this tutorial, we show you how to use our "FnRetrieverOpenAI"
implementation to build an agent on top of OpenAI's function API and
store/index an arbitrary number of tools. Our indexing/retrieval
modules help to remove the complexity of having too many functions to
fit in the prompt.
Initial Setup
Let's start by importing some simple building blocks.
The main thing we need is:
1. the OpenAI API
2. a place to keep conversation history
3. a definition for tools that our agent can use.
   import json
   from typing import Sequence
   from llama_index.tools import BaseTool, FunctionTool
   /Users/suo/miniconda3/envs/llama/lib/python3.9/site-packages/deeplake/util/check_latest_version.py:32: UserWarning: A newer version of deeplake (3.6.7) is available. It's recommended that you update to the latest version using `pip install -U deeplake`.
     warnings.warn(
Let's define some very simple calculator tools for our agent.
   def multiply(a: int, b: int) -> int:
       """Multiply two integers and returns the result integer"""
       return a * b
   def add(a: int, b: int) -> int:
       """Add two integers and returns the result integer"""
       return a + b
   def useless(a: int, b: int) -> int:
       """Toy useless function."""
       pass
   multiply_tool = FunctionTool.from_defaults(fn=multiply, name="multiply")
   useless_tools = [
       FunctionTool.from_defaults(fn=useless, name=f"useless_{str(idx)}")
       for idx in range(28)
   ]
   add_tool = FunctionTool.from_defaults(fn=add, name="add")
   all_tools = [multiply_tool] + [add_tool] + useless_tools
   all_tools_map = {t.metadata.name: t for t in all_tools}
Building an Object Index
We have an "ObjectIndex" construct in LlamaIndex that allows the user
to use our index data structures over arbitrary objects. The
ObjectIndex will handle serialiation to/from the object, and use an
underying index (e.g. VectorStoreIndex, SummaryIndex,
KeywordTableIndex) as the storage mechanism.
In this case, we have a large collection of Tool objects, and we'd
want to define an ObjectIndex over these Tools.
The index comes bundled with a retrieval mechanism, an
"ObjectRetriever".
This can be passed in to our agent so that it can perform Tool
retrieval during query-time.
   # define an "object" index over these tools
   from llama_index import VectorStoreIndex
   from llama_index.objects import ObjectIndex, SimpleToolNodeMapping
   tool_mapping = SimpleToolNodeMapping.from_objects(all_tools)
   obj_index = ObjectIndex.from_objects(
       all_tools,
       tool_mapping,
       VectorStoreIndex,
   )
Our "FnRetrieverOpenAIAgent" Implementation
We provide a "FnRetrieverOpenAIAgent" implementation in LlamaIndex,
which can take in an "ObjectRetriever" over a set of "BaseTool"
objects.
During query-time, we would first use the "ObjectRetriever" to
retrieve a set of relevant Tools. These tools would then be passed
into the agent; more specifically, their function signatures would be
passed into the OpenAI Function calling API.
   from llama_index.agent import FnRetrieverOpenAIAgent
   agent = FnRetrieverOpenAIAgent.from_retriever(obj_index.as_retriever(), verbose=True)
   agent.chat("What's 212 multiplied by 122? Make sure to use Tools")
   === Calling Function ===
   Calling function: multiply with args: {
     "a": 212,

Retrieval-Augmented OpenAI Agent

     "b": 122
   }
   Got output: 25864
   ========================
   Response(response='212 multiplied by 122 is 25,864.', source_nodes=[], metadata=None)
   agent.chat("What's 212 added to 122 ? Make sure to use Tools")
   === Calling Function ===
   Calling function: add with args: {
     "a": 212,
     "b": 122
   }
   Got output: 334
   ========================
   Response(response='212 added to 122 is 334.', source_nodes=[], metadata=None)


ReAct Agent with Query Engine Tools

In this section, we show how to setup an agent powered by the ReAct
loop for financial analysis.
The agent has access to two "tools": one to query the 2021 Lyft 10-K
and the other to query the 2021 Uber 10-K.
We try two different LLMs:
* gpt-3.5-turbo
* gpt-3.5-turbo-instruct
Note that you can plug in any LLM that exposes a text completion
endpoint.
Build Query Engine Tools
   from llama_index import (
       SimpleDirectoryReader,
       VectorStoreIndex,
       StorageContext,
       load_index_from_storage,
   )
   from llama_index.tools import QueryEngineTool, ToolMetadata
   try:
       storage_context = StorageContext.from_defaults(persist_dir="./storage/lyft")
       lyft_index = load_index_from_storage(storage_context)
       storage_context = StorageContext.from_defaults(persist_dir="./storage/uber")
       uber_index = load_index_from_storage(storage_context)
       index_loaded = True
   except:
       index_loaded = False
   if not index_loaded:
       # load data
       lyft_docs = SimpleDirectoryReader(
           input_files=["../data/10k/lyft_2021.pdf"]
       ).load_data()
       uber_docs = SimpleDirectoryReader(
           input_files=["../data/10k/uber_2021.pdf"]
       ).load_data()
       # build index
       lyft_index = VectorStoreIndex.from_documents(lyft_docs)
       uber_index = VectorStoreIndex.from_documents(uber_docs)
       # persist index
       lyft_index.storage_context.persist(persist_dir="./storage/lyft")
       uber_index.storage_context.persist(persist_dir="./storage/uber")
   lyft_engine = lyft_index.as_query_engine(similarity_top_k=3)
   uber_engine = uber_index.as_query_engine(similarity_top_k=3)
   query_engine_tools = [
       QueryEngineTool(
           query_engine=lyft_engine,
           metadata=ToolMetadata(
               name="lyft_10k",
               description="Provides information about Lyft financials for year 2021. "
               "Use a detailed plain text question as input to the tool.",
           ),
       ),
       QueryEngineTool(
           query_engine=uber_engine,
           metadata=ToolMetadata(
               name="uber_10k",
               description="Provides information about Uber financials for year 2021. "
               "Use a detailed plain text question as input to the tool.",
           ),
       ),
   ]
Setup ReAct Agent
Here we setup two ReAct agents: one powered by standard gpt-3.5-turbo,
and the other powered by gpt-3.5-turbo-instruct.
   from llama_index.agent import ReActAgent
   from llama_index.llms import OpenAI
   llm = OpenAI(model="gpt-3.5-turbo-0613")
   agent = ReActAgent.from_tools(query_engine_tools, llm=llm, verbose=True)
   response = agent.chat("What was Lyft's revenue growth in 2021?")
   print(str(response))
   Thought: I need to use a tool to help me answer the question.
   Action: lyft_10k
   Action Input: {'input': "What was Lyft's revenue growth in 2021?"}
   Observation: Lyft's revenue growth in 2021 was 36%.
   Response: Lyft's revenue growth in 2021 was 36%.
   Lyft's revenue growth in 2021 was 36%.
Run Some Example Queries

ReAct Agent with Query Engine Tools

We run some example queries using the agent, showcasing some of the
agent's abilities to do chain-of-thought-reasoning and tool use to
synthesize the right answer.
We also show queries.
   response = agent.chat(
       "Compare and contrast the revenue growth of Uber and Lyft in 2021, then give an analysis"
   )
   print(str(response))
   Thought: I need to use a tool to help me compare the revenue growth of Uber and Lyft in 2021.
   Action: lyft_10k
   Action Input: {'input': "What was Lyft's revenue growth in 2021?"}
   Observation: Lyft's revenue growth in 2021 was 36%.
   Thought: I need to use a tool to help me compare the revenue growth of Uber and Lyft in 2021.
   Action: uber_10k
   Action Input: {'input': "What was Uber's revenue growth in 2021?"}
   Observation: Uber's revenue growth in 2021 was 57%.
   Response: In 2021, Lyft's revenue growth was 36% while Uber's revenue growth was 57%. This indicates that Uber experienced a higher revenue growth compared to Lyft in 2021.
   In 2021, Lyft's revenue growth was 36% while Uber's revenue growth was 57%. This indicates that Uber experienced a higher revenue growth compared to Lyft in 2021.
**Async execution**: Here we try another query with async execution
   # Try another query with async execution
   import nest_asyncio
   nest_asyncio.apply()
   response = await agent.achat(
       "Compare and contrast the risks of Uber and Lyft in 2021, then give an analysis"
   )
   print(str(response))
Compare gpt-3.5-turbo vs. gpt-3.5-turbo-instruct
We compare the performance of the two agents in being able to answer
some complex queries.
Taking a look at a turbo-instruct agent
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   llm_instruct = OpenAI(model="gpt-3.5-turbo-instruct")
   agent_instruct = ReActAgent.from_tools(
       query_engine_tools, llm=llm_instruct, verbose=True
   )
   response = agent_instruct.chat("What was Lyft's revenue growth in 2021?")
   print(str(response))
   Thought: I need to use a tool to help me answer the question.
   Action: lyft_10k
   Action Input: {'input': "What was Lyft's revenue growth in 2021?"}
   Observation: Lyft's revenue growth in 2021 was 36%.
   Response: Lyft's revenue growth in 2021 was 36%.
   Lyft's revenue growth in 2021 was 36%.
Try more complex queries
~~~~~~~~~~~~~~~~~~~~~~~~
We compare gpt-3.5-turbo with gpt-3.5-turbo-instruct agents on more
complex queries.
   response = agent.chat(
       "Compare and contrast the revenue growth of Uber and Lyft in 2021, then give an analysis"
   )
   print(str(response))
   Thought: I need to use a tool to help me compare the revenue growth of Uber and Lyft in 2021.

ReAct Agent with Query Engine Tools

   Action: uber_10k
   Action Input: {'input': "Please provide information about Uber's revenue growth in 2021."}
   Observation: Uber's revenue grew by 57% in 2021 compared to the previous year. This growth was primarily driven by an increase in Gross Bookings, with Delivery Gross Bookings increasing by 71% and Mobility Gross Bookings growing by 38%. The increase in Delivery Gross Bookings was due to higher demand for food delivery orders and expansion across U.S. and international markets. The growth in Mobility Gross Bookings was a result of increased Trip volumes as the business recovered from the impacts of COVID-19.
   Thought: I have information about Uber's revenue growth in 2021. Now I need to use a tool to get information about Lyft's revenue growth in 2021.
   Action: lyft_10k
   Action Input: {'input': "Please provide information about Lyft's revenue growth in 2021."}
   Observation: Lyft's revenue increased by 36% in 2021 compared to the prior year.
   Response: In 2021, Uber experienced a higher revenue growth rate of 57% compared to Lyft's growth rate of 36%. This indicates that Uber had a stronger performance in terms of revenue growth during that period. The growth in Uber's revenue was primarily driven by an increase in Gross Bookings, with both Delivery and Mobility segments contributing to the growth. The increase in Delivery Gross Bookings was due to higher demand for food delivery services, while the growth in Mobility Gross Bookings was a result of increased trip volumes as the business recovered from the impacts of COVID-19.
   In 2021, Uber experienced a higher revenue growth rate of 57% compared to Lyft's growth rate of 36%. This indicates that Uber had a stronger performance in terms of revenue growth during that period. The growth in Uber's revenue was primarily driven by an increase in Gross Bookings, with both Delivery and Mobility segments contributing to the growth. The increase in Delivery Gross Bookings was due to higher demand for food delivery services, while the growth in Mobility Gross Bookings was a result of increased trip volumes as the business recovered from the impacts of COVID-19.
   response = agent_instruct.chat(
       "Compare and contrast the revenue growth of Uber and Lyft in 2021, then give an analysis"
   )
   print(str(response))
   Response: The revenue growth of Uber was higher than Lyft in 2021, with Uber experiencing a 74% growth compared to Lyft's 48%. This indicates that Uber may have had a stronger financial performance in 2021. However, further analysis is needed to fully understand the factors contributing to this difference.
   The revenue growth of Uber was higher than Lyft in 2021, with Uber experiencing a 74% growth compared to Lyft's 48%. This indicates that Uber may have had a stronger financial performance in 2021. However, further analysis is needed to fully understand the factors contributing to this difference.
   response = agent.chat(
       "Can you tell me about the risk factors of the company with the higher revenue?"
   )
   print(str(response))
   Thought: I need to find out which company has higher revenue before I can provide information about its risk factors.
   Action: lyft_10k
   Action Input: {'input': 'What is the revenue of Lyft in 2021?'}
   Observation: The revenue of Lyft in 2021 is $3,208,323,000.

ReAct Agent with Query Engine Tools

   Thought: Now that I know Lyft has higher revenue, I can find information about its risk factors.
   Action: lyft_10k
   Action Input: {'input': 'What are the risk factors of Lyft?'}
   Observation: Lyft faces numerous risk factors that could potentially harm its business, financial condition, and results of operations. These risk factors include general economic factors such as the impact of the COVID-19 pandemic, natural disasters, economic downturns, and political crises. Operational factors such as limited operating history, financial performance, competition, unpredictability of results, uncertainty regarding market growth, ability to attract and retain drivers and riders, insurance coverage, autonomous vehicle technology, reputation and brand, illegal or improper activity on the platform, accuracy of background checks, changes to pricing practices, growth management, security and privacy breaches, reliance on third parties, and ability to operate various programs and services. Additionally, Lyft faces risks related to its evolving business, including forecasting revenue and managing expenses, complying with laws and regulations, managing assets and expenses during the COVID-19 pandemic, capital expenditures, asset development and utilization, macroeconomic changes, reputation and brand management, growth and business operations, geographic expansion, talent acquisition and retention, platform development, and real estate portfolio management. Furthermore, Lyft's financial performance in recent periods may not be indicative of future performance, and achieving or maintaining profitability in the future is not guaranteed. The Express Drive program and Lyft Rentals program also expose Lyft to risks related to vehicle rental partners, residual value of vehicles, and payment processing.
   Response: Lyft faces numerous risk factors that could potentially harm its business, financial condition, and results of operations. These risk factors include general economic factors such as the impact of the COVID-19 pandemic, natural disasters, economic downturns, and political crises. Operational factors such as limited operating history, financial performance, competition, unpredictability of results, uncertainty regarding market growth, ability to attract and retain drivers and riders, insurance coverage, autonomous vehicle technology, reputation and brand, illegal or improper activity on the platform, accuracy of background checks, changes to pricing practices, growth management, security and privacy breaches, reliance on third parties, and ability to operate various programs and services. Additionally, Lyft faces risks related to its evolving business, including forecasting revenue and managing expenses, complying with laws and regulations, managing assets and expenses during the COVID-19 pandemic, capital expenditures, asset development and utilization, macroeconomic changes, reputation and brand management, growth and business operations, geographic expansion, talent acquisition and retention, platform development, and real estate portfolio management. Furthermore, Lyft's financial performance in recent periods may not be indicative of future performance, and achieving or maintaining profitability in the future is not guaranteed. The Express Drive program and Lyft Rentals program also expose Lyft to risks related to vehicle rental partners, residual value of vehicles, and payment processing.
   Lyft faces numerous risk factors that could potentially harm its business, financial condition, and results of operations. These risk factors include general economic factors such as the impact of the COVID-19 pandemic, natural disasters, economic downturns, and political crises. Operational factors such as limited operating history, financial performance, competition, unpredictability of results, uncertainty regarding market growth, ability to attract and retain drivers and riders, insurance coverage, autonomous vehicle technology, reputation and brand, illegal or improper activity on the platform, accuracy of background checks, changes to pricing practices, growth management, security and privacy breaches, reliance on third parties, and ability to operate various programs and services. Additionally, Lyft faces risks related to its evolving business, including forecasting revenue and managing expenses, complying with laws and regulations, managing assets and expenses during the COVID-19 pandemic, capital expenditures, asset development and utilization, macroeconomic changes, reputation and brand management, growth and business operations, geographic expansion, talent acquisition and retention, platform development, and real estate portfolio management. Furthermore, Lyft's financial performance in recent periods may not be indicative of future performance, and achieving or maintaining profitability in the future is not guaranteed. The Express Drive program and Lyft Rentals program also expose Lyft to risks related to vehicle rental partners, residual value of vehicles, and payment processing.

ReAct Agent with Query Engine Tools

   response = agent_instruct.query(
       "Can you tell me about the risk factors of the company with the higher revenue?"
   )
   print(str(response))
   Response: The risk factors for the company with the higher revenue include competition, regulatory changes, and dependence on drivers.
   The risk factors for the company with the higher revenue include competition, regulatory changes, and dependence on drivers.
**Observation**: The turbo-instruct agent seems to do worse on agent
reasoning compared to the regular turbo model. Of course, this is
subject to further observation!


import json

   from typing import Sequence, List
   from llama_index.llms import OpenAI, ChatMessage
   from llama_index.tools import BaseTool, FunctionTool
   from llama_index.agent import OpenAIAgent
   def add(a: int, b: int) -> int:
       """Add two integers and returns the result integer"""
       return a + b
   add_tool = FunctionTool.from_defaults(fn=add)
   def useless_tool() -> int:
       """This is a uselss tool."""
       return "This is a uselss output."
   useless_tool = FunctionTool.from_defaults(fn=useless_tool)
   llm = OpenAI(model="gpt-3.5-turbo-0613")
   agent = OpenAIAgent.from_tools([useless_tool, add_tool], llm=llm, verbose=True)
"Auto" function call
The agent automatically selects the useful "add" tool
   response = agent.chat("What is 5 + 2?", function_call="auto")
   === Calling Function ===
   Calling function: add with args: {
     "a": 5,
     "b": 2
   }
   Got output: 7
   ========================
   print(response)
   The sum of 5 and 2 is 7.
Forced function call
The agent is forced to call the "useless_tool" before selecting the
"add" tool
   response = agent.chat("What is 5 * 2?", function_call="useless_tool")
   === Calling Function ===
   Calling function: useless_tool with args: {}
   Got output: This is a uselss output.
   ========================
   === Calling Function ===
   Calling function: add with args: {
     "a": 5,
     "b": 2
   }
   Got output: 7
   ========================
   print(response)
   The product of 5 and 2 is 10.
"None" function call
The agent is forced to not use a tool
   response = agent.chat("What is 5 * 2?", function_call="none")
   print(response)
   The product of 5 and 2 is 10.