Multi-Document Agents

In this guide, you learn towards setting up an agent that can
effectively answer different types of questions over a larger set of
documents.
These questions include the following
* QA over a specific doc
* QA comparing different docs
* Summaries over a specific odc
* Comparing summaries between different docs
We do this with the following architecture:
* setup a "document agent" over each Document: each doc agent can do
  QA/summarization within its doc
* setup a top-level agent over this set of document agents. Do tool
  retrieval and then do CoT over the set of tools to answer a
  question.
Setup and Download Data
In this section, we'll define imports and then download Wikipedia
articles about different cities. Each article is stored separately.
We load in 18 cities - this is not quite at the level of "hundreds" of
documents but its still large enough to warrant some top-level
document retrieval!
   from llama_index import (
       VectorStoreIndex,
       SummaryIndex,
       SimpleKeywordTableIndex,
       SimpleDirectoryReader,
       ServiceContext,
   )
   from llama_index.schema import IndexNode
   from llama_index.tools import QueryEngineTool, ToolMetadata
   from llama_index.llms import OpenAI
   wiki_titles = [
       "Toronto",
       "Seattle",
       "Chicago",
       "Boston",
       "Houston",
       "Tokyo",
       "Berlin",
       "Lisbon",
       "Paris",
       "London",
       "Atlanta",
       "Munich",
       "Shanghai",
       "Beijing",
       "Copenhagen",
       "Moscow",
       "Cairo",
       "Karachi",
   ]
   from pathlib import Path
   import requests
   for title in wiki_titles:
       response = requests.get(
           "https://en.wikipedia.org/w/api.php",
           params={
               "action": "query",
               "format": "json",
               "titles": title,
               "prop": "extracts",
               # 'exintro': True,
               "explaintext": True,
           },
       ).json()
       page = next(iter(response["query"]["pages"].values()))
       wiki_text = page["extract"]
       data_path = Path("data")
       if not data_path.exists():
           Path.mkdir(data_path)
       with open(data_path / f"{title}.txt", "w") as fp:
           fp.write(wiki_text)
   # Load all wiki documents
   city_docs = {}
   for wiki_title in wiki_titles:
       city_docs[wiki_title] = SimpleDirectoryReader(
           input_files=[f"data/{wiki_title}.txt"]
       ).load_data()
Define LLM + Service Context + Callback Manager
   llm = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context = ServiceContext.from_defaults(llm=llm)
Building Multi-Document Agents
In this section we show you how to construct the multi-document agent.
We first build a document agent for each document, and then define the
top-level parent agent with an object index.
Build Document Agent for each Document
In this section we define "document agents" for each document.
We define both a vector index (for semantic search) and summary index
(for summarization) for each document. The two query engines are then
converted into tools that are passed to an OpenAI function calling
agent.
This document agent can dynamically choose to perform semantic search
or summarization within a given document.
We create a separate document agent for each city.
   from llama_index.agent import OpenAIAgent
   from llama_index import load_index_from_storage, StorageContext
   from llama_index.node_parser import SimpleNodeParser
   import os
   node_parser = SimpleNodeParser.from_defaults()
   # Build agents dictionary
   agents = {}

Multi-Document Agents

   query_engines = {}
   # this is for the baseline
   all_nodes = []
   for idx, wiki_title in enumerate(wiki_titles):
       nodes = node_parser.get_nodes_from_documents(city_docs[wiki_title])
       all_nodes.extend(nodes)
       if not os.path.exists(f"./data/{wiki_title}"):
           # build vector index
           vector_index = VectorStoreIndex(nodes, service_context=service_context)
           vector_index.storage_context.persist(persist_dir=f"./data/{wiki_title}")
       else:
           vector_index = load_index_from_storage(
               StorageContext.from_defaults(persist_dir=f"./data/{wiki_title}"),
               service_context=service_context,
           )
       # build summary index
       summary_index = SummaryIndex(nodes, service_context=service_context)
       # define query engines
       vector_query_engine = vector_index.as_query_engine()
       summary_query_engine = summary_index.as_query_engine()
       # define tools
       query_engine_tools = [
           QueryEngineTool(
               query_engine=vector_query_engine,
               metadata=ToolMetadata(
                   name="vector_tool",
                   description=f"Useful for questions related to specific aspects of {wiki_title} (e.g. the history, arts and culture, sports, demographics, or more).",
               ),
           ),
           QueryEngineTool(
               query_engine=summary_query_engine,
               metadata=ToolMetadata(
                   name="summary_tool",
                   description=f"Useful for any requests that require a holistic summary of EVERYTHING about {wiki_title}. For questions about more specific sections, please use the vector_tool.",
               ),
           ),
       ]
       # build agent
       function_llm = OpenAI(model="gpt-4")
       agent = OpenAIAgent.from_tools(
           query_engine_tools,
           llm=function_llm,
           verbose=True,
           system_prompt=f"""\
   You are a specialized agent designed to answer queries about {wiki_title}.
   You must ALWAYS use at least one of the tools provided when answering a question; do NOT rely on prior knowledge.\
   """,
       )
       agents[wiki_title] = agent
       query_engines[wiki_title] = vector_index.as_query_engine(similarity_top_k=2)
Build Retriever-Enabled OpenAI Agent
We build a top-level agent that can orchestrate across the different
document agents to answer any user query.
This agent takes in all document agents as tools. This specific agent
"RetrieverOpenAIAgent" performs tool retrieval before tool use (unlike
a default agent that tries to put all tools in the prompt).
Here we use a top-k retriever, but we encourage you to customize the
tool retriever method!
   # define tool for each document agent
   all_tools = []
   for wiki_title in wiki_titles:
       wiki_summary = (
           f"This content contains Wikipedia articles about {wiki_title}. "
           f"Use this tool if you want to answer any questions about {wiki_title}.\n"
       )
       doc_tool = QueryEngineTool(
           query_engine=agents[wiki_title],
           metadata=ToolMetadata(
               name=f"tool_{wiki_title}",
               description=wiki_summary,
           ),
       )
       all_tools.append(doc_tool)
   # define an "object" index and retriever over these tools
   from llama_index import VectorStoreIndex
   from llama_index.objects import ObjectIndex, SimpleToolNodeMapping
   tool_mapping = SimpleToolNodeMapping.from_objects(all_tools)
   obj_index = ObjectIndex.from_objects(
       all_tools,
       tool_mapping,
       VectorStoreIndex,
   )
   from llama_index.agent import FnRetrieverOpenAIAgent
   top_agent = FnRetrieverOpenAIAgent.from_retriever(
       obj_index.as_retriever(similarity_top_k=3),
       system_prompt=""" \
   You are an agent designed to answer queries about a set of given cities.

Multi-Document Agents

   Please always use the tools provided to answer a question. Do not rely on prior knowledge.\
   """,
       verbose=True,
   )
Define Baseline Vector Store Index
As a point of comparison, we define a "naive" RAG pipeline which dumps
all docs into a single vector index collection.
We set the top_k = 4
   base_index = VectorStoreIndex(all_nodes)
   base_query_engine = base_index.as_query_engine(similarity_top_k=4)
Running Example Queries
Let's run some example queries, ranging from QA / summaries over a
single document to QA / summarization over multiple documents.
   # should use Boston agent -> vector tool
   response = top_agent.query("Tell me about the arts and culture in Boston")
   === Calling Function ===
   Calling function: tool_Boston with args: {
     "input": "arts and culture"
   }
   === Calling Function ===
   Calling function: vector_tool with args: {
     "input": "arts and culture"
   }
   Got output: Boston is known for its vibrant arts and culture scene. The city is home to a number of performing arts organizations, including the Boston Ballet, Boston Lyric Opera Company, Opera Boston, Boston Baroque, and the Handel and Haydn Society. There are also several theaters in or near the Theater District, such as the Cutler Majestic Theatre, Citi Performing Arts Center, the Colonial Theater, and the Orpheum Theatre. Boston is a center for contemporary classical music, with groups like the Boston Modern Orchestra Project and Boston Musica Viva. The city also hosts major annual events, such as First Night, the Boston Early Music Festival, and the Boston Arts Festival. In addition, Boston has several art museums and galleries, including the Museum of Fine Arts, the Isabella Stewart Gardner Museum, and the Institute of Contemporary Art.
   ========================
   Got output: Boston is renowned for its vibrant arts and culture scene. It is home to numerous performing arts organizations, including the Boston Ballet, Boston Lyric Opera Company, Opera Boston, Boston Baroque, and the Handel and Haydn Society. The city's Theater District houses several theaters, such as the Cutler Majestic Theatre, Citi Performing Arts Center, the Colonial Theater, and the Orpheum Theatre.
   Boston is also a hub for contemporary classical music, with groups like the Boston Modern Orchestra Project and Boston Musica Viva. The city hosts major annual events, such as First Night, the Boston Early Music Festival, and the Boston Arts Festival, which contribute to its cultural richness.
   In terms of visual arts, Boston boasts several art museums and galleries. The Museum of Fine Arts, the Isabella Stewart Gardner Museum, and the Institute of Contemporary Art are among the most notable. These institutions offer a wide range of art collections, from ancient to contemporary, attracting art enthusiasts from around the world.
   ========================
   print(response)
   Boston has a rich arts and culture scene, with a variety of performing arts organizations and venues. The city is home to renowned institutions such as the Boston Ballet, Boston Lyric Opera Company, Opera Boston, Boston Baroque, and the Handel and Haydn Society. The Theater District in Boston is a hub for theatrical performances, with theaters like the Cutler Majestic Theatre, Citi Performing Arts Center, Colonial Theater, and Orpheum Theatre.
   In addition to performing arts, Boston also has a thriving contemporary classical music scene, with groups like the Boston Modern Orchestra Project and Boston Musica Viva. The city hosts several annual events that celebrate the arts, including First Night, the Boston Early Music Festival, and the Boston Arts Festival.
   Boston is also known for its visual arts scene, with a number of art museums and galleries. The Museum of Fine Arts, the Isabella Stewart Gardner Museum, and the Institute of Contemporary Art are among the notable institutions in the city. These museums offer a diverse range of art collections, spanning from ancient to contemporary art, and attract art enthusiasts from around the world.

Multi-Document Agents

   # baseline
   response = base_query_engine.query("Tell me about the arts and culture in Boston")
   print(str(response))
   Boston has a rich arts and culture scene. The city is home to a variety of performing arts organizations, such as the Boston Ballet, Boston Lyric Opera Company, Opera Boston, Boston Baroque, and the Handel and Haydn Society. Additionally, there are numerous contemporary classical music groups associated with the city's conservatories and universities, like the Boston Modern Orchestra Project and Boston Musica Viva. The Theater District in Boston is a hub for theater, with notable venues including the Cutler Majestic Theatre, Citi Performing Arts Center, the Colonial Theater, and the Orpheum Theatre. Boston also hosts several significant annual events, including First Night, the Boston Early Music Festival, the Boston Arts Festival, and the Boston gay pride parade and festival. The city is renowned for its historic sites connected to the American Revolution, as well as its art museums and galleries, such as the Museum of Fine Arts, Isabella Stewart Gardner Museum, and the Institute of Contemporary Art.
   # should use Houston agent -> vector tool
   response = top_agent.query("Give me a summary of all the positive aspects of Houston")
   === Calling Function ===
   Calling function: tool_Houston with args: {
     "input": "positive aspects"
   }
   === Calling Function ===
   Calling function: summary_tool with args: {
     "input": "positive aspects"
   }
   Got output: Houston has many positive aspects that make it an attractive place to live and visit. The city's diverse population, with people from different ethnic and religious backgrounds, adds to its cultural richness and inclusiveness. Additionally, Houston is home to the Texas Medical Center, which is the largest concentration of healthcare and research institutions in the world. The presence of NASA's Johnson Space Center also highlights Houston's importance in the fields of medicine and space exploration. The city's strong economy, supported by industries such as energy, manufacturing, aeronautics, and transportation, provides numerous economic opportunities for residents and visitors alike. Furthermore, Houston has a thriving visual and performing arts scene, including a theater district and a variety of museums and galleries. Overall, Houston's diverse community, cultural attractions, and economic prospects make it an exceptionally appealing city.
   ========================
   Got output: Houston has numerous positive aspects that make it a desirable place to live and visit. Some of these include:
   1. **Diversity**: Houston is known for its diverse population, with people from different ethnic and religious backgrounds. This diversity adds to the city's cultural richness and inclusiveness.
   2. **Healthcare and Research Institutions**: The city is home to the Texas Medical Center, the largest concentration of healthcare and research institutions in the world. This makes Houston a hub for medical innovation and healthcare services.
   3. **Space Exploration**: Houston is also known for NASA's Johnson Space Center, highlighting the city's significant role in space exploration.
   4. **Strong Economy**: Houston's economy is robust and diverse, supported by industries such as energy, manufacturing, aeronautics, and transportation. This provides numerous economic opportunities for its residents.
   5. **Arts and Culture**: The city has a thriving visual and performing arts scene, with a theater district and a variety of museums and galleries. This makes Houston a vibrant place for art lovers and creatives.
   Overall, these aspects contribute to making Houston an appealing and dynamic city.
   ========================
   print(response)
   Houston has numerous positive aspects that make it a desirable place to live and visit. Some of these include:
   1. Diversity: Houston is known for its diverse population, with people from different ethnic and religious backgrounds. This diversity adds to the city's cultural richness and inclusiveness.
   2. Healthcare and Research Institutions: The city is home to the Texas Medical Center, the largest concentration of healthcare and research institutions in the world. This makes Houston a hub for medical innovation and healthcare services.

Multi-Document Agents

   3. Space Exploration: Houston is also known for NASA's Johnson Space Center, highlighting the city's significant role in space exploration.
   4. Strong Economy: Houston's economy is robust and diverse, supported by industries such as energy, manufacturing, aeronautics, and transportation. This provides numerous economic opportunities for its residents.
   5. Arts and Culture: The city has a thriving visual and performing arts scene, with a theater district and a variety of museums and galleries. This makes Houston a vibrant place for art lovers and creatives.
   Overall, these aspects contribute to making Houston an appealing and dynamic city.
   # baseline
   response = base_query_engine.query(
       "Give me a summary of all the positive aspects of Houston"
   )
   print(str(response))
   Houston has several positive aspects that contribute to its reputation as a thriving city. It is home to a diverse and growing international community, with a large number of foreign banks and consular offices representing 92 countries. The city has received numerous accolades, including being ranked as one of the best cities for employment, college graduates, and homebuyers. Houston has a strong economy, with a broad industrial base in sectors such as energy, manufacturing, aeronautics, and healthcare. It is also a major center for the oil and gas industry and has the second-most Fortune 500 headquarters in the United States. The city's cultural scene is vibrant, with a variety of annual events celebrating different cultures, as well as a reputation for diverse and excellent food. Houston is known for its world-class museums and performing arts scene. Additionally, the city has made significant investments in renewable energy sources like wind and solar. Overall, Houston offers a high quality of life, reasonable living costs, and abundant employment opportunities.
   # baseline: the response doesn't quite match the sources...
   response.source_nodes[1].get_content()
   response = top_agent.query(
       "Tell the demographics of Houston, and then compare that with the demographics of Chicago"
   )
   === Calling Function ===
   Calling function: tool_Houston with args: {
     "input": "demographics"
   }
   === Calling Function ===
   Calling function: vector_tool with args: {
     "input": "demographics"
   }
   Got output: Houston is a majority-minority city with a diverse population. According to the U.S. Census Bureau, in 2019, non-Hispanic whites made up 23.3% of the population, Hispanics and Latino Americans 45.8%, Blacks or African Americans 22.4%, and Asian Americans 6.5%. The largest Hispanic or Latino American ethnic group in the city is Mexican Americans, followed by Puerto Ricans and Cuban Americans. Houston is also home to the largest African American community west of the Mississippi River. Additionally, Houston has a growing Muslim population, with Muslims estimated to make up 1.2% of the city's population. The city is known for its LGBT community and is home to one of the largest pride parades in the United States. The Hindu, Sikh, and Buddhist communities are also growing in Houston. Overall, Houston is considered one of the most ethnically and culturally diverse metropolitan areas in the country.
   ========================
   Got output: Houston is a majority-minority city with a diverse population. According to the U.S. Census Bureau, in 2019, non-Hispanic whites made up 23.3% of the population, Hispanics and Latino Americans 45.8%, Blacks or African Americans 22.4%, and Asian Americans 6.5%. The largest Hispanic or Latino American ethnic group in the city is Mexican Americans, followed by Puerto Ricans and Cuban Americans.
   Houston is also home to the largest African American community west of the Mississippi River. Additionally, Houston has a growing Muslim population, with Muslims estimated to make up 1.2% of the city's population. The city is known for its LGBT community and is home to one of the largest pride parades in the United States. The Hindu, Sikh, and Buddhist communities are also growing in Houston.

Multi-Document Agents

   Overall, Houston is considered one of the most ethnically and culturally diverse metropolitan areas in the country.
   ========================
   === Calling Function ===
   Calling function: tool_Chicago with args: {
     "input": "demographics"
   }
   === Calling Function ===
   Calling function: vector_tool with args: {
     "input": "demographics"
   }
   Got output: Chicago has a diverse demographic makeup. It experienced rapid population growth during its early years, becoming one of the fastest-growing cities in the world. Waves of immigrants from various European countries, as well as African Americans from the American South, contributed to the city's population growth. Over time, Chicago's population has fluctuated, with a decline in the latter half of the 20th century followed by a rise in recent years. As of the latest census estimates, the largest racial or ethnic groups in Chicago are non-Hispanic White, Black, and Hispanic. Additionally, Chicago has a significant LGBT population and is known for its cultural diversity.
   ========================
   Got output: Chicago is known for its diverse demographic makeup. The city experienced rapid population growth during its early years, with immigrants from various European countries and African Americans from the American South contributing significantly to this growth. Over time, the population has fluctuated, with a decline in the latter half of the 20th century followed by a rise in recent years.
   As per the latest census estimates, the largest racial or ethnic groups in Chicago are non-Hispanic White, Black, and Hispanic. The city also has a significant LGBT population and is celebrated for its cultural diversity.
   ========================
   print(response)
   Houston has a diverse population with a demographic makeup that includes non-Hispanic whites (23.3%), Hispanics and Latino Americans (45.8%), Blacks or African Americans (22.4%), and Asian Americans (6.5%). The largest Hispanic or Latino American ethnic group in Houston is Mexican Americans. Houston is also home to the largest African American community west of the Mississippi River and has a growing Muslim population.
   On the other hand, Chicago is also known for its diverse demographics. The city has a significant non-Hispanic White population, along with a substantial Black population and Hispanic population. Chicago is celebrated for its cultural diversity and has a significant LGBT population.
   Both Houston and Chicago have diverse populations, with a mix of different racial and ethnic groups contributing to their vibrant communities.
   # baseline
   response = base_query_engine.query(
       "Tell the demographics of Houston, and then compare that with the demographics of Chicago"
   )
   print(str(response))
   Houston is the most populous city in Texas and the fourth-most populous city in the United States. It has a population of 2,304,580 as of the 2020 U.S. census. The city is known for its diversity, with a significant proportion of minorities. In 2019, non-Hispanic whites made up 23.3% of the population, Hispanics and Latino Americans 45.8%, Blacks or African Americans 22.4%, and Asian Americans 6.5%. The largest Hispanic or Latino American ethnic group in Houston is Mexican Americans, comprising 31.6% of the population.
   In comparison, Chicago is the third-most populous city in the United States. According to the 2020 U.S. census, Chicago has a population of 2,746,388. The demographics of Chicago are different from Houston, with non-Hispanic whites making up 32.7% of the population, Hispanics and Latino Americans 29.9%, Blacks or African Americans 29.8%, and Asian Americans 7.6%. The largest Hispanic or Latino American ethnic group in Chicago is Mexican Americans, comprising 21.6% of the population.
   Overall, both Houston and Chicago have diverse populations, but the specific demographic composition differs between the two cities.
   # baseline: the response tells you nothing about Chicago...

Multi-Document Agents

   response.source_nodes[3].get_content()
   response = top_agent.query(
       "Tell me the differences between Shanghai and Beijing in terms of history and current economy"
   )
   === Calling Function ===
   Calling function: tool_Shanghai with args: {
     "input": "history"
   }
   === Calling Function ===
   Calling function: vector_tool with args: {
     "input": "history"
   }
   Got output: Shanghai has a rich history that dates back to ancient times. However, in the context provided, the history of Shanghai is mainly discussed in relation to its modern development. After the war, Shanghai's economy experienced significant growth, with increased agricultural and industrial output. The city's administrative divisions were rearranged, and it became a center for radical leftism during the 1950s and 1960s. The Cultural Revolution had a severe impact on Shanghai's society, but the city maintained economic production with a positive growth rate. Shanghai also played a significant role in China's Third Front campaign and has been a major contributor of tax revenue to the central government. Economic reforms were initiated in Shanghai in 1990, leading to the development of the Pudong district and its classification as an Alpha+ city.
   ========================
   Got output: Shanghai's history is rich and complex, dating back to ancient times. However, its modern development is particularly noteworthy. After the war, Shanghai experienced significant economic growth, with a boost in both agricultural and industrial output. The city's administrative divisions were restructured, and it became a hub for radical leftism during the 1950s and 1960s.
   The Cultural Revolution had a profound impact on Shanghai's society, but despite this, the city managed to maintain economic production with a positive growth rate. Shanghai also played a significant role in China's Third Front campaign and has been a major contributor of tax revenue to the central government.
   In 1990, economic reforms were initiated in Shanghai, leading to the development of the Pudong district. This has helped Shanghai to be classified as an Alpha+ city, indicating its influence on the global economic stage.
   ========================
   === Calling Function ===
   Calling function: tool_Beijing with args: {
     "input": "history"
   }
   === Calling Function ===
   Calling function: vector_tool with args: {
     "input": "history"
   }
   Got output: Beijing has a rich history that spans several dynasties. It was the capital of the Ming dynasty, during which the city took its current shape and many of its major attractions, such as the Forbidden City and the Temple of Heaven, were constructed. The Qing dynasty succeeded the Ming dynasty and made Beijing its sole capital. During this time, the Imperial residence and the general layout of the city remained largely unchanged. However, the city faced challenges during the Second Opium War and the Boxer Rebellion, resulting in the looting and destruction of important structures. In the early 20th century, Beijing saw the signing of a peace agreement between the Eight-Nation Alliance and the Chinese government, which led to the restoration of Qing dynasty rule. However, the dynasty eventually collapsed in 1911.
   ========================
   Got output: Beijing has a rich and complex history that spans several dynasties. It served as the capital during the Ming dynasty, during which the city took its current shape and many of its major attractions, such as the Forbidden City and the Temple of Heaven, were constructed. The Qing dynasty succeeded the Ming dynasty and made Beijing its sole capital. During this time, the Imperial residence and the general layout of the city remained largely unchanged.
   However, the city faced significant challenges during the Second Opium War and the Boxer Rebellion, which resulted in the looting and destruction of important structures. In the early 20th century, Beijing saw the signing of a peace agreement between the Eight-Nation Alliance and the Chinese government, leading to the restoration of Qing dynasty rule. However, the dynasty eventually collapsed in 1911. Despite these tumultuous events, Beijing has managed to preserve its historical heritage while also evolving into a modern metropolis.

Multi-Document Agents

   ========================
   === Calling Function ===
   Calling function: tool_Shanghai with args: {
     "input": "current economy"
   }
   === Calling Function ===
   Calling function: vector_tool with args: {
     "input": "current economy"
   }
   Got output: The current economy of Shanghai is strong and thriving. It is a global center for finance and innovation, and a national center for commerce, trade, and transportation. The city has a diverse economy, with its six largest industries comprising about half of its GDP. Shanghai has experienced rapid development and has been one of the fastest-developing cities in the world. It has recorded double-digit GDP growth in almost every year between 1992 and 2008. As of 2021, Shanghai had a GDP of CN¥4.46 trillion ($1.106 trillion in PPP), making it one of the wealthiest cities in China. It is also the most expensive city in mainland China to live in. Shanghai is a major player in the global financial industry, ranking first in Asia and third globally in the Global Financial Centres Index. It is home to the Shanghai Stock Exchange, the largest stock exchange in China and the fourth-largest in the world. The city has attracted significant foreign investment and has been a hub for the technology industry and startups. Overall, the current economy of Shanghai is robust and continues to grow.
   ========================
   Got output: The current economy of Shanghai is robust and thriving. It is a global center for finance and innovation, and a national center for commerce, trade, and transportation. The city has a diverse economy, with its six largest industries comprising about half of its GDP.
   Shanghai has experienced rapid development and has been one of the fastest-developing cities in the world. It has recorded double-digit GDP growth in almost every year between 1992 and 2008. As of 2021, Shanghai had a GDP of CN¥4.46 trillion ($1.106 trillion in PPP), making it one of the wealthiest cities in China.
   Shanghai is also the most expensive city in mainland China to live in. It is a major player in the global financial industry, ranking first in Asia and third globally in the Global Financial Centres Index. The city is home to the Shanghai Stock Exchange, the largest stock exchange in China and the fourth-largest in the world.
   The city has attracted significant foreign investment and has been a hub for the technology industry and startups. Overall, the current economy of Shanghai is robust and continues to grow.
   ========================
   === Calling Function ===
   Calling function: tool_Beijing with args: {
     "input": "current economy"
   }
   === Calling Function ===
   Calling function: vector_tool with args: {
     "input": "current economy"
   }
   Got output: The current economy of Beijing is dominated by the tertiary sector, which includes services such as professional services, wholesale and retail, information technology, commercial real estate, scientific research, and residential real estate. This sector generated 83.8% of the city's output in 2022. The secondary sector, which includes manufacturing and construction, accounted for 15.8% of output, while the primary sector, which includes agriculture and mining, contributed only 0.26%. The city has also identified six high-end economic output zones that are driving local economic growth, including Zhongguancun, Beijing Financial Street, Beijing Central Business District (CBD), Beijing Economic and Technological Development Area (Yizhuang), Beijing Airport Economic Zone, and Beijing Olympic Center Zone. These zones are home to various industries and sectors, such as technology companies, financial institutions, office buildings, industrial parks, and entertainment and sports centers.
   ========================
   Got output: The current economy of Beijing is primarily driven by the tertiary sector, which includes services such as professional services, wholesale and retail, information technology, commercial real estate, scientific research, and residential real estate. This sector generated 83.8% of the city's output in 2022. The secondary sector, which includes manufacturing and construction, accounted for 15.8% of output, while the primary sector, which includes agriculture and mining, contributed only 0.26%.

Multi-Document Agents

   Beijing has also identified six high-end economic output zones that are driving local economic growth. These include Zhongguancun, Beijing Financial Street, Beijing Central Business District (CBD), Beijing Economic and Technological Development Area (Yizhuang), Beijing Airport Economic Zone, and Beijing Olympic Center Zone. These zones are home to various industries and sectors, such as technology companies, financial institutions, office buildings, industrial parks, and entertainment and sports centers.
   ========================
   print(str(response))
   In terms of history, both Shanghai and Beijing have rich and complex pasts. Shanghai's history dates back to ancient times, but its modern development is particularly noteworthy. It experienced significant economic growth after the war and played a major role in China's economic reforms. Beijing, on the other hand, has a history that spans several dynasties and served as the capital during the Ming and Qing dynasties. It has preserved its historical heritage while evolving into a modern metropolis.
   In terms of current economy, Shanghai is a global center for finance and innovation. It has a diverse economy and has experienced rapid development, with a high GDP and significant foreign investment. It is a major player in the global financial industry and is home to the Shanghai Stock Exchange. Beijing's economy is primarily driven by the tertiary sector, with a focus on services such as professional services, information technology, and commercial real estate. It has identified high-end economic output zones that are driving local economic growth.
   Overall, both cities have thriving economies, but Shanghai has a stronger focus on finance and global influence, while Beijing has a diverse economy with a focus on services and high-end economic zones.
   # baseline
   response = base_query_engine.query(
       "Tell me the differences between Shanghai and Beijing in terms of history and current economy"
   )
   print(str(response))
   Shanghai and Beijing have distinct differences in terms of history and current economy. Historically, Shanghai was the largest and most prosperous city in East Asia during the 1930s, while Beijing served as the capital of the Republic of China and later the People's Republic of China. Shanghai experienced significant growth and redevelopment in the 1990s, while Beijing expanded its urban area and underwent rapid development in the last two decades.
   In terms of the current economy, Shanghai is considered the "showpiece" of China's booming economy. It is a global center for finance and innovation, with a strong focus on industries such as retail, finance, IT, real estate, machine manufacturing, and automotive manufacturing. Shanghai is also home to the world's busiest container port, the Port of Shanghai. The city has a high GDP and is classified as an Alpha+ city by the Globalization and World Cities Research Network.
   On the other hand, Beijing is a global financial center and ranks third globally in the Global Financial Centres Index. It is also a hub for the Chinese and global technology industry, with a large startup ecosystem. Beijing has a strong presence in industries such as finance, technology, and pharmaceuticals. The city is home to the headquarters of large state banks and insurance companies, as well as the country's financial regulatory agencies.
   Overall, while both Shanghai and Beijing are important economic centers in China, Shanghai has a stronger focus on industries such as finance, retail, and manufacturing, while Beijing has a strong presence in finance, technology, and pharmaceuticals.


Fine Tuning GPT-3.5-Turbo

In this notebook, we walk through an example of fine-tuning
gpt-3.5-turbo.
Specifically, we attempt to distill GPT-4's knowledge, by generating
training data with GPT-4 to then fine-tune GPT-3.5.
All training data is generated using two different sections of our
index data, creating both a training and evalution set.
We then finetune with our "OpenAIFinetuneEngine" wrapper abstraction.
Evaluation is done using the "ragas" library, which we will detail
later on.
   # !pip install llama-index pypdf sentence-transformers ragas
   import os
   import openai
   os.environ["OPENAI_API_KEY"] = "sk-..."
   openai.api_key = os.environ["OPENAI_API_KEY"]
Data Setup
Here, we first down load the PDF that we will use to generate training
data.
   !curl https://www.ipcc.ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
   100 20.7M  100 20.7M    0     0   397k      0  0:00:53  0:00:53 --:--:--  417k84k      0  0:00:55  0:00:24  0:00:31  406k    0   395k      0  0:00:53  0:00:48  0:00:05  403k0   396k      0  0:00:53  0:00:53 --:--:--  406k
The next step is generating a training and eval dataset.
We will generate 40 questions on different sections of the PDF we
downloaded.
We can use GPT-3.5 on the eval questions to get our baseline
performance.
Then, we will use GPT-4 on the train questions to generate our
training data. The training data will be collected with out
"OpenAIFineTuningHandler".
This step is entirely optional if you don't want to spend the
time/tokens -- the eval and training questions are also provided in
this folder, as well as the training data!
Train Generation
   from llama_index import SimpleDirectoryReader, ServiceContext
   from llama_index.llms import OpenAI
   from llama_index.evaluation import DatasetGenerator
   documents = SimpleDirectoryReader(
       input_files=["IPCC_AR6_WGII_Chapter03.pdf"]
   ).load_data()
   # Shuffle the documents
   import random
   random.seed(42)
   random.shuffle(documents)
   gpt_35_context = ServiceContext.from_defaults(
       llm=OpenAI(model="gpt-3.5-turbo", temperature=0.3)
   )
   question_gen_query = (
       "You are a Teacher/ Professor. Your task is to setup "
       "a quiz/examination. Using the provided context, formulate "
       "a single question that captures an important fact from the "
       "context. Restrict the question to the context information provided."
   )
   dataset_generator = DatasetGenerator.from_documents(
       documents[:50],
       question_gen_query=question_gen_query,
       service_context=gpt_35_context,
   )
   # NOTE: this may take some time. Go grab a coffee!
   questions = dataset_generator.generate_questions_from_nodes(num=40)
   print("Generated ", len(questions), " questions")

Fine Tuning GPT-3.5-Turbo

   Generated  40  questions
   with open("train_questions.txt", "w") as f:
       for question in questions:
           f.write(question + "\n")
Eval Generation
Now, lets generate questions on a completely different set of
documents, in order to create our eval dataset.
   dataset_generator = DatasetGenerator.from_documents(
       documents[
           50:
       ],  # since we generated ~1 question for 40 documents, we can skip the first 40
       question_gen_query=question_gen_query,
       service_context=gpt_35_context,
   )
   # NOTE: this may take some time. Go grab a coffee!
   questions = dataset_generator.generate_questions_from_nodes(num=40)
   print("Generated ", len(questions), " questions")
   Generated  40  questions
   with open("eval_questions.txt", "w") as f:
       for question in questions:
           f.write(question + "\n")
Initial Eval with GPT-3.5-Turbo Query Engine
For this eval, we will be using the "ragas" evaluation library.
Ragas has a ton of evaluation metrics for RAG pipelines, and you can
read about them here.
For this notebook, we will be using the following two metrics
* "answer_relevancy" - This measures how relevant is the generated
  answer to the prompt. If the generated answer is incomplete or
  contains redundant information the score will be low. This is
  quantified by working out the chance of an LLM generating the given
  question using the generated answer. Values range (0,1), higher the
  better.
* "faithfulness" - This measures the factual consistency of the
  generated answer against the given context. This is done using a
  multi step paradigm that includes creation of statements from the
  generated answer followed by verifying each of these statements
  against the context. The answer is scaled to (0,1) range. Higher the
  better.
   questions = []
   with open("eval_questions.txt", "r") as f:
       for line in f:
           questions.append(line.strip())
   from llama_index import VectorStoreIndex
   # limit the context window to 2048 tokens so that refine is used
   gpt_35_context = ServiceContext.from_defaults(
       llm=OpenAI(model="gpt-3.5-turbo", temperature=0.3), context_window=2048
   )
   index = VectorStoreIndex.from_documents(documents, service_context=gpt_35_context)
   query_engine = index.as_query_engine(similarity_top_k=2)
   contexts = []
   answers = []
   for question in questions:
       response = query_engine.query(question)
       contexts.append([x.node.get_content() for x in response.source_nodes])
       answers.append(str(response))
   from datasets import Dataset
   from ragas import evaluate
   from ragas.metrics import answer_relevancy, faithfulness
   ds = Dataset.from_dict(
       {
           "question": questions,
           "answer": answers,
           "contexts": contexts,
       }
   )
   result = evaluate(ds, [answer_relevancy, faithfulness])
   print(result)
   evaluating with [answer_relevancy]
   100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [01:02<00:00, 20.69s/it]
   evaluating with [faithfulness]
   100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [03:52<00:00, 77.37s/it]

Fine Tuning GPT-3.5-Turbo

{'role': 'user', 'content': 'Context information is below.\n---------------------\npage_label: 410\nfile_name: IPCC_AR6_WGII_Chapter03.pdf\n\nIt is challenging to apply this experimental approach to communities or ecosystems (see Figure \nBox\xa03.1.1).To date, most research on community or ecosystem response to climate-induced drivers has been in large-volume (>10,000 l) \nmesocosms (Riebesell and Gattuso, 2014), or at natural analogues such as CO 2 seeps, in which only one driver (ocean acidification) is \naltered (see (4) in Figure Box\xa03.1.1).Only very recently have two drivers been incorporated into climate-change manipulation studies \nexamining responses of primary producers to secondary consumers (see (5) in Figure Box\xa03.1.1a; Nagelkerken et\xa0al., 2020).Therefore, \n‘natural experiments’ from the geological past (Reddin et\xa0al., 2020) provide insights into how food webs and their constituents respond to \ncomplex change involving multiple drivers.Contemporary observations are occasionally long enough (>50\xa0years) to capture community \nresponses to complex climate change.For example, Brun et\xa0al.(2019) reported a shift in zooplankton community structure in the North \nAtlantic (1960–2014), with major biogeochemical ramifications.Conducting sufficiently long manipulation experiments to study the effect of adaptation on organisms is equally difficult (see Figure \nBox\xa03.1.1b), with much research restricted to multi-year studies of the microevolution of fast-growing (more than one division per day) \nphytoplankton species responding to single drivers (Lohbeck et\xa0al., 2012; Schaum et\xa0al., 2016).In a few experimental evolution studies \n(see (7) in Figure Box\xa03.1.1a; Brennan et\xa0al., 2017), multiple drivers have been used, but none have used communities or ecosystems (see \nFigure Box\xa03.1.1b).Nevertheless, the fossil record provides limited evidence of adaptations to less rapid (relative to present day) climate \nchange (Jackson et\xa0al., 2018).Despite the need to explore ecological or biogeochemical responses to projected future ocean conditions, \nlogistical challenges require that assessments of climate-change impacts at scales larger than mesocosms use large-scale, long-term in \nsitu observational studies (as documented in Section\xa03.4).\n\npage_label: 409\nfile_name: IPCC_AR6_WGII_Chapter03.pdf\n\n3\n409Oceans and Coastal Ecosystems and Their Services  Chapter 3\nunderlies inhibited thermal adaptation under nitrogen-limited \nconditions (low confidence) (Aranguren-Gassis et\xa0 al., 2019).When \nselection is strong due to unfavourable environmental conditions, \nmicrobial populations can encounter functional and evolutionary \ntrade-offs evidenced by reducing growth rates while increasing \ntolerance and metabolism of reactive oxygen species (Lindberg and \nCollins, 2020).Other trade-offs can be observed in offspring quality \nand number (Lindberg and Collins, 2020).These findings contribute \ntowards a mechanistic framework describing the range of evolutionary \nstrategies in response to multiple drivers (Collins et\xa0al., 2020), but other \nhazards, such as extreme events (e.g., MHWs), still nee

Fine Tuning GPT-3.5-Turbo

   {'ragas_score': 0.8356, 'answer_relevancy': 0.9725, 'faithfulness': 0.7325}
GPT-4 to Collect Training Data
Here, we use GPT-4 and the "OpenAIFineTuningHandler" to collect data
that we want to train on.
   from llama_index import ServiceContext
   from llama_index.llms import OpenAI
   from llama_index.callbacks import OpenAIFineTuningHandler
   from llama_index.callbacks import CallbackManager
   finetuning_handler = OpenAIFineTuningHandler()
   callback_manager = CallbackManager([finetuning_handler])
   gpt_4_context = ServiceContext.from_defaults(
       llm=OpenAI(model="gpt-4", temperature=0.3),
       context_window=2048,  # limit the context window artifically to test refine process
       callback_manager=callback_manager,
   )
   questions = []
   with open("train_questions.txt", "r") as f:
       for line in f:
           questions.append(line.strip())
   from llama_index import VectorStoreIndex
   index = VectorStoreIndex.from_documents(documents, service_context=gpt_4_context)
   query_engine = index.as_query_engine(similarity_top_k=2)
   for question in questions:
       response = query_engine.query(question)
Create "OpenAIFinetuneEngine"
We create an "OpenAIFinetuneEngine": the finetune engine will take
care of launching a finetuning job, and returning an LLM model that
you can directly plugin to the rest of LlamaIndex workflows.
We use the default constructor, but we can also directly pass in our
finetuning_handler into this engine with the "from_finetuning_handler"
class method.
   finetuning_handler.save_finetuning_events("finetuning_events.jsonl")
   from llama_index.finetuning import OpenAIFinetuneEngine
   finetune_engine = OpenAIFinetuneEngine(
       "gpt-3.5-turbo",
       "finetuning_events.jsonl",
       # start_job_id="<start-job-id>"  # if you have an existing job, can specify id here
   )
   # finetune_engine = OpenAIFinetuneEngine.from_finetuning_handler(
   #     finetuning_handler,
   #     "gpt-3.5-turbo",
   #     "tmp.jsonl"
   # )
   finetune_engine.finetune()
   Num examples: 61
   First example:
   {'role': 'system', 'content': "You are an expert Q&A system that is trusted around the world.\nAlways answer the query using the provided context information, and not prior knowledge.\nSome rules to follow:\n1. Never directly reference the given context in your answer.\n2. Avoid statements like 'Based on the context, ...' or 'The context information ...' or anything along those lines."}
   {'role': 'assistant', 'content': 'Several approaches are used to assess ecological responses to multiple climate-induced drivers. These include laboratory- and field-based experiments, field observations such as natural gradients and climate analogues, the study of paleo-analogues, and the development of mechanistic and empirical models. Experimental studies often focus on individual drivers, but recent manipulations have used large-volume mesocosms to explore ecological responses to both warming and acidification. Observations from time series longer than modes of natural variability are essential for revealing and attributing ecological responses to climate change. Paleorecords also provide insights into the influence of multiple drivers on marine biota. Multi-species and integrated end-to-end ecosystem models are powerful tools to explore and project outcomes to the often-interacting cumulative effects of climate change and other anthropogenic drivers. These models can integrate some aspects of the knowledge accrued from manipulation experiments, paleo- and contemporary observations, help test the relative importance of specific drivers and driver combinations, and identify synergistic or antagonistic responses.'}

Fine Tuning GPT-3.5-Turbo

   No errors found
   Num examples missing system message: 21
   Num examples missing user message: 0
   #### Distribution of num_messages_per_example:
   min / max: 2, 3
   mean / median: 2.6557377049180326, 3.0
   p5 / p95: 2.0, 3.0
   #### Distribution of num_total_tokens_per_example:
   min / max: 229, 2011
   mean / median: 1274.27868852459, 1385.0
   p5 / p95: 533.0, 1848.0
   #### Distribution of num_assistant_tokens_per_example:
   min / max: 11, 334
   mean / median: 72.36065573770492, 37.0
   p5 / p95: 23.0, 193.0
   0 examples may be over the 4096 token limit, they will be truncated during fine-tuning
   Dataset has ~77731 tokens that will be charged for during training
   By default, you'll train for 3 epochs on this dataset
   By default, you'll be charged for ~233193 tokens
   As of Augest 22, 2023, fine-tuning gpt-3.5-turbo is $0.008 / 1K Tokens.
   This means your total cost for training will be $0.621848 per epoch.
   Waiting for file to be ready...
   finetune_engine.get_current_job()
   <FineTuningJob fine_tuning.job id=ftjob-u9T7BF5zRxVX4n5b9Jtbb5cR at 0x2c641fe20> JSON: {
     "object": "fine_tuning.job",
     "id": "ftjob-u9T7BF5zRxVX4n5b9Jtbb5cR",
     "model": "gpt-3.5-turbo-0613",
     "created_at": 1693254044,
     "finished_at": null,
     "fine_tuned_model": null,
     "organization_id": "org-1ZDAvajC6v2ZtAP9hLEIsXRz",
     "result_files": [],
     "status": "running",
     "validation_file": null,
     "training_file": "file-j1fwmqIAoqZXWZQ8EqwHucXs",
     "hyperparameters": {
       "n_epochs": 3
     },
     "trained_tokens": null
   }
   ft_llm = finetune_engine.get_finetuned_model(temperature=0.3)
Evaluation
After some time, your model will be done training!
The next step is running our fine-tuned model on our eval dataset
again to measure any performance increase.
   from llama_index import ServiceContext
   from llama_index.llms import OpenAI
   from llama_index.callbacks import OpenAIFineTuningHandler
   from llama_index.callbacks import CallbackManager
   # Option 1: pass in ft_llm directly into ServiceContext
   ft_context = ServiceContext.from_defaults(
       llm=ft_llm,
       context_window=2048,  # limit the context window artifically to test refine process
   )
   # # Option 2: you can also specify the model name manually
   # ft_model_name = "ft:gpt-3.5-turbo-0613:..."
   # ft_context = ServiceContext.from_defaults(
   #     llm=OpenAI(model=ft_model_name, temperature=0.3),
   #     context_window=2048,  # limit the context window artifically to test refine process

Fine Tuning GPT-3.5-Turbo

   # )
   questions = []
   with open("eval_questions.txt", "r") as f:
       for line in f:
           questions.append(line.strip())
   from llama_index import VectorStoreIndex
   index = VectorStoreIndex.from_documents(documents, service_context=ft_context)
   query_engine = index.as_query_engine(similarity_top_k=2)
   contexts = []
   answers = []
   for question in questions:
       response = query_engine.query(question)
       contexts.append([x.node.get_content() for x in response.source_nodes])
       answers.append(str(response))
   from datasets import Dataset
   from ragas import evaluate
   from ragas.metrics import answer_relevancy, faithfulness
   ds = Dataset.from_dict(
       {
           "question": questions,
           "answer": answers,
           "contexts": contexts,
       }
   )
   result = evaluate(ds, [answer_relevancy, faithfulness])
   print(result)
   evaluating with [answer_relevancy]
   100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [00:49<00:00, 16.34s/it]
   evaluating with [faithfulness]
   100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 3/3 [04:04<00:00, 81.44s/it]
   {'ragas_score': 0.8680, 'answer_relevancy': 0.9607, 'faithfulness': 0.7917}
Exploring Differences
Let's quickly compare the differences in responses, to demonstrate
that fine tuning did indeed change something.
   from llama_index import VectorStoreIndex
   index = VectorStoreIndex.from_documents(documents)
   questions = []
   with open("eval_questions.txt", "r") as f:
       for line in f:
           questions.append(line.strip())
   print(questions[12])
   What is a key barrier globally for ocean health, governance, and adaptation to climate change, according to the report?
Original
   from llama_index.response.notebook_utils import display_response
   from llama_index import ServiceContext
   from llama_index.llms import OpenAI
   gpt_35_context = ServiceContext.from_defaults(
       llm=OpenAI(model="gpt-3.5-turbo", temperature=0.3),
       context_window=2048,  # limit the context window artifically to test refine process
   )
   query_engine = index.as_query_engine(service_context=gpt_35_context)
   response = query_engine.query(questions[12])
   display_response(response)
**"Final Response:"** A key barrier globally for ocean health,
governance, and adaptation to climate change, according to the report,
is the availability of technology, knowledge, and financial support,
as well as existing governance structures.
Fine-Tuned
   from llama_index import ServiceContext
   from llama_index.llms import OpenAI
   ft_context = ServiceContext.from_defaults(
       llm=ft_llm,
       context_window=2048,  # limit the context window artifically to test refine process
   )
   query_engine = index.as_query_engine(service_context=ft_context)
   response = query_engine.query(questions[12])
   display_response(response)
**"Final Response:"** The report identifies a broad range of barriers
and limits for adaptation to climate change in ecosystems and human
systems. These include the availability of technology, knowledge, and
financial support, as well as existing governance structures. Existing
ocean-governance structures are already facing multi-dimensional,
scale-related challenges because of climate change.

Fine Tuning GPT-3.5-Turbo

As we can see, the fine-tuned model provides a more thorough response!
This lines up with the increased faithfullness score from ragas, since
the answer is more representative of the retrieved context.
Conclusion
So, in conclusion, finetuning with only ~61 questions actually helped
improve our eval scores!
**answer_relevancy: 0.9725 -> 0.9607**
The answer relevancy dips slightly but it's very small.
**faithfulness: 0.7325 -> 0.7917**
The faithfulness appears to have been improved! This mains the anwers
given better fuffil the original question that was asked.


Fine Tuning with Function Calling

In this notebook, we walk through how to fine-tuning gpt-3.5-turbo
with function calls. The primary use case here is structured data
extraction. Our main focus is distilling GPT-4 outputs to help improve
gpt-3.5-turbo function calling capabilities.
We will walk through some examples, from simple to advanced:
1. Fine-tuning on some toy messages/structured outputs logged through
   our OpenAI Pydantic Program object.
2. Fine-tuning on context-augmented queries/structured outputs over an
   entire document corpus. Use this in a RAG system.
   import nest_asyncio
   nest_asyncio.apply()
   import os
   import openai
   os.environ["OPENAI_API_KEY"] = "sk-..."
   openai.api_key = os.environ["OPENAI_API_KEY"]
Fine-tuning Using GPT-4 Pydantic Programs
In this section we show how to log inputs/outputs through our low-
level Pydantic Program module. We use that dataset to fine-tune an
LLM.
Defining Pydantic Model + Program
Here, we define the GPT-4 powered function calling program that will
generate structured outputs into a Pydantic object (an Album).
   from llama_index.program import OpenAIPydanticProgram
   from pydantic import BaseModel
   from llama_index.llms import OpenAI
   from llama_index.callbacks import OpenAIFineTuningHandler
   from llama_index.callbacks import CallbackManager
   from typing import List
   class Song(BaseModel):
       """Data model for a song."""
       title: str
       length_seconds: int
   class Album(BaseModel):
       """Data model for an album."""
       name: str
       artist: str
       songs: List[Song]
   finetuning_handler = OpenAIFineTuningHandler()
   callback_manager = CallbackManager([finetuning_handler])
   llm = OpenAI(model="gpt-4", callback_manager=callback_manager)
   prompt_template_str = """\
   Generate an example album, with an artist and a list of songs. \
   Using the movie {movie_name} as inspiration.\
   """
   program = OpenAIPydanticProgram.from_defaults(
       output_cls=Album,
       prompt_template_str=prompt_template_str,
       llm=llm,
       verbose=False,
   )
Log Inputs/Outputs
We define some sample movie names as inputs and log the outputs
through the function calling program.
   # NOTE: we need >= 10 movies to use OpenAI fine-tuning
   movie_names = [
       "The Shining",
       "The Departed",
       "Titanic",
       "Goodfellas",
       "Pretty Woman",
       "Home Alone",
       "Caged Fury",
       "Edward Scissorhands",
       "Total Recall",
       "Ghost",
       "Tremors",
       "RoboCop",
       "Rocky V",
   ]
   from tqdm.notebook import tqdm
   for movie_name in tqdm(movie_names):
       output = program(movie_name=movie_name)
       print(output.json())
     0%|          | 0/13 [00:00<?, ?it/s]
   {"name": "The Shining", "artist": "Various Artists", "songs": [{"title": "Main Title", "length_seconds": 180}, {"title": "Opening Credits", "length_seconds": 120}, {"title": "The Overlook Hotel", "length_seconds": 240}, {"title": "Redrum", "length_seconds": 150}, {"title": "Here's Johnny!", "length_seconds": 200}]}
   {"name": "The Departed Soundtrack", "artist": "Various Artists", "songs": [{"title": "Gimme Shelter", "length_seconds": 272}, {"title": "Comfortably Numb", "length_seconds": 383}, {"title": "I'm Shipping Up to Boston", "length_seconds": 166}, {"title": "Sweet Dreams (Are Made of This)", "length_seconds": 216}, {"title": "I'm Shipping Up to Boston (Instrumental)", "length_seconds": 166}, {"title": "The Departed Tango", "length_seconds": 123}, {"title": "Thief's Theme", "length_seconds": 201}, {"title": "Well Well Well", "length_seconds": 126}, {"title": "Comfortably Numb (Live)", "length_seconds": 383}, {"title": "Sail On, Sailor", "length_seconds": 181}]}

Fine Tuning with Function Calling

   {"name": "Titanic Soundtrack", "artist": "James Horner", "songs": [{"title": "My Heart Will Go On", "length_seconds": 273}, {"title": "Rose", "length_seconds": 120}, {"title": "Hymn to the Sea", "length_seconds": 365}, {"title": "Southampton", "length_seconds": 180}, {"title": "Take Her to Sea, Mr. Murdoch", "length_seconds": 150}]}
   {"name": "Goodfellas Soundtrack", "artist": "Various Artists", "songs": [{"title": "Rags to Riches", "length_seconds": 180}, {"title": "Gimme Shelter", "length_seconds": 270}, {"title": "Layla", "length_seconds": 270}, {"title": "Jump into the Fire", "length_seconds": 240}, {"title": "Atlantis", "length_seconds": 180}, {"title": "Beyond the Sea", "length_seconds": 180}, {"title": "Sunshine of Your Love", "length_seconds": 240}, {"title": "Mannish Boy", "length_seconds": 240}, {"title": "Layla (Piano Exit)", "length_seconds": 120}]}
   {"name": "Pretty Woman Soundtrack", "artist": "Various Artists", "songs": [{"title": "Oh, Pretty Woman", "length_seconds": 178}, {"title": "King of Wishful Thinking", "length_seconds": 253}, {"title": "It Must Have Been Love", "length_seconds": 250}, {"title": "Show Me Your Soul", "length_seconds": 285}, {"title": "No Explanation", "length_seconds": 244}]}
   {"name": "Home Alone Soundtrack", "artist": "John Williams", "songs": [{"title": "Somewhere in My Memory", "length_seconds": 180}, {"title": "Holiday Flight", "length_seconds": 120}, {"title": "The House", "length_seconds": 150}, {"title": "Star of Bethlehem", "length_seconds": 135}, {"title": "Setting the Trap", "length_seconds": 165}, {"title": "The Attack on the House", "length_seconds": 200}, {"title": "Mom Returns and Finale", "length_seconds": 240}]}
   {"name": "Caged Fury", "artist": "The Fury Band", "songs": [{"title": "Caged Fury", "length_seconds": 240}, {"title": "Prison Break", "length_seconds": 180}, {"title": "Behind Bars", "length_seconds": 210}, {"title": "Escape Plan", "length_seconds": 195}, {"title": "Fight for Freedom", "length_seconds": 220}]}
   {"name": "Edward Scissorhands Soundtrack", "artist": "Danny Elfman", "songs": [{"title": "Introduction", "length_seconds": 120}, {"title": "Ice Dance", "length_seconds": 180}, {"title": "Edwardo the Barber", "length_seconds": 150}, {"title": "The Grand Finale", "length_seconds": 240}]}
   {"name": "Total Recall", "artist": "Various Artists", "songs": [{"title": "Recall", "length_seconds": 240}, {"title": "Mars", "length_seconds": 180}, {"title": "Memory", "length_seconds": 210}, {"title": "Rebellion", "length_seconds": 300}, {"title": "Escape", "length_seconds": 270}]}
   {"name": "Ghost", "artist": "Various Artists", "songs": [{"title": "Unchained Melody", "length_seconds": 218}, {"title": "Oh My Love", "length_seconds": 156}, {"title": "Ditto's Theme", "length_seconds": 92}, {"title": "Love Inside", "length_seconds": 180}, {"title": "Ghostly Encounter", "length_seconds": 120}]}

Fine Tuning with Function Calling

   {"name": "Tremors Soundtrack", "artist": "Various Artists", "songs": [{"title": "Main Theme", "length_seconds": 180}, {"title": "Graboids Attack", "length_seconds": 240}, {"title": "Val and Earl's Theme", "length_seconds": 200}, {"title": "Burt's Arsenal", "length_seconds": 220}, {"title": "Nest of the Graboids", "length_seconds": 190}]}
   {"name": "RoboCop: The Soundtrack", "artist": "Various Artists", "songs": [{"title": "Main Theme", "length_seconds": 180}, {"title": "Murphy's Death", "length_seconds": 240}, {"title": "RoboCop's Training", "length_seconds": 210}, {"title": "ED-209", "length_seconds": 195}, {"title": "Clarence Boddicker", "length_seconds": 220}, {"title": "RoboCop Saves the Day", "length_seconds": 240}, {"title": "RoboCop's Theme", "length_seconds": 180}]}
   {"name": "Rocky V", "artist": "Various Artists", "songs": [{"title": "Measure of a Man", "length_seconds": 240}, {"title": "Can't Stop the Fire", "length_seconds": 210}, {"title": "Go for It!", "length_seconds": 180}, {"title": "Take You Back (Home Sweet Home)", "length_seconds": 200}, {"title": "The Measure of a Man (Reprise)", "length_seconds": 120}]}
   finetuning_handler.save_finetuning_events("mock_finetune_songs.jsonl")
   Wrote 14 examples to mock_finetune_songs.jsonl
   !cat mock_finetune_songs.jsonl
Fine-tune on the Dataset
We now define a fine-tuning engine and fine-tune on the mock dataset.
   from llama_index.finetuning import OpenAIFinetuneEngine
   finetune_engine = OpenAIFinetuneEngine(
       "gpt-3.5-turbo",
       "mock_finetune_songs.jsonl",
       # start_job_id="<start-job-id>"  # if you have an existing job, can specify id here
       validate_json=False,  # openai validate json code doesn't support function calling yet
   )
   finetune_engine.finetune()
   finetune_engine.get_current_job()
   <FineTuningJob fine_tuning.job id=ftjob-uJ9kQ9pI0p0YNatBDxF3VITv at 0x172a5c9a0> JSON: {
     "object": "fine_tuning.job",
     "id": "ftjob-uJ9kQ9pI0p0YNatBDxF3VITv",
     "model": "gpt-3.5-turbo-0613",
     "created_at": 1696463378,
     "finished_at": 1696463749,
     "fine_tuned_model": "ft:gpt-3.5-turbo-0613:llamaindex::8660TXqx",
     "organization_id": "org-1ZDAvajC6v2ZtAP9hLEIsXRz",
     "result_files": [
       "file-Hbpw15BAwyf3e4HK5Z9g4IK2"
     ],
     "status": "succeeded",
     "validation_file": null,
     "training_file": "file-MNh7snhv0triDIhsrErokSMY",
     "hyperparameters": {
       "n_epochs": 7
     },
     "trained_tokens": 22834,

Fine Tuning with Function Calling

     "error": null
   }
Try it Out!
We obtain the fine-tuned LLM and use it with the Pydantic program.
   ft_llm = finetune_engine.get_finetuned_model(temperature=0.3)
   ft_program = OpenAIPydanticProgram.from_defaults(
       output_cls=Album,
       prompt_template_str=prompt_template_str,
       llm=ft_llm,
       verbose=False,
   )
   ft_program(movie_name="Goodfellas")
   Album(name='Goodfellas Soundtrack', artist='Various Artists', songs=[Song(title='Rags to Riches', length_seconds=180), Song(title='Gimme Shelter', length_seconds=270), Song(title='Layla', length_seconds=270), Song(title='Jump into the Fire', length_seconds=240), Song(title='Atlantis', length_seconds=180), Song(title='Beyond the Sea', length_seconds=180), Song(title='Sunshine of Your Love', length_seconds=240), Song(title='Mannish Boy', length_seconds=240), Song(title='Layla (Piano Exit)', length_seconds=120)])
Fine-tuning Structured Outputs through a RAG System
A use case of function calling is to get structured outputs through a
RAG system.
Here we show how to create a training dataset of context-augmented
inputs + structured outputs over an unstructured document. We can then
fine-tune the LLM and plug it into a RAG system to perform retrieval +
output extraction.
   !mkdir data && wget --user-agent "Mozilla" "https://arxiv.org/pdf/2307.09288.pdf" -O "data/llama2.pdf"
   --2023-10-04 23:46:36--  https://arxiv.org/pdf/2307.09288.pdf
   Resolving arxiv.org (arxiv.org)... 128.84.21.199
   Connecting to arxiv.org (arxiv.org)|128.84.21.199|:443... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: 13661300 (13M) [application/pdf]
   Saving to: ‘data/llama2.pdf’
   data/llama2.pdf     100%[===================>]  13.03M   229KB/s    in 45s
   2023-10-04 23:47:25 (298 KB/s) - ‘data/llama2.pdf’ saved [13661300/13661300]
   from pydantic import Field
   from typing import List
   class Citation(BaseModel):
       """Citation class."""
       author: str = Field(..., description="Inferred first author (usually last name")
       year: int = Field(..., description="Inferred year")
       desc: str = Field(
           ...,
           description="Inferred description from the text of the work that the author is cited for",
       )
   class Response(BaseModel):
       """List of author citations.
       Extracted over unstructured text.
       """
       citations: List[Citation] = Field(
           ...,
           description="List of author citations (organized by author, year, and description).",
       )
Load Data + Setup
   from llama_hub.file.pymu_pdf.base import PyMuPDFReader
   from llama_index import Document, ServiceContext
   from llama_index.node_parser import SimpleNodeParser
   from pathlib import Path
   loader = PyMuPDFReader()
   docs0 = loader.load(file_path=Path("./data/llama2.pdf"))
   doc_text = "\n\n".join([d.get_content() for d in docs0])
   metadata = {"paper_title": "Llama 2: Open Foundation and Fine-Tuned Chat Models"}

Fine Tuning with Function Calling

   docs = [Document(text=doc_text, metadata=metadata)]
   chunk_size = 1024
   node_parser = SimpleNodeParser.from_defaults(chunk_size=chunk_size)
   nodes = node_parser.get_nodes_from_documents(docs)
   len(nodes)
   89
   # setup service context
   finetuning_handler = OpenAIFineTuningHandler()
   callback_manager = CallbackManager([finetuning_handler])
   gpt_4_context = ServiceContext.from_defaults(
       llm=OpenAI(model="gpt-4-0613", temperature=0.3),
       callback_manager=callback_manager,
       chunk_size=chunk_size,
   )
   gpt_35_context = ServiceContext.from_defaults(
       llm=OpenAI(model="gpt-3.5-turbo-0613", temperature=0.3),
       callback_manager=callback_manager,
       chunk_size=chunk_size,
   )
   eval_context = ServiceContext.from_defaults(
       llm=OpenAI(model="gpt-4-0613", temperature=0), chunk_size=chunk_size
   )
Generate Dataset
Here we show how to generate a training dataset over these
unstructured chunks/nodes.
We generate questions to extract citations over different context. We
run these questions through a GPT-4 RAG pipeline, extract structured
outputs, and log inputs/outputs.
   # setup dataset generator
   from llama_index.evaluation import DatasetGenerator
   from llama_index import SummaryIndex, PromptTemplate
   from tqdm.notebook import tqdm
   from tqdm.asyncio import tqdm_asyncio
   fp = open("data/qa_pairs.jsonl", "w")
   question_gen_prompt = PromptTemplate(
       """
   {query_str}
   Context:
   {context_str}
   Questions:
   """
   )
   question_gen_query = """\
   Snippets from a research paper is given below. It contains citations.
   Please generate questions from the text asking about these citations.
   For instance, here are some sample questions:
   Which citations correspond to related works on transformer models?
   Tell me about authors that worked on advancing RLHF.
   Can you tell me citations corresponding to all computer vision works? \
   """
   qr_pairs = []
   node_questions_tasks = []
   for idx, node in enumerate(nodes[:39]):
       num_questions = 1  # change this number to increase number of nodes
       dataset_generator = DatasetGenerator(
           [node],
           question_gen_query=question_gen_query,
           text_question_template=question_gen_prompt,
           service_context=eval_context,
           metadata_mode="all",
           num_questions_per_chunk=num_questions,
       )
       task = dataset_generator.agenerate_questions_from_nodes(num=num_questions)
       node_questions_tasks.append(task)
   node_questions_lists = await tqdm_asyncio.gather(*node_questions_tasks)
   node_questions_lists
   gpt4_index = VectorStoreIndex(nodes, service_context=gpt_4_context)
   gpt4_query_engine = gpt4_index.as_query_engine(output_cls=Response, similarity_top_k=1)
   from json import JSONDecodeError
   for idx, node in enumerate(tqdm(nodes[:39])):
       node_questions_0 = node_questions_lists[idx]
       for question in node_questions_0:
           try:
               # note: we don't need to use response, events are logged through fine-tuning handler
               gpt4_query_engine.query(question)
           except Exception as e:
               print(f"Error for question {question}, {repr(e)}")
               pass
     0%|          | 0/39 [00:00<?, ?it/s]
   Error for question Which citations are referred to in the discussion about safety investigations into pretraining data and pretrained models?, ValidationError(model='Response', errors=[{'loc': ('__root__',), 'msg': 'Expecting value: line 1 column 1 (char 0)', 'type': 'value_error.jsondecode', 'ctx': {'msg': 'Expecting value', 'doc': 'Empty Response', 'pos': 0, 'lineno': 1, 'colno': 1}}])

Fine Tuning with Function Calling

   finetuning_handler.save_finetuning_events("llama2_citation_events.jsonl")
   Wrote 83 examples to llama2_citation_events.jsonl
Setup Fine-tuning
We kick off fine-tuning over the generated dataset.
   from llama_index.finetuning import OpenAIFinetuneEngine
   finetune_engine = OpenAIFinetuneEngine(
       "gpt-3.5-turbo",
       "llama2_citation_events.jsonl",
       # start_job_id="<start-job-id>"  # if you have an existing job, can specify id here
       validate_json=False,  # openai validate json code doesn't support function calling yet
   )
   finetune_engine.finetune()
   finetune_engine.get_current_job()
   <FineTuningJob fine_tuning.job id=ftjob-ATYm4yZHP1QvXs1wx85Ix79F at 0x1752b6b60> JSON: {
     "object": "fine_tuning.job",
     "id": "ftjob-ATYm4yZHP1QvXs1wx85Ix79F",
     "model": "gpt-3.5-turbo-0613",
     "created_at": 1696497663,
     "finished_at": 1696498092,
     "fine_tuned_model": "ft:gpt-3.5-turbo-0613:llamaindex::86EwPw83",
     "organization_id": "org-1ZDAvajC6v2ZtAP9hLEIsXRz",
     "result_files": [
       "file-wabcIIxjLqvhqOVohf4qSmE7"
     ],
     "status": "succeeded",
     "validation_file": null,
     "training_file": "file-WbYcsinIbH8vyCAstcoFEr92",
     "hyperparameters": {
       "n_epochs": 3
     },
     "trained_tokens": 132678,
     "error": null
   }
Use within RAG Pipeline
Let's plug the fine-tuned LLM into a full RAG pipeline that outputs
structured outputs.
   ft_llm = finetune_engine.get_finetuned_model(temperature=0.3)
   ft_service_context = ServiceContext.from_defaults(llm=ft_llm)
   from llama_index import VectorStoreIndex
   vector_index = VectorStoreIndex(nodes, service_context=ft_service_context)
   query_engine = vector_index.as_query_engine(output_cls=Response, similarity_top_k=1)
   # setup baseline as well
   base_index = VectorStoreIndex(nodes, service_context=gpt_35_context)
   base_query_engine = base_index.as_query_engine(output_cls=Response, similarity_top_k=1)
   query_str = """\
   Which citation is used to measure the truthfulness of Llama 2? \
   """
   # query_str = """\
   # Which citation corresponds to the concept of collecting data that represents \
   # empirically sampled human preferences in RLHF?\
   # """
   # query_str = "Which citations in the paper discuss the development and release of Llama 2?"
   # query_str = "Which citations are mentioned in the section on RLHF Results?"
   # query_str = "Which citation discusses the carbon output related to the production of AI hardware?"
   response = query_engine.query(query_str)
   print(str(response))
   {"citations": [{"author": "Lin et al.", "year": 2021, "desc": "TruthfulQA, used for LLM hallucinations to measure whether a language model is truthful in generating answers to questions while being informative at the same time."}]}
   base_response = base_query_engine.query(query_str)

Fine Tuning with Function Calling

   print(str(base_response))
   {"citations": [{"author": "Lin et al.", "year": 2021, "desc": "TruthfulQA"}]}
   # view sources
   print(response.source_nodes[0].get_content())
   # as a reference, take a look at GPT-4 response
   gpt4_response = gpt4_query_engine.query(query_str)
   print(str(gpt4_response))
   {"citations": [{"author": "Lin et al.", "year": 2021, "desc": "TruthfulQA, used for LLM hallucinations to measure whether a language model is truthful in generating answers to questions while being informative at the same time."}]}


Finetune Embeddings

In this notebook, we show users how to finetune their own embedding
models.
We go through three main sections:
1. Preparing the data (our "generate_qa_embedding_pairs" function
   makes this easy)
2. Finetuning the model (using our
   "SentenceTransformersFinetuneEngine")
3. Evaluating the model on a validation knowledge corpus
Generate Corpus
First, we create the corpus of text chunks by leveraging LlamaIndex to
load some financial PDFs, and parsing/chunking into plain text chunks.
   import json
   from llama_index import SimpleDirectoryReader
   from llama_index.node_parser import SimpleNodeParser
   from llama_index.schema import MetadataMode
   TRAIN_FILES = ["../../../examples/data/10k/lyft_2021.pdf"]
   VAL_FILES = ["../../../examples/data/10k/uber_2021.pdf"]
   TRAIN_CORPUS_FPATH = "./data/train_corpus.json"
   VAL_CORPUS_FPATH = "./data/val_corpus.json"
   def load_corpus(files, verbose=False):
       if verbose:
           print(f"Loading files {files}")
       reader = SimpleDirectoryReader(input_files=files)
       docs = reader.load_data()
       if verbose:
           print(f"Loaded {len(docs)} docs")
       parser = SimpleNodeParser.from_defaults()
       nodes = parser.get_nodes_from_documents(docs, show_progress=verbose)
       if verbose:
           print(f"Parsed {len(nodes)} nodes")
       return nodes
We do a very naive train/val split by having the Lyft corpus as the
train dataset, and the Uber corpus as the val dataset.
   train_nodes = load_corpus(TRAIN_FILES, verbose=True)
   val_nodes = load_corpus(VAL_FILES, verbose=True)
   Loading files ['../../../examples/data/10k/lyft_2021.pdf']
   Loaded 238 docs
   Parsing documents into nodes:   0%|          | 0/238 [00:00<?, ?it/s]
   Parsed 349 nodes
   Loading files ['../../../examples/data/10k/uber_2021.pdf']
   Loaded 307 docs
   Parsing documents into nodes:   0%|          | 0/307 [00:00<?, ?it/s]
   Parsed 418 nodes
Generate synthetic queries
Now, we use an LLM (gpt-3.5-turbo) to generate questions using each
text chunk in the corpus as context.
Each pair of (generated question, text chunk used as context) becomes
a datapoint in the finetuning dataset (either for training or
evaluation).
   from llama_index.finetuning import (
       generate_qa_embedding_pairs,
       EmbeddingQAFinetuneDataset,
   )
   train_dataset = generate_qa_embedding_pairs(train_nodes)
   val_dataset = generate_qa_embedding_pairs(val_nodes)
   train_dataset.save_json("train_dataset.json")
   val_dataset.save_json("val_dataset.json")
   100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 349/349 [11:24<00:00,  1.96s/it]
   100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 418/418 [14:54<00:00,  2.14s/it]
   # [Optional] Load
   train_dataset = EmbeddingQAFinetuneDataset.from_json("train_dataset.json")
   val_dataset = EmbeddingQAFinetuneDataset.from_json("val_dataset.json")
Run Embedding Finetuning
   from llama_index.finetuning import SentenceTransformersFinetuneEngine
   finetune_engine = SentenceTransformersFinetuneEngine(

Finetune Embeddings

       train_dataset,
       model_id="BAAI/bge-small-en",
       model_output_path="test_model",
       val_dataset=val_dataset,
   )
   finetune_engine.finetune()
   Epoch:   0%|          | 0/2 [00:00<?, ?it/s]
   Iteration:   0%|          | 0/71 [00:00<?, ?it/s]
   Iteration:   0%|          | 0/71 [00:00<?, ?it/s]
   embed_model = finetune_engine.get_finetuned_model()
   embed_model
   LangchainEmbedding(model_name='test_model', embed_batch_size=10, callback_manager=<llama_index.callbacks.base.CallbackManager object at 0x17743fd60>)
Evaluate Finetuned Model
In this section, we evaluate 3 different embedding models:
1. proprietary OpenAI embedding,
2. open source "BAAI/bge-small-en", and
3. our finetuned embedding model.
We consider 2 evaluation approaches:
1. a simple custom **hit rate** metric
2. using "InformationRetrievalEvaluator" from sentence_transformers
We show that finetuning on synthetic (LLM-generated) dataset
significantly improve upon an opensource embedding model.
   from llama_index.embeddings import OpenAIEmbedding
   from llama_index import ServiceContext, VectorStoreIndex
   from llama_index.schema import TextNode
   from tqdm.notebook import tqdm
   import pandas as pd
Define eval function
**Option 1**: We use a simple **hit rate** metric for evaluation:
* for each (query, relevant_doc) pair,
* we retrieve top-k documents with the query,  and
* it's a **hit** if the results contain the relevant_doc.
This approach is very simple and intuitive, and we can apply it to
both the proprietary OpenAI embedding as well as our open source and
fine-tuned embedding models.
   def evaluate(
       dataset,
       embed_model,
       top_k=5,
       verbose=False,
   ):
       corpus = dataset.corpus
       queries = dataset.queries
       relevant_docs = dataset.relevant_docs
       service_context = ServiceContext.from_defaults(embed_model=embed_model)
       nodes = [TextNode(id_=id_, text=text) for id_, text in corpus.items()]
       index = VectorStoreIndex(nodes, service_context=service_context, show_progress=True)
       retriever = index.as_retriever(similarity_top_k=top_k)
       eval_results = []
       for query_id, query in tqdm(queries.items()):
           retrieved_nodes = retriever.retrieve(query)
           retrieved_ids = [node.node.node_id for node in retrieved_nodes]
           expected_id = relevant_docs[query_id][0]
           is_hit = expected_id in retrieved_ids  # assume 1 relevant doc
           eval_result = {
               "is_hit": is_hit,
               "retrieved": retrieved_ids,
               "expected": expected_id,
               "query": query_id,
           }
           eval_results.append(eval_result)
       return eval_results
**Option 2**: We use the "InformationRetrievalEvaluator" from
sentence_transformers.
This provides a more comprehensive suite of metrics, but we can only
run it against the sentencetransformers compatible models (open source
and our finetuned model, *not* the OpenAI embedding model).
   from sentence_transformers.evaluation import InformationRetrievalEvaluator
   from sentence_transformers import SentenceTransformer
   from pathlib import Path
   def evaluate_st(
       dataset,
       model_id,
       name,
   ):
       corpus = dataset.corpus
       queries = dataset.queries
       relevant_docs = dataset.relevant_docs
       evaluator = InformationRetrievalEvaluator(queries, corpus, relevant_docs, name=name)

Finetune Embeddings

       model = SentenceTransformer(model_id)
       output_path = "results/"
       Path(output_path).mkdir(exist_ok=True, parents=True)
       return evaluator(model, output_path=output_path)
Run Evals
OpenAI
~~~~~~
Note: this might take a few minutes to run since we have to embed the
corpus and queries
   ada = OpenAIEmbedding()
   ada_val_results = evaluate(val_dataset, ada)
   Generating embeddings:   0%|          | 0/418 [00:00<?, ?it/s]
     0%|          | 0/836 [00:00<?, ?it/s]
   df_ada = pd.DataFrame(ada_val_results)
   hit_rate_ada = df_ada["is_hit"].mean()
   hit_rate_ada
   0.8779904306220095
BAAI/bge-small-en
   bge = "local:BAAI/bge-small-en"
   bge_val_results = evaluate(val_dataset, bge)
   Downloading (…)ab102/.gitattributes:   0%|          | 0.00/1.52k [00:00<?, ?B/s]
   Downloading (…)_Pooling/config.json:   0%|          | 0.00/190 [00:00<?, ?B/s]
   Downloading (…)2d2d7ab102/README.md:   0%|          | 0.00/78.9k [00:00<?, ?B/s]
   Downloading (…)2d7ab102/config.json:   0%|          | 0.00/684 [00:00<?, ?B/s]
   Downloading (…)ce_transformers.json:   0%|          | 0.00/124 [00:00<?, ?B/s]
   Downloading model.safetensors:   0%|          | 0.00/133M [00:00<?, ?B/s]
   Downloading pytorch_model.bin:   0%|          | 0.00/134M [00:00<?, ?B/s]
   Downloading (…)nce_bert_config.json:   0%|          | 0.00/52.0 [00:00<?, ?B/s]
   Downloading (…)cial_tokens_map.json:   0%|          | 0.00/125 [00:00<?, ?B/s]
   Downloading (…)ab102/tokenizer.json:   0%|          | 0.00/711k [00:00<?, ?B/s]
   Downloading (…)okenizer_config.json:   0%|          | 0.00/366 [00:00<?, ?B/s]
   Downloading (…)2d2d7ab102/vocab.txt:   0%|          | 0.00/232k [00:00<?, ?B/s]
   Downloading (…)d7ab102/modules.json:   0%|          | 0.00/229 [00:00<?, ?B/s]
   Generating embeddings:   0%|          | 0/418 [00:00<?, ?it/s]
     0%|          | 0/836 [00:00<?, ?it/s]
   df_bge = pd.DataFrame(bge_val_results)
   hit_rate_bge = df_bge["is_hit"].mean()
   hit_rate_bge
   0.7930622009569378
   evaluate_st(val_dataset, "BAAI/bge-small-en", name="bge")
   ---------------------------------------------------------------------------
   FileNotFoundError                         Traceback (most recent call last)
   Cell In[59], line 1
   ----> 1 evaluate_st(val_dataset, "BAAI/bge-small-en", name='bge')

Finetune Embeddings

   Cell In[49], line 15, in evaluate_st(dataset, model_id, name)
        13 evaluator = InformationRetrievalEvaluator(queries, corpus, relevant_docs, name=name)
        14 model = SentenceTransformer(model_id)
   ---> 15 return evaluator(model, output_path='results/')
   File ~/Programming/gpt_index/.venv/lib/python3.10/site-packages/sentence_transformers/evaluation/InformationRetrievalEvaluator.py:104, in InformationRetrievalEvaluator.__call__(self, model, output_path, epoch, steps, *args, **kwargs)
       102 csv_path = os.path.join(output_path, self.csv_file)
       103 if not os.path.isfile(csv_path):
   --> 104     fOut = open(csv_path, mode="w", encoding="utf-8")
       105     fOut.write(",".join(self.csv_headers))
       106     fOut.write("\n")
   FileNotFoundError: [Errno 2] No such file or directory: 'results/Information-Retrieval_evaluation_bge_results.csv'
Finetuned
   finetuned = "local:test_model"
   val_results_finetuned = evaluate(val_dataset, finetuned)
   df_finetuned = pd.DataFrame(val_results_finetuned)
   hit_rate_finetuned = df_finetuned["is_hit"].mean()
   hit_rate_finetuned
   evaluate_st(val_dataset, "test_model", name="finetuned")
Summary of Results
Hit rate
~~~~~~~~
   df_ada["model"] = "ada"
   df_bge["model"] = "bge"
   df_finetuned["model"] = "fine_tuned"
We can see that fine-tuning our small open-source embedding model
drastically improve its retrieval quality (even approaching the
quality of the proprietary OpenAI embedding)!
   df_all = pd.concat([df_ada, df_bge, df_finetuned])
   df_all.groupby("model").mean("is_hit")
InformationRetrievalEvaluator
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
   df_st_bge = pd.read_csv("results/Information-Retrieval_evaluation_bge_results.csv")
   df_st_finetuned = pd.read_csv(
       "results/Information-Retrieval_evaluation_finetuned_results.csv"
   )
We can see that embedding finetuning improves metrics consistently
across the suite of eval metrics
   df_st_bge["model"] = "bge"
   df_st_finetuned["model"] = "fine_tuned"
   df_st_all = pd.concat([df_st_bge, df_st_finetuned])
   df_st_all = df_st_all.set_index("model")
   df_st_all