SQL Join Query Engine

In this tutorial, we show you how to use our SQLJoinQueryEngine.
This query engine allows you to combine insights from your structured
tables with your unstructured data. It first decides whether to query
your structured tables for insights. Once it does, it can then infer a
corresponding query to the vector store in order to fetch
corresponding documents.
Setup
   # NOTE: This is ONLY necessary in jupyter notebook.
   # Details: Jupyter runs an event-loop behind the scenes.
   #          This results in nested event-loops when we start an event-loop to make async queries.
   #          This is normally not allowed, we use nest_asyncio to allow it for convenience.
   import nest_asyncio
   nest_asyncio.apply()
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SimpleDirectoryReader,
       ServiceContext,
       StorageContext,
       SQLDatabase,
       WikipediaReader,
   )
   INFO:numexpr.utils:Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   INFO:numexpr.utils:NumExpr defaulting to 8 threads.
   NumExpr defaulting to 8 threads.
Create Common Objects
This includes a "ServiceContext" object containing abstractions such
as the LLM and chunk size. This also includes a "StorageContext"
object containing our vector store abstractions.
   # # define pinecone index
   # import pinecone
   # import os
   # api_key = os.environ['PINECONE_API_KEY']
   # pinecone.init(api_key=api_key, environment="us-west1-gcp")
   # # dimensions are for text-embedding-ada-002
   # # pinecone.create_index("quickstart", dimension=1536, metric="euclidean", pod_type="p1")
   # pinecone_index = pinecone.Index("quickstart")
   # # OPTIONAL: delete all
   # pinecone_index.delete(deleteAll=True)
   from llama_index.node_parser.simple import SimpleNodeParser
   from llama_index import ServiceContext, LLMPredictor
   from llama_index.storage import StorageContext
   from llama_index.vector_stores import PineconeVectorStore
   from llama_index.text_splitter import TokenTextSplitter
   from llama_index.llms import OpenAI
   # define node parser and LLM
   chunk_size = 1024
   llm = OpenAI(temperature=0, model="gpt-4", streaming=True)
   service_context = ServiceContext.from_defaults(chunk_size=chunk_size, llm=llm)
   text_splitter = TokenTextSplitter(chunk_size=chunk_size)
   node_parser = SimpleNodeParser.from_defaults(text_splitter=text_splitter)
   # # define pinecone vector index
   # vector_store = PineconeVectorStore(pinecone_index=pinecone_index, namespace='wiki_cities')
   # storage_context = StorageContext.from_defaults(vector_store=vector_store)
   # vector_index = VectorStoreIndex([], storage_context=storage_context)
Create Database Schema + Test Data
Here we introduce a toy scenario where there are 100 tables (too big
to fit into the prompt)
   from sqlalchemy import (
       create_engine,
       MetaData,
       Table,
       Column,
       String,
       Integer,
       select,
       column,
   )
   engine = create_engine("sqlite:///:memory:", future=True)
   metadata_obj = MetaData()
   # create city SQL table
   table_name = "city_stats"
   city_stats_table = Table(

SQL Join Query Engine

       table_name,
       metadata_obj,
       Column("city_name", String(16), primary_key=True),
       Column("population", Integer),
       Column("country", String(16), nullable=False),
   )
   metadata_obj.create_all(engine)
   # print tables
   metadata_obj.tables.keys()
   dict_keys(['city_stats'])
We introduce some test data into the "city_stats" table
   from sqlalchemy import insert
   rows = [
       {"city_name": "Toronto", "population": 2930000, "country": "Canada"},
       {"city_name": "Tokyo", "population": 13960000, "country": "Japan"},
       {"city_name": "Berlin", "population": 3645000, "country": "Germany"},
   ]
   for row in rows:
       stmt = insert(city_stats_table).values(**row)
       with engine.begin() as connection:
           cursor = connection.execute(stmt)
   with engine.connect() as connection:
       cursor = connection.exec_driver_sql("SELECT * FROM city_stats")
       print(cursor.fetchall())
   [('Toronto', 2930000, 'Canada'), ('Tokyo', 13960000, 'Japan'), ('Berlin', 3645000, 'Germany')]
Load Data
We first show how to convert a Document into a set of Nodes, and
insert into a DocumentStore.
   # install wikipedia python package
   !pip install wikipedia
   Requirement already satisfied: wikipedia in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (1.4.0)
   Requirement already satisfied: beautifulsoup4 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from wikipedia) (4.12.2)
   Requirement already satisfied: requests<3.0.0,>=2.0.0 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from wikipedia) (2.28.2)
   Requirement already satisfied: certifi>=2017.4.17 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2022.12.7)
   Requirement already satisfied: charset-normalizer<4,>=2 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (3.1.0)
   Requirement already satisfied: idna<4,>=2.5 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (3.4)
   Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (1.26.15)
   Requirement already satisfied: soupsieve>1.2 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from beautifulsoup4->wikipedia) (2.4.1)
   [notice] A new release of pip available: 22.3.1 -> 23.1.2

SQL Join Query Engine

   [notice] To update, run: pip install --upgrade pip
   cities = ["Toronto", "Berlin", "Tokyo"]
   wiki_docs = WikipediaReader().load_data(pages=cities)
Build SQL Index
   sql_database = SQLDatabase(engine, include_tables=["city_stats"])
Build Vector Index
   # Insert documents into vector index
   # Each document has metadata of the city attached
   vector_indices = {}
   vector_query_engines = {}
   for city, wiki_doc in zip(cities, wiki_docs):
       vector_index = VectorStoreIndex.from_documents([wiki_doc])
       query_engine = vector_index.as_query_engine(similarity_top_k=2)
       vector_indices[city] = vector_index
       vector_query_engines[city] = query_engine
Define Query Engines, Set as Tools
   from llama_index.query_engine import SQLJoinQueryEngine, RetrieverQueryEngine
   from llama_index.tools.query_engine import QueryEngineTool
   from llama_index.tools import ToolMetadata
   from llama_index.indices.vector_store import VectorIndexAutoRetriever
   from llama_index.query_engine import SubQuestionQueryEngine
   from llama_index.indices.struct_store.sql_query import NLSQLTableQueryEngine
   sql_query_engine = NLSQLTableQueryEngine(
       sql_database=sql_database,
       tables=["city_stats"],
   )
   from llama_index.query_engine import SubQuestionQueryEngine
   query_engine_tools = []
   for city in cities:
       query_engine = vector_query_engines[city]
       query_engine_tool = QueryEngineTool(
           query_engine=query_engine,
           metadata=ToolMetadata(
               name=city, description=f"Provides information about {city}"
           ),
       )
       query_engine_tools.append(query_engine_tool)
   s_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=query_engine_tools)
   # from llama_index.indices.vector_store.retrievers import VectorIndexAutoRetriever
   # from llama_index.vector_stores.types import MetadataInfo, VectorStoreInfo
   # from llama_index.query_engine.retriever_query_engine import RetrieverQueryEngine
   # vector_store_info = VectorStoreInfo(
   #     content_info='articles about different cities',
   #     metadata_info=[
   #         MetadataInfo(
   #             name='title',
   #             type='str',
   #             description='The name of the city'),
   #     ]
   # )
   # vector_auto_retriever = VectorIndexAutoRetriever(vector_index, vector_store_info=vector_store_info)
   # retriever_query_engine = RetrieverQueryEngine.from_args(
   #     vector_auto_retriever, service_context=service_context
   # )
   sql_tool = QueryEngineTool.from_defaults(
       query_engine=sql_query_engine,
       description=(
           "Useful for translating a natural language query into a SQL query over a table containing: "
           "city_stats, containing the population/country of each city"
       ),
   )
   s_engine_tool = QueryEngineTool.from_defaults(
       query_engine=s_engine,
       description=f"Useful for answering semantic questions about different cities",
   )
Define SQLJoinQueryEngine
   query_engine = SQLJoinQueryEngine(
       sql_tool, s_engine_tool, service_context=service_context
   )
   response = query_engine.query(
       "Tell me about the arts and culture of the city with the highest population"
   )
   Querying SQL database: Useful for translating a natural language query into a SQL query over a table containing city_stats, containing the population/country of each city
   INFO:llama_index.query_engine.sql_join_query_engine:> Querying SQL database: Useful for translating a natural language query into a SQL query over a table containing city_stats, containing the population/country of each city

SQL Join Query Engine

   > Querying SQL database: Useful for translating a natural language query into a SQL query over a table containing city_stats, containing the population/country of each city
   INFO:llama_index.indices.struct_store.sql_query:> Table desc str: Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)) and foreign keys: .
   > Table desc str: Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)) and foreign keys: .
   SQL query: SELECT city_name, population FROM city_stats ORDER BY population DESC LIMIT 1;
   SQL response:
   Tokyo is the city with the highest population, with 13.96 million people. It is a vibrant city with a rich culture and a wide variety of art forms. From traditional Japanese art such as calligraphy and woodblock prints to modern art galleries and museums, Tokyo has something for everyone. There are also many festivals and events throughout the year that celebrate the city's culture and art.
   Transformed query given SQL response: What are some specific cultural festivals, events, and notable art galleries or museums in Tokyo?
   INFO:llama_index.query_engine.sql_join_query_engine:> Transformed query given SQL response: What are some specific cultural festivals, events, and notable art galleries or museums in Tokyo?
   > Transformed query given SQL response: What are some specific cultural festivals, events, and notable art galleries or museums in Tokyo?
   Generated 3 sub questions.
   [Tokyo] Q: What are some specific cultural festivals in Tokyo?
   [Tokyo] Q: What are some specific events in Tokyo?
   [Tokyo] Q: What are some notable art galleries or museums in Tokyo?
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=3069 request_id=eb3df12fea7d51eb93300180480dc90b response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=3069 request_id=eb3df12fea7d51eb93300180480dc90b response_code=200
   [Tokyo] A:
   Some specific cultural festivals in Tokyo include the Sannō at Hie Shrine, the Sanja at Asakusa Shrine, the biennial Kanda Festivals, the annual fireworks display over the Sumida River, picnics under the cherry blossoms in Ueno Park, Inokashira Park, and the Shinjuku Gyoen National Garden, and Harajuku's youth style, fashion and cosplay.
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=3530 request_id=ae31aacec5e68590b9cc4a63ee97b66a response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=3530 request_id=ae31aacec5e68590b9cc4a63ee97b66a response_code=200
   [Tokyo] A:
   Some specific events in Tokyo include the 1964 Summer Olympics, the October 2011 artistic gymnastics world championships, the 2019 Rugby World Cup, the 2020 Summer Olympics and Paralympics (rescheduled to 2021 due to the COVID-19 pandemic in Japan), the Asian Network of Major Cities 21, the Council of Local Authorities for International Relations, the C40 Cities Climate Leadership Group, and various international academic and scientific research collaborations.

SQL Join Query Engine

   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=5355 request_id=81bff9133777221cde8d15d58134ee8f response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=5355 request_id=81bff9133777221cde8d15d58134ee8f response_code=200
   [Tokyo] A:
   Some notable art galleries and museums in Tokyo include the Tokyo National Museum, the National Museum of Western Art, the Nezu Museum, the National Diet Library, the National Archives, the National Museum of Modern Art, the New National Theater Tokyo, the Edo-Tokyo Museum, the National Museum of Emerging Science and Innovation, and the Studio Ghibli anime center.
   query engine response:
   Some specific cultural festivals, events, and notable art galleries or museums in Tokyo include the Sannō at Hie Shrine, the Sanja at Asakusa Shrine, the biennial Kanda Festivals, the annual fireworks display over the Sumida River, picnics under the cherry blossoms in Ueno Park, Inokashira Park, and the Shinjuku Gyoen National Garden, Harajuku's youth style, fashion and cosplay, the 1964 Summer Olympics, the October 2011 artistic gymnastics world championships, the 2019 Rugby World Cup, the 2020 Summer Olympics and Paralympics (rescheduled to 2021 due to the COVID-19 pandemic in Japan), the Asian Network of Major Cities 21, the Council of Local Authorities for International Relations, the C40 Cities Climate Leadership Group, various international academic and scientific research collaborations, the Tokyo National Museum, the National Museum of Western Art, the Nezu Museum, the National Diet Library, the National Archives, the National Museum of Modern Art, the New National Theater Tokyo, the Edo-Tokyo Museum, the National Museum of Emerging Science and Innovation, and the Studio Ghibli anime center.
   INFO:llama_index.query_engine.sql_join_query_engine:> query engine response:
   Some specific cultural festivals, events, and notable art galleries or museums in Tokyo include the Sannō at Hie Shrine, the Sanja at Asakusa Shrine, the biennial Kanda Festivals, the annual fireworks display over the Sumida River, picnics under the cherry blossoms in Ueno Park, Inokashira Park, and the Shinjuku Gyoen National Garden, Harajuku's youth style, fashion and cosplay, the 1964 Summer Olympics, the October 2011 artistic gymnastics world championships, the 2019 Rugby World Cup, the 2020 Summer Olympics and Paralympics (rescheduled to 2021 due to the COVID-19 pandemic in Japan), the Asian Network of Major Cities 21, the Council of Local Authorities for International Relations, the C40 Cities Climate Leadership Group, various international academic and scientific research collaborations, the Tokyo National Museum, the National Museum of Western Art, the Nezu Museum, the National Diet Library, the National Archives, the National Museum of Modern Art, the New National Theater Tokyo, the Edo-Tokyo Museum, the National Museum of Emerging Science and Innovation, and the Studio Ghibli anime center.
   > query engine response:
   Some specific cultural festivals, events, and notable art galleries or museums in Tokyo include the Sannō at Hie Shrine, the Sanja at Asakusa Shrine, the biennial Kanda Festivals, the annual fireworks display over the Sumida River, picnics under the cherry blossoms in Ueno Park, Inokashira Park, and the Shinjuku Gyoen National Garden, Harajuku's youth style, fashion and cosplay, the 1964 Summer Olympics, the October 2011 artistic gymnastics world championships, the 2019 Rugby World Cup, the 2020 Summer Olympics and Paralympics (rescheduled to 2021 due to the COVID-19 pandemic in Japan), the Asian Network of Major Cities 21, the Council of Local Authorities for International Relations, the C40 Cities Climate Leadership Group, various international academic and scientific research collaborations, the Tokyo National Museum, the National Museum of Western Art, the Nezu Museum, the National Diet Library, the National Archives, the National Museum of Modern Art, the New National Theater Tokyo, the Edo-Tokyo Museum, the National Museum of Emerging Science and Innovation, and the Studio Ghibli anime center.

SQL Join Query Engine

   Final response: Tokyo, the city with the highest population of 13.96 million people, is known for its vibrant culture and diverse art forms. It hosts a variety of cultural festivals and events such as the Sannō at Hie Shrine, the Sanja at Asakusa Shrine, the biennial Kanda Festivals, and the annual fireworks display over the Sumida River. Residents and visitors often enjoy picnics under the cherry blossoms in Ueno Park, Inokashira Park, and the Shinjuku Gyoen National Garden. Harajuku's youth style, fashion, and cosplay are also notable cultural aspects of Tokyo. The city has hosted several international events including the 1964 Summer Olympics, the 2019 Rugby World Cup, and the 2020 Summer Olympics and Paralympics (rescheduled to 2021 due to the COVID-19 pandemic).
   In terms of art, Tokyo is home to numerous galleries and museums. These include the Tokyo National Museum, the National Museum of Western Art, the Nezu Museum, the National Diet Library, the National Archives, the National Museum of Modern Art, the New National Theater Tokyo, the Edo-Tokyo Museum, the National Museum of Emerging Science and Innovation, and the Studio Ghibli anime center. These institutions showcase everything from traditional Japanese art such as calligraphy and woodblock prints to modern art and scientific innovations.

   print(str(response))
   Tokyo, the city with the highest population of 13.96 million people, is known for its vibrant culture and diverse art forms. It hosts a variety of cultural festivals and events such as the Sannō at Hie Shrine, the Sanja at Asakusa Shrine, the biennial Kanda Festivals, and the annual fireworks display over the Sumida River. Residents and visitors often enjoy picnics under the cherry blossoms in Ueno Park, Inokashira Park, and the Shinjuku Gyoen National Garden. Harajuku's youth style, fashion, and cosplay are also notable cultural aspects of Tokyo. The city has hosted several international events including the 1964 Summer Olympics, the 2019 Rugby World Cup, and the 2020 Summer Olympics and Paralympics (rescheduled to 2021 due to the COVID-19 pandemic).
   In terms of art, Tokyo is home to numerous galleries and museums. These include the Tokyo National Museum, the National Museum of Western Art, the Nezu Museum, the National Diet Library, the National Archives, the National Museum of Modern Art, the New National Theater Tokyo, the Edo-Tokyo Museum, the National Museum of Emerging Science and Innovation, and the Studio Ghibli anime center. These institutions showcase everything from traditional Japanese art such as calligraphy and woodblock prints to modern art and scientific innovations.
   response = query_engine.query(
       "Compare and contrast the demographics of Berlin and Toronto"
   )
   Querying SQL database: Useful for translating a natural language query into a SQL query over a table containing city_stats, containing the population/country of each city
   INFO:llama_index.query_engine.sql_join_query_engine:> Querying SQL database: Useful for translating a natural language query into a SQL query over a table containing city_stats, containing the population/country of each city
   > Querying SQL database: Useful for translating a natural language query into a SQL query over a table containing city_stats, containing the population/country of each city
   INFO:llama_index.indices.struct_store.sql_query:> Table desc str: Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)) and foreign keys: .
   > Table desc str: Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)) and foreign keys: .

SQL Join Query Engine

   SQL query: SELECT city_name, population, country FROM city_stats WHERE city_name IN ('Berlin', 'Toronto');
   SQL response:  Berlin and Toronto are both major cities with large populations. Berlin has a population of 3.6 million people and is located in Germany, while Toronto has a population of 2.9 million people and is located in Canada.
   Transformed query given SQL response: What are the age, gender, and ethnic breakdowns of the populations in Berlin and Toronto?
   INFO:llama_index.query_engine.sql_join_query_engine:> Transformed query given SQL response: What are the age, gender, and ethnic breakdowns of the populations in Berlin and Toronto?
   > Transformed query given SQL response: What are the age, gender, and ethnic breakdowns of the populations in Berlin and Toronto?
   Generated 6 sub questions.
   [Berlin] Q: What is the age breakdown of the population in Berlin?
   [Berlin] Q: What is the gender breakdown of the population in Berlin?
   [Berlin] Q: What is the ethnic breakdown of the population in Berlin?
   [Toronto] Q: What is the age breakdown of the population in Toronto?
   [Toronto] Q: What is the gender breakdown of the population in Toronto?
   [Toronto] Q: What is the ethnic breakdown of the population in Toronto?
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=934 request_id=b6a654edffcb5a12aa8dac775e0342e2 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=934 request_id=b6a654edffcb5a12aa8dac775e0342e2 response_code=200
   [Berlin] A:
   It is not possible to answer this question with the given context information.
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=1248 request_id=c3023af7adbb1018a483467bba6de168 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=1248 request_id=c3023af7adbb1018a483467bba6de168 response_code=200
   [Toronto] A:
   The gender population of Toronto is 48 per cent male and 52 per cent female. Women outnumber men in all age groups 15 and older.
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=2524 request_id=3a00900922f785b709db15420d83205b response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=2524 request_id=3a00900922f785b709db15420d83205b response_code=200

SQL Join Query Engine

   [Berlin] A:
   It is not possible to answer this question with the given context information.
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=4415 request_id=273aa88ce1189e6f09a7d492dd08490a response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=4415 request_id=273aa88ce1189e6f09a7d492dd08490a response_code=200
   [Toronto] A:
   The median age of the population in Toronto is 39.3 years. Persons aged 14 years and under make up 14.5 per cent of the population, and those aged 65 years and over make up 15.6 per cent. Women outnumber men in all age groups 15 and older.
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=4960 request_id=4cb35c8f2cd448297321211f8e7ab19e response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=4960 request_id=4cb35c8f2cd448297321211f8e7ab19e response_code=200
   [Berlin] A:
   The ethnic breakdown of the population in Berlin is primarily German, Turkish, Polish, English, Persian, Arabic, Italian, Bulgarian, Russian, Romanian, Kurdish, Serbo-Croatian, French, Spanish, Vietnamese, Lebanese, Palestinian, Serbian, Indian, Bosnian, American, Ukrainian, Chinese, Austrian, Israeli, Thai, Iranian, Egyptian and Syrian.
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=5783 request_id=5293a02bb62560654072ab8cc3235663 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/completions processing_ms=5783 request_id=5293a02bb62560654072ab8cc3235663 response_code=200
   [Toronto] A:
   The ethnic breakdown of the population in Toronto in 2016 was: European (47.9%), Asian (including Middle-Eastern – 40.1%), African (5.5%), Latin/Central/South American (4.2%), and North American aboriginal (1.2%). The largest visible minority groups were South Asian (Indian, Pakistani, Sri Lankan at 12.6%), East Asian (Chinese at 12.5%), and Black (8.9%).
   query engine response:
   Berlin:
   Age breakdown: It is not possible to answer this question with the given context information.
   Gender breakdown: It is not possible to answer this question with the given context information.
   Ethnic breakdown: The ethnic breakdown of the population in Berlin is primarily German, Turkish, Polish, English, Persian, Arabic, Italian, Bulgarian, Russian, Romanian, Kurdish, Serbo-Croatian, French, Spanish, Vietnamese, Lebanese, Palestinian, Serbian, Indian, Bosnian, American, Ukrainian, Chinese, Austrian, Israeli, Thai, Iranian, Egyptian and Syrian.
   Toronto:
   Age breakdown: The median age of the population in Toronto is 39.3 years. Persons aged 14 years and under make up 14.5 per cent of the population, and those aged 65 years and over make up 15.6 per cent. Women outnumber men in all age groups 15 and older.

SQL Join Query Engine

   Gender breakdown: The gender population of Toronto is 48 per cent male and 52 per cent female. Women outnumber men in all age groups 15 and older.
   Ethnic breakdown: The ethnic breakdown of the population in Toronto in 2016 was: European (47.9%), Asian (including Middle-Eastern – 40.1%), African (5.5%), Latin/Central/South American (4.2%), and North American aboriginal (1.2%). The largest visible minority groups were South Asian (Indian, Pakistani, Sri Lankan at 12.6%), East Asian (Chinese at 12.5%), and Black (8.9%).
   INFO:llama_index.query_engine.sql_join_query_engine:> query engine response:
   Berlin:
   Age breakdown: It is not possible to answer this question with the given context information.
   Gender breakdown: It is not possible to answer this question with the given context information.
   Ethnic breakdown: The ethnic breakdown of the population in Berlin is primarily German, Turkish, Polish, English, Persian, Arabic, Italian, Bulgarian, Russian, Romanian, Kurdish, Serbo-Croatian, French, Spanish, Vietnamese, Lebanese, Palestinian, Serbian, Indian, Bosnian, American, Ukrainian, Chinese, Austrian, Israeli, Thai, Iranian, Egyptian and Syrian.
   Toronto:
   Age breakdown: The median age of the population in Toronto is 39.3 years. Persons aged 14 years and under make up 14.5 per cent of the population, and those aged 65 years and over make up 15.6 per cent. Women outnumber men in all age groups 15 and older.
   Gender breakdown: The gender population of Toronto is 48 per cent male and 52 per cent female. Women outnumber men in all age groups 15 and older.
   Ethnic breakdown: The ethnic breakdown of the population in Toronto in 2016 was: European (47.9%), Asian (including Middle-Eastern – 40.1%), African (5.5%), Latin/Central/South American (4.2%), and North American aboriginal (1.2%). The largest visible minority groups were South Asian (Indian, Pakistani, Sri Lankan at 12.6%), East Asian (Chinese at 12.5%), and Black (8.9%).
   > query engine response:
   Berlin:
   Age breakdown: It is not possible to answer this question with the given context information.
   Gender breakdown: It is not possible to answer this question with the given context information.
   Ethnic breakdown: The ethnic breakdown of the population in Berlin is primarily German, Turkish, Polish, English, Persian, Arabic, Italian, Bulgarian, Russian, Romanian, Kurdish, Serbo-Croatian, French, Spanish, Vietnamese, Lebanese, Palestinian, Serbian, Indian, Bosnian, American, Ukrainian, Chinese, Austrian, Israeli, Thai, Iranian, Egyptian and Syrian.
   Toronto:
   Age breakdown: The median age of the population in Toronto is 39.3 years. Persons aged 14 years and under make up 14.5 per cent of the population, and those aged 65 years and over make up 15.6 per cent. Women outnumber men in all age groups 15 and older.
   Gender breakdown: The gender population of Toronto is 48 per cent male and 52 per cent female. Women outnumber men in all age groups 15 and older.
   Ethnic breakdown: The ethnic breakdown of the population in Toronto in 2016 was: European (47.9%), Asian (including Middle-Eastern – 40.1%), African (5.5%), Latin/Central/South American (4.2%), and North American aboriginal (1.2%). The largest visible minority groups were South Asian (Indian, Pakistani, Sri Lankan at 12.6%), East Asian (Chinese at 12.5%), and Black (8.9%).

SQL Join Query Engine

   Final response: Berlin and Toronto are both major cities with large populations. Berlin, located in Germany, has a population of 3.6 million people. The ethnic breakdown of the population in Berlin is primarily German, Turkish, Polish, English, Persian, Arabic, Italian, Bulgarian, Russian, Romanian, Kurdish, Serbo-Croatian, French, Spanish, Vietnamese, Lebanese, Palestinian, Serbian, Indian, Bosnian, American, Ukrainian, Chinese, Austrian, Israeli, Thai, Iranian, Egyptian and Syrian. Unfortunately, the age and gender breakdowns for Berlin are not available.
   On the other hand, Toronto, located in Canada, has a population of 2.9 million people. The median age of the population in Toronto is 39.3 years. Persons aged 14 years and under make up 14.5 per cent of the population, and those aged 65 years and over make up 15.6 per cent. The gender population of Toronto is 48 per cent male and 52 per cent female, with women outnumbering men in all age groups 15 and older. The ethnic breakdown of the population in Toronto in 2016 was: European (47.9%), Asian (including Middle-Eastern – 40.1%), African (5.5%), Latin/Central/South American (4.2%), and North American aboriginal (1.2%). The largest visible minority groups were South Asian (Indian, Pakistani, Sri Lankan at 12.6%), East Asian (Chinese at 12.5%), and Black (8.9%).

   print(str(response))
   Berlin and Toronto are both major cities with large populations. Berlin, located in Germany, has a population of 3.6 million people. The ethnic breakdown of the population in Berlin is primarily German, Turkish, Polish, English, Persian, Arabic, Italian, Bulgarian, Russian, Romanian, Kurdish, Serbo-Croatian, French, Spanish, Vietnamese, Lebanese, Palestinian, Serbian, Indian, Bosnian, American, Ukrainian, Chinese, Austrian, Israeli, Thai, Iranian, Egyptian and Syrian. Unfortunately, the age and gender breakdowns for Berlin are not available.
   On the other hand, Toronto, located in Canada, has a population of 2.9 million people. The median age of the population in Toronto is 39.3 years. Persons aged 14 years and under make up 14.5 per cent of the population, and those aged 65 years and over make up 15.6 per cent. The gender population of Toronto is 48 per cent male and 52 per cent female, with women outnumbering men in all age groups 15 and older. The ethnic breakdown of the population in Toronto in 2016 was: European (47.9%), Asian (including Middle-Eastern – 40.1%), African (5.5%), Latin/Central/South American (4.2%), and North American aboriginal (1.2%). The largest visible minority groups were South Asian (Indian, Pakistani, Sri Lankan at 12.6%), East Asian (Chinese at 12.5%), and Black (8.9%).


Pandas Query Engine

import logging
   import sys
   from IPython.display import Markdown, display
   import pandas as pd
   from llama_index.query_engine import PandasQueryEngine
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
Let's start on a Toy DataFrame
Very simple dataframe containing city and population pairs.
   # Test on some sample data
   df = pd.DataFrame(
       {"city": ["Toronto", "Tokyo", "Berlin"], "population": [2930000, 13960000, 3645000]}
   )
   query_engine = PandasQueryEngine(df=df, verbose=True)
   response = query_engine.query(
       "What is the city with the highest population?",
   )
   > Pandas Instructions:
   ```
   df['city'][df['population'].idxmax()]
   ```
   > Pandas Output: Tokyo
   display(Markdown(f"{response}"))
Tokyo
   # get pandas python instructions
   print(response.metadata["pandas_instruction_str"])
   df['city'][df['population'].idxmax()]
Analyzing the Titanic Dataset
The Titanic dataset is one of the most popular tabular datasets in
introductory machine learning Source: https://www.kaggle.com/c/titanic
   df = pd.read_csv("../data/csv/titanic_train.csv")
   query_engine = PandasQueryEngine(df=df, verbose=True)
   response = query_engine.query(
       "What is the correlation between survival and age?",
   )
   > Pandas Instructions:
   ```
   df['survived'].corr(df['age'])
   ```
   > Pandas Output: -0.07722109457217768
   display(Markdown(f"{response}"))
-0.07722109457217768
   # get pandas python instructions
   print(response.metadata["pandas_instruction_str"])
   df['survived'].corr(df['age'])


CitationQueryEngine

This notebook walks through how to use the CitationQueryEngine
The CitationQueryEngine can be used with any existing index.
Setup
   import os
   from llama_index.llms import OpenAI
   from llama_index.query_engine import CitationQueryEngine
   from llama_index.retrievers import VectorIndexRetriever
   from llama_index import (
       VectorStoreIndex,
       SimpleDirectoryReader,
       StorageContext,
       load_index_from_storage,
       LLMPredictor,
       ServiceContext,
   )
   /home/loganm/miniconda3/envs/llama-index/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
   service_context = ServiceContext.from_defaults(
       llm=OpenAI(model="gpt-3.5-turbo", temperature=0)
   )
   if not os.path.exists("./citation"):
       documents = SimpleDirectoryReader("../data/paul_graham").load_data()
       index = VectorStoreIndex.from_documents(documents, service_context=service_context)
       index.storage_context.persist(persist_dir="./citation")
   else:
       index = load_index_from_storage(
           StorageContext.from_defaults(persist_dir="./citation"),
           service_context=service_context,
       )
Create the CitationQueryEngine w/ Default Arguments
   query_engine = CitationQueryEngine.from_args(
       index,
       similarity_top_k=3,
       # here we can control how granular citation sources are, the default is 512
       citation_chunk_size=512,
   )
   response = query_engine.query("What did the author do growing up?")
   print(response)
   Before college, the author worked on writing short stories and programming on an IBM 1401 using an early version of Fortran [1]. They later got a TRS-80 computer and wrote simple games, a program to predict rocket heights, and a word processor [2].
   # source nodes are 6, because the original chunks of 1024-sized nodes were broken into more granular nodes
   print(len(response.source_nodes))
   6
Inspecting the Actual Source
Sources start counting at 1, but python arrays start counting at zero!
Let's confirm the source makes sense.
   print(response.source_nodes[0].node.get_text())
   Source 1:
   What I Worked On
   February 2021
   Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.
   The first programs I tried writing were on the IBM 1401 that our school district used for what was then called "data processing." This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.
   The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.
   I was puzzled by the 1401. I couldn't figure out what to do with it. And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but I didn't know enough math to do anything interesting of that type. So I'm not surprised I can't remember any programs I wrote, because they can't have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn't. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager's expression made clear.

CitationQueryEngine

   With microcomputers, everything changed. Now you could have a computer sitting right in front of you, on a desk, that could respond to your keystrokes as it was running instead of just churning through a stack of punch cards and then stopping.
   print(response.source_nodes[1].node.get_text())
   Source 2:
   [1]
   The first of my friends to get a microcomputer built it himself. It was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer.
   Computers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started programming. I wrote simple games, a program to predict how high my model rockets would fly, and a word processor that my father used to write at least one book. There was only room in memory for about 2 pages of text, so he'd write 2 pages at a time and then print them out, but it was a lot better than a typewriter.
   Though I liked programming, I didn't plan to study it in college. In college I was going to study philosophy, which sounded much more powerful. It seemed, to my naive high school self, to be the study of the ultimate truths, compared to which the things studied in other fields would be mere domain knowledge. What I discovered when I got to college was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.
   I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI.
   AI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven't tried rereading The
Adjusting Settings
Note that setting the chunk size larger than the original chunk size
of the nodes will have no effect.
The default node chunk size is 1024, so here, we are not making our
citation nodes any more granular.
   query_engine = CitationQueryEngine.from_args(
       index,
       # increase the citation chunk size!
       citation_chunk_size=1024,
       similarity_top_k=3,
   )
   response = query_engine.query("What did the author do growing up?")
   print(response)
   Before college, the author worked on writing short stories and programming on an IBM 1401 using an early version of Fortran [1].
   # should be less source nodes now!
   print(len(response.source_nodes))
   3
Inspecting the Actual Source
Sources start counting at 1, but python arrays start counting at zero!
Let's confirm the source makes sense.
   print(response.source_nodes[0].node.get_text())
   Source 1:
   What I Worked On
   February 2021
   Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.
   The first programs I tried writing were on the IBM 1401 that our school district used for what was then called "data processing." This was in 9th grade, so I was 13 or 14. The school district's 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain's lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.

CitationQueryEngine

   The language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.
   I was puzzled by the 1401. I couldn't figure out what to do with it. And in retrospect there's not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn't have any data stored on punched cards. The only other option was to do things that didn't rely on any input, like calculate approximations of pi, but I didn't know enough math to do anything interesting of that type. So I'm not surprised I can't remember any programs I wrote, because they can't have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn't. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager's expression made clear.
   With microcomputers, everything changed. Now you could have a computer sitting right in front of you, on a desk, that could respond to your keystrokes as it was running instead of just churning through a stack of punch cards and then stopping. [1]
   The first of my friends to get a microcomputer built it himself. It was sold as a kit by Heathkit. I remember vividly how impressed and envious I felt watching him sitting in front of it, typing programs right into the computer.
   Computers were expensive in those days and it took me years of nagging before I convinced my father to buy one, a TRS-80, in about 1980. The gold standard then was the Apple II, but a TRS-80 was good enough. This was when I really started programming. I wrote simple games, a program to predict how high my model rockets would fly, and a word processor that my father used to write at least one book. There was only room in memory for about 2 pages of text, so he'd write 2 pages at a time and then print them out, but it was a lot better than a typewriter.
   Though I liked programming, I didn't plan to study it in college. In college I was going to study philosophy, which sounded much more powerful. It seemed, to my naive high school self, to be the study of the ultimate truths, compared to which the things studied in other fields would be mere domain knowledge. What I discovered when I got to college was that the other fields took up so much of the space of ideas that there wasn't much left for these supposed ultimate truths. All that seemed left for philosophy were edge cases that people in other fields felt could safely be ignored.
   I couldn't have put this into words when I was 18. All I knew at the time was that I kept taking philosophy courses and they kept being boring. So I decided to switch to AI.
   AI was in the air in the mid 1980s, but there were two things especially that made me want to work on it: a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. I haven't tried rereading The


Retriever Query Engine with Custom Retrievers - Simple Hybrid Search

In this tutorial, we show you how to define a very simple version of
hybrid search!
Combine keyword lookup retrieval with vector retrieval using "AND" and
"OR" conditions.
Setup
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SimpleKeywordTableIndex,
       SimpleDirectoryReader,
       ServiceContext,
       StorageContext,
   )
   from IPython.display import Markdown, display
   INFO:numexpr.utils:Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   INFO:numexpr.utils:NumExpr defaulting to 8 threads.
   NumExpr defaulting to 8 threads.
   /home/loganm/miniconda3/envs/llama-index/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
Load Data
We first show how to convert a Document into a set of Nodes, and
insert into a DocumentStore.
   # load documents
   documents = SimpleDirectoryReader("../data/paul_graham").load_data()
   # initialize service context (set chunk size)
   service_context = ServiceContext.from_defaults(chunk_size=1024)
   node_parser = service_context.node_parser
   nodes = node_parser.get_nodes_from_documents(documents)
   # initialize storage context (by default it's in-memory)
   storage_context = StorageContext.from_defaults()
   storage_context.docstore.add_documents(nodes)
Define Vector Index and Keyword Table Index over Same Data
We build a vector index and keyword index over the same DocumentStore
   vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
   keyword_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 17050 tokens
   > [build_index_from_nodes] Total embedding token usage: 17050 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens
   > [build_index_from_nodes] Total embedding token usage: 0 tokens
Define Custom Retriever
We now define a custom retriever class that can implement basic hybrid
search with both keyword lookup and semantic search.
* setting "AND" means we take the intersection of the two retrieved
  sets
* setting "OR" means we take the union
   # import QueryBundle
   from llama_index import QueryBundle
   # import NodeWithScore
   from llama_index.schema import NodeWithScore
   # Retrievers
   from llama_index.retrievers import (
       BaseRetriever,
       VectorIndexRetriever,
       KeywordTableSimpleRetriever,
   )
   from typing import List
   class CustomRetriever(BaseRetriever):
       """Custom retriever that performs both semantic search and hybrid search."""
       def __init__(
           self,

           vector_retriever: VectorIndexRetriever,
           keyword_retriever: KeywordTableSimpleRetriever,
           mode: str = "AND",
       ) -> None:
           """Init params."""
           self._vector_retriever = vector_retriever
           self._keyword_retriever = keyword_retriever
           if mode not in ("AND", "OR"):
               raise ValueError("Invalid mode.")
           self._mode = mode
       def _retrieve(self, query_bundle: QueryBundle) -> List[NodeWithScore]:
           """Retrieve nodes given query."""
           vector_nodes = self._vector_retriever.retrieve(query_bundle)
           keyword_nodes = self._keyword_retriever.retrieve(query_bundle)
           vector_ids = {n.node.node_id for n in vector_nodes}
           keyword_ids = {n.node.node_id for n in keyword_nodes}
           combined_dict = {n.node.node_id: n for n in vector_nodes}
           combined_dict.update({n.node.node_id: n for n in keyword_nodes})
           if self._mode == "AND":
               retrieve_ids = vector_ids.intersection(keyword_ids)
           else:
               retrieve_ids = vector_ids.union(keyword_ids)
           retrieve_nodes = [combined_dict[rid] for rid in retrieve_ids]
           return retrieve_nodes
Plugin Retriever into Query Engine
Plugin retriever into a query engine, and run some queries
   from llama_index import get_response_synthesizer
   from llama_index.query_engine import RetrieverQueryEngine
   # define custom retriever
   vector_retriever = VectorIndexRetriever(index=vector_index, similarity_top_k=2)
   keyword_retriever = KeywordTableSimpleRetriever(index=keyword_index)
   custom_retriever = CustomRetriever(vector_retriever, keyword_retriever)
   # define response synthesizer
   response_synthesizer = get_response_synthesizer()
   # assemble query engine
   custom_query_engine = RetrieverQueryEngine(
       retriever=custom_retriever,
       response_synthesizer=response_synthesizer,
   )
   # vector query engine
   vector_query_engine = RetrieverQueryEngine(
       retriever=vector_retriever,
       response_synthesizer=response_synthesizer,
   )
   # keyword query engine
   keyword_query_engine = RetrieverQueryEngine(
       retriever=keyword_retriever,
       response_synthesizer=response_synthesizer,
   )
   response = custom_query_engine.query("What did the author do during his time at YC?")
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: What did the author do during his time at YC?
   > Starting query: What did the author do during his time at YC?
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['time', 'yc', 'author']
   query keywords: ['time', 'yc', 'author']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['time', 'yc']
   > Extracted keywords: ['time', 'yc']
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1919 tokens
   > [get_response] Total LLM token usage: 1919 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens

   print(response)
   The author worked on YC, wrote essays, worked on a new version of Arc, wrote Hacker News in Arc, wrote YC's internal software in Arc, and dealt with disputes between cofounders, figuring out when people were lying to them, and fighting with people who maltreated the startups.
   # hybrid search can allow us to not retrieve nodes that are irrelevant
   # Yale is never mentioned in the essay
   response = custom_query_engine.query("What did the author do during his time at Yale?")
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 11 tokens
   > [retrieve] Total embedding token usage: 11 tokens
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: What did the author do during his time at Yale?
   > Starting query: What did the author do during his time at Yale?
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['yale', 'time', 'author']
   query keywords: ['yale', 'time', 'author']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['time']
   > Extracted keywords: ['time']
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 0 tokens
   > [get_response] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   print(str(response))
   len(response.source_nodes)
   None
   0
   # in contrast, vector search will return an answer
   response = vector_query_engine.query("What did the author do during his time at Yale?")
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 11 tokens
   > [retrieve] Total embedding token usage: 11 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1871 tokens
   > [get_response] Total LLM token usage: 1871 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   print(str(response))
   len(response.source_nodes)
   The author did not attend Yale. The context information provided is about the author's work before and after college.
   2


Ensemble Query Engine Guide

Oftentimes when building a RAG application there are different query
pipelines you need to experiment with (e.g. top-k retrieval, keyword
search, knowledge graphs).
Thought: what if we could try a bunch of strategies at once, and have
the LLM 1) rate the relevance of each query, and 2) synthesize the
results?
This guide showcases this over the Great Gatsby. We do ensemble
retrieval over different chunk sizes and also different indices.
**NOTE**: Please also see our closely-related Ensemble Retrieval
Guide!
Setup
   # NOTE: This is ONLY necessary in jupyter notebook.
   # Details: Jupyter runs an event-loop behind the scenes.
   #          This results in nested event-loops when we start an event-loop to make async queries.
   #          This is normally not allowed, we use nest_asyncio to allow it for convenience.
   import nest_asyncio
   nest_asyncio.apply()
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().handlers = []
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SimpleDirectoryReader,
       ServiceContext,
       StorageContext,
       SimpleKeywordTableIndex,
       KnowledgeGraphIndex,
   )
   from llama_index.response.notebook_utils import display_response
   from llama_index.llms import OpenAI
   Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   NumExpr defaulting to 8 threads.
Load Data
We first show how to convert a Document into a set of Nodes, and
insert into a DocumentStore.
   from llama_index import SimpleDirectoryReader
   # try loading great gatsby
   documents = SimpleDirectoryReader(
       input_files=["../../../examples/gatsby/gatsby_full.txt"]
   ).load_data()
Define Query Engines
   # initialize service context (set chunk size)
   from llama_index.llms import OpenAI
   llm = OpenAI(model="gpt-3.5-turbo")
   service_context = ServiceContext.from_defaults(chunk_size=1024, llm=llm)
   nodes = service_context.node_parser.get_nodes_from_documents(documents)
   # initialize storage context (by default it's in-memory)
   storage_context = StorageContext.from_defaults()
   storage_context.docstore.add_documents(nodes)
   keyword_index = SimpleKeywordTableIndex(
       nodes,
       storage_context=storage_context,
       service_context=service_context,
       show_progress=True,
   )
   vector_index = VectorStoreIndex(
       nodes,
       storage_context=storage_context,
       service_context=service_context,
       show_progress=True,
   )
   # graph_index = KnowledgeGraphIndex(nodes, storage_context=storage_context, service_context=service_context, show_progress=True)
   Extracting keywords from nodes:   0%|          | 0/77 [00:00<?, ?it/s]
   Generating embeddings:   0%|          | 0/77 [00:00<?, ?it/s]
   from llama_index.prompts import PromptTemplate
   QA_PROMPT_TMPL = (
       "Context information is below.\n"
       "---------------------\n"
       "{context_str}\n"
       "---------------------\n"
       "Given the context information and not prior knowledge, "
       "answer the question. If the answer is not in the context, inform "
       "the user that you can't answer the question - DO NOT MAKE UP AN ANSWER.\n"
       "In addition to returning the answer, also return a relevance score as to "
       "how relevant the answer is to the question. "
       "Question: {query_str}\n"
       "Answer (including relevance score): "
   )
   QA_PROMPT = PromptTemplate(QA_PROMPT_TMPL)

Ensemble Query Engine Guide

   keyword_query_engine = keyword_index.as_query_engine(text_qa_template=QA_PROMPT)
   vector_query_engine = vector_index.as_query_engine(text_qa_template=QA_PROMPT)
   response = vector_query_engine.query(
       "Describe and summarize the interactions between Gatsby and Daisy"
   )
   print(response)
   Gatsby and Daisy's interactions are described as intimate and conspiring. They sit opposite each other at a kitchen table, with Gatsby's hand covering Daisy's hand. They communicate through nods and seem to have a natural intimacy. Gatsby waits for Daisy to go to bed and is reluctant to leave until he knows what she will do. They have a conversation in which Gatsby tells the story of his youth with Dan Cody. Daisy's face is smeared with tears, but Gatsby glows with a new well-being. Gatsby invites Daisy to his house and expresses his desire for her to come. They admire Gatsby's house together and discuss the interesting people who visit. The relevance score of this answer is 10/10.
   response = keyword_query_engine.query(
       "Describe and summarize the interactions between Gatsby and Daisy"
   )
   > Starting query: Describe and summarize the interactions between Gatsby and Daisy
   query keywords: ['describe', 'interactions', 'gatsby', 'summarize', 'daisy']
   > Extracted keywords: ['gatsby', 'daisy']
   print(response)
   The interactions between Gatsby and Daisy are characterized by a sense of tension and longing. Gatsby is visibly disappointed when Daisy expresses her dissatisfaction with their time together and insists that she didn't have a good time. He feels distant from her and struggles to make her understand his emotions. Gatsby dismisses the significance of the dance and instead focuses on his desire for Daisy to confess her love for him and leave Tom. He yearns for a deep connection with Daisy, but feels that she doesn't fully comprehend his feelings. These interactions highlight the complexities of their relationship and the challenges they face in rekindling their romance. The relevance score for these interactions is 8 out of 10.
Define Router Query Engine
   from llama_index.tools.query_engine import QueryEngineTool
   keyword_tool = QueryEngineTool.from_defaults(
       query_engine=keyword_query_engine,
       description="Useful for answering questions about this essay",
   )
   vector_tool = QueryEngineTool.from_defaults(
       query_engine=vector_query_engine,
       description="Useful for answering questions about this essay",
   )
   from llama_index.query_engine.router_query_engine import RouterQueryEngine
   from llama_index.selectors.llm_selectors import LLMSingleSelector, LLMMultiSelector
   from llama_index.selectors.pydantic_selectors import (
       PydanticMultiSelector,
       PydanticSingleSelector,
   )
   from llama_index.response_synthesizers import TreeSummarize
   TREE_SUMMARIZE_PROMPT_TMPL = (
       "Context information from multiple sources is below. Each source may or may not have \n"
       "a relevance score attached to it.\n"
       "---------------------\n"
       "{context_str}\n"
       "---------------------\n"
       "Given the information from multiple sources and their associated relevance scores (if provided) and not prior knowledge, "
       "answer the question. If the answer is not in the context, inform "
       "the user that you can't answer the question.\n"
       "Question: {query_str}\n"
       "Answer: "
   )
   tree_summarize = TreeSummarize(
       summary_template=PromptTemplate(TREE_SUMMARIZE_PROMPT_TMPL)
   )
   query_engine = RouterQueryEngine(
       selector=LLMMultiSelector.from_defaults(),
       query_engine_tools=[
           keyword_tool,
           vector_tool,
       ],
       summarizer=tree_summarize,
   )
Experiment with Queries

Ensemble Query Engine Guide

   response = await query_engine.aquery(
       "Describe and summarize the interactions between Gatsby and Daisy"
   )
   print(response)
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=1590 request_id=b049001384d0e2f2d96e308903351ca3 response_code=200
   Selecting query engine 0: Useful for answering questions about this essay.
   Selecting query engine 1: Useful for answering questions about this essay.
   > Starting query: Describe and summarize the interactions between Gatsby and Daisy
   query keywords: ['interactions', 'summarize', 'describe', 'daisy', 'gatsby']
   > Extracted keywords: ['daisy', 'gatsby']
   message='OpenAI API response' path=https://api.openai.com/v1/embeddings processing_ms=75 request_id=3f76f611bb063605c3c2365437480f87 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=4482 request_id=597221bd776638356f16034c4d8ad2f6 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=5773 request_id=50a6030879054f470a1e45952b4b80b3 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=6478 request_id=9171e42c7ced18baedc77cc89ec7478c response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=6166 request_id=f3218012e3f9a12e00daeee0b9b06f67 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=4808 request_id=ab6887cbec9a44c2342d6402e28129d6 response_code=200
   Combining responses from multiple query engines.
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=4506 request_id=5fd128dab043f58111521d19e7c4f59a response_code=200
   The interactions between Gatsby and Daisy are portrayed as intense, passionate, and filled with longing and desire. Gatsby is deeply in love with Daisy and throws extravagant parties in the hopes of winning her back. Despite Daisy's marriage to Tom Buchanan, they reconnect and begin an affair. They spend time together at Gatsby's lavish house and even plan to run away together. However, their relationship ends tragically when Daisy accidentally kills Tom's mistress, Myrtle, while driving Gatsby's car. Gatsby takes the blame for the accident and is later killed by Myrtle's husband. Overall, their interactions explore themes of love, wealth, and the pursuit of happiness.
   response.source_nodes
   []
   response = await query_engine.aquery(
       "What part of his past is Gatsby trying to recapture?"
   )
   print(response)
   Selecting query engine 0: Keywords: Gatsby, past, recapture.
   > Starting query: What part of his past is Gatsby trying to recapture?
   query keywords: ['gatsby', 'past', 'recapture']
   > Extracted keywords: ['gatsby', 'past']
   KeyboardInterrupt
Compare Against Baseline
Compare against a baseline of chunk size 1024 (k=2)
   query_engine_1024 = query_engines[-1]

Ensemble Query Engine Guide

   response_1024 = query_engine_1024.query(
       "Describe and summarize the interactions between Gatsby and Daisy"
   )
   display_response(response_1024, show_source=True, source_length=500)


FLARE Query Engine

Adapted from the paper "Active Retrieval Augmented Generation"
Currently implements FLARE Instruct, which tells the LLM to generate
retrieval instructions.
   import os
   from llama_index.llms import OpenAI
   from llama_index.query_engine import FLAREInstructQueryEngine
   from llama_index import (
       VectorStoreIndex,
       SimpleDirectoryReader,
       StorageContext,
       LLMPredictor,
       ServiceContext,
   )
   /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
   service_context = ServiceContext.from_defaults(
       llm=OpenAI(model="gpt-4", temperature=0), chunk_size=512
   )
   documents = SimpleDirectoryReader("../data/paul_graham").load_data()
   index = VectorStoreIndex.from_documents(documents, service_context=service_context)
   index_query_engine = index.as_query_engine(similarity_top_k=2)
   flare_query_engine = FLAREInstructQueryEngine(
       query_engine=index_query_engine,
       service_context=service_context,
       max_iterations=7,
       verbose=True,
   )
   flare_query_engine
   <llama_index.query_engine.flare.base.FLAREInstructQueryEngine at 0x1475ba7a0>
   response = flare_query_engine.query(
       "Can you tell me about the author's trajectory in the startup world?"
   )
   Query: Can you tell me about the author's trajectory in the startup world?
   Current response:
   Lookahead response: The author began their journey in the startup world by [Search(What did the author do in the startup world?)]
   Updated lookahead response: The author began their journey in the startup world by co-founding Y Combinator (YC), a startup accelerator that provided funding and support to startups in batches. They aimed to fix issues in the venture capital industry by making a larger number of smaller investments, funding younger and more technical founders, and allowing founders to remain as CEOs. The author also wrote Hacker News, a news aggregator initially for startup founders, in a new version of Arc programming language.
   Current response:  The author began their journey in the startup world by co-founding Y Combinator (YC), a startup accelerator that provided funding and support to startups in batches. They aimed to fix issues in the venture capital industry by making a larger number of smaller investments, funding younger and more technical founders, and allowing founders to remain as CEOs. The author also wrote Hacker News, a news aggregator initially for startup founders, in a new version of Arc programming language.
   Lookahead response: Since then, the author has been involved in mentoring and advising numerous startups, helping them grow and succeed in their respective industries. [Search(What are some notable startups the author has worked with?)]
   Updated lookahead response: Since then, the author has been involved in mentoring and advising numerous startups, helping them grow and succeed in their respective industries. Some notable startups the author has worked with include Reddit, Justin Kan and Emmett Shear (who went on to found Twitch), Aaron Swartz (who helped write the RSS spec), and Sam Altman (who later became the second president of YC).

FLARE Query Engine

   Current response: The author began their journey in the startup world by co-founding Y Combinator (YC), a startup accelerator that provided funding and support to startups in batches. They aimed to fix issues in the venture capital industry by making a larger number of smaller investments, funding younger and more technical founders, and allowing founders to remain as CEOs. The author also wrote Hacker News, a news aggregator initially for startup founders, in a new version of Arc programming language. Since then, the author has been involved in mentoring and advising numerous startups, helping them grow and succeed in their respective industries. Some notable startups the author has worked with include Reddit, Justin Kan and Emmett Shear (who went on to found Twitch), Aaron Swartz (who helped write the RSS spec), and Sam Altman (who later became the second president of YC).
   Lookahead response: done

   print(response)
   The author began their journey in the startup world by co-founding Y Combinator (YC), a startup accelerator that provided funding and support to startups in batches. They aimed to fix issues in the venture capital industry by making a larger number of smaller investments, funding younger and more technical founders, and allowing founders to remain as CEOs. The author also wrote Hacker News, a news aggregator initially for startup founders, in a new version of Arc programming language. Since then, the author has been involved in mentoring and advising numerous startups, helping them grow and succeed in their respective industries. Some notable startups the author has worked with include Reddit, Justin Kan and Emmett Shear (who went on to found Twitch), Aaron Swartz (who helped write the RSS spec), and Sam Altman (who later became the second president of YC).
   response = flare_query_engine.query(
       "Can you tell me about what the author did during his time at YC?"
   )
   Query: Can you tell me about what the author did during his time at YC?
   Current response:
   Lookahead response: During his time at YC, the author [Search(What did the author do at YC?)]
   Updated lookahead response: During his time at YC, the author worked on selecting and helping founders at YC, solving their problems, and engaging with their startups. They also wrote all of YC's internal software in Arc and managed Hacker News, which was a source of stress for them.
   Current response:  During his time at YC, the author worked on selecting and helping founders at YC, solving their problems, and engaging with their startups. They also wrote all of YC's internal software in Arc and managed Hacker News, which was a source of stress for them.
   Lookahead response: done

   print(response)
   During his time at YC, the author worked on selecting and helping founders at YC, solving their problems, and engaging with their startups. They also wrote all of YC's internal software in Arc and managed Hacker News, which was a source of stress for them.
   response = flare_query_engine.query(
       "Tell me about the author's life from childhood to adulthood"
   )
   Query: Tell me about the author's life from childhood to adulthood
   Current response:

FLARE Query Engine

   Lookahead response: The author grew up in a small town, where they [Search(What did the author do during their childhood?)] and later went on to attend college, majoring in [Search(What did the author major in during college?)].
   Updated lookahead response: The author grew up in a small town, where they mainly worked on writing and programming outside of school. They wrote short stories and tried programming on the IBM 1401 using an early version of Fortran and later went on to attend college, majoring in
   Current response:  The author grew up in a small town, where they mainly worked on writing and programming outside of school. They wrote short stories and tried programming on the IBM 1401 using an early version of Fortran and later went on to attend college, majoring in
   Lookahead response: computer science and English literature. After college, they [Search(What did the author do after college?)]
   Updated lookahead response: computer science and English literature. After college, they wrote essays on various topics, worked on spam filters, did some painting, and hosted dinners for friends. They also bought a building in Cambridge to use as an office. Later, the author applied to art schools, got accepted into RISD, and attended their foundation classes. They also received an invitation to take the entrance exam at the Accademia di Belli Arti in Florence.
   Current response: The author grew up in a small town, where they mainly worked on writing and programming outside of school. They wrote short stories and tried programming on the IBM 1401 using an early version of Fortran and later went on to attend college, majoring in computer science and English literature. After college, they wrote essays on various topics, worked on spam filters, did some painting, and hosted dinners for friends. They also bought a building in Cambridge to use as an office. Later, the author applied to art schools, got accepted into RISD, and attended their foundation classes. They also received an invitation to take the entrance exam at the Accademia di Belli Arti in Florence.
   Lookahead response: During their time at RISD and the Accademia di Belli Arti, the author honed their artistic skills and further developed their writing, eventually transitioning into a successful career as an author and artist. [Search(What did the author achieve in their career?)]
   Updated lookahead response: During their time at RISD and the Accademia di Belli Arti, the author honed their artistic skills and further developed their writing, eventually transitioning into a successful career as an author and artist. The author achieved several things in their career, including publishing essays online, writing a book called "Hackers & Painters," working on spam filters, doing some painting, and hosting dinners for friends. They also discussed ideas about venture capital and how it could be improved.
   Current response: The author grew up in a small town, where they mainly worked on writing and programming outside of school. They wrote short stories and tried programming on the IBM 1401 using an early version of Fortran and later went on to attend college, majoring in computer science and English literature. After college, they wrote essays on various topics, worked on spam filters, did some painting, and hosted dinners for friends. They also bought a building in Cambridge to use as an office. Later, the author applied to art schools, got accepted into RISD, and attended their foundation classes. They also received an invitation to take the entrance exam at the Accademia di Belli Arti in Florence. During their time at RISD and the Accademia di Belli Arti, the author honed their artistic skills and further developed their writing, eventually transitioning into a successful career as an author and artist. The author achieved several things in their career, including publishing essays online, writing a book called "Hackers & Painters," working on spam filters, doing some painting, and hosting dinners for friends. They also discussed ideas about venture capital and how it could be improved.

FLARE Query Engine

   Lookahead response: done

   print(response)
   The author grew up in a small town, where they mainly worked on writing and programming outside of school. They wrote short stories and tried programming on the IBM 1401 using an early version of Fortran and later went on to attend college, majoring in computer science and English literature. After college, they wrote essays on various topics, worked on spam filters, did some painting, and hosted dinners for friends. They also bought a building in Cambridge to use as an office. Later, the author applied to art schools, got accepted into RISD, and attended their foundation classes. They also received an invitation to take the entrance exam at the Accademia di Belli Arti in Florence. During their time at RISD and the Accademia di Belli Arti, the author honed their artistic skills and further developed their writing, eventually transitioning into a successful career as an author and artist. The author achieved several things in their career, including publishing essays online, writing a book called "Hackers & Painters," working on spam filters, doing some painting, and hosting dinners for friends. They also discussed ideas about venture capital and how it could be improved.
   response = index_query_engine.query(
       "Can you tell me about the author's trajectory in the startup world?"
   )
   print(str(response))
   The author's trajectory in the startup world began with their involvement in various projects and activities, such as writing essays on different topics, working on spam filters, and painting. They also hosted dinners for friends, which helped them learn how to cook for groups and network with people from various backgrounds.
   In October 2003, the author met Jessica Livingston at a party, who later became a significant figure in their startup journey. Jessica worked in marketing at a Boston investment bank and was intrigued by the stories of startup founders she met through the author. She decided to compile a book of interviews with these founders.
   In early 2005, Jessica interviewed for a marketing job at a Boston VC firm, which led the author to discuss the issues with venture capital and how it could be improved. The author also gave a talk at the Harvard Computer Society about starting a startup, which made them realize they should start angel investing.
   On March 11, the author, Jessica, and their friends Robert and Trevor decided to start their own investment firm, implementing the ideas they had discussed. They founded Y Combinator, an angel investment firm that made unconventional choices in the startup world. The author's trajectory in the startup world has been marked by their involvement in various projects, networking, and eventually co-founding a successful investment firm.
   response = index_query_engine.query(
       "Tell me about the author's life from childhood to adulthood"
   )
   print(str(response))
   The author's life from childhood to adulthood includes a variety of experiences and interests. They wrote numerous essays on various topics, which were later compiled into a book called Hackers & Painters. They also worked on spam filters and pursued painting as a hobby. The author used to host dinners for friends every Thursday night, which taught them how to cook for groups. They bought a building in Cambridge, which was a former candy factory and later a porn studio, to use as an office.
   In October 2003, the author met Jessica Livingston at a party, and they started dating a few days later. Jessica worked in marketing at a Boston investment bank and later decided to compile a book of interviews with startup founders. When she was looking for a new job, the author shared their thoughts on how venture capital should be improved.
   The author also attended the Accademia, a prestigious institution, to study painting. However, they were disappointed with the lack of teaching and learning taking place there. The author painted still lives in their bedroom at night, using leftover scraps of canvas.


SQL Router Query Engine

In this tutorial, we define a custom router query engine that can
route to either a SQL database or a vector database.
Setup
   # NOTE: This is ONLY necessary in jupyter notebook.
   # Details: Jupyter runs an event-loop behind the scenes.
   #          This results in nested event-loops when we start an event-loop to make async queries.
   #          This is normally not allowed, we use nest_asyncio to allow it for convenience.
   import nest_asyncio
   nest_asyncio.apply()
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SimpleDirectoryReader,
       ServiceContext,
       StorageContext,
       SQLDatabase,
       WikipediaReader,
   )
   INFO:numexpr.utils:Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   INFO:numexpr.utils:NumExpr defaulting to 8 threads.
   NumExpr defaulting to 8 threads.
   /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
Create Database Schema + Test Data
Here we introduce a toy scenario where there are 100 tables (too big
to fit into the prompt)
   from sqlalchemy import (
       create_engine,
       MetaData,
       Table,
       Column,
       String,
       Integer,
       select,
       column,
   )
   engine = create_engine("sqlite:///:memory:", future=True)
   metadata_obj = MetaData()
   # create city SQL table
   table_name = "city_stats"
   city_stats_table = Table(
       table_name,
       metadata_obj,
       Column("city_name", String(16), primary_key=True),
       Column("population", Integer),
       Column("country", String(16), nullable=False),
   )
   metadata_obj.create_all(engine)
   # print tables
   metadata_obj.tables.keys()
   dict_keys(['city_stats'])
We introduce some test data into the "city_stats" table
   from sqlalchemy import insert
   rows = [
       {"city_name": "Toronto", "population": 2930000, "country": "Canada"},
       {"city_name": "Tokyo", "population": 13960000, "country": "Japan"},
       {"city_name": "Berlin", "population": 3645000, "country": "Germany"},
   ]
   for row in rows:
       stmt = insert(city_stats_table).values(**row)
       with engine.begin() as connection:
           cursor = connection.execute(stmt)
   with engine.connect() as connection:
       cursor = connection.exec_driver_sql("SELECT * FROM city_stats")
       print(cursor.fetchall())
   [('Toronto', 2930000, 'Canada'), ('Tokyo', 13960000, 'Japan'), ('Berlin', 3645000, 'Germany')]
Load Data
We first show how to convert a Document into a set of Nodes, and
insert into a DocumentStore.
   # install wikipedia python package
   !pip install wikipedia
   Requirement already satisfied: wikipedia in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (1.4.0)
   Requirement already satisfied: requests<3.0.0,>=2.0.0 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from wikipedia) (2.28.2)

SQL Router Query Engine

   Requirement already satisfied: beautifulsoup4 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from wikipedia) (4.12.2)
   Requirement already satisfied: idna<4,>=2.5 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (3.4)
   Requirement already satisfied: charset-normalizer<4,>=2 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (3.1.0)
   Requirement already satisfied: certifi>=2017.4.17 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2022.12.7)
   Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (1.26.15)
   Requirement already satisfied: soupsieve>1.2 in /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages (from beautifulsoup4->wikipedia) (2.4.1)
   [notice] A new release of pip available: 22.3.1 -> 23.1.2
   [notice] To update, run: pip install --upgrade pip
   cities = ["Toronto", "Berlin", "Tokyo"]
   wiki_docs = WikipediaReader().load_data(pages=cities)
Build SQL Index
   sql_database = SQLDatabase(engine, include_tables=["city_stats"])
   from llama_index.indices.struct_store.sql_query import NLSQLTableQueryEngine
   sql_query_engine = NLSQLTableQueryEngine(
       sql_database=sql_database,
       tables=["city_stats"],
   )
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens
   > [build_index_from_nodes] Total embedding token usage: 0 tokens
   /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages/langchain/sql_database.py:227: UserWarning: This method is deprecated - please use `get_usable_table_names`.
     warnings.warn(
Build Vector Index
   # build a separate vector index per city
   # You could also choose to define a single vector index across all docs, and annotate each chunk by metadata
   vector_indices = []
   for wiki_doc in wiki_docs:
       vector_index = VectorStoreIndex.from_documents([wiki_doc])
       vector_indices.append(vector_index)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens

SQL Router Query Engine

   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 20744 tokens
   > [build_index_from_nodes] Total embedding token usage: 20744 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 21947 tokens
   > [build_index_from_nodes] Total embedding token usage: 21947 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 12786 tokens
   > [build_index_from_nodes] Total embedding token usage: 12786 tokens
Define Query Engines, Set as Tools
   vector_query_engines = [index.as_query_engine() for index in vector_indices]
   from llama_index.tools.query_engine import QueryEngineTool
   sql_tool = QueryEngineTool.from_defaults(
       query_engine=sql_query_engine,
       description=(
           "Useful for translating a natural language query into a SQL query over a table containing: "
           "city_stats, containing the population/country of each city"
       ),
   )
   vector_tools = []
   for city, query_engine in zip(cities, vector_query_engines):
       vector_tool = QueryEngineTool.from_defaults(
           query_engine=query_engine,
           description=f"Useful for answering semantic questions about {city}",
       )
       vector_tools.append(vector_tool)
Define Router Query Engine
   from llama_index.query_engine.router_query_engine import RouterQueryEngine
   from llama_index.selectors.llm_selectors import LLMSingleSelector
   query_engine = RouterQueryEngine(
       selector=LLMSingleSelector.from_defaults(),
       query_engine_tools=([sql_tool] + vector_tools),
   )
   response = query_engine.query("Which city has the highest population?")
   print(str(response))
   INFO:llama_index.query_engine.router_query_engine:Selecting query engine 0: Useful for translating a natural language query into a SQL query over a table containing: city_stats, containing the population/country of each city.
   Selecting query engine 0: Useful for translating a natural language query into a SQL query over a table containing: city_stats, containing the population/country of each city.
   INFO:llama_index.indices.struct_store.sql_query:> Table desc str: Schema of table city_stats:
   Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)) and foreign keys: .
   > Table desc str: Schema of table city_stats:
   Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)) and foreign keys: .
   INFO:llama_index.token_counter.token_counter:> [query] Total LLM token usage: 347 tokens
   > [query] Total LLM token usage: 347 tokens
   INFO:llama_index.token_counter.token_counter:> [query] Total embedding token usage: 0 tokens
   > [query] Total embedding token usage: 0 tokens
    Tokyo has the highest population, with 13,960,000 people.
   response = query_engine.query("Tell me about the historical museums in Berlin")
   print(str(response))
   INFO:llama_index.query_engine.router_query_engine:Selecting query engine 2: Useful for answering semantic questions about Berlin.

SQL Router Query Engine

   Selecting query engine 2: Useful for answering semantic questions about Berlin.
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 8 tokens
   > [retrieve] Total embedding token usage: 8 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 2031 tokens
   > [get_response] Total LLM token usage: 2031 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   Berlin is home to many historical museums, including the Altes Museum, Neues Museum, Alte Nationalgalerie, Pergamon Museum, and Bode Museum, which are all located on Museum Island. The Gemäldegalerie (Painting Gallery) focuses on the paintings of the "old masters" from the 13th to the 18th centuries, while the Neue Nationalgalerie (New National Gallery, built by Ludwig Mies van der Rohe) specializes in 20th-century European painting. The Hamburger Bahnhof, in Moabit, exhibits a major collection of modern and contemporary art. The expanded Deutsches Historisches Museum reopened in the Zeughaus with an overview of German history spanning more than a millennium. The Bauhaus Archive is a museum of 20th-century design from the famous Bauhaus school. Museum Berggruen houses the collection of noted 20th century collector Heinz Berggruen, and features an extensive assortment of works by Picasso, Matisse, Cézanne, and Giacometti, among others. The Kupferstichkabinett Berlin (Museum of Prints and Drawings) is part of the Staatlichen Museen z
   response = query_engine.query("Which countries are each city from?")
   print(str(response))
   INFO:llama_index.query_engine.router_query_engine:Selecting query engine 0: Useful for translating a natural language query into a SQL query over a table containing: city_stats, containing the population/country of each city.
   Selecting query engine 0: Useful for translating a natural language query into a SQL query over a table containing: city_stats, containing the population/country of each city.
   INFO:llama_index.indices.struct_store.sql_query:> Table desc str: Schema of table city_stats:
   Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)) and foreign keys: .
   > Table desc str: Schema of table city_stats:
   Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)) and foreign keys: .
   INFO:llama_index.token_counter.token_counter:> [query] Total LLM token usage: 334 tokens
   > [query] Total LLM token usage: 334 tokens
   INFO:llama_index.token_counter.token_counter:> [query] Total embedding token usage: 0 tokens
   > [query] Total embedding token usage: 0 tokens
    Toronto is from Canada, Tokyo is from Japan, and Berlin is from Germany.