Composable Graph

   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import VectorStoreIndex, SimpleKeywordTableIndex, SimpleDirectoryReader
Load Datasets
Load both the NYC Wikipedia page as well as Paul Graham's "What I
Worked On" essay
   # fetch "New York City" page from Wikipedia
   from pathlib import Path
   import requests
   response = requests.get(
       "https://en.wikipedia.org/w/api.php",
       params={
           "action": "query",
           "format": "json",
           "titles": "New York City",
           "prop": "extracts",
           # 'exintro': True,
           "explaintext": True,
       },
   ).json()
   page = next(iter(response["query"]["pages"].values()))
   nyc_text = page["extract"]
   data_path = Path("data")
   if not data_path.exists():
       Path.mkdir(data_path)
   with open("../test_wiki/data/nyc_text.txt", "w") as fp:
       fp.write(nyc_text)
   # load NYC dataset
   nyc_documents = SimpleDirectoryReader("../test_wiki/data/").load_data()
   # load PG's essay
   essay_documents = SimpleDirectoryReader("../paul_graham_essay/data/").load_data()
Building the document indices
Build a tree index for the NYC wiki page and PG essay
   # build NYC index
   nyc_index = VectorStoreIndex.from_documents(nyc_documents)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 28492 tokens
   > [build_index_from_nodes] Total embedding token usage: 28492 tokens
   # build essay index
   essay_index = VectorStoreIndex.from_documents(essay_documents)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 17617 tokens
   > [build_index_from_nodes] Total embedding token usage: 17617 tokens
Set summaries for the indices
Add text summaries to indices, so we can compose other indices on top
of it
   nyc_index_summary = """
       New York, often called New York City or NYC,
       is the most populous city in the United States.
       With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2),
       New York City is also the most densely populated major city in the United States,
       and is more than twice as populous as second-place Los Angeles.
       New York City lies at the southern tip of New York State, and
       constitutes the geographical and demographic center of both the
       Northeast megalopolis and the New York metropolitan area, the
       largest metropolitan area in the world by urban landmass.[8] With over
       20.1 million people in its metropolitan statistical area and 23.5 million
       in its combined statistical area as of 2020, New York is one of the world's
       most populous megacities, and over 58 million people live within 250 mi (400 km) of
       the city. New York City is a global cultural, financial, and media center with
       a significant influence on commerce, health care and life sciences, entertainment,

Composable Graph

       research, technology, education, politics, tourism, dining, art, fashion, and sports.
       Home to the headquarters of the United Nations,
       New York is an important center for international diplomacy,
       an established safe haven for global investors, and is sometimes described as the capital of the world.
   """
   essay_index_summary = """
       Author: Paul Graham.
       The author grew up painting and writing essays.
       He wrote a book on Lisp and did freelance Lisp hacking work to support himself.
       He also became the de facto studio assistant for Idelle Weber, an early photorealist painter.
       He eventually had the idea to start a company to put art galleries online, but the idea was unsuccessful.
       He then had the idea to write software to build online stores, which became the basis for his successful company, Viaweb.
       After Viaweb was acquired by Yahoo!, the author returned to painting and started writing essays online.
       He wrote a book of essays, Hackers & Painters, and worked on spam filters.
       He also bought a building in Cambridge to use as an office.
       He then had the idea to start Y Combinator, an investment firm that would
       make a larger number of smaller investments and help founders remain as CEO.
       He and his partner Jessica Livingston ran Y Combinator and funded a batch of startups twice a year.
       He also continued to write essays, cook for groups of friends, and explore the concept of invented vs discovered in software.
   """
Build Keyword Table Index on top of tree indices!
We set summaries for each of the NYC and essay indices, and then
compose a keyword index on top of it.
   from llama_index.indices.composability import ComposableGraph
   graph = ComposableGraph.from_indices(
       SimpleKeywordTableIndex,
       [nyc_index, essay_index],
       index_summaries=[nyc_index_summary, essay_index_summary],
       max_keywords_per_chunk=50,
   )
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens
   > [build_index_from_nodes] Total embedding token usage: 0 tokens
   # set Logging to DEBUG for more detailed outputs
   # ask it a question about NYC
   query_engine = graph.as_query_engine()
   response = query_engine.query(
       "What is the climate of New York City like? How cold is it during the winter?",
   )
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: What is the climate of New York City like? How cold is it during the winter?
   > Starting query: What is the climate of New York City like? How cold is it during the winter?
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['cold', 'new york city', 'winter', 'new', 'city', 'climate', 'york']
   query keywords: ['cold', 'new york city', 'winter', 'new', 'city', 'climate', 'york']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['new', 'city', 'york']
   > Extracted keywords: ['new', 'city', 'york']
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 18 tokens
   > [retrieve] Total embedding token usage: 18 tokens

Composable Graph

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 3834 tokens
   > [get_response] Total LLM token usage: 3834 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 282 tokens
   > [get_response] Total LLM token usage: 282 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   print(str(response))
   The climate of New York City is humid subtropical, with hot and humid summers and cold, wet winters. The average temperature in the winter is around 32°F (0°C), but temperatures can drop below freezing. Snowfall is common in the winter months, with an average of 25 inches (63 cm) of snow per year.
   # Get source of response
   print(response.get_formatted_sources())
   > Source (Doc id: b58b74a6-c0c8-4020-8076-fdcd265dc7a3):
   The climate of New York City is humid subtropical, with hot and humid summers and cold, wet win...
   > Source (Doc id: e92aafcf-08c2-4a8c-897b-930ad420179a): one of the world's highest. New York City real estate is a safe haven for global investors.
   ===...
   # ask it a question about PG's essay
   response = query_engine.query(
       "What did the author do growing up, before his time at Y Combinator?",
   )
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: What did the author do growing up, before his time at Y Combinator?
   > Starting query: What did the author do growing up, before his time at Y Combinator?
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['growing up', 'y combinator', 'time', 'growing', 'author', 'combinator']
   query keywords: ['growing up', 'y combinator', 'time', 'growing', 'author', 'combinator']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['author', 'combinator']
   > Extracted keywords: ['author', 'combinator']
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 17 tokens
   > [retrieve] Total embedding token usage: 17 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 3947 tokens
   > [get_response] Total LLM token usage: 3947 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 218 tokens
   > [get_response] Total LLM token usage: 218 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens

Composable Graph

   print(str(response))
   The author likely grew up doing a variety of activities, such as writing essays, painting, cooking, writing software, and hosting dinners for friends. He may have also been involved in giving talks and was likely driven by the idea of working hard to set the upper bound for everyone else.
   # Get source of response
   print(response.get_formatted_sources())
   > Source (Doc id: 92bc5ce3-3a76-4570-9726-f7e0405ec6cc):
   Before his time at Y Combinator, the author worked on building the infrastructure of the web, wr...
   > Source (Doc id: ed37130a-3138-42d4-9e77-1c792fe22f4e): write something and put it on the web, anyone can read it. That may seem obvious now, but it was ...


Composable Graph with Weaviate

   import logging
   import sys
   import weaviate
   from pprint import pprint
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SimpleKeywordTableIndex,
       SummaryIndex,
       VectorStoreIndex,
       SimpleDirectoryReader,
   )
   from llama_index.vector_stores import WeaviateVectorStore
   resource_owner_config = weaviate.AuthClientPassword(
       username="<username>",
       password="<password>",
   )
   client = weaviate.Client(
       "https://test-weaviate-cluster.semi.network/",
       auth_client_secret=resource_owner_config,
   )
   # [optional] set batch
   client.batch.configure(batch_size=10)
Load Datasets
Load both the NYC Wikipedia page as well as Paul Graham's "What I
Worked On" essay
   # fetch "New York City" page from Wikipedia
   from pathlib import Path
   import requests
   response = requests.get(
       "https://en.wikipedia.org/w/api.php",
       params={
           "action": "query",
           "format": "json",
           "titles": "New York City",
           "prop": "extracts",
           # 'exintro': True,
           "explaintext": True,
       },
   ).json()
   page = next(iter(response["query"]["pages"].values()))
   nyc_text = page["extract"]
   data_path = Path("data")
   if not data_path.exists():
       Path.mkdir(data_path)
   with open("../test_wiki/data/nyc_text.txt", "w") as fp:
       fp.write(nyc_text)
   # load NYC dataset
   nyc_documents = SimpleDirectoryReader("../test_wiki/data/").load_data()
   # load PG's essay
   essay_documents = SimpleDirectoryReader("../paul_graham_essay/data/").load_data()
Building the document indices
Build a tree index for the NYC wiki page and PG essay
   # build NYC index
   from llama_index.storage.storage_context import StorageContext
   vector_store = WeaviateVectorStore(weaviate_client=client, index_name="Nyc_docs")
   storage_context = StorageContext.from_defaults(vector_store=vector_store)
   nyc_index = VectorStoreIndex.from_documents(
       nyc_documents, storage_context=storage_context
   )
   # build essay index
   vector_store = WeaviateVectorStore(weaviate_client=client, index_name="Essay_docs")
   storage_context = StorageContext.from_defaults(vector_store=vector_store)
   essay_index = VectorStoreIndex.from_documents(
       essay_documents, storage_context=storage_context
   )
Set summaries for the indices
Add text summaries to indices, so we can compose other indices on top
of it
   nyc_index_summary = """
       New York, often called New York City or NYC,
       is the most populous city in the United States.
       With a 2020 population of 8,804,190 distributed over 300.46 square miles (778.2 km2),
       New York City is also the most densely populated major city in the United States,
       and is more than twice as populous as second-place Los Angeles.
       New York City lies at the southern tip of New York State, and
       constitutes the geographical and demographic center of both the
       Northeast megalopolis and the New York metropolitan area, the
       largest metropolitan area in the world by urban landmass.[8] With over
       20.1 million people in its metropolitan statistical area and 23.5 million
       in its combined statistical area as of 2020, New York is one of the world's
       most populous megacities, and over 58 million people live within 250 mi (400 km) of

Composable Graph with Weaviate

       the city. New York City is a global cultural, financial, and media center with
       a significant influence on commerce, health care and life sciences, entertainment,
       research, technology, education, politics, tourism, dining, art, fashion, and sports.
       Home to the headquarters of the United Nations,
       New York is an important center for international diplomacy,
       an established safe haven for global investors, and is sometimes described as the capital of the world.
   """
   essay_index_summary = """
       Author: Paul Graham.
       The author grew up painting and writing essays.
       He wrote a book on Lisp and did freelance Lisp hacking work to support himself.
       He also became the de facto studio assistant for Idelle Weber, an early photorealist painter.
       He eventually had the idea to start a company to put art galleries online, but the idea was unsuccessful.
       He then had the idea to write software to build online stores, which became the basis for his successful company, Viaweb.
       After Viaweb was acquired by Yahoo!, the author returned to painting and started writing essays online.
       He wrote a book of essays, Hackers & Painters, and worked on spam filters.
       He also bought a building in Cambridge to use as an office.
       He then had the idea to start Y Combinator, an investment firm that would
       make a larger number of smaller investments and help founders remain as CEO.
       He and his partner Jessica Livingston ran Y Combinator and funded a batch of startups twice a year.
       He also continued to write essays, cook for groups of friends, and explore the concept of invented vs discovered in software.
   """
   index_summaries = [nyc_index_summary, essay_index_summary]
   nyc_index.set_index_id("nyc_index")
   essay_index.set_index_id("essay_index")
Build Keyword Table Index on top of vector indices!
We set summaries for each of the NYC and essay indices, and then
compose a keyword index on top of it.
Define Graph
   from llama_index.indices.composability import ComposableGraph
   graph = ComposableGraph.from_indices(
       SimpleKeywordTableIndex,
       [nyc_index, essay_index],
       index_summaries=index_summaries,
       max_keywords_per_chunk=50,
   )
   custom_query_engines = {
       graph.root_id: graph.root_index.as_query_engine(retriever_mode="simple")
   }
   query_engine = graph.as_query_engine(
       custom_query_engines=custom_query_engines,
   )
   # set Logging to DEBUG for more detailed outputs
   # ask it a question about NYC
   response = query_engine.query(
       "What is the weather of New York City like? How cold is it during the winter?",
   )
   print(str(response))
   # Get source of response
   print(response.get_formatted_sources())
   # ask it a question about PG's essay
   response = query_engine.query(
       "What did the author do growing up, before his time at Y Combinator?",
   )
   print(str(response))
   # Get source of response
   print(response.get_formatted_sources())


DeepLake + LlamaIndex

Look at financial statements
   !pip install llama-index deeplake
   Requirement already satisfied: llama-index in ./GPTIndex/lib/python3.9/site-packages (0.6.37)
   Requirement already satisfied: deeplake in ./GPTIndex/lib/python3.9/site-packages (3.6.7)
   Requirement already satisfied: urllib3<2 in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (1.26.7)
   Requirement already satisfied: numpy in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (1.24.2)
   Requirement already satisfied: sqlalchemy>=2.0.15 in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (2.0.17)
   Requirement already satisfied: pandas in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (2.0.0)
   Requirement already satisfied: typing-inspect==0.8.0 in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (0.8.0)
   Requirement already satisfied: langchain>=0.0.218 in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (0.0.219)
   Requirement already satisfied: fsspec>=2023.5.0 in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (2023.6.0)
   Requirement already satisfied: beautifulsoup4 in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (4.12.2)
   Requirement already satisfied: tiktoken in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (0.3.3)
   Requirement already satisfied: typing-extensions==4.5.0 in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (4.5.0)
   Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (8.2.2)
   Requirement already satisfied: dataclasses-json in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (0.5.7)
   Requirement already satisfied: openai>=0.26.4 in ./GPTIndex/lib/python3.9/site-packages (from llama-index) (0.27.4)
   Requirement already satisfied: mypy-extensions>=0.3.0 in ./GPTIndex/lib/python3.9/site-packages (from typing-inspect==0.8.0->llama-index) (1.0.0)
   Requirement already satisfied: pillow in ./GPTIndex/lib/python3.9/site-packages (from deeplake) (9.5.0)
   Requirement already satisfied: boto3 in ./GPTIndex/lib/python3.9/site-packages (from deeplake) (1.24.59)
   Requirement already satisfied: click in ./GPTIndex/lib/python3.9/site-packages (from deeplake) (8.1.3)
   Requirement already satisfied: pathos in ./GPTIndex/lib/python3.9/site-packages (from deeplake) (0.3.0)
   Requirement already satisfied: humbug>=0.3.1 in ./GPTIndex/lib/python3.9/site-packages (from deeplake) (0.3.1)
   Requirement already satisfied: tqdm in ./GPTIndex/lib/python3.9/site-packages (from deeplake) (4.65.0)
   Requirement already satisfied: numcodecs in ./GPTIndex/lib/python3.9/site-packages (from deeplake) (0.11.0)

DeepLake + LlamaIndex

   Requirement already satisfied: pyjwt in ./GPTIndex/lib/python3.9/site-packages (from deeplake) (2.6.0)
   Requirement already satisfied: aioboto3>=10.4.0 in ./GPTIndex/lib/python3.9/site-packages (from deeplake) (10.4.0)
   Requirement already satisfied: nest_asyncio in ./GPTIndex/lib/python3.9/site-packages (from deeplake) (1.5.6)
   Requirement already satisfied: aiobotocore[boto3]==2.4.2 in ./GPTIndex/lib/python3.9/site-packages (from aioboto3>=10.4.0->deeplake) (2.4.2)
   Requirement already satisfied: wrapt>=1.10.10 in ./GPTIndex/lib/python3.9/site-packages (from aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (1.15.0)
   Requirement already satisfied: aioitertools>=0.5.1 in ./GPTIndex/lib/python3.9/site-packages (from aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (0.11.0)
   Requirement already satisfied: botocore<1.27.60,>=1.27.59 in ./GPTIndex/lib/python3.9/site-packages (from aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (1.27.59)
   Requirement already satisfied: aiohttp>=3.3.1 in ./GPTIndex/lib/python3.9/site-packages (from aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (3.8.4)
   Requirement already satisfied: attrs>=17.3.0 in ./GPTIndex/lib/python3.9/site-packages (from aiohttp>=3.3.1->aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (22.2.0)
   Requirement already satisfied: aiosignal>=1.1.2 in ./GPTIndex/lib/python3.9/site-packages (from aiohttp>=3.3.1->aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (1.3.1)
   Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in ./GPTIndex/lib/python3.9/site-packages (from aiohttp>=3.3.1->aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (4.0.2)
   Requirement already satisfied: charset-normalizer<4.0,>=2.0 in ./GPTIndex/lib/python3.9/site-packages (from aiohttp>=3.3.1->aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (3.1.0)
   Requirement already satisfied: frozenlist>=1.1.1 in ./GPTIndex/lib/python3.9/site-packages (from aiohttp>=3.3.1->aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (1.3.3)
   Requirement already satisfied: multidict<7.0,>=4.5 in ./GPTIndex/lib/python3.9/site-packages (from aiohttp>=3.3.1->aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (6.0.4)

DeepLake + LlamaIndex

   Requirement already satisfied: yarl<2.0,>=1.0 in ./GPTIndex/lib/python3.9/site-packages (from aiohttp>=3.3.1->aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (1.8.2)
   Requirement already satisfied: jmespath<2.0.0,>=0.7.1 in ./GPTIndex/lib/python3.9/site-packages (from boto3->deeplake) (1.0.1)
   Requirement already satisfied: s3transfer<0.7.0,>=0.6.0 in ./GPTIndex/lib/python3.9/site-packages (from boto3->deeplake) (0.6.0)
   Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in ./GPTIndex/lib/python3.9/site-packages (from botocore<1.27.60,>=1.27.59->aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (2.8.2)
   Requirement already satisfied: requests in ./GPTIndex/lib/python3.9/site-packages (from humbug>=0.3.1->deeplake) (2.28.2)
   Requirement already satisfied: numexpr<3.0.0,>=2.8.4 in ./GPTIndex/lib/python3.9/site-packages (from langchain>=0.0.218->llama-index) (2.8.4)
   Requirement already satisfied: pydantic<2,>=1 in ./GPTIndex/lib/python3.9/site-packages (from langchain>=0.0.218->llama-index) (1.10.7)
   Requirement already satisfied: PyYAML>=5.4.1 in ./GPTIndex/lib/python3.9/site-packages (from langchain>=0.0.218->llama-index) (6.0)
   Requirement already satisfied: langchainplus-sdk>=0.0.17 in ./GPTIndex/lib/python3.9/site-packages (from langchain>=0.0.218->llama-index) (0.0.17)
   Requirement already satisfied: openapi-schema-pydantic<2.0,>=1.2 in ./GPTIndex/lib/python3.9/site-packages (from langchain>=0.0.218->llama-index) (1.2.4)
   Requirement already satisfied: marshmallow-enum<2.0.0,>=1.5.1 in ./GPTIndex/lib/python3.9/site-packages (from dataclasses-json->llama-index) (1.5.1)
   Requirement already satisfied: marshmallow<4.0.0,>=3.3.0 in ./GPTIndex/lib/python3.9/site-packages (from dataclasses-json->llama-index) (3.19.0)
   Requirement already satisfied: packaging>=17.0 in ./GPTIndex/lib/python3.9/site-packages (from marshmallow<4.0.0,>=3.3.0->dataclasses-json->llama-index) (23.1)
   Requirement already satisfied: six>=1.5 in ./GPTIndex/lib/python3.9/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.27.60,>=1.27.59->aiobotocore[boto3]==2.4.2->aioboto3>=10.4.0->deeplake) (1.16.0)
   Requirement already satisfied: certifi>=2017.4.17 in ./GPTIndex/lib/python3.9/site-packages (from requests->humbug>=0.3.1->deeplake) (2022.12.7)

DeepLake + LlamaIndex

   Requirement already satisfied: idna<4,>=2.5 in ./GPTIndex/lib/python3.9/site-packages (from requests->humbug>=0.3.1->deeplake) (3.4)
   Requirement already satisfied: soupsieve>1.2 in ./GPTIndex/lib/python3.9/site-packages (from beautifulsoup4->llama-index) (2.4.1)
   Requirement already satisfied: entrypoints in ./GPTIndex/lib/python3.9/site-packages (from numcodecs->deeplake) (0.4)
   Requirement already satisfied: pytz>=2020.1 in ./GPTIndex/lib/python3.9/site-packages (from pandas->llama-index) (2023.3)
   Requirement already satisfied: tzdata>=2022.1 in ./GPTIndex/lib/python3.9/site-packages (from pandas->llama-index) (2023.3)
   Requirement already satisfied: dill>=0.3.6 in ./GPTIndex/lib/python3.9/site-packages (from pathos->deeplake) (0.3.6)
   Requirement already satisfied: ppft>=1.7.6.6 in ./GPTIndex/lib/python3.9/site-packages (from pathos->deeplake) (1.7.6.6)
   Requirement already satisfied: pox>=0.3.2 in ./GPTIndex/lib/python3.9/site-packages (from pathos->deeplake) (0.3.2)
   Requirement already satisfied: multiprocess>=0.70.14 in ./GPTIndex/lib/python3.9/site-packages (from pathos->deeplake) (0.70.14)
   Requirement already satisfied: regex>=2022.1.18 in ./GPTIndex/lib/python3.9/site-packages (from tiktoken->llama-index) (2023.3.23)
   WARNING: You are using pip version 21.2.4; however, version 23.1.2 is available.
   You should consider upgrading via the '/Users/adilkhansarsen/Documents/work/LlamaIndex/llama_index/GPTIndex/bin/python3 -m pip install --upgrade pip' command.
   # My OpenAI Key
   import os
   import getpass
   os.environ["OPENAI_API_KEY"] = getpass.getpass("OpenAI token: ")
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SimpleKeywordTableIndex,
       SimpleDirectoryReader,
       LLMPredictor,
       ServiceContext,
       download_loader,
       Document,
   )
   from llama_index.vector_stores import DeepLakeVectorStore
   from llama_index.llms import OpenAI
   from typing import List, Optional, Tuple
   from pathlib import Path
   import requests
   import tqdm
   INFO:numexpr.utils:Note: NumExpr detected 10 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   Note: NumExpr detected 10 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   INFO:numexpr.utils:NumExpr defaulting to 8 threads.
   NumExpr defaulting to 8 threads.
   /Users/adilkhansarsen/Documents/work/LlamaIndex/llama_index/GPTIndex/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm

DeepLake + LlamaIndex

Ingest Data (PDFs of Financial Statements)
   # financial reports of amamzon, but can be replaced by any URLs of pdfs
   urls = [
       "https://s2.q4cdn.com/299287126/files/doc_financials/Q1_2018_-_8-K_Press_Release_FILED.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/Q2_2018_Earnings_Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_news/archive/Q318-Amazon-Earnings-Press-Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_news/archive/AMAZON.COM-ANNOUNCES-FOURTH-QUARTER-SALES-UP-20-TO-$72.4-BILLION.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/Q119_Amazon_Earnings_Press_Release_FINAL.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_news/archive/Amazon-Q2-2019-Earnings-Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_news/archive/Q3-2019-Amazon-Financial-Results.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_news/archive/Amazon-Q4-2019-Earnings-Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/2020/Q1/AMZN-Q1-2020-Earnings-Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/2020/q2/Q2-2020-Amazon-Earnings-Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/2020/q4/Amazon-Q4-2020-Earnings-Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/2021/q1/Amazon-Q1-2021-Earnings-Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/2021/q2/AMZN-Q2-2021-Earnings-Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/2021/q3/Q3-2021-Earnings-Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/2021/q4/business_and_financial_update.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/2022/q1/Q1-2022-Amazon-Earnings-Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/2022/q2/Q2-2022-Amazon-Earnings-Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/2022/q3/Q3-2022-Amazon-Earnings-Release.pdf",
       "https://s2.q4cdn.com/299287126/files/doc_financials/2022/q4/Q4-2022-Amazon-Earnings-Release.pdf",
   ]
   # hardcoding for now since we're missing q3 2020
   years = [
       2018,
       2018,
       2018,
       2018,
       2019,
       2019,
       2019,
       2019,
       2020,
       2020,
       2020,
       2021,
       2021,
       2021,
       2021,

DeepLake + LlamaIndex

       2022,
       2022,
       2022,
       2022,
   ]
   months = [1, 4, 7, 10, 1, 4, 7, 10, 1, 4, 10, 1, 4, 7, 10, 1, 4, 7, 10]
   zipped_data = list(zip(urls, months, years))
   PDFReader = download_loader("PDFReader")
   loader = PDFReader()
   def download_reports(
       data: List[Tuple[str, int, int]], out_dir: Optional[str] = None
   ) -> List[Document]:
       """Download pages from a list of urls."""
       docs = []
       out_dir = Path(out_dir or ".")
       if not out_dir.exists():
           print(out_dir)
           os.makedirs(out_dir)
       for url, month, year in tqdm.tqdm(data):
           path_base = url.split("/")[-1]
           out_path = out_dir / path_base
           if not out_path.exists():
               r = requests.get(url)
               with open(out_path, "wb") as f:
                   f.write(r.content)
           doc = loader.load_data(file=Path(out_path))[0]
           date_str = f"{month:02d}" + "-01-" + str(year)
           doc.extra_info = {"Date": date_str}
           docs.append(doc)
       return docs
   def _get_quarter_from_month(month: int) -> str:
       mapping = {1: "Q1", 4: "Q2", 7: "Q3", 10: "Q4"}
       return mapping[month]
   docs = download_reports(zipped_data, "data")
   100%|██████████| 19/19 [00:13<00:00,  1.44it/s]
Build Vector Indices
   llm_chatgpt = OpenAI(temperature=0, model="gpt-3.5-turbo-16k-0613")
   service_context = ServiceContext.from_defaults(llm=llm_chatgpt)
   /Users/adilkhansarsen/Documents/work/LlamaIndex/llama_index/GPTIndex/lib/python3.9/site-packages/langchain/llms/openai.py:769: UserWarning: You are trying to use a chat model. This way of initializing it is no longer supported. Instead, please use: `from langchain.chat_models import ChatOpenAI`
     warnings.warn(
   # Build city document index
   from llama_index.storage.storage_context import StorageContext
   # build vector index for each quarterly statement, store in dictionary
   dataset_root = "amazon_example/amazon_financial_"
   vector_indices = {}
   for idx, (_, month, year) in enumerate(zipped_data):
       doc = docs[idx]
       dataset_path = dataset_root + f"{month:02d}_{year}"
       vector_store = DeepLakeVectorStore(
           dataset_path=dataset_path,
           overwrite=True,
           verbose=False,
       )
       storage_context = StorageContext.from_defaults(vector_store=vector_store)
       vector_index = VectorStoreIndex.from_documents(
           [doc], storage_context=storage_context, service_context=service_context
       )
       vector_indices[(month, year)] = vector_index
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 1023 tokens
   > [build_index_from_nodes] Total embedding token usage: 1023 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens

DeepLake + LlamaIndex

   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 1118 tokens
   > [build_index_from_nodes] Total embedding token usage: 1118 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 917 tokens
   > [build_index_from_nodes] Total embedding token usage: 917 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 793 tokens
   > [build_index_from_nodes] Total embedding token usage: 793 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 927 tokens
   > [build_index_from_nodes] Total embedding token usage: 927 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 1021 tokens
   > [build_index_from_nodes] Total embedding token usage: 1021 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 845 tokens
   > [build_index_from_nodes] Total embedding token usage: 845 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 866 tokens
   > [build_index_from_nodes] Total embedding token usage: 866 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 895 tokens
   > [build_index_from_nodes] Total embedding token usage: 895 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 846 tokens
   > [build_index_from_nodes] Total embedding token usage: 846 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens

DeepLake + LlamaIndex

   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 702 tokens
   > [build_index_from_nodes] Total embedding token usage: 702 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 715 tokens
   > [build_index_from_nodes] Total embedding token usage: 715 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 763 tokens
   > [build_index_from_nodes] Total embedding token usage: 763 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 848 tokens
   > [build_index_from_nodes] Total embedding token usage: 848 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 842 tokens
   > [build_index_from_nodes] Total embedding token usage: 842 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 842 tokens
   > [build_index_from_nodes] Total embedding token usage: 842 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 839 tokens
   > [build_index_from_nodes] Total embedding token usage: 839 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 638 tokens
   > [build_index_from_nodes] Total embedding token usage: 638 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 701 tokens
   > [build_index_from_nodes] Total embedding token usage: 701 tokens
Test Querying a Vector Index
   response = (
       vector_indices[(1, 2018)]
       .as_query_engine(service_context=service_context)
       .query("What is the operating cash flow?")
   )
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens

DeepLake + LlamaIndex

   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 7 tokens
   > [retrieve] Total embedding token usage: 7 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1081 tokens
   > [get_response] Total LLM token usage: 1081 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   print(str(response))
   print(response.get_formatted_sources())
   The operating cash flow for the trailing twelve months ended March 31, 2018, was $18.2 billion.
   > Source (Doc id: e764aa30-7451-4c93-aac3-402bb1dd7aba): 1
   AMAZON.COM ANNOUNCES FIRST QUARTER SALES UP 43% TO $51.0 BILLION
   SEATTLE —(BUSINESS WIRE...
   > Source (Doc id: 934a2360-2fa6-4fbc-9706-2e0cc743d9be): and Insignia brands, available for purchase in 2018 through Best Buy stores,
   BestBuy.com, and Am...
   response = (
       vector_indices[(1, 2018)]
       .as_query_engine(service_context=service_context)
       .query("What are the updates on Whole Foods?")
   )
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 8 tokens
   > [retrieve] Total embedding token usage: 8 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1073 tokens
   > [get_response] Total LLM token usage: 1073 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   print(response)
   The given context information does not provide any updates on Whole Foods.
Build Graph: Keyword Table Index on top of vector indices!
We compose a keyword table index on top of all the vector indices.
   from llama_index.indices.composability.graph import ComposableGraph
   # set summary text for city
   index_summaries = {}
   for idx, (_, month, year) in enumerate(zipped_data):
       quarter_str = _get_quarter_from_month(month)
       index_summaries[
           (month, year)
       ] = f"Amazon Financial Statement, {quarter_str}, {year}"
   graph = ComposableGraph.from_indices(
       SimpleKeywordTableIndex,
       [index for _, index in vector_indices.items()],
       [summary for _, summary in index_summaries.items()],
       max_keywords_per_chunk=50,
   )
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens
   > [build_index_from_nodes] Total embedding token usage: 0 tokens
   from llama_index.indices.query.query_transform.base import DecomposeQueryTransform
   decompose_transform = DecomposeQueryTransform(
       service_context.llm_predictor, verbose=True
   )
   # TMP
   query_str = "Analyze revenue in Q1 of 2018."

DeepLake + LlamaIndex

   # with query decomposition in subindices
   from llama_index.query_engine.transform_query_engine import TransformQueryEngine
   custom_query_engines = {}
   for index in vector_indices.values():
       query_engine = index.as_query_engine(service_context=service_context)
       transform_metadata = {"index_summary": index.index_struct.summary}
       tranformed_query_engine = TransformQueryEngine(
           query_engine, decompose_transform, transform_metadata=transform_metadata
       )
       custom_query_engines[index.index_id] = tranformed_query_engine
   custom_query_engines[graph.root_index.index_id] = graph.root_index.as_query_engine(
       retriever_mode="simple",
       response_mode="tree_summarize",
       service_context=service_context,
   )
   query_engine_decompose = graph.as_query_engine(
       custom_query_engines=custom_query_engines,
   )
   from llama_index.indices.query.query_transform.base import DecomposeQueryTransform
   decompose_transform = DecomposeQueryTransform(
       service_context.llm_predictor, verbose=True
   )
   response_chatgpt = query_engine_decompose.query("Analyze revenue in Q1 of 2018.")
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Analyze revenue in Q1 of 2018.
   > Starting query: Analyze revenue in Q1 of 2018.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['revenue', '2018', 'q1', 'analyze']
   query keywords: ['revenue', '2018', 'q1', 'analyze']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['2018', 'q1']
   > Extracted keywords: ['2018', 'q1']
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the total revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 13 tokens
   > [retrieve] Total embedding token usage: 13 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the total revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1082 tokens
   > [get_response] Total LLM token usage: 1082 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the total revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 13 tokens
   > [retrieve] Total embedding token usage: 13 tokens

DeepLake + LlamaIndex

   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 974 tokens
   > [get_response] Total LLM token usage: 974 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the total revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 13 tokens
   > [retrieve] Total embedding token usage: 13 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the total revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 852 tokens
   > [get_response] Total LLM token usage: 852 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the total revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 13 tokens
   > [retrieve] Total embedding token usage: 13 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the total revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1179 tokens
   > [get_response] Total LLM token usage: 1179 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens

DeepLake + LlamaIndex

   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 901 tokens
   > [get_response] Total LLM token usage: 901 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 982 tokens
   > [get_response] Total LLM token usage: 982 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 774 tokens
   > [get_response] Total LLM token usage: 774 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q1 of 2018.

DeepLake + LlamaIndex

   > New query: What was the revenue of Amazon in Q1 of 2020?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze revenue in Q1 of 2018.
   > New query: What was the revenue of Amazon in Q1 of 2020?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 950 tokens
   > [get_response] Total LLM token usage: 950 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 218 tokens
   > [get_response] Total LLM token usage: 218 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   print(str(response_chatgpt))
   Based on the given context information, the revenue of Amazon in Q1 of 2018 was $51.0 billion.
   response_chatgpt = query_engine_decompose.query("Analyze revenue in Q2 of 2018.")
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Analyze revenue in Q2 of 2018.
   > Starting query: Analyze revenue in Q2 of 2018.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['2018', 'revenue', 'analyze', 'q2']
   query keywords: ['2018', 'revenue', 'analyze', 'q2']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['2018', 'q2']
   > Extracted keywords: ['2018', 'q2']
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the total revenue of Amazon in Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 13 tokens
   > [retrieve] Total embedding token usage: 13 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the total revenue of Amazon in Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1177 tokens
   > [get_response] Total LLM token usage: 1177 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

DeepLake + LlamaIndex

   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the revenue of Amazon in Q3 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the revenue of Amazon in Q3 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 972 tokens
   > [get_response] Total LLM token usage: 972 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the total revenue of Amazon in Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 13 tokens
   > [retrieve] Total embedding token usage: 13 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the total revenue of Amazon in Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 852 tokens
   > [get_response] Total LLM token usage: 852 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1081 tokens

DeepLake + LlamaIndex

   > [get_response] Total LLM token usage: 1081 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1078 tokens
   > [get_response] Total LLM token usage: 1078 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 898 tokens
   > [get_response] Total LLM token usage: 898 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2021?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze revenue in Q2 of 2018.

DeepLake + LlamaIndex

   > New query: What was the revenue of Amazon in Q2 of 2021?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 818 tokens
   > [get_response] Total LLM token usage: 818 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2020?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze revenue in Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2020?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 901 tokens
   > [get_response] Total LLM token usage: 901 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 210 tokens
   > [get_response] Total LLM token usage: 210 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   print(str(response_chatgpt))
   Based on the given context information, the revenue of Amazon in Q2 of 2018 was $52.9 billion.
   response_chatgpt = query_engine_decompose.query(
       "Analyze and comapre revenue in Q1 and Q2 of 2018."
   )
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > Starting query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['comapre', 'q1', 'revenue', 'analyze', '2018', 'q2']
   query keywords: ['comapre', 'q1', 'revenue', 'analyze', '2018', 'q2']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['q1', '2018', 'q2']
   > Extracted keywords: ['q1', '2018', 'q2']
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens

DeepLake + LlamaIndex

   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1080 tokens
   > [get_response] Total LLM token usage: 1080 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1175 tokens
   > [get_response] Total LLM token usage: 1175 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q1 and Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 15 tokens
   > [retrieve] Total embedding token usage: 15 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q1 and Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 780 tokens
   > [get_response] Total LLM token usage: 780 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

DeepLake + LlamaIndex

   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q1 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 901 tokens
   > [get_response] Total LLM token usage: 901 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q1 and Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 15 tokens
   > [retrieve] Total embedding token usage: 15 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q1 and Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1001 tokens
   > [get_response] Total LLM token usage: 1001 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q1 and Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 15 tokens
   > [retrieve] Total embedding token usage: 15 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.

DeepLake + LlamaIndex

   > New query: What was the revenue of Amazon in Q1 and Q2 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 960 tokens
   > [get_response] Total LLM token usage: 960 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q3 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q3 of 2018?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 972 tokens
   > [get_response] Total LLM token usage: 972 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q4 of 2018?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q1 and Q2 of 2018 according to their financial statement?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 860 tokens
   > [get_response] Total LLM token usage: 860 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2019?

DeepLake + LlamaIndex

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2019?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1078 tokens
   > [get_response] Total LLM token usage: 1078 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2022?
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 12 tokens
   > [retrieve] Total embedding token usage: 12 tokens
   > Current query: Analyze and comapre revenue in Q1 and Q2 of 2018.
   > New query: What was the revenue of Amazon in Q2 of 2022?
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 894 tokens
   > [get_response] Total LLM token usage: 894 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 351 tokens
   > [get_response] Total LLM token usage: 351 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   print(str(response_chatgpt))
   Based on the given context information, we can analyze and compare the revenue in Q1 and Q2 of 2018 for Amazon.
   The revenue of Amazon in Q1 of 2018 was $51.0 billion, while the revenue in Q2 of 2018 was $52.9 billion. Therefore, the revenue in Q2 of 2018 was higher than the revenue in Q1 of 2018. The difference between the two quarters is $1.9 billion.


Defining a Unified Query Interface over your Data

This notebook shows how to build a unified query interface that can
handle:
1. **heterogeneous data sources** (e.g. data about multiple cities)
   and
2. **complex queries** (e.g. compare and contrast).
   import logging
   import sys
   # logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   # logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   # Uncomment if you want to temporarily disable logger
   logger = logging.getLogger()
   logger.disabled = True
   from llama_index import (
       VectorStoreIndex,
       SimpleKeywordTableIndex,
       SimpleDirectoryReader,
       ServiceContext,
   )
   /Users/suo/miniconda3/envs/llama/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
Load Datasets
Load Wikipedia pages about different cities.
   wiki_titles = ["Toronto", "Seattle", "Chicago", "Boston", "Houston"]
   from pathlib import Path
   import requests
   for title in wiki_titles:
       response = requests.get(
           "https://en.wikipedia.org/w/api.php",
           params={
               "action": "query",
               "format": "json",
               "titles": title,
               "prop": "extracts",
               # 'exintro': True,
               "explaintext": True,
           },
       ).json()
       page = next(iter(response["query"]["pages"].values()))
       wiki_text = page["extract"]
       data_path = Path("data")
       if not data_path.exists():
           Path.mkdir(data_path)
       with open(data_path / f"{title}.txt", "w") as fp:
           fp.write(wiki_text)
   # Load all wiki documents
   city_docs = {}
   for wiki_title in wiki_titles:
       city_docs[wiki_title] = SimpleDirectoryReader(
           input_files=[f"data/{wiki_title}.txt"]
       ).load_data()
Building Vector Indices
Build a vector index for the wiki pages about cities.
   from llama_index.llms import OpenAI
   chatgpt = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context = ServiceContext.from_defaults(llm=chatgpt, chunk_size=1024)
   gpt4 = OpenAI(temperature=0, model="gpt-4")
   service_context = ServiceContext.from_defaults(llm=gpt4, chunk_size=1024)
   /Users/suo/miniconda3/envs/llama/lib/python3.9/site-packages/langchain/llms/openai.py:687: UserWarning: You are trying to use a chat model. This way of initializing it is no longer supported. Instead, please use: `from langchain.chat_models import ChatOpenAI`
     warnings.warn(
   # Build city document index
   vector_indices = {}
   for wiki_title in wiki_titles:
       # build vector index
       vector_indices[wiki_title] = VectorStoreIndex.from_documents(
           city_docs[wiki_title], service_context=service_context
       )
       # set id for vector index
       vector_indices[wiki_title].set_index_id(wiki_title)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 20744 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 16942 tokens

   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 26082 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 18648 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 21844 tokens
   index_summaries = {
       wiki_title: (
           f"This content contains Wikipedia articles about {wiki_title}. "
           f"Use this index if you need to lookup specific facts about {wiki_title}.\n"
           "Do not use this index if you want to analyze multiple cities."
       )
       for wiki_title in wiki_titles
   }
Test Querying the Vector Index
   query_engine = vector_indices["Toronto"].as_query_engine()
   response = query_engine.query("What are the sports teams in Toronto?")
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 8 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1904 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   print(str(response))
   The sports teams in Toronto include:
   1. Toronto Maple Leafs (NHL - ice hockey)
   2. Toronto Blue Jays (MLB - baseball)
   3. Toronto Raptors (NBA - basketball)
   4. Toronto Argonauts (CFL - Canadian football)
   5. Toronto FC (MLS - soccer)
   6. Toronto Marlies (AHL - ice hockey)
   7. Toronto Six (NWHL - women's ice hockey)
   8. Toronto Rock (NLL - lacrosse)
   9. Toronto Rush (AUDL - ultimate frisbee)
   10. Toronto Wolfpack (Rugby league, playing in the North American Rugby League tournament)
Build a Graph for Compare/Contrast Queries
We build a graph by composing a keyword table index on top of all the
vector indices. We use this graph for compare/contrast queries
   from llama_index.indices.composability import ComposableGraph
   graph = ComposableGraph.from_indices(
       SimpleKeywordTableIndex,
       [index for _, index in vector_indices.items()],
       [summary for _, summary in index_summaries.items()],
       max_keywords_per_chunk=50,
   )
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens
   # get root index
   root_index = graph.get_index(graph.root_id)
   # set id of root index
   root_index.set_index_id("compare_contrast")
   # define decompose_transform
   from llama_index.indices.query.query_transform.base import DecomposeQueryTransform
   from llama_index import LLMPredictor
   decompose_transform = DecomposeQueryTransform(LLMPredictor(llm=chatgpt), verbose=True)
   # define custom retrievers
   from llama_index.query_engine.transform_query_engine import TransformQueryEngine
   custom_query_engines = {}
   for index in vector_indices.values():

       query_engine = index.as_query_engine(service_context=service_context)
       query_engine = TransformQueryEngine(
           query_engine,
           query_transform=decompose_transform,
           transform_metadata={"index_summary": index.index_struct.summary},
       )
       custom_query_engines[index.index_id] = query_engine
   custom_query_engines[graph.root_id] = graph.root_index.as_query_engine(
       retriever_mode="simple",
       response_mode="tree_summarize",
       service_context=service_context,
       verbose=True,
   )
   # define graph
   graph_query_engine = graph.as_query_engine(custom_query_engines=custom_query_engines)
Test querying the graph
   query_str = "Compare and contrast the arts and culture of Houston and Boston. "
   response = graph_query_engine.query(query_str)
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the arts and culture of Houston and Boston.
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['contrast', 'houston', 'arts', 'boston', 'culture', 'compare']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']
   > Current query: Compare and contrast the arts and culture of Houston and Boston.
   > New query: What are some notable cultural institutions or events in Houston?

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 11 tokens
   > Current query: Compare and contrast the arts and culture of Houston and Boston.
   > New query: What are some notable cultural institutions or events in Houston?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1877 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > Current query: Compare and contrast the arts and culture of Houston and Boston.
   > New query: What are some notable cultural institutions or events in Boston?

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 11 tokens
   > Current query: Compare and contrast the arts and culture of Houston and Boston.
   > New query: What are some notable cultural institutions or events in Boston?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 2130 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 885 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 885 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens

   print(response)
   Houston and Boston both have rich arts and culture scenes, with a variety of cultural institutions and events that cater to diverse interests. Both cities have a strong presence of performing arts organizations, such as the Houston Grand Opera and Houston Ballet in Houston, and the Boston Ballet and Boston Lyric Opera Company in Boston. They also have renowned symphony orchestras, with the Houston Symphony Orchestra and the Boston Symphony Orchestra.
   Both cities host annual events that celebrate their unique cultural identities, such as the Houston Livestock Show and Rodeo, Houston Gay Pride Parade, and Houston Greek Festival in Houston, and the Boston Gay Pride Parade and Festival, Italian Summer Feasts, and Fourth of July events in Boston. Additionally, both cities have thriving theater districts, with Houston's Theater District and Boston's Theater District housing several historic and modern theaters.
   In terms of visual arts, both Houston and Boston have notable art museums, such as the Museum of Fine Arts in both cities, as well as the Houston Museum of Natural Science and the Contemporary Arts Museum Houston in Houston, and the Isabella Stewart Gardner Museum and the Institute of Contemporary Art in Boston. Houston also has unique institutions like the Menil Collection, Rothko Chapel, and the Byzantine Fresco Chapel Museum, while Boston has historic sites related to the American Revolution preserved in the Boston National Historical Park and along the Freedom Trail.
   While both cities have a strong focus on arts and culture, Houston's cultural scene tends to be more diverse, with events like the Art Car Parade, Houston International Festival, and Bayou City Art Festival showcasing the city's eclectic mix of cultures. On the other hand, Boston's cultural scene is deeply rooted in its history and traditions, with events like the Boston Early Music Festival and historic sites along the Freedom Trail reflecting the city's rich past.
Build a router to automatically choose between indices and graph
We can use a "RouterQueryEngine" to automatically route to the vector
indices and the graph.
To do this, first build the query engines, and give each a description
to obtain a "QueryEngineTool".
   from llama_index.tools.query_engine import QueryEngineTool
   query_engine_tools = []
   # add vector index tools
   for wiki_title in wiki_titles:
       index = vector_indices[wiki_title]
       summary = index_summaries[wiki_title]
       query_engine = index.as_query_engine(service_context=service_context)
       vector_tool = QueryEngineTool.from_defaults(query_engine, description=summary)
       query_engine_tools.append(vector_tool)
   # add graph tool
   graph_description = (
       "This tool contains Wikipedia articles about multiple cities. "
       "Use this tool if you want to compare multiple cities. "
   )
   graph_tool = QueryEngineTool.from_defaults(
       graph_query_engine, description=graph_description
   )
   query_engine_tools.append(graph_tool)
Then, define the "RouterQueryEngine" with a desired selector module.
Here, we use the "LLMSingleSelector", which uses LLM to choose a
underlying query engine to route the query to.
   from llama_index.query_engine.router_query_engine import RouterQueryEngine
   from llama_index.selectors.llm_selectors import LLMSingleSelector
   router_query_engine = RouterQueryEngine(
       selector=LLMSingleSelector.from_defaults(service_context=service_context),
       query_engine_tools=query_engine_tools,
   )
Asking a compare and contrast question should route the query to the
graph.
   # ask a compare/contrast question
   response = router_query_engine.query(
       "Compare and contrast the arts and culture of Houston and Boston.",
   )
   INFO:llama_index.query_engine.router_query_engine:Selecting query engine 5: This tool contains Wikipedia articles about multiple cities, which allows for comparison and analysis of different cities, such as Houston and Boston..
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: Compare and contrast the arts and culture of Houston and Boston.

   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['contrast', 'houston', 'arts', 'boston', 'culture', 'compare']
   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['houston', 'boston']
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 11 tokens
   > Current query: Compare and contrast the arts and culture of Houston and Boston.
   > New query: What are some notable cultural institutions or events in Houston?
   > Current query: Compare and contrast the arts and culture of Houston and Boston.
   > New query: What are some notable cultural institutions or events in Houston and Boston?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1835 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > Current query: Compare and contrast the arts and culture of Houston and Boston.
   > New query: What are some notable cultural institutions or events in Boston?

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 11 tokens
   > Current query: Compare and contrast the arts and culture of Houston and Boston.
   > New query: What are some notable cultural institutions or events in Boston?

   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 2134 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 772 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 772 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   print(response)
   Based on the context information provided, both Houston and Boston have rich arts and cultural scenes, with a variety of institutions and events catering to diverse interests.
   Houston's cultural institutions and events include the Houston Theater District, the Museum District, the Houston Livestock Show and Rodeo, the Houston Gay Pride Parade, the Houston Greek Festival, the Art Car Parade, the Houston Auto Show, the Houston International Festival, and the Bayou City Art Festival.
   In contrast, Boston's cultural institutions and events include the Boston Symphony Hall, New England Conservatory's Jordan Hall, Boston Ballet, various performing-arts organizations, contemporary classical music groups, the Theater District, First Night, Boston Early Music Festival, Boston Arts Festival, Boston Gay Pride Parade and Festival, Italian Summer Feasts, Fourth of July events, art museums such as the Museum of Fine Arts and Isabella Stewart Gardner Museum, the Institute of Contemporary Art, art gallery destinations like the South End Art and Design District (SoWa) and Newbury St, and the Boston National Historical Park.

   Both cities have theater districts, gay pride parades, and arts festivals. However, Houston has unique events such as the Livestock Show and Rodeo, the Greek Festival, the Art Car Parade, and the Houston Auto Show. On the other hand, Boston has a strong focus on classical music with venues like the Symphony Hall and Jordan Hall, as well as historical sites related to the American Revolution.
Asking a question about a specific city should route the query to the
specific vector index query engine.
   response = router_query_engine.query("What are the sports teams in Toronto?")
   INFO:llama_index.query_engine.router_query_engine:Selecting query engine 0: This content contains Wikipedia articles about Toronto, which can provide information about the sports teams in the city..
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 8 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1905 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   print(response)
   The sports teams in Toronto include:
   1. Toronto Maple Leafs (NHL - ice hockey)
   2. Toronto Blue Jays (MLB - baseball)
   3. Toronto Raptors (NBA - basketball)
   4. Toronto Argonauts (CFL - Canadian football)
   5. Toronto FC (MLS - soccer)
   6. Toronto Marlies (AHL - ice hockey)
   7. Toronto Six (NWHL - women's ice hockey)
   8. Toronto Rock (NLL - lacrosse)
   9. Toronto Rush (AUDL - ultimate frisbee)
   10. Toronto Wolfpack (Rugby league, currently playing in the North American Rugby League tournament)


Using LlamaIndex with Pinecone

Test complex queries over both text-davinci-003 and ChatGPT
   !pip install llama-index
   # My OpenAI Key
   import os
   os.environ["OPENAI_API_KEY"] = ""
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SimpleKeywordTableIndex,
       SimpleDirectoryReader,
       LLMPredictor,
       ServiceContext,
   )
   from llama_index.vector_stores import PineconeVectorStore
   from llama_index.llms import OpenAI
Load Datasets
Load Wikipedia pages
   wiki_titles = [
       "Toronto",
       "Seattle",
       "San Francisco",
       "Chicago",
       "Boston",
       "Washington, D.C.",
       "Cambridge, Massachusetts",
       "Houston",
   ]
   pinecone_titles = [
       "toronto",
       "seattle",
       "san-francisco",
       "chicago",
       "boston",
       "dc",
       "cambridge",
       "houston",
   ]
   from pathlib import Path
   import requests
   for title in wiki_titles:
       response = requests.get(
           "https://en.wikipedia.org/w/api.php",
           params={
               "action": "query",
               "format": "json",
               "titles": title,
               "prop": "extracts",
               # 'exintro': True,
               "explaintext": True,
           },
       ).json()
       page = next(iter(response["query"]["pages"].values()))
       wiki_text = page["extract"]
       data_path = Path("data")
       if not data_path.exists():
           Path.mkdir(data_path)
       with open(data_path / f"{title}.txt", "w") as fp:
           fp.write(wiki_text)
   # Load all wiki documents
   city_docs = {}
   for wiki_title in wiki_titles:
       city_docs[wiki_title] = SimpleDirectoryReader(
           input_files=[f"data/{wiki_title}.txt"]
       ).load_data()
Initialize Pinecone Indexes
   import pinecone
   import os
   api_key = ""
   environment = "eu-west1-gcp"
   index_name = "quickstart"
   os.environ["PINECONE_API_KEY"] = api_key
   llm = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context = ServiceContext.from_defaults(llm=llm)
Recommended Option:  Pass API key via env variable, and index_name & environment as argument
   # Build city document index
   from llama_index.storage.storage_context import StorageContext
   city_indices = {}
   for pinecone_title, wiki_title in zip(pinecone_titles, wiki_titles):
       metadata_filters = {"wiki_title": wiki_title}
       vector_store = PineconeVectorStore(
           index_name=index_name,
           environment=environment,
           metadata_filters=metadata_filters,
       )
       storage_context = StorageContext.from_defaults(vector_store=vector_store)
       city_indices[wiki_title] = VectorStoreIndex.from_documents(
           city_docs[wiki_title],
           storage_context=storage_context,
           service_context=service_context,
       )
       # set summary text for city
       city_indices[wiki_title].index_struct.index_id = pinecone_title
Alternative Option: instantiate pinecone client first, then pass to PineconeVectorStore
   pinecone.init(api_key=api_key, environment=environment)
   pinecone_index = pinecone.Index(index_name)
   # Build city document index
   city_indices = {}
   for pinecone_title, wiki_title in zip(pinecone_titles, wiki_titles):
       metadata_filters = {"wiki_title": wiki_title}
       vector_store = PineconeVectorStore(
           pinecone_index=pinecone_index, metadata_filters=metadata_filters

Using LlamaIndex with Pinecone

       )
       storage_context = StorageContext.from_defaults(vector_store=vector_store)
       city_indices[wiki_title] = VectorStoreIndex.from_documents(
           city_docs[wiki_title],
           storage_context=storage_context,
           service_context=service_context,
       )
       # set summary text for city
       city_indices[wiki_title].index_struct.index_id = pinecone_title
Query Index
   response = (
       city_indices["Boston"]
       .as_query_engine(service_context=service_context)
       .query("Tell me about the arts and culture of Boston")
   )
   print(str(response))
   print(response.get_formatted_sources())
Build Graph: Keyword Table Index on top of vector indices!
We compose a keyword table index on top of all the vector indices.
   from llama_index.indices.composability.graph import ComposableGraph
   # set summaries for each city
   index_summaries = {}
   for wiki_title in wiki_titles:
       # set summary text for city
       index_summaries[wiki_title] = f"Wikipedia articles about {wiki_title}"
   graph = ComposableGraph.from_indices(
       SimpleKeywordTableIndex,
       [index for _, index in city_indices.items()],
       [summary for _, summary in index_summaries.items()],
       max_keywords_per_chunk=50,
   )
   custom_query_engines = {
       graph.root_id: graph.root_index.as_query_engine(
           retriever_mode="simple", service_context=service_context
       )
   }
   query_engine = graph.as_query_engine(
       custom_query_engines=custom_query_engines,
   )
Compare Queries (text-davinci-003 vs. ChatGPT)
**Simple Query**
   query_str = "Tell me more about Boston"
   response_chatgpt = query_engine.query(query_str)
   print(response_chatgpt)
   print(response_chatgpt.get_formatted_sources())