Knowledge Graph Query Engine

Creating a Knowledge Graph usually involves specialized and complex
tasks. However, by utilizing the Llama Index (LLM), the
KnowledgeGraphIndex, and the GraphStore, we can facilitate the
creation of a relatively effective Knowledge Graph from any data
source supported by Llama Hub.
Furthermore, querying a Knowledge Graph often requires domain-specific
knowledge related to the storage system, such as Cypher. But, with the
assistance of the LLM and the LlamaIndex KnowledgeGraphQueryEngine,
this can be accomplished using Natural Language!
In this demonstration, we will guide you through the steps to:
* Extract and Set Up a Knowledge Graph using the Llama Index
* Query a Knowledge Graph using Cypher
* Query a Knowledge Graph using Natural Language
Let's first get ready for basic preparation of Llama Index.
   # For OpenAI
   import os
   os.environ["OPENAI_API_KEY"] = "sk-..."
   import logging
   import sys
   logging.basicConfig(
       stream=sys.stdout, level=logging.INFO
   )  # logging.DEBUG for more verbose output
   from llama_index import (
       KnowledgeGraphIndex,
       LLMPredictor,
       ServiceContext,
       SimpleDirectoryReader,
   )
   from llama_index.storage.storage_context import StorageContext
   from llama_index.graph_stores import NebulaGraphStore
   from llama_index.llms import OpenAI
   from IPython.display import Markdown, display
   # define LLM
   # NOTE: at the time of demo, text-davinci-002 did not have rate-limit errors
   llm = OpenAI(temperature=0, model="text-davinci-002")
   service_context = ServiceContext.from_defaults(llm=llm, chunk_size_limit=512)
   INFO:numexpr.utils:Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   INFO:numexpr.utils:NumExpr defaulting to 8 threads.
   WARNING:llama_index.indices.service_context:chunk_size_limit is deprecated, please specify chunk_size instead
   # For Azure OpenAI
   import os
   import json
   import openai
   from llama_index.llms import AzureOpenAI
   from llama_index.embeddings import OpenAIEmbedding
   from llama_index import (
       VectorStoreIndex,
       SimpleDirectoryReader,
       KnowledgeGraphIndex,
       LLMPredictor,
       ServiceContext,
   )
   from llama_index.storage.storage_context import StorageContext
   from llama_index.graph_stores import NebulaGraphStore
   from llama_index.llms import LangChainLLM
   import logging
   import sys
   from IPython.display import Markdown, display
   logging.basicConfig(
       stream=sys.stdout, level=logging.INFO
   )  # logging.DEBUG for more verbose output
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   openai.api_type = "azure"
   openai.api_base = "INSERT AZURE API BASE"
   openai.api_version = "2022-12-01"
   os.environ["OPENAI_API_KEY"] = "INSERT OPENAI KEY"
   openai.api_key = os.getenv("OPENAI_API_KEY")
   lc_llm = AzureOpenAI(
       deployment_name="INSERT DEPLOYMENT NAME",
       temperature=0,
       openai_api_version=openai.api_version,
       model_kwargs={
           "api_key": openai.api_key,
           "api_base": openai.api_base,
           "api_type": openai.api_type,
           "api_version": openai.api_version,
       },
   )
   llm = LangChainLLM(lc_llm)
   # You need to deploy your own embedding model as well as your own chat completion model
   embedding_llm = OpenAIEmbedding(
       model="text-embedding-ada-002",

Knowledge Graph Query Engine

       deployment_name="INSERT DEPLOYMENT NAME",
       api_key=openai.api_key,
       api_base=openai.api_base,
       api_type=openai.api_type,
       api_version=openai.api_version,
   )
   service_context = ServiceContext.from_defaults(
       llm=llm,
       embed_model=embedding_llm,
   )
Prepare for NebulaGraph
Before next step to creating the Knowledge Graph, let's ensure we have
a running NebulaGraph with defined data schema.
   # Create a NebulaGraph (version 3.5.0 or newer) cluster with:
   # Option 0 for machines with Docker: `curl -fsSL nebula-up.siwei.io/install.sh | bash`
   # Option 1 for Desktop: NebulaGraph Docker Extension https://hub.docker.com/extensions/weygu/nebulagraph-dd-ext
   # If not, create it with the following commands from NebulaGraph's console:
   # CREATE SPACE llamaindex(vid_type=FIXED_STRING(256), partition_num=1, replica_factor=1);
   # :sleep 10;
   # USE llamaindex;
   # CREATE TAG entity(name string);
   # CREATE EDGE relationship(relationship string);
   # :sleep 10;
   # CREATE TAG INDEX entity_index ON entity(name(256));
   %pip install ipython-ngql nebula3-python
   os.environ["NEBULA_USER"] = "root"
   os.environ["NEBULA_PASSWORD"] = "nebula"  # default is "nebula"
   os.environ[
       "NEBULA_ADDRESS"
   ] = "127.0.0.1:9669"  # assumed we have NebulaGraph installed locally
   space_name = "llamaindex"
   edge_types, rel_prop_names = ["relationship"], [
       "relationship"
   ]  # default, could be omit if create from an empty kg
   tags = ["entity"]  # default, could be omit if create from an empty kg
   Requirement already satisfied: ipython-ngql in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (0.5)
   Requirement already satisfied: nebula3-python in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (3.4.0)
   Requirement already satisfied: pandas in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython-ngql) (2.0.3)
   Requirement already satisfied: Jinja2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython-ngql) (3.1.2)
   Requirement already satisfied: pytz>=2021.1 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python) (2023.3)
   Requirement already satisfied: future>=0.18.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python) (0.18.3)
   Requirement already satisfied: httplib2>=0.20.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python) (0.22.0)
   Requirement already satisfied: six>=1.16.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python) (1.16.0)
   Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from httplib2>=0.20.0->nebula3-python) (3.0.9)

Knowledge Graph Query Engine

   Requirement already satisfied: MarkupSafe>=2.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from Jinja2->ipython-ngql) (2.1.3)
   Requirement already satisfied: tzdata>=2022.1 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pandas->ipython-ngql) (2023.3)
   Requirement already satisfied: numpy>=1.20.3 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pandas->ipython-ngql) (1.25.2)
   Requirement already satisfied: python-dateutil>=2.8.2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pandas->ipython-ngql) (2.8.2)
   WARNING: You are using pip version 21.2.4; however, version 23.2.1 is available.
   You should consider upgrading via the '/Users/loganmarkewich/llama_index/llama-index/bin/python -m pip install --upgrade pip' command.
   Note: you may need to restart the kernel to use updated packages.
Prepare for StorageContext with graph_store as NebulaGraphStore
   graph_store = NebulaGraphStore(
       space_name=space_name,
       edge_types=edge_types,
       rel_prop_names=rel_prop_names,
       tags=tags,
   )
   storage_context = StorageContext.from_defaults(graph_store=graph_store)
(Optional)Build the Knowledge Graph with LlamaIndex
With the help of Llama Index and LLM defined, we could build Knowledge
Graph from given documents.
If we have a Knowledge Graph on NebulaGraphStore already, this step
could be skipped
Step 1, load data from Wikipedia for "Guardians of the Galaxy Vol. 3"
   from llama_index import download_loader
   WikipediaReader = download_loader("WikipediaReader")
   loader = WikipediaReader()
   documents = loader.load_data(
       pages=["Guardians of the Galaxy Vol. 3"], auto_suggest=False
   )
Step 2, Generate a KnowledgeGraphIndex with NebulaGraph as graph_store
Then, we will create a KnowledgeGraphIndex to enable Graph based RAG,
see here for deails, apart from that, we have a Knowledge Graph up and
running for other purposes, too!
   kg_index = KnowledgeGraphIndex.from_documents(
       documents,
       storage_context=storage_context,
       max_triplets_per_chunk=10,
       service_context=service_context,
       space_name=space_name,
       edge_types=edge_types,
       rel_prop_names=rel_prop_names,
       tags=tags,
       include_embeddings=True,
   )
Now we have a Knowledge Graph on NebulaGraph cluster under space named
"llamaindex" about the 'Guardians of the Galaxy Vol. 3' movie, let's
play with it a little bit.
   # install related packages, password is nebula by default
   %pip install ipython-ngql networkx pyvis
   %load_ext ngql
   %ngql --address 127.0.0.1 --port 9669 --user root --password <password>
   Requirement already satisfied: ipython-ngql in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (0.5)
   Requirement already satisfied: networkx in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (3.1)
   Requirement already satisfied: pyvis in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (0.3.2)

Knowledge Graph Query Engine

   Requirement already satisfied: Jinja2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython-ngql) (3.1.2)
   Requirement already satisfied: pandas in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython-ngql) (2.0.3)
   Requirement already satisfied: nebula3-python in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython-ngql) (3.4.0)
   Requirement already satisfied: jsonpickle>=1.4.1 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pyvis) (3.0.1)
   Requirement already satisfied: ipython>=5.3.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pyvis) (8.10.0)
   Requirement already satisfied: backcall in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (0.2.0)
   Requirement already satisfied: pickleshare in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (0.7.5)
   Requirement already satisfied: prompt-toolkit<3.1.0,>=3.0.30 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (3.0.39)
   Requirement already satisfied: appnope in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (0.1.3)
   Requirement already satisfied: pygments>=2.4.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (2.15.1)
   Requirement already satisfied: traitlets>=5 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (5.9.0)
   Requirement already satisfied: pexpect>4.3 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (4.8.0)
   Requirement already satisfied: stack-data in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (0.6.2)
   Requirement already satisfied: decorator in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (5.1.1)
   Requirement already satisfied: jedi>=0.16 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (0.18.2)
   Requirement already satisfied: matplotlib-inline in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from ipython>=5.3.0->pyvis) (0.1.6)

Knowledge Graph Query Engine

   Requirement already satisfied: parso<0.9.0,>=0.8.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from jedi>=0.16->ipython>=5.3.0->pyvis) (0.8.3)
   Requirement already satisfied: MarkupSafe>=2.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from Jinja2->ipython-ngql) (2.1.3)
   Requirement already satisfied: ptyprocess>=0.5 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pexpect>4.3->ipython>=5.3.0->pyvis) (0.7.0)
   Requirement already satisfied: wcwidth in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from prompt-toolkit<3.1.0,>=3.0.30->ipython>=5.3.0->pyvis) (0.2.6)
   Requirement already satisfied: six>=1.16.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python->ipython-ngql) (1.16.0)
   Requirement already satisfied: pytz>=2021.1 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python->ipython-ngql) (2023.3)
   Requirement already satisfied: future>=0.18.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python->ipython-ngql) (0.18.3)
   Requirement already satisfied: httplib2>=0.20.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from nebula3-python->ipython-ngql) (0.22.0)
   Requirement already satisfied: pyparsing!=3.0.0,!=3.0.1,!=3.0.2,!=3.0.3,<4,>=2.4.2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from httplib2>=0.20.0->nebula3-python->ipython-ngql) (3.0.9)
   Requirement already satisfied: python-dateutil>=2.8.2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pandas->ipython-ngql) (2.8.2)
   Requirement already satisfied: numpy>=1.20.3 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pandas->ipython-ngql) (1.25.2)
   Requirement already satisfied: tzdata>=2022.1 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from pandas->ipython-ngql) (2023.3)
   Requirement already satisfied: executing>=1.2.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from stack-data->ipython>=5.3.0->pyvis) (1.2.0)
   Requirement already satisfied: pure-eval in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from stack-data->ipython>=5.3.0->pyvis) (0.2.2)

Knowledge Graph Query Engine

   Requirement already satisfied: asttokens>=2.1.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from stack-data->ipython>=5.3.0->pyvis) (2.2.1)
   WARNING: You are using pip version 21.2.4; however, version 23.2.1 is available.
   You should consider upgrading via the '/Users/loganmarkewich/llama_index/llama-index/bin/python -m pip install --upgrade pip' command.
   Note: you may need to restart the kernel to use updated packages.
   Connection Pool Created
   INFO:nebula3.logger:Get connection to ('127.0.0.1', 9669)
   [ERROR]:
    'IPythonNGQL' object has no attribute '_decode_value'
            Name
   0  llamaindex
   # Query some random Relationships with Cypher
   %ngql USE llamaindex;
   %ngql MATCH ()-[e]->() RETURN e LIMIT 10
   INFO:nebula3.logger:Get connection to ('127.0.0.1', 9669)
   INFO:nebula3.logger:Get connection to ('127.0.0.1', 9669)
                                                      e
   0  ("A second trailer for the film")-[:relationsh...
   1  ("Adam McKay")-[:relationship@-442854342936029...
   2  ("Adam McKay")-[:relationship@8513344855738553...
   3  ("Asim Chaudhry")-[:relationship@-803614038978...
   4  ("Bakalova")-[:relationship@-25325064520311626...
   5  ("Bautista")-[:relationship@-90386029986457371...
   6  ("Bautista")-[:relationship@-90386029986457371...
   7  ("Beth Mickle")-[:relationship@716197657641767...
   8  ("Bradley Cooper")-[:relationship@138630731832...
   9  ("Bradley Cooper")-[:relationship@838402633192...
   # draw the result
   %ng_draw
   nebulagraph_draw.html
   <IPython.lib.display.IFrame at 0x2b43757f0>
Asking the Knowledge Graph
Finally, let's demo how to Query Knowledge Graph with Natural
language!
Here, we will leverage the "KnowledgeGraphQueryEngine", with
"NebulaGraphStore" as the "storage_context.graph_store".
   from llama_index.query_engine import KnowledgeGraphQueryEngine
   from llama_index.storage.storage_context import StorageContext
   from llama_index.graph_stores import NebulaGraphStore
   query_engine = KnowledgeGraphQueryEngine(
       storage_context=storage_context,
       service_context=service_context,
       llm=llm,
       verbose=True,
   )
   response = query_engine.query(
       "Tell me about Peter Quill?",
   )
   display(Markdown(f"<b>{response}</b>"))
   Graph Store Query:
   ```
   MATCH (p:`entity`)-[:relationship]->(m:`entity`) WHERE p.`entity`.`name` == 'Peter Quill'
   RETURN p.`entity`.`name`;
   ```
   Graph Store Response:
   {'p.entity.name': ['Peter Quill', 'Peter Quill', 'Peter Quill', 'Peter Quill', 'Peter Quill']}
   Final Response:

Knowledge Graph Query Engine

   Peter Quill is a character in the Marvel Universe. He is the son of Meredith Quill and Ego the Living Planet.

Peter Quill is a character in the Marvel Universe. He is the son of
Meredith Quill and Ego the Living Planet.
   graph_query = query_engine.generate_query(
       "Tell me about Peter Quill?",
   )
   graph_query = graph_query.replace("WHERE", "\n  WHERE").replace("RETURN", "\nRETURN")
   display(
       Markdown(
           f"""
   ```cypher
   {graph_query}
   ```
   """
       )
   )
MATCH (p:"entity")-[:relationship]->(m:"entity") WHERE
p."entity"."name" == 'Peter Quill'
RETURN p."entity"."name";
We could see it helps generate the Graph query:
   MATCH (p:`entity`)-[:relationship]->(e:`entity`)
     WHERE p.`entity`.`name` == 'Peter Quill'
   RETURN e.`entity`.`name`;
And synthese the question based on its result:
   {'e2.entity.name': ['grandfather', 'alternate version of Gamora', 'Guardians of the Galaxy']}
Of course we still could query it, too! And this query engine could be
our best Graph Query Language learning bot, then :).
   %%ngql
   MATCH (p:`entity`)-[e:relationship]->(m:`entity`)
     WHERE p.`entity`.`name` == 'Peter Quill'
   RETURN p.`entity`.`name`, e.relationship, m.`entity`.`name`;
   INFO:nebula3.logger:Get connection to ('127.0.0.1', 9669)
     p.entity.name           e.relationship  \
   0   Peter Quill  would return to the MCU
   1   Peter Quill  was abducted from Earth
   2   Peter Quill             is leader of
   3   Peter Quill            was raised by
   4   Peter Quill            is half-human
                               m.entity.name
   0                                May 2021
   1                              as a child
   2                 Guardians of the Galaxy
   3  a group of alien thieves and smugglers
   4                          half-Celestial
And change the query to be rendered
   %%ngql
   MATCH (p:`entity`)-[e:relationship]->(m:`entity`)
     WHERE p.`entity`.`name` == 'Peter Quill'
   RETURN p, e, m;
   INFO:nebula3.logger:Get connection to ('127.0.0.1', 9669)
                                                 p  \
   0  ("Peter Quill" :entity{name: "Peter Quill"})
   1  ("Peter Quill" :entity{name: "Peter Quill"})
   2  ("Peter Quill" :entity{name: "Peter Quill"})
   3  ("Peter Quill" :entity{name: "Peter Quill"})
   4  ("Peter Quill" :entity{name: "Peter Quill"})
                                                      e  \
   0  ("Peter Quill")-[:relationship@-84437522554765...
   1  ("Peter Quill")-[:relationship@-11770408155938...
   2  ("Peter Quill")-[:relationship@-79394488349732...
   3  ("Peter Quill")-[:relationship@325695233021653...
   4  ("Peter Quill")-[:relationship@555553046209276...
                                                      m
   0             ("May 2021" :entity{name: "May 2021"})
   1         ("as a child" :entity{name: "as a child"})

Knowledge Graph Query Engine

   2  ("Guardians of the Galaxy" :entity{name: "Guar...
   3  ("a group of alien thieves and smugglers" :ent...
   4  ("half-Celestial" :entity{name: "half-Celestia...
   %ng_draw
   nebulagraph_draw.html
   <IPython.lib.display.IFrame at 0x2ae69fdf0>
The results of this knowledge-fetching query could not be more clear
from the renderred graph then.


SQL Auto Vector Query Engine

In this tutorial, we show you how to use our SQLAutoVectorQueryEngine.
This query engine allows you to combine insights from your structured
tables with your unstructured data. It first decides whether to query
your structured tables for insights. Once it does, it can then infer a
corresponding query to the vector store in order to fetch
corresponding documents.
   import openai
   import os
   os.environ["OPENAI_API_KEY"] = "[You API key]"
   openai.api_key = os.environ["OPENAI_API_KEY"]
Setup
   # NOTE: This is ONLY necessary in jupyter notebook.
   # Details: Jupyter runs an event-loop behind the scenes.
   #          This results in nested event-loops when we start an event-loop to make async queries.
   #          This is normally not allowed, we use nest_asyncio to allow it for convenience.
   import nest_asyncio
   nest_asyncio.apply()
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SimpleDirectoryReader,
       ServiceContext,
       StorageContext,
       SQLDatabase,
       WikipediaReader,
   )
   INFO:numexpr.utils:Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   INFO:numexpr.utils:NumExpr defaulting to 8 threads.
   NumExpr defaulting to 8 threads.
Create Common Objects
This includes a "ServiceContext" object containing abstractions such
as the LLM and chunk size. This also includes a "StorageContext"
object containing our vector store abstractions.
   # define pinecone index
   import pinecone
   import os
   api_key = os.environ["PINECONE_API_KEY"]
   pinecone.init(api_key=api_key, environment="us-west1-gcp-free")
   # dimensions are for text-embedding-ada-002
   # pinecone.create_index("quickstart", dimension=1536, metric="euclidean", pod_type="p1")
   pinecone_index = pinecone.Index("quickstart")
   /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages/pinecone/index.py:4: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from tqdm.autonotebook import tqdm
   # OPTIONAL: delete all
   pinecone_index.delete(deleteAll=True)
   {}
   from llama_index.node_parser.simple import SimpleNodeParser
   from llama_index import ServiceContext, LLMPredictor
   from llama_index.storage import StorageContext
   from llama_index.vector_stores import PineconeVectorStore
   from llama_index.text_splitter import TokenTextSplitter
   from llama_index.llms import OpenAI
   # define node parser and LLM
   chunk_size = 1024
   llm = OpenAI(temperature=0, model="gpt-4", streaming=True)
   service_context = ServiceContext.from_defaults(chunk_size=chunk_size, llm=llm)
   text_splitter = TokenTextSplitter(chunk_size=chunk_size)
   node_parser = SimpleNodeParser.from_defaults(text_splitter=text_splitter)
   # define pinecone vector index
   vector_store = PineconeVectorStore(
       pinecone_index=pinecone_index, namespace="wiki_cities"
   )
   storage_context = StorageContext.from_defaults(vector_store=vector_store)
   vector_index = VectorStoreIndex([], storage_context=storage_context)

SQL Auto Vector Query Engine

Create Database Schema + Test Data
Here we introduce a toy scenario where there are 100 tables (too big
to fit into the prompt)
   from sqlalchemy import (
       create_engine,
       MetaData,
       Table,
       Column,
       String,
       Integer,
       select,
       column,
   )
   engine = create_engine("sqlite:///:memory:", future=True)
   metadata_obj = MetaData()
   # create city SQL table
   table_name = "city_stats"
   city_stats_table = Table(
       table_name,
       metadata_obj,
       Column("city_name", String(16), primary_key=True),
       Column("population", Integer),
       Column("country", String(16), nullable=False),
   )
   metadata_obj.create_all(engine)
   # print tables
   metadata_obj.tables.keys()
   dict_keys(['city_stats'])
We introduce some test data into the "city_stats" table
   from sqlalchemy import insert
   rows = [
       {"city_name": "Toronto", "population": 2930000, "country": "Canada"},
       {"city_name": "Tokyo", "population": 13960000, "country": "Japan"},
       {"city_name": "Berlin", "population": 3645000, "country": "Germany"},
   ]
   for row in rows:
       stmt = insert(city_stats_table).values(**row)
       with engine.begin() as connection:
           cursor = connection.execute(stmt)
   with engine.connect() as connection:
       cursor = connection.exec_driver_sql("SELECT * FROM city_stats")
       print(cursor.fetchall())
   [('Toronto', 2930000, 'Canada'), ('Tokyo', 13960000, 'Japan'), ('Berlin', 3645000, 'Germany')]
Load Data
We first show how to convert a Document into a set of Nodes, and
insert into a DocumentStore.
   # install wikipedia python package
   !pip install wikipedia
   Requirement already satisfied: wikipedia in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (1.4.0)
   Requirement already satisfied: beautifulsoup4 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from wikipedia) (4.12.2)
   Requirement already satisfied: requests<3.0.0,>=2.0.0 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from wikipedia) (2.31.0)
   Requirement already satisfied: idna<4,>=2.5 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (3.4)
   Requirement already satisfied: charset-normalizer<4,>=2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (3.2.0)
   Requirement already satisfied: certifi>=2017.4.17 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (2023.5.7)
   Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from requests<3.0.0,>=2.0.0->wikipedia) (1.26.16)
   Requirement already satisfied: soupsieve>1.2 in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from beautifulsoup4->wikipedia) (2.4.1)

SQL Auto Vector Query Engine

   WARNING: You are using pip version 21.2.4; however, version 23.2 is available.
   You should consider upgrading via the '/Users/loganmarkewich/llama_index/llama-index/bin/python3 -m pip install --upgrade pip' command.
   cities = ["Toronto", "Berlin", "Tokyo"]
   wiki_docs = WikipediaReader().load_data(pages=cities)
Build SQL Index
   sql_database = SQLDatabase(engine, include_tables=["city_stats"])
   from llama_index.indices.struct_store.sql_query import NLSQLTableQueryEngine
   sql_query_engine = NLSQLTableQueryEngine(
       sql_database=sql_database,
       tables=["city_stats"],
   )
Build Vector Index
   # Insert documents into vector index
   # Each document has metadata of the city attached
   for city, wiki_doc in zip(cities, wiki_docs):
       nodes = node_parser.get_nodes_from_documents([wiki_doc])
       # add metadata to each node
       for node in nodes:
           node.metadata = {"title": city}
       vector_index.insert_nodes(nodes)
   Upserted vectors: 100%|██████████| 20/20 [00:00<00:00, 22.37it/s]
   Upserted vectors: 100%|██████████| 22/22 [00:00<00:00, 23.14it/s]
   Upserted vectors: 100%|██████████| 13/13 [00:00<00:00, 17.67it/s]
Define Query Engines, Set as Tools
   from llama_index.query_engine import SQLAutoVectorQueryEngine, RetrieverQueryEngine
   from llama_index.tools.query_engine import QueryEngineTool
   from llama_index.indices.vector_store import VectorIndexAutoRetriever
   from llama_index.indices.vector_store.retrievers import VectorIndexAutoRetriever
   from llama_index.vector_stores.types import MetadataInfo, VectorStoreInfo
   from llama_index.query_engine.retriever_query_engine import RetrieverQueryEngine
   vector_store_info = VectorStoreInfo(
       content_info="articles about different cities",
       metadata_info=[
           MetadataInfo(name="title", type="str", description="The name of the city"),
       ],
   )
   vector_auto_retriever = VectorIndexAutoRetriever(
       vector_index, vector_store_info=vector_store_info
   )
   retriever_query_engine = RetrieverQueryEngine.from_args(
       vector_auto_retriever, service_context=service_context
   )
   sql_tool = QueryEngineTool.from_defaults(
       query_engine=sql_query_engine,
       description=(
           "Useful for translating a natural language query into a SQL query over a table containing: "
           "city_stats, containing the population/country of each city"
       ),
   )
   vector_tool = QueryEngineTool.from_defaults(
       query_engine=retriever_query_engine,
       description=f"Useful for answering semantic questions about different cities",
   )
Define SQLAutoVectorQueryEngine
   query_engine = SQLAutoVectorQueryEngine(
       sql_tool, vector_tool, service_context=service_context
   )
   response = query_engine.query(
       "Tell me about the arts and culture of the city with the highest population"
   )
   Querying SQL database: Useful for translating a natural language query into a SQL query over a table containing city_stats, containing the population/country of each city
   INFO:llama_index.query_engine.sql_join_query_engine:> Querying SQL database: Useful for translating a natural language query into a SQL query over a table containing city_stats, containing the population/country of each city
   > Querying SQL database: Useful for translating a natural language query into a SQL query over a table containing city_stats, containing the population/country of each city

SQL Auto Vector Query Engine

   INFO:llama_index.indices.struct_store.sql_query:> Table desc str: Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)) and foreign keys: .
   > Table desc str: Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)) and foreign keys: .
   SQL query: SELECT city_name, population FROM city_stats ORDER BY population DESC LIMIT 1;
   SQL response:
   Tokyo is the city with the highest population, with 13.96 million people. It is a vibrant city with a rich culture and a wide variety of art forms. From traditional Japanese art such as calligraphy and woodblock prints to modern art galleries and museums, Tokyo has something for everyone. There are also many festivals and events throughout the year that celebrate the city's culture and art.
   Transformed query given SQL response: What are some specific cultural festivals, events, and notable art galleries or museums in Tokyo?
   INFO:llama_index.query_engine.sql_join_query_engine:> Transformed query given SQL response: What are some specific cultural festivals, events, and notable art galleries or museums in Tokyo?
   > Transformed query given SQL response: What are some specific cultural festivals, events, and notable art galleries or museums in Tokyo?
   INFO:llama_index.indices.vector_store.retrievers.auto_retriever.auto_retriever:Using query str: cultural festivals events art galleries museums Tokyo
   Using query str: cultural festivals events art galleries museums Tokyo
   INFO:llama_index.indices.vector_store.retrievers.auto_retriever.auto_retriever:Using filters: {'title': 'Tokyo'}
   Using filters: {'title': 'Tokyo'}
   INFO:llama_index.indices.vector_store.retrievers.auto_retriever.auto_retriever:Using top_k: 2
   Using top_k: 2
   query engine response: The context information mentions the Tokyo National Museum, which houses 37% of the country's artwork national treasures. It also mentions the Studio Ghibli anime center as a subcultural attraction. However, the text does not provide information on specific cultural festivals or events in Tokyo.
   INFO:llama_index.query_engine.sql_join_query_engine:> query engine response: The context information mentions the Tokyo National Museum, which houses 37% of the country's artwork national treasures. It also mentions the Studio Ghibli anime center as a subcultural attraction. However, the text does not provide information on specific cultural festivals or events in Tokyo.
   > query engine response: The context information mentions the Tokyo National Museum, which houses 37% of the country's artwork national treasures. It also mentions the Studio Ghibli anime center as a subcultural attraction. However, the text does not provide information on specific cultural festivals or events in Tokyo.
   Final response: Tokyo, the city with the highest population of 13.96 million people, is known for its vibrant culture and diverse art forms. It is home to traditional Japanese art such as calligraphy and woodblock prints, as well as modern art galleries and museums. Notably, the Tokyo National Museum houses 37% of the country's artwork national treasures, and the Studio Ghibli anime center is a popular subcultural attraction. While there are many festivals and events throughout the year that celebrate the city's culture and art, specific examples were not provided in the available information.

   print(str(response))
   Tokyo, the city with the highest population of 13.96 million people, is known for its vibrant culture and diverse art forms. It is home to traditional Japanese art such as calligraphy and woodblock prints, as well as modern art galleries and museums. Notably, the Tokyo National Museum houses 37% of the country's artwork national treasures, and the Studio Ghibli anime center is a popular subcultural attraction. While there are many festivals and events throughout the year that celebrate the city's culture and art, specific examples were not provided in the available information.

SQL Auto Vector Query Engine

   response = query_engine.query("Tell me about the history of Berlin")
   Querying other query engine: Useful for answering semantic questions about different cities
   INFO:llama_index.query_engine.sql_join_query_engine:> Querying other query engine: Useful for answering semantic questions about different cities
   > Querying other query engine: Useful for answering semantic questions about different cities
   INFO:llama_index.indices.vector_store.retrievers.auto_retriever.auto_retriever:Using query str: history of Berlin
   Using query str: history of Berlin
   INFO:llama_index.indices.vector_store.retrievers.auto_retriever.auto_retriever:Using filters: {'title': 'Berlin'}
   Using filters: {'title': 'Berlin'}
   INFO:llama_index.indices.vector_store.retrievers.auto_retriever.auto_retriever:Using top_k: 2
   Using top_k: 2
   Query Engine response: Berlin's history dates back to around 60,000 BC, with the earliest human traces found in the area. A Mesolithic deer antler mask found in Biesdorf (Berlin) was dated around 9000 BC. During Neolithic times, a large number of communities existed in the area and in the Bronze Age, up to 1000 people lived in 50 villages. Early Germanic tribes took settlement from 500 BC and Slavic settlements and castles began around 750 AD.
   The earliest evidence of middle age settlements in the area of today's Berlin are remnants of a house foundation dated to 1174, found in excavations in Berlin Mitte, and a wooden beam dated from approximately 1192. The first written records of towns in the area of present-day Berlin date from the late 12th century. Spandau is first mentioned in 1197 and Köpenick in 1209, although these areas did not join Berlin until 1920.
   The central part of Berlin can be traced back to two towns. Cölln on the Fischerinsel is first mentioned in a 1237 document, and Berlin, across the Spree in what is now called the Nikolaiviertel, is referenced in a document from 1244. 1237 is considered the founding date of the city. The two towns over time formed close economic and social ties, and profited from the staple right on the two important trade routes Via Imperii and from Bruges to Novgorod. In 1307, they formed an alliance with a common external policy, their internal administrations still being separated. In 1415, Frederick I became the elector of the Margraviate of Brandenburg, which he ruled until 1440.
   The name Berlin has its roots in the language of West Slavic inhabitants of the area of today's Berlin, and may be related to the Old Polabian stem berl-/birl- ("swamp"). or Proto-Slavic bьrlogъ, (lair, den). Since the Ber- at the beginning sounds like the German word Bär ("bear"), a bear appears in the coat of arms of the city. It is therefore an example of canting arms.

   print(str(response))
   Berlin's history dates back to around 60,000 BC, with the earliest human traces found in the area. A Mesolithic deer antler mask found in Biesdorf (Berlin) was dated around 9000 BC. During Neolithic times, a large number of communities existed in the area and in the Bronze Age, up to 1000 people lived in 50 villages. Early Germanic tribes took settlement from 500 BC and Slavic settlements and castles began around 750 AD.
   The earliest evidence of middle age settlements in the area of today's Berlin are remnants of a house foundation dated to 1174, found in excavations in Berlin Mitte, and a wooden beam dated from approximately 1192. The first written records of towns in the area of present-day Berlin date from the late 12th century. Spandau is first mentioned in 1197 and Köpenick in 1209, although these areas did not join Berlin until 1920.

SQL Auto Vector Query Engine

   The central part of Berlin can be traced back to two towns. Cölln on the Fischerinsel is first mentioned in a 1237 document, and Berlin, across the Spree in what is now called the Nikolaiviertel, is referenced in a document from 1244. 1237 is considered the founding date of the city. The two towns over time formed close economic and social ties, and profited from the staple right on the two important trade routes Via Imperii and from Bruges to Novgorod. In 1307, they formed an alliance with a common external policy, their internal administrations still being separated. In 1415, Frederick I became the elector of the Margraviate of Brandenburg, which he ruled until 1440.
   The name Berlin has its roots in the language of West Slavic inhabitants of the area of today's Berlin, and may be related to the Old Polabian stem berl-/birl- ("swamp"). or Proto-Slavic bьrlogъ, (lair, den). Since the Ber- at the beginning sounds like the German word Bär ("bear"), a bear appears in the coat of arms of the city. It is therefore an example of canting arms.
   response = query_engine.query("Can you give me the country corresponding to each city?")
   Querying SQL database: Useful for translating a natural language query into a SQL query over a table containing: city_stats, containing the population/country of each city
   INFO:llama_index.query_engine.sql_join_query_engine:> Querying SQL database: Useful for translating a natural language query into a SQL query over a table containing: city_stats, containing the population/country of each city
   > Querying SQL database: Useful for translating a natural language query into a SQL query over a table containing: city_stats, containing the population/country of each city
   INFO:llama_index.indices.struct_store.sql_query:> Table desc str: Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)) and foreign keys: .
   > Table desc str: Table 'city_stats' has columns: city_name (VARCHAR(16)), population (INTEGER), country (VARCHAR(16)) and foreign keys: .
   SQL query: SELECT city_name, country FROM city_stats;
   SQL response:  Toronto is in Canada, Tokyo is in Japan, and Berlin is in Germany.
   Transformed query given SQL response: What countries are New York, San Francisco, and other cities in?
   INFO:llama_index.query_engine.sql_join_query_engine:> Transformed query given SQL response: What countries are New York, San Francisco, and other cities in?
   > Transformed query given SQL response: What countries are New York, San Francisco, and other cities in?
   INFO:llama_index.indices.vector_store.retrievers.auto_retriever.auto_retriever:Using query str: New York San Francisco
   Using query str: New York San Francisco
   INFO:llama_index.indices.vector_store.retrievers.auto_retriever.auto_retriever:Using filters: {'title': 'San Francisco'}
   Using filters: {'title': 'San Francisco'}
   INFO:llama_index.indices.vector_store.retrievers.auto_retriever.auto_retriever:Using top_k: 2
   Using top_k: 2
   query engine response: None
   INFO:llama_index.query_engine.sql_join_query_engine:> query engine response: None

SQL Auto Vector Query Engine

   > query engine response: None
   Final response: The country corresponding to each city is as follows: Toronto is in Canada, Tokyo is in Japan, and Berlin is in Germany. Unfortunately, I do not have information on the countries for New York, San Francisco, and other cities.

   print(str(response))
   The country corresponding to each city is as follows: Toronto is in Canada, Tokyo is in Japan, and Berlin is in Germany. Unfortunately, I do not have information on the countries for New York, San Francisco, and other cities.


JSON Query Engine

The JSON query engine is useful for querying JSON documents that
conform to a JSON schema.
This JSON schema is then used in the context of a prompt to convert a
natural language query into a structured JSON Path query. This JSON
Path query is then used to retrieve data to answer the given question.
   # First, install the jsonpath-ng package which is used by default to parse & execute the JSONPath queries.
   !pip install jsonpath-ng
   Requirement already satisfied: jsonpath-ng in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (1.5.3)
   Requirement already satisfied: ply in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from jsonpath-ng) (3.11)
   Requirement already satisfied: six in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from jsonpath-ng) (1.16.0)
   Requirement already satisfied: decorator in /Users/loganmarkewich/llama_index/llama-index/lib/python3.9/site-packages (from jsonpath-ng) (5.1.1)
   WARNING: You are using pip version 21.2.4; however, version 23.2.1 is available.
   You should consider upgrading via the '/Users/loganmarkewich/llama_index/llama-index/bin/python3 -m pip install --upgrade pip' command.
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   import os
   import openai
   os.environ["OPENAI_API_KEY"] = "YOUR_KEY_HERE"
   openai.api_key = os.environ["OPENAI_API_KEY"]
   from IPython.display import Markdown, display
Let's start on a Toy JSON
Very simple JSON object containing data from a blog post site with
user comments.
We will also provide a JSON schema (which we were able to generate by
giving ChatGPT a sample of the JSON).
Advice
Do make sure that you've provided a helpful ""description"" value for
each of the fields in your JSON schema.
As you can see in the given example, the description for the
""username"" field mentions that usernames are lowercased. You'll see
that this ends up being helpful for the LLM in producing the correct
JSON path query.
   # Test on some sample data
   json_value = {
       "blogPosts": [
           {"id": 1, "title": "First blog post", "content": "This is my first blog post"},
           {
               "id": 2,
               "title": "Second blog post",
               "content": "This is my second blog post",
           },
       ],
       "comments": [
           {"id": 1, "content": "Nice post!", "username": "jerry", "blogPostId": 1},
           {
               "id": 2,
               "content": "Interesting thoughts",
               "username": "simon",
               "blogPostId": 2,
           },
           {
               "id": 3,
               "content": "Loved reading this!",
               "username": "simon",
               "blogPostId": 2,
           },
       ],
   }
   # JSON Schema object that the above JSON value conforms to
   json_schema = {
       "$schema": "http://json-schema.org/draft-07/schema#",
       "description": "Schema for a very simple blog post app",
       "type": "object",
       "properties": {
           "blogPosts": {
               "description": "List of blog posts",
               "type": "array",

JSON Query Engine

"items": {
                   "type": "object",
                   "properties": {
                       "id": {
                           "description": "Unique identifier for the blog post",
                           "type": "integer",
                       },
                       "title": {
                           "description": "Title of the blog post",
                           "type": "string",
                       },
                       "content": {
                           "description": "Content of the blog post",
                           "type": "string",
                       },
                   },
                   "required": ["id", "title", "content"],
               },
           },
           "comments": {
               "description": "List of comments on blog posts",
               "type": "array",
               "items": {
                   "type": "object",
                   "properties": {
                       "id": {
                           "description": "Unique identifier for the comment",
                           "type": "integer",
                       },
                       "content": {
                           "description": "Content of the comment",
                           "type": "string",
                       },
                       "username": {
                           "description": "Username of the commenter (lowercased)",
                           "type": "string",
                       },
                       "blogPostId": {
                           "description": "Identifier for the blog post to which the comment belongs",
                           "type": "integer",
                       },
                   },
                   "required": ["id", "content", "username", "blogPostId"],
               },
           },
       },
       "required": ["blogPosts", "comments"],
   }
   from llama_index.indices.service_context import ServiceContext
   from llama_index.llms import OpenAI
   from llama_index.indices.struct_store import JSONQueryEngine
   llm = OpenAI(model="gpt-4")
   service_context = ServiceContext.from_defaults(llm=llm)
   nl_query_engine = JSONQueryEngine(
       json_value=json_value, json_schema=json_schema, service_context=service_context
   )
   raw_query_engine = JSONQueryEngine(
       json_value=json_value,
       json_schema=json_schema,
       service_context=service_context,
       synthesize_response=False,
   )
   nl_response = nl_query_engine.query(
       "What comments has Jerry been writing?",
   )
   raw_response = raw_query_engine.query(
       "What comments has Jerry been writing?",
   )
   display(Markdown(f"
Natural language Response

{nl_response}"))
   display(Markdown(f"
Raw JSON Response

{raw_response}"))
   # get the json path query string. Same would apply to raw_response
   print(nl_response.metadata["json_path_response_str"])
   $.comments[?(@.username=='jerry')].content


Query Engine with Pydantic Outputs

Every query engine has support for integrated structured responses
using the following "response_mode"s in "RetrieverQueryEngine":
* "refine"
* "compact"
* "tree_summarize"
* "accumulate" (beta, requires extra parsing to convert to objects)
* "compact_accumulate" (beta, requires extra parsing to convert to
  objects)
In this notebook, we walk through a small example demonstrating the
usage.
Under the hood, every LLM response will be a pydantic object. If that
response needs to be refined or summarized, it is converted into a
JSON string for the next response. Then, the final response is
returned as a pydantic object.
**NOTE:** This can technically work with any LLM, but non-openai is
support is still in development and considered beta.
Setup
   import os
   import openai
   os.environ["OPENAI_API_KEY"] = "sk-..."
   openai.api_key = os.environ["OPENAI_API_KEY"]
   from llama_index import SimpleDirectoryReader
   documents = SimpleDirectoryReader("../data/paul_graham").load_data()
Create our Pydanitc Output Object
   from typing import List
   from pydantic import BaseModel
   class Biography(BaseModel):
       """Data model for a biography."""
       name: str
       best_known_for: List[str]
       extra_info: str
Create the Index + Query Engine (OpenAI)
When using OpenAI, the function calling API will be leveraged for
reliable structured outputs.
   from llama_index import VectorStoreIndex, ServiceContext
   from llama_index.llms import OpenAI
   llm = OpenAI(model="gpt-3.5-turbo", temperature=0.1)
   service_context = ServiceContext.from_defaults(llm=llm)
   index = VectorStoreIndex.from_documents(documents, service_context=service_context)
   query_engine = index.as_query_engine(output_cls=Biography, response_mode="compact")
   response = query_engine.query("Who is Paul Graham?")
   print(response.name)
   print(response.best_known_for)
   print(response.extra_info)
   Paul Graham
   ['working on Bel', 'co-founding Viaweb', 'creating the programming language Arc']
   Paul Graham is a computer scientist, entrepreneur, and writer. He is best known for his work on Bel, a programming language, and for co-founding Viaweb, an early web application company that was later acquired by Yahoo. Graham also created the programming language Arc. He has written numerous essays on topics such as startups, programming, and life.
   # get the full pydanitc object
   print(type(response.response))
   <class '__main__.Biography'>
Create the Index + Query Engine (Non-OpenAI, Beta)
When using an LLM that does not support function calling, we rely on
the LLM to write the JSON itself, and we parse the JSON into the
proper pydantic object.
   import os
   os.environ["ANTHROPIC_API_KEY"] = "sk-..."
   from llama_index import VectorStoreIndex, ServiceContext
   from llama_index.llms import Anthropic
   llm = Anthropic(model="claude-instant-1.2", temperature=0.1)
   service_context = ServiceContext.from_defaults(llm=llm)
   index = VectorStoreIndex.from_documents(documents, service_context=service_context)
   query_engine = index.as_query_engine(
       output_cls=Biography, response_mode="tree_summarize"
   )
   response = query_engine.query("Who is Paul Graham?")
   print(response.name)
   print(response.best_known_for)
   print(response.extra_info)
   Paul Graham
   ['Co-founder of Y Combinator', 'Essayist and programmer']
   He is known for creating Viaweb, one of the first web application builders, and for founding Y Combinator, one of the world's top startup accelerators. Graham has also written extensively about technology, investing, and philosophy.

Query Engine with Pydantic Outputs

   # get the full pydanitc object
   print(type(response.response))
   <class '__main__.Biography'>
Accumulate Examples (Beta)
Accumulate with pydantic objects requires some extra parsing. This is
still a beta feature, but it's still possible to get accumulate
pydantic objects.
   from typing import List
   from pydantic import BaseModel
   class Company(BaseModel):
       """Data model for a companies mentioned."""
       company_name: str
       context_info: str
   from llama_index import VectorStoreIndex, ServiceContext
   from llama_index.llms import OpenAI
   llm = OpenAI(model="gpt-3.5-turbo", temperature=0.1)
   service_context = ServiceContext.from_defaults(llm=llm)
   index = VectorStoreIndex.from_documents(documents, service_context=service_context)
   query_engine = index.as_query_engine(output_cls=Company, response_mode="accumulate")
   response = query_engine.query("What companies are mentioned in the text?")
In accumulate, responses are separated by a default separator, and
prepended with a prefix.
   companies = []
   # split by the default separator
   for response_str in str(response).split("\n---------------------\n"):
       # remove the prefix --  every response starts like `Response 1: {...}`
       # so, we find the first bracket and remove everything before it
       response_str = response_str[response_str.find("{") :]
       companies.append(Company.parse_raw(response_str))
   print(companies)
   [Company(company_name='Yahoo', context_info='Yahoo bought us'), Company(company_name='Yahoo', context_info="I'd been meaning to since Yahoo bought us")]


Sub Question Query Engine

In this tutorial, we showcase how to use a **sub question query
engine** to tackle the problem of answering a complex query using
multiple data sources.It first breaks down the complex query into sub
questions for each relevant data source, then gather all the
intermediate reponses and synthesizes a final response.
Preparation
   import os
   import openai
   os.environ["OPENAI_API_KEY"] = "sk-..."
   openai.api_key = os.environ["OPENAI_API_KEY"]
   import nest_asyncio
   nest_asyncio.apply()
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   from llama_index.tools import QueryEngineTool, ToolMetadata
   from llama_index.query_engine import SubQuestionQueryEngine
   from llama_index.callbacks import CallbackManager, LlamaDebugHandler
   from llama_index import ServiceContext
   # Using the LlamaDebugHandler to print the trace of the sub questions
   # captured by the SUB_QUESTION callback event type
   llama_debug = LlamaDebugHandler(print_trace_on_end=True)
   callback_manager = CallbackManager([llama_debug])
   service_context = ServiceContext.from_defaults(callback_manager=callback_manager)
   # load data
   pg_essay = SimpleDirectoryReader(input_dir="../data/paul_graham/").load_data()
   # build index and query engine
   vector_query_engine = VectorStoreIndex.from_documents(
       pg_essay, use_async=True, service_context=service_context
   ).as_query_engine()
   **********
   Trace: index_construction
       |_node_parsing ->  0.394271 seconds
         |_chunking ->  0.393344 seconds
       |_embedding ->  0.753133 seconds
       |_embedding ->  0.749828 seconds
   **********
Setup sub question query engine
   # setup base query engine as tool
   query_engine_tools = [
       QueryEngineTool(
           query_engine=vector_query_engine,
           metadata=ToolMetadata(
               name="pg_essay", description="Paul Graham essay on What I Worked On"
           ),
       ),
   ]
   query_engine = SubQuestionQueryEngine.from_defaults(
       query_engine_tools=query_engine_tools,
       service_context=service_context,
       use_async=True,
   )
Run queries
   response = query_engine.query(
       "How was Paul Grahams life different before, during, and after YC?"
   )
   Generated 3 sub questions.
   [pg_essay] Q: What did Paul Graham do before YC?
   [pg_essay] Q: What did Paul Graham do during YC?
   [pg_essay] Q: What did Paul Graham do after YC?
   [pg_essay] A:
   Before YC, Paul Graham was a hacker, writer, and worked on Arc, a programming language. He also wrote essays and worked on other projects.
   [pg_essay] A:
   Paul Graham stopped working on YC in March 2014 and began painting. He spent most of the rest of the year painting and then in November he ran out of steam and stopped. He then began writing essays again and in March 2015 he started working on Lisp again.
   [pg_essay] A:
   Paul Graham worked on YC in a variety of ways. He wrote essays, worked on internal software in Arc, and created Hacker News. He also helped select and support founders, dealt with disputes between cofounders, and fought with people who maltreated the startups. He worked hard even at the parts he didn't like, and was determined to make YC a success. In 2010, he was offered unsolicited advice to make sure YC wasn't the last cool thing he did, which set him thinking about his future. In 2012, he decided to hand YC over to someone else and recruited Sam Altman to take over. He worked on YC until March 2014, when his mother passed away, and then he checked out completely.

Sub Question Query Engine

   **********
   Trace: query
       |_query ->  13.064431 seconds
         |_llm ->  2.499768 seconds
         |_sub_question ->  2.05934 seconds
           |_query ->  2.059142 seconds
             |_retrieve ->  0.278184 seconds
               |_embedding ->  0.274593 seconds
             |_synthesize ->  1.780895 seconds
               |_llm ->  1.740488 seconds
         |_sub_question ->  5.364061 seconds
           |_query ->  5.363695 seconds
             |_retrieve ->  0.230257 seconds
               |_embedding ->  0.226763 seconds
             |_synthesize ->  5.133343 seconds
               |_llm ->  5.091069 seconds
         |_sub_question ->  2.148964 seconds
           |_query ->  2.14889 seconds
             |_retrieve ->  0.323438 seconds
               |_embedding ->  0.319841 seconds
             |_synthesize ->  1.825401 seconds
               |_llm ->  1.783064 seconds
         |_synthesize ->  5.198214 seconds
           |_llm ->  5.175849 seconds
   **********
   print(response)
   Before YC, Paul Graham was a hacker, writer, and worked on Arc, a programming language. During YC, he wrote essays, worked on internal software in Arc, and created Hacker News. He also helped select and support founders, dealt with disputes between cofounders, and fought with people who maltreated the startups. After YC, Paul Graham stopped working on YC and began painting. He then began writing essays again and in March 2015 he started working on Lisp again. Paul Graham's life was different before, during, and after YC in that he changed his focus from programming and writing to painting and then back to programming and writing.
   # iterate through sub_question items captured in SUB_QUESTION event
   from llama_index.callbacks.schema import CBEventType, EventPayload
   for i, (start_event, end_event) in enumerate(
       llama_debug.get_event_pairs(CBEventType.SUB_QUESTION)
   ):
       qa_pair = end_event.payload[EventPayload.SUB_QUESTION]
       print("Sub Question " + str(i) + ": " + qa_pair.sub_q.sub_question.strip())
       print("Answer: " + qa_pair.answer.strip())
       print("====================================")
   Sub Question 0: What did Paul Graham do before YC?
   Answer: Before YC, Paul Graham was a hacker, writer, and worked on Arc, a programming language. He also wrote essays and worked on other projects.
   ====================================
   Sub Question 1: What did Paul Graham do during YC?
   Answer: Paul Graham worked on YC in a variety of ways. He wrote essays, worked on internal software in Arc, and created Hacker News. He also helped select and support founders, dealt with disputes between cofounders, and fought with people who maltreated the startups. He worked hard even at the parts he didn't like, and was determined to make YC a success. In 2010, he was offered unsolicited advice to make sure YC wasn't the last cool thing he did, which set him thinking about his future. In 2012, he decided to hand YC over to someone else and recruited Sam Altman to take over. He worked on YC until March 2014, when his mother passed away, and then he checked out completely.
   ====================================
   Sub Question 2: What did Paul Graham do after YC?
   Answer: Paul Graham stopped working on YC in March 2014 and began painting. He spent most of the rest of the year painting and then in November he ran out of steam and stopped. He then began writing essays again and in March 2015 he started working on Lisp again.

Sub Question Query Engine


Defining a Custom Query Engine

You can (and should) define your custom query engines in order to plug
into your downstream LlamaIndex workflows, whether you're building
RAG, agents, or other applications.
We provide a "CustomQueryEngine" that makes it easy to define your own
queries.
Setup
We first load some sample data and index it.
   from llama_index import (
       VectorStoreIndex,
       SimpleDirectoryReader,
   )
   # load documents
   documents = SimpleDirectoryReader(
       "../../../examples/paul_graham_essay/data"
   ).load_data()
   index = VectorStoreIndex.from_documents(documents)
   retriever = index.as_retriever()
Building a Custom Query Engine
We build a custom query engine that simulates a RAG pipeline. First
perform retrieval, and then synthesis.
To define a "CustomQueryEngine", you just have to define some
initialization parameters as attributes and implement the
"custom_query" function.
By default, the "custom_query" can return a "Response" object (which
the response synthesizer returns), but it can also just return a
string. These are options 1 and 2 respectively.
   from llama_index.query_engine import CustomQueryEngine
   from llama_index.retrievers import BaseRetriever
   from llama_index.response_synthesizers import get_response_synthesizer, BaseSynthesizer
Option 1 ("RAGQueryEngine")
   class RAGQueryEngine(CustomQueryEngine):
       """RAG Query Engine."""
       retriever: BaseRetriever
       response_synthesizer: BaseSynthesizer
       def custom_query(self, query_str: str):
           nodes = self.retriever.retrieve(query_str)
           response_obj = self.response_synthesizer.synthesize(query_str, nodes)
           return response_obj
Option 2 ("RAGStringQueryEngine")
   # Option 2: return a string (we use a raw LLM call for illustration)
   from llama_index.llms import OpenAI
   from llama_index.prompts import PromptTemplate
   qa_prompt = PromptTemplate(
       "Context information is below.\n"
       "---------------------\n"
       "{context_str}\n"
       "---------------------\n"
       "Given the context information and not prior knowledge, "
       "answer the query.\n"
       "Query: {query_str}\n"
       "Answer: "
   )
   class RAGStringQueryEngine(CustomQueryEngine):
       """RAG String Query Engine."""
       retriever: BaseRetriever
       response_synthesizer: BaseSynthesizer
       llm: OpenAI
       qa_prompt: PromptTemplate
       def custom_query(self, query_str: str):
           nodes = self.retriever.retrieve(query_str)
           context_str = "\n\n".join([n.node.get_content() for n in nodes])
           response = self.llm.complete(
               qa_prompt.format(context_str=context_str, query_str=query_str)
           )
           return str(response)
Trying it out
We now try it out on our sample data.
Trying Option 1 ("RAGQueryEngine")
   synthesizer = get_response_synthesizer(response_mode="compact")
   query_engine = RAGQueryEngine(retriever=retriever, response_synthesizer=synthesizer)
   response = query_engine.query("What did the author do growing up?")
   print(str(response))
   The author worked on writing and programming outside of school before college. They wrote short stories and tried writing programs on an IBM 1401 computer using an early version of Fortran. They also mentioned getting a microcomputer, building it themselves, and writing simple games and programs on it.
   print(response.source_nodes[0].get_content())
Trying Option 2 ("RAGStringQueryEngine")
   llm = OpenAI(model="gpt-3.5-turbo")

Defining a Custom Query Engine

   query_engine = RAGStringQueryEngine(
       retriever=retriever, response_synthesizer=synthesizer, llm=llm, qa_prompt=qa_prompt
   )
   response = query_engine.query("What did the author do growing up?")
   print(str(response))
   The author worked on writing and programming before college. They wrote short stories and started programming on the IBM 1401 computer in 9th grade. They later got a microcomputer and continued programming, writing simple games and a word processor.


Retriever Router Query Engine

In this tutorial, we define a router query engine based on a
retriever. The retriever will select a set of nodes, and we will in
turn select the right QueryEngine.
We use our new "ToolRetrieverRouterQueryEngine" class for this!
Setup
   # NOTE: This is ONLY necessary in jupyter notebook.
   # Details: Jupyter runs an event-loop behind the scenes.
   #          This results in nested event-loops when we start an event-loop to make async queries.
   #          This is normally not allowed, we use nest_asyncio to allow it for convenience.
   import nest_asyncio
   nest_asyncio.apply()
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SummaryIndex,
       SimpleDirectoryReader,
       ServiceContext,
       StorageContext,
   )
   INFO:numexpr.utils:Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   INFO:numexpr.utils:NumExpr defaulting to 8 threads.
   NumExpr defaulting to 8 threads.
   /Users/jerryliu/Programming/gpt_index/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
Load Data
We first show how to convert a Document into a set of Nodes, and
insert into a DocumentStore.
   # load documents
   documents = SimpleDirectoryReader("../data/paul_graham").load_data()
   # initialize service context (set chunk size)
   service_context = ServiceContext.from_defaults(chunk_size=1024)
   nodes = service_context.node_parser.get_nodes_from_documents(documents)
   # initialize storage context (by default it's in-memory)
   storage_context = StorageContext.from_defaults()
   storage_context.docstore.add_documents(nodes)
Define Summary Index and Vector Index over Same Data
   summary_index = SummaryIndex(nodes, storage_context=storage_context)
   vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens
   > [build_index_from_nodes] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 17038 tokens
   > [build_index_from_nodes] Total embedding token usage: 17038 tokens
Define Query Engine and Tool for these Indices
We define a Query Engine for each Index. We then wrap these with our
"QueryEngineTool".
   from llama_index.tools.query_engine import QueryEngineTool
   list_query_engine = summary_index.as_query_engine(
       response_mode="tree_summarize", use_async=True
   )
   vector_query_engine = vector_index.as_query_engine(
       response_mode="tree_summarize", use_async=True
   )
   list_tool = QueryEngineTool.from_defaults(

Retriever Router Query Engine

       query_engine=list_query_engine,
       description="Useful for questions asking for a biography of the author.",
   )
   vector_tool = QueryEngineTool.from_defaults(
       query_engine=vector_query_engine,
       description="Useful for retrieving specific snippets from the author's life, like his time in college, his time in YC, or more.",
   )
Define Retrieval-Augmented Router Query Engine
We define a router query engine that's augmented with a retrieval
mechanism, to help deal with the case when the set of choices is too
large.
To do this, we first define an "ObjectIndex" over the set of query
engine tools. The "ObjectIndex" is defined an underlying index data
structure (e.g. a vector index, keyword index), and can serialize
QueryEngineTool objects to/from our indices.
We then use our "ToolRetrieverRouterQueryEngine" class, and pass in an
"ObjectRetriever" over "QueryEngineTool" objects. The
"ObjectRetriever" corresponds to our "ObjectIndex".
This retriever can then dyamically retrieve the relevant query engines
during query-time. This allows us to pass in an arbitrary number of
query engine tools without worrying about prompt limitations.
   from llama_index import VectorStoreIndex
   from llama_index.objects import ObjectIndex, SimpleToolNodeMapping
   tool_mapping = SimpleToolNodeMapping.from_objects([list_tool, vector_tool])
   obj_index = ObjectIndex.from_objects(
       [list_tool, vector_tool],
       tool_mapping,
       VectorStoreIndex,
   )
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 59 tokens
   > [build_index_from_nodes] Total embedding token usage: 59 tokens
   from llama_index.query_engine import ToolRetrieverRouterQueryEngine
   query_engine = ToolRetrieverRouterQueryEngine(obj_index.as_retriever())
   response = query_engine.query("What is a biography of the author's life?")
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 10 tokens
   > [retrieve] Total embedding token usage: 10 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
   > [retrieve] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 2111 tokens
   > [get_response] Total LLM token usage: 2111 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
   > [retrieve] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 2148 tokens

Retriever Router Query Engine

   > [get_response] Total LLM token usage: 2148 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.query_engine.router_query_engine:Combining responses from multiple query engines.
   Combining responses from multiple query engines.
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1063 tokens
   > [get_response] Total LLM token usage: 1063 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   print(str(response))
   The author is a creative person who has had a varied and interesting life. They grew up in the US and went to college, but then decided to take a break and pursue their passion for art. They applied to two art schools, RISD in the US and the Accademia di Belli Arti in Florence, and were accepted to both. They chose to go to Florence, where they took the entrance exam and passed. They then spent a year living in Florence, studying art at the Accademia and painting still lives in their bedroom. After their year in Florence, the author returned to the US and completed their BFA program at RISD. They then went on to pursue a PhD in computer science at MIT, where they wrote a dissertation on the evolution of computers. During their time at MIT, they also did consulting work and wrote essays on topics they had been thinking about. After completing their PhD, the author started a software company, Viaweb, which was eventually acquired by Yahoo. They then went on to write essays and articles about their experiences in the tech industry. They also wrote an essay about how to choose what to work on, which was based on their own experience. The author then moved back to Florence, where they found a rent-stabilized apartment and continued to pursue their interest in art. They wrote about their experiences in the art world, and experienced the reactions of readers to their essays. The author is now a successful writer and continues to write essays and articles about topics they are passionate about.
   In summary, the author's life has been a journey of exploration and creativity. They have experienced a wide range of different things in their life, from art school to computer science to the tech industry, and have used their experiences to inform their writing. They have pursued their passion for art, and have used their knowledge and experience to create meaningful work.
   response
   "\nThe author is a creative person who has had a varied and interesting life. They grew up in the US and went to college, but then decided to take a break and pursue their passion for art. They applied to two art schools, RISD in the US and the Accademia di Belli Arti in Florence, and were accepted to both. They chose to go to Florence, where they took the entrance exam and passed. They then spent a year living in Florence, studying art at the Accademia and painting still lives in their bedroom. After their year in Florence, the author returned to the US and completed their BFA program at RISD. They then went on to pursue a PhD in computer science at MIT, where they wrote a dissertation on the evolution of computers. During their time at MIT, they also did consulting work and wrote essays on topics they had been thinking about. After completing their PhD, the author started a software company, Viaweb, which was eventually acquired by Yahoo. They then went on to write essays and articles about their experiences in the tech industry. They also wrote an essay about how to choose what to work on, which was based on their own experience. The author then moved back to Florence, where they found a rent-stabilized apartment and continued to pursue their interest in art. They wrote about their experiences in the art world, and experienced the reactions of readers to their essays. The author is now a successful writer and continues to write essays and articles about topics they are passionate about. \n\nIn summary, the author's life has been a journey of exploration and creativity. They have experienced a wide range of different things in their life, from art school to computer science to the tech industry, and have used their experiences to inform their writing. They have pursued their passion for art, and have used their knowledge and experience to create meaningful work."

Retriever Router Query Engine

   response = query_engine.query("What did Paul Graham do during his time in college?")
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 11 tokens
   > [retrieve] Total embedding token usage: 11 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
   > [retrieve] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1947 tokens
   > [get_response] Total LLM token usage: 1947 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 0 tokens
   > [retrieve] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 1947 tokens
   > [get_response] Total LLM token usage: 1947 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   INFO:llama_index.query_engine.router_query_engine:Combining responses from multiple query engines.
   Combining responses from multiple query engines.
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 316 tokens
   > [get_response] Total LLM token usage: 316 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   print(str(response))
   Paul Graham studied philosophy in college, but he did not pursue AI. He continued to work on programming outside of school, writing simple games, a program to predict how high his model rockets would fly, and a word processor. He eventually convinced his father to buy him a TRS-80 computer, which he used to further his programming skills.