Building Evaluation from Scratch

We show how you can build evaluation modules from scratch. This
includes both evaluation of the final generated response (where the
output is plain text), as well as the evaluation of retrievers (where
the output is a ranked list of items).
We have in-house modules in our Evaluation section.
Setup
We load some data and define a very simple RAG query engine that we'll
evaluate (uses top-k retrieval).
   !mkdir data
   !wget --user-agent "Mozilla" "https://arxiv.org/pdf/2307.09288.pdf" -O "data/llama2.pdf"
   mkdir: data: File exists
   --2023-09-19 00:05:14--  https://arxiv.org/pdf/2307.09288.pdf
   Resolving arxiv.org (arxiv.org)... 128.84.21.199
   Connecting to arxiv.org (arxiv.org)|128.84.21.199|:443... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: 13661300 (13M) [application/pdf]
   Saving to: ‘data/llama2.pdf’
   data/llama2.pdf     100%[===================>]  13.03M  1.56MB/s    in 9.3s
   2023-09-19 00:05:25 (1.40 MB/s) - ‘data/llama2.pdf’ saved [13661300/13661300]
   from pathlib import Path
   from llama_hub.file.pymu_pdf.base import PyMuPDFReader
   loader = PyMuPDFReader()
   documents = loader.load(file_path="./data/llama2.pdf")
   from llama_index import VectorStoreIndex, ServiceContext
   from llama_index.node_parser import SimpleNodeParser
   from llama_index.llms import OpenAI
   llm = OpenAI(model="gpt-4")
   node_parser = SimpleNodeParser.from_defaults(chunk_size=1024)
   service_context = ServiceContext.from_defaults(llm=llm)
   nodes = node_parser.get_nodes_from_documents(documents)
   index = VectorStoreIndex(nodes, service_context=service_context)
   query_engine = index.as_query_engine()
Dataset Generation
We first go through an exercise of generating a synthetic evaluation
dataset. We do this by synthetically generating a set of questions
from existing context. We then run each question with existing context
through a powerful LLM (e.g. GPT-4) to generate a "ground-truth"
response.
Define Functions
We define the functions that we will use for dataset generation:
   from llama_index.schema import BaseNode
   from llama_index.llms import OpenAI
   from llama_index.prompts import (
       ChatMessage,
       ChatPromptTemplate,
       MessageRole,
       PromptTemplate,
   )
   from typing import Tuple, List
   import re
   llm = OpenAI(model="gpt-4")
We define "generate_answers_for_questions" to generate answers from
questions given context.
   QA_PROMPT = PromptTemplate(
       "Context information is below.\n"
       "---------------------\n"
       "{context_str}\n"
       "---------------------\n"
       "Given the context information and not prior knowledge, "
       "answer the query.\n"
       "Query: {query_str}\n"
       "Answer: "
   )
   def generate_answers_for_questions(
       questions: List[str], context: str, llm: OpenAI
   ) -> str:
       """Generate answers for questions given context."""
       answers = []
       for question in questions:
           fmt_qa_prompt = QA_PROMPT.format(context_str=context, query_str=question)
           response_obj = llm.complete(fmt_qa_prompt)

Building Evaluation from Scratch

           answers.append(str(response_obj))
       return answers
We define "generate_qa_pairs" to generate qa pairs over an entire list
of Nodes.
   QUESTION_GEN_USER_TMPL = (
       "Context information is below.\n"
       "---------------------\n"
       "{context_str}\n"
       "---------------------\n"
       "Given the context information and not prior knowledge, "
       "generate the relevant questions. "
   )
   QUESTION_GEN_SYS_TMPL = """\
   You are a Teacher/ Professor. Your task is to setup \
   {num_questions_per_chunk} questions for an upcoming \
   quiz/examination. The questions should be diverse in nature \
   across the document. Restrict the questions to the \
   context information provided.\
   """
   question_gen_template = ChatPromptTemplate(
       message_templates=[
           ChatMessage(role=MessageRole.SYSTEM, content=QUESTION_GEN_SYS_TMPL),
           ChatMessage(role=MessageRole.USER, content=QUESTION_GEN_USER_TMPL),
       ]
   )
   def generate_qa_pairs(
       nodes: List[BaseNode], llm: OpenAI, num_questions_per_chunk: int = 10
   ) -> List[Tuple[str, str]]:
       """Generate questions."""
       qa_pairs = []
       for idx, node in enumerate(nodes):
           print(f"Node {idx}/{len(nodes)}")
           context_str = node.get_content(metadata_mode="all")
           fmt_messages = question_gen_template.format_messages(
               num_questions_per_chunk=10,
               context_str=context_str,
           )
           chat_response = llm.chat(fmt_messages)
           raw_output = chat_response.message.content
           result_list = str(raw_output).strip().split("\n")
           cleaned_questions = [
               re.sub(r"^\d+[\).\s]", "", question).strip() for question in result_list
           ]
           answers = generate_answers_for_questions(cleaned_questions, context_str, llm)
           cur_qa_pairs = list(zip(cleaned_questions, answers))
           qa_pairs.extend(cur_qa_pairs)
       return qa_pairs
   qa_pairs
   [('What is the main focus of the work described in the document?',
     'The main focus of the work described in the document is the development and release of Llama 2, a collection of pretrained and fine-tuned large language models (LLMs) ranging in scale from 7 billion to 70 billion parameters. The fine-tuned LLMs, called Llama 2-Chat, are optimized for dialogue use cases. The document also provides a detailed description of the approach to fine-tuning and safety improvements of Llama 2-Chat.'),
    ('What is the range of parameters for the large language models (LLMs) developed in this work?',
     'The range of parameters for the large language models (LLMs) developed in this work is from 7 billion to 70 billion.'),
    ('What is the specific name given to the fine-tuned LLMs optimized for dialogue use cases?',
     'The specific name given to the fine-tuned LLMs optimized for dialogue use cases is Llama 2-Chat.'),
    ('How do the models developed in this work compare to open-source chat models based on the benchmarks tested?',
     'The models developed in this work, specifically the fine-tuned LLMs called Llama 2-Chat, outperform open-source chat models on most benchmarks tested.'),
    ('What are the two key areas of human evaluation mentioned in the document for the developed models?',
     'The two key areas of human evaluation mentioned in the document for the developed models are helpfulness and safety.'),
    ('What is the purpose of providing a detailed description of the approach to fine-tuning and safety improvements of Llama 2-Chat?',
     'The purpose of providing a detailed description of the approach to fine-tuning and safety improvements of Llama 2-Chat is to enable the community to build on their work and contribute to the responsible development of Large Language Models (LLMs).'),

Building Evaluation from Scratch

    ('What is the intended benefit for the community from this work?',
     'The intended benefit for the community from this work is to enable them to build on the work and contribute to the responsible development of large language models (LLMs). The team provides a detailed description of their approach to fine-tuning and safety improvements of Llama 2-Chat for this purpose.'),
    ('Who are the corresponding authors of this work and how can they be contacted?',
     'The corresponding authors of this work are Thomas Scialom and Hugo Touvron. They can be contacted via email at tscialom@meta.com and htouvron@meta.com respectively.'),
    ('What is the source of the document and how many pages does it contain?',
     'The source of the document is "1" and it contains 77 pages.'),
    ('Where can the contributions of all the authors be found in the document?',
     'The contributions of all the authors can be found in Section A.1 of the document.')]
Getting Pairs over Dataset
**NOTE**: This can take a long time. For the sake of speed try
inputting a subset of the nodes.
   qa_pairs = generate_qa_pairs(
       # nodes[:1],
       nodes,
       llm,
       num_questions_per_chunk=10,
   )
[Optional] Define save/load
~~~~~~~~~~~~~~~~~~~~~~~~~~~
   # save
   import pickle
   pickle.dump(qa_pairs, open("eval_dataset.pkl", "wb"))
   # save
   import pickle
   qa_pairs = pickle.load(open("eval_dataset.pkl", "rb"))
Evaluating Generation
In this section we walk through a few methods for evaluating the
generated results. At a high-level we use an "evaluation LLM" to
measure the quality of the generated results. We do this in both the
**with labels** setting and **without labels** setting.
We go through the following evaluation algorithms:
* **Correctness**: Compares the generated answer against the ground-
  truth answer.
* **Faithfulness**: Evaluates whether a response is faithful to the
  contexts (label-free).
Building a Correctness Evaluator
The correctness evaluator compares the generated answer to the
reference ground-truth answer, given the query. We output a score
between 1 and 5, where 1 is the worst and 5 is the best.
We do this through a system and user prompt with a chat interface.
   from llama_index.prompts import (
       ChatMessage,
       ChatPromptTemplate,
       MessageRole,
       PromptTemplate,
   )
   from typing import Dict
   CORRECTNESS_SYS_TMPL = """
   You are an expert evaluation system for a question answering chatbot.
   You are given the following information:
   - a user query,
   - a reference answer, and
   - a generated answer.
   Your job is to judge the relevance and correctness of the generated answer.
   Output a single score that represents a holistic evaluation.
   You must return your response in a line with only the score.
   Do not return answers in any other format.
   On a separate line provide your reasoning for the score as well.
   Follow these guidelines for scoring:
   - Your score has to be between 1 and 5, where 1 is the worst and 5 is the best.
   - If the generated answer is not relevant to the user query, \
   you should give a score of 1.
   - If the generated answer is relevant but contains mistakes, \
   you should give a score between 2 and 3.
   - If the generated answer is relevant and fully correct, \
   you should give a score between 4 and 5.
   """
   CORRECTNESS_USER_TMPL = """
   ## User Query
   {query}
   ## Reference Answer
   {reference_answer}
   ## Generated Answer
   {generated_answer}

Building Evaluation from Scratch

   """
   eval_chat_template = ChatPromptTemplate(
       message_templates=[
           ChatMessage(role=MessageRole.SYSTEM, content=CORRECTNESS_SYS_TMPL),
           ChatMessage(role=MessageRole.USER, content=CORRECTNESS_USER_TMPL),
       ]
   )
Now that we've defined the prompts template, let's define an
evaluation function that feeds the prompt to the LLM and parses the
output into a dict of results.
   from llama_index.llms import OpenAI
   def run_correctness_eval(
       query_str: str,
       reference_answer: str,
       generated_answer: str,
       llm: OpenAI,
       threshold: float = 4.0,
   ) -> Dict:
       """Run correctness eval."""
       fmt_messages = eval_chat_template.format_messages(
           llm=llm,
           query=query_str,
           reference_answer=reference_answer,
           generated_answer=generated_answer,
       )
       chat_response = llm.chat(fmt_messages)
       raw_output = chat_response.message.content
       # Extract from response
       score_str, reasoning_str = raw_output.split("\n", 1)
       score = float(score_str)
       reasoning = reasoning_str.lstrip("\n")
       return {"passing": score >= threshold, "score": score, "reason": reasoning}
Now let's try running this on some sample inputs with a chat model
(GPT-4).
   llm = OpenAI(model="gpt-4")
   # query_str = "What is the range of parameters for the large language models (LLMs) developed in this work?"
   # reference_answer = "The range of parameters for the large language models (LLMs) developed in this work is from 7 billion to 70 billion."
   query_str = "What is the specific name given to the fine-tuned LLMs optimized for dialogue use cases?"
   reference_answer = "The specific name given to the fine-tuned LLMs optimized for dialogue use cases is Llama 2-Chat."
   generated_answer = str(query_engine.query(query_str))
   print(str(generated_answer))
   The fine-tuned Large Language Models (LLMs) optimized for dialogue use cases are specifically called Llama 2-Chat.
   eval_results = run_correctness_eval(
       query_str, reference_answer, generated_answer, llm=llm, threshold=4.0
   )
   display(eval_results)
   {'passing': True,
    'score': 5.0,
    'reason': 'The generated answer is completely relevant to the user query and matches the reference answer in terms of information. It correctly identifies "Llama 2-Chat" as the specific name given to the fine-tuned LLMs optimized for dialogue use cases.'}
Building a Faithfulness Evaluator
The faithfulness evaluator evaluates whether the response is faithful
to any of the retrieved contexts.
This is a step up in complexity from the correctness evaluator. Since
the set of contexts can be quite long, they might overflow the context
window. We would need to figure out how to implement a form of
**response synthesis** strategy to iterate over contexts in sequence.
We have a corresponding tutorial showing you how to build response
synthesis from scratch. We also have out-of-the-box response synthesis
modules. In this guide we'll use the out of the box modules.
   EVAL_TEMPLATE = PromptTemplate(
       "Please tell if a given piece of information "
       "is supported by the context.\n"
       "You need to answer with either YES or NO.\n"
       "Answer YES if any of the context supports the information, even "
       "if most of the context is unrelated. "
       "Some examples are provided below. \n\n"
       "Information: Apple pie is generally double-crusted.\n"
       "Context: An apple pie is a fruit pie in which the principal filling "

Building Evaluation from Scratch

       "ingredient is apples. \n"
       "Apple pie is often served with whipped cream, ice cream "
       "('apple pie à la mode'), custard or cheddar cheese.\n"
       "It is generally double-crusted, with pastry both above "
       "and below the filling; the upper crust may be solid or "
       "latticed (woven of crosswise strips).\n"
       "Answer: YES\n"
       "Information: Apple pies tastes bad.\n"
       "Context: An apple pie is a fruit pie in which the principal filling "
       "ingredient is apples. \n"
       "Apple pie is often served with whipped cream, ice cream "
       "('apple pie à la mode'), custard or cheddar cheese.\n"
       "It is generally double-crusted, with pastry both above "
       "and below the filling; the upper crust may be solid or "
       "latticed (woven of crosswise strips).\n"
       "Answer: NO\n"
       "Information: {query_str}\n"
       "Context: {context_str}\n"
       "Answer: "
   )
   EVAL_REFINE_TEMPLATE = PromptTemplate(
       "We want to understand if the following information is present "
       "in the context information: {query_str}\n"
       "We have provided an existing YES/NO answer: {existing_answer}\n"
       "We have the opportunity to refine the existing answer "
       "(only if needed) with some more context below.\n"
       "------------\n"
       "{context_msg}\n"
       "------------\n"
       "If the existing answer was already YES, still answer YES. "
       "If the information is present in the new context, answer YES. "
       "Otherwise answer NO.\n"
   )
**NOTE**: In the current response synthesizer setup we don't separate
out a system and user message for chat endpoints, so we just use our
standard "llm.complete" for text completion.
We now define our function below. Since we defined both a standard
eval template for a given piece of context but also a refine template
for subsequent contexts, we implement our "create-and-refine" response
synthesis strategy to obtain the answer.
   from llama_index.response_synthesizers import Refine
   from llama_index import ServiceContext
   from typing import List, Dict
   def run_faithfulness_eval(
       generated_answer: str,
       contexts: List[str],
       llm: OpenAI,
   ) -> Dict:
       """Run faithfulness eval."""
       service_context = ServiceContext.from_defaults(llm=llm)
       refine = Refine(
           text_qa_template=EVAL_TEMPLATE,
           refine_template=EVAL_REFINE_TEMPLATE,
       )
       response_obj = refine.get_response(generated_answer, contexts)
       response_txt = str(response_obj)
       if "yes" in response_txt.lower():
           passing = True
       else:
           passing = False
       return {"passing": passing, "reason": str(response_txt)}
Let's try it out on some data
   # use the same query_str, and reference_answer as above
   # query_str = "What is the specific name given to the fine-tuned LLMs optimized for dialogue use cases?"
   # reference_answer = "The specific name given to the fine-tuned LLMs optimized for dialogue use cases is Llama 2-Chat."
   response = query_engine.query(query_str)
   generated_answer = str(response)
   context_list = [n.get_content() for n in response.source_nodes]
   eval_results = run_faithfulness_eval(
       generated_answer,
       contexts=context_list,
       llm=llm,
   )
   display(eval_results)
   {'passing': True, 'reason': 'YES'}
Running Evaluation over our Eval Dataset
Now let's tie the two above sections together and run our eval modules

Building Evaluation from Scratch

over our eval dataset!
**NOTE**: For the sake of speed/cost we extract a very limited sample.
   import random
   sample_size = 5
   qa_pairs_sample = random.sample(qa_pairs, sample_size)
   import pandas as pd
   def run_evals(qa_pairs: List[Tuple[str, str]], llm: OpenAI, query_engine):
       results_list = []
       for question, reference_answer in qa_pairs:
           response = query_engine.query(question)
           generated_answer = str(response)
           correctness_results = run_correctness_eval(
               query_str, reference_answer, generated_answer, llm=llm, threshold=4.0
           )
           faithfulness_results = run_faithfulness_eval(
               generated_answer,
               contexts=context_list,
               llm=llm,
           )
           cur_result_dict = {
               "correctness": correctness_results["passing"],
               "faithfulness": faithfulness_results["passing"],
           }
           results_list.append(cur_result_dict)
       return pd.DataFrame(results_list)
   evals_df = run_evals(qa_pairs_sample, llm, query_engine)
   evals_df["correctness"].mean()
   0.4
   evals_df["faithfulness"].mean()
   0.6


Building a Router from Scratch

In this tutorial, we show you how to build an LLM-powered router
module that can route a user query to submodules.
Routers are a simple but effective form of automated decision making
that can allowing you to perform dynamic retrieval/querying over your
data.
In LlamaIndex, this is abstracted away with our Router Modules.
To build a router, we'll walk through the following steps:
* Crafting an initial prompt to select a set of choices
* Enforcing structured output (for text completion endpoints)
* Try integrating with a native function calling endpoint.
And then we'll plug this into a RAG pipeline to dynamically make
decisions on QA vs. summarization.
1. Setup a Basic Router Prompt
At its core, a router is a module that takes in a set of choices.
Given a user query, it "selects" a relevant choice.
For simplicity, we'll start with the choices as a set of strings.
   from llama_index import PromptTemplate
   choices = [
       "Useful for questions related to apples",
       "Useful for questions related to oranges",
   ]
   def get_choice_str(choices):
       choices_str = "\n\n".join([f"{idx+1}. {c}" for idx, c in enumerate(choices)])
       return choices_str
   choices_str = get_choice_str(choices)
   router_prompt0 = PromptTemplate(
       "Some choices are given below. It is provided in a numbered "
       "list (1 to {num_choices}), "
       "where each item in the list corresponds to a summary.\n"
       "---------------------\n"
       "{context_list}"
       "\n---------------------\n"
       "Using only the choices above and not prior knowledge, return the top choices "
       "(no more than {max_outputs}, but only select what is needed) that "
       "are most relevant to the question: '{query_str}'\n"
   )
Let's try this prompt on a set of toy questions and see what the
output brings.
   from llama_index.llms import OpenAI
   llm = OpenAI(model="gpt-3.5-turbo")
   def get_formatted_prompt(query_str):
       fmt_prompt = router_prompt0.format(
           num_choices=len(choices),
           max_outputs=2,
           context_list=choices_str,
           query_str=query_str,
       )
       return fmt_prompt
   query_str = "Can you tell me more about the amount of Vitamin C in apples"
   fmt_prompt = get_formatted_prompt(query_str)
   response = llm.complete(fmt_prompt)
   print(str(response))
   1. Useful for questions related to apples
   query_str = "What are the health benefits of eating orange peels?"
   fmt_prompt = get_formatted_prompt(query_str)
   response = llm.complete(fmt_prompt)
   print(str(response))
   2. Useful for questions related to oranges
   query_str = "Can you tell me more about the amount of Vitamin C in apples and oranges."
   fmt_prompt = get_formatted_prompt(query_str)
   response = llm.complete(fmt_prompt)
   print(str(response))
   1. Useful for questions related to apples
   2. Useful for questions related to oranges
**Observation**: While the response corresoponds to the correct
choice, it can be hacky to parse into a structured output (e.g. a
single integer). We'd need to do some string parsing on the choices to
extract out a single number, and make it robust to failure modes.
2. A Router Prompt that can generate structured outputs
Therefore the next step is to try to prompt the model to output a more
structured representation (JSON).
We define an output parser class ("RouterOutputParser"). This output
parser will be responsible for both formatting the prompt and also
parsing the result into a structured object (an "Answer").
We then apply the "format" and "parse" methods of the output parser

Building a Router from Scratch

around the LLM call using the router prompt to generate a structured
output.
2.a Import Answer Class
We load in the Answer class from our codebase. It's a very simple
dataclass with two fields: "choice" and "reason"
   from dataclasses import fields
   from pydantic import BaseModel
   import json
   class Answer(BaseModel):
       choice: int
       reason: str
   print(json.dumps(Answer.schema(), indent=2))
   {
     "title": "Answer",
     "type": "object",
     "properties": {
       "choice": {
         "title": "Choice",
         "type": "integer"
       },
       "reason": {
         "title": "Reason",
         "type": "string"
       }
     },
     "required": [
       "choice",
       "reason"
     ]
   }
2.b Define Router Output Parser
   from llama_index.types import BaseOutputParser
   FORMAT_STR = """The output should be formatted as a JSON instance that conforms to
   the JSON schema below.
   Here is the output schema:
   {
     "type": "array",
     "items": {
       "type": "object",
       "properties": {
         "choice": {
           "type": "integer"
         },
         "reason": {
           "type": "string"
         }
       },
       "required": [
         "choice",
         "reason"
       ],
       "additionalProperties": false
     }
   }
   """
If we want to put "FORMAT_STR" as part of an f-string as part of a
prompt template, then we'll need to escape the curly braces so that
they don't get treated as template variables.
   def _escape_curly_braces(input_string: str) -> str:
       # Replace '{' with '{{' and '}' with '}}' to escape curly braces
       escaped_string = input_string.replace("{", "{{").replace("}", "}}")
       return escaped_string
We now define a simple parsing function to extract out the JSON string
from the LLM response (by searching for square brackets)
   def _marshal_output_to_json(output: str) -> str:
       output = output.strip()
       left = output.find("[")
       right = output.find("]")
       output = output[left : right + 1]
       return output
We put these together in our "RouterOutputParser"
   from typing import List
   class RouterOutputParser(BaseOutputParser):
       def parse(self, output: str) -> List[Answer]:
           """Parse string."""
           json_output = _marshal_output_to_json(output)
           json_dicts = json.loads(json_output)
           answers = [Answer.from_dict(json_dict) for json_dict in json_dicts]
           return answers
       def format(self, prompt_template: str) -> str:
           return prompt_template + "\n\n" + _escape_curly_braces(FORMAT_STR)
2.c Give it a Try
We create a function called "route_query" that will take in the output
parser, llm, and prompt template and output a structured answer.
   output_parser = RouterOutputParser()
   from typing import List
   def route_query(query_str: str, choices: List[str], output_parser: RouterOutputParser):
       choices_str
       fmt_base_prompt = router_prompt0.format(
           num_choices=len(choices),
           max_outputs=len(choices),
           context_list=choices_str,
           query_str=query_str,
       )
       fmt_json_prompt = output_parser.format(fmt_base_prompt)
       raw_output = llm.complete(fmt_json_prompt)
       parsed = output_parser.parse(str(raw_output))
       return parsed
3. Perform Routing with a Function Calling Endpoint
In the previous section, we showed how to build a router with a text
completion endpoint. This includes formatting the prompt to encourage
the model output structured JSON, and a parse function to load in

Building a Router from Scratch

JSON.
This process can feel a bit messy. Function calling endpoints (e.g.
OpenAI) abstract away this complexity by allowing the model to
natively output structured functions. This obviates the need to
manually prompt + parse the outputs.
LlamaIndex offers an abstraction called a "PydanticProgram" that
integrates with a function endpoint to produce a structured Pydantic
object. We integrate with OpenAI and Guidance.
We redefine our "Answer" class with annotations, as well as an
"Answers" class containing a list of answers.
   from pydantic import Field
   class Answer(BaseModel):
       "Represents a single choice with a reason." ""
       choice: int
       reason: str
   class Answers(BaseModel):
       """Represents a list of answers."""
       answers: List[Answer]
   Answers.schema()
   {'title': 'Answers',
    'description': 'Represents a list of answers.',
    'type': 'object',
    'properties': {'answers': {'title': 'Answers',
      'type': 'array',
      'items': {'$ref': '#/definitions/Answer'}}},
    'required': ['answers'],
    'definitions': {'Answer': {'title': 'Answer',
      'description': 'Represents a single choice with a reason.',
      'type': 'object',
      'properties': {'choice': {'title': 'Choice', 'type': 'integer'},
       'reason': {'title': 'Reason', 'type': 'string'}},
      'required': ['choice', 'reason']}}}
   from llama_index.program import OpenAIPydanticProgram
   router_prompt1 = router_prompt0.partial_format(
       num_choices=len(choices),
       max_outputs=len(choices),
   )
   program = OpenAIPydanticProgram.from_defaults(
       output_cls=Answers,
       prompt=router_prompt1,
       verbose=True,
   )
   query_str = "What are the health benefits of eating orange peels?"
   output = program(context_list=choices_str, query_str=query_str)
   Function call: Answers with args: {
     "answers": [
       {
         "choice": 2,
         "reason": "Orange peels are related to oranges"
       }
     ]
   }
   output
   Answers(answers=[Answer(choice=2, reason='Orange peels are related to oranges')])
4. Plug Router Module as part of a RAG pipeline
In this section we'll put the router module to use in a RAG pipeline.
We'll use it to dynamically decide whether to perform question-
answering or summarization. We can easily get a question-answering
query engine using top-k retrieval through our vector index, while
summarization is performed through our summary index. Each query
engine is described as a "choice" to our router, and we compose the
whole thing into a single query engine.
Setup: Load Data
We load the Llama 2 paper as data.
   !mkdir data
   !wget --user-agent "Mozilla" "https://arxiv.org/pdf/2307.09288.pdf" -O "data/llama2.pdf"
   mkdir: data: File exists
   --2023-09-17 23:37:11--  https://arxiv.org/pdf/2307.09288.pdf
   Resolving arxiv.org (arxiv.org)... 128.84.21.199
   Connecting to arxiv.org (arxiv.org)|128.84.21.199|:443... connected.
   HTTP request sent, awaiting response... 200 OK
   Length: 13661300 (13M) [application/pdf]
   Saving to: ‘data/llama2.pdf’
   data/llama2.pdf     100%[===================>]  13.03M  1.50MB/s    in 9.5s

Building a Router from Scratch

   2023-09-17 23:37:22 (1.37 MB/s) - ‘data/llama2.pdf’ saved [13661300/13661300]
   from pathlib import Path
   from llama_hub.file.pymu_pdf.base import PyMuPDFReader
   loader = PyMuPDFReader()
   documents = loader.load(file_path="./data/llama2.pdf")
Setup: Define Indexes
Define both a vector index and summary index over this data.
   from llama_index import ServiceContext, VectorStoreIndex, SummaryIndex
   service_context = ServiceContext.from_defaults(chunk_size=1024)
   vector_index = VectorStoreIndex.from_documents(
       documents, service_context=service_context
   )
   summary_index = SummaryIndex.from_documents(documents, service_context=service_context)
   vector_query_engine = vector_index.as_query_engine()
   summary_query_engine = summary_index.as_query_engine()
Define RouterQueryEngine
We subclass our "CustomQueryEngine" to define a custom router.
   from llama_index.query_engine import CustomQueryEngine, BaseQueryEngine
   from llama_index.response_synthesizers import TreeSummarize
   class RouterQueryEngine(CustomQueryEngine):
       """Use our Pydantic program to perform routing."""
       query_engines: List[BaseQueryEngine]
       choice_descriptions: List[str]
       verbose: bool = False
       router_prompt: PromptTemplate
       llm: OpenAI
       summarizer: TreeSummarize = Field(default_factory=TreeSummarize)
       def custom_query(self, query_str: str):
           """Define custom query."""
           program = OpenAIPydanticProgram.from_defaults(
               output_cls=Answers,
               prompt=router_prompt1,
               verbose=self.verbose,
               llm=self.llm,
           )
           choices_str = get_choice_str(self.choice_descriptions)
           output = program(context_list=choices_str, query_str=query_str)
           # print choice and reason, and query the underlying engine
           if self.verbose:
               print(f"Selected choice(s):")
               for answer in output.answers:
                   print(f"Choice: {answer.choice}, Reason: {answer.reason}")
           responses = []
           for answer in output.answers:
               choice_idx = answer.choice - 1
               query_engine = self.query_engines[choice_idx]
               response = query_engine.query(query_str)
               responses.append(response)
           # if a single choice is picked, we can just return that response
           if len(responses) == 1:
               return responses[0]
           else:
               # if multiple choices are picked, we can pick a summarizer
               response_strs = [str(r) for r in responses]
               result_response = self.summarizer.get_response(query_str, response_strs)
               return result_response
   choices = [
       "Useful for answering questions about specific sections of the Llama 2 paper",
       "Useful for questions that ask for a summary of the whole paper",
   ]
   router_query_engine = RouterQueryEngine(
       query_engines=[vector_query_engine, summary_query_engine],
       choice_descriptions=choices,
       verbose=True,
       router_prompt=router_prompt1,
       llm=OpenAI(model="gpt-4"),
   )
Try our constructed Router Query Engine
Let's take our self-built router query engine for a spin! We ask a
question that routes to the vector query engine, and also another
question that routes to the summarization engine.
   response = router_query_engine.query(
       "How does the Llama 2 model compare to GPT-4 in the experimental results?"
   )
   Function call: Answers with args: {
     "answers": [
       {
         "choice": 1,
         "reason": "This question is asking for specific information about the Llama 2 model and its comparison to GPT-4 in the experimental results. Therefore, the summary that is useful for answering questions about specific sections of the paper would be most relevant."

Building a Router from Scratch

       }
     ]
   }
   Selected choice(s):
   Choice: 1, Reason: This question is asking for specific information about the Llama 2 model and its comparison to GPT-4 in the experimental results. Therefore, the summary that is useful for answering questions about specific sections of the paper would be most relevant.
   print(str(response))
   The Llama 2 model performs better than GPT-4 in the experimental results.
   response = router_query_engine.query("Can you give a summary of this paper?")
   Function call: Answers with args: {
     "answers": [
       {
         "choice": 2,
         "reason": "This choice is directly related to providing a summary of the whole paper, which is what the question asks for."
       }
     ]
   }
   Selected choice(s):
   Choice: 2, Reason: This choice is directly related to providing a summary of the whole paper, which is what the question asks for.
   print(str(response))


import nest_asyncio

   nest_asyncio.apply()
   import logging
   import sys
   import os
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       SimpleDirectoryReader,
       ServiceContext,
       StorageContext,
   )
   from llama_index import VectorStoreIndex, SummaryIndex, SimpleKeywordTableIndex
   from llama_index.llms import OpenAI
   from llama_index.response.notebook_utils import display_response
Load Documents
   reader = SimpleDirectoryReader("../paul_graham_essay/data")
   documents = reader.load_data()
Parse into Nodes
   from llama_index.node_parser import SimpleNodeParser
   nodes = SimpleNodeParser().get_nodes_from_documents(documents)
Add to Docstore
   TABLE_NAME = os.environ["DYNAMODB_TABLE_NAME"]
   from llama_index.storage.docstore.dynamodb_docstore import DynamoDBDocumentStore
   from llama_index.storage.index_store.dynamodb_index_store import DynamoDBIndexStore
   from llama_index.vector_stores.dynamodb import DynamoDBVectorStore
   storage_context = StorageContext.from_defaults(
       docstore=DynamoDBDocumentStore.from_table_name(table_name=TABLE_NAME),
       index_store=DynamoDBIndexStore.from_table_name(table_name=TABLE_NAME),
       vector_store=DynamoDBVectorStore.from_table_name(table_name=TABLE_NAME),
   )
   storage_context.docstore.add_documents(nodes)
Define & Add Multiple Indexes
Each index uses the same underlying Node.
   # https://gpt-index.readthedocs.io/en/latest/api_reference/indices/list.html
   summary_index = SummaryIndex(nodes, storage_context=storage_context)
   # https://gpt-index.readthedocs.io/en/latest/api_reference/indices/vector_store.html
   vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
   # https://gpt-index.readthedocs.io/en/latest/api_reference/indices/table.html
   keyword_table_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)
   # NOTE: the docstore still has the same nodes
   len(storage_context.docstore.docs)
Test out saving and loading
   # NOTE: docstore, index_store, and vector_index is persisted in DynamoDB by default when they are created
   # NOTE: You can also persist simple vector store to disk by using the command below
   storage_context.persist()
   # note down index IDs
   list_id = summary_index.index_id
   vector_id = vector_index.index_id
   keyword_id = keyword_table_index.index_id
   from llama_index.indices.loading import load_index_from_storage
   # re-create storage context
   storage_context = StorageContext.from_defaults(
       docstore=DynamoDBDocumentStore.from_table_name(table_name=TABLE_NAME),
       index_store=DynamoDBIndexStore.from_table_name(table_name=TABLE_NAME),
       vector_store=DynamoDBVectorStore.from_table_name(table_name=TABLE_NAME),
   )
   summary_index = load_index_from_storage(
       storage_context=storage_context, index_id=list_id
   )
   keyword_table_index = load_index_from_storage(
       storage_context=storage_context, index_id=keyword_id
   )
   # You need to add "vector_store=DynamoDBVectorStore.from_table_name(table_name=TABLE_NAME)" to StorageContext to load vector index from DynamoDB
   vector_index = load_index_from_storage(
       storage_context=storage_context, index_id=vector_id
   )
Test out some Queries
   chatgpt = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context_chatgpt = ServiceContext.from_defaults(llm=chatgpt, chunk_size=1024)
   query_engine = summary_index.as_query_engine()
   list_response = query_engine.query("What is a summary of this document?")
   display_response(list_response)
   query_engine = vector_index.as_query_engine()

import nest_asyncio

   vector_response = query_engine.query("What did the author do growing up?")
   display_response(vector_response)
   query_engine = keyword_table_index.as_query_engine()
   keyword_response = query_engine.query("What did the author do after his time at YC?")
   display_response(keyword_response)


import nest_asyncio

   nest_asyncio.apply()
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       SimpleDirectoryReader,
       ServiceContext,
       LLMPredictor,
       StorageContext,
   )
   from llama_index import VectorStoreIndex, SummaryIndex, SimpleKeywordTableIndex
   from llama_index.composability import ComposableGraph
   from llama_index.llms import OpenAI
   from llama_index.response.notebook_utils import display_response
Load Documents
   reader = SimpleDirectoryReader("../paul_graham_essay/data")
   documents = reader.load_data()
Parse into Nodes
   from llama_index.node_parser import SimpleNodeParser
   nodes = SimpleNodeParser.from_defaults().get_nodes_from_documents(documents)
Add to Docstore
   from llama_index.storage.kvstore.firestore_kvstore import FirestoreKVStore
   from llama_index.storage.docstore.firestore_docstore import FirestoreDocumentStore
   from llama_index.storage.index_store.firestore_indexstore import FirestoreIndexStore
   kvstore = FirestoreKVStore()
   storage_context = StorageContext.from_defaults(
       docstore=FirestoreDocumentStore(kvstore),
       index_store=FirestoreIndexStore(kvstore),
   )
   storage_context.docstore.add_documents(nodes)
Define Multiple Indexes
Each index uses the same underlying Node.
   summary_index = SummaryIndex(nodes, storage_context=storage_context)
   vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
   keyword_table_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)
   # NOTE: the docstore still has the same nodes
   len(storage_context.docstore.docs)
Test out saving and loading
   # NOTE: docstore and index_store is persisted in Firestore by default
   # NOTE: here only need to persist simple vector store to disk
   storage_context.persist()
   # note down index IDs
   list_id = summary_index.index_id
   vector_id = vector_index.index_id
   keyword_id = keyword_table_index.index_id
   from llama_index.indices.loading import load_index_from_storage
   kvstore = FirestoreKVStore()
   # re-create storage context
   storage_context = StorageContext.from_defaults(
       docstore=FirestoreDocumentStore(kvstore),
       index_store=FirestoreIndexStore(kvstore),
   )
   # load indices
   summary_index = load_index_from_storage(
       storage_context=storage_context, index_id=list_id
   )
   vector_index = load_index_from_storage(
       storage_context=storage_context, vector_id=vector_id
   )
   keyword_table_index = load_index_from_storage(
       storage_context=storage_context, keyword_id=keyword_id
   )
Test out some Queries
   chatgpt = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context_chatgpt = ServiceContext.from_defaults(llm=chatgpt, chunk_size=1024)
   query_engine = summary_index.as_query_engine()
   list_response = query_engine.query("What is a summary of this document?")
   display_response(list_response)
   query_engine = vector_index.as_query_engine()
   vector_response = query_engine.query("What did the author do growing up?")
   display_response(vector_response)
   query_engine = keyword_table_index.as_query_engine()
   keyword_response = query_engine.query("What did the author do after his time at YC?")
   display_response(keyword_response)


import nest_asyncio

   nest_asyncio.apply()
   import logging
   import sys
   import os
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       SimpleDirectoryReader,
       ServiceContext,
       LLMPredictor,
       StorageContext,
   )
   from llama_index import VectorStoreIndex, SummaryIndex, SimpleKeywordTableIndex
   from llama_index.composability import ComposableGraph
   from llama_index.llms import OpenAI
   from llama_index.response.notebook_utils import display_response
Load Documents
   reader = SimpleDirectoryReader("../paul_graham_essay/data")
   documents = reader.load_data()
Parse into Nodes
   from llama_index.node_parser import SimpleNodeParser
   nodes = SimpleNodeParser.from_defaults().get_nodes_from_documents(documents)
Add to Docstore
   MONGO_URI = os.environ["MONGO_URI"]
   from llama_index.storage.docstore import MongoDocumentStore
   from llama_index.storage.index_store.mongo_index_store import MongoIndexStore
   storage_context = StorageContext.from_defaults(
       docstore=MongoDocumentStore.from_uri(uri=MONGO_URI),
       index_store=MongoIndexStore.from_uri(uri=MONGO_URI),
   )
   storage_context.docstore.add_documents(nodes)
Define Multiple Indexes
Each index uses the same underlying Node.
   summary_index = SummaryIndex(nodes, storage_context=storage_context)
   vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
   keyword_table_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)
   # NOTE: the docstore still has the same nodes
   len(storage_context.docstore.docs)
Test out saving and loading
   # NOTE: docstore and index_store is persisted in MongoDB by default
   # NOTE: here only need to persist simple vector store to disk
   storage_context.persist()
   # note down index IDs
   list_id = summary_index.index_id
   vector_id = vector_index.index_id
   keyword_id = keyword_table_index.index_id
   from llama_index.indices.loading import load_index_from_storage
   # re-create storage context
   storage_context = StorageContext.from_defaults(
       docstore=MongoDocumentStore.from_uri(uri=MONGO_URI),
       index_store=MongoIndexStore.from_uri(uri=MONGO_URI),
   )
   # load indices
   summary_index = load_index_from_storage(
       storage_context=storage_context, index_id=list_id
   )
   vector_index = load_index_from_storage(
       storage_context=storage_context, vector_id=vector_id
   )
   keyword_table_index = load_index_from_storage(
       storage_context=storage_context, keyword_id=keyword_id
   )
Test out some Queries
   chatgpt = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context_chatgpt = ServiceContext.from_defaults(llm=chatgpt, chunk_size=1024)
   query_engine = summary_index.as_query_engine()
   list_response = query_engine.query("What is a summary of this document?")
   display_response(list_response)
   query_engine = vector_index.as_query_engine()
   vector_response = query_engine.query("What did the author do growing up?")
   display_response(vector_response)
   query_engine = keyword_table_index.as_query_engine()
   keyword_response = query_engine.query("What did the author do after his time at YC?")
   display_response(keyword_response)


import nest_asyncio

   nest_asyncio.apply()
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import SimpleDirectoryReader, ServiceContext, LLMPredictor
   from llama_index import VectorStoreIndex, SummaryIndex, SimpleKeywordTableIndex
   from llama_index.composability import ComposableGraph
   from llama_index.llms import OpenAI
Load Documents
   reader = SimpleDirectoryReader("../paul_graham_essay/data")
   documents = reader.load_data()
Parse into Nodes
   from llama_index.node_parser import SimpleNodeParser
   nodes = SimpleNodeParser.from_defaults().get_nodes_from_documents(documents)
Add to Docstore
   from llama_index.storage.docstore import SimpleDocumentStore
   docstore = SimpleDocumentStore()
   docstore.add_documents(nodes)
Define Multiple Indexes
Each index uses the same underlying Node.
   from llama_index.storage.storage_context import StorageContext
   storage_context = StorageContext.from_defaults(docstore=docstore)
   summary_index = SummaryIndex(nodes, storage_context=storage_context)
   vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
   keyword_table_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)
   # NOTE: the docstore sitll has the same nodes
   len(storage_context.docstore.docs)
   6
Test out some Queries
   llm = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context_chatgpt = ServiceContext.from_defaults(llm=llm, chunk_size=1024)
   WARNING:llama_index.llm_predictor.base:Unknown max input size for gpt-3.5-turbo, using defaults.
   Unknown max input size for gpt-3.5-turbo, using defaults.
   query_engine = summary_index.as_query_engine()
   response = query_engine.query("What is a summary of this document?")
   query_engine = vector_index.as_query_engine()
   response = query_engine.query("What did the author do growing up?")
   query_engine = keyword_table_index.as_query_engine()
   response = query_engine.query("What did the author do after his time at YC?")
   print(response)
   After his time at YC, the author decided to take a break and focus on painting. He spent most of 2014 painting and then, in November, he ran out of steam and stopped. He then moved to Florence, Italy to attend the Accademia di Belle Arti di Firenze, where he studied painting and drawing. He also started painting still lives in his bedroom at night. In March 2015, he started working on Lisp again and wrote a new Lisp, called Bel, in itself in Arc. He wrote essays through 2020, but also started to think about other things he could work on. He wrote an essay for himself to answer the question of how he should choose what to do next and then wrote a more detailed version for others to read. He also created the Y Combinator logo, which was an inside joke referencing the Viaweb logo, a white V on a red circle, so he made the YC logo a white Y on an orange square. He also created a fund for YC for a couple of years, but after Heroku got bought, he had enough money to go back to being self-funded. He also disliked the term "deal flow" because it implies that the number of new startups at any given time


Redis Docstore+Index Store Demo

   import nest_asyncio
   nest_asyncio.apply()
   import logging
   import sys
   import os
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       SimpleDirectoryReader,
       ServiceContext,
       LLMPredictor,
       StorageContext,
   )
   from llama_index import VectorStoreIndex, SummaryIndex, SimpleKeywordTableIndex
   from llama_index.composability import ComposableGraph
   from llama_index.llms import OpenAI
   from llama_index.response.notebook_utils import display_response
   INFO:numexpr.utils:Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   INFO:numexpr.utils:NumExpr defaulting to 8 threads.
   NumExpr defaulting to 8 threads.
   /home/loganm/miniconda3/envs/llama-index/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
Load Documents
   reader = SimpleDirectoryReader("../paul_graham_essay/data")
   documents = reader.load_data()
Parse into Nodes
   from llama_index.node_parser import SimpleNodeParser
   nodes = SimpleNodeParser.from_defaults().get_nodes_from_documents(documents)
Add to Docstore
   REDIS_HOST = os.getenv("REDIS_HOST", "127.0.0.1")
   REDIS_PORT = os.getenv("REDIS_PORT", 6379)
   from llama_index.storage.docstore import RedisDocumentStore
   from llama_index.storage.index_store import RedisIndexStore
   /home/loganm/miniconda3/envs/llama-index/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
   storage_context = StorageContext.from_defaults(
       docstore=RedisDocumentStore.from_host_and_port(
           host=REDIS_HOST, port=REDIS_PORT, namespace="llama_index"
       ),
       index_store=RedisIndexStore.from_host_and_port(
           host=REDIS_HOST, port=REDIS_PORT, namespace="llama_index"
       ),
   )
   storage_context.docstore.add_documents(nodes)
   len(storage_context.docstore.docs)
   20
Define Multiple Indexes
Each index uses the same underlying Node.
   summary_index = SummaryIndex(nodes, storage_context=storage_context)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens
   > [build_index_from_nodes] Total embedding token usage: 0 tokens
   vector_index = VectorStoreIndex(nodes, storage_context=storage_context)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 17050 tokens
   > [build_index_from_nodes] Total embedding token usage: 17050 tokens

Redis Docstore+Index Store Demo

   keyword_table_index = SimpleKeywordTableIndex(nodes, storage_context=storage_context)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens
   > [build_index_from_nodes] Total embedding token usage: 0 tokens
   # NOTE: the docstore still has the same nodes
   len(storage_context.docstore.docs)
   20
Test out saving and loading
   # NOTE: docstore and index_store is persisted in Redis by default
   # NOTE: here only need to persist simple vector store to disk
   storage_context.persist(persist_dir="./storage")
   # note down index IDs
   list_id = summary_index.index_id
   vector_id = vector_index.index_id
   keyword_id = keyword_table_index.index_id
   from llama_index.indices.loading import load_index_from_storage
   # re-create storage context
   storage_context = StorageContext.from_defaults(
       docstore=RedisDocumentStore.from_host_and_port(
           host=REDIS_HOST, port=REDIS_PORT, namespace="llama_index"
       ),
       index_store=RedisIndexStore.from_host_and_port(
           host=REDIS_HOST, port=REDIS_PORT, namespace="llama_index"
       ),
   )
   # load indices
   summary_index = load_index_from_storage(
       storage_context=storage_context, index_id=list_id
   )
   vector_index = load_index_from_storage(
       storage_context=storage_context, index_id=vector_id
   )
   keyword_table_index = load_index_from_storage(
       storage_context=storage_context, index_id=keyword_id
   )
   INFO:llama_index.indices.loading:Loading indices with ids: ['24e98f9b-9586-4fc6-8341-8dce895e5bcc']
   Loading indices with ids: ['24e98f9b-9586-4fc6-8341-8dce895e5bcc']
   INFO:llama_index.indices.loading:Loading indices with ids: ['f7b2aeb3-4dad-4750-8177-78d5ae706284']
   Loading indices with ids: ['f7b2aeb3-4dad-4750-8177-78d5ae706284']
   INFO:llama_index.indices.loading:Loading indices with ids: ['9a9198b4-7cb9-4c96-97a7-5f404f43b9cd']
   Loading indices with ids: ['9a9198b4-7cb9-4c96-97a7-5f404f43b9cd']
Test out some Queries
   chatgpt = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context_chatgpt = ServiceContext.from_defaults(llm=chatgpt, chunk_size=1024)
   query_engine = summary_index.as_query_engine()
   list_response = query_engine.query("What is a summary of this document?")
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 26111 tokens
   > [get_response] Total LLM token usage: 26111 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   display_response(list_response)
**"Final Response:"** This document is a narrative of the author's
journey from writing and programming as a young person to pursuing a
career in art. It describes his experiences in high school, college,

Redis Docstore+Index Store Demo

and graduate school, and how he eventually decided to pursue art as a
career. He applied to art schools and eventually was accepted to RISD
and the Accademia di Belli Arti in Florence. He passed the entrance
exam for the Accademia and began studying art there. He then moved to
New York and worked freelance while writing a book on Lisp. He
eventually started a company to put art galleries online, but it was
unsuccessful. He then pivoted to creating software to build online
stores, which eventually became successful. He had the idea to run the
software on the server and let users control it by clicking on links,
which meant users wouldn't need anything more than a browser. This
kind of software, known as "internet storefronts," was eventually
successful. He and his team worked hard to make the software user-
friendly and inexpensive, and eventually the company was bought by
Yahoo. After the sale, he left to pursue his dream of painting, and
eventually found success in New York. He was able to afford luxuries
such as taxis and restaurants, and he experimented with a new kind of
still life painting. He also had the idea to create a web app for
making web apps, which he eventually pursued and was successful with.
He then started Y Combinator, an investment firm that focused on
helping startups, with his own money and the help of his friends
Robert and Trevor. He wrote essays and books, invited undergrads to
apply to the Summer Founders Program, and eventually married Jessica
Livingston. After his mother's death, he decided to quit Y Combinator
and pursue painting, but eventually ran out of steam and started
writing essays and working on Lisp again. He wrote a new Lisp, called
Bel, in itself in Arc, and it took him four years to complete. During
this time, he worked hard to make the language user-friendly and
precise, and he also took time to enjoy life with his family. He
encountered various obstacles along the way, such as customs that
constrained him even after the restrictions that caused them had
disappeared, and he also had to deal with misinterpretations of his
essays on forums. In the end, he was successful in creating Bel and
was able to pursue his dream of painting.
   query_engine = vector_index.as_query_engine()
   vector_response = query_engine.query("What did the author do growing up?")
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 8 tokens
   > [retrieve] Total embedding token usage: 8 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 0 tokens
   > [get_response] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   display_response(vector_response)
**"Final Response:"** None
   query_engine = keyword_table_index.as_query_engine()
   keyword_response = query_engine.query("What did the author do after his time at YC?")
   INFO:llama_index.indices.keyword_table.retrievers:> Starting query: What did the author do after his time at YC?
   > Starting query: What did the author do after his time at YC?
   INFO:llama_index.indices.keyword_table.retrievers:query keywords: ['action', 'yc', 'after', 'time', 'author']
   query keywords: ['action', 'yc', 'after', 'time', 'author']

Redis Docstore+Index Store Demo

   INFO:llama_index.indices.keyword_table.retrievers:> Extracted keywords: ['yc', 'time']
   > Extracted keywords: ['yc', 'time']
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 10216 tokens
   > [get_response] Total LLM token usage: 10216 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   display_response(keyword_response)
**"Final Response:"** After his time at YC, the author decided to
pursue painting and writing. He wanted to see how good he could get if
he really focused on it, so he started painting the day after he
stopped working on YC. He spent most of the rest of 2014 painting and
was able to become better than he had been before. He also wrote
essays and started working on Lisp again in March 2015. He then spent
4 years working on a new Lisp, called Bel, which he wrote in itself in
Arc. He had to ban himself from writing essays during most of this
time, and he moved to England in the summer of 2016. He also wrote a
book about Lisp hacking, called On Lisp, which was published in 1993.
In the fall of 2019, Bel was finally finished. He also experimented
with a new kind of still life painting, and tried to build a web app
for making web apps, which he named Aspra. He eventually decided to
build a subset of this app as an open source project, which was the
new Lisp dialect he called Arc.


LongContextReorder

Models struggle to access significant details found in the center of
extended contexts. A study observed that the best performance
typically arises when crucial data is positioned at the start or
conclusion of the input context. Additionally, as the input context
lengthens, performance drops notably, even in models designed for long
contexts.
This module will re-order the retrieved nodes, which can be helpful in
cases where a large top-k is needed.
Setup
   import os
   import openai
   os.environ["OPENAI_API_KEY"] = "sk-..."
   openai.api_key = os.environ["OPENAI_API_KEY"]
   from llama_index import ServiceContext
   from llama_index.llms import OpenAI
   llm = OpenAI(model="gpt-3.5-turbo-instruct", temperature=0.1)
   ctx = ServiceContext.from_defaults(llm=llm, embed_model="local:BAAI/bge-base-en-v1.5")
   /home/loganm/miniconda3/envs/llama-index/lib/python3.11/site-packages/torch/cuda/__init__.py:546: UserWarning: Can't initialize NVML
     warnings.warn("Can't initialize NVML")
   from llama_index import SimpleDirectoryReader
   documents = SimpleDirectoryReader("../data/paul_graham/").load_data()
   from llama_index import VectorStoreIndex
   index = VectorStoreIndex.from_documents(documents, service_context=ctx)
Run Query
   from llama_index.indices.postprocessor import LongContextReorder
   reorder = LongContextReorder()
   reorder_engine = index.as_query_engine(
       node_postprocessors=[reorder], similarity_top_k=5
   )
   base_engine = index.as_query_engine(similarity_top_k=5)
   from llama_index.response.notebook_utils import display_response
   base_response = base_engine.query("Did the author meet Sam Altman?")
   display_response(base_response)
**"Final Response:"** Yes, the author met Sam Altman when they asked
him to be the president of Y Combinator. This was during the time when
the author was in a PhD program in computer science and also pursuing
their passion for art. They were applying to art schools and
eventually ended up attending RISD.
   reorder_response = reorder_engine.query("Did the author meet Sam Altman?")
   display_response(reorder_response)
**"Final Response:"** Yes, the author met Sam Altman when they asked
him to be the president of Y Combinator. This meeting occurred at a
party at the author's house, where they were introduced by a mutual
friend, Jessica Livingston. Jessica later went on to compile a book of
interviews with startup founders, and the author shared their thoughts
on the flaws of venture capital with her during her job search at a
Boston VC firm.
Inspect Order Diffrences
   print(base_response.get_formatted_sources())
   > Source (Doc id: 81bc66bb-2c45-4697-9f08-9f848bd78b12): [17]
   As well as HN, I wrote all of YC's internal software in Arc. But while I continued to work ...
   > Source (Doc id: bd660905-e4e0-4d02-a113-e3810b59c5d1): [19] One way to get more precise about the concept of invented vs discovered is to talk about spa...
   > Source (Doc id: 3932e4a4-f17e-4dd2-9d25-5f0e65910dc5): Not so much because it was badly written as because the problem is so convoluted. When you're wor...
   > Source (Doc id: 0d801f0a-4a99-475d-aa7c-ad5d601947ea): [10]

LongContextReorder

   Wow, I thought, there's an audience. If I write something and put it on the web, anyone can...
   > Source (Doc id: bf726802-4d0d-4ee5-ab2e-ffa8a5461bc4): I was briefly tempted, but they were so slow by present standards; what was the point? No one els...
   print(reorder_response.get_formatted_sources())
   > Source (Doc id: 81bc66bb-2c45-4697-9f08-9f848bd78b12): [17]
   As well as HN, I wrote all of YC's internal software in Arc. But while I continued to work ...
   > Source (Doc id: 3932e4a4-f17e-4dd2-9d25-5f0e65910dc5): Not so much because it was badly written as because the problem is so convoluted. When you're wor...
   > Source (Doc id: bf726802-4d0d-4ee5-ab2e-ffa8a5461bc4): I was briefly tempted, but they were so slow by present standards; what was the point? No one els...
   > Source (Doc id: 0d801f0a-4a99-475d-aa7c-ad5d601947ea): [10]
   Wow, I thought, there's an audience. If I write something and put it on the web, anyone can...
   > Source (Doc id: bd660905-e4e0-4d02-a113-e3810b59c5d1): [19] One way to get more precise about the concept of invented vs discovered is to talk about spa...


File Based Node Parsers

The combination of the "SimpleFileNodeParser" and "FlatReader" are
designed to allow opening a variety of file types and automatically
selecting the best NodeParser to process the files. The "FlatReader"
loads the file in a raw text format and attaches the file information
to the metadata, then the "SimpleFileNodeParser" maps file types to
node parsers in "node_parser/file", selecting the best node parser for
the job.
The "SimpleFileNodeParser" does not perform token based chunking of
the text, and one of the other node parsers, in particular ones that
accept an instance of a "TextSplitter", can be chained to further
split the content.
Let's look at an example of using the "FlatReader" and
"SimpleFileNodeParser" to load content. For the README file I will be
using the LlamaIndex README and the HTML file is the Stack Overflow
landing page, however any README and HTML file will work.
   from llama_index.node_parser.simple_file import SimpleFileNodeParser
   from llama_index.readers.file.flat_reader import FlatReader
   from pathlib import Path
   /Users/adamhofmann/opt/anaconda3/lib/python3.9/site-packages/langchain/__init__.py:24: UserWarning: Importing BasePromptTemplate from langchain root module is no longer supported.
     warnings.warn(
   /Users/adamhofmann/opt/anaconda3/lib/python3.9/site-packages/langchain/__init__.py:24: UserWarning: Importing PromptTemplate from langchain root module is no longer supported.
     warnings.warn(
   reader = FlatReader()
   html_file = reader.load_data(Path("./stack-overflow.html"))
   md_file = reader.load_data(Path("./README.md"))
   print(html_file[0].metadata)
   print(html_file[0])
   print("----")
   print(md_file[0].metadata)
   print(md_file[0])
   {'filename': 'stack-overflow.html', 'extension': '.html'}
   Doc ID: a6750408-b0fa-466d-be28-ff2fcbcbaa97
   Text: <!DOCTYPE html>       <html class="html__responsive
   html__unpinned-leftnav" lang="en">      <head>          <title>Stack
   Overflow - Where Developers Learn, Share, & Build Careers</title>
   <link rel="shortcut icon" href="https://cdn.sstatic.net/Sites/stackove
   rflow/Img/favicon.ico?v=ec617d715196">         <link rel="apple-touch-
   icon" hr...
   ----
   {'filename': 'README.md', 'extension': '.md'}
   Doc ID: 1d872f44-2bb3-4693-a1b8-a59392c23be2
   Text: # 🗂 LlamaIndex 🦙 [![PyPI -
   Downloads](https://img.shields.io/pypi/dm/llama-
   index)](https://pypi.org/project/llama-index/) [![GitHub contributors]
   (https://img.shields.io/github/contributors/jerryjliu/llama_index)](ht
   tps://github.com/jerryjliu/llama_index/graphs/contributors) [![Discord
   ](https://img.shields.io/discord/1059199217496772688)](https:...
Parsing the files
The flat reader has simple loaded the content of the files into
Document objects for further processing. We can see that the file
information is retained in the metadata. Let's pass the documents to
the node parser to see the parsing.
   parser = SimpleFileNodeParser()
   md_nodes = parser.get_nodes_from_documents(md_file)

File Based Node Parsers

   html_nodes = parser.get_nodes_from_documents(html_file)
   print(md_nodes[0].metadata)
   print(md_nodes[0].text)
   print(md_nodes[1].metadata)
   print(md_nodes[1].text)
   print("----")
   print(html_nodes[0].metadata)
   print(html_nodes[0].text)
   {'filename': 'README.md', 'extension': '.md', 'Header 1': '🗂 LlamaIndex 🦙'}
   🗂 LlamaIndex 🦙
   [![PyPI - Downloads](https://img.shields.io/pypi/dm/llama-index)](https://pypi.org/project/llama-index/)
   [![GitHub contributors](https://img.shields.io/github/contributors/jerryjliu/llama_index)](https://github.com/jerryjliu/llama_index/graphs/contributors)
   [![Discord](https://img.shields.io/discord/1059199217496772688)](https://discord.gg/dGcwcsnxhU)
   LlamaIndex (GPT Index) is a data framework for your LLM application.
   PyPI:
   - LlamaIndex: https://pypi.org/project/llama-index/.
   - GPT Index (duplicate): https://pypi.org/project/gpt-index/.
   LlamaIndex.TS (Typescript/Javascript): https://github.com/run-llama/LlamaIndexTS.
   Documentation: https://gpt-index.readthedocs.io/.
   Twitter: https://twitter.com/llama_index.
   Discord: https://discord.gg/dGcwcsnxhU.
   {'filename': 'README.md', 'extension': '.md', 'Header 1': '🗂 LlamaIndex 🦙', 'Header 3': 'Ecosystem'}
   Ecosystem
   - LlamaHub (community library of data loaders): https://llamahub.ai
   - LlamaLab (cutting-edge AGI projects using LlamaIndex): https://github.com/run-llama/llama-lab
   ----
   {'filename': 'stack-overflow.html', 'extension': '.html', 'tag': 'li'}
   About
   Products
   For Teams
   Stack Overflow
   Public questions & answers
   Stack Overflow for Teams
   Where developers & technologists share private knowledge with coworkers
   Talent
   								Build your employer brand
   Advertising
   Reach developers & technologists worldwide
   Labs
   The future of collective knowledge sharing
   About the company
   current community
               Stack Overflow
   help
   chat
               Meta Stack Overflow
   your communities
   Sign up or log in to customize your list.
   more stack exchange communities
   company blog
Furter processing of files
We can see that the Markdown and HTML files have been split into
chunks based on the structure of the document. The markdown node
parser splits on any headers and attaches the hierarchy of headers
into metadata. The HTML node parser extracted text from common text
elements to simplifiy the HTML file, and combines neighbouring nodes
of the same element. Compared to working with raw HTML, this is
alreadly a big improvement in terms of retrieving meaningful text
content.
Because these files were only split according to the structure of the
file, we can apply further processing with a text splitter to prepare
the content into nodes of limited token length.
   from llama_index.node_parser import SimpleNodeParser
   from llama_index.text_splitter import SentenceSplitter
   # For clarity in the demo, make small splits without overlap
   splitter = SentenceSplitter(chunk_size=200, chunk_overlap=0)
   splitting_parser = SimpleNodeParser(text_splitter=splitter)

File Based Node Parsers

   html_chunked_nodes = splitting_parser.get_nodes_from_documents(html_nodes)
   md_chunked_nodes = splitting_parser.get_nodes_from_documents(md_nodes)
   print(f"\n\nHTML parsed nodes: {len(html_nodes)}")
   print(html_nodes[0].text)
   print(f"\n\nHTML chunked nodes: {len(html_chunked_nodes)}")
   print(html_chunked_nodes[0].text)
   print(f"\n\nMD parsed nodes: {len(md_nodes)}")
   print(md_nodes[0].text)
   print(f"\n\nMD chunked nodes: {len(md_chunked_nodes)}")
   print(md_chunked_nodes[0].text)
   HTML parsed nodes: 67
   About
   Products
   For Teams
   Stack Overflow
   Public questions & answers
   Stack Overflow for Teams
   Where developers & technologists share private knowledge with coworkers
   Talent
   								Build your employer brand
   Advertising
   Reach developers & technologists worldwide
   Labs
   The future of collective knowledge sharing
   About the company
   current community
               Stack Overflow
   help
   chat
               Meta Stack Overflow
   your communities
   Sign up or log in to customize your list.
   more stack exchange communities
   company blog
   HTML chunked nodes: 87
   About
   Products
   For Teams
   Stack Overflow
   Public questions & answers
   Stack Overflow for Teams
   Where developers & technologists share private knowledge with coworkers
   Talent
   								Build your employer brand
   Advertising
   Reach developers & technologists worldwide
   Labs
   The future of collective knowledge sharing
   About the company
   current community
               Stack Overflow
   help
   chat
               Meta Stack Overflow
   your communities
   MD parsed nodes: 10
   🗂 LlamaIndex 🦙
   [![PyPI - Downloads](https://img.shields.io/pypi/dm/llama-index)](https://pypi.org/project/llama-index/)
   [![GitHub contributors](https://img.shields.io/github/contributors/jerryjliu/llama_index)](https://github.com/jerryjliu/llama_index/graphs/contributors)
   [![Discord](https://img.shields.io/discord/1059199217496772688)](https://discord.gg/dGcwcsnxhU)
   LlamaIndex (GPT Index) is a data framework for your LLM application.
   PyPI:
   - LlamaIndex: https://pypi.org/project/llama-index/.
   - GPT Index (duplicate): https://pypi.org/project/gpt-index/.
   LlamaIndex.TS (Typescript/Javascript): https://github.com/run-llama/LlamaIndexTS.
   Documentation: https://gpt-index.readthedocs.io/.
   Twitter: https://twitter.com/llama_index.
   Discord: https://discord.gg/dGcwcsnxhU.
   MD chunked nodes: 13
   🗂 LlamaIndex 🦙
   [![PyPI - Downloads](https://img.shields.io/pypi/dm/llama-index)](https://pypi.org/project/llama-index/)
   [![GitHub contributors](https://img.shields.io/github/contributors/jerryjliu/llama_index)](https://github.com/jerryjliu/llama_index/graphs/contributors)
   [![Discord](https://img.shields.io/discord/1059199217496772688)](https://discord.gg/dGcwcsnxhU)
Summary
We can see that the files have been further processed within the
splits created by "SimpleFileNodeParser", and are now ready to be

File Based Node Parsers

[TextNode(id_='e6236169-45a1-4699-9762-c8d3d89f8fa0', embedding=None, metadata={'filename': 'README.md', 'extension': '.md', 'Header 1': '🗂 LlamaIndex 🦙'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='e7bc328f-85c1-430a-9772-425e59909a58', node_type=None, metadata={'filename': 'README.md', 'extension': '.md', 'Header 1': '🗂 LlamaIndex 🦙'}, hash='e538ad7c04f635f1c707eba290b55618a9f0942211c4b5ca2a4e54e1fdf04973'), <NodeRelationship.NEXT: '3'>: RelatedNodeInfo(node_id='51b40b54-dfd3-48ed-b377-5ca58a0f48a3', node_type=None, metadata={'filename': 'README.md', 'extension': '.md', 'Header 1': '🗂 LlamaIndex 🦙'}, hash='ca9e3590b951f1fca38687fd12bb43fbccd0133a38020c94800586b3579c3218')}, hash='ec733c85ad1dca248ae583ece341428ee20e4d796bc11adea1618c8e4ed9246a', text='🗂 LlamaIndex 🦙\n[![PyPI - Downloads](https://img.shields.io/pypi/dm/llama-index)](https://pypi.org/project/llama-index/)\n[![GitHub contributors](https://img.shields.io/github/contributors/jerryjliu/llama_index)](https://github.com/jerryjliu/llama_index/graphs/contributors)\n[![Discord](https://img.shields.io/discord/1059199217496772688)](https://discord.gg/dGcwcsnxhU)', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), TextNode(id_='51b40b54-dfd3-48ed-b377-5ca58a0f48a3', embedding=None, metadata={'filename': 'README.md', 'extension': '.md', 'Header 1': '🗂 LlamaIndex 🦙'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='e7bc328f-85c1-430a-9772-425e59909a58', node_type=None, metadata={'filename': 'README.md', 'extension': '.md', 'Header 1': '🗂 LlamaIndex 🦙'}, hash='e538ad7c04f635f1c707eba290b55618a9f0942211c4b5ca2a4e54e1fdf04973'), <NodeRelationship.PREVIOUS: '2'>: RelatedNodeInfo(node_id='e6236169-45a1-4699-9762-c8d3d89f8fa0', node_type=None, metadata={'filename': 'README.md', 'extension': '.md', 'Header 1': '🗂 LlamaIndex 🦙'}, hash='ec733c85ad1dca248ae583ece341428ee20e4d796bc11adea1618c8e4ed9246a')}, hash='ca9e3590b951f1fca38687fd12bb43fbccd0133a38020c94800586b3579c3218', text='LlamaIndex (GPT Index) is a data framework for your LLM application.\n\nPyPI: \n- LlamaIndex: https://pypi.org/project/llama-index/.\n- GPT Index (duplicate): https://pypi.org/project/gpt-index/.\n\nLlamaIndex.TS (Typescript/Javascript): https://github.com/run-llama/LlamaIndexTS.\n\nDocumentation: https://gpt-index.readthedocs.io/.\n\nTwitter: https://twitter.com/llama_index.\n\nDiscord: https://discord.gg/dGcwcsnxhU.', start_char_idx=None, end_char_idx=None, text_template='{metadata_str}\n\n{content}', metadata_template='{key}: {value}', metadata_seperator='\n'), TextNode(id_='ce269047-4718-4a08-b170-34fef19cdafe', embedding=None, metadata={'filename': 'README.md', 'extension': '.md', 'Header 1': '🗂 LlamaIndex 🦙', 'Header 3': 'Ecosystem'}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={<NodeRelationship.SOURCE: '1'>: RelatedNodeInfo(node_id='953934dc-dd4f-4069-9e2a-326ee8a593bf', node_type=None, metadata=

File Based Node Parsers

ingested by an index or vector store. The code cell below shows just
the chaining of the parsers to go from raw file to chunked nodes:
   md_chunked_nodes = splitting_parser.get_nodes_from_documents(
       parser.get_nodes_from_documents(reader.load_data(Path("./README.md")))
   )
   print(md_chunked_nodes)


LLM Reranker Demonstration (2021 Lyft 10-k)

This tutorial showcases how to do a two-stage pass for retrieval. Use
embedding-based retrieval with a high top-k value in order to maximize
recall and get a large set of candidate items. Then, use LLM-based
retrieval to dynamically select the nodes that are actually relevant
to the query.
   import nest_asyncio
   nest_asyncio.apply()
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SimpleDirectoryReader,
       ServiceContext,
       LLMPredictor,
   )
   from llama_index.indices.postprocessor import LLMRerank
   from llama_index.llms import OpenAI
   from IPython.display import Markdown, display
Load Data, Build Index
   # LLM Predictor (gpt-3.5-turbo) + service context
   llm = OpenAI(temperature=0, model="gpt-3.5-turbo")
   chunk_overlap = 0
   chunk_size = 128
   service_context = ServiceContext.from_defaults(
       llm=llm,
       chunk_size=chunk_size,
       chunk_overlap=chunk_overlap,
   )
   # load documents
   documents = SimpleDirectoryReader(input_files=["lyft_10k.pdf"]).load_data()
   index = VectorStoreIndex.from_documents(documents, service_context=service_context)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 226241 tokens
   > [build_index_from_nodes] Total embedding token usage: 226241 tokens
   > [build_index_from_nodes] Total embedding token usage: 226241 tokens
Retrieval Comparisons
   from llama_index.retrievers import VectorIndexRetriever
   from llama_index.indices.query.schema import QueryBundle
   import pandas as pd
   from IPython.display import display, HTML
   from copy import deepcopy
   pd.set_option("display.max_colwidth", -1)
   def get_retrieved_nodes(
       query_str, vector_top_k=10, reranker_top_n=3, with_reranker=False
   ):
       query_bundle = QueryBundle(query_str)
       # configure retriever
       retriever = VectorIndexRetriever(
           index=index,
           similarity_top_k=vector_top_k,
       )
       retrieved_nodes = retriever.retrieve(query_bundle)
       if with_reranker:
           # configure reranker
           reranker = LLMRerank(
               choice_batch_size=5, top_n=reranker_top_n, service_context=service_context
           )
           retrieved_nodes = reranker.postprocess_nodes(retrieved_nodes, query_bundle)
       return retrieved_nodes
   def pretty_print(df):
       return display(HTML(df.to_html().replace("\\n", "<br>")))
   def visualize_retrieved_nodes(nodes) -> None:
       result_dicts = []
       for node in nodes:
           node = deepcopy(node)
           node.node.metadata = None
           node_text = node.node.get_text()
           node_text = node_text.replace("\n", " ")
           result_dict = {"Score": node.score, "Text": node_text}
           result_dicts.append(result_dict)
       pretty_print(pd.DataFrame(result_dicts))
   /var/folders/1r/c3h91d9s49xblwfvz79s78_c0000gn/T/ipykernel_58458/2502541873.py:8: FutureWarning: Passing a negative integer is deprecated in version 1.0 and will not be supported in future version. Instead, use None to not limit the column width.

     pd.set_option('display.max_colwidth', -1)
   new_nodes = get_retrieved_nodes(
       "What is Lyft's response to COVID-19?", vector_top_k=5, with_reranker=False
   )
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 11 tokens
   > [retrieve] Total embedding token usage: 11 tokens
   > [retrieve] Total embedding token usage: 11 tokens
   > [retrieve] Total embedding token usage: 11 tokens
   > [retrieve] Total embedding token usage: 11 tokens
   visualize_retrieved_nodes(new_nodes)
   <IPython.core.display.HTML object>
   new_nodes = get_retrieved_nodes(
       "What is Lyft's response to COVID-19?",
       vector_top_k=20,
       reranker_top_n=5,
       with_reranker=True,
   )
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 11 tokens
   > [retrieve] Total embedding token usage: 11 tokens
   > [retrieve] Total embedding token usage: 11 tokens
   > [retrieve] Total embedding token usage: 11 tokens
   > [retrieve] Total embedding token usage: 11 tokens
   visualize_retrieved_nodes(new_nodes)
   <IPython.core.display.HTML object>
   new_nodes = get_retrieved_nodes(
       "What initiatives are the company focusing on independently of COVID-19?",
       vector_top_k=5,
       with_reranker=False,
   )
   visualize_retrieved_nodes(new_nodes)
   <IPython.core.display.HTML object>
   new_nodes = get_retrieved_nodes(
       "What initiatives are the company focusing on independently of COVID-19?",
       vector_top_k=40,
       reranker_top_n=5,
       with_reranker=True,
   )
   visualize_retrieved_nodes(new_nodes)
   <IPython.core.display.HTML object>


PII Masking

   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index.indices.postprocessor import (
       PIINodePostprocessor,
       NERPIINodePostprocessor,
   )
   from llama_index.llms import HuggingFaceLLM
   from llama_index import ServiceContext, Document, VectorStoreIndex
   from llama_index.schema import TextNode
   INFO:numexpr.utils:Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   Note: NumExpr detected 16 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   INFO:numexpr.utils:NumExpr defaulting to 8 threads.
   NumExpr defaulting to 8 threads.
   /home/loganm/miniconda3/envs/llama-index/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
   # load documents
   text = """
   Hello Paulo Santos. The latest statement for your credit card account \
   1111-0000-1111-0000 was mailed to 123 Any Street, Seattle, WA 98109.
   """
   node = TextNode(text=text)
Option 1: Use NER Model for PII Masking
Use a Hugging Face NER model for PII Masking
   service_context = ServiceContext.from_defaults()
   processor = NERPIINodePostprocessor(service_context=service_context)
   from llama_index.schema import NodeWithScore
   new_nodes = processor.postprocess_nodes([NodeWithScore(node=node)])
   No model was supplied, defaulted to dbmdz/bert-large-cased-finetuned-conll03-english and revision f2482bf (https://huggingface.co/dbmdz/bert-large-cased-finetuned-conll03-english).
   Using a pipeline without specifying a model name and revision in production is not recommended.
   /home/loganm/miniconda3/envs/llama-index/lib/python3.11/site-packages/transformers/pipelines/token_classification.py:169: UserWarning: `grouped_entities` is deprecated and will be removed in version v5.0.0, defaulted to `aggregation_strategy="AggregationStrategy.SIMPLE"` instead.
     warnings.warn(
   # view redacted text
   new_nodes[0].node.get_text()
   'Hello [ORG_6]. The latest statement for your credit card account 1111-0000-1111-0000 was mailed to 123 [ORG_108] [LOC_112], [LOC_120], [LOC_129] 98109.'
   # get mapping in metadata
   # NOTE: this is not sent to the LLM!
   new_nodes[0].node.metadata["__pii_node_info__"]
   {'[ORG_6]': 'Paulo Santos',
    '[ORG_108]': 'Any',
    '[LOC_112]': 'Street',
    '[LOC_120]': 'Seattle',
    '[LOC_129]': 'WA'}
Option 2: Use LLM for PII Masking
NOTE: You should be using a *local* LLM model for PII masking. The
example shown is using OpenAI, but normally you'd use an LLM running
locally, possibly from huggingface. Examples for local LLMs are here.
   service_context = ServiceContext.from_defaults()
   processor = PIINodePostprocessor(service_context=service_context)
   from llama_index.schema import NodeWithScore

PII Masking

   new_nodes = processor.postprocess_nodes([NodeWithScore(node=node)])
   # view redacted text
   new_nodes[0].node.get_text()
   'Hello [NAME]. The latest statement for your credit card account [CREDIT_CARD_NUMBER] was mailed to [ADDRESS].'
   # get mapping in metadata
   # NOTE: this is not sent to the LLM!
   new_nodes[0].node.metadata["__pii_node_info__"]
   {'NAME': 'Paulo Santos',
    'CREDIT_CARD_NUMBER': '1111-0000-1111-0000',
    'ADDRESS': '123 Any Street, Seattle, WA 98109'}
Feed Nodes to Index
   # feed into index
   index = VectorStoreIndex([n.node for n in new_nodes])
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 30 tokens
   > [build_index_from_nodes] Total embedding token usage: 30 tokens
   response = index.as_query_engine().query("What address was the statement mailed to?")
   print(str(response))
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 8 tokens
   > [retrieve] Total embedding token usage: 8 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total LLM token usage: 71 tokens
   > [get_response] Total LLM token usage: 71 tokens
   INFO:llama_index.token_counter.token_counter:> [get_response] Total embedding token usage: 0 tokens
   > [get_response] Total embedding token usage: 0 tokens
   [ADDRESS]


Time-Weighted Rerank

Showcase capabilities of time-weighted node postprocessor
   from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
   from llama_index.indices.postprocessor import (
       TimeWeightedPostprocessor,
   )
   from llama_index.node_parser import SimpleNodeParser
   from llama_index.storage.docstore import SimpleDocumentStore
   from llama_index.response.notebook_utils import display_response
   from datetime import datetime, timedelta
   /home/loganm/miniconda3/envs/llama-index/lib/python3.11/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
Parse Documents into Nodes, add to Docstore
In this example, there are 3 different versions of PG's essay. They
are largely identical **except** for one specific section, which
details the amount of funding they raised for Viaweb.
V1: 50k, V2: 30k, V3: 10K
V1: -1 day, V2: -2 days, V3: -3 days
The idea is to encourage index to fetch the most recent info (which is
V3)
   # load documents
   from llama_index.storage.storage_context import StorageContext
   now = datetime.now()
   key = "__last_accessed__"
   doc1 = SimpleDirectoryReader(
       input_files=["./test_versioned_data/paul_graham_essay_v1.txt"]
   ).load_data()[0]
   doc2 = SimpleDirectoryReader(
       input_files=["./test_versioned_data/paul_graham_essay_v2.txt"]
   ).load_data()[0]
   doc3 = SimpleDirectoryReader(
       input_files=["./test_versioned_data/paul_graham_essay_v3.txt"]
   ).load_data()[0]
   # define service context (wrapper container around current classes)
   service_context = ServiceContext.from_defaults(chunk_size=512)
   node_parser = service_context.node_parser
   # use node parser in service context to parse docs into nodes
   nodes1 = node_parser.get_nodes_from_documents([doc1])
   nodes2 = node_parser.get_nodes_from_documents([doc2])
   nodes3 = node_parser.get_nodes_from_documents([doc3])
   # fetch the modified chunk from each document, set metadata
   # also exclude the date from being read by the LLM
   nodes1[14].metadata[key] = (now - timedelta(hours=3)).timestamp()
   nodes1[14].excluded_llm_metadata_keys = [key]
   nodes2[14].metadata[key] = (now - timedelta(hours=2)).timestamp()
   nodes2[14].excluded_llm_metadata_keys = [key]
   nodes3[14].metadata[key] = (now - timedelta(hours=1)).timestamp()
   nodes2[14].excluded_llm_metadata_keys = [key]
   # add to docstore
   docstore = SimpleDocumentStore()
   nodes = [nodes1[14], nodes2[14], nodes3[14]]
   docstore.add_documents(nodes)
   storage_context = StorageContext.from_defaults(docstore=docstore)
Build Index
   # build index
   index = VectorStoreIndex(nodes, storage_context=storage_context)
Define Recency Postprocessors
   node_postprocessor = TimeWeightedPostprocessor(
       time_decay=0.5, time_access_refresh=False, top_k=1
   )
Query Index
   # naive query
   query_engine = index.as_query_engine(
       similarity_top_k=3,
   )
   response = query_engine.query(
       "How much did the author raise in seed funding from Idelle's husband (Julian) for Viaweb?",

Time-Weighted Rerank

   )
   display_response(response)
**"Final Response:"** $50,000
   # query using time weighted node postprocessor
   query_engine = index.as_query_engine(
       similarity_top_k=3, node_postprocessors=[node_postprocessor]
   )
   response = query_engine.query(
       "How much did the author raise in seed funding from Idelle's husband (Julian) for Viaweb?",
   )
   display_response(response)
**"Final Response:"** The author raised $10,000 in seed funding from
Idelle's husband (Julian) for Viaweb.
Query Index (Lower-Level Usage)
In this example we first get the full set of nodes from a query call,
and then send to node postprocessor, and then finally synthesize
response through a summary index.
   from llama_index import SummaryIndex
   query_str = "How much did the author raise in seed funding from Idelle's husband (Julian) for Viaweb?"
   query_engine = index.as_query_engine(similarity_top_k=3, response_mode="no_text")
   init_response = query_engine.query(
       query_str,
   )
   resp_nodes = [n for n in init_response.source_nodes]
   # get the post-processed nodes -- which should be the top-1 sorted by date
   new_resp_nodes = node_postprocessor.postprocess_nodes(resp_nodes)
   summary_index = SummaryIndex([n.node for n in new_resp_nodes])
   query_engine = summary_index.as_query_engine()
   response = query_engine.query(query_str)
   display_response(response)
**"Final Response:"** The author raised $10,000 in seed funding from
Idelle's husband (Julian) for Viaweb.


Forward/Backward Augmentation

Showcase capabilities of leveraging Node relationships on top of PG's
essay
   from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
   from llama_index.indices.postprocessor import (
       PrevNextNodePostprocessor,
       AutoPrevNextNodePostprocessor,
   )
   from llama_index.node_parser import SimpleNodeParser
   from llama_index.storage.docstore import SimpleDocumentStore
Parse Documents into Nodes, add to Docstore
   # load documents
   from llama_index.storage.storage_context import StorageContext
   documents = SimpleDirectoryReader("../data/paul_graham").load_data()
   # define service context (wrapper container around current classes)
   service_context = ServiceContext.from_defaults(chunk_size=512)
   # use node parser in service context to parse into nodes
   nodes = service_context.node_parser.get_nodes_from_documents(documents)
   # add to docstore
   docstore = SimpleDocumentStore()
   docstore.add_documents(nodes)
   storage_context = StorageContext.from_defaults(docstore=docstore)
Build Index
   # build index
   index = VectorStoreIndex(nodes, storage_context=storage_context)
Add PrevNext Node Postprocessor
   node_postprocessor = PrevNextNodePostprocessor(docstore=docstore, num_nodes=4)
   query_engine = index.as_query_engine(
       similarity_top_k=1,
       node_postprocessors=[node_postprocessor],
       response_mode="tree_summarize",
   )
   response = query_engine.query(
       "What did the author do after handing off Y Combinator to Sam Altman?",
   )
   print(response)
   After handing off Y Combinator to Sam Altman, the author decided to take up painting. He spent most of the rest of 2014 painting and eventually ran out of steam in November. He then started writing essays again and wrote a few that weren't about startups. In March 2015, he started working on Lisp again and wrote a new Lisp, called Bel, in itself in Arc. He banned himself from writing essays during most of this time and worked on Bel intensively. In the summer of 2016, he and his family moved to England and he continued working on Bel there. In the fall of 2019, Bel was finally finished and he wrote a bunch of essays about topics he had stacked up. He then started to think about other things he could work on and wrote an essay for himself to answer that question.
   # Try querying index without node postprocessor
   query_engine = index.as_query_engine(similarity_top_k=1, response_mode="tree_summarize")
   response = query_engine.query(
       "What did the author do after handing off Y Combinator to Sam Altman?",
   )
   print(response)
   The author decided to take up painting and spent the rest of 2014 painting. He wanted to see how good he could get if he really focused on it.
   # Try querying index without node postprocessor and higher top-k
   query_engine = index.as_query_engine(similarity_top_k=3, response_mode="tree_summarize")
   response = query_engine.query(
       "What did the author do after handing off Y Combinator to Sam Altman?",
   )
   print(response)
   After handing off Y Combinator to Sam Altman, the author decided to take a break and focus on painting. He also gave a talk to the Harvard Computer Society about how to start a startup, and decided to start angel investing. He also schemed with Robert and Trevor about projects they could work on together. Finally, he and Jessica decided to start their own investment firm, which eventually became Y Combinator.
Add Auto Prev/Next Node Postprocessor
   node_postprocessor = AutoPrevNextNodePostprocessor(
       docstore=docstore, num_nodes=3, service_context=service_context, verbose=True
   )
   # Infer that we need to search nodes after current one

Forward/Backward Augmentation

   query_engine = index.as_query_engine(
       similarity_top_k=1,
       node_postprocessors=[node_postprocessor],
       response_mode="tree_summarize",
   )
   response = query_engine.query(
       "What did the author do after handing off Y Combinator to Sam Altman?",
   )
   > Postprocessor Predicted mode: next
   print(response)
   After handing off Y Combinator to Sam Altman, the author decided to take a break and focus on painting. He spent most of 2014 painting and was able to work more uninterruptedly than he had before. He also wrote a few essays that weren't about startups. In March 2015, he started working on Lisp again and wrote a new Lisp, called Bel, in itself in Arc. He had to ban himself from writing essays during most of this time in order to finish the project. In the summer of 2016, he and his family moved to England and he wrote most of Bel there. In the fall of 2019, Bel was finally finished. He then wrote a bunch of essays about topics he had stacked up and started to think about other things he could work on.
   # Infer that we don't need to search previous or next
   response = query_engine.query(
       "What did the author do during his time at Y Combinator?",
   )
   > Postprocessor Predicted mode: none
   print(response)
   The author did a variety of things during his time at Y Combinator, including hacking, writing essays, and working on YC. He also worked on a new version of Arc and wrote Hacker News in it. Additionally, he noticed the advantages of scaling startup funding and the tight community of alumni dedicated to helping one another.
   # Infer that we need to search nodes before current one
   response = query_engine.query(
       "What did the author do before handing off Y Combinator to Sam Altman?",
   )
   > Postprocessor Predicted mode: previous
   print(response)
   Before handing off Y Combinator to Sam Altman, the author worked on writing essays, working on Y Combinator, writing all of Y Combinator's internal software in Arc, and fighting with people who maltreated the startups. He also spent time visiting his mother, who had a stroke and was in a nursing home, and thinking about what to do next.
   response = query_engine.query(
       "What did the author do before handing off Y Combinator to Sam Altman?",
   )
   > Postprocessor Predicted mode: previous
   print(response)
   Before handing off Y Combinator to Sam Altman, the author worked on YC, wrote essays, and wrote all of YC's internal software in Arc. He also worked on a new version of Arc with Robert Morris, which he tested by writing Hacker News in it.