Metadata Replacement + Node Sentence Window

In this notebook, we use the "SentenceWindowNodeParser" to parse
documents into single sentences per node. Each node also contains a
"window" with the sentences on either side of the node sentence.
Then, during retrieval, before passing the retrieved sentences to the
LLM, the single sentences are replaced with a window containing the
surrounding sentences using the
"MetadataReplacementNodePostProcessor".
This is most useful for large documents/indexes, as it helps to
retrieve more fine-grained details.
By default, the sentence window is 5 sentences on either side of the
original sentence.
In this case, chunk size settings are not used, in favor of following
the window settings.
   %load_ext autoreload
   %autoreload 2
Setup
   import os
   import openai
   os.environ["OPENAI_API_KEY"] = "sk-..."
   openai.api_key = os.environ["OPENAI_API_KEY"]
   from llama_index import ServiceContext, set_global_service_context
   from llama_index.llms import OpenAI
   from llama_index.embeddings import OpenAIEmbedding, HuggingFaceEmbedding
   from llama_index.node_parser import SentenceWindowNodeParser, SimpleNodeParser
   # create the sentence window node parser w/ default settings
   node_parser = SentenceWindowNodeParser.from_defaults(
       window_size=3,
       window_metadata_key="window",
       original_text_metadata_key="original_text",
   )
   simple_node_parser = SimpleNodeParser.from_defaults()
   llm = OpenAI(model="gpt-3.5-turbo", temperature=0.1)
   embed_model = HuggingFaceEmbedding(
       model_name="sentence-transformers/all-mpnet-base-v2", max_length=512
   )
   ctx = ServiceContext.from_defaults(
       llm=llm,
       embed_model=embed_model,
       # node_parser=node_parser,
   )
   # if you wanted to use OpenAIEmbedding, we should also increase the batch size,
   # since it involves many more calls to the API
   # ctx = ServiceContext.from_defaults(llm=llm, embed_model=OpenAIEmbedding(embed_batch_size=50)), node_parser=node_parser)
Load Data, Build the Index
In this section, we load data and build the vector index.
Load Data
Here, we build an index using chapter 3 of the recent IPCC climate
report.
   !curl https://www..ch/report/ar6/wg2/downloads/report/IPCC_AR6_WGII_Chapter03.pdf --output IPCC_AR6_WGII_Chapter03.pdf
     % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                    Dload  Upload   Total   Spent    Left  Speed
     0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: www..ch
   from llama_index import SimpleDirectoryReader
   documents = SimpleDirectoryReader(
       input_files=["./IPCC_AR6_WGII_Chapter03.pdf"]
   ).load_data()
Extract Nodes
We extract out the set of nodes that will be stored in the
VectorIndex. This includes both the nodes with the sentence window
parser, as well as the "base" nodes extracted using the standard
parser.
   nodes = node_parser.get_nodes_from_documents(documents)
   base_nodes = simple_node_parser.get_nodes_from_documents(documents)
Build the Indexes
We build both the sentence index, as well as the "base" index (with
default chunk sizes).
   from llama_index import VectorStoreIndex
   sentence_index = VectorStoreIndex(nodes, service_context=ctx)

   base_index = VectorStoreIndex(base_nodes, service_context=ctx)
Querying
With MetadataReplacementPostProcessor
Here, we now use the "MetadataReplacementPostProcessor" to replace the
sentence in each node with it's surrounding context.
   from llama_index.indices.postprocessor import MetadataReplacementPostProcessor
   query_engine = sentence_index.as_query_engine(
       similarity_top_k=2,
       # the target key defaults to `window` to match the node_parser's default
       node_postprocessors=[
           MetadataReplacementPostProcessor(target_metadata_key="window")
       ],
   )
   window_response = query_engine.query("What are the concerns surrounding the AMOC?")
   print(window_response)
   There is low confidence in the quantification of Atlantic Meridional Overturning Circulation (AMOC) changes in the 20th century due to low agreement in quantitative reconstructed and simulated trends. Additionally, direct observational records since the mid-2000s remain too short to determine the relative contributions of internal variability, natural forcing, and anthropogenic forcing to AMOC change. However, it is very likely that AMOC will decline for all SSP scenarios over the 21st century, but it will not involve an abrupt collapse before 2100.
We can also check the original sentence that was retrieved for each
node, as well as the actual window of sentences that was sent to the
LLM.
   window = window_response.source_nodes[0].node.metadata["window"]
   sentence = window_response.source_nodes[0].node.metadata["original_text"]
   print(f"Window: {window}")
   print("------------------")
   print(f"Original Sentence: {sentence}")
   Window: Nevertheless, projected future annual cumulative upwelling wind
   changes at most locations and seasons remain within ±10–20% of
   present-day values (medium confidence) (WGI AR6 Section  9.2.3.5;
   Fox-Kemper et al., 2021).
    Continuous observation of the Atlantic meridional overturning
   circulation (AMOC) has improved the understanding of its variability
   (Frajka-Williams et  al., 2019), but there is low confidence in the
   quantification of AMOC changes in the 20th century because of low
   agreement in quantitative reconstructed and simulated trends (WGI
   AR6 Sections 2.3.3, 9.2.3.1; Fox-Kemper et al., 2021; Gulev et al., 2021).
    Direct observational records since the mid-2000s remain too short to
   determine the relative contributions of internal variability, natural
   forcing and anthropogenic forcing to AMOC change (high confidence)
   (WGI AR6 Sections 2.3.3, 9.2.3.1; Fox-Kemper et al., 2021; Gulev et al.,
   2021).  Over the 21st century, AMOC will very likely decline for all SSP
   scenarios but will not involve an abrupt collapse before 2100 (WGI
   AR6 Sections 4.3.2, 9.2.3.1; Fox-Kemper et al., 2021; Lee et al., 2021).
    3.2.2.4 Sea Ice Changes
   Sea ice is a key driver of polar marine life, hosting unique ecosystems
   and affecting diverse marine organisms and food webs through its
   impact on light penetration and supplies of nutrients and organic
   matter (Arrigo, 2014).  Since the late 1970s, Arctic sea ice area has
   decreased for all months, with an estimated decrease of 2 million km2
   (or 25%) for summer sea ice (averaged for August, September and

   October) in 2010–2019 as compared with 1979–1988 (WGI AR6
   Section 9.3.1.1; Fox-Kemper et al., 2021).
   ------------------
   Original Sentence: Over the 21st century, AMOC will very likely decline for all SSP
   scenarios but will not involve an abrupt collapse before 2100 (WGI
   AR6 Sections 4.3.2, 9.2.3.1; Fox-Kemper et al., 2021; Lee et al., 2021).
Contrast with normal VectorStoreIndex
   query_engine = base_index.as_query_engine(similarity_top_k=2)
   vector_response = query_engine.query("What are the concerns surrounding the AMOC?")
   print(vector_response)
   The concerns surrounding the AMOC are not provided in the given context information.
Well, that didn't work. Let's bump up the top k! This will be slower
and use more tokens compared to the sentence window index.
   query_engine = base_index.as_query_engine(similarity_top_k=5)
   vector_response = query_engine.query("What are the concerns surrounding the AMOC?")
   print(vector_response)
   There are concerns surrounding the AMOC (Atlantic Meridional Overturning Circulation). The context information mentions that the AMOC will decline over the 21st century, with high confidence but low confidence for quantitative projections.
Analysis
So the "SentenceWindowNodeParser" +
"MetadataReplacementNodePostProcessor" combo is the clear winner here.
But why?
Embeddings at a sentence level seem to capture more fine-grained
details, like the word "AMOC".
We can also compare the retrieved chunks for each index!
   for source_node in window_response.source_nodes:
       print(source_node.node.metadata["original_text"])
       print("--------")
   Over the 21st century, AMOC will very likely decline for all SSP
   scenarios but will not involve an abrupt collapse before 2100 (WGI
   AR6 Sections 4.3.2, 9.2.3.1; Fox-Kemper et al., 2021; Lee et al., 2021).
   --------
   Direct observational records since the mid-2000s remain too short to
   determine the relative contributions of internal variability, natural
   forcing and anthropogenic forcing to AMOC change (high confidence)
   (WGI AR6 Sections 2.3.3, 9.2.3.1; Fox-Kemper et al., 2021; Gulev et al.,
   2021).
   --------
Here, we can see that the sentence window index easily retrieved two
nodes that talk about AMOC. Remember, the embeddings are based purely
on the original sentence here, but the LLM actually ends up reading
the surrounding context as well!
Now, let's try and disect why the naive vector index failed.
   for node in vector_response.source_nodes:
       print("AMOC mentioned?", "AMOC" in node.node.text)
       print("--------")
   AMOC mentioned? False
   --------
   AMOC mentioned? False
   --------
   AMOC mentioned? True
   --------
   AMOC mentioned? False
   --------
   AMOC mentioned? False
   --------
So source node at index [2] mentions AMOC, but what did this text
actually look like?
   print(vector_response.source_nodes[2].node.text)
   2021; Gulev et al.
   2021)The AMOC will decline over the 21st century
   (high confidence, but low confidence for
   quantitative projections).4.3.2.3, 9.2.3 (Fox-Kemper

   et al. 2021; Lee et al.
   2021)
   Sea ice
   Arctic sea ice
   changes‘Current Arctic sea ice coverage levels are the
   lowest since at least 1850 for both annual mean
   and late-summer values (high confidence).’2.3.2.1, 9.3.1 (Fox-Kemper
   et al. 2021; Gulev et al.
   2021)‘The Arctic will become practically ice-free in
   September by the end of the 21st century under
   SSP2-4.5, SSP3-7.0 and SSP5-8.5[…](high
   confidence).’4.3.2.1, 9.3.1 (Fox-Kemper
   et al. 2021; Lee et al.
   2021)
   Antarctic sea ice
   changesThere is no global significant trend in
   Antarctic sea ice area from 1979 to 2020 (high
   confidence).2.3.2.1, 9.3.2 (Fox-Kemper
   et al. 2021; Gulev et al.
   2021)There is low confidence in model simulations of
   future Antarctic sea ice.9.3.2 (Fox-Kemper et al.
   2021)
   Ocean chemistry
   Changes in salinityThe ‘large-scale, near-surface salinity contrasts
   have intensified since at least 1950 […]
   (virtually certain).’2.3.3.2, 9.2.2.2
   (Fox-Kemper et al. 2021;
   Gulev et al. 2021)‘Fresh ocean regions will continue to get fresher
   and salty ocean regions will continue to get
   saltier in the 21st century (medium confidence).’9.2.2.2 (Fox-Kemper et al.
   2021)
   Ocean acidificationOcean surface pH has declined globally over the
   past four decades (virtually certain).2.3.3.5, 5.3.2.2 (Canadell
   et al. 2021; Gulev et al.
   2021)Ocean surface pH will continue to decrease
   ‘through the 21st century, except for the
   lower-emission scenarios SSP1-1.9 and SSP1-2.6
   […] (high confidence).’4.3.2.5, 4.5.2.2, 5.3.4.1
   (Lee et al. 2021; Canadell
   et al. 2021)
   Ocean
   deoxygenationDeoxygenation has occurred in most open
   ocean regions since the mid-20th century (high
   confidence).2.3.3.6, 5.3.3.2 (Canadell
   et al. 2021; Gulev et al.
   2021)Subsurface oxygen content ‘is projected to
   transition to historically unprecedented condition
   with decline over the 21st century (medium
   confidence).’5.3.3.2 (Canadell et al.
   2021)
   Changes in nutrient
   concentrationsNot assessed in WGI Not assessed in WGI
So AMOC is disuccsed, but sadly it is in the middle chunk. With LLMs,
it is often observed that text in the middle of retrieved context is

often ignored or less useful. A recent paper "Lost in the Middle"
discusses this here.
[Optional] Evaluation
We more rigorously evaluate how well the sentence window retriever
works compared to the base retriever.
We define/load an eval benchmark dataset and then run different
evaluations over it.
**WARNING**: This can be *expensive*, especially with GPT-4. Use
caution and tune the sample size to fit your budget.
   from llama_index.evaluation import (
       DatasetGenerator,
       QueryResponseDataset,
   )
   from llama_index import ServiceContext
   from llama_index.llms import OpenAI
   import nest_asyncio
   import random
   nest_asyncio.apply()
   len(base_nodes)
   428
   num_nodes_eval = 30
   # there are 428 nodes total. Take the first 200 to generate questions (the back half of the doc is all references)
   sample_eval_nodes = random.sample(base_nodes[:200], num_nodes_eval)
   # NOTE: run this if the dataset isn't already saved
   eval_service_context = ServiceContext.from_defaults(llm=OpenAI(model="gpt-4"))
   # generate questions from the largest chunks (1024)
   dataset_generator = DatasetGenerator(
       sample_eval_nodes,
       service_context=eval_service_context,
       show_progress=True,
       num_questions_per_chunk=2,
   )
   eval_dataset = await dataset_generator.agenerate_dataset_from_nodes()
   eval_dataset.save_json("data/ipcc_eval_qr_dataset.json")
   # optional
   eval_dataset = QueryResponseDataset.from_json("data/ipcc_eval_qr_dataset.json")
Compare Results
   import asyncio
   import nest_asyncio
   nest_asyncio.apply()
   from llama_index.evaluation import (
       CorrectnessEvaluator,
       SemanticSimilarityEvaluator,
       RelevancyEvaluator,
       FaithfulnessEvaluator,
       PairwiseComparisonEvaluator,
   )
   from collections import defaultdict
   import pandas as pd
   # NOTE: can uncomment other evaluators
   evaluator_c = CorrectnessEvaluator(service_context=eval_service_context)
   evaluator_s = SemanticSimilarityEvaluator(service_context=eval_service_context)
   evaluator_r = RelevancyEvaluator(service_context=eval_service_context)
   evaluator_f = FaithfulnessEvaluator(service_context=eval_service_context)
   # pairwise_evaluator = PairwiseComparisonEvaluator(service_context=eval_service_context)
   from llama_index.evaluation.eval_utils import get_responses, get_results_df
   from llama_index.evaluation import BatchEvalRunner
   max_samples = 30
   eval_qs = eval_dataset.questions
   ref_response_strs = [r for (_, r) in eval_dataset.qr_pairs]
   # resetup base query engine and sentence window query engine
   # base query engine
   base_query_engine = base_index.as_query_engine(similarity_top_k=2)
   # sentence window query engine
   query_engine = sentence_index.as_query_engine(
       similarity_top_k=2,
       # the target key defaults to `window` to match the node_parser's default
       node_postprocessors=[
           MetadataReplacementPostProcessor(target_metadata_key="window")
       ],
   )
   import numpy as np
   base_pred_responses = get_responses(
       eval_qs[:max_samples], base_query_engine, show_progress=True
   )
   pred_responses = get_responses(eval_qs[:max_samples], query_engine, show_progress=True)
   pred_response_strs = [str(p) for p in pred_responses]
   base_pred_response_strs = [str(p) for p in base_pred_responses]
   evaluator_dict = {
       "correctness": evaluator_c,
       "faithfulness": evaluator_f,
       "relevancy": evaluator_r,
       "semantic_similarity": evaluator_s,
   }
   batch_runner = BatchEvalRunner(evaluator_dict, workers=2, show_progress=True)
Run evaluations over faithfulness/semantic similarity.

   eval_results = await batch_runner.aevaluate_responses(
       queries=eval_qs[:max_samples],
       responses=pred_responses[:max_samples],
       reference=ref_response_strs[:max_samples],
   )
   base_eval_results = await batch_runner.aevaluate_responses(
       queries=eval_qs[:max_samples],
       responses=base_pred_responses[:max_samples],
       reference=ref_response_strs[:max_samples],
   )
   results_df = get_results_df(
       [eval_results, base_eval_results],
       ["Sentence Window Retriever", "Base Retriever"],
       ["correctness", "relevancy", "faithfulness", "semantic_similarity"],
   )
   display(results_df)
                          names  correctness  relevancy  faithfulness  \
   0  Sentence Window Retriever     4.366667   0.933333      0.933333
   1             Base Retriever     4.216667   0.900000      0.933333
      semantic_similarity
   0             0.959583
   1             0.958664


Rerank can speed up an LLM query without sacrificing accuracy (and in

fact, probably improving it). It does so by pruning away irrelevant
nodes from the context.
   from llama_index import VectorStoreIndex, SimpleDirectoryReader
   # load documents
   documents = SimpleDirectoryReader("../data/paul_graham").load_data()
   from llama_index import ServiceContext, set_global_service_context
   from llama_index.llms import OpenAI
   llm = OpenAI(model="gpt-3.5-turbo")
   ctx = ServiceContext.from_defaults(embed_model="local")
   set_global_service_context(ctx)
   /home/jonch/.local/lib/python3.10/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
   # build index
   index = VectorStoreIndex.from_documents(documents=documents)
   from llama_index.indices.postprocessor import SentenceTransformerRerank
   rerank = SentenceTransformerRerank(
       model="cross-encoder/ms-marco-MiniLM-L-2-v2", top_n=3
   )
First, we try with reranking. We time the query to see how long it
takes to process the output from the retrieved context.
   from time import time
   query_engine = index.as_query_engine(similarity_top_k=10, node_postprocessors=[rerank])
   now = time()
   response = query_engine.query(
       "Which grad schools did the author apply for and why?",
   )
   print(f"Elapsed: {round(time() - now, 2)}s")
   Elapsed: 4.03s
   print(response)
   The author applied to three grad schools: MIT and Yale, which were renowned for AI at the time, and Harvard, which the author had visited because a friend went there and it was also home to Bill Woods, who had invented the type of parser the author used in his SHRDLU clone. The author chose these schools because he wanted to learn about AI and Lisp, and these schools were known for their expertise in these areas.
   print(response.get_formatted_sources(length=200))
   > Source (Doc id: 08074ca9-1806-4e49-84de-102a97f1f220): been explored. But all I wanted was to get out of grad school, and my rapidly written dissertation sufficed, just barely.
   Meanwhile I was applying to art schools. I applied to two: RISD in the US,...
   > Source (Doc id: 737f4526-2752-45e8-a59a-e1e4528cc025): about money, because I could sense that Interleaf was on the way down. Freelance Lisp hacking work was very rare, and I didn't want to have to program in another language, which in those days would...
   > Source (Doc id: b8883569-44f9-454c-9f62-15e926d04b98): showed Terry Winograd using SHRDLU. I haven't tried rereading The Moon is a Harsh Mistress, so I don't know how well it has aged, but when I read it I was drawn entirely into its world. It seemed o...
Next, we try without rerank
   query_engine = index.as_query_engine(similarity_top_k=10)
   now = time()
   response = query_engine.query(
       "Which grad schools did the author apply for and why?",
   )
   print(f"Elapsed: {round(time() - now, 2)}s")
   Elapsed: 28.13s
   print(response)
   The author applied to three grad schools: MIT and Yale, which were renowned for AI at the time, and Harvard, which the author had visited because a friend went there and was also home to Bill Woods, who had invented the type of parser the author used in his SHRDLU clone. The author chose these schools because he was interested in Artificial Intelligence and wanted to pursue it further, and they were the most renowned for it at the time. He was also inspired by a novel by Heinlein called The Moon is a Harsh Mistress, which featured an intelligent computer called Mike, and a PBS documentary that showed Terry Winograd using SHRDLU. Additionally, the author had dropped out of RISD, where he had been learning to paint, and was looking for a new challenge. He was drawn to the idea of pursuing AI, as it was a field that was rapidly growing and he wanted to be part of the cutting edge of technology. He was also inspired by the idea of creating something unique and innovative, as he had done with his SHRDLU clone, and wanted to continue to explore the possibilities of AI.

   print(response.get_formatted_sources(length=200))
   > Source (Doc id: 08074ca9-1806-4e49-84de-102a97f1f220): been explored. But all I wanted was to get out of grad school, and my rapidly written dissertation sufficed, just barely.
   Meanwhile I was applying to art schools. I applied to two: RISD in the US,...
   > Source (Doc id: 737f4526-2752-45e8-a59a-e1e4528cc025): about money, because I could sense that Interleaf was on the way down. Freelance Lisp hacking work was very rare, and I didn't want to have to program in another language, which in those days would...
   > Source (Doc id: b8883569-44f9-454c-9f62-15e926d04b98): showed Terry Winograd using SHRDLU. I haven't tried rereading The Moon is a Harsh Mistress, so I don't know how well it has aged, but when I read it I was drawn entirely into its world. It seemed o...
   > Source (Doc id: 599f469b-9a92-4952-8753-a063c31a953b): I didn't know but would turn out to like a lot: a woman called Jessica Livingston. A couple days later I asked her out.
   Jessica was in charge of marketing at a Boston investment bank. This bank th...
   > Source (Doc id: c865f333-b731-4a8b-a99f-eec54eaa1e6b): Like McCarthy's original Lisp, it's a spec rather than an implementation, although like McCarthy's Lisp it's a spec expressed as code.
   Now that I could write essays again, I wrote a bunch about to...
   > Source (Doc id: 69c6b190-2d4e-4128-b9c4-4fd31af2df65): 1960 paper.
   But if so there's no reason to suppose that this is the limit of the language that might be known to them. Presumably aliens need numbers and errors and I/O too. So it seems likely the...
   > Source (Doc id: c9c95028-a49e-440e-a953-7aabe6b9996d): What I Worked On
   February 2021
   Before college the two main things I worked on, outside of school, were writing and programming. I didn't write essays. I wrote what beginning writers were supposed...
   > Source (Doc id: 7f0c11db-d6f0-41f9-95bc-1feab914f58f): that big, bureaucratic customers are a dangerous source of money, and that there's not much overlap between conventional office hours and the optimal time for hacking, or conventional offices and t...
   > Source (Doc id: c143a6c2-5f5d-49c5-bc1e-b9caa0ce4931): must tell readers things they don't already know, and some people dislike being told such things.
   [11] People put plenty of stuff on the internet in the 90s of course, but putting something online...
   > Source (Doc id: 6e281eec-6964-414b-be61-bcc509d95903): which I'd created years before using Viaweb but had never used for anything. In one day it got 30,000 page views. What on earth had happened? The referring urls showed that someone had posted it on...
As we can see, the query engine with reranking produced a much more
concise output in much lower time (4s v.s. 28s). While both responses

were essentially correct, the query engine without reranking included
a lot of irrelevant information - a phenomenon we could attribute to
"pollution of the context window".


Sentence Embedding Optimizer

   # My OpenAI Key
   import os
   os.environ["OPENAI_API_KEY"] = "INSERT OPENAI KEY"
Setup
   from llama_index import download_loader
   WikipediaReader = download_loader("WikipediaReader")
   loader = WikipediaReader()
   documents = loader.load_data(pages=["Berlin"])
   from llama_index import VectorStoreIndex
   index = VectorStoreIndex.from_documents(documents)
   <class 'llama_index.readers.schema.base.Document'>
   INFO:root:> [build_index_from_documents] Total LLM token usage: 0 tokens
   INFO:root:> [build_index_from_documents] Total embedding token usage: 18390 tokens
Compare query with and without optimization for LLM token usage,
Embedding Model usage on query, Embedding model usage for optimizer,
and total time.
   import time
   from llama_index import VectorStoreIndex
   from llama_index.indices.postprocessor import SentenceEmbeddingOptimizer
   print("Without optimization")
   start_time = time.time()
   query_engine = index.as_query_engine()
   res = query_engine.query("What is the population of Berlin?")
   end_time = time.time()
   print("Total time elapsed: {}".format(end_time - start_time))
   print("Answer: {}".format(res))
   print("With optimization")
   start_time = time.time()
   query_engine = index.as_query_engine(
       node_postprocessors=[SentenceEmbeddingOptimizer(percentile_cutoff=0.5)]
   )
   res = query_engine.query("What is the population of Berlin?")
   end_time = time.time()
   print("Total time elapsed: {}".format(end_time - start_time))
   print("Answer: {}".format(res))
   print("Alternate optimization cutoff")
   start_time = time.time()
   query_engine = index.as_query_engine(
       node_postprocessors=[SentenceEmbeddingOptimizer(threshold_cutoff=0.7)]
   )
   res = query_engine.query("What is the population of Berlin?")
   end_time = time.time()
   print("Total time elapsed: {}".format(end_time - start_time))
   print("Answer: {}".format(res))
   Without optimization
   INFO:root:> [query] Total LLM token usage: 3545 tokens
   INFO:root:> [query] Total embedding token usage: 7 tokens
   Total time elapsed: 2.8928110599517822
   Answer:
   The population of Berlin in 1949 was approximately 2.2 million inhabitants. After the fall of the Berlin Wall in 1989, the population of Berlin increased to approximately 3.7 million inhabitants.
   With optimization
   INFO:root:> [optimize] Total embedding token usage: 7 tokens
   INFO:root:> [query] Total LLM token usage: 1779 tokens
   INFO:root:> [query] Total embedding token usage: 7 tokens
   Total time elapsed: 2.346346139907837
   Answer:
   The population of Berlin is around 4.5 million.
   Alternate optimization cutoff
   INFO:root:> [optimize] Total embedding token usage: 7 tokens
   INFO:root:> [query] Total LLM token usage: 3215 tokens
   INFO:root:> [query] Total embedding token usage: 7 tokens
   Total time elapsed: 2.101111888885498
   Answer:
   The population of Berlin is around 4.5 million.


Cohere Rerank

   from llama_index import VectorStoreIndex, SimpleDirectoryReader, pprint_response
   /Users/suo/miniconda3/envs/llama/lib/python3.9/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
   # load documents
   documents = SimpleDirectoryReader("../data/paul_graham").load_data()
   # build index
   index = VectorStoreIndex.from_documents(documents=documents)
Retrieve top 10 most relevant nodes, then filter with Cohere Rerank
   import os
   from llama_index.indices.postprocessor.cohere_rerank import CohereRerank
   api_key = os.environ["COHERE_API_KEY"]
   cohere_rerank = CohereRerank(api_key=api_key, top_n=2)
   query_engine = index.as_query_engine(
       similarity_top_k=10,
       node_postprocessors=[cohere_rerank],
   )
   response = query_engine.query(
       "What did Sam Altman do in this essay?",
   )
   pprint_response(response)
   Final Response: Sam Altman agreed to become the president of Y
   Combinator in October 2013. He took over starting with the winter 2014
   batch, and worked with the founders to help them get through Demo Day
   in March 2014. He then reorganised Y Combinator to be controlled by
   someone other than the founders, so that it could last for a long
   time.
   ______________________________________________________________________
   Source Node 1/2
   Document ID: c1baaa76-acba-453b-a8d1-fdffbde1f424
   Similarity: 0.845305
   Text: day in 2010, when he was visiting California for interviews,
   Robert Morris did something astonishing: he offered me unsolicited
   advice. I can only remember him doing that once before. One day at
   Viaweb, when I was bent over double from a kidney stone, he suggested
   that it would be a good idea for him to take me to the hospital. That
   was what it ...
   ______________________________________________________________________
   Source Node 2/2
   Document ID: abc0f1aa-464a-4ae1-9a7b-2d47a9dc967e
   Similarity: 0.6486889
   Text: due to our ignorance about investing. We needed to get
   experience as investors. What better way, we thought, than to fund a
   whole bunch of startups at once? We knew undergrads got temporary jobs
   at tech companies during the summer. Why not organize a summer program
   where they'd start startups instead? We wouldn't feel guilty for being
   in a sense...
Directly retrieve top 2 most similar nodes
   query_engine = index.as_query_engine(
       similarity_top_k=2,
   )
   response = query_engine.query(
       "What did Sam Altman do in this essay?",
   )
Retrieved context is irrelevant and response is hallucinated.
   pprint_response(response)
   Final Response: Sam Altman was one of the founders of Y Combinator, a
   startup accelerator. He was part of the first batch of startups funded
   by Y Combinator, which included Reddit, Justin Kan and Emmett Shear's
   Twitch, and Aaron Swartz. He was also involved in the Summer Founders
   Program, which was a summer program where undergrads could start their
   own startups instead of taking a summer job at a tech company. He also

Cohere Rerank

   helped to develop a new version of Arc, a programming language, and
   wrote a book on Lisp.
   ______________________________________________________________________
   Source Node 1/2
   Document ID: abc0f1aa-464a-4ae1-9a7b-2d47a9dc967e
   Similarity: 0.7940524933077708
   Text: due to our ignorance about investing. We needed to get
   experience as investors. What better way, we thought, than to fund a
   whole bunch of startups at once? We knew undergrads got temporary jobs
   at tech companies during the summer. Why not organize a summer program
   where they'd start startups instead? We wouldn't feel guilty for being
   in a sense...
   ______________________________________________________________________
   Source Node 2/2
   Document ID: 5d696e20-b496-47f0-9262-7aa2667c1d96
   Similarity: 0.7899270712205545
   Text: at RISD, but otherwise I was basically teaching myself to paint,
   and I could do that for free. So in 1993 I dropped out. I hung around
   Providence for a bit, and then my college friend Nancy Parmet did me a
   big favor. A rent-controlled apartment in a building her mother owned
   in New York was becoming vacant. Did I want it? It wasn't much more
   tha...


LLM Reranker Demonstration (Great Gatsby)

This tutorial showcases how to do a two-stage pass for retrieval. Use
embedding-based retrieval with a high top-k value in order to maximize
recall and get a large set of candidate items. Then, use LLM-based
retrieval to dynamically select the nodes that are actually relevant
to the query.
   import nest_asyncio
   nest_asyncio.apply()
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SimpleDirectoryReader,
       ServiceContext,
       LLMPredictor,
   )
   from llama_index.indices.postprocessor import LLMRerank
   from llama_index.llms import OpenAI
   from IPython.display import Markdown, display
Load Data, Build Index
   # LLM Predictor (gpt-3.5-turbo) + service context
   llm = OpenAI(temperature=0, model="gpt-3.5-turbo")
   service_context = ServiceContext.from_defaults(llm=llm, chunk_size=512)
   WARNING:llama_index.llm_predictor.base:Unknown max input size for gpt-3.5-turbo, using defaults.
   Unknown max input size for gpt-3.5-turbo, using defaults.
   # load documents
   documents = SimpleDirectoryReader("../../../examples/gatsby/data").load_data()
   documents
   index = VectorStoreIndex.from_documents(documents, service_context=service_context)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   > [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 49266 tokens
   > [build_index_from_nodes] Total embedding token usage: 49266 tokens
Retrieval
   from llama_index.retrievers import VectorIndexRetriever
   from llama_index.indices.query.schema import QueryBundle
   import pandas as pd
   from IPython.display import display, HTML
   pd.set_option("display.max_colwidth", -1)
   def get_retrieved_nodes(
       query_str, vector_top_k=10, reranker_top_n=3, with_reranker=False
   ):
       query_bundle = QueryBundle(query_str)
       # configure retriever
       retriever = VectorIndexRetriever(
           index=index,
           similarity_top_k=vector_top_k,
       )
       retrieved_nodes = retriever.retrieve(query_bundle)
       if with_reranker:
           # configure reranker
           reranker = LLMRerank(
               choice_batch_size=5, top_n=reranker_top_n, service_context=service_context
           )
           retrieved_nodes = reranker.postprocess_nodes(retrieved_nodes, query_bundle)
       return retrieved_nodes
   def pretty_print(df):
       return display(HTML(df.to_html().replace("\\n", "<br>")))
   def visualize_retrieved_nodes(nodes) -> None:
       result_dicts = []
       for node in nodes:
           result_dict = {"Score": node.score, "Text": node.node.get_text()}
           result_dicts.append(result_dict)
       pretty_print(pd.DataFrame(result_dicts))
   /var/folders/1r/c3h91d9s49xblwfvz79s78_c0000gn/T/ipykernel_44297/3519340226.py:7: FutureWarning: Passing a negative integer is deprecated in version 1.0 and will not be supported in future version. Instead, use None to not limit the column width.
     pd.set_option('display.max_colwidth', -1)
   new_nodes = get_retrieved_nodes(

       "Who was driving the car that hit Myrtle?", vector_top_k=3, with_reranker=False
   )
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 10 tokens
   > [retrieve] Total embedding token usage: 10 tokens
   visualize_retrieved_nodes(new_nodes)
   <IPython.core.display.HTML object>
   new_nodes = get_retrieved_nodes(
       "Who was driving the car that hit Myrtle?",
       vector_top_k=10,
       reranker_top_n=3,
       with_reranker=True,
   )
   visualize_retrieved_nodes(new_nodes)
   <IPython.core.display.HTML object>
   new_nodes = get_retrieved_nodes(
       "What did Gatsby want Daisy to do in front of Tom?",
       vector_top_k=3,
       with_reranker=False,
   )
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 14 tokens
   > [retrieve] Total embedding token usage: 14 tokens
   visualize_retrieved_nodes(new_nodes)
   ****Score****: 0.8647796939111776
   ****Node text****
   : got to make your house into a pigsty in order to have any
   friends—in the modern world.”
   Angry as I was, as we all were, I was tempted to laugh whenever he
   opened his mouth. The transition from libertine to prig was so
   complete.
   “I’ve got something to tell you, old sport—” began Gatsby. But Daisy
   guessed at his intention.
   “Please don’t!” she interrupted helplessly. “Please let’s all go
   home. Why don’t we all go home?”
   “That’s a good idea,” I got up. “Come on, Tom. Nobody wants a drink.”
   “I want to know what Mr. Gatsby has to tell me.”
   “Your wife doesn’t love you,” said Gatsby. “She’s never loved you.
   She loves me.”
   “You must be crazy!” exclaimed Tom automatically.
   Gatsby sprang to his feet, vivid with excitement.
   “She never loved you, do you hear?” he cried. “She only married you
   because I was poor and she was tired of waiting for me. It was a
   terrible mistake, but in her heart she never loved anyone except me!”
   At this point Jordan and I tried to go, but Tom and Gatsby insisted
   with competitive firmness that we remain—as though neither of them had
   anything to conceal and it would be a privilege to partake vicariously
   of their emotions.
   “Sit down, Daisy,” Tom’s voice groped unsuccessfully for the paternal
   note. “What’s been going on? I want to hear all about it.”
   “I told you what’s been going on,” said Gatsby. “Going on for five
   years—and you didn’t know.”
   Tom turned to Daisy
   ****Score****: 0.8609230717744326
   ****Node text****
   : to keep your
   shoes dry?” There was a husky tenderness in his tone … “Daisy?”
   “Please don’t.” Her voice was cold, but the rancour was gone from it.
   She looked at Gatsby. “There, Jay,” she said—but her hand as she tried

   to light a cigarette was trembling. Suddenly she threw the cigarette
   and the burning match on the carpet.
   “Oh, you want too much!” she cried to Gatsby. “I love you now—isn’t
   that enough? I can’t help what’s past.” She began to sob
   helplessly. “I did love him once—but I loved you too.”
   Gatsby’s eyes opened and closed.
   “You loved me too?” he repeated.
   “Even that’s a lie,” said Tom savagely. “She didn’t know you were
   alive. Why—there’s things between Daisy and me that you’ll never know,
   things that neither of us can ever forget.”
   The words seemed to bite physically into Gatsby.
   “I want to speak to Daisy alone,” he insisted. “She’s all excited
   now—”
   “Even alone I can’t say I never loved Tom,” she admitted in a pitiful
   voice. “It wouldn’t be true.”
   “Of course it wouldn’t,” agreed Tom.
   She turned to her husband.
   “As if it mattered to you,” she said.
   “Of course it matters. I’m going to take better care of you from now
   on.”
   “You don’t understand,” said Gatsby, with a touch of panic. “You’re
   not going to take care of her any more.”
   “I’m not?” Tom opened his eyes wide and
   ****Score****: 0.8555028907426916
   ****Node text****
   : shadowed well with awnings, was dark and cool. Daisy and
   Jordan lay upon an enormous couch, like silver idols weighing down
   their own white dresses against the singing breeze of the fans.
   “We can’t move,” they said together.
   Jordan’s fingers, powdered white over their tan, rested for a moment
   in mine.
   “And Mr. Thomas Buchanan, the athlete?” I inquired.
   Simultaneously I heard his voice, gruff, muffled, husky, at the hall
   telephone.
   Gatsby stood in the centre of the crimson carpet and gazed around with
   fascinated eyes. Daisy watched him and laughed, her sweet, exciting
   laugh; a tiny gust of powder rose from her bosom into the air.
   “The rumour is,” whispered Jordan, “that that’s Tom’s girl on the
   telephone.”
   We were silent. The voice in the hall rose high with annoyance: “Very
   well, then, I won’t sell you the car at all … I’m under no obligations
   to you at all … and as for your bothering me about it at lunch time, I
   won’t stand that at all!”
   “Holding down the receiver,” said Daisy cynically.
   “No, he’s not,” I assured her. “It’s a bona-fide deal. I happen to
   know about it.”
   Tom flung open the door, blocked out its space for a moment with his
   thick body, and hurried into the room.
   “Mr. Gatsby!” He put out his broad, flat hand with well-concealed
   dislike. “I’m glad to see you, sir … Nick …”
   “Make us a cold drink,” cried Daisy.
   As he left the room again she got up and went over to Gatsby and
   pulled his face
   new_nodes = get_retrieved_nodes(
       "What did Gatsby want Daisy to do in front of Tom?",
       vector_top_k=10,
       reranker_top_n=3,
       with_reranker=True,
   )
   INFO:llama_index.token_counter.token_counter:> [retrieve] Total LLM token usage: 0 tokens
   > [retrieve] Total LLM token usage: 0 tokens

   INFO:llama_index.token_counter.token_counter:> [retrieve] Total embedding token usage: 14 tokens
   > [retrieve] Total embedding token usage: 14 tokens
   Doc: 2, Relevance: 10
   No relevant documents found. Please provide a different question.
   visualize_retrieved_nodes(new_nodes)
   ****Score****: 10.0
   ****Node text****
   : to keep your
   shoes dry?” There was a husky tenderness in his tone … “Daisy?”
   “Please don’t.” Her voice was cold, but the rancour was gone from it.
   She looked at Gatsby. “There, Jay,” she said—but her hand as she tried
   to light a cigarette was trembling. Suddenly she threw the cigarette
   and the burning match on the carpet.
   “Oh, you want too much!” she cried to Gatsby. “I love you now—isn’t
   that enough? I can’t help what’s past.” She began to sob
   helplessly. “I did love him once—but I loved you too.”
   Gatsby’s eyes opened and closed.
   “You loved me too?” he repeated.
   “Even that’s a lie,” said Tom savagely. “She didn’t know you were
   alive. Why—there’s things between Daisy and me that you’ll never know,
   things that neither of us can ever forget.”
   The words seemed to bite physically into Gatsby.
   “I want to speak to Daisy alone,” he insisted. “She’s all excited
   now—”
   “Even alone I can’t say I never loved Tom,” she admitted in a pitiful
   voice. “It wouldn’t be true.”
   “Of course it wouldn’t,” agreed Tom.
   She turned to her husband.
   “As if it mattered to you,” she said.
   “Of course it matters. I’m going to take better care of you from now
   on.”
   “You don’t understand,” said Gatsby, with a touch of panic. “You’re
   not going to take care of her any more.”
   “I’m not?” Tom opened his eyes wide and
Query Engine
   query_engine = index.as_query_engine(
       similarity_top_k=10, node_postprocessors=[reranker], response_mode="tree_summarize"
   )
   response = query_engine.query(
       "What did the author do during his time at Y Combinator?",
   )
   query_engine = index.as_query_engine(similarity_top_k=3, response_mode="tree_summarize")
   response = query_engine.query(
       "What did the author do during his time at Y Combinator?",
   )
   retrieval =


Recency Filtering

Showcase capabilities of recency-weighted node postprocessor
   from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
   from llama_index.indices.postprocessor import (
       FixedRecencyPostprocessor,
       EmbeddingRecencyPostprocessor,
   )
   from llama_index.node_parser import SimpleNodeParser
   from llama_index.storage.docstore import SimpleDocumentStore
   from llama_index.response.notebook_utils import display_response
   /Users/jerryliu/Programming/llama_index/.venv/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from .autonotebook import tqdm as notebook_tqdm
Parse Documents into Nodes, add to Docstore
In this example, there are 3 different versions of PG's essay. They
are largely identical **except** for one specific section, which
details the amount of funding they raised for Viaweb.
V1: 50k, V2: 30k, V3: 10K
V1: 2020-01-01, V2: 2020-02-03, V3: 2022-04-12
The idea is to encourage index to fetch the most recent info (which is
V3)
   # load documents
   from llama_index.storage.storage_context import StorageContext
   def get_file_metadata(file_name: str):
       """Get file metadata."""
       if "v1" in file_name:
           return {"date": "2020-01-01"}
       elif "v2" in file_name:
           return {"date": "2020-02-03"}
       elif "v3" in file_name:
           return {"date": "2022-04-12"}
       else:
           raise ValueError("invalid file")
   documents = SimpleDirectoryReader(
       input_files=[
           "test_versioned_data/paul_graham_essay_v1.txt",
           "test_versioned_data/paul_graham_essay_v2.txt",
           "test_versioned_data/paul_graham_essay_v3.txt",
       ],
       file_metadata=get_file_metadata,
   ).load_data()
   # define service context (wrapper container around current classes)
   service_context = ServiceContext.from_defaults(chunk_size=512)
   # use node parser in service context to parse into nodes
   nodes = service_context.node_parser.get_nodes_from_documents(documents)
   # add to docstore
   docstore = SimpleDocumentStore()
   docstore.add_documents(nodes)
   storage_context = StorageContext.from_defaults(docstore=docstore)
   print(documents[2].get_text())
Build Index
   # build index
   index = VectorStoreIndex(nodes, storage_context=storage_context)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 84471 tokens
Define Recency Postprocessors
   node_postprocessor = FixedRecencyPostprocessor(service_context=service_context)
   node_postprocessor_emb = EmbeddingRecencyPostprocessor(service_context=service_context)
Query Index
   # naive query
   query_engine = index.as_query_engine(
       similarity_top_k=3,
   )
   response = query_engine.query(
       "How much did the author raise in seed funding from Idelle's husband (Julian) for Viaweb?",
   )
   INFO:llama_index.token_counter.token_counter:> [query] Total LLM token usage: 1813 tokens
   INFO:llama_index.token_counter.token_counter:> [query] Total embedding token usage: 22 tokens

Recency Filtering

   # query using fixed recency node postprocessor
   query_engine = index.as_query_engine(
       similarity_top_k=3, node_postprocessors=[node_postprocessor]
   )
   response = query_engine.query(
       "How much did the author raise in seed funding from Idelle's husband (Julian) for Viaweb?",
   )
   # query using embedding-based node postprocessor
   query_engine = index.as_query_engine(
       similarity_top_k=3, node_postprocessors=[node_postprocessor_emb]
   )
   response = query_engine.query(
       "How much did the author raise in seed funding from Idelle's husband (Julian) for Viaweb?",
   )
   INFO:llama_index.token_counter.token_counter:> [query] Total LLM token usage: 541 tokens
   INFO:llama_index.token_counter.token_counter:> [query] Total embedding token usage: 22 tokens
Query Index (Lower-Level Usage)
In this example we first get the full set of nodes from a query call,
and then send to node postprocessor, and then finally synthesize
response through a summary index.
   from llama_index import SummaryIndex
   query_str = "How much did the author raise in seed funding from Idelle's husband (Julian) for Viaweb?"
   query_engine = index.as_query_engine(similarity_top_k=3, response_mode="no_text")
   init_response = query_engine.query(
       query_str,
   )
   resp_nodes = [n.node for n in init_response.source_nodes]
   INFO:llama_index.token_counter.token_counter:> [query] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [query] Total embedding token usage: 22 tokens
   summary_index = SummaryIndex(resp_nodes)
   query_engine = summary_index.as_query_engine(node_postprocessors=[node_postprocessor])
   response = query_engine.query(query_str)
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total LLM token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [build_index_from_nodes] Total embedding token usage: 0 tokens
   INFO:llama_index.token_counter.token_counter:> [query] Total LLM token usage: 541 tokens
   INFO:llama_index.token_counter.token_counter:> [query] Total embedding token usage: 0 tokens


Tree Summarize

Load Data
   from llama_index import SimpleDirectoryReader
   reader = SimpleDirectoryReader(
       input_files=["../data/paul_graham/paul_graham_essay.txt"]
   )
   docs = reader.load_data()
   text = docs[0].text
Summarize
   from llama_index.response_synthesizers import TreeSummarize
   summarizer = TreeSummarize(verbose=True)
   response = await summarizer.aget_response("who is Paul Graham?", [text])
   6 text chunks after repacking
   1 text chunks after repacking
   print(response)
   Paul Graham is a computer scientist, writer, artist, entrepreneur, investor, and essayist. He is best known for his work in artificial intelligence, Lisp programming, and writing the book On Lisp, as well as for co-founding the startup accelerator Y Combinator and for his essays on technology, business, and start-ups. He is also the creator of the programming language Arc and the Lisp dialect Bel.


Refine

Load Data
   from llama_index import SimpleDirectoryReader
   reader = SimpleDirectoryReader(
       input_files=["../data/paul_graham/paul_graham_essay.txt"]
   )
   docs = reader.load_data()
   text = docs[0].text
Summarize
   from llama_index.llms import OpenAI
   llm = OpenAI(model="gpt-3.5-turbo")
   from llama_index import ServiceContext
   service_context = ServiceContext.from_defaults(llm=llm)
   from llama_index.response_synthesizers import Refine
   summarizer = Refine(service_context=service_context, verbose=True)
   response = summarizer.get_response("who is Paul Graham?", [text])
   > Refine context: making fakes for a local antique dealer. She'd ...
   > Refine context: look legit, and the key to looking legit is hig...
   > Refine context: me 8 years to realize it. Even then it took me ...
   > Refine context: was one thing rarer than Rtm offering advice, i...
   print(response)
   Paul Graham is an individual who has played a crucial role in shaping the internet infrastructure and has also pursued a career as a writer. At one point, he received advice from a friend that urged him not to let Y Combinator be his final noteworthy achievement. This advice prompted him to reflect on his future with Y Combinator and ultimately led him to pass on the responsibility to others. He approached Jessica and Sam Altman to assume leadership positions in Y Combinator, aiming to secure its continued success.


Refine with Structured Answer Filtering

When using our Refine response synthesizer for response synthesis,
it's crucial to filter out non-answers. An issue often encountered is
the propagation of a single unhelpful response like "I don't have the
answer", which can persist throughout the synthesis process and lead
to a final answer of the same nature. This can occur even when there
are actual answers present in other, more relevant sections.
These unhelpful responses can be filtered out by setting
"structured_answer_filtering" to "True". It is set to "False" by
default since this currently only works best if you are using an
OpenAI model that supports function calling.
Load Data
   texts = [
       "The president in the year 2040 is John Cena.",
       "The president in the year 2050 is Florence Pugh.",
       'The president in the year 2060 is Dwayne "The Rock" Johnson.',
   ]
Summarize
   from llama_index.llms import OpenAI
   llm = OpenAI(model="gpt-3.5-turbo-0613")
   from llama_index import ServiceContext
   service_context = ServiceContext.from_defaults(llm=llm)
   from llama_index.response_synthesizers import get_response_synthesizer
   summarizer = get_response_synthesizer(
       response_mode="refine", service_context=service_context, verbose=True
   )
   response = summarizer.get_response("who is president in the year 2050?", texts)
   > Refine context: The president in the year 2050 is Florence Pugh...
   > Refine context: The president in the year 2060 is Dwayne "The R...
Failed Result
As you can see, we weren't able to get the correct answer from the
input "texts" strings since the initial "I don't know" answer
propogated through till the end of the response synthesis.
   print(response)
   I'm sorry, but I don't have access to information about the future.
Now we'll try again with "structured_answer_filtering=True"
   from llama_index.response_synthesizers import get_response_synthesizer
   summarizer = get_response_synthesizer(
       response_mode="refine",
       service_context=service_context,
       verbose=True,
       structured_answer_filtering=True,
   )
   response = summarizer.get_response("who is president in the year 2050?", texts)
   Function call: StructuredRefineResponse with args: {
     "answer": "There is not enough context information to determine who is the president in the year 2050.",
     "query_satisfied": false
   }
   > Refine context: The president in the year 2050 is Florence Pugh...
   Function call: StructuredRefineResponse with args: {
     "answer": "Florence Pugh",
     "query_satisfied": true
   }
   > Refine context: The president in the year 2060 is Dwayne "The R...
   Function call: StructuredRefineResponse with args: {
     "answer": "Florence Pugh",
     "query_satisfied": false
   }
Successful Result
As you can see, we were able to determine the correct answer from the
given context by filtering the "texts" strings for the ones that
actually contained the answer to our question.
   print(response)
   Florence Pugh
Non Function-calling LLMs
You may want to make use of this filtering functionality with an LLM
that doesn't offer a function calling API.
In that case, the "Refine" module will automatically switch to using a
structured output "Program" that doesn't rely on an external function
calling API.
   # we'll stick with OpenAI but use an older model that does not support function calling
   davinci_llm = OpenAI(model="text-davinci-003")

Refine with Structured Answer Filtering

   from llama_index import ServiceContext
   from llama_index.response_synthesizers import get_response_synthesizer
   davinci_service_context = ServiceContext.from_defaults(llm=davinci_llm)
   summarizer = get_response_synthesizer(
       response_mode="refine",
       service_context=davinci_service_context,
       verbose=True,
       structured_answer_filtering=True,
   )
   response = summarizer.get_response("who is president in the year 2050?", texts)
   print(response)
   > Refine context: The president in the year 2050 is Florence Pugh...
   > Refine context: The president in the year 2060 is Dwayne "The R...
   Florence Pugh is the president in the year 2050 and Dwayne "The Rock" Johnson is the president in the year 2060.
"CompactAndRefine"
Since "CompactAndRefine" is built on top of "Refine", this response
mode also supports structured answer filtering.
   from llama_index.response_synthesizers import get_response_synthesizer
   summarizer = get_response_synthesizer(
       response_mode="compact",
       service_context=service_context,
       verbose=True,
       structured_answer_filtering=True,
   )
   response = summarizer.get_response("who is president in the year 2050?", texts)
   print(response)
   Function call: StructuredRefineResponse with args: {
     "answer": "Florence Pugh",
     "query_satisfied": true
   }
   Florence Pugh


Pydantic Tree Summarize

In this notebook, we demonstrate how to use tree summarize with
structured outputs. Specifically, tree summarize is used to output
pydantic objects.
   import os
   import openai
   os.environ["OPENAI_API_KEY"] = "sk-..."
   openai.api_key = os.environ["OPENAI_API_KEY"]
Load Data
   from llama_index import SimpleDirectoryReader
   reader = SimpleDirectoryReader(
       input_files=["../data/paul_graham/paul_graham_essay.txt"]
   )
   docs = reader.load_data()
   text = docs[0].text
Summarize
   from llama_index.response_synthesizers import TreeSummarize
   from llama_index.types import BaseModel
   from typing import List
Create pydantic model to structure response
   class Biography(BaseModel):
       """Data model for a biography."""
       name: str
       best_known_for: List[str]
       extra_info: str
   summarizer = TreeSummarize(verbose=True, output_cls=Biography)
   response = summarizer.get_response("who is Paul Graham?", [text])
   5 text chunks after repacking
   1 text chunks after repacking
Inspect the response
Here, we see the response is in an instance of our "Biography" class.
   print(response)
   name='Paul Graham' best_known_for=['Writing', 'Programming', 'Art', 'Co-founding Viaweb', 'Co-founding Y Combinator', 'Essayist'] extra_info="Paul Graham is a multi-talented individual who has made significant contributions in various fields. He is known for his work in writing, programming, art, co-founding Viaweb, co-founding Y Combinator, and his essays on startups and programming. He started his career by writing short stories and programming on the IBM 1401 computer. He later became interested in artificial intelligence and Lisp programming. He wrote a book called 'On Lisp' and focused on Lisp hacking. Eventually, he decided to pursue art and attended art school. He is known for his paintings, particularly still life paintings. Graham is also a programmer, entrepreneur, and venture capitalist. He co-founded Viaweb, an early e-commerce platform, and Y Combinator, a startup accelerator. He has written influential essays on startups and programming. Additionally, he has made contributions to the field of computer programming and entrepreneurship."
   print(response.name)
   Paul Graham
   print(response.best_known_for)
   ['Writing', 'Programming', 'Art', 'Co-founding Viaweb', 'Co-founding Y Combinator', 'Essayist']
   print(response.extra_info)
   Paul Graham is a multi-talented individual who has made significant contributions in various fields. He is known for his work in writing, programming, art, co-founding Viaweb, co-founding Y Combinator, and his essays on startups and programming. He started his career by writing short stories and programming on the IBM 1401 computer. He later became interested in artificial intelligence and Lisp programming. He wrote a book called 'On Lisp' and focused on Lisp hacking. Eventually, he decided to pursue art and attended art school. He is known for his paintings, particularly still life paintings. Graham is also a programmer, entrepreneur, and venture capitalist. He co-founded Viaweb, an early e-commerce platform, and Y Combinator, a startup accelerator. He has written influential essays on startups and programming. Additionally, he has made contributions to the field of computer programming and entrepreneurship.


LlamaIndex + DeepEval Integration

This code tutorial shows how you can easily integrate LlamaIndex with
DeepEval. DeepEval makes it easy to unit-test your LLMs.
You can read more about the DeepEval framework here: https://docs
.confident-ai.com/docs/framework
Feel free to check out our repository here: https://github.com
/confident-ai/deepeval
[image: Framework][image]
Set-up and Installation
We recommend setting up and installing via pip!
   !pip install -q -q llama-index
   !pip install -U -q deepeval
This step is optional and only if you want a server-hosted dashboard!
(Psst I think you should!)
   !deepeval login
Testing for factual consistency
   from llama_index.response.schema import Response
   from typing import List
   from llama_index.schema import Document
   from deepeval.metrics.factual_consistency import FactualConsistencyMetric
Setting Up The Evaluator
Setting up the evaluator.
   from llama_index import (
       TreeIndex,
       VectorStoreIndex,
       SimpleDirectoryReader,
       LLMPredictor,
       ServiceContext,
       Response,
   )
   from llama_index.llms import OpenAI
   from llama_index.evaluation import FaithfulnessEvaluator
   import os
   import openai
   api_key = "sk-XXX"
   openai.api_key = api_key
   gpt4 = OpenAI(temperature=0, model="gpt-4", api_key=api_key)
   service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)
   evaluator_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4)
Getting a LlamaHub Loader
   from llama_index import download_loader
   WikipediaReader = download_loader("WikipediaReader")
   loader = WikipediaReader()
   documents = loader.load_data(pages=["Tokyo"])
   tree_index = TreeIndex.from_documents(documents=documents)
   vector_index = VectorStoreIndex.from_documents(
       documents, service_context=service_context_gpt4
   )
We then build an evaluator based on the "BaseEvaluator" class that
requires an "evaluate" method.
In this example, we show you how to write a factual consistency check.
   from typing import Any, Optional, Sequence
   from llama_index.evaluation.base import BaseEvaluator, EvaluationResult
   class FactualConsistencyEvaluator(BaseEvaluator):
       def evaluate(
           self,
           query: Optional[str] = None,
           contexts: Optional[Sequence[str]] = None,
           response: Optional[str] = None,
           **kwargs: Any,
       ) -> EvaluationResult:
           """Evaluate factual consistency metrics"""
           if response is None or contexts is None:
               raise ValueError('Please provide "response" and "contexts".')
           metric = FactualConsistencyMetric()
           context = " ".join([d for d in contexts])
           score = metric.measure(output=response, context=context)
           return EvaluationResult(
               response=response,
               contexts=contexts,
               passing=metric.is_successful(),
               score=score,
           )
   evaluator = FactualConsistencyEvaluator()
   query_engine = tree_index.as_query_engine()
   response = query_engine.query("How did Tokyo get its name?")
   eval_result = evaluator.evaluate_response(response=response)
   /usr/local/lib/python3.10/dist-packages/transformers/convert_slow_tokenizer.py:470: UserWarning: The sentencepiece tokenizer that you are converting to a fast tokenizer uses the byte fallback option which is not implemented in the fast tokenizers. In practice this means that the fast version of the tokenizer can produce unknown tokens whereas the sentencepiece version would have converted these unknown tokens into a sequence of byte tokens matching the original piece of text.
     warnings.warn(
   {'success': True, 'score': 0.97732705}
   /usr/local/lib/python3.10/dist-packages/deepeval/metrics/metric.py:42: UserWarning: API key is not set. Please set it by visiting https://app.confident-ai.com

LlamaIndex + DeepEval Integration

     warnings.warn(
Other Metrics
We recommend using other metrics to help give more confidence to
various prompt iterations, LLM outputs etc. We think ML-assisted
approaches are required to give performance for these models.
* Overall Score: https://docs.confident-
  ai.com/docs/measuring_llm_performance/overall_score
* Answer Relevancy: https://docs.confident-
  ai.com/docs/measuring_llm_performance/answer_relevancy
* Bias: https://docs.confident-
  ai.com/docs/measuring_llm_performance/debias


Relevancy Evaluator

This notebook uses the "RelevancyEvaluator" to measure if the response
+ source nodes match the query.This is useful for measuring if the
query was actually answered by the response.
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       TreeIndex,
       VectorStoreIndex,
       SimpleDirectoryReader,
       LLMPredictor,
       ServiceContext,
       Response,
   )
   from llama_index.llms import OpenAI
   from llama_index.evaluation import RelevancyEvaluator
   import pandas as pd
   pd.set_option("display.max_colwidth", 0)
   # gpt-3 (davinci)
   gpt3 = OpenAI(temperature=0, model="text-davinci-003")
   service_context_gpt3 = ServiceContext.from_defaults(llm=gpt3)
   # gpt-4
   gpt4 = OpenAI(temperature=0, model="gpt-4")
   service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)
   evaluator = RelevancyEvaluator(service_context=service_context_gpt3)
   evaluator_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4)
   documents = SimpleDirectoryReader("./test_wiki_data").load_data()
   # create vector index
   vector_index = VectorStoreIndex.from_documents(
       documents, service_context=ServiceContext.from_defaults(chunk_size=512)
   )
   # define jupyter display function
   def display_eval_df(query: str, response: Response, eval_result: str) -> None:
       eval_df = pd.DataFrame(
           {
               "Query": query,
               "Response": str(response),
               "Source": response.source_nodes[0].node.text[:1000] + "...",
               "Evaluation Result": "Pass" if eval_result.passing else "Fail",
           },
           index=[0],
       )
       eval_df = eval_df.style.set_properties(
           **{
               "inline-size": "600px",
               "overflow-wrap": "break-word",
           },
           subset=["Response", "Source"]
       )
       display(eval_df)
Evaluate Response
Evaluate response relative to source nodes as well as query.
   query_str = "What battles took place in New York City in the American Revolution?"
   query_engine = vector_index.as_query_engine()
   response_vector = query_engine.query(query_str)
   eval_result = evaluator_gpt4.evaluate_response(
       query=query_str, response=response_vector
   )
   display_eval_df(query_str, response_vector, eval_result)
   <pandas.io.formats.style.Styler at 0x7f9f219aa970>
   query_str = "What are the airports in New York City?"
   query_engine = vector_index.as_query_engine()
   response_vector = query_engine.query(query_str)
   eval_result = evaluator_gpt4.evaluate_response(
       query=query_str, response=response_vector
   )
   display_eval_df(query_str, response_vector, eval_result)
   <pandas.io.formats.style.Styler at 0x7f9f00c28c70>
   query_str = "Who is the mayor of New York City?"
   query_engine = vector_index.as_query_engine()
   response_vector = query_engine.query(query_str)
   eval_result = evaluator_gpt4.evaluate_response(
       query=query_str, response=response_vector
   )
   display_eval_df(query_str, response_vector, eval_result)
   <pandas.io.formats.style.Styler at 0x7f9f00c40790>
Evaluate Source Nodes
Evaluate the set of returned sources, and determine which sources
actually contain the answer to a given query.
   from typing import List
   # define jupyter display function

Relevancy Evaluator

   def display_eval_sources(
       query: str, response: Response, eval_result: List[str]
   ) -> None:
       sources = [s.node.get_text() for s in response.source_nodes]
       eval_df = pd.DataFrame(
           {
               "Source": sources,
               "Eval Result": eval_result,
           },
       )
       eval_df.style.set_caption(query)
       eval_df = eval_df.style.set_properties(
           **{
               "inline-size": "600px",
               "overflow-wrap": "break-word",
           },
           subset=["Source"]
       )
       display(eval_df)
   # NOTE: you can set response_mode="no_text" to get just the sources
   query_str = "What are the airports in New York City?"
   query_engine = vector_index.as_query_engine(similarity_top_k=3, response_mode="no_text")
   response_vector = query_engine.query(query_str)
   eval_source_result_full = [
       evaluator_gpt4.evaluate(
           query=query_str,
           response=response_vector.response,
           contexts=[source_node.get_content()],
       )
       for source_node in response_vector.source_nodes
   ]
   eval_source_result = [
       "Pass" if result.passing else "Fail" for result in eval_source_result_full
   ]
   display_eval_sources(query_str, response_vector, eval_source_result)
   <pandas.io.formats.style.Styler at 0x7f9f17afce20>
   # NOTE: you can set response_mode="no_text" to get just the sources
   query_str = "Who is the mayor of New York City?"
   query_engine = vector_index.as_query_engine(similarity_top_k=3, response_mode="no_text")
   eval_source_result_full = [
       evaluator_gpt4.evaluate(
           query=query_str,
           response=response_vector.response,
           contexts=[source_node.get_content()],
       )
       for source_node in response_vector.source_nodes
   ]
   eval_source_result = [
       "Pass" if result.passing else "Fail" for result in eval_source_result_full
   ]
   display_eval_sources(query_str, response_vector, eval_source_result)
   <pandas.io.formats.style.Styler at 0x17d6eab00>


Guideline Evaluator

This notebook shows how to use "GuidelineEvaluator" to evaluate a
question answer system given user specified guidelines.
   from llama_index.evaluation import GuidelineEvaluator
   from llama_index import ServiceContext
   from llama_index.llms import OpenAI
   # Needed for running async functions in Jupyter Notebook
   import nest_asyncio
   nest_asyncio.apply()
   GUIDELINES = [
       "The response should fully answer the query.",
       "The response should avoid being vague or ambiguous.",
       "The response should be specific and use statistics or numbers when possible.",
   ]
   service_context = ServiceContext.from_defaults(llm=OpenAI(model="gpt-4"))
   evaluators = [
       GuidelineEvaluator(service_context=service_context, guidelines=guideline)
       for guideline in GUIDELINES
   ]
   sample_data = {
       "query": "Tell me about global warming.",
       "contexts": [
           "Global warming refers to the long-term increase in Earth's average surface temperature due to human activities such as the burning of fossil fuels and deforestation.",
           "It is a major environmental issue with consequences such as rising sea levels, extreme weather events, and disruptions to ecosystems.",
           "Efforts to combat global warming include reducing carbon emissions, transitioning to renewable energy sources, and promoting sustainable practices.",
       ],
       "response": "Global warming is a critical environmental issue caused by human activities that lead to a rise in Earth's temperature. It has various adverse effects on the planet.",
   }
   for guideline, evaluator in zip(GUIDELINES, evaluators):
       eval_result = evaluator.evaluate(
           query=sample_data["query"],
           contexts=sample_data["contexts"],
           response=sample_data["response"],
       )
       print("=====")
       print(f"Guideline: {guideline}")
       print(f"Pass: {eval_result.passing}")
       print(f"Feedback: {eval_result.feedback}")
   =====
   Guideline: The response should fully answer the query.
   Pass: False
   Feedback: The response does not fully answer the query. While it does provide a brief overview of global warming, it does not delve into the specifics of the causes, effects, or potential solutions to the problem. The response should be more detailed and comprehensive to fully answer the query.
   =====
   Guideline: The response should avoid being vague or ambiguous.
   Pass: False
   Feedback: The response is too vague and does not provide specific details about global warming. It should include more information about the causes, effects, and potential solutions to global warming.
   =====
   Guideline: The response should be specific and use statistics or numbers when possible.
   Pass: False
   Feedback: The response is too general and lacks specific details or statistics about global warming. It would be more informative if it included data such as the rate at which the Earth's temperature is rising, the main human activities contributing to global warming, or the specific adverse effects on the planet.


QuestionGeneration

This notebook walks through the process of generating a list of
questions that could be asked about your data. This is useful for
setting up an evaluation pipeline using the "FaithfulnessEvaluator"
and "RelevancyEvaluator" evaluation tools.
   import logging
   import sys
   import pandas as pd
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index.evaluation import DatasetGenerator, RelevancyEvaluator
   from llama_index import (
       SimpleDirectoryReader,
       VectorStoreIndex,
       ServiceContext,
       LLMPredictor,
       Response,
   )
   from llama_index.llms import OpenAI
   reader = SimpleDirectoryReader("../data/paul_graham/")
   documents = reader.load_data()
   data_generator = DatasetGenerator.from_documents(documents)
   WARNING:llama_index.indices.service_context:chunk_size_limit is deprecated, please specify chunk_size instead
   chunk_size_limit is deprecated, please specify chunk_size instead
   chunk_size_limit is deprecated, please specify chunk_size instead
   chunk_size_limit is deprecated, please specify chunk_size instead
   chunk_size_limit is deprecated, please specify chunk_size instead
   eval_questions = data_generator.generate_questions_from_nodes()
   eval_questions
   ['What were the two main things the author worked on before college?',
    'How did the author describe their early attempts at writing short stories?',
    'What type of computer did the author first work on for programming?',
    'What language did the author use for programming on the IBM 1401?',
    "What was the author's experience with programming on the 1401?",
    'What type of computer did the author eventually get for themselves?',
    "What was the author's initial plan for college?",
    'What made the author change their mind about studying philosophy?',
    "What sparked the author's interest in AI?",
    'What did the author realize about AI during their first year of grad school?',
    'What were the two art schools that the author applied to?',
    'How did the author end up at RISD?',
    'What was the purpose of the foundation classes at RISD?',
    'How did the author manage to pass the entrance exam for the Accademia di Belli Arti?',
    'What was the arrangement between the students and faculty at the Accademia?',
    "What was the author's experience painting still lives in Florence?",
    'What did the author learn about visual perception while painting still lives?',
    'Why did the author decide to leave the Accademia and return to the US?',
    'What did the author learn about technology companies while working at Interleaf?',
    'What lesson did the author learn about the low end and high end in the software industry?',
    "What was the author's motivation for writing another book on Lisp?",
    'How did the author come up with the idea for starting a company to put art galleries online?',
    'What was the initial reaction of art galleries to the idea of being online?',
    'How did the author and his team come up with the concept of a web app?',
    'What were the three main parts of the software developed by the author and his team?',
    'How did the author and his team learn about retail and improve their software based on user feedback?',
    'Why did the author initially believe that the absolute number of users was the most important factor for a startup?',
    "What was the growth rate of the author's company and why was it significant?",
    "How did the author's decision to hire more people impact the financial stability of the company?",
    "What was the outcome of the company's acquisition by Yahoo in 1998?",
    "What was the author's initial reaction when Yahoo bought their startup?",
    "How did the author's lifestyle change after Yahoo bought their startup?",
    'Why did the author leave Yahoo and what did they plan to do?',

QuestionGeneration

    "What was the author's experience like when they returned to New York after becoming rich?",
    'What idea did the author have in the spring of 2000 and why did they decide to start a new company?',
    "Why did the author decide to build a subset of the new company's vision as an open source project?",
    "How did the author's perception of publishing essays change with the advent of the internet?",
    "What is the author's perspective on working on things that are not prestigious?",
    'What other projects did the author work on besides writing essays?',
    'What type of building did the author buy in Cambridge?',
    "What was the concept behind the big party at the narrator's house in October 2003?",
    "How did Jessica Livingston's perception of startups change after meeting friends of the narrator?",
    'What were some of the ideas that the narrator shared with Jessica about fixing venture capital?',
    'How did the idea of starting their own investment firm come about for the narrator and Jessica?',
    'What was the Summer Founders Program and how did it attract applicants?',
    "How did Y Combinator's batch model help solve the problem of isolation for startup founders?",
    "What advantages did YC's scale bring, both in terms of community and customer acquisition?",
    'Why did the narrator consider Hacker News to be a source of stress?',
    "How did the narrator's role in YC differ from other types of work they had done?",
    'What advice did Robert Morris offer the narrator during his visit in 2010?',
    'What was the advice given to the author by Rtm regarding their involvement with Y Combinator?',
    'Why did the author decide to hand over Y Combinator to someone else?',
    "What event in the author's personal life prompted them to reevaluate their priorities?",
    'How did the author spend most of 2014?',
    'What project did the author work on from March 2015 to October 2019?',
    'How did the author manage to write an interpreter for Lisp in itself?',
    "What was the author's experience like living in England?",
    "When was the author's project, Bel, finally finished?",
    'What did the author do during the fall of 2019?',
    "How would you describe the author's journey and decision-making process throughout the document?",
    "How did the author's experience with editing Lisp expressions differ from traditional app editing?",
    'Why did the author receive negative comments when claiming that Lisp was better than other languages?',
    'What is the difference between putting something online and publishing it online?',
    'How did the customs of venture capital practice and essay writing reflect outdated constraints?',
    'Why did Y Combinator change its name to avoid a regional association?',
    "What was the significance of the orange color chosen for Y Combinator's logo?",
    'Why did Y Combinator become a fund for a couple of years before returning to self-funding?',
    'What is the purpose of Y Combinator in relation to the concept of "deal flow"?',
    'How did the combination of running a forum and writing essays lead to a problem for the author?',
    "What was the author's biggest regret about leaving Y Combinator?"]
   # gpt-4
   gpt4 = OpenAI(temperature=0, model="gpt-4")
   service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)
   evaluator_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4)
   # create vector index
   vector_index = VectorStoreIndex.from_documents(
       documents, service_context=service_context_gpt4
   )
   # define jupyter display function
   def display_eval_df(query: str, response: Response, eval_result: str) -> None:
       eval_df = pd.DataFrame(

QuestionGeneration

           {
               "Query": query,
               "Response": str(response),
               "Source": response.source_nodes[0].node.get_content()[:1000] + "...",
               "Evaluation Result": eval_result,
           },
           index=[0],
       )
       eval_df = eval_df.style.set_properties(
           **{
               "inline-size": "600px",
               "overflow-wrap": "break-word",
           },
           subset=["Response", "Source"]
       )
       display(eval_df)
   query_engine = vector_index.as_query_engine()
   response_vector = query_engine.query(eval_questions[1])
   eval_result = evaluator_gpt4.evaluate_response(
       query=eval_questions[1], response=response_vector
   )
   display_eval_df(eval_questions[1], response_vector, eval_result)
   <pandas.io.formats.style.Styler at 0x7fcb78d7f130>


HotpotQADistractor Demo

This notebook walks through evaluating a query engine using the
HotpotQA dataset. In this task, the LLM must answer a question given a
pre-configured context. The answer usually has to be concise, and
accuracy is measured by calculating the overlap (measured by F1) and
exact match.
   from llama_index.evaluation.benchmarks import HotpotQAEvaluator
   from llama_index import ServiceContext, VectorStoreIndex
   from llama_index.schema import Document
   from llama_index.llms import OpenAI
   llm = OpenAI(model="gpt-3.5-turbo")
   service_context = ServiceContext.from_defaults(
       embed_model="local:sentence-transformers/all-MiniLM-L6-v2",
       llm=llm,
   )
   index = VectorStoreIndex.from_documents(
       [Document.example()], service_context=service_context, show_progress=True
   )
   Parsing documents into nodes: 100%|██████████| 1/1 [00:00<00:00, 129.13it/s]
   Generating embeddings: 100%|██████████| 1/1 [00:00<00:00, 36.62it/s]
First we try with a very simple engine. In this particular benchmark,
the retriever and hence index is actually ignored, as the documents
retrieved for each query is provided in the dataset. This is known as
the "distractor" setting in HotpotQA.
   engine = index.as_query_engine(service_context=service_context)
   HotpotQAEvaluator().run(engine, queries=5, show_result=True)
   Dataset: hotpot_dev_distractor downloaded at: /Users/loganmarkewich/Library/Caches/llama_index/datasets/HotpotQA
   Evaluating on dataset: hotpot_dev_distractor
   -------------------------------------
   Loading 5 queries out of 7405 (fraction: 0.00068)
   Question:  Were Scott Derrickson and Ed Wood of the same nationality?
   Response: No.
   Correct answer:  yes
   EM: 0 F1: 0
   -------------------------------------
   Question:  What government position was held by the woman who portrayed Corliss Archer in the film Kiss and Tell?
   Response: Unknown
   Correct answer:  Chief of Protocol
   EM: 0 F1: 0
   -------------------------------------
   Question:  What science fantasy young adult series, told in first person, has a set of companion books narrating the stories of enslaved worlds and alien species?
   Response: Animorphs
   Correct answer:  Animorphs
   EM: 1 F1: 1.0
   -------------------------------------
   Question:  Are the Laleli Mosque and Esma Sultan Mansion located in the same neighborhood?
   Response: Yes.
   Correct answer:  no
   EM: 0 F1: 0
   -------------------------------------
   Question:  The director of the romantic comedy "Big Stone Gap" is based in what New York city?
   Response: Greenwich Village
   Correct answer:  Greenwich Village, New York City
   EM: 0 F1: 0.5714285714285715
   -------------------------------------
   Scores:  {'exact_match': 0.2, 'f1': 0.31428571428571433}
Now we try with a sentence transformer reranker, which selects 3 out
of the 10 nodes proposed by the retriever
   from llama_index.indices.postprocessor import SentenceTransformerRerank
   rerank = SentenceTransformerRerank(top_n=3)
   engine = index.as_query_engine(
       service_context=service_context,
       node_postprocessors=[rerank],
   )
   HotpotQAEvaluator().run(engine, queries=5, show_result=True)

HotpotQADistractor Demo

   Dataset: hotpot_dev_distractor downloaded at: /Users/loganmarkewich/Library/Caches/llama_index/datasets/HotpotQA
   Evaluating on dataset: hotpot_dev_distractor
   -------------------------------------
   Loading 5 queries out of 7405 (fraction: 0.00068)
   Question:  Were Scott Derrickson and Ed Wood of the same nationality?
   Response: No.
   Correct answer:  yes
   EM: 0 F1: 0
   -------------------------------------
   Question:  What government position was held by the woman who portrayed Corliss Archer in the film Kiss and Tell?
   Response: No government position.
   Correct answer:  Chief of Protocol
   EM: 0 F1: 0
   -------------------------------------
   Question:  What science fantasy young adult series, told in first person, has a set of companion books narrating the stories of enslaved worlds and alien species?
   Response: Animorphs
   Correct answer:  Animorphs
   EM: 1 F1: 1.0
   -------------------------------------
   Question:  Are the Laleli Mosque and Esma Sultan Mansion located in the same neighborhood?
   Response: No.
   Correct answer:  no
   EM: 1 F1: 1.0
   -------------------------------------
   Question:  The director of the romantic comedy "Big Stone Gap" is based in what New York city?
   Response: New York City.
   Correct answer:  Greenwich Village, New York City
   EM: 0 F1: 0.7499999999999999
   -------------------------------------
   Scores:  {'exact_match': 0.4, 'f1': 0.55}
The F1 and exact match scores appear to improve slightly.
Note that the benchmark optimizes for producing short factoid answers
without explanations, although it is known that CoT prompting can
sometimes help in output quality.
The scores used are also not a perfect measure of correctness, but can
be a quick way to identify how changes in your query engine change the
output.


Pairwise Evaluator

This notebook uses the "PairwiseEvaluator" module to see if an
evaluation LLM would prefer one query engine over another.
   # attach to the same event-loop
   import nest_asyncio
   nest_asyncio.apply()
   # configuring logger to INFO level
   import logging
   import sys
   logging.basicConfig(stream=sys.stdout, level=logging.INFO)
   logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))
   from llama_index import (
       VectorStoreIndex,
       SimpleDirectoryReader,
       ServiceContext,
       Response,
   )
   from llama_index.llms import OpenAI
   from llama_index.evaluation import PairwiseComparisonEvaluator
   import pandas as pd
   pd.set_option("display.max_colwidth", 0)
   INFO:numexpr.utils:Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   Note: NumExpr detected 12 cores but "NUMEXPR_MAX_THREADS" not set, so enforcing safe limit of 8.
   INFO:numexpr.utils:NumExpr defaulting to 8 threads.
   NumExpr defaulting to 8 threads.
Using GPT-4 here for evaluation
   # gpt-4
   gpt4 = OpenAI(temperature=0, model="gpt-4")
   service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)
   evaluator_gpt4 = PairwiseComparisonEvaluator(service_context=service_context_gpt4)
   documents = SimpleDirectoryReader("./test_wiki_data/").load_data()
   # create vector index
   service_context1 = ServiceContext.from_defaults(chunk_size=512)
   vector_index1 = VectorStoreIndex.from_documents(
       documents, service_context=service_context1
   )
   service_context2 = ServiceContext.from_defaults(chunk_size=128)
   vector_index2 = VectorStoreIndex.from_documents(
       documents, service_context=service_context2
   )
   query_engine1 = vector_index1.as_query_engine(similarity_top_k=2)
   query_engine2 = vector_index2.as_query_engine(similarity_top_k=8)
   # define jupyter display function
   def display_eval_df(query, response1, response2, eval_result) -> None:
       eval_df = pd.DataFrame(
           {
               "Query": query,
               "Reference Response (Answer 1)": response2,
               "Current Response (Answer 2)": response1,
               "Score": eval_result.score,
               "Reason": eval_result.feedback,
           },
           index=[0],
       )
       eval_df = eval_df.style.set_properties(
           **{
               "inline-size": "300px",
               "overflow-wrap": "break-word",
           },
           subset=["Current Response (Answer 2)", "Reference Response (Answer 1)"]
       )
       display(eval_df)
To run evaluations you can call the ".evaluate_response()" function on
the "Response" object return from the query to run the evaluations.
Lets evaluate the outputs of the vector_index.
   # query_str = "How did New York City get its name?"
   query_str = "What was the role of NYC during the American Revolution?"
   # query_str = "Tell me about the arts and culture of NYC"
   response1 = str(query_engine1.query(query_str))
   response2 = str(query_engine2.query(query_str))
By default, we enforce "consistency" in the pairwise comparison.
We try feeding in the candidate, reference pair, and then swap the
order of the two, and make sure that the results are still consistent
(or return a TIE if not).
   eval_result = await evaluator_gpt4.aevaluate(
       query_str, response=response1, reference=response2
   )
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=5536 request_id=8a8f154ee676b2e86ea24b7046e9b80b response_code=200

Pairwise Evaluator

   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=5536 request_id=8a8f154ee676b2e86ea24b7046e9b80b response_code=200
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=9766 request_id=dfee84227112b1311b4411492f4c8764 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=9766 request_id=dfee84227112b1311b4411492f4c8764 response_code=200
   display_eval_df(query_str, response1, response2, eval_result)
   <pandas.io.formats.style.Styler at 0x2bfe67010>
**NOTE**: By default, we enforce consensus by flipping the order of
response/reference and making sure that the answers are opposites.
We can disable this - which can lead to more inconsistencies!
   evaluator_gpt4_nc = PairwiseComparisonEvaluator(
       service_context=service_context_gpt4, enforce_consensus=False
   )
   eval_result = await evaluator_gpt4_nc.aevaluate(
       query_str, response=response1, reference=response2
   )
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=6714 request_id=472a1f0829846adc1b4347ba4b99c0dd response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=6714 request_id=472a1f0829846adc1b4347ba4b99c0dd response_code=200
   display_eval_df(query_str, response1, response2, eval_result)
   <pandas.io.formats.style.Styler at 0x2b9a9bd30>
   eval_result = await evaluator_gpt4_nc.aevaluate(
       query_str, response=response2, reference=response1
   )
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=9252 request_id=b73bbe6b10d878ed8138785638232866 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=9252 request_id=b73bbe6b10d878ed8138785638232866 response_code=200
   display_eval_df(query_str, response2, response1, eval_result)
   <pandas.io.formats.style.Styler at 0x2b9a98a00>
Running on some more Queries
   query_str = "Tell me about the arts and culture of NYC"
   response1 = str(query_engine1.query(query_str))
   response2 = str(query_engine2.query(query_str))
   eval_result = await evaluator_gpt4.aevaluate(
       query_str, response=response1, reference=response2
   )
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=6053 request_id=749fdbde59bf8d1056a8be6e211d20d9 response_code=200
   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=6053 request_id=749fdbde59bf8d1056a8be6e211d20d9 response_code=200
   INFO:openai:message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=7309 request_id=ba09bb38320b60cf09dbebb1df2c732b response_code=200

Pairwise Evaluator

   message='OpenAI API response' path=https://api.openai.com/v1/chat/completions processing_ms=7309 request_id=ba09bb38320b60cf09dbebb1df2c732b response_code=200
   display_eval_df(query_str, response1, response2, eval_result)
   <pandas.io.formats.style.Styler at 0x2baf33850>


BEIR Out of Domain Benchmark

About BEIR:
BEIR is a heterogeneous benchmark containing diverse IR tasks. It also
provides a common and easy framework for evaluation of your retrieval
methods within the benchmark.
Refer to the repo via the link for a full list of supported datasets.
Here, we test the "all-MiniLM-L6-v2" sentence-transformer embedding,
which is one of the fastest for the given accuracy range. We set the
top_k value for the retriever to 30. We also use the nfcorpus dataset.
   from llama_index.embeddings import HuggingFaceEmbedding
   from llama_index.evaluation.benchmarks import BeirEvaluator
   from llama_index import ServiceContext, VectorStoreIndex
   def create_retriever(documents):
       embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-small-en-v1.5")
       service_context = ServiceContext.from_defaults(embed_model=embed_model)
       index = VectorStoreIndex.from_documents(
           documents, service_context=service_context, show_progress=True
       )
       return index.as_retriever(similarity_top_k=30)
   BeirEvaluator().run(
       create_retriever, datasets=["nfcorpus"], metrics_k_values=[3, 10, 30]
   )
   /home/jonch/.pyenv/versions/3.10.6/lib/python3.10/site-packages/beir/datasets/data_loader.py:2: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
     from tqdm.autonotebook import tqdm
   Dataset: nfcorpus downloaded at: /home/jonch/.cache/llama_index/datasets/BeIR__nfcorpus
   Evaluating on dataset: nfcorpus
   -------------------------------------
   100%|███████████████████████████████████| 3633/3633 [00:00<00:00, 141316.79it/s]
   Parsing documents into nodes: 100%|████████| 3633/3633 [00:06<00:00, 569.35it/s]
   Generating embeddings: 100%|████████████████| 3649/3649 [04:22<00:00, 13.92it/s]
   Retriever created for:  nfcorpus
   Evaluating retriever on questions against qrels
   100%|█████████████████████████████████████████| 323/323 [01:26<00:00,  3.74it/s]
   Results for: nfcorpus
   {'NDCG@3': 0.35476, 'MAP@3': 0.07489, 'Recall@3': 0.08583, 'precision@3': 0.33746}
   {'NDCG@10': 0.31403, 'MAP@10': 0.11003, 'Recall@10': 0.15885, 'precision@10': 0.23994}
   {'NDCG@30': 0.28636, 'MAP@30': 0.12794, 'Recall@30': 0.21653, 'precision@30': 0.14716}
   -------------------------------------
Higher is better for all the evaluation metrics.
This towardsdatascience article covers NDCG, MAP and MRR in greater
depth.