what is the difference and similarities between this project: ""# Cohere Rerank Endpoint (public-beta)
In the past months, we engineered a novel relevance endpoint, that takes a query and a list of documents and predicts the relevance between the query and each document.

It can be used in a two-stage retrieval setup: First you take the user question, and retrieve the top-100 documents from your collection by either using lexical search or semantic search.

You then pass the question and these top-100 documents to our relevance-endpoint to get a score for each document. You can then rank these documents based on these scores.

In our benchmarks across 20 datasets, we **saw significant improvements compared to lexical and semantic search**, especially for use-cases where no training data is available.

We will demonstrate the rerank endpoint in this notebook.



! pip install cohere

import cohere
import requests
import numpy as np
from time import time
from typing import List
from pprint import pprint

# Set up your cohere client
API_KEY = "{YOUR API KEY HERE}"
co = cohere.Client(API_KEY)

# Example query and passages (data taken from http://sbert.net/datasets/simplewiki-2020-11-01.jsonl.gz)
query = "What is the capital of the United States?"
docs = [
    "Carson City is the capital city of the American state of Nevada. At the 2010 United States Census, Carson City had a population of 55,274.",
    "The Commonwealth of the Northern Mariana Islands is a group of islands in the Pacific Ocean that are a political division controlled by the United States. Its capital is Saipan.",
    "Charlotte Amalie is the capital and largest city of the United States Virgin Islands. It has about 20,000 people. The city is on the island of Saint Thomas.",
    "Washington, D.C. (also known as simply Washington or D.C., and officially as the District of Columbia) is the capital of the United States. It is a federal district. The President of the USA and many major national government offices are in the territory. This makes it the political center of the United States of America.",
    "West Virginia is a state in the Appalachian region of the United States. Its capital and largest city is Charleston. It is often abbreviated W. Va. or simply WV.",
    "Capital punishment (the death penalty) has existed in the United States since before the United States was a country. As of 2017, capital punishment is legal in 30 of the 50 states. The federal government (including the United States military) also uses capital punishment.",
    "North Dakota is a state in the United States. 672,591 people lived in North Dakota in the year 2010. The capital and seat of government is Bismarck.",
    "Kentucky is a state in the United States. Its capital is Frankfort. It touches the states of Missouri (by the Mississippi River), Illinois, Indiana, Ohio, West Virginia (by the Ohio River), Tennessee and Virginia. There are many rivers in Kentucky",
    "Micronesia, officially the Federated States of Micronesia, is an island nation in the Pacific Ocean, northeast of Papua New Guinea. The country is a sovereign state in free association with the United States. The capital city of Federated States of Micronesia is Palikir.",
    "Utah is a state in the west United States. The capital and largest city is Salt Lake City. Utah became a state in the U.S. on January 4, 1896."]

## Using the Endpoint
In the following cell we will call rerank to rank `docs` based on how relevant they are with `query`.



results = co.rerank(query=query, documents=docs, top_n=3, model='rerank-english-v2.0') # Change top_n to change the number of results returned. If top_n is not passed, all results will be returned.
for idx, r in enumerate(results):
  print(f"Document Rank: {idx + 1}, Document Index: {r.index}")
  print(f"Document: {r.document['text']}")
  print(f"Relevance Score: {r.relevance_score:.2f}")
  print("\n")

## Search on  Wikipedia - End2end demo
The following is an example how to use this model end-to-end to search over the Simple English Wikipedia, which consists of about 500k passages.

We use BM25 lexical search to retrieve the top-100 passages matching the query. We then send these 100 passages and the query to our rerank endpoint to get a re-ranked list. We then output the top-3 hits according to BM25 lexical search (as used by e.g. Elasticsearch) and the re-ranked list from our endpoint.


!pip install -U  rank_bm25

import json
import gzip
import os
from rank_bm25 import BM25Okapi
from sklearn.feature_extraction import _stop_words
import string
from tqdm.autonotebook import tqdm


!wget http://sbert.net/datasets/simplewiki-2020-11-01.jsonl.gz

# As dataset, we use Simple English Wikipedia. Compared to the full English wikipedia, it has only
# about 170k articles. We split these articles into paragraphs and encode them with the bi-encoder
wikipedia_filepath = 'simplewiki-2020-11-01.jsonl.gz'

passages = []
with gzip.open(wikipedia_filepath, 'rt', encoding='utf8') as fIn:
    for line in fIn:
        data = json.loads(line.strip())
        passages.extend(data['paragraphs'])

print("Passages:", len(passages))

print(passages[0], passages[1])

# We  compare the results to lexical search (keyword search). Here, we use
# the BM25 algorithm which is implemented in the rank_bm25 package.

# We lower case our text and remove stop-words from indexing
def bm25_tokenizer(text):
    tokenized_doc = []
    for token in text.lower().split():
        token = token.strip(string.punctuation)

        if len(token) > 0 and token not in _stop_words.ENGLISH_STOP_WORDS:
            tokenized_doc.append(token)
    return tokenized_doc


tokenized_corpus = []
for passage in tqdm(passages):
    tokenized_corpus.append(bm25_tokenizer(passage))

bm25 = BM25Okapi(tokenized_corpus)

# This function will search all wikipedia articles for passages that
# answer the query. We then re-rank using our /rank endpoint

def search(query, top_k=3, num_candidates=100):
    print("Input question:", query)

    ##### BM25 search (lexical search) #####
    bm25_scores = bm25.get_scores(bm25_tokenizer(query))
    top_n = np.argpartition(bm25_scores, -num_candidates)[-num_candidates:]
    bm25_hits = [{'corpus_id': idx, 'score': bm25_scores[idx]} for idx in top_n]
    bm25_hits = sorted(bm25_hits, key=lambda x: x['score'], reverse=True)

    print(f"Top-3 lexical search (BM25) hits")
    for hit in bm25_hits[0:top_k]:
        print("\t{:.3f}\t{}".format(hit['score'], passages[hit['corpus_id']].replace("\n", " ")))


    #Add re-ranking
    docs = [passages[hit['corpus_id']] for hit in bm25_hits]

    print(f"\nTop-3 hits by rank-API ({len(bm25_hits)} BM25 hits re-ranked)")
    results = co.rerank(query=query, documents=docs, top_n=top_k, model='rerank-english-v2.0')
    for hit in results:
        print("\t{:.3f}\t{}".format(hit.relevance_score, hit.document["text"].replace("\n", " ")))



search(query = "What is the capital of the United States?")

search(query = "Number countries Europe")

search(query = "Elon Musk year birth")

search(query = "Which US president was killed?")

search(query="When is Chinese New Year")

search(query="How many people live in Paris")

search(query="Who is the director of The Matrix?")""

and this project: ""I'm going to talk about allsort it's a technique that uses smaller language models to identify which pieces of very long text should be injected into a final prompt you send for evaluation now I know what you're thinking this might be slower and it might be more costly but there are some interesting tricks and I think they're going to give you a performance speed trade-off you're going to enjoy the easiest way to explain elsort is just by going through an example now the idea overall is to break the input database into chunks which is the same as doing traditional Vector database search but instead of using Vector search to determine which chunks to include we're going to use a language model to determine whether the chunks are relevant and the reason this is good is because language models are very good at telling if a chunk is going to be relevant to a question so the problem at hand is given a chunk of text and given a question can we ask the language mod model if that chunk is relevant to the question so here I have laid out prompt and I'm just going to run it uh using open AI so I have a sample text which you can think of just as a text file and a snippet from that file here it's very short typically it might be say 500 tokens so the snippet is the ostrich is 4T tall and I want the language model to tell me whether or not on a scale of 1 to 5 this is going to be relevant to the question which is how tall is the ostrich so you can see I've got this nicely formatted prompt I'm asking the language model to carefully consider the relevance of the snippet to the question and provide a score from 1 to five where one is least relevant and five is most relevant and use the full range of scores to indicate the degree of relevance so when you run that query H you already saw the answer it'll output five and indeed that's correct because it's very relevant now just to show you this works if I take the snippet and I just put something in here like like apples so now I'm asking if the question how tall is the ostrich is relevant to Apples and here when I submit it gives back a value of one so you can see the language model is correctly telling us the relevance of the snippet to the question now you'll notice I'm asking the language model to only respond with the number and this is important because by doing this the language model only has to respond with one single token either a one a two a three a four or a five five and this makes it really fast for the language model to rate this snippet much faster then if we're asking the language model to extract what's relevant from the snippet and so now you can think of us chaining many of these requests for larger Snippets and very quickly hitting a GPU in parallel and getting the relevance for each of those Snippets that allows us to build a Cascade of Snippets and rank them and then take the most relevant ones and put those into to a question that is going to be part of our final evaluation so here's how that works in graphical form you have a database this is just text it could be many documents the very same as you'd have a database in Vector search and you split this database into chunks again not unlike typical rag based Vector search but instead of using Vector search you're going to use a language model to rate their relevance using the method I just described and what this allows then is to have a list of sorted chunks instead of being sorted by something like cosign similarity or dot product they're just sorted by the relevance that the language model has provided them so now that you have this sorted list of chunks we can take the most relevant chunks and use them as the context for the query that we want to answer now the specific way I'm going to choose the chunks is I'm going to include all chunks that are deemed to be four or five out of five so that means we'll include all of the most relevant and you'll find the language model doesn't find too many of those so we don't get overcrowded and I'm also going to insist on a minimum of three chunks so even if there's only one chunk that's five that means that my algorithm is going to go down all the way even as far as three or two as one to make sure to include at least three chunks and if there are any ties it's just going to choose chunks at random but hopefully that only happens towards the bottom of the pecking order if there really aren't many relevant chunks now when you pull all of this together and if you use a model that's strong enough for doing the relevance grading you're able to get some very nice results and I'll go through this in a little more detail later but here are the final results I have three columns I have first off allsort which is the method I just described I then have a full context text test so here I've put in the full about 100K of context into GPT 4 Turbo just straight away and then I also have taken a simpler rag based approach where I use Vector search to pull out the three most relevant Snippets that are 500 tokens in length and what you can see is across this body of nine questions that I've selected and I'll describe them in more detail I'm able to get all of them correct using this all sort method uh which is better than just putting all that context into GPT 4 and it's also better than the rag technique clearly you should note that this is a limited set of questions and this isn't necessarily going to generalize to all use cases but my goal today is to explain to you the intuition of why this kind of technique can perform well and what its advantages and disadvantages are against using full context or against using traditional rag I'm going to go through some of these prompts in more detail later but to give you a very cursory overview of why there's a difference in performance in short rag sometimes will miss out on specific pieces of text text and so it doesn't provide the context that's needed for GPT 4 to answer and in the case of providing the full context to GPT 4 while it's generally a strong model it sometimes just focuses on the wrong thing and it gets confused by one relevant piece of information and answers on that basis rather than specifically what's been referred to in the question so there are a few nuances um and details I want to go through in implementing this approach I'll start out by describing how I developed the nine test questions I just showed and that'll help you understand how you could develop your own questions and set up your own database I then will show you how I've generated the answers in the case of providing the full context and that's quite simple I literally dump the full context into the prompt then I'll describe the standard rag setup that I've used where I pick the three uh closest Snippets of 500 tokens and then I'll go into detail on the all sort approach there's something here a trick that's really useful and it's called reject responses there is a package that has come out it's been very popular on GitHub called outlines and this functionality has also been integrated into text generation inference by hugging face which is the API I typically use when I run servers and this API now has a grammar field that allows you to specify specifically the format that the response has to take and this avoids the llm response bonding uh in any spous way you might have been wondering if that would be an issue when you sh saw my first example but this is how we can 100% get the llm to respond with 1 2 3 four or five there's no other response that the llm can possibly give I will talk then about how once I've got the relevant scores I classify the chunks and sort them and I'll show you how I deploy an endpoint that's used as my helper API I'm using the smog 34 beam model uh it actually performs quite a bit better than the underlying e 34b model um so I'll show you quickly how to do that then I'll do a live demo running all sort on a prompt it takes um just around a minute to run the prompt now it is unoptimized and I think there could be quite a lot of time squeezed from that last off I'll go through the costing of using this all sort type of approach versus gp4 and I'll go through the latency as well compared ing it to gb4 Turbo using the full context and also using a vector search based rag method last I'll show you a few quick helper model ablations so I did try and use open chat 7B as a helper model it doesn't perform quite as well although it's somewhat similar to rag in terms in terms of score and I also looked at using Mixr which was a bit weaker than Smog and also the E model which was quite a bit weaker than Smog on which ye is originally based and for this video we'll be working out of the advanced inference repository it's available for purchase you can just buy the scripts for this video on tr.com or you can also buy lifetime access which includes future projects that get uploaded to this repository so looking at the repo contents here um we can take a quick look at the different API uh call scripts that I have and the one we're focused on here is going to be llm rag which is data retrieval using uh all sort so I'm going to go into the API calls folder and then I'm going to go into llm Rag and here I have the readme file that will explain everything on getting set up after setting up a virtual environment then I'm going to install the packages uh so in vs code is what I recommend and next prepare the input files so here's what I'm going to do for input files and I'm going to try and make it a bit tricky um I have two pieces of context one piece of context is the Burkshire hathway annual meeting 23 this is a 5-hour video and I've taken the transcript from YouTube um actually I've taken the sound and I've converted it into text using whisper which results generally in a better quality transcript so I have this and it's maybe about 55,000 tokens in length and the second thing I've done is I've taken the annual report uh for 23 which is much more recent it was just released within the last month so I think around February March of this year and this provides commentary which is roughly about let's say 8 nine months later than when the annual meeting was and so there are some subtle differences uh because of changes in business and markets between what was discussed in the meeting last year which I'm calling meeting and the report which is what I'm showing here right now and this allows a basis for providing both of these as context to the language model and asking some kind of tricky questions about what sentiments have changed um between the LA meeting last year and the meeting this year okay so with those two um files in hand and as I said I extracted the text using whisper from uh YouTube you can check out the video on transcription from trellis if you want to see how to do that and then I took the PDF and I ran a PDF to text script that's in this repository to convert it into text so I have about 55,000 tokens of the annual meeting text and I've about 13,000 tokens from the annual report and these are the two sources of context I'm going to use for my queries okay so next I generated a few questions manually and I can show these best if I go over to VSS code um quite simply within the input files in LM rag I've generated a CSV I've got nine questions I've got the questions and I've got the answers now it's easier actually to see if I just show you in a web browser so let's take a look through the questions here I'll very quickly go through them so you can appreciate the difficulty the first question here is a retrieval question H asking why the manager didn't want to sell uh to a competitor and of course he wanted to he preferred to sell to birkshire which he did um because selling to a competitor uh the argument is that maybe the staff would have been fired next question is where Buffett feels more comfortable with his Capital either Taiwan or Japan so here uh his thoughts have kind of changed because of the geopolitical situation the third question is pure retrieval it's asking what Vicki hb's first job was Vicki holl is the CEO of ocidental petroleum and her first job was at City service it's kind of an obscure reference that's just deep within uh the annual report Vicki Hollow is also mentioned quite a lot so that makes it kind of tricky because the language model has to pick out the ex right reference by the way her name is mentioned many times throughout the report and that makes it kind of tricky to answer the question next question here is B Buffett sentiments on a new tax law and in particular does he understand the implications for the tax Burk here will pay uh the answer here is that it's a bit unclear because uh there's no regulation that says whether unrealized gains will be treated as taxable or not uh so a retrieval based question but a little bit more nuanced the next one here is about critical differences between the business of ocidental compared to traditional oil ocidental is a Shale business so the production generally will fall very fast from those Wells compared to traditional uh the next question is the question of industrial capacity and why does Buffett talk about industrial capacity of the US during the second world war and the reason is and it's a little bit oblique but the reason is because he's trying to draw analogy with what needs to happen for the energy transition and and why that's going to be difficult because it would be difficult to operate in the way the US state operated during the second world war now I have three final questions that ask specifically the model to do they asked specifically for a comparison between sentiment last year and this year one is around birkshire hathway as a business so both birkshire haway and the rail business in the next question were doing very well last year um but actually during the end of the year there were significant concerns that Buffett saw and that he mentions in the report so Buffett sentiment changes significantly and we can test whether the language model is able to pick that up and last of all there's a similar question around apple apple is not actually mentioned very much in the annual report uh if the model was really smart and appreciated all of the previous uh birkar hathway meetings which should be in the training context then it would realize that Buffett not mentioning apple is maybe significant in the current report because he mentions them a lot in last year's meeting anyway we can see if the language model is able to appreciate that kind of nuance with that little overview of the questions we can take a look at how I prepare the prompt when I inject the full context in and send it to GPT 4 Turbo just to be clear in every single case in this video I'm using GPT 4 to do the final evaluation the difference between these three columns is how I prepared it context in the full case I literally inject the full context as we'll see in the r case I inject the top three 500 token Snippets and in the allsort case I insert the Snippets that are either rated four or five but I make sure I've got a minimum of three Snippets that are included I've now described how I prepared the questions and I'm going to move to show you quite quickly how I prepare the prompt for the case of full context and I can actually look at a prompt that I have saved uh it's available here in the output files Under full contact GPT 4 Turbo preview and this here is the first prompt and quite simply the way I set up the prompt is I start by having tags for the context and then within that I have tags for each document so here you can see the annual meeting and I've got the full 55 or so thousand tokens that are included in that now once I've included all that content I close the tag on the meeting and I open a tag for the annal report and then I inject the full contents of the report right here now I haven't done any formatting other than my conversion from PDF to text with that all injected we then have the prompt question itself so you can see I close here with a tag for the annual report and then I close the tag on the context and I say in answering the following question you may choose if appropriate to make use of the above context and then I say why didn't the manager of TTI want to sell the business to a competitor so really simple I'm literally injecting the full context in this scenario here next up I'm going to show you how I do standard rag search which is a vector-based approach if you want to use a vector-based approach for retrieval augmented generation then you need a model to generate the vectors or embeddings that's the same word that's been used often and what an embedding model does is it takes in tokens and it converts them so all the tokens within a phrase into a vector representing the meaning of that and now you can think of all of your text split into 500 token chunks is a series of arrows so each 500 chunk is an arrow and quite simply to determine which parts of the text are relevant to the question we convert the question into an arrow and compare the direction of the question with each of the chunks in the text The Way We compare them is by looking at the angle between them if the angle between them is small then that means they're probably closer in meaning and so we're going to pick the ones that are closest in angle and closest in meaning which is why we talk about cosine search now technically you can look at either cosine search which is the angle between them or you can look at the dot product the dot product considers not just the angle between the query and the passage Vector but also the degree of content and whether the amount of content within the passage is small or large now as it turns out the model that we're using has been trained to just use cosine search because all of the vectors are normalized to a unit basis so it doesn't matter which approach we use for the model that I picked but in general just be aware that often if you care about not just finding relevant data but finding a large quantity you might want to do a DOT product instead in any case here I am on the leaderboard the mte leaderboard it's a leaderboard for the best embedding models and many of these models here are open source but some of them are not now I just didn't think too much about it I looked at a few models and ended up picking one that I felt was in a reasonable position and also was open source so I could download it also you should note there's not all that much difference in the scores between a lot of these models and the one I settled on was E5 large V2 which is this model here you can see it has uh half a million uh downloads which is a good sign so I'm going with this model here and I've set up a script in in order to create embeddings uh for my database so typically the way we do this is we take the text files before we even do any llm work we take the text files and we chunk them and once the text file is chunked we're then going to convert each of the chunks into a vector that's what I mean when I say create embeddings so here for example I'm loading a vector database it's just a database I'm saving on my computer in pickle format there are better ways to do this for Speed uh like using Milas but I wanted to keep it simple so what I do as you'll see in the code is I chunk the text first of all into chunks of 500 tokens and then I encode the chunks into vectors so that's what model encode does here and you'll see above that I have loaded the inflat E5 large V2 model so now I have a database that has all of these chunks in text and beside each text representation is a vector it's a very high dimensional Vector I think maybe a th Dimensions uh so you can think of an arrow quite easily there in a th000 Dimensions now that I have those uh embeddings for my data set I'm able to create a vector for each query and compare that query Vector to all of the vectors in my database so here I have some some helpers uh for my embeddings one of the helper functions I have is called search similar texts and quite simply it takes in a collection name The Collection being the set of vectors for a given document so I have a collection for the birkar report and I have a collection for the birkar meeting so it takes in the collection name and it takes in the query embedding so I've already converted the question into a vector by using model do and code as well elsewhere in the code so given this uh collection name which is pointing to a database and the embedding for the query I'm able to make uh this cosine comparison and you'll see the cosine comparison uh is just down here P torch cosine similarity uh so this calculates the similarities between all of the vectors and then I just filter out the top results top K and I'm setting top K to three so I'm picking the three closest vectors so what that gives me is it gives me three chunks for each of the documents so I will have six chunks in total because I'm injecting two documents here the Burkshire meeting which is going to have three chunks and then I'll do the same for the Burkshire Port which will be three chunks so in total for the rag based context there are going to be six chunks and each of the three chunks as I said will be wrapped in tags to indicate what the document name is that's very important because it helps guide the language model and together those chunks or rather those documents will be wrapped in context tags just to make it really clear that that's context for the language model now let me show you what that looks like in practice so here is the setup for rag context so again I open up with a context tag to make it clear what the context is I've have a tag then for the Burkshire annual meeting and then I start off with the first chunk you'll see that the chunk here doesn't start at the start of a sentence you probably could optimize that if you wished um and by the way my chunks are 500 tokens long with an overlap of 50 tokens so I have a tiny bit of overlap just to make sure that I don't completely cut meaning in half and weaken the signal so here I have the first chunk it's 500 tokens and then you see dot dot dot and then the second chunk and dot dot dot and then the third chunk here so I've three chunks then I close the tag and then I open a tag for the next document and now again I'm going to have one chunk then I'll have dot dot dot then the second chunk dot dot dot and the third chunk then I close the tag for the second document close the tag for the context and ask the question same question and same format question as in full context and indeed all of these calls in answering the following question you may choose if appropriate to make use of the above context why didn't the manager of TTI want to sell the business to a competitor now I just want to do a search here um because this is instructive on why rag isn't working well for this question it's easier for me to search in vs code so I'm just opening up the very same file here and I'm searching for TTI because the question of course is why didn't the manager want to sell it to a business that is a competitor and we can see the the passages that are picked out by the rag system and there are a few mentions of TTI so at the very end here of one of these Snippets there's a reference to Paul Andrews who was the former manager of TTI and you can see that's probably why this specific snippet was pulled out because it does refer to TTI but actually there's no relevant information in this snippet here um it just doesn't talk about the questions of why to sell or not here's a second reference to TTI and it's talking about TTI as an example of a business that Burkshire bought but it again doesn't refer to the question of why or not to sell to a competitor and actually those are the only references to TTI the rest here is TTI within sitting and so that shows you that there isn't a relevant snippet that's been picked out talking about TTI and the question of selling it to a competitor and that's why rag fails on this question if you go back and look at our table you'll see that rag is unsuccessful whereas when we feed in the full context the answer is correct and also the allsort method picks out the right snippet as we can take a look at later on next up we're ready to take a look at how the allsort algorithm was built starting off with this question of enforcing rejects so when I showed you in chat GPT one of the questions in your mind is well what if the language model doesn't respond exactly with one What If instead it gives a longer answer like based on all of this above we feel the relevance is one out of five or what if it says one or it says a or something else well actually now you can enforce this specific response here and force it to only respond with a number going from 1 to five and as I mentioned this is something that has been pioneered by groups such as outlines you can check out outlines on GitHub the repo has become very popular over the last few weeks and this functionality is also included in text generation inference right now what I'm going to do is take a look at that in the code base so if I go back to visual code and I take a look here at um the classify function this is a function that allows us to classify Snippets as being 1 2 3 four or five in fact it's a series of functions but within it there's a call to the language model this is the helper language model that I'm running on runpod and when we send the payload so when we send instructions to runp pod to generate a response as usual we send in a prompt the max number of tokens which can be small because we're only going to get one or two back um we're also going to add this new parameter which is grammar and the type of grammar we're going to put in is reject and the value is uh square bracket 1-5 closing square bracket and what this tells the language model is when it generates the probability distribution for the next token it restricts the possible tokens to only those that are represented by 1 2 3 4 or five and this is really beautiful because it means the language model must respond with 1 to five and it's only going to consider the probabilities of those tokens in giving its response and so we can absolutely guarantee when we use this method that we're going to get back a one a two a three a four or a five and when we have back that number we're simply able to put it into a table where we have the chunk and then we have the relevance beside it and with that we have a prepare all sort context script so here I have prepare all sort context and I have a classification function so what this classification function does is it sends requests to my GPU and it sells them very fast and they stack up because the GPU with text generation inference is able to run uh continuous batching of these requests so I'm able to use the GPU at very high Effectiveness which is very good for cost so I'm sending these requests in and they're being processed continuously and they're coming back to me with the relevance being noted so I'm getting uh the score by getting the result of that check which is being done making use of the classifi text function I just showed earlier and then I get out uh relevant chunks and the way I'm picking the relevant chunks is I'm saying let's consider all of those with a score of four or greater because the language model is saying those are very relevant and this is a very nice way to do it because it means if you have a lot of information that's relevant it's going to give you more chunks and if you've less that's relevant in the data set it's going to give you less and this is um a lot more flexible than using something like vector comparison because it's very hard to set a cut off with Vector comparison where something is or isn't relevant if the cosine similarity is 08 or 085 like where do you set it so that you get the right values whereas here if you set it at four or five you know that if the helper model is doing a good job it's going to be a quite relevant snippet now just as a backup I make sure that there are always at least three Snippets included so even if there's no four and FES there's going to be at least three it'll just go down the pecking order in that list and that's in a nutshell how we get the relevance for the chunks um how we send them in in parallel and I can't overemphasize the import of being able to send them in and continuously batching the requests because this allows us to make very good use of the GPU and get costs that actually you'll see later are probably just as good or maybe even better than if you were to pay on a per token basis so the last thing I want to show you is how to set up a GPU and that's reasonably simple we'll head over to uh the GitHub repo here and take a look at the readme if you scroll down to setting up a server for all sort you'll see well you'll see tr's affiliate link for runp pod which uh supports the channel if you're signing up for the first time and you'll also see a one-click template here uh for smog 34b chat now smog 34b is a fine tune of the ye model which itself is quite a strong chat chat fine-tuned model the smog model is itself uh it's developed by Abacus Ai and it's built on top of some work by John Durban who has used uh his everything bagel data set I believe in order to develop this model you can see here this model is a fine- tune of John Duran's excellent Bagel model this model has not utilized any form of merging and the thing that I really like about the E 34b and then this even stronger 34b model is that it fits in 16-bit Precision on a single a100 and that means you can get quite good inference speeds in fact you probably can in some cases get better than Mixr inference speeds because your just able to fit it on one GPU and run in that 16bit format so if you go back and you click on that runp pod uh link to the one-click template you'll see that this is uh ready to go smog 34b chat and you then just want to pick an a100 now I would like to pick this a100 here because it's 189 but usually I pick this uh X sxm One um it's got a larger GPU and typically it has the right drivers I need Cuda 12.1 um so typically I'll pick this one here even though it's a bit more expensive and I'll deploy it just note that you need to have a disc of at least 75 um maybe a tiny bit less in order to fit the full weights and I also like to limit uh the max input length and tokens to be just enough for my application um so that I can save on memory where possible and I'll click on continue and that will allow me to deploy the pod now it'll take a little bit of time for the weights to download although HF transer is enabled on these pods the weight downloads are very fast um it uses a rust based download approach um you'll see when the GPU memory is up to the top and when it says invalid host name right here that the Pod is ready for you to hit um from your inference scripts so with the Pod set up uh I'm going to head over to my scripts and I need need to set the environment variables there are two things that I need to set first off I need to set up the endpoint you just put in the Pod ID right here um so copy this pod ID and paste it in here and second of all you need to specify the model because this specifies the tokenizer that's going to be used uh when we call the model and just to show you as an illustration here you can see the model has been read in from the environment variables as is the API endpoint and then that's ultimately being used in order to make the calls here for the classification now before I actually run this and show you an example I just want to explain a more detailed prompt that's actually being used so we went through earlier this basic prompt which literally provides the snippet and then the question and then it gives a little bit of guiding text but actually this isn't good enough it took me some time to iterate on the prompt and the reason is is because when you have sentiment-based questions if you ask how did Buffett feel about the energy business last year versus this year or more specifically what was Buffett's concern with the energy business last year versus this year or was Buffett concerned the language model will tend to look for instances of concern but perhaps Buffett is overjoyed and if Buffett is Overjoyed it won't necessarily pick out the examples of overjoyed as Snippets that it include in the context and so you need to make the model aware of this when it's evaluating the relevance and so we'll go through this prompt uh step by step here so again the snippet is the ostrich is 4ot tall and again we'll ask it to comment on a scale of 1 to 10 but we're going to feed it some examples now this just generally improves performance so we've got an example here the company's Revenue increased by 10% what was the financial performance and the relevance here is four it's definitely highly re relevant but not like a completely clear picture of financial performance then another example here the CEO now pron to expand in New Markets what were the company's Revenue figures for the quarter so you can see I'm just giving some examples and what this gets the model to do is get used to using different levels get used to using 1 2 3 4 five if um you don't give it this kind of example it may kind of get locked in on giving all ones or threes I found that this helps it to be a little bit more elective and deliberate about the layer that it's picking for that relevance now here's uh where that sentiment comes into play so if the question involves sentiment or opinion consider the following Snippets that directly Express or imply sentiment should be given High relevance Snippets that contradict or provide evidence against so this is where if he's Overjoyed it would provide that snippet as an example if it's being if the question is about being concerned and then Snippets that are neutral or do not address the sentiment should be given a lower relevant score and for general questions not involving sentiment focus on the snippet's relevant relevance to the main topic and details mentioned in the question carefully consider the relevance and use the full range of scores to illustrate the degree of relevance so this is my further detailed prompt actually when I use this prompt I don't include respond only with the number that isn't necessary because I'm using the reex um I'm forcing it to be a number so actually I don't include that but you can see for the purpose of illustration if I try and run this without it the model is going to blab on and give me more than just that number with the explanation of all sort done we're ready now to do a live demo so I'm going to head over to VSS code and take a look at the readme running any evaluation is done with python eval py and then specifying the input files of which there are two here the two text files and then specifying the method and for method you can actually run with no context whatsoever you can run with full context the standard rag context or all sort context which is what I'm going to do now and a few other flags here just for testing you can add Skip eval if you just want to view the prompts you can also just test with one question which is what I'm going to do now and you can also add on inspect prompts if you want the prompts to be saved so you can take a look at them so I'm going to run with uh the allsort context and I'm going to do it using um test one Q only um because I want to find the time for just running one question so here I'll put that on and why not inspect the prompts as well we'll save them for looking at now I have the runp Pod smog 34b model up and running uh so we're ready to go now you'll see that this is now going to prepare to hit that end point and then we'll start seeing the scores coming back and here we have the results coming back you can see the API being hit and the scores we can see a lot of ones so this is filtering out pieces that are not relevant at all and you'll see intermittently there might be a higher score let's see you see a five here so there's one very relevant snippet and if we keep going here and just to note that the total time taken is higher at the start because there's a period where the reject needs to be set up I haven't looked into it in detail but that requires a little bit of one-off time but once your reject is set any request that you make afterwards will be a lot faster so actually that will even knock time off the estimates that I'm going to share a bit later okay so the process is complete you can see how most of these Snippets were scored as one because they're just not relevant which is exactly what you want um even further I can go in and take a look at the results so if I go into the output files and I go to all sort and this should be under allsort using gp4 for preview and we're just running one question which is the first question and it's that same question on uh TTI so let's do a search for TTI and you can see already there's a difference if you remember when I talked about rag here there's an extra snippet that's being included that wasn't picked up by Rag and it's the key snippet because it says um and he said this is the business owner talking I've got all the money I need the family's got all the money but what do I do with the business these people have helped me throughout my life I could sell it to a competitor if I sold to a competitor they'd fire my people so this is the exact answer and it just shows you how um the language model is picking this out by giving it a high rating and that's why it's included within the context that's going to this prompt and being sent to the language model now you probably saw this automatically open up it's the evaluation results and so we can go through a few of these to take a look at the performance and here's the question why didn't the manager of TTI want to sell the business to the competitor and you can see the full context answer is correct uh because of doing so would result in the competitor firing tti's employees and keeping their own there's also the no context answer which is not too interesting because it has to make things up there's the rag context answer and rag actually just picks the wrong um it picks out the wrong Snippets and then includes uh context that's not relevant so it talks about the manager of TTI not the owner and goes on to talk about that which is not correct now allart um gets this correct because it says the owner was concerned the competitor would likely fire his employees so it gets that uh perfectly so yeah this is uh an example of where we have better performance from um the rag moving on to the next question uh where does Buffett feel more comfortable with his Capital um and that means money Taiwan or Japan and the full context answer is Japan because of geopolitical reasons the rag context answer is correct as well and uh no context we can ignore the all sort answer is also correct Vicky Hollow so the full context answer uh the first job was City service it's actually city services um City Service Company yeah so yeah that's correct uh the rag context it makes up Dairy Queen so it doesn't pick out that reference um which was difficult because the spelling I think was not exactly right in the Raw text uh but you can see that with the uh allsort context it gets this uh question correct next up how does buffer feel about the new tax law um that impos a minimum rate of 15% so here's the 15% answer so Warren Buffett expresses a pragmatic view that's correct uh he indicates Berk is well prepared to handle it uh he says that he'd prefer the company's earnings to be calculated without including the unrealized capitals gains or loss um which can be highly volatile okay so this actually he doesn't say with respect uh to the tax question he says that that towards the start of the meeting it's a comment that he nearly always makes um and the comment in reports is that he doesn't agree with the accounting system but this is actually a separate point to what's happening here so the logic here is actually not exactly correct uh from the full context answer if you look at the rag context answer it says something similar here but the specific piece I'm looking for is around unrealized gains uh we'll see if that's included the piece I'm specifically looking for is this one here which says Buffett acknowledges there's still unanswered questions particularly how it will affect the treatment of unrealized capital gains um and impact on renewable energy tax incentives which was actually a question from the audience and the reason I'm looking for this little snippet is because it directly answers the second part of the question which is does he understand the implications for taxes Burkshire will pay and that part is not exactly answered correctly in the full context answer now it is answered correctly down here in the allsort answer you you can see that uh unrealized capital gains and marketable securities this is mentioned in the correct context here um he explains that the specifics of how the new tax law app law applies to this are not mentioned he says that the specifics of how the new tax law are applied here are not entirely clear okay differences between accidental and traditional oil so the difference is that in Shale oil the production of a well Falls very fast that's what I'm looking for a comment that Charlie Monger talked a lot about in the last meeting and um yeah you can see that's included here in the full answer production rate deines rapidly in the rag context answer you can see um I think it gets it as well the rapid decline rates of the oil in the all sort context answer again the answer is correct here now why does Buffett bring up the matter of industrial capacity during the second world war in the full context answer uh it says Buffett brings it up uh to highlight the productive capacity and resilience of the American economy when faced with significant challenges and this isn't actually true because the whole point Buffett is making is that the American economy was very resilient uh during the second world war but the system is very different it's now much more kind of um a spread out system it's less centralized than it was during the second world war and he's specifically saying it will be difficult to coordinate and achieve the same uh directed goals as it was during second world war so this view is not entirely correct now moving to rag um here Buffett says it's developed to illustrate an historical example of rapidly mobilized and reoriented and yes that is correct it is a historical example and the key point I'm looking for though is he uses the example to discuss the difficulties of achieving similar Feats when facing modern challenges that require significant re-engineering of systems such as the energy system in the US so that's the kind of contrast I'm looking for and it's there in the rag context now here with the no context answer again I don't care by the way I'm including this just because you might say oh well doesn't the model already know about the Burkshire meeting so it's not actually picking anything up about from the rag context or the context at all and this is why I have it because you can see if you read through the no context answer it's just going to tell you about the positives of the US during the war but it doesn't really give anything clear uh with respect to the current uh challenges and how that's going to be difficult because of a new context today so moving on to the allord question um this actually kind of mirrors the rag question in that it gets it right and it does mention again energy He suggests that while the capital and knowledge exists to make significant progress such as in transitions in renewable energy the lack of a unified National effort and the mechanisms to direct resources effectively are major OB obstacles and this is the Nuance that we're trying to get from the model so I know I'm talking in a lot of detail but this is what you need to do if you want to appreciate the difference in performance between different approaches moving to the last three questions is Berkshire more concerned about Burkshire hathway energy last year in the meeting or this year in the report and the answer is I'll just go straight to the answer at the bottom it's that um Buffett expresses some concerns in the annual meeting um about the cost and coordination and coordination of upgrading infrastructure um but he's much more concerned in in the annual report um so let's just very quickly see uh what the answers say here so here in the full context he's got more concern in the report than in the meeting which is correct um the rag context Buffett does not express a specific concern in either document so clearly rag is not able to pick up uh the nuances in this case here and then moving to the allsort um Warren expresses a more ounc concern in the annual report compared to the annual meeting so this is an example of where rag doesn't work so well but uh works well with all sort or the full context a similar question then on the rail business and the answer with respect to the rail business is that Buffett is more cautious in the annual report and highlights operational and financial challenges I'm specifically looking for the model to mention a drop in earnings if I scroll up here uh in the allore question you can see uh he gives a mix But ultimately optimistic view um he does mention the falling uh revenues here and also wage increases as well so he is uh describing some of the challenges there uh if you go up and look at the rag context you can see that um here he does mention the decline in earnings and in the full context as well you can see here he's mentioning some of the negatives to decrease in Revenue so all three approaches worked well on that question and the last question here around Apple comparing his sentiment on Apple between last year's meeting and this year's and the key thing I want all answers to see is that he doesn't actually mention Apple at all in the report I won't go through the detail on this question but all three the full context Rag and allsort do well on this question I was actually hoping though they might do even better if you really know the Burkshire history you'll know that he often is very complimentary about Apple so it's somewhat unusual that he didn't mention Apple in the meeting uh r in the report this year so there's probably significance this but to nuanced it seems based on the language models knowledge to be able to comment on that just by means of summary we go back to the table here so you see all nine of these questions and all sort it gets all of the questions correct the full model it does make that mistake on the tax law um further around uh the industrial capacity during the second world war it doesn't catch on to the Nuance that's being applied there meanwhile on rag it makes some retriev errors here around Vicky hb's job and also around TTI doesn't provide the language model with the correct snippet that's needed and likewise uh or rather not likewise but in the case of doing the comparison between Burkshire hathway Energy's performance last year and this year it's just not able to pick out the right sentiment around the matter concerned and so the model uh the final model is just lacking context when it needs to aine on whether that sentiment has changed I did try a few ablations notably around using a different model than smog I tried a 7B model which has got about three times the inference speed so you'll see it takes about 75 seconds with my unoptimized approach to do one question using allsort but um the inference time is about a third if you use open chat and you can see it performs reasonably well um it's probably better than using rag although it gets slightly different questions wrong it has some difficulty in identifying the sentiment around the question of concern in the energy question with relative um matters of last year versus this year and also the sentiment around uh what Buffett thinks of B berkshire's rail business nonetheless uh it does very well on retrieving factual information and and does better than the rag question on this the other ablation I did was to look at using Mixr 8x7 B I ran that model on two uh A1 100s and it is faster than Running Y it's uh only a little bit faster though because y can be run on a single GPU a single a100 but the quality is not quite as good um and perhaps most interestingly when I ran the raw y model which is before being uh fine-tuned and improved by John Durban and Abacus AI I got significantly worse results um so it really matters what model is being used as a Helper and what I see is that basically some of the base chat models that are being being released they seem not to have the same amount of instruction fine-tuning data as some of these later fine- tunes and this instruction tuning data seems to improve performance quite a lot I showed in an earlier video on performance evaluation that's the video where I used um Nicholas Carini's repository you'll see that open chat is a lot better than the base model Mistral and I think likewise we're seeing a big Improvement here in the smog model versus the base ye model let's take a quick look at costing here I'm going to compare the cost of running with the full long context then the rag approach and the allsort approach I'm going to assume that the database is 100k tokens and you can scale it accordingly next um we have GPT 4 which is about a dollar per 100K tokens so $10 per million tokens of input and for the allsort assumptions let's assume I'm running on runp pod with this uh a100 add it's actually SRX or something like that but it's an a100 for 225 per hour and it takes 75 seconds for the prompt preparation um and that's unoptimized I think that can be cut down in a few different ways um by improving how I'm running TGI on runp pod but in any case for 75 seconds at 225 an hour that's about 5 cents for prompt preparation and just as an aside you could also in principal query um a model ye that's been hosted by some other company there are companies out there there that will just host e models and you can hit them and pay for them on a per token basis typically it might be about uh 50 cents per million tokens so for 100,000 tokens you might also be talking about at 5 cents so this goes to show you that actually in this case because we're making such good use of the GPU we're able to get down to similar cost levels as the commercial um and this makes sense because the whole thing about commercial inference is that they can make use at very high percentage rates of the gpus and therefore get good economics just to note though that I'm not aware of um any apis although I'm sure there will be soon that actually support this uh re reject forcing as I've shown you um typically they aren't running TGI but I assume that this type of reject uh formatting will be supported by more and more apis including uh potentially open AI um quite soon which would allow you to actually use GPT 3.5 so here we for costing um in the full context case there's no cost to prepare because you're just injecting the data so it costs uh $1 for input tokens I'm ignoring the output token cost so very Loosely $1 uh for rag there's no cost I'm assuming for preparing embeddings so the evaluation is just uh the eval tokens I'm assuming they're context of 2,000 tokens um for all sort uh in the self-hosted case there's 5 cents uh to pay for that run pod 75 seconds uh so a total of 7 cents and in the holed case if there was an API that supported rejects I'm saying potentially 5 cents of prep and then 2 cents of eval so 7 cents overall you can see that the cost here is greater than rag because you are doing using an llm which is a more expensive way and the cost is going to go up for Preparation as you're using more and more data but I think you have to weigh that off against the fact that using a good llm can get a lot more accurate retrieval than using a rag based technique so if you are targeting a high value application this could be a good retrieval method last off on latency and this is extremely unoptimized I'm just giving it because I think it's useful to know but it it probably isn't representative once you improve gb4 can have very different times for making the same request in my experience so it's hard to say exactly what the latency is um for the full context uh I ran a request it took 25 seconds with a rag that's including the embeddings and the time to hit gp4 um it took me 40 seconds but again that could be just variance in the time for the open AI request um for all sort it took 75 seconds again there's an initial period to set up the reject that contributes probably about even 20 seconds of that and there are some optimizations on how I'm hitting the API that probably could cut that down to be a bit lower if you're using a hosted service that will be able to provide 200 tokens per second uh for average generation that's faster than what I would have been getting from the a100 so you could by using a hosted service probably GPT 3.5 when they support for this re when they support this reject matching it will be possible uh to get this timed down probably to something overall that's comparable with rag keep in mind that my rag approach is very unoptimized as well I'm not using a database like milis or something like that so all of these um possibly with the exception of the full context approach can be improved significantly all of these possibly with the exception of uh the full context approach which is in the control of open AI or Cloud um and I would say potentially cloud is even maybe a little bit stronger than what I've shown with gp4 turbo depending on the questions but all of this here can be greatly improved and optimized and that's it for this video on language model assisted retrieval or what I've called all sort as always you can go to tr.com and find the Advanced inference page there you can buy the scripts specific to this video or you can buy lifetime access to that repository which will contain future inference update scripts that I upload by the way as of roughly yesterday we reached the 300 person Mark that's at least 300 people who have purchased lifetime access to one of trellis Advanced repositories there is the advanced fine-tuning the advanced inference Advanced Vision and advanced transcription you can find them all on tr.com and the support is much appreciated because that is what keeps this channel going all right folks let me know the comments on this video on all sort all below cheers""