I think it should be.


Extractive Q&A With Haystack and FastAPI in Python

OK.


Yeah, so it's just starting up again now.


OK, looks good.


So I think that's good.


And let's try and open up.


Over here.


We should get loads of text.


Yeah.


So that's cool.


So now we're getting the query.


What did your grandfather teach you?


And this is pretty hard to see what you see.


We get this list.


And then this goes all the way through here.


And we get all of these different ads.


So the first, like the top rated answer is what did your grandfather teach you?


And it's act and speak.


OK.


In sleep, seems act and speak.


It's all like children from their parents.


OK, well, you know, that's interesting.


So it's not amazing.


So let me try.


Instead of using.


Instead of using the DPR retriever, let's try.


Swapping out for a BM25 retriever, which is it's basically just an algorithmic retriever.


And it should actually run faster as well.


Elasticsearch retriever.


And all we do is swap those around.


And I think the only parameter we need is the document.


So.


Yeah.


So, OK, let's let that start up again.


OK, just waiting.


Let's try this one, see how it compares, and then we'll go one of these, I think.


So that's let's go.


And just refresh the page again.


OK.


Yeah, this is a lot better.


Which is weird.


I kind of thought the other one would do better, but.


That's fine.


Maybe we need to fine tune it a bit more than what we have.


So what we have here is what your grandfather teach you and we get good morals and the government of my temper, which is, I think, exactly what I wanted it to return.


So that's good.


I see it is like a new line character as well, which I think we need to format the text before we put it in to remove those.


But that's cool.


Pretty cool.


So that's really good.


It's working, but we're not actually making our own queries.


So we need to add that.


And to do that, all we need to do is add Q in here.


And this is our query string.


And in here, we just made that equal to query.


And now if we do that.


Rather than opening in the browser, I'm going to open insomnia here.


So insomnia, what you can see now is a HTTP client.


So it allows us to send requests really easily and just format everything in a nice way.


I should have opened this before. I'm not sure why I didn't.


I just forgot about it.


So localhost 8000, we're going to the query endpoint.


Then in our query, we have the Q parameter.


And in here, I want to say, what did your grandfather teach you?


And then we send this and we should get a response on the right.


Cool. And this looks a lot better.


So it's good.


And let's try something else.


So I think I've read a lot of the time, he just talks about the universe.


So what is the universe?


And send that.


Takes a while to run it the most.


Cool. So we get a query and then we get answers.


And it's a well-arranged universe.


Universe loves to make whatever it is about to be.


Interesting.


That which knows beginning and end and there's the real world.


That which knows beginning and end and there's the reason which pervades all substance and through all time by fixed periods, revolutions, administers the universe.


Yeah, it's pretty deep.


Yeah, that was cool.


So I think that's everything. I don't want to include anything else.


So that's pretty cool.


So now we have a look at this again.


We've just done like this entire section here and the API.


I mean, we'll probably make changes in the future, but that's kind of all this done.


So you can cross that off.


And then the next bit is going to be probably the most difficult bit, at least for me, which is going to be the front end in Angular.


So should be pretty interesting.


So let's see.


Let's see how that goes.


But for now, I think that's it.


Building a PlotLy $GME Chart in Python

Hi and welcome to the video.


Building a PlotLy $GME Chart in Python

We're going to do something a little bit different today


Building a PlotLy $GME Chart in Python

and we're going to build a Python project from scratch.


Building a PlotLy $GME Chart in Python

And what I want to do is build a Plotly chart


Building a PlotLy $GME Chart in Python

and feed stock information into this chart


Building a PlotLy $GME Chart in Python

from the AlphaVantage API,


Building a PlotLy $GME Chart in Python

which is like very similar to the Yahoo Finance API


Building a PlotLy $GME Chart in Python

as far as I'm aware.


Building a PlotLy $GME Chart in Python

I haven't used either of them that much to be honest.


Building a PlotLy $GME Chart in Python

But they seem to have pretty good documentation


Building a PlotLy $GME Chart in Python

and it's a little bit different


Building a PlotLy $GME Chart in Python

and I just want to explore using this API rather than Yahoo Finance,


Building a PlotLy $GME Chart in Python

which is like the sort of de facto standard almost.


Building a PlotLy $GME Chart in Python

So after we give some stock information to this chart,


Building a PlotLy $GME Chart in Python

we want to actually chart it and plot a few sort of interesting indicators.


Building a PlotLy $GME Chart in Python

So I've kind of, you can see on the screen,


Building a PlotLy $GME Chart in Python

I've gone for a few different indicators here that,


Building a PlotLy $GME Chart in Python

I mean, I haven't really seen any of these.


Building a PlotLy $GME Chart in Python

I'm not a finance person,


Building a PlotLy $GME Chart in Python

so I don't claim to know what all of this means exactly.


Building a PlotLy $GME Chart in Python

Obviously, the simple moving average is,


Building a PlotLy $GME Chart in Python

or it's a moving average of the stock price over time.


Building a PlotLy $GME Chart in Python

So that one is quite simple.


Building a PlotLy $GME Chart in Python

And I just want to include that in there


Building a PlotLy $GME Chart in Python

so that we can see that we're actually doing things right


Building a PlotLy $GME Chart in Python

and we're plotting these indicators correctly.


Building a PlotLy $GME Chart in Python

Then we have this thing called the unbalanced volume,


Building a PlotLy $GME Chart in Python

which sort of measures positive and negative volume flow.


Building a PlotLy $GME Chart in Python

So, the reason I'm including all these volume and balancing


Building a PlotLy $GME Chart in Python

and momentum-based indicators,


Building a PlotLy $GME Chart in Python

at the moment in the stock market,


Building a PlotLy $GME Chart in Python

we have the big WallStreets versus WallStreets bets.


Building a PlotLy $GME Chart in Python

I want to sort of use this opportunity to just build sort of a cool project


Building a PlotLy $GME Chart in Python

that at least gives a little bit of information


Building a PlotLy $GME Chart in Python

on what is actually happening in terms of the actual stock market.


Building a PlotLy $GME Chart in Python

Now, a lot of these are, you know, traditional indicators.


Building a PlotLy $GME Chart in Python

They've been around for a few years,


Building a PlotLy $GME Chart in Python

so I don't know if they're actually going to be useful or not,


Building a PlotLy $GME Chart in Python

because obviously the market conditions at the moment are not normal.


Building a PlotLy $GME Chart in Python

But it'll be interesting to see.


Building a PlotLy $GME Chart in Python

And then we have this balance of power,


Building a PlotLy $GME Chart in Python

which is the market strength of the buyers versus sellers.


Building a PlotLy $GME Chart in Python

So again, pretty relevant rate of change.


Building a PlotLy $GME Chart in Python

Maybe we'll include this, I'm not sure.


Building a PlotLy $GME Chart in Python

This one called Arun, which looks quite interesting actually.


Building a PlotLy $GME Chart in Python

So if we go on here,


Building a PlotLy $GME Chart in Python

and you can see that it essentially gives us this.


Building a PlotLy $GME Chart in Python

So we get like a downtrend and uptrend indicator line.


Building a PlotLy $GME Chart in Python

And whenever these cross over,


Building a PlotLy $GME Chart in Python

that's typically when you see an uptrend or downtrend.


Building a PlotLy $GME Chart in Python

And I think that'd be quite interesting.


Building a PlotLy $GME Chart in Python

And I think that'd be quite interesting to see as well.


Building a PlotLy $GME Chart in Python

And this, so this is based on the time between highs and lows


Building a PlotLy $GME Chart in Python

in a specific time period.


Building a PlotLy $GME Chart in Python

So what you're doing here is measuring the lowest point.


Building a PlotLy $GME Chart in Python

And that gives you your sort of the points, the edges,


Building a PlotLy $GME Chart in Python

or the peeps and troughs of this indicator here.


Building a PlotLy $GME Chart in Python

So I think that'll be pretty interesting.


Building a PlotLy $GME Chart in Python

And then we have this Chaikin oscillator.


Building a PlotLy $GME Chart in Python

And I thought this would also be quite interesting


Building a PlotLy $GME Chart in Python

because this is tracking the strength of price movements


Building a PlotLy $GME Chart in Python

and the underlying buying and selling pressure


Building a PlotLy $GME Chart in Python

to estimate demand.


Building a PlotLy $GME Chart in Python

So that should also be pretty interesting.


Building a PlotLy $GME Chart in Python

So let's go ahead and we're going to start coding this up.


Building a PlotLy $GME Chart in Python

So the first thing we need to do is figure out how to use the AlphaVantage API.


Building a PlotLy $GME Chart in Python

So basically all the data that we want, we can get from here.


Building a PlotLy $GME Chart in Python

So the main part of that is going to be this time series intraday API.


Building a PlotLy $GME Chart in Python

So we'll just go ahead and figure this out.


Building a PlotLy $GME Chart in Python

Okay, so if we just take this,


Building a PlotLy $GME Chart in Python

let's figure out the format that data is actually going to come in with.


Building a PlotLy $GME Chart in Python

Okay.


Building a PlotLy $GME Chart in Python

So we have our interval, which we included here.


Building a PlotLy $GME Chart in Python

Symbol, we're going to change that to GME.


Building a PlotLy $GME Chart in Python

And then we have this time series,


Building a PlotLy $GME Chart in Python

which is in the dictionary and inside here,


Building a PlotLy $GME Chart in Python

we have our open, high, low, close and volume as well, which is interesting.


Building a PlotLy $GME Chart in Python

So, okay, let's pull that together.


Building a PlotLy $GME Chart in Python

So first, I do want to have the candlestick format.


Building a PlotLy $GME Chart in Python

So I came up with this.


Building a PlotLy $GME Chart in Python

So candlestick is this, whereas if we were just tracking the sort of average,


Building a PlotLy $GME Chart in Python

it would look like this.


Building a PlotLy $GME Chart in Python

So first, we're just going to do this with Plotly.


Building a PlotLy $GME Chart in Python

And then we'll figure out how to build the candlestick.


Building a PlotLy $GME Chart in Python

So we're going to go ahead and try and build that.


Building a PlotLy $GME Chart in Python

So for now, we're just going to use this data.


Building a PlotLy $GME Chart in Python

And we'll get into a data frame.


Building a PlotLy $GME Chart in Python

So import pandas.


Building a PlotLy $GME Chart in Python

And just clean it up a little bit.


Building a PlotLy $GME Chart in Python

So


Building a PlotLy $GME Chart in Python

Okay, so I'm going to initialize the data frame first and then we'll just loop through


Building a PlotLy $GME Chart in Python

each of these entries.


Building a PlotLy $GME Chart in Python

We'll have the date and time in one column, the open, high, low, close and volume in other


Building a PlotLy $GME Chart in Python

columns and we'll just append each one of these as we work through.


Building a PlotLy $GME Chart in Python

So here we're just going to print out each of the date time keys because here we need


Building a PlotLy $GME Chart in Python

to actually access this in order to get into our actual values here.


Building a PlotLy $GME Chart in Python

So just print this out and then we need to also add this.


Building a PlotLy $GME Chart in Python

There we go.


Building a PlotLy $GME Chart in Python

So these five minute intervals and this is what we're going to work with.


Building a PlotLy $GME Chart in Python

Let's just make things a little bit easier to read.


Building a PlotLy $GME Chart in Python

And now we're going to do here is append a dictionary which is going to contain each


Building a PlotLy $GME Chart in Python

one of these columns.


Building a PlotLy $GME Chart in Python

So I'm going to order this one, the Bugün returns it.


Building a PlotLy $GME Chart in Python

That's good.


Building a PlotLy $GME Chart in Python

Whenever we're doing this append method we also need to add ignore index onto the end


Building a PlotLy $GME Chart in Python

otherwise we'll get a index error which is really annoying and it just happens every


Building a PlotLy $GME Chart in Python

time. Usually I forget to add this in before seeing the error but this time apparently not.


Building a PlotLy $GME Chart in Python

Okay so we also need to add in dt here because at the moment we're trying to access the the


Building a PlotLy $GME Chart in Python

whole dictionary the top level of it whereas we need to access each data frame section of it.


Building a PlotLy $GME Chart in Python

See look I remembered it this time and I so got it wrong. Okay so now we have all of our


Building a PlotLy $GME Chart in Python

data in an easy to read format. I just want to see why these are all the same. Maybe that's normal.


Building a PlotLy $GME Chart in Python

Okay so they're not all the same they do change. So we are also looking at IBM and IBM I don't


Building a PlotLy $GME Chart in Python

think is a particularly volatile stock. Okay so I'm gonna switch this data over to GME for that


Building a PlotLy $GME Chart in Python

I do have my keys somewhere. Okay so I have my key here now it's just in a text file. So when


Building a PlotLy $GME Chart in Python

you're reading from text files like your keys which is generally I think what you'll do a lot


Building a PlotLy $GME Chart in Python

of the time when you're just developing something or I mean you could put straight into your code


Building a PlotLy $GME Chart in Python

I wouldn't recommend but I mean it's up to you. So you seem to do with open specified file,


Building a PlotLy $GME Chart in Python

R for read and just put key like this and that should work. So now take this and let's swap for


Building a PlotLy $GME Chart in Python

GME and API key I want to use what I defined here. I'm gonna put key in there convert this


Building a PlotLy $GME Chart in Python

into an S-string. Okay let's see okay that worked it's cool and let's compare this. So these were


Building a PlotLy $GME Chart in Python

hardly changing as far as the change a little bit here between these but actually in within these


Building a PlotLy $GME Chart in Python

are high and low within five minutes wasn't really changed much and maybe it went for game


Building a PlotLy $GME Chart in Python

survivor let's see. I need to refresh here a little more movement interesting. Okay so I


Building a PlotLy $GME Chart in Python

haven't used Plotly that much before so let's just try and figure out. I mean if you're using


Building a PlotLy $GME Chart in Python

the creating the candlestick a graph I'm not really sure how I'm gonna go about that yet.


Building a PlotLy $GME Chart in Python

See I don't even have it installed on this instance. Oh so before I did so I got my API


Building a PlotLy $GME Chart in Python

key I didn't really explain how you can get one too. So all you need to do I think it says on


Building a PlotLy $GME Chart in Python

here I look so you just go to this page and you click claim your free API key go on there and


Building a PlotLy $GME Chart in Python

or you just put whatever you're doing so I suppose for me software developer maybe put student if


Building a PlotLy $GME Chart in Python

you're learning but either way it doesn't really matter because you're gonna get this free API key


Building a PlotLy $GME Chart in Python

anyway it's not they give it to you straight away I think from what I can remember so it's


Building a PlotLy $GME Chart in Python

not really an issue there. So let's go back to Plotly and let's just install that quickly.


Building a PlotLy $GME Chart in Python

Yeah that looks like just pip install Plotly and I think as well because I haven't installed it in


Building a PlotLy $GME Chart in Python

this environment I'm going to have to install a load of Jupyter extensions as well to actually


Building a PlotLy $GME Chart in Python

show the plot. But let's see if it works first.


Building a PlotLy $GME Chart in Python

Okay so let's get that data again.


Building a PlotLy $GME Chart in Python

Don't need to see that again. Okay so


Building a PlotLy $GME Chart in Python

see that.


Building a PlotLy $GME Chart in Python

Let's just go with the open for now.


Building a PlotLy $GME Chart in Python

Okay yeah so you can see that it's not showing anything and this is an issue with one of the


Building a PlotLy $GME Chart in Python

Plotly extensions. So we just need to install those it's not difficult though from what I remember.


Building a PlotLy $GME Chart in Python

Yeah this Plotly extension. Yeah so it's the Jupyter lab extension so that's checking what we have.


Building a PlotLy $GME Chart in Python

Okay so I need these two. Jupyter lab extension, Jupyter lab Plotly and Plotly widget. So let's


Building a PlotLy $GME Chart in Python

go and install those two. I realize maybe this is too small for you to see so these two we're


Building a PlotLy $GME Chart in Python

going to do. Although these are the uninstallable we're going to install them.


Building a PlotLy $GME Chart in Python

Okay so then we need to install the other one as well.


Building a PlotLy $GME Chart in Python

Then we might need to restart Jupyter but let's see if it works. I'm also going to need


Building a PlotLy $GME Chart in Python

a Jupyter lab build as well just in case I need it. I don't think I will but let's try.


Building a PlotLy $GME Chart in Python

Okay so let's go back to our lab and let's just refresh and try. Let's see if it works.


Building a PlotLy $GME Chart in Python

If not we'll just restart. Yeah okay so let's just restart it quickly.


Building a PlotLy $GME Chart in Python

Let's try all this again. Here we go so it's working now. So yeah that's pretty cool. Although


Building a PlotLy $GME Chart in Python

what is this? We have, yeah that's weird. Do we need to, okay let me check what these


Building a PlotLy $GME Chart in Python

are. I assume these must be strings they're not actual numbers. Yeah so we need to convert


Building a PlotLy $GME Chart in Python

these into numbers as well. Okay so we can do that with the to numeric function and then


Building a PlotLy $GME Chart in Python

just pass the columns that we want in there. I'm not sure if we can do all the ones or


Building a PlotLy $GME Chart in Python

if we have to do one at a time. Let's try all. Okay yeah so just one at a time. So let's


Building a PlotLy $GME Chart in Python

just take this list. Also not sure if it does it in place or not as well. Let's just have


Building a PlotLy $GME Chart in Python

a quick look. Okay so I think it does not do it in place. Let's see.


Building a PlotLy $GME Chart in Python

Okay so.


Building a PlotLy $GME Chart in Python

So no it doesn't. Right there we go so now it's a float. This should work now. There


Building a PlotLy $GME Chart in Python

we go. So that looks pretty cool. We have this sort of interactive, interactivity with


Building a PlotLy $GME Chart in Python

the chart which this is why I wanted to use Plotly over like a map.lib. I think this is


Building a PlotLy $GME Chart in Python

definitely something we want to have in the chart. So we have that now. Let's, before


Building a PlotLy $GME Chart in Python

we do the candlesticks, let's just try putting on the a few of the indicators and see how


Building a PlotLy $GME Chart in Python

that looks. And see if they're straightforward or not. So they're just simple moving average.


Building a PlotLy $GME Chart in Python

So let's add that in.


Building a PlotLy $GME Chart in Python

Okay so let's just clean this API requesting up a little bit.


Building a PlotLy $GME Chart in Python

Okay so that is our API and then we just want to change these. Just use it like this instead.


Building a PlotLy $GME Chart in Python

Okay so yeah that's May. We want GME weekly interval. No we want so five minutes. Oh no


Building a PlotLy $GME Chart in Python

we don't. We want so say it's a moving average. So yeah let's go weekly and see how that looks.


Building a PlotLy $GME Chart in Python

Time period this is like 10 days. What do we have here? We don't have a time period.


Building a PlotLy $GME Chart in Python

So we do want to adjust that a little bit. So maybe we can zoom in a little bit here.


Building a PlotLy $GME Chart in Python

Probably can't see anything. So the function interval. So yeah we have a few. Let's go


Building a PlotLy $GME Chart in Python

like a 60 minute because then we can actually see that on the time period that we're already


Building a PlotLy $GME Chart in Python

looking at. Number of data points. But okay let's change this to a 60 minute moving average.


Building a PlotLy $GME Chart in Python

And then we don't want to change that one. We change this one. And add in the API key


Building a PlotLy $GME Chart in Python

as well. Okay let's just see what we get. Okay cool. So we have this. Okay we have this


Building a PlotLy $GME Chart in Python

technical analysis. Okay so we want to pulling data out of here. Okay cool. So I wonder if


Building a PlotLy $GME Chart in Python

you can add like multiple functions. I don't know. But let's stick with this. I'm basically


Building a PlotLy $GME Chart in Python

going to do the same thing again where we take these. But this time we are getting our


Building a PlotLy $GME Chart in Python

moving average. So technical analysis. Let's put this in. And all we want here is the date


Building a PlotLy $GME Chart in Python

time and then this SMA value. So technical analysis I think that's fine. So now we just


Building a PlotLy $GME Chart in Python

change this to SMA. I think that should be okay. Let's see. Okay let's put that in here


Building a PlotLy $GME Chart in Python

so I don't do that again. Now just overwrote all of that. I mean I need to change this.


Building a PlotLy $GME Chart in Python

Okay. Okay looks cool. So we will also need to do this again right where we convert it


Building a PlotLy $GME Chart in Python

to numeric. Let's just add SMA head here. Okay. So now we have our moving average. So


Building a PlotLy $GME Chart in Python

let's just do that. Then let's plug this. And let's just see what we get. Okay. So we


Building a PlotLy $GME Chart in Python

get a lot more data here. So we need to restrict that to only within what we have here. But


Building a PlotLy $GME Chart in Python

at the same time I think it would be cool if we got a little more data than what we


Building a PlotLy $GME Chart in Python

have. But let's just keep it dynamic for now. So we just take the minimum date time that


Building a PlotLy $GME Chart in Python

we have within the F and then we'll just apply that to all of our technical indicators. So


Building a PlotLy $GME Chart in Python

we're only pulling it from then onwards. We could probably get it from the API but let's


Building a PlotLy $GME Chart in Python

just filter the data frame. Yeah because here I'm not sure they do actually give you like


Building a PlotLy $GME Chart in Python

a date time limit that you can just extract from that point. So yeah I think we need to


Building a PlotLy $GME Chart in Python

do it from the data frame anyway. So after we do all of this let's also get the min date


Building a PlotLy $GME Chart in Python

time. That would just be df.DateTime.min. We also need to re-extract all of that because


Building a PlotLy $GME Chart in Python

I overwrote it earlier. Okay and then let's check our date time. Should be something reasonable.


Building a PlotLy $GME Chart in Python

So yeah I think that makes sense. It's like 12 hours before. No. About eight hours before


Building a PlotLy $GME Chart in Python

this point. So that's fine. So eight hours of data in there. So when we do this all we


Building a PlotLy $GME Chart in Python

need to do is just filter out anything that is below that date time. So all we do is SMA


Building a PlotLy $GME Chart in Python

where the date time is greater than or equal to min DT. Okay so let's just check. So we


Building a PlotLy $GME Chart in Python

have the length of SMA here. So we've got 560 rows and then if we do this we filter


Building a PlotLy $GME Chart in Python

it down to nine which is more reasonable. So we have I think we had around eight hours


Building a PlotLy $GME Chart in Python

in there so nine hours maybe and we're taking it in 60-minute intervals so that makes sense.


Building a PlotLy $GME Chart in Python

And then if we do that. Cool so we get our SMA indicator here. So let's plot those both


Building a PlotLy $GME Chart in Python

together and see how it looks. Okay so apparently you need to also add fig, add scatter and


Building a PlotLy $GME Chart in Python

then you add mode lines onto the end to add another line. So maybe I didn't see this.


Building a PlotLy $GME Chart in Python

Okay so line is supposed to be lines I think. Okay cool. So that looks good. We have our


Building a PlotLy $GME Chart in Python

SMA and our actual plot. So that seems pretty straightforward but now we actually want to


Building a PlotLy $GME Chart in Python

maybe add candlesticks into this because that would be pretty cool in my opinion. No idea


Building a PlotLy $GME Chart in Python

how we can do that in Plotly so let's try and figure it out.


Building a PlotLy $GME Chart in Python

It actually looks really simple. So you need to use the graph objects instead of express.


Building a PlotLy $GME Chart in Python

Let's try and add that then. So if it's really that easy then this is super easy.


Building a PlotLy $GME Chart in Python

Oh well. Okay so that's our candlestick. That's really cool. I don't really know if you want


Building a PlotLy $GME Chart in Python

this bit but I suppose it's kind of cool. Let's just go with it for now. So we have


Building a PlotLy $GME Chart in Python

that and then we also want to add scatter again to add the SMA. Okay that's pretty cool.


Building a PlotLy $GME Chart in Python

So now we just want to add in a few of those other indicators. So what do we have? We have


Building a PlotLy $GME Chart in Python

this on balance volume, balance of power, rate of change, a rune and trike in. So let's do


Building a PlotLy $GME Chart in Python

this on balance volume one first. I'm going to add in a couple of functions to actually


Building a PlotLy $GME Chart in Python

do things as well. So like here every time that we want to convert these to numeric I


Building a PlotLy $GME Chart in Python

think we want to convert anything that's just not called date time. So we can just add that


Building a PlotLy $GME Chart in Python

as a function. Let's do two numeric. Then just add our data frame.


Building a PlotLy $GME Chart in Python

And then we just go for col in df.columns. We only want to do this if col is not equal


Building a PlotLy $GME Chart in Python

to date time. So every other column we do want to convert to numeric. And just return


Building a PlotLy $GME Chart in Python

that. Okay so instead of needing to write it every time we just write to numeric when


Building a PlotLy $GME Chart in Python

Okay.


Building a PlotLy $GME Chart in Python

So here we're getting the time series data. We also want to convert to numeric here. We


Building a PlotLy $GME Chart in Python

can leave it as df. Okay let's just test that and see how it is. Okay looks good. So we


Building a PlotLy $GME Chart in Python

can remove that and then we are getting our SMA data. So let's just move that into one


Building a PlotLy $GME Chart in Python

box as well. And this we just want SMA and SMA.


Building a PlotLy $GME Chart in Python

Okay let's move this up to the top as well. I want to add in a few more here. So we have


Building a PlotLy $GME Chart in Python

which one is it? It's the on balance I think. Yeah the on balance volume. Let's do that


Building a PlotLy $GME Chart in Python

one. And we're probably going to do something similar here again right. So let's go to our


Building a PlotLy $GME Chart in Python

presentation and let's get the on balance obv. So we have this. So paste that there.


Building a PlotLy $GME Chart in Python

Do something pretty similar again. So we have the API key. It's going to be our key. The


Building a PlotLy $GME Chart in Python

interval should probably check what that actually means for this. So this one's probably the


Building a PlotLy $GME Chart in Python

one that I was most interested in seeing. It's like it measures the positive and negative


Building a PlotLy $GME Chart in Python

flow with the intention of identifying where you've got a load of sort of big investors


Building a PlotLy $GME Chart in Python

putting a lot of money into it. Or sorry when you've got a lot of smaller investors putting


Building a PlotLy $GME Chart in Python

a lot of money into it. So basically the volume is higher because obviously the retail investors


Building a PlotLy $GME Chart in Python

are putting a lot more bets on but they're smaller buys or sells. So that I imagine would


Building a PlotLy $GME Chart in Python

be how this works. Although I've just read this very briefly before so it could be completely


Building a PlotLy $GME Chart in Python

wrong but it sounds pretty interesting from reading this. So let's just try and implement


Building a PlotLy $GME Chart in Python

it. I'm going to go with shorter time period than what it's suggesting here because I don't


Building a PlotLy $GME Chart in Python

know if we will see anything with that. Or shorter interval sorry. Then GME. So let's


Building a PlotLy $GME Chart in Python

put this in. Now we need to see what we actually get here. OK so we have this technical analysis


Building a PlotLy $GME Chart in Python

and it's basically the same as before when we had the SMA. So we could probably write


Building a PlotLy $GME Chart in Python

a function that does that for us but let's just do it like this for now. So we just change


Building a PlotLy $GME Chart in Python

this to ABV. It's a pretty easy change. OK. So let's try that.


Building a PlotLy $GME Chart in Python

I think this one should be pretty interesting to see. And I don't know to be honest I don't


Building a PlotLy $GME Chart in Python

we're going to need to put these on another chart because obviously that isn't going to


Building a PlotLy $GME Chart in Python

work. I imagine there isn't an easy way of doing it but let's just check quickly. Yeah.


Building a PlotLy $GME Chart in Python

OK. So we should probably have a look at how people normally do this.


Building a PlotLy $GME Chart in Python

Hmm. OK.


Building a PlotLy $GME Chart in Python

okay and let's try and stick with the graph objects


Building a PlotLy $GME Chart in Python

so I don't know if it's gonna go as yeah this just goes as express so


Building a PlotLy $GME Chart in Python

okay so we're actually just gonna use a scatter okay cool


Building a PlotLy $GME Chart in Python

so run something like this we go scatter then we do we add make some Y


Building a PlotLy $GME Chart in Python

okay so just swap our ABV okay cool so we have this interesting


Building a PlotLy $GME Chart in Python

so this is our on balance value is that right unbalanced volume sorry so I


Building a PlotLy $GME Chart in Python

wonder if this would make more sense because it's volume to have it as a like


Building a PlotLy $GME Chart in Python

bar charts at the bottom because I think that's how these things usually work


Building a PlotLy $GME Chart in Python

hmm


Building a PlotLy $GME Chart in Python

and I'll be interesting I think


Building a PlotLy $GME Chart in Python

okay so I have subplots here we have space very similar to okay perfect so


Building a PlotLy $GME Chart in Python

that looks kind of good but we want to stack so I mean it's just the same as far


Building a PlotLy $GME Chart in Python

as I know yeah anything that I'm not doing here is matching on the x-axis but


Building a PlotLy $GME Chart in Python

again that shouldn't really be an issue for us because we do actually have we


Building a PlotLy $GME Chart in Python

have already much on the x-axis so let's give that a go


Building a PlotLy $GME Chart in Python

so we want subplots make and we append trace this time instead so


Building a PlotLy $GME Chart in Python

so this time I'm gonna append trace I don't know if we can add scatter okay no


Building a PlotLy $GME Chart in Python

so we need to yeah I don't know if we should have rather been doing that


Building a PlotLy $GME Chart in Python

anyway because it needs to be more like this and then I suppose we can add mode


Building a PlotLy $GME Chart in Python

here or maybe not less that's not for now it's just a pen trace then we do


Building a PlotLy $GME Chart in Python

that for SMA as well


Building a PlotLy $GME Chart in Python

let's see how how that goes just want two rows and we've already imported this


Building a PlotLy $GME Chart in Python

so let's okay so we have data why okay and we also need to specify the rare in


Building a PlotLy $GME Chart in Python

the column as well okay so this will be row Sean this yeah this one could be


Building a PlotLy $GME Chart in Python

wrote to this one here will be very one cold well cold one and we just need to


Building a PlotLy $GME Chart in Python

remove the data part that we added in here so I suppose yeah


Building a PlotLy $GME Chart in Python

just pen trace and just remove that


Building a PlotLy $GME Chart in Python

okay so yeah it's pretty messy what okay so it's because we have this so I kind


Building a PlotLy $GME Chart in Python

of want to just get rid of that for now so I don't know what it is like a


Building a PlotLy $GME Chart in Python

excited slider maybe okay so this is kind of what we need I think yeah so


Building a PlotLy $GME Chart in Python

it's need to update the layout and just made us folks and I think suppose up to


Building a PlotLy $GME Chart in Python

here type date okay this should work let's see


Building a PlotLy $GME Chart in Python

okay cool so we have so now I have this


Building a PlotLy $GME Chart in Python

what is it most useful how do you normally let's have a look at how this


Building a PlotLy $GME Chart in Python

unbalanced volume is normally plotted


Building a PlotLy $GME Chart in Python

okay so this is the unbalanced volume here so it's just like in the line chart


Building a PlotLy $GME Chart in Python

underneath the other line chart so it's basically what we have already this is


Building a PlotLy $GME Chart in Python

interesting so here's like a confirming trade when they're both in the same


Building a PlotLy $GME Chart in Python

direction okay so this is what we want to look at here when the price and OBV


Building a PlotLy $GME Chart in Python

are making higher piece and higher troughs upward trend is likely to


Building a PlotLy $GME Chart in Python

continue so when they're both going up when I'm making lower and lower they


Building a PlotLy $GME Chart in Python

there's a downward trend if the OBV is rising accumulation might maybe taking


Building a PlotLy $GME Chart in Python

place so if we have a look at this so you see that both going down so that's


Building a PlotLy $GME Chart in Python

like okay the downward trend is likely to continue and here they're both going


Building a PlotLy $GME Chart in Python

up so upward trend is likely to continue and then they're just leveling out here


Building a PlotLy $GME Chart in Python

so it's not really anything of interest but I suppose here it's kind of balanced


Building a PlotLy $GME Chart in Python

and we see the momentum so pushing up again just before it rises so maybe that


Building a PlotLy $GME Chart in Python

is what we're sort of looking for either way it's pretty interesting to to


Building a PlotLy $GME Chart in Python

actually look at and then I also want to probably have a look at this shaking


Building a PlotLy $GME Chart in Python

oscillator as well so it looks pretty interesting I think so if we go back into


Building a PlotLy $GME Chart in Python

the API documentation so I think it's this one yeah so it's the tracking ad


Building a PlotLy $GME Chart in Python

line okay so when this is measuring big investors I think so when you're looking


Building a PlotLy $GME Chart in Python

for where the big investors are putting their money this is supposed to be an


Building a PlotLy $GME Chart in Python

indicator of it as far as I understand so it examines the strength of price


Building a PlotLy $GME Chart in Python

moves and the underlying buying and selling pressure to try and measure the


Building a PlotLy $GME Chart in Python

demand then divergent between the price and the oscillator is indicated as most


Building a PlotLy $GME Chart in Python

frequent single not and flagged a short term reversal in price so that'd be


Building a PlotLy $GME Chart in Python

interesting to see if we can maybe find that happening around here or around


Building a PlotLy $GME Chart in Python

here so let's let's figure that one out as well


Building a PlotLy $GME Chart in Python

so so this time we're just replacing the function of ad interval this one's doing


Building a PlotLy $GME Chart in Python

daily so maybe this is the sort of thing we want to be looking at more frequently


Building a PlotLy $GME Chart in Python

so you have the key GME and I think that's pretty much what we want let's


Building a PlotLy $GME Chart in Python

put this down to let's go 60 minutes again and then we can maybe change that


Building a PlotLy $GME Chart in Python

okay so let's see what we get so it's a striking ad okay cool


Building a PlotLy $GME Chart in Python

so let's pull that through so I'm gonna use this code again so replace this


Building a PlotLy $GME Chart in Python

I'm gonna call it ad or call it chat chiken ad okay and what was it just


Building a PlotLy $GME Chart in Python

striking ad right yeah so sorry one


Building a PlotLy $GME Chart in Python

replace this with striking ad again and a chat down here as well okay


Building a PlotLy $GME Chart in Python

so I think so this bit we need what do we on this bit to be so just striking ad


Building a PlotLy $GME Chart in Python

okay we've got two numeric so that all looks pretty good to me let's add that


Building a PlotLy $GME Chart in Python

in so obviously I was gonna need another another row let's just add that in


Building a PlotLy $GME Chart in Python

Chad we'll put this at the bottom we need to add another another row here


Building a PlotLy $GME Chart in Python

let's see how that looks so okay cool


Building a PlotLy $GME Chart in Python

doesn't look bad it's interesting so the point was that one day diverge is a


Building a PlotLy $GME Chart in Python

indication of a sort of a movement change and that indicates a reversal in price


Building a PlotLy $GME Chart in Python

so I suppose here's like the only bit where it kind of doesn't follow the


Building a PlotLy $GME Chart in Python

price because the price is pretty level if I was even going down here right and


Building a PlotLy $GME Chart in Python

here it starts to go up so maybe that's what that's indicating the reversal in


Building a PlotLy $GME Chart in Python

price change but it's also kind of difficult to because we're only looking


Building a PlotLy $GME Chart in Python

at our time here so we should probably increase the you know how much review


Building a PlotLy $GME Chart in Python

and so let's just go to like 10 minutes and see how that looks


Building a PlotLy $GME Chart in Python

okay was our bet a bit we can't do that


Building a PlotLy $GME Chart in Python

yes not returning anything it's probably the interval now you see okay so we need


Building a PlotLy $GME Chart in Python

15 minutes let's go five see what it's like okay so that's more interesting so


Building a PlotLy $GME Chart in Python

now we just see like this big movement up here but then I bet if we if we look


Building a PlotLy $GME Chart in Python

at long term it's pretty probably pretty stable so let's try and increase how


Building a PlotLy $GME Chart in Python

much data actually looking at


Building a PlotLy $GME Chart in Python

so the interval


Building a PlotLy $GME Chart in Python

okay so what's our Mendy T okay so this is returning okay so this is actually


Building a PlotLy $GME Chart in Python

returning Friday's I was wrong earlier when I said this is a Saturday's data


Building a PlotLy $GME Chart in Python

she kind of stupid but does this actually return us anything more than


Building a PlotLy $GME Chart in Python

what we have here


Building a PlotLy $GME Chart in Python

you


Building a PlotLy $GME Chart in Python

okay cool so if you want this full rather than compact so output size going


Building a PlotLy $GME Chart in Python

to be full so that should give us a lot more data so before we were heading it


Building a PlotLy $GME Chart in Python

11 on the 29th let's see what we have now okay that's way further back that's


Building a PlotLy $GME Chart in Python

cool we probably I don't know if we'll be able to even see that much though so


Building a PlotLy $GME Chart in Python

maybe it's worth maybe not doing a five minute intervals for going like a daily


Building a PlotLy $GME Chart in Python

let's see what that looks like of course that changes this as well


Building a PlotLy $GME Chart in Python

so let's change it here


Building a PlotLy $GME Chart in Python

hopefully that should work okay now so let's check what we are returning


Building a PlotLy $GME Chart in Python

okay so because we are doing the entry day we can go 60 minutes so let's change


Building a PlotLy $GME Chart in Python

that to 60 minutes okay on a good zoom further back I wasn't expecting that


Building a PlotLy $GME Chart in Python

it's pretty good there's a lot of data so now I'm thinking what do we want here


Building a PlotLy $GME Chart in Python

maybe daily do we want daily weekly for now


Building a PlotLy $GME Chart in Python

and these maybe we can get daily


Building a PlotLy $GME Chart in Python

so I saw look at these how they are now so here's our city so as a volume it's


Building a PlotLy $GME Chart in Python

pushing up loads and everyone's buying into it but you can see this is massive


Building a PlotLy $GME Chart in Python

increase here this is back in at the start of January it means yeah so start


Building a PlotLy $GME Chart in Python

January and yeah it was going up but not loads you can just see kind of pushing


Building a PlotLy $GME Chart in Python

up pushing up and then here you have same again it starts to go a little bit


Building a PlotLy $GME Chart in Python

crazy so here when over as it went greater than zero and this is January 20


Building a PlotLy $GME Chart in Python

seconds so here this is before it all went crazy so I think it kind of went


Building a PlotLy $GME Chart in Python

down here and then back up let's see it's pretty interesting so I mean I


Building a PlotLy $GME Chart in Python

think that's it for this video for now it's probably I mean obviously there's a


Building a PlotLy $GME Chart in Python

lot we could do like tidying up but just in getting a few indicators together and


Building a PlotLy $GME Chart in Python

the actual sock data in in candlesticks that was pretty easy I mean I've never


Building a PlotLy $GME Chart in Python

done most of this before with plotly and it's super easy to figure out so it's


Building a PlotLy $GME Chart in Python

pretty good so I suppose that's it so thank you for watching the video I'll


Building a PlotLy $GME Chart in Python

leave all this code on github as well I'll add a link in the description if it


Building a PlotLy $GME Chart in Python

helps I mean it's not too much anyway but might help you pull things together


Building a PlotLy $GME Chart in Python

a bit quicker so anyway thank you for watching and I will see you again next


Building a PlotLy $GME Chart in Python


Metadata Filtering for Vector Search + Latest Filter Tech

Hi, welcome to the video. We're going to be exploring two of the common methods that we can


use to filter indexes in vector similarity search. And then we're also going to explore


Pinecone's new solution to filtering in vector search.


Now, in vector similarity search, what we do is build representations of data. So that could be


text, images or cooking recipes, and we convert them into vectors. We then store those vectors in


an index. And what we typically want to do is perform some kind of search or comparison of


all the vectors within that index. So for example, if you found this video or article through Google


or YouTube, you will have typed something, some sort of query into one of those search engines.


So maybe something like, how do I filter in vector similarity search? Then whichever search engine


you use most likely converted your query into a vector representation. And then it took that vector


representation and compared it to all of the other vectors in its vector index. So those could be


pages, videos, and so on. It could be anything really. And out of all of those index vectors,


it was this video or this article, which seemed to be one of the most similar vectors to your


query vector. And so you were served this video or this article near the top of your search.


You clicked on it and now here we are. In search and recommender systems, there's almost always


a need to apply some sort of filter. On Google, we can search based on categories such as news


or shopping. We can search by date. We can search by language or region. And likewise, Netflix,


Amazon, Spotify might want to compare users in specific regions. So restricting the search scope


to relevant vectors is in many cases an absolute necessity. And despite the very clear need for


filtering, there isn't a particularly good approach for doing so. So let's start having a look at the


different types of filters available to us. So during the video, we'll be covering each one of


those. We have pre-filtering, post-filtering, and Pinecones new single stage filtering.


Now, what I want to just do here is have a look at what metadata filtering actually is. So when we


have a vector index, each vector is going to be assigned some sort of metadata. And that can be


anything. It could be a number, a date, text. It can really be anything that we can use to filter


our search. And what we want to do is search where this or some condition is true. So for example,


say we have a big corporation and they have all these different departments and there are loads


of internal documents within that corporation. Some of those documents are assigned to the


engineering department. Some of them are assigned to HR and so on. A user in that company might want


to go into their search and sometimes they may want to search across all departments, but sometimes


they might want to apply some sort of filter. So they might want to say, I want the top K documents


where the department is equal to engineering, or I want the top K where the department is not HR.


And we can apply anything in these metadata filters. So we may want documents that are quite


recent. So what we might say is we want the top K documents where the date is greater than or equal


to 14 days ago. And then we can sort of mix and match all of those different metadata filters


as well. Now to implement a metadata filter, we need two things really. We need our vector index,


which is what you can see at the top here. And we also need our metadata index. Now each of these


will be paired one by one. So each vector will have its own metadata record. And what we would do


is we'd apply a condition to our metadata index and that would remove a few of these. So we'd get rid


of these. And then based on what we have removed here, we would also remove those equivalent


vectors from our vector index. And that's how our filter would get applied. But there are


different orders and different ways of doing this, as we will take a look at now. So the first


one of those is we could have a look at a post filter. So a post filter is nothing more than we


take our query vector, which is over here. And we also take our metadata query over here. Now we start


by performing a approximate search between our query vector and all of our index vectors over


here. And we get the top k matches there. So let's say we wanted maybe 10 of those.


And then what we do is we then add in our metadata query. So we add that in over here.


And that creates a filter for us. So then we filter those remaining vectors through our new


filter. And that leaves us with the filtered top k. Now in this case, top k is not going to be,


usually, it's not going to be the number we asked for. So say over here we had 10 top k matches.


Here we filter some of those out. So we might end up with four. And then we also have pre-filtering.


Now pre-filtering, we change the order a little bit. We apply our filter before we do the search.


So we have our metadata query over here. We use that to create a filter over here. We then apply


that to our full vector index over here. And that leaves us with so many of our vectors. And then


what we do is we search based on that. But because we are not searching through the full data sets,


we can't do approximate nearest neighbor search. So we have to do an exhaustive search at this


point. Now let's start with the pre-filter process. So here, like we saw before, we start


with our metadata index. We apply our filter to this, identifying which positions satisfy our


filter condition. And then we use this filtered metadata index to filter out the vectors which


do not satisfy our condition. And as we kind of saw before, this is where the issue with pre-filtering


comes in. Because we have just filtered out some of the vectors or many of the vectors in our index,


we no longer have the same index as what we started with. And we need full index to apply


our approximate search on. But as soon as we filter, we change the structure of that index.


So we can no longer perform an approximate nearest neighbor search, which means we're just doing


a brute force, exhaustive k-nearest neighbor search. Now, if our index is very small or the


number of vectors that we have output after our filter is very small, this is probably okay. But


as soon as we start working with big data sets, this is not going to be very manageable. And the


only other alternative that we have here is to build an index for every possible filter outcome,


which is not really an option because it's just simply not realistic to build that many indexes.


So pre-filtering, we have good accuracy, but it's very slow. Now, post-filtering is of course


slightly different. So in this case, we start with our vector index. Now we can perform our


approximate nearest neighbors search because we have the full index. We haven't filtered anything


yet. And that returns the top k vectors that we want. So say we want 10 vectors at this point.


And then what we do is we find all the vectors through our metadata index that satisfy whatever


metadata condition we have set. And then we apply the filter to those top k vectors, so 10 vectors


maybe. And of course, at this point, what we are doing is we're reducing the number of vectors that


we get out. So we don't actually get 10 vectors. We, for example, could get four vectors. And in


the worst case scenario, that filter could rule out all of the vectors that we've returned. And


in the end, we return nothing. Even when in the index, there could be some relevant vectors,


which is obviously not very ideal. We can try and eliminate this problem by just increasing k a lot.


So of course, if we use a low k value, the chances of all of them being excluded when we apply our


filter post search is reasonably high. But if we increase k up to 1 million, it's much lower.


But the only problem with that is that our search becomes very slow. And the more we increase k to


eliminate the problem, the slower it gets. So in this case, we have unreliable accuracy or performance.


But it is faster unless we increase k. So now let's introduce single stage filtering by Pinecone.


Now we are going to go through some code and test this. But first, I just want to introduce


what it is at a high level. So it's a new filter built by Pinecone. And at a high level, it works


by merging the vector and metadata indexes. And it allows us to filter and then do an approximate


nearest neighbor search. So what we get there is the accuracy of pre filtering. And at the same time,


the search speed is often faster than post filtering as well. So we rarely do get the best of both with


this new filter. But let's go and actually try it out. OK, so we're going to be using Pinecone here.


And all I've done here is imported Pinecone, imported JSON, and I've imported my data here.


So this data I've already uploaded or asserted to my Pinecone client. And what it is, is just


the squad data set in both English and Italian. Now in there we have a few different items.


So we have the record ID. We have the text that I've just sold this locally. We have the vector,


which has been observed to Pinecone. And then we also have the metadata, which is with Pinecone as


well. Now if we take a look at what we have in the metadata. We see that we have the language,


so we have either English or Italian. And then we also have the topic. Now what I want to do is


just test the new filtering. So we're going to be filtering based on language, topic. And we also


have another metadata item here, which I don't have locally, which is just a randomly generated


date. So we can have a look at using some of the greater than or equal to, less than equals to


filters that we can use in Pinecone. So the first thing I'm going to do is initialize


my connection to Pinecone. So I write pinecone.init. I need to pass my API key, which I've loaded above.


And also the environment that I am working in. Now of course this will be different and depend


on which environment you are using. So I've initialized there and what I can do is I can


now create a direct connection to a specific index within my Pinecone environment. Now I'm going to


be connecting to one I've already made, which is called squad test. And now what I'll be able to do


is use this index object to perform my queries. So we're going to be performing a vector search here.


So what we need first is a query vector to perform our search with. Now I use the sentence


transformers library to encode the already indexed vectors. So what we're going to do is use the same


model to encode our query vector. So write sentence transformer and that embedding model


is sentence transformer.


And I use the STSP xlm r multilingual model. So I will need to download this.


Okay so that is downloading now.


And then what we want to do is create our query vector. So I'm going to assign it to xq.


And all I need to do is write embedder.


And dot encode. And then I pass in the query that I would like to perform. So in this case I'm going


to search for contexts in our data set which mentioned something along the lines of early


engineering courses provided by American universities in the 1870s. So I will execute that.


And note that we're using a multilingual model here so we should find that we will return


both English and Italian results. But both of them should be something similar to this topic.


So we will return our results. We just write index dot query.


And what we can also do before we even do that is we'll just convert xq into the format that we need.


So like this. And then in here we just pass xq and I'll say that I want the top k value to be 3.


Now remember if we were doing if we were using post filtering here if we set top k value of 3


we would probably return less than 3. So with post filtering we would want to set something


stupidly high here just to get maybe three samples if we're lucky. But as we're using


single stage filtering we only need to set top k equal to 3. So we'll execute that and let's return


the results. And you see here that we get our ids. So we get this id 1, 2 and 3. So what we now want


to do is we want to map that back to the data that we have sold locally. To do that we're going to


write ids equals i id for i in the results. So we're just getting these ids here. So we're going


results we need to enter into the results key. We want to access the first position in that list


and then in there we want to access the matches.


And then from there we'll just print ids see what we get. Okay we get those three ids.


And now what we want to do is use the data that I imported up here.


Just here. And we're going to use that to print out whatever it is that these ids are referring to.


Now what we have in our data at the moment is a big list which is not that useful.


So it would be more useful if we just reformat that into a dictionary.


So I'll do that quickly. We're just going to write get sample


is equal to x id and then in here we will store our context and metadata. So context


and metadata. We definitely saw the vector in here


because we can't read that anyway so it's not that useful for us. Let me say for x in data.


Okay now we can. Okay so it's not context. Let me come up here and have a look at what we have.


Look at what we have. I think it's text. Yeah okay so let's change that to text here and here.


Okay so now we can do for i in ids.


I want to get the sample. So get sample i.


And we'll just print that out.


So we see here that first one we get is Italian and the translation for this is


something to do with the College of Engineering was instituted in 1920. So we have college,


engineering that's good. And then we also have something along the lines of the College of


Science from the 1870s. So generally this looks pretty relevant I think. And then down here we have


this is Italian again but we also have the English translation of this here as well. So we can see


straight away School of Engineering, public engineering school founded 1891 and it offered


engineering degrees as early as 1873. So that's again pretty relevant. Now I don't understand


Italian and so my first filter here would probably be okay I only want to return the English results.


So let's go ahead and do that. So I'm going to say results equals and let's just copy what we had


up here so we're not repeating ourselves. We just want to take the index query we can include


result no we don't need it. Let's just take that.


Just take that and all we need to do now is add our filter. So we just write filter


and then in here we want to write so we have our metadata and we have our language


and we want to say that this must be equal to so we use eq


en which is English. So we get our results and we're going to want to do the same thing again.


So we'll want to get our ids and we want to print those out.


And there we go. So now we're just getting English results. Now that was pretty fast so let's I think


what is quite useful is to see how fast those two searches were. Now obviously we're getting


pretty relevant results here where again we're returning three results even with our filter


applied. So that's good. So it seems like we're getting the accuracy of pre-filtering here.


And let's have a look at the speed difference between the two approaches. Now we shouldn't


see anything particularly major because this is a very small index. We only have I think 40,000


vectors here. So we won't see anything significant but at least we can check that we're not going


anything slow. So let's have a look and you see here that we're actually getting a slightly faster


response when we filter and this is typical with pinecone single stage filtering. When we add a


filter usually we'll actually get faster results which is pretty insane. So not only are we getting


good speed like post-filtering but we're actually making our search faster by adding a filter which


is neither post-filtering nor pre-filtering can do that. And again at the same time we're still


getting that accuracy of pre-filtering. So this is in my opinion pretty impressive. Now we might also


want to add another filter. So at the moment we're just adding one filter which is fine.


It works but let's say I look at my results and I know this is hard to read but in here we have


the topic we have University of Kansas. Okay fine maybe I'm not interested in the University of


Kansas. So how about here we have University of Notre Dame. Let's say I'm not even interested in


these guys either. Institute of Technology let's say okay yeah we can keep them that's fine.


So I want to say okay I want everything that is one in English and two not from the University of


Kansas and not from the University of Notre Dame. So to do that I need to add another condition to


my filter. So to do that all I need to do is say topic is not in this time. So we're going to say


not in and then we pass a list here. So a list of what we don't want to see which was the University


of Notre Dame and also the University of Kansas. Okay so let's add those two and let's see what


let's see what we get. So again seeing pretty fast and we're getting University of Canvas here so


that must mean that I have written something wrong. And so I think here the topic filter


up in my pankoindex is actually maybe called title. Let's see and this is also wrong so let's


correct that. Okay so now we're getting something different. So yes this should be title in reality.


So Institute of Technology, Institute of Technology and where is our other one? Institute of Technology


here. Now we're not returning University of Kansas and we're not returning University of Notre Dame


which is what we wanted. Now there was also the date filter that I wanted to show you as well.


So we don't only need to filter based on strings but can also filter based on numeric data times.


And for me to show you this I think it's best if we it would also be better if we


it would also be better if we include our metadata here as well so we can just see it


directly from our results. So we don't have a returning relevant text. So now let's just have


a look at the metadata. So I'm going to include metadata there. Let's just see what we get.


So we see now that we actually include the metadata in our results which is also pretty cool.


Now we have date which is just a numeric value here. It's just something very simple. It's just


randomly generated. There's nothing, there's no actual relation between the date and this


record. It's completely random and we can see okay we have a date from 2016, 2008 and we also have


2020 here as well. Now the first thing that I might want to do is say okay I want to return


only the more recent date. So let's say okay we add, we keep all of our other filters in there


and we might say okay but we also want date to be greater than or equal to


let's say what do we have here? We have


what is the most recent? So it's 2021. Let's say we want to go for ones that are


let's say 2018 onwards for now. So 2018001. Okay so the very first day of 2018.


Let's search and see what we get and we can see yep it's definitely filtering correctly there.


Now let's have a look at what the search time is for that.


So adding quite a few filter conditions here. So let's just see what we get.


We should also exclude that.


And you see it's actually slightly faster again which is again it's pretty cool.


But like I said it's a small data set. When we do this on bigger data sets the difference can be


huge. Now what we can also do is we can actually add another condition within our date here. So


we can say okay we want it to be greater than or equal to 2018. But let's say we want to search


for records only in 2018. So we might also say okay we want it to be greater than 2018 or the


first day of 2018. We also want it to be less than or equal to the very last day in 2018.


So 20181231 and we will filter and we see that now we're only returning records for 2018.


We see that now we're only returning records from 2018. So again super cool and I think


an incredibly useful functionality for vector similarity search.


Now we were just using a very small data set there. So I could really show you


how impressive the speed up can be when we're applying filters. But I do have this other index.


Now I'm not going to go through just coding everything because I mean it's pretty straightforward.


We have a index here which is 1.2 million vectors and it has a single metadata field in there which


I've called tag1. And that's just a randomly generated number or integer from 0 to 100.


So we of course initialize the connection to our index in the first cell up here.


And then over here I'm just creating a random query vector. So first this here


is our unfiltered search. So we get this 79.2 milliseconds. Now again most of this is network


latency rather than the search time in the actual index. But we will see the search time decrease


pretty dramatically here. So first we'll say okay we want tag1 to be greater than 30. So we're going


from 0 to 100. So we're roughly removing probably about 30 percent of the vectors from our search.


And we can see okay we've just shaved off eight milliseconds which is impressive.


And then we take that even further. So we say okay we want greater than 70. So now we're shaving off


around 70 percent of our vectors. And our search time goes down to 56.6 milliseconds.


Do it even further. So about 90 percent here. We go down to 54 milliseconds. And then here I'm using


the equals sign here. So I'm only searching for about one percent of the index. And it goes down


to 51.6 milliseconds. So incredibly impressive speed up there. And this is kind of what it looks like.


So we have the tag1 gt value or greater than value on the left. And as we increase that


up this way our time our search time milliseconds goes down. Now it is a little bit bumpy. It goes


up and down a lot. I've tried to showcase that in this graph. But the trend is quite clearly downwards.


So the more we filter the faster our search which is incredible. Now that's it for this video.


Covering pre-filtering post-filtering and Pinecone's new single stage filtering.


I hope this has been useful and insightful. If you are interested in testing Pinecone out yourself


there is a link to Pinecone's website in the description. But we'll leave it there for now.


Stoic Philosophy Text Generation with TensorFlow

Hi, in this video we're going to go through, or what I'm going to sort of redesign this code here,


which is the code that we are currently using to actually train a recurrent neural network model.


It's been trained on these two datasets here, which I can actually show you over here.


So the model is actually training on meditations by Marcus Aurelius, and also letters from Stoic


by Seneca, which is in Latin this, I'm not going to pronounce it.


So we're pulling them from these two sources, both open source, which is really good.


The only thing with letters from Stoic is that this is the parent page, and then we're using


Beautiful Soup to pull the individual letters, so there's 124 of them.


So we put them all in, put them into a dictionary, and then that dictionary, we extract the text we


need from that, join it all together, and then join that with meditations, and that is our training


data. It's literally just a big huge string containing all the meditations and all the letters,


which you can see here. I'm just drawing it together.


So this is the formula we get letters in.


Okay, so we have the letter name, and then we have the local web address and the text itself.


So we join all those together into one big string, and then after that we join meditations and letters


into the data that we use for training. So I'll quickly summarize what we do to pre-process the


data and actually build a neural network, and then I'll get started with it. So we take a vocab,


which is a unique set of all of the unique verbs that we have within our data, but actually sorry


for this, we're using characters and not verbs. So you can use either in NLP, sometimes you use


words, sometimes you use characters. Characters are just kind of easier to set up, you don't


really have to do as much with that. Let's draw back some benefits for both. Usually you probably


use verbs, but in this case we're going to use characters. Then we create a character to


index dictionary. So when we feed the characters into the data, into the model, they need to be


numbers, they need to be integers. Those integers represent an index in an array. So we will have


the first layer in the model is a embedding array, which is like a three-dimensional model,


a three-dimensional tensor, which has the indices, the numbers, the integers, the numbers,


the indices of each letter going down here, and these across here is the embedding dimension.


So character A in our case will have 256 float point values that represent it.


And then it's three-dimensional because we have a batch size of 64, so we feed in 64 of everything


at any one point. So that's why we have the character index to convert from a character


into an index, which can then be read into that embedding layer. And then we have the pulling it


back. So after we've trained and we want to convert our predictions into readable characters again,


we use this, which goes from the index to the character. And this is just converting our data


into indices. Here is a sequence lamp, so that is the window of text that we read at any one point.


So imagine, let me give you an example. Imagine our window is four characters, and we have


the input of hello with two Ls, not three. So our input has a window size of four. So it reads hell.


Our outputs, because we are using this to predict or generate text, our outputs will be pushed


one index along. So where is our input is hell, our output will be hello. That will be our target.


And we do this throughout the entire text. The only difference for us is that we are doing it in


sequences of 101. So define these. So define here. So you can set 100. And then a few other things,


like I said, the batch size 64, embedding dimensions 256. So we have a few things in there.


I'll come down to here. So here we're using a TensorFlow dataset object, which is really useful.


And that, essentially, we put our data into this dataset object. And then we can use the functions,


useful methods on our dataset object, like batching, shuffle, and also batching again here.


So that's really useful. This here, that's what I said before with the hello. So that chunk


can be hello. Input data, if I let me actually put comment. So we're inputting hello. This input


data will be hell. And this will be hello. So that's why we're splitting into our input and


target data here. And then we map the dataset object built here. We map the dataset object


into this function to get our correctly formatted input output. And then we can use the function


to get our correctly formatted input output dataset. So we're using shuffle. So here,


sorry, let me just go this quickly. So this is where we are splitting into the sequence


that for 101 at a time. So you can see sequence length plus one. And then anything remaining


after that because obviously the dataset is very unlikely to correctly squeeze into sets of 101.


So we just drop the remaining characters. If there are any, there probably will be,


but it's not going to be many. So it's fine. So do that here mapping, like I just said,


and then here we shuffle the dataset. Both sides we define up here. So we shuffle the dataset.


And the reason we do that is to give a better representation of the data with every single


batch. So we're going to train on 64 sequences at any one time, and then update the model weights.


So shuffling dataset means that especially with this data as well. So we have the first part data


meditations, second part letters. If we didn't shuffle the data, it would be training on 64


sequences of meditations at any one time. And then within that, it would probably also be speaking


about a specific topic at any one time, or a specific few topics. Now it doesn't give a very


good overview of everything we're training on. So it updates those weights according to that specific


topic and that specific book, specific text, so meditations or letters. So we shuffled dataset


to give a better representation of the data in every single batch. So now instead of having


just meditation, just one topic, it's going to have a few different topics from meditations and


a few different topics from letters. So that works a lot better. And then batch. So we already


use batch here to split into sequence lengths. And then here we use batch again to split into


batches of 64 sequence lengths. So here we're building the model.


So I have two different, I'm going to change all of this by the way, this is what we're doing.


So we have a GRU unit model or LSTM unit model. So they change here.


So I'm not going to go into too much, I want to be quick. But the embedding layer, as I said before,


is a FOCubsize. So FOCubsize is how many characters we have this way. So there are two,


I think 85. The embedding dimension is how, it's 256, it's how detailed every single character


is. And then the batch input shape is the number of sequences that we're going to put in there in


one time, which is 64 in this case. Then we have our LSTM unit. This is where the actual


learning, sequence learning comes into play. So it stands for long short term memory unit.


This is a gate recurrent unit. The point of using these over a recurrent neural network is that they


retain a sense of memory long term. They do that through these different gates within the units,


which is really useful, obviously, with text. And then the other point is that we're using dropout


of 10% on both of these. So these units are naturally very deep, so they can outfit really


easily. So having a dropout of 10% means that we mask 10% of the inputs at any one time.


So in a sentence, or in a word, say hello again, this will mask


one of what we'll mask 10%. There isn't, you know, we can't really split that. So


we'll mask one letter. Okay, so hello will become H blank, L, L, O, right.


And that just helps the model generalize. And then here, this is our classification. So it's


just a typical neural network densely connected, hence dense. And that outputs into our


our folk-up size, which is probably, I think, 85, which essentially means output zero will map to A,


output one will map to B, and so on. Okay, and then that's after that, we use the index to


character dictionary again. Okay, so here we're just building models, this is in a function.


Okay, summarizing model, creating a loss function, piling the model using Adam optimizer,


and for the loss function, sorry, sparse categorical cross entropy. Okay, here we are


saving the model weights, every epoch, defined by this here, which is fed into TensorFlow


TensorFlow during training. Okay, so callbacks, checkpoint callback, which is here. Okay.


Then at the end, we restore the final checkpoint, rebuild the model. Okay, so here building it


again, but this time instead of a batch size of 64, it has a batch size of one. Okay. So that


is that. So the reason we do that is we don't want a batch size of 64 when we're predicting,


because then we'd have to put in a list of 64 starting strings. And we don't want to do that.


So we'd have to put like from 64 times in the list, which doesn't make sense. So instead of


doing that, and that would take a lot more computing power as well. Okay. So we essentially


flatten the model a little bit. And then it will only have one batch at a time, or a batch size of


one at a time. So then we just feed in the word from and it will predict, okay, it will generate


text. So we rebuild it here, load the weights into it and build it again. Okay. Now we just


summary the model, which is the same as above, but like I said, one instead of 64 for the batch size.


Here, I'm just clearing out the memory of the checkpoints, because there's a lot of them that


take up a lot of memory. So I do that after it's loaded or the most recent one. And then here,


I'm saving the model and the character to index dictionary, which I'll go through that later.


Then here we generate text. This is an old text generation function. It's updated a lot now,


which I can show you later. So here, data writer, saving the model, saving the character to index,


and then here. So what we have down here, which is kind of interesting,


I haven't seen this done that often before.


In this scenario, I don't really think I've seen it happen. I've seen someone else do it for


a chatbot. But other than that, I haven't seen anyone else do it. It's using multiple


kernel networks, scoring them based on their output, based on their English and how grammatically


correct everything is and so on. Scoring their outputs and then choosing a winner. Okay.


So you can see here, there's some really rubbish ones because they're not properly trained.


Okay, but some of them are obviously a lot better. So these ones, you can see where it says


meditations, they've only been trained on meditations by Marcus Aurelius and they


tend to do worse a lot of the time. These are trained both. Okay. So this one has the top score


of almost 20. What is a cloth vessel if it were no useful book? Okay. And then here


has a score of 14.12. These are a pleasure and a mock of all those in which it were something. Okay.


And actually you can see, so it's these ones here. These are the scores. So if we scroll up,


we can see one of them went a little bit crazy. Got a really bad score, minus 120.


I don't know why this happens. It's very weird. But sometimes I just go crazy, which is why I


sort of built this ensemble, Recruit No Network learning thing method, because some of them


occasionally go a bit crazy. So by doing that, you have four other models or three other models


in this case, which are there to back it up. So one of them goes crazy. One of the other models


takes over. Okay. Which is really useful. I saw the output from one of these that I haven't really


trained very much and it was so much better already. It was so much better. So the thing


that you might think is, okay, using different models, so how do you keep them speaking about


the same thing? And they do that by the winning. So we split in sentences, we rate a sentence,


the winning sentence goes back in to the models. And then that is used to generate more text. So


they're continuously being updated with the best new sentence. Okay. Which is pretty good.


So this is the new text generation function. It still needs some work. It's still really


quickly put together at the moment. The rating function. So I'll just kind of go over really


quickly if the text is empty, because I was getting some errors before where text would


be empty every now and again. I'm not sure if that was an error in my code or the models


were just being weird. I think it must have been an error in the code. Somehow I need to remove


this and then actually figure it out. Okay. So if text is empty, just return it because otherwise


it'll throw an error when rating the rest of it. So then we normalize text as in we remove all


punctuation and lowercase everything. Okay. And then here we check for correct punctuation. So


at the end, right, at the end is it full stop, exclamation mark, question mark, or near the end,


e.g. if there's a full stop and then a new line character, is there a full stop, exclamation mark,


or question mark? This doesn't fully work at the moment because it's splitting. It stops the


text generation when there's a full stop. For example, so I need to update that so that it


stops the text generation when there's a full stop, exclamation mark, question mark, or new line


character. The other alternative when it stops text generation is when it's generated too many


characters, which is a limit of like 500 at the moment. So it does that too. Okay. And then we


check for too much repetition. Like if it's going to say, you know, there, there, there, there,


there, there, there, it's probably a problem, I think. So then that happened like occasionally,


not so much with these models. But I have seen it happen quite often before. So that's quite a good


rating, a wave rating as well. And then here checking all the words, the actual words according


to the vocabulary that we had. So we actually saved the vocab. So I built this separately.


I don't know if I still, I have the code somewhere, it's something common. So I just


read in all the data that we have, have and then split into words and split that into a text


document. Right. Which is here. Can we see this? I don't know. I don't know if you can see this


in the PyCharm. What have I done? Okay, so here it is. So you can see just a list of all of the


words. Okay. And then we just put that into a reg ex here. And we, you know, search it. Okay.


It's in there. If it's real, or it's in there already, it gets a good rating. Otherwise, no.


Yeah. Okay. And then here is our ensemble class. So here is where, so this predict method here.


So initialize a prediction dictionary called self-dominantations. So this will have the model


name, the score, and then the text. Okay. Which you can see here. And then that is used by Gladiator


predict, which also controls this function here. So this function actually generates text and scores


it. This one here just controls that function. So it runs that function. Yeah. And then it finds


the highest scoring sentence or sequence. Okay. So the highest scoring one is added to the text.


And then the highest scoring one is set as the new start sequence. And then initially we keep


the start string because it's something we typed in like from, and we need to keep that in for the


first iteration to make sense. But then after the first iteration, we don't want to keep what we're


feeding back into it because it's a previous sentence. So we just want new text after that


point. So we just set that to false. And then it goes through, loops through, depending on how many


times we have set it to. So here's 10. Okay. That's going to loop through 10 times and produce


our winning output. Okay. So that's what we have so far. What I'm going to do now,


what we need to do now is refactor this into something that is clean and


not so messy. So I'm going to rebuild it into, I think, a new model, a new, sorry,


a new Python file called train, I think. Yeah, I think train is fine. We're going to call it train.


And then we will refer to that whenever we are building a new model and training the model.


So then everything is segregated a bit nicer. Okay. So I'm going to go ahead and get on with that.


I'll describe what I'm doing every now and again, but for the most part, I'm just going to fast


forward, the basic code being made. It should be okay.


Okay.


You


Okay, so what we have done now is just refactored


just rebut that code into a class here as you can see so now the model is initiated


initialize here we format the data and then we build a model so my intention


like splitting it like this is that we can format the data okay and then I can


pass multiple model build parameters to it and build several models at once so I


haven't built that up yet I'm going to go so it's just been like a loop here


and it will loop through different model parameters and build them all up


throughout you know over a night or something so we have several different


models that we can then use in the ensemble class up here okay and then


after that we will have you know several good models hopefully good models or


competing to get the best text right so I think that is pretty good so far it's


one thing I just noticed actually here


Composite Indexes and the Faiss Index Factory

Hi and welcome to the next video in our similarity search series.


So far we've worked on quite a few different indexes


that we can use in FI's for similarity search


and through the past few videos and articles


we've tested a lot of different indexes


and we've sort of tested to see what works and what doesn't work


and the sort of pros and cons of each of those indexes.


Now that's very good and we've had some I think pretty interesting and cool results


but in reality the best or the highest performing indexes are all composite indexes.


Now composite simply means an index which is built of multiple parts


and all safety art performances are produced by these composite indexes.


So what we'll cover in this video is composite indexes at a high level.


We'll just introduce them.


We'll also learn how we can use the FI's index factory function.


Now when we're building composite indexes this function is incredibly useful


and I would say almost essential.


You can build composite indexes without it


but it's very difficult and the code is pretty horrific to actually work with.


So I think index factory is pretty essential in building these sort of indexes


and we'll also explore how we can build a few of the more popular composite indexes as well.


So IVF-ADC, Multi-D-ADC and IVF-HNSW indexes.


And let's explain what all of those are because I know it's just a load of words and letters.


So first we want to have a look at the composite components.


So the components that we can have within a composite index.


So there are two sort of main components that we're pretty much always going to have


and that is so in the middle we have what is typically a course quantizer.


So I'll just put course here.


This is stuff like IVF and we can also use HNSW here.


And then we pretty much always have another component which is our fine quantizer.


Now in terms of fine quantization you're looking at things like product quantization


but in this sort of space we would also just include flat vectors.


Although that's not really a form of quantization it's just using flat vectors.


They would be sort of interchangeable.


So flat vectors get put into this box not because they are a form of fine quantization


but simply because flat vectors would be placed into that position within the index structure.


Then the most common addition to these two would be the preprocessing step.


So the preprocessing of our vectors.


And here you have a few things.


You can have PCA which is dimensionality reduction or we can also have OPQ.


Now OPQ is optimized product quantization


and in order to integrate both of these we would have to have PQ down here as well.


And all this OPQ step would do is rotate the vectors that are being placed into our index


in such a way that it optimizes them for product quantization later on.


And then these three are probably the most common ones.


But we also have another block as well and that comes at the end here.


And this is a sort of like a refinement step.


All throughout here we've sort of built our approximate search


and then we return top 10k or top 10,000 most relevant results according to our approximate search.


And then we perform an exhaustive search at the end


which will re-rank them based on a more definitive I would say search.


Here's just a few examples of that.


So we have the vector transform, coarse quantizer, fine quantizer and refinement steps there.


So for example what we might do is when the vectors are incoming we transform them,


we rotate them using OPQ.


Then we perform coarse quantization and assign all of those incoming vectors,


those rotated vectors now into different lists within an IVF index.


Once they've been assigned to their different clusters or cells


we use product quantization to compress them and reduce the memory usage of our index.


And then when it comes to search time what we might want to do is use a refinement step


which would take the full vectors which in this case would not allow us to reduce the memory


usage of our index significantly.


But we could add a we could add like a PQ refinement step here so those vectors are


stored as their quantized forms or compressed forms rather than the flat vectors,


whichever way works for your index.


During our search we would go through each of these steps and turn the top 10,000 items,


so the top 10k and using our final refinement step rather than using IVF we would just take


the Euclidean distance or the L2 distance between all of them and find the ones that are most similar.


And this is an example of that.


So this is this is IVF PQ. This is an example of the two middle parts of that so IVF and PQ.


We are excluding the re-ranking and excluding the OPQ part.


And what we're doing here is we take our original vector we assign it to an IVF cell


and then we also take that original vector in a separate almost stream.


We quantize it, convert them into the codes and place that under the IVF cell assignment.


Now I think that's plenty for an introduction to Composite Index.


Let's move on to how we start building them using the Index Factory.


Okay so here I'm just importing our data.


We're using the SIFT1M data set as we usually do.


If you don't have that there will be a link in the description to the download script that


and I've also just imported FICE up here.


So the first thing I'm going to do is show you the difference between


building an index with the normal method that we've been using in every previous article


and video and how we can do the same using the Index Factory.


So what we need to do is we write quantizer,


FICE and in here I'm going to write index flat L2.


So this is because we're using the flat vectors.


Again like I said before the flat vectors are in the place of where we would put a


fine quantizer usually but they themselves are not actually a fine quantizer.


And then to build our index from that we would write FICE and this is our index flat IVF or IVF flat.


And in here we first pass our quantizer, then our dimensionality, then we pass nlist which


is the number of our Arnois cells or oculuses that we want to include in our IVF list.


Okay after that we add our vectors.


Actually we train them first so we train on those vectors and then we add them.


Okay do I need to oh I need to rerun this okay.


Run that and run this again.


Once we have those we can we can do search like we usually do.


And that's how we build our index and then add data to it.


So that's the usual approach.


Using the Index Factory approach is slightly different.


So what we do is instead we write I'm going to write index F just so we can


compare the two indexes later on.


And all we need to write is FICE.


We call it index factory function and in here we need to pass the dimensionality of our vectors


like we usually do but then we pass this it's called an index factory string.


And in here we specify the structure of the index that we'd like to build.


So we're going to say IVF 256 so a IVF index with 256 cells followed by or which contains flat vectors.


And that's it.


So we can run that and then we do the same thing again.


We do index F train xb and index F add xb like so.


And that should work.


So once we've done that let's what I want to show you is that they do in fact


output the exact same results.


So we need to do DI so what we normally do index.search xq k.


k I haven't defined let's go with 100.


Oh let's go 10 because so we can actually see the results.


And the results okay and then let's do the same but for index F which is our index factory index.


So here we just replace index with index F and you should see the same thing again.


I'm going to just replace these as well.


Okay and I mean we can see they're exactly the same.


There's no difference between them.


So we were returning the same nearest neighbors using both.


So they are the same indexes is what I'm trying to point out here.


The search time is pretty much the same.


I always find it's like very slightly faster when you use index factory.


But I mean very slightly it's on like micro scale value like maybe five microseconds faster


which is obviously not that much.


And in terms of memory uses are both exactly the same as well.


So that's a very simple example.


But obviously when we start building more complex indexes the index factory becomes a lot more useful.


So let me show you a good example of why we'd use index factory.


E.g. let's put together a more complex index and we'll see the difference between the two approaches.


Okay so I've just pulled this code in here and this is I said before I have those kind of like


four different components that we can have.


We have pre-processing and we have like the course and find quantizer.


So define quantizer here I've just renamed as Vect because it's not really quantizer.


So it's just flat vectors.


And then we also have the sort of the set that comes after where we're re-ranking the vectors as well.


And I also need to add that in so it's just index equals vice index refine flat.


And in here we have to pass all of this.


So I'm going to rename that to Q and add it in here.


So here we're building our index like we usually do.


So we've got our rotation OPQ rotation which is a pre-processing set.


And then we go on to our IVF.


So we're putting sorting those out into different cells.


And we are using the define quantizer it's not really quantizer it's the flat vectors.


And then we're merging all of those together into this sort of merged almost proto-index.


We could use this as an index but then we want to add another post-processing set which is a re-ranking.


Which is what we've done there.


Okay and I mean all this code is quite complicated.


I mean at least to me it's not that easy to read there's a lot going on.


So run that it can take a little bit of time to build to be honest.


Maybe it's better if I use less data.


I think it's fine for now.


If it stays too long I'll reduce how much data we have.


And if we wanted to rebuild this in using the index factory it's super simple.


All we do is we write index equals fice index factory.


So literally we're going to do this on one line and pretty much all of this.


Index factory and we have our dimensionality D and then we have our string.


So first we have the obq set so the pre-processing set.


In there using an m value of day two.


Then that goes on to our ivf set so we're using ivf there.


We are using a n list value 256.


256 that's flat.


Okay I've just added these so it's not training for too long.


I've added these in here.


So here we're not using flat so we're using flat vectors here.


But then we're using ivf pq so this is actually pq and we're using m here.


So m is day two up here.


Okay and then the next step after that is our re-indepthing or refinement step.


And that's just our flat like this.


And that's it.


That's the whole...


That's the whole... what we wrote all of this.


We compress it into the single line with the index factory.


Which is why when you're doing the more complicated stuff.


Or at least the comps when you start using comp indexes.


I think index factory is practically essential.


Missing it is... or trying to do without is...


You mean you can do it of course like we showed here you can.


But it's difficult and it gets a lot more complicated than this as well.


So I think it's probably the best approach in my opinion at least.


Now if we train and do the same again where I'm just doing first 50k.


Which is still taking forever.


Maybe because we're recording at the same time.


Okay and one thing to do here is di equals index search.


And then we can again we'll just see what we return.


And make sure they match.


Okay and let's have a look at what we have.


Let's print out i before we lose it.


Before we lose it we do the same here.


So and we'll just do the search as well.


Index search xqk.


And again we just want to see i.


Okay and that's finished and we can see.


So we compare those results and they are exactly the same.


Which is what we'd expect because we are building the same index.


So you know from this to this using the index factory.


So I think it's fair to say there is good reason to use it when you're building these composite indexes.


Now in the next video we're going to be covering the composite indexes.


And how we can build them using the index factory.


Now if you are watching this on YouTube it's the next video in the playlist.


Otherwise if you're on if you're over on Pinecone reading the article it's below.


So you can read about it or watch it there.


Testing the New Haystack Doc Store

Today we're going to continue going through the Haystack and Pinecone


Testing the New Haystack Doc Store

integration that I've been building. We're just going to go through like a demo notebook


Testing the New Haystack Doc Store

that I put together and see how everything works. This is essentially how


Testing the New Haystack Doc Store

I'm testing the document store as I build it and then in or maybe not this