So if we...


HNSW for Vector Search Explained and Implemented with Faiss (Python)

We will have a million values in here once we add our SIFT1M dataset.


So what we do is we just write npbin count levels to count the number,


to count basically how many vertices we have in each level.


When I say level, by the way, it's the same as earlier on when I was saying layers.


They're the same.


So when I say levels, layers, same thing.


So obviously at the moment, we don't have anything in our graph.


So how do we add that?


We already know.


We just do index add xb.


Okay, nothing else.


So we add that.


That might take some time because we're building the graph we saw before.


It's a reasonably intensive process.


So it does take a little bit of time.


I think maybe around 40 seconds for this.


So I will fast forward.


Okay, so it's done.


52 seconds.


Not too bad.


Now, what we now want to do is I want to have a look at these again.


So let's see if we have a structure in our graph yet.


So max level.


So that's cool.


That means we have five levels, 0, 1, 2, up to 4.


And then if we do this again, we'll be able to see the sort of distribution of vertices across each one of those levels.


So I'll just write this.


There we go.


So we just ignore this first one here.


That doesn't mean anything.


So this is our layer 0.


Like I said, most vectors get assigned to this layer.


Now we have layer 1 all the way up to layer 4 here or level 4.


Now, FICE will always add at least one vertex to your maximum level because otherwise we don't have an entry point to our graph.


So there's always at least one in there.


And we can also check which vector is being used there.


So we can write index HNESW entry points.


Yes.


And there we have it.


So that's the number of our vector that we have added as our entry point up here.


You can also change that if you want, but I wouldn't probably, especially with this, I definitely wouldn't recommend it.


Now, I was super curious as to how it decides where to insert everything.


So I went through the code, which you can find over here.


So this is the FICE scale repo.


And we have this HNESW CPP file.


And in here we come down and it might take me a little while.


So this is where we initialize our index.


And we have this set default probas.


So if we go into set default probas, we will find this here, which is not really too hard to read.


I don't read, I don't know C++, but this is pretty simple.


And what I translated that to in Python is this here with a few comments because there is no comments in the other file for some reason.


Now, what we're basically doing is just building a probability distribution for each of our layers.


And we do it using this function here.


So we're taking the exponential of the level divided by that ML function or value that I mentioned earlier.


So that's the multi-level, the level multiplier value.


And what we get is a probability for the current level.


Now, what we do is once we reach a particularly low probability, we stop creating new levels.


So if we run this.


Oh, another actually interesting thing is here.


We are saying on every level we will assign M neighbors as our M max.


Except from level zero where we will assign 2M.


OK, so this is M times like 2.


So for our use case with 32, we have M set to 32.


So that means in all of our layers, except from layer zero, we'll have each vertex will have 32 neighbors or friends.


And on layer or level zero, it will have 64 friends.


So I thought that was also interesting.


So we can go ahead and run that.


So the 32 here is our M. We can put it in place of M if we want.


So we run that and we get this probability distribution.


So this is for layer zero, the probability of it being added using a random number generator.


So we're saying if our randomly generated number is less than this value, we will assign or we will insert our vertex at level zero.


And then we continue that logic all the way down to here, which is our level four.


And then here is the total number, so the cumulative number of neighbors per level.


So if we insert level four, we will have this many neighbors for that vertex in total.


If we insert level zero, we have 64. Again, a little bit interesting.


Now, in the code, when we're inserting vertices, we use another function.


This is the last one. I'm not going to go through all of them. We're just going to go through this one.


We insert or we decide where to insert our vertex using this function here, which is the random level function.


This takes our assigned probus from here. So this here and also a randomly generated number.


So here I'm using the NumPy's random number generator.


And we go through and that's how you assign a different level for each of our values.


So if I let's run that and we'll see that we get a very similar distribution to what we get in FICE,


which is, I think, pretty cool to see. So if we do the bin count again, we get this.


Now, if you take a look at this, these values are so close to what we produce up here.


So where is it? So here, look at these values. Almost exactly the same.


So, yeah, it's pretty close. They're both working in the same way.


Now, I also visualise that in this chart here. So you can see, OK, on the right,


we have our Python version, on the left, we have our FICE version. And I've included the numbers in there.


Basically, it's pretty much exactly the same. It's off by a few here and there, but not much.


So I thought that was pretty cool to see. Now, OK, that's cool. That's how everything works.


That's interesting. But what I think many of us are probably most interested in is


how do I build the best HNS sub U index for my use case, for what I want?


So I ran some codes to go through all of that. And pretty much I did this.


So we have, this is also a good thing for me to show you, is how we get out EF construction and EF search values.


So I think a good starting point for both of these values is probably 1632 is reasonable.


But you can play around with them. I think that's usually the best way.


And for most of what I'm going to show you, I was using this. So I was performing a thousand queries


and calculating the recall at one value for those. Also, I was checking how long our search is taking


and the memory usage of each of these indexes. So the first thing I want to tell you is a recall, of course.


So in what we have here, on the left we have with a EF constriction value of two, which is very low.


I don't know, never use that. Never use an EF constriction value that low.


Well, you can see the recall is pretty terrible. But, you know, surprisingly, even though this is a very bad EF constriction value,


it's still got up to 40% recall. So I was impressed with that. That was cool.


And so the EF search values are what you can see on the bottom. So it went from, I think, two all the way up to 256.


256. Yeah, that's right. And we did that again for the EF constriction value of 16 here and also 64.


I did it for more EF constriction values, but these are probably the ones I think showed the relationship best.


And then all the colors you can see here, we have a few different colors. We're using different M values.


So what I found from this is that you basically you want to increase your M and your EF search values to improve your recall.


And EF constriction also helps with that. So higher EF constriction value means that we need lower EF search and M values to achieve the same recall.


Now, of course, we can just increase all of these and we get better recall, but that isn't for free.


We have to sacrifice some of our search time as well. As always.


So we, well, this is what we've got. So same graphs as last time. This time we have search time on the Y axis.


Now, EF constriction here. It does actually make a bit of a difference when we start using higher M values.


So I found before in previous videos and articles that EF constriction didn't really have too much of an effect on the search time,


which seemed fair enough because it is mostly an indexed constriction parameter.


But that it does seem to have some effect when you start adding or increasing the number of queries that you are searching for.


And in particular, where you have high M values. Now, one thing also to note here is that we're using a log scale on the Y axis.


So the high values are definitely a fair bit higher. But again, I mean, if we go for a sort of reasonable EF search value,


all of these are pretty high. I mean, 256 for search values is high. If we go for sort of EF search of 32, let's say around here.


We're still getting reasonable search time. So this is on the millisecond, well it's a bit more on the millisecond scale.


And if we take a look at that on Recall, we're sort of getting up into the 60 plus area at that point, 6 plus.


And look, obviously it's not bad. It's reasonably good. But you can definitely get better than that when you start merging different indexes,


which we have covered all of that in another video, another article. So you'll be able to find that in the, either at the bottom of the article,


if you read the article, or in the video description if you're watching this on YouTube.


Now, having a look at the EF construction values again, no effect on the search time.


We find that EF construction, when we use lower end values, like I mentioned, it doesn't have too much of an impact on the search time.


So if you are using a lower end value, I think EF construction is a good value to increase because it doesn't tend to push up your search time too much.


And you can improve your Recall by increasing it. So it's definitely one to try and increase them and see how it goes.


For me, I found it's been quite useful at least.


And then finally, the sort of negative point of H and W is the memory usage is just through the roof, as you can see here.


So using the lowest end value, which I think was two, which is you would never ever use that.


We were getting memory usage for the SIFT1M dataset of more than half a gigabyte.


If you push M up a lot, I mean, 512 is maybe a little bit too much, but we're sort of pushing four and a half gigabytes, five gigabytes.


So it's something to be aware of, I think.


But like I said before, we can still improve the memory usage and the search speed by simply using different indexes alongside our H and W index.


So if memory usage is a concern, you just compress the vectors using product quantization.


If the speed, if you need it to be even faster, you can use the IVF index on top of H and W to improve the search times.


And if you want both of those, you just put them all together and you can get that as well.


Now, like I said, we have covered all of that in a lot more detail in another article and video.


But for this video on H and W, I think we're going to leave it there.


Text Summarization with Google AI's T5 in Python

Hi and welcome to this video on text summarization.


We're going to go ahead and build a really simple easy to use text


summarizer using Google AI's T5 model. So this is insanely easy to do.


All together we actually only need seven lines of code and with that we can


actually summarize tech using Google's T5 model which is


actually the cutting edge in terms of text summarization at the moment.


So it's really impressive that we can do this so easily


and we'll just run through really quickly and


see what we can do. So we need to import Torch and the Transformers library.


And from the Transformers library we just need the auto tokenizer


and auto model with LMHead.


Whilst they're importing we can initialize our tokenizer and model.


So all we do for this is tokenizer and then we load our tokenizer from


pretrained.


And we will be using the T5 base model.


And then we do the same for our model except with the auto model with LMHead


class.


We also need to make sure that we return a dictionary here as well.


Okay for this we're going to take some text from the


Wikipedia page about Winston Churchill.


And we will just take this text here.


I've already formatted it over here so I'm just going to take


this and paste it in. But this is exactly the same


as what I highlighted just here without the


numbers and headers. So we run that and we simply build our input ids.


So all we're doing here is taking each of these words


splitting them into tokens. So imagine if we


split one sentence into a list of words we would have


this his first speech as Prime Minister. Each one of these words would be a


separate token. So we split them into those tokens and


then we convert those tokens into unique identifier numbers. Each of


these identifying numbers will be used by the


model to map that word which is now a number to a


vector that has been trained and represents that word.


We just add summarize at the front here


followed by our sequence.


Because we are using PyTorch we want to return pt


tensors.


And we set a max length of 512 tokens which is the maximum number of


tokens that t5 can handle at once. Anything longer than this we would like


to truncate.


So now we can have a look at those inputs and we can see we have


our tensor of input ids.


Now we need to run these input ids through our


model. So we do model generate and this will output a this will


generate a certain number of output tokens


which are also numeric representations of the


words. Now all we need to do here is pass our inputs.


And then we give a max length and minimum length as well.


Let's just tell as the model we do not want anything longer than


we're going to use 150 characters


and anything less than 80 words.


Now we have a length penalty parameter here as well.


So the higher the number the more the model would be penalized for


going either below or above that min and maximum length. We're going to use


quite a high value here of five.


And we use two beams.


Now what we also need to do here is actually pass these into another


variable outputs and then when we want to access these


outputs we will use outputs


zero as this is the tensor containing our numeric word ids. Now we can use


tokenizer again to decode our outputs. So this is converting


our outputs from the numeric ids into text.


And we also want to give that to another variable.


Finally we can print our summary.


And here we can see that the model has taken


some of the information


I think entirely from this second paragraph here


and created a summary of the full text. Out of the boxes it's pretty good


because if you read through this it includes a lot of the main points.


Now the first paragraph isn't that relevant and I would say the


final paragraph is not either. Most of the information that we want


from my point of view is in the second and third paragraph.


Now the model has quite clearly only extracted information from the second


paragraph which is not ideal but for an out of


the box solution it's still performed pretty well. So


that's it for this model. I hope this has been pretty useful and insightful as to


how quickly we can actually build a pretty good text summarizer and


implement Google's T5 model in almost no lines of code.


How to Build an AI-Powered Video Search App

Today I want to have a look at how we can build a NLP powered intelligent search for video.


And more specifically we're going to have a look at how we can do that for YouTube.


Now YouTube is obviously huge, but it started with a very simple video titled I'm at the Zoo.


This video was a 19 second clip of YouTube's co-founder literally at the zoo showing you some elephants in the enclosure behind him.


And it seems really silly now, but when you think about this, that was back in 2005.


And up until that point the only video or insights into other people's lives that we would have seen is that of celebrities or politicians.


And all of it was very orchestrated and not natural.


So YouTube sort of brought this very unique age where we're able to actually see into other people's lives and normal people's lives like me or you.


And see just what they are doing.


And okay, I'm at the Zoo is a kind of odd, very short video, but it really marks the beginning of this almost explosion of normal people made content.


Which I think is really cool.


And now what YouTube brings to us is a lot more than just an insight into someone's life.


It can really range from anything.


You're watching this video right now to learn about how to build a search tool for video using natural language processing.


And without YouTube, sure there'd probably be something else, but YouTube was really the place where this all started.


So building a search tool that allows you to search through some of the most engaging content on the planet, I think is really cool.


And of course we can just use normal Google search or YouTube's actual search bar.


But as we'll see, we can actually build something that maybe does quite well in comparison to those tools.


So before we really jump into the technology and how to build it, let's have a quick look at a few examples of what we are going to actually be able to build after following through this video.


So this is just a really simple interface built using Streamlit.


And we can just say, OK, what is deep learning?


And OK, we're going to have here a few videos. I think we have five.


And we can just kind of click on one of these.


So I mean, all of these seem pretty relevant, but I just want to show you something really cool.


If we click on one of these, it will take us straight to the point of the video.


So here it's sorted at 329 and it's going to show us what is deep learning.


OK, so that's really cool. Like straight away, it's almost like a very intelligent and fast search tool for YouTube.


And YouTube is full of amazing content. So this is, I think, really cool.


I was playing around with this quite a lot. If we're interested in something else.


So what is a transformer? What is a transformer model?


What is a BERT transform model?


OK, so BERT, this looks pretty promising. Transformers here looks pretty good.


And then if we have a look at this one, also the first result, this is the most relevant.


And this explains BERT pretty well as well. So that's really cool.


And what I think is coolest about this is we just use off the shelf models and put this together super quickly.


It is not complicated, as you will see, to put all this together.


You can probably do it within the next 20 minutes after following this video.


So we're going to have a look at this data set here.


So if we have a look in here, we have 116,000 text files.


And we also have the audio files as well. So we're only going to use the text files.


And these are basically for every small segment of video.


We have the subtitles from that segment. And that's what we're going to be performing our search on.


Now, let's have a look at how we can download this data from Kaggle and also what the data actually looks like.


So to start, you will need an account. So once you have created an account on Kaggle, you go to the right here.


You go to, I think, Account. We have this nice picture of me with hair.


And we can scroll down and there's this little API section here. And we can create a new API token.


Once you've done that, that's going to download a kaggle.json file to your computer.


And we use that kaggle.json file to authenticate the Kaggle Python client.


So the Kaggle Python client will just pip install kaggle. And then when you run this import kaggle the first time,


if you don't have that kaggle.json downloaded onto your system, this will come up with an error.


And it will tell you where to place your kaggle.json. So you just, whatever directory it tells you to put it in,


put your kaggle.json in there, rerun import kaggle, and it should work.


And then to use this, we just run this. So this is in the command line. You can also do this in a terminal window.


Kaggle datasets download. And this is that dataset I just showed you. We will need to unzip that.


And once we unzip that, you will find everything in this data directory here.


Now you can see in this data directory, there are a lot of files. So all of these represent a video ID.


So if we go into here, we have all these timestamps. And this is just a range of a timestamp.


And in there we have the audio that represents the audio from that range. And we also have the subtitles from that range.


So if we look over here, so within the range from three seconds in to about six seconds in,


the word or the words spoken in the video are machine learning. It's a buzzword, but I, and then we're on to the next timestamp,


which would be here. We have look. But I would also claim it's a lot more than, and then I assume they're going to talk about how machine learning is more than a buzzword.


That's fine. So yeah, that's what we have there. But that's very little. You saw in that little app that we had thumbnails, we had the video title,


which we don't have in that data set. And we even, I think we had the video description as well. OK, so you can see all that here.


We didn't have any of this in that in that data set. So how do we do that? How do we scrape that data? To do that, we need a few things.


So we need Beautiful Soup, TQDM, you don't necessarily need it, but it's best we do. And data sets, we use data sets here because rather than having all this data


stored locally, we can actually save it to the Hug & Face hub. And if you do, if you're not really interested, you just want to build this up really quickly,


you can download that data and I will show you how you can do that in what is probably the next chapter of this video.


So you can hover over the timeline of this video and you'll be able to see the next chapter. And at the start of that, you'll be able to see how we can load that data set directly from Hug & Face


rather than going through all this preprocessing. I want to show you because this is how you like a big part of machine learning and building these things is the preprocessing.


So it would be a shame to miss it. So we can see what I showed you before. We have video IDs, which are just a list of the directories in each one of those.


We have the timestamps and we can load them. So we have in data, video ID, splits, and then we have the subtitles.txt file.


Okay, cool. And what I want to do is just loop through all these files to give us that, you know, give us what we can get from those files.


So that is the video ID from the directory names, the text from subtitles.txt, the start second, end second, which we can get from the directory timestamps,


and also the URL because the URL is actually we can pull that from the video ID. Okay. So I won't go through this in too much depth.


But what we're doing is we're going through each one of those splits, extracting that small chunk of text, it's a very small chunk of text.


If you remember, it's just a few seconds long. So with Q&A, we really want to have longer chunks of text than like five words.


So what I've done here is said, okay, once we reach about three or four sentences, we are going to save that as a chunk. Okay.


There are better ways to do this. It's like a very good approach is to have overlapping chunks so that you're not missing anything at all there


because we might be cutting things off right in the middle of a sentence. It's probably not the best idea.


But just for this demo, it's good enough. It's not a problem. So we create our start and end seconds using the timestamps that we have in that directory name.


Okay. So that's literally just the number of seconds into the video that we are at.


And then we create a document, which is just all of the details for that particular chunk of text, including the video that comes from the text itself,


the sign end seconds of that chunk of text, and also this, which is the URL. So the URL directs us to the video, the specific video,


and also the start of that chunk of text that we're going to be returning later on.


And yeah, so we create a list of documents like that. It shouldn't take long.


So this is one example. And you can see, okay, like here, we kind of just cut off in the middle of a sentence.


It's not perfect as well, but that's fine. It works well enough.


So it would be better to have some sort of window where we overlap. So we, for example, took this paragraph, and then we maybe took half the first,


the second half of the paragraph, followed by the next half of the next document. But this is good enough.


You see a few of those there, and we have the start and end seconds URL, which you can see we have the 41 seconds, 41.


Okay. Now, as I said before, there's that other metadata that we don't have in here.


Now, you might need to, if you're on Mac, you might need to PIP or Conda install LXML, and that is for Beautiful Soup.


Beautiful Soup is a like a data scraping library or HML processing library, almost.


So it's really good when we're scraping information from websites, which we can do.


So we can, okay, we're going to go to each video and we're going to capture the data, thumbnail and any other information we can from there.


So in this case, we just have the title and the thumbnail.


And we saw those within this metadata dictionary.


Okay, so we have title, thumbnail. If there's an error, so there were a couple of exceptions here, rather than just throwing an error


and not returning anything or stopping the process, just returning an empty title and thumbnail.


There's two out of the 127 that we scraped there. So I think we might see it's too much of an issue.


Okay, so now we have the document, which is what we originally pulled, and then we have the title and thumbnail.


So what we can do here is pull those together and there we have our full document.


Okay, so when we are saving things to the Hug & Face Hub, we can just save it as a JSON lines file.


Okay, so it's like this list of dictionaries. We save that to a JSON L file and then you can actually just upload that directly to Hug & Face.


So that's pretty straightforward.


Okay, so as promised, this is how you can download the data that we just created.


So super easy. It's exactly the same as what you saw before. We have video ID, text, start and end, URL, title and thumbnail.


And we have 11,000 items there. Okay, and so we have 11,000 sort of documents and that is spread across 127 videos or unique videos, as far as I can remember.


And we can just see one of those and you can see we've already seen this example from before. So it's exactly the same, but obviously a lot easier for us to actually use because we're just pulling it from Hug & Face Hub, which I think is really cool.


Now, the next thing we need to do is actually index all those documents within Effective Database. Of course, we're using Pinecone here.


So first thing we need to do to begin doing that is initialize the Sentence Transformer. So the Sentence Transformer is a model that is going to take the text and convert it into a vector, which we can then place inside our vector database and use that to perform our sort of intelligent semantic search through all of those documents.


So we have the match sequence length, 128 here. So I use this to come up with this three to four sentence length of our paragraphs because typically a token, which is what this is, 128 tokens, is typically going to be something like three to five characters.


So we'll go with three characters here. And so that's why we got the 360 from. And we have this 768 sentence embedding dimensionality. So that's important. So pull that in here to the embedding dim variable.


And we're going to use that when we are initializing our index. So we need to get an API key for this. So you go to app.pinecone.io, create an API key, and then you can just go here and take that.


So if you're just looking at where to actually get your API key. OK, so you go within your project. It's probably going to be called someone's default project. You click in there, you go to API keys, and then you have this. You just copy this and then you would paste it inside this.


Pretty simple. And then we can create our index, whatever you can call it, whatever you want. I call it YouTube search because that's what this is for me. Again, you can call it whatever you want.


For this model, we need to use cosine similarity and we need to align the model embedding dimension and the dimensionality of our index. OK, so that's what I've done there.


And then we connect to our new index using the name that we gave it here. So what I'm going to do is index our data in batches of 64. The data we insert into our index will include the document ID, the embedded vector that we create using the sentence transform model, and any metadata we'd like to include.


So that's what we saw before. The title, start seconds, the text itself. We're going to include all of that in there. So to do that, we create this loop. This is where I'm using TQDM. This is just a progress bar so that we can see how long this is going to take.


It shouldn't take too long, though, by the way. And what we do so in batches of 64, I'm going to encode all of the text. I'm going to create the IDs. So we have two sets of... so we have a video ID, but video ID is not unique for every single snippet.


So what I'm doing here is taking video ID and a start second because that is unique and placing those together. OK. And then what we're doing is creating the metadata. So I just want these items from the metadata from before.


So I'm just pulling those in. There's nothing, nothing particularly unique there. OK. And then I'm just upsetting all those. So inserting everything into Pinecone, the IDs, the embeddings and the metadata. And yeah, just adding all that to Pinecone. Super easy.


And then we describe our index size. You don't need to do this, but this is so we can see what is in there. So we have the dimensionality, how full our index is. We don't have much in there at the moment, so it's not very full.


We can usually fit about a million embeddings into a typical pod, which is like a hardware unit of the vector database. You can have more pods if you need more than that. And yeah, so this is our vector count.


We have just over 11,000 items in there. And then we go, this is what we're doing before, what is deep learning. And then we return those results. So this is just a text. And yeah, everything is pretty relevant there.


So that is how it works. It's pretty, I think, simple. The only other thing to do is actually put all that into a streamer app, which is not hard at all. OK, so this is our code. I'm going to zoom out a little bit. I'm sorry if this is kind of small, but it's hard to get that all squeezed into the screen.


So yeah, I mean, the same as before, we're just initializing things. We have this little card, which is like each component or result that we return. Just HTML, really basic HTML, so nothing complicated there.


And then, yeah, we're just using Streamlit components. Streamlit right, Markdown, there's nothing really in there. We have the search bar. Whenever the search bar has something inside it, we search and just return that information in the format of a card.


And all that together is how we build this. So it's really, I think, super straightforward and really easy to use. Yeah, that's it for this video. I hope this has been useful and at least insightful as to how you can build something like this.


Of course, this is just one example, like YouTube search, video search. You can search through anything that you can imagine as long as you have some sort of data that represents it and some way to represent that sort of data as a question and answer format.


You can build something like this super easily.


IndexLSH for Fast Similarity Search in Faiss

Hi and welcome to the video.


We're going to be covering how we can implement LSH in FICE.


Now, we covered LSH and how it works using random projection


in a previous video, which I'm going to release at the same time as this video.


So I'll link to that in the description if you want to understand how LSH works.


In this video, we're just going to cover how we implement it in FICE.


So we're going to be using the SIFT 1M dataset,


which you can download using, I'll put a link to the script in the description,


but it's just a big dataset of dense vectors.


Now, from this, we'll get our 1 million samples, xB,


and we also have our queries, which I'm going to just,


we're just going to use one query at a time.


Okay, so we'll have one query, 1 million samples,


and we perform our search using that.


So what we first want to do is import FICE.


I'm going to get my dimensionality, which is xB.shape,


one, I think, yep, 128.


So that's our dimensionality of each vector within the SIFT 1M dataset.


And initially, let's use an n bits value of four,


which gives us, if we think about it, how many buckets does that give us?


We can do this, right?


So it gives us a total of 16 buckets.


Okay, these are also potential buckets.


Our vectors aren't necessarily going to be separated between all of them.


They might, we might end up only using like 10 of those buckets, for example.


Okay, and then what we want to do is initialize our index.


So FICE index LSH, and in here we need to pass our dimensionality


and also our n bits value.


So how many binary values we're going to include within each bucketed item.


So do we, first thing, it's always good to check,


do we need to train the index?


Index is trained as true, so that means we don't need to train it.


There's no optimizations or anything we need to do here.


So that's good. Don't need to worry about that.


And then, so without needing training, all we do is index add and we add our data.


Okay, and now we're actually just ready to go ahead and search.


So we do DI equals index dot search, execute, and let's return,


let's set k up here actually.


So let's set k equal to 10.


Okay, so we do k and then we get the,


these are the, apparently the most similar indexes to our query.


Okay, and there's one good thing to check here,


which makes it really easy to figure out if we have enough buckets or not,


is d, which is the distance between those vectors.


Now, obviously if we have all of our vectors in a single bucket


and we search and our query ends up in the same bucket,


all of the distances here are going to be zero


because they're all going to be exactly the same Hamming distance,


which is what we're returning.


So let's look and that's what we get.


I mean, we have very few numbers here, so that's not that surprising.


Let's push k up to 10k.


You'd think at this point, surely we'd get some ones, but no.


So at least the first 10,000 nearest neighbors are all within one bucket.


So it clearly, there's an issue here,


we can't differentiate between all of those.


But if we just increase this to a very high number,


let's say like 500,000.


So this is returning half of the index.


At this point, we should, I imagine, see a few ones.


So at some point within the first 500,000 items,


we do actually, we move from a single bucket to at least one other bucket.


Now, if we think about it,


we can actually estimate roughly how,


what n bits value we need in order to kind of get a better distribution


between our buckets.


So to not have everything in a single bucket.


So let's try these.


We have for n bits in these values.


Let's give it a go, see what we get.


And we see, okay, n bits equals two.


So this gives us an average of 250,000 items within each bucket.


So obviously that's not useful.


Here we get 62k, again, not useful, 4k, 15k, that's a lot better, right?


But it's still not useful because we kind of want,


if we set k equal to 10,


this isn't really going to give us anything particularly useful.


So, and we also need to consider the fact that we,


in our buckets, not all of them are going to get used


and what we will find is that it's almost like there will be a majority of,


or there will be a minority of buckets that end up with the majority of vectors.


So we need to go sub one for this to work.


So I would say something probably n bits of 24,


we're getting that, I mean, that value actually looks pretty good, 0.05.


So on average, we should get 0.05 items within each bucket.


Okay, so I mean, let's test that.


So we're going to copy this and we'll just write a similar code


to what we wrote before.


Although one thing before we do that,


let's just check the average similarity.


Let's change k to like 100 or 10.


Let's go with 10.


Yeah, let's do 10.


And what I want to do is, given that,


I want to calculate the cosine similarity, average cosine similarity.


So we'll do that, we use from sklearn metrics pairwise,


import cosine similarity, that's just super quick way for us to calculate it


and then check sklearn.


And we'll do cosine similarity.


And we have I0, xq.


I'm so sorry, we need to get the indexes.


So I just contains the indexes, we need to pull them out of xb.


Okay, and we have this.


So we also need to take the mean of that.


So cos mean, we get 0.44,


which is just the average of those, the average similarity of those.


So it's pretty low.


Let's compare that to if we did that for the whole data set.


So like this, see what we get.


Okay, so actually these are, yeah, I mean, these are slightly more similar.


So we even, you know, get slightly better results,


even with that terrible sort of booketing.


And so what I want to do is repeat that process we just did there,


but I want to do it for different n bits values.


So we need to take our index, bring it down here,


and we're going to say this, we do index.add our data,


and then we do di equals index.search as we usually do.


k is 10, and then we want to do cos equals cos and similarity again.


We're going to do xbi.


So we're pulling out those indexes that we've got.


We're doing this different n bits values, hence loop and execute.


Okay, and then what we're going to do is just print n bits is equal to


whatever it's equal to at that point.


And then new line, I'm going to put, well, the similarity.


So similarity is equal to cos.mean, like that.


Let's just see what we get.


So, okay, cool.


So we look here, and so n bits of 2 is terrible, gets better, gets better.


And then at 24, so that bit where we had the 0.05,


we get the maximum value.


So we get a 7.3 here, 7.4, which is the best we get.


And then it kind of comes off a little bit here.


And I've kind of visualized that here.


So this is the same structure.


I've gone a little bit high with n bits as well.


And this is essentially the sort of structure we get.


So the bit that we just kind of saw was this section here,


where we're increasing n bits and reducing the number of, or reducing,


like heavily reducing the number of vectors within each bucket on average.


And the cosine similarity just shoots up really quick.


But then afterwards, we still get this kind of slow increase.


And that's because, so the buckets are now quite spread,


like the vectors are spread very nicely between different buckets.


But as we increase the resolution even further and further,


all it means is that we're adding more of the original information,


the original positional information of those vectors,


and essentially improving the resolution of those vectors.


So they're just getting slightly more accurate in whether the hashed buckets


are closer to other very similar or not similar vectors.


Essentially, the binary vectors are more heavily representing


the original dense vectors.


And that's why we see that slight increase that continues


even after we get to n bits of 32.


Now, one final thing I just wanted to quickly cover is,


say we do have issues and we want to sort of visualize the actual buckets themselves.


How can we extract those buckets and view the number of vectors that we have in each one?


Now, there isn't any way of doing this in FICE.


So we kind of just have to put our own code together.


So what we need to do is, I'm going to create, first create another index,


because I want to visualize these.


So I'm going to do, and if I visualize it with n bits of 32,


there's going to be, well, how many buckets would we have there?


It would be 2 to the power of 32.


So, yeah, we can't visualize that many buckets.


So we're going to go with 4 and n bits of 4, which gives us 16 buckets.


So index LSH, we have D and we're going to use a n bits value of 4.


n bits equals 4, just put that there.


Okay.


We're going to add our vectors.


And now what we want to do is actually extract those binary vectors themselves.


So to do that, we write array.


So we're going to store them in the array variable.


And we do FICE.VectorToArray, like this.


And in here we do index.Codes.


So codes, just that word there is essentially,


that means, you know, the binary vectors that we've built, they are called codes.


That's all it means, but we use this word a lot in similarity search.


So it's worth remembering.


Also n bits is used a lot as well.


So in future videos, we will, in this series, you will see them quite a lot.


So then let's have a look at what we have.


We have, okay, so we wanted to pull out the binary codes


and we're getting these numbers,


which is not exactly what we expect or is it, right?


Well, okay, let's have a look at the min value here.


Zero.


Okay. Array. Max.


Okay. So we get zero to 15, which means we get 16 values.


And okay, let's do n to the power of n bits.


Okay.


So that means we have 16 unique buckets that all of our vectors are spread between.


So that's interesting.


Let's see how many vectors we have.


We have one million.


So we have one million spread between 16 buckets.


So actually that sounds about right.


But how do we, so these values here,


these are just the integer versions of our binary vectors.


So, well, like this is represented by zero


and this is represented by one, right?


So what we can do is use this little script here


to convert them into the actual binary vectors.


So here, this value is our first one, which is a five.


And the next one is a 12, which you see here.


So, I mean, that's, that is our, they're our binary vectors.


And we can visualize how they are spread,


how many items are within each one of those buckets using this array.


And what we'll get is something that looks like this.


So this is what I mean where we have, so we have 16 buckets,


which you can see on the left here.


So all of these, we've 16 of those,


and not all of them even have any vectors in.


So we have one, two, three, four that don't even contain any vectors.


And then the majority, the vast majority are contained within these and these.


Right, so these four buckets contain almost everything.


And they have loads, they have, so the count is to the thousands there.


So here, that number, add a few more zeros onto the end.


So, yeah, these four buckets have far too many,


far too many vectors in for it to be useful.


So that's why we, before we increase it, increase our InBits value up to 24.


Now, I think, I mean, that's basically everything I wanted to cover,


but just quickly in terms of performance.


So visualize these using, this is using the FICE LSH index.


So we have InBits, so obviously we just increase InBits loads.


We increase the recall performance.


So that's one thing about LSH is that to get good performance,


you really have to increase the InBits value a lot.


And at that point, it actually gets, it can get really inefficient.


So this is the time for each query as compared to a flat index.


So obviously, I mean, you kind of want to be careful with a LSH index.


Obviously it can be useful, but, you know,


high dimensionality is difficult to balance well with speed and accuracy.


But obviously, you know, it can be done.


It just depends on how much accuracy you're willing to sacrifice.


Because in most cases with LSH,


you will be sacrificing a reasonable amount of accuracy.


So the recall here is about 50%.


So that's, it's managing to identify 50% around here


of the same items that a flat index would identify.


And it's only slightly faster.


So yeah, you just have to weigh, you know,


what is most important for your use case with that.


But anyway, that's it for LSH.


This is the last video we'll cover for LSH for now.


In the next few videos and articles,


we're going to be covering a few different indexes


and how we use them effectively.


So it should be pretty interesting,


but we're definitely getting a bit more advanced than LSH now.


So that should be good.


But that's it for this video.


Fine-tune High Performance Sentence Transformers (with Multiple Negatives Ranking)

Hi, welcome to the video. We're going to be having a look today at something called a multiple


negatives ranking loss or MNL loss for training sentence transformers. Now if you're new to


sentence transformers, they're essentially NLP models using transformers that have been trained


specifically for building dense vector representations of text. And the highest performing


sentence transformers at the moment are all trained using this method. So they're trained


on a natural language inference data set and they are trained with MNL loss. So in this video we


are going to learn about MNL loss and how it works. We're going to work through a PyTorch


example very quickly and then we're going to look at how we can implement it ourselves using the


sentence transformers library. So let's jump straight into it. So if you saw our previous


article and video, you remember that we had a NLI data set which is built from the Stanford NLI


data set and the multi-genre NLI data set. We're using the same data set here. So in essence all it


is is a load of sentence pairs and there is a label 0, 1 or 2 which indicates whether those sentence


pairs either infer each other, are not really related like they could both be true but not


necessarily because of each other or if they contradict each other. So they're the three labels


and what we covered in the previous video and article was something called softmax loss. With


softmax loss we use those labels 0, 1 or 2 to produce a classification. We optimize on that


label. Now with MNL loss we don't actually use those labels. We just use the sentence pairs and


what I'm going to show you is where we use something called a Siamese network which means


we only have an anchor and a positive sentence. Now what that means is an anchor is a you can


think of it as like a base sentence and a positive to that anchor sentence is just a sentence that


indicates that the anchor is true. Then we could also have negatives and a negative would indicate


that that anchor is not true. We can extract that from the NLI data set. So let's start


pre-processing our data and have a look at what we need to do. Okay so this code here we already


wrote it in previous video article but we're going to go through it. So if you've never seen


any of this before it's fine. We're going to go through it. I'm going to explain everything. It's


not a problem. So basically up here all I'm saying is what I just told you. So these are our labels


in our data. We have this zero entailment, one neutral and two contradiction and then I'm just


saying okay M and R we don't actually need those labels. All we need are anchor positive pairs e.g.


where pair a pair where the premise suggests the hypothesis so where the positive


indicates the the anchor or the other way around.


So essentially what we need are just rows where we have the label zero. So we're going to do that


but first we need to actually get our data. So we're using the Huginface data set library here


which is very good and we're getting the Stanford Natural Language Influence data set here. Now


down here this format you can see is the format of the data set. So we have these three features


premise, hypothesis and label. And if we come down here we can see what one of those looks like. So


we have premise, a person on a horse jumped over a broken down airplane and this hypothesis a


person is training his horse for a competition and the label for that is one. Come up here one


means neutral so basically this here a person's training his horse for competition does not


necessarily mean a person is on a horse is jumping over a broken down airplane. So this


is a person jumping over a broken down airplane. Now we come down here and I think this one is


this one's entailment so this one is a pair that we want. This is an anchor positive pair


and we have a person's outdoors on a horse and a person on a horse jumps over a broken down


airplane. If they're jumping over a broken down airplane they're probably outside or in fact they


are almost definitely outside. So this indicates it so this is an entailment means it's an anchor


positive pair. So I said before we have two data sets not just one so we have the Stanford


Natural Language Infants Data Set and we also have the Multi-Genre Natural Language Infants


Data Set and that's what we're getting here MLI. So I'm just loading that with the the load data


set here. We're learning it from the GLUE data set. Within GLUE there is this MLI which is the


data that we want and we're just getting the training data from that. No we don't because I


think there's also maybe there's validation data and there's also test data and we don't want that


we just want the training data. So we can see the format for this data set almost exactly the same.


Now we just have this extra id. So what we can do is we just remove those columns so we


MLI.removeColumns and we specify that we don't want the id column because we're going to merge


these two data sets and for us to merge them they both need to have the exact same format.


So after that what we do is we perform this CAS function. Now if we didn't perform this CAS


function let me show you. So I need to run these anyway so start running them now. Come down here


I'm going to load data set removed columns and okay let's say I'm not going to do this CAS.


I'm just going to concatenate the data sets. I'm going to merge those two data sets together.


I do that and we get this arrow invalid error and the reason for that is because although those two


data sets look very similar they're not actually exactly the same. The data sets used in one of them


includes this not null specification whereas the other one does not include that not null specification.


So I assume that means that this other data set can include null specification.


So we can't merge them both because they're not the same data type. So what we do we come up here


and we have to do this CAS. So we're casting the features of the SMI data set to be the same as the


MNI data set features. So we do that run it now come down here we can run it again and it will work.


And now in data set we have the full data sets that's both MNI and SNNI.


You can see that because we have 943,000 rows. If you come up here we only have 392


in the MNI data set and up here we have 550 in the SNLI. So we can run that data set.


So we have all the data and now in the in the previous video an article I mentioned we have these


negative one labels. Now this is an error in the data or it's not an error but it's where


whoever labeled the data they couldn't decide whether this was they couldn't decide on the nature


of the relationship between the pair of sentences. So they just put minus one there's not very few


I think it's 700 or so sentences in there or pairs in there that are labeled with this


minus one but we that's not that's not a label we can't do anything with that in our data. Now


what we do is we use this filter method to remove so we have say false for the row so the


row is false if the label is equal to minus one which is saying that row is false e.g. we do not


keep it. So we can then use the filter method to remove the data. So we can then use the filter


method if its label value is equal to minus one otherwise we do keep it. Now the things are a bit


different now because we actually only want anchor positive pairs which means we want to remove


or we only want to keep the rows which have a zero label we want to remove everything else.


So we need to modify this to just keep the zero values so we'll say okay false if x label is not


equal to minus and to zero else it's true. So this is going to keep only zero values and remove


everything else so that removes those erroneous rows and also removes the neutral and constant


and contradiction rows as well. Now we remove that and let that run. Now while that's running let me


show you some some visuals of how this will work. Okay so what we have here is we have that


anchor positive pair from before so here our anchor would be I think from eyes and the positive would


be our hypothesis. But it would obviously only be rows where the label is equal to zero which is the


entailment label. Now we have our our anchor and positive and as we usually would with a transform


model we tokenize them. So we tokenize them we're going to do that with the just a tokenizer method


a pre-trained tokenizer from the base transformers library or we do that if we're using PyTorch.


If we're using the sentence transformers library it's a lot easier and we don't actually need to


do that it will will deal with that for us which is it's quite nice. And what that produces is just


a tokenized version of the anchor and the positive. So here we have our so I'm just going to put a for


anchor and over here p for positive. And then what happens next is we have a single BERT model


but we actually kind of visualize it as two BERT models because we're processing the anchor and


then after we've processed the anchor data we move on to processing the positive data. So it's like


we're using for every single training step we're using the same BERT model twice. So we process


both of those through our BERT model and that produces token embeddings. So token embeddings are


512 dense vectors which contains 768 dimensions. And then we use something called a mean pooling


function. So mean pooling function is say we have I don't know we have some vectors here


one two three. What we're going to do is is take the mean value across each dimension. So


let's say we have three dimensions here in our known as a bad idea because we have three vectors.


Let's say we have five dimensions here and we take the average across each of those dimensions


and what we produce from that mean pooling operation is a single five dimensional vector.


So we have like the mean of each of those across each of those dimensions. That's the mean pooling


operation and obviously from that we produce what is our sentence embedding which is a single


one dimensional dense vector. And we produce that sentence embedding both for a our anchor


and for p our positive. And what we have here so I don't know if I mentioned it but we


so this is a Siamese network this double network and this triple network is a triplet


network and it works in the exact same way but we also have a negative. So where before we had that


the contradiction label I think I think the label is two. I could be wrong I think it no is I think


it is two. That would be a negative sentence because it contradicts the anchor. And what we do


is we process that as well and then we also get a negative sentence embedding at the end.


And what we do with that during training so the whole MNR loss thing multiple negative ranking


thing is we take all of those vectors that we produce the a which is the anchors the p and the


n for the positive negatives if we're using negatives if not we just basically you know


blank out the dark blue part of this visual. All we do is we calculate the cosine similarity


between our anchor and our positive and maybe our neutral negative as well but I'm just going to say


I'm going to go through the anchor and positive version of it for now. So we calculate the cosine


similarity between the anchor and positive and we do that for every anchor so anchor zero for the


first one we would calculate the cosine similarity between anchor zero and positive zero one two


two three and four and so on up until the batch size. So what we have there is the actual so the


number of sentence embeddings that we have there is equal to the batch size that we are using.


And obviously what we would expect with our anchor and positive pair is that we would expect


the cosine similarity between a zero and p zero to be greater than the cosine similarity between


let's say a zero and p one or p two or p three or p four. And likewise if we had a three we would


expect the cosine similarity between a three and p three to be greater than that between a three


and p zero or p one or p two or p four. So that's how we optimize we say okay the target label


for a zero is just going to be to have the greatest the greatest arg max value with the


value of p zero with the pair p zero and for a three it's going to be with p three. So the labels


for this are actually just zero one two three four up until the batch size which we will see


I'm going to show you that. Okay so here we have what is our sort of involved MNR loss training


notebook. So I'm just going to take you through this we're just using PyTorch rather than the


sentence transformers library here. Now I'm going to go through and we're going to come to


where we actually start training. So we have a mean pooling operation here so I mentioned before


with that mean pooling function where we're getting the average across dimensions that's what


this is dealing with. The reason it looks more complicated than just taking the average is


because we need to consider where we have padding values where the mask value is zero because we


don't want to consider those in our average function because then it's going to obviously


bring down our average a lot just for having more padding tokens which doesn't make sense obviously.


So that's why it looks more complicated than you probably expect from for an averaging function.


And then so we're using PyTorch here so we're moving some set model it's just a BERT model


that we're using here plain you know straight BERT model nothing nothing special about it


I think it's BERT based on case and just moving it to a CUDA GPU if we have one right so it is


available checking if it's available. And then here we're defining some layers that we're going


to be using in MNR loss so we have the cosine similarity I said before we're doing that we're


checking and calculating similarity between pairs that's what we're doing here just initializing


that function and I also have a loss function of course so here we're using a categorical cross


entropy loss and we'll see we'll see how that works. We can also use a scale function as well


I think I thought I did use it a bit later on I think but it's fine. It's not really too important


but we just use that we multiply our similarity score value by the scale value later on.


Down here so we're using transforms optimization so what we're doing here is we're saying for the


first 10 percent of the training data I want to warm up to this learning rate of 2e to minus 5.


So we're not going to start at that learning rate initially we're going to slowly build up to it.


And yeah that's all what we're doing there.


And we can see our batch here so we have attention mask.


So sorry we have the anchor and in the anchor we have our attention mask and we have our input ids.


And then in positive we also be able to say we have attention mask and input ids.


Right so input ids and attention mask use the transformers library before you probably


recognize these it's just the input tensors that we use when we're feeding text into a model.


So that's how our transformer understands our text.


Yeah obviously just mentioned here what is in there and let's come down is there anything


important I don't think so okay let's come down to the training loop so yeah using a scale value of


20 there. And we come to here so here we have our anchor embeddings so these are sentence embeddings


so we have the anchor sentence embeddings and we have positive sentence embeddings which is


output from our BERT model. And what we do we we do the mean pooling so when I said sentence embeddings


here they are the token embeddings sorry so we have 512 token embeddings and then we do the mean


pooling to get the sentence embeddings. And then what we do is we calculate the cosine similarity


between each of our anchors and all of the positive values so we create that array of values of


of cosine similarities. And on each row obviously we would expect the true pair to have the highest


cosine similarity so that is what we will do a little further down here so labels.


What this value does here is just outputs a tensor which is 0 1 2 3 4 up until the batch size of the


data. So that's where you'd expect the argmax value to be. So for A3 you'd expect the maximum


cosine similarity to be in the third index where P3 where it's been compared to P3.


And then here we are calculating the loss so we're calculating between the scores


which we have up here and the labels. So this is taking the but we're basically looking for


this value here to be the maximum value in a specific row at the index equal to the current


label so that's the the A3 P3 paired I'm talking about. Now we're just optimizing on that and


and that's it that's the order is 2mR loss in PyTorch.


But when I say that's all there is there's actually quite a lot I mean there's a lot of


code going into this and it's very confusing. Oh here is the that's the labels tensor that I


just mentioned by the way so you can see it's just counting up to our batch size which is 32.


So that's the PyTorch implementation but it's it's complicated and to be honest if you do the


same in with sentence transformers you're probably going to get better results so I'm going to show


you how to do that with with sentence transformers it's just it's a lot easier and a lot more


effective. Okay so in sentence transformers so we're back in the the first node of the


notebook now so we have our data set 314 rows and it is just anchor positive pairs.


So with sentence transformers we we use something called an input example so we have a big list of


input examples which is the data format that our sentence transformers library training methods


would expect. So we want to do from sentence transformers import input example and what we'll


do is we'll just initialize the list here so samples and this is very simple just going to go


for for sample in for row let's say for row in data set we want to say samples dot append


and in here we have our input example so that you know the thing we just the thing we just


imported the object the special sentence transformers object and this in in the previous


video and article this accepted two different parameters it accepted text and label.


Now like I said we don't have labels we just generate them as we're performing the the training


so we don't need the label here so we just write text and in here we set list and in there we want


our row we want the premise which is our anchor and we also want the hypothesis which is our


our positive okay and that's that's all we need okay.


So that oh that will take a while so so what you can do is if I just go to the


data loader and I'm going to do that and what I can do is if I just go from tqdm.auto import tqdm


so we can add this to our loop here so that we have like a nice progress bar so we can see


how far along we are how long it's going to take and I mean it's very quick anyway but I think it's


now here we need to we need to initialize what is a data loader usually we use the PyTorch


data loader this time we're not going to this time we're going to use a special


data loader from the sentence transformers library and I'll explain why in just a moment so


from sentence transformers I want to import datasets yeah we set the back size as we usually


do we're going to set that equal to 32 and and we're creating a data loader so we're just going


to call it data we're going to call that loader again as we usually do and in here we want to


write datasets and we want to do no duplicates data loader so you can see that there okay and


what this is going to do is unlike a normal data loader in PyTorch which would just feed you


it would just feed you like 32 in this case samples at once it wouldn't check the data in


there or no duplicates data loader checks that you have you don't have any duplicates within the same


batch now realistically this data set probably not going to get that anyway but if you do have


data sets which might have duplicates you can you can use this to avoid that issue because if you


think okay if you have a duplicate and our labels are saying that pair A1 and P1 should be the same


but in reality over here we have A7 and P7 which are exactly the same it's going to confuse the


model and it's going to say that A1 and P7 should be matching but in reality it should just be


A1 and P1 so that's why we use this no duplicates data loader to remove any possibility of that


happening now as well if that happens occasionally it's not really an issue anyway but it's nice to


just be be careful with it so we have our samples and we want the batch size which is equal to our


batch size okay so that's that's our data loader now what we need to do is initialize a model so


in in sentence transformers we we can do the same thing as we do with hug and face transformers where


we load like a pre-trained model but we can also sort of initialize a new model using what are


called modules so in this case we're going to use two modules which is going to be the the


transformer module so so the actual bear itself and that's going to be followed by a pooling


module for the mean pooling that we that we do so we're going to write from sentence transformers


import models models and also sentence transformer we say okay that is equal to


two models and it's a transformer model so we initialize that and we're using a vert base


on case model okay and we also have our pooling module and that is models pool pooling yes and in


here the pooling approach needs to know the dimensionality of the the vectors that you're


dealing with and we can get that from vert we can say get word embedding dimension which is the 768


and as well as we also need to know which which type of pooling we want to use and for that we


can write pooling and we can see we have all these different methods in here so we have the pooling


mode we we can use the cls token we can take the maximum value we can take the mean of the


we can take the mean and consider the square root of the length of the tokens i don't i don't know


this one's completely wrong and we also have the mean so this is a mean pooling method that i


mentioned before but we're using that one so we say true okay and then to actually initialize the


model using those two parts or two modules we're going to write model equals sentence transformer


and like i said before this is how you would usually load a pre-trained model so if you want


to load a pre-trained model you'd be like vert based on case like in here we're not doing that


we are initializing a new model using these two modules so we write modules equals and we have


vert followed by the pooling function or module and then we can we can have a look at what we


have there so we have the model so hopefully it doesn't take too long okay we get all this this


is fine it's just coming from i think hunting face and then here we have the the structure


of our model so we can see i would transformer we're using the vert model and in here the pooling


we have the the embedding dimension so you see and we see that the only one of these values that is


true is a pooling mode mean tokens and the rest of them are false because we're not using those


those methods now all we need to do is is do the we need to initialize the the loss function so


from sentence on summers import losses and our our loss function is going to be equal to


losses and we have the multiple negatives ranking loss which we can we can see there


and all we need to all we need to pass to that is a model so it knows what model it is what


it knows the model parameters that it's dealing with so let's run that and with that we're ready


to actually start training or fine-tuning the model so we'll say okay we want to use one epod


one epod and a number of warm-up steps so like i said before we have that 10 of warm-up steps


that we that we want to use um but isn't as complicated to you know set that this time


we just want to say okay one ten percent zero point one multiplied by the total number of steps


and what total number of steps well it would be the length of the length of the loader okay and


we write int there okay and and that's that was our warm-up steps and then we can call model.fit


so like tennis flow and then in here we just pass a few you know that we pass our model configuration


and setup and so on so first thing we want to do is set our train objectives


which is just a list containing train objective pairs you could say for us we just have one of


those and that is loader followed by loss so our objectives and then we have the the epochs so


epochs just put one there it's probably easier we have a number of warm-up sets so we write warm


steps is equal to warm-up and then we have the output path so where where we're going to save


our model um so i think i use bert mnr2 or something for the for the final one i put together


um and then after that final thing if you want so this will come up with the like a progress bar like


we saw before but the progress bar for me it will just print every single step update so it's quite


annoying so you can set share with progress bar equal to faults if you want and then you just run


this okay we we run this this will fine-tune our model now i've already run it so i'm not going to


run it again it takes well it takes a while um so i'm going to show you that just see the results


from that okay so this is my other netbook where i i well already ran all of what you just saw


uh we have our sentences here so it's a load of random sentences but a couple of them do match up


right so we have this sushi one here and there's another sushi one here but what i've done is not


use the same words in you know both sentences and got noodles and weaving spaghetti we also have


dental specialists with construction materials and dentists for chewing bricks right so similar


in concept but they don't show any of the same words and it's quite abstract as well


so down here we've so we have our model we're just encoding those sentences and then what i'm


doing is i'm just creating a similarity matrix between all of these different sentences or the


encodings produced by our model the sentence embedding is produced by a model and all of the


equivalent embeddings from this list of sentences and we've seen so we just use map.lib and seaborn


to actually plot that and we see that we get this nice sort of visualization and we see that we have


four and three that align and also nine and one and seven and five and those are the the three


pairs i mentioned before that are very similar and the rest of these values are all very low


which is obviously very good because those pairs are not similar like maybe they have some sort of


similarity but they're not similar so we can obviously see straight away where the the true


pairs are or the true semantic pairs are there okay so i think i think that's pretty much it


for this video we've kind of gone through multiple negative ranking loss and and and how to implement


it and like i said before this is really if you're going to train a sentence transformer or fine-tune


a sentence transformer this is the approach i would probably go with depending on the data that you


have now i mean that's everything for the video so thank you very much for watching i hope you've


Train Sentence Transformers by Generating Queries (GenQ)

Today we're going to have a look at how to fine-tune or build a sentence transformer


or by encoder model using a very limited dataset.


So when I say limited dataset all we actually need to fine-tune this by encoder model is


unstructured text data from our target domain.


So the domain or the area that we want to apply the model to.


So the reason that we might want to do this is I think quite clear for anyone who has


trained sentence transformers by encoders before.


We in most cases will find that we don't have that much data and the data that we do have


is usually not in the correct format for actually training a model.


So we're going to look at one of the most recent methods for actually doing this and


that is called GenQ.


At a very high level GenQ relies or the main benefits of GenQ are these three points here


or reasons for using GenQ.


Our training data is simply unstructured text.


So when I say unstructured text it's if you have for example PDF documents or web pages


that you're scraping data from you can use that as your training data.


It's specifically for asymmetric semantic search e.g. where your queries that you're


searching with are generally smaller than the passages or context that you're searching


for.


We'll explain that a little bit more pretty soon.


And one of the very impressive things with GenQ is that the performance can approach


the performance of models trained using supervised methods.


Now that's not going to be the case for every single use case and I'd say you're probably


going to be relatively lucky if you start getting towards supervised performance but


it does happen.


So that's I think a pretty good indication that this is a good technique to use or at


least try.


So at a high level let's go through what GenQ actually does.


So we start with our unstructured text over here so our passages of text.


We take a T5 model so a T5 is a model from Google and the general philosophy of T5 is


that every problem in NLP is a text to text problem.


And we'll explain that a little more in a moment.


But what we're doing with this T5 model is we take our long passages, unstructured text


and use T5 to generate queries which are these short chunks of text over here.


Now these queries are basically questions that the passage should answer.


So what we end up getting are these query passage pairs.


Now if you have trained a lot of buying coder models you probably recognize that a query


passage pair like this is basically the format we need to actually train a model using multiple


negatives ranking loss which is actually what we do here.


So we're synthetically generating these queries and then we're actually using a supervised


training method to train on those queries.


So it's almost unsupervised because we're not needing to label data or anything here


but in reality we are actually labeling data it's just automated.


So it's a bit of a unsupervised supervised learning technique.


And from that we get our buying coder or sentence transform model at the bottom here.


Now don't worry if none of that makes sense yet we are going to go through everything


in a lot more detail.


So let's start at the very start with our unlabeled data.


We can describe our data as either being in domain or out of domain.


So what I mean by that is for example we want to train a model that is going to allow us


to query or search through German financial documents.


So in domain for that particular use case might be what we have over here.


So we have a German finance news articles probably in domain and then we also have German


finance regulation documents probably in domain.


Out of domain but not too far out of domain is English financial documents over here.


So in some cases it can be okay to train your model on some things that are slightly out


of domain like in this area here if that's all you have.


But ideally you want things to be very specifically in your domain.


And then other things like PyTorch documentation PyTorch whether it's in German or English


it's out of domain because it's not finance documents.


Doom books are definitely getting out of domain here.


And even if we had for example something like German pirate metal even though it's in German


it's still out of domain because there's probably very little crossover between that and financial


documents.


So ideally the data that we do need for training this it can be structured it can just be documents


or web pages but it should really be within the topic of if we're doing German financial


a German financial model we would ideally want something like the finance news articles


or the finance regulations in German.


So that's in domain and out of domain.


And GenQ is specifically built for where we have a lot of in domain data but it's unstructured


like documents or web pages as I said before.


So another thing that GenQ is very specifically built for is the what I called asymmetric


semantic search.


Now asymmetric semantic search is basically where you have asymmetry between the sizes


or the length of the text between what you are querying so your search query and what


you're searching for e.g. trying to pull from your database.


So your queries might be very small let's say on average they are 32 words long that


is probably actually very long your queries are probably even shorter.


Imagine when you type into Google so maybe more like 5 to 10 words at most and then what


you return from Google are pretty usually little chunks of text and web pages and so


on so your passages are kind of like those chunks of text and they're bigger.


So there's asymmetry in the sizes between what you are searching with and searching


for.


That's where you get the asymmetric in asymmetric semantic search.


Now the next point is that we're synthetically generating queries.


So we have that unstructured text it's just passages but to train the model we're actually


going to use a supervised training method and to train with a non-supervised training


method we need to have queries that match up to those passages.


So to generate those queries we use a T5 model that has been fine-tuned specifically for


query generation.


So as I mentioned before the philosophy of T5 is every problem is a text to text problem.


So T5 is not just used for query generation we can use it and say okay translate English


to German that's good and get the German translation for that.


You can use it for scoring two sentences on how similar they are and you get like 3.8


for these two because they're relatively similar and the range there is from 0 to 5.


And then you might get a covalent sentence so that is where you are saying is this sentence


correct does it make sense and the course is jumping well it doesn't make sense so it's


saying okay this is not acceptable.


All of these although okay for example in the semantic similarity example we do output


3.8 it's still actually text so all of this is just text to text and that's the core philosophy


behind T5.


It's just an encoder decoder model everything is handled every problem is handled text to


text.


The same applies to our problem we have a passage and we want to generate a query for


that passage okay it's just text to text.


And with T5 it might look something like this so the passage up at the top here we have


this really long you know this from Wikipedia and the expert examples on GenQ.


Just a paragraph saying Python is interpreted high level etc etc programming language so


it's just a little intro to Python.


T5 takes this and it will say okay let's generate three queries okay and they're randomized


so it can take this one passage and apply a little bit of randomness and generate different


queries.


So we get this define Python program.


What is Python program and what is Python useful?


Like they're all queries that we would probably search in Google for example and return a


pass that looks like this.


Now for this we do need a T5 model that has been trained to produce queries and I'll show


you later that we are going to use a specifically trained model for that and it is worth noting


that your queries or the queries that are generated can be a little bit noisy and you


might maybe we'll see an example later of this.


What I mean by noisy is that sometimes they can be nonsensical or just weird queries and


like not good quality queries and that is probably the main issue with GenQ that you


need to be aware of is that because we're generating queries using a language generation


model and that's pretty much the only step in generating the training data.


If there's any noise that will be translated into the performance of our model later on


that we are training and will obviously make the performance less than it could be if the


generate queries were all perfect.


But that's to be expected nonetheless it's in most cases I think works pretty well.


So let's have a look at an example of this in Python.


So here we have the generation code for just one example so this paragraph here.


So I'm taking this example from the S-PAT.NET web page on GenQ.


So all we're doing here is we're downloading a model we're using a via model over here


that's been trained specifically for this although as far as I know it's been trained


for GPL and not GenQ but nonetheless this works for this.


We initialize a tokenizer and a model using the transformers library here and we're using


t5 for conditional generation basically to generate text.


This is our import text and it's just the same paragraph we saw before and all we're


going to do is we're going to create our input ids by tokenizing this text and then we're


going to generate three sequences here and down here all I'm doing is decoding the sentences


or sequences that we generated into text.


And I think this is the same example this is where I actually pulled the example that


we saw before from so we generate these queries we get to find Python program what is Python


used for what is Python program.


That's all there is to the query generation step.


So it's not really complicated we just need to do it for a lot of queries or a lot of


passages.


So that's the query generation step and what we get after the query generation obviously


is a set of a query passage pairs and then we can use those to fine-tune our model.


Now for fine-tuning our model we use multiple native ranking loss.


Now that's sort of a relatively abstract animation going through what MNR is actually doing.


So MNR at a high level works by placing all those pairs that we created the query passage


pairs into batches.


And each one of these batches you imagine your query at position 0 should have the highest


similarity score with your passage at position 0 because they're the pair.


But then you have a batch so you're actually going to compare the similarity between the


query at position 0 and the passage at position 1 and 2 and 3 and 4 and so on.


And for each batch the model weights are going to be optimized so that the pair in position


0 have the highest similarity and for example a query 0 and query 3 or 4 or 5 and so on


have a lowest similarity.


And we're not going to go super in-depth on MNR here because we've covered it before


and I'll make sure there's a link to that actual article and video in the description.


But at a high level it's basically just a ranking optimization.


We are optimizing so that the pairs that should be together, those blocks that you see that


are more obvious in the diagonal line in that batch, those should have the highest similarity


scores.


So with that we will have a bind encoder that has been fine tuned to our specific use case.


Now let's have a look at the actual code that we can use to do that.


So the very first step that we'll need to go through is actually getting our data.


So to get our data we're just going to use the squad data set.


The reason I'm using this is because the test set of this has both the questions and context


pairs so we can look or we can assess the quality of our model at the end of this.


I'm just checking here, a lot of these contexts are duplicated because we have a sample of


one context we might have 30 questions.


So all we're doing here is removing those duplicates so that we just have the passages.


So we can see that the final length or the final number of passages that we have is just


under 19,000 here.


Which is not that many.


I think for a lot of use cases if you're using unstructured data you can probably get more.


But in this case it does work very well with just a small amount.


But it's a very generic data set so for more specific use cases maybe you might want to


have more data, maybe not you'll just have to play around with it and see what works.


So to generate the queries you already saw before what we were doing.


So all we're doing is loading this model again, tokenizer and the model.


We're setting the model to evaluation so for example if there's any normalization layers


in there they are set to their evaluation mode.


Ready for inference.


So we're going to ship that and we move on to the inference step where we are going to


generate the queries for our passages.


So here in TQDM we're just using this as a progress bar.


There's nothing fancy going on there.


Just so we can actually see the progress because it can take a little bit of time particularly


if you have a very large amount of data.


We're setting torches to no grad so we're not calculating the gradients of our model


because it takes more time.


We're not updating or optimizing the model we're just using inference here.


And then what we need to do is get our passage.


So for P in TQDM passages we are doing that so if I just show you here.


So look at passage 0 just one example.


Of course I'm just going to run this.


And this is just our passage.


So we're going through those.


If there's any tab characters which I generally don't think there are but maybe there's the


odd one.


I'm going to replace those because later we're going to be using a tab separated values for


a TSV file for storing our pairs.


And obviously if we have more than one tab in those pairs separating the query and the


passage it's going to mess up the formatting of the file.


So I'm just removing those placing them in spaces just in case.


Then creating the input IDs using tokenizer and code as we did before.


Then we're generating three queries per passage.


We decode the queries to human readable language so we can read it and so the next model can


actually can tokenize that text and understand it otherwise we're using token IDs from another


model which can't be read by our next buying code and model when we're fine tuning it.


So we decode them into human language and then we are going to add the pairs.


So that's our query just in case any tab characters end up in there.


We're generating them we just replace any tab characters with space again and then we


use a tab character to separate the query from our passage.


And then what I'm doing here is I'm saying every 1024 pairs we create I want to save


that or write that to file.


Then we increase the file count which is because every new file is going to save as pairs.


For example the first one is pairs 0 and it will be pairs 1 and pairs 2 and so on.


And then we just reset or refresh the pairs list.


Finally if we because here we're doing it in batches of 1024 maybe the final batch only


has some 600 pairs in there I'm just saying if pairs is not non e.g. the list is not empty


I want you to save that final batch.


And with that we've generated our data so that's the generative or query generation


step.


And we can then move on to the actual fine tuning of the buying code and model.


So to do that obviously we're in a new notebook now so first thing I want to do is get a list


of all the files or the pair files I created before.


So what we're doing here we saved it into the data directory and they're all.tsv files.


Then because we're going to use sentence transformers to train here we need to import the input


example object.


And this is just a specific object type used by the Sentence Transformers library.


Just a standardized format so that we can easily read data that we're going to train


with.


So all we're doing here is looping through all of those files we're opening all of them


reading all the lines as all pairs and here if there's no tab character in the line we're


just kind of avoiding it just in case for whatever reason the tab character isn't in


there.


And then here we're splitting getting the query and passage and we're creating the input


example and we're appending that to pairs.


So we have this big list of input examples of our queries and passages.


And then we set up the data loader for multiple negative ranking loss.


We've done this before but if you need to you can go through it.


So I'm using a batch size of 24.


With MNR the greater the batch size the better generally because you're basically making


it harder for the model because if you have more pairs in that batch the harder it is


going to be to rank that one pair that is the correct pair at position number one.


If there's only for example two in your batch your model has like a 50 50 chance if it's


just guessing of ranking the correct pair as position one.


If there's a hundred in your batch it has a pretty small chance of actually guessing


that correctly.


So that's why we try and use a larger batch size but obviously it will depend on your


available compute that you have.


And then we are going to use this node duplicates data loader from sentence transformers.


So this is specifically built to avoid having duplicate queries or contexts in your batch


because obviously if you're trying to assess or you're trying to rank your query and passages


correctly and you have duplicate passages in there your models won't get confused and


it can't actually rank them correctly.


So we avoid having duplicates in there using this.


And then we are just going to define the bind coder model that we're going to be fine tuning.


So I'm using MPNet model here.


MPNet is generally quite good as a sentence transformer bind coder model.


So it's a good one to use.


We are using the mean token or mean pooling method here.


So all the word embeddings that the model outputs are being compressed into a single


sentence embedding by taking the average across all of the dimensions.


And I'm putting those both together the actual transform model followed by the pooling layer


to create our bind coder model.


And then after we initialize our model we can initialize the MNL loss function.


Super easy again with sentence transformers.


All we do is losses multiple negative ranking loss and then pass the model in there.


And then we train, I put single epoch here.


We're actually going to train, you can train for a single epoch or I think that's what


they do in the GenQ paper.


But I've also seen good results from using three epochs here which is I think the default


in the sentence transformer documentation.


So you can go single epoch, maybe try it if it works otherwise go with three epochs.


In this case I use three.


Here ignore that.


So we just fit, we pass our loader loss like typical sentence transformers training here.


And then I save the model as MPNet GenQ squad.


We warm up for 10% of those training sets as well.


And then yep the model is saved in that directory and we can just load it as it would any other


sentence transformer model.


So I think that's it for this walkthrough.


What I'm going to do is in a separate video we'll have a look at how we can test this


model and evaluate it.


So that'll be interesting.


I'll release it pretty soon after this.


So I hope that's been useful.


Thank you very much for watching and I will see you again in the next one.


Streamlit for ML #1 - Installation and API

Today we're going to start working on a NLP app for open-debating question


answering built using Streamlit. Now this will be a series of videos and


articles and in this video we're just going to introduce the overall plan


for the app. We're going to have a look how we install Streamlit and everything


else we need and then we're going to put together a few simple components that


will basically act as the core of our app. So let's first have a look at what


we're going to build. So what I essentially want to do is have like a


web page where we have a search bar nothing particularly complicated here


and that's supposed to be magnifying glass and what we're going to do is type


like a question in here so anything question mark and then we enter and we


want to return a load of results. Now these results we want them to be


intelligent so we're going to ask a question we're going to be using the


Wikipedia data set so we're going to have paragraphs or chunks of text from


Wikipedia pages indexed somewhere which we'll explain in a minute. I want to


return the most relevant of those so if I ask something like who are the Normans


it's probably going to come back with Wikipedia page about Normans saying


Normans were people in north of France in this certain time in the 10th 11th


century and so on. So it's going to be like a really big chunk of text and for


now we're going to do that we're going to create like a we're going to return a


big chunk of text but then in the end what I actually want to do is kind of


use a what's called a reader model to specify a smaller part of that text. It's


going to say okay who are the Normans it's going to say there were people in the


10th 11th century in northern France and that's why it's going to return like a


very specific segment from a larger paragraph and we're going to make it


look nice so we're going to be able to press like a little down button here


when we display just this little answer and it would then expand to show us a


full answer the full paragraph. So that is in essence what we are going to do. Now


to do that we need a few components as Streamlit which acts as the front end


user interface for our app which is what we're going to be able to see and


then the back end we're going to build and open the main Q&A app. Now I'm going


to really breeze over this so super quick we're going to have a vector


database just store the paragraphs I mentioned. For that we're going to be


using Pinecone. We're going to have a retriever model we're going to use a


BERT retriever model that's been trained on question answering and when we want


to include this little short snippet we're going to also have a reader model


okay and that's sort of the back end and then we have Streamlit on the front end so


it should be pretty cool. Now let's actually go ahead and install Streamlit.


Okay so to create your environment to do that I'm using Anaconda here. Anaconda


distribution just makes things super easy and all you're really going to need


to do I'm going to breeze through this very quickly is you want to do


condocrate new your environment name I'm going to use Streamlit and your Python


version which is 3.8 that I'm using followed by Anaconda so you enter that


into your your terminal window so far I can copy this okay I would do this


Python version 3.8 and for the environment name I will use Streamlit.


Now I've done this already so I'm not going to do it again and from there we


just conda activate that environment name I will show you this so conda


activate Streamlit okay you see over here we're now in this Streamlit


environment and from there we are going to install a few things so we're going


to pip install Streamlit for one, Pinecone client so this is going to be how we


set up or create our vector database which is going to store our paragraphs our


Wikipedia paragraphs and we also want to use sentence transformers okay so you


have to go ahead and pip install all of those again I've already done it I'm not


going to redo it and then we can check that Streamlit is installed with


Streamlit hello okay and this should I if I write it correctly this should pop


up with a like a default or template app in Streamlit in our browser window so


we'll take a moment to load okay and here we have it okay so we can see we're


in our browser it just opened it automatically localhost 8501 so that's


the default port for Streamlit apps and there we go so Streamlit is installed


now I'm going to ctrl C to close that and what I'm going to do is switch over


to VS code and start building an app so in VS code now I have this Streamlit


dash NLP directories empty the moment so first I'm going to do is create a new


file app.py okay now app.py is going to be well it's going to be our app


it's going to be where we where we do everything now to actually use


Streamlit we want to import Streamlit as st now we want to build our like our


search box so what we'll first do is create like a little description so we


can write or we can generate HTML code through Streamlit by typing markdown you


can also use HTML which I will show you at some point but for now let's create a


little header we're going to call it AI Q&A and I'm going to say ask me a


question okay and Streamlit will go ahead and actually convert that into HTML code


now let's go ahead and initialize this app so what we first need to do is


navigate to that directory so for me it's going to be in documents projects


and I call it Streamlit NLP okay and then we just want to write Streamlit


run app.py and this will initialize our app in the same way that we did Streamlit


hello earlier so you can see we have this localhost 8501 so if we go


ahead and open that in our browser we'll see our web page okay now it's super


simple at the moment we just have this AI Q&A our header and this little


description ask me a question okay now what I'm going to do is place this on


one side and I want VS code on the other side okay now what we can do in here


let's change something first make it easy for ourselves so I want to also add