a label which is our similarity score. Now that similarity score varies from between zero to five


Making The Most of Data: Augmented SBERT

where zero is no similarity no relation between the two sentence pairs and five is they mean that


same thing. Now see here these two mean the same thing as we


Now we can see here that these two mean the same thing as we would expect. So we first want to


modify that score a little bit because we are going to be training using cosine similarity loss


and we would expect our label to not go up to a value of five but we would expect it to go up to


a value of one. So all I'm doing here is changing that score so that we are dividing everything by


five normalizing everything. So we do that and no problem and now what we can do is load our


training data into a data loader. So to do that we first need to load our training data into a


data loader. So to do that we first form everything into a input example and then load that into


into our PyTorch data loader. So I'll run that and then at the same time during training I also


want to output a evaluation source. So how the cross encoder do on the evaluation data.


So to do that I import. So here we're importing from sentence transformers cross encoder


evaluation. I'm importing the cross encoder CE correlation evaluator.


I again am using input examples with working sentence transformers library


and I am importing both text and labels.


And here I am putting all that development or I'm putting all that validation of that data


into that evaluator. Okay now I can run that and then we can move on to initializing a cross encoder


and training it and also evaluating it. So to do that we're going to import from sentence


transformers. So from sentence transformers and I'll just make sure I'm working in Python.


I'm going to import from cross encoder


a cross encoder. Okay and to initialize that cross encoder model I'll call it C.


All I need to do is write cross encoder very similar to when we write sentence transformer


initializer and model. We specify the model from the face transformers that we like to


initialize a cross encoder from. So that based on case and also a number of labels that we'd like to


use. So in this case we are just targeting a similarity as well between 0 and 1. So we just


want a single label there. If we were doing for example NLI labels where we have entailment


contradiction and neutral labels or some other labels we would change this to for example 3.


But in this case 1. We can initialize our cross encoder and then from there we move on to actually


training. So we call model or C.fit and we want to specify the data loader. So this is slightly


different to the fit function we usually use with sentence transformers. So we want train data loader.


We specify our loader that we initialize just up here the data loader. We don't need to do this


but if you are going to evaluate your model during training you also want to add in evaluator as well.


So this is from the C correlation evaluator. Make sure here using a cross encoder evaluation class.


We would like to run for say one epoch and we should define this because I would also like to


while we're training I would also like to include some warm up sets as well. I'm going to include a


lot of warm up sets actually. Although I'll mention it I'll talk about it in a moment.


So I would say number of epochs


is equal to one and for the warm up I would like to take integer. So the length of loader. So the


number of batches that we have in our data set. I'm going to multiply this by 0.4. So I'm going to do


a warm up or do warm up sets for 40 percent of our total data set size or batch or 40 percent of our


total number of batches. And we also need to multiply that by number of epochs. Say we're


training two epochs we multiply that in this case just one. So not necessary but it's there.


So we're actually performing warm up for 40 percent of the training steps and I found this


works better than something like 10 percent 15 percent 20 percent. However that being said I


think you could also achieve a similar result by just decreasing the learning rate of your model. So


by default. So if I write in the epochs here we'll define the warm up sets.


So by default this will use optimizer params with a learning rate of 2e to the minus 5.


OK. So if you say want to decrease that a little bit you could go let's say go to the minus 6 5e to minus 6.


And this would probably have a similar effect to having such a significant number of warm up


sets. And then in this case you could decrease this to 1 or 10 percent. But for me the way I've


tested this I've ended up going with 40 percent warm up sets and that works quite well. So the


final step here is where do we want to save our model. So I'm going to say I want to save it into


BERT base cross encoder or let's say BERT STSB cross encoder. And we can run that and that will


run everything for us. I'll just make sure it's actually. Yep there we go. So see it's running


but I'm not going to run it because I've already done it. So let me pause that and I will move on


to the next step. OK. So we now have our gold data set which we have pulled from HuginFace data sets


and we've just fine tuned a cross encoder. So let's cross both of those off of here.


This and this. And now so before we actually go on to predicting labels with the cross encoder


we need to actually create that unlabeled data set. So let's do that through random sampling


using the gold data set you already have. And then we can move on to the next steps.


OK. So I'll just add a little bit of separation in here. So now we're going to go ahead and create


the augmented data. So as I said we're going to be using random sampling for that. And I find that


the easiest way to do that is to actually go ahead and use a Pandas data frame rather than


using the HuginFace data set object that we currently have. So I'm going to go ahead and


initialize that. So we have our gold data. That will be pde.data frame.


And in here we're going to have sentence one and sentence two. So sentence one.


That is going to be equal to stsb sentence one. OK. And as well as that we also have sentence two


which is going to be stsb sentence two. Now we may also want to include our


label in there. Although I wouldn't say this is really necessary.


Or add it in. So our label is just label.


And if I have a look here. So we have. I'm going to overwrite anything called gold.


So OK. I'm going to have a look at that as well. So we can see a few examples of what we're actually


working with. I'll just go ahead and actually rerun these as well.


OK. So there we have our gold data. And now what we can do because we've


reformatted that into a kind of data frame. We can use the sample method to randomly sample


different sentences. So to do that what I will want to do is create a new data frame.


So this is going to be our unlabeled silver data set. It's not going to be a silver data set.


Because we don't have the labels or scores yet.


But this is going to be where we will put them. And in here we again will have sentence one.


And also sentence two. But at the moment they're empty. There's nothing in there yet.


So what we need to do is actually iterate through all of the rows in here. So before that I'm just


going to do from or import TQDM.auto from TQDM.auto import TQDM. And that's just a progress bar.


So we can see where we are. I don't really like to wait and have no idea how long this is taking to


process. And for sentence one in TQDM. So we have the progress bar. And I want to take a list


of a set. So we're taking all the unique values in the gold data frame for sentence one.


Okay so that will just loop through every single unique sentence one item in there. And I'm going


to use that and I'm going to randomly sample five sentences from the other column sentence two


to be paired with that sentence one. And here I'll sample the sentence two phrases that we're


going to sample are going to come from the gold data of course. And we only want to sample from


rows where sentence one is not equal to the current sentence one because otherwise we


are possibly going to introduce duplicates. And we're going to remove duplicates anyway but let's


just remove them from the sampling in the first place. So we're going to take that so all of the


gold data set that where sentence one is not equal to sentence one. And what I'm going to do is just


sample five of those rows like that. Now from that I'm just going to extract sentence two. So the five


sentence two phrases that we have there. And I'm going to convert them into a list. And now for


sentence two in the sampled list that we just created I'm going to take my pairs. I'm going to


append new pairs. So pairs are append and I want sentence one to be sentence one.


And also sentence two is going to be equal to sentence two. Now this will take a little while.


So what I'm going to do is actually maybe not include the full data set here.


So let me possibly just go maybe the first 500. Yeah let's go to the first 500.


See how long that takes. And I will also want to just have a look at what we get from that.


So yes it's much quicker. So we have sentence one. Let me remove that from there.


And let's just say that top 10. So because we are taking five of sentence one every time and


random sampling it we can see that we have a few of those. And another thing that we might want to


do is remove any duplicates. Now there probably isn't any duplicates here but we can check. So


pairs equals pairs.drop duplicates. And then we'll check the length of pairs again.


And also print. Let me run this again and print.


Okay so there were not any duplicates anyway but it's a good idea to add that in just in case.


And now what I want to do is actually take the cross encoder. In fact actually let's go back to


our little flowcharts. So we have now created our larger unlabeled data set. So it's good. And now


we go on to predicting the labels of our cross encoder. So down here what I'm going to do is


take the cross encoder code here. And what I've done is I've trained this already and I've


uploaded it to the Hugin base models. So what you can do and what I can do is this. So I'm going to


write James Callum and it is called BERT STSB cross encoder. Okay so that's our cross encoder.


And now what I want to do is use that cross encoder to create our labels. So that will create


our silver data set. Now to do that I'm going to call it silver. For now I mean this isn't really


the silver data set but it's fine. And what I'm going to do is create a list and I'm going to zip


both of the columns from our pairs. So pairs sentence one, pairs sentence two. Pairs sentence


one and pairs sentence two. Okay so that will give us all of our pairs again. You can look at those.


Okay so it's just like this. And what we want to do now is actually create our score. So just


take the cross encoder. What did we load it as? CE.predict and we just pass in that silver data.


So do that. Let's run it. It might take a moment. Okay so it's definitely taking a moment. So


let me pause it. I'm going to just do let's say 10 because I already have the full data set so I


can show you that somewhere else. And let's have a look at what you have in those scores. So three


of them. So we have an array and we have these scores. Okay so that they are our predictions,


our similarity predictions for the first three. Now because they're randomly sampled a lot of


these are negative. So if we go silver, say negative. I mean more. They're not relevant.


So yeah we can see not particularly relevant. And that's just one must first issue with this.


And you can try and modify that by after creating your scores. If you oversample and


got a lot of values or a lot of records and then just go ahead and remove most of the low scoring


samples and keep all of your high scoring samples that will help you deal with that imbalance in


your data. So what I'm going to do is I'm going to add to the labels column those scores which


will not actually cover all of them because we only have 10 in here. So let me maybe multiply that.


So this isn't, you shouldn't do this obviously. It's just so they fit.


Okay and let's have a look. Okay so we now have sense one, sense two and some labels.


And what you do, although I'm not going to run this, is you would write pairs.to csv.


Don't necessarily need to do this if you're running everything in the same notebook.


But it's probably a good idea. So with csv, I'm going to say the silver data is a tab separated


file. And obviously the separator for that type of file is a tab character. And I don't want to


include those. Okay and that will create the silver data file that we can train with.


Which I do already have. So if we come over here, we can see that I have this file and


we have all of these different sentence pairs and the scores that our encoder has assigned to that.


So I'm going to close that and I'm going to go back to the demo.


And what I'm now going to do is actually, well first go back to the flow chart that we had.


I'm going to cross off predict labels.


And we're going to go ahead and fine tune the buy encoder on both gold and silver data.


So we have the gold data. Let's have a look at what we have. Yes and the silver. I'm going to


load that from file. So pd.read csv. Silver.tsv. And separator is a tab character. And let's have a look.


What we have. Make sure it's all loaded correctly. Looks good. Now what I'm going to do is


put both those together. So all data is equal to gold.append silver. And we ignore the index.


So we're going to get an index error. Sorry. True. And all data.head. Okay we can see that we


hopefully now have all of the data in there. So let's check the length.


Yeah so it's definitely a bigger data set now than before with just gold.


Okay so we now have a larger data set. We can go ahead and use that to fine tune the


the buy encoder or sentence transformer. So what I'm going to do is take the code from up here.


So we have this train data. And I think I've already run this before so I don't need to


import the import example here. But what I want to do here is for row in all data.


And what we actually want to do here is for i row in all data because this is a data frame.


It iterates through each row. We have row, sentence one, sentence two, and also a label.


So we load them into our train data. And we can have a look at that train data.


See what it looks like.


Okay we see that we get all these input example objects. If you want to see what one of those


has inside you can access the text like this. Should probably do that on a


in a new cell. So let me pull this down here. And you can also access a label to see what we


what we have in there. Okay so that looks good. And we can now take that like we did before and


load it into a data loader. So let me go up again and we'll copy that. Where are you?


Take this. Bring it down here. And we run this. Creates our data loader. And we can move on to


actually initializing the sentence transformer or by encoder and actually training it. So


once you run from sentence transformers.


We're going to import models and we're also going to import sentence transformer. Now to initialize


our sentence transformer if you've been following along with the series of videos and articles.


You will know that we do something looks like this. So we're going to convert and we're going


to import the sentence transformer. And we're going to import the sentence transformer.


So we're going to convert and that is going to be models.transformer.


And here we're just loading a model from copy paste transformers. So


that base in case. And we also have our pooling layer. So models again and we have pooling.


And in here we want to include the dimensionality of the vectors that the pooling


layer should expect. Which is just going to be vert.get word embedding dimension.


And also it needs to know what type of pooling we're going to use. Are we going to use


CLS pooling? Are we going to use mean pooling, max pooling or so on. Now we are going to use


pooling and we're going to use a mean. So mode mean tokens. Let me set that to true.


So there are the two let's say components in our sentence transformer. And we need to now put


those together. So we're going to call model equals sentence transformer. And we write modules.


And then we just pass as a list vert and also pooling. Okay. So we run that. We can also have


a look at what our model looks like. Okay. And we have a sentence transformer object. And inside


there we have two layers or components. First one is our transformer. It's a vert model. And the


second one is our pooling. And we can see here the only pooling method that is set to true is the


mode mean tokens. Which means we're going to take the mean across all the word embeddings


output by vert and use that to create our sentence embedding or vector. So with that model now


defined we can initialize our loss function. So we do want to write from sentence transformers


dot losses import cosine similarity loss. So cosine similarity loss. And in here we


need to pass the model so it understands which parameters to actually optimize.


And initialize that. And then we sell our training function or the fit function. And


that's similar to before the cross encoder although slightly different. So let me take that.


That's a little further up from here.


Then take that and we're just going to modify it. So warm up. I'm going to warm up for 15% of the


number of steps that we're going to run through. We change this to model. It's not C anymore.


And like I said there are some differences here. So we have a training objectives. That's different.


And this is just a list of all the training objectives we have. We are only using one.


And we just pass loader and loss into that. Evaluator. We could use an evaluator. I'm not


going to. For this one I'm going to evaluate everything afterwards. The epochs and warm


steps are the same. The only thing that's different is the output path which is going to be vert


stsp.org. That's it. So go ahead and run that. It should run. Let's check that it does.


Okay so I've got this error here. So it's lucky that we checked. And this runtime error found


dtype long but expected to float. And if we come up here it's going to be in the data loader or in


the data that we've initialized. So here I've put int for some reason. I'm not sure why


that is. So this should be a float. The label in your training data. And that should be the same


up here as well.


Okay so here as well the cross encoder. We would expect a float value. So just be aware that I'll


make sure there's a note in the video earlier on for that. Okay and okay let's continue through


that and try and rerun it. Should be okay now. Oh I need to actually rerun everything else as well.


So rerun this. Okay label 1.0. Okay it's better. This is this. I'll just leave this for a moment.


Just to be sure that is actually running this time. But it does look good. So yeah that's fine.


So it looks good. When for some reason in the notebook I'm actually seeing the number of


iterations. But okay yeah pause it now and we can see that it's actually running.


I'm actually seeing the number of iterations. But okay pause it now and we can see that yes it did


run through two iterations. So it is running correctly now. That's good. So that's great.


What I want to do now is actually show you okay evaluation of these models. So back to our flow


chart quickly. Okay so fine tune by encoder. We've just done it. So we've now finished with our in


the main augmented expert training strategy. And yeah let's move on to the evaluation. Okay so my


evaluation script here is maybe not the easiest to read.


But basically all we're doing is we're importing the embedding similarity evaluated from down here.


I'm loading the the glue data. SDSP again and we're taking the validation split which we didn't


train on. We are converting it into input examples feeding it into our embedding similarity evaluator.


And loading the model. The model name I pass through some command line arguments from up here.


And then it just prints out the score. So let me switch across to the command line.


We can see how that actually performs. Okay so just switch across to my other desktop because


this is much faster. So I can actually run this quickly. So python and 03. So we're going to run


that evaluation script. And what we're going to pass here is we have all three. All three the


cross encoder the sentence transformer train using augmented expert and also a sentence


transformer trained purely on the gold data set. So first let's have a look at the STSB gold data


set trained model. So run this. It might take a moment to download it. Okay so everything's


downloaded and then we've got a score of 0.506. So it correlates to the predictions of the model


correlate to the actual scores with a 50% correlation. So they do correlate. It's not bad.


It's not great either. Let's have a look at the cross encoder. So again cross encoder.


Okay and we get score of 0.58. So as we would expect training on just the gold data


the cross encoder does outperform the the by encoder or sentence transformer. And the final


one would be okay with the augmented data how does the sentence transformer perform.


So let's run that again. Not a way to download. And we get a much better score of 0.69. So yeah


the correlation there is much higher than this for the augmented data set than if we had just


used a gold data set. So it really really has improved the performance a lot. Now this is


maybe an atypical performance increase. It's something like 90% or 19 point increase in


performance and that's good. But if you look at the the original paper from Nils Reimers and Co.


they found a sort of expected performance increase of I believe seven or nine points.


So this is definitely pretty significant. This is definitely a bit more than that. But I think it


goes to show how good these models or this training strategy can actually be. So that's it


for this video. I hope this has been useful and I hope this helps a few of you kind of overcome the


sometimes lack of data that we find in I think a lot of our particular use cases.


How-to use the Kaggle API in Python

Hi and welcome to this video where we are going to go through setting up and using the Kaggle API.


How-to use the Kaggle API in Python

So the first thing we want to do is actually pip install kaggle.


How-to use the Kaggle API in Python

Now I already have it installed so I'm not going to go ahead and install it again.


How-to use the Kaggle API in Python

But once you do have it installed.


How-to use the Kaggle API in Python

You can try and import the kaggle module.


How-to use the Kaggle API in Python

And you will get this error here.


How-to use the Kaggle API in Python

So this error simply tells you that you could not find the kaggle.json and you need to add it to this location here.


How-to use the Kaggle API in Python

Now the reason it's telling you this is because we use kaggle.json to authenticate our API access.


How-to use the Kaggle API in Python

Obviously kaggle is not going to let anyone access their API.


How-to use the Kaggle API in Python

You need to have a account before you start downloading their data.


How-to use the Kaggle API in Python

So to get our kaggle.json credentials we simply go over to kaggle.com.


How-to use the Kaggle API in Python

Now if you don't have an account you'll have to go ahead and create one.


How-to use the Kaggle API in Python

Once you've created your account you simply go over to this little icon over here in the top right.


How-to use the Kaggle API in Python

Click account.


How-to use the Kaggle API in Python

And scroll down until you see this API section.


How-to use the Kaggle API in Python

Now all you need to do is create a new API token.


How-to use the Kaggle API in Python

And this creates the kaggle.json credentials and allows me to save them to my computer.


How-to use the Kaggle API in Python

So I'm just going to save them in my documents for now.


How-to use the Kaggle API in Python

And then head back to the notebook.


How-to use the Kaggle API in Python

And we're going to see that we need to save it here.


How-to use the Kaggle API in Python

So I'm going to copy and paste that across.


How-to use the Kaggle API in Python

And here we have the directory that we need to put our kaggle.json.


How-to use the Kaggle API in Python

I'm going to take my kaggle.json and simply move it into here.


How-to use the Kaggle API in Python

OK so to check that it's worked we simply rerun this cell.


How-to use the Kaggle API in Python

And there we can see that our kaggle API is now functional.


How-to use the Kaggle API in Python

Now we don't actually need this import kaggle.


How-to use the Kaggle API in Python

Instead we need to import the kaggle API class from the kaggle API extended module.


How-to use the Kaggle API in Python

Once we've imported that we simply initialize our API.


How-to use the Kaggle API in Python

And then authenticate it.


How-to use the Kaggle API in Python

Now we're ready to start downloading datasets.


How-to use the Kaggle API in Python

And the kaggle API gives us several options for doing this.


How-to use the Kaggle API in Python

The two that you're most likely to use are for downloading the competition datasets or standalone datasets.


How-to use the Kaggle API in Python

Now a competition dataset is related to a current or past competition.


How-to use the Kaggle API in Python

So for example there is a sentiment analysis on movie reviews competition.


How-to use the Kaggle API in Python

We can actually find it over here.


How-to use the Kaggle API in Python

And you can see here in the URL kaggle.com is followed by this C.


How-to use the Kaggle API in Python

And this C essentially means that this is a competition.


How-to use the Kaggle API in Python

And we can also see playground prediction competition.


How-to use the Kaggle API in Python

Everything is telling us that this is a competition.


How-to use the Kaggle API in Python

And in this competition it comes with some data.


How-to use the Kaggle API in Python

Now this is different to a standalone dataset.


How-to use the Kaggle API in Python

And these standalone datasets can simply be uploaded by anyone.


How-to use the Kaggle API in Python

So if we go to sentiment 140 dataset here.


How-to use the Kaggle API in Python

You look in the URL and you can see that this dataset has been uploaded by Casanova.


How-to use the Kaggle API in Python

And there is a slightly different structure to the dataset page as well.


How-to use the Kaggle API in Python

And you can see here it's a dataset.


How-to use the Kaggle API in Python

First tab takes us to data.


How-to use the Kaggle API in Python

And we can scroll down and see the data that we can get here.


How-to use the Kaggle API in Python

So there are two different methods for downloading each one of these.


How-to use the Kaggle API in Python

We can't download competition datasets with the standalone dataset method.


How-to use the Kaggle API in Python

And we can't download standalone datasets with the competition dataset method.


How-to use the Kaggle API in Python

So we'll start with the competition dataset.


How-to use the Kaggle API in Python

And to download one of these all we need to do is use the competition download file method.


How-to use the Kaggle API in Python

And then we need to pass the competition name followed by the dataset.


How-to use the Kaggle API in Python

So head back over here.


How-to use the Kaggle API in Python

And you can see the competition name is this.


How-to use the Kaggle API in Python

And the data that we would like is train.tsv.zip.


How-to use the Kaggle API in Python

And that is downloaded into our current directory. You can see it here.


How-to use the Kaggle API in Python

Okay so that's how we download the competition datasets.


How-to use the Kaggle API in Python

We can also download the standalone datasets.


How-to use the Kaggle API in Python

To do so we use the dataset download file method.


How-to use the Kaggle API in Python

And then here we need to pass the username followed by the dataset name.


How-to use the Kaggle API in Python

So if we head over here.


How-to use the Kaggle API in Python

You can find both in the URL. So this one is Casanova slash Sentiment 140.


How-to use the Kaggle API in Python

We also need to specify the file name.


How-to use the Kaggle API in Python

Which in this case is this text here.


How-to use the Kaggle API in Python

And then we just execute that.


How-to use the Kaggle API in Python

And now we can see that we have downloaded both files here.


How-to use the Kaggle API in Python

Now you will notice that both of these files are actually zipped.


How-to use the Kaggle API in Python

So we can just quickly unzip them using Python.


How-to use the Kaggle API in Python

All we need to do is import zip file.


How-to use the Kaggle API in Python

And with zip file.


How-to use the Kaggle API in Python

We specify the path to the data.


How-to use the Kaggle API in Python

Which in this case is just the file name.


How-to use the Kaggle API in Python

And we specify that we are simply reading it.


How-to use the Kaggle API in Python

And then we simply call the extract all method.


How-to use the Kaggle API in Python

And we have our dataset here.


How-to use the Kaggle API in Python

And we see everything is in the right format.


How-to use the Kaggle API in Python

So that's everything for this tutorial on using the Kaggle API.


How-to use the Kaggle API in Python

If you have any questions, let me know in the comments below.


How-to use the Kaggle API in Python


API Series #2 - Building an API with Flask in Python

In this video, we are going to explore how we can put together a pretty simple API using Flask.


Now, what we're going to do is go through and build essentially what you can see here. So,


all of this code here is a single API. We're not going to go through all of it in this one video,


but we're going to go through the essentials and we're going to


step by step put an API together that is very similar to this.


Now, what this API is doing is pulling information from these two CSV files and either


presenting that information to us, that data, allowing us to modify, allowing us to add new entries


or allowing us to delete it. So, we get to use get, post, put and delete methods. Now, if you are


following along with this tutorial, you'll probably want this data so you can get it using


this code over here. So, over here we have, I'll make sure there's a link to this in the description


so you can get it. All we're doing is downloading the code or those two data files from here


and then storing it in a local directory. I mean, you can change that directory to whatever you


want, of course. Inside those two files, we have locations here, which is just a list of,


I think these are actually real coffee cafes, but I put this, I made this CSV quite a long time ago,


so I'm not 100% sure. Then over here, we just have some made up user data as well.


And the end result will be something like this. So, this is a program called Insomnia. I'll leave


a download link to that in the description, but it just allows us to send API requests very easily.


So, I'm going to send an API request to this address here, which is the API I just showed you.


And it's a user's endpoint. We will have two endpoints, which are essentially just separations


of the API. So, users and that's going to return all the user's data. And then we also have locations.


And we'll send that as well. And we return all the locations.


Now, we're going to start from scratch. And the first thing that we'll need to do is import


everything we need. So, we will need to import Flask. So, from Flask, import Flask.


We'll also need to import Flask RESTful. So, this is another Flask library that gives us a few


very useful tools for building an API. So, right from Flask RESTful, import resource API and rec


pass. Now, specific to what we're doing here, we will also want to import pandas and import AST.


Now, you will probably have to pip install these as well. So, the pip for those is pip install Flask


and Flask RESTful. Nothing weird there. All right. So, the first thing we need to do is initialize


our Flask app and then initialize our Flask API. So, we do that. So, this is typical Flask here.


So, we write Flask name and then the API is API equals API app. Okay. Super easy. Now,


we already touched on it, but our app is going to have two different endpoints.


So, we're going to have the users endpoints and we're also going to have the locations.


Endpoints. And we're going to go through building the users endpoint, but you'll be able to find the


code for both of those in the description. But the way that we separate both of these is we use a


class object. So, we're going to create our use class first. So, we write class users. And then


to initialize this as an endpoint for our API, we actually need to pass the resource object into it.


That will inherit the resource object and expose it to different like HTTP requests like get,


post and so on. For now, we're not going to add anything in there, but to make it easier,


we need to write API add resource. And then we need to specify the web address of this or the,


or the, it's almost like a page of a web address that the resource or endpoint will be.


So, we'll say users, which is our class and then the actual endpoint location, which is users.


Okay. So, this is saying I want you to map the class users here to this place in our API.


So, our web address. So, if for example, the user is a user, then we're going to pass the


API. So, our web address. So, if for example, the API was located at, let's say, API.com,


this resource would be located at API.com slash users. Okay. And we'll also create another one,


although we're not going to fill it out in this video, that will be called locations.


So, again, exactly the same thing. Yeah. Resource, we'll just pass for now and we'll copy that


and do the same for locations. Okay. Now, that's sort of the structure or the very high level


structure of our API itself. So, what I like to do is whilst we are building or writing the code


for our API, I like to also test it as we go along. I think it makes things a lot easier.


So, to run our API, we need to write this. If name is equal to main,


we write app.run. Okay. It's that easy. And now what we can do is just press the execute button up here.


Okay. Okay. So, we got this error because I, okay, I don't know why I did that. So, this is,


this needs to be the class, not a string of the class. Okay. So, now let's run that again.


It should work. Okay. So, we get this. This is a development server. Do not use it in production


environment. That's fine because we're just testing it. And then we can see this is running


at this address. Okay. So, if we take this, we'll copy it and we're going to take it over into


Insomnia. Okay. And I'll just clear everything that I was doing before. And we'll go into here.


Okay. And if we send this, we won't return anything yet. So, we'll just get this not found.


But if we go to users, we should at least return this internal server error. So, this means that,


okay, we are not getting anything back because there's an error on server side. So, in our API,


but there is something there. So, before when we just had this, so remember we have those two


endpoints. We're not actually using the base URL. So, there's not actually anything there.


We get this not found because there just isn't anything there. But if we type in the locations


or users, we will return this internal server error. Now, we're getting this because we haven't


written any code. So, it doesn't know what to do when we send that request. So, let's go back to


our code and we'll start writing something. So, first thing I want to do is we'll define a get


request. So, write find get bus self. And in here, what I want to do is load the users CSV. So,


to get that, I'm going to write users path.


So, this is the specific directory that I have stored my users data.


Of course, it might be different for you.


And what I'm going to do is down here, I'm going to use pandas to just read in that data. Whenever


I call this get request, I'm going to read it in. So, this is very similar to if we, for example,


had a database on a server somewhere. When the API gets a request, it would read that data quickly.


And return it to us. So, this is a very simple version of that. So, I'm going to do data equals


pd.read CSV. And we just go users path. And hopefully that's the right path. I think it is.


We can see. So, I come up to here. Yes, we have data. So, I guess that should be fine.


Yes, that should be fine. Users CSV. Cool. And then what we want to do, we can't pass a


we can't pass a pandas, a data frame object through an API. So, we need to convert this


into a dictionary. So, write data.toDate. Now, what we want to do is return. We want to return


the data. So, we're going to write data, which is equal to data, our dictionary. And we're going to


return the 200 code, which means the API request was successful. Here's your data. Okay, I'll save that.


And I believe it should, should update automatically. Let me check.


So, come over here. Let's send that again to users.


Okay, no. So, I think we need, we need to restart the API.


So, come over here.


Here, I'm just going to control C and run it again.


Now, you can, you can turn debug mode on. So, I think that will automatically reload everything.


So, to let's do that first. So, down here in app run, we're just going to set debug


equals true. Save that. I'm going to press control C down here and execute again.


And now, when we make changes, it should reload automatically.


So, come over here. Let's send that request again.


Now, we see user path is not defined. So,


see why that is wrong. So, it's user's path. So, I'll save that and that should reload automatically.


Let's try again. Okay, perfect. So, now we get a 200 response there. We get all of our users data.


Okay, so we're just returning that data frame in addition to the data frame.


Okay, cool. So, that's how get requests. Let's move on to, let's put requests, I think. So,


adding more data. Okay, so this time, what we're going to need to do is, so using post request,


I said put a second ago on and post. And in here, the first thing we need to do is,


we need to retrieve information from the user that they want to upload. So, the first thing


we need to think about here is we are trying to get data from the user. So, they are going to send


us like a new entry. So, what did we have before in our users?


We have user ID, name, city, locations. Okay, so we want to use it to be able to add a new


row. So, each one uses essentially a row in a data frame. So, we have locations, city, name,


and user ID. So, we're going to send this to the user. So, we're going to send this to the user.


Name and user ID. Now, we want the user to be able to add locations or location, locations,


name, and city. Now, I think the user ID, we don't really want the,


well the user isn't going to add that, that's going to be generated


automatically. So we ignore that, but we do want the user to be able to pass those


three things. Now to do that we're going to use the rec parser, so request pass


that we use up here. And to initialize that we need to write parser equals


request, sorry what is it up here? Rec pass, sorry, rec pass dot request parser here.


It initializes our parser and what this is going to do is when we add


arguments into the request this is going to read them and what we're


going to do is parse them out into variables. So we are going to, within


this request we want to allow those three, so location, name and city.


So write parser add argument, this one will be location and we're just going to


do one location at a time. So do location ID and then of course we use a put


request later if they want to add more locations. You say required is true,


they do need to add one of these and the type of this is a integer so we'll write


int. Okay we need to add another one I'm just going to copy this so let's copy


here, here and we have name and city. Name, city and the type is for both of those


strings which I think might be the default so we probably don't even need


to include that type string there but it is there so leave it. So now we have the


three arguments that I use you can use to post information to our API and what


we need to do to extract whatever the user has sent us is we do this we're


going to extract them into this dictionary here which is called args and


we write parser, parse args like that. Okay so now we have those arguments and


let's first just make sure this is working so I want to just I'm going to


return what we what we just tried to send that so let's return the location


I'm just going to put lock because args location ID, name and the city. Okay so let's just make sure


it's actually working and we'll set 200 save it and that should reload


automatically let's go to insomnia and we are sending a post request here so


post we have users we enter this question mark so this allows us to start


adding our parameters so location ID is going to be equal to I'm gonna say five


and so we we added the ampersand symbol there to add another parameter so


location ID and what are the other ones user user or name


James it's me and for the city oh let's put London okay send that cool and then


we we're just returning that request back to azure you notice here we put


lock and not location ID so we can tell it's not just returning what we sent


it's returning what we have processed through our function and return back to


ourselves okay cool so now we know we're passing those arguments correctly now


let's use them so again what I want to do is read our data so do data equals


data pandas dot read CSV again and we're reading the users path okay so users path


and we're doing users here so actually do we want location in there at all I


don't know


hmm


yeah maybe so now let's let's say the user can specify that user ID we're not


returning anymore so let's remove that so I want the user to I want to be able


to post this because then if we have a duplicate of a previous user ID we're


not we can say we can check if that it already exists in the data or not now


which we'll do now so right if ARG's user ID in the data that we just like so


in data is it user ID I think okay I think it's I want to return an error


message saying this already exists right message and all we do is write ARG's user


ID already exists and what will return here is the 409 code which indicates


that there's some sort of conflict and there is a conflict we already this user


ID already exists so we're saying there's conflict you can't you can't


create a new one it's already there if it doesn't already exist that's great


and we go ahead and we create that new user ID so what we do is write data


equals a data dot append then we're just appending the new data that we've


retrieved from our ARG's so it's going to be we need user ID which is going to be


ARG's user ID let me copy this make it a little bit quicker for locations it's


just empty for now city and we have name okay and then we need to I think ignore


yes ignore index true true sorry okay and then we'll save that data so to CSV


it is is it uses path and I'm going to set index false so we're not saving them


the index the row numbers and then I just want to return once we return the


data again we can't return a data frame we need to convert it into a dictionary


so write to date and it's a 200 response I'm gonna save that let's test it see if


it works so I'm going to let's see modify this to user ID now send that


internal server error so let's see a date frame no attribute to date so I


need to add an underscore save again and let's send that again okay so now we can


see we have more items in there the only thing is I've added five for all of them


oh did I it's in the code right what did I do yeah very smart okay so change that


name and this is city and one other thing is that these are all this should


also be a string one thing I notice and this should also be a string I'm gonna


leave it for now it's fine so let me send that again five already exists


because so at least we know that our conflict code works so let's create


another one number six I'm going to call this Jim let's go again okay so now


we've created number six as well to create gosh we create a few there it's


knowing oops so yeah now we've now we've created number six here which is Jim


lives in London if you use your ID is six okay so I mean formatting isn't


perfect but that's good for now we've created our first post request okay so


it's pretty annoying that we have those kind of messy entries that we added to


our datasets or data frame so what we really could do with having in our API


is a way to actually delete those so let's go ahead and build that so we need


to first define a delete function method and all we really need here is to say


okay which which entry do you want to delete so we don't need to have a name


or a city or anything we just want the user ID so we'll copy that into here so


we pass the input from the user and then we say okay and argument user ID but


that's all we need we don't need anything else and then and then what we


need to do is pass those as well so just like before parser pass orgs yeah there


we go so now we have so at this point the user has given us a user ID that


they want to delete and we need to go ahead and delete that so to delete it


first we need to say okay does it does it exist so we say if orgs user ID is


in our data so we need to load the data first so let me let me load it here okay


so if that is in data user ID then then we can go ahead and delete it and to do


that we were just going to use typical like pandas data frame logic so we just


write data equals data data user ID and we just want to select the rows that are


not equal to the user ID that we just pulled in so the orgs user ID right so


that's just going to return all of the rows except from the one that we've


specified okay so we deleted that now we just need to save that to file again to


CSV it is I keep forgetting the the variables users path and then indexes


false okay so we've opened file deleted the entry saved it back to CSV if we


have done all that we can return to user at 200 codes that will just return


let's return all the data again we don't need to do this every time we could just


return you know the statement saying hey you've deleted it everything so I'm just


going to return so we can see what we're actually doing in reality probably pretty


sure you don't want to wouldn't want to actually do this but I suppose it


depends maybe maybe you would who knows so return to 100 code because it's


successful now in the case that the user ID does not exist here we want to say


okay we want to check well we've already checked if it exists and then from there


we're just like okay it doesn't exist what are you doing so we just return we


say what did we use before when we sent a message message just message okay cool


message and the message is going to be user ID does not exist something like


those lines so orgs user ID mm-hmm does not exist let me format a little nicer


and then here we need to add the code which is suppose not found so for


zero four okay so I think that's it really so let's save it again it should


reload automatically head over to insomnia let's again so let's go to users


we're just going to look at what we have so get users what do we get we have all


these now we want to delete user IDs four and five okay these two are messy


we don't want those so we want to specify a user ID don't want four so we


change that to a delete request we don't want four let's send that see what


happens why why did that not change a data I didn't know change use ID is it


because because this needs to be a string I think okay I think yeah


probably let me try again send no so nothing okay so that took me far too


long to actually figure out what was wrong but there wasn't anything wrong


there there was the string issue so we did need to convert a verse to a string


to compare them but also so what I was doing here was reading the key value


here so the row number and thinking for as user ideas is not we have we have two


fives that we saw mess up and made those earlier so we we there is no for for us


to remove now we don't need to don't need to worry about it so let's test it


with five instead which are the the two messy ones that we need to remove so


let's send that and now you can see that we've actually removed those entries so


that's pretty good now the one thing I did also notice is if I try and send


that again we do get the 404 not found so that's that's pretty cool right we've


got everything that we would expect it now I think that's it for the code the


only other thing I want to show you was this other so this is a the actual full


API script here so in here I also included a another another method here


or another endpoint locations and we run that and we can mess around with the


cafe locations as well but that's everything so I think nothing else


want to cover in this video so we'll leave it there so thank you very much


Locality Sensitive Hashing (LSH) for Search with Shingling + MinHashing (Python)

Hi and welcome to the video. Today we're going to be covering another technique in similarity


search called Locality Sensitive Hashing or LSH. Now LSH is a hugely popular technique used in


efficient similarity search. Now there are a huge number of companies that use similarity search.


I mean you have big names like Google. I mean Google is built from similarity search. Then you


have Netflix, Amazon, Spotify. All of them are constantly recommending you different products,


films, music and they do that by comparing you to other customers. So they are performing a


similar search between you other customers and identifying the most similar ones. Now


you have two approaches. You have exhaustive which is comparing all of the data points. I'm just


going to call them vectors from now on because that's what we'll be using. So comparing all


these vectors and obviously it's slow. Approximate search allows us to approximate those vectors,


restrict our scope to a more relevant range of vectors and so on. So it cause a lot of different


techniques. It's not just one technique here. The one we're going to be covering today is


low-pass sensitive hashing. So at its core LSH is a hashing algorithm which attempts to


maximize hash collisions. So what we see on screen right now is a dictionary, like a typical


Python dictionary in the way that it hashes different items. So we have our keys which are


items that we're hashing. We process them through a hashing function and that hashing function


attempts to minimize hashing collisions e.g. to not put keys in the same bucket. It wants every


key to go to a separate bucket and then these are connected. Then they don't contain the values but


they're connected to the values that we relate back to our keys. So that's a Python dictionary.


That's our Python dictionary. But we're not wanting to minimize collisions. We are wanting


to maximize the collisions. So what we see here is a hashing function that maximizes those


collisions. So this is essentially what LSH is doing. So we are attempting to for any similar keys,


so these here and these here, they're all similar enough for us to want to put them


into the same buckets. So we've put two of them into here and then the other three into this


bucket. Now there are quite a few different ways of doing this and there are a lot of different


LSH methods. In fact, LSH is a very generic term that applies to a lot of different algorithms.


And the one that we will be covering is what I see as the traditional version. So it is the


original version of LSH. And what we'll be covering in this video is shingling, min hashing,


and that LSH function. So we'll get to understand why very soon. So here is the overview of the


process that we're going to be walking through. So we have shingling. So we have at the very start,


we have this text. So flying fish flew by the space station. Now that's just a string. And


what we want to do is extract all of the unique pairs of texts. So when we say shingling, it's


k shingling. And in this case, our k value is two because we're taking two characters at once. If


we were to take k equals four, for example, then we would take like pace and then move on, we take


ace and a space and so on. So that's the shingling. And from that, we create a set. So


if we have duplicate shingles, we remove those. So we just end up with one. So in this, I don't


know if we do have any duplicates, but say maybe down here, we have i n again, because we also have


it up here. We would end up with just a single i n in the set. You wouldn't have two. And then we


want to encode those. So that means we take a vocabulary from all of our texts and not just


this one sentence, but we'll have more than one sentence, obviously, that we're comparing. And


we'll use that to build a one vector from the vocab and our shingle set. Then we process that


through something called a min hash function, which produces this dense vector or signature.


So this thing down here, OK, that is what's called a signature.


And then we band that into this final bit here. This is our actual LSH process.


So we band that vector into multiple sub vectors and then we hash them. So where we find that we


have any two sub vectors go to the same hash bucket, then that means that the full vector


that they both come from is considered, or the two full vectors that they both come from are


considered a candidate pair. And we take those and we then calculate some other, we calculate


the similarity between them. OK, so first step in our process, like we discussed, is the shingling


operation. So shingling is simply where we take a window of length K characters and we simply move


that down through our text, like you can see here. And from that, we create the shingle set.


So in Python, what we do to shingle these three sentences we have here is we'll create a


shingle function here. And this is going to take some text, which is a string. And we're going to


say we're going to define the K values of the number of characters we take within each window,


which is obviously an integer. Now we initialize the value of the string.


Now we initialize our shingle set here. We'll make a string initially. And then what we do is


for i in range. And then here we want to go from the, or we want to go to the length of our text


minus K. So minus that window length plus one, because we want to go right up to the end of that.


And then here, all we do is shingle set dot append. And then we write,


so we have the text and we want to go from i up until i plus K.


Okay. That's our shingle list, I suppose. And then we want to return a set. So this will remove


any duplicates that we have. So shingle set. Okay. So that's our shingle function. And we


just want to process each one of our sentences through that. So we'll go A equals shingle A.


Also, we need to define K, which can be two. I'll just define K here.


Okay. And then let's have a look at what we have. And we see that we have this, it's shuffle.


There's no order to our set here. And we see that we have all of the pairs of words in there.


So we have S for the sort of the space part here or station actually could be either.


And if we try and find, okay, so here we have the very sorts of fly or flying.


You have the L Y there as well. Iron. So that's our shingle set. And with this,


we have all of our shingles. So the next step is to create our vocabulary, which is just


all of our shingles, our shingle sets, a union together. So to create that, all we do is


we go A union B dot union like that. You can see again, we have just a lot more text in there now,


or a lot more, many more shingles. That is our vocab. So now we have our shingle set and we have


our vocab. So we can tick both of those off. Now what we need to do is create our one encoding


over here. And the only other thing we need is a zero vector. So there's two, well, I mean,


there's more than two ways to do this, but I think there's two ways of thinking about it.


Normally the more efficient way would be to create a NumPy array full of zeros and then just


add the ones in where we have matches between our vocab and shingle set. But I'm not going to do


that. I'm just going to keep things like incredibly simple in the code that we're writing.


So I'm going to do A, it's one hot. Or the one thing we should do is make this a list because


we want order in our vocab and not have it shuffled. So what we do here is we say one for


x in A, or sorry, no, one if x is in A else zero for x in vocab. So what we're doing here is


what we're doing here is looping through the vocab and every single shingle within there,


we're saying if that exists in our signature, make that point in our list a one, otherwise


make it a zero. So that's simply our one hot encoding. So if we do ABC and then we have a look


at our A one hot, we see that we have this one hot encoded or this sparse array. Now,


min hashing is the next step in our process and it allows us to convert our what are currently


sparse vectors into dense vectors, which we call signatures. Now, what you see here is a run through


of how we do this for maybe one signature. We want to do that for multiple signatures though.


So we would actually run through this process multiple times. So what we're doing here is we're


creating a randomly permuted array, which counts from one to the length of our vocab. And then


what we are essentially doing, I know so in this web, we're basically shuffling it and then we're


counting through until we find the first alignment to one within our vector. In reality, you just


take all of your values and you find a minimum one that aligns to one. So that's if you're using


NumPy, which we'll see later on. I'll just show you the code, I'm not going to actually write all


of it though. So in code, that would look something like this. So we would start with a list, which is


the range from one to the length of our vocab. And if we have a look at that, we just say account.


We're going to shuffle that. So from random import shuffle.


And we just do it like this. So it modifies it in place. So we don't need to do anything there. So


let's view that.


Okay, so now we've shuffled that, shuffled it twice now, but that's fine.


And let's just loop through five of those. So four, we can look for more. Four i in range


from one to ten. What we're going to say is, I just want to print i, which aligns to the hash


sample index for that value. Okay, if we print that, we see so one, the value one, where is it?


Here is at index 85. Two is at 53 and so on. And essentially what we're doing here is saying,


loop through these, identify this index and this index in our one-hot vector. Does it align to a one?


And see that here, we find the first one at eight. And that means our signature value for this


point is for this min hash vector and our one-hot sparse vector here.


That signature value will be eight. And we repeat that for multiple min hash vectors.


Which is what you can see here. So if we were to work through this, so we saw one here,


that does not align to a one. So we work up to two and we find that it does align to a one. So that


is why we have this here. And then we go on to this one here. We find one and we work up to two.


We have this here. And then we go on to this one here. We find one does not align. Two,


still does not align. Three does not align. And four does align. So then we assign a four


in our min hash function. We go along and keep doing that to create our signature.


Okay. So I'm going to use these functions here. It's just what we wrote before, but put into a


cleaner format. And what I'm going to do is create 20 min hash vectors.


Run that. And then here we are going to run each of our one-hot sparse vectors


through our create hash function, which is here. And it's going to convert them into our signatures


as we described before. And we see here that we have also what did I mention. So here we have


20 min hash vectors, which means we have a length of 20 for each signature.


So what we see here are our dense vectors. And these are just compressed versions of our sparse


vectors. And we can check that that is true by we'll define a, we'll create a jaccard


similarity function. So we take, and here we take x and y both will be sets. And we just return the


length of the intersection between both of those. So the intersection between those


is divided by the union of both of those. So that is how you calculate jaccard similarity.


This should be a y.


Okay. And then if we do jaccard on both of those, so we have a sig, b sig.


These will have to be converted into sets. Forgot. So like that.


And then if we also take the jaccard for, I think it's just a and b, right?


So copy that. Okay. So we get, this is 0.6 and this is 1.4. Now, if we look up here, I think it's


a and b are not supposed to be very similar. So that's fine. And then b and c should be similar.


So if we swap this for c and then c here, we should both get higher values. And they should be


and they should be roughly in the same ballpark. I mean, they're not perfect because we're using a


very low number here. We're only using 20 values and typically use a lot more. But that's fine.


So you can see that they're both aligned, right? So despite converting these into the signature


vectors, it recognizes that they are very similar. And converting these into signature vectors,


it still recognizes that they are reasonably similar. So that's good. That's what we want.


Now the final step in our whole LHS process is the LHS function itself.


So this is essentially what it does. So we have our signature over here, which we built using the


steps that we just went through, which you can see here. And from that signature, we take a certain


number of equal length subvectors. So we define that using this here, this b. So b is 3. So that


means we split a signature into three different subvectors, which we see over here. And ideally,


what we want to be doing here is saying, okay, we process our subvectors each through a,


either a different hash function or it can be the same hash function, just as long as we use that


same hash function for the equivalent subvector in another signature, which you'll see in a moment,


it'll make sense. And once we have multiple signatures going together through those hash


functions, you can see here that they're equivalent on both sides, hash one, hash one here. These can


all just be a single hash function as well, which is what we're going to do. We're not really going


to use a hash function. And what we get here is three opportunities to identify these signatures


as being potential candidate pairs, which is where we consider it for further similarity


comparisons. In this case, hash threes both collide down here. So we say, okay, that means that a


and b are candidate pairs. I'm just going to put cand pairs. So this act of splitting our


signatures up into multiple subvectors just gives us more opportunities to identify similarities


because if we were to use the full vectors, the full vector would have to be very similar


for them to be put into the same hash bucket. With this, we only part of it to be very similar. So


increases the chances of us finding those similar signatures. So we're going to implement a very


simple version of this. I'm going to keep this very simple. Here we're just splitting our signature


vector. So we add our signature and b, which is the number of bands. And the first thing we do is


just make sure that our signature can be split into b bands equally. So where we take the remainder


after the division here, it must be equal to zero. And then we say we need to calculate the rows. So


the number of rows within each band, which obviously just the length of the signature divided by b.


And then we initialize a subvector array or list. And then we loop through and append subvectors.


Really simple, simple implementation. And let's apply that to b and c. So we have said


that we want 10 bands. So we only have 20 items or 20 numbers within our signature vectors. So


obviously we only get bands of two rows at a time. And we should find that at least one of those


match. So what we do is we loop through and we say if b rows equals c rows, break. Okay. And we


find very quickly that there is a candidate pair there. So that means that b and c, the full vectors


will be considered as a candidate pair. Let's do the same for a. And we should find, okay, so for


both a and b and a and c, it's not considered a candidate pair because there's just no similarity


there. So that's good. That's exactly what we wanted to happen. So we can now see that


that is our implementation of this. So the LSH, traditional LSH approach. Now a few other things


that we haven't covered, but we should just touch on quickly. And you can find, so there's an article


link in the description that which covers this. I walked through all of this and there will also be


a notebook where I'm getting these results from in the first place. So you can also look at that.


That includes the NumPy implementations of what we've just done, which is slightly more efficient,


although not super efficient because I want it to still be readable. So what we have here is


visualization shows the similarity, the cosine similarity of our signature vectors and whether


they were considered as candidate pairs or not. So these up here, these are our candidate pairs.


This is just a random sample. I think the actual full dataset is really big. So running this,


all of them is super inefficient because we're also running everything else through. So I can


just have the visualization here. But if you run just that on it, it does work. It's fine.


So at the top there, we have our candidates. At the bottom, we have our non-candidates.


We have some like, so we can see that high similarity does correlate with them being


classified as candidate pairs, which is good. It's obviously what we want. And there is this formula


that I did not write down, which I should have done, which is P equals one minus one minus S,


which is our similarity down here to the power of R, which is the number of rows in each band.


And all of this to the power of B, which is number of bands. Now that correlates to this


line here, this probability. Obviously it's P, capital P. So that's where it's coming from.


And if we run this with different similarity values, this is the pattern that we get.


And obviously that correlates, you can see, with whether something is classified as a candidate


pair or not. And what we can do is we can modify B to push the number of candidate pair


classifications either up or down. So here we have different B values. The side we have black,


which is 50. Then we go 25, 20, which is what we used before, and five. So let's say we found that


we're not identifying enough candidate pairs. We could push that down a little bit. Maybe we don't


do too much. So we could change B from 20 to 25. And if we do that, we see this. So in green, you


have our old results and our old probability line. And then in blue and pink, we have the new ones


again, or blue and magenta. So what we see here is we've pushed that down. So we've changed B to 25.


And now we're returning more results. So over here, we have these, for example, which we were


not returning before. And there are also more values in here as well. And there are less values


down here. So that's the result of us modifying B. So we can visualize that like so. So if we


increase B, we move it in this direction, which increases the number of candidate pairs,


which also increases the number of false positives that we're going to return. This line, by the way,


is our threshold. So similarity threshold is basically where we want the cutoff to be between


things being identified as candidate pairs and not candidate pairs. It's like our target almost.


Or if we wanted to reduce the number of candidate pairs, because maybe we're getting too many false


positives, we can push it this way, which will result in less candidate pairs, but also results


in more false negatives. So non candidate pairs where we should have candidate pairs. So it's just


a case of balancing both of those. But that's everything for this video. I hope it's been useful.


Sentence Similarity With Sentence-Transformers in Python

I welcome to this video on using the sentence transformers library to compare similarity


between different sentences. So this is going to be a pretty short video. I'm not going to go


really into depth. I'm just going to show you how to actually use the library. Now, if you do want


to go into a little more depth, I have another video that I'll be releasing just before this one.


And that will go into what is actually happening here, how we are calculating similarity or pulling


the how the BERT model that we'll be using is actually creating those embeddings and then how


we're actually calculating the similarity there. So if you're interested in that, go check it out.


Otherwise, if you just want to get a quick similarity score between two sentences,


this is probably the way to go. So we have these six sentences up here and this one,


three years later, the coffin was still full of jello. And this one, the person box was packed


with jelly many dozens of months later. They're saying the same thing, but the second one is


saying it in a way that most of us wouldn't normally say it. Instead of saying coffin,


we're saying person box instead of jello, we're saying jelly. I think that's kind of normal,


actually. And instead of years, we're saying dozens of months. So it's not really sharing


the same words, but we're going to see that we can actually find that these two sentences are


the most similar out of all of these. So we're taking those and we're going to be importing the


sentence transformers library. And we want to import the sentence transformer. And then from that,


we want to initialize a sentence transformer model. So we write sentence transformer.


And then in here, we're going to be using this model that I've already defined a model name,


which is a BERT base MLI mean tokens model. So initialize that. I need to rerun that.


So we have our model and I'll just show you really quickly. This model is coming from the


Hugging Face Transformers library behind sentence transformers. So this is the actual model we are


using. Now, first thing we do here is create our sentence vectors or sentence embeddings.


So we'll call this sentence Vets equals model and code. And all we need to do here is pass our


sentences. So we can pass a single sentence or a list of sentences. It's completely fine.


And then let's just have a quick look at what we have here. So you see that we have this big array.


And if we look at the shape, we see that we have a six by 768 array. So the six


refers to our six sentences here. And the 768 refers to the hidden state size within


the BERT model that we're using. So each one of these sentences is now being represented


by a dense vector containing 768 values. And that means that we already take those and


compare similarity between them. So to do that, we're going to be using the sklearn


implementation of cosine similarity, which we can import like this. So sklearn pairwise or metrics


pairwise. And we import cosine similarity.


And to calculate our cosine similarity, all we do is take that function. And inside here, we pass


our first sentence. So this three years later, the coffin is still full of jello. I want to pass


that sentence vector, which is just an index zero of our sentence Vets array.


And because we are extracting that single array value. So if we just have a look at this, you see


that we have a almost like a list of lists here. If we just extract this, we only get a list. So


what we want to do is actually keep that inside a list, otherwise we'll get dimension error.


And then we can also use the same method to extract the array. So we can extract the array


and then we do sentence Vets one onwards. So this will be the remaining sentences.


Okay, so let's take these or let's just bring them down here.


Calculate this. And we can see that our highest similarity by quite a bit is just 0.72.


Now, that means that between this sentence and this sentence, we have a similarity score of 0.72.


So clearly, it's working, it's scoring high similarity. And you can play around this and


and test multiple different words and sentences and just see how it works. But that's the easy


way of putting all this together. So I think it's really cool that we can do that so easily.


But I don't think there's really anything else to say about it. So


How to Index Q&A Data With Haystack and Elasticsearch

Okay so in this video what we're going to do is actually index our data so at the moment we just


have all of our paragraphs from Meditations by Marcus Aurelius and to do this we are going to


be using the Elasticsearch document store. So of course if we're using Elasticsearch we first need


to actually download and install it so I'm just going to take you through those steps now.


And all we need to do is head on over to this website up here


and it's elasticssearch.co and you can see the address just there. Now I'm going to follow the


instructions for Windows but of course if you're on Linux or Mac just follow through it's very


similar either way. So here we're going to install it on Windows using the MSI installer.


So just scroll down here and we can see we can download the package from this link so download


that and once you download it just open it and we'll see this window pop up. So once you see


this window pop up we just go through with all of the default settings. So install as a service


and continue through obviously if you do need to change anything change it but for me there's


nothing here that I want to modify. Notice here we have the HTTP port and we're using 9200 we'll


be using that later. We just continue through here default settings and then we click install


and we just let that install. Okay so now that we've installed Elasticsearch we can go ahead


and actually check that it's running. So to do that we're going to import Python requests


and whenever we interact with Elasticsearch it's either going to be through Haystack or it will be


through the request library and we'll just interact with the Elasticsearch API. So to check the health


of our cluster so essentially check that it's actually up and running. What we need to do is


send a GET request to localhost and if you remember earlier we had it was port 9200. Of course if the


port on yours was different modify it this is just the default value and after this we need to reach


out to the cluster endpoint and we are checking the health and then we'll just format that as a


JSON. So what you should see here is we have our cluster which is Elasticsearch. It may have a different name


if you modified it but by default it's Elasticsearch. The status is yellow which basically just means we


have one node up and running. You can have multiple nodes in Elasticsearch and for your cluster health


to be green it will expect your shards of indexes to have backup shards across different nodes and


obviously we can't do that if we only have one node but it's completely fine for us because we're


just in development. If you're in production yes you'd probably want it to have those backup shards.


If none of that made any sense don't worry about it we really don't need to know any of that for what we're doing here.


Now what we can also do is we can check if we have any indices already.


Now if I take a look at mine I will already have some indices set up which I've just set up prior


to recording this and to check that we go to localhost again


and this time we want to call the cat API which is what we would call whenever we want to see


data in a table human readable format rather than JSON and what we're checking here are the indices.


And we'll just add text on there so we can actually see that and this is quite messy so if we just


print it instead look a bit cleaner. Okay so you can see I have these two indices you shouldn't I


don't think have either of those no you won't have either of those so don't worry about that.


Now what we are going to do is create a new index which will be called Aurelius and that is where we


will put our documents. Now to actually implement that we will be going through the Haystack library


which you can pip install farm Haystack and what we want to do is from Haystack dot document store


elastic search import


elastic search document store. So this is our document store instance and of course


this is not aware of our elastic search instance we need to initialize that so


we'll store it in a variable called doc store


and all we write is elastic search document store.


Now we need to initialize it with the parameters so it knows where to connect to our elastic search


instance. So to do that we write host and this is local host. Now if you have a username and


password set which you don't by default you will need to enter them in here. I don't have any set so


no worries.


And then we also need to specify our index and at the moment we don't have an Aurelius index and


that's fine because this will initialize it for us. So we'll just call it Aurelius.


Now if we go down here we can see what it actually did so it sent a put request to here.


localhost 9200 Aurelius. So that's how you create a new index. After that what we want to do is


first import our data. So we have the data here which I got from this website


and process with this script which you can find on GitHub. I'll keep a link in the description so you


can just go and copy that if you need to. Now I haven't really done much pre-process it's pretty


straightforward and all you need to do here is actually open that data. So we do that with open


and from here that data file is located two folders up in a data folder it's called meditations.txt.


I'm going to be reading that


and all we do is data equals f.read


and then if we just have a quick look at the first 100 characters there we see that we have


this newline character and that signifies a new paragraph from the text. So what we want to do here


is split the data and then we can see that we have a newline character.


So what we want to do is split the data by newline


and then if we check the length of that see that we have 508 separate paragraphs in there.


So what we now want to do is we want to modify this data so that it's in the correct format


for Haystack and Elasticsearch. So that format looks like this so it expects a list of dictionaries


where each dictionary looks like this from the text and inside here we would have our paragraph.


So each one of these items here and then there's another optional field called meta


and meta contains a dictionary and in here we can put whatever we want.


So for us I don't think at the moment there's really that much to put into here other than


where it came from so the book or maybe the source is probably a better word to use here


and all of these are coming from Meditations. Now later on we will probably add a few other


books as well and then the source will be different and when we return that item from


our retriever and our reader we'll at least be able to see which book came from him.


It would also be pretty cool to maybe include like a page number or something but at the moment with


this there are no page numbers included so we're not doing that at the moment.


So that's the format that we need and it's going to be a list of these.


So to do that we'll just do some list comprehension.


So we're going to write this and let's just copy this I think yeah it should be fine we'll copy this


and just indent that and in here we have our paragraph and sources Meditations for all of them


and then we just write for paragraph in and data okay so yeah that should work and if we just


and if we just check what we have here okay so that's that's what we want so we have text


we have the paragraph and then in here we have this meta with a source which is always Meditations


at the moment so that looks pretty good and we'll just double check the length again it should be


508 okay perfect now what we need to do is index all of these documents into our Elasticsearch


instance and to do that it's super easy all we do is call docstore because we're doing this through


Haystack now and we do write documents and we just pass in our data.json and that should work.


Okay cool so we can see here what it's done as it's sent a POST request to the Bulk API and sent


two of them I assume because it can only send so many documents at once so that's pretty cool and


now what I want to check is that we actually have 508 documents in our Elasticsearch instance


so to do that we're going to revert back to requests so we'll do requests.get


again go to our localhost


9200 and here we need to specify the index that we want to count the number of entries in


and then all we do is add count onto the end there and this will return a JSON object so we do this


so that we can see it and sure enough we have 508 items in that document store.


So if we head on back to our original plan so up here we had meditations we've now got that


and we've also set up the first part of our sack over here so Elastic now has meditations in there


so we can cross that off now the next step is setting up our retriever which we'll cover in the


Training BERT #5 - Training With BertForPretraining

Hi, welcome to the video. Here we're going to have a look at how we can pre-train BERT.


So what I mean by pre-train is fine-tune BERT using the same approaches that are used to actually


pre-train BERT itself. So we would use these when we want to teach BERT to better understand the


style of language in our specific use cases. So we'll jump straight into it, but what we're going


to see is essentially two different methods applied together. So when we're pre-training,


we're using something called mass language modeling or MLM and also net sentence prediction or NSP.


Now in a few previous videos, I've covered all of these. So if you do want to go into a little more


depth, then I would definitely recommend having a look at those. But in this video, we're just


going to go straight into actually training a BERT model using both of those methods using


the pre-training class. So we need first to import everything that we need. So I'm going to


import requests because I'm going to use request download data we're using, which is from here. You


find a link in the description for that. And we also need to import our tokenizer and model classes


from transformers. So from transformers, we're going to import BERT tokenizer


and also BERT for pre-training. Now, like I said before, this BERT for pre-training class contains


both an MLM head and an NSP head. So once we have that, we also need to import torch as well. So let


me import torch. Once we have that, we can initialize our tokenizer and model. So we initialize


our tokenizer like this. So BERT tokenizer and from pre-trained. And we're going to be using the


BERT base uncased model. Obviously, you can use whichever BERT model you'd like. And for our model,


we have the BERT for pre-training class. So that's our tokenizer model. Now let's get our data.


Don't need to worry about that warning. It's just telling us that we need to train it, basically,


if we want to use it for inference predictions. So we get our data. We're going to pull it from


here. So let me copy that. And it's just requests.get. And paste that in there. And we should


see a 200 code. That's good. And so we just extracted data using the text attribute.


So text equals that. We also need to split it because it's a set of paragraphs that are split


by a new line character. And we can see those in here. Now we need to power data both for NSP and


MLM. So we'll go with NSP first. And to do that, we need to create a set of random sentences. So


sentence A and B. And then we need to create a set of random sentences. So we need to create a set


of random sentences. So sentence A and B, where the sentence B is not related to sentence A. We


need roughly 50% of those. And then the other 50% we want it to be sentence A is followed by sentence


B. So they are more coherent. So we're basically teaching BERT to distinguish between coherence


between sentences. So like long term dependencies. And we just want to be aware that within our text,


so we have this one paragraph that has multiple sentences. So we split by this. We have those. So


we need to create essentially a list of all of the different sentences that we have that we can just


pull from when we're creating our training data for NSP. Now to do that, we're going to


use this comprehension here. And what we do is write sentence. So for each sentence,


for each paragraph in the text. So this variable.


For sentence in para.split. So this is where we're getting our sentence variable from.


And we just want to be aware of if we have a look at this one, we see we get this empty sentence,


we get that for all of our paragraphs. So we want to not include those. So we say if


sentence is not equal to that empty sentence. And we're also going to need to get the length


of that bag for later as well. And now what we do is create our NSP training data.


So we want that 50-50 split. So we're going to use the random library to create that 50-50 randomness.


We want to initialize a list of sentence a's, a list of sentence b's,


and also a list of labels. And then what we do is we're going to loop through each paragraph in


our text. So for paragraph in text. We want to extract each sentence from the paragraph. So


we're going to use it similar to what we've done here. So write sentences. This is going to be a


list of all the sentences within each paragraph. So sentence for sentence in para.split


by a period character. And we also want to make sure we're not including those empty ones. So if


sentence is not equal to empty, then once we're there, what we want to do is we want to get the


number of sentences within each sentence or sentences variable. So just get length.


And the reason we do that is because we want to check that a couple of times in the next few


lines of code. And first time we check that is now. So we check that the number of sentences is


greater than one. Now this because we're concatenating two sentences to create our


training data, we don't want to get just one sentence. We need it where we have, for example,


in this one, we have multiple sentences so that we can select like this sentence followed by this


sentence. We can't do that with these because there's no guarantee that this paragraph here


is going to be talking about the same topic as this paragraph here. So we just avoid that.


And in here, first thing we want to do is set out start sentence. So this is where


sentence A is going to come from. And we're going to randomly select,


say for this example, we want to randomly select any of the first one, two, three sentences.


Okay, we'd want to select any of these three, but not this one, because if this sentence A,


we don't have a sentence B which follows it to extract.


So we write random, randint 0 up to the length of num sentences minus two. Now we can now get


our sentence A, which is append, and we just write sentences start. And then for our sentence B,


50% we want to select random one from bag up here, 50% of time we want to select the genuine


next sentence. So say if random.random, so this will select a random float between 0 and 1,


it's greater than 0.5. And sentence B is going to be, we'll make this our coherent version. So


sentences start plus one. And that means our label will have to be zero because that means


that these two sentences are coherent. Sentence B does follow sentence A. Otherwise,


we select a random sentence for sentence B. So do append, and here we would write bag,


and we need to select a random one. So we do random, same as we did earlier on for the start,


we do random, randint from zero to the length of the bag size minus one. So we also need to do the


label, which is going to be one in this case. We can execute that. Now that will work. I go a


little more into depth on this in the previous NSP video. So I'll leave a link to that in the


description if you want to go through it. And now what we can do is tokenize our data. So to do that,


we just write inputs and we use a tokenizer. So this is just normal, you know,


hugging face transformers. And we just write sentence A and sentence B. So hugging face


transformers will know what we want to do with that. It will deal with formatting for us,


which is pretty useful. We want to return PyTorch tensors. So return tensors equals pt.


And we need to set everything to a max length of 512 tokens. So max length equals 512.


The truncation needs to be set to true. And we also need to set padding equal to max length.


Okay. So that creates three different tensors for us.


Impart IDs, token type IDs, and attention mask. Now for the pre-trained model, we need two more


tensors. We need our next sentence label tensor. So to create that, we write inputs,


next sentence label. And that needs to be a long tensor


containing our labels, which we created before in the correct dimensionality. So that's why we're


using the list here and the transpose. And we can have a look at what that creates as well.


So look at the first 10. We get that. Okay. And now what we want to do is create our mask data. So


we need the labels for our mask first. So when we do this, what we'll do is we're going to clone


the input IDs tensor. We're going to use that clone for the labels tensor. And then we're going to go


back to our input IDs and mask around 15% of the tokens in that tensor. So let's create that labels


tensor. It's going to be equal to inputs, input IDs, detach, and clone. Okay. So now we've got


our mask data. Okay. So now we'll see in here, we have all of the tensors we need, but we still need


to mask around 15% of these before moving on to training our model. And to do that, we'll


use, we'll create a random array using the torch rend. That needs to be in the same shape as our


input IDs. And that will just create a big tensor between values of zero to one. And what we want


to do is mask around 15% of those. So we write something like this. Okay. And that will give us


our mask here, but we also don't want to mask special tokens, which we are doing here. We're


masking classification tokens and also masking padding tokens up here. So we need to add a little


bit more logic to that. So let me just add this to a variable. So we add that logic, which says,


and input IDs is not equal to one zero one, which is our CLS token, which is what we get down here.


See the impact. See we get faults now. And we also want to do the same for our separator tokens,


which is one zero two. We can't see any of those. And our padding tokens, we use zero.


So you see these are all that will go false now, like so. So that's our masking array.


And now what we want to do is loop through all of these, extract the points at which they are not


false. So where we have the mask and use those indices values to mask our actual input IDs up


here. To do that, we go for i in range inputs, input IDs dot shape zero. This is like iterating


through each row. And what we do here is we get selection. So these are the indices where we have


true values and mask array. And we do that using torch flatten mask array at the given index,


where they are non-zero. And we want to create a list from that. Okay. So we have that. Oh, and so


I want to show you what the selection looks like quickly. So it's just a selection of indices to


mask. And we want to apply that to our inputs, input IDs. So at the current index, and we


select those specific items and we set them equal to one zero three, which is the masking token ID.


Okay. So that's our masking. And now what we need to do is we need to take all of our data here and


load it into a PyTorch data loader. And to do that, we need to reform our data into a PyTorch


data set object. And we do that here. So main thing to note is we pass our data into this


initialization that assigns them to this self encodings attribute. And then here we say, okay,


given a certain index, we want to extract the tensors in a dictionary format for that index.


And then here we're just passing the lengths of how many tensors or how many samples we have in


the full data set. So run that. We initialize our data sets using that class. So right, data set


equals meditations data set, pass our data in there, which is inputs. And then with that,


we can create our data loader like this. So torch utils data data loader. And we have data set.


Okay. So that's ready. Now we need to set up our training loop. So first thing we need to do is


check if we are on GPU or not. If we are, we use it and we do that like so. So device equals torch


device cuda if torch cuda is available. Else torch device CPU. So that's saying use the GPU


if we have a cuda enabled GPU, otherwise use CPU. And then what we want to do is move our model


over to that device. And we also want to activate the training mode of our model.


And then we need to initialize our optimizer. I'm going to be using Adam with weighted decay.


So from transformers import Adam w. And initialize it like this. So optim equals Adam w.


We pass our model parameters to that. And we also pass a learning rate. So learning rate


is going to be 5e to the minus 5. Okay. And now we can create our training loop.


So you're going to use TQDM to create the progress bar. And we're going to go through


two epochs. So for epoch in range two, we initialize our loop by wrapping it within TQDM.


And in here we have our data loader. And we set leave equal to true so that we can see that progress


bar. And then we loop through each batch within that loop. Up here, so I didn't actually set the


batches. My mistake. So up here we want to set where we initialize the data loader. We want to


set batch size equal to 16. And also shuffle the data set as well. Okay. So for batch in loop,


here we want to initialize the gradient on our optimizer. And then we need to load in


each of our tensors, which there are quite a few of them. So we have inputs.keys. We need to load


in each one of these. So input IDs equals batch. We access this like a dictionary. So input IDs.


We also want to move each one of those tensors that we're using to our device.


So we do that for each one of those. And we have attention mask.


And next sentence labels and also labels.


Labels and also labels. Okay. And now we can actually process that through our model.


So in here, we just need to pass all of these tensors that we have. So input IDs.


And then we have token type IDs. Just copy this.


Attention mask.


Next sentence label.


And labels.


Okay. So that's quite a lot going into our model. And now what we want to do is extract the loss


from that. Then we calculate loss for every parameter in our model. And then using that,


we can update our gradients using our optimizer. And then what we want to do is print the relevant


info to our progress bar that we set up using TQDM and loop. So loop set description.


And here I was going to put the epoch info. So the epoch we're currently on.


And then I also want to set the post fix.


Which will contain the loss information. So loss.item. Okay. We can run that.


And you see that our model is now training. So we're now training a model using both


assignment modeling and net sentence prediction. And we haven't needed to take any structured data


to set up the model. So we can just run that. And we can see that our model is now training.


We haven't needed to take any structured data. We've just taken a book and pulled all data and


formatted it in the correct way for us to actually train a better model, which I think is really


Q&A Document Retrieval With DPR

Okay, so in the previous video what we did was set up our Elasticsearch document store to contain all of our


Q&A Document Retrieval With DPR

paragraphs from meditations


Q&A Document Retrieval With DPR

so we did that in this script here and


Q&A Document Retrieval With DPR

All together we only have not that much data, 508 paragraphs or documents within our document store


Q&A Document Retrieval With DPR

so


Q&A Document Retrieval With DPR

What we now want to do is set up the next part of our


Q&A Document Retrieval With DPR

Retriever reader stack, which is the retriever and


Q&A Document Retrieval With DPR

What the retriever will do is


Q&A Document Retrieval With DPR

given a query it will communicate with our Elasticsearch document store and


Q&A Document Retrieval With DPR

return a


Q&A Document Retrieval With DPR

Certain number of contexts which are the paragraphs in our case that it thinks are most relevant to our query


Q&A Document Retrieval With DPR

So


Q&A Document Retrieval With DPR

That's what we are going to be doing here and


Q&A Document Retrieval With DPR

The first thing that we need to do is initialize our document store again, so I'm just going to copy these


Q&A Document Retrieval With DPR

and