Unicode Normalization for NLP in Python

normal text and we're comparing it to someone who's got hello world in some weird text with


Unicode Normalization for NLP in Python

circles around every letter, we can't actually compare them like for like because our models


Unicode Normalization for NLP in Python

or code in general is not going to be able to compare those two different Unicode character


Unicode Normalization for NLP in Python

sets.


Unicode Normalization for NLP in Python

And the issue with diacritics is that those characters always have this hidden property


Unicode Normalization for NLP in Python

in that we have one Unicode character, which is the capital C with Cedilla.


Unicode Normalization for NLP in Python

But then we have an identical set of characters, which is, for example, the Latin capital C


Unicode Normalization for NLP in Python

immediately followed by something called the combining Cedilla character.


Unicode Normalization for NLP in Python

And they together look exactly like the other Unicode character.


Unicode Normalization for NLP in Python

And this is quite difficult to deal with.


Unicode Normalization for NLP in Python

So we have these two problems and we use Unicode normalization to actually deal with those


Unicode Normalization for NLP in Python

when we're building NLP models.


Unicode Normalization for NLP in Python

So I kind of said there's two forms of equivalent characters that are not really equivalent.


Unicode Normalization for NLP in Python

The first of those is the compatibility equivalence.


Unicode Normalization for NLP in Python

That's where we have stuff like font variants.


Unicode Normalization for NLP in Python

We have different line break sequences, circled variants, superscripts, subscripts, fractions


Unicode Normalization for NLP in Python

and a few other things as well.


Unicode Normalization for NLP in Python

Now we want our model to see both hello world with those weird circles and also just hello


Unicode Normalization for NLP in Python

world as one because that's how we read it and that's how it's supposed to be interpreted.


Unicode Normalization for NLP in Python

And that is what the compatibility equivalence is for.


Unicode Normalization for NLP in Python

And we'll look at how we actually deal with that pretty soon.


Unicode Normalization for NLP in Python

And then we also have the canonical equivalence, which is the thing with the accents and the


Unicode Normalization for NLP in Python

glyphs I mentioned before.


Unicode Normalization for NLP in Python

So you have a few different reasons for that.


Unicode Normalization for NLP in Python

But two that I think are most relevant is where you have the combined characters.


Unicode Normalization for NLP in Python

So we have that C with cedilla character and then we also have the capital C plus the combining


Unicode Normalization for NLP in Python

cedilla characters merged together.


Unicode Normalization for NLP in Python

And then we also have conjoined creating characters, which I think are pretty common as well.


Unicode Normalization for NLP in Python

Canonical equivalence is much more to do with characters that we can't really see that they


Unicode Normalization for NLP in Python

are different, but they are in fact different.


Unicode Normalization for NLP in Python

Whereas compatibility equivalence is more to do with the purpose that made them different


Unicode Normalization for NLP in Python

and in reality their meaning is the same.


Unicode Normalization for NLP in Python

So we have two different directions for how we can transform our text between these two


Unicode Normalization for NLP in Python

different forms.


Unicode Normalization for NLP in Python

So we have decomposition, which is breaking down Unicode characters into smaller parts


Unicode Normalization for NLP in Python

or more normal parts.


Unicode Normalization for NLP in Python

And then we have composition, which is taking multiple Unicode characters and merging them


Unicode Normalization for NLP in Python

into a single accepted Unicode character.


Unicode Normalization for NLP in Python

So I've got this example here.


Unicode Normalization for NLP in Python

So this U00C7, if we take a look here, this is our C with cedilla.


Unicode Normalization for NLP in Python

And we can see here this is what it looks like.


Unicode Normalization for NLP in Python

It has this C and it's got a little cedilla at the bottom.


Unicode Normalization for NLP in Python

And then the other side, we have these two characters here.


Unicode Normalization for NLP in Python

And if we just take a look here, we can see, okay, this is the C plus cedilla.


Unicode Normalization for NLP in Python

So these are two separate Unicode characters.


Unicode Normalization for NLP in Python

And then we see, okay, they actually look exactly the same again.


Unicode Normalization for NLP in Python

And obviously that's where our problem is.


Unicode Normalization for NLP in Python

So what we can do is we can decompose them into their different parts.


Unicode Normalization for NLP in Python

Now these are already separated.


Unicode Normalization for NLP in Python

So when we decompose them, we just get the same thing again.


Unicode Normalization for NLP in Python

Whereas for our C with cedilla character, we decompose that and we basically get these


Unicode Normalization for NLP in Python

two different parts, which is the Latin capital C and the combining cedilla character.


Unicode Normalization for NLP in Python

And then we can perform canonical composition to put those both together and merge them


Unicode Normalization for NLP in Python

back into the capital C with cedilla.


Unicode Normalization for NLP in Python

And that's essentially how decomposition and composition works.


Unicode Normalization for NLP in Python

Obviously it's slightly different for the compatibility decomposition, but we'll talk


Unicode Normalization for NLP in Python

about that quite soon.


Unicode Normalization for NLP in Python

When we take the fact that we have these two different directions, composition, decomposition,


Unicode Normalization for NLP in Python

and we have our two types of transformations, which is compatibility and canonical equivalence,


Unicode Normalization for NLP in Python

we get these four forms.


Unicode Normalization for NLP in Python

So we have form D, which is canonical decomposition, which is what I showed you here, where we're


Unicode Normalization for NLP in Python

decomposing those characters into its individual parts.


Unicode Normalization for NLP in Python

And if we just take a look at how to actually do this in Python.


Unicode Normalization for NLP in Python

So we'll take this Unicode here and we'll just place it here.


Unicode Normalization for NLP in Python

This is our C with cedilla character.


Unicode Normalization for NLP in Python

So we just print that out.


Unicode Normalization for NLP in Python

We see we have that character.


Unicode Normalization for NLP in Python

Now the other one is where it's kind of both together.


Unicode Normalization for NLP in Python

So I'm just going to call that C plus cedilla.


Unicode Normalization for NLP in Python

And that is the Latin capital C, which is 0043, which if I just print this out so we


Unicode Normalization for NLP in Python

can just see it before we put the cedilla on the end, we just have a C.


Unicode Normalization for NLP in Python

And then for the cedilla, we just put 0327 and we get that.


Unicode Normalization for NLP in Python

And obviously these look the same, but if we compare them, we'll see that they are not


Unicode Normalization for NLP in Python

the same.


Unicode Normalization for NLP in Python

We get faults.


Unicode Normalization for NLP in Python

So to deal with that, this is where we need to use our canonical decomposition or NFD,


Unicode Normalization for NLP in Python

that we can see here.


Unicode Normalization for NLP in Python

So to do all this, we're going to need to import the Unicode data library.


Unicode Normalization for NLP in Python

And then we use Unicode data normalization.


Unicode Normalization for NLP in Python

In this case, we're using NFD, which is canonical decomposition.


Unicode Normalization for NLP in Python

And then what we want to do is pass in our C with cedilla, because we're going to want


Unicode Normalization for NLP in Python

to break this down into the two different parts.


Unicode Normalization for NLP in Python

So that's the one that we need to transform.


Unicode Normalization for NLP in Python

And then the other side, we're going to have our C plus cedilla, which is our two characters.


Unicode Normalization for NLP in Python

And we'll see, if we just change this to normalize, now we have true.


Unicode Normalization for NLP in Python

So now what we've done is converted the single character into the two separate characters


Unicode Normalization for NLP in Python

here.


Unicode Normalization for NLP in Python

And that is because we've used normal form decompositions, we've decomposed those, we've


Unicode Normalization for NLP in Python

wrote them apart.


Unicode Normalization for NLP in Python

Now, on the other side of that, we have the canonical composition, where we build them


Unicode Normalization for NLP in Python

back up into one.


Unicode Normalization for NLP in Python

And to do that, we use NFC.


Unicode Normalization for NLP in Python

And obviously, if we try it with this, we're not going to get the right answer, because


Unicode Normalization for NLP in Python

we're not going to find that they match, because we're compositioning this back into itself.


Unicode Normalization for NLP in Python

So it's just going to be this again, against this, which are separate.


Unicode Normalization for NLP in Python

So we actually need to switch which side we have this function on.


Unicode Normalization for NLP in Python

So if I just remove this, and copy this across, and we'll see that now we get true, because


Unicode Normalization for NLP in Python

what we've done is converted these into this.


Unicode Normalization for NLP in Python

And that's how we normalize for canonical equivalence, which is essentially where we


Unicode Normalization for NLP in Python

can't actually see the difference.


Unicode Normalization for NLP in Python

On the other side, we have where people are using the weird text.


Unicode Normalization for NLP in Python

So in our abbreviations, we have these two with the K.


Unicode Normalization for NLP in Python

And that K means compatibility.


Unicode Normalization for NLP in Python

Where there isn't a K, that means we're using the canonical equivalence.


Unicode Normalization for NLP in Python

Where there is a K, we're using the compatibility equivalence.


Unicode Normalization for NLP in Python

Now the first of those is normal form KD, which is compatibility decomposition.


Unicode Normalization for NLP in Python

Now this breaks down the fancy or alternative characters into their smaller parts, if they


Unicode Normalization for NLP in Python

do have smaller parts.


Unicode Normalization for NLP in Python

So for example, fractions, if we have the one over two fraction, that will get broken


Unicode Normalization for NLP in Python

down into the numbers one and two, and also a fraction slash character, which you can


Unicode Normalization for NLP in Python

actually see down here.


Unicode Normalization for NLP in Python

And we also have our fancy characters.


Unicode Normalization for NLP in Python

So where we have this fancy capital H, and we decompose it into just a normal Latin capital


Unicode Normalization for NLP in Python

letter H.


Unicode Normalization for NLP in Python

And that's how the compatibility decomposition works.


Unicode Normalization for NLP in Python

And to apply that, we want to use NFKD.


Unicode Normalization for NLP in Python

So if we just take what we have here, and we're just going to switch what we're actually


Unicode Normalization for NLP in Python

using.


Unicode Normalization for NLP in Python

So I'm going to switch out the su siddilla for this fancy H.


Unicode Normalization for NLP in Python

So your fancy H.


Unicode Normalization for NLP in Python

In fact, we can just leave it like that because we can at least see what we're doing now.


Unicode Normalization for NLP in Python

So we're going to put that here.


Unicode Normalization for NLP in Python

And we want to compare that to just a normal letter H.


Unicode Normalization for NLP in Python

Obviously it's false, doesn't match.


Unicode Normalization for NLP in Python

What we need to do is normalize this and decompose it into the capital H character.


Unicode Normalization for NLP in Python

So let's take this.


Unicode Normalization for NLP in Python

And we're going to use our normalize function again.


Unicode Normalization for NLP in Python

But this time, we want to use compatibility equivalence, which is the K, and we're decomposing


Unicode Normalization for NLP in Python

it using D.


Unicode Normalization for NLP in Python

And now we can see that we are getting true.


Unicode Normalization for NLP in Python

So we just print out the results of this function.


Unicode Normalization for NLP in Python

You can see, okay, great.


Unicode Normalization for NLP in Python

It's just taking that H and converting it into something normal.


Unicode Normalization for NLP in Python

And then that leads us on to our final normal form, which is normal form at KC.


Unicode Normalization for NLP in Python

So normal form KC consists of two sets.


Unicode Normalization for NLP in Python

We have the compatibility decomposition, which is what we've just done.


Unicode Normalization for NLP in Python

And then there's a second set, which is a canonical composition.


Unicode Normalization for NLP in Python

So we're building that back up, those different parts, canonically.


Unicode Normalization for NLP in Python

This allows us to normalize all variants of a given character into a single shared form.


Unicode Normalization for NLP in Python

So for example, with our fancy H, we can add the combining Cedilla to that in order to


Unicode Normalization for NLP in Python

just make this horrible monstrosity of a character.


Unicode Normalization for NLP in Python

And we would write that out as we have H here.


Unicode Normalization for NLP in Python

So we can just put that straight in.


Unicode Normalization for NLP in Python

And then we can just come up here and get our Cedilla Unicode and put that in.


Unicode Normalization for NLP in Python

And if we put those together, we get this weird character.


Unicode Normalization for NLP in Python

Now, if we wanted to compare that to another character, which is the H with Cedilla, which


Unicode Normalization for NLP in Python

is a single Unicode character, we're going to have some issues because this is just one


Unicode Normalization for NLP in Python

character.


Unicode Normalization for NLP in Python

So if we use NFKD, we can give it a go.


Unicode Normalization for NLP in Python

So we'll add this in.


Unicode Normalization for NLP in Python

Let's try and compare it to this.


Unicode Normalization for NLP in Python

Okay, we get false.


Unicode Normalization for NLP in Python

That's because this is breaking this down into two different parts.


Unicode Normalization for NLP in Python

So a H and this combining Cedilla.


Unicode Normalization for NLP in Python

So if I just remove this and print out, you see, okay, they look the same, but they're


Unicode Normalization for NLP in Python

not the same because we have those two characters again.


Unicode Normalization for NLP in Python

So this is where we need canonical composition to bring those together into a single character.


Unicode Normalization for NLP in Python

So that looks like this.


Unicode Normalization for NLP in Python

So we have, initially, we have our compatibility decomposition.


Unicode Normalization for NLP in Python

If we go across, we have this final work, which is the canonical composition.


Unicode Normalization for NLP in Python

And this is the NFKC normal form.


Unicode Normalization for NLP in Python

So normal form KC.


Unicode Normalization for NLP in Python

And to apply that, all we need to do is, obviously, adjust this to KC.


Unicode Normalization for NLP in Python

And, okay, we run that.


Unicode Normalization for NLP in Python

We seem to get the same result.


Unicode Normalization for NLP in Python

And then if we add this, we can see, okay, now we're getting what we need.


Unicode Normalization for NLP in Python

And in reality, I think for most cases, or almost all that I can think of anyway, you're


Unicode Normalization for NLP in Python

going to use this NFKC to normalize your text.


Unicode Normalization for NLP in Python

Because this is going to provide you with the cleanest, simplest dataset that is the


Unicode Normalization for NLP in Python

most normalized.


Unicode Normalization for NLP in Python

So when going forward with your language models, this is definitely the form that I would go


Unicode Normalization for NLP in Python

with.


Unicode Normalization for NLP in Python

Now, of course, you can mix it up.


Unicode Normalization for NLP in Python

You can use different ones.


Unicode Normalization for NLP in Python

But I would definitely recommend, if this is quite confusing and hard to get a grasp


Unicode Normalization for NLP in Python

of, just taking these Unicode characters, playing around them a little bit, applying


Unicode Normalization for NLP in Python

these normal form functions to them and just seeing what happens.


Unicode Normalization for NLP in Python

And I think it will probably click quite quickly.


Unicode Normalization for NLP in Python

So that's it for this video.


Unicode Normalization for NLP in Python

I hope it's been useful and you've enjoyed it.


Unicode Normalization for NLP in Python

So thank you for watching.


Unicode Normalization for NLP in Python


Sentence Similarity With Transformers and PyTorch (Python)

Today we're going to have a look at how we can use transformers like BERT to create embeddings for sentences


and how we can then take those sentence vectors and use them to calculate the semantic similarity between different sentences.


So at a high level, what you can see on the screen right now is a BERT base model.


Inside BERT base we have multiple encoders and at the bottom we can see we have our tokenized text.


We have 512 tokens here and they get passed into our first encoder to create these hidden state vectors,


which are of the size 768 in BERT.


Now these get processed through multiple encoders and between every one of these encoders, there's 12 in total,


there are going to be a vector of size 768 for every single token that we have.


So 512 tokens in this case.


Now what we're going to do is take the final tensor out here, so this last hidden state tensor,


and we're going to use mean pooling to compress it into a 768 by 1 vector.


And that is our sentence vector.


Then once we've built our sentence vector, we're going to use cosine similarity to compare different sentences


and see if we can get something that works.


So switching across to Python, these are the sentences we're going to be comparing and there's two.


So there's this one here, which is three years later the coffin was still full of jello.


And that has the same meaning as this here.


I just rewrote it, but with completely different words.


So I don't think there's really any words here that match.


So in years we have dozens of months, jelly, jello, coffin, person box.


No normal human would even say that second.


Well, no normal human would probably say either of those.


But we definitely wouldn't use person box for coffin and many dozens of months for years.


So it's reasonably complicated, but we'll see that this should work for similarity.


So we'll find that these two shared highest similarity score after we've encoded them with BERT and calculate our cosine similarity.


And down here is the model we'll be using.


So we're going to be using sentence transformers and then BERT based NLI mean tokens model.


Now there's two approaches that we can take here.


The easy approach using something called sentence transformers.


I'm going to be covering that in another video.


And this approach was a little more involved where we're going to be using transformers and PyTorch.


So the first thing we need to do is actually create our last hidden state tensor.


So, of course, we need to import the libraries that we're going to be using.


So transformers, we're going to be using the auto tokenizer and the auto model.


And then we need to import Torch as well.


And then after we've imported these, we need to first initialize our tokenizer model.


Which we just do auto tokenizer.


And then for both these, we're going to use from pre-trained.


And we're going to use the model name that I've already defined.


So these are coming from face library, obviously.


And we can see the model here. So it's this one.


And then our model is auto model from pre-trained again.


Run those.


And now what we want to do is tokenize all of our sentences.


Now, to do this, we're going to use a tokens dictionary.


And in here, we're going to have input IDs.


And this will contain a list. And you'll see why in a moment.


And attention mask, which will also contain a list.


Now, when we're going through each sentence, we have to do this one by one.


For sentence in sentences.


We are going to be using the tokenizers encode plus method.


So tokenizer encode plus.


And then in here, we need to pass our sentence.


We need to pass the maximum length of our sequence.


So with BERT, usually we would set this to 512.


But because we're using this BERT based NLIME tokens model,


this should actually be set to 128.


So we set max length to 128.


And anything longer than this, we want to truncate.


So we set truncation equal to true.


And anything shorter than this, which they all will be in our case,


we set padding equal to the max length.


To pad it up to that max length.


And then here, we want to say return tensors.


And we set this equal to PT, because we're using PyTorch.


Now this will return a dictionary containing input IDs


and attention masks for a single sentence.


So we'll take the new tokens, assign it to that variable.


And then what we're going to do is access our tokens dictionary.


Which inputs IDs first.


And append the input IDs for the single sentence


from the new tokens variable.


So input IDs.


And then we do the same for our attention mask.


Okay.


So that gives us those.


There's another thing as well.


These are wrapped as vectors.


So we also want to just extract the first element there.


Because they're almost like lists within a list, but in tensor format.


And we want to extract the list.


Now that's good.


But obviously we're using PyTorch here.


We want PyTorch tensors, not lists.


So within these lists, we do have PyTorch tensors.


So in fact, let me just show you.


So if we have a look in here.


We'll see that we have our PyTorch tensors.


But they're contained within a normal Python list.


So we can even check that.


We do type.


We see that we get lists.


And inside there, we have the Torch tensor.


Which is what we want for all of them.


So to convert this list of PyTorch tensors into a single PyTorch tensor.


What we do is we take this Torch.


And we use the stack method.


And what the stack method does is takes a list.


And within that list we'll expect PyTorch tensors.


And it will stack all of those on top of each other.


Essentially adding another dimension.


And stacking them all on top of each other.


Hence the name.


So take that.


And we want to do it for both input IDs and attention mask.


And then let's have a look at what we have.


So let's go attention or input IDs.


And now we just have a single tensor.


Okay, so we do type.


And now we just have a tensor.


Now, that's great.


Check its size.


So we have six sentences that have all been encoded into the 128 tokens.


Ready to go into our model.


So to process these through our model.


We'll output the outputs to this outputs variable.


And we take our model.


And we pass our tokens as keyword arguments into the model input there.


So we process that.


And that will give us this output object.


And inside this output object, we have the last hidden state tensor here.


And we can also see that if we print out keys.


You see that we have the last hidden state.


And we also have this pooler output.


Now, we want to take our last hidden state tensor.


And then perform the mean pooling operation to convert it into a sentence vector.


So to get that last hidden state, we will assign it to this embeddings variable.


And we extract it using hidden or last hidden state.


Like that.


And let's just check what we have here.


So we'll just hold it at shape.


And you see now we have the six sentences.


We have the 128 tokens.


And then we have the 768 dimension size.


Which is just the hidden state dimensions within BERT.


So what we have at the moment is this last hidden state tensor.


And what we're going to do is now convert it into this using a mean pooling operation.


So the first thing we need to do is multiply every value within this last hidden state tensor by zero.


Where we shouldn't have a real token.


So if we look up here, we've padded all of these.


And obviously there's more padding tokens in this sentence than there are in this sentence.


So we need to take each of those attention mass tensors that we took here.


Which just contain ones and zeros.


Ones where there's real tokens.


Zeros where there are padding tokens.


And multiply that out to remove any activations where there should just be padding tokens.


E.g. zeros.


Now the only problem is that if we have a look at our attention mass.


So tokens attention mass.


If we have a look at the size.


We get a six by 128.


So what we need to do is add this other dimension which is the 768.


And then we can just multiply those two tensors together.


And this will remove the embedding values where there shouldn't be embedding values.


And to do that, we'll assign it to mass.


But we'll do it later actually.


So attention.


And what we want to do is use the unsqueeze method.


And if we start looking at the shape.


So we can see what is actually happening here.


See that we've added this other dimension.


And then what that allows us to do is expand that dimension out to 768.


Which will then match to the correct shape that we need to multiply those two together.


So we do expand.


And here what we want is we'll take embeddings.


And we want to expand it out to the embeddings shape that we have already used up here.


So that will compare these two.


And see that we need to expand this one dimension out to 768.


And if we execute that, we can see that it has worked.


So the final thing that we need to do there is convert that into a float tensor.


And then we assign that to the mask here.


So this float at the end, that's just converting it from integer to float.


So now what we can do is apply this mask to our embeddings.


So we'll call this one mask embeddings.


And it is very simple. We just do embeddings multiplied by mask.


And now if we just compare embeddings, have a look at what we have here.


So it's quite a lot.


And now we have a look at mask embeddings.


And you see here that we have the same values here.


So looking at the top, these are the same.


But then these values here have been mapped to zero.


Because they are just padding tokens. We don't want to pay attention to those.


So that's the point of the masking operation there.


So remove those.


And now what we want to do is take all of those embeddings.


Because if we have a look at the shape that we have, we still have this 128 tokens.


We want to convert this into one token.


And there's two operations that we need to do here.


So we're doing a mean pooling operation.


So we need to calculate the sum within each of these.


So if we summed all these up together, that's what we are going to be doing.


And pushing them into a single value.


And then we also need to count all of those values.


But only where we were supposed to be paying attention.


So where we converted them into zeros, we don't want to count those values.


And then we divide that sum by the count to get our mean.


So to get the summed, we do torch.sum.


And then it's just mass embeddings.


And this is in the dimension one, which is this dimension here.


That's how I look at the shape that we have here.


Okay, so now we can see that we've removed this dimension.


And now what we want to do is create our counts.


And to do this, we use a slightly different approach.


We just do torch clamp.


And then inside here, we do mass.sum.


Again, in the dimension one.


And then we also have, we also add a min argument here.


Which just stops us from creating any divide by zero error.


So we do one e.


And all this needs to be is a very small number.


I think by default it's one e to the minus eight.


But I usually just use one e to the minus nine.


Although in reality, it shouldn't really make a difference.


And sorry, just put counts there.


Okay, so that's our sum and our counts.


And now we get the mean pulled.


So we do mean pulled equals summed divided by the counts.


And we'll just check the size of that again.


Okay, so that is our sentence vector.


So we have six of them here.


Each one contains just 768 values.


And let's have a look at what they look like.


We just get these values here.


Now, what we can do is compare each of these


and see which ones get the highest cosine similarity value.


Now, we're going to be using the sklearn implementation.


Which is metrics.pairwise.


We import cosine similarity.


And then this would expect numpy arrays.


Obviously, we have PyTorch tensors.


So we are going to get an error.


I'm going to show you so you at least see it.


I'll show you how to fix it.


So we cosine similarity.


And in here, we want to pass a single vector


that we are going to be comparing.


So I'm going to compare the first text sentence.


So if we just take these and put them down here.


So I'm going to take the very first one of those,


which is mean pulled 0.


And because we are extracting this out directly,


that means we get a list format.


We want it to be in a vector format.


So it's a list within a list.


And then we want to extract the remaining five sentences.


So go one all the way to the end.


So that's those last five there.


Now if we run this, we're going to get this runtime error.


We go down and we see comment column numpy on tensor that requires grad.


So this is just with PyTorch.


This tensor is currently within our PyTorch model.


And we need to detach it from PyTorch


in order to convert it into something that PyTorch cannot read anymore.


And it actually tells us exactly what we need to do.


So use tensor, detach numpy instead.


So we take detach and numpy.


And all we need to do is write mean pulled because that rerun it.


And we get our similarity scores.


So straight away, we got 0.33174455.


This one is the one the highest similarity, 0.72, by a fair bit as well.


So that is comparing this sentence and sentence at index one of our last five,


which is this one.


So there we've calculated similarity and it is clearly working.


So that's it for this video.


I hope it's been useful.


I think this is really cool.


Faiss - Introduction to Similarity Search

Hi, welcome to this video. We're going to be covering Facebook AI Similarity Search


or FICE. And we're going to be covering what FICE is and how we can actually begin using


it and we'll introduce a few of the key indexes that we can use.


So just as a quick introduction to FICE, as you can probably tell from the name, it's


a similarity search and it's a library that we can use from Facebook AI that allows us


to compare vectors with a very high efficiency. So if you've seen any of my videos before


on building sentence embeddings and comparing sentence embeddings, in those videos I just


added a generic Python loop to go through and compare each embedding and that's very


slow. Now if you're only working with maybe 100 vectors, it's probably OK, you can deal


with that. But in reality, we're probably never going to be working with that smaller


data set. Facebook AI Similarity Search can scale to tens, hundreds of thousands or up


to millions and even billions. So this is incredibly good for efficient similarity search.


But before we get into it, I'll just sort of visualize what this index looks like. So


if we imagine that we have all of the vectors that we have created and we put it into our


similarity search index. Now they could look like this. So this is only a three dimensional


space, but in reality, there would be hundreds of dimensions here. In our use case, we're


going to be using dimensions of 768. So, you know, there's a fair bit in there. Now when


we search, we would introduce a new vector into here. So let's say here this is our query


vector. So x, q. Now if we were comparing every item here, we would have to calculate


the distance between every single item. So we would calculate between our query vector


and every other vector that is already in there in order to find the vectors which are


closest to it. Now we can optimize this. We can improve, we can decrease the number of


dimensions in each of our vectors and do it in a intelligent way so they take up less


space and the calculations are faster. And we can also restrict our search. So in this


case, rather than comparing every single item, we might restrict our search to just this


area here. And these are a few of the optimizations at a very high level that we can do with FICE.


So that's enough for the introduction to FICE. Let's actually jump straight into the code.


Okay, so this is our code. In here, this is how we are loading in all of our sentence


embedding. So I've gone ahead and processed some already because they do take a little


bit of time to actually build. But we're building them from this file here. We'll load this


into Python as well. But I mean, it's pretty straightforward to say load of sentences that


have been separated by a newline character. And then here we have all of those NumPy binary


files. Now there's NumPy binary files. Like I said, we're getting them from GitHub, which


are over here. That's where we're pulling them all in using this cell here. Now that


saves everything to file. And then we just read in each of those files and we append


them all into a single NumPy array here. And that gives us these 14.5 thousand samples.


Each embedding is a vector with 768 values inside. So that's how we're loading in our


data. I'll also load in that text file as well. So we just want to do with open sentences.txt.


And then we'll just read that in as a normal file. And we just write, I'm going to put


lines equals fp.read. And like I said, we're splitting that by newline characters. So we


just write that. Sorry, sentences. And we see a few of those as well. Okay. Now to convert


from those sentences into those sentence embeddings, I need to import this anyway for later on


when we're building our query vectors. I'll just show you how I do that now. What we do


is from sentence transformers, which is the library we're using to create those embeddings,


import sentence transformer. And then our model, we're using sentence transformer again.


And we're using the BERT and base NLI mean tokens model. Okay. So that's how we initialize


our model. And then when we're encoding our text, we'll see in a moment, we just write


model encode. And then we write something in here, hello world. Okay. And that will


encode, that will give us a sentence embedding. Okay. So that is what we have inside here.


We just have the sentence embeddings of all of our lines here. Now, I think we have everything


we need to get started. So let's build our first FICE index. So the first one we're going


to build is called the index flat L2. And this is a flat index, which means that all


the vectors are just flat vectors. We're not modifying them in any way. And the L2 stands


for the distance metric that we're using to measure the similarity of each vector or the


proximity of each vector. And L2 is just Euclidean distance. So it's a pretty straightforward


function. Now, to initialize that, we just write FICE. So we imported, no, so we need


to import FICE. And then we write index equals FICE dot index flat L2. And then in here,


we need to pass the dimensionality of our vectors or our sentence embeddings. Now, what


is our dimensionality? So each one is 768 values long. So if we'd like a nicer way of


writing out, we put sentence embeddings. And we write shape one. OK. And our index requires


that in order to be properly initialized. So do that. That will be initialized. Let


me run it again. I think my notebook just restarted. It did restart. It's weird. OK,


one minute. So that's going to initialize the index. And there is one thing that we


need to be aware of. So sometimes with these indexes, we will need to train them. So if


the index is going to do any clustering, we would need to train that clustering algorithm


on our data. And now in this case, we can check if an index needs training or is trained


already using the is trained attribute. And we'll see with this index, because it's just


a flat L2 index, it's not doing anything special. We'll see. Because it's not doing anything


special, we don't need to train it. And we can see that when we write is trained, it


says it's already trained. Just means that we don't actually need to train it. So that's


good. Now, how do we add our vectors, our sentence embeddings? All we need to do is


write index add. And then we just add embeddings like so. So pretty straightforward. So add


sentence embeddings. And then from there, we can check that they've been added properly


by looking at the end total value. So this is number of embeddings or vectors that we


have in our index. And with that, we can go ahead and start querying. So let's first create


a query. So we'll do xq, which is our query vector. And we want to do the model and code


that we did before. Now, I'm going to write someone sprints with a football. OK. That's


going to be our query vector. And to search, we do this. So we write di equals index search


xq. And then in here, we need to add k as well. So k, let me define it above here. So


k is the number of items or vectors, similar vectors that we'd like to return. So I'm going


to want to return 4. So with here, with this, we will return 4 index ids into this i variable


here. I'm going to time it as well, just so we see how long it takes. And let's print


i. You can see that we get these four items. Now, these align to our lines. So the text


that we have up here, that will align. So what we can do is we can print all of those


out. So let's do i. And then in here, we want to write lines i for i. Sorry, let me end


that. For i in i. OK. Ah, sorry. So this is 0 here. OK. So these are the sentences or


the similar sentences that we've got back. And we see, obviously, it seems to be working


pretty well. All of them are talking about football or being on a football field. So


that looks pretty good, right? Only problem is that this takes a long time. We don't have


that many vectors in there. And it took 57.4 milliseconds. So it's a little bit long. And


something that we can actually improve. OK. So before we move on to the next index, I


just want to have a look at the sort of speed that we would expect from this when we are


this is a very small data set. So what else could we expect? So if we go over here, I've


already written all this code. If you'd like to go through this notebook, I'll leave a


link in the description. So come down here, we have this flat L2 index. And this is the


query time. So this is for a randomly generated vector with a dimension size of 100. And this


is a number of vectors within that index. So we go up to 1 million here. And this is


a query time in milliseconds. You can see it increases quite quickly. Now, this is in


FICE, but it's still an exhaustive search. We're not really optimizing how we could do.


We're not using that approximate search capabilities of FICE. So if we switch back over to FICE,


we can begin using that approximate search by adding partitioning into our index. Now,


the most popular of these uses a technique very similar to something called Voronoi cells.


I'm not sure how you pronounce it. I think that's about right. And I can show you what


that looks like. So over here, if we go here, we have all of these. So this is called a


Voronoi diagram. And each of the sort of squares or the cells that you see are called Voronoi


cells. So here we have Voronoi cells. And that is just what you see here. So this, this,


all of these kind of squares are each a cell. Now, as well as those, we also have our centroids.


So I'm just going to write this down. So centroids. And these are simply the centers of those


cells. Now, when we introduce a new vector or our query vector into this, what we're


doing is essentially, so we have our query vector and let's say, let's say it appears


here. Now, within these Voronoi cells, we actually have a lot of other vectors. So we


could have, we could have millions in each cell. So there's a lot in there. And if we


compare that query vector and this thing here to every single one of those vectors, it would


obviously take a long time. We're going through every single one. We don't want to do that.


So what this approach allows us to do is instead of checking against every one of those vectors,


we just check it against every centroid. And once we figure out which centroid is the closest,


we limit our search scope to only vectors that are within that centroid Voronoi cell.


So in this case, it would probably be this centroid here, which is the closest. And then


we would just limit our search to only be within these boundaries. Now, what we might


find is maybe there's the closest vector here is actually here, whereas the closest vector


here is right there. So in reality, this vector here, this one, might actually be a better


approximation or a better, it might be more similar to our query. And that's why this is


approximate search, not exhaustive search, because we might miss out on something, but


that is kind of outweighed by the fact that this is just a lot, a lot faster. So it's


sort of pros and cons. It's whatever is going to work best for your use case. Now, if we


want to implement that in code, first thing that we want to do is define how many of those


cells that we would like. So I'm going to go 50. So use this endless parameter. And


then from there, we can set up our quantizer, which is almost like another step in the process.


So with our index, we are still going to be measuring the L2 distance. So we still actually


need that index in there. So to do that, we need to write FICE index flat L2. And we pass


out dimensions again, just like we did before. And like I said, that's just a step in the


process. That's not our full index. Our full index is going to look like this. So we write


index. And in here, we're going to have our FICE. And this is a new index. So this is


the one that is creating those partitions. So we write index IVF flat. And in there,


we need to pass our quantizer, the dimensions, and also the endless.


OK. Now, if you remember what I said before, in some cases, we'll need to train our index.


Now, this is an example of one of those times. Because we're doing the clustering and creating


those foreign noise cells, we do need to train it. And we can see that because this is false.


Now, to train it, we need to just write index train. And then in here, we want to pass all


of our sentence embeddings. So sentence embeddings, like so. Let's run that. It's very quick.


And then we can write, it's trained. And we see that's true. So now, our index is essentially


ready to receive our data. So we do this exactly the same way as we did before. We write index


add. And we pass our sentence embeddings again. And we can check that everything is in there


with index and total. OK. So now, we see that we have our index. It's ready. And we can


begin querying it. So what I'm going to do is use the exact same query vector that we


used before. Going to time it so that we can see how quick this is compared to our previous


query. And we're actually going to write the exact same thing we wrote before. So can I


actually just copy it? So take that. Bring it here. There we go. So now, let's have a


look. So total, 7.22. So bring it up here. And we have 57.4. Now, this is maybe a little


bit slow. So we'll see that the times do vary a little bit quite randomly. But maybe that's


a little bit slow. But it's probably pretty realistic. So that took 57 milliseconds. This


one, 7. Now, let's have a look. So these are the indexes we've got. Let's compare them


to what we had before. And I believe they're all the same. So we've just shortened the


time by a lot. And we're getting the exact same results. So that's pretty good. Now,


sometimes we will find that we do get different results. And a lot of time, that's fine. But


maybe if you find the results are not that great when you add this sort of index, then


that just means that this search is not exhaustive enough. Like we are using approximate search.


But maybe we should approximate a little bit less and be slightly more exhaustive. And


we can do that by setting the nprobe value. So nprobe, I'll explain in a minute. So let


me actually first just run this. And we can see it will probably take slightly longer.


So yeah, we get 15 milliseconds here. Of course, we get the same results again, because there


were no accuracy issues here anyway. But let me just explain what that is actually doing.


So in this case here, what you can see is a IVF search where we are using an nprobe


value of 1. So we're just using, we're just searching one cell based on what the first


nearest centroid to our query vector. Now, if we increase this up to 8, let's use a smaller


number in this example. So maybe we increase it to 4. Our four nearest centroids. So I


would say probably these, this one, this one, this one, and the one we've already highlighted.


All of those would now be in scope because our nprobe value, so the number of cells that


we are going to search is 4. Now, if we increase again to say 6, these two cells might also


be included. Now, of course, when we do that, we are searching more. So we might get a better


performance, better accuracy. But in terms of performance in time, it's also not, it's


also going to increase and we don't want time to increase. So there's a trade off between


those two. In our case, we don't really need to increase this. So don't really need to


worry about it. So that is the index IVF. And we have one more that I want to look at.


And that is the product quantization index. So this is actually, so we use IVF and then


we also use product quantization. So it's probably better if I try and draw this out.


So when we use product quantization, imagine we have one vector here. So this is our vector.


Now the first step in product quantization is to split this into sub vectors. So we split


this into several and then we take them out. We pull these out and they are now their own


sort of mini vectors. And this is just one vector that I'm visualizing here, but we would


obviously do this with many, many vectors. So there would be many, many more. So in our


case, that's one, 15,000, just under 15,000. Now that means that we have a lot of these


sub vectors. And what we do with these is we run them through their own clustering algorithm.


So what we do is we end up getting clusters and each of those clusters is going to have


a centroid. So this one would also be run through one. So each subset of vector slices


is going to be run through its own clustering algorithm, creating these centroids. And these


centroids are smaller in size than the original sub vectors here. And what we do is for each


of these sub vectors, so each of these sub vectors, they get pulled into here. So maybe


this one is here and it gets assigned to its nearest centroid. And then we take that assignment


all the way back over here and add it into our vector. So this is centroid three, for


example. And when I say assign it back, it's probably the wrong way to think about it.


Maybe it's more like this. So it becomes a new vector built from those centroid IDs.


Okay. So this would be three. Now, what that does is essentially reduces the size of our


vectors, but pretty significantly, depending on what dimensions we use there. So let's


go back to the code. Let's implement that. Now, we need to define two new variables here.


So m, which is going to be the number of centroids in the final vector. So one thing that we


do need to know with m is that m must be, we must be able to multiply m into d. So what


is our d value? It's 100. I can't remember. Where are we? Let me check. So 768. Now we


should be able to divide that into eight, I think. Yeah. So this is good. We can use


eight for m, but we couldn't use something like five. Because if we do five, we see that


d doesn't fit. So five doesn't fit nicely into d, whereas eight does. So m or d must


be a multiple of m. Otherwise we're going to get an error. And that's because of the


way that those vectors are broken down into the final centroid ID vectors. And we also


need to specify the number of bits within each of those centroids. So this value, we


can use what we want. I'm going to use eight. And then we can set up our index and also


the quantizer. So we use a quantizer as we did before. So the quantizer is going to be


fice.index flat L2 d. And also our index here is going to be, so this is a new one. This


is index IVFPQ. So it's not a flat vector anymore, which is the full vector. It's a


quantized vector. So where we have reduced the size of it through this, through the method


I explained before, where we drew it out. Now we need to pass a few arguments into here.


First one is the quantizer. So the quantizer d, which is our own dimensionality, n list,


m and bits. So pass all those to our index. Sorry, we need to put fice there as well.


And there we go. So we now have our index again. You may have guessed that we might


need to train this one. There we go. So to train it, we just write index.train, that's


our sentence embeddings. Okay. So it might take a little bit longer this time. There


we go. And then we can add our vectors. Now, after adding those, let's see how quick this


is. Should be a lot quicker or a fair bit quicker. It's hard to get much quicker than


the last one. So we're going to use the same code as before. So I'm going to take this


down here. See, 2.86. So we've gotten a lot faster. So we've gone from, what's up here,


57 milliseconds down to two. Now, there is one thing here. These values are now different.


So the accuracy has decreased. So if we, where is the last one here? So you can see that


we are getting, so we have the 190. We still have that one. And we have the 12, 4, 6, 5.


But these two at the front are now different. And this is just, it's on the trade-offs of


accuracy versus speed. So if we come down here, let's give that a go. Let's have a look


at what we are pulling through. So I'll copy this again. Let's just see. So we have these.


I mean, although the accuracy has decreased technically because it's not getting the same


results as the exhaustive search, they're still pretty good results. So I mean, nonetheless,


I think that is pretty cool. So let's have a look at, let's compare this to our previous


two other methods in terms of, as we did before, the graphs. So here is that final one. So


we have IVF PQ along the bottom. Yeah, it's a lot faster, right? And then we have IVF


flat with a end period value of 10, much faster than L2, but still not quite as fast as PQ.


And then we have flat L2 at the top, which obviously, and just as well, just be aware


on the left here, we have a log scale. So the differences are pretty significant when


we get up to the 1 million mark. So I think that's it for this video. So I think obviously


FIESS is pretty cool, definitely really useful. And I think we're definitely going to explore


it more in the future. So for now, that's it. So thank you for watching and I'll see


API Series #3 - How to Deploy Flask APIs to the Cloud (GCP)

Hi, in this video we're going to be having a look at how we can deploy a plastic API


using Google's cloud platform or GCP.


So it's actually a very simple process and we're going to jump straight into it.


So we want to go to Google and first thing I'm going to do is actually go to the GCP console.


So we just write GCP console and we just click here.


And that should bring you through to this page.


If you already have an account, if you don't, you will need to obviously create an account.


But once you have your account, you come over here, you can either come up here to create a project,


or you just click on this button here, create project.


And I'm just going to call mine, flask API.


You will need to set up a billing account if you haven't already.


Mine is already set, so I don't need to do anything there.


Now just come to create.


Okay, that will take a moment for it to actually create a project.


Okay, but once it is done, we will see we have this project ID here.


We're going to need to keep that.


So I'm just going to copy it now.


So we're going to be using it later.


And we need to initialize the cloud build API.


So we come up here.


And we just want to click on this first link here.


And what we do is enable this API for this specific project.


So you need to make sure at the top here you have your your flask API project set.


So again, that will take a moment to actually install.


And then we can go ahead and create a project.


So I'm going to go ahead and create a project.


That will take a moment to actually install.


And the final thing that we need to set up is we need to install the GCP SDK.


So all I'm going to type into Google is GCP SDK install.


And you see, okay, we have this first link here, install in cloud SDK.


Click on that.


And you just follow through.


These instructions are pretty basic for whichever OS you're using.


So for me, I'm Mac.


Click here, install this.


And then I have the G Cloud SDK on my terminal.


So I'm going to switch over to VS Code.


And we'll take a look at what all this actually looks like in our code


and actually creating our API and deploying it.


Okay, so I'm in VS Code now.


I have this little directory.


It's just an API.


We created this in the previous video series and an article.


So you can have a look at that if you want.


But the content of this isn't particularly important.


As a note, we're importing some data from some files here.


And we have these two endpoints, users and locations.


That's really it.


There isn't nothing else particularly important in here.


It's just a Flask API.


So what I want to do is open a terminal window here.


Okay, so I'm assuming my base environment here.


And I've already installed the cloud SDK.


So that means I can use these G Cloud calls.


So I'm going to write G Cloud app create project.


And I need this to be equal to the project ID that I copied before.


So I have abstract, abstract ring 230710.


Okay, and this will just create an app inside the project


that we created in the console before.


So I'm going to enter.


And you also need to select your region that you are going to be using.


For me, I'm just Europe West.


So I'll put 11.


And that will create the App Engine app in that chosen project,


which is an abstract ring, which we called Flask API.


That's the actual, this is the ID of that project.


The actual name is Flask API.


So the app is now created.


Please use G Cloud app deploy to deploy your first app.


Now, before we actually do that,


we need to install the App Engine extension for Python.


Because we're on Python right now.


And by default, the Python version of that is not installed or included.


So we need to do G Cloud components install App Engine.


Sorry, I realize you probably can't see.


So G Cloud components install App Engine, Python.


Okay, that's it.


And we see, so in my case, I've already installed the App Engine Python in my G Cloud SDK.


So it's just saying, I already have it installed.


If you just install it, I'll say it will install for you.


So that's everything we need for deploying our app.


I think the last few things we need to include here.


So we have our actual script out, API script.


But we do need a few other things as well.


So before we actually try and deploy our API, we should just test that actually works.


So we can simulate the same environment that the App Engine will be running our API within


by using something called Goonicorn or Gunicorn maybe.


So to install that, we just pip install Goonicorn.


I already have it installed, so I'm not going to do it again.


And once you have installed that, you just need to write the Goonicorn.


And then you select a host port.


So I'm going to go with a 1000.


And then you need to write the name of your Python API file.


So mine is main.


And then you come down here, you can see that we have this app run.


So where did I define app?


Should be at the top here.


So app, which is the Flask app.


So after this colon here, we need to include that.


And that will initialize and run our API.


Okay.


Now you can see here, let's make this a little smaller.


You can see here that we have the actual location.


So let's open that.


And if we just open it on this first page, we will see that it just says not found.


Now this isn't actually a problem.


It's because we only have the two endpoints, users and locations.


So let's open that and you'll see that we get this JSON response, which is what we'd expect.


Now, the content of that is not important.


We just want to see that, okay, the API is actually running when we run it with Goonicorn.


So that's good.


So let's switch back over to VS Code.


We'll stop that from running.


And that means that the core of the API will run in the cloud App Engine environment.


So it's good.


The only thing is, okay, we have our Python environment here.


So I'm using Python 3.8 down here.


And up here, I have a few modules I've imported as well.


So Flask, Flask RESTful, Pandas.


How is the App Engine going to know that we're using this environment?


It doesn't.


We need to tell the App Engine what our environment looks like and what environment it should set up.


So to do that, we need to use two files.


So we'll come over here and then create a new file.


And I'm going to call it app.yaml.


And inside here, we just want to set our runtime.


So the runtime is just going to be Python 3.8.


That's it.


Nothing else is needed there.


Now I'm going to create another new file.


And I'm going to call it requirements.txt.


And in here, we're going to call it run time.


And I'm going to call it requirements.txt and in here, I need to write which modules are using


which packages.


Ours is pretty simple.


So Flask, Flask RESTful, and we also have Pandas.


Now save that.


And beyond that, there isn't anything else.


So we can switch back over to the terminal window.


And we can now do the gcloud deploy.


So write gcloud app deploy.


And you see also just within the API folder.


So this is the folder that ordered directory that contains all of these files here.


Deploy.


This will take a moment.


I'm going to tell it where I want to deploy.


Okay, this is actually quite useful.


So I've got this error that's popped up because I am currently in another project,


which no longer exists.


So I'm getting this error.


So I need to make sure that I'm trying to deploy to the right project.


So I just write gcloud app deploy.


I can specify the app.yaml.


And I just write project equals.


And then in here, I need the project name from before.


So I'm just going to go and copy that again.


So come to here.


So it's this abstract ring.


Abstract ring.


And I just write in here abstract ring.


Okay.


Now run that.


That should come up with this here.


So we have come down here.


We have the script and then the target project.


So it's correct.


We're now going to the correct place.


And this URL here is where we will be deploying the API to.


So I'm going to click continue.


Yes.


And this can take a little bit of time for it to run.


So just wait a moment.


One thing that's actually pretty useful is when you try,


when you first deploy, gcloud will create this gcloud ignore file.


It's like a git ignore.


And in here, it will just ignore the typical things that you do actually want to ignore.


So you have PyCache here, which has been created up here.


So we don't need to exclude that from the upload by default, which is useful, I think.


So see how that is going.


Still uploading.


Okay.


So good to go.


I'm going to copy this.


Open in the browser.


And we should see, so what we saw before with this, we should end up seeing the same.


So we'll go first to that address.


We get that not found that we saw before.


So expected.


And we'll just go to users.


There we go.


So we see that our API is now deployed to Google Cloud.


I can also see that over here.


So if I refresh, we should see the request pop up over here.


Or they will do eventually.


So this will give you a summary of how many requests are made to your API.


That's it for this video.


I hope it's been useful.


And I will see you in the next one.


Training BERT #3 - Next Sentence Prediction (NSP)

Hi, welcome to the video. Here we're going to have a look at using next sentence prediction or NSP


for fine tuning our BERT models. Now a few of the previous videos we covered mass language modeling


and how we use mass language modeling to fine tune our models. NSP is like the other half of fine


tuning for BERT. So both of those techniques during the actual training of BERT, so when Google train


BERT initially, they use both of these methods and whereas MLM is identifying or almost training on


the relationships between words, next sentence prediction is training on more long-term relationships


between sentences rather than words. And in the original BERT paper it was found that without


NSP because they tried training BERT without NSP as well, BERT performed worse on every single metric.


So it is pretty important and obviously if we take this approach, we take mass language modeling and


NSP and apply both those to training our models, fine tuning our models, we're going to get better


results and if we just use MLM. So what is NSP? NSP consists of giving BERT two sentences, sentence


A and sentence B and saying paper does sentence B come after sentence A and then BERT will say okay


sentence B is the next sentence after sentence A or it is not the next sentence after the sentence


A. So if we took these three sentences that are on the screen, we have one, two and three, right?


One and two, if you ask BERT does sentence two come after sentence one, then we'd kind of want BERT to


say no, right? Because clearly they're talking about completely different topics and the type


of language and everything in there just doesn't really match up. But then if we have a look at


sentence three and sentence one, they do match up. So sentence three is quite possibly a


the follow-on sentence after sentence one. So in that case we would expect BERT to say this is the


next sentence. So let's have a look at how NSP looks within BERT itself. So here we have the core


BERT model and during fine-tuning or pre-training we add this other head on top of BERT. So this is the BERT for pre-training head.


And the BERT for pre-training head contains two different heads inside it.


And that is our NSP head and our mass language modeling head.


Now we just want to focus on the NSP head for now. And as well we don't need to fine-tune or


train our models with both of these heads. We can actually do it one by one. We could use it mass


language modeling only or we could use NSP only. But the full approach to pre-training BERT is using both.


So if we have a look inside our NSP head we'll find that we have a feed forward neural network


and that will output two different values. Now these two values are our is not the next sequence


there and our is the next sequence which is there. Okay so value zero is the next sentence value one is not the next sentence.


Now we have the final outputs from our final encoder in BERT at the bottom here.


And we don't actually use all of these activations. We only use the CLS token activation which is over at the left here.


So this here is our CLS token. Okay and when I say this is our CLS token I mean more that this is not our CLS token.


The CLS token is down here so we input the CLS token and this output is the subsequent output after being processed by 12 or so encoders within BERT itself.


So this is the output representation of that CLS token. Now the activations from that get fed into our feed forward neural network.


And the dimensionality that we have here is 768 for that single token. This is in the


BERT base model by the way. And that gets translated into our dimensionality here which is just the two outputs.


So that's essentially how NSP works. Once we have our two outputs here we just take the arg max of both of those.


So we take both over here and we just take an arg max function of that and that will output us either 0 or 1.


Where 0 is the is next class and 1 is the not next class.


And that's how NSP works. So let's dive into the code and see how all this works in Python.


Okay so we're going to be using HuggingFace's transformers and PyTorch. So we'll import both of those.


And from transformers we just need the BERT tokenizer class and the BERT for next sentence prediction class.


And BERT for next sentence prediction.


Now we also want to import Torch and we're going to use two sentences here.


So both of these are from the Wikipedia page on the American Civil War and these are both consecutive sentences.


So going back to what we looked at before we would be hoping that BERT would output a 0 label for both of these.


Because sentence B is the next sentence after sentence A.


This one being sentence B this one being sentence A.


So execute that and we now have three different steps that we need to take.


And that is tokenization, create a classification label so the 0 or the 1 so that we can train the model.


And then from that we calculate the loss. So the first step there is tokenization.


We tokenize, it's pretty easy. All we do is inputs, tokenizer and then we pass text and text2.


And we are using PyTorch here so I want to return a PyTorch tensor.


Make sure that's PT.


We need to also initialize those so tokenizer equals BERT tokenizer from pre-trained.


And we'll just use BERT base encased for now. Obviously you can use another BERT model if you want.


And copy that and initialize our model as well.


Okay now rerun that.


And we'll get this warning that's because we're using these models that are used for training or for fine tuning.


So it's just telling us that we shouldn't really use this for inference. You need to train it first.


And that's fine because that's our intention.


Now from these inputs we get a few different tensors.


So we have input ids, token type ids and attention mask.


Now for next sentence prediction we do need all of these.


So this is a little bit different to mass language modeling.


With mass language modeling we don't actually need token type ids.


But for next sentence prediction we do.


So let's have a look at what we have inside these.


So input ids is just our tokenized text.


And you see that we pass these two sentences here.


And they're actually both within the same sentence or the same tensor here, input ids.


And they're separated by this 102 in the middle which is a separated token.


So before that all these tokens that is our text variable or sentence A.


And then afterwards we have our text 2 variable which is sentence B.


And we can see this mirrored in the token type ids tensor as well.


So all the way along here up to here that's our sentence A.


So we have zeros for sentence A.


And then following that we have ones representing sentence B.


And then we have our attention mask which is just ones because the attention mask is a one where it's a real token and a zero where we have padding token.


So I don't need to really worry about that tensor at all.


Now the next step here is that we need to create a labels tensor.


So to do that we just write labels and we just need to make sure that when we do this we use a long tensor.


OK. So we use a long tensor and in here we need to pass a list containing a single value which is either our zero but is the next sentence or one for is not the next sentence.


In our case our two sentences are supposed to be together.


So we would pass a zero in here.


And run that.


And if we have a look at what we get from there you see that we get this integer tensor.


So now we're ready to calculate our loss which is really easy.


So we have our model up here which we have already initialized.


So we just take that.


And all we do is pass our inputs from here into our model as keyword arguments.


So that's what these two symbols are for.


And then we also pass labels to the labels parameter.


OK. And that will output a couple of tensors for us so we can execute that and let's have a look at what we have.


So you see that we get these two tensors we have the logits and we also have the loss tensor.


So let's have a look at the logits and we should be able to recognize this from early runway.


We saw those two nodes and we had the two values on for the index zero for is next and index one for is not next.


So let's have a look.


You see here that we get both of those.


This is our activation for is the next sentence.


This is our activation for is not the next sentence.


And if we were to take the argmax of those outputs logits we get zero which means it is the next sentence.


OK. And we also have the loss and this loss tensor that will only be output if we pass our labels here.


Otherwise we just get a logits tensor.


So when we're training obviously we need labels so that we can calculate the loss.


And if we just have a look at that we see it's just a loss value which is very small because the model is predicting a zero and the label that we've provided is also a zero.


So the loss is pretty good there.


So that is how NSP works.


Obviously it's slightly different if you're actually training your model and I am going to cover that in the next video.


So I'll leave a link to that in the description.


But for now that's it for this.


Streamlit for ML #5.3 - Publishing Components to Pip

Today we're going to look at how we can actually publish the component that we've been building for Streamlit.


So what that means is we can actually pip install the component and then use it in any Streamlit app.


Just as we would a normal Python framework.


So we pip install it and then we just import and use it.


So this is an article I've been putting together that kind of covers this.


So we're basically just going to be going through this.


And what you're going to see at the end is we can actually do this.


So you see here we have this from st card component import card component.


And then we just do card component title subtitle body link.


And that will create our card.


So there's not that much to go through. It's pretty straightforward I think.


So let's jump into it.


So a little bit of background on how pip is actually working here.


When you pip install something you're actually installing it from this here.


The Python package index.


So put all pi b here. I think that's how you pronounce it.


And I can go in here and I can search for pandas.


And it's going to show us pandas or it's going to show us a lot of different pandas.


I think it's this one.


OK. Yeah. Pip install pandas at the top there.


Now we can also find the st card component I've already built beforehand.


So if we open this. I think it's this one. Yeah.


It's like version 0.10 at the moment.


So this is the current component. Look there's me.


And you can go ahead and install that.


It's slightly different to the component that we have been putting together.


Not as generic.


I built it for a particular use case which you'll probably see pretty soon.


So what we're going to do is create another st card component 2 or something.


I don't know. You have to give it a unique name so we can't use the same name again.


So we'll go ahead and start with that.


So we'll go to our project. We have our template st card component.


Now there's a few things we need to do for this.


So the first one is create a build distribution for the react component or react part of our component.


So we'll come into I think here and we'll do that at the moment.


There's no build directory. We're going to make one.


So we need to open a terminal window for that.


So I'm just pulling one up now.


OK. And I'll just navigate to this directory. So documents.


Projects.


Component. Component template.


Template.


And then in there we want to go st card component.


We might need to change the name of this by the way. I'm not sure.


And front end.


OK. So from within here we need to use node package manager to create the build distribution.


So come to the article.


Oh that is something we also need to do. So let's not forget that.


So before we before we do that we also need to.


Go to init.py.


And up here we need to make sure we set release to true.


Now release is false whilst we're putting everything together but as soon as we want to actually distribute it


or publish it we need to set it to true. So beforehand this is this was being run.


OK. So it was saying you need to load st card component from local host 3 0 0 1.


Obviously when people are using this component in the future they're not going to be running the component on the local host.


Instead what they are going to do is they're going to download a package which is going to contain this front end build distribution directory.


And from there they'll get the component they will be built from there.


OK. So I think I don't know.


So anything else. Oh and also this this will not be run anymore.


So this is like the demo app that we were putting together before.


That will no longer be run. So yeah that's fine.


We just need to change that nothing else needs to be changed.


Then we're now our front end directory. We just write this npm run build.


Let's do that. Let me open this so we can see what's going on here.


Hopefully this will go well and we'll see the build directory appear in the front end on the left.


Might take a moment as well.


OK. So the build folder is ready to be deployed and we can come over here and we see we have build and it has all of these.


So this is our this is our component in JavaScript.


OK. Now with that we can we have the build distribution for the actual react component.


Now we need to create the distribution for the Python distribution.


OK. So let's go ahead and do that. Scroll down a little bit.


So this is one thing I think we need might need to change.


So changing the name parameter in setup.py because at the moment it's stcardComponent.


I don't think it needs to be that anymore.


Yeah. I think I'm going to need to change this to stcardComponent2.


So let's go ahead and do that.


OK. So we have initpy here. We're going to go to setup.py.


So here. We have the name of component.


And I think we'll also need to change this template name.


So this will be stcardComponent2.


OK. Now we should be able to go ahead and create the component or create the Python component.


OK. First we need to Python setup.py.


We need to write sdist and bdistwheel like that.


OK. We run that and we should find a...


So we need to go one more.


Fold it up. Then run it again.


So from this we'll find that we get a dist folder here. So dist.


So in here we should find we have this tar file and this wheel file.


And it's those two that we're going to upload to the Python package index.


And to do that we're going to need something called twine.


So we first install that pip install twine.


I already have it downloaded so I won't run it again.


And then we run Python.


m twine upload.


And then we need to specify that we're going to upload it to the Python package index.


And the only other option you have here is the test Python package index which can be quite...


I would recommend you look into that.


Like if you're putting a package out there that's quite good to test that the package is going to work.


I have a video and article I think on Python packages.


So maybe... I'll put a link to that in the description so you can take a look if you're interested in figuring this out a bit more.


So we're just going to go with the Python package index rather than the test Python package index.


And we want to specify that we're going to be pulling everything from the distribution folder.


So let's run that.


And it will ask us to login.


So my username...


Oh this is another thing. So you will need to sign up on the same website I mentioned before.


It's I think at pipi.org.


So you need to sign up and then you can go ahead and do this.


File already exists. OK.


This is because I already have...


You see here it's using stcard component.


So I need to make this stcard component 2.


So we'll go through everything in here.


So this actually I didn't need to rename that all.


So we can ignore that.


So let's rename all this to 2.


It's also in init.py.


This one we don't really need to change but we'll do anyway.


2. Is there anything else in there?


I don't think so. So delete this.


I'll delete the distribution folder and the build folder. I don't know.


OK. I'll delete that as well.


And then in here we need to change this because the directories change.


So the manifest and also in setup.py changes to 2 as well.


OK. Now let's rerun this again.


So we go to this.


Python. Yep. Run that.


Create our distribution folder.


And then we're going to use twine again. So hopefully this time it'll work.


OK. So now we have stcard component number 2.


We can go ahead and actually have a look at that website.


So let's come over here.


OK. So we can pip install.


And this is a version number. We don't actually need that.


OK. Here we just pip install stcard component number 2.


Let's copy that across.


I'm going to do that in my streamlit environment.


OK. Now while that's downloading.


I'm going to come over here. I'm going to open or create a new directory.


Actually. So.


I'll do it from here. OK.


So make directory.


We'll call it stdemo.


I'll open that in VS Code.


OK. And then in here I'm just going to create a new file app.py.


In here. Let me copy it across from the article.


A little quicker.


So where are we here.


It's actually open in the gist. OK. So I'll copy this.


Put it into here. Stcard component number 2.


OK. Save that. And now let's run it from here.


So cd stdemo. And we run streamlit.


Run. App.py.


OK. Localhost. And here we go. So this is our.


Package or distributed. Stcard component.


Click here. Make sure it's working.


There we go. Perfect. So.


Yeah that's everything for this kind of short series on.


Custom components in streamlit.


I hope it's been useful and.


The NEW Match-Case Statement in Python 3.10

Okay so we're going to go through the new match case statement in Python 3.10. So the new match


statement they're actually calling it structural pattern matching and what it does is basically


allows us to create a switch case statement in Python. So switch case statements are pretty


common in most languages but it's something that Python just hasn't had and we can actually see in


this pep here pep 3.10.3 that they were going to add it back in 2006 or it's been considered at


least but then it was just rejected because basically no one wanted it. So since then it


just hasn't been added to Python but now with this new pep pep 6.3.4


it looks like it's going to be added in 3.10. So what I've done is downloaded Python 3.10


it's alpha 6 and we're just going to have a play around and see how this works.


So this first quick example of how it's going to look and it's just a super easy simple example.


I'm just going to comment these out because I'll cause an error and you can see here that we have


this HTTP code 418 and what we're doing here is we're matching the subject which is this head


which is this HTTP code and then we go into our cases. So in the case of that subject being equal


to this we do whatever is within this block and we do the same if it is 404 or if not we go down


to 418 and at the end of that we can also add a catch all case statement. So in this case we just


say okay case we put anything here and we just say okay code not found. So this is what will run


if none of these are executed. So we can also remove this and it will just carry on with the


code. So that's our first example you see here we get I'm a teapot for code 418. Obviously that's a


super simple example and that's essentially how it works. I also have this graph here to try and


explain more visually the actual flow of information here. So we have a subject which we


set to match at the top. So in this case we are using HTTP code and then we check for truthy or


falsy whether that subject matches the pattern that we've provided in our case. If it is truthy


we execute the block if it is falsy we continue to the next case. So yeah it's pretty much how it


works but let's have a look at a few other examples which I think demonstrate the actual use case and


benefits of this a bit better. So the first one though that I want to quickly show you is this one


which is actually from this pep here pep635 and I thought this was a pretty cool example. So


what I showed you is just a really simple you know is this equal to this if not go on to the next one


whereas this I think better demonstrates that we can check the structure of the data that we're


feeding in. So here we're checking the host port and mode of a connection. So for example if we're


connecting through HTTP we might set our mode to HTTP and then we also have a host and a port and


this will all be within a tuple. However in some cases maybe let's show you here so we can have our


host here and then we have the port and then here we may sometimes have the mode of the connection


so maybe it's FTP and sometimes we might not and if we don't see anything here we just assume it's


HTTP and that's essentially what this code here is doing. So it's taking that tuple as x and in this


case we can see we're using if elif and it's fine like there's nothing wrong with this but then if


we look at how we write this using match in case it does look a lot cleaner. So that's just one


example and then I have an actual use case example. I've been using the squad2 data set which


is a very popular data set for training question answering models in machine learning and I think


that's actually a really cool example of how the case match statement might actually be pretty


beneficial. So let's just separate this out and I'll walk you through how we can apply that. So


first I'm just going to import requests and also import json because we're going to be pulling a


json file from the internet and we're going to be reading that and the url for that is this


and the file that we're going to be reading is the training data from the squad data set.


Okay so the data that we're looking for is at this address. So if we want to just download that


just download that we can with requests and we'll just get url plus the file


and then after we've pulled that using requests we're just going to save it to file.


So I'm going to file write binary as f and I'm going to write it to file in chunks. It's quite


a big data set. So it's going to use it's a content chunk size. I mean we can kind of go


for anything here let's go for 100 and then just write the chunk. Okay so that's our data set


downloaded and we can open it over here and if we just look through a few of these we can see that


there's quite a few layers to the data set. So this is something we're going to have to consider


when we're building out this function for parsing it both with the if else version and the match


case version. So what we're going to want to do here is loop through each one of these and you


can see there's quite a few of them and we want to get the question and the text here which is the


answer to that question and we just want to pull those out and nothing else we'll just put them out


as a tuple and create a big list. Now the complexity of this comes from so if I sorry open that again


you can see here we have okay this is fine and we have a question and we have a question and then we


have answers which contains our answer. So just remember we've got answers here okay this is the


name of the key that contains our answer. If we go right down to the bottom the actual format is


different and this is the case for quite a few of them. So if we go into here okay see straight away


we have the question and we have answers but it's empty and instead we have this plausible answers


and in here we have our answer. So it's slightly different format in that rather than using answers


some of them use plausible answers I'm not sure why but some of them do and for some of those as


well they also include this answers key which is just an empty list. So we just need to write some


logic to actually deal with that. Okay so we've saved this file so I'm just going to open it back up again


and it is only just here just read binary it's f and we just save it into the squad variable.


Let's have a quick look at what we have in here. So you see over here everything is contained


within this data key so if we close out it closes everything. So first off we want to access that


and then it's a list and we'll loop through each one of those. So let's just have a look at the


first version or the first item in that list and it's the Beyonce group. So we can just have a


quick look it's just quite messy I'm not going to go into it but essentially to work through this


data we're going to have to write something like this and this is for both the if else statement


or if else version and the match case version and we write squad data. So this is going through each


group so Beyonce or Matters we are the two that we saw before. Now I'm going to go through each paragraph


in the group that is paragraphs


and then we're going to go through each question and answer here within that or each question and


the information that has next to that question and that is in the paragraph


QAS.


And I'm just going to pass for now but that will loop through everything that we need so we're


going to use that for both the if else version and the match case version. So let's take this


and we'll build out the if else version first. So in this case we can just get our questions


straight away so we are at the moment we've gone through paragraphs and we're looping through each


of these so we're in QAS at the moment and we're going to pass through the first one.


So we're in QAS at the moment and we want to get the question which is this so QA


question and then this is where we have our if else logic. So answers in here and


first we want to check okay is there an answers key within the dictionary here.


So we write that like if answers in QA.keys but as we saw before the answers key can be there


but it can also be empty and if that's the case we also obviously we can't pull anything out from


that answers list. So we also need to say and length of QA answers is greater than zero. So we


have two conditions there and if both are satisfied then we want to pull the answers from that key.


So it's answers QA answers and then it's always with this data set we are always entering the


or there's always only one item within the list of answers. So we just pull out the first item at


index zero and then we pull out the text. So if I show you we have index zero and then we have the


text. So entering this dictionary or list sorry and then we're entering index zero because all


index zero and then we're pulling out the text which can be in the late 1990s for this first one.


Okay so that is our first part of the if else statement. Second part we want to say okay


else if we have plausible answers. So this is what I showed you at the end plausible answers


in QA keys and we also want to say the same again where QA plausible answers needs to have some


values in it as well. So in this case we pull our answers from plausible answers.


So in this case we pull our answers from plausible answers.


And it's the same again there. Okay so that's our if and elif and then we just want one final


else statement at the end. So if for some reason we can't find the answers I'm just going to put


answers equal to none. Then after all of that I'm going to just initialize a new squad list here.


And we're just going to append our question answer pairs to that. So new squad dot append.


Let's have question and answers. Should really be answers answer rather than answers but it's fine.


Okay so let me just put these into a two port.


Okay great so let's just have a quick look at what we get here. Make sure we're getting the


right thing. So we're going to take a look at the first five and the last five as well.


Okay so this looks good. Yep so we have two pulls question in the first index and the answer in the


second index. Okay so that's great and it's worked for both the answers and plausible answers.


It's all of these down here using the plausible answers format. So this is what it looks like with


if, elif and else. So let's rewrite this. But we're going to do it with the match case statement


instead. So take this and what we can do is just remove all of that and maybe we can just keep in


the question and what we'll do is say match QA and then in here what we are looking for is in the


case of having a dictionary that contains the answers key and that answers key contains a list


which also contains a dictionary which contains a key called text. We want to pull out the answers


from whatever that text key is pointing to. So the value there and this will pull whatever is


within this part of the dictionary or the part of the data structure into this new variable answers.


Which is I think really cool about the new syntax. So what we've done here is actually


already assigned our variable answers so we don't actually need to do anything within this code block.


So that's pretty cool and then we just write the same thing but we do it for our plausible answers.


And then we just write the same thing but we do it for our plausible answers. So if we write that out


and this in my opinion is pretty cool as well.


We just write it like that and that's all there is to it and we just write pass again


and in the final case of it not working we just say okay just set answers equal to none.


And let's run that and let's just take what we we have here and we should get the same.


And we should get the same. Okay so yep we can see exact same output. So that's the comparison.


I mean to me this does look way cleaner and this is I suppose a little bit complicated


but generally I think this is kind of easier to read than this. Or at least at first glance this


does look cleaner in my opinion but you know I'm not 100% sure on which one I would go for.


At the moment I'm kind of leaning towards this one but let's see. Let's see how people start using


this new syntax going forward. It'll be pretty interesting to see at least. So that is all I


wanted to cover on that. So that's it for this video. Thank you very much for watching. I hope


CLIP Explained | Multi-modal ML

Today we're talking about what is quite possibly the future of machine learning in both computer


CLIP Explained | Multi-modal ML

vision and NLP. We're talking about a combination of both into a single multimodal model. That model


CLIP Explained | Multi-modal ML

is Clip which was built and trained by OpenAI. Now Clip is open source and it has been around


CLIP Explained | Multi-modal ML

for a little while. It's been around from since the start of 2021 but in the past few months


CLIP Explained | Multi-modal ML

we've seen the adoption of Clip grow at a pretty insane rate. It has found uses in a load of things


CLIP Explained | Multi-modal ML

so what we will be focusing on is text or multimodal search across text and images.


CLIP Explained | Multi-modal ML

We're also going to talk a little bit about zero shot classifications, zero shot object detection


CLIP Explained | Multi-modal ML

and at some point in the future we'll also talk about the use of models like Clip or even Clip


CLIP Explained | Multi-modal ML

itself in the diffuser models that have become incredibly popular recently like DALY 2, Imagen


CLIP Explained | Multi-modal ML

and Mid-Journey. Now to understand why Clip is so important we can use a paper that was released


CLIP Explained | Multi-modal ML

in 2020 called Experience Grounds Language. In Experience Grounds Language the authors define


CLIP Explained | Multi-modal ML

these five different world scopes, corpus, internet perception, embodiment and social.


CLIP Explained | Multi-modal ML

Now most of the models that we're aware of in NLP, this is a sort of NLP language focused paper,


CLIP Explained | Multi-modal ML

these were sort of the state of the art very recently so you have BERT, T5 and all these


CLIP Explained | Multi-modal ML

different models GPT and so on. In perception this is where we start to see not just NLP but also


CLIP Explained | Multi-modal ML

computer vision so here is where we are now with Clip. Okay and quite interestingly as well in the


CLIP Explained | Multi-modal ML

next world scope this is where you start to get reinforcement learning and then you continue going


CLIP Explained | Multi-modal ML

so you almost see this culmination of all the different disciplines and machine learning coming


CLIP Explained | Multi-modal ML

together into one larger discipline. Now the focus between these different world scopes is


CLIP Explained | Multi-modal ML

mostly on the data that is being used to train the models so over here world scope one you have


CLIP Explained | Multi-modal ML

the first models that we saw which would have been things like Word2Vec which is probably the


CLIP Explained | Multi-modal ML

one of the earliest examples of deep learning in NLP and that consisted of training a neural network


CLIP Explained | Multi-modal ML

on a smaller or not small amount of data but small compared to future data sets so a single corpus


CLIP Explained | Multi-modal ML

you know for example 100,000 sentences from Wikipedia maybe one example of that.


CLIP Explained | Multi-modal ML

Then going forwards we have the internet sized corpuses so these are based on a huge web


CLIP Explained | Multi-modal ML

scraping from across the internet from many different sources and the models trained on this


CLIP Explained | Multi-modal ML

were able to obviously kind of pull in more general understanding of language from


CLIP Explained | Multi-modal ML

purely text data. Okay so it's a lot of data but it's so purely text. The next one which we're


CLIP Explained | Multi-modal ML

focusing on world scope 3 is not just that it's in our example text and image data. Okay so we're


CLIP Explained | Multi-modal ML

training a model to understand both of these different modalities of information and this is


CLIP Explained | Multi-modal ML

almost like AI moving from a purely purely digital very abstract space to a more realistic real world


CLIP Explained | Multi-modal ML

space because in the real world we don't just rely on text data we rely on a huge number of sensory


CLIP Explained | Multi-modal ML

inputs we have visual audio touch and everything else. Okay so we're sort of moving more towards


CLIP Explained | Multi-modal ML

that more broad scope of inputs from different modalities okay where modality would be something


CLIP Explained | Multi-modal ML

like text or image or visual and so on. For us in the real world that chaotic ensemble of different


CLIP Explained | Multi-modal ML

sensory inputs is what kind of creates or trains our internal model of the outside world so it


CLIP Explained | Multi-modal ML

makes sense that that is the sort of direction that machine learning and AI may also go in.


CLIP Explained | Multi-modal ML

So to achieve this multi-modality in CLIP we actually use two models that are trained to


CLIP Explained | Multi-modal ML

almost speak the same language. So with these two models one of them is a text encoder one of them


CLIP Explained | Multi-modal ML

is an image encoder. Both of these models create a vector representation of whatever they are being


CLIP Explained | Multi-modal ML

input so the text encoder may get a sentence that sentence could be two dogs running across a frosty


CLIP Explained | Multi-modal ML

field and then we have a image of two dogs running across a frosty field and CLIP will be trained so


CLIP Explained | Multi-modal ML

that the text encoder consumes our sentence and outputs a vector representation that is very very


CLIP Explained | Multi-modal ML

closely aligned to what the image encoder has output based on the image of the same concept.