maybe you could invert it by just looking for all of the opposite values but to do that you'd have


Implementing Filters in the New Haystack Doc Store

to pull out all of the metadata from a pinecone and i don't i don't think it's possible to do that


but someday i need to check so for now i'm just going to


ignore this one we'll deal with it later so yeah let's that looks good so let's


so yeah let's that looks good i think um the only thing now i want to test it on some other


filters this is just one filter maybe it doesn't work on on the others so let me uh find some of


those so i think which which notebook were those uh testing filters and yeah we come down here we


have a few other filters as well so let's try those hopefully it works so filter down here


we pass them and then we convert them to pinecone


so convert to pinecone


so let's see um


and just remove that so we see it straight away


okay so we have um we have a lot of stuff in here let me just remove this


more Zelda


okay um


yeah i mean that looks looks right to me let's try another


there's a testing okay yeah


uh we we just did that one so let's try this one


let's see what we get so convert to pinecone just this again


yeah and this looks good as well


so i think i think we're probably ready to actually go ahead and test that in here so


let me take one of these um i'll take some of these filters i'm just going to apply directly


to pinecone to make sure it's actually working so to do this i have to create a load of vectors


and i don't want to need to do that i just want to create a new filter


so if i go to


pinecone demo come here okay so this one this one's easier


yeah so this one only does it with like six um so i'm gonna run through this and see if it works


so initialize i think we've already created this dictionary this document store


um oh the other thing that we need to do is actually implement it in pinecone so that's


important so in here at the moment when we go to filter i'm using this uh build filter clause


i don't want that anymore so i actually want to just remove this


and then let's find where we're applying filters okay let's have a look at right documents see


what i do okay so here we're using self build filter clause so we need to change this to use


the same logic as the other document source so so look at um we aviate and i think


okay uh ignore that the aviate filters see why they okay here this is what we want to do


this is what we want to do so the filter this just use this nothing more than that so it's


really simple so the filters are just going to be converted into that so they we use filter


dict here but just use filters convert to pinecone


and i'm just going to search for this that will tell us where else we are


using that so search for this oh it's only the only here oh really okay so that's it


it's the only bit we need to change we also need to import this so it's what is it logical filter


okay import this and we should be good to go


okay now let's go ahead and pip install that


and then we can test it and hopefully it will work okay so come over here


pinecone demo let's restart that rerun it


um so we've already got a document sort here so i don't think we need to write these documents again


um yeah i can ignore that dense passage retrieval we do need


don't need to update the embedding so let me just move this to a new cell


run it we do need this


and then okay let's make sure it's working and those answers and then we will try and apply the


metadata filtering so filters just something really simple here and then maybe we can like


modify them a little bit to see what else you can get okay it looks good um


it's going to run that just so i can see maybe some of the metadata we have here


it's just this one again i want to return another name so i can add it like a grain or


statement you see what that looks like but for now it's fine okay let's run it okay cool


okay let's run it okay cool so looks like filtering actually worked


we're only returning two here because um based on that filter there's only two items in the out of


the six so that that's really cool uh let's get all the documents so


and let's get some other names so we have this one


let's add that into our into our filter so we're going to go we're going to use all here so


we want or


this of this


okay let's run that i think that's in the right syntax


there we go so now we're returning more samples returning more than now because


they're either from um from this one or from this one which is


really cool and we can we can double check that is the case so i can't see anything there


i return all documents and uh we'll go forward for doc in all docs


is it doc can i do let's just check that this works no okay so


let's see what's in there so document


meta cool document meta and then name


so let's just print all those out to make sure that is the case


um print cool so it looks like it's working so that is i think most


of the issues on the four requests i think there's one other maybe one other big one at least


okay so the typing with uh mypi so we can see that if we i'm not going to go through that now


uh but i can at least show you what that is so


you come to type check here


and we can see in the test with mypi there's a load of um typing issues


like here all of these all typing issues so there's something to fix as well but we're not


going to go through that now i think it's not that interesting yeah i think that's it for


for this video i hope it's been useful to just kind of go through that and like interesting to


see how haystack actually works and particularly haystack filtering works in the in the library


so yeah thank you very much for watching i hope it's been useful and i will see you in the next one


Training BERT #2 - Train With Masked-Language Modeling (MLM)

Okay in this video what we're going to do is take a look at how we would train a model,


a transform model, using mass language modeling or MLM. Now MLM typically would use it when we want


to teach a transform model like BERT to better understand the specific style of language in our


specific use cases and it consists of taking a input sentence or sequence, masking a few of the


tokens within that input sequence and asking BERT to predict the words that we have masked. So this


is pretty useful because we can take any chunk of text and process it through a masking function


and we can use that for training. We don't need to get label data which is really really useful.


So let's jump straight into it and what we first need to do is import everything we need. So we


need our tokenizer and model from transformers and we also need to import PyTorch. So do from


transformers import BERT tokenizer and BERT for mass lm. Then we also want to import torch


and then what we want to do is initialize our tokenizer and model. So our tokenizer is a BERT


tokenizer from Pretrained and we're using the BERT base uncased model. Let's copy that


and our model will be pretty similar so this time it's using BERT for mass lm.


Mass lm is just mass language modeling or MLM that I mentioned before. So


it's great. Now I'm going to be training this on a book that you can just get from the internet,


it's Meditations by Marcus Aurelius. The language in that is pretty unique so I figure this is quite


a good example. So I already have it downloaded and I've cleaned up a little bit. I will include


a link to that clean version of this so you can follow along if you want. So for me of course


I already have it downloaded here. So Meditations clean.txt and we are reading that in.


So p and all I need to do here is read and what I've done is split each paragraph within


Meditations by a newline character. So I will just split by newline and that should get us what we


want. Okay so we have this text now and what I want to do with this is actually tokenize it.


And this is just like we normally would with the transformers library. So we have our tokenizer up


here and we just pass our text into that. Now we're using PyTorch here so we want to return


PyTorch tensors pt and we also need to set the maximum length which for this bird model is 512


and then we need to set truncation to true and padding equal to max length. So this will


either truncate or pad each one of these sentences to the length of 512 tokens.


This should be returned tensors so there we go and here we are. So we see we have our input ids.


We don't need to worry about token type ids here and we have our attention mask which bird just


uses for calculating attention. I'm not going to really go into depth on any of that.


Now as I said before we need two things for training our bird model here. We need the


input ids which will have a mask token. Now we haven't created that mask token yet


and we also have our output labels which will not include that mask token. So before we mask


our input ids we need to create a copy of that which we will use as our labels.


So we write inputs labels and we set that equal to inputs input ids so our input ids tensor


input ids tensor and we clone that by first attaching it and then cloning it.


Okay and that that's all we need. So have a look at inputs again. I mean now we have input ids at


the top and if we go down to bottom we have a copy of those in this labels tensor.


Now what we need to do is create our mask. So with BERT when they are pre-training BERT they


use a few rules but at the core of that the main rule is that each token that is not a special


token has a 15% chance of being masked. So when I say special token I mean the separator and


classifier tokens which look like this and I'll point those out in a minute. In fact we can have


a look here. This is our classifier token this is 101 and you see that at the start of every sequence


and then at the end here we also have padding tokens. We also don't want to mask those.


So to create that 15% probability for each token what we do is use the torch rand function


and we use this to create a tensor of floats that have the equal dimensions to our inputs ids here.


Inputs input ids tensor.


Like so and if we check the shape of rand we see this 507 which is the number of sequences we have


and 502 which is the number of tokens that each sequence has. So we can see that we have


507. So if we were to use take this we see we get the same. Now we have a look in there.


It's just a set of floats from the value zero up to one. Now what we want to do


is mask roughly 15% of these or give each one there's a 15% probability of being masked


and the way we do that is mask anything that is under the value 0.15. So for example these ones


here they will be masked whereas these ones up here will not be masked. To do that all we write


is rand and we do less than 0.15. Now if we have a look at mask array we see that now these values


that were less than 0.15 have this true value which is what we'll be using to mask our tokens


later on. But at the same time if you remember the classifier token is always in the first


position within each tensor. So here we would have a classifier token here too and in fact all of


these would also be padding tokens. We don't want to mask any of those. So what we do to avoid that


is we add some extra logic. So I put that in brackets but actually we're going to just first


test the logic so I can show you what it's actually doing. So we have our inputs input ids.


Okay so these are the padding tokens these are the classifier tokens and what we do is just say


inputs input ids not equal to 101 which is our classifier token. Okay and now you see that we


get a fault wherever there is a classifier token and we want to do the same but for our padding


and to do that we multiply that so that is essentially adding it to the logic here.


It's like a like an and statement and now we are removing the padding tokens from that mask


and there's one more we can't see here but there's also a separate token which is represented by the


token id 102. So we also include that in here as well. Now all of these together we want to add


these onto the logic up here. Okay and now we will get our mask array. You see now we have faults


wherever we had the padding tokens we have faults wherever we had the classifier tokens but we still


have a few mask tokens in there so these true values here. Okay so that's our mask array


and now what we want to do is take the indices of each true value within each one of the


each one of these almost rows of the tensor. Now let's first do that with just one of them


so you can see how it works. So you take this one check the shape it should be 512 yeah so this is


just one row here and what we'll do is we'll say non-zero and this will return the indices where we


have the well where we have non-zero values that give you the true values.


But this is like a vector so what we want to do here is flatten that so we do torch flatten


and now we get almost a list but it's still a tensor we want an actual list and we just write


two lists. Okay so now these are the index positions for the true values within this first


row but we want to do for every row and to do that all we do here is we just use a for loop.


So we initialize our selection list here and we say for let's call it row or for i in mask array


shape zero.


So mask array shape zero let me just show you


it's the 507 rows that we have we want to do selection append and then we already have our


logic here so we want to append this but we're going to append this for every single one of


those rows so let's


for every single one of those rows so let's oh so sorry let's add a range on here


and let's have a look at what we get in the selection so we'll just have a look at the first


let's go to the first five ah sorry I just need to replace that with i there we go so now we have


indices for the first five rows here we have it for all of them of course we're showing you the


first five and there we go that's what we that's what we want and then what we want to do is we


can just copy this we want to set the values at each one of these indices equal to 103 which is


our mass token within each row of our input ids tensor so we go inputs input ids then here we need


to select those specific values and that is at row i followed by selection so a selection of indices


at i as well and we set those equal to 103 like so and now let's have a look at what we have in


our input ids tensor so now we can see we have these mass tokens where we saw the true values


before in our mask array and we haven't touched any declassifier or the padding tokens or the


separated tokens which are in there as well now our tensors here in the correct format but we


still need to process them through something called a data loader during training now to process them


through a data loader we need to convert them into a pytorch a dataset object and to do that


what we can do is write a create a class here which will handle this for us so it's going to


be meditations dataset and create the dataset object we need to pass the dataset class into


here so this is torch utils data dataset now there's a few things we need here we need the


initialization function which is just init and we pass self and encodings and here we're just


going to assign encodings to a attribute within this class


this class coding equals encodings now the data loader expects two additional functions


or methods that is the get item method and the length method length method is so that you can


check the length of the dataset that it's looking at and the get item is so that you can get a


dictionary formatted batch of those items so for get item we write this and we need self and then


we also specify the index and what we do is we return a dictionary and this is just going to


pass so we have we have the input ids key we have the labels key attention mask key and token type


ids key it's going to pass those back to the data loader when it requests this get item method


so we write torch tensor and we pass the values and the index of those values for key vowel


in self encodings dot items so that should should be okay


and the only thing left is the length method so we define length and here there's no input


parameters all we need to do is return the length of our dataset so it's return


length and we're doing self encodings and then we can just use any of the


tensors that we have in there but we'll do input ids


and we could even we could modify this to be like shape zero and get rid length at the end there


but i'll just stick with length for now


so that is our class which will handle the formatting of our data into a dataset object


and all we need to do is rewrite data set so this can be our new dataset variable


we have meditation dataset our class and in here we just pass our encodings or our inputs


like so okay and now we can initialize our data loader which pytorch will be using to load


our data during training so we write data loader equals torch utils data and data loader


here we want to pass our data set and then we also want to specify our batch size so i'm going to go


with 16 you can modify this depending on your your gpu or your your computer whatever however


much memory you have and then we also want to shuffle the data within their results so that


we're not extracting say the first 16 paragraphs all at once we're actually going to be extracting


16 from random parts of the book okay now we're ready to move on to actually training so first


we need to set up the or set up all the training parameters so so we first want to move the model


so we first want to move the model to gpu if you have a gpu and we check if we have a gpu using


let me show you first so torch device cuda


if torch cuda is available else torch


device cpu so we're saying here if we have a cuda enabled gpu use that otherwise we just use cpu


and you can see here that i do have it so we have this device type cuda and what we do is


assign that to the device variable here and we use that to move our model and everything across


to that device and we do that using model to device and we should get a big output here


we get all of this information that we don't need to look into that now we need to activate our


model our model is training mode so we just do model train let's make sure it's ready


and the final thing before we we set up our actual training loop is we need to initialize


our optimizer we're going to be using adam with weighted decay here so that's the adam optimizer


with weighted decay weighted decay just reduces the chance of overfitting especially with big


models like transform models so we're going to do from transformers import


import adam w and our optimizer is going to be adam w passing our model parameters and we also


need to passing a learning rate and we'll do one e to the minus five


so model parameters brackets at the end there okay


okay now we're fully set up we can actually begin training which is is set up as a normal


training loop in pytorch and first thing i want to do is just import tqdm this is


this allows us to create a progress bar during training otherwise we just sat down we don't see


any updates on training so we can actually start training again and we can see the progress bar


any updates on training which we don't want obviously i'm going to say so we do two epochs


you can obviously modify this as you want i'm just we're just seeing how this all works so


i'm not going to train it that much and you want to be careful of training transform models for


too many epochs they over fit very easily and we'll do four epoch in range epochs


and then here we want to set up our training loop so to do that we want to wrap it within a tqdm


function there and we just pass our data loader what did i call it data loader up here


and that leave equals true this just leaves the progress bar rather than replacing it with every


new epoch and then we run through each batch within our loop so the style batches of 16 items at time


and we first want to initialize initialize our calculated gradient so with every loop we will


calculate gradients and we first we don't want to start with with gradients already calculated we


want to initialize them or set them to zero so we do optim zero grad then we want to pull all of


all of our tensors that we require for training so input ids of course first one


and that will be equal to batch then in here we access our input ids


and additionally you see before we moved our model to our gpu we also want to do that for


our tensors here as well so we say to device okay and we follow this structure for our


other tensors as well now for mass language modeling we don't need to


do anything with token type ids so we just ignore those we have our attention mask we do need that


and we also have our labels which we we do need of course and with that we can process everything


so now we do outputs model and we pass out input ids so inputs we want to specify the attention mask


so just copy that and we also need to specify our labels which is labels


okay now let's just extract the loss from those outputs so we get a loss tensor there and


what we do here is we use the backward method which calculates loss for every parameter


from our in our model and from that we can calculate the gradient update using our optimizer


so using that we have optin and we call the step and this will take a step to optimize all of the


weights within our model based on the loss now final little bit here this is just


you know aesthetics i want our loop i want to actually see certain bits of information in that loop


so all i do is loop set description


and here i just want to show the epoch which is just epoch


and then i also want to see the loss in the post-processing


so we do loop set postfix and we do loss loss item so item here just pulls out the exact value


within that within that loss tensor up here okay that should be everything let's let's go


see what we have there we go so now we're training see the loss is going down slowly


and that's that's it so we're now training our transform model using meditations by Marcus Aurelius


with mass language modeling it's really not that hard i mean there is quite a bit to it


but i think once you once you do it it's reasonably straightforward and the fact that you can do this


on basically any set of text using just a mastering function is incredibly so so useful


so we don't need to you know go out looking for labeled data anywhere which is amazing


so that's that's it for this video i hope you enjoyed it and i hope you enjoyed this video


that's it for this video i hope it's been useful i know it's a bit of a long one but


Intro to Dense Vectors for NLP and Vision

Hi and welcome to this video. We're going to start a new series on embedding methods for NLP,


but we're also going to have a look at other embedding methods as well. So mainly we're going


to be focusing on language dense embeddings. We might have a look at sparse embeddings,


but we've already covered that before. So I'm not 100% sure on that, but definitely dense embeddings.


We're going to also have a look at how we can build dense embeddings for images and maybe some


other media formats as well. So I think this series of articles and videos will be pretty


exciting. Now what I want to start with is having a look at, we're basically quickly introducing what


dense vectors and dense embeddings are. And whilst we do that, I'm going to refer a lot to


Word2Vec because that's the first widely adopted version of this. And then we're going to have a


look at sentence embeddings. So how we can build sentence embeddings using the sentence transformers


library. And we're going to go through the code for that as well. Then we're going to have a look


at Q&A. So Q&A is quite interesting I think. And we're going to focus on Facebook AI's dense


passage retriever for that. And again, we are going to go through the code for that as well.


And then another thing that I think is quite exciting is image and text embeddings. So to do


that, we're going to have a look at the new vision transformer. So I think all of that's pretty cool.


So let's jump straight into it. So I think the first question we want to ask is why would we


use dense vectors in the first place? Now we have two options when it comes to representing text.


And that is we can represent it as a dense vector or as a sparse vector. Now sparse vectors are good


if we're going to focus on the syntax and the words that we're comparing. So if we had two sentences,


Bill ran from the draft towards the dolphin. And then we said the opposite. So Bill ran from the


dolphin towards the draft. Both of these sentences have the exact same words in it, but they have


different meanings, right? So in one of them, Bill is running away from a draft and the other one is


running away from a dolphin. Now when it comes to sparse vector representation, we'd find it difficult


to correctly identify these as not being the same sentence because we tend to represent


words one by one in some sort of one-hand coding and then compare those vectors. Now we can also


use n-grams so we can put two words together. And in that case, we would identify that there is a


difference, but it's not that effective. And then we also want to consider where we have different


words for the same meaning. So for example, if you want to say hello to someone, say hi, hello,


hey, I'm sure there's a million other ways of saying it. And sparse vector representations


would view these as different words. So whereas sparse vectors are very good for or comparing the


syntax of text, it's not very good at comparing the semantics or the meaning behind text. And


that's where we want to start using dense vectors. So we can see dense vectors as pretty much a


numerical representation of the semantic meaning behind some text. And we can actually


visualize a lot of these relationships. So towards around 2013, we had Word2Vec, which was the first


very popular dense vector embedding for words. And around that time, we had a lot of people


showing that you had things like this, what you can see on the screen, where, for example,


we'd have days of the week clustered together, or we would have months or other related abstract


topics represented or clustered together in our highly dimensional space. Now, of course,


this is a 3D graph. When we're actually building these dense vectors, we have many more dimensions,


more towards 500, 700, 800 or so. So this is obviously a simplified version of that.


And not only will we find that similar words are clustered in the same area, but we also find


that we can perform what I think is best described as arithmetic on words. So this is a very popular


example that came from around the same time as Word2Vec. If you want references and everything,


you'll be able to find them at the bottom of the article that this video is attached to.


If you need the article, it's in the description. Now, what we'd find is if we took the vector for


king, subtracted the vector for man, added the vector for woman, we would not get the exact


vector for queen, but we'd get very, very close. So the nearest vector would be the vector for queen.


And I mean, I think that's super interesting. And this is from the start of when we had these vector


embeddings. So this is eight years ago now, and they've just gotten a lot more advanced in that


time. So as I said, these examples are coming from the era of the Word2Vec. And Word2Vec was


one of the earliest versions of these dense representations. And going from the name,


we know it's word2vector, so we're converting words into vectors. Now, how this worked, there


were two different methods. We had the skipgram method, which is what you can see now, which is


where we take one word, and we would take the sparse vector encoding for that word on the left


that you can see. And then we would, in the vector on the right, we would have a one-hot encoding for


all of the words that surround that first word. So in this case, we have fox, and that's surrounded


by the words quick, brown, jumped, and over. And this would be run through a simple feedforward


neural network. And we would go through this compression stage, and it is within that


compression stage that we would build our dense vector representation for fox. And that would


simply be a neural network being optimized to go from fox and predict the quick, brown, jumped,


and over. And this would be done many times over for every time that word appears in a big corpus


of text with its multiple contexts. And what that does is it just builds up like a numerical


representation of that word. And then there was the other approach, which words they also use,


which is called continuous bag of words. And it's basically the same. We're just swapping the order


of the transformation. So on the left, we have all of our context words. And then on the right,


we would have the word that we're focusing on and we're building the embedding for.


Now, Word2Vec really seemed to act as the catalyst for a lot of other vector embeddings from Word2Vec.


For example, we have Sentence2Vec, Doc2Vec. We even had this one that I found when I was


researching for this, which is called Batter Picture2Vec, which is a vector embeddings for


the Major League Baseball players. So you've got a lot of different 2Vec methods that came out of


the woodworks after the original Word2Vec. And then we also had other ones like Glove as well,


which is worth a mention. Now, nowadays, Word2Vec is pretty outdated and we wouldn't really go ahead


and use that. So I'm not going to spend any more time on it. And we'll just move on to having a


look at Sentence Similarity. So you can see Sentence Similarity as very similar to Word2Vec


in that we're building these dense representations, but rather than representing a single word,


we're representing a sentence or a paragraph. And the way that this would be done is using


the current transform models. So BERT was the first example of doing this. And BERT by itself,


you can build embeddings, but it's based on a token by token embedding. So within BERT,


you have all of these different embeddings, but they each represent a single token. So


what the guys at Sentence Transformers did is they trained like a Siamese, they call it a Siamese


BERT, where they had two BERTs and they were trained in parallel and they output a single


vector for the full input that was input into the model, which was around 128 tokens at max.


Now, this allowed us to build a single vector for sentences. And that's very good because then we


can start comparing sentences and paragraphs. So let's have a look at how we can actually build


that in code. So the first thing you'll need to do is pip install Sentence Transformers. Now,


I've already done this, so I'm not going to rerun it. But if you don't have Sentence Transformers,


you will need to install it. Then after that, all you want to do is you want to write from


Sentence Transformers. We want to import the Sentence Transformer object. And from there,


we can just initialize our model super easy. We just write model, Sentence Transformer.


And then in here, we just need to type our model name. Now, if you Google Sentence Transformers


or Spert, you will find the web page for this library. And it has loads of different models


on there. One of the highest performing ones that I found on there at the moment is called all


mp-net base v2. Okay, so we just execute that. And usually, you will need to download the model.


So you will see a load of loading bars or progress bars. That's fine. It's just downloading the model


for you. I already have it downloaded, so I don't need to run it again. And then what we need is a


set of sentences so that we can actually compare, we can compare these and look at what the Sentence


Transformer believes is the most similar. Now, all of these are completely random, but we have


this one here. The bees decided to have a mutiny against their queen. And I just rewrote that in


a way that we don't have any matching words between the two sentences. So we have flying,


singing insects, rebellion, opposition to the matriarch. Now, the meaning there is pretty much


the same, maybe not exactly the same, but pretty much. But there are no shared words other than,


I think, to and the. Yeah, to and the. So in terms of sparse vector encoding,


this wouldn't score very well. But we'll see that with dense vectors, it will. So the first thing


we want to do is encode our embeddings. So we write embeddings because model dot encode sentences.


Okay, and then let's have a look at what the outputs or at least the shape of what outputs.


And we see that we get seven vectors or seven embeddings, each one with the dimensionality


of 768 values. And we can use cosine similarity to compare all of these. Now,


the easiest way to do this is we just import cosine similarity from sentence transformers.


So sentence transformers dot util import cos sim. Okay, and then what we do is calculate the


the cosine similarity scores between all of our vectors. Now, I want to compare the final item


here. So this last one against the rest of them, because I want to see that this is the most


similar. So I'm just going to select that. So embeddings and we're just taking the last vector.


And then I want embeddings, well the remaining of them. So all the vectors except from the last one.


And let's just have a look. We have seen that we have something that seems pretty obvious. So


this one here is the most similar by quite a bit. 0.6. The closest is 0.19 here. So it's


definitely calculating that is a lot more similar than the other ones. So if I take the argmaps of


that, we should see, so three and take the item. And if we go sentences and we put that sentences,


we index number three. We see, okay, the bees decided to have mutiny against our queen.


So it correctly identified that these two, this


and this are far more similar than the rest of the sentences, which I think is very cool because


there's not even any similar words in there. And even as a human, it's kind of, you know, the bees


flying, singing insects and matriarch and queen. You know, it's not obvious. So I think that's


really cool. Another popular use of embeddings for language applications is question answering.


Now, question answering can be done with a few different, let's say architectures. And one of the,


I think, most popular ones is open domain question answering. Now, the structure of open domain


question answering is what you can see on the screen at the moment. So we ask a question that


gets passed to something called a retriever model. The retriever model contains a question encoder,


which encodes the question, passes it along to our index database. And within there, we will have a


set of contexts. Now, contexts are usually a paragraph that contains the answer to our question.


And DPR both encodes our questions and encodes our context into the same vector space. So what we


would get is, for example, if we had a question, what is the capital France? And then we also had


a context, the capital France is Paris. DPR would attempt to encode both of those into the same


vector space or very, very close by. So the vectors produced by both of those would be very, very


similar. So all we're doing in that index database is finding the most similar embeddings to our


question embedding. And then from there, we pass that along, we pass our context and the question


again to our reader model. Here I've used a BERT Q&A model. It doesn't have to be BERT, it can be


any reader for question answering. And then that outputs the specific part of our context, which


contains our answer. So in the previous example, it would output Paris, hopefully. Now, we had that


DPR retriever model. DPR is Facebook AI's dense passage retriever. And it actually consists of


two smaller encoders. We have question encoder and context encoder. Now, during training, what


we do is we train both of these encoders in parallel, and we pass questions and their equivalent


context to the question and context encoder, respectively. And we optimize based on the


based on a contrastive loss function. So we compare the vectors from our question encoder


and the context encoder, and we try to minimize the difference between them, the question and


context pairs. And that's how we build the DPR model. That's why it works for question answering.


So it's not like our sentence transformers where they are just a single model and they're


used to identify very similar sentences. This is used to identify not very similar sentences,


but very similar question and context pairs. And we will see difference in a moment when we


when we go through the code. So let's get started with that. So come down here. And first thing we


probably want to do is initialize our context encoder and our question encoder from DPR. Now,


we're going to use the Hocking Face Transformers library for this. So if you do not already,


you'd have to pip install transformers. Now, if you pip installed sentence transformers,


that does include transformers as a prerequisite. So if you installed that already, you should


already have transformers as well. So first thing we want to do is from transformers,


we want to import a fair few classes here. So we need both the model or the encoder and tokenizer


for each for both our context encoder and our question encoder. So let's do the context encoder


first. So write DPR context encoder tokenizer and DPR context encoder here. And then as well as that,


we also want the question encoder tokenizer and question encoder. So we write DPR question


encoder tokenizer and also DPR question encoder. And that's all we need to import. So let's run


that and then we can go along. We can go ahead and initialize our tokenizer model.


So we have the context model. Now this is going to be the DPR context encoder from Pretrained.


If you've ever used Hugging Face transformers before, you should recognize this from Pretrained.


We're just going to load in a model, which we can find on the HuggingFace.co slash models website.


So if you go to that address and you type in what I'm about to type in, you will find


that it comes up. So I'm going to type Facebook slash DPR context. So ctx encoder and we want


single enqueue base. And I'm going to copy this because we are going to use it again in just a


moment for our context tokenizer. So context tokenizer equals DPR context encoder tokenizer


from Pretrained again, from Pretrained. And then again, we want the same model name in there.


Okay, so they are our context side of the model, but we also need to get the question side. So


we've got our context encoder and tokenizer. Now we want the question encoder and tokenizer.


So write question here and here, and we're just replacing everything where we've put ctx with


question in here. So it says question and this as well. And then in the model, we are just


replacing ctx again with question. So pretty straightforward. Now I'm going to run that with


you. If you haven't already got these cached on your machine, it can take a little bit of time


because we're downloading four sets of models and tokenizers. So it can take a little bit of time.


Now I already have them, so I don't need to wait for that.


Now, first thing I want to do is set up a set of questions and context. So I have three questions


here. Well, you can read them. I'm not going to go through them all. And then we have context.


Each question has a couple of contexts that are kind of relevant, but then just one that


is actually the answer. And we also have inside here, I've also put in the questions themselves,


because I want to prove that this is not just a sentence transformer where it's finding the most


similar sentence. So it should, when we have these questions, it shouldn't return like this one,


it shouldn't return what is the best-selling sci-fi book. It should instead return the


best-selling sci-fi book is Dune. So we should see that there is a difference between using DPR


and using sentence transforms. So run that. And then what we want to do is tokenize everything.


So we're going to tokenize our context. So I'm going to write xb tokens and we want the context


tokenizer. And then in here, we're going to pass our context. And then if you use


hooking face transformers, you should recognize this as well. So max length here. So for this,


I'm going to put 256 and I'll set padding equal to max length. We don't need to truncate anything.


I don't think. No, they're all very short. So this max length, we could even reduce it to


something pretty small, but I'm going to leave it at that. So we'll pad up to the max length


and the only thing we do need to include here is that we want to return PyTorch tensors.


So return tensors equals pt. Okay. And then what we can do is we write xb embeddings. So this is


how we build our embeddings, our context embeddings. xb embed, I'll just call it xb,


is equal to model, the context model, ctx model. And then in here, we pass our tokens,


xb tokens like that. And then for our questions, we do exactly the same thing,


but of course we just replace the context part of it with questions. So here we have


question tokenizer, we have questions and we have the question model. And then here I'm going to


rename xb to xq. So I'll query. Okay. Let's have a look at what we get. So first let's have a look


at what we have inside xq. So we'll see that we have a few different tensors in here. So I'll


just write xq keys to see what we have. You can see that we actually only have one output here,


so the pooler output, which is fine because that's what we need. So we write xq pooler output.


And these here are our embeddings. So we can write shape to see the shape of those embeddings. So we


have three vectors. So the number of questions that we passed up here and each one of those


questions has been encoded into a embedding of 768 dimensions. So that looks good. And we can


do the same for xb if we want as well. It's exactly the same. So write xb and we'll see the shape.


Just at this time we have nine vectors because obviously we have more context than we do with


questions. So what we want to do now, I'm going to import torch. So again, this should have been


installed already with Hugging Face transformers and also Sentence transformers. So if you've got


a NIST file, you don't need to worry about installing this. What I'm going to do is go 4i


and then the query vector in xq pooler outputs. So I'm going to enumerate that.


So what I'm doing here is I'm going to run through, I'm going to create a loop to go through each


query and to get the most similar vector from xb. So from our encoded context.


So we write probs equals cosine similarity. So these are our similarity scores.


Doing exactly the same as we did before. We're just going to write xqvec. So the single vector


at the moment and from here we just want xb pooler outputs. And from there we want to get the argmax.


So the maximum argument, so the highest score in our probability array here. So torch argmax


and here we have probs. And then what I'm going to do is I'm going to print the current question


that we're asking. So questions i. Now I'm going to print the context which has been chosen from


our argmax. So we just write context argmax. And then I'm just going to put this in here so we


have a little bit of separation. Let's have a see what we have. So we get what is the capital city


Australia. Now remember this exact question was also in our context and it's not returning the


the exact sentence back to us or the exact question back to us. It's actually returning us the answer.


So Canberra is the capital city of Australia. Now second one as we had hoped the best-selling sci-fi


book has been chosen to do in here. And then I just wanted to include this one as well to point


out that it's not perfect. It doesn't always get things right. So in this case it didn't find the


correct answer of how many searches are performed on Google. If we have a look at the context.


So the correct answer should have been this one here. So Google serves more than two trillion


queries annually. So it didn't get that one but the other two it did get despite


having the actual questions in there as well.


So again I think that's really cool and I think Q&A is something that has a lot of potential


in many businesses around the world. So I think that's a really a very cool one to use.


Okay so the next one to cover is a mix of of language and also a vision. So


recently computer vision has had a few advances from the discipline of NLP. So in NLP we've been


using transformers for a reasonable amount of time now and transformers have proven to be


incredible models for language. And very recently transformers have been applied


to computer vision as well which is very cool I think. And what we're finding is that a model or


an architecture that can be used for language can also be used for computer vision. And that's


I think that's super cool. So I want to show you one of those models or as briefly touch upon one


of those models we will go into in more detail in a future article and video. But for now I'm


just going to mention it we have the the vision transformer which is a very recent I think the


paper is January 2021 if I'm not wrong. And although we don't need a vision transformer to


build an embedding for an image I think the fact that we can use it is a great thing.


I think the fact that we can use it is pretty cool and we can really do it very easily with


Hocking Face transformers as we will see when we when we go through the code. Now a very interesting


use of this is to actually take two different encoders both transformers. The text encoder is


more of a traditional transformer obviously and the image encoder is our new vision transformer.


And we can actually train them together like we did with DPR Dubai encoder architecture.


And what we can do is train it to put images and language so language that describes an image


and map them to the same point in a vector space or very close at least. And that's what I've tried


to visualize as you can you can see on the screen now. So we have two logs running we process that


through our text encoder and we get a very similar vector to if we took the picture of two dogs running


and process that through an image encoder which would be our vision transformer.


So I think that's I don't know for me I think that's so cool. Now


I'm going to be using these three pictures that I got from Unsplash. If you want to see the


photo credits they will be in the either in the article if you're reading the article or they'll


be in the video description if not. And what I'm going to do is we have these three pictures we're


going to encode those and I'm also going to encode these three captions and a few other captions as


well. And we're going to see if they match so we're going to perform a similarity or cosine


similarity search across them and see which pairs match the closest and well we'll see it's the


results are pretty pretty cool in my opinion. So let's let's jump into it we again we're going to


be using transformers and we're going to be using a new model from from OpenAI which is for the


image and text. It's similar to DPR where DPR is in question and context encoding. Clip is using


two encoders to do image and and caption encoding which is pretty cool. So we're going to do from


transformers import clip processor. So I'm kind of viewing this processor as what we could call a


tokenizer in the in typical language transformers and then we want the clip model. Okay so this


contains both encoders for us so we don't have to mess around like we did DPR where we were importing


you know we imported four classes here we're just importing the two.


And then what I want to do is we'll just initialize those so again very similar so we do


clip model from pre-trained and in here we write OpenAI


OpenAI clip VIT so it's the vision transformer this VIT you see here refers to the


the vision transformer which clip is using or is based on at least the vision aspect


and we want to write base patch 32. So I mean we'll go into more detail but the patch part of


that is referring to the way that the model almost tokenizes your images it splits an image


into into different patches and that's the patch size for patch 32 there. So we also want the


processor which again I can we can kind of see that as akin to our or equivalent to our tokenizer


and we're just doing this for language models and again I'll just copy that across.


Okay so model processor looks good let me run it.


Okay again I already have it cached so it won't it won't download for me and you'll get this


thing here don't worry about it still works. Now I'm going to copy in the the code I'm using to


get the photos so I have the photo URLs here I'm using a pill to create the image object


and I'm using requests to actually get the image from the URL that we have here and then down here


I'm just going to show you what what images we have so I actually need to get my live in there


as well so import live in a pyplot. That's prt do you need and numpy as well.


Okay and we'll see those images that we saw before so we have the dog the puppy or dog running


the dog hiding the entry and then we have the two dogs running. Okay so they are images and


we've stored them in in images here okay and the next part are captions so I've just written these


six captions the first three are actually the captions and then the other three I just I made


it up I included like trees and parking there because they look like well there's a tree here


and there's a park here so make it try and make it a little bit more difficult but I mean that


reasonably straightforward still I think and then to create our imagine you can see these as tokens


we do inputs so processor similar to our tokenizer again and we have we have a few inputs here so we


have the text and we want to input our captions and then we also have images and of course we


just input our images and then we want to return the return tensors or tensor equal to pt


and we set padding to true okay


return return tensors pt


okay and if we just can we have a quick look at what we have here so we have our input ids


pixel values and so on so input ids we also have attention here as well so these first two


are for our text and then pixel values are for the images


and now what we want to do is create our encodings so in here because we're using the


the clip model we're actually going to perform the encodings and it's also going to do the whole


similarity checking for us as well and identify which ones of which images and captions are


the closest pairs or what it's going to do is go through each image and find the caption that


it believes belongs to it so like before we just write inputs here and i think maybe let's have


a look at what we have in our outputs so we can see we'll have a few things here that i think are


pretty pretty useful so the logics per image and per text so for these we can for each of our texts


we can use this to get the most probable image that is assigned to each each caption and in


logics per image we can use these to find the most probable caption for each image and then


so what we were doing before where we're just extracting the embeddings we can also do that


and maybe i'll just copy in the code for that as well so we have the text output here and then


and so we have the text embeddings here so we can extract those if we want and we also have the


image embeddings in here as well and then a little further down we have the the logics somewhere pool


output here yeah so i've did the pooler output to my logics okay so let me just close that and i


do believe we also have a few more so let me let me just show you those quickly yeah we have we


have a few a few tensors there as our vision model output text model output as well now what we'll do


is i'm going to paste this code in and so here i'm going to go for for image in in each image


i'm going to iterate through i'm going to get the arguments so the the the caption that it believes


or is predicted for that image and then we're going to show it and we're going to print both


out and let's see if they if they match oh so i'm getting ahead of myself there so we also need to


so the probability there is the probes equals outputs and we want the logics pair image


and we'll take the arg max while we're here so dim equals one for that and let's have a look


what we get we'll see that we get this so it's predicting caption two caption zero and then


caption one for our three images let's loop through that and we'll see we get a dog running


cool a dog hiding behind a tree and then two dogs running as well which i don't know for me


maybe because i'm usually working with language i think seeing both language and images together is


i don't know really cool super i don't know fascinating that it actually works like that


so easily so another thing that we i want to show you very quickly i'm just going to copy the code


in because i don't want to go through all of it um it'll take a while so we just have the embeddings


so these are the embeddings if we wanted to extract them and do what we did before with them


or if you wanted to take these embeddings put them in a like a vector index somewhere a vector database


we can we can get our query so i'm going to do a dog hiding behind a tree and we can get our


the context or not the context the the images the image embeddings again like before we do the


the similarity so the cosine similarity we get the highest one is the second one here so it's


looking pretty good and from there we get our prediction which is the arg max so we'll take


number one and let's have a look what our prediction is then so we will plot that we'll


show you the image again we have prediction so it's showing us the dog hiding behind a tree


for our query which is a dog hiding behind a tree so again super cool now that's it for this video


we've i think covered quite a lot of embedding methods we've had a look at some introduction


to den inspectors with where to back and where it sort of came from how it quickly evolved


and we've had a look at sentence embeddings and sentence transformers moved on to q a with


facebook ai's dpr and now we've had a look at the new vision transformer and how we can use that


with other transformer models to build these really cool sort of cross media embeddings that


we can compare which is sort of blown me away a little bit now that's it for this video but


like i said this is the first video and article in what will be a series on embeddings so there's


a lot more to come but for now thank you very much for watching and i'll see you in the next one


Multi-Class Language Classification With BERT in TensorFlow

Hi, welcome to this video on multi-class classification using transformers and TensorFlow.


So I've done a video very similar to this before, but I missed a few of the final steps at the end,


which were saving and loading models and actually making predictions with those models.


So I've made this video to cover those points as a lot of you were asking for how to actually do


that. So we're going to cover those and we're also going to cover all the other steps that lead up to


that. So if this is the first video on this that you've seen, then I'm going to take you through


everything. So I'm going to take you through sourcing data from Kaggle that we'll be using,


pre-processing that data, so that's tokenization and encoding the label data.


Then we're also going to be looking at setting up a TensorFlow input pipeline, building and training


the model. And then we go on to the few extra things I missed before, so the saving and loading


the model and also making those predictions. So that's everything that we'll be covering. And


what I've done is left chapters in the timeline so you can skip ahead to whatever you need to see.


So we'll jump straight into the code. Okay, so here we're just going to download the data. We're


going to be using the sentiment analysis on the movie reviews data set, which you can see over


here on Kaggle. We can download it from here if you just click on train.tsv.zip, download it and


unzip it. I'm just going to do it through the Kaggle API here and then unzip it with zip file


here, it's just a little bit easier. So I'll run that. And I do have a video on the Kaggle API.


So I'll put a link to that in the description. Okay, so we've got our data, you can see in the


left here. And it's going to import pandas and view what we have. Just read from CSV.


And we're using a tab limited file here. So we need to use the tab separator.


And let's just see what we have. Okay, we have the sentiment here and we have the phrase and that's


all we really need. So on the phrase here, we're going to be tokenizing this text to create two


input tensors, our input IDs and the attention mask. Now, at the moment, we're going to contain


these two tensors within two non-py arrays, which will be of dimensions, the length of data frame


by 512. 512 is the sequence length of our tokenized sequences when we're putting them


into BERT. So when tokenizing, all we're going to do is iterate through each sample one by one,


and assign each sample or each tokenized sample to its own row in the respective non-py array.


And we'll just first initialize those as empty zero arrays. So we'll do that. So first we need


to import non-py. And then like I said before, the sequence length is 512. And then our number


of samples is just equal to the length of our data frame. And with that, we can initialize those


empty zero arrays. So one will be xids, which will be our token IDs. And that is initialized with


empty zeros. And then we pass the size of that array here. So number of samples or the length


of the data frame by the sequence length of 512. And then we can copy that. And we do the same


thing for xmask, which is our attention mask. And then let's just confirm we have the right shape


as well. And do that like this. Okay, so we have 156,000 samples and 512 tokens within each sample.


So now that we have initialized those two arrays, we can begin populating them with the actual


tokenizer text. So we're going to be using transformers for that. And we are using BERT,


so we'll import BERT tokenizer. And then we just want to initialize the tokenizer.


And what we'll do here is just load it from pre trained. So I'm using BERT base case.


And then what we can do here is we'll just loop through every phrase within the phrase column.


And just enumerate that so we get the row number in i and the actual phrase in the phrase variable.


So here phrase. And then we want to pull out our tokens using the tokenizer. So we get encode plus.


And then we have our phrase, the max length, which is sequence length. So that's 512.


We're going to set truncation to true. So here if we have any text which is longer than 512 tokens,


it will just truncate and cut it off at that point because we can't feed in arrays of or


tensors of different shapes into our model. It needs to be consistent. Likewise, if we have


something that is too short, we want to pad it up to 512 tokens. And for that we need to use


padding equal to max length, which will just pad up to whatever number we pass into this max length


variable here. And then I want to add special tokens. So in BERT we have a few tokens like


this, which means the start of a sequence. This, which means separator, and that is either


separating different sequences or marking the end of a sequence. And also we have a


pattern token, which we'll be using because we set padding equal to max length. So if we have


an input, which is maybe 412 tokens in length, or let's say 410, both of these will be added to it,


which will push it up to 412. And then we'll add 100 padding tokens onto the end, push it up to


512. So that's how that works. So obviously we do want to add those special tokens, otherwise BERT


will not understand what it is reading. And then because we're using TensorFlow here,


we're going to return tensors equals tf for TensorFlow. And then what we want to do here,


so we've pulled out our tokens into this dictionary here. So this will give us a dictionary


that will have different tensors in there. So we'll have our input ids and attention mass.


And we want to add both of those to a row within each one of these zero arrays. So to do that,


we want to say x ids. And then this is why we have enumerated through here. So we have this i.


So this tells us which row to put it in. And then we want to add a row of these zero arrays.


So we'll add this row to put it in. And then we want to select the whole row.


And we set that equal to tokens, because input ids.


Now, as well, we do the same for the x mask.


But this time, rather than input ids, we are going to say equal to attention mask.


Okay, so let's just have a quick look at what we have in x ids now.


Okay, it's just zero. So now let's run this. And now if we rerun this, we can see okay, now


we have all these other values. So first, this 101 is the CLS token that I mentioned before.


So the start sequence token, and these zeros here, they are all padding tokens. So obviously,


at the end here, we almost always have paddings unless the text is long enough to come up to this


point, or if it has been truncated. But here, we can see sort of structure that we would expect.


And we see some duplication here. So we have 101, then we have this 138, 1326. And if we look at our


data here, we can see okay, that's why because the first few samples we have like segments of


the full sequence here. So there is some duplication there. So that all looks pretty good.


And I think let's just have a quick look at the x mask. So here you can see something different.


We have these ones and zeros. So x mask is essentially like a control for the attention


layer within BERT. Wherever there's a one, BERT will calculate the attention for that token.


Wherever there's a zero, it will not. So this is to avoid BERT making any kind of connection with


the padding tokens. And actually, these padding tokens are in reality not there. We want BERT to


just completely ignore them. And we do that by passing every padding token as a zero within


the attention mask array or tensor. Now, as well as that, we also want to encode our label data.


So at the moment, we can see here we have the sentiment. And this is a set of values from zero


to four, which represent each of the sentiment classes. So here we have sentiment labels of zero,


which is very negative, one somewhat negative, two neutral, three somewhat positive, and four positive.


So we're going to be keeping all those in there, but we're going to be one-hot encoding them.


So to do that, we will first extract that data. So we'll go


put into array. We want to put df sentiment, which is the column name.


And we want to take the values. And if I just show you what that gives us,


just gives an array. And we have up to four, so zero, one, two, three, four.


And that's good. And now what we need to do is initialize again, a zero array. So we'll do empty zeros.


And in here, we're going to do it the length of the number of samples, because again,


we have the same number of samples in our labels here. So the length of the data frame.


And then we want to say the array max value plus one. Now, this works because in our array,


we have zero, one, two, three, four. So the max value of that is four. Okay. And we have


five numbers here. So if we take the four plus one, we get five, which is the total number of values


that we need. And this essentially gives us a number of columns in this array. And we need one


column for each different class. So that gives us what we need. And we can just check that as well.


So we have labels.shape. And here we see we have the length of the data frame, or number of samples.


And we have our five classes. Now, let's have a quick look at what we're doing here. So


print out labels. Okay, we just had zeros. Now, what we're going to do is we're going to do some


fancy indexing to select each value that needs to be selected based on the index here. And set


that equal to one. So for these first three examples, we will be selecting this, making


it equal to one. And then this because this is in the second. So we've got zero, one, two,


these are the column indices. So first one, that will be set to one because we have one here.


This is number two. So we select number two, and two here as well. And then for three down here,


we'd have a one here. Okay. So to do that, we need to specify both the row. So once again,


one row at a time. So all we need here is a range of numbers, which covers from zero all the way


down to 156, 0, 6, 0, which is our length here. So to do that, we just go MP a range,


we have a number of samples. And then here, we need to select which column we want to set each


value to, or select each value for. And that's easy, because we already have it here. This is our


array. So we just write array. And then each one of those selects once that equal to one.


So let me just put it there, I want to put it here. Okay, and now let's rerun this cell as well.


Okay, and now we can see we have those ones in the correct places. So that's our one hot encoding.


And now what we want to do is sell our data here, and put into a format that TensorFlow will be able


to read. So to do that, we want to import TensorFlow first. And what we're going to do is


use the TensorFlow data set object. So this is just a object provided by TensorFlow, which just


allows us to transform our data and shuffle and batch it very easily. So it just makes our life


a lot easier. So it's a data set. And we're creating this data set from tensor slices,


which means arrays. And in here, we're going to pass a tuple of x ids, x mask, and labels.


And to actually view what we have inside that data set, we have to use specific data set methods,


because we can't just print it out and view everything. It's a generator. So what we can do


is just take one and that will show us the very top sample or after we batch it, the very top batch,


and just print that out. And you see here, okay, we have this take data set shapes. And we have this


tuple here. So this is one sample. And inside here, we have a tensor, which is of shape 512.


So this is our very first x ids array or row, sorry. So this is like doing this and getting


this. So this is what this value is here. This is the size or the shape. Okay. And then we are


the same for the x mask, which is the second item here in index one. And we also have four labels as


well. You can see here. Okay, and that's all good. But what we need now is to merge both of our input


tensors into a single dictionary. So the reason that we do that is that when TensorFlow reads


data during training, it's going to read or it's going to expect a tuple where it has the input


in index zero and the output or target labels in index one. It doesn't expect a tuple with three


values. So to deal with that, what we do is we merge both of these into a dictionary, which will


be contained within tuple index zero. So first we put in that function. And what we're going to do


is just apply whatever is inside this function to our data set. And it will reformat everything


that data set to this format that we set up here. So our input IDs, just change that to input.


And the masks and we have the labels. And all we want to do is return input IDs and masks together.


And we're also going to give them these key names so that we can map the correct tensor to the


correct input later on in the model. And we go input IDs. We have our attention mask,


which goes to our masks. And then that is the first part of our tuple.


And then we also have the labels, which is the second part. Okay. And that's all we need to do


for creating that map function. And then like I said before, data set makes things very easy for


us. So to actually apply that mapping function, all we need to do is data set dot map, map function.


So now let's have a quick look at what we have. See if the format has changed or the shape.


Okay. You can see now we have, so it's all within a tuple. This is the one index of that tuple.


This is a zero index of that tuple. And in the zero index, we have a dictionary with input IDs,


which goes to our input ID tensor and attention mask, which maps to our attention mask tensor.


Okay. So that's great. And now what we want to do is actually batch this data.


So I'm going to use batch size of 16. You might want to increase or decrease this,


probably mostly dependent on your GPU. I have a very good GPU. So I would say this is at the


upper end, the size that you want to be using. And what we do here is data set dot shuffle.


So this is going to shuffle our data set and the value that you should input in here. I


tend to go for this sort of value for this type of size data set. But if you notice that your data


is not being shuffled, just increase this value and then batch. And then we have the batch size.


Okay. So split into batches of 16. So first shuffle the data and then we batch it. Otherwise,


we would get batches and just end up shuffling the batches, which is not really what we want to


do. We just want to actually shuffle the data within those batches. And then we want to drop


the remainder and set that equal to true. So that is dropping. So we have batch size of 16.


If we had, for example, 33 samples, we would get two batches out of that, 16 and 16, and then we'd


have one leftover. And that wouldn't fit cleanly into a batch. So we would just drop it. We would


get rid of it. And that is what we're doing there. And then let's just have a look at what we have


now. So now see this has changed. So we still have that tuple shape where we have the labels here


and the inputs here. But our actual shapes, our tensor shapes has changed. So now we have 16


samples for every tensor. Okay. So that is what we want. That's good. We've got our full data set.


Our training data set here. And what we might want to do is split that into a training and


validation set. And we can do that. It's set in the split here. So we're going to use 90%.


You can change this. So we're going to 90% training data, 10% validation data. And what we


need to do is just calculate the size of that split. So the size of the training data in this


case. So we'll take the x ids shape. Okay. Or actually we can just set the SQL to the


value that we defined up here. Okay. The number of samples. So these are the same. Let me show you.


So number of samples and the x ids shape. Characterities are the same thing. We've already


defined that. So let's just go with that. And we're going to divide that by the batch size.


Okay. And this gives us the new number of samples within our data set because we've batched it here.


Okay. Now, one, we only want 90% of these. So we multiply by split.


So I do need to just run that quick. Okay. So that's our 90% mark. And when we're saying in a


moment that we want this number of samples from the data set, we can't give it a float because we


can't have 0.375 of a batch. It doesn't matter. So we're going to do that. And then we're going to


have 0.375 of a batch. It doesn't work. So what we need to do is set that equal to a integer to


round off. So we'll do that here. So move that and run that. So now we have our size.


We can split our data set. So we're going to have the trained data set, which is just going to be


data set. And as we did up here, where we do this take method to take a certain number of samples,


we do the same. But now we're going to take a lot more than just one. We're going to be taking


the size that we calculated here, which is 8,700 or so. And then for the validation data set,


we kind of want to do the opposite, right? We want to skip that many samples. So that's exactly


what we write here. We just write skip size. So we're going to skip the first 8,700 or so.


And we're just going to take the final ones out of that final 10%. And then we're not going to be


using data set anymore. And it's quite a large object. So we can just remove that from our


memory. Okay, so now we're on to actually building and training our model. So we're going to be using


transformers again to load in a pre-trained BERT model. And we're going to be using the


TF auto model. It's using the TensorFlow version. And we'll just set BERT equal to TF auto model


from pre-trained. And it is BERT base


and case. And just like we would because here we're using the TensorFlow version here,


if we got rid of this, we'd be using PyTorch. So here we're using TensorFlow, just like we would


have any other TensorFlow model. We can use the summary method to print out what we have.


Okay, and we see we just have this one layer, which is a BERT layer. And this is just because


everything within that BERT layer is a lot more complex than just one layer. But it's all embedded


within that single layer. So we can't actually see inside of it just by printing out the summary.


Now, we have BERT and that is great. But that's kind of like the core or the engine of our model


book. We need to build a frame around that based on our use case. So the first thing we need to do


is, okay, we have our two input layers. Okay, we have that one for input IDs and one for the


attention mask. So first, we need to define those. So we've already imported TensorFlow earlier for


the data set. So we don't need to do that again. And what we do is input IDs, and we say tf.keras


layers. And we're using an input layer here. And the shape of the input layer is equal to the


sequence length. So sequence length, and then we just add a comma here. Okay, so that's the same


shape as that we were seeing up here. Okay. Now, we need to set the name. And we do this because


as we have seen up here, we have this dictionary, right? And we need to know which import each of


these input tensors are going to go into. And that's how we do it. We map input IDs to this


name here, import IDs, and we'll do the same for attention mask as well in a moment. And we set


the data type equal to integer 32. And we do that because these are tokens here. So BERT expects


them to be integer values. And we do the same for mask tf.keras layers. Input. And then we


add the shape. And sequence length again. We have the name, which is where we use our attention


mask to map that across. And again, it's just the same D type, which is int 32. Okay, so they are


our input layers. And then what we need is we need, well, after the input, what do we have? We


want to feed this into BERT, right? So what we're doing there is we're creating our embeddings


from BERT. And what we need to do is access the transformer within that BERT object. So to do that,


for BERT, we just write BERT.BERT, which accesses that transformer. And in there, we want to pass out


input IDs and our attention mask, which is going to be the mask. So these two input layers here.


And we have two options here. We can pull out the raw activations or the raw activation tensors


from the transformer model here. And these are three dimensional. And as I said, just take out


that raw activation from the final layer of BERT. Or what they also include here is a pooled layer


or pooled activations. So these are those 3D tensors pooled into 2D tensors. Now what we're


going to be using is denselys here. So we are expecting 2D tensors and therefore we want to


be using this pooled layer. We could also pool it ourselves, but the pooling has already been done.


So why do it again? Now, what we want to do here for our final part of this model is we need to


convert these embeddings into our label predictions. Okay, so to do that, we want two more layers.


And these are going to both be densely connected layers. And for the first one, I'm going to use


1024 units or neurons. And the activation will be ReLU. And we're passing the embeddings into that.


And then our final layer, which is our labels, that is going to be the same thing again.


So dense. But this time, we just want a number of labels here. So we did calculate this before.


It was array, max, plus one, right? So that is just five. Okay, so we have five output labels.


And what we want to do here is calculate the probability across all five of those output


labels. So to do that, we want to use a softmax activation function. And we just want to say,


we're going to call this the outputs layer because it is our outputs. Now, that is our model


architecture. They are all of our layers, but we haven't actually put those layers into a model


yet. They're all kind of just floating there by themselves. Obviously, they do have these


connections between them, but they're not initialized into a model object yet. So we


need to do that. And to do that, we go tf.keras model. And we also need to do here, we need to set


the input layers, so our inputs. And we have two of those. So we put them in a list here,


input ids and the mask. So this is those two.


And then we also need to set the outputs. And we just have one output and that is a y. Yeah. So just


setting up our boundaries of our model. We have the inputs and they lead to our outputs.


Everything else within this is already handled. To go to x, we consume embeddings and embeddings


consume input ids and mask. So those connections are already set up. And let's just see what we


have here. And I just realized that this is input. This should be mask. And let's see what this error


is. Okay. So here I've got to add this connection. So I've got to add this connection.


So we need to add x there. Okay. So now what do we have? It's a little bit messy, but so we have our


input ids and we have the shape here, the attention mask. So these are our two input layers.


They lead into our vert layer. Then we have our pre-classifying layer here. This is our


densely connected URANET with 1024 units. And we have our outputs, which is the softmax.


And we have five of those. Now, if you would like to, what you can do if you don't have enough


compute to train the vert layer as well, you can also write this in the output layer.


So we go model layers and we select number two because we have zero, one and two. So vert is


number two in there. And we can set trainable equal to false. And this would just freeze the


parameters within this vert layer and just train the other two here. Okay. But I will be keeping


that so they can be trained as well. Although you don't need to. It will probably give you a small


performance increase, but not a huge amount in most cases. Now, I want to set up the model training


parameters. So we need a optimizer, which is optimizers. And for this, we're going to set


a pretty small learning rate of one E to the minus five. This is because we've got our vert model in here.


We also want to set a weight decay as well. So this is Adam with a decay. And what we also want to add


is a loss function. So we want to do tf.keras losses. And because we're using categorical outputs


here, we want to use categorical cross entropy. And then we're going to set our accuracy as well.


And that is tf.keras metrics this time. And we're using categorical accuracy for the same reason.


We're going to need to pass accuracy in there as well.


And let me change that to metrics. And then we want to do model compile optimizer equal to the


optimizer, loss to loss. And metrics is going to be equal to a list containing our accuracy.


So that's our model training parameters. So the final thing to do now is train our model.


So to do that, we call model fit like we would in any TensorFlow training. And we have our training


data set, which we're going to use to train our model. And then we're going to use our model


training. And we have our training data set, which we built already. The validation data we'll be


using is our validation data set. And we'll train that for three epochs. And that will take some


time. And immediately after we do train that, I'm also going to save that model to sentiment model.


Okay, that will just create a directory here and store all the files that we need for that model


in there. So I will go ahead and run that. And I will see you when it's done. Okay, so


we've finished training our model, we go up to an accuracy of 75%, still moving, and also


validation accuracy of 71% here. So inside the sentiment model directory that we just created


when we saved our model, we have everything that we need to load our model as well. So I'm just


going to show you how we can do that. So what we can do is let's start a new file here, no notebook.


Okay, so let's just close this. And what we'll do here is we need to first import TensorFlow.


And after we have imported TensorFlow, we need to actually load our model. So we use tf.keras.models


load model. And then we're loading this from the sentiment model directory here. And then


let's just check that what we have here is what we built before.


Okay, great. So you can see now, we have our input layers, but and then we have our pre classifier


and classifier layers as well. So that's exactly what we did before. And now we can go forwards and


start making predictions with this model. So before we make our predictions, we still need to convert


our data or input strings into tokens. To do that, I'm going to create a function to handle it for us.


First, we are going to need to import the tokenizer from transformers.


We're using vert tokenizer just like we did before. And we're going to be using tokenizer,


vert tokenizer from pre trained, vert base case. Okay, so exactly the same as what we used before.


So that is our tokenizer. And all we need to do is we're going to say, call it prep data. And here


we would expect a text which is a string. And we'll return our tokens. This is just the same as


what we were doing before. Okay, so we do encode plus, our text, we set a max length, which is going


to be 512, as always, but we are going to drink it anything longer than that.


And we're going to pad anything shorter than that. And we'll pad it up to the max length.


And then we want to add the special tokens, that is true. And then there's one other thing


that we don't need, which are the token type IDs.


So token type IDs, and this is just another tensor that is returned and we don't need them.


So we can just ignore them. And we're going to return the TensorFlow tensors.


Okay. And that is all we need to do.


And now we can just return our tensors in the correct format. So the format that we need is,


like we used before, with the input IDs. But if you remember before, within the data set,


we were working with TensorFlow float 64 tensors. So we're going to use that.


Within the data set, we were working with TensorFlow float 64 tensors. So we also need


to convert these, which will be integers, I believe, into that format. So to do that,


we do tf.cast tokens, input IDs. And we say we want to cast that to tf.float 64.


And we can copy that across and we'll repeat the same thing, but for our attention mask.


So attention mask, we just copy that across. And that is all we need to prepare our data.


And just fit set. Okay. And now what we can do is we can use prep data.


We'll just put something like hello world. Okay. And we get all these values.


And you can see here, I have entered that wrong. We don't even need this necessarily,


but just to be explicit. And we just need to add the s onto ids. So rerun that and remove that


error. And you can see here we have our CLS token, hello world, separated token,


followed by lots of padding. Okay. So our prep data works. And let's just


put that into a value there. And what I want to do now is get the probability


by calling model predict. And we do test here. So we've already done prep data.


And let's see what we get. Okay. So we have these predictions, which is not that easy to read.


And we also need to access the zero index. So just get a simple array there.


And what we can do to get the actual class from that is we'll just import numpy as np,


because we just want to take the maximum value out of all of these. And to do that,


we just do np argomax, probes, and zero. Okay. So it's had a neutral sentiment.


But something like this movie is awful. And we should get zero. Okay.


And we'll do this and we'll get four. Okay. So it's working. So that I know is pretty long,


but that is really everything you need from start to finish. We've preprocessed the data,


set up our input data set pipeline. We've built, trained our model, saved it, loaded it,


and made predictions. So that's really everything you need. So I hope that has been a useful video.


HNSW for Vector Search Explained and Implemented with Faiss (Python)

Hi, welcome to the video.


We're going to be having a look at the Hierarchical Navigable Small Worlds graph, or HNSW graph,


and how it's used in vector search.


Now, HNSW is one of the best performing algorithms for Approximate Nearest Neighbours Search.


And what I want to do with this video is explore how it works,


and how we can implement it in FICE.


So, let's go ahead and get started.


Now, we'll start with the sort of foundations of HNSW and where it comes from.


So, we can split Approximate Nearest Neighbours algorithms into three broad categories.


Trees, hashes, and graphs.


Now, HNSW, or HNSW graph, so we can figure out, probably belongs in the graph category.


And more specifically, it's a type of proximity graph, which simply means that vertices,


which are just the vectors in our graph, or the nodes in our graph,


are connected to other nodes or vertices based on their proximity.


And we would typically measure this proximity using Euclidean distance.


I don't have to, but that's the sort of standard.


Now, going from a proximity graph, which is pretty simple, to a HNSW graph is a pretty big leap.


And there's two main algorithms that really helped me understand what HNSW actually does.


And that is the Probability Skip List and the Navigable Small World Graph,


which are just the predecessors of HNSW.


So, what I'm going to do is just quickly run through those two algorithms.


So, this is a Probability Skip List.


It's reasonably old. It was introduced in 1990 by a guy called William Pugh.


And it works by building several layers of linked lists, which is what you can see up here.


So, what we do with this is we're looking for the number 11, or the key 11.


And we saw over here the top layer of our start block.


And what we do is we say in our layer 3, which is our entry layer, and we move across.


So, we go over to the next block here.


And we say, okay, number 11 is not 5, and 5 is not greater than 11.


So, we continue and we go all the way over to the end block over here.


Once we reach the end block, we know that we need to go to the next layer.


So, one layer down.


And so, we do that. We go to layer 2 at the start block again.


And we continue moving across.


Now, at this point, we get to block 19 over here.


And we say, okay, 19 is greater than 11.


So, now we know that we need to go on to our next layer.


Layer 1 down here.


And then we go across and we find number 11.


Now, the whole point of this is when we have a lot of values,


this can reduce the number of steps we need to take in order to get to our final value.


Rather than just going along our layer 0 and going along every single value within our list.


And HNSW borrows this hierarchical layered format or levels, as we will see pretty soon.


Navigable Small World or NSW graphs were developed over the course of a few papers around 2011 up to 2014.


And the basic idea is that we can have a very large graph with many vertices.


And because we have both short-range links,


so the sort of links that you see on the outside of our circle here,


and we also have these long-range links, which is what you see going across the circle,


we make our graph very navigable.


So, it's very easy to get from one point to another in very few steps.


And there's some terminology that we should cover here,


which is all the vertices we have here, all of their connections,


so all the other vertices that they all connect to,


we refer to these as that vertex's friends list.


And when searching this NSW graph, we begin at a predetermined entry point.


So, in here we see we're sort of starting at a few different points,


but in reality with these big graphs, we always start with one specific entry point,


or maybe sometimes we can use different entry points based on some sort of logic.


But we have either one or only a few entry points in our graph.


And then what we do is we perform what is called greedy routing.


Now, routing just refers to the route that we take through our graph,


and greedy refers to the fact that we are always going to choose


out of our vertices friend list, we're going to choose the friend


that is the closest to our target vertex.


Now, if we find that there are no near vertices in our friend list,


this is a local minimum, and this is our stopping condition.


We cannot get any closer. We're kind of stuck in that local minimum.


Now, if our vertex has very few friends, we can get stuck quite easily.


So, we can get stuck in the local minimum if there are less connections.


And we actually refer to vertices that have fewer friends as a low degree vertex.


So, it's low degree, it means it has a low number of links or edges.


And then we also have high degree vertices.


Now, with these, if we are currently on a high degree vertex,


it's harder to get stuck in the local minimum because we have more options


on where we can move.


Now, the routing through a navigable small world graph is split into two phases.


So, we have typically the earlier zoom out phase,


where we are passing through vertices that have a low number of links.


And then we also have the zoom in phase, where we are passing through


high degree vertices, which have a lot of links.


And what we tend to find is that high degree vertices will usually have


more long range links, so we can move further across our graph


with these high degree vertices.


Now, to minimize the probability of stopping early,


so avoiding those local minima, we can increase the average degree of our vertices.


Now, this also increases the complexity of our network and therefore slows down our search time.


So, we have to find a balance between both of those.


Now, another approach to avoiding hitting a local minima too early


is to sort search on high degree vertices, so the vertices which have a lot of connections.


Now, for NSW, we can do this and it works for low dimensional data,


but it doesn't work perfectly.


And it's actually one of the key factors that HNSW improves upon


and uses to improve its own performance.


So, let's move on to HNSW and how both of those fit into HNSW.


So, HNSW is obviously an evolution of NSW and it borrows inspiration


from Poob's probability skip list structure with the hierarchy.


So, you can see that here we have a few different layers.


So, we have our entry layer just like we did with the probability skip list structure.


And we also have an entry point.


And what we've basically done here is taken a navigable small NSW graph


and split it across multiple layers.


So, high degree vertices will tend to be spread across more layers


because when we're building our graph, we add the number of friends


based on which layer it gets inserted at.


The higher the layer that the vertex is inserted at, the more friends it's going to have,


which makes it a high degree vertex.


And what that does is if we start at the highest layer,


that means we're on high degree vertices.


So, we are less likely to get stuck in the local minima.


And that means we're not going to stop early.


So, that allows us to start at the top layer and we move through our graph.


And what we do is we have our query vector and on each layer,


we keep moving or traversing across different edges in that layer


in essentially the same way we did for NSW.


So, we greedily identify the friend of our current vertex


that has the least distance from our query vector.


And we move to that and then we keep doing that.


And once we hit local minimum, we don't stop the whole thing.


We just move down to the next layer.


And then we do it again.


And we keep doing that until we reach the local minimum of layer zero.


And that is our stopping condition. That's where we stop.


Now, that's how we search. That's what our graph looks like and how we search.


But let's have a quick look at how we actually build that graph.


So, what we do is we have a probability function,


which essentially says we're pretty much going to put every vertex


or every vector on layer zero.


Not every, but a pretty high number of them as we'll see later.


And we say, okay, we get a random number from uniform distribution.


And if that number is greater than our probability for that level,


we move on to the next layer. So, we increment our layer.


So, let's say this turns out to not be true.


We move up a layer and we try again.


Now, this next layer is going to have low probability,


but we're also going to subtract the probability value from our lower layer


from the random number that we've produced.


And we keep doing this again and again until we get up to our top layer.


At that point, there are no other layers. This is very unlikely to happen.


Say if we have a billion, maybe not a billion, like a hundred,


I think it was a hundred million vectors that I tried this with.


I got, I think, four vectors on the top layer.


So, it is very unlikely for one vector to go all the way up to the top.


But if it does, we insert it at that layer,


and it has the possibility of becoming a entry point for our graph.


One other thing that we also have here is the ML at the top.


Now, this is called the level multiplier.


And this value is just a normalization value,


which allows us to modify the probability


or the probability distribution across each one of our layers.


Now, the creators of HSW, they found that the best value for ML,


the optimum value, was to use one over the natural logarithm of M.


And M, we'll have a look at that pretty soon.


That's just the number of neighbors each vertex is assigned.


So, here we have M. So, we set M to three.


And we're saying here, so we want to insert this blue vector.


So, that blue vector you see, we've decided, okay,


we're going to insert it at layer one.


That means it gets inserted at both layer one and layer zero,


but not at layer two.


And we go through our graph just like we do when we search.


But this time, we're trying to find many neighbors of our new vertex


that we can assign to its friend list.


And what we have here is, so with an M value of three,


we would add three neighbors to that vertex,


both on layer one and also on layer zero.


Now, we can also have a M max value and an M max zero value,


which is what we have down here.


And basically, as more vertices are added,


we may find that this vertex ends up with more than three friends.


And maybe that's fine, but what we essentially do is use M max


to say, okay, if we have any vertices with more than this number of friends,


we need to trim it and we need to just keep the closest,


like three in this example.


And then M max zero is another value, and that is the same,


so it's the maximum number of friends of vertex you have,


but for layer zero, which is, I think, always a higher number


than the other layers.


Now, one other thing that we should cover very quickly


is we also have two more variables, ef-construction and ef-search.


Now, ef-construction is used when we're building our graph,


and it's basically the number of friends of our inserted vertex


that we will use when traversing across layers.


So in layer one up here, we have these three friends,


and that means that during our search for friends in the lower layers,


we would go down like this, and we would start our search


from each of these three vertices in the lower layer.


Okay, now that would be the case for all of those.


And ef-search is the same, but it's used during search time


to search through our graph.


And I mean, that's pretty much all we need to know


on the theory side of HNSW, so let's dive into the implementation


of HNSW in FICE.


Okay, so we're in Python now, and all I've done is taken our data,


so I'm using the sift1m data set as we usually do,


and all I'm going to do is use this to search through our index, as always.


Now, the first thing we want to do is actually create


or initialize our index, so I'm going to use two parameters.


So we have d, which is the dimensionality of the sift1m data set,


so the length of our vectors in there, and also m.


Now, m, if you remember, that's the number of neighbors


that we will assign to each vertex in our graph.


Now, for this, I'm going to use a value of 32.


And what we can now do is rewrite index equals FICE,


index flat HNSW, and then we just pass d and m.


And that's, you know, we've initialized...


Okay, sorry, so it's not flat HNSW, it's HNSW flat.


Okay, there we go.


So with that, we have initialized our index.


Of course, there's nothing in there at the moment, and we can see that,


so we can go ahead and write index HNSW max level


to see, okay, how many levels do we have in our index.


And we'll see that we get this minus one, which means that the levels in there,


the structure has not been initialized yet.


And we also see that if we do this.


So this is how we get the number of vertices that we have within each layer.


So we write the FICE vector to array.


And in here, we want to write index HNSW levels.


And then let's print out.


And we see that we just get this empty array.


Now, what we'd also want to do when we do...


So this is just a massive array.