So each one of our samples here will need to be processed by this Flare.data.sentence


How-to do Sentiment Analysis with Flair in Python

method before we pass it into the actual model.


Once we do have this, so let's call this sample as well.


We will pass it to our model for prediction.


Which is really easy. All we need to do is call the predict method.


On the sample.


And now this doesn't output anything.


Instead, it actually just modifies the sentence object that we have produced.


So it modifies sample.


And we can see now that our sample, we solved the sentence and we solved the number of tokens.


But we also have these additional labels which are the predictions.


We have the label which is positive, which means it's a happy or it's a positive sentiment.


And then what we have here is actually the probability or the confidence in that prediction.


That's great, but realistically we want to be extracting these labels.


So we're actually able to extract these by accessing the labels method.


So you have labels here and this produces the positive and the confidence.


To access each one of these we access the positive and the confidence.


We access index zero followed by dot value.


Okay, so this will give us the positive.


And then we can also do the same to get the confidence called the score.


Like that.


So what we can do now is just create a simple for loop that will go through each sample in our test


data and assign a probability for each one.


So we will initially create a sentiment and confidence list.


And then we will just as we are looping through the data we will append our sentiment values.


So the positive or negative and the confidence to each one of these lists.


So here we are first tokenizing our sentence.


Then we are making a prediction using that tokenized sentence which we are calling sample.


And as we did before we have now got this labeled sentence and we just need to extract the two labels that we have here.


Okay, so we can see here that one of our sentences was just blank so we will add in some logic to avoid any errors there.


Okay, so looking at this it's also whenever there's a space as well.


So we just need to trim this which we can do easily using the strip method.


Okay, so it took a little bit of time but we now have our predictions.


So what we want to do is actually add what we have here in the sentiment and confidence list to our data frame.


So to do that we just add df sentiment to create a new sentiment column.


And we made that equal to the sentiment list that we have created.


And then we also do the same for confidence as well.


Then we can see our data frame.


Okay, so initially looking at this it looks pretty good.


So intermittently pleasing but mostly routine effort.


Incredibly negative but basically saying it's occasionally okay but generally nothing special.


So obviously it's a negative sentiment which is matched up to negative sentiment here.


Here we're saying okay Kidman's the only thing that's worth watching in birthday girl.


And it serves as another example of the sad decline of British comedies in the post full monty world.


Fair enough, also negative.


So this one is our first positive.


Once you get into it it's relevant.


The movie becomes a heady experience.


Yeah, I mean sounds pretty positive to me.


So it's quite good even here where we're not saying anything particularly like a negative or positive word.


We're just saying that the movie is or the movie delivers on the performance of striking skill and depth.


Which must be pretty hard for a machine to understand and actually get right.


But looking at all these it's doing really well.


And I think it's really cool that we can actually do this with so little effort.


And we've only actually written a few lines of code in reality.


And it's producing really good accurate results which is really impressive to me.


So that's it for this video.


How to Build Custom Q&A Transformer Models in Python

Hi and welcome to the video. Today we're going to go through how we can fine-tune a Q&A Transformer model.


So for those of you that don't know, Q&A just means question answering and it's one of the biggest topics in NLP at the moment.


There's a lot of models out there where you ask a question and it will give you an answer.


And one of the biggest things that you need to know how to do when you are working with transformers,


whether that's Q&A or any of the other transformer based solutions, is how to actually fine-tune those.


So that's what we're going to be doing in this video. We're going to go through how we can fine-tune a Q&A Transformer model in Python.


So I think it's really interesting and I think you will enjoy it a lot. So let's just go ahead and we can get started.


Okay, so first thing we need to do is actually download our data.


So we're going to be using the SQuAD dataset, which is the Stanford question answering dataset,


which is essentially one of the better known Q&A datasets out there that we can use to fine-tune our model.


So let's first create a folder.


It's going to use OS and OSMaker.


We'll just call it SQuAD. Obviously, call this and organize it as you want. This is what I will be doing.


Now, the URL that we are going to be downloading this from is this.


Okay, and there are actually two files here that we're going to be downloading, but both will be coming from the same URL.


So because we're making a request through URL, we're going to import requests.


We can also use the Wget library as well, or if you're on Linux, you can just use Wget directly in the terminal.


It's up to you what we're going to be using requests.


Okay, and to request our data, we're going to be doing this.


So it's just a get request, use a FString, and we have the URL that we've already defined.


And then the training data that we'll be using is this file here.


Okay, requests. Okay, we can see that we've successfully put that data in there.


Okay, so like I said before, there's actually two of these files that we want to extract.


So what I'm going to do is just put this into a for loop, which will go through both of them.


Just copy and paste this across.


Rename this file.


And the other file is the same, but instead of train, we have dev.


Okay, so here we're making our request.


And then next thing we want to do after making our request is actually saving this file to our drive.


Okay, and we want to put that inside this squad folder here.


So to do that, we use open.


And again, we're going to use a string here.


I want to put inside the squad folder here.


And then here we are just going to put our file name, which is file.


Now we're writing this in binary because it's JSON.


So we put wb for our flags here.


And then within this namespace, we are going to run through the file and download it in chunks.


So we do for chunk.


And then we iterate through the response.


Like this.


We'll just use a chunk size of four.


And then we just want to write to the file like that.


So that will download both files.


Just add the colon there.


We should be able to see them here now.


So in here we have data.


We have essentially a lot of different topics.


So the first one is Beyonce.


And then in here, we will see, if we just come to here, we get a context.


But alongside this context, we also have QAS, which is question and answers.


And each one of these contains a question and answer pair.


So we have this question, when did Beyonce start becoming popular?


So this answer is actually within this context.


And what we want our model to do is extract the answer from that context by telling us


the start and end token of the answer within that context.


So we go zero and it is in the late 1990s.


And we have answer start 269.


So that means that a character 269, we get I.


So if we go through here, we can find it here.


OK, so this is the extract.


And that's what we will be aiming for our model to actually extract.


But there will be a start point and also the end point as well, which is not included in here.


But we will add that manually quite soon.


So that's our data, and then we'll also be testing on the dev data as well,


which is exactly the same.


OK, so we move on to the data prep.


So now we have our files here.


We're going to want to read them in.


So we're going to use the JSON library for that.


And like we saw before, there's quite a complex structure in these JSONs.


There's a lot of different layers.


So we need to use a few for loops to fill through each of these and extract what we want,


which is the context, questions and answers all corresponding to each other.


So in the end, we're going to have a list of strings, which is going to be all of these.


And in the case of the answers, we also have the starting position.


So it will be a list of dictionaries where one value is a text and one value is the starting position.


So to do that, we're going to define a function called Rebsquad.


We'll define our path here as well.


And the first thing we need to do is actually open the JSON file.


So we do with open path.


And again, we are using a binary file.


So we're going to have B as a flag.


But instead of writing, we are reading.


So use R here.


So RB.


I'm just going to do JSON load F here.


So now we have our dictionary within this squad dict here.


So maybe whilst we're just building this function up, it's probably more useful to do it here.


So we can see what we're actually doing.


So let's copy that across.


And then we'll fill this out afterwards.


Of course, we do actually need to include the path.


So let's take this.


And now we can see what's inside here.


Maybe we can load just a few rather than all of them.


Or we can investigate it like this.


Okay, so we have the version and data, which we can actually see over here.


Version and data.


So we want to access the data.


And within data, we have a list of all these different items, which is what I was trying


to do before.


So we go into data and just take a few of those.


Okay, and then we get our different sections.


First one, let's just take zero, which is Beyonce.


And then we have all of these.


So we're going to want to loop through each one of these because we have


this one, the next, and we're going to keep needing to just run through all of these.


So let's do that.


We want to do for group in squad dict.


And remember, we need to include the data here.


Let's just see how, say group title.


So group title, so we can see a few of those.


Okay, we're going to go through each one of those.


So the second part of that are these paragraphs.


And within the paragraphs, we have each one of our questions.


So let's first go with paragraphs and we'll do a chop in here.


Sorry, it's a list.


There we go.


And the first thing we need to extract is the easiest one, which is our context.


However, that is also within a list.


So now if we access the context, we get this.


So we're essentially going to need to jump through or loop through each one of these here.


Now we're going to access the paragraphs and loop through each one of those.


And then here, we're going to access the context.


So let's write that.


So we already have one group here.


So let's just stick with that.


And we're going to run through the passage in the paragraphs.


So already here, we're going through the for loop on this index.


And now we're going to go through the for loop.


And now we're going to go through a loop on this index.


Let's keep that.


So that means that we will be able to print the passage context.


And there we go.


So here we have all of our context.


So that's one of our three items that we need to extract.


Okay, so that's great.


Let's put that all together.


So we're going to take this, put it here.


And then we have our context.


Okay, that's great.


But obviously for each context, we have a few different questions and answers.


So we need to get those as well.


Now, that requires us to go through another for loop.


So let's go this passage.


We need to go into the QAS key and loop through this list of question and answers.


So we have this, and then we have our list.


So another layer in our for loop will be for question answer in passage QAS.


And then let's take a look at what we have there.


Okay, great.


So we have plausible answers, question and answers.


So what we want in here is the question and answers.


So question is our first one.


Perfect.


So we have the questions now.


And then after we have extracted the question,


we can move on to our answers.


As we see here, the answers comes as another list.


Now each one of these lists all just have one actual answer in there,


which is completely fine.


So we can access that in two ways.


We can either loop through or we can access the zero value of that array.


Either way, it doesn't matter.


So all we need to do here


is loop through those answers or if we want just go with QA answers 0.


So in most cases, this should be completely fine.


As we can see here, most of these question and then they have the answers dictionary.


The question and then they have the answers dictionary, which is fine.


However, some of these are slightly different.


So if we scroll right down to the end here,


see, okay, we have this which is talking about physics.


And then rather than having our answers array, we have these plausible answers,


which is obviously slightly different.


And this is the case for a couple of those.


So from what I've seen to state, it's like the best way to deal with this


is simply to have a check.


If there is a plausible answers key within the dictionary,


we will include that as a check.


Within the dictionary, we will include that as the answer


rather than the actual answers dictionary.


So to do that, all we need to do is check if QA keys contains plausible answers.


If it does, we use that.


Otherwise, we use answers.


Okay.


Then we use this one.


Otherwise,


we will use answers.


So let's just add all of that into our for loop here.


So we have our context, and then we want to loop through the question answers.


And this is where we get our question.


Then once we're here, we need to do something slightly different,


which is this plausible answers.


And then we use this access variable in order to define what we're going to loop through next.


So here we go for answers.


Answer, sorry, in QA access, because this will switch to implausible answers or answers.


And then within this for loop, this is where we can begin adding this context, question, and answer


to a list of questions, contexts, and answers that we still need to define up here.


So each one of these is just going to be an empty list.


And then all we do is copy this across,


and we just append everything that we've extracted in this loop.


And the context.


And then we just add the context.


And the context, question, and answer.


And that should work.


So now let's take a look at a few of our contexts.


Okay, and we can see we have this and because we have multiple question and answers for each context,


the context just repeat over and over again.


But then we should see something slightly different when we go with answers.


And questions.


Okay, so that's great.


We have our data in a reasonable format now.


But we want to do this for both the training set and the validation set.


So what we're going to do is just going to put this into a function like we were going to do before.


Just read squad.


So here we're going to read in our data, and then we run through it and transform it into our three lists.


All we need to do now is actually return those three lists.


And answers.


So now what we're going to do is just go back to our training set.


And we're going to do this for both the training set and the validation set.


And answers. So now what we can do is execute this function for both our training and validation sets.


So we're going to train context, questions and answers.


Okay, so that is one of them.


And we can just copy that.


And we just want this to be our validation set.


Like so.


We now have the training context and the validation context, which we can see right here.


So here let's hope that there is a slight difference in what we see between both.


Okay, great.


That's what we would expect.


Okay, so now we have our data almost in the right format.


We just need to add the ending position.


So we already have the start position if we take a look in our train answers.


Okay, we have the answer start, but we also need the answer end.


And that's not included within the data.


So what we need to do here is actually define a function that will go through each one of our


answers and context and figure out where that ending character actually is.


And of course, we could just say, okay, it's the length of the text.


We add that onto the answer start and we have our answer end.


However, that unfortunately won't work because some of the


answer starts are actually incorrect and they're usually off by one or two characters.


So we actually need to go through and one, fix that and two, add our end indices.


So to do that, we're just going to define a new function.


It's going to be add end index.


And here we will have our answers and the context.


And then we're going to just feed these in.


So first thing we do is loop through each answer and context pair.


And then we extract something which is called the gold text, which is essentially the


answer that we are looking for.


It's called the golden text or gold text.


So simply our answer and within that the text.


So we are pulling this out here.


So we should already know the starting index.


So what we do here is simply pull that out as well.


And then the end index ideally will be the start plus the length of the gold text.


However, that's not always the case because like I said before, they can be off by one or two characters.


So we need to add in some logic just to deal with that.


So in our first case, let's assume that the characters are not off.


So if context start to end.


Start to end equals the gold text.


This means everything is good and we don't need to worry about it.


So we can modify the original dictionary and we can add answer end into there.


And we made that equal to our end index.


However, if that's not the case, that means we have a problem.


It's one of those dodgy question answer pairs.


And so this time what we can do is we'll add a else statement.


So we're just going to go through when the position is off by one or two characters because


it is not off by any more than that in the squad data set.


Loop through each of those and we'll say, OK, if the context.


And then in here, we need to add the start index and this again.


So let's just copy and paste that across.


Be easier.


But this time we're checking to see if it is off by one or two characters.


So just minus N.


And it's always minus and it isn't shifted.


It's always shifted to the left rather than shifted to the right.


So that's this is fine.


So in this case, the answer is off by end tokens.


And so we need to update our answer start value and also add our answer end value.


So start index minus N and we also have the end.


So that's great.


We can take that and we can apply it to our train and validation sets.


So all we do here is call the function.


And we just see train answers and train context.


And of course, we can just copy this and do the same for our validation set.


OK, perfect.


So now if we have a quick look, we should be able to see that we have


a few of these ending points as well.


OK, so I think that looks pretty good.


And that means we can move on to actually encoding our text.


To tokenize or encode our text, this is where we bring in a BERT tokenizer.


So we need to import the Transformers library for this.


And from Transformers, we are going to import the Distilbert.


So Distilbert is a smaller version of BERT, which is just going to run a bit quicker,


but it will take a very long time.


And we're going to import the FAST version of this tokenizer because this allows us to more


easily adjust our character and then start locations to token and start locations later on.


So first, we need to actually initialize our tokenizer, which is super easy.


All we're doing is loading it from a pre-trained model.


And then all we do to create our encodings is to load the


tokenizer. So we'll do the training set first.


Let's call it tokenizer.


And in here, we include our training context.


And the training questions.


So what this will do


is actually merge these two strings together.


So what we will have is our context and then there will be a separator token


followed by the question.


And this will be fed into Distilbert during training.


I just want to add padding there as well.


And then we'll copy this and do the same for our relation set.


And this will convert our data into encoding objects.


So what it can do here is


Okay, and this will convert our data into encoding objects.


So what we can do here is print out different parts that we have within our encodings.


So in here, so you have the input IDs.


So let's access that.


And you'll find in here we have a big list of all of our samples.


So check that we have 130k.


And let's open one of those.


Okay, and we have these token IDs.


And this is what Bert will be reading.


Now if we want to have a look at what this actually is in sort of human readable language,


we can use the tokenizer to just decode it for us.


Okay and this is what we're feeding in.


So we have a couple of these special tokens.


This just means it's the sort of sequence.


And in here we have a process form of our original context.


Now you find that the context actually ends here.


And like I said before, we have this separator token.


And then after that we have our actual question.


And this is what is being fed into Bert, but obviously the token ID version.


So it's just good to be aware of what is actually being fed in and what we're actually using


here.


But this is a format that Bert is expecting.


And then after that we have another separator token followed by all of our padding tokens


because Bert is going to be expecting 512 tokens to be fed in for every one sample.


So we just need to fill that space essentially.


So that's all that is doing.


So let's remove those and we can continue.


So the next thing we need to add to our encodings is the start and end positions because at


the moment we just don't have them in there.


So to do that we need to add an additional bit of logic.


We use this character to token method.


So if we just take out one of these.


Let's take the first one.


We have this.


And what we can do is actually modify this to use the character token method.


Remove the input IDs because we just need to pass it the index of whichever encoding


we are wanting to modify or get the start and end position of.


And in here all we're doing is converting from the character that we have found a position


for to the token that we want to find a position for.


And what we need to add is train answers.


We have our position again because the answers and encodings the context and question needs


to match up to the answer of course that we're asking about.


And we do answers start.


So here we're just feeding in the position of the character and this is answer.


So feeding in the position of the character and we're expecting to return the position


of the token which is position 64.


So all we need to do now is do this for both of those so for the start position and end


position.


See here we should get a different value.


OK but this is one of the limitations of this.


Sometimes this is going to return nothing as you can see it's not returning anything


here.


And that is because sometimes it is actually returning the space and when it looks at the


space and the tokenizer see that and they say OK that's nothing we're not concerned


about spaces and it returns this non value that you can see here.


So this is something that we need to consider and build in some added logic for.


So to do that again we're going to use a function to contain all this.


And call it add token positions.


Here we'll have our encodings and our answers.


And then we just modify this code so we have the encodings.


We have the answers.


And because we're collecting all of the token positions we also need to initialize a list


to containers.


So we do start positions empty list and end positions.


And then we just want to loop through every single answer and encoding that we have.


Like so.


And here we have our start position so we need to append that to our start positions


list.


And we just do the same for our end positions which is here.


Now here we can deal with this problem that we had.


So if we find that the end positions the most recent one so the negative one index is none.


That means it wasn't found.


That means there is a space.


So what we do is we change it to instead use the minus one version.


And all this needs to do is update the end positions here.


Okay that's great.


But in some cases this also happens with the start position.


But that is for a different reason.


The reason it will occasionally happen with start position is when the passage of data


that we're adding in here so you saw before we had the context and separated token and


then the question.


Sometimes the context passage is truncated in order to fit in the question.


So some of it will be cut off.


And in that case we do have a bit of a problem but we still need to just allow our code to


run without any problems.


So what we do is we just modify the start positions again just like we did with the


end positions.


Obviously only if it's a none.


And we just set it to be equal to the maximum length that has been defined by the tokenizer.


It's as simple as that.


Now the only final thing we need to do which is because we're using the encodings is actually


update those encodings to include this data because as of yet we haven't added that back


in.


So to do that we can use this quite handy update method.


We'll just add in our data as a dictionary.


So you have the start positions, start positions, then we also have our end positions.


And then again we just need to apply this to our training and validation sets.


And let's just modify that.


Let's add the training encodings here and the train answers.


We do that again the validation set.


So now let's take a look at our encodings.


And here we can see great now we have those start positions and end positions.


We can even have a quick look at what they look like.


What we've done is actually not included the index here so we're just taking it for the


very first item every single time.


So let's just update that.


So obviously that won't get us very far.


And just update that as well.


And now this should look a little bit better.


So it's lucky we checked.


Okay so our data at the moment is in the right format.


We just need to use it to create a PyTorch dataset object.


So to do that obviously we need to import PyTorch.


And we define that dataset using a class.


And just pass in the torch.


Utils data dataset.


We need to initialize that.


Like so.


And this is coming from the Houdin Face Transformers documentation.


Don't take credit for this.


And we essentially need to do this so that we can load in our data using the PyTorch


data loader later on.


Which makes things incredibly easy.


And then we just have one more function here.


One method.


Okay and return.


And also this as well.


That should be okay.


So we apply this to our datasets to create dataset objects.


Our encodings and then the same again for the validation set.


Okay so that is our data.


Almost fully prepared.


All we do now is load it into a data loader object.


But this is everything on the data side done.


Which is great because I know this bit does take some time and I know it's not the most


interesting part of it.


But it's just something that we need to do and need to understand what we're doing as


well.


So now we get to the more interesting bit.


So we'll just add the imports in here.


So we need our data loader.


And then we're going to import the atom optimizer with weighted decay.


Which is pretty commonly used for transformer models when you are fine tuning.


Because transformer models are generally very large models and they can overfit very easily.


So this atom optimizer with weighted decay essentially just reduces the chances of that


happening which is supposed to be very useful and quite important.


So obviously we're going to use that.


And then final bit is TQDM.


So TQDM is a progress bar that we are going to be using so that we can actually see the


progress of our training.


Otherwise we're just going to be sat there for probably quite a long time not knowing


what is actually happening.


And trust me it won't take long before you start questioning whether anything is happening


because it takes a long time to train these models.


So they are our imports and I'm being stupid again here.


So it's from, did that twice.


Okay so that's all good.


So now we just need to do a few little bits for the setup.


So we need to tell PyTorch whether we're using CPU or GPU.


In my case it will be a GPU.


If you're using CPU this is going to take you a very long time to train.


And it's still going to take you a long time on GPU.


So just be aware of that.


But what we're going to do here is say device cuda, if cuda is available.


Otherwise we are going to use the CPU.


And good luck if that is what you're doing.


So once we've defined the device we want to move our model over to it.


So we just model.to device.


So this.to method is essentially a way of transferring data between different hardware


components at your CPU or GPU.


It's quite useful.


And then we want to activate our model for training.


So there's two things we have here.


So we have.train and eval.


So when we're in train mode there's a lot of different layers and different parts of


your model that will behave differently depending on whether you are using the model for training


or you're using it for inference which is predictions.


So we just need to make sure our model is in the right mode for whatever we're doing.


And later on we'll switch it to eval to make some predictions.


So that's almost everything.


So we just need to initialize the optimizer.


And here we're using the weighted decay atom optimizer.


We need to pass in our model parameters and also give it a learning rate.


And we're going to use this value here.


All these are the recommended parameters for what we are doing here.


So the one thing that I have somehow missed is defining the actual initializing the model.


So let's just add that in.


And all we're doing here is loading again a pre-trained one.


So like we did before when we were loading the transformers tokenizer.


This time it's for question answering.


So this the Silbert question answering is a Silbert model with a question and answering


head added onto the end of it.


So essentially with transformers you have all these different heads that you add on


and they will do different things depending on what head it has on there.


So let's initialize that from pre-trained.


And we're using the same one we use up here which is the Silbert base uncased.


And sometimes you will need to download that.


I don't need to as I've already done that but this can also take a little bit of time.


Not too long though and you get a nice progress bar hopefully as well.


Okay so now that it's all set up we can initialize our data loader.


So all we're doing here is using the PyTorch data loader object and we just pass in our


training dataset, the batch size.


So how many we want to train on at once in parallel before updating the model weights


which will be 16.


And we also would like to shuffle the data because we don't want to train the model on


a single batch and it just learned about Beyonce and then the next one it's learning about


Chopin and it will keep switching between those but never within a single batch having


a good mix of different things to learn about.


So it is data set, seems a bit of a weird name to me so I'm just going to change it.


And I also can't spell.


There we go.


And that is everything we can actually begin our training loop.


So we're going to go for three parts.


And what we want to start with here is a loop object.


So we do this mainly because we're using TQDM as a progress bar otherwise we wouldn't need


to do this.


There would be no point in doing it.


And all this is doing is kind of like pre initializing our loop that we are going to


go through.


So we're going to obviously loop through every batch within the train loader.


So we just add that in here.


And then there's this other parameter which I don't know if we...


So let's leave it.


But essentially you can add leave equals true in order to leave your progress bar in the


same place with every epoch.


Whereas at the moment with every epoch what it will do is create a new progress bar.


We are going to create a new progress bar.


But if you don't want to do that and you want it to just stay in the same place you add


leave equals true into this function here.


So after that we need to go through each batch within our loop.


The first thing that we need to do is set all of our calculated gradients to zero.


So with every iteration that we go through here every batch at the end of it we are going


to calculate gradients which tells the model in which direction to change the weights within


the model.


And obviously when we go into the next iteration we don't want those gradients to still be


there.


So all we're doing here is reinitializing those gradients at the start of every loop.


So we have a fresh set of gradients to work with every time.


And here we just want to pull in our data.


So this is everything that is relevant that we're going to be feeding into the training


process.


So everything within our batch.


And then in here we have all of our different items.


So we can actually see go here.


We want to add in all of these.


And we also want to move them across to the GPU in my case or whatever device you are


working on.


And we do that for the attention mass start positions and end positions.


So these start and end positions are essentially the labels that are targets that we want our


model to optimize for.


And the input IDs and attention mass are the inputs.


So now we have those defined.


We just need to feed them into our model for training.


And we will output the results of that training batch to the outputs variable.


Model.


Put IDs.


We need the attention mass.


We also want our start positions and end positions.


Now from our training batch we want to extract the loss.


And then we want to calculate loss for every parameter.


And this is for our gradient update.


And then we use the step method here to actually update those gradients.


And then this final little bit here is purely for us to see.


This is our progress bar.


So we call the loop.


We set the description which is going to be our epoch.


And then it would probably be quite useful to also see the loss in there as well.


We will set that as a post fix.


So it will appear after the progress bar.


Okay, and that should be everything.


Okay, so that looks pretty good.


We have our model training.


And as I said, this will take a little bit of time.


So I will let that run.


Okay so we have this non type error here.


And this is because within our end positions we normally expect integers.


But we're also getting some non values because the code that we used earlier where we're


checking if end position is non essentially wasn't good enough.


So as a fix for that we'll just go back and we'll add like a while loop which will keep


checking if it's non and every time it is non reduce the value that we are seeing by


one.


So go back up here and this is where the problem is coming from.


So we're just going to change this to be a while loop.


And just initialize a essentially a counter here.


And we'll use this as our go back value.


And every time the end position is still non we'll just add one to go back.


And this should work.


Need to remember to rerun anything we need to rerun.


Okay and that looks like it solved the issue.


So great we can just leave that training for a little while and I will see you when it's


done.


Okay so the model is finished and we'll go ahead and just save it.


So obviously we'll need to do that whenever actually doing this on any other projects.


So I'm just going to call it the Silbert custom.


And it's super easy to save we just do save pre trained and the model path.


Now as well as this we might also want to save the tokenizer so we have everything in


one place.


So to do that we also just use tokenizer and save pre trained again.


Okay so if we go back into our folder here see models and we have this Silbert custom


and then in here we have all the files we need to build our PyTorch model.


It's a little bit different if we're using TensorFlow but the actual saving process is


practically the same.


So now we've finished training we want to switch it out of the training mode so we use


model eval.


And just get all this information about our model as well we don't actually need any of


that.


And just like before we want to create a data loader so for that I'm going to call it val


loader and it's exactly the same code as before in fact it's probably better if we just copy


and paste some of this.


At least the loop.


So what we're going to do here is take the same loop and apply it as a validation run


with our validation data.


Paste that there initialize this data loader this time of course with the validation set.


We'll stick with the same batch size.


Now this time we do want to keep a log of accuracy so we will keep that there and we


also don't need to run most for epochs because we're not training this time we're just running


through all the batches within our loop of validation data.


So this is now validation loader and we just loop through each of those batches so we don't


need to do anything with the gradients here.


And because we're not doing anything to gradients we actually add this in to stop PyTorch from


calculating any gradients.


This will obviously save us a bit of time when we're processing all of this.


And we put those in there.


The outputs we do still want this but of course we don't need to be putting in the start and


end positions so we can remove those.


And this time we want to pull out the start prediction and end prediction.


So if we have a look at what our outputs look like before.


You see we have this model output and within here we have a few different tensors which


each have an accessible name.


So the ones that we care about are startLogits and that will give us the logits for our start


position which is essentially like a set of predictions where the highest value within


that vector represents the token ID.


So we can do that for both.


You'll see we get these tensors.


Now we only want the largest value in each one of these vectors here because that will


give us the input ID.


So to get that we use the argmax function.


And if we just use it by itself that will give us the maximum index within the whole


thing.


But we don't want that we want one for every single vector or every row.


And to do that we just set dim equal to one.


And there you go we get a full batch of outputs.


So these are our starting positions and then we also want to do the same for our ending


positions.


So we just change start to end.


So it's pretty easy.


Now obviously we want to be doing this within our loop because this is only doing one batch


and we need to do this for every single batch.


So we're just going to assign them to a variable.


And there we have our predictions.


And all we need to do now is check for an exact match.


So what I mean by exact match is we want to see whether the start positions here which


we can rename to the true values.


Whether these are equal to the predicted values down here.


And to calculate that so let me just run this so we have one batch.


That shouldn't take too long to process and we can just write the code out.


So to check this we just use the double equal syntax here.


And this will just check for matches between two arrays.


So we have the start predictions and the start true values.


So we'll check for those.


Okay so if we just have a look at what we have here we get this array of true or false.


So these ones don't look particularly good but that's fine we just want to calculate


the accuracy here.


So take the sum and we also want to divide that by the length.


Okay so that will give us our accuracy within the tensor and we just take it out using the


item method.


But we also just need to include brackets around this because at the moment we're trying


to take item of the length value.


Okay and that gives us our very poor accuracy on this final batch.


So we can take that and within here we want to append that to our accuracy list.


And we also want to do that for the end prediction as well.


And we'll just let that run through and then we can calculate our accuracy from the end


of that.


And then we can have a quick look at our accuracy here.


And we can see fortunately it's not as bad as it first seemed.


So we're getting a lot of 93, 100%, 81% that's generally pretty good.


So of course if we want to get the overall accuracy all we do is sum that and divide


by the length.


We get 63.6% for an exact match accuracy.


So what I mean by exact match is say if we take a look at a few of these that do not


match so we have a 75% match on the fourth batch.


Although that won't be particularly useful because we can't see that batch right now.


So let's just take the last batch because we have these values here.


Now if we look at what start true is we get these values.


Then if we look at start pred we get this.


So none of these match but a couple of them do get pretty close.


So these final four, all of these count as 0% on the exact match.


But in reality if you look at what we predicted for every single one of them it's predicting


just one token before.


So it's getting quite close but it's not an exact match so it scores 0.


So when you consider that with our 63.6% accuracy here that means that this model is actually


probably doing pretty well.


It's not perfect of course but it's doing pretty well.


So overall that's everything for this video.


We've gone all the way through this.


If you do want the code for this I'm going to make sure I keep a link to it in the description


so check that out if you just want to copy this across.


But for now that's everything so thank you very much for watching and I will see you


again soon.


How to Build Q&A Models in Python (Transformers)

Hi and welcome to this video on question answering with Bert.


So firstly we're going to have a look at the transformers library


and we're going to look at how we can find a Q&A model.


And then we're going to look at the Q&A pipeline.


So we're going to look at actually loading a model in python using the transformers library.


We're going to look at tokenization, how we load a tokenizer and what exactly tokenizer is actually doing.


And then we're going to take a look at the pipeline class,


which is essentially a wrapper made available by the Hugging Face Transformers library.


And it basically just makes our job in terms of building a Q&A pipeline incredibly easy.


So we're going to cover all those, it's going to be quite straightforward and quite simple.


So let's just get straight into it.


Okay so when we're doing question answering, we're essentially asking the model a question


and passing a context, which is what you can see here for the model to use to answer that question.


So you can see down here we have these three questions.


So what organization is the IPCC a part of?


And then the model will read through this and use its language modeling to figure out


which organization the IPCC is part of, which is not inherently clear from reading this.


We can see we've got IPCC here and is a scientific intergovernmental body


under the auspices of the United Nations.


So clearly the IPCC is a part of the United Nations, but it's not clear.


It's not definitively saying that in this, but once we've actually built this model,


it will quite easily be able to answer each one of these questions without any issues.


So the first thing we want to do is go over to the Hugging Face website.


And on the Hugging Face website, we just want to go over to the Models page.


So it's here.


Okay and on this Models page, the thing that we want to be looking at


is this question and answering task.


So here we have all these tasks because when you're working with transformers,


they can work with a lot of different things.


Text summarization, text classification, generation, loads of different things.


But what we want to do is question answering.


So we click on here and this filters all of the models that are available to us


just purely for question and answering.


So this is the sort of power of using the Hugging Face Transformers library.


It already has all these pre-trained models that we can just download and start using.


Now, when you want to go and apply these to specific use cases,


you probably want to fine tune it, which means you want to train it a little bit more


than what it is already trained.


But for actually getting used to how all of this works,


all you need to do is download this model and start asking questions


and understanding how everything is actually functioning.


So obviously there's a lot of models here.


We've got 262 models for question answering,


and there's new ones being added all the time.


A few of the ones that I would recommend using are the DeepSets models.


So here are the DeepSets models, eight of them for question answering.


The one that we will be using is this BERT Base Case Squad 2.


Another one that I would definitely recommend trying out is this Electra Base Squad 2.


But we will be sticking with BERT Base.


Now, it's called DeepSet here because it's from the DeepSet AI company,


and this model is being pulled directly from their GitHub repository.


So DeepSet is actually the GitHub organization,


and then this is the repository BERT Base Case Squad 2.


BERT is obviously the model BERT from Google AI.


Base is the base version of BERT.


So you can see here we have BERT large, that's just a large model. We're using the base model.


Case just refers to the fact that this model will differentiate between


uppercase and lowercase characters or words.


The alternative to this would be uncased here where there's no differentiation


between uppercase and lowercase.


And then Squad 2 refers to the question answering data set that this model has been trained on,


which is the Squad 2 version.


So we're going to take this model.


So you see DeepSet BERT Base Case Squad 2, and we are going to load it into here.


And all we need to do to do that is from transformers.


So this is the Hug & Face Transformers library.


We want to import BERT for question and answer.


So this is a specific class and using this class we can initialize a few different models,


not just this specific model.


So you can see here we have this BERT base case.


We can also initialize this BERT large uncased Roberta.


And if there's a Distill BERT as well, we can also load those in.


And what this does is it loads that BERT base case.


And what this does is it loads that specific model with its question and answering layer


added on there as well.


So this model has been trained with the extra layer specifically for question answering.


And we need to use BERT for question answering to load that.


Otherwise, if you are not using it with a specific use case and you're just wanting


to get the model itself, you can just use the auto model class like that.


But we want it for question answering, so we load this one.


Another thing to note is that we are using the PyTorch implementation of BERT here.


So Transformers works by having both TensorFlow and PyTorch as alternative frameworks working


behind the scenes.


In this case, we're using PyTorch.


If you want to switch over to TensorFlow, all you do is add TF in front of that class.


So that is our model.


And to actually load that in, all we do is copy this and we use the from pre-trained


method.


And then this is where the model name from over here comes into play.


So we've got deep set BERT base case squad 2.


And we just enter that in there.


Okay, and with that, we've actually just loaded the model.


That's all we had to do.


Of course, there are a few other steps.


This is just a model.


But there are a few steps before we actually get the data to the model.


So we need to actually process the data.


So we have this context here and this is just a string.


BERT doesn't understand strings.


BERT understands an array of integers where each integer represents a token ID.


And that token ID is very specific to BERT.


And each one is unique and represents a specific word or piece of syntax, punctuation, or so


on.


So we need to convert this string into that specific BERT ready format.


And to do that, we need to use a tokenizer.


So again, we're going to go from transformers.


I'm going to import the auto tokenizer class.


Here we can use, for example, the BERT tokenizer.


But for this, we don't need anything specific.


It's quite generic.


It will just load all of those mappings from the string or the word into the tokens.


There's no real issue there.


So we import our auto tokenizer.


And to initialize it, we just see this.


It's practically the same syntax as what we used for the first one.


The same syntax as what we used before.


We use this fromPretrain method.


And then again, we're using the same model.


OK.


And then with this, we can actually tokenize our data.


So all we need to do is write tokenizer and code.


And then let's just pass in one of these questions.


So we'll pass in the first one.


So questions.


And the first question there.


And two variables that we will need to add in here are the truncation, which we will set to true.


And the padding, which we also set to true.


So when we are setting up these models and the data going into them,


Bert in particular will expect 512 tokens with every input.


Now here, when we look at this, we can see there's probably one.


So each one of these words is most likely to be truncation.


One, so each one of these words is most likely to be a token.


And then this question mark at the end will also be a token.


So we have around 10 tokens in there.


Now, because we have padding, this will add a set of padding tokens onto the end of it


to bring that total number of tokens up to 512.


Now, alternatively, say if we had 600 tokens in there,


we would be relying on the truncation to cut the final 88 tokens to make it a total of 512.


And that's why we need those two arguments in there.


So let's see what we get from this.


You can see here that we have our tokenized input.


So Bert will be able to read and understand this.


And essentially what we have, so this 1327 is the equivalent to what?


This 2369 is equivalent to organization and so on and so on.


Now, what you might not see here is why we have this 101.


So 101 for Bert actually refers to a special token, which looks like this.


And this just signifies the start of any sequence.


So if we were to just take this, we can see that, OK, we get the same again.


We get this 101, which is the start sequence.


Then we get the start sequence token again, because that's all we've put into here.


And the Bert, the tokenizer is reading that and converting into the 101.


And then we also get this final special token as well.


And we can also see that it's here.


So this is another special token which signifies the end of a sequence,


or it signifies a separator point.


So if we write this out, you see here that the separator is 102.


And what I mean by it signifies a separation point or a separator.


So when we feed this context and this question into our Bert,


model Bert will expect it to be within the format something like this.


So we have the start sequence token.


Then we will have our context tokens.


So this will just be a list of integers, which are the token IDs.


And then what we will see is a separator token here, followed by our question.


Which again, after this is followed by a separator token.


And again, after this, we get a set of padding tokens, which look like this.


And that will just take us up to the 512 token amount.


And that's how the data going into Bert will look like.


We have that start sequence, we have the context, we have a separate,


we have a question, we have a separate and then we have padding.


It's always going to look like that when it's going into a Bert Q&A model.


So if we just remove that and this here.


And what we want to do now is actually set up this tokenizer


and our model into a pipeline, into a Q&A pipeline.


So again, we get this pipeline from the Transformers library.


So we come down here, do from Transformers, import pipeline.


And now what we want to do is just initialize a pipeline object.


So to do that, we just write pipeline.


And then in here, what we need to add is a model type.


So obviously, you can see up here, we have all of these different tasks.


So summarization, text generation and so on.


The Transformers library needs to understand, or this pipeline object


needs to understand which one of those pipelines or functions


we are intending to use.


So to tell it that we want to do question answering,


we just write question answering.


And that basically sets the wrapper of the pipeline


to handle question answering formats.


So we'll see our input and for our input, we will be passing a context and a question.


So we'll see that it will convert into the right structure that we need


for question answering, which is the CLS context separator,


question separator and padding.


It will convert into that, feed it into our tokenizer.


And the output of that tokenizer, our token IDs will be fed into BERT.


BERT will return us a span start and span end,


which is essentially two numbers, which signify the start position


and end position of our answer within the context.


And this pipeline will take those two numbers and apply them to our context


to get the text, which is our answer from that.


So it's essentially just a little wrapper and it adds a few functionalities


so that we don't have to worry about converting all of these things.


So now we just need to pass in our model.


And the tokenizer as well.


And it's as simple as that.


That's our pipeline set up.


So if we want to use that now, all we need to do is write NLP.


And then here we pass a dictionary.


And this dictionary, like I said before, needs to contain our question and context.


So the question.


And for this, we will just pass the first of our questions up here again.


So this questions at the index zero.


And then we also pass our context,


which is inside the context variable up here.


Okay.


And this will output a dictionary containing the, well, we can see the score of the answer.


So that is the model's confidence that this is actually an answer.


Like I said before, the start index and end index and what those start index and end index map to,


which is United Nations.


So our question was, what is the answer?


And we got United Nations, which is correct.


So let me just show you what I mean with this start and end.


So if we do 118 here, we get the first letter of our answer because we are going through here


and it is pulling out this specific character.


If we then add the first letter of our answer,


if we then add this and go all the way up to our end, which is at 132,


we get the full set because what we're doing here is pulling out all the characters from you


or character 118 all the way up to character 132, which is actually this comma here.


But obviously with Python list indexing, we get the character before.


And that gives us United Nations, which is our answer.


So let's ask another question.


We have what UN organizations established the IPCC?


And we get this WMO and United Nations Environment Program, UNEP.


So if we go in here, we can see it was first established in 1988


by two United Nations organizations, the World Meteorological Organization,


WMO, and the United Nations Environment Program, UNEP.


So here we have two organizations and it is only actually pulling out one of those.


So I think the reason for that is all that is reading is WMO and United Nations Environment


Program. So it is pulling out those two organizations in the end, just not the full name of the first one.


So it's still a pretty good result.


And let's go down to this final question.


So what does the UN want to stabilize?


And here we're getting the answer of greenhouse gas concentrations in the atmosphere.


So if we go down here, we can see the ultimate objective of the UNFCCC


is to stabilize greenhouse gas concentrations in the atmosphere at a level that would prevent


dangerous anthropogenic interference with the climate system.


So again, we are getting the answer, stabilize greenhouse gas concentrations.


So our model has gone through each one of those questions and successfully answered them.


And all we've done is written a few lines of code.


And this is without us fine tuning them at all.


Now, when you do go and apply these to your own problems, sometimes you won't need to do


any fine tuning and the model as is will be more than enough.


But a lot of the time you will need to fine tune it.


And in that case, the answer is yes.


You fine tune it and in that case, there are a few extra steps.


But for this introduction, that's everything I wanted to cover there.


In terms of fine tuning, I have covered that in another video.


So I will put a link to that in the description.


But that's everything for this video.


So thank you very much for watching.


I hope you enjoyed and I will see you again next time.


Thanks.


Language Generation with OpenAI's GPT-2 in Python

Hi and welcome to the video. We're going to go through language generation using GPT-2.


Now this is actually incredibly easy to do and we can build this entire model including the imports,


the tokenizer model and outputting our generated text with just seven lines of code which is


pretty insane. Now the only libraries we need for this are PyTorch and Transformers. So we'll go


ahead and import them now. Now all we need from the Transformers library are the GPT-2 LMHead


model and GPT-2 tokenizer. So we can initialize both of those as well now and both will be from


pre-trained. So now we have initialized our tokenizer and model. We just need a sequence of


text to feed in and get our model going. So I've taken a snippet of text from the Wikipedia page


of Winston Churchill which is here and it's just a small little snippet talking about when he took


office during World War II. Now from this I've tested it briefly and it seems to give some


pretty interesting results. So we will go ahead use this. All we need to do is tokenize it.


Now all we're doing here is taking each of these words, splitting them into tokens. So that would


be a list where each word is its own item. So he began his premiership. Each one of those would be


a separate value within that list. Once we have them in that tokenized format our tokenizer will


then convert them into numerical IDs which map to a word vector that's been trained to work with GPT-2.


Now because we're using PyTorch we just need to remember to return PT tensors here.


So now we have our inputs we just need to feed them into our model. So we can do that using


model.generate. We add our inputs. Now we also need to tell PyTorch how long we want our generate


sequence to be. So all we do for that is add a max length. And this will act as the cut off point.


Anything longer than this will simply be cut off. And now here we are just generating our output.


We also need to pass this into the outputs variable here so that we can actually read from it and


decode it. So to decode our output IDs because it will output numerical IDs representing words just


like we fed into it we need to use the tokenizer decode method.


And our output IDs are in the zero index of the outputs object. And we also want to skip any


special tokens. So this would be stuff like end of sequence tokens, padding tokens, unknown word


tokens and so on. And then we can print the text. Now we can see here that it's basically just going


over and over again saying the same things which is not really what we want. So this is a pretty


common problem and all we need to do to fix this is add a new output. So we can just add a new


argument to our generator method here. So we simply do sample equals true. And then we can rerun this.


And this looks pretty good now.


So we can add more randomness and restrict the number of possible texts.


So we can add more randomness and restrict the number of possible tokens for the model to use


using the temperature and top k parameters respectively. Now temperature acts as the amount


of randomness input into the model. So a high temperature above one will create more random


tokens than the default. Anything below one makes the model less random. So say if we put a


super high number like five we will probably get a pretty weird output.


Okay so we can see here initially skimming over it doesn't look too bad but then when you start


reading it it's practically impossible to follow. There's no structure and there's just a couple of


random words in there that are just completely irrelevant. Now we can also see here as an end


bracket and there's starting bracket that pairs with it. And generally just some really weird


syntax. So we turn the temperature down to 0.7 and we will actually decrease the randomness from


the original model. Now you can toy around with this and see what produces more interesting


results. Generally higher temperature will create more creative outputs. And the other parameter we


can also use is the top k parameter. Now top k limits the sample tokens to the top rated tokens


that the model is predicting. So we can add 50 for example and this will alter our output.


Generally I've found top k tends to make the text a little more coherent and I would assume this is


because it is sticking within a smaller space of possible tokens or words that you can output.


So now here we can see pretty understandable logical text again. And we can see here it


mentions Lord a lot which makes sense because this is Britain. So if we put the temperature back up to


one we should get a slightly more random output again. And then here we can see that there's a


little more weird text coming in. So we have here that the first Australian Prime Minister


was sacked by a Labour Minister which obviously is a little bit strange. But it just shows that


we can add more randomness or we can try and restrict our model to become more coherent.


And we can do this super easily using the generate parameters. So with just a few more


lines of code we built our model up and running and actually generating text incredibly easily.


So I hope this has been insightful and useful. If you have any questions or suggestions please just


How-to Structure a Q&A ML App

Okay so I had an idea. I thought maybe what we could do is actually put


How-to Structure a Q&A ML App

together a few videos and what we'll do in the next few videos is build out a


How-to Structure a Q&A ML App

full Q&A project and this is something that I've actually been wanting to do


How-to Structure a Q&A ML App

for quite a while now and I think it will actually be really cool. I was


How-to Structure a Q&A ML App

thinking through this morning and I thought okay today we're gonna start it.


How-to Structure a Q&A ML App

So what you can see on the screen right now is the GitHub repo for this project


How-to Structure a Q&A ML App

which is completely empty almost. There's a read me a gitignore and this one


How-to Structure a Q&A ML App

architecture drawing that I sell literally you can see four minutes ago.


How-to Structure a Q&A ML App

So I'm gonna show you that and I'm going to explain what we're actually going to


How-to Structure a Q&A ML App

go through and I think we're gonna cover a lot of different things. So I think


How-to Structure a Q&A ML App

this is I think that is one of the reasons why I think this is such a cool


How-to Structure a Q&A ML App

project. So at the moment this is like the basic architecture of what we would


How-to Structure a Q&A ML App

need to build a Q&A model and the end goal is to have is to have a front end


How-to Structure a Q&A ML App

which looks kind of like this. So we'll have like a search bar here and we'll


How-to Structure a Q&A ML App

have some visual up here. It's gonna be a little bit better than the six man. I'm


How-to Structure a Q&A ML App

gonna show you what I already have and maybe we can use that or maybe we'll do


How-to Structure a Q&A ML App

something different I don't know. And what we're gonna be able to do is ask a


How-to Structure a Q&A ML App

question here and we're going to be able to answer the question based on a


How-to Structure a Q&A ML App

Soic philosophy books. So I haven't really read any of these I've read like


How-to Structure a Q&A ML App

little bits but they're pretty interesting and I think quite unique. So


How-to Structure a Q&A ML App

as far as I know there's definitely not anything like this out there at the


How-to Structure a Q&A ML App

moment where you ask a question and you get the answer back from some ancient


How-to Structure a Q&A ML App

Soic philosophy book. And there's only really two books that I've thought of so


How-to Structure a Q&A ML App

far which is Meditations by Marcus Aurelius and Letters from a Stoic by Seneca.


How-to Structure a Q&A ML App

And the good thing with both of these is that we can find both of them online for


How-to Structure a Q&A ML App

free so we can use Python requests to get these. So I'll just kind of put a


How-to Structure a Q&A ML App

little list here what I think we're gonna need. So the first one is actually


How-to Structure a Q&A ML App

extracting and downloading this data so we're using requests for that. And then


How-to Structure a Q&A ML App

once we have actually got that data we need to pre-process it. And when we're


How-to Structure a Q&A ML App

pre-processing it I think that will just be a case of using regex more than


How-to Structure a Q&A ML App

anything else but I'm not sure yet so let's see. So after we pre-process it and


How-to Structure a Q&A ML App

that's when we get into this stuff over here so this whole sort of stack that you


How-to Structure a Q&A ML App

can see without the API. So this is a typical it's called a reader or


How-to Structure a Q&A ML App

retriever reader and what we do is we use this up here this is our database


How-to Structure a Q&A ML App

it's a document store elasticsearch document store and what we're gonna do


How-to Structure a Q&A ML App

is feed all of these. Sorry it's getting a little bit messy with the color so let


How-to Structure a Q&A ML App

me change it. So what we're gonna do is take these and we're gonna feed them


How-to Structure a Q&A ML App

into our document store. And once we have that what we want to do is build this


How-to Structure a Q&A ML App

retriever reader stack and it will allow us to query the retriever down here and


How-to Structure a Q&A ML App

what the retriever will do is send that query to elasticsearch here which is


How-to Structure a Q&A ML App

what you can see happening there and returning from that we'll get so many


How-to Structure a Q&A ML App

different contexts. So all of the texts from meditations and letters from Stoic


How-to Structure a Q&A ML App

will split them up by maybe paragraph and store them in here and what these


How-to Structure a Q&A ML App

contexts will be are the most relevant paragraphs. And once we've done that this


How-to Structure a Q&A ML App

retriever will pass on the context to our reader model down here and what the


How-to Structure a Q&A ML App

reader model will do is say it's given a long sentence like this or paragraph it


How-to Structure a Q&A ML App

will say okay the actual answer that you want is actually only these three words


How-to Structure a Q&A ML App

here and it will return those three words and what we want to do is return


How-to Structure a Q&A ML App

those three words in our answer back to our API but alongside the answer we're


How-to Structure a Q&A ML App

also going to include the the full context here as well. So we get a few


How-to Structure a Q&A ML App

things back and I think that that's like going to be more machine learning side


How-to Structure a Q&A ML App

of it but obviously we need to support the machine learning side and I mean the


How-to Structure a Q&A ML App

very first part of that that you can obviously see here is the API so let me


How-to Structure a Q&A ML App

so let me write down so we have the ML part and for that we're going to use in


How-to Structure a Q&A ML App

something called Haysec and once we get out of that part we move on so we use


How-to Structure a Q&A ML App

different color here move on to our API the API we'll just we'll use probably


How-to Structure a Q&A ML App

fast API to set that up then once we set that up we go on to our front end part


How-to Structure a Q&A ML App

and the front end I don't I'm not a front-end developer I just mainly use


How-to Structure a Q&A ML App

Python but I do know Angular a little bit so what we're gonna do is build all


How-to Structure a Q&A ML App

of this part using Angular so this will be our angular front end


How-to Structure a Q&A ML App

and that's essentially everything we'll be covering but there's no there's quite


How-to Structure a Q&A ML App

a lot in here in particular as well what I've missed is alongside you know we


How-to Structure a Q&A ML App

have our reader model down here but what I want to try and do is rather than just


How-to Structure a Q&A ML App

taking the reader model from Hugging Face transformers as we normally would I


How-to Structure a Q&A ML App

want to actually try training it and for that we need to use something called MLM


How-to Structure a Q&A ML App

which is mass language modeling so we would need to train a BERT model using


How-to Structure a Q&A ML App

MLM or fine-tune the BERT model I should say on the data from our books up here


How-to Structure a Q&A ML App

and then we'd also want to train it so that it performs Q&A and for that we


How-to Structure a Q&A ML App

need to use the squad data set probably squad anyway so you know there's quite a


How-to Structure a Q&A ML App

lot that I think we would have to do to build this and I think it'd be pretty


How-to Structure a Q&A ML App

interesting so that is what we're going to be covering in sort of the next few


How-to Structure a Q&A ML App

videos and the one or the final little thing okay so over here we have the


How-to Structure a Q&A ML App

Marcus Aurelius Superman and I thought maybe something like this would be cool


How-to Structure a Q&A ML App

I don't know this is something I drew ages ago and this is this sorry so this


How-to Structure a Q&A ML App

is like this is Marcus Aurelius and I think something like this maybe this or


How-to Structure a Q&A ML App

something like it would be pretty cool to just have in the middle of the web


How-to Structure a Q&A ML App

page and underneath we have a search bar and keep it pretty simple so I think


How-to Structure a Q&A ML App

that's everything really for the plan and I mean the first thing we're going to


How-to Structure a Q&A ML App

do in the in the next video is actually sell requests and download that data and


How-to Structure a Q&A ML App

maybe pre-process it as well or they they might be two videos so that's


How-to Structure a Q&A ML App

everything for this video I hope that you're as excited about this as I am


How-to Structure a Q&A ML App

because I'm really looking forward to actually building all of this I think


How-to Structure a Q&A ML App

it'll be super cool and I mean ideally at the end it's one it's gonna look


How-to Structure a Q&A ML App

cool and two we're gonna learn like a huge amount of stuff if you even put all


How-to Structure a Q&A ML App

this together there's so many different things that you need to know in order


How-to Structure a Q&A ML App

to make everything work so it should be really cool and I'm looking forward to


How-to Structure a Q&A ML App

getting started with it so I will see you in the next video where we'll


How-to Structure a Q&A ML App


Streamlit for ML #5.1 - Custom React Components in Streamlit Setup

Today we're going to have a look at how we can build custom components in Streamlit using React and external libraries like Material UI.


So there are already plenty of components that you can use directly from Streamlit and also from the Streamlit community.


Obviously you may find yourself needing to use something that doesn't already exist.


So we'll go through that process. Fortunately it's not too difficult. It does require a little bit of front end code.


But Streamlit does make it pretty straightforward.


So let's have a quick look at what this card might look like. This component is a card.


So all we have for our custom component is this little card in the middle here.


The rest of this is just Streamlit. And then this bit here is based on a Material UI component.


So if I go back we just have a title, subtitle, text or body of text and a link here which is a Material UI button.


Click on it and for now it's just taken to Google.


But we can essentially like an object in Python we can pass whatever values we like to this component.


So here are the actual cards from Material UI that we're basing this on.


And with these cards we can include much more than what I have shown you.


This is just the absolute basics. Like we can include these little profile images, drop downs, options and so on.


Even little expanders, pictures. So there's a lot we can do with it.


And as far as I know there isn't anything like this available directly within Streamlit.


So let's go ahead and set up our environment for developing this component.


So if you are on Mac and you have Homebrew installed you can just go brew, install node.


And this will install Node.js and the node package manager.


I already have it installed so I'm not going to rerun it.


If you are not on Mac or you don't have Homebrew you can download it from the Node.js website.


So go over here. This here. Node.js.org. And then download.


OK. And you have Windows, Mac and so on.


So once you have that installed you also need to install Streamlit.


I'm going to assume you already have it installed.


But if not you just pip install Streamlit.


Now I have Streamlit in another environment so I'm going to activate that.


I'm going to activate Streamlit.


And the Streamlit side of things I will run from this terminal window.


Now to build our component for Streamlit we need to follow a set of structure.


And fortunately Streamlit provides us with a few templates that we can start from to make our lives a little bit easier.


So let me show you where you can find those.


So just type in Streamlit GitHub or maybe we can go with Streamlit Components template.


It's probably easier.


Click on here.


OK. And we have the Streamlit Component Template repo.


We just want to git clone this so we can code over here. Copy this.


And then switch back to our terminal.


Navigate to whatever folder you're going to store the template folder within.


So I'm going to go to Document Projects and I'm going to write git clone.


And then I'm going to clone the component template repository.


OK. Let's open that VS Code and we'll have a look at what we have in here.


OK. So.


On the right over here we have our directory structure. We have a few different templates.


So we're going to go this template here which uses React.


And within this template directory this is our actual project.


So we can almost ignore the rest of this stuff here.


Just anything within template is what we care about.


We have set up py. So we're going to use this and manifest for creating a pip package which will contain our components.


So to actually use that component we put in sol.app.component and then we just import it into our streamlit app script.


Or app.py usually.


And then use it.


And within here we just have the default file names here or default directory names of my component.


We're going to change some of these inside.


So here we have initpy. So that's where we're initializing our streamlit server or app from.


We'll go through that. It's fine.


We have this front end which is anything in here is the React side of things.


So if we go a little further we have some styling. The index.


And then in here we have these TSX files.


So in here we're using TypeScript. So if you know TypeScript it's really good.


If not it's not too much different from Python. I mean it's fairly different but it's not unbelievably different.


So I think if you know Python you'll probably read this at least and kind of follow what we're doing.


But I also don't really know TypeScript. I can just get through putting something simple together with this.


So most of the work we're going to do is going to be my component TSX.


But we are going to modify a lot of these files as well.


So first let's rename everything because everything at the moment is using default names.


And we obviously are building a custom component. We want to give it our own name. That makes it a little more identifiable.


So we'll start by going to my component up here. Rename that to stCardComponent.


So st is just Streamlit Card Component.


Down here we have myComponent. Rename that to cardComponent.


Basically anywhere that we have myComponent we modify to cardComponent. We update these imports.


So inside cardComponent here if we just find and replace. So find myComponent and replace that with cardComponent.


Replace all of those. See if we have anything for myComponent.


No. OK. Save that and let's have a look at the index. I think we also have something here.


Just myComponent here.


OK. Replace those.


So we have everything in there.


And in set of Pi we also want to update the name here.


So this will change it to Streamlit Card Component.


OK. So this defines it for our package later on.


And this actually should align with the directory or the folder that we have here.


So not Streamlit but st.


And in our manifest here we also need to update this to be stCardComponent.


So this needs to point to the front end.


We don't have the build directory yet but we will. So that needs to go stCardComponent.


OK. So that should pretty much be everything we need to rename.


Maybe other little bits are actually in init.


Here we also want to change this to stCardComponent.


So here stCardComponent.


Anything else?


OK. That looks good.


So let's go ahead and actually initialize the current or basically default other than the things we've renamed version of this component and see how that looks.


So we'll go to and here we have the component template.


Inside the template.


See the template.


And then we have stCardComponent.


So in here we also have front end directory and inside here we have all of our packages and everything.


So like the node side of stuff.


So first thing we need to do is actually install the node packages we need for our CARD component.


So. So we just do npm install and this is just going to install everything from this package.json file.


So run that. It might take a moment.


So just give it a second.


OK. So that's done.


And next thing we want to do is install the node packages that are required specifically for the CARD that we're going to build.


So these are all the node packages we've just installed.


They're extremely to function like the core of the packages that we need.


But because we are using this material UI CARD thing we need a few extra things.


So we need to npm install again.


This is kind of similar to like a pip install.


Although specific to this directory.


So when we pip install we install them to our Python environment.


In this case it's almost like the environment is this directory in this project.


So npm install.


We need MUI material.


MUI icons material.


Icons material.


Emotion react.


And also emotion styled.


Now I think this is probably going to give us an error.


Let's see.


OK. So yes we get this error.


We have a dependency conflict with I think if I remember correctly.


So we have this 3mic component template and this throws some dependency conflicts with the MUI material stuff.


So it's annoying but we can just get around it by adding this legacy pia-deps.


Now I haven't been playing around with this and I haven't noticed any issues pop up from using this legacy pia-deps.


But obviously you just need to be aware. Don't just throw it in there all the time.


But in this case it seems to work fine.


OK.


So with that we should be able to run everything.


So let's go ahead and do that.


First thing we want to do is so we're going to run two things here.


We're going to run within this front end directory.


We write npm start.


This is going to initialize or execute the server that hosts our react component.


And then we also need to open another terminal window.


We also need to navigate to our template directory.


So cd documents projects.


It is component template.


Templates.


cd template.


And in here we have a sd card component.


OK. We have this initpy.


Initpy is like a template app for just running and testing our component.


So we just write streamlit run init.py.


So previous videos you've probably seen me write streamlit run app.py.


This is kind of acting as our app.py whilst we're in the development stage.


But once we switch to a release version this initpy we will modify a little bit.


And it will not be for doing this streamlit run.


It will do something slightly different.


It will extract everything from a compiled build distribution.


So now we have these.


We're hosting two servers.


We have this localhost 3001.


If we go over there.


Open that.


We're going to see nothing.


So this is just hosting our React component.


But our React component by itself doesn't actually show anything.


So we actually need to go over and open the streamlit localhost.


So that is in 8.5.0.1.


Now we have this template component.


So we can click here and it updates this basically.


Maybe even change this.


Enter. We get this.


So that's cool.


But obviously we want to build a custom component.


And we'll go ahead and we'll do that in the next video.


For now I think we'll leave it there.


We've set up the environment.


We've started running the default component.


And I think that's good enough for now.


And in the next video we'll be able to really focus on actually building the component itself.


Which means we're going to be toying with code over in the card component TSX quite a bit.


And cleaning this up.


And yeah.


Creating our card.


So I hope that's been useful.


Thank you very much for watching.


Making The Most of Data: Augmented SBERT

In this video we're going to have a look at how we can make the most of limited data using


language data augmentation strategies and training approaches.


More specifically we're going to focus on something called augmented experts.


So you may or may not be aware that the past decade has been sort of a renaissance or explosion


in the field of machine learning and data science and


a lot of that, especially the early progress with things like


perceptron and neural networks, a lot of that was researched and discovered


back in the 50s and 60s and 70s, but we


didn't see that really applied in industry or


anywhere really until the past decade and


there are two main reasons for this.


So the first is that we didn't have enough compute power back in the


50s, 60s, 70s to train the models that we needed to train and we also didn't have the data to actually


train those models. Now


compute power is


not really a problem anymore. We sort of look at this graph, it depends on what model you're training


of course if you are open AI and you're training GPT-4 or 5 or whatever


yeah, maybe compute power is pretty relevant, but for most of us we can get access to


cloud machinery, personal machines and we can wait


a few hours or a couple of days and fine tune or pre-train a transform model


that is


good performance for what we need.


Now that obviously wasn't always the case until very recently back in


1960s, you see on this graph here we have the IBM


704 and


you can see under the Y axes we have floating point operations per second


and that's a logarithmic scale.


So linear scale just basically looks like a straight line until a few years ago and then shoots up.


It's


pretty impressive how much progress is made in terms of


computing power. Now


like I said, that's not really an issue for us anymore.


We have the compute in most cases to do what we need to do and


and data is not as much of a problem anymore, but we'll talk about that in a moment.


So data again, we have a very


big increase in data, not quite as big as the computing power and this graph here doesn't go quite as far back.


It's only 2010


where I believe it was at 2 zettabytes and now


71 or


so in 2021. So there's a fairly big increase, not quite as much as computing power


over time, but still pretty massive. Now the


thing with data is


yes, there's a lot of data out there, but is there that much data out there for


what we need to train models to do and


in a lot of cases, yes, there is.


But it really depends on what you're doing. If you are


focusing on


a more niche domain. So what I have here on the left over here are a couple of niche domains.


There's not that much data out there on


sentence pairs for climate evidence and claims, for example. So where you have a


piece of evidence and a claim and whether the claim supports evidence or not, there is a very small data set called climate fever data set, but it's not big.


For agriculture, I assume within that industry, there's not that much data, although I


have never worked in that industry. So I


am not fully aware. I just assume there's probably not that much.


And then also niche finance, which I do


at least have a bit more experience with and I imagine this is probably something that a lot of you will find useful as well.


Because


finance is a big industry. There's a lot of finance data out there, but there's a lot of niche


projects and problems in finance where you find much less data.


So yes, we have a lot more data nowadays, but we don't have enough for a lot of the data that we need.


On the right here, we have a couple of examples of low resource data sets. So we have


Adave from the Maldives and also the Navajo languages as well. So with these


we kind of need to find a different approach. Now we can


investigate, depending on your use case, unsupervised learning, TSEA, which we have covered in a previous video article and


that does work when you're trying to build a model that recognizes


generic similarity. It works very well as well.


But for example with the climate claims data, we are not necessarily trying to


match sentence A and B based on their semantic similarity.


But we're trying to match sentence A, which is a claim, to sentence B, which is a claim.


As to whether that evidence supports the claim or not.


So in that case, unsupervised approach like TSEA doesn't really work.


So what we have is very little data and there aren't really


any alternative training approaches that we can use. So basically what we need to do is


create more data. Now data orientation is


difficult, particularly for language. So data orientation is not specific to NLP.


It's used across ML and it's more established in the field of computer vision.


And that makes sense because computer vision, say you have an image, you can modify that image using a few different approaches.


And a person can still look at that image and think, OK, that is the same image.


It's just maybe it's rotated a little bit. We've changed the color grading, the brightness,


or something along those lines. We just modified it slightly. But it's still in essence the same image.


Now for language it's a bit difficult because language is very abstract and nuanced.


So if you start randomly changing certain words, the chances are you're going to produce something


that doesn't make any sense. And we, when we're augmenting our data, we don't want to just throw


rubbish into our model. We want something that makes sense. So there are some data augmentation


techniques. And we'll look at a couple of the simpler ones now. So there is a library called NLP.org


which I think is very good for this sort of thing. It's essentially a library that allows us to do


data augmentation for NLP. And what you can see here is two methods using word2vec vectors and


similarity. And what we're doing is taking this original sentence. So the quick brown fox jumps


over the lazy dog. And we're just inserting some words using word2vec. So we're trying to find


what words word2vec thinks could go in here, which words are the most similar to the surrounding


words. And we have this al-Ziari, which I don't know. I think it seems like a name to me. But I


am not sure. That I don't think really fits there. So it's not great. It's not perfect.


Lazy superintendents dog. That does kind of make sense. I feel like a lazy superintendents dog is


maybe a stereotype or I'm sure it's been in The Simpsons or something before. So,


okay, fair enough. I can see how that can fit in there. Which again, it's a bit weird. It's not


great. Substitution for me seems to work better. So rather than the quick brown fox, we have the


easy brown fox. And rather than jumping over the lazy dog, jumps around the lazy dog. Which


changes the meaning slightly. Easy is a bit weird there to be fair. But we still have a sentence


that kind of makes sense. So that's good, I think. Now we don't have to use words to vet. We can also


use contextual word embeddings like with Bert. And for me, I think the results look better. So


for insertion, we get even the quick brown fox usually jumps over the lazy dog. So we're adding


some words there. It makes sense. That's I think good for substitution. And we're only doing one


word here. And we're changing that to a little quick brown fox instead of just quick brown fox.


So I think that makes sense. And this is a good way of augmenting your data and bring more data


from less. But for us, because we are using sentence pairs, we can basically just take


all of the data from say we have A and B over here. Imagine this is a data frame.


And we have all of these sentence A's and we have all these sentence B's. Now if we take one sentence


A, it's already matched up to one sentence B. And what we can do is say, OK, I want to randomly


sample some other sentence B's and match them up to our sentence A. So we have three more pairs now.


OK, so if we did this, if we took three sentence A's, three sentence B's, and we made new pairs


from all of them, not really random sampling, just taking all the possible pairs, we end up with


nine new or nine pairs in total, which is much better if you


extend that a little further. So from just a thousand pairs, we can end up with one million pairs.


So you can see quite quickly, you can take a small data set and very quickly create a big data set


with it. Now this is just one part of the problem though, because our smaller data set


will have similarity scores or natural language inference labels, but the new data set


that we've just created, the augmented data set, doesn't have any of those, just randomly sampled


new sentence pairs. So there's no scores or labels there and we need those to actually train and model.


So what we can do is take a slightly different approach or add another step into here.


Now that other set is using something called a cross encoder. So in semantic similarity,


we can use two different types of models. We can use a cross encoder, which is over here,


or we can use a bi-encoder or what I would usually call a sentence transporter.


Now a cross encoder is the sort of old way of doing it and it works by simply putting


sentence A and sentence B into a BERT model together at once. So we have sentence A,


separate a token, sentence B, feed that into a BERT model and from that BERT model we will get all of our


embeddings, output embeddings over here and they all get fed into a linear layer,


which converts all of those into a similarity score up here. Now that similarity score is


typically going to be more accurate than a similarity score that you get from a bi-encoder or a


sentence transformer. But the problem here is from our sentence transformer we are outputting


sentence vectors and if we have two sentence vectors we can perform a cosine similarity or


a Buclidean distance calculation to get the similarity of those two vectors. And the cosine


similarity calculation or operation is much quicker than a full BERT inference set,


which is what we need with a cross encoder. So I think it is something like a BERT model


so I think it is something like for maybe 10 maybe clustering 10,000 vectors using a cross encoder,


an expert cross encoder, would take you something like 65 hours whereas with a bi-encoder it's going


to take you about five seconds. So it's much much quicker. And that's why we use bi-encoders


or sentence transformers. Now the reason I'm talking about cross encoders is because we get


this more accurate similarity score which we can use as a label. And another very key thing here


is that we need less data to train a cross encoder. With a bi-encoder if we I think the


SBERT model itself was trained on something like one million sentence pairs and some new


models are training a billion or more. Whereas a cross encoder we can train a reasonable cross


encoder on something like 5k or maybe even less sentence pairs. So we need much less data and


that works quite well what we've been talking about with data orientation. We can take a small data set


we can augment it to create more sentence pairs and then what we do is train on that original


data set which we call the gold data set. We train our cross encoder using that and then we use that


fine-tuned cross encoder to label the augmented data set without labels and that creates a augmented


label data set that we call the silver data set. So that sort of strategy of creating a silver


data set which we would then use to fine-tune our bi-encoder model is what we refer to as the


in-domain augmented SBERT training strategy. And this sort of what you can see this flow diagram


is basically every set that we need to do to create an in-domain or SBERT training process.


So we've already described most of this so we get our gold data set, the original data set.


That's going to be quite small let's say one to five thousand sentence pairs that are labeled.


From that we're going to use something like random sampling which I'll just


call random sample. We're going to use that to create a larger data set. Let's say we create


something like a hundred thousand sentence pairs but these are not labeled. We don't have any


similarity scores or natural language inference labels for these. So what we do is we take that


gold data set and we take it down here and we fine-tune a cross encoder using that gold data


because we need less data to train a reasonably good cross encoder. So we take that and we


fine-tune cross encoder and then we use that cross encoder alongside our unlabeled data set


to create a new silver data set. Now the cross encoder is going to predict the similarity scores


or NLI labels for every pair in that data set. So with that we have our silver data. We also have


the gold data which is up here and we actually take both those together and we fine-tune the


by encoder or the sentence transformer on both the gold data and the silver data. Now one thing I


would say here is it's useful to separate some of your gold data at the very start so don't even


train your cross encoder on those. It's good to separate them as your evaluation or test set


and evaluate both the cross encoder performance and also your by encoder performance on that


separate set. So don't include that in your training data for any of your models. Keep that


separate and then you can use that to figure out is this working or is it not working. So


that is in the main org fermented experts and sort of see this is the same as what you saw before


just another this is the training approach. So we have the gold trained cross encoder.


We have our unlabeled pairs which have come from random sampling our gold data.


We process those for a cross encoder to create the silver data set and then the silver and the gold


come over here to fine-tune a by encoder. So that's it for the theory and the concepts


and now what I want to do is actually go through the code and and we'll work through an example


of how we can actually do this. Okay so we have downloaded the both the training and the validation


set for our scsb data and let's have a look at what some of that data looks like. So scsb


zero. So we have sentence pair sentence one sentence two just a simple sentence and we have