data. Write our text data but we also so this is a list and what we want to do is


How to Build a Bert WordPiece Tokenizer in Python and HuggingFace

we want to join every item in that list separated by a newline character so we


write this and that creates our files. Now at this point we've created a file


and our text data still has 5,000 items in it and we're going to start looping


through and populating it with even more items. So what we need to do now is


re-initialize or empty our text data variable. So that's empty again it can


start counting from zero all the way up to 5,000 again. Okay and so at this point we're


saving our file so this will initially be file underscore zero.txt but if we


loop through again and do it again it's still going to be zero so we need to


make sure we are increasing that file count so that it's not remaining the


same just overwriting the same file over and over again. Okay and what you can also


do if you want is you can add another so this down here with open you can add


that just in case there's any leftover items at the end there that haven't


been saved into this neat 5,000 chunk. I'm not going to do that now but you can


add that in if you want to. Okay so it looks pretty good the only thing that we


we do need here is actually make sure the encoding is UTF-8 otherwise we'll


get I think we'll get an error if we miss that. Okay so that will or should create all of our


data so let's let me open that directory here on the left so we have this empty


OSCA directory. I'm going to run this and we should see it get populated so it's


pretty quick. There we go so we're building all these plain text files here


and if we open that ignore that and we see that we get all of these so each row


here is a new sample. Okay and as you can see it's all all Italian so that's our


data it's ready and what we can do is move on to actually training the


tokenizer. So the first thing we actually need to do is get a list of all


those files and that we can pass on to our our tokenizer. So to do that we'll


use the pathlib library so from pathlib import path and we just go string x for


x in path so so here we need to specify the directory where our files will be


found so that is just OSCA and then the end here we just add this glob and here


we don't fit in this case we don't need to do this because if we if we just use


path here it will just select all of the files in that directory and in our case


we we can actually do that because there's no other files other than the


text files but it's good practice just in case there is anything else in there


we can use a we can use this function here to say within this directory just


select all text files okay and then let's have a look so in paths we have we


should have all of our files and let's see how many of those we have okay so we


have ten of those so in total yep 50,000 in total samples there because you have


5,000 in each file okay so now let's initialize our plain tokenizer so we


want to do from tokenizers so if you don't have tokenizers installed super


easy all you have to do is do pip install tokenizers again this is another


HuginFace library like transformers or datasets which we used before and from


transformers want to import the BERT word piece tokenizer which is showing


us there so we load that and then our tokenizer we initialize it with BERT


tokenizer again and then in here we have a few a few different variables which


are useful to to understand so first one is clean text so this just removes


obvious characters that we don't want and converts all white space into spaces


so we can say that's true we have handle Chinese characters now this I'll leave


it as false but but what this does is if it sees a Chinese character in your


training data what it's going to do is just add spaces around that character


which as far as I know allows but at least when we're tokenizing those


Chinese characters it allows them to be better represented I assume but I


obviously I don't know Chinese and I have never trained anything in Chinese


so I don't know but that's what it does has strip accents so this is a pretty


relevant one for us so this is safe we have like a E like this it will convert


it into this obviously for romance languages like Italian we those accents


are pretty important so we don't want to we don't want to strip those it's also


strip not string and then the final one the lowercase so this is if we want to


if we want to view this is equal to this we would set low case equal to true in


this case we I you know for me I'm happy to have those capital characters as


being equal to low case characters that's completely fine so that initializes


handle Chinese sorry handle Chinese characters like this so that initializes


our tokenizer now we train it so tokenizer dot train in here we need to


first pass our files so the is it pass that we use up here yeah pass so


training it with those we want to set the vocab size so this is the number of


tokens that we can have within our tokenizer it can be very small for us


because we don't have that much data in there I want to set the min frequency


which initially I thought oh that must mean that you know the minimum number of


times a token must be found in the data for it to be added to the vocabulary but


it's not it's actually the the minimum number of times that the it must see two


different tokens or characters together in order for it to consider these as


actually a token by themselves so submerged together so typically I think


people use two for that which is fine special tokens so these are the special


tokens I use by BERT special underscore tokens and for that we will have padding


so the padding token the unknown token the classified token which we put the


sort of every every sequence let me put this on a new line we have the separator


token which we put the end of of a sequence and then we also have the mask


token which is pretty important we are training that core model we also have


limit the alphabet so this is the number of different characters that we can see


within our vocab so limit alphabet so we'll go with 1000 and word pieces


prefix so this is what we saw before in the example where we had the two the two


hashes and this like I said it just indicates a piece of a word rather than


a full word and that should be it actually so I don't think there's really


anything else that is important for us so we'll train that make sure hopefully


it will work again this can take a lot that what this will take a little bit of


time even without our smaller smaller data set so let's see what it's sharing


us it's not I don't know why I think I need to install something here because


I just get this blank output I think it's supposed to be a loading bar and


then what we we do at this point is we probably want to save our new tokenizer


so I'm going to save as new tokenizer and I just write tokenizer dot save


model and that is going to go to the new tokenizer directory so new tokenizer


okay and that will save this vocab text file and if we just have a quick look at


what that has inside so come over here let's we have new tokenizer vocab text


and then in here we can see all of our all of our tokens okay so the way that


this the way that this works yeah so you can actually see you know how we use


that alphabet the limit alphabet we can see that there are 1000 tokens so this


stops at 1005 and then go all the way up here so it begins at row 6 so that's the


1000 alphabet characters so single characters that are within or allowed


within our tokenizer now the oscdata set is just pulled from the internet so you


do get a lot of random stuff in there so we have we have a lot of Chinese


characters when we're dealing with Italian but if we come down here we start


to see some of those Italian words and these are the tokens so our text our


tokenizer is going to read our text it's going to split it out into these tokens


so like the ABB, OP, FIN, it's going to split out into those and then the next


step is to convert those into token IDs which are represented by the row numbers


of those tokens so if it saw FIN in the text it would replace that with


the FIN token and then it would replace the FIN token with this 2201 now let's


let's see how that works so first thing is how do we load that tokenizer we do


it as we normally would so from transformers import we're using a BERT


tokenizer so import BERT tokenizer and we'll say tokenizer equals BERT tokenizer


from pre-trained and all we do is we point that to where we saved it so


it's a new tokenizer and that should load okay so first I want to say tokenizer


and I'm going to tokenize ciao come here and this is just hi how are you and we


see that we get these these tokens here so we have number two here which if you


probably don't remember in our text file at the top we had our special


tokens row number two we had the CLS token so the classified token which we


always put at the start of a sequence and at the end we also have this three


which is the separator token now if we just go and open that vocab file that


we built so it's new tokenizer new tokenizer vocab.txt if we read that in


so let's write vocab fp.read and we want to split by newline characters so


split like so because every token is separated by a newline we can see let's


have a look at the special tokens so we have padding unknown CLS at


position number two and separate a position number three so if I were to go


to we'd get CLS which aligns to what we have here so what we can do is we could


take all of these values and we could use them to identify from this vocab and


we can do it using tokenizer decode by the way as well but I'm going to do it


by indexing in that vocab file that we built just to you know show that that's


what a vocab file actually is that's how it's used so if I write that out so we


have take this I want to access input IDs


okay and what I'm going to do is say for I in that list can you print vocab vocab


I and at the end we'll just add a space okay and then we get this so CLS


deciding classifier token child estimation mark come Eva question mark


separate a token which marks the end of our our sentence which I think is is


pretty cool now let's let's try that with something else so take this again


do I want okay so yeah we'll just do this so what I say a lot when in Italy


is a computer me and thing and okay Peter and the entei means I understood


nothing which is very useful so if I print that out we see that we get CLS


a computer niente separator now what we're seeing here is you know full


words not we're not seeing any word pieces so if I can find it I think this


will hopefully return a word piece so response it's a billy time like this let


me try this will hopefully return a few word pieces yes there we go okay so we


see respond see Billy tap and these are our different word pieces so this gets


separated into not just a single token but four tokens which is pretty cool now


I think I mean that's it for for this video don't think yeah that means I


think nothing else on to I think we need to cover that's pretty much everything


really needs no for building a word piece so Kaiser for using with with


Sentiment Analysis on ANY Length of Text With Transformers (Python)

In this video we're going to take a look at how we can apply sentiment analysis to longer pieces


of text. So if you've done this sort of thing before in particular with transformers or even


LSTMs or any other architecture in NLP you will find that we have an upper limit on the number of


words that we can consider at once. In this tutorial we're going to be using BERT which is


a transformer model and at max that consumes 512 tokens. Anything beyond that is just truncated


so we don't consider anything beyond that limit. Now in a lot of cases maybe for example you're


analyzing sentiment of tweets it's not a problem but when we start to look at maybe news articles


or reddit posts they can be quite a bit longer than just 512 tokens and when I say tokens tokens


typically maps to words or punctuation. So what I want to explore in this video is how we can


actually remove that limitation and just consume as many tokens or words as we'd like and still get


a accurate sentiment score whilst considering the full length of text. And at a high level this is


essentially what we are going to be doing. We're going to be taking the original tensor which is


the 1361 tokens and we're going to split into different chunks so we have chunk one, chunk two


and chunk three here. Now we want most of these chunks or all of these chunks in the end are going


to be 512 tokens long and you can see with chunk one, chunk two they are 512 already.


However of course 1361 can't be evenly split into 512 so the final chunk will be shorter.


And once we have split those into chunks we will need to add padding, we need to add the


start sequence and separator tokens. If that is new to you then don't worry we'll explain that


very soon. And then we calculate the sentiment for each one of those, take the average and then use


that as a sentiment prediction for the entire text. And that's essentially a high level what we're


going to be doing but that is much easier said than done. So let's just jump straight into the


code and I'll show you how we actually do this. Okay what we have here is a post from the investing


subreddit. It's pretty long I think it's something like 1300 tokens when we tokenize it and obviously


that is far beyond the 512 token limit that we have with BERT. So if we want to consider the full


text we obviously have to do something different. And the first thing that I think we want to do is


actually initialize our model and tokenizer. Because we're using BERT for sequence classification


we will import the BERT for sequence classification model or class. And we are importing that from


the transformers library. So that is going to be our model class. Then we also need the tokenizer


as well which is just a generic BERT tokenizer. So those two are our imports and then we actually


need to initialize the tokenizer and the model. So the BERT tokenizer is pretty straightforward.


And then we are going from pre-trained. So we're using a pre-trained model here. And if we just


open the Hugging Face Transformers Models page. So Hugging Face dot co slash models.


And we can head over here and we can actually search for the model that we'd like to use.


We're doing text classification so we head over here and filter by text classification. And then


the investing subreddit is basically full of financial advice. So we really want to if possible


use a more financially savvy BERT model which we can find with FinBERT. And we have two options


FinBERT here. I'm going to go the process AI FinBERT model. And all we actually need is this


text here. We go back to our code. We'll just enter it here. So process slash and we all just


want this on the same line like that. And we're also going to be using the same model for our


BERT for sequence classification. So BERT sequence classification.


And we do the from pre-trained process AI FinBERT again. And that's all we need to do to actually


initialize our model and tokenizer. And now we're ready to actually tokenize that input text.


So when it comes to tokenizing input text for those of you that have worked with Transformers


before it typically looks something like this. So we write tokens or whichever variable name


you'd like to use. We use tokenizer encode plus. We pass our text here. We add special tokens. So


this is the CLS, separator tokens, padding tokens. So anything


from this list here. So all these tokens are used specifically within BERT for different purposes.


So we have the padding token which we use when a sequence is too short. So BERT always requires


that we have 500 tokens within our inputs. If we are feeding in 100 tokens then we add 412 padding


tokens to fill that empty space. Unknown is just when a word is unknown to BERT. And then we have


the CLS token here and this appears at the start of every sequence. And the token ID for this is


101. So we'll be using this later so it's important to remember that number. And then we also have the


septoken which indicates a separator which indicates the point between our input text and


the padding. Or if there is no padding it would just indicate the end of the text. And they're the


only ones that we really need to be concerned about. So typically we add those special tokens


in there because BERT does need them. We specify a max length which is the 512 tokens that BERT


would expect. And then we say anything beyond that we want to truncate. And anything below that we


want to pad up to the max length. And this is typically what our tokens will look like. So now


we have it's a dictionary. We have input IDs. We have this token type IDs which we don't need to


worry about. And we have the attention mask. And that's typically what we would do. But in this


case we are doing things slightly different because one we don't want to add those special


tokens immediately because if we add this special token we have a CLS or sort of sentence token. And


then we also have a separator token at the end and start of our tensor. And we don't want that


because we're going to be splitting our tensor up into three smaller tensors. So we actually don't


want to add those yet. We're going to add those manually later. And then we also have this max


length truncation and padding. Obviously we actually don't want to be using any of these because if we


truncate our 1300 token text into just 512 then that's just what we would normally do. We're not


actually considering the whole text. We're just considering the first 512 tokens. So clearly we


also don't want any of those variables in there. In our case we actually do something slightly


different. We still use the encode plus method. So tokenizer encode plus.


We also include text. This time we want to specify that we don't want to add those special tokens.


So that's false. And that's actually it. We don't want to include any of those other arguments in


there. The only extra parameter that we do want to add, which we want to add whenever we're working


with PyTorch, is we want to add return tensors equals pt. And this just tells the tokenizer to


return PyTorch tensors. Whereas here what we had are actually just simple Python lists.


And if we're using TensorFlow we switch this over to tf. In our case using PyTorch.


And let's just see what that gives us. Okay so here we get a warning about the sequence length


and that's fine because we're going to deal with that later. So we're going to use the


that later. And then in here we can see okay now we have PyTorch tensors rather than the


lists that we had before. Which is great that's what we want. Now we have that we actually want


to split each of our tensors or the input ids and the attention mass tensors. We don't need to do


anything with the token type ids. We can get rid of those. We want to split those into chunks of


length 510. So the reason we're using 510 rather than 512 is because at the moment we don't have


our CLS and separator tokens in there. So once we do add those that will push the 510 up to 512.


So to split those into those chunks it's actually incredibly easy. So we'll just write input id


chunks and we need to access our tokens dictionary. So tokens and then we want to


access the input ids here. And you'll see here that this is actually a tensor. It's almost like


a list within a list. So to access that we want to access a zero index of that. And then we're


just going to split which is a PyTorch method by 510. And that is literally all we need to do


to split our tensor into batches. And we repeat this again but for the mask.


And just change this to attention mask. Again we don't need token type ids so we can just


ignore that. And then let's just print out the length of each one of our tensors here.


So for tensor and input id chunks. Just print the length of it so we can check that we are actually


doing this correctly. Okay so we can see we have 510, 510 and the last one is shorter of course


like we explained before at 325. So that's pretty ideal. That's what we want. And now we can move


on to adding in our CLS and separate tokens. I'll just show you how this is going to work.


So I'm going to use a smaller tensor quickly just as an example.


So we just need to also import torch. So we do that here.


Okay so we have this tensor and to add a value on either side of that we can use the torch cap


method which is for concatenating multiple tensors. In this case we would use torch cat.


And then we just pass a list of all the tensors that we would like to include here.


Now we don't have a tensor for our tokens so we just create it within this list.


And that's very easy we just use torch tensor.


And then if you remember before the CLS token is the equivalent of 101 when it's converted to


the token ID. So that's going to come at the start of our tensor and then the middle we have our


actual tensor and at the end we want to append our 102 tensor which is the separator token.


Okay and we'll just print that out we can see okay we've got 101 and then we have our sequence


and 102 at the end. Then after we add our CLS and separator tokens we will use the same method


for our padding as well. But we want to write this logic within a for loop which will iterate


through each chunk and process each of our tokens. So we'll just use our for loop here.


So we'll use our for loop here. And then we'll use our for loop here. And then after we add


our CLS and separator tokens we will use the same method for our padding as well.


So we want to write this logic within a for loop which will iterate through each chunk and process


each one individually. So first I'm going to create a variable to define the chunk size which


will iterate through each one of those. So we'll just go through a range of the length of the


number of chunks that we have. This will go 0, 1 and 2. And then we can access each chunk using


the i index here. So first we want to add the CLS and separator tokens just like we have above.


That we go input id chunks. We get the current index and then we just do torch


cat which is just concatenate. And then we pass a list just like we did before which is going to be


torch tensor. And then in the middle we have a. We're going to replace that with this.


OK. And then we want to do the same for our attention mask. But of course in our attention


mask if we look up here it's just full of ones. And the only two values that we can actually have


in our attention mask is either 1 or 0. And the reason for this is whenever we have a real token


that Bert needs to pay attention to we have a 1 in this attention mask. Whereas if you don't have


1 in this attention mask. Whereas if you have a padding token that will correspond to a 0 in this


attention mask. And the reason for this is just so Bert doesn't process attention for the padding


tokens within our inputs. So it's essentially like telling Bert to just ignore the padding.


So in our case here both of these are not padding tokens. So both of them should be 1.


OK. And then that gets us our sequences with the CLS separator and added attention


mask tokens in there. So now we need to do the padding. And realistically with padding we're


actually going to do that for the final tensor. So what we will do to make sure that we don't


try and pad the other tensors is just check the length of each one. First we'll calculate


the required padding length. Which is just going to be equal to the trunk size minus the input ID


trunk. Then we want the index shape 0. So this is like taking the length of the tensor. OK. And


for chunks 1 and 2 this would just be equal to 0. Whereas for the final chunk it will not it will


be something like 150 or 200. So what we want to do is say if the pad length is greater than 0


then this is where we add our padding tokens. So first we do the input ID trunk.


And again we're just going to use the torch concatenate method.


Concatenate method. This time we have our input ID


trunk at the start. I think it's a chunks not chunk.


And also here this should be mask chunks. So let's just fix that quickly.


And here we first have this and then the parts following this need to be our padding tokens.


And to create those we are going to do the torch tensor again. And then in here we're going to just


add one zero in a list. But then we're going to multiply that by the pad length. So if the


pad length is 100 this will give us a tensor that has 100 zeros inside it. Which is exactly what we


want. And then we'll copy and paste this and do the exact same thing for our masking tensor as well.


Okay so now let's just print out the length of each one of those tensors.


So for chunk input ID chunks print the length of that trunk. And then we'll also just print out


the final chunk as well so we can see everything is in the right place.


And here so just copy so this here needs to have an s on the end.


And up here so when we first build these so if we just print out one of them.


You see that the input ID chunks is actually a tuple containing three of our tensors.


So what we actually want to do, just close that, is before we start this whole process


we just want to convert them into lists so that we can actually change the values inside. Because


otherwise we are trying to change the values of a tuple which we obviously can't because tuples are


immutable in iphone. Which means you can't change the values inside them. So we just convert those to lists.


And then we also need to add an s on here. And there we go we finally got there. So now we can see okay here we have 514.


So let me just rerun this bit here and then rerun this. Okay so it's because I was running it twice.


It was adding these twice. So now we have 512. And then we can see we have our tensor.


So this is just printing out the input ID chunks. You see here we have all these values.


And this is just the final one. So you can see at the bottom we have this padding. If we go up here


we have our starter sequence token 101. And down here we have the end of sequence separator.


So now what we want to do is stack our input IDs and attention mass tensors together.


So we'll create input IDs. We'll use torch stack for that. And that's going to be input ID chunks.


And then we also have the attention mask that we need to create. So we do the same thing there.


And that is the mask chunks.


And then the format that BERT expects us to be feeding in this data is a dictionary.


Where we have key value pairs. So we have a key input IDs which will lead to our input IDs tensor


here. And then another one called attention mask that will have the attention mask as its value.


So we'll just define that here.


And this is just the format that BERT expects. So the input IDs and then we have the input IDs there.


And then we also have the attention mask. We have the attention mask in there.


Now as well as that BERT expects these tensors to be in a particular format. So


the input IDs expects it to be in a long format. So we just add long on to the end there. And then


for the attention mask we expect integers. So we just add int on to the end of there.


And then we just print out input bit so we can see what we are putting in there.


Okay great. So that is exactly the format that we need. Now we can also use the


BERT expect to get our outputs. So we pass these into our model as keyword arguments.


So we just add these two asterisk symbols. That means it's a keyword argument.


And then in there we pass our input dict. And this allows the function to read these keywords.


Take them as variables and assign these tensors to them.


So there we have our outputs. You can see here that we have these logits.


These are our activations from the final layer of the BERT model.


And you see okay we have these values. What we want in the end is a set of probabilities.


And of course this is not a set of probabilities because probabilities we would expect to be


between the variables. So we can just pass these into our model.


Probabilities we would expect to be between the values of 0 and 1.


Here we have negatives. We have values that are over 1.


That's not really what we would expect. So to convert these into probabilities


all we need to do is apply a softmax function to them.


Now softmax is essentially sigmoid but applied across a set of categorical or output classes.


And to implement that we just do torch and then functional.


And then we just add softmax onto the end there. And we need to access the output logits


which is in index 0 of the outputs variable. So that is just accessing this tensor here.


And then we access dimension minus 1. So the dimension negative 1 is just accessing


the final dimension of our tensor. So in this case we have a 3D tensor.


So this is like accessing the second dimension or dimension number 2.


Because when we have 3D tensor we have dimension 0, 1 and 2.


Minus 1 of 0 is just the dimension 2 if that makes sense.


So imagine we have 0, 1 and 2 here. If we go here and we take negative 1.


We come around here to the back of the list. And that is accessing the second dimension.


OK. So that is going to take a softmax function across each one of these outputs.


And then we can print that out. So now we have our probabilities.


So the outputs of the Finbert model. These ones here in the first column are all positive.


So this is the prediction of the chunks having a positive sentiment.


These are all negative. So the prediction of the chunk having


a negative sentiment. And these are all neutral.


So if it has a neutral sentiment. So we see here the first and second chunks


are both predicted to have a negative sentiment. Particularly the first one.


And the final one is predicted to have a positive sentiment.


Now if we want to get the overall prediction. All we do is take the mean.


So the probabilities. And we just want to take the mean.


And we take that in the 0th dimension. Which would just go from here down.


Take the mean of those three. Take the mean of these three.


And take the mean of these three as well. Print it out.


You see here. OK.


The negative sentiment is definitely winning here. But only just. It's pretty close to the positive.


So it's a reasonably difficult one to understand. And this is because over here we have mostly


negative. Kind of negative. And mostly positive.


So it's a bit of a difficult one. But the negative sentiment does win out in the end.


Now if you'd like to get the specific category that won.


We'll just take the argmax of the mean. And that will give us its tensor.


If we want to actually get the value out of that tensor.


We can just add item on to the end there. And that is it.


We have taken the average sentiment of a pretty long piece of text.


And of course we can just use this code. And iterate through it for multiple long pieces of text.


And it doesn't really matter how long those pieces of text are.


This will still work. So I hope this has been an interesting and useful video for you.


I've definitely enjoyed working through this and figuring it all out.


How to Build TensorFlow Pipelines with tf.data.Dataset

Hi, we're going to go through the TensorFlow datasets.


Essentially these are a more efficient


built-in way to build our import pipelines.


So we can see the documentation here. If you'd like


to go through it you can do. I'll leave a link in the


description. But we are just going to go and dive


right into it.


So to use the pipeline dataset object


we need to actually import TensorFlow of course.


And we're


also going to be using pandas and numpy for a few examples here.


So we'll go ahead and import those as well.


Okay, so there's a few different ways that we can


read data into our data pipelines, datasets.


So the first of those is from in memory, which is probably the way


that most of you, if you have seen this before, will have seen it.


We'll go ahead and put that together quickly.


So the first one is a list. So we can


take a couple of python lists, put them together, and build


our dataset using that. So we'll just


put some together really quickly here.


Okay, so we just have input and outputs. They're both


lists. And then to create our


dataset object from these two, we just


type tf.data.dataset


with a capital D.


And we are taking these


as tensor slices. Like this.


And one thing


that we're doing here, we're putting both of these into a tuple.


Because this only accepts one


input parameter. And that input parameter is basically


all of the data that we are going to be feeding into our model later on.


The default format for


the dataset object when it's feeding into a model is


simply one input tensor and


one output tensor. Or target or label tensor.


Whatever you'd like to call it.


Here, once it loads, we will have built


our first dataset.


Took a lot longer than it should have done.


And for item and dataset. So we're just going to see


what it looks like. And we'll see it's like


a list of tensor arrays.


So we can see the tuple format that we created


here. So this is the first item we have.


And the first tensor object is a NumPy array


or NumPy integer, which is zero.


Which matches up to this. And then next to that we have the output value


which is one here. And then it's the same


for the following three rows in there.


So we can also do the same with NumPy arrays.


It's literally pretty much exactly the same


format. Like this.


And this will produce the exact same thing.


And then if we want to use


a dataframe, which I assume a lot of you will do.


This time, before we were passing inputs and outputs,


this time we just do the dataframe.


And we will see


we'll create a slightly different format here.


And with this we would reformat or restructure


the dataset here.


Before feeding into our model for it to read everything


correctly. But for now we're just going to leave it like that and then we'll go over


the mapping and everything pretty soon.


So the other option we have for reading


the data into our dataset is actually reading it directly


from file. So from file the benefits


we get from doing that is that we are reading in data from an out of memory


source. And


because we're reading from an out of memory source, TensorFlow will


read data batch by batch rather than


pulling in the entire data source or the entire


dataset all at once. So if we


have like a big dataset, then this is pretty useful because


in a lot of cases we're working with a big dataset and we can't actually bring


everything into our memory all at once.


So this allows us to get around that and it does it in an efficient way.


And it's just super easy as well.


I also put this together.


Zoom out a little bit. And this is my


attempt at demonstrating the difference


or demonstrating what the


from file version of this does. So this is our


dataset here and we batch it into three batches here. Obviously you'd have way more than


this. And at any one time we


feed in a single batch, apply our dataset transformations


feed it into the model for training and then once we're done with that


we go onto the next batch and then we feed it into dataset transformations


and go and train the model with it.


So let's do that quickly.


So I've got this train.tsv here.


And that file is actually from the


sentiment analysis on the movie reviews from


Kaggle. So you can download it here. You can see the link. I'll put it in the


description. And


we're going to read from this. Read from it directly.


So it's slightly different.


We do tf.data.


experimental. And then we make


csv dataset.


So I know that we are actually using a tab


separated values here rather than comma separated values.


So all we're doing here is we're going to change the field


delimiter to a tab character instead.


So it's train.tsv.


And then


also in here we actually define our batch size.


So we're just going to do something really small for now but obviously


when you are using this


for your actual models you would probably be doing something like a batch size of 64


or 128 or whatever it is you're using.


But we're just going to go for 8 now so we can like really


easily visualize everything. And then next, oh I don't know why it's


doing that. It's fine. Next we do


field delimiter. So this is where we tell it's actually


a tab delimited file rather than comma.


Let me try and


sort that out.


It's really annoying. And then


we also need to


we need to set the


label for our dataset.


Which if we


look here our label


is this sentiment field.


So we go sentiment. And then


actually another really useful argument here


is the select columns. So


with this we just pass a list of the columns that we want to keep. And then


it will drop all the other columns.


So for now I mean


it depends on what you're doing obviously here. But we're just going to keep the


id. And then we're also


going to keep the input and target data.


So the id is


phrase id. And then


the input data is phrase


yep. And then sentiment for the


label. Let's execute that.


And then let's just have a


quick look at what we have. So we


use this take to just take the first


batch within our dataset.


If we say take 20 then it will take the first 20 data. The first 20


batches and nothing else. But we just want


to see the first one so we can see the actual format within the


within the dataset.


So you can see here we have the phrase id. And this is


why I wanted to keep the phrase id in. We don't actually need it for


training model. But I


wanted to show you that it actually shuffles the data. So you can see


in here phrase id is 1, 2, 3, 4, 5.


It's in order. But then when we


read data in with this it actually automatically


shuffles everything which is a pretty cool feature.


So yeah it's pretty useful. And then here we have phrases


which is our would be our


input data. And then here we have the sentiment


ratings which would be our target data.


And so that's everything for the


reading into our dataset.


And we'll move on to


performing a few operations on it.


So I'm going to go


back and just assume that we're not reading


it from here. Actually let's use this


but we're going to load it into our memory


first.


Like this.


Let's do this.


So I'm going to do this because I want to show you the


shuffle and batch methods. And obviously if it's already shuffled and batch


there's no point in showing you. But this is


useful to know if you're reading things from in memory.


Obviously if you are reading things from your disk then there's


no point in doing this part.


And then we're


TSV so we need to keep the separator as


a tab. Okay let's just make sure


I've read it in correctly.


Okay cool. So


what I want to show you here. We actually need to


read it into our dataset. So we're going to use


the same here.


And then let's do the


for item in


print item


Okay so this is because we have these phrases in here.


So I mean we don't really need them.


So let's just go sentence ID to


make things a bit easier for now.


So I mean


if you're using strings obviously for machine learning you're going to


tokenize it. So you would


do that first.


But we're not going to go all the way through to actually training the model.


We're just going to have a look at the pipelining.


Okay cool.


So our first row


is 1 1 1, 1 1 1. That's what we expect.


First thing is to


actually do the shuffling and the batch like I said before.


So we're going to do more


no we stick with a batch of eight. Just make things a bit more readable.


So what we do is it's like


super easy. So we shuffle the data set


and we just add in a large number here to


make sure it shuffles everything like as far away


from it's neighboring samples as possible.


And the sort of standard number here is actually


10,000. I don't know exactly why


but almost every time I've seen shuffling


I've seen people use 10k so I'm going to stick with it.


And then I'm so used to putting like 64 or 128.


We'll just put a batch of eight here.


So


if we take, so now we've batched it so we should actually see


more than one because it's taking the first


like the one of the highest level


record or batch within the data set.


So now we should see quite a few and we can see okay cool it's


definitely mixed up the phrase IDs


because these were one two three four before and now they're all


mixed up. So that's cool we shuffled and batched it


like incredibly easily. So


you know that's one of the benefits of doing it and as well


writing this code is one it's incredibly easy and simple


to remember like it's not hard to remember that. It's very


obvious when you're reading it like what is happening. Your data set shuffle


batch into eight. Super easy I mean maybe some people might get a bit


confused by this number here. But otherwise it's super easy


to read. And it's really quick and efficient so


it's pretty good.


Next thing I want to show you is the


map method. So for any


more complex data transformations


this is probably what you'd use.


It's really really useful.


So what can we do? We can maybe add


or multiply everything in the labels by two. I mean obviously we wouldn't


do this in reality but it's just an example.


And we'll also reformat the data.


So we're going to build it as if we have two


input fields. So for example


when you're working with transformers or a lot


of the time you have input ID field or


layer and you also have an attention mask field or layer


don't worry if that doesn't really make sense. But essentially


we just have two input layers and fields. So we'll format


this to be formatted in the correct way to have


two inputs and then one output and we'll also change the number


of the output just so we can see how it works.


So generally the best way of doing this is to


create a function. So I'm just going to call it map


func and pass x. So this is just going to pass every single


record within our data set.


So one thing I actually just realised is that we


should batch this afterwards.


Because otherwise we have to consider the batching. So let's


move these after.


And let's write this. So we're going to return


the data set.


So when we are working with multiple inputs


or outputs even, the best way to


let TensorFlow know where each input is


supposed to be going is to give the input layers or output layers a name


when you're defining the model


and building it and match that to the names that we


give to this dictionary here. So


with the transformer example, I think most people


just do input ids.


And then I'm just going to make this up so our input id for this is going to be


this value here.


And we're going to put it into a list.


Because typically you'd have


an array or a list of numbers coming in here.


And we're just going to take the first


value of x.


And then the mask


we're going to write it like this.


Number one.


And then on the outside of this dictionary


we only have one label


or one output or one target, however you want to call it.


And we're just going to perform a really basic operation on it just


to show that we can. Just multiply by two and


nothing special.


So to


apply this mapping function, all we


need to do, again it's incredibly easy,


is data set.map.


And then we map the map func


like this. And then we did batch it before so


let's just re-run this bit of the code so that we have it all


unbatched again.


And this. Ok, so let's have a look at what we have now.


So you can see here we have this format.


Let's do that again. Let's see what we have.


Ok, it's kind of hard to read but


inside a tuple


so this is the


index one of the tuple and this is index


zero. So inside


index zero we have this dictionary that


we defined, which has input ids


and it also has the mask.


And that is what


this tuple format here with the input and


the output here


is what TensorFlow will be expecting when we


fit to the model. And then if it sees


a dictionary in either the input or the output


it will read the key values


of the dictionary and


the values that match to those keys will be provided to


a corresponding key. So essentially


you would have to have a layer called input ids and it would pass


this to that layer and then it would also pass


this to the layer of mask.


And then we would also have the outputs being passed


to our output layer. We wouldn't necessarily


need to mark this one out though.


Ok, then we want to batch it


like we did before and then we can just view what we have here.


So we have the dictionary and then we have everything else as well.


Ok, so that's pretty good. That's what we want.


Ok, so we need to define


a, just really quickly define a model.


So it's going to have a inputs


input ids, sorry, input ids


layer.


Ok,


and


so I'm not actually going to define all of this. I'm just going to show you how it would work.


So you define your shape.


You define it here and then you would also have your name.


And then it's this name that would have to match up to the dictionary


that we have fed in previously.


So we go like, input ids


or was it inputs ids? Ok, so inputs ids


Alright, they would have to match.


And then we would have a mask as well.


Because we have two inputs, remember? And then this one


would be called mask. And obviously you would call this mask or anything


else you want. We've got like input one, input two.


It doesn't matter.


Later on, when we actually fit the model,


we would do it like this.


Obviously we would have an output here as well.


I'm not defining the rest of it. But we would have two inputs


and then the output. We would have the two inputs


something in the middle, the output. And then


all we would have to do with that model architecture is


fit in data set like this.


And then you'd have however many epochs you're training for.


And then that's everything for that.


So actually there's one other thing that I wanted


to mention as well.


Obviously a lot of the time you're going to want to


split your data for training into like a


a training


and a validation set. So to do that


it's actually super easy. So before


we mention the take method. So we


use data set.take


and we can also use data set.skip


and these are like equal and opposite.


So if we do data set.take


10, it would take the first 10 batches of the data set


and nothing else. Then if we do data set.skip


10, it will skip the first 10 batches of the data set and nothing else.


So if we


this is not the most efficient way of doing it.


But let's just do it like this for now.


So if we just take the length


of the data set.


So I say this isn't efficient because it's


to take the list of the data set we're loading everything in.


Or we're generating it. Because the data set is generated


and we're putting everything into memory


as a list and then taking the length of it. So it's better


just to know how many batches you're building


from the start. If it's a big data set. I mean for this it's fine


because we don't have a lot of data. But normally it would be better


not to do that. So let's see.


I mean you can see even with this data set it's pretty small


it's still taking quite a long time. And then say if we want to take a


70% split. So 70-30, 70% for the


training data, 30% for the validation and probably test data as well.


But you just split that after.


We would take, so


training size


would be


0.7. And remember this


is taking the batches. So it would actually be


the length


divided by


the batch size which is 8.


Okay and then we're going to have to round this


to the nearest batch, or the nearest integer.


Because we can't take 10.2 or something like that. So


we just round it here.


And then let's just see what we have for train size.


Okay so we have 1707


batches for the training data.


Okay so we want to take that number of batches.


So


train size and we create another data set


which is the train data set.


And then for the validation data set


we just skip those


1707 data sets, batches.


Like that.


Super simple.


So let's take the length of those


so we can see again, I know it's not efficient but


it's the easiest way to do it quickly.


Okay so yeah we get 1707


and then our 30% value is when it finally


loads.


When it finally loads


Ah okay so


we already have


the batch size here, I don't know why, that was really stupid.


So we've already considered the batches in here. So we didn't


need to consider it here as well. That's why it was


taking me so long. Okay. So


it would actually be, it should actually be that way.


So let's do that. So yeah we have a training size of


13.6k and then the remaining


batches will feed into our validation data.


Which will be around


600 values, sorry just under


6000 values. But that's everything


I think that I wanted to go through.


So we've covered all of the essentials


of the TensorFlow dataset object.


How we can load in memory data and or read


into datasets from file.


How to batch and shuffle the ones that we read from


in memory sources. How to transform the


datasets with map. How we can feed them into models.


One thing to note that if we just have an input


and output like you probably will for most models


you don't need to do anything you just have it in the


tuple format and you have inputs and outputs.


And then you would just


do model.fit dataset like that.


You don't have the named layers or anything you just feed it straight in.


Then after that


we went through the


oh the splits we just did here.


So yeah that's everything. I hope it's been a


useful video and I hope you enjoyed the


Long Form Question Answering (LFQA) in Haystack

Today we are going to talk about a subdomain of question answering called


long-form question answering. Now before we get into the specifics let's just


talk very quickly about question answering as a subdomain of NLP. Question


answering has I think exploded as a subdomain of NLP in the past few years


mainly because I think question answering is an incredibly widely


applicable use case for NLP but it wasn't possible to do question answering


or not anything good with question answering until we had transformer


models like BERT. So that means that as soon as we got something like BERT


the question answering became viable and with the huge number of use cases for


question answering it obviously kind of took off. Now question answering is quite


complicated but at its core it's basically just the retrieval of


information in a more human like way and when we consider this I think it makes


it really clear how broadly applicable question answering is because almost


every organization in the world if not all are going to need to retrieve


information right and for a lot of companies and particularly larger


organizations I think the act of information retrieval is actually a big


component of their day-to-day operations. Now at the moment most organizations do


information retrieval across a suite of tools so they will have people using


some sort of internal search tools which are typically keyword based which is


generally not always that helpful. Sometimes it's useful but a lot of the


time it's not great. Then another key form of information retrieval in most


organizations is literally person-to-person so you go and ask someone


who you think will probably know where some information is like a document or


so on and obviously you know this sort of patchwork of information retrieval to


an extent sure it works but it's inefficient. Now if we consider that many


organizations contain thousands of employees each of those employees


producing pages upon pages of unstructured data e.g. pages of


documents and texts that are meant for human consumption in most cases all of


that information is just being lost in some sort of void. Rather than that


information being lost in a void that we're never going to see again it


becomes useless to the organization or the company we can instead place it in a


database that a question answering agent has access to and when we ask a question


to that Q&A agent which we ask in a human-like way it will go and retrieve


the relevant information for us instantly or not instantly but pretty


close. The majority of data in the world is unstructured and there's a few


different sources for this but I think places like Forbes estimate that number


to be around 90% of the world's data so in your organization you probably have a


number similar to this so 90% of your data is unstructured that means it's


meant for human consumption not machines and it means it's liable to get lost in


that void where we're just never going to see that information ever again. Now


that's massively inefficient. Question answering is an opportunity to not lose


that and actually benefit from that information. Now in question answering


there are two main approaches in both cases of question answering we saw those


documents in or not usually we saw those documents in a document store or vector


database so these documents are what we would call sentences or paragraphs


extracted from your for example PDFs or emails or whatever unstructured data you


have out there and we retrieve data from that and then the next step is where we


have the two different forms of question answering with that relevant information


that we have from our sort of document store based on a query that we've


passed through we either generate an answer or we extract an answer so


obviously when we're generating an answer we look at the whole all the


context that we've retrieved and we use an NLP model to generate some sort of


human answer to our query based on that information otherwise we use an


extractive model which is literally going to take a snippet of information


from the data that we have retrieved. So there's a few components that I just


described there there was that document store at the start when we're using a


document store which we will in most cases I'd imagine we call that open book


question answering now the reason it's called open book is it is like students


in an exam okay we have a typical exam you don't have any outside materials to


refer to it's just you have to rely on what is in your brain that's very


similar to using for example a generator model that given a question it doesn't


refer to any document store it just refers to what is within its own memory


or its own model memory and that model memory has been built during model


training so that would be referred to as closed book generative or abstracted Q&A


on the other hand we have a document store so that document store is


like we are in our exam as students and we have a open book that we can refer to


for information so we're not just relying on what is in our head we're


looking at the information in this book and we still need to rely on the


knowledge in our head in order to apply what is in that book to the questions


were given in the exam it's exactly the same for open book abstractive question


answering in that you have the generator model but we're not just relying on a


generator model to answer our questions we are also relying on a document store


which is our book and what is called a retrieval model and this retrieval model


is going to take our question it will encode it into a vector embedding takes


it to that document store which is actually just a vector database in in


our scenario of what we're doing and in a vector database what you have is lots


of other vector embeddings which are essentially numerical representations of


the documents that you stored in it before so remember documents are those


chunks of paragraph or sentences from different sources that vector database


has loads of these these local context vectors and we pass our query vector


into that document store or vector database and we retrieve the most


similar context vectors from there and pass them back to our retrieval pipeline


then that is passed to our generator model our generator model is you want


to see the query followed by the set of retrieved relevant hopefully context and


it uses all of that to generate an answer okay so we can see with this open


book format we are passing a lot more information into the generator which


allows the generator to answer more specific questions now long form


question-answering which is what we are going to go through is one form of this


abstractive question-answering the only difference with only the one thing that


makes a long form question-answering long form question-answering is that the


generator model has been trained to produce a multi sentence output so


rather than just outputting a maybe a answer of three or four words or one


sentence it is going to try and output a full paragraph answer to you okay so


that's long form question-answering or LFQA so we are going to implement LFQA


in HaySac and HaySac is a very popular NLP library mainly for question


answering now to install HaySac and the other libraries that we need today we do


this so we have pip install we need the pinecone client farm HaySac specify


pinecone in their data sets and pandas actually I think you can ignore pandas


let's remove that so just these three here with farm HaySac we are going to


be using something called a pinecone document store so for that you need


either version 1.3 or above now to initialize that pinecone document store


so remember the document store is that thing that you saw on the right before


so they were storing all of our context vectors we will do this so we first need


an API key from a pinecone so there's a link here I'll just open it and show you


quickly and that will bring you to this page here now you can sign up for free


you don't have to you don't need to pay for anything and we don't need to pay


for anything to do what we're doing here either it's all completely free so you


just sign up and once you've signed up you will see it should just be one


project on your home page so for me it is the default project James's default


project so you can go into that and then on the left over here we have API keys


so we open that and we get our default API key we can just copy it so come over


here and we use that to authenticate our pinecone document store back in our code


so I would paste that here and with that we just run this so we are initializing


our document store we are calling our index so remember document store is


actually vector database in this case and inside that vector database we have


what's called an index the index is basically the list of all the context


vectors that we have we call that index haystack LFQA now you can call it


whatever you want and just when you are wanting to load this document store


again you need to specify the correct index that's all that's all that's the


only difference it makes similarity we're using cosine similarity and we're using


embedding dimensions 768 now it's important to align this to whatever the


similarity metric and embedding dimension of your retriever model is in


our case cosine and since it's a these are pretty typical retriever model


metrics and dimensionalities now we can go down we can check our metric type we


can also see the number of documents and the embeddings that we have in there now


we don't have any at the moment because we haven't pushed anything to our


document store we don't have any data so we need to get some data for that we are


going to use hugging face datasets so over here we're going to use this data


set here which is a set of snippets from Wikipedia there are a lot of them in


full this data set is 9 gigabytes now to avoid downloading this full data set


what we do is set streaming equal to true and what this will do is allow us to


iteratively load one record at a time from this data set and we can check what


we have inside that data set by running this so next create a iterable from our


data set and we see this so the main things to take note of here are section


title and passage text passage check text is going to create our context or


that sort of document and there are a couple of things so history is going to


be what we are going to filter for in our data set this is a very big data set


and I don't want to process all of it so I'm restricting our scope to just


history and we're going to only return a certain number of records from that


section that's important to us purely for that filtering out of other sections


or titles in there section titles and we will include article title as metadata


in our documents although it's not really important because we're not


actually going to use it it's just so you can see how you would include


metadata in there in case you did want to use it so here what we're doing is


filtering only for documents that have the section title history okay and we


just get this iterable object because we're streaming so it just knows now


when we're streaming one by one when it's pulling an object it's going to


check if that object section title starts with history if it does it will


it will pull it if not it will move on to the next one okay so we're just going


to pull those with history now what we need to do is process those and add them


to our document store now what I've done here is said okay we are only going to


pull 50,000 of those and no more at that point we cut off and it's actually it


cuts off just before 50,000 and we're going to do is we're going to add in a


single batch so we're going to loop through all of all going to pull all of


these records we're going to collect 10,000 of them and then we're going to


add them to our document store and this is a haystack document object so we have


a content the content is the document text that big paragraph you saw before


meta is any metadata that we'd like to add in there now with the pinecone


document so we can use metadata filtering although I won't show you how


to do that here but that can be really useful if it's something you're


interested in so that's how you'd add metadata to your document as well and


all I'm doing is create adding that doc to a docks list and we increase the


counter and once a counter hits the batch size which is the 10,000 we write


those documents to our document store now you will remember I said document


store is a vector database and inside the vector database we have vectors at


the moment when we write those documents we're not actually creating those


vectors because we haven't specified the retriever model yet we're going to do


that later so at the moment what we're doing is kind of adding the documents as


just plain text to almost be ready to be processed into vectors to put into that


vector database so it's almost like they're in limbo waiting to be added to


our to our database so we add all those that can take a it can take a little bit


of time not not too long though and then once we hit or get close to 50,000 we


break so we stop the loop and then we can see if we get document account we


see that we have the almost 50,000 documents in there but then when we look


at the embedding count zero and that's because we you know they're waiting to


be added into the vector database those the text documents so they are existing


documents they just don't exist as embeddings yet okay so what we now need


to do is convert those documents into vector embeddings now to do that we need


a retriever model now at this point it's probably best to check if you have a


GPU that is available like a CUDA enabled GPU if you don't this step will


take longer unfortunately but if you do that's great because this will be


pretty quick in most cases depending on your GPU of course so we initialize our


retriever model so we're using the embedding retriever and this allows us


to use what are called sentence transformer models from the sentence


transformers library now I'm using this model here and we can find all the


sentence transformer models over on the Huginface model hub so let's have a


quick look at that so we are here Huginface dot co slash models and I can


paste that model name maybe I just do flight sentence embeddings now flight


sentence embeddings are a set of models that were trained on a lot of data using


the flights library but there are a lot of other sentence transform models see


the one we're using here so for example if we go sentence transformers you see


all of the default models used by the sentence transformers library so we are


using this MP net model we also specify that we're using sentence transformers


a model format and when we initialize our retriever we also need to add the


document store that will be retrieving documents from so we've already


initialized our document source so we just add that in there and at this point


it's time for us to update those embeddings so when we say update


embeddings what this is going to do is look at any of all the documents that


are ready and with your document store and it's going to use the retriever


model that you passed here and embed them into vector representations of


those contents and then it's going to store those in your pinecone vector


database that will be processed and at this point we could run this get


embedding count again and we would get this 49995 value now another way that


you can also see this number is if we go back to our pinecone dashboard we can


head over to our index so hey stack LF QA we click on that scroll down and we


can click on index info and then we can see the total number of vectors which is


the same okay so that will that number will be reflected in your vector database


once you have updated the embeddings using a retriever model okay and at that


point we can just test the first part of our LF QA pipeline which is just a


document sort and a retriever so we initialize this document search pipeline


with our retriever model and we can ask a question when's the first electric


power system built okay and all this is going to do is retrieve the relevant


context it's not going to generate an answer yet it's just going to retrieve


what things is the relevant context so we have here electrical power system in


1881 two electricians built the world's first power system in gold and Ming in


England which is pretty good so that's pretty cool and what we now need to do


is we have our document store or vector database and then we have our retriever


model now we need to initialize our generator model to actually generate


those answers so come down here we are going to be using a sequence to sequence


generator and we are going to be using this model here so this again you can


find this on the hugging face model hub and there are different generator models


you can use here but you do want to find one that has been trained for long form


question answering so for example we have the bar LF QA that you can find


here or you have the bot explain like I'm five model that we can find here now


I think the bar left you a model seems to perform better so we have gone with


that also it's been trained with a newer data set and yeah we just initialize it


like that now when we say sequence to sequence that's because it is taking in


a sequence of characters or some some input you're going to output a sequence


of characters eg the output the answer and if you are curious that the input


will look something like what you see here okay so we have the question then


we have the users query it's followed by context and then we have this sort of P


token here and that P token indicates in the model to start of a new context that


has been retrieved from our documents on so in this case we've retrieved three


contexts and all of that is being passed to the generator model where it will


then generate an answer based on all of that okay so yeah that we just


initialize that the generator model and then we initialize a generative Q&A


pipeline we pass in the generator and retriever model we don't need to include


document store here because the document store has already been passed


to their retriever model when we're initializing that so it's almost like


it's embedded within the retriever so we don't need to worry about in there and


then we can begin asking questions on this is where it starts to get I think


more interesting now one thing to make note of here is we have this top k


parameter and that's just saying how many contexts to retrieve in in the


context of our retriever model and then for the generator how many answers to


generate so in this case we're retrieving three contexts and then we


are generating one answer based on the query and those three contexts like we


saw in the example so in this I'm asking what is a war of currents it's good to


be specific to test this and if we have the data within our data set it seems


to be pretty good at pulling that out and producing a relatively accurate


answer so the war of currents was a rivalry between Thomas Edison and


George Westinghouse's companies over which form of transmission DC or AC was


superior okay that's the answer which is pretty cool and we can see what it's


pulled that from so it's pulled it from this content this content and this


content okay so there were three parts that got fed into the into the model and


that's that's good we can see a lot of information there but maybe we can see


a little bit too much information so we can actually use a print answers utility


to minimize more outputting them and here we get just this which is also a


lot easier to read so we just pass our result into print answers and specify


details of minimum the rest of that is the same as what we asked before okay so


it's much more readable now one thing to point out here is that this is actually


a very good answer but maybe there's not that much detail now if we find that


we're not getting much detail in our answers or that the answer is just wrong


what the issue might be is first the retrieved context may not contain any


relevant information for the model to actually view and answer the question


correctly okay so it's not retrieving relevant information from that external


sort of open book document source and the second is if it's not also not


retrieving information from there and it's also not retrieving information


from you remember I mentioned that these models can have a memory if it's not


able to find any relevant information within its memory for your particular


query if both of those conditions are not satisfied so we don't have


information or relevant information coming from the external source and we


don't have relevant information coming from the model memory the generator is


going to output usually something nonsensical okay so in this scenario we


have two options really the generator model we can increase its size so we can


use a larger generator model because larger generator models have more model


parameters which means they have basically more memory that they have


learned during training or we can increase the amount of data that we are


pulling from the document store okay so if we are just returning three documents


or contacts we can increase it to ten because then the generator is being fed


a lot more information and it might be that the correct information that we


need may come in maybe context five or context six and nine and the generator


will see that and be like okay that's the answer I'm going to you know reform


like this into a into my answer okay so we can try that here now we already got


a good answer but we can just see what we get if we increase the retriever so


all the retrieved number of documents so it increased up to ten and we see that


we get this much longer chunk of text now and I think the first half of this


is relatively accurate so we have this in 1891 first power system was installed


in the United States I think that's relatively relatively correct and then


it starts to get a little bit silly after that because you know we pulled


more context from our documents or but with that we have pulled in more


irrelevant information because we're retrieving ten now so there's a good


chance that the last few of those are not relevant so we're feeding love


irrelevant information into our generator model and so it starts to get


confused and then it can start to ramble like we like we see here so that's what


we see happening another thing I want to point out is that the generator has this


memory so a lot of people always think when they hear okay the generator has


memory does that mean I don't need the documents or because we have this memory


can't just fine-tune the model so that it knows everything within my particular


use case in some cases yes you might be able to do that but it generally only


works for more general questions or general knowledge if you start to get


specific it tends to fail with that sort of memory pop because the memory can


only source so much information and in the end what you will probably need is


you want a model with good memory so it can kind of maybe pull out some facts


from there but for anything specific it's probably going to need to refer to


its document store so what we have done here is we've asked the same question


but this time I've replaced the retrieved documents with just nothing


and we can see the result of that straightaway so the answer is I'm not


sure what you mean by war so it's it has no idea what the war of currents is it


doesn't have that information within its memory so without that external document


source it it doesn't know what to say it's just okay I don't even know what


war is but like I said in some cases particularly when you're asking more


general knowledge query it will be able to pull that out from its memory so who


was the first person on the moon it it knows this because it's it's such a


common thing to know it's probably seen it in the training data that the model


has been trained on a million times maybe not a million but a few times at


least so that is the first person man to walk on the moon was Neil Armstrong okay


cool so I think that's pretty much it we can ask a few more questions when was


the first electrical power system built so we asked this and it's up and it will


give us this answer if we want to confirm that this is correct so this is


what I did with this I was a bit confused because Google is telling me


something different you can print out the content using this so we loop through


the results documents and we just print dot content and this okay so two


electricians built first power system at gold amming in England so that


information is actually coming from somewhere it's not just making it up so


that can be really useful another thing just to be aware of with generators is


that they can generate misleading information so you need to be careful


with that so for example in this one I asked where did kovat 19 originate now


this is pretty unfair because the generator probably hasn't seen anything


about kovat 19 and at the same time it doesn't have any kovat 19 information


within its document so because we looked at history not not anything else so it


just says kovat 19 isn't a virus which it is it's a bacterium okay so


straightaway it's pretty pretty wrong so it's just one example of where you need


to just be cautious with this sort of thing because it can just give


completely wrong answers if it doesn't have the relevant information available


to it so with that there's a couple of things you could do to mitigate that you


can one just include the sources of information if you know if you build


some sort of search interface make sure you include those so users can look at


that and see where this information is coming from and two there are self


confidence scores that are given to these answers so you could put threshold


like you say anything be below 0.2 confidence we just don't show or we show


I'm not confident in this answer but it might be this or something along those


lines okay so that's just one drawback let's we'll just go through a few final


questions so what was NASA's most expensive project I would say the space


shuttle project that's correct tell me something interesting about history of


the earth in this case it really it's nothing it's not really history I don't


think but it does give us a an interesting fact about the magnetic


field being weak compared to the rest of the solar system I don't know if that's


true or not it seems like it might not be when it says compared to the rest of


the solar system I'm thinking is it weak compared to Mars I don't think so so


that might not be true another thing to be wary of who created a Nobel Prize and


why so this one is correct and I think quite interesting and how is the Nobel


Prize funded we kind of see it down here so I know the information is in there


hence why I've asked a question and it it tells you that as well with a little


bit more information so that is it for long form question answering with


Hayesak as I said at the start I think question answering is one of the most


widely applicable forms of NLP or use cases of NLP it it can be applied almost


everywhere so it's a really good one to just go away and see you know maybe I


can implement like document search in my organization or I can create some source


internal search engine that helps people in some way and I think in a lot of


organizations it's very possible to do this and add a lot of benefit and reduce


a lot of friction in day-to-day processes of most companies so that's it


FREE 11 Hour NLP Transformers Course (Next 3 Days Only)

I want to introduce you to this course I've been working on.


I've just released it and I wanted to give a lot of you guys who subscribed


and follow me on medium or Twitter. I wanted to give you guys


a chance to get this course for free. So for the next three days it is


completely free. You just use this code. But I just want to talk very quickly


about what it actually covers. Now obviously you can see from the


title it's NLP and it's with Transformers and


Python. Now if we scroll down a little bit


we've come to this course overview video and I'll just quickly you know go


through this because it's quite long and I don't want to take too much


of your time. And we cover a lot of things so first


thing is NLP and Transformers where I give it a quick summary of


NLP in general the history of NLP leading up to Transformers.


Then move into a bit of pre-processing for NLP. Now this is just your basic


stuff. I think the most relevant one here for us


and Transformers is Unicode Normalization and Tokenization


Special Tokens. Then I move through a few lectures on


Attention, how Attention works and describing the logic behind it


before moving into what I always see this as like the


hello world of NLP which is sentiment analysis.


I think it's a great introduction and we introduce Transformers


in this section here.


And it's worth pointing out as well that I use a lot of different frameworks


throughout this course so Flare is the very first one.


We also we use Hockey Face Transformers that's obviously the primary one that


we'll be using throughout the course. TensorFlow, PyTorch,


NLTK, spaCy and many others as well. So there's a lot in there.


Of course using a lot of BERT. Then there's a few so there's two projects


in the course as well. The first of those is Sentiment Analysis.


The second one is Question Answering. Both of them I think are great because


they take you all the way through from the very start of your


project so getting data all the way through to actually building


your model and applying it to your data.


Then we moved on to Named Entity Recognition,


Question Answering, how we measure the performance of our models which


is of course very important.


A full Question Answering stack using some another library called Haste Stack


which I think this is one of the coolest things


in the course in my opinion and it just in NLP in general.


This sort of stuff is incredibly cool. Then like I said there's that second


project, the Q&A project before we move on to


Similarity. Now similarity is super important in NLP and I think


probably one of the most promising areas in the future for


further research and just impact that it could have on industry. I think


this is really a super cool place to be. Then finally


we move on to Fine Tuning. So that's the course in a nutshell


all together there's 11 hours of content so it's


I think comparatively long when you look at other


NLP courses so you'll see this 11, 10, 10, 3 and 6.


As far as I'm aware it's the first course that focuses on transformers on


Udemy. So if you're into NLP


obviously transformers are really the models that you want to be using.


Check out the course in the next three days it's completely free using this


code. So thank you for watching and I hope


Computer Vision Basics + More deeplearning.ai Progress! | Learning Intelligence 16

Woohoo! Check it out guys, just passed the second course on Coursera's Deep Learning


Specialization, Improving Deep Neural Networks, Hyperparameter Tuning, Regulization and Optimization


and look how beautiful that certificate is and that's something I can put on my LinkedIn or


other profiles and whatnot. You're going to have about four or five of these.


Five, there you go, by the end of this course. So welcome to Learning Intelligence 15.


This week or this episode is going to be all about the Deep Learning Specialization


on Coursera. I just passed course two and tomorrow I'm going to be getting into


course three which is on, I think, Improving Machine Learning Models but we'll get into that


tomorrow. I'll show you then. But otherwise, the last project of part two was on TensorFlow and


now I've got a special guest for this episode. So I'm going to go try and find him and then he's


going to explain what exactly TensorFlow is. Whoa! I cannot believe that worked. Shh! Don't tell


Daniel I'm here but I'm from the year 2045. You can consider me future Dan and I just found a


time portal. I know why I'm talking like a back man but I must say the future is great and I'm


going to explain TensorFlow for you. So if you imagine, TensorFlow is a library, it's a framework,


right? So if you wanted to build something out of Lego, you don't build the Lego blocks yourself.


You buy the Lego set and you put it together and you make something amazing. So if you want to


build a really advanced deep learning model like they do at Google, you don't build it from the


ground up when you have TensorFlow available which is your Lego set and you can use the same library


as Google, TensorFlow, to build your deep learning models. It's like creating a Lego feature out of


your brand new Lego set but you're doing it with your deep learning model. Ah, PS from the future,


all the Rick and Morty episodes are practically true. Those guys are like fortune tellers.


Back to Dan. All right so I couldn't find my friend and maybe TensorFlow will have to be explained in


another episode but that's all right. Tomorrow I'm going to get into part three of the deep learning


specialization. Let's do it. I'm going to finish up today about halfway through week two of course


three on the Coursera deep learning specialization. Check it out here. What are we up to? ML Strategy


2. So this part of the course or course three of the deep learning specialization, let me just put


you down here, is all about improving your machine learning model. So once you've sort of built it


and once you've hit the target of or got some good results at least or maybe you've got some bad


results, this week is teaching us all about how to improve those results or how to take the results


you've got from hitting the wrong thing and adjust them so that they can be more towards the thing


you're or towards the answer you're really after. Actually one of my favorite takeaways from this


so far has been that I know there's there's a lot of content on sort of look at this actually I'll


show you my notes on a lot of technical stuff on how to improve your models whatnot doing tests


evaluating ideas in parallel a whole bunch of quiz notes there there's look there's a quiz question


that I got wrong and then we go up here I can't even pronounce this almost or orthogonalization


orthogonalization comment below if you can pronounce that better than I can. So there's a


lot of in-depth stuff and a lot of stuff that you can definitely definitely use to improve your


models but one of my favorite parts of it was the interview with Andrej Kapathy. If you haven't heard


of Andrej Kapathy, he's a computer science PhD who taught a course at Stanford on computer


vision. I'll put a link below to his to his course actually. His insight he's currently


head of AI at Tesla and one of his insights that he said at the end towards the end of the interview


was that when they were trying to he thinks the field of AI going in two different directions


one will be more so building models for different tasks and the models keep getting better and


better and better and there'll be another sort of another branch off of AI which will be


the teams working towards artificial general intelligence and what his insight was was he


said that when they were first getting into computer vision and he's definitely an expert


on computer vision he taught a course on it at Stanford he said they were trying to do it in


compartmentalized pieces so for example they do facial structures and then do maybe animals and


then do cars or something so different subsets of computer vision tasks and I may be getting that


wrong but just imagine different subsets rather than doing it as a whole piece and he thinks


that's that's maybe the key to artificial general intelligence is rather than nailing natural


language nailing speech nailing computer vision and then trying to merge them all together it


might be better to work on it as a system as a whole. That was really cool for me actually


there's a blog post let me find it and I'll show it to you quickly because I'm going to read it


and I'll link it in the description so you can read it too and comment below what you think.


So here it is a short story on AI a cognitive discontinuity I can't speak tonight so just


just pretend I pronounced that word correctly but this is on Andrej Kapathy's blog I'll link


this below I'm going to read it and I'll let you know what I think. That's it for me to study


tonight I'm actually I'm not going to read that because I don't like looking at screens while


I'm in bed this is the book that I'm reading at the moment it's incredible absolutely highly


recommended it's a short read it's only about 150 or so pages but Victor Frankl was actually a


prisoner in law switch and some of the insights in this book are incredible so if you're looking


for a new book to read I'd check this one out it's only I think nine dollars or something on


Amazon or something like that or book depository but I'll leave a link in the description so you


can check it out and if you have read it comment below what do you think. I want to show you


possibly one of my favorite parts of this course so far so this is a quiz or actually it's a case


study so at the end of week three or the no sorry course three and it's an autonomous driving you


can imagine it as a flight Andrew refers to it as a flight simulator for machine learning


and as you can see I failed at the first attempt so I got 11 out of 15 but the beauty is I can


try it three times every eight hours so this is an autonomous driving case study so to help you


practice strategies for machine learning and this week we'll present another scenario and ask you


how you would act. We think this simulator of working in a machine learning project will give


you a task of what lending of what leading machine learning project could be like I'm reading this


through the camera screen by the way I should really just read it normally so essentially you're


employed by a startup building self-driving cars amazing you're in charge of detecting road signs


stop signs for the tune signs construction signs ahead etc and traffic signals red and green lights


and images the goal is to recognize which of these objects appear in each image as an example


the above image or the below image it should be contains a pedestrian crossing assigned with red


traffic lights so there we go example image example output and so what the questions are is


imagine you're in this scenario say for example I'm going to show you one rather than me explain it


let's go here first question would be you were just getting started with this project what is the


first thing you would do assume each of the steps below would take an equal amount of time a few


days and I got this one right so the first thing you should do is spend a few days training a basic


model and see what mistakes it makes so correct as discussed in the lecture applied ml is highly


is a highly iterative process if you train a basic model and carry out error analysis see what


mistakes it makes it will help point you in the promising directions that's one example of the


questions you go through and I'm really enjoying it because it's it's sort of it really feels like


as if I'm working on the self-driving car project it's like someone's coming to me and asking hey


we've got this scenario what should we do here and of course I haven't got it all down pat yet


because I only got 14 I mean 11 out of 15 I'm going to do the quiz again in a second but I think


it's really good to get this type of experience and as Andrew says he's I haven't he hasn't come


across any kind of scenarios like this before in other courses and neither have I actually and so


it's really good to get hands-on experience with what a case study would be like in in working in


some sort of startup or some sort of company on these deep learning projects which is which is


what do I what I intend to do in the near future so let's look at an example of what I got wrong


I got two wrong in a row for example your goal is to take road signs stop signs pedestrian crossing


signs a bunch of signs and traffic signals the goal is to recognize which of these objects appear


in each image you plan to use a deep neural network with relu units in the hidden layers for


the output layer a soft soft max activation would be a good choice for the output layer because this


is a multi-task learning problem true or false so I selected true but it should be false so maybe


a soft max shouldn't be used for the the final layer I'll find out where I went wrong and catch


up on the next attempt all right we made it so second attempt I got 100% because I went over my


errors I took some screenshots of the questions I got wrong and and put them in here so that when I


was going back over the question I could choose a different answer follow up to the question I got


wrong before question two so it was actually false so soft max would be a good choice if one and only


of the possibilities stop signs speed bump pedestrian crossing green light and red light was


present in each image but we need more than one possibility soft max to be good if if this was red


light traffic light was zero and there was only one output but soft max doesn't help when we need


more than than one potential output on to the next course so before getting into course four I was


watching this interview with Andrew and Ruslan Salak Hatdinov probably pronouncing that wrong


but Ruslan is the head of AI and or director of AI research at Apple which is really cool and it's


amazing to get all these insights from some of the what's called it's heroes of deep learning


in this course and it's good to get their perspective and how they got into the field


because I know they all came from such different backgrounds like for example Ruslan did his


masters in computer science and then he went and worked in the finance sector and then he randomly


bumped into Jeffrey Hinton if you remember from either last video the one before who's the


godfather of deep learning and then Hinton got him into deep learning and what he was working on in


artificial intelligence and whatnot maybe not deep learning at the time but got him into AI


and that's when he did his PhD fast forward a few years he's now head of director of research


AI research at Apple so that's amazing and I took a few notes on on what his favorite favorite parts


of deep learning are and his advice on on how to get into deep learning one of his main takeaways


was that when he teaches a class um where is it yeah when Ruslan teaches a class on deep learning


he practices he gets his students to practice coding a back propagation algorithm algorithm


for a convolutional network yourself so I'm excited to do that I think I'm doing that in


the next course and he says deep reinforcement learning is very exciting training agents in


virtual environments that's something I'm really excited about too I wrote an article on medium


actually about deep reinforcement learning it's it's a high level article so it's not


it doesn't go in depth on what deep reinforcement learning actually is but it was on how DeepMind


trained AlphaGo Zero sorry AlphaGo not AlphaGo Zero and they've released a new environment for


StarCraft 2 to help train deep reinforcement learning agents but I'm on to the fourth course


of the deep learning specialization which is on convolutional neural networks and we're going to


start off with some computer vision so I'm going to jump into this and then once I've made some


progress I'll let you know. Alright guys so we just finished the first project of week one of


course four on the Coursera deep learning.ai deep learning specialization and it was essentially


coding a convolutional neural network by hand forward propagation and back propagation so we


see all the way down that scroll look how small that scroll bar is so this was this is probably


the hardest project yet in the whole whole specialization just because it was probably


the most programming intensive one so far as well and the fact that convolutional neural nets


are if you've they're kind of hard to understand if you haven't been over them before so what I've


done is I've drawn a very basic overview or how I see convolutional neural nets so far and mind you


I'm still a beginner so don't take this as gospel but this is how I think about it and maybe if you


check it out you could leave a comment and you know more about it than I do you could tell me


how to improve on it. Here's what we have see a convolutional neural net by the way is used to


recognize images so that's that's the main thing or video or images just imagine it for convolutional


C for computer vision so that's what it's used for so imagine if we had a million pictures of trees


and convolutional neural network starts off by making a little filter a little box and it's


going to go across this image we'll fill in the gap there and then it's going to keep going going


going moving across and all the way through the image till it's done it layer by layer and then


what it's going to produce over here is a matrix of zeros and significant values so if you notice


the ones treat them as significant values it may not necessarily be ones but it could be something


else so this means that the filter has picked up that there's something here in the image


so the rest is all zeros and this is very roughly drawn so forgive me for that and then


it's going to convert it even further into eventually being a vector so it's going to be


zeros zeros zeros and then some ones throughout there because after all what a computer's understand


best they understand numbers so a computer can't necessarily understand a picture of a tree so it


turns it into numbers and then what do we do with those numbers well say we did that with a


tree well say we did that this process a million times right and from what from that what you would


get is a whole bunch of these that you can use to recognize other pictures of trees what the


computer could do is find patterns in those vectors so find patterns in the zeros and ones


and then use those patterns to recognize future images or or scan over future images or images


and then use those patterns to recognize the future data and then use those patterns to


repeat it and it will give back whether there's a tree in there or not because that new image


will have similar significant features aka same amount of ones and zeros as other images you've


fed it aka the millions of of images you've fed it as as the initial data and remember deep


learning network or especially deep learning network it's get more data get some good data


make sure it's good label data because it bad data will make it worse of course and understand


that computers don't understand what what a picture of a tree exactly is so or anything


really for that matter the facial recognition on your iphone 10 or iphone x doesn't recognize your


face it recognize it turns your face into numbers and it recognizes the numbers so that's the two


takeaways for deep learning as a whole and convolutional neural nets and that's how i


understand them if you have a better understanding of them please leave a comment below and help some


other people out as i learn more about it i'll bring more to you after after this i've got to


all my next step in the deep learning specialization is another another project programming


assignment prog i can't speak programming assignment but instead of coding it convolutional


neural networks by our hand or step by step we're going to use tensorflow and as i've said before