Testing the New Haystack Doc Store

then we'll leave that for later. So for now if you wanted to test this at the


Testing the New Haystack Doc Store

moment it's not in Haystack yet the pull request hasn't been accepted so if you want


Testing the New Haystack Doc Store

to clone this which I'm going to be going through. I'm going to git clone and you're


Testing the New Haystack Doc Store

coming from pinecone.io slash Haystack. Okay and then from now we want to


Testing the New Haystack Doc Store

activate the pinecone.store branch so I'm going to actually do that here so


Testing the New Haystack Doc Store

I'm going to cd into the correct Haystack directory so documents


Testing the New Haystack Doc Store

projects Haystack and then in here I can I can check this so I am already on this


Testing the New Haystack Doc Store

branch okay so it's not not really an issue for me. I have a Haystack


Testing the New Haystack Doc Store

environment I've set up so I'm going to cd activate Haystack. Okay


Testing the New Haystack Doc Store

was last thing okay and then pip install so pip install so I think this should


Testing the New Haystack Doc Store

already be installed the most recent one for me so it should be pretty quick. Okay


Testing the New Haystack Doc Store

so that has installed and then we can go ahead and actually use this so set this


Testing the New Haystack Doc Store

to make sure using the Haystack environment and then let's go through


Testing the New Haystack Doc Store

everything. So this is not so I have the Haystack version that I've been


Testing the New Haystack Doc Store

working on in this directory now in this Haystack test directory. I've just been


Testing the New Haystack Doc Store

making a load of notebooks here to test how things work and figure everything


Testing the New Haystack Doc Store

out and this is a notebook I'm using to test the pinecone.doc.


Testing the New Haystack Doc Store

So I initialize it first with this pinecone from Haystack.doc.


Testing the New Haystack Doc Store

import pinecone.doc so I can run that and this is just using so


Testing the New Haystack Doc Store

we have the environment so these are pinecone specific arguments here so the


Testing the New Haystack Doc Store

environment I want to run it on so I'm using the default one which is US West


Testing the New Haystack Doc Store

one on GCP and then my key set a key I am getting from in here so I have my app


Testing the New Haystack Doc Store

that pinecone.io this is my Haystack project if you if you like have just


Testing the New Haystack Doc Store

signed up for this it will come up with like your name default project so you


Testing the New Haystack Doc Store

just click on that and then you go to API keys here I said I'm using my


Testing the New Haystack Doc Store

default one I can just copy over here and then I've placed it it's not the


Testing the New Haystack Doc Store

best way to do it but I just find it easy I've just placed in a in a file


Testing the New Haystack Doc Store

called secret I'm reading from there you can also put it in you know slightly


Testing the New Haystack Doc Store

worse way of doing this you can put it in directly in the notebook as well I


Testing the New Haystack Doc Store

would I wouldn't even recommend doing this so I would recommend doing that


Testing the New Haystack Doc Store

even less otherwise you want to enter the API key into your environment


Testing the New Haystack Doc Store

variables and then you can just load it like OS environment something like this


Testing the New Haystack Doc Store

like API key okay


Testing the New Haystack Doc Store

but I'm just loading it from file I find it easier so okay so we've initialized


Testing the New Haystack Doc Store

the document store here we can see so we have the embedding dimensions by


Testing the New Haystack Doc Store

default this is 768 so if you're using a model that doesn't use that dimension


Testing the New Haystack Doc Store

which most models do but maybe you're using for example like the mini LM


Testing the New Haystack Doc Store

models they use dimensionality of 368 maybe so I think the bedding what is it


Testing the New Haystack Doc Store

vector no okay let me have a look at the the document source I can see which


Testing the New Haystack Doc Store

argument is okay so we come down to here and the oh yeah okay it's vector dim as


Testing the New Haystack Doc Store

expecting it to come up so that weird so vector dim is three would be like 368


Testing the New Haystack Doc Store

for example but in this case it's fine the default value of 768 is it's not


Testing the New Haystack Doc Store

problem and then we see record count is zero because we haven't inserted


Testing the New Haystack Doc Store

anything into our document store yet so we're gonna go ahead and actually do


Testing the New Haystack Doc Store

that but to do that we do need to download some documents so important a


Testing the New Haystack Doc Store

few things here this is just using one of a site's like demo notebooks so I'm


Testing the New Haystack Doc Store

simply putting these these things here clean the text all these things and then


Testing the New Haystack Doc Store

we convert these to these files to dictionaries I run that and then we can


Testing the New Haystack Doc Store

see those dictionaries in here so we have this content and yeah we get all of


Testing the New Haystack Doc Store

our documents here we can see we the documents have this particular format


Testing the New Haystack Doc Store

where we have content which is like the length of text or the big chunk of text


Testing the New Haystack Doc Store

let me go up to this one actually and then we also have metadata and we have


Testing the New Haystack Doc Store

the name of the file in here and we could put other metadata in here as well


Testing the New Haystack Doc Store

so the align aligns quite well with the format that pinecone consumes so that's


Testing the New Haystack Doc Store

useful at least now one thing that we have here at the moment I don't want is


Testing the New Haystack Doc Store

too much data because at the moment I'm working from my Mac and it's not


Testing the New Haystack Doc Store

particularly powerful so what I'm going to do is we have a lot of bits here so


Testing the New Haystack Doc Store

if I just look we have yeah 2.4 or almost 2.5 thousand examples it's gonna


Testing the New Haystack Doc Store

take a long time to embed all of those all the contents into vectors on on my


Testing the New Haystack Doc Store

Mac so I am going to just take like the first maybe 10 or just six examples


Testing the New Haystack Doc Store

because it will be quicker and I can still test everything with that okay so


Testing the New Haystack Doc Store

I'll do that and then here we we come to writing the dictionaries containing


Testing the New Haystack Doc Store

documents or database now this doesn't do anything this doesn't upset them into


Testing the New Haystack Doc Store

pinecone because we and the current implementation uses both a sequel


Testing the New Haystack Doc Store

database that saw the long contents and also pinecone to saw the vectors and the


Testing the New Haystack Doc Store

metadata at the moment we don't have a retriever set we do that next so this is


Testing the New Haystack Doc Store

just going to write everything to the local sequel database sorry I run that


Testing the New Haystack Doc Store

we've written those documents very quick obviously we're in here six and then we


Testing the New Haystack Doc Store

come here and I can initialize the dense retriever so in this example we're using


Testing the New Haystack Doc Store

this Facebook DPR model and we'll stick with that obviously you can you can spot


Testing the New Haystack Doc Store

the other models as well so we'll leave that there and turns it update so once


Testing the New Haystack Doc Store

I've initialized the retrieve model I can update the embeddings using that


Testing the New Haystack Doc Store

retriever model and a small batch size now 16 is pretty big we don't actually


Testing the New Haystack Doc Store

need to do that many so two just doing batches of two so yeah let's run that


Testing the New Haystack Doc Store

shouldn't take too long okay so we've just processed those that's


Testing the New Haystack Doc Store

yes finished running so now we have those vectors and the metadata and


Testing the New Haystack Doc Store

everything in pinecone and also in the sequel database we can we can see that


Testing the New Haystack Doc Store

as well so if we go to our pinecone console again so if I okay so loading


Testing the New Haystack Doc Store

and we should see in a moment okay we have this index document here so if I


Testing the New Haystack Doc Store

click on that will give me index information zoom out a little bit and


Testing the New Haystack Doc Store

I'm seeing number of vectors in there at the moment is six okay so we've we have


Testing the New Haystack Doc Store

those six items or documents in pinecone as expected now from there we can set up


Testing the New Haystack Doc Store

our QA pipeline so we're forming extractive QA or open main question


Testing the New Haystack Doc Store

answering so we run that this might take a moment just to initialize everything


Testing the New Haystack Doc Store

in in haystack okay so that has run and now I can actually start asking


Testing the New Haystack Doc Store

questions now at the moment this is going to just return a load of rubbish


Testing the New Haystack Doc Store

because we we only have six documents in there none of those documents talk about


Testing the New Haystack Doc Store

any of this so we're not going to return anything relevant here but we just want


Testing the New Haystack Doc Store

to release during testing all I'm doing is making sure that everything is


Testing the New Haystack Doc Store

pointing to the right place and actually processing in a way that I would expect


Testing the New Haystack Doc Store

so when I run this I would expect a prediction to be returned I would expect


Testing the New Haystack Doc Store

five answers from that prediction and I would expect it to run so here we can


Testing the New Haystack Doc Store

see it's inference six examples here so come up here okay so we have so because


Testing the New Haystack Doc Store

we are retrieving ten here where we retrieve those ten examples but in our


Testing the New Haystack Doc Store

case there's only six examples so far hi let me reduce this to five and we'll do


Testing the New Haystack Doc Store

three here but let me explain it quickly so we're returning all six examples and


Testing the New Haystack Doc Store

then this inference in samples where it has loading bar that's referring to the


Testing the New Haystack Doc Store

reader model take a look at that single example and scoring it in it and pulling


Testing the New Haystack Doc Store

out a specific answer from that obviously we only have six examples here


Testing the New Haystack Doc Store

so we only see inferencing examples six times okay so now if I reduce top K in


Testing the New Haystack Doc Store

the retriever to five and read I'm just reducing a little bit because typically


Testing the New Haystack Doc Store

that is a lower value than your retriever top K we should see now that


Testing the New Haystack Doc Store

does inferencing samples five times rather than six so with one two three four


Testing the New Haystack Doc Store

and five okay so now we're running or looping through each of those five


Testing the New Haystack Doc Store

return examples inferencing the reader model extracting answers and then we


Testing the New Haystack Doc Store

should be able to use this print answers and we should get something from there


Testing the New Haystack Doc Store

now we can see that although it's not returning the great answer because the


Testing the New Haystack Doc Store

great answer is not in there we only have those six contents it is at least


Testing the New Haystack Doc Store

returning something that would make sense from a syntax point of view so


Testing the New Haystack Doc Store

we're saying who specifying who created this vocabulary and it's returning the


Testing the New Haystack Doc Store

name of a person okay so in one of these in one of these contents there's a name


Testing the New Haystack Doc Store

person so it's pulling out the name person because it knows that you were


Testing the New Haystack Doc Store

asking a question about who created something so the answer that would


Testing the New Haystack Doc Store

naturally be the name of a person it's not the right answer but at least it's


Testing the New Haystack Doc Store

you know almost being logical in the answer that is returned okay so as well


Testing the New Haystack Doc Store

as that I want to I wanted to test the other functions not just the querying


Testing the New Haystack Doc Store

in Haystack also getting your documents by IDs so if we run this these are just


Testing the New Haystack Doc Store

the IDs that have been assigned to the different documents we should get two


Testing the New Haystack Doc Store

sets of answers so we have document we have the content here and we also have


Testing the New Haystack Doc Store

we have the embedding as well if you need that metadata and it goes on for a


Testing the New Haystack Doc Store

little while so if I open in the text editor so we can see everything so we


Testing the New Haystack Doc Store

have all of these and then we have the second so just here see on the right we


Testing the New Haystack Doc Store

have the second document so it's returning two of those so that's good


Testing the New Haystack Doc Store

that seems to be working as we expect


Testing the New Haystack Doc Store

okay so let's minimize that and the last thing that I want to check is okay is


Testing the New Haystack Doc Store

them is a metadata filtering working so I actually tested this a lot more than


Testing the New Haystack Doc Store

what we have here so we can maybe you can switch over to that notebook because


Testing the New Haystack Doc Store

then we can at least see you know what level of filtering we can actually do


Testing the New Haystack Doc Store

here now again this this will change slightly as well so let's make sure this


Testing the New Haystack Doc Store

is a right notebook so what I would do rather than running all this again I'm


Testing the New Haystack Doc Store

just going to take this this is using a different a different index however so


Testing the New Haystack Doc Store

I'll just show you this rather than running through it okay so in this


Testing the New Haystack Doc Store

notebook I had more time to run this I think I didn't run it on my Mac I run it on


Testing the New Haystack Doc Store

my my other computer which is a lot faster so in this notebook what we're


Testing the New Haystack Doc Store

doing is using the squad data set only 4,000 examples I wanted I still wanted


Testing the New Haystack Doc Store

it to be quick where initially specifying that we want to create a


Testing the New Haystack Doc Store

squad index in the pinecone document store rather than default document index


Testing the New Haystack Doc Store

we're using a different model as well so vector dim is 384 in this example and


Testing the New Haystack Doc Store

then here I'm just getting the all the context from squad so I'm getting all


Testing the New Haystack Doc Store

the context and all the titles the titles are just for my reference so I


Testing the New Haystack Doc Store

can modify the metadata filters later the context of what we actually saw or


Testing the New Haystack Doc Store

encode and store see I'm writing them there and then here I'm initializing


Testing the New Haystack Doc Store

dense passage retriever this time Robin so we're using dense passage retriever


Testing the New Haystack Doc Store

here DPR but we can replace this with other sentence transformer models as


Testing the New Haystack Doc Store

well and it's so still works I don't know if there's any particular reason


Testing the New Haystack Doc Store

not to do this but it works so you can do it at least in this case it works so


Testing the New Haystack Doc Store

I use sentence transformers here it's a smaller model so I thought it gets easier


Testing the New Haystack Doc Store

a little bit faster and then I updated the embeddings for read a model just


Testing the New Haystack Doc Store

use a default stuff here again and then I run this so which college at Notre


Testing the New Haystack Doc Store

Dame ad in 1921 we have all the top case of here and in this case it should


Testing the New Haystack Doc Store

return a relevant answer okay and we see so it's the way that haystack returns I


Testing the New Haystack Doc Store

was a little confused at this first haystack returns like a small segment of


Testing the New Haystack Doc Store

your context or contents not the full contents so we get the answer which is


Testing the New Haystack Doc Store

College of Commerce and then we get the context that has pulled from right


Testing the New Haystack Doc Store

so that's quite cool so by 1921 the addition of the College of Commerce so


Testing the New Haystack Doc Store

is returning the answer that that's I think really cool to see and then I


Testing the New Haystack Doc Store

wanted to obviously begin adding filters using specifically with haystacks


Testing the New Haystack Doc Store

filtering syntax so there's a lot of testing here to make sure different


Testing the New Haystack Doc Store

filters work with the code that I wrote so this is haystack filtering context or


Testing the New Haystack Doc Store

syntax the only difference between this and for example pinecones is that we


Testing the New Haystack Doc Store

need to have this okay so we would have that and I also think that this here


Testing the New Haystack Doc Store

would be like a single dictionary and this would also be a single dictionary


Testing the New Haystack Doc Store

right and then we would remove that dictionary I think I think that's the


Testing the New Haystack Doc Store

pinecone version of it but the haystack syntax slightly different obviously


Testing the New Haystack Doc Store

they're inspired by the same original like syntax or filtering and they just


Testing the New Haystack Doc Store

use this dictionary okay so there's a method in the document store at the


Testing the New Haystack Doc Store

moment which I can open show you this is going to change though so if I come down


Testing the New Haystack Doc Store

here we built filter built filter clause so this is what is handling the


Testing the New Haystack Doc Store

translation from haystack syntax to pinecones syntax it's relatively messy


Testing the New Haystack Doc Store

so there's probably some iterations to make it a bit cleaner possibly if


Testing the New Haystack Doc Store

possible but there's a lot of you know if and elif stuff in here depending on


Testing the New Haystack Doc Store

what syntax we're looking at so it's kind of hard to make this particularly


Testing the New Haystack Doc Store

clean but that's what's handling that and that is called whenever you have a


Testing the New Haystack Doc Store

filter specified in your in your query so let's go back to this so this yep we


Testing the New Haystack Doc Store

have a very simple filter in haystack syntax does it make sense probably


Testing the New Haystack Doc Store

probably not but I wanted to make sure it worked and and this will just be


Testing the New Haystack Doc Store

translated into pinecones syntax and actually work you can see that when when


Testing the New Haystack Doc Store

you actually run this you have I think one oh you have this or so you have only


Testing the New Haystack Doc Store

items from the Age of Enlightenment or you have this single context that


Testing the New Haystack Doc Store

returned so you can see those which is pretty useful and then I'm just


Testing the New Haystack Doc Store

modifying those filters testing some different things just going through


Testing the New Haystack Doc Store

those so it's relatively simple testing there's not there's not a lot going on


Testing the New Haystack Doc Store

here a little more complex here where these almost like layered statements but


Testing the New Haystack Doc Store

it's still pretty pretty straightforward so that's the filtering testing and then


Testing the New Haystack Doc Store

I think from now it's pretty much it and the only other thing was get all


Testing the New Haystack Doc Store

documents so if I can run this and if I run that we see that we actually do


Testing the New Haystack Doc Store

return all those documents so that's another method I wanted to make sure it's


Testing the New Haystack Doc Store

working so I think that's it for like going through how the how you would


Testing the New Haystack Doc Store

actually use this document so again there's still things that will be


Testing the New Haystack Doc Store

changed but none of these methods as far as I know should be possibly the I think


Testing the New Haystack Doc Store

I did see in a comment that maybe the vector dimension so here this might


Testing the New Haystack Doc Store

change but otherwise everything else should stay the same so yeah that's it


Testing the New Haystack Doc Store

for this walkthrough I hope it's been useful interesting so thank you very


Testing the New Haystack Doc Store


Building MLM Training Input Pipeline - Transformers From Scratch #3

Hi, welcome to this video.


So this is the third video in our Transformers from Scratch mini-series.


And in the last two videos, we basically got a load of data and trained our tokenizer,


which is what you can see here.


So I'm going to have to rerun these.


Okay. And we just tokenize some Italian.


So this is a Italian BERT model, Roberta model.


So we can write something like this, which means, hello, how are you?


And it will tokenize our text here.


Now, this is where we got up to.


What we now want to do is build out a input pipeline.


And train a model.


Now, this is reasonably, I would say, involved.


Because we need to do a few things.


First thing is we need three different tensors in our model here.


We need the input IDs and attention mask.


We also need the labels tensor as well.


So the labels tensor is actually just going to be our input IDs as they are at the moment.


But input IDs are not going to be that.


Our input IDs, they need to be passed through a mass language modeling script,


which will mask around 15% of the tokens within that tensor.


And then whilst we're training, our model is essentially going to be


trying to guess what those mass tokens are.


And we'll optimize the model using the loss between the guesses that the model outputs


from the input IDs and the real values of our labels.


So that's essentially how it's going to work.


I suppose it's a lot easier said than done.


So first thing we're going to do is create our mass language modeling function.


If you have watched some of my videos before, we quite recently did, I think, two videos,


maybe two videos on mass language modeling.


I'll leave a link to those in the description,


because the code is pretty much the same as what we cover there.


And I will cover it very quickly here, but not too in depth.


So if you're interested, those links will be there in the description.


So the very first thing we need to do, which I haven't done yet,


is import torch.


So using PyTorch here.


And we need to create a random array.


So we write torch, and this random array needs to be in the same shape as our tensor


that we've input up here.


So let's write tensor.shape.


And then we want to mass around 15% of our model.


And then we want to mass around 15% of those.


So to do that, we use rand, where rand is less than 0.15.


Because what we've created here is an array in the shape of our input tensor,


which is going to be our input IDs, where every value is within the range of 0 to 1.


So and that's completely random.


That means there should be, or for each token, there's a roughly 15% chance of that value


being under 0.15.


So that's our first criteria for our random masking array, the roughly 15%.


We can call this mask array as well.


But there's a few other criteria as well.


And again, I covered that in the other videos.


But in short, we don't want to mask special tokens.


So we can see up here we have two special tokens.


If we add padding, so if I'm just going to add a little bit of padding, not loads.


Let's go max length 10.


And we write padding equals max length.


If we do that, we get these extra ones here.


They are padding tokens.


So we basically would say we don't want to mask our zeros, twos or ones, because they're


special tokens that we don't want to mask.


So we also just put where tensor is not equal to zero.


And let's just copy that.


It's a little bit easier.


And also not equal to one, and it will also not equal to two.


So, and I do wonder if, yeah, we could make that a bit nicer.


So if we just do, so you can either do this, right?


Where you specify each token and you will want to do that sometimes.


Maybe like with BERT, because your special tokens are in the range of like 100, 0, 101,


so on.


There's a few different ones.


Because we've got everything, so either two or below, we could just write this.


So we could say where it is not, or where it is greater than two.


So this is like an and statement saying, we're going to mask tokens that have a randomly


generated value of less than 0.15.


That's our 15% criteria.


And they're not a special token, e.g. they are greater than the value two, because our


special tokens are 0, 1, and 2.


So that's cool.


Now what we want to do is loop through each row in our tensor.


So we want to do for i in range tensor.shape 0.


So this is how many rows we have in our tensor.


And we can't do this in parallel, because each row is going to have a different number


of tokens that will be masked.


So if we did this in parallel, we'd end up trying to fit different sized rows into an


equally sized tensor.


So we can't do that.


Again, if this is confusing, I have those videos.


But I mean, you don't need to specifically know everything that's going on here.


This is just how we mask those roughly 15% of tokens.


So we want torch, flatten, and this is a bit confusing, but we want to take the mask


array at the current position.


I want to say where it's not 0.


So when we create this mask array, we essentially get a load of true or false values in the


size of our tensor shape.


Where we have 1s, that is a mask.


And what we're doing here is we're saying, get me a list of all the values that are not


0, e.g.


They're 1s.


And that gives us a list within a list.


So we get something like this, and it will say indices 2, 4, 18.


I don't know.


I said 4.


It's 5.


2, 5, 18.


They are where your mask tokens will be.


And then we use torch flatten here to remove that outer list.


And at the end here, we're going to convert it to a list so that we can do some fancy


indexing in a moment.


And that fancy indexing looks like this.


So we have our tensor.


We're specifying the current row because we're going row at a time.


And then we want to specify that selected number of indices where we're going to place


our mask.


Now, what does the mask token look like?


Well, we can actually find it over here.


Over here in our vocab.json.


Yeah.


So scroll to the top and we see our mappings here.


So the mask token is number 4.


So that's what we're going to use.


Switch back over.


So we're going to make those values equal to 4.


That's our mask.


Then at that point, we have successfully masked our input IDs and we want to return the sensor.


So that's our masking function.


That's a big part of this video.


That's one of the harder parts.


So now what we're going to do is I'm going to scroll up a little bit to here.


So we have...


I'm just going to take this.


So this will give us a list of all of our training files.


So here.


And we just need to do from path lib, import path.


Import path.


Okay.


Let's have a look at what we have.


So this is just a list of everything that we have over here.


So these are text files containing our Talion samples.


Each sample is separated by a newline character.


And each file also contains about 10,000 samples.


So we have quite a bit of data.


And what we're going to do here is we're going to create our three tensors.


So the three tensors that I mentioned before, we have...


Thought I'd make a list.


I didn't make a list.


So we have the labels and input IDs, and then we also have the attention mask as well.


So let's first initialize those as a list.


So input IDs, attention mask, or I'm just going to call it mask.


And labels.


And what we're going to do is...


So we're going to use a progress bar here.


So I'm just going to import.


So from TQDM, auto import TQDM.


So you have to import that as well.


And what I'm going to do is loop through each path in our...


I'm going to wrap it in TQDM.


This creates our progress bar in our paths.


For each path, we're going to load it, extract all data,


convert it into the correct format that we need here,


and append each one of those to the list,


and then create a big tensor out of that.


So we want to write with open.


And then here we have our path, our reading, and the encoding is UTF-8 as F.


We want to write text equals F.read.split like that.


So I'm going to rename it lines.


So this is just a big list of 10,000 samples that are all Italian.


So then we want to encode that.


So we write sample equals tokenizer lines on our max length,


which is going to be 512.


We want padding up to that max length.


And we also want to truncate anything that is further than that.


So truncation equals true.


Okay, that's our tokenization done.


Then we want to extract all of those and add them to our list.


So we get our labels first.


Now, the labels are just the input IDs produced by our sample.


So sample, input IDs.


And I'm thinking here we can do return tensors, use PyTorch.


So append our input IDs to labels.


And then we have our mask.


We want to append the sample attention mask.


And then we can also see that here, by the way.


Here, this is what we're doing.


We're taking those out, putting them into our list.


And then, so we have labels, mask, and we want to create our input IDs.


Now, input IDs, that's what we built this mass language modeling function for.


And in there, we need to pass our tensor.


So to do that, we just want to write sample, input IDs.


And before I forget, that needs to go within MLM, like that.


Now, I don't want to modify that tensor because it's been appended to labels.


So I'm going to create a clone of that.


And that will be done using attach and dot clone, like that.


So it's pretty good.


It's pretty good. Let's run that.


Okay. And it's going to take a long time.


So, yeah, I'm not going to use all of it.


It was going up as well, so I have no idea how long that'll take.


Let's leave that for a little bit.


Let's go to first 50 for now.


I've still got to wait a little while, but at least not as long.


So I'll leave that to run.


Hopefully, it shouldn't take too long.


And yeah, I'll see when it's done.


Okay. So that's done. It wasn't too long.


And if we just have a look.


So input IDs at the moment is just a big list.


I don't know if it's a good idea, but here we go.


So we just have like a list of tensors.


What we can do is rather than having a list of tensors,


we can use something called torch cat.


And torch cat expects a list of tensors to be passed through it,


which is why I've done this.


But we have lists and we just append tensors to it.


And we can do that and it will concatenate our tensors,


which is pretty cool.


So what we want to do now is we write input IDs


and we're just going to concatenate all of our tensors.


So then they're ready for formatting into a dataset.


So we have mask here and labels here.


We can also see just worth pointing out,


we have that mask token there.


So we know that we have mask tokens in our input IDs now.


Let's run that and let's just compare.


So let's go input IDs, zero.


That's quite a lot.


So can I just do the first 10?


And then let's do the same for labels.


We'll see that we don't have these fours


or we hopefully shouldn't have those fours.


Okay. So that's essentially a masking operation.


So cover this with a mask here and same here and here.


Okay, cool.


Now the format that our dataset needs and our model needs


is a dictionary where you have input IDs,


which maps to input IDs obviously.


And you can guess either two as well.


So input IDs, this one attention mask to mask and the final one is labels.


So there are encodings.


Now we create a dataset object to create a dataset object.


In fact, actually we create a dataset object


to create a data loader object,


which is what we use to load data into our model.


And that's essentially our input pipeline.


So, but to create that data loader,


we need to create a dataset object.


Now the dataset object, we create that by, well, like this.


So we do class dataset, call it whatever you want.


And we want torch, utils, data, dataset, like that.


We need a initialization function,


which is going to store our encodings internally.


Don't forget the def there.


So we want to write self encodings equals encodings.


So this is initializing our dataset object.


And then there's two other methods that this object needs.


We need a length method.


So that we can say length dataset,


and it will return the number of samples


that are in the dataset.


And we also need a get item method,


which will allow the data loader to extract a certain,


so say, if it says, give me number one,


it's going to go into this dataset object


and extract the tensors, the input IDs, attention,


mass and labels at position one.


So that's, yeah, that's what we need to do there.


So we'll do length first.


And length, we don't need to pass anything in there.


We'll just call it length.


So from that, we just want to return the self encodings,


do input IDs.


And remember before we did this shape


and we took the first one, which is the length.


So if I took second input ID,


if I took second input IDs,


if I can just do it here, so I'll copy that.


If I go here, we get that 500k,


which is the number of samples we have.


That's what we want to return.


Okay, so that's our length.


And then we also have the get item.


So here we do want to pass a index value.


So this is going to be,


our data load is requesting a certain position.


And for that, we want to return.


So we're going to return dictionary.


It needs to be in this format here,


but we need to specify the correct index.


Now, what we could do is we could do like self encodings


and then access our input IDs like that.


We also need to change that here.


So it will give us an error.


Dot shape.


And we could do that.


So we could take that like so,


and then just say index position.


That's fine.


You can do that if you want.


But an easier way of doing it


where we don't need to specify the,


we don't care about the structure of the dataset.


We just want to get it out.


We don't need to specify it.


We can just do this.


You write key tensor.


So the specific index of that tensor for key tensor


in self encodings the items.


So if we were to go encodings the items,


so we can do that here.


See, we get essentially everything in our dataset.


So we're just looping through that,


returning it and specifying which index we're returning here.


So once we have written that,


we can initialize our dataset.


So our dataset equals dataset.


And then we just pass in our encodings there.


So just move that and encodings.


That's it.


So that's our dataset.


And now we initialize our data loader.


So this is pretty much it for our input pipeline.


So data loader equals torch utils.


It's just coming from the same area as our dataset.


Data loader.


Now we pass in our dataset object.


We want to specify batch size.


So I typically go with 16.


This will depend on how much your computer can handle it once as well.


So just play around that, see what works.


And we also want to shuffle our dataset as well.


So yeah, that's our input pipeline.


After that, obviously we want to feed it in


and train our model with it.


So that's what we're going to do.


So we're going to cover that in the next video.


Angular App Setup With Material - Stoic Q&A #5

Okay, so in the last video, we put together our API using Python, and that included our


sort of full Q&A stack where we can begin asking questions and getting answers.


So now what we're going to do is start working on the front end, which is going to be in


Angular.


Now if you haven't used Angular before, that's fine.


I'm really definitely not an expert, so we can learn together.


And if you haven't installed Angular before or will use it or I assume if you don't use


it, you probably don't have it installed, then I'm going to link right at the top of


this video description to a article or like a guide.


I'll find one or if I can't find a good one, I'll just write one.


Describing exactly what you need to do in order to get to this point that you can see


on the screen now, which is the node.js command prompt.


And on Windows, you can also run this from command, from the command line or from your


bash instance if you've installed everything properly.


And I think it's pretty much the same.


It's probably a lot easier on Linux or Mac.


So now that we're here, I'm going to switch across into the directory that I want to create


the front end application.


So I'm going to be using two different directories, one for the back end, one for the front end.


And we're going to navigate into the directory that we want to put our front end directory.


So it will be documents projects for me.


In here, I'm going to create a new directory called Aurelius front end.


Now I'm not going to do that the usual way with make dirt.


Instead what we're going to do is initialize our application, which will also initialize


our directory using the Angular CLI.


Now anything Angular CLI begins with ng, which is just Angular.


Ng is sort of short for Angular.


And what I want to do is create a new application.


I'm going to call it Aurelius front end.


And then we're going to get a load of questions here.


I'm going to say yes to all of them.


And then when it asks us what style sheet format we'd like to use, we're going to be


using material, Angular material here.


And if we want to later on modify the color theme with Angular material, we really want


to be using SCSS.


So we'll just start straight away with that.


And as well SCSS, we can just write normal CSS and it will work as it normally would.


So there's really nothing to lose.


OK, and that's just going to print loads of stuff to the screen and hopefully tell us


that everything's good.


OK, so that's all ready.


So what I'm going to do now is head over to VS Code and open this directory.


OK, so now we're in VS Code, opened the new Aurelius front end directory.


And first thing I want to do is open our terminal.


And all I want to do is ng serve to make sure that everything is set up correctly.


So this will initialize an instance of our app, which we can then open up in our browser.


So it might take a little bit of time the first go.


It should be pretty quick.


OK, it's cool.


Now we go to our browser, localhost 4200.


And we should get this.


So this is like the initial screen that you get with any Angular app.


So that just means that it works.


So that's good.


And what we're going to do is we're going to take this down here to start initializing


different parts of our app.


After all, we're not actually going to do that right now.


Because the first thing I want to do is actually just get it set up and get Angular Material


set up because Angular Material is going to be essentially our design of the entire app,


which is really easy to use.


But it can be really annoying if you don't set up first thing.


So I'm going to stop that.


And I'm going to add Angular Material to our application.


To do that, we do ng add Angular Material.


And this will install Angular Material in our app.


And we don't really have to do anything.


The only thing we actually have to do here, so we have to say which theme we'd like to


use.


We're going to go with Indigo Pink.


So these are the default colors that you can use.


And I think we're probably going to change these later on anyway.


But let's see.


For now, at least we're going to say Indigo Pink.


Here we say yes, yes, and let install.


That's installed now.


So if we head over to src app and app module TS, we should be able to see everything in


here.


So there are a couple of things that we installed just now, like extra things.


And that was the browser animations.


And that's what you see here.


Okay.


Now when we're using Angular Material, we tend to add another module.


So Angular has all these different modules.


And this is like the almost like the control panel for where we import things.


So we import different packages into this script, and then that allows us to use it


across all of our other web page files.


And what we can do is create two of these scripts so that we have two versions of it.


One of them is just going to be used for our material, Angular Material imports, which


is anything to do with how our app looks and essentially works when we're clicking around.


The reason we do that is because it can just get really messy if you have everything in


one.


So to do that, all we do is add material module TS.


And because this is like a duplicate of this package, all I do is take that code and just


modify it a little bit.


So in here, we basically want to get rid of all of these imports except for an ngModule.


And we can also get rid of declarations.


So we just have imports.


And we also want to add an exports array as well.


So this is like whenever we import a certain module from material, Angular Material, so


things like buttons, menus, all these sort of things, we're going to import it at the


top.


And I'm going to show you how in just a moment.


And then we're going to add it to the imports array here and also the exports array.


And that will allow us to use it in the rest of our app.


Now the only thing we need to change here is modify this to be material module.


And let's go ahead and import our first component, which is going to be mat button module.


And we're importing that from at Angular Material.


So this is the package that we installed before.


And Kite is finding the right directory there, so it's material button.


We'll copy this.


And we remove these because we're not importing these in this file.


Just add it to mat button module and also the exports.


So now we can use this module in the rest of our code.


And this is a material button, so it's like a fancy looking button.


So now let's exit that and go over to our app module TS.


So this is like the mothership of all of our code.


Everything goes through this, including that other module, even though it's like a duplicate.


So to include that, we do what we've done here.


So you see we have this browser animations module.


We're going to do that same thing.


We have it there.


We also have it in this imports array.


So to import that module that we just made.


So if I have a look here, we see that it's called material module.


So let's copy that.


Send it over here because you see over here we're exporting it to the rest of the code.


So that means it can be read by this app module TS.


So import it.


And we do from.


And this will probably figure out maybe, maybe not material material module there.


OK, so that is now imported that into this script.


And what we want to do is include it in the import array there.


Now it's accessible by the rest of our code.


So that's I know it's quite a lot.


But let's material basically set up and just so we can see that's actually working.


Let's just add that material button into the very front page of this app.


So the very front page of this app is app component HTML.


So you saw before we had that kind of fancy initial page where it gave us a few hints


on what we should do next.


And that is everything that you see in here.


So we're going to just select all and delete this.


And all I want to add in here is a button and it's a material button.


So mat button like this.


And we're going to put search.


OK now if we go back into our browser let's make sure it's I don't think it's running.


I think we have to restart here.


So ng serve again.


Spin up the server.


If we head over.


So this should go in a moment and we'll just get a button in the top left.


OK.


So now we have our button and you see we get this kind of nice little we go over it gets


highlighted and when we click it we have that nice ripple effect.


So that's our material button.


And I think that's it for this first video on the front end.


So we've initialize our front end and also included material or angular material in there


as well.


Now from now on it's I think we will probably just build how this front page looks a little


bit to make it look nice.


And then we won't get crazy by the way we'll just make it look bearable and then we'll


add in functionality so that we can search on here rather than through the API.


So it should be pretty cool I think.


Building Transformer Tokenizers (Dhivehi NLP #1)

Today, we're going to have a look at how we can design tokenizers or more specifically


a BERT tokenizer for low resource languages.


So when I say low resource language, I mean a language where we don't really have that


much data and there is not already a tokenizer or transformer model out there built for that


specific language.


Now there are transform models and tokenizers for a lot or even most languages, but there's


still a lot of languages that are, well, simply less common and get less attention than something


like English or Chinese.


So you may find if you're working one of these languages, there are very few models out there


or maybe no models out there.


So that's what we're going to focus on today.


We are going to more specifically focus on building a tokenizer and in the future we're


going to do more of this as well, more models and so on.


We're going to focus on building a tokenizer for the Devei language.


Now Devei is the language of the Maldives.


So the Maldives islands and the Indian ocean, I'm sure a lot of you probably know them,


but they're very beautiful islands, incredibly amazing weather.


And generally I would not usually think of the Maldives and NLP in the same context,


but there are a lot of people that do.


And in particular, there's a guy called Asherak or Ismail Asherak.


And we've been speaking a lot about NLP and through that I've been made aware of how difficult


it is to actually build models for languages where there is not much attention out there


already.


So what we decided to do is kind of team up and try and put together some transform models


for the Devei language.


And the first step to that is obviously the tokenizer.


You need tokenizer before you build anything else.


So there are a few hurdles that we kind of need to overcome when we're building these


models.


So there are some pre-trained models for Devei, but they're not necessarily that useful for


what we want to apply them to.


And you may think, okay, there's those multilingual models with hundreds of languages included.


Now from at least what I've seen, all of them miss Devei.


Another thing that is difficult with Devei and other low resource languages is actually


finding text data.


Now labeled data is practically impossible for a lot of these languages, but unlabeled


unstructured data, maybe that's a bit more reasonable.


You can use the web scraper and scrape loads of text from whichever language you are using,


the websites in that language on the internet.


And that is what Ashrack has done to get the data set that we're using today.


And as well as that, another really difficult thing is that Devei uses a very unique writing


system that is known as Thana.


And Thana looks really cool and is definitely very unique and beautiful, but it's not really


very well supported even by those multilingual language models.


As you can see here, it looks really cool.


We're also going from right to left, which is quite interesting.


Everything's also in the same case.


There's no uppercase or lowercase, which is again, quite interesting, I think.


So there's a lot of unique properties with this language that we need to deal with.


And from what I've seen, there are no current tokenizers which actually support these characters.


So we really do need a tokenizer.


First step is getting data.


So like I said before, Ashrack went and did the hard part and took or scraped a load of


Devei text from the Devei internet.


And he managed to put together 16 or more than 16 million samples of Devei paragraphs


and sentences.


Now, obviously, there are going to be little things in there that we maybe don't want,


but for the most part, it's actually very good.


So step number one is downloading the Devei corpus from Ashrack.


And you can see here, so it's hosted over here on Hugging Face.


So we can use the data sets library to go ahead and actually get that data, which is


pretty useful.


So we just do pip install if you don't have it already.


Data sets.


OK, now, once that's installed, we go ahead and we just write this.


So we are going to Ashrack's Devei corpus and we are taking the train data split from


that corpus.


Now it's fairly big, like I said, it's more than 16 million samples.


So what you can do if you don't want to load it all at once, you can set streaming equals


true and what that will do is create more iterable objects, which you can see down here.


And that means we can form a for loop and iterate through the data one sample at a time


and not download the full thing in one go.


We just download each sample as we look through the whole list.


So you can see here that's what I've done.


So the row here in the train, so going through the train split, we're just printing one at


a time.


And again, you can see that really cool Devei text in here.


Now there's just three samples here, so it's very simple.


We just have almost like a dictionary with text and then we have the text following it.


So that is it.


All I've done here, so I'm creating a generator object because later on when we are loading


this data into the tokenizer, we're going to be loading it in as a generator and that


generator will expect us to load the text in almost like lines.


So we'll expect to iterate through and just receive the text.


It will not expect to receive a dictionary which contains the key text which actually


reads the text.


It expects just this here.


OK, nothing else.


So I've done is create the generator object here.


And you don't need numerate much for why that's there.


And all this does is iterate through each row and extract text from each row.


So it's literally going through and extracting this and nothing else.


So now when we actually use that, that's what we get.


So here we're using this Devei text function or generator.


We are using the exact same code we did before.


We're just printing the row.


You see now we don't get dictionary, we get just the plain text which is what we want.


Now that will be fed into our tokenizer train from iterating method.


Now that's kind of skipping ahead a bit.


So let's go back and take a look at where this tokenizer is coming from in the first


place.


So we have in our tokenizer, we have a few different sets.


So let's say we have the tokenizer.


It's not just a tokenizer.


There are different components or subprocessors.


The very first is the normalization.


Now this is optional.


You don't need to do this.


But it's useful.


So given maybe we have something like we have these two words.


So this is specifically European languages here.


But we have C and C. Now these are pretty similar and a lot of people may actually use


these interchangeably although as far as I know are actually different words.


But people may use them interchangeably.


And if we would expect that then we can use something called NFKD, Unicode Normalization


to actually take both of these and create the same word which would just be SI.


And this is particularly useful if you have weird characters.


So for example rather than the letter F like this, you see people on social media using


like the weird ones.


Not like kind of like this.


It's all these weird characters and stuff.


But basically what they mean is an F. And because we're people we can understand that's


what they're talking about.


But a machine learning model is not going to see those as the same thing because they


have different Unicode patterns.


So what we do is use NFKD, Unicode Normalization and we convert them to being the same thing.


So that can be quite useful.


Another very useful thing is you can also lowercase within your normalization step.


So that's also useful now like I said before with DeValue you don't have uppercase and


lowercase so it's not really necessary for that but if we do have for example English


text in our data at any point it's probably better we normalize it or lowercase it because


that means we will create less non-DeValue tokens.


So for example we have the word hello and we also have the word hello like this.


Now without lowercasing these may become two separate tokens and take up more space in


our tokenizer vocabulary where we only really want to keep DeValue characters or words.


So what we can do is lowercase that and that reduces the likelihood of getting duplicate,


lowercase and uppercase non-DeValue characters in there.


So that's another thing we will do.


So let me remove all of this.


So we are going to lowercase and also use NFKD.


So that's the first component.


The next component after that is pre-tokenization.


Now pre-tokenization is a tokenization step that happens before we actually tokenize.


So what I mean by that is pre-tokenized let's say we have a string hello world.


It's probably easiest.


Now what this is going to do is just split this into very simple tokens.


So we might say anything with a space will get split or anything with space in between


it will get split and maybe punctuation as well will get split.


So then we would end up with the token hello.


We get a list from this.


We get hello.


We get world and we also get this summation model.


It's messy but I think that makes sense.


So that is the pre-tokenization step and then after that we actually have the model for


the tokenizer itself.


Now right in model.


Now this is where you have something like a word piece tokenizer which is what we're


going to use.


Now word piece is the tokenizer that BERT uses and later on we're going to be using


BERT models so that's why I'm stuck with word piece.


And what this will do is it will take all of the tokens we've created and it will merge


them into the most or the largest logical components that it can think of.


So this isn't a very good example of the hello world but let's say we have being maybe.


Yeah being we can do that.


So being for BERT specifically this would probably get split into something like a B


and then a sub-word token which is always prefixed by the double hashtag symbol and


that would be ING.


So we get two tokens from that because it's a BERT or the BERT word piece tokenizer sub-word


tokenizer so it doesn't split every word like hello probably just be hello but it will split


some words like it seems being more relevant if you split them which can be can be useful.


For example if you had like it snowed it is snowing and snow or snowboarding you would


get snow in all of those and then you get boarding or board and ING and you get ING for


snowing and snowed so that can be quite useful to find patterns in the words or sub-words.


Next up is post-processing so post-processing and this is where we would add any special


tokens to our text so BERT specifically would use something like a what's called a classify


token followed by hello world followed by a separate token and you probably have padding


tokens and all of these different things so we're adding in any special tokens and we're


also going to create different tensors so we will have an input IDs tensor which will


be the ID or integer values that represent each one of these word tokens or sub-word


tokens. We have token type IDs which is usually when you have sentence pairs and you will


also have a attention mask tensor which tells BERT or the transform model which tokens to


actually pay attention to for example ignore any padding tokens. So I mean they're the


main components of a tokenizer but there is also another which we will I think almost


always add at least which is a decoder so when your model outputs let's say it outputs


a word prediction so it's predicting you've masked the word you know let's say here we've


covered hello and we said BERT what is that word and BERT is trying to tell us hello but


BERT doesn't know the string hello it just knows token ID value so it's going to give


us like it's been safe. Yeah this is my prediction is the number three and obviously we're like


okay great I don't know what number three means so we need a decoder to take that number


three and translate that into something more readable for us which would be hello okay


and that's that's what a decoder is for so obviously we don't need it for the input into


BERT but we do need it if we want to understand the output from BERT. So that's our tokenizer


components reasonably high level but let's let's go into the code and we'll have a look


at how this how we've implemented that for our BERT workpiece tokenizer. So in this we


are using the Hugging Face tokenizers library which is is very good and I would definitely


recommend you do the same otherwise this will be very difficult. Now to install this you


would just pick install tokenizers there's nothing more to it than that and what I've


done here is imported everything that we're going to be using. Now you can see a few of


the components I mentioned earlier so we have we have decoders the models which is the tokenizer


itself normalizes the pre-tokenizers and the post processes as well. Further on we have


these other two classes and we'll go we'll get to them soon so I'll explain them later.


Now first thing we want to do is actually we're not going one two three four like I


said before because then we would be initializing the normalization step first. Instead what


we actually do is we initialize tokenizer and the main component of that tokenizer is


the model so in this case word piece. So we initialize that using tokenizer here so you


see up here we have tokenizer this is just a wrapper so that we create a hud and face


tokenizers object and in there we need to pass the type of tokenizer that we're using


and we are taking that from models up here and we're using word piece. Now in here we


also have the unknown token so you specify that so that's where your tokenizer if it


sees a word it doesn't know it will place this okay it will put new nk rather than just


leaving it empty or not understanding this is like the only thing it can put there instead


of raising an error or something so we put that in there. Now after we have initialized


that tokenizer instance we need to set the tokenizer attributes which are going to relate


to the components that we listed just a moment ago. So here we set the normalizer attribute


in our tokenizer which we just initialized and we're setting that equal to normalizers


sequence so there's a sequence of normalization sets here and we are using the lowercase first


we locate everything and then we are using nfkd unicode normalization. So there is for


that so we've set normalization set then we do the pre tokenization set now pre tokenization


is where we're splitting our string into tokens so words or punctuation. Now the simplest


way to do this is we use this whitespace tokenizer so all that does is splits on either whitespace


or on punctuation like commas or exclamation marks or something like that. So that's it


we've set out pre tokenizer and then we have this so this was another part that we didn't


mention so we imported this trainers so we imported it here now trainers is basically


the method that or function that we'll be using to train or control the training process


of our tokenizer and all we're doing here is we're using a word piece trainer we have


a word piece tokenizer model we are setting the vocabulary size so that's the maximum


number of tokens that our tokenizer will contain and then we also need to set any special tokens


so we already set one earlier but we need to make sure that is included so with special


tokens we probably won't see them or we hopefully won't see them in our input text so or the


text that we'll be training on so what we do is we insert them now beforehand okay so


we have this unknown token padding token classified token separated token and masking token then


here we set the min frequency so the minimum number of times we need to see a token to


add it to the vocabulary and then we also earlier on you saw we had the sub words in


our word piece tokenizer here we're just setting the prefix identified those so it's just a


to symbols there now at this point we're back to the start we're downloading the debate


corpus again we already covered it so we can skip that we already have the debate corpus


and we are storing it in the generator db text and then that's where we get onto our


training step so the reason that we put db or we create this db text generator is because


we're training from iterator here which expects to receive lines of text so we have db text


and then the trainer is equal to the trainer so the trainer we define just up here so that


controls the training process and that's it so this once you run that that will run through


all of the text in your db text generator and train to the specifications you set in


your trainer object we can check so the tokenizer you can get the vocab size we set that earlier


and you can see this is at 30,000 so as we as we would expect now before when I showed


you that list there were five steps and we've covered one which is normalization two pre


tokenization and three which would be our actual tokenization model now after that we


we train the tokenizer so we train using those first three steps or components and then we


still need to define the next step which is the post processing step now the first thing


we do is we get our classifier id so the the integer value that represents cls and the


integer value that represents scp we get both of those and we use this processing or processes


template processing method to create this here which is looks quite messy but if you


take a look at this here so we have the one input at the top and we have the two inputs


at the bottom so when you are feeding just a single sentence into your BERT model what's


going to happen is looking at code we are going to we are going to use this format so


we need to have our cls token followed by a which is sent into one followed by the separated


token now back to the image we have on the bottom we have two sentences now if we feed


those into BERT they need to be understood as separate sentences to BERT or separate


maybe question and context for example now in that case back to our code we would use


or we would format it as a cls token followed by sentence a first one followed by a separated


token separate the two sentences and finally we would have sentence b after that and then


again we do finish with a separated token again now another thing that's different here


is that we have this colon one for the pair now that's for the token type ids array so


token type ids tells us where we have sentence a and sentence b now in the case of single


sentence everything in token type ids is equal to zero so it's just all zero zero and zero


the alternative where we have two sentences is that sentence a is going to be zero and


everything related to sentence b is going to be one it's as simple as that and then


we need to specify the special tokens that we're using here so we have cls and we're


specifying it again here and mapping that to the actual token id integer so we that's


the post processing step that's all sorted we don't need to do anything else with that


now and we can move on to so after this our tokenizer will feed the input ids that it


decorates into the model and then we would move on to the decoder step so after Bert


has finished processing whatever it's processing it's going to output you a number like a word


prediction maybe it's going to give you a token id and we need to know okay how do


we how do we decode that in something we can understand and maybe it gives us a load of


token ids and what we need to do is say okay we're using a word piece tokenizer and we're


going to decode from word piece and we also because we're using word piece as this prefix


this is already set by default by the way but i'm just putting in there in case you


want to use a different prefix although i wouldn't recommend it but if you if you have


a reason to you can change that so that's our tokenizer and that that's it so we've


initialized or we've created our tokenizer and after that we move on to saving it into


a format that is most useful to us so i think most of us when we are using a tokenizer model


we're probably going to use hugon base transformers not home base tokenizers because they are


two different libraries and when we load a tokenizer with hugon base transformers we


tend to use this pre-trained tokenizer or pre-trained tokenizer fast class now what


we can do is save our tokenizer into the format that is compatible with hugon base transformers


and compatible with this class specifically and to do that we first actually initialize


the tokenizer the hugon base tokenizer tokenizer using this class so we are using pre-trained


tokenizer fast and if you've used this before in our base transformers we usually write


from pre-trained and we load a model from here so like based on case at this time we're


not using that not using any methods we're just initializing the object directly we pass


the tokenizer to tokenize the object then we also have the unknown token padding token


and the other special tokens in there as well okay so that is ready for us to save it's


in the correct format now so what we do is take full tokenizer which we've initialized


here and we save it as a pre-trained model and what i'm doing here is saving it to the


BERT base db directory and once that's saved we're going to save these three files here


and from there we can actually just load it from pre-trained like we normally would now


just one thing when we load from pre-trained normally we probably are going to be loading


from the hugging base models hub in this case we're not loading from a local directory so


in some cases maybe you have this in a different directory so you would write something like


your path here which would go to a different directory and then you would have your model


directory there okay so just be aware of that and then we can test test our tokenizer with


some debate text and we see okay cool this is the very text no idea what any of it means


but it looks great and from that we get three tensors so we have the impact these tensors


which you know two represents i think this cls token and three here would represent our


separate a token the two special tokens everything in between is the va or the punctuation like


these brackets open closing brackets this comma or something else and then we have token


type ids now we only have one sentence here so just sentence a all of that should be zero


and then we have the attention mass tensor as well in this case we don't have any padding


that i've been padding so the attention mass should just be one and that's it our tokenizer


is actually ready now if you do want to go ahead and actually load this tokenizer directly


rather than going through all of this you can just write james callum and load it like


this because this is the under your hugging face models hub we we have a model and this


also includes a bird model as well so you have the bird model which can load and also


the tokenizer so that is it for this guide or walk through to building a bird word piece


tokenizer for a loan resource or a language which does not have any currently supported


tokenizer so i hope this has been useful as i said the tokenizer is really just the first


step in in what we hope will be a greater way to support the debate language and particularly


the ai community over there in what they are building and doing by putting together a few


bird models that are fine tuned or built for specific or different purposes and like this


is the very first step in that so i hope it's all been very useful thank you very much for


🤗 Hugging Face just released *Diffusers* - for models like DALL-E 2 and Imagen!

Today we're going to have a look at a new library from Huginface called Diffusers.


And it covers what are called Diffuser Models.


Now Diffuser Models, if you don't know what they are, you've probably heard of a few of them already.


They are like OpenAI's DALI2 is a Diffuser Model.


Google's Imogen, MidJourney's Image Generation Model as well.


There are a lot of these Diffuser Models and they are producing pretty insane things.


If you know what GANs are, you can kind of think of them as doing that same thing of generating


something. In the case of the three models I just mentioned, it's generating images.


But they can also generate other things as well. They can generate audio, video


and I imagine probably a lot more.


So they are pretty cool and also pretty new.


But already, you know, everyone pretty much everyone knows about DALI2


and the other Diffuser Models as well.


They're already making a big impact.


So Huginface's decision to create a library to support this in a more open source way


in contrary to the OpenAI, Google and MidJourney approach where they have everything behind closed


doors, which fair enough, they're probably massive models and most of us can't run them anyway.


This decision by Huginface I think is a pretty good one.


It means that normal people like me and you can actually start playing around with these models,


which is really cool.


So let's have a look at what we can do with the first version of this library.


So to install the library, we just pip install diffusers.


You may also need to add transformers onto the end there.


I'm not sure. I already had transformers installed, so it might not be necessary.


And yeah, let's just have a look at this first example.


So I've taken this example from the GitHub repo,


modified it a little bit and played around with the prompts.


But all we're doing is taking this Convice text to image model.


So this is a diffusion model, diffuser,


and we're just creating this diffusion pipeline using this model ID.


And the example they gave was something like a painting of a squirrel eating


maybe a mango or something along those lines.


I don't remember.


So I just I want a banana just to see what happens.


And we get this pretty cartoony image here of a...


I mean, I suppose it's not necessarily eating the banana,


but it's definitely a squirrel with a banana.


So with just a few lines of code here, we got that, which is pretty cool.


So yeah, already something to be pretty impressed with.


So that's cool.


But obviously I wanted to play around with these prompts


and just see what it can do.


Now, just to be very clear, this is a very small and basic model.


So we're not going to get like Dali 2 standard images from this


or Imogen standard images from this, which is to be expected.


But nonetheless, we can play around with it and we can just do it.


This is on, I ran all this on my MacBook Pro.


So yeah, it didn't even take long.


It may be a minute or a couple of minutes for each image at most.


So with this one, again, it's the same prompt again.


All I did was change the number of inference steps.


So I played around with this a little bit.


I'm not familiar with the Fuser models.


So I don't know the best approach


dealing with different parameters here.


But number of inference steps, you can increase or decrease that.


And then what I was probably more interested in here


was like the prompt engineering side of things.


So how can I modify the prompt to make it more like what I want it to be?


And I see a lot of people saying, well, I'll show you in a moment.


But I started off with this photorealistic image


of a squirrel eating a banana.


And yeah, you can see straight away,


like there's a bit more detail in this image.


The banana and the squirrel kind of like melded together here.


It's a bit weird.


But it's a kind of more realistic image.


And I noticed there's also this reflection down here


from the banana, which is kind of cool.


Yeah, and yeah, with this prompt engineering stuff,


I see people adding things like in 4K


to try and make something like higher quality.


Now this is with like Dali 2.


So it's not necessarily the same with this model.


But I thought I'd give it a go anyway.


We get this and this is like,


it seems to be trying to pull in more detail around here.


You know, I wouldn't say it's not a good idea.


You know, I wouldn't say it necessarily works.


But it's kind of cool.


It's kind of weird that has these two eyes staring at us here.


But nonetheless, interesting.


So this, I thought I'd go away from the squirrel examples


and go for an Italian person


eating pizza on top of the Colosseum in Rome.


And we get this, which is pretty good actually.


It's not on top of the Colosseum.


But I thought this was pretty cool anyway.


It looks a lot like the Colosseum.


And he looks relatively Italian.


Some interesting sunglasses here as well.


And the pizza is a bit strange, but yeah, pretty cool.


Here, photorealistic image of a squirrel eating a banana


rendered in Unity.


This is taking inspiration from what I've seen people do


with OpenAI's Dali here.


And yeah, we get this.


I assume they must have trained on a lot of stock images


because I got this a few times.


You can kind of see the overlay,


like the watermark from the stock image here.


And then like down here, I think that's pretty...


Like you usually have some information


about the stock image at the bottom


or the company that owns the stock image.


So I thought that was kind of funny that comes through.


It came through on a fair few images.


Not a lot, but a few.


I thought I'd try it from real engine as well.


Yeah, similar sort of thing.


So one thing that I have noticed with this


is that it seems to be able to...


It's like here I put a giant squirrel destroying a city,


which is a pretty abstract thing.


There's probably not any pictures of that


that it has seen in the past.


And I think when there's no pictures of that,


of something that it's seen in the past,


it's true it's put two concepts together,


at least this model.


So yeah, I kind of got this.


I think this is the best image


and I tried a lot of prompts with this.


So we've got this squirrel here


and he's on some kind of blocks.


It looks a bit like C.


That one, okay, it's not so bad.


And then there was this one as well.


It kind of looks like a city in the background.


I thought that was pretty cool.


But then the remaining ones here,


just like a squirrel or a squirrel thing


in a natural environment.


Or this one.


So here I'm playing around the inference steps.


I just turned it right down


and you just kind of get this interesting.


Again, I don't know anything about diffusion models.


So I don't know why it's doing that.


I assume it's just, well,


I know that it's generating the image from noise.


So I assume this needs a few inference sets


in order to take that noise


and create an image out of it.


So at this point,


it's still kind of stuck halfway


in between the noise and an actual image, I assume.


Yeah, we kept going through these.


Just a squirrel, squirrel again.


And then I eventually got this image


of a squirrel thing inside what seems to be a city.


Yeah.


And then I got a picture of an actual city.


So it seems, yeah, it knows what a squirrel is.


It knows what a city is.


But putting those two things together,


particularly a giant squirrel,


doesn't seem to do so well with that.


Here I modified it a little bit.


So a landscape image.


And then the same prompt,


slightly different number of inference sets.


No, no, actually the same.


So I just rerun this same prompt twice,


same parameters,


and it just got like a weird squirrel again.


Kind of cool with this,


like the grass detail in front here though.


Anyway, yeah, that's that one example.


I want to tell you something else.


One example, I wanted to have another go


with some other models as well.


These are less impressive, I'll be honest.


But, you know, this is literally,


I feel like they released this library


like a week ago maybe at the time of me recording this.


So it's really new.


And there's all these different pipelines by the way.


As far as I understand this,


this pipeline uses a different something called a scheduler,


which is like the algorithm that denoises the noise,


the input noise that the image is generated from.


Don't quote me on that.


I think that is what these different pipelines are doing.


But I'm really not sure.


It's something along those lines.


Okay, so with this one,


this is like a Pokemon trained by this guy here.


I think he's called Manuel something.


He has a load of models that he's built through Hugging Face.


And this is supposed to be a Pokemon generating image,


image generation model.


So yeah, I mean, you can kind of see


it definitely has that Pokemon style to it.


But like, it's not, yeah, it's not really a good,


I wouldn't say it's a good Pokemon.


It's kind of messed up.


And then I thought I'd try the other pipeline.


So I read on the Hugging Face Diffusers GitHub repo that,


and I don't know if this is completely accurate or not,


but this DDIMP or DDIM,


our algorithm is supposed to be slower.


Yes, slower, but more accurate


or can produce better quality images than the DDPM pipeline.


But that's not always true.


And in some case, in a lot of cases,


I imagine these different models have been trained


with one of these algorithms in mind.


And you can't always, for some models you can,


but you can't always switch between the two algorithms


and get them to work with a particular model.


That's how I've understood it.


Again, could be wrong, I don't know.


But it's definitely faster.


This took me like two minutes, 30.


This is running for a thousand inference steps


by default.


I'll show you, we can change that in a moment.


The same here was seven seconds,


but then we just got noise.


So I thought, okay, maybe it just needs to be ran for more steps.


Actually, sorry.


This was run for 50 inference steps by default, not a thousand.


And the same for this one here.


And then here I turned that up to a thousand


just to see if it would produce anything


because I expected that maybe if I run it


for more inference steps, it might do something.


But no, unfortunately not.


So clearly this model can't,


we can't use different algorithms


or denoising algorithms, maybe that's what they're called.


I don't know.


With that model.


So then I thought, okay, let's try a different model.


This is from Google.


Google have put out a load of these diffusion models


in the last couple of days on Hugging Face, which is pretty cool.


And obviously this, you need to use a DDPM pipeline.


You see in the model name or model ID.


And yeah, this one takes ages to run.


At least on my M1 Max without MPS,


it was about 25 minutes, really long.


And then I got this.


So it's a bit disappointing,


but you can kind of see there's a sort of a cat face in there.


But there's definitely the feeling of a cat in this image.


I wouldn't say that is actually a cat in this image though.


And the same for this one.


You can kind of see a cat over here.


Yeah, interesting.


And there's, you know, again, the feeling of a cat in this image.


But there's definitely not a cat.


Yeah, maybe there's a cat, but it doesn't look very healthy.


And then the last thing I just had a look at here is this config.


So the model or pipelines here,


they are set up using this configuration dictionary.


And you can modify different parts of your pipeline


by changing this config and loading your pipeline with a different config.


Similar to the configs in the Transformers library,


where you have like a BERT config and you have a BERT config.


And you load that into your BERT model.


I assume they're going for a similar sort of thing here.


So I thought I'd just point that out.


But yeah, it's literally very early days with this library.


I haven't really been through anything.


I've just, you know, taken a very high level look at the library


and played around with a few image generation pieces.


So I hope this is interesting to see.


I'm pretty excited for this library. I definitely want to play with it a lot more in the future.


Very soon I'm also going to be having a look at some other Diffuser models.


So hopefully I will understand them better in the very near future.


So thank you very much for watching.


I hope this video has been interesting.


And I will see you again in the next one.


Adding New Doc Stores to Haystack

Today we're going to do something a little bit different and


Adding New Doc Stores to Haystack

We're going to have a look at the project that I'm currently working on now


Adding New Doc Stores to Haystack

this project is a


Adding New Doc Stores to Haystack

Very high level and if you know the Haystack framework, it's to implement a new document store within that framework


Adding New Doc Stores to Haystack

If you don't know it, no worries. I'm going to explain it


Adding New Doc Stores to Haystack

so I


Adding New Doc Stores to Haystack

Left a couple of notes which you can see here to remind myself what I actually want to talk about in this video


Adding New Doc Stores to Haystack

So to start I want to just give you know an overview of what is ODQA or open domain question answering


Adding New Doc Stores to Haystack

And the three components and that you probably if you watch a couple of my videos


Adding New Doc Stores to Haystack

You've definitely seen me talk about this before


Adding New Doc Stores to Haystack

But for those of you that don't know it comes very quickly go over that


Adding New Doc Stores to Haystack

And I'm going to explain how Haystack fits into that


Adding New Doc Stores to Haystack

So if you already know what Haystack is and you're already know what open domain question answering is you


Adding New Doc Stores to Haystack

Probably just want to skip ahead a little bit. It's up to you


Adding New Doc Stores to Haystack

I'll leave some chapter markers in the in the timelines that you can skip ahead if you want


Adding New Doc Stores to Haystack

And then we're going to have a look at the current document stores


Adding New Doc Stores to Haystack

within the


Adding New Doc Stores to Haystack

Haystack framework


Adding New Doc Stores to Haystack

and


Adding New Doc Stores to Haystack

Why I'm going to implement a pinecone document store and then we're going to have a look so this is probably kind of


Adding New Doc Stores to Haystack

relevant especially if you want to


Adding New Doc Stores to Haystack

contribute to open source


Adding New Doc Stores to Haystack

I'm going to just go through


Adding New Doc Stores to Haystack

set up my own


Adding New Doc Stores to Haystack

Git repo for this and and set it up so that it


Adding New Doc Stores to Haystack

works with the other official Haystack Git repo as well


Adding New Doc Stores to Haystack

so I'll show you how you can do that and how you can set up your Git for open source projects when you're


Adding New Doc Stores to Haystack

planning on contributing and


Adding New Doc Stores to Haystack

Then we'll have a look at the document stores or the specific code in document stores


Adding New Doc Stores to Haystack

In the Haystack


Adding New Doc Stores to Haystack

library, so


Adding New Doc Stores to Haystack

Yeah, I think it's a it's a different video, but should be should be interesting. Oh


Adding New Doc Stores to Haystack

and one last thing actually if you do want to


Adding New Doc Stores to Haystack

Test document store go ahead and just you can get cloned from here


Adding New Doc Stores to Haystack

to your own


Adding New Doc Stores to Haystack

local machine


Adding New Doc Stores to Haystack

so I'm going to


Adding New Doc Stores to Haystack

Get cloned the above okay


Adding New Doc Stores to Haystack

And then you want to


Adding New Doc Stores to Haystack

Do within that same repository, so let's say that downloads it to your documents Haystack folder


Adding New Doc Stores to Haystack

Navigate to that folder in your terminal window or command line


Adding New Doc Stores to Haystack

And then you just need to write pip install


Adding New Doc Stores to Haystack

dot right and


Adding New Doc Stores to Haystack

That will


Adding New Doc Stores to Haystack

Install this version of Haystack and then you can test the pine cone document store now. I'll show you


Adding New Doc Stores to Haystack

How you can start doing that?


Adding New Doc Stores to Haystack

so if you if you do want to test that and maybe even if you want to contribute and


Adding New Doc Stores to Haystack

You see some like terrible code that I've written you want to make it more efficient. Please do go go ahead and do that


Adding New Doc Stores to Haystack

So


Adding New Doc Stores to Haystack

Let's start with a quick overview of open domain question answering and the three components that we see in there


Adding New Doc Stores to Haystack

So I'm going to assume you've already seen some of my videos on this


Adding New Doc Stores to Haystack

Or that you're aware of Haystack and open domain question answering so I'm gonna be really quick


Adding New Doc Stores to Haystack

So open domain question answering or ODQA


Adding New Doc Stores to Haystack

It's basically you have


Adding New Doc Stores to Haystack

load of text data or some other data in a database somewhere and


Adding New Doc Stores to Haystack

And let's say Wikipedia. That's a good example


Adding New Doc Stores to Haystack

so we have our Wikipedia articles over here called Wiki and


Adding New Doc Stores to Haystack

We ask a question so we have let's say like a search bar like Google


Adding New Doc Stores to Haystack

Over here, and we ask question like who are the Normans?


Adding New Doc Stores to Haystack

Question mark and then we press search


Adding New Doc Stores to Haystack

right


Adding New Doc Stores to Haystack

We have in Wikipedia we have the answer to that again the Normans are


Adding New Doc Stores to Haystack

people from Normandy in the north of


Adding New Doc Stores to Haystack

France but we need a way of translating that question into


Adding New Doc Stores to Haystack

That specific little part of


Adding New Doc Stores to Haystack

Information and pulling that from our larger database this can contain millions or more of


Adding New Doc Stores to Haystack

Paragraphs of information so how do we pull the right bit of information?


Adding New Doc Stores to Haystack

So this query gets converted into a vector using what is called a retriever model


Adding New Doc Stores to Haystack

So we go into our retriever model


Adding New Doc Stores to Haystack

It's converted into a vector all of these Wikipedia snippets are also vectors


Adding New Doc Stores to Haystack

So we go to our database. This is a vector database over here and


Adding New Doc Stores to Haystack

We compare our query vector which will call execute with all of our context vectors


Adding New Doc Stores to Haystack

And that will return a set of relevant


Adding New Doc Stores to Haystack

Context vectors so we'll call them XC and


Adding New Doc Stores to Haystack

These context vectors are quite big they're like paragraphs now. We don't need a paragraph of text tellers


Adding New Doc Stores to Haystack

So we pass this into what's called a reader model


Adding New Doc Stores to Haystack

Over here


Adding New Doc Stores to Haystack

Also passing in our question and that reader model is going to take our context


Adding New Doc Stores to Haystack

So let's say this is one of them. It's really long piece of text


Adding New Doc Stores to Haystack

And it's gonna say okay the specific answer is here. Okay, so it's gonna say


Adding New Doc Stores to Haystack

north of


Adding New Doc Stores to Haystack

France or Normandy, okay, and


Adding New Doc Stores to Haystack

That's open the main question answering super


Adding New Doc Stores to Haystack

You know very high level. It's how it works


Adding New Doc Stores to Haystack

Now how does haste that fit into that?


Adding New Doc Stores to Haystack

So I will show you it makes my sense to just go to their repo, okay?


Adding New Doc Stores to Haystack

And we actually have this little about here, it's nice open source NLP framework


Adding New Doc Stores to Haystack

uses transformer models. We all know what they are and


Adding New Doc Stores to Haystack

Allows us to


Adding New Doc Stores to Haystack

Implement production ready search question answering


Adding New Doc Stores to Haystack

Semantic document certain summarization. Okay, so it's quite a lot the main bit. We're focusing on at least


Adding New Doc Stores to Haystack

here is


Adding New Doc Stores to Haystack

this kind of like search and


Adding New Doc Stores to Haystack

Question answering so this is exactly what I just showed you. All right, so we're


Adding New Doc Stores to Haystack

Entering our question this produces a query back to a press search


Adding New Doc Stores to Haystack

And then we'll get these context which is a long chunks of text and then the answers which are the those little


Adding New Doc Stores to Haystack

Smaller snippets that you saw that highlighted


Adding New Doc Stores to Haystack

All right, so the retriever gets retriever and vector database gets those big paragraphs


Adding New Doc Stores to Haystack

The reader model gets a little answer


Adding New Doc Stores to Haystack

Okay


Adding New Doc Stores to Haystack

And haste is a very nice way to do that


Adding New Doc Stores to Haystack

and


Adding New Doc Stores to Haystack

haste is basically a way to do that and it's


Adding New Doc Stores to Haystack

incredibly easy and


Adding New Doc Stores to Haystack

And straightforward like you don't really need to know too much about opening question answering


Adding New Doc Stores to Haystack

Probably you never even need to know that it's open the main question answering and you can implement


Adding New Doc Stores to Haystack

This pipeline and get something that works really well. It's honestly a very good framework for this sort of thing


Adding New Doc Stores to Haystack

So I've been earlier on I mentioned this document sorting


Adding New Doc Stores to Haystack

document store is basically where you're storing the


Adding New Doc Stores to Haystack

The information so I'm earlier on I said vector database document stories


Adding New Doc Stores to Haystack

Basically, you can see them as being both the same thing just different names at the same thing


Adding New Doc Stores to Haystack

Now


Adding New Doc Stores to Haystack

We kind of have this


Adding New Doc Stores to Haystack

Overview here. We're not going to worry about indexing pipeline so much


Adding New Doc Stores to Haystack

And that's basically just you know you get your data and you convert it into a readable format for your document store


Adding New Doc Stores to Haystack

But we do have a document store and then we have to search pipeline


Adding New Doc Stores to Haystack

So we have the retriever and then the reader over here. We have a generator


Adding New Doc Stores to Haystack

I haven't mentioned that we're not going to talk about it here. We got answer


Adding New Doc Stores to Haystack

But let me show you an example so


Adding New Doc Stores to Haystack

We'll go to here better retrieval via DPR notebook and I scroll down a little bit


Adding New Doc Stores to Haystack

Okay, so here we're using FICE as a document store


Adding New Doc Stores to Haystack

Okay, so this is where we're going to store all of our information


Adding New Doc Stores to Haystack

Scroll down a little bit more


Adding New Doc Stores to Haystack

And over here we're pulling in these


Adding New Doc Stores to Haystack

These files which are is a wiki. I think it's the wiki for Game of Thrones


Adding New Doc Stores to Haystack

Like the fan wiki or something like that


Adding New Doc Stores to Haystack

It's just all the text from those. So you have loads of paragraphs talking about Game of Thrones and this random like trivia


Adding New Doc Stores to Haystack

about it


Adding New Doc Stores to Haystack

So once we


Adding New Doc Stores to Haystack

So we've got the data and we write that into our document store


Adding New Doc Stores to Haystack

now


Adding New Doc Stores to Haystack

We've written the text or document store at this point, but not the actual vectors


Adding New Doc Stores to Haystack

Because like I said, there's a vector database


Adding New Doc Stores to Haystack

behind this as well and


Adding New Doc Stores to Haystack

What has happened here is we're storing text of the context or that Game of Thrones wiki data


Adding New Doc Stores to Haystack

That you know our database, but we've not covered them into


Adding New Doc Stores to Haystack

Vectors yet because we need a retriever model to do that and we can't search until we have


Adding New Doc Stores to Haystack

That those vectors


Adding New Doc Stores to Haystack

Okay, because we're searching using a vector search


Adding New Doc Stores to Haystack

so


Adding New Doc Stores to Haystack

What we do is initialize a retriever model here here they're using dpr


Adding New Doc Stores to Haystack

You can use dpr if you want probably I would look into retriever models on


Adding New Doc Stores to Haystack

In sentence transformers personally, it's probably more efficient and and generally I think the performance is better as well


Adding New Doc Stores to Haystack

But in this example, it's just dpr


Adding New Doc Stores to Haystack

And then so we initialize that dpr retriever model and then update the embedding so we


Adding New Doc Stores to Haystack

create


Adding New Doc Stores to Haystack

the vectors


Adding New Doc Stores to Haystack

and then we


Adding New Doc Stores to Haystack

create the vectors from the text that we've already pushed to our document store


Adding New Doc Stores to Haystack

Okay, and this creates vectors stores them in the vector database, which is by fice in this case


Adding New Doc Stores to Haystack

Although fice is just an index. It doesn't have all of the other stuff that would make it a vector database


Adding New Doc Stores to Haystack

But i'm going to just call it that


Adding New Doc Stores to Haystack

for the sake of simplicity


Adding New Doc Stores to Haystack

And then we have a reader so


Adding New Doc Stores to Haystack

The reader model is that


Adding New Doc Stores to Haystack

the final part that you saw so


Adding New Doc Stores to Haystack

You know after we after we have retrieved a set of relevant context vectors


Adding New Doc Stores to Haystack

We pass them through our reader model and that extracts a specific snippet of an answer


Adding New Doc Stores to Haystack

Okay, like a few words that answers our question


Adding New Doc Stores to Haystack

And then so haystack allows us to do all this super easy, you know, we've


Adding New Doc Stores to Haystack

I don't know how many lines of code that is


Adding New Doc Stores to Haystack

It's you know, we have our imports and then there is something like four lines of code


Adding New Doc Stores to Haystack

to initialize all those components


Adding New Doc Stores to Haystack

And then we are initializing a extractive qa pipeline


Adding New Doc Stores to Haystack

So open domain question answering we are extracting


Adding New Doc Stores to Haystack

The answers from a set of contexts. So it's extractive qa here


Adding New Doc Stores to Haystack

Um and in that pipeline we just pass our read and retrieve


Adding New Doc Stores to Haystack

We don't need to pass a document store or the vector database because if we have a look up here


Adding New Doc Stores to Haystack

Retriever already contains the document store. Okay, so it's already in there


Adding New Doc Stores to Haystack

Okay, so we've initialized the pipeline and then


Adding New Doc Stores to Haystack

We can ask questions. So we have this pipeline run query


Adding New Doc Stores to Haystack

To create dot thrackey vocabulary


Adding New Doc Stores to Haystack

We have this top k we won't focus on those right here. It's just how many answers to return basically