Adding New Doc Stores to Haystack

to initialize all those components and


Adding New Doc Stores to Haystack

Then we are initializing a extractive QA pipeline


Adding New Doc Stores to Haystack

So open domain question answer and we are extracting


Adding New Doc Stores to Haystack

The answers from a set of contexts. So it's extractive QA here


Adding New Doc Stores to Haystack

And in that pipeline we just pass our read and retrieve


Adding New Doc Stores to Haystack

We don't need to pass the document store or the vector database because if we have a look up here


Adding New Doc Stores to Haystack

Retriever already contains the document store. Okay, so it's already in there


Adding New Doc Stores to Haystack

Okay, so we've initialized the pipeline and then


Adding New Doc Stores to Haystack

We can ask questions. So we have this pipeline run query to create Dothraki vocabulary


Adding New Doc Stores to Haystack

We have this top K. We won't focus on those right here. It's just how many answers to return basically


Adding New Doc Stores to Haystack

And then we print the answers although


Adding New Doc Stores to Haystack

In this notebook, we don't actually see them print the answers. I have an example so I'll show you quickly


Adding New Doc Stores to Haystack

Okay, so I'm just going to come down here and you can I have an example. This is with the


Adding New Doc Stores to Haystack

My so, you know what I built so far the pine cone


Adding New Doc Stores to Haystack

document so it's not finished yet, but it is kind of working so


Adding New Doc Stores to Haystack

Same thing again, it's tried to create pipeline. I've just copied this code from the


Adding New Doc Stores to Haystack

from that notebook you saw


Adding New Doc Stores to Haystack

And come down here and this is a bit that we were missing from the other one. So print answers prediction


Adding New Doc Stores to Haystack

And we get this now in this example, it's a really bad answer because I haven't I've only


Adding New Doc Stores to Haystack

uploaded like six contexts because I was just testing it


Adding New Doc Stores to Haystack

But we'll basically you'll get something like this. So you'll get an answer


Adding New Doc Stores to Haystack

It's like who created the Dothraki vocabulary and it's just pulling out someone's name because it's like it knows that you're asking who created something


Adding New Doc Stores to Haystack

But there aren't any good answers here. So it's like okay, that's all I can give you


Adding New Doc Stores to Haystack

So again, Elio Garcia from here. Okay, so


Adding New Doc Stores to Haystack

That's the sort of format that we're expecting this sort of workflow when you're using Haystack. Which is


Adding New Doc Stores to Haystack

It's really good. It makes this very easy and


Adding New Doc Stores to Haystack

And Haystack is it's all about making open the main question answering really simple


Adding New Doc Stores to Haystack

And


Adding New Doc Stores to Haystack

That's kind of why I want to bring pine cone as a document store into that as well because pine cone makes vector search


Adding New Doc Stores to Haystack

incredibly simple


Adding New Doc Stores to Haystack

You know from what I can see simpler than any other option out there. So


Adding New Doc Stores to Haystack

That's why I think they go together very well


Adding New Doc Stores to Haystack

And you yeah you get good performance and and everything else with


Adding New Doc Stores to Haystack

Pinecone is awesome. There's a there's a lot of benefits


Adding New Doc Stores to Haystack

to including it


Adding New Doc Stores to Haystack

Now what I want to just show you quickly is okay. How how do you get started when you're


Adding New Doc Stores to Haystack

setting up um, when you're hoping to contribute


Adding New Doc Stores to Haystack

Open source project. How do you set that up in in git?


Adding New Doc Stores to Haystack

so


Adding New Doc Stores to Haystack

I mean you can see here I made a little little chart


Adding New Doc Stores to Haystack

Which kind of explains why you know why why the setup is different so you have your local machine where you're going to do


Adding New Doc Stores to Haystack

development work


Adding New Doc Stores to Haystack

Typically you would only have what you you wouldn't have this


Adding New Doc Stores to Haystack

Like haystack repo upstream thing you just have this when you're working on a project


Adding New Doc Stores to Haystack

Okay, you have your remote repository your origin and you pull and push to your origin


Adding New Doc Stores to Haystack

It's pretty simple


Adding New Doc Stores to Haystack

So if you plan on contributing to your project


Adding New Doc Stores to Haystack

It's a little bit different because you need to first you need to fork


Adding New Doc Stores to Haystack

your new repository so you have the official repository and then you have your forked repository


Adding New Doc Stores to Haystack

That becomes your origin


Adding New Doc Stores to Haystack

And then you pull and push to that origin that's been forked


Adding New Doc Stores to Haystack

but you also


Adding New Doc Stores to Haystack

If you're working on this for a long time, you want your repository to stay up to date


Adding New Doc Stores to Haystack

with the upstream or the


Adding New Doc Stores to Haystack

official repository


Adding New Doc Stores to Haystack

so you need to also set up a an upstream remote as well so that you can


Adding New Doc Stores to Haystack

Pull and merge from that upstream to your to your local and your your forked repo to keep it up to date


Adding New Doc Stores to Haystack

So let me show you quickly


Adding New Doc Stores to Haystack

How you might do that?


Adding New Doc Stores to Haystack

Okay, so on my right I just have terminal window and on the left I have


Adding New Doc Stores to Haystack

Haystack repository


Adding New Doc Stores to Haystack

Run that notebook we run before let me so we'll go to the top level first thing you need to do


Adding New Doc Stores to Haystack

Is where are we?


Adding New Doc Stores to Haystack

Me comes in here view code no


Adding New Doc Stores to Haystack

Make it bigger. Here we go. So on this green button


Adding New Doc Stores to Haystack

and


Adding New Doc Stores to Haystack

We are going to no we're not we're going to fork here


Adding New Doc Stores to Haystack

Okay, I already have a fork so i'm going to okay fork again to a different account


Adding New Doc Stores to Haystack

Okay, i'm not going to do i'm just going to use this one i've already created


Adding New Doc Stores to Haystack

But anyway, you fork it will create a fork on your account


Adding New Doc Stores to Haystack

Okay, and then you go


Adding New Doc Stores to Haystack

To that to that fork. So actually I can go up here


Adding New Doc Stores to Haystack

Just replace deep set ai with my username


Adding New Doc Stores to Haystack

And then you go to the fork


Adding New Doc Stores to Haystack

Here just replace deep set ai with my username


Adding New Doc Stores to Haystack

Okay


Adding New Doc Stores to Haystack

And you can see I also fetch upstream here. You can also use that


Adding New Doc Stores to Haystack

I'm gonna we're gonna set up on on git


Adding New Doc Stores to Haystack

So you can see how it works


Adding New Doc Stores to Haystack

so this is my


Adding New Doc Stores to Haystack

My personal version of haystack i'm that I am modifying right


Adding New Doc Stores to Haystack

So what I want to do


Adding New Doc Stores to Haystack

Is code i'm going to copy this


Adding New Doc Stores to Haystack

And i'm going to


Adding New Doc Stores to Haystack

Come over here. Um, i'm going to cd


Adding New Doc Stores to Haystack

Documents, I think that's fine. I'm going to git clone


Adding New Doc Stores to Haystack

That okay, so it's going to clone haystack into or onto my local machine


Adding New Doc Stores to Haystack

It's actually going to take a really long time so let me just sail for another random project


Adding New Doc Stores to Haystack

So can I cd haystack


Adding New Doc Stores to Haystack

Okay, i'm just gonna make the haystack no problem, um


Adding New Doc Stores to Haystack

The cd into it i'm gonna pretend it's a


Adding New Doc Stores to Haystack

a repo


Adding New Doc Stores to Haystack

So i'm gonna get in it


Adding New Doc Stores to Haystack

Okay


Adding New Doc Stores to Haystack

And let's just pretend this is now our haystack repository that we've just


Adding New Doc Stores to Haystack

Done a git clone for okay


Adding New Doc Stores to Haystack

So


Adding New Doc Stores to Haystack

First thing you need to do is you need to add our remote so


Adding New Doc Stores to Haystack

git remote


Adding New Doc Stores to Haystack

Add origin. So the origin remote


Adding New Doc Stores to Haystack

Is going to be our personal


Adding New Doc Stores to Haystack

Like repo so it's here the this one james callum haystack


Adding New Doc Stores to Haystack

So let me copy this


Adding New Doc Stores to Haystack

Pull into here


Adding New Doc Stores to Haystack

And then what you need to do is set up a upstream


Adding New Doc Stores to Haystack

remote so


Adding New Doc Stores to Haystack

add upstream


Adding New Doc Stores to Haystack

And this is going to be the original


Adding New Doc Stores to Haystack

So if I go to here, so deep set ai haystack, so this is the original repo i'm going to copy this


Adding New Doc Stores to Haystack

And paste it in here


Adding New Doc Stores to Haystack

Okay, so now I think it's git remote yeah, so you can see the origin the upstreams


Adding New Doc Stores to Haystack

And that's that's what we that's what we want


Adding New Doc Stores to Haystack

So now whenever there's an update to the upstream that we want to merge without without


Adding New Doc Stores to Haystack

The version of the project we're working on we would just write git pull


Adding New Doc Stores to Haystack

upstream


Adding New Doc Stores to Haystack

Okay, and it will pull


Adding New Doc Stores to Haystack

Any updates and we'll have to merge and commit everything


Adding New Doc Stores to Haystack

into our own repo


Adding New Doc Stores to Haystack

at the same time if I


Adding New Doc Stores to Haystack

Let me move out of this one and move into the real haystack that i'm actually working on


Adding New Doc Stores to Haystack

so


Adding New Doc Stores to Haystack

projects haystack


Adding New Doc Stores to Haystack

Okay


Adding New Doc Stores to Haystack

There's nothing I want to change at the moment, but what I can do is I can git pull upstream


Adding New Doc Stores to Haystack

Git pull upstream


Adding New Doc Stores to Haystack

And maybe


Adding New Doc Stores to Haystack

Maybe we'll see some changes. I'm not sure


Adding New Doc Stores to Haystack

I think I did it yesterday. So maybe not


Adding New Doc Stores to Haystack

Oh, maybe yeah. Yep there is


Adding New Doc Stores to Haystack

Okay, so this is actually going to


Adding New Doc Stores to Haystack

Update everything


Adding New Doc Stores to Haystack

Okay, so I need to actually get pull


Adding New Doc Stores to Haystack

upstream


Adding New Doc Stores to Haystack

master I think they use


Adding New Doc Stores to Haystack

So not just git pull upstream need to specify the branch as well


Adding New Doc Stores to Haystack

Okay, and this is up to date. So there's no no issues there


Adding New Doc Stores to Haystack

Okay, so that's pretty good. But obviously when we are committing things so, you know git commit to do everything we're doing


Adding New Doc Stores to Haystack

And then we do a git push


Adding New Doc Stores to Haystack

We don't go to upstream


Adding New Doc Stores to Haystack

We go to origin. Okay, and we should also have a branch checked out as well


Adding New Doc Stores to Haystack

So we would get check out you can actually see mine here


Adding New Doc Stores to Haystack

If I do git status you can see i'm on the pinecone doc store branch


Adding New Doc Stores to Haystack

So when i'm when i've committed some changes


Adding New Doc Stores to Haystack

And I want to push that to my


Adding New Doc Stores to Haystack

Repository i'm going to do git push u


Adding New Doc Stores to Haystack

Origin and then i'm going to do pinecone


Adding New Doc Stores to Haystack

Doc


Adding New Doc Stores to Haystack

So, okay


Adding New Doc Stores to Haystack

And that'll make changes in my own repository


Adding New Doc Stores to Haystack

So yeah, that's really it for the like when you're working with open source you you have that so you're able to pull


Adding New Doc Stores to Haystack

From the original repository, which is this deep set


Adding New Doc Stores to Haystack

Haystack and you're also working and pulling and pushing


Adding New Doc Stores to Haystack

To your personal version of that repo


Adding New Doc Stores to Haystack

So I thought it's useful might be useful someone


Adding New Doc Stores to Haystack

out there


Adding New Doc Stores to Haystack

And that's really you know how you get started with your project. So once you have that set up you have your


Adding New Doc Stores to Haystack

Haystack repository, which i'll go and open in the s code over here


Adding New Doc Stores to Haystack

Okay, so


Adding New Doc Stores to Haystack

This is the haystack repository, um, or the local version of it


Adding New Doc Stores to Haystack

And in here we have the document source, okay, and there are there are a few so let's go through those


Adding New Doc Stores to Haystack

Now i'm not saying you you have to use pinecone or anything like that, but i'm just saying why I


Adding New Doc Stores to Haystack

Why I want to include it so we have a deep set cloud, which I haven't i'm I think is very new


Adding New Doc Stores to Haystack

I'm not sure even sure it's fully implemented yet. I'm not i'm not sure


Adding New Doc Stores to Haystack

Um, so I can't say anything about that. I have no idea


Adding New Doc Stores to Haystack

um, I know that's


Adding New Doc Stores to Haystack

Deep sets offering and i'm pretty excited to see what that is to be honest. I'm sure it's very cool


Adding New Doc Stores to Haystack

Elastic search and now with elastic search. You're not really doing a vector search


Adding New Doc Stores to Haystack

you're doing like a


Adding New Doc Stores to Haystack

sparse retrieval followed by a


Adding New Doc Stores to Haystack

dense vector re-ranking


Adding New Doc Stores to Haystack

so


Adding New Doc Stores to Haystack

Yeah, you can use this but you're not doing a an actual semantic search or


Adding New Doc Stores to Haystack

Or full on or full open the main question answering


Adding New Doc Stores to Haystack

With this so it's not great


Adding New Doc Stores to Haystack

Files you have to handle the infrastructure and stuff yourself. So i'm not too keen on that


Adding New Doc Stores to Haystack

Yeah


Adding New Doc Stores to Haystack

It's five is good, but you have to you have to understand what you're doing


Adding New Doc Stores to Haystack

Otherwise, it's a nightmare especially you have a lot of vectors if you get to like a million plus


Adding New Doc Stores to Haystack

Buys can be difficult


Adding New Doc Stores to Haystack

You have milvus. Milvus is


Adding New Doc Stores to Haystack

You know, I think they can host stuff for you


Adding New Doc Stores to Haystack

um


Adding New Doc Stores to Haystack

You can definitely use that I found it


Adding New Doc Stores to Haystack

difficult to set up and for me, I want everything to be super simple and easy and just


Adding New Doc Stores to Haystack

Good, um, I struggled with milvus a bit so


Adding New Doc Stores to Haystack

Can use it of course


Adding New Doc Stores to Haystack

I did find it like great and I had the same thing with weaviate


Adding New Doc Stores to Haystack

Again I hear I hear good things


Adding New Doc Stores to Haystack

But you're going to struggle a little bit


Adding New Doc Stores to Haystack

Um another thing


Adding New Doc Stores to Haystack

On these two so I think milvus and weaviate are close


Adding New Doc Stores to Haystack

You're going to get to pinecone at the moment in haystack


Adding New Doc Stores to Haystack

Uh, but they also doesn't we don't they don't have the the full


Adding New Doc Stores to Haystack

Metadata filtering capabilities of pinecone as well


Adding New Doc Stores to Haystack

Metadata filtering pinecone is very good


Adding New Doc Stores to Haystack

Because they have something called single stage filtering


Adding New Doc Stores to Haystack

Not pre or post filtering


Adding New Doc Stores to Haystack

And it is definitely I think it outperforms the others


Adding New Doc Stores to Haystack

So that's another thing


Adding New Doc Stores to Haystack

To to consider as well


Adding New Doc Stores to Haystack

So there are a few there are a few document stores that you can already use


Adding New Doc Stores to Haystack

Um, there are others as well as memory graph db. I haven't used these so I can't comment on them. Um,


Adding New Doc Stores to Haystack

But yeah, they're the current offerings that you have


Adding New Doc Stores to Haystack

And then we have pinecone which i'm i'm currently implementing


Adding New Doc Stores to Haystack

Okay, now let's have a look at


Adding New Doc Stores to Haystack

I want to have a quick look at this


Adding New Doc Stores to Haystack

Core functionalities. Um, we already really had a look at them, but I just want to cover them very quickly again


Adding New Doc Stores to Haystack

so


Adding New Doc Stores to Haystack

Basically what i'm going to need to work on


Adding New Doc Stores to Haystack

To actually get this working is


Adding New Doc Stores to Haystack

Write documents or or initialize actually


Adding New Doc Stores to Haystack

um initializing your document store


Adding New Doc Stores to Haystack

so if I


Adding New Doc Stores to Haystack

Open this more


Adding New Doc Stores to Haystack

Okay, so you can ignore this it's not relevant so this bit here


Adding New Doc Stores to Haystack

Got a document store a pinecone document store initializing that with a api key environment like you would with panko normally


Adding New Doc Stores to Haystack

Okay, so that's just initializing your document store. Like how how does that work? And that's pretty simple. It's just the init over here


Adding New Doc Stores to Haystack

And we're connecting to it


Adding New Doc Stores to Haystack

So that's the initialization


Adding New Doc Stores to Haystack

Um, and we're connecting to it. Nothing


Adding New Doc Stores to Haystack

Nothing special there


Adding New Doc Stores to Haystack

And then the first thing that you saw before is write documents. So this is where we're writing text and


Adding New Doc Stores to Haystack

This is we kind of have to do something


Adding New Doc Stores to Haystack

We can't just use pinecone for this because we can have very long


Adding New Doc Stores to Haystack

chunks of text and pinecone doesn't accept


Adding New Doc Stores to Haystack

More than five kilobytes of metadata at the moment


Adding New Doc Stores to Haystack

Okay, so we can't store long pieces of text in that metadata


Adding New Doc Stores to Haystack

So we actually have to use a sql a local sql


Adding New Doc Stores to Haystack

instance


Adding New Doc Stores to Haystack

Um to store those big contexts and then the smaller bits of metadata. We just throw into pinecone


Adding New Doc Stores to Haystack

So that's basically populating our sql database


Adding New Doc Stores to Haystack

Like I said before we just


Adding New Doc Stores to Haystack

Initializing a retriever model. It's not I haven't had to do anything there


Adding New Doc Stores to Haystack

And then we update embeddings. So this is where we're looking at those contexts converting them into vectors and storing them in pinecone


Adding New Doc Stores to Haystack

And then there's just a querying so


Adding New Doc Stores to Haystack

We're asking question here now you can't see here but this is actually calling method called query


Adding New Doc Stores to Haystack

And that's a standard method. So you have to the way we have to implement everything in our document store


Adding New Doc Stores to Haystack

Has to fit with the standard haystack


Adding New Doc Stores to Haystack

Ways of doing things


Adding New Doc Stores to Haystack

And then we have to be able to delete things delete documents


Adding New Doc Stores to Haystack

Get documents by ids


Adding New Doc Stores to Haystack

Um and so on and also include metadata, which is important now


Adding New Doc Stores to Haystack

I'm not going to actually dive into details of that in this video. I want to do that in the in maybe the next video


Adding New Doc Stores to Haystack

We'll walk over a lot of that stuff


Adding New Doc Stores to Haystack

However now just an overview of what there is in there how all this works


Adding New Doc Stores to Haystack

Why we'd even want to do it in the first place and so on so yeah, I don't think there's anything else


Adding New Doc Stores to Haystack

That's on to cover in this first video. Um, well, let me let me check


Adding New Doc Stores to Haystack

Yeah, that looks like everything


Adding New Doc Stores to Haystack

so


Adding New Doc Stores to Haystack

Yeah, that's everything for this video. We'll we'll


Adding New Doc Stores to Haystack

I think explore this in a lot more depth in the next video


Adding New Doc Stores to Haystack

So that should be pretty interesting. Uh, if you if you do want to contribute and help


Adding New Doc Stores to Haystack

implement this, please


Adding New Doc Stores to Haystack

Feel free go ahead and do it. Um, so over here


Adding New Doc Stores to Haystack

This is a repo at the moment. Um, I'll update in the description if that changes


Adding New Doc Stores to Haystack

But yeah, that would be really cool so


Adding New Doc Stores to Haystack

And as well, you can just go ahead and test it. That's also also really helps


Adding New Doc Stores to Haystack

So yeah, that's everything for this video. Um, thank you


Adding New Doc Stores to Haystack

very much for watching I hope is


Adding New Doc Stores to Haystack

I know it's a bit different. It's not really a tutorial


Adding New Doc Stores to Haystack

It's more just like walking through what we're doing, but I hope it's useful to see


Adding New Doc Stores to Haystack

You know how this sort of stuff might work


Adding New Doc Stores to Haystack

Um, maybe you know, maybe it's something that you want to do as well. Um, so that would be cool


Adding New Doc Stores to Haystack


Faiss - Vector Compression with PQ and IVFPQ (in Python)

Hi, welcome to this video.


We're going to have a look at how we can implement a product quantization index


or index PQ in FICE.


And we're also going to have a look at how we can implement a composite index


using an inverted file list, IVF, and also PQ together


to improve our results or our performance even further.


Now, we just have a quick look at this.


If you watched the previous video on the logic behind PQ, you'll recognize this.


But this is just demonstrating the speed increase that we can get


and also mainly the memory increase, which is an important factor in PQ.


And obviously, there's a huge increase here.


So the speed increase is a five to six times increase with just PQ


and the memory decrease is huge.


It's like a 97% reduction in memory.


Now, I'm not going to go through all of the, you know, how all that works


because we already did it in the last video.


So there is a link to that video in the description.


So if you'd like to go through that, go ahead and go through it.


But if you just want to see how this works in FICE, then continue


and we'll jump straight into it.


Okay, so the first thing that we need to do is actually get our data.


So we're going to be using this, the SIFT1M dataset.


Again, in the description, there's a link to a notebook where you can download


and also load that into your own notebook if you'd like to do that.


But just to say, it's a pretty straightforward, very popular dataset


for these sort of examples I use all the time.


And I'll just show you the sort of shape.


So XB is, it is like, it is the vectors that we'll be indexing.


So we're going to index these vectors in our index in FICE and search through them.


Typically has one million vectors.


I'm going to stick with 500k because it can just take a little bit of time


for everything to load.


Although that being said, I don't think it's that bad.


So let me, yeah, you know what, let's just go with one million.


It's more interesting.


And then XQ, I'm just using a single vector here.


So we just have one vector in there.


As you see, dimensionality for all of our vectors here is 128,


which is pretty typical, but it's definitely on the lower end for dense vectors.


So let's first initialize our PQ index.


So we need to import FICE and we'll get our dimensionality, which is XB.shape1.


Now I'm using this slightly different syntax to usual to align better with the PQ notation.


Usually you can go with low case D and you can go that's fine.


It's not really an issue.


I'm just going to go with up case for now to align better to that notation


if you are following on from the previous video.


And we're going to set M equals to eight.


So this is how many sub vectors we have.


Now, typically we assert that D is divisible by M,


but we already know it is.


So I'm going to skip that.


But that's something that you always need to,


you do need to make sure that D is divisible.


In fact, you know, I'm going to put it in anyway.


So let's assert that D is divisible by M.


Okay, just put this in because otherwise we don't,


we are not splitting our vectors or sub vectors equally.


And we will not be allowed to do that.


So we do need to make sure that's always the case.


And then we want to set the n bits value.


So this is the number of bits per sub quantizer.


So if you were watching the previous videos,


we would see this as,


so the k star value is equal to two to the power of n bits.


Okay, so the number of centroids that we have within each sub quantizer.


And then we initialize our index.


So index equals vice index pq d M and n bits.


Okay, that's our index ready to go.


And what we need to do now is actually add our vectors.


Now, you have to think with product quantization,


we are clustering our sub quantizers are using clustering.


So we do actually need to train it.


So we can see if an index needs to be trained or not by using this.


So we write index is trained.


In this case, it's trained as false.


So that means we need to train it before we add our vectors.


So to do that, we need to write index train,


and then we pass it xp.


We're in that.


And this can take a little bit of time.


It shouldn't be too long.


But if you start increasing the n bits value,


which is probably a good idea.


I mean, n bits of maybe 11 is actually what is recommended in the pq paper.


But I'm not using it here because it takes a long time


to train and build your vectors.


So I'm avoiding it here.


But if you're using this,


I definitely recommend trying those larger n bits values.


It's also worth noting that when we're using composite index of IVF and pq,


we can't use an n bits value of greater than eight,


but we'll move on to that anyway.


That has to be something that they've hardcoded into FICE.


For now, I think they should be removing that limitation at some point.


So we have now trained and now we can add our vectors.


Okay.


So now what we want to do is we'll return,


say the top k most similar vectors,


we'll go with k of 100.


And then we want to do our search.


So we're going to go distance.


I so typically would write d here.


But obviously we already using d, so I'm going to write distance


equals index.search.


And then in here, we get past our query and we also pass k.


And that will return the k nearest neighbors in that search.


So in here, if we write shape, we see that we have 100 of those.


And then distances here are the actual distances.


So i are the indices of the nearest neighbors,


distances, the actual distances that they have returned.


Now let's go ahead and what I'm going to do is I'm going to time it.


So we can use this time function to essentially run this cell multiple times


and get the average time taken to run it,


which is also quite useful when we're comparing different indexes.


We'll do index.search and we want xqmk.


Okay.


Now we see that that took 3.34 milliseconds.


So reasonably fast.


Let's compare that to a flat index.


So we'll do l2index equals fice index flat l2


and d, oh d with a capital, l2index.add.


And we want to add all of our data to xb.


And let's time that again.


So I'm just going to copy this here and we'll just replace that with l2index.


And let's see what we get.


Okay. So an average of 19.5 milliseconds.


So already this is pretty quick, right?


Which is cool.


Now we can check the performance of our index as well.


Now this varies from all the times I've tested it,


but we should get something around maybe 50.


So 50% recall isn't exactly cutting edge, but that's peaky.


That is how it is.


If we want high recall, we can increase n bits to get that.


But it's definitely, you know, it's a drawback of using PQ.


Okay. So we want to sum for every one value for every value that we find in our index.


So in the return indexes for our PQ results, if they are in the l2 results.


So we're just saying, you know, how many of the results returned PQ index


were also returned by our l2 index, which we are seeing as the 100% recall.


So l2 index, 100% recall.


And here we're seeing, okay, how many of those did we match to using our PQ index?


If i in l2 i.


So I actually need to just copy this, pull it down here.


So when we use a time it, we can't get any variables out from it,


which is, I don't know why, but it is a little bit annoying.


So we will get l2 this and l2 i.


Okay. So this time we get 38% recall.


Okay. So let's now have a look at the memory usage of each of these.


So I'm just going to define a function, get memory, so we can more easily compare those.


And what we'll do is we'll just write our index file like so.


Index. And I'll just call it temp index.


Now, because we're going to delete it straight away afterwards,


we're just writing it to file so we can read, see how large the index is.


And then we delete it again.


A file size equals os path dot get size to get size of that index in bytes, I think.


So we'll do temp dot index.


And then we don't really, we're just using that to check the size.


So we just want to delete that afterwards.


So like so.


And then we just return the file size.


Okay. So let's do that for both of our indexes.


So we get the memory for the l2 index first.


Let me, so that's remove, not remote.


Okay. So we get pretty big.


That's, it's half a gigabyte, which is massive.


And then we want to get memory again for our index.


Let's see what we have.


So now we have, we have eight megabytes.


So that's a massive control.


That's eight over five, 12.


So it's a 98.4% reduction in size, which is huge.


Which is really cool.


Now we, like I said before, we have this composite index where we have both the IVF step and a PQ step.


So we're going to have a look at how we can implement that.


And we'll see that memory decrease will be a slightly less than the memory decrease that we had before.


So we'll just do that.


And then we'll do the same thing for the other indexes.


We'll see that memory decrease will be slightly less significant because it's a more complex index.


So that would be very small still.


And then the speed decrease will be absolutely insane.


So we'll go into that.


So now we have, we have end list, which is the number of,


the number of cells within our IVF index.


So if we just have a quick look at what that might look like.


So imagine these are our PQ vectors on a 2D space.


Adding IVF essentially does this.


So we add all these separating cells called Voronoi cells.


And each of the vectors within each cell is assigned to the vector to that cell centroid.


Now here, this, you see the magenta dot at the bottom, that's our query vector.


And it's landed within that cell that you see highlighted.


Now, because it's landed in there, we only compare it to those vectors within that cell


that are also assigned to the same cell.


And that's what we're doing with end lists.


We're saying how many of those cells we would like to have.


Now we're going to go with 2048.


And it's worth noting that this must be greater than, greater than or equal to k star,


which is equal to two to the power of n bits.


Okay, so in this case, we could go with 256 to be fair.


Let's go with that for now and see how it goes.


And then we initialize our index like this.


So we go index equals FICE, index IVFPQ.


And in here, we need to pass our vectors.


So in this case, we will just, when I say vectors, I mean the index that contains our vectors.


So you typically see this written as quantizer,


but I'm going to write it as vectors because I think it's a little bit less confusing.


So we'll do FICE, index, flat, L2.


So we're just going to use these as our starting point.


But that doesn't mean we're storing these as they are in our index.


They actually get processed using the PQ step as we already did before.


So we have VEX and D, endless, M and n bits.


Okay, so we now have our index and we can see if it's trained like we did before.


Obviously, it will not be because we are still using PQ again.


So we need to train that and we do the same as we did before.


So train, xp.


After that, we add our vectors and we can get our results.


So index.search, xqk.


Okay, now we haven't seen how fast that is.


So let's have a look. We'll just time it again.


Okay, and although the actual search took longer, it's because we're looping through.


See, we had 10,000 loops for this test and we got this number, which is insanely small.


So 0.001.


No, that's not right.


We are on... What are we on?


This. 0.1 milliseconds. Sorry.


So 0.1 milliseconds. What did we get before? It was...


Yeah. So this is using our flat index, 20 milliseconds.


If we go up a little further, we see 3.3 milliseconds for that search up there.


So yeah, super fast.


And then if we check the recall, so we do use the same as what we used before.


So where is our sun? Here.


And like I said, this can go up and down.


So at 24 this time, it's pretty bad.


But I mean, like I said, pq isn't the best for recall.


But what we can do is actually, because we're using IVF,


we can increase the number of cells that we search through.


So you can kind of see that happening here.


So using the example from before, we start with an end probe value of one,


which is the default, increase it to two, three, four, five, and so on and so on.


We can increase that all the way up to search all of our end probe values.


Of course, that is kind of a waste of time,


because if you're searching through all of your cells,


you're basically just doing a flat search


with the additional overhead of having an IVF index.


So it's even slower.


We can use that information.


So we know that if we set our index.endprobe equal to 2048,


which is the number of cells that we,


the maximum number of cells that we set.


Or did we use 256?


No, we used 256.


So the endless value.


So we can just put in this here.


If we do that and we do a search again,


xq and k, and we go this i.


Do that again.


And let's come up here, get our recall.


So we see that the maximum recall we're going to get with this instance is 30%


Now this is pretty low working on it.


Before I was getting sort of 50, 52%.


So it's also a case of, you know,


there is some randomness in what you're going to output from your index.


But of course, again, we can also increase n bits


and do some other things.


So we can increase n bits, try decreasing m,


and see how things work.


But it's also worth noting with IVFPQ,


the n bits value does have to be 8, at the moment at least.


It's just, it's something that's hard coded into FICE.


Unless you want to go into the FICE code and then change that,


you can remove it if you want.


Now, what we want to do is obviously not search through the entire index.


So we're going to change endprobe.


Let me just bring this down.


I'm going to change endprobe to something that is not quite as high.


So maybe 20, we get 33.


So that's high enough for us to get our maximum performance.


But obviously it's going to be a lot quicker


because we're searching through 20 cells rather than 256.


Let's try 10, 32.


So increase it a little bit to 12.


So 32, 16, 33.


So around this area here, so 14, 13.


So 13 is our optimum endprobe value


where we get the max recall out of that the quickest time.


But of course, which parameters you use, completely up to you.


It depends on your use case as well.


So that's it for this video.


Thank you very much for watching.


I hope it's been useful and I will see you in the next one.


How-to Build a Transformer for Language Classification in TensorFlow

Hi, and welcome to this video on implementing transformer


models with TensorFlow.


So we're going to go through six steps, which


are downloading and preprocessing data,


initializing the Hugging Face totalizer model.


And by Hugging Face, I mean the Transformers framework.


Then we encode input data to get input ID and attention tensors.


Then we build the full model architecture.


So that is our input layers, which go into BERT,


and then the output layers post-BERT.


Then it's back to the normal TensorFlow process,


where we sell our optimizer, metrics, and loss.


And then we begin training.


And we will cover each one of these steps in this video.


So first, we need to actually get our data.


So we're going to use the IMDB Movie Review data set,


or it may actually be rotten tomatoes.


I'm not sure.


Now, this data set provides us with sentiment ratings


from 0, which is terrible, up to 4, which is amazing.


You can get the data set from Kaggle,


or we can just download it using the Kaggle API, which


is what we're going to do here.


Now, if you haven't used Kaggle API before, that's fine.


All you need to do is pip install kaggle.


And then you need to head over to the Kaggle website,


go to your account page, scroll down to, I think,


it's API integration, download kaggle.json.


And then you need to place kaggle.json


in the correct Kaggle folder, which


will have been created when you did the pip install.


Now, if you're not sure where that is,


all you need to do is import kaggle, like that.


And when you execute this, a OS error will appear.


And it will say you need kaggle.json, which you don't


have, you need to put it in this folder.


You just go to the kaggle folder,


and then you are ready.


Now, we just need to initialize our API and authenticate it.


And now we can use the competition download file


method to download our data.


And then we can use the


kaggle.json file method.


And we are going to import the data into this directory.


Now, let's refresh up here, and we can see it.


Now, obviously, it's a zip file, so we


need to quickly unzip that.


We can also do this in Python, or you can do it manually.


We will just do it manually.


It's easier, well, quicker at least.


And there we go.


We now have our data here.


It's a tab delimited file.


So if we open it, you can see here, we're delimiting by the


tab character.


And we can see we have our phrase and our sentiment, which


are the two that we care most about.


Now, we'll use pandas to read data.


And because it's a tab delimited file, we use


read CSV, and then we just set the separator to tab.


Now, this data set has a full sentence, which we can see


with the phrase ID is one and sentence ID is one.


That is our full sentence or full phrase.


And then we have lots of parts of that same phrase cut down


into different pieces and given a sentiment value.


Now, I mean, you can use this, but I'm going to avoid it


because I'm just going to be using the training data for


both the training set and the validation set.


And I don't want to pollute the validation set with very


similar phrases that we have used in the training data.


So we're just going to drop duplicates and keep the first


element of every unique sentence ID.


And here you can see that we have now removed those


duplicates or segments.


With the segments removed, we only have 8500.


Now we need to move on to encoding our data.


So for that, we are going to be using the Transformers


framework, which we will also be using for the transformer


itself.


And this works by providing a tokenizer and the model


itself for each transformer.


So we're going to be using BERT, and that means that we


are going to import or initialize a BERT model and also


the BERT tokenizer, which is already pre-built.


Now, before we do this encoding, we need to figure out


how long we want each sequence to be because the encoding


method also acts as our padding or truncation method.


So to do that, we will get the sequence length in words of


each sentence and plot out and just kind of go by eye and


say, OK, around here, won't cut off too much data.


So first, we need to get the sequence length of every


sentence in here.


And all we're going to do here is get a length of each


sentence split.


Now, split will, by default, split by spaces.


So we will get a list of words here.


Now, we need to actually visualize this, so we will use


Matplotlib and Seaborn just because they're super easy


and quick to use.


And then we will also set the Seaborn style just to make it


a bit more visually appealing.


And we will also increase the figure size so we can actually


see what's going on.


And then we will use a distribution plot.


OK, and here we can see the distribution of the length of


each sequence in our data set.


Now, we could cut off maybe around 40 or even 50.


I think we'll go with 50 just so we get as much data in


there as possible.


So we'll set sequence length equal to 50.


Now, we need to initialize our tokenizer.


And before we do that, actually, we need to import it


from the Transformers library.


And we are getting our model from a pre-trained model.


And we're using BERT base cased.


Now, cased here refers to whether BERT distinguishes the


difference between uppercase and lowercase characters.


The alternative would be uncased, and this would just


not distinguish the difference between uppercase and


lowercase. Everything would just become lowercase.


But when people are on the Internet and they want to


shout and seem angry, people put everything in capital


letters. So BERT can probably pick up on this and tell


that someone is being dramatic or shouting at you over


the Internet or whatever else. And because we are


classifying sentiment here, it's probably quite important.


Now that we've initialized our tokenizer, we can go on


to the encoding. So we use the encode plus method,


which looks like this.


So you'll see here we've just defined or hard coded a


single line, which is Hello World. We are using a max


length of 50. We want the encoder to truncate any text


that is longer than 50 tokens. Obviously, this one will


not be. But when we are feeding all of our data through


it, we need this in there.


And on the other hand, we also want anything shorter


than 50 to be padded with pad tokens. In this case, we


will end up with 48 of these padding tokens.


And here we are just telling the tokenizer to pad up to


the value that we have given in the max length argument.


Now BERT comes with several special tokens. We have the


start sequence, end of sequence, padding, unknown and


mask tokens. In order to add these in during the encoding


method here, we need to write add special tokens true.


And in this case, all it's going to do is add the start


sequence token, the end of sequence token, and then it's


going to add all of our padding values.


Now, there are several different outputs that we can get


from this encode plus method. By default, we have input


IDs and the return token type IDs. Now, the token type


IDs we don't really need, so we can tell the tokenizer to


not return those.


But we do need input IDs, which is fine, we get them by


default, and also the attention mask tensor. To return


this, we just write return attention mask equals true.


Finally, because we are working in TensorFlow, we need to


return TensorFlow tensors.


OK, so here we have our outputs. So we get two tensors.


One of those are the input IDs and then also our attention


mask here. So we input the sequence Hello World.


And we can see that this value here and input IDs is Hello


and this is World. Now, the 101 and 102 you see here are


the start sequence and end of sequence tokens used by BERT.


And the remaining zeros are simply the padding tokens.


We also have the attention mask and this is used to tell BERT


what tokens to calculate attention for and which tokens


to just completely ignore. So where we have a one that


tells BERT, yep, pay attention to this. Where there's a zero,


it means just ignore it. So we have zeros for every padding


token because padding tokens aren't important to us. It's


just padding. But then we have ones for the end of sequence


and start sequence tokens and also Hello and World because


they are actually important for BERT to pay attention to.


Now, of course, this is just one and we need to do this for


every sample in our data set. So we'll go ahead and do that.


Now we're just going to use a simple for loop to add each


value or sequence into a NumPy array, which we will initialise now.


So first import NumPy.


And then initialise our two arrays.


Both of them are going to be the same size, so it's going to be


the length of our data frame by the sequence


that we have defined, which is 50.


And we do this twice. We also want one for the attention mask.


And we can see the shape of our arrays here. Now we're just going to use a


for loop to do this. It's only a small data set, so


whatever we use doesn't really matter. We are only processing and encoding


this data one time and then we will save it and then load it from memory


when we're actually training our model.


And there we have our for loop. This will go through each sequence


and add each one of those sequences into the respective index


of our initialised zero arrays here.


OK, and then here we can see our complete arrays.


So at the top we have our input ids in x ids and we can see each one


starts with our sort of sequence token 101 followed by a few words.


Then there will be the end of sequence token somewhere in the middle there. And then at the end we have our


padding tokens. And then we can also see in x mask, which is our attention mask.


We have the ones to pay attention to and the zeros to ignore, which


obviously correspond to the respective values in the ids array.


Now for our labels, we are actually going to one-hot encode them.


So at the moment, let's see what we have.


So we will get values of 4, 1, 3, 2


and 0. Though you can't see it here.


Now this is a pretty good format to go straight in and one-hot encode.


So all we need to do here is create an array from the


data frame column.


And then here, like we did before, we are just initializing a empty zero array.


And what we are doing here is we are taking the array size.


So if I. Let's do this a little differently.


Do this a little differently.


So the array size is just the length of our data frame.


And array.max is what we see maximum value within


our array, which is the number 4. And then we are adding 1 onto that, which is saying here that we


want a zero array of 8,529 rows


by 5 columns.


Now at the moment, we just have an empty zero array. So we just need to add ones


in the indices where we have a value. And we do that very


easily like this.


So we create a range of values from 0 to


8,528, which is the array size here. And then


within that we add array because in array we


have each sentiment value, so 0, 4, 3, 2. And this will add a


one in that index. So 0, 3, 4. And that


produces our one-hot encoding.


So here our rating was 1, here


4, here 1, and so on. Now I said before that


typically we'd save these before loading them in. So we are going to save them.


Obviously we don't need to load them back in. But I will show you how to anyway


because going forwards, if you want to retrain data, you have to


do this. Otherwise you'd have to do everything all over again. And when you're working with bigger data sets, that


will take quite some time. So first we'll just save them.


So now we've saved them and we've just removed all of them from memory.


So now we will not be able to access any of them.


Going forwards, we are just going to load them back in, which is exactly the same process.


Quickly write that out.


And now we have our data back in.


OK, so now we need to pull all of our arrays


into a TensorFlow dataset object. So we'll use a


dataset object because it makes things a lot easier so we can restructure the data,


shuffle and batch it in just a few lines of code, which is a lot


faster in terms of performance and also a lot faster for us to write down.


It's much simpler code. So we'll import TensorFlow.


And also, if you're using a


GPU, you can check that it is being detected by your system with this.


And there you can see I have a GPU being picked up by TensorFlow.


So first we need to restructure the data. TensorFlow expects our data


to be input as a tuple. So that tuple consists of


our inputs and our target or output labels. Now,


because we are using BERT, our data structure is slightly different because in the input


tuple, we actually have a dictionary and that dictionary contains a key


that is input underscore ids, which maps to our x ids array.


We also have another key, attention underscore mask, which maps to


our x mask array. So first we actually need to create our


dataset object, which we do like so.


And this just creates


a generator which contains all of our data in the


tuple like format where each tuple contains one x id array,


x mask array and label array. So we can actually view


one of those like so.


And here we can see each one of our arrays.


So first one x ids, x mask and then labels.


Now TensorFlow expects our input data as a dataset


to be in a tuple format. The zero index of that tuple


needs to be our input values and the one index of that


tuple needs to be our labels. Now, in our case, it's also slightly


different because we are using two inputs. So within that


tuple, we have our input and within that input, we have a dictionary


and that dictionary consists of a key input underscore ids,


which maps to our x ids array and attention underscore mask, which must map to our


x mask array. So to create this structure, we need to


build a mapping function like so.


And here we return the format we need. So input underscore ids,


which will map to input ids


and attention underscore mask, which will map to our


attention underscore mask, which will map


to mask. Then, because we are still


expecting to use the input output tuple, we also need to add labels onto the end there.


And to apply this mapping function to our dataset object,


we use the map method.


So now we can view a single


row in this dataset.


And now we can see a slightly different format.


So we have our input dictionary here where we have input ids


and attention mask. And then we have our output tensor here.


Now, I said before that this makes shuffling and batching our


dataset very easy. So we can do it in a single line


like this.


So here we're going to put our samples into batches of 32 and then the


value that I've given to the shuffle method here essentially just needs to be


very large. The larger your dataset, the larger this number needs to be.


I take a sample of my dataset and increase the number


if I can see that this is not shuffling the dataset properly.


OK, so now we have our shuffle batch dataset in the correct format.


So now we just need to split it into our training and validation


sets. OK, and to do that, we need to get the total


size of our dataset now that it is batched. So we can do that


like so.


Now, because the dataset object is a generator, we can't


just take the length of it directly, so we have to convert it into a list.


Now, if you're working with a very large dataset, this is probably not the right


method to use, and you should instead take the size of the dataset that you know


already and calculate the current length of it, including the batch size


from that. But for us, this is fine and we can see that


dataset length is 267. Now, if


we want a 90-10 split between the training and validation


set, we simply use a split value of 0.9.


And then we get our


training and validation sets as two different datasets.


To split the dataset, we use take and skip.


We used take earlier on, and what it does is simply takes the


specified number of, in this case, batches and nothing else. Skip, on


the other hand, does the opposite. It skips a specified number of batches and then takes


the remainder.


And then at the end here, we can delete dataset if space is an issue.


OK, so now data is ready, we can go on to actually


building our model architecture. So first we initialize BERT,


and to do that, we need to import tf.automodel from the Transformers library.


And we initialize BERT like so.


So here, remember


to use the same model that you are using to initialize your tokenizer.


And here we can see that we have now imported BERT.


So we have BERT, but we need to build a network


around BERT as well. First we need to do is define our input layers.


And of course, there are two because we have the input ids and the attention mask.


And the shape is simply the sequence length that we are using.


And the name here is very important. This needs to match up to


the dictionary value that we have defined in our dataset here.


So input ids, input ids and attention mask.


So these need to match up. Otherwise TensorFlow does not know


where these are going.


And we do the same here, but we are doing this for the attention mask.


So those are our two input layers. And now we need to


pull the embeddings from our initialized BERT model.


BERT consumes our two input layers.


Like so, and BERT will return two tensors to us.


One of those is the last hidden state, which is what we are interested in. That is a


3D tensor, which provides all the information from the last hidden state of the


BERT model. The second tensor


that we are going to ignore is called the pooler output. And the


pooler output is essentially the last hidden state run through a


feedforward or linear activation function and pooled. So that


creates a 2D tensor, which can be used for classification


if you want. But we are going to pull it ourselves, so we will not be using


that tensor.


OK, so here you can experiment with adding LSTM layers,


convolutional layers or anything else. But for now, to keep things


simple, we are just going to add a global max pooling layer, which will convert our


output 3D tensor into a 2D tensor. Again, you could


skip this and you could just output the pooler output tensor


like this by changing the 0 to a 1, but we are not going to be using that.


OK, so up here, we just need to define the input data types as well, which I missed.


And that will remove the type error.


Now we need to normalise our outputs here.


This is what we are going to do. We are going to use the


input data types as well.


And we are going to use the input data types as well.


This will almost always give better results when we are actually training the model.


And then following this, we will go into our densely connected neural network layers,


which are in charge of actually figuring out the classification of our


input embedding outputs.


And then we want to add a dropout layer here. This just prevents


any overfitting or too much overfitting.


Then we add another densely connected neural network.


And finally, we are creating our output layer,


which is going to be a densely connected neural network with a Softmax activation function.


Now we use Softmax here because we have our three labels or no, sorry,


five labels in the output. So let me change this.


So we have our five labels in the output because we have one hot encoded the 0,


1, 2, 3 and 4.


And finally, we just give it a name of outputs.


Now that is our model architecture, but we still need to tell


TensorFlow what our input layers are and what our output layer is.


So to do that, we define our model like so.


And to the inputs, we pass input IDs and mass.


And then to the outputs, we just pass y.


Finally, we have a model so we can actually


execute that and produce a model summary here so we can see what we have


built.


OK, and here we can see our model. Now, if we scroll down to the


bottom, we can see the number of parameters in our model, and we have quite a lot,


108 parameters, almost all of them are trainable. Now,


BERT is a very big model, and I wouldn't recommend training


it unless you have a specific reason to.


Now, for this data set, it's definitely overkill.


So what we can do is actually freeze the BERT model


by freezing the third layer at index two of our


model layers. And we simply set trainable equal to false to do that.


So if we now look at our model summary, we will see that the number


of parameters is exactly the same, but the number of trainable parameters


will have reduced by a lot.


So here we now have 104,000


trainable parameters rather than 108 million trainable parameters.


We can go ahead and put together our optimizer, loss


and accuracy, compile our model and begin training.


Now for our optimizer, we're just going to use Adam.


With a learning rate of 0.01.


For the loss, because we are using one-hot


encoding for our outputs, we are going to use the categorical cross entropy loss.


And finally, for accuracy, we are also going to use categorical accuracy for the same reason.


And we can compile our model.


So here I've just missed the R.


And now we can actually train our model.


So we just do model fit.


We have our training data, our validation set


and our model.


Our training data, our validation set.


And I found that for this model, we have to use a lot of


epochs to actually get a good accuracy out of it. So I'm going to train it for total of


maybe 140 epochs. Now, depending on your


GPU, if you're using a GPU, this will still take quite some time.


So if you're using a CPU, it will take a very long time. So maybe reduce


the number of epochs or reduce the data set size a little further if you want.


So I'm going to start training this and I will see you on the other side.


OK, so we have finished training and you can see


that the accuracy over time is actually quite good.


It goes up very slowly, which is why using a lot of epochs has been quite useful here.


But then if we take a look at the accuracy at the end, it's 82 percent,


which is not bad. But I think more importantly is the fact that it was still going up very gradually.


I think with further training, this could quite easily get to about 90 percent


on this data set. And considering it's a very small data set, that's pretty good.


Now, if we look at the validation accuracy, we actually get a high number by quite a bit.


So this actually went up to 94 percent here, almost 95.


And I would assume that this is because within the validation set,


there are more easy examples, whereas in the training set, we have some more difficult examples.


But nonetheless, these are pretty good results for quickly putting together a model.


It's not a particularly big model other than the BERT encoder in the middle.


But otherwise, it's a pretty straightforward, simple model.


So it's pretty cool that you can get these results on sentiment analysis in so little time.


And going forward with other data sets with more data can definitely do a lot better.


At the same time, we can also improve the model. We can add LSTM layers to the classifier


or convolutional neural net layers or even just more densely connected neural network layers as well.


So there is a lot that we can actually do with this. But for now, that's everything.


So I hope you enjoyed the video. I hope it's useful to you.


If you have any questions, let me know in the comments below. But otherwise, thank you for watching


6 of Python's Newest and Best Features (3.7-3.9)

Hi and welcome to this video. We're going to go through and take a look at some of the


coolest more interesting features that have been added into Python in the past few years.


So we'll just go straight into it and code a few examples as well.


Okay so the first one and the only example we're going to look at from Python 3.7 is


the addition of breakpoints. So obviously a breakpoint is where we typically click on


the left side of our editor to add in the little red circle which means we want the


code to break at this point. What the breakpoint function allows us to do is actually use Python


debuggers within our code. So the function uses a BDB debugger by default but we can


also use other alternatives as well. So for example if we had a list of functions here,


a list of print statements. Okay so they just print out hello. If we for whatever reason


wanted to put in a breakpoint to try and figure out what was going wrong, save this code was


probably a little more complicated. We could add in this. Okay and then we go straight


into the Python BDB debugger at the bottom here. And this is essentially like an interactive


shell where we can just write Python and figure out what is going on. So obviously there's


not really much for us to figure out in this case but when you are actually writing more


complex functions or classes or modules in your code this can be really really useful.


So here we can print or do anything. It's simply an interactive Python debugger window.


So the second of our new features actually comes from Python 3.8. Now this is the warvis


operator which was the go-to for the majority of 2019 for anyone covering what is new in


Python 3.8. So it's a new operator added to Python. It's genuinely very useful. It essentially


allows us to assign a value to a variable on the fly. So it looks like this which looks


like a waris hence the name and we can use it to write more compact code. So say we have


a list here and we want to assign the length of this list to a variable which we'll call


tlen and then we want to write an if statement checking if tlen is greater than 3. And if


so we want to print out the length tlen and a little statement saying that this is greater


than 3. That's great. With the waris operator we can actually make this more compact and


include our variable assignment within the if statement. So we put this within a bracket


and then we change our equal sign here to a waris operator and we can run that again


and now we have 4 is greater than 3. So then we've shortened our code, made it more compact.


Obviously you know whether this is useful and more readable is completely up to who


is writing the code. Nonetheless it's a very useful operator to understand and use. So


a more realistic use case for this could be if we are writing a regit statement. So I'll


just write out a sentence here quickly. Ok so we want this sentence to go through and


use regex to find a currency followed by a few numbers. So I'll just switch this to dollars


here. So we're going to go through each of these sentences and only print out the ones


that contain a currency. We also need to import re for regex. Now for those of you that have


used regex a fair bit before you will know that when we use a regex search it will return


either none or it will return a set of groups which contain our matches. And typically what


we'd have to do is we'd have to write amount equals re.search and then we would have our


regex in here. So here I'm just going to do a lookup. So this is just checking for a dollar


symbol before a set of digits. And this will return just digits not the dollar symbol.


Ok and usually you would have to do this and then we would have to write if amount or just


if amount is true and then print out the amount and the group which would be zero and then


we would get our results. This should be sentences. Ok so we get 100. But what we can do instead


is use our walrus operator. Write if amount and now we've shortened our code. So that's


a more probably more useful example of using the walrus operator. Another really cool feature


that was added in 3.8 is the fstring equal sign specifier. So I'll give a quick example.


So we have these two variables which are just two numbers. And maybe we are trying to figure


out maybe we're doing something wrong and we want to see where our values are going


and what variables are being assigned to. And what we might typically do is this. We


might write x equals x and y equals y. Ok and this is probably what a lot of us would


do. But rather than doing this we can actually just write x equals like this. And this will


print out the same thing. So we show the variable name and the value within it. So it's a smaller


but I think very useful addition to Python. Ok so the final feature from Python 3.8 is


the inclusion of syntax to specify the input parameters of functions that cannot be called


by name but instead by the position. So what I mean by that is if we have this function


here. We can either write this which is using the position of the parameters in order to


fill in those variables. Or we can also do this. And we'll get the exact same output.


Now this is great but when you're writing code that other people are using and relying


on and maybe that code isn't completely finished yet. You may find that you are not entirely


sure if you'll keep this variable or input parameter as op or this is x or this is y.


You might change these around and if people are using your code like this that will cause


problems when you do change your code to use different variable names. So maybe at a later


point you decide you would change your code to use method instead of op. If we took that


and then tried to use this function again we would get this type error. Obviously we


don't want that. So what we can do instead is define method as a positional only parameter.


And we do that by adding this slash. So now if we try to run this but with operator again.


We see okay we have a type error. Count got some positional only arguments. Passes keyword


arguments. So although we haven't changed operator to method yet we are preventing the


user from using op here. So that forces the user to use this instead. Now the user can


still define these using their names. That's completely fine. But they cannot use anything


that comes before this slash. So if we do rewrite our function to use method instead


of op. And the user runs that code that they've been using before we won't get any errors


because we prevented the user from being able to use this keyword parameter. Okay so that's


everything for Python 3.8 and we're going to move on to Python 3.9. Now the main one


of those is the addition of more type hinting. So type hinting is something that has been


around for a while and it seems to be a recurring theme with every Python update that we will


get some new type hinting functionality. For Python 3.9 they've essentially merged all


the different type hinting features that they've been adding over time and brought them into


the core Python functionality. So there's no more importing typing modules or annotation


modules just works as it is for the most part. So for those of you that don't know what typing


is in Python say we have this function here. It's quite clear that this function add int


shouldn't be used with strings. Okay but it can be used because we're just using a plus


here so it's essentially telling Python to concatenate these two strings. But our intention


is actually to only allow integers. So what we can do is add a type annotation or a type


hint and that's what this is here. So we add this colon followed by the data type that


we intend this value to be. Then we return plus a vowel. Now when you are writing this


code out in your editor and you have your Python linter so pylint for example it will


come up with an error if you write something like this later on. And this isn't an error


that will stop your code. We can still run it. It is just a hint it is a warning. Okay


you can't see it here because we're in Jupyter but in a normal editor with pylint installed


you will see this bit here get highlighted with a note saying we expected type int got


string instead. Okay so that's the warning that will pop up. Now this is just very useful


for complex code bases where you're not entirely sure what values are supposed to be going


in and out of different functions and classes. Just makes things a lot easier to follow.


Another syntax addition in terms of the type hinting that was added is the ability to define


the type that should be coming out of a function not just into a function. Say here we want


to sum up all the values within a dictionary. We want to input a dictionary and what we


would expect to get out of this which is defined by this arrow syntax here is an integer. Now


we just return the sum of the values within our dictionary. And then we define our dictionary


here and we can also add the type here as well. And then we call our function and we


should get a integer out of that as expected. Now if we change this and these were the other


way around and we had strings instead of numbers here we would obviously get a full blown error.


But before we even run that we would see a type in come up saying okay we are expecting


a integer here not a string. So it can be very useful in just in terms of readability


it can depending on how you do this be very very useful. If you look into complex libraries


like you have the TensorFlow library or Transformers library those libraries with type hints can


make things a lot easier. In fact some of the more recently written code in the Transformers


library has been using type hinting and it makes it so much easier to read. So it is


a very good addition to Python in my opinion. Now something that is very useful if we take


this function again we can actually use more complex types. So we know that we want a dictionary


but maybe you want to define what is within that dictionary as well. So we can actually


do that like this. So now we know that our dictionary should be built of string keys


and integer values. And this is just an extra layer of detail so we can be more specific.


There we go. Now the final feature in Python 3.9 that we will go over is dictionary unions


operators. There is quite an interesting addition to the syntax and it gives us two new operators.


One is called the merge operator and the other is the update operator. So the merge operator


looks like this and we use it to merge two dictionaries. So let's write out two dictionaries


quickly. So we have these two dictionaries and we can merge them using our merge operator.


Ok and now we can see that those two dictionaries are now one within our new variable C. The


update operator is slightly different in that it allows us to do the merge in place. So


what that means is rather than adding this to a new variable C we actually assign it


to the variable on the left of the statement. And this here is the update operator. Now


if we print a we see that we get the same function. Ok so that's it for this video covering


some of the biggest changes and upgrades to Python that we actually see and use on a daily


basis. But that's it for this video. I hope you've enjoyed and as always thank you for


Intro to Sentence Embeddings with Transformers

Hi, welcome to the video. We're going to explore how we can use sentence transformers and sentence


embeddings in NLP for semantic similarity applications. Now in the video we're going


to have a quick recap on transformers and where they came from. So we're going to have a quick


look at recurrent neural networks and the attention mechanism and then we're going to move on to


trying to define, you know, what is the difference between a transformer and a sentence transformer


and also understanding, okay, why are these embeddings that are produced by transformers or


sentence transformers specifically so good? And at the end we're also going to go through how we


can implement our own sentence transformers in Python as well. So I think we should just jump


straight into it. Before we dive into sentence transformers I think it would make a lot of sense


if we pieced together where transformers come from with the intention of trying to understand


why we use transformers now rather than some other architecture. And I think it's also very important


if we try and figure out the difference between a transformer and a sentence transformer as well.


So we're going to start with recurrent neural networks and more specifically I want to have


a look at machine translation. So machine translation use something called a encode decoder


network where you would have a encoder which is a set of, well, recurrent units usually something


like LSTMs or GRUs and information or that encoder would encode some sort of input text. So in let's


say English it would encode that English text into something called a context vector. And then


this context vector would be passed along to a decoder network which again is just another set


of LSTM or GRU units and it would decode those into our target language say like French or in


this case actually Italian. That is how machine translation worked back with recurrent neural


networks. The only issue is that we're trying to pass a lot of information through that single


point between the encoder and the decoder. Now that creates what is called information bottleneck.


There's too much information trying to be crammed through that single one point. So what they came


up with is something called the attention mechanism. What the attention mechanism does is for every


step or every token that is decoded by our decoder that token is sent to the attention mechanism


and the alignment between the decoder at that time step is compared to all of the encoder


units or on all of the encoder hidden states. And what that does is essentially builds this


type of attention. So it tells the decoder which tokens from the encoder to focus on. So it's


literally saying you know where do I need to pay attention to whatever my current unit is. And


this attention mechanism what it produces is something like this. So this is from another


very well-known paper in 2015 and what you can see is a matrix of the we have the French words or the


French translation on the on the left on the y-axis and then on the top we have the English


translation. And all of these boxes you see are the activations of the attention mechanism. So


we can see essentially which words are the most aligned and that is essentially what the attention


mechanism did. It allowed the decoder to focus on the most relevant words in the encoder part


of the network. Now moving on in 2017 there was a another paper called Attention is All You Need and


this really marked I think what is a turning point in NLP. What was found in this paper is that they


could remove the recurrent part of the encoder decoder network and maintain just the attention


mechanism. And what they produced with a few modifications to the attention was a high


performing model than any of the recurring neural networks with attention or without attention that


came before it. And what they named this new this new model was a transformer. So this is this 2017


paper Attention is All You Need is where transformers came from. And they actually came


from a mechanism that was aimed to help improve recurrent neural networks. Now of course like I


said the attention mechanism was not just the same plain attention mechanism that was used before


in recurrent neural networks. It had been modified a little bit as well. And those modifications


really came down to three key changes. And those were positional encoding which replaced the key


advantage of recurrent neural networks in NLP which was the ability to consider the order of a


sequence because they were recurring neural networks. They considered one word or one time


set after the other. So there was a there was a sense of order to those models that does not appear


in for example convolutional neural networks. And this positional encoding worked by adding a set of


varying sine wave activations to each input embedding. And these activations varied based


on the position of the word or token. So what you have there is a way for the network to identify


the order of the tokens or the token activations or embeddings that are being processed.


The next change was self-attention. Now self-attention is where the attention mechanism


is applied between a word and all the other words in its own context. So the sentence or the paragraph


that it belongs in. Now we saw with the encoder decoder that attention was being applied between


the decoder and the encoder. This is like applying attention to the encoder and the encoder again.


And what this did is rather than just embedding the meaning of a word it also embeds the context


of a word into its vector or word representations which obviously greatly enriched the amount of


information that you have within that embedding. And then the third and final change that they


made was the addition of multi-head attention. And we can see multi-head attention as several


parallel attention mechanisms working together. And using these multiple attention heads allowed the


representation of several sets of relationships rather than just a single set. Now these new


transform models also had the benefit of generalizing very well. So what we find with


transform models and of course you could do this to an extent with your current neural networks as


well but it was far less effective. So with transform models we take the core of the model


which has been trained using a significant amount of time and computing power by the likes of Google


and OpenAI. We just take that core model add a few layers onto the end of it that are designed


in a way for our specific use case and train it a little bit more so we fine tune it. And


I think one of the most widely known of these or the most popular of these models is probably BERT.


And of course there are other models as well later in this video we're going to have a look at one


called the MPNET model. But BERT is certainly one of the I think most popular of those.


Now so far we've sort of explained that transformers have a much richer embeddings


or word or token embeddings than anything that came before. And that's good but we're


interested in not word or token level embeddings but sentence embeddings because we


want to do semantic similarity between sentences or paragraphs. And of course transformers they


work the inside of the transformer works based on word level or token level embeddings. So that


doesn't really help us that much. So what would happen before sentence transformers were introduced


is we would use something called a cross encoder. So we would have a BERT cross encoder model. And


like I said before we have the core BERT model and then we just add a couple of layers onto the end


of it and fine tune it for our specific use case. And that's what a cross encoder is. It's the core


BERT model as you can see on the screen right now. It's a core BERT model followed by a feedforward


neural network. We pass two sentences into the BERT model at once. The BERT model embeds them


with very rich word and token level embeddings. The feedforward network takes those and decides


how similar those two sentences are. Now this is fine and it is actually accurate. It works well


but it's not as scalable because essentially every time you want to compare a pair of sentences


you have to run a full inference computation on BERT. So let's say we wanted to form a semantic


similarity search across just 100,000 sentences which is a reasonably small data set.


We would have to run the BERT inference computation 100,000 times to actually go through


and identify the similarity between all of those sentences. And that's going to take a lot of time.


And it gets worse as well. I mean if you consider clustering all of those 100,000 sentences


we would end up with just under 500 million comparisons there. So running a full BERT


inference prediction 500 million times just to cluster 100,000 sentences. Obviously that's not


scalable. That will take a very very long time. So ideally what we need is something like word


or token embeddings but for sentences. Now with the original BERT we could actually produce these.


They just were not very good. So what we can do is take the mean value across all of our word


embeddings typically BERT is 512 those being output by the model. We take the average across


all those and take that average as what we call sentence embedding. Now there were other methods


of doing this as well. That was probably the most popular and effective one. And we could take that


sentence embedding, store it somewhere and then we could just compare it to other sentence embeddings


using something that's a bit simpler than a full BERT computation like cosine similarity.


And that is much faster. That's fast enough for us. But it's just not that accurate. And it was


actually found that comparing average average love embeddings which were produced in 2014


were actually more accurate. So we can't really do that. We can't use this what is called a mean


pooling approach. Or we can't use it in its current form. Now the solution to this problem was


introduced by two people in 2019 Nils Reimers and Irenia Gurevich. They introduced what is the first


sentence transformer or sentence BERT. And it was found that sentence BERT or S BERT outformed all


of the previous Save the Art models on pretty much all benchmarks. Not all of them but most of them.


And it did it in a very quick time. So if we compare it to BERT, if we wanted to find the most


similar sentence pair from 10,000 sentences in that 2019 paper they found that with BERT that


took 65 hours. With S BERT embeddings they could create all the embeddings in just around five


seconds. And then they could compare all those with cosine similarity in 0.01 seconds. So it's


a lot faster. We go from 65 hours to just over five seconds which is I think pretty incredible.


Now I think that's pretty much all the context we need behind sentence transformers. And what we do


now is dive into a little bit of how they actually work. Now we said before we have the core transform


models and what S BERT does is fine tunes on sentence pairs using what is called a Siamese


architecture or Siamese network. What we mean by a Siamese network is that we have what we can see,


what can view as two BERT models that are identical and the weights between those two


models are tied. Now in reality when implementing this we just use a single BERT model. And what we


do is we process one sentence, a sentence A through the model and then we process another


sentence, sentence B through the model. And that's the sentence pair. So with our cross-linked


we were processing the sentence pair together. We were putting them both together, processing them


all at once. This time we process them separately. And during training what happens is the weights


within BERT are optimized to reduce the difference between two vector embeddings or two sentence


embeddings that are produced for sentence A and sentence B. And those sentence embeddings


are called U and V. Now to actually create those sentence embeddings we do what we did before with


BERT where we do the mean pooling operation. Now the reason that this works better is because we're


fine tuning it. So with BERT we didn't fine tune it. This time we are fine tuning it. Now there are


several different ways of training S BERT but the one that was covered most prominently in the


original paper is called the Softmax loss approach. And that's what we are going to be


describing here. Now for the Softmax loss approach we can train on natural language inference data


sets. Now the 2019 paper used two of those, the Stanford natural language inference corpus and


the multi-genre natural language inference corpus. Now both of these sets are very similar.


Both of these were merged together and what we have inside there are sentence pairs. One is a


premise which suggests a certain hypothesis. So those two sentence pairs are in some cases related.


And we can tell whether they're related or not using the label feature. Now the label feature


contains three different classes. We have zero which is called entailment and what this is is


it indicates that the premise sentence suggests the hypothesis sentence. Then we have class one


or label one that means the two sentences are neutral. So they could both be true but they are


not necessarily related. And then we have number two which is contradiction which means the premise


and hypothesis sentences contradict each other. Now given this data we feed sentence A into


BERT first for our fine tuning and then we feed sentence B into BERT. So that's our premise and


hypothesis. The Siamese BERT or BERT outputs two sentence embeddings U and V from this process.


And what we do is concatenate those two sentence embeddings. Now the paper explained a few different


ways of doing that but the most effective was take U and V and what we do is we take the absolute


value of U minus V which is an element wise operation. So basically finding the difference


between U and V and that produces this other vector which is the the bar U minus V bar.


And then we concatenate all of those together. So they all get concatenated together and then they


are passed into a feed forward neural network. Now this feed forward neural network tapes the


concatenated vector length or sentence embedding length as its input and outputs just three


activations. Now those three activations align with our three labels. So what we do from there


we have those three activations we then need to calculate the softmax loss between those


predicted labels and the true labels. Now softmax loss is nothing more than a cross entropy loss


function and that's really all it is all there is to training one of these models. Now we are going


to cover all of this in full. We're going to go through the code and everything and train our own


sentence burden model but for now we're just describing the process and we'll cover that in


another well probably the next video and article. Now that's really I think everything we need to


know for now on how they work and where sentence transformers and transformers come from. So let's


jump into Python and what we'll do is actually implement some of these models using the sentence


transformers library which was built by the same people who designed the first sentence transformer


as BERT. So let's go ahead and do that now. So the first thing that you will need to do if you


do not already have sentence transformers installed is just pip install sentence transformers.


Like so I already have it installed so I'm not going to to go ahead and run that. So all I'm


going to do now is from sentence transformers I'm going to import the sentence transformer class


class object and then from there we can initialize a sentence transformer. Super easy.


So all we need to write is model equals sentence transformer and then in here we just need to


write our model name. Now if you go to this website


sbert.net you will find a load of different models. Now the one that we will be using is this.


So this is the original SBERT model and if we just come down here and print out what that will


return to us we will see a few different components.


So you can see that we have we have well we have two components we have the transform model


and we have the pooling layer which is the mean pooling that I mentioned before.


Now the transformer we see that the max sequence length so the maximum number of tokens that we


can input there is 128 and we see that we're using the base model is a SBERT model from the


Hugging Face library. Then in pooling we can see the output dimension of our sentence


vector which is 768 and we can also see the way that the tokens have been pulled to create the


sentence embedding which is just the mean pooling approach. Now given these sentences here


I'm going to run that. All we need to do to actually encode those is write model dot encode


and then we just pass sentences and that will create our embeddings. I'm just going to call


this embeddings and let's see what they look like. So embeddings again


and we see that we have these which are our embeddings. So each one of these is for a single


sentence so up here we have the first sentence here and these are each of dimensionality 768.


So we can check the shape of that if we want as well just to confirm that it's true. So you see


that we have five embeddings, five sentences, each one 768 dimensions. Now that we have our


sentence embeddings we can use these to quickly compare sentence similarity for


quite a few different use cases. The most popular are semantic textual similarity which is a


comparison of sentence pairs which is what we are going to do here and generally this is probably


most often used for benchmarking these kinds of models. Then we have semantic search and


we've covered semantic search a lot already in other articles and videos and this is information


retrieval using semantic meaning. So given a set of sentences we can search using a query sentence


and identify the most similar records. So this enables us to search based on concepts rather than


specific words which is pretty cool. Now we also have clustering so we can cluster our sentences


which is obviously useful for things like topic modeling. Now we can put together a very fast STS


sort of semantic textual similarity example using nothing more than a cosine similarity function


and numpy. So we just want to import numpy as np and we also I'm going to use the sentence


transformers cosine similarity function. So write cosine so write sentence transformers util import


cos sim. Okay so from there what I'm going to do is I'm going to initialize a empty array


of zeros and I want that to be the length of our sentences so I want it to be a five by five array.


And what we're going to do is loop through all of these sentence embeddings that we produce from


Spert and compare those with the cosine similarity. So we just want to write for i in a range


and we'll let the sentences and we just want to say the similarity function or the similarity array


from i to the end on this specific column is going to equal to cosine sim and then here we


want to put our embeddings we just want the the current session followed by the other other


embeddings. Okay and that will populate our our similarity array. So we can print that out and we'll


see okay so down the middle here so I've not populated these because these well we've already


got those pairs on this side of the array you'll see we're going to visualize it so that'll probably


make more sense. So let's do that now so we're going to import map plot lib not pyplot as plt


and I'm also going to import seaborne just makes it a little bit easier and nicer as well and I'm


just going to write sns dot heatmap in here sim and we want annotations set to true.


Okay so this is just a visual that array that we produced just now and we can see here so we have


the sentence values or sentence positions so sentence zero if we go here actually let me


print them out here instead that makes more sense so if I print sentences maybe it's verifying


it so I'll put them like this yeah so we have a number zero is obviously this first sentence


and then we have number one two three and four and that correlates to zero to four and zero to four


here on the on the two axes so if we want to look at the most similar pair according to our


expert model is this four and three which gets a cosine similarity of 0.64 and if we have a look


three and four are these two so these two are the only ones that kind of mean the same thing


and I've written these so that they are basically they basically carry or they share none of the same


descriptive words so for dentists we have dental specialists chewing bricks I put flossing so not


even the same thing and construction materials as well it's not even the same thing there


but very similar sort of concepts that we're talking about there so you can see that it's


identifying those two as the most similar concept which is pretty cool and then we get some other


similarity scores which are kind of high here and they're not really related so we have three and


zero talking about eggplants and mannequin heads so it's pretty different I suppose in reality


someone being an eggplant and this sort of thing is they're both kind of like weird strange things


to happen so maybe that's why it's capturing similarity there but it's not obviously similar


so generally I think it's good that it identifies this as being the most similar but it could be


better and we do find that with the more recent models it is in fact better so what we can do is


I'm going to get this other model and we'll just call it model


and we're going to do sentence transformer again and this time we are using the MP net


base model now this is basically the highest performing model at the moment although I was


told on the channel's discord by Ashrac that there is actually when they were training this


MP net model they also trained a Roberta model and although the Roberta model is not shown on the


sentence transformers homepage you can see in the competition where they train both these models


that they did also train that model and it does have slightly higher performance as well it might


be slower I'm not sure probably because it's Roberta but the performance of that is actually


is actually higher than the MP net version of that model which is pretty cool now here we can


see something well a few things that's slightly different besides the max sequence length this


three times as long as it was with the BERT model using the MP net base model and we also have this


additional normalization function now let's just take what we we wrote up here and we'll


just take what we we wrote up here so I'm just going to take this


bring it down here and let's just use the heat map straight away so sns heat map


sim and anapt equals true and we'll see something slightly different or we should do this isn't


so I haven't processed the similarity yet or the embeddings so it's right embeddings equals


model dot encode sentences now if we run it we'll see that the similarity of these ones of these


other sentence pairs is now a lot lower but it's still identifying foreign three is pretty similar


so we can see straight away there's a decent performance increase from from this MP net model


which is it's like the most recent model and the original SBIRT model here so I think that's


pretty cool to see as well now that's it for this model introducing sentence embeddings and the


sentence transformers library and models now going forwards obviously this is a series of videos so


we're going to be covering a lot more than just sentence transformers but next we are actually


going to cover how we can train a sentence BERT model SBIRT model which I think will be will be


pretty cool so until then that that's it for now so thank you very much for watching and I will see


How to Build Python Packages for Pip

Okay so we're going to work through actually building a Python package that we can install


How to Build Python Packages for Pip

through pip. So when we're installing a package through pip we're actually installing a package


How to Build Python Packages for Pip

from this website here the python package index. So if we just search for something we all know


How to Build Python Packages for Pip

pandas we'll see here that we actually have all these different pandas projects. Now the reason


How to Build Python Packages for Pip

there's so many of these is because loads of people upload projects that say pandas in the name


How to Build Python Packages for Pip

as you can see. The one that we use and we know is this pandas 1.2.3 and we head on here and we can


How to Build Python Packages for Pip

actually see all of the information about that package. Now what we want to do is actually build


How to Build Python Packages for Pip

our own package and upload it to this so that we can pip install it and build it in a way that


How to Build Python Packages for Pip

other people can use it without any issues. So to start I am going to use this package that I put


How to Build Python Packages for Pip

together quickly for this and what it does is it simply generates a ASCII visual based on a few


How to Build Python Packages for Pip

images that I've included in here. So these synthwave visuals and all this package does is


How to Build Python Packages for Pip

you initialize it and you press generate and it will generate ASCII versions of these pictures


How to Build Python Packages for Pip

at random. So it's pretty useless but it's a good example. Now the first thing we want to look at