to compare how many of them match.


Best Indexes for Similarity Search in Faiss

And what I'm also going to do is I'm going to change K to 100.


And remember we're using 10.


That's a little bit low.


So let's change it to 100.


And then we can compare those a little easier.


And the reason I want to do that


is to show you the performance difference we'll get


when we start modifying another variable called the emperor value.


So let me wait for this to finish training


and we'll jump into that.


Okay, so that's finished now.


And we sort of run this.


I can compare or do it.


Yeah, yeah, I want to do this.


And I'm going to call recall on here


and we'll see what it is.


So we get 5%.


Okay, that's pretty terrible.


And the way that we can improve this


is by increasing the emperor value.


So the number of cells in our IVF index that we're searching.


The only problem, so well, let me show you


what we normally do is this.


So we'd write index nprobe and we set this to a higher number.


But we can't actually do that anymore


because our index consists of, let's say, multiple indexes.


So if we were just using this part, it would be fine.


We could do that.


But because we've added obq,


we can't access nprobe in a normal way.


So we have to use something else.


And that is, if I write it out, it's 5.


So we can add price, extract index, and we need to write IVF.


I know we're not using IVF, we're using IMI,


but we use this here for both of those,


IMI or IVF, whichever one you're using.


And then in here, we just pass our index.


And then from here, so this is kind of extracted


the IMI part or the IVF part of our index.


And that means that we can now set the nprobe value.


So we're going to set that to 14.


And what we can do is di equals index.search again.


We can do xqk.


And we do recall.


Hi.


We get 44%.


So it is increasing.


Now, what happens if we set increase that to 100?


We get 69.


So you see, as we keep increasing this number,


the performance or the recall increases as well.


Okay, so 69 looks like the max we're going to get there.


Although in reality, you've self-tested this before,


you can definitely get higher than that.


You should, in fact, well, slightly higher.


You should be able to get up to about 75%.


So, I mean, that is that index.


And through this, we do get very fast search times.


So you see here 0.2 is faster, I think.


So time it.


Index.search.


And we want to say xqk.


And let's reduce this.


Let's say to 50.


See if we still get a...


Okay.


Okay, so I'm going to go for 60 for a 66% recall.


Then index.search.


And this loops through multiple times to get the average speed.


And we get 379 microseconds, which is pretty fast.


So that is the last two of our Composite indexes.


Let me, before we move on to the next one,


let me actually show you the performance


you can expect from this one.


So this is what I got when I was testing on this index.


So in the middle there, the magenta line,


that's what we just built.


You also have the non-optimized version,


which is the blue line.


And then I wanted to also include the flat index in there as well for comparison.


The flat index is slower though.


And plus again, the memory requirements for that flat index are sort of through the roof.


This one is incredibly small.


Not as small as IVF, PQ.


Or IVF-ADC, but still pretty small.


Okay, so this is our final Composite index.


And I think the most, well, the most performing one from what I've seen,


as long as you don't mind increasing the memory usage quite a bit.


So before we're using product quantization,


we can use product quantization with this


and still get pretty good results actually.


But the memory usage when we're using this with flat vectors,


is of course more.


So what we're doing here, we still have that IVF component.


So we're not using IMI this time, we're using IVF.


Like we were in the first Composite index.


And then we're also having a HN-SW,


or hierarchical navigable small worlds index or graph.


And what it does is it uses IVF,


splits our data into all of its different cells.


Or cells which each have a centroid to identify which cells are closest to your query.


And during search time,


rather than comparing all of those centroids to your search query,


so an exhaustive search across all those centroids,


we use HN-SW to approximate that and optimize or speed it up.


So let's just go into HN-SW very quickly.


First, so it's based on the small worlds graph theory.


And I mean, this is where you can see on the screen right now,


there's this circle or the nodes or vertices on the side.


And then we have all the edges or links between them.


And the theory is that in a very large network,


so something like Facebook where there's billions of users,


you will find the average degree of separation,


the average number of links between any two people is very small.


I think with the Facebook one, it's something like incredibly small, like four or six.


And that's from the average between any one,


any two random people on Facebook are connected by only four to six friends.


And with this, what we find is that in these small world graphs,


we have sort of long range and short range links.


Now, when we're traversing around that graph,


so from one node or one vertex to another,


we obviously move more quickly across the graph when it's a long range link.


So let's say your friend, you're American,


your friend has a friend over in India,


and they have friend who is someone important in one of the villages in India, for example.


You know, between you and that important person in a village in India,


even though you don't know anyone in India,


you have your friend, their friend over in India, and him.


So there's three sets to get to him.


Now, the set between you and your friend, you're pretty close,


you have a very high proximity, so it's a short range link.


Between your friend and their friend over in India, that's a long range link.


Now, HNSW takes this theory and applies it.


So what it does is it takes long range links and splits them across different layers.


So in this case, we define long range as geometrically far apart.


And on the highest level of our HNSW graph, we have all of these long range links.


And it's on that first level that we input our query vector


and we first find those nearest neighbors.


So we take our first few traversals across the links.


And as we go down each layer, so each traversal, we go down a layer,


the links become shorter and shorter range, so it becomes finer.


And at the bottom layer, we finally get to what is our approximate nearest neighbor.


Now, that's what you can see here.


So we saw the entry point on layer two, which is our entry layer.


That only has long range links, so we can make it a big jump


from our entry point over here, over to here.


Then we're on to the next step.


So we move down the layer, down here.


And then we go, OK, where's the nearest neighbor now?


It's over here.


OK, so we go there and then we're on to the next step.


So we again want to move down a layer.


So now we're on our bottom layer and this is our final set.


And so, OK, where's our nearest neighbor?


OK, and then that's what we assume.


We assume that this vertex or node is our nearest neighbor to our query vector.


And the reason we do this is if you think, OK, up here, we only have long range links,


that means there's very few nodes.


Whereas down here, we probably have loads of nodes.


So if we saw it down here, we'd be making loads of steps.


Whereas doing it like this, we can make big steps early on and then smaller steps later,


which just minimizes the number of steps we need to take


or the number of comparisons that we make.


Now, how does this apply to IVF?


We said, OK, this is an IVF and HNSW composite index.


And this is what you can see now.


So that HNSW process that we just went through,


imagine all those points or those vertices,


all of those are cell centroids from IVF.


So what we're doing is rather than comparing,


exhaustively comparing all of those cell centroids,


we're just comparing a few of them


or we're finding the approximate closest or nearest neighbor very quickly.


And then we limit our exhaustive search scope to that single cell,


if the n-perf value is one, obviously.


Usually it's not going to be one, it's going to be greater.


Yeah, but that's pretty much what we're doing with this IVF-HSW index.


Now, because IVF-HSW optimizes this centroid


or focuses on optimizing the centroid search portion,


it makes more sense if we increase the number of centroids.


Because if we increase the number of centroids,


we reduce the number of vectors within each cell.


And because our optimized search point here


or the approximate search is the centroid comparison,


and then once we're in those centroids,


we're actually comparing all of the vectors in there.


So we're performing an exhaustive search on the scope that we do at Crate.


So what we do is we increase the number of centroids


because that's the optimized portion


and we reduce the number of vectors that will return in our final scope


because that's the unoptimized part of this search.


So to put all that together, we're going to use this index factory string.


In terms of increasing speed, like I said,


we would just increase this number here, which we will see.


I'll show you another graph in a minute.


Okay, so let's go up here.


I'm going to modify this.


So we're going to go IVF 4096 followed with an underscore.


And underscore by HSW 32.


And what this means is we're creating this many cell centroids or cells.


And this 32 is how many connections or links each node or vertex has in the HSW graph.


And then after that, we're using flat index here.


Now, I said before, we can use PQ 32, for example, here.


But the accuracy, the recall won't be as good.


But you can use that.


So if you need to reduce the memory usage with this,


and it is very good, by the way, using PQ,


so you still get a very good recall.


You can do like that.


Now let's run that.


We will do the search again.


We'll check the recall, see what we get.


Okay, so that has just finished and we get this recall of 25%.


And now again, we're using the emperor value of 1 by default.


We probably want to increase this.


So come down here.


We don't need to change the variable name.


I'm just doing it because it bothers me, leaving it as IMI.


And let's go 60 again.


I mean, we do have a high number of cells here.


So probably want to increase it a bit more than that.


But let's go with that.


Now let's go with that, see what we get.


And we get 93%, which okay, fair enough.


That's pretty good.


Like straight out 93%.


That's nice.


So let's time it, see how fast that is.


And you see, okay, so before we were getting with the multi-D ADC index,


we were getting I think 3.64 microseconds.


Now we're getting 5.4.


A bit more, but it's hardly anything.


And I mean, we're getting 93%.


Before we were getting, I think, what do we get?


But of course, at the same time, the memory usage for this is like half a gigabyte.


But we can reduce that by instead of using flat index,


we just change it to PQ32.


Not like that.


We change it to PQ32.


And that improves it a lot.


But we're not doing that here.


We're just going to leave it flat.


And let me let's just...


So if we, for example, we know that the maximum number of cells here is 4096.


Let's do an insanely high number.


And this is the maximum performance that we're going to get out of it.


So 100%, right?


Because we're using flat vectors, we can get that 100% point.


So yeah, we can get very good performance, 89% there.


And yeah, it's super easy.


So look, 432 microseconds.


I mean, for me, I think that's very good.


So for that reason, this is definitely one of my favorite indexes.


It's just, again, the memory usage is high.


So let's have a look at the actual performance, search time, recall.


So I've included a few different number of cells here for us to compare.


So like I said before, the optimal, the optimized part is the IVF part


or the centroid search part, the number of cells.


So we can increase the number of cells.


And that will decrease the search time, but also decrease the recall a little bit as well.


But not that much.


So, you know, you can modify it.


You can increase the number of cells, IVF, varioin recal.


Varioin recales to improve the search time if you need to.


Now, I mean, that's it.


I didn't want to cover anything else in the video.


We've covered those three indexes, the IVF-ADC, Multi-D-ADC, and IVF-HSN.


So with those, I think you can build some pretty cool indexes.


So thank you very much for watching.


3 Vector-based Methods for Similarity Search (TF-IDF, BM25, SBERT)

We're going to cover three different vector based similarity search methods.


We're going to explain how they work and try and get an intuition for why they work.


And we're also going to actually go ahead and implement each of these in Python.


Now, for TF-IDF and BM-25, they're both pretty similar.


And BM-25 is actually an improved version of TF-IDF.


And we can classify both of these as being sparse vector methods.


So these both use sparse vectors, which essentially means that vectors


with a lot of zeros in with the occasional value in there.


And then our bottom one down here, which is sentenceBert.


That's an example of a dense vector.


Now, dense vectors are quite interesting as they allow us to consider semantics


or the meaning behind what we are saying rather than just the syntax


and the words that we're using.


So that's pretty interesting, but all of these can work better than


the others in different situations.


I've had dense representations of language in some cases work worse


than sparse representations.


And of course, the other way around as well.


So we'll have a look and we'll cover all of those.


So let's get started with TF-IDF.


Now, TF-IDF consists of two different components, as you can see the name TF-IDF.


Now, the first of those components is the TF, which is the term frequency.


And term frequency does what it says on the tin.


It looks at a sentence or a paragraph and given a certain query, which is the queue


here, it will tell you compared to the length of that document, how frequent


your query is.


So what we have here is this D, which is our document.


So we're saying, okay, what is the frequency of the query queue in our document D?


Then on bottom down here, we're saying, what is the frequency of all terms


in our document D?


So if we were to calculate this, we would get the following.


If we were to calculate this, we would get one because we only have one query.


Our query in this case is bananas divided by 18, which is the total number of terms


in that document.


And that will give us 0.056.


So that's our term frequency, which is half of TF-IDF.


We still have IDF.


So what is IDF?


How do we calculate that?


So IDF is the inverse document frequency, and it's calculated for each document.


So when we say document here, we mean it can be a sentence or a paragraph.


Essentially, what you see here, we have ABC.


Each one of those are documents.


In this case, each one of them is a sentence, could be a paragraph,


could be a mix of different things, right?


In this case, we have these sentences.


And what we're saying here is the inverse document frequency is the log of the number


of documents.


So in this case, we would have three over the number of documents that


contain the query is or the word is, which in this case is all of them.


So we have all three documents there.


And so what we get from that is the value one.


Now, don't forget that this is a log here.


So it's not actually one.


It's if I left myself a little bit of space, it's log and then one log one, which is equal to zero.


So is is a very common word.


And because it's so common, we literally do not care about it.


And this IDF is multiplied by our term frequency.


So we'd have TF down here.


So if our term that we were searching for was is, then all of these sequences would


get a value of zero.


Which might seem counterintuitive.


But if your query is something like is the best city in the forest, right?


You know, from the top sentence there, you wouldn't want to be prioritizing the word is or the or in,


because those words are everywhere.


So we don't really care about those words.


We want to care about the more unique words.


And that's what you'd get down here.


So we have a query for forests.


The number of documents is actually the same.


So three.


And how many documents contain the word forest?


Only one of them.


We have a at the top.


So we would have log three.


And what is log three?


Log three is zero point four eight.


So then we times that by TF and we get a larger number than we would for is.


So that is where the IDF or inverse document frequency comes in.


It essentially gives us a multiplier for less common words because chances are they're more


relevant to our query.


Now, what we can see here is just a work through of what we just looked at.


So we're calculating the TF for each document and the word is up here.


We calculate the IDF and then we multiply those two together.


And because is is everywhere, the IDF is equal to zero and therefore the outcome is always


zero because it's just not a relevant word that we care about.


Now, on the other hand, the word forest.


When we calculate the TF IDF for each of these, the TF for B and C is zero because the term


just doesn't appear in those documents.


And then the IDF is higher because forest is a rarer word.


So it's zero point four eight.


Now, the outcome of that is that only this value here.


So this is the TF IDF for the document A is not zero.


And that's essentially how TF IDF works.


Now, let's jump across to Python.


Let's have a quick look at how we'd write our code.


So we have we have those three sentences again up here.


So let's run that.


And then just here we can see, OK, we input not a single word.


We input numpy, we use a numpy here.


We're merging all the documents here.


Now, the reason we do this, so they merge into a list of lists is just so we can more


easily calculate the terms N and also the term NQ equals our query, which you can see


just here in the IDF section.


Now, once we do get to TF IDF here, so we calculate the IDF, which I just explained


anyway, and then also the term frequency, which is just the number of matches for our


specific word in a given sentence.


So let's run that and let's have a look.


TF IDF for the word.


Let's go over is like we did before in the previous section.


Like we did before in A.


What do we get? We get zero as we as we calculated before.


And how about forest?


And here we get the zero point zero six.


Obviously, it's more precise here, but before we got zero point zero six in our other screen.


Now that's TF IDF reasonably straightforward.


However, I did say that these are vectors, right?


These are sparse vectors.


Obviously, this doesn't look much like a vector.


So how do we turn this into a vector?


Well, what we do is we take our vocab.


So all of the words that we have in all of our documents, so A, B and C.


So if we so we create our vocab and all we want to do is we take a set of our three


documents, A plus B plus C.


And this is just going to create a set containing all of the words across all of our documents.


So a quick look at what we what we get from that.


So vocab, this is just all every single word that we have in there.


Now, to create our vector, what we want to do is take that vocab and mirror it into a vector.


So for every word here, we're going to calculate the TF IDF for each document and sort that in a


list, which creates our TF IDF vector.


So let me show you how that works.


So let's first initialize our vector.


So vector A and what we're going to do is say for word in vocab.


Vector A.


.append.


And then here we're going to call TF IDF and then we have our word.


So we define that as word and our sentence.


In this case, we're doing it sentence A.


So write sentence A like that.


And we can do this for each of our vectors.


Let's just do A and B for now.


So B here and here and here.


OK, and let's have a look at what we get.


So we have vector A and we see that we get essentially a vector.


Now, we don't have that many documents or that many words here.


So we're going to use the same method.


We're going to use the same method for all of our documents.


Now, we don't have that many documents or that many words here, so we're just getting


the same number here, which is meaning that it appears in one document and that document is A.


Now let's have a look at B.


So you get a few different values here and that's our TF IDF vector.


That's how we build it.


So that's TF IDF.


Let's move on to BM25.


Now, BM25 is a optimized version of TF IDF.


So one of the problems with TF IDF is as the frequency of queries found in a document


increase, the score increase linearly.


So imagine you have an article and in that 1000 word article, it has the word dog maybe 10 times.


Now, there's a good chance that the article is talking about dogs, right?


Now, if you double the number of words that mention that are dog to 20, the TF IDF score


for that document will double as well.


And that, if you think about it logically, is a document or an article that has the word


dog 10 times, half as relevant as the same article that has the word dog 20 times.


Probably not.


Maybe it's a look, maybe it is slightly less relevant, but not quite that much.


And that's why BM25 comes in.


It almost normalizes that value and we'll visualize that soon as well.


But let's explain this because it looks pretty horrific.


Now, up here at the top, we have very similar to TF IDF.


We have our essentially term frequency here and then we have our IDF here.


Now, in our term frequency, we can see that we have these two values here.


And this is the frequency of the query, frequency of terms.


And this is pulled straight from TF IDF.


But then we have a few new terms around here.


So we have K, which you can also see here as well.


Now, K and B here are adjustable special parameters.


K would typically be in the range of 1.25 by default.


And B around 0.75.


And we can modify these based on our use case to optimize the algorithm.


Now, another new one that we have over here is this D average.


Now, this D here, that is the current document length.


So I'll put D length.


And then over here, we have the average document length.


So that's all of our documents.


So it's all of the documents, their length divided by the number of documents.


So that would be N. Now, that is the TF component and how it's been modified.


And then let's have a look at the IDF component.


Now, the IDF component, we see these terms here and here, which are exactly the same


as the two terms that we have here.


So it's the number of documents versus the number of documents that contain that query.


And then over here, we just have a few constant values.


So although the BM25 algorithm looks pretty complex, it's not that much different from


the TF IDF algorithm.


So let's go ahead and implement that in Python.


Now, over in Python, we have these sentences or documents here.


We're going to add all these together in docs, just makes it a little bit easier for us.


And we still have this TF IDF function.


Now, I want to look at this and compare it to what we would do for BM25.


So the first thing is, let me just make this a little more readable.


So sentence account here, we turn it into frequency.


Okay, so change that to frequency like so.


Now we do the same here.


We sort our frequency.


And then this is our term frequency component of TF IDF and BM25.


Now let's have a look at these.


So we have k and b.


Those are our adjustable parameters.


The only sort of new thing we have here really is the average dl, which is the average length


of all documents.


So how do we calculate that?


We just calculate it outside function because this is for all documents.


It doesn't matter which sentence or which word we're looking at, it's going to be the


same for each one.


So we just sum the length of each sentence for every sentence in our docs.


In fact, this here, we can change that to docs like so.


So we change that.


Okay, now let's just run it quickly, see what we get.


So BM25, purple, and in sentence b, there is no purple.


So we would expect zero, which is what we get.


So we know this is at least working, right?


So let's do the same for sentence a and we should get a score.


So 1.76, which is what we'd expect.


So that looks good.


Now the other bit that is slightly different, this is really quite simple, is this bit here,


so the IDF component.


Now IDF up here, what do we do?


We are just taking the length of the documents, dividing by the sum here.


And that's exactly what we do here.


So this here, we could rewrite if we want to match to the below as n underscore q, like


so.


And that's the only real difference.


The other part is, so here we're using log to the base 10, and down here we're using


the natural logarithm.


So there's a small, there's also that difference as well.


But otherwise we're just adding the 0.5 and the ones and rearranging the n and the q.


And of course this here would be n.


Now that's BM25 and TF-IDF.


But what does that actually look like?


So when we compare these two algorithms, well TF-IDF, the score for TF-IDF increases linearly


with the frequency for number of words or number of matching terms.


So it goes up like this.


Okay, so that's TF-IDF.


Now BM25 is slightly different.


BM25, instead of going up like that, it does something which looks more like this.


So it increases a lot quickly and then it kind of levels off.


So if you have four words for BM25 or you have eight, eight will become slightly more


relevant.


The score will show that slightly more relevant, but not that much.


Now for TF-IDF, the difference is the relevance is doubled.


And depending on your use case, maybe that is more along the lines of what you want.


But I do believe in most cases, BM25 is probably more realistic.


So let's move on to our final one, dense vectors with sentence birds.


Now our final algorithm is S-Bird.


So S-Bird uses something called dense representations of language.


And what that means is rather than having a vector like we saw with TF-IDF and which


is the same for BM25, where it has a lot of zeros and then the odd number here and there,


which is called a sparse vector, we have something which has many more values in there.


So there's essentially more information crammed into a smaller space.


And the effect of this is that you can represent language in a more meaningful manner.


So the word, for example, hi, would be in a very similar space to the word hello.


And so would words like days of the week, so Monday, Tuesday, and so on.


And the way it's sentence bird works is that we have a transform model, we have a bird.


And our words or our query is processed, it comes in through here, it's processed by many


bird encoding layers.


And then we get this dense vector.


Now as well as our query, so Q, so are our documents.


They are also processed through the same encoder network.


And that produces dense vectors.


Now once we have both of those dense vectors, so we have this can be Q and then we also


have D. We use cosine similarity between both of those to calculate how similar they are.


So how close to each other they are, the angle between them.


And that works pretty much like this.


So we have over here, we have sort of a lone vector.


And then over here, we have two vectors which are much more similar.


They all at least share the same direction.


And these two have a much smaller angle between them than either of those two do with this


other vector, which is all the way over there.


And that's essentially how cosine similarity works.


It just finds the angle between two vectors and where the angle is smaller, they are more


similar.


And where the angle is greater, they are less similar.


Now we're not going to implement that from scratch because that would take a long time


and be pretty hard to do, to be honest.


So what we're going to do is use the sentence transformers library, which is a very, very


good library that uses Hugging Faces transformers under the hood and has super easy implementations


of sentence transformers, which is essentially what we just saw, but we're using it.


I was using the example of this specific one called sentenceBert, which is also what we're


going to be using here.


So we run this.


We're using this BertBaseNLIMeanTokens model.


And we have all of our sentences here.


So first thing we need to do is we initialize our model here.


And then after that, we encode all of our sentences with model.encode.


So we do that.


And we come down here.


So here we've encoded.


So what this has produced is the sentence embedding.


So once the text has been processed by our Bert model, it outputs these sentence embeddings.


And they are vectors that represent the full sentence or the full document that we have


input.


Now once we get those out into our sentence embeddings variable there, we use the cosine


similarity function.


Now here we're just going to use it.


We're going to use scikit-learn's implementation.


And what we're going to do, because we have many sentence embeddings here, let's just


have a look.


So we have sentence embeddings.


Let's have a look at the shape.


And you see that we have seven sets of embeddings here.


This here, this 768, is the size of a single sentence embedding vector.


Now we have seven.


Why do we have seven?


Because we have seven sentences up here.


So what we're going to do here is loop through each one and calculate the cosine similarity


between that sentence and all of the other sentences.


So run that.


And we will get this array, which shows us all of our scores between all of our sentences.


All possible combinations.


Now that's pretty hard to look at.


So let's visualize it using matplotlib.


We get this nice visual here.


So we can see here we have a on the left.


And it has a very high similarity to the vector a.


So I haven't filled those out or anything.


We are comparing the same vector to the same vector.


And as you would expect, they get a very good score because they are exactly the same.


They all score one.


So there's no problem there.


We just ignore those ones in the middle.


Now let's go on to the next one.


So we have these here.


So c and b.


OK.


Those two seem to have very high similarities.


Super high.


So let's have a look.


So b and c.


Now these are the two that both talk about throwing bananas.


Or actually no.


b talks about throwing bananas onto the street.


c talks about finding bananas on the street.


So yeah, they're pretty similar, right?


But they have a lot of the same words.


So fair enough.


tfidf and bm25 both identify those as being similar as well.


So that's good.


But it doesn't blow you away.


Now, how about b here and g.


So what I've done for g here is I've just taken the sentence b and swapped the words


around so that we don't really have matching words.


So throwing bananas instead of throwing, I'm using the word bombard.


Instead of the word street, I'm using road.


Instead of bananas, I'm using yellow fruit.


Right?


So it has a similar sort of meaning, but they're not using the same words.


So tfidf and bm25 would really struggle here.


Now let's go down and let's have a look.


So b and g.


So we have b here, g at the end.


And yeah, they have a really good score.


0.66 as their second highest behind b and c.


So that just goes to show that sentence transformers in general don't require the same words to


be used.


They rely more on the semantic meaning of those words, which is, I think, incredibly


cool.


So that's it for this video.


We've covered quite a lot.


The sparse vectors tfidf and bm25 and also dense vector sentence burr or sentence transformers


in general.


So I hope you've enjoyed the video.


Training BERT #1 - Masked-Language Modeling (MLM)

Okay, so in this video, we're going to have a look at what I think is the more interesting side of transformers,


which is how we actually train those.


So typically with transforming what we do is we download a pre-trained model from Hugging Face and


then at that point we can either use a pre-trained model as is, which in a lot of cases


it will be good enough to actually do that.


But then at other times we might want to actually fine-tune the model and


that is what I'll be showing you how to do here. So


core of BERT, there are two different training or fine-tuning approaches that we can use.


And we can even use both of those together.


But for this video, what we're going to have a look at is how to use a Master Language modeling, which is called MLM.


And MLM is really the


probably the most important of those two


core training approaches.


The other one being next sentence prediction.


So


what MLM is,


is we essentially give BERT a


input sequence.


like this, so this would be our input sequence and we ask BERT to predict the input sequence.


To predict the same input sequence


as the output. And BERT will optimize the weights within its


encoder layers


in order to produce this output.


Now


obviously that's pretty easy. So what we do


is we mask some random tokens within the input. So here we might mask one


and


what we do is replace that with another token,


which is a special token called the mask token.


Which looks like that. And


when we're doing MLM,


we would typically mask around 15 percent


of the input tokens.


So


if we take a look at how that looks, so this might look a little complicated,


but it's pretty straightforward. So down here


we have


our input from the previous slide.


We process that through our tokenizer like we normally would with transformers.


And then in the middle here, I haven't drawn it, but in the middle there's a masking function.


And that masking function will mask


around 15 percent of the tokens in the input ID's tensor.


So here we have a mask token and


they will then get processed by BERT in the middle here.


And BERT will output a set of vectors which all have the length


Usually there's different BERT models. They have different lengths.


We'll go with the 768 here and then we pass them through a feed forward network.


And that will output our output logits up here.


And


each one of those is of the size equal to the vocab size.


with this model, I think the vocab size is something around I think three


or thirty thousand five hundred, something like that.


And then from there to get the predicted token,


for each one of those logits, we apply a softmax function to get a probability distribution.


And then we


apply a argmax function, which is what you can see here. So this is just an example of one of those logits


over here.


We have the softmax,


we get the probability distribution,


and then we apply our argmax to get our final token ID, which we can then map or we can then decode using our tokenizer


to get an actual word in English.


So that's how it works. Let's have a look at how we actually do that in code.


Okay, so first we'll need to import everything we need. So we're using transformers here,


where we're using the BERT tokenizer and


BERT for mass lm classes. And then we'll also be importing torch as well. So from transformers,


import our tokenizer.


And


also our BERT for mass lm, which is mlm, the mass language modeling.


And then we also want to import torch as well.


Okay.


And then what I want to do here is initialize our two models, well our tokenizer and model.


And I do that


just as we normally would with Hugging Face Transformers.


So we do BERT tokenizer.


And then we also want to import torch.


So we do BERT tokenizer


from pretrained.


And here we have BERT base and case.


And then we also want our model, which is BERT for mass lm.


And this will also be from pretrained.


Again using the same model, so BERT base and case.


So that's our tokenizer and model. And I'm also going to use this example text here.


So


we see here, so this should be


election, this mass, and this one here should be attacked.


Now


we'll execute that.


I've done, I've made a typo here.


Okay.


And now what we want to do is actually tokenize that chunk of text.


So to do that we would write inputs. We have our tokenizer and all we do is pass our text in there.


We're using PyTorch here, so we want to return tensors.


PT.


Okay, and let's have a look at what tensors we return from that.


So you see we have our input ids, token type ids, and attention mask. Now


we don't need to worry about token type ids whatsoever for mln.


And attention mask


attention mask.


We mlm does use that but I'm not going to go into any details.


So all we want to focus on this video is input ids.


So


let's have a look at what we have there.


So there's a few things that I want to point out.


First we have our special tokens. So we have the CLS or classified token here. We have the separated token scp.


And we also have our mass tokens. One here.


And one here.


And everything in between are actual real tokens


from our text.


what we have now, we have our


inputs.


And what we do is use these inputs initially to create our labels.


But what I've done here is already massed


our inputs. So what I'm going to do is just actually replace these with the actual words.


So this is election.


And this one is attacked.


So just re-run that and that.


Okay.


And now what we can do with that is actually create our target labels.


So the target labels needs to be contained within a tensor called labels.


We should create like that. And it just needs to be a copy of this input ids tensor.


And to create a copy of that,


we write detach.


And then we clone it.


So that creates our copy which is


not going to be connected to our input ids.


And now if we just have a look at our inputs, we can see


input ids at the top and we have labels at the bottom. And they are just copies. Okay.


Now what we want to do is mask a random number of input ids or tokens within the input ids tensor,


but not the labels tensor.


Now to do that, what we can do is use the PyTorch random function and


using that, what we'll do is create a random array of flows.


Create a random array of floats that have equal dimensions to input ids tensor.


So all we do is we pass inputs, input ids, dot shape into there.


And if we can check the shape of it afterwards,


we get this one by 62, which is equal to this here.


And we can have a look at what we have there. It's just a set of floats between zero and one.


Now, if we want to select a random 15% of those,


what we do is we'll create a new array, mask array, and this will be equal to rand,


where rand is greater than or less than 0.15.


Okay. And this will select 15% of those.


And let me show you what that looks like. So this will create a Boolean array.


And here we say all of these faults.


And then these true values are, that's where we will put our mask tokens later on.


Now there's one here, and this one is covering our separator token.


Now we don't want to mask our separator or classified token.


We don't want to mask any special tokens.


So what we can do is add an extra little bit of logic there, which will look like this.


So we do inputs, input ids, and we say not equal to 101, which is our classified token.


And let's just have a look at what this looks like and see that now we get through everything


except for my classified token.


And we multiply this by the same rule, but for our separator token.


So now you see that we have faults here and faults here.


Now all we need to do is add this to our mask array logic up here.


And we also put brackets around this.


And this will make sure that these two are always faults no matter what.


Okay.


Now what I want to do is actually get the index positions of each one of these true values.


And to do that, we write torch flatten.


So this is going to just flatten the tensor that we will get out from this next bit of code.


And maybe it would make sense.


Okay, let's start from the first part of the code.


So we're going to go myMaskArray.


Here, that gets us our mask array.


We want to say non zero.


And that will get us a vector of indices where we have true values or non-zero values.


And what we want to do is convert that into a list.


Like that. But you see that we have a list within a list.


So this is where the torch flatten comes in.


So we add another bracket around this.


And we do torch flatten.


And then we convert it to a list.


And that gives us a list of true values.


And then we convert it to a list and that gives us a list of indices where we have these true values.


So that's our selection.


And now what we want to do is use that selection to select a certain number or select those specific


indices within our input's input ID tensor.


So we want to select the first part of that.


So the zero index followed by selection.


And we set those equal to 103.


And then let's have a look.


See if that works.


So 103 is our mass token, by the way.


And you can see here now we have those mass tokens in those positions.


So we've just massed roughly 15% of those tokens.


And then from there we can pass all of this into our model.


And the model will calculate our loss and the logits that we saw before.


So we do that as we normally would when we're using Hugging Face and Torch.


So we have models.


We pass our inputs as keyword arguments.


So look at what output is given us.


And we see we have these two tensors.


We have loss and we have logits.


Now let's have a look at what that loss looks like.


OK, so we get this value here.


So that is our loss.


And of course, with that loss, we can actually optimize our model.


OK, so that's how mass language modeling works.


Now, when we're actually training a model using mass language modeling,


obviously the code is slightly different.


But there's also a reasonable amount of depth that we need to go into for that.


So I'm not going to include it in this video, but I am going to do a video on that.


Actually training a model using mass language modeling pretty soon.


And I'll leave a link to that in the description because I know some of you probably want to


watch that to understand how to actually train your own models using this.


But that's it for this video.


I hope it's been useful.


How to use Color Histograms for Image Retrieval

Browsing, searching and retrieving images has never been a particularly easy thing to do.


Traditionally, many technologies relied on manually annotated metadata and searching through that using the more established or better established text retrieval techniques.


This approach works when you have a dataset that has high quality annotation, but this is really not the case for most larger datasets.


That means that any large image dataset has to rely on content based image retrieval.


Search with content based image retrieval relies on searching through the content of the image rather than relying on any metadata.


Content can be colour, shapes or textures or with some of the latest advances in machine learning, deep learning, we're actually able to search based more on the meaning of an image, like the human meaning rather than even the textures or edges in the image or so on.


But today we're not going to be looking at that. We're going to be looking at one of the earliest methods for image retrieval, which is colour histograms.


Colour histograms represent one of the earliest content based image retrieval techniques and they allow us to search through images and retrieve similar images based on their colour profiles.


And we can see an example of what we're going to learn about here. So we have this query image. It's just a city. You can see it's very blue and orange.


To the right here we have this. It was basically this is what the colour histogram is.


Now, when we actually build our colour histograms and search with them, we are going to convert them into a single vector.


So an image embedding. And with that image embedding, we're going to return these five images. Now, this is using a very small dataset.


We have I think it is literally 21 images. It's tiny. So with that in mind, this is returning pre relevant images.


When we speak about the colour content of these images, they all seem to have the similar sort of aesthetic, a lot of blues and oranges. So this is pretty much what we're going to do.


And underneath these images, you can actually see the colour profiles as well. So the one to the right is actually just a duplicate image.


It's the same image. But then the other ones we can see that they all have this kind of or some of them have this peak in the red and then they're pretty flat with the rest of the colours.


So this example demonstrates the core idea of colour histograms. That is we take an image, translate it into the histograms that you saw, which we then translate into a image embedding vector.


And then use our colour profile embedding to retrieve other similar images based on their colour profiles.


Now, there are many pros and cons to this technique. I mean, as you'd expect, it's one of the earliest techniques for image retrieval.


But there are a lot of benefits and we'll discuss a lot of those as we go through this video and just understand how to actually implement a colour histogram.


It will become quite clear what the pros and cons are if you haven't figured a few of those out already.


Now, we're going to work through two notebooks. And the first one is going to show us how we actually build colour histograms so we can actually see step by step the actual process.


And the second one is where we'll implement the search component. Now, you can find links to both of these notebooks in the description of this video.


So if you want to follow along, go ahead, open those and you'll be able to go through the code live.


So a few things that we need to install. So just pip install, we have OpenCV library, NumPy and datasets.


Now, OpenCV is just a popular computer vision library. NumPy, I'm sure you probably know what it is.


It's just focused on numerical operations for arrays and datasets is Hugging Face datasets. Hugging Face is like an NLP library.


And datasets is their way of allowing people to store datasets and download them super easily.


So for us, we are going to use that to get this image set here. Now, you can use your own images if you like.


You don't have to do this one. But this is what I'm going to go with. And this is, like I said, I think 21 images.


It's really not that many, but it's all we need for this example. So you can load datasets from datasets, load dataset and we have this pinecone image set.


If you'd like to see where this dataset is, you can go to Hugging Face.co datasets.


And then you just type in the dataset name that you see here, pinecone image set.


And it will take you to where this website here, which is where this dataset is actually hosted.


Now, inside, actually, let me run this. So inside here, we have this image bytes feature.


And this is just a base 64 encoded representation of our image bytes. So when we download these, we need to decode them.


Again, using base 64. So we do that here. So we're just going to create this processing function and we're going to decode everything.


I don't think there's really much to go through there. But from that, we'll get these images.


OK, and we can actually check. So let's just see how many images there are. 21.


OK, and this should align to appear as always the number of rows 21. OK, cool.


And we can display the images now with matplotlib. And you see we get this image with these three dogs, but they're pretty blue.


That's because we have loaded them, we've decoded them using the OpenCV library.


And OpenCV library, it reads images using the color channels blue, green and red in that order.


And I don't know why they do that. It's kind of the opposite way of what most things do, which is red, green and blue or RGB.


So what we need to do is actually flip those color channels if we'd like to see the true color version of this image.


So let me show you. So at the moment, this is a shape of that image. So this image is actually an array.


These two here are like the actual pixel values. So you can see 2560 at the bottom here and then you can see 1600 over here.


OK, so it's the Y axis or the height of the image. And this is the width of the image.


And then this three, these are the color channels. So it's the blue, green and red color channels.


And here are the color values. So blue, green and red for the very top left pixel right of there in the corner.


OK, so we see these values, by the way, so they go from zero, which is no color, up to 255, which is full color.


And we'll have a look at that in a moment. I'll show you what I mean.


So first, let's just flip those color channels. So you do NP flip. We are going to flip the whole array, but we are going to flip it from axis two, which is a color channel axis.


So let's do that. And you can see the shape is exactly the same because all we've done is flipped it.


But these, the pixel values here, so it was blue, green, red is now red, green, blue.


So now we can visualize that and we actually get the actual picture, which is these three dogs that are very much not blue.


So first, we're going to go through building a histogram this slow way, just so we can understand exactly what is actually going on.


So let's take a look at image zero again. So the three dogs and we're going to have a look at pixel zero.


So we have those values. Again, this is the other way around. It's blue, green, red.


Now, each pixel, like I said, has those blue, green, red activation values from zero, no color to 255, max color.


So if you had, let's say these were zero, zero and zero, there's no color or there.


That means we would get just black because there's no color. If we had 255, 255, 255, then we have white, which is just all the color you can possibly have in one.


And in between them, you have everything. So here's a few examples. We have blue, green, red and a few other things.


And I'm just going to plot those in this array here and I'll show you.


OK, so we can see we have blue, green, red and so on and so on.


OK, you can see all I'm doing is swapping the 255 for the orange one.


We've got like half of the green color in there as well. But that's pretty much all we're doing.


So every single color can be represented by these values.


So back to the previous values, we have that the top left corner pixel of that picture of three dogs.


We have blue, green and red. So you can think, OK, blue, green and red, that means it's going to be sort of a greeny blue color.


Which can be quite neutral because you have all of those colors kind of in the middle between zero and 255.


So we can see that here. So this this top left block here is actually that pixel.


Right, because here I'm just displaying the three by three pixels in the top left corner of the image.


And this is a true color image by the way. That's why it's an RGB image. So that's the one we flipped.


Now what we are going to want to do when we're comparing these images is actually we don't want an array.


We want a vector. So I'm going to reshape this.


And now we have these 409 or 4 million values, which is just all of the rows in our image.


All the rows concatenated together. That's what this reshape is doing.


Now if we plot that again, we can still see that those top three values are the same.


But there's no more rows underneath those pixels anymore. It's just a single row.


Now we have this one row, but it's still actually an array because we have the three color channels.


So we need to also extract those out.


Now we literally just have three vectors that represent our image and we can visualize each of these with a histogram.


So let's do that. And this is the color profile.


The red, green and blue color profile of that image of the three dogs.


So what we have on the X axis here is the color activation value from 0 up to 255.


And then what we have on the Y axis here is the count of the number of pixels that have that particular value.


So we can see this is a pretty neutral color.


Most of the values, and this is probably the case for most images as well, most of the values are kind of in the middle point.


So it's pretty neutral.


You can see that there's a lot of pixels here that don't have any blue in whatsoever.


But beyond that, I don't think there's much to note from there.


OK, so let's put everything we've done so far into a single function so that we can replicate these charts for a few images.


So what did we do before? We have our image.


We can also change the number of bins that we use.


So you see up here we've got an individual bin for every single color activation value.


We can push those together so that we have less bins and we are going to do that later on because it doesn't really affect the retrieval performance that much unless you go really low.


So first we're going to convert to a true color image.


So from BGR to RGB.


I'm just going to show the image so we can see what's actually happening whenever we call this function.


Convert it into a vector with the three channels.


And then what we're going to do is...


So here I'm breaking the values or dividing the values by the number of bins and then converting them back to integer values.


This is basically just a really quick way of creating the bins.


So if I divided this by two, for example, so I had two bins.


Here we'd get 128 in this division parameter.


And we divide everything by 128.


So we're kind of pushing everything together, all these pixel values into discrete categories or bins.


And then we want to get the red, green and blue channels.


And then we plot it.


So I'm going to run that and we're just going to try it on a few images.


So we'll start with this one.


So we have this image of a city.


And this is, I think, where you can see color histograms are a bit more interesting than the last one.


So it's a very blue image.


And we can see that here, like it is super blue.


And then we look at the histograms and the blue histogram, there's a lot of high values for blue pixels over here.


Now, what you can also do, which I mentioned before, is use the bins parameter.


So we just want, let's say we want 64 bins here.


OK, and you can see that we have these sort of bars now where you can really see those before.


That's because we have 64 values.


And we can reduce that even more.


Let's say we wanted to go really low and we want to go to two.


OK, and we just get these two bins now.


OK, and you can sort of see even from these two bins, it's a very blue image.


But of course, that's a little bit too much.


So we're sick with values between 32 and 64 because you can actually see what's going on with that.


Now, if you have a look at this image, we can see there's very little blue in this image.


It's a lot of green, a little bit of red as well.


If you take a look, we can see that almost all of the blue values are pushed right to the left,


which means there's very few high value blue pixels in this image.


We can do this again here. We see, OK, again, it's very green.


So we don't really get those blues.


And we can keep going through all of that.


See a few more.


And this one, this one's also interesting because it's a very colour specific image.


There's a lot of orange there.


And you can kind of see that because it just got these really big spikes in particular areas of your histograms.


So we can see that represented by the histograms.


Now, that's a slow way of building histograms.


Like I said before, we've gone through that to understand exactly what is going on.


But it's not the most efficient way of doing it because there are already functions for creating histograms


built into the OpenCV library.


So let's have a look at how we would use that.


So we have this CV2.


We've imported the CV library earlier.


Just here. So just import CV2.


And then what we're going to do is we create, we use this calc hiss function.


We pass in an image.


We pass in the colour channel. We can only do this one colour channel at a time.


Whether we want to mask anything, I'll explain that in a minute.


And so on, I'll explain the rest in a moment as well.


So we'll run that. And you see that we get this 64 by 1 shape.


And this is actually our histogram.


So we don't even need to use, before we were using matplotlib, the histogram function there.


Now we don't even need to use that. We can just plot this directly.


Now, there are a few things in here.


So we have this calc hiss function.


What are all these values?


So we have images, channels, mask, hiss size and ranges.


What does that mean exactly?


So the images.


So this is a list of CV2 loaded images with the channels blue, green and red.


So if you look up here, that's why I've taken the red histogram from the third channel or position 2.


And green obviously in 1 and blue in 0.


It can load multiple images.


So that's why we have put that single image inside the square brackets.


Where is it? Here.


And then we have channels.


So this is what channels you want to create your histogram for.


I'm wanting to extract one at a time here.


So I've said, OK, I want the red channel or the channel in position 2 and so on.


Mask is another image or array which just consists of zeros and ones.


Now that allows us to mask a part of the image if we would like to.


So imagine you had half of the images zeros, half the images ones.


It means that you would literally remove half of the image when you add that mask to this.


OK, imagine you multiply all the values in your image by the zero and ones.


The ones that become zero there.


You can't see them anymore because their colour activations are zero.


OK, so they just become black. That's how it works.


And then bins is the number of bins that we would like to add in there like we did before.


And then the histogram range is the range of colour values that we would expect.


Because we're using RGB we expect 0 to 255.


So we write 0 to 256 because the top value is not inclusive.


So that is not included so it actually just goes up to 255.


OK, and that's how we look at what we get from that.


So like I said, we don't need to use the histogram plot from Matplotlib anymore.


We can just plot it directly.


And yeah, we get this.


OK, so I think this is the last image.


Yeah, so it's this image here.


You can see that we get the same thing.


So that's the end of the building histograms notebook.


Let's have a look at how we actually create our embeddings with this.


And then how we actually search using these colour histograms.


OK, so we're now in the search histogram notebook.


And what we're going to do is create a function to basically do everything we've done so far.


So using the CV2CalcHist function.


We're going to use that and then we're going to actually concatenate the red, green and blue channels into a single vector.


So you've seen this before.


We have red, green and blue.


Then we concatenate red, green and blue together along axis 0.


And then we reshape this as a minus 1.


That's just to remove...


When you run this there's always like an extra dimension in there.


So we're just removing that extra dimension.


Now if we run that we should see that we get this.


So we have a 96 dimension vector.


Now why is it 96?


We just set the default number of bins to 32 at the top here.


So that means we get 32 values for the red channel, for the green value and the blue channel.


And then we concatenate them all together.


So in the end we have all those 32 together.


It's three of them.


So we get a vector of dimension 1996.


So...


If you also imagine this...


Let me visualise it even so we run this.


You can see here we have the vector.


The values from up to 32.


Here from 32 to 64.


And from 64 to 96.


And these are red, green and blue.


And we get this.


It's the same thing as what we saw before.


But we just separated them all into a vector.


Rather than an array with three colour channels.


Now let's go through and we'll use this loop to do that for all of our images.


So we're going to create all these image vectors.


And we can compare vectors with Euclidean Distance.


Although I found this didn't work very well.


At least not compared to Cosine Similarity.


Which we calculate like this.


So...


What we're going to do is use Cosine Similarity


to find the most similar matches for each image.


And we're going to put all this within a function to keep everything clean.


During visualisation we're also going to use the flipped arrays.


So the true colour arrays.


So we can actually see what they are a little bit better.


So I'm going to run that.


If I go through here...


All we're doing is getting the query vector which is marked by this index value here.


So for example those three dogs were at position 0 within our images.


So if I wanted to use that as our query vector I would just pass 0 to this function here.


And that will retrieve the query vector from the image vectors we've already created.


And I'm going to calculate the distance between that query vector and that query image.


And every other vector within our image vectors.


That also includes the same image.


It's not hard to remove that from here but just for the sake of simplicity I've kept it in.


Plus it tells us if this is actually working.


Because we should always return that one as the most similar image.


And then we're using this NumPyArgPartition function.


Which is just based on the distances we calculate.


It's going to retrieve those indexes.


And then we can use those indexes to retrieve the images.


The actual images themselves that are the most similar.


So we just run that using the dog image.


We see that we return the dog image first.


Oh sorry.


This is reversed so this is actually the first most similar image.


And that is the dog image itself.


And then we have these other ones.


We don't know what those are so let's go ahead and we're going to write another function.


This is not so important but this is just going to help us visualize these results.


So there's a lot of NumPy here.


We don't really need to go through this.


But you can go through this code if you want to.


So run that and then I am going to get some results.


Same as what we saw before and I just want to visualize those.


So here I'm actually doing it for another image so we can ignore that bit there.


So for image six.


Okay cool.


So we have this is the image that we're using as our query.


And you can see this is what I showed you at the start.


So it's the city image.


And we have these.


This is the color histogram for it.


And then these are the images that it's returning.


So very similar in terms of the color scheme for each of these.


Let's try another one.


Okay so this one I mean it's pretty obvious what this should return.


Probably another image as we can see.


I like that one so we have this yellow background with the dog.


And then we have the yellow background with the cat.


Okay and then some other images.


These ones are not really that similar but there aren't that many images in this dataset.


So that's why there aren't any others that are more similar.


And this one I think is really good because we can really see that these two color histograms are very aligned.


They're very similar.


If we want to just have a look at so at the moment we're using a default number of bins.


I'm not sure what the value is.


Anyway we can modify the number of bins.


So we'll go 96.


Let's see what we get.


Okay we get this.


So now we're modifying the bins.


We're still returning the same images though.


There isn't I think one of these might be slightly different.


Okay so this one here.


Actually reducing the number of bins here did damage the quality of this retrieval.


So in this case we still have that relevant image but we have these other images that are actually being placed before it.


Which is surprising but it's just one of the limitations of this technique is it's not particularly robust.


Now this is how we would actually search using these color histograms.


Now I think it's probably quite clear what some limitations of this are.


This retrieval technique isn't perfect and I think these results highlight some of those drawbacks.


So the key limitation here is we come up here and we can see we're returning this image of a cat with a yellow background.


But what if for your use case you're not actually looking for a similar color.


You're actually looking more for similar content.


So you'd actually want to return this is a dog.


I want to return another dog.


So probably this image over here.


In that case this is a really bad technique because it's basing everything purely on the color.


It's not even looking at textures.


It's not looking at the edges of an image.


It's not looking at what is inside the image.


It's just looking at the color profile of each image.


So it's very limited with that.


Now there was further work on color histograms to improve some of these.


So for example different techniques that consider the texture of images.


They consider the edges within the images and a few other things.


But still all of these things they're not going to get you as far as some of the more recent deep learning methods that allow you to actually consider what is inside the image from a very human perspective.


But that being said if you just want to return images or retrieve images that have a similar sort of aesthetic to a particular image that you're searching with.


So you want a similar color profile like we saw with the earlier results up here.


We are returning pictures that kind of look like the first image like the query image.


So that's one pro to using this technique.


Another one is that it's incredibly easy to implement.


We've just done it and in reality we don't even need to go through that sort of slow building a histogram part.


We can just do it super quickly using OpenCV.


And another key benefit is the results are very interpretable.


So with like a lot of deep learning methods you have a black box.


You put in some data. You return some results.


Why did you get those results?


A lot of the time you don't really know.


With this you know exactly why you're returning a particular result.


You can see that the color profile is very similar to the one the particular color profile that you're querying with.


So we understand what is actually happening here.


It's not a black box like neural networks.


So with all that in mind this approach to image embedding and retrieving images is great if your use case is looking at the aesthetics or the color profile of images.


If you want something more advanced of course you're going to want to go for something else.


Now that's it for this video.


I hope this sort of introduction to 1D earlier image embedding techniques and methods for content based image retrieval has been interesting.


But for now we're going to leave it there.


Training Parameters - TensorFlow Essentials #3

Hi and welcome to the video on training parameters. So these are essentially settings that we can use


to fine-tune the learning process and there are essentially three parts to remember.


We have the optimizer, the loss function and our performance metrics. Now the optimizer is used to


minimize the model loss and popular optimizers that you may have heard about are Adam or


stochastic gradient descent or SGD for short. And then we have the loss function. This is essentially


saying how do we calculate the difference between our predicted values and the true or real values.


And for this we use a loss function and the most popular of those that you will see is categorical


cross entropy. But there are a lot of different ones that you will see used in different places


and it would not be weird to use another one of those functions. And then finally we have the


performance metrics and this is what we use to measure our model's actual performance.


There are a lot of ways to do this but the simplest and often most useful metric is to just


use accuracy. So let's start with the optimizer. So we'll use Adam as it is the most popular.


And all we want to do is create our optimizer variable.


And then into that we pass our optimizer which is Adam. And then we also need to pass a learning


rate. So this is essentially a number which tells TensorFlow or tells Adam how quickly or how slowly


to update model parameters. So a larger number means larger updates whereas a smaller number


means smaller updates. This learning rate is actually one of the most important high parameters


in our model because we need to find a balance between not too high and not too low. And I have


an image over here to try and explain why. So you can see here we have three different plots.


First of those is where we have a very low learning rate. And what we have is a step which is


quite small which goes in the direction downwards on our slope. So further down, down here is where


we want to be because this is a lower level of loss and therefore a higher accuracy. So we want


to get down to here which is a global minima. But because our steps are too low we get to here


and our model thinks okay we need to go back because the gradient is negative in this direction.


So it goes back and then because the steps are too small it just gets stuck in this small little


local minima. Okay so where there's a dip but it's not the the lowest dip it's called local minima.


And even if the learning rate is perfect you can still get stuck in the local minima it's


completely normal. But obviously the lower the learning rate the more likely that is to happen.


So obviously you don't want to be too low. Alternatively we also don't want to be too


high because if our learning rate is too high then yes we will miss more of the local minima.


But once we get to the point where we are at a global minimum we can still get stuck in the


minimum. We want to go down into this sort of descent. Our steps are just simply too large to


actually do that. They can't go down because we need a smaller learning rate in order to actually


get down there. Otherwise what you would see during training is that your loss function or loss


value is just going up and down. It's sort of all over the place but it's not really going anywhere


it's just going up and down. And that is generally because your learning rate is too high. However if


we do get a learning rate that is sort of in the middle of these two and it's just right then we


will hopefully get a graph that looks something like this. So we don't get stuck in the local


minima because the step is large enough to miss it but it's not too large to not be able to converge


into the global minima which is what you can see here. And that is the ideal learning rate that we


want to try and get. A lot of the time it's just a case of trying things out and trying a few


different learning rates and see what works. But this is the logic between not going too high and


not going too low. So that is our optimizer which is in the optimizers attribute.


Now we need to go to our loss function. So the loss is the y-axis in the plot here.


So the lower the loss the higher the model performance or the higher the accuracy.


But obviously we need to actually calculate that. So we will create a loss variable.


And we do tf keras losses. And then we want to calculate the categorical cross entropy loss.


There's no other hyperparameters or anything that we need to add into this. So we're going to


add a tf keras. So that's all we need for that. And then we also want to initialize a accuracy


metric. So we'll put that into a variable called accuracy or acc.


Then we do tf keras metrics and categorical accuracy.


So the reason we are using categorical accuracy here is because in most cases you probably have


a set number of outputs rather than just a single output. So it will most likely look something


like this rather than like this. So when we have different categories we need to use categorical


accuracy. Then we just pass in the string accuracy there as well. So we've now set up all of our


training parameters and we just need to compile our model now. So we just call model compile.


And then we pass the optimizer loss and accuracy metric. So for metrics we also always put these


within the list because typically you're probably going to want more than one metric. But for now


we're just sticking with accuracy. We don't need to add anything else. But sometimes you may have


other things in here like an f1 score or some other metrics as well. And you'd want to pass all


of these in as a list. But we're just going to go with accuracy. And now that is our model. It is


completely ready for us to start training with it. But for now that's everything on the training parameters.


Identify Stocks on Reddit with SpaCy (NER in Python)

Hi, welcome to this video on Name Entity Recognition with spaCy.


So I'm going to take you through how we can use spaCy to perform Name Entity Recognition or NER.


And first, what we're going to do is take this text that you can see here.


And what we are aiming to do is extract all the organizations I mentioned in this text.


So in our case, with this, we are wanting to extract ARC.


So we're going to look at how we do that.


I'm also going to show you how we can visualize that process as well using this spaCy,


which is a visualization package embedded within spaCy.


It's super cool.


Now I'm going to show you how we programmatically extract entities.


So obviously visualization is great,


but we do want to just pull those out in a more programmatic fashion.


So we're going to do that.


And then once we have done that process with our single example,


obviously, we are going to want to scale that to a ton of examples.


So what I have is a sample of I think it's 900 posts from the Investing subreddit.


And we're going to build out a process that will take all of those,


pull out all the entities that are being mentioned,


prune out a few that we don't really want,


and then give us the most frequently mentioned samples or organizations within that data set.


So let's just jump straight into it.


We have our text data here.


And this is just a single extract from the Investing subreddit.


And I'm just going to use this to show you how spaCy works


and how we can do name entity recognition on this single piece of text.


We want to start by first importing spaCy.


And if you don't already have spaCy, it's very easy to install.


You just pip install spaCy and enter and that will install the module for you.


And we're going to be using both spaCy.


And I'm also going to show you something called displayCy,


which is like a visualization package that comes with spaCy.


So that is from spaCy, import displayCy.


And once we've imported those, we also want to load in our model.


So spaCy comes with a lot of options in terms of models.


And we can see those on spaCy.io slash models.


If we come down here, we see this.


This is the model that we will actually be using.


I just want to quickly cover what we are actually looking at here.


So come down here, we have the naming conventions and see that the name is built up of the language


and the name.


The language for us, of course, English, which you can see here.


And this last part is the name.


Now, the name consists of the model type, genre, and size.


The type here is core, which is a general purpose model,


which includes vocabulary, syntax, entities, and word vectors.


We're interested in using entities for the NER tasks that we are looking at.


Web is the type of data that the pipeline or the model has been trained on.


So the two examples I give here is web or news.


Web includes stuff like blogs and Reddit fits pretty well with that.


So we're going to use web.


And then we just have the model size here.


We're just going to go with small.


To download that, we go back into the browser.


To download that, we go back into our command line interface and we type this.


I would type Python M, spacey, to access the spacey module.


And then download and then your model name here, which in our case, so English core web small model.


I'm not going to enter this because I have already downloaded it.


But that is what you need to do.


And once that is downloaded, we can then load it into our notebook.


So we'll load it into this NLP variable and we'll do spacey load.


And then again, we just enter the model name.


There we go.


That is our model.


Now to actually process this data, it's super easy.


We will assign it into this doc variable.


And we just take NLP and add in the tips.


Now print that out.


Okay.


And we can see, okay, we have this, which just kind of looks like the text that we pass in there.


So it's a bit, it looks like nothing has actually happened, but that is not the case.


This is actually a spacey document object.


So if we click help doc here and you say, okay, we have doc object and then we've got all these


different methods, attributes, everything in there.


So it has worked.


And that's good because we can then use doc.ents to access the entities that spacey has identified


within this document object or within this text.


And we can see here, although it doesn't say what type these labels are, we have arc, arc,


arc, ETF, and this bear cave.


This doesn't tell us that much information, but the information is here.


So I want to quickly show you displayC because it's pretty cool and we can visualize what


is actually happening here.


So we just do displayC, render.


We pass in our doc.ent.


We pass in our document object and the style for the visualization.


There's a few different styles.


We are going to be using the entity style.


And this is pretty cool.


It shows us the text and then we have these labels on top and we see that arc and ETF are


identified as organizations.


ETF, we don't really want that in there.


ETF is a exchange traded fund and it's not really what we're looking for in terms of


the organizations.


Nonetheless, it is identifying arc correctly three times, which is pretty good.


Now, work of arc, when I first saw this, I had no idea what that meant.


To me, that seems like it's Picasso painting or a statue from Michelangelo.


You know, I really had no idea what it meant by work of arc.


So what we can do to get a small description of each label, if we don't know what it means,


is we just type spacey.


Explain and we just do work of art.


And then we see, okay, it's titles, books, songs, etc.


So that makes a lot more sense than what I was initially thinking.


And it also fits quite well to what this is.


So this bear cave thing here is actually an article and it's not quite a book, but it


is something that someone has written, just like a book or a song.


So it fits in with that category, in my opinion.


So that's great, visualize these entities and the text.


But we also want to process this in a more efficient way.


We can't just visualize it.


So this is where we go back to our doc ends here.


And what we want to do is actually work through each one of these in a for loop.


And although these look just like they are text or something along those lines, they're


not, they're actually entity objects.


So let me just show you how we deal with that.


So we go for entity in doc end.


So we can print out the label, which is this org or work of art.


And we print that out by accessing the entity object.


And going into the label attribute and just notice that there's a underscore at the end


of that attribute name.


Just remember that.


And that will give us the label, so org or work of art.


And then we can also find the entity text.


So we just go entity and then we can type in text there as well.


And then we see here, OK, that's pretty good.


So we can go back to our doc end.


And then we see here, OK, that's pretty cool because now we've got the organization work


of art and then we have what it is talking about, which part of the text is actually


extracting out for us there.


So that is really cool and really useful.


And that is actually all we need to start extracting the data out and processing it.


So if we just come down here and take this loop and let's see how to modify it a little


bit and we're going to extract the organizations from this list.


So we're going to initialize a org list.


And then here, we want to add some logic which says, OK, if this is a org organization label,


we want to add that to our org list.


So to that we say, OK, if label label.


It's equal to org.


Org list.


Append entity text.


And let's just view our org list at the bottom here.


OK, so here it's entity not label.


And here we get our list of all the organizations.


So it's excluded Bear Cave because Bear Cave is not an org, it's a work of art.


So that's pretty cool.


But ideally, from my perspective on what we want here is we don't need to have arc popping


up three times.


We just want to say, OK, what organizations have been mentioned?


We don't care about how frequently they've been mentioned in a specific item.


So to do that, we just convert this to a set, which will remove any duplicates.


And then we convert it back into a list.


So your org list equals list set org list.


And then let's just see what that looks like.


So we now just have ETF and arc.


And that's exactly where I wanted this to be.


OK, so we've applied this to a single piece of text.


But we want to apply this to a full data frame.


So first thing we need to do is actually import that text.


So I pulled this from Reddit.


So this is the data that we're going to be using.


OK, so we're pulling this from the investing subreddit.


And we're using the right API to do that.


Now, if you haven't used the right API before, I do have a video on that.


So I will leave a link to that in the description.


Otherwise, you can also just get this data directly if you don't want to go through the whole ready API thing.


And I will leave a link to that in the description as well.


So just separate this.


And now we just want to import pandas.


And now I just need to read in our data.


Let's pd read csv.


And this is in the data directory for me.


And is redditinvesting.csv.


And the separator we're using here is the pipe delimiter.


So let's just make sure we've read that incorrectly.


And there we go.


So we have our data.


And the thing that we really focus on here is this self text column.


So in here, we just have 836 posts.


And we'll just apply our NER to all of those and just see what people are talking about.


So we need to convert what we did up here into a function that we can then apply to our data frame.


So let's take that and just going to convert this into a function.


So we'll call it get entities.


And then here we'll pass in a single string.


Add that in there.


And we'll say here we need to create our document object.


So we'll just call it document.


Create our document object. So NLP text.


We've already defined the model up here as NLP.


Is this variable.


Initialize our entity organization list.


And then work through each one of those and append it to our list.


And then we just want to return that list.


But we also do want to remove any duplicates.


So we'll just return the set list version.


Now we can run that.


And let's just apply that to our data frame.


So we'll create a new column called organizations.


And we will just take the self text column and apply our get entities function to it.


Let's just see what we get.


So this will take a little bit of time because we're processing a lot of entries here.


Obviously if you're doing this for a larger data set,


you're probably going to want to batch this a little bit.


So keeping it on file somewhere.


Reading maybe up to a thousand samples at once.


Applying this and then saving it back to file and just working through like that.


So for us, we can see straight away we have some things that we probably don't really want in there.


So I'm not sure what these are.


And then we also have this SMP 500 PE.


SMP 500, you know, loads of things.


So it's not really what we want in there because we just want actual company names.


So what we can do is create a black box.


So what we can do is create a blacklist.


What I mean by a blacklist is we just create a list full of anything that we don't want to be included.


For example, these here, we really don't want those.


Now we don't necessarily need to do this as well for everything


because what we will find with a lot of these items that we don't really want to include in there.


In fact, actually, I think I'll probably keep these two in as an example.


What we will find with a lot of these is that they only appear maybe once or twice in the whole data set.


So we can actually filter those out by only searching for organizations that appear at least sort of three or four times within our data set.


That just filters out all the rubbish that we get with these ones.


But in other cases like SCC, that will appear quite a lot.


And we don't necessarily want to be finding where it comes up with the SCC.


And in some cases, maybe you do want to, but in this case, I'm going to remove it.


I'm going to remove the SMP 500 as well.


And maybe leave it like that.


I'm not sure where to, I assume Lemonade isn't a company.


So I'm just going to put that in there as well.


And then there's a few others that I've noticed before that come up quite a lot.


We get like the FDA, Treasury, Fed appears all the time, CNBC always appears, EU always appears.


And I think that's probably a fair few, the ones that we don't want in there.


So we'll include those.


And to exclude those from our search, we just add another condition here.


So we add an AND condition and say AND entity text.


And you'll see here everything is in lowercase.


So we also apply a lower here.


So this means we don't have to type out Fed in capital and Fed in lowercase.


We do entity text lower, not in the blacklist.


And that would just exclude any that are included in there.


And then we can just update that as we go along.


So just rerun that and we'll rerun this as well.


And we start writing out the next part of what we're doing here.


So what I want to create is essentially a frequency table.


So we want to have each one of these companies and we want to see how often they're going to


how often or how frequently they are mentioned.


So to do that, we can use a counter object from the collections library.


So what we can do with that is we simply pass a list, for example,


and it will go through and count all the instances of a specific value.


And then organize them into the counter object,


which gives us a few useful methods for actually viewing that data.


For example, viewing the most common values in that data set.


So that's pretty useful. That's what we are going to be using.


So to use that, we need to import it from the collections library.


That's the counter object.


And like I said before, this needs a list.


And at the moment, we have a column in the data frame.


So it's not really the right format that we need to transform it into a counter object.


Instead, what we need is just simple flat list.


So first thing we can do is take that column and convert it into a list.


So we do organizations.


To list.


You can see here, okay, we do have list, but it's actually a list of lists.


So we've got a list and within that list, we have all these lists.


And we don't want that for our counter object.


We actually just want a plain straight list.


So we need to add in those step to the process, which is flattening that list.


So we'll call it orbs flat.


And here we're just going to use list comprehension to loop through each list within the list


and pull out each item into our new list.


What I mean by that is org here is like a separate list.


It's like a single item within the sub list.


So if I just view the first two here.


So org is like the SMP 500 P here.


And then that will be our item that makes up this new list that we are making.


And they will come from a sub list.


And these sub lists are these lists here.


And we need to iterate through each one of those sub lists.


For each one that is within our orgs list, which is the full thing.


And at the end here, we're just saying go through each org.


So each item in the sub list, which is kind of a confusing syntax.


But it works.


And it's just something that you get used to if you're not already.


So then let's view the first five entries in that.


So it's orgs flat.


And there we go.


We have a few companies.


So now we can pass this into our counter object.


So do frequency counter orgs flat.


Okay.


And then we can view the most frequent of those by using the most common method.


And then here we just pass the number of the most common items that we'd like to see.


So if we'd like to see the top 10, let's pass in 10.


And then here we can see that we have the most frequently mentioned organizations from the


investing subreddit data that we have.


There's a few things in here that we probably wouldn't get into.


Like EV, ETF, COVID, you've got socket strange, SPAC.


There's a few items in there that we can definitely prune out with the blacklist.


But overall, I think that looks pretty good.


And this very quickly shows us how easy it is to apply named entity recognition to a data set to


actually extract what the text within that data set is actually talking about.


Now, if you start pairing this with things like sentiment analysis, it can get pretty cool.


So, I mean, that's definitely something that I think we will cover soon.


But for this video, I'm just going to leave it with NER.


So I hope this has been useful.


Using Semantic Search to Find GIFs

Today we are going to have a look at how we can use semantic search to


Using Semantic Search to Find GIFs

intelligently search GIFs. And I think the best way for me to explain what I


Using Semantic Search to Find GIFs

mean by that is to simply show you what I mean by that. So here we have very


Using Semantic Search to Find GIFs

simple streamlit front end for our search pipeline and I'm going to


Using Semantic Search to Find GIFs

search for dogs talking on the phone. Now this is a natural language query.


Using Semantic Search to Find GIFs

There is some dependence on keywords here or there will be some keywords


Using Semantic Search to Find GIFs

that are shared. You know we have dogs, phone. But because we're using semantic


Using Semantic Search to Find GIFs

search we can try and avoid using these keywords. So if we go for something that


Using Semantic Search to Find GIFs

has the same meaning but just doesn't include the word dog. So for legged mammal.


Using Semantic Search to Find GIFs

We see that we actually still return this one here. We've lost this one.


Using Semantic Search to Find GIFs

It's a little further back now. But for legged mammal talking on the phone still


Using Semantic Search to Find GIFs

works pretty pretty well. Now that's pretty cool but what I want to really do


Using Semantic Search to Find GIFs

here is show you how you can do this. And in particular okay this is GIF search.


Using Semantic Search to Find GIFs

Maybe it is actually pretty useful particularly when you are searching


Using Semantic Search to Find GIFs

for GIFs to use on social media and so on. But I think the real power of


Using Semantic Search to Find GIFs

this is its potential. We can use the same pipeline to do a lot more than just


Using Semantic Search to Find GIFs

GIF search. We can adapt it super easily to financial documents, image search,


Using Semantic Search to Find GIFs

video search and a lot more. So let's just have a quick high-level view of


Using Semantic Search to Find GIFs

what this pipeline actually looks like. So we start with our query at the top.


Using Semantic Search to Find GIFs

So dogs on the phone or dogs talking on the phone. Okay that is our query. It's going to


Using Semantic Search to Find GIFs

come down here and it's going to go into what we call a retriever model. A


Using Semantic Search to Find GIFs

retriever model is basically a transformer model. Let me call it


Using Semantic Search to Find GIFs

retriever. It's a transformer model that's specially trained to


Using Semantic Search to Find GIFs

output meaningful vector representations of that text. So from here we're going to


Using Semantic Search to Find GIFs

get this big long vector. Plenty of numbers in there. I think the


Using Semantic Search to Find GIFs

retriever we use is going to output a dimensionality of I think 374 if I'm


Using Semantic Search to Find GIFs

not wrong or something along those lines. That's actually a very small vector.


Using Semantic Search to Find GIFs

Usually they are much larger. So we get that vector embedding and it's going to


Using Semantic Search to Find GIFs

go into a vector database. Now the vector database for us is going to be Pinecone.


Using Semantic Search to Find GIFs

And in Pinecone we have all of these. So the GIF you just saw, each one


Using Semantic Search to Find GIFs

of those has a little description. Okay so the one you saw before might say I


Using Semantic Search to Find GIFs

think it says something like vintage dogs talking on the phone or something


Using Semantic Search to Find GIFs

along those lines. And we take that description, we pass it through the


Using Semantic Search to Find GIFs

retriever model, so the same retriever model we use here, and store it as a


Using Semantic Search to Find GIFs

vector embedding inside Pinecone. Now this has been done for hundreds of


Using Semantic Search to Find GIFs

thousands of GIFs and their descriptions. So then what we do is we look inside


Using Semantic Search to Find GIFs

Pinecone so we have all of these different GIF descriptions and what we


Using Semantic Search to Find GIFs

do, we're going to use a different color here, we introduce our query. So this is actually


Using Semantic Search to Find GIFs

this vector here. And we say okay which of these other items are the


Using Semantic Search to Find GIFs

closest to our query and maybe we say okay it's these two here. So what we do


Using Semantic Search to Find GIFs

is return those over here and we don't really care about the vectors themselves


Using Semantic Search to Find GIFs

but we care about the metadata that's been attached to those vectors. So in


Using Semantic Search to Find GIFs

that metadata we are going to have the URL. Okay so the URL where we can find


Using Semantic Search to Find GIFs

that GIF image or the GIF file. So that's included in here. Okay so that comes


Using Semantic Search to Find GIFs

along with us and we use that to display the GIF on our screen. So that's what you


Using Semantic Search to Find GIFs

saw before. We just have some HTML, it's just like an image tag and we just


Using Semantic Search to Find GIFs

have the URL in there. Okay and we display them. And that's what we are going to


Using Semantic Search to Find GIFs

build. So let's take a look at how we can do that. Now you will be able to


Using Semantic Search to Find GIFs

find all of this code in a link in the video description or if you're watching


Using Semantic Search to Find GIFs

this on the article it will be at the bottom of the article. So with that in


Using Semantic Search to Find GIFs

mind I'm not going to go really in depth on the code. I'm just going to go


Using Semantic Search to Find GIFs

through it quite quickly and just give you an idea of what we're actually doing.


Using Semantic Search to Find GIFs

So the first thing we would need to do is install any of these libraries if we


Using Semantic Search to Find GIFs

don't have them installed already. So we have our connection to Pinecone here. We


Using Semantic Search to Find GIFs

have Sentence Transformers which is our retriever model. TQDM is just a progress bar.


Using Semantic Search to Find GIFs

Pandas is just Pandas data frames. Now here not really that important if


Using Semantic Search to Find GIFs

you're building a app but if you're doing this in the notebook you will want


Using Semantic Search to Find GIFs

this because this will allow us to display HTML within our Jupyter


Using Semantic Search to Find GIFs

notebook which will seem important if we want to see what GIFs we are returning.


Using Semantic Search to Find GIFs

And then we obviously need a data set. So we have this data set here and if we go


Using Semantic Search to Find GIFs

down we see it's a Tumblr GIF data set. 100,000 animated GIFs and 120


Using Semantic Search to Find GIFs

sentences describing those GIFs. Now what that means, so there's a bit of an


Using Semantic Search to Find GIFs

imbalance there, that means that there are multiple, in some cases not all, there


Using Semantic Search to Find GIFs

are multiple descriptions for a single GIF and we can go down here we can we


Using Semantic Search to Find GIFs

can have a look at that in a moment. But this is the data set structure. We have


Using Semantic Search to Find GIFs

the URL and then we have a description and we'll print these out in a moment.


Using Semantic Search to Find GIFs

So let me show you this first. So we come down here and this is an example right.


Using Semantic Search to Find GIFs

So we have this image which is from the URL. So I've pulled that in from the URL.