where we have two images that are exactly the same okay so these appear in the middle here


Bag of *Visual* Words for Image Classification and Retrieval

the cosine similarity for those is going to be is going to be one okay and here and here


this image there's maybe there's like a building in the background there's some trees so it's kind


of maybe halfway in between the church image and the dog image now because the dog image has some


trees the church image has some it's a building so it shares some similarities both of them with this


this image so it would probably be somewhere in the middle of that similarity scale okay somewhere


around here and then images like the church and the dog they there's not really any similarity


there so they have a lower closer to zero cosine similarity okay so that is the


the theory behind bag of visual words and how we actually retrieve similar images using bag


of visual words now like i said the two parts this video now onto the second part which is


where we're going to have a look at the implementation this all of this in python now


if you want to follow along with this uh there will be a link to a colab notebook with all of


this in in the description or if you're reading this on pinecone in the resources at the bottom


of the article so what we first need to consider okay we need a data set of images so we're going


to keep this super quick light and simple first we we get a we're not uploading so we download


the hooking face data set we're using this one so it's like what they describe it as a french


image net i'm not sure what makes it french but it's just how they have described this


but it's a almost 10 000 pretty high resolution images that we can we can just download immediately


like this is all we do and we can see straight where we have that church image okay and yeah


we can see all the images just by printing them like this so first thing we want to do is translate


these images so these are what are called pill image objects at the moment we want to translate


those into numpy arrays so we can do that we just go through the entire data set and we create this


array or this list sorry of items which is going to be a list of numpy array versions of our pill


objects now these images some of them are gray scales some of them are color we're going to drop


the color from these images and we're just going to have grayscale so we can do that here so we


use the open cv library okay and we just go through and we convert all of these images if they are


rgb images to grayscale if they're already grayscale we don't need to transform them now how


do we tell if they're grayscale or not well the shape of a of an image that is grayscale


is different so let me show you images okay so we see here that we have this


two-dimensional array okay whereas if we had a look at so in images training these are all the


color images that we still have if we have a look at beforehand it's a three-dimensional image because


you also have the red green and blue color channels included in that array so that's how we


can tell that's why the the length of the image shape here when it is two we know that it's


greater than two sorry we know that it is a red rgb image whereas if not it's grayscale


so we can see okay these are the new grayscale images now the next step now that we have our


images and they're ready and pre-processed we need to go through that process of creating our


eventually creating our sparse vectors so we start by creating the visual features now we move on to


visual words and then we create the the frequency vectors and then we create the tf idf vectors okay


so number one visual visual features we do that like so so it's actually pretty simple we are


using the using sift from open cv and what we're doing here is we're creating a new


using sift from open cv and what we're doing here is just initializing lists where we're going to


sort all of our key points and descriptors so remember we have both uh the key point is kind


of pointing to where the where the visual feature is and the the descriptor is a description of the


visual feature 128 dimensional vector and yeah we would so we initialize this extractor the sift


extractor and here we're just for every image in our black and white images we're going through


we're extracting the kim key points and the descriptors and we're adding those key points


and descriptors to our big list of key points and descriptors here now you notice that we're


doing this image by image so in here we're actually going to have um so for example the


descriptors um we're going to have maybe i'll show you in a moment actually um i'm sure i will show


you later why don't i show you now so let's have a look at the descriptors for i know say image 100


um let me have a look at the shape so for image 100 let me see that we have 436 descriptors


and each of those descriptors is um of dimensionality 128 because it's a sift descriptor


okay now this won't happen with this sample but just be aware that some images um for example


imagine you have like a picture of the sky and there's not really anything there it's just like


the color of the sky um your the sift algorithm might just return nothing because there's no


nothing for it to there's no key points for it to look at there's like no change just a flat image


in that case the the image the sift descriptors will just be none um so we have to be careful


in the next step and just avoid those um so and so what we're doing here if there are no descriptors


in a particular image descriptor list um we just drop it we just drop it and we're just


drop it we just drop that image okay but for this sample it's not really a problem but


just letting you know because in other data sets you might come into this problem okay so um


yeah all we're doing now we're just dropping any of those descriptors that don't have any


descript any of those images that don't have any descriptors um as i said not an issue for this


for this sample now let's have a look at what these what these key points and descriptors


actually look for look like um so these here so these are the key points so in each in the


middle of each of these circles um we have we have a descriptor okay so we have the sift algorithm


as identified a visual feature in each one of these circles okay and we see here and here and


and so on all right so the the center of these circles is the position of the key point uh the


line that you can see is like the direction of the the feature that was detected and the size of the


circles are essentially like the scale of the feature that was uh was detected so with all of


our we have all of our visual features now remember in that list we had visual features


visual words um frequency vectors and tf idf vectors now we're on to the visual words part


so first um all we're doing is creating 1000 random items a random we're going to use this to


select 1000 random images we've added this random seed in here so you can reproduce what we what we


do here now the reason that we are taking a sample in the first place is just to speed up the k means


part of this the training part you can leave this if you want and just train everything it just it


will take longer and so we extract the descriptors and the key points for our training sample


and we put everything together all the all the descriptors when we're training k means we don't


care which image a descriptor is in all we care is that the descriptors are there so we take all


of our descriptors and we put them all into a single numpy array we can check the shape so we


have is it one point one point one million of these descriptors it's quite a lot um for less than 10


less than 10 000 10 000 images i think i think it's fairly large and yeah so um here i'm just


showing you okay number of descriptors contained in each image um sorry each image um so we have


like image zero here it has 544 descriptors second one quite a lot more quite a lot less


and so on it's kind of you know each image has a different set of items in it so it's normal


now you have to set the number of visual words that you would like um from your from your images


so it's like the vocab size in nlp were in nlp terms nlp lingo um for this i just set k equal


to 200 so we're going to have 200 visual words um from this we don't have that many images uh if


you have more images you should probably increase this particularly if they're very varied in what


they are if they are not varied they're all kind of like the same thing you can probably use less


use less um visual words and yeah so we are performing k means here we just do for one


iteration and that outputs our codebook which is essentially just a big list of all of our


all of our clusters now uh if you are i don't know maybe you're doing you're creating your codebook


on on a machine and then you're going to be using it later on like over and over again i figured


it's probably very important to show you how to actually save and load that codebook now it's


pretty easy anyway it's not it's not like it's difficult so we use joblib here and we just


dump k which is you know the the number of visual words that we have you don't even necessarily need


to put that in there you could just use the codebook alone and we're going to save it into


this pickle file and then when you're loading it at later later point we just use joblib load


there's nothing else to it now at this point we have here we've we've constructed our codebook


um so we have gone we've finished we've created our visual features we created our visual words


um and they're all just stored in this codebook and now what we want to do is take the visual words


visual features from our images process them through the visual words that we have in our


codebook and use that to create our sparse frequency vector so that's what we are doing here


so we go through each of our images in descriptors or the set of descriptors image descriptors in


descriptors and what we do is we use the vector quantization quantization function from from scipy


to extract the the visual words from those and what that's going to do it's not going to extract


the actual like the the vectors of visual words you're going to extract the like ids of the visual


words um so we actually get all of these id values each of which um represents one of our visual


words like it's like a token right in nlp you have word tokens um in in bag of visual words you have


visual word tokens uh so that's what we that's what we have here okay so in image zero uh we know


that the first five of our visual words um have these tokens okay so number 84 22 and so on and


and in total there are five hundred three hundred and ninety seven visual words in image zero okay now this here so


we can take a look at the the centroid so the the vector of the visual word by looking at the


codebook we go to visual word or use visual word id index 84 and we can see the shape that is 128


okay that's like our centroid position now how do we do the we need to do like a frequency count so


how many of each visual word um based on the visual word ids how many of those are in each image okay so


uh we know so we know that the length so that's how look at the length of our codebook should be 200


okay so what we're going to do here is we're going to create a frequency vector each one of these is


going to be of dimensionality 200 because that's all the possible visual words within our codebook


okay so we're going to loop through all of our visual words and we're going to initialize the


frequency vector for each image and then we're going to loop through each word found in the


each word found in the image's visual words and for each one of those visual word index values


we use that to add a one in the position of that visual word in the image's frequency


uh vector okay after we've done all that we we have basically we have a single


frequency vector for every single um image each one of dimensionally dimensionality 200


and at the end here we're just reformatting that big list of numpy vectors into a single


numpy array okay and at the end we can see that we have 9469 of these frequency vectors each one


dimensionality 200 aligning to our codebook now from above we saw that we had these uh we had


these visual word tokens in image i think image zero yeah so we had these visual word tokens you'd


see we have one two one seven two twice here okay because that that visual word appears in the image


more than once um so let's go down here and let's have a look at each one of those so actually i


don't know why i needed to count i just copy and pasted it across um so we can actually even remove


that if we want but i'll just leave it so if we have a look at those we should we would expect


all of these values to have at least a value of one because we've already seen that they are there


and for one one seven two we know there's at least two of them in there um and we can see that here


so we look at the position frequency vector zero uh we look at each one of these positions and we


can see that there are numbers there okay and let's have a look at more of image zero's frequency


vector so we we have this here we can increase this we can show everything i mean so we have this


200 values uh each of these are the frequency of those visual words in that image and we can


represent that in a big histogram like this okay now we want to move on to tf idf part so you know


we have all of these um these frequency counts in a big vector but some of those visual words are


irrelevant they're like the vision equivalent of the word the or the word a or the word and


we don't want those in there they're not not important they tell us anything


so we use tf idf to reduce the the score of those items reduce their sort of dominance in our


frequency vectors so we are going to use the the tf idf formula that you can see here um so first


we want to calculate um once calculate well and we already know it's the number of images e.g.


the size of the data set um and then df is the number of images that a visual word actually


appears in okay so basically what we're doing here is we're looking at all of the frequency


vectors and we're saying okay if there's if there is um in a single vector if the value is greater


than zero e.g. one or more e.g. that visual word appears in that particular image we will assign


that position one okay and then we take the sum across all of those um all of those frequency


vectors along the zero axis so basically let's say we have visual word zero um in image zero


maybe that visual word is uh the frequency is three in image one the frequency is two and in


image three the frequency is zero if we perform this operation across those just those three


frequency vectors the value in that from that first column would be equal to two because it


only appears it is only not zero in two of the three images or frequency vectors okay so we can


see what's happening here so df shape df shape it's still going to be the same um we are going


like uh we have the 200 dimensional frequency vectors we have 9000 of those um and we're


compressing those 9000 into a single a single um i don't know what we call this like a df vector


okay um now let's have a look at this so the inverse document frequency i suppose we could


call it the inverse image frequency um is just the log of n over df okay nothing nothing special


there and then uh to calculate tf idf we also need the term frequency part of tf idf now the


term frequency part we actually already have those frequency vectors that we've been uh we've been


building they are the term frequencies so that's that's all we do is the frequency vectors multiplied


by idf idf by the way we can see here it's just a single vector for all of our entire data set


okay the only part that changes is the frequency vectors okay and we get this now if we plot that


you can see now this is the same the same image as before and so if we go up you see that we had


all of these like kind of high values now we come here and there's just this this really high value


here maybe that's a visual word of the the cross on top of the church that's probably like a


distinguishing feature if we're looking for maybe looking for religious buildings or churches or


something like this lines um whereas the rest of the visual words kind of get pushed down because


most visual words are probably not going to be that relevant okay now we've generated our tf idf


vectors here okay so we have the more stored in tf idf here i'll just point out here we have just


over 9000 of those and we can begin searching so to search we're going to use cosine similarity so


we're going to solve this this image of a dog to sort cosine similarity we calculate like this


now we're going to search through the entire search space for b so in this case b is not going


to be a vector and we're just going to calculate the cosine similarity for all of the all of the


items when compared to a so one of those items is going to be the exact match to a because


a for search i so set maybe search i is what is it 6595 that image is still within b here


so we're going to get at least at the top we're going to get that image returned to us first


okay so the minimum and maximum cosine similarities zero to pretty much one we're


using floating point numbers so it's not actually exactly one but it is basically one okay i'm going


to go quite quickly through this so we use numpy argsort to get the most similar items


the first image should obviously correspond to the image itself we don't expect the cosine


similarity to be pretty much one and yeah that's what we get here so we're going to just display


a few of those so using this as our search term we get so using this was our search term now this


is the first result cosine similarity is equal to one go down we get another dog pretty similar


looking we get this you know it's not it's not always perfect and then we get more of these uh


very similar looking dogs here okay now that looks pretty good so we're going to test for


a few more images uh we have this one here it's like some old record player at first we get the


same record player and i don't think this record player actually appears again in this state set


but i thought it was interesting because we get this printer and we get this image


because we get this printer um i assume it's a printer that looks very similar it has this


like top bit on top and despite the background being very different and you know this one's just


like an image a picture like a blurry image it manages to pull through this object that to me


looks pretty similar and then the rest of them is nothing uh of relevance maybe this one's like a


box it's kind of similar okay this one uh we search for the sky with a fish uh obviously we get


the exact image back first and then we get the same guy with fish in a slightly different position


it's like open his mouth a little bit as you go into the side and get another guy with fish


and then here uh i thought this was kind of interesting so here okay there's no guy with fish


but i mean you look at the background of these images right the background these images is like


a some like leafy foresty place um and obviously here leafy foresty place so kind of makes sense


that it would pull through this image here as well uh for this one i suppose there's kind of


like a forest in the background but otherwise i don't know why it's why this one is being pulled


through this one i thought was interesting so this one didn't perform well on or at least it didn't


um return what i would expect it to return okay so we have these two parachutists and then we get


golf balls a dog of the golf ball um but if you look at the this parachute here it has this like


really thick um black line and then the rest of the parachute is probably white and here on this


uh on this golf ball you have like a very similar black line so i assume for these uh you know


these golf balls has actually seen that and been like okay that's uh they're two very similar


features we have another one here we have this another golf ball um return obviously the same


one again and then return of the golf ball like another golf ball the same brand as well and these


as well and then we return this truck i thought maybe the reason it's returning this truck is it


like this white ridge pattern on it which the golf balls also have a similar sort of white ridge


pattern i wasn't too sure um but anyway that one performed quite pretty well i thought


this one we have this garbage truck the garbage truck here obviously return the garbage truck


and then we return this image and initially i thought okay it's not very good um first result


back but then we get a garbage image here then if you actually look at this image


we look in the background uh that actually does seem to be a garbage truck here it's


either a garbage truck or like some big industrial bin i'm not really sure um but i thought that was


kind of cool that i picked up on this thing in the background i i missed the first time i looked


at this as well um so that's cool uh but yeah so there are all the results as sort of go through


those quickly to have a look at how it how it did and you can probably optimize this to be much


better than what i have here now we use a very small data set for example um but we've done


something pretty cool i suppose we used k-means so it's lined a little bit there um in terms of


the visual words but otherwise it's it's very simple okay and for me to see this sort of


performance on you know without like some big neural network or anything um i think it's it's


pretty cool so i hope this has been useful um thank you very much for watching and i will see


Input Data Pipelines - TensorFlow Essentials #4

Hi and welcome to this video on TensorFlow datasets.


So TensorFlow dataset is an object type that TensorFlow supplies that essentially makes our


job a lot easier when building an input data pipeline. First, when we are actually putting


our data into our model, a dataset object makes it much, much easier. And we literally just feed in


dataset to our model training and everything, as long as it's set up correctly, will run very


smoothly. Second, we can batch and shuffle our dataset with a single line of code, which is


honestly incredible. And finally, we can also adjust our dataset very easily without taking


up too much disk space. And alongside all of those, the dataset object is incredibly well optimized.


So it's definitely well worth learning how to use this. So we'll just open up a new Jupyter


notebook here. And we want to import TensorFlow, pandas, and NumPy.


So there are a few different ways of reading in data into our dataset. The first of those is from


in memory. So if we, for example, have our inputs and labels, which are our outputs,


we can very quickly create a dataset with this, like so.


So we just pass a tuple containing the inputs and labels.


So incredibly easy. So now we can loop through the dataset to see what we have inside.


Okay, so we can see that we have these tensor objects, which are the data types or the array


types that TensorFlow will read. And we see, okay, it's a NumPy integer, integer 32 here,


and the value is zero. And we have one, two, three. Okay, so these are obviously our inputs.


Now on the other side, we have our labels, where we have NumPy one, zero, one, and zero.


So incredibly easy to get that set up. But obviously, we're not always going to read in


as a list. We can also read in as a NumPy array as well, which again is incredibly easy. In fact,


it's essentially the exact same piece of code. So just going to convert our initial list into an


array. And also labels as well.


See what we have here. Okay, so now we have an array rather than a list.


And all we do is the exact same thing again.


There we go, we have the exact same output again. So let's go back to our list.


There we go, we have the exact same output again. So lists and arrays, we can deal with them in the


exact same way. There's no need to do anything differently for them both.


Now in the case of us using a pandas dataframe, we may want to do something a little bit different.


So we have our inputs.


And our labels.


Okay, so now we have a pandas dataframe rather than an array or a list. The best way to do that


is to use a Pandas dataframe. So we're going to use Pandas dataframe.


Rather than an array or a list. The best way to deal with this and the way that we will be dealing


with this throughout this course is to simply convert these into a NumPy array. Which we can do


really easily like this. And we just add values onto the end. That creates our array.


So if we wanted to again create a data set from that, you just take this.


Split each into a column. Take the values.


And then we get our data set.


Incredibly easy. So the only difference here is that we are creating a 64-bit integer rather than


a 32-bit integer. And this is because this is the preferred data type for pandas. But for us, I mean


that doesn't really make much difference. Now we can also read data from file. And to do that we


again use TensorFlow data. But we use something slightly different here. So this is a newer


feature. So it's not within the data set attribute. It's within experimental.


We make a CSV data set.


And all we would do here is pass a CSV data set into here. So if we quickly get some.


Okay, so we've got this train.csv here. And this is just an extract from the Kaggle Titanic data set.


So you can find that by searching Titanic Kaggle and it will come up. And all we have are all of


these rows which each lists a passenger on the Titanic. So all we do is train.csv. And then here


we can actually specify our batch sizes, what columns we want, the field delimiter, and all


these different things, which is incredibly useful. So first we'll do a batch size. And


let's go with a batch size of 16. So this simply means that we will read our data into our model


16 rows at a time. Our field delimiter, because we're using a comma separated file, we don't


really need to specify this. But it's there just in case you are using something else like a pipe


delimited file or a tab delimited file. And then we can also specify specific columns as well.


So let's select a few. So let's go with passenger ID, survived, and P class.


And then we also add in our label name, which is the target or output column.


Now one thing to note here is that our label name must also be within our selected column list.


And there we go. So now we can view what we have.


So we can use this dot take method, just take a certain number of columns or batches, sorry.


Let's print item. Okay, so here we can see we have passenger ID and we have a few values there.


P class, a few values there. And then at the end we have our labels, which are not assigned a name,


but they are here. Now, if we take a look at the passenger ID values here, we have 489826 and 273.


If we look at the top here, these are obviously not the same. So the reason that is, is because


when we use this experimental makes CSV data set method, we actually automatically shuffled data,


which is a good thing. Realistically, we always want to try and shuffle data before feeding it


into our model. We want every batch that we have to essentially be a representative sample of the


full data set. However, we're not always necessarily going to be reading from file.


So if we're not doing that, we do something else. So let's read in our train.csv.


We're going to read it in with pandas.


And then we want to create our data set using the original method that we learned,


which is from tensor sizes.


So this time, let's take a few different inputs and create a array here.


So we might want pclass, age, and parche. So this will create an array with three columns and the


number of rows equal to the number of samples that we have within our data set. And then for the


output label, which is always the second part of the tuple, we want to have survived.


So here we just need to change df to data


to match up to our data frame up here. And now we have our data set. So the only other thing


is that we want to have a batch of samples. So the only issue is now none of this is shuffled


and none of it is batched. So we do for item in data set, take one.


We see that we just have one sample. We want everything to be shuffled and batched.


All we need to do is write a single line of code with data set dot shuffle. And then here,


we just add a large number. The larger the data set and the less shuffled it is, the larger we need


to add in this number. So if we have a very unrepresentative sample or batch, we need to


increase this number. And then we sell batch. And let's go with 16 again. And there is also


another argument that we can add here. So obviously, 16 probably won't fit perfectly into


the number of samples that we have. So say if we had batches of two, and we had a total data set


size of nine, four of those twos would fit perfectly into eight, but then we'd have one


sample left over at the end. So what we can do if that is the case, is we can just drop the


remainder to avoid any shape problems coming up later on. So just drop remainder equals true.


And there we have it. We have shuffled and batched our data set. So that is everything on


the TensorFlow data sets object. This is incredibly useful. So definitely get familiar with it. And


we will also be using it a lot in this course as well. So I hope that has been useful. As always,


NER With Transformers and spaCy (Python)

Okay in this video we're going to have a look at using named entity recognition through the spaCy


library which I've already done a video on before but this time rather than using the


traditional spaCy models we're going to use a transformer model. Now the setup for this is


pretty similar to normal spaCy so the very first thing we want to do is obviously install spaCy


transformers. So to do that all we need to write is pip install spaCy and then in square brackets


transformers. Now alongside this we can also install what we need to run it with CUDA so if


you have CUDA you can check your version number like this. Okay and so for me I can see that I


have CUDA 11.1 and so what I would do rather than just writing pip install spaCy transformers


I would write this transformers and then after this I include my CUDA version so I write CUDA


and I'm on 11.1 so I put 111 and then I'd enter this and install spaCy but I already have it


installed so I'm not going to do that again. Now once we have installed spaCy transformers


we also need to download the transformer model from spaCy and to do that we would write this


so we write python and we specify the spaCy module and then we write download and the


transformer model is very similar to the other spaCy models if you've used them before. We have


to do this for every spaCy model not just the transformer model. So it's depending on which


model you want I'm going to be using the English one there are other models as well for other


languages. We're using the English core web and I think there's only a web transformer model but


I'm not 100% sure on that so don't take my word for that and then the transformer part of this


is trf at the end so usually the the models we typically use are sm for small and lg for large


this time we're using the transformer version of that model. Again I already have it installed so


I'm not going to run it again and now to start using this we do what we would normally do with


spaCy so all we actually need to do is import spaCy and I'm also going to import displayCy so


that I can visualize the name entity recognition so from spaCy import displayCy and then what we


do is this is almost exactly the same as what we always do with spaCy there's really no difference


here so we do nlp spaCy.load and then here we include our model name so usually what we do is


write encore web sm or lg this time we're just going to write trf for transformer. Run that and


once we've loaded that we can perform name entity recognition as we normally would so all we do is


doc nlp and then in here we pass our tps and then we want to visualize that and we can do that in


Jupyter just like this so render and then we pass in the document for the doc that we just created


and we want the style to be entities because we're doing n-e-r here. Okay so there we have our trf


model now I'm going to swap these around because I want to compare this to what we would normally


use which is the let's use the large model because it's probably the closest match to the transform


model and I'll just call this lg and modify this to be lg as well and what we'll do is just do the


same thing again here and here and rerun that and you can see that in this case there's actually no


difference now I think this is to be expected because the traditional spaCy model especially


the large model is still really good in terms of performance so in most cases or at least a fair


few cases we shouldn't really see that much difference because this is still a pretty solid


model and we can see this as well here so this is a longer text I got this from the investing


subreddit and let's do the same thing again so we're going to take the transformer model also


the traditional model we'll convert both of those to dots so just do it like this let's try and keep


it reasonably tidy okay and then we just want this and this again okay now again with this one


we don't see any difference because like I said this traditional spaCy model is still pretty good


but what we can do is start adding more complex or or at least longer pieces of text and then


let's compare how they perform because in this case the transformer model will outperform the


traditional model okay so this text is quite a bit longer and now let me let me copy these two


bring them down here and let's do that again I'm not sure why sorry I'm loading the reloading the


model that I didn't realize okay and here and here now let's compare these two so the transformer


model is correctly identifying fastly as an organization the large model is identified


fastly as a person so you can see a little bit of difference there then up here we have this quarter


121 identified as a date by a transformer model not at all with the large traditional model


and go a little bit further so this one I'm not really sure if this is a good thing or bad thing


so the transformer model is identifying a whopping 27% as the percent which I don't think that's a


I don't think that's really a good thing but it depends on what you're doing maybe maybe you kind


of want that exaggeration of the percentage in there I don't think so so I suppose that depends


but I would say probably transformer model is actually performing worse with that single that


single use that single case the rest of these the money percentages they're all matching between the


two models here the large model is pulling out this one cardinal I'm not sure we really want


that one in there to be honest and then if we continue the transform model is getting the q1


date and the facility organization down here usual model missing q1 we do have facet person


and then down here the large model is seeing CFO as an organization which is obviously not the case


it's actually a position obviously so yeah I mean they're not far from each other for sure but I


think in this case the transformer model is definitely outperforming at least by a little


bit and I think the more complex the language gets the better the transformer model can form


so I mean that's it for this video I just wanted to give you a quick demonstration of spacey and


transformers which I think is really cool that you can actually use both of those packages


How-to Use The Reddit API in Python

Hi and welcome to this video on how to use the Reddit API in Python.


How-to Use The Reddit API in Python

So I'm going to keep this really short and we'll just get straight into the code in just a moment.


How-to Use The Reddit API in Python

But I just want to describe what we're actually going to cover in this video.


How-to Use The Reddit API in Python

So the first thing we need to do is obviously get access to the API.


How-to Use The Reddit API in Python

So I'll just take you through how we can do that.


How-to Use The Reddit API in Python

And then I'll explain how we authenticate ourselves when accessing the API.


How-to Use The Reddit API in Python

After that, I'll take you through some of the most common uses of the API


How-to Use The Reddit API in Python

that I think most of you are probably going to be most interested in.


How-to Use The Reddit API in Python

So that's stuff like getting the most popular threads from a subreddit


How-to Use The Reddit API in Python

or just a steady stream of all the threads being posted onto a subreddit.


How-to Use The Reddit API in Python

So let's just get straight into it and we'll start putting together our API.


How-to Use The Reddit API in Python

OK, so the first thing we need to do is head over to this page here, which is reddit.com


How-to Use The Reddit API in Python

slash prefs slash apps.


How-to Use The Reddit API in Python

Now we just want to scroll down here and find this create another app or create an app button.


How-to Use The Reddit API in Python

And you click on that.


How-to Use The Reddit API in Python

Now you just give it a name.


How-to Use The Reddit API in Python

It doesn't really matter what you call it, just something that you recognize.


How-to Use The Reddit API in Python

We are using this as a script for personal use.


How-to Use The Reddit API in Python

Obviously, if you are using this API for something else,


How-to Use The Reddit API in Python

then tick one of the other options that is relevant.


How-to Use The Reddit API in Python

You can give it a quick description.


How-to Use The Reddit API in Python

And then here you need to give it a redirect URI.


How-to Use The Reddit API in Python

So for me, I'm just going to enter my Twitter address because basically you can put anything


How-to Use The Reddit API in Python

you want in here, but it's so that when people are wanting to find out something about your API,


How-to Use The Reddit API in Python

they will be directed to whatever you put in this box.


How-to Use The Reddit API in Python

So obviously, if someone's find out about my API,


How-to Use The Reddit API in Python

they'll come to here and they know that they can ask me about it.


How-to Use The Reddit API in Python

Okay, and then here, this is our secret key, which we are going to need later.


How-to Use The Reddit API in Python

So make sure you keep note of this and also this personal use script as well.


How-to Use The Reddit API in Python

So I'm just going to copy those across and put them into my JupyterLab here.


How-to Use The Reddit API in Python

And I'm just going to call it client ID.


How-to Use The Reddit API in Python

So I'll identify and this is the public key.


How-to Use The Reddit API in Python

And here we have our secret key as well.


How-to Use The Reddit API in Python

So this one you need to keep secret.


How-to Use The Reddit API in Python

Obviously, I'm showing you this, but this API won't exist by the time I upload the video.


How-to Use The Reddit API in Python

And we just enter those.


How-to Use The Reddit API in Python

So now we have those.


How-to Use The Reddit API in Python

The next step is to request a temporary auth token from Reddit.


How-to Use The Reddit API in Python

And the first thing we need to do is actually import the request library.


How-to Use The Reddit API in Python

Then we get our authorization like so.


How-to Use The Reddit API in Python

And here we enter our client ID and secret key.


How-to Use The Reddit API in Python

Now, once we've done that, we are going to need to actually log in.


How-to Use The Reddit API in Python

So to do that, we can first initialize a dictionary where we specify that we are going to be logging


How-to Use The Reddit API in Python

in with a password, which we do like this.


How-to Use The Reddit API in Python

And then we pass in our username and password as well.


How-to Use The Reddit API in Python

And for my password, I'm just going to read it in from this text file here.


How-to Use The Reddit API in Python

You can, if you want, and this is just a simple script, you can just enter your password here.


How-to Use The Reddit API in Python

It's not recommended.


How-to Use The Reddit API in Python

It's recommended that you read it from elsewhere, but it's completely up to you how you deal with this.


How-to Use The Reddit API in Python

But this is how you can read it in from a text file.


How-to Use The Reddit API in Python

Just make sure you put R there instead of W for read.


How-to Use The Reddit API in Python

Okay, so that is the dictionary that we will need to pass along to Reddit in just a moment.


How-to Use The Reddit API in Python

So we also need to essentially identify the version of our API.


How-to Use The Reddit API in Python

And for this, you can literally put anything you want, but we'll put something that is at least slightly descriptive.


How-to Use The Reddit API in Python

We'll just call it my API.


How-to Use The Reddit API in Python

I'll put this is the version number.


How-to Use The Reddit API in Python

Now, all we need to do is actually send a request for our OAuth token.


How-to Use The Reddit API in Python

And we send this request to this address.


How-to Use The Reddit API in Python

We are accessing the API version one and the access token endpoint.


How-to Use The Reddit API in Python

And in there, we also need to include our auth that we received earlier.


How-to Use The Reddit API in Python

We need to include our login data.


How-to Use The Reddit API in Python

And we also need to include the headers.


How-to Use The Reddit API in Python

And this will return us hopefully everything that we need.


How-to Use The Reddit API in Python

Okay, and then here we can see our access token.


How-to Use The Reddit API in Python

So we need to access that and we'll just store it in a variable here.


How-to Use The Reddit API in Python

So this token is something that we will need to add to our headers whenever we're using the API.


How-to Use The Reddit API in Python

So to do that, we just write this.


How-to Use The Reddit API in Python

And we need to add that within authorization.


How-to Use The Reddit API in Python

And the token itself needs to be formatted in a string that contains the word bearer space,


How-to Use The Reddit API in Python

and then the token itself.


How-to Use The Reddit API in Python

So then we just print out headers.


How-to Use The Reddit API in Python

This is what we get.


How-to Use The Reddit API in Python

So now we can access every endpoint within the Reddit API.


How-to Use The Reddit API in Python

So beforehand, if we had tried to access this endpoint,


How-to Use The Reddit API in Python

the auth reddit.com, then API v1 me, if we'd have tried to access this, we would have not been allowed.


How-to Use The Reddit API in Python

So let's say we just put the headers and we will just put this user agent API that we had before.


How-to Use The Reddit API in Python

Okay, and we get a 401 response.


How-to Use The Reddit API in Python

So let's copy this and try again.


How-to Use The Reddit API in Python

But this time use headers, which includes our authorization.


How-to Use The Reddit API in Python

And so you get a 200, which means everything is okay.


How-to Use The Reddit API in Python

And then we can add Jason onto the end here.


How-to Use The Reddit API in Python

And we get all of this information.


How-to Use The Reddit API in Python

So that's great.


How-to Use The Reddit API in Python

We now have access to anything and we can start accessing what I think is probably the more


How-to Use The Reddit API in Python

relevant, important information.


How-to Use The Reddit API in Python

So the first one that I want to focus on is retrieving the most popular posts on a subreddit.


How-to Use The Reddit API in Python

So if I head over to the Reddit API documentation over here.


How-to Use The Reddit API in Python

Okay, so we can see here we have this get subreddit hot and this returns the most popular posts.


How-to Use The Reddit API in Python

We have this get subreddit hot and this returns all of the hot posts on that subreddit.


How-to Use The Reddit API in Python

So in our case, let's go with the hot threads in the Python subreddit.


How-to Use The Reddit API in Python

So to do that, we send the get request.


How-to Use The Reddit API in Python

And like you can see here, it's this R subreddit hot.


How-to Use The Reddit API in Python

So we can copy that across and we start the request with the oauth reddit.com.


How-to Use The Reddit API in Python

And then we have our R subreddit get rid of this end bracket.


How-to Use The Reddit API in Python

Hot.


How-to Use The Reddit API in Python

And of course, the subreddit that we want to look at is Python.


How-to Use The Reddit API in Python

And then we can just add our headers in here.


How-to Use The Reddit API in Python

So this is request not reddit.


How-to Use The Reddit API in Python

And then we can see what is in there using this JSON method.


How-to Use The Reddit API in Python

And then here we get all this data.


How-to Use The Reddit API in Python

So this is obviously not very clean at the moment.


How-to Use The Reddit API in Python

So let's clean this up and we can put it into a Pandas data frame.


How-to Use The Reddit API in Python

So it's a bit more readable.


How-to Use The Reddit API in Python

So first, let's figure out how to access each post within the response.


How-to Use The Reddit API in Python

So let's open this again.


How-to Use The Reddit API in Python

Now, within this JSON, all of our posts are contained within this data key here.


How-to Use The Reddit API in Python

So let's add data.


How-to Use The Reddit API in Python

And then once we get into data, we have a few different options.


How-to Use The Reddit API in Python

So we have this mod hash, which is nothing we need to care about.


How-to Use The Reddit API in Python

We have this, which is 27.


How-to Use The Reddit API in Python

That's not the post that we want.


How-to Use The Reddit API in Python

And then we have this one here, which is children.


How-to Use The Reddit API in Python

And then you'll see that this is a list.


How-to Use The Reddit API in Python

And within this list, we have all the information about all of the hot posts within the Python subreddit.


How-to Use The Reddit API in Python

So that is where we want to extract data from.


How-to Use The Reddit API in Python

So let's do that.


How-to Use The Reddit API in Python

Let's print that post.


How-to Use The Reddit API in Python

Okay, and now we are getting somewhere.


How-to Use The Reddit API in Python

And you can see there's quite a lot of data in each one of these.


How-to Use The Reddit API in Python

So it's probably worth us cleaning this up a little bit more.


How-to Use The Reddit API in Python

So you can see here, this is our other, the next entry in this list.


How-to Use The Reddit API in Python

So what we probably want to do here is extract the data within the post.


How-to Use The Reddit API in Python

So this is giving us this other dictionary, which contains all the data.


How-to Use The Reddit API in Python

This is our other dictionary, which contains all the relevant information we want.


How-to Use The Reddit API in Python

And then it is within here that we are going to want to extract the different parts of information into our data frame.


How-to Use The Reddit API in Python

So just as an example, we have the title.


How-to Use The Reddit API in Python

Okay, and then here we can see all of these titles of the most popular threads in the subreddit.


How-to Use The Reddit API in Python

So this is essentially the syntax that we're going to use to populate our data frame.


How-to Use The Reddit API in Python

First, let's just import pandas.


How-to Use The Reddit API in Python

Maybe install it.


How-to Use The Reddit API in Python

Okay.


How-to Use The Reddit API in Python

And then we need to initialize our pandas data frame.


How-to Use The Reddit API in Python

So we do it like so.


How-to Use The Reddit API in Python

Okay, and that just gives us an empty data frame.


How-to Use The Reddit API in Python

And then we're going to use the for loop like we did before to loop through each one of the posts


How-to Use The Reddit API in Python

and just extract them as a row into this data frame.


How-to Use The Reddit API in Python

So we'll do df equals append.


How-to Use The Reddit API in Python

And then within this, we create a dictionary, which is going to contain everything that we would like to include.


How-to Use The Reddit API in Python

And at the end of that as well, we also need to remember to ignore index,


How-to Use The Reddit API in Python

otherwise we'll end up with a load of errors and we want to avoid doing that.


How-to Use The Reddit API in Python

So first, let's include the subreddit.


How-to Use The Reddit API in Python

Just so we know where this data is actually coming from.


How-to Use The Reddit API in Python

So just like before, we want to do the post data.


How-to Use The Reddit API in Python

And then we just access the subreddit.


How-to Use The Reddit API in Python

Okay, and let's just have a look at what we have there.


How-to Use The Reddit API in Python

So, okay, perfect.


How-to Use The Reddit API in Python

As expected, we're getting all of these entries through.


How-to Use The Reddit API in Python

So that's great.


How-to Use The Reddit API in Python

But obviously, we're probably going to want a little bit more than just the subreddit.


How-to Use The Reddit API in Python

So let's just add a few more items as well.


How-to Use The Reddit API in Python

So we have the title, like we did before.


How-to Use The Reddit API in Python

And another pretty important one, in my opinion.


How-to Use The Reddit API in Python

So let's just go this.


How-to Use The Reddit API in Python

Another important one is the self text, which contains the actual content of the thread,


How-to Use The Reddit API in Python

or the text content of that thread.


How-to Use The Reddit API in Python

So let's just add a couple of more items.


How-to Use The Reddit API in Python

So that one is pretty important if you're wanting to extract any information about,


How-to Use The Reddit API in Python

well, anything from Reddit.


How-to Use The Reddit API in Python

Okay, so this is starting to look a little bit better.


How-to Use The Reddit API in Python

Let's see what we have.


How-to Use The Reddit API in Python

Okay, it looks good.


How-to Use The Reddit API in Python

And maybe we want to also include a few other items, maybe the number of upvotes, the downvotes,


How-to Use The Reddit API in Python

and the score of the posts.


How-to Use The Reddit API in Python

So we can do a few different things here.


How-to Use The Reddit API in Python

We have the upvote ratio, which is of course the number of upvotes it is getting in comparison


How-to Use The Reddit API in Python

to downvotes.


How-to Use The Reddit API in Python

And maybe we'd also just like to include the actual number of upvotes and downvotes as well.


How-to Use The Reddit API in Python

And again, it's pretty straightforward.


How-to Use The Reddit API in Python

We just include these, and we can include downs like so.


How-to Use The Reddit API in Python

And finally, we can also include the score of the post.


How-to Use The Reddit API in Python

Okay, so that gives us quite a lot of information that we can go ahead with this.


How-to Use The Reddit API in Python

Now, if there are other things that you're interested in adding in here,


How-to Use The Reddit API in Python

you can just do this to actually see what keys you can include.


How-to Use The Reddit API in Python

So it's access to data and then keys, and this will just return lists of everything in there.


How-to Use The Reddit API in Python

Now, this is pretty useful for actually finding the most relevant or the most popular posts,


How-to Use The Reddit API in Python

but a lot of the time what you might want to do is actually stream the newest post.


How-to Use The Reddit API in Python

So you essentially get a real time update of what is actually going on.


How-to Use The Reddit API in Python

And I would say this is probably what most people are going to want to use the API for.


How-to Use The Reddit API in Python

So we can take a quick look at that as well.


How-to Use The Reddit API in Python

And we can find it just over here.


How-to Use The Reddit API in Python

We have this R subreddit new.


How-to Use The Reddit API in Python

Okay, so essentially all we actually need to do here is adjust our old call to,


How-to Use The Reddit API in Python

instead of reaching out to the hot endpoint, we reach out to the new endpoint.


How-to Use The Reddit API in Python

So let's just modify our code to do that.


How-to Use The Reddit API in Python

Okay, so up here where we have hot, we just change that to new.


How-to Use The Reddit API in Python

Okay, seems to have worked.


How-to Use The Reddit API in Python

And then we just do the same thing again.


How-to Use The Reddit API in Python

So we just rerun this code.


How-to Use The Reddit API in Python

Okay, great.


How-to Use The Reddit API in Python

And then we do this and we get all of the latest posts on our subreddit,


How-to Use The Reddit API in Python

which of course is pretty useful.


How-to Use The Reddit API in Python

Now, this is returning around 27 to this one is 25 posts at once.


How-to Use The Reddit API in Python

And of course, you're probably going to want maybe a few more than that.


How-to Use The Reddit API in Python

So what we can do is actually add a limit parameter.


How-to Use The Reddit API in Python

And this limit parameter, we just add like so, add params.


How-to Use The Reddit API in Python

And then in here, we add limit and we can go up to 100 items.


How-to Use The Reddit API in Python

So if we run that and let's take a look at what we had before.


How-to Use The Reddit API in Python

We had this JSON and we had this dis equals 25, which is a little bit


How-to Use The Reddit API in Python

dis equals 25, which means that we return 25 items before.


How-to Use The Reddit API in Python

Now, if we run that, we will see 100.


How-to Use The Reddit API in Python

So now we're returning 100 items.


How-to Use The Reddit API in Python

And of course, that's pretty useful.


How-to Use The Reddit API in Python

So now we're getting more data back.


How-to Use The Reddit API in Python

And we can essentially just keep running this again and again,


How-to Use The Reddit API in Python

and extracting as much data as we would like.


How-to Use The Reddit API in Python

So if we just rerun this, so you can see we go up to 24 here.


How-to Use The Reddit API in Python

Rerun that and we will go over to 99.


How-to Use The Reddit API in Python

Okay, so that again is pretty useful.


How-to Use The Reddit API in Python

Now, there's also one more thing that is pretty important to understand with this.


How-to Use The Reddit API in Python

And that is how we can extract the IDs of a post from the ready API.


How-to Use The Reddit API in Python

So if we go into post here,


How-to Use The Reddit API in Python

we have these two different items. We have kind, which is actually, I think, here.


How-to Use The Reddit API in Python

So we have this T3.


How-to Use The Reddit API in Python

So Reddit posts just have these different types or kinds.


How-to Use The Reddit API in Python

And it's essentially a code that says whether it's a thread or some other type of post,


How-to Use The Reddit API in Python

which I think is something like ads or videos or something along those lines.


How-to Use The Reddit API in Python

But generally, we're always going to be working with T3, which are threads.


How-to Use The Reddit API in Python

But if you are working with something else, of course, that may change.


How-to Use The Reddit API in Python

And then as well as that, we also have the ID, which is here.


How-to Use The Reddit API in Python

And we can put both of these together in order to create the Reddit post ID.


How-to Use The Reddit API in Python

So we add this with a underscore in the middle.


How-to Use The Reddit API in Python

And this, that is the unique ID and that is unique for every post on the subreddit.


How-to Use The Reddit API in Python

And in the API documentation, you will see this referred to as the full name.


How-to Use The Reddit API in Python

So what we can do with this is actually,


How-to Use The Reddit API in Python

essentially loop back in time with the API. So one of the things we can do is only request


How-to Use The Reddit API in Python

threads that are further back in time than a post given a specific full name,


How-to Use The Reddit API in Python

which it would be this T3 mix of letters.


How-to Use The Reddit API in Python

So if we would like to do that, so let's take this final one we have here.


How-to Use The Reddit API in Python

And all we do is add that into another variable after like this.


How-to Use The Reddit API in Python

And this will only take 100 new threads that have appeared after this post.


How-to Use The Reddit API in Python

So we can do that.


How-to Use The Reddit API in Python

And then what we can do rather than actually initializing our post ID,


How-to Use The Reddit API in Python

we can avoid doing that and we can actually loop through and add all of these new posts


How-to Use The Reddit API in Python

to our data frame. And then we end up with even more data.


How-to Use The Reddit API in Python

And here we go.


How-to Use The Reddit API in Python

Okay, so that's how we can walk through and keep extracting more and more data


How-to Use The Reddit API in Python

from the subreddit.


How-to Use The Reddit API in Python

Now, at some point, it will stop allowing you to do this. You can only go so far back in time,


How-to Use The Reddit API in Python

which depends on the volume of requests that you're making, the volume of threads on a specific


How-to Use The Reddit API in Python

subreddit. But that is essentially all you need to actually do that.


How-to Use The Reddit API in Python

So like I said, it's the red API,


How-to Use The Reddit API in Python

is incredibly powerful and unlike most other APIs on social networks, it's free to use.


How-to Use The Reddit API in Python

So definitely something to take advantage of and see how you can implement it in your own projects.


How-to Use The Reddit API in Python

So I hope you've enjoyed the video and thank you for watching.


How-to Use The Reddit API in Python


How LSH Random Projection works in search (+Python)

Okay, so we're on to LSH or locality sensitive hashing with random projection.


In the previous video in the series, if you're following it, we covered LSH,


but we covered the more traditional version of it with shingling, min hashing, and LSH.


What we're covering here is, I suppose, more of like a modern implementation of it.


And this is what you'll see in libraries like FICE, which we'll work through later on how to


actually implement this in FICE. So what we're going to cover in this video is specifically


LSH random projection. We're going to work through a few visualizations, try and get a grasp of


what it is actually doing. And whilst we're doing that, we'll also work through how we implement


that in very simple Python code. For the FICE implementation of this, which is obviously much


more efficient than what we'll be covering in this video, we will cover that in another separate


video to this, because otherwise it's just huge, a very long video. So if you would like to see that,


I'll make sure there's a link to that in the description. So let's just recap very quickly on


what is LSH. So at the top here, we have this first kind of row, if you like. Here we're


minimizing collisions. So this is a hash function, this blue bit in the middle, and these are vectors


on the left. And we're passing through a hash function and it's making sure that we put both


of those into separate buckets. That's minimizing collisions. LSH tries to do the opposite and what


does is it maximizes collisions, but it only tries to maximize collisions for similar vectors, not


just everything. So LSH in short is a hashing function that tries to bucket similar vectors


together. Now when we're performing search with LSH, we have all of our, I would say,


database vectors indexed already. So they've all been booketed. Imagine they've all been booketed.


And we introduce a new vector, which would be our query vector. And we process that through the same


booketing mechanism and we see where it lands. And then essentially what we can say is, okay, this has


landed in this bucket and that means that the most similar other vectors to it are the ones that are


either in the same bucket or in the neighboring buckets. So what it does is allows us to compress


our vectors into low resolution vectors, which makes our search a lot faster. And that is what


you can see here. So we have our dense vectors at the top. Typically we're using dense vectors for


this and they can contain hundreds or thousands of values. And they're all typically floating point


numbers. So memory wise it's pretty heavy. And what we do is convert them into a very small


binary vector, which is what you can see at the bottom. So it's a lot more memory efficient and


usually it should be faster to search. Although sometimes it can actually go the other way and


it can become slower to search than just comparing, than just using a flat index.


Now at the same time, because we are approximating these vectors, the search accuracy is obviously


going to decrease. But what we want to do really is to maintain decent accuracy whilst speeding up


the search. And our aim is using XQ, which is our query vector, is to return the k nearest neighbors


as accurately as possible. Obviously we're approximating so it won't be perfect, but that's


fine as long as you get a decent speed up. Now LSH with random projection works by splitting


our vector space, which is obviously a highly dimensional vector space, using hyperplanes.


So I mean it's what you can see right here. And the way that it works is that given a single


hyperplane on the positive side of the hyperplane, if your vector appeared on that side,


it would be assigned a positive dot product value. And then we would process that in our


binary vector, this would be assigned a one. On the other side of that you have the negative side of


the hyperplane. And if your vector is on that side of it, it will be assigned a negative value


with the dot product. And with that in our binary vector would be assigned a zero.


And the reason that that works is we're using the dot product value. So imagine this green line


down here. Imagine this is the hyperplane that I just showed you. We have that normal vector,


the n that comes out here. And using the dot product, if the dot product finds that both of


these are in the same direction, e.g. anything in this sort of angle,


then it will take that as a positive. If it's on the other side, so anywhere here,


it would take that as a negative dot product, which is what you can see over here.


Now a single binary value isn't going to tell us much about the direction of our vector or the


position of our vector. So what we do is just add more hyperplanes in there. So we add more


hyperplanes and that gives us more binary values within our vector. So what you can see here is we


have those, we have two hyperplanes now. The magenta obviously correlates to the


zero index in these vectors. And then the teal hyperplane correlates to these ones,


the number one indexes. So what we would do is essentially just use loads of hyperplanes,


which is kind of what you see here. So these are the big arrows and the points that you see there,


the normal vectors. So where we're actually calculating dot product,


and then these are our hyperplanes. Okay, so here for blue, we would get a value of zero.


And then here, for example, for blue, we would get a value of one. Now let's start building this out


in code so we can actually see how this works. So what I'm going to do is set the number of


hyperplanes that we would like. And we set that using a parameter here called n bits.


I'm going to say we have four hyperplanes. Just for this example in reality, we'd use more,


but for now we're going to go with four. So we have four binary values here. And all I'm going to


do is create our vector dimensionality as well. So we're just going to use 2D vectors to make it


easier and so we can sort of visualize stuff as well. So all we need to do is we're going to


create the plane norms, and they are going to be numpy random, rand, and the dimensionality there


will be n bits and d. So the number of hyperplanes that we want, so n bits, and the dimensionality


that we're actually using. And we're just going to minus 0.5, and then we're going to


minus 0.5 because we want them to center around the origin, the zero axis. You don't need to do


this by the way, it's just so we can kind of see the effects of it a little bit better.


Okay, and we get these values. Now those vectors don't align to this visualization,


but it's essentially the same thing. What we have done is we've created four 2D hyperplanes,


or we've actually created four 2D normal vectors that we're going to use to build our binary


vectors. So we're going to create these three vectors, a, b, and c, and what we're going to do


is calculate the dot products for each one of those. So then we know whether they are positive


or negative behind each of our plane norms. So to do that we're just going to do np. and then we


just add our vector and we add plane norms and we just transpose those. Okay, so let's see what we


get there. Okay, so we see that we get negative, negative, positive, positive. Okay, so when we


convert this into our binary vector that will be 0 0 1 1. Now we want to do this not just for a,


but also for b and c. So okay, now what we want to do is say okay if it's negative it's 0,


if it's positive it's a 1. So to do that all we want to do is write a dot and we say well greater


than 0 it's a 1, so it's positive. And we do that again for each one of our vectors. Let's see what


we get. Okay, so now we get false, false, true, true. So the final thing to do there, although I


don't necessarily need to, is just convert them. In fact we do need to purely to create our


the binary vector string. So we'll see why in a moment. It's fine.


As type int. So it's essentially easier for us to visualize. That's it.


Okay, and we should get something that more looks like this here. So you see here obviously the


values, the positions are slightly different. But we have a is on the positive side of the


teal hyperplane, so 1 is of course 1. And then on c and b are both on the other side, so they are


of course negative. Now if we consider a, b and c, b and c are kind of in the more in the similar


position. So we would hope that they kind of align in there in the values that they have a bit better


than a. But let's have a look, see what we get. Okay, so they're the same. So that's good because


they are very much in a very similar place, which you can see from here. So a, b and c in this case


match up to what we're writing in the code. It's just the hyperplanes that are different. Now it's


these binary vectors that we use to create our LSH buckets. So what we're going to do is actually


implement that just using a Python dictionary to make it easy. So what we'll do is I'm going to put


each of our vectors a dot and b dot and c dot into this vectors list. It's just so we can iterate


through them a little bit easier. I'm going to initialize our buckets which is going to be like


I said a Python dictionary. And here I'm just going to set i equal to zero. So this is just so


we can loop through each of those vectors. So we're going to do for i in range length of vectors. So


yeah we don't need either. So the first thing we want to do is create a hash string using the


vectors that we have up here. So I'm going to do hash string equals I'm going to do join like that.


And what we want to do is join all of those numbers together but we need to to join them as strings.


So when vectors i as type string. Okay so if I make sense let me show you what that does hash string.


Okay so we just get something like that. Now then what we want to do is say if the hash string is


not already within our buckets is not in buckets.keys we want to initialize a new list. So


buckets hash string equals a new list. So we're initializing a list essentially initializing a


bucket to put our vectors in. So initialize it and then after that we just add we append it


to that bucket. So this is essentially what we need. Let me okay yeah should should be fine.


So let's print the buckets and see what we get. If hash string is what did I write there?


Not in there we go. So now we see we have these two hash buckets and one and two have both been


been put into the same one. All right and that's essentially how LSH works but just on a


much bigger scale. So now let's say you know we have our buckets and what we want to do is


is given a new vector we want to search using that I want to hash it and search. This is our


query vector xq. So what we see here so we've got like two examples one on the left is an example.


So this is we're comparing our query vector against two samples in our LSH buckets.


So xq is this zero it's been hashed 0 1 1 1 and what we do is first we compare it to


this vector and we see and we're using Hamming distance here which is essentially you know


do these two equivalent values match or not. If they do it's zero there's no distance between them


if they do not match then the distance is one and then we add up all of the distance values


at the end there. So with this one none of them match so we get like one plus one plus one plus


one which is obviously four. So the Hamming distance between those two is four which is


the biggest you're going to get with this dimensionality of binary vectors.


And then we have the second one and these ones match a bit better so zero is equal to zero one to one


one to one so all those zeros and then so like zero zero zero and then this final one is one


so then that equals one. Okay so that's Hamming distance and when we consider that with our code


over here we also have to consider that there's a degree of information being lost because


this is how we're storing the vectors we don't store the original vectors anymore this is you


know that essentially like final form in the LSH index. So say we have our query vector and it


comes through a zero zero one zero like great we get a perfect match but in reality is it


closer to one or is it closer to two we don't actually know so we have to be careful when


we're building our buckets to make sure that there's not too many items within each bucket


we need them to be reasonably spread out but not spread too thin because if we spread them


too thin across too many buckets the the index becomes absolutely huge so it's is definitely like


a balancing act between having enough buckets enough granularity in there to differentiate


between a reasonable number of vectors but not too granular that we just make the index bigger


because then it's slower than just doing a flat search. Now what you what we see here is


what we just discussed right so given these two vectors a and b they're reasonably far from each


other if we use a value of n bits value of two our vectors are not big enough to get bugged into


the same place we can't differentiate them but what we can do obviously increase the number of


hyperplanes or increase n bits and then we can differentiate them so so for you know these two


here exactly the same buckets these are not we have differences here here and here so we can


differentiate between them which obviously is what we need but at the same time we are increasing


the size of our index so it means we are becoming more accurate but we're also getting slower so


it's yeah finding the middle ground between them both. Now that's it for the implementation


details behind FICE what we'll do is we're going to leave this video here and we'll cover the FICE


implementation in the next video which I will leave a link to in the description so you can


BERTopic Explained

Today, it's estimated that up to 90% of the world's data is unstructured,


BERTopic Explained

meaning that it's built specifically for human consumption rather than for machines.


BERTopic Explained

And that's great for us, but it's also kind of difficult because when you're trying to organize all of that data,


BERTopic Explained

a lot of data, it is quite simply impossible to get people to do that.


BERTopic Explained

It's too slow and it's too expensive.


BERTopic Explained

Fortunately, there are more and more techniques that allow us to actually understand text or unstructured text.


BERTopic Explained

We're now able to search based on the meaning of text, identify the sentiment of text,


BERTopic Explained

extract human entities and a lot more.


BERTopic Explained

Transformer models are behind a lot of this.


BERTopic Explained

Now, these transform models are the sort of thing that make sci-fi from a few years ago look almost obsolete


BERTopic Explained

because being able to communicate in a human-like way with a computer was regulated just in sci-fi.


BERTopic Explained

But now these models can actually do a lot of the stuff that was deemed impossible not that long ago.


BERTopic Explained

In machine learning, this act of organizing or clustering data is generally referred to as topic modeling.


BERTopic Explained

It is the automatic clustering of data into particular topics.


BERTopic Explained

What we're going to do in this video is take high level view of that topic.


BERTopic Explained

The library, how we would use Bert Topic's default parameters to perform topic modeling on some data.


BERTopic Explained

Then we'll go into the details.


BERTopic Explained

So what is Bert Topic actually doing?


BERTopic Explained

So we're going to have a look at the Transformer Embedding Model, a UMAP for damage and reduction.


BERTopic Explained

We're going to take a look at HDBSCAN for clustering.


BERTopic Explained

And now we're going to look at CTF-IDF for basically extracting topics from your clusters.


BERTopic Explained

Each of those are really interesting.


BERTopic Explained

And once you understand how they work, you can use that understanding to improve the performance of your Bert Topic Topic Modeling.


BERTopic Explained

Once you've been through all that, we're going to then go back to Bert Topic and see how our understanding of these different components


BERTopic Explained

allows us to improve our topic modeling with Bert Topic.


BERTopic Explained

So we're going to cover a lot, but let's start with that high level overview of the Bert Topic library.


BERTopic Explained

So we're going to start with this reddit topics data set.


BERTopic Explained

And in here we have the subreddit that the data has been pulled from.


BERTopic Explained

We have the titled thread and we have the actual text within this self text feature.


BERTopic Explained

We have a few different subreddits in here.


BERTopic Explained

We have Investing, Python, Language Technology and PyTorch.


BERTopic Explained

So we have four subreddits.


BERTopic Explained

And what I want to do is use just the text and automatically cluster each of these threads into a cluster that represents that particular subreddit.


BERTopic Explained

So Investing, Python, PyTorch and Language Technology.


BERTopic Explained

So in here we have very similar data sets, actually not the same.


BERTopic Explained

We're going to use that one later.


BERTopic Explained

But this one's a little bit shorter and it just includes data from the Python subreddit.


BERTopic Explained

We have 933 rows in there.


BERTopic Explained

And I'm just going through this one as a quick example of how we might use the Bert Topic library using the basic parameters.


BERTopic Explained

So we can remove rows where self text is NAN or very short.


BERTopic Explained

And this is if you're using a data frame, otherwise we would use this.


BERTopic Explained

OK. So you can come up here and you just pip install the data sets library for that.


BERTopic Explained

And then we come down here and we get to Bert Topic.


BERTopic Explained

Now Bert Topic, you would pip install Bert Topic at the top here.


BERTopic Explained

And all we're doing here, so I've added this count vectorizer, which is essentially just going to remove soft words in one of the later steps of Bert Topic.


BERTopic Explained

And all you do is first just convert your data to a list if you need to.


BERTopic Explained

Actually, this is for a Pandas data frame.


BERTopic Explained

So this is what we'd be running with the Hug and Face data sets object.


BERTopic Explained

And all we do is add these arguments.


BERTopic Explained

So we make sure we have language set to English.


BERTopic Explained

So using an English language model for embedding the text and then we just fit transform.


BERTopic Explained

OK, so that's like fit and predict for this text here.


BERTopic Explained

OK, and we can see here we've just, all we've done here is embedded the text to embeddings.


BERTopic Explained

And we, so those embeddings are quite a very high dimensional.


BERTopic Explained

And it will depend on the model that we're using, but you're thinking like hundreds of dimensions.


BERTopic Explained

And then we reduce the dimensionality of those embeddings and cluster those.


BERTopic Explained

And then there is the finding of topics based on the text within each one of those clusters.


BERTopic Explained

So from that we get those two lists, topics and props.


BERTopic Explained

And if we take a look, so we have, this is just printing, it's going to print out five of them.


BERTopic Explained

But the size is too big, so it's fine.


BERTopic Explained

We're not going to look through all of them.


BERTopic Explained

So we have this predicted topic, two.


BERTopic Explained

So this is just a cluster.


BERTopic Explained

We don't know what the actual cluster is.


BERTopic Explained

And we come down and it's, what is it?


BERTopic Explained

I think Python for sure.


BERTopic Explained

Yeah, Python.


BERTopic Explained

Now we have another one, a negative one in BERT topic is used where that data doesn't seem to fit into any of the clusters that BERT topic has identified.


BERTopic Explained

So it's an outlier.


BERTopic Explained

Now, in our case, we can say, okay, this is probably Python, but BERT topic with the default parameters hasn't identified that.


BERTopic Explained

So, but we're going to learn how to improve those default parameters.


BERTopic Explained

So we'll figure that out.


BERTopic Explained

And then what we can do is see the top words found within each topic.


BERTopic Explained

So basically the topic using this get topic info.


BERTopic Explained

And you see that we do actually have a lot of topics here.


BERTopic Explained

Now we're trying to break the Python subreddit down into multiple topics here.


BERTopic Explained

So I'm not really sure how would be best to do that.


BERTopic Explained

We can definitely see some of these make sense.


BERTopic Explained

So here you have the based on, I assume, image processing with Python, learning Python, web development with Python and a few other things as well.


BERTopic Explained

I'm curious about this one.


BERTopic Explained

It was like some sort of game. I don't know.


BERTopic Explained

But anyway, so we go down, we have 16 of these topics.


BERTopic Explained

And then we can also go ahead and visualize those.


BERTopic Explained

So this is quite useful when trying to see how each topic might relate to another or how similar they are.


BERTopic Explained

And then if we come here, we can see how each of those clusters actually kind of grouped together in this hierarchy.


BERTopic Explained

And we can also visualize the words within each one of those topics as well.


BERTopic Explained

But, you know, this is just really simple BERT topic at the moment.


BERTopic Explained

We've just used default parameters.


BERTopic Explained

We haven't really done anything.


BERTopic Explained

What I want to do is actually go into the details of BERT topic.


BERTopic Explained

So let's start having a look at each of the components that I mentioned before within the BERT topic pipeline.


BERTopic Explained

So as I said, BERT topic consists of a few different components.


BERTopic Explained

So we start off with our text.


BERTopic Explained

Now, our text can come in a lot of different formats.


BERTopic Explained

But typically we're going to go with something like paragraph or sentence size chunks of text.


BERTopic Explained

And we typically call those paragraph or sentence size chunks of text documents.


BERTopic Explained

So our documents, they come down here and pass into a NLP, typically a sentence transformer model.


BERTopic Explained

So we have, we'll just draw that as like a box here.


BERTopic Explained

And that sentence transformer is going to output for each document a single vector.


BERTopic Explained

Now this vector or this embedding is very high dimensional.


BERTopic Explained

So it is, I'm going to write it as a list.


BERTopic Explained

It's just going to be really, really big.


BERTopic Explained

And we're going to have loads of them as we have loads of documents.


BERTopic Explained

But within that vector, we essentially have a numerical representation of the meaning behind its respective document.


BERTopic Explained

So it's like we're translating human meaning into machine meaning for each of these documents.


BERTopic Explained

That's pretty much how we can see it, which is already, I think, really cool.


BERTopic Explained

So once we have those vectors, the problem with these is that they're very high dimensional.


BERTopic Explained

So sentence transformers, typical dimension size is 768 dimensions.


BERTopic Explained

You have some sort of like half or sort of mini models, which will output something like 384 dimensions.


BERTopic Explained

And then you have some models that output many more like OpenAI's.


BERTopic Explained

Embedding models, I think, can go up to 200 or...


BERTopic Explained

No, no, they can go way higher. They can go to, I think, it's just over 10K.


BERTopic Explained

So it's pretty huge.


BERTopic Explained

Now, what we need to do is compress the information within those vectors into a smaller space.


BERTopic Explained

So for that, we use something called UMAP.


BERTopic Explained

Now, UMAP is...


BERTopic Explained

Well, there's a lot to talk about when it comes to UMAP.


BERTopic Explained

And we're only going to cover a very high level in this video.


BERTopic Explained

But even that, we're going to leave it until later to really sort of dig into it to an extent.


BERTopic Explained

But what that will do is allow us to compress those very high dimensional vectors into a smaller vector space.


BERTopic Explained

So what we would typically do is go from 768 or any of those to something like 3D or 2D.


BERTopic Explained

OK, so we can have something like 2D or 3D vectors now.


BERTopic Explained

And what's really useful is that we can actually visualize that.


BERTopic Explained

So we can have a look at our output from UMAP and try and figure out, OK,


BERTopic Explained

is this producing data that we can cluster or not?


BERTopic Explained

Because it's very... not always very easy, but it's easier to see or easier to assess


BERTopic Explained

whether there are clusters in the data or some sort of structure in our data when we can actually see it.


BERTopic Explained

So that's really useful.


BERTopic Explained

And then after that, we're going to pass those into another step.


BERTopic Explained

So HDB scan.


BERTopic Explained

Now, HDB scan, again, like UMAP, there's a lot to talk about.


BERTopic Explained

We're going to cover it at a high level.


BERTopic Explained

But what that will do is actually identify clusters in our data.


BERTopic Explained

And it's, from what I have seen so far, it's very good at it.


BERTopic Explained

And again, it's super interesting.


BERTopic Explained

Both UMAP and HDB scan, in my opinion, are very interesting techniques.


BERTopic Explained

And then after that, move on to another technique or component, which is called CTF IDF.


BERTopic Explained

Now, a few, or many of you, may recognize the TF IDF part.


BERTopic Explained

The C part is a modification of TF IDF.


BERTopic Explained

TF IDF is almost like a way of identifying relevant documents.


BERTopic Explained

Remember, documents are those chunks of text based on a particular query.


BERTopic Explained

What we do here is identify particular words that are relevant to a particular set of documents,


BERTopic Explained

e.g. our clusters or our topics.


BERTopic Explained

And then it outputs those.


BERTopic Explained

So what we do is we'd end up with like four clusters, for example,


BERTopic Explained

which is what we're going to be using.


BERTopic Explained

So we'd end up with these four clusters and we'd be able to see, okay,


BERTopic Explained

this one over here is talking about money, stocks, and so on.


BERTopic Explained

So that would be its topic.


BERTopic Explained

And that would hopefully be the investing subreddit.


BERTopic Explained

And then over here, we would have, for example, Python and we would see things like, well,


BERTopic Explained

Python, programming, code, and so on as those terms that match up to that topic the best.


BERTopic Explained

Now, at a very high level, that is BERT topic.


BERTopic Explained

But of course, there's not much we can do in terms of improving our code


BERTopic Explained

if this is all we know about BERT topic.


BERTopic Explained

We really want to see it in practice.


BERTopic Explained

So what we're going to do is actually go through each of these steps one by one


BERTopic Explained

and see what we can do to improve that component for our particular data set.


BERTopic Explained

And at least have a few ideas on how we might go about tuning each of these components


BERTopic Explained

for other data sets as well.


BERTopic Explained

Okay, so we'll start with the transformer embedding step.


BERTopic Explained

So what we're going to do first is actually get our data set.


BERTopic Explained

So this time we're using the full data set, the reddit topics data set.


BERTopic Explained

I'm specifying this revision because in the future, I'll probably add more data to this data set.


BERTopic Explained

Adding this in will make sure that you are using the current version,


BERTopic Explained

e.g. the version I use of that data set rather than any new versions.


BERTopic Explained

So now we have just in the 4000 rows in there.


BERTopic Explained

Again, I'm just going to remove any short items in there and that leaves with just over 3000.


BERTopic Explained

You can shuffle data, although you don't really need to.


BERTopic Explained

It's good practice anyway.


BERTopic Explained

And then here, so if you find this is taking a long time to run on your computer,


BERTopic Explained

just reduce this number.


BERTopic Explained

Okay, and if you do use this, then definitely shuffle beforehand,


BERTopic Explained

but otherwise you probably don't need to.


BERTopic Explained

So if it's taking a long time to run, just reduce this to like 1000.


BERTopic Explained

Just so you're working through this with less data.


BERTopic Explained

Again, also you can find this notebook and all the other notebooks


BERTopic Explained

I'm going to work through in the description below.


BERTopic Explained

So the first step, as I said, is going to be that embedding step.


BERTopic Explained

So what we're doing here is actually pulling in one of those mini sentence transform models I mentioned.


BERTopic Explained

We can see here that it creates a word embeddings or sentence embeddings of 384 tokens,


BERTopic Explained

which is nice rather than using the larger models.


BERTopic Explained

It just means it's going to run quicker,


BERTopic Explained

particularly if we're just not using any special machines, which I'm not.


BERTopic Explained

I'm on my laptop right now.


BERTopic Explained

So we come down and we can create the embeddings.


BERTopic Explained

Now, as mentioned, we probably don't want to do all this at once


BERTopic Explained

unless you have some sort of supercomputer.


BERTopic Explained

So I'm running through those in batches of 16.


BERTopic Explained

So I'm taking 16 of those documents I mentioned,


BERTopic Explained

putting those into our sentence transform model


BERTopic Explained

and outputting 16 of these sentence embeddings


BERTopic Explained

and then adding all those to this embeds array here.


BERTopic Explained

You can see that.


BERTopic Explained

OK, so we encode the batch, which we pull from there, which is 16 at a time.


BERTopic Explained

We get our embeddings to that batch and then we just add them to the embeds array.


BERTopic Explained

And that's actually all there is to the sentence transformer or sentence embeddings bit.


BERTopic Explained

I'm not going to go into detail on that because there are many articles and videos on that


BERTopic Explained

that we have covered already.


BERTopic Explained

So we're going to keep that really light.


BERTopic Explained

And we move on to UMAP, which we haven't covered before.


BERTopic Explained

So we'll go into a little more detail on that.


BERTopic Explained

Now, the UMAP dimensionality reduction step is very important


BERTopic Explained

because at the moment we have these 384 dimensional vectors


BERTopic Explained

and that's a lot of dimensions.


BERTopic Explained

And we don't really need all of those dimensions


BERTopic Explained

to fit the information that we need for just clustering our data


BERTopic Explained

or clustering the meaning of those documents.


BERTopic Explained

Instead, we can actually try and compress as much meaning as possible


BERTopic Explained

into a smaller vector space.


BERTopic Explained

And with UMAP, we're going to try and do that to two or three dimensions.


BERTopic Explained

Now, another reason we use this is one, we can visualize our clusters,


BERTopic Explained

which is very helpful if we're trying to automatically cluster things.


BERTopic Explained

And two, it makes the next step of clustering much more efficient.


BERTopic Explained

So there are several reasons that we do this and that's a few of them.


BERTopic Explained

So to better understand what UMAP is actually doing


BERTopic Explained

and why it might be better than other methods, which we'll talk about,


BERTopic Explained

it's probably best that we start with some data that we can visualize.


BERTopic Explained

So the word embeddings or sentence embeddings


BERTopic Explained

that we just created, 384 dimensional,


BERTopic Explained

we can't visualize them at the moment.


BERTopic Explained

So what I'm going to do is actually switch across to another dataset


BERTopic Explained

and start with three dimensions.


BERTopic Explained

And what we'll do is reduce that to two dimensions


BERTopic Explained

and we'll see how that works


BERTopic Explained

or we'll see the behavior of those reduction techniques.


BERTopic Explained

Not just UMAP, but the other ones as well.


BERTopic Explained

So for that, I'm going to use this World Cities Geo dataset.


BERTopic Explained

In there, what we have is we have cities, their countries, regions, and continent.


BERTopic Explained

Now, what we want to focus on is a continent.


BERTopic Explained

Now, we also have these other features, so latitude and longitude.


BERTopic Explained

And using the latitude and longitude, what I've done is


BERTopic Explained

correctly use X, Y, and Z coordinates.


BERTopic Explained

So we now have a coordinate system


BERTopic Explained

in order to place each of these cities on a hypothetical globe.


BERTopic Explained

And we have just over 9,000 of those.


BERTopic Explained

So let's have a look at those.


BERTopic Explained

Okay, so I'm just plotting these with Plotly.


BERTopic Explained

You can see the code there. I'm not going to go through it.


BERTopic Explained

And what we can see is what looks pretty much


BERTopic Explained

like the outline of the world.


BERTopic Explained

Okay, so it's not perfect.


BERTopic Explained

Like for some reason, the United States of America


BERTopic Explained

just isn't in this dataset.


BERTopic Explained

So we have the North American continent


BERTopic Explained

and it only contains countries in Central America


BERTopic Explained

and then includes Canada right at the top there.


BERTopic Explained

So there's some missing chunks.


BERTopic Explained

And I believe Russia is also missing.


BERTopic Explained

Obviously, again, quite a big place.


BERTopic Explained

And I'm sure there are many other places missing as well.


BERTopic Explained

But all that to say, it's not a perfect dataset.


BERTopic Explained

But we have enough in there to kind of see what we can do


BERTopic Explained

with reducing this 3D space into 2D space.


BERTopic Explained

Now, essentially what we're doing there is


BERTopic Explained

we're trying to recreate a world map.


BERTopic Explained

Now, it's definitely not going to replicate a world map


BERTopic Explained

or not any that any sensible person would use.


BERTopic Explained

But we should be able to see some features


BERTopic Explained

of the different continents, the countries and so on.


BERTopic Explained

So what we're going to do with this world map


BERTopic Explained

is reduce it into 2D space using not just UMAP,


BERTopic Explained

but also two other very popular dimensionality reduction techniques,


BERTopic Explained

PCA or principal component analysis and TC.


BERTopic Explained

Now, we can see when we try and do this with PCA,


BERTopic Explained

we kind of like a circular globe-like thing


BERTopic Explained

and we get these clusters that kind of overlap, which is not ideal.


BERTopic Explained

I don't think a clustering algorithm is going to perform particularly well here.


BERTopic Explained

So, but what we do have is the distances between those clusters


BERTopic Explained

or those continents has been preserved relatively well,


BERTopic Explained

at least the ones that were further apart from each other.


BERTopic Explained

And that is because PCA is very good at preserving large distances.


BERTopic Explained

But it's not so good at maintaining the local structure of your data.


BERTopic Explained

And in our case, we kind of want to go and maintain to an extent


BERTopic Explained

both the local and the global structure of our data as much as possible.


BERTopic Explained

So what if we take a look at tcne?


BERTopic Explained

Now, tcne is almost the opposite.


BERTopic Explained

Rather than focusing on the global structure of data,


BERTopic Explained

it focuses on local structure because it essentially works


BERTopic Explained

by creating a graph where each sort of neighbor node is linked


BERTopic Explained

and it tries to maintain those local links between neighbors.


BERTopic Explained

And this is very good at representing the local structure of our data.


BERTopic Explained

And through the local structure, it can almost indirectly infer


BERTopic Explained

to an extent the global structure, but it's still not that great at it.


BERTopic Explained

And we get this sort of messy overlapping of some clusters.


BERTopic Explained

And generally, with our data set at least,


BERTopic Explained

the clusters are just not really tied together very well.


BERTopic Explained

Now, another issue with tcne is that it's very random.


BERTopic Explained

So every time you run it, you're going to get a very...


BERTopic Explained

you can get a very wildly different result, which is again not very ideal.


BERTopic Explained

So what we use instead is UMAP.


BERTopic Explained

Now, UMAP kind of gives us the best of both worlds.


BERTopic Explained

It performs well with both local and global structure.


BERTopic Explained

And we can apply it incredibly easy using the UMAP library,


BERTopic Explained

which we can just install.


BERTopic Explained

And then in a few lines of code, we can actually apply UMAP to our data.


BERTopic Explained

But there's much more to UMAP than just a few lines of code.


BERTopic Explained

There are many parameters that we can tune in order for UMAP


BERTopic Explained

to work with our data in a particular way


BERTopic Explained

or just fit to the topology of our particular data set.


BERTopic Explained

So there's a lot to cover,


BERTopic Explained

but I think by far the key parameter that can make the biggest impact


BERTopic Explained

is the nNeighbours parameter.


BERTopic Explained

Now, the UMAP algorithm, it does a lot of things,


BERTopic Explained

but one of the key things it does is it identifies the density