query positive passage p-plus and assumed negative passage p-minus and


Is GPL the Future of Sentence Transformers? | Generative Pseudo-Labeling Deep Dive

the result of this is that our model is going to have to learn very nuanced


differences between that negative or assume negative and that positive in


order to be able to separate them out and understand that okay the positive is


a pair for our query but this negative even though super similar is not the


pair for our query okay so just making it giving our model an extra task to


perform here now with all that in mind we also need to understand that we're


assuming that all the passages are returning we're returning on negatives


we don't actually know that they are okay and again this is something that


we're going to handle later on in the pseudo labeling step but for now we just


assume that they are in fact negative passages to our query so let's move on


to the actual implementation of negative mining so the first thing we need to


consider for this negative mining step is that there are two parts to it there


is an embedding or retrieval model again we're going to use something that has


been trained on data from pre-covid times and what I was going to do is


taking out our passages and our queries and translate them into into vectors okay


but we need somewhere to sort those vectors so we need a vector database to


do that so we're going to use a vector database to store all those vectors and


then we're going to perform a search through all of those with our queries in


order to identify the most similar passages and then we're going to say


okay are these is this a positive passage for this query if not then great


we're going to assume it's a negative passage for our query we're just going


to go through and do that so the first thing we're going to do is load our


model so we're using this MS Marquardt silver base TASB again that is it's a


pre pre-covid data that's been trained on and we'd like that into sentence


transformers and just setting the match sequence length to 256 here okay and


then we're going to initialize a pinecone index so this is going to be


our vector database we're going to store all of our vectors and so for that we do


need an API key it's all free you don't need to pay anything for this all the


infrastructure and stuff's handled for us which is great so we get a pinecone IO


make an account if you need to and then you'll get an API key now for me it's


not the best way to do it it's just easy I'm just storing my API key in a file


called secret in the same directory as my code I'm just reading that in here


okay so my API key oh as well another thing you need to do here is to install


that you need to pip install pinecone client okay not pinecone pinecone


client so we initialize pinecone we with our API key this is just a default


environment there are other environments available if you want to use them but I


don't think you really need to and then we create the index okay so we have a


negative mining index so here I'm just saying you can check your currently


running indexes with pinecone list indexes if negative mine is not in there


then I'm going to create negative mine index and there's a few things you need


to pass so dimension the dimension is what we have here so that the embedding


dimension that our model embedding model is outputting so we specify that 7 6 8


the metric this is important we need to be using dot product metric here for


GPL and also the number of pods so by default this is one what you can do is


increase this so I think I increase it to 70 which is probably like massively


overkill for this but that shortened so with one I had a runtime I think of one


hour 30 or one hour 40 with 70 which again it's probably overkill maybe 40


would do the same I don't know with that it was I think 40 minutes so a lot


faster but you do have to pay for anything that's more than one pod so I


know if you're in a rush then fair enough fine like go with that otherwise


you just sit with one pod it's free okay and then you connect so you this creates


your index and then you connect to your index so pinecone index negative mine


okay and then we want to go through we're going to use our model actually


this bit here we're creating our file reading generator so it's just going to


yield the query and passage pairs I include this in here because I got a


value error because there's some like we had like literally one very weird data


in there so I just added that in there to skip that one and yeah initializing


that generator and then here I'm going to go through and actually encode the


the passages so you see it will get recreating a passage batch here we're


doing in batches again make it faster so I'm adding my passages of the passage


batch also adding IDs to the ID batch and then once we reach the batch size


which is 64 in this example we encode our passage batch we also convert them


to a list because we need it to be a list when we're upsetting everything's


upset just means it means update or insert just like database lingo we need


it to be a list when we're pushing this because the pushing it through through


an API call here so JSON requests so we need to convert that to a list rather


than numpy array which is I think the default and then we just create a list


of ID and vectors and upload that to to pinecone our index and then we just


refresh those batches so we start again and then we do node 64 and do then do


the encode and upset again and then the end here I just wanted to check number of


vectors we have an index so we see the vector the the dimensionality of the


index saw as the index fullness this will tell you pretty much how quick it's


going to run at zero is perfect it means it's basically empty or pretty quick


and then you have the vector count here so remember we have 200,000 examples or


pairs but not all of them are being used so they are all being used but not all


of them are unique because for each passages that we have we have three


queries okay so three unique queries but three


duplicated passages so obviously you know out of those 200,000 passages that


we have we need to divide that by three to get the actual number of unique


passages that we that we have in there which is something like this 76840


so the database is now set up for us to begin there to mine step it's full of


all of our passages so what we're going to do is loop through each of the


queries in in pairs which we created here so in loop through all of those


here in batches of batch size again so 100 and we're going to initiate this


triplets list where we're going to store our query positive and negative at the


moment we just have query positive remember so we're gonna go through there


get our queries get our positives and we're gonna create the query embeddings


okay so we that's in batches again then we search for the top 10 most similar


matches to our query and then we loop through all of those okay so for query


positive passage and query response this query response will actually have the


10 possible or the 10 highest and narrative passages for that query inside


there so extract those remember 10 in there and I'm just gonna shuffle those


so I'm gonna shuffle them and then we're gonna loop through them now we do this


so that we're not just returning the most similar one all the time but we're


returning one of the most similar top ten instead and then we extract native


passage from that one record that hit and then one thing we rarely consider


here is okay if we've got all of our passages in there we're also going to


have the positive passage for our query as well in there so it's pretty likely


that we were going to return that at least for a few of our careers for sure


so we need to check that negative passage we're looking at does not match


to the positive passage for that query and then if not then that means it's a


it's a negative passage or assume negative passage and we can append it to


our triplets so if query tab positive tab negative and when we say then file


okay now one last thing before we before we move on we should delete our index so


if if you're under the free tier you're just using that one pod it's fine you're


not gonna pay anything anyway but I guess it's also good practice to remove


it after you're done with it but if you are paying of course you want to do this


so you're not spending any more money than you need to so that's a negative


mining step and now we can move on to what is the final data preparation step


which is a pseudo labeling now pseudo labeling is essentially where we use a


cross encoder model to kind of clean up the pre the data from the previous two


steps so what this cross encoder is going to do is generate similarity


scores for both the query positive and query negative pairs okay so we pass


both those into cross encoder model and it will it will output predicted


similarity scores for the for the pairs so first thing you need to do is


actually initialize a cross encoder model again this should be a pre it


should have been trained on pre-covid data and then we're going to generate a


we're going to use a generator function as we have been doing throughout this


entire thing just to read the data or the pairs triplets in this case from


file and then yield them and what we're gonna do is use that cross encoder using


this sort of function here to calculate both the similarity the positive and the


negative and then we subtract those to get a margin between them so like the


separation between them okay which is ideal for when we're performing margin


MSC loss which will be very soon for fine-tuning our model so we go through


we using our generator here align a time query positive negative we get a


positive score negative score and then we calculate the margin between them and


then we're going to append those to label lines and save it all to file so


it's pretty quick step actually for the for the pseudo labeling step but it's


this final step is also very important in ensuring that we have high quality


training data without it we would need to assume that all passages returning


negative mining step are actually negatives and they're irrelevant for our


for a particular query in reality this is obviously never the case because some


negative passages are obviously going to be more negative or less negative than


others okay and maybe some of them are not even negative in the first place so


there was this chart or example from the GPL paper that I really liked I'm just


adapted it for our particular use case so the query is what are the symptoms of


kovat 19 okay the positive said first row there is


kovat 19 symptoms include fever coughing loss of sense of smell or taste okay so


that positive passage is the passage that we started with we generated that


query in the generative or crew generation stage and then we retrieve


these negatives in the negative mining stage now just have a look at each one


of these so the first one fever coughing and a loss of sense of smell a common


kovat symptoms that is not actually a negative it's a positive but this is


very likely to happen in the negative mining stage because we're returning the


most similar or the passages and those other very similar passages it could


very easily be actual genuine positives that are just not marked as being the


pair for our query okay we don't actually know that the negatives so in


this case this is a false negative because it should be a positive now the


next one okay we have these these easy negatives the next one symptoms are


physical or mental features that indicate a condition of disease now


this is a pretty typical easy negative because it's it has some sort of


crossover so the the question is what are the symptoms of kovat 19 and this is


just defining what a symptom is but it's not it's not about kovat 19 or the


symptoms of kovat 19 this is an easy negative because it's it mentions like


one of those keywords but other nice it's not really similar also it'll be


quite easy for our model to separate that so it will be easy for it to look


at this easy negative and say okay it's not relevant for our query okay it's


very different to the positive that I have here okay that's why it's an easy


negative next one another easy negative is kovat 19 is believed to spread from


animals to humans in late 2019 again it's talking about kovat 19 but it's not


the symptoms of kovat 19 so maybe it's slightly harder for our model to


separate this one but it's still a super easy negative okay and then we have the


final one so this is a hard negative costs are a symptom of many illnesses


including flu kovat 19 and asthma in this case it's actually a partial answer


because it tells okay cough is one of the symptoms but it's not still not an


answer we're asking about the symptoms we want to know about multiple symptoms


this is we kind of partially answers but it doesn't really and what I really like


about this is we look at the scores on the right here we have the scores when


you're using something like pseudo labeling from GPL and scores when


you're not using pseudo labeling like with you using gen q now gen q even so


like for the for the folds negative gen q is seeing this as a full-on negative


so you're just gonna confuse your model if you're doing this if it has two


things that are talking about the same thing and then you're telling your model


actually one of them is not relevant your model was just like you're really


gonna damage the performance of your of your model from doing that and it's the


same with the other ones as well okay there's there's almost like a sliding


scale of relevance here it's not just black or white there's there's all these


shades of gray in the middle okay and when using GPL and pseudo labeling we


can we can fill in those shades of gray and we can see that there are there is


a sliding scale of relevance without pseudo labeling we really missed that so


it's a very very simple step but really important for the performance of our


model okay so we now have the fully prepared data and we can actually move


on to actually fine-tuning our sentence transformer using margin MSC loss now


this fine-tuning portion is not anything new or unique it's actually very common


margin MSC loss is used for a lot of sentence transformer models so looking


back at the generate data we have the format of Q, positive passage, negative


passage and margin let's have a look at how those fit into the margin MSC


function so we have the query positive negative we pass those through we get


the similarity here so this is our similarity but the query and the


positive okay and this is this by encoder here is the one we're training


by the way so this by encoder is the final model that we're fine-tuning that


we're trying to adapt to our new domain and then we also calculate using the


query and negative vectors here we calculate the similarity between those


to get the similarity of Q and negative we subtract those to get this here


which delta hat is our predicted margin with our predicted margin we can compare


that against our true margin or we assume it's true margin from our cross


encoder and we feed those into the margin MSC over here loss function so we


have the predicted minus a true for each each sample we square all that to get


the squared error and then we're going over for all of our samples taking the


average and that is margin MSC loss now what is really nice is that we can


actually use the default the default training methods in sentence transformers


library for this because margin MC loss is just a normal method there's nothing


nothing new here at all okay so from sentence transformers I'm


going to import the input example which is just a data format we always use and


we're training in here so we're opening up the triplets margin here I'm just


reading it all the lines here I'm not gonna use a generator this time I'm


going to just create our training data which is just a list of these input


examples we pass the query positive and negative as a text in here so as our


triplet then the label we take the float with margin because margin is a string


from our TSV and from this we get this 200,000 training examples and we can


load these pairs into a generator data loader here so we're just using normal


torch data loader nothing special here we use empty cache here just to clear


our GPU in case we have anything on there so if you keep running this


script again and again trying to like get it working or just modifying


different parameters to see how it goes you'll want to include this in here


batch size is important with margin MSC loss the larger you can get


the better I did see I think in the sentence transformers documentation I


used something like 64 and that's hard you're gonna need like a pretty good


GPU for that and to be fair even 32 you need a good GPU this is using Tesla


one Tesla V100 for this okay and it works that's fine so as high as you can


get 32 I think is good even better if you can get 64 and yeah just initialize


data loader we set the batch size and we shuffle it's true and next we so next


step we want to do here is initialize the buying the buying code or sentence


transformer model that we're going to be fine-tuning using domain adaption so


this is the same one we use earlier in the retrieval step okay we're actually


going to be taking that and fine-tuning it so we set the model match sequence


length here again as we did before and then we initialize the margin MSC loss


function as we would with normal sentence transformers and then we just


run like model fit we train for one epoch or actually it depends you can get


that performance if you train for more I'll show you and I just do 10% of the


warm steps here I usually do and I'm going to save it same model name but


I've added kovid on to the end there and this doesn't take long I think this was


maybe it was 40 minutes per epoch on on the Tesla V100 so it's not bad now in


terms of the the amount of training steps you want to actually put your


model through with GPL they did find so in the paper there's a really nice chart


that shows that after around 10,000 steps if they're particularly use case


it kind of leveled off the performance didn't really improve that much more


from there however I did find so I'm using 200,000 examples so one epoch for


me is already 200,000 training steps which is more and I found that actually


trained for 10 epochs I didn't really I didn't test in between I just went from


one and I also tested 10 found that 10 actually the performance seemed to be


better I didn't really quantitatively assess this I just looked at it


qualitatively and it was performing or is returning better results than what


the model that I've been trained for one epoch so for rather than 200,000 steps


they've been trained for 2 million steps okay which is a fair bit more than a


hundred thousand mentioned here but that's just what I found I don't know


why I suppose it will depend very much on the data set you're using in


particular but I would definitely recommend you just you try and see test


a few different models the training doesn't take too long for this step so


you have the luxury of actually being able to do that and testing these


different numbers of training steps so I mean once training is complete we we


can actually test that model as we usually would we we have the model as


well so I can show you so we have the model over on if we go to Hugging Face


CO models so if you search in here you go pinecone and one of these is


the model that has been trained on COVID so hit this one here okay so you can


actually copy this and you can just write I think it is it should maybe have


an example I say you write this so you from sentence transformers import


sentence transformer and you just do model equals sentence transformer and


your model name which in this case is what you can see up here pinecone MS


Marco and you can test the model that I've actually trained here again this


was trained on 10 epochs not one okay so in this evaluation here you can see


what I was looking at so load the old model it's just a MS Marco the silver


based test B and then here I've got the model trained for one epoch and here the


model trained for 10 epochs okay which is equivalent to the model I just showed


you on Hugging Face HUB so I've taken these tests so we have these queries and


three answers or three passages for each one of those now these are not even


really perfect matches to the data we were training on but they're kind of


similar in that they just include COVID-19 okay so it's not a perfect match


it's not even perfectly in their main but it's good enough I think but I think


that really shows how how useful this technique is because it does work even


with this kind of not perfect match between what I'm testing on and what I'm


actually training on so here I'm just going through those I'm getting the dot


product score between those and sorting based on that so that the highest rated


passage for each query is returned so for the old model how is COVID-19


transmitted we get Ebola HIV and then very the very bottom we have corona okay


again what is the latest name variant of corona and then it's these here again


like it's not a COVID-19 right at the bottom we have chronic lager right at


the top I don't know if the rest of the world calls it lager or or beer but I


put lager there anyway and then we say okay what symptoms COVID-19 then we get


flu corona come second here and then we have symptoms at definition again and


then how will most people identify that have contracted the coronavirus and then


after drinking too many bottles of Corona beer most people are hungover


right so obviously it's not working very well and then we do the GPL model so


this is the trainer one epoch we see slightly better results so Corona it's


certain number one here and then and then here is it's number two so it's not


not great but fine what symptoms of COVID-19 this one it


gets right so it's good and then here how most people identify that it


contracted coronavirus again it's doing the drinking too many bottles of Corona


again then you have COVID-19 second place then we have the the model it's


trained to ten epochs this one every time so number one number one and


number one okay so it's pretty cool to see that and yeah I think it's actually


like it's super interesting that this technique can actually do that using


nothing more than just unstructured text data so that's it for this video on


generative pseudo labeling or GPL now I think it's already very impressive what


you can do and more importantly I think or what is more interesting for me is


where this sort of technique will go in the future like is this going to become


like a new standard in in training the sentence transformers where you're


building this synthetic data I feel like there's there's really a lot that we


covered in this in this video and from the from the GPL paper to unpack and


think about like we have negative mining we have pseudo labeling of all these


different techniques that kind of go together and produce this really cool in


my opinion technique that we can use super powerful for me it's just like


super impressive that you can actually do this so I hope there is more in in


this sort of field in GPL or some sort of new form of GPL in the future I'm


very excited to see that but for now of course that's it I hope this video is


been useful I hope you're excited about this as I am so thank you very much for


Streamlit for ML #3 - Make Apps Fast with Caching

In this video, we are going to continue with the app that we've been building.


So far, we, as a quick summary, we have what you can see over here.


We have this AI Q&A, it's just the very basics of Streamlit.


And we have this search bar or text input that we've built with Streamlit over here.


And then we put together the backend of our app.


So that is initializing a connection to our vector database.


We also created all of our context vectors and put those in there.


And also initializing our retrieval model, which takes care of encoding the query here.


And then Pinecone over here takes care of finding the most relevant context vectors based on that query vector.


And then we had a look at how we can iterate through all the contexts that we returned from Pinecone and then display them.


Now at the moment, it's very ugly.


And at the same time, another really bad thing that we need to solve in this video is this takes forever to do anything.


Right. If I just maybe even if I just remove that and press enter, I'm not even searching for anything.


And this is going to take, I don't know, like a minute.


I'm going to cut forwards so you don't have to wait as long as I do.


OK, so it's just finished. That took way too long.


So what we want to do or the reason for that is mainly the retrieval model download over here.


So every time we rerun or change anything in our app, the way stream it works is it reexecutes everything in your script.


And that's really good because it makes developing an app super simple.


But when you have something like that, you're downloading an ML model.


You don't want to redo that every time a tiniest little thing changes in your app.


You only want to do it once, like when the user opens your app the first time, then you download it and then you don't download it again.


That's what you want to happen.


And we also want the same to happen with our Pinecone connection.


Here we just want to initialize that connection once and not every time something changes in the app.


So we're going to do that. We're going to figure that out.


So we can go over to the Streamlit docs and we can scroll up to the top or go to this menu and we go to advanced features.


Now, and it says advanced, but it's not hard to do this.


OK, so we can optimize performance with stcache.


OK, let's have a look at that.


So we can scroll down.


It's a caching mechanism that allows your app to save performance when loading data from the web, load datasets or performing expensive computations.


And that sounds pretty much like what we want.


OK, so let's go down and we see basic usage.


So we have this is a good example, right?


So we have this function here.


It takes a long time to run every time and therefore makes the app very slow.


Every time anything changes, the whole thing is reloaded.


So this expensive computation is rerun every time.


We don't want that to happen.


What we want to do is, OK, you can just add this.


And that means that the output from that expensive computation is just stored.


OK, it's not it's not reloaded every single time.


OK, so let's let's try.


Not saying it's going to work, but let's try and do that.


So we're going to put define init retriever.


OK, and that is just going to return the retriever model.


So.


Return that.


And we do the same for our pinecone stuff here as well.


So define init pinecone.


OK, and then obviously we need to actually call those.


So let's do that.


Let's do that here.


OK, so we're going to call those.


We want the model is equal to init retriever.


And we want the index.


Is equal to init pinecone.


OK, let's save that.


Let's have a look at our app.


OK, so it's running again.


Let's wait a moment.


Actually stop that because we.


Here we're returning nothing, so we do.


We actually want to return the index.


That would be that would be useful.


And now we do we have to press this.


Rerun.


OK, now the first time we do this, it's going to take a while.


And.


First, OK, we want to make sure this is actually.


Is it working like it was before?


Let's see.


We getting any errors?


OK, now also it always seems to be working fine.


OK, so let's do.


Let's add that stcache that we saw in the documentation.


Let's add that to both of these.


OK.


Save.


We get this nice little spinner running init retriever.


It's not very descriptive for our users, so later on we'll have a look at making that a little


more interesting or descriptive.


But for now we'll stick with that.


And this is quite useful because we can see OK, you know what are the slow parts of our model to load?


OK, so we get this error.


OK, why is that?


So when we are caching with.


Streamlit.


What it is doing is.


Well, it basically checks if whatever's been cached changes with every rerun.


So it's putting the function or putting some values into your function or rerunning it and


having a look at what the hash code is that comes out of it.


Now in.


In this case, we're calling an API.


We don't.


Or we cannot actually hash the.


The connection to our pinecone index.


OK, and we shouldn't really do that for our retriever model either.


So what we can do is something which is kind of new from Streamlit.


OK, so whereas stcache is always going to check the hash code to see if anything is changing,


there are these new experimental caches.


And one of those in particular is this.


We have experimental memo.


It's fine.


So we use that to store expensive computations.


That's fine.


No, no, no.


Try that with some things, but that's not what we want.


We want this experimental singleton.


So basically what that means, experimental singleton is whatever you're running should


just be run once and it should not change.


Right, we're assuming that this will not change.


Right, so it's not going to check if it's changed and therefore it is not going to create


that hash representation of whatever it is you're running.


So we can write st experimental singleton.


Put it here as well.


Oops.


Copy it.


Put it here.


And OK, we've just saved it.


Let's have a look, see what happens.


OK, again, it's going to take a little while to rerun everything.


Hopefully not too long.


OK, there we go.


So now we have our search.


Let's say who are the Normans?


OK, I'm not going to skip ahead.


Straight away.


OK, so there's no waiting anymore, which is really good because before it just took so


long.


So, yeah, that's that's how we improve the performance of our Streamlit app using caching


and these new experimental caching primitives that Streamlit have developed.


So that's incredibly useful.


And what I want to look at in the next video is OK, over into our app.


Yes, the performance is there now, but it doesn't look so good.


So maybe we can have a look at actually improving this look here.


And to do that, we're actually going to not use what we are going to use Streamlit, but


we're going to pull in what are called Bootstrap card components, which are another sort of


HTML or CSS library.


And using the style from them to display our information.


It will look a lot nicer than it does now.


So that's it for this video.


I hope it's been useful.


Thank you very much for watching and I'll see you in the next one.


Today Unsupervised Sentence Transformers, Tomorrow Skynet (how TSDAE works)

In this video we're going to have a look at how we can train sentence transformers


without needing any labeled data. So if you're new to sentence transformers,


sentence embeddings or vectors, a sentence vector as we'll call it is


simply a numerical representation of a sentence or paragraph. If you think about


language it's a very human centric concept. It's not built for computers so


computers really struggle to get the meaning or concepts that we as humans


find very easy to communicate using language. Now the modern-ish computers


appeared during and around World War II. The first application of NLP came soon


after in 1954 with the Georgetown machine translation experiment. In that


first decade of research those involved were pretty optimistic that they were


going to solve the problem of machine translation in just a few short years.


Obviously they were a little bit optimistic and that's still a problem


that's still not solved. We still haven't solved machine translation and the same


goes for anything in NLP but in the past decade especially there have been a lot


of breakthroughs. The field of NLP has progressed at an incredible rate in just


the past decade and we now have an incredible ecosystem of language models


and techniques that we can use for a lot of different use cases. Now a lot of this


recent success is in part thanks to the dense vector representations of language.


So those are vectors, so numerical vectors that a machine can understand


but are built in such a way that they actually provide a numerical


representation of the semantics or the meaning behind whatever is those vectors


represent whether that be tokens, words or sentences, paragraphs and so on.


With those dense vectors we now have a way for computers to comprehend and


understand to an extent the semantic meaning behind language. To build those


given a lot of data and a lot of compute we tend to use transform models.


In NLP transformers are the de facto standard and for building representations


of sentences or paragraphs there is a subcategory of transformers called


sentence transformers. Now the training process to build a transformer begins


with something called pre-training that produces a generic transform model and


then we fine-tune that so we train that further using special methods to build


sentence transformers that can produce these very information rich and accurate


sentence vectors. Now whereas pre-training tends to use unsupervised


training methods fine-tuning tends to be more along the lines of supervised


training and what that means is that we need a lot of labeled data and for some


domains and languages there simply is not enough labeled data out there to


actually build a sentence transformer for those specific domains or languages.


So that means that you can always spend a long time gathering data and labeling all


data to get tens of thousands of labeled samples or you can go ahead and try


fine-tuning model using unsupervised training. Now unsupervised training I


will tell you straight away is not going to get you the performance that you


would get from a supervised training approach. However if you do not have the


labeled data to train using a supervised approach, unsupervised training is your


best bet and it still works pretty well. So in this video that's what we're going


to cover we're going to cover how we can train a sentence transformer or fine-tune


sentence transformer using a unsupervised training method called


Transformer-based Sequential Denoising Autoencoder. So what we'll do is jump


straight into it and take a look at where we might want to use this training


approach and and how we can actually implement it. So the first question we


need to ask is do we really need to resort to unsupervised training? Now


what we're going to do here is just have a look at a few of the most popular


training approaches and what sort of data we need for that. So the first one


we're looking at here is Natural Language Inference or NLI and NLI


requires that we have pairs of sentences that are labeled as either contradictory,


neutral which means they're not necessarily related or as entailing or


as inferring each other. So you have pairs that entail each other so they


are both very similar pairs that are neutral and also pairs that are


contradictory. And this is the traditional NLI data. Now using another


version of fine-tuning with NLI called a multiple negatives ranking loss


you can get by with only entailment pairs so pairs that are related to each


other or positive pairs and it can also use contradictory pairs to improve the


performance of training as well but you don't need it. So if you have positive


pairs of related sentences you can go ahead and actually try training or


fine-tuning using NLI with multiple negative ranking loss. If you don't have


that fine. Another option is that you have a semantic textual similarity data


set or STS and what this is is you have so you have sentence A here, sentence B


here and then you have a score from from 0 to 1 that tells you the similarity


between those two scores and you would train this using something like cosine


similarity loss. Now if that's not an option and your focus or use case is on


building a sentence transformer for another language where there is no


current sentence transformer you can use multilingual parallel data. So what I


mean by that is so parallel data just means translation pairs so if you have


for example a English sentence and then you have another language here so it can


it can be anything I'm just going to put XX and that XX is your target language you can


fine-tune a model using something called multilingual knowledge distillation and


what that does is takes a monolingual model for example in English and using


those translation pairs it distills the knowledge the semantic similarity


knowledge from that monolingual English model into a multilingual model which


can handle both English and your target language. So they're three options


quite popular very common that you can go for and as a supervised methods the


chances are that probably going to outperform anything you do with


unsupervised training at least for now. So if none of those sound like something


you can do the data sets or if it sounds like you probably can't get data that


seems like that and it doesn't match your use case then we would have to move


on to unsupervised training. So like I've written here you want to go for


unsupervised if you have little to no data in your unique domain or your


low resource language and with low resource language you also have no


translation data so from a source language like English or you can use


other languages as well just as long as there's a monolingual model in that


source language to your target language. So if you if we can't do that we move


on to unsupervised learning and one of the the best approaches at the moment is


this transformer based and sequential denoising auto encoder. Now there are


other approaches as well but we're not going to cover those and I think for now


this is probably your best bet although there is other methods being


researched that do look quite promising as well. So the way that TSDE works is


if you have let's say a sentence here what you do is you take your


sentence and you corrupt that data so we have this and what we're going to do


is it's just kind of remove parts of that or modify it in a different way and


do some other things to it so it's slightly different but not too different


that it should not be similar and we take both of these so we take the


modified input and we feed it into our encoder model so our transformer. Our


transformer outputs a set of tokens and we use some sort of pooling method to


convert those into a single sentence vector. So with that sentence vector we


process that through another model here which is a decoder model. So that goes


into the decoder model and what that decoder model must do is actually


optimize to produce a sentence here to produce the same text. So it has to try


and predict this original sentence and these weights in the decoder and encoder


are optimized in order for the decoder to be able to actually do that and that


is TSDAE. It's not particularly complex when you sort of look at a very


high level and it's certainly a very intelligent way of building a sentence


transformer without any label data. Now let's have a little graphic here to


compare TSDAE to MLM. Now MLM is Mass Language Modeling and that is a


pre-training approach that a few of you will probably be familiar with and


that's why I wanted to include this comparison in here. So with MLM which


is the bottom down here we take some input text and we mask one of the tokens


in that text. We pass it through an encoder which outputs all these token


vectors and basically these token vectors for every word almost, every


word or sub word here will be represented by one of these token vectors


or token embeddings. All those are passed into the decoder and the decoder


attempts to predict which word or token is behind that masked token here. So it's


trying to optimize for that mask to become an elephant and that's how you


pre-train a transformer or one of the ways that you can pre-train a transformer.


TSDAE is different for quite a few reasons but I think the main reasons


you can think of is one we are not necessarily masking the input and it is


found that the best way or the best approach is to actually delete the token.


So you see here we should have that mask here but it's not there anymore we just


removed it. So one we delete rather than mask though in the TSDAE paper


they did test both. We have an encoder as before but that is followed by this


pooling step and this pooling step takes the token vectors that we see down here


and it converts them into a single sentence vector. So that sentence vector


is passed on to the decoder so if you compare both of these steps here these


two the decoder in muscle language modeling is getting a lot more


information it's getting token level information whereas the decoder in TSDAE


is dealing with a lot less data and it's dealing with sentence level


information rather than token level information and it is then optimizing


for the same thing to try and predict that we should have this text here with


elephant rather than the missing or corrupted text that was input into the


encoder initially. So that's the main difference between both of those. Now in


the TSDAE paper from Wang, Reimers and Gurevich they tested different


approaches to fine-tuning. So the first of those is the noise type. So when we


take that original text and we corrupt it what is the best approach in you know


how do we corrupt it and they found that deleting tokens so this box


here produces the best results. Other options you can swap tokens so swap one


word for another you can mask as we saw with muscle language modeling can place


those tokens you can add new tokens you do different things right but by far


the best here was to just delete the token. Now there's also you know how


many tokens do you delete so going through each token you assign a


probability of that token being deleted and that's what we see here with this


noise ratio. So again best best approach there is 0.6 okay so we're going through


each token and assigning probability of 60% that that token will be deleted. So


you are removing quite a lot of data and then we have the the pooling method so


the the little circle after the encoder and the best or the highest


performing approach here was was using mean pooling okay but to mean pool you


have another step of actually taking the average across all those word vectors or


token vectors whereas with CLS pooling you don't do that you just take the CLS


token which is a it's a classified token from from BERT so if you've seen it


before it looks like this okay and then you have your other tokens following it.


So that is the approach that they they stuck with they they went and used


deletion only in encrypting that data ratio of 0.6% with a probability of 60%


and at the end they use CLS pooling. You can use mean pooling as well or even


max it's up to you but we later on we're going to stick with CLS to follow


along with the with the paper. It's just a actually quick explanation of CLS and


mean pooling if it's new to you. So we have our encoder we output loads


of token vectors CLS pooling we just take that single vector and


that creates that is our sentence vector so we're not really doing anything there


we're just extracting that vector whereas with mean pooling we're taking an


average over all of the output token vectors to create our sentence vector. So


that's it for the sort of visual explanation so let's jump into how


we actually build or fine-tune a model using this approach. Okay so we have here


I'm just loading a data set so we're using Huggian-based datasets here.


If you haven't used it before you would want to pip install datasets and what


we're doing is getting the oscut dataset oscut corpus which is basically a


massive multilingual corpus it has a lot of different languages in there. I'm sure


not every single language but if you can think of a language it's probably in


there. And we're taking the English portion of that just so I can actually


read and understand things and we're taking the training data. Okay and this


is important so the oscut dataset or at least for English the size of that is


quite massive it's 1.8 terabytes of data. If you don't include the streaming


true all that's going to download to your computer and I assume you probably


a lot of us probably don't have that much memory available on machines anyway


so it won't work. So you need to add streaming equals true because what that


will do is as we request a sample from the dataset it will download it and pull it


through for us one at a time not the full thing. So it's obviously a lot more


efficient it's not going to break our computer or anything. Now because we're


streaming it we have to kind of iterate through it so if I want to show you part


of that so I'm going to go for row in oscar print that row and just break. Okay


so we're just going to print the first item there so we have these two


features id which is just an index id value and we also have text. Now in here


we you can see quite a lot so there's this text and it's it's pretty long so


there's multiple paragraphs in there multiple sentences and when we're


training with tstae we only want smallish sentences we just want one


sentence for each sample. So we need to split that we need to split that up into


just sentences. So to do that I'm going to import re because I want to split


for just periods here full stops periods and I also want to split on new line


characters and I want to remove spaces at the same time so I just want to remove


anything that indicates that this is a new sentence or paragraph and split


based on on that character. So I'm gonna create a regex re.compile and that


is going to be any full stop followed by a space that's an optional thing so it


doesn't have to be a space and also optionally followed by a new line


character. Okay so this is just going to match any full stop for sure and it will


also allow for that to be a space included in that and there was allow


that for that to include a new line character as well. So that's gonna capture


everything for us and let's see what that looks like so we write splitter.split


and we'll go re so the re that we just pulled through text and yeah okay so now


we see that we have all these nice sentences rather than just one massive


paragraph. You see some here are not very long and what we'll do is we'll remove


them later on because they're not really sentences. So let's do that. I'm gonna


create a number here which is going to count the number of sentences we manage


to capture. So in reality or at least in the TSDA paper they found that 10k is


pretty much all you need and you can go up to 100k as well if you want. So


we're going to go up to 100k we probably don't necessarily need to


probably stick 10k maybe even lower. English is probably a reasonably easy


one for this to figure out so you could possibly go even lower and still get


decent results. So that's one thing about TSDA is that you need very


little data which is pretty cool especially when it's not labeled. So we


have sentences I'm going to create a list and we're going to just iterate through


so for row in Oscar we need to create our new sentences so new sentences equals


splitter dot split and that will be row text like we did before and we also want


to say we want to remove a sentence so say line for line and sentence if sorry


in new sentences new sentences if that line or length of that line is less than


greater than 10. Okay so we're saying we only want to include strings that


have a character length of greater than 10 and we can maybe even increase up


because if we look at this this is definitely more than 10 so let's just go


with 20 for now and see how that goes. Now there are new sentences from a


single sample and we want to extend our sentences list with those new sentences


so we just write send new sentences. Okay and like I said Oscar is a massive


data set if we run through this for the whole data set we're going to end up


with a lot of sentences and we don't need that many we're only 10 to 100,000


sentences so what I'm gonna say is number of sentences is going to be equal


to the length of new sentences so the number of new sentences that we've just


added we're going to add that onto new num number of sentences so once that


exceeds 100k then we want to break so we want to stop it's num sentence. Okay and


with that we should be able to run that should be quite quick. Okay pretty


nice and easy there and the next step as we usually would with PyTorch is we


want to put this data into a data set object and then we want to load that


data set object into a data loader. Now because we're doing this thing where we


corrupt our data by adding noise to it we either need to do that manually when


we're building our data set object or what we can do is just use the sentence


transformers denoising auto encoder data set object so to use that we just write


from sentence transformers dot datasets import denoising auto encoded data set


and we're also going to create the data loader now as well so as we usually


would in PyTorch we'll just import that as well so it's from torch dot data or


utils dot data import data loader and we want to create a data set so data set is


equal to denoising auto encoded data set and we just pass our data into that so


sentences and that is all we need for our data set so from now we can create


our data loader so it's loader equals sorry data loader and we pass in our


data set and we also want to say okay what the batch what is the batch size


and do we want to shuffle the data so for the batch size we're going to put


eight we don't shuffle data shuffle is true and we also want to drop the last


batch because this will it will not be the same same batch size as the rest of


our batches or most likely it will not be so we just drop it it's easier so we


run that so we now have that's our data is prepared for fine-tuning and we now


need to move on to the final preparation before we actually fine-tune the model


which is setting up the loss function and the actual training function itself


so before we actually even do the loss function we need to define the model


that we're going to be training so we're using BERT based on case from the


Hug and Face Transformers library and to initialize that or we will initialize


that through sentence transformers so from sentence transformers import from


sentence transformer and we also also want models now the BERT model is going


to be models.transformer and then in here we just pass the name BERT based on


case so this will just download models directly from Hug and Face models


repository as we would with the Transformers library so we want to write


BERT based on case and then we also need a pooling layer so right pooling equals


models dot pooling and as we saw before when we were working through everything


we want to use pooling using the CLS or classified token so to do that we will


need to pass CLS in there but before we do that we also actually need to tell


the pooling layer what number what dimensionality to expect from that vector


and to get that I'll show you we can just write BERT equals get word


embedding dimension and we'll see 7 6 8 but we will once it's actually defined


so let me put it in there so I'll add that in there okay run this you're seeing


a sec okay that's just initializing everything and we have that 768 now what


we have here are two separate layers and we need to combine them both or merge


them both into a single sentence transformer model so to do that we want


to write model equals this is where sentence transformer part comes in we


have modules and that will be BERT followed by the pooling layer and then


we can also print out the sort of a description of that model as well so see


we have transformer using BERT model and we also see the pooling we have the


well-invented dimension and we also have the pooling mode CLS token is true


whereas the rest of them are false because we're not using those pooling


methods now with the model defined we can we can find our loss function so


right from sentence transformers dot losses and we're going to import the


denoising auto encode loss and we'll use that to define our loss function so we


just write denoising auto encode loss and we pass in our model so it knows


what to what to actually optimize and we also need to make sure that we tie the


encoder and decoder weight so we have true in that okay because we have that


encode decoder and we're tying those weights together because the performance


is better we can also set faults but the performance will not be as good so that


is everything we we can go ahead and actually move on to this training


function so model dot bit so what we're gonna be doing here is we write model


dot bit and I just had a few points here so we're gonna be using an atom optimizer


we are going to be using a learning rate so LR of 3e to minus 5 and that learning


rate is constant so we're going to be using a constant learning rate so if


you've watched through the last videos we tend we have tended to use a warm-up


before we actually move on to that learning rate so we warm up to that


learning rate with this we're just going to choose constant learning rate all the


way through and there's no also no weight cake in there so model up fit we


need to say what are what are our training objectives so we just write


train objectives and then in there we pass a list and in here we need pairs of


data loaders and the loss functions we're going to use to optimize with that


data in our case we just have one these so it's just load up the data and also


loss we're going to train for one epoch so you put sequence one we are going to


be using the atom optimizer the default optimizer here is a Adam W or Adam


weight decay optimizer from the transformers library so if we want to


use Adam we just set the weight decay to zero so we write weight and decay equals


zero we also need to send scheduler so this is why I mentioned before so


scheduler equals constant learning rate okay so we're not doing any any warm-up


we're just going with a constant learning rate all the way through and


then we want to pass our optimizer params so here we're just passing the


learning rate nothing more so that is three three e to the minus five and


while training probably gonna want to see the progress bar so I'm gonna set


that equals true as well now after that after we finish training you're probably


gonna want to save the model so you can you can save that as wherever you would


like so I'm gonna write TSDAE but I'm going to start running that and I'm


gonna stop it and I'm because it takes a it doesn't take long to be fair with a


hundred thousand samples this took 20 minutes on a reasonably good GPU so


that's a RTX 3090 so it's obviously a decent GPU but it's nothing like it's


not a test or anything like that so this is very quick I was pretty impressed


okay you can see that this is training so I'm gonna do is pause that or stop


that and I'm going to switch over to the other notebook where I have that


training and the evaluation I perform afterwards we'll just run through it


really quickly okay so we're here now yep you can see everything is the same


as before model save I'll say that and oh there was one thing I did want to


mention so if you do get an error with an LTK you just need to do this you just


need to have a pip install an LTK view you haven't already got installed and


then you just run this input in LTK and that's going to download this punt


tokenizer which is used in the in the denoising process so that we're having


noise and then and denoising and whatever else it uses that tokenizer so


that's why we why we need that in there so if you do get that error just run


that so after that we so we train the model and we want to evaluate the performance


of the model because we want to see that has actually worked so one benchmark


that you use probably seen you use a few times already is the semantic textual


similarity benchmark or STSB which again we use we can use hug and face data sets


to pull that so it's from the blue data set it's the STSB part of that and we're


gonna take the validation split so I'm not taking the training data though we


can if we want because we haven't trained on that just in case and that


contains one sentence two sentence and a label this label is a score so if you


remember earlier we were talking about STS data that we can train using cosine


similarity loss that this is the data that we would use so we have that label


score and in this data set it ranges from 0 to 5 we want to normalize that


from 0 to 1 so that's what I'm doing here I'm using the data sets map function


and then a lambda in here and we're just dividing all those by 5 mapping them all


from 0 to 1 then we are reformatting that that benchmark data STSB data using


the sentence transformers input example class so we need this because we're


using the evaluators from the sentence transformers library later so all we do


there is we loop through create a list and we append input example objects


that contain the two sentences and the label so the score and then we


initialize a similarity evaluator so we can see this is called embedding


similarity evaluator so that's for this type of data the STS data and we're just


passing all the samples we have there and you can write to CSV if you want but


I'm not doing that because I just want to see the the score in here I don't


really I don't want to see on the detail I just want to have a look at the


overall score and then we just evaluate the model so you just pass the model to


your evaluator and using the model that we train TSDAE we get 0.75


which is the Spearman's coefficient so basically saying how where our model is


scoring high does that correlate to where the true scores are high or the


true labels are high and 0.75 means yes there is correlation there's its reason


it's pretty strong right it's not the strongest as we'll see in a minute but


it's pretty strong so that's a good score and we can see that it is is


working now if we compare that to an untrained BERT model so what we had


before before we actually fine-tuned it with TSDAE so we scroll down I've just


been reinitialized something here the same model as before and evaluated it


scroll down a little bit to find a score and you see that it's like 0.32 which is


obviously way lower and yeah there's some correlation there but it's not not


not great so TSDAE is giving us pretty good results I think like it is really


not bad now something we mentioned earlier yes TSDAE works it's an


unsupervised method but the unsupervised method is or cannot really be compared


to supervised methods in terms of the performance that you'll get from your


sentient transformer and so I wanted to show you that here so first thing I've


done it is taking the original SBIRT so the first one and okay we get 0.8 or 0.81


if you want to round up so it's about 7% better than the other than the


unsupervised method which is a fair bit but it's not massive so at least our


unsupervised model is up there in like a good area of performance it's not the


best but it does work and as well in the paper they did do better than this I


think they got to I think 78 maybe so they they also did better than what I


got here anyway so you can probably do better and then I want to tell look at a


more advanced model one more recent model was at least like MP net so the


MP net model scored pretty much 89 or 0.89 so we get okay result I would I


think pretty good results here saying but obviously it can't compare it to


those unsupervised models now that's it for this video I think at least for me


this unsupervised approach to training is actually one of the coolest things


or at least one of the coolest approaches to training and building


models that I've seen in sentence transformers possibly only really


paralleled with multi-lingual knowledge of distillation for training


multilingual models both of these together for me I think are really


incredible I know this isn't like the best performance when you think about


all of the low resource languages out there that don't really have that much


data they just have one structure text states like this or the or the domains


very specific domains where they just have loads of text data but they don't


have labeled data and they can't afford to pay someone to go and create all that


data or don't have the time to even I think this is a really cool method to


actually be able to to use I mean you don't really need anything so for that


reason I think this is really cool and is definitely one of the most


interesting ways of training in my opinion just training something without


labeled data that actually works pretty well so yeah that's it for this video I


hope it's been useful and thank you very much for watching and I'll see you in


How I work from anywhere

Today I'm going to talk about the way that I work in terms of working from anywhere.


How I work from anywhere

So for the past two years, pretty much since we had the first COVID lockdowns in the UK,


How I work from anywhere

I've been working remotely.


How I work from anywhere

In those two years, I've learned a lot about working from anywhere and in particular the


How I work from anywhere

sort of things that you need to consider or the things that you need to take with you


How I work from anywhere

when you're working from anywhere.


How I work from anywhere

Because in reality, it's very different when you are working at home at your desk.


How I work from anywhere

It's you have built that workspace to suit you or even in an office that workspace, maybe


How I work from anywhere

it's not built to suit you, but it's built to suit working.


How I work from anywhere

When you start working remotely from different places, the chances are most places you go


How I work from anywhere

or a lot of places you go are not really going to have good workspaces.


How I work from anywhere

That's just how they are because most places are not really built for working.


How I work from anywhere

So you really have to get a little bit creative sometimes, but you can avoid that to an extent.


How I work from anywhere

If you actually take the right kit, if you take pretty much a remote desk setup.


How I work from anywhere

So I'm going to show you my remote desk setup and we'll just talk through it.


How I work from anywhere

Now obviously I know that not everyone is able to work remotely, but I think in the


How I work from anywhere

past two years it has become much more common and I'm sure at least a few of you are probably


How I work from anywhere

working remotely either full time or at least part time.


How I work from anywhere

So yeah, I hope this is useful.


How I work from anywhere

This sort of setup for me seems to be working really well.


How I work from anywhere

I can pretty much go anywhere now and make myself a little workspace, which is pretty


How I work from anywhere

cool.


How I work from anywhere

Okay, so what you can see here is pretty much my workspace and obviously as I do all these


How I work from anywhere

YouTube videos all the time, I also have this big microphone here, which this week I have


How I work from anywhere

replaced or not replaced, but I have now have an alternative microphone, which should make


How I work from anywhere

it a little bit easier for traveling because taking this massive like road stand that you


How I work from anywhere

see here around with me, it's not a lot of fun.


How I work from anywhere

So yeah, this shotgun mic should be a lot better for me, but of course most people are


How I work from anywhere

not really going to need a microphone, so you don't need to worry about that anyway.


How I work from anywhere

So instead what you actually need is what you can see here pretty much.


How I work from anywhere

So I have this little stand here and this is super important because you're going to


How I work from anywhere

be working in a lot of weird places and most of them are not going to have good workspace


How I work from anywhere

setups.


How I work from anywhere

So you need your laptop to ideally be at your sort of eye level.


How I work from anywhere

This is really important and another thing that's super, super crucial about this is


How I work from anywhere

how it packs down to pretty much nothing.


How I work from anywhere

You fit in this little bag here and it's incredibly lightweight.


How I work from anywhere

I don't remember how lightweight exactly, but it weighs nothing.


How I work from anywhere

I just put it in my bag and go.


How I work from anywhere

It's great.


How I work from anywhere

So another thing that's so important, at least for me, a second screen.


How I work from anywhere

So I have this second screen.


How I work from anywhere

Obviously your MacBook doesn't come with a second screen.


How I work from anywhere

I've seen some that do attach to it, so maybe that's worth looking into, but they're super


How I work from anywhere

expensive.


How I work from anywhere

So this screen here, honestly in terms of how much productivity you're going to get


How I work from anywhere

out of a second screen, in most cases, at least for me, I think it's incredibly important.


How I work from anywhere

So the one thing I will say this is if you have a first generation MacBook where they


How I work from anywhere

have literally like the two USB ports and nothing else, this specific screen has to


How I work from anywhere

be attached to your, to the power adapter.


How I work from anywhere

So you're probably going to get like an hour, maybe two hours use out of it before you need


How I work from anywhere

to un-attach it and plug in your actual power adapter to charge your MacBook again.


How I work from anywhere

But with the newer MacBooks, maybe the older ones as well, that's not really an issue anymore.


How I work from anywhere

So that's just to bear in mind.


How I work from anywhere

Okay.


How I work from anywhere

Another really important part of the setup here is the separate keyboard and trackpad


How I work from anywhere

or a mouse, right?


How I work from anywhere

This is obviously if you have your laptop stand, you need separate keyboard and a mouse


How I work from anywhere

or trackpad.


How I work from anywhere

But that's, I think this is a pretty obvious one anyway.


How I work from anywhere

So other essential things that may be a less obvious.


How I work from anywhere

So this power bank here, I think this is pretty important.


How I work from anywhere

I mean, it's a new thing for me and I've only used it a couple of times, but if you're traveling


How I work from anywhere

for long periods of time or you have unreliable electricity, this is really useful.


How I work from anywhere

So that's really good.


How I work from anywhere

I'd recommend you buy like a reliable one, like a good one, even if it's a bit more money


How I work from anywhere

because you're probably going to need it.


How I work from anywhere

All right.


How I work from anywhere

Another super important thing is cable management, but not in the same way that you would manage


How I work from anywhere

the cables on your desk.


How I work from anywhere

So when you're traveling, you probably might not even realize, but you have a lot of cables


How I work from anywhere

and so it's really important to have those organized in a way.


How I work from anywhere

Otherwise they're just going to end up in your suitcase or your rucksack all over the


How I work from anywhere

place and it becomes an absolute nightmare, I promise.


How I work from anywhere

So I have this Gibbet bag.


How I work from anywhere

I don't know what you call it.


How I work from anywhere

It's just a bag and you just put in all of your cables so you can see everything here.


How I work from anywhere

Right.


How I work from anywhere

So, yeah, this is useful.


How I work from anywhere

It's not full yet, but I have everything in here.


How I work from anywhere

So like I have all the normal cables.


How I work from anywhere

These are like what everyone's going to have, like your laptop charger, phone charger, a


How I work from anywhere

few other things.


How I work from anywhere

And then I like to keep other things in here.


How I work from anywhere

So these are the instructions for the new shotgun mic.


How I work from anywhere

It's better I just leave it there for when I do actually need to read them.


How I work from anywhere

And yeah, I have another lens for the camera here.


How I work from anywhere

So that's pretty cool.


How I work from anywhere

Just slides in there.


How I work from anywhere

The battery.


How I work from anywhere

So you can add a lot of things here, right?


How I work from anywhere

Doesn't have to just be charging cables.


How I work from anywhere

So I really mean it when I say this is super important.


How I work from anywhere

Of all the things here, this is one of the best.


How I work from anywhere

Okay.


How I work from anywhere

Another thing super important is this bag.


How I work from anywhere

So obviously you can use any bag you want, but this is a Peak Design bag.


How I work from anywhere

And you see loads of people, like camera people using them, which obviously helps me because


How I work from anywhere

I have a camera.


How I work from anywhere

But it's useful for like anything because it's really well built bag.


How I work from anywhere

When I bought this bag, like it's expensive, but I felt like I've never had a good bag


How I work from anywhere

before as soon as I got this.


How I work from anywhere

It's really useful.


How I work from anywhere

So you have all of these different compartments.


How I work from anywhere

So you see in here, you have these different compartments.


How I work from anywhere

You have the top bit here for, I just usually put like a water bottle and some random things


How I work from anywhere

in there.


How I work from anywhere

Here I usually put my camera.


How I work from anywhere

And then in the bottom here, you can fit notepads, a book, other things if you really want.


How I work from anywhere

And then on the sides, you have all these other things as well.


How I work from anywhere

Like the cables usually, or before they were for cables.


How I work from anywhere

Now I use this.


How I work from anywhere

So it's not really as needed.


How I work from anywhere

So for example, this, my sand will usually go in the side here.


How I work from anywhere

So I just kind of throw this in.


How I work from anywhere

Water bottle.


How I work from anywhere

It's either on the side or on the inside here.


How I work from anywhere

So either in here or I can also put it over here.


How I work from anywhere

So at the moment I also have this because we're doing like a road trip kind of around


How I work from anywhere

Europe at the moment.


How I work from anywhere

I'm not driving to Madrid by the way, that's on a flight just for the conference.


How I work from anywhere

But everywhere else we're driving.


How I work from anywhere

So this is just a light.


How I work from anywhere

I figured if the car ever broke down, this would be really useful.


How I work from anywhere

I haven't needed it so far.


How I work from anywhere

But it's just a headlight.


How I work from anywhere

Nothing special.


How I work from anywhere

But what it's used for is in here.


How I work from anywhere

Or is it the other side?


How I work from anywhere

Might be the other side.


How I work from anywhere

Ah, it is here.


How I work from anywhere

So they have this little weird thing here.


How I work from anywhere

I don't know how well you can see that.


How I work from anywhere

This is a little cable.


How I work from anywhere

Like elastic cable thing.


How I work from anywhere

But if you don't want anyone to see your fancy water bottle, this is an expensive water bottle.


How I work from anywhere

Which is kind of stupid but it's fine.


How I work from anywhere

You use this so you just throw that over.


How I work from anywhere

Mainly it's to make sure nothing falls out if you're walking around with your bag upside


How I work from anywhere

down or something.


How I work from anywhere

I don't know.


How I work from anywhere

And then second screen.


How I work from anywhere

I don't always put this in my bag.


How I work from anywhere

It depends.


How I work from anywhere

Sometimes I do, sometimes I don't.


How I work from anywhere

I probably, I don't even know if I'm going to take it to, I probably won't even take


How I work from anywhere

this to Madrid.


How I work from anywhere

But if I were taking it, it goes in the back here because we have all these multiple components,


How I work from anywhere

compartments.


How I work from anywhere

So you see in here, it's nothing special to be honest.


How I work from anywhere

But it does fit a lot of stuff in which is pretty cool.


How I work from anywhere

So this actually, probably not going to leave it in there.


How I work from anywhere

But these, I do.


How I work from anywhere

So again being light, being thin, these are really good to just put in here.


How I work from anywhere

You can put the cable bag in here.


How I work from anywhere

I'm going to this time.


How I work from anywhere

I just have all my cables with me but you really don't need to.


How I work from anywhere

Usually I would just keep like essential cables.


How I work from anywhere

Like if I feel, if we're going for a long trip, like if you ever fly to Asia, usually


How I work from anywhere

a couple of flights at least during the past two years, it's been a couple of flights to


How I work from anywhere

get there.


How I work from anywhere

And it means that you're going to be traveling for ages, like more than a day in some cases.


How I work from anywhere

So taking a laptop charger in this bag separately to the Gibbet bag has what I've been doing


How I work from anywhere

and then just putting the Gibbet bag in like a suitcase.


How I work from anywhere

So it goes straight in here.


How I work from anywhere

And usually, despite us traveling for the past, most of the past two years, Lara still


How I work from anywhere

doesn't have like a decent laptop bag because she just likes to have like this tiny backpack


How I work from anywhere

which is, I'm not joking, about this size.


How I work from anywhere

So you can't fit anything inside it.


How I work from anywhere

Or she has a handbag.


How I work from anywhere

And neither of those you can put your laptop in.


How I work from anywhere

So she usually, I usually am carrying that in this bag.


How I work from anywhere

So two MacBooks, the smaller dimension ones, it's like 13 inch maybe.


How I work from anywhere

They both go in here and it's fine.


How I work from anywhere

Okay.


How I work from anywhere

And yeah, that's how I pack everything.


How I work from anywhere

So this works pretty much anywhere, at least for me, it works.


How I work from anywhere

So that's really good.


How I work from anywhere

It's all super lightweight, you know, as you can see, there's hardly anything there.


How I work from anywhere

But with that, you are pretty much going to be able to work anywhere, which today's world


How I work from anywhere

is pretty useful for a lot of people, I think.


How I work from anywhere

So yeah, that's it for this video.


How I work from anywhere

I hope it's been useful and I will see you again in the next one.


How I work from anywhere


Bag of *Visual* Words for Image Classification and Retrieval

Today we are going to talk about something called bag of visual words.


Now bag of visual words, we can use it for essentially converting images into


vector embeddings or image embeddings and using those image embeddings in image classification


and image retrieval. Now this video is going to be split into two parts. First part we are going to


look at the theory of bag of visual words and how it works and then in the second half of the video


we'll actually look at an implementation of it and see how we can implement something like bag


of visual words in python. So let's get started with trying to understand what exactly is bag of


visual words. Now for those of you that are coming from NLP you may have heard of bag of words before


it's a very common technique used in using NLP and information retrieval but it focuses on


sentences it focuses on language and text. So we can we can pretty much visualize it as this so we


take a sentence and we pull out all of the individual words from that sentence and then we


put them into a bag. This bag is unordered so imagine it's just like an unordered list and


that that's what bag of words is. Bag of visual words is pretty much the same thing but applied


to images which is quite interesting. So how does that possibly make sense? How do we extract


words or visual words from an image? Well at a very high level it's pretty much what you can see


here. So we take a load of images and we extract almost like patches from those images and this is


not exactly what happens we don't just put those patches into a bag we actually convert these


patches into what are called the visual words so these patches here we would call visual features


and then we do some pre-processing in the middle here you can't see it right now but


we'll fill all that in throughout the video and what we get out of that processing is a load of


what we call visual words. Now as with language there are a limited number of visual words.


There are so many words in the English language and there are also so many visual words in our


image data sets. Now a visual feature let's say this over here is a visual feature.


It consists of two things consists of a key point and a descriptor.


Now key points are the points in an image that the visual feature is located at.


So it's like the coordinates and they do not change or they should not change as an image


is rotated scaled or transformed in some way and descriptors as you might have guessed are almost


like a description of that particular visual feature. There are multiple algorithms for


extracting these features and creating these descriptors. You can use algorithms like sift


orb or surf. Now we're not going to go into detail on what those algorithms are doing right now at


least but we'll just understand that they are pretty much looking at looking at these images


they're finding edges changes in color changes in light and just identifying those and running us


through a set algorithm to create what we call the descriptor. Now the most common of these which


and the one that we are going to use is called sift. Now sift is pretty useful because it's


invariant to scale rotation translation illumination and blur to an extent of course it's not perfect


but it can deal with all of these things. Now sift first looks through the image and it identifies


a key point so something of that it believes is of relevance. So just as one example it will it


would look probably look up here and there's not really anything in that image right you like there's


no difference it's just one like flat color so it it probably won't extract a feature from here.


Okay but when it comes over here it's going to see it's going to see some edges you can see some


changes in color and it's like okay well there's something there so we put a key point there and


then it's going to go through all of its like algorithmic magic and based on the edges the


colors whatever else in that particular patch it is going to produce a 128 dimensional vector.


Now it doesn't stop there it doesn't just find the first patch it goes through an image it's


going to find like a load of patches especially for this image there's a lot of information


everywhere it's going to be like okay look there's a window there's this edge over here


something interesting over here is all these colors and edges like it's going to find


these key points these visual features everywhere throughout this image and we'll actually see


what these look like a little bit later on. So what we get is a load of these visual features


the order of those visual features doesn't matter they're all just kind of placed or


ordered suspended in the air over here okay we can almost imagine there's like another bag here


which is our bag of visual features before it becomes a bag of visual words. So we're kind


of at this point right now so every image just becomes this massive essentially an array with


all these 128 dimensional sift vectors. Now once we have pulled out all of our visual features from


an image we move on to the next step which is how do we go from you know visual feature to visual


word. Well we have to use what's called a codebook so a codebook is essentially it's going to be like


our vocabulary if you come from NLP again you'll recognize vocabulary is essentially like a big


list of all the words in your data set okay. This codebook is going to be a big list of all of the


visual words from our images but of course we don't have our visual words yet we need to create


them so that's what we're going to do next we're going to build our codebook which is basically


going to create all of our visual words at the same time. So how do we construct this codebook?


Well the idea is that we take a load of similar visual features and we actually cluster them


together and we get a centroid out of that cluster. Now that centroid is what we would call a visual


word. So we look at this image here right we on the left over here these are visual features


right we have multiple visual features there's all these things that look kind of like a dog's


nose from a slightly different position but you know they pretty much look like super similar


okay and what we do is we process all of these we use k-means clustering and we get a single


centroid out of this cluster of points that all look very similar okay so we go from what is this


this is five 128 dimensional visual features to one still 128 dimensional centroid or visual word


okay and we do that using k-means clustering. Now this does not just happen for a single image okay


we actually feed in all of our images from our let's call it a training data set okay so we'll


actually have loads and loads of these images not just one okay it's very very bad drawing and


okay we have like a dog's nose over here but maybe we have another dog over here another image of a


dog and he has a nose obviously maybe so these centroids here are also going to be very similar


to the centroids of this other dog's nose that are over here so in reality we don't just have


these five visual features that create our our single cluster over here our dog nose visual


word we actually probably have quite a lot so that's that's the process of creating code but we


basically go through all of the visual features that we've created we're going to create a lot of


them and we compress them into a set of particular points which become our visual words now this


process is called vector quantization it's where you take a vector and you take a vector


you take it's where you take a whole range of vectors and you do something like clustering in


order to create to convert this massive range of vectors into just a select few vectors okay


so again we can use the example or the parallel of language there are pretty much an infinite number


of sentences out there in the world but if we perform bag of words we extract every word from


let's leave stick with the english language sentences at the moment if we extract every


single word and we put them all in the bag and we remove all duplicates we're going to end up with


far less than an infinite number of items there all right that is that's quantization i suppose


quantization of language i'm going from an infinite number of possible sentences to a


limited number of possible words that make up those those sentences that's exactly what we're


doing here okay so just taking this one example we have like a church building over on the left


we have these two visual features right these these two dots they're visual features maybe


we can also have another visual feature over here over here right all of those should kind of be


very similar they're like the screen window they have all these similar edges and similar colors


so they probably all end up being clustered together into this single visual word which is


you know a window this one over here all right we have this kind of like bell tower thing


and we just take different parts of this bell tower and they would probably all feed into a


single visual word now one thing we should point out here is this is just one image now imagine we


took another image with other windows inside it that other image with with windows


let's say the window is here we would take the visual features from that and they would also


be translated into this over here so what we get out of that is rather than our image being


represented by all all these 128 dimensional visual features they are actually represented by these


technically 128 dimensional visual words but because there's a limited number of these visual


words we can number them so rather than so for example this one would be zero we this one we


selected here would be one two three and four and let's for now pretend this is our entire codebook


that means that this image here would now be represented by the number of visual words that


we found so the like the frequency of these visual words so at position zero we have a dog's nose we


didn't see any of those in this image so that would be zero this number one here so we found


like these two visual features over here another one another one here so we found four of these


right so then this would become four okay again we don't have like a i think it's a dog's leg there


i think it's a dog's leg there in this case we found two of these features so this is a this


one we selected here and then here we have zero again okay so what we've done just there is we've


gone from this representing this image with a very large number of visual features so 128 dimensional


and also there were six of them in this image in reality there'd be a lot more okay so quite a few


values there and then we've kind of compressed it down into this five dimensional sparse vector now


in reality we're not going to just have five possible visual words in the later examples


we're going to use around 200 visual words to represent just over 9000 images


okay but that's still a big improvement over you know what we're doing here so this massive


matrix that we have has just been translated into a 200 dimensional vector and as well i should say


the images that we that we take later when we actually process for example this image we won't


just get around six visual features we'll get more like say 800 visual features so in reality


be something like this 128 dimensions and then we would compress that down into a single 200


dimensional vector okay that's what we're going to do now if we consider okay how how is this


useful to us so we're going to focus on you know the information retrieval side of things but it


works as well with image classification where we basically look at so for now we're going to add


another step in in a moment but for now we're creating what are these frequency vectors so this


thing that you saw here so frequency vector and that frequency vector is basically telling us okay


we have you know in it we have this many i'm just going to use this example um this is i think okay


two or something like that we have two windows in this image in this image we also have these


two crosses on top of a bell tower whereas this image of a dog we have we have a nose we have a


leg we have a ear right so if you were to plot these in a high dimensional space we'll just you


know for for the sake of simplicity we'll pretend okay this is actually a two-dimensional vector


so our our dog image could be like here okay and our church image could be over here now


they're not similar okay we'll use cosine similarity so that basically means we're


going to measure the angle between vectors so if we maybe we had another dog image and because it


has many of the similar visual words it'll be plotting in a similar space in a similar vector


space so that means that the angle between these two here will be a lot smaller right when you


compare it to the angle between either of those and this one here okay and then we're going to


calculate cosine similarity between them and we'll see that the cosine similarity between our more


similar images with similar visual words they are more similar okay so we have our our sparse vectors


but this is you know there's another step that we need to we need to take and i'll return to


the language example in language we in nlp we have something called stop words now stop words


are basically super common words that don't really tell you much about what it is you're looking at


so the word the or the word a the word and all of these words appear in language all over the place


but they don't necessarily tell you what the topic of whatever it is you're talking about is


okay and in natural language processing these stop words can be overpowering that you know you can get


200 instances of the in a in an article and if for example you are reading an article about


like roman history you want to be looking for things like the word rome or the word julius caesar


or the words julius and caesar you don't want to be looking for the word v it doesn't tell you


anything so these sub words are usually they're either removed from your sentences or we can use


something called tf idf which allows us to look at the relevance of words based on how common they


are across different documents so uh in in the example i just mentioned we we let's say we have


articles about a lot of different things and one of those happens to be this roman history article


and it has the word rome in it all right so as a user when i'm when i'm searching imagine i search


something like the history of rome right so we have the history of rome four words there when


we're searching we don't want to give equal relevance to the word the as we do to rome and we


don't want to give equal relevance to the word of as we do to history right we basically want to


just ignore of and ignore the and just focus on rome and history right so tf idf is a very


useful tool for that right so tf idf allows us to do that by looking at a frequency of words across


all these different documents and basically scaling the score of our the frequency scores


that we we've just retrieved based on the frequency of these words throughout all the all


your other documents and the same is possible with visual words so we have all of our images


documents and we have all the visual words in all those all of those images we use tf idf to


look at the visual words found in each frequency vector and we use that to essentially lower the


score of visual words that appear all the time and increase the score of visual words that are


very rare and very unique to a particular image so in the church image that you see here in a lot


of cases sift isn't going to pick up patch number one but let's say it did in in this example so


it's picked up a patch of sky a patch of sky is probably pretty common in a lot of pages so we


would want to bring the score of that patch of sky down we'd want to lower it whereas this patch


number two we want the score of that to be increased or at least just maintained okay so


this is how we do that this is a typical tf idf formula now tf idf typically you know we think of


it with documents and terms which is just words so the notation maybe it's a bit a bit weird to


translate over into image images here but we'll do it anyway t refers to term in in typical tf idf


lingo but we are using it to represent visual word d usually means document


in this case our documents are images so it's an image okay and then we have tf td and that refers


to the term frequency of a sort of the frequency of a particular visual word in a particular image


okay so the frequency or term frequency of visual word in image okay let's say we took


over here we have visual word one so tf of visual word one in image we'll call this image like image


zero in a zero would be equal to two okay because there are two of these patches or if we want to


include the other ones it would actually be four n that you can see over here that is the total


number of images so number of images dft here so that stands for document frequency of term so


essentially how many documents does our particular visual word appear in or how many images does our


visual word appear in and all of this is so we we take both of those and we take the log of them


okay so this here becomes the this is the inverse document frequency this this whole sort of part


here this is the term frequency we multiply those together and we get this and that gives us the tf


idf now the idf part if you if you look at this so just consider this it's useful to understand


we have a number of images number of images does not change based on which image you're looking at


and the document frequency for a particular term that also does not change depending on whether


you're looking at a particular image or not so this idf vector is actually the same across our


entire data set okay it is not going to change the only thing that changes is the tf idf which


is the term frequency so using this we can go from what we have here so all these all of these


visual words are kind of on the same level that it's not really any variation there


to something more like this so maybe we've decided that a nose is a pretty good indicator


of something relevant okay whereas like this leg thing here it's not a very good


indicator or anything okay so they've been adjusted in accordance with their their relevance


okay so we've just been through how to create our visual words how to create a it's like a sparse


vector representation of our images based on those visual words and then how to scale the values in


those sparse vectors based on the relevance of each of each visual word that the values represent


so after we've done all that we're sort of ready to move on to the next step which


maybe you want to feed all these vector representations into a classifier model in


order to perform some classification or what we're going to go through is how we can


actually retrieve similar images based on these vectors now there are different


distance or similarity metrics that we can use when we're comparing vectors you know you have


dot product similarity you have Euclidean distance but we're going to focus on using


cosine similarity now this is really useful because the tf-idf scores that we've output here


the magnitude of those of those vectors can vary significantly so by using cosine similarity it


means that we don't need to normalize anything so we can just take these vectors and we can compare


them immediately which is really useful so cosine similarity is calculated like this so let's say we


I'm going to draw another one these 2D graphs right let's say we have vector a here and we have


vector b over here okay the dot product on the top here this is going to increase as these


as these two vectors are more similar but that by itself this dot product value it works poorly


when we have other vectors of different magnitude in here so if we had all these different vectors


so we had one over here although we might say that we're going to call this one c c i'm going


to call this one d to me it seems like b and c are probably more similar than c and d but because


d and c have a greater magnitude the dot product which considers both angle and magnitude is going


to produce a higher score okay and that's even true if maybe let's say we had like b and another


let's say we had like b and another item here even though these are basically the same vector


they would score lower than d and c purely because d and c have a larger magnitude


okay so what we do is we add this item here which looks at the magnitude of a and b together


and we we divide by that so we're essentially normalizing everything in this in this calculation


and that's how we calculate the cosine here so we're actually just calculating the cosine between


between angles okay and that gives us our cosine similarity so let's have a look at another example


so here if we had two exactly matching vectors the cosine similarity between those would be equal


to one if we had two vectors that were at a 90 degree angle so vector a is perpendicular to vector


b we would return a value of zero now because we're using tf-idf all of our vector values are


positive so in reality all of our vectors would have to be placed within this space here so within


like the i'm assuming this is the positive of the y-axis and this is the positive of the x-axis


so all of our vectors would actually appear in this space here so this vector a down here


would not happen but just using it for the sake of showing you that we can have a cosine similarity


of zero now in this image here i'm trying to point out the fact okay again it's the same thing