And the answer is, you should somewhat mix them. And how much that's that's detailed in the plot here. So you can see right here, R is the ratio of self supervised to supervised pre training data.


ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (Paper Explained)

And then the y axis is the performance when fine tuned on superglue. Now to the if you go to the right, you can see that the blue line approaches the dashed line and the dashed line is simply the performance without any supervised data at all.


So if you simply pre trained on the masked language modeling, and then you find tune on superglue, it makes sense as R goes to infinity, the proportion of supervised data approaches zero. So that blue line must meet the dashed line.


However, it's pretty interesting what happens at the beginning note that if R is zero, the performance is terrible. That means if you only have like supervised tasks, that is crap. That is just not good at all.


And I think this, this might even be like this, I think this is super important recognition, you know, for all the while you need a lot of tasks and so on. If you don't have your language model pre training, if you don't have yourself supervised pre training, even if you have the same amount of data points, it is going to be crap, right?


So there is something for having the self supervised tasks right here and having them all actually learn these, like grammar, and continuing just bland pieces of text that are not geared towards some specific task.


Because, you know, if you then have to pick up on new tasks, it kind of helps you that you've just been around language, it seems.


And this is also a hypothesis is that these supervised data sets by means of all being on the same task, right, every one of these hundred and seven data sets contains data points that is like structured exactly the same.


The amount of world knowledge in there is quite big, yet the amount of language knowledge in there is quite small, right?


So all the all the fungal all the fungal data set prompts are going to look not only are they all going to start with like parse to fungal colon, but they're going to kind of look the same like they're going to be like the sentence they're going to be very much the same.


The answers are going to be well, they're not even text, but also they're all like, very much the same, you're not going to get random text in such a query, you're going to get text, you know, for the, for the entities it knows, and for the kinds of facts that it wants to express.


So the supervised data sets while being good in having labels, they're quite bad at actually expressing language, which is, is interesting, right? Because it is something that is especially pronounced in language, I feel, because if we, for example, look at images, would it, I guess, I guess it might hurt to if we if we had like ImageNet, but so if we had like huge, huge ImageNet,


maybe it's still hurt because it's only like the thousand classes and images tend like ImageNet images that are made for classification tend to like display one particular object in the middle. Yeah, I don't know.


But in language, certainly like NLP tasks, they have super limited language. So it's important towards here, we approach, obviously the performance without supervised data, if we don't have any self supervised data, it just does not work or it just works very poorly.


However, there is an interesting regime right here. As you can see, as soon as we hit one, which means equal amount of self supervised and supervised data, we are performance improves over just using self supervised data.


And then if the ratio is two, that seems to work quite well. As you can see right here, that is almost 80. That is, that's quite a big jump in superglue performance. However, as we know, if r grows, this must come down again, it comes down pretty quickly, already at four, it seems to be almost back to where it started and then kind of wobbling around.


Or, or so it seems, of these experiments are big, so you can't exactly fault the authors for not evaluating many times and giving error bars, though, here in this case, it would have been quite nice.


So, you can see the window is quite narrow. And that is a little bit of a disappointment by me or a bit of a criticism here is that, yes, it's good that you can get something if you have this many pre training tasks and so on.


But in a real world scenario, right? Um, you know, I don't, I don't know how many how much I need of that, right? I could go with a good guess and say, well, probably other data sets have similar behaviors as superglue.


But, you know, who knows? And generally, if windows are so small for hyper parameters, it's a bit of a risk. And if in practice, I always have to do giant experiments to evaluate whether that hyper parameter setting is still appropriate.


And it's not really a game because, like, doesn't save me anything, because I always have to do this evaluation.


But maybe, you know, maybe this turns out to be fairly robust. This number two, who knows, in their experiments, at least it did.


Then they go on and they ask, Okay, so we got 107 tasks. If we simply pick random subsets of these tasks, how does that impact performance?


As you can see, there is a trendy trend ish thing sort of upwards, there is a bit of a downwards here, but nobody knows because as you can see, here we do kind of have error bars, and they're kind of big, the standard deviations.


What you can also see is that for larger batch sizes, the trend seems to be kind of more consistent and better. We've known for a while that larger batch sizes help these things.


But I guess especially if you have so many different tasks together, having a big batch size means that it's much more probable that you have sort of balanced data that you don't end up with a batch with only data from one particular task, which kind of wanks all the other weights out of out of whack.


Or if you have like two or three batches, one after another of these, it can be quite detrimental.


So that is maybe one but yeah, that like the standard deviations are quite big, as you can see, but clearly the higher batch size is better, even if they have the same same compute budget, right?


So it's not like the higher batch size experiment, like some more data, at least I don't think so, or had more compute.


And the last thing is they compare, when they compare during training, so they ask themselves, which is this really sample efficient? And the answer is, compared to like a T five model? Yes, it's quite, it's quite sample efficient.


As you can see right here, comparing these two models, you can see that even after the same compute amount of compute after the same amount of steps, the EXT five model is quite a bit higher than the T five model.


So it's not just at the end, it's in fact, during all of training, so you could potentially stop earlier and get like the same performance as the T five model would get much later in the training.


So for example, this point here is already reached here way earlier, of course, the longer you go out, the more extreme this effect gets.


And that's basically it for the ablation experiments. This is the part of the paper that I quite liked, because it's kind of investigative and it sort of justifies their, their choices.


And the choices are pretty simple, right? It's just like throw everything in there and do everything at the same time. But instead of just doing it and then evaluating which they do down here, obviously, they're kind of better at most things in in distribution tasks out of distribution tasks, yada, yada, yada.


Like they're just they're better, like, trust me, though it is it is interesting to see that very often they're not that much better, like, they're kind of like, all right, T five gets like 29.01. And you get like 29.49.


Like who knows how big of a difference this is, it is, I guess, for a machine translation, that is a difference.


Still, it's not that much, honestly, on other tasks, you can see right here, the T five gets, what is this 55. And this is 63. This is quite a big difference.


I think I saw other tables where the differences were even more drastically. So it seems to really depend on the task as well, whether or not this can get you a gain or not.


Yeah, so I quite liked, I quite liked this investigation into are we doing the right thing, even though, even though what they want to do is the most simple thing is just like throw everything together into one giant model and add some self supervised as well at the same time.


But yeah, it is interesting to see.


And if you want to learn more and dig into their exact results right here, this is all available in the appendix, there is a split of exactly how the 107 tasks are, if I understand correctly, here you can see the different data sets used to construct ex mix implementation details and so on.


It is, it is quite thorough.


Yeah.


So that was it for ex T five. In summary, this is a T five model that has been supercharged by pre training it on a combined language modeling objective and supervised objective.


That is a mixture of 107 different NLP tasks, all at the same time in a ratio of two parts of self supervised and one part of supervised data. And that turns out to perform extremely well if you fine tune it on downstream tasks.


And in fact, it's not really easily possible to outdo that recipe by doing something smarter out of the box. So by selecting a good subset of tasks, or by, or was my point ready, or by somehow staggering the things like doing this, like, first the pre training and then the pre fine tuning, and so on.


It's not easy to beat it.


That was my point, my, my two cents on this paper. If you enjoyed it, leave a like, if you didn't enjoy it I guess you can leave a dislike but you know what's it gonna do honestly, YouTube.


I still see it right I still see how many dislikes I got but if you dislike the video tell me in a comment what you dislike.


Synthetic Petri Dish: A Novel Surrogate Model for Rapid Architecture Search (Paper Explained)

Hi there, today we're looking at Synthetic Petri Dish, a novel surrogate model for


rapid architecture search by Adi Tarawal, Joel Leeman, Felipe Petroski-Suc, Jeff


Klune and Kenneth O. Stanley. This paper on a high level, it basically says if you


want to do neural architecture search, if you for example search for a better


non-linearity, you should be able to extract that non-linearity instantiated


in a very small network and then evaluate that very small network in


order to predict the performance of a large network and therefore you can find


a better non-linearity in much less time. Now the exact procedure how you do this


in the small network is the topic of this paper. As always if you like content


like this I encourage you to subscribe if you are not already and to share out


the video so other people can experience the joy themselves. All right let's


dive in. So they say in the abstract neural architecture search explores a


large space of architectural motives which basically so it basically means


you want to find a neural architecture let's say you have a multi-layer


perceptron right here a couple of layers okay and they're all connected by you


know feet forward weights whatnot and each of these weights basically is a


multiplication so each one of these is a multiplication of x by your weight w and


then there is a non-linearity so the non-linearity could be a sigmoid so the


sigmoid would be something like 1 over 1 plus e to the negative x now there's a


bit of an extension in a sigmoid where you can do a sigmoid that has like a


temperature parameter attached or a slope parameter where you go cx so in


one case you can set c such that the sigmoid has a shape like this and in


then if you put c to a different value you can make this slope you can make it


like a shape well this is terrible like this you know what I mean okay so this


this C right here can potentially change the behavior of your network and you


want to find a good parameter C and this is a hyper parameter now there are many


hyper parameters like this for example how many units you have in a particular


layer in a CNN it could be your filter size in a transformer could be the


number of heads and so on it could actually be not only the slope of the


non-linearity but the actual non-linearity itself or famously in


recurrent neural networks you have these recurrent cells and they're like okay we


have an input signal and a carry signal and then the in the input here is like


dot multiplied here and then there is like a gate with a non-linearity and


then it's kind of like multiplied by the carry but then there is also a like a


forget gate and whatnot there's a minus right here it's very complicated and so


people do architecture search over these kind of problems to find better


architectures for particular problems now the problem is of course that how do


you know if if a given architecture is good what you have to do is you'll have


to go take that cell that you have dreamed up you think well I think that's


a good cell and you have to train it on the full training data set this is a


data set a database right this is a full training data set then you need to


evaluate it on your validation data set and then you have like a number you have


like okay this is eight good and then you go back and you say okay what if I


change this cell here what if I change it to a plus instead of a minus and you


do the entire thing again train it for I don't know how much validate it and then


this is like a nine and you can say oh cool that's a nine so this is a very


basic architecture search and there has been a lot of development in this space


so like evolutionary search and so on but they most of the time they require


pretty much evaluating the entire thing on the full data so you get a good you


get a good estimate of what your final performance back here is going to be now


people have come up with methods to counter that and they say well if we can


sort of encode the cell structure let's go with the let's go with the RNN cell


if we could encode the cell structure in in a sort of a continuous way so you


know we can encode text in a continuous way let we could also encode a cell


structure because the cell structure I can write it down as an equation I can


say like okay it's the forget gate of the carry times the sigmoid output of X


plus the so this is the plus here and plus the sigmoid output of X multiplied


by the input let's call that I something like this right this is text I can like


write it down and then I can encode that into a vector much I can for example


build another RNN ironically or something to to encode that or I can


represent it as a computation graph like it is here and use a graph neural


network to encode that into a single vector and then I have sort of an


embedding space where each cell that I could build is a point in that embedding


space and then I can evaluate a couple of them I can for example say okay this


one here this one here this one here this one here I'm going to train them


these cells I'm going to do the full training eval and so on get their


scores and then I can learn basically in this latent space I can learn a


predictor I can say okay here I get I got an eight I got a nine I got a two


and I got a four so it appears to be that in this direction that the good


cells are in this direction and then I can do it again I can sample or I can do


gradient descent in this space since this is now a continuous space and the


gradient descent on the model that gives me this space it's right so this this


method basically tries to take in the building plan of a cell and learn to


predict the performance just by looking at it if if you're thinking of the


touring machine right now then you like I I immediately thought of like this


this halting problem because it appears to be exactly what it is so you're


trying to build a machine that takes the building plan of another machine and


tries to predict its performance now in a general sense we can already state


that this problem is sort of the difficulty of this problem is


equivalent to the difficulty of the original problem so I'm not sure but it


appears to you know it appears to work if you throw lots of compute at it but


of course that's a problem you need lots of compute right so either your your


option one is to run all of these things and kind of iterate them in a neural


sorry in an evolutionary way or your second option is to take the building


plan and predict the performance from that both are not satisfactory and both


use lots of compute now neural petri dish is a or synthetic petri dish is a


way to combine the two together it says can we take the building plan here but


actually run on the data on data to predict the performance so what they're


saying is basically if I have this cell right here and usually this cell you know


it deals with vectors of let's say size 512 and so on it will say it since this


is let me draw it again up here so here you have the cell and here you have


somehow the connections in there when you carry and the input and the input


here okay and this is the output or the carry I have 512 embedding like size of


this so this is a giant cell there's 512 the vector has 512 dimensions going in


basically can't I take the exact same thing but and keep the connection


pattern so I would keep the entire pattern of connection right here but I


only do it for one or two so this is 512 and this is just two right just the I


just reduce the dimensionality but I sort of keep the connection pattern


alive if I only do that I have like a very small network right now and the


same goes for if this is so a lot of times these RN ends they have multiple


layers of these things so they have another exactly equal box up here and


then another up here I can just reduce this to one layer and out of the


regularity of these neural network things it is known that or one can make


the assumption that the performance on this thing will sort of kind of be


correlated to the performance of the entire thing and that's one of the


things that this Petri dish paper does so we try to take out what we are


trying to search over namely the connection pattern we keep that as it


is up here but we reduce everything else we reduce the dimensionality we reduce


the number of layers and so on now they don't actually reduce the number of


layers here but you can reduce the number of units and so on okay so this


in essence this works whenever you can do that whenever you can keep the


structure you're searching over but can reduce the rest so that's one


precondition doesn't work for everything the second part here is that


you don't want to use this particular training data and then this particular


validation data because first of all it's a lot of training data and second


of all it won't give you that good of a prediction instead of what you're trying


to do and this is the second part of the idea of Petri dish is you're trying to


abstract the training data to get you a very small data set such that and the


validation data as well as well such that if you train on this data and


evaluate on this data the performance that you get will be very predictive of


the performance had you trained the big model on the big data set okay in fact


in this Petri dish paper these little data sets they have nothing to do with


the with the original train and validation data and that's I think it's


one of the cool things here these things this training data and this validation


data they are optimized as well by the procedure they're optimized special data


points that are trained these are trained parameters such that if you


train on the training on the small training data and evaluate on the small


eval data the you will be able to predict this performance back here with


high accuracy okay and this I think is where previous approaches have or might


have failed because it's you know the the idea of scaling down your network in


order to do the architecture search is probably has it has appeared to many


people before that's not you know that's not really genius idea but probably they


have all found that now we can't really do it it doesn't really give us accurate


enough numbers but in this case the addition of adding these synthetic data


sets that are much smaller but can still if you train and evaluate on them can


still predict with high accuracy the full score of the full model that I


think makes this idea work all right so I guess we're already through the idea


and problem setting and everything without actually reading the paper so


they give this example right here at the beginning that if you have a a two layer


100 wide so 100 dimensional MNIST networks it's two layers it's I think


it's an MLP two layer MLP with a non linearity that is this the sigmoid right


here this okay now you can see it has this temperature parameter here it has


this slope parameter and you you want to do neural architecture search to find


the best slope parameter now usually you would just do a grid search but this is


an this is an example because this can be of course much higher dimensional


things and then you don't want to do grid search anymore okay so what do we


do if you look at how the 100 white MNIST network so we can draw it right


here so this is a 100 dimensional MNIST network so this is 100 and each cell


each connection here first has a weight and then has the sigmoid non-linearity


and the sigmoid non-linearity is parameterized by the parameter C okay


and you have you have many of them right you have one here and so on and each one


has a different C and each of these networks represents one blue dot here


so if you let C vary so this sigmoid slope value right here that's your


parameter C if you let this vary and train the big network on the entire


data set to convergence and then you eval on the validation data set you get


the slope like the blue curve so if you see the blue curve the blue curve is if


you start over here if you reduce this slope you'll gain in performance but if


you reduce it too much you drop drastically okay until it's if it's if


it's zero it's basically you know the the X is not the signal doesn't propagate


anymore and you you have no learning occurring okay so that's the original


performance now what if I only give you training data in this range right here I


only showed you this particular range I can can't actually zoom in that much but


if if I give you this and I ask you to build one of these please take the


architecture and predict the performance that we saw at the beginning like one of


these good little machines or Turing Turing machines you would it's basically


say well that looks to me like a line so I'm gonna predict the red thing here and


even if you can you know evaluate a bunch of these it just looks like a line


and you're going to predict it's probably a slope like this right this


happens almost independently of which model you choose to predict right here


the the data of training is simply doesn't give away that the fact that


there is a there is this break down here which happens in the real world so if you


just give this as training data there is no way so so the the criticism about


these models is valid that they will only work where you give them training


data they can at best interpolate their training data but they can't really


extrapolate now here since the synthetic Petri dish method which is the green


thing here uses the actual not the actual non-linearity that this thing


characterizes so it it instantiates the sigmoid with the parameter C that you


give it just not on the large network but on a small network in fact their


network is just one unit sorry one unit and then another unit so it's just a two


hidden layer but just with one unit instead of 100 and of course you can't


feed in MNIST right here right but we said they don't feed in the data they


actually feed in their synthetic data that they learn so you give them the


points here and they learn the synthetic data to to evaluate to evaluate the


others and then once you ask them well if if my C is right here what's the


performance going to be it's going to instantiate that in its small network it


is going to use the training data that it has learned from this region right


here in order to train this and then it's going to evaluate this on the


synthetic validation data that is also learned on the training data and it is


going to come up with a performance metric it says okay this is how good


it's going to be and since it is an approximation in its building plan to


the entire network it will react similarly so it will get that there is


this performance dip right here okay so it you can see how this sort of makes


sense you are actually running an approximation to the actual program


instead of just looking at the plan of the program and trying to predict it


which you know halting problem says hello okay so that is the motivating


example of their MNIST thing and here is the entire algorithm all right so you


take MNIST training and validation data and you instantiate a bunch of really


big networks this is ground truth okay you you need this you need this to learn


from you instantiate a bunch of really big networks now if I draw the graph


from before right we had this was the performance of the actual networks you


want you this comes from here from this region right here this is the training


data okay so you instantiate a bunch of these networks each one you instantiate


in one of them right each one gives rise to a different non-linearity and you do


the full training ground truth training and evaluation on the full training set


and the full validation set and you get validation losses right for each of


these and these are the points right here now you that's the training data


for your neural for your neural architecture search so for your Petri


dish method what the Petri dish does is it says it extracts the motive and the


motive is the thing that you optimize over so as I said you want to keep that


thing in its essence but you want to reduce everything else so it reduces it


instead of from a two layer on hundred YML MLP it reduces that to a two layer


single neuron wide MLP okay and it now this over here is the training data for


the procedure that we're going to do now so what it would take is it would take


it would take one of these values it would instantiate and we have that here


would instantiate the neural network in the small form of that and now we know


that if I train the full data and evaluate if I train on the full training


data and evaluate on the full validation data I should get this accuracy all


right so I will create and we're going to look at in a second I will create


training and validation data such that if I train on this training data and


then validate on this validation data I get the same validation loss as if I had


trained the big network with the same you know the same C parameter on the


full training data and evaluate on the full validation data okay so in this


step I'm optimizing the data here the training and validation data all right


and now in the second step once I have this training and validation data such


like that I can basically reproduce this this graph right here then I can go and


actually ask my model okay now please tell me what happens over here so what


am I going to do I'm gonna take that I'm gonna instantiate it I'm going to use my


training data that I learned to train it I'm going to use my validation data that


I learned to evaluate it and it's going to give me a number and that number is


going to be like close to hopefully close to do this so this is how we can


extrapolate using that method okay now there are a number of assumptions right


here and you can imagine this doesn't work in any situation this works if if


you you know if you basically you have to get lucky in that you have to


abstract the correct things right I said you need to reduce everything else so


they reduce notably you see they reduce the 100 the 100 layer width to a single


neuron wide MLP and they sort of guess that that doesn't change the fundamental


thing but you can also see they leave the two layer right they leave the two


the two layer neural network and I'm can almost guarantee you that they tried


this reducing this to a one layer neural network and it did not work and so you


have to be sort of very careful of what quantities you abstract and what


quantities you don't because okay now you might always think oh I can reduce


the you know number of dimensions or channels that's also not always the case


so I think that's kind of the crux of the method you have to actually


engineer this down compressing of the architecture such that it's its


properties are still kept and yeah but yeah in other things how do you how do


you actually produce training and validation data to match these and there


are a number of ways but what comes to mind is is meta learning right so


because what you're doing they initialize the training and validation


data at random points so these are just random at the beginning and then they


optimize the data itself using gradient descent okay now see synthetic training


data and they are randomly initialized okay and they use gradient descent they


have it somewhere yes so they have this inner training loop okay which is many


steps of inner training and then they have the outer loss which is the it's


the validation loss after the inner training loop and the difference from


that to the true validation loss and then they do gradient descent on this


outer loss now this outer loss is a result of the inner loss and the inner


loss is a result of the inner training procedure and the inner training


procedure is n steps of feeding in the training data every step you feed in the


training data so your computational graph is going to look like so here's


your training data as train and here are your initial parameters you at random


lies initialize them randomly in the first step use the training data to


produce theta one then in the second step you use your training parameter


your training data again to produce theta two and then you use it again to


use theta three and so on each time you feed the training data in order to


evolve your parameters to give you a better prediction right so the gradient


since somewhere back here there's a loss the gradient here will have to flow back


through all of these paths and through all of these connections to the training


data this is kind of you back propagate through an optimization procedure and we


have this a bunch of times here and I've looked at the code and the code is like


really crazy and it looks like proper research code but it appears to be that


that's actually what's happening they back prop through the optimization


procedure to find this synthetic training and validation data now that's


I mean that's crazy but it also kind of limits how far you can go with this


because usually you can't back prop for more than a couple of steps doing this


now that the model the fact that the model inner model is small helps but


also this introduces very very much like these things are very brittle if you


back prop through an optimization procedure like this these things tend to


be very brittle and so I think there's another thing there where you have to


pay careful attention all right that's it's basically it the last thing they


say is that they can combine this with architecture search in that so not only


can you predict good architectures what you can do is you can actually predict


the which architectures are good and then you can use that prediction to get


new to basically input this into your neural architecture search to inform it


so instead of the neural architecture search having to evaluate all of the


candidates that it produces it only has to now evaluate the very small subset of


candidates that the synthetic petri dish training deems most worthy of being


evaluated in this case here instead of evaluating all of the things here it


would limit itself to whatever the synthetic petri dish says are the


highest performing ones because if the synthetic petri dishes and is any good


then it will you know give accurate predictions of how they're performing


and then that can go in multiple rounds so the architecture search can find new


come up with new things that it thinks are better through like an evolutionary


mutation algorithm the petri dish can evaluate them in the synthetic way and


then suggest the like ten candidates to evaluate on the full test set and you


see that way you don't have to evaluate all the like thousand candidates all


right all right cool they do this for this MNIST and they also do it for


finding a RNN cell for the pen treebank this is a language modeling task and the


this is a benchmark for neural architecture search where you're trying


to find a good RNN cell to get the perplexity really low and here you can


see if they give the same amount of data to all the methods then the benchmark


neural architecture search is worse than the synthetic petri dish informed


architecture search now one has to say on the full date I believe the NAO gets


to about here but of course if you give all of them the same data the neural the


petri dish beats this method and I think still this method here uses way more


compute because it always has to evaluate all the candidates and that's


exactly one of these where I learn an architecture to predict the other


architecture by just looking at it so it works but it doesn't work as well as


actually running the architecture in an abstract fashion this also shows you the


importance of selecting your experimental evaluation in a smart way


like they argue they argue for very long why it makes sense to evaluate


everything on reduced data such that their method here can be better and they


don't have to compare to the full thing it's easier for them to work on reduced


data and they argue you know it's it's it's what people usually do in practice


and that's the the task they focus on so you know good good good good paper


writing right here yeah that's basically it to the paper there's a lot of things


to be said here I think this works in very very limited settings it seems to


me that it's sort of brittle with respect to how you abstract and also it


it's always the case like how many how large is this synthetic training data in


their case there they like abstract this to 20 or 30 data points or something like


this so it seems to me since you're optimizing this training data with


gradient descent what you would mainly find are adversarial sort of adversarial


examples to this architecture here so I'm going to guess that the inner


optimization is very noisy and that's because if you really let your


optimizer run then it will abuse every single thing it can to match that


validation loss and that will usually lead to an adversarial example since


you're optimizing the data itself okay so I think this suffers from that and


this is we had this in the in the planning you know the planning in in


learned world models and reinforcement learning where if you have a really


really good planner it will just abuse the mistakes that you make in


approximating the true world and the same here you're going to make mistakes


approximating this architecture here and the better your your optimizer is for


producing this synthetic data the probably the worse the worse the result


is going to match the worst that these losses are going to actually match now


okay these losses will match because they're that's what you train for but


the worse these two curves will match each other because now you're just


finding adversarial examples for your particular training data another concern


I have here is with respect to the double descent phenomenon so if you know


the double descent phenomenon if here you have your number of parameters and


here you have your validation loss let's say and you know that if I add parameters


I can make my validation loss go down so this is assuming I have a model with p


parameters and I always train it on the train data to like to convergence if I


add parameters I can generalize better until a point where I add too many


parameters and I start overfitting and my validation loss goes up again but the


double descent phenomenon and I think I've done a video on this shows that


after a certain threshold you get the interpolation threshold the validation


loss goes actually down again it goes down even further here now I'm so this


is a very strange phenomenon by itself but I'm sort of concerned that if you do


this abstraction that this paper proposes so you read your let's say your


full model is here with a large number of parameters so it is past this


interpolation threshold if you now seriously reduce the number of


parameters because you want to go into this petri dish you will get maybe you


will cross this interpolation threshold and actually be on this side of the


curve right here now of course at the same time you reduce the amount of data


which would push you over here again but it is different data so I'm not sure how


all of this is going to play out it appears to work in these settings right


here but I I think this is it's it's sort of it's sort of applicable in some


situations and it's it'd be very cool if we develop this further such that we


understand when it applies and when we can use it because I feel this can be a


very cool thing if we understand it better and if we can apply it throughout


all right that's the end if you like this paper leave a comment if you didn't


BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Hello everyone, today we're looking at BERT, Pre-training of Deep Bidirectional Transformers


for Language Understanding by Jacob Devlin and Ming-Huai Chang, Kenton Lee, Christina


Tatanova.


These are people from Google AI Language, so you're about to see the most hyped model


currently.


So basically BERT is a model that takes as an input language, so token sequences, and


outputs various things.


So it can be made to do various things, almost any NLP task, with basically little training


because the BERT model comes pre-trained on a very large corpus, and we're going to see


how that's done.


Alright, so the paper introduces basically the kind of current state of the art of language


models and they say okay, what they want to do new is they want to do bidirectional training.


I'm going to go down here and see their comparison.


So here they compare three models and these are representative of three types of models.


So first here is for example the OpenAI Transformer.


So this is the classic, or one of the classic transformer models.


We've talked about transformers before in the Attention is All You Need video.


So what a transformer does is it uses attention and for those who forgot what attention is,


if you have like a token sequence A, B, C, D, E, then a classic model to use that would


be an LSTM.


So the LSTM would go here, it would like have a vector representation, a hidden state, and


then it would take this A, it would take this hidden state and compute a new hidden state,


and then it would go on and take the B and incorporate this into the hidden state.


The hidden state kind of always stays the same size, but the recurrent model will update


the hidden state as it goes over the input sequence.


So this is one way of dealing with language, but people have kind of done another way and


that's the attention based mechanism is where basically for each of these you compute a


vector independently of each other.


So each one has a vector representation and then you have a vector representation of what


you want, which is called an attention head and you can have multiple of these.


But in the simplest case, let's just say we are looking for the subject in this sentence.


So A, B, C, D, E is a sentence and one of the words is the subject of the sentence.


Then we could have a vector here that's called a query vector.


So these are called values v and this is called a query q and then these vectors are the same


size.


I know I'm very poor at this.


You're going to compute the inner product with each of these.


So the inner product you want to do.


Okay I already screwed this up.


You're actually computing two vectors for each token.


So one is the, but this is not too important for this step.


One is the key and one is the value.


All right, value and this is called the key and you have your query q and you compute


the inner products actually with the key.


Sorry.


The values aren't too important for what I want to demonstrate but you compute key with


query.


All right and that gives you basically for each key it's going to give you an output


and so you're going to have a for this a, b, c, d, e you're going to have like this


much inner product, this much inner product, this much, this much, this much inner product.


So after maybe a softmax you have like a nice distribution and then you can say aha here


this is the biggest alignment of the particular key with my query and my query is which key


is the subject.


Of course you're going to train all these queries and keys producing procedures.


So this is this attention mechanism and if you then want that's where the value comes


in you can if your query is not only which one's the subject but it's actually a generic


query that okay I'm going to extract some information from some token that I'm going


to use later then you would actually take b and say ah b is the best one okay I'm going


to take the value of b or basically going to take a weighted average of the values according


to these values here.


So this is very shortly what attention is.


If you want a lengthy explanation go to the attention is all you need video.


So OpenAI GPT uses attention here and it's a left to right transformer that's what it


says here and what that means is it goes also step by step but in each step it uses attention.


So here is the input tokens and as you can see it goes in this direction so each one


of the and these are multiple layers of attention so you can also layer these of course.


So each one of the attention intermediate steps can only attend to whatever is to the


left of it.


All right you can see this here.


So it goes step by step and it goes left to right basically so you can you can kind of


take the sequence in as a left to right input.


Basically what that means is whenever you interpret a particular token you can your


context is only to the left of that token you don't know what's coming yet.


It's like when you when you read a sentence from left to right but then as humans unconsciously


we probably go and at the end of the sentence kind of make sense of the thing as a whole


but here the model is forced to make sense of what the thing only from whatever is to


the left of it.


So that's a basic limitation of these left to right models.


Then there is another approach which is called Elmo which has been popular recently as a


substitute for word vectors.


So if you know word vectors, word vectors are basically the kind of first stage in most


language processing task where for each word say the cat sat on something.


For each word you have you have a big giant table and for each word you associate a vector


of fixed size dimension.


So you place every word in a vector space and these vectors you pre-compute with something


like word2vec or glove and it gives you a nice way to basically deal with these words


in a canonical way and you can pre-train the word vectors.


It's all really nice but people have realized okay words can have multiple meanings and


words can kind of slightly change meaning depending on words around them and so on.


So what Elmo does is Elmo uses two LSTMs.


One LSTM goes into this direction, one LSTM goes into this direction and basically a single


LSTM as we saw before it takes in the input sequence one by one.


So here e1, e2, e3, en it produces hidden states at each step.


It produces a hidden state that is a result of a previous hidden state and the current


token.


And then what it says is okay now these hidden states here basically these are now the embeddings


of the token e1, e3 and so on.


These are the embeddings.


So the word vectors as to say are no longer just one vector per word so they're not in


isolation anymore.


But basically you need the entire sequence to compute the word vectors as a result of


this LSTM.


This is more powerful because it can give individual words multiple or each basically


each word has kind of a unique embedding depending on the surrounding words.


You would still hope that a given word would have similar embedding or similar word vector


all across the language but you can kind of fine tune it to the particular sentence it


is in and also you can completely change its meaning if it's kind of a word that has a


completely new meaning in that sentence.


So basically it uses two LSTMs one as I said here forward one backward these also have


multiple layers and so on.


And each of these produce one such hidden vector per token and you simply concatenate


the two from the from so from here this LSTM on the left produces one this LSTM on the


right produces maybe here another one.


And you simply concatenate the two to get the final embedding the final word vector


for each token.


So the fundamental limitation here is that this is kind of you have information from


the left and you have information from the right.


So other than here the original transformer you actually have you actually can condition


on the left context and the right context but it's very it's it's very shallow because


it's simply a concatenation of the left facing LSTM and the concatenation of the right facing


LSTM.


And these ultimately intrinsically they have nothing to do with each other.


So you simply concatenate the two things that the left facing LSTM still can only see to


the left and the right facing LSTM still can only see to the right.


So you basically have two half blind models and then you kind of concatenate so the it's


still suboptimal because of what you want is you want a single model to output your


word vectors or to interpret the language that can look at both the left and the right


at the same time and then incorporate information from both of them simultaneously and not just


at the end by concatenation.


This is what BERT does.


So BERT here and this is kind of what they claim is the new contribution.


BERT at each in each layer here of the model the the let's look at this and for a particular


token they look at all of the context.


So every every other token in the in the input they look at that.


And so the the basically it seems kind of it seems kind of obvious but it's it's actually


there's reasons why these other models don't do this.


But so this is the entire point of BERT is at each layer in this in this transformer


architecture is still an attention mechanism by the way.


So there's there's the mechanism of attention here and here is exactly the same or almost


the same.


They actually keep it close on purpose in order to compare.


But now we have attention not only to the left but also to the right to everything.


Right.


So why do these other model why do for example the OpenAI transformer only look to the left.


That's because somehow you need a task to train on.


And most of the time if you especially if you want unsupervised training you going to


do something like language modeling and language modeling what you have is a sentence A B C


D and you're asking what comes next here.


All right.


So by by the definition of the task you can only look to the left.


That's that's just how these like how the task works.


So it makes sense that these other models kind of do this because they they pre-training


this BERT has a different pre-training because they can they can only they have to look to


the left and the right.


The other thing is what you want to use the model for.


So the good thing if you if you go left to right is you can use the model now for generating


language in the same vein.


If you have a B C D and you ask and the model is trying to produce the next character only


looking to the left.


Right.


Then you can you can say what's the next character.


The model says E and then you can feed the same thing into the model and say OK what's


now the next character.


Gee so there's pretty useful if you only look to the left you can actually use the model


then for generating language which is something you can't do with BERT or it's not it's not


really obvious now how to do it with BERT.


People are I know people are investigating into language producing producing entire sequences


with BERT.


But as yet it's not super clear how to do this with this model.


That being said the model is pretty good at pretty much everything else.


So let's jump in to how they train.


They train let's see where we are here.


They train using masked basically masked language modeling.


So I want to actually go into that first masked language modeling.


What they do is they basically replace some words by the mask token and they don't have


a good nice right there.


They have one here.


All right here if you just look at kind of the top sentence here the man went to mask


store.


Right.


Don't don't don't worry about the set and so on.


Just this man went to mask store and the model simply asked to predict what's here which


word is there.


So it needs to incorporate information from the right and from the left to do this.


So that's basically how you train it.


They simply drop out some of the words some of the time and they have different techniques.


So you can clearly tell a lot of work has gone into kind of fine tuning everything in


this model like how to train it and so on.


Let's say we don't always do this.


Sometimes we do this other thing and sometimes we do that.


There are several ways of biasing this model but basically you do this masked language


modeling and then because they also want to evaluate on let's say entire sequence tasks


or tasks that span multiple sentences what they do is the second pre training task at


the same time as you can see here where they feed two sentences.


So that's the first sentence.


That's the second sentence.


They feed these two sentences as an input.


So at first they have this token and these separate the sentences and then they ask the


model to predict a label is next and is next is true if the second sentence follows the


first sentence.


If it's like a logical continuation and the way you do this unsupervised is really easy.


You take a big giant corpus and you take a sentence for the first sentence and then 50


percent of the time you take the next sentence in the corpus and the label is true and 50


percent of the time you take some random sentence.


Here you say for example the man mask to the store and the next sentence is penguin mask


or flightless birds and that's kind of a random sentence.


So the model is asked to predict.


Well that's probably not the next sentence following this first sentence.


So you do these two tasks.


You pre-train and you can do this unsupervised you don't need supervised data for that.


You just need a corpus and they do this for a long time with a lot of data and the model


itself is giant has 24 I think of these transformer layers so it's giant and then you kind of


pre-train this model.


Here is a here's an illustration of some extra things.


So what they do is they they first this is the input up here.


So the first token is this CLS token which is kind of the start token and then this is


the first sentence then the set is the separator of two sentences.


And this is the second sentence.


And then again we'll get to these hashtags in a second.


But first they say OK first we have the token embedding so they kind of start with with


the original concept of of word vectors at the very basis because you need to start with


actually going into a vector space to use these models.


But they don't they then they then kind of transform these through the transform layers.


They also use segment embeddings and segment embeddings as you can see here is simply kind


of a binary label.


EA being the label for the first sentence and EB being the label for the second sentence.


So just the model can differentiate which one's the first and which one's the second


because it's kind of hard to learn for a transformer architecture that the set tokens kind of separate


the sentences so you kind of want to help it.


And the last thing is positional embeddings and we've already talked about these in Attention


is All You Need.


This is where you can kind of the model since it's a transformer it doesn't go step by step


it doesn't go one done done done done.


So it's kind of hard for the model to make out how far things are apart from each other.


How far two tokens if they're neighbors or if they're really far apart.


And these positional embeddings kind of help the model decide if two tokens are close to


each other in input if they're right if they're just neighbors or if they are actually really


far apart.


So this is this is how the kind of first input is constructed out of these embeddings and


then it's fed through these transformer layers as we saw with the mask the lem task and the


is next task.


I want to quickly get to these hashtags what what they mean.


So the input here is separated into word pieces so-called word pieces.


And what that is is so in language processing tasks you have kind of a choice you have you


have a choice of how to tokenize your input.


So what let's look at a sentence here subscribe to PewDiePie.


So this is a sentence and the sentence is rather let's say word wise complicated.


So why why might a language model have problem with this.


So first you need to tokenize this sentence.


All right.


So what most people do is they say OK here are the word boundaries we're going to tokenize


this into three segments first is subscribe to PewDiePie.


OK so three things and each of these now needs a word vector associated with it.


Now to the thing is the word vectors let's assume you have them pre-trained or something.


In any case you need a big table a big big table and this goes down here where for each


word a the to I you you have a vector associated with it.


So you need to keep this in your model.


And as you can as you know English has a lot of words here.


So this table is going to be really big and the problem is how do you make this table.


Right.


OK you could make it kind of dynamically and so on but in general you're going to create


this table with all the words you know and that's going to be too big because English


has so many words and then you can say all right we'll only take the top whatever is


used in 90 percent of the language which turns out to be this kind of burrito distributed


so it turns out to be like 5 percent of the words are used in 90 percent of the language.


So you just take these but then you're going to have the problem.


OK here to two is not a problem.


Why not to is used super often.


We're going to have it at the very top somewhere and we're going to have a vector.


Type is it's already it's not so common.


So maybe you have a word for it somewhere down but then PewDiePie is a name.


So there is no there's not even a word like that's not even a word.


It's just so what you what you usually do what people usually do is they have this out


of vocabulary token and then they have a vector associated somewhere here with out of vocabulary


tokens.


Whatever and I don't know what it is I just know that I don't have it in my vocabulary


and the model kind of deals with that.


That's kind of it's not it's not really ideal especially if you don't want to generate language.


Also your model tends to generate out of vocabulary tokens if you allow that if you don't allow


that you have a problem during training.


So it's all kind of messy.


What's the alternative.


The alternative is to go character level.


So let's look at character level in character level.


You say all right my words are obviously made of characters and characters.


I'm just going to split at each character right here the white space can be a character


too.


So I'm going to split at each character and then I'm simply going to have one vector for


each character and there's only like 20 something six of those and I'm so I can keep 26 vectors.


But this tends to be rather problematic because a character by itself having a meaning that


that can be encapsulated by a vector is kind of it's kind of shady because the character


by itself usually doesn't mean it doesn't have a meaning.


So what's the solution here.


The solution is to go in between.


The solution is to say well let's actually go for word pieces and you can kind of think


of them as syllables but you can you can split you can make them in a way that you have a


fixed size vocabulary say OK I have 4000 entry places in my big table.


I can afford 4000 size table.


So first of all I'm going to have for each character A B C D E and so on.


I'm going to have a vector but then I only have 26 I have 3000 some left.


I'm going to have also the most common words.


Now A is already here but maybe I can have to and from.


And so the most common words they also get there.


And then for the other things I'm going to split the words maybe in sub scribe.


Right so these are two syllables and sub can be pre kind of a prefix to many things and


I only need then one one.


So I've sub here sub I only need one vector for that and then the rest if scribe scribes


by the way also a word so I can have that.


But if scribe weren't in my vocabulary I can divide scribe then up into into characters


and then describe them with the character level.


So basically I can mix and match here I can sub that's that I have that and then scribe


I don't have it.


I don't have any of the pieces but so I can just use the character the character.


So this would would be sub and then S C R I B E.


So these these would be the tokens that I work with now as as my input and this these


tags here so so this is what would happen to PewDiePie.


You could simply split along each character so you basically this kind of an interpolation


between the token model and the character model and it's really neat and usually works


quite well.


The as I said the the hashtag sign here simply means that these two have originally been


one word and now this this in here is just a word piece token.


This is a really good example where word piece come in because play by itself is a word and


I can make play in instead of having an own vector for that I can divide it into play


which already has a meaning and presumably playing and play would have similar meaning


so it makes sense to have play as the token singled out here and then ing as a suffix


also makes sense to have a token for that in my table and then I simply have these two


tokens here and that probably already gives me more information than simply having the


word playing.


By the way you should subscribe to PewDiePie.


Just FYI.


All right let's go on.


So we do word piece tokenization we do the masked language model we do the next sentence


prediction pre-training.


What do we have now?


We have a model that can really really well predict some masked words.


Now how do we use it?


Now they evaluate on these I believe it's 11 tasks 11 different tasks of or is it I


don't know how many it is it is a lot with the same model so this pre-trained model they


now claim can be fine-tuned to do all of these tasks and it gets up it's like state-of-the-art


on everyone it's crazy.


So how do they fine-tune it?


So the easiest tasks are the one are the so-called sequence level task where you basically have


the sequence and you're you're about to predict one class label for the entire sequence so


here we have the sentence pair classification tasks.


For example the task we saw before the is next task but there is more sophisticated


tasks that you need kind of supervised data for and so with the supervised data you'd


have a class label that you could train on.


So what do you do is let's look at one of them MLLI they had it up here nope here multi-genre


natural language inference crowdsourced entailment classification task so given a pair of sentences


the goal is to predict whether the second sentence is an entailment contradiction or


neutral with respect to the first one.


Alright two sentences and you're about to predict which one of these three labels it


is so you put the two sentences here BERT can already take two sentences as an input


as we saw the embeddings are the A and B embeddings and the position embeddings are left out


of the picture here but they would be added to it and these would be the embeddings for


it and then you pass this through the BERT model and this is the final layer and what


they do is they simply take now the embedding the final embedding for this first one corresponding


to this start token and they simply put a single layer of classification so basically


a logistic regression on it and that's how they then get a class label so if this is


whatever let's say this is this this gives you here a hidden vector of 512 dimensions


512 and you have three labels to output here one two three you simply need a matrix that's


512 by 3 of size and these are the these are the weights that you would then have to train


in addition to BERT so BERT is pre-trained and you have to simply only now learn these


weights of course they also kind of fine-tune the entire BERT model but that's really fine-tuning


the only thing you have to learn from scratch is is this these weights here that's pretty


first of all it's pretty neat because you can be very quick at learning new tasks because


you simply start from the pre-trained BERT and then you go and learn a single classifier


layer on top and astonishingly this works extremely well for these tasks a bit of a


bit of a more challenging task is this here SQUAD is a question answering task and we're


going to jump down here where they explain the task so you have an input question oops


you have an input question and the input question is where do water droplets collide with ice


crystals to form precipitation and you have an input paragraph which is kind of a paragraph


from Wikipedia page and you know that the answer is somewhere in this paragraph right


the data set is constructed such that the answer is in the paragraph so the in paragraph


reads precipitation forms as small as smaller droplets coalesce via collision with other


raindrops or ice crystals within a cloud so your the question is where do water droplets


collide to form precipitation the answer here is within a cloud so that's this this thing


here so usually what SQUAD models do is they predict a span they predict where's the start


of the answer and where's the end of the answer that's also what kind of BERT's trained to


do so in order to do this what you do is again you already have the ability to input two


sequences so we've trained with two sentences but here they simply say oh well the first


sequence is going to be the question our second sequence is going to be the entire paragraph


from Wikipedia and then for each output for each output for the output of each token remember


there's as many outputs as there's inputs because the transformer will always transform


to the same length of sequence for each token in the output we classify it is this token


the start token or is this token the end token or is this token none of all now what they


do effectively is that here each each one outputs each one is a vector and they as we


said at the beginning of finding out which one's the subject now here we have two queries


namely query one which is is this the start let's call it query s and query e is is this


the end token so these are two queries and I'm going to just produce compute the inner


product of each query with each of these outputs right and over my sequence here this is going


to give me a distribution so start for start maybe this token is not much than this token


is a lot and so on there's five tokens and for the end not so much not so probable not


so probable very probable not so probable so what you get gonna get is from these inner


products is a distribution over which ones to start and which ones to end and you're


gonna say okay this one's probably the start and this one's probably the end so that's


how you predict the span and again what you have to ultimately learn is these these queries


here and so not that much and this is named entity recognition and named entity recognition


you have a sentence and you're supposed to recognize named entities like up here we saw


subscribe to PewDiePie and the named entity would be PewDiePie right this is a name and


you're supposed to recognize that this is a name and they do it the same same way that


they do the squat basically or a similar way sorry they basically for each of the outputs


here they simply classify whether or not is it's part of an entity or not so what they


have to do is they have to simply train if they you also have different labels for which


kind of entity is this this is like a person and this is this is no entity so if you have


10 of the labels then each for each thing you would classify it into one of 10 classes


you need a classifier of input size versus number of classes that's all you have to train


in addition to pre to fine-tuning BERT itself all right so they kind of evaluate on all


of these tasks they get super duper numbers on all of them here BERT large wins on pretty


much everything and this model is big just saying and they trained it on TPUs which is


available in kind of Google Cloud infrastructure so far and it trained it on a lot of data


so to a way it's kind of expected that you would outperform but it's very surprising


that you outperform everyone else by this much and they've done a lot of kind of ablation


studies where they show that it's really due to the fact that they do this left and right


context they take into account the left and right context of a given token when doing


the attention that it's that that's why it's better so here for example they compare the


BERT base model and they say okay what if we don't do the NSP the next sentence prediction


task then you can see the numbers they already kind of they drop on these tasks and what


if we then additionally do only left to right training and the numbers they drop pretty


seriously again you see sometimes here for example you see a pretty serious drop in the


number also here so there really seems to be real value in doing this kind of left and


right context attention so it's not just about the model size and the amount of data that's


basically what they show here and it's really cool that the paper actually shows this because


usually people have an idea and they throw a lot more resources at it and they're better


you'd never know why and this is pretty cool that they actually show all right so this


is all I have to say about this paper but check it out the models are here pre-trained


you can actually download them you can fine tune in for yourself for your own task and


they're pretty pretty powerful there are smaller models for if you don't have a TPU that are


Creating Stable Diffusion Interpolation Videos

Welcome everybody to a quick video in a new series called


This is now a thing you can do


probably something like that


on making interpolation videos with stable diffusion


like the one playing right now


So stable diffusion is put super simply


a text to image generation AI


which runs on only a few gigs of memory


and which all by itself is absolutely insane


given the vast number and types of imagery that it can create


effectively having compressed countless terabytes of imagery down to just a few gigs


That's incredible, but maybe more on that later


For now, we can make images of just about whatever we want


but from this model's inherent capabilities and understanding of language and text


and mapping that to pictures


one of the more interesting things that we can do is interpolate between two or more images


To do this, I'll be using a package from Nate Raw on GitHub


which makes this whole process a breeze to do


This package is also in active development


and I am sure many things will be changing and improving over time


so if there is, especially with something in like my workflow


my guess is things will change and will be added to that package


and you won't have to do it exactly as arduously as I have


So first of all, we need two or more images to start with to shift between


My personal workflow has been to come up with a prompt


and then generate 50 to 100 or a few hundred images with randomly selected seeds


and then manually review the images and pick the ones that I think would be cool to use


I'll put a gist or something like that in the description so you can copy me


but at some point, I think in the near future


I really expect this package from Nate Raw to contain a lot of this workflow for you


so you don't have to be doing this like separately


So for images that you go with, you can do whatever you want


For me, I think things that work most obviously are anything that like grows or changes over time


and then stitch those things together


So things like cities or maybe like a story or an evolution of something over time


that sort of thing I think makes a whole lot of sense


One of the things I really wanted to do was like a music video


and it's funny because I was talking with Nate Raw


and that was one of the things he even suggested


but it's like I like my YouTube channel so I can't do that


but I think that would be a really cool application


I just don't think, I just can't do it


So anyways, instead I decided to just go with an example of a Porsche 911 model over the years


So from 1965 to current


but I encourage you to try all kinds of crazy things


because there's so many like unique and fascinating things that you can do


because you don't, it doesn't have to be a thing that grows over time


or it doesn't even have to be two things that are even related


You could interpolate between a car and a dog if you wanted


and that might be pretty cool to see


So I encourage you guys to really kind of tinker around and try to be creative with it


Like I said, all this stuff runs on very, like you don't need that beefy of a GPU


like 10 gigs ish is all you need


So definitely, and you can even get by with less than that


So anyway, first I have a script that runs through a list of prompts


Then for each prompt it iterates through and in this case 200 examples


each with a randomly selected seed


From there, each image is generated with StableDiffusion


and it's saved in a directory with the name of the prompt


and then the name of the actual image in that directory is the seed that was used to generate that image


Then I just look through the images and either select the ones that I want


or I come up with another prompt or try some more seeds


until I have a good sequence of images that I want to make a video out of


From here, we can use this StableDiffusion videos package


to go through and interpolate between all of these making pretty cool videos


One of the more recent modifications to this package was the addition of an upscaler


which you can use by just adding upscale equals true in your walk command


and then also installing the real ESRGAN package


This is just a general upscaler


so it may not work precisely for the task or the type of imagery that you're attempting to use it for


but it's there if you want it


and it will get you from 512x512 imagery to 2048x2048


So go forth and create


Also, right now I have a CEO of NVIDIA, Jensen, signed 3090 TI to be given away


All you need to do to be eligible is to sign up for free for GTC,


the all digital NVIDIA GPU and AI oriented conference


and all you need to do is sign up using the link in the description


and you just have to attend one or more sessions


The 3090 TI is an absolute beast of a card


so congratulations in advance for whoever gets it


I am jealous of you


and the sign box is a just cool piece of history, honestly


Anyway, that's all for now


Hopefully you guys have enjoyed the shorter-ish format of videos


There are so many like little projects that are kind of coming out


and things that you can do with AI like stable diffusion


It's not just stable diffusion stuff


There's other stuff as well


I don't really think it would be great to just like throw them all in like one giant video


and then each one is not really big enough for like a full feature length


a full 40 minutes syntax video


So let me know what you think about the shorter format


Also, if you make your own stable diffusion interpolation videos


please put them down in the comments


I'd be really interested to see what other people are coming up with


with this sort of like technology


There's so much room for creativity here


and I just I'm super curious to see what other people come up with


So anyway, that's all for now


Thanks for watching and I will see you all in another video


∞-former: Infinite Memory Transformer (aka Infty-Former / Infinity-Former, Research Paper Explained)

Hello there, today we'll look at Infinityformer infinite memory transformer by Pedro Enrique


Martins, Zita Marino and Andre F. T. Martins. On a high level, this paper proposes a transformer


that can attend to unbounded memory in the past. It does so by building up what he calls a long


term memory, which is a continuous signal rather than a discrete signal as most of the other


transformers do. It uses continuous attention to do so and that enables it essentially to


continuously compress the past into this continuous long term memory and then attend to


it as it predicts next tokens. It also introduces the concept of sticky memories, which essentially


are events in the past that are of particular importance to the future. So by keeping those


sticky memories specifically around, they increase performance yet again. So we'll go through the


paper, what the model looks like, how it works, and what it does in the experimental results.


Ha caught you, you wouldn't have guessed it, but this video is sponsored by Weights and Biases.


If you're in the ML space and you don't know about Weights and Biases, what are you doing?


Please, if you track your experiments using a spreadsheet, a piece of paper,


TensorBoard, weird folder names, like I used to do, stop that. Use Weights and Biases. It's one


line of code and you can log any of your experiments to the cloud, not just metrics, but


models, datasets, output images, little videos, anything you want. Say hello to Zyrk.


Believe me, when I started the PhD, I was looking for something like Weights and Biases and I tried


every single thing there is. I tried every productivity tool, every note taking tool,


and I just couldn't get anything to work for one part because the features were just lacking,


for the other part because I was just too lazy. And Weights and Biases solves both of those


problems. It has all the things that I need to track my experiments, collaborate with others,


and so on. But also it's just a single line of code and everything else works automatically.


It even boosts my productivity because whenever I have logged a model, I can just call a function


to download that model from the Weights and Biases website. I don't need to place it in a correct


folder or keep track of it myself. It's just there. On top of that, it relieves me from the


stress of writing stupid Overleaf reports because I can write a Weights and Biases report and share


that with the people that I want to show my work to. The Weights and Biases report is so much more


useful than a PDF. It's essentially a website, but you don't need to code any HTML or CSS or whatnot.


You can include dynamic content, you can reference the runs you did, you can pull out data from the


runs, you can present that in a neat fashion. And it gets even more easy, you don't even need to...


And it gets even more simple, you don't need to even set up anything. In fact, Weights and


Biases runs in the cloud by default, you can host it on premise, but it really wants to live in the


cloud. All you have is an API key, you log in, and you're good to go. So please check it out.


Accounts are completely free for personal use. I promise you will not be disappointed. Give it


a try. And now let's get into the video. Bye bye. Cool. So there are a couple of good things and a


couple of questionable things about this paper. Also, there are a lot of engineering choices in


this paper, which I don't necessarily want to go into. There are a lot of things that one could do


differently, I feel, which in influences the experimental results as well, I guess, but we'll


just take it for what it is. The other thing is that I believe this should be called not infinity


former, but inf t former, that's actually how you find it on. If you Google for this, you have you


can enter inf t former, inf t being of course, the abbreviation in LaTeX for this symbol right here.


And I think, you know, to make it more unique, we should just call this the inf t former.


Alright, so what does the inf t former propose, they say in the abstract right here that


transformers struggle when attending to long context, since the amount of computation grows


with the context length, and therefore cannot model long term memories effectively. So there


are a number of things written hidden right here, they say the amount of computation grows with the


context length. Now for classic transformers, it's actually worse, right, the amount of computation


grows quadratically with the context length. But even for some of these, let's say linear


transformers, the amount of computation still grows linearly with the context length. So they


they see even this as a problem, they say they cannot model long term memories effectively.


Now, they say several variations have been proposed to alleviate this problem, but they


all have a finite memory capacity, being forced to drop old information. In this paper, we propose


the inf t former, which extends the vanilla transformer with an unbounded long term memory.


By making use of a continuous space attention mechanism to attend over the long term memory,


the inf t formers attention complexity becomes independent of the context length. Now already


remember right here, there is rarely a free lunch, I don't want to say there is no free


lunch, because I've definitely eaten free lunches before. But there is rarely a free


lunch in these kinds of things. If we have a finite computation, we cannot pack infinite


information in there. So if we are attending to unbounded long term memory, that means something


else will have to give. And of course, the thing that gives here is just the amount of information


you can retain. Now this can be a good thing to trade off sort of boundedness in time for boundedness


in information. Yet still, you have to keep that in mind. As I said, they also introduced this thing


called sticky memories that keep important things around. Now, as we go through this, this gets it


in my mind, at least this gets more and more into just like a classic LSTM model. So the classic


LSTM model, of course, takes in some sort of a, a input, then models a hidden state, then propagates


that hidden state when it inputs the next thing, and so on. And it sort of has to keep track of


what's important in its own hidden state, as to decide what it wants to remember what it doesn't


want to remember. So as with the transformer, the LSTM has in fact an unbounded memory, right, it


can remember things for arbitrarily long, yet it only has finite capacity to do so it needs to


overwrite some memory every now and then. So this is a bit how you can think of this model is


essentially the same principle as an LSTM trading off unboundedness for finite representation space.


I'm not saying this is an LSTM, it is a little bit different, it might be a smarter way to do


unbounded computation, it might not be, but in concept, it is the same the similar thing. Okay,


so what's up with this continuous attention that they keep talking about?


This is, in essence, quite a simple concept. Namely, if you have a sequence of let's say tokens,


right, and every token has an embedding vector, so every token is associated with a vector that


is its embedding. And this can be the first layer, but this can be also the intermediate,


the intermediate values of the computation. So from one layer to the next, you always in the


transformer have number of tokens of these embedding vectors that travel through the model,


they get transformed into by the next layer into new embedding vectors, and so on, and so on.


Now, the NFT former, what it does is it takes this signal right here, and, and changes that


from a discrete signal into a continuous signal. So you would no longer have dimensions that,


you know, the first, the topmost dimension here, the first dimension of all these vectors might be


whatever 459.13. That's no longer the case, what you would have is like a continuous signal.


Okay, now, how do you do that pretty easily? What the NFT former does is it takes each of


these dimensions separately, okay, each of these dimensions, it plots these points up


on a sort of continuous plane. So this, this here, so this, it labels it from zero to one. So you


divide this interval into, I guess, five different points, because we have five tokens. For the first


one you label, sorry about that, you label with a four, where is a four? I suck at this. So here is


a four, so dot here, then here is a five, I guess, so dot here, 9.1 and three, like here, okay, so


here's three. Cool. And then what it does is it calculates an interpolation. So the interpolation


would be this, approximately, right? So it calculates an interpolation of these points,


and then it simply stores that interpolation, it forgets about the embedding vectors themselves,


and it simply stores that signal. And that is it's so called long term memory, simply this signal.


Now, you might wonder, why don't we just store the embedding vectors, right? Instead of the signal?


And that is, of course, a good question. The goal is, of course, that you can store the signal more


efficiently than the embedding vectors. So if we can describe the signal here with less than five


numbers, then we might be able to then we might be able to save some space, right? Like what,


like this is reasonable, this could be a polynomial of degree three, right? If, for example,


like, if I draw this, you know, this is reasonably a polynomial of degree three, ergo, we'd have to


store like three numbers, maybe plus a bias of four. But if we agree that we always store


polynomials of degree three, then no matter how many embedding vectors we have, we're always going


to store the signal as three numbers or four numbers, right as a constant amount of numbers.


And that is essentially the trick right here on how we get away from the sequence length,


we simply commit to a representation, a fixed representation of a signal, and then we interpolate


the embedding vectors using this fixed representation. Now the fixed representation


here isn't a degree polynomial, but it is in fact, a series of radial basis functions.


So we associate each point in time, which is the the here the one the two, like the the interval


from zero to one, we index this into a radial basis function. And radial basis functions are


nothing more than so this is one, this is one, this is one, okay, so these are these are three


essentially, these are three radial basis function spaced out right here. And how could we represent


the signal from up here? Using that maybe we can say okay, that's plus, you know, if here is one,


like that's plus 4.5 of that of of, let's call that psi one, then minus, you know, it goes down


like minus three of psi two, and then it goes up again, like plus four of psi three, maybe some


sort of a bias plus two, okay, so four numbers, three radial basis functions. Alright, so these


things here are completely independent of the data, they're not learned, they're simply fixed


once, like this is going to be the our basis for representing all of the signals. And then the way


we transform the discrete signal into the continuous one is we run a regression. So the


regression you can run by solving this system right here, by figuring out what is the matrix B


here. And that's a linear system. What is the matrix B? How do I have to mix the radial basis


functions here in order to match my signal as closely as possible? The way they do it is they


run a ridge regression, ridge regression is simply a regression with an L2 penalty, I think,


is that the case? Yes, I think so. So you run y is equal to x times w. So you're trying to find w


x times w, you're trying to find that so your loss is going to be the distance of these things


squared, and then you have some sort of a regularization constant. And on the L2 norm of


the weights. So you solve this, there's a closed form solution. This is the closed form solution


for ridge regression with f being the matrix containing these basis vectors, this one right here.


And there you get your B matrix. So you transform x, which is dependent on the length of your sequence,


right into B, which is only of the length of how many basis vectors you decide to have, in this


case, three, or three plus one, if we want to buy us again. All right, so and that's how you have a


continuous signal, you might already hear, you might already say, wait, isn't this just a special


case of a system that simply compresses a sequence into a variable length sequence into a fixed length


sequence? Like, isn't this just a way to embed, like a continuous, like an unbounded sequence?


And I'd say yes, absolutely. That's the first thing. The second thing is, is certainly


the whole procedure is certainly not independent of length, as this system right here is absolutely


dependent on the length of your signal. And you can also see that the longer your sequence gets,


the more mistakes you'll actually make in representing it, because you only represented


using the same basis vector. So here is where the trade offs happen by going from length L2 length,


I believe they call it n, the length here of the number of basis vectors is n. So that's the first


thing, here's where the trade off happens. The second thing, which really kind of interests me,


and here you see this again, right? So by the way, this, then they consider their their memory,


right? So you can technically do this with all of the past, right? You take all of the past,


you remember the vectors right here, and then you interpolate. Or what you can do is you can,


what they call, you know, if you really go to unbounded memory, you take the past, you take


the current sequence, you can do what you can do is you can contract the past, which means you can


interpolate the interpolation. So you can sample it in a more coarse grained fashion at the


handy, you can sample it in a more coarse grained fashion than you originally produced it, which


leads to samples like here, and then you concatenate with the new signal. And then you


simply interpolate again into the whole signal. So you can see the more distant past is now


compressed to that. And the more recent past is appended to that. And of course, in the next step,


you'll contract this whole thing to a shorter sequence and append the more recent thing right


here and interpolate again, how this is conceptually no different from an LSTM. It brings


about the same problems as an LSTM, namely more recent things are more likely to be in memory than


way past things and so on. So calling this, you know, being able to attend to all of the past


things, this, you know, being able to attend to unbounded, unbounded memory and so on, is a like,


it's a bit shady. Like that just, that's just my opinion, you have to be aware of the trade offs.


Second of all, second is the fact that in order for this to work, right, and we haven't even


done the extension part yet, we're just representing our signal as a continuous signal. In order for


this to work, you're counting on the fact that there is some kind of a regularity, right? Here,


I've drawn these points specifically such that I could draw a neat line through them. Yet there is


absolutely no reason why the embeddings of the continuous, you know, next to each other tokens


anyway, continuous such that you can interpolate it, right? You count on the fact that you can


compress the signal, because the signal like the samples go like, right, then you're like,


whoa, I can, I can represent this by one line, right? One radial basis function goes through all


of them. Cool. But there is no reason why this should be like the signal could be like,


tuk tuk tuk tuk, like completely, completely random in terms of what the real floating point


numbers are in the individual dimensions. Yeah, they mitigate this a little bit by smoothing the


signal first before they, before they interpolate it. But in my mind, that kind of only makes it


less accurate. It doesn't make the problem go away, it just makes it sort of less accurate, because


if there is an actual value to having a pattern like this, if that's actually an important,


an important pattern, then neither interpolating it very coarsely with only few basis functions,


nor first smoothing it will, will necessarily help. So, you know, I just from a principled


standpoint, I am skeptical that this is the case that signals that these signals here are


necessarily such that they are easily interpolatable. But of course, I might be wrong. So,


you know, that's it, I might be wrong, right? Okay. So what do we do with it? Alright, let's say


we have the past in this long term memory, right? This is all of the past, we've interpolated it


into this fixed long term memory, this continuous signal that we represent as a superposition


of a fixed set of basis functions, we have our short term memory here, which is simply whatever


we would put anyway, into the context of the transformer, right? And then we have our sequence


that we actually want to deal with. So the attention within the discrete part of the


transformer is as you know it, this is self attention, a training, I guess, masked self attention


for certain tasks. This is as you know it, the question is, how do we make use of this long term


memory right here? And here is how we do it. So for each location in where we want some sort of


a prediction, right, we produce a query, as you know, if in a transformer layer, every single


token produces to go from one layer to the next produces a query vector, the query vectors tell


what this token wants to know about the sequence in the last layer. Now every token also emits a


key and a value vector, so key and value, key and value, and so on. I'm only drawing the keys and


then this is routed by inner product. Now the query, of course, we can keep the query simply


tells what does this token wants to know. So the query is also taken to go to the long term memory,


right? So the query vector of each discrete token now goes to the long term memory down here.


And we'd have to find a way to ask the long term memory something according to this query. So how


do we do it? What we need is we need some sort of a notion of a key and a value for this long term


memory. And here's how we compute it. Remember, we have, it's not the continuous signal is described


continuous signal is described by this matrix B right here. So if the continuous signal is described


by the matrix B, then of course, we can compute keys and values from B, these W matrices right


here are learned parameters that take B and make it into keys and values. Now, the keys and the


values are of different length, they are sequences, they're discrete sequences, right, they're of


different length than the length of the sequence we're dealing with. But that doesn't matter.


Nothing in a transformer actually specifies that the next layer always have to has to have the same


length of sequence. So what you can imagine, the way you can imagine this is from the long term


memory, essentially, what we're doing is we're building another sequence, it's not as long


as the sequence that generated the long term memory. But essentially, we're building another


sequence of tokens, they are, you know, not necessarily corresponding to individual tokens


in the input, they're corresponding to how the thing is constructed. But nevertheless, and from


those, we can certainly generate keys and values as we do regularly. Okay. So we essentially compress


the past into this pseudo sequence of fixed length via a continuous representation. And then we just


use attention again, to map the keys here with the queries. Now, when it comes to actually


computing the thing, it's not, it's not as easy. So this is in concept. But when it comes to actually


computing the thing, what we want to do is we don't want to really abstract this into series,


we would like to use continuous attention. So continuous attention essentially means that


our attention doesn't go directly to one particular token. So it's not like, we know this


token and this token and this token. But since we have a continuous signal, our attention should be


something more like, well, I want to attend to this part of the sequence. And we model that as a


probability density over the sequence, specifically, we restrict ourselves to a Gaussian. So what I can


say is I can my query, the interactions between the queries and the keys will give me a Gaussian,


where I say I would like to attend to this particular part of the sequence, right, this is


where in the past I want to attend. And this is how broadly, let's say I want to attend,


you know, how many, how much of the surrounding I want to consider. So this, this ultimately defines


a Gaussian, like where it is, and how, how far the Gaussian is spread. Right, so I can attend to per


per query, per token per head, I can attend to one location in the past, and its surrounding


and the width I can also specify. And this is also learned. So as I understand it, these fine


transformations right here are also learned transformations, maybe I'm wrong in that it


just says a fine. But yeah, and then the sigmoid and the soft plus are just regular functions. But


you can see right here, this is essentially, as you're used to multiplying keys and queries.


But then instead of attending to the tokens themselves, because we don't have tokens, right,


we, we specify a Gaussian to attend over the continuous signal. And ultimately, we can


integrate, essentially, we can integrate the two things. So we can integrate the values that we


obtain from the from the sequence, this these values, we integrate them according to the


probability distribution that we get. And that's going to be our output values. So these here are


going to be our output values. Now, once we have the output values from the long term memory,


we add them to the output values that we get from the short term memory and the sequence itself,


add them together, I think they go through another affine transformation after that.


And there is your output. And the output is going to be one output per token in the sequence that


you're interested in. Okay, so I know this was fairly lengthy. But to recap, we take the past,


we do, we do a regression, a ridge regression in order to determine the coefficients to represent


the past as a continuous signal with respect to a fixed set of radial basis functions. This gives us


a fixed size representation, independent of how long the past is, then the way we use the past is,


we take the queries that come from the attention mechanism, we transform the representation of the


past, which is this B matrix right here, into keys and values, we take the inner product between the


queries and the keys. And this determines a Gaussian window for us where in the past we want


to attend to, we integrate the values from that region according to the Gaussian. And that's going


to be our output signal from the long term memory. This gets added to the output signal of the regular


attention mechanism. And that gives us the output signal as a whole. Okay, this is essentially,


essentially it. And if we do this one after another, right, we could simply always go to the


past and compress it. But we can also do this trick that I mentioned before, this unbounded memory


trick, where you always take the signal from the past, you compress it essentially by subsampling


it, you concatenate the new signal, and then you interpolate again. And on top of this, they


introduce these sticky memories. And the sticky memories simply say, look here, the points that I


have sampled, the points that I have sampled this past signal on here, I simply, well, don't believe


my drawing, but I simply did that uniformly, I sampled this uniformly, that kind of gives me a


good sampling of the of the signal, right, I can also sample this differently, I can oversample


certain regions and under sample certain regions. So here they say, why don't we oversample


according, why don't we sample according to these Gaussians that we've determined during the


attention mechanism. So the Gaussians, of course, are summed up over all the attention heads, and


over all the sequences in, so we're sorry, all over all the tokens in the current sequence that


you're looking at, because all of these things attend to the same past. If we sum up all these