Each number now becomes a variable that each step of the way, these numbers are being assigned to this variable each number.


Python 3 Programming Tutorial - For loop

Obviously example list is corresponding to example list.


We'll hit enter.


If you didn't notice already by now with the while loop and all that, when you make one of these loops and you hit enter, it automatically tabs it over for you.


If you're coming from maybe another language or not at all from any programming language, you have to usually do this.


Actually you don't have to really do it in Python.


You have to do it.


You must have this space here.


In other languages, typically, you've got curly braces.


You come down and you could put all you wanted over here.


Then most people say you should have good standards and you should automatically indentate yourself.


It is not necessary in other languages.


In Python it is, but luckily, usually, it's going to automatically indent for you.


It is just fine.


It makes your code pretty automatically.


That's kind of nice.


Anyway, you must have that indentation.


This indentation will apply throughout the entire loop.


In order to get out of that loop, I'll show you in a moment what you would need to do.


For now, let's just say print each number.


We'll save and run that.


As you can see, it just iterated through the entire list in order.


1, 5, 6, 1, 6, 7, 8, 9, 345, 535.


Pretty cool.


Now what you can do too is then you come down here and we'll just say print continue program.


Like that.


Run it.


You'll see here that it continues the program after the for loop because it's not under the same indentation.


But if you hit tab and you indent this along with this for loop, you're going to see continue program for every number.


When we run this, you can see that at each step of the way it continued the program.


Just keep that in mind that that's how your indentation really matters.


Another quick thing I'll show you guys real fast is first what can happen sometimes is this.


You've copied and pasted a lot of code maybe and it's not indented.


So what you can do though is you can highlight all the code like what's called a block of code and you can highlight it and then hit tab.


And it will tab the code over.


By that same token, if your code is too far tabbed over, that also will not work.


It literally won't run.


Well, Python 3 just keeps throwing me for a loop.


Apparently that's okay.


It's going to go ahead and run.


Hold on.


Let me try something else.


I'll come down here, paste, and then let's do this.


All right, Python 3.


That was okay as well.


Well, at least in Python 2.7, you can't do that.


But anyway, I'll tell you anyways how you can fix this just so it does look right.


Because even though apparently Python 3 is more accepting of improper standards than Python 2.7, you should still try your best to give good standards.


Because it gets very messy when you want to do maybe another for loop down here and do some stuff, right?


So anyways, what you can do to move a block the other way is highlight it, hold control, and it's the open bracket.


So you can hit that a couple times and it tabs it over.


If you don't know what I'm talking about, it's this symbol right here, like this key.


You hit that while holding control and that will shift it over.


So that's that.


But now back to some more stuff about the for loop.


I was telling you guys that you can use the for loop much like you use while loop as a counter.


And it's much better as a counter.


And here's why.


So if you remember the for loop that we wrote, or the while loop rather, I believe that came out to be four total lines of code.


But instead with the for loop, you can do 4x in range.


And range is a built-in function in Python.


So first I'll just show you guys what it's going to do.


1, 11.


And all we want to do is just print x for now.


So let's run that real quick.


And you'll see that what it did is obviously our previous program.


But then it did 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, not 11.


And the way range works is it starts at the first number and goes all the way up to the ending number or whatever it is.


Now, a couple of, or one thing to note, range is a built-in Python function.


So you don't have to import it.


And it will literally make that range of numbers for you.


In Python 2.7, Python's 3.0, or Python 3.3's version of range here has actually been backported to Python 2.7.


And it is xrange.


It's known as xrange in Python 2.7, so xrange like that.


And what is beneficial about this range function over the Python 2.7's range function is when you do something like,


in Python 3.3, and you say range 1 and like this number, what it's going to do in Python 2.7 is literally generate a list that is this,


from 1 to this number, whatever it is.


And obviously that's going to use a lot of your virtual memory, so your RAM.


And it's probably going to max out your RAM before it completes the operation.


But in Python 3.3, or if you use what's called xrange in Python 2.7, it turns this into what's known as a generator.


And eventually we'll talk more about generators later on.


But what a generator does is it only visits that operation once.


It doesn't store it into memory, but it will count through.


So pretty interesting stuff there.


But more on that kind of stuff later since this is meant to just be basics.


So that pretty much covers the for loop.


A couple of examples of the for loop here, both as a counter and then also as an iterator through a list.


But really this iterated through a list only actually used a generator function to do so.


So anyways, that is that.


So hopefully you guys learned something new.


Hopefully you guys are enjoying my tutorials.


If you have any questions, feel free to post them below.


Otherwise, as always, thanks for watching.


Thanks for all the support and the subscriptions.


Evaluating NLP Models via Contrast Sets

Hi there, today we're looking at evaluating NLP models via contrast sets. And these are too many


Evaluating NLP Models via Contrast Sets

authors from too many places for me to read out. So we'll just jump right into the problem. So what


Evaluating NLP Models via Contrast Sets

is the problem? Or let's jump into the solution. Here you see a visual question answering task,


Evaluating NLP Models via Contrast Sets

visual question answering. In this case, you have two pictures right here, picture one,


Evaluating NLP Models via Contrast Sets

picture two, and a sentence. Two similarly colored and similarly posed chow dogs are


Evaluating NLP Models via Contrast Sets

face to face in one image. Right? And I guess the task here is to see to have the system answer,


Evaluating NLP Models via Contrast Sets

is this correct or incorrect? And as you see here, I believe that's a correct statement. Or


Evaluating NLP Models via Contrast Sets

you're, you're maybe tasked to ask, which is the image that this applies to write? Is it image one


Evaluating NLP Models via Contrast Sets

or image two? And of course, here, it's image one. Now, the, the problem with such systems is that


Evaluating NLP Models via Contrast Sets

there are a lot of easy things that the models can do that will usually get them the answer. What we


Evaluating NLP Models via Contrast Sets

like to imagine is that the model will look at this and recognize that this is a dog here, right?


Evaluating NLP Models via Contrast Sets

This is a dog, here is its face, and this is a dog and here is its face. And it will see aha,


Evaluating NLP Models via Contrast Sets

there's a count, there's two of them, right? There's two of them. And there's the notion of


Evaluating NLP Models via Contrast Sets

face. And there's notion of pose and so on. But usually, there are tricks that the models can do


Evaluating NLP Models via Contrast Sets

to get this easier. For example, I know that in a particular visual question answering system,


Evaluating NLP Models via Contrast Sets

whenever there is a question of what is the ground covered in? Or something like this? Answer is


Evaluating NLP Models via Contrast Sets

always snow. Right? You don't even have to look at the image. And similarly, there are a lot of


Evaluating NLP Models via Contrast Sets

these kind of tricks that the models learn. And the authors recognize correctly that this is


Evaluating NLP Models via Contrast Sets

mostly a data set problem, right? So the data set, usually what you do in these data sets is you have


Evaluating NLP Models via Contrast Sets

an image, right? You have an image that you scrape from the web or something, and it has some


Evaluating NLP Models via Contrast Sets

mountains and maybe, right, and then there's snow, right? On the mountains, on the ground. And maybe


Evaluating NLP Models via Contrast Sets

it has. And you give this to a bunch of to a mechanical Turk or someone like a raider,


Evaluating NLP Models via Contrast Sets

and you instruct them, you produce a question to this image, you give them a couple of examples,


Evaluating NLP Models via Contrast Sets

right? And they're usually kind of lazy. And they will just look at it and be like, oh, what


Evaluating NLP Models via Contrast Sets

questions could I ask? Okay, I'll ask, you know, you need to ask something, usually the instructions


Evaluating NLP Models via Contrast Sets

are it must be visual. And it must maybe be answerable with a one word answer or something


Evaluating NLP Models via Contrast Sets

like this, right? Or it must, you know, it must be a multiple choice question. So there are these


Evaluating NLP Models via Contrast Sets

number of instructions, and they will usually be like, okay, what's kind of special about this


Evaluating NLP Models via Contrast Sets

picture? Okay, there's snow, so I'm gonna ask about that snow, right? So the, the problem is


Evaluating NLP Models via Contrast Sets

mainly the process of data set generation, right? That is, that will lead to kind of biases and easy


Evaluating NLP Models via Contrast Sets

solutions for the models where the models, they will simply kind of learn statistical correlations


Evaluating NLP Models via Contrast Sets

between things and the intention. So we have a big divern, di divergence between the intention of


Evaluating NLP Models via Contrast Sets

what the data set creators want, right, the intention is, in this case, is visual understanding,


Evaluating NLP Models via Contrast Sets

visual of the world. And there's a big, big difference between this and between how the


Evaluating NLP Models via Contrast Sets

data set is really constructed. So the authors are trying to address this with what they call


Evaluating NLP Models via Contrast Sets

contrast sets. Now they say, okay, you get out of this process here, you get a data set, right,


Evaluating NLP Models via Contrast Sets

you get a training data set and the test data set, maybe here, a smaller test data set. What they say


Evaluating NLP Models via Contrast Sets

is what we should do is we should additionally have these things called contrast sets, right? So


Evaluating NLP Models via Contrast Sets

this is train. This is test. And usually, these two come from the same distribution, right, you


Evaluating NLP Models via Contrast Sets

simply make them and then you split them somehow. And you take the test from the train. But these


Evaluating NLP Models via Contrast Sets

here are not from the same distribution. This is the contrast. What they argue is that the authors


Evaluating NLP Models via Contrast Sets

of the data set should create the contrast set, right, so you see that there's a split here,


Evaluating NLP Models via Contrast Sets

where the data set comes from, they argue that the authors of the data set with knowing what


Evaluating NLP Models via Contrast Sets

intention they have, they should create the contrast data set manually, by hand, in order


Evaluating NLP Models via Contrast Sets

to make kind of really hard examples that show what they want out of the system. They capture


Evaluating NLP Models via Contrast Sets

this here in their example. So if we go back to the example, here are things and they suggest to


Evaluating NLP Models via Contrast Sets

do this via perturbations. So what they would do is they would start at this example up here,


Evaluating NLP Models via Contrast Sets

right? They would start and they would perturb it textually, or via image. So they would perturb it


Evaluating NLP Models via Contrast Sets

to make it change the label, the gold label. This is different from adversarial examples. In


Evaluating NLP Models via Contrast Sets

adversarial examples, you want to perturb a sample such that it is still the same, but to the


Evaluating NLP Models via Contrast Sets

classifier, it's different. Here you have the opposite goal, you want to make something that


Evaluating NLP Models via Contrast Sets

is means kind of the opposite, but you want to test whether your classifier can pick up on that.


Evaluating NLP Models via Contrast Sets

So in this case, the one example will be too similarly colored and similarly post cats instead


Evaluating NLP Models via Contrast Sets

of dogs, right, are face to face in one image that would change the label, right? Whereas before,


Evaluating NLP Models via Contrast Sets

the answer was yes, that's a correct sentence. Now it's no, that's not a correct sentence. There are


Evaluating NLP Models via Contrast Sets

no cats in these images. Also, here, three similarly colored dogs. So the intention of the


Evaluating NLP Models via Contrast Sets

authors, you have to view it through this lens. The intention here is that the system can recognize


Evaluating NLP Models via Contrast Sets

the species, right of the entities in the images, the system can count. And the system can compare,


Evaluating NLP Models via Contrast Sets

right compare in this case colors. So you want to kind of make perturbations on these attributes


Evaluating NLP Models via Contrast Sets

from a test image, right? You can also think about image perturbations where you keep the sentence,


Evaluating NLP Models via Contrast Sets

but you modify the image such that there's still two dogs and they're simply they're still facing


Evaluating NLP Models via Contrast Sets

each other. But and but they're not similarly colored anymore, right? So the similarly colored


Evaluating NLP Models via Contrast Sets

here would be the the attribute that where before it was true. Now it's false with the new image,


Evaluating NLP Models via Contrast Sets

right? So you get the gist that the people that created the data set that know their intention


Evaluating NLP Models via Contrast Sets

will create manually these samples. And the authors, they propose a new metric to track this,


Evaluating NLP Models via Contrast Sets

but essentially, the authors propose how well models do on these contrast sets will be a


Evaluating NLP Models via Contrast Sets

reflection should be, it should be kind of an additional thing that people do with their NLP


Evaluating NLP Models via Contrast Sets

models. Alright, so you get the picture I that is, I believe the the entire gist of this paper.


Evaluating NLP Models via Contrast Sets

And I have some problems. First of all, here, they say, Alright, let's give a toy example in two


Evaluating NLP Models via Contrast Sets

dimensions. Say you have this data set, right, and the red one is the correct decision boundary,


Evaluating NLP Models via Contrast Sets

right? And you want to capture that. But because you only have limited training data, and because


Evaluating NLP Models via Contrast Sets

you in in this generation processes, you have a systematic biases. So if we had non systematic


Evaluating NLP Models via Contrast Sets

biases, we would think that, okay, we maybe pick this and this one, and this one here, and this one


Evaluating NLP Models via Contrast Sets

here, and this one here, right? We don't get all of them, but we kind of get an IID sample, right?


Evaluating NLP Models via Contrast Sets

That wouldn't be so much of a problem, you could still kind of recover the decision boundary. But


Evaluating NLP Models via Contrast Sets

because we have systematic biases, the authors argue, we, we actually introduce biases. So the


Evaluating NLP Models via Contrast Sets

systematic bias here is that we would have the blue ones, we would only capture things on this,


Evaluating NLP Models via Contrast Sets

sorry, on the on on this layer up here, and of the red ones, orange ones, we'd only capture


Evaluating NLP Models via Contrast Sets

things of the level down here. And thereby, we introduce the kind of data set the bias, it


Evaluating NLP Models via Contrast Sets

doesn't require this complex decision boundary anymore. Right? And if we now the problem is,


Evaluating NLP Models via Contrast Sets

if we collect the data set like this, and we simply say, well, these ones are the test set,


Evaluating NLP Models via Contrast Sets

and these ones are the train set, right? It will generalize well to the test set, but it will not


Evaluating NLP Models via Contrast Sets

generalize well to what we actually want. And therefore, the author say, well, if we introduce


Evaluating NLP Models via Contrast Sets

these contrast sets here, then you see that the decision boundary that we found will not perform


Evaluating NLP Models via Contrast Sets

well on these contrast sets, right? So they would say, we take one example of the test set, right?


Evaluating NLP Models via Contrast Sets

This is you can see this is this example right here. And we would perturb it to make it change


Evaluating NLP Models via Contrast Sets

its label, or in this case, one of them where the label remains, that we would kind of perturb it


Evaluating NLP Models via Contrast Sets

meaningfully. And yeah, so as I said, I have multiple problems with this first 2d toy examples,


Evaluating NLP Models via Contrast Sets

very, very bad for NLP models. First of all, low dimensional intuition does not generalize to high


Evaluating NLP Models via Contrast Sets

dimensional intuition, like very, very little. Second of all, even worse, usually these NLP


Evaluating NLP Models via Contrast Sets

models have so many parameters, much more parameters than you have data set, that which


Evaluating NLP Models via Contrast Sets

means that your decision boundary is instantly going to be simple, even if you had all the data,


Evaluating NLP Models via Contrast Sets

you could possibly want this, it's just a very different kind of problem. And then the next


Evaluating NLP Models via Contrast Sets

problem is if even with by doing this contrast set, and you already see it here, right? You


Evaluating NLP Models via Contrast Sets

already see it. You can only kind of bicker about the data, okay, but with the contrast set, you


Evaluating NLP Models via Contrast Sets

only really capture this one aspect. So if that was actually well adhered to, you could measure


Evaluating NLP Models via Contrast Sets

very locally whether or not this this would work or not. And the ability to come up with meaningful


Evaluating NLP Models via Contrast Sets

contrast sets to ever capture what the model is doing is almost impossible because you have to


Evaluating NLP Models via Contrast Sets

create them manually. And then you suggest that the authors themselves make these contrast sets.


Evaluating NLP Models via Contrast Sets

Remember, the authors are the ones that gave these instructions, right? These instructions


Evaluating NLP Models via Contrast Sets

right here, the authors provided them to the to the data set annotators. So the authors will


Evaluating NLP Models via Contrast Sets

probably be even more biased if they have to do their own right, if they have to now create their


Evaluating NLP Models via Contrast Sets

own contrast examples, they will probably, even though they know their intention, they will


Evaluating NLP Models via Contrast Sets

probably be like more biased than if you at least this here, at least this here is a distributed


Evaluating NLP Models via Contrast Sets

process across people, right? So you get things that you wouldn't have thought of. But if just


Evaluating NLP Models via Contrast Sets

the three authors of the date of the paper, make the contrast examples, I would argue that that's


Evaluating NLP Models via Contrast Sets

an even more biased measure often. So all of this, it just strikes me as, as the paper is basically


Evaluating NLP Models via Contrast Sets

saying, let's try on a few things. And I think the fundamental problem is much, much deeper. And


Evaluating NLP Models via Contrast Sets

it goes with this intention part, like I get it, the the visual question answering data set doesn't


Evaluating NLP Models via Contrast Sets

capture the doesn't capture what you want. It doesn't make the model suddenly understand that


Evaluating NLP Models via Contrast Sets

there are dogs and there are species of animal and so on. It simply makes it correlate things. But


Evaluating NLP Models via Contrast Sets

that's what deep learning and especially NLP does. So right, it's like, it's like saying you


Evaluating NLP Models via Contrast Sets

build a build an image net classifier, and it can't fly. And if I if I try it on my tests, that it


Evaluating NLP Models via Contrast Sets

requires my computer to fly and my image net model can do this, then it doesn't serve my intention,


Evaluating NLP Models via Contrast Sets

right? And I mean, it's, it's a crass example. But ultimately, you the correct approach should


Evaluating NLP Models via Contrast Sets

be to better encapsulate your intention into the data set generating process and then correctly


Evaluating NLP Models via Contrast Sets

interpreting the results. That mean, okay, on this data set, as far as we can tell the way we created


Evaluating NLP Models via Contrast Sets

it, this is the performance of the model, it doesn't, the model will never learn to fulfill


Evaluating NLP Models via Contrast Sets

your intention. And I get it, that's what you're saying. But still, even with this contrast set,


Evaluating NLP Models via Contrast Sets

I think it's a really bad measure to formally propose it's I think you should much more propose


Evaluating NLP Models via Contrast Sets

how is the data set generating process different from what you want and what are the limitations


Evaluating NLP Models via Contrast Sets

there, right? And so that's that that I think that will lead to much more meaningful, meaningful


Evaluating NLP Models via Contrast Sets

results than simply the authors providing a few manually put examples that they feel capture their


Evaluating NLP Models via Contrast Sets

intention, it will not will not the reason we do deep learning, instead of straightforward, if


Evaluating NLP Models via Contrast Sets

else programming is because we cannot capture even our intentions. And therefore, data set


Evaluating NLP Models via Contrast Sets

generation is the only is the only method we have, so to say. Right. So, ultimately, I believe


Evaluating NLP Models via Contrast Sets

these these whole NLP, especially the visual question answering and so on the natural language


Evaluating NLP Models via Contrast Sets

understanding part needs to have a grounding. So, ultimately, I think grounding, grounded NLP, it


Evaluating NLP Models via Contrast Sets

means basically that you're not only doing NLP, which is simply you take text and you take images


Evaluating NLP Models via Contrast Sets

and you correlate them somehow, right? You just make a statistical connection. Grounded NLP models


Evaluating NLP Models via Contrast Sets

is the hope that you could build something that actually understands the world understands that


Evaluating NLP Models via Contrast Sets

there's entities that is interacted, there's something like a pose that there is something like


Evaluating NLP Models via Contrast Sets

something like what the color means, right? What a dog is, and so on. And as entities, I think we're


Evaluating NLP Models via Contrast Sets

not there yet. And I think that will be the ultimate solution to these kinds of tasks, not not


Evaluating NLP Models via Contrast Sets

any sort of local, very local, very low dimensional perturbation. I mean, yeah, let's say you create


Evaluating NLP Models via Contrast Sets

a contrast set, you will be able to capture one tiny little bit of your intention one tiny little


Evaluating NLP Models via Contrast Sets

bit, even though you know your intention, you will capture a tiny little bit, all of the 1000 other


Evaluating NLP Models via Contrast Sets

degrees of freedom of your own intention, you won't be able in the to capture in the contrast


Evaluating NLP Models via Contrast Sets

set, I guarantee you. All right, that was my quarrels with that I invite you to read the whole


Evaluating NLP Models via Contrast Sets

paper. They actually do this for NLP data sets. It's a lot of work. And they show that the models


Evaluating NLP Models via Contrast Sets

perform much worse on their contrast sets. And interestingly, the humans don't the humans are


Evaluating NLP Models via Contrast Sets

able to solve the contrast set. Of course, of course, because you tell the humans what the task


Evaluating NLP Models via Contrast Sets

is right? That's like, humans succeed on contrast set, like how surprising. What you should do is


Evaluating NLP Models via Contrast Sets

you should just provide the humans with the data set, not tell them what the task is. Even worse,


Evaluating NLP Models via Contrast Sets

just provide them with the encoded data set, like not the text itself, but actually the token IDs,


Evaluating NLP Models via Contrast Sets

right? And then and then make them do the thing. And the humans will just as well make a statistical


Evaluating NLP Models via Contrast Sets

correlation between the tokens and the images or whatnot. And the humans will fail just as well on


Evaluating NLP Models via Contrast Sets

the test on these contrast sets, because the humans, maybe they'll figure out what the task is,


Evaluating NLP Models via Contrast Sets

but probably not. So humans succeed on contrast set, how surprising you tell them the intention,


Evaluating NLP Models via Contrast Sets

while you don't tell it to the model. Yes. I seem a bit critical. But yeah, please read the paper.


Evaluating NLP Models via Contrast Sets


The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks

Hi there, today we're looking at the lottery ticket hypothesis finding sparse trainable neural


networks by Jonathan Frankel and Michael Carbin. So this paper is sort of an empirical paper into


what makes neural networks train successfully. And it comes out of the literature of pruning. So they


say neural network pruning techniques, right, they have been around for a while, they can reduce the


parameter counts of trained networks by over 90%, decreasing storage requirements and improving


computational performance or inference without compromising accuracy. Right. So what does this


mean? If you have a neural network, let's say you just have three nodes, each layer, you have two


layers here, you have a neural network here. If you have a fully connected neural network, every


node is going to be connected with every node in the next layer, right. And these are these


connections are your weights, your thetas. And you're going to train them, which means you have


a you have a number of steps in this direction. And let's say you have a test set accuracy, right


here. So here is steps, you're going to train them. And if you train them, your accuracy will reach a


certain point, right. I'm just going to draw the the endpoint here, let's say you reach a 90% test


accuracy. So your network generalizes pretty well. That's pretty good. So people have been wondering,


this these networks, they they require quite a lot of storage, you know, this is nine connections


right here. So three times three, and this is also nine connections, can we make it smaller,


but still retain the accuracy. And this is where pruning comes in. So with pruning, people would


go and after you train them. So the first step is train the full network, right. And then the second


step is prune. Now when when you prune, you basically select among the weights that you have,


that you have trained, you select the best ones in some form or another. In this case,


people just select the ones with the largest magnitudes. But there are multiple techniques


to do this. And this is very related to things like quantization or distillation. So with pruning,


you just leave away some of the weights or most of the weights. And you hope that you still retain


a pretty good accuracy right here, right? Sorry, actually, we don't need these steps thing. So you


leave away weights, and you retain a good accuracy. So pruning methods have been deployed successfully


to make networks use less space or be faster to evaluate because of course, with less numbers,


you need to do less calculations. Right. So this paper builds on top of this. And it basically


says, All right, if if we do the following, if if we now take this network that we identified


after training, and we, we just take this network, and we train it from the beginning, only this sub


network, right, so three is retrain, then it will also perform pretty well or even better, under one


condition, right. So if you only train this thing, it will perform well under one condition. And the


condition is that you transfer over the initial weights. So right, the question is, can we train


just the small network from the beginning so that we don't have to train the big network, right. And


the paper identifies that this works if if your initial weights, theta zero of the small network,


are equal to the initial weights of the large network, right, just so just the ones where you


pour them over. But basically, the short answer is no. And the reason is, if you only want to


train this small network, you need to know the good initialization of these of these weights all


here. And the good initialization, you only know after you've trained the large network and actually


identified which of these connections make sense. So you can't just take a smaller network from the


beginning, you have to train the larger one, then you know which weights and which initializations


make sense. So this is the winning lottery ticket hypothesis. Basically, it states and we can read


it out in full. The lottery ticket hypothesis is a randomly initialized dense neural network


contains a sub network that is initialized such that when trained in isolation, it can match the


test accuracy of the original network after training for at most the same number of iterations.


Right? Now, the important part here is that it contains a sub network, right, that is initialized


such that when trained in isolation. So two things are important. It is important the structure of


the network of the sub network, but it is also important, what are the initialization of the


connections. So the, the paper kind of hints at why neural networks work at all. And the reason why


neural networks work is because we've often thought of, hey, neural networks have so many parameters,


how can they even generalize? The reason is the following. If we have a neural network, we throw


so many parameters at it, some of the parameters, one subset of the parameters, namely the red ones


here are going to be initialized in such a way in such a beneficial way, that training will perform


will make the network perform well, right? So it's initialization plus SGD on that, that sub


network, right. So it is actually only a very small sub network that is responsible for the


performance of the neural network. But, but that sub network needs to be initialized at the correct


position. And by over parameterizing these neural networks so much, we actually give it


combinatorically many sub networks to choose from where the initialization could be well. So So


because of this combinatorics, it means that if we over parameterize by some margin, then there's


almost guaranteed to be a good sub network in there that can then perform well, right. So I hope


this makes sense. It is basically it is not a way. It is not a magic thing where you now we can we


now can train the smaller networks. It is it is an explanation of why the over parameterization in


neural networks makes sense. Because by over parameterizing, we allow the neural networks


to combinate to exploit the combinatorics to find a good, well initialized sub network that will


perform well. And the evidence for this is exactly the fact that if we transfer over the sub network,


it by itself will reach the same performance or actually exceed the performance, but only if we


initialize it at the same point as it was initialized in the original network. So here is


how these these sub networks are identified. We've already hinted at that, but here is how the the


paper does it. Right. So says identifying winning tickets first randomly initialize a neural network,


this is the full neural network, right? Then train the network for J iterations arriving at


some parameters, right? These are the trained parameters. prune p percent of the parameters,


right? So of these parameters prune some, right? And this is for in order to know which ones you


prune, you need to have first trained the full neural network, right? So this is the catch here,


you need to train the full neural network to know which ones you must prune. And thereby,


you create a mask m, right. And then they say, reset the remaining parameters to their value in


theta zero, actually, you don't need to say remaining, you can just say reset the parameters


to their values in theta zero. Now, this is also important, this is the same theta zero, as it was


at the beginning of the training, right? So you need to actually set them back to those exact


values. And thereby, you create the winning ticket. This, okay, actually, if you if you just want to


end up with a trained network, then you then this, this, this remaining thing here is important. But


if you then want to retrain, you can only you can set everything back and only train the masked


version of the network, right. And they, they say this, this will identify these winning tickets,


and it will actually work better if you don't do this in what they call one shot. But if you do


this iterative pruning, that means it repeatedly trains, prunes and resets the network over n


rounds, each round prunes p to the one over n percent of the weights that survive the previous


round. Now, why might that be? It might be and this is this is, I think, a somewhat valid hypothesis


that I myself put forth here. It might be that if you prune some of the weights, right, let's say


you prune this one, and this one, what you'll do is you'll put the responsibility of these weights


on two other weights. So maybe on this one, and this one. So as we said, they prune by, by looking


at which weights are large. So let's say here, we have the weights of the layer. And these are the


magnitudes of the of the weights. Right? So, okay. So you would prune, let's say you only want to keep


two of those around. So you would prune this one and this one because these are pretty small, right?


Here's the magnitude. And you would also prune that one, right? If you just do this one shot,


and then you would retrain and maybe these weights would end up somewhat different. But if you do


this in multiple rounds, right, let's say you first prune one of them, you only prune the smallest one,


right, this one here, and then you retrain, and then your weights actually change. And all of the


responsibility that this weight carried before is now transferred onto this, right? So your new,


your new weights look like this, and you prune another one like this. And again, all the


responsibility of this would in my hypothetical example, fall on this one, right? And now if you


prune a third one, you would actually prune this one because you realize, ah, this weight here,


in absence of these two other weights is actually important. So you would prune this one as well,


right? So I think that is why this kind of iterative pruning method might work a bit


better than the one shot pruning method that they say here. So they do a lot of empirical


investigation. And I just want to highlight very few of them. But so that you get the gist,


and then the paper goes into a lot of detail and a lot of different architectures that you can check


out yourself. Alright, so here we have a plot, sorry, that that deals with percent of weights


remaining. So as you go to the right here, they drop in more and more weights and realize this is


a log plot. Right? So if the dashed lines here are random pruning, which means you just drop out a


certain number of weights, and then you you retrain, right? And you can see that the dashed


line here, it starts dropping and just becomes worse as you as you have less and less weights


remaining, which is exactly what's expected, right? You prune the network, you make it smaller,


you make it less performant. And the more weight you take away, the less performing it is. But


interestingly enough, right? If you do this pruning that they suggest, and then retrain with


the correct initialization, not only do you retain the same level of accuracy for very long,


you see here, this is 2.9 or 1.2% of weights remaining, but you actually go higher, right? So


you can see here, when you have 16% of weights remaining, you there's actually a significant


difference between the full network and the pruned network. And that's only by simply training this


winning hypothesis. So this I find very, very fascinating. And again, this is not a magic bullet


that you can do from the beginning, but it does give a clue that if you could train, sorry, if you


could train these from the beginning, then then you might actually end up at a better point. So it


does actually give a practical application. Also, you see they train faster. So the blue line here


is the full network over the course of training. Sorry, this should be blue. So here is training


iterations. And this is test accuracy. So you see the full network does something like this. Now,


if you prune to what 20% of the weights actually train faster, and you go higher, and even if you


have 7% of the weights, you go almost as high. So this is very interesting. Only when you go to like,


1.9% of the weights does your performance degrade again, and eventually actually go lower than the


original network. So that that is pretty, pretty, pretty cool. I think. Now, they do, as I said,


they do a lot of investigation. And I think one of the one of the main takeaways is that it is not


only the structure of the winning hypothesis. So it's not only the structure of the sub network


that makes it to be a winning hypothesis, it is actually the initialization. Here, I want to show


one of these plots, they have lots of plots. You can see here, for example, sorry, this is from my


own annotations. Again, this is percent of weights remaining, and this is test accuracy at the final


iteration. And if we initialize the sub network at its original position, like this method suggests,


you see, we first increase the accuracy and then decrease it after a long time. If we take the same


sub network, right, but we randomly reinitialize it, then it drops much faster and actually


immediately drops. So it really is about not only the structure of the sub network, but about its


initialization. I think that is that is the core of the hypothesis here. A very interesting related


finding that I just want to mention I find to be that they actually discover that the weights so if


you have a weight of the so if you have two kinds of weights, let's actually go up to my original


drawing here. If you have if you compare how fast or how far do the weights travel in optimization


space, right, so you can you can basically look at how far weights travel during optimization. So


you take the full neural network here, and you look at a parameter that ends up being in the


winning hypothesis theta, theta zero, and it goes to theta end, which let's say theta final. And you


also look at parameters that don't end up in the winning hypothesis. Let's call these theta one,


two, theta, also final prime, not too good at labeling. And you look at how far they travel,


you'll find that the weights that end up in the winning hypothesis, they during optimization,


they travel much further in optimization space than weights that are not in the winning hypothesis,


right, they just stay around much more. So it's not that the kind of good network is already


contained in initialization, it's much more than than the good network lens itself. Very


favorably to be initialized by SGD, right, because it travels farther. It means SGD has a bigger,


bigger pull on it, right. I think there is a lot of things that are yet to be explored in this


space. And I think this paper is a very cool contribution to our understanding of how neural


networks work. Alright, I invite you to check out all the experiments, they do a very thorough job.


Training more effective learned optimizers, and using them to train themselves (Paper Explained)

Hi there, today we'll look at tasks, stability, architecture and compute,


training more effective learned optimizers and using them to train themselves by Luke Metz,


Nero Mahesvaranathan, C. Daniel Friedman, Ben Poole and Jascha Sol Dikstein. So on a high level,


this paper deals with sort of a meta problem. It deals with learning optimizers that learn


machine learning models. Learned optimizers is kind of a new field of research. And the goal is


to obtain an optimization function that can be used to train all kinds of machine learning models.


And this paper builds on a line of research and kind of extends that research. It doesn't,


it's not the first one to do this, but it is so far the largest and most compute intensive and


most task encompassing notion of learned optimizers. And the optimizer they end up with


has some nice properties as they're going to show. And also, it can be used to train itself. So it


can iteratively be used to train itself, ending up with a even better learned optimizer. So we're


going to go through the paper and we're going to find out how much of these claims are kind of


wishful thinking and how much are actually true. I have mixed feelings about this paper, though,


in all of this, remember, my opinion is my opinion. And they are very open about their results,


which is something I really, really appreciate. I feel that if more papers were as open as these


people are about what worked and also what didn't work, we would be in a better place as a research


community. That being said, as I said, I do have some mixed feelings about the statements being


made here and about how the results are interpreted. So stick around if you're interested


into that. Also, I find the broader impact statement to be a bit funny, but we'll come to that


at the very end. If you like content like this, as always, don't hesitate to share it out. I've


been on a bit of a break. It feels good to be back making videos after after right paper deadlines.


Let's dive in. They say, much as replacing hand design features with learned functions has


revolutionized how we solve perceptual tasks, we believe learned algorithms will transform how we


trained how we train models. So lots of packings in this sentence. For those for you young kids


that have been growing up with deep learning, there was a time before deep learning. And


basically, what we would do is we would use hand design features. And this works really well, if


you have like a database of customer data, it worked moderately well, if you have like a picture.


So if you have a picture, whatever of your cat, what people used to do is they used to run these


kind of very handcrafted detectors feature extractors over this. So these might be like


fixed filters, like three by three Sobel filters, gradient filters, and so on, run them over the


image, try to detect corners, try to detect very small things. And then once they had a couple of


features like this, they would feed this into a classic kind of classification algorithm like a


logistic regression, and so on. There were sophisticated approaches, but most required the


hand engineering of features. Of course, deep learning transformed all of this deep learning


basically, if you want to take a cynical look at deep learning, it's simply replacing the part that


creates the features, the classifier is still like a logistic regression. However, deep learning


knows how itself can extract good features, in fact, better features than humans ever could for


perceptual tasks. So for images for sound, in the latest iterations, also for language. These people


say that this can also this kind of thinking can also be applied to this optimization algorithms.


So in optimization, what you want to do is you want to train your deep network, right? Whatever


goes from your image from this thing right here to your final output, you want to train this and


we train this using gradient descent. So what this has is usually there's like many, many layers in


your deep neural network, and each one has parameters, well, let's call them theta, theta,


theta one, theta two, and so on. These are all vectors or matrices, your convolutional filters,


your batch norm parameters, and so on. We can collect all of these into a big parameter vector,


let's call that theta. And the task is now to find the best theta, I think you're introduced to that.


So in optimization, what you want to do is you have a theta, you feed an x, you feed an example


through it, you get some sort of output, let's call that f, that gives you some sort of loss,


you back propagate that loss, and what you end up with is a gradient of theta. If we were just


doing gradient descent, we would update theta right here, we would update theta to be theta


minus the gradient of theta given some step size right here. This is classic gradient descent. And


most algorithms are something like this. In, for example, gradient descent with momentum considers


has like some additional term right here, where they consider the last steps. AdaGrad, for example,


considers a factor down here where they divide by some kind of the square norm of past gradient.


So D, sorry, the D, this, you add up the past gradient square norms like this, or you average


over them. There are many variants, you can do this averaging right here also with momentum in


kind of a decaying way. There are all sorts of algorithms to optimize these functions. And the


sense behind this is that ultimately deep learning is a non convex problem. So instead of your


classic classifiers, they look something like this as a loss function in your parameters or more,


maybe more to say something like this, if we look at it in 2d, and you can just do gradient descent,


basically go to the optimum. However, in deep learning, it's a bit of a different situation.


So you might have many different optima, many local optima. And we know by now that we can go


to either one of them. And that should be fine. So let's do some level sets right here, maybe here,


here. Okay, but so you can see right here, you have multiple optima where these dots are, but in


between, it's kind of shaky. So you might have like a major flat area right here, and you might


have a major flat area right here. But then as you get close to this optima, maybe the steepness


increases. So if you look at a cross section, there might be like some sort of a flat area,


and then it increases again, and you want an optimization algorithm to kind of automatically


adjust to the steepness, and to changes in steepness, and so on. And that's what these


modifications to gradient descent are supposed to do. So AdaGrad, for example, adjusts automatically


to a shape like this. So even if it's convex, you can see that the scale of this parameter is much


flatter than of this parameter. AdaGrad would automatically kind of stretch one out and make


the other smaller, such that it transforms it to a nice kind of all the all dimensions are equal


problem because you only have one learning rate per dimension. If you go further and go into the


dimensions of Adam or RMS, these now can also kind of change over time. AdaGrad also to a degree,


but much more so these other algorithms can adapt to like changes in steepness. And once it goes


flat again, they can kind of recognize, oh, now it's flat again. So I might do some bigger steps.


Once it goes steep again, they're like, okay, I should probably be kind of concerned right here.


So the first notion of momentum, that's really useful. The kind of counters stochasticity of


stochastic gradient descent. It's a big field, but what they all have in common, it's humans


sitting down coming up with this particular, like a particular formula because they feel, ah, if I,


you know, do this thing, then it might do this, it might stretch out these dimensions that might


be artificial. These are humans sitting down. Now, the analogy here that these people make is


we used to do this for classifiers, we used to hand design features that we felt make sense,


like the image gradient and so on, or the FFT for let's say for sound. And and that that worked so


far, but it worked better when we let deep learning do its thing. And the goal, of course,


is also that we let machine learning come up with the optimization procedure. So what exactly goes


so if we try to update theta, we might update it not as a fixed formula, but we might take the old


theta, we might take the gradient of theta, and we might take a bunch of features that we calculate


from these things like things like the sum over the norm of old gradients, and so on. And we put


all into a big function. So f and f is, you know, in the classic sense, that's what the humans


define. But now the goal, of course, is to learn f. So do you have a set of meta parameters,


let's call them whatever that thing is. And, and and phi, maybe, psi, I know psi, let's call it


like this. And now have a have a meta parameters. So let's use it, let's parameterize f as a neural


network that learns to output the next weight for the underlying neural network. Now, the F itself,


of course, has to be learned somehow. But the idea is, is kind of since it's a meta algorithm,


meta algorithms tend to be much more general and much more smooth, and therefore they themselves


could be optimized fairly generally. And once we have a good F, we can apply it to all sorts of


tasks. And that's exactly what they do. So they consider three problems in learning optimizers.


So first of all, computational scale, learning optimizers is hard. And this paper here, invests


a lot of compute into learning one meta optimizer. Second, training tasks. And this I feel, this is


the kind of the core here, in that what they do is that they now you have to pay attention. So


if we talk about data sets, it's it's very confusing now, because on one hand, you have data


sets like MNIST. And you have data sets like CIFAR-10, right? So these are data sets. But


in the in the task of learning an optimizer, a data set is something like this. So in MNIST,


let's just make the analogy here, we have following samples, this image, this image,


this image, right? In CIFAR-10, we have like this airplane right here. This is an airplane,


it's an airplane, believe me, with the truck, right truck, and so on, we have this. Now,


these are the classic data sets. However, in this paper, a data set consists of the following. And


this data set they use here is called task set. So one sample in the task set data set is I take


the MNIST data set, I use like a five layer CNN on MNIST. And I use a batch size of 32. And I let


it run for 10k steps, and so on. That's one sample, right? The next sample could be I take CIFAR-10.


I use a ResNet-50 on it, my batch size is 64. And I let it run for 50k steps. Right, so this,


these are now samples in this task set data set. And the task set data set consists of a wide


variety of tasks, I believe over 6000 different samples, which include things like RNN tasks,


image recognition tasks, very simple, like 2d optimization, or sorry, quadratic optimization


tasks, and so on. So there's all these kind of different tasks. And the goal you can see now,


the goal is that if we find, so here, what's the goal when we learn MNIST? What the goal is, if


our output is going to be a CNN that we can input any sort of digit into, and it gives us the label


2. The goal here in task set is, if we find F, an optimizer that works for all of these samples in


the data set, then we can give any sort of new sample. So let's say we will give we'll have a


new problem, right, we'll have our medical, medical data set. And we have this ResNet-101 that we want


to train on it, not a pre train, but that we want to train on it, we want to train with a batch size


of 64. And so on, we can input that, and the the optimizer will spit out good parameters for that


particular date for that ResNet-101, the optimizer will be good. So it's important to stress that we


are looking for one single optimizer, one single function that can optimize all these kinds of


different tasks, right? That's a challenge, of course. And that's what this paper attempts.


And then the last thing here they say is the inductive bias of optimizer architecture. The


parameterization of the learned optimizer and the task information fed to it strongly affect


performance. In this work, we propose a new hierarchical learned optimizer architecture


that incorporates additional task information such as validation loss, and show that it outperforms


the previous learned optimizer architectures. So I think you get the overview right now. So let's


actually jump right in. So what does their optimizer look like? Their optimizer, here is


kind of the contrast to previous work. Let's actually jump into their optimizer. Their


optimizer consists of each parameter is associated with one LSTM and one feedforward network. Okay,


so the LSTM gets the following. Actually, let's look at the the feedforward network.


Where do they say what these output? At some point, they say what they output.


One second. Nope, nope. So here,


such as training loss, validation loss, normalized to have a relatively consistent scale to compute


zero to compute a weight update, the per parameter MLP outputs two values, A and B,


which are used to update inner parameters. So their formula to update this is what we call theta


right here. Their formula to update theta is this thing right here, x of A and B. So


for each parameter, their optimizers outputs A and B. So that's this feedforward network. It doesn't


actually, as I can tell, this paper is very confusing. Like there are multiple points where


it's not clear what they do and their notation differences doesn't help. So here, if I had to


guess, I would say they don't output delta w, they actually output A and B. Okay. So


into their feedforward network goes the most important thing is the gradient. Okay, if if this


network were to do something very trivial, it would simply output the gradient right here,


it would, it would make a equal to one, no, what's x of one? No, that doesn't work. Zero, sorry,


he would output A equal to zero, and B equal to the gradient. And then you just get gradient descent


back. But we also want to feed it with information that it could use, right, that it could use to


to make better decisions, such as momentum, right now, if it can, it could technically reproduce


SGD with momentum, we give it the second moment, well, now it can, it can do things like AdaGrad,


because that uses the second moment, it's noted, notice, like, note that this algorithm doesn't do


it symbolically, there are other papers that try to come up with a symbolic algorithm,


with a symbolic expression for a better optimizer, right? Like I've shown you with Adam, like you can


write it down as a symbolic expression. This is not that paper, this paper, really, the output of


the feedforward network is a number, or two numbers per parameter, or two vectors, whatever you,


you want to look at it like this is a numerical procedure, you're really trying to find this thing


is this F, it's really a vector goes in and a vector goes out. Okay, and these are the features,


gradient momentum, second moment, and so on. There are more features that go into the model,


namely training and validation loss. So since you are training an underlying model, you have access


to the labels at all time. This is what you have to think even at test time. So when you test your F


with a test task, that test sample will have an associated training data set with it, right?


And you're going to have the loss of that training data set. And you're also going to have the


validation loss. I guess you could split it yourself if you wanted to. But the goal that's,


we're going to come how we exactly optimize F and what the loss for us is. But intuitively,


you want to train your F such that the validation loss of the inner task is as small as possible.


And we're going to see how that works. So yeah, the tensor shape as well. So it could technically do


something like implicit batch norm, right? It could do that, depending on how big the current


tensor is that it optimizes gradient norm, and so on. So the total norm of the total gradient,


they just feed all this kind of information in here. And you can already see kind of my first,


my first bummer with this is that if this were really modeled after classic deep learning, what


you would input is two things. Okay, maybe like the current step. No, not even that. So what you


would input is two things, you would input your sample x, and you would input the gradient. Okay,


like you would input your, your or sorry, not the sample, you would input the current weight,


yes, the W that you're changing, and you would input the gradient, which is


the gradient that you get from backprop from the underlying system. And this technically,


since the LSTM goes over time, right, so in each step, the LSTM technically remembers the last


steps. If this is a neural network, it's a universal function approximator, it could


technically calculate the momentum, it could technically calculate the second moment of these


things. I guess these things here, you you could feed in, I agree, couldn't do that.


conceivably, but these other things, you could, you know, this, it could calculate this. So we're


back into the business of feature engineering. And this is going to, and they say this at the


beginning, right, as I said, this paper is quite honest, they say that these things that they feed


in also these things, they make a lot in terms of the final performance of this model. So this kind


of bugs itself with the analogy of, hey, remember, when we replaced handcrafted features with learned


features in computer vision, let's do the same. It's only halfway there, as yes, we are replacing


the symbolic operation, but we are still inputting a lot of the handcrafted features that we think


are useful. Okay, so as you can see, there's an LSTM going over the time steps. And for each,


for each parameter, there's a small feed forward network, the output of the feed forward network


is going to be sent back to the next step of the LSTM, the LSTM, of course, is recurrent,


and so on. So I hope you can see how this works. So what this what this does is is you have a neural


network that you input a data set into your letter data set run through it, it gives you a loss, and


you are using F to optimize that loss, right? F is a function that takes in the W of the current


neural network, that's the W here, and it outputs the W at the next step t plus one, you do this


for a bunch of steps. So a bunch of steps until you have like, I don't know n steps, then you


take your validation data set of the inner task, your validation data set, and you calculate your


final loss, loss of your validation data set given W, so loss given W of the validation data,


this is disconnected right here. And what you want is you want to optimize the psi of the F


such that that loss is as small as possible. I hope you can see the problem in this, even if this


is all differentiable, which it can be right, you are going to have to back propagate through


n inner steps of optimization since each of these steps is a forward propagation through F,


right? And only at the end, you have an actual loss right here, a validation loss. So you're


going to have to back prop through all these n steps, which is simply not possible. Currently,


we can't back prop through 1000s of steps, and we need 1000s of steps currently to optimize deep


learning architectures. So they are opting for something different. Okay. So we have this model,


the model is acting as an optimizer. At the end, there's a validation loss. And we are wondering,


how should we optimize this model to make the validation loss as small as possible, given an


n step rollout of the underlying thing, while we can't back propagate through the entire rollout.


And if you have guessed reinforcement learning, you're almost correct. So the answer here is


going to be evolution strategies. They say at right here, we deal with these issues by using


derivative free optimization, specifically evolutionary strategies to minimize the outer


loss, obviating the need to compute derivatives through the unrolled optimization process.


Previous work has used unrolled derivatives and was thus limited to short numbers of unrolled


steps. Using evolution strategies, we are able to use considerably longer unrolls.


Okay, so they use these evolution strategies and later these persistent evolution strategies,


which are modification. So evolution strategies, really briefly, there are many, many variants of


it. But ultimately, what you can do is you are here with your guess of the best parameters,


you are going to perturb these parameters by a little bit in multiple directions. So since


evolution kind of the the, there are many ways of evolutionary strategies. And this, I feel what


they do here is sort of the weakest way, because I've had people flame me before because they're


saying that these are not really evolution strategies. And I agree, is basically glorified


random search. So you kind of perturb it in each direction, you end up with this population,


then you evaluate each of these new data points. And maybe you'll find that this one, this one,


this one, this one, these are actually good. This is like meh, meh, and these ones are really bad,


okay, or like worse. So you want to shift your guess of the best parameters into the direction


of the of the good ones, and away from the direction of the bad ones. And you can kind of


see this green thing here as a pseudo pseudo gradient is kind of a finite difference method,


if you really think about it. And I know evolutionary strategies and so on, they contain


things like crossover and whatnot inspired by biology. Honestly, they don't say much here, but


I have read the the kind of other papers or I've not fully read them, but looked at them. And it


looks to me like that they're doing something like this. And they're using kind of the same trick to


calculate the pseudo gradient as the reinforce algorithm. So this is kind of the log derivative


trick to differentiate something that is not differentiable. And yeah, so again, this is not


really written well, because here I would expect that they just take a step into the direction of


these good perturbed points. But what it seems like just from the abstract, because in the


abstract, they say, Oh, we optimize all our things using Adam. Right. And so in terms of the outer


grade, I can actually show you this is so here is a, again, not to rag on these, maybe I'm just a


poor reader. But this is a wildly confusing paper to read. And I have still have not really a clue


what's going on. Because things are just described vaguely, then there's this pseudo code, which


doesn't help like it does not help. I like it just, it basically just specifies how they named


their variables. It doesn't show you most of the actually important logic. At least that's what I


feel. Okay, so here, outer optimization details. We optimize all models with Adam, right, we swept


the learning rates, yada, yada, yada, we find the optimal learning rate is very sensitive and


changes, depending on how long the outer training occurs. So it's clearly they say outer training,


and Adam, which means they use Adam for the outer training. But before they say, Oh, we use derivative


free methods, like evolution strategies. And they don't say anything about Adam up here.


So what I'm guessing is that they use the evolution strategies to find these pseudo gradients


right here, because in the paper that I've looked up from them, which is their own older work,


that they use these evolution strategies to obtain a gradient. And then I'm going to guess


they take this gradient right here, and they feed that as the task gradient into Adam.


And then they use Adam to to basically optimize their their outer thing. But instead of back


propping to get the gradient, they use yes to get the gradient. I'm guessing that's what's happening.


Yeah, so that for that, then task distributions, as we said, they have this task data set.


6000 tasks designed after this task set data set. It's not exactly task set, I think it's inspired


by task set. These tasks include RNN, CNNs, masked autoregressive flows, fully connected networks,


language modeling, various variational auto encoders, simple 2d test functions, quadratic


bowls, and more. For tasks that require them, we additionally sample a data set batch size


network architecture initialization scheme. So there are multiple issues here. One issue is that


right next sentence to keep outro training efficient, we ensure that all tasks take less


than 100 milliseconds per training step. For each task that makes use of a data set, we will create


four splits to prevent data leakage. This is very cool that they, you know, really separate


inner training, inner validation, outer training, outer validation, and so on. Sorry, not outer


training, outer validation, and then outer test that they only look at at the end. Of course,


outer training is the inner task. But you can see that even Google research hasn't doesn't have


really enough compute here to really thoroughly survey deep learning as a field and and take all


the tasks into consideration. So they have to like settle for rather small tasks like c410,


MNIST, and so on, and various small architectures, of course, that go along with it. And if you know


much about deep learning, you know that there are considerable effects of scale in these things,


namely optimization has, I think optimization honestly has kind of gone back a step in terms


of complexity. It used to be much more of a debate like, oh, should you know this optimization


algorithm that one. Now, most people use Adam. And also a lot of people just use SGD with momentum,


SGD with momentum, and especially in the larger models, like let's say, BERT or even larger models.


SGD with momentum seems to be the way to go not only because it's it's easy to implement,


but because it actually performs well, especially in large models with large data. So


there are considerable effects of scale and by only training on small models and data, that


is very big hindrance. And we're going to see it in the results right after writing the next step


right here, that this, this is limited to that. This is limited to that, let's say,


to that domain, they also say up here, unfortunately, directly utilizing these


large scale models is computationally infeasible. Therefore, we have to train on proxy tasks for


speed. Yeah, not, not really representative in terms of how optimization interacts with the task.


The the Yeah, so that's, that's kind of my my comment right here. And one that I see like the


biggest weakness of this paper. Okay, so we went off to that. And I would say we jump now into the


results. So the results here are the following. So here they compare with various handcrafted


optimizers, right? And it's a bit of a weird thing to let me just say this, this task is a very big


and very, very hard engineering tasks, because all of these tasks have to implement them, then


their loss or of different scales, you have to take care of that, and so on. So this is considerable


engineering effort. And it's like, I don't, I don't want to diss the work, I just kind of want to


point out where the limits are, in terms of where they might not have pointed it out so much. So


here they compare two different things. The top ones are algorithms that have like a fixed learning


rate, it's like, whatever in for Adam, like I suggest your three minus four, if that doesn't


work, at least a little bit, you're screwed, right? So you take that. So one trial, then you might


want to use Adam, but you might want to kind of search over the learning rate. So they do 14 trials


to search over for a good learning rate in Adam. And it goes on until like this, this here is 2000


trials, trying out different parameter combinations. While their optimizer, their learned optimizer,


only ever has one trial, because it's it's learned, it has no hyper parameters. And that's


one thing they point out that once they have learned their optimizer, it itself has no hyper


parameters, it you can you I mean, it can't it's a learned function, right? So there's nothing to search over and therefore


that's a that's a, you know, something you save. So you can see that if it's over this middle line,


the learned optimizer improves over the other optimizer for train and test sets in solid and in


shaded. You can see for most things, there is a bit of a movement to the right, except in these,


you know, very, very grid searchy things. So if you do grid search heavily, and you have lots of


parameters to tune, it seems you can outperform this thing, but it can outperform things where you


do not grid search, at least on these kinds of tasks, which is pretty cool. To say it does use


more memory. And I don't know exactly if it uses more time, it certainly uses like five times as


much memory as Adam, I think they say, yeah, time, I don't know, Adam is doing considerable amount of


work as well. So don't underestimate that compared to like one LSTM forward pass. They analyze what


their learned optimizer remember, this is one learned optimizer out of all these, like they have


one data set, they end up with one learned optimizer. And now they look at it, and they feed


this loss function right here, x minus y squared. If you look at the trajectories of the atom


optimizer, if you like start here, it'll go this this way, if you start here, it'll go this way,


of course, because this whole line here is a global optimum of this function. So Adam seems


to be doing something sensible. And in fact, I've tried them in a little collab, all of the


classic algorithms do this. However, the learned optimizer does something else, namely it pulls


towards zero zero, right, it pulls towards kind of the origin. So they claim that this optimizer


has learned something like implicit regularization, which does make sense, right? This optimizer is


optimized for giving as good of a validation loss as possible. Okay, now what do we know,


especially about small tasks, small data set, small architectures on on deep learning, what do


we know about the validation loss is that a little bit of regularization might be a good idea, because


overfitting in these regimes is still a problem. So it makes sense that something that is trained


to optimize for as low validation loss as possible, will learn to implicitly regularize the


parameters, right, I think that's, it's sensible. And they analyze this right here. And they show


that this optimizer has in fact, learned by itself to kind of pull the weights towards this point


zero. That's one take on it. The other take on it could be, it could be that simply in the tasks,


it's given setting most weights close to zero was actually just a good idea, per se. And maybe the


scale right here, or the shape of the loss function is too broad for this, and it pulls it towards zero


for other reasons. Ultimately, we can't know it seems though that the explanation is somewhat


plausible. I have to say there's one exception, the atom w. So atom w optimizer will explicitly


do the same thing. So if you start with atom w here, let's do that in a different color,


it will kind of go towards or yeah, depending on the step size, it can go like this, or it can go


like this, it will pull towards zero because it also has this kind of built in. So it's cool to


see that the learned optimizer has learned this though, in a chapter titled understanding optimizer


behavior, I would expect honestly, something more interesting than like clearly we have already come


up with with this in atom w. And clearly, the notion that we should kind of pull weights towards


zero, and that might be some sort of a good idea as a regularization isn't new to humans, right?


What I would have expected here is that they say, wow, our learned optimizer has now learned kind of


a complex but sensible way to deal with steepness changes in the landscape, or something like this,


that that is not achievable, or not easily achievable by kind of these classic algorithms,


it's more complex, but it makes sense. That's what I want to learn optimizer for, I don't want to


learn the optimizer to tell me, well, maybe you should like, add a bit of the norm to the loss


like gee, thanks. So yeah, again, they don't make claims about superior behavior of their optimizer,


but still, that's kind of what I would expect from a learned function. Again, if you look at the


generalization along different things, you see the the gray band here is where the up where the


training tasks lie in terms of the number of hidden units batch size and data set size. And


they say, sometimes our learned optimizer, which is in red, generalizes, like, yeah, sometimes it


does, but sometimes it just like screws up completely. And more often than not, it seems like


here, here, okay, here, it's better, but then here, it's worse. So I would not yet take this off the


shelf, though, I agree, it has some it has some promising value. Lastly, they say, okay, now we've


we've done this on all these small models, let's go let's go bigger. And bigger for them actually


means a small resnet on c410, which is like 14 layer resnet, and a small resnet on resized image


net. So these are still small things. And I don't know exactly why they can only once they have the


optimizer, why they can only feed these maybe because the LSTM itself also has like an internal


memory constraint when you have to feed in all of the weights of the network. However, look at this.


So this is c410, right? This is c410 on a resnet resnet. So this is fairly big, but you can see


Adam and momentum, they overfit. So here's the training loss, I'm gonna guess this is the


validation loss, they overfit, while the learned optimizer Wow, it doesn't overfit. But you see,


so first of all, it ends up here, okay, ends up here. When Adam and momentum were here,


their validation loss was here, which is pretty much where this ends up. So better.


And then you can make two claims, you can say this is because it's whatever implicitly regularizing,


but also you can say, this is because it's crap, right? It like it doesn't actually manage,


at least your optimizer should be able to get the training loss down, right? If if like any optimizer,


I get it, they say it's implicitly regularizing, but no, like, why? Like, I'd rather have explicit


regularization, but have an optimizer that actually gets the training loss down as much as I


want it. If I run it longer, I don't care about overfitting, it should peg down the training loss.


And this one doesn't do it. I think the explanation here isn't that it's super duper regularizing here,


it's just crap. And again, not to say that the paper is crap, but the learned function they get


isn't as good as Adam or momentum. Here, the same thing on a bigger this is ImageNet on a ResNet on


a bigger ResNet, I believe. And you can see that, yeah, you maybe can say that the learned optimizer


is on par with the others. But you see a trend, right? You see the trend that this, it gets so


when it's small, right? Small problems, the learned optimizer here outperforms. Okay, when it's a bit


bigger problems, the learned optimizer is still outperforms in validation loss. When it's even


bigger, the learned optimizer is the same size, right? And here you can see, if you grid search,


you can outperform the learned optimizer 3e minus four. Look at that. Look at that. It's like jackpot.


So this high suspension is if you go to even higher problems, right, then this learned optimizer will


just get worse and worse and worse. And this is the ultimate dichotomy in this paper. It says, look,


there are no hyper parameters and our learned optimizer, you don't have to do grid search. Well,


where can I do grid search on small problems? Where can't I do grid search on big problems? Where does


this learned optimizer work on small problems? I don't care if I don't, if I if I can or can't do


grid search on small problems, I care about big problems, which have fundamentally different


optimization properties than small models. So the last experiment here is where they take this


optimizer, this learned optimizer, and they use it to train itself. So they train it once and then


they, you know, apply it to itself, like the analogy is the compiler that can compile itself.


So you can see that yeah, at the beginning, it's kind of faster. But then it kind of flattens out.


And you can see that it can't train itself, right? That's the answer. Because it doesn't matter,


like this, this part here, except in very in limited circumstances, where you want to like train


to okay performance really fast. It doesn't matter if it doesn't end up in the same place, right? And


you can clearly see here, it's not going to end up in the same place. I'm going to show you the


full graph in a second. But even from that, you can see that it cannot train itself. It in fact,


Adam can train it sell it this optimizer better than it can train itself. And this, you know, that


yeah, just take it take that for for what it is. They have a full plot, like the longer plot in the


appendix right here. And where is it? Here. So, you know, you decide if this algorithm can be used


to train itself or not. I get it is pixelated right now, it's gonna load in a second, but you can see


all right, so the as I said, there is this this giant. Yeah, here, there you go. This this pseudo


code in this paper right here in the appendix is supposed to be helpful, I guess. But yeah, so what


it actually shows is how is like their variables and how they interact. And again, I find it's


correct what they when they say there are no hyper parameters once you've trained the optimizers,


but gee, are there a giant amount of hyper parameters in actually training that learned


optimizer. So just deciding which features go into that. And then so you you have whatever your


your your embeddings this list like it like okay, there are no hyper parameters in this procedure,


I get it. I'm a bit hyperbolic here. But there are no hyper parameters, except for, you know,


this list, the fact that you use a sine function. These gradient clipping values right here, this


clipping thing right here, the fact that you use a square root right here. Whatever you scale that


by this constant right here, this thing, the fact that you use log apps here. You can have all kinds


of things. There are not many hyper parameters right here. But it goes on right the G norm again,


we clip by something that is completely arbitrary. You can you can see that the architecture Oh,


another clipping value that is just set to five. The the arbitrariness of how you train this


optimizer itself is is is riddled with hyper parameters. And I get it the sense is that this


has own has to be done once. But given the result, I feel that this Yeah, I think that


uh, yeah, there's lots of room and I feel whatever you input into these whatever rolling features


there are, has is going to have a giant amount of influence over the over the what comes out


over the optimizer comes which is again is something they admit, right? So much code in this.


Yeah. Okay, lastly, let's go to the broader impact statement, which I find to be


amusing for a simple reason. So the broader impact statement, what is it supposed to do I


maintain that what it's supposed to do is you I don't agree that these things have to be in but


if you want to put one in and the way that the people who require it frame it is you think about


your method, the thing you have suggested, and you think about the ethical societal implications of


that and you really think about the good and the bad implications of this. And my meme is is the


broader impact statement is technology good technology bad technology biased. And I say good


bad biased because you want to think about what's good you want to think about what's bad and then


there is it's really in fashion to say that everything is biased and of course your model is


as a result also biased or your method or whatnot. This is a fashion at the moment expect this maybe


to go away in a couple of years. The other thing part of the meme is the technology part. So


I say technology because what people usually do is they've just presented a method. They don't want


to trash it right they like you you're not going to say my method is potentially bad what you want


to say is you're going to make it easy for yourself and say well my method is part of machine


learning or if you if you have something for optimizing GANs you say well GANs can be used for


GANs can be used for good and bad and are biased right so you you make it both easier for yourself


and you take yourself out of the crosshairs by simply going one or two layers up and the ultimate


layer up of course is just a statement technology. So I intended this to be a meme until I read


improving technology to do machine learning will accelerate its impact for better or worse.


We believe machine learning technologies will be beneficial to humanity on the whole


that's improving the ability to optimize models are moving towards like literally the meme has


become reality by them explicitly saying well this is part of technology and technology can


be good or bad. None of none of this is actually about their the specifics of their method like


in my mind if you are seriously doing this you should think about what differentiates my particular


paper from other papers and how does that particular differentiation manifest good or bad


as a consequence like how what are the consequences of that particular differentiation


however technology good technology bad technology is of course biased so


yeah that's that. All right I hope this was I think it's cool work right this is cool work


and you know Google is one of the very few places where this even can be done it is certainly it is


a paper that fully admits its limitations and that's also extremely cool and interesting and


it's written very unclear at times honestly but yeah that was my commentary I hope you enjoyed


this if you did share it out leave a comment tell me what you think including what you think


Jetson Nano 2GB from NVIDIA

What's going on everybody and welcome to a review of the 2 gigabyte Jetson Nano


Jetson Nano 2GB from NVIDIA

So the 2 gigabyte Nano is actually the little brother to the 4 gigabyte Nano the original one


Jetson Nano 2GB from NVIDIA

Which we also reviewed on this channel some time ago


Jetson Nano 2GB from NVIDIA

The only significant difference is this has 2 gigabytes of RAM


Jetson Nano 2GB from NVIDIA

This one has 4 the GPU is a 128 core Maxwell GPU


Jetson Nano 2GB from NVIDIA

And that is the same on both devices also both devices have I believe the exact same 4 core ARM CPU


Jetson Nano 2GB from NVIDIA

The other differences that I could see by looking at least at specifications online is that the 4 gigabyte Nano


Jetson Nano 2GB from NVIDIA

Apparently has 2 CSI connectors but when I look at this board I just see the one


Jetson Nano 2GB from NVIDIA

So I don't know I don't know if this is like an early board that I happened to get


Jetson Nano 2GB from NVIDIA

And they added another connector I don't know


Jetson Nano 2GB from NVIDIA

But I only see the one I'm not really sure you need two


Jetson Nano 2GB from NVIDIA

Two is kind of cool I guess so you could run like the two Raspberry Pi cameras and so on


Jetson Nano 2GB from NVIDIA

But if you need more cameras you can always use the USB ports


Jetson Nano 2GB from NVIDIA

So yeah so on this 4 gigabyte again there's just the one CSI connector that I can see


Jetson Nano 2GB from NVIDIA

But again that's all you really need I think


Jetson Nano 2GB from NVIDIA

The other difference is this board is actually powered by USB-C


Jetson Nano 2GB from NVIDIA

Does still have the micro there and also there's I think one fewer USB ports yeah


Jetson Nano 2GB from NVIDIA

This one also the 4 gigabyte has a display port I don't see that here this one just has HDMI


Jetson Nano 2GB from NVIDIA

Again I don't think that's a big deal at all I think most projects aren't going to need that


Jetson Nano 2GB from NVIDIA

And if you desperately need a display port I think you could just use HDMI to display port converter I don't know


Jetson Nano 2GB from NVIDIA

So I don't really think it was necessary so it's kind of cool that they've


Jetson Nano 2GB from NVIDIA

What they've done here really all they've done is kept that same powerhouse that 128 core Maxwell core GPU


Jetson Nano 2GB from NVIDIA

And instead just trimmed away some more things to the dev kit to make it more affordable to people


Jetson Nano 2GB from NVIDIA

So this was $99 and this one is $59


Jetson Nano 2GB from NVIDIA

And to be honest $59 for that 128 core Maxwell core GPU


Jetson Nano 2GB from NVIDIA

That's pretty cool I mean Maxwell core kind of old architecture


Jetson Nano 2GB from NVIDIA

But still quite powerful powerful enough to do things like object detection and so on


Jetson Nano 2GB from NVIDIA

And so actually what we did when we got the 4 gigabyte board was do object detection


Jetson Nano 2GB from NVIDIA

With USB camera on the board just to see what kind of you know specs could we get


Jetson Nano 2GB from NVIDIA

Or what kind of frames per second basically as a metric


Jetson Nano 2GB from NVIDIA

And then I tested that same exact thing the exact same program on the 2 gigabyte nano


Jetson Nano 2GB from NVIDIA

And unsurprisingly got basically identical results


Jetson Nano 2GB from NVIDIA

Obviously again the only kind of performance downgrade that we get here is the RAM


Jetson Nano 2GB from NVIDIA

Now 2 gigabytes of RAM is it's tight I'm not going to lie there


Jetson Nano 2GB from NVIDIA

But still in practice at least with the one test that we ran


Jetson Nano 2GB from NVIDIA

It still produced identical results obviously you can also use swap


Jetson Nano 2GB from NVIDIA

Which I had to do to build some packages on this device


Jetson Nano 2GB from NVIDIA

But I also had to use swap on the 4 gigabyte nano so


Jetson Nano 2GB from NVIDIA

Honestly at least for me it hasn't made a big difference


Jetson Nano 2GB from NVIDIA

But again I just know in my head that 2 gigs of RAM is tight


Jetson Nano 2GB from NVIDIA

So yeah so the board's actually pretty impressive


Jetson Nano 2GB from NVIDIA

The only gripe I have is actually the same gripe I had with the 4 gigabyte nano


Jetson Nano 2GB from NVIDIA

Which is the heatsink so I recognize maybe I mean well with the GPU


Jetson Nano 2GB from NVIDIA

I kind of think especially this GPU I think it's required that they send it with a heatsink


Jetson Nano 2GB from NVIDIA

I recognize they've sent it with a heatsink thank you Nvidia


Jetson Nano 2GB from NVIDIA

But recently I reviewed the Xavier Annex dev kit


Jetson Nano 2GB from NVIDIA

So this is one of the more pricey dev kits this one's like I want to say 384 core


Jetson Nano 2GB from NVIDIA

I can't remember off the top of my head if it's Pascal


Jetson Nano 2GB from NVIDIA

And I think it had 384 CUDA cores and then like 40 something Tensor Cores


Jetson Nano 2GB from NVIDIA

I can't remember perfectly definitely a more pricey device so cool it gets a nicer heatsink


Jetson Nano 2GB from NVIDIA

But I can't imagine the heatsink and fan combination is too expensive


Jetson Nano 2GB from NVIDIA

Again I recognize that this is supposed to be a trimmed down device


Jetson Nano 2GB from NVIDIA

I respect that I like that goal of getting cheaper cheaper devices


Jetson Nano 2GB from NVIDIA

Especially deep learning devices you literally can do deep learning on this device


Jetson Nano 2GB from NVIDIA

It's a $59 device you can do deep learning out of the box


Jetson Nano 2GB from NVIDIA

But you can only do deep learning out of the box for about 10 minutes before this device will overheat


Jetson Nano 2GB from NVIDIA

Now on the 4 gigabyte nano I actually wound up corrupt like basically it overheated


Jetson Nano 2GB from NVIDIA

And then like I don't know it just shut off and it turned out the SD card got corrupted


Jetson Nano 2GB from NVIDIA

I had to redo everything at least on this device


Jetson Nano 2GB from NVIDIA

It just displayed this message that it was throttling and everything else it was like okay


Jetson Nano 2GB from NVIDIA

But that's not good for the hardware


Jetson Nano 2GB from NVIDIA

So I wish that they would send it with one of these kind of heatsink and fan combos


Jetson Nano 2GB from NVIDIA

Because this works very well actually this device stays within a decent range of temperatures


Jetson Nano 2GB from NVIDIA

Whereas this device simply does not you can't use this dev kit without a fan


Jetson Nano 2GB from NVIDIA

And you can't even dev on it without a fan I think that's kind of a problem


Jetson Nano 2GB from NVIDIA

So I recognize you know if you use it in a project you would just add a fan to it and blah blah blah


Jetson Nano 2GB from NVIDIA

But I think that's the one thing I would change and just add a fan just can't be that bad


Jetson Nano 2GB from NVIDIA

Especially because they're adding people were wanting Wi-Fi on the board


Jetson Nano 2GB from NVIDIA

And there is no built-in Wi-Fi but apparently they're sending these out with Wi-Fi dongles to everybody


Jetson Nano 2GB from NVIDIA

And in my opinion rather than the Wi-Fi dongle send the fan


Jetson Nano 2GB from NVIDIA

We don't actually need not every project will need Wi-Fi like for me at least in my use case


Jetson Nano 2GB from NVIDIA

I recognize I'm not everybody but I just plug in the cable and I SSH to the device it's really simple


Jetson Nano 2GB from NVIDIA

And then if I need Wi-Fi well yeah I've got Wi-Fi dongles all over the place


Jetson Nano 2GB from NVIDIA

So if you needed Wi-Fi you could add it but not everybody will need the Wi-Fi dongle


Jetson Nano 2GB from NVIDIA

Everyone will need the fan so anyway I do wish they added the fan


Jetson Nano 2GB from NVIDIA

But other than that I think everything I think this board is really cool


Jetson Nano 2GB from NVIDIA

I hope Nvidia continues obviously to make you know cool powerful dev boards like this one


Jetson Nano 2GB from NVIDIA

Also the super high-end GPUs but I also kind of like that Nvidia is what I would think now


Jetson Nano 2GB from NVIDIA

Is kind of targeting the smaller kind of edge truly edge devices that pretty much almost everyone


Jetson Nano 2GB from NVIDIA

Can afford to kind of tinker with and play with I think that's really cool and bringing AI


Jetson Nano 2GB from NVIDIA

To these kind of like smaller cheaper boards like the Raspberry Pi only good things can come from


Jetson Nano 2GB from NVIDIA

That getting that in the hands of as many people as possible I think only good things can come


Jetson Nano 2GB from NVIDIA

So I really like that like I said my only complaint get a fan on there I hope the next one I review


Jetson Nano 2GB from NVIDIA

Has the fan I was so excited when I saw this this one I don't know how much it adds to the cost


Jetson Nano 2GB from NVIDIA

And this one's actually kind of really nice maybe you can make a cheaper version of it or something


Jetson Nano 2GB from NVIDIA

I don't know anyway really like it I'll put a link to Nvidia's page for this in the description


Jetson Nano 2GB from NVIDIA

You can get it off of Amazon and like all the other places it is currently I believe still in pre-order


Jetson Nano 2GB from NVIDIA

They ship late October so I don't even think we have an exact date yet but anyway pretty cool


Jetson Nano 2GB from NVIDIA

Anyway pretty cool technology I'm really happy with it I got to come up with I have some small projects


Jetson Nano 2GB from NVIDIA

I'd like to do as we complete the Neural Networks from Scratch book I've got some small projects


Jetson Nano 2GB from NVIDIA

And I'd like to kind of put these little dev boards more to use so maybe we'll see more of these


Jetson Nano 2GB from NVIDIA

Especially this one I feel like it's a sin to not be using this awesome board right here so


Jetson Nano 2GB from NVIDIA

So stay tuned hopefully we'll have some content on this if you've got some ideas that


Jetson Nano 2GB from NVIDIA

Of projects that you'd like to see with this let me know but I definitely have some ideas


Jetson Nano 2GB from NVIDIA

Whizzing around in my head so anyway that's all for now again like I said the board releases


Jetson Nano 2GB from NVIDIA

In late October for $59 which in my opinion is a steal just have a fan you literally could just


Jetson Nano 2GB from NVIDIA

Like put it on a like a stationary fan or something just have a fan but yeah pretty cool


Jetson Nano 2GB from NVIDIA


Dream to Control: Learning Behaviors by Latent Imagination

Hi there, today we're looking at dream to control learning behaviors by latent imagination by


Donnie Gerhofner, Timothy Lillicrop, Timmy, sorry, Jimmy Baugh and Muhammad Nerozi. This is a


reinforcement learning paper that iterates on a kind of a series of previous papers where the goal


is to learn a policy. In this case, they want to learn policies for these kind of continuous control


tasks. So these physics based robots, these hopper or walker types of tasks where you have to control


this robot, these joints in order to move forward. And so the goal is that you have multiple


observations as you do in in reinforcement learning. And from each observation, you need


to somehow come up with an action of what to do. And then that will give you the next observation,


as well as a reward, a reward for, you know, if you if your goal is to move this spider,


maybe the reward is proportional to how far you move. So your goal is to collect the maximum reward,


which would mean you have to move the spider as far as possible, simply by doing the correct


actions. The goal of this paper now is to do this by learning how the by learning to sort of plan


ahead in this latent space. So as you can see here, the way they do it is they take the observation,


and they feed it through an encoder. Now you can think of this as maybe a convolutional neural


network or something, anything that can work that can take an image as an input and give you a


hidden representation. So now this here is the hidden representation. From this hidden representation,


you can determine what the next action is going to be. And then you get a new observation. And


then again, you can feed that along with the last hidden state into a new hidden state. So this this


is already previous previous models do this a lot, right? You encode your observation, and you have


a sort of an, let's say, a recurrent neural network that incorporates all of the observations.


into hidden state along with the actions you take. And then you always decide on a next action to do.


So what does this model do differently? This model wants to do this all in hidden space. So


what this model wants to do is it wants to say, okay, I am here, I have this observation.


Now my encoder tells me that this is going to give me this hidden state. And now what it wants to do


is it wants to take in the action that it's doing. And without seeing the next observation, right,


it wants to predict it. Right, he wants to say, well, if I am here, and I do this action, what


might the action be, the action might be to put the joystick to the right, it will learn the hidden


state corresponding to the spider being a bit more to the right, right. So this is a bit more to the


right than it is right now. And it will need to do so a number of time steps into the future. And


it will kind of learn from its own imagination. So this this is what it will imagine into the future,


how the hidden states look. And then it will learn from that, instead of having to really do the


actions in the real world. Now, we've already looked at a number of papers, including something


like mu zero or I to a or something like this. This now is only is slightly different. So you can see


so what's different here, what is different is in mu zero, we had this, we use this latent model


in order to plan ahead, like in order to do our decision tree, planning ahead, and so on. This


model doesn't do this, this model still wants to come up with a single policy where you encode


your state, right? This is on the right is the final result, you encode your state, and then you


determine what your actions going to be. And you have your next state and so on. So the final goal