from the causality literature, and using them to then develop the new architectures and neural


Dynamic Inference with Neural Interpreters (w/ author interview)

networks. But then really, you know, that there's no causality test, right. So you can only pursue


some of the side benefits that you would expect from from a causal model. And this be this ability


of recomposing and using knowledge in without having to use a lot of examples, right, that's


really one of them. And it's inspired by the paper of Joshua. And so that here you you track


a little bit how tokens go through this computation graph. And I have my so just I love this line here,


just there are variations, but also similarities between samples in how their constituent set


elements are routed through the network, which I was like, well, what am I supposed to make of this?


Do you want to maybe tell a little bit what we can see in in this analysis? So these are three


different through I'll let you explain it. It's probably Yeah, it's a controversial plot. I think


I should take the fall. So the idea here is down to appendix. No. Yeah, you're sure was like, yeah,


why do we need this in the main paper? Like, does it but so I'm just saying what I see what I see


is that there appears to be you say you say the colored dots identify functions right same color


implies shared parameters. So here I see that the same function appears twice. Yeah, so these seem


to be this seems to be one of these scripts. And this seems to be another one of these scripts.


Exactly. And, and I'm so then the funk that colors tells me the functions are repeated. Okay,


I can I can accept that that's exactly as you said. And then I'm somehow supposed to read something


from the line from the connection line. So what you're supposed to read is that they're not all


the same. If you put three samples, you have three samples, they're routed differently to your


network, right. So I mean, it's kind of putting our money where our mouth is when we say that,


you know, like samples are routed differently through the network, you know, like, this is kind


of like a visualization of that, right? If you have three different samples, they're routed


differently to the network, here it is. And I think here it's important, like, you don't want to like


over interpret what these functions do on the visual domain, right? Because you don't like,


I mean, that's the power of deep learning, right? Like, that you have this this cascade of


computation, and maybe the result in between is not particularly interpretable. And you don't want


to read too much into it. And we also don't want to do that, not to over constrain the network. But


then it's important to really show like, if I give you concretely three images, yeah, is the


computation identical? Because if it is, then we're doing something wrong, right? Yes, exactly. That's


what I like, this is this is because I think in these works, it's always really important to sort


of try to check yourself, right? If you're really doing what you claim you're doing. And I see, for


example, here, the first the first sample has a lot of connections, like in this area, and the last


sample, not at all. And so, okay, that's, that's kind of what I thought I was supposed to see. But


I just wanted to check in with you. And this is really, let's say this is to address the haters


that say, all your architecture, it essentially it, you know, it just does the same thing. It's


just kind of, okay, I see. Cool. And another, another thing that I want to get to is your sort


of claims that now I have sort of this dynamic, I can add functions, I can remove functions, and so


on. Do you could you explain a little bit? How do you imagine that? How does that work? What does


what do you mean by add functions, remove functions? Is this during training? Can I do


something at inference time? Can I can I ramp up computation at inference time? What did you what


did you had in mind? And what did you do experimentally? So this is my I think this is my


favorite part of this model, like, you know, like, so the way you can think of functions is kind of


like, I don't know, I like to call it smart batteries. Right? So you can like install new


smarts into your model. And like, so let's say you pre train your so that like, one angle is that you


pre train your model on some data set, right? And then a test time or like an adaptation time, you


realize, okay, I want to kind of apply, there's a new data set that I want to apply my model to,


right. And so you can add like a new function that like, or a bunch of new functions that would kind


of nest itself in with the other existing functions and kind of synergize and work together in order


to solve the new problem, right. So you can kind of increase the capacity really easily, because


like nothing, the interpreter does not care how many functions there are. Right. So you know,


like, like the way it's parameterized, it's kind of, it does not really care. And so you can add


new functions. But what we also found, and I think this was one of the cool rebuttal,


rebuttal phase ideas was to, was to, you know, like, you can also remove functions at test time


without any additional training. So you'd like train with five functions and then at test time,


you just randomly drop three of them and the performance or like two of them and the performance


does not like immediately tank, you know, like it does not catastrophically fail, which is kind of


tells you that the function right here. Yeah, exactly. So it tells you that, you know, that


the system is kind of one dropped function is still fine. Yeah. Two, two, two is pushing it,


right? But, but it's still not, you know, like still not rock bottom, but okay. I mean,


three and four, I mean, there's nothing left after three and four, right. But, but, but yeah,


that's nice, right? Because like going back to like validating your hypothesis, right.


This is something that normally is not possible with distributed representation and the typical


neural network to use, right. And then, you know, it becomes important to check this, even if it,


you know, you can only remove one function. If you remove two, the performance is not great,


but just the fact that you can do it is something you really need to check when you propose


an architecture like this, because it's part of your hypothesis, right. And you need to design


an experiment to test it. Now, when you add and remove functions, do you retrain after you did


this or do you just rip them out and let and evaluate? So when we remove functions, we just


rip them out and evaluate nothing extra at inference time, no parameters are updated,


except the functions that are kind of needed. So there's nothing like no extra training at all.


The model is also not trained with function dropout, which is something one could arguably do,


but we don't do that. I mean, the model is trained with all functions and then it's still kind of,


yeah. I think it tells us that the functions are kind of a little bit autonomous and they


can kind of like, yeah, like they're not. Yeah. Somehow magically they happen to be so,


which is kind of cool. And when you add functions, let's say you have a bunch of pre-trained model,


then you need to sort of fine tune a little bit in order to, okay, in order to incorporate that.


Do you have extension ideas to this, maybe to make it fully modular that I can, you know,


grab together my functions of different models or is this anything you have on your radar?


Yeah, that would be kind of, yeah, I think that'd be nice. You know, like where you have like a


library where you can kind of pick out the books and like compose your own thing. That would be


nice. I mean, I don't fully know how to do it. I don't know. Maybe you guys have ideas.


I mean, it also goes into the direction of like architecture search. Sorry, go ahead.


Yeah. Sorry. I was just mentioning that it can also go in the direction of continual learning. So


basically you keep on adding new parameters as new concept comes in and keep adopting the


previous model, like without losing or catastrophically forgetting the previous


knowledge. So we have like some very preliminary, yeah, exactly. Sorry.


You could freeze like the codes to some of the functions, right? As you keep on adding new tasks


and potential, okay. And even like just having more like diversity in terms of data sets, right?


Yeah. Like you first have no train on these digits and then you start doing, I don't know,


any models and then you just keep adding to your collection of functions.


And you have the model, you said before you did it on some real data. Can we do classification


and reasoning? And could you just briefly tell us how do you assess that the model can do abstract


reasoning? It has to do with these matrices right here, right? Yeah. This is like a fun task,


which is for me personally is surprisingly difficult, honestly. You need to look at these


pictures very, very hard and detect patterns. And then you have a list of possible answers


and you need to decide which one completes the sequence. And apparently I'm not very smart,


so I cannot do this very well. But somehow, yeah, neural networks can do it. And it's a really


fun task because it requires you, it requires a network to really reason about abstract entities


and relate them across the different panels. So there are some logical rules that determines


whether a panel is the correct answer or not. And if you have access to the logical rules,


it's extremely easy. It's like some quantities are constant or it's the end of these shapes.


But if you don't have the right abstraction, it becomes a very difficult task. And it's really


nice that neural networks can do it. And especially they can then extrapolate to new


relations that have not been seen in the training set. And that's the, of course, the performance is


bad, at least compared to like the in-distribution performance. But the fact that it's not completely


random is pretty nice. Yeah. And do you have any way or idea? I haven't seen it in this paper,


but any idea how you could train a network on this and then sort of inspect these functions


and really see that these logical rules are somehow learned by these individual functions


in some way? Because essentially that's what you would hope happens, right? That somehow these


functions, they learn, these individual modules learn to represent these individual independent,


I'm going to guess the data set makes it clear that these are independent logical rules. Is there


any way you could think of to inspect this? Or is like, how would you go about this?


I mean, you can try to look at circuits, like the Antropic AI style, which I find absolutely


fascinating. But it also shows how much energy that takes. Right? So you have like really a team


working as soon as you have these distributed system. It's kind of like, and I'm not even sure


it even makes, I'm not sure to what extent these would make sense at all to us humans.


I'm not sure what we'll find there. Absolutely. I mean, even if at all we'll find the same


primitives or not, I mean, I don't know. And I think that's kind of what makes neural


networks so exciting. They might be solving it in a way that is totally own to the neural


network, totally autogonal to how we think about things. Right? And that's, but we kind of made it


at the same time. So we're making something and we don't know what it's doing, which is


fascinating. I think that's kind of like, that makes the whole deep learning journey worth it,


I think. So sorry, I went on a tangent, but long story short, I don't see an easy way,


except the whole like circuits line of analysis, which is kind of, yeah.


And we, cool. Excellent. Is there, is there another thing that you would like to mention


regarding experiments or, or regarding, regarding the paper itself? Cause we, yeah. Oh yeah. Yeah.


For a while. Yeah. Yeah. So, so like if you go up, I mean a little bit up, just the figure that


just kind of hiding. I think this is also super neat figure. Yeah. That one. So, so yeah. So like


we'll need to have the idea of kind of reducing the number of function iterations. And you know,


like instead of dropping functions, we just reduce the amount of compute at test time without any


extra training, like kind of like recurrent applications of the functions. Exactly. So we


were kind of squeezing in height and not in width. Like previously we were squeezing in width by


reducing the number of functions, but now we're squeezing in height and it works. And like I was


super surprised to like, like it, it caught us by surprise, I think like so that was, that was a


fantastic lead. So yeah, this shows essentially as you, you can, you can sort of, you train with


eight iteration, eight recurrent iterations, but you can get away at inference time doing seven or


six or even five and performance essentially stays sort of the same. It's only when you go down to


one or two that you really, you know, really drag the performance down. And think about it. Of course,


it's not something you would want to do with this particular architecture, but the idea of this


conditional compute and, and it's really nice because it, it allows you to, you train your,


your, your big library of functions, right? And now you have a task and, and, and inference time


is very important, right? So, okay, I'm not going to do eight iterations. I'm just going to do half


and of course it's going to be worse, but I don't need to retrain anything.


Or I want to add capacity. I plug in this new module or I have memory constraints and then I


cut half of them, right? This is, this is fun. And I would like to see more of it in neural networks.


Did you try more than you trained with? Like doing 16 when you trained with eight?


Actually, I think, yeah, we, yeah, can you guys hear me? Yeah, we did do that. And we expected


somewhat like, it might increase the accuracy. Not sure. I mean, it was unrealistic expectation


and it didn't do much. Like it dropped a bit, a little. So we did go 16.


So it's, it's, yeah. It's fair to say like, it's works the best when it is as trained,


but there is like, there is a, like a variation you can do. Yeah, I see.


But at the same time, I think it'll be fun to figure out ways of, you know, like breaking that


pattern, you know, like not having dropped down, but at least, you know, like at least saturate,


that would be nice. Yeah. Right. Yeah. It might be that if you train with only like two or three,


you might have maybe a bit of a chance because it seems like just great gauging from the plot,


right? It seems that at eight, even you train at eight, you already seem to be in kind of a regime


where it's, you know, it has, has done its work. Right. I enjoyed, yeah, I enjoyed, I enjoyed reading


this and I very much enjoyed having you here. So thank you. Thank you so much for being here. And


[Classic] Playing Atari with Deep Reinforcement Learning (Paper Explained)

Hi there, today we'll look at playing Atari with deep reinforcement learning by Vladimir Mni et al


of DeepMind. So this is another one of our series of impactful past papers. This paper right here


kicked off an entire revolution in reinforcement learning specifically, it sort of started the deep


reinforcement learning hype. Before that I reinforcement learning was kind of this weird


field of Markov decision processes and so on. Now I know there were successes and all and


stuff was happening. But this really made a lot of waves because it brought the power of deep


neural networks to reinforcement learning and with a pretty simple application of convolutional


networks managed to solve these reinforcement learning games where previous algorithms really


couldn't either or were heavily reliant on hand engineered features. So we'll take a look here and


what people did back then, what was the state of the art and what they are telling us about that,


kind of set it in relation to today. Alright, if you do like papers like this, commentary like this,


share it out, leave a like and tell me in the comments what you think. So let's dive in. They


say we present the first deep learning model to successfully learn control policies directly


from high dimensional sensory input using reinforcement learning. The model is a


convolutional neural network trained with a variant of Q learning, whose input is raw pixels


and whose output is a value value function estimating future rewards. We apply our method


to seven Atari 2600 games from the arcade learning environment with no adjustment of the architecture


or learning algorithm. We find that it outperforms all approaches on six of the games and surpasses


a human expert on three of them. So there's a lot packed into this. First of all, I wanted to


recognize the absolute LaTeX savagery right here. Yeah, you know, just just something. I'm not I'm


not OCD about that kind of stuff. I think we should ditch LaTeX honestly. But so there's a


lot of information packed in this abstract right here. So they say this is the first deep learning


model to learn control policies. So that's the task of these reinforcement learning algorithms


directly from high dimensional sensory input using using reinforcement learning. So what do they mean


this are called learning environment. If you don't know what it is, it's basically these old games


right here, you can kind of emulate them and run them. And the inputs are always sort of the same.


So you have one joystick, I believe. So you have kind of this joystick and it can go into various


directions like left, right, up, down, and then also the intermediate directions. And then you


have a I think also a button that you can push. And that gives you a total of some somewhere around


16 or 20 actions in each of the games. So the good thing about this environment is that the actions


are always the same. But of course, they mean different things in different games. So the


games here, you know, for example, pong or this breakout, the these games are kind of really low


pixel games, as you can see, and they come in form of an image, right. So this is an image, this is


like 180 pixels, and this is like 150 pixels. And the task here is to learn a policy, which means


which buttons and directions you need to push, depending on the observation right here, these


pixels and achieve the maximum amount of reward. So reward is given in each game differently as


well. For example, in this pong game, the reward is every time you score kind of a goal against


your opponent. In breakout, you get a reward every time you manage to hit or kill a one of these


blocks and so on. So reward is different, but your objective is always to maximize the reward


in in a formal framework, you have an agent and an environment. And the environment would always


give you an observation, which in this case, the observation is one of these images. And the agent


will give back an action. So the action in this case, would be the which button to press or which


direction to move the joystick into. And then the environment would get back give back a reward. So


the reward is could be, you know, it could be this, you scored a goal. So that's zero most of the


time, and sometimes it's one or, or nine, or it could be like how long you're alive and so on. This


is very, very, very variable. So the difficulty of reinforcement learning very often is that these


episodes can go for a while. So this whole process here will repeat over time. And the this can go on


this can go on for hundreds of steps or 1000s of steps until you're done, you know, playing a game


like this, like this right here. And the reward can be very sparse. So you might only get a reward


at the very end of the game. Sometimes, most often in these games, you get one in between, but still


there can be multiple time steps where you don't have a reward. And your task is to figure out


the actions were the good ones. This is known as the credit assignment problem. And to do the credit


assignment problem just from pixels alone, that was unheard of at the time this paper came out.


That's why they say we are the first deep learning model to successfully learn the directly from


high dimensional sensory input. Okay, so the power of deep learning, they argue is that a deep neural


network, a convolutional neural network can extract these high level features by itself.


However, at this time, people only knew that it could do so for supervised learning, basically,


for every input of an image, you had a label. And that's how you train these convolutional neural


network. Here, it's very different here, you will get maybe 1000 of those images. And you'll simply


say, well, you got a score of 1100. And somehow, you need to figure out which ones of these were


the ones that gave you the good score, and how to generalize that. So there are various difficulties


here to apply convolutional neural networks to this problem. And they have detail right here,


how they did it. So they say the model is a convolutional neural network, which, you know,


have been demonstrated. So this is after things like Alex net, though before res net,


trained with a variant of Q learning, whose input is raw pixels, and whose output is a value function


estimating the future. So we'll get into Q learning, Q learning is a reinforcement learning algorithm


that has been around, you know, for a long time, but just not combined with deep neural networks.


And yeah, then they say, the last cool thing here is that they apply them to seven games with no


adjustment of the architecture or learning algorithm. So that they apply the same algorithm,


the same hyper parameters to all of the seven games. And the model learns all of the seven


games not as a single model, but as seven different models. However, they all have the


same hyper parameters, so they don't need to tune them, which is an additional benefit like this


would have been a cool paper, even if they had to sort of tune the algorithm to each of the seven


games. But, you know, they didn't, which makes it even more impressive. It's kind of their point


here that these that this reinforcement deep reinforcement learning can be some sort of a


general learning mechanism. Of course, you know, later, there has been like giant amount of


development since this. And people have come up with all kinds of giant architectures and


whatever corrections of policy corrections seem to real continuous control. This is, this is most


of it is a derivation of this work right here. And this work, it reads surprisingly simple, I have to


say, and it's almost like they had little idea what problems were to be tackled later in RL,


because it kind of reads like, you know, we have this thing and you can learn general things. So


yeah, that was, we're still not done with reinforcement learning, I guess we we've just


started. Okay, so in a bit of more formal setting, what you will have in reinforcement learning are


these rewards, right? So at each time step, you perform an action, you're in a state. So the,


we said you get back an observation, and we call the observation the state. Now these two things


aren't a sorry, these two things aren't entirely the same thing. So the observation is what you


get back directly from the environment. And then the state it can be something more like if you


remember something from the last observation, that can be part of your state, the state is


basically what you base your decision on. And the observation is the pure thing you get from the


environment. Now, in this case, they do some processing, but essentially, will will regard


them as the same thing. So the state is what you see of the environment. Then in each of these


steps, you perform an action, we'll call that this a thing right here. And in each step, you also get


a reward for the last action that you took. And the reward is going to be lowercase r. Now, what


you want to do formally is you want to maximize the the reward that you get in time t. Sorry,


you want to, that's the reward you get at time t prime, you what you want to do. If you play an


episode, you're here and you perform an action, you go here, you perform an action, you go here,


you perform an action, you go here, and then your episode is done. For each action, you'll get a


reward reward one reward two reward three, what you want to do is you want to maximize the total


the sum of all of these rewards. So over the course of your episode, you want to collect as


much reward as you can. There is a discount factor right here, which is sort of saying that


rewards that are very far in the future, they're not as important as rewards right now, however,


you can set this to one if you want. So you see you want to maximize the future the sum of future


rewards, which is this thing right here. Okay. So how do you how do you do this, there are two


main methods in reinforcement learning. The first one is called a policy gradient method. And very


briefly, a policy we'll call it pi takes in a state s. And it gives you back an action. Okay.


And I mean, this is the same for for Q learning. But in a policy gradient method, which is not


this paper, but it's like a little bit easier to understand, I believe, will simply say, well,


we'll simply train a neural network to do that. Right. And there is this, there's this policy


gradient trick where you can back propagate, even though the reward isn't back propagatable, and so


on. You simply say, we'll learn a neural network to give us the action that's best, right. So we'll


have a neural network, the state goes in neural network, and then we'll just have as many outputs


as there are actions like action one, action, two, action, three, action four, and we'll treat it as


a classification problem, right. So you simply train the network to pick the action that is best


in this case, and you are irregardless of how you know which action is best right now. In Q


learning, you do something else, namely, you train this thing called the Q function. The Q function


is a function that takes in a state and an action, and it gives you what the reward is going to be


in the future, if you are in this state and perform this action. Okay, so you are in a given state,


right, and you have three actions at your proposal. Okay, you have action, one, action, two, and action


three, and you are in state s. You what you would do is you would call the Q function if you had a


perfect Q function, it could give you the reward, you would call the Q function three times, you


would call the Q function first with a one, you say what's Q of s and a one, and the Q function


would maybe say that seven. And here you say what's the Q function of s and a two, and it will


maybe say that's four. And here, same thing for a three, and it may be say that's one, then you


would know aha, if I take action one, my reward from not only the reward for this step, but my


reward from here on until the end of the episode is going to be seven. That is if you had a perfect


Q function. Now the Q function is always of course conditioned on a policy right here. So there's


what it basically says if I take action a one right now, and after that I follow policy pi,


then I'm going to get the reward of seven. It's a bit of a multi layered reasoning approach.


But ultimately, you don't have to you don't have to worry much about this being this being


conditioned on a policy. Ultimately, the Q function says if you take this action right now,


what will the reward be for the entire rest of the episode? Okay, so if you had a perfect Q function,


you could simply ask it about all the actions as we did here, and then pick the action with the


highest number, okay. And then you're guaranteed because, you know, there could be a situation


where, you know, your your reward in a single step is going to be very high here, like 100. And here


00. Okay, and you would be tempted to take that action right here. But But after that, it's just


going to be 00. So your total reward is going to be 100. But here, even though it's zero now,


it could be that after that, it's 50, and then 40, and then 2000, and so on. So your total reward


is going to be much, much more. If you were simply to train a function that tell you what's the


reward in the next step, then you would lose because that function would not be able to look


ahead sufficiently. What we're trying to do with the Q function is we're trying to predict to


train a function that will tell us not only what's the reward in the next step, but what is the


reward in all the steps to come from here, right? Of course, conditioned on all the decisions we


make in the future, but that's this policy pi right here. I hope it's somewhat clear what it


I hope it's somewhat clear what a Q function is. Interestingly, we can take the same network


architecture for this. So what you would do naively is you would build a neural network


where you say, okay, a Q function takes a state and an action. So I'll put those into a neural


network. And then outcomes, this estimation of the reward, which we usually call Q. So that the value


Q is this estimation of the future reward if you take this action in this state. Now, the


disadvantage here is that we have to call this neural network once for each action in every state


that we're in. So that's like, if there's 10 actions, that's like 10 forward passes. What we


could do is we could simply take the same neural network we had for our initial or very initial


policy method. And we use that and we simply input state and will train it to output the Q


for action one and the Q sorry for the state s and action one, the Q for the state s and action two,


and so on. So there is going to be this kind of shared encoder. And then that is basically going


to encode the state into a latent space and then classify for each of the actions how valuable this


particular action would be in that state. So the this here is called a deep Q network. Okay, it's


a network that takes in a state and gives you back the Q value. Now the problem right here is we,


you know, here we said if we had a perfect Q function, a Q function that was always right,


right, then the problem would be solved because we could just ask the Q function what to do.


Of course, we don't have a perfect Q function, we need to train it. So how do we train a Q function?


And the answer is surprisingly simple. So what you want to do is you want you are in this,


you're in this state. And you want to estimate right what your Q value is, you want to train your


Q network, what you can do is you can simply play an episode according to the Q function you have,


and you'll maybe play this episode right here, right? Like, you go here and you collect all of


this reward. So this entire thing now goes into your data set. And then you have a sample, you


know, I was here in this state s I took action one, and I got in total 2090 as a reward. So that


is going to be your labeled sample, right? Your labeled sample is going to be s, I was an s, I did


a one. And now I have I then got 2090 reward. Cool. And in the next episode, so you're going on


playing, and you maybe go down here, and then you get a next training example, I was in state s,


I so you keep restarting the episodes, so you can get into the same state multiple times,


I performed a three, and I got only 100 reward. So that's another training sample. So these training


samples right here, you can use to train your Q function. This is called online reinforcement


learning, you play the game at the same time as you train your neural network, and you use that


improved neural network to play more games. And with time, there is this well known there is a


there are theorems around Q learning that say, if you do that iteratively, then your Q function


will converge to the optimal Q functions under some assumptions, which of course not given if


this is a deep neural network, but you know, who cares? Yeah. So formally, your Q function,


as you can see right here, is going to be there is this Bellman recurrence kind of recurrence


property of the Q function. So if I am, if I am in a state s, and I'm wondering, what is my Q


of my state s and my an action a. And I said with respect to a policy, which the star policy is


going to be the policy where we always select the highest Q function. So we'll basically say,


we're in state s, we select action a. And after that, we'll just always select whatever the highest


scoring action is right, like right now, action a might not be the highest scoring, but we'll take a


right now. And after that the highest scoring, that's Q star, it's a Q function conditioned on


the policy where after we perform the first action, which is a, we'll take always the best one


according to the Q function. Right, that's right here. So we're in state s, we perform a, and s


prime is going to be the state that we are going to. So we're in s, we perform a, we get to s prime.


So in s prime, which is a function of your environment, we're always going to take the maximum action


and r is going to be the reward of the next step. So you can see this recurrence equation right here


that Q star can be framed in terms of Q star. So the Q star of this state is going to depend on the


Q star of the next state. And you can use that fact and you can, you know, prove that pretty, we've


already done it basically, you can use that fact to now train your neural network. So your neural


network loss function is going to be the following. It's going to say, look, this here is the Q


function for state s and action a. That's my and this is my neural network telling me how much


that's worth. And this is the label, right? So here you have to think in terms of back classic


supervised supervised learning, this here is going to be your f of x. And this here is going to be


your y. And we will take the squared loss between the two, except your input x is going to be which


state am I in and which action am I taking. And your label is going to be bootstrapped by your own


Q function. So your label is going to be the reward you got. Remember, this comes from a replay


buffer, we already played that game. And we already know what happened after we perform this


action. Right. And what happened is we got this reward, and we got into this state. So we can


simply ask our own Q function again, what's the best action to take in this state? And what reward


would we get? And then we have our label, right? So our label y is going to be Yeah, I was I was I


was pretty confused when I learned this the first time. So I'm going to assume some of you are


confused as well. So your Q function is supposed to tell you what's going to be the reward from


here until the end of the episode, okay, that you can decompose in the reward that you get from this


very next action, plus the sum from then to t plus one until the end of the episode, okay, so t prime,


so that's t prime equals t plus one. Alright, so it's pretty simple, the total reward from now until


the end, you can decompose in the reward now, plus the reward after that until the end. Now,


this here, we know we've played the episode, we know what happened. This here, we can simply ask


our Q function again, because we also know what state we got into. And this, as you can see,


is very much this, but just one step later. So we can simply ask our own Q function, which might be


imperfect, right? But it's certainly a good guess. We say, okay, this reward from now should be equal


to the reward we got plus whatever we reward we get later. And yes, you might be astounded by the


fact that we are using our own neural network, though be with the parameters one time step ago,


in order to produce our label. But that is exactly what these Q learning theorems are about. They


basically say under some assumptions, if you do this, and you iterate, then this will converge to


the optimal Q function. So as you can see right here, this is the this is the gradient of the loss.


It's astounding that back then they still wrote down the gradient of the loss, like almost no one


does this. Now you just say, put this into TensorFlow and go. Yeah, so they make some remarks


here, namely that this algorithm is model free, right? There's no model of the environment, you


simply learn a function that for each state tells you the Q value for each action. That's, that's all


everything, everything that the all the logic needs to be within the neural network itself.


So that's pretty cool. And they say it's also off policy. It learns about the greedy strategy while


following a behavior distribution that ensures adequate exploration of the state space. So while


while training, they do this epsilon greedy strategy that follows the greedy strategy,


which is where you always take the maximum with one minus epsilon selects a random action with


probability epsilon. So while you do your experience, you follow your Q function,


you always ask the Q function, what's the best thing to do right here? But you know, that's,


that gets you into too much of exploitation. So in epsilon amount of time, you want to do a bit


of exploration and just take a random action. Alright, so that's basically the algorithm. So


the algorithm is right here. And they have some tricks to get it to work. And the biggest trick


they got it to work is the so called replay buffer, this experience replay, because what


happens if you play a game of Atari, right, of pong specifically, then, you know, you have this


and you're here and your opponent is here and the ball is here. And then the next frame, you are


here again, your opponent might be a bit up, and the ball is here. Okay, and so on. So these samples


here, they are all very, very correlated, right, the ones after another, especially if you now


build mini batch, let's say our mini batch sizes to this mini batch has almost no variability in


it. So if you have something like batch norm or whatnot, this, this will be like terrible, because


these data samples are correlated. And we, in supervised learning, we make a pretty big deal


out of, you know, shuffling our data set and all the data points being IID and so on. So what they


say is, rather than using the data samples, as we collect them, we put them into a big, big buffer,


a big replay buffer. And from that replay buffer, we basically sample at random. Okay, so that means


that, you know, some samples can be used multiple times, other samples can be never sampled, because


there is a fixed size and the new ones will always kick out the oldest ones. So some samples might


not be used, some samples might be used twice or three times, we can also learn, you know, four


times as fast as we sample, and then every sample on average will be used four times. So this, this


experience replay proved very, very important for this algorithm to work. That's why they say


deep Q learning with experience replay. So they have this replay memory D right here to capacity


n. And you initialize your Q function with random weights as you do with a neural network. And then


you play these episodes for each episode, you start out with s one, the state one, and you do


pre processing. So in pre process, they have some more tricks where they downscale the image, they


concatenate four images in a row. Because sometimes in Atari get these flicker things. And


also, if you concatenate four things in a row, you for example, can tell it with in which direction


the ball is moving and so on. So give a little bit of history. So one sample technically would


be four frames, they also do sticky actions, and so on all of these things that you can find today


in these emulators that are almost default now like sticky actions, they invented right here.


So for the time steps within the episode, we want to, we've probability epsilon select a random


action. Otherwise, just ask your Q function, what should I do right here, give me the best action


in this particular state, then you would execute that action and observe a reward and the next


state. So the next image right here, you would set the next state to this transition. Okay, so


in the state there can be more as I said, there can be more than the image that you want to


as I said, there can be more than the image like the previous state and the action you took. But


right here, I believe it's like purely the current last four frames. And then you store that transition


in the replay buffer. After that you sample a random mini batch of transitions from the replay


buffer. So here you can see this here is where we correlate the inputs because if we simply were to


use our last transition for learning, then we would run into a problem. But right here,


we sample from that replay buffer. So this is going to be your input, this is going to be your


x for your supervised learning of the deep neural network, what's going to be your sorry, without


the reward, of course, what's going to be your y, your y, if you're at the end of the episode,


it's simply the reward that you got, because there's no more reward coming. However, if you're


not at the end, it's the reward that you got from this last step, plus all of the reward that you're


going to get in the future. Now you aren't in the future yet, but you can ask yourself, you can ask


your Q function, what that reward is most likely going to be. If your Q function gets better,


then this estimate gets better, and your labels get better, then your Q function gets better,


and so on in a big circle. And then you perform a gradient descent step on this L2 loss between the


label and your prediction. Note that there, if you are in a deep learning framework, there is like a


stop gradient on this label right here. So the back propagation only happens with respect to


this right here, which makes sense, right? So this is your x, this is your input, and f of x is


usually what we back propagate into. Okay, there's no notion yet of like a second Q network and so on,


which proved very valuable in the future of this paper. This paper simply applied kind of the most


basic version of this, and they simply got it to work. They just got deep neural networks to work


with reinforcement learning. And yeah, there's a big, a big chance that this was due to this


experience replay, which I believe they did not invent. I mean, the, the, this has, of course,


around the been around before, but they were the ones to realize and combine and do that. It's also


pretty interesting, the neural network that they actually used was like super duper small.


The input to the neural network consists of 84 by 84 by four image produced by this. So this is the


pre processing at the first hidden layer convolves 16 eight by eight filters with stride four with


the input image and applies the rectifier non linearity. So the relu, the second hidden layer


convolves 32 four by four filters with stride two again followed by rectifier non linearity.


The final layer hidden layer is fully connected and consists of 256 rectifier units. The output


layer is a fully connected linear layer with single output for each valid action. Number of


valid actions is very between four and 18 on the games we considered. Okay, you can see that neural


network is pretty small. It's two conv layers. And as was in fashion back then, you had like big


filters. So you know, big filters from like Alex net. Yeah, big filters, but fewer than today. So


today, the trend is more like deeper layers, more filters, but they are not as big. They're like


three by three filters today only. Yeah, pretty interesting how they did it back then. Interesting


also no max pooling and so on. So pretty cool. And then we have the


experiments. And here they go into experiments. So they show that their average reward in these


games is kind of noisy, but it improves over time, especially also if you look at the average


queue of the max action, it continuously goes up during training. So this is really a successful


training, especially this investigative experiment they did right here, where you can see one example


of how the queue function what the queue function says, remember, the queue function gives us the


whatever the future reward is going to be, okay, and here we always look at the max action. So in


this first frame, you can see this enemy had just appeared. And you can see that from here to here,


there's a spike in the queue value because you can shoot enemies. And that gives you reward


the a this is already so the enemy isn't shot yet, by the simple appearance of the enemy,


the queue function also like already jumps in value, because it anticipates a future reward,


right, then the the agent shoots. And you can see here, the shot is about to land at the enemy. And


that's when we're here. So this now the queue function is very sure that in the future, there's


going to be a high reward. But then once the once the enemy is shot, then there is no more enemy to


be shot. And the queue function drops drastically, because it doesn't see a future reward as being


as likely as at the beginning, when there was this new enemy to be shot. So that's, you know,


pretty interesting. And you can see pretty directly that there is a correlation between


what's happening in the game and this learned queue function. If you compare this to other


methods, and they really say that these other methods, most of them have some kind of very


special feature engineered, like, so their method just takes RGB, but the other methods recognize


that oh, in these Atari games, most of the time, you know, there are unique colors for the things.


So you know, the enemies are all like green, like, and they make unique channels for those green


enemies, or they even have handcrafted object detectors, and tell the algorithm where these


objects are. So the comparison really isn't fair. Yet, the DQN outperform these others, like almost


everywhere. And they also evaluated against a against a human. And I don't actually know,


they just say an expert human. I have no idea, maybe just put David silver in front of computers


like, okay, David, here you go. And you can you can, like what happened in pong? Like, come on,


David. But you can see there, there were still problems where the humans were vastly superior,


and they mainly attribute this to the difficulty of the problem. And it could also be because,


for example, in breakout them, there's this kind of the most famous example, where the agent kind


of figured out this strategy of shooting the ball, shooting like a hole into this wall that you have


to break, and then shooting the ball up here. So the ball bounces up and down. And basically, you


win. From then on, you just watch the ball go. And the agent does nothing anymore. So this DQ


networks figured out that strategy, and you need to pull it off very precisely, which of course,


the computer can do very well. So it sometimes achieves these super high scores by pulling


something off precisely. But in games where they say where you have to plan ahead for longer,


it kind of fails. And we know that this long planning was about to be a problem for years


to come. And it's still not solved. So still, Go Explore is highly controversial that can solve


these kind of long exploration games. And those are still games, right? So we are basically not,


we are very much further than they were in this paper. But also, we are basically nowhere yet.


Yeah, if I'm if I'm allowed to say that. So I enjoyed reading this paper. This is it's very,


it's very well written. If you somehow know how to think about reinforcement learning, like this,


this Q function, what the Q function means, and why you would learn it in this way. I find this


is not super well described, this kind of requires a bit of a knowledge of not of RL, but just of how


to think of RL. But apart from this, everything else is written incredibly well, easy, straightforward.


And this was just a nice work of its time. And I appreciate it for that. All right,


ACCEL: Evolving Curricula with Regret-Based Environment Design (Paper Review)

Check this out.


What you're seeing here is a bunch of agents that have all never seen this level before.


This level is in fact procedurally generated and the agents must somehow overcome the obstacles right here.


You can see there's stumps, there's gaps.


The green one is performing pretty well right here.


Coincidentally, the green one is also what we're going to look at in today's paper.


The idea here is, as I said, these agents have never seen these environments


and the environments are procedurally generated.


Every time I hit reset here, a different environment is created.


And also notably, on the right side right here, I have these sliders


with which I can control the different properties of the procedurally generated environments


such as how wide the gaps are, how many steps to the stairs there are.


And as I modify these, you can see the environments they get more and more challenging


as I slide these things to the right hand side.


Now, they get super challenging at some point.


And the question is, how do we train an agent using reinforcement learning


in order to be able to solve these challenging environments?


Because it's pretty clear that if I want an agent to solve an environment like this


and remember, it's a procedurally generated environment.


So I can't just train it on the same environment over and over and over again until it gets it.


If I want to train an agent to solve the family of environments that are very hard here,


it's almost impossible to do so using from scratch reinforcement learning


because there's just never any success of any of the agents like they never finish an episode,


they never get good reward, they always stumble at the first obstacle.


And so what's the way we, I still want the green one to actually make this.


Come on, green one, come on.


It's not going to make it right.


So the idea is that what we want to do is we want to develop a curriculum.


So a curriculum means that we're going to use this ability to create levels of different difficulties


to guide the agent to learn more, no, to learn more and more difficult environments.


So we're going to start with very easy environments, very flat environments,


not many gaps in them, not many stairs in them.


So fairly easy environments like this, and we use reinforcement learning


and try to teach the agent just to solve this level.


Now, most of them will do a fairly good job at that level.


As you can see, not too much of a problem.


Some stumble, some don't, but you know, this is solvable.


And then we will progressively as the agent gets better and better,


increase the difficulties of the level and using that, using that difficulty increase over time,


there is a chance that the agents, they learn more and more and more,


what they learn more and more to go and solve these levels.


So from scratch learning of the difficult environment, environments might not be possible.


However, there is a chance if we design a curriculum in the correct sequence of difficulties for the agents to learn.


This is not unlike humans learn in, you may have heard of this,


what you want to do is train in the zone of proximal development or something like this,


which essentially means that you want to always challenge yourself just outside of your current abilities.


And that's how you maximize your progress in learning.


That's the same idea that we have here with these evolving curricula over time.


So the paper we're going to look at is called evolving curricula with regret based environment design


by Jack Parker Holder and Minki Jiang and others, mainly by Meta AI,


but there's a bunch of collaborations with UC Berkeley, University of Oxford and yeah, I guess that's it.


So this paper combines the recent developments in regret based algorithms that go about making a curriculum


and evolution, which is another way that people go about this.


So the paper proposes to train a single agent, not a family of agents,


a single agent that is generally capable of solving all kinds of difficulties and levels


and to do that via an automated curriculum that is given by a teacher algorithm.


The teacher algorithm itself is not learned.


The teacher algorithm is actually defined by this schematic right here.


And all of this is regret based, which makes it independent of kind of domain specific heuristics.


So the goal of this algorithm right here is to have a general algorithm to design these curricula


without being reliant on essentially creating a new heuristics for all of the different tasks it needs to solve.


So we're going to look at it.


Here's a brief overview over the algorithm itself.


How does it do it?


How does it get an agent to learn step by step?


And the most difficult question is, you know, how fast do you increase with the difficulties of your levels?


Because if you increase not fast enough that you're essentially stuck in learning,


if you increase the difficulty too fast, you have the same problem again,


in that the agent will not be capable of keeping up.


So what you want to do is you want to have some sort of a level generator.


And that is what we just saw before in this web demo.


By the way, you can go look at try out this web demo for yourself at accelagent.kithub.io.


I'll obviously I'll link it in the description to this video.


But you want to have some sort of a level generator,


which is essentially the thing that I have here on the right.


I want to have the ability to create different levels.


This doesn't need to be parameterized like it is here.


For example, in this maze world that they portray right here, all I have is an empty room.


And then I have the ability to place blocks in it.


So every pixel can either be a wall or not a wall.


And that's it. That's a generator.


The generator can just place blocks and that's it.


There's no need for some sort of a slider here that controls the difficulty.


That's going to be done completely automatically as you'll see.


So once we have the generator,


we could already build some sort of a curriculum algorithm, right?


We could just sample different levels from the generator


and then just train the agent on all of them.


However, that wouldn't amount to much of a curriculum


as it would probably generate easy and hard levels all throughout each other.


And the agent would be able to solve the easy levels maybe a little bit


and then maybe a bit of the harder levels.


But if you don't sequence this correctly,


there's a big chance that you're going to fail mostly because


as the level design space gets higher and higher,


most levels are either going to fall in the too easy or way too hard section


and not a lot are going to be in that zone of proximal development


and therefore you don't have much of a learning signal.


So we need to somehow filter and curate these levels that we generate.


So we have a generator and the generator simply gives us the starting bunch of levels


and I believe you can also go to the generator within the algorithm and so on.


But imagine the generator gives us just a bunch of starting levels.


This is one of these starting levels.


I'm going to take a different color right here.


Otherwise, you won't see.


That's even worse. Thank you.


So the generator gives us a bunch of starting levels


and these go to the student.


Again, the student here, that's a single agent.


That is not a family of agents.


The evolutionary methods here are not with regard to the student,


but to the levels themselves.


So there's one student that trains on all the different levels.


So what we do is we simply evaluate, we ask,


we let the student run on this level and we see how well it does.


We're going to measure its regret.


So the regret of a student, we're going to get to that measure.


It's essentially an estimate of how far the student is away


from the optimal policy on that particular level.


And what we want to do is we want to strictly select for levels that have high regret.


So levels where the student is far away from the optimal policy


because those are the levels where the student can still learn something.


And if we do that correctly,


then this automatically sequences these levels in the sequence of difficulty


such that they're always just at the edge of what the student can do.


And you'll see how that works in a bit.


So we want to measure their regret and we have the buffer right here.


The buffer is where all the levels that we currently think are interesting


for the student to learn at reside.


This buffer is managed by the curator.


The curator is essentially just a bucket of levels that we think are interesting.


What we then do is we can replay those levels


so we can actually train the student on the levels.


But we also, if we just train the students on these levels,


that's not much of an interesting thing.


So we also need a way to update that buffer.


And the way we update the buffer is we select some of the levels


for editing. So some of the levels we think, okay, these are good levels,


but could we make them like just a bit more difficult


because the student can solve them now.


So what's a way to make them more difficult?


Then we send them through an editor.


And the editor, again, this can be pretty much anything.


So in our example up here, the editor could simply either place another block right here


or remove a block.


What is important is that different from the generator,


the generator just generates a new thing while the editor modifies the existing things.


And the assumption is that if I modify something that has a difficulty X,


right, then if I modify it to X hat,


then the difficulty of X hat will not be too much different.


So what I'm going to do is I'm at, let's say here is the student's starting point


and the student increases its ability round by round.


So maybe this is the zone that the student can solve right now.


And I select a level that is here.


So the student can just about solve it.


And then I modify that with the editor a little bit


and I maybe produce a produce different offspring like here, here, here and here.


So what I want to do is I want to select for the offspring


and here's where that's where the evolutionary method comes in.


I want to select for the offspring that will make progress for the students


so that the student just can't solve right now


and add that to the buffer of things where I do reinforcement learning on.


So with the editor, I create a bunch of different offspring for this level


as we see right here and I evaluate the student on them.


I measure the student's regret.


And if the regret is high, I put that back into the buffer.


So in this way, I always keep the buffer filled with levels


that the student just can't like it's just at the zone of where the student can solve them.


So if I now add the blue circled levels, obviously the next, you know,


I'm going to increase my ability to out here a little bit in this direction, right?


And then maybe here is another level that I modify with these two


and that increases the student's ability to here.


And then from these levels, I will again create offspring maybe to here and here.


Again, I will filter out the ones that become easier.


And so as you can see the student's abilities,


they will continually increase guided by this metric of this regret.


So that's the entire algorithm essentially.


You'll have one student that is generally capable


and the buffer right here will always contain levels that the student just can't


or just about can solve by measure of these regret and continuously editing.


Obviously, this doesn't work everywhere like there needs,


there's a lot of preconditions for this to work.


For example, you need to be able to have this level generator and level editor.


You need to be able to create levels of various difficulties,


not out of the box, but it like should be possible in principle.


There should be the possibility of creating a curriculum in the first place,


which is not possible for all the tasks,


especially with the condition that if I modify the problem a little bit,


like this thing right here, if I modify the problem a little bit,


then the difficulty should only be modified by a little bit.


Like that is not a given for many, many tasks.


However, if this is all given, then it becomes suddenly possible.


And of course, we run into all the problems of having a single student,


like there's catastrophic forgetting and so on,


but we don't worry about this right here.


As you might have seen previously that the Axel agent right here,


this the green agent, no matter kind of what the terrain is,


its strategy is always sort of the same.


So its strategy is always to kind of hold one leg out and bounce on the hind leg.


And okay, that might not have been.


So it will always, it's not going to make that,


it will always bounce on the hind leg.


Actually, most of them will do it.


Bounce on the hind leg and kind of wiggle the front leg.


And that's how it bridges gaps and stairs and ladders and so on.


Okay, most of them do that.


But you'll see that this is a problem of I think having a single,


single agent solve these things.


If you want a single agent to be solved, to solve all the environments,


that means that implicitly kind of one strategy or one set of strategies,


must be enough to solve all the environments,


which is also not a given for much of the world of reinforcement learning.


However, this can all be fixed.


So this was the overview.


Now let's dive into a little bit more into the algorithm itself.


Again, we have not yet, there's still a crucial element


and that is this regret that we haven't talked about yet.


But the algorithm in code looks like this.


I want to initialize a policy that this is the student policy pi


and this level buffer.


So the buffer is lambda, I guess lambda.


Okay, so I'm going to sample some initial levels


and I'll just assume that the initial levels here,


they're going to be mixed in difficulty.


So they're going to be some easy levels and some hard levels


and some levels that the student might just be able to solve out of the box


or not.


So, well, then we're going into a while loop,


the big like while not converged.


We're going to sample a replay decision


and the replay decision is essentially, it's a binary variable that tells me,


do I want to take a level from the buffer


or do I want to take a level from the, a new level from the generator?


Because if you only have initial levels in your buffer, right,


then you're kind of limited by the evolution of these levels.


So much unlike we have non-convex optimization problems in deep learning,


these landscapes of levels might be super duper non-convex.


And that's why if you just evolve a bunch of levels,


there is obviously the danger that you get,


that you sort of narrow, like you never,


so if you teach the agent to go like down a bunch of stairs


and you go ever more and more stairs, more and more stairs,


but the initial levels never had like a big cliff like this,


your agent will not be able to solve it even with this method


because no amount of adding stair steps will get you to the big cliff.


And that's why it's important to every now and then actually sample a level


from the level generator to bring some diversity in there


because that's what I see with this method,


it's probably pretty easy to teach yourself into a corner.


So if we have something from the level generator,


we collect the trajectory and it's important that we have two different modes right here.


We have the student in evaluation mode.


So every time that we have some level, some new level,


we first evaluate the student on it.


We want to know whether the student can actually solve it or not


on how well it can solve it.


So what do we do?


We compute the approximate regret.


We don't actually train on this level.


We just evaluate it and that is a property.


I think that reduces the signal to noise ratio tremendously.


We want to pre-filter what levels we train on.


We don't just want to train on all of them.


So this is a, interestingly enough, a method where even though we have the training data available,


it seems to be better if we filter the training data.


It's still good training data, right?


Any of these levels is good training data for reinforcement learning.


It's not like there's noisy data or the label is wrong or something,


but it seems to be quite important to accurately select the levels we want to train on.


So that is an interesting thing by itself.


But what you'll see in this algorithm is that they always will first evaluate a level,


determine whether the regret is high or whether it is in the zone of proximal development


and only then use that level to actually train the agent on.


That is interesting.


So we compute this regret and we add the level to the buffer.


So the level here is this theta.


So these are the parameters again here that we evolve.


We evolve two sets of parameters, the parameters of pi, which is the student's policy.


But that is just a very simple proximal policy optimization reinforcement learning algorithm right here.


We don't actually care what kind of RL algorithm it is as long as it can learn.


The interesting parameters here are the parameters of the levels.


And this could be the level itself in case of this maze or it could be the parameters.


No, actually it would be the level itself.


It needs to be an actual instantiation of the level,


not just the parameters that you enter into the generator unless the generator is deterministic.


And we only add it to the buffer if the score meets a threshold.


So that is where we filter out things where the regret is too low.


So only if it is a hard level for the student to solve, we put it into the buffer


and we'll get to how we actually filter out the levels that are too hard in a second.


So that's just if we decide we need a new level.


If we decide actually that we want to go into the buffer,


we're going to sample a level that we've previously added into the buffer.


And remember, we've determined that all of these are in the zone of proximal development.


We train, we collect the policy and we actually train.


So this is where we train.


We train on a level that we sampled from the buffer in the first place.


It's the only time we train the agent at all.


And then we are not done with this level yet.


What we do is we take the same level that we just sampled and we actually edit it.


So here edit to produce theta prime.


And the editing can be as I said anything as long as you can reasonably assume


that any edit will not distort the difficulty too much, right?


So it needs to distort the difficulty somewhat but not too much.


Again, we collect the trajectory. We do not train it.


We do not, we simply run the student on the new levels exact same way we did before.


We compute the regret and we add it to the buffer if the score meets a threshold.


Optionally update the editor using the score so that that can be the editor itself could be some sort of dynamic algorithm or not.


So that is the algorithm in a nutshell. It's pretty simple.


There is a buffer. I train on levels inside the buffer and only on levels that are inside the buffer.


How do levels get into the buffer? Two ways.


They can be sampled either from the level generator or they can be edited from levels that are already in the buffer.


However, both of them will only get into the buffer if they, if we evaluate the agent on them first,


compute its regret and if the regret is higher than some threshold, that's how we curate the buffer.


And that's it. That's the entire algorithm. So they have a bunch of experiments right here.


And that's, it's probably better to go back to the website to look at the experiments.


So, oh no, we need to look at what the regret is, obviously.


So regret is just, the way it's formulated right here, the regret is the difference between the expected rewards of two policy.


So if I have a, this here is regret. So the regret of theta and now you know theta is a level, right?


So the regret specific to a level would be, and here is policy one and policy two.


Now in this case, it's the current policy and the optimal policy, but you can see down here,


the regret can be defined over any two arbitrary policies.


It is simply the difference in the values of the two policies. And what's the value?


The value is the expected future reward.


And if I pose it like this, it's probably just the expected reward.


So the formulation right here, where I plug in the optimal policy would simply be, you know, what?


I have some sort of level, right? And I have my current agent right here.


And the agent expects to get some sort of reward, like maybe it gets onto here and then it crashes.


So that's a reward of, I don't know, 50.


And the optimal policy, if the level is solvable at all, could actually go to the end and solve it and get a reward of 100.


So my regret in this case would be 50.


And that is a good measure of how difficult the level is, or let's say how much you can still learn from that level.


Because if a level is too difficult, and that's the catch, if a level is too difficult,


then not even the optimal policy will be able to achieve much in that level.


And therefore, you know, why are you like, what point is it to go to that level and actually solve it?


Or if there is any stochasticity, if a level needs a lot of luck, right, then as well, the expected reward,


the expected future reward of the optimal policy will also be not super high.


So by selecting things that have high regret, meaning that have a high difference between the optimal policy and the current policy,


we select for levels that where the current student can still learn a lot of things.


So there's still headroom to learn.


Now, the optimal policy is obviously hard to compute because if we had it, we wouldn't have to solve the problem.


So that there is an approximation we need to do because we don't have access to the optimal policy.


And the approximation is this thing right here, which is called the positive value loss.


This is from previous work.


By the way, this work is essentially a combination of two previous works.


This PLR, I don't, it's okay.


I don't remember exactly right now what it stands for, but what PLR does is it also uses this regret objective,


but it simply applies it to randomly generated levels.


So it randomly generates and it just curates that random, those randomly generated levels.


And the other thing that it borrows from is evolutionary methods, which maintain,


the evolutionary methods always maintain a population and they do this sort of editing the population and then evaluating their fitness.


However, most of the evolutionary methods, they are very hand tailored things of what it means to be fit.


So the fitness function could be quite specific to a given environment.


And remember, we're not evolving the agents here, of which fitness would obviously just be like how well can you solve a level.


We're evolving the levels themselves.


So the idea of this paper right here is to simply use the regret as a fitness function and then curate the levels according to the regret.


So it brings in evolution into the PLR algorithm with regret being the fitness.


That's, I guess, formulated in two different ways.


So the positive value loss, let's unpack that real quick.


It stems from this thing right here, delta K, delta K is the TD error at time step T.


So if I'm in a level and I'm at some time, these are the time steps and the observations that I make through the time steps.


The TD error is I can compute after I've completed the episode.


So at each step, I've gotten some sort of reward.


Maybe my reward here is R1, my reward here is R2, R3, R4 and so on.


So in temporal difference learning, what I do is I always at each point in time, let's say I'm here, I want to estimate my future reward that I'm going to make.


And that would be my value function.


Right. So my value function tells me what the future reward will hold.


Now, I can estimate the reward one step into the future or two steps into the future or three steps and so on.


My temporal difference error is simply and if it's written like this, I think that's I'm not entirely sure if that's like a TD lumped or a TD one error.


But in general, what I can do is I can just predict all of my future rewards and the difference between what I predict my future rewards to be and what they actually are,


which I know after I've completed the episode.


That's my TD error. That's my temporal difference error.


I can use the temporal difference error to learn a value function because otherwise I'd have to learn the value function just from the rewards that I get.


And the TD error is a bit more of a smooth objective and I believe it converges to the same thing ultimately.


But you can reduce the variance a little bit under certain assumptions.


The TD error that we're interested in right here, it doesn't matter if the agent uses it to learn or not,


but the agent simply predicts the future rewards along the way as it solves the level.


After the level is completed, we compare that to the actual rewards that it got.


We calculate the difference of that and that becomes the TD error.


Then we sum up the TD error across from each time step. I can calculate a TD error, right?


So I can do that from each time step. If I'm at time step T, I can look ahead.


Okay, yeah, I can look ahead from each time step until the end and probably possibly the TD error could be looking from either from or to that particular time step.


That is not exactly specified. I would have to go and read this paper possibly or the PLR paper.


It's not super important. We can add that up.


Here are some discount factors that we use for that, but you can disregard these for now.


Essentially, it simply means, okay, from time step T on, you know, how wrong am I about the future?


And what we're going to do is we're going to apply a ReLU to that.


So essentially, we're going to cap it at zero, which means that I'm only going to be interested in wherever I under


or overestimate. Now, let's think about this. Wherever I overestimate.


So the TD error, as far as I know, is the value minus the reward. Correct me if that's a different way around, but it's what I estimate minus what it truly is.


Now, if this is high, it means that I completely overestimated my ability to achieve reward in this level.


And that could be, you know, a good level to train on. If I underestimated my ability to achieve reward,


then I'm going to guess that that level might be easier than I had anticipated.


So, but if I overestimated, that level might be harder than I anticipated.


And that's exactly the levels that I want to train at. So I'm going to cap that at zero.


I'm going to sum that up across all the time steps. And if this number is very high, it means that throughout the level,


I consistently overestimated my ability to make progress in this level to get reward.


And therefore, that level should go into the buffer. So this is the approximation to regret that we're going to use right here.


And now you have the entire algorithm. Generate levels, give them to the student, evaluate them, evaluate this measure.


Does the student under or overestimate its ability? If it overestimates its ability, put it into the buffer,


then take stuff from the buffer, train the student on it, give it to the editor, modify it, and evaluate the student again on it.


If the student overestimates its ability on the edited levels, put them back into the buffer and train on them. That's it.


You can also see a little bit why this doesn't necessarily suggest levels that are way too hard.


Because if you had a level that was way too hard, the student might even correctly estimate that it's not going to make a lot of progress there.


Because it's pretty easy to recognize that you're not going to make a lot of progress if the level is super duper hard.


Right. So the levels that this is going to select again is exactly the levels where the student thinks it should do well, but it doesn't really do well.


So let's look a bit into the experiments. The experiments, as I said, are best probably viewed on this website,


because they're a bit interactive. So what they first do is they come up with these lava grid levels and has the website crashed again.


So the lava grid levels, they're procedurally generated.


The agent must get to the goal while avoiding the lava grids.


And as the experiments show, these get progressively harder and harder.


They next go to these mazes and Excel starts from just empty rooms.


So they start from empty rooms. And up here, I believe you can see some of the generated levels by this algorithm.


And the website has indeed crashed. Let's refresh. So if we look at what levels it generates,


you can see that the levels are, they're fairly difficult, right? But they're also kind of random.


They don't really look like human levels. So you might be a bit doubtful of whether that's going to help in mazes that we typically know.


But you can clearly see the progress from the initially empty rooms to it filling up and to actually becoming harder and harder and harder.


And if you then evaluate these things on levels that humans have designed, there's this benchmark right here.


It will do pretty well, especially against these other methods that also do curriculum evolution of levels.


So especially things here like large corridors. So these are very difficult.


The agent only gets a little window around itself to view. It doesn't get an overview over the entire level.


And therefore, it needs to sort of keep in mind things that it did previously.


And that is a hard task. And they even, this is really cool.


What they do is they have the agent generalize, I believe, from 16 by 16 grids, which they train on, to this 51 by 51 grid.


And you can see that the agent, it kind of follows, it goes like left, always left.


And that works because this maze has no loops. At least I believe it has no loops.


So it in the end, it actually finds the goal. Why this is exactly 51 by 51?


I don't know. Maybe because the inside then is 50 by 50 or because that was just the largest maze that it worked on.


But it is astounding that it can sort of generalize to much, much larger things because in the small mazes,


it is conceivable that it could kind of keep all of its history and memory.


But here you can really see that it has learned to develop an actual algorithm for what it does.


So there is an algorithm like always go left. Yeah, pretty, I could, you know, watch forever.


Then they go on to these terrains. And again, the thing here is that without hand crafting fitness functions or anything like this,


just purely based on these regret measures, these levels, they continuously evolve,


which you can see right here in what directions the levels evolve.


So first steps are increased, then stair heights and so on.


And at the end, you have a generally capable agent. They compare this.


So they do some ablations. But interestingly, they compare this to Poet.


And Poet is an interesting algorithm because Poet trains a population of agents.


So Poet will always pair environments and agents and try to get the best achieving population of agents,


which leads to very specialized agents for very specialized types of environments.


So the comparison is not exactly accurate, but they do.


I believe they do show that their algorithm takes a lot less interactions, obviously, because it's only one student.


And Poet has an entire population of students.


And they also analyze over the course of training how their levels would fall into Poet's,


because Poet has a categorization of levels of which ones are easy and hard and so on.


And as you can see right here, it starts off with a lot of easy levels on the left and quite a bit of challenging levels,


but not very many very challenging or extremely challenging levels.


And as time progresses, you can see that at least a little bit, the proportion of easy levels,


it sort of takes a backseat and then the proportion of extremely challenging levels increases.


What is also interesting, at least for me, is that there's not a monotone,


monotonic development into the direction of challenging levels.


And that is what, you know, I believe maybe this might be a little bit of a sign of this catastrophic forgetting,


because this is only a single agent.


Essentially, if you train it into one direction, it might forget the other directions that exist.


And specifically, it might forget how to do easy levels because there's always a hill in the challenging levels.


It might fall over once it just encounters a flat plane.


I've actually seen this a bunch of times in the trial runs that I did on the website.


So it's pretty interesting to see that even though extremely challenging levels get added


and there are certainly more very challenging level than at the beginning and less easy levels,


it does not converge to only having extremely challenging levels.


So that is also interesting.


Here you can see a little bit of a comparison, notably the top row, a poet is a population based algorithm,


as you can see here, which is what makes it different here and not super-duper comparable.


Then the other ones are, so the PLR, as you can see,


it also uses the minimax regret strategy to curate levels.


However, there is no, it simply relies on random sampling from the generator,


whereas Excel uses the random sampling plus evolution,


which essentially means that it pairs the PLR algorithm with the poet algorithm.


And that appears to work quite well.


So that is all that I wanted to say on this work.


There's a lot more to say, but I hope that is being clarified in the interview with the authors.


What is a bit worrisome to me about this paper is just the fact that while they frame it as,


this is very general, this needs essentially no heuristics and so on.


I believe that is not entirely the case.


I believe there's a lot of domain knowledge that kind of gets sneaked inside.


For example, we need this threshold, right?


We need this threshold on the regret.


So there is a threshold.


Only if it hits the threshold, we put it into the buffer.


Like they criticize poet for filtering levels where the agent gets between 50 and 300 reward


and they kind of say, well, that's kind of really arbitrary and is really made for that level.


And I agree.


But then there is kind of a regret threshold, which is again,


that is kind of a hyper parameter that I'm going to guess that you have to tune.


And the same thing goes for, how do I edit these levels and so on?


I believe them that it can be an arbitrary editor.


But again, this is, it's very specific.


And I believe what is most specific here is just the choice of tasks that you go about.


Not every task.


And I would argue that very few tasks are actually lend themselves to this kind of evolution


because again, you need to be able to create a very smooth trajectory from easy


to hard where the same or similar strategies will solve all the different difficulties.


And in addition, you need also to be able for the editor to edit levels in such a way


that such a path can be created, right?


And you need to avoid the catastrophic forgetting.


You can't evolve into too many different things at the same time and so on.


But I do think it's a cool method and there's certainly applications and curriculum learning,


I think is one of the most interesting things that we can currently do because gone are the days of,


like you essentially shift some responsibility from the agent algorithm to the environment creation algorithm,


which I like, right?


Because we've seen scaling up of agents dramatically, drastically,


and maybe we can end up with a leaner agent if we shift some of that learning difficulty to the environment.


All right. That's what I had to say.


Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (Paper Explained)

Transformers are quickly coming for your favorite models. Yesterday, they replaced LSTM in NLP.


It used to be good at NLP, but we now have transformers. Think again. Today, we're going


to see that maybe in the near future, transformers will replace convolutions in image processing. So


this paper is a step in toward towards this direction, you just wonder what is it going to


be tomorrow? Maybe linear regression is going to be replaced just by giant transformers trained on


5000 TPUs? Who knows? We'll see. In any case, we're looking at axial deep lab standalone axial


attention for pan optic segmentation by Huiyue Wang, Yukun Chu, Bradley Green, Hartwig Adam,


Alan Yuel and Liangjie Chen of Johns Hopkins University and Google Research. So this paper


combines a bunch of techniques that have been introduced recently, to deal with attention in


problems where you would traditionally use a convolution. So in this particular case,


they deal with this problem of pan optic segmentation, which basically, you'll see,


you'll get an image, and there's a bunch of stuff on the image, like a cat here,


and a house right here. And you're supposed to color the pixels of the same object the same. So


you see all these pixels here are house, and then all these pixels, these pixels right here are cat,


and so on. And then there's also the background. So all these pixels right here, I know beautiful,


beautiful, beautiful, our background. So for this problem, it's kind of important that


you're very precise, first of all, so you can look at, you know, pixels or clusters of pixels,


and also that you take long range dependencies into account. Because if you, for example,


recognize that this is a house, and you recognize that here's a wall right here,


you might be able to much better classify what is wall over here and what isn't. Okay, so the kind


of long range dependencies play a role in these problems across images. And usually attention


mechanisms are pretty good for these long range dependencies. But they're also expensive. And


that's what this paper deals with. So they use this axial attention that has been introduced for


exactly resolving this problem in types of data like images or higher order tensors. And they also


combine this together with learned positional encodings, which we've also seen time and time


again, throughout the kind of transformer and attention literature. So the combination of axial


attention, these learn positional embeddings, allows them to replace the ResNet backbone that


usually is found in panoptic segmentation models with the with a standalone attention. So they


build models that are partial replace the convolutions with attention modules, or replace


them entirely. So the entire model is going to be just an attention model, so no more convolutions


in it. And they perform pretty well in classic tasks like they test on ImageNet classification,


they perform pretty well. And they achieve state of the art on some of these segmentation tasks.


So we'll go through the model right here. This is a very, very extensive paper in terms of


experimental evaluation. What I want to get into is mainly how the method works and show you what


their model looks like. So we'll go through it. And as always, let me know what you think in the


comments. And tell me if you liked it or not. Share it out if you did. Alright, so let's go


right. So they go over a very long list of prior work, which is, you know, pretty, pretty cool.


And here they say their contributions. So their contributions are four fold. First of all,


the proposed method is the first attempt to build standalone attention models with larger,


large or a global receptive field. And we'll see what that means. We propose position sensitive


attention layer that makes better use of positional information without adding much computational cost.


We show that axial attention works well not only as a standalone model on image classification,


but also as a backbone on pan optic segmentation, instance segmentation and semantic segmentation.


Maybe what I did before described before was instance or semantic segmentation and not pan


optic segmentation. Excuse me if that's the case. As you can see, it can be used for various various


image tasks. Lastly, our axial deep lab improved significantly over bottom up state of the art on


cocoa achieving comparable performance of two stage methods. We also surpassed previous state


of the art methods on mapillary vistas and city scapes. So these are various tasks, as I said,


and also what they don't mention here is that they perform fairly well on image net. In fact,


in the abstract, they formulate this as in particular, our model outperforms all existing


standalone self attention models on image net like that's, you know, that's a way to phrase it.


You just exclude all of the other models until you're the best outperforms all existing standalone


self attention models on image net. Yeah, I mean, that's good. I mean, there's something to be said


of comparing apples to apples. But you can also you can also go overboard if you want to make your


work look as good as possible. Of course, you know, everyone, everyone does that. And there's


no particular shame in it. Okay, so if we're going to build up our model right here, and the basic


element of this model is going to be this self attention mechanism. Now, quickly, because I know


you all know what it is, but very quickly, you want to perform this action right here,


over a region right here. So there is always a query. And now the subscripts here are going to


be important in this paper. Okay, so the query is at a given position, position O, and you can see


that's the O right here. That's the I'm going to call it the output, I guess that's what they said


as well. So the output position, you want to go over all of the input positions, and you want to


and you want to aggregate data from all of the input positions. So that's right here. And how do


you aggregate data by this softmax operator right here, you can see the key also has a P right here,


and the softmax is over the axis of P. So in particular case of the images, what does that mean?


If you have an image right here, it's made into pixels, okay, so you have pixels. Now a transformer


or gen in generally these attention models, what you can imagine is they always transform


a data point into a data point of the same dimensions. Now this doesn't have to be actually,


and I think one of the developments that is going to come in coming years or months or weeks,


maybe someone's already doing it is in fact, to play more with this with this arbitrary constraint


that we're imposing on ourselves, because it's not really clear that this is the best thing. But for


now, an attention layer is always transforming a data point here a four by four image into a data


point of the same size, also a four by four image right here. Now this is as I said, this is quite


simplified. But it is true in NLP, where we always transform our whatever 512 sequence, a token


sequence into a 512 token sequence, and it is true here. Now the output is going to be here on


the right. And the question always is, okay, so I'll go over these, these, these pixels right here.


And for every pixel, let's say for this pixel, I'm going to ask what data goes there, what's the


output of the layer at that particular pixel. And the output of the layer is going to be somehow


dependent on on the input right here. Now, if you know classic convolutional models, what the


classic convolutional model says, the output of this is going to be dependent on this region right


here, if it's like a three by three filter, okay, so you have this convolutional filter. And that


means that blue dot on the right is going to pay attention to, you know, its own location in the


input, plus everything around it, okay. And then every single data point here is going to do that.


So for example, the screen data point is going to pay attention to this region right here. Now


there's a border. So there's maybe some padding. But the question is always, where does the


information come from? And how is it aggregated? Okay, in a convolution layer, what happens in a


convolution layer, in a convolution layer, you simply have your filter, right, you have your


filter, and the filter has numbers in it, like three, and five, and eight, and so on. And what


you're going to do is you're going to take this region right here, this blue region of the lower


layer. And that's maybe that's also, you know, filled with numbers like seven, what's a good


number? Zero, zero is a nice number. And you're going to multiply those. And then you're going to


sum them up. And then you're going to put that on where the blue dot is, okay. So where does the


information come from in the convolution from around the location from around the output


location, but in the input, okay, so you go to the input at the same location as where you want the


output to be, you take the neighborhood, and there is a fixed a fixed scheme of aggregating the


neighborhood, okay, and then you sum, you multiply and you sum across it. In contrast to this, in a


fully attentional model, where does the information come from? Let's again, look at the blue dot. And


let's consider it fully attentional. Okay, where does the information come from? Everywhere,


anywhere, anywhere at all, okay, the information comes from everywhere. Now, how do I know how to


aggregate the information? So it's no longer in a neighborhood, how do I know how to aggregate the


information? That's also different. So two things are different. Now, in a convolution, I would have


another four by four grid here that's pre specified. But in the attention model, this here


is basically all filled with question marks, question mark, question mark, where what number


goes here? How do I in the end, I also do this multiply, and I sum it up, and I put it right here.


Okay. But how do these numbers come to be? Well, these numbers also come, these are dynamically


computed also from from the input. It's a bit special. But this is how attention works. Okay.


So every pixel gets to decide where information comes from, and how it is aggregated. It basically


it comes from anywhere. And how it is aggregated is dynamic depending on the pixel. If you don't


still don't understand it, maybe pay out to watch a video on attention itself, I happen to have made


one. But you can watch any one when you understand that you will understand the the extension here to


the image is the exact same thing as with the sequence except the pixels are basically one long


sequence in the image. Okay, so these would be a fully attentional model down here. Now, what's the


problem here? The problem is that pictures are pretty large. So even even something like MNIST,


which is like 28 by 28 is like 700 pixels plus I don't remember exactly, but it's like


about 700 pixels. And our big transformers now so BERT, a very famous transformer takes inputs


that are like 512 in length, and you already need pretty decent hardware to run this. And the


requirements on memory and compute scale quadratically with the input length. So already


with MNIST, you're in pretty, pretty shady territory. If you go up to something like image net,


which is like 225 by 225, you're, that's bad, right? That's not good. So you have to come up


with something else. So people have been playing around. The reason why I introduced it this way


is people have been playing around a bit with sort of coming up with an intermediate with a


compromise between the two. So the compromise that this paper here focuses on is going to be


is going to be a compromise where we you remember when I said where does the information for a given


pixel come from? And we said, okay, it can come from anywhere in the attention framework. And


that's good, because that allows us to make super long range connections. So any pixel can aggregate


information from any other pixel, and not even in a fixed way, but in a dynamic way. So depending


on the pixel value itself and the other values, it can decide how it wants to aggregate information.


That turns out to be expensive, right? Every pixel together with every pixel, well, that's quadratic.


Okay, so what do we do? We make a third method that's going to be a compromise. And the compromise


is going to be the following. The compromise is going to be all right, we still do the dynamic


aggregation, which means that we still do the attention thing. However, however, we're going to


restrict back to this neighborhood region of the convolution. So in this model, where does


information for the blue dot come from, it again comes from this neighborhood right here. And this


number, the size here is going to be called m. So it still comes from that m by m neighborhood. So


pixel can only aggregate information from its neighbors. But contrary to a convolution, how it


aggregates the information like this, what in convolution would be a kernel, the kernel is made


dynamically by the attention module. And it's made dynamically on a case by case basis. Okay, so we


restrict it to a neighborhood, multiply, sum it up, and then put it into the output. And we do


that for every pixel. Now it resembles much more a convolution, simply a convolution with this


dynamic with this dynamic matrix right here. And that's the starting point for this paper. So this


paper does two things to this. It says, okay, we can augment this by so called positional embeddings,


a positional embedding you might know from the sequence transformers. So if I have a sequence,


my cat is tall, don't even know what that means for a cat. But okay, what in a positional encoding,