which basically means that it will compute entry by entry or block by block.


GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding (Paper Explained)

So these are the matrices we have to multiply. And you can see that if I want to do multiply this by this, that's fine.


I can do this on one machine. But and that will give me the block up here.


But if I want the block up here, I have to multiply this by this, which is across two different machines.


So what this system does is it's going into a while loop because it realizes there's not enough memory.


And it kind of sends around these different slices to the different parts, each time computing a little piece.


So here, first we do this by this, this is fine. But then we grab ourselves from the, we grab ourselves this one here,


calculate the next little piece up here, then we grab ourselves the number two, calculate the piece here.


And then so this is from zero, this is from two, the one we already had, and then we grab ourselves piece three,


and multiply that until here, until we have this final slice that we want.


Okay, so this goes in a while loop in multiple rounds, the system gets knows itself when it has to do this,


and when it can calculate the full thing at once, because it fits into in memory.


It's even smarter than that, in that it can do these halo exchanges. So if you have to do something like this, a convolution.


Now in a convolution, what you'll do if you think of a, think of an image, and you want to do a convolution on it,


but the image happens to be sharded, let's say the image is so large, it's sharded across nine different machines like this.


Now if you want to do a convolution, that's pretty cool, you know, here, here, here, but here, all of a sudden,


your convolution is across two different machines. So this system G shard will adapt automatically,


and do these halo exchanges where it kind of sends around from this machine, it'll send something to this machine,


such that it can do the convolution in that step, and vice versa. And then this can be a padded accordingly, as you can see.


This, I think this is this is this was like super ugly to implement, if you just imagine that for each of these operations,


you have to think about, okay, how can I express this with these MPI primitives, like dynamic slice and collective permute, and so on.


It's just an absolute nightmare. And I'm very happy that other people have done this, and I will probably just get to use it.


So there is a lot more to this system than I've just explained, I just tried to give you a flavor of what building a system like this


means and how easy it is to use it like this. In order to implement all of this mixture of experts things,


you simply go from this, which is a one a single machine implementation, how you would write it to this, which is now the same,


it's almost the same code. But this now you can run on however many machines. And if you compile it with the system,


it will do what you expect it to do in this sharded way. Completely crazy.


Okay, so they apply this to massively multilingual massive machine translation.


So two things, it's massively multilingual, and it's massive machine, which means, I guess, a lot of machines.


And the reason here is twofold. So what they say is, we have massively multilingual translation,


why don't they just look at, you know, single machine translation. And it has a very specific reason, namely,


if you have massively multilingual translation, which means that you have a lot of different languages,


and you all have to translate them, ideally, to all the other languages, or, you know, every language pair.


But in this case, they only look at all the languages to English. I don't exactly know why, but I guess there must be some kind of reason.


If you do this, then you can make use of two of a thing where there are languages that you just don't have much data on, like, I don't know, Basque or something like this.


There's not that many people speaking Basque or Swiss German, there's not even a written form, a standard written form of Swiss German.


So you just don't have as many resources. And for other languages, you have giant amounts of resources.


And what you can make use of is this phenomenon called positive language transfer, where it happens that, for example, Swiss German is very close to German.


Now they can't understand us, which is a giant advantage for us. But still, it shares a lot of similarities with German.


So if you learn a lot about German, you can sort of transfer learn to Swiss German pretty easily.


So if you have a system that does German and Swiss German at the same time, you can perform better on both languages because the Swiss German part of your model,


the part of your model that does Swiss German profits from the German inputs as well. Now, don't understand me wrong, there is not an individual part of your model that for each language, it's all done at the same time.


But still, you can imagine that, you know, some of these things will specialize in some of the languages.


But the hope is that if you have German and Swiss German in the same training set, that if the model realizes what a question construct is in German, it will be able to apply that also to Swiss German with some minor modification.


So there is a benefit of having these many languages, especially for the low resource languages.


Okay, so as the number of languages sorry as the number of language pairs to be modeled within a single translation model increases positive language transfer starts to deliver large gains for low resource languages.


Given the number of languages considered, which I believe is 100 here, M4 has a clear advantage on improving the low resource task. On the contrary for high resource languages, the increased number of tasks limit per task capacity within the model,


resulting in lower translation quality compared to a models to to a models trained on a single language pair. This capacity bottleneck for high resource languages can be relaxed by increasing the model size to massive scale in order to satisfy the need for additional capacity.


So basically, you're saying, we can, if we train all these languages together, that will help a lot for these low resource languages, but it might hurt the high resource languages because now, we would have enough data technically to train and French to English model on in, you know, this giant model, we could train that.


And now that we have all these other languages in there, it just hurts us because we don't have enough parameters. And we can solve this, of course, by simply adding more parameters. So that's the solution, add more parameters, and you increase the capacity of the model, and you still get the benefits of the positive language transfer.


So their investigations is going to be into how much can we scale this? And is there like a sweet spot? Where because if you if you increase the parameters too much, you counteract this positive language transfer again.


So since, you know, since Swiss German and German can sort of benefit from each other. However, if we have too many parameters, so, and then we end up having all of these experts right here, and the tokens are always routed to these experts, and it always happens that all the Swiss German tokens are always routed to this expert and all the German tokens are always routed to that expert.


There will be no sharing of weights, there will be this positive language transfer will not happen because we have too much capacity. So the goal is to find a sweet spot between positive language transfer and this capacity bottleneck.


They do use an in house data set, which we don't have access to. But they say, the training corpus mined from the web contains parallel documents for 100 languages to and from English adding up to a total of 25 billion training examples.


However, they only use from 100 languages to English, this result in approximately 13 billion training examples to be used for model training. So that's a lot. It's a lot of data, especially for translation is kind of noisy translation because it's mined from the web, but still, it's a lot of data.


They have baselines. So the baselines are first of all, in order to form our baselines, we trained separate bilingual neural machine translation models for each language pair.


So that means a single model for each language to English, to depending on the available training data per language.


And then they also have a they also have a baseline, where they try open AI style to build as deep as single transformer as possible. And by that they mean, we also include a variant of a dense 96 layer transformer encoder decoder network trained with g pipe pipeline parallelism on the same data set as another baseline.


So the difference again here is that this 96 layer is a dense transformer, which means that all of the tokens go through the same computation, and we don't shard the computation out to these experts, right?


You shard according to the batch, but all of them go through the same parameters. And that means we can, we can only scale up the number of layers. And that severely limits the that severely limits the computational efficiency, even if we have, you know, your pipeline parallelism and so on.


That hurts. They say training to convergence took over six weeks on 2000 TPU course.


That's crazy.


But I guess, yeah, you know,


I was saying earlier that that I always thought we were happy.


I always thought we were happy and machine learning because kind of the the hip science fields being biology, like genetics and and machine learning was thought like, oh, but these biology people, they always need like million dollar grants from government to run their experiments and we can just sit down with a laptop.


This time is over. If you start a PhD now start applying for money to get TPUs.


Yeah. Okay. In any case, here you can see what this does. So they compare a bunch of models right here.


So this T, this is this big dense transformer, that's going to be one of our baselines. And the other baseline here is going to be the zero axis. The zero axis means this is the single model for that language pair.


So only so for each language, they trained one model only on data from that language.


And that's going to be the worst thing here, because this this multi language translation in one model will generally help you if you have enough parameters, you can see all the models here have enough parameters, such that the difference here that this is difference in blue is positive.


Including this baseline model right here. So the baseline model, as you can see, has 2.3 billion parameters, even though it takes that much longer to train. And that's, as we said, a function of the fact that it's dense and deep, that hurts in training efficiency.


And then you have these mixture of expert models. They always consider two things they consider different numbers of experts, you can see it goes from 128 to 2048 experts.


And they consider a number different number of layers from 12 layers to 36 layers, 36 layers still being way smaller than the 96 layer transformer here.


And that's the reason why it trains faster. So it, it doesn't train faster.


So the reason it trains faster is because it has less layers. And then the reason it has more parameters is because it has a lot of these experts.


And the art here is to constrain how much these more experts hurt you. So, you know, you could run into the same problem where if you scale up the experts, in fact, you do, it doesn't fit into memory anymore, and it's going to hurt you a lot in training efficiency, kind of like if you increase the number of layers.


But the G shard system prevents that it lets you up the number of experts without incurring the cost. That being said, it does not let you up the number of layers, you're going to incur the same cost if you up the number of layers, as you have with the dense transformers.


So does this help? It helps a lot. As you can see right here, there's a general trend upwards and what's the x axis, the x axis is low resource languages.


So you can see that as we as we go to lower and lower resource languages, this multitask training, this multilingual translation improves significantly over the baseline where we only train a system for that language specifically.


And these 10k examples, it's a it's a it's quite a bit, but it's not that much, especially since it's noisy data.


So this is specifically good for low resource languages. But you can see also the high resource languages here benefit from the multilingual translation.


And that's a function of the fact that we have, you know, large enough models. In fact, you can see the larger the models, the more the difference in blue is, and there's not really an end in sight.


And they also see it say that they haven't seen convergence in training. So you can technically train this, you know, forever.


Yeah, you can also see that the the lowest mixture of experts right here is almost on par with their big, dense transformer that took so much longer to train.


Right. So this lowest model right here, I believe it took I don't want to go back, but it took it took hours or so or few hours to train.


Whereas this 96 layer dense transformer took these six weeks to train.


Though has to be said, the number of TPUs is not to be neglected. But if you're Google, you know, you just have them laying around.


What's also interesting here, and you can start seeing this two things. First of all, you can see that the difference between here in between the dense transformer and this baseline model is very low for high resource languages, but gets larger for low resource languages.


This is an indication that the dense transformer, it does more to share parameters between the languages because it shares parameters between all the things because all the tokens go through the same computation.


So it is going to be a bit better in low resource languages, but still the general trend upwards holds even for the mixture of experts.


The second thing is that you see there is a crossover here in these in these big in these biggest models.


And what are the big models? One, the blue one is the one with 2048 experts, and the green one is the one with 500 experts.


They're both as deep models. But all of a sudden, over here for the high resource languages, it's still true that if you up the number of parameters, you get a benefit.


So up the number of experts as well, you get a benefit. But over here for the low resource languages, it's it, you see, it actually hurts you to up the number of experts.


And that's the phenomenon. Exactly. We talked about before, if you have too many of these experts, and you do a hard routing, that means all the tokens go a different way.


And that means you don't get any sharing benefit from the multilingual translation.


And they investigate a lot. And they basically claim that their sweet spot of expert in their particular task appears to be somewhere in between these 2000 and this 500 expert number, where you can see it doesn't always help you to scale up the model.


Though I have to say, maybe the transformers, maybe they need a resonant moment. So I believe in computer vision, it was sort of the same problem that we try to build deeper models and why like, okay, this, this is more width, but


I think there might be some breakthrough on the horizon where someone just figures out how to train these giant models, even more giant transformer models with deeper layers.


And then there's a new new era of transformers. However, this is not that effect. I'm sorry, I said this at the wrong place. This is not that effect.


This is to show that in this case, we do benefit for the high resource languages because we increase capacity. But for the low resource languages, we suffer if we up the number of experts too much, because they don't share any parameters anymore between the languages or between the different parts.


Like it's not a necessity that the different languages are going to be routed to different experts. But it's probably going to happen right there's no no hard coded thing that says if it's this language, it needs to go there.


It just probably is going to happen this way because the different languages are going to be needed to be treated differently and therefore the system learns to route first and foremost, those two different experts.


Here you can see the model sizes, including this 60 layer models model with 2000 experts that they didn't manage to train they said they had numerical instability, but that had 1 trillion parameters.


I'm pretty sure they're, they're cool. They must be quite mad about this right? Like you have the trillion parameters, even though it's not that much bigger than the 600 billion that the trillion, it would be cool to write a paper like a trillion parameter model.


But for now, they are at the 600 billion mark, and they simply want to tell you that they have actually compiled a model that's that big just didn't manage to train it. And yeah, that's here, here is where I wanted to say that maybe we're waiting for the resonant moment, where all of a sudden someone figures something out that makes the training of basically infinitely deep transformers possible like we made the training for almost infinitely deep.


CNNs possible with Resnets.


Okay, so they conclude this, and so they, that's the the investigation of what the number of experts and so on gives you. And here is a bit of a different investigation, where they more care about training efficiency.


So they ask themselves, how many billion tokens of input do we need to reach a given cross entropy. So here, the more tokens you need, the lower your efficiency is right.


You can see that the general trend is the following. If, if you up the number of layers, you get more efficient, you can see and just look at this column for now this point seven column, you can see it already pretty clearly.


So here you go from 12 layers to 36, you gain efficiency. Here you gain here you gain pretty predictable, if you up the number of layers, you need to see fewer tokens to get to the same cross entropy.


And in fact, you can get to a lower cross entropy all together at the end. We've known this for language models already.


The other effect is of course, what happens if we go not deeper, but wider, if we increase these number of experts, if we increase this sparse computation.


So here you can see, let's just look at the 12 layers for now, let's look at all the rows where there's 12 layers.


So here, you get a significant advantage by upping the number of experts from 100 to 500. But then you hurt upping the number of experts to 2000. Right?


So that's that's sort of your you're hurting efficiency by upping the number of experts too much. And the same if we look at the 36 layer, so you gain massive efficiency by upping the number of experts, but you lose that efficient part of that efficiency again, by increasing it even more.


Now, we saw that the this model is still the best model, but it's not as efficient as that model. And that gives you another indication that there is sort of a sweet spot between these two things between the positive transfer and the bottleneck capacity that appears to be somewhere in between right here.


So that's pretty interesting. Because we know about depth that you can basically up and up and up and get more efficient, but with not that much.


Yeah. The largest model can be trained in under four days to achieving the best quality. Yes, yes, yes, but this is just a yeah.


So here, oh, you can see the batch size in tokens is quite, quite a bit.


So yeah, if you have a thousand, if you have a context window of a thousand, that means the batch size here was about 4000.


So as as expected.


Yeah, this is just easy peasy 22 TPU core years. I've seen someone on Twitter saying this. This is the new measure for computers.


It's no longer like flops. It's TPU core years. Just mad, mad. And yeah.


So 42 days to train this thing right here. Crazy, crazy, crazy.


All right. They also have a number of investigations in other parts of efficiency, like per device memory consumption.


You can see here that as you up the as you up the number of experts, you can see here, here, here, your weights don't go up because as you up the number of experts,


you can just up the number of machines and the per machine weight usage will be the same, right?


Because the experts are independent of each other. Each one has their own weight matrix.


So you can just add machines and you keep your weight requirements the same.


However, if you go deeper, then your weights increase because you're now deeper. You have more layers.


You have your so also your transformer weights will be higher and so on.


So you go deeper right here. You see 3660 layers. Your memory consumption increases for the weight.


And also, this is the other big part in transformers, right?


The activations that you have to save, because as we said, if you have a transformer and I have layer, layer, layer, layer, I basically have to keep around each of these signals in order to do back propagation.


And that's why also the activation here increases as I go deeper.


Now you can see percentually it decreases again here.


So what's happening? Technically, you don't have to keep these things around.


You can also once the signal comes back, you can recompute them from the beginning or from an intermediate point.


Now this increases computation, but saves the need to store the activations.


And apparently G shard yet another thing it does is it will recompute as necessary.


The activations if it realizes that you don't have enough memory to store them.


So all of this is pretty crazy, honestly.


And they look at where the different computations go.


And I don't want to go into this.


And they have these micro benchmarks where they really show that the increase in complexity is really according to square root of n, because that's how long it takes to distribute along these experts.


There's a lot to this paper and there's no time to go through all of it.


I think this video is already way too long.


I hope I have given you an impression of what's possible with this system.


And as I said, I'm excited what people can come up with.


Just to say that in the appendix here, they detail that they have done this for all the operations in XLA.


So, for example, convolution.


This is so ugly how you have to implement the convolution because you have to padding must be correct across these expert across the sharded machines.


So there are no experts anymore.


This is just G shard.


The padding has to be correct.


The strides have to be correct.


Data needs to be exchanged according to the machines.


The window size needs to be correct.


Blah, blah, blah.


It's.


Thank you for doing this and not having to do it myself.


Yeah, I'm excited.


As soon as the code's out, if I get hold of it, I'll, you know, link it or you'll find it once it's out.


If it's already out, I'm just too dumb to see it.


I enjoyed reading this.


It's different than a machine learning paper.


It kind of shows you what goes into engineering a system like this and how easy it can be if it's engineered well to then apply it.


I think this is going to be extremely helpful to the community.


And with that said, 23 pages later.


See you next time.


Dreamer v2: Mastering Atari with Discrete World Models (Machine Learning Research Paper Explained)

Hi there, what you're seeing here are predictions by a world model learned for Atari reinforcement


learning.


On the top you see what really happened during an episode of play.


And on the bottom, you see the predictions of this world model, the world model just


gets five frames at the beginning, which you don't even see here, as a conditioning, and


then it predicts 45 frames of gameplay.


It's astounding how accurate it is not only in terms of how the game evolves, but also


in terms of what the agent will actually do.


So the world model, the specific world model you see here is part of the dreamer v2 algorithm


from the paper mastering Atari with discrete world models by Danny Scharhoffner, Timothy


Lillicrop, Muhammad Nerozi, and Jimmy Ba of Google Brain, DeepMind and the University


of Toronto.


So these kind of world models, they enable you to do very quick reinforcement learning.


Once you have the model, you can use it to imagine yourself playing the game instead


of actually playing the game.


And therefore, you can do much more efficient reinforcement learning.


And this paper details how to get an accurate world model for Atari, which was sort of out


of reach until now, especially considering that they only do single GPU reinforcement


learning.


So the result, as you can see here, is going to be an algorithm that is the top single


GPU agent right now, competing, you know, outperforming other.


So here's dreamer v2 outperforming other algorithms such as rainbow, IQN, DQN.


And the special thing here is that dreamer v2 is a model based algorithm, whereas the


current or the previous best ones, especially single GPU best ones, were model free algorithms.


And you can see the next best model based algorithms were are not really competitive


in Atari, right, this is specifically Atari.


So dreamer v2 is an evolution of dreamer v1, which worked well for things like continuous


control, but Atari still seemed a bit out of reach.


So the difference between model based reinforcement learning and model free reinforcement learning


is that model based reinforcement learning first learns a wall model of the world, it


learns how the world acts, and then it uses that model to learn what actions to perform,


whereas model free algorithms, they simply act in the world, and they learn to predict


the best actions as they act in the world.


So there's your difference.


And how does dreamer v2 do that on the high level, it has two stages.


Stage one is learn a world model from past experience.


And then stage two is use that world model, as we said, for reinforcement learning.


And the reinforcement learning here is going to be just actor critic learning.


Very straightforward.


There's a little modification with a pass through estimator.


But the real difference is going to be in how the world model is learned.


And the novel contribution or the main contribution here is this latent state, which consists


of a stochastic latent state, which other than other world models, which model the latent


states as something like Gaussian random variables.


This paper models the latent state as categorical random variables.


And that turns out to work pretty well for Atari.


So that's step one, learn world model step two, do a reinforcement learning in the model.


So not using any data anymore.


And you can repeat those two steps as many times as you want.


So you start out with a, you know, set of data, then you learn an actor.


And then you use that actor to collect more data, and so on until you have a really good


actor and the world model is really accurate for that actor.


So that's the overview.


And you know, it's going to turn out as we already saw to to beat other at least single


GPU models by quite a bit.


So we'll go through the paper through the individual steps and discuss what's new and


how it all works.


The code is also available.


I'll link to it.


And the blog post I've shown you here has some more explanatory graphics.


If you like content like this, as always, don't hesitate to click like and shared with


all your friends, especially the Atari gamers, because they are outperformed as you can see


here.


All right.


So world models, pretty quickly in reinforcement learning you as you all hopefully, or maybe


no, you have an agent that is interacting with an environment.


And the agent can so the environment always provides the agent with an observation Oh,


here, which would be an image in an Atari game.


And the agent decides to do one of many available actions in response to receiving the observation.


The environment then responds with a reward for that action.


So either you die, which is like negative reward, or you collect the coin, which is


positive reward, or you win the game, which is like 1000 reward.


And it also gives the agent a new observation, the next observation, and the agent again


response by performing another action, and so on.


So you have this cycle and the goal of reinforcement learning agent is usually to maximize all


the rewards that it collects during playing with the environment.


And you want to repeat that many times for many episodes to have the agent learn to get


as to do the actions that are as good as possible in terms of reward.


Now, in classic, let's say classic in model for your reinforcement learning.


One way to do this is to take this right here, as you play the game.


As you play the game, you collect data, right?


So let's assume we collect data as we act in the world.


And from this data, we can, we can learn something.


So model free learns from the raw experience.


So an episode will always be a series of images, right?


And actions you have performed.


So here is an image and I have performed action one and then came the next image and I've


performed action two.


So what classic reinforcement learning would do is it would say, okay, from this transition,


doing this action, I have gotten five, five reward.


And from this transition in this action, I've gotten negative three reward.


So I'm going to have to do this action one more often, because it gave me a lot of reward


after I observed this thing here, right?


The combination of this thing, I need to do action one more.


And when I'm in this situation, I need to do action two less, and so on.


Okay, so you're simply trying to put this image that you get into a neural network that


tries to predict action one as often as possible.


And you want the same network when you input this next image to not predict action two.


So what anything but action two.


Okay.


So that's going to be that's kind of the logic between of the classic model free reinforcement


learning.


Usually, this is implemented in a sort of an LSTM fashion, or it's one way of doing


it.


So you have an LSTM that tracks a hidden state.


Why do you need a hidden state?


Because you might not see everything in the image there is right, this is not necessarily


Markovian.


So there might be information that you need to remember for a long time, like when an


enemy leaves the screen and then comes back, you want to track it.


Do you have an LSTM or some kind of RNN, and then you want to feed the images into that


one by one.


And then you simply so with an encoder, which is usually kind of a convolutional neural


network, I'm going to draw it like this.


And then you try to predict the here the good actions and here you try to not predict the


bad action, and so on.


So this is a simple classifier.


Ultimately, it's an LSTM with a classifier on top.


And the classifier simply tries to either predict a class of action one or not or predict


anything else, right.


So and you train it via backpropagation through time.


And that's it.


Now here is a little bit different.


So why?


Why is this maybe not a good idea?


Well, all you have is the signal of the reward for given actions.


And that means it is, it is fairly hard to generalize in these kinds of things.


So when you imagine you have your screen right here, and there's an opponent kind of here,


there's an opponent here, and you are down here, and the opponent shoots, right, you


have to move out of the way you have to move over here.


Now, RL is completely capable of learning that.


However, take the next situation over here.


Now the opponent is here shoots, and you are down here.


You have to again, learn to move out of the way.


For a classic RL algorithm, these two things are identical are completely different states.


Like this, this is there's nothing equal about the two, like this is a completely different


thing.


And it has to sort of learn by force, look, in this situation there, you know, you need


to move and then this situation, you also need to move.


Now given that that is a convolutional neural network, it might after a while learn the


fact that it you know, these two situations have something in common.


But in essence, these are two different things.


And you have to learn purely from the reward purely from the fact that you're going to


die if you don't move to get out of the way in two situations.


And of course, this situation can be replicated all over.


However, if you have a world model, right, imagine now we have a world model over here.


And the world model accurately learns to predict the future.


Now we know that, you know, we are here, this is here.


Now we can imagine ourselves forward.


And we were going to see we're going to get hit.


And that means we need to go out of the way.


So doing this explicitly would be called planning.


We are not going to do planning in this paper.


Okay, we are still going to do the classic RL.


But you can see what advantages a world model could do.


Now the advantage of the world model we have in this paper is that it is going to enable


this left hand process much faster, because we don't even we don't need to interact with


the world anymore to learn all of this stuff.


We can simply do this in imagination while dreaming.


So to say that's why it's called dreamer and learn the stuff on the left.


So it's not that the world model is used for explicit planning for explicit thinking ahead,


it's just going to rapidly speed up this process on the left.


It's technically model free reinforcement learning in a learned model, which is I guess


why it's called model based.


Okay, so how do we learn the world model?


This is quite a complex thing.


So the backbone, as you can see, is this H chain right here.


So the H chain, that is your classic keep where the model keeps track of a latent state.


So you everything that's kind of going on in the game right now, you want to save into


the latent state.


So the model is going to learn a latent state transition.


And this specifically is using a GRU recurrent neural network with a gated recurrent unit.


So it's not an LSTM.


But it's kind of a little brother of the LSTM.


That is sometimes a bit easier to train, sorry, Jurgen.


But this is the backbone.


Okay, so from step to step, we somehow we get an observation, and we somehow want to


incorporate that information and keep track of it.


Now how we do it, usually, this is it, right?


Usually you just feed this into an encoder, which in this case is going to be a convolutional


neural network, and then you combine that you put that as an input into your recurrent


cell.


Let's disregard everything else for a moment.


How do you actually train the thing?


Usually in model free reinforcement learning, you would simply predict the reward or the


action that maximizes the reward, like you would predict the best action to do in actor


critic or you can actually predict the Q value and Q learning.


Not in model based, we're trying to learn a model.


So what we're going to do is we're going to try to predict here, the we're going to try


to predict the image.


Now this can be in fact, the next image or it can be the same image and I don't even


remember which one it is.


Okay, it predicts.


I don't know.


So it can I'm going to guess it.


I'm going to guess it reconstructs the same image.


Okay, so here you can see the image predictor.


Oh, yeah.


So xt is predicted from ht and zt.


So we want to reconstruct the same image first and foremost.


Okay.


So we input an image and we want to get out the same image.


This is like an like an auto encoder.


So the representation we're going to get in the middle here, somehow needs to be able


to represent the image very well.


And we also want to predict the reward.


Here we're also going to get an action.


It's you can see it here more.


So we're going to get an action.


Remember we are learning from experience.


We have done this here a bunch of times and we have a data set of experience.


So we know what actions we took.


We're going to learn a model that tells us given we're in this state and performing certain


action, what's going to happen.


So we're going to learn the reward and the image.


And it might not make too much sense with the same frame.


But if you look at the next frame, it makes a bit more sense.


So given image x one, we want to encode it somehow, right.


And then through the GRU over here, we are informed well while after x one happened,


we did in this episode, we did a one.


And then we got reward, or two, and the resulting image was x two.


Okay, so we're trying to predict given observation and a latency.


Given state, this h1, we're trying to end an action, we're trying to predict what reward


we got, and what the game looked like after we performed the action.


This is trained in back propagation through time.


So not only do we predict, you know, one future image, but we actually predict a sequence


of rewards and images.


Okay, so that's how we're going to learn a world model, input, observations and actions


and output rewards and observations.


Okay.


And that's exactly what you saw at the beginning in these videos.


So they the model was simply input a bunch of frames here, and then rolled out for a


number of steps.


And it, you know, we looked at the output of this, this is by the way, this is a deconvolutional


neural network, like a deconvolutional, you know, like in a DC GAN type of type of network.


Okay, now what are these special parts right here?


The special parts are what makes this model work.


So the hidden state, as you can see, the thing I circled in red in the middle, is not just


the recurrent neural network hidden state, it is actually a combination of two things.


They call this a combination of a fixed state of a deterministic state and a stochastic


state.


So what you're going to have is you're going to have the state which is a vector, this


is the h, let's call that h zero, okay, of the of the LSTM.


Now you're going to get an action into this, as we saw before, the action is combined with


this, and you ask yourself, given that action and the hidden state, and now, we don't just


want to know what's the next hidden state, like in a normal RNN, what we're going to


predict is actually this z variable right here.


And this z variable is a description of the current state, a stochastic description of


the current state in a very specific form.


So the h is simply a vector, right, you can store in it whatever you want.


But the z, which is going to be concatenated to the h, it's going to be both, it's going


to be predicted from the h, and it is also going to be concatenated to the h for further


processing.


So you're going to predict this thing, together with the image x down here, you're going to


predict that z thing.


And you're also going to concatenate it to h for further processing.


So the red circle is going to be the concatenation, and not even that.


Okay, maybe I should explain what it is.


So it is going to be of this form, it is going to be a collection of categorical variables,


each having, you know, 32, so it's 32 categorical variables, each having 32 possible classes.


And the model can decide absolutely by itself, what the categorical variables are for, and


what each of the classes mean.


So for example, in the Space Invaders game, right, one categorical could be the location


of the agent location, right.


And the 32 different values it could take are maybe going to be, you know, if this value


is if it's this value, then it means the agent is somewhere down here in this quadrant or


in this tile.


If it's this value right here, the agent is going to be in here, and so on.


So these are categorical values, and they can, you know, take one of these 32 different


values, they can only take one.


So that's the difference between these and like a Gaussian latent variable.


Because these stochastic states used to be modeled in like, say, you know, we have 32


Gaussians, like in a in a VAE, we have 32 of these latent variables.


And now we make them categorical, and that turns out to be pretty good for this Atari


games.


So the other could be the enemy, does the enemy shoot?


Is, you know, has the enemy fired a shot?


Now maybe we don't need 32 variables right here.


Like this could simply mean, this could simply mean yes, and this could simply mean no.


But also, you know, we can make use, we can encode actually 16 different enemies.


So we can encode has this enemy shot that we see here, or has an enemy that is potentially


here fired a shot or has an enemy that is potentially here fired a shot, right, we can,


we can encode this in that.


Now I can see that you can see the problem, right?


Two enemies can shoot at the same time.


And in a categorical variable, you can only have one value.


However, it might still be enough to just encode, you know, whichever enemy has shot


most recently, or least recently, into this variable, and you can still play the game


with that information.


Okay, so you can see here that so it's 32 variables.


So 32, we can have 32 here, and each can have 32 different values.


And you know, the state is going to be described by


by having each of these 32 variables be, you know, in one position or another, as you can


see right here.


And hey, it's Yannick from the future.


I forgot the whole video to show you this.


So I'm doing it now, they have a pretty good explanation of why categorical variables might


be important for a thing like Atari.


And that is because sometimes you have pretty big junctures in the world state.


So maybe, you know, you do very similar actions, or maybe slightly different actions from the


same states.


But you know, the slightly different action results in different changes in the world.


And that means your prediction sort of has to capture all of that.


So when your predictions is just a Gaussian, a Gaussian can only sort of have a mean and


a variance, it cannot predict multimodal distributions.


However, a categorical distribution can like it can be spiky, it can be very concentrated


on one particular thing, or it can actually be a superposition of many different states.


And when you sample from that, you actually have your multi modality.


So it's again, something that is kind of very suited to certain environments, but not others.


And you know, when it fits, then it seems to work pretty well.


But this is in the blog post, if you want to look at this graphic yourself.


All right, back to past Yannick.


Bye bye.


You can see that the entire observation sequence, the observations, they never get into the


system except through these z variables.


So this is an extreme compression, every observation that you get in is going to be described by


this extremely compressed format.


And they hypothesize that, you know, because it's so compressed, because it's so sparse,


it might actually force the model to learn pretty good latent variables.


And that's also why it's so fast, because you never touch the observations again, you


only work in this latent space.


So what actually happens is the CNN is going to predict a distribution.


So for each of the 32 variables, is going to predict a distribution of the 32 values


that variable could take and one here, and one, and so on.


Okay, it's going to predict 32 distributions of that.


And then there is a sampling step.


So this is now sampled from this is the sign for sampling from and that gives you not 32


distributions, but it actually gives you 32 just straight, okay, here, here, here.


So this is why it's called the stochastic part.


So and that will actually make that blue.


So you realize that is going to be fed here.


So this this deterministic state h is going to be used to predict this distribution, the


distribution is going to be sampled from, and then this sample is going to be concatenated


together with h, and that will finally make our actual latent state.


So the latent state here is this concatenation out of the deterministic and out of a sample


of the stochastic, and that ensures that you sort of keep your your options because it's


sampled about the world model, you always draw from this distribution, which you can


entropy regularize, right?


But you also have the deterministic information that you pull through.


Okay, so that's how the hidden state comes to be.


And there is one node we haven't left out right yet.


Okay, during learning during actual reinforcement learning, what you want to do is the following


you simply want to start off with a single observation or actually a hidden state that


you've seen during training of the world model.


And from that point on, you don't want to have anything to do with observation.


So you see right here, since we, we learned a reward predictor, right, we can simply use


that reward predictor instead of the real environment.


So and we don't want observations anymore.


So what you want to do is you simply want to use this backbone here to predict the these


latent states, so you simply want to unroll these latent states.


Now usually, in order to do that, you need the observation you can see here, clearly,


the next latent state is a result of the previous one, and the action and the observation.


Now, if you don't want to do this, it means you have to predict the observation.


But you can't predict the observation because that will be slow.


And we already know that doesn't really work.


So you want to predict this Z variable, we've said that observation, the next observation


is going to be fed into the algorithm through this by means of constructing such a Z variable.


So if you could predict that variable, without seeing the observation, you could you don't


need the observation anymore.


And that's exactly the last output right here, you can see each H state, not is not only


used to construct that Z variable together with the observation, we also predict the


same Z variable, but without looking at the observation, okay.


Of course, that's going to be not as good, like, the latent representation is going to


be much better when you actually see what happens in the game.


However, in order to do dream reinforcement learning, we need to be able to completely


detach from the observations.


And that's why we also predict at the same time, we predict the same variable, but without


seeing the observation.


And then we're going to introduce a loss function that makes it such that these two are going


to be very close together.


So the agent now has to do a trade off.


And the trade off is, do I want to get the best information out of my observation?


Do I want to represent it as accurately as possible in order to reconstruct it really


well and in order to predict the reward really well?


Or do I want to be able to predict this thing without seeing the observation, which means


that you know, I have to, I have to not rely as much on the image, I have to rely more


on learning the actual dynamics of the world and what happens when I perform actions in


them.


That's what exactly what this KL divergence here is going to do.


So the model has to find a trade off between the two.


And if you engineer that trade off correctly, you are able to use the just the predicted


z variables instead of the true ones, at least for a certain number of steps, I think they


do 15 steps into the future during learning.


And of course, the errors accumulate because you never able to predict that z exactly.


However, it's enough to do good reinforcement learning.


And this sparsity here, it helps very much.


Okay, I know this is a lot.


But you know, to shortly recap, learning world model means that you input observations, and


you learn to predict the future.


So you learn to predict the future observations, you learn to predict the future rewards, given


actions, given actions that you performed, you start off with a random agent or any agent


you want, you simply want to learn what happens when I do something.


Now the way you predict that is going to be through a recurrent neural network, the latent


state of which is going to be a combination of a classic latent state of an RNN and concatenated


with a sample from a stochastic, very, very compressed state that you obtain from a CNN


encoder combined with the last hidden state.


So the combination of a sample from this, and the deterministic state is going to be


your compact world model state from which you predict the future.


And in addition to that, you also try to predict this stochastic state, just from the deterministic


hidden state and the action without knowing what the actual next observation is, or the


current observation, I guess.


And that means you can then use those prediction values at reinforcement learning time in order


to be completely decoupled from the observations.


And now, yeah, we we we sort of have it.


So what if you learn a world model like this, what you can do now is you don't need the


observations anymore, you maybe need one start observation, and you simply unroll into the


future.


And you do reinforcement learning in this completely imaginary, like this is a dream


now, this is a dream.


This is just dream, a dream.


Now, it's a it's also completely not cheated.


Yeah, so the reinforcement learning they do right here is going to be something like,


you know, A2C or A3C, it's going to be an actor critic method, an advantage actor critic


method.


It's a pretty basic, but very strong reinforcement learning algorithm, where you learn sort of


two models, you learn the critic that accumulates that tries to predict the future rewards.


So they tries to predict these values right here.


And you learn an actor that is trying to make the critic really, really happy.


Now, you swap this once you have a good agent, you go back, you collect more data, because


your world model is never going to be accurate, it's never going to replace actually playing


the environment, your world model only has data from where the agent goes, right?


That's where it learns from.


So it's crucial that once you have a better agent, you update your world model, because


now the agent does different things.


And it goes places that the world model has never seen, right?


If you know, if you have this, if you have like a maze game, okay, and the maze is, I


don't know, I'm not good at mazes.


But you know, you're here.


And once you crash into a wall, you're done.


So the agent, it will just be random at the beginning.


So like crash a lot into these walls, and so on, just do random actions.


So the world model, if it just learns from that experience, it is going to learn maybe


that there's a wall right here.


But this thing, we don't know, right?


Now, if you get a little bit of reward, maybe there's a coin right here, okay.


And every now and then this stupid random agent actually finds the coin, right?


It walks over here and finds the coin gets a reward.


The reinforcement learning means that it's going to do that more often.


So now, the agent is going to walk over here more and more often.


But you only do that in the world model, the world model only knows up until here, because


that's where the agent has gone the farthest.


Now that the agent goes further, right, you actually need to go back to the environment


and let the agent run in the true environment.


Because now that agents going here, you know, it's going to explore a bit more.


Because you know, it learned it learned only seeing this.


And now it learns a bit more you record, you build out your world model, it's like, ah,


there's the wall goes until here, but then there's a free space.


And then maybe something comes here, and so on.


So working with world model is not is not super easy.


And it only is going to this is very specific.


And this is going to be my my criticism right here, in that all of this seems quite specific


to Atari, and a reinforcement learning is such a big field, and such a general algorithm


that you're going to build in some kind of prior knowledge about the world.


But it seems like the some reinforcement learning papers, I never know, you know, how much is


this all applicable to other RL environments, it seems like this, you know, is specifically


for Atari.


And learning these world models in this fashion is only going to work.


If you know, every now and then you find a reward, you still have the explore exploit


dilemma, if your world model isn't accurate, then, you know, you're not going to do accurate


RL, and so on.


And maybe the density of rewards isn't going to be enough for you to actively push yourself


up in these cycles.


And you know, there's another problem with these latent variables, they're categorical,


which I think, you know, is super cool, because it gives you a sparse representation.


But you only learn it from the images.


In fact, they say they can even leave away the reward predictor for the world model.


So you learn to reconstruct the images.


However, if two images are very close to each other, but they mean different things in the


game.


So, you know, two images can be super duper close, like an enemy can be here, or slightly


off, right.


But if it's slightly off, it doesn't hit you.


And therefore, you know, you're all good.


Now these two states are still pretty close, because if you move a bit, you're likely to


get hit.


But sometimes a little bit of a change in image can mean actually a big change in game


state, and vice versa, which is actually even worse, a big change in image can mean like


it doesn't matter, like if everything in the image rotates around, but your agent still


has nothing and is at the same place.


It means nothing to you as a human, yet an algorithm like this, that whose goal it is


to predict the future as accurately as possible, it will devote a lot of attention to accurately


predict the future, or predict variances in the future, even though they might not be


relevant.


So in this in this task of or in this bottleneck of encoding everything into a very compact


state, you might actually lose important information.


And that means all of all of the like two states that are very, very far, like need


to be differentiated, are going to be just the same in this representation.


And there that means your agent will will never really learn because one is bad and


one is good.


So the mean reward is zero.


And it says, Well, when I get to that state, my mean reward is kind of zero, and it's just


kind of a big variance.


And then the world model will never learn the difference because it has bigger things


to worry about.


So this is, it's all very specific.


And you'll see this in the in the loss term right here.


So this is the loss function for learning the world model.


And you can see they have an image reconstruction loss right here.


This is a, this is cross entropy loss.


So it's, this is your approximation distribution.


This is what really happened.


Yeah, it's a, it's kind of a probabilistic way of writing things.


So these are cross entropy losses, when you see log p of the expectation of under Q.


They have a loss predicting the reward, they have a loss predicting the discount, which


is, you know, mainly made for predicting when an episode ends in the in the imagine trajectory.


And then they have this transition loss, coupled with the entropy regularizer.


So the transition loss is going to be for predicting these z states.


And the entropy regularizer is for keeping the distribution in the z states not peaked.


So you want to kind of retain that stochasticity.


And this together, you might recognize as the KL divergence between the p and q.


And that's this connection right here.


So I'm going to minimize the KL, which is the same as saying, I want this thing to be


as accurate as sorry, I want, I want, I want these things to be as close as possible to


each other.


But the entropy should, should still be given.


And yeah, as you can see, or you can you can you can decompose that.


So this is going to be, this is going to be the KL divergence between the two distributions.


I don't have a better way of explaining that without writing it down.


You can already see, they have a massive amount of hyper parameters, right?


Like here's one, here's one, here's one, here's one, here's one.


Okay.


So even within the KL divergence, they have actually two, one hyper parameter for the


KL divergence and one to trade off the entropy with the actual cross with the transition


log loss with the cross entropy there.


And they do ablations and see that that is really important that you have that trade


off that you're able to make that trade off.


And it's the same as the beta variational auto encoder.


By the way, it's an entire paper about why you need an additional hyper parameter here.


Like that's the entire paper of beta VAEs, which I found funny, but you know, it seems


to be important.


You can see right here, this is KL balancing.


So you have one, you have one term for making the prior close to the posterior, the prior


being the one where you just see H and the posterior being the one where you see H and


X.


And you have another term for making the posterior close to the prior and you trade them off


with these variables right here.


Then the reinforcement learning itself, again, has a bunch of hyper parameters.


So it is doing TD lambda learning and you can look that up TD lambda learning basically


means you're here in your state and you're going to predict the value, sorry, the reward


going to the next state and you're going to predict the value at that state.


And then you're also going to predict from the same state, the reward two steps forward


and the value at that state.


And you're also going to predict the reward three steps forward and the value at that


state.


And at the end, you're going to sum all of that up into one number that is kind of an


aggregate of all of this.


And that's going to be your prediction.


That's what you regress on in your value predictor and the actor tries to maximize that.


So there's another parameter lambda that tells you how you aggregate these things, right?


And also H for how many steps you do that.


There's also going to be in the actor loss function, they decided not only do they want


the classic reinforce loss as you have, you actually want the a straight through estimator


of the distribution.


And so a straight through estimator is when you want to back prop through sampled things.


Normally the reinforce gradients, what they do is, if your actor outputs a distribution,


let's say over three actions, right?


You don't all you can say is that I did action two here, and it gave me seven reward, right?


So you want to make that more likely because seven is pretty good.


Actually you subtract the baseline, but you know, let's say after the baseline, it's seven.


So you simply act like you have a target distribution of this and scale it by seven.


That's reinforced gradients.


What you could also do is you could actually regress on directly through the softmax operation


right here, because this here is a sampling step.


You cannot back prop through sampling steps.


The way you can do it is that you take the signal, the loss signal here, but you act


as if this was your output and not this, okay?


So you act as if you had made actions in proportion to their distribution and not actually sampled


one particular action.


This is going to give you a biased signal, but it has much lower variance.


Whereas if you sample and then scale, it's going to be unbiased, but much higher variance.


So they do these straight through estimators, not only here, but actually also in this step


up here.


And you can see how that works in modern deep learning frameworks.


So you have your distribution in terms of your logits.


So what you can do is you sample from them and forward propagate should be the sample,


right?


So the trick is to do plus and minus the same thing.


So the forward propagation signal is simply your sample, as you can see right here.


Now the sample, this operation, it has no gradient.


Oh, you can't see that.


It has no gradient.


So deep learning framework will simply not back prop through it.


So if you were to just use the sample in your graph, you won't get a gradient.


But what you can do is you can actually calculate the probabilities here, like the thing you


want to back propagate, and then do plus that and minus stop gradient of that.


You can see right here, this has no gradient.


This has no gradient.


So the gradient is going to be as if you had forward propagated this probes variable.


But on the forward pass, the probes variable exactly cancels out with itself.


And the sample is forward propagated.


This is called a straight through estimator gives you a bias gradient, but much less variance


than if you had to, you know, if you scale the sample like the reinforced gradients.


So these is in the world model, and they use this actually in the actor loss right here.


And you can see, there is another hyper parameter.


Here is another hyper parameter.


And then they have an entropy regularizer to facilitate exploration, which is normal,


but gives you another regularizer.


And not only do they have sorry, hyper parameter.


Not only do they have these three additional hyper parameters, they scale two of them


during training.


So they now have a schedule to scale them.


So they this straight through estimator, they actually scale it to zero over the course


of training, but yet, two more hyper parameters, namely how fast you want to decay those things.


So this whole thing is a giant bucket of hyper parameters.


And so they say, while the unbiased reinforced gradients can help add a better final solution,


however, we find that using only reinforced gradients for optimizing the policy also works


well, it might just not work as fast or as well, but it also works well.


You know that, in general, this is reinforcement learning, but this is a bit, you know, the


amount of hyper parameters here is quite staggering.


And I'm going to guess that this took a lot of work to even get off the ground, right.


So here you can see how this compares to other algorithms, specifically blue here is dreamer


And they do suggest a bunch of different things.


So they have task median gamer normalized.


So gamer is a professional human level gamer.


And gamer normalized means you simply divide by what that professional gamer can do.


So you can see that it can even exceed, you know, this gamers are here is over 1.5 times


over 55 different Atari games.


Very good.


However, these Atari games, some of them are actually unbounded.


And in some of them, a machine can just be so much better than a human, that usually


these scores are dominated by very, very few games where the machine just excels, you know,


hugely, and other games are like zero, and both the median score, and the mean score,


they are not really meaningful, at least that's what this paper here argues.


So they propose two modifications.


So the first modification, actually, this is from a different paper as well, says you


shouldn't normalize by, you know, kind of a professional gamer, you should actually


normalize by the human world record.


So this is record normalized, you can see it gives a cleaner score.


And then they say, well, given that a few games still the the machine can just outperform


humans so much, what you should do is actually, you should never allow the show you just,


you should just clip the machine score at where the human world record is.


So the reasoning behind this, I can imagine is something like, what's the difference between


the human world record and the professional gamer world record?


Well, the human world record, the professional gamer is already pretty good at gaming in


general, let's say, but the human world record holder has probably, you know, figured out


every single detail of that particular game and is, you know, is pushing it with like


exploits and whatnot.


You know, I don't know if you've seen legend like Ocarina of Time speedruns lately, but


they're crazy.


So that is going to be human world record.


And it's probably going to be better to normalize by this because you know, the machine will


necessarily find these kind of exploits, they will, it will probably find them as well.


However, there are some things that where the machine you have to be where you have


to be like pixel and microsecond accurate where the machine can do it and the human


can't.


So clipping it might make sense.


I'm not really sure about this.


Like there's arguments to be made that you maybe shouldn't normalize by the human world


record because, you know, you don't want to give credence to like, exploits, but the gamer


kind of represents more how the game is intended to be played.


I don't know.


They just suggest this new score just so happens to be that in this new score they are, you


know, other than here, they are just dominating at all time points.


Yeah, let's let's leave them that they do a quite a number of ablations, especially


they find out that, for example, if they do latent variables as categorical that outperforms


Gaussian latent variables by a lot.


So and that's, you know, that's kind of a reasoning why they use the categorical variables.


The KL balancing simply means that additional parameter in the KL term, if they enable it,


you can see it helps a lot.


Image gradients.


So when they they wonder, can we learn the world models from predicting images or from


predicting rewards or from both?


So they do both as a default, but if they leave away the image gradients, it doesn't


work anymore.


However, if they leave away the reward gradients, you can see it still works pretty well.


Again, this is all quite Atari specific.


And it also means that you can see right here, right, the Atari game lends itself to this


kind of to exactly this kind of model.


So how much this is a success for general reinforcement learning is questionable.


However, what you can say is that if an environment lends itself to be world model learned by


this kind of latent categorical variables, like so if the image state is going to be


if changes in the image are going to be a good indicator of actual changes in relevant


world variables, then you know, you might you might be very suited with a model like


this.


And so they compare this to other algorithms, for example, to mu zero, which doesn't run


on a single GPU, I think it is better, but it doesn't run on a single GPU.


And it uses kind of a lot more Atari frames than the the dreamer algorithm.


So you see, again, that you just need to find the correct category and you can be state


of the art.


So if this is like single GPU, Atari, no, I don't want to I don't want to dunk on this.


This is pretty cool work.


And if you look at the code, it took a lot of effort like you can see that from the code.


Okay, the last thing I want to look at is where does it succeed?


And where does it fail?


So you can see a comparison, for example, dreamer v2 versus IQN or dreamer v2 versus


rainbow.


And you can see, and particularly interesting is where does it fail, and it fails in video


pinball.


And actually, I don't have it pulled up right here.


But if you look it up, so if you look it up, you can probably see why.


Because this video pinball thing.


Thanks, thanks, YouTube.


This video pinball thing, it has a lot of changes in image without really doing much,


you know, changes in the world state.


So what actually matters is like this little tiny ball, this little tiny, you know, it's


kind of a bunch of pixels, and the rest, you know, kind of moves around.


And okay, maybe it doesn't move too much right here.


But still, you know, there's this new cross that appears and so on.


So a world model that learns to, you know, there's kind of flashes over the whole image,


a world model that learns to accurately predict the world, maybe is going to not focus so


much on that little ball, but maybe is going to focus more on the rest of the image if


that changes well.


And also, you can see maybe the reward now and again, a flash, the reward doesn't change


all too much.


Yeah, it does maybe but you know, any any time it bumps somewhere.


So my hypothesis is going to be that, you know, in games, where what actually matters


consists of very few changes in the actual image, and there are lots of other big image


changes that don't really matter so much for the immediate reward, maybe for the future,


but not for the immediate, this algorithm is going to not be as good.


And that is one example is this video pinball.


And I might be wrong on this, but it's kind of a hypothesis.


So the code for this is going to is available right here.


Check it out as well as you should check out the blog post.


They have a lot of ablations right here, as you can see, and graphs for the individual


games turning off and on different variables.


And you might as well give it a try if you have a reinforcement learning problem that


has an environment similar to Atari.


Alright, that was everything I had to say for this pretty cool paper.


Check it out.


[ML News] Hugging Face course | GAN Theft Auto | AI Programming Puzzles | PyTorch 1.9 Released

Huggingface releases a course you can now play GTA inside of an AI's mind and spot turns


one. Welcome to ML news. Good evening. Huggingface the famous NLP startup releases a course that


teaches you how to use their models, libraries and other code they release. This goes from


introduction of how to use transformers and what transformers are how to fine tune them


to the diving in area about the data sets and tokenizers library up to advanced things


like speeding up training and training your custom training loop. Of course, the course


is highly integrated with the Huggingface ecosystem, but it requires quite little and


it seems like a good place. If you don't know a lot, but you know how to program, you can


get into deep learning and specifically NLP pretty easily with that course. So the course


consists of videos, collabs, code demonstrations, and so on. This should be specifically interesting


for practitioners or data scientists that know a little bit about machine learning,


but really want to get into the applications of retrained NLP models, maybe want to fine


tune them a little bit, give it a try, check it out. It's up there for free. Next up the


popular YouTubers send decks releases a GTA version that is played entirely in the mind


of a neural network. All the environment you see is entirely generated by a neural network


that responds to your action. The network has been trained by random agents driving


around on this stretch of road so you can't actually go further than this. To run the


demo you do need a GPU that is CUDA capable though the code is available and you're probably


very free to extend this to also work on CPU and extend the level beyond this stretch of


road through all of this experience the neural network actually learns something about the


physics of the game itself, even though you never teach it physics. So go check out the


demo if you can check out the code give the video watch and the like, I'll provide the


links to the GitHub in the description of this video and you're able to take it from


there. Next up Facebook is testing AI to get you to stop fighting in its group CNN business


sites. Apparently Facebook is introducing new moderator tools for group admins that


get notified whenever there is a conflict argument happening in their groups. This allows


them to go in and limit how often users can post or maybe block some users in order to


de escalate the conflict. I love the examples they give here going like lol what shut up


you're so dumb. Stop talking about organic food you idiot idiots. If this nonsense keeps


happening I'm leaving the group. I mean I get they can't show the worst arguments happening


on Facebook in their product demo. It's still kind of fun. Now of course this is not the


first time that moderation tools are used or that AI is supposed to help moderation.


You can always be a bit skeptical about AI regulating speech somewhere as long as this


is just used to send notifications to moderators. It's one thing if this is also used then to


automatically moderate content, I would be a little more skeptical. Also the bigger problem


with these things I think is always the conflict between are we simply detecting toxicity and


conflicting opinions or are we detecting opinions that we don't like. Now today's social media


giants have a bit of a tendency to be in that second category. And that's something that


I would advise strongly against. However, there is an easier way to moderate toxicity


on Facebook. You don't want to get into toxic arguments on Facebook, I suggest you just


don't use Facebook. No one else does. You're welcome. You know on this show, which is an


irregular show, we do get our fair share of comments and feedback and thank you all so


much for that. Some are though just a little bit silly, like this one. Now that I think


about it, we see a strong gradient from the north. In this area, huge actions. And this


little piece, high, high accuracy. So take your time, train efficiently and avoid huge


saddles. Huge saddles are bad for you. Also, don't take your kids to saddles. They're dangerous.


Dangerous for you and your panel. For me, it's all. And now the word to Yannick.


Alright the Washington Post writes, an autonomous ships first effort to cross the Atlantic shows


the difficulty of the experiment. Apparently there is a ship called the Mayflower 400 that


is built by a British company and is supposed to cross the Atlantic Ocean in a purely autonomous


fashion. Now I'm not sure how much of this is technically AI, as it seems to be mostly


a lot of control theory and classic robotics, but it is an autonomous vehicle. So pretty


cool at that. So the applications of autonomous ships are going to be according to this article,


going and measuring some chemical composition of faraway ocean lands, ocean waters, generally


doing reconnaissance and listening to whale sounds. And surely there are no other applications


for this. Not at all. Can't strap anything to it, then you can then. However, there is


a problem in that the ship had a technical difficulty and had to return to shore. So


the actual crossing of the Atlantic will have to wait for another couple of weeks, it seems.


Now there is a website where you can track in real time what the ship is doing. So as


you can see right here, this is the route the ship was supposed to take with a few historical


landmarks of when famous other ships sank, and the target is in Massachusetts. Now what


you can also see is the path that the actual ship took until now. So it is still apparently


out in the ocean somewhere. And you can see the point where it had to turn around. But


it seems like it had some problems already before what exactly happened here, dotted


line is the course and it just kind of decided to get away from it. And then of course, here


it had to turn around due to the technical difficulties. However, once it turned around,


it just decided to go into a couple of formations just for giggles, I guess. So is it now still


going to America? Or is it returning to shore? No one knows. It seems like our long term


goal of building self deciding AI has finally succeeded. And the AI just decides to stay


in the water for a little bit longer. All right, next news, Pytorch releases the 1.9


million release. Among other things, it migrates some of previously experimental libraries


to stable such as torch dot linoc and complex autograd. Specifically torch dot linoc is


supposed to replicate whatever numpy dot linoc has in it and bring this to Pytorch tensors.


This should enable a lot more easy applications of classic linear algebra routines in Pytorch


effectively. Another big improvement is the mobile interpreter of Pytorch, which makes


it possible to reduce binaries that you ship to mobile devices by up to 75% for typical


applications. So if you want to get into mobile development with Pytorch, now is a good time


to check out the new 1.9 release. There are also a lot of other improvements, for example,


relates to the Pytorch RPC framework that allows you to send data around between distributed


workers. So check it out, give it a try. Let's go on. All right, ZDNet writes, I just watched


McDonald's new AI drive through, and I've lost my appetite. So apparently this tick


tock by user soup master 2000 is going around showing what the new automated drive through


machines at McDonald's are capable of. Welcome to McDonald's. We're currently serving a limited


menu. So please review the menu before ordering. Let me know what I can get for you. Can I


get two medium Oreo McFlurries? All right. Would you like anything else? That's it. Okay,


your total will be 658. Please go forward. Now people are calling this robot a bit dystopian


or whatnot. As ZDNet here writes, the voice is exactly the same robot voice you've heard


in every disturbing sci fi movie. It's as if Siri's daughter has just got her first


job. Welcome to McDonald's. It reminds me of GLaDOS in portal. So instead of this feeling


dystopian, I get a bit of a warm feeling in my heart. But as you can see, like the recognition


of speech works just fine. And that's honestly all I want from an ordering robot. I don't


want it to give me heartwarming emotions or anything like this. I'm just fine with that.


But it kind of shows you how hard it is to actually make a human interaction AI work.


And it seems like the more human you make it, the less people are forgiving of mistakes.


No one bothers if a automated train voice takes a little too long to announce the next


station. But when it's supposed to be more human, people get freaked out if it's like


just a little off. It's a very special phenomenon. But honestly, I'm not too bothered. Next news


CNBC writes artificial intelligence won't replace the role of financial advisors UBS


CEO says. So apparently, UBS CEO Ralph Hamer said artificial intelligence is better suited


to handling day to day functions like opening an account or executing trades. Apparently


he said that if it comes to these basic tasks, AI is better. And by AI, I guess he just means


software. Where is AI in opening an account or executing a trade? So apparently the opinion


here is that our financial advisors should be supported by the technology and their advisors


they should advise. So the advisors shouldn't take care of low level tasks such as opening


accounts. Instead, they should be informed by the AI to make decisions. He also said


UBS is looking to adopt a Netflix experience where clients can access a dashboard of different


research and product like everybody wants dashboards. Why? Why? Like I get it but technologies


like AI can help financial advisors figure out the best way to serve clients according


to Hamer's. If you ask me, this just sounds like an industry that's a bit in decline and


a bit threatened by the general rise of digitalization and software and AI. So all the tasks he describes


that AI is able to do is pretty much things that just software are able to do while AI


is going to actually replace these humans. So this kind of rests on the assumptions that


you think we still want to be advised by those bankers. Now if memory serves me right, didn't


you just kind of recently advise everyone to buy into the housing markets and then not


tell everyone that everything is full of crap until you sold your own stuff and then punch


the entire world into a big recession? Yeah, are you sure we want to be advised by those


people? I think I'll take my chances with an AI any day. Thank you. All right, Jürgen


Schmidt Huber released a new blog post celebrating the 90th birthday of Kurt Gödel's 1931 paper,


which he says laid the foundations of theoretical computer science and the theory of artificial


intelligence. Now whatever opinion of Schmidt Huber you have, he is a pretty good historian.


And his blog posts are generally quite interesting to read. So it's pretty short and concise


and filled with references that allow you to go deeper if you want to invite you to


go check it out and read it up. Next news Facebook releases oddly an oddly named data


augmentation library to help build more robust AI models. Data augmentation is an important


topic especially in things like computer vision research, but the library allows you to go


even beyond that into NLP data augmentation and others. So if you're doing anything that


uses augmentations, I invite you to check out this library. All right, a team from MIT,


the Allen Institute for AI and Microsoft Research have released a set of programming puzzles


along with a paper and there is a big GitHub repo filled with puzzles that are supposed


to accelerate the research into AI coding. So AI that is able to solve coding problems.


In these problems, the AI gets a piece of code which contains a function that it has


to satisfy and the rest is up to the imagination of whoever builds the algorithm. The cool


thing about this approach is that it's pretty general. So the examples here contain things


like towers of Hanoi, finding optimal strategies for tic tac toe, shortest path problems, and


even some open problems in computer science and mathematics. You can even contribute your


own puzzles. And I think the repository is meant as sort of a collective effort to collect


pieces of code that AI might be able to solve in the future, or that AI is already able


to solve. If you're into AI generated code, and AI generated problem solutions, check


out this repository and try yourself to come up with an AI that solves some of these problems.


And last news spot turns one beloved machine dog and carrier of various military items,


Boston Dynamics robot spot turns one year old as deployed in the real world. So Boston


Dynamics has released a little video of where spot is used throughout the world. Now, of


course, there are some pretty cool applications for this technology. Like it can go into mines


and check out dangerous areas, it can go into high voltage areas or into Chernobyl to make


sure radiation. And it seems like the applications of drones like these are pretty, pretty numerous,


it can save a lot of humans from doing either very tedious work or very dangerous work.


Now of course, this being produced by Boston Dynamics, it displays the robot in the best


possible light. But with any technology, there are good applications, there are bad applications.


I think it's cool that technology is being pushed forward. And I'd rather have spot in


this world than not. But this was it for this week's ML news. I hope you enjoyed this one


and I'll see you next time. Bye bye.


Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (+Author)

Hello there, today we're looking at language models as zero shot planners extracting actionable


knowledge for embodied agents. And I'm going to interview the first author Wenlong Huang


in a few minutes. So first, there's an explanation of the paper 1015 minutes or so I'm gonna


try to keep to it. And then we jump into the interview where we can discuss this paper


at length. On a high level, this paper asks, can we use the knowledge that is inherent


in large language models like GPT-3, or surprisingly, open AI's codecs in order to do planning in


what they call embodied agents. Ultimately, it's going to be this environment right here,


the I don't even know what it's the virtual home environment. And it's about a virtual


home, you have to fulfill some tasks like brushed your teeth, then the model has to


come up with a sequence of steps that are admissible by the environment. So there's


a level of admissibility of action, predefined actions that are admissible, the model has


to come up with these actions in order to fulfill the task, the model is then rated


based on executability and correctness of their plans. And it turns out that the larger


the models get, as you can see right here, the less executable the plans become, which


means that the actions they generate aren't admissible by the environment, probably because


the models are more, let's say, powerful, they can express themselves in more ways,


they have different ideas of how to reach goals. However, the correctness, this is human


evaluated of these models rise as they grow larger. So this gives you an indication that


the large models seem to have quite a lot of knowledge. And we have to say these are


not trained, the entire paper just works except for one baseline evaluation just works with


pre trained models, they are not fine tuned at all on this environment right here. So


what this paper does is it says, well, given that the larger the models get, the more correct


their plans are, can we do something to fix the issue with the executability to that they


develop these translation procedure right here, these are three specific improvements


they do to the models in order to get their executability up, you can see they sacrifice


like a little bit of the correctness, but they do make the plans largely executable


in the environment. And therefore, procedures like this could be applied in many different


ways. It's not only about the virtual home environment and so on. It's essentially anywhere


where you bring together the knowledge that is inherent in large language models with


some sort of a domain specific language or a grammar or any anything like this, like


where you have to transfer that knowledge into a new domain, but you don't want to train


a model to do so. So we're going to see how they do it really briefly. First of all, the


environment itself, as I already said, is this now this is visualized, although they


never work, you know, actually in 3d, just a small correction here, because I messed


this up. There are actually two versions of the virtual home environment. One is a Python


version that focuses on the textual interaction with the environment. The other one is implemented


in Unity and actually does work in 3d. The developers of the environment mostly focus


on the Unity environment because it's more real. But as of yet, that has a subset of


the actions available that the Python environment has. And the authors of the paper use the


Python environment and the data set that comes along with that. We're going to go into this


more in the interview. Stay tuned. They simply grab the data set of possible tasks, some


tasks you can see right here, a task could be throwaway paper, another task could be


brush teeth, and there there'd be a sequence of steps. This environment is made by humans.


So the tasks are made by humans, and then other humans have to come up with the steps


that are admissible, admissible actions in this environment. There are, I believe, a


number of objects that are defined, they're predefined. Yeah, so there are a number of


objects, for example, living room, television, sofa, and so on. And there are a number of


verbs. So walk, find, switch on, and so on. And not every verb object combination is possible.


Some verbs have two objects and so on. But essentially, you combine the predefined verbs


and the predefined objects, and then the state of the world changes. So the world keeps track


of states, there are certain preconditions. For example, you can probably only sit on


the sofa if you are in the vicinity of it. So you need to first find the sofa, you can


only switch on the television. Similarly, if you have first founded the television or


walked to the television or something like this, if the television is in the living room,


you first need to go to the living room, and so on. So there's a hidden kind of a state.


But all of this is constructed. And we talked about this in the interview, like, what's


the appropriate granularity of actions like this? And isn't this a major issue, but it


is made all with the humans in the loop. So the data set is supposed to be kind of the


most natural expression of these tasks as split into steps that a human would come up


with. So this is the grammar of the environment. And the language models, they don't, they


don't know about this grammar, they're just language models. So what they do is they take


something like GPT-3, and they make a prompt. Now the prompt, as you might know, in GPT-3,


you have to give a prompt. So the prompt could just be like, here's the task, you know, blah,


blah, blah, brush your teeth, then what's step one, right? And then GPT-3 will probably


it will probably even generate step two, and three, and four, but it will probably not


be according to the these actions in these templates, you can help this a little bit


by putting a prompt up here. So the prompt they use is one, I believe, one specific plan.


So they have already like task up here, some task, and then some number of steps, so that


the model kind of knows what is expected. We also talked about this in the interview,


and this could potentially be improved by multiple, multiple prompts and so on. But


in the baseline, they have one particular prompt, and then one of the improvements is


actually to select a more optimal prompt. This is the basic setup, you have a goal in


this environment with a fixed grammar, and you task you input this right here to your


language model. And the language model will spit out the plan. Now, what do you do with


the plan, the plan, you score, like how good is the plan, and they have two different scorings


available. One is executability. And executability is just like, it's essentially parsability


by the environment. So in executability, you ask yourself, can it be correctly parsed,


which means that is the syntax according to the syntax of the environment, and they do


have a little translation procedure, like a little heuristic translation procedure for


the baseline in place, so that the language model probably can't get it exactly right.


But they do sort of translate to the closest action there. But also one of the improvements


is related to this. And then also does it satisfy the common sense constraints of the


environment. And these would be programmed in like, for example, you can only pour yourself


a glass of milk, if you first open the fridge and grab the milk, this can be measured directly,


what cannot be measured that well is correctness. So these models, they would come up with plans


and independent of whether they're executable or not, they could be correct, right. And


that's where they ask humans. So they use human evaluations, they conduct human evaluations


in order to score the correctness of whatever these models output. So they give it to human


ask the human, does this look like a sensible plan in order to brush your teeth, and the


human would either say yes or no, when they do like ablations, and so on. They also use


like longest common sub sequences between two programs and so on in order to not spend


ginormous amounts of money on humans. But essentially, the correctness metric is a human


metric. It's also interesting, because you thought you could just execute like the plan


in the environment, and that gives you like, does it succeed or not, but they say correctly


that for a task like make breakfast, there's not really a defined end condition that you


could program into the environment to give a reward. So it's more accurate to ask humans


whether a plan is correct. As you might have guessed, this environment is very human centric.


It's made by humans, with humans in the loop and so on. It's supposed to really be sort


of a representation of human tasks and human plans to human tasks. Alright, so now we're


going into the improvements, there are three distinct improvements they make. So if they


just do this, if they just do what we've described so far, then the graph up here results, excluding


the two models on the right, you can see the larger the models get, the higher their correctness,


but the worse their executability. So now the thought is, can we change that? Can we


raise the executability? And so this is the baseline right here, zero shot planning via


causal large language model, you put in a task as a prompt, and along with like the


format you expect, which is this one right here, which is some other task from the data


set, then you use the pre trained language model like GPT three or something. And that


will give you a plan. And that's it. So the next thing they do is they do what they call


a translation model. So they introduce a second model, which is also pre trained. And this


is it's not trained on translation, it's just trained on masked large language modeling.


So think of this, like, this is just BERT. In fact, I believe they use sentence BERT,


just pre trained on English language. And what they do is they make a big vocabulary


of all the admissible actions. So all the admissible actions would just be like any


combination between any verb, and any object that would actually go with that, that is


admissible to this verb. So from this, they make like a giant list of all of the admissible


actions. And then they embed that giant list. So they put this into some embedding space


using the sentence BERT model, pre trained, right. And then whenever the large language


model output something, they don't implement it into the plan directly, they first embed


whatever the model outputs. Let's put this over here, they embed it, let's say that becomes


this right here, then they see what's the nearest neighbor of my admissible actions


to this thing. And then they simply replace whatever the model output with the nearest


neighbor. And they call that they call that translation. So essentially, it translates


from general natural language space into the space of the admissible actions or the grammar


of the model. Now, this has some problems on its own. For example, if the model outputs


the compound actions. So if it says, for example, squeeze out the glob of lotion and put it


in your mouth or so or on your face, I guess, then well, it's apply lotion, it's anywhere,


squeeze out the glob of lotion and put it on your skin, that would be still one action.


Now which one would be the closest right here, there's going to be somewhere like, squeeze


out a bit of lotion, and the other one is going to be like, put the lotion on your skin.


Yet you only have one action, like it's, it's, it's one line. So one action, it just contains


like an end. Now the end might be easy to recognize, but there are other, there are


going to be other like compound actions. And this is going to be a problem here, because


you just map one action to one admissible action. But in any case, doing this already


helps a lot, even though there are still some problems. To alleviate the rest of the problems,


they have two more improvements. The first improvement they do is they say, well, if


there's a compound action, we can still kind of alleviate that a little bit. So in the


original method, what they did is they simply took this through the through the language


model, and they got out just a list of steps, right? Here is step one, here is step two,


here is step three, and so on. That is just a list of steps. And they would translate


even when they use the translation model, they would translate each of them to a admissible


action, translate this one to an admissible action. Well, now you have no idea of whether


that sequence of admissible actions even makes sense, right? For example, one could be a


compound action, and it just gets translated to one of the two actions, and then the next


action doesn't have a precondition. So what they do is they interleave the two steps,


right, they interleave this translation with the generation. So they would only generate


one step at a time, like step one, then they were translated, and then they would use the


translated version and put it back into the language model to get step two. That way,


the language model always is conditioned on admissible actions instead of just being free


form and then translating after the fact. So this is autoregressive generation. The last


improvement they make, which is, I guess, more of a minor improvement, that's why it's


not in this diagram. However, what they do is instead of having a generic prompt, what


they do is they take the task, they embed it using the same sentence bird embedding,


and they compare it to embeddings of all of the tasks that they have in the data set.


And they just pick the closest task in the data set to act as a prompt, which could still


transfer some in context knowledge in, you know, for the current task. So that is essentially


the method, they investigate this, they have a algorithm right here, they also like they,


it's I'm, I formulated it in a rather easy way, but they do not only consider the closest


action they consider actually a weighting of in so in the translation, they consider


a weighting between how close it is it to an admissible action, and how likely is that


action that they that they output. So they would generate not only one action and then


translate it, they would actually generate a bunch of variants. And they consider each


one of them like how close is it to an admissible action, and also how likely is it and then


they take the best combination of the two that is obviously modulated by a hyper parameter.


They have early stopping and all of this kinds of stuff. And this results in this results


in a neat, in a neat algorithm that and we're going to talk about these things in a bit


and also the also the results right here, I just I want to highlight that if you look