meaning you can try to solve one task but in the meantime do Q learning against the other task that has a different reward function and that's the key idea here we're gonna take reward functions that are not the ones we care about that are auxiliary reward functions that are easy to automatically extract from the environment and the Q&A against those who word functions and by doing so the core structure the core of the neural net will learn things that are also useful for the task we actually care about okay


so what's actually that a little deeper here so basically she agent is the core thing the sparse awards means you just I mean here's the cake from now on Laocoon you get this one T round the cake from the real reward that we care about that's not enough we want more rewards that's exactly what this cue function thing is going to do we're going to define many other rewards and there's many other rewards are going to allow us to learn from a lot more signal and if you only had our one reward okay so this reward function that was defined here by

L12 Representation Learning for Reinforcement Learning --- CS294-158 UC Berkeley Spring 2020


the office of the paper is called a pixel control reward function and so what they do is they turn what the agency's first-person vision of the maze into a courser this gives grayscale and a representation of what it's seeing and you get rewarded in this auxilary reward task for how much you're able to change the discourse pixel value so what does that mean if your agent turns into a direction where things are much brighter than the direction they're facing right now then the pixel values will change a lot and that would be a high reward


thing or other way around if right now things look very bright in a certain pixel in it it in turns and makes that pixel darker that would be a high reward again that's not what we actually care about but it's a very simple auxilary loss that we can impose and that we can run q-learning against and so that's it also turns out that this is the one that mattered the most for improving the learning there are other Attilio losses with is the one that matters the most intuition here why this one matters the most is that in Q


learning you are learning about the effect of your actions and so because you're learning about the effect of many possible actions you could take would be the Q value if I turn to the left well did you follow that turn to the right what if we the Q value I look up look down and you're really learning something about how the world works and not just how the world works with how your actions interact is what happens in the world another usually loss is reward prediction so what you do here is for your current pause that you're executing


you can try to predict a future time steps how much real reward you're going to get so maybe you get rewards for eating an apple and so when you see an apple in the distance you should be able to predict that if you keep running forward for three more steps you'll get that Apple and so learning to predict that in three steps you gonna get that Apple is an auxilary loss that's introduced here and then elastic zero loss is value function replay so it's saying from the current time how much reward am I going to get over the next


few steps so this applies in theory in the base every see agent actually all right so if you look at results here this is undefined lab which is that 3d navigation environment where you collect apples and other fruits as a reward we can look at different approaches so the bottom curve we are looking at is the base a tree C agent so that's the dark blue bottom curve and the hope is by having auxiliary losses we can do better if we incorporate all the ideas that we just covered you get the unreal agent you get this top lead curve here and now


we see here is various operations to see well which one of these matters the most which ones might not contribute very much what we see is if he does do the human 4 pixel control it's a loss that's the yellow curve you get almost all the juice of these area losses but if an addition you have the reward prediction and the value replayer you have yet a little better performance so actually another thing I want to highlight here the butts on the top of the graph here says average of the top three aging so there's a way to evaluate things in the


paper usually reinforcement learning because you need to explore and there's a lot of random is an exploration the results are somewhat unpredictable of meaning that some runs will be more lucky than other rooms it'll be high variance and so with the baby here they pick the top three grunts might say why the top three that's a lot crazy shouldn't you look at the average performance anything like that yeah you could argue should look at the average performance it's what's done in most papers that our thinking here


was then imagine what you're interested in is finding a policy and you have maybe a budget of doing 20 runs then maybe what matters is what's the best one among those 20 runs or it could be a little more robust aberrancy you know how do the best three runs do and so an approach where the best three runs are consistently great then that's an approach where if you won't afford 20 runs total you'll have a good one among them and so that's a it's kind of a funny way to score things but it happens to be how they do things in this paper


so another thing that compared with which we as unsupervised learning students of course we're very clear about if you do pixel control why not do feature control why not cue function for kind of later layers in the network if for later lays in the network I want to see if I take an action you know can I change the future value and maybe layer five or layer six instead of just pixel value change well we see a 3c plus feature control in green and it received plus pixel control in orange you can see the pixel control actually works better


of course this might depend on the environment this might depend on exactly only architecture that the experiments that we're done in this paper showed that pixel control actually slightly outperformed feature based control and again control here means it's the auxiliary loss using the auxiliary cue functions that the ultimate reward function that you're actually optimized for and score against on the vertical axis here is the real reward function of collecting the fruits in the maze then here are a couple of unsupervised RL


baselines so what are some other things we're looking at again a principal pixel control shown in yellow that's the top curve and both plots then input input change frequency just try to have an auxiliary law that says can I predict how what I see will change as a function of my action so that's really learning a dynamics model that's shown in blue and then shown in green is in Petri constructions that's a bit like a Dae I have an input make a little representation reconstructed back out and so what we see is that these


things that might seem maybe more natural and more advanced like infantry construction input change prediction are actually less effective than pixel control and of course I mean there could be many many factors have plenty here but the high level intuition that most people have is that the reason these are clearly cute functions work so well is that what's happening here is that as we as we work with exilic q functions we are we are actually we're actually learning about not just how the world works which is the input change


prediction but how we are able to affect what happens in the world and that's really what matters for learning to achieve high reward than the task you care about now domain they looked at rather than first-person maze navigations Montezuma's in French on this month's range of famous at our game where expression is very difficult there are many many rooms in every room there's complicated things you have to do collecting keys jumping over things they make one mistake you're dead and you start back at the beginning


shows you that unreal outperforms it you see quite quite a bit a 3 CR at the bottom I think black is really not getting anywhere where the unreal approach is actually doing a lot better now let's take a look at this maze navigation agent in action so this is deep mind lab let's take a look at the agent plane you agents collecting the apples here not collecting the lemons that's apparently it's not good in this particular game to collect the lemons and so this agent has learned to navigate mazes the way it can


do that by de Rais because it has a lsdm which gives a memory and so you can remember places been before and things has tried before to more efficiently find the next new location where there might be a fruit it hasn't collected yet and so the reason I'm showing these specific results here is because the space of well reinforced planning in general but especially representation and reinforcement learning does the evaluations aren't all in the same type of environments there's a lot of variation how these things get evaluated


and so having a good feel for what these experiments actually look like is important to get a sense for how advanced might this method really be and so we see here as well this first person navigation well that's pretty complicated so this might be a pretty advanced method that play here here we see a bit of an inside look into the agent itself where on the top right you see the pixel control Q values is something depending on which action that take this for actions available how much how high will my Q value be which really


is corresponds to an understanding of how the world works what will change in what I see as a function of actions I take all right so to summarize the unreal law since the original ATC loss which is a state policy gradient most value function loss then there is value replay loss which look seven replay about her Valarie prediction and then there's the pixel control two functions for different course pixels in the view of the agent and then finally there's the reward prediction loss small opening in the word prediction they ensured was


an equal portion of rewarding and non rewarding examples so a balanced training set the pixel control did split into 20 by 20 rid of cells all right so in the entire results we see that unreal also helps over a 3 C but not nearly as much as in the deep mind lab environments but still a significant improvement the vertical axis here is human normalized to form a server the way deep mind is evaluating this is they look at what you missed in 13 Natalie for every game that's gonna be a different score because every game is


very different scoring system and then they normalize it across games another total score across all games how well the agent learns so it is across many many Atari games in terms of Atari games on average how fast is the learning curve go up so you cannot overfeeding onto one game or the other and do well on this score you need to be able to learn well on all of the games to do well on this score alright and then also look here at robustness because there's many agents being trained and this top three curves on


this le expose performance of all the agents here is a little evaluation of robustness and we'll see that you know there's a bit of decaying performance not all agents learn equally well but it's not that there's just one death as well and then nobody else as well so this looks pretty good okay so that's the first thing I want to cover which is auxilary losses and unreal is a very good example of that does more work happening all the time in this space but that was kind of the big initial result that's showed this is something that can


be very beneficial let's switch gears to state representation an ominous will have many many subsections it turns out first one we have is how would go from observation to state so the the kind of paper that most people might be most familiar with is the world models paper by David ha and collaborators and here's very kind of a simple diagram showcasing what they investigated so what you have is you have an environment the environment leads to an observation in this case pixel values and that can be very high dimensional fear but I'll take


100 by 100 image that's 10,000 pixels that's a very high dimensional input then they say well that's kinda Michelin we want our agent to work on something lower dimensional because we know under the hood there is a state of the world and the state of dog might be summarized wearing just a small set of numbers maybe 10 20 30 numbers is enough so my agent shouldn't have to operate shouldn't do reinforcement on those 10,000 number input should be doing reinforced learning on dusty 30 number input and might be able to learn a lot


more quickly because credit assignment should be easier if we only have to look at 30 numbers instead of 10,000 numbers and so it is a you risk a strain of racial auto-encoder which will of course cover them in their release of this course to find a latent representation now from which we can reconstruct there is more but then we'll use the latent representation as the input to the reinforcement learning agent which now hopefully will be more efficient so will them do in this approach is train a recurrent neural network now learns to


predict the next latent state so what's gonna happen here is we're gonna learn a way to simulate how the world works in this recurrent neural network but not by directly simulating and pixel space but by simulating in its latent space which could go a lot faster if there's a lot lower dimension we don't have to render the world at all times we can just simulate how the latent variables evolve over time of course this will also depend on the actions thing so it's a Mulligan's interaction and previous latent state generate the next life in


state and of course you want it to be the case that that matches up with the actual next latent state that you're VA you would output when you get to observe the next live in state and then he actually gets fed into the environment also and this so you have kind of two paths here at the actual environment path and you have the RNN prediction path and you hope that they line up or you training really to make this line up the thing in blue is called the world model is a thing that looks at the latent state see turns it into by


looking at the action latent state turns it into next light instead alright so they looked at this in the comics of car racing so in the left you see the environment that drains the roads you're supposed to stay on the road here the way they were rewarded shut up and race down this road as quickly as possible this is from pixel input so you get a lot of numbers as input and somehow you hope that would get turned into effectively an understanding of roughly where the road is where your cars on that road and which direction it's


facing your car and then understand how to steer it to go down the road as quickly as possible procedure that followed is that click 10,000 robots from a random policy and the trendler view to encode frames into Z space just thirty two dimensional z space so low dimensional compared to the pixel open space then they train a ironing model to predict next lengthen state from previous latent state action and there's this additional hit late instead inside the Arnim then they use evolution but it's just one of many possible RL


approaches to train a linear controller to maximize expected reward of a robot so step one two and three is the unsupervised learning that can happen ahead of time and then you can run RL on that representation that you've learned so one thing that's real interesting here is that remember the Omnitech where yaw would say oh well you know reinforced learning is a cherry on the cake which is tiny compared to the cake and why are something for spent in the chair because it's not a lot of rewards it's just small amount of reward signal


the following york response to be bound a signal there's a lot of signal coming from self supervised learning and that's the foundation of the kick and so if you look at what's happening here the VMA neural network has four million parameters the RN and dynamics model network says no network has four hundred thousand parameters and then the controller the thing that is learned with RL only has eight hundred something parameters there's a massive difference in that RL only has to learn a small number of parameters which Maps do it


only have a less lesser amount of signal whereas the saw survived part has to learn most parameters millions of parameters and that's done from one to device theta okay so here's an example of an input frame the 64 by 64 pixels here and a frame reconstruction which kind of roughly matches up not perfectly but it gets to just then here we have when we use just C or Z on h h is the ardennes in that state so it shows that it's important that the RNA and hidden state captures something important about the world let's look at results so what we


see here in the table is scores obtained with the model described highest score in this car is the environment compared to previous methods obviously in principle unlimited she be able to learn this too but when you limit the amount of time you get to train then using cell scribes learning to learn a representation combined with reinforcement to learn the control allow us to get seemingly higher scores than previous methods there were pure RL were able to do so this is the model we looked at before one experiment we saw


so far I had the car racing environment the second experiment where you have to dodge things being shot at you in a fist doom environment so the input will look something like we see on the left but sometimes you'll see fireballs coming at you when they're shooting at you and you got to dodge those fire bullets to get to stay alive and get high reward same approach train of uni training our end world model then linear controller train with our L on top of that and so again this linear controller train on top of


that is trained in the Arnon simulator itself so you don't need you don't need to simulate what things will look like rendering is often expensive computationally if you need to go all the way to rendering to train your policy I'll take a lot longer to do the same number of rollouts their oldest thing that low dimensional latent space to train the policy so it's called doom take cover here's a higher resolution version of what this looks like if you were to play this game yourself same approach laid out here again


unsupervised learning does all the stuff at the top here millions of parameters learn then the RL only needs to learn about a thousand forever again beautiful association of the non-linear cake idea so here is what here's what this what this looks like one thing to to keep in mind here is that it actually sometimes you can you can have some quirky results where the simulator of the world allows you to do things you can now do in the real world and so that's something to look out for that they're highlighting in on their


website so if you go look at the kind of normal temperature higher temperature things you'll see some differences there so here are the results we have depending on the temperature different discrepancies so for low temperature we see a very high virtual score but the actual score not so great for higher temperatures we have a closer match between the virtual score in the actual score so actually actually I should I quickly highlight what would meet with temperature here so typically in RL you have a policy that has stochastic output


so you would have a distribution over actions and that solution over actions can have a temperature parameter in terms of how much you favor your favorite action and so that temperature parameter if you make it small low close to zero then you'll always think your preferred out most preferred action when you have then you end up with a close to the domestic policy we have a close to domestic policy you can often explored quirks in your simulator whereas if you have some random is in your policy you have a higher temperature


if ourselves a little bit of randomness then you could not exploit the very specific quirks and lauren simulator because the randomness will prevent you from being able to go to that very very quirky path where you all of a sudden get a high score even though you know really you can't do that in the real world but your simulator has a small little bug you won't be able to trigger that small little bug and that's what's going on here with temperature at higher temperature we are not able to exploit tiny little bugs my learned simulator we


have to learn something more robust and that leads to a better match between performers in the real environment relative to the learned simulator ok so that was the world models paper by David hein collaborators now one question you could ask yourself if we're going to learn a world model we're going to learn a simulator some Lincoln space simulator couldn't make sense to try to learn a latent space such that control becomes easier what I mean with that so if you look at the control literature some control problems are easy to solve some


control problems are very hard to solve and maybe we can map our pixel observations and world and amps and pixel space into a blatant space dynamics that satisfy certain properties that make the resulting control problem easier to solve a good example of this is linear dynamical systems if you have a linear dynamical system then the control problem tends to be relatively straightforward to solve so how about this dinner this is what this paper we're going to cover here is going to do is hold on give me one second here


[Music] let me cover something else here first so one thing that might happen is as you train the world model on your randomly collected data and then turn your policy and test it in the real world it might not always work the reason might not work is because the randomly collected data might not not have been interesting enough to maybe cover the parts of the space where you would get high reward and so what then you'd want to do is iterate this process at this point you effectively a model-based reinforcement


procedure you collect data you learn a model you find a policy in the learned model you deploy that policy you take a new data and prove your world model and repeat so that's what they did and this gives for the carpal swing up and so after about twenty trations with this we would be able to learn to swing it up now a couple of other world models papers is the actions additional video prediction using deep networks in atari games just at the top here worth checking out model-based reinforced planning for atari it's another one


worth checking out and then learning the dynamics from for planning some pixels planet which we'll look at a little bit later also so if you want to look more closely at the specific it which is covered there's a really nice website world model start github dot which has the code which has many demos for you to play with the play with what actually the latent variables are doing in the VAD and so forth for these alarms so highly recommend checking that out and here is a video of the chris doom cover in action so you get these fireballs


coming at you and the agent has learned to get out of the way to not get killed all right so we looked at so far is we looked at how to go from observation to state and then learn a model in that latent state space now we're gonna take a look at program division to state but also from state action to next state so and this was that earlier alluded to an hour i jumped the gun a little bit on this yes we're gonna now be representational and that's not ahead of time learning your position and pixel to hopefully state or


something like state but that is already when it's doing representation looking at the dynamics and so when we look at the dynamics to representation learning why not learn a representation where the dynamics is such that control becomes easier for example learn a representation such that in this new representation space that an Amex is linear because if the dynamics is linear then all of a sudden control becomes easy and you turn your original pixel space problem might be highly nonlinear very complex to have a control methodology for into a


latent space problem where it's just linear and very simple to solve that's the main idea behind is embed to control paper we're covering now so the existence they considered were pendulum card pull and three linked arm but again this is from pixel soda a pixel input when the lid representation where hopefully dynamics is close to linear and hence control becomes easy so it's called stochastic control the methods they apply it's kind of a standard control method then you can apply to linear systems and embed to


control will learn a latent space model using pressure on encoder while forcing a locally linear latent space dynamics model so once you have a local inner model you can apply stochastic optimal control is an example that in action where once you have such a model is very easy to find the controller that brings you to a target and say a stable fixed point thanks to that controller or just to work well locally along this trajectory you seem to have linear dynamics models and in fact the way this methods work is they


tend to linearize the dynamics along trajectories but when if you learn a latent space model where it's already linear we already good to go and that linearization will not be an approximation or actually be the action model that you learn to that be nice to have a very big fit of your linear model to the absolute veneks so the costs are often assumed to be quadratic so that's that's an assumption to make you know this class of problems called lqr problems later through out of control problems sometimes LQG problems if you


also have some use to bestest be in there and these problems assume that you have linear dynamics and quadratic costs without a cost meaning there's a quadratic penalties for being away from the state where you're supposed to be okay so of course we can't just throw from our original pixel observations to some space where the NamUs is linear and ignore the real-world dynamics esta map button my lab back out to real world to them so let's look at the complete loss function to look at first of all go to latent space see you need to be able to


reconstruct the original language so Z should not lose important information about what is happening or what's the situation here then we have this temporal aspect here and I'll Polly want to reach a goal and want to have long-term act prediction that in the end put the sequence of actions that are keys the goal it also predicts that's going to be the case so every step along the way we're gonna have prediction for when and use linear models so prediction must be locally analyzable for all valid control magnitudes such


that when we optimize our controls we get something then when it works in simulation also works in real world now we're going to force that to be true by learning a model that does this by construction so let's look at that model here's the next component we already have our encoder decoder we have our control input u so in controls usually he was used for control input and reinforcement often a is used for action controls need for controls then we have our next latent state CP plus 1 now for this to be meaningful the same decoder


should be able to reconstruct the image input at time T plus 1 if that's the case then that latent space dynamics was correct okay so we're going to learn a locally linear model here in that transition to make that work okay then once we have all that in place we pretty much good to go we're going to use this model over long horizon C to make sure that actually that we don't just do this over one step we actually lay this out over and longer horizons and as we've trained the model we have multi-step predictions over which we


have this loss function you might say why do we need why do we need all this well it turns out that if you make a small mistake and your prediction for the next state then you might say nah Bob just a small mistake no big deal but if you make a small mistake the problem is that you land in a new latent state for which your model might not have been trained and when you make the next prediction to go to time T plus 2 you're doing it from a time T plus 1 latent state that you're not familiar with that doesn't lie in


your training distribution and now you might make a not so good prediction make it even worse and this is an accumulation of errors over time can lead to divergence and explicitly avoided any kind of simulations to run over longer horizon need some mechanism to avoid that okay so one mechanism is to explicitly have a loss of a multi-step another mechanisms to ensure that your next state prediction comes from the correct label distribution so if you embed into my Tate unit Gaussian spins then after you do your next state


prediction they should also that what you get there should come from a Gaussian unit gosh and distribution to ensure that when you go from there to the next one you're ready to make your predictions all right so those are the components we have an autoencoder tutoring image X into latent state with accurate long-term vision of latent states because we ensure that the next latent state comes from the correct distribution a unit Gaussian just like our auto encoder it forces it to be and then the prediction must be locally in


erisa bowl so we don't get some fans in their network to predict the next flight instead from the current day and see if it has to be feasible with just a linear prediction okay so this is the full system that they proposed all the last term's shown at the bottom now let's take a look at how all this works so they apply this to cart poll that showed a good amount of success in car pool and then here are some evaluations on on that showing that embed to control indeed can do in virtual pendulums swing up pretty well I can do carp or balance


you can do three link arm so good results on three environments that they experiment with and here's this environments look like this is from broad images so what we're watching is effectively also what the agencies the agent will often see the down sampling so they can actually look at the the environments themselves so really much on the left because worked on the right and here we have cardboard balancing in action and so this can use some idea of how capable this approach is so it does very well at the same time clearly these


environments they're not nearly as complicated as what we saw in the Unreal environments where it was deep mind lab navigation tasks versus these kind of 2d relatively low-resolution single robot that you fully control kind of tasks now in embed to control the idea was to have a single linear system and for your full of enemies that might be difficult but it's been shown in controls is that very often even though your real system is highly nonlinear locally it can be linearized and so you might ask the question can we instead follow the same


philosophy and I was in bed to control but instead of learning the single linear model can we learn just a collection of linear models in some way that allows us to apply time varying linear control methods which are also extremely efficient and maybe have a richer set of environment that we can solve for because time varying linear models can cover more than just a single linear model can that's actually what we did in this work called solar showing an action on the right here we now have different linear models at different


times and so we learn to embed into space where at each time a local linear model can capture the transition very well so you still get initial random rollouts followed by learning representation and latent dynamics but now I'm not a simple linear model but the sequence of linear models and then from that once we've done that we can start doing a robot infer where we are in this thing's getting out of in your models find the sequence of controllers execute that get new data and repeat the smaller base to draw in action a model-based reinforced


waiting in action on the distant if we're setting where we make the latent space very efficient to find optimal policies and so might not succeed on the first time around so get the new data update the representation infer where we are in terms of linear dynamics models and trying to updated policy and repeat and this can actually learn in about 20 minutes to stack it like a block learning it from from pixels as input okay so we looked at state representation which is how to go from broth acceleration to state learning


ahead of time with a few in the world models paper that we looked at after we learned dynamics model and mapping from pixels who stayed at the same time and maybe benefit from that now here's another way we can think about this which is we could think of putting some prior information so when we have pixels as inputs and we know that under the hood there is a state thing we know that state is just a bunch of real numbers so we did here in this papers is said okay when a plug data we're going to learn a latent


representation which is curated by sequels of column filters then we're going to apply a spatial softmax meaning we're going to look for each of these 16 filters where each filter is most active through a spatial softmax and output the corresponding coordinates those coordinates should allow us to reconstruct the original image because they captured essence they recorded to the objects in the scene if you know the courts of the objects we've seen at least as the home you can reconstruct the scene and then once we have learned


that representation we could learn to control with just a 32 dimensional input rather than needing to take in 240 by 240 input which is much higher dimensional and much more expensive to do reinforced fighting against there's actually capable of learning a pretty wide range of skills here so here is the data collection so it's just randomly moving around collecting data that data is used to train that spatial auto encoder and sir then we learn we look actually we imprinted the goal situation and then we do reinforce


learning in the feature space the thirty two dimensional feature space and learn in a relatively short amount of time how to push the block to the target location it's not ready you can follow and how to go from image observations to state or something likes hearing kind of interesting method here them it actually doesn't bother reconstructing it says all we need to do is think about physics what is physics tell us well we're gonna want to find an encoding of state underlying state coming through the observation fine here will be the big


neural network that turns image observation into underlying state well what do we know about state we know that in physics then there will be coordinates and then derivatives of coordinates which are the velocities of these objects so there is a state variable corresponding to velocity and other severe I'll compare corresponding to position and the change in position is velocity that's we know that velocity is derivative of position then what else do we know we know that when the world to be in different states we're going to need you


know different state values so by default if we you know get random stuff presented to us we want the embeddings as in a field two different situations to be far apart so that's what this law is is saying we want embeddings to be far apart but in all you do is play the embedding is far apart well then that's not enough to get any structure so then the next loss here says that four consecutive times the position state variables should be close it also says that between time T and t minus 1 the velocity state variables should be close


because philosophy cannot change quickly so this is saying acceleration people is going to be small on average an acceleration is gonna be small then conservation of momentum and our energy is captured in here and in the last part here Singer we need a representation where the actions are able to influence what the next state is going to be so wanted correlation between action and state all right so it tested is on a couple of environments you know where they would just collect data in these environments pixel input and then learn a state


representation that doesn't do reconstruction just try to satisfies those invariants that are expected to be no good loss functions based on physics and one pretty interesting state representations that way here's another example of state learning in action going relatively quickly earth-2 just gonna give it a lot of different ideas across we've covered the beta tae and one of the early lectures beta V is a very solid encoder we'll put a coefficient beta in front of the KL loss on the prior and by making that


collision beta bigger than one effective what we're doing is we're trying to make the latent variable Z maximally independent so we're trying to find a disentangled representation of the scene and so the thinking here is that well if we want to find something that we think of our state from raw pixel values and probably we need to find something that's really strongly disentangled and so it's putting that prior into it and they show that by having this beta V you actually get much better transfer so they train a beta vini and then do Q


learning and bit in a network that takes the embeddings from the beta ba and compare it with regular Q learning and so on the left here we see what happens in the training environments the training environments regular Chi learning and Darla's which is few learning with the beta V representation learning do about equally well but when we look at a new task related task but looks very different by doing the representation line which is shown at the bottom right we have to get much better performance top left is


actually not getting a job done it's not collecting these yellow targets whereas bottom varieties we look at at collecting yellow targets and what's changed while the walls in the background have changed to pink rather than green and the ground has changed the blue rather than yellow so a relatively small change originally QN this doesn't do representation learning per se hasn't learned those notions whereas stability has somehow learned a representation that allows it to transfer here at zero shop to this new environment much better


it's not our idea of representing first state and dynamics we looked up Gantz and I think lecture four of this class now if you just train again what happens is that you just transfer each frame independently what we want this we want to learn but there's intentions that are consistent over time as we're going to do is we're gonna have a discriminator here that looks at two consecutive observations and decides whether those two consecutive observations are consecutive observations from the real world and welcome secular observations generated


by a generator and so if a generator here trying to generate fake sequence of observations trying to fool the discriminator and at convergence what that means that this generator is trying to generate observation sequences that are indistinguishable from real-world observation sequences once you have done you can use that generator as a simulator and learn that simulator or planning that similar in this case we did planning to try to achieve goals we will see on the right here is we didn't actually did this for rope manipulation


so on the left is the initial spin of the road on the right design end state of the rope and we see us with causal info again it thinks that these are the interpolated states so it thinks that this is a sequence of states you have to go through to go from the initial state to the end state same for the next one next one next one compare that with VC GM which we also which would currently just doesn't look at transitions just looked at individual frames we see that the interpolation here doesn't necessarily lead to


intermediate states that are that meaningful for a robot to try to follow that you know sequence of intermediate states and rope configurations to get some start to go and so we're able to by training in Foca which looks at realism of transitions rather than just realism of individual frames is able to learn a dynamics model in a latent space that we can use for robot to make plants now one of the first things we covered was the world models which showed that you can learn a latent space and then learn all right on top of


the latent space for dynamics and then learn a linear control on top of that of course that's a very new you think it's almost surprising it works in there what's so interesting that it actually does work in a range of environments but hopefully it's not not likely to be the final answer to keep it that simple and so here's a paper called planet learning latent announced models from pixels lesson planning in it so what's what's new here is after learned laden space 10-ounce model it's actually risk is not deploying a policy of learning it's


using a planner look using look ahead as a function of which sequence of actions do I get the most reward taking that first action that sequence of actions repeat and here learns the latent space encoding together with learning D dynamics also is joint learning of encoding and dynamics recently has been an improver that is called dreamer from the same office roughly and what they show is that you can actually run a limb planning in in latent space you can actually train a active critic and Leyton space simulator and that'll


actually do better than i'm the planet he also showed that the dynamics model you learn it's better in these environments to learning stochastic dynamics model rather than in the domestic dynamics model and that there's a two big differences between planned a dreamer going from planning to learning active critic agent and using a stochastic model now so far we talked about latent space models and directly learning to control in the latent space there is also work that actually goes back the image space and so here are


some example executions by robot moving objects to target locations and this is done by this the system here learned a video prediction model so i learn as a function of action the robot takes what will be the next frame i see and i long to the next action will be the next frame RC and so forth once you have a action conditional video prediction model and if you have a target frame or target property that you want to achieve you can now use this action traditional bigger prediction model as your simulator and this can give really good


results actually some examples shown here the slide get the downside of this that often planning to take a long time because to generate an actual traditional video prediction it can be fairly expensive we need to generate actually many of them because you're trying different sequence of actions to see which one might work the best and then after you find one that might work the best it might be a sequence of ten actions you take the first of those ten actions and you repeat that whole process and so these things tend to be


not as real time as some of the other things we looked at but it's very surprising how all this works you can do full a traditional video fliction and manipulate objects that way now one thing you might wonder is it's all good and well to do full detailed video prediction but is it always meaningful imagine you drop the bottle of water class ball of water drops on the floor how are you gonna do video prediction 1/2 for what happens there very very hard I mean you you're never gonna have access to all the details of


the state of the water in the ball all the little defects that might be in the mid you know water bottle materials and so forth that will determine how exactly this thing fractures the best you can be able to do is probably why I think it's gonna break into a lot of pieces and pieces of different sizes and you know maybe the the net tongue stays together because it doesn't hit the ground it's the bottom that's hitting the ground and so forth and you also don't need the details like to make decisions you just


need to know it's going to break and so what you could do say hey instead of learning a fully dynamics model and say I need to learn just what the future will look like be able to predict that you say hey what if I can predict what action I took for example seeing this shattered bottle and say well the action taken was dropping the bottle and so if I can go from I can make that prediction then I can also understand you want to achieve a certain goal what action might leave me there and not with me there this is called inverse dynamics and so


that's at the core of many other dynamics models that being learned throughout that for dynamics learn an inverse dynamics more effectively like learning a goal condition action strategy so no is it a paper here what if it said as follows it says we want to learn a Ford model and latent space I want to sleep in space - of course in fact will represent the things that matter but if all we care about is live in space predictions then the problem is that maybe we'll make our little space always zero and we picked always zero


we're always good but we don't have anything interesting and so they're gonna say well we want to learn a little space we would complete the next latent state but to avoid it being all zeroes or any other way of being degenerate we're going to require that from the latest state of the next time 50 plus one and the light instead of the current times et which we offered to predict the action that was taken at time T and so we went to dynamics models at the same time we're learning a inverse dynamics in a fluid dynamics


model at the same time in this light in space this is applied to learn to poke object so well you see here on the left is data collection you can set this up for autonomous data collection on the right where you see is the learned control so it's learned that the Namek law and now I can look at the current state and look at the goal state and it can do a prediction of which action is going to help the most to get close to that goal state and can repeatedly do that well it finally reaches something very close to the goal State okay now


reinforced planning is about reward so far ration mostly ignored the rewards when we learned representations and so we'll switch that up now let's not just learn to predict next state but also learn to predict future reward kind of first paper down or first recent paper that looked at this and the deep reinforcement in convicts is a predictor on paper so enter learning and planning and what they said is well it's difficult to know what needs to go into the latent state and so because we don't really know what


it has to when laid instead and we don't necessarily want to reconstruct the full observation because that's just so many things to reconstruct them we really want to focus on the essence well if what we care about is getting high reward should we just focus on predicting future rewards for every sequence of actions we can predict the future reward well we should be good to go then we can just thick secretive action that leaves the highest feature award and we're good to go predictor on did this for some relatively simple environments showing


here billiards as a function of which actually you tink how old the billiards balls and up and I should have pretty well on that task and also they looked at it for maze navigation now the most threesome is all the scores that you might have heard of that builds on top of these very directly is mu 0 mu 0 is also learning a blatant dynamics model that predicts rewards and doesn't worry about reconstruction and so this one here doesn't just given one action in the beginning what's the sequence of latent states that allow me to predict


reward in the future and use 0 same thing but now action conditional and was able to solve a very wide range of game situations I'm on a variation is successful feature so you might say it's enough predicting reward which is just one number what if reward consists of many components gave a clear about location of the robot maybe I care about energy expend maybe I care about other things these are all features and so the idea here is then if I had a set of features that relate the reward why not learn to predict well learn a latent space model


that allows me to predict the future sequence of features encountered we looked at this ourselves in the comics of navigation actually so when you have a robot that's navigating a world it does some convolutional processing of its observations then they'll be some lsdm because when you're navigating you currently see something we want also remember things you've seen in the past it's in memory here and then some that should try to predict observations features of observations they might in the future for example might have a


pollution or something like that here's this system in action so we're gonna so what we have here let me fast forward this a little bit to the experimental setup so what we see here is this is inside a simulator actually for now but also real world experiments coming later you see the kind of visual inputs this is processing and it's trying to predict things about speed hiding collision those are features it's trying to predict so I put it know this many steps in the future well my heading B will my speed be my


collision be based on what I see right now and based on the actions I will take any intervening time through that is able to learn in a total internal representation of how the world works but most importantly how the world works as it relates to features that matter for navigation versus try to learn everything about the world which might be a lot to learn relative to what you actually need to learn be successful at your task and so based on its able to learn to navigate these environments pretty well then so that the real robot


so here we have the actual robot that's going to learn to navigate the hallways in quarry all over at the electrical engine electrical engineering building at Berkeley so we see here and actually when it's still learning has a lot of collisions but it learns to predict that it learns something say if I think if I see this take that sequence of actions I will have a collision in five time steps or my heading will change in that way and so forth and so after training its internalize a lot of how the world works


now I can plan against a transition well I need to act now let's go to this is test let's learn now we can see that it's learned to avoid collisions and in terms of what it's doing it it knows to predict as a function of the actions taking whatever which is likely to happen or not well heading it might end up with and then take actions accordingly and again the reason I'm showing all these videos here is because as you see different approaches are testable very different environments and this by no means a converged research field and there's a


lot of variation how things get tested and by looking at how its tested to give you a sense of how complex an environment a certain approach might be able to handle now a natural question you might have is well this is all great there's all these different ways of learning representations but could we come up with a way of optimally representing the world what would that even mean what does it mean to have an optimal reclamation of the world well there's some worried especially trying to get up this so here are some fairly theoretical


to be fair references on trying to understand what it means the popularization of the world and one thing you'll often see come back is his word oma morphism and when it refers to is that essentially you have the real world you have a simulator and you want it to be the case ad if you go from real world to some weight and space simulator so you have a one-to-one match that's happening you go from from lit from real world to this latent space representation at that point you simulate in both worlds and then after a


while he tried to map back and see if it still corresponds but homomorphism would mean that you had still the correspondents many steps if he's are any number of steps in the future and so that would be kind of a by simulation homomorphism type approach and the question of course becomes what's the minimal life in space that you need to be able to do that just the more middle that laden spaces the less variables you want to do was as a reinforcement winner or a planner who tried to learn achieve good reward in the environment now one


thing that's very well-known in traditional controls is something called separation principle and separation principle in traditional control says the following and it's not well it's very specific snare it says if I have a linear dynamical system and I have noisy observations of this state so I don't have access to state I only have noisy observations and these noisy observations are linear functions of the state so linear dynamics observations linear function of this state then to do optimal control in this


environment where I don't have full access to the state all I need to do is to find the optimal estimator of state which will be a common filter and use data out with my best estimate of the state at every time combine them in the optimal controller assuming I have full access to state so the separation panel says I could several enzyme an estimator and a controller design them separately and then combine and that's actually optimal and that's actually very related things we've been talking about one of the representation I wanted to control want


to be the case I would do the right can representation when decision comes out of it and just be used with optimal control we get the optimal result and so you some work now trying to look at what you have a nonlinear system might apply deep neural networks and so forth what does it mean to have you know optimal estimation of state from your observations and how you know when is that compatible with your control and so forth so very interesting theoretical direction if you're more Theory inclined so another way to think of it is to say


well shouldn't I just think about it end to end so often in deep learning you have kind of two paths one path is you're gonna try to design something and the other pattern you say hey I'm just think about the result that one is the result that one let me define a loss function on the result I want and then training a staff instead of putting all the modules in more detail together myself so in this case what it might mean well instead of learning representation for a dynamic smaller and then bolt it on a planter or bolting on


a reinforcement agent why not say hey when I learned my dynamics model I should train it end to end such that what I learned is maximal compatible with a planner that I will use in the future so this goes a little bit back to the early thing we cover the embed to control we said if we can learn a linear dynamics model in latent space planning comes easy you're gonna say what a feel a more general plan and my mom and so that general planner might work well in a wide range of situations now can we learn a representation that if we combine it


with a more general planner together they function well if so then we learned a good representation so when we did this and some early work validation that works led by then post hoc Aviv tomorrow now professor at Technion we showed that actually the validation a very common way of doing planning for toddler Markov decision processes actually this validation process can be turned into a neural network representation and so we can then bolt this validation network onto a representation Learning Network and optimize them together to try to get


good performance out turning image input into a representation on which validation runs and the encoding of image input will need to be such down the validation process actually gives good results and we even gave the validation process and flexibility to learn parts of that which showed that this way you can actually get very good performance on planning tasks they might say well planning with visual inputs shouldn't just choose you just be able to learn a confident that just kind of looks at it and makes the right decision


well it turns out really if sometimes what we're doing here is building a very strong prior intercom by building the validation aspect into it that's a bit like why do we use a confident we'll use a continent to encode translation invariance and once we can learn more efficiently than if we were to use a fully connected Network it's kind of the same idea here we're learning that work that should solve a control problem that under the hood uses planning well then we should just put that planning into network the planning structure into the


network so we can learn it all and when and now one question that has often come up in this in this context as well she we ever do pixel level video prediction that's a good question I mean awfully you're just looking at noise and what's the point in trying to predict that what really matters is predicting the things that effect so how do you do that more directly so we're going to use plan ability as a criterion for representation learning now so validation that works as I just described go into a little more detail


it says have an observation observation it's turn goes into a module that outputs a value function which is how good a certain state is they put that out for every every state that you can hang on in parallel then an attention mechanism will look at the currents observation and understand which of all these possible stages should index into then make a decision on what to do in the current state the value iteration module looks at essentially validation cannot remember that sense what it does it needs to look


at reward and dynamics model and then this and rewarding dynamic smaller can do a recurrent calculation to get out the value of each state so this is just a recurrent calculation repeating applying the same operation so some recurring moment and it's a return on that work with local calculation because states next to each other can be visited from each other and happen to show off in this dynamic programming calculation so turns out that there is a recurrent component is enough to represent validation calculation but we don't need


to do it with some evaluation which only applies to situation we can have a tabular representation of the world which means for very relatively small discrete states places he knows more generally so we're looking at here is universal planning Network universal fine network says okay we have an observation we want to achieve a goal observation we take our initial observation we're turning to the late instead we're gonna encoded then we take an action new let instead looks we take an action new let me say we're not


actually late in state and so forth taken on articulated state and after that series of actions we want our little state here to match up the delays in state of the goal of the rich that water key so we can do is within search over actions that will get us close and so if we had already trained this live in space dynamics model all we would need to do is to optimize this sequence of actions and if this is a continuous space we can optimize the sequence of actions and back to the dishes a look around standard vacuum Gatien define a


sequence of actions that optimizes how close we'll get to the goal so that's the planning part assuming we have this dance model we can run back obligation to play how do you get the dynamics model well here's we're going to do we're going to learn the dynamics model such that so we're going to try to find parameters in this dynamics model such that if we use those parameters to run this optimization to find actions then the sequence of actions we find corresponds to what was shown in a demonstration that we're given so we're


given a demonstration a sequence of actions and we'll have an imitation loss and that will say we want to be able to imitate the sequence of actions by writing this very specific process of optimizing with aggregation our sequence of actions against a dynamics model that we're going to learn once we have learned that the nameks model this way what it means is then onwards we can learn we can use this latent space dynamics model to find sequence of actions to optimize how close we get to some other goal in the future so benefit


here is that internalization that our inductive bias than just learning some backbone black box for imitation now she also learns a metric in this abstract space that's useful for reinforced learning in the future so we're comparing us with a reactive imitation learning it just says okay I need to just imitate a sequence of actions but this black box known that doesn't know that when you imitate probably the demonstrator had a goal and you're trying to find something of actions that it keeps that goal so it doesn't have that inductive bias it's


not I do as well and something closes the architecture we use is something that's also a recurrent neural network but doesn't have the internal optimization process in the inner loop to find a sequence of actions that optimizes how close we get to a goal so task we looked at here was some maze navigation tasks and also reaching between obstacles to a target the courtesy here on the horizontal axis number of demonstrations the bring boxes is the average test success rate oh it seems Universal planning networks


outperforms the baselines that I just described but that means that building that inductive bias helps significantly in learning to solve these problems now note that you can says well what did it actually learn we said to build an inductive bias we say with building inductive bias to learn to plan in that inner loop but is it really learn to plan here's experiment doing say what if we learn with 40 iterations of gradient descent to find a civil actions and then we test with a very number of planning steps meaning we vary the number of Grandison


steps in the inner loop when we do plan if our thing is doing planning then the hope is that by writing more planning iterations it would keep refining the plan and end up with a better plan than if it always access to 40 iterations that's indeed what we see here after the horizontal actually we increase the number of planning steps the test success rate goes up for the same amount same training same training just different number of planning steps of tests on so this indicates that likely is something like planning is really


happening under the hood and if you plan longer you can do better nothing that happens is when you do this you learn a representation that ties into how an agent should make decisions that representation can be used by a reinforcement learning agent to learn more quickly what makes me a force wink typically hard is that the reward is sparse but if you map your world into this latent space in that latent space where you're running this optimizer grading descent to find good actions well again bring descend assumes that


there's some smoothness so once you've learned that we can space where there are smoothness you can optimize against that probably means that in that latent space distances are more meaningful I think now do reinforce learning against distances in that waking space you're doing it against the reward that's better that's not sparse but it's dense and it's giving a signal locally on whether you're improving or not improving on what you're doing and so we showed in a wide range of environments I did indeed reinforcement learning can be


a lot more effective when using the distance in Laytonsville earning in the process I just described but then you do reinforcement in a new environment example we did imitation in three link and 4 link environments switch to a 5 link environment Rand reinforced wanting the file of environments faced and the latent space there is used for reward shaping and I guess you learn a lot more quickly same thing here where the initial learning happened with a point mass and a to sub-point mass and then actually have the controller and robot and thanks to


these shaping that comes from learning the slave representation where distances are meaningful learning can be a lot more efficient okay so at this point we've covered quite a few different ways of combining representation learning with reinforced learning to be more efficient and the general theme so far has been that or at least in our state for positions been done raw pixels sure it has the information but it's embedded in a very high dimensional space is million megapixel image million dimensional input we wanted in a more


compact position with and learn against more efficiently and all these ascribe observations available to state and state actually the next date and so forth all tried to get a handle of that problem now nothing you might observe is down what we covered so far is fairly complex is a wide range of ideas at play and so the question we asked ourselves recently is is it possible with a relatively simple idea to maybe get a lot of leverage that we have seen here and let's take a look at that and see how far agree with relatively simple


idea and actually we'll see out the form essentially all the approaches we've covered so far that doesn't mean the ideas and the approaches we've covered so far are not important so we're not important with colleges to skip them there's a lot of good ideas we've covered that we probably want to bring into this next approach we're about to cover but what I'm about to cover curl will really focus on simplicity and see how far I can get with something very simple our stunning exhibition here was if you look at the learning curves the


vertical axis here is reward and higher is better horizontal axis number of trials in this environment and so see like at the end here 1e a to a hundred million steps have been taken in this environment and so we see a blue learning curve here that learns very quickly and then green learning curves that take a long time to learn what's different blue learns from states green learns from pixels same thing here blue learns from stayed very flowers green from pixels not nearly so fast and this isn't this case the RL


algorithm is soft is a d4 PG which is still yard are a logger so if you think about the essence here reinforced winning is about learning to achieve goals and if the underlying space is low dimensional there is a low dimensional state shims will be able to recover that low dimensional state and then learn just as efficiently from pixels as from state and how might we do that well we've seen a lot of success in past lectures with contrast of learning for computer vision in fact we saw with CTC that it was possible by using


unlabeled data is on image net to constantly out the form learning with label data so unlabeled plus so there's the same amount of label data but the blue curve also has unlabeled data you see that the unlabeled data consistently helps outperform having only access to that amount of labeled data then of course very recently Sinclair came out and as actually getting equally good performance has supervised learning on image net when using a linear classifier just a linear classifier on top of a self supervised representation so that means


that almost all the learning happens in self supervision and then a little bit of learning habitat the M of course to get the meaning of the labels but it just needed a linear classifier if that's the case then the hope is if we do something similar in reinforcement all we need to do is do something where we do representation learning that extracts the essence and I've gained a little bit of extra information the reward to do the rest of the learning so would it simply or do it essentially said I have an image I'm going to turn


into two versions of that same image and when I then embed them linear neural network the symbol networking the left hand the right upper channels then the embedding should be close as measured with some cosine similarity and of course over another image that I embed then I'm betting should be far away and those are the negatives in the denominator so for more details and that of course go back to our self supervised learning lectures from a few weeks ago what's important here is done this is a very simple idea it's just saying turn an


image into two images and the embedding should be close take a different image it's embedding should be far from this and what's surprising about this even though it's relatively simple it enables representation learning that then on top of that all you need is a linear classifier to get a really good image that classification performance and they actually looked at many types of augmentations cropping cut out color surveilled filter Norris blur rotate and what they found is that crop matters the most and color matters quite a bit too


but really cropping is the one that matters the most so now the curl curl looks like a nice representation learning with RL so what did we do here we have our a replay buffer on which we normally would just run reinforcement learn and so we have our replay buffer we take on my observations now to replay buffer since this is a dynamical system we need to look at a sequence of frame and consider that a single observation otherwise we cannot observe philosophy and a single frame acknowledge their velocity so we'll have a stack of sequential frames


that we together consider a single observation let's pack of frames then gets undergoes did augmentation to that augmentation in this case two different crops then one goes into the query encoder I'm gonna go key and predators could actually be the same or different you can choose and then ultimately you do two things with this in the top path it just goes to the reinforcements in law so if you run B for PG again or you run soft actor critic you run PPO and so forth that happens along the top path so what it means is along the top


path you run your standard our logarithm the only thing that's changed is that we take this rare replay buffer you do some data on English now in the bottom path you have another data images in the same frame' you have a contrast of loss so essentially the same loss not exactly same details but at a high level same as we saw in the Sinclair slide okay so a couple of things that were important to make this work Sinclair uses a cosine laws what we found is that having a a weighting matrix here between the key McQuarrie is Ashley Borden then we'll


see in the red curve the bilinear and waiting is us secretly outperform using just cosine the other thing we notice is that using momentum Indian and one of the encoder pass is very important to which was actually dawn actually saw herself as learning lecture in the moko work we also have momentum and one of their past same thing was important here again big difference so once we do that we can see that curl outperformance both prior model based and model free state-of-the-art methods so we look at here is medians course on deep mind


control one hundred kid you might control firing the kiss it's hundred can steps are firing the kid steps and so it's checking really can you learn to bounce it's not about after one hundred million steps where you ask is about 100 thousand firing down steps where are you at and so we see here after winter and kid steps from state ship access to state this is how far you get curl on one hundred K steps is a little bit behind what you can do from state but no family K stuff is actually all the way there so we see that we can learn almost


as well from pixels as from stayed with curl for Kepner prior methods that also tried to learn from pixels we see that they consistently we're not doing nearly as well after firing the kid steps and Sam with after hundred clear steps so both after hundred K M cavity steps curl up of homes or prior our elephant pixels on deep mind control Sweden and after getting very close to take this learning here we have the learning curves in gray we see state based learning and red we see curl we see that in many of these red is matching gray


there are a few exceptions within most of them red matches fray meaning that with curl are elfin pixels can be almost as efficient as RL fringe state at least for these deep mind control tasks and here we'll look at you know a table of results you see in boldface the winner compared with all prior methods favorite methods of learning from pixels and you see that consistently curl outperforms the other methods for the tyranny and hierarchy not just an average but on essentially all of the individual tasks except for


that no one here and one here dark public iris with curl doesn't learn as fast and we look at the details what happens there these are environments where the dynamics is fairly complex so this requires some more research with our hypothesis here has been that in those environments learning from pixels is particularly difficult because if you just look at the pixels the dynamic is not well captured in the sequence of frames you get to see for example if contact forces matter a lot and it's you can easily read those off from pixels


and so having access to state makes a pretty big difference in terms of being able to learn looking at the entire benchmark we are looking at median human on normalized score across 26 Atari games at 100k frames and we see that compared to Paris today our rainbow rainbow dqm simple and well at least rainbow DQ and simple and rainbow DQ and curl seen every out performs prior and state-of-the-art and it's getting at about 25 percent of human normalized score here is a broken out for the individual games and curls


outperforming proxy they are fairly consistently what's simple coming in first on it's still two of them so computers are all matched human data efficiency it's good question human normalized algorithm score we see on the freeway and on Janus bond that we get pretty much the level of human efficiency for the other games is a little bit of a way to go but it is not rotates night and don't know it's already double-digit percentage performance relative to human on almost all of them okay so we looked at two main directions in representation


learning in reinforced going so far using auxiliary losses and doing things down if I couldn't come down to trying to recover on the line state with a self supervised type loss now there are only ways representation that I can help mainly an exploration which is one of the big challenges in reinforced learning and in Austin for unsupervised feel discovery so let's look at those two now first we can help exploration is through exploration bonuses so what's the idea here in a tabular scenario meaning a very small


reinforcement problem where the number of states you can visit you can count but say there's only you know a good grid world addition to being only one of sixteen squares that's it one of sixteen possible states a very simple thing new is you give a bonus to the agent for visiting grid squares it hasn't been to before or hasn't been frequently before that encourages going and checking things out that you have don't have much experience with yet that can be very effective in is small environments but its impact of a large continuous state


space is because in a large but they infinite States build infinitely many splits well there's always more stuff you haven't seen so you need a different way of measuring what makes something new versus something already may be understood so one big breaker in the space wants to look at using generic model in this case a pixel CN n for density estimation so the idea here is you planet our game or the agents playing at target you want to measure how often has the agent been in the stick but if you'd never special


specific stage there's too many of them so still we're gonna do is women train a pixel CNN model on what you see on the screen and things you've seen so far the more often you've seen something the higher the log-likelihood under that pixel CNN model but when you let's say enter a new room in this game first time you enter the new room the log likelihood of that new thing you see on the screen will be very very low it'll be a bad score then it's a signal that this is something you need to explore because you're unfamiliar with it as


measured by the flow log likelihood score as you can effectively give exploration bonuses now based on the log likelihood scores under your pixel CNN model that you're trained online has your age and this acting in the world there's a comparison here between using the odds versus just using random exploration and it helps a lot another way to do this you can train a variational honor encoder which leads to an embedding and then you can mount these embeddings into a hash table and just do counting in that a hash table


and that's something we did a couple years ago Amin helps a lot in terms of giving out the right kind of exploration incentives to explore difficult to explore environments more efficiently another thing you can do that kind of gets maybe more at the core of what you really want but it's a little more complicated to set up is for information maximizing exploration so the idea here is the following when you are in a new situation what what makes a deal what makes it interesting about it being new well one way to measure this


is to say hey if I'm in a situation where after taking Archie I cannot predict what's happening next very well then I'm not familiar with this so I should give a bonus for you know having gone into unfamiliar territory that's called curiosity we'll cover that in a moment especially been pretty successful but actually it's also a little defective because if you just have something that's too passive in the world let's say you roll some dice well it's gonna be unpredictable so to make this more charitable one thing you can do is you can say hey I


don't want to be getting exploration bonuses when something is inherently unpredictable how I'm going to get them what it's something that's unpredictable because I have not learned enough yet about this and so the way we did this environment COK you can set up a dynamics model that you're learning and as you learn the dynamics model as Nydia that comes in you can see we actually set up a a posterior over dynamics policy of the distribution over possible dynamics models as new data comes in you get that posterior


if that updated posterior is very different from a previous posterior it means that you got interesting information it allows you to learn something about how the world works so that should give you an exploration bonus because you did something interesting to learn about the world but when throwing the dice addition guys rolled many many times and then rolls again and you couldn't predict it because that's just awareness you cannot predict but your model for the dice will already say it's uniform you know over


all possible outcomes that model will not see much update if any and you will not be given an exploration products and so that's the idea in vine only get exploration bonuses when it updates your posterior over how the world works and again showing here that that helps a lot in terms of exploring more efficiently under the hood that's really self supervising type ideas for a dead and small ensembles or based on representations of the AMEX models and been given exploration bonuses based on that the simple version of that is


called curiosity we're more directly look at you know was something pretty cool or not pretty quiet more the domestic environment often that's actually enough and that's in a lot of success in many of these game environments another thing you could do with self that's learning a representation learning for exploration is to think about it in a more deliberate way you could say hey it's notice about getting bonuses after it's being something new it should also be about thinking about what I should even do before I experience it can set a goal


for myself that makes for a good goal when I'm trying to explore train goal again what's done is the idea is the following you have a in this case let's look at iteration 5 down here the other a set of points that you've reached in this maze you start the bottom left you did a bunch of runs to reach a set of points and where you notice is that the way ascetic goals in the green area unable to consistently achieve your goals we accepted in the blue area it's high variance and some in the red area you should don't achieve your goals we


can induce and say oh actually in the future set my goals in the blue / red area cuz that's the frontier of what I know how to do and so how you're gonna do that you're gonna learn some kind of generative model to generate goals in that regime then go again did you ever have a cell network strained to them generate goals at the frontier of what you're capable of and this allows you to explore Avars much more efficiently because Mary is setting goals to go to places at the frontier of your capability so you continue expanding


your skills you can also do this with a various auto-encoder that's done in rig where the traditional auto-encoder is generating new goals it's those goals and I'm silly not this frontier in the same where they're essentially goals that are similar to things you've seen in the past but the hope is that frequently enough you are the frontier that you learn relatively quickly no can also read way those goals based on how you know how much they're at the frontier measured in something called skew fit which is an expansion to this


paper that sometimes changes the sampling in late in space to get closer to sampling from the frontier rather than just from what you've seen in the past so brick itself here are some examples of this in action you see robots learning to breach and to push so that's the kind of thing that channel is pretty hard to explore for because normally a robot would just be waving in the air and so forth here you can you know set goals that relate to moving objects around and then it would be inclined to move towards object and


wisdom now another thing you can do in terms of exploration and leveraging chariot of models or once about smalls is skill transfer and this should remind you of how we initially motivated unsupervised learning or some of the motivation which was no transfer learning can be very effective with deep fuel nuts now would it be nice if it'll be translating from a task that does not require labels on to a task that requires labels that's transfer from one surprise learning task to them fine tuned when a supervised task well


similar ideas can be a planned reinforcement money so what's going on here so far we mostly talked about going from observations to state those kind of representation money but there's another type ropes and fish line that matters for position learning around objectives behaviors tasks the question here is how do you supervisors and their learning for these things what's the contrast what's done now to explore you maybe put some noise in your action and that way you have some random behavior you might explore something


interesting gonna take a long time sometimes that's shown to be a bit more effective on your explore by putting random is on the whate near no network not only kind of consistently deviate in one way or the other so the good example for the thing on the right works better than thing on the left is let's say is posted for explore a hallway but when I'm left with random walk left right will take very long to get to the end of the hallway and explore both ends of the hallway the one on the right would induce a bias to mark to the right and


maybe with not a random perturbation and do some bars to go to the left and maybe after a couple of robots would have gone to both ends and that's it but it's still really counting on markets it's not it's not really using any more knowledge about experience from the past to explore something new more quickly and that's where the question we're after can we use experience from the past to now learn to do something more quickly for example if you have been in environments shown on the left ear where when you're in the environment and you


don't get to see the red dots the red dots are just for us but imagine we cannot see us or about the red dots and any time you get dropped in environment the reward is I have a spot on that semi circle but you don't know which spot and so you have to go find that reward after a while you should realize done I should go to the semi circle and see which Pollan's semi circle has the reward and that will be a more efficient exploration than to just randomly walk around in this 2d world and then randomly maybe run into the reward on


that semi circle or shown on the right imagine they're supposed to push a block onto the red flat target in the back but you don't know which block you're supposed to push well you'd have a very good strategy saying I push the purple one mmm no reward okay I'm gonna try to push the green one you know her would try to push the silent one Norway which is the yellow one I reward I push the yellow one again and keep collecting reward that's what we would do I see much but how do we get that kind of exploration behavior that's much


more targeted than random motions into an edge and how to get to learn to do that well what we really want then is somehow a representation of behaviors for example pushing objects makes for an interesting behavior that often relates to reward whereas random motion that where the gripper does not interact with objects will rarely be interesting and rarely lead to rewards that's the kind of thing we want to learn in our representation of behaviors it is one way we can do that this is supervised the bridge but just


doesn't just set some context or not ones about supervised for now but it will go from supervised and transfer to an unsupervised and transfer very soon imagine of many many tasks for each task you have a discrete indexing through the top which is turn into an embedding then I've been expended to the policy the currents data observation fed into the policy nopales take action if you train this policy for many many tasks at the same time then it'll learn depending on what task representative with this index D to take a good action for that task


but now the additional thing done here is that this latent code Z here is forced to come from a normal distribution what does that do the normal distribution means that even the future we don't know what the task is nobody tells us what the task is there might be a new task we can actually sample from this distribution to get exploratory behavior so you say oh let's sample this you know sample Batsy and the possible still do something very directed something that relates to maybe interacting with objects as opposed to


just some random chittering to make this even stronger the case there's a mutual information objective between each directory and the latent variables see here so turns out there's actually help so you learn in a bunch of tasks this way and they have a new task and you explore by generating Latham code see and then someone you'll finally I can carry that actually leads to good behavior and you'll start collecting higher reward coming Bank is low less supervise where there's a little differently let me canoes and say well let's not


even have discreet tasks to mixing let's just have a late until it going in and when a policy that pays attention to the latent code while collecting your work why would that happen well there will still be many tasks under the hood but we're not telling it the indexes of the task we're just letting experience reward and so what I'll learn to do is they'll learn to sample a see he would have got zv4 successful behavior with dust and it'll reinforce that see if it doesn't it'll have to sample a different Z and so forth so here's some tasks


family is every dot in the semester Birkin spreads to a different task so we hope here that it would learn to associate different Z's with different spots in the semicircle such that when it later explores by sampling different G's it would go to different spots in the semicircle I mean the one that's successful be able to reinforce that same for the wheeled robot here and here's the block pushing tasks looking at the learning curves we see that indeed by getting to because it getting to pre trained on this notion of


indexing into tasks or a dispute over tossed and then be able to explore by sampling possible tasks it's able to them in blue here learn very quickly to solve new tasks compared to other approaches the generated behaviors we see are also very explored at Reseda we explored their behaviors indeed respond to visiting the semicircle and this gives the wheeled robot in the mill here it's Thea walking robot on the right is the block pushing what is look like in a human projet didn't do representation learning for exploration


behaviors you and instead of having this nice push behavior should have just some jittery behavior of the robot gripper that wouldn't really interact with the blocks or get any block to the target area after it's done those exploratory behaviors of course the next single will happens a policy grand update that will update the policy to essentially sample Z from a more focused distribution that focuses on the part of the session will correspond to the part a semi-circle what the target is or the block that needs to be pushed okay now what we did


here was transfer from having a set of tasks to now solving a new task relatively quickly by having good exploration behavior but we still needed to define a set of tasks and then transfer from that to question how is going to completely unsupervised school we just have the robot we're on its own to learn a range of behaviors and another test time Explorer in a meaningful way to Zone in on specific skill quickly take a look so it's actually multiple lines of work that effectively do the same thing but try a different object there's but the same


high-level idea so the hang of ideas we're still gonna have a policy PI that conditions his actions on the observation the current stayed near and a latent code which might or might not come from a discreet code bubbly has come from a latent critical to a normal distribution so we can resample this in the future this will solving trajectories and so the way we're going to pre train this is by saying that there needs to be high mutual information between its directory that results from this policy and the latent code is acting based upon so you


start you roll out at the beginning of you roll out your sample Z you keep z fixed for the entire roll out to get a trajectory you want the trajectory to say something about whatever Z was that you used for this trajectory what does it mean that high missile bases in Turkey Thailand see what we measured in many ways and that's what these four different papers are the first paper which are actually discrete variable in a directory and the second paper looks at B and the final state the third paper looks at Z and every


intermediate state independently some together and then the fourth one looks at Z and the full trajectory as a whole and they all get fairly similar results actually so here's the third tapered Eisenbach a tall paper showing a range of behaviors that comes out of this when you apply this to the cheetah robot so for different disease you get different behaviors here I mean we see how our mission information people use different Z's and the trajectories look very different to us may not indeed a different Z results in a very different


directory and of course the beauteous ones is learn to check out all these behaviors for different Z's now at test time you need to do something else but they need to run out of certain speed either will be Z's that already correspond to running forward and then you can fine-tune the Z directly around with to learn a policy it isn't to figure out the Z that will result in the behavior that you want here are some videos from the paper that make a model paper looking at and curating all kinds of different trajectories correspond to


responding to all kinds of different latent variables see so we see pretty same latent variable see same kind of directory gets output and here's some more videos well some of these cannot be played for some reason but here's a cheetah robot the eggs I'm a tall approaching so this kind of just not too sure that you know they camera at all might be better than the awesome burger dollar I think it's just a show that actually is very similar that so the the difference in those four objectives might not be too important


actually some limitations in this approach this coaster comes from an ATM at all paper is that when you have a humanoid which is very high dimensional compared to cheetah which is essentially just kind of stands up or runs on its head humanoid is high dimensional you try to find financial information behaviors between Si and trajectories you can it can take a long time or it can have a lot of mutual information with all trajectories actually being on the ground because there's a lot of different things you can do on the


ground and it's not something where you necessarily automatically get it to run around this running is very hard to learn where I was doing all kind of different tricks on the ground is much much easier okay so let me summarize what we covered today would cover a lot of ground much more quickly than in most of our other lectures because this lecture here is more of a sampling of ideas of how representation learning and reinforcing have come together in theorists place and you know a very deep dive in any one


of them as we've done in previous lectures the big high level ideas are that we attain a neural network and beep reinforcement learning your mind is looking at auxilary losses and if those losses are related to your task well it might help you to learn more quickly than if you did not have those exact losses and of course the most economical paper there was the Unreal paper under the hood a lot of disguise to state representation if we have high dimensional image inputs well hopefully under the hood in this task often there


is a low dimensional state and so there's many things you can do to try to extract latent representation that is closer to state than there is no pixels once you're working without lady representation closer to state or maybe even match to a state learning might go along more quickly and in fact we've seen that with the curl approach it's possible to learn almost as quickly from pixels as from spit it's not just about turning a raw sensor observation into a state there's other things you can do with representation lying in our own you


can have it help with exploration you can have it help an exploration by helping you generate exploration bonuses especially measured things that are new canonically and tabular environment she measured by you know visitation rates but in high dimensional spaces you'll always visit new states so you need to measure how different that new state is from past is which which you can do it to narrative models and my clearance another thing you can do in terms of exploration is you can think about generic models for behaviors that are


interesting such that mount exploration becomes a matter of behavior generation rather than random action all the time or you can learn to narrative models for goals that might be interesting to set and then set goals with your generic model for a reinforcement agent to try to achieve to expand its frontiers of capabilities and I'm not if you can do is ultra by skill discovery don't suppose skill discovery what we do is we essentially have no reward at all in a pre training phase but the hope is that the agent nevertheless starts exhibiting


to be able to introduce Alec Radford Alec Radford is a research scientist at open the eye alec has pioneered many of the latest advances in AI for natural language processing you might be familiar already with GPT and GPT to which Alec led those efforts at open AI and of course earlier in the semester we covered BC GN which was the first Gann incarnation that could start generating realistic looking images and that was also led by Alex it's a real honor to have Alec with us today and yeah now Mike please take it away from here

L11 Language Models -- guest instructor: Alec Radford (OpenAI) --- Deep Unsupervised Learning SP20


yeah totally I'm super excited to be here and present because this course is like my favorite research topic unsupervised learning and yeah just really excited to chat with you all today so today I'm gonna focus on the NLP and tech side and I'm just gonna start the timer and today I'll be talking about just kind of generally learning from text in a scalable unsupervised kind of fashion kind of give a history of the field and some of the you know main techniques and approaches and kind of walk through the methods and kind of where we are today


as well as providing some commentary on kind of supervised learning versus on supervised learning in NLP and why I think you know unsupervised methods are so important in this space yeah so let's I guess get started so learning from text you know one of the I think prerequisites to kind of start with is standard supervised learning requires kind of you know what we'd say is machine learning great data and what I mean by that is kind of your canonical machine learning data set is something at least in an academic context is


something like you go use a crowd worker pipeline and you very carefully curate gold labeled standards for some data you're trying to annotate and this is a pretty involved expensive process and you often are emphasizing kind of quality and specificity and preciseness to the thing you care about the task you're trying to predict and maybe a very specific targeted data distribution and what this often means is you get a small amount of very high quality data and even some of the largest efforts in space just because you have paid human