could be put into that that is that is tricky but there's no fallback that's a good fallback because let's say a fallback is like you know go constant velocity well you might just run into something or like yeah but you can but you could for example say oh I'm gonna stop carrying about any of my other high-level stuff and I just want to like come to a stop with the obstacle


avoidance if it's a really bad scenario yeah okay just another video of simulation and the kind of cool thing about simulation so this is this is actually the from an SDK that we have it's kind of in a private beta SDK where people sort of we let people work on it that we think make a lot of sense to to kind of develop on our drone but the the cool thing is to share that simulation

Lecture 23 Guest Lecture: Drones -- CS287-FA19 Advanced Robotics at UC Berkeley


between like our purposes and actually being able to turn the drone into something that you can program so at a high level like make it move around make it do things but not have to care about obstacle avoidance and and again the low-level stuff so we're it's it's a hard problem because controlling a drone like you have to understand a lot of the internals of the system to do it so


we're kind of always thinking about what is the way to try to design an API such that it's it's intuitive for people to understand especially people that aren't experienced drones or even robotics to be able to control things and get what they want and sort of understand why it might not be able to do something and why it might go do something okay cool let me show a couple of other


cool things so we estimate the wind online which is really useful for to have as a model to do more effective planning and aerodynamic control the drone this is something neat it's a lot less relevant for this drone but the first drone had perimeter guards and the reason I had perimeter guards is to have cameras that can see like on the outside that's why we very carefully designed


the geometry of this drone to try to get rid of the perimeter guards because they're so expensive and bulky and and bad for performance while still having visibility but the cool thing was it could bump into things and not have the propellers touch it and in this case like we go we fly into this glass intentionally and we actually detect that collision and we add that obstacle


into our map there so we won't do it again which is a pretty neat thing to do and then yeah we look a lot at sort of recovery from dynamic new verse so we call this inverted pizza toss where you your initial condition is like upside down and then spinning quickly and then the drone kind of has to come down and recover this is another fun one where there's prior work on this as like its


own controller but something neat that we found was if you actually basically just with parameter changes to our planning system if we shut off one rotor we find this kind of mode of flight where it's still semi stable where you're not controlling yaw anymore but you're able to sort of hover and come down to Al and this is all in simulation these so far because the question of


state estimation in this in this scenarios is a lot harder but that's pretty cool that like by just spinning really fast you're able to stay hovering with three rotors and in fact what we did was sure that we could again in simulation actually follow somebody and avoid obstacles while just having three propellers and spinning like a like a madman so I don't think anyone's shown that


before okay these are slides from cvpr it's a vision conference so I'm gonna skip those I wouldn't say that in fact I already talked about yeah don't work on deep learning benchmarks and try to improve them by one percent that's a sucky thing to do work on hard things and real robots and that's I think what this class is all about so that's awesome okay and I'm


just gonna show a one last video which is one of my favorites just for drones following Jack here as he goes down a waterslide just in in every direction and it's just a fun thing to watch all right cool yeah so we get a lot of sweet videos and again so our goal is to like bring these drones to doing useful things not just for video capture for you know content for consumers and


action sports but also for inspection and mapping and construction and all all these places where you need a trustworthy reliable robot that can move around anywhere and get and sense sense things basically all right that's that's it yeah thank you cool and then yeah if any of you guys are like looking at internships or I know a lot of your graduate students so I'd be happy to


hi everyone welcome to lecture 12 of deep unsupervised learning spring 2020 today we'll cover representation learning in reinforcement learning first before I start I wanna give a big thank you to many colleagues and friends who have contributed to this lecture through sharing their insights illustrations slides and videos that all very directly contributed to well I'll be sharing in

L12 Representation Learning for Reinforcement Learning --- CS294-158 UC Berkeley Spring 2020


this lecture here today thank you so Terry what class has been about unsupervised learning today we're actually going to look at how one surprise learning and reinforcement can be brought together in a way that inspires learning to make reinforced learning more efficient but we haven't covered reinforcement learning yet in this class and so what I'm first gonna


do is step through some of the very basics of reinforced learning obviously can't cover everything it could be a course in itself but we'll go through some of the basics from the recent successes and then from there look at successes where unsurprised learning and then reinforced winning or brought together so what is reinforcement lis reinforce learning is a problem setting


where you have an agent agents supposed to take actions in the world ask the agent take actions the world will change so for example the world could be the robot body and the environment around the robot body and after the world has changed because of the agents action this process repeats over and over and over and the goal for the agent is to maximize reward collected in the process


for example imagine our agent is supposed to control a self-driving car then the reward might be positive for reaching destination and negative forgetting to an accident maybe our agent is a robot chef and then the road might be positive for a good meal even more positive for an excellent meal and negative for making a total mess in the kitchen and the goal in reinforcement


learning is for this agent to figure out through its own trial and error how to get high reward so as the human designer who gives a specification with the statement I'd like high reward and you say reward is high for the things I just described and then the agent specific items on how to achieve that another example could be a video game the score in the video game


could be the reward and so the agents supposed to figure out how to play that game to maximize reward where are some challenges in reinforcement learning man let me contrast it with supervised links and supervised learning what happens is you have an input and a corresponding outputs and the way you supervise your learning systems bias named for this input that should be output for this


other input that should be up and so forth in reinforcement learning your robot chef might be busy in the kitchen for half hour comes out with its meal and you might say good or bad meal but that's not reflective of the last action their robots have took it's reflective that whole half and half an hour of working a kitchen that somehow result in a higher board or a low reward and now


when that chef robot chef cooks multiple times and sometimes has good outcome sometimes bad outcomes you could start looking at what's common between the good outcomes what's common among the bad outcomes that process is a seizing apart what might have positively concrete or might have negatively con treated that's solving the credit assignment problem it's one of the big


challenges for a reinforcement winning agent another big challenge of stability when let's say you have an agent learned to fly a helicopter well helicopters are naturally unstable so if you're not careful during the learning process you might crash your system and I might just stop the whole thing another big challenge is exploration for a reinforced training


agent to learn to do things it actually if it actually doesn't know how to do anything has to try things it's never done before it has to explore and this is many many challenges when you try things you never tried before well how do you even know what you should be trying it could be so many things to try what's more interesting what's less interesting and it also brings back the


stability challenge is how do you make sure we try something you don't destroy the system and so forth now one example that many people would know in real life all three enforce planning is how to train a dog when you train a dog the dog is the reinforced running agent and you as a human provide rewards you might give the dog positive reward when it does well and negative reward when it


does poorly and you don't control what the dog does it's not supervising against you cannot tell the dog do this do that do that all just all its muscle muscles will follow your commands no the dog will just do some stuff and you'll say good or bad depending on how happy you are with what the dog did so one of the things that we want to do in today's lecture is give you a bit of


overview of successes of reinforcement learning but then also from there look at limitations and take a look at how representation when it can help out so one of these successes of probably success that foot deep reinforcement on the map was in 2013 when deepmind came out with the DQ end results deepmind showed that it's possible for a neural network to learn to play a wide range of


Atari games from its own trial and error now this was a big surprise until then if you looked at reinforced learning results they would typically be on relatively small simple environments and the input would not be images the input to the agent would be a very well-crafted representation of whatever world the agent is in summarizing in a small set of features or state variables and so a


big surprise all of a sudden reinforcement works with pixels as input from there a lot of progress was made of course including the progress list on this slide here a lot of it coming out of also deep mind Berkley open AI and much much higher scores and faster learning has been achieved in the Atari was on on the Atari benchmark sense wasn't just Atari deep mind also should


even learn to play the game of go long-standing challenge many people thought it would be another 20 years if we asked him in 2013 2014 but sure enough in 2015 a computer beat the world champion in doe then the first version alphago was a combination of imitation learning and reinforcement learning the second version alphago zero was pure reinforcement learning it just learned from playing us


itself and over time became better than the best human players and then was a big result in more advanced gameplay or video game play opening showed that the game of dota 2 can be mastered by reinforced learning engine into 2017 he was shown a reinforcement agent to master the one-on-one version of the game and be some of the best human players and then Lola was shown that reinforcement


learning enables playing a very competitive not necessarily beating the human world champion team just yet but at a very competitive level with some of the best human teams through pure reinforcement learning at Berkeley in parallel we're exploring reinforcement learning for robotic control and so here is actually some reinforcement in action thus far we just talked about results


what does it look like what you see it in action here we see an agent that's learning to control and this kid is learning to run and we give it positive reward the more positive the more moves to the right and it's negative reward for falling up to the ground and what we see is that over time it figures out a strategy a control policy that gets it to run off to the


right now the beauty here is that it's able to learn this in a way that is not specific to this two legged robot meaning that we can run the exact same deep reinforced landing code and run it on the four-legged robot it'll learn to control this four-legged robots and in fact it can also learn to play at our games the exact same code in this case is precision policy optimization


TRP o combined with generalized advanced estimation tae that's able to learn to master all their skills in this case that the robots learning to get up the reward is based on how close the head is to standing head height the closer the head of standing head height the higher the reward and then this was generalized to a much wider range of skills so what you see here is reinforced winning edge


that has massive very wide range of locomotion skills and then here we see it in action on a real robot this is Bret the Berkeley robots for the elimination of the tedious tasks because we you must don't want to do the tedious tasks with want robots to do those students tasks for us when we see here is this robot is learning to put the Block in it imagine opening and indeed


over time it figures out how to get the block did the matching opening now what it's doing under the hood it's learning a vision system and a control system all together to learn to complete this task what's the catch in all of this data inefficiency well monasteries achieved an Atari Ingo in robot locomotion robot manipulation and so forth this mastery requires an


enormous amount of trial and error and so the big question is can we somehow bring that down and reduce the amount of trial and error that's required to master these skills it turns out I believe in many others believe that representation learning can play a big role in getting there to get to much more efficient reinforcement learning and it's not something that is fully


understood yet this is a domain with a lot of room for more research and so we'll cover today is a pretty wide range of highlights of relatively recent results people have achieved by looking at combining representation learning with reinforcement to make RL reinforce learning more efficient we'll look at is long for directions auxilary losses state representation exploration and


unsurprised field discovery and we'll unpack these as we go along one thing you'll notice is that it's not some kind of linearly building up thing and at the end you know it culminates in what is the most important piece really what we're going to be covering is a wide range of highlights that each has their own interesting factors to it and probably a solution that will be the


final solution in the future we'll combine ideas from many of the research results that we cover today into one system so let's start with those hilary losses the paper wanna start with is the unreal paper by deep mine so the idea here is done reinforce money as a sense can be very data hungry especially there's only sparse rewards and the question is can we make an agent learn


more efficiently by having auxiliar prediction and control test by having it not just learn from reward signal because it might be very sparse but have the agent absorb other signals of course nothing else we want to supervise as a human because then it becomes very tedious but self supervising those things that are available in the environment that the agent could try to


learn from even if it's not exactly rewards signal so the unreal agent which stands for unsupervised reinforced planning and auxiliary learning showed it tenants improvement in data efficiency over a three C which was can be standard RL approach deep mind abuses at the time on the 3d deep mind lab which is a navigation first-person vision navigation task and sixty percent


improvement the final scores so faster learning and converging to better final scores so what does the architecture look like when we see at the top here in the middle is the base a QC agent so again this is not a reinforcement in lecture let me give you a little bit of background what's going on here in reinforced when you have experiences the agent without any given time and has to


make a decision taking the current input process it and then through the policy output try to make a decision on what to do and through the value function output try to predict how much reward is gonna get in the future from this moment in time onwards so there's two output predictions here and that's the standard base a through C editor already predicts two things how much reward that's V


value the cumulative reward over time that's coming and policy PI action it should be take so both of those are outputs of the same network that's the basis of this edge then all the data gets put into a replay buffer and it's reused in other ways and the same neural net that is the a through C agent will give it multiple heads even more head so it has to make even more predictions and


by giving an additional prediction tasks if these prediction paths are related to learning to solve the original problem which is achieve high reward and these prediction tasks are real ended and hopefully it'll learn something that will transfer over to the real task who care about and be able to learn the real task more quickly so what are these auxiliary tasks the first one is


auxilary q functions so the idea here is you're given additional heads to the neural network that are q functions a q function is predicting for the current situation how much you work well I get in the future if I currently take a specific action so for each possible action I'll predict how much reward might I get now the interesting thing about Q function learning is that you


can do Q function learning of policy meaning you can try to solve one task but in the meantime do Q learning against the other task that has a different reward function and that's the key idea here we're gonna take reward functions that are not the ones we care about that are auxiliary reward functions that are easy to automatically extract from the environment and the Q&A


against those who word functions and by doing so the core structure the core of the neural net will learn things that are also useful for the task we actually care about okay so what's actually that a little deeper here so basically she agent is the core thing the sparse awards means you just I mean here's the cake from now on Laocoon you get this one T round the cake from


the real reward that we care about that's not enough we want more rewards that's exactly what this cue function thing is going to do we're going to define many other rewards and there's many other rewards are going to allow us to learn from a lot more signal and if you only had our one reward okay so this reward function that was defined here by the office of the paper is called a


pixel control reward function and so what they do is they turn what the agency's first-person vision of the maze into a courser this gives grayscale and a representation of what it's seeing and you get rewarded in this auxilary reward task for how much you're able to change the discourse pixel value so what does that mean if your agent turns into a direction where things are much brighter


than the direction they're facing right now then the pixel values will change a lot and that would be a high reward thing or other way around if right now things look very bright in a certain pixel in it it in turns and makes that pixel darker that would be a high reward again that's not what we actually care about but it's a very simple auxilary loss that we can impose and that we can


run q-learning against and so that's it also turns out that this is the one that mattered the most for improving the learning there are other Attilio losses with is the one that matters the most intuition here why this one matters the most is that in Q learning you are learning about the effect of your actions and so because you're learning about the effect of many


possible actions you could take would be the Q value if I turn to the left well did you follow that turn to the right what if we the Q value I look up look down and you're really learning something about how the world works and not just how the world works with how your actions interact is what happens in the world another usually loss is reward prediction so what you do here is for


your current pause that you're executing you can try to predict a future time steps how much real reward you're going to get so maybe you get rewards for eating an apple and so when you see an apple in the distance you should be able to predict that if you keep running forward for three more steps you'll get that Apple and so learning to predict that in three steps you gonna get that


Apple is an auxilary loss that's introduced here and then elastic zero loss is value function replay so it's saying from the current time how much reward am I going to get over the next few steps so this applies in theory in the base every see agent actually all right so if you look at results here this is undefined lab which is that 3d navigation environment where you collect


apples and other fruits as a reward we can look at different approaches so the bottom curve we are looking at is the base a tree C agent so that's the dark blue bottom curve and the hope is by having auxiliary losses we can do better if we incorporate all the ideas that we just covered you get the unreal agent you get this top lead curve here and now we see here is various operations to see


well which one of these matters the most which ones might not contribute very much what we see is if he does do the human 4 pixel control it's a loss that's the yellow curve you get almost all the juice of these area losses but if an addition you have the reward prediction and the value replayer you have yet a little better performance so actually another thing I want to highlight here


the butts on the top of the graph here says average of the top three aging so there's a way to evaluate things in the paper usually reinforcement learning because you need to explore and there's a lot of random is an exploration the results are somewhat unpredictable of meaning that some runs will be more lucky than other rooms it'll be high variance and so with the baby here they


pick the top three grunts might say why the top three that's a lot crazy shouldn't you look at the average performance anything like that yeah you could argue should look at the average performance it's what's done in most papers that our thinking here was then imagine what you're interested in is finding a policy and you have maybe a budget of doing 20 runs then


maybe what matters is what's the best one among those 20 runs or it could be a little more robust aberrancy you know how do the best three runs do and so an approach where the best three runs are consistently great then that's an approach where if you won't afford 20 runs total you'll have a good one among them and so that's a it's kind of a funny way to score things but it happens


to be how they do things in this paper so another thing that compared with which we as unsupervised learning students of course we're very clear about if you do pixel control why not do feature control why not cue function for kind of later layers in the network if for later lays in the network I want to see if I take an action you know can I change the future value and maybe layer


five or layer six instead of just pixel value change well we see a 3c plus feature control in green and it received plus pixel control in orange you can see the pixel control actually works better of course this might depend on the environment this might depend on exactly only architecture that the experiments that we're done in this paper showed that pixel control actually slightly


outperformed feature based control and again control here means it's the auxiliary loss using the auxiliary cue functions that the ultimate reward function that you're actually optimized for and score against on the vertical axis here is the real reward function of collecting the fruits in the maze then here are a couple of unsupervised RL baselines so what are some other things


we're looking at again a principal pixel control shown in yellow that's the top curve and both plots then input input change frequency just try to have an auxiliary law that says can I predict how what I see will change as a function of my action so that's really learning a dynamics model that's shown in blue and then shown in green is in Petri constructions that's a


bit like a Dae I have an input make a little representation reconstructed back out and so what we see is that these things that might seem maybe more natural and more advanced like infantry construction input change prediction are actually less effective than pixel control and of course I mean there could be many many factors have plenty here but the high level intuition that most


people have is that the reason these are clearly cute functions work so well is that what's happening here is that as we as we work with exilic q functions we are we are actually we're actually learning about not just how the world works which is the input change prediction but how we are able to affect what happens in the world and that's really what matters for learning to


achieve high reward than the task you care about now domain they looked at rather than first-person maze navigations Montezuma's in French on this month's range of famous at our game where expression is very difficult there are many many rooms in every room there's complicated things you have to do collecting keys jumping over things they make one mistake you're dead and


you start back at the beginning shows you that unreal outperforms it you see quite quite a bit a 3 CR at the bottom I think black is really not getting anywhere where the unreal approach is actually doing a lot better now let's take a look at this maze navigation agent in action so this is deep mind lab let's take a look at the agent plane you agents collecting the apples here not


collecting the lemons that's apparently it's not good in this particular game to collect the lemons and so this agent has learned to navigate mazes the way it can do that by de Rais because it has a lsdm which gives a memory and so you can remember places been before and things has tried before to more efficiently find the next new location where there might be a fruit it hasn't collected yet


and so the reason I'm showing these specific results here is because the space of well reinforced planning in general but especially representation and reinforcement learning does the evaluations aren't all in the same type of environments there's a lot of variation how these things get evaluated and so having a good feel for what these experiments actually look like is


important to get a sense for how advanced might this method really be and so we see here as well this first person navigation well that's pretty complicated so this might be a pretty advanced method that play here here we see a bit of an inside look into the agent itself where on the top right you see the pixel control Q values is something depending on which action that


take this for actions available how much how high will my Q value be which really is corresponds to an understanding of how the world works what will change in what I see as a function of actions I take all right so to summarize the unreal law since the original ATC loss which is a state policy gradient most value function loss then there is value replay loss which look seven replay


about her Valarie prediction and then there's the pixel control two functions for different course pixels in the view of the agent and then finally there's the reward prediction loss small opening in the word prediction they ensured was an equal portion of rewarding and non rewarding examples so a balanced training set the pixel control did split into 20 by 20 rid of cells all right so


in the entire results we see that unreal also helps over a 3 C but not nearly as much as in the deep mind lab environments but still a significant improvement the vertical axis here is human normalized to form a server the way deep mind is evaluating this is they look at what you missed in 13 Natalie for every game that's gonna be a different score because every game is


very different scoring system and then they normalize it across games another total score across all games how well the agent learns so it is across many many Atari games in terms of Atari games on average how fast is the learning curve go up so you cannot overfeeding onto one game or the other and do well on this score you need to be able to learn well on all of the games


to do well on this score alright and then also look here at robustness because there's many agents being trained and this top three curves on this le expose performance of all the agents here is a little evaluation of robustness and we'll see that you know there's a bit of decaying performance not all agents learn equally well but it's not that there's just one death as


well and then nobody else as well so this looks pretty good okay so that's the first thing I want to cover which is auxilary losses and unreal is a very good example of that does more work happening all the time in this space but that was kind of the big initial result that's showed this is something that can be very beneficial let's switch gears to state representation an ominous will


have many many subsections it turns out first one we have is how would go from observation to state so the the kind of paper that most people might be most familiar with is the world models paper by David ha and collaborators and here's very kind of a simple diagram showcasing what they investigated so what you have is you have an environment the environment leads to an observation in


this case pixel values and that can be very high dimensional fear but I'll take 100 by 100 image that's 10,000 pixels that's a very high dimensional input then they say well that's kinda Michelin we want our agent to work on something lower dimensional because we know under the hood there is a state of the world and the state of dog might be summarized wearing just a small set of numbers


maybe 10 20 30 numbers is enough so my agent shouldn't have to operate shouldn't do reinforcement on those 10,000 number input should be doing reinforced learning on dusty 30 number input and might be able to learn a lot more quickly because credit assignment should be easier if we only have to look at 30 numbers instead of 10,000 numbers and so it is a you risk a strain of


racial auto-encoder which will of course cover them in their release of this course to find a latent representation now from which we can reconstruct there is more but then we'll use the latent representation as the input to the reinforcement learning agent which now hopefully will be more efficient so will them do in this approach is train a recurrent neural network now learns to


predict the next latent state so what's gonna happen here is we're gonna learn a way to simulate how the world works in this recurrent neural network but not by directly simulating and pixel space but by simulating in its latent space which could go a lot faster if there's a lot lower dimension we don't have to render the world at all times we can just simulate how the latent variables evolve


over time of course this will also depend on the actions thing so it's a Mulligan's interaction and previous latent state generate the next life in state and of course you want it to be the case that that matches up with the actual next latent state that you're VA you would output when you get to observe the next live in state and then he actually gets fed into the environment


also and this so you have kind of two paths here at the actual environment path and you have the RNN prediction path and you hope that they line up or you training really to make this line up the thing in blue is called the world model is a thing that looks at the latent state see turns it into by looking at the action latent state turns it into next light instead alright so


they looked at this in the comics of car racing so in the left you see the environment that drains the roads you're supposed to stay on the road here the way they were rewarded shut up and race down this road as quickly as possible this is from pixel input so you get a lot of numbers as input and somehow you hope that would get turned into effectively an understanding of roughly


where the road is where your cars on that road and which direction it's facing your car and then understand how to steer it to go down the road as quickly as possible procedure that followed is that click 10,000 robots from a random policy and the trendler view to encode frames into Z space just thirty two dimensional z space so low dimensional compared to the pixel


open space then they train a ironing model to predict next lengthen state from previous latent state action and there's this additional hit late instead inside the Arnim then they use evolution but it's just one of many possible RL approaches to train a linear controller to maximize expected reward of a robot so step one two and three is the unsupervised learning that can happen


ahead of time and then you can run RL on that representation that you've learned so one thing that's real interesting here is that remember the Omnitech where yaw would say oh well you know reinforced learning is a cherry on the cake which is tiny compared to the cake and why are something for spent in the chair because it's not a lot of rewards it's just small amount of reward signal


the following york response to be bound a signal there's a lot of signal coming from self supervised learning and that's the foundation of the kick and so if you look at what's happening here the VMA neural network has four million parameters the RN and dynamics model network says no network has four hundred thousand parameters and then the controller the thing that is learned


with RL only has eight hundred something parameters there's a massive difference in that RL only has to learn a small number of parameters which Maps do it only have a less lesser amount of signal whereas the saw survived part has to learn most parameters millions of parameters and that's done from one to device theta okay so here's an example of an input frame the 64 by 64 pixels here and a


frame reconstruction which kind of roughly matches up not perfectly but it gets to just then here we have when we use just C or Z on h h is the ardennes in that state so it shows that it's important that the RNA and hidden state captures something important about the world let's look at results so what we see here in the table is scores obtained with the model described highest score


in this car is the environment compared to previous methods obviously in principle unlimited she be able to learn this too but when you limit the amount of time you get to train then using cell scribes learning to learn a representation combined with reinforcement to learn the control allow us to get seemingly higher scores than previous methods there were pure RL were


able to do so this is the model we looked at before one experiment we saw so far I had the car racing environment the second experiment where you have to dodge things being shot at you in a fist doom environment so the input will look something like we see on the left but sometimes you'll see fireballs coming at you when they're shooting at you and you got to dodge those fire bullets to get


to stay alive and get high reward same approach train of uni training our end world model then linear controller train with our L on top of that and so again this linear controller train on top of that is trained in the Arnon simulator itself so you don't need you don't need to simulate what things will look like rendering is often expensive computationally if you need to go all


the way to rendering to train your policy I'll take a lot longer to do the same number of rollouts their oldest thing that low dimensional latent space to train the policy so it's called doom take cover here's a higher resolution version of what this looks like if you were to play this game yourself same approach laid out here again unsupervised learning does all the stuff


at the top here millions of parameters learn then the RL only needs to learn about a thousand forever again beautiful association of the non-linear cake idea so here is what here's what this what this looks like one thing to to keep in mind here is that it actually sometimes you can you can have some quirky results where the simulator of the world allows you to do


things you can now do in the real world and so that's something to look out for that they're highlighting in on their website so if you go look at the kind of normal temperature higher temperature things you'll see some differences there so here are the results we have depending on the temperature different discrepancies so for low temperature we see a very high virtual score but the


actual score not so great for higher temperatures we have a closer match between the virtual score in the actual score so actually actually I should I quickly highlight what would meet with temperature here so typically in RL you have a policy that has stochastic output so you would have a distribution over actions and that solution over actions can have a temperature parameter in


terms of how much you favor your favorite action and so that temperature parameter if you make it small low close to zero then you'll always think your preferred out most preferred action when you have then you end up with a close to the domestic policy we have a close to domestic policy you can often explored quirks in your simulator whereas if you have some random is in your policy you


have a higher temperature if ourselves a little bit of randomness then you could not exploit the very specific quirks and lauren simulator because the randomness will prevent you from being able to go to that very very quirky path where you all of a sudden get a high score even though you know really you can't do that in the real world but your simulator has a small


little bug you won't be able to trigger that small little bug and that's what's going on here with temperature at higher temperature we are not able to exploit tiny little bugs my learned simulator we have to learn something more robust and that leads to a better match between performers in the real environment relative to the learned simulator ok so that was the world models paper by David


hein collaborators now one question you could ask yourself if we're going to learn a world model we're going to learn a simulator some Lincoln space simulator couldn't make sense to try to learn a latent space such that control becomes easier what I mean with that so if you look at the control literature some control problems are easy to solve some control problems are very hard to solve


and maybe we can map our pixel observations and world and amps and pixel space into a blatant space dynamics that satisfy certain properties that make the resulting control problem easier to solve a good example of this is linear dynamical systems if you have a linear dynamical system then the control problem tends to be relatively straightforward to solve so how about


this dinner this is what this paper we're going to cover here is going to do is hold on give me one second here [Music] let me cover something else here first so one thing that might happen is as you train the world model on your randomly collected data and then turn your policy and test it in the real world it might not always work the reason might not work is because the randomly collected


data might not not have been interesting enough to maybe cover the parts of the space where you would get high reward and so what then you'd want to do is iterate this process at this point you effectively a model-based reinforcement procedure you collect data you learn a model you find a policy in the learned model you deploy that policy you take a new data and prove your world model and


repeat so that's what they did and this gives for the carpal swing up and so after about twenty trations with this we would be able to learn to swing it up now a couple of other world models papers is the actions additional video prediction using deep networks in atari games just at the top here worth checking out model-based reinforced planning for atari it's another one


worth checking out and then learning the dynamics from for planning some pixels planet which we'll look at a little bit later also so if you want to look more closely at the specific it which is covered there's a really nice website world model start github dot which has the code which has many demos for you to play with the play with what actually the latent variables are doing in the


VAD and so forth for these alarms so highly recommend checking that out and here is a video of the chris doom cover in action so you get these fireballs coming at you and the agent has learned to get out of the way to not get killed all right so we looked at so far is we looked at how to go from observation to state and then learn a model in that latent state space


now we're gonna take a look at program division to state but also from state action to next state so and this was that earlier alluded to an hour i jumped the gun a little bit on this yes we're gonna now be representational and that's not ahead of time learning your position and pixel to hopefully state or something like state but that is already when it's doing representation looking


at the dynamics and so when we look at the dynamics to representation learning why not learn a representation where the dynamics is such that control becomes easier for example learn a representation such that in this new representation space that an Amex is linear because if the dynamics is linear then all of a sudden control becomes easy and you turn your original pixel


space problem might be highly nonlinear very complex to have a control methodology for into a latent space problem where it's just linear and very simple to solve that's the main idea behind is embed to control paper we're covering now so the existence they considered were pendulum card pull and three linked arm but again this is from pixel soda a pixel input


when the lid representation where hopefully dynamics is close to linear and hence control becomes easy so it's called stochastic control the methods they apply it's kind of a standard control method then you can apply to linear systems and embed to control will learn a latent space model using pressure on encoder while forcing a locally linear latent space dynamics


model so once you have a local inner model you can apply stochastic optimal control is an example that in action where once you have such a model is very easy to find the controller that brings you to a target and say a stable fixed point thanks to that controller or just to work well locally along this trajectory you seem to have linear dynamics models and in


fact the way this methods work is they tend to linearize the dynamics along trajectories but when if you learn a latent space model where it's already linear we already good to go and that linearization will not be an approximation or actually be the action model that you learn to that be nice to have a very big fit of your linear model to the absolute veneks so the costs are


often assumed to be quadratic so that's that's an assumption to make you know this class of problems called lqr problems later through out of control problems sometimes LQG problems if you also have some use to bestest be in there and these problems assume that you have linear dynamics and quadratic costs without a cost meaning there's a quadratic penalties for being away from


the state where you're supposed to be okay so of course we can't just throw from our original pixel observations to some space where the NamUs is linear and ignore the real-world dynamics esta map button my lab back out to real world to them so let's look at the complete loss function to look at first of all go to latent space see you need to be able to reconstruct the original language so Z


should not lose important information about what is happening or what's the situation here then we have this temporal aspect here and I'll Polly want to reach a goal and want to have long-term act prediction that in the end put the sequence of actions that are keys the goal it also predicts that's going to be the case so every step along the way we're gonna have prediction for


when and use linear models so prediction must be locally analyzable for all valid control magnitudes such that when we optimize our controls we get something then when it works in simulation also works in real world now we're going to force that to be true by learning a model that does this by construction so let's look at that model here's the next component we already


have our encoder decoder we have our control input u so in controls usually he was used for control input and reinforcement often a is used for action controls need for controls then we have our next latent state CP plus 1 now for this to be meaningful the same decoder should be able to reconstruct the image input at time T plus 1 if that's the case then that latent space dynamics was


correct okay so we're going to learn a locally linear model here in that transition to make that work okay then once we have all that in place we pretty much good to go we're going to use this model over long horizon C to make sure that actually that we don't just do this over one step we actually lay this out over and longer horizons and as we've trained the model we have


multi-step predictions over which we have this loss function you might say why do we need why do we need all this well it turns out that if you make a small mistake and your prediction for the next state then you might say nah Bob just a small mistake no big deal but if you make a small mistake the problem is that you land in a new latent state for which your model might not have been trained


and when you make the next prediction to go to time T plus 2 you're doing it from a time T plus 1 latent state that you're not familiar with that doesn't lie in your training distribution and now you might make a not so good prediction make it even worse and this is an accumulation of errors over time can lead to divergence and explicitly avoided any kind of simulations to run


over longer horizon need some mechanism to avoid that okay so one mechanism is to explicitly have a loss of a multi-step another mechanisms to ensure that your next state prediction comes from the correct label distribution so if you embed into my Tate unit Gaussian spins then after you do your next state prediction they should also that what you get there should come from


a Gaussian unit gosh and distribution to ensure that when you go from there to the next one you're ready to make your predictions all right so those are the components we have an autoencoder tutoring image X into latent state with accurate long-term vision of latent states because we ensure that the next latent state comes from the correct distribution a unit Gaussian just like


our auto encoder it forces it to be and then the prediction must be locally in erisa bowl so we don't get some fans in their network to predict the next flight instead from the current day and see if it has to be feasible with just a linear prediction okay so this is the full system that they proposed all the last term's shown at the bottom now let's take a look at how all this works so


they apply this to cart poll that showed a good amount of success in car pool and then here are some evaluations on on that showing that embed to control indeed can do in virtual pendulums swing up pretty well I can do carp or balance you can do three link arm so good results on three environments that they experiment with and here's this environments look like this is from


broad images so what we're watching is effectively also what the agencies the agent will often see the down sampling so they can actually look at the the environments themselves so really much on the left because worked on the right and here we have cardboard balancing in action and so this can use some idea of how capable this approach is so it does very well at the same time clearly these


environments they're not nearly as complicated as what we saw in the Unreal environments where it was deep mind lab navigation tasks versus these kind of 2d relatively low-resolution single robot that you fully control kind of tasks now in embed to control the idea was to have a single linear system and for your full of enemies that might be difficult but it's been shown in controls is that very


often even though your real system is highly nonlinear locally it can be linearized and so you might ask the question can we instead follow the same philosophy and I was in bed to control but instead of learning the single linear model can we learn just a collection of linear models in some way that allows us to apply time varying linear control methods which are also


extremely efficient and maybe have a richer set of environment that we can solve for because time varying linear models can cover more than just a single linear model can that's actually what we did in this work called solar showing an action on the right here we now have different linear models at different times and so we learn to embed into space where at each time a local linear


model can capture the transition very well so you still get initial random rollouts followed by learning representation and latent dynamics but now I'm not a simple linear model but the sequence of linear models and then from that once we've done that we can start doing a robot infer where we are in this thing's getting out of in your models find the sequence of controllers execute that get


new data and repeat the smaller base to draw in action a model-based reinforced waiting in action on the distant if we're setting where we make the latent space very efficient to find optimal policies and so might not succeed on the first time around so get the new data update the representation infer where we are in terms of linear dynamics models and trying to updated policy and repeat and


this can actually learn in about 20 minutes to stack it like a block learning it from from pixels as input okay so we looked at state representation which is how to go from broth acceleration to state learning ahead of time with a few in the world models paper that we looked at after we learned dynamics model and mapping from pixels who stayed at the same time and


maybe benefit from that now here's another way we can think about this which is we could think of putting some prior information so when we have pixels as inputs and we know that under the hood there is a state thing we know that state is just a bunch of real numbers so we did here in this papers is said okay when a plug data we're going to learn a latent representation which is curated by


sequels of column filters then we're going to apply a spatial softmax meaning we're going to look for each of these 16 filters where each filter is most active through a spatial softmax and output the corresponding coordinates those coordinates should allow us to reconstruct the original image because they captured essence they recorded to the objects in the scene if you know the


courts of the objects we've seen at least as the home you can reconstruct the scene and then once we have learned that representation we could learn to control with just a 32 dimensional input rather than needing to take in 240 by 240 input which is much higher dimensional and much more expensive to do reinforced fighting against there's actually capable of learning a pretty


wide range of skills here so here is the data collection so it's just randomly moving around collecting data that data is used to train that spatial auto encoder and sir then we learn we look actually we imprinted the goal situation and then we do reinforce learning in the feature space the thirty two dimensional feature space and learn in a relatively short amount of time how


to push the block to the target location it's not ready you can follow and how to go from image observations to state or something likes hearing kind of interesting method here them it actually doesn't bother reconstructing it says all we need to do is think about physics what is physics tell us well we're gonna want to find an encoding of state underlying state coming through the


observation fine here will be the big neural network that turns image observation into underlying state well what do we know about state we know that in physics then there will be coordinates and then derivatives of coordinates which are the velocities of these objects so there is a state variable corresponding to velocity and other severe I'll compare corresponding


to position and the change in position is velocity that's we know that velocity is derivative of position then what else do we know we know that when the world to be in different states we're going to need you know different state values so by default if we you know get random stuff presented to us we want the embeddings as in a field two different situations


to be far apart so that's what this law is is saying we want embeddings to be far apart but in all you do is play the embedding is far apart well then that's not enough to get any structure so then the next loss here says that four consecutive times the position state variables should be close it also says that between time T and t minus 1 the velocity state variables should be close


because philosophy cannot change quickly so this is saying acceleration people is going to be small on average an acceleration is gonna be small then conservation of momentum and our energy is captured in here and in the last part here Singer we need a representation where the actions are able to influence what the next state is going to be so wanted correlation between action and


state all right so it tested is on a couple of environments you know where they would just collect data in these environments pixel input and then learn a state representation that doesn't do reconstruction just try to satisfies those invariants that are expected to be no good loss functions based on physics and one pretty interesting state representations that way here's another


example of state learning in action going relatively quickly earth-2 just gonna give it a lot of different ideas across we've covered the beta tae and one of the early lectures beta V is a very solid encoder we'll put a coefficient beta in front of the KL loss on the prior and by making that collision beta bigger than one effective what we're doing is we're trying to make


the latent variable Z maximally independent so we're trying to find a disentangled representation of the scene and so the thinking here is that well if we want to find something that we think of our state from raw pixel values and probably we need to find something that's really strongly disentangled and so it's putting that prior into it and they show that by having this beta V you


actually get much better transfer so they train a beta vini and then do Q learning and bit in a network that takes the embeddings from the beta ba and compare it with regular Q learning and so on the left here we see what happens in the training environments the training environments regular Chi learning and Darla's which is few learning with the beta V representation


learning do about equally well but when we look at a new task related task but looks very different by doing the representation line which is shown at the bottom right we have to get much better performance top left is actually not getting a job done it's not collecting these yellow targets whereas bottom varieties we look at at collecting yellow targets and what's


changed while the walls in the background have changed to pink rather than green and the ground has changed the blue rather than yellow so a relatively small change originally QN this doesn't do representation learning per se hasn't learned those notions whereas stability has somehow learned a representation that allows it to transfer here at zero shop to this new environment much better


it's not our idea of representing first state and dynamics we looked up Gantz and I think lecture four of this class now if you just train again what happens is that you just transfer each frame independently what we want this we want to learn but there's intentions that are consistent over time as we're going to do is we're gonna have a discriminator here that looks at two


consecutive observations and decides whether those two consecutive observations are consecutive observations from the real world and welcome secular observations generated by a generator and so if a generator here trying to generate fake sequence of observations trying to fool the discriminator and at convergence what that means that this generator is trying


to generate observation sequences that are indistinguishable from real-world observation sequences once you have done you can use that generator as a simulator and learn that simulator or planning that similar in this case we did planning to try to achieve goals we will see on the right here is we didn't actually did this for rope manipulation so on the left is the initial spin of


the road on the right design end state of the rope and we see us with causal info again it thinks that these are the interpolated states so it thinks that this is a sequence of states you have to go through to go from the initial state to the end state same for the next one next one next one compare that with VC GM which we also which would currently just doesn't look


at transitions just looked at individual frames we see that the interpolation here doesn't necessarily lead to intermediate states that are that meaningful for a robot to try to follow that you know sequence of intermediate states and rope configurations to get some start to go and so we're able to by training in Foca which looks at realism of transitions rather than just realism


of individual frames is able to learn a dynamics model in a latent space that we can use for robot to make plants now one of the first things we covered was the world models which showed that you can learn a latent space and then learn all right on top of the latent space for dynamics and then learn a linear control on top of that of course that's a very new you think it's


almost surprising it works in there what's so interesting that it actually does work in a range of environments but hopefully it's not not likely to be the final answer to keep it that simple and so here's a paper called planet learning latent announced models from pixels lesson planning in it so what's what's new here is after learned laden space 10-ounce model it's actually risk is not


deploying a policy of learning it's using a planner look using look ahead as a function of which sequence of actions do I get the most reward taking that first action that sequence of actions repeat and here learns the latent space encoding together with learning D dynamics also is joint learning of encoding and dynamics recently has been an improver that is called dreamer from


the same office roughly and what they show is that you can actually run a limb planning in in latent space you can actually train a active critic and Leyton space simulator and that'll actually do better than i'm the planet he also showed that the dynamics model you learn it's better in these environments to learning stochastic dynamics model rather than in the


domestic dynamics model and that there's a two big differences between planned a dreamer going from planning to learning active critic agent and using a stochastic model now so far we talked about latent space models and directly learning to control in the latent space there is also work that actually goes back the image space and so here are some example executions by robot moving


objects to target locations and this is done by this the system here learned a video prediction model so i learn as a function of action the robot takes what will be the next frame i see and i long to the next action will be the next frame RC and so forth once you have a action conditional video prediction model and if you have a target frame or target property that you want to achieve


you can now use this action traditional bigger prediction model as your simulator and this can give really good results actually some examples shown here the slide get the downside of this that often planning to take a long time because to generate an actual traditional video prediction it can be fairly expensive we need to generate actually many of them because you're


trying different sequence of actions to see which one might work the best and then after you find one that might work the best it might be a sequence of ten actions you take the first of those ten actions and you repeat that whole process and so these things tend to be not as real time as some of the other things we looked at but it's very surprising how all this works you can do


full a traditional video fliction and manipulate objects that way now one thing you might wonder is it's all good and well to do full detailed video prediction but is it always meaningful imagine you drop the bottle of water class ball of water drops on the floor how are you gonna do video prediction 1/2 for what happens there very very hard I mean you you're never


gonna have access to all the details of the state of the water in the ball all the little defects that might be in the mid you know water bottle materials and so forth that will determine how exactly this thing fractures the best you can be able to do is probably why I think it's gonna break into a lot of pieces and pieces of different sizes and you know maybe the the net tongue stays together


because it doesn't hit the ground it's the bottom that's hitting the ground and so forth and you also don't need the details like to make decisions you just need to know it's going to break and so what you could do say hey instead of learning a fully dynamics model and say I need to learn just what the future will look like be able to predict that you say hey what if I can predict what


action I took for example seeing this shattered bottle and say well the action taken was dropping the bottle and so if I can go from I can make that prediction then I can also understand you want to achieve a certain goal what action might leave me there and not with me there this is called inverse dynamics and so that's at the core of many other dynamics models that being learned


throughout that for dynamics learn an inverse dynamics more effectively like learning a goal condition action strategy so no is it a paper here what if it said as follows it says we want to learn a Ford model and latent space I want to sleep in space - of course in fact will represent the things that matter but if all we care about is live in space predictions then the problem is


that maybe we'll make our little space always zero and we picked always zero we're always good but we don't have anything interesting and so they're gonna say well we want to learn a little space we would complete the next latent state but to avoid it being all zeroes or any other way of being degenerate we're going to require that from the latest state of the next time


50 plus one and the light instead of the current times et which we offered to predict the action that was taken at time T and so we went to dynamics models at the same time we're learning a inverse dynamics in a fluid dynamics model at the same time in this light in space this is applied to learn to poke object so well you see here on the left is data collection you can set this up


for autonomous data collection on the right where you see is the learned control so it's learned that the Namek law and now I can look at the current state and look at the goal state and it can do a prediction of which action is going to help the most to get close to that goal state and can repeatedly do that well it finally reaches something very close to the goal State okay now


reinforced planning is about reward so far ration mostly ignored the rewards when we learned representations and so we'll switch that up now let's not just learn to predict next state but also learn to predict future reward kind of first paper down or first recent paper that looked at this and the deep reinforcement in convicts is a predictor on paper so enter learning and planning and what


they said is well it's difficult to know what needs to go into the latent state and so because we don't really know what it has to when laid instead and we don't necessarily want to reconstruct the full observation because that's just so many things to reconstruct them we really want to focus on the essence well if what we care about is getting high reward should we just focus on predicting


future rewards for every sequence of actions we can predict the future reward well we should be good to go then we can just thick secretive action that leaves the highest feature award and we're good to go predictor on did this for some relatively simple environments showing here billiards as a function of which actually you tink how old the billiards balls and up and I should have pretty


well on that task and also they looked at it for maze navigation now the most threesome is all the scores that you might have heard of that builds on top of these very directly is mu 0 mu 0 is also learning a blatant dynamics model that predicts rewards and doesn't worry about reconstruction and so this one here doesn't just given one action in the beginning what's the sequence of


latent states that allow me to predict reward in the future and use 0 same thing but now action conditional and was able to solve a very wide range of game situations I'm on a variation is successful feature so you might say it's enough predicting reward which is just one number what if reward consists of many components gave a clear about location of the robot maybe I care about energy


expend maybe I care about other things these are all features and so the idea here is then if I had a set of features that relate the reward why not learn to predict well learn a latent space model that allows me to predict the future sequence of features encountered we looked at this ourselves in the comics of navigation actually so when you have a robot that's navigating a world it


does some convolutional processing of its observations then they'll be some lsdm because when you're navigating you currently see something we want also remember things you've seen in the past it's in memory here and then some that should try to predict observations features of observations they might in the future for example might have a pollution or something like that here's


this system in action so we're gonna so what we have here let me fast forward this a little bit to the experimental setup so what we see here is this is inside a simulator actually for now but also real world experiments coming later you see the kind of visual inputs this is processing and it's trying to predict things about speed hiding collision those are features it's trying to


predict so I put it know this many steps in the future well my heading B will my speed be my collision be based on what I see right now and based on the actions I will take any intervening time through that is able to learn in a total internal representation of how the world works but most importantly how the world works as it relates to features that matter


for navigation versus try to learn everything about the world which might be a lot to learn relative to what you actually need to learn be successful at your task and so based on its able to learn to navigate these environments pretty well then so that the real robot so here we have the actual robot that's going to learn to navigate the hallways in quarry all over at the electrical


engine electrical engineering building at Berkeley so we see here and actually when it's still learning has a lot of collisions but it learns to predict that it learns something say if I think if I see this take that sequence of actions I will have a collision in five time steps or my heading will change in that way and so forth and so after training its internalize a lot of how the world works


now I can plan against a transition well I need to act now let's go to this is test let's learn now we can see that it's learned to avoid collisions and in terms of what it's doing it it knows to predict as a function of the actions taking whatever which is likely to happen or not well heading it might end up with and then take actions accordingly and again the reason I'm showing all these videos


here is because as you see different approaches are testable very different environments and this by no means a converged research field and there's a lot of variation how things get tested and by looking at how its tested to give you a sense of how complex an environment a certain approach might be able to handle now a natural question you might have is well this is all great there's all these


different ways of learning representations but could we come up with a way of optimally representing the world what would that even mean what does it mean to have an optimal reclamation of the world well there's some worried especially trying to get up this so here are some fairly theoretical to be fair references on trying to understand what it means the popularization of the world and one


thing you'll often see come back is his word oma morphism and when it refers to is that essentially you have the real world you have a simulator and you want it to be the case ad if you go from real world to some weight and space simulator so you have a one-to-one match that's happening you go from from lit from real world to this latent space representation at that point you


simulate in both worlds and then after a while he tried to map back and see if it still corresponds but homomorphism would mean that you had still the correspondents many steps if he's are any number of steps in the future and so that would be kind of a by simulation homomorphism type approach and the question of course becomes what's the minimal life in space that you need to


be able to do that just the more middle that laden spaces the less variables you want to do was as a reinforcement winner or a planner who tried to learn achieve good reward in the environment now one thing that's very well-known in traditional controls is something called separation principle and separation principle in traditional control says the following and it's not well it's


very specific snare it says if I have a linear dynamical system and I have noisy observations of this state so I don't have access to state I only have noisy observations and these noisy observations are linear functions of the state so linear dynamics observations linear function of this state then to do optimal control in this environment where I don't have full


access to the state all I need to do is to find the optimal estimator of state which will be a common filter and use data out with my best estimate of the state at every time combine them in the optimal controller assuming I have full access to state so the separation panel says I could several enzyme an estimator and a controller design them separately and then combine and that's actually optimal


and that's actually very related things we've been talking about one of the representation I wanted to control want to be the case I would do the right can representation when decision comes out of it and just be used with optimal control we get the optimal result and so you some work now trying to look at what you have a nonlinear system might apply deep neural networks and so


forth what does it mean to have you know optimal estimation of state from your observations and how you know when is that compatible with your control and so forth so very interesting theoretical direction if you're more Theory inclined so another way to think of it is to say well shouldn't I just think about it end to end so often in deep learning you have kind of two paths one path is


you're gonna try to design something and the other pattern you say hey I'm just think about the result that one is the result that one let me define a loss function on the result I want and then training a staff instead of putting all the modules in more detail together myself so in this case what it might mean well instead of learning representation for a dynamic smaller and


then bolt it on a planter or bolting on a reinforcement agent why not say hey when I learned my dynamics model I should train it end to end such that what I learned is maximal compatible with a planner that I will use in the future so this goes a little bit back to the early thing we cover the embed to control we said if we can learn a linear dynamics model in latent space planning


comes easy you're gonna say what a feel a more general plan and my mom and so that general planner might work well in a wide range of situations now can we learn a representation that if we combine it with a more general planner together they function well if so then we learned a good representation so when we did this and some early work validation that works led by then post hoc Aviv tomorrow


now professor at Technion we showed that actually the validation a very common way of doing planning for toddler Markov decision processes actually this validation process can be turned into a neural network representation and so we can then bolt this validation network onto a representation Learning Network and optimize them together to try to get good performance out turning image input


into a representation on which validation runs and the encoding of image input will need to be such down the validation process actually gives good results and we even gave the validation process and flexibility to learn parts of that which showed that this way you can actually get very good performance on planning tasks they might say well planning with visual inputs


shouldn't just choose you just be able to learn a confident that just kind of looks at it and makes the right decision well it turns out really if sometimes what we're doing here is building a very strong prior intercom by building the validation aspect into it that's a bit like why do we use a confident we'll use a continent to encode translation invariance and once we can learn more


efficiently than if we were to use a fully connected Network it's kind of the same idea here we're learning that work that should solve a control problem that under the hood uses planning well then we should just put that planning into network the planning structure into the network so we can learn it all and when and now one question that has often come up in this in this context as well she


we ever do pixel level video prediction that's a good question I mean awfully you're just looking at noise and what's the point in trying to predict that what really matters is predicting the things that effect so how do you do that more directly so we're going to use plan ability as a criterion for representation learning now so validation that works as I just


described go into a little more detail it says have an observation observation it's turn goes into a module that outputs a value function which is how good a certain state is they put that out for every every state that you can hang on in parallel then an attention mechanism will look at the currents observation and understand which of all these possible stages


should index into then make a decision on what to do in the current state the value iteration module looks at essentially validation cannot remember that sense what it does it needs to look at reward and dynamics model and then this and rewarding dynamic smaller can do a recurrent calculation to get out the value of each state so this is just a recurrent calculation repeating


applying the same operation so some recurring moment and it's a return on that work with local calculation because states next to each other can be visited from each other and happen to show off in this dynamic programming calculation so turns out that there is a recurrent component is enough to represent validation calculation but we don't need to do it with some evaluation which only


applies to situation we can have a tabular representation of the world which means for very relatively small discrete states places he knows more generally so we're looking at here is universal planning Network universal fine network says okay we have an observation we want to achieve a goal observation we take our initial observation we're turning to the late


instead we're gonna encoded then we take an action new let instead looks we take an action new let me say we're not actually late in state and so forth taken on articulated state and after that series of actions we want our little state here to match up the delays in state of the goal of the rich that water key so we can do is within search over actions that will get us close and


so if we had already trained this live in space dynamics model all we would need to do is to optimize this sequence of actions and if this is a continuous space we can optimize the sequence of actions and back to the dishes a look around standard vacuum Gatien define a sequence of actions that optimizes how close we'll get to the goal so that's the planning part assuming we


have this dance model we can run back obligation to play how do you get the dynamics model well here's we're going to do we're going to learn the dynamics model such that so we're going to try to find parameters in this dynamics model such that if we use those parameters to run this optimization to find actions then the sequence of actions we find corresponds to what was shown in a


demonstration that we're given so we're given a demonstration a sequence of actions and we'll have an imitation loss and that will say we want to be able to imitate the sequence of actions by writing this very specific process of optimizing with aggregation our sequence of actions against a dynamics model that we're going to learn once we have learned that the nameks model this way


what it means is then onwards we can learn we can use this latent space dynamics model to find sequence of actions to optimize how close we get to some other goal in the future so benefit here is that internalization that our inductive bias than just learning some backbone black box for imitation now she also learns a metric in this abstract space that's useful for reinforced learning in


the future so we're comparing us with a reactive imitation learning it just says okay I need to just imitate a sequence of actions but this black box known that doesn't know that when you imitate probably the demonstrator had a goal and you're trying to find something of actions that it keeps that goal so it doesn't have that inductive bias it's not I do as well and something closes


the architecture we use is something that's also a recurrent neural network but doesn't have the internal optimization process in the inner loop to find a sequence of actions that optimizes how close we get to a goal so task we looked at here was some maze navigation tasks and also reaching between obstacles to a target the courtesy here on the horizontal axis


number of demonstrations the bring boxes is the average test success rate oh it seems Universal planning networks outperforms the baselines that I just described but that means that building that inductive bias helps significantly in learning to solve these problems now note that you can says well what did it actually learn we said to build an inductive bias we


say with building inductive bias to learn to plan in that inner loop but is it really learn to plan here's experiment doing say what if we learn with 40 iterations of gradient descent to find a civil actions and then we test with a very number of planning steps meaning we vary the number of Grandison steps in the inner loop when we do plan if our thing is doing planning then the


hope is that by writing more planning iterations it would keep refining the plan and end up with a better plan than if it always access to 40 iterations that's indeed what we see here after the horizontal actually we increase the number of planning steps the test success rate goes up for the same amount same training same training just different number of planning steps of


tests on so this indicates that likely is something like planning is really happening under the hood and if you plan longer you can do better nothing that happens is when you do this you learn a representation that ties into how an agent should make decisions that representation can be used by a reinforcement learning agent to learn more quickly what makes me a force wink


typically hard is that the reward is sparse but if you map your world into this latent space in that latent space where you're running this optimizer grading descent to find good actions well again bring descend assumes that there's some smoothness so once you've learned that we can space where there are smoothness you can optimize against that probably means that in that latent


space distances are more meaningful I think now do reinforce learning against distances in that waking space you're doing it against the reward that's better that's not sparse but it's dense and it's giving a signal locally on whether you're improving or not improving on what you're doing and so we showed in a wide range of environments I did indeed reinforcement learning can be


a lot more effective when using the distance in Laytonsville earning in the process I just described but then you do reinforcement in a new environment example we did imitation in three link and 4 link environments switch to a 5 link environment Rand reinforced wanting the file of environments faced and the latent space there is used for reward shaping and I guess you learn a lot more


quickly same thing here where the initial learning happened with a point mass and a to sub-point mass and then actually have the controller and robot and thanks to these shaping that comes from learning the slave representation where distances are meaningful learning can be a lot more efficient okay so at this point we've covered quite a few different ways of combining representation learning


with reinforced learning to be more efficient and the general theme so far has been that or at least in our state for positions been done raw pixels sure it has the information but it's embedded in a very high dimensional space is million megapixel image million dimensional input we wanted in a more compact position with and learn against more efficiently and all these ascribe


observations available to state and state actually the next date and so forth all tried to get a handle of that problem now nothing you might observe is down what we covered so far is fairly complex is a wide range of ideas at play and so the question we asked ourselves recently is is it possible with a relatively simple idea to maybe get a lot of leverage that we have seen here


and let's take a look at that and see how far agree with relatively simple idea and actually we'll see out the form essentially all the approaches we've covered so far that doesn't mean the ideas and the approaches we've covered so far are not important so we're not important with colleges to skip them there's a lot of good ideas we've covered that we probably want to bring


into this next approach we're about to cover but what I'm about to cover curl will really focus on simplicity and see how far I can get with something very simple our stunning exhibition here was if you look at the learning curves the vertical axis here is reward and higher is better horizontal axis number of trials in this environment and so see like at the end here


1e a to a hundred million steps have been taken in this environment and so we see a blue learning curve here that learns very quickly and then green learning curves that take a long time to learn what's different blue learns from states green learns from pixels same thing here blue learns from stayed very flowers green from pixels not nearly so fast and this isn't this case the RL


algorithm is soft is a d4 PG which is still yard are a logger so if you think about the essence here reinforced winning is about learning to achieve goals and if the underlying space is low dimensional there is a low dimensional state shims will be able to recover that low dimensional state and then learn just as efficiently from pixels as from state and how might we do


that well we've seen a lot of success in past lectures with contrast of learning for computer vision in fact we saw with CTC that it was possible by using unlabeled data is on image net to constantly out the form learning with label data so unlabeled plus so there's the same amount of label data but the blue curve also has unlabeled data you see that the unlabeled data consistently


helps outperform having only access to that amount of labeled data then of course very recently Sinclair came out and as actually getting equally good performance has supervised learning on image net when using a linear classifier just a linear classifier on top of a self supervised representation so that means that almost all the learning happens in self supervision and then a little bit


of learning habitat the M of course to get the meaning of the labels but it just needed a linear classifier if that's the case then the hope is if we do something similar in reinforcement all we need to do is do something where we do representation learning that extracts the essence and I've gained a little bit of extra information the reward to do the rest of the learning so


would it simply or do it essentially said I have an image I'm going to turn into two versions of that same image and when I then embed them linear neural network the symbol networking the left hand the right upper channels then the embedding should be close as measured with some cosine similarity and of course over another image that I embed then I'm betting should be far away and


those are the negatives in the denominator so for more details and that of course go back to our self supervised learning lectures from a few weeks ago what's important here is done this is a very simple idea it's just saying turn an image into two images and the embedding should be close take a different image it's embedding should be far from this and what's surprising about this even


though it's relatively simple it enables representation learning that then on top of that all you need is a linear classifier to get a really good image that classification performance and they actually looked at many types of augmentations cropping cut out color surveilled filter Norris blur rotate and what they found is that crop matters the most and color matters quite a bit too


but really cropping is the one that matters the most so now the curl curl looks like a nice representation learning with RL so what did we do here we have our a replay buffer on which we normally would just run reinforcement learn and so we have our replay buffer we take on my observations now to replay buffer since this is a dynamical system we need to look at a sequence of frame and consider


that a single observation otherwise we cannot observe philosophy and a single frame acknowledge their velocity so we'll have a stack of sequential frames that we together consider a single observation let's pack of frames then gets undergoes did augmentation to that augmentation in this case two different crops then one goes into the query encoder I'm gonna go key and predators


could actually be the same or different you can choose and then ultimately you do two things with this in the top path it just goes to the reinforcements in law so if you run B for PG again or you run soft actor critic you run PPO and so forth that happens along the top path so what it means is along the top path you run your standard our logarithm the only thing that's changed is that we


take this rare replay buffer you do some data on English now in the bottom path you have another data images in the same frame' you have a contrast of loss so essentially the same loss not exactly same details but at a high level same as we saw in the Sinclair slide okay so a couple of things that were important to make this work Sinclair uses a cosine laws what we


found is that having a a weighting matrix here between the key McQuarrie is Ashley Borden then we'll see in the red curve the bilinear and waiting is us secretly outperform using just cosine the other thing we notice is that using momentum Indian and one of the encoder pass is very important to which was actually dawn actually saw herself as learning lecture in the moko


work we also have momentum and one of their past same thing was important here again big difference so once we do that we can see that curl outperformance both prior model based and model free state-of-the-art methods so we look at here is medians course on deep mind control one hundred kid you might control firing the kiss it's hundred can steps are firing the kid steps and so


it's checking really can you learn to bounce it's not about after one hundred million steps where you ask is about 100 thousand firing down steps where are you at and so we see here after winter and kid steps from state ship access to state this is how far you get curl on one hundred K steps is a little bit behind what you can do from state but no family K stuff is actually all the way


there so we see that we can learn almost as well from pixels as from stayed with curl for Kepner prior methods that also tried to learn from pixels we see that they consistently we're not doing nearly as well after firing the kid steps and Sam with after hundred clear steps so both after hundred K M cavity steps curl up of homes or prior our elephant pixels on deep mind control Sweden and


after getting very close to take this learning here we have the learning curves in gray we see state based learning and red we see curl we see that in many of these red is matching gray there are a few exceptions within most of them red matches fray meaning that with curl are elfin pixels can be almost as efficient as RL fringe state at least for these deep mind control tasks and


here we'll look at you know a table of results you see in boldface the winner compared with all prior methods favorite methods of learning from pixels and you see that consistently curl outperforms the other methods for the tyranny and hierarchy not just an average but on essentially all of the individual tasks except for that no one here and one here dark public iris with curl doesn't learn


as fast and we look at the details what happens there these are environments where the dynamics is fairly complex so this requires some more research with our hypothesis here has been that in those environments learning from pixels is particularly difficult because if you just look at the pixels the dynamic is not well captured in the sequence of frames you get to see for example if


contact forces matter a lot and it's you can easily read those off from pixels and so having access to state makes a pretty big difference in terms of being able to learn looking at the entire benchmark we are looking at median human on normalized score across 26 Atari games at 100k frames and we see that compared to Paris today our rainbow rainbow dqm simple and well at least


rainbow DQ and simple and rainbow DQ and curl seen every out performs prior and state-of-the-art and it's getting at about 25 percent of human normalized score here is a broken out for the individual games and curls outperforming proxy they are fairly consistently what's simple coming in first on it's still two of them so computers are all matched human data


efficiency it's good question human normalized algorithm score we see on the freeway and on Janus bond that we get pretty much the level of human efficiency for the other games is a little bit of a way to go but it is not rotates night and don't know it's already double-digit percentage performance relative to human on almost all of them okay so we looked at two


main directions in representation learning in reinforced going so far using auxiliary losses and doing things down if I couldn't come down to trying to recover on the line state with a self supervised type loss now there are only ways representation that I can help mainly an exploration which is one of the big challenges in reinforced learning and in Austin for


unsupervised feel discovery so let's look at those two now first we can help exploration is through exploration bonuses so what's the idea here in a tabular scenario meaning a very small reinforcement problem where the number of states you can visit you can count but say there's only you know a good grid world addition to being only one of sixteen squares that's it one of sixteen


possible states a very simple thing new is you give a bonus to the agent for visiting grid squares it hasn't been to before or hasn't been frequently before that encourages going and checking things out that you have don't have much experience with yet that can be very effective in is small environments but its impact of a large continuous state space is because in a large but they


infinite States build infinitely many splits well there's always more stuff you haven't seen so you need a different way of measuring what makes something new versus something already may be understood so one big breaker in the space wants to look at using generic model in this case a pixel CN n for density estimation so the idea here is you planet our game or the agents


playing at target you want to measure how often has the agent been in the stick but if you'd never special specific stage there's too many of them so still we're gonna do is women train a pixel CNN model on what you see on the screen and things you've seen so far the more often you've seen something the higher the log-likelihood under that pixel CNN model but when you let's say


enter a new room in this game first time you enter the new room the log likelihood of that new thing you see on the screen will be very very low it'll be a bad score then it's a signal that this is something you need to explore because you're unfamiliar with it as measured by the flow log likelihood score as you can effectively give exploration bonuses now based on the log


likelihood scores under your pixel CNN model that you're trained online has your age and this acting in the world there's a comparison here between using the odds versus just using random exploration and it helps a lot another way to do this you can train a variational honor encoder which leads to an embedding and then you can mount these embeddings into a hash table and


just do counting in that a hash table and that's something we did a couple years ago Amin helps a lot in terms of giving out the right kind of exploration incentives to explore difficult to explore environments more efficiently another thing you can do that kind of gets maybe more at the core of what you really want but it's a little more complicated to set up is for


information maximizing exploration so the idea here is the following when you are in a new situation what what makes a deal what makes it interesting about it being new well one way to measure this is to say hey if I'm in a situation where after taking Archie I cannot predict what's happening next very well then I'm not familiar with this so I should give a bonus for you know having


gone into unfamiliar territory that's called curiosity we'll cover that in a moment especially been pretty successful but actually it's also a little defective because if you just have something that's too passive in the world let's say you roll some dice well it's gonna be unpredictable so to make this more charitable one thing you can do is you can say hey I


don't want to be getting exploration bonuses when something is inherently unpredictable how I'm going to get them what it's something that's unpredictable because I have not learned enough yet about this and so the way we did this environment COK you can set up a dynamics model that you're learning and as you learn the dynamics model as Nydia that comes in you can see we actually


set up a a posterior over dynamics policy of the distribution over possible dynamics models as new data comes in you get that posterior if that updated posterior is very different from a previous posterior it means that you got interesting information it allows you to learn something about how the world works so that should give you an exploration bonus because you did something


interesting to learn about the world but when throwing the dice addition guys rolled many many times and then rolls again and you couldn't predict it because that's just awareness you cannot predict but your model for the dice will already say it's uniform you know over all possible outcomes that model will not see much update if any and you will not be given an exploration products and


so that's the idea in vine only get exploration bonuses when it updates your posterior over how the world works and again showing here that that helps a lot in terms of exploring more efficiently under the hood that's really self supervising type ideas for a dead and small ensembles or based on representations of the AMEX models and been given exploration bonuses based on


that the simple version of that is called curiosity we're more directly look at you know was something pretty cool or not pretty quiet more the domestic environment often that's actually enough and that's in a lot of success in many of these game environments another thing you could do with self that's learning a representation learning for exploration is to think about it in a more


deliberate way you could say hey it's notice about getting bonuses after it's being something new it should also be about thinking about what I should even do before I experience it can set a goal for myself that makes for a good goal when I'm trying to explore train goal again what's done is the idea is the following you have a in this case let's look at iteration 5 down here the other