open-endedness is essentially you know we're studying systems that can generate their own data in an infinite uh capacity and so it's systems that essentially if you run it for longer and longer they get more and more complex they generate more and more quote unquote interestingness or interesting data um and so if we can actually you know crack this snut of how do we actually come up with a self-improving system in the sense that keeps generating interesting data uh we can then use that data to train further train our models but of course you get into this perpetual ual uh data machine type of uh idea where obviously you know there's how do you generate more data uh if you know the data is ultimately coming from a model that you probably trained on previous data how do you get net new information from that well I think a lot of this is actually just resolved purely again going back to this idea of the reward function right or a preference function where there is outside information coming in through some sort of filtering criteria for example human designers in the loop uh or designers designing some sort of preference model that could essentially automat Ally rate the kinds of automatic uh data that's being generated by these open-ended systems what does Waker stand for right so Waker stands for weighted acquisition of knowledge across environments for robustness fantastic and what was the title of the paper oh right yeah reward free curricular oh God what was the title of the paper reward free curricular for training robust World models that was it okay so um give us the elevator pitch yeah totally so basically like the overarching um question that we're trying to answer with this paper is like how should we go about training like very general agents so in the context of the paper we think of a general agent as being one that's able to perform a lot of different tasks so we might think of these as different reward functions if we're think thinking of it from a reinforcement learning perspective um but also be able to perform those tasks in lots of different environments so you know we don't want a robot to just be able to do you know pick up tasks or do tasks in my like my kitchen specifically we want the robot to be able to go into like arbitrary Apartments and also be able to do those tasks in like arbitrary environments and so we we kind of thought about like yeah how do we want to create an agent that can do such a thing and we argue in the paper that a good way of doing it would be to have an agent that um has a very general World model um so a world model meaning that it can predict the outcome of sequences of actions and predict what will happen if it does certain actions and so we argue if we have a very general World model that can lead to a very general agent that's able to um perform you know a variety of tasks in different environments and so then you know once we've established that we kind of asked the question of how do we get a very general World model and what does it mean to have a good World model that works um well in a very general setting across different environments and different tasks like how do we Define that and how should we gather data to do that beautiful so I really enjoyed reading the paper and um it reminded me a lot of um Kenneth Stanley's poet paper so he was uh doing this thing called curriculum learning and it's it's really related to machine teaching as well there's quite a few things in in M machine learning where you say well if we had a really principled way of of selecting the best training data and presenting it to the Learner in the best possible order could the learner be better and in that poet paper Stanley was kind of generating a diverse set of environments and like training um a learner on those things and you're doing something very similar and you're using this Mini Max regret which is a concept from decision Theory can you bring that in yeah absolutely so um so I guess we have this notion of like wanting to be um to perform well across a wide range of scenarios right so scenarios in our context mean like different environments and different tasks and kind of like the most standard way of thinking about that especially in reinforcement learning or in machine learning in general is you think about like the average performance so so how do I optimize like the expected reward across all of these different scenarios um and a lot of the work that that M's done as well kind of argues that just just optimizing for expectation isn't necessarily the best um the best objective so you know we can imagine in the real world we don't really know like the distribution over possible tasks or or anything well you know in most situ we don't know things like that and so maybe a better objective is to try and be robust instead and robust basically we can think of that as meaning like we should do reasonably well in every situation we could be in and that that's kind of what a robust objective is um and one of the ways that you can define a robust objective is via Minx regret and so regret means like suboptimality like how well did I do relative to the best I could have possibly done so it means basically the same thing as it does in normal English um and so the Minimax regret objective basically says across all possible situations I want to try and do um minimize the regret across all possible situations minimize the maximum regret I should say so that means in all possible situations we should do almost as well as the best we could have possibly done um and I guess just to contrast this against the standard objective for robustness so the more common objective for robustness at least traditionally is like maximan performance that means maximize the performance while the environment's like minimizing and choosing the most adversarial environment or the most adversarial scenario um but but the problem with kind of the max objective is that in some environments you just can't do anything let's say like some s situations is too hard you're doomed and so if in some situations you're doomed and you always get like zero reward or negative Infinity reward that means there's no incentive to try and do better in any other environment because your maximan reward is always going to be zero and so therefore I think like minchi argues as well as Michael Dennis and a lot of these recent papers argue that Minimax regret so minimizing the maximum sub optimality is actually like a better objective for a general agent that's robust fascinating so um if I understand correctly is is it a way of saying I want to have the best case worst um expected regret uh yes so basically Minimax regret is saying that uh if you assume that you know the environment is adversarial to you in in some way like when you're training or at inference time when you're actually testing your policy out in the real world um Minx regret is saying the agent should behave the model should behave in a way that minimizes its worst case possible regret uh over all the possible conditions of the world uh that that this adversary could choose what's really interesting about this paper is we are talking about the reward free exploration phase and we're also talking about the domain of modelbased reinforcement learning as opposed to um you know let's say value based reinforcement learning where um you get this entanglement right so the the Dynamics the model of the world it's still in there but it's kind of in meshed with this with this value model whereas in model-based reinforcement learning in a principal way we kind of separate out the parts so that we can do explicit planning and a and simulations and stuff like that so we're very much in this model based domain right yeah absolutely yes we focused on yeah model based reinforcement learning or some people like to call this like the world model setting more recently but yeah like you said we you know in typical like model fre reinforcement learning we we we typically aim to learn a policy in a value function and yeah as you said like that value function is kind of implicitly encoding the Dynamics through the fact that we learn the value function using the Balman equation so so the bman equation kind of propagates the information between like transition and the environments through the value function so so the value function will like implicitly have the Dynamics in it um but in modelbased reinforcement learning we want to very explicitly model the Dynamics of the environment and so what I mean by that is we want to be able to take some previous sequence of observations perhaps those are images and then also condition on the next action we want to take in the environment and then be able to predict the distribution over the next observation or state so we're very explicitly modeling the Dynamics of the environment okay now this is really interesting because you know people think about reinforcement learning and in reinforcement learning you don't so much care about having a model of the world you care about building trajectories that lead to some you know task or goal or whatever that you're interested in so like I mean just just in broader terms what what what do we get from explicitly modeling the world so there there are a few Arguments for why we would want to explicitly model the environment so so one of which is um a lot of people would argue that you get better sample efficiency by modeling the environment and the argument for this is you know the reward function might be quite sparse and so if you're just relying on like the propagation of rewards backwards to try and learn the optimal behavior that might not be as efficient as actually learning the Dynam DS because the Dynamics can be learned from every single transition that you have it's kind of like a standard supervisor run supervised learning problem so so you kind of have like a richer signal to learn from which might arguably lead to better sample efficiency um but I think like the more concrete arguments that I would argue for are that if you have a model of the environment it's it's some kind of more General thing that you can then use to develop better decision- making later on so so if you just learn a value function you're kind of only learning how to optimally do that specific reward function or optimize that specific reward function um but if we have a model of the environment we can kind of arbitrarily be given some task later down whether it be a reward function or goal state or something like that and we can then plan to optimize that task later down the road so I would think that um you know it's kind of a much more General way of having a powerful decision-making agent rather than just specifying like one task and learning the optimal kind of policy for one task and I guess another thing that I'll add to that is um rather than only learning like a feed forward policy like you would and reinforcement learning so something that Maps directly to actions the other thing that a world model allows you to do is also to do online planning so you can imagine at test time we're trying to deploy in the environment but we can actually do a bit more further planning through the world model to then work out what the best action is rather than relying on just a newal network to immediately output an action and there's kind of a lot of work showing that if you can do this like planning at test time you can kind of get a lot of a better performance on a lot of environments especially things that that really rely on um search to do well things like go and like these kind of games where you do have to think explicitly ahead in the environment and so I would think the main reasons you would want to consider um learning a world model and maybe a last point I'll just add is that I think this is kind of a um again like unclear whether this is true necessarily but but I think some people would argue that a world model will generalize better than learning a value function so you can imagine like a world model is learning things like you know State transitions so you can imagine if you if you're training on state transitions the model is kind of implicitly being forced to learn something like physics or something like that and so if you're like very explicitly forcing the model to learn something like physics You could argue you know we'll go to some new state and the rules of physics will still hold and therefore the world model will still be quite good at the new state potentially whereas if you learn a value function I guess it's a little bit less clear as to whether you're put on a new situation will the same kind of structure of that value hold as it would a model anyway sorry that was a bit of a long answer but no no it's fascinating I mean when I was reading the paper the one of the reads I got is um in machine learning we are often overcoming the curse of sparcity so of course like in trajectories and reinforcement learning that that's quite intuitive but even in learning the world model itself the model um just because of the way they're trained it it tends to compress the world into small little motifs and actually the world is quite complicated and we need to combine the motifs together in lots of interesting and and Rich ways and by exploring through the world model we're almost kind of like making we're forcing it to make those connections yeah and I think um you know to follow up on Mark's um uh Mark's point I think it's also interesting because especially in the Waker paper uh the world model setting we're looking at specifically reward free world models and so essentially there's this uh EXP decision to separate separate out the two components of a world model which is essentially the Dynamics function which tells you how things transition from state to state how does a state transition state of the world transition to the next state of the world given an action that the model or the agent is taking in that world and the reward that it receives so this latter part the reward is defined by the reward function and so uh you know I think Mark was uh to follow up on his point a lot of the benefits of the world model is in this design Arrangement is that you can compositionally separate out this Dynamics aspect from the reward aspect so the general idea would be why should an agent trained in such a world model be able to generalize to a new setting well maybe if that setting shares a lot of the underlying Dynamics in that version of the world for example rules of physics and the agent has learned how to exploit those to accomplish um navigation around that environment or reach different types of tasks uh achieve different kinds of tasks in that environment then you can um sort of superimpose a different reward function that essentially defines a different task because the reward function defines what task success is so you can essentially superimpose different tasks on top of that Dynamics model and you would you know you could expect that the agent could learn more quickly because it's already mastered sort of the foundational skills of navigating or manipulating different aspects of the Dynamics of that world we've been on a bit of a journey here um I think over the last few years in in the literature of um we we want to have robust models and we're doing that by kind of perur and you know making a bunch of manipulations to the environment and there there was this domain randomization and there's like unsupervised environment design and of course your your iteration now is doing this in in in the domain of um reward free exploration but can you take us on on that Journey sort of maybe starting with um domain randomization kind of just to uh elaborate on something that Mark was previously talking about which is that the typical you know standard setup in machine learning is to uh essentially optimize a model's performance uh over a uniform distribution uh over the data points and so this is really just randomly sampling data points and we try to minimize the loss over those data points for whatever objective we're trying to minimize or maximize in reinforcement learning um we want to train agents that can perform well in lots of different uh versions of the environment and so um you can think of each environment uh almost as a bundle of data points right it's kind of the set of trajectories that the agent can um can encounter within that version of the world and we essentially in reinforcement learning we want to learn to maximize uh the reward of the agent uh in that set of uh trajectories so we want to specifically start to uh actively pursue those trajectories that give us the highest reward and we learn from the reward signal as the feedback signal for figuring out you know which actions uh and therefore which trajectories will lead to maximizing that reward and so typically um when we operate in the multitask setting uh we essentially randomly sample different versions of the environment and essentially have the agent try to maximize its performance its reward on that random sample of environments uniformly uh sampled from you know the set of possible environments um and this is essentially uh causing the agent it'll cause the agent to learn a policy that's optimal for essentially uniform distribution over those environments um but of course this is kind of a naive assumption because we essentially are assuming that every possible version of the environment is equally likely which is obviously not true because some versions of the world will not be as likely as others uh for example like if you walk outside the sky is usually blue and not green and so you know when the sky is orange maybe that happens if you're in California and there's a wildfire but that's not usually the case and so instead what we can do is we can turn to decision Theory and think of um sort of more sensible approaches to what it means to act optimally uh when you're uncertain about uh what state of the world uh the world will be in and so the thing that we focus on in this paper um is this idea of Mini Max regret where it is this idea again of having the agent act in a way that essentially minimizes its worst case regret um in any possible uh state of the world so largely you know this is a shift from randomly sample what it means in practice is you want to shift from randomly sampling environments during training to essentially uh sampling environments that maximize the agent regret and what this means is you're now actively sampling for those environment settings where the agents um experiencing the most regret and here regret is defined just simply as what does the optimal agent do in that version of the environment and what did this current agent that's learning do in that environment and so there's this Gap in performance and you want to actively find those environments where that Gap is maximal and if you view this as this adversarial game now between you know uh an adversary like nature that's choosing the environment and the agent that's learning to solve the environment um you can think of the adversary as you know having a a payoff function in that game or it's rewarded for the based on the regret that the agent experiences and the agent is trying to shrink that regret so the agent you can think of as being rewarded for you know um the the negative of that reward so the agents reward signal is you can think of as the negative of the regret and so now you have the setting where you can essentially view this training process this active sampling process as a two player zero some game where the adversary is you know rewarded for the regret of the agent in each environment it chooses and the agent is rewarded based on the um the agent receives the negative regret as its uh payoff and so um we know that in two player Zer some games there's always a uh this there's always a solution called a Nash equilibrium and so this is an idea in Game Theory where basically this is um a choice of behaviors on both parties or a choice of strategies on both parties in the game such that um no play can do better unless the other player changes their strategy and so you can think of this as a situation where you know I'm not neither player is incentivized to deviate from their behavior uh once they reach this choice of mutual strategies and so we know that all two players zero some games have a Nash equilibrium uh set of strategies between the two players and in this case uh we know there's additional theorem called the Mini Max theorem which says that when in a two player Zero Sum game specifically two players and zero sum when um you are at the snash equilibrium setting then each player must be playing what's called U the Mini Max um the Mini Max strategy which means that each player is minimizing the maximum um minimizing the maximum reward for the other player and so here the reward again is the regret and therefore just based on this known you know theorem about two player zero some games we know that um the agent which is you know receiving the payoff of negative regret it's the Min player it must be implementing the Minimax regret strategy and so this is how we essentially can shape the training process to essentially um arrive at an agent that performs Mini Max regret decision-making rather than decision-making that optimizes um just a uniform sample of environments okay so can I play back um some of those things as I understand it so um essentially we we are we're building a model which will learn to select the environments where we perform badly on and then we fine on those environments because we're leaning into the gaps we're saying where where do I perform badly let's fine tune on that and then you're saying that if we continue to do this as a kind of adversarial sampling game that we will reach a Nash equilibrium so it will converge in a good place but help me understand that why would it you know it seems to me intuitively that it might be unstable or it might not quite what why does it converge so there's no guarantees around convergence and and so I think this is an area where there's a lot of room for innovation around these methods a lot of this is um this is more I would say like theoretical motivation around why we think actively sampling environment settings based on um estimates of regret is a good idea and another Point related to that around sort of this gap between the theory I I just um explained and in practice is that uh regret itself is a pretty hard quantity to actually uh measure in practice because you know knowing regrets defined as what's Optimal Performance um minus my agent performance so you kind of have to know what Optimal Performance is and in general you don't know the optimal Behavior therefore you don't really know the Optimal Performance on any environment unless it's like a very toy setting and so um in practice we also use approximations for the regret uh in order to do this kind of active sampling and so um there's a lot of deviations between theory and practice um so there's no guarantees you know that different forms of gradient-based optimization uh for RL training would actually lead to converging to Nash equilibria uh a lot of the theory is just stating that if you were to run the system the learning system for a long time if we make the assumption that the optimization algorithm is uh fairly good at producing you know an improved response to the other player in this type of Zero Sum game you if you're assuming that if the successive sort of series of best responses uh that the optimization algorithm is generating um continues to improve over the previous ones you could make the assumption that maybe eventually it does get to that equilibrium but um there is no mathematical guantee that this actually happens what we want to do is um uh you know build this latent uh Dynamics um uh you know predictive model which is a simulacrum of of what the idealized version is but we don't have a way of directly Computing the regret so we kind of perform um you know we learn a proxy for that regret how does that work so we think of regret in in the following way so so there's kind of this um old school result from like um mdp Theory or maybe it's not that old but like 20 years ago or something like that called the simulation lur and that basically says that you know if if we let's assume for now that we we have like an optimal planner so we can give our like model of the world to this optimal planner and and some reward function let's say later down the road we get given some reward function and so we give the model and the reward function to our optimal planner and we assume that this planner can return the optimal policy in our model um so we kind of have this you know planning Oracle um and if we assume that we can do that then we can think about the difference between like how good the policy would be from um a planning Oracle in the model versus the truly optimal policy in the real world and so what the simulation L tells us is that you know the difference between these two policies so the one found by acting optimally in the model versus the truly optimal one is bounded essentially by the error between the model and the real world under the distribution of states that the policy uh would generate so so you know it only it only matters that we have low eror where the policy would go essentially because you know if there are some states that are just completely irrelevant what the policy is going to do it's not really going to matter if the if the models not accurate there um so we kind of use this result to think about the regret so that that gives us like you know if we have like one um one true mdp and one model of an mdp and one reward function the simulation limit can tell us you know what would kind of be the regret if we did this optimal planning um within this one model of the um of the mdp um but then in our work we're we're not really interested in the sitting of like one mdp one reward function um so we start to think about you know what happens if we have arbitrarily many environments as well as arbitrarily many reward functions which we don't know in advance and then I guess the other thing that I should say like you you alluded to like latent Dynamics is you know these existing results are assuming that we have an mdp that's fully observable meaning you know exactly what the state of the environment is um but usually when we think about like World models or even or just maybe more modern reinforcement learning we're really interested in learning from like quite high-dimensional signal so images or maybe well probably images but maybe there are other high high dimensional signals we we want to reason about um and because we're just using image observations this means like the world is like partially observable like we can't infer everything we need to know about the world just from one image you know for for basically any physical task like the velocity of objects is important but you can't infer that just from one image um so in this partially observable environment we really want to take um a sequence of observations because we need to to use those sequence of observations to infer what the state is so you know viewing a sequence of images will help me to infer what the um the velocities are for example and so we can think of this as inferring like a belief a belief over what the state is in a partially observable nvp um so we need this full sequence of images and we need to use the full sequence images to then to be able to predict ahead what the next observation will be um that's kind of what you know most World models are attempting to do um but if we just like Tak in a bunch of images and then try and directly predict images again that's like quite a hard problem um to just like just predict straight an image space and so the most common thing to do is kind of to take your previous sequence of images and then try and get like some compressed representation of the history of images into like the latent State um and then predict the Dynamics in the latent state so yeah so I have my sequence of images I kind of compress these somehow into some vector and then I give it a new new action and I try and predict what the next kind of latent Vector will be given this new action and this now represents my prediction of the the Dynamics in the world and then if I want to um you know predict what the next observation would be in image space then I can also decode that back to an image um but then a lot of works also argue that maybe we don't want to actually learn to predict the entire image so maybe you don't want to actually decode the entire image but that's that's another aspect that we might want to get into but there's this whole broad story of of um working in the latent space and um in reinforcement learning there was that paper called World models by you know David Haron and Schmid Huber and it it also I think has a relationship with you know what laon's doing with jeer and these like you know joint embedding prediction architectures so there seems to be something magical about working in in the latent space and also you were talking about um you know partially observable Markov decision processes and you know that seems to be this idea that we need to have a modeling framework for the world and I I guess like the ideal situation would be is that like we just we we knew exactly what would happen you know every single time step in every single state um but we don't you know so so so we model it as a partially observable Markov decision process and the Markov bit is quite interesting as well I mean maybe um you guys can just introduce why do we use that as a model so markovian basically just means you only need to look at like the current state to be able to infer all the information about the system um so so in a Markov decision process we have some State and then we assume that we're able to take some actions and given some State and some action we get some distribution over next states of the system and then the the system will transition according to that distribution to the next state and this is just like kind of a general framework for modeling like systems that we might want to control so you know it kind of dates back to like early work and control theory but then it's also the main framework used in reinforcement learning um yeah in the reinforcement learning setting because it's the decision process we we also add in a reward function which tells us how good it is to be in a certain state or to execute a certain State action pair um but yeah as you said with relating to like partial observability in a lot of like systems we we don't actually know what the true like state of the world is so so you can imagine you know if we want to think of the entire world as a partially observable mdp we can't just have some Vector telling us exactly what the true configuration of the world is or or maybe that exists but we can't we definitely can't just know that and so we usually think of it as being a partially observable system um so this means that like given given the state um you know at each step we'll basically get some distribution over observations and we just get to observe that observ so you know the state of the world could be what it currently is in here and maybe my observation is like a camera image so I only get some camera image of the world that allows me to infer a bit of information about the state um and because it only allows me to infer a bit of information about the state it doesn't tell me the whole state it really you need to keep track of all the observations you have to be able to keep track of all the information you have about the world so you know you could imagine um if the task is for me to remember how to get out the door a while ago um you know I I I don't just need need to be able to like look at my current image of the world to be able to infer that information I need to have kept track of like all my previous information as well um so that's kind of why we think about often want to think about like partially observable environments as opposed to fully observable ones amazing amazing so so so mention maybe you can um uh bring in this this latent idea sure and and sort of contrast that to what Lon is doing as well sure I mean so I think in machine learning and deep learning uh there's this General Paradigm that's been around you know since the Inception which is learning uh late latent representations of data and one of the benefits of learning latent representation is that um you know ideally your objective uh that leads to learning these latent representations is that you are ultimately learning a lower dimensional representation of the data or dynamics that you're modeling like in our case with the world model um that captures just what is necessary it's a more compact representation of just the information that's necessary to predict the task you're trying to predict and so um with uh with our case uh or Laten space World models a lot of the benefit of working in the latent space is that if uh as opposed to working in the full image space for example if your observations are images like in a video game is that there could be a lot of sporus features or you know a lot of additional information that you could be expending lots of compute and um you know gradient updates just to learn those patterns when they don't actually impact the ultimate um transition Dynamics or reward dynamics that you need to learn in order to do well in that environment so one example is if you have a game where you know maybe the background is different uh because it's daytime or nighttime or it's close to Sunset um but ultimately you know the background doesn't really impact uh how the player moves around in the environment or whether they've reached the end goal of the task and so if you're training a uh model where it needs to compress a lot of this information first into a smaller dimensional latent Vector latent representation um you don't really need you would expect that latent representation not to actually capture it would start to ignore the background color and it might only capture certain features of the environment that can um essentially if you were to decode it back out it might only capture certain information about the environment that's predictive of the actual task that you want to solve um so maybe if the task is to say reach a coin at the end of a level then maybe the lant representation would capture the presence of the coin or whether the the proximity of the character you're controlling to the coin um and so uh with the JEA related work I think a lot of this is also you know motivated with with this idea where if we can learn a better latent space representation um of images or videos or whatever modality we're trying to model um it's a much lower dimensional computationally efficient representation uh that you can um you can effectively use for Downstream tasks um I'm not s I'm actually not super familiar with exactly you know the the visual JEA uh uh objective so I don't think I can say too much about that oh that's okay yeah I mean but but yeah I mean you pretty much nailed it so um I mean Lon even gives the example of like um you know in self-driving cars you might not be interested in the leaves on on on the road you know so like with increasing levels of of nesting you kind of like learn to ignore the things that are not relevant and focus on the things that that are relevant but we we're almost getting to the center of the bullseye here so in intelligence to me is all about model building and and and that's what these abstractions are they're models that kind of are predictive about the thing that that that's relevant and kind of like ignoring what is not relevant and we build better models when we have a curriculum apparently this happens in nature as well Max Bennett I was talking to him the other day and he said you know our genome doesn't encode all of our skills um explicitly because it would be too inefficient to do so but they do encode a kind of curriculum so we teach babies you know we Babble with babies and we teach babies how to talk and stuff like that so so the curricula is is really important and then we we're getting to the center of the bullseye which is intelligence in in general now I think Lon thinks that it's specialized and and what that means is that there there motifs that statistically generalize and what that means is that you do need environments you need to find motifs that are present in in as many environments as possible and those are the generalizing features do would you agree with that yeah definitely I think that a lot of um so a lot of really powerful machine learning methods for example uh are trained in simulation and when you're training in simulation there's a concept in control from control literature uh called the sim2 real Gap and essentially this is essentially quantif in a performance difference between uh well it's quantifying a few things one is just how different is the are the actual physical or other other kinds of Dynamics captured by your simulator compared to reality so if you have a physics simulator how accurate are for example the friction Dynamics or different kinds of contact Dynamics uh in your robotic simulator compared to those actual Dynamics in the real world with a real robot um and this also leads to a Sim tooreal Gap in terms of performance so if you train in the simulator you know a lot of times what machine learning is really good at is is really good at learning to exploit whatever system you're training the uh the model in and so it's fairly um common for you know systems that or models that are trained within a simulator to learn to eventually exploit the simulator and so actually like one big area of um games AI is using is actually leveraging this idea where they essentially use ml models they optimize ml models to within a certain game environment to try to find bugs within that environment to look for exploits automatically um so ml system is very good at finding exploits and whatever system you have but then the issue is those exploits are usually where exactly where the gap between your simulator and re reality resides and so you actually don't want your model to learn to exploit these differences between the simulator and reality to get a high performance uh because that kind of defeats the purpose of then later transferring your model that's trained in simulation to reality because now in reality obviously the model can't exploit those same those same glitches within the simulator um yeah so yeah yeah I mean because the reason this is really interesting is is that the the premise of your paper is that it is possible to build a generalist agent which means it's an agent that can be fine-tuned and worked really well on a on a whole bunch of Downstream tasks and to me that implies that at least in our physical world in any situation you might use this agent that there are General motifs that it could have learned during pre-training that it could like you know become activated in any situation um does that is is that fair yeah May I can say something about um just the way that we should could think about like the different like latent Dynamics objectives so so I think I agree that like at least when I try and think about how I think or how people think I think I agree that like you know a truly intelligent system should kind of think through the world and like a very compressed representation of the world like if I'm trying to like think through how to go to the airport like I'm definitely not like predicting ahead in terms of like the raw image space of trying to predict every image I might observe on the way to the airport and things like this and so I think we have this kind of like trade-off between you know um like you said with the VF paper like should should we just try and like um kind of basically model like the minimum information we need about the world to try and you know do the do the relevant task in the world and I think what you're saying I think that probably is maybe more what we think about when we think about like human intelligence or something like that um but then there's also this other way where we just say we're going to just like train enforce the model to be able to predict ahead every single image and so in our paper we do actually enforce that the model has to predict the next image um and so um basically what this might mean is yeah maybe the model does you know hopefully it does like like you said like kind of capture the underlying like true things that matter in in in the environment but it might also mean like what we were saying with like the leaves example like this might Force the model to kind of capture a lot of irrelevant details that don't really matter like the leaves on the ground and things like this and so you know maybe that means it isn't actually capturing the underlying motifs it's actually just getting good at image Generation Um but then or or image prediction I should say um but then I've also heard arguments kind of saying you know so what if people don't really think in terms of like image prediction you know I you know we think in terms of like like these high level motives but people have other people would argue that you know kind of the machine learning Machinery is there to do really good image prediction so so if if we if we can get a model that can actually just like predict images ahead really well um and not really worry so much about whether it's reasoning about these like high level features you know if you can predict images ahead really well you know that's enough to make to do good decision- making a lot of context so I think there's this kind of like contrasting ways of thinking about you know image prediction is good enough we'll just predict like really visually good scenes and that will be good enough for decision- making or do we want to the model to try and reason about like more abstract features of the environment and that's kind of a more intelligent way of reasoning about the world um and yeah I think that's a very interesting tradeoff um yeah yeah I mean like it's um like the biggest problem in machine learning is overfitting you know so as you say like that there all of these statistically generalizing features but they generalize within a domain and the domain might be like your your simulator or like you know how you're training it rather than how it's being used in in production and then as you say that there's also this Almost Human chauvinistic or puritanical view on this which is that well um you know it does the right thing for for the wrong reasons or or I I use different motives to do the reasoning so that thing must be doing it wrong you know what I mean and um I was talking with Chris Bishop at MSR the other day and and you know he's um big on symmetries and you know the kind of stuff that like Max Welling and Tak Cohen and bronstein and um deep M have done loads of cool stuff on on this but it's this idea that like we know the world um has a certain geometry it has certain physical prior so like we can deliberately um you know kind of construct the approximation class in machine learning method so so that like we it an easier problem right because because we know we know the thing is in there yeah so I mean I guess sort of the uh slight tangent I went into around the simt Gap I guess part of the point I wanted to make there is that um you know one way around the Sim to Gap is you could try to train um you could try to parameterize a very large space of possible versions of reality and this is kind of the motivation behind this method of domain randomization where you sort of say this is the you know this is the specific task domain I care about I can parameterize the different uh vers of the task with a few parameters and I basically want to search over the space of parameters and train my model or my agent on all possible variations of this world but obviously that's not very sample efficient because that design space could be huge could be massive and so instead we like these active sampling strategies like we were talking about earlier uh around Mini Max regret style um active sampling where you sample those environments that maximize your regret or some other type of objective maybe like uncertainty uh similar to what we do in the Waker paper um but ultimately these things this active sampling process it leads to uh what we like to call an auto curricul automatic curriculum um and this is in contrast to Prior curriculum learning works because here this is um an automatically generated curriculum so you you can kind of not have any predefined notion of what is easy or hard it's purely fixed to what is easy or hard for the model in terms of how good the model is at performing at those tasks and so it's nice it's an automatic curriculum so you can think of it as almost like weaving a path through this high-dimensional design space automatically such that if the uh agent or model were to train on data along this path of environments through its experiences in this path of environments during the training curriculum it'll basically be maximizing some sort of Information Gain objective um because you know for example regret if there's a high regret that's that means there's a high uh ceiling there's a high Gap in terms of how much the agent can improve which implies that there's a lot more for the agent to learn in those environments so it's sort of this like Optimal you want to find this optimal path weaving through this High dimensional design space of environments now the danger here is that as you do this uh Auto curriculum the auto curriculum uh could also get go Haywire very easily because the design space is so big if you're training and simulation which we have to do because these methods are so sample inefficient we need so much data to train them um you want to train in simulation but if you're doing the auto curriculum in the simulation design space it could start to Veer very easily and quickly into different corners or niches of the design space where um you know the parameters no longer really make sense in terms of mapping to a physical reality or a real world scenario that we as human users uh actually care about and so kind of it would be you know it would defeat the purpose of spending all this compute to train this model that could then help us in the real world because now it's veering off into parts of the design space that don't really matter for humans it's kind of noisy parts of the design space and so this kind of leads us to this question of grounding how do we ground curricula how do we align the curricula such that you know they can still do their exploration through this active sampling type of procedure over the environment design space but at the same still at the same time maintain at least some proximity to the parts of that design space that are relevant to what humans care about in terms of the actual tasks they represent I've been speaking with Kenneth Stanley a lot recently and we're talking about open-endedness and in general I've been trying to come at this problem from multiple angles and I've been using the lens of agency because I think agency is something that happens in the real world and that's why we have this Divergent process because we have multiple agents you know kind of like you know undirected following their own gradient of interestingness so in in evolution that's a great example of that it is this Divergent process but it's also grounded it's physically grounded you know so it's like the physical world creates some kind of constraints on on on the things that that are found and um I mean you know Clon called this AI generating algorithms there's quite a few different takes on this but the idea is that um to search this complex search space we we need to have a Divergent search and that's like we actually need to create the problems and the solution so like in the real world the the the you know the drafts had the problem of like eating the leaves from from from the trees and the problems and the solutions get generated in Tandem and this whole thing just kind of grows and grows and grows and that seems to be the most important feature that is missing in current AI systems and the grounding or the um Stanley calls it the gradient of interestingness I'm not sure whether You' agree with that but um I mean what Mark what what what what do you think about the importance of like this Divergence in in AI kind of the current Paradigm of machine learning of kind of like you know Gathering some data set beforehand or specifying some simulator beforehand if it's reinforcement learning is kind of good enough to do like a lot of reasonable tasks that we might care about um you know like obviously like predicting language or generating simulated language or performing very well at some simulated task and RL but it definitely seems like the next step towards like very general agents that are kind of you know I guess maybe I don't know if we want to use the term AGI but there something something more along the lines of a general agent that's kind of you know able to kind of self-improve and learn in more diverse environments um it definitely seems like that's kind of the next step of where machine learning will go and if we're going to get to that point I kind of agree with the idea that you know it certainly doesn't make sense to have some agent just randomly trying to gather completely random new knowledge like it certainly seems to make sense that you know you know even as a human to improve your intelligence you kind of selectively try and find out the areas in which like you can gather more more information or more knowledge and things like this and this is kind of what you know leads to this kind of I guess branching or you know like you said like the diverse set of things um that you might want to learn more about and so yeah I think like it clearly seems to make sense that like this kind of more openend this thinking is probably going to be like the next Paradigm of how we think about these kinds of systems but I I think M will had more to say about this I think the reason open-endedness is so interesting now is I think we're uh there's there's a few reasons why I think it's like newly relevant to this current ERA of machine learning because these ideas have been around for quite a while like um Ken Stanley Joel Layman um Jeff cloon uh Lis asaurus these a lot of these researchers they've they've been thinking about open-endedness and novel tbas search Divergent search for decades um I think it's really interesting to think about why there's sort of this Resurgence of these ideas now and I think a lot of it is because um it is again you know it's it's sort of following the same um sort of uh Tailwinds that have been driving a lot of the ml industry which is just like uh much better compute much larger data sets and I think what we're seeing now is that we know that modern deep learning methods work best when we can scale up the compute and the data that's how you get them to work um to to their Max small capabilities um at some point we're going to run out of data and a lot of people are now starting to talk about you know this as sort of a pending issue on the horizon which is you know at the current rate of consuming data for training our foundation models at some point we're going to run out of data we're where are we going to get the next trillion tokens from um and so I think a lot of this uh now points a lot of the interest to open-endedness because open-endedness is essentially you know we're studying systems that can generate their own data in an infinite uh capacity and so it's systems that essentially if you run it for longer and longer they get more and more complex they generate more and more quote unquote interestingness or interesting data um and so if we can actually you know crack this nut of how do we actually come up with a self-improving system in the sense that keeps generating interesting data uh we can then use that data to train further train our models but of course you get into this Perpetual uh data machine type of uh idea where obviously you know there's how do you generate more data uh if you know the data is ultimately from a model that you probably trained on previous data how do you get net new information from that well I think a lot of this is actually just resolved purely again going back to this idea of the reward function right or a preference function where there is outside information coming in through some sort of filtering criteria for example human designers in the loop uh or designers designing some sort of preference model that could essentially automatically rate the kinds of automatic uh data that's being generated by these open-ended systems and if we can do this kind of filtering we can essentially automatically find start to automatically find uh useful net new data net new trajectories net new even you know maybe um sentences like tokens or uh net new content to train our models on I've been thinking a lot about creativity recently and and I I think creativity is is is the other half of the coin of intelligence so in the world we live in I think that the intelligent process is is us we are a Divergent search and we are um basically tackling a complex search space and we are building knowledge and we we are mimetically sharing them in our society we're embedding them in our language and then language models come and acquire all of that knowledge so the cynical take is that AI today doesn't you know generalize and you it doesn't it doesn't creatively find new knowledge it just is a representation of the knowledge that we have found but it's not black and white is it so the work that you're doing is a great example of no no no you can generate new knowledge by exploring these complex search spaces and even though you're exploring existing models you're discovering interesting and novel combinations of those models that have not been found before so it's creating a novel margin on something that was not there before but I suppose the ideal future we want to get into is that we really can just from a far deeper level generate new knowledge yeah I think one interesting thing that I've been thinking about more recently you know is that um sort of the you know the high level question is just right now all of the state-of-the-art AI systems from chat gbt to stable diffusion style models for text image Generation all these systems they're they're amazing very impressive you know like 5 years ago I would not have believed that these systems could exist at this level of performance today but uh ultimately uh what they do is they're in the they're they're in the QA business so I basically ask these systems a question or I give them a command and they give me an answer um and so I think the next Frontier of AI is really how do we Design Systems that don't just answer questions but they actually are the ones that start to ask the questions and I think once we can have ai systems that start to ask interesting questions um that's when we start to get closer to I think traditional Notions of what uh strong AGI might be okay so so again really really interesting now so we're getting into agency and and people think that oh you could give a language model agency you just like you know run it in a loop and interesting things will happen well well it that's not true because the whole point of open-endedness is to prove that existing systems converge they don't diverge they don't accumulate information so we would need to create a kind of agent that like you know it would just keep running and it would just keep doing interesting and novel things it would keep accumulating information and I think that the reason why language models don't have agency is because they are essentially um a low entropy model and what what that means is during training a lot of the the sort of like the unnecessary um you know complexity was snipped off so the models only know about relevant things in the next step what's the next best token and it it feels like we would need to have not only a higher entropy search but we we would also need to have um a diverse set of models that are actively continually learning and and diverging from from each other but that's just my take I mean what do you guys think about that yeah I think that so I guess this relates quite a lot to this idea of like intrinsic motivation which is something that we utilize in our paper and I guess I guess the idea with that is like you know if we're trying to like gather new data in an environment like we shouldn't necessarily be constraint to just trying to new gather new data that's like good for a specific task um and so I I guess this kind of you know so intrinsic motivation basically says I should just gather new information because it's novel um and things like this and so we can basically like specifically try and gather information that you know reduces our uncertainty about the environment and and um or or similar objectives that that don't rely on some external reward signal and I think we when you get to the situation where the model is able to like self-improved in the absence of an external reward signal so intrinsic meaning that the the signal for what you should get is just purely generated by the model so it's purely intrinsic to the model um so I think the situation where you know you have the model that's able to self-improve without any external signal without a human having to Define what the reward is or what the objective is or this was good data this was bad data um I feel like that does feel like a lot closer to the notion of agency because of the fact you don't have kind of some external person defining what's good and what's bad and so yeah I think like this like and you also mentioned the word like creativity because I think at least in the context of things that I've done in terms of machine learning and reinforcement learning I think like intrinsic motivation feels like the closest thing related to creativity so you're basically like trying to gather information because it's novel or because you think it's or the model thinks it's interesting rather than because um you know it it satisfies some objective and so I think we could maybe say like intrinsic motivation is in some sense like an objective for being creative as well um I don't know if you have any thoughts about this yeah I think I think that uh it's I think there's definitely a a hugely deep connection between intrinsic motivation and uh creativity um in the literature intrinsic motivations also sometimes called artificial curiosity so this is a term that was coined by Jurgen Schmid Huber um could you could you explain it just what it is yeah yeah so oh yeah so taking a step back intrinsic motivation is essentially um in in reinforcement learning we train on reward signals and as Mark was saying um we typically train on external reward signal by external we mean that this is a task-based reward so this is um external in the sense that something outside of the agent that's learning like the human system designer decided that this is what the reward signal is for the task uh intrinsic means that we want to we don't design directly the reward signal but we're actually using some aspect of the model itself in order to drive the model's learning forward and so one example of this could be prediction error so if the model U has a large prediction error on a certain task like averaged over each time step we can use that as a reward signal and say hey you want to visit more parts of the environment where you're bad at predicting um how the state will transition when you act in that part of the environment and so uh as you can see this is very similar to maybe like intu Notions of what curiosity is uh curiosity and different forms of play um in the psychology literature A lot of people actually argue that you know different forms of play uh and curiosity really they they amount to you can model these behaviors as essentially a person trying to uh engage in activities where you know they're not very good at predicting the outcome and that's kind of what makes you could argue that's kind of what makes certain kinds of uh entertainment fun because or entertaining because you can't actually predict what will happen um you know in a few frames of the movie like like a movie wouldn't be very interesting or a book would not be very interesting if you can predict what will happen in the rest of the book just by reading the first few pages uh and so intrinsic motivation is really saying let's guide the model towards parts of the environment or the world or experiences where it's similarly unpredictable Stanley speaks about this this concept of deception or he calls it the false Compass which is this idea that any objective and and even you you could say exploring all of the search bace is an objective so he said every objective has deception and if you monotonically optimize any objective you will always lead into you know like a a deceptive part of the search page but then like the counterargument is say okay well let's let's not um let's not have any principles for doing the um you know the exploration let's just do something completely random and that doesn't seem very good so so then you know there's this concept of well how how do I how do I imuse some concept of what's interesting without falling victim to deception yeah so Ken Stanley uh has a famous essay in the realm of open endedness where he points out um that this notion of interestingness is uh ultimately a subjective concept and so even in the case of intrinsic motivation which I think is you know in practice we can get a lot of mileage out of this um and we've seen this in a lot of domains where uh exploration helps a lot like even in the Waker paper it's largely founded on this idea on how we exploit intrinsic motivation uh for learning World models but um ultimately you know these these modelbased uh measures of intrinsic motivation they are by definition based on the particular model at play and so um at some point you know you're you're starting to overfit to what that specific model finds interesting and of course what that model finds interesting if your measure of interestingness is something like a prediction error um is going to be a function of you know the specific architecture of the model the actual inductive biases of that model uh the capacity of that model to learn and so you could imagine a model where you know at the beginning it's looking for lots of interesting parts of a particular video game environment but at some point you know it might saturate what it can represent and what it can learn and at some point it might start to find things it's explored before interesting just because it's starting to forget those parts of the environment you know if you have like a very rich stream of different kinds of environments that it's exploring so ultimately this is like an example of deception because now it's like I I think that my model is the model thinks it's exploring parts of the environment that it finds interesting based on this prediction error But ultimately it might actually start to go back to other parts of the environment because of issues of model Capac capity and another really famous example of this issue would be like the noisy TV so like if your environment has you know this this um noisy TV where it's just showing random noise random RGB pixels um you know that's you know that's not something you can actually predict because it's just noise and so the model if your intrinsic motivation is really just to search for novelty in the form of prediction error it might just start staring at this TV forever because it's something that it just can't predict and it'll just by looking at that TV it'll be maximizing its prediction error yeah yeah it's so interesting um so so just coming into Rich suton a little bit so he had this idea called um reward is enough and and essentially that that's making the case that you know just using um implicit uh motivation all the stuff that that you've just been speaking about using this trajectory um you know optimization process that we can do everything we need to do and in in your paper you're kind of making an argument similar to what lcon has been making for years about self-supervised image learning that what we should do guys is let's let's kind of pre-train a base model so this model um understands environmental Dynamics really well and then we stick a reward in there and we build um agents after that so does it in any way reenforce or pun intended uh Sutton or or or or do you think it's still complimentary I think it's still complimentary at least if I understand the the meaning of the reward is enough paper because my understanding of that um line of thought is basically saying that you know we can kind of specify you know any task that we might might want an intelligent agent to do as optimizing a reward in some like mdp or prdp so Mark of decision process or something like that and I think our work isn't contrary to that in the sense of like you know I I do think that that probably is a sufficient framework to be able to model any any kind of behavior that we might want an agent to do but I think when it comes to actually like practically implementing that idea there's a lot of difficulties so the first one might be um you know how do we even specify that reward function so you know if the reward function is to um have a good life or something like this like there's obviously like you know maybe there is some like numerical way of defining that in terms of an mdp but there's like not actually a good way of of writing down that function that Maps what I do to whether I'm getting good rewards and so I think there's this kind of like you know I think that's a good framework for like thinking about any problem but then you have these kind of like practical issues of how do you actually Define rewards and how do you how do you say like whether an agent's doing well or not doing well and things like this um and so I think that's still um even with the world models lines of work I think that's still like kind of quite a difficult issue so so so the world models lines of work kind of you know allow you to model you know predicting a in the environment which is a very useful thing for doing a lot of tasks um but then if you actually want to optimize some specific task you still have this problem of like how do you define the reward and so we eventually want to get to this point of being able to like inject a reward into the world model so we're kind of in agreement with that kind of line of thinking in the sense we're eventually going to use a reward to derive the the the desired intelligent Behavior so I don't think there's any conflict in that sense but we still have this kind of problem of how do we inject that reward into the the world model how do we Define what that reward should be um and the case of um you know one of the easiest things to do for example would just be to label each image with a reward and then you can kind of encode that image into the latent space of the world model and then use that to Define how good a certain thing is and that's kind of the style of thinking we think of on our work um but I don't think that overcomes this like overarching issue of in general it's you know rewards can Define everything but how do you in practice like get that function is pretty hard yeah yeah I mean in a sense reward is enough is sort of a tonology because once you know the reward um if you know the the reward function for your environment you can essentially compute the value function which gives you the optimal policy um and so reward has to be enough if you know the reward function and so uh I think the more interesting question is definitely like what is enough for the reward what is enough to actually have a system automatically figure out what are interesting new rewards for us to train new agents or new models on or continue training existing models on um and I think this goes back to the question of environment Design This is largely the motivation of that line of work this autoc curricular environment design where essentially if we can automatically weave through this path of possible environments of the design space of the environments the design space uh clearly will Encompass like a big part of the design space is also encompassing the reward for those tasks and so essentially we want to find a curriculum automatic curriculum or path through the possible reward functions in which we can start to train a more and more General agent but then the interesting question is again like what exactly is the right notion of interestingness in order to drive that curriculum that path through the design space of possible things we could be training our model or agent on and um and that's I think one of the most interesting open questions and it relates to the question as well of how do we get the model to ask the questions um because really what drives humans in terms of asking further questions uh is our own implicit notion of interestingness which is informed by things like the scientific method and you know being able to create explanations about the world and we find things interesting when we can't actually explain some phenomenon about the world uh B based on existing theories or explanations and so I think what's really missing for a well-grounded you know human interpretable version of interestingness is having models that can essentially come up with their own theories about the world and start to probe those theories for where there's mismatch between you know the their learned theory of the world and evidence that new evidence that they find from experiences in the world yeah it's so interesting and and um I mean when I make the argument that agent should be physically and socially embedded it's it's actually quite a simple argument which is just the guard it it's that interestingness thing I think that that is how you know having um agency but with the guard rails of our physical and social embedding so you know we're we're sampling things that make sense because they're already there that you know but but but obviously we can go off piece a little bit as individual agents I I feel that that that's what helps that process just coming back to suton it's entirely possible that I've misunderstood suton by the way so my my interpretation of of reward is enough and it might be true as you say that it's tautological given that if you already knew the reward function for particular environment then it could do everything that it needed to do but my interpretation of of reward is enough is that it would lead to um a general intelligence and you know General in the kind of magical sense that it would work in in any possible situation but if it is specialized in the way that we agreed earlier that there exists a a reward function which would in you know codify motifs and things that you know you need to know or optimize in a particular environment or set of environments then to me that's still specialized intelligence and I would great yeah yeah that's that I think that aligns with my take as well where I think if you have a reward function um it's already sort of applying uh largely applies to at least the examples in that position paper about reward is enough it seems like most of the reward functions they discussed are largely um grounded in a specific task and I think that if you have the reward function for a specific task then it definitely seems that you can have some optimization or learning algorithm that essentially learns to optimize that reward and therefore achieve that task um so I do think sort of the open question that uh it I think saying reward is enough I think it kind of passes the buck up further one level to the question of where that reward comes from and I do think that having systems that can automatically design interesting new rewards that seems like the frontier yeah I I agree and and you know to me intelligence is about discovering the knowledge and the knowledge is the reward function so it feels like kind of baking the knowledge in into the system um okay so another sort of Galaxy brain take is um I was talking to Bishop about this the other day and um do you think of like deep learning models as one model or do you think of them as a sort of like intrinsic Ensemble of models because they they get you they behave differently in an input sensitive way so you know like depending on the prompt you put into language into a language model you might find that like a different part of the weight space gets activated and essentially it's like retrieving a mini program and that program is being run but it's not it's not model building it's like model retrieving but would you agree with that H I guess I'm not sure about the like like within like subsets of a single homogeneous model but I guess the thing that I like to think about that's I think quite related to this is this idea of like and I think Yan Lon also kind of well a lot of people have laid out like a similar architecture as like you know should we think of intelligent agents as having kind of like separate subsystems that can maybe like be thought of as different your networks and so you know we could have like you know um the standard notion of a policy which is like outputting actions and maybe we also want to have the notion of like a prediction model more like a world model that predicts what might go ahead in the world as well as maybe like a planner that is somehow good at like optimizing in that model and so we could kind of think of all these things as like separate subcomponents that we assume an intelligent you know an intelligent thing would have like an intelligent thing should be able to predict ahead on the world it should also be able to Output actions it should hopefully maybe be able to infer like why other things happened and things like this and so I guess as to whether we think that should you know be just like one homogeneous model um for which maybe you query it and maybe you know different aspects of that model would kind of um you know handle different aspects of the query or that we should think of those as separate components I'm not really sure as to whether it matters whether they're separate components or not because yeah I agree that you probably could just have like one massive model that does all of these things and I think at least from the the trend that I've been seeing um in kind of the world models literature and and also just like I guess the RL lit or maybe just we should call it the foundation model literature is you kind of don't want to have like a a separate model that does the prediction for actions and a separate model that does the prediction of observations like why not just have one massive model that's jointly trained to predict everything you might want to query and then depending on the different query you know it will just either predict an action or predict a video sequence or it can be condition on actions or condition on language so I think in this sense like this kind of model like you said is more like just one massive model but it kind of has like lots of different subtasks that it's able to do um and so maybe this is actually like the more effective way of training a model because then you kind of get generalization across these different subtasks as well yeah and the reason I'm asking the question is um it seemed I mean like you know for for an outside are coming in it looks like statistics is broken you know in the olden days we used to talk about the my fre lunch theorem used to say like you know you need to have specialized models for different situations and now the narrative is that we have generalist models we have Foundation models and and they are better than the specialized models in a strong sense and you know and I like to sort of push on this a little bit and see well when when does it break because we know that there are like these physics inspired models with inductive priors that you know know know about invariance of you know like molecules in drug Discovery and stuff like that and surely they would be better than a language model but no no no now they're training language models on mathematical conjecturing and like you know like um drug formula using tokens and so on so you know as an outsider you might just think well we can just use a big transformers model for everything I I think a lot of this does come from um well so I think the attention-based Transformer architecture is proven empirically to just be highly scalable highly effective at learning lots of different kinds of data distributions um but I think also part of it is just that we're just starting to enter this regime where we're just training these models on an insanely large amount of data and I think that a lot of times we need to sort of take a step back and really consider the amazing performances on different tasks and really think about you know how much information was actually leaked into uh this task in the training data because um right now we're really just training uh these huge models on I think I would say that we're largely training them on the test distribution in many cases um I do there I have seen like lots of examples of uh truly impressive behaviors from these models that that do seem like uh truly novel like zero shot General ization to unseen tasks uh like there was a recent example I saw on Twitter where someone uh had like a very low resource like rare language and they gave it a few gave they gave I think the cloud 3 Model A few examples and it was able to essentially perfectly reproduce uh new utterances in that language uh so that does seem very impressive um but it does seem at the same time you know a lot of the performances for example on elsat or like AP biology exams I imagine a lot of that is really a function of just uh literally giving the model uh the test domain in terms of information during the training step okay okay so there are like two schools of thought on this when we talk about world models you know people are talking about Sor and is it building a world model and and it certainly seems to be it seems to be doing I mean obviously it's not doing Navia Stokes it's not doing like fluid dynamics but but it seems to be doing something like that so like one one extreme view is that it it is just a hash table and you know it's it's kind of doing some diffused approximate retrieval or whatever another school of thought is that it's like a simulator and you know people talk about the simulator's view of large language models and you know like it's like it's modeling not only you know just just just the words and the language but it's also implicitly learned to model the world and the people and and all of us so that's the Spectrum I mean like Mark where where do you think these things are on that Spectrum yeah I think we like it would be great to be able to play around with it and kind of see what we can get out of it but I think I think if you can for example you know after each kind of you know so it's a language condition model so if after each kind of frame you could for example put in a different langu um language kind of conditioning and say like you know what happens here if you know the mug was pushed off the table instead of whatever else was originally happening in the video and so if you can basically do this kind of like counterfactual or like Interventional predictions where you kind of give some new action and then you're able to see like the alternative outcome of that new action I think if the model's able to do that then I would think that it does have a pretty good understanding of how the world works in the sense of you know I really think like if you can predict the outcome of any action given some sequence of observations I do think that's a pretty good proxy for being to say if you can do that you really do understand how the world Works um and so I think if the model can do that I I would be kind of inclined to say that it does have like a kind of world model in the sense of understanding the underlying world but then there might also be a chance that you know you know these models aren't like you said it's more just like a diffuse retrieval and and perhaps if you try and do like a very fine grain conditioning on a slightly different outcome um different like conditioning maybe it won't actually give you the correct kind of counterfactual prediction and so I think maybe we'd have to see how good these models are at generalizing to to slightly different inputs and things like that to really see if it understands things well or it is just like kind of generating some arbitrary video yeah I think it's a double whammy because our colloquial use of language and like you know use of models and intelligence is so static that like you know we we we um we think of of that as being intelligence but but we're still going like we we're now create we're creating knowledge right now we're creating models because we're exploring we're doing exactly what you said mchi we're like we're exploring the search space and we're building models and we're combining them together and you know presumably would diverge quite quickly from from from the language models but I mean what what's your take on on this idea that they are you know potentially World simulators yeah um so just regarding the the sort of lookup analogy for these large models I I think it's so my mental model is similar to that um although I think it's it's very close to um I think a really good write up of of the of this alternative take um which is more like there's an alternative take which is that it is kind of like a lookup table but the prompt itself is a key that maps not to a specific sort of response but to potentially like a function aast space of functions and fr had a really good um sort of blog post where he kind of goes more into the details of this viewpoint but I think that that really you know resonates with my intuition of how these things behave where it's not literally looking up like um a key value in a hash table it it seems more like it's these models have learned over tremendous amounts of data to compress that data they have to learn um I think more abstract functions that helped to explain that data and therefore they're learning functions so they're approximating some kind of function uh or a vast family of functions and I think the prompt really acts like as a key that essentially activates a particular function and so you can kind of think of you know in the classical world where one neural network equals one function like basically it's mapping from images to image net labels now like Foundation model in the foundation model regime it's like one Foundation model is essentially kind of like a giant database of lots and lots of lot different functions um that's basically activated selectively based on the input or the prompt um and I I do think that you know based on this I think it's definitely possible that with enough data from the world enough experiential data that these Foundation models can learn sort of a basis set of Dynamics and transitions that explain how the world Works um and essentially if it does learn these transitions um for example in like the massive amount of video data that sore is trained on um I would say that yeah I I would agree that they are essentially starting to approximate uh World models sure yeah so yeah these are two um separate papers so so the first one being dreamer led by like Dan and jar half so this is um you know example of work in the space of world models and so basically what dreamer involves doing is like a way of training a world model and then also showing that you can just generate synthetic data in this mod model and then optimize decision- making like purely using the synthetic data um so we talked a little bit earlier about like partially observable mdps so we want to like take kind of the sequence of observations um and then be able to predict like the next a distribution over the next observation given some action and so so we also talked about how you might want to like compress this into like a um more compressed representation of of the previous observation so basically what dreamer proposes to do and a lot of works on world modeling is to take your previous sequence of observations and then you map them to some compressed representation and then could predict ahead in this latent space um the next uh latent um latent State condition on the action and then yeah the really interesting about this is that now um you know we can in general predict what's going to happen to condition on different actions so now if you want to get like interesting Behavior out of something like dreamer you can then go ahead and generate a lot of synthetic data using dreamer or the dreamer World model and then use that to optimize behavior and so in Dreer basically the way it's done is by doing like on policy reinforcement learning in the world model so a lot of people call this like reinforcement learning in imagination so it's basically you know you're imaginating a bunch of synthetic data then using that to like use some standard reinforcement learning algorithm and then optimize um behavior in some sense um and then you could also do other things like Monti research which is like closer to like the works on on muso and things like this creativity is a little bit like a cloud and all the creativity only happens on the surface of the cloud so there's this interesting thing that like Creative Discovery depends on the history of all the things that are discovered before and typically like new discovery only happens at the end of the chain not back in in the middle tinkering exactly and and there's also this notion that creativity happens through knowledge so like knowled new knowledge doesn't come from The Ether it's kind of there's some creative component to it but it it's it's on the um the the trodden path of existing knowledge that we already have yeah that wasn't a very good question but so when when we talk about imagination through like you know like reinforcement learning policies and so on what we're saying is like you know I'm I'm imagining all of these like possible you know worlds and so on but I'm using the cognitive Primitives of all of the stuff that I already know yeah I think knowledge is definitely U compounding uh compounding um artifact uh that's basically like the culmination of everything all the experiences that we uh that we encounter like throughout our whole life and through also like Beyond you know going backwards Beyond like even our individual lives into like the cultural knowledge that's shared and uh what's really cool about language models is that they are essentially um a codification of cultural knowledge and so uh Jeff cloon has this concept of AI generating Ai and so he's got multiple pillars of essentially what it takes for uh you to have ai systems generate General AI systems and he recently added actually like as a fundamental piece of this in in his framework uh this idea of building on top of foundation models and so he says he calls it like standing on the shoulders of giant Foundation models um which is I think really just um sort of the ml equivalent of building on top of cultural knowledge there's there's a real shift recently towards talking about um synthetic data and as we were just saying like you know synthetic data doesn't come from the epha so we already know stuff about the world we we build simulators and we kind of generate new information but in the neighborhood of things that we already know and then we kind of like iterate and fine-tune on the generated data um what what what do you think about that process yeah no I think yeah maybe I'll bring it back to this like the plan to explore line of work so yeah um so so basically like the motivation of that kind of work is like kind of saying you know we might have some like previous data set or something and we've trained our world model on that data set but we really want to go out and like gather more data and then like improve the world model um um by gathering more data and so we can use things like intrinsic motivation to then give us like a reward signal within the world model so in the sense of something like prediction error which me mentioned earlier so now we can basically like train a policy in the world model that's now not trained for a specific task but it's trained to go out and gather information in the world so basically now you know you do this imagining in the world model to imagine aead but instead of imagining ahead how do I do a task well you're imagining ahead how do I get to states that I don't know what happens and therefore will learn more and that's basically like the motivation behind plan to explore um and then in our um paper Waker it's it's kind of like inspired by plan to explore as well as works on like Auto curricular and so basically what we're trying to say is you know plan to explore is good for for getting an agent to go out and gather data um within a single environment and you know and presumably once you've gathered enough data within a single environment then you can generate a bunch of synthetic data in that single environment and then do what we discussed with dreamer in terms of like optimizing a policy for that very specific environment um but what we're really interested in is saying you know let's not assume that we have like one specific environment beforehand let's assume that you know there's some space of you know broad range of scenarios like we want a very like General agent there might be a bunch of different environments and then within that those different environments we kind of want to be able to to handle absolutely any task and so in the Waker paper we're basically saying like you know how should we gather the data within um within this like broad space of possible environments and tasks such that we can train a very good World model and then once we have that world model that's kind of like capable across environments and tasks you know the assumption is that we can then use that to generate good synthetic data which we can then um use to optimize behavior and so maybe to talk a little bit about like how we formalize this problem um so you know we mentioned earlier this idea of like the simulation Lemma so we basically say that or an existing work that says like in a single environment we can bound the gap between the optimal policy that's trained in the world model so trained in the synthetic data to the to the truly optimal policy by the error in the world model and the distribution of States generated by that policy so it's kind of intuitive like the world model should have you know low error and then we will get a good policy out of it but then what we're trying to say is like now let's assume we don't know what the environment is beforehand and we also don't know what the task is beforehand so how do we get like a good World model that can handle like all of those situations when we later want to go ahead and optimize some task um and so the way that we do this is we basically yeah we then use this notion of min max regret to say that the policy should have like low maximum regret across this entire space of environments and then using the simulation Lim we can basically say now now the um the world model has to have low error across all environments under the distribution of States generated by the optimal policy for any future task um so we're going to say like yeah the world model has to be good for any environment and under you know in any area that the policy might go to that's relevant to the Future tasks and then what we kind of say in the paper is you know if we want a truly General agent we're not going to know what the distribution of tasks is beforehand so we don't know we don't know what the reward function is we don't have a set of reward functions um you know we're just going to kind of assume the agent has to do anything later down the line and this is kind of like related to this idea of like open-endedness that we've talked a lot about and so if we don't know what the task is going to be like later down the line um then the best assumption we can do is say that you know it could be any reward function later down the line um which is maybe not the best assumption because as we talked a bit earlier if you're just kind of you know we talked about a bit about intrinsic motivation and interestingness and if you kind of assume the task can be absolutely anything later down the line you're kind of assuming that you know the agent might want to do something completely ridiculous later like it if you do this in robotics that might mean the task is just to do like back flips later or something like that but you have no interest in doing that so it's it's not clear if that's really a good assumption about how we should think about what tasks might be interesting later but that's the Assumption we make so we assume the task can be absolutely anything later down the line um so so now we have to get a to the point where we have the world model which is good for any environment and under the distribution of States generated for any task or any optimal reward function um and to do this we basically like Leverage two different techniques so to generate this state um so to handle the aspect that we don't know what the task is later down the line we assumed that um we have an intrinsically motivated policy that's basically seeking out the maximum uncertainty in any single environment and so basically if if this um if this intrinsically motivated policy is seeking out the maximum certainty in every environment um it's kind of like estimating for us what the maximum uncertainty is in every environment because it's like actively finding uncertainty in every environment so now we have a policy that's finding like the maximum uncertainty in every environment and then if we want to um optimize this like Minimax Criterion across environments we kind of need the maximum uncertainty to be low across all environments so so we kind of have to have like um you know this policy isn't able to find like lots of big errors across all different environments um and so basically you know what we could think might might might what happened in practice is you know you could imagine there are a bunch of different environments some which are like low complexity and some of which are high complexity and if we just kind of naively sampled from those two different environments data you know our world model is going to very quickly get good at the low complexity environment and then it's going to leave lot more data from that high complexity environment to eventually get the errors low in the high complexity environment so to bring it back to the title of the paper which is weighted acquisition of knowledge across environments for robustness so the idea here is that we're basically going to change how we sample that distribution of data across environments to make sure that maximum uncertainty stays low across environments so what this ends up looking like is you know we're going to sample less data from the environment that has lower complexity and then we're going to actively sample more data from the environment that has higher complexity such that we we bring those errors down on the higher complexity environments and I guess this is a little bit different to existing works on curricular because normally in curricular like automatic curriculum learning you kind of assume that you have some reward function which is telling you how well the policy is doing in each environment and you use use that specific like metric of how well the policy is doing to determine um you know where the policy has more potential to learn but because we're making this assumption that you know we don't know what the reward function is we're we're trying to get a general agent that can kind of do any task any reward function um we don't assume that we know that reward function beforehand so we can't use reward as a metric of saying like I need more data from here or I need more data from here but then kind of the main argument of the paper is showing that you know if if we just think about this in terms of prediction era in the world model like we can actually use that as like an intrinsic motivation signal to say you know does the agent need to gather more data from this environment or from this environment without access to a reward function and so we could kind of think of um this work as kind of a more General approach to automatic curriculum learning in the sense of like we're not assuming that that you have a reward function beforehand we're kind of agnostic to what the task is um and because and to kind of distill that knowledge that's that's gathered without the reward function we use World model as a mechanism to like distill that knowledge because if you just like naively have an agent gathering information with a reward function um you know how do you how do you kind of put that knowledge into the agent and we kind of argue the best way of doing that is the world model um so that's kind of a summary of like the Waker paper and like what the ultimate algorithm ends up doing so I mean essentially you're doing a high entropy search so you're you're leaning into um areas of complexity and you're building a higher complexity model which goes against the grain of of the intuition of like oam's Razer that we should have simple models so you're you're almost deliberately saying no I I want I want to model the the complexity and and have more of that and then the other interesting thing is like from from a a curriculum learning point of view I think traditionally we did explicit curriculum learning and you know we might have some principles around having a monotonically increasing curriculum of complexity whereas here by leaning into um environments where we do worse on so we're selecting them based on prediction error we're actually implicitly getting a kind of monotonically increasing complexity which just happens to work really well yeah I I guess actually it actually almost ends up being in the opposite direction so so by leaning into the the the higher complexity environments more we're kind of saying let's prioritize the harder environments more to begin with so let's like gather more data in in the higher complexity environments um you know cuz I guess in intuitively if you kind of want to be good across all environments you kind of need more data from the higher complexity environments and we don't really explicitly think about an ordering of going first from easy to hard um I guess that maybe there is a good something to look into there because you know like a lot of these Works go from low complexity to high complexity because it's kind of easier to learn an initial policy that can kind of do something in the low complexity environment and then you build up the complexity gradually um but I think that that idea is most useful when you know what the task is so you could imagine if the task is like Locomotion if it's walking you kind of want to First loan a policy that's able to walk on flat ground and then maybe gradually build up the complexity like add and bumps and then eventually it can walk on like very complicated terrain so it kind of makes sense to go from low to high complexity um but in this work we're focusing on Purely intrinsic motivation meaning that the policy is not trying to learn a specific task it's trying to just seek out um uncertainty and like reduce uncertainty and so we don't really have the the notion of you know you first need to be able to learn how to do something on an easy an easy environment and then towards harder environments because there is no specific task that we're trying to learn and so I think for this reason you know we wouldn't didn't really focus on this notion of moving from easier to hard environments that actually you know we're consistently samping more data from the hard environments and I guess I think this relates or I think this is something that you brought up when we when we worked on this is like you know I think we can really relate this idea to like a lot of different contexts including things like like language models for example um so you know you could imagine if I'm training in LM I don't really necessarily have this you know not really a reward function in some sense you're just trying to do like unsupervised prediction um and so you know we could for example take the prediction era of like a language model in a in a bunch of different domains and say you know the language model is not very good at predicting a language about some certain task or something like that and and you know we could say you know and intuitively the same thing kind of holds if it's not very good at predicting you know um what the next token is in French like we should presumably gather more data in French um and so that kind of gives us a way of like actively Gathering the appropriate data um and so yeah I think this idea of like Gathering more datab based uncertainty obviously is a very general idea like the idea of like Active Learning um but we kind of like specialize that into thinking about how do we think about this in terms of the reinforcement learning setting it might be um interesting to talk about as well like sort of because we looked at some of the metrics as well right the environment complexity metrics we don't have the external notion of difficulty but we we also did look at sort of the emergent uh curriculum yeah yeah yeah gotcha yeah so I guess um so it kind of dependent on the environment so in some environments you just kind of got this like very straightforward behavior of like you know consistently gather more data in the more complex environment um but because we're we're actively trying to gather data um of the the environments for which the uncertainty is the highest kind of this curriculum could change o over over the course of training so so what happened in some of the other environments for example is that initially all the environments are just like high un certainty like there's like all environments are kind of misunderstood therefore like sample all environments like equally more or less to just get a rough understanding uh and then you know as as the model would improve on the simplest environments then we would see like more and more emphasis towards sampling the highest complexity environments so I guess in that sense we would get something to more like kind of what you said in terms of like a standard curriculum but but a bit different in the sense of like initially everything is uncertain so we're just going to sample everything uniformly um but then we kind of get a better understanding of which of the environments you know the uncertainty remains high on these higher complexity ones and those are the ones we need to like go and gather more data yeah I mean I I can see this both ways I mean certainly from a like a basian optimization point of view that there's something to be said for um you know this is where I'm uncertain go and gather more data where where I have highest uncertainty and as you say like traditionally in curriculum learning we are told that we need to have monotonic increasing complexity but as you just said that's when we have a particular task in mind now neuron networks they're a little bit like a block of clay aren't they so you know it's it starts off with abject complexity and then we do stand you know we do um stochastic gradient descent and we chip away at the clay and we kind of build we sculp a statue that that that we want to build and I'm just trying to get an intuition here so like with with this maximum entropy um search you know like high entropy search what we're doing is is we're saying okay well here are some complex models and these models must contain motifs that tell us a lot of information it's a little bit like the ELO algorithm in chess you know you actually get Information Gain when something surprising happened so here's a big um block of complexity and I'm going to try and infer what the motifs are in that complexity that that explain the information that I'm missing I think that um a lot of this ultimately traces back to sort of there's like this like fundamental pattern towards uh I think that like ties a lot of these ideas around active um active experiment design or like active sampling which is and all all these autoc curricular methods which is you essentially want to devise uh what you know nowadays we call a self-supervised objective or self self-supervised training algorithm um where essentially you have the system essentially use signals it produces itself um during the training or evaluation process in order to drive itself forward in terms of deciding what future data to train on and so you know we sometimes call these kinds of systems Auto curricula as well because it's automatically generating this curriculum of tasks to train on and I think the sort of like the fundamental connecting um uh pattern here is just that this the signal that we use to drive the training it's always going to be based on something like uh an uncertainty signal or um going back to the open-endedness literature something like a classic notion of interestingness and I think there's just a lot of different possible choices for this metric and so one for example we talked a lot about Mini Max grab so regret could be one of these driving signals because it measures the existence of a performance Gap and therefore probably an information Gap as well in terms of learning to master those tasks with high regret um but also uncertainty is also another one it ties back to novelty because novel environments you will be more uncertain within and so there's fundamentally lots of different sort of branches of these autoc curricula that you could use depending on this search objective that you use to drive this exploration process can we contrast this to you know like large language models they are self-supervised learning so you know we we do this self-supervised objective you know which is like you know typically predicting the next word and it's a similar thing with um self-supervised um image um learning now the difference is with that is you're talking about a principled way of you know seeking specific information you know with um let's say high high entropy and that would lead to an imp curricular whereas with language modeling language modeling there is no implicit curricular but I might argue that there kind of is because the way the model does this continual learning um it might regularize itself so if you give it sort of surprising and weird information the language model might just kind of brush it off and if you reinforce things that it already knows then it's almost like a a stream of channels you know it'll say okay you know go go and go and and pay attention to that so it's almost like it's implicit yeah and I would say that in some ways it's almost explicit in terms of how we design these systems um a lot of times like if you look at uh for example open ai's job listings they're actually hiring specifically for experts in different domains to essentially create the next batch of supervised data to train or instruction tune their models on uh for example they hire biologists or they hire people with legal expertise to generate this data and um you can think of this essentially as a human steered or human driven version of this active sampling process right because it's essentially they know that the model uh tends to get high perplexity or they don't it doesn't perform as well on this domain of tasks it doesn't get as high of an LSAT score as it could and so you can essentially you know it's it's beyond an algorithm at this point right it's kind of the super algorithm where you have the system designers now also being part of the data collection process and in a way um supervised learning is really just sort of one point in a continual learning process where you know classically we just looked at one step of this which is here's a batch of data train on that but really um building machine Learning Systems especially nowadays everything's in production these are all live systems you have have to keep it up to date you have to keep it continually generalizing to new knowledge um like Chachi PT or Claude or Gemini and so really it's sort of this pattern over and over again in sequence where you collect a batch of data train your model on that collect the next batch of data you know continue training your model on that um and really you want to be selective about what the next batch of data is because obviously if you just retrain it on the previous batch of data um it's going to overfit to that data uh Beyond a few epochs or uh it's not going to you know get as much novel information from it just because it's already trained on it so you do want to selectively actively collect the data and so I think we kind of almost explicitly already do this at a systems level um and I think the next Frontier is really just having systems that self-improve in this way where they can start to guide more of their own active data collection I love this way of thinking about it you know like gbt for is a mtic intelligence it's not just like you know a bunch of weights on on a on a on a server somewhere and so you could argue you know there this concept called graduate student descent which is what happens in in Academia or even as you just articulated with open AI it's a little bit like an epic Mechanical Turk right where you know um they are monitoring the logs they know when things go go badly and then they lean into it in the same way you are they they go in higher experts and they kind of like add more and more data in all of the holes and eventually there are no more pockets of like abject failure it just it just appears to work really well for everyone and people start to say that it's you know generally intelligent so yeah so there's there interesting systems if you of of of intelligence yeah it kind of starts to mimic just the scientific process in a way uh where we're sort of we we're putting a lot of Hope in the model to basically be able to distill uh information from sort of the net new batch of data that we collect um you know that we know the model currently doesn't explain well and we we we put a lot of faith in gradient descent in order to basically be able to come up with updates to the weights that better explain that data so we're kind of we're kind of already treating this system as almost like an automated um scientist or an automated version of this like continual process of creating theories and explanations about the world um but of course you know um humans are still much better at language models at doing this uh or large models at doing this so I do think there clearly seems like a huge gap in terms of what we still of work that needs to be done in order to build systems that can actually build much more robust theories uh based on like net new data and even seeking that out as humans do interesting and and certainly you know in this broader mtic intelligence we are still the sources of a gency but um we we were just sort of talking a minute ago about there being two types of AI you know that there's there's an AI where we are the generating sources of agency but there might potentially be another AI in the future where that that is the generating source of agency yeah I I so I think that um this kind of ties into my my the framework I personally use to think about open-ended systems as well uh where I think that you know at a high level you can you can study AI sort of in silicone you can study it in systems that you control that you design and that you try to like have the AI model self-improve within and so you can try to build uh systems that self-improve within silico and that's going to lead to potentially some issues around like the grounding problem where essentially it starts to the auto the autoc curricular exploratory process starts to Veer into Parts pockets of the design space that are not relevant to tasks you care about um and so this kind of the danger of like generating open-ended systems in silico and I think it's very similar to potential dangers of generating AGI in silico um and I think the alternative is really just what are existing intelligent systems and how do we actually amplify the efficiency the efficacy of those systems the intelligence within those systems and so you can kind of think of like sort of the entire Enterprise of AI research as do we want to generate like AI or intelligence from scratch or do we want to build tools uh you know motivated or inspired by human intelligence and other intelligence systems and use that to further amplify existing intelligence like human creativity human intelligence could you argue because if intelligence is a Divergent search process you might be tempted to think that well if we had loads of tools to help us share the models and help other people discover the models that I've created and that will help us generally be more intelligent but could you make the counter argument that I'm actually sequestering agency or stealing agency from other people because rather than thinking for themselves and discovering novel models they're just going to use my model yeah I mean I think that in the best case scenario you're Building Systems that essentially you know not not you know to to think about how you know as existing systems nowadays can build on the shoulders of foundation models you really want the to build models where even humans can stand on their shoulders where the humans can basically leverage the existing expertise or automative capabilities of those models to then like move further beyond what they're naturally capable of doing and really that pushes the frontier of the knowledge that we can create as a civilization and so you're already starting to see this where there's some recent studies that show for example like Junior software Engineers that use systems like um chaty BT to help them with coding at work they actually now are starting to match the performance of more senior Engineers uh because it sort of levels the playing field but that also translates into just like uh net more productivity per software engineer and so um I think that it's more just unlocking sort of uh existing bottleneck and how productive each individual can be and really just means that each individual can create a lot more value can discover a lot more knowledge um than before okay but I mean do do you think that it creates a tendency towards boilerplate though so we're more we're more efficient at doing things that exist but you know like on on the frontier we might have a Slowdown there's definitely the danger that it can lock you in to certain patterns right so basically if chpt always returns a certain boiler plate that might have an anti- pattern in it um if that stays around it could self- amplify and then future generations of programmers might just adopt that by default because it's what's already generated by autocomplete so I think that that's also another really interesting realm of questions which is basically how do you um how do you avoid these kinds of uh these local Optima when you start to train a model on its own outputs and I think again like sort of the solution will start to look like some form of novelty search or exploration makes sense okay um what do you guys think about like um you know acade Academia versus industry and um some say there's a bit of a brain drain from Academia totally yeah I think there's like a very very clear trade-off between the two in the S they both have like fantastic things going for them and I guess the trade-off being you know academic freedom and Academia and be able to like individually pursue ideas like purely for curiosity sake and um you know that's something I've really loved about Academia but I guess you know I guess the general Trend and and machine learning research at the moment is kind of towards like larger scale projects especially you know a lot of the properties that we might want to see kind of only emerge when you expend a lot of compute and therefore you know a lot of interesting research can kind of maybe not only be done on an industry but it's a lot easier to do some kinds of research in industry and so I think this kind of leads this trade-off of do you want freedom or do you want to be on these like larger projects that are potentially more impactful and so yeah I've really struggled with that trade-off I think they they both have big pros and cons I don't know what you think mie yeah I I think that um industry is I I think like at a very like first word rough approximation would be to say that industry focuses much more on um exploitation and Academia is where you know in principle you should get a lot more exploration um but I I do think that currently uh both systems are kind of like entwined in the same sort of reward function at a high level where essentially um you know if if if you're if you care a lot about citations and a short-term greedy algorithm for maximizing C citations would be to focus your research efforts on uh sort of whatever topic is uh trendy or hyped at the current time and so like I think you see tons of people obviously working on language models partly because it really is a fascinating subject and it really is like the most powerful form of deep learning we have so I understand why everyone's working on it but I also think that um a lot of it is kind of you you do get this sort of Rich gets richer effect around different topics that people tend to gravitate towards and you lose a lot of the exploration that you should otherwise have um and it's partly because you know like both industry and Academia are at some level optimizing for a similar um sort of reputational status or citation count sort of metric um and so I think that's an issue but I also think that in some ways uh industry you could say has a additional benefit where I do think that from like a short-term point of view industry is better poised to make certain um higher impact research not just because of the resources available to industry but also partly because um sort of Industry uh you know rides or dies based on whether the actual research artifact you produce uh is useful and so I think that's like a very powerful reward function that is not necessarily true for Academia um and then sort of on the to take the counter position I think Academia obviously you know you have a lot more freedom to just explore ideas that don't need to be on that critical path for Value creation immediately and so gives you a lot more scope to potentially find like the next big thing and so I think really it's about like if you want to if you want to take the bet that you can you know play a part in disc the next big thing then and that's that's suited to your taste for research then Academia makes more sense uh but if you know um you want to you want to maximize the probability you'll have a higher impact in sort of like a near horizon line of work then industry is definitely I think a better bet Rich Sutton you know he he had this bitter lesson essay and he made the argument that it's just all computation and there are no shortcuts and you can even think of you know maybe we're not very intelligent um Evolution has just been running for a very very long time and we are the result of that so in in a sense do you think that we could make strides in intelligence you know just through Ingenuity or are we always going to need loads of computer power this definitely like makes me think of like the recent Trend that we've been seeing even in like kind of the reinforcement learning literature lately which is like these kind of large scale like mostly industry projects that are kind of they're even ditching the idea of doing like sequential decision Mak so you know you have all these Al that are like you know optimal planning and so forth but we're kind of seeing a trend towards you know even ditching that complexity of algorithm and just going straight to just copy what the human did and so kind of reducing the problem to you know essentially no real algorithmic um Innovation and more just like can you gather enough expert data and I think yeah I guess the reason why that trend is occurring is is I guess like you said there kind of been you know the B lesson kind of said that you know just being able to SC with more data and more compute as kind of the most important thing and a lot of the more complex algorithms especially around like reinforcement learning are actually like quite challenging to scale up especially like online reinforcement learning if you want to go out and like actually have an agent like actively collecting data in a bunch of different environments and updating itself online like that's so much like engineering infrastructure to set up and so I think there's this this trend towards just like the simplest algorithm possible which is like not even reinforcement learning not even planning just copy an expert but I think that that's like you kind of said um earlier with like this kind of like short-term exploitation I think this is you know it it kind of makes sense to exploit this now and push it as far as possible because you know it's very easy to just train a large Transformer and then gather as much data as possible and I think in areas like robotics we haven't really seen like how far can that go like can you actually get a generally useful robotics platform just by gathering more expert demonstrations and training a larger and larger Transformer and so I think it does kind of make sense that why like a lot of Industry projects are pursuing that because we don't really know you know will will that actually hit a bottleneck or or if you just gather enough data will that will that kind of be sufficient and I guess like you know you could argue that I think it's probably true that there must be a better algorithm out there that that can in principle do this in a more efficient way but I guess if it's just easier to just gather more data and just do imitation learning I can see that there's at least a business case for trying that um so I guess I'm on the the opinion of like you know there must be a more efficient way of getting to like a more intelligent system but it's not necessarily clear that just scaling like raw supervised learning or unsupervised learning like won't get you there and so it it does make sense to pursue that first but kind of what I hope and expect to see is that eventually pure imitation learning or pure unsupervised learning will kind of run out of steam and everything will Plateau and I think at that point you know then these like more complicated algorithms about Gathering more data reinforcement learning planning Etc will really come into their own and so I guess this again relates back to like the Academia industry trade-off like you know a lot of the products in Industry are just going to kind of be exploiting gathering data right now whereas maybe there's a lot of scope to do these kind of more expor exploratory projects where maybe that will get you to like the next Frontier a few years down the line um I don't know what you think about this yeah I definitely think that um yeah just like treating everything as just supervised learning it does tend to work because we have large data sets but um I think again like the challenge is just at some point we will run out of tokens we'll run out of data to train on um and so that's why these self-improving more self-exploratory systems will be more and more I think Paramount to like driving performance even further so if we want to sort of break Beyond sort of the token limit of like the data that's available now we actually need these systems to generate their own tokens their own synthetic data um and that's that's where like the self-play autoc curricular exploration types of algorithms will start to um become more and more prominent and obviously you need an environment in which to do that exploration and that's where the world model um line of research is going to be very powerful just because uh that allows you to really sort of milk all of the value within uh the existing previous data you have seen by creating these Role Models where you might be able to do like counterfactual trajectories and really learn much more um amplify the existing data you had yeah I mean I think one of one of the key things for me um is modeling Dynamics so um it's quite interesting actually with the human knowledge thing so even looking at the Innovations from from Deep Mind you know early versions of of alphao were bootstrapped with human knowledge and then there was the alpha zero so it was actually doing what we're were talking about it was actually discovering knowledge on its own and um in principle that's a great idea but of course like any restricted domain it's tractable but in in the real world it isn't and I'm not sure whether it makes sense to use the the computation and met you know information metaphor for the real world and humans and so on but but the basic idea is that we are all real agents the universe is a massive computer we're discovering all of this knowledge and then we're bootstrapping that into um a machine learning algorithm and then the question is well if you kind of just capture the thing now without the Dynamics that produced it um will the system be robust and could you still um you know kind of carry on as we were in the real world if if that makes sense so um but yeah the interesting thing with the work you've done is is that you are modeling agential systems and you are modeling Dynamics but could that be used for you know much more complex tasks like the real world like simulating much more complex systems in the real world exactly yeah I think that if you if you so I think that just purely imitation learning alone is not really going to get you there um but I think that if you can if you can uh imitate so one is sort of finding the set of tasks I think that uh if you find the set of tasks or reward functions that could be relevant then you can start to simulate things that are otherwise really hard to capture by just purely imitating historical trajectories so for example strategic adaptation type of behaviors are really hard because those are sort of an open-ended space of behaviors where if you basically have like a stock market for example that's a really good example where if you have a stock market that's a very open-ended system and like different Traders will have different strategies that are best responses to each other and then over time the set of strategies evolves over time in an open-ended way um you know trading strategies that worked 10 years ago probably won't work very well today because people have sort of um they they've sort of figured out uh those strategies and so they won't be very competitive and so um I don't see an IM an imitation learning system being able to sort of um generalize to that level of complexity just because by definition it's imitating previous uh trajectories and therefore strategies so I think you need some of like a um a more uh more interactive trial and error learning that allows for strategic adaptation and that requires some notion of a payoff or a reward and so you kind of need to have this this idea of um you you can't just purely I think learn uh a model of something like the stock market just based on previous data you really need to have more inductive biases around uh sort of you know what creates a payoff or what the actual reward function is for each of the Traders uh but that might be something that you could um you could learn over time but I but maybe not in the yeah so sry this is kind of like this not very coherent but I feel like uh you might need something that looks more like learning over a space of programs that starts to Encompass different kinds of uh tasks and then you can basically simulate those tasks to completion with agents that can essentially uh try to self-improve against other agents the stock market I think is a wonderful metaphor for what we're talking about and for for two reasons first of all from the grounding reason because you know like the the the mtic world is very ungrounded and that's why we develop as humans lots of weird shared delusions about things because it's actually like you know it can go in it can go in almost any direction and also the concept of alpha I think is really important because a trading strategy works really well today and then when other people learn about it it no longer provides an advantage because everyone else knows about it and I feel it's the same with language models so you know like GPT 4 Pros was really novel and cool it it was great to you know have like a TED Talk speech when it came out and now it doesn't seem cool anymore because everyone's using it on LinkedIn so it's almost like that that we need to have this in like continuous creative evolving process producing new sources of Alpha and the Paradox is that if everyone has access to the same model it can't be a source of alpha by definition yeah I guess on that like topic because we kind of talked about like synthetic data earlier and you kind of said like you know one one mechanism towards getting like a kind of self-improving system that is able to kind of you know continue to improve is to kind of like fil so the synthetic data for example so we might kind of you know have the the new system and then we generate some more data and then we kind of have some like filtering mechanism to say that you know in the current stock market this is this is good data or what you know whatever system we're thinking about and and then we can kind of like use that to enable the model to improve um you know and adapt to the new system but something I've always like like thought about is like or I guess one is is it really trivial to be able to like filter that you know new synthetic data and then two it feels like if you're just relying on like fil filtering existing synthetic data like is isn't that inevita going to kind of plateau and so I guess event you know we talked about how you kind of said that you do actually actively not need to go out and get real more real data but I guess I'm kind of asking you do you think this idea of just like filtering synthetic data from a model is kind of sufficient to always be able to adapt and improve or is it always going to be a mixture of like more real data plus synthetic data filtering I think it's the letter just because um at some point you would expect that uh the synthetic data you do generate it'll start to sort of uh saturate like what's already in the model um just cuz the model is trained on a finite amount of information so at some point you're just going to start to see more and more um especially like the more likely trajectories or sequences of samples you'll start to see that uh more and more and so you're not really going to be very sample efficient in terms of searching for the synthetic data so can can you tell us about the results of the paper totally Yeah so basically we evaluate this um this algorithm on a bunch of like synthetic simulated domains kind of like robotics related tasks um and kind of yeah environments where there's like varying levels of complexity so you know you might have a robot pushing around a variable number of like objects or maybe you have different terrain that the robot might want to um um learn to kind of you know do Locomotion over and things like this um and so kind of you know the main comparison we make is like how well does Waker work relative to like naive domain randomization so how well does it work if you just like uniformly sample the space of environments versus if you do actively seek out the environments that have this like higher uncertainty um and so basically what we show is that you know if we do the Waker approach we still do like very well on average but we consistently do better in terms of robustness and so robustness by robustness I mean here that that we do better in terms of the worst environments um that the agent is evaluated under and so this kind of means you know if the agent is able to do well in the worst environments that it that it is evaluated under that kind of shows that it's able to do well across all environments because its worst performance is still good um so you kind of this this shows that we we achieve this like robustness property um which we talked about in terms of like Mini Max regret um but we EV we don't evaluate it in terms of like the true notion of Minimax regret because as as we talked about earlier actually evaluating regret exactly is difficult because that that that requires knowing the exact true Optimal Performance um which isn't something we can really know so instead we we just show that you know the agent performs well across all environments more so than if you just like naively sampled the environments uniformly and in terms of decomposing the performance across the spectrum of possible environments so like you know the ideal situation is that we have a very simple model which just generalizes so we happen to have found the golden Motif you know there's a spectrum of correlations almost all of them are Furious but we've just you know just by through some sheer magic we found the best Motif to work in all situations probably that's not quite true probably there are some good generalizing motifs and the model has also kind of like memorize the long tail and that there's some degree of like you know it works really well on on the test set but might not out of domain distribution do you have any like way of reasoning about what that is um so yeah I agree I guess there's like not necessarily it's not necessarily the case that by like focusing more on these like longtail examples that's necessarily the best way of of training the best model because like you said like maybe it happens to be the case that if the model is trained on some certain subset of the task like that will actually generalize better but but I think in practice that's not something we can really um really know how to you know like optimally select the the best kind of set of tasks that will generalize well and so we do focus more on on like you know these these kind of longtail tasks like the ones that we might see rarly and therefore have high uncertainty about um in terms of like the the out of distribution generalization so so we do also do some experiments like looking at how well does the model generalize out of distribution um and basically what we show is that if we train the model in this way and then we give it some more environments that hasn't seen at test time um if the environments are more complex then then we've seen sorry hasn't seen it training time basically like this model then generalizes better to outof distribution environments that are like more complex which is kind of what you'd expect cuz we've kind of bias something towards more complexity we're able to generalize better to out of distribution environments that have higher complexity and then guess the question is like do we care about out of distribution environments that have higher complexity like what about the out of distribution environments that have lower complexity um and I would argue that you know basically the lower en out dist distribution environments that have lower complexity like we would already expect that the model is able to do very well at so so there's not really much of a difference there because you know almost any reasonably trained model can handle the very simplest environment so what we really care about is can we generalize out of distribution to like higher complexity environments and so by biasing the suling towards the higher complexity environments we do show that we're able to generalize further out of distribution to even higher complexity environments okay but is there any way of knowing whether it's kind of like memorizing the high complexity instances or whether it's still learning abstract motifs and generalizing between them yeah that's a great question I think that's a really interesting question generally for ML as a field right now which is better evaluation benchmarks for uh generalization within different kinds of models um and like we we alluded to earlier there's kind of this uh issue of data leakage between training and test set which is um which is definitely an issue that is currently happening with large language models um it doesn't take away from the impressiveness of these models CU clearly there is a strong generalization aspect to their behavior but I do think that in terms of measuring performance on specific benchmarks um we really need to solve this problem how do we have these clean data sets uh that allow us to to truly test on inputs that the model hasn't seen at training um I think in the case of uh reinforcement learning um that's a bit more difficult just because usually we focus on a particular task domain and so there's always going to be some shared similarities within task but obviously uh we didn't do this in this paper but we could try things where we have more um more controlled uh settings where we you know change one aspect of the environment and really uh see if it's learning specific causal relationships between um things that have to be accomplished in that task uh but we didn't do that um that actually think would be a really interesting idea for uh a new evaluation environment for RL yeah I mean the benchmarks thing is just a huge challenge in in machine learning um in general but just just to kind of round off off the interview I mean M you were talking about you're doing some work with um Ed grinstead and he's an amazing guy I'm get getting Ed back on and um you said that um been looking into this kind of the interface between humans and and machine learning can you tell me about that yeah so just to not say too much about it because um it's related to current work that's happening at Deep Mind um is just that you know I think from personally from a high level point of view I'm very interested you know talking about this divide sort of this fork in the road in terms of what's the path to open studying open-endedness studying it in silico or studying it in situ in the setting of an actual open-ended system like a user um app interaction or you know the interaction between a user and a piece of software on the web uh or potentially with many other users there are such Rich um existing systems online that are already open-ended because they amplify or connect the creativity and knowledge of humans um to create more knowledge and more creative artifacts and so I think what's really uh interesting in my mind now is sort of studying uh systems or algorithms that allow us to better steer the creativity of humans uh as they are uh mediated by software um and basically allow us to essentially amplify existing intelligent or creative systems that are open-ended so amplify existing open open-endedness rather than try to build it from scratch amazing guys it's been an honor to have you on MLS thank you so much thanks so much yeah great cool yeah we're done