at for example, vanilla GPT-3 has a really low executability, it does have a high correctness.


Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (+Author)

However, if you look at the translated version, which is after their improvements, you can


see the executability has risen dramatically, while the correctness is a bit lower, like


you get a bit lower in correctness because of the whole translation procedure and so


on you're mocking with the outputs, humans may not like it as much. This is all stuff


we're going to touch on in the interview. Just interestingly highlighting that codecs


like the codecs model seems to be scoring quite well on these tasks. So also the translated


codecs is much smaller. However, it scores high, really high. So parameter for parameter,


the codecs model is actually pretty, pretty good at this, which was a surprise to me.


So I think this is an exciting paper. It it except as I said, for a fine tuning baseline,


it turns out to work with completely without any training. It's just evaluation, so to


say. And I liked it. And I think this does have applications like getting the knowledge


out of these large language models is something we should, you know, be getting better at


doing. Otherwise, I don't think we make full use of them. Alright, so now I want to jump


into the interview with Wenlong. I hope you enjoy that as well. Tell me how you like these


these videos with the interviews without the interviews, anything you want in the comments.


I'll see you. Bye bye. Welcome, everyone. Today with me here is Wenlong Huang, who is


the first author of the paper about language models as zero shot planners and very, very


happy to have you here. Welcome, Wenlong. Thank you, Yaning. Yeah, super, super happy


to be here. And this is I've already told you, but this paper is different. And I like


different papers. And it's it's different in a way that maybe wasn't expected. Every,


it seems like every day, we find a new applications for these large language models and yet another


thing that they can do here. And when I when I saw this, I was reminded of a friend of


mine who had like similar ideas, but it never really materialized. I tried some of this


stuff as well, combining large language models with planning with telling me what to do in


the real world. I even made a video where GPT three told me a recipe and then I cooked


the rest like me and my friend, we cooked the recipe and so on. But it seemed like always


a bit a bit out of place a bit a bit off just to give you detailed instructions. And when


I saw a paper that was really trying to make this work in a real environment. I was I was


very happy to see that. And yeah, that's that is that is this paper. And also to be said,


you have a stellar board of of co collaborators right here. How did this come about? Like,


how did you even get to the idea? Hey, I could use these language models to do planning.


Was it like, did it immediately come to you? Did it sort of build up from some basic idea


or what was the process?


So yeah, thanks for the briefing. So I think that's actually came out to be really surprising


to us as well. So first, we were just having when we just playing around with the largest


language models on the many of the web interface, we found that like, actually, there's something


there like you said, if you ask it for a recipe, or we actually originally study, like whether


you can offer the staffs for making coffee, etc. So we found that like, when the most


get large enough, there's actually something there. And this is the sign of life, I think,


for us to kind of go on and investigate how we can make that actually useful for agents.


So we kind of just started from there. And actually, it came out to be pretty surprising.


Originally, we felt like maybe we need some training data set to maybe like train something


a translator or something to actually make it useful. But it turns out like, but we really


trying to constrain ourselves in the meantime, because we don't want it to be tailored to


a specific environment. So we would just want to see, like just the language model itself,


like how well can do how far it can go. So this is what got us in the end. We just like


explored for like two months and then found like, you can actually do this without any


any training. And it Yeah, it's actually truly surprising. And actually, actually a really


fun process for me as well.


It sounds like fun.


Yeah, just trying to see whether you can offer something like really realistic and really


fun. Yeah.


Yeah. So you came you came across this, this environment right here, this virtual home


environment? Was this always the plan? Or why did you choose like, there are a million


environments open AI gym and, and they're they're moving, you know, these Mujoko kind


of robot simulations? Why was this one particularly useful? Did you immediately think of this


one? Or how did this came about?


Thanks. Yeah. So actually, I wasn't doing too much research in this embodied agents


area, especially for this, like really high level tasks. And then I actually went went


to the like, Google Scholar and then search for appropriate environments for this. And


we found this virtual home environment. And we really liked it because it actually can


model any, any tasks, if you can express them in terms of this, like, textual language plan,


like a like just just like textual plan. So and actually, there are many, many other environments


as well. But some of them are limited by I think a lot of people also use Alfred environment.


That's a really good environment too. And I think it's a bit more structured there. But


the tasks are often come from like a template. So it's usually like pick something, pull


something. But actually, there are a lot of challenges there. I think it's a different


set of challenges. And we found like, what the virtual home tackles is exactly what we


look for, because it can model like any task expressed in freeform language, especially


those like really challenging tasks like people do actually every day, like make breakfast,


make tea, make coffee, and then a particular cares about the common sense constraints in


them. So specifically, this environment has a set of like, preconditions and post conditions


for each action. So for example, if you want to grab a glass of milk from from the fridge,


you can just like say, go to the fridge and grab glass of milk, because you first got


to open the fridge first. And then, like, preferably you want to close the fridge afterwards.


So it's really this like, this constraints, I think, are really useful and really interesting


to study whether language models can handle this. And you've, you've investigated several


different language models. And just to be clear, this environment, it has this kind


of syntax, it has very defined things you can do. And somewhere, I think you say it's


about 50,000 actions that are ultimately possible. It's kind of a combination of a bunch of verbs,


which are grab, open, go to and lift or things like this, and a bunch of objects like kitchen,


fridge, and so on. So any plan would consist of a sequence of verb object, verb object,


like here, walk to kitchen, open fridge, grab milk. So the the the any planner in this environment


would have to output this syntax directly. Now, you had a plan of not training anything,


right? You didn't want to train anything, you simply wanted to investigate what knowledge


is already there in the language models. And you came up with kind of a way to translate


that you want to maybe elaborate how do you how do you query these language models? And


how do you make them actually conform to the to the the the syntax here?


Of course, yeah. So the the way that virtual home expresses this actions are via like this


specific format where you put a square bracket, like for the action, atomic action, like grab


food open, and then you put I think it's a parenthesis, or Yeah, something for for the


arguments. And but the problem is like, we can't just like, expect language models to


handle this. Because I mean, even if we put an example in front, maybe they can do it.


But it's definitely not the way that usually humans produce language. So and after all,


this language models are trained on human text. So we decide like, maybe it's not the


the right way to query this models. Maybe we just want to have you ever tried? Have


you tried letting them output directly the syntax? Or was it just like, Yeah, it's not


gonna work anyway.


I tried briefly, but it's definitely not thoroughly investigated. And yeah, like it, like intuition


wise, I think it's definitely sure to to to use like natural language, but we did adopt


for the the most basic approach that we can we can think of, which is like, just define


a straight up like, template for each atomic action. And actually, because this atomic


actions are simple enough, like just walk, grab and those things. So this atomic action,


I mean, the template, the templates we actually came up with our, I think, actually, just


natural way like people, people say things. So like, turn off something, turn off something,


and then add some some words in between like in, on, on top of, etc.


And then and then you just query these models. And you have multiple ways of evaluating this,


right? You care about two things, you care about correctness, and you care about executability.


And in at least, so you also make use of humans, like, how did you how did you design? Like,


what was your thinking behind designing the evaluation?


Yeah, so actually, it came up to be really challenging to evaluate these things. Like


I said, so like, this this task are because they're expressing free form language. So


that means they're really open ended. So it might be deterministic, whether like, if you


want to grab a glass of milk, you just want to look in the end, whether you have a glass


of milk. But if you really think about it, if we don't want to constrain anything in


the task that we want to want to do, like making breakfast, like what is the correct


way to make breakfast, everyone has different preferences. So it's hard for us, actually,


I think it's still a challenge. In this sort of task is like really determined the correctness.


I'm sorry, it's the success rate for each task. So you can't really tell if a task is


really successful, depending on how open ended it is. So we decided that, okay, so if it's


hard to computationally produce a metric for success rate, like, but as humans, we can


definitely tell if it's making something semantically meaningful. So this use part of like human


evaluations to do this, but we don't want to entirely rely on humans. Because as you


can tell, for the for the text that like, for the action plan that real language models


generate, they're so realistic that like, they can even fool many humans that like,


are too realistic. So you can just entirely rely on humans to say if it's successful.


So we also use this metric executability, which is also used in in past papers from


in like that uses virtual home. So we just use this metric as well to basically determine


whether the plan satisfy the common sense constraints in this environment, namely just


like, whether you like open, make sure to open the fridge before grabbing something


from it. Yeah, like this. It's interesting, because when the humans rate it, the humans


would also skip a bunch of steps, right? If you said if you tell a human, go to the fridge


and grab a glass of milk, the human will go like, Oh, yeah, of course. All right, which


is, which is one of my maybe this is jumping ahead a little bit. But one of the questions


I had most when I read this was just, there is a level of specificity that is required


right here, which is kind of ambiguous, right? You have a high level description, which is


like make breakfast, right? And then you have a bunch of steps, which you need to follow.


And sure, these steps correspond to actions in the environment. So they're kind of given


by that. But the language model doesn't know that, right? The language model just knows


I need to produce a plan. So how is the language model? You know, why do why do we expect the


language model to figure out that it needs to like that it can't that it needs to say,


open the fridge before you get a glass. But it for example, it doesn't need to say, put


one foot in front of the other foot, you know, in order to walk. So, you know, did you have


any insights or concerns with like, there seems to be like a very specific level of


specificity of these plans?


Yeah, so that's a really good question. Actually, this granularity actually comes from the data


set, or the virtual environment itself, because the way because we essentially follow the


format of virtual environment, and also this data set they collected from humans of how


to do this really, like human activity task. So the way they collect, they build this environment


is the first ask many humans to come up with a set of tasks that they do in everyday household.


And then they ask a different group of human to come up with a detailed plan that can drives


a robot to do to perform this tasks. And, and it's after that they build this environment


based on the verbs used by those humans. So you can think of like this environment is


really built on top of what humans says. Now, now, the developers who would just say like,


okay, we want this granularity, you want this like, walk, grab and those etc. So they actually


asked this humans to give those words and verbs and then build those actions according


to those verbs. And they did make sure to, for each of the verb to develop a set of common


sense constraints, which completely makes sense. And I think they're actually, like,


reasonably exhaustive for those actions. So if you want to grab something, you definitely


need to make sure the things you grab is not within a closed container, for example. So


in this case, the fridge is a container and it has this attribute of being open or being


closed. So they internally keep track of the attributes for each of the objects. And then


to make sure that like, if you do something like this, you don't violate the common sense


constraints. So to answer your question, so like this granularity really depends on the


humans. And like, I think this is where language models really shine because it essentially


language model is trained on human produced text. So my hypothesis, although this is definitely


not something they're only tested by, my hypothesis is that because it's trained on human produced


text and humans after all produce these actions. So if you do it careful enough, and then use


some techniques to properly translate them or doing something else, you can essentially


get back something similar to what humans produce in the beginning.


Yeah, I mean, you would imagine that sort of the humaness of how the environment was


built would also be present a little bit in these language models, which makes sense.


I don't have a better idea, like of how to build an environment like this.


So yeah, I think it's pretty reasonable.


Yeah, it's actually not to be really like, interesting to me, because it's like, it's


just super hard for me, if I were to develop this environment, like how would you even


animate like, all this like really, like human tasks, even just in a household setting, it's


super difficult. And I think they did a really good job here. And then I think this is also


what makes like language models particularly useful for this task, because these are basically


just human tasks. And language models are really good at like mimicking humans.


Yeah. Yeah. So on the on the left here, we see a bunch of models that you've evaluated


right here. So again, executability is sort of how like, if it if it matches the syntax


of the environment, if I can map it to that, and also, I guess if it if it violates any


of these common sense constraints. So just like how executable is the plan in the environment,


no matter whether it's the wrong thing, right? And that comes in, in a minute, second, and


correctness is a thing that is rated by human annotators, they look at the plan that was


produced, and they just, from their own intuition, are like, Well, is this a good plan to make


breakfast? Yes or no. And we clearly see like, there is there's this downward trend, if we


exclude the models on the right, there is this trend line here where the larger models,


they seem to produce more correct plans, which means plans that the humans like more, but


they are less, less executable. Whereas the smaller models, they are less correct, which,


you know, we can best correct, I would have expected that, but they're more executable.


Yeah. And you've noticed in the paper that very often, they just produce plans that have


nothing to do with the task description, they will just produce like a plan. And that is


according to the syntax of the examples that you give in the prompt, right? But how can


you explain that? Like, even on the top here, like the large models, it's even better than


humans at correctness. So humans rating other humans think that GPT-3 produces more correct,


correct plans. Why is it so bad at executability?


Yeah. So there are actually two questions that like, I think you erased one is why this


like smaller models like, like, when I say smaller, it's actually still pretty large,


the large GPT-2 model. So why do they produce like more executable plans? And the second


question is why the GPT-3, the large GPT-3 model is actually better than human? So to


answer the first question, I think that's because we did find some failure modes here


for smaller models. I think the two most prominent ones are first, it frequently tries to like


repeat the given example. For example, you give it like how to browse internet instead


like go out to the computer and use type on the keyboard, etc. And then you ask it to


brush teeth, it still goes like goes to the computer and then type out on the keyboard.


So it's totally nothing like sensible here. And the second source of error is sometimes


it just outputs really short plans. If you say like, sleep, task, go to sleep, it's just


like go to go to the bedroom and just stop.


So that's this right here, brush teeth, it's just like, go to bathroom.


Yeah, exactly. So when these plans are short enough, even though it can be executed, like


if you just say like walk to bathroom and walk to the bedroom, just one single action,


like for walk, there's not much like common sense constraints there. So like, you can


totally imagine like it's super executable. But if you present them to humans, of course,


like humans will spot this and then say, Okay, this is not correct. Because we when we do


human evaluations, we were trying to make it simple so that the the error here is not


too big. Because we don't ask like hundreds of humans to evaluate this. We only ask got


to ask 10 evaluators in this case. So so that's why like, did this smaller models are now


really good at executability. And, and the second question that you ask is why this like


larger models are actually better than humans. So we actually this is not a completely fair


comparison if you just look at one axis. So all the results here will be look at from


two axis that we care about. So one is the semantic correctness, which is evaluated by


humans. And the second is the executability. So this human plans that we use are from this


data set that virtual home developers, like crowdsource from from from Amazon Turkers.


So this plans they make sure that like these are executable plans. So which means that


they're they have one like here. Yeah, they'd be over here. Yeah, but but we don't want


to put a spot right there. Yeah, on the right because it's it's hard to see because humans


are a big baseline and reference here. It's not the baseline that we're trying to beat,


of course, like GPD three is out there yet, in terms of like, at the same time, outputting


correct action plans and semantic, semantically correct action plans and also be able to really


ground them in the environment. But using this to access, we can really see, for example,


which is the which axis is the place that as a community that we may want to work more


on to get it better to to get the human levels and with this paper that we kind of find this


result actually a bit interesting to us is that like, for this larger models, like in


terms of semantic correctness, you don't need to worry too much about it. It's kind of already


there if you if you if you do it, extract them. But the real question is, how do we


make them executable for agents that we that we care about? And this is exactly what you


do right in the in like the meat of the paper and the result or these these translated models


right here, that, you know, notably, they do drop a little bit in terms of their correctness


as rated by humans, but they gain massively in executability. And this is the result of


a bunch of different ingredients, like three main ingredients, as far as I could tell,


you quickly want to go like, tell what, like what the ingredients are to make whatever


these models output into what something that I mean, you know, the virtual home is maybe


a test bed, right? It's not I don't see this paper being about virtual home. It's more


like here is a model that outputs something yet I need the output in some other form,


right in in this is very general problem has many applications. And if we could solve that


bridge, that technically is, you know, is a big game. That's exactly what you do. So


how did you go about this?


Yeah, so I say, I just want to make sure that actually, this paper just present a really


like preliminary staff, I don't think it's anything particularly. I mean, it does like


if this problem, but it's a big step, I believe, like, I mean, you the executability raises


pretty, pretty high. I don't I didn't want to oversell you, but not not undersell you


certainly.


Yeah. So, um, but to answer the question, so, so so we actually found like, there's


actually, as you said, there are three ingredients, but central to this is one simple, really


simple technique that we found, that's the most useful, which is action translation.


So because in this virtual home environment, the actions that it supports are are a limited


set. I mean, it's not small, but it's something that we can definitely enumerate with our


computational hardware and in like in a really quick manner. So like just like one 10th of


a second or something, something like that. So let's say if we can enumerate all the actions


that are supported by by the environment, then the question now becomes, how do we translate


the this really sensible action plans generated by language models for but not really executable


plans? How can we translate that into those actions supported by environment or if you


want to deploy something, deploy something in the in the real world, let's say your robot


only supports 10 actions. How do you map those tasks into the 10 actions that the robot supports?


So what we found is that you first need to enumerate all the actions. And then we found


that you can again leverage the the world knowledge in this language models by using


another language model. Namely, here we reuse Roberta, which is a language model really


similar to BERT. And it's a different language model because it essentially it's a mass language


model. So it's really good at outputting a useful embedding like in terms of about the


semantic meaning for that sentence. So what we do is that we take the sentence output


by GPT three or codecs and then we just compare that against all the possible admissible actions,


allowed actions by the environment. And then we found the most similar one in terms of


like this distance in the embedding space. Yeah, we actually use just cosine distance


and found that to work decently well. Yeah, I have I've like, there's like an entire


space somewhere and you just place all the actions. I guess you can even pre compute


those right you can pre compute the embedding of all possible actions there. And once my


language model outputs anything at all, all I need to do is ship it through the Roberta


model get its embedding, put it somewhere, get the nearest neighbor. And that's my kind


of translated action. So here we have an example that would that would translate like squeeze


out a glob of lotion into poor lotion into right hand. Yeah, this it would map. Yeah,


action into and poor, poor, it would be the verb lotion kind of the object and right hand


also one of the objects. So maybe like there's two arguments to pour. Yeah, I mean, this


makes it is it seems very simple. But I was at a talk by the people who made the first


version of the, you know, in Gmail, you you have these always like three options to respond


to like the quick quick options to respond, right? Yeah. And and I think the first I'm


not sure how it is done now. But the first version of this, we were like, wow, this is,


you know, you know, cool. It actually takes, you know, it takes into account the the email


message that was there, we always thought it was kind of like a language model generative


model somewhere. So I went to talk, and they were just like, No, we just have a big list


of responses, we just classify, right? Whatever. We just take your message, right? And we just


put it through a model. And then we just classify into this big, big bucket of possible answers.


So I mean, this is even though it is, it is simple. It's, it's, it's, it's very powerful,


powerful method. And that being said, you don't even train this, you take an off the


shelf embedding model, and you compute nearest neighbors. And it does turn out quite well,


you do. However, you talk about this in the paper, there is a bunch of problems. And one


of the problems I see is whenever a step contains like multiple steps, right? Is that like,


is that a big if you found this to be a big problem, because this just maps one action


to one other action. But if it's like, you know, open the fridge and take a glass of


milk, then I have essentially no way of translating that into an admissible sequence.


Yeah, that's a that's a good question. And I think that's one of the main errors that


like, this, this operator model that we use, it's actually a sentence operator model, because


it's trained with a different objective such that it can really, you can actually calculate


this cosine distance between the embeddings they generate. So it's a like, we found like,


it's pretty difficult to map a compounded action, like you said, like, like two actions


in one sentence into one admissible action. But this is partly mitigated by how you tune


the temperature set the sampling parameter, just the temperature for the GPT-3 or codex


models. Because we found that if you do increase the temperature, then it tends to output something


more verbally expressive answers for each step. So that means it's harder to translate.


And we if you if you try like all this, like different settings, we did we in the end,


we found like, usually you want to use like, lower temperature than than what people mostly


use for language generation, for example. Yeah, so that like each action is like small


enough and succinct enough. And then and then after we translate this action, so that it's


easier for this bird model, Roberta model to translate. And yeah, something I forgot


to mention, like, after we got this translated action, we found that it's too useful to put


that back to the original prompt for the translated action back instead of like the original action,


so that you can lie the GPT-3 and codex model to to reason, like, how am I going to do based


on this like action already performed? So yeah, like you said, like you pointed, this


is the third sub figure here.


So we would take instead of instead of generating the entire plan at once, we just generate


one action, then we translate it, and then we substitute essentially whatever GPT-3 output


with whatever the translated thing is, and then based on that create the next action,


it makes sense because you it's like almost like a guiding like a bit of a guardrail for


for the language model. Instead, if you were to let it generate all at once, and then you


translate each action individually, they almost like lose connection to each other, right?


Because this here might mitigate some of this, this stuff ready for a compound action, like


go to the fridge and grab a glass. And the closest I hope that the closest sentence is


the go to fridge, right? The language model might still recover and recognize aha, I haven't,


you know, grabbed, haven't grabbed a glass yet. So that is so these are improvements


one and two. And then the third, the third thing you found that really helps is the prompt


up here. So the priming, which I think in GPT-3, it's very common to have these priming


prompts to tell the model what kind of stuff you you expect as an output. I was surprised


to see that you only have one priming prompt. Whereas in general, people put more than one,


usually people put like three or something like this. Is there a particular reason why


you used just one?


There is actually not a particular reason I actually found like, I mean, in the beginning


we were we know that we have this data set, right? And then we, we found, originally,


we actually tried to train something to achieve this. But in the end, we found out like, we


don't even need to train something. And like, now the question becomes like, like, can you


even leverage this data set to some extent to make it useful? Of course, this is something


like additional, I mean, it would definitely be better without any of this. But if you


have this data set, you can actually found like, this most similar example to the query


task here. For example, like this is apply lotion. So like shave, task shave is determined


to be most similar. Again, judged by this Roberta model using the same technique. Yeah.


So I think that that's the main motivation for using this. But we didn't thoroughly investigate


it like how you structure the prompt, whether you add like multiple things there and then


or you change the template here. Because I just defined this template from day one, like


task something, step one, something, step two, something. Maybe there is a better template.


Maybe you want to add some instruction there to make it better. And so I like, I mean,


this is definitely possible. And we don't investigate them here because we don't just


want to get the best performance out of this. We want to show people like this is something


possible and it's really interesting to us. So that's why we ended up like just using


the most simple technique here. Yeah. And to answer your question, why we don't put


multiple things there. I think one important reason is like because this example plans


that we put in front are produced by humans. And this is because there's a space constraint.


I'm using a like oversimplified version in this figure specifically. But in practice,


these plans are actually pretty long. And they actually already take up a lot of space


in the prompt. So if you put more than one, sometimes it gets too long. And I mean, it's


maybe something handleable by larger models, but we just opt for the most simple case.


And I actually read this, like there's a recent paper investigating why in context learning


works, they frame this as a implicit Bayesian inference problem. And they did come to a


conclusion that the longer the prompt, if I remember correctly, it helps the model.


So in this way, you kind of like trade off the number of examples you put and the length


of each example. So in those cases, I think you mentioned many people put many examples


before the query. Those are usually the cases where the tasks they care about are smaller.


So for example, you want to ask Einstein was born somewhere, then this is just a sentence.


So you probably want to put more than one sentence there. But this case, our case is


an extensive action plan. So it's already pretty lengthy. And we don't want to go too


crazy over here.


I mean, it's Yeah, sorry, that the recording has stopped on the screen side, but we can


still see it. Yeah, so yeah, I was I was I was quite interested in in that, in the sense


of the prompt structuring, because I know that can also make a big difference. But I


also like the sort of approach of not having too many moving parts, you know, in, in one


single, in one single thing, because it makes things complicated. And for many papers, it


it makes you wonder, like, what was exactly the thing that gave the improvement here?


Now you you do very good ablations of all of these different improvements, which I really


liked. And you showed that kind of the translation is the main part right here, although the


other things certainly also help. Have you ever? So it reminds me a bit of this, you


know, this retro model, these language models that retrieve from the internet as they produce


text, it reminds a little bit of this, right in that you, you, you produce, you go and


retrieve the closest samples in the data set, as you produce the text. It Yeah, I think


the this combination of retrieval and generation is picking up steam. And and it looks pretty


interesting. My question is a little bit, have you tried also, because essentially,


you now rely on this translation procedure to to produce the correct actions? Have you


tried any way to like, let the model know what the possible actions are, like something


like, you know, I can imagine, maybe I, you know, I asked the model first, and then I


get maybe the five closest actions are the 10 closest actions in embedding space. And


then I somehow put these in the prompt here, like, you know, in between, you know, what


am I going to do next? Is it this or this or this or this, right? And then the model


could, maybe I could prime the model to output one of them. And, you know, is there, did


you try any, any way of telling the model more what's even possible in the environment?


Because right now, you're essentially relying on on just the language model itself? Yeah,


that's a really good question, too. So like, we actually didn't try the specific thing


that you talk about, like generate a bunch of possible actions and then ask the model


again, which of these are best. But we did try something similar, which is like being


searched. So essentially, in beam search, you look ahead to see, like what the organs


are, are like having, in the end, get the highest likelihood. So we did try to constrain


the vocabulary that can be used in the beam search. But this is only conducted on smaller


models because obviously, the GPT-3 and codex models are now open to fully open to public.


So we can't, we don't really have full access to different features. Like, you can't restrict


the vocabulary dynamically. Yes. So I've only done this on smaller mode, relatively smaller


models like the GPT-Neo. And then I think I might have tried on GPT-J as well, which


is a 6 billion parameter model. And it actually turns out that they don't do really well with


if you really just constrain the vocabulary that way. Yeah, specifically just the beam


search constraining the vocabulary you can generate. But so my hypothesis, this is not


thoroughly tested because it's not invested on larger models as well. But my intuition


why it doesn't work so well is that this language models are really trained on human text. So


it really is, they're really used to how humans speak a certain language in English. So like


people don't speak things in this way, step one, something, two, something, step three,


something. So that's why if you really constrain the models this way, a lot of the world knowledge


encoded in these models are lost. So basically, and personally, just a personal opinion, I


don't think these models are doing like super intelligent reasoning here. It's basically


just doing kind of retrieving what's, what is trained on. So retrieving this like large


scale text. So if you want to retrieve better, you better adopt the same way that humans


speak a language. So like if you don't constrain the vocabulary, you can get the most out of


a language model. And you can really tell if you adjust a temperature, like if you go


different temperature, they can tell you like different levels of things and they can be


really realistic. But if you really constrain it, a lot of this knowledge is lost and it's


can't really do too much like common sense reasoning here.


You mentioned this a bunch of times, I was surprised to find codecs as a model. And so


you have these are sort of vanilla models. And then you have the translated ones where


all your improvements are in there. So there is the action translation, there is the sampling


even according according to the probability and executability there, there is the retrieval


of the closest prompt and so on. And these translated models, they perform really well.


What I was surprised by and also by the results is that codecs, I mean, that it's even in


here, it's like a code model, but also that comparably, it holds up, right? It's, it's


not as good as the GPT-3 model, but it's also very, very much smaller. So you know, parameter


by parameter codecs is outshining GPT on this task very well. How did you, how did you even


consider using codecs? And how can you explain that this model is doing so well?


Yeah, so one intuition why we actually, this actually came out to be pretty surprising


to us as well. So we did find like this codecs models are really good at generating this


plans and actually from my own experience playing with this models, I did find like


codecs thinks that this is part of some doc stream. So it's actually imagining like people


just like asking the doc stream here, but instead of letting keep generating the code,


we kind of just stop here. So, okay, yeah, finish the doc stream for us. That's enough.


So, so yeah, so it's actually doing some of this kind of doc stream. It generates this


doc stream thing. And I, the reason I think the smaller codecs model are actually better


than the same size GPT-3 model is that because it's trained on a more structured data. So


like code and specifically many of this like this code examples in the dataset, in the


training dataset consists of doc stream and the code. So not only can handle code really


well, you can also generate really realistic doc streams. So, and people in doc stream,


they don't write in like,


Yeah, they don't write a novel.


Yeah. So they write something really step by step and have more structure in it. So


that's my intuition why actually does really well with this test. So you can really process


this sequential, like logical reasoning better than the same size GPT-3 model. But of course,


if you use a larger model, that potentially be more helpful. Yeah.


Or I mean, there is, as you said, there is still a lot of open like questions about how


exactly you structure the prompts. Like maybe this step one, step two, step three isn't


ideal for these language models. Maybe you need to more let them write like a Reddit


post or something about how they went and got a glass of milk yesterday and then translate


that somehow. But yeah, it's pretty cool. So one thing that just came to my attention


right here is this top row right here, which I found hilarious. So this is the task is


complete Amazon Turk surveys. So the four steps apparently that you need to do is walk


to home office, sit on chair, switch on computer, look at computer. Like, is this the is this


is this the this is the description of complete Amazon Turk? It's a pretty accurate description


maybe of what Amazon Turk workers do.


So like I said, this task are generated by crowdsource from humans. And this the humans


here happen to be Amazon Turkers. So one of them decided that, OK, if you want me to generate


some tasks, I would say like just complete surveys on Amazon. So they decided to put


one of this here. And we found this hilarious, too. So like I said, so this language model,


so they can't really handle anything that you wanted to. So because we did put the example


in the front. So I think in this case, the example happens to be something related to


computer and the models actually happen to reason or potentially you could just repeat


the example. But depending on other tasks, it doesn't seem like that's the case. But


it does come to the reasoning that like this might be something related to computer, too.


And I'm like this staffs here.


Yeah, yeah. I mean, this is I mean, it has something like melancholic. And it also has


something a bit, as you said, rebellious of like, you know, I'm here doing my my Amazon


Turk work. I'm gonna you know, I'm just gonna put my Easter egg in there in this in this


data set or, or like, show you but it also shows something I think about the interaction


with this environment because, you know, if you ask me, you know, what did you do today,


I could tell you, you know, I programmed this I viewed a pull request, I sent some email


and so on. But in the action space of this environment, this would all just be characterized


as go to desk, sit on chair, switch on computer, look at computer. And yeah, so so it is really


maybe also a constraint of the environment itself. And and and it's, as I said, I think


the challenge is going to be there's so much knowledge in these language models, and we


somehow need to get it out into the domain that we care about. And yeah, I guess I guess


many opportunities are still there. And in this particular environment, is it so the


way I see it, we have this environment, it's a 3d environment, but you never actually for


the your studies, you never actually had to actually execute anything in the environment.


Is that correct? Or do I see something wrong here?


I think those when you say ask you do you mean like, like run in the environment?


Yeah, like run the 3d environment, like actually give it to the environment because you evaluate


executability you can do with a parser right to see whether it matches the actions and


constraints and the correctness you evaluate with the humans. Because my question was also


a little bit like, why can't I just run it and see if you know, at the end, there's breakfast,


but you already you already said that the tasks are so, so like, how would you how would


you detect there's breakfast, right?


So so in terms of so a bit background here for the virtual environment. So it comes in


two versions. One is the I think they call the evolving graph version, which is a pure,


like you said, a state machine, a Python, like reading Python. So it just goes in and


then checks which whether the actions can be parsed, and then we satisfy the common


sense constraint. And the other version they implement is this is this visual visualized


version, where they actually only implement a subset of the act the total action supported


environment. So I think they so in the evolving graph version, the Python version, there are


42 actions and in the visualized version, there are only 10 actions. So it's limited.


Like the plans we can generate, we can really visualize are limited. So that's also part


of the reason we don't show the visualized version to humans. Like can you tell us whether


this is successful or not? So yeah, that's that's a, that's indeed something we can do


right now. And I think that's like as a community, as we go go on, like, to this next step with


more complex tasks that humans do every day, instead of just like, lower level tasks. As


a community, I think more efforts can be can be put here and to develop better simulator


and also maybe beyond even household environment. So yes, just as a as a story here, I did play


around with the codecs and then GPT three models to have a generate something out of


the household domain. And seems like they do have some a lot of knowledge for those


as well. So if you can ask, how do how do I pay bills at a restaurant? And how do I


work out at the gym? And I think in on Twitter, there's also someone tries to after the posting


of this paper, they try to ask the GPT three model, how do I start a company? So yeah,


they do have a lot of knowledge for this. And as long as you can provide a set of actions


that are necessary to complete these tasks, I think, no matter what, what the granularity


is, ideally, it should be at the same granularity as of humans. So ideally, it should be this


models should be able to generate something, something sensible and reasonable. But yeah,


definitely right now is something that you definitely can trust to put on a robot. Of


course, yeah, yeah.


Yeah, I mean, it's, I've always, I've always seen people thinking when they think GPT three


or so they they and they think, for example, of video games, they always imagine, you know,


we can have our NPC, our characters, that the dialogue be generated by GPT three. So


it the dialogue is more realistic. But I think this shows that it can go further if we are


able to map sort of GPT threes knowledge into a sort of structured domain that we choose,


we could potentially also let these models generate the action sequences of like, of


characters, for example, let's say in video games, because that's like common complaint


that, you know, the guards, they always walk up and then down and then left and then right


and then up and then down. And right, they have these, even if the dialogue gets really


good, their their behavior is still kind of lame, either that or they they cheat, they


know where you are at all times. But with I feel with models like this, we can almost


like take this common sense knowledge and maybe have the hopes of transferring that


to to various domains and infuse a lot of areas with common sense. And I find that to


be I find that to be pretty cool.


Yeah, in itself, that would be really exciting and interesting application. Yeah. Yeah.


So I mean, there's a lot of a lot of things to be gay. So I what I did, I was specifically


intrigued about clip. I don't know if you are thinking about this or not. But what I


what I tried to do is I tried to take like a frame of Pac Man, like and and you know,


there's like walls here and here and and and here and I had Pac Man be like, you know,


here facing a wall and then there's like a ghost behind Pac Man, right? And and then


there's like these little dots over here to to eat. And so it was like super clear what


you have to do. So I tried to feed that to clip. And you know, you can make a clip classify


things by just evaluating a bunch of different strings with it. So I like try to I try to


evaluate the strings, go left, go up, go right, go down, or like Pac Man should go left, Pac


Man should go up, but it never worked out. So if you can, if you could get something


like this running, this would be amazing. Like, with maybe with your knowledge, maybe


Pac Man isn't the right environment because clip was trained on whatever picture scraped


from Instagram. But I think just this this type of, you know, thinking beyond just the


strings in terms of language, but thinking in terms of I have some structured environment


and I want to leverage this, this knowledge of these models is is super cool.


Yeah, that will be a super interesting application. I think using clip here, like, because it


fills in another modality, which is image could be really interesting. So I think it


kind of solves one of the major limitations of this paper, namely just the because currently


we generate plans, regardless of the environment state, so it doesn't condition on environment


state and potentially using clip, you can encode something there because you can also


take image as input to to an image can serves can serve as states for for for the environment.


I think that would be really cool. And yeah, so yeah.


So just to be to be clear to the listeners that the basic idea for this I have from from


a PhD student that was partially in our lab called John Battista for scandal. So the credit


fully goes to him of of this whole idea. I didn't want to but I just it got me thinking


so much about, you know, we can extract this knowledge into into other modalities. And


that's, that's pretty cool. Is there anything you want to maybe say about the experiments?


Is there anything that was very surprising to you or, you know, something you didn't


expect or something you particularly want to highlight?


Um,


I actually I think we covered most things. But I think I might say something about the


the baseline here, I assume you can probably see except for the human references, we also


got to find a GPD three version. And we did find that fine tuning can can be a really


strong baseline here. Because as you can probably tell the one of the measures here LCS, which


is the longest common subsequence. Yes, this measure here is much higher than the others.


This measure, basically, calculates how much overlapping there is in your generated plans


against those plans written by humans. So it's kind of calculating this IOU score. So


we did find this to be a strong baseline. And I think it still actually makes sense


to be a strong baseline because this is trained on such data. And so this is kind of to illustrate


that like, if you do have domain data, it's still really helpful to train your models,


fine tune your models this way. But if you don't have something like this, you can potentially


just leverage the knowledge already in this language models.


Cool. Yeah, so where, where does your future lie? What are you? Are you going to, are you


going more into this direction? Or was this sort of like a one off thing? Or do you have?


I mean, what are the interesting questions that you are asking now, maybe as a follow


up to this?


Yeah, so I personally, I haven't decided because I am in a stage where, like, I'm applying


to PhD programs and, and also other positions. So like, but but as a follow up, I think it


would be really interesting, as I mentioned, one limitation, major limitation of this work


is that we haven't found a clear way to condition on the environment state. So that like, if


you really place an agent in in the household, for example, there is no, if you want to make


coffee, but there's no coffee, but there's no, there isn't a automatic coffee machine,


how would you make a coffee with some maybe similar devices. So the agent can really reason,


if you just put it this way, because it doesn't condition on the environment state. So I think


it would be really interesting to, like, investigate how you can also condition on the current


environments and then and then reason from there. But this might require some training


data. And I think that's part of the reason why we don't, like go full length here to


investigate this, because this is something just for us to tell people like, this is an


interesting finding, and we may be able to leverage something here. But I think this


will be really exciting and like interesting future work.


Cool, excellent. Wenlong, thank you very much for being here. This was awesome. So great


to hear from, you know, from always from the people who made the stuff. So yeah, thanks


a lot.


Yeah, thank you so much. Yeah. And in the end, I think I also want to like point that


like, this is a group effort and really a lot of thanks goes to three of my advisors,


Peter Bill, Deepak Pathak and Igor Mordech.


Excellent. All right. Thank you. And I hope to see you again.


Yeah, I'm like, it would be an honor to always be here. Yeah.


Reinforcement Learning, Fast and Slow

Hi there, today we're looking at reinforcement learning fast and slow by Matthew Botvinnik,


Reinforcement Learning, Fast and Slow

Sam Ritter, Jane X. Wang, Zebkert Nielsen, Charles Spondel and Demis Hassabis.


Reinforcement Learning, Fast and Slow

These people are from Google DeepMind and this is a review of kind of a development


Reinforcement Learning, Fast and Slow

in reinforcement learning, especially as it pertains to kind of how humans learn or what


Reinforcement Learning, Fast and Slow

we can understand from the RL world that translates over to human learning.


Reinforcement Learning, Fast and Slow

All right, so basically their argument here is that the first wave of deep RL, as you


Reinforcement Learning, Fast and Slow

see here, is powerful but slow.


Reinforcement Learning, Fast and Slow

And they give examples of this.


Reinforcement Learning, Fast and Slow

So in box one, box one is this.


Reinforcement Learning, Fast and Slow

So they believe there's an image missing here.


Reinforcement Learning, Fast and Slow

This is backgammon, TD Gammon.


Reinforcement Learning, Fast and Slow

This is the famous DeepMind Atari playing bot.


Reinforcement Learning, Fast and Slow

And this is kind of the 3D labyrinth playing bots.


Reinforcement Learning, Fast and Slow

So there's been a number of advances in RL, and especially what they talk about is deep


Reinforcement Learning, Fast and Slow

RL.


Reinforcement Learning, Fast and Slow

So when we talk about reinforcement learning, the easiest case is where you have an agent


Reinforcement Learning, Fast and Slow

and an environment.


Reinforcement Learning, Fast and Slow

All right, so the agent will observe some observation Oh, from the environment and then


Reinforcement Learning, Fast and Slow

based on that the agent will perform an action a and then the environment will give back


Reinforcement Learning, Fast and Slow

a reward and also a next observation.


Reinforcement Learning, Fast and Slow

So this is a zero, a one, right, and then this is a zero and then here you give a one


Reinforcement Learning, Fast and Slow

a I.


Reinforcement Learning, Fast and Slow

So basically, this goes back and forth and back and forth.


Reinforcement Learning, Fast and Slow

The agent performs an action, the environment gives a reward and the next observation.


Reinforcement Learning, Fast and Slow

So this could be for example, here in the in the Atari world, the observation is the


Reinforcement Learning, Fast and Slow

screen itself, right?


Reinforcement Learning, Fast and Slow

And then the agent needs to perform an action, which is an input of the joystick or pressing


Reinforcement Learning, Fast and Slow

some button.


Reinforcement Learning, Fast and Slow

You can see the individual actions actually listed here.


Reinforcement Learning, Fast and Slow

And then the reward will be given to the agent via a number, which I guess is the same number


Reinforcement Learning, Fast and Slow

as up here.


Reinforcement Learning, Fast and Slow

So the task is to maximize the reward simply by so the the difference is you're not doing


Reinforcement Learning, Fast and Slow

this in a supervised manner.


Reinforcement Learning, Fast and Slow

So you're not telling the agent what would be the correct action to do, you simply tell


Reinforcement Learning, Fast and Slow

it that whether what it did was good or bad by giving it a high or a low reward.


Reinforcement Learning, Fast and Slow

Right, so that's reinforcement learning.


Reinforcement Learning, Fast and Slow

So what is deep reinforcement learning deep reinforcement learning simply means the agent


Reinforcement Learning, Fast and Slow

maps the observation to the action via a deep neural network.


Reinforcement Learning, Fast and Slow

So deep neural network that that that's deep reinforcement learning, where the mapping


Reinforcement Learning, Fast and Slow

or some part of the agent consists of a deep neural network.


Reinforcement Learning, Fast and Slow

And that's you see, for example, here, there is a deep neural network mapping the observation


Reinforcement Learning, Fast and Slow

to the action, as well as down here, but it's a bit more complicated.


Reinforcement Learning, Fast and Slow

Alright, so they argue that the first wave of this was powerful, but slow, meaning kind


Reinforcement Learning, Fast and Slow

of, you need a lot of samples.


Reinforcement Learning, Fast and Slow

And they give two sources of why it's slow, why you need a lot of samples.


Reinforcement Learning, Fast and Slow

They say the two factors are incremental parameter adjustment, and weak inductive bias.


Reinforcement Learning, Fast and Slow

So incremental parameter adjustment means basically, that you have to update or train


Reinforcement Learning, Fast and Slow

your neural network in a very small incremental way, in order to basically, because you train


Reinforcement Learning, Fast and Slow

it one by one, right?


Reinforcement Learning, Fast and Slow

You train your neural network, step by step, you have to make small steps in order to not


Reinforcement Learning, Fast and Slow

forget what, what came before, right, you can't fundamentally readjust your neural network


Reinforcement Learning, Fast and Slow

to every new batch of observations, because then that's going to destroy all the information


Reinforcement Learning, Fast and Slow

you've learned of the old one.


Reinforcement Learning, Fast and Slow

And then we conductive bias here is, is a basically an understanding of these neural


Reinforcement Learning, Fast and Slow

networks, they are general function approximators, and they can approximate any function.


Reinforcement Learning, Fast and Slow

So if you just think in terms of kind of, I don't know, let's say polynomials, and what


Reinforcement Learning, Fast and Slow

kind of polynomials are this polynomial, this polynomial, this polynomial, this weird polynomial.


Reinforcement Learning, Fast and Slow

If I have a function that can approximate all of these, then I have a weak inductive


Reinforcement Learning, Fast and Slow

bias.


Reinforcement Learning, Fast and Slow

Whereas if I kind of know, okay, all my polynomials are the polynomial that I'm looking for, ultimately,


Reinforcement Learning, Fast and Slow

I'm very sure it's a third degree polynomial, right?


Reinforcement Learning, Fast and Slow

So something like this, or like this, or like this, so this is much less of a much less


Reinforcement Learning, Fast and Slow

of a class of functions that I can fit.


Reinforcement Learning, Fast and Slow

But if I'm sure that the function that I'm trying to fit falls in this category, then


Reinforcement Learning, Fast and Slow

I'm much faster.


Reinforcement Learning, Fast and Slow

So this is then called a strong inductive bias is where I build into the model, basically,


Reinforcement Learning, Fast and Slow

I tell it beforehand, here is a very restricted class of functions that you can fit.


Reinforcement Learning, Fast and Slow

Whereas in a weak inductive bias, I won't tell it that I'll simply say, well, model,


Reinforcement Learning, Fast and Slow

you could fit any function you want.


Reinforcement Learning, Fast and Slow

And I'm just giving you training samples.


Reinforcement Learning, Fast and Slow

So this is a classic example of a bias variance trade off, where there is, there's a lot of


Reinforcement Learning, Fast and Slow

variance in these models, meaning you can fit also a lot of functions.


Reinforcement Learning, Fast and Slow

But here, because you buy as the model towards a certain set of functions, it can lower this


Reinforcement Learning, Fast and Slow

variance.


Reinforcement Learning, Fast and Slow

And in this case, here, they argue, it speeds up learning.


Reinforcement Learning, Fast and Slow

Because you don't have you don't have as much variance, that means you can basically go


Reinforcement Learning, Fast and Slow

faster while learning.


Reinforcement Learning, Fast and Slow

Alright, so they propose to two solutions to this problem of this kind of to mitigate


Reinforcement Learning, Fast and Slow

these problems that make reinforcement learning faster, or have made reinforcement learning


Reinforcement Learning, Fast and Slow

faster.


Reinforcement Learning, Fast and Slow

But this is a review remember, remember.


Reinforcement Learning, Fast and Slow

So the first one is episodic deep reinforcement learning.


Reinforcement Learning, Fast and Slow

And this episodic deep reinforcement learning is specified here fast learning through episodic


Reinforcement Learning, Fast and Slow

memory.


Reinforcement Learning, Fast and Slow

So what the suggestion in this field of research is to augment the neural network, or the agent


Reinforcement Learning, Fast and Slow

by a memory.


Reinforcement Learning, Fast and Slow

And the memory could look something like this.


Reinforcement Learning, Fast and Slow

So in a lot of these RL frameworks, what you what a principal component of the agent is,


Reinforcement Learning, Fast and Slow

so the agent will get an observation Oh, and one of the things it has to do is estimate


Reinforcement Learning, Fast and Slow

the value of this observation of this state.


Reinforcement Learning, Fast and Slow

So basically, the agent isn't some state, let's say you play pong, right, and you are


Reinforcement Learning, Fast and Slow

here down, and the ball comes your way up there, right, this little arrow, sorry.


Reinforcement Learning, Fast and Slow

So the ball flies away from you, and you're all the way down, which basically means draw


Reinforcement Learning, Fast and Slow

this figure.


Reinforcement Learning, Fast and Slow

So here, you are down here, and the ball is here flying up there.


Reinforcement Learning, Fast and Slow

So one task in these in these agents, that occurs often is to estimate the value of this


Reinforcement Learning, Fast and Slow

observation basically means how much reward am I expecting from this state going into


Reinforcement Learning, Fast and Slow

the future.


Reinforcement Learning, Fast and Slow

In this case, I probably will not expect a lot of reward since I can't move up fast enough,


Reinforcement Learning, Fast and Slow

right to catch the the ball.


Reinforcement Learning, Fast and Slow

So this, I would assign this state a pretty low value, whereas if I were up here, I would


Reinforcement Learning, Fast and Slow

assign this state quite a high value.


Reinforcement Learning, Fast and Slow

So as we've already seen, this is a this is a deep neural network mapping, we learn to


Reinforcement Learning, Fast and Slow

assign value to different states.


Reinforcement Learning, Fast and Slow

And this is one of the parts that takes a long time.


Reinforcement Learning, Fast and Slow

And these methods, they are the one that's depicted here replaces this value estimation


Reinforcement Learning, Fast and Slow

by saying, Okay, we have an observation, we somehow need to estimate its value.


Reinforcement Learning, Fast and Slow

Why Why don't we look for similar observation, so we have some kind of memory.


Reinforcement Learning, Fast and Slow

Right?


Reinforcement Learning, Fast and Slow

And we go with our observation and we retrieve O prime one, O prime two, O prime three, that


Reinforcement Learning, Fast and Slow

are somehow similar, right.


Reinforcement Learning, Fast and Slow

So in our in our pong example, I'm down, I'm up here, ball moves here, I could be looking


Reinforcement Learning, Fast and Slow

now at, at states where I was here, or where I was here, like really close or where the


Reinforcement Learning, Fast and Slow

ball flew a bit differently, but still in the same direction or down here, right?


Reinforcement Learning, Fast and Slow

So all these states are kind of close to my state.


Reinforcement Learning, Fast and Slow

And I can, I already have, I already have played these since they're in my memory, right?


Reinforcement Learning, Fast and Slow

So with every one of them, I can also retrieve the rewards that I got.


Reinforcement Learning, Fast and Slow

So I already because I already know the problem in reinforcement learning is before you do


Reinforcement Learning, Fast and Slow

an action, you don't know what the reward will be.


Reinforcement Learning, Fast and Slow

But here, I already know, because I've played it, I've already experienced it.


Reinforcement Learning, Fast and Slow

It's in the past, so I know what reward I got.


Reinforcement Learning, Fast and Slow

So, and this is exactly what they say over here, they basically say, here, we have time,


Reinforcement Learning, Fast and Slow

time runs this way, we're in state one, then in state two, and so on.


Reinforcement Learning, Fast and Slow

And we perform actions and, and get rewards.


Reinforcement Learning, Fast and Slow

And what we can do is we can save these states into this memory, as along with their sum


Reinforcement Learning, Fast and Slow

of discounted rewards that we collect from that state on.


Reinforcement Learning, Fast and Slow

And then later, this is like a SpongeBob reference.


Reinforcement Learning, Fast and Slow

If we want to estimate the value of some new state, right, what we do is we retrieve all


Reinforcement Learning, Fast and Slow

of these states from memory, calculate a similarity score over them.


Reinforcement Learning, Fast and Slow

And with with we wait, basically, we add their rewards weighted by how similar they are to


Reinforcement Learning, Fast and Slow

the state that we want to compute.


Reinforcement Learning, Fast and Slow

So this basically amounts to averaging over states, respective by how close they are to


Reinforcement Learning, Fast and Slow

the current state, right, this is kind of a soft, a soft way of saying I only select


Reinforcement Learning, Fast and Slow

the states which are close.


Reinforcement Learning, Fast and Slow

And that gives you a value estimate for the new states.


Reinforcement Learning, Fast and Slow

So basically, this means you just got rid of having to train a value function.


Reinforcement Learning, Fast and Slow

And this will speed up your reinforcement learning quite a bit.


Reinforcement Learning, Fast and Slow

If you don't have to train that, if you already have good value estimations from your previous


Reinforcement Learning, Fast and Slow

experience, that's great.


Reinforcement Learning, Fast and Slow

Of course, there are a number of problems associated with that, namely, if this memory


Reinforcement Learning, Fast and Slow

here, for example, becomes stale, it doesn't represent the future rewards quite as well.


Reinforcement Learning, Fast and Slow

There is also a question of which states do you keep in memory, just the good ones?


Reinforcement Learning, Fast and Slow

Or do you have to have a certain property?


Reinforcement Learning, Fast and Slow

Do you have to have some diversity in there?


Reinforcement Learning, Fast and Slow

And of course, the biggest problem here, the biggest problem is how do you know when two


Reinforcement Learning, Fast and Slow

states are similar or when they aren't?


Reinforcement Learning, Fast and Slow

It might be easy in a situation like pong, where I only have like three variables like


Reinforcement Learning, Fast and Slow

position, y position of my, of my paddle and position of the ball and velocity of the ball.


Reinforcement Learning, Fast and Slow

Those are like, I can specify those in five numbers.


Reinforcement Learning, Fast and Slow

But if it gets harder than that, if it's like this labyrinth setting, full 3d environment,


Reinforcement Learning, Fast and Slow

then we have no clue which states are similar to each other.


Reinforcement Learning, Fast and Slow

And what these what most end up doing is they will train, you guessed it, they will train


Reinforcement Learning, Fast and Slow

a deep neural network to give you this similarity score between states, right?


Reinforcement Learning, Fast and Slow

How they do it is, is a different question.


Reinforcement Learning, Fast and Slow

But presumably, you can train this network offline, basically meaning you can pre train


Reinforcement Learning, Fast and Slow

it, you could pre train it, and then the so we have two stages, stage one, pre train,


Reinforcement Learning, Fast and Slow

train similarity, dnn, right?


Reinforcement Learning, Fast and Slow

And then once we've done that second stage, do reinforcement learning using this.


Reinforcement Learning, Fast and Slow

And the claim here is that by having this done this, the second stage will become faster.


Reinforcement Learning, Fast and Slow

So it, it doesn't really solve the problem of the sample efficiency.


Reinforcement Learning, Fast and Slow

But what it says is, okay, the actual reinforcement learning part will become faster, because


Reinforcement Learning, Fast and Slow

we've already done the work previously.


Reinforcement Learning, Fast and Slow

But basically, by by including this similarity score, sorry, whatever, dnn, by including


Reinforcement Learning, Fast and Slow

this in the language of the review here, we have successfully introduced an inductive


Reinforcement Learning, Fast and Slow

bias into the RL procedure, because the RL procedure now can just fit any function we


Reinforcement Learning, Fast and Slow

say we tell it your value function is one that conforms to our notion of similarity


Reinforcement Learning, Fast and Slow

that we've pre trained.


Reinforcement Learning, Fast and Slow

This restricts the RL algorithm.


Reinforcement Learning, Fast and Slow

And we give it an inductive bias and as long as our similarity score is useful for the


Reinforcement Learning, Fast and Slow

RL algorithm, it can speed up its learning because it doesn't have to learn the value


Reinforcement Learning, Fast and Slow

function itself.


Reinforcement Learning, Fast and Slow

All right, cool.


Reinforcement Learning, Fast and Slow

So the second part here is a bit more abstract called meta reinforcement learning speeding


Reinforcement Learning, Fast and Slow

up deep RL by learning to learn.


Reinforcement Learning, Fast and Slow

These kind of learning to learn approaches are quite abundant in the literature people


Reinforcement Learning, Fast and Slow

try this usually there's a as I mean, it's, it's very large scale experiments.


Reinforcement Learning, Fast and Slow

Basically, you have I think I believe they show it somewhere here.


Reinforcement Learning, Fast and Slow

Yeah, you have like some, some outer loop, where you would say that's this thing here.


Reinforcement Learning, Fast and Slow

What the outer loop does is in each loop, it samples one environment.


Reinforcement Learning, Fast and Slow

So it samples one environment from a distribution of environment.


Reinforcement Learning, Fast and Slow

So now you not only have one environment, but you say, Okay, if I'm going to navigate


Reinforcement Learning, Fast and Slow

this maze, one trying to learn to navigate this maze, I'm going actually to learn to


Reinforcement Learning, Fast and Slow

learn to navigate many mazes, right.


Reinforcement Learning, Fast and Slow

So it's not like you train one agent to learn, you train one agent to navigate many mazes.


Reinforcement Learning, Fast and Slow

That would just be classic reinforcement learning.


Reinforcement Learning, Fast and Slow

But you want to train an algorithm that helps an agent learn as a particular maze.


Reinforcement Learning, Fast and Slow

And you do that by training your helper algorithm on a variety of agent maze combinations.


Reinforcement Learning, Fast and Slow

So in each step, you sample one environment like this, this here, and you then have an


Reinforcement Learning, Fast and Slow

inner loop here.


Reinforcement Learning, Fast and Slow

You fully reinforcement learn train an agent in the classic sense on this environment,


Reinforcement Learning, Fast and Slow

right, you see here, action, action, observation reward, right, but the agent receives some


Reinforcement Learning, Fast and Slow

kind of signal from outside.


Reinforcement Learning, Fast and Slow

So the outside algorithm will kind of tell the agent how to approach the problem, right.


Reinforcement Learning, Fast and Slow

This could be that it initializes the, the weights here, you see that the outer loop


Reinforcement Learning, Fast and Slow

trains the parameter weights, which determine the inner learner that interacts with an environment


Reinforcement Learning, Fast and Slow

during the duration of the episode.


Reinforcement Learning, Fast and Slow

For every cycle of the outer loop, new environment environment is sampled from a distribution


Reinforcement Learning, Fast and Slow

of environments which share some common structure.


Reinforcement Learning, Fast and Slow

So basically, the one would expect when you train this, that these parameters here, this


Reinforcement Learning, Fast and Slow

could be, for example, it could be the initial weights of the network that the agent uses


Reinforcement Learning, Fast and Slow

that this one possibility, right, this is very abstract here, this meta reinforcement


Reinforcement Learning, Fast and Slow

learning, it could be literally anything that the outer model teaches the inner model or


Reinforcement Learning, Fast and Slow

gives to the inner model.


Reinforcement Learning, Fast and Slow

Right. And you, you train both of these with reinforcement learning. So the inner you train


Reinforcement Learning, Fast and Slow

with reinforcement learning on the individual rewards. And then you can train the outer


Reinforcement Learning, Fast and Slow

loop on the reward that the entire app agent environment episode achieved.


Reinforcement Learning, Fast and Slow

So the, that's kind of a two loop situation. And yeah, so that's meta reinforcement learning,


Reinforcement Learning, Fast and Slow

again, is very unspecified what it does. But as you can already see, if you now have such


Reinforcement Learning, Fast and Slow

an algorithm, that kind of tells the inner agent, just as an example, how to initialize


Reinforcement Learning, Fast and Slow

its weights, right, how to initialize the weights of its deep neural network, if you


Reinforcement Learning, Fast and Slow

have that here, then the agent you will technically bias it, this is again, an inductive bias.


Reinforcement Learning, Fast and Slow

So you will give it inductive bias towards what you think are good weights to generally


Reinforcement Learning, Fast and Slow

learn these maze structured environments, right. Since the outer loop, you can update


Reinforcement Learning, Fast and Slow

it way slower, because it needs to learn over a longer time horizon. And it needs to learn


Reinforcement Learning, Fast and Slow

things for a different variety of environments. But once you have good kind of initial weights


Reinforcement Learning, Fast and Slow

for a particular environment, then this agent in here can learn much faster, given an individual


Reinforcement Learning, Fast and Slow

environment. So the agent you instantiated, and then you give it good starting weights


Reinforcement Learning, Fast and Slow

or some other kind of signal about the environment. And then it can go much, much faster at learning


Reinforcement Learning, Fast and Slow

the environment. Thereby, you have just sped up this inner agent by providing it an inductive


Reinforcement Learning, Fast and Slow

bias. And that's basically what what the the these, what the claim of the review is that


Reinforcement Learning, Fast and Slow

by providing these models with a larger inductive bias, you may then speed up their learning


Reinforcement Learning, Fast and Slow

because you've kind of told them what good functions are on the out from the outset.


Reinforcement Learning, Fast and Slow

Of course, you see the problem again here. Well, the problem, the problem is, of course,


Reinforcement Learning, Fast and Slow

you actually need to train this outer sorry, this this outer loop. And the outer loop may


Reinforcement Learning, Fast and Slow

actually take much, much longer to train than a single on an unbiased reinforcement learning


Reinforcement Learning, Fast and Slow

thing. But again, what you could do is you could pre train on a distribution of environments.


Reinforcement Learning, Fast and Slow

And then once a new environment shows up that is similar to this distribution, you can then


Reinforcement Learning, Fast and Slow

have the agent instantiated and learn much faster. So again, kind of this two step process,


Reinforcement Learning, Fast and Slow

you could pre train this outer loop, and then the inner loop will be much faster than if


Reinforcement Learning, Fast and Slow

you didn't have the outer loop. Alright, so those are basically the kind of the kind of


Reinforcement Learning, Fast and Slow

outlines they do here, they then kind of do a connection to like the brain and so on.


Reinforcement Learning, Fast and Slow

And they relate this to biology and biological learning. But ultimately, their conclusion


Reinforcement Learning, Fast and Slow

is here, that whenever you want to do whenever you have slow RL, or this is at least my conclusion


Reinforcement Learning, Fast and Slow

from their article, whenever you have slow RL, you can transform it to fast RL, RL, but


Reinforcement Learning, Fast and Slow

you have to outsource the slow RL, slow, something else slow x, you have to outsource the slowness


Reinforcement Learning, Fast and Slow

to some other part. So if you want to do fast RL, you have to outsource the slowness. And


Reinforcement Learning, Fast and Slow

what the slowness provides is an inductive bias. Which means, yeah, if you want to do


Reinforcement Learning, Fast and Slow

like fast RL with episodic memory, you have to learn the similarity function, which again,


Reinforcement Learning, Fast and Slow

which might be slow in itself, but then the RL will be fast. And if you want to do this


Reinforcement Learning, Fast and Slow

via kind of a an outer meta learner, again, this learning of the outer meta learner might


Reinforcement Learning, Fast and Slow

be slow, but then the inner learner will be fast. In a connection to the kind of biological


Reinforcement Learning, Fast and Slow

aspect of this, they do make a connection which which I find is appropriate in that,


Reinforcement Learning, Fast and Slow

for example, the human brain, the reason we can learn things fast, let's say in the physical


Reinforcement Learning, Fast and Slow

world, picking things up, dropping things down or navigating our paths, we're incredibly


Reinforcement Learning, Fast and Slow

good at this, navigating through like a weird terrain with rocks in the way is because,


Reinforcement Learning, Fast and Slow

of course, our brains have been adapted to these kinds of environment over generations.


Reinforcement Learning, Fast and Slow

So there is an outer process, like evolution, which is this kind of outer loop. And it instantiates


Reinforcement Learning, Fast and Slow

the inner loop, which are the humans that kind of live or die by their ability to to


Reinforcement Learning, Fast and Slow

navigate better. So the if if the outer loop does a good job of only keeping the humans


Reinforcement Learning, Fast and Slow

alive that can navigate well, then the individual human in here that that does this, the individual


Reinforcement Learning, Fast and Slow

human given a landscape with rocks will then be much faster at learning to navigate it.


Reinforcement Learning, Fast and Slow

Alright, so that was it for that I it's an interesting article to read, especially the


Reinforcement Learning, Fast and Slow


[ML News] Google introduces Pathways | OpenAI solves Math Problems | Meta goes First Person

Google introduces pathways, their next generation AI architecture, open AI solves high school


math problems.


And Facebook goes all on first person view.


Welcome to ML news.


Before the video starts, a quick thanks to our sponsor weights and biases.


I want to show you this one feature that I just learned about.


Did you know you can embed a weights and biases report in notion, it's actually not only reports,


but also other stuff by weights and biases.


So they have this neat little page here.


Ironically, it is actually a notion and it is super easy to embed live weights and biases


stuff into notion.


So for example, here I have a sweep and you can see the sweep is interactive.


So you can do all the kinds of things you're used to analyzing the weights and biases sweep.


Now I can just grab that URL, get over to notion and create a new embed, paste the link.


And there we go.


Look at that.


This is a fully functional weights and biases report inside of notion.


So you have all the interactivity here that you would usually have, as you can see, so


I can look at my runs, I can activate them, I can even go and look at my sweep controls


and various things.


This is really cool if you work together with other people and you work on more than just


weights and biases reports, you can take your notes and notion and then embed the report


and the sweep whatever into notion page.


I love notion I love weights and biases and it's very cool to go together.


If you don't know weights and biases, it is your one stop shop for all your machine learning


experimental needs from trying out models optimizing hyper parameters all the way to


saving your models deploying them and so on.


It runs in the cloud.


It's free for personal users and for education.


There are plans for teams and for self hosted setups.


So all the more reason to go try it out.


Thanks again to weights and biases for sponsoring this video.


And now let's get into it.


Bye bye.


Hello and welcome to ml news.


Let's dive into our first story.


Jeff Dean has released a blog post on the Google blog.


Note this is not the Google AI blog.


This is the main Google blog.


He's also given a TED talk about the subject and the subject is this model called pathways,


a next generation AI architecture.


We don't actually know much about this architecture because all we have is that TED talk and this


illustration right here.


And essentially Jeff Dean imagines Google's future AI projects to rely on this new architecture,


where instead of having single task neural networks that train, you have this giant multitask


neural network that can do all the tasks at once.


And that would also be sparsely activated.


As you can see here, different tasks would leverage different paths through this network.


This goes along with a few criticisms on today's architectures.


So he says, for example, today's AI models are typically trained to do only one thing.


Pathways will enable us to train a single model to do 1000s or millions of things.


So the goal is to have one model do many, many tasks at once.


Second, he says, today's models mostly focus on one sense pathways will enable multiple


senses.


This refers to the fact that the input to current neural networks are single modalities.


Sometimes they're two modalities, but mostly they're single modalities, like images or


text or sound.


This pathway architecture naturally being multitask will also be multimodal, which means


that it could input any sort of modality in this TED talk, he gives the example whether


or not you see a leopard or hear the word leopard or hear someone say the word leopard


or see video of a leopard that should essentially evoke the same concept in your brain and therefore


also in the pathway model.


And lastly, he says today's models are dense and inefficient pathways will make them sparse


and efficient.


This refers to the fact that our current networks are densely activated, everything's connected


to everything.


And that's very, very inefficient.


He imagines this future pathways architecture to be sparsely activated, meaning that only


very small sub parts of the network will be activated for a given input sample.


And therefore the different parts of the network doing different things, they don't always


have to be active at the same time.


This can also make the model much more efficient in terms of parameters and computation.


Now as I said that there's not a paper to go along with this or an implementation or


even a plan of how to get there.


This is essentially a wish list and it's not a particularly new wish list like people have


dreamed of, oh, can't we just make multimodal multitask models where one model learns everything?


Well, yeah, everyone wishes that but you still have the problems.


Namely, for example, catastrophic forgetting.


If you try to teach the model many tasks and then one task more, you still have to ensure


that it doesn't forget the old tasks, which is very, very difficult, especially in this


picture, it seems like this is a rather feed forward architecture right here without any


sort of memory modules or anything like this.


So how they're going to achieve that, I don't know.


Secondly, they say there are many different tasks here.


However, huge data architectures mostly rely on self supervision, and then fine tuning


for individual tasks and not having different tasks in parallel, though multitask training


is a thing.


And lastly, the sparse activations are not trivial to achieve.


Again, people have been saying this forever, like, well, can we just have a sparse neural


network, probably the brain is sparse, blah, blah, blah.


But how are you going to get there?


This is just a wish list how we're going to get there.


I don't know.


The main problem with sparsity being that if you have a sparse forward signal, then


your backwards gradients are also going to be sparse.


You may never learn the correct sparse way through your network.


If you only activate sparsely in the forward pass.


These are all challenges that have existed forever.


But it seems like Google is determined to solve these challenges.


I mean, if they can all the better, but for now, it's just a plan and an idea.


And I'm excited to see what happens.


Open eyes released a blog post called solving math word problems where they train a language


model to solve math problems.


This goes along with a paper saying training verifiers to solve math word problems by people


at open AI, you can read it if you want.


Essentially, it is a data set of about 8000 of these high school math problems, where


you mainly need the basic addition, subtraction, multiplication and division in order to solve


the problem.


They're usually stated as little stories, and they have some sort of an answer.


Now large language models such as GPT three are usually kind of bad at this type of stuff,


mainly because they are not accurate enough.


They don't do these simple steps that are required enough.


They're more like a language model.


They're more like a conversation model or a thing that simply repeats some of the stuff


it has already seen.


So the first approach the paper takes is to fine tune such a language model on these tasks.


And it turns out that doesn't go too well.


Very often that makes a lot of mistakes as well.


And the solution comes in the form of what they call verifiers.


So verifiers are model that are not trained to produce the solution, but they are trained


to rate whether a solution to a problem is likely to be the correct solution or not.


So now what they do is they use one model that they fine tuned to produce like 100 solutions,


and then they use the verifiers to rank the solution and pick the best one.


And that turns out to be very, very powerful.


So we've seen approaches like this before you remember the Dali model of open AI also


not only used a generative model for the avocado chair, but it also used the clip model in


order to rank the outputs of the generative model.


So this could be a more general recipe for improving generative models is train verifiers


and then generate a bunch of solutions and rank them with the verifiers.


As I said, you can read the paper and the data set of these math questions is available


to download.


Sam Altman tweeted neural networks really, truly learn.


It's not a fancy trick.


This is one of the most remarkable things humans have ever figured out and the implications


are difficult to overstate.


Now I'm not sure if he just wanted to start like a fire with this kind of things.


There are many ways of going about this, but it seems like the truth or veracity of the


statement entirely depends on how you define learning.


But it seems like Sam Altman and in general, that's what we see out of open AI is of the


opinion that learning that humans do isn't that much different from the learning that


current large scale neural networks inherently do.


Now this is to be set a little bit into contrast with what people from the more symbolicist


camp may think about these neural networks and about the nature of learning and intelligence


in general.


But again, I guess it only depends on the definition of words here and just putting


the modifiers really and truly in front of a non defined word doesn't suddenly make it


defined.


But what do you think?


Let me know in the comments after you hit the subscribe button.


See what I did there.


Next news business insider writes Google's AI researchers say their output is being slowed


by lawyers after a string of high level exits getting published really is a nightmare right


now.


So the article starts off with a bunch of Google controversies.


Obviously some famous people were fired from Google recently and there were a bunch of


scandals around that.


And now one senior AI researcher who spoke with insider on the condition of anonymity


comes forward and says, well, the lawyers are essentially up our necks right now.


It's so difficult to publish.


This is really stifling publishing inside of Google and so on.


And the article backs this up by saying, according to Google's online records, the company published


925 pieces of AI research in 2019 and 962 in 2020.


But the company looks to have experienced a moderate slowdown this year, publishing


just 618 research papers in 2021.


Thus far.


Now, this is the only thing where they actually back anything up that they say.


Now I've no doubt that this is the case inside of these big companies, they give examples


whenever they write words such as bias or fairness, then the lawyers, they would just


have like tons of questions or want to cross them out because they just don't understand


the technical terms behind these things.


Now noteworthy terms like bias and fairness actually have about 60 technical definitions


and they're all in conflict with each other.


So can't exactly fault the lawyers.


What I found funny is that in the last section here, a spokesperson from Google took a statement


and said we're publishing papers at the same rate we did last year.


At this time last year, there were 815 approved papers and this year there are 820 so far.


The spokesperson said adding our website doesn't reflect all papers and is typically updated


a few months after publications.


So they had to bury this on the very bottom of the article right here because they want


to like tell a story about how lawyers are so terrible and about how this exit stifled


Google so much and don't get me wrong, lawyers are terrible.


And I'm pretty sure that they're a pain in the neck.


But the claim that this is especially ramped up now doesn't seem to hold apart from like


one or two anonymous people inside of Google coming forward.


And the fact that they have to hide this thing at the very bottom, which is pretty clear,


like that's a much more likely explanation than Google now suddenly ramping up their eyeballs


of the lawyers like lawyers have always been like this.


So insider, I'm calling crap on you.


DeepMind releases their reinforcement learning lecture series 2021.


This is a lecture series about introduction to reinforcement learning by DeepMind researchers


at the University College London and you can in fact watch all of them.


They're freely available on YouTube.


The slides are available and it's pretty cool if you want to get into reinforcement learning.


It starts out with the simple frameworks and it ends with deep reinforcement learning.


David Ha tweeted the following out a pop up shop in Shibuya will sell clothing with adversarial


patches printed on them to make a fashion statement.


Now while I can't understand this exactly, I do think it's pretty cool.


So the label or the brand or the store is called camouflage against the machines unlabeled


and the clothing features adversarial patches.


Now whether that will help in any way or form like I'm quite doubtful, but it is a pretty


cool inside joke if you meet other researchers.


The next one isn't really machine learning news, but it is quite important.


A contributor to PyTorch has released a viable solution for Python concurrency.


So if you don't know, CPython, the reference implementation for the Python language has


this problem that in a multi threaded application in order to keep track of all the objects


flying around, it essentially is forced to do this reference counting.


And in order to do proper reference counting, it essentially means that every time a reference


is incremented or decremented, it has to lock down all the threads.


This is known as the GIL, the global interpreter lock.


And it is the reason why you can program multi threaded applications in Python, but they


will never be able to use the interpreter at the same time, which means that if you


have CPU bound applications, multi threading will just not help, it will not speed up your


application at all, you need to go to multi processing.


So the rule for the longest time has been if your application is IO bound, then you


can use multi threading because it's easier to program, it's easier to reason about, you


have shared state and so on.


However, if your application is CPU bound, then you have to go to multi processing, which


is quite a bit more heavy, more error prone, so on.


Many attempts have been made previously to remove the GIL, but every single actual implementation


of a Python without a GIL had the advantage of being able to run multi threaded applications


really concurrently, but also the disadvantage that single threaded applications, which most


Python programs are single threaded applications would slow down due to these changes.


But now this new suggestion by Sam Gross, who as I already said, is a major contributor


to PyTorch is actually a viable solution and is being evaluated currently, which is pretty


cool.


And it may be that in the future, Python concurrent programming will get a lot easier.


Big science has released t zero plus plus, which is a model that is a multitask trained


text to text model don't even exactly know how I should call this.


But essentially, they took t five, and they trained it with a bunch of different NLP tasks


that you all frame as a really a text input.


So if you don't know what t five is, t five is this concept that when I have an NLP task,


rather than encoding it somehow in a smart way, I simply encode it as a natural language


prompt.


For example, if I want to translate from French to English, I simply say, please translate


the following from French to English, and then I put the French sentence and then I


train the model to autoregressively predict the English sentence.


This means I can use pre trained language models as a start for these models.


And namely, that is what GPT three does zero shot out of the box.


So the idea here is that if GPT three can do this in a zero shot fashion, these natural


language tasks that are formulated in the language of let's say of the input of English,


can't we achieve the same or better zero shot performance if we don't retrain the model


on language modeling as GPT three is, but if we instead pre train the model on other


tasks.


So T zero is this model that takes a bunch of different NLP tasks puts them all into


the language as a human would input them or type them up.


So they're compatible with a language model trains all of them at the same time.


And it turns out that the resulting model can actually do new NLP tasks in a zero shot


fashion, much like GPT three, but is way more parameter efficient at that.


So this is pretty cool.


And the model is available on hugging face.


A bunch of examples of what that can look like they have different versions of this


model, you can import it in the hugging face API, you can even try it out here on the website.


And the thing I want to highlight is that big science isn't some research lab or a company,


it's actually a one year long research workshop on large multilingual models and data sets.


This is simply a conglomeration of a bunch of researchers from all over the world that


is loosely organized together for one year to investigate these large models.


So it's pretty cool that something outside of traditional academia or corporate research


labs also comes into the game and provides lots of cool stuff for the community.


Definitely check it out.


Check out their paper, check out their models.


Speaking of the hugging face hub, hugging face released this tweet saying that the data


set viewer is available in hugging face hub is essentially a preview where you can for


any data set go and see what kind of samples are in there, not for any data set, but for


any that supports the hugging face streaming API, which are like half the data sets on


the hugging face hub.


This works for images.


So here you can see MNIST and you already saw some NLP things.


So pretty cool.


Hugging face hub is getting more and more useful by the day.


Sight is a sort of a Google scholar ish type of thing where you can look for publications


and then inside the publications, every citation will be annotated first of all with the context


of where it goes.


So any citation target if you click on it, you'll see sort of the context of the citation.


And second of all, it is annotated with the fact of whether the citation actually supports