into that slot, you can just super easily predict a class from it.


Object-Centric Learning with Slot Attention (Paper Explained)

So we're almost at the end.


So you now predict for each slot a class or a description of the object, whatever you


want to predict.


And this is the exact same thing as in this Facebook paper now, where for each for each


of these slots, we've predicted a bounding box.


The question is, how do you assign this to the labels?


And that's pretty easy that there's this thing called the Hungarian matching that basically


what you're saying is you want to be as forthcoming as possible, right?


So if you predict a gray cube somewhere, and there is a gray cube somewhere here, you want


to match them, you'll say, Okay, I'm going to give you the benefit of the doubt.


And I'm going to do your model, I'm going to assume with the gray cube, you meant that


gray cube right here.


And if there is the yellow rectangle, and yellow rectangle somewhere over there, you


don't incur any penalty as long as you predict the correct things.


Now only whenever you predict like a second yellow rectangle, so both of these slots now,


so this slot and this slot for some reason, they predict a yellow rectangle, this one


correctly, and this one was assigned this object, and it incorrectly predicts a yellow


rectangle, sorry, other way around.


This one incorrectly predicts a yellow rectangle where there's no second yellow rectangle in


our label set, there's only this, maybe this green cube, then this will be a mistake because


it can't be matched, it will be matched to the one where it has the least loss, but it


will be matched to something that's not a yellow rectangle.


And therefore, that's going to be a mistake.


So this is how you calculate the loss function with this matching algorithm.


And you can calculate that matching in a deterministic fashion, so you can back propagate through


it.


So you can see, if this slot assignment works, we'll have a pretty easy time then calculating


the classes coming up with a loss, the same for the unsupervised object discovery.


What we'll do is we'll run these things through this slot decoder.


Now this slot decoder is very similar to an generator in GANs.


For example, it takes a hidden representation as input.


Now the hidden representation here is going to be these, these slots, and it's going to


up sample it into an image.


If we train the whole if if we have a good slot assignment mechanism, we can pretty easily


train a decoder like this, right with any method you want.


In this case, I believe they use some yeah, some sort of up sampling up convolution architecture


right here.


And they use the L2, they minimize the reconstruction error between the end the output image and


the input image.


So it's sort of like a variational auto encoder or just auto encoder objective in this case.


Alright, so we know how to encode a picture into hidden representation using a standard


convolutional neural network.


And we know once our slot attention mechanism works, we pretty much know how to go from


there.


So the question is, what is this slot attention mechanism?


Now what we're supposed to do is we're supposed to again assign each one of the features into


a slot and in a very specific fashion.


So if you think about the pixels right here, there can be multiple of these pixels or multiple


of the regions, multiple features can be assigned to one slot.


But we'd rather not have the same feature assigned to multiple slots.


So each slot takes in many features, but the features should be this divided between the


slots such that only one slot attends to a feature.


And by me saying attend, you probably already know where this is going.


So if you have the features, and you consider the slots, right, and we just look at a single


feature for now, what we'll do is we'll have an attention mechanism from the slots going


into the features.


So if you don't know what an attention mechanism is, I have this video called attention is


all you need, where I explain this.


But briefly, the features, they will emit something that's called a key, which is a


vector, and then the slots will emit a query, which are also vectors.


And the sir, the the information is now routed by agreement of key and query.


In this case, this thing, this this feature right here would be routed to this slot.


Now, it would be routed to both slots, but it wouldn't be routed as much to the bottom


slot.


And we make sure that this happens by using a softmax assignment.


So if this is like nine, and this is four, what we'll do is a softmax assignment such


that after that, so we have a proper distribution, which would be something like after the softmax


be something like 0.9 and 0.1 right here.


So you can see that the attention is fairly hard.


So this is basically it's a differentiable way to assign these things.


Okay, so an attention mechanism fulfills the property that we want to basically assign


features to the slots in a way that the slots compete for the features.


As you can see right here, if this slot here matches the feature the best it come it out


competes the other slot because at the end, this has to be normalized to one because of


the softmax.


So this competition is the heart of the slot attention mechanism.


And this is this is how it works.


So this is the slot attention module, as you can see.


So you'll take your inputs, and they have lots of layer norms in here, but disregard


the layer norms.


So what you'll do is you'll calculate the agreement between the inputs and the slots.


Now you might wonder in a standard attention mechanism, you'll have input signal coming


from here, which is like maybe these are the input signals.


And then you construct the keys and the queries for the next layer, you construct all from


that input signal, right?


And also the values, by the way, you construct everything from that input signal.


But in this case, we'll have many features and we'll only have a fixed amount of slots


right here.


So where do these slots come from?


Where do the the signal for the keys come from?


In the Facebook DETR paper, we saw that these are learned embeddings.


However, in this case right here, these are not learned, the slots are initialized randomly.


So at the beginning of each thing, the slots are initialized randomly.


You can think of this as an attention mechanism, where you have the attention module right


here.


And then at the beginning, you simply have randomly initialized positional embedding


or randomly initialized slots.


And then the image is going to be encoded through a CNN right here, giving you a bunch


of these features.


And then you'll have cross attention between these features and the slots.


And that will give you the next layer right here.


Okay.


All right, so you want to calculate the routing between the inputs and the slots.


And then you want to perform a softmax over the slots, which will give you this competitive


nature between the slots.


So all the slots are going to compete for the to be routed for the features to be routed


to them.


And then this is simply the second part of the attention mechanism.


So you will have a weighted mean.


Now this is a slightly different from an attention mechanism, because in a real attention mechanism,


you'll have a weighted sum right here, here you will have a weighted mean, but it's basically


such that you can have a different amount of slots and the kind of values will stay


the same.


That's why you do the mean.


So you weight them up and the values are simply a function of the inputs.


This is like in a standard attention mechanism, then what you'll do, you can see that this


is now called updates.


Okay, so you start with the slots randomly.


And then you use the slots to route the information.


You take the inputs, and you use that information routing to calculate the updates.


Now you put the updates through a GRU with the state being the previous slots.


And then you add that to the slots, either this says optional residual MLP.


So what you can do is you will have a residual MLP or not.


This is a fairly complicated thing.But if you think of it, it is just a transformer.


So what they describe here, sorry, the purpose of this GRU here, of course, is that the GRU


is a recurrent unit.


And you can see right here that they do this multiple times.


So once you start with the random slots, right, but then you update the slots and you go,


you go again.


Okay, so you do this.


First of all, you'll have the features, and you'll just have random slots.


And then you do a bit of routing.


Okay, okay, so now we have a bit of routing.


Cool.


You update these slots to be the next set of slots.


And then you take the same features and route them again.


So you you route them again.


And this is supposed to be kind of this iterative procedure.


You might have seen this in capsule networks.


I've done a video on capsule networks, where exactly this type of iterative routing, you


always have the same routing functions, right?


These functions for value, and key and query are always the same.


But you do this iteratively many times in a row.


This is like a transformer with weight sharing.


It's exactly the same, right?


So you have these slots, you initialize them randomly, you do your query times keys, your


softmax times the value right here.


And this transformer even has this plus this MLP layer right here, right?


This, the transformer has that in there.


And then you simply do it again.


So up here, you have the next transformer layer, but instead of being its own layer,


you'll copy the so it's it's weight shared.


It's a transformer with weight sharing between the modules, and the inputs, they are also


copied up here.


This the side inputs.


All right, it's otherwise it's the it's the same thing except that these aren't produced


by an encoder that is also a transformer, they're actually produced by a CNN.


And the weights here are shared.


The only difference is that in between here, they also have like this gru, this gru thing,


but they do an ablation on it.


And it's actually not that important.


So you could might as well just leave it away.


It brings only very few benefits.


So this is how I want how I think of this model.


This is a multi layer a T layer transformer with weight sharing in for the individual


layers where the inputs the input positional encoding are randomly initialized each time.


Okay, now they they really stress this random initialization, because this differs from


the detr paper in that in the detr paper, these things here are learned.


And the detr paper, we have also this kind of object detection thing.


And it what will happen when you learn these is that for example, this one right here,


might specialize in objects that are sort of on the top left of the image.


And this one might specialize in objects that are kind of long and in the middle and so


on.


And this one might specialize to something else.


Now I can't tell you what works better or whatnot.


It seems like you can, if I were to implement something like this, I might want to go with


the Facebook one and then just have more right here.


In this paper, they opt for having fewer, but because they're fewer, if you learn them,


they become, I guess, too specialized, and you will need to keep them agnostic.


So you don't want to learn them, you simply want to randomly initialize them each time


and via via the iterative routing via the weight sharing, they will be sort of assigned


correctly.


Alright, I hope you could follow this.


Yeah, if you if you want to anthropomorphize this, you could think of each of these slots


starts out just randomly.


And then just by sheer coincidence through this attention mechanism, they happen to be


assigned a couple of these features.


Now, because we train the model to perform well, because they're already assigned these


features in the next layer, they'll basically ask through the query function, they'll ask


for more of that, they'll basically say, Oh, I'm now responsible kind of for the gray pixels,


give me more of the gray pixels, right?


Give me give me more of that.


And then in the next layer, even more of that even more of that.


And you'll see in the, in the investigations into what happens exactly this type of thing


happening.


So if we skip ahead to the experiments, where they show what happens through the iterations,


you can see this right here.


So the attention, the attention maps of these slots, you can see that in after the first


step, you can see that, you know, it's slot two right here is assigned kind of these both


of these objects slot three is already pretty.


So the first step kind of learns to segment a little bit of the image, but not, you know,


too well slot for it.


Also the attention map here is pretty, pretty wonky.


But if you in the next step, and this is kind of crucial.


Actually the these slots, they specialize the slot to realize as well, I have a lot


of these, these blue pixels, I'm gonna give me more of those, right?


Give me more of those.


So it gets all the blue pixel while slot four has a lot of these golden pixels says, give


me more of that of those golden stuff.


That's also regionally right next to that.


And since these two compete, I'm pretty sure slot two would also ask for more of the golden


pixels because it has a lot of golden pixels.


But it competes with slot four because of the softmax.


So all of the golden pixels are assigned to slot four and not slot two, while all of the


blue pixels that slot for surely asks for as well, are assigned to slot two in the next


iteration.


So I actually consider iteration one is for you take the randomly initialized slots and


you kind of assign them stuff.


So this is mainly the, this is now mainly the transformer layer learning to segment.


But then step two is where the magic really happens is where the slots, they kind of realize


what's assigned to them and they ask for more of it.


And through the competition, you'll get this separation into objects, right?


So the whole thing is trained end to end, which basically means that these functions


get really good at doing this kind of segmentation.


And then in subsequent iterations, you can just see this effect, multiplying even more


and more, right?


But I might even you might even be able to think that you might want to separate step


one and the subsequent steps, because step one is sort of seems fundamentally different


from steps two, three, four, and so on.


Because step one is this kind of assignment process, and then the other steps are refinement.


So if I were to take this model and make it better, I would try to have a special like


not weight sharing between steps, the first step and the subsequent steps.


But what do I know these?


Apparently it works.


Okay, you can also look at the reconstruction since their objective is to reconstruct.


So basically what each slot outputs, each slot out, if you reconstruct each slot here,


these are the different slots, each slot is supposed to output a picture of reconstruction.


Now if we consider that each slot is responsible for an object, you might very well say, okay,


this slot here gives me a picture with just the object in it that it's supposed to reconstruct.


And then this slot here gives me a picture with just the object that it is supposed to


reconstruct.


Now, how do you know how to combine these pictures, especially since they might be overlapping


and so on?


So the way you do it is you actually output four channels.


So you output R, G, B, and A. So A being the alpha channel, so each slot also has to decide


where the object is that it reconstructs.


And so each, so this here might be, okay, everything here is alpha one, and including


the shadow, maybe, maybe there's a little shadow, and everything else is alpha zero.


And then the alpha maps you combine also via a softmax to ensure that they sum up to one.


So you combine the pictures, including their alpha maps.


That means you can basically reconstruct from the slots where the where the objects are.


Now you'll know, you'll notice this is this thing here, you'll notice that they often


use, for example, here, four different slots, because even though the image has three different


objects, why is that?


Because you need to reconstruct the entire image.


So you need at least one slot for the background, and that's always what you can see right here.


So if you have the, sorry, the reconstruction, you'll see that slot two, with time, with


iterations, it reconstructs this cube, slot three reconstructs the ball, slot four reconstructs


the yellow cylinder, and slot one reconstructs the background.


Okay, also here, if you see the attention masks, you see that the slot one will be responsible


for the background here, the background is significantly darker than in these others,


though they do say the background doesn't really tend to go to one slot in particular,


it tends to kind of spread out across all the slots.


And this might mean more investigation.


Yeah, so they have these different tasks right here, for example, to segment these Tetris


blocks here, and you can see the segmentations, it works pretty, pretty well.


Now why does this work so well?


It's probably because of the data sets.


So these kinds of data sets, they come, you know, they're produced by a generator and


the generator specifically has these these objects right here, and it sort of in it arranges


them in an independent fashion, the background is really clean, right, the objects themselves


are really clean and geometric and so on.


And they're, they're kind of arranged in a random fashion.


And then there's a render of that.


So this is like super, super duper clean data set.


And I guess that has a lot to do with why these methods work so well, because they can


just assume okay, an object is generally, you know, spatially something, some geometric


shape that I know, it's close together, it's pretty independent from its surrounding, and


it's trained with objects that are almost zero correlated, like there's zero correlation


between the objects in the training data set.


So I wouldn't yet apply this much to to real world problems, but it is an interesting thought


right here.


So that's the sort of idea behind the paper, I hope you got that they do a lot of experiments.


And here is a bit where my quarrels start.


So they say that they compare, for example, with the with these in the unsupervised object


discovery experiments, they have this data set called clever.


And this data set has these images with sort of, I believe, clever six has all up to six


different objects.


Now this is already one of the things this is not a specifically quarrel with this model.


But if your data set has six things, they all they give like, they give seven slots


because they know that the data set has at most six things, which means they can always


cover all the things now it works when there's less objects.


But I think the knowledge of how many objects there's going to be is also a big part of


why these models work and why maybe it's not entirely ready yet for the real world.


But anyway, they compare to these two baselines.


They're called iodine, which also employs kind of a recurrent architecture, but not


with an attention mechanism, and Monet, and they say, yada yada yada, this replaces lot


of them.


No, that's not it.


For the Monet iodine DSP and baselines, we compare with the published numbers, as we


use the same experimental setup.


So they say they use the same experimental setup.


And that's why they don't reimplement these models.


But they use the published numbers in their respective papers, which is something you


can do.


This is often, I guess, these machine translation papers, and so on, they do this, just because,


you know, it's a lot to run these things.


However, here, I'm a bit skeptical first, yes, because it is it is Google.


So they do have a lot of resources available to technically run these things.


I've seen at least Monet has an implementation by the author, or I've seen one of them the


other one also, there is an implementation and there's eight authors on this particular


paper.


So I yeah, I, this would be okay.


They say as we use the same experimental setup.


So in that case, if you have the same setup, it's more okay.


But it really depends on you really having the same setup.


And this is a bit where it kind of falls.


So for example, one example right here is, they say we train the woman using the atom


optimizer with a learning rate and 60.


So on they use a single GPU.


We further make use of learning rate warm up to prevent early saturation of the attention


mechanism and an exponential decay, decay schedule in the learning rate, which we found to reduce


variance.


So I've checked these other models, and none of them talks about learning rate warm up and


nowhere in their code is their learning rate warm up.


Now, you might you might argue, okay, this is specific to this model, it might need this.


But if you look at the results, right here, for example, you see that they don't outperform


these other models by too much.


So you can see right here, this is on par.


This here outperforms this one a little bit.


But then also the star here means that they have left out one of the denotes that one


outlier was excluded from evaluation, I guess, which is valid if it's a super outlier.


But in this case, I would categorize this model as a different way of doing doing things


and not necessarily outperforming the others.


So this also, if you look at the ablations, the differences here are minuscule.


And in these ablations that they show, every single thing they do, like gives them a little


bit of a boost, and you just make it kind of across the line to reach state of the art.


I'd rather have research move in a direction where we just show cool ideas and that they


work and that's what this paper does to to be fair.


What I do have more of a problem with a little bit is this here.


On Clever 6, we can use a batch size of up to 64 on a single v100 GPU as opposed to four


in this iodine baseline, right, compared to iodine, our model is significantly more efficient


in terms of both memory consumption and runtime, which is, you know, something I believe.


But this characterization that they use a batch size of four, and here in this paper,


they can use a size up to 64 on a single v100 GPU.


I've read the iodine paper, and the iodine paper says, yes, they do use a batch size


of four on one GPU.


So they also use one GPU.


But they say their GPU has a RAM of 12 gigabytes.


And 12 gigabyte RAM GPUs that points to something like Ti, either I guess a 1080 or a 2080 or


something like this.


This is not a v100.


The v100 come in 16, or probably Google has the 32 gigabyte version.


So this is a 32 gigabyte GPU that is significantly better than the Ti GPUs.


These v100 they cost like five or 10 times more than the Ti GPUs.


And to simply say, we have also one GPU and we can run up to a batch size of 64 and they


can only run a batch size of four.


It seems, I don't know, it seems sort of overstating what you can do now.


Maybe I'm wrong.


Maybe they have actually tested this other model and concluded also on their GPUs, it


can only run to a batch size of four.


But I highly doubt it, because in their, here the paper is cited and in their paper, they


explicitly name that they use for a batch size of four for their 12 gigabyte GPUs.


So yeah, in this, that just kind of pulls through.


So there's the, there's the minuscule improvements.


And then there's the ablations of all these tricks where everyone just gives you a little


bit.


And then there is this kind of very, very, very favorable comparison, wordsmithy a bit,


which gives a bit of a bitter taste to what I think is actually a very, very cool method.


Because why is this method so cool?


Because for example, these slots here, they are trained to also absorb the background,


right?


So you can technically at inference time, increase the number of slots, even though


you've trained with just with a few slots, right?


You can, you can increase the number of slots, and the model can just handle it and they


show it right here in these results.


Here you can, this data set has six objects, and this data set has 10 objects.


Now the model has only been trained over on six objects, and they can just up the number


of slots at inference time, and it'll work also very well.


Also they can now up the number of iterations since these are all weight shared these iterations,


right, we've looked at it, there's there's weight sharing between the iterations.


There is nothing stopping you from just piling on here because it's weight shared, you don't


need any more weights, you can just refine this iteration.


And since the iteration themselves are refining these attention masks, anyway, you might as


well at inference time, refine them some more.


They have an ablation where they show that technically, like three, two or three iterations


at training time gives them the best result, I guess just because of gradient propagation,


because more layers means you have to propagate the gradient back more.


But at inference time, you can just up these iterations.


And as you can see right here, you get better and better.


So this these results are, are pretty cool.


They respect the property that sets should be permutation invariant, and, and so on.


This routing view of the transformer is pretty cool, even though it's you can look at it


as a transformer with weight sharing or an iterative routing protocol, like in capsules.


So all of this I find to be very, very cool idea.


And I think that's how we should look at this, this paper.


So before I am too critical of this paper, I want to say that I really like the idea


and the algorithm here, the implementation.


Yeah.


So that was the paper at last, I actually want to look at the broader impact statement


just because I've I've complained about brought the need for broader impact statements.


So I just want to kind of go just read them and just like look how how the different companies


have the different institutions, how the different people how the community reacts to them, crafts


them and so on.


So this one I find particularly interesting.


Let's go through it.


It says the slot detention module allows to learn object centric representation from perceptual


input, okay.


As such, it is a general module that can be used in a wide range of domains and applications.


In our paper, we only consider artificially generated data set under well controlled settings,


where slots are expected to specialize to objects.


However, the specializations of our model is implicit and fully driven by the downstream


task.


We remark that as a concrete measure to assess whether the module is specialized in unwanted


ways.


One can visualize the attention masks to understand how the input features are distributed across


the slots.


While more work is required to properly address the usefulness of the attention coefficients


in explaining the overall predictions of the network, especially if the input features


are not human interpretable, we argue that they may serve as a step towards more transparent


and interpretable predictions.


This is a I mean, it's a fine statement, but it's not a broader impact statement, right?


If you followed a bit what the broader impact statement is supposed to be, this is not one.


Okay, the closest this comes to a broader impact statement is said, as such, it is a


general model that can be used in a wide range of domains and applications, and maybe a little


bit that you can visualize the attention masks to understand how the input features are distributed.


But the broader impact statement is supposed to give you a preview of how this might affect


society at large, while this here just kind of lists properties of the model for the research


community and sort of for this, for the application of this model, as, as you know, the introspection


of the model itself, this says nothing about society as such.


So maybe, you know, maybe that's, I think that the smarter people will turn the broader


impact statement into more of an introduction section, because that's something you usually


put in a conclusion, or in an introduction, where you say, look, here are some things


our model can do.


And this is what we might be useful for.


And this is how you could introspect it, and so on.


And since the broader impact statement, especially at NeurIPS, you were allowed to put the broader


impact statement on the main paper, so not in the appendix, but it wouldn't count towards


your page limit.


It's, I guess, pretty foreseeable that what people are going to start to do is simply


put more of their paper into the broader impact section kind of cloaked in the veneer of a


broader impact statement.


But I this is, this is clearly not what what the broader impact statement was originally


supposed to be.


Now, I don't know if this is good or bad.


I just think these authors are, you know, they're doing, I think, a good thing here


by simply telling us actually something useful about the model.


But that's just my opinion.


I do thank you for being with me here.


I know this was a bit ranty, flip flopping back and forth between the different things.


We haven't looked at set prediction at all.


We've only looked at these kind of masks, but I invite you to go through the paper yourself


and check it out.


It's pretty cool.


And they do describe a lot of things in pretty detail.


The appendix is very long and has very many ablations.


And this is something I do appreciate.


This Team won the Minecraft RL BASALT Challenge! (Paper Explanation & Interview with the authors)

If we just do behavior cloning using this data, you know, won't cut it like we don't have enough data.


Hello there. Today we're going to look at this right here. This is an agent in Minecraft that's trying to build a waterfall.


So the goal is to go up a mountain, find a good spot, put down some water, turn around and then take a beautiful picture of the waterfall.


That is one of the four tasks of the mineRL Basalt competition. This is what we're going to talk about today.


And not only are we going to talk about the challenge, the competition, as you can see, make waterfall is one of the four subtasks.


We're actually going to talk to the winning team, to the Kairos team in just a second.


This is just the intro. I want to tell you a little bit about what's going on.


So that later in the interview with the authors, you can follow if you don't know what Minecraft is or sort of the basics of these competitions.


If you do, feel free to skip ahead. This is just going to take five to 10 minutes right here.


So I want to show you another one to give you a little bit of the impression of what these agents can do.


I haven't actually looked at many of them. I don't know what's going to happen right here, whether that's successful or not.


These are the actual videos that the judges saw that that were part of these competitions.


So the competition is human judged. There's no reward function.


It's literally you just give 10 videos to a human and they're supposed to rate how good these things are, how human like they are and so on.


Ah, it missed the waterfall a little bit right there. Let's see whether I can turn around.


Yeah, it can not spot on, as you can imagine, and not spot on in any of the 10 things, but good enough to win this competition.


So how did this team go about this? If you don't know what Minecraft is, Minecraft is this game that it looks like, you know, it's it looks like it's from 1990 or so.


Everything is made of blocks, but it is a really cool game. It's a completely open world game.


You can do anything and everything. You can craft items. All of these blocks you can destroy and build up somewhere else.


You can collect items and craft new, better items from it. For example, you can craft a pickaxe with which you can mine things, mine stone.


From that you can build like an oven, a smelter and smelt iron ore. From that you can build iron tools and so on.


This world is completely procedurally generated. So there is there's no the level is never the same.


And that's one of the things that makes these challenges so hard. And the other thing is just the sheer amount of freedom that you have right here.


So the agent now has spent quite a bit of time looking for a good place to build the waterfall.


It looks like it got stuck right here. That must that that's kind of one of the failure cases I imagine. Or it's going to get out.


It's going to get out. What what a what a clench, clench, clench play there.


It looks like here it's a good spot for waterfall. Yes. Put it down. Walk away from it. Turn around. Snap picture with the sheep in it.


Beautiful. So this has actually led to a paper as well by the winning team called combining learning from human feedback and knowledge engineering to solve hierarchical tasks in Minecraft along with open source code that you can check out.


So you can retrain their agent. You can look at their code and you can improve it. It's MIT licensed.


Therefore, you know, all good to go for you. So what did this team do that gave them the winning submission?


The challenge in itself is you're given the tasks in just a short string. So there's not a reward function or anything like this.


The short string literally is, for example, the find cave. It's the agent should search for a cave and terminate the episode when it is inside one.


That is the entire description of the task. As I said, no reward functions. You do get 40 to 80, I believe playthroughs 40 to 80 human demonstrations for each task.


Not all of them completing the task though, and a bit of a code base. And that's it.


This team came up with the following solution they built at the core, they built what they call a state machine, but I want to start somewhere else I want to start from how they used the human demonstrations.


So they had humans and demonstrations of humans solving this task. And then they trained a navigation policy. This is trained via behavior cloning.


So you try to make an agent that just kind of clones the human movements. They did cut out all of the sort of interacting with the environment things from the human demonstrations, such that it was just only navigation going from point A to point B.


This is a policy that they can activate at any time. So as you can see right here, this gives rise to these two one of what they call learned or engineered subtasks.


So you they have a stack of these subtasks. One of them is this navigation subtask that is obviously learned. They have other ones that are just hard coded.


For example, when it's time to actually place the waterfall at that point when you think you're at a good point to build a waterfall, this movement of stacking up the blocks and then putting the waterfall on top, that is a hard coded policy.


So these subtasks are hard coded partially and partially learned, and they're controlled by this state machine.


On top of that state machine, which we're going to get to in a minute, the state machine itself is controlled by this state classifier.


So the state classifier is a thing that they came up with.


They take pictures from the game, frames from the game, and they collect additional human labeled data where for each picture they let the humans label, for example, is this inside a cave, which you can see right here, that's inside a cave.


If you play Minecraft, you'd know. Is there danger ahead, which means kind of a large body of water that you should avoid or something like this?


Do you have animals, which is relevant for some of the tasks? So they build up the state classifier, which is also learned, and that state classifier is now going to control this state machine.


I'm not sure if they actually have it somewhere for one of the tasks in the paper. They do have it in the accompanying presentation.


The state machine controls what the age or which sub policy is active at any given point. Let's see. It's not here.


Well, I can maybe maybe I can I can draw it a little bit you're going to see in the presentation.


So you start and then you, for example, if it's the make waterfall task, you go you get to a point where you want to ask, is there a good spot to place the waterfall is a good spot in sort of the view of the agent?


If no, then you go to the explore sub policy. And if yes, then you go to the go there, the go there sub policy is activated.


These are these sub policies that we saw are either learned or hard coded.


For example, the Explorer one, you can imagine, maybe it's just sort of walking around until the state class classifier tells you that there is actually a good spot.


So what makes the decision between no and yes, that is exactly this state classifier, this trained state classifier, at some point, it will tell you, ah, now you found a good spot.


And then you can switch policy. So from there, if after the go there, you get to another decision point, and the decision point might be like, are you in front of a big wall?


If yes, use the jump policy. If no, use the walk policy or something like this.


So, as you can see, the state machine itself is hard coded. So the humans came up with, what do we need to do to complete the tasks, but the individual steps, they can be either learned, or hard coded policies.


And that's how they go through fulfilling these tasks, they use the state classifier to always tell them what specific sub task here should be activated at any given point, controlled by the state machine.


And, you know, with that, they finish the task. One additional thing that they sometimes need is this estimated odometry.


This is where they just look at the actions they've performed so far. And they build this overhead map of the agent. As you as the agent walks through the environment, they're able to sort of remember things.


For example, this here is has animals. So they're remember, they're going to remember locations of animals, of bodies of water, and so on. And that allows them later if on in the later stages, if they need to go back to something, they can efficiently find it again.


For example, in the waterfall sub task, they have to go away from the waterfall, turn around to put the waterfall inside of their field of view, and then take a picture or finish the episode.


That could be controlled by this overhead map that they build up. It's pretty interesting. All the while they only have access to the image of the simulator, they do not have access to like the F3 menu, or anything like this.


All they have is the image, they do have some information on their inventory and their current item, but not much more than that.


Alright, that was it from me. If you're interested read this paper, it's a pretty good write up. And also it has a lot of evaluation, they did a lot of human evaluation as well, computing these true skill ranking scores and so on to compare their system and do various ablations.


It's really interesting. But now I want to give over to the interview part of this. Let me know how you like these more interviewee style of ways of presenting papers.


This one is obviously a very, very applied paper, very visual paper. But yeah, let me know what you think and now enjoy.


Hi, everyone. Welcome. Welcome. This is this is a really, really awesome opportunity. Right here. I'm joined by the winning team of the mine RL basalt challenge 2021 by David Watkins, Nick way to which and Vinicius Gooks who managed to somehow lock their way into winning this competition.


No, I'm kidding. I'm kidding. This is really awesome. I've seen the videos of your agent. And congratulations, first of all on winning and welcome to the channel.


Thanks for having us. Yeah, thank you very much for having us. We're excited to talk about the work.


So if you could describe in your words, the challenge itself, the challenge is about just sort of a bunch of tasks and then humans rate these tasks.


How have you what make what made you decide to take part in this challenge? Even how did you find it? Did you just stumble across each other? How did you form your team?


Like what was your interest in this? Well, I can say that so we all work together. So that's it wasn't like we kind of find each other.


We've had prior experience working together at the Army Research Lab. And, you know, I think Vinicius was actually the one that stumbled upon this challenge.


And what we liked about this challenge was that it's, you know, it's different from most other machine learning challenges out there, different from other AI competitions.


And the fact that, you know, you don't have an objective function to optimize over. Right. So it immediately makes it harder.


You know, the challenge, again, like it's in Minecraft with these very free form, you know, almost lifelike tasks where really you just have a description, a human readable description of what that task is.


There's no reward function, no objective function. So automatically means you can't just apply standard reinforcement learning techniques and you have to employ some sort of clever measures and potentially learning from humans, which is really what the core of the challenge is about learning from humans.


And that's actually, you know, each of us have machine learning backgrounds and the research that we do is kind of human guided machine learning.


So this challenge is almost like perfect for us. Like, oh, this is this is a great challenge. We knew it was going to be hard. But yeah, that was kind of the calling for us.


And just so for I will have introduced this, but the challenge was there were four tasks and every task was just given, if I understand correctly, like a very short description of what to do.


So, for example, find cave is the agent should search for a cave and terminate the episode when it is inside one. That is that is all.


And all you have as an input, if I understand this correctly, is the screen. Right. Not nothing more.


Well, you do have the screen and you do have your inventory and the item that you have currently equipped and the screen 64 by 64 RGB.


That that is a horrible resolution. But you do not you do not have because in Minecraft for people who play it, there's F3, right?


You can press it. You see your coordinates. You see sort of your bio and so on. Not you have none of that. You have to sort of do everything from from the screen alone.


And you're given 40 to 80 human demonstrations, if I know this correctly, but not all of them successful. Right.


Is that that was a surprise for us as well when we were using those demonstrations in our Asian and we realized like, look at this guy. He just walked around and through the snowball to end the episode.


How is that even useful? Like, it was a surprise for us as well.


And and sometimes you get some items. So one of the challenges, for example, is to it's called create village animal pen, where it is after spawning in a village, build an animal pen next to one of the houses in a village.


Animal pens must contain two of a single kind of animal. You're only allowed to pen chickens, cows, pigs or sheep. Don't harm the village.


And you're in this case, you'd be given also some sort of a fence and fence gates in order to build the pen.


So it's not like you would have to go collect resources, but the task is still quite challenging.


Exactly. Yeah. You don't have to collect any resource or build anything. You were given everything on your inventory, but like completing all those tasks was already a huge challenge.


So, yeah. And especially given that, again, to remind people, the reward here is not some function you can compute.


The reward is at the end, it's given to human raters. The human reads the description and then the human decides how well did your agent perform it.


And most striking, I find this in a third task that is build waterfall, where the goal is that you have to, I can maybe read the description, after spawning in a mountainous area, the agent should build a beautiful waterfall.


That's part of the description, a beautiful waterfall and then reposition itself to take a scenic picture of the same waterfall.


The picture of the waterfall can be taken by orienting the camera and then throwing a snowball when facing the waterfall at a good angle.


So there is even an essence of sort of subjectivity, judgment, beauty and so on in it.


So that just, you know, that is the challenging part, I think here. What was your first, you saw this, you thought, I want to do this challenge, we want to do this challenge.


What was your first try? Like, what was the first thing you threw at the problem?


Well, I can speak a little bit about it. Like, at least me, myself, like when I read the challenge, I had no idea how to approach it.


Because I was thinking, OK, we have a few demonstrations, but like from my experience, research and everything, I thought if we just do behavior cloning using this data, you know, won't cut it.


Like we don't have enough data. And then we like it took us like a month to solidify like an approach. We thought about behavior cloning.


We talked about Gale. We thought about like, OK, let's hardcore this whole thing.


We definitely thought about different approaches. And then I guess in the end, it was a mix of everything.


And that's what you make clear. So there is a paper about you wrote a paper about your approach as well. And the paper's title is combining learning from human feedback and knowledge engineering to solve hierarchical tasks in Minecraft, sort of pointing out that the best approach will be one where learned elements are mixed with hand engineered elements.


How did you so my question is sort of how did you come about this? Was this an iterative process or did you you said you scrambled with a bunch of things at the beginning?


Did you add and add and add? What was your what was your process? What was the first thing that maybe you realized, ah, this works now a little right.


And then how did you build up your your end solution?


Well, so I can add a little bit to that. So, you know, we were motivated like the nice thing about the competitions were motivated to try to do well.


And and so we we knew from the beginning that we didn't want we want to take a different approach.


Probably a lot of people would just try to apply and machine learning, you know, throw a lot of compute at it.


And, you know, we kind of realized that really, if we want a solution that is a little less just academic and more that works for this particular application, we're going to need to really use everything right, including, you know, try to inject our own domain bias about the problem into the framework, into the solution.


So that really led us to these, you know, OK, well, we could have a hierarchy of different modules. Some of those are hand engineered. Some of those are learned, you know, the things that we can't engineer.


And then we can have like, you know, a state machine where we we know the agent should be doing this.


So, you know, let's let's not have the the, you know, our own machine learning component learn the things that we already know how to do from scratch. Right. And just make the job harder.


Let's add that information to the agent and let's, you know, save the learning for the things that we can't easily do. Right. And then have them work together.


Yeah, I think you make this clear and I'm just going to share a screen for a bit right here. You make this clear in sort of this diagram, which is an overview over your system.


And at the core here is this state machine. You want to maybe talk a little bit about why a state machine might make sense right here.


For example, this here is the state machine for for the waterfall task.


I can talk a little bit about it. So if you saw like those tasks. So, for example, let's talk about the build waterfall tasks since we have the diagram open.


There's there's really like a hierarchy of subtasks that needs to be complete in order to, you know, to finish this whole task. For example, for the make waterfall.


You first you need to find a good spot to build your waterfall. Right. And that that means you need to climb up somewhere.


You need to be like at the edge of a cliff. Right. And then you have to actually build a waterfall. You know, you got to equip your water bucket and, you know, pointed down through the water bucket. Right.


And then hopefully this waterfall will be beautiful. Right. Assuming you got like a good spot.


Then you have to go really far away from this waterfall and then position your camera just right to get like the best, you know, the best view of this waterfall and throw a snowball to finish it.


Right. So there's this whole hierarchy of tasks. It needs to be completed like one step at a time.


And there's like this logical order. So the state machine was our approach to make sure that the agent would actually follow this order, you know, without coming back and forth.


Like if you do like, for example, some just an end to end machine learning approach, the agent might, you know, let's say go find a spot and then we'll go back, take a picture, you know, come back again.


Right. Equip the water bucket to build the waterfall. So the state machine was our solution to make sure the agent would follow kind of this logic for each task.


And I think you profit from the fact that all of these tasks can be sort of described quite well in this state machine fashion, as I think a lot of, you know, if you play Minecraft as a human, that's sort of the same thing you do.


Right. You if you want to beat the ender dragon, you okay, first I need to do this, then this, then this. And it's quite the same thing with a few decision nodes in between.


And these decision nodes here in the in the green, those are now decided by a classifier, if I understand this correctly.


So you build this, this little interface here where humans could rate, you were allowed in the competition to collect a little bit like a limited amount of different human feedback.


And you chose among other things, you chose to have humans label different images from the game with such a with with such maybe you can describe it a little bit, what were you interested in? And why did you choose to put the additional human labeling into this task and not any other task?


Well, like, why did you prefer this? Something important to keep in mind is that you're allowed to include 30 megabytes of additional data in this competition.


And the Minecraft simulator is such that if you were to record a bunch of actions or steps that the player took and try to replay them, it's not currently designed to handle RNG the same way every time.


So if I go break a block, that block is going to fly differently depending on the state, the internal state of the random number generator.


And we have no control over that. So you can't seed it necessarily. We can't seeding it just doesn't work.


So we couldn't just collect more demonstration data other than videos. And that would eat into 30 megabytes very quickly, as I'm sure you could imagine.


So dividing up each of the tasks into a bunch of shared states made the most sense to us.


It's something we've used in previous research to handle navigation tasks before.


And it works reliably. And I think there's a lot of research in making state classifiers work really well.


So it was more just us as a team, while we're watching TV, labeling a bunch of Minecraft screens.


The most difficult part, of course, though, is it's 64 by 64.


And there are many situations where maybe you want to recognize that there's an animal in the frame and it's a chicken and it's the small white blob, but it could be confused with a flower.


And you're kind of fighting yourself to make sure that this actually works.


And so there were some different strategies we were looking to employ to make sure that the state was classified correctly.


But it worked pretty well.


Cool. And I think people can see here maybe at this graphic, but you have such things like, for example, good waterfall view, which makes sense, right?


This is a subjective thing of the reward function.


So it makes total sense to include that in the human annotated data and not code a heuristic.


But you also have things like a danger ahead, which you then use.


So I think once you know which node you're in, right?


In this state machine, very often the blue blocks right here, which are the actions, the blue blocks involve going somewhere.


Right. For example, if has mountain, then, you know, if you don't have a mountain, find a mountain.


If you do have a mountain, go to the mountain.


And that part means that your Minecraft agent has to go from point A to point B.


And that's where you build a specialized navigation, navigation subroutine.


And you said right now you've already done this in the past.


Can you tell maybe a little bit in general, what does it take to make agents navigate around?


So can I just mention one more thing about the state classifier?


Sure.


So with the state classifier, like David and Anisha were saying, it's really the core of the state machine.


So we knew we wanted, you know, it's the thing that makes the drives our entire solution.


So it has to be, you know, more or less somewhat accurate.


And we needed a lot of data. So we actually collected around, I think, eighty eight thousand labels, which sounds like a lot.


But of course, you know, that type of manual annotating, no one really wants to do, you know, as machine learning scientists,


we'd rather spend that time trying to, you know, code up a solution to do that instead of doing it ourselves.


But what we did, we tried to make it as easy as possible by, you know, we're not HCI experts, but, you know,


we tried to come up with a kind of intuitive labeling interface to make it as quick as possible to kind of


you know, like one demonstration that's three minutes long at a, you know, FPS of 20 frames per second.


You know, that's a lot of images. And we try to take advantage of the fact that the images are somewhat correlated through time.


Right. So the way we designed our labeling interface is kind of just a step through each image of the trajectory.


And if you hold down a button, let's say one of the buttons is, you know, there's nothing ahead, it's just open fields.


So you can just hold down that button and it's going to traverse, you know, through the demonstration until something else comes up.


And then you can just move a different button. So very quickly, you know, you can, you know, label five thousand images in one trajectory


in like less than a minute because you're just holding down these buttons instead of like, you know, showing an individual image


and then selecting the label and then the next image and selecting the label. I think that really allowed us to get it.


It sacrifices a little bit of accuracy. Maybe when you're transitioning, you might miss, you know, get a few misclassifications,


but you're able to get a lot more more labeled images.


I think this is a recurring theme sort of in real world tasks, the efficiency of data labeling when you include humans.


I've just recently watched sort of Elon Musk's appearance on Lex Friedman. And before that, I've commented on Karpati's talk about the autopilot there.


It's a thing that you see again and again that the easier you make it for humans to annotate data, the more benefit you have later.


It's almost an unfair like multiplier that you have on your system. I think it's neglected currently by academia.


So it's pretty cool that you thought about this as well.


Yeah, I think it is neglected because it is not easy and takes a lot of time.


And like manual labor, nobody wants to do manual labor, but definitely having like high quality labeled data labeled by humans makes totally the difference.


So and now we'll let's go to the navigation subroutine. How do you navigate? Wait, that is here.


So you have a navigation policy which essentially says the agent needs to go from A to B.


And what does it take to build that? Like it seems very complicated in a game so complicated as Minecraft.


So, well, so the behavioral cloning part, right? So that part is, you know, unfortunately just very simple.


It's not any secret sauce or anything complicated.


You know, we again just prefacing by this, you know, was a competition and we had a deadline.


We had so much more that we wanted to do with this particular part, right?


For the solid navigation part, we wanted to do something way more than just standard behavioral cloning, you know, things like generative adversarial imitation learning, you know, trying to have better architectures.


In the end, we didn't have enough time. We were scrambling. And for this component, we just did behavioral cloning.


But the way that we did that is, you know, as you can see in this model, it's like, okay, the agent only has the image as input and its output, you know, are more or less just the direction keys.


So it can go forward, it can turn left, it can turn right, it can strafe left, strafe right, and then it can move its camera.


And really the way that we did that is we just we had all these demonstrations for each of these tasks.


We kind of the only kind of trick that we applied was that we realized, right, this is just a navigation component.


So we only want to learn to imitate the part of the demonstrations that we're navigating.


Right. So let's just chop off that demonstration just to that navigation part and then feed that into our navigation policy.


And so that's basically what we did was, you know, any time where the agent was building, like building the pen or the village or the waterfall, we cut those segments out.


The remaining segments are where the agent is just trying to go from one point to the next.


We kept those in and use that as our training data for the behavioral cloning module.


And in this model here, it says image input. Do you also give the model access to, let's say, the results of your state classifier and maybe the current state machine state or something like this?


So the agent knows where to go or do you rely on behavior cloning for the entirety of navigation?


Yeah, that's a really good point.


It's our this particular navigation policy is just terribly simple. It's really just the image input.


It's being driven by the state classifier in the sense that it allows, you know, the state classifier decides when to start and stop the navigation policy.


But we're not feeding in any information directly from the state classifier or other other more interesting information that that certainly would help.


If we had more time, we could probably do that. It would make sense to do that.


But right now, the state classifier just decides when to start that navigation policy and when to terminate the navigation.


I think I just just went and I had a little bit on top of that.


Like the main reason we didn't add anything else on this is because we didn't have.


So like the so this navigation sub task policy was trained from the demonstrations provided by the competition.


So that data didn't have any like state machine. So the state machine was everything on our side.


So we really only had access to the actions that the agent took right and the camera data.


And again, like I think the using that demonstration data provided by the competition to train only the navigation sub test made sense because let's say think about it.


Let's say we want to do an end to end behavior cloning. Right.


And then you were doing the fine cave task and the fine cave task.


At some point the human will throw a snowball when the agents inside the cave.


And that's only one data sample. And the whole episode has about two to three thousand.


So you haven't one same sample to throw in the snowball on over, like, you know, 3000 samples.


But to find the cave, it took a lot of steps. And this is all really useful for navigation.


So we did this like Nick said, this preprocess to remove all those actions, leave only the navigation part and use that to train this navigation sub task.


And I think that was pretty helpful to in our approach.


So it's fair to say that, for example, you're here and you you are your has mountain classifier says yes, then the state machine would simply activate the navigation.


Does it? Yeah. But it doesn't it doesn't necessarily tell it where to go.


You just rely on the fact that your demonstration in your demonstration, people have generally gone towards the mountain and therefore the navigation policy would have learned that implicitly.


Exactly. Let me I guess let me explain this diagram a little bit.


So what you said is correct. So the green diamonds are decision nodes. Right.


And that's that's based on the output of the state classifier. Right.


So like has mountains, you know, if it's over, let's say 90 percent confidence, we'll take that as a yes. Right.


And then we go to those blue rectangles and each blue rectangle is a sub task.


And those sub tasks can be either learned or coded or like hard coded.


So, for example, go to go or find go actually find go was learned from the human demonstration.


So we would not say like something like, oh, go to this coordinate like we didn't have.


Right. We would just use the human the policy that was trained from human demonstrations to navigate, let's say, going up the mountain.


Right. And then let's say on that part of the diagram where you have the dash line, you know, there's a green diamond there written at the top.


So let's say if the state classifier detect that we're on top of the mountain.


Right. Then we would switch to this place waterfall sub task and this place waterfall sub task was hard coded.


So that was not learned from the human demonstrations. And what the sub task does is basically point your camera down, keep the water bucket and throw it.


You know, that's kind of placing the waterfall. So those blows are a mix of learned sub tasks and hard coded.


What my question is a little bit. You have, for example, this danger ahead state.


Right. But you don't feed any state to the navigation policy. Where is the danger ahead used inside the state classifier somewhere?


Like you say, if there's danger ahead, then we don't even want to activate navigation.


Exactly. So that's something that it's like a safe critical sub task that takes priority over everything.


So it doesn't matter if you're looking at the mountain, whatever you need to do, if there's danger ahead, just avoid it.


Right. So it's like a sort of a safe override that's always on no matter which sub task we're doing, if you're following the human or not, because, you know, just avoid danger.


Because our first iterations of Asian and even the final one still does some time when you fall on one of those lakes, you just can't escape.


It's just too hard. Like sometimes you're like two blocks tall.


Then it's hard to like teach the Asian to break the blocks and jump, like do all those things that us humans do pretty well for the Asian is pretty hard.


So our Asian got stuck a bunch of times. Then we had to add like some safety sub tasks to help a little bit the Asian to escape those things.


And at some point, you also built in this odometry estimation because you only had the image and you thought it would be maybe you can explain this.


What led you because it's not a straightforward thing to include. Right. If I think about how would I solve this task?


What is the odometry estimation? What is it for and why did you include it?


I can talk about it. So like you mentioned at the beginning of the video, we could not like in Minecraft, we do know where the Asian is like when you were playing the game.


You can press like F3, you can see everything. Right. But in the competition, we were not allowed to use that. Right.


So we had some ideas. OK, let's use the simulator. But we were not allowed to do that.


But we're thinking like, what do we know about this problem? Right. So we do have access to the actions that the Asian took. Right.


And we do have access to the image. Not only that, we know a little bit of Minecraft.


So we know that the simulator runs at 20 frames per second. So each frame is one over 20, 0.05 seconds.


So we know this this this time interval between each frame. Right. And from Minecraft, we know that, for example, the walking distance is actually, I think, 4.4 point thirty two meters per second.


So we had this information from the wiki. So let's say if the Asian send the command to move forward.


Right. And not considering inertia or anything. Right. We could assume that in one frame, the Asian walked four point thirty two times, 0.05.


Right. So like this velocity times this DT, this time interval.


So we know how much the Asian walk in the X direction. Right. And then we had the actions.


We had access to the actions for the camera control. So we could estimate the heading.


So just based on the actions that the Asian took and knowledge of the simulator. Right.


We're able to sort of estimate velocity X, Y and heading. And then we integrate that over time because, you know, your time interval.


So you can come up with estimates of X, Y and heading for the Asian. And that's what you see on this kind of this black diagram on the right, which I can explain everything in more details to you.


You know, but I mean, you build this sort of map almost like this is an overhead map of the agent in its environment annotated with.


First of all, what you've done so far, right, your position that's that's been going on.


Maybe if this here loads this year is different trajectories.


But you also annotate this map with various things that you find like whenever your state classifier says something.


Where is this information used? I guess it's you said it's not in the navigation because that it doesn't get any additional features.


Where is the information that you estimate from from this this overhead map? Where is it used?


The best example for this is to make waterfall task. So when the agent places a waterfall, you know, something we were thinking is maybe we'll try the behavioral cloning.


But often, you know, the behavioral cloning doesn't really stay still very often because they really learned.


Well, the navigation sub policy. So instead, we we sort of use that heading estimation to move the agent away a fixed amount and then rotate around to look at it.


So there are just certain tasks that is really important that whatever the final view is, the line with some landmark in the environment that we don't have a ground truth information for.


Yeah, so it's really the odometry is mainly used in various places in the state classifier.


I mean, start at the state machine in some of the subtasks like David saying. Another example is the animal pen, right?


The challenging part of that task is you really have to build. You first got to find an open location, then build the pen.


And then you have to leave that pen and go find the animal somewhere.


Right. They could be anywhere and then lure them back to the pen. So you have to remember where you built that pen.


And so that that's, you know, the odometry comes into play for that place.


So we were using the state classifier to kind of classify, OK, here's an open location.


Now we switch to pin building mode. OK, the pen is built. Let's go find some animals.


We remember the location of that pen, you know, based on our estimated odometry.


And then once we find some animals, then we try to go back to that location.


And just to say that the try to go back will be a hard coded policy that takes as an input the remembered location of the pen and your guess of where you are in relation to that pen.


Exactly. Yeah. So, yeah, at that stage you have a X, Y coordinate of the pen and you have an X, Y and had the estimates of your position.


Right. So you can basically compute the angle between like where you're looking and where the pen is. You can compute this angle.


Right. And the policy was literally kind of close this angle and then keep moving to kind of reduce this distance over time and go back to that location.


So the simple policy. There are a few limitations, though, on the odometry side, which I just want to comment just to don't say this was like a God dear approach for that.


So, for example, since we only use the actions, right, if you think about it, the odometry is just seeing the actions.


Right. And then, OK, the agent is moving forward. So we've seen this moving forward action.


Right. So we're integrating that over time, increasing the distance and everything.


Right. Well, what if the agent gets stuck like behind the rock, behind the tree and it is still moving forward like in Minecraft?


You can still kind of walk forward sort of sliding. Right. But you're still stuck in place.


But the odometry does not know that. Like we had some some ideas to integrate like different in the pixels.


Right. Using this camera data to know when when the agent is stuck. So we can order that.


But we didn't have time to do that at the end. But this approach, our current approach still works for a short, short distance.


Right. So, of course, the longer you walk, you know, like the the drift will be just higher on this estimation.


But for short distances actually actually works pretty well. And I guess it's sorry.


I was going to say that a slam approach in a 64 by 64 image that's only RGB is incredibly challenging and probably not the right approach for this particular challenge.


And it might also be fair to say that you said you had a lot of ideas.


I guess if you were to go further, you'd probably let's say try to come up with a navigation policy that's both learned but also controllable in some way.


Try to come up with an odometry estimation that takes into account the picture which could recognize when you're stuck and so on.


I think there's there's a lot of stuff to improve, but I'm very impressed by sort of your your your pragmatism of, OK, this works well enough.


Let's go on. Was there was there moments when I guess there's moments in every project when when you're or what was the moment when you most thought, ah, this is not going to work.


Let's give up. Like, did you have a moment like this and what did you do?


You guys want to comment on that? Well, there's there I guess a lot of those moments we if you go back to the main overall diagram, we definitely like had went back and forth on on what should the solution be?


You know, we were still toying around at some points with with, you know, a more, you know, end to end approach in some places and whether we should put our eggs in that basket or whether we should do this kind of approach.


Ultimately, you know, this is the one that we landed on and we we designed this.


The nice thing about this approach is it's hierarchical, but it's very modular. Right. And the idea is that each of these subtasks, you know, their individual modern modules that we can improve upon or replace.


And so, like, you know, if we had more time, some of the things that we would do is start to try to replace some of these hand engineered subtasks with more learning based subtask and or, you know, replace the navigation module with a more advanced learning module that uses more information.


One of the things we spent a lot of time on that never made into or at least was was kind of using generative adversarial imitation learning as our core algorithm for learning the navigation module.


And, you know, with Gale, it's it's basically using again. And as we found out, like everybody knows, GANs are notoriously difficult to stabilize, including GANs for Minecraft.


It didn't ultimately end up making it. We had to revert back. So that was one of our scenarios where we're like, oh, this is this is definitely not going to work. You know, we spent a ton of time doing that and we had to kind of, you know, replace with our with our backup, which is just, you know, standard behavior.


Oh, I think so. Go ahead. Also, the at one point, my brothers are very good at Minecraft and the Minecraft speed running community is a it's a pretty big thing.


So at one point we were considering why don't we just get somebody to play Minecraft really well? But that stupid Minecraft simulator limitation.


And also, you know, it's it's one thing to get a bunch of people to play the game better than maybe the demonstrators were playing. But that also means that, you know, that data won't necessarily be very rich because they can't play the game well and label the data at the same time.


And I think it comes back to this problem of labeling data really conveniently is difficult, especially when you're driving the agent simultaneously. So it becomes a very difficult challenge to use human data when the the amount of data you can actually collect is small.


And this being Minecraft, I think I like I'm fascinated by this because I wonder how much world knowledge is in inside a human when they play Minecraft and how much is sort of learned because the world is different, like literally different every time.


And I can learn Minecraft by just watching someone do it a few times, right? I can perfectly not perfectly, but I can well generalize to other worlds. Is that because I've watched someone I've done it a bunch of times? Or is that because I know from my life what sand is and what water is and how it behaves?


And I think that I don't know. Yeah, I think I guess the main advantage of like, you know, humans is that, you know, we've lived, you know, 20, 30, 70 years already right in the real world.


And then Minecraft tries to mimic that. So we humans have a huge kind of baggage that we can use.


But we have to always remember, like, those Asians, they start from scratch. They literally start from nothing. Right. We had to collect data to teach what danger was for those Asians.


Like, like, had to teach, oh, don't jump in the water, you know, don't don't drown there, you know, things like that. So that is very challenging as well.


And I have I have your sort of so for videos that you upload, and they have side by side the agent view the classifier, but also the odometry estimation.


Do you want to maybe so this is for example, do you have one that is your favorite of these four?


Probably the waterfall, I think, will look pretty nice. So this is the build house was pretty challenging.


This is 30 seconds. I'm going to I'm going to slow it down to like point two five right here.


Do you maybe? Oh, yeah, I can. Oh, yeah. I can like call anyone to comment a little bit on what's happening right here. So which state is it in? What's happening?


Yeah. So so this is a video of the Asian solving the make waterfall task. Right. And then you mainly see in the screen in the screen two panels.


So on the left side, that's the RGB. So this is like a camera view of the Asian. Right. And on the right side, this black panel is the estimated odometry.


So if we start there on top left, you see like action and then a huge tensor. Right. So that's the I think 12 or 13 actions that the Asian was performing.


So they're mostly binaries. So like move forward or not move back or not, you know, things like that.


And below that, you see the raw output of the state classifier. So we had 12 classes or I guess 13 with, you know, the non class.


And you see like the confidence of the classifier, you know, for classifying the state like this camera, this camera image.


So you see like right now, you know, facing wall is pretty much almost 100 percent.


I think it is from all the stone that the Asian is seeing. So it thinks it is a wall. Right. And on the right side, the odometry.


So we can start there on the on the top part there. You see a X, a Y and a heading. So X, Y. So that's the estimated position of the Asian.


So that's not the ground truth. So again, we didn't have the ground truth. Same with the heading.


So that's estimated. And that camera angle there is like a vertical angle. Right. And then on the right side, you have like some time.


So we kind of just have a keep track of time. And then you have a legend. So the legend there is for all the colors you see in the odometry.


So the red one, the red dot is the Asian. So right now it is down at the bottom of the screen.


Whenever the way the Asian walks around, it leaves like this trace. So that's the Y dashed line that you see on the screen.


And then like right now, you see, for example, it just saw that cyan, I think, blob at the bottom there.


That's when the state classifier detect that we are on the top of the waterfall. So you see that.


That's the last thing on the legend there. So basically, yeah, the Asian walks around and some of the relevant states that we classify,


we sort of drop a pin in the map kind of just to keep track of it. So in the video, the first like 25 seconds or so, what this is,


the map, you know, it starts off basically with the navigation policy, right?


So the behavioral cloning module that we trained is is in control and it's driving and it's basically, you know, trying to mimic all of the other human demonstrators that did this task,


you know, which is more or less kind of walk around and look for a good spot. And then when the state classifier detects like, OK, this is a decent spot,


that's when you saw it switch to the, all right, let's build the waterfall. And then after build the waterfall, the state classifier switched to the now go take a picture sub task.


And so that's basically what you see in this video. And one thing I'll say with this, the interesting thing with the navigation policy is


this is something we kind of noticed and it's just a theory. We don't have any proof on it. But like the, you know, the agent jumps around a lot.


But we think that's because the agent is mimicking the human demonstrators.


So like so jumping for the sake of jumping, not necessarily jump over stuff like, you know, there's there's some players faster if you jump.


Yeah, yeah, exactly. And that's seen in the demonstrations or some players like me, I just jump idly, you know, just a fixation.


So I'm just like randomly jumping, not to particularly jump over anything. You kind of see that in the agent's behavior.


So it's almost, you know, makes it more human like, at least in our opinion, versus, you know, a hard coded navigation policy, which mainly, you know,


you might expect it to just walk without jumping unless it needs to jump right over something here.


You know, the agent is kind of just more pseudo randomly jumping like a human would.


And I thought that was pretty cool because, you know, another part of this competition that we haven't talked about yet is not just, you know,


developing agents that can do the task the best, but also there was a sub thread to the competition of who can build the most human like agent,


which we also won that prize. So, you know, this would potentially I mean, really our whole system, you know,


is sort of aims at the human like because we added a lot of human knowledge to it.


But like the behavioral cloning part, you know, that might also add to that because it kind of moves around more or less like it like a human would move around.


And it looks a little less robotic, like if it were kind of a more hand engineered.


Except like here when it's like a good spot for a waterfall, you immediately point down and start like, I guess this is the hard coded part.


You see right now immediately point down, build a bunch of blocks, place the bucket.


And then it's interesting. So this part here is hard coded as well.


It's just like move the agent away. And we see the agent kind of slide on the left a little bit because I've noticed that later when it turns around,


it sort of almost misses a little bit the angle. Right. So this is this could be this drift that you have in the odometry estimation.


So it's trying to make a picture of the waterfall directly misses like a little bit.


So I guess that would be that would sort of be the problems that you get in just having the just having the estimation from the action which you mentioned.


Yeah. So, for example, when you throw the water down, right.


Sometimes the agent will float in the water and that will turn the agent a little bit left and right.


But the odometry doesn't see that because the agent didn't command the camera movement.


So it doesn't update your headings. So that can also cause problems later.


But yeah, like you said, that part was hard coded like the place waterfall sub task was hard coded.


But all the way up to that thing up to that part was learned from human demonstrations, which is the navigation sub task.


What I think what you what you need to do is you just need to train the navigation thing on, you know, dream.


So you just want to you just want to train it on like a bunch of videos of dream and then just see what happens.


I would be so curious to see what happens. Well, that's what we wanted to do that initially is we thought, oh, look at all of this awesome data on YouTube that we could maybe try to learn from.


But there's no actions associated with it. Yes. OK. True. You sort of have to estimate the actions almost a little bit.


And you'd also have to like there's a lot of things you'd have to guess at what's actually going on.


Which where do we crop the video? Right. There's all this stuff they have overlaid and it becomes more challenging to use YouTube data.


But I see. OK. You wait. What was I was I going to?


One thing that yeah, one thing that I was a little bit like a tiny bit dissatisfied with with this competition.


Obviously, it's already super duper challenging. Right. And Minecraft is so much more complicated than this thing.


But there were these four tasks and you knew them ahead of time. Right.


That's why you were able to sort of build the state machine. The descriptions were very clear ahead of time.


Let's say that I come and I'm the organizer and I change the challenge for next year and next year.


It's still the same thing. It's human rated. It's described in just like a simple string.


But I won't tell you what the string is. Right. I won't tell you ahead of time.


How would you how would you go about designing a system like this?


Like what would you would you do? Would you try to go the same route?


Or let's say you also had very limited resources like you had now.


You can't train like a giant RL system.


I think we would definitely be forced to go a different route, which I think would be good.


You know, one of the things I like about this competition, again, is that it's you know, I think it's important for the field because, you know,


it's these tasks again that you can't just, you know, do this black box optimization over because there's no objective function.


So you're forced to really try to learn from a human. Right. Or do something. Right.


And, you know, we really took that to heart. We knew like, OK, in order to do wellness competition,


we cannot just use the human provided demonstrations like the majority of the other teams.


We had to add our own additional human input and feedback.


And we did that with the design of our state machine and in the labeling, the human exhaustive human labeling that we added.


But, you know, to take it a step further, really, I think the interesting thing would be to have a system where you have you learn from real time human feedback,


which our system didn't do because, you know, well, one is that's more challenging and we didn't have time.


And because all the the tasks are known ahead of time, you don't have to have real time human feedback.


You can, you know, collect your human feedback or human labeling beforehand and then use it.


But if you have now a new iteration of this competition where you do not know the the tasks ahead of time,


then you now might need a system where your agent needs to learn from human feedback in real time and kind of interact with the human to kind of get that learning.


Because, you know, you're just seeing what you need to do the task at competition time.


So I think that would be really interesting and that would force more solutions to use something that that uses real time human feedback.


What set you apart if you you probably seen sort of the other teams that competed and so on.


And I'm sure they were also they were also engaged and motivated and tried a bunch of things.


What do you think was sort of the or maybe the most defining factor that let you win?


Was it I'm sure there was a level of stochasticity in the evaluation.


But, you know, you won I think not one, but two of the three subcategories even.


So it must mean that you had a considerable, let's say, edge over most of the competition.


What in your estimation was that?


I have a guess. You guys can comment on that.


I think in my opinion, I think our edge was actually using human feedback data.


So like the other teams, if I remember correctly, I think number two used sort of improved algorithm that would improve on Gale.


So that was kind of sort of full RL approach.


The third team tried to use some of kind of learning from human preference, if you remember that paper.


But they didn't use a human to rate the trajectories. They used like heuristic.


And we were the only team that actually used human data.


So we, you know, we label a bunch of data.


You know, we added kind of our knowledge, our bias on the task and everything.


So I think really using the human, I think, was the key factor that allowed us to win two of three of the awards.


One hundred percent. Like, you know, yeah, we had a state machine approach with, you know, these modular hierarchical design.


We wouldn't have been able to do that if we didn't have, you know, this classifier that was generated with additional human feedback and human labeling.


And so it's really the thing that stood us out.


And like we said, it was, you know, the other teams, they just used the human demonstrations.


And even I think the third place team, they used a simulated human, right?


Instead of doing the hard work of actually getting that human feedback, they just defined this simple heuristic.


And I think that right there is like, you know, the important thing, like the field sometimes can just like, oh, well, let's just it's easier to kind of simulate out the human.


Let's come up with a better algorithm.


But it really just shows like we should do a better job trying to incorporate human feedback because it's definitely, you know, valuable information and can really improve the way we develop our algorithms.


And I think it's important as well to when you look at Minecraft, it's very much feels like an open world sandbox problem, very similar to using a robot in the real world.


And collecting real world data is about as difficult as I would say.


Well, it's a little more challenging in some ways, but challenging to collect lots of good, rich human demonstrations in this particular environment.


And so if we were looking at this as a generalized approach to solving this kind of navigation problem, I think we would have used a similar approach for handling this on a robot where, you know, a robot going to go pick something up somewhere


can be broken down into a bunch of discrete steps and we solve each of those steps really well.


Whereas an end to end approach, we risk having situations where the neural network is doing something that we can't debug at all.


And I think that hierarchical approach really let us debug each step really well, as opposed to the monolithic approach.


Now, just to say on the leaderboard website, there is a team that has like a better score than you.


Is that an artifact of the one leaderboard or is it a late entry after the competition?


So that's the public leaderboard, right? And it's an unofficial leaderboard.


This highlights the other difficulty of this competition.


Like, again, there's nothing to just automatically grade everything. You have to just get volunteers to literally just sit down and look at pairs of videos of different agents and see which one is better.


Very, very arduous task, right? And the public leaderboard is just any random person with a web browser can go on and start rating all the people.


We provided some ratings. It's completely unofficial, but it was just used to kind of determine who would go to the next round.


So the top 10 teams and then the competition organizers actually hired professional contractors, you know, but actually had, you know, not just random people, but like contractors go and do official evaluations to determine the winners.


And on that one, that's where we won first place. But on the public leaderboard, we're not showing us first place because of the stochasticity of all the human raiders.


I love that the professional contractors were probably like they had to know Minecraft, right?


So they're like the most competent people in it were probably like some 13 year olds.


A bunch of kids to watch some videos, give some ratings.


Excellent. Yeah. Is there anything you'd like to? That was my exhaustive list of questions that I had about this.


Is there anything you feel is important to add for people to know if they want to do something like this themselves?


I think during the presentation we had the slide about that. So this competition might happen again next year or I guess this year already, 2022.


So if you're really interested on that, make sure to go ahead and start playing with the mineRL package now because it took us a long time to figure that out.


I think I can speak for all three here. I think that was our first time working with the mineCraft package, like the reinforcement learning package.


So it took us some time to learn all the, you know, how to work with that, their actual space, observation space and everything.


So if you want to like an extra edge this next year, you can maybe start playing with the package now.


And I think that's it. Maybe play a lot of mineCraft. I think that helped. Yeah.


You mentioned the paper that we have, but we also have made our code available for anybody that wants to try it themselves or improve upon our solution.


I think the paper got the link to the code. Yeah, I'm pretty sure it's there. So yeah, go ahead, play with our code, maybe make it better.


Let us know. Maybe make some pull requests.


Cool. Awesome. Well, in this case, thank you so much for being here and sharing this.


It's really I love I like it. I think it's really cool when when things like this get get out into the well, not real world, but mineCraft world, which is close enough.


It's incredibly hard task. And just from the videos I saw, I was surprised by, you know, just how far you can get with how little sort of resources and data.


But it's just one last thing like the definitely, you know, after this first year's competition, the you know, this is far from solved.


And I think the competition organizers realize that, too. So out of the four tasks, which are, you know, that you already mentioned, you know, basically advancing in difficulty, the fine cave and the make waterfall the easiest.


Those are pretty much solved. The create animal pen and especially the build the village.


None of those solutions came even close to really solving that.


You know, I'm sure the human raiders are just looking at two really junk agents doing random stuff and trying to pick which one's better.


Right. But, you know, it's still like on that build village task, but still a very simple task out of the range of tasks that you can conceive in Minecraft is still far from from solving.


And I mean, yeah, there's there's no crafting yet. There is no fighting. There is no exploring. And this isn't even like this. This is where Minecraft starts.


The actual game of Minecraft is where you sort of set your own goals. Right. And you try to achieve something new.


Yeah, it's it's cool to see that there's still a lot of a lot of stuff to do. Awesome.


Thank you so much for being here. And yeah, I hope to see you next year again.


Thank you very much for having us, Yannick. Like I said, I watch a bunch of your videos. I really like your channel. I'm excited to see.


Hey there, it's Yannick. I'm going to leave you with the submissions of the team to the competition that were actually judged by the human annotators.


So you can see what the human saw and what it takes to win such a competition.


We'll show you all the submissions for each of the tasks in parallel. Let me know if you like this video.


Leave a like if you did and leave a comment if you have comments, suggestions, anything at all. See you next time.


Oh, oh, oh, oh.


Oh, oh, oh, oh, oh.


Inconsistency in Conference Peer Review: Revisiting the 2014 NeurIPS Experiment (Paper Explained)

Hi there! Today we'll look at inconsistency in conference peer review


revisiting the 2014 NeurIPS experiment by Corinna Cortez and Neil D. Lawrence


which were actually the chairs of the 2014 NeurIPS conference. So they are


going to have access to some data that the rest of us sadly don't have access


to but also it allows them to make pretty cool research on how conference


reviewing works and whether or not it actually can determine the quality of a


paper or how much of it is just random subjective reviewer decisions. Now this


paper particularly here takes up the papers that were subject to the 2014


NeurIPS experiment and tracks them over time. So it looks at the


papers that were submitted, how did they perform in the subsequent years, meaning


how many citations that they accumulate both for the accepted and for the


rejected papers and they find some pretty interesting results right here. So


we'll dive into this. The paper is not too long and the conclusions are fairly


straightforward. I still think it's really cool that people actually follow


up on this work. So for those of you who don't know the 2014 NeurIPS experiment


that is the wrong color. The 2014 NeurIPS experiment was an experiment in


assessing how much of review of conference review is random essentially.


So what you did was and I think they have a little section about this here


yeah so they selected about 10% of the submissions these were 170 papers and


these would undergo review by two separate committees. So whereas usually


you have a paper that goes into a review let's call that a committee which is a


bunch of reviewers and an area chair and they make the decisions of whether to


accept or to reject and yeah at the end you have a decision. So in this


experiment you would take a paper you would actually give it to two different


committees committee one and committee two. Committee one would only be selected


from kind of one half of the reviewer pool and committee two would only be


selected from the other half. These were random assignments and to the two


pools and also the papers who participated were randomly selected. So


each of these committees would reach their own decision accept or reject and


of course the interesting part is how many of those agree or how many of those


disagree with each other and by the way the paper would be accepted finally if


the max so if either of the committees would accept the paper and if I recall


correctly this year's NeurIPS conference actually repeats that experiment from


2014. So we're going to have another data point in hopefully assessing how


conference reviewing has developed over the years whether it's gotten better or


actually worse. Alright so that was the experiment in 2014 but by the way the


authors here have decided that the name change is retroactive. I never know I


never know when talking about old NeurIPS conferences whether I'm supposed


to say it was NIPS 2014 or NeurIPS in any case in this in this paper we're


doing we're doing NeurIPS. So what was the outcome of that experiment and


that's pretty interesting namely here you can see these are still 2014


numbers committee one and committee two split up so it's not the same committee


one of course but committee one would always be reviewers selected from kind


of the first half of the population committee two from the second half. They


did agree on most of the papers as you can see here for 101 papers they agreed


to reject for 22 they agreed to accept however for 43 of the papers one


committee would accept and the other one would actually reject. So for about 25%


of the papers the two committees would disagree. 25% it's you know it sounds


it's a lot but it doesn't sound like that much but if you look at it in a


different way where they say right here if the conference reviewing had been run


with a different committee only half of the papers presented at the conference


would have been the same. So this is looking at if you for example always go


with committee one you would have these papers but if you would always go with


committee two you would have these papers therefore but the simple


selection of the committee determines about half the papers at the conference


so if you're at the conference you walk through the the big halls of posters or


you look at the proceedings you you have to keep in mind that half of the papers


are there only purely because of the random choice of or not purely but they


wouldn't be there had the reviewing committee been a different one. Half the


papers that's kind of crazy and of course this sparked a lot of discussion


right here. So this is the outset this was the results from that time and now


we're going into new analysis. So they do three different distinct points of


analyses. The first one is they do the title is called reviewer calibration. So


they try to figure out what portion of a reviewers assessment of a paper is let's


say objective and what portion is subjective. So what portion of a score is


simply due to the reviewers subjective feelings about the paper that doesn't


match with any other reviewers scores. So here you can see this for example what