If you don't know, gradio makes it really easy to build like simple interfaces to your


[ML News] AI learns to search the Internet | Drawings come to life | New ML journal launches

model, you don't need to code a lot super easy to get a text box running where people


can enter a bunch of text or an image uploader.


So people can interact with computer vision models.


It's also super easy to host that in the cloud back it with a GPU and a lot of the demos


these days are done via gradio.


It's even simpler than a collab.


So it seems hugging faces ever becoming more powerful.


I mean, it's pretty cool for now.


But can you imagine if hugging face will be like, you know, the dystopian overlord company


at some point, you know, for Google or Microsoft, you can imagine it their logo is kind of,


you know, like the Google logo is colorful, but you can definitely imagine it in like


a dystopian setting where you know, everything's controlled by them and so on.


But you know, hugging, hugging face, you know, as you are beaten down and imprisoned for


thought crime, you'll just you'll see that.


I'm not sure if they've branded themselves into a corner right here, but it would be


an interesting future.


Please make it happen.


Alright, some helpful things for this week.


MinDali is code base and checkpoint that is named after min GPT.


It is a 1.3 billion text to image generation model trained on 14 million text image pairs.


Now as far as I understand it, this is not to be mixed up with Dali mini, which is another


project that attempts to reproduce Dali.


Dali mini is quite a bit older and more advanced if I see this correctly, but cool that both


exist.


MinDali releases version three of Arnheim, which is a generative art model that uses


neural visual grammars.


I've reported on this previously, this is essentially a model that doesn't just generate


the images pixel by pixel, but has a neural grammar like you need to do paint strokes,


or you need to place objects or something like this.


And this gives for pretty interesting generative art.


So version three is out, you can make collages and anything like this, check it out.


This is a new benchmark called the document understanding benchmark where the goal is


to understand documents not only in their textual content, but also in their layout,


there can be tables in documents, there can be what type is the document, there can be


our two documents of the same type, where's the document from all kinds of stuff.


There's GitHub org to go along with it, including a JSON schema, an evaluator and some baselines.


There's also a NeurIPS paper, check it out if you're interested.


NeurIPSy is a benchmark for question answering with long input text comma yes.


So there's also a paper to go along with this.


And this is a multiple choice QA data set with context passages in English that have


an average length of about 5000 tokens.


So this is much longer than typically current models can process the paper rights.


So if you want to compete here, you have to be a little bit tricky.


Perceiver IO is now in the Hugging Face hub, I believe I've made a video about Perceiver


IO maybe not actually remember if it wasn't Perceiver IO or the original Perceiver.


But in any case, this is a multimodal attention model that can ingest essentially any data.


I love how this block here just says self attention, self attention, self attention,


self attention, self attention.


Try saying self attention a bunch of times in a row.


I mean, is this what five times self attention and then n times five times self attention.


This new paper called self attention does not need of n squared memory by Google Research


presents an algorithm for attention and an extension for self attention that does not


require the old n squared memory that everyone claims.


So the algorithm is here depicted in these formulas, it essentially notes that you can


pull out the normalization of the softmax out until the end until after you've multiplied


with the value matrix.


And therefore, you can trade off the n squared memory requirement for doing it all in parallel


with an iterative algorithm that uses less memory.


If you're interested, check out paper.


Michael Bronstein has a cool blog post called deriving convolution from first principles.


So in this he goes through what a convolution is and how you can represent it as a circulant


matrix.


But not only that, he shows that if you want an operator that is naturally shift invariant,


and you view this through the lens of the circulant matrices, and what happens if you


shift them around,


if you want an operator like this, then naturally, it has to be the convolution operator.


It's pretty cool.


It draws on some fundamental math and Fourier transforms enter the picture.


So if you're interested, I definitely invite you to check it out.


And it is also a very good gateway into the entire literature of equivariant deep learning,


of course, of which Michael Bronstein is an expert in the Google AI blog has an entry


on training machine learning models more efficiently with data set distillation, I believe I've


previously also made a video on this, but now there is a blog post about it.


And I think more importantly, the distilled data sets have been released, you don't know


what this is, this is essentially you want to train a classifier with as little data


as possible.


However, you get to make the data.


So you try to sort of make kind of adversarial examples or uber super prototypes of data


so that the classifier can learn from as little data as possible.


Here you see a CIFAR 10 distilled into just 10 images.


So you have one single image per class.


So you see at the top, you simply try to select the best images from each class.


And that will give you a final test accuracy of 16.3%.


Again, this is the entire data set.


But if your entire data set is this crafted data set at the bottom, again, only 10 images,


you will get a test set accuracy of 50%, which is pretty respectable for only having 10 images


to train on.


Again, there are papers to go along with it.


But there are also now the data sets available online.


Hebo is a library for Bayesian optimization released by Huawei.


So this was the winning submission to the NeurIPS 2020 black box optimization challenge.


So if you're into this field, and you're looking for a very, very performant library, maybe


this is it.


Ruda Lee has released their big model we've previously reported on a Ruda Lee, which is


a Russian version of Dali.


And they have released their small model previously.


However, now they are releasing their big model, but they don't release the weights or anything


like this.


Of course, as everyone else, they release it via an API.


So you can call the API and you'll get a bunch of outputs.


So here you can see chic living room with green armchairs by the window.


This is by the way, this is Google translated the model is in Russian, or you can see a


bunch of other images, they do look awfully like cut out a lot of them look they have


super sharp edges for some reason.


It's really interesting.


And the humans all of which have slightly weird faces is pretty impressive from Dali


model.


We've previously announced the net hack challenge and the report is now out the results of the


net hack 2021 challenge at NeurIPS are out and it turns out that symbolic methods are


still better than neural methods.


But the neural methods are also advancing pretty quickly.


So in gray, you see last year's baseline and you see the progress that has been made.


For those of you who don't know the net hack challenge is a reinforcement learning challenge


adapted from the net hack game, which is very fast to simulate because it's only ASCII based,


but you can render it in a pretty way like this.


It has a procedurally generated levels and is known for being very, very, very, very,


very complicated.


So the challenge has finished, but the environment is still up.


So if you want to give it a try, you know, go for it.


Lastly, MIT News writes characters for good created by artificial intelligence.


So this is a piece that initially features here a picture of Albert Einstein being brought


to life.


So check this out here.


Here's Albert.


This is just uber.


This is uber creepy.


No, this is just mega creepy.


Yeah, well, I guess the the idea is more that you get inspired for what's going to be possible


in the future.


The article takes a surprisingly positive view on sort of digital characters and virtual


characters and will people be able to sort of lend their appearance to things?


Can you make psychotherapy more accessible to people with mental health issues and so


on, which is surprising because usually these articles all have sort of a negative slant


in them.


And of course, there is a paragraph about legal and ethical challenges, which obviously


no one wants to deny.


But it's good to see other people also being a little bit more optimistic about the future


like you know, look at all the cool things we could do with such technologies.


Now whether or not all these benefits will materialize like whether or not it really


matters that Albert Einstein explained something to you.


I'm not entirely sure but it's a neat short article if you're interested, check it out.


And this was already it for ML news.


Thank you so much.


Remember to stay hydrated.


It's always best to do so from a Weights and Biases cup.


Thanks so much again to Weights and Biases for sponsoring this video and I'll see you


next time.


Context R-CNN: Long Term Temporal Context for Per-Camera Object Detection (Paper Explained)

Hi there, today we'll look at Context R-CNN, Long Term Temporal Context for Per Camera


Object Detection by Sarah Beery, Guan Hong Wu, Vivek Rathod, Ronny Votel and Jonathan


Huang. So on a high level, this paper tries to do object detection for cameras where you


the camera is in the same place for a long time. For example, these wild trap cameras


or traffic cameras right here. It proposes to do object detection by incorporating data


from the images that the camera has seen in the past to help the detection in the current


frame. And it does so via an attention mechanism that it runs over a memory of past data. So


we're going to take a look at how this is done and how well it works. And yes, stick


around if you want to know. As always, if you enjoy content like this, then consider


sharing it out telling your friends about it. Subscribe if you haven't and tell me what


you think in the comments. So the paper starts off and describes the problem and the problem


is fairly simple. You want to do object detection in images. Object detection is the task of


basically if you if I give you an image, you should tell me what is on the image and where.


So in this case, here, you would have to draw me this bounding box and say, this is a deer.


On the bottom, you would have to draw bounding boxes. Maybe they have to be rectangular,


maybe not and say this is a bus. And here is a truck and here is another truck and here


is a car and so on. So there can be many objects in an image, there can be one object, there


can be objects of different classes, or there can be no objects at all. So this is just


object detection. And there have been many papers on this. And specifically, there has


been this rCNN. And this is the model that we're going to extend. So the rCNN model or


specifically the faster rCNN model that we're going to build on is a model that simply detects


these bounding boxes in single in single images. But now we consider the situation where we


have a camera that records images for a long, long time. So in these wild trap cameras,


they often sit there for months. And it's not that easy to make use of them. Because


in addition to there being a lot of data, they have motion triggers. So that could be


there is no nothing for a long time. And then there's a the the animal walks in the trap.


And then you have a bunch of images, like one per second for 10 seconds. And then you


have nothing again for like a day or two days. And then you have 10 images again, because


another animal walks in or maybe doesn't, and so on and another. So you have irregular


sampling frequencies, you have very, very different distance between the frames. All


of this makes it very, very not suited for models like temporal convolutions, or things


like LSTMs, because they don't work super well with data like this. Now, I know there


are formulations where LSTMs can do this, but they don't work very well with these super


long contexts and irregular sampling frequencies and so on. So the idea is if we have a frame


right here, like this one, and we want to detect what's on it, we should be able to


pull information from other frames that the same camera has seen, like from this one,


or from this one, or from this one right here. And we should be able to do so in a dynamic


way. Now, why could that help? If you look at, for example, down here, these images have


been taken, they say images were taken on separate days. But you can see this thing


right here is in both images. So or a very similar thing is probably that bus's regular


route. So in in order to classify if whether or not this here is a bus, it might be very


helpful to also look at this picture right here and see, ah, you know, it's about at


the same location, it looks the same. And also it looks like a bus. So, you know, that


that kind of gives evidence that this could be this other thing could also be a bus. Then


also there are background objects. So sometimes the single frame detectors get confused, it


might be labeling this here as a car, because just the lighting the exact lighting in this


picture is just off by the correct amount that it is confused. But considering this


picture over here, maybe it recognizes here, no, that's not a car. And it can bring over


this evidence to that frame and, and consider, ah, maybe, maybe, you know, this is the same


thing. So it's it's not a car. So this is this is not the same than simply adding training


data, we really consider the fact here that these images, they come from the same camera


that is in the same location, or maybe, you know, that is filming the same thing. So this


all of this is going to be within the same camera, not just adding adding, ah, I ID training


data. And with animals as well, like, often the same animal has like its regular route,


or within these 10, these bursts of 10s, the same as animal will kind of walk around a


bit and maybe, you know, here, it's half occluded, but maybe in a different image, you see, ah,


here I see the nose. So it helps you make a better prediction. Also, animals are often


in kind of crowds in and that helps if you see that there are other deer around the probability


that this is a deer increases rapidly. So how are we going to do this, what we're going


to do is we're going to build an attention mechanism that can do these kinds of look


into the past, and some also a little bit of the future, as we will see, but mainly


will look into other images from the same camera in a dynamic way. And we'll learn how


to address those other images from a memory bank. So the architecture is described right


here. Now, as you can see, we are still in the business of doing object detection. So


what we'll do is we'll sort of hijack a existing object detector. And the object detector we're


going to hijack is going to be this FRCNN, this faster R-CNN object detector. That's


an object detector for a single frame problem. So that means you have one image, and you're


supposed to detect what's on it. It has two stages, as you can see. So stage one, if you


have an image, and let's say there's some stuff on it, stuff, stuff, stuff, there's


stuff, okay. This is what stage one is supposed to do is it's supposed to extract regions


of interest. This could be, okay, all of these are regions of interest. So it simply says,


well, there is something, there might be something right here in these regions of interest. And


then it describes each of these regions of interest using features. Okay, so it extracts


these regions of interest, and each region of interest gets features assigned to it.


So what these are, I think these are like seven by seven by 2048 features. But let's


just say for the sake of describing it, that these are just a vector of features for each


region of interest. So each region of interest is going to be associated with one vector


of features that this model extracts. Okay, and the next region of interest also has a


vector and the next region of interest also has a vector and so on. Stage two then takes


each one of these takes each one of these vectors and assigns a class to it. So this


would be deer right here. Okay, so stage one proposes regions of interest along with features,


then stage two takes each of these regions of interest and classifies them basically.


And I guess there's there's many in between stage like this is massively simplified. There's


no maximum suppression, there is kind of an alignment stage where you can refine the the


bounding box and so on. But in essence, these are two stages. And you can see that this


system here, it goes in between the two stages. So all of this right here, we shove in between


the two stages. So we'll still use the stage one, and we'll still use the stage two. But


in between in this thing right here, we'll try to sort of pimp these features such that


the stage two detector has an easier time classifying. Okay, so we're going to pimp


these features by incorporating in because these features right now, if we just do it


with vanilla, these are just from the current frame. And we're going to add to them information


from other frames of the same camera. And we're going to do it in two different ways.


So the first way, as you can see here, the first way is this short term memory. And the


second way is the long term memory. Now, the two are slightly different. As you can guess,


the short term memory is going to be only over a short time period around the current


frame. And the long term memory is going to be basically across a very long time horizon


into the past. You can see we're trying to classify this blue frame right here that what


we call the key frame. So what we'll do is we'll run it through stage one, cool. So we


have features for each region of interest. And then you can see this goes here and through


these residual connections, this goes into stage two over here. So basically, stage two


still receives the same input, it receives whatever stage one outputs for the keyframe.


But we're going to add to that twice. So we're going to add two things as I as I said, so


the short term memory is added right here. Now, how do we build the short term memory,


we build the short term memory simply by considering all the frames around the keyframe. And this


you can see right here, the current window around the keyframe, which can be like one


frame around it, or two frames or three frames, just a few frames around the current frame.


And this can be fairly helpful, as we said, for example, if the deer moves a bit, the


car moves a bit, you know, it gets into a slightly different lighting and so on. This


can help us very much to classify the current keyframe if we also have features from the


surrounding frames. So for each of these surrounding frames, we also run them through the stage


one detector to also extract regions of interest. And that all of these features go into this


memory, short term memory bank right here. There's different strategies, you don't always


have to extract all of the regions of interest, you can also extract just the top one and


so on. Or you can extract the mean, since these are fairly, you know, consistent, the


cameras at the same place. There are many ways you can do this. But what you ultimately


end up with is a short term memory bank that kind of is, so you'll have a bank and you


have lots of these feature vectors in here for your region, your regions of interest


of the surrounding frames. Now, if this here, if this here is your half occluded deer, right,


so this is the half occluded deer, and you want to consider information from the surrounding


frames, maybe in the next frame, so maybe this is three frames, like 123. And two is


the key frame, maybe in the next frame, the deer moves a bit, and you see its nose. And


that, this particular region of interest here is relevant. So how do you know, how do you


now get from this entire memory, this feature vector that would be helpful? And the answer


is you get it through an attention mechanism. You can see that right here, the way the short


term memory is added is through this attention block. They describe the attention block right


here, it is a fairly standard attention mechanism. So I've done a video on attention is all you


need. If you don't know what an attention mechanism is, go check it out. But you can


see it's it's very, very standard. So you have these input features, which are the features


that come from the key frame. And you have the context features, which are all the features


in your memory bank, you encode the input features with into a query using a fully connected


layers and the context features into keys. And then you match the queries with the keys


and the softmax in order to get a weighting over the context features. And then you aggregate


the values from the context features. So this is a standard attention mechanism. What does


it mean? It basically means that each of these vectors right here, they will emit a key that


kind of describes what kind of information is contained in that vector. The vector over


here will emit a query that describes what sort of information it is looking for to in


order to describe what's in the region of interest as well as possible. And then you


simply match the query with the keys to determine which key fits best to that query. And whichever


one fits best, let's say this one here, then you take that vector from the memory bank


and incorporate it together with your current information that you already have. So that's


how you address things that from other frames using an attention mechanism. Okay, now, if


this were all, you know, we could train this right now. We could train all of this because


all of this is differentiable, right? This stage one detector right here is differentiable.


It goes here and here, you know, the information, the attention mechanism is differentiable.


The stage two detector is differentiable, all differentiable. Cool. We can train this


end to end. Now what's the problem? The problem is this long term memory right here. So in


this memory, ideally, we would want to fit, let's say an entire day, an entire week, or


even an entire month of data from one of these cameras. And it's just not feasible that we


expand this current window here to an entire month or an entire week. For many of those


of those cameras, because even though they have a low frame rate, and so on, it's still


too much in order to then be all differentiable, all back propagatable, and so on. So we can't


really back prop in for these long term memory. In essence, what we want to do is exactly


the same, we want to build up a memory of all of the regions of interest, or maybe selected


regions or all of the best regions of interest, whatever heuristic strategy we have, of the


past, whatever this camera has seen, let's say in the last month or in the current week,


or something like this, we want to build all of this up, and then use an attention mechanism,


just the same in order to incorporate it. But we, we have to come up with these things


right here in some other way, than a way where we can back prop. So we can't really use this


stage one detector right here, because this is this is the one we're training. And so


we have to back prop through it. Now, an easy proposal is to simply use it anyway, but do


like a stop gradient on it. So we don't back prop through it. That is one way but this,


the paper decides on a different way, the paper decides that all of the all of the past,


basically, right here, right here, and so on. We'll take a pre trained object detector.


So not the one we're training currently, but we'll take take a pre trained one that was


pre trained either on something like cocoa, which is an object detection data set, or


you can pre train it on cocoa and then fine tune it on a task you're interested in in


a single frame fashion, in what for whatever way we'll take a pre trained object detector,


or region of interest extractor. And that will give us for each frame in the past will


give us also the regions of interest, along with the features. Okay. And these are the


features that we then go and put into put into the memory bank. Sorry, my tablet just


crashed a bit. There we go. Okay, so we'll take a pre trained extractor right here. That


will give us features for regions of interest, we'll put that into the memory bank. And then


we will use an attention mechanism to incorporate it. Now the attention mechanism we can train,


but we cannot train the extractor for the features. And this is the difference to the


short term memory where we can actually train the feature extractor in order to help us


with you know, building the memory. Now the memory is simply built without a goal in mind,


basically. And the attention mechanism, the attention mechanism basically has to learn


that it doesn't work with features that are meant for its task, it works with features


that have been originally created for a different tasks, and they're not going to change. But


as we'll see, this, you know, can be handled. So that's what they do, they incorporate short


term and long term memory into their stage two prediction, and then the stage two prediction


simply takes in all of those features and classifies the class of the object. And that's


the architecture of context or CNN, it's our CNN with long and short term context. So they


describe very different ways of you know, how they built the memory and so on how they


built the features. I didn't I kind of glossed over this right now. There's a lot of consideration


in building these thing and you have to look at the paper, how they exactly do this. I'm


more interested in the high level architecture and the sort of ideas behind it. So when they


do this, they do outperform the, they do outperform the current or the single frame baselines


by quite a bit. So this SS and this CCT are these wildlife data sets, whereas this CC,


I think this is the city something city cam, these are this street data set. As you can see, they


do outperform the single frame baseline by quite a bit. Now interesting, as you can see


right here, as they increase the time horizon of this long term memory, so they, they can


all they can now choose how much information do they want to put in that long term memory,


as they increase the time horizon for one minute, one hour, one day, and so on, the


performance goes up and up and up, which is, you know, a strong indication that these features


actually help from the from the time horizon, because you don't have more parameters, you


simply increase the amount of information in the memory bank. And if the performance


goes up, you can make a very strong claim that these these this is actually due to the


fact that you have more information in that memory bank couldn't really guess any other


explanation right here. So they they do it, they do investigate different memory strategies,


they do a lot of ablations right here, where they also say, Okay, what if we only have


the short term attention? What if we only have the long term attention? What do only


if we only have self attention, that means attention only into the current frame, but


of across regions of interest, that's interesting, if you have like a herd of animals and so


on, and they all help. But as you can see, the long term attention tends to help the


most in this data set. And short term attention help helps a lot in this data set, if you


just compare to the other owner, these are two different metrics, not data sets, sorry


about that. But in essence, it helps the most when you combine the two. And that's, you


know, that's pretty cool to see. So they do some qualitative results, which I find very


interesting. For example, they can visualize what the attention weights of their models


are. So here, you always have a very long timeframe, I think an entire month in this


in this memory bank of the long term memory. Now, in the top classification, you see the


large thing here, the large frame is the one you actually want to classify. And the other


frames are the frames where the top attention score is so that the attention weights are


the highest. So here, in order to classify this, what does the model pay attention to?


Or which other frames does the model pay attention to? And you can see right here, they are all


spread across the entire month, here is the timeline, the most attended two pictures are


spread across the entire month, and almost all of them actually have that war talk in


here. So this must be like its regular route. And the model recognizes that and pulls in


information from all these other images in order to correctly classify it here. On the


other hand, on the next example, this gazelle and tablet crashed right here. It also puts


all the all the weight on top of the of images of that same gazelle. But you can see maybe


that gazelle was only there for this one particular moment. And all the pictures this camera has


of it is, you know, in the very few moments that the gazelle was around, you can see they


all come here from the same point in time, or very, very close points in time. And you


can see that it puts a lot of weight on wherever the gazelle is. So, you know, that's a pretty


strong indication that it actually learns to pull in the correct information be that


from long time horizon, or from a short time horizon, if necessary. You can also see right


here, they visualize the top attention, where the top attention weights go in terms of how


long the frames where the attention goes to is away from the frame that they're trying


to classify. So these graphics are somewhat kind of weird to interpret. This here always


means how much is the total time of the buffer. So the memory buffer here contains always


pictures from the total from one hour before, until one hour after the keyframe you want


to classify. So this is the frame you want to classify at minute zero. And the memory


buffer contains images from 60 minutes before to 60 minutes after. So it's not real time,


right? You go back to your through your footage and you try to classify. So you can also pull


out images from the future. You can see there's most attention is on the current frame, which


makes sense. You're trying to classify the current frame, and it kind of falls off as


you go further and further away. And this is across the entire data set. So this is


not a specific example, which also makes sense. Probably in most of the time, the relevant


information is, you know, next closer in time rather than farther away. But also you can


see that the distribution is pretty spread out. So it makes the model makes use of the


entire range of time. And you can see that throughout even if you have an entire day


in the buffer, or two days, even if you have entire week before and week after in the buffer,


and even if you have an entire month here. And especially if you look at when you have


an entire week in the buffer, you can see the periodicity through the days. So that


means the model tends to pay attention to images that are from the same time of day


compared to the current keyframe. That's a fairly, fairly good indication that the model


has actually learned to address these this memory by its content, right? Now, night and


day isn't super difficult, because you can just go on the brightness and so on. But still,


it's pretty cool to see that this is actually happening. They do have some failure cases


of the single frame model that their model is able to handle up here. And they make a


lot of sense. So here you can see that there is an object that's moving out of frame. And


the single frame detector wasn't able to recognize this, probably because it's moving out of


frame. Whereas this new this context RCNN is able to detect it probably because it looked


at the frame just before it, where the car was somewhere back here. And it could correctly


classify it. Well, that's, well, just disregard my drawings. Here, it managed to recognize


this animal in the back, whereas this old model, the single frame model hasn't also


probably by looking either at frames next to it or by looking at other frames of herds


of animals and realizing that usually when there's two elephants, there's more. Here,


you can see that the object highly occluded. So we're talking about the object like at


the very edge of the frame object poorly lit, this is particularly impressive. And also


an example where the animals are often in herds. And if you see one deer, the likelihood


that there's other deer is very high in this particular camera. And by aggregating information


from different frames, you can see that maybe it's always the same patch of deer that comes


by. And here, the single frame detector detects this patch here as a vehicle where it shouldn't


and of course, the new model, the Context RCNN is able to recognize that this is present


in all of the frames. And in most frames, the single object detector doesn't detect


it as a vehicle. And so it can kind of carry over that information. Now, you can already


see sort of what the downsides might be if the single object detector is like very, very,


very sure that this is in in a single frame that this is a car, it could carry over that


information to the other frame. So even though the single frame detector might have failed


in that particular frame, if it fails super hard, it might, you know, shout that to all


the other frames basically dominate the memory saying like, look, this is a car, I'm like


pretty sure. And it will carry over that information to all of the other frames. And they say,


in one of the in one of these high confidence mistakes, it basically detected the same tree


as a giraffe over and over again. What I find particularly interesting is they do look at


so here they have this this curve of on the bottom, you have confidence threshold, so


how confident the model is. And on the y axis, you have the number of false positives. And


you can see that in the low confidence regime, the context or CNN has lower false positives


than the single frame detector. And the green line here is when you only have positive boxes.


So when you only include regions of interest where there is an actual object, which, in


this case is sort of hurtful, you also want the regions of interest where there is nothing,


because that helps you avoid false positives in other frames. That's why the orange line


is below the green line. But strangely, here in the high confidence regime, you can see


that the single frame model has fewer false positives than the context RCN. And I like


the text that they have to this. In figure seven, we can see that adding empty representations


reduces the number of false positives across all confidence threshold compared to the same


model with only positive representations. We investigated the 100 highest confidence


false positives from context RCN and found that in almost all of them in 97 out of 100,


the model had correctly found and classified animals that were missed by human annotators.


So basically, these, these graphs are even underestimating how good that model is because


the model is seems appears to be better than the human annotators of the test set. I find


that I find that to be pretty, pretty impressive. And here you can see failure modes, where


they say, for example, when exploring the confident false positives on the snapshot


Serengeti data set, yada, yada, yada, the three out of 100 images, so whatever was not


a fail human failure, where, where context RCN and erroneously detected an animal where


we're all of the same tree, highly confidently predicted to be a giraffe. So this is a failure


mode when, when the model is highly confident, it might spill that over to other frames,


because the we now aggregate the information within the same camera across the frames.


To be said, of course, their train test split is such that there's not the same camera in


the training data as in the testing data, they have different, different, entirely different


cameras in the testing data than in the training data, just so there is no information leakage.


Yeah, so that is the that's the model right here. How it works, it's it's pretty cool.


It kind of wedges itself in between any single frame object detector that has these two stages.


And you know, it's a pretty neat idea to bring in context from from the past or even the


future of the same camera. Just a quick glance at the appendix, they have lots of different


examples right here. In one example, their camera kind of fell over and they say, well,


it still worked the camera with the system was still able to kind of do attention across


this failure, this kind of tipping over of the camera. They have more examples right


here, which I find pretty impressive, like the super low light things where it correctly


detects like the possum. And yeah, I invite you to check out the paper, the code they


Research Conference ICML drops their acceptance rate | Area Chairs instructed to be more picky

Good morning, I hope you had a good night's sleep. It's just another day where the review system in


machine learning is completely and utterly broken this time courtesy of the ICML chairs, apparently


notifying the senior area chairs to reduce the number of accepted submissions by about 10%.


According to current meta review statistics, we need to raise the acceptance bar also saying we


plan to reduce the number of accepted papers, please work with your senior area chair to raise


the bar area chairs and senior area chairs do not have to accept a paper only because there is


nothing wrong with it. So the ICML conference is trying to raise the bar on scientific publication


in their venue by just accepting a little bit less papers than they would do according to


current trajectory of the review process. ICML currently is in the post review post rebuttal


process where the actual acceptance decisions are made. Now, why is this important? This is


important because there are only about three or four large conferences in machine learning each


year depending on your subfield bit more or even a bit less. For many places, if you want to get a


PhD, if you want to get tenure, if you want to achieve anything in academia, you need to publish


papers at those venues. And given that the field is exploding currently getting a paper there is


quite difficult. acceptance rates have been dropping steadily in the past few years, though


you can see the number of accepted papers has actually risen. This is a consequence of the


exponential growth of the machine learning field. Now there's a growing concern that the review


process isn't really good. And what gets published and what doesn't get published is just kind of a


wash and the noisy process which is true. I've made quite a number of videos about the really


flawed review process in machine learning. Essentially, here is what we know. If your paper


is really good, then it's going to get accepted very probably you might get unlucky, but with high


probability, it's going to get there. If your paper is really bad, also with a high probability,


it's going to get rejected. However, for most papers, which aren't extremely good, which aren't


extremely bad, there's just this middle area, most papers fall into this middle area. And it's really


a roll of a dice, you get some reviewers, they might know what they're talking about, they might


not know what they're talking about, they have their favorite data set, you didn't evaluate on


it, they reject or they weak except because they just don't want to deal with your rebuttal. It's


an all around fun process, but it can ruin your life. And for a conference such as ICML, it is


important that it keeps up its reputation for only publishing the best papers and really good


scientific results. So by reducing the acceptance rate, what they'll do is they'll put more focus


on the really good papers that stand out, which can be interpreted as a good thing. Because


ultimately, the really good papers will still stay while some of the borderline papers will drop out,


that gives you a stronger signal that whatever comes from this conference is a valuable


scientific publication. On the other hand, you can say given how noisy that review process is,


you simply compress a little bit the amount of people that draw a lucky lottery ticket. And


given that the field is growing, and there is huge pressure on people to publish, and also the fact


that large corporations throw extreme amounts of money of getting papers published at these


conferences, weeding out the academics that don't have as much resources, it is a bit of a


controversial decision. Essentially, reviewers and area chairs are even more incentivized to


just find anything wrong with a paper and rejected because of it. And the downside of that is that


if you don't have as much resources to train on every data set, you're probably going to be out


much more likely. And also if you have some really cool idea that just doesn't work yet quite well


doesn't beat state of the art yet, but is quite interesting. Also very probably you're not going


to get there. So while the optimist might see a stronger signal for an acceptance rating at that


conference, and just higher quality output, and the pessimist might see the noisy process and say,


Well, what is it all worth? It doesn't mean anything to get accepted anyway. And now it's


just less papers that do and also large companies are going to dominate the field. And also academics


are going to draw the short stick. The optimist and the pessimist are no match for the PhD student.


See what they seem to be doing right here is specify the acceptance their target in percent,


which means number of accepted papers divided by number of submitted papers. I hope you see


where this is going. Your target acceptance rate in the eyes of the conference means that the


numerator should be smaller. However, you can reach that same acceptance rate by just making


the denominator larger. Now hypothetically, if just everyone would submit more papers,


we could drop the acceptance rate, but also raise the chances that our actual papers are going to


get in. Now, in this hypothetical scenario, I would not be advocating for submitting fake papers or


just empty PDFs. But you might have some papers in the drawer, like this duty right here that I wrote


back in I don't know when, where I designed a method to defend against black box model theft


attacks, which I thought was pretty smart. But honestly, it needs a lot of work to actually


make it work. And I just did not bother. It's an archive right now. But even though I am not happy


with it, as it is, it is certainly better than a lot of stuff that I've seen submitted to ICML,


that I've read as a reviewer, and even some stuff that actually got accepted at the end. So compared


to that, I don't see a reason why this should not be worthy. So you my friend are going to ICML


next year. How about that? Of course, all just a hypothetical. I'm not advocating for you to


mess with a system that's clearly broken and needs to be renewed. And we should reinvent


the whole thing. However, it's fun to think about. If you have some thoughts on hypothetical


scenarios or stories about how your papers got rejected that we all love to tell,


Author Interview: SayCan - Do As I Can, Not As I Say: Grounding Language in Robotic Affordances

So today we're here with three of the authors of this paper with I have to say a lot of authors. It seems like a giant work just from what I could gather from the from the paper itself and the data collection and the evaluation and so on. So this was a huge thing, but the results are pretty cool. So here with me today are Fei Xia, Brian Ickter and Karol Hausmann, who are three of the authors of this work. Welcome to the channel, everyone.


Thanks.


Thank you for having us.


It's great to have you here. The I like I love the title, because it's a bit of a mantra on the do as I do as I say not as I do, which is kind of the other way around right here. And this idea of connecting robots and language, it seems pretty natural, I have to say, I've, I've seen a number of paper attempt to do something like this, like, can we maybe translate what the language model says into the space of what the robot understands and things like this. But this here, it's


seems like a bit of a of a new approach. Why? Why did you try? Why did you attempt to do this? Like, why does this seem promising? And why did no one else do this thing yet?


Yeah, I think, to start like the, to, I guess, like prior work on, like, using a language model to kind of translate it down, I think we first started out with sort of like playing around with that. And I realized, I guess, how much information is imbued in these language models, and how well they're able to reason over sequences and remember what they've done. But when we really, like, started thinking about applying it to the world, it was sort of like odd that there's no way to basically make sure that whatever it's saying actually makes sense for the environment that it was


in. And so I think, like, after playing around with that for a while, we were sort of like stuck there, like, okay, we have these, like, interesting plans, but they don't actually make sense for everything that the robot can do. And so we started kind of like shifting towards towards that problem.


Yeah, I think also separately, we've been trying to get robots to do many things and learn multiple skills. And this is a very difficult problem. And we were debating kind of the best way to do this, whether we should predefine the skills upfront, or whether we should just demonstrate kind of anything that comes to mind and label it afterwards. And just connecting these two dots, the language models with the skills that we already have on the robot, seems like a nice way of factorizing this


problem.


Did you always, could you so you have this robot in this environment? And is if I understood correctly, maybe here is a good demonstration of that. They have the robot in these two environments. And these are the environments that exist to understand this correctly. So it's only these two environments, there's no


generalization across environments.


Yeah, so we've been collecting data in a few different environments. These are the two environments that we use for evaluation. We also have a separate environment that is right next to the environment that's marked as B here, where robots are practicing, but it looks fairly similar to at least the stations that the robots practice on are fairly similar to the stations that you see here.


The backgrounds are changing, the objects are changing that we practice with and things like that. We also use simulation as an additional environment that we then try to make look similar to the real world. But we don't really focus in this paper on generalization to completely new environment. We rather try to focus on kind of having a robot do as many things in a single environment.


When we talk about robot practicing things, I guess that's where your method starts with robots practicing things. And by things, I guess we mean a bunch of very low level, let's call them unit unitary skills, like here, for example, find a coke can pick up the coke can bring it to you something like this. So these, these could be things that conceivably we could learn with something like behavior cloning or something like this.


How did you decide on what actions are possible for these robots to do on their own, like as a unit?


Some of it is based on what the robot is capable of. Some of it's like what gives us a like a easy reward function. And some of it was sort of motivated by what composes well into long horizon behaviors that you really want to do in the world. Like, if we have a robot operating in the kitchen, what would I ask it to do? What's required of it to do that? And how would I break down the task? I think was like, part of the motivation, like really, how this robot is going to operate in the world.


Yeah, it also it's interesting to see how this picture changes. So initially, we kind of have to come up with these. And we kind of have to think upfront, what would that person ask a robot to do? But now that we have something running, we can actually ask people and see how they interact with the robot and decide on which skills we should be learning next based on that.


So I want to add that at the beginning, we choose pick and place because these are two fundamental skills that can unlock a large number of instructions that we are able to solve. But it is also very easy to add new skills into the picture. Like we only need to have a have a language language description for the skill. And we also need a policy and value function. So these are all the three, three things you need to learn.


And these are the three, three things you need to import a new skill into the same framework.


So what I like here is that you said you need a policy and a value function that policy doesn't even have to be like neural network based policy. conceivably one skill can be a very classic control problem. I believe when you pick up things, you is that correct that you classically control where the actuator should go? And when you move the robot, you kind of plan in space. So not everything is like reinforcement learned or behavior cloned.


Yeah, so different skills are learned differently. In this case, pick was learned through behavioral cloning on real data. But the efforts and some, for instance, moving around, this is not trained with reinforcement learning or behavior cloning. So you can compose, you can have different algorithms train different skills.


And these skills, just to to round out the picture right here, the input is whatever the camera sees, plus, you know, kind of all the states of the actuators. So that conceivably, there's an apple in front of you, and the task is pick up an apple. And that that'd be kind of the state from from where you operate.


That's right. Yeah, we are going to value function, the value function describes kind of how likely you are to fulfill that task.


That's right. Yeah. So the input to the policy is the image that the robot sees that you get at every after every action. We actuate the arm by doing and the factor position control. Yeah, these are the inputs and outputs.


And also, there's a terminate action, right? Sorry, that so the robot can say it itself when it's done.


Yes. So one of the actions of the robot can command this terminate, which basically means I'm done. Now we can move on to the next one.


And okay, so now I guess that this is one part of the puzzle, you have robots, you have all these policies for all the little things that the robots could do these little things were developed by you by the community conceivably, you could also use the large language models itself to suggest new things to train right on the basic level, you could ask gpd three, what would you do right here and then the little steps you could conceivably make into like train into little actions, but you have this line of code that you can use to do that.


And so you have all these little actions, but you have this library of things. And now the question is, how do you how do you compose them? And that's where the large language models comes in? Do you want to comment maybe a little bit on like, how does that look in a base in a basic way? How do we combine the knowledge of language models with these skills that the robots can do?


So this is at a high level. So the language model already has so much knowledge about the world. And how to do things in order and memory and things like that. And the way to get it to like really speak in the way that is amenable to the robot. First, we showed a few like prompt examples.


We showed solving, you know, like about 10 problems, and breaking it down from the query into the sequence of steps that it would take to solve that. It turns out you can actually not use that and you still actually get like, some level of performance, maybe like half the performance.


So the language model just like comes out of the box with pretty good understanding of these tasks. We then show it these examples, this kind of brings it into the right frame of thought. But if you do that, and you ask for something new, it doesn't like fully constrain it in a way that the robot will be able to understand it.


So our tasks, along with the image and the states that we mentioned before, also takes in like a task ID. So it says like, like pick up the apple. So really what we needed to do is like output, pick up the apple. It can't say like pick up the fruit because the low level policies are not generalizing to that.


So to make sure that every time we actually output things you can do, we instead of like taking the generative output of a language model, we use what's called a scoring model. So when a language model outputs some text, it also comes with a probability that it would output that text.


So instead, we can just like force it to only respond in these ways and say basically how likely it is to respond in that way. So in this case, we get like a score of if I were going to pick up the apple or put the apple somewhere, these are the things I'd likely to respond.


These are the things there's no way I would respond. And this gives us some like probability that the language model thinks this is really useful to the downstream task.


On the other side, we have these value functions and policies that we've talked about. They're actually the value functions outputting how likely it is to achieve a task. I think actually there's another slide one, like one more down.


But this is basically saying basically, these are possible from this state. And so on one hand, we have a language model saying, this seems really useful for the task. And on the other hand, we have the value function saying, this seems possible. And together, they give some probability that this is what you want to do to basically accomplish the high level instruction.


I have a number of, okay, let's just start at the beginning at the language model level, I see the high level picture, you ask both the language model and the value functions, what they think should happen next.


And the combination of the two is what then you really do, which makes a lot of sense. When you do ask these language models what to do right here, you said you use the you use the essentially you ask the language model for the likelihood of an output instead of letting it generate the output.


Was this your first try? Because one could also imagine, you know, saying something like, of the following options, which one would you pick, right? And then you list all the options, which would conceivably be more general, because you could option you could add options over time and stuff like I guess you could do that here as well.


But was this your first attempt? Or did you did you have some prompt engineering attempts before that?


Yeah, I think at first we tried just like prompt engineering to see like, basically what the generative model would output. I think like our, our initial thinking was we just want the generative model to basically plan as much as we can.


But that runs into two problems. One is that it doesn't constrain the output fully. So if I give it all these examples, and then I said, how would you put a fruit on the table instead of an apple on the table, the generative model will actually respond with like, number one, find a fruit number two, pick up the fruit.


And then you need to figure out how to like take that and project it into the final, like thing that the robot can actually handle. You can project this in some sort of like embedding space, and that works sort of well, but you actually lose some context on the overall query.


So I guess the way that we do it is a little bit more like, well founded, so to speak. But the other really nice benefit of this is it gives us scores for everything, which is really interpretable. It lets us like see the trade off between these two options.


So in your example, you said, you know, what if I just said, here are your options, pick one and the language model would probably pick one. But now you only know that this is its favorite option. You don't know the probability that it would have done maybe maybe it's actually okay with the next three options.


So this gives us this like interpretable score that we can then combine with the value functions.


Yeah. I mean, there are some caveats to this, I feel in that, for example, we know that by definition, longer outputs are less likely. Right. So I guess it's not too much of a problem for you, because most of yours are like three or four words.


But have you noticed any of kind of these effects of just how these probabilities are constructed as kind of multiplications of softmax outputs, like that's got to bring its own bias into the into the picture? Have you observed any of that? Have you had problems with any of that? Or was it was it generally okay?


Yeah, it's, it's definitely a little bit of an issue. I mean, I think in general, it's also very particular to the to these, like, if you were to misspell a word in there, or like have an A versus an M, it's not particularly robust to those in the options, it is in the query, like to what the user might say, but not when you're scoring these options, because if one word is off, then this like multiplication of each word just kind of tanks the entire score.


So we did have to be somewhat careful with what we have. One way to kind of like get around this a little bit is if you have some like, end of statement token, and if it if it adds extra words on the end, then it's saying, if there's like more to come that end of token will basically like kind of normalize the rest of it, like you can't end a statement early.


The, I think in the other thing that we did try to do is like, potentially normalize them. So knowing that this query is longer, perhaps we need to upweight it or have some normalization on the language output. But we found that it wasn't particularly consistent. And there wasn't like, just a constant effect across one or the other. And it depends on the way you like referred to the query. And so at the end of the day, we just took the outputs as they were. So it was an issue, but it wasn't like, it was a problem.


It was an issue, but it wasn't like a huge one.


I imagine that there's another factor here. For example, if you if you say, you said before, pick up a fruit or please bring me a fruit or something of this, you're essentially relying on the ability of the large language model to sort of recognize that Apple is a fruit and, and kind of interpret that in the same way and, and so on. So the kind of close as the language model estimates how close the things are, did you find this generally an agreement of in how how


humans find how close to things are? And maybe, yeah, I'm just I'm just wondering about this notion of how how close things in language are together. Also, what happens if you, for example, have an apple and an orange in your scene, these two things would be quite close together. So even if you said, you know, please pick up an apple, the pick up an orange thing would conceivably score quite high in the language model, which might perturb


your things. So I can kind of, I can sort of make out that you have an ideal environment right here, in that you probably picked objects that are distinct from each other, locations that are fairly distinct from each other, right, such that there's a nice semantic gap between the things. What like, do you think this is well applicable to a real world setting? Or what kind of hurdles could there be with connecting language models and the


set of actions in this way?


So I guess the first question was about, do these families kind of align with what you would expect? And that was actually that was one of the first things that I was looking at was like, how well do these scores sort of match up to what you think it's going to be? So yeah, it turns out that like apple and orange and banana are all going to score quite highly when you're asking for fruit. If you ask for a snack, all the food options are going to score highly. Similarly drink soda, any category like that. It performs about, yeah, as you said, it's a very, very close to the same thing as the


apple and orange. And it performs about, yeah, as you would expect as a human, which is good. But then yeah, it comes to this problem of what if there's an apple and orange? Or what if there is an orange but not an apple? And that's where these value functions come in. This is actually like, one of the key reasons why we have to do this embodiment grounding. Because if you just asked a regular language model that doesn't know what's there, then how does it make that decision? Maybe it chooses the wrong one, then your plan isn't really correct. And our policies may not actually work.


But the value function tells you, if there is an apple in the scene and no orange, then you're going to see a high value function on the apple because the pick apple command could work, versus the orange command is going to be quite low. And so that actually lets you sort of like disambiguate this. So in the in figure B, if it had a pick up the Red Bull, if you said, bring me a drink, and there's a Red Bull, but no water, it's going to pick up the Red Bull, because that's actually what's there.


And if not, then then the instruction itself is ambiguous, right? If you say pick pick up a drink, and there's two drinks. And both are affordable according to the value function. Yeah.


Yeah, then we think like either is completely fine.


I think it's also interesting, because then the robot is making the trade off itself, depending maybe on the value function. So for instance, if you ask for a fruit, and there's an orange and an apple, but it's much better at picking up apples, maybe it will pick up the apple because the value function will just tip the scale. So it will make some errors in that sense. But since this is interpretable, and you can kind of look back and see why it decided for that, it can also inform us as to what the value function is.


What skill we should train a little bit more, or which value functions are a little underfitted, and things like that. So it will make some sort of mistake. But maybe that's, that's okay. Maybe that's acceptable.


I think one like really nice feature that too, is it's not necessarily always like it's better at picking up oranges or apples. But you can see like these objects are in different locations, one may be better for the policy than the other. So we're gonna end up doing the one that's a little more robust, a little more likely to succeed as long as it still fulfills the high level query.


Yeah, I like the fact that you have success probability as sort of the the ultimate score, because I was I also thought one failure mode here is that some tasks are inherently harder than others, right. And so naturally, your value function would be lower. And therefore, you can misinterpret just by the fact like, well, like this, this, this is me, the procrastinator, like, this thing seems really hard. I'm gonna do this other thing that it doesn't really help, but it's really easy to do.


Easy. So it's almost it's almost too human how the robot would act in this way. So yeah, you have these. What I like here as well is that you have the bank of value functions on one hand, the language model on the other hand, and they are never, if I understand correctly, trained together, right? They're never in fact, the language model is probably just frozen.


So they're never trained together, which means that you could conceivably just add a skill to the robot train its value function for it, and just plug it in and and go.


Yeah, we can scale this fairly easily. So we can continue adding skills, we can also change the underlying algorithm on how we train the skills, or how we train the particular skill that we want to add if we if suddenly there is a really good script that allows to, I don't know, swipe the floor or something like that. We can we can also add that as long as we have a value function for it. And also at the same time, if the language model becomes better, we can add more skills.


We can also swap out the language model and get improvements through that.


I want to add that. So currently value function is one way that we instantiate affordance. But there are many other ways that we can instantiate affordance, like for example, we can directly do prediction. We can also use classical motion planning, like to calculate, for example, length of the trajectory is also or the probability of success if you do like sampling based motion planning. So there are many ways that we can come to the affordance and the method is really easy to use.


And it's also really flexible to plugging any type of affordance.


I guess a big topic in maybe, maybe it's more the space of blockchains and things like this is agents that do an action for you, but also optimize, for example, for cost, or for resources or something like this, this could directly flow into that where you can tell the robot, you know, do whatever fulfills my task, but also costs very little. And this could if this directly flows into affordance, there might be a normalization issue. But if this directly flows into a cost, then it's a good thing.


So if this directly flows in, you'd have you could tune the knobs on these on these functions fairly easily. So this is the full algorithm, I guess we haven't talked yet about how you extend this to multiple steps. But it is, as far as I can tell, fairly easy in that you do this in sort of a a stepwise fashion. So first, you ask your language model your value functions at the current state and the current camera position, where what should be done, then you can do the same thing.


Then you try to whatever should be done, according to both scores combined, you execute that. And after you execute it, you ask the same thing again, but now the prompt changes. And it's simply that you so here, the prompt is essentially I would first, and then first action is decided. And once you go on, the prompt now says, I would first and then whatever was decided on and then second and then


it's simply the same thing with the next action. Did I get this approximately correct? Do you pay any attention to whether or not the task was fulfilled successfully?


So right now we don't, we assume it will successfully execute, I think some things could happen, like, if it fails at a navigation task, say was trying to navigate to an apple, and the and it doesn't get there, then the value functions at that next state are going to be quite low. So you're not going to be able to basically pick something up or whatever. So maybe then you end up selecting navigate to the apple again, or navigate to a table instead. But we don't have any like explicit success.


Detection. I think this is like one area that we're like pretty interested in going basically like finishing the job and closing the loop entirely on when you try to do something, did you succeed telling the language model and then having the language model adapt accordingly.


I want to show one video from from your website, which in this case, if I got this right, it confused me, I guess a little bit, because this thing right here, if you see, it kind of looks around, sees all the things, right, like looks and sees, and then it kind of scores the actions. And like this. So pick Apple, I can't do that. Pick Apple, I can't do that.


Pick sponge, okay. Bring you a sponge. No, not go to trashcan place the sponge place the sponge is good. And that's the place the sponge kind of up ways to bring you a sponge. Or like what's going on right here, because in my in my estimation, the robot shouldn't even look around. Initially, the robot should just have its camera position fixed. And then it


in first instance, it should probably figure out like, find a sponge or something like this. And then it would move and then it would see consider these next actions. Like, what is what is this video supposed to to show?


Yeah, I think your understanding is completely correct. So this is more like a conceptual video where we wanted to kind of get across that it can accomplish longer tasks. But you're right that the way it would happen is that it would look at the current image, then it will decide that at first needs to find a sponge or maybe pick up the sponge if the sponge is already available, then append that to prompt and continue. So we just wanted to make it short so that you can still get get that idea across. But only


by having a single image. Yeah, so it might be a little bit confusing. It doesn't, I think, depict fully how the method works.


Yeah, I think we just got excited by the visual of a language model sort of seeing nothing and then waking up and saying, Oh, I'm a robot. Okay, here's my history of what I've done before. Okay, depending on that, what I thought I made a lot of sense doesn't make any sense anymore. So it's more like excitement than anything else.


It does look pretty sweet. Like, it looks pretty cool, especially like the effects on like the zoom, seeing what's around. You use by the way, we've we've not shown this yet you use these use these everyday robots, constructions, which look, look semi creepy, but also quite cool. And I think that's a really cool thing.


Especially when they pick up stuff, they like hold it behind their back, like it's like a mixture of a butler and someone who just has a knife and wants to stab you. But pretty sweet. And it works surprisingly well.


So maybe we can talk about the results of a little bit. Next, because my next question would sort of be okay, how well does it actually work in the environments where you tested on? Do you maybe want to comment a little bit on what was the what were the general results? And then you have some ablations?


If you want to take this or do you? Yeah, I think I can take this. So we test this on two environments. One is the real office kitchen. And another one is a kind of a mock office kitchen, showing in figure five, I think. And we tested on 101 instructions from like six categories. Yeah, so here here are the six categories.


Yeah, so here here are the test environment that the A is a real kitchen and B is a mock kitchen. There are 15 objects that we focus on and also five semantic semantic locations like these locations are semantically meaningful, like table trashcan, close counter, far counter, and a robot operator location where we define like, bring back to you. That's where it is supposed to bring it back to.


We test on 101 instructions from six or seven categories. If you scroll down a little bit, it's mainly to test different capabilities of the robot. For example, can it understand synonyms like non synonyms of verb synonyms like what does throwaway mean? Throwaway means bring something to the trashcan like recycle means bring something to the trashcan. And also structure language, which is just like verb non compositions. And also we test on the robot.


We test embodiment, which means we test if the robot understand what its current embodiment is, for example, if I already pick up something, I shouldn't try to find it again, because I already have it. Also, we test on crowdsourced. Basically, it's unstructured human queries from like co workers, for example, and long horizon tasks, which are some of the really challenging instructions, such as I spilled my coke on the table, how would you throw it away, and then bring me something to clean. So that's a really interesting thing to test.


So these are really challenging tasks, or the robot need to understand what does spill mean, like what tools you can use to clean up a spill. So these are the instructions that we tested. And overall, I think we achieved 71% planning success rate and 66% execution success rate. And it's the hardest question is do the longer horizon tasks. So I think we only have about like 30 or 40% success rate.


And yeah, we are working on improving those, like other success rate on those harder questions. Right, if you have anything to add,


yeah, the only thing I was gonna say is that the long horizon ones are particularly challenging, both from like, reasoning and language side, but a lot of the issue comes with if you have like a 90% success rate manipulation policy, which is still quite high. Every time you do this, you reduce the probability that your overall plans going to succeed. And so that starts to like, both it's a big challenge, and we want to get our manipulation policies better and better, and each of our low level skills better and better. But also,


having some sort of like closed loop that so the language model knows to retry would be really helpful here.


And you, I saw I saw in the results that it was pretty interesting in that you did ablate a lot of these things. For example, you did ablate what, for example, if we don't have the language model, and these, okay, these are the overall success rate, you play, what if we don't have the language model? And what if we don't have the scoring model? And generally, they were worse, much worse in both cases, which was pretty interesting.


And it's pretty cool to see, and not always the same, except in this one. It is one thing to understand this correctly, if you drop the generative model, or no generative uses it uses a large language on a projects the nearest to the nearest skill via an embedding, that is actually better than your original policy. Is that just noise? Or is there something behind it? If you use this verbs category?


And my guess is, I think it's more noise than, than anything else. But there were definitely times where, so I mean, we see it like really fail in certain circumstances. So embodiment, because there's no value function there to tell it that it can't do something, there's a real issue for it. And so there are a lot of failures for anything that didn't have a value function there.


I do think we saw some like, some pretty interesting differences between the no value function. So this is the scoring model only without a value function, and the generative model. And so some of the issues with a generative model came around with like nouns, for instance.


And this is because when you do this projection, so the say I said I just worked out I want a snack, it then projects to or then these, the plan will say, bring me a snack. But really, what I want is a snack to help me recover from my workout. And so that like little bit of information is enough to say it's probably not like potato chips, but maybe something like healthier similarly like a drink there would lose a lot of its information.


And so on the noun ones, we saw that it ended up like losing this information that cost a lot of the success rate, whereas the scoring model did okay across the board, but maybe not as like smoothly in the verb category.


Another really fascinating thing here is, at least in my opinion, just the scale of data collection in this project. I have I have made a few notes. And at one point, it says something like, you use a lot of human labelers for, for example, the success rate of these little policies. So even when when you train these little or small, small


unit, let's call them unit policies, you use humans to see whether they're correct or not. And you use three human raters per execution. And you have to train these little human raters to see whether they're correct or not. So that's a really interesting thing.


It says something like, let's call them unit policies, you use humans to see whether they're correct or not. And you use three human raters per execution, and you get it, it get give it one single sparse reward. If two out of three agree, so like this scale seems immense.


the best way to spend the human time and not maybe to gather more noisy but three times


more labels or something like this? Like how did this come to be?


Yeah, this is a good question. I think we are still figuring this out, a lot of these


questions and how to spend how to spend human time in the most efficient way that helps


the policies the most. And I think there's a question of crowd labeling, as you as you


mentioned. So how much noise can you tolerate in the reward function compared to like the


throughput of that? Also, how much time you should spend collecting human demonstrations


versus how much time humans maybe should be just supervising robots collecting data autonomously?


How much should we be spending time developing assets and policies in simulation and transferring


them to the real world? So we are still kind of trying to find the trade offs between all


of these. I don't think we have any any very good answers right now. As for labeling itself,


we noticed in previous projects that the noise and the reward signal is can be really can


have a big influence on performance. So that's why we decided to have three labelers to agree


on the two of which we have to agree to market the reward. And we also had additional questions


such as was the behavior undesirable or unsafe? And these are sometimes quite ambiguous. So


it's actually it helps quite a lot to have multiple people look at the video and and


tell us what they think.


Did you always have these additional things in? So you have, as you say, and also wrote


this down somewhere, a unsafe, undesirable or infeasible? Did you always have this in


or was this kind of a development that happened over time that you realized, oh, crap, we're


asking people, how likely is the robot to pick up an apple, but there's no apple in


sight and things like this?


Yeah, so some of them we added. So initially, we knew that safety is a is a big problem.


So we started with with that question. And we noticed that sometimes the robot would


do something that isn't necessarily unsafe, but we still don't want it to do it. For instance,


it will touch the object that it wasn't supposed to touch or it will just poke something and


it will fall off the table. So then then we added the undesirable, which is like has a


slightly different definition. And we can also optimize for it differently in the reward


function. And then regarding the the last one, the infeasibility, this is something


that we noticed with reinforcement learning algorithms, that if you add a lot of data


where the task wasn't feasible, even though the data is technically correct, right, the


robot didn't accomplish the task, it got reward zero, but it seems to be influencing the real


algorithms in a bad way. So we added this in addition to prevent that and potentially


filter for this data or see how we can change the algorithms to handle that kind of data


better.


And why do you only give a single reward? I mean, presumably a human watching a video


like this could be, you know, every couple of frames could be like, yeah, good job, robot.


Yeah, that's the right way. Yeah, oh, no, don't do that. Like, essentially, like Peter


Pan or like, you know, warmer, warmer, warmer, colder, colder, which would give sort of a


much more dense label space. Was this like a technical limitation? Or did you also consciously


choose to say, No, we got that it's one single reward. And that's only it's one when you


fulfill the task and zero everywhere else. Yeah, so there's, I think, a few reasons for


this. First, I think the ambiguity that comes with it, you know, it's already sometimes


difficult to decide whether the task was accomplished correctly, or whether it was undesirable or


not. If in addition to this, you have to add this continuous signal, whether the robot


is going the right direction, I think it can be fairly ambiguous, depending on what the


robot is doing. Secondly, we made a decision some time ago, that optimizing for sparse


reward tasks would be just more scalable for the future. There are some tasks where it's


quite difficult to say whether the robot is actually going in the in the right direction.


And sometimes that accomplishes a task in a surprising way. And we don't necessarily


want to kind of eliminate that and introduce human bias of like, well, I think it should


go that way. So our array algorithms that we've been developing have also been optimized


for the sparse reward setting. So that was kind of another factor that we that we thought


about when when considering the reward function.


So speaking about doing it like humans, there's a yet another set of data collection in this


project. And that is that not only do you collect the labels, but you also do quite


a considerable amount of behavior cloning from essentially learning from demonstrations


from humans with another set of data gathered from you call the tele operated tele operator


sessions. How can we how can we imagine such a tele operator session? Like how many of


these kitchens and robots do you have? And how long does this take to gather a data set


that you could conceivably do behavior cloning from?


Yeah, so I think we specified in the paper that we gathered at that point around 70,000


demonstrations for all these different tasks. This is across 11 robots, I believe. We built


a little we built little stations where the robots like the stations that you can see


in the picture here, where the robots can can practice these things and people can demonstrate


how to how to how to do things. I think we are still trying to see how much of this if


we if we filter the data set, for instance, how much can we filter it and still get really


high result. So I think we we don't have very good answers to that yet. Yeah, but this is


something we're looking into kind of the trade offs between how much demonstration how many


demonstrations you're collecting how much autonomous data and so on.


Where is this just because this is at Google, which is a company and sure there's like a


cash cow that generates infinite money, but there's got to be some kind of constraint


on you just or how do you how do you how does this work maybe what robotics at Google? What


is your mission there? And how do you pitch such a thing to to management like yeah, essentially


we want to collect 70,000 sessions of tele operated things every time a human presumably


not a random human because they would just crash the robot out of spite. But like a trained


trusted human needs to sit down and spend their time and there's robots are quite slow


as of now. Like this, there's got to be a considerable budget behind all of this data


collection and labeling and so on. How do you do you have to make a case for that? Or


are you relatively free in doing this? How does how does your work in the let's say it


in the business perspective look like?


Yeah, I think in any company, you kind of have to make a case or even in in academia,


you have to make a case for your project why you think this is how the money should be


spent and where the resources should go. So usually the way we we kind of justify this


by showing kind of step by step results and showing if we extrapolate this where this


is going to go. So we we've done some projects previously where we showed reinforcement learning


at scale with six robots or behavior cloning at scale with just two or three robots. And


then we start seeing that with the amount of data that we collected there, we already


can see some interesting results. And now if we want to get these robots to do many


more things, we need more robots, we need more data. And this is kind of one big bet


that we that we have in robotics at Google is that this large scale machine learning


could be a way to really help robotics. So we want to we want to be able to de-risk some


of those questions for the for the community, right? Like if we can actually buy a lot of


robots and provide a lot of demonstrations, how does it scale? How does it work?


I think one of the slides or one of the figures in the appendix actually has somewhat like


the way that we built up these skills one by one, it's maybe I don't know what page


it's on, but it's a little higher than that. Yeah, this one sort of shows like how these


were built up over time and, and how more one more and more skills were added, more


and more data was collected each time seeing signs of life for the algorithms and performance


and improving upon that.


And you can see that from time to time, there's a new skill being added. So that kind of goes


from zero up. In the meantime, there's also like underlying code is changing. So it's


kind of like improvements over time.


So this goes it goes up and to the right, which is what we all love. And was there was


there major downturns in this project, like times where, you know, things didn't seem


to work out or you didn't exactly know what the problem was things like this, could you


get us a bit behind the scenes into when when things go wrong?


No problem. See, I think there's quite a lot. I'm just trying to think which one to tell


you. There's quite a lot also from previous projects. But I think one thing that was quite


surprising to me personally, and I think we are still kind of working on that as that


if you spent in if you classify approaches into, let's say, imitation learning and reinforcement


learning approaches, if you spend enough time and data on either of them, you can get them


to work.


So we some of the the results that you see here, most of them are from behavior calling,


but we can achieve very comparable results with reinforcement learning either by transferring


policies from simulation and then continue collecting with that policy and kind of fine


tuning it to a high performance, or by just bootstrapping from real data and improving


upon that. But what is quite surprising is that combining these these two have has been


quite tricky. So kind of having a single algorithm that can digest all of that data that can


digest all of the demonstrations, as well as the autonomous data that was collected,


data that we collect in simulation and so on, and have it have all the properties that


we want. So it performs at least as good as behavior cloning, but it can also improve


autonomously and so on. This has been this has been quite surprising and tricky.


I want to maybe have a bit of an or make a bit of an outlook right here because it seems


we have a pretty cool way to go from skills that are described by language, but you have


to define them. Let's just scroll to one of them. You have to define them ahead of time,


right? You have to define pick up the Coke can bring it to you find the Coke can and


so on you have to just you have to design these even though they're described by language,


they're pretty fixed set. Now the first thing that maybe one can think about is how to extend


that set and not necessarily extended just linearly. But I'm thinking of something when


I say please clean up the table, you might not know what's on the table. So we need this


kind of a concept of like almost like a variable or an unknown. You know, like, so the plan


could be go to the table and then kind of decide what to do next. So the language model


could get even or has to get a feedback almost from either the the value functions or from


the picture itself, right? Is that anything that's on your your radar? Sort of what if


I don't what if I have to adjust my plan on the fly to the state that I'm going to encounter?


How could this model be extended to to handle that? Let's say all the actions are in your


action space, right? But you just don't know at the beginning which ones you're going to


take.


Yeah, I guess right now we kind of like count on the value functions to sort of like collapse


whatever your plan is into the thing that is actually possible in the world. I think


like one of the most, I guess, straightforward ways to do it, though maybe not straightforward


in practice, is to use things like visual transformers, or like structured scene representations


that actually tell the language model what's possible so that they can start like reasoning


over it earlier on. The other thing is to add in something like these success rates,


success detectors that say, okay, you tried to do this and it wasn't possible. So maybe


you tried to find an apple that wasn't possible. Perhaps the next thing to do is try to find


an orange that may actually be in the scene. So there's some like combination of value


functions giving it feedback about the scene. But right now we don't have anything that


like has the language model really, really reasoning over the steps because the value


functions take care of that like interaction. But one could fine tune it on some data that


allows it to do that is probably the most straightforward way to do it. But whether


that works is open question.


I guess the other thing is, and this would really also close the loop or close one of


the loops is if I imagine that I also had a model that could take any visual input and


then kind of describe that, describe what's happening in the visual input. So I'm going


to give it a video of pick up the of something picking up the Coke can and the thing would


come up with like a label for it like, oh, this video shows pick up a Coke can, right?


Then I'd have almost limitless possibilities. I could just let a robot move at random essentially


let the language model or let this model describe what it's doing, then kind of feed that to


the language model and so on. So instead of you designing the actions that it should train,


I could just let it do stuff and then have a model describe that stuff and then use that.


Is that a plan or is there like a major hurdle on the way there? Because that will kind of


result in a almost autonomously learning system if you give it a good language model. The


language model could even also prompted what to try next, right? Like the language model


could be like, okay, what should I learn next? I should probably learn to pick up an orange.


And then you just ran them around until the thing, the description model says, ah, this


looks like picking up an orange.


I guess I can say something first and then I will ask like Carol because he has previously


worked, Carol and Brian worked a little bit on like learning from play data. So what you


described kind of similar to that. What I want to mention is that we find language is


a great kind of state obstruction because people invent language because they obstruct


some state, right? Like every word, every sentence is meaningful. So there are some


work in language showing that like using language obstruction can improve exploration. For example,


you can use that to guide your exploration and summarize current state. So that's one


potential direction that we can go.


Yeah, I think there's kind of multiple ways you can see pushing this to an extreme. I


think like one small step in the direction would be rather than having these predefined


skills, label everything in hindsight, as I think you're describing as well, and train


policies based on the hindsight labels. So it's not just pick up an apple, but you know,


no matter kind of however the person that looked at that video described that that's


the skill that the robot was performing. And then you maybe don't have to constrain the


language model to pick across the skills that you train, but maybe you can just take the


generative output and see how that works. I think there is also a potential research


to be done in how much can language actually take from the robotics problem and how much


can it help solving it. So right now we are operating at a certain level of obstruction.


Like you command things like pick up the cocaine and then the language model can operate on


that. But you can also imagine operating on much lower level, which is just like, you


know, move this direction or that direction or something like that. And the language model


commands all of that. And you kind of, you can choose where in that obstruction you want


to be. And I think it's quite interesting that we at least can try things like this


because of how good language models are today.


Yeah, and I think I guess to that there's also works on using language basically to


predict rewards like over states. And so that's like one way to kind of like hook it all together.


We have this like general framework.


What's the biggest hurdle? Like what's the biggest, let's say unsolved problem to push


the sort of everyday robots, not the company, but like the the the expression, the robots


that help us doing our tasks? What where's the like the biggest roadblock in getting


these to a point where they could actually be usable?


I think right now, given kind of how much time we spend on different parts of the system,


it's the skills themselves, the bottleneck is still the robot actually doing the thing


that you ask it to do, even though these skills are simple to get them to the place where


they generalize to any environment can kind of pick up any object, even the object that


it wasn't trained on and do these tasks with large diversity of objects, environments and


so on to very high performance. This is still really, really hard. So I think if if we get


much better skills, underlying skills, then we'll have made a big step towards this actually


being very useful.


I was going to say the along with those skills, like the way that we use the value functions


is that as the skill improves, so does the like value functions estimate of what it can


do. So it's kind of nice. We're like positioned both to use these skills, but it also improved


the overall algorithm by having a better estimate of a success probability. So I think we're


like, I think SACAN itself is at least set up in a good way to sort of like scale along


with as this bottleneck is relieved.


Last question from from my side, what do you think of the Tesla bot? And when I give you


the short pro in in in briefly, and that it is the ultimate platform, because the world


is designed for designed for humans, right? So if you have the humanoid robot, conceivably,


it could do anything the human can at least mechanically. Do you does this sound good


to you? Or is there like major skepticism?


No comments. You can you can wager wager bets right now.


I think one thing that is maybe that I'm excited to see is I think Tesla has the ability to


scale things up quite well. They seem to be a really good hardware company. And so it


would be interesting to see how some of the problems change. This is also things that


we are researching as well how problems change and how solutions change when you have many,


many of these robots. So I would be I would be excited to see they have any any good insights


there.


Is there last things that we maybe haven't touched on yet that you would like people


to know? Here, just for visuals, I'm showing what some of the successful episodes at the


end, which are quite impressive, like very multi. So there's just one robot just this


is this is a collage, but very multi step things. And I think that's just really impressive.


Very long horizon planning things down to these individual actions. Yeah, that's that's


pretty cool. Anything any last thing you want to let people know how can they get started?


Where can they find out more information?


Just want to mention that we have the website on the website, we have a couple of videos


demo demonstrating how the robot works and how the inference process works along with


the decision process, all the scores we have calculated, along with the robot execution.


So if there are anyone interested in like how our algorithm works, check, definitely


check that out.


I think like, I guess what I'm most excited about with it is like, how interpretable it


is that you can actually see how the decision is being reached by the robot that you can


see that the language model likes these things and that the affordance model understands


that these tasks make sense or do not make sense in a given world embodied environment.


I think it's like nice that it scales really well to adding in new tasks as we go. And


then I guess towards how people would use it, I think to start Yeah, I mean, the paper


and the website is a good place to go. I think we're planning to open source a version of


it on a more kind of toy environment in the coming months. So hopefully, that'll be like


an exciting, like easy way to sort of like, get in the mix with both this and language


models. I think there's a lot of power in, in leveraging language models and kind of


giving them these like hands and eyes to execute real world tasks. I also think you had a point


earlier about basically, like we use affordances, but really, it's just a value function. It's


this value function doesn't necessarily have to map to an affordance. And I think that's


a really powerful idea that we're basically taking all the knowledge in a language model


and then hopefully applying it with a value function that isn't even necessarily normalized


to can you do this or not? It's sort of what's helpful, what's possible for whatever the


RL train policy is doing. So I think that's like a really, I don't know, open space.


Yeah, I'm also quite excited about how language can kind of chip away a little bit from the


robotics problem. I think that's something that we haven't really thought about that


much before. And we see that we can handle much more, much longer horizon commands, abstract