for maybe like a second or less than a second humans are actually not even that sensitive to it so it needs to be a little bit of displacement okay so now we trained this representation for a long long time you know weeks of GPU time and then we have a representation that has the whole thing working right here hope the whole thing working and then we can think we can look at what we


can do with that representation so one thing we can do is we can now visualize the source of the sound because what we can do is we can say okay given this task of you know is it is it aligned or not aligned we can actually just use the kind of classic class activation visualization maps to see what pixels is it using to tell if things are aligned or not because those are the pixels that

Alexei Efros: Self-supervision, Meta-supervision, Curiosity: Making Computers Study Harder


are the sound producing pixels okay so here is an example of some of the places where it thought those were the important places that it's that it decided where the sound producing places and here is some some some visualizations of this over time [Music] okay so this is again completely automatic no labels of any kind okay another thing we can do is we can just


plug this in into your stand the kind of action recognition data said a lot of those data sets have audio in them so we thought okay can we use this audio channel to improve our our method and we are actually definitely getting improvement over audio and we are improving other self supervised method we're still not as good as something that is kind of trained with lots and


lots of relevant semantic data but again the hope is that as we get to harder and harder data sets that semantic labels will be harder and harder to get and so the self supervised methods hopefully will get better okay and finally kind of a cool fun thing we could do is to see if we can do off on screen separation of the audio sources so an example would be if you have a


speaker speaking in on to the camera and then there's somebody else speaking that is not being seen our feature is only going to focus on the on the speaker that is being seen and so we can subtract away the speaker who is not being seen okay so let's let's see what I so unfortunately we thought oh this is such a cool idea nobody would ever think about it at the same time like four more


groups who are basically doing the same thing luckily we heard about each other in time so we actually cross sighted in so it's all fine our method most of these folks actually that was the goal of their papers so they could have they basically work on that particular problem for us it's just basically one application of our feature that just kind of falls out of


our method so we'd say that we're kind of a it's it's more of an application of our stuff but hopefully we are also able to to do other other things as well and the idea here is basically we take our representation and then we seed it into kind of a standard encoder/decoder representation that starts with a spectrogram and basically learns the separated into into things that part of


the spectrogram that have evidence in the image or in the video and the part that does not have evidence in the video and then you can then you can play either one or the other right once you separate you can play either one and so here is an example then asking about it because they're not interesting facts to you that's not true I have a plenty of questions integration you a sample about


your by talking about your flight now I know there's a bunch of people talking one is old screen one is off screen let's see what we can do so this is just the on screen been asking about it because they're not interesting facts to you and then people all about no I'm not I don't want to okay and this is the off screen that's not true I have a plenty of questions integration you have to


disability Oscar by talking about your flight so there's there is a little bit of noise there but mostly does the right thing thank you so much thank okay and this is midnight is no second have day let's talk about digital omens there are some fears so much were able to show to the rest of the world the unshakable japan-us alliance okay we have both speakers here okay


you're saying that they're not and we can we can hide one of the speakers and then gets away okay could even do something all right laughter okay all right all right okay most of it is gone but not all okay so so that is that is kind of a various ways of trying to get data to supervise itself and hopefully learn the representation that that is that is you know useful for other tasks the second


topic I want to mention briefly is what we call meta supervision and the idea here is instead of telling what the correct answer should be we tell how that correct answer should behave so what do I mean by that so the kind of the direct supervision is a the direct supervision is is you have input X and you train a function f of X and you want that function f of X to produce Y's okay


that's direct supervision you know from X's to eyes what are other ways we can we can set up this problem one way is we train a function of f of X that produces something in the domain Y in the set capital y so we don't tell it what particular y we want we just want it to be in the set of Y's okay and one example of this is generative adversarial networks I'll give a brief


overview based on a paper we had a couple last year the colorization example that I showed it had a lot of kind of a ways of hacking it to make it look good and some what you want is you really want this white wall to be white on the back but because we kind of told it to all you need to be more colorful the wall becomes not white okay so it's kind of overshooting it and the annoying


thing is that there is no way for the algorithm to look at this that's just not looking realistic you know do something better make it better have a try to optimize for things looking realistic so we don't know how to do that well actually we do so you know for any kind of a problem we do have like colic or is polarization or super-resolution or whatever it would be


nice if we had this function that would tell us you know make something realistic enough it is this loss function this commune universal it says make make images look real we do have that function right now that's that's called a graduate student okay that's where the graduate student basically keeps hacking all algorithm until you know enough of the pictures look good


and then then we send it to to publish but it would be better if the computer were doing it themselves okay so one way that we we can do this is we can basically have somehow the computer send the resulting images to Amazon Mechanical Turk ask a whole bunch of people if this is good looking or not good looking and use that signal to update the algorithm very very expensive


okay but what what we can do is we can use this idea that that recently came out that kind of does something similar okay because remember we have a lot of real images so what we could do is we can have another network that can act as as an amazon turk are deciding if something looks good or not okay and that network is basically going to tell us if the image that we generated if it


looks real that is does it can you distinguish that image from a set of real images and if their answer is no that means that we are doing well okay and this is the idea behind this giant material models in Goodfellow and colleagues that that has kind of really energized this whole field of image synthesis let's think about it you know what is it actually doing


quickly so we have our function that translates say from greyscale to color okay and now what we want is we want to add another network on top of that and that network we want to decide how to decide if the image here if it looks real or it looks fake okay so what we want is we want G the network G to fool the network D so we want D to think that this is a real image whereas


in fact it was generated by by by G okay and of course G who doesn't want to get fooled so do you really trust ooh to figure out if it can if it can if it can tell and the idea is to basically have these two networks battle it out in a kind of a duel like an arms race and the idea with as just any arms races when you have competition when you have a duel both


get better and better and better that's that's that's the beautiful story Oh of gas right so here is a little bit of math so let's say that in this particular context if we're trying to see what G wants to do d wants to have a high probability for generated images if the image was generated by G we wanted to have a high probability of saying that it's it's a fake image okay whereas


if the image is actually a real image then we want to have it say no this is a low probability that it was faked okay so we want to have a G such that it maximizes this quantity okay at the same time of course what we want G to do is to do the opposite it wants to minimize that same quantity right so d G is going to get back signal from D that says okay I figured you out


and then G is gonna say okay let me see what I can do better now to improve my generator generated image so that G will have a hard time so maybe I'll something that looks a little bit better okay and try to fool it but of course I don't want to fool just this particular G I want to fool a the best possible D so this is where we get this whole minimax formulation where you want to


have the best D and then minimize the the G to - to do the best of that okay so one way to think about it is that now this D you can think of it as a loss function it's kind of like l1 or l2 it basically tells what you need to do how do you optimize G such that it gets closer and closer to the goal the only difference is that instead of being nel - now this G is learnt the G learns what


does that mean to get closer to the goal for that particular problem and what it means is that it basically means to be indistinguishable from the real samples from this from this data domain okay so we're almost done but not quite because here is an example imagine that my G went completely crazy and started producing cats for any input image it got it produced the cat okay now is this


a real immature a fake image it's a real image it's a cat it's a it's my Student Union's cat named Aquarius very nice yet so it's a real image but it's not really what we met so we need to give it a little bit more constraints so basically what we want to do is we want to give D not just the generate image but also the input X so it can look at the pair of both of them together to say is it is


the G of X the result of starting with X so that is is this a real pair or a fake pair and now we're all good now this is the conditional gann case and now we're we're able to to get it to work and this is the the final thing that is being optimized now I'm of course hiding a whole bunch of things under the rug here this is not a pretty optimization as you might imagine it is very complex


complicated to optimize this thing and and and there is a lot and a lot of work on trying to make it simpler so far it's still it's it's it's it's more art than a science how do you optimize this thing but my graduate students are really amazing at doing this so so we were able to get this working and so now we can we can you know plug in greyscale and call her pair and then we can colorize images


we can do the exactly the same code we plug in Google Streetview and satellites and then we can you know we can basically hallucinate satellites from from maps or we can do it the other way around exactly the same code but because the G is getting optimized for every different pair it basically learns what is important for every domain we can generate from labels we can generate


facades we can go from day to night we can go to from thermal imaging to to normal RGB imaging we can take edges image edges and produce images that could have come from those things okay this kind of looks cool but actually it's not that complicated H maps actually contain a lot of the information the cool thing is that we can then train on this and test on just


you know human sketches and even there it's actually doing something reasonable which is which is quite kind of neat and then we we put this online the code online and and a lot of kind of artists decided to do cool things with this and so this is this is kind of a neat thing where you you don't even need to do results of your papers anymore you just kind of post the code online and then


just download results from from that other peoples have done and and somebody even did a little edges to cats thing you can try to yourself you you you you draw something you hit the the pigs button and then it will get you and get your cat okay there you go the the the best the best yeah the best use of company computer technology in my at least for me I don't


know this is yeah this is the the pinnacle of my research career I think I get lots of cats on the Internet so so this is an example of Gann so as a again we we talked about direct supervision again basically cell we supervise of not on the particular label but on a set why there are other types of metal supervision we can think about so one is one of my favorite ones its


cycle consistency okay the idea is that we don't know the answer why we don't have label for that but what we know is that if we if we have our f of X which produces some Y and then we apply a G of that of that we should get back to X okay and this is a constraint that people have used a lot especially in tracking in computer vision you track forwards you get somewhere you don't


know where you are then you track backwards in time along the video and the idea is that you you should end up where you started with and if you don't then something is wrong okay but we can use this as a constraint to again for optimization so for example let's say that we want to do this kind of a pics to pix image to image translation but we don't have labeled pairs so let's say we


want to translate from horses into zebras right there is no possible label data for this so how do we do this well we can take an inspiration from actually Mark Twain and ID of back-translation in in in in in in language and the idea of back translation is that if you want something translated in a foreign language you don't know you hire one translator to translate to that language


and you hire another one to decide back into language you do know and then you double check that it kind of still make sense right and Mark Twain wrote this book jumping frog in English then in French then clawed back into a civilized language once more by patient and renew merited toil so here he was showing that in this particular case the translation was not a good one so he translated back


and and sure that was it was it was not not looking good okay and so what we're gonna do here is we're going to basically do the same idea we're going to now have a translator G that goes from domain acts of the main why okay and then translator F that goes back okay and and and and that's all now because we one do we don't want it to cheat and just stay where it is


we want it to also have this adversarial loss this this disg and loss it says that when you get to domain why you better be indistinguishable from a real thing and why and when you get to domain X you better be indistinguishable from something real in X okay so what we're doing here is we're since starting with an image X we translate into a zebra domain again we don't have a label for


that but we know that it should look like a real zebra and then we translate it back and then if we didn't if we don't get exactly where we started with well that's our loss that's what we want to minimize that is exactly this the thing that we are going to want to minimize okay and if you kind of a step back and squint at this thing what does it look like it's


an hour old friend out encoder right you have the input you reconstruct that input the only difference is that it's an out in Golder that instead of a bottleneck it just has a different domain in the middle so it basically just has go through something else it's forced to go from some other representation and then come back and then you also do it the other way around


okay and so then we can turn horses into zebras and vice versa we can even do it in in in videos just one frame at a time the failures are kind of fun I showed this picture in Moscow last year and I thought that's it I've they'll not let me out but they did we could also do kind of nice things on going from images to paintings okay by just kind of matching from domain of


photographs the domain of in this case Suzanne paintings and I'm particularly happy about those clouds and you've probably seen a lot of the stylization papers and results they usually look at a single single image that you want to stylize for a particular image here we can take a whole domain we can take all of Suzanne's painting all the thousands of them and learn the representation


that kind of models the whole Cezanne okay and and the clouds look pretty nice we can also go the other way around we can go from a painting into something that kind of hopefully hard to distinguish from a real image now if this was a perfect talk this would also be Suzanne and Suzanne didn't work Monet is simpler so so I will show you Monet but we're still working on Suzanne


hopefully we can get Suzanne - okay we can also apply this to translating between video games in the real world so this is Grand Theft Auto and this is making it look like kitty so now you can see it's like old German looking and you can go the other way around which is even cooler they kind of walking around with a with with like make your reality like a video game so here is an example


of reality as a video game I can let me show you just because this is just such a cool such a cool result this is this is again people have been just playing around with with these things all right I know it's not alright never mind it's forgot this is like artists just taking our code and running with it and okay I don't have the okay nevermind I will I will I'll give give you the


pointer but yeah finally I want to show a very cute example of you know I in in my you know in my group we have been playing around with a lot of making fake imagery from from early on and so now a lot of people are worried about you know all this fake news and you know put in screwing things up so we thought we could try to play on the other side of the fence as well and see if we can


detect if image is not being realistic okay and the top one is actually from my old paper with James Hayes where we learn to kind of fill in holes and create these image composites and here we can finally you know detect this and and and recover the that this image was what's fake and we are also going to do it with this the same idea of of self supervision and metal supervision this


is with a couple of wonderful former Berkeley undergrads and and under Owens so given this image it might not might look reasonable to you but in fact of course it is fake and how do we how do we detect this well if we had enough fake examples we can just again do this you know supervise direct supervision they're out but we just don't have enough of these positive


examples and so what we're going to do is we're trying to think about this as a normally detection so we're going to find to see if we can learn if an image is consistent with itself okay and the idea here is the following we can look at a couple of patches of this image and we see is there some sort of some kind of fingerprint in these in this patches that tells us that they might have come


from different imaging systems that they are not from the same camera or and not from the same image okay now if we had access to the actual images that this was taken from created with then we could actually look at the the metadata that comes with the image and then we can realize that it's actually you know the cameras are different the focal lengths and different cetera et cetera


but of course in real life we don't have access to any of that and so what we're going to do is we're going to train an algorithm to see if for a pair of images if those images if we can learn if a pair of patches comes from the same image or not now that by itself you could also do that that doesn't work as well because then again you don't have enough enough data for it for that so


instead what we're gonna do is we're gonna train if a pair of patches have the same is it metadata tag there's a many different metadata exif metadata tags like camera brand focal length JPEG compression etc etc and for each one we can predict not the value of that tag but is it the same or is it different okay and so the idea is that we have a whole bunch of we train


with a whole bunch of real images we don't need any fake images for this so for every pair of real images we take a couple of random patches and then we look at the exif tags that are similar in this images and we train those things to say okay yes those are similar and the different ones we say no those are those are different so basically we train something like 80 different


classifiers that says for every different for every single active tag is this going to be the same or is it going to be different okay and so now we have a kind of a way to establish if a pair of images if pair of patches are consistent and more along one of the dimensions the dimension being you know are they do they come from the same camera do they have the same resolution


do they have the same jpg etc etcetera okay so here is the different task tags and how well the how well we can predict if they if they come from the same image or not so you can say that the lens make is one of top-performing months so it is basically like the the what who produced the lens then you have a custom renderer is basically some Apple iPhone thing it


basically says iPhone then then then you have a bunch of various things that really code for a different processing that's done by different cameras so they're all kind of things that different cameras do differently whereas things like image date and time or GPS coordinate are basically a chance level as you would expect okay and then what we do is we combine them all together oh


and we also have some other consistencies that are like try to do it like blurring and and our Erie JPEG in etc etc and then a test time here is an image here is a manipulated image and what we do is we just have a pair go and find a whole bunch of different pairs of images and for every single active tag we could predict a map of if those two images are consistent or


not consistent with each other these two patches okay and so we have a kind of consistency map for every single tag like camera or or focal length etc and then we combine them together into an overall consistency heat map and then once we have a heat map we can we can run normalized cuts and actually cut it into inter and thing or not and so here we can predict that all look at this


this is this is a this is the inconsistent part and here is here is what it found and consistency is here here actually we didn't even notice it but it detected that the shadow on the on the floor was also painted in not just the guy on the top not sure nice nicely it works it you know for normal images it doesn't fire usually which is good and you know we're beating most of


the pretty pretty much all all the other methods that are have been supervised we are beating them with this kind of self supervised method and for some images we don't have the ground truth so we don't know so I don't know who knows maybe this is how conspiracy theories are are born so I think I'm over time there so I I will I will I will skip the last part


but I'm happy to talk about curiosity 101 so I think oops there you go thank you very much [Applause] any questions actually all know on the last part detecting detecting fake images so manipulated images so I would I wonder what would happen if you actually plug it into this game cycle where you're trying to actually make make wadding image look like it was taken with


different objects with different camera and then having this discriminatory trying to discriminate it it which of the sides would win eventually if any so yeah so in I think in in you know longer so I think what they were saying is what happens if you have the the the critic the guy who is trying to find fakes connect with the generator that tries to make them better


right and then have them battle with each other with this because there's an ongoing thing where people are trying to find fake images and then on the other hand improving that's right that's right that's right no I think this is this is actually that that's that's what we are talking you're thinking about doing doing doing next this kind of to have the the the fake detector not be a kind


of a static one but to to to learn by having a generator generate better and better fakes and so hopefully then the detector becomes better and better and better technology to improve the detectors of face is exactly the same technology that you will use to improve the producing of the images right that is that is true but at least for now with these grand formulations in the end so that it this


is kind of a weird thing that began really the Khmers converges when the detector cannot know the difference between the real and the fake right so that would be when the the thing actually converges in reality though the detector always wins so we we cannot fool the detector so those gaps never really converge so for now and it kind of may be reasonable because it's always


easier to criticize them to create right always easy to be critic than it would then than a than a painter right so for now that might be okay but but in generally I think it's a it's it's it's an arms race it's an arms race and and and yeah there is no there is no gonna be a perfect solution there is always going to be something that the generator can do that that defeats the the


defenses that's why I think it needs to be active that's why the the the this the the the fake detector needs to keep thinking of some other ways that the generator could potentially be fooling it and and be prepared for that if I may just smoke before so many of these fake images are created by copying you know patches of the same image somewhere else in the


image is that something that this detector will be able to detect no no we actually we looked at this and most of those most of the fakes are not copied from from the same image at least the ones that we have looked at they're usually you know you go you find something on the internet you put you know picture of the Pope or Putin or whatever or Trump yes yes so in this thing we're basically


where we're focusing on what's called image splicing where you have two sources and then you create an image out of those two sources yeah so if you if you move things within the same image it might still possibly detect things we have seen a couple of examples where that happens of kind of the copy/move thing because it sometimes it screws up the the JPEG compression for example but


in general it's not it's not trained for that it's not it's that's not what it's looking for it's looking for really different imaging pipelines so it's not it's not supposed to be working for that example for that thing yeah mm-hm allow me I don't hear it that close doing videos I will I will I will repeat yes so the question is if we have tried it on on videos with people imitating


other people different voices we haven't but I think this is this is yeah we can or any of the the papers online the code is online so anyone can actually run it and see that would be an interesting thing to to run and see we have played with them very very ventriloquist's so when you have a puppet and you talk as a puppet and then you talk as as yourself and it kind of works it kind of works


for that so hopefully it should be it should be doing something like this yes it's not it's not meant to be kind of a detective detecting detecting fakes it's it's really mad it's meant to be fooled by the same things humans are fooled by so if the impersonator is a good one hopefully our method will work too yeah yeah question about this domain to domain mappings and you have to like one


directional and cycling example Google Maps in the wandering back to them the paintings were in the cytosine in the cytosine you care about the consistency that you come back to the center and you say the painting were actually working only in one direction the pay they the the paintings work in both directions paintings two two two photos they just don't look so good so


III didn't well I didn't show the results that didn't didn't work very well so remember in them in the Google Maps you go to the Google page and you can you can copy you can you can you can you can copy the the Google map and then you switch to the satellite and you can copy the satellite of that same thing so you have aligned inputs and outputs right so this is a much simpler problem


so there you don't need any consistency because you have the X in the Y's given to you and so then you can just do it directly with the paintings and them the photographs there is no alignment you have the suzanne paintings and you have photographs and you don't have a one-to-one correspondence and so for that the cycle is is is imported so that's the only thing we have basically


we there is no other constraint so then it's very portable and we come back to the polarization for example okay then the colored image is not strange today oh it is it is because there is a there is a hold on the the gun yeah so so let me see let me see zouri so in this in this setup you are given the input and the output okay and you are your basically where is the picture your


training on the grayscale and the color okay so you're given the input and the output and the Gann is just making sure that the output that the generator produces is similar in the perceptual space to what we humans expect okay because if you don't have the Gann then then you get weird results like this one right this one right this is what happens if you just kind of do


regression with the sake something like l2 loss yes yeah I I did not mention that I I'm sorry yes it does have the kind of this standard l2 loss plus the gamblers yes you're right I'm sorry I yeah I dropped out that that's like yes so for this you have the Delta loss plus the Gann loss in the second one with the cycle we don't have in the l2 loss because there


is no there is no data for that yes yes I'm sorry yes this good good point yes mm-hmm no it'sit's true I think I think it's it's it's not it's not clear what heat map what do you want to call her because again the the the producer of the sub for example when you're when you're playing the the organ the producer of the sound is the big the big pipe that's the thing produces the sound but of


course what what it's being it's it's coloring is the guy player pressing the keys and so I think it's it's not a very clear what is it that we want you know do we want the actual physical thing that produces the sound or do we want the the actuator and so here I think we are not we're not we just wanted to see what what would it be sighs so we are happy with anything that connects with


with sound production in some way or the other but yeah it you're completely right it's not I think if I think it's really it's it's going to be looking at correlations not causations and I think actually I would be happy if the dance party thing if it shows the people dancing I think that would be actually pretty cool but yeah you're completely right it's not a it's just kind of a


type of visualization that that shows okay this these are the pixels that connect [Music] so the question is I think this is a very very very good question that that there was a little bit of a slide of hand I said that you know the the tasks were too easy and the computers were easily cheating on the kind of the classification tasks and then I ended up making tasks harder but also changing


the tasks to something like you know pretty pictures or connecting audio and visual etc etc what about going back to you know detecting car detecting dogs and and and classification so I think I personally am NOT a big fan of the classification task to begin with I think it's a it's a it's a task that is designed to to be cheated on because it's a task that basically assumes that


you have a close do you have a close the world your world is a thousand classes and you're basically deciding one of the thousand things right so your your your your your chance performance is actually not that bad your chance performance on on something like image net is is one in one and two hundred right it's it's actually the chance is pretty high in the real world we're in the open world


where the potential number of that you need to recognize is almost infinite okay and so I think that actually a lot of the problems with with with the with with the come with these networks cheating is because we're testing them on tasks that are very constrained we're cheating is actually actually the right thing to do so we're testing them on something that is that is not that that


that that that is a kind of a very it's a it's a it's a it's a specialist task right what I think these methods will excel at is the generalist tasks something where you train on something and then you you you apply to something completely different and hopefully it will work better and we have already seen this like for example the colorization feature that we have


trained it does better than imagenet if the task is for example then to predict depth from a single image right so if the task is very different from the task it was trained on the cell supervised features work better if the task is similar to what it was trained on then the semantic tasks work better so I think that the the big goal is that we want to produce a generalist computer


something that is able to deal with novel situations in a reasonable matter not something that we already have if your goal is to just learn a specialist that will tell you no different types of of you know of Viennese pastries from each other you know you have a thousand different pastries and you want to tell name all of them then I think the current direct supervision methods are


exactly the thing you need to do give in this kind of closed world but if you want to have a general algorithm that can do the pastries it can do the flavors of gelato or it can you know tell cooker from from Malevich then the hope is that these kind of more general methods should get you a better of doing this so it's the difference between doing well on the exam versus


I want to talk to you about the future of medicine. But before I do that, I want to talk a little bit about the past. Now, throughout much of the recent history of medicine, we've thought about illness and treatment in terms of a profoundly simple model. In fact, the model is so simple that you could summarize it in six words: have disease, take pill, kill something.

Soon We'll Cure Diseases With a Cell, Not a Pill | Siddhartha Mukherjee | TED Talks


Now, the reason for the dominance of this model is of course the antibiotic revolution. Many of you might not know this, but we happen to be celebrating the hundredth year of the introduction of antibiotics into the United States. But what you do know is that that introduction was nothing short of transformative. Here you had a chemical, either from the natural world


or artificially synthesized in the laboratory, and it would course through your body, it would find its target, lock into its target -- a microbe or some part of a microbe -- and then turn off a lock and a key with exquisite deftness, exquisite specificity. And you would end up taking a previously fatal, lethal disease -- a pneumonia, syphilis, tuberculosis --


and transforming that into a curable, or treatable illness. You have a pneumonia, you take penicillin, you kill the microbe and you cure the disease. So seductive was this idea, so potent the metaphor of lock and key and killing something, that it really swept through biology. It was a transformation like no other. And we've really spent the last 100 years trying to replicate that model over and over again


in noninfectious diseases, in chronic diseases like diabetes and hypertension and heart disease. And it's worked, but it's only worked partly. Let me show you. You know, if you take the entire universe of all chemical reactions in the human body, every chemical reaction that your body is capable of, most people think that that number is on the order of a million.


Let's call it a million. And now you ask the question, what number or fraction of reactions can actually be targeted by the entire pharmacopoeia, all of medicinal chemistry? That number is 250. The rest is chemical darkness. In other words, 0.025 percent of all chemical reactions in your body are actually targetable by this lock and key mechanism. You know, if you think about human physiology


as a vast global telephone network with interacting nodes and interacting pieces, then all of our medicinal chemistry is operating on one tiny corner at the edge, the outer edge, of that network. It's like all of our pharmaceutical chemistry is a pole operator in Wichita, Kansas who is tinkering with about 10 or 15 telephone lines. So what do we do about this idea?


What if we reorganized this approach? In fact, it turns out that the natural world gives us a sense of how one might think about illness in a radically different way, rather than disease, medicine, target. In fact, the natural world is organized hierarchically upwards, not downwards, but upwards, and we begin with a self-regulating, semi-autonomous unit called a cell.


These self-regulating, semi-autonomous units give rise to self-regulating, semi-autonomous units called organs, and these organs coalesce to form things called humans, and these organisms ultimately live in environments, which are partly self-regulating and partly semi-autonomous. What's nice about this scheme, this hierarchical scheme building upwards rather than downwards,


is that it allows us to think about illness as well in a somewhat different way. Take a disease like cancer. Since the 1950s, we've tried rather desperately to apply this lock and key model to cancer. We've tried to kill cells using a variety of chemotherapies or targeted therapies, and as most of us know, that's worked. It's worked for diseases like leukemia.


It's worked for some forms of breast cancer, but eventually you run to the ceiling of that approach. And it's only in the last 10 years or so that we've begun to think about using the immune system, remembering that in fact the cancer cell doesn't grow in a vacuum. It actually grows in a human organism. And could you use the organismal capacity, the fact that human beings have an immune system, to attack cancer?


In fact, it's led to the some of the most spectacular new medicines in cancer. And finally there's the level of the environment, isn't there? You know, we don't think of cancer as altering the environment. But let me give you an example of a profoundly carcinogenic environment. It's called a prison. You take loneliness, you take depression, you take confinement,


and you add to that, rolled up in a little white sheet of paper, one of the most potent neurostimulants that we know, called nicotine, and you add to that one of the most potent addictive substances that you know, and you have a pro-carcinogenic environment. But you can have anti-carcinogenic environments too. There are attempts to create milieus, change the hormonal milieu for breast cancer, for instance.


We're trying to change the metabolic milieu for other forms of cancer. Or take another disease, like depression. Again, working upwards, since the 1960s and 1970s, we've tried, again, desperately to turn off molecules that operate between nerve cells -- serotonin, dopamine -- and tried to cure depression that way, and that's worked, but then that reached the limit.


And we now know that what you really probably need to do is to change the physiology of the organ, the brain, rewire it, remodel it, and that, of course, we know study upon study has shown that talk therapy does exactly that, and study upon study has shown that talk therapy combined with medicines, pills, really is much more effective than either one alone. Can we imagine a more immersive environment that will change depression?


Can you lock out the signals that elicit depression? Again, moving upwards along this hierarchical chain of organization. What's really at stake perhaps here is not the medicine itself but a metaphor. Rather than killing something, in the case of the great chronic degenerative diseases -- kidney failure, diabetes, hypertension, osteoarthritis -- maybe what we really need to do is change the metaphor to growing something.


And that's the key, perhaps, to reframing our thinking about medicine. Now, this idea of changing, of creating a perceptual shift, as it were, came home to me to roost in a very personal manner about 10 years ago. About 10 years ago -- I've been a runner most of my life -- I went for a run, a Saturday morning run, I came back and woke up and I basically couldn't move.


My right knee was swollen up, and you could hear that ominous crunch of bone against bone. And one of the perks of being a physician is that you get to order your own MRIs. And I had an MRI the next week, and it looked like that. Essentially, the meniscus of cartilage that is between bone had been completely torn and the bone itself had been shattered. Now, if you're looking at me and feeling sorry,


let me tell you a few facts. If I was to take an MRI of every person in this audience, 60 percent of you would show signs of bone degeneration and cartilage degeneration like this. 85 percent of all women by the age of 70 would show moderate to severe cartilage degeneration. 50 to 60 percent of the men in this audience would also have such signs. So this is a very common disease.


Well, the second perk of being a physician is that you can get to experiment on your own ailments. So about 10 years ago we began, we brought this process into the laboratory, and we began to do simple experiments, mechanically trying to fix this degeneration. We tried to inject chemicals into the knee spaces of animals to try to reverse cartilage degeneration,


and to put a short summary on a very long and painful process, essentially it came to naught. Nothing happened. And then about seven years ago, we had a research student from Australia. The nice thing about Australians is that they're habitually used to looking at the world upside down. (Laughter) And so Dan suggested to me, "You know, maybe it isn't a mechanical problem.


Maybe it isn't a chemical problem. Maybe it's a stem cell problem." In other words, he had two hypotheses. Number one, there is such a thing as a skeletal stem cell -- a skeletal stem cell that builds up the entire vertebrate skeleton, bone, cartilage and the fibrous elements of skeleton, just like there's a stem cell in blood, just like there's a stem cell in the nervous system.


And two, that maybe that, the degeneration or dysfunction of this stem cell is what's causing osteochondral arthritis, a very common ailment. So really the question was, were we looking for a pill when we should have really been looking for a cell. So we switched our models, and now we began to look for skeletal stem cells. And to cut again a long story short,


about five years ago, we found these cells. They live inside the skeleton. Here's a schematic and then a real photograph of one of them. The white stuff is bone, and these red columns that you see and the yellow cells are cells that have arisen from one single skeletal stem cell -- columns of cartilage, columns of bone coming out of a single cell. These cells are fascinating. They have four properties.


Number one is that they live where they're expected to live. They live just underneath the surface of the bone, underneath cartilage. You know, in biology, it's location, location, location. And they move into the appropriate areas and form bone and cartilage. That's one. Here's an interesting property. You can take them out of the vertebrate skeleton, you can culture them in petri dishes in the laboratory,


and they are dying to form cartilage. Remember how we couldn't form cartilage for love or money? These cells are dying to form cartilage. They form their own furls of cartilage around themselves. They're also, number three, the most efficient repairers of fractures that we've ever encountered. This is a little bone, a mouse bone that we fractured and then let it heal by itself.


These stem cells have come in and repaired, in yellow, the bone, in white, the cartilage, almost completely. So much so that if you label them with a fluorescent dye you can see them like some kind of peculiar cellular glue coming into the area of a fracture, fixing it locally and then stopping their work. Now, the fourth one is the most ominous, and that is that their numbers decline precipitously,


precipitously, tenfold, fiftyfold, as you age. And so what had happened, really, is that we found ourselves in a perceptual shift. We had gone hunting for pills but we ended up finding theories. And in some ways we had hooked ourselves back onto this idea: cells, organisms, environments, because we were now thinking about bone stem cells, we were thinking about arthritis in terms of a cellular disease.


And then the next question was, are there organs? Can you build this as an organ outside the body? Can you implant cartilage into areas of trauma? And perhaps most interestingly, can you ascend right up and create environments? You know, we know that exercise remodels bone, but come on, none of us is going to exercise. So could you imagine ways of passively loading and unloading bone


so that you can recreate or regenerate degenerating cartilage? And perhaps more interesting, and more importantly, the question is, can you apply this model more globally outside medicine? What's at stake, as I said before, is not killing something, but growing something. And it raises a series of, I think, some of the most interesting questions about how we think about medicine in the future.


Could your medicine be a cell and not a pill? How would we grow these cells? What we would we do to stop the malignant growth of these cells? We heard about the problems of unleashing growth. Could we implant suicide genes into these cells to stop them from growing? Could your medicine be an organ that's created outside the body and then implanted into the body?


Could that stop some of the degeneration? What if the organ needed to have memory? In cases of diseases of the nervous system some of those organs had memory. How could we implant those memories back in? Could we store these organs? Would each organ have to be developed for an individual human being and put back? And perhaps most puzzlingly, could your medicine be an environment?


Could you patent an environment? You know, in every culture, shamans have been using environments as medicines. Could we imagine that for our future? I've talked a lot about models. I began this talk with models. So let me end with some thoughts about model building. That's what we do as scientists. You know, when an architect builds a model, he or she is trying to show you a world in miniature.


But when a scientist is building a model, he or she is trying to show you the world in metaphor. He or she is trying to create a new way of seeing. The former is a scale shift. The latter is a perceptual shift. Now, antibiotics created such a perceptual shift in our way of thinking about medicine that it really colored, distorted, very successfully, the way we've thought about medicine for the last hundred years.


But we need new models to think about medicine in the future. That's what's at stake. You know, there's a popular trope out there that the reason we haven't had the transformative impact on the treatment of illness is because we don't have powerful-enough drugs, and that's partly true. But perhaps the real reason is that we don't have powerful-enough ways of thinking about medicines.


It's certainly true that it would be lovely to have new medicines. But perhaps what's really at stake are three more intangible M's: mechanisms, models, metaphors. Thank you. (Applause) Chris Anderson: I really like this metaphor. How does it link in? There's a lot of talk in technologyland about the personalization of medicine, that we have all this data and that medical treatments of the future


will be for you specifically, your genome, your current context. Does that apply to this model you've got here? Siddhartha Mukherjee: It's a very interesting question. We've thought about personalization of medicine very much in terms of genomics. That's because the gene is such a dominant metaphor, again, to use that same word, in medicine today, that we think the genome will drive the personalization of medicine.


But of course the genome is just the bottom of a long chain of being, as it were. That chain of being, really the first organized unit of that, is the cell. So, if we are really going to deliver in medicine in this way, we have to think of personalizing cellular therapies, and then personalizing organ or organismal therapies, and ultimately personalizing immersion therapies for the environment.


So I think at every stage, you know -- there's that metaphor, there's turtles all the way. Well, in this, there's personalization all the way. CA: So when you say medicine could be a cell and not a pill, you're talking about potentially your own cells. SM: Absolutely. CA: So converted to stem cells, perhaps tested against all kinds of drugs or something, and prepared.


I'm excited to be here because I looked at the class syllabus and it looks really good so it's super relevant so my name is hike I work at Sky do I leave the autonomy team there Skye do is a start-up in Redwood City that makes vision based fully autonomous drones this is the drone we just launched the sky do - and it's it's it does a lot of stuff that you've been learning in this

Lecture 23 Guest Lecture: Drones -- CS287-FA19 Advanced Robotics at UC Berkeley


class and so I mostly just want to show a bunch of visualizations talk about the way we think about things and please ask lots of questions because that's that's the whole point so I'll just pass this around if you guys want to take a look oh we go I don't think I'm allowed to fly in here but I would and you'll have to excuse my moustache but it is November so okay so just to get you guys


interested I'm gonna show the first like I don't know 40 seconds of our of our launch video for this guy which we launched just boy a month ago and we're shipping out right about now so yeah all right so you can watch the rest it's pretty good but I'm just going to talk very quickly about the point of our company and then just jump into technical stuff so basically drones are


really useful because you can have an eye in the air moving around in 3d and that's useful for just a ton of different industries and this is nothing new the problem is that I think the technology for them to be safe and trustworthy and intelligent in the wild in enough places hasn't been there and so that's the point of sky do from day one it's like let's make the technology


for this type of robot work such that it can go out and do things in the world so basically we've we've been invested in this like building the whole stack and the reason we build the hardware is because like the whole thing just matters like you care a ton about weight and size and power and cost but also just about the the quality of the cameras the exact like vibration


characteristics of the thing the electronics in play the power system the embedded system and the software on top that runs like controlling the whole thing makes it possible to make to like push the limits of robots and and that's why we do it and so so not just you know this is the consumer launch that that videos regarding you know like this awesome video capture for anyone without


needing to be an expert pilot but we're but it also makes a ton of sense for any task where you're inspecting things like bridges and cell towers and pipelines and rooftops anywhere that's dangerous anywhere that's costly and you need to build a map of things so for construction for mining it it it's it's a long list so as long as you have something that it's trustworthy intelligent can


automate tasks and get get even just video data it's incredibly useful so that's that's our goal this is another slide that says the same thing okay and then here's just a view of kind of the more commercial side of this so the idea of being able to get up inside kind of these high voltage trusses and power lines and take close inspection photos this is something that's very hard


to do manually and it's and it's dangerous and these things need to get inspected all the time it's incredibly expensive and then on the right is us being inside kind of just this narrow bridge truss with metal on all three sides so something you really can't do with GPS it it's just it's not possible cool and then finally I'll show that this product that we just kind of


announced that we're super super excited about which is all the dock also commonly referred to as the box which is essentially just a cozy home for the drone that is weatherproof and basically it pops out the drone flies and does a task comes back and lands and goes in and the things cloud connected and the whole idea is like you have this robot out somewhere on your construction side


along along a pipeline and the drones go out and do tasks on a regular schedule and they come back and the whole thing is hands-off and it's truly like just just an autonomous thing that's that's doing work and so it becomes more of a cloud platform where people can use that rather than you know somewhat someone is piloting this robot and in control of it and that's kind of that's how we see the


the long long-term play of this going cool so here's a video that just shows it's just a clip of a mountain biker that we think is a pretty good shot of just autonomous visual tracking following so the drone is doing all this computation of course onboard in real-time based on visual tracking and we're able to kind of get some nice cinematic video here with tree


occlusions with fast motion and here we're getting a shot from the front which we call lead mode which is pretty tough because it means you have to do a lot better job anticipating someone's emotions so I want to just talk about aspects of like how this works and just build it up and again please ask lots of questions okay so if if you look at the drone it's got so


it's got the main 4k camera on a 3-axis gimbal and that's kind of what we were looking at here and then it has six navigation cameras so you can look at the geometry of the drone there's there's three on top and three on the bottom and they're actually these crazy super fisheye cameras so they see 200 degrees so they see like beyond a hemisphere which means the three on the


top can basically see the upper hemisphere in trinocular view and then the three on the bottom can see the bottom hemisphere in trinocular view and the goal is to get like 360 coverage so we can do 360 you know obstacle avoidance and awareness of the scene so this is what the this is what one of the cameras looks like on the bottom and one on the top the other four are pretty


similar to these two so you can see the it's the view is pretty funky like this is this is one of the propellers that's the battery on the top that you're seeing from the bottom right side where this is forward and then the this fin is actually if you get the drone you can see it's got this little by the camera it's got this fin which protects the camera lens if you were to like drop it


down and it looks a lot bigger here than on the drone itself but that's because it's just really right next to the camera and then this is the other yeah I think the other arm is over there that you're seeing from one of the top cameras so obviously it's this kind of tiny planet distorted view but this this is what we work with and the cameras are rolling shutter so doing computer vision


on rolling shutter cameras is is a really hard problem because what basically what that means is every row is kind of taken at a different time which means if you're undergoing fast rotation especially but even fast motion you're the contents of the image are as if you know time is changing in them so to account for that correctly you have to you have to be very careful


so here's here's what it kind of looks like with the navigation cameras put together to create a 360 view so if you see the clip we were watching before is on the top left and then but that is of the full sort of sphere where you can see it's just this yellow outline is that does that make sense so having so the navigation cameras give a lot more context and that's you know that's


that's what we use for all the algorithms so the core the core things we basically need to do to make this drone fly our one state estimation so we need to estimate our trajectory how how we're moving in the world and the second thing is obstacle avoidance building a dense map of the environment in a way that we can use to to navigate the third thing is detection and tracking of


objects so in this case the biker seeing them in the image estimating than 3d predicting their trajectory and the fourth thing is motion planning so with that sort of understanding of the scene want to figure out what is the best flight path that's sort of in this case provides a really cinematic video and is also not crashing it's also smooth and respects the physical dynamics of of the


vehicle like being able to push the limits of aggressiveness but with if you need it but without sort of commanding something that is not possible by the physical vehicle okay and then here is the same 360 view with with a depth estimate at the bottom so this is one way of visualizing the like full output of our of our depth estimation system where the close pixels are more yellow


and so you can imagine how that is really useful for obstacle avoidance because you can directly kind of feed into the the planning system as a 3d map I'm visualizing you here in this we call us an echo rectangular view because it's a nice way to show the 360 but this is not actually how we use the system online okay and then here's one more way of viewing like a single


instance in time here it's a it's a point cloud view that is the kind of colored by the pixels themselves this guy appear that's the drone that's the trajectory the drone was flying and then here is the person's trajectory that makes sense and then we have you have a scene around us and then it's going to switch to a depth view so instead of being colored by the pixels themselves


they'll be colored by the range from the vehicle so the distance of each point from the vehicle similar to what we were looking at before but now in 3d okay cool so I'm going to talk a little bit about what we need to do in terms of state estimation and this was actually our first drone the r-1 it's a picture of but anything that is you know could potentially be variable on the drone we


want to model an estimate so the the the obvious one is the trajectory of the robot which be like a pose in space so a position velocity orientation angular velocity over time of the robot and we do this with our visual inertial odometry system I'll show one slide after this we also have several I am use and so accelerometer gyroscope which is super useful for propagating you know


short term data it really only works to integrate the IMU for a matter of at least the accelerometer for maybe a second beyond that on a drone and with the sort of size and and cost constraints it is it's too noisy but I'm use have biases and estimating them correctly is is key to being able to effectively use them and then we have the poses of the cameras on the vehicle


so we you know obviously have CAD of the of the lenses and they get calibrated Factory time and we take that and we put that into our model of you know how we're gonna go from things in the image in 2d to 3d and the problem is those move around so every vehicle is a little bit different just because the physical characteristics so they get calibrated in the factory by a robot that waves


them around a bunch of tags and we know how to model that we have a model for the camera lenses and we calibrate that and also their poses the problem with that is that they also move around when as the vehicle undergoes temperature changes especially so it's a magnesium frame with plastic pieces and those things expand and contract and any small amount of rotation like even a small


fraction of a degree significantly changes the actual results you get when something is 20 meters away so that the problem is that the ability of estimating depth or really doing odometry from things that distance it depends on the inverse range of the the object you're looking at and so as you get that far out you're really looking for fractional differences between a


fractional differences of a pixel between multiple images and that makes you that makes it so that you have to be very careful about your calibration the yes so so we we calibrate the poses of the cameras and particular it's mostly about the rotation like the actual translation of the cameras moving it does not have a large effect on the results it's mostly it's mostly


rotations but we calibrate those things online sore joint ly with our odometry system so there there are many variables in the optimization some of which are that the pose of the robot but also the poses of the cameras and the biases the IMU and additionally we calibrate the intrinsic parameters of our lenses which is which is kind of nuts thing to do on line but those also change I'll show a


video of that here so this is showing a temperature sweep of a fixed lens and you can actually see sort of these circles are fixed but you can see how much it moves is actually set pixels of motion there over the change and those are the like intrinsic lens parameters and the distortion model of the camera and so to really handle this we also estimate these online with


everything else which is a cool and a hard problem and then of course we have propeller spinning so we've got to estimate the rates of those propellers and then we have the gimbal and we estimate a combination of ways exactly where the user cameras pointed because while we have you know we have sensors on the gimbal that tell us where we think it is and we've commanded it to be


somewhere it's also on a an isolation system and there are other changes there so we also additionally do visual estimation of between the user camera and the navigation cameras to to line those things up cool so here's just a quick visualization of the visual inertial Adamo tree system so on the bottom left are some features that are being tracked in this case they're their sparse


features just tracked over time in one camera and the right is the estimated trajectory of the robot so I I think you've guys have gone through kind of a traditional slam pipeline is that right in this class sort of maybe ok well anyway it's yes so so what you get is a bunch of 2d observations in your camera images by tracking in your images and you can track either between cameras at


a single time instant or you can track over time and really you want to kind of do both and there's different characteristics of what works well but then you take that and you are jointly optimizing for a 3d locations of those landmarks that agree with the observations you've made in the images while also solving for your own pose and to the the residual and this kind of


objective will be the reprojection of that 3d point into your camera model for your pose and those are all variables in your optimization result in a pixel coordinate and then you would compare that pixel coordinate to the pixel coordinate at which you actually observed that feature in that camera and with multiple observations of what you think is the same 3d point that starts


to constrain things in this system okay still know okay so here is here's a little bit of what rolling shutter can look like in this case it's rolling shutter combined with vibration it slowed down a lot it's a little subtle to see here but you can see these like Wiggles we call it wibblewobble the yes so the the car you know you can you can see the windshield of the car sort of


waving around and that if you're not modeling that if you're not accounting for it that's just pure noise that's going to mess things up for you in a way that does not match geometric constraints the the the way you if this is happening to you you have two choices choice one is to like improve your hardware improve your system such that you don't get this issue like there


might be a certain safe frequency of vibration that is related to the the time it takes to read out your camera and you get some periodic signal option two is to have a way of estimating exactly how the camera moved during this time period and accounting for that which requires very careful work and we do both of these things but it's once once you're trying to fly like at high


speeds in aggressive scenarios like we really really have to be pushing the edge of what what we can model there and what we can estimate okay cool this is one of my favorite slides because I think it talks about what is what is really hard about computer vision at least computer vision in a production robot so the so a here is just thin thin branches so flying in


a forest right now it's winter we're gonna have a bunch of customers they're gonna take their drones and I'm gonna try to follow themselves riding a bike in the forest and there's going to be all kinds of terrifying thin branches that you can barely see and we we work really really hard on being able to see that from the cameras I'll make the bold claim that I think we're almost


definitely the best in the world that seeing really fine thin cameras from their branches from camera images and when you're flying at you know you kind of thinking about like what's the resolution of my camera how fast am i flying what's my physical reaction time to either steer away or stop it's obviously not guaranteed you have a way to steer away so you could really think


you'd be thinking about okay if I'm flying at 35 miles per hour and there is it like what how far away do I have to see an object to be able to stop if that's like if it suddenly pops in and and that will that kind of plays into how how faint does it like what is the smallest perceptible signal you can have in your camera images before it like you have a chance of seeing that


and I think we're we're pretty close to the limit there of like if if you can as a human like look at the camera image and and see something like we probably see it as well but it's still like we're we're up against that edge of just how much signals can we get out of that so be here is is a really fun scenario where there's a bright Sun so the the tough lighting conditions make


everything harder right so in this case there's a low Sun and we're in a forest and particularly are actually images from our one the first the first drone but it's it's very similar so the slide on the left the camera on the left is I think has a smudge on it like a fingerprint on the lens which basically makes just like light goes differently into the camera and and the


left and right cameras look very different there what that means is that if you're looking to if you're looking between the images for a photometric signal that they're they're not agreeing at all so typically what you you know might do for stereo vision algorithm is look at two cameras and look for you know look along and if you pull her line and look for something where the you


know say a small patch of pixels looks the same in the left and right camera the problem is if you're up over here here you just got these crazy vertical sunrays that will match other sun rays in a totally inconsistent way but not your actual contents so it's a huge source of noise and especially when it's it's different between the two cameras and then C is just motion blur so that


drone can you know like that's kind of our decision unless it hits something and flips over but to allow very aggressive rotations up to several hundred degrees per second which induces motion blur in the cameras D is just reflections and mirrors and I really don't have a good solution for that like if you if you fly this drone in front of a panel of glass that's like three


meters by three meters wide like it it will you know it will see what's inside and and think it can it can fly in the the hope of doing this like semantically is it's it's tough it's really tough and then mirrors of course like in this case a reflective building and so we typically hope is that these vertical struts we're able you know those make geometric sense and would prevent us


from flying into it but the part of the mirror that's just reflecting the building behind well that just looks like an opposite world that's geometrically consistent and so our depth map will reflect that it thinks that these are that these are basically poles and then there's something behind there if you look closely you can kind of see there's some blinds in here so horizontally and


so at some at some point of like if you can see what's in the window and you can see what's behind you have to kind of there's like two potential realities for what that could be and it's it's an interesting problem to think about when when that's which would happen and which taking a stronger signal okay so E is just big textualist surfaces so let's say the drones like right up against the


ceiling is typically the the hardest of cases and really only sees what he looks like so it's just white and that's also what the sky looks like in a lot of scenarios now sometimes the sky is bright blue but not not always often it's just basically saturated whiteness from the cameras and deciding which of those two is is the case is a really hard problem the best thing that can


help you there is like the fact that we have full 360 context on the drone means that the rest of the scene can tell us oh is this more likely to be an indoor scene with a room or is this going to be you know we're flying out in an outdoor scene and that the reason it matters so much is because well the sky is one of the best directions to fly in like it's it's it's usually very safe and so if


we're if we're trying to get away from something and try to figure out how to move like going up is usually a very good decision but not if it's a ceiling then you're gonna have a bad time okay and then f is water so there's just water droplets on the lens I mentioned smudges from your fingerprint also dirt also dust also cracked lenses just all manner of things that inhibit the sensor


and this is something that we don't particularly control right the user is is responsible they get for cleaning the lenses but they take this drone somewhere we've never been and we asked them to wipe the thing and we and we detect these things online like that's the best thing we can do so actually as a drone is flying we look for potential issues with what the


cameras see and and having dirty lenses and we have sort of we have a lot of these types of warnings but this is an interesting one particular is like we have multiple levels of like oh it looks like your lenses are dirty like you should probably land and clean them and try again and you can dismiss that and most people just dismiss it and don't listen and then there's another level


that's like okay you really have dirty cameras like we're not gonna let you say track anymore so the drone just kind of like goes into a hover it's like oh it's not safe but we can't just make the drone land or something because the drone could be you know a couple of kilometres out over the ocean or something and just hovering there or landing is not a good acceptable


solution and just coming back on its own is also dangerous so what we sort of chose to do there is for the drone to just hover and you can control it manually on your phone or with a controller but it won't like track you because then it's sort of more autonomous and you're oblivious to the fact that it's it's performance is degraded for sure so we we care about


lots of those types of robustness issues because it's just like getting getting to a demo that works of like okay you know we have we have this drone it films somebody it avoids obstacles like that is much much easier obviously than getting to the point where you can actually have something where someone paid for a product and they take it somewhere new and they like initially


don't trust it but what we typically see is after like one or two flights of you know no like no problems it works and in the vast majority of you know scenes you're not gonna have any issue but you quickly just gain way too much confidence in the robot just to work anywhere anytime and that's something that we kind of struggle with is how to like communicate limitations in an


effective way because it does work so well in so many cases okay talk a little bit about mapping so and by talk I mean show this video so these are just you know vark like a voxel representation over the user camera video basically just to to look cool so in terms of dense mapping so that the goals from state estimation are different right because for estimating


how we're moving basically we have we have cameras in every direction and you don't need a lot of points like you need a little bit of information about other things are moving that are static and you track them well the which means if something is very difficult like crazy tangle of thin branches you can just ignore that part and look at the ground or look at the tops of the trees and


that's enough to tell you how you're moving but for actually mapping things like we need to pay attention to every single direction and make a decision about that saying oh this directions like too hard to look at like you can decide to never fly in that direction but that's that's very limiting so we published one paper here a couple of years ago with research in turn and Alex


Kendall and the the paper was basically trying to combine a classical stereo matching pipeline like geometric reasoning about stereo vision with with deep learning to learn features and do the matching so at the time we we did really well on these the kiddie self-driving data set which we submit it to and the basically the bottom one here is the ground truth of that data set


which is a lidar scan and in the middle is a as a prediction of our of our network and so it it worked really well in in these scenes I'm not going to go into details here but essentially the the network was structured in a way to take advantage of sort of intuition about stereo matching and classical stare matching algorithms where we have we compute shared learned weights on


each of the two images with two towers and then we have a cost volume approach where kind of mix those features together at different possible disparities and then we regularize the cost volume with 3d convolutions and then we we pick a winner so this this worked really well but I'll kind of give a couple of caveats the first caveat is it ran at one frame per second on a Titan X GPU


which meant like obviously we can't use it it's orders of magnitude off and the second caveat is that frankly the the kitty data said or really like my experience any self-driving data set that I've seen are are incredibly easy compared to the types of scenes that drones tend to see in in the wild and the reason is because of regularity so most of the images just look very


similar like you have a road scene you have cars there's always a ground plane there's things on the sides there are street signs that are vertical and there's just a ton of semantic priors you can do this is what monocular depth estimation works really well on driving scenes as well when things are you know when things are normal now that's not always the case but for a drone you can


be flying really like really in any nook or cranny of any building or while and and you really can't tell what's in a lot of the camera images and there's just no there's no good monocular prior a lot of the time because you could just be in inside some random geometric shape in a sculpture and art gallery or something and so it's it's a much harder problem for that and and specifically


the the metrics on a dataset like kitty where you're typically measured on like how how accurate is like the average pixel error across the image right like I can't emphasize enough just how useless that is for robotic navigation purposes like being being 1% closer like it's basically just dominated by you know how well do you estimate the ground plane and the exact pixel values of the


depths and if you have something like a thin object that you miss or an entire vehicle but it's further away or something like it has very little to impact on the score and so focusing on metrics that really matter for autonomous navigation were things like 3d occupancy and just thinking about only the parts of scene that matter and in the end like in the in the Berkley


way that kind of one of the most powerful ways to do that is to think about the sort of action space like what are the parts of the scene that actually matter for making a decision rather than doing it allocating your error metric by pixel values which is totally simple to think about but but pretty meaningless okay so since then we've made it a lot faster and a lot better and it's also


changed in many many ways but basically we are flying deep networks that produce depths and build an obstacle map for us and it runs at approximately a thousand times faster than the previous sort of first thing that I mentioned and if she think we were the first ones to fly a robot based in a complex scene based on sort of a deep learned obstacle avoidance system about a year and a half


ago this was our little ATV flight yeah yeah that's a good question basically we we do a bit of both like we aggregate the images but the the dominant part of the deep learning estimation right now is is between the sort of cam like the camera is at an instant in time but we have definitely we definitely do things that are between like temporarily as well there are different challenges into


doing that one of the challenges is moving objects so moving objects you know you you have to estimate that motion to geometric we get the correct thing right like if you're doing optical flow and then you assume things are static you'll get in correct depth in particular like if you're flying the same speed as somebody and following a biker say and they and and you do that


and you think they're static like it'll seem like they're far away because basically they don't move with respect to you and so the pixels you stay in the same pixel location which means which implies that they're they're basically at infinity so there's tricky things like that to take into account yeah I'm going to show some a couple of just clips of the sort of hard things we


focus on so this is just a forest with a bunch of branches and and looking like really what you care about here is like did you see this object or not and generally the like 3d extent of it but the exact the exact details of of the shape really don't matter very much although it does tend to be correlated with you know the quality of your predictions here's a here's a 3d view of


that same thing where we have the camera image on the bottom left and then the depth on the bottom right okay power lines are real and something to be worried about so okay you can see it's a little hard to tell here oh these aren't matching are they okay well so you can kind of see if you trace this out there there are a bunch of power lines in this image so these ones


obviously are very clear you can see these this is a faint but you can see it over here it kind of fades into the background this is a power line that's quite faint to see and it disappears entirely behind the Sun here and then actually up here there's one that goes all the way across like that and it's it's very very hard to see so it's just giving some some intuition for that and


there's a second one here and there's a third one here and so really like it does depend a lot on the background and the lighting on everything just when it gets in your camera mijin and how much that works so again that's that's kind of the walls we tend to be up against trying to push the limits of where we can fly what we can see and how fast we can go especially and then about


textualist surfaces so here's actually flying through our office and you can see on the left here example it's a scene with you know specularity is on the walls which will be lying to you photo metrically and just a lot of a lot of white texture and then here is the same thing in 3d it's funny this blue glow on the bottom is actually because there's there are some different lights


on the drone but the in this case this this LED was on which is very close to this camera and it basically just shines this blue halo under right there on the ground but we we don't keep this one on in flight normally it's just these guys which have been you know designed to not be seen by the cameras the whole kind of cemented the whole geometry of the drone


and the inversion and everything is kind of built around enabling as much visibility as possible of the world without obstructions yeah the frequency so I can't say exactly what frequency is part of the reason is because it's sort of internal stuff and part illusion is because it it can change based on some different things so we we are pretty clever about some of the things we do in


terms of so you know what what resolution do you process stuff at how complicated of algorithms do you run or intensive and which like which directions do you focus on so all those things in terms of like down sampling and foliation and choosing the the extent of the algorithm you run are pretty important to finding the the best balance and so for example if you're


flying really fast at such that it's really dynamically infeasible for you to even move in these other directions for the purpose of navigation it doesn't really make sense to focus your efforts there you really want to kind of like pick the cone of just do tunnel vision there now the kind of in a different case like for example let's say you're let's say you're barreling down this way


at max speed but you're also filming something that's over there now you may want to focus on those two directions instead of just the one because you need to estimate the scene over there to make some decisions about your high-level objectives and then obstacle avoidance over here so things like that are our important games to play okay here's another fun one that's just artistic


this is just an example of someone basically pushing back on the joystick so we have we have a bunch of accessories but one is just it's a controller and you can fly and the idea is basically like you control a drone like normal you know kind of if you've ever flown these MOTU joysticks and the idea is that you don't have to be an extra pile like you can just you can


just push it forward and won't crash right it'll go around things it will it will do its thing so it's it's a really fun way to fly because it means you can you can either just have fun or do your task film something but you're not worried about physics or the drone or the crashing or things like that and that that's what this shot was basically just pushing pushing back on the sticks


okay yeah so this is can't tell if you guys can see this at all but this is kind of a typical example of what a thin branch looks like in an image so these are actually two images it's a quite a close branch and so get a sense of like here's here's the branch and then here's the branch so if you look closely it's it's coming out of the ground over here and then it comes all the way down very


close to the camera and so the actual correct sort of match of this stick is moving from here to here and then there's another one over here that's moving like that so if you if you look closely you can kind of get a sense of that and so this is just now maybe conveying a sense of the the type of problem here which is one of the hardest things we need to do which is just


carefully pick out objects and and there's kind of the outline in this image of what what we're looking at and actually the a real key here is like if you flip between two images you can tell but if you're just looking at a single image you can't like no no human can and really no algorithm can either like you can look at this kind of scene all day I'm a tangle of branches and not have


any idea what the closest thing in there is which really showed that that's kind of what I was getting out of the difference between a road type scene which is very regular and this type of scene here's here's another fun one where we're actually above powerlines so there's there's one power line here you can see that's moving there's another think really faint one here and then


there's two right here that you can see very clearly on this part of them disappear mostly up up in that section okay awesome so we have you know we can play with simulated data synthetic data and the cool thing about that is you know we can we know everything about it and we can try to make it realistic we also have real data and the problem with real data is we don't have any ground truth


for the depth which is a which is a very tough problem and there's a whole body of research that focuses on trying to learn from unsupervised data which is really interesting the problem is that slide I showed about all the visions the things that make vision hard none of those things are photo metrically consistent so if that's your cue to learn depth it's that's not a solved


problem right because the that that cue won't get you the really hard things so here's it's kind of a fun example of something where we took we took Suns like synthetic Suns that we generated and we just put them in real images and the kind of fun thing you can do there is to say oh well whatever you predicted before the Sun was there you should predict the same thing with the Sun so


that's kind of one way of getting at just without knowing everything at least gaining some robustness to a really big bright Sun okay let me show a little bit about tracking and detection so this is kind of a fun video of just tracking somebody through a forest with a bunch of lighting changes and occlusions and pose change so people change shape a lot and we're reproject ejector ii into the


image which you need the a pretty accurate estimate of the robots trajectory to do you can and then on the top right is actual output of a deep network that predicts the pixels of this person instance at that time step and then the bottom right is just that's just the top-down view of where where the drone was and where the person was and then the red vector was it's just an


estimate of the velocity so like what's in a linear sense where where's that person gonna go in the next next a little bit cool so the way we do this is kind of the way we do everything else which is a combination of geometric reasoning and and and deep learning for semantics and the images come to be combined with like 3d motion and things we know tend to be true about the world


so oh yeah well so multi object tracking is even if you're just tracking one person which is more common tracking multiple objects is really useful because you can then discriminate between people rather than just saying everyone else's background you can say oh this person a disappeared the person B matches person B from before rather than person B also just appeared and


which one's the right person so these are our three co-founders just running among each other and we're doing a pretty good job of keeping them in frame or keeping him as the correct estimate and then on the bottom oh boy oh boy okay on the bottom right here it's kind of a fun failure mode where it's hard to see but so able run behind Matt here and then or Adam and Matt pops out at the


same instant so like there's a person one person goes behind and hides and the other person that was hiding pops out and they're kind of at a similar velocity and so the the things you're combining are like where are they their positions they're their velocities someone's unlikely to just suddenly stop but it can happen and then they're like estimate the visual estimate of their by


appearance which we do with with deep networks and so in this case the the visual estimate is similar enough that like the chance of one person disappearing and another appearing and then being at the same velocity is like very low and it clearly yeah they're basically learned features so essentially it's based on the appearance of an instance of an object and there's


pretty solid body of literature about how to have to try to learn those things but you tend to have learned through examples of people that look there's different constraints based on what type of like use you want to do but typically it's like you have similar instances you have the same instance of a person but from many different views and poses and and lighting and all that and then you


have multiple people that look similar and you train positive and negative examples and you you learned that kind of task this is very common for like yes like surveillance camera datasets they tend to be easy to collect where they're just fixed now again the drone is a lot harder because it's moving and people tend to be moving very quickly and running away and doing flips and stuff


which make the problem a lot harder but the concept is pretty similar yeah oh yeah so in the just for the sort of consumer follow you skate based use take off and then whatever is in the view of the user camera you'll see just icons on the phone so we must we do people in cars and so you'll just see a little plus icon and you just tap and then and then it works and it shows that halo


that was in the launch video earlier on the phone we also have this thing called the beacon it's another accessory so this guy just a little puck that lets you basically fly without your phone and it has a it has a GPS that we trust a lot more and so what this does is if it lets the drone if it loses visual track and you view like you get stuck or something fails


it can basically try to just combine between that and the GPS estimate and then find you again and pick you up visually and so the idea is to just to make it much more robust in terms of if you're skiing or something that you you're just hands you don't have to worry about it and so the other way we do it which is kind of fun is if you just if you take off like this it will


take off and if there was someone standing right there it will just track that person okay cool say a little bit about planning so I think generally we we have a bunch of knowledge of the environment it's all uncertain there's no guarantees about anything because you know we're not sure if we saw the thin branch where we think we know where we are there's uncertainty


about all these things we try to reason about that and the best way possible in the end like we have some measurements and we need a policy that decides what the propeller should do such that the drone accomplishes our goals you know some of which are are down here like obstacle avoidance dynamic limits smoothness effective tracking and cinematic quality of the


video and these are all you know that there are hard trade-offs with each other it's it's not easy to achieve these things in all but the simplest of cases so I think it breaks down a way that I think you you've you've all kind of seen before so we won't go into this one but basically like we have the the biggest the most interesting part of the problem is figuring out what the cost


function should be based on everything we know and so based on the current state of things in the full environment what what are the actions we should choose that that meant like that give us the best result and defining that function is a lot of work and trial and error and investigation and that's that's something we just work at on just iteratively like trying things out


flying it seeing what fails trying to like figure out what our discrepancies between we really want and what we saw which is which is a really fun problem and then you have to find you actually you need to find an action that accomplishes that goal and that that's the cycle and then either you didn't have the right objective or you didn't find the minimum for that objective well enough to do


what you wanted and in terms of solving it I think you guys all know but like you can you can learn you know that's the the promise of deep reinforcement learning is that you can well potentially learn both of these things from data and you can do sampling techniques you need a trajectory library you can do a tree based search or you can do optimization and you can you can


do trajectory optimization and mile predictive control to solve these things and so we we visit we do a combination of all these things in a way but but as cool is like we we run something on board that runs at 500 races per second that is reasoning about kind of the whole stack between all the way up to like obstacle ones and cinematic quality all down to like what are the dynamic


limits on the rotors and the more you can reason about the whole system together rather than in separate little layers that have their own boundaries the more you can get out of the robot I think that's kind of universally true the trick is that you have to it's you have to be a lot more careful it's a more dangerous game to play and again we we also use sampling techniques and they


have they have a good role to play within the system then I'll just add a note about simulation because it's it's awesome and super useful especially for motion planning there's like for computer vision you want is like really photorealistic images that you've carefully curated for planning you don't care you just need something that like is good enough to see your behaviors and


try things out on so this is like you know we literally drive this guy around with a joystick at our computers and he just kind of slides around the world and we just used that to test basic things and then even without images we have these scenes where basically we generate a random forest and we honor we want to say compare we're going to figure out okay how far


away from obstacles should the drone stay to not crash into them versus like how often is it going to get stuck in a dense forest if we do that so we can like we tend to run logs against hundreds and thousands of with regressions against a ton of just recorded log data right so we record all of our flights all of the images and we can rerun all our vision algorithms and


check like those types of things but for planning you can't change what the robot did right it's it's it's all kind of off policy and so simulation is really helpful there and so we can run an experiment like I would generate a bunch of different random forests and have the subject run the same path through them and then see how often the drone hit something or get stuck and have a


statistical sense of what to do there the problem is always like something bad happens we look at that log and then like okay we can tweak this thing to fix it but then did we just make five other things worse and we can't like do we can't do a thousand real flights to test that in any sane way so maintaining the sort of infrastructure to handle all those things is really critical yeah


yeah yeah I mentioned I would fly in here but I don't think I'm allowed to yeah that's a good question we identified the dynamics model which is somewhere between them but we definitely model the dynamics of this drone it's not like a model free black box and the reason because it's definitely model vulnerable so it is you know the propellers there are aerodynamic effects


that we understand quite well there's the inertia's of the drone and properties of it and so we we model everything that we can and we try to fit parameters to it based on flight data and then I think that's that's pretty different from other types of robots that are more like especially when you're dealing with contacts and more complicated things but this is not that


many degrees of freedom there I think the complicated part comes from aerodynamics so once you're once you're flying fast and you're in turbulent flow and you get these types of like pitching moments and the accelerations your rotors play into things so that there is a lot of complexity there but it's also complexity that has been pretty well modeled in aerospace at least in terms


of the structure of something that you can learn so yeah yeah uh-huh we don't I mean you can't you can't ensure anything because well it yeah I guess it's a complicated question I does not exactly sometimes people ask like how do you guarantee that it's going to be obstacle free for example and and the answer to that is we can't because of perception and so nothing is ever guaranteed but we


we do our best so in terms of like if you have a nominee MVC how do you make sure that it converges well it depends a lot on how you're solving it if you there are good ways to know if you've made it worse than your initial guess for example so if you if you do that you can try to fall back to something else or you just like try to work towards robustness so


have have ways of just managing what you're doing such that you at least have some path that moves you forward and is not catastrophic but the best way to I think the best way to do that is just to try to carefully design your problem there constraints in the setup of what you have to to have you know stability and convergence and and there's there's a lot of there's a ton of work that