oh how likely so against the timid benchmark is a data set that many speech researchers work on this is one of those datasets where you know if you do point 1 percent better you write a paper um and a few years ago Holland was able to you know make what correspondent so I think we worked out something like like two thirds of a decade work for progress or something on this data set just by learning in this chart is outdated I made this child I think back when pauillac was publishing this paper since publishing this paper geoff hinton and


others have surpassed this also using deep learning techniques right um and then I was referring as was preparing this talk I best prefer practice I ask my students to help me put together a chart of the social results where you know we or others or whatever holders through the odd benchmark result using deep learning and there were surprisingly many of them from us Stanford from other groups on I say yeah I worked on machine learning for a long time I've never in my life seen anyone technology not go over benchmarks like this quickly this is the

Andrew Ng: "Deep Learning, Self-Taught Learning and Unsupervised Feature Learning"


whole view the deep learning is like knocking over benchmark like nobody's business um there's actually a lot more than fits on one slide I think if I put all the ones I'm aware of it'd be about three slides like this what's left to be done right so and I know that some of you are you know here because uh you want to learn how to apply these things and I know some of you are here because you might be even interested in doing research yourselves and writing research papers yourselves in deep learning and future learning so I want to share of


you I'll do this later to talk more about the state later as well I'll share of you what I think of as a as a good way as one as one of many promising directions in which to you know take research for for deep learning I think that's a scaling up so and happy how do we build effective deep learning algorithms right how do you get these animals work well well in fact how do you build effective machine learning algorithms you know so let's not back in history right about 20 years ago oh there were these debates about you know


where these different supervised learning algorithms so no feature learning yourself or learning pursue these supervised learning algorithms and they used to be all these debates about you know is this your album better is my album better so um michelle banko and eric bro did one of the studies that most influenced my thinking where they took maybe four of the state of the art learning algorithms of the day I guess back in 2001 azn's were not yet popular so didn't actually study SVM's but they took a natural


language processing house on which they had a effectively unlimited source of label data and they trained for learning algorithms and plotted on the x-axis is a training set size parts on the y-axis is the performance is the accuracy all the algorithms do about the same is the amount of data you have and even a quote superior algorithm often lose to a quote inferior algorithm if only you can give the inferior algorithm or data to train on yeah so I think this results like these that has led to this Maxim in machine learning


that you know says that often is not who is the best album that witnesses who has the most data and then I definitely see this over and over and slip the value of you if you look at think about the most commercially successful websites you know the ones making large amounts of money in that you use per every day many of those algorithms are incredibly simple is like logistic regression but the secret is that those albums will fit far more data than anyone else has so how about so this is supervised learning um how about unsupervised learning so


Adam coats as you who who helped prepare this handout a few years ago actually and a half ago did this interesting study right where he took all of the unsupervised feature learning algorithms of the day that you know that there's guys like us I guess debate as my averin better as your oven better weather and he took a bunch of these algorithms and ran all of them and they read the model size so for unsupervised feature learning all of us have a have a large amount of data right if you know you're doing if you're learning from unlabeled


images from natural images um you have an infinite amount of data right new and so the parameters the very is not the amount of data is the size of the models of how many features do you learn those coefficients a 1 just in the or the example we had 64 coefficients a 1 through a 64 for sparse coding but let's set that bigger let's learn a thousand features instead or 10,000 whatever let's learn much larger number features and what Adam found was that you know the album does matter maybe it matters more than know maybe matters more than


we respond supervised learning because these albums are less patrol but they can be sort of silver result where the bigger the model the better it does and in fact one interesting historical aside so see part n we actually went back and historically creates though you know I mean we like to publish paper and saying my oberyn's better than yours yours valine whatever right went back and traced all the sequence of papers with you know person a published as a result on C far person B published paper saying oh I did better than person C published


an LP say even better and then the one says oh I do even better a new idea invert we traced though a couple sequences of look of that of supposedly benchmarks of advances in benchmark that was supposedly do 200 miles better than yours as I do better and we believe that a lot of those results of the supposed progress was actually because the models got bigger right it's not that my album is actually better it just I don't know most law had more time to work and so I trained mine better and so I'm going to write a paper saying


my albums better in the stuff I've done the one most reliable way to get better results has been to train a bigger model if I change the algorithm sometimes it makes it better sometimes not but in fact I you know look at the literature I feel like a lot of work by a lot of different research groups has been in some ways on on trying to get these models to just train bigger right so in this world of a supervised learning where all of us have an infinite amount of data you know I feel like we're not limited by what data we


have were much more limited by our ability to process the infinite amount of data that all of us have alright so you know many attempts to come more efficient algorithms parallelization I say John's done very cool work on the FPGA and a second plantation oh I think I brought I'm going to take credit for bringing GPUs to the deep learning world and and so on as well work like this um and in fact looking at this chart my personal interpretation which others will disagree with is that those results were achieved to a very large part


because of scalability issues right but this is my personal interpretation which others may disagree with could you through the questions 7 & 8 on the handout so question 7 whether you have which which ones did you check off 2 & 3 cool I'll take your word for it and for question 8 oh and I just actually a question I checked off everything except DNA computing them it off my answer anyway yeah I thought of floating quantum in there too but I think someone actually is working on quantum computing yes yeah all right cool so um let's see you know


what there's something else I could talk about I think I'll do that towards the end um so you know just to wrap up this piece I think talk about the high level vision of less learning rather than manually designing our features but again kind of for me you know this isn't just about machine learning anymore this is I feel like um can we really learn something about a I especially perceptual ai ai ai and human intelligence is very broad I think you know we're we're starting to get a handle maybe the perceptual part of AI


which is maybe like you know 40 to 60 percent of many animal brains right so this big part of the brain and so what I'd like to do is as I state you know thank you for your attention and for your patience or need to do these things I hope that was somewhat fun what I like to do is let's break and later on in the in the next couple sessions where you know die slightly deeper into technical details go for the basics talk about neural networks and builders the algorithms also to point out that you know after you later today or


have you ever thought about joining a cago competition but don't really know where to start well you're in the right place I'm title data scientist to Rachel's hatman and today I'm gonna show you how to enter a competition alright to start off with we're gonna pick a competition and I already know I want to enter the housing prices advanced regression techniques so before I join I'm probably gonna want to read through the rules but I actually read through them earlier and you can do that on your own so I've accepted the rules for this

How to Enter a Kaggle Competition (using Kernels) | Kaggle


competition and I've joined the competition let's find a kernel to work off of so let's see kernels that people have recently run so looking through here oh look here's an example submission rachel is that you yeah that's me I did I made this earlier so I'm going to click on this example submission here and this is gonna walk me through all of the steps I need in order to enter a cankle competition so rather than just reading through it here I'm actually gonna copy and edit this kernel so I have my own copy that I can


work in it's gonna take just a second to spit up when you enter a competition there's a couple of basic steps you need to do first you need to train your model I'm doing all of this in kygo kernels which means that our code is going to be run on Kaggle rather than on my local computer so this competition this kernel already has a lot of the information we need in it so we are reading in some helpful libraries this is using R you can also write a kernel in Python if you prefer loading in my data and I'm setting a scene for reproducibility from


here I want to actually train my model so it looks like this person it's me has already started training a very simple model we're doing a test train split for cross-validation so I can check that my models not overfitting and then there's a little bit of pre-processing we're taking the training data we're removing the column that we are trying to predict to sale price and then he are we are converting strings to factors and then we are doing some label encoding so this is just taking categorical factors like I don't know


zoning and turning that in to a number so that we can create a matrix and from there we can use our matrix as input to an XG boost model so let's train our model really quick and if we look at the mean average error which is what this competition is going to be graded on I can see that it looks like it's actually still going down as we continue to do lots of boosting rounds so this is an X G boost model and the number of rounds you train it for will affect how much complexity in the data you can model and let's make sure that we're not


overfitting by running the same model on some of our held-out data so doing some cross validation and it looks like the test error is actually smaller than the Train error so what I'm gonna do is just increase the number of training rounds and see if that's still the case all right yeah and it looks like it is so I think just by increasing the number of training rounds I should get a better performance than this kernel originally had so once we have trained our model we can actually make our predictions so our


predictions here are going to be based on this test CSV which if we look at our data set we can see that the Train CSV has 81 columns and the test CSV has 80 columns that's because the test data set has all of the features that you would need but it does not have the feature that you're trying to predict so I don't know what the answers are over here that's what I'm trying to guess I will take these this test data that doesn't have the target column I'm gonna do the exact same pre-processing as I did before and then I'm gonna make some


predictions using the model that I've trained up here to several more rounds than original submission and then I am going to make predictions using this test data matrix and I'm going to save it as a CSV so what I'm saying here is I want a data frame with two columns one column is called ID and this has the ID information from the test data and one column is called sale price and this has my predictions from the submission prediction and I know that those are the columns that I need because if I look at the sample submission here I have the


columns ID and sale price and if I don't have the right column names here when I go to submit my file I will get an error all right once I have made my predictions I'm going to write those predictions to a CSV file so I'm saving a file out that has all of my submission information in it from here I need to commit my notebook so committing runs all of your code top to bottom and it creates a stable version that you can refer back to later so if I'm making multiple submissions I can go back and I can look at the


committed version I used for that specific submission and it looks like it's complete so let's open the version and here if I scroll down you can see at the bottom we have output files so I have one output file submission CSV that has the predictions that I saved out in my kernel and I'm going to submit this file to the competition ah and it's submitted and the submission is complete and it looks like my score is 0.16 and if I jump to my position on the leaderboard I might have to wait a second for that to be that's to be done but you can see


that I have my score I've made my submission and from here all I need to do is keep trying new things and improving my model and then making new submissions as I think the things that I did will improve my overall model so that's it that's all you need to do to enter a kegel competition obviously I wrote some of the code off-screen we're all so of course welcome to use the kernels that people have made public for a various for specific competition and also check out the discussion and see what people are talking about about that


hi there check out these clusters of images right here and just have a look at how all of them are pretty much showing the same object so here's balloons here's birds here's sharks or other fish these are from images from the image net data set and you can see that these clusters are pretty much the object classes themselves so there's all the frogs right here here all the all the people that have caught fish so this the astonishing thing about this is that these clusters have been obtained without any labels of the image net

Learning To Classify Images Without Labels (Paper Explained)


dataset of course the data set has labels but this method doesn't use the labels it learns to classify images without labels so today we're looking at this paper learning to classify images without labels by water from Guns Becca Simon Van Daan hender stung stamatis Georg Ulis mark pro Simmons and Luke fungal and on a high level overview they have a three-step procedure basically first they they use self supervised learning in order to get good representations second they do a clustering so they do a sort of k


nearest neighbor clustering but they do clustering on top of those things but they're doing in a kind of special way and then third they do a refinement through self labeling so if you know what all of these are you basically understand the paper already but there's a bit of tricky steps in there and it's pretty cool that at the end it works out like you just saw so before we dive in as always if you're here and not subscribed then please do and if you liked the video share it out and leave a comment if you feel like commenting cool


so as we already stated the problem they ask is it possible to automatically classify images without the use of ground truth annotations or even when the classes themselves are not known a priori now you might seem like you might think that this is outrageous how can you class high when you don't even know what the classes are and so on so the way you have to imagine it going forward and they're sort of they don't explicitly explain it but it's it's sort of assumed that if you have a data set dataset ba-da-da-da-da and you learn to


classify it what basically that means is you cluster it right you put some of the data points in in the same clusters okay and then of course the data set I'm gonna draw the same data set right here the same data set would have an actual classification thing so this would be class zero this here maybe class one and this year might be class 2 now you can't possibly know how the classes are like called or something which one is the first which one is a second so at test time basically if you have a method like


this that doesn't use labels what you're going to do is you're basically going to find you're going to be as generous as possible and in the assignment of these and say I'll look if I assign this here to cluster zero and this here to cluster 2 and this year to cluster 1 and I just you know carry over the labels what would my accuracy be under that labeling so you've asked generous as possible with the assignments of the labels so that's how it's going to work right that's a way you have to keep in mind we're basically developing an algorithm


that gives us this kind of clustering of the data and then if that clustering partitions the data in the same way as the actual labeling would the actual labeling with the test labels then we think it's a good algorithm okay so they claim they have a okay in this paper we deviate from recent works and advocate a two-step approach and it's actually a three step approach but we're feature learning and clustering are decoupled okay why why is that so they argue what you could do what people have done is and I'm going


to well this is just a wall of text so what you could do is you could just basically cluster the data like who says you can't use clustering algorithms but then the question is what what do you cluster them by like you need a distance so if I have points in 2d it sort of makes sense to use the Euclidean distance here but if I have images of cats and dogs and whatnot then the Euclidean distance between the pixels is really not a good a good thing but also so you might think we could actually we could you use a deep neural network and


then basically send the image that's the image right here send the image through the deep neural network and then either take this last state right here so it goes through and through and through and we could get take either of the hidden states or we could just take you know the last state that is the sort of hidden representation right here and do the clustering with that but then of course the question is what do you what which neural network do you take how do you train that neural network and there have been a few approaches such as a


deep cluster which try to formulate basically an objective for that neural network where you first you send all the images through right you send a bunch of images through to get you in embedding space you get you points and then in embedding space you think well the features that are in the embedding space they are somehow latent and they you know if basically the entire thing is if this neural network was used to classify images you would have a classification head on top and a classification head this is like a five class classification


that is nothing else than a linear classifier boundary that you put on top of this hidden representation so if you were to use this neural network for classification it must be possible to draw a linear boundary between the classes and therefore the either things like the inner product distance or the Euclidean distance must make sense in that space if they don't make sense in the picture space but they must make sense in the hidden representation space because what you're going to do with them is exactly linear classification


the last classification head of a neural network is just a linear classifier so the assumption is that and the conclusion is well in this space you should be able to cluster by Euclidian distance so what deep cluster does alternate like is first get the representations you start off with a random neural network then cluster these representations then basically label self label the images in a way now Oh way oversimplifying that technique right here but you have these alternative steps of clustering and then kind of


finding better representation and then clustering these representations and what it basically says is that the CNN itself is such a is like a prior because it's the translation invariant works very good for very well for natural images the CNN itself will lead to good representations if we do it in this way and there's some good results there but this paper argues that if you do that then the the algorithm tends to focus a lot on very low-level features so if the pixel on the bottom right here is blue right then you can and the neural


network by chance puts two of those images where the blue pixel on the bottom right it puts them close together then in the next step it will because they're close together will cluster them together and then it will basically feed back the new representation should put the two in the same class right it will feed back that it should focus even more on that blue pixel so it's very very dependent on initializations and it can jump super easily onto these low-level of features that have nothing to do with what the high level task


you're ultimately trying to solve which is to classify these images later so what this paper does is it says we can eliminate this we can eliminate this the fact that these methods will predict will produce neural networks that focus on low-level features and how do we do that we do that by representation learning so representation you're learning you might know this as self supervised learning and this is the task they solve in the first step of their objective so let's go through this this right here is an


image now the T is a transformation of that image and in self supervised learning there are several methods that you can transform an image so for example you can random crop an image you can just cut out like a piece right here and scale that up to be as large as the original image or you can use for example data augmentation which means you take the image and you basically so if there is I don't know the cat right here you kind of convolve it with some things it's there's like a very squiggly cat okay I'm terrible is you can you can


rotate it for example so it's like this okay so these are all these are all sets including the crop sets of this transformation T so you transform it in some way and you want after you've transformed it you send your original image that it should be red you send your original image and the transformed image through a neural network each one by themselves okay and then after then this you say the hidden representation here should be close to each other Oh this is this is basically the self supervised training task it's it's been


shown to work very very well as a pre-training method for classification neural networks you you have an image and it's augmented version and you minimize the inner product or the Euclidean distance between the two versions in the hidden space and the rationale is exactly the same the rationale is that this hidden space of course should be linearly classifiable and so the distance between those should be close and the rationale between having these tasks is that well if I flip the image right if I flip the image


to the right it cannot focus on the pixel on the bottom right anymore because that's not going to be the pixel on the bottom right here and I'm not always going to flip it into the same direction and sometimes I'm gonna crop it so it also can't focus on the pics on the bottom right because in the crop that pixel is like out here it's not even in the crop so basically what you're looking to do with these self supervised methods is you are looking to destroy this low level of information that's that's all you're looking to


build a pipeline of a neural network here that destroys deliberately low level information and you do that by coming up with tasks like this self supervision tasks that this that deliberately exclude this information from being used I think that's what's going on generally in the self supervised learning thing okay so this here as you can see is the neural network that you train you send both images the original and the Augmented version through the same neural network and then you minimize some distance which is usually like the inner product


or the Euclidean distance in this embedding space okay and what you train you can see right here you train the parameters of this neural network so the transformations are fixed or sampled and the distance is fixed you train the neural networks such that your embeddings minimize this task now this is nothing new this has been this has been used for a couple of years now to get better representation self supervised learning the thing but they basically say we can use this as an initialization step for this clustering procedure because if we


don't do that we focus on these low-level features okay and notice you don't need any labels for this procedure that's why it's called self supervise okay so the second second part is the clustering now they cluster but they don't just cluster these representations that would be that doesn't perform very well in their in their experiments what they instead do is they minimize this entire objective right here and we'll go through it step by step so they train a new neural network okay this thing right here this is a new neural network so


first you have you already have the neural network which was called what was it even called the one that gives you the embedding with the theta okay it's called five theta it's the same architecture and I think they initialize one with the other so in step 1 you get Phi Theta 5 theta goes if from from X gives you a representation of X ok let's call it hidden X so that's via self supervised learning but in step 2 you train an entirely new new neural network this phi ada here and you initialize it with this one but now you


train it to do the following again you want to minimize sorry you want to maximize the inner product right here see that's the inner product you want to maximize the inner product between two things now that's the same thing as before we want to minimize the distance between two things and the dot product distance in that case you maximize the dot product between two things and the two things are two images that go through the same neural network as before right this and this now what's different here is that here


input and one image of the data set that's the same as before okay so we input one image but here before in the self supervised learning we input an Augmented version of that and now we input something else we input this K right here now what's K what K comes from this neighbor set of X okay this is the set of neighbors of X and these neighbors are determined with respect to this neural network right here okay so what you do after step one is you take your neural network with the good embeddings and here is your data set X


your data set X that's really another your data set X is this list basically of all the images in your data set and what we're going to do is you're going to take all of them using that neural network that you just trained and embed them into a latent space right here okay this is the latent space where you have done this self supervised training and now for each image right here so if this is X eye you're going to find its K nearest neighbors and they use I think they use five as a benchmark so you're going to find its nearest neighbors it's


five nearest neighbors and you do this for each image so this image has these five nearest neighbors and so on so in step two what you're trying to do is you're going to try to pull together each image and its nearest neighbors in that in this this not in this space directly but you determine which ones are the nearest neighbor from this neural net where can you keep it constant that's how you determine what the nearest neighbors are in the first task and that is your NX set for X I and in the second step you're trying to make


the representations of any image and its nearest neighbors closer to each other okay so with with this thing right here you maximize the inner product between X in after this neural network and a nearest neighbor of X that was was a nearest neighbor after the first task now the way they cluster here is not just again by putting it into an embedding space like we saw before but this thing right here this neural network as you can see here is is a C dimensional vector in 0 1 now C is the number of classes that you can either


know that so you don't know which class is which you don't have labels but you could know how many classes there are or you could just guess how many classes there are and as long as you as you over guess you can still like build super clusters later so this they simply say it's in 0 1 but they also say it performs a soft assignment so we're also going to assume that this is normalized so for each for each data point X here you're going to you're going to have an image you're going to put it through this new neural network ok this new


neural network new and it's going to tell you it's going to give you basically a histogram let's say class 1 2 or 3 we guess there are 3 class and it's going to give you an assignment of the 3 and you also take a nearest neighbor here is your dataset you also take a nearest neighbor of that so you so you look for this set n of X and you take a nearest neighbor maybe that's that's a maybe that's a dog I can't I really can't draw a dog yeah that's the best I can do I'm sorry and you also put that through the same network and you're


saying since they were nearest neighbor and task 1 they must share some sort of interesting high level features because that's what the first task was for therefore I want to make them closer together in in the in the light of these of this neural network right here so this is also going to give you an assignment like maybe like this okay and now you object you you train this network right here to basically match these two distributions okay so this is this is now a classifier into c classes but we guess c and we don't have labels


we simply our label is going to be my neighbors from the first task must have the same labels that's our label now they say they also have this term right here which is the entropy over assignments okay as you can see so they minimize the following they minimize this quantity which has a negative in front of it so that means they maximize this log inner product and they also maximize the entropy because sorry so they minimize this thing and but the entropy is a negative quantity right so they maximize the interview


because here is a plus and now they minimize the entropy let's see what they say by minimizing the following objective now entropy is the sum of the negative sum of P log P and this if this is P yes this is the probability that an image is are going to be assigned to cluster C over the entire dataset so they're going to mmm yes so it's negative this quantity negative minus P log P and this is the entropy so they're going to minimize the entropy let's see what they say we include an entropy term the second term in equation two which


spreads the predictions uniformly across clusters C ok so what we want is a uniform assignment over class which means we should maximize the entropy oh yes okay they minimize this thing and this here is the negative entropy right okay so they want basically what they want of over the whole dataset that not all of the images are going to be in the same cluster well this is cluster one and then this is cluster two and then this is cluster three so that term counteracts that basically the more evenly spread the


entire dataset distribution is the the the higher the entropy the lower the negative entropy and that's the goal right here I'm sorry this this was I was confused by the too many negative signs and then you minimize the entire thing all right now they say they say different thing right here they say here this bracket denotes the dot product operator as we saw it's the dot product between these two distributions right here the first term in equation two imposes this neural network to make consistent predictions for a sample X I


and its neighboring samples the neighbors of X I and here is an interesting thing note that the dot product will be maximal when the predictions are one hot that means confident and assigned to the same cluster consistent so they basically say the objective encourages confidence because it encourages predictions to be one hot and it can Courage's consistency because it you know that because the distributions need to be the same they should be in the same cluster right now I agree with the consistency like if you make the inner


product high then of the up to date of two of these histograms of course they'll look the same right because these are ultimately vectors these are three-dimensional vectors let's call them two-dimensional vectors right so here is class one here is class two if you you know make the inner product small or high they will agree on their predictions but I I disagree that this encourages anything to be one hot like in and if you have two vectors that are both zero one times zero one the inner product is going to be one and if you


have two assignments that are 0.5 and 0.5 then it is also going to result in an in an inner product of is it 0.5 right is also going to to be no so what's the inner product here the inner product is 0.5 times 0.5 plus 0.5 times 0.5 which is 0.5 am i dumb an embarrassingly long time later oh it's because the l1 norm okay okay we got it we got it I am I am okay I am too dumb yes of course I was thinking of these vectors being normalized in L 2 space where their inner products would always be 1 but of course if you have


assignments between classes and it's a probability distribution a histogram then all of the possible assignments lie on this on this thing right here now the inner product with yourself of course is the length of the vector and the length of a vector that points to one class or the other class is longer than a vector that points in between so okay I see that's where they get this that's where they get this must be one hot from so okay I'll give that to them it is actually encouraging one hot predictions as long as these things are normalized


in l1 space which they probably are because they're histograms right yes that was that was dumbness of me I was trying to make a counter example I'm like wait a minute this counter example is a counter example to my counter example okay so yeah that's that's that so as you can see they are of course correct here and they now make the first experiments so they say basically after the first step of the self supervised training they can already retrieve sort of nearest neighbors and the nearest neighbors the nearest neighbors of these images right


here are the ones that you see on the right and after the self supervised one these nearest neighbors are already pretty good at sharing the high level features actually crazy-crazy good right this flute here is in different sizes as you can see the fishes aren't aren't all exactly the same the birds so you can see it really focuses on sort of higher level of features but I guess it's really dependent on this higher level tasks and they what they also investigate this quantitatively but I just want to focus on how how good is


this after only the self supervised thing and now they do this clustering and they can already sort they could already evaluate it right here because now they have a clustering right after this step they've basically pulled together the neighbors and they have this neural network that is not assigning classes so they could already evaluate this and they are going to do that but that's not good enough yet then they do a third step which is fine tuning through self labeling now self labeling is pretty much exactly what


it's what it says it's you label your own data with your own classifier now that might be a bit outrageous and you basically saying wait a minute if I label my own data and learn a classifier on these labels isn't isn't it just going to come out the same and the answer is no right if you have a dataset because your classifier doesn't give you just first of all if your classifier is something like this right just happens to be and you label and you learn a new classifier it is going to be more like this right because it sort of maximizes a lot


of classifiers maximize these distances between the classes so even if it's like that and then the second step they do is they say okay there are some points where we are actually more confident about such as this one we're more confident about that one also this one and then this one here is pretty close like we're not super neither this one but we're very confident about these two so we're only going to use the ones where we are in fact confident about to learn to learn the new classifier or basically we you can also weigh them and


so on but they go by confidence right here as you can see in this final algorithm so this is the entire algorithm and I got kicked away at our algorithm there we go all right so semantic clustering by adopting nearest neighbors they're scan algorithm so in the first step you do this pretext task this is the self supervision the representation learning okay for your entire data set no sorry this is this is this your optimized optimizer neural network with task T that's just self supervised representation learning okay then the


second thing we're going to determine the nearest neighbor set for each X now they also in that's that they also augment the data they do a heavy data augmentation and so on also in this in the third step in the self labeling they do date augmentation there's a lot of tricks in here but ultimately the base algorithm goes like this so you find your neighboring sets for each X and then what you do while you're clustering loss decreases you update this clustering neural network by with this loss that we saw so this is the loss


where you make the nearest neighbors closer to each other while still keeping the entropy high okay and then in the last after you've done this you go through when you say while the length of Y increases what's why why is all the data points that are above a certain threshold now you going to filter the data set that is above a certain threshold and that's your data set Y and you Terrain this same neural network you basically fine-tune it with the cross entropy loss on your own labels so now you only have labels Y


okay so it's not it's not labels you have the cross entropy loss between the assignments of this and the assignments of your data set okay so you basically do the same task but you filter by confidence and they use a threshold I think of 0.7 or something like this now let's go into the experiments the experiments or look as follows so they do some ablations to find out where in their methods kind of the the gains come from and we'll just quickly go through them if they just do these self supervision at the beginning and then


just do k-means clustering on top of that that will give them on C for ten a thirty five point nine percent accuracy so not very good so the clustering you can't just cluster on top of these representations and then be done if they do what they say so this is sample and batch entropy loss this basically means you do not care about the nearest neighbors you do this entire thing but you only make an image close to the prediction close to itself and it's augmentations so you don't use any nearest neighbor information also


doesn't work like I wouldn't pay too much attention that the numbers are ten twenty or thirty it just it like doesn't work now if you use the scan loss you all of a sudden you get into a regime where there is actual signal so this is um this is now significantly above the this is significantly above random guessing and if you use strong data augmentation as I said is a lot of this is has these tricks in it of what kind of data augmentation you do and so on so never forget that that these papers besides their idea they put in all the tricks


they can so you get 10% more and then if you do this self labeling step you get another 10% more and this is fairly respectable like 83.5 without ever seeing labels it's fairly good but of course there are only ten classes right here so keep that in mind but they will do it on image net later and they investigate what kind of self supervision tasks at the beginning are important and they investigate things like rot net feature decoupling and noise contrastive estimation which noise contrastive estimation is the best a


noise contrastive estimation I think is just where you as we said you input an image and then it's kind of noisy versions with augmented in various ways and then you classify them together and this has been like this these methods have been very successful in the last few years yeah so this they they have various investigations into their algorithm I want to point out this here this is the accuracy vs. confidence after the complete clustering step so this is now the third step is self labeling and you can see right here as


these confidence of the network goes up the actual accuracy goes up as well so that means the network after the classroom is really more confident about the points that it can classify more accurately there's like a correlation between the network is confident and the actual label of the point which is remarkable because it has never seen the label but also see how sort of the range here is soup is quite small so with the standard augmentation that goes like from here to here so where you set that threshold is


fairly important that might be quite brittle here because you need to set the threshold right such that some points are below it and some are above it and you you don't want to pull in points where where you're not because if you pull in points from here you're only you only have the correct label for 75% or something like them of them and that means if you now self label and learn on them you're going to learn the wrong signal so this this step seems fairly brittle honestly but I don't know of course they go on and investigate


various things such as how many clusters do you need or how many nearest neighbors sorry do you need this number K here and you can see that if you have zero neighbors then you're doing a lot worse than if you have let's say five nearest neighbors so the jump here as you can see is fairly high in all the data sets but after that it sort of doesn't really matter much so it seems like five nearest neighbors should be enough for most things and here they just show that when they remove the false positives that their algorithm


actually converges to the correct clustering the correct accuracy which is not surprising like if you remove the wrong samples that are wrong then the rest of the samples are going to be right I think that's just showing that it doesn't go into some kind of crazy downward spiral loop or something like this but still it's just kind of funny okay so they do you investigate how much they improve and they improve by quite a lot above the kind of previous methods so they have a lot of previous methods but a manner


this includes things like k-means and so on ganz deep cluster that we spoke about and this method it already gets as you can see fairly close to good accuracy so you have like eighty eight point six percent accuracy and that's you know fairly remarkable on C 410 without seeing the labels but we'll go on and now they go into image net now image net of course has way more classes it has a thousand classes compared to see four tenths ten classes so if you if you think you know clustering ten classes might and they're fairly apart from each


other might work with various techniques image net a thousand classes that's way more difficult that I do subsample this to 50 100 and 200 classes and they get okay accuracy as you can see they they get 81 percent in for 50 classes where a supervised baseline would get 86 percent into 200 classes they get 69 percent where a supervised baseline would get 76 percent so it's fairly it's it's there and and that's quite remarkable for this low number of classes and they figure out that if they look for these samples


that are kind of in the most of the middle of their cluster they get these prototypes right here you can see all of these images I don't if you know imagine that some of the images we really only have the part of the object and so on so here with the prototypical things you really get center clear shot of the object with clearly visible features and so on so this sort of re sort of repeats the fact that this clustering really does go on what sort of semantic information of course the labels here are you know


from the from the test label set the network can't figure that out and and then they go for a thousand classes and in a thousand classes it doesn't really work because there might be just too many confusions right here but they do have this confusion matrix of their of their method and it shows that the confusion matrix is pretty much a long like block diagonal along these super clusters right here so you can you can see the dogs the network confuses the dogs fairly often and then insects with each other but not really across here


which is still quite remarkable but I mean that's you get the same thing for a lot of these methods so I don't know I don't know how much different this would be in other methods but certainly it's interesting to look at now they go into one last thing and that is what if we don't know how many clusters there are right if we don't know anything so say so far we have assumed to to have knowledge about the number of ground truth classes the model predictions were valid losing the Hungarian matching algorithm we already saw this in the DET


är by facebook if you remember however what happens if the number of clusters does not match the number of ground truth classes anymore so they now say Table three reports the results when we overestimate the number of ground truth classes by a factor of two okay so now they build just twenty classes for C for ten instead of ten classes and we'll going to look at table three real quick where's Table three this is Table three okay so when they over cluster you get the thing here on the bottom and you can see there is a drop in accuracy right


here now what I don't actually they don't actually say how they do the over cluster matching so if you imagine if I now have I don't know six clusters but I any need to assign them to three clusters you know here do I still use this most optimistic thing so they are still used I think they still use this most optimistic matching right where you assign everything to its best fitted cluster right you compute all the permutations and then you give it the best benefit of the doubt now if you imagine the situation where I over


cluster to the point that I have each image in its own cluster and I run this algorithm to evaluate my clustering I give it basically the most beneficial view then I would get a hundred percent accuracy okay so like in in in one of the in this over cluster approach I would sort of expect that you actually get a better score because you can like there is more generosity of the matching algorithm involved now that's counteracted by the fact that you can't group together things that obviously have similar features because they are


in the same class so there's kind of two forces pulling here but I was kind of astounded that it's going down and the evaluation method of this matching algorithm it sort of breaks down when you have more classes at least in my opinion yeah but but it's interesting to see that you can just overshoot and but then you need some sort of heuristic to reconcile that in any case I think this paper is pretty cool it brings together a lot of things that we're already present and introduces this kind of this step approach but what you have to keep


in mind and by the way there's lots of samples down here what you have to keep in mind is there are a lot of hyper parameters in here there are like this threshold and you know the first of all yeah the number of classes the thresholds the architectures and so on and and all of this has been tuned to get these numbers really high right all of these steps all of the augmentations and so on the chosen data argumentations it has been chosen to get this number as high as possible so you know to interpret this as oh look we can close classify without


knowing the labels is you know if yes in this case but the hyper parameter choices of the algorithm are all informed by the labels so it is still very very unclear of how this method will actually work when you really don't have the labels when you actually have to choose the hyper parameters in absence of anything and yeah I think the future might tell if they continue to work on this alright thanks for listening looking watching and bearing with me through my wrestling with with various math basic math in this video I


I'm excited to be here because I looked at the class syllabus and it looks really good so it's super relevant so my name is hike I work at Sky do I leave the autonomy team there Skye do is a start-up in Redwood City that makes vision based fully autonomous drones this is the drone we just launched the sky do - and it's it's it does a lot of stuff that you've been learning in this class and so I mostly just want to show a bunch of visualizations talk about the way we think about things and please ask lots of questions because that's that's

Lecture 23 Guest Lecture: Drones -- CS287-FA19 Advanced Robotics at UC Berkeley


the whole point so I'll just pass this around if you guys want to take a look oh we go I don't think I'm allowed to fly in here but I would and you'll have to excuse my moustache but it is November so okay so just to get you guys interested I'm gonna show the first like I don't know 40 seconds of our of our launch video for this guy which we launched just boy a month ago and we're shipping out right about now so yeah all right so you can watch the rest it's pretty good but I'm just going to talk very quickly about the point of our


company and then just jump into technical stuff so basically drones are really useful because you can have an eye in the air moving around in 3d and that's useful for just a ton of different industries and this is nothing new the problem is that I think the technology for them to be safe and trustworthy and intelligent in the wild in enough places hasn't been there and so that's the point of sky do from day one it's like let's make the technology for this type of robot work such that it can go out and do things in the world so


basically we've we've been invested in this like building the whole stack and the reason we build the hardware is because like the whole thing just matters like you care a ton about weight and size and power and cost but also just about the the quality of the cameras the exact like vibration characteristics of the thing the electronics in play the power system the embedded system and the software on top that runs like controlling the whole thing makes it possible to make to like push the limits of robots and and that's


why we do it and so so not just you know this is the consumer launch that that videos regarding you know like this awesome video capture for anyone without needing to be an expert pilot but we're but it also makes a ton of sense for any task where you're inspecting things like bridges and cell towers and pipelines and rooftops anywhere that's dangerous anywhere that's costly and you need to build a map of things so for construction for mining it it it's it's a long list so as long as you have something that it's trustworthy intelligent can


automate tasks and get get even just video data it's incredibly useful so that's that's our goal this is another slide that says the same thing okay and then here's just a view of kind of the more commercial side of this so the idea of being able to get up inside kind of these high voltage trusses and power lines and take close inspection photos this is something that's very hard to do manually and it's and it's dangerous and these things need to get inspected all the time it's incredibly expensive and then on the right is us


being inside kind of just this narrow bridge truss with metal on all three sides so something you really can't do with GPS it it's just it's not possible cool and then finally I'll show that this product that we just kind of announced that we're super super excited about which is all the dock also commonly referred to as the box which is essentially just a cozy home for the drone that is weatherproof and basically it pops out the drone flies and does a task comes back and lands and goes in and the things cloud connected and the


whole idea is like you have this robot out somewhere on your construction side along along a pipeline and the drones go out and do tasks on a regular schedule and they come back and the whole thing is hands-off and it's truly like just just an autonomous thing that's that's doing work and so it becomes more of a cloud platform where people can use that rather than you know somewhat someone is piloting this robot and in control of it and that's kind of that's how we see the the long long-term play of this going


cool so here's a video that just shows it's just a clip of a mountain biker that we think is a pretty good shot of just autonomous visual tracking following so the drone is doing all this computation of course onboard in real-time based on visual tracking and we're able to kind of get some nice cinematic video here with tree occlusions with fast motion and here we're getting a shot from the front which we call lead mode which is pretty tough because it means you have to do a lot better job anticipating someone's


emotions so I want to just talk about aspects of like how this works and just build it up and again please ask lots of questions okay so if if you look at the drone it's got so it's got the main 4k camera on a 3-axis gimbal and that's kind of what we were looking at here and then it has six navigation cameras so you can look at the geometry of the drone there's there's three on top and three on the bottom and they're actually these crazy super fisheye cameras so they see 200 degrees so they see like beyond a


hemisphere which means the three on the top can basically see the upper hemisphere in trinocular view and then the three on the bottom can see the bottom hemisphere in trinocular view and the goal is to get like 360 coverage so we can do 360 you know obstacle avoidance and awareness of the scene so this is what the this is what one of the cameras looks like on the bottom and one on the top the other four are pretty similar to these two so you can see the it's the view is pretty funky like this is this is one of the propellers that's


the battery on the top that you're seeing from the bottom right side where this is forward and then the this fin is actually if you get the drone you can see it's got this little by the camera it's got this fin which protects the camera lens if you were to like drop it down and it looks a lot bigger here than on the drone itself but that's because it's just really right next to the camera and then this is the other yeah I think the other arm is over there that you're seeing from one of the top cameras so obviously it's this kind of


tiny planet distorted view but this this is what we work with and the cameras are rolling shutter so doing computer vision on rolling shutter cameras is is a really hard problem because what basically what that means is every row is kind of taken at a different time which means if you're undergoing fast rotation especially but even fast motion you're the contents of the image are as if you know time is changing in them so to account for that correctly you have to you have to be very careful so here's here's what it kind of looks


like with the navigation cameras put together to create a 360 view so if you see the clip we were watching before is on the top left and then but that is of the full sort of sphere where you can see it's just this yellow outline is that does that make sense so having so the navigation cameras give a lot more context and that's you know that's that's what we use for all the algorithms so the core the core things we basically need to do to make this drone fly our one state estimation so we need to estimate our trajectory how how


we're moving in the world and the second thing is obstacle avoidance building a dense map of the environment in a way that we can use to to navigate the third thing is detection and tracking of objects so in this case the biker seeing them in the image estimating than 3d predicting their trajectory and the fourth thing is motion planning so with that sort of understanding of the scene want to figure out what is the best flight path that's sort of in this case provides a really cinematic video and is also not crashing it's also smooth and


respects the physical dynamics of of the vehicle like being able to push the limits of aggressiveness but with if you need it but without sort of commanding something that is not possible by the physical vehicle okay and then here is the same 360 view with with a depth estimate at the bottom so this is one way of visualizing the like full output of our of our depth estimation system where the close pixels are more yellow and so you can imagine how that is really useful for obstacle avoidance because you can directly kind of feed


into the the planning system as a 3d map I'm visualizing you here in this we call us an echo rectangular view because it's a nice way to show the 360 but this is not actually how we use the system online okay and then here's one more way of viewing like a single instance in time here it's a it's a point cloud view that is the kind of colored by the pixels themselves this guy appear that's the drone that's the trajectory the drone was flying and then here is the person's trajectory that makes sense and then we have you have a


scene around us and then it's going to switch to a depth view so instead of being colored by the pixels themselves they'll be colored by the range from the vehicle so the distance of each point from the vehicle similar to what we were looking at before but now in 3d okay cool so I'm going to talk a little bit about what we need to do in terms of state estimation and this was actually our first drone the r-1 it's a picture of but anything that is you know could potentially be variable on the drone we want to model an estimate so the the the


obvious one is the trajectory of the robot which be like a pose in space so a position velocity orientation angular velocity over time of the robot and we do this with our visual inertial odometry system I'll show one slide after this we also have several I am use and so accelerometer gyroscope which is super useful for propagating you know short term data it really only works to integrate the IMU for a matter of at least the accelerometer for maybe a second beyond that on a drone and with the sort of size and and cost


constraints it is it's too noisy but I'm use have biases and estimating them correctly is is key to being able to effectively use them and then we have the poses of the cameras on the vehicle so we you know obviously have CAD of the of the lenses and they get calibrated Factory time and we take that and we put that into our model of you know how we're gonna go from things in the image in 2d to 3d and the problem is those move around so every vehicle is a little bit different just because the physical characteristics so they get calibrated


in the factory by a robot that waves them around a bunch of tags and we know how to model that we have a model for the camera lenses and we calibrate that and also their poses the problem with that is that they also move around when as the vehicle undergoes temperature changes especially so it's a magnesium frame with plastic pieces and those things expand and contract and any small amount of rotation like even a small fraction of a degree significantly changes the actual results you get when something is 20 meters away so that the


problem is that the ability of estimating depth or really doing odometry from things that distance it depends on the inverse range of the the object you're looking at and so as you get that far out you're really looking for fractional differences between a fractional differences of a pixel between multiple images and that makes you that makes it so that you have to be very careful about your calibration the yes so so we we calibrate the poses of the cameras and particular it's mostly about the rotation like the actual


translation of the cameras moving it does not have a large effect on the results it's mostly it's mostly rotations but we calibrate those things online sore joint ly with our odometry system so there there are many variables in the optimization some of which are that the pose of the robot but also the poses of the cameras and the biases the IMU and additionally we calibrate the intrinsic parameters of our lenses which is which is kind of nuts thing to do on line but those also change I'll show a video of that here so this is showing a


temperature sweep of a fixed lens and you can actually see sort of these circles are fixed but you can see how much it moves is actually set pixels of motion there over the change and those are the like intrinsic lens parameters and the distortion model of the camera and so to really handle this we also estimate these online with everything else which is a cool and a hard problem and then of course we have propeller spinning so we've got to estimate the rates of those propellers and then we have the gimbal and we


estimate a combination of ways exactly where the user cameras pointed because while we have you know we have sensors on the gimbal that tell us where we think it is and we've commanded it to be somewhere it's also on a an isolation system and there are other changes there so we also additionally do visual estimation of between the user camera and the navigation cameras to to line those things up cool so here's just a quick visualization of the visual inertial Adamo tree system so on the bottom left are some features that are being tracked


in this case they're their sparse features just tracked over time in one camera and the right is the estimated trajectory of the robot so I I think you've guys have gone through kind of a traditional slam pipeline is that right in this class sort of maybe ok well anyway it's yes so so what you get is a bunch of 2d observations in your camera images by tracking in your images and you can track either between cameras at a single time instant or you can track over time and really you want to kind of do both and there's different


characteristics of what works well but then you take that and you are jointly optimizing for a 3d locations of those landmarks that agree with the observations you've made in the images while also solving for your own pose and to the the residual and this kind of objective will be the reprojection of that 3d point into your camera model for your pose and those are all variables in your optimization result in a pixel coordinate and then you would compare that pixel coordinate to the pixel coordinate at which you actually


observed that feature in that camera and with multiple observations of what you think is the same 3d point that starts to constrain things in this system okay still know okay so here is here's a little bit of what rolling shutter can look like in this case it's rolling shutter combined with vibration it slowed down a lot it's a little subtle to see here but you can see these like Wiggles we call it wibblewobble the yes so the the car you know you can you can see the windshield of the car sort of waving around and that if you're not


modeling that if you're not accounting for it that's just pure noise that's going to mess things up for you in a way that does not match geometric constraints the the the way you if this is happening to you you have two choices choice one is to like improve your hardware improve your system such that you don't get this issue like there might be a certain safe frequency of vibration that is related to the the time it takes to read out your camera and you get some periodic signal option two is to have a way of estimating


exactly how the camera moved during this time period and accounting for that which requires very careful work and we do both of these things but it's once once you're trying to fly like at high speeds in aggressive scenarios like we really really have to be pushing the edge of what what we can model there and what we can estimate okay cool this is one of my favorite slides because I think it talks about what is what is really hard about computer vision at least computer vision in a production robot so the so a here


is just thin thin branches so flying in a forest right now it's winter we're gonna have a bunch of customers they're gonna take their drones and I'm gonna try to follow themselves riding a bike in the forest and there's going to be all kinds of terrifying thin branches that you can barely see and we we work really really hard on being able to see that from the cameras I'll make the bold claim that I think we're almost definitely the best in the world that seeing really fine thin cameras from their branches from camera images and


when you're flying at you know you kind of thinking about like what's the resolution of my camera how fast am i flying what's my physical reaction time to either steer away or stop it's obviously not guaranteed you have a way to steer away so you could really think you'd be thinking about okay if I'm flying at 35 miles per hour and there is it like what how far away do I have to see an object to be able to stop if that's like if it suddenly pops in and and that will that kind of plays into how how faint does it like what is


the smallest perceptible signal you can have in your camera images before it like you have a chance of seeing that and I think we're we're pretty close to the limit there of like if if you can as a human like look at the camera image and and see something like we probably see it as well but it's still like we're we're up against that edge of just how much signals can we get out of that so be here is is a really fun scenario where there's a bright Sun so the the tough lighting conditions make everything harder right so in this case


there's a low Sun and we're in a forest and particularly are actually images from our one the first the first drone but it's it's very similar so the slide on the left the camera on the left is I think has a smudge on it like a fingerprint on the lens which basically makes just like light goes differently into the camera and and the left and right cameras look very different there what that means is that if you're looking to if you're looking between the images for a photometric signal that they're they're not agreeing


at all so typically what you you know might do for stereo vision algorithm is look at two cameras and look for you know look along and if you pull her line and look for something where the you know say a small patch of pixels looks the same in the left and right camera the problem is if you're up over here here you just got these crazy vertical sunrays that will match other sun rays in a totally inconsistent way but not your actual contents so it's a huge source of noise and especially when it's it's different between the two cameras


and then C is just motion blur so that drone can you know like that's kind of our decision unless it hits something and flips over but to allow very aggressive rotations up to several hundred degrees per second which induces motion blur in the cameras D is just reflections and mirrors and I really don't have a good solution for that like if you if you fly this drone in front of a panel of glass that's like three meters by three meters wide like it it will you know it will see what's inside and and think it can it can fly in the


the hope of doing this like semantically is it's it's tough it's really tough and then mirrors of course like in this case a reflective building and so we typically hope is that these vertical struts we're able you know those make geometric sense and would prevent us from flying into it but the part of the mirror that's just reflecting the building behind well that just looks like an opposite world that's geometrically consistent and so our depth map will reflect that it thinks that these are that these are basically poles and then


there's something behind there if you look closely you can kind of see there's some blinds in here so horizontally and so at some at some point of like if you can see what's in the window and you can see what's behind you have to kind of there's like two potential realities for what that could be and it's it's an interesting problem to think about when when that's which would happen and which taking a stronger signal okay so E is just big textualist surfaces so let's say the drones like right up against the ceiling is typically the the hardest of


cases and really only sees what he looks like so it's just white and that's also what the sky looks like in a lot of scenarios now sometimes the sky is bright blue but not not always often it's just basically saturated whiteness from the cameras and deciding which of those two is is the case is a really hard problem the best thing that can help you there is like the fact that we have full 360 context on the drone means that the rest of the scene can tell us oh is this more likely to be an indoor scene with a room or is this going to be


you know we're flying out in an outdoor scene and that the reason it matters so much is because well the sky is one of the best directions to fly in like it's it's it's usually very safe and so if we're if we're trying to get away from something and try to figure out how to move like going up is usually a very good decision but not if it's a ceiling then you're gonna have a bad time okay and then f is water so there's just water droplets on the lens I mentioned smudges from your fingerprint also dirt also dust also cracked lenses just all


manner of things that inhibit the sensor and this is something that we don't particularly control right the user is is responsible they get for cleaning the lenses but they take this drone somewhere we've never been and we asked them to wipe the thing and we and we detect these things online like that's the best thing we can do so actually as a drone is flying we look for potential issues with what the cameras see and and having dirty lenses and we have sort of we have a lot of these types of warnings but this is an


interesting one particular is like we have multiple levels of like oh it looks like your lenses are dirty like you should probably land and clean them and try again and you can dismiss that and most people just dismiss it and don't listen and then there's another level that's like okay you really have dirty cameras like we're not gonna let you say track anymore so the drone just kind of like goes into a hover it's like oh it's not safe but we can't just make the drone land or something because the drone could be you know a couple of


kilometres out over the ocean or something and just hovering there or landing is not a good acceptable solution and just coming back on its own is also dangerous so what we sort of chose to do there is for the drone to just hover and you can control it manually on your phone or with a controller but it won't like track you because then it's sort of more autonomous and you're oblivious to the fact that it's it's performance is degraded for sure so we we care about lots of those types of robustness issues because it's just like getting getting


to a demo that works of like okay you know we have we have this drone it films somebody it avoids obstacles like that is much much easier obviously than getting to the point where you can actually have something where someone paid for a product and they take it somewhere new and they like initially don't trust it but what we typically see is after like one or two flights of you know no like no problems it works and in the vast majority of you know scenes you're not gonna have any issue but you quickly just gain way too much


confidence in the robot just to work anywhere anytime and that's something that we kind of struggle with is how to like communicate limitations in an effective way because it does work so well in so many cases okay talk a little bit about mapping so and by talk I mean show this video so these are just you know vark like a voxel representation over the user camera video basically just to to look cool so in terms of dense mapping so that the goals from state estimation are different right because for estimating


how we're moving basically we have we have cameras in every direction and you don't need a lot of points like you need a little bit of information about other things are moving that are static and you track them well the which means if something is very difficult like crazy tangle of thin branches you can just ignore that part and look at the ground or look at the tops of the trees and that's enough to tell you how you're moving but for actually mapping things like we need to pay attention to every single direction and make a decision


about that saying oh this directions like too hard to look at like you can decide to never fly in that direction but that's that's very limiting so we published one paper here a couple of years ago with research in turn and Alex Kendall and the the paper was basically trying to combine a classical stereo matching pipeline like geometric reasoning about stereo vision with with deep learning to learn features and do the matching so at the time we we did really well on these the kiddie self-driving data set which we submit it


to and the basically the bottom one here is the ground truth of that data set which is a lidar scan and in the middle is a as a prediction of our of our network and so it it worked really well in in these scenes I'm not going to go into details here but essentially the the network was structured in a way to take advantage of sort of intuition about stereo matching and classical stare matching algorithms where we have we compute shared learned weights on each of the two images with two towers and then we have a cost volume approach


where kind of mix those features together at different possible disparities and then we regularize the cost volume with 3d convolutions and then we we pick a winner so this this worked really well but I'll kind of give a couple of caveats the first caveat is it ran at one frame per second on a Titan X GPU which meant like obviously we can't use it it's orders of magnitude off and the second caveat is that frankly the the kitty data said or really like my experience any self-driving data set that I've seen are are incredibly easy


compared to the types of scenes that drones tend to see in in the wild and the reason is because of regularity so most of the images just look very similar like you have a road scene you have cars there's always a ground plane there's things on the sides there are street signs that are vertical and there's just a ton of semantic priors you can do this is what monocular depth estimation works really well on driving scenes as well when things are you know when things are normal now that's not always the case but for a drone you can


be flying really like really in any nook or cranny of any building or while and and you really can't tell what's in a lot of the camera images and there's just no there's no good monocular prior a lot of the time because you could just be in inside some random geometric shape in a sculpture and art gallery or something and so it's it's a much harder problem for that and and specifically the the metrics on a dataset like kitty where you're typically measured on like how how accurate is like the average pixel error across the image right like


I can't emphasize enough just how useless that is for robotic navigation purposes like being being 1% closer like it's basically just dominated by you know how well do you estimate the ground plane and the exact pixel values of the depths and if you have something like a thin object that you miss or an entire vehicle but it's further away or something like it has very little to impact on the score and so focusing on metrics that really matter for autonomous navigation were things like 3d occupancy and just thinking about


only the parts of scene that matter and in the end like in the in the Berkley way that kind of one of the most powerful ways to do that is to think about the sort of action space like what are the parts of the scene that actually matter for making a decision rather than doing it allocating your error metric by pixel values which is totally simple to think about but but pretty meaningless okay so since then we've made it a lot faster and a lot better and it's also changed in many many ways but basically we are flying deep networks that produce


depths and build an obstacle map for us and it runs at approximately a thousand times faster than the previous sort of first thing that I mentioned and if she think we were the first ones to fly a robot based in a complex scene based on sort of a deep learned obstacle avoidance system about a year and a half ago this was our little ATV flight yeah yeah that's a good question basically we we do a bit of both like we aggregate the images but the the dominant part of the deep learning estimation right now is is between the sort of cam like the


camera is at an instant in time but we have definitely we definitely do things that are between like temporarily as well there are different challenges into doing that one of the challenges is moving objects so moving objects you know you you have to estimate that motion to geometric we get the correct thing right like if you're doing optical flow and then you assume things are static you'll get in correct depth in particular like if you're flying the same speed as somebody and following a biker say and they and and you do that


and you think they're static like it'll seem like they're far away because basically they don't move with respect to you and so the pixels you stay in the same pixel location which means which implies that they're they're basically at infinity so there's tricky things like that to take into account yeah I'm going to show some a couple of just clips of the sort of hard things we focus on so this is just a forest with a bunch of branches and and looking like really what you care about here is like did you see this object or not and


generally the like 3d extent of it but the exact the exact details of of the shape really don't matter very much although it does tend to be correlated with you know the quality of your predictions here's a here's a 3d view of that same thing where we have the camera image on the bottom left and then the depth on the bottom right okay power lines are real and something to be worried about so okay you can see it's a little hard to tell here oh these aren't matching are they okay well so you can kind of see if you trace this


out there there are a bunch of power lines in this image so these ones obviously are very clear you can see these this is a faint but you can see it over here it kind of fades into the background this is a power line that's quite faint to see and it disappears entirely behind the Sun here and then actually up here there's one that goes all the way across like that and it's it's very very hard to see so it's just giving some some intuition for that and there's a second one here and there's a third one here and so really like it


does depend a lot on the background and the lighting on everything just when it gets in your camera mijin and how much that works so again that's that's kind of the walls we tend to be up against trying to push the limits of where we can fly what we can see and how fast we can go especially and then about textualist surfaces so here's actually flying through our office and you can see on the left here example it's a scene with you know specularity is on the walls which will be lying to you photo metrically and just a lot of a lot


of white texture and then here is the same thing in 3d it's funny this blue glow on the bottom is actually because there's there are some different lights on the drone but the in this case this this LED was on which is very close to this camera and it basically just shines this blue halo under right there on the ground but we we don't keep this one on in flight normally it's just these guys which have been you know designed to not be seen by the cameras the whole kind of cemented the whole geometry of the drone


and the inversion and everything is kind of built around enabling as much visibility as possible of the world without obstructions yeah the frequency so I can't say exactly what frequency is part of the reason is because it's sort of internal stuff and part illusion is because it it can change based on some different things so we we are pretty clever about some of the things we do in terms of so you know what what resolution do you process stuff at how complicated of algorithms do you run or intensive and which like which


directions do you focus on so all those things in terms of like down sampling and foliation and choosing the the extent of the algorithm you run are pretty important to finding the the best balance and so for example if you're flying really fast at such that it's really dynamically infeasible for you to even move in these other directions for the purpose of navigation it doesn't really make sense to focus your efforts there you really want to kind of like pick the cone of just do tunnel vision there now the kind of in a different


case like for example let's say you're let's say you're barreling down this way at max speed but you're also filming something that's over there now you may want to focus on those two directions instead of just the one because you need to estimate the scene over there to make some decisions about your high-level objectives and then obstacle avoidance over here so things like that are our important games to play okay here's another fun one that's just artistic this is just an example of someone basically pushing back on the joystick


so we have we have a bunch of accessories but one is just it's a controller and you can fly and the idea is basically like you control a drone like normal you know kind of if you've ever flown these MOTU joysticks and the idea is that you don't have to be an extra pile like you can just you can just push it forward and won't crash right it'll go around things it will it will do its thing so it's it's a really fun way to fly because it means you can you can either just have fun or do your task film something but you're not


worried about physics or the drone or the crashing or things like that and that that's what this shot was basically just pushing pushing back on the sticks okay yeah so this is can't tell if you guys can see this at all but this is kind of a typical example of what a thin branch looks like in an image so these are actually two images it's a quite a close branch and so get a sense of like here's here's the branch and then here's the branch so if you look closely it's it's coming out of the ground over here and then it comes all the way down very


close to the camera and so the actual correct sort of match of this stick is moving from here to here and then there's another one over here that's moving like that so if you if you look closely you can kind of get a sense of that and so this is just now maybe conveying a sense of the the type of problem here which is one of the hardest things we need to do which is just carefully pick out objects and and there's kind of the outline in this image of what what we're looking at and actually the a real key here is like if


you flip between two images you can tell but if you're just looking at a single image you can't like no no human can and really no algorithm can either like you can look at this kind of scene all day I'm a tangle of branches and not have any idea what the closest thing in there is which really showed that that's kind of what I was getting out of the difference between a road type scene which is very regular and this type of scene here's here's another fun one where we're actually above powerlines so there's there's one power line here you


can see that's moving there's another think really faint one here and then there's two right here that you can see very clearly on this part of them disappear mostly up up in that section okay awesome so we have you know we can play with simulated data synthetic data and the cool thing about that is you know we can we know everything about it and we can try to make it realistic we also have real data and the problem with real data is we don't have any ground truth for the depth which is a which is a very tough problem and there's a whole body


of research that focuses on trying to learn from unsupervised data which is really interesting the problem is that slide I showed about all the visions the things that make vision hard none of those things are photo metrically consistent so if that's your cue to learn depth it's that's not a solved problem right because the that that cue won't get you the really hard things so here's it's kind of a fun example of something where we took we took Suns like synthetic Suns that we generated and we just put them in real images and


the kind of fun thing you can do there is to say oh well whatever you predicted before the Sun was there you should predict the same thing with the Sun so that's kind of one way of getting at just without knowing everything at least gaining some robustness to a really big bright Sun okay let me show a little bit about tracking and detection so this is kind of a fun video of just tracking somebody through a forest with a bunch of lighting changes and occlusions and pose change so people change shape a lot and we're reproject ejector ii into the


image which you need the a pretty accurate estimate of the robots trajectory to do you can and then on the top right is actual output of a deep network that predicts the pixels of this person instance at that time step and then the bottom right is just that's just the top-down view of where where the drone was and where the person was and then the red vector was it's just an estimate of the velocity so like what's in a linear sense where where's that person gonna go in the next next a little bit cool so the way we do this is


kind of the way we do everything else which is a combination of geometric reasoning and and and deep learning for semantics and the images come to be combined with like 3d motion and things we know tend to be true about the world so oh yeah well so multi object tracking is even if you're just tracking one person which is more common tracking multiple objects is really useful because you can then discriminate between people rather than just saying everyone else's background you can say oh this person a disappeared the person


B matches person B from before rather than person B also just appeared and which one's the right person so these are our three co-founders just running among each other and we're doing a pretty good job of keeping them in frame or keeping him as the correct estimate and then on the bottom oh boy oh boy okay on the bottom right here it's kind of a fun failure mode where it's hard to see but so able run behind Matt here and then or Adam and Matt pops out at the same instant so like there's a person one person goes behind and hides and the


other person that was hiding pops out and they're kind of at a similar velocity and so the the things you're combining are like where are they their positions they're their velocities someone's unlikely to just suddenly stop but it can happen and then they're like estimate the visual estimate of their by appearance which we do with with deep networks and so in this case the the visual estimate is similar enough that like the chance of one person disappearing and another appearing and then being at the same velocity is like


very low and it clearly yeah they're basically learned features so essentially it's based on the appearance of an instance of an object and there's pretty solid body of literature about how to have to try to learn those things but you tend to have learned through examples of people that look there's different constraints based on what type of like use you want to do but typically it's like you have similar instances you have the same instance of a person but from many different views and poses and and lighting and all that and then you


have multiple people that look similar and you train positive and negative examples and you you learned that kind of task this is very common for like yes like surveillance camera datasets they tend to be easy to collect where they're just fixed now again the drone is a lot harder because it's moving and people tend to be moving very quickly and running away and doing flips and stuff which make the problem a lot harder but the concept is pretty similar yeah oh yeah so in the just for the sort of consumer follow you skate based use take


off and then whatever is in the view of the user camera you'll see just icons on the phone so we must we do people in cars and so you'll just see a little plus icon and you just tap and then and then it works and it shows that halo that was in the launch video earlier on the phone we also have this thing called the beacon it's another accessory so this guy just a little puck that lets you basically fly without your phone and it has a it has a GPS that we trust a lot more and so what this does is if it lets the drone if it loses visual track


and you view like you get stuck or something fails it can basically try to just combine between that and the GPS estimate and then find you again and pick you up visually and so the idea is to just to make it much more robust in terms of if you're skiing or something that you you're just hands you don't have to worry about it and so the other way we do it which is kind of fun is if you just if you take off like this it will take off and if there was someone standing right there it will just track that person okay cool


say a little bit about planning so I think generally we we have a bunch of knowledge of the environment it's all uncertain there's no guarantees about anything because you know we're not sure if we saw the thin branch where we think we know where we are there's uncertainty about all these things we try to reason about that and the best way possible in the end like we have some measurements and we need a policy that decides what the propeller should do such that the drone accomplishes our goals you know some of which are are


down here like obstacle avoidance dynamic limits smoothness effective tracking and cinematic quality of the video and these are all you know that there are hard trade-offs with each other it's it's not easy to achieve these things in all but the simplest of cases so I think it breaks down a way that I think you you've you've all kind of seen before so we won't go into this one but basically like we have the the biggest the most interesting part of the problem is figuring out what the cost function should be based on everything


we know and so based on the current state of things in the full environment what what are the actions we should choose that that meant like that give us the best result and defining that function is a lot of work and trial and error and investigation and that's that's something we just work at on just iteratively like trying things out flying it seeing what fails trying to like figure out what our discrepancies between we really want and what we saw which is which is a really fun problem and then you have to find you actually you need


to find an action that accomplishes that goal and that that's the cycle and then either you didn't have the right objective or you didn't find the minimum for that objective well enough to do what you wanted and in terms of solving it I think you guys all know but like you can you can learn you know that's the the promise of deep reinforcement learning is that you can well potentially learn both of these things from data and you can do sampling techniques you need a trajectory library you can do a tree based search or you


can do optimization and you can you can do trajectory optimization and mile predictive control to solve these things and so we we visit we do a combination of all these things in a way but but as cool is like we we run something on board that runs at 500 races per second that is reasoning about kind of the whole stack between all the way up to like obstacle ones and cinematic quality all down to like what are the dynamic limits on the rotors and the more you can reason about the whole system together rather than in separate little


layers that have their own boundaries the more you can get out of the robot I think that's kind of universally true the trick is that you have to it's you have to be a lot more careful it's a more dangerous game to play and again we we also use sampling techniques and they have they have a good role to play within the system then I'll just add a note about simulation because it's it's awesome and super useful especially for motion planning there's like for computer vision you want is like really photorealistic images that you've


carefully curated for planning you don't care you just need something that like is good enough to see your behaviors and try things out on so this is like you know we literally drive this guy around with a joystick at our computers and he just kind of slides around the world and we just used that to test basic things and then even without images we have these scenes where basically we generate a random forest and we honor we want to say compare we're going to figure out okay how far away from obstacles should the drone


stay to not crash into them versus like how often is it going to get stuck in a dense forest if we do that so we can like we tend to run logs against hundreds and thousands of with regressions against a ton of just recorded log data right so we record all of our flights all of the images and we can rerun all our vision algorithms and check like those types of things but for planning you can't change what the robot did right it's it's it's all kind of off policy and so simulation is really helpful there and so we can run an


experiment like I would generate a bunch of different random forests and have the subject run the same path through them and then see how often the drone hit something or get stuck and have a statistical sense of what to do there the problem is always like something bad happens we look at that log and then like okay we can tweak this thing to fix it but then did we just make five other things worse and we can't like do we can't do a thousand real flights to test that in any sane way so maintaining the sort of infrastructure to handle all


those things is really critical yeah yeah yeah I mentioned I would fly in here but I don't think I'm allowed to yeah that's a good question we identified the dynamics model which is somewhere between them but we definitely model the dynamics of this drone it's not like a model free black box and the reason because it's definitely model vulnerable so it is you know the propellers there are aerodynamic effects that we understand quite well there's the inertia's of the drone and properties of it and so we we model


everything that we can and we try to fit parameters to it based on flight data and then I think that's that's pretty different from other types of robots that are more like especially when you're dealing with contacts and more complicated things but this is not that many degrees of freedom there I think the complicated part comes from aerodynamics so once you're once you're flying fast and you're in turbulent flow and you get these types of like pitching moments and the accelerations your rotors play into things so that there is


a lot of complexity there but it's also complexity that has been pretty well modeled in aerospace at least in terms of the structure of something that you can learn so yeah yeah uh-huh we don't I mean you can't you can't ensure anything because well it yeah I guess it's a complicated question I does not exactly sometimes people ask like how do you guarantee that it's going to be obstacle free for example and and the answer to that is we can't because of perception and so nothing is ever guaranteed but we we do our best so in terms of like if


you have a nominee MVC how do you make sure that it converges well it depends a lot on how you're solving it if you there are good ways to know if you've made it worse than your initial guess for example so if you if you do that you can try to fall back to something else or you just like try to work towards robustness so have have ways of just managing what you're doing such that you at least have some path that moves you forward and is not catastrophic but the best way to I think the best way to do that is just to