try to carefully design your problem there constraints in the setup of what you have to to have you know stability and convergence and and there's there's a lot of there's a ton of work that could be put into that that is that is tricky but there's no fallback that's a good fallback because let's say a fallback is like you know go constant velocity well you might just run into something or like yeah but you can but you could for example say oh I'm gonna stop carrying about any of my other high-level stuff and I just want to like


come to a stop with the obstacle avoidance if it's a really bad scenario yeah okay just another video of simulation and the kind of cool thing about simulation so this is this is actually the from an SDK that we have it's kind of in a private beta SDK where people sort of we let people work on it that we think make a lot of sense to to kind of develop on our drone but the the cool thing is to share that simulation between like our purposes and actually being able to turn the drone into something that you can program so at a

Lecture 23 Guest Lecture: Drones -- CS287-FA19 Advanced Robotics at UC Berkeley


high level like make it move around make it do things but not have to care about obstacle avoidance and and again the low-level stuff so we're it's it's a hard problem because controlling a drone like you have to understand a lot of the internals of the system to do it so we're kind of always thinking about what is the way to try to design an API such that it's it's intuitive for people to understand especially people that aren't experienced drones or even robotics to be able to control things and get what they want and sort of understand why it


might not be able to do something and why it might go do something okay cool let me show a couple of other cool things so we estimate the wind online which is really useful for to have as a model to do more effective planning and aerodynamic control the drone this is something neat it's a lot less relevant for this drone but the first drone had perimeter guards and the reason I had perimeter guards is to have cameras that can see like on the outside that's why we very carefully designed the geometry of this drone to try to get


rid of the perimeter guards because they're so expensive and bulky and and bad for performance while still having visibility but the cool thing was it could bump into things and not have the propellers touch it and in this case like we go we fly into this glass intentionally and we actually detect that collision and we add that obstacle into our map there so we won't do it again which is a pretty neat thing to do and then yeah we look a lot at sort of recovery from dynamic new verse so we call this inverted pizza toss where you


your initial condition is like upside down and then spinning quickly and then the drone kind of has to come down and recover this is another fun one where there's prior work on this as like its own controller but something neat that we found was if you actually basically just with parameter changes to our planning system if we shut off one rotor we find this kind of mode of flight where it's still semi stable where you're not controlling yaw anymore but you're able to sort of hover and come down to Al and this is all in simulation


these so far because the question of state estimation in this in this scenarios is a lot harder but that's pretty cool that like by just spinning really fast you're able to stay hovering with three rotors and in fact what we did was sure that we could again in simulation actually follow somebody and avoid obstacles while just having three propellers and spinning like a like a madman so I don't think anyone's shown that before okay these are slides from cvpr it's a vision conference so I'm gonna skip those I wouldn't say that in fact I already


talked about yeah don't work on deep learning benchmarks and try to improve them by one percent that's a sucky thing to do work on hard things and real robots and that's I think what this class is all about so that's awesome okay and I'm just gonna show a one last video which is one of my favorites just for drones following Jack here as he goes down a waterslide just in in every direction and it's just a fun thing to watch all right cool yeah so we get a lot of sweet videos and again so our goal is to like bring these drones to doing useful


things not just for video capture for you know content for consumers and action sports but also for inspection and mapping and construction and all all these places where you need a trustworthy reliable robot that can move around anywhere and get and sense sense things basically all right that's that's it yeah thank you cool and then yeah if any of you guys are like looking at internships or I know a lot of your graduate students so I'd be happy to hear from basically anybody in this class we have you know we have a we have


hi there today we're looking at supervised contrast of learning by people from Google research and MIT now this paper proposes a new loss for supervised learning and you might recognize that this is a big claim so for ever now we basically use this cross-entropy loss in order to do supervised training of neural networks this paper proposes to replace that with the supervised contrastive loss and let's jump straight into the results here they say our supervised contrastive loss outperforms the cross-entropy loss

Supervised Contrastive Learning


with standard data augmentations such as Auto augment and Rand augment so these are some of the previous state-of-the-art data augmentation techniques used together with the cross entropy loss and they say there are supervised contrastive loss outperforms them you can see here on image net which is the biggest vision benchmark or the most famous one this new loss the supervised contrastive loss outperforms these other methods by something like a percent one percent is a big improvement on image net right now so they claim it

Supervised Contrastive Learning


is a big claim right you recognize if this is true this could be a game-changer basically for all of supervised learning and supervised learning is really the only thing right now in deep learning that works so it could revolutionize the field but so here's the but it is actually not a new loss to replace the cross entropy loss and that's they do they do come about this pretty quickly some and you don't think they're they're dishonest or lying or anything here but it is it is sort of if you start reading you like what this

Supervised Contrastive Learning


is a new loss it is not it is a new way of pre training the network for a classification task and so let's look into this so if you look at what does what does mean to build a classifier in this is what you usually do this is supervised cross-entropy training you have an image and the image here is of a dog you put it through your network and you obtain a representation so the representation here R is this last layer or the second-to-last layer and you put that through a classification layer and then a soft Max and what you get as an output

Supervised Contrastive Learning


is basically a probability distribution and let's say you have three classes here there's dog there's cat and there's horse and let's say the network doesn't yet isn't yet trained very well so the probability for dog here is fairly low so this is basically what the network thinks of that image like which class does it belong to with what probability I also have this label right here so the labeled dog for that image what you do with that is you do a one hot vector so that would look like this so the one is at the position where the correct class

Supervised Contrastive Learning


is and then the cross-entropy loss takes all of this and does the following there's a sum over all your classes in this case you have three classes and let's call these the labels l and you want to always take the label of the class times the log probability that the network thinks belongs to this class so you can quickly see that this if the label is 0 so for all the incorrect classes that means this entire term drops away and only if the label is 1 so only the correct class that will result in the log probability of the class

Supervised Contrastive Learning


where the label is the correct label right so in order to make this a loss you actually have to put a negative sign in front of here because you want to this so this entire thing reduces to the log probability of the correct class this is what you want to max semi's there for you if you want to minimize something you need so you minimize the negative log probability of the correct class which means you maximize the probability a a if you've never looked at the cross entropy loss like this it is important to notice that

Supervised Contrastive Learning


you're gonna say hey all this does is pull this here up right and it doesn't do anything to the other ones but you have to realize that this softmax operation since this is a probability distribution all of this is normalized to sum up to one so implicitly you will push these down through the normalization right so what this does is it pushes the correct class up and it pushes the other classes down so this to look at this is going to be important later because if you look at what this representation here does so again you

Supervised Contrastive Learning


have the network produces a representation here this is 2,000 dimensional and then it does it adds on top this classification layer this classification layer is simply a linear layer and then a softmax on top so how you have to imagine this is that there is a representation space this 2010 shell space and the representations are made in such a way that the labels such that sorry let's have three classes here the representations are made in such a way that a linear classifier can separate them correctly right so here

Supervised Contrastive Learning


this would be like a boundary and then this would be another boundary and this maybe would be another decision boundary so you can see that the linear classifier can separate the classes well that is the goal if you use this soft Max cross-entropy loss that is implicitly what will happen in the representation space W all the cares about is that the classes are on one side of the decision boundary and everything else is on the other side of a decision boundary so if if you have the network isn't trained very well at the beginning and

Supervised Contrastive Learning


you maybe have a sample of the green class here it will push the network such that the representation of that sample will go on to the other side of this decision boundary and it will push the decision boundary at the same time to make that happen more easily right so it will optimize all of this at the same time that's what you do that's how you optimize representations so this work here and another work has said wouldn't it be great if the representation and decision boundaries weren't just trained at the same time for this but we learn

Supervised Contrastive Learning


good representations first such that classifying them becomes very simple and in essence what this paper says is if we have a representation space W shouldn't images of the same class shouldn't we just make them close together you know so without caring about decision boundaries we just want them to be close to each other and we want them to be far apart from other classes if that happens you can see that a linear classifier is going to have a very easy time separating these classes later so that's exactly what this paper does it has a

Supervised Contrastive Learning


pre training stage and a training stage so in the pre training stage this is over here su provides contrastive in the pre training stage it simply tries to learn these representations right like over like down here such that without the decision boundaries class think images of the same class are close together and images of different classes are far apart which notice the the subtle difference right to the cross-entropy loss where you just care about them being on one or the other side of a decision boundary and in stage this so

Supervised Contrastive Learning


this stage one and then in stage two and there is where where it comes in you basically freeze the network so you freeze these weights down here these are frozen you don't train them anymore all you train is this one classification layer so the represent actually freeze also the representation layer here you only train the classifier on top in stage two but you train it using soft Max and using the cross-entropy loss so you you train the classifier in the old cross entropy way using just normal supervised learning the difference here

Supervised Contrastive Learning


is that the stage one free training is is what's training the network and the cross entropy dose only trains the classifier right so let's look at how this pre training actually worked what is using what it's using is a method called contrastive pre-training now in contrastive pre training and they have a little diagram up here what this does is if you look at the classic way of doing contrastive pre train you have to go to the unsupervised pre-training literature people have kind of discovered that they can improve a neural network by

Supervised Contrastive Learning


pre-training it first in an unsupervised way and this is also called some of these methods are called self supervise so the advantage here of self supervised or unsupervised pre training is that you don't need labels what you want to do is simply to make the representation space somewhat meaningful right so you simply want the network to learn representations of images that are somehow meaningful right that are there and here's how you do it so you want to take an image like this dog here and then you want to randomly augment this

Supervised Contrastive Learning


image which just means you want to produce different versions of the same image in this case down here this is a random crop it's cropped about here it's still the same image but it's a different version of it in the case here you can see that it's flipped left right and the brightness is slightly increased so these are just different versions of the same image and what you also want are what's called negatives negatives are simply different images from your data set right for example this or this or this you don't

Supervised Contrastive Learning


care as long as they're different right you just sample a bunch and what you want so you're you're embedding space and they make it big a deal here that they are normalized and that seems to work better but this is not necessary for the idea to work the big idea is here that if you have an image right here let's say this is this is the dog and the blue dots here are the augmented versions of the same dog and the green dots are all the other images in the data set what you want is that the all the images that come from the original

Supervised Contrastive Learning


same image are pulled close together and everything else is pushed apart right so that's why these are called positives and these are called negatives so the contrast of training basically means that you always want to have a set that you pull together in representation space and assets called the negatives that you push apart so the network basically learns about these random transformations that you have here the network kind of learns what it means to come from the same image it learns to be robust to these kind of transformations

Supervised Contrastive Learning


it learns about the data in general and how to kind of spread the data and embedding space with these transformations so this usually ends up in a pretty good representation space and people have been using this in recent years in order to gain significant improvements now the problem here if you specifically do this to pre train a classifier is the thing they show on the right so on the left here you have a picture of a dog right but if you just do this self supervised you do it without the labels so it can happen

Supervised Contrastive Learning


that this image here shows up in the negatives but it is also of a dog right and now this image here is going to end up maybe being this image here and you see what happens to it it's a green one so it's gonna get pushed apart and this is going to make the entire task for the later classifier much harder because if they are pushed apart from each other how is a linear classifier going to have them on the same side of the decision boundary while having everything else on a different side right so the the task

Supervised Contrastive Learning


here is implicitly making the task for the later classifier harder by pushing apart samples that should be of the same class and so this is this is not happening if you introduce a labels to the pre training objective that's what they do the supervised contrast objective now you still all you want to do is here we're going to draw the same embedding space and we're going to draw this original dog image and we're going to draw the Augmented version of the original dog image but now we also have the following we also have these images

Supervised Contrastive Learning


which are images of the same class so we're going to put them in black here and let's say the augmented versions around them in smaller black thoughts augmented versions of those right you can augment them as well and then you have the negative samples and the negative samples are not just any images but just images of different classes so you just go over your mini batch and all everything that's of the same class we could becomes positives including their augmentations and everything that is not in the same class becomes negatives and

Supervised Contrastive Learning


also you can augment them as well so now we have a bunch of things in our embedding space and our objective is going simply going to be again we want to push away all the images that not of the same class as our original as our red original image which is called the anchor so all of this needs to be pushed away but now we want to pull together all the Augmented versions of the original image but also we want to pull together all of the other images of the same class including also their augmented version so all of this is

Supervised Contrastive Learning


going to be pulled together so not only does the let work learn about these augmentations which again for this idea the augmentations aren't even necessary the network there learns a representation space where images of the same class are close together which again is going to make the task of later linear classifiers that needs to separate this class from other classes very very easy and again the other images aren't just going to be pushed away but if they're from the same class let's say this and this image are from

Supervised Contrastive Learning


the same class all of those are going to be pushed apart from a red dot but by themselves being pushed together to their own cluster here of their own class I hope this makes sense and I hope the difference to the cross-entropy objective is sort of clear the cross-entropy objective simply from the beginning just cares about which side of the decision boundary iran while this pre-training objective first cares to put things close together that are in the same class and then the decision classifier will have a much easier time

Supervised Contrastive Learning


the reason why this works better than the because because it's not entirely clear from the beginning that why this should work better because it's working with the same information it's just because people have generally found that these pre-training contrastive pre-training objectives they just are somewhat better at exploiting the information in the data set then if you just hammer on hammer with the contrastive sorry with the cross-entropy loss from the beginning so but it is not fully explained yet why this works better be

Supervised Contrastive Learning


as it's working with the same data again the difference here is that the previous methods of contrastive pre-training the self supervised ones they did not have access to the labels and the advantage of that is you can have a giant database of unlabeled additional data that you do the free training on whereas here we do the pre training including the labels so here the label dog is an intrinsic part because we need to know which of these samples we need to pull together but that also means we cannot leverage the

Supervised Contrastive Learning


may be that we have more on labelled data and unlabeled data is pretty cheap to obtain so that's the advantages and disadvantages here so this new loss so they they do compare this here and usually in these contrastive objectives you have somewhat like two encoders want to encode the the anchor and want to encode the Augmented versions and this one is like a momentum with shared weights and so on it all of this isn't really important if you want to look into that look into papers like momentum contrast or I did one on curl for

Supervised Contrastive Learning


reinforcement learning I think the the general gist of it is clear so they compare the formulation of their loss to the self supervised one usually it takes the form of things like this so the one is the the anchor here and then the ZJ I would be the positive example and you see here that the inner product between the anchor and the positive example sorry about that the inner product should be high because here the loss is the negative of whatever is here so if you minimize the loss you say I want the inner product

Supervised Contrastive Learning


between my anchor and whatever is the positive sample to be high and all everything else here which includes the thing on the top but it also includes everything else I the inner product to be low and which is exactly the thing where you push you pull together the positives and you push apart everything else the that is the standard objective that you had before they they extend this but looks almost the same so compared to the unsupervised objective now first of all they extend this such that you can have more than

Supervised Contrastive Learning


one positive sample now this is also possible in the unsupervised way so they just augmented by this and they also now this is the crucial part they include the labels into the pre turning objective so they say everywhere where I and J have the same label should be maximized in the inner product so should be pulled together while everything else is being pushed apart yes so they say we generalize to an arbitrary number of positives and I also say contrastive power increases with more negatives I think that's just a finding that they

Supervised Contrastive Learning


have that when they add more negative so when they increase the batch size that contrastive power increases they do analyze their gradient which I find it's pretty neat you can already see that if you formulate a loss of course the gradient is going to go in the negative direction but they make it clear that if you look at the gradient for the positive cases what appears is this one - pIJ quantity and the pIJ quantity is exactly the inner product between I and J normalized of course so if you minim so the gradient is going to point into

Supervised Contrastive Learning


the negative direction of that for the positives which means you you're gonna pull them together and it's going to push into this direction with for the negative classes which means you you push them up and they also analyze what happens a in with relation to hardness so they say there are two kinds of if you just look at the positive samples there are two kinds there are easy positives where the network has already learned to match them closely where the inner product is almost one if you look at them that means the pIJ quantity is large

Supervised Contrastive Learning


right because that is basically the inner product and you look at this term this term is exactly what we saw in the gradient then you see that this here since this is one this entire thing is zero this is all so highs this is close to one so this entire thing is zero this is almost zero but if you have a hard positive where the network hasn't learned yet to align the inner product properly or align the representation properly then the angle between the things again these are normalized the angle is they are approximately

Supervised Contrastive Learning


orthogonal so the gradient magnitude is going to be this here is going to be approximately 0 so this is close to 1 and this here since this is also 0 is also close to 1 so this is going to be larger than 0 which means that their loss focuses on the examples that are that the network cannot yet represent well according to their objective which makes sense right first of all but second of all it that is exactly the same thing as in the cross entropy loss if you if you look at the cross entropy loss and you have a

Supervised Contrastive Learning


situation where the network is really good already for a given sample so it already puts a dog into the dark class then the the gradient will not be pulling much for that sample it might mainly focuses on where you're still wrong so it is like I appreciate the analysis but it is not a notable difference I think what they want to show is that their loss if you do gradient descent really does what it is supposed to do namely first of all it does this polling together pushing a part of inner products for the positive and negative samples and it

Supervised Contrastive Learning


mainly focuses on samples where you not yet have found a good representation to align them with others it focuses on pairs that are not yet correctly close or together or far apart they also connect this to the triplet loss where they can show after some approximation that if their loss only has one positive and one negative sample it is going to be proportional to the triplet loss the triplet loss is basically where you have an image and you find one positive I think there's going to be of the same class right here and you find one

Supervised Contrastive Learning


negative of a different class and you try to push those apart while pulling those together the problem here they say is the problem of hard negative sampling in order for this to make sense you need the negative sample to be what's called a hard negative sample so this the call is hard negative mining because you only have one negative sample you better make this something where the network can learn from right and if it's too easy the network can't learn anything and they're more thereby you have the problem of hard negative mining where

Supervised Contrastive Learning


you often have to filter through your mini batch or even through your data set to find a good negative sample to go along with this pair of positive samples but I don't I don't really see how their method except that you know it has a bunch of positives and negative samples except for that which I guess you could also apply to the triplet loss there's not really a difference here again your if your method is a contrastive method you do have the problem that if you simply sample at random your negative samples are going to be become easier

Supervised Contrastive Learning


and easier over the training over the course of training and you get the problem of at some point you're gonna have to do actively sample hard negatives I think this paper just gets around it by having huge batch sizes so yeah but again they do get state-of-the-art on imagenet for these types of networks and augmentation strategies and they do look at how their loss appears to be more hyper parameter stable so if they change out the augmentation if they change the optimizer or the learning rate you can see here that the spread in accuracy is

Supervised Contrastive Learning


much smaller than for the cross entropy loss except here but it is it is hard to compare variances of things that don't have the same means in terms of accuracy so take this on the right here with a grain of salt they also evaluate this on corrupted image net so there's an image net data set where you it has several levels of corruptness of the data set and you can see your accuracy goes down but the accuracy for the cross entropy loss goes down faster than for the supervised contrastive loss you see they start

Supervised Contrastive Learning


together like this and they go further apart now it is not clear to me whether that's just an effect like if you just trained a supervised contrastive loss also to this level whether it would fall off at the same speed or whether because it is the supervised contrastive loss it would kind of match that curve it's not clear whether that's really an effect of the difference of the losses or is just an effect of the fact that they aren't the same accuracy to begin with again this kind of shifting you can't really

Supervised Contrastive Learning


compare things that have different means in the first place but that's it is an interesting finding that their method is more stable to these corruptions I just want to point out at the end their training details and just highlight they train for up to seven hundred epochs during the pre training stage which is I think standard but mad and they trained up models with batch sizes up to 8192 so you need like a super TPU cluster to run these kind of things and I am never exactly trusting of numbers like this even though it's

Supervised Contrastive Learning


it's kind of a good improvement it is still like a 1% improvement and in these small numbers I feel I just feel the there might be sir there might be a big effect that things like batch sizes and how much you put into computing how much compute you put into it and what else you're doing there might be so much influence of that that I first want to see this replicated multiple times across the entire field before I'm going to really trust that this is a good thing to do alright so I hope you like this if you're still here

Supervised Contrastive Learning


hi there today we'll look at mix-up beyond empirical risk minimization by a Hong Yi Chang Mustapha sis Yong and dolphin and David Lopes Pass so this paper is actually pretty simple but it introduces a technique that apparently helps with training classifiers and it I have it seen it used in practice so there must be at least something to it it is ultimately very simple so usually you input a data point X into your neural network in deep learning so f of X that's your neural network your known neural network has parameters

mixup: Beyond Empirical Risk Minimization (Paper Explained)


theta you get some output Y hat and along with the X you also have a y:i true label and then you have a loss function that compares what you output with your true label and then you just try to make that loss smaller you want to adjust your parameters so next time you see data point X its output will be a little closer to the true label Y and we call this empirical miss empirical risk minimization because you don't actually what you think is that your X comes from some distribution from some data distribution D like the space of


all natural images or the space of all of language but what you actually have is you have a data set of a finite amount of data that you can put a that you can sample x and y from and so instead of your minimizing your true risk you minimize your empirical risk the empirical risk minimization right here now what's the problem with that the problem is that you can get overly confident about your data points and nothing else and that will hurt your generalization so if you have a data point let's say right here and another


one right here your network is basically so this this is a this is class 1 this is class 2 your network is going to maybe make decision boundaries like this and like this where it says ok here is class 1 and here is class 2 but it could you know it's very conceivable that here it says ah here is class 4 and over here is class 7 and right here through is class 9 and by the way here class 4 again so the the empirical risk minimization leaves everything in between the data points open now what this paper proposes is that we should


not only train our our classifier on these data points but on all the data points sort of in between the two and this is the mix of data points so this data point here might be constructed if this is a and this is B from zero point one times B right and plus 0.9 times a because it's mostly a and it's a little bit B and now you think what are the labels here if a belongs to class one and B belongs to class two then of course the label of this data point is zero point one times the class of B which is 2 plus 0.9 times the class of a


which is 1 ultimately because what you do is you input a class like class number two if you want to input this into a machine learning model you just you don't just say it's class number two what you input is a distribution that is basically has zeros everywhere so these small things there zero zero zero one zero and this here is at class number two so this would be class number one class number two class number three right you input a distribution like this if you want to express class number two now in our sample right here what we


would input as a label is simply a mix between class so 0.9 0.9 of class 1 0.1 of class 2 and then zero everywhere else so this would be our label for the data point that we construct right here this will be our sorry the top one would be our data point formally you take two data points and you mix them using this lambda mixing factor that'll give you a new data point that's in between the other data points and you take the two corresponding labels and you mix them accordingly as well and that will give you the label for that data point and


now your model will learn to basically smoothly interpolate so you will teach your model the thing on the left here is class number one right that's class number one the thing on the right is class number two this here is a half of class one and a half of class two so the model basically learns a smooth interpolation where the situation that's here on top is probably not going to happen anymore but what it would do is it would sort of create these ISO lines around class two and then around class one where it's sort of smoothly getting


less and less sure about the class of the data points but on the way it is always either class 1 or class 2 and they say that can help the generalization performance and it's visible or why right it's just the only thing that's not it it's not clear from the beginning is that this kind of interpolation actually makes sense because if this means we sort of linearly interpolate between two images so if we have two images we just take half of one and half of the other and that will be not a natural image it will


be kind of a blurry thing otherwise you know all our problems would be solved and we could just linearly classify things but in any case in practice it actually seems to help probably because interpolations of two images linear interpolations are still much more like something like a natural image then any random noise you could come up with so they say it isn't code right here code is pretty simple simply want to mix the two things and the mixing factor this lambda here comes from a beta distribution and they use a beta I


believe of 0.4 or something just want to quickly show you this is the red line here so the red line as you can see mostly most of the time they're going to either sample the thing on the very left or the thing on the very right that means the either sample the first or the second data point but some of the time they actually sample something in the middle and it's it's fairly uniform in the middle so it appears like a good distribution to sample from if you want to sample these mixing coefficients and by adjusting the the actual number of


alpha and beta here you can determine how many times you sample the original data points versus how many times you sample something in the middle okay on this toy data set right here they showcase what mix up can do so in a classic model you have the orange and the green data points and blue is basically where the classifier believes its class one you see this very hard border here it's quite a hard border now you only have two classes here and so the hard border is sort of a problem in itself because if you think of for


example adversarial examples all they have to do is basically get over that one inch and the classifier is already super duper sure it's the orange class right whereas if you use mix up your border is much much much more fuzzy it's like yeah it's only really sure here and out out here everywhere but in the middle it's sort of like me I don't know and so that's kind of a more desirable situation and of course this here works particularly in this in this linear to the setting but as we can see the same reasoning applies to sort of higher


higher layers and higher dimensionality data points right I have to seem to lost the ability to zoom Oh No that's back okay and that's basically it for this paper this is all they do they propose this method and then they test it they say something interesting here that mix-up converges to the classical method as off approaches zero so that would push your beta distribution basically in the middle all the way down and you would only sample from the very left or the very right so you can smoothly interpolate between this mixing


and the classic method they so their main results are we apply this to classifiers and what I like is since again is also a classifier so the discriminator is a class fair they also apply it to Ganz and they outperform unstable eyes the classic training on ganz they show that it's more robust towards adversarial attacks because it's not so sure about intermediate things and they generally outperform other methods but also they do this nice investigation here where they measure the prediction error of in between data


and what it means is they say a prediction is counted as a miss if it does not belong to Y I or Y J so you have a sample right here X I in the sample right here X J and you look at what the classifier says in between the two data points so you just interpolate the two data points and just measure what the classifier says and whenever the classifier either says y I or Y J either either a label of those two data points you count it as correct and you only count it as incorrect if it says something else and you can see here if


you train with the classic method erm the these errors happen much more often that's exactly the situation I pointed out at the beginning where in the high dimensions it can you know occur that all sorts of decision boundary sneak here in between the two data points and by interpolating between them during training you sort of much reduce that you reduce that effect a lot now this they also say that the gradient norm of the gradients of the model with respect to input in between training data it happens the same thing the norm


of the gradients in the middle is also much much lower and this yeah this investigation I find pretty cool I have to say I have seen mixup in practice so it might be useful I've read a paper where they basically say oh it was a big transfer paper yeah where they basically say it is useful if you have for example if you have little data and a big model so you can sort of regularize the model and is also useful to know that they did test this with drop out so we can they compared it with drop out and the conclusion is basically that this is


something else than drop out so it's not doing the same thing drop out of course it means you drop out some of the data points in intermediate activations and that sort of gives you a noisy version of the data point this here can actually be combined with drop out which means that it gives you an additional benefit you see right here most of the best numbers happen when you use mix up plus drop out so it seems to be just an additional regularization on top of drop out pretty cool pretty cool investigation awesome alright so if you


and us in the car today we are going to talk about AI machine learning and deep learning the reason why is because AI is everywhere in the news the government loves to put money on AI project a pinch of cube todo investors love to put money on machine learning powered companies investment and everybody's talking about deep learning algorithms we are going to define what is AI what is machine learning what is good learning at the end I'm going to tell you how a full-stack programmer like me that doesn't like math and a dozen really

머신러닝 vs 딥러닝 vs 인공지능? A.I. 개념정리!


like calculus in all that can get started working with artificial intelligence and machine learning ai ai is divided in two categories narrow AI in general area Hollywood and Netflix and all the movies are in general AI general AI is machines that do everything that humans do and better that can do a general purpose AI that can do anything so it can talk it can learn to play games it can communicate it can make a judgment it can just be exactly like a human in varies general AI right now in the world in the industry we are in a narrow AI

머신러닝 vs 딥러닝 vs 인공지능? A.I. 개념정리!


narrow AI are machines that can only do one thing and one thing very well where the focus of the machine is narrow the applications are narrow the machine can do one thing well and one thing only an example of narrow AI would be for example the Facebook AI at finding out faces in photos but that's it this AI is not able to learn how to find out dogs in photos because is narrow is general is narrow AI it can only do one thing well and one thing only that's what we are right now but now we need to understand how we teach those

머신러닝 vs 딥러닝 vs 인공지능? A.I. 개념정리!


hey guys here is where machine learning is the way to accomplish a a I so how the machines learn there are many categories of machine learning but the most famous ones are to on supervised learning and supervised learning let's say that we are going to make an application that detects if a food is a hot dog or not a hot dog if we did it in a supervised way what we will do is that we will label what a hot dog is okay a hot dog is a sausage a hot dog is long a hot dog has some sauce on top a hot dog has is between a bun that is a

머신러닝 vs 딥러닝 vs 인공지능? A.I. 개념정리!


label we're going to label what a hot dog is we're going to tell the Machine what a hot dog is right and then we are going to get millions of photos of any kind of food we're going to put that into the machine in the machine based in our labels the machine is going to say ok this photo has 60% chance of being a hot dog so the machine is not thinking the machine is just telling us in a probability way with statistics and mathematics it just train us this has 95% chance of being a hot dog based on the labels that the humans gave to the

머신러닝 vs 딥러닝 vs 인공지능? A.I. 개념정리!


machine one example of supervised machine learning will be for example a music recommendation system the user US will label the songs a good night so we will tell the Machine these are the snow that I enjoy this is the ton of beat that I enjoy this is the kind of artists that I like next time a new song comes the Machine now has labeled data into what is a song that Nikolas likes and then you will know if Nikolas has a 90% chance of liking this new song that's coming up that is supervised learning humans label the data now in

머신러닝 vs 딥러닝 vs 인공지능? A.I. 개념정리!


unsupervised learning the humans do not label the data so for example to do this hot dog app what we are going to do is that we are going to give to the Machine millions of photos of only hot dogs we're going to give that to the Machine we're gonna give the machine any leg and we are going to let the Machine by itself figure out a label that makes a hot dog so when you're going to tell any description of a hot dog we're just gonna give the Machine a lot of hot dog photos in the machine we'll figure out by itself after a lot of time and a lot

머신러닝 vs 딥러닝 vs 인공지능? A.I. 개념정리!


of processing power and a lot a lot of data all right now of course I know this is just a very simple explanation if you want this if you like this topic we can talk more about it in super in future videos but now I need to move on to the last topic which is learning deep learning is just a way to accomplish machine learning machine learning is a way to accomplish a deep learning is called deep learning because it makes use of something called neural networks very smart scientists very smart mathematicians and computer scientists

머신러닝 vs 딥러닝 vs 인공지능? A.I. 개념정리!


and all that they came up with this algorithm that works like a brain you need a lot of data to train that and you need a lot of processing power because it's a very long and computer intensive process but that's it deep learning is a way to accomplish machine learning machine learning is a way to accomplish AI all right now deep learning is being used by companies for example like Google or Google or Tesla for example because they can process and have massive amounts of data and they have massive amounts of money so how does a

머신러닝 vs 딥러닝 vs 인공지능? A.I. 개념정리!


full-stack programmer come up and start working with machine learning and deep learning well if you want to get started in machine learning you have to learn Python that is like the best way to get started if you know Python then you can move on and look into something called tensorflow thankfully you don't have to do all these things by yourself manually you don't have to create a neural networks by OSHA the community has already built tons and tons of things the most popular framework for artificial intelligence is

머신러닝 vs 딥러닝 vs 인공지능? A.I. 개념정리!


called tensor flow tensor flow is in JavaScript and Python so those two languages are super super easy to do get start to do it and you can get started tomorrow if you wanted to also there is this thing called brain jail on brain j/s they already have neural net works deep learning algorithms activation functions I'll on the things already done for you to work with deep learning and no GS so if you ask me it's an amazing thing like I said the community has already built so many things so if you're scared of math

머신러닝 vs 딥러닝 vs 인공지능? A.I. 개념정리!


calculation on that you can go and start playing around with tensorflow or with brain shaders and that will just give you an introduction of what is machine learning without having to take care of all the math in the calculus Python and JavaScript those two languages are gonna be big big big big on machine learning I think Python already is it's a massive in JavaScript is a start in there because it's on the web thank you for watching I hope that you enjoyed this video let me know what you think let me know in the comments

머신러닝 vs 딥러닝 vs 인공지능? A.I. 개념정리!


The "Dirty Jobs" crew and I were called to a little town in Colorado, called Craig. It's only a couple dozen square miles. It's in the Rockies. And the job in question was sheep rancher. My role on the show, for those of you who haven't seen it -- it's pretty simple. I'm an apprentice, and I work with the people who do the jobs in question. And my responsibilities are to simply try and keep up, and give an honest account of what it's like to be these people for one day in their life. The job in question: herding sheep.

Learning from dirty jobs | Mike Rowe


Great. We go to Craig and we check into a hotel, and I realize the next day that castration is going to be an absolute part of this work. Normally, I never do any research at all. But this is a touchy subject, and I work for the Discovery Channel, and we want to portray accurately whatever it is we do. And we certainly want to do it with a lot of respect for the animals. So I call the Humane Society and I say, "Look, I'm going to be castrating some lambs. Can you tell me the deal?" And they're like, "Yeah, it's pretty straightforward."

Learning from dirty jobs | Mike Rowe


They use a band, basically, a rubber band, like this, only a little smaller. This one was actually around the playing cards I got yesterday -- (Laughter) But it had a certain familiarity to it. And I said, "Well, what exactly is the process?" And they said, "The band is applied to the tail, tightly. And then another band is applied to the scrotum, tightly. Blood flow is slowly retarded; a week later the parts in question fall off. "Great -- got it." OK, I call the SPCA to confirm this. They confirm it. I also call PETA just for fun,

Learning from dirty jobs | Mike Rowe


and they don't like it, but they confirm it. OK, that's basically how you do it. So the next day I go out. And I'm given a horse and we go get the lambs and we take them to a pen that we built, and we go about the business of animal husbandry. Melanie is the wife of Albert. Albert is the shepherd in question. Melanie picks up the lamb, one hand on both legs on the right, likewise on the left. Lamb goes on the post, she opens it up. Alright. Great. Albert goes in, I follow Albert, the crew is around. I always watch the process done the first time before I try it.

Learning from dirty jobs | Mike Rowe


Being an apprentice, you know, you do that. Albert reaches in his pocket to pull out, you know, this black rubber band, but what comes out instead is a knife. And I'm like, "Hmm, that's not rubber at all," you know? (Laughter) And he kind of flicked it open in a way that caught the sun that was just coming over the Rockies, it was very -- (Laughter) It was ... it was impressive. In the space of about two seconds, Albert had the knife between the cartilage of the tail, right next to the butt of the lamb, and very quickly, the tail was gone and in the bucket that I was holding.

Learning from dirty jobs | Mike Rowe


A second later, with a big thumb and a well-calloused forefinger, he had the scrotum firmly in his grasp. And he pulled it toward him, like so, and he took the knife and he put it on the tip. "Now, you think you know what's coming, Michael, You don't, OK?" (Laughter) He snips it, throws the tip over his shoulder, and then grabs the scrotum and pushes it upward, and then his head dips down, obscuring my view. But what I hear is a slurping sound, and a noise that sounds like Velcro being yanked off a sticky wall,

Learning from dirty jobs | Mike Rowe


and I am not even kidding. Can we roll the video? No, I'm kidding, we don't -- (Laughter) I thought it best to talk in pictures. I do something now I've never, ever done on a "Dirty Jobs" shoot, ever. I say, "Time out. Stop." You guys know the show, we use take one; we don't do take two. There's no writing, there's no scripting, there's no nonsense. We don't fool around, we don't rehearse -- we shoot what we get! I said, "Stop. This is nuts." I mean -- (Laughter) "This is crazy. We can't do this." And Albert's like, "What?"

Learning from dirty jobs | Mike Rowe


And I'm like, "I don't know what just happened, but there are testicles in this bucket, and that's not how we do it." He said "Well, that's how we do it." I said, "Why would you do it this way?" And before I even let him explain, I said, "I want to do it the right way, with the rubber bands." And he says, "Like the Humane Society?" I said, "Yes, like the Humane Society. Let's do something that doesn't make the lamb squeal and bleed. We're on in five continents, dude! We're on twice a day on the Discovery -- we can't do this."

Learning from dirty jobs | Mike Rowe


He says, "OK." He goes to his box and pulls out a bag of these little rubber bands. Melanie picks up another lamb, puts it on the post, band goes on the tail, band goes on the scrotum. Lamb goes on the ground, lamb takes two steps, falls down, gets up, shakes a little, takes another couple steps, falls down. I'm like, this is not a good sign for this lamb, at all. Gets up, walks to the corner. It's quivering, and it lies down and it's in obvious distress. And I'm looking at the lamb and I say, "Albert, how long?

Learning from dirty jobs | Mike Rowe


When does he get up?" He's like, "A day?" I said, "A day! How long does it take them to fall off?" "A week." Meanwhile, the lamb that he had just done his little procedure on is, you know, he's just prancing around, bleeding stopped. He's, you know, nibbling on some grass, frolicking. And I was just so blown away at how completely wrong I was, in that second. And I was reminded how utterly wrong I am, so much of the time. (Laughter) And I was especially reminded of what a ridiculously short straw I had that day,

Learning from dirty jobs | Mike Rowe


because now I had to do what Albert had just done, and there are like 100 of these lambs in the pen. And suddenly, this whole thing's starting to feel like a German porno, and I'm like -- (Laughter) Melanie picks up the lamb, puts it on the post, opens it up. Albert hands me the knife. I go in, tail comes off. I go in, I grab the scrotum, tip comes off. Albert instructs, "Push it way up there." I do. "Push it further." I do. The testicles emerge. They look like thumbs, coming right at you. And he says, "Bite 'em.

Learning from dirty jobs | Mike Rowe


Just bite 'em off." (Laughter) And I heard him, I heard all the words -- (Laughter) Like, how did I get here? How did -- I mean -- how did I get here? It's just -- it's one of those moments where the brain goes off on its own, and suddenly, I'm standing there in the Rockies, and all I can think of is the Aristotelian definition of a tragedy. You know, Aristotle says a tragedy is that moment when the hero comes face to face with his true identity. (Laughter) And I'm like, "What is this jacked-up metaphor? I don't like what I'm thinking right now."

Learning from dirty jobs | Mike Rowe


And I can't get this thought out of my head, and I can't get that vision out of my sight, so I did what I had to do. I went in and I took them. I took them like this, and I yanked my head back. And I'm standing there with two testicles on my chin. (Laughter) And now I can't get -- I can't shake the metaphor. I'm still in "Poetics," in Aristotle, and I'm thinking -- out of nowhere, two terms come crashing into my head, that I hadn't heard since my classics professor in college drilled them there. And they are "anagnorisis" and "peripeteia."

Learning from dirty jobs | Mike Rowe


Anagnorisis and peripeteia. Anagnorisis is the Greek word for discovery. Literally, the transition from ignorance to knowledge is anagnorisis. It's what our network does; it's what "Dirty Jobs" is. And I'm up to my neck in anagnorises every single day. Great. The other word, peripeteia, that's the moment in the great tragedies -- Euripides and Sophocles. That's the moment where Oedipus has his moment, where he suddenly realizes that hot chick he's been sleeping with and having babies with is his mother. That's peripety, or peripeteia.

Learning from dirty jobs | Mike Rowe


And this metaphor in my head -- I've got anagnorisis and peripeteia on my chin -- (Laughter) I've got to tell you, it's such a great device, though. When you start to look for peripeteia, you find it everywhere. I mean, Bruce Willis in "The Sixth Sense," right? Spends the whole movie trying to help the little kid who sees dead people, and then -- boom! -- "Oh, I'm dead." Peripeteia. You know? It's crushing when the audience sees it the right way. Neo in "The Matrix," you know? "Oh, I'm living in a computer program.

Learning from dirty jobs | Mike Rowe


That's weird." These discoveries that lead to sudden realizations. And I've been having them, over 200 dirty jobs, I have them all the time, but that one -- that one drilled something home in a way that I just wasn't prepared for. And, as I stood there, looking at the happy lamb that I had just defiled -- but it looked OK; looking at that poor other little thing that I'd done it the right way on, and I just was struck by -- if I'm wrong about that, and if I'm wrong so often, in a literal way, what other peripatetic misconceptions might I be able to comment upon?

Learning from dirty jobs | Mike Rowe


Because, look -- I'm not a social anthropologist, but I have a friend who is. And I talk to him. (Laughter) And he says, "Hey Mike, look. I don't know if your brain is interested in this sort of thing or not, but do you realize you've shot in every state? You've worked in mining, you've worked in fishing, you've worked in steel, you've worked in every major industry. You've had your back shoulder to shoulder with these guys that our politicians are desperate to relate to every four years, right?" I can still see Hillary doing the shots of rye,

Learning from dirty jobs | Mike Rowe


dribbling down her chin, with the steel workers. I mean, these are the people that I work with every single day. "And if you have something to say about their thoughts, collectively, it might be time to think about it. Because, dude, you know, four years." So, that's in my head, testicles are on my chin, thoughts are bouncing around. And, after that shoot, "Dirty Jobs" really didn't change, in terms of what the show is, but it changed for me, personally. And now, when I talk about the show, I no longer just tell the story you heard and 190 like it.

Learning from dirty jobs | Mike Rowe


I do, but I also start to talk about some of the other things I got wrong; some of the other notions of work that I've just been assuming are sacrosanct, and they're not. People with dirty jobs are happier than you think. As a group, they're the happiest people I know. And I don't want to start whistling "Look for the Union Label," and all that happy-worker crap. I'm just telling you that these are balanced people who do unthinkable work. Roadkill picker-uppers whistle while they work, I swear to God -- I did it with them.

Learning from dirty jobs | Mike Rowe


They've got this amazing sort of symmetry to their life. And I see it over and over and over again. So I started to wonder what would happen if we challenged some of these sacred cows? Follow your passion -- we've been talking about it here for the last 36 hours. Follow your passion -- what could possibly be wrong with that? It's probably the worst advice I ever got. (Laughter) Follow your dreams and go broke, right? I mean, that's all I heard growing up. I didn't know what to do with my life, but I was told if you follow your passion, it's going to work out.

Learning from dirty jobs | Mike Rowe


I can give you 30 examples right now. Bob Combs, the pig farmer in Las Vegas who collects the uneaten scraps of food from the casinos and feeds them to his swine. Why? Because there's so much protein in the stuff we don't eat, his pigs grow at twice the normal speed, and he's one rich pig farmer. He's good for the environment, he spends his days doing this incredible service, and he smells like hell, but God bless him. He's making a great living. You ask him, "Did you follow your passion here?" and he'd laugh at you.

Learning from dirty jobs | Mike Rowe


The guy's worth -- he just got offered like 60 million dollars for his farm and turned it down, outside of Vegas. He didn't follow his passion. He stepped back and he watched where everybody was going, and he went the other way. And I hear that story over and over. Matt Freund, a dairy farmer in New Canaan, Connecticut, who woke up one day and realized the crap from his cows was worth more than their milk, if he could use it to make these biodegradable flowerpots. Now he's selling them to Walmart, right? Follow his passion? The guy's -- come on.

Learning from dirty jobs | Mike Rowe


So I started to look at passion, I started to look at efficiency vs. effectiveness. As Tim talked about earlier, that's a huge distinction. I started to look at teamwork and determination. And basically, all those platitudes they call "successories" that hang with that schmaltzy art in boardrooms around the world right now, that stuff -- it's suddenly all been turned on its head. Safety. Safety first is ... Going back to OSHA and PETA and the Humane Society: What if OSHA got it wrong? I mean -- this is heresy, what I'm about to say --

Learning from dirty jobs | Mike Rowe


but what if it's really safety third? Right? (Laughter) No, I mean, really. What I mean to say is: I value my safety on these crazy jobs as much as the people that I'm working with, but the ones who really get it done -- they're not out there talking about safety first. They know that other things come first -- the business of doing the work comes first, the business of getting it done. And I'll never forget, up in the Bering Sea, I was on a crab boat with the "Deadliest Catch" guys -- which I also work on in the first season.

Learning from dirty jobs | Mike Rowe


We were about 100 miles off the coast of Russia: 50-foot seas, big waves, green water coming over the wheelhouse, right? Most hazardous environment I'd ever seen, and I was back with a guy, lashing the pots down. So I'm 40 feet off the deck, which is like looking down at the top of your shoe, you know, and it's doing this in the ocean. Unspeakably dangerous. I scamper down, I go into the wheelhouse and I say, with some level of incredulity, "Captain -- OSHA?" And he says, "OSHA? Ocean." And he points out there.

Learning from dirty jobs | Mike Rowe


(Laughter) But in that moment, what he said next can't be repeated in the Lower 48. It can't be repeated on any factory floor or any construction site. But he looked at me and said, "Son," -- he's my age, by the way, he calls me "son," I love that -- he says, "Son, I'm the captain of a crab boat. My responsibility is not to get you home alive. My responsibility is to get you home rich." (Laughter) You want to get home alive, that's on you." And for the rest of that day -- safety first. I mean, I was like -- So, the idea that we create this sense of complacency

Learning from dirty jobs | Mike Rowe


when all we do is talk about somebody else's responsibility as though it's our own, and vice versa. Anyhow, a whole lot of things. I could talk at length about the many little distinctions we made and the endless list of ways that I got it wrong. But what it all comes down to is this: I've formed a theory, and I'm going to share it now in my remaining 2 minutes and 30 seconds. It goes like this: we've declared war on work, as a society -- all of us. It's a civil war. It's a cold war, really. We didn't set out to do it

Learning from dirty jobs | Mike Rowe


and we didn't twist our mustache in some Machiavellian way, but we've done it. And we've waged this war on at least four fronts, certainly in Hollywood. The way we portray working people on TV -- it's laughable. If there's a plumber, he's 300 pounds and he's got a giant butt crack, admit it. You see him all the time. That's what plumbers look like, right? We turn them into heroes, or we turn them into punch lines. That's what TV does. We try hard on "Dirty Jobs" not to do that, which is why I do the work and I don't cheat.

Learning from dirty jobs | Mike Rowe


But, we've waged this war on Madison Avenue. So many of the commercials that come out there in the way of a message -- what's really being said? "Your life would be better if you could work a little less, didn't have to work so hard, got home a little earlier, could retire a little faster, punch out a little sooner." It's all in there, over and over, again and again. Washington? I can't even begin to talk about the deals and policies in place that affect the bottom-line reality of the available jobs, because I don't really know; I just know that that's a front in this war.

Learning from dirty jobs | Mike Rowe


And right here, guys -- Silicon Valley. I mean -- how many people have an iPhone on them right now? How many people have their BlackBerry? We're plugged in; we're connected. I would never suggest for a second that something bad has come out of the tech revolution. Good grief, not to this crowd. (Laughter) But I would suggest that innovation without imitation is a complete waste of time. And nobody celebrates imitation the way "Dirty Jobs" guys know it has to be done. Your iPhone without those people making the same interface,

Learning from dirty jobs | Mike Rowe


the same circuitry, the same board, over and over -- all of that -- that's what makes it equally as possible as the genius that goes inside of it. So, we've got this new toolbox. You know? Our tools today don't look like shovels and picks. They look like the stuff we walk around with. And so the collective effect of all of that has been this marginalization of lots and lots of jobs. And I realized, probably too late in this game -- I hope not, because I don't know if I can do 200 more of these things -- but we're going to do as many as we can.

Learning from dirty jobs | Mike Rowe


And to me, the most important thing to know and to really come face to face with, is that fact that I got it wrong about a lot of things, not just the testicles on my chin. I got a lot wrong. So, we're thinking -- by "we," I mean me -- (Laughter) that the thing to do is to talk about a PR campaign for work -- manual labor, skilled labor. Somebody needs to be out there, talking about the forgotten benefits. I'm talking about grandfather stuff, the stuff a lot us probably grew up with but we've kind of -- you know, kind of lost a little.

Learning from dirty jobs | Mike Rowe


Barack wants to create two and a half million jobs. The infrastructure is a huge deal. This war on work that I suppose exists, has casualties like any other war. The infrastructure is the first one, declining trade school enrollments are the second one. Every single year, fewer electricians, fewer carpenters, fewer plumbers, fewer welders, fewer pipe fitters, fewer steam fitters. The infrastructure jobs that everybody is talking about creating are those guys -- the ones that have been in decline, over and over.

Learning from dirty jobs | Mike Rowe


Meanwhile, we've got two trillion dollars, at a minimum, according to the American Society of Civil Engineers, that we need to expend to even make a dent in the infrastructure, which is currently rated at a D minus. So, if I were running for anything -- and I'm not -- I would simply say that the jobs we hope to make and the jobs we hope to create aren't going to stick unless they're jobs that people want. And I know the point of this conference is to celebrate things that are near and dear to us, but I also know that clean and dirty aren't opposites.

Learning from dirty jobs | Mike Rowe


transformers are quickly coming for your favorite models yesterday they replaced lstms in nlp they used to be good at nlp but blah we now have transformers think again today we're going to see that maybe in the near future transformers will replace convolutions in image processing so this paper is a step in toward towards this direction you just wonder what is it going to be tomorrow maybe linear regression is going to be replaced just by giant transformers trained on 5 000 tpus uh who knows we'll see in any case we're looking at

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (Paper Explained)


axial deep lab stand-alone axial attention for pan-optic segmentation by hui wang yukon chu bradley green heart week adam alan yul and liang chia chen of john hopkins university and google research so this paper combines a bunch of techniques that have been introduced recently uh to deal with attention in problems where you would traditionally use a convolution so in this particular case they deal with this problem of pan-optic segmentation which basically you'll see you'll get an image and there's a bunch of stuff on the


image like a cat here and a house right here and you're supposed to color the pixels of the same object the same so you see you see all these pixels here are house and then all these pixels these pixels right here are cat and so on and then there's also the background so all these pixels right here i know beautiful beautiful beautiful our background so for this problem um it's kind of important that there you you you're very precise first of all so you can look at you know pixels or clusters of pixels and also that you


take long-range dependencies into account because if you for example recognize that this is a house and you recognize that here's a wall right here um you might be able to much better classify what is wall over here and what isn't okay so the kind of long-range dependencies play a role in these problems across images and usually attention mechanisms are pretty good for these long-range dependencies but they're also expensive and that's what this paper deals with so they use this axial attention that has been


introduced for exactly resolving this problem in types of data like images or higher order tensors and they also combine this together with learned positional encodings which we've also seen um time and time again throughout the kind of transformer and attention literature so the combination of axial attention these learn positional embeddings allows them to replace the resnet backbone that usually is found in panoptix segmentation models with the with a standalone attention so they build models that are partial


replace the convolutions with attention modules or replace them entirely so the entire model is going to be just an attention model so no more convolutions in it and they perform pretty well in classic tasks like they they test on imagenet classification they perform pretty well and they achieve state-of-the-art on some of these segmentation tasks so we'll go through the model right here this is a very very extensive paper in terms of experimental evaluation what i want to get into is mainly how the method works


um and show you what their model looks like so we'll go through it and as always let me know what you think in the comments and tell me if you liked it or not uh share it out if you did all right so they go over a very long list of prior work which is you know pretty pretty cool and here they say their contributions so their contributions are four fold first of all the proposed method is the first attempt to build standalone attention models with larger large or a global receptive field and we'll see what that means


we propose position sensitive attention layer that makes better use of positional information without adding much computational cost we show that axial attention works well not only as a standalone model on image classification but also as a backbone on pan-optic segmentation instant segmentation and semantic segmentation maybe what i did before described before was instance or semantic segmentation and not panoptix segmentation excuse me if that's the case as you can see it can be used for various various image tasks


lastly our axial deep lab improves significantly over bottom-up state-of-the-art on cocoa achieving comparable performance of two-stage methods we also surpassed the previous state-of-the-art methods on mapillary vistas and cityscapes so these are various tasks as i said and also what they don't mention here is that they perform fairly well on imagenet in fact in the abstract they formulate this as um in particular our model outperforms all existing standalone self attention models on imagenet like that's you know that's a way to phrase it uh


you just exclude all of the other models until you're the best outperforms all existing standalone self-attention models on imagenet yeah i mean that's good i i mean there's something to be said of comparing apples to apples but you can also you can also go overboard if you want to make your work look as good as possible of course you know everyone everyone does that and there's no particular shame in it okay so if a we're going to build up our model right here and the basic element of this um model is going to be this self-attention


mechanism now quickly because i know you all know what it is but very quickly you want to perform this action right here over a region right here so there is always a query and now the subscripts here are going to be important in this paper okay so the query is at a given position position o and you can see that's the o right here that's the i'm going to call it the output i guess that's what they said as well so the output position you want to go over all of the input positions and you want to aggregate data from all


of the input positions so that's right here and how do you aggregate data by this softmax operator right here and you can see the key also has a p right here and the softmax is over the axis of p so in particular case of the images what does that mean if you have an image right here it's made into pixels okay so you have pixels now a transformer or gen in generally these attention models what you can imagine is they always transform a data point into a data point of the same dimensions now this doesn't have to be


actually and i think one of the developments that is going to come in coming years or months or weeks maybe someone's already doing it is in fact to play more with this with this arbitrary constraint that we're imposing on ourselves because it's not really clear that this is the best thing but for now an attention layer is always transforming a data point here a four by four image into a data point of the same size also a four by four image right here now this is as i said this is quite simplified but it is true


in nlp where we always transform our whatever 512 sequence token sequence into a 512 token sequence and it is true here now the output is is going to be here on the right and the question always is okay so i'll go over these um these these pixels right here and for every pixel let's say for this pixel i'm going to ask what data goes there what's the output of the layer at that particular pixel and the output of the layer is going to be somehow dependent on on the input right here now if you know classic convolutional


models what the classic convolutional model says the output of this is going to be dependent on this region right here if it's like a three by three filter okay so you have this convolutional filter and that means that blue dot on the right is going to pay attention to you know its own location in the input plus everything around it okay and then every single uh data point here is going to do that so for example this green data point is going to pay attention to this region right here now there's a border um so there's maybe some padding


but the question is always where does the information come from and how is it aggregated okay in a convolutional layer what happens in a convolution layer in a convolution layer you simply have your filter right you have your filter and the filter has numbers in it like three and five and eight and so on and what you're going to do is you're going to take this region right here this blue region of the lower layer and that's maybe that's also you know filled with numbers like seven what's a good number zero zero is a


purple is a good nice number and you're going to multiply those and then you're going to sum them up and then you're going to put that on where the blue dot is okay so where does the information come from in the convolution from around the location from around the output location but in the input okay so you go to the input at the same location as where you want the output to be you take the neighborhood and there is a fixed a fixed scheme of aggregating the neighborhood okay and then you sum you multiply and you sum across it in


contrast to this in a uh fully attentional model where does the information come from let's again look at the blue dot and let's consider it fully attentional okay where does the information come from everywhere anywhere anywhere at all okay the information comes from everywhere now how how do i know um how to aggregate the information so it's no longer in a neighborhood how do i know how to aggregate the information that's also different so two things are different um now in a convolution i would have another four by four


grid here that's pre-specified but in the attention model this here is basically all filled with question marks question mark question mark where how what what number goes here how do i in the end i also do this multiply and i sum it up and i put it right here okay but how do these numbers come to be well these numbers also come these are dynamically computed also from um from the input it's a bit special but this is how attention works okay so every pixel gets to decide where information comes from and how it is aggregated it basically it


comes from anywhere and how it is aggregated is dynamic depending on the pixel if you don't still don't understand it maybe pay out to watch a video on attention itself i happen to have made one but you can watch any one when you understand that you will understand the um the extension here to the image is the exact same thing as with the sequence except uh the pixels are basically one long sequence in the image okay so this would be a fully attentional model down here now what's the problem here the problem is


that pictures are pretty large so even even something like mnist which is like 28 by 28 is like 700 pixels plus i don't remember exactly but it's like about 700 pixels and our big transformers now so bert a very famous transformer takes inputs uh that are like 512 in length and you already need pretty decent hardware to run this and the requirements on memory and compute scale quadratically with the input length so already with mnist you're in pretty pretty shady territory um if you go up to something like imagenet which is like


225 by 225 you're that's bad right that's not good um so you have to come up with something else so people have been playing around the reason why i introduced it this way is people have been playing around a bit with sort of coming up with an intermediate with a compromise between the two so the compromise that this paper here focuses on is going to be is going to be a compromise where we you remember when i said where does the information for a given pixel come from and we said okay it can come from anywhere


in the attention framework and that's good because that allows us to make super long-range connections so any pixel can aggregate information from any other pixel and not even in a fixed way but in a dynamic way so depending on the pixel value itself and the other values it can it decide how it wants to aggregate information that turns out to be expensive right every pixel together with every pixel well that's quadratic okay so what do we do we make a third method that's going to be a compromise and the compromise is going to be the


following the compromise is going to be all right we still do the dynamic aggregation which means that we still do the attention thing however however we're going to restrict back to this neighborhood region of the convolution so in this model where this information for the blue dot come from it again comes from this neighborhood right here and this number the size here is going to be called m so it still comes from that m by m neighborhood so a pixel can only aggregate information from its neighbors but contrary to a convolution


how it aggregates the information like this what in convolution would be a kernel the kernel is made dynamically by the attention module and it's made dynamically on a case-by-case basis okay so we restrict it to a neighborhood multiply sum it up and then put it into the output and we do that for every pixel now it resembles much more a convolution simply a convolution with this dynamic with this dynamic matrix right here and that's the starting point for this paper so this paper does two things to this it says okay


um we can augment this by so-called positional embeddings uh a positional embedding you might know from the sequence transformers so if i have a sequence my cat is tall i don't even know what that means for a cat but okay what in a positional encoding so if you use a transformer and you transform this as we said into a sequence of equal length and then transform is basically information routing the transformer simply sees the lower layer sequence as a set not as a sequence it has no notion of what's neighboring to


what what comes from where so it pays to tell the transformer by the way this is word one this is word two this is word three this is word four there are various ways to do it um transformers usually have fairly complicated kind of sine wave based positional encodings that bring many advantages with them um in this case they say well it might pay a payoff to learn where actually these things are in this neighborhood so they experiment with relative positional encoding which means they annotate this neighborhood with


something like look here in the middle it's a zero zero here it's like zero one here it's zero negative one negative one zero and so on so they annotate it with uh these positional encodings now this is this would be the easy way what they actually do is they simply they give the model a matrix like this and they learn that matrix by heart let's say um so the positional encodings are relative positional encodings and they are learned okay so you can do that you can learn position on coding so if you don't want to do the one two three


four right here you simply say well here is a vector here is a vector here is a vector and here is also a vector now model you're already learning like all the ways to make this thing here happen and you're already learning your output weights up here right using back propagation why don't you learn yourself what you would like for position one like what kind of information you would like to be to have there using back propagation right so the model you provide them you always provide the same vector so this is the same vector


for position one and you have a different vector for position two and you have a different vector for position three right so um but across all of the data points these vectors are going to be the same so the vector one is always going to be that same vector for all of the data points so the model somehow must learn independent of the data point what it means to be in position one so the model must learn how it wants to fill that vector that's called a learned positional embeddings we've seen this in many models so far it usually works


pretty well and i guess here it works especially well if you have these relative positional encodings and so this thing here is not going to be an actual matrix filled with these numbers it's going to be a learned matrix a trainable matrix that is filled that the network is allowed to fill with numbers right like three five eight and you might be you might notice that we've seen this before right um so ultimately the information in this blue thing right here is going to depend on this dynamically created aggregating of


information through the neighborhood and this statically learned aggregation of information throughout the neighborhood which is a con which is sort of a convolution right um because in the convolution you've already seen here this is a statically learned map of how to aggregate information from the neighborhood of a pixel so i think even though there are slight differences um they for example say this these are the same across attention heads and so on um however i suspect that you you can think of these learned positional embeddings


to be um to be kind of like what you learn in a convolution not exactly so um no i i think i made a mistake and we'll see it in the formula we'll see it in the formula yeah okay so here they introduce these positional embeddings okay so you see that we previously we had the soft max previously we had this and this okay so this is the lower layer this is the information that comes into the layer and now it's it's transformed into values by a linear matrix but essentially this is the lower layer and for each of


the output locations you want to know how should i aggregate information from that lower layer and you do this by this thing here this thing here is this dynamically constructed attention matrix using also the softmax okay so how should you aggregate information this comes from this query at the output position and the keys at the input position and now you add to that this method this thing right here which is again an inner product between the carrier query and the positional encodings okay so the positional encodings are going to


be learned and hard coded but they still are modified by the queries so the query can still pay attention the difference is the keys depend on the input while the positional encoding does not depend on the input so the queries can decide i want to gather information from this and this and this type of information so that would be the key or it can decide i would like very much to look at pixels that are somehow on the bottom right of the pixel that i am now that would be the um positional encodings and that's that's the mistake


i made when i said it's equivalent to a convolution it is not because the query can still it's still modulated by that query vector um of how to aggregate information otherwise you would have this to be a standalone multiplied by the input right here but it sort of pays off to think of it like what you do in the convolution so in the convolution you learn how to aggregate information basically based on on position um relative position to the position that you want to output and here you do a similar thing you


learn static position embeddings that you then can attend to with your queries all right so these are the position embeddings and they make use of those position embeddings in fact they attend them to the following in this work we enable the output to retrieve relative positions beside the content based on query key affinities formally so the problem up here is that okay you have these position embeddings um and here are the outputs but if you do this in multiple layers right if you do let's let's go with 1d


sequences if you do this in multiple layers and here you annotate the position let's just go one two three four um and okay this layer can make use of that right we gather stuff from here but then when this layer when this layer gathers information from here the where the information comes from in the layer below is some is how somehow getting lost right so it cannot kind of pull through this information to here or at least it's very complicated this model extends this positional embeddings in order to pull through that


information so as you can see there are two new things right here the biggest important new thing is that right here we don't so here is how we aggregate information okay and here is the information that we aggregate over now you can see previously this was just this value vector and now it is extended to the position to positional embeddings learned positional embeddings okay so the this with this you're able to route the positional embeddings to the output and also here you can see the attention gets fairly complex


so you have query key attention which is classic attention the queries can attend to positional codings but also the keys can attend to positional encodings so not only can uh not only can the the node on top say i would like to attend to position three um position three can also say well together with me uh positions two and four are are fairly important i guess that's what that's what that is maybe i'm mistaken here but you can see right here there is an interaction between the keys and the positional encoding right here


now these position encodings they are different for the queries keys and values but um ultimately we don't it doesn't make too much of a difference so here is a contrast between what a traditional attention layer would do and what they would do so a traditional attention layer gets the input x and transforms it by means of these linear transformations right here into the queries these are the queries let's call them q into the keys and into the values okay then it does a matrix multiplication with the keys and the queries and puts


that through a softmax so this here is going to be our attention matrix this is the attention matrix and the attention matrix is multiplied here by the values and that determines our output okay again the attention matrix defines how we aggregate information and the values is what information do we aggregate you know for the output in contrast when we introduce these positional encodings you can see right here again we have query key and value now it gets a little bit more more more complex right here namely we do this


query key multiplication right here but we also multiply the query by these uh positional embeddings for q we also multiply the keys by the positional embeddings for k and all of this together so this is a big plus right here all of this together is routed through the softmax okay and now the diagram is a little bit complicated uh now you can see the softmax aggregates information from here and from this learn position embeddings i would rather have they would just use it like they did in the formula uh do v plus r and


say that's going to be the information that we are aggregating and the soft max here the output of the softmax is going to be how we aggregate information this is the attention all right i hope that's sort of clear you introduce these positional embeddings for queries keys and values and that allows the model to have a sense of where the information is coming from basically what positions which if you drop the convolutions so the convolution had this intrinsically because in your convolutional kernel right uh can i


i'm i'm dumb if in your convolutional kernel the number right here if there was a seven right here that meant that wherever you are whatever is on the bottom right is seven important okay so that's that was the the convolution had this intrinsically here if you just do attention we as humans we see it in a in this kind of grid form but the machine doesn't the machine simply sees a set of pixels it simply sees you can this is to the attention mechanism this is exactly the same as a long list of pixels or a discontinued set


it doesn't matter to the machine so it's like the problems a feed forward network has so we need to annotate it we have to give it positional information and learned positional information seems to work very well right here though you could think of static positional information okay this is the first thing the positional embeddings um that now help the attention mechanism see where the information is coming from that's really important in pictures uh so we add that the second thing they do is this so-called axial


attention now axial attention is a sort of a let's say a trick in order to reduce the [Music] load on a the load on an attention mechanism so what does it mean we've already we've already seen in sequences right if i have a sequence a sequence layer that's going to be n squared connections between the two now there are various ways to restrict that so instead of having all of these connections let's say from onenote we've already seen wait if we just restrict it to let's say only this thing right here only this


stuff that can be that is lower right that is lower in complexity and this in this case it would be just a neighborhood so that's what we've done that's this this m thing right here however we can also do it in different ways since this is a set anyway we can simply say uh maybe we should just always skip one we could like do attention like this and that would be just fine too right that would also leave away some of the information but you gain in computational efficiency there are various trade-offs now in a picture you have


the same options right so you can do the neighborhood thing as we did or you can say where should the green pixel pay attention to axial attention says the green pixel should pay attention to only the row where it is in okay that's it should ignore the rest of the input it should only pay attention to that row where it is in and then in the next layer we'll flip it then the green pixel the same green pixel will pay attention to only the column it is in okay so that's that's called axial attention but don't think like don't don't


there is nothing special about this being an axis or whatnot you could also define and it would not be called axial attention but you could define it it makes the same sense to say well that green pixel just depends on this diagonal right here just in the in this layer it just does this diagonal and then in the next layer it does like the anti-diagonal um you can say uh i just choose five random pixels in this layer and five random pixels in the next layer and that would work as well we've already seen this in this paper called


big bird right the big big big bird but big bird so big bird explicitly used random connections in the attention mechanism and their argument was well if we use different random connections in each layer then information can travel pretty fast through the network so what's the problem with these neighborhoods right here what's the problem with neighborhood attention like this the problem is that you break the long range dependencies so let's see what happens if information needs to go from this pixel or to this pixel or this node to this


node but if information needs to travel from this node to this node in a classic attention mechanism everything's connected to everything so that node in the next layer can simply aggregate information from here well that's not possible if you do this kind of neighborhood attention as we've done here if i do neighborhood attention then at most right because the neighborhood is three long at most this node right here can aggregate information from this node and then again it's three long in the next step so now


this node can aggregate information from this node okay because the in the neighborhood is three long and you can only attend to within your neighborhood this means that if i want to send information to something that's really far away i need to um i need to go many many layers right i need to go layer layer layer layer and this has been well known this has already been a like a problem this has already been a property of convolutional neural networks so convolutions specifically traded off the fully connectedness of fully connected layers


two local connections convolutions but that means that you have to go very deep in order to make long range connections you can't just make them in one step the same problem right here now this paper big bird argued that if you have random connections instead of neighborhood connections just the property of random graphs mean that um you you are pretty fast in sending information around so because in a random graph of size n you on average all two nodes are connected by path lengths of log n this is much faster


because in this neighborhood thing two nodes are connected in a path length of order of n right you can you can pretty easily see that if i make the sequence longer i need that many more steps in order to send it around in fact it's like something like n divided by m this neighborhood size in a random graph it's log n and in this axial attention that's why i introduced it it's two okay every uh every two nodes are connected by two steps if if node if this node right here needs to send information to this node right here


in a classic attention mechanism you could do some one step because every pixel attends to every other pixel however right now we have to um we have to see so this node attends in this layer sorry i have to think so how do we send information between the two we select this node right here in the first layer this node pays attention to this row okay which includes the red dot so the red dot can send information to the x in this layer in the next layer we select this node right here which is our target node where the information


should go to it pays attention to all of this column which includes that x that before right this this x right here where we send information to so it takes two layers two steps to send information from any node to any other node that's pretty good so this um axial attention if you stack them on top of each other you sacrifice a little bit of uh of being able to send information from anywhere to anywhere for the pleasure of not having this quadratic attention anymore as you can see your attention mechanism is now as long


or as big as your column or is wide or your row is high again this isn't this isn't specific to rows or columns you could do this as i said with these kind of uh diagonals you could do it with any other sort of sub pattern where you can sort of guarantee that the overlap between the layers is enough so you can send information around pretty efficiently and they use this right here so this axial attention you can see the formula is exactly the same the only change from before is this part right here you can see that


the neighborhood that they aggregate over is no longer m by m it is now 1 by m so we've seen them going from if this is the the full input image and you wanna you wanna see where to attend what this paper does is it says a classic sorry a convolutional neural network would be attending to some sub part right this is convolution an attention mechanism pure attention would attend to everything right this is attention then what we are doing sorry that was a mistake what other people were doing were reverting back this attention


um to a subpart this kind of neighborhood attention okay but that was still you know you still have m squared you still have o of m squared because of the attention mechanism now what we are doing is we are going even lower we're actually going one by m okay this this is with with axial attention so in general it's one by m and then in the next layer we can go one by m in this direction and have that property um and because it's so cheap now right because it's now o of m to compute this we might as well make


m as long as the row itself okay so their last step is going to be to say okay we have one by m right here and that's going to be the row itself now you can see right here that they say axial attention reduces the complexity to hwm this enables global receptive field which is achieved by setting the span m directly to the whole input features optionally one could also use a fixed m value in order to reduce memory footprint on huge feature maps which is something that they're going to do later on imagenet i believe so when they