something like this let's say this is an example so the task right here would be to only do right turns now this is not a really nice example we're going to see some nicer examples later on but you can see that now the environment is specified as a vector of rewards and you can create the specific tasks like turning right simply by adjusting how you mix these different things by this vector w and this is going to be the key ingredient here so they discuss your general reinforcement learning the reinforcement learning lingo and i


think we've gone through this a number of times just very very quickly uh in reinforcement learning you're given these transitions you are in a state you take an action and that leads you to get a reward or prime and you get into a state s prime in the next state they say the reward is given by the reward function so the reward is purely a function of where you are and what you do and where you get to now most reinforcement learning problems you can actually kind of forget about this part right here because well it isn't it is kind of

Fast reinforcement learning with generalized policy updates (Paper Explained)


important but you could um most reinforcement learning problems the reward is simply a matter of where you are and what you do and this can be a random variable there can be randomness but maybe it's easier if you for now think about the reward simply as a function of these two things so what you want to discover is a policy pi where you input you input where you are and the output is going to be what should you do in that situation okay uh that is a policy and associated with each policy is this thing called a q function


so you can see right here the q function of a policy um is going to be a function of where you are and what you do and this is a bit confusing but it basically means that you are in state s so you are here and you have let's say three options action one action two action three to do now the q function tells you the q function this is s and the a's are the numbers okay so let's say we plug in the state s and for a we plug in number two what it will tell you is if i am in state s and i perform action number two


then how valuable is that for me and value is defined by all the reward that i'm going to pick up from now until the end of time or the end of the episode it depends um but let's say until the end of time well how much how much reward am i going to pick up from now until the end of time is a bit of a vague not a vague question but a difficult question i can tell you how much i could estimate how much reward i'm going to pick up in the next step because i know what action i'm doing i'm performing action number


two but what happens after that who knows so that's where this policy right here comes in this policy right here says so the full definition of the q function is if i'm in state s and i perform action a right now and after that i follow policy pi what is my reward going to be well now it's well defined so right now you do action a and after that you do whatever action the policy tells you in that specific situation okay so that's the q function and you can pretty easily see that if you have a q function right if


you have an accurate q function you can get a good policy by simply always going with the action that gives you the highest q value because um it's because of a recurrence relationship called the the bellman equation uh this thing right here so your q function basically decomposes into the reward in the next step as we said plus whatever happens after that and whatever happens after that is just by the nature of how the things are defined is going to be the q function of whatever the policy is telling you so you can get a pretty good policy by


always doing whatever action your q function tells you is best this step of calculating the q function is called a policy evaluation and this paper here is going to generalize these notions um sorry so this is a policy evaluation and then the act of selecting an action is going to be a policy improvement these are just names okay but we need to know them because the paper introduces two new things i'm going to where do i highlight policy evaluation i don't know but here they say this is the policy improvement


okay ah here policy evaluation policy improvement these are the two steps so the first step is calculate the queue function the second step is to select an action and you can see how these things interlock namely we can calculate the q function of a given policy and we can improve that policy by selecting whatever action is best for the q function this paper generalizes this and you can see that there is a little a little r right here so the r is just a specific way to reference the reward function used right here okay and


you can see it here as well now usually we have one policy and one reward right and so what we do is we improve the policy and that leads us to better evaluate the q function for a given reward function and that leads us to improve the policy now this paper is going to transform this into the following we have many policies so we have policy one policy two and so on until policy i don't know p and we also have many reward functions reward 1 reward 2 reward 3 and so on until reward let's call that r so we have


many different tasks right here and we have many policies now in essence they don't need to have some anything to do with each other for the theory of this paper but i can simplify this a bit of how they see the world so let's say you have an agent and the agent has been trained on simply that first task right here and has been trained using classic q learning reinforcement learning what not and that results in this particular policy and then the agent just from scratch you restarted again you run reinforcement learning just


on reward number two and obtained policy number two and so on so you do this for all these rewards individually okay so you give the agent a new task and you ask it to learn a policy for that task now you're in a situation where if you are have a new task so are new the question is do you again need to train a new policy and the answer for this paper is no because we have all these policies we don't need to train a new we can simply mix and match these policies that we already know to obtain a good solution for the new


task so how does the paper do it it does it yeah it does it in the following it defines the successor features okay maybe it since maybe it's better if we first go to an example so the example they give here is the following otherwise this i guess this might sound just a bit too abstract okay so you have this world here the agent is the thing here in yellow and it can just move so its actions are move left up right down this this is one step okay in the environment there are two different objects one object is a triangle and one


object is a square okay so um there are a number of tasks we can define right now in this thing so we define tasks according to a reward function so the reward let's say the reward one is going to be um one if if it picks up a square sorry the square and zero else just if it picks up a square on any given step we give it a reward of one it we don't care about the blue triangles okay and then reward two is going to be the opposite it's going to be one not the opposite but one if it picks up a triangle and zero else so you can see the um


good policies right here so pi one is a is a good policy for reward one because it just goes and and collects these red things doesn't care about the blue things just goes and collects them pi two it goes and collects the blue things doesn't care about the red things okay so let's imagine that you have run reinforcement learning twice once for reward one and once for reward two and now you have two policies okay so you have two policies this will lead to pi one this will lead to pi two and now i give you the third task now the third task is a


bit special it's one if you pick up a square and it's um it's zero else except it's negative one if you pick up a blue thing well the order of these is kind of wrong but it just for visual representation okay so now you're asked to um pick up the red things but avoid the blue things okay pick up as many red things as you can avoid the blue things and again as we said the question is do you now have to run reinforcement learning again in this agent with your simulator using like q learning or something like this from


the start or can you come up with a solution just given these two policies that will perform well on the on this new task okay and we're going to see how they do it so what they do is they use successor features so these successor features um i've done a video about successor features and i'll link to that you can look at that but essentially essentially the successor features are defined like this and for that we need to know what this thing is right here they simply call this a feature function okay it's very it's very um


ambiguous term a feature function is a function that takes in a transition so state action next state and maps it to a high dimensional vector note this is almost the same as a reward function except the reward function simply maps it to a number now this is mapped to a higher dimensional thing again i wanna i kind of wanna leave out the next state right here just to make things easier on you so a feature here can be many many things but the structure of the features is going to be such that the reward function is going to be


this feature times this w vector so it was a bit a bit not correct before when i said the reward is now a vector the reward of a particular task w can be seen as the inner product between the features and the task vector so w specifies the task and the features well they specify the features in our case it can be it can be fairly simple namely yes i was i was definitely wrong at the beginning so the feature functions right here is which object do you pick up okay so we define the feature function as 1 0 if you pick up a square and we define


the feature function as 0 1 if you pick up a triangle and now you can and we define it as we define it as 0 0 if you pick up nothing and now you can fairly easily see that the reward of each task can be simply calculated by mixing the features accordingly okay so reward one is going to be simply the feature a 1 0 which is the w vector so i can specify a task by giving the appropriate w vector and now you can see that if this is my reward function my agent can go out into the world if it collects a square it is going to


be rewarded right here if it collects a triangle even though the features indicate that it collected a triangle it doesn't care about it because the w is 0 right here if i now want to give it the new tag the same is true for r2 if and i want to give it the new task r3 right and you remember the reward function right there i can achieve that reward function by simply multiplying the same features the exact same feature functions by this vector right here okay remember there is a slight difference between the reward function and the feature function


in this particular example the idea of the paper is that the feature function can be rich in in expressivity and you know tell you all sorts of things about your current state and the reward function is just a number right and then the the reward is specified by simply linearly mixing these features so the structure imposed by the paper here is that there are such a thing as a feature and any task can be described by mixing these same features okay that's that's the issue right here so the features are going to be constant


across tasks whereas the w defines the task all right so the the goal here is that if you have learned many many things um during your tasks what you want to do is you want to learn this feature representation that is the same across all tasks and then you want to simply have the w specify how to mix these features to get the reward now of course this is a very strict very very definition not not a lot of things will fall into this unless you make the features like exponentially big of course um however they do


discuss whenever a task doesn't fall into that so i hope you're with me so far this is the first kind of restriction we impose on our worlds that we can tackle with this framework namely that all of our worlds have all of our tasks in this world have to be a linear mix of the same features if that's given then our um then we can derive policies for tasks that we have never seen we can derive good policies by doing zero learning simply by specifying the task we can have a good policy for that task from the policies we've already learned


for the other tasks okay so the reward three is now simply this and yeah notice it's not the same as the reward function because the reward function had one if you pick up the square negative one if you pick up the triangle and zero else so the zero we don't have to specify here because it's not part of our features right so you can see that the reward function is given simply by that and we can now as i said derive a good policy for this reward by looking at the other policies even though none of these policies has ever learned to avoid


anything so it makes it defines these successor features right here so the successor features is much like the q function you can see the signature is almost the same so as a q function tells you um how much reward you're going to get if you do the action a and then follow policy pi the successor features almost the same thing however it doesn't tell you what rewards you're going to get it tells you which features you're going to get and which features by that we mean the sum of future features now you can see


this some this a little bit this uh it of course it comes from the fact of the linearity up here so it's not really an additional restriction but simply to clarify what this means for your environment your environment has to be able to be looked at in terms of these features and these features they need to be cumulative again that comes from the fact that it's linear but to see so a feature like i want an an even number of steps or something like this would be terrible uh because and they're going into things


like this later but it would be terrible because here we have the sum and um as soon as you if you have a feature that is very high if you have an even number of steps then um or if you have a feature that counts the steps you will never be able to to do well because if you have a feature that counts the steps it simply counts up and up and up and up depending on how many steps you do and your reward can never be specified in terms of a mix of these features and therefore your successor features are going to be useless


but in our case where it's where feature one is pick up is how many of the sorry after rephrase our feature one is whether or not you pick up a square therefore if we sum it up our successor feature one is going to be the number of this is this is a pound sign the number of squares that you pick up okay similarly our feature too is whether or not you pick up a triangle in a particular step so our successor feature number two is going to be the number of triangles that you pick up over time you can see that the successor features


is kind of the analogous of your q function but it is not in terms of a single number the reward it is going to be in terms of these features which is an entire vector okay and because we've constructed this in a linear way you can also pretty clearly see that the q function is inherently related to the uh to the successor features you can obtain the q function by simply multiplying the successor features by your task vector w now a lot of you might be wondering where does this w come from and in our initial case we're


just going to frame everything as being given right so we're we're given this this w we're we're defining everything from our god-like perspective for now so don't think all of this is learned by now um yeah all right so how can you now derive this this magical new policy okay so we let's say we have this policy one and we have the policy two and they and you have the this features that you've learned constantly over both tasks in fact your it's given right it this pi function we give it we impose it that the feature one is


whether you pick up a red square feature two is whether you pick up a blue square then we know that the reward functions can be achieved by doing the w so this here your w is going to be one zero and your w here is going to be zero one and we now we want a good policy for task three and we know we can achieve this by the one negative one w how can we derive a good policy and this is this algorithm this general policy evaluation a general policy improvement so it assumes that you as we said you have many many different


many different policy so here you can see policy one where's policy two here's policy two and so on it assumes that you have many different features and therefore many different successor features in fact you have a vector of them right so here you can see feature one feature two and so on and it also assumes that you're in a current state and you have many actions at your disposal right now action one action two and so on okay so this is all the past you've already defined your features you have learned these policies


and now you're given a new w w new in our case it's this one negative one and we want the best action so we are in state s we are given this w we want the best action now here is a method where we can simply calculate the best action in terms by by not reinforcement learning at all in this new task so by structuring things like this here so what does it really say here it this thing says we are going to evaluate all of these different cells of this tensor right here so we're going to determine what is the successor feature


number two for policy pi one um in state s if i right now do a2 this is very abstract so let's say you're here and action action two is actually going to the right okay so you're here oh this was yellow it doesn't matter so this is so this is action one this is action two so action two is you go to the right okay you can you can see that this will let you pick up um we'll let you pick up a triangle now here that's action three and so on okay so what's this number going to be so we are in state s as we said and we do action


two so action two is going to pick up a triangle the triangle the picking up of a triangle means that our pi for this step or sorry our five for the step is going to be zero one okay so our successor features this is not the features itself this is the successor features the successor features decompose into the next step plus all the next steps that we can follow okay so all the steps that will come so what are these features going to be is it's going to be the sum over that plus everything that follows and i can take a little bit of a guess


here which means that this number so we only care about feature two right here this feature feature two this number is going to be one for the next step because we are going to pick up a triangle if we do action two but then after that we're going to follow policy one and policy one has been trained to pick up the red squares and not care about triangles so i'm going to guess that every now and then it will kind of step over a triangle but it won't fall we won't you know explicitly go look for them so let's say the episode


was 10 more steps but the board has like 100 squares so and it has like three triangles on it so let's say that's like three-tenths um in expectation okay so this is going to be this is going to be the number that we're looking for we're doing this for every single one of these cells okay this this thing is going to do for every single one of these cells and this is very similar to evaluating q functions except we're evaluating an entire vector right here that's the difference to simply learning many q functions so if you were to evaluate


only a q function then you would only have this first matrix this first block right here okay but you have feature one feature two and so on so you calculate everything in terms of these features and then by linearity you can mix it with that vector so in our case this is going to be the one negative one which will give you the q functions right from what we've seen before you obtain a q function by simply mixing your successor features with your um with this task vector and if you have a q function you can pretty easily determine uh which


action you should take now you have here a q function with respect to every single policy but you can simply take the max right so the max across all of this will determine um will determine so you could take the max across all the policies which will give you the q function for a particular action over all policies that you consider and then you can simply take the arg max of that and determine the action you should take okay so it's a pretty big evaluation but if you do this that means you don't have to do reinforcement learning on


this task it simply determines which action right now is the best given everything that i know from these old policies about the task and that's not going to be like the optimal policy uh per se but it's going to be one policy that's pretty pretty good and you can actually prove some things across that so they do this right here and you can see that here is what q learning does on this new task of picking up the squares and avoiding the triangles q learning takes a while to get there however if you do what they are suggesting


and you know you give the w you can supply the w almost from the beginning you see right here almost from the beginning it is at a high reward now q learning surpasses it eventually but um it's pretty impressive that without doing any learning you are immediately good right now the caveat here of course is that they already need these policy pi 1 and pi 2 given to the algorithm and that comes from previous reinforcement learning trials and they say that they give these trials as many steps as q learning uses so they give them this


these amounts of steps on these other tasks so the comparison here is a bit shaky if you ask me but the point made is that if you have a new task right now you can obtain very good solutions uh and you don't have to do anything okay and these solutions can be the basis for new reinforcement learning right you could start q learning off right here and then get here much faster potentially and so on so the next objective right here is that now we have defined the tasks and we had we know what these features are and we know how


to mix these features as imposers of the task so what happens if we only have the reward function we specify the task only in terms of the reward functions but we're kind of looking at the features and we're like agent please figure out yourself how to to apply these features in order to make the reward high and that's what this thing is right here this gp and gpi with regress w so you don't no longer tell it what the w is um it needs to infer it through reinforcement learning right and it's not really reinforcement


learning but what it does where is it yeah it simply it because all of this is linear and this thing here is given so always remember this thing here is given and these are the rewards that you obtain you can simply do a regression to figure out the w of the task now that's going to take some time but as you can see right here it is going to take um a lot less time than sim than doing q learning from scratch notably because you have good features so this is some this is this gets closer and closer to transfer


learning right if you imagine that this right here is your pre-trained neural network and you simply learn the last layer of it um you you freeze this you do transfer learning fine tune the last layer here we are so um it gets closer and closer and you'll see this trend right here so it's pretty cool what you can do but basically i think it's a lot of math around a framework and the more and more you relax the kind of impositions uh that they need for their framework the more it gets back to simply well we do reinforcement learning at


least in my um estimation so before we look at that this here is a pretty pretty cool experiment where they they look at how the how the different tasks can be achieved if you give different policies so you'll have noticed that we have always given these two two tasks one zero and zero one these were our tasks that we trained on and then one z negative one is task we evaluated on okay and you might object and say wait a minute these these two tasks you know they're pretty good as let's say pre-training tasks


because and it's basically the standard basis right and any other task can be mixed from those so these are orthogonal vectors in this vector space so you're being pretty generous to this system what happens if we're not as generous so that's what they do here so they have different um policies and they evaluate how much you can learn with these different policies so the way you have to read this diagram is right here is going to be the one zero axis as they well they label it right here and this is going to be the zero one axis and


this is evaluation so every direction on this circle defines a task for example this task right here as you can see is going to define the task of picking up both the squares and the triangles right whatever you pick up you get a reward however the task down here is going to be please pick up the squares but avoid the triangles at all cost okay and now they're going to look what happens if we supply different policies to choose from remember we're in this situation we're again in this situation where we give


everything and we give initial policies we give the task vector and now it's about deriving a good policy just from looking at the old policy so no learning as a baseline you have q learning which into a given direction um tells you basically how how long q learning or takes or how far q learning gets with a given amount of steps indicated by this one two three four and so on um yeah you see i think this is this is this in how far q learning gets with these amounts of steps is the dotted lines right here so q learning


gets this far with 10 to the i don't know 4 and then this far 10 to the 5 and so on so these are comparisons you can see that on the outside q learning is going to beat this these methods but our hope is going to be that of course if we have this zero shot generalization it's much better than running q learning for really long if we get close to it so the green thing is what we've already seen policies one and two will give you a fairly you know good um fairly good extent right here so what does it mean it means it can solve


it can solve pretty much everything from here here this task this this task this task it kind of falls off once we go down here so once we go to the avoid section it sort of falls off because it has never learned to avoid now still we can of course do the avoidance by simply imposing a negative collection but negative collecting and avoiding aren't exactly the same thing in these um in these environments right because avoiding can also be going really close to something but not hitting it while collecting it's not the inverse of


collecting the inverse of collecting would be like run away as far as as far as possible so we can expect that we've only ever learned to collect we're not going to be super good at avoiding um then the other extreme is when we give policies three and four and i haven't told you but you can see it right here uh policy three is explicitly to collect one and avoid the other while policy four is the opposite right here avoid the squares collect the triangles and now this policy this policy is should be pretty good on


all of the tasks in between as you can see it has the biggest extent right here and that also makes sense by the way there's nothing down here because the task of avoiding both things doesn't really make sense because you can just stay where you are because there are also these these squares where there's nothing but you can see that the mixture of those is quite potent so already we can see even though these span a bases in fact an orthogonal basis as much as these because of the nature of the features that we define for the


task they are not equivalent in mixing after so we can be more generous we can also be less generous if we only provide policy five and policy five is simply to pick up to pick up both objects then we're going to have a pretty hard time when it comes to avoiding things so you can see it can do fairly well picking up the various things in a positive manner but as soon as we cross this line into the like this horizontal line into where it's about avoiding a particular object um it's not it's not the the choices of actions we have from


policy five aren't going to be super good at that and um they do another they do another thing right here so that the left thing is where they say it's important which policies we provide and the right thing they want to say something like it's important um so they want to say if we provide more policies that can be advantageous because we basically have more options to choose from okay so now they start off with policy four and policy four is simply avoid these squares collect the triangle you can see it performs


fairly well over here where it's all about avoiding the uh squares and collecting the triangles as soon as you get into you know collecting or even here the opposite directions it's pretty bad right that's the red thing and now they add policy two to policy four so policy two is going to be also to collect um the the triangles but to just neglect the squares and that will also do a bit better why does it do better because it's better at collecting uh because this policy here also needs to avoid um and this policy here doesn't care so


in the regimes where it's better to not care than to avoid adding this policy adding these options is going to be good and you can see that there's a general expansion here as we add more policies however i want to point out that for example here this black thing which should be technically superior to the blue thing because it contains as you can see here all the policies that the blue thing contains plus another policy um i don't i don't know if my vision but i'm pretty sure here the black thing is inside the blue


thing uh so that means there can also be a disadvantage to adding more policies right here because maybe you got you have too much to choose from and so right here what we say is we add a policy that is all about collecting the squares and it is performing it is actually decreasing the perform the addition of this is decreasing the performance on tasks where you have to avoid the squares which i'm not sure if if that makes sense again the opposite of collecting isn't avoiding but i'm just pointing this out and this


isn't really mentioned in the paper the paper simply says see we add policies and therefore we are getting better i'm not i don't agree with this given these results or maybe it the plotting the plotting is bad all right so they say okay more policies better which i disagree with they also say ho we can as as much as we can regress the w right we regress w we figure out the task we can even learn the successor features okay we can not the successor features um the pi functions that lead to the succession successor features and you


can see if you do it with the true w you're really good at the beginning if you do it with a regress w we can see that before you can you so this is the small version of this plot right here this is like this uh section i think yeah you know you improve however we can also learn this pi function we can also learn the features where if we're not given the features maybe we can learn the features and they say well we can do this with but also by regression so here what we can do is we can find the function that minimizes the function


and the w along with it that minimizes this error right here okay so you're finding the function and the w that that matches this error and this now really is like learning a neural network i mean you know um so i get i get it you have the i here and the w doesn't depend on the i and so on um but you're getting more and more back to actually simply learning non-linear functions mixing them linearly right here and i think that's going to be kind of the crux of this method uh the fact that the more complicated


your problems are the less you are going to be able to do this kind of stuff and they even go as far as to say well what if like before we the reward is actually something like whether or not you have collected an even number of triangles or squares then they say well you can simply not have a single w but you can find a function w and now the policy is a function of the function of w and you can do potentially the same regression problem but as you can see it gets so now you um this right here is going to be a


function of state and so you can see that it more and more it simply goes back to basically q learning again the only difference here is that you have this intermediate features but i think you can simply view this let's say as a hidden layer in a neural network i get it some are held constant across sums and so on but you know i i like the method in terms of um you know in terms of the analysis so if you are given all this stuff it seems pretty cool that you can derive new policies uh it's implication for lifelong


learning they say look here um you have a bunch of tasks in your database that you've already learned on your agent is going out into the world it faces a new task it can use this thing you can use this thing to obtain a new good policy for that task it can then use reinforcement learning rl to refine that policy and then it can simply save that policy into the database so it keeps expanding and expanding this thing so it keeps adding rows and rows and rows right here of new policies that it's learned over the course of its life so


once it's facing a new task it can just kind of draw from its experience and derive a good initial solution however uh the actual analysis only works i feel in quite limited circumstances and if you want to relax these limited circumstances then you need to basically regress and regress and regress away from away from their setup and i'm not sure i'm not sure where this is going to go if this is going to be a general framework for people it seems like it because it's pretty easy but then also it seems like most of the world doesn't really fall


into this category in fact this divide and conquer approach um i'm not sure but from divide and conquer i almost imagine something like you subdivide and subdivide and subdivide until you know you are at some kind of basic task they still only go for you know single tasks like this here the tasks are somehow in sequence and i'm not i think we should really think about hierarchical rl now this can be a good first step right here but most hierarchical rl even the ones that specify themselves as fully hierarchical like we can do


many layers they rarely go above two layers or three like like one one metal layer and one actual layer like this one right here uh they rarely go further maybe they go two layers but that's about it um i've seen very little in actual hierarchical or dividing conquer reinforcement learning just because it's so hard to train yeah all in all cool paper and if you want to get it into the math a little bit i think it's pretty easy math uh once you kind of set your goals on what it's actually meant to achieve um if you just read


hi there today we're going to look at end to end object detection with transformers by Nicolas carrion Francisco masa and others at Facebook AI research so on a high level this paper does object detection in images using first a CNN and then a transformer to detect objects and it does so via a bipartite matching training objective and this leaves you basically with an architecture that is super super simple compared to the previous architectures that had all kinds of engineering hurdles and thresholds and hyper

DETR: End-to-End Object Detection with Transformers (Paper Explained)


parameters so really excited for this as always if you like content like this consider leaving a like comment or subscribe let's get into it so let's say you have a picture like this here and you're supposed to detect all the objects in it and also where they are and what they are this task is called object detection so a good classifier here would say there's a bird right here and so this is a bird and then this here is also a bird right they can be overlapping these bounding boxes so this is you see the first problem that bird


why is that green nevermind okay and those are the only two objects so there's a number of very difficult things here first of all you need to sort of detect the objects you need to know how many there are it's all it's not always the same in each image there can be multiple objects of the same class there can be multiple objects of different classes they can be anywhere of any size that can be overlapping in the background small or across the entire image they can include each other partially so the problem is a


very very difficult problem and previous work has has done a lot of engineering on this like building detectors and then kind of you want to classify every single pixel here and then you you you get like two detection right here that are very close for the same classes they are that must maybe be the same instance right so there's only one thing here and not things and so on so there there used to be very complicated architectures that solve these problems and this paper here comes up with a super simple architecture and we'll kind of go from


the high level to the low to the implementation of each of the parts so what does this paper propose how do we solve a task like this first of all we put the image and the image here without the labels of course we put it through a convolutional neural network encoder since this is an image task it's you know kind of understandable that we do this mostly because CNN's just works so well for images so this gives us this set of image features and I think this this vector here is not really representative of what's happening so


let's actually take this picture right here and throw it in kind of an angled way and what what we'll do with the CNN is we'll simply sort of scale it down but have it multiple so here it's three channels right it's red green and blue like this three channels but we'll scale it down but we make it more channels so yeah so more channels okay but it's still sort of an image right here it still has the image form okay so the the CNN basically gives us this thing which which is sort of a higher level representation of the image with


many more feature channels but still kind of information where in the image those features are and this is going to be important in a second because now this thing which is this set of image features goes into a transformer encoder decoder and this is sort of the magic thing here as as a component we'll look into that in this in a second but we'll take it out right here is this set of box predictions so outcomes each of these boxes here is going to be consisting of a tuple and the tuple is going to be the class and the bounding


box okay so an example for this could be bird bird at x equals two y equals five okay that that's an example another example of this could also be there is nothing at x equals seven y equals nine okay so nothing the nothing class is a valid class right here and that's also important but safe to say there is this set of box predictions and then that is basically your output right these things are your output if you have those things you can draw these bounding boxes you can assign the labels the question is how do you train it now what you're


given is a database of images and these images as you see here on the right these images already have by human annotators drawn these bounding boxes in and also labels so this here would be annotated with bird and this here would be annotated with bird but it doesn't have any of these like it doesn't annotate the nothing classes or and so on so the question is how do you compare the two can you simply say okay if the first one here is the bird and then and the second one is this bird then it's good but then you know that the ordering


shouldn't matter you simply simply care whether you have the correct bounding boxes you don't care whether you have put them in the correct order and also what if your classifier does something like this it outputs those two boxes we see here but it also outputs this here and says bird or like one that is slightly off and says bird and so on so how do you deal with all of these cases so the way that this paper deals with all of these cases is with their bipartite matching loss this thing right here so how does it work let's say your


where can we go let's say your classifier so here is an image I'll have to wait for this to catch up here is an image and we put it through this entire pipeline and we get a set of predictions right and they're going to be class bounding box class bounding box class bounding box now the first thing you need to know is that there are always the same amount of predictions right there are always this size here is fixed that's large n okay that is sort of that's kind of a maximum of predictions since you can always


predict either a class or the nothing class in in this case you could predict anywhere from zero to five objects in the scene right okay and then the second thing is from your from your database you get out an image with its bounding box annotations right that are made by human laborers let's say these two and you also do class bounding box class bounding box but now you see we only have two two instances so here we just pad with the nothing class so I don't know what the bounding box should be for the nothing class it doesn't really


matter nothing no bounding box nothing no bounding box no bounding box so your ground truth labels if you will are also of size n so you always compare n things here on the left that your classifier output with n things on the right now as we already said the question is how do you deal with you can't simply compare one by one because the the ordering should not be important but also you don't want to encourage your classifier to always kind of if there is if if the one bird is very prominent right you don't want to


encourage your classifier to say do you say that here's a bird here's a bird there's a bird right here hey hey there's a bird there's a bird there's a bird and basically just because the signal for that bird is stronger and basically ignore the other bird what you want to do is you want to encourage some sort of your classifier to detect if it has already detected an object it shouldn't detect again in a slightly different place so what the way you do this is with this bipartite matching loss so at the time when you compute the loss you go here


and you compute what's called a maximum matching now what you have to provide is a loss function so we can there's a loss function L and L will take two of these things L will take the read the predicted thing of your model and L will take the true under one of the true underlying things and L will compute a number and we'll say how well do these two agree so you can say for example if either of them is the nothing class then I have no loss like I don't care about them that gives you no loss but if the two if the two classes agree and the two


bounding boxes agree then it's very good right and we maybe even gives like some negative loss or give loss zero but if if the bounding boxes agree but the classes don't agree then you say that's bad or the other way around if the classes agree in the bottom or even if everything disagrees it's the worst what what you're basically saying is if if these two would correspond to each other right if the thing on the left were the prediction for the thing on right which we don't know right it could be that the thing on the right refers to the bird on


the right and the thing on the Left refers to the bird on the left so would be natural that the bounding boxes aren't the same but you say if these were corresponding to each other what what would the loss be how well would they do and now if you compute this bipartite matching what you want I guess it's a it's a minimum matching in this case what you want is you four to find an assignment of things on the left two things on the right a one to one assignment this is an example of a one to one assignment everything on the left


is assigned exactly one thing on the right such that the total loss is minimized right so you're going to say I'm going to align the things on the left with the things on the right such that it's maximally favorable right I give you the maximum benefit of the doubt by aligning these things and what so in the best possible case what's the loss okay hope this is this is somehow clear so this you're trying to find the assignment from the left to the right that makes that basically is the best case for this output right here where


you really say oh okay here you output the output a bird very close to the bird ear in the ground truth label that's this here so I'm going to connect I'm going to connect these two because that's sort of it's it's it gives the model the most benefit of the doubt and the loss that you have at the end of that matching so this loss here would only then count wherever these connections are that loss is going to be your training loss okay so this solves the problems we had before it is not dependent on the order because if you


reorder the things your minimum matching will simplify it will simply swap with it it is it is um if you output the same bird multiple times only one of these is going to be assigned so if if this here is that bird only one of them only this one maybe is going to be assigned to that one and the other ones can't be assigned to that one are forced to be assigned to a different one let's say this one here and are going to incur a loss so you encourage your model to output let's say diverse bounding boxes different bounding boxes for things okay


so this D solves these problems and it's very clever and there are algorithms to compute these these minimum matchings and they use the Hungarian algorithm which will give you exactly such a matching again this is possible because you have n things on each side and the N is in effect here is a the maximum of objects that you can detect at once I guess if there is less you can simply pad right here and then the model of course is encouraged to come up with the equal number of no class predictions because if it outputs a prediction when it


shouldn't right if it already predicts two things and these are assigned to these two things and then it outputs one more thing it is going to be penalized because it should output three things with no class but it has output one-to-many with a with a class is going to be penalized okay so the this is a pretty pretty cool thing it again it relies on the fact that you have n on both sides but you can make n so large that basically it covers all of the cases so you can make n like 50 so you can detect up to 50 things in a scene


alright that's the algorithm in a high-level they do show their loss here you see the loss ultimately is going to be so it's going to be over this matching right here that's the minimum bipartite assignment that basically minimizes this total loss over your prediction and label matchings and the loss they come up with here I said you have to give the algorithm a loss is this one and they kind of go into how they do it I don't think it's super important so the the class algorithm sorry the loss on the class labels I


think it's going to be a soft Max or a sorry a cross-entropy loss like in usual classification and the loss on the to say whether to bounding boxes agree is a mixture of the l1 loss that compares to bounding boxes and this iou loss which is not dependent on the scale of the bounding boxes it kind of computes how much fraction of the two bounding boxes overlap but in any case the lost base they consist of saying how eyeli how much do the labels agree and how much do the bounding boxes agree okay again this is only possible because after that you


compute this matching otherwise you would have no clue which boxes to come which predictions to compare to which other predictions so let's look at this architecture a bit more in detail as we said you have this what they call the backbone which is a convolutional neural network and with that you put in some positional encodings now I already said the you should look at the these features right here as just smaller feature versions of the image but they still have some image nature then they are flattened so once they are put in


the transformer encoder because the transformer is naturally a sequence processing unit okay so it takes in just a sequence of vectors right here and since an image is not a sequence what you'll do is if you have your image features and we said we have a bunch of channels let's say we have four channels and their height and width and see you're going to unroll and flatten that into one sequence so this is height times width you basically unroll across these axes right here into this axis and it's channels I so


basically you have a sequence here of of C dimensional feature vectors that you then put into your encoder okay so your encoder will now transform this sequence into an equally long sequence yet again of features and the good thing about a transformer because why do you use a transformer the good thing about the transformer is that in such a sequence and I've done videos on transformers it you can basic mainly look at the video attention is all you need if you want to under than this more fully this thing can


basically have a tension so it has attention layers it can attend from each position to each position in a one-shot manner so as it transforms this representation up the transformer layers at each step it can basically aggregate information from everywhere in the sequence to anywhere else and therefore it's very it's very powerful if you have a sequence and you need sort of global connections across the sequence this is very good for a language processing because in a sentence let's look at this sentence the input images are matched


together all right applying blah blah blah blah blah blah blah blah blah blah and then there is they write the word they and you need you need to know that they refers to the input images okay and but you see this is very very far away in the sentence so you need a model that makes use of long range dependencies and they make the case that in such a task right here you also need the long range dependencies because these bounding boxes as you see right here there can be quite large so if you have an image you need that this


part here communicates with these and this and this and this part basically anywhere in the bounding box and these bounding boxes can be quite large so the transformer architecture actually makes sense here now I want to go a bit later into why I think it actually makes even more sense for a bounding box detection but right now I just want to keep going through this through this architecture right here so if my computer here decides to come back yes we can go on so what will get out is yet another so in here we put this thing we put down here


we put into the transformer encoder and we get an equally sized equally shaped sequence out of the transformer encoder you see that this thing here goes as a side input into this transformer decoder so the transformer encoder here is just a bit more of a feature mapping technically just for the architecture you could think of just putting this into here but of course it's gonna go better with the transformer encoder the transformer decoder now does something similar but you see it has the encoder as a side input this is very much like


this is not like Burt Burt is like a only encoder transformer whereas this is much like the original attention is all you need transformer that has an encoder and then the decoder as a side input basically as conditioning information has the encoder output what does the decoder do again since it's a transformer it's going to take a sequence and output a sequence the sequence it takes is right here is what they call object queries and this also is different from the attention is all you need papers and they don't do it


autoregressive lee they just do it one shot what does it mean it means that you start with a sequence here of four things and this is these are the this is this big n and you out you output the sequence of a sequence of four things and it's important to see what they're going to end up so these things are then directly going through a classifier that now outputs the so these things here are these class label bounding box outputs okay so each of these things is going to after transformation end up being one of


these bounding boxes either defining an object or saying that there isn't an object somewhere okay you see here this bounding box refers to this bird this bounding box refers to this bird so each of these things is going to to be one bounding box and the what they call object queries is the question of course is what do you input here right actually I want to transform this image information that comes from the left here I want to transform that into the bounding boxes what do I input here and the answer is you just


input at the start you just input n random vectors because what's that gonna give you is basically n outputs you want and outputs because you want n of these bounding box classifications so you need n things and if I input n things into a transformer it's going to give me n things as an output and then in each step I can simply condition on the information that comes in the images and it it'll give me right then I can incorporate that information it's a very deep learning way of thinking about it actually that you just need the


information somewhere in there and I need n things now they go more into detail into this transformer architecture help help in the helpful fashion in the appendix and will go there quickly so this I think here makes more sense so the image features come in here right and you see this is just a transformer stack an encoder stack of multi-head self attention and feed-forward in instants wise or like token wise feed-forward network and then that information is taken and is given as conditioning information over here now in here as I


said you input these object queries which at the beginning are just n random vectors and what you're going to do you Argos are going to feature and code them and then you combine it with this image information so ultimately if you think of this one of these things one of these things is going to be a vector right and then that vector is going to be transformed and then it will have as it is transformed it will have the opportunity to basically look at features that come from here now the arrow is in the wrong direction so you


have already taken the image and you've transformed it into a feature representation which is also a vector right you have the features of the image right here now as you transform this vector this object query queue you have the opportunity to look at the image features right and that's how do you get the image information in there so the image features will come in here transform that through attention so this is an attention mechanism on the image and then what you will output is a bounding box and a little class label it's really


hard to explain I would guess you need to understand really what attention mechanisms are and of course the crucial part of of course is what what's this what do you input at the beginning and these object queries aren't actually random as I said they are learned so what you're going to do is you're going to learn independent of the input image you're going to learn n different object queries and these object queries now it's very it's very interesting because these object queries are sort of going to be different it's like you have


different people that can ask the input image different questions right and this they have so there n is 100 but they show 20 of these object queries that they learn and so did they have visualization of all bounding box predictions on all images so it's it's sort of like you have n different people at your disposal and you train these n different people to kind of ask different questions of the input image ok you say this person up here will always ask irrespective of what the input image is will always ask sort of


hey input image what's what's on your bottom left right that's I'm really interested what's on your bottom left and sometimes I'm a bit interested in what's here but I'm mainly interested what's on the bottom left of the image whereas this person right here sorry this person right here is more interested in what's in the center that the different colors here is refer to different sizes of bounding boxes so this person is also interested so the person on the top-left is interested mainly in I think small bounding boxes that are on the bottom


left and the person here is mostly interested in what I'm really interested what's in the center what's large in the center I want give me large things that are in the center right and then this person right here is really interested on stuff that's on the right side of the image so you see in order to get different sort of a difference in bounding box predictions you train n different people to ask different questions of the of the input image and this asking of questions is exactly what an attention mechanism is so this person


right here let's let's take this this person and I'm saying person these are vectors these are learned object queries but this person first they will simply ask the question what's on what's on the right side and then the image features right I'm getting poor drawing the image features it will have an attention mechanism to this part of the image features and then it will get back some signal right and then it will transform that with its own signal up and then it will ask maybe again okay now that I know more because you see that person is


interested in multiple things it's interested in those things and those things so at first it will focus on these things but then it says oh now I'm now I know more right there is there I know I see there is actually something on the right side so in the higher layers it can then go back and ask the image more questions by sending these cue vectors of the attention mechanism and it will get back the V vectors from the image features that correspond to these cue things so up and up the layers this person can ask more refined


questions about what that particular person is interested in okay and since you have the different people here that ask different questions you basically learn the people in a way such that across the data set they all together they cover every possible image pretty well again these people what they're interested in initially is not dependent on the picture you simply learn this in a global manner all right this is the best way I have of describing it you basically learn n people that are each one is interested in different things


different classes and different regions in the image and each one of these people is going to output their best guess of what is where based on what they're interested in so that person might say I'm you know I'm the person that's interested kind of in the left side of things so I am going to output that there is a bird right here now these people if this is a transformer right and everything can attend to everything they can actually communicate with each other as they incorporate information from the image so in each


layer they can do both they can incorporate information from the image and they can communicate with each other and then in the next layer that can do it again and again and again and thereby they can sort of they can sort of say well you already got the left side I will take the right side you already got the bird class I will take the elephant class and so on so you see here how the the architecture of the transformer actually is also very conducive to doing this bounding box prediction in that these different things can sort of


attend to each other and therefore communicate with each other all right I hope that sort of makes sense now before we get into the experiments I want to list a third reason of why the transformer especially the encoders might actually also make a giant amount of sense here since you on the image into height and width and you have to imagine what does the transformer do the transformer as we said here has this notion of a tension where from any point in the sequence it can gather information from any other point in the sequence and this that's


usually one of the downsides of the Transformers is done via a quadratic attention mechanism so if I just list one feature channel go over here if I just list one feature Channel right here this is height times width of the image right this is this is the entire image unrolled in one vector height times width and here I unroll it again height times width then this this matrix that I can build right here which is called the attention matrix right here it will tell me which parts of the sequence attends to which other parts okay so if you have


an image that has the let's say the number three and you really want to figure out whether or not this is a three then the bow up here must communicate with the bow down here right they need to share information you say oh there's a bow here there's a bow here and there is a spiky thing here that must be a three so you want something this is rather at the beginning of the sequence you want this to attend first of all it will attend itself so you get fairly high values along the diagonal maybe 1010 1011 1112 and I saw this all


eg skated a hundred million nine nine but it also like this this part here at the beginning of the sequence let's say it's here because this is unrolled right needs to attend to the end so this needs to attend to the end which we will put an 11 here and the other way around doesn't always need to be symmetrical by the way okay but in any case this is going to be a H times W squared matrix because everything can attend to everything and that's the attention mechanism why do I think this is so good for bounding boxes because let's let's


imagine you actually have a matrix that is like this okay height times width times height times width every single point in here actually defines a bounding box because this point this point right here in this dimension corresponds to one location in the image and on this axis it corresponds to another location now in the attention matrix simply means these two points need to communicate but if you have two pixels you actually have defined a bounding box right here right you you were actually you're defining a bounding


box and the fact that this is happening in the exact same matrices could mean that the Transformers are uniquely well the Transformers across sequences of these height times with unrolled images are uniquely well conducive to these bounding box prediction tasks I'm actually a bit astounded because when I first just read the title this immediately popped to my mind I'm like oh yes of course and they're going to predict the bounding boxes by simply training so what you would do what I thought this was gonna be as out you


output an actual matrix like this and then you simply each point you can you can simply classify right so you can classify here whether whether or not like at in this direction there is a bird right and then if you have two points like this for example you and you also classify whether in this direction there is a bird right and this naturally defines a bounding box or you could like take this matrix and actually just classify individual points in this matrix to be the bounding boxes because they already define bounding boxes so I


just I think these these quadratic things are are uniquely I mean someone must have thought of this or if not like the YouTube channel it would be funny first paper ever to actually have to cite the YouTube channel but again yeah so transformers seem to be a good idea for these kinds of things so how do they do of course they do well they are on par where with these other much much much more complex architectures these faster our CNN models they are apparently much more complex but they are on par with this they do however train forever


I think they train for like six days on eight GPUs is not that much if you compare to like language models on hundreds of TP use but still okay I don't want to go into the numbers of experiments but what is really cool is that they can now visualize this sort of attention and you can see right here that if they look at a particular point in the image and visualize the attention it will actually attend to the instance itself so it will like these are usually the problems for these detection algorithms when things overlap and are


partially occluded but you can see right here that the attention is on the part of the image that makes the instance in the back and the attention here is on the part of this and it doesn't sort of overlap into the others so that is one thing that's pretty impressive about these architectures the other thing they show is for example it can generalize to many many instances so here it has never seen 24 giraffes in one image but yet it can absolutely do that and giraffe giraffe to rupture after of and the one of the coolest images I find are these


here where you can see right here again attention visualisation and you see that even within the bounding box of the front elephant here you see that the attention on this foot of the back elephant is is is assigned to this blue bounding box so this is the blue basically the blue bounding box person that is attending to that back foot that means they they these things really sort of understand or they learn these things like occlusion and you know just hard I have a hard time describing it but you can see it visually here right like how


it clearly learns that these are two instances that are sort of occluding each other but this this this instance can actually appear within the bounding box of the other instance and the same goes for the zebra here that are partially occluding each other and you can see that the attention is correctly like even here this back foot of this zebra is correctly labeled so all in all that is pretty cool and they take it a step further and they say well with this architecture we can actually pretty easily do pixel wise classification so


this is this cocoa stuff and things data set where I don't know which one is the stuff and which one is the things I think things is the objects and stuff is like sky and mountains and so on and so this is a classification task where you actually have to label every single pixel so what they do is they simply input this through their detector and they detect the instances they take the attention maps of the instances and then they scale it up this right here is just a CNN sort of in Reverse that scales up the image because they have scaled it


down as we said they scale it up again and then they scan simply classify each pixel where each of these you remember we had these different people here that it that cared about different things in the image each of these people will classify their respective pixels the pixels they feel responsible for and then you simply merge all of these people's predictions together into this prediction and again this gives pretty pretty impressive results I am I mean this is this is fun this looks like it sort of works I haven't


they do quantitative analysis of course but I'm just impressed by the examples right here alright that was sort of it I really enjoyed reading this papers the simplicity is pretty cool they do have not only do they have code in the paper to show how ridiculously easy it is to get this to run this is all you need in pi torch but they do actually have code and as I understand they also have pre trained models so they have this model Zoo right here where they give you the pre trained models so you can play with


[Music] hi guys this is Laker from a Judaica the evolution of AI has changed the entire 21st century in terms of technology ai has told in the spotlight and it's advancements are quicker than we predicted with such an exponential growth in AI machine learning is becoming the most training field of the 21st century it is starting to redefine the way we live and it's time we understood what it is and why it matters in this session we'll be discussing the different types of machine learning and we'll compare them to each other so let

Supervised vs Unsupervised vs Reinforcement Learning | Data Science Certification Training | Edureka


me run you through today's agenda we're going to begin the session with an introduction to machine learning next we will discuss the types of machine learning after that we'll compare supervised unsupervised and reinforcement learning based on a few key parameters we'll finally end the session by discussing a few example problems that can be solved using supervised unsupervised and reinforcement learning algorithms so without any further delay let's get started so guys machine learning is the science of getting computers to act by


feeding them data and letting them learn a few tricks on their own without being explicitly programmed now this sounds awfully a lot like a human child so let's consider a small scenario to understand machine learning now as a child if you had to distinguish between fruits such as cherries apples and oranges you wouldn't even know where to start because you're not familiar with how the fruits look now as we grow up we collect more information and start developing the capability to distinguish between various fruits the only reason


why we are able to make this distinction is because we absorb our surroundings we gathered more data and we learn from our past experiences it's because our brain is capable enough to think and make decisions since we have been feeding it a lot of data and this is exactly how machine learning works it involves continuously feeding data to a machine so that it can interpret this data understand the useful insides detect patterns and ident my key features to solve problems this is very similar to how our brain works


now let's move ahead and take a look at the different types of machine learning so first of all we have supervised learning now guys supervised means to oversee or direct a certain activity and make sure it's done correctly in this type of learning the machine learns under guidance so at school or teachers guided us and taught us similarly in supervised learning machines learn by feeding them label data and explicitly telling them hey this is the input and this is exactly how the output must look okay so the


teacher in this case is the training data next we have unsupervised learning unsupervised means to act without anyone's supervision or without anybody's direction now here the data is not labeled there is no guide and the machine has to figure out the data set given and it has to find hidden patterns in order to make predictions about the output an example of unsupervised learning is an adult like you and me we don't need a guide to help us with our daily activities we can figure things out on our own without any supervision


finally we have reinforcement learning now guys reinforcement means to establish or encourage a pattern of behavior let's say that you were dropped off at an isolated island what would you do now initially you'd panic and you'd be unsure of what to do where to get food from how to live and so on but after a while you will have to adapt you must learn how to live in the island adapt to the changing climates learn more to eat and what not to eat so here you're basically following the hit and trial concept because you new to the


surrounding and the only way to learn is experience and then learn from your experience this is what reinforcement learning is it is a learning method wherein an agent which is basically you stuck on the island interacts with its environment which is the island by producing actions and discovers errors or rewards and once the agent gets trained it gets ready to predict the new data presented to it now let's move ahead and look at the differences between supervised answer and reinforcement learning so let's begin by looking at their definitions


now like I mentioned earlier supervised learning is a type of machine learning wherein we teach the machine using label data so an input and your output is label next we have unsupervised learning over here the data provided to the machine is not labeled and the machine has to learn without any supervision so that's why it should discover hidden patterns and trends in the data finally we have reinforcement learning now the basic concept behind reinforcement learning is that there is an agent now this agent is put in an


unknown environment so the agent has to explore the environment by taking actions and transitioning from one state to the other so that he can get maximum rewards now the next parameter to consider is the type of problems that are solved using supervised unsupervised and reinforcement learning so under supervised learning we have two main categories of problems we have regression problems and we have classification problems now guys there is an important difference between classification and regression basically


classification is about predicting a label or a class whereas regression is about predicting a continuous quantity now let's say that you have to classify your emails into two different routes so here basically we'll be labeling our emails as spam and non-spam mails for this kind of problem where we have to assign our input data into different classes we make use of classification algorithms on the other hand regression is used to predict a continuous quantity now a continuous variable is a variable that has infinite number of


possibilities for example a person's weight so someone could be 180 pounds or they could be 180 point 10 pounds or 180 point 1 1 0 pounds now the number of possibilities for weight are limitless and this is exactly what a continuous variable is so regression is a predictive analysis used to predict continuous variables here you don't have to label data in two different classes instead you have to predict a final outcome like let's say that you want to predict the price of a stock over a period for such problems you can make use of


regression algorithms coming to unsupervised learning this type of learning can be used to solve association problems and clustering problems association problems basically involve discovering patterns in data finding co-occurrences and so on a classic example of Association rule mining is a relationship between bread and jam so people who tend to buy bread also tend to buy jam over here it's all about finding associations between items that frequently co-occur or items are similar to each other apart from Association problems unsupervised


learning also deals with clustering and anomaly detection problems clustering is used for cases that involve targeted marketing wherein you are given a list of customers and some information about them and what you have to do is you have to cluster these customers based on their similarity now guys Digital AdWords use a clustering technique to cluster potential buyers into different categories based on their interests and their intent anomaly detection on the other hand is used for tracking unusual activities an example of this is credit


card fraud where in various unsupervised algorithms are used to detect suspicious activities then there is reinforcement learning now this type of learning is comparatively different in reinforcement learning the key difference is that the input itself depends on the actions we take for example in robotics we might start in a situation where the robot does not know anything above the surrounding it is in so after it performs certain actions it finds out more about the world but the world it sees depends on whether it chooses to


move right or whether it shows to move forward or backward in this case the robot is known as the agent and its surrounding is the environment so for each action it takes it can receive a reward or it might receive a punishment now the next parameter is the type of data used to train a machine when it comes to supervised learning it's quite clear and simple the machine will be provided with a label set of input and output data in the training phase itself so basically you feed the output of your algorithm into the system this means


that in supervised learning the machine already knows the output of the algorithm before it starts working on it now an example is classifying a data set into either cats or dogs alright so if the algorithm is fed an image of a cat the image is labeled as a cat similarly for a dog so guys this is how the model is taught it's told that this is a cat by labeling it after the algorithm is taught it is then tested using a new data set but a point to remember here is that in the training phase for a supervised learning algorithm the beta


is labeled alright the input is also labeled and the output is also labeled in unsupervised learning the machine is only given the input data so here we don't tell the system where to go the system has to understand itself from the input data that we give to it so it does this by finding patterns in the data so if we try to classify images into cats and dogs in unsupervised learning the machine will be fed images of cats and dogs and at the end it will form two groups one containing cats and the other containing dogs now the only difference


here is that it won't add labels to the output okay it will just understand how cats look and cluster them into one group and similarly for dogs coming to reinforcement learning there is no predefined data the input depends on the actions taken by the agent now these actions are then recorded in the form of matrices so that it can serve as a memory to the agent so basically as the agent explodes the environment it will collect data which was then being used to get the output so guys in reinforcement learning there is


no predefined data set given to the machine the agent does all the work from scratch the next parameter to consider is training in supervised learning the training phase is well defined and very explicit the machine is fed training data where both the input and output is labeled and the only thing the algorithm has to do is map the input to the output so the training data act like a teacher or a guide over here now once the algorithm is well trained it is tested using the new data when it comes to unsupervised learning the training phase


is big because the machine is only given the input and it has to figure out the output on its own so there's no supervisor here or there's no mentor over here in reinforcement learning there is no predefined data and the whole reinforcement learning process itself is a training and testing phase since there is no predefined data given to the machine it has to learn everything on its own and it starts by exploring and collecting data the next parameter we're going to discuss is the aim of each of these machine learning


types the main aim or the end goal of a supervised learning algorithm is to forecast an outcome now obviously that is the basic aim of all these machine learning types but the whole supervised learning process is built in such a way that it can directly give you a predicted outcome because supervised learning algorithms have a very well-defined training phase unsupervised learning is all about discovering patterns and extracting useful insights now since the algorithm is only fed the input it has to find a way to get to the


output by finding trends and associations in the data coming to reinforcement learning the agent here is a lot like a human child just like how a baby is clueless about the world initially the agent also has no idea about its environment but as it explores the environment it starts learning it learns from the mistakes it makes and it basically learns from its experience now let's look at the approach followed when it comes to supervised learning it's quite simple like I mentioned earlier all that the algorithm has to do is map


the known input to the known output in unsupervised learning the algorithm has to find patterns in data trends in data and keep exploring the data until it reaches the output the approach followed by reinforcement learning is a trial and error method the trial and error method best explains reinforcement learning because the agent has to try out all possible actions to learn about its environment and to get maximum rewards the next parameter is feedback now in supervised learning there is a direct feedback mechanism since the machine is


trained with build input and output for unsupervised learning there is no feedback mechanism because the machine is unaware of the output during the training phase now in reinforcement learning the feedback is in the form of rewards or punishments from the environment so when an agent takes a suitable action it will get a corresponding reward for that action but if the action is wrong then it gets a punishment so rewards and punishments can be thought with respect to a game now in a game when you win a state you


get extra coins but when you fail you have to go back to the same state and try again now let's look at some of the popular algorithms supervised learning has algorithms like linear regression which is mainly used for regression problems it also has algorithms like support vector machines decision trees and so on and these can also be used for classification problems coming to unsupervised learning we have algorithms like key means C means for clustering analysis and algorithms like a priori and Association rule mining to deal with


Association problems now reinforcement learning is just being explored recently a few algorithms include Q learning and the state action reward state action algorithm next up we have applications so guys supervised learning is widely used in the business sector for forecasting risks risk analysis predicting sales profit and so on coming to unsupervised learning so guys the recommendations you see when you shop online like for example if you buy a book on Amazon right you get a list of recommendations now these are all done


by unsupervised learning algorithms other applications include anomaly detection credit card fraud detection and so on now reinforcement learning is used in self-driving cars in building games and all of that one famous example is the alphago game I'm sure all if you have heard of that so guys those were the major differences between supervised unsupervised and reinforcement learning so now let me give you a few examples of problems that can be solved using supervised unsupervised and reinforcement learning algorithms all


right so our first use case is to study a bank credit data set and make a decision about whether to approve the loan of an applicant based on his profile so here we are going to be given a bank credit data set now the information that you see over here is for each of the customers so every customer's account balance purpose credit amount value savings everything is given in the data set and you have to predict whether you can approve the loan of an applicant based on his bank account balance based on his purpose his


credit amount and his savings so for this problem you can make use of the supervised learning algorithm known as key and an algorithm or key in your is neighbor algorithm now let's look at our next use case now here we have to establish a mathematical equation for distance as a function of speed so basically over here you're going to predict the distance that a car can travel based on its speed so guys the best algorithm to use for such a problem is the linear regression algorithm so the linear regression algorithm is


basically used to predict continuous quantities and in this case we have to predict the distance which is a continuous quantity and like I mentioned earlier a linear regression is a type of supervised learning algorithm okay moving on to our next few skills now the problem here is to cluster a set of movies as either good or average based on a social media outreach all right now if you read the problem statement properly you can see the word cluster alright this clearly means that this is a clustering problem and clustering


problems fall under unsupervised learning so here we're going to make use of a algorithm known as k-means algorithm to form two clusters okay one cluster is going to contain popular movies and the other is going to contain non popular movies based on their likes on social media now moving ahead the our next problem statement is to perform Market Basket analysis by finding association between items bought at the grocery store again over here you can see the keyword association this means that this is an association problem now


Association problems fall under the unsupervised learning algorithms and here we can make use of the a priori algorithm to do this so here what you have to do is basically if and find association between different items so if a person bought bread and butter together it means that there is an association between these two items so in this problem you just going to find the association between different items and you're going to make use of the unsupervised learning algorithm cause the a priori algorithm so guys


this is the last use case and over here the problem statement says that you're going to place an agent in any one of the rooms and basically the rooms are represented as 0 1 2 3 4 & 5 and the goal here is to reach the outside of the building now this is clearly a reinforcement learning problem all right to solve this you can make use of the cue learning algorithm and your end goal is to reach room number 5 so guys here you can see that there is no data set because the data set is going to be developed by the agent


itself so guys over here the agent is responsible for collecting the data all right he's going to explore the environment collect useful information and then he's going to use this information to get to room number 5 so guys that was it for our use cases and with this we come to the end of today's video I hope all of you enjoyed it if you have any doubts or any queries regarding the session please leave them in the comment section and we'll get back to you at the earliest so guys thank you so much for watching this


hey guys and welcome to another fun and easy machine learning video on support vector machines so the other day I was walking through the park where I saw a lot of people with their pets dogs as well as cats and then I came across this strange creature just really challenging for me to tell whether it was a dog or get but I eventually figured it out that it was a cat groom like a dog now if it was challenging for me to figure out imagine how difficult and challenging it would be for a computer to precisely classify between a dog and a cat a

Support Vector Machine (SVM) in 7 minutes - Fun Machine Learning


really great algorithm for these types of applications is the support vector machine algorithm or SVM it looks at the extremes of the data sets and draws a decision boundary also known as a hyperplane near the extreme points in the data set so essentially the support vector machine algorithm is a frontier which best segregates the two classes so how does it work to understand SVM Zapatera let's first take a look at why they called support vector machines so say we got some sample data over here of features that classify whether an


observed picture is a dog or cat so we can for example look at the snout length or the ear geometry if we assume that dogs generally have longer snouts and gets have much more pointy ear shapes so how would we decide where to draw our decision boundary well we can draw it over here or here or like this any of these would be fine but what would be the best if you do not have the optimal decision boundary we could incorrectly classify a dog with a cat so if we draw an arbitrary separation line and we use intuition to draw it somewhere between


this data point for the dhoklas and this point for the cat glass these points are also known as support vectors which are defined as data points that the margin pushes up against all points that are close to the opposing glass so the algorithm basically implies that only support vectors are important whereas training examples are ignore an example of this is so that if we have in our case of a dog that loser cat or get that is chrome like a dog we want our classifier to look at the extremes and set our margins based on these support


vectors so we have D plus which is the shortest distance to the closest positive point and D minus which is the shortest distance to the closest negative point and then we have the margin of a separating hyperplane which is the positive plus the negative the line or decision boundary that segregates the two classes is commonly referred to as a hyperplane because s VMs can be used in multi dimensional data sets and the data points are referred to as vectors as they have coordinates within the space of data so


what we discussed so far is also known as linear support vector machines or L SVM because the class are linearly separable but what happens if we have a dataset that is not linear separable so say we are presented with data that looks like this where it looks almost impossible to use a single line to separate the two classes we can use a function to transform our data into high dimensional space so you can see over here we go from one dimensional to two dimensional space we can apply a simple polynomial function to get a parabola and now you


can easily see how we can draw our hyperplane we can do the same for this data set where it's easy to draw the hyperplane or line but for a machine will use a function to transform our data from two-dimensional to three dimensional feature space now the only problem with transformation into higher dimensional feature space is that it's computationally expensive we can use a kernel trick to reduce the computational costs a function that takes as inputs vectors in the original space and returns the dot product of the vectors


in the feature space is called a kernel function also referred to as kernel trick using a kernel function we can apply the dot product between two vectors so that every point is mapped into a high dimensional space via some transformation so essentially we use it to transform a non linear space into a linear space if you look at some popular kernel types here are some popular kernel types that you can use transform our data into high dimensional feature space there are polynomial kernel radial basis function RPF or RBF


kernel sigmoid kernel amongst others unfortunately choosing the correct kernel is a non-trivial task and may depend on specific task at hand no matter which kernel you choose you need to tune the kernel parameters to get good performance from a classifier a popular parameter tuning technique includes k-fold cross-validation you'll deal some of these parameters in our Python labs so the advantages of support vector machines are that they are effective in high dimensional spaces they are so effective in cases where a


number of dimensions is greater than the number of samples they use a subset of training points in the decision function or support vectors so it's also memory efficient suppose factors are first down so different kernels can be specified for the decision function common kernels are provided but it's also possible to specify custom kernels we can add kernel functions together to achieve even more complex hyperplanes the disadvantages however of sports vector machines include if the number of features is greater than the number of samples the


method is likely to give poor performance support vector machines do not directly provide probability estimates these are calculated using an expensive five fold cross validation if you take a look at the applications of support vector machines the support vector machine algorithm has numerous applications and can be a quite popular alternative to artificial neural networks or a ends here are some applications from published journal papers so we can use support vector machines in medical imaging there's one application for SVM based regression


models to study the Equality in urban areas in the city of Oviedo in Spain support vector machines is also used for image interpolation as well as medical classification tasks in the financial industry support vectors are used for time series predictions as well as financial and there's one paper on the application of neural networks mixed with support vector machines in coding theory and practice there's also for pattern recognition for machine volts diagnosis which also uses support vector machines as well as page ranking algorithm and


hi there today we'll look at distributed representations of words and phrases and their compositionality by thomas McAuliffe Ilya sutskever chi-chan Greg Corrado and Jeffrey Dean this is another historical paper it's one of three papers it's the middle one that introduces the original word to Veck algorithm and if you as you might know work to back was extremely influential in NLP since this paper basically until recently where it's sort of gone out of fashion a bit in research with the rise of things like Elmo and Bert but it's

[Classic] Word2Vec: Distributed Representations of Words and Phrases and their Compositionality


still a very very relevant so we'll look at this historical paper today with kind of the hindsight of being a couple years into the future in fact as you see right here this was released in 2013 so it's seven years later now and we'll look back and we'll see what they said back then about the system this is not going to be like a very you know well enhanced PowerPoint presentation of how we're to Veck works this we're going to look at the paper and read it together if if you if you like you know content like this if you like historical paper


readings let me know in the comments share it out if you do like it and of course subscribe because this these kind of historical papers I enjoy them but you know many people might already know what these things are so yeah ok let's you know go through the paper and pick up their ideas and kind of put them in context they say the recently introduced continues skip graham model is an efficient method for learning high-quality distributed vector representations that capture a large number of precise syntactic and semantic


world relationships so the Skip Graham model was already introduced by mclubbe in an earlier paper that came out I believe not like one or two months prior to this one as I said where to veck is a series of papers I don't think there is a paper called were to vector they here have released the code along with the with the paper in the code was called - vac so the Skip Graham model was introduced previously but it is replicated right here so this in the Skip Graham model what you're trying to do is you're trying to get a distributed


word representation so what does that mean that means that for each word in your language let's take these words right here for each word in the language you want to come up with a vector that somehow describes that word in a continuous fashion so that with in a the - like me might be mapped to I don't know 0.1 0.9 and 0.3 Learn might be mapped to negative 0.5 and so on so each word gets assigned a vector in the same dimensional space and what the previous paper kind of discovered is that if you do this correctly then these vectors


they have they have some kind of properties and we can already kind of jump ahead because this was already a bit a bit researched in the last paper the semantics of these vectors will be something like this so here they have a two dimensional PCA so these are the first two dimensions of the one thousand dimensional skip gram vector so the vectors they obtain they can do things like this where they can show that in these spaces for example there appears to be a vector Direction that characterizes the capital of a country


so if you take a few countries and their capitals and you average that vector you get a kind of a direction for capital Ness of a city given a country you can see that there is a a pretty clear relation here now some of these things have later been revised to such that they they are ultimately ended up being not that impressive for example there was always this kind of math with vectors and I don't I believe this is this might not be in this this is in the last paper where they discovered that if you take the vector for King and you


subtract the vector for man and you add the vector for a woman then that would result in the vector for Queen so the way they did it was basically they did this calculation right here and then they searched in the point they ended up they searched for the nearest neighbor in their vocabulary and that turned out to be Queen but in order to make it Queen actually you have to exclude the original word King and people quickly discovered that if you don't exclude the original word it you know the result of this kind of


arithmetic will almost always lead back to the original word and then a lot of these analogy tasks are simply the result of you then discarding that word during the nearest neighbor search and then Queen just happens to be one of the closest words and it's it sort of much less dependent on which exact calculation you do here so there's been a lot of follow-up work kind of analyzing criticizing these vector maths but definitely we know that these word vectors turned out to be extremely extremely helpful and syntactically and