not a ton of features and a ton of input variables and things like that so the last one we're going to talk about is reinforcement learning and this definitely could do with this whole other a whole other video all about reinforcement learning but I just wanted to give a basic introduction to it because I think that's something people are usually trying to figure out what's


the difference between supervisor unsupervised and reinforcement learning but basically this is a form of supervised learning the the main factor here is the system focuses on maximizing a reward signal so the reward signal is supervision that that is a convenient so we have an agent and it's said that into the feature space so all of the input variable options are there and it was it has to

Supervised vs Unsupervised vs Semi / Self Supervised vs Reinforcement Learning | Machine Learning


learn what actions it can take in this feature space in order to maximize the reward signal so one example would be AI chess or any form of AI game playing it's generally done with reinforcement learning so it's if you example if you think about the chess board the the thought I guess that's playing chess it has all of these options of the different moves they can take and the


reward they're trying to maximize is winning the game and that's all that really matters and this is whether they win the game or not so it'll play this game hundreds and hundreds of times until it learns what exact moves in both scenarios so that the environment also plays a role here so you can see in the graph we've got the agent actions environment and so here the agent would


be the ball playing chess the action will be a move that the bot takes the environment is in this case we have somebody else playing chess so the move that they take is the sort of environment that they need to relate to as well as the chess board that results from these actions so when the bot makes a move their pieces move so they need to update their understanding of the moves


that are available based on the move that they've taken and then again by the moves that the other person has taken and then so the state will be the game the board game the layout of the board essentially and the reward again will just be the whether they're winning or not their probability of winning will probably be included in the some way based on the layer of the board


and they learned by trial and error what is the best move to make in what situation and there's also usually a delayed rewards obviously in this case we have that there is the winning of the game and that can be delayed in the sense that some moves that you take will have a knock-on effect on your probability of winning so like if this game is tracking the probability of


winning your next move might not effect that necessarily but you'll give you access to the move after and that could increase by a lot another example is things like a but that has to travel around and collect things obviously every move that it takes won't necessarily collect something but a few moves from there if we continue along the right track they'll pick up something and that will


be a reward so the delayed ward is a big characteristic there and then because that's how it learns how this set of moves are important as not just a single move but like a series of moves and then the trial by error is how it learns and yeah usually there'll be like hundreds and hundreds and thousands of instances of this and then that's how they learn so again reinforcement learning can do


its whole other video I just wanted to give a basic introduction to all of these things so I really hope that that was helpful for you guys I hope you enjoyed this video and if you want to see more videos like this again do be sure to give it a thumbs up onto um comment down below because this is a new style of video on this channel and when we're trying get new things here I need


to know they're working so buying you guys giving the video a thumbs up on commenting below it then that is way for me to understand whether it's working or not and then in future that means I can sort of delve into the individual algorithms that you'll see a machine learning and probably I'll do one week on the actual algorithm and explaining that and then the following week on the


hi there people so a lot of you have asked me how I read papers and honestly I don't think there is any super special method to it but you know I thought because people have asked me to make a video on it so I'll make a video on it and I'll try to share my method of reading papers and hopefully this is going to be somewhat of a miniseries or a series where I every now and then

How I Read a Paper: Facebook's DETR (Video Tutorial)


discuss how I read one of the papers that I make videos about and I'll try to select them such that different things are highlighted now I've selected this one right here for really for no particular reason other than I sort of remembered it and going to try to go with you through how I read this and how I encountered this and kind of try to honestly share what I thought at the


first time when I read it and I hope this helps some of you if it does help you and if you like content like this of course feel free to share this out and subscribe if you haven't seen my original video on this paper it might be worth to go watch it I'll link it and with that let's dive in so again this might be not really something new but I'll just go through it okay so first


thing I do is of course read the title so the title has three parts and to end object detection with transformers so what I notice that I do myself is I like through reading a paper it's like read the paper with an open mind I don't do that I almost immediately form an opinion and a hypothesis of what's going on like so I see transformers so I know what transformers are if you don't I've made


a video if made lots of videos on transformers attention is all you need is the base paper for that so I know what a transformer is okay and I know that transformers are usually in NLP are usually used in NLP door there are things like you know other things with transformers but usually an NLP model then I read object detection and I know object detection is


a computer vision tasks so immediately this here is sort of a a difference and I immediately try to assess what's the new idea in this paper and in this case it might be okay it might be applying transformers to object detection but then I also see end to end and the only reason to put that in a title is because that's the novelty because usually in deep learning we're sort of used to


systems being and to end and even if they aren't if most systems aren't end-to-end a lot of people don't is like end to end image classification on image net like Thanks so I I was guessing that the reason they put in end-to-end into the title was because that's actually something that's special about the model so now I have like two competing hypotheses of why this paper matters


first of all because it does it with transformers and second because it does it end to end and the of course the true fear is that the combination of end to end transformers all of that is what makes this model and I already form like a hypothesis of whether I like this or not like I have to be have to be honest I have very quick judgment of papers of whether I like them or not and then I


sort of catch myself each time and I still try to so they're for most papers actually that I have sort of a negative opinion at the beginning where i will- there are papers where I think like there is no way this is going to you know work or something like this I'm actually positively convinced throughout the paper so for most for most papers where I that I read I'm trying to find


the positive things in there but I do form an opinion pretty quickly usually alright so the second thing this part right here I like I don't even see this is like advertisements on on like Twitter I like you you just I have always had issues with author names like people would come to me and be like oh have you seen the new vineos paper and no clue and then when they say like oh


that's where they use this character level model to do that and I'm like oh that paper so I like do not care who the authors are of a paper like I don't I can't remember the papers by their author names I've gotten better at it I have to say but i-i've always had trouble with this now that's not to say that a name doesn't pop out to me like if this would be like it like yoshua


bengio or some someone like really famous then of course that would catch my eye and but I also know that you know yoshua bengio is paper yoshua bengio slab is huge so just because a big name is on the paper doesn't mean that the paper is going to be of any good or bad quality sometimes the author's give you an indication of what kind of research is there like if


you see a Najaf Clun or Ken Kenneth Oh Stanley you know that there's there's going to be the certain type of of you know learning to explore and and and kind of a bit more out-of-the-box thinking in their papers which I really like but it doesn't immediately give you a clue maybe if you go by first authors it's much more indicative if you have already read some of their papers but


most often I just ignore authors and go on the affiliation sometimes matters in in that it's a bit of a vicious cycle if there's a big name affiliation like Facebook AI Google ai and so on these papers also they get more exposure in like the press and so on so whatever Google a publish that paper all of these all these pops eye magazines like verge and this and life hacker and hacker news


and whatnot they like like write a blurb about it so often they get much more scrutinized for these papers they get much more they get much more the public attention but they also get much more scrutiny which in turn means that there is a bit more pressure on them to do good experiments so that biases meet like a little bit into the direction of believing their experimental evidence


more now usually this is also backed up by the fact that I am actually convinced by their experiments usually so these these big-name papers often I find myself even without or disregarding the affiliation to be convinced more than of like regular papers my most often issue with papers is that I don't believe the experiments and I make no difference like even if


it's Facebook I still my prior is the experiments or crap and I don't believe them and they have to convince me of the opposite but some like I can't say that it doesn't affect me that it's like a big name affiliation okay so then the second thing is I sometimes I see the paper on archive and I skim the abstract sometimes the abstract is informative and sometimes not so here it's like blah


blah blah a new method that views object detection as a direct set prediction problem I'm like oh yeah okay so streamlines the detection effectively removing the need for many hand design components like non maximum suppression yada yada yada the main ingredients called detection transformer asset based global loss that forces unique prediction via bipartite matching and


the transformer encode or decode or architecture so they make it clear here why it matters and that's that's what I what I want to get at is sort of what's the new thing in this paper most papers are even though they're all very long and have lots of math and so on they often have like one or maybe two new core things that they really tell you sometimes zero but a lot of times


it's like one thing that they really do and you you sort of have to but they're trying to cloak it often because they need to make their research as impactful as possible right but you need to sort of figure out what it is they're doing they make it fairly easy for us in that they say okay they remove the need for many hand designed components like non maximum suppression which tells me that


they're building something that's easier than what came before them and that already tells me it it's not necessarily going to be better their argument is more that it's going to be easier right there there are sort of two kinds of experimental results the ones where you try to beat what came before you and the ones where you're trying to say look our thing works just as well as this other


thing while being more advantageous in some other metric so I would place this already in the sort of second category and then they say what are the actual ingredients it's a set based global loss that forces unique predictions via bipartite matching now I at this point I know what these terms mean but at this point actually don't have to know what the terms mean what I need to recognize


is that I simply have to go later and figure out what that is and a transformer based encoder/decoder architecture okay so there are two things right here that I remember I need to pay attention to later there is this loss which seems to be special and there is the transformer architecture which seemed which they say okay that that's the model basically consists of those


two things and then they have a short description of what it does given a fixed small set of learned object queries that your reasons about the relations of the objects and the global image context to directly output the final set of predicted in parallel that almost tells me nothing of yeah okay the model reasons maybe this in parallel is something but the model is conceptually


simple and does not require specialized library unlike many other modern detectors this sort of repeats this enforces my hypothesis that they're going with the hey this is a much easier way of doing things approach detter demonstrates accuracy and runtime performance on par with well established that further confirms my hypothesis that this is on are right they runtime performance on


par with the current state of the art and at the end they say moreover data can easily be generalized to proceed to produce Panoptix segmentation in a unified manner we show that it significantly outperforms competitive baselines training code and praetor models are available as part when I first read it is like ok can easily be generalized to produce this Panoptix


segmentation this is I didn't know yet whether this is like a central claim of their paper that it can do this segmentation or whether this is like an added benefit to their paper because you can read it in both ways and I'm just ready to find this out in the paper now after I've read the abstract and sort of already form the hypothesis of what's going on so here I already in my mind I


already sort of have a model of how would I do that right how would I how would I do that and then what would I do so right now what I might be thinking is if I have a transformer over images that directly outputs the the predictions in parallel I'm imagining like an image and the image somehow needs to go into a transformer so maybe there's like an encoder like a CNN encoder that gives me


image features and then it's it's so maybe you sample this down this image this is just me hypothesizing what could be going on right and then I might be unrolling that right this image into a vector of these lower pixels and then so in my mind what I would do right here if without knowing anything more would be to do something like Bert's pan predictions so I would have Bert right


here and I so for I would input this sequence right here and then to detect an object I would sort of think that maybe the Bert you know Bert has an output that is the same length as the input right so it's it's very good at sequence tagging and things like this so maybe how it detects an object is going to be that it sort of like tags the tags the center location in the pixel of an


object right here or a tag somehow the corners of the of the bounding box but then I don't know how this is going to be in parallel maybe Bert outputs like a score for each location and then you do some kind of matching right here so this is my initial hypothesis of what's going on and then I scroll through and honestly the first thing I do is I go and find the pictures and know no


different in all like since since you first book you read that's what you do I go and find the pictures because usually if someone proposes anything new that they're gonna try to make a picture of it luckily I don't do like super theoretical what not your Bayesian generalization bounds and I don't know so most often papers I read have some sort of picture and that's very helpful


to me I know I know but yeah so I find this picture and here I see okay you have image you have CNN okay gives you a set of image features or so far so good then transform or encoder decoder then set of box predictions so all of them come out here and I already read they're in parallel and then bipartite matching laws so here they I can see they color these in different ways and these color


appear to match with these colors right here right in the green here and these they they also this is a very good graphic but from this I can already read that these here go to the no object a lot of times the graphics aren't very good so this this is where I'm not saying in every paper you can learn by looking at the graphics like sometimes the graphics are terrible and you're


like what's going on here I like I don't this this makes no sense this happens a lot in this paper right here this happens to be very very good explanatory graphics so I'll take advantage of that and I do the same thing in the other papers right but then later when it doesn't match what I read in the text I'll have to you know update my belief and so on but here I see that these go


to no object and this goes to no object so I don't know yet that this is the test set at the point where I read this I was sort of confused by this but I recognized that each of these boxes right here is going to be either resulting in a bounding box or in the no object prediction so from that I could conclude that these things here are maybe some sort of a fixed set right but


I still thought that you know these that this would actually be the output of these image features so that in this case you'd have like six set of image features and then you'd have like Bert here even though that's not an encoder decoder I still this was still my running hypothesis that somehow you'd map these image features to these boxes right here so and I didn't know what to


what to make of this this thing right here so then I went through some more and look for more pictures and there are not sometimes I also kind of glanced at the formulas but okay when I Everest I see this this is just I mean this is kind of useless like okay cool you minimize the loss thanks this okay didn't really pay attention to that ah new picture cool so this picture is much


more informative than the other picture yeah I believe with the other picture they were trying to show case this loss how they do the matching and even though I could read a lot from that picture I did not get that part and that therefore I felt when I saw this and I just glanced at it I'm like wait what what's different then up here it seems like the same but okay let's look at


this so again we see okay you have set of image features that comes out of the CNN so that conforms with my belief but then this here goes into a transformer encoder and this comes out so immediately I see oh this is not the same as these boxes here right that was my hypothesis that these things here would be the colored boxes so I I say okay obviously that's not what happens this


thing here seems to be sort of the encoded image information then that's somehow fed into here and that then there are these object query things and they seem to correspond to this so I'm a bit more confused right now what I can see is that these then will result in these in these boxes okay so being confused by that I look for more pictures so I go look for more pictures


and this here seems to be like of a visualization a lot of these papers have some sort of ablation experiments or so and so on this I just find a really cool picture for now I don't know yet what it means this I don't know yet what it means and I go down skip off this and then back here in the appendix I find this here which I immediately map to the previous where this is the anchor and


this is a decoder and I've already read the attention is all you need paper and at that point it clicked and means like this is not a Burt transformer this is one of these transformers that has an encoder and the decoder even though they told me like 50 billion times already I was too stupid until this point so now I know okay okay I see what's going on so the image goes through here and then


this goes as a side input like as an attention from the decoder to the encoder like I know in NLP right so in NLP this here would be a source sequence like maybe if you do translation and this here would be a target sequence so now whenever I see a transformer like this and it outputs something this I I look at it as okay this here is sort of the input that goes as like a side input


over here and usually here you have the target sequence but that's not the case right here right you have these object queries so this is how far I get from the pictures now I go up so I have a sort of I have questions now I have questions and that's when I start reading the paper only now do I start reading the paper after I've looked through all the images form the


hypothesis and sort of have questions on how this works and we'll go a bit faster from now on - just not bore you with all the things so the introduction is often very important even though it's called introduction and maybe you know if you read a book like if there's like introduction or a prologue or something like this it's often kinda pointless introduction in these research papers is


one of the most important points because all of these papers they try basically all of them try to convince a reviewer to accept them and in order to do that they will set up their main points and their main story immediately in the introduction so what you'll usually have is a problem statement which is here like why what's what's wrong right now and then you have like a story of how


their paper addresses the issue okay and that's that's here we streamline the training pipeline by viewing object prediction yada yada yada this is often formulates in words what the paper is about and what contribution the paper makes right this is like a this is like a longer abstract the abstract is often very very cryptic very dense this here is often much more


informative of what the paper does so for understanding the paper and a high level the introduction is the best place but given that I've already looked at the images and so on I don't actually draw many new much new information from this thing then is related work and honestly I I skip it like unless I'm the actual reviewer of a paper like when I'm the reviewer of a paper I read the


related work but often related work is just like you first of all you site a bunch of your friends and then you cite the mandatory papers and then you cite every single person that you think could be a reviewer because or you've actually been rejected from a conference with a reviewer claiming that your you haven't compared or you haven't cited data or that paper you can pretty much be sure


that that's the if if it's not a glaring of may omission if it's like a niche paper and you haven't cited it then you're like okay I'm gonna cite it just because the next conference you could be Mary viewer again so I'm not I'm not sure that these related work sections they're necessary like if someone wants to write their theses and they go and read this paper and they want references


oftentimes this is a good place but a lot of it is just blah blah blah blah blah okay I know disagree with me if you want oh yeah - maybe - reading quality so I tend to at this point I tend to not skim so at first I skim but at this point I tend to read every sentence and read it closely and understand it and when I realize like I'm tired or something I don't just skim the paper I've tried to


skim papers and it doesn't doesn't work try to read every sentence understand every sentence and okay if you don't understand it don't stop reading because of that but try to not skim and be like oh yeah yeah yeah okay I gotta go to go to get away that is not helpful except related work skip completely cool then a lot of times in this paper now is the the model and this is the section I'm


actually interested in right so I read very very closely here and then I find out what their their loss is all about and again I stress read these things and understand them right sometimes it's hard but if you're if you're confused that means you either if they've done a bad job or they made a mistake or that you haven't under stood something if you can't understand


the sentence try to read on maybe it's clarified later and then you know go back but again do not do not like just start a lot of times when I read paper previously like I wouldn't understand something quite well yet and then I would be like oh yeah yeah yeah and then I noticed that I start skipping and skimming more and more because that would you know pop up again and again


and I wouldn't understand it again and again and then at the end I would just be kind of glancing at the paper and I don't want to do that right here so I want to read every sentence and understand it okay so here then I find out about the loss and then I if I don't know something here then I'll go and look it up on maybe on Wikipedia or something like this now I don't need to


understand every single part of it right that's maybe I should correct myself so for example this bounding box loss here they talk about the second part of the max across the Hungarian Falls is this box loss that scores bounding boxes unlike many detectors that do box prediction with some Englishmen the other yada-yada they say the most commonly used l1 loss will have


different scales for a small so here they basically talk about how they mix the losses they say overall our box loss is that defined as this and this now I haven't I don't know what these losses are I just assume there are some bounding box losses so when I it's not true when I say understand everything understand the things that are integral to the story of the paper right how


exactly they compute bounding box losses at this point I don't care I just assume that there is some loss that I can back propagate right I what is important is that they do this Hungarian matching thing right as soon as I get that I'm like ah that was this you know this um this thing no this thing up here this thing this with the matching thing now I get it now I know uh there are always


the same amount of boxes here there are always the same amount of labels here and all we need to do is somehow match them and I immediately think why is that relevant oh because when something is already matched to an object some other thing cannot be matched to the same object and that's how we you know prevent the fact that all the things predict the same thing right and so that immediately


becomes clear and as I said there is usually like one or two ideas in a paper I don't assume or I don't care what their exact loss function is because I've sort of gotten the idea up here of what the loss is about alright so I hope that's clear under very closely read the things and understand the things that are necessary for the story if you find if you think something's not necessary


for the story and then later end up not understanding that maybe come back and you know read it again in any case I would I would rather I would rather skip something and assume it's not necessary if I think so and then come back then trying to understand every everything but the things I do read I try to understand thoroughly okay then there's the architecture okay and


that again I read closely and get backbone ok transformer encoder ok and now I understand much more closely a decoder ok and here I get now finally I get what this is about because n objects in parallel yada yada yada these input embeddings are learned positional encodings that we refer to as object queries and similarly to the encode we add them to the input at each attention


layer so now they name i've already seen these object queries here and the only word i actually need from this sentence are learnt the fact that their positional encodings I just kind of ignore as soon as they say learnt I know AHA these things here are learned they have actually they're always the same for each of the images they're just over all learned okay so now I feel I


understand the entire model and yeah so then they say auxiliary decoding losses and this sometimes you have to pay attention to like exhilarating because those are the things that here they say explicitly we found helpful to use auxiliary losses sometimes they they won't say why they did it they will just say our loss consists of three things and you know if you look at the three


things only one of the things is really a part of their story so far and that you should immediately conclude that they've put in the other things because they tried it and it didn't work right so you can also kind of get an estimate of the brittleness and so on of the system in that you see how many unnecessary things are there or how many things are not straightforward how many


things aren't the easiest thing that you would do when you would go about and do what they did okay so then you this concludes this model or method usually the student section is called like method or model or something like this and you go to experiments now the main question I have so far or I have maybe I have some more questions about the model itself that I haven't been able to pick


up from this section which is not the case here but I simply keep those questions in mind and see whether they are resolved later right so I keep an awareness of what I don't understand but from here on my main issue is are they demonstrating that their story works right so they're here they're they're proposing a loss and a model and in my mind they now need to convince me that


that works and that's that's it's not as easy as simply to show me some numbers that they are good at some benchmark they need to show me that they get those numbers because of what they claim so here they claim well okay they propose a new they propose a new architecture so what they need to convince me of is that the architecture itself makes sense right but in other papers when when you


propose like and when you say like oh we for example in an L STM when you build in an attention mechanism and you claim oh we you know the attention mechanism can look back at the source sequence in one step then you need to convince me that that actually happens right so you need to not only need you need to perform well you need to convince me that you perform well because of what


you claim your model does right so and that's often difficult and I specifically look out in the experiments for usually the question is like where are they trying to me right where are they trying to are or are they trying to me are they trying to cover up the fact that something doesn't work now all the experiments are always in the best light possible of course and


you have to keep that in mind but a lot of times you can also already see from the experiments that okay are they doing something weird are they not showing me some obvious experiment or and that's a lot of time because is there an easier explanation for why they get the results that they get other than their explanation right and it is it is their job to convince you that their


explanation is the correct one for these numbers and especially if there is an easier one that they haven't excluded and then I don't believe the experiments if that's the case right if there is an easier explanation for the effect I'm I'm very skeptical but some papers have an easier job here than other papers so in this paper they basically show results on a on a on a task and since


their paper is about hey our pipeline is just easier than other pipelines what they first of all need to do is they need to like match the numbers of other pipelines and here I see that okay in these results often in the aftermath maybe a table or something here you see like this their model other models and their model is the best model in a lot of cases now if the best thing is of


course if is if their model throughout is the best the worst thing is if it's like scattered like this even if their model is the best but in every single benchmark a different configuration of their model is the best that's that's sort of a bad sign unless they can explicitly explain why that is and it's also not that good of a sign if these things are spread out like this like


sometimes this baseline is good sometimes their model is better and so on so pay attention to that now in this paper it doesn't matter so much that's actually fine because what they're trying to show is that their model is on par and way easier and they've already made the case in what way it is easier it's easier in terms of architecture if there were to say it's much faster then


after that I would expect you know an experiment in speed while these numbers are matched so but since they say it's easier I've already seen the architecture I'm convinced of that now that they show okay our numbers match actually I'm surprised they even outperform a lot of times then I'm quite happy with these experiments so also look for differences between numbers and


the spread of numbers now it's not easy to say what if like point 1 is a bigger a small difference that depends on the task but if you know pay attention to these things pay attention to the fact that these results are noisy and oftentimes there is a lot more hyper parameter tuning going into the model of the paper then into the baseline model so why do you want to make your look


your stuff look as good as possible and here is a little bit where the institutional credibility of someone like Facebook comes in in that I tend to believe their results a bit more than other results megha but a bit more yeah also look at patterns that they don't point out in the text so if there is like a pattern if you see like an interaction between the number of parameters and the score


or something like this just try to be on the lookout of that and see if you can spot something that you think or think about whether that makes sense or not in what your hypothesis would be so here we go on and okay then they go into ablations and a lot of a lot of these papers do appellations and i generally appreciate that so here they visualize that the attention mechanism in their


model actually refers to different instances right encoder self attention for a set of reference points the encoder is able to separate individual instances and you can see that pretty clearly right here where and even here with the overlapping cows and this is the sort of experiment that I would expect that actually convinces me that their architecture does what it says


that it does right and something like this where you see like totally overlapping things with the attention of the individual things visualized so telling me like especially this one right here the the foot of the back elephant actually being focused by the attention of the bounding box of the back elephant that's the sort of experiment that convinces me that their


claims like that their numbers really come from what they claim it comes from okay so at the end of the experimental section you should always ask yourself have they really convinced me that their story is true right that the improvement or winnig whenever they get an improvement or whatever they get whether is is due to the story that they want to sell me or could there be like an easier


explanation or does something not fit is like other are the experiments different than from what you would SPECT here okay so these are these are my main questions are they are they convincing me of their story it's not do they have state of the art numbers I don't care I don't care even though like sometimes so there is a bit of a catch I I don't care about state of the art


numbers now let's say you have a table like this and you have a computer vision model and one of the models is like on the C for 10 data set now if your baseline model has like a ninety one ninety two percent accuracy on C for ten when I know the state of the art is 96 I don't care right I know like I've done C for ten I know with like I don't know five six layer CNN you can reach these


91 92 93 % accuracy and to get to the 96 97 you would actually be like in the region of a wide ResNet and whatnot so I you know I know that even though you're a few points behind state of the art I know you know this this is valid still so I don't care but if you were to be like at 80 percent accuracy on C for 10 then I then I get a bit like I like it's pretty easy to get to 90 percent plus


with like a standard CNN so there I immediately start to wonder why is there an explanation now this could be like a theoretical paper that says oh we investigate MLPs and that's why we only get that number so that's that would be fine but if something is out of the ordinary like this then I pay attention but never because something isn't like the latest and greatest state of the


earth that's just dumb ok and also if only evaluate what the paper claims it does right if the paper says we want to show that we are on par with current models then don't be mad if the paper doesn't outperform these models they didn't claim that right so yeah so after these ablations I'm actually pretty happy right here with the results and this right here when I saw this I didn't


I didn't expect that but I read the experiment description that these are these different learned object queries and what they do and that gave me an increased understanding of how these object queries actually work right so at that point I still had like a vague I knew that these are learned but reading this and sort of looking at it studying it a bit I was like oh okay then I


understood even better what they are so again when I say understand everything in the method section you can still have questions and but you just have to keep it in mind for later and then here I go on and there's this de TR for Panoptix segmentation and they here they propose like a new model so I first look at it and I'm like okay they proposed a new model they can do stuff like this now


this is not object detection and again I'm not sure is this like a is this like a an add-on to the method or is was was this up here just an intermediate step to this and honestly after reading that I still wasn't sure it seems like something in between of course the paper is also a bit longer than other papers it just it seems it's too long for just being a side note but it's too short for


being its own thing so that was just a bit weird and I treated it as just like a oh we can also do this with our model but I didn't pay like too much attention to that okay so at the end I you know look at conclusions now the conclusions of a paper are much much often they are not nearly as informative as the introduction the conclusions they all often tend to be very generic and kind


of hedging a bit against criticisms saying what would be up for future work which is again hedging against criticism because you could simply say well we didn't do this that's future work yes so again I read it but I don't really pay attention to it and then I gloss over the abstract I just would kind of scroll through the abstract if there's something that catches my eye I would


look at it and if not then not and then I basically go to the start and whenever I didn't understand something I go back I look at it again and I try to think are all my questions answered and have they sufficiently convinced me that their story is the thing that really has the effect right here and then if I now were to make a video of this I've often found it useful to just put the paper


away for a while and it's I usually get the best result when I read the paper the day before and then make a video the day after or if not I'll just you know put it away do something else do some email responding programming going outside eating lunch just some kind of a break between first read or between your first couple of reads and just I don't even think about the paper I just it's


kind of it's in the subconscious it kind of bruised right and I happen to think about the paper every now and then but I don't make a conscious effort to be like oh how am I gonna explain this and so on but I just found the the worst videos are the ones where I immediately make the video after reading a paper and I've just discovered that if I kind of take a


break and then I look at it again right I look I don't read it fully again but I if I have if I have the feeling I've understood it I don't read it fully again but I just kind of look at it and go again through this story and I think that's even if you you know wanna if you want to talk about a paper in a reading group or tell you know explain it to your friends or whatnot this is often


very useful just put it away for a while let it Mellow and I find that helps a lot okay that was my process of reading this particular paper now we again this this is a high quality paper so it's I find it's a pretty easy read in that I simply need to understand what they did and I'm pretty happy with their experiments I maybe next time I can find an experiment


or a paper where I'm initially more skeptical and not as happy with what I find but yeah let me know if you enjoy this or if you would like to see any other explanation I don't exactly know if this is what you expect it from a video like this so let me know maybe I have misunderstood you completely or it's way too long way too detailed or way too undetailed yeah leave me a


all right welcome everyone welcome to lecture 13 of CS 287 last week on Thursday there was no live lecture but we actually did cover the material so ignosi did they recording at home and posted a video on Piazza so you can watch lecture 12 on video it will mean no live version of that lecture but this way we can keep up with the course and not lose a lecture slot today lecture 13

Lecture 13 Kalman Smoother, MAP, ML, EM -- CS287-FA19 Advanced Robotics at UC Berkeley


we'll look at Kalman smoothers maximum a-posteriori estimation maximum likelihood and expectation maximization all right so this is our menu for today let's start with smoothing what's the main idea behind smoothing let's go back to filtering which we've been doing in filtering what you have is you try to find a distribution over variable XT after you have observed some sensor


measurements Z 0 through ZT and ideally over time I keep track of this when you go to the next time you again find a distribution for XT plus 1 given all observations up to time T plus 1 and repeat now if you look at this very symmetric you only use information from the past and often that's indeed the best you can do because right now you need to know the state of the robot as


well as possible so the best you can do is filtering in that situation but what if you're post-processing your data you already collected your data I want to know where was my robot back at time T we should also use the information that happened after and so that's called smoothing and smoothing you use all the sensor observations before and after to come to a conclusion about the


distribution over state of your robot or your environment now and the figures had true here I ignored the actions and in principle it could be action everywhere for every time slice but it does not change the math in any way what does the action do the only thing the action does if we're doing estimation is that the conditional distribution of XT plus 1 given X T becomes conditional XT plus 1


given XT and the action but once the action is fixed because you have the whole sequence is just indexing into a conditional distribution and putting it there index by the action and so let's assume that already happened we already have a conditional distribution or XT given XT minus 1 and maybe what's indexed by the action maybe there was no actions it doesn't really matter the


math is the same all right so now to think about smoothing and compared to filtering let's work out the basics on the board so we'll do this by example rather than having a you know just very general like any kind of horizon problem we'll pick a very specific horizon just horizon 2 so we'll be interested in this up [Music] we'll be interested in the probability


distribution over state at time two so that's our example in general would be any time T but we're going to do by example for time to given observations at time zero one and two now one thing we know is that this is proportional to the Joint Distribution over x2 and the observation Z 0 Z 1 Z 2 again what does this sign mean the proportional sign what it means is that we're looking at a


distribution over x2 and what it means then we can evaluate this quantity here for all values of x2 and once we have all the numbers for every possible value of x2 we can just sum those together divide by that sum and that will normalize it make this sum to 1 and get the actual conditional now the model we have is this hmm like models so how do we write out this thing over here it's


equal to the sum over variables we have ignored because we have a chain in how the probability distribution is set up and we left out x0 and x1 but as you write out the Joint Distribution they'll appear in it so we'll sum out over x0 and x1 and then we can write out the full chain rule for this hmm there is z2 given x2 before that there was x2 given x1 then with that there was


right before that Z 1 given x1 and then before that there was x1 given x0 and before that there was z 0 given x0 and then distribution for X 0 and the graph corresponding to this is X 0 as to capitals x0 x1 x2 and then observed Z 0 C 1 Z 2 so we just throw it out the fool joint over all six variables here but then we only care about these four so we sum out over X 0


and X 1 to get distribution over just a 4 we care about now what do we do in filtering well what can I should start looking at some reorganization here X 0 where does it appear X 0 only appears at the end here so as we look at this summation we can actually move it we can say this is really X 0 summation can happen in the back here so sum over X 0 P X 1 given X 0 p 0 given X 0 P X 0


because as far as X 0 is concerned everything up front is a constant and bringing up a constant up front is fine to do that's just saying multiplying every term with a constant or first summing it all together and then multiplied with the constant is the same thing then how about X 1 X 1 does not appear in here so we can bring this up front so we can bring P Z 2 given X 2 up


front and then we have the summation over X 1 and then this one does have X 1 in it and so that's this one and then this one after we sum out over X 0 we'll have X 1 and Z 0 in it C 0 is a constant that X 1 will be in it so we have to keep it in the back behind this summation now let's give these some meaning and how we would actually run the filtering algorithm


we'd say okay this thing here is actually the joint is to join between Z 0 well let's write X first join between X 0 and Z 0 then we multiply with X 1 given X 0 and we sum out over X 0 we've seen this last week this will give us the joint between X 1 and Z 0 then we multiply with Z 1 given X 1 so multiply with this conditional at this point we have C 1 comma X 1 comma Z


0 then we multiply with X 2 given X 1 so here we'd have P X 2 comma Z 1 comma X 1 comma Z 0 then we sum out over X 1 so it disappears after here we have P X 2 comma Z 2 comma sorry C 1 comma C 1 C 0 and then here we multiply Z 2 given X 2 so at this point we have P Z 2 comma X 2 comma Z 1 comma Z 0 which is indeed what we have over here and so what we see is


that we can recursively compute the Joint Distribution between the latest state variable X 2 and all past observations and the calculation we do is pretty much the same every time we just multiply in an observation they will multiply in the dynamics model and sum over the past variable then we multiply in the next observation multiplied and dynamics some out over


the past state variable and repeat so we have a general recursive approach to finding these things and let's see we're going to put this let me put it over here we have P XT plus 1 comma Z 0 through ZT is equal to sum over X T the dynamics model XT plus 1 given X T times X T comma Z 0 through CT and then to bring in that last observation ZT plus 1 which is ultimate what we want here for


that update now we have Z 0 through Z T n ZT plus is going to be multiplying this thing with the condition of observation given everything else so P Z T plus 1 given X T plus 1 times what we have here P X T plus 1 comma Z 0 through CT now these are bit equations are very easy to run and that's all you need to do to run filtering that's a quick reminder when we go from the joint so here we have the


joint when XT plus 1 is e 0 through Z T and here we have a baton ZT plus 1 in general we need to multiply with the conditional of ZT plus 1 given all the variables already present here not just this one variable need to multiply with the new variable condition all variables you already have but in this model we have a conditional independence we don't need a condition on the past


observations if we have the current state and so that's why we just have XT plus 1 here and not XT plus 1 and all observations to condition on because once we know XT plus 1 those past observations don't need to be conditioned on anymore they're independent of Z T plus 1 and that's shown in the graph structure here and we're looking at we're getting z2 given


everything in the past is the same thing as z2 just given x2 and that's exactly what's happening here why we only condition on the XT plus 1 so that's filtering and we cover them but we covered it now in a different way that last I'm slightly a different way so we can now match up smoothing with what we saw here in filtering I'm gonna have to use a slightly smaller font for this


example actually what a mm-hmm how much smaller brand is in your memories I'll use the entire width of the board as needed to cover smoothing so you've got this and we'll use the full width so smoothing now we care about let me try out here the context we will have x2 again and after the Y should be x3 and x4 and there will be observations Z for Z 3 Z 2 and I should all be things


before it also x1 x0 and observation z1 c0 now we're just it in the distribution for x2 given all observations this is something you can do after the fact I have the fact analysis what happened to my system now to have all the information available so the quantity we're after is P x2 given Z 0 Z 1 Z 2 Z 3 Z 4 now again we know this is proportional to the joint between all


these variables x2 comma Z 0 Z 1 Z 2 Z 3 Z 4 now we don't directly have this kind of joint available what we have available here is something of the form that involves all of these variables so all the conditionals multiplied together in that graph over there is what we have available and then some variables we're not going to care about X 0 X 1 X 3 and X 4 we don't care about we will write


out the full joint over all 10 variables and then sum out the four variables we don't care about so what's the fool joint the fool joint would be Z 4 given X 4 that's happening all the way at the end then X for given X 3 then Z 3 given X 3 x3 given X to Z 2 given X 2 and it will continue on this other board X 2 given X 1 the C 1 given X 1 X 1 given X 0 z 0 given X 0 and then the prior P X 0 this


is the full joint now for this full joint we know we don't care about X 0 X 1 X 3 and X 4 so we're going to sum them out to get rid of them X 0 X 1 X 3 X 4 we sum out over to get this quantity over here alright now we're going to play the same trick it's during when we sum out over these variables is their way to rearrange this into smaller calculations we're going to first


multiply everything together and then finally get to some out but we can do smaller bite-sized calculations where we sum some variables out make them disappear and that way not have this exponential blow-up as the thing becomes bigger if you do it naively this way this summation that the variables are binary let's say then there will be two to the horizon number of terms in this


summation but by bringing them in in the right spots you reduces and you get a linear competition in the length of the network rather than an exponential calculation so can we do the same thing here well let's see how about X 0 actually we can play the same trick as before X 0 instead of summing over it over here where does it appear only in the back here so let's just insert that


summation over here how about X 1 all of these are constants as far as X 1 is concerned so we can put the summation over X 1 over here and get rid of it over here then as we think of this thing here so we have this calculation happening as a first calculation and then after that we can do this calculation which we know from filtering this is giving us the quantities we saw in filtering because


everything happening here so we did in filtering we know what those quantities are going to be then how about these here what can we reorganize here to make this the summations kind of move in well it's going to be similar to what we have happening here x0 is the furthest out in the chain and so it gets on the most inner side simply will be through 4 x4 that's the furthest


away so actually going to be able to first look at X 4 and say we're just going to sum over X 4 let's let's forget about X chief for now we'll need to squeeze in there but we'll sum over Explorer so we're left with summation over X 4 and we can actually just do this part there's no Explorer anywhere else the result of that will be something that does involve X 3 because as you some out


X for Z's are constants but there's still an extreme here so we need to put on the outside still a summation over X 3 it's still in there and multiply in every appearance of X 3 this one this one and so we have this thing over here now let's look at the quantities we have after we group things this way so let's start again over here what do we get if you look at this quantity over here we


looked at it in filtering this is computing P x1 comma Z 0 comma Z one simple recursive calculation then we look at this one over here this quantity here we have not seen it in filtering but we can interpret it what is Z Z for given X for X 4 and X 3 this is really then Z 4 comma X for given X 3 it's what we have over here then we sum over X 4 so we sum out X 4 so we end up with here


is P Z 4 X 4 is being summed out given X 3 all right now as we keep processing this and this will in the future call a backward message be X 3 X 3 is the variable then once we multiply in Z 3 given X 3 what do we what do we get we get I should we multiply multiplying so multiplying this one Z 3 given X 3 which would give us pc3 comma Z forgiven X 3 if we go up to here then we'll apply X 3


given X 2 which will make it multiply next we give an X to graze X 3 to the front P X 3 Z 3 Z 4 given X 2 then we sum out X 3 and we're left with P z3 z4 given x2 so from one side of the chain we get the probability of the evidence that comes after x2 given x2 that's living here over here we have to join between X 1 Z 0 Z 1 but actually we can multiply this thing and some out


over X 1 so we'll get P x2 comma Z 0 z 1 so you have the joint of x2 with the past evidence then we have bringing in the current evidence at time 2m we know the conditional submission for the later evidence given x2 so if you multiply all three of these together we get exactly this quantity over here which is to join between x2 and all the evidences and so he think to observe here is that


evidence that comes from the past is just a standard forward filter being run that's shown over here Adams coming from the future is some kind of backwards filter running that does these updates here that work from the back of the chain back to x2 and give you a condition of all future evidence given x2 and of course you need the evidence at the actual time to also incorporate


it to get the full evidence alright so in terms of math that we did here ultimately all it is is writing out the full joint distribution and moving around the summations and discovering the structure of how can do calculations from the front and from the back to bring in to the time where we're at in a way that is not exponential in the number of variables we're considering


it's linear every calculation is simple we do one simple calculation per time slides to work our way to time - all right any questions about this then let me project in typeset lay tag the equations we just derived by example on the board but we're going to look at see how these on the slides this is what we did we did this the whole smoothing thing this is the full filter equations


at the bottom going to magnify them there's a backward and a forward we can combine them to get the local so here's the full thing we can run a filter forward and we'll call those things a messages in some sense indexed by time so very simple set of update equations incorporate the dynamics model and the next observation and repeat then backward pass does something very


similar but works its way from the back you initialize with just uniform because you have nothing really to go out there's no prior at the end you have just nothing to start from and then you start bringing in evidencing is bringing the dynamics that got you there so the dynamics into the time step you were working from but otherwise it looks exactly the same


and once you have those you can combine them to get the distribution for the variable XT jointly with all evidence at all times now one thing that might come up in practice is that as you run this the way it's shown here even though mathematically is the simplest way you might run into numerical problems because actually compute a joint over more and more variables the actual


probability value will keep going down and down and you might get under flow where you get numbers that are below numbers you can represent in floating point so in practice even though the math is kind of simple and cleanest when working with the joint often you would renormalize as you work along so just say okay I have maybe currently my this thing is really a joint with all past


evidence but I can also just renormalize it and forget about it being a joint and just say hey I'm just going to renormalize and know that it's now a conditional instead of a joint what do you lose you lose that probability you don't know anymore the probability of all the evidence you just have a conditional now for x2 or whatever time slice it is given everything else if you do care


about the actual value because you want to say was this a likely or unlikely run that I just saw happen then you can keep track of these in log space you can just keep track of the log probabilities instead of the actual probabilities and that way avoid the underflow alright so last thing to do if you do it this way is just a normalization but again as I said for numerical reasons often you'll


be the new you'll be doing the normalization as you work along to make sure things stay in the range that your floating point is computation is happy with now we can do other things the same ideas we can use to find pairwise posteriors for example the posterior between XT and XT plus 1 jointly with all the evidence from what we derived on the board it should be clear what's


going to happen here XT and XT plus 1 sitting next to each other we're going to work our way from the front and the back towards them and we're gonna stop right before each one of them coming from each side and then multiply in the middle the conditional XT plus 1 given XT and the evidences for that time slice in fact one way you could Mathematica think of it as just the thing of it is


whatever make XT and XT plus on one variable as if it's one variable we ignore that it's really two variables then exactly the same calculation can be done and then when you unpack this one variable you'll see that you'll have to introduce an XT plus 1 given XT into it because you're unpacking the details otherwise it's just the same thing to compute these forward and backward


messages and then when you are hitting the middle point you essentially just have the backward coming in XT plus 1 the forward coming into XT you're multiplying the conditional and the one observation conditional that you hadn't incorporated yet now you might wonder why would we care about pairwise posterior I shall come clear later in this lecture for now you might say well


why would ever care but we'll clarify that alright so these are just the laws apply but the same as we did on the board oh you might take this to the next level a little harder to do you can do as an exercise can I find the joint between XT and XT plus K which is not a neighboring variable with all the evidence you can imagine that the relief definitely again


messages coming from left and right but you'll have to a little bit of thinking about what you do with the stuff in the middle you still have to sum over those variables in a way but not lose XT as you work your way to XT plus K ik is then you don't recover the joint you need to keep somehow XT around so you'll essentially do something like we did right on something out over XT you just


keep it around and you skip the summing out over X T and keep working your way forward all the way till XT plus K and the XT will just still be in the air because she never some doubt over it what is the common smoother the common smoother is exactly what we just covered applied to the situation where these probability distributions are conditional gaussians where conditional


of XT plus 1 given X T is a linear Gaussian and a conditional of ZT given XT is a linear Gaussian and then their concrete they're not just like these abstract distributions but in that specific case you get the common smoother so you find that the math for common smoother is very similar to come on filter which agnostic covered will be very similar equations happening that


are really just matrix updates you don't need to do any explicit integration any kind of weird integrals that are hard to compute no these closed forms you just manipulate matrices and you'll find updates for your covariance matrix and for your mean and this case will come from both sides and then I'll come together and give you the smooth estimate based on evidence from both


sides so well you can do it as an exercise see if you can work through that if you want to check if you really understood the derivations that we're done in the previous lecture for the common filter you could see if you can find the derivations for the backward pass the forward pass will stay the same the backward pass will be the new thing can you find what that looks like and if


you can do that then means really understood how this works now we can also look at the results the imagine we run a Kalman filter or a common smoother how well does it work so a natural comparison would be something along the lines of let's say I have some dynamical system and I don't get to observe the state directly but I get to see some observations so I would run a


Kalman filter but since I'm running an experiment that could say well okay let me actually give myself access to this state see what it is and see how precise my filter is how well does it track this state and that might be but for debugging and just understanding how well a Kalman filter could work but then you could also sending for the smoother you could say oh let me also run the


smoother and what would you hope for you'd hope for the smoother let's move your estimate to have a mean that is closer to the real state than the filter it doesn't always have to be closer but an expectation it should be closer because it brings in more information by bringing more information it should be able to do better where might this would be most pronounced at time 0 because at


time 0 the filter will have no information yet but its mood will have incorporated everything from the future to estimate the state at time 0 will not be pronounced at all at the very end is at the very end of your time sequence the smoother and the filter uses the exact same information and they should have the same estimate otherwise something funny going on I mean maybe


some numerical things going on but overall they should have the same estimate because they use exact same information to get the estimate at the last time slice in between you can think of it as the smoother having roughly twice as much information is it's not necessarily exactly the Pens in the exact conditional probabilities observer and it can depend on a lot of things but


in general you can think about is having twice as much information especially roughly in the middle and so you'd expect the variance to be about half meaning that the average deviation from the real estate for the smoother should be about half in terms of variance compared to the average squared deviations be about a half compared to the filter well let's take a look here's


some MATLAB code wrote awhile back and ran this experiment and so what we have here is a plot we just did 20 time steps we see in solid line the state or a two state very both losing one state variable shown in blue once the rebels shown in green two-dimensional space and we see the state the greens variable starts at the top there blue starts at the bottom here


then we can look at the smoother in dotted and the filter in dashed we look at the estimates for example early on here we see that well the filter really has no clue when it's just starting out and it's not really close to this state but the smoother is very close because as seen all the future to understand what the state might be now then at the very end we see that they're very close


together because that's just the way it is you might say why is it never perfectly on the state why does it not at the end noticed it perfectly maybe you've seen some things where it says a common thought that will converge to the correct state that's only true is no noise in the system if there's no noise then over time it'll nail the state but because there is noise in this


simulation because noise on the observations noise in the dynamics you can never perfectly know the state because you never get access to it all you get is noisy measurements but we will see that over time the common filter will converge to a kind of fixed variance a fixed expected squared error around the state you'll get that kind of convergence but we won't converge to the


actual state per se any questions about this yes Suroosh actually when you use the mic so you talked about the normalization at the very end is that trivial every in every instance or are there certain instances in which you can't do it analytically or it's computationally intractable yeah so I would say there's even a more general question as we look at these update


equations for filter and smoother are they tractable in general and in general they are not and we'll actually see approximations later this lecture where we it's not tractable because they're integrals and the integrals you need to do it numerically and in high dimensions you can't do it precisely because you need to populate the high dimensional space to get a reasonable approximation


your integral not gonna be able to do it or in a discrete space if your state space is very large imagine I don't know imagine a state variable has is a vector so X is a vector let's say x0 is a vector and each entering that x0 vector can take on I don't know 100 values and maybe 100 entries now you have 100 to 200 possible values for your state and you can't enumerate that in


this summation because 100 to 100 will be far too much to work with and so things we'll see later this lecture is how to deal with this when I would say the equivalent of iterative lqr when it's a nonlinear system but maybe locally it's close to linear and then locally can approximate with linear gaussians and that's the extended column under we already covered that that's


extended Kalman filter I said a common file that you covered last lecture and we'll cover next lecture is particle filters which will essentially do sample based approximations to this entire calculation so they'll say I can't cover everything let me just run a particle filter which is much like the sampling based approaches to value iteration that we saw it's the counter part you just


sample bunch of states look at the valuation update particle filters the equivalent you sample a bunch of possible states you don't know which ones correct you propagate them all re-weight them based on what are the evidence compatible with them or not and that way get an approximate estimate of the distribution and say yes absolutely in general these filter calculations are


not possible to do exactly but in special cases discrete small number of values a state can take on yes very feasible and linear Gaussian distributions for a next state given current state and observation given current state again we can do it in closed form those are the only ones that are tractable the other ones you'll do approximations yes I say let me use this what's an example


of a smoother being useful in that you want to know the posterior you're giving future evidence okay yeah so a question is about the smoother being useful I'm gonna defer the answer to that till the second half of a lecture because we're kind of building up to where we're going to use it and so let's see if it's still that question after lecture but it's a very valid question just a little bit of


patience so what we've covered so far is filtering and smoothing which returns at distribution for the marginal what is the distribution for State at time T given all observations or given all past observations but sometimes we care about it some a little different so top is filtering middle is smoothing bottom you see in red all the states are marked we want to know what is the most likely


joint across all of them now typically adjoins this reason of our many variables not easy to represent so typically what you would do instead of trying to find the fool joint over all of them given all the evidence you'd say let's find the single most likely state combination over all times so what is the single most likely path in state space that was followed based on the


observations I had that's maximum a-posteriori estimation it's about finding the max instead of the distribution now we won't work through the math on the board for this one but it is a bit exercise to try it on your own and the results are on the slides but effectively you'll see happen in these slides is that instead of looking at a summation over the variables we


look at a max over two variables and a max will interact with this whole set of equations essentially the same way as a summation would and we'll play the exact same trick we'll see like which factors have a dependence on each on the variable we're maxing over bringing out in a smaller group together and so we'll recursively be able to calculate the max while running along the sequence so


there'll be a max that start at X 0 and y observation Z 0 what is the X 0 that's most likely given the observation so far but actually we'll do a little more than if what we do is find an egg Sierra that's most likely based on the observation it's not in the silicone paddle with everything that's following so they've just computed the most likely x0 we'll say for every x0 we're going to


calculate how likely it is given the evidence we've seen so far from there we'll then say once we have that we can combine that with the model for x1 given x0 and observation for z1 given x1 to find how likely each x1 is if we match it with the best x0 for that x1 so essentially saying for each x1 how likely is it if I get to match it with the best the most compatible x0 that's


what lives in m1 x1 do the same thing we'll find what how likely is each x2 assuming I get to match it with the best possible choice for x0 and x1 and so it's exactly the same thing instead of saying was a probability for some x2 value summed over x0 and x1 we're just saying if we got to pick the best x0 and x1 so there's a replacing that sum with a max otherwise the same thing is


happening same for them x3 and so forth now generally this would be the update equation just as simple as the ones we saw for filtering but now the summation becomes a max that's the only difference because we're not saying was the probability combined over all possible values of the other variables it's if I got to choose the best choice of value for the other variables now one thing


that happens when you run this at the end of the day where you have is for the last variable X H capital H at the very end you'll say for each value that can take on how likely is it the assuming the other ones take on the best matching but that's in all you have so I should have to keep some pointers around whenever you do this max here you have to keep track of for each value of XT


which value of XT minus 1 is the one that was chosen as the max so you can work your way back along the chain along those pointers to find the full sequence so details are shown here but essentially very simple it's just like the filtering operations except that now we have for all X T we have to store the Arg max to remember the most compatible value for the proof


from the previous time slice so when at the very end we're done at capital T we can see okay for all values of K of X capital T which one is the most likely if it gets to be completed in its optimal way you pick that one followed from that value to what the previous values should be previous value all the way back to the front so very efficient algorithm and you can do this for


example in a tabular case you can do it in general as long as the competition is tractable as long as you can do that maximization sometimes a maximization is easier to do that an integral so sometimes this thing is more tractable to run than doing the actual filtering because well maxing you can run gradient descendant maybe at least find the local maximum whereas if you need to do an


integration it kind of to sum over everything in the space and can be less tractable very often now one special case is the common filter or the linear Gaussian setting so summations become integral sure can't enumerate the overall associations but we can find solutions efficiently we know that when we have because we have multivariate gaussians everywhere the the crazy thing


is in some sense that for the common filter if you think about it if you're on your common filter you find the mean everywhere that sequence of means is actually also the most likely sequence so there is no difference in a common filter between the maximum a posteriori and the means that you find at every step from these from the smoother not the filter from the smoother because you


want the most likely full sequence so either account for everything why is that well think about it what if you do an exact calculation do an exact calculation forget about any kind of algorithms you say I'm gonna compute the full joint over all access given the evidence that's gonna be a Gaussian the Gaussian for all X is given the evidence well if we have a Gaussian for all X is given


the evidence what's the thing that's most likely it's the means all the means and if you wonder what's the most likely for this single time slice given all the evidence it's also the mean for that single time slice so it's a very special case where the means and the fool correlated maximum a-posteriori are actually the same it's because the I mean big part is essentially it's a very


simple distribution compared to most distributions and it happens to simplify that way an alternative you can do in situations like this often is to in this case in particular you can solve an optimization problem because essentially trying to find the set of variables that maximizes the objective namely the log probability some of the log probabilities of the evidences is just a


Comics optimization problem you can also find it that way all right so so far we looked at estimation we are given a model for dynamics and a model for measurements and from that we estimate distribution over state or max most likely sequence over state second half lecture will actually start looking at how we can estimate the parameters in this distribution we assuming they're


given to us we assume we're given dynamics model assume we're given the observation model and practice YouTube we now given them you have to come up with them by hand might be hard more convenient might be to collect data and estimate them and so we'll look at that in when we restart in two minutes let me mute it for a moment [Music] [Music] [Music] alright let's restart so let's look at


estimating some parameters Oh wrong one so simple example let's say we have a thumbtack and you want to build a probability distribution when you throw it up and it lands on the table or in the ground will it I should be pointing up or will it be lying on its side with the kind of neatly thing pointing diagonally down well what do you think was the probability of up or down


probably you know in principle you could think about if first principle say well the air flow around this thing what might happen and so forth not going to be easy to come up with a very precise number so how do we get this parameter then to know probability of up versus down what we can run an experiment imagine we do it 10 times and with this as the results we get we see it's up


eight times and down twice if that's the outcome of our experiments that we might just say well probability of up is 0.8 and we might just work with that now I might say well too small an experiment need to run this for longer sure that's just somebody did this and so any thoughts what the probability is gonna be zero point eight you think zero point eight now maybe I don't know yet well no


on the next slide any other guesses two-thirds it's never hard to know I mean it's very empirical so it turns out total up seventy-seven total down 23 so they tossed up ten of them every time and then looked at how many we're up versus down so yeah seventy-seven percent chance according to this experiment that you land at the pointy thing up okay so that might be our best model we can make for


this short of somebody collecting even more data and getting a more precise estimate of this thing but then I mean this kind of a somewhat specifically designed scenario it's very hard to just you do some first principles but even when you do have first principles available for the dynamical system you're trying to model often a lot of details you won't know very precisely


and so very often you'll still want to run experiments to get a more precise estimate of the dynamics model of the sensor model they you can get from just first principles so let's take a more general look at this math works out and how we can generalize this to other things [Music] so the first thing is that we said okay 77 up 23 down 77% chance that that seems


pretty reasonable but what if your distribution is more complex what are you gonna do I mean maybe there's no way to just do counting so what are you going to do then well the general principle ideally we'd find a general principle that always applies and in the case of the thumbtack experiments still simplifies and gives us the same solution we already know so how can we


generalize this notion that we were just counting to get our best estimate well there's something called likelihood so imagine we observe eight up to down and let's say the probability of up we call theta then we're going to say you know what's the probability of a sequence that we observe maybe we have up down down up up up up up until the end what's the probability of that well we can


write it out if we say it probably of up is theta then the probability of up down down all up would be theta times one minus theta times one minus theta times theta and this would be seven times one time in this two times and so total we'd have data to the eight times one minus theta squared as the probability of that particular sequence happening we'll call


it as the likelihood of what we saw happen if we choose a parameter vector theta then we could say well how should we choose theta again we chosen by doing counts but we're hoping to find a more general principle that will reduce the counts in this case but in other cases still be possible to apply so you could say well a more general principle would be to say I want to find the parameter


theta that maximizes this score because whichever theta maximizes this score is the theta that makes what I saw happen in the world more likely to happen than any other theta I would have made it so it's the best explanation of how the world works at least the part of the world that I observe so you can say okay well this thing look like you can plot this thing look something like this data will


live between zero and one of course this is a probability and then dysfunction data to aid one minus theta squared what does that look like there 0.5 over here it'll look like this with it turns out the peak will be at 0.8 and so that's nice because that means that the principal we intuitively thought was pretty good which is just counting corresponds to something more general


which is looking at the likelihood of one of the experiment under the parameter and then finding parameter that maximizes the likelihood now in general plotting will not be really an option you'll need to somehow find this thing without having to plot it but we've covered optimization already in the class we can look at derivatives gradients and find the optimum this


thing for this very simple objective we can just say derivative of this thing with respect to theta is equal to well what is it it's something like 8 theta to the power 7 1 minus theta squared plus theta to the 8 times 1 minus theta well 2 times 1 minus theta and then there's a minus here so I have another negative 1 appearing here ok and then me if the function really looks like this I


mean the derivative is actually 0 over here and over here so hopefully we can find where this is equal to 0 easily and really we find hopefully at 0.8 so let's see this thing equal to 0 while there's a theta to the 7 here theta up to the 8th over there so we can rewrite this as so I want this equal to 0 but this is equal to theta to D seven times I should have plotted it wrong it's like


gonna go like this here theta to the seven times one minus theta we can bring up front and then there is left eight one minus theta plus 2 times theta equal to zero so we see that this thing will be equal to zero when theta is equal to 0 theta equal to 1 so those are actually minima rather than Maxima they're bad places to be but they have a derivative that's 0 and then the other thing is


whenever this ting is equal to 0 which is 8 minus 8 theta plus 2 theta equal to 0 is that working out for us hopefully it's working out oh the minus -1 - there's a minus sign lost somewhere minus 2 here minus 2 theta so I have a minus over here and so then we have 8 equals 10 theta so theta equals 0.8 so we've got the three places where derivative equals 0 0 1 and 0.8 and this


is of course the one we want we can verify this by plotting or we can verify by taking the second derivative at that spot and seeing that it's a negative second derivative which gives us that shape now this math was kind of ok we can do it but actually practice people often prefer to do the math slightly differently they'll say ok in general we have a likelihood maybe of the type L


theta equals theta to the power N 1 how often we saw the first outcome 1 my state and 0 when we saw outcome 0 and we can work through the same kind of math we saw over there but instead we could actually look at also at the log of L theta the log likelihood which will be log of theta to the N 1 1 minus theta to the M 0 which is equal to n1 log theta plus and 0 log 1 minus theta why


is the kid to look at the log instead of the original thing when you're trying to maximize or minimize by taking the log at every point on the function you are doing monotonic transformations something was the highest point it'll still be the highest point for that function because the lowest still be the lowest the ordering stays the same so it's okay to take the log then the


derivative becomes simple because we don't have this product of stuff anymore we have a sum of things because the log of the product is sum of the logs and we take derivatives which is the sum of derivatives which is simpler than this thing where if we have many complicated terms multiplied together they'd all like stay together in complicated ways and be a lot more hairy to work with and


we can do this thing derivative respect to theta equal to 0 is what we want let's look at the derivative so N 1 times 1 over theta plus and 0 times 1 over 1 minus theta and then a negative 1 here that equal to 0 they were multiplying by theta and 1 minus theta so we have N 1 1 minus theta plus well - and 0 theta equals 0 so now we need to reorganize this a little bit we end up


with N 1 minus N 1 theta minus n 0 theta equals 0 so theta equals N 1 over N 0 plus N 1 which is what we hoping for because intuitive result we thought should be the right one but we're covering it in a principled way that does not depend at all on a distribution of this format you can have any kind of distribution and apply the same principle you could say I have a


distribution as a complicated functional form very hairy form but I can still say this is the likelihood score under the form let me find the parameters that make this maximally likely now this plot over here is the reason people like the logs it simplifies the math that you do by hand that salsa simplifies the math you do it numerically this numerically once you


take the log this plot on log scale on this axis right on the original scale will actually look more like this so it's a nice concave shape with a single optimum there's not this weird curvature happening where you also because this tends to be difficult to optimize with you don't have that show up it's much nicer behaved and so by taking the log you get something numerically easier to


work with just as well as often analytically easier to work with we always talked about convex problems which are problems shaped like this and those are easy to minimize well these are concave problems in this case they're easy to maximize the same thing same algorithms can be applied guaranteed to find the one maximum that exists for this thing it was a question


there so intuitively yeah so it's always I mean if you take a class in convex optimization you'll see that you know half of the class is dedicated to building intuition of how you highball something's comebacks or not and same thing with concave I mean it's the same kind of thing it's hard to say like how intuitively you would do that sort of like working through all those


principles and starting to recognize all the patterns in this case we can just plot it so I made an actual precise plot of what it looks like and we'll see that it looks beautifully concave but yeah I'm practice you would look at second derivatives eigen values of the Hessian if they're all they're all negative then you have a nice concave shape there's no no magic recipe short of all those


tricks that they teach in the convex optimization classes so we covered this we covered this we've covered the log is a monotonic transformation we can just work with the log instead of the original thing and then these are the two plots and again I just generated those plots and so we know in this case it's true because these are the precise plots for that objective but generally


it's going to be true that the log will do where the log of shape I'll essentially help you getting maximum likely problems often will become better condition when you take the log compared to keeping the original and we said here I remember convex concave convex was it once we've covered before any line between two points on the function should be above the function concave is


the other way around and that means you have a unique optimum unique maximum when you have that now effectively you can apply this principle to any kind of distribution we saw just a Bernoulli distribution up or down outcome for the thumbtack well how about multinomial where it can take on different values one two three and so forth up to capital K well we received some samples x1


through XM we can just see what's the log likelihood of these samples well is the log of the product of theta 1 to the power how often we have outcome 1 theta to the power half when we have the outcome 2 and so forth and this is what we end up with and then we can do the math and find that again it comes down to Counting how about an hmm imagine some sample human hmm we see


both the state X and the observation Z at all times if we have that we could estimate the model the dynamics model and the observation model again by doing counts but they were precisely we could look at the made to principally derived this we can look at these are the models we want to estimate let's look at the likelihood of this sequence of observation State and sensor observation


write out the likelihood under the Joint Distribution then we can run the kind of optimization in this case can be done in closed form and we'll find that indeed we'll get the counts for conditional of State at time T given state at time t minus 1 and the counts for the condition of observation given state doesn't need to be count based distributions or discretize regions here is a continuous


distribution exponential distribution and exponential is of the form lambda e to the negative lambda X X can only take on positive values here and lambda is a choice that determines how quickly this thing decays versus maybe having a heavier tail you get some examples some samples from distribution three point one eight point two one point seven you can just say well what's the probability


of each of these samples under this density multiply all of them together or take the log of the product of all of them and then see what maximizes it and in this case lambda is three over 13 and that might not have been as easy to read off by just looking at these numbers might not have said oh it's 3 over 13 you have to do a little bit more math derive what it is and you find that you


know the equation comes down to what you see here which is some summation of all the values you've got in the denominator and the number of samples on top so this is the general version your lambda will be some sense the one over the average of the x-values that you perceive do the same thing for the distributions how about uniform distribution what do you think can be


the outcome there we can plug in the math but actually for uniform it's intuitive is much simpler for uniform you want to maximize mobility of things you saw but it has to be uniform so essentially you look at the farthest out samples and it's a last spot where you assign any probability and everything in between has the same probability so uniform thing that maximize it would be


this thing when your highest sample is be your lowest sample as a it with all the mass between a and B and this uniform has to be equal how about gaussians can do the same thing the math is again the same we're not going to work through the details but essentially you said that's my density function when I have a sample or multiple samples I maximize the product of the


probabilities of these samples or the sum of the log probabilities of these samples log is convenient here because the Gaussian is an exponential in it and the exponential cancels with the log and you work to the math what do you see well the mean of the Gaussian will be the mean of your samples that's the maximum likelihood estimate and the variance parameter of your Gaussian will


actually be the empirical variance on your samples not too surprising but formally derived that that is actually the right thing to do to do maximum likelihood estimation for a Gaussian how about conditional Gaussian but this litigation would be where you have a distribution where Y is effectively a linear regression of X but it could be higher dimensional of course y equals a0


plus a1x plus noise that's a linear Gaussian from one D to one D you can get a bunch of samples work through it and find the maximum likelihood estimate what is it going to be well you have to do some math you'll have a bunch of y's and x's and this will be the probability of their combination and then you'll have to look at okay what maximizes the product of those probabilities you do a


bunch of math what comes out well you see that effectively you get a least square solution that you have to do to find the parameters of this linear Gaussian and you will find that the variance is essentially the empirical variance left when estimating Y from X based on your best estimate of that linear fit you can do this for multivariate gaussians again the math is going to be more hairier to


this higher dimensional and so forth but only it's just very linear map there's no trickery happening you're just saying this is my density there's my data just plug away at it and out comes some result for these kind of solutions in nice closed form and again conditional multivariate Gaussian y equals C times X will come something that looks like least squares solution for the C matrix


and then for the covariance matrix we'll get the empirical covariance on the samples if you actually want to work through this and get this result here are some key matrix identities that are useful otherwise you probably won't get to this result and so these are just kind of things that you at some point might have derived in a previous class or might have never drived it might be a


surprise right now but these are true quantities that can come in very handy when doing these multi well multidimensional derivations with gaussians you'll see these tricks will help you out probably one of the more intriguing ones it's like the gradient of the log of the determinant of a matrix with respect to the entries in that matrix is just the inverse of that


matrix why does it matter well remember a multivariate Gaussian has that determinant of the covariant main covariance matrix up front they all need to find derivative respect to the entries in the covariance matrix in a maximizing the likelihood and the covariance matrix is a parameter you try to find the right setting of that whole matrix will have to take derivatives of


that thing and so turns out nice closed form for this if you don't know about this you might say oh well no closed-form impossible is gonna have to do this numerically but it turns out you can do it as in closed form alright so how about a full we observed linear Gaussian Basin filter setting so you have XT plus 1 equals ax t plus bu t + WT ZT plus 1 equals CX T plus D plus V T


there's a standard common filter type setting if everything is observed you can actually apply maximum likelihood to find a B C D and the covariance matrix Q for W and the covariance matrix R for V and that we have a model of your system now one thing you might want to be wary of is that sometimes you don't want to just do the maximum likely estimate you might


want to pay attention to something else so think about thumbtack example let's say I had five ups would you say that the probability of down is zero probably you would not because you might say well you never know it could be down sometimes and so what that means is that you have some prior information that is not present yet or reflected yet in this data data is too small you have


knowledge about the world that you've condensed in this notion that actually sometimes it could fall the other way just hasn't happened yet in this experiment so we can do you can introduce a prior explicitly to account for that you can say well my prior is something that some probability on theta some 1 1 minus theta I raise them to the same power here theta times 1 minus


theta it's as if as if I've seen one time theta come out which is up 1 times 1 minus theta which is which is down I said I assumed that already happened ahead of time hasn't happened yet but I know it could happen let's assume it already happened and then multiplied in with everything else those those kind of priors are particularly convenient come up with a lot of priors if you take


priors that look like as if you already ran an experiment say assume I already ran the experiment that I already saw a few times is a few times that then I'll be in the same form factor as the likelihood and if your derivation for maximum likelihood came out nice and closed form then this will come out nationally in closed form too because they'll be the same derivation just


introduce some fake experiments in the mix but otherwise everything's the same for this kind of bear knew your experiment this is what it could look like if you have data to the power alpha minus 1 1 minus 2 power beta minus 1 and then closed form on the right there you'll see the effectively add pseudo counts alpha minus 1 beta minus 1 pseudo counts as Zima this happened but hasn't


happened for some of these choices by the way like simple ones you might think about like the sign one here that's all faint go to Baker at beta equal to it's like you put as if each side has already happened once but then in the extreme where alpha and beta are smaller than one it's as if you have a negative version of it happening it's like you think it's not likely both have already


happened it's actually more likely the only one could have happened and your prior comes out the opposite way where you see that it puts a lot of weight on either one or zero not a lot of weight in the middle that's possible too you don't have to make it uniform your prior it's whatever you think might be likely and so if you think I think it's always gonna be the same it I don't know which