mild pneumonia they give her the usual treatment of antibiotics and all's good but then her condition starts to worsen on day six she develops what's called tachycardia that means in medical speak her heart rhythm has accelerated dramatically she then has trouble breathing on day seven she experiences septic shock that means her body is in crisis incidentally mortality in shock is one in two now it's only at this point that the doctors get really concerned and they transfer her to the intensive care unit I see use other units where the most


critically ill patients get cared for they here they give her every possible treatment to stabilize her but her condition only worsens first her kidneys start to fail then her lungs fail and on day 22 she dies mrs. mani did receive the right set of treatments the problem is she received them only too late what mrs. mani experienced was an infection that turned into sepsis let me tell you a little bit about what substance is sepsis occurs when infection releases chemicals in your blood to tackle the infection so your body releases

Better Medicine Through Machine Learning | Suchi Saria | TEDxBoston


chemicals to fight the infection now this chemical can trigger an egg of inflammatory response when this in when this inflammation triggers this negative inflammatory response what it can then do is cause a cascade of changes leading your organs to fail leading to depth sepsis is the 11th leading cause of death more than breast cancer and prostate cancer combined turns out substance is preventable if treated early okay so then what's the catch doctors find it very hard to recognize sepsis in fact a Harvard study shows


with 93 leading academic experts that when they were given several cases of patients with and without sepsis they couldn't agree two years ago my nephew he was admitted to the best Hospital in India and he died of sepsis my family was devastated I'm a machine learning expert and what I do is study ways in which we can use large messy datasets to enable intelligent decision-making so natural question for me was could machine learning of health could machine learning of help mrs. Manny and my nephew so this led to a massive effort


with my colleagues at Hopkins to design what we call the targeted real-time early warning system or trues based on machine learning I'll give you a sneak peek into what chooses and how we're using it to tackle sepsis let me take a step back and tell you a little bit about what machine learning is and what's AI artificial intelligence is a field of study very design very teach computers how to learn okay just like you teach your kids machine learning is one way of doing this by designing code or programs that teach computers stuff


over time by by interacting with the environment or watching okay so I'm going to show you a video of some robots learning how to walk I find it funny how it shudders so you're probably now are thinking this is hopeless well so the question is how can we teach robots or machines how to walk intuitively you can think of it as designing a game the goal of the game is for the computer or the robot to learn how to walk for as long as possible without following ok so to do this first we have to design write down the goal in


a language the computer understands for this we'll use math okay so now you're wondering well how do we write the goal of walking without falling as long as possible in math well that's often hard for different tasks but you can think of it as writing down a formula and what this formula does is it scores so in the case of walking it'll score every move the robot makes if the move it makes helps the robot walk it gets a high score if the move that the robot speaking makes the robot unstable it gets a low score and now the robots goal


is to experiment with the sequence of moves in order to be able to maximize its score so how does it know which moves to try right well there are two strategies for doing it first it expected learns by interacting with the environment okay so here the robot will just make a guess it guesses it makes a move if the move gets a high score that's positive feedback and the robot builds on it okay the second strategy is by watching other robots in other words the robot finds data from past robots that are similar to this robot it watches what


moves that robot did when it was in very similar positions and now it emulates or replicates those moves okay so those are the two strategies so I'm going to show you a video of robot learning how to walk using the strategy I just described okay so in the beginning it's going to look hopeless but I promise you it gets better and just to be clear this is not so this is the skeleton off the robot and so this is not a human animator going there and just moving or animating this video this is really the robot the algorithm


choosing which moves to make by moving the joints of the skeleton that you're seeing and you can see it's already getting better now suddenly the robots a will blue walk and run for a lot longer than it was doing right so essentially the basic principle is as follows you figure out a game that the computer can play you write it down using a language it understands and then we train it to optimise the score right this is how we teach cars how to drive computers how to play the game of go an Alexa to understand say your preference of


coconut water so let's go back in our case stir the problem of sepsis so the goal here is to identify sepsis as quickly as possible right and for this truths learns by watching in other words using data from past patients this avoids the need for tools to have to experiment on new patients right so to do that what are the pieces truth needs to do so one big change that has happened in medicine that's interesting to note is in the past five years the introduction of electronic health records in EHRs every single measurement


every single lab test that is ever done when you walk into the clinic or you're in the hospital gets collected trues analyzes this data from thousands of patients to identify subtle signs and symptoms that appear in patients with sepsis than those without okay but that's not alone what truths also needs to do is to figure out how to think about every signal in the context of every other signal let me give you an example let's look at the example of creatinine in SOL creatinine is a waste molecule okay and your kidneys filter it out okay


but here's the catch so when your body is septic it affects your kidneys it deteriorates your kidneys ability to filter out creatinine so creatinine level rises but there are many other things that can affect your kidneys ability to filter out creatinine for example if you have chronic kidney disease you're very likely to have high creatinine levels so now what truth has to do is to figure out is your creatinine high because of sepsis or because of chronic kidney disease or the numerous other factors that need to high


creatinine levels but that's not enough it needs to do this for every single signal that exists in the electronic health record and chooses think about every signal in the context of every other signal to identify signs and symptoms that occur more often in patients with sepsis than those without let's return to mrs. Manny research by Kumar and colleagues have shown that for every hour treatment is delayed mortality goes up by seven to eight percent so timing is critical we went and took mrs. Manny's data and we ran


throughs on it and here's what we found truth would have detected mrs. Manny's sepsis 12 hours before doctors currently did as my clinical colleagues would say that is the difference between life and death last year we showed using data from 16,000 patients that throughs on average would have detected on most patients on average 24 more than 24 hours prior to the shock onset that's not 24 hours in two-thirds of these patients their sepsis was detected prior to any organ dysfunction whatsoever and to put this result in context that's 60%


increase in performance over state of the art so what truth is really doing is doctors a much longer window to come in and intervene in order to prevent organ dysfunction and mortality this year the independently validated truths in data from Howard County General Hospital in Maryland and now we're working to do real-time integration in order to make something like truths available to every doctor at Hopkins I'm also really excited because after we've published our papers several other health systems are now already


implementing the published version of truths in order to be able to develop it in their own environment so I'm going to highlight like a few perhaps three salient characteristics that I think makes a strategy like truths very powerful ok first truths runs 24/7 what it does is it gives doctors a second pair of reliable eyes right - it's hard to scale-up doctors it's easier I think much easier to scale up computers and what truth is really doing is allowing us to get expertise from the best doctors everywhere here's the third one


which i think is very interesting in many cases like we see in substance we might not need new measurements the signs and symptoms were already in your data and what truth is really doing is discovering these signs and symptoms to learn something that we couldn't see by eye finally there's been a lot of buzz about big data and I want to make a little subtle point about a technical problem that I think truces solving that is very interesting it truths would be able to learn learn much faster if it had a lot of data on you or it could get


more data by experimenting on you but we don't want that right so what truths really has to do is leverage your limited data to figure out what's right for you right so in other words what truths really has to solve ism is a challenging small data problem in other words it has limited data on you and has to figure out what is the right treatment for you and for that it has to let it leverages vast amounts of data from other patients and figures out what information to borrow in order to make these assessments reliably and


precisely so I also want to tell you a little bit about how the strategy is not unique to sepsis so very broadly if you think about it in many diseases essentially where you have profile of symptoms and the response to treatments varies a great deal across individuals you can use the strategy like truth in order to target treatment so you're wondering like for example if you consider cancer diabetes multiple sclerosis Parkinson's lupus so there are many such diseases on which a strategy like Truths is amenable in fact in our


own lab with experts in rheumatic diseases or immune diseases in particular we're looking at how in scleroderma for instance we can use strategies similar to tools to avoid giving strong immunosuppressants to patients who don't need them other colleagues this is William Pelham Susan Murphy and their team they're studying kids with ADHD and looking at how using similar data-driven strategies they can identify when kids can be benefit from behavioral therapy and we can avoid the need for giving them psychostimulants


altogether so the strategy is very very powerful so I was speaking about substance so let's go back to substance again so I said it was sepsis Awareness Month and the CDC has declared substance to be a medical emergency rightfully so remember 750,000 people annually are affected by sepsis a patient's family recently asked me what will it take to bring this to a hospital near us I think that can be done in fact it can even be done within a year but we don't want to stop there we want it to be possible to bring strategy


like truth - hospitals everywhere and so the question is to do that what will it take right so I think the three key things we need your help for one we need super smart engineers to be working in healthcare we need your help in building and scaling up such technologies don't go to wall street healthcare needs you right we need policymakers to create incentives to open up electronic medical records as an expert at a leading health institution it's taken me more than a year because the EMR is so closed in order to be able to figure out how to


implement rules against the EMR it really should be easier than this three we need a healthcare system that's based on quality our current healthcare system is incentivized to optimize volume rather than quality right now you can choose which restaurants to go to based on the quality of food should you be able to choose the hospitals you go to based on quality of care part of the problem is that quality data at the moment is not very visible to consumers and we really need to make a bigger effort to make this quality a visible so


all right welcome everyone welcome to lecture 13 of CS 287 last week on Thursday there was no live lecture but we actually did cover the material so ignosi did they recording at home and posted a video on Piazza so you can watch lecture 12 on video it will mean no live version of that lecture but this way we can keep up with the course and not lose a lecture slot today lecture 13 we'll look at Kalman smoothers maximum a-posteriori estimation maximum likelihood and expectation maximization all right so this is our menu for today

Lecture 13 Kalman Smoother, MAP, ML, EM -- CS287-FA19 Advanced Robotics at UC Berkeley


let's start with smoothing what's the main idea behind smoothing let's go back to filtering which we've been doing in filtering what you have is you try to find a distribution over variable XT after you have observed some sensor measurements Z 0 through ZT and ideally over time I keep track of this when you go to the next time you again find a distribution for XT plus 1 given all observations up to time T plus 1 and repeat now if you look at this very symmetric you only use information from the past and often that's indeed the


best you can do because right now you need to know the state of the robot as well as possible so the best you can do is filtering in that situation but what if you're post-processing your data you already collected your data I want to know where was my robot back at time T we should also use the information that happened after and so that's called smoothing and smoothing you use all the sensor observations before and after to come to a conclusion about the distribution over state of your robot or your environment now and the figures had


true here I ignored the actions and in principle it could be action everywhere for every time slice but it does not change the math in any way what does the action do the only thing the action does if we're doing estimation is that the conditional distribution of XT plus 1 given X T becomes conditional XT plus 1 given XT and the action but once the action is fixed because you have the whole sequence is just indexing into a conditional distribution and putting it there index by the action and so let's assume that already happened we


already have a conditional distribution or XT given XT minus 1 and maybe what's indexed by the action maybe there was no actions it doesn't really matter the math is the same all right so now to think about smoothing and compared to filtering let's work out the basics on the board so we'll do this by example rather than having a you know just very general like any kind of horizon problem we'll pick a very specific horizon just horizon 2 so we'll be interested in this up [Music] we'll be interested in the probability


distribution over state at time two so that's our example in general would be any time T but we're going to do by example for time to given observations at time zero one and two now one thing we know is that this is proportional to the Joint Distribution over x2 and the observation Z 0 Z 1 Z 2 again what does this sign mean the proportional sign what it means is that we're looking at a distribution over x2 and what it means then we can evaluate this quantity here for all values of x2 and once we have all the numbers for every possible value


of x2 we can just sum those together divide by that sum and that will normalize it make this sum to 1 and get the actual conditional now the model we have is this hmm like models so how do we write out this thing over here it's equal to the sum over variables we have ignored because we have a chain in how the probability distribution is set up and we left out x0 and x1 but as you write out the Joint Distribution they'll appear in it so we'll sum out over x0 and x1 and then we can write out the full chain rule for this hmm


there is z2 given x2 before that there was x2 given x1 then with that there was right before that Z 1 given x1 and then before that there was x1 given x0 and before that there was z 0 given x0 and then distribution for X 0 and the graph corresponding to this is X 0 as to capitals x0 x1 x2 and then observed Z 0 C 1 Z 2 so we just throw it out the fool joint over all six variables here but then we only care about these four so we sum out over X 0 and X 1 to get distribution over just a 4 we care about now what do we do in


filtering well what can I should start looking at some reorganization here X 0 where does it appear X 0 only appears at the end here so as we look at this summation we can actually move it we can say this is really X 0 summation can happen in the back here so sum over X 0 P X 1 given X 0 p 0 given X 0 P X 0 because as far as X 0 is concerned everything up front is a constant and bringing up a constant up front is fine to do that's just saying multiplying every term with a constant or first summing it all together and then


multiplied with the constant is the same thing then how about X 1 X 1 does not appear in here so we can bring this up front so we can bring P Z 2 given X 2 up front and then we have the summation over X 1 and then this one does have X 1 in it and so that's this one and then this one after we sum out over X 0 we'll have X 1 and Z 0 in it C 0 is a constant that X 1 will be in it so we have to keep it in the back behind this summation now let's give these some meaning and how we would actually run the filtering algorithm


we'd say okay this thing here is actually the joint is to join between Z 0 well let's write X first join between X 0 and Z 0 then we multiply with X 1 given X 0 and we sum out over X 0 we've seen this last week this will give us the joint between X 1 and Z 0 then we multiply with Z 1 given X 1 so multiply with this conditional at this point we have C 1 comma X 1 comma Z 0 then we multiply with X 2 given X 1 so here we'd have P X 2 comma Z 1 comma X 1 comma Z 0 then we sum out over X 1 so it disappears after here we have P X 2


comma Z 2 comma sorry C 1 comma C 1 C 0 and then here we multiply Z 2 given X 2 so at this point we have P Z 2 comma X 2 comma Z 1 comma Z 0 which is indeed what we have over here and so what we see is that we can recursively compute the Joint Distribution between the latest state variable X 2 and all past observations and the calculation we do is pretty much the same every time we just multiply in an observation they will multiply in the dynamics model and sum over the past variable then we multiply in the next observation


multiplied and dynamics some out over the past state variable and repeat so we have a general recursive approach to finding these things and let's see we're going to put this let me put it over here we have P XT plus 1 comma Z 0 through ZT is equal to sum over X T the dynamics model XT plus 1 given X T times X T comma Z 0 through CT and then to bring in that last observation ZT plus 1 which is ultimate what we want here for that update now we have Z 0 through Z T n ZT plus is going to be multiplying this thing


with the condition of observation given everything else so P Z T plus 1 given X T plus 1 times what we have here P X T plus 1 comma Z 0 through CT now these are bit equations are very easy to run and that's all you need to do to run filtering that's a quick reminder when we go from the joint so here we have the joint when XT plus 1 is e 0 through Z T and here we have a baton ZT plus 1 in general we need to multiply with the conditional of ZT plus 1 given all the variables already present here not just this one variable need to multiply with


the new variable condition all variables you already have but in this model we have a conditional independence we don't need a condition on the past observations if we have the current state and so that's why we just have XT plus 1 here and not XT plus 1 and all observations to condition on because once we know XT plus 1 those past observations don't need to be conditioned on anymore they're independent of Z T plus 1 and that's shown in the graph structure here and we're looking at we're getting z2 given everything in the past is the same thing


as z2 just given x2 and that's exactly what's happening here why we only condition on the XT plus 1 so that's filtering and we cover them but we covered it now in a different way that last I'm slightly a different way so we can now match up smoothing with what we saw here in filtering I'm gonna have to use a slightly smaller font for this example actually what a mm-hmm how much smaller brand is in your memories I'll use the entire width of the board as needed to cover smoothing so you've got this and we'll use the full width


so smoothing now we care about let me try out here the context we will have x2 again and after the Y should be x3 and x4 and there will be observations Z for Z 3 Z 2 and I should all be things before it also x1 x0 and observation z1 c0 now we're just it in the distribution for x2 given all observations this is something you can do after the fact I have the fact analysis what happened to my system now to have all the information available so the quantity we're after is P x2 given Z 0 Z 1 Z 2 Z 3 Z 4 now again we know this is


proportional to the joint between all these variables x2 comma Z 0 Z 1 Z 2 Z 3 Z 4 now we don't directly have this kind of joint available what we have available here is something of the form that involves all of these variables so all the conditionals multiplied together in that graph over there is what we have available and then some variables we're not going to care about X 0 X 1 X 3 and X 4 we don't care about we will write out the full joint over all 10 variables and then sum out the four variables we don't care about so what's the fool


joint the fool joint would be Z 4 given X 4 that's happening all the way at the end then X for given X 3 then Z 3 given X 3 x3 given X to Z 2 given X 2 and it will continue on this other board X 2 given X 1 the C 1 given X 1 X 1 given X 0 z 0 given X 0 and then the prior P X 0 this is the full joint now for this full joint we know we don't care about X 0 X 1 X 3 and X 4 so we're going to sum them out to get rid of them X 0 X 1 X 3 X 4 we sum out over to get this quantity over here alright now we're going to play the same trick it's during when we


sum out over these variables is their way to rearrange this into smaller calculations we're going to first multiply everything together and then finally get to some out but we can do smaller bite-sized calculations where we sum some variables out make them disappear and that way not have this exponential blow-up as the thing becomes bigger if you do it naively this way this summation that the variables are binary let's say then there will be two to the horizon number of terms in this summation but by bringing them in in the


right spots you reduces and you get a linear competition in the length of the network rather than an exponential calculation so can we do the same thing here well let's see how about X 0 actually we can play the same trick as before X 0 instead of summing over it over here where does it appear only in the back here so let's just insert that summation over here how about X 1 all of these are constants as far as X 1 is concerned so we can put the summation over X 1 over here and get rid of it over here then as we think of this thing here so we


have this calculation happening as a first calculation and then after that we can do this calculation which we know from filtering this is giving us the quantities we saw in filtering because everything happening here so we did in filtering we know what those quantities are going to be then how about these here what can we reorganize here to make this the summations kind of move in well it's going to be similar to what we have happening here x0 is the furthest out in the chain and so it gets on the most inner side simply


will be through 4 x4 that's the furthest away so actually going to be able to first look at X 4 and say we're just going to sum over X 4 let's let's forget about X chief for now we'll need to squeeze in there but we'll sum over Explorer so we're left with summation over X 4 and we can actually just do this part there's no Explorer anywhere else the result of that will be something that does involve X 3 because as you some out X for Z's are constants but there's still an extreme here so we need to put on the outside still a summation over X


3 it's still in there and multiply in every appearance of X 3 this one this one and so we have this thing over here now let's look at the quantities we have after we group things this way so let's start again over here what do we get if you look at this quantity over here we looked at it in filtering this is computing P x1 comma Z 0 comma Z one simple recursive calculation then we look at this one over here this quantity here we have not seen it in filtering but we can interpret it what is Z Z for given X for X 4 and X 3 this is really


then Z 4 comma X for given X 3 it's what we have over here then we sum over X 4 so we sum out X 4 so we end up with here is P Z 4 X 4 is being summed out given X 3 all right now as we keep processing this and this will in the future call a backward message be X 3 X 3 is the variable then once we multiply in Z 3 given X 3 what do we what do we get we get I should we multiply multiplying so multiplying this one Z 3 given X 3 which would give us pc3 comma Z forgiven X 3 if we go up to here then we'll apply X 3 given X 2 which will make it multiply


next we give an X to graze X 3 to the front P X 3 Z 3 Z 4 given X 2 then we sum out X 3 and we're left with P z3 z4 given x2 so from one side of the chain we get the probability of the evidence that comes after x2 given x2 that's living here over here we have to join between X 1 Z 0 Z 1 but actually we can multiply this thing and some out over X 1 so we'll get P x2 comma Z 0 z 1 so you have the joint of x2 with the past evidence then we have bringing in the current evidence at time 2m we know the conditional submission for the later


evidence given x2 so if you multiply all three of these together we get exactly this quantity over here which is to join between x2 and all the evidences and so he think to observe here is that evidence that comes from the past is just a standard forward filter being run that's shown over here Adams coming from the future is some kind of backwards filter running that does these updates here that work from the back of the chain back to x2 and give you a condition of all future evidence given x2 and of course you need the evidence


at the actual time to also incorporate it to get the full evidence alright so in terms of math that we did here ultimately all it is is writing out the full joint distribution and moving around the summations and discovering the structure of how can do calculations from the front and from the back to bring in to the time where we're at in a way that is not exponential in the number of variables we're considering it's linear every calculation is simple we do one simple calculation per time slides to work our way to time - all


right any questions about this then let me project in typeset lay tag the equations we just derived by example on the board but we're going to look at see how these on the slides this is what we did we did this the whole smoothing thing this is the full filter equations at the bottom going to magnify them there's a backward and a forward we can combine them to get the local so here's the full thing we can run a filter forward and we'll call those things a messages in some sense indexed by time so very simple set of update equations


incorporate the dynamics model and the next observation and repeat then backward pass does something very similar but works its way from the back you initialize with just uniform because you have nothing really to go out there's no prior at the end you have just nothing to start from and then you start bringing in evidencing is bringing the dynamics that got you there so the dynamics into the time step you were working from but otherwise it looks exactly the same and once you have those you can combine them to get the distribution for the


variable XT jointly with all evidence at all times now one thing that might come up in practice is that as you run this the way it's shown here even though mathematically is the simplest way you might run into numerical problems because actually compute a joint over more and more variables the actual probability value will keep going down and down and you might get under flow where you get numbers that are below numbers you can represent in floating point so in practice even though the math is kind of simple and cleanest when


working with the joint often you would renormalize as you work along so just say okay I have maybe currently my this thing is really a joint with all past evidence but I can also just renormalize it and forget about it being a joint and just say hey I'm just going to renormalize and know that it's now a conditional instead of a joint what do you lose you lose that probability you don't know anymore the probability of all the evidence you just have a conditional now for x2 or whatever time slice it is given everything else if you do care


about the actual value because you want to say was this a likely or unlikely run that I just saw happen then you can keep track of these in log space you can just keep track of the log probabilities instead of the actual probabilities and that way avoid the underflow alright so last thing to do if you do it this way is just a normalization but again as I said for numerical reasons often you'll be the new you'll be doing the normalization as you work along to make sure things stay in the range that your floating point is computation is happy


with now we can do other things the same ideas we can use to find pairwise posteriors for example the posterior between XT and XT plus 1 jointly with all the evidence from what we derived on the board it should be clear what's going to happen here XT and XT plus 1 sitting next to each other we're going to work our way from the front and the back towards them and we're gonna stop right before each one of them coming from each side and then multiply in the middle the conditional XT plus 1 given XT and the evidences for that time slice


in fact one way you could Mathematica think of it as just the thing of it is whatever make XT and XT plus on one variable as if it's one variable we ignore that it's really two variables then exactly the same calculation can be done and then when you unpack this one variable you'll see that you'll have to introduce an XT plus 1 given XT into it because you're unpacking the details otherwise it's just the same thing to compute these forward and backward messages and then when you are hitting the middle point you essentially just


have the backward coming in XT plus 1 the forward coming into XT you're multiplying the conditional and the one observation conditional that you hadn't incorporated yet now you might wonder why would we care about pairwise posterior I shall come clear later in this lecture for now you might say well why would ever care but we'll clarify that alright so these are just the laws apply but the same as we did on the board oh you might take this to the next level a little harder to do you can do as an exercise can I find the joint between XT


and XT plus K which is not a neighboring variable with all the evidence you can imagine that the relief definitely again messages coming from left and right but you'll have to a little bit of thinking about what you do with the stuff in the middle you still have to sum over those variables in a way but not lose XT as you work your way to XT plus K ik is then you don't recover the joint you need to keep somehow XT around so you'll essentially do something like we did right on something out over XT you just keep it around and you skip the summing


out over X T and keep working your way forward all the way till XT plus K and the XT will just still be in the air because she never some doubt over it what is the common smoother the common smoother is exactly what we just covered applied to the situation where these probability distributions are conditional gaussians where conditional of XT plus 1 given X T is a linear Gaussian and a conditional of ZT given XT is a linear Gaussian and then their concrete they're not just like these abstract distributions but in that


specific case you get the common smoother so you find that the math for common smoother is very similar to come on filter which agnostic covered will be very similar equations happening that are really just matrix updates you don't need to do any explicit integration any kind of weird integrals that are hard to compute no these closed forms you just manipulate matrices and you'll find updates for your covariance matrix and for your mean and this case will come from both sides and then I'll come together and give you the smooth


estimate based on evidence from both sides so well you can do it as an exercise see if you can work through that if you want to check if you really understood the derivations that we're done in the previous lecture for the common filter you could see if you can find the derivations for the backward pass the forward pass will stay the same the backward pass will be the new thing can you find what that looks like and if you can do that then means really understood how this works now we can also look at the results the imagine


we run a Kalman filter or a common smoother how well does it work so a natural comparison would be something along the lines of let's say I have some dynamical system and I don't get to observe the state directly but I get to see some observations so I would run a Kalman filter but since I'm running an experiment that could say well okay let me actually give myself access to this state see what it is and see how precise my filter is how well does it track this state and that might be but for debugging and just understanding how


well a Kalman filter could work but then you could also sending for the smoother you could say oh let me also run the smoother and what would you hope for you'd hope for the smoother let's move your estimate to have a mean that is closer to the real state than the filter it doesn't always have to be closer but an expectation it should be closer because it brings in more information by bringing more information it should be able to do better where might this would be most pronounced at time 0 because at time 0 the filter will have no


information yet but its mood will have incorporated everything from the future to estimate the state at time 0 will not be pronounced at all at the very end is at the very end of your time sequence the smoother and the filter uses the exact same information and they should have the same estimate otherwise something funny going on I mean maybe some numerical things going on but overall they should have the same estimate because they use exact same information to get the estimate at the last time slice in between you can think


of it as the smoother having roughly twice as much information is it's not necessarily exactly the Pens in the exact conditional probabilities observer and it can depend on a lot of things but in general you can think about is having twice as much information especially roughly in the middle and so you'd expect the variance to be about half meaning that the average deviation from the real estate for the smoother should be about half in terms of variance compared to the average squared deviations be about a half compared to


the filter well let's take a look here's some MATLAB code wrote awhile back and ran this experiment and so what we have here is a plot we just did 20 time steps we see in solid line the state or a two state very both losing one state variable shown in blue once the rebels shown in green two-dimensional space and we see the state the greens variable starts at the top there blue starts at the bottom here then we can look at the smoother in dotted and the filter in dashed we look at the estimates for example early on


here we see that well the filter really has no clue when it's just starting out and it's not really close to this state but the smoother is very close because as seen all the future to understand what the state might be now then at the very end we see that they're very close together because that's just the way it is you might say why is it never perfectly on the state why does it not at the end noticed it perfectly maybe you've seen some things where it says a common thought that will converge to the correct state that's only true is no


noise in the system if there's no noise then over time it'll nail the state but because there is noise in this simulation because noise on the observations noise in the dynamics you can never perfectly know the state because you never get access to it all you get is noisy measurements but we will see that over time the common filter will converge to a kind of fixed variance a fixed expected squared error around the state you'll get that kind of convergence but we won't converge to the actual state per se any questions about


this yes Suroosh actually when you use the mic so you talked about the normalization at the very end is that trivial every in every instance or are there certain instances in which you can't do it analytically or it's computationally intractable yeah so I would say there's even a more general question as we look at these update equations for filter and smoother are they tractable in general and in general they are not and we'll actually see approximations later this lecture where we it's not tractable because they're


integrals and the integrals you need to do it numerically and in high dimensions you can't do it precisely because you need to populate the high dimensional space to get a reasonable approximation your integral not gonna be able to do it or in a discrete space if your state space is very large imagine I don't know imagine a state variable has is a vector so X is a vector let's say x0 is a vector and each entering that x0 vector can take on I don't know 100 values and maybe 100 entries now you have 100 to 200 possible values for your


state and you can't enumerate that in this summation because 100 to 100 will be far too much to work with and so things we'll see later this lecture is how to deal with this when I would say the equivalent of iterative lqr when it's a nonlinear system but maybe locally it's close to linear and then locally can approximate with linear gaussians and that's the extended column under we already covered that that's extended Kalman filter I said a common file that you covered last lecture and we'll cover next lecture is particle


filters which will essentially do sample based approximations to this entire calculation so they'll say I can't cover everything let me just run a particle filter which is much like the sampling based approaches to value iteration that we saw it's the counter part you just sample bunch of states look at the valuation update particle filters the equivalent you sample a bunch of possible states you don't know which ones correct you propagate them all re-weight them based on what are the evidence compatible with them or not and


that way get an approximate estimate of the distribution and say yes absolutely in general these filter calculations are not possible to do exactly but in special cases discrete small number of values a state can take on yes very feasible and linear Gaussian distributions for a next state given current state and observation given current state again we can do it in closed form those are the only ones that are tractable the other ones you'll do approximations yes I say let me use this what's an example of a smoother being useful in that you


want to know the posterior you're giving future evidence okay yeah so a question is about the smoother being useful I'm gonna defer the answer to that till the second half of a lecture because we're kind of building up to where we're going to use it and so let's see if it's still that question after lecture but it's a very valid question just a little bit of patience so what we've covered so far is filtering and smoothing which returns at distribution for the marginal what is the distribution for State at time T


given all observations or given all past observations but sometimes we care about it some a little different so top is filtering middle is smoothing bottom you see in red all the states are marked we want to know what is the most likely joint across all of them now typically adjoins this reason of our many variables not easy to represent so typically what you would do instead of trying to find the fool joint over all of them given all the evidence you'd say let's find the single most likely state combination over all times so what is


the single most likely path in state space that was followed based on the observations I had that's maximum a-posteriori estimation it's about finding the max instead of the distribution now we won't work through the math on the board for this one but it is a bit exercise to try it on your own and the results are on the slides but effectively you'll see happen in these slides is that instead of looking at a summation over the variables we look at a max over two variables and a max will interact with this whole set of


equations essentially the same way as a summation would and we'll play the exact same trick we'll see like which factors have a dependence on each on the variable we're maxing over bringing out in a smaller group together and so we'll recursively be able to calculate the max while running along the sequence so there'll be a max that start at X 0 and y observation Z 0 what is the X 0 that's most likely given the observation so far but actually we'll do a little more than if what we do is find an egg Sierra that's most likely based on the


observation it's not in the silicone paddle with everything that's following so they've just computed the most likely x0 we'll say for every x0 we're going to calculate how likely it is given the evidence we've seen so far from there we'll then say once we have that we can combine that with the model for x1 given x0 and observation for z1 given x1 to find how likely each x1 is if we match it with the best x0 for that x1 so essentially saying for each x1 how likely is it if I get to match it with the best the most compatible x0 that's


what lives in m1 x1 do the same thing we'll find what how likely is each x2 assuming I get to match it with the best possible choice for x0 and x1 and so it's exactly the same thing instead of saying was a probability for some x2 value summed over x0 and x1 we're just saying if we got to pick the best x0 and x1 so there's a replacing that sum with a max otherwise the same thing is happening same for them x3 and so forth now generally this would be the update equation just as simple as the ones we saw for filtering but now the summation


becomes a max that's the only difference because we're not saying was the probability combined over all possible values of the other variables it's if I got to choose the best choice of value for the other variables now one thing that happens when you run this at the end of the day where you have is for the last variable X H capital H at the very end you'll say for each value that can take on how likely is it the assuming the other ones take on the best matching but that's in all you have so I should have to keep some pointers around


whenever you do this max here you have to keep track of for each value of XT which value of XT minus 1 is the one that was chosen as the max so you can work your way back along the chain along those pointers to find the full sequence so details are shown here but essentially very simple it's just like the filtering operations except that now we have for all X T we have to store the Arg max to remember the most compatible value for the proof from the previous time slice so when at the very end we're done at capital T we


can see okay for all values of K of X capital T which one is the most likely if it gets to be completed in its optimal way you pick that one followed from that value to what the previous values should be previous value all the way back to the front so very efficient algorithm and you can do this for example in a tabular case you can do it in general as long as the competition is tractable as long as you can do that maximization sometimes a maximization is easier to do that an integral so sometimes this thing is more tractable


to run than doing the actual filtering because well maxing you can run gradient descendant maybe at least find the local maximum whereas if you need to do an integration it kind of to sum over everything in the space and can be less tractable very often now one special case is the common filter or the linear Gaussian setting so summations become integral sure can't enumerate the overall associations but we can find solutions efficiently we know that when we have because we have multivariate gaussians everywhere the the crazy thing


is in some sense that for the common filter if you think about it if you're on your common filter you find the mean everywhere that sequence of means is actually also the most likely sequence so there is no difference in a common filter between the maximum a posteriori and the means that you find at every step from these from the smoother not the filter from the smoother because you want the most likely full sequence so either account for everything why is that well think about it what if you do an exact calculation do an exact


calculation forget about any kind of algorithms you say I'm gonna compute the full joint over all access given the evidence that's gonna be a Gaussian the Gaussian for all X is given the evidence well if we have a Gaussian for all X is given the evidence what's the thing that's most likely it's the means all the means and if you wonder what's the most likely for this single time slice given all the evidence it's also the mean for that single time slice so it's a very special case where the means and the fool


correlated maximum a-posteriori are actually the same it's because the I mean big part is essentially it's a very simple distribution compared to most distributions and it happens to simplify that way an alternative you can do in situations like this often is to in this case in particular you can solve an optimization problem because essentially trying to find the set of variables that maximizes the objective namely the log probability some of the log probabilities of the evidences is just a Comics optimization problem you can also


find it that way all right so so far we looked at estimation we are given a model for dynamics and a model for measurements and from that we estimate distribution over state or max most likely sequence over state second half lecture will actually start looking at how we can estimate the parameters in this distribution we assuming they're given to us we assume we're given dynamics model assume we're given the observation model and practice YouTube we now given them you have to come up with them by hand might be hard more


convenient might be to collect data and estimate them and so we'll look at that in when we restart in two minutes let me mute it for a moment [Music] [Music] [Music] alright let's restart so let's look at estimating some parameters Oh wrong one so simple example let's say we have a thumbtack and you want to build a probability distribution when you throw it up and it lands on the table or in the ground will it I should be pointing up or will it be lying on its side with the kind of neatly thing pointing diagonally down well what do you think


was the probability of up or down probably you know in principle you could think about if first principle say well the air flow around this thing what might happen and so forth not going to be easy to come up with a very precise number so how do we get this parameter then to know probability of up versus down what we can run an experiment imagine we do it 10 times and with this as the results we get we see it's up eight times and down twice if that's the outcome of our experiments that we might just say well probability of up is 0.8


and we might just work with that now I might say well too small an experiment need to run this for longer sure that's just somebody did this and so any thoughts what the probability is gonna be zero point eight you think zero point eight now maybe I don't know yet well no on the next slide any other guesses two-thirds it's never hard to know I mean it's very empirical so it turns out total up seventy-seven total down 23 so they tossed up ten of them every time and then looked at how many we're up versus down so yeah


seventy-seven percent chance according to this experiment that you land at the pointy thing up okay so that might be our best model we can make for this short of somebody collecting even more data and getting a more precise estimate of this thing but then I mean this kind of a somewhat specifically designed scenario it's very hard to just you do some first principles but even when you do have first principles available for the dynamical system you're trying to model often a lot of details you won't know very precisely


and so very often you'll still want to run experiments to get a more precise estimate of the dynamics model of the sensor model they you can get from just first principles so let's take a more general look at this math works out and how we can generalize this to other things [Music] so the first thing is that we said okay 77 up 23 down 77% chance that that seems pretty reasonable but what if your distribution is more complex what are you gonna do I mean maybe there's no way to just do counting so what are you


going to do then well the general principle ideally we'd find a general principle that always applies and in the case of the thumbtack experiments still simplifies and gives us the same solution we already know so how can we generalize this notion that we were just counting to get our best estimate well there's something called likelihood so imagine we observe eight up to down and let's say the probability of up we call theta then we're going to say you know what's the probability of a sequence that we observe maybe we have up down


down up up up up up until the end what's the probability of that well we can write it out if we say it probably of up is theta then the probability of up down down all up would be theta times one minus theta times one minus theta times theta and this would be seven times one time in this two times and so total we'd have data to the eight times one minus theta squared as the probability of that particular sequence happening we'll call it as the likelihood of what we saw happen if we choose a parameter vector theta then we could say well how should


we choose theta again we chosen by doing counts but we're hoping to find a more general principle that will reduce the counts in this case but in other cases still be possible to apply so you could say well a more general principle would be to say I want to find the parameter theta that maximizes this score because whichever theta maximizes this score is the theta that makes what I saw happen in the world more likely to happen than any other theta I would have made it so it's the best explanation of how the world works at least the part of the


world that I observe so you can say okay well this thing look like you can plot this thing look something like this data will live between zero and one of course this is a probability and then dysfunction data to aid one minus theta squared what does that look like there 0.5 over here it'll look like this with it turns out the peak will be at 0.8 and so that's nice because that means that the principal we intuitively thought was pretty good which is just counting corresponds to something more general which is looking at the likelihood of


one of the experiment under the parameter and then finding parameter that maximizes the likelihood now in general plotting will not be really an option you'll need to somehow find this thing without having to plot it but we've covered optimization already in the class we can look at derivatives gradients and find the optimum this thing for this very simple objective we can just say derivative of this thing with respect to theta is equal to well what is it it's something like 8 theta to the power 7 1 minus theta squared


plus theta to the 8 times 1 minus theta well 2 times 1 minus theta and then there's a minus here so I have another negative 1 appearing here ok and then me if the function really looks like this I mean the derivative is actually 0 over here and over here so hopefully we can find where this is equal to 0 easily and really we find hopefully at 0.8 so let's see this thing equal to 0 while there's a theta to the 7 here theta up to the 8th over there so we can rewrite this as so I want this equal to 0 but this is


equal to theta to D seven times I should have plotted it wrong it's like gonna go like this here theta to the seven times one minus theta we can bring up front and then there is left eight one minus theta plus 2 times theta equal to zero so we see that this thing will be equal to zero when theta is equal to 0 theta equal to 1 so those are actually minima rather than Maxima they're bad places to be but they have a derivative that's 0 and then the other thing is whenever this ting is equal to 0 which is 8 minus 8 theta plus 2 theta equal to


0 is that working out for us hopefully it's working out oh the minus -1 - there's a minus sign lost somewhere minus 2 here minus 2 theta so I have a minus over here and so then we have 8 equals 10 theta so theta equals 0.8 so we've got the three places where derivative equals 0 0 1 and 0.8 and this is of course the one we want we can verify this by plotting or we can verify by taking the second derivative at that spot and seeing that it's a negative second derivative which gives us that shape now this math was kind of ok we


can do it but actually practice people often prefer to do the math slightly differently they'll say ok in general we have a likelihood maybe of the type L theta equals theta to the power N 1 how often we saw the first outcome 1 my state and 0 when we saw outcome 0 and we can work through the same kind of math we saw over there but instead we could actually look at also at the log of L theta the log likelihood which will be log of theta to the N 1 1 minus theta to the M 0 which is equal to n1 log theta plus and 0 log 1 minus theta why


is the kid to look at the log instead of the original thing when you're trying to maximize or minimize by taking the log at every point on the function you are doing monotonic transformations something was the highest point it'll still be the highest point for that function because the lowest still be the lowest the ordering stays the same so it's okay to take the log then the derivative becomes simple because we don't have this product of stuff anymore we have a sum of things because the log of the product is sum of the logs and we


take derivatives which is the sum of derivatives which is simpler than this thing where if we have many complicated terms multiplied together they'd all like stay together in complicated ways and be a lot more hairy to work with and we can do this thing derivative respect to theta equal to 0 is what we want let's look at the derivative so N 1 times 1 over theta plus and 0 times 1 over 1 minus theta and then a negative 1 here that equal to 0 they were multiplying by theta and 1 minus theta so we have N 1 1 minus theta plus well -


and 0 theta equals 0 so now we need to reorganize this a little bit we end up with N 1 minus N 1 theta minus n 0 theta equals 0 so theta equals N 1 over N 0 plus N 1 which is what we hoping for because intuitive result we thought should be the right one but we're covering it in a principled way that does not depend at all on a distribution of this format you can have any kind of distribution and apply the same principle you could say I have a distribution as a complicated functional form very hairy form but I


can still say this is the likelihood score under the form let me find the parameters that make this maximally likely now this plot over here is the reason people like the logs it simplifies the math that you do by hand that salsa simplifies the math you do it numerically this numerically once you take the log this plot on log scale on this axis right on the original scale will actually look more like this so it's a nice concave shape with a single optimum there's not this weird curvature happening where you also because this


tends to be difficult to optimize with you don't have that show up it's much nicer behaved and so by taking the log you get something numerically easier to work with just as well as often analytically easier to work with we always talked about convex problems which are problems shaped like this and those are easy to minimize well these are concave problems in this case they're easy to maximize the same thing same algorithms can be applied guaranteed to find the one maximum that exists for this thing it was a question


there so intuitively yeah so it's always I mean if you take a class in convex optimization you'll see that you know half of the class is dedicated to building intuition of how you highball something's comebacks or not and same thing with concave I mean it's the same kind of thing it's hard to say like how intuitively you would do that sort of like working through all those principles and starting to recognize all the patterns in this case we can just plot it so I made an actual precise plot of what it looks like and we'll see that


it looks beautifully concave but yeah I'm practice you would look at second derivatives eigen values of the Hessian if they're all they're all negative then you have a nice concave shape there's no no magic recipe short of all those tricks that they teach in the convex optimization classes so we covered this we covered this we've covered the log is a monotonic transformation we can just work with the log instead of the original thing and then these are the two plots and again I just generated those plots and so we know in this case


it's true because these are the precise plots for that objective but generally it's going to be true that the log will do where the log of shape I'll essentially help you getting maximum likely problems often will become better condition when you take the log compared to keeping the original and we said here I remember convex concave convex was it once we've covered before any line between two points on the function should be above the function concave is the other way around and that means you have a unique optimum unique maximum


when you have that now effectively you can apply this principle to any kind of distribution we saw just a Bernoulli distribution up or down outcome for the thumbtack well how about multinomial where it can take on different values one two three and so forth up to capital K well we received some samples x1 through XM we can just see what's the log likelihood of these samples well is the log of the product of theta 1 to the power how often we have outcome 1 theta to the power half when we have the outcome 2 and so forth and


this is what we end up with and then we can do the math and find that again it comes down to Counting how about an hmm imagine some sample human hmm we see both the state X and the observation Z at all times if we have that we could estimate the model the dynamics model and the observation model again by doing counts but they were precisely we could look at the made to principally derived this we can look at these are the models we want to estimate let's look at the likelihood of this sequence of observation State and sensor observation


write out the likelihood under the Joint Distribution then we can run the kind of optimization in this case can be done in closed form and we'll find that indeed we'll get the counts for conditional of State at time T given state at time t minus 1 and the counts for the condition of observation given state doesn't need to be count based distributions or discretize regions here is a continuous distribution exponential distribution and exponential is of the form lambda e to the negative lambda X X can only take


on positive values here and lambda is a choice that determines how quickly this thing decays versus maybe having a heavier tail you get some examples some samples from distribution three point one eight point two one point seven you can just say well what's the probability of each of these samples under this density multiply all of them together or take the log of the product of all of them and then see what maximizes it and in this case lambda is three over 13 and that might not have been as easy to read off by just looking at these numbers


might not have said oh it's 3 over 13 you have to do a little bit more math derive what it is and you find that you know the equation comes down to what you see here which is some summation of all the values you've got in the denominator and the number of samples on top so this is the general version your lambda will be some sense the one over the average of the x-values that you perceive do the same thing for the distributions how about uniform distribution what do you think can be the outcome there we can plug in the


math but actually for uniform it's intuitive is much simpler for uniform you want to maximize mobility of things you saw but it has to be uniform so essentially you look at the farthest out samples and it's a last spot where you assign any probability and everything in between has the same probability so uniform thing that maximize it would be this thing when your highest sample is be your lowest sample as a it with all the mass between a and B and this uniform has to be equal how about gaussians can do the same thing the math


is again the same we're not going to work through the details but essentially you said that's my density function when I have a sample or multiple samples I maximize the product of the probabilities of these samples or the sum of the log probabilities of these samples log is convenient here because the Gaussian is an exponential in it and the exponential cancels with the log and you work to the math what do you see well the mean of the Gaussian will be the mean of your samples that's the maximum likelihood estimate and the


variance parameter of your Gaussian will actually be the empirical variance on your samples not too surprising but formally derived that that is actually the right thing to do to do maximum likelihood estimation for a Gaussian how about conditional Gaussian but this litigation would be where you have a distribution where Y is effectively a linear regression of X but it could be higher dimensional of course y equals a0 plus a1x plus noise that's a linear Gaussian from one D to one D you can get a bunch of samples work through it and


find the maximum likelihood estimate what is it going to be well you have to do some math you'll have a bunch of y's and x's and this will be the probability of their combination and then you'll have to look at okay what maximizes the product of those probabilities you do a bunch of math what comes out well you see that effectively you get a least square solution that you have to do to find the parameters of this linear Gaussian and you will find that the variance is essentially the empirical variance left when estimating Y from X based on your


best estimate of that linear fit you can do this for multivariate gaussians again the math is going to be more hairier to this higher dimensional and so forth but only it's just very linear map there's no trickery happening you're just saying this is my density there's my data just plug away at it and out comes some result for these kind of solutions in nice closed form and again conditional multivariate Gaussian y equals C times X will come something that looks like least squares solution for the C matrix and then for the covariance matrix we'll


get the empirical covariance on the samples if you actually want to work through this and get this result here are some key matrix identities that are useful otherwise you probably won't get to this result and so these are just kind of things that you at some point might have derived in a previous class or might have never drived it might be a surprise right now but these are true quantities that can come in very handy when doing these multi well multidimensional derivations with gaussians you'll see these tricks will


help you out probably one of the more intriguing ones it's like the gradient of the log of the determinant of a matrix with respect to the entries in that matrix is just the inverse of that matrix why does it matter well remember a multivariate Gaussian has that determinant of the covariant main covariance matrix up front they all need to find derivative respect to the entries in the covariance matrix in a maximizing the likelihood and the covariance matrix is a parameter you try to find the right setting of that whole


matrix will have to take derivatives of that thing and so turns out nice closed form for this if you don't know about this you might say oh well no closed-form impossible is gonna have to do this numerically but it turns out you can do it as in closed form alright so how about a full we observed linear Gaussian Basin filter setting so you have XT plus 1 equals ax t plus bu t + WT ZT plus 1 equals CX T plus D plus V T there's a standard common filter type setting if everything is observed you can actually apply maximum


likelihood to find a B C D and the covariance matrix Q for W and the covariance matrix R for V and that we have a model of your system now one thing you might want to be wary of is that sometimes you don't want to just do the maximum likely estimate you might want to pay attention to something else so think about thumbtack example let's say I had five ups would you say that the probability of down is zero probably you would not because you might say well you never know it could be down sometimes and so what that means is that


you have some prior information that is not present yet or reflected yet in this data data is too small you have knowledge about the world that you've condensed in this notion that actually sometimes it could fall the other way just hasn't happened yet in this experiment so we can do you can introduce a prior explicitly to account for that you can say well my prior is something that some probability on theta some 1 1 minus theta I raise them to the same power here theta times 1 minus theta it's as if as if I've seen one


time theta come out which is up 1 times 1 minus theta which is which is down I said I assumed that already happened ahead of time hasn't happened yet but I know it could happen let's assume it already happened and then multiplied in with everything else those those kind of priors are particularly convenient come up with a lot of priors if you take priors that look like as if you already ran an experiment say assume I already ran the experiment that I already saw a few times is a few times that then I'll be in the same form factor as the


likelihood and if your derivation for maximum likelihood came out nice and closed form then this will come out nationally in closed form too because they'll be the same derivation just introduce some fake experiments in the mix but otherwise everything's the same for this kind of bear knew your experiment this is what it could look like if you have data to the power alpha minus 1 1 minus 2 power beta minus 1 and then closed form on the right there you'll see the effectively add pseudo counts alpha minus 1 beta minus 1 pseudo


counts as Zima this happened but hasn't happened for some of these choices by the way like simple ones you might think about like the sign one here that's all faint go to Baker at beta equal to it's like you put as if each side has already happened once but then in the extreme where alpha and beta are smaller than one it's as if you have a negative version of it happening it's like you think it's not likely both have already happened it's actually more likely the only one could have happened and your prior comes out the opposite way where


you see that it puts a lot of weight on either one or zero not a lot of weight in the middle that's possible too you don't have to make it uniform your prior it's whatever you think might be likely and so if you think I think it's always gonna be the same it I don't know which side is going to be but so it's gonna be the same that you have this alpha beta equals 0.5 as a reasonable prior it's only called irrational a distribution which generalizes this to multinomial variables but the high level I mean there's a lot of symbols here Ohio was


the same thing you're just saying I pretend I already saw a few experiments I have pseudo counts for those pretend experiments and I multiply the probability of those into the likelihood of the actual experiments do the same thing for a Gaussian to make the math work out you want the prior for your mean of your Gaussian they also be a Gaussian because then you have a Gaussian multiplied with a Gaussian and we know that's again a Gaussian and the math will be easy if you said well actually I think the prior for the mean


is that I know it's guaranteed to be positive can never be negative so a Gaussian is not a right fit because the Gaussian even if up without positive it'll still have something running negative well then it's not going to work out as nicely with your math and your product to make a trade-off you might say you know what it's fine I take a Gaussian far enough positive a small enough variance there's very low probability mass and a negative and the math will work out cleanly and that's what I'm going to use or you might say


no I'm going to use some other kind of prior and now I have to do some numerical optimization to find the maximum likely or maximum a posteriori estimate because it's not closed for him anymore so if typically we'll make a trade-off between convenience and precision of the prior that you are imposing your problem you can do the same thing for conditional linear gosh you can have priors they're gonna priors over the linear coefficient a but more general kind of priors of the matrix there goes some X to Y or from XT to XT plus one


that matrix a here are some examples of this worked out so the slides can just work through what it looks like when you have a prior so there are some points shown in blue the true relation is shown in green so that's what we were hoping to recover but the data is noisy so the maximum likely this within a small amount of data shown in red is actually pretty far off from green but if it had a prior that in this case thinks the coefficients are more likely to be small rather than be large it'll kind of regularize that and you'll find the


black line which is running closer to horizontal compared to the red line now one thing you also want to do and don't want to forget about is cross-validation so whenever you have some data and you just fit to the data it's possible that you're memorizing data over fitting it rather than paying attention to the real pattern we saw this in valuation sample based valuation you don't want to just over fit the few samples and make sure that you fit your neural net in a way that it generalizes to other data so people we do is you split your data into


train and validation data and then for a range of priors you can compute the maximum posterior e and then C on the validation data which estimate of your maximum a-posteriori parameter gives the best performance on the validation data and that tells you that the prior you use to estimate that was the better prior that's the same thing in standard neural net learning you'd say I put some like coefficient in front of weights square because I want to keep the weights small the coefficient in front of that is a choice it's a hyper


parameter it's a prior over your weights effectively a Gaussian prior over your weights that you're putting in same thing would be happening here you put in a prior and then in cross-validation find out which prior yielded the best results now what we covered so far assumed in all of the maximum life here the maximum posterior that we observe all the data we can write out the density or the probability of all that data and there's no unobserved variables what we're going to cover next week Tuesday because I'm Thursday we'll do


hi there today we'll look at hopfield networks is all you need by researchers from the johannes kepler university in linz and the university of oslo so on high level this paper proposes a new type of hopfield networks that generalizes modern hopfield networks from binary patterns to continuous patterns and then shows that the retrieval update rule of these new hopfield networks is equivalent to the attention mechanism that's used in modern transformers and it's actually a more general formulation of the attention mechanism and therefore

Hopfield Networks is All You Need (Paper Explained)


it can be used to do kind of a variety of things to improve modern deep learning and uh it also has a companion paper where it applies this to some kind of immunology research and gets uh achieves state of the art in a task that is specifically suited to this type of attention all right let's dive in together we'll go over what this paper does what it proposes and so on if you like pay if you like videos like this uh consider subscribing you know sharing it out and i hope you're enjoying this all right also thanks to my discord community


um for you know very helpful bringing me up to speed on this paper uh super interesting discussions there if you're not on our discord yet uh i invite you to join it's fun okay so what is a hopfield network a hot field network is a pretty kind of old style old conceptualization of a neural network so in a hopfield network what your goal would be is you can conceptualize it as a bit of a neural network so let's say we have five neurons or something like this uh your what your goal would be is to have a neural network where you


can store so-called patterns and a pattern in this case would be a binary string of size five so for example one zero one zero zero or one one zero one zero and you'd have a list of these patterns and what your goal would be is to store these patterns in the neural network such that and here you know we'll just consider everything to be sort of connected to everything else and um what your goal would be in this is that you can kind of store patterns inside this neural network and you adjust the weights somehow so this was as i said this was


this was this is kind of an old model um you store you you adapt the weights such that you store these patterns and what does it mean for a pattern to be stored if you have stored a pattern you can you will then be able to retrieve it and you retrieve a pattern in these kind of old style hopfield networks by providing a partial pattern so what you'll say is for example i i want a pattern that starts with one one zero and you give that to the network and there would be a so-called update rule and the update rule is kind


of an internal uh rule so let's just go through this so here this one one zero maybe this is one one zero and then they would kind of send messages around so this update rule would somehow adjust the value of this and this neuron here to what's most compatible with the network weights and if if the network weights have been adjusted correctly this will turn out then at the end of applying this update rule that this is a one and this is a zero and therefore this pattern here is retrieved now had i input uh one zero one at the beginning then


the outcome would be different hopefully this pattern here would have been retrieved okay so you can see the applications of this like you can have the first three digits as sort of a database key and then the last ones as sort of the value that you store along with it and then you can simply provide the first few you can also provide you don't always have to provide three um so this all depends this is this is sort of an as i said an old conceptualization of neural networks so people were imagining that this is kind of how


the brain works you know fire together wire together and also with research into this it it turns out that you know you might think you know there's there's kind of five neurons so maybe i can store five different patterns you know accurately because if i store too many patterns right if i have many many many many patterns then i can't expect to be able to retrieve all the patterns again because some of them will just be so equal that you know many will start maybe with this and and i won't have a chance to to


retrieve the one i want or if the update rule will make a mistake so you might think this might be like five because i have five neurons or maybe ten because i have ten connections but it turns out that um in modern hopfield networks with the appropriate update rule you can store exponentially many patterns in these networks exponentially many in the in the dimension of the um in the dimension of the patterns and here i guess that would be the length of the pattern so this is a little bit surprising the kind


of storage capacity of these networks and we'll this um this paper here generalizes that to continuous uh to continuous states so what do we mean with continuous states i guess i mean continuous patterns so no longer is a pattern a binary string but a pattern now is a string of floating point numbers okay like 0.5 1.3 and so on and you know a string of floating or a sequence of floating point numbers is naturally depicted as a vector okay so our patterns are going to be different vectors that we store and um you know in


high dimensions that the the vectors will be kind of separated well from each other as long as we don't have too many but this paper shows that all these properties for the modern hopfield networks that hold for binary strings still hold if you go to these kind of um continuous to these vector patterns that means you can store exponentially many patterns in the dimensions of the vector which is pretty surprising right because you'd think like you know after you have one vector per dimension that you know after that it might get a bit shaky but


no you can actually store exponentially many that's pretty surprising and this paper is a lot about how to do that and the fact that that happens and so on so we've talked about update rules for these um kind of hopfield networks and i haven't really specified what that is i've just said that you know i enter a pattern and then the network does something and out comes out comes the whatever the pattern that matches my query so this here is called a query you might already um this is on purpose like the kind of overlap between the attention


mechanism lingo and the hopfield network lingo we're going to conflate the two to kind of make clear where the two overlap if you don't know what an attention mechanism is or aren't familiar with it watch my video on attention is all you need uh once you watch that this video will make a lot more sense all right so in what the update rule does is specifically and the update rule there isn't only one right there are many different proposals of hobfield networks and they all lead to different properties but what an


update rule does ultimately is it minimizes what's called an energy so every type of hopfield network is associated with an energy function and this the energy function of the modern hopfield network for binary strings is this energy function right here so with x x is the pattern um the pattern this is the kind of state of the hopfield network and so these are the the whatever is stored in the network and then design here is the query that you enter into the network and then the energy here tells you this quantity you have to minimize this


quantity in order to retrieve the pattern that you want okay now we are never directly working with the energy as such uh so what you could do is for example use back prop or something to use gradient descent to decrease the energy but usually along with an energy function comes an update function and the update function is what i've talked about here like you do something and then the network does something and then you get the pattern out what the network does is it minimizes its energy function and the update rule


is made such that the corresponding energy function is minimized so the energy function is more like a theoretical consideration that you say okay here is my energy function of my hopfield network and the there will be a corresponding update rule that minimizes that energy function and if you use that update rule maybe multiple times then the energy function will be minimized and you will have retrieved your pattern or not if if you have too many patterns stored it might also fail right so they they say what the update rules are


in the um in the text here for the old hopfield networks but we're not really interested in the old ones we're interested in the ones that this paper cares about namely where the patterns that you store in the hopfield network are these vectors over our vector patterns and the query is also a vector pattern so you want to store all of these patterns into the hopfield network so i'm going to draw it like this here i'm going to store it into the hopfield network and then after that you want to come up with a query


and the query is like this and in the case of the binary strings we had something like well i sort of know half of my binary string now in the vector hop field network it's more like well i sort of kind of know the direction that my vector should point in okay and you will re what you want to retrieve is the vector that has kind of a large inner product okay so if i enter this query into my hopfield network what i hope is that this vector here is retrieved now you see it's not exactly the same vector like they do point if i translate that


here by i it's maybe something like this but so they are different but you want to say well i kind of know what i want i kind of want something like this and then the hopfield network would answer with ah i have something like this it's this right here okay so you that the connection to attention mechanism should become pretty pretty obvious right now but you know the um to actually establish this formally is the kind of the point of this paper and you know it's pretty cool to see so they formulate this new energy right


here this is the energy of this new continuous hopfield network um specifically they have to have this term right here because they now have continuous states and continuous queries uh this if you minimize the energy it basically means that your query can never you know go to infinity because you have the the query right here in the energy function um the update rule is this right here and we'll look at that in a moment but remember the update rule is what you actually implement in code so if i if i have a query right here i plug it in


here this is the state of my hopfield network and i apply this rule multiple times and out comes the kind of answer of the hopfield network to my question so the i input this and the out comes this after i update after i apply the update rule maybe multiple times right and interestingly you can already see that this here if you rewrite a bunch of these quantities so if you rewrite the beta here which is the softmax temperature in a way to be one over square root of d and if you take the query design here to be the query


matrix and if you take the x here to be the key matrix then this is equivalent to the update or sorry the attention mechanism of a modern transformer so that's the point of the paper is that we can look at the transformer attention mechanism as a hopfield network and they have this interesting this interesting diagram at the end right here uh so the appendix you know this is typical i guess sep ho haita i remember this sailor paper had like 60 pages of machine proof appendix this also this has like 70 page appendix


crazy but at the end of the appendix you'll find this diagram right here now usually in an attention mechanism um you have whatever the the input is so you have an input right here so this is attention mechanisms or at least transformers they work on sequences or sets of objects and from these you'll generate three things you'll generate the you'll generate the queries the keys and the values now you can either generate the queries from the same objects which would be self-attention or you can generate the queries from like a different object


over here uh it doesn't it doesn't matter too much for our discussions but either you you know have a reference input or you have you know this kind of same input all the way and then what you do is you use three different heads or you know three different matrices to transform that input into queries keys and values so i often conceptualize this as you have kind of your input set and each of the input sets outputs a key and also each one which would be a vector and also each one outputs a query so i often draw this here


the same sequence and each one outputs a query and the query sort of the query is kind of a request for information so the key exposes uh sort of what exposes something about the input here so this could be a sentence down here this could be my cat is very pretty and the the the vector the key vector right here could encode something like this is a noun or this is an animal or anything like this right and the query here it could ask for for other things so for example since this is cat this vector right here the query vector


is generated from that you know token cat now it could recognize that cat is a noun and it could ask the other nodes to basically say are there any adjectives around here because um you know adjectives because i it itself is a noun it's the object of the sentence right it could ask are there any kind of adjectives that describe the object because that would be naturally a thing to ask if you were the noun you would want to know are there any kind of modifiers um for for me so it could output the query and the query here


could mean you know this direction could mean adjectives and you see here the word pretty is an adjective so it itself would output a key that says by the way i'm an adjective right so if the cat asks then if this node asks for an adjective and this outputs uh the adjective vector then because the inner product between the two things is high this will be routed here so attention mechanisms basically information routing that's how i always describe it but in this paper we look at it more like these here are the patterns that are


stored in a hopfield network and i by inputting a query and the dot product being the update rule of the hopfield network i retrieve from the hopfield network i retrieve the appropriate pattern that i ask for okay and then you know the values the values are simply a modification of the keys in this form but a lot of people also do keys and values to be the same thing but this routing of information happens here where you multiply the queries and the keys and then you put a soft max over them okay so if you just look from the perspective


of a single node like this node here this cat node what it would do is it would inner product its own query vector with all of the key vectors right so it would build an inner product with all of these and then it would normalize it would put it through a soft max which will kind of give it a distribution right so here would give it like uh so this this actually matches because ma well my is also very important for cat this this this is just an accident i did not plan this uh this here this also well many things match


but but uh in our example we would just say that this last one it's it's not only higher it's also wider um it matches very well right and so the information routing would route mostly information from this pretty token to the cat token which makes sense in our case right this is the attention mechanism now sin if we are interpreting this as a hopfield network and the update rule here is the dot product you can actually think of applying this rule multiple times so what happens now if we and this is where this update rule


comes in what happens if we take this distribution and we don't aggregate the values like usually we would aggregate the values by this distribution what if we aggregate the keys by this distribution okay what comes out well if we look at this and you know let's just assume that this key right here matches really well but the others also match a little bit what would come out would be a weighted average where a lot of weight is put on this particular key so what will turn out would be something like something that's


very close to that key you can see um i'm going to draw the old key here in green and i want to draw the old query in blue so you see that it's then whatever comes out is not the query but it's also not that only key that matches right it's kind of a weighted average but with that key dominating okay now since you know in a hopfield network what we would do is we would go again we would put this new thing the red thing instead of the query vector okay so we would use this aggregated keys this weighted average


as a new query vector for that node right here so duplicate that node over here i'll use that query vector again and do the same thing again okay inner product with all of the query vectors and now since this is already an aggregate of the query vectors what's going to happen of course the distribution that's going to come out is going to be weighted even more heavily into the direction so let's make it even wider into the direction of that key that matches okay and you can pretty clearly see if i do that iteratively


um then that will lead to a situation where everything is like very low except that one key will sort of dominate the distribution and ultra high and ultra wide okay and that's how that's exactly how a hobfield network works right i would input the query which would be sort of what i want right i kind of know what i want okay and then i apply this rule multiple times right and with each time i refine refine refine until i decide on a pattern right the hopfield network is made for pattern retrieval and these here are the patterns that i


want to retrieve so here the patterns aren't kind of stored in the network beforehand but the patterns are also generated like in an attention layer so the keys are generated by the previous layer or by these matrices but that doesn't matter for the hopfield network update rule so you see here that the attention mechanism can be interpreted as simply one step making one step of this update rule but you can think of making actually multiple steps and and retrieving the particular key so you know deciding on a sort of a hard


routing of particular information now that only works if if there are no other vectors that are close to that particular key right so if the query is this and you know the way i drew it here you can see that there are many there is this one and this one and this one that matches so technically the way i drew it um what would happen most likely is no many no matter how many times you apply your update rule it would sort of result in kind of the average of the three keys right so because they're all matching and they would all contribute


to that weighted average of the query in the next step and then that means basically the convergence would be to something in the middle and that's going to be a central point of this paper um in which situation we are so they call the first part is retrieving a single pattern and they call the second situation where you have multiple patterns that all match that are not well separated from each other they call this a meta-stable state and it's going to be pretty interesting to look at um transform like bert language models


and look at where they actually are are they actually operating in this single pattern retrieval mode or are they operating in the meta stable state mode all right so here you can see it in the diagram the only thing differing this from a hopfield network sorry from an attention mechanism is this branch right here so here you ask do you want to do multiple updates after you've you've multiplied the queries and the keys do you want to do multiple updates if yes so if you're in a this hot field network situation you want to do


multiple updates then you go back as you can see and you do you use the keys together with the output of the softmax to generate a new query so this query q here is now generated from the output here and the key so the keys are the same these are this is the same thing it's just put here twice okay this is exactly what we discussed okay i hope that's some somehow clear that these the the attention mechanism is simply a one step uh hub field network pattern retrieval algorithm with a particular update rule that is that is matches this energy


function that they propose right here of course they do this you know particularly because the update rule that turns out is the transformer update rule but um i actually don't know if they backwards engineered the energy function to match the transformer or if they first came up with a continuous help field networks and then just kind of discover that it's like the transformer we'll maybe never find out okay so um let's go there are a couple of theorems i believe there are four five theorems right here that uh show


that kind of make some points about this about this stuff and we'll go through them we won't go through the proofs or any you know super in-depth meaning but it's pretty cool to go through them and they are proved very rigorously as i said there's a 70 page appendix so have a look at that if you're up for it okay so they say here we have an update rule this is our update rule for our new hop field networks so the first theorem they say is the update rule that we propose converges globally if we apply the update rule repeatedly


the energy for t goes equals infinity and the energy will converge sorry the energy will converge to a fixed point this being a fixed point for t of course sorry for t goes to infinity yeah if this is a fixed point basically saying that if i apply this update rule here over and over and over again it will it will make this energy function converge to a fixed it will make this energy function converge don't want to say anything mistakenly here or claim too much but that basically connects the update rule to the energy


okay so just showing like this really is the update rule for that particular energy function okay now as as itself it's not super duper interesting yet but um now we get to theorem two so theorem two for the iteration that's the update rule that we just looked at we have we have that um this convergence holds as t goes to infinity for some stationary point furthermore this quantity here goes to zero so that means this is the um the update at t plus one and this is the update at t and the difference between them


goes to zero so that means not only does the energy converge but the iterates themselves convert so the algorithm actually converges the individual updates of the algorithm so this e new at some point that will no longer change because the the norm between it and the previous one will go to zero you can see that either the sequence here converges or in the other case the set of limit points yada yada is a connecting subset this is a bit over the top here they say okay it can either converge to a point or it can converge to a


connected subset but if the loss is finite then any sequence generated by the iteration equation three converges to some fixed point so you know basically saying that here we oh this is not the loss i'm sorry um no this is the domain never mind i'm an idiot this is basically saying that this algorithm will converge okay and they define here what it means for a pattern to be stored and retrieved and that's for establishing what the kind of storage capacity of a hopfield network is so we've established that the update


rule minimizes the appropriate energy and the update rule will converge at some point which means that we can you know if it converges we can retrieve the pattern that it converges to so now we define how many patterns can we actually store for that we need to know what does it mean for a pattern to be stored so we assume that we have patterns and these patterns are called x okay x i we have n different patterns each one is called um x with a subscript we assume that around every pattern a sphere is given so how do we imagine this um


we have these patterns and this is this is just a space now they consider patterns of the uh on on a sphere but we'll just conceptualize it as this we'll have a space there are patterns we want to store okay and we'll say around every pattern there is a sphere okay sphere like this and naturally the patterns are going to be there's going to be a notion of well separated patterns and you can imagine this a little bit like these spheres won't be touching each other if these spheres aren't touching each other that means that the patterns are kind of


well separated and that means that if we initialize the query remember the query here is a vector that kind of sort of looks like a pattern and that means the query is kind of close to the pattern in some notion of distance so if we initialize the query somewhere in that sphere then it might if it converges to that sphere to that pattern then we retrieve the pattern okay now it gets a bit more complicated than this but not much more so we say a pattern is stored if there is a single fixed point inside the sphere to which all points