**Medical Decision Making and Decision Analysis**

**October 24, 2012**

**Transcript of Cyberseminar**

**HERC Cost Effectiveness Analysis Course**

**Medical Decision Making and Decision Analysis**

**Presenter: Jeremy D. Goldhaber-Fiebert, Ph.D**

**October 24, 2012**

Paul:  Well, it’s my great pleasure to introduce Jeremy Goldhaber-Fiebert who’s our speaker today.  Jeremy graduated from Harvard in History and Literature and more recently he got his Ph.D. there in Medical Decision—in Health Policy.  And with his specialty in Medical Decision Making.  He received a student award for best research from the Society for Medical Decision Making while he was in his Ph.D. program.  Now four years later, he’s one of the trustees for the Society for Medical Decision Making.  He’s assistant professor of Medicine at Stanford and where his research focuses on infectious disease, Tuberculosis and Hepatitis C, and also cervical cancer and he’s done this work both modeling outcomes and costs in these areas and cost effectiveness as applies to both developing and developed countries.  I—personally am very grateful to be involved with him on a study of looking at cost effectiveness of new hepatitis C treatments in VA.  So without any further ado, I welcome you Jeremy and look forward to the talk.

Jeremy Goldhaber-Fiebert:  Thank you very much, Paul.  So the title of today’s talk is Introduction to Medical Decision Making and Decision Analysis and so I’m going to hope to go through the following set of topics today:  decision analysis and cost effectiveness analysis.  I sort of group those two together for a reason and the latter four topics listed on the agenda: decision trees, sensitivity analysis, Markov models, and micro simulations.  And the reason why I kind of group those two things together is that decision analysis and cost effectiveness analysis are general techniques that don’t necessarily require models.  They often do, but not always.  And so you can perform a decision analysis or cost effectiveness analysis using empirical study data.  But often we don’t have all of the data that we need and we use models in order to sort of help us get the information that we need to perform these analyses.  As I go through the talk, that dichotomy will come up in terms of how the talk is set out.  And my real goal is to get through and to spend enough time on the first five topics today and towards the end I’ll touch on a few more advanced topics, so that some of the attendees who might have additional interests, are aware of some of these more advanced topics and that might raise questions for additional thinking and reading and work.  

So without further ado, the first question is what is a decision analysis?  Before we talk about how one might perform it, we’d like to know what it is.  The typical definition of a decision analysis is a quantitative method for evaluating decisions between multiple alternatives in situations of uncertainty.  So that’s a mouthful and so let’s break that apart.

First we’ll focus on this thing that I’ve underlined here, decisions between multiple alternatives.  So this is critical to a decision analysis.  If you do not have multiple alternatives, you don’t have a decision and therefore you don’t have—you have nothing to analyze.  Now almost all this all problems have can be thought of as do it or don’t do it, so in general that’s not a problem, but alternatives and getting at those alternatives is key.  And what it means to decide between things in the context of a decisional analysis is that one is going to allocate resources to one of those alternatives and not to the others, right?  Whether that’s—I’m going to spend my time working on something and not something else or whether I’m going to deliver a given treatment and not a different treatment, what resources means is very broad, but the idea is I’m going to pick one of these from multiple that is what I mean by a decision.

Second part is quantitative method for evaluating decisions.  So what does that mean?  That involves gathering information, assessing the consequences of each alternative action, specifically information about that, clarifying the dynamics and tradeoffs involved in selecting one choice versus another, and then to select an action that gives the best expected outcomes, so the way I’m going to evaluate the decision and just choose between these alternatives is based on the one that maximizes the expected outcome that I care about. So the decision maker—this is not a normative exercise.  The decision maker is going to specify what he or she or they care about and then the analyst, the decision analyst is going to do a quantitative exercise in order to say if you choose alternative A you will get the most of what you told me you care about.

If you do alternative B you’ll get less.  If you do C you’ll get even less.  Therefore you should choose A.  All right.  So let’s unpack a number of these concepts.  Before I go on, I should say that we often employ probabilistic models to do this so that gets on the idea that things are uncertain and that we don’t have all the data readily available to us in one nice empirical study that we can then analyze statistically.  And so I’ll talk about modeling, as I said before, a little bit further down.

Let’s talk about the steps of a decision analysis.  First we want to enumerate all relevant alternatives.  Decision analysis involves a comparison between the expected outcomes under one intervention versus the expected outcomes on another.  So we know we can set up a straw horse comparison where we take something and compare it to something that’s terrible and that first something will look great, but that doesn’t really tell us how much better it is than the alternative.  It tells us how much better it is than something that is a straw horse or terrible. What we want to do is we want to simultaneously consider all of the relevant alternative strategies, actions, treatments, and screening tests, whatever it is you’re deciding between.  So that’s key.  So we spend some time thinking about that.  We think about what people are doing now.  What’s the clinical status kind of state of the art care?  What are FDA approved, maybe, or what might be FDA approved.  What studies and trials tell us may be efficacious and reasonable alternatives?  Sometimes things aren’t feasible.  Even though an MRI might be useful in some for some condition in the particular hospital that we might be conducting this analysis for—they might not have an MRI machine.  So therefore the MRI is not a relevant alternative in that context.

Okay.  Then we’ll identify the important outcomes.  Whether this is costs, or life years, or cases of a given disease, or quality adjusted life years, which I’ll talk a little bit more about further down in the talk.  We want to identify the outcomes that are important and again we want to try to think about as many of these outcomes as possible in our steps for designing this analysis.  We ultimately might collapse some of these, combine some of these, and focus on some of these, but we want to think about what are the outcomes that are important.

We want to determine the relevant uncertain factors.  How sensitive or specific a diagnostic test is.  How efficacious a given treatment is, so what we might know about the efficacy of a given treatment is from a trial and that trial has an estimate and it has an uncertainty band and that’s what I mean by uncertain factors.  There are certain things—chance events or things that we haven’t measured to absolute precision that we should reflect because that will—that will matter to us in computing our expected the expected outcomes under each of the different alternatives.  We want to encode the probabilities for these uncertain factors.  So we say, the efficacy of treatment is uncertain.  From these studies we know that the relative risk reduction from this treatment is such and such and this is the range or the distribution of uncertainty around that point estimate.

We’ll do that for the relevant uncertain factors.  We’ll specify a value for each outcome so for life years, each year is a year.  For quality adjusted life year, a life year lived in perfect health is valued differently than a life year lived with substantial disability.  And the way we value these outcomes depends on what those outcomes are.  

We will then combine these elements to analyze the decision and I will talk a bunch about this in the coming slides, but ultimately we’re going to combine these elements, and we’re going to multiply the values of the outcomes by the probabilities of those outcomes occurring for each alternative and then we’re going to compare the expected outcomes under each alternative and find the alternative that gives us the most of the outcomes that we or the decision makers that we are working with desire.  Decision trees and related models are important for this.

You can use a decision tree even if the probabilities and outcomes all come from one given study.  And so—what I’ll talk about next are decision trees. 

Before I do that, I want to say one thing.  So decision analysis can focus exclusively from say in the context of medical decision making on health.  In the context of choosing variety of chemo and radio therapy or what not for giving cancer, it might just be about extending survival.  That might be the thing that the physician and patient care about so we can conduct a decision analysis focused exclusively on asking which treatment option maximizes survival for this particular patient.  We might also in a variety of resource allocation problems ask about cost effectiveness and when costs are included as one of the important outcomes, a typical way to do that is with a cost effectiveness analysis, a type of decision analysis that includes cost as one of its outcomes.  So what is a cost effectiveness analysis?  It’s a type of decision analysis that includes cost as one of its outcomes.  In the context of health and medicine a cost effectiveness analysis is a method for evaluating tradeoffs between health benefits and costs resulting from alternative courses of action and a cost-effectiveness analysis or CEA supports decision makers; it’s not a complete resource allocation procedure.  So when I talk about what I’m talking about in the next few slides, this is not meant to say that decision makers should always choose the strategy that maximizes—that provides the most health benefit for a given amount of money.  There may be other concerns that are very relevant in a—for a given decision making context beyond cost effectiveness, which is a measure only of efficiency.  It’s not a measure of equity—who gets what—being distributionally fair.  And there are other concerns that people may have.

All right, so how to compare two strategies in a CEA.  I just—sort of talked in general about two different alternatives and saying how do I maximize you know life years or survival in the context of cancer.  So in cost effectiveness we have two outcomes: a cost outcome and a measure of health benefit or effectiveness.  We typically look at this as a ratio.  So the ratio is shown at the bottom of the slide and says CER or cost effectiveness ratio, also known as an incremental cost effectiveness ratio.  And the numerator of that ratio which is denoted Ci-Calt is supposed to express the idea of the difference between the costs of the intervention or strategy if you will and the costs of the alternative under study.  So this might be new treatment versus old treatment and what are—what is the incremental costs or the additional costs, let’s say of doing the new treatment.  This is not just the cost of the new drug versus the cost of the old drug but that new drug may lead to fewer events, fewer costly hospitalizations or what not, so it’s the difference in total costs, both those averted as well as the cost of the new treatment, under the new treatment versus those total costs under the alternative.  The denominator likewise is sort of very similar.  It’s the difference between the health outcomes or effectiveness of the intervention and the health outcomes of the alternative.  Right? And so this forms a ratio of differences.  

So you can think of the numerator as the incremental resources required by the intervention and the denominator is the incremental health effects gained with the intervention.  And that is the way that we sort of think about that and we ask the question is the amount of additional money that we need to get that amount of additional benefit good value for money?  What good value for money means is beyond what I’m going to talk about kind of for the majority of today but I’m happy to speak about it briefly in the question period if that is something that people are interested in.

Okay.  So often time we need models for conducting decision analysis and cost effectiveness analysis and so what do I mean by a decision model.  It’s a schematic representation of all the clinically and policy relevant features of the decision problem that we’re considering.  So that’s a nice short definition, but let’s unpack it again a bit.  So it includes the following in its structure: the decision alternatives, the strategies, the treatments, what we’re deciding between, the clinically and policy relevant outcomes.  This is the life years or quality adjusted life years, the QALYs, or cases averted, costs etc.  And sequences of events or passive events that may have things that are uncertain about whether they will occur. There may be chains of events that have some chance attached to them.  It enables us to integrate knowledge about the decision problems from many sources, so we might have one study that talks about the probabilities, a different study that talks about the values associated with outcomes, and another study that talks about the relative efficacy of the various alternatives in terms of how they change the probabilities of some events that lead to outcomes and so this is in some ways you can think about this model as a form of synthesizing evidence very much focused on deciding between alternative courses of actions and then we use this model to help us compute the expected outcomes averaging across the uncertainties for each decision alternative so that we can compare the alternatives to each other on the agreed upon outcome or metric of a desired metric that we’re trying to get the most of.

Let’s talk about building a decision-analytic model.  So building a decision-analytic model you can think of in steps.  We’re going to define the model structure.  Now these steps are often iterative so in practice we typically will define a structure and then we’ll have conversations with domain experts if they’re not involved in initially doing that and also use it to interrogate the available data and will sort of refine the structure so that it in an iterative approach so that structure represents a reasonable view of reality and is feasible to populate with data.  We’ll assign probabilities to all the chance events in the structure.  We will assign values, utility weights for quality adjusted life, for costs of associated with each outcome and we’ll encode that into the structure.  And we’ll evaluate the expected utility of each of the decision alternatives or each of the expected outcomes—the value of the outcomes of each of the decision alternatives doing something called averaging out and holding back which I will illustrate in detail in a few slides.  Then we’ll perform, and this is extremely important, a set of sensitivity analyses with the goal being to understand if our decision changes with reasonable plausible alternative assumptions typically about the probabilities or values of outcomes.

So we want to build a model that’s simple enough to be understood and complex enough to capture the problems, elements in a convincing way.  In a convincing way.  In a way—to lay our assumptions very bare on the table and another value of such a modeling analysis is that we’re making everything very explicit.  We’re saying this is how we arrive at the conclusion that we arrive at.  So an important quote from George Box and Norman Draper is that “All models are wrong.”  And they were talking in this case about statistical models, not really decision models, “but some models are useful.”  So the way we think about models, there is no way to capture all of the exquisite detail about the real world and any tractable parametrical computable model.  So what we want to do when we build this model is build a model that captures all of the salient features of the decision problem in sufficient detail but is understandable and is feasible to build and to analyze.  So I remind myself of this often because there’s often the temptation to kind of model in exquisite detail a set of things which we don’t know whether they’ll even matter very much.

Let’s talk about defining the model structure and what do I mean by the model structure.  So what are the elements of a decision tree structure?  A decision node is the first element.  It’s a place in the decision tree at which there’s a choice between several alternatives and typically the way we represent this when we diagram out this tree is with a square.  In this case, in this hypothetical example there’s a decision between surgery for some particular patient or patient population and medical management or medical treatment and the decision maker at this point gets to decide or the patient in consultation gets to decide surgery or medical management and the question is what are the expected outcomes under surgery or medical management.  Now in the example that I’ve given I’ve shown two alternatives, surgery and medical management.  As I said before, we want to kind of think about all relevant alternatives so the decision node can accommodate as many alternatives as we want between and these alternatives have to be mutually exclusive.  I’ll define that term in a second.

The second type of element that goes into a decision tree is what’s called a chance node.  It’s a place in the decision tree at which chance determines the outcome based on a probability.  So in this example which—so we typically denote a chance node as a circle.  The chance is either the patient has no complications or the patient dies.  Now again, the chance node can have more than two branches.  You can have no complications, complications, and dies.  And that probably would make sense in this context because we want the chance node to represent outcomes that are mutually exclusive and collectively exhaustive.  So with these two terms mutually exclusive and collectively exhaustive —I’ll define those on this next slide.

So what do mutually exclusive and collectively exhaustive mean?  So mutually exclusive means only one alternative can be chosen or only one event can occur.  And collectively exhaustive, the term that we that we use in the chance node, is that at least one event must occur—one of the possibilities must happen and taken together the possibilities make up the entire range of outcomes.  Finally, the last element of a decision tree for decision analysis is something called the terminal node.  And that sort of defines the final outcome associated with each pathway of choices and chances.

So in this—and that’s typically denoted with a sort of sideways triangle.  And it’s a final outcome must be valued in relative terms so cases of disease, life years, quality adjusted life years, cost, etc., so that they can be used for comparison. In this case, this particular terminal node ultimately leads to thirty years of remaining life expectancy.

Let’s combine these elements around a very stylized exemplar problem.  I should say as you saw in the opening slide, I’m a Ph.D.,  I’m not a clinician and my example, as I say is extraordinarily stylized.  Please, I’m sure there are a number of clinicians on the line.  Don’t beat me up on all of the clinical mistakes that I’ll make in going through this example.  I apologize in advance for any clinical deficiency I might have.

So to summarize, before we jump into this example.  We have decision nodes which enumerate a choice between alternatives for the decision maker to make.  Chance nodes which enumerate possible events determined by a chance or probability.  And terminal nodes which describe outcomes associated with a given pathway.  A set of choices and chances.

The entire structure of the decision tree can be described with only these elements.  So let’s take an example.  We have some set of patients who are symptomatic.  We don’t know what their underlying disease is. It’s a likely serious disease and we won’t know whether they actually have that serious disease until treatment is provided.  There are two treatment alternatives.  Surgery, which is potentially risky, and medical management, which has a low success rate but is substantially less risky.

With surgery there are essentially two types of surgery—whether it’s curative or palliative and that is assessed once the surgery is—is underway that that’s done.  So the goal in this particular decision analysis is to maximize life expectancy for the patient.

Okay, so we start with the initial decision.  Should we choose surgery or should we choose medical management?  If medical management is chosen there’s a chance which we don’t observe at the beginning that the disease is present or the disease is absent.  If the disease is present, given medical management there is a chance that the person is cured and a chance that the person is not cured.  

Likewise, with surgery there’s a chance that disease is present and disease is absent and this presence or absence should be the same across the surgery branch.  These should be the same, which we’ll talk about in a second and these should be the same.  Since these don’t influence the prevalence of disease.  But what they do influence is the chance of living or dying in the absence of disease.  Surgery is risky and there’s a chance of surgical death denoted here with the chance node for surgical death.  And if the disease is present, then there’s essentially whether the surgery is going to be a curative surgery or a palliative surgery. And again, there’s a chance of surgical death or living, surviving the surgery, and there’s a chance of cure or no cure depending—which is different depending on whether this is curative surgery or palliative surgery.

All right, so I’ve used this term a number of times, this pathway term to the terminal node.  So this pathway I’ve drawn here as a dashed orange line and so that is the pathway the outcome that we’d have to say what is the remaining life expectancy for someone who has surgery, has disease, is given curative surgery, survives the surgery, and is cured by the surgery.  And that clearly would be different for somebody who is given surgery, has the disease, and the type of surgery is curative, and the person dies on the operating table.  Right, and so that’s why these pathways are going to tell us what the outcomes—you know what the outcomes will be.  And that’s what I mean by pathways, a combination of choices and chances.  All right, so the first thing I do is attach a probability.  So as I said before, the prevalence of disease is not impacted by the choice of treatment.  As we can see the cure rate is low for medical management.  There’s a chance of surgical death even for people without the disease.  Depending upon whether you’re doing curative surgery or palliative surgery, there is a higher chance of surgical death.  Right?  But also a higher chance of cure with curative surgery than with palliative surgery.  So and where we get these probabilities might include studies, experts, primary analyses that we might conduct ourselves using administrative data or trial data or what not.

Finally, we need to attach outcomes.  So let’s add outcomes.  So anybody—so surgical deaths which happen sort of on the day of surgery gets zero additional life expectancy, so zero y.  Zero years.  If cure is not effected, then there’s two years of remaining life expectancy in this example.  And if cure is effected or if the person didn’t have the disease at all then the remaining life expectancy is twenty years.  Right.  And I sort of made these the same across all of them.  If there was a reason—because we have these pathways, if one pathway led to a shorter life expectancy, you could reflect that.  So this could in theory be 19.  You’d have to have a reason why you’d make twenty this—nineteen different than all the different twenties, but that, in theory, can happen.  So we now have a structure with decision nodes and chance nodes and terminal nodes.  We’ve attached probabilities to all of our chance nodes and we’ve attached values or outcomes to all of our terminal nodes and now we can do something called averaging out and folding back.  And I’m going to talk about—I’m going to illustrate this.  So averaging out means that when we have a chance of one—one thing occurring or another, we can multiply the chance by the value plus the other chances by the other value and get an expectation.  So let’s take a simple one here.  We’re going to average out—so if I have medical management and the disease is present there’s a ten percent chance of cure and a ninety percent chance of not curing.  So if you multiply ten percent by twenty plus ninety percent by two, and get 3.8 years on expectation.  And I replace that little chance node with its expectation, which is 3.8 y, shown in blue.  Notice that this is essentially the same calculation with the same values here.  So we should do that again.  Right?  .1 times 20 plus .9 times two equals 3.8 years on expectation.  And then we can repeat. So now medical management with the disease present had a ten percent chance of getting 3.8 years on expectation because sometimes it cures and sometimes it doesn’t and a ninety percent chance of the disease being absent in which case we have twenty years, so we can average and fold back.  .1 times 3.8 plus .9 times 20.  Equals 18.38 years and medical management on average, on expectation yields 18.38 remaining life—years of life expectancy.  We can now focus on our surgical branch and again perform that same computation.  Two percent times zero.  Ninety eight percent times 3.8 years for palliation.  3.72 years.  Again, .9 times—2 times 20 plus .1 times 2.  18.2 years.  And again.  .1 times 0 plus .9 times 18.2, 16.38 years.  

Here we come to an interesting spot.  This one is different.  It’s a choice node.  It’s a—it’s a decision node.  And the surgeon or the surgeon and patient have picked whether to do curative or palliative surgery.  In this case since we want to choose the thing that maximizes life expectancy we’re going to choose curative surgery because here it delivers the highest expected gain, 16.38 years as compared with 3.72 years.  So this is called folding back.  This is not a chance.  I’m just going to cut this part off of the tree and I’m going to choose that.  Now we continue averaging out and folding back and so we’re at our next chance node over here which is 19.8 years and again, .9 times 19.8 .1 times 16.38 and we get 19.46 years on expectation with surgery that is curative, or is intended to be curative.  Now we compare these two alternatives based upon their expected outcomes.  The decision is between surgery and medical management and we take the difference between those two, sort of like the denominator in our cost effectiveness ratio.  We’re not concerned with cost here, so we get a difference of 1.08 years and we recommend choose surgery with try cure surgical option because it maximizes remaining life expectancy.  If we were conducting a cost effectiveness analysis we would have done the same thing except we would have attached cost to all of those different outcomes.  When we averaged out and folded back we would have two numbers that we were averaging out and folding back at each chance node and choice node.

Let’s suppose that surgery costs $10,000 on expectation and medical management costs $100 on expectation so we would have computed the same 1.08 years of remaining life expectancy and $9,900 as the incremental cost with surgery and then we take the ratio to compute the incremental cost effectiveness ratio.  $9,900 divided by 1.08 years gives me $9,167 per year of life gained.  So if we are willing to pay at least $9,167 per life year gained, we would choose surgery, otherwise we would choose medical management.

All right.  So let’s focus a bit on sensitivity analyses.  So we have a lot of probabilities.  They came from studies or expert opinion or our own analyses and some of them may be very uncertain.  For example, the chance of cure or of surgical death given type of treatment.  So sensitivity analysis is a systematic way of asking what if questions to see how the decision might change as a result.  We’re—another way to think of it is it determines how robust the decision is.  Basically if I can change the uncertain probability across—or wide range across the range of uncertainty and it doesn’t change my choice of surgery over medical management then my decision is robust to the uncertainty in that parameter.  In a one way sensitivity analysis, also called a threshold analysis, we’ll vary one parameter across some range denoting its uncertainty and see if our decision changes somewhere in that range or if it stays the same.  In a multi way sensitivity analysis you can imagine this, for more than one parameter we’ll do this simultaneously for two—two way sensitivity analysis and we’ll see if that changes.

So let’s assume for a second that the probability of surgical death with curative surgery is highly uncertain.  So what we’ll do is we’ll do that same averaging out and folding back, changing this value between zero and one hundred percent.  We don’t have to do it across that wide of a range, if we do it more tightly.  But let’s just do that and what we see is that the expected benefit with medical management denoted by the red line here remains at roughly 18.38 years of life because surgical death doesn’t change the expected benefit for medical management.  There’s no surgery inside medical management.  However, surgery that’s curative if expected life years declines as the likelihood of dying on the operating room table increases, so up here if there’s no chance of surgical death, it’s well above 19.5 remaining years of life, and at 100% it’s well below that of medical management.  So our base case is 10%.  Right?  And clearly the blue line is higher than the red line which is essentially the reason why we chose surgery.  This difference is the 1.08 years that we saw previously.  And another key point is the threshold, the point on the graph where the decision changes.  So if our uncertainty about the probability of curative surgical death is that it goes between zero and .3, our decision is robust in that range.  Right?  So if we’re only uncertain over this range about that probability then the blue line is always on top of the red line.  However, if we’re really uncertain about it and death might be .8, in our uncertainty range then in fact the decision might be somewhat sensitive to or depend upon the probability of curative surgical death.  

All right, likewise I can vary the probability of curative surgical death shown on the Y axis here and the prevalence of the disease and this might show us where surgery is preferred.  This region.  And this is where medical management is preferred. And if you think about the one way it’s essentially the line that we saw—the lines that we saw on the previous slide would be fixing the prevalence at our base—at our base prevalence and looking at the effect of this parameter.  So in our base case is somewhere over here.  So if we were simultaneously uncertain about each of these parameters enough it might be the case for example if we think that the true prevalence of disease in symptomatic patients is substantially lower then we might be uncertain about whether we should do surgery or medical management.

All right, so I’ve tried to highlight as advanced topics that I’m just going to touch on briefly.  An additional type of sensitivity analysis called a probabilistic sensitivity analysis or 2nd order Monte Carlo simulation or sensitivity analysis.  The idea is that you have a whole bunch of probabilities in your—values in your decision tree and you attach uncertainty distributions to all of them, draw from those distributions, run the analysis with each of those draws and look at the percentage of time that one strategy is preferred to another. I’m not going to say more about it but just to note there is this more advanced technique and if some of you are going on to conduct these analyses, this is becoming increasingly used and required by journals like Annals of Internal Medicine.

All right, so sometimes we use decision trees and sometimes and often in published analyses Markov Models are used.  I’m going to use my remaining time and I know I don’t have a lot of remaining time to talk about Markov Models.  So what to do when there is a possibility of repeated events or decisions. When there’s a possibility of repeated events or decisions, that’s when we want to use a Markov Model.  So if a decision is one time, has immediate action with immediate kind of acute outcome, say, do an intervention and increase the probability of survival, the decision tree is often appropriate.  However, decisions can involve choices about diseases that have either repeated actions or time dependent events.  So let’s look at this.  So let’s say a person at time zero has a chance of remaining well or becoming somewhat sick.  If they have a chance of becoming somewhat sick then they have a chance of becoming well again, remaining sick, or becoming very sick where serious outcomes could occur.

If a person is well at this time period one, they may become sick before time period two or they may stay well and again, if they become sick they have this chance again—you can imagine.  If you’re thinking about on the order of weeks or months and you’re looking over a person’s lifetime, just how big and intractable such a tree that has a recursive property is to build as a decision tree.  So we use Markov models when we have this chance of events occurring and recurring repeatedly in time.

So what is a Markov Model?  It’s a mathematical modeling technique derived from matrix algebra that describes the transitions a cohort of patients make among a number of mutually exclusive and collectively exhaustive health states during a series of short intervals or cycles.

So that’s the formal definition.  Let’s take this apart and illustrate it.  So there are some properties that go along with this Markov Model.  Patients are always in one or individuals in one of a finite number of health states.  Events are modeled as transitions from one state to another.  The time spent at each health state determines the overall expected outcome.  So living longer without disease yields higher life expectancy and higher quality adjusted life expectancy than living with disease.  During each cycle of the model, individuals may make a transition from one state to another or they remain in the state that they’re in.  So to construct a Markov model we’ll define a set of mutually exclusive health states and we’ll determine possible transitions between these health states—often called state transitions or transition probabilities and then we’ll determine a clinically valid cycle length.  That means the time between allowed transitions.

One slide on cycle length.  You want to make the cycle length short enough for a given disease that’s being modeled that the chance of two events—meaning two transitions occurring within that one cycle is essentially zero or very small.  So for some applications that might be weekly or monthly.  Say for chronic diseases and major events that occur with chronic diseases and some situations like thinking about management and care of patients in ICUs you might want to use a much shorter cycle length.

That’s different depending on the problem.  Let’s take a simple model, natural history model of disease and we’ll talk about health states. In this simple model we have healthy, sick, and dead.  So these states are mutually exclusive and collectively exhaustive.  A patient is either healthy or sick or dead.  And in no more than one of the states and when deciding on states one should try to choose the relevant biologically or pathophysiologically plausible sense of states, clinically relevant states.  And then we’ll have two Markovian assumptions. The first is, and these are very important, homogeneity.  This means all individuals in the same state—in the same health state have the same cost, quality of life, and risk of transition out of that state.  So if I’m in the sick state, I look like everybody else in the sick state.  If there are relevant gradations of sick then I need multiple, that either have different costs or different qualities or different risks of transitions, then we need multiple levels within sick or multiple health states: Very sick, not so sick, etc.  The other important Markov assumption is memorilessness, which is that my risk of transition from, to future states only depend upon where I am now.  So if my history matters for my transitions then I need to have more states that capture differences in history.

And the way we sort of get around both of these problems, as I’ve alluded to, is via stratification and/or tunnel states which are advanced topics and again I can take those up in question period if people want.

The next things we need are transitions.  So I’m showing blue arrows which show the transition from healthy to dead, from sick to dead, from healthy to sick, and from sick to healthy.  And the allowed transitions obviously depend upon the disease.  And of course, there are for many health states since dead you can’t transition out of, essentially that sort of implies that anybody who’s dead stays dead and there’s a transition back into the state.

Of course an important part of Markov models in the context of health is that people have a risk of death at all times and from all states and that they are not protected from death by being in a state.  That would be a mark of an incorrect model.  If you have a state like dead that has no transitions out of it it’s called an absorbing state. And over time, everybody will ultimately flow into that state.

So you can think about these transitions as a set of probabilities, the probability of transitioning from one state to another with during a given cycle.  So pHH here is the probability of going from healthy to healthy, pHS the probability of going from healthy to sick, pHD the probability of going from healthy to dead and likewise for these states and that’s just the way I’ve denoted these things and this is called my transition matrix.

All right.  So at the start of the model and at each time step or each cycle in the model, there’s a proportion of our population that are in each of these states of healthy, of sick, and of dead.  And when we compose this matrix of transitions on the proportions of people in the states, we get a new sort of state vector or proportion of the population in different states.  And these transition probabilities, this transition matrix, can also depend upon time as well, so the probability of transitioning might be different for thirty year olds than for forty year olds or something like that.  And that’s an advanced topic, which again, I’m happy to talk about.

So what happens is from a matrix multiplication point of view is that we multiply the probability of healthy going to healthy by the proportion of people that are healthy.  The probably of people who are sick becoming healthy by the proportion of people who are sick.  And the probability of people who are dead going to healthy is obviously multiplied by zero since that’s not an allowed transition and we’ll get the new proportion of people who are healthy.  So multiply across the arrows and sum—in practice nobody does this matrix multiplication.  We do it with software for building these things.

So over time these proportions in healthy and sick and dead change.  So if you think of t evolving on the X axis the proportion in each of these states always sums to one and changes over time and of course since dead is the absorbing state, dead rises with time and the others change in a variety of ways.  So is the proportion of people who are healthy the same thing as the prevalence?  The answer to that question is no since prevalence is the proportion in a given state divided by those who are not dead and here the proportion who are not dead both declined in time and is model time the same thing as age?  Again the answer is not necessarily.  We could start our cohort at age twenty so model times zero is really equivalent to age twenty but there is an easy way to transform between model time and age of our cohort.

So these proportions which I show graphically here can be shown on a table. Paul you should just signal me when I should sort of cut off.  I have a couple of more slides and I’m going to then not cover micro simulation.  I’ll just leave those in the slides and for questions if that’s all right.

So underlying the trace are these proportions... 

Paul:  That sounds fine.

Jeremy Goldhaber-Fiebert:  ...that are in each of our health states at a given time.  And death of course is going up and the proportions in each of the other ones are different.  So when we want to value these outcomes, because again, we’re going to use this to make a decision, we’re going to attach, if we’re doing quality adjusted life years, we’re going to attach let’s say a value of one for each cycle lived in healthy and a value of .6 for each cycle lived in sick and a value of 0 for each cycle lived in dead.  And so what you can think about it is that these values, these QH or 1, QS or .6 are multiplied by the proportion of the people in the population who are in that state at that cycle and we’ll sum across and I’m not showing any discounting here.  Again that’s something that I will happily talk about in the question period.  But that’s how we would compute the QALYs using this model, which has these repeated chances of getting sick and getting well and getting sick, which is why we wouldn’t necessarily have a decision tree.  And likewise we could do this with cost where the costs of being healthy are different than the costs of being sick.

So interventions, essentially what they do is they modify some of these probabilities.  They might change my probability of going from sick to healthy.  It makes me more likely to become well or there might be a preventative intervention that reduces my chance of going from healthy to sick.

I have an example I have on the slide which I’m not going to go through because of time which involves a screening test followed by a treatment.  And essentially what happens in the example is that people get screened and those who screen positive receive treatment and that treatment effectiveness reduces their chances of becoming sick or remaining sick.

This is the transitions in natural history.  Our transitions which I showed calculated above in the presence of the intervention, right, which would be these numbers.  This would be the life expect—the proportion of the population that’s alive at different times right, without intervention and the increased proportion of the population that’s alive with intervention and that would give our life expectancy increase or QALY increase with the intervention and once we did that then we’re back to treatment A versus treatment B what’s the life expectancy.  What’s the other life expectancy?  Take the difference and decide.  So these Markov models are just ways of computing those blue boxes with life expectancy to make the decision ultimately.

This is showing how we would diagram this out with—where all the nodes are essentially the same as before except for this M node which stands for the fact that we’re in Markov Model so you cycle back to into a state in the Markov Model.

All right.  So I’m not going to touch on Monte Carlo Simulations versus Markov Cohort simulations.  I’m happy to say something briefly about that.  There are slides here.  And with that I’ll turn it back to Paul and take any questions that people may have.  Thank you for the opportunity to present to you all.

Paul:  Well, that’s great, Jeremy.  There was one question asking about what software do you use?

Jeremy Goldhaber-Fiebert:  That’s a great question.  So there are a variety of software that are available.  People often use Tree Age.  Other people use software called decision maker.  You can do this in Excel and at the Society for Medical Decision Making which offers short courses, they actually have run a course the last couple of years, showing how to do this in Excel.  For more complex models, people will use mat lab.  I have built such models using straight statistical software like Stata.  There are specialized software for doing very fancy things and sometimes people build really complicated models using general programming tools like Java or C++.  Many courses use Tree Age as a teaching tool and many models that you’ll see published in the last five years if you looked across them you’d see generally Tree Age.

Paul: I guess the one additional piece of information to mention is that Tree Age is not free.  It has an annual license fee.

Jeremy Goldhaber-Fiebert:  That’s right.

Paul:  Whereas Decision Maker is a shareware program and not nearly as friendly.

Jeremy Goldhaber-Fiebert:  That’s right.  So there are tradeoffs between sort of what you pay for, what you get, how easy is it for you to use and also along the dimension of how complicated and sophisticated the problems that you’re considering are.

Paul:  And then someone asked if you might speak a little bit about micro simulation.

Jeremy Goldhaber-Fiebert:  I’m happy to.  So if—micro simulation, I think is best shown by this picture that I have up here.  On the top I have healthy, sick, dead just like our Markov model but a micro simulation basically generates a whole bunch of little individual synthetic patients who then flow through the model where instead of the those numbers in our transition matrix being proportions of the population flowing smoothly at each time that represent the chance that that individual in that next cycle will transition from healthy to sick.  So in this first case, this individual becomes sick, stays sick, becomes healthy, becomes sick again, and dies.  Another ind—and those are those probabilities of events where we sort of flip a coin with those pseudo random number generator.  Another individual with the same probabilities might become healthy, healthy, sick, healthy, healthy, dead.  And another individual might be very unlucky and die in the first cycle.  And the treatment might change those probabilities for the individual in the model.  Again, if you think about it, if you do this for a large number of individuals, you can ask for the proportion of this large number of simulated individuals what proportion are healthy at time one.  Looking across all those paths, what proportion are sick and what proportion are dead and once you’ve done that and you’ve filled in the numbers in this table then the computation of the expected QALYs and expected costs and the decision analysis are the same.

So the reason—one other thing that I’ll sort of say is why not just always do micro simulation?  That’s sort of how the world works, you know, individual people experiencing individual events.  So micro simulations are even more challenging to build and code and so and they’re more computationally intensive and you have to deal with the fact that you have a finite number of individuals that you simulated so you have uncertainty in your estimates of QALYs and of cost and so that’s the reason why you might not want to do a micro simulation and the reason to do a micro simulation is that your transitions depend upon complex patient history or your interventions are complicated, screen people until they have screened three times consecutively negative and then stop screening them for five years and start again is a very hard rule to represent in a Markov Cohort model and it’s much easier to represent that in a micro simulation.  So there is sort of this tradeoff.  Don’t make the model more complex than it needs to be for certain problems, you need to add that extra level of complexity and micro simulation is a way of doing that.

Paul:  Great.  So there was a question about one of the slides.

Jeremy Goldhaber-Fiebert:  Yes.

Paul:  And the slide was the one that was—let’s see—right after the graph of proportion by model time.

Jeremy Goldhaber-Fiebert:  Yes.  I’ll get to there in a second.  We had proportion by model time.  Right?  I’m sorry—let me get back here.  Let me see if I can do it this way.  Sorry guys.  Hopefully I’m not making—so we have proportion by model time.  And then I asked those two questions.  Is this the one or is it the...

Paul:  They said, I think it’s why do you have seven stages in the final slide?  I thought there were three.

Jeremy Goldhaber-Fiebert:  No.  No.  So underlying the trace—I’m showing the—so the columns are the proportion who are healthy, the proportion who are sick and the proportion who are dead and not dead is just one minus the dead.  It’s just a way of keeping track of what percentage are either healthy or sick.  So you can ignore this final column.  That’s a great question and the stage is the same thing as the time T.  And the reason why I’m showing seven here is because by stage seven in this highly stylized example, everybody is dead, so there will be no transitions because everybody is dead, there will be no transitions out of dead so there’s no point in running the model past dead.  So typically what we do in reality is we take a lifetime horizon for the model and we more or less run the model until the cohort could be up to like 120 years old.  At which point everybody is dead.  In fact, in practice, it is almost always the case that long before 120 years; the overwhelming majority of people are dead in the model and therefore what happens those last two years doesn’t really matter.

The number of rows is dictated by the cycle length times a reasonable amount of time until everybody in the model is dead and then nothing more is happening.  I’m not accruing any more QALYs.  I’m not accruing any more costs.  And no transitions are occurring.

Paul:  Somebody asked if you had a template for how you’d build a model in Excel and it occurs to me this page could be one.

Jeremy Goldhaber-Fiebert:  Yes.  So—so that’s right. So you can think about instead of having these numbers just be output numbers, these numbers could be equations.  Right?  So multiplying this row by a matrix in Excel to get this row.  Right?  That’s one way of doing it.  What I would say is that what I just said is a way to do it.  But it’s not best practice in the context of Excel.  I know that—so one thing that we can do is look up on the SMDM website and identify who the course leaders were of that Excel based course and we can see—we can at least link to that so people can sort of see who’s talking about doing that and it may be the case that those instructors make some teaching materials freely available in which case we can link to them or what not.  So I’m pretty sure some of the instructors are at Stanford so we may be able to prevail upon our colleagues.

Paul:  Heidi can distribute that to the course attendees.  I think we have time for about one more question or about two minutes before the hour. Somebody wrote: for a disease like tuberculosis history of past disease increases the risk of future disease and can a Markov Model deal with that?

Jeremy Goldhaber-Fiebert:  Sure.  Typically people who are modeling tuberculosis are using sort of dynamic transmission models which are an entirely separate thing to talk about.  Infectious disease modeling.  My chance of becoming infected right, is—it depends upon the prevalence of the disease in the population, let’s say.  But so the way I would think about it is that you have a—so instead of having healthy, sick, healthy again.  Right.  Which is—no tuberculosis.  tuberculosis, no tuberculosis.  Since the risk of subsequently becoming sick again depends on having been sick previously, I would have a state called post sick.  Or you know in this case, previously with active tuberculosis.  So the risks from previously sick with tuberculosis to becoming, having active disease again can be different because it’s a different health state.  And that’s exactly an example where the homogeneity assumption is violated. Not all people who are healthy right have the same risk. Those who have prior disease have a different risk and so we need a different state in the Markov cohort model context to capture that. So that’s for relapse or reinfection or these sorts of things in the context of TB that would be a relevant way to do it.

Paul:  It might be worth just mentioning a tunnel state.

Jeremy Goldhaber-Fiebert:  Yes.  So in tunnel states, the idea behind a tunnel state is that it’s a way to capture so let’s say you’re on treatment for—you have a monthly model right?  And the idea is I enter treatment and treatment is six months long.  So basically the way you think of tunnel states is I have sort of six states: treatment first month, treatment second month, treatment third month, treatment fourth month, fifth month, sixth month.  Then I’m off treatment.  And so you can’t just have one treatment state because if I enter treatment at a different time, I’m different than the people who have already been on for five months.  So—in the sense that the people who have been on for five months are only on for one more month and then they need to stop.  And so the tunnel state allows you to kind of keep track of things delivered across cycles in a model that are not cohort time.  Right?  So time on treatment and what you can then do is for example, suppose that the first month of treatment involves some additional drugs that are really expensive and the second month of treatment is different.  Now you can give different costs for different places in the tunnel. The idea of the tunnel is that you enter at the beginning and you exit at the end other than via death and that all along the tunnel, different things can happen to you: the tunnel can be wider or narrower, you have a greater chance of dying, a smaller chance of dying, higher cost, lower cost, greater quality, less quality, etc.  So if you imagine treatments that are different along their six month or eight month or twelve month course you know so that we use this in sort of therapies for hepatitis C that we’re modeling where different you know drugs are given for some chunk of the time.  There are these lead in periods for some drugs and different drugs for some chunk of time and different drugs after that.

Paul:  We appreciate.  This is a lot of material to try to cover in just one course and in fact even one semester.  It’s hard to get by, but I think it’s a great orientation and gives people some idea of what the words are and I see you had a last slide that gave some additional references, right—

Jeremy Goldhaber-Fiebert:  Yes.  There are some additional references at the back. These are what I would call classic texts or classic articles on decision science and decision modeling that were done generally in the US.  There are additional newer books that have been done and there’s some really nice books that have come out of UK relatively recently including those by Andy Briggs and colleagues in the UK, and other books that have been published.  This is by no means an exhaustive list but a good place to start and read and enter into the literature a little further.

Paul:  I’ll also mention that Mendhol Singer gave two lectures and I believe also we still have on the cyber seminar—they’re still on the cyber seminar archive about how to use Tree Age, that specific software.

Jeremy Goldhaber-Fiebert:  Fantastic.  That’s a very useful—that’s very useful and those complement the tutorials that Tree Age itself provides with it.  So I think that’s excellent.  Thanks again for giving me a chance to talk to you all.

Paul:  So Heidi has some routine announcements.

Heidi:  Yes, for everyone, thank you for joining us today.  As you leave the session today you will be prompted with a feedback survey.  If you could take just a few moments to fill that out, we would definitely appreciate getting your feedback on today’s session.  We also have the next session in this series which is scheduled for next Wednesday, October 31.  You should have all received the registration information in your e-mail today.  That session is Introduction to Effectiveness: Patient Preferences and Utilities.  We hope you can join us for that session.  Thank you everyone for joining us today and we hope to see you at a future HSR&D Cyber Seminar.  Thank you.

**[End of Recording]**