CHI_April6_5pm_Roman_Ps

**Transcript of Cyberseminar**

**HERC Econometrics with Observational Data**

**Propensity Scores**

**Presenter: Todd Wagner, PhD**

**October 23, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact HERC at herc@va.gov.*

Todd Wagner:  I just wanted to welcome everybody today. The Cyber Seminar, it is part of our cyber course so, econometrics. I am Todd Wagner. I am one of the health economists here. My voice is back to normal if you were here two lectures ago. I was just in the beginning of laryngitis. Thank you all for your patience. 

Let me get started. Just to give you a heads up on where we are headed for today’s lecture; I am going to talk a little bit about the backgrounds on assessing causation because that is hopefully why you are interested in using propensity scores. 

I am going to define what we mean by propensity score. How you are going to calculate it. There are different ways to calculate it and use it, and the limitations of it. There are definitely some important limitations. If you hang out for three and four, please also hang out for the last part as well. Jean Yoon is my co-conspirator today. If you have questions, Jean is going to be monitoring the questions and answers. Jean, feel free to jump in if you want to make clarifications or have anything else to add. It is always good to have a team doing this.

Hopefully, you are interested in understanding causal relationships. Whether you are using observational data or if you are doing randomized trials. Hypothetical questions, you might be interested in red wine. Does red wine, drinking it affect health?  Perhaps you want to justify drinking more red wine. Or, perhaps you are interested in whether a new treatment improves mortality. These would be examples of things that you might be interested in asking. 

The randomized trial provides a methodological approach for understanding causation. By that I mean is that you are implicitly setting up the study to sort of follow this treatment practice. You are recruiting participants. You are randomly sorting people. It is not their choice about whether they are getting A or B. It is, I think sort of a random coin flip is determining it. Then you are following those folks. You are looking at the outcomes. You are going to have this outcome Y. 

Often, we will use the terms Y. You will see Y throughout as our outcome variable. You will see it throughout many times regression where Y is on the left-hand side. That is what we mean by the outcome variable. What we are really interested in sort of the difference between the outcomes for groups A and B.  The randomization provides a very nice way of controlling for what is happening here. We know that the only way that they would be different is through the randomization, that the treatment assignments.

The expected effect of the treatment; if you were to go back and say what would you expect?  You can even go back a slide. I am going to use my little pointer. I seem to have lost Heidi, my drawing tools.

Moderator:You have to click on draw at the top of the screen.

Todd Wagner:That is, and thank you.

Moderator:There you go.

Todd Wagner:What you might end up saying this. What is the expected effect here?  When we talk about expectations, it is just a weighted average. What we might be interested in just is what is the weighted average between treatment group A and B. You are going to hear throughout this talk the idea of expectations. I just want to make sure you understand. This is another way of writing that expectation. The expectation of what we expect the treatment to be is the difference for the groups A and B. That is going to be your expected outcome. In this case it is just the average of A minus the expected act and effect of group B; the mean difference, the weighted mean difference. 

You can use it and....  not two weeks or lectures ago; I introduced the idea of a formula. Hopefully that was not alien to you. You can use this same formula here. This would be a very simple way of analyzing the data in a linear regression type framework. Where you have got A as the intercept; B is just the effect of the treatment types, the random assignments. E is your error term. You can, in this case as we think about trial as being a patient level trial. The I is the unit of analysis for a person. This is quite a simple framework in part because of the randomization really helps here. 

You obviously can expand this model to control for baseline characteristics. One of the things that we often do in clinical trials when we are running them is do things to ensure that the randomization happens correctly. There is a block randomization and so forth to make sure that we are getting the balance as we are proceeding through the trials. But you can find times where there is imbalance. 

You can control for those baseline characteristics where you see imbalance. In this case, I have just changed that model to include X, which in this case you could think of as a vector or a baseline predetermined or a characteristics that were prior to randomization. You would not want to take things that were after the randomization in part because those could be effected by the randomization itself. The assumption here is that the right-hand side variables are measured without noise. They are considered fixed and repeated samples. If we do this time and time again, there is going to be no noise in them. 

What I think was probably a more important assumption I strategy there is no correlation between the right-hand side variables and the error returns. Again, here is our expectation notation here. We expect that covariate X right here; and the error term U, to have no variance and have an average of zero, a weighted average of zero. Now in a clinical trial where we are actually flipping a coin and randomly assigning people that is actually quite a normal assumption. Because how else could if it is a good randomization could you violate that assumption. If these conditions hold we say that the beta, the effects of the treatments. What we go back to our line there is an unbiased assessment of the causal effect of the treatment on the outcome. 

Boy, randomized trials are nice. They are very simple but observational studies are not so. You may have times where randomization is unethical, and feasible, and practical. Or not scientifically justified. The other situation that I think of and probably this falls into the unfeasible or practical. It is the relative comparative advantages of big data. There are times where it is very easy to look at our VA data sets and pull huge samples that have – sort of to address the questions that were interesting. But it is not randomized. A whole host of other questions come into play. I am going to stand on the shoulders of giants here. This is Matt Maciejewski’s and Steve Pizer’s work. That was the Cyber Seminar. You could actually see that back in 2007. I am going to use this framework because I like the way they presented it. You have got in this case keep an eye on that zero or that circle in the middle, the sorting. Because you have got patient characteristics that go into this. That provider characteristics that go into this. 

The sorting is not randomized. This becomes a really important feature of observational studies. You have got the treatment group and the comparison group. But it is a choice. It really is not a randomized coin flip. That is going to create all sorts of problems when we are trying to compare outcomes for these two groups. Now, if everything is fully observed. If you had a crystal ball and you – or you are the NSA. You could fully observe everything and keep track of all of the characteristics. You would fully understand the sorting process. You could fully control for it. 

I say in red here that never really happens. The other way you could – it. Christine, I think it was two weeks ago talked about there have been structural models. Like the Heckman selection model, a very structural models. It is about people placing large assumptions on how they think that sortage should work. Then you can – you can back up these – your outcome differences. Again, you are relying on assumptions there. 

But in reality we never fully observe everything. Maybe you observe and sorting without randomization. Maybe you observe some things. These things are what I am going to say is unobserved characteristics. Maybe you would think that teamwork or provider communication, or patient education is important. You do not observe them. But they are only associated with the outcomes. In this case, you could have a fixed affect that would perhaps be able to take out that effect. We often include in our statistical analysis a fixed effect for the facility that would keep in line things that were fixed at the facility level and remove those from the analysis. Fixed effects are typically very useful. We are not going to talk about a whole lot about fixed effects today.

The other thing that you could have. Well, this is probably the more problematic situation is where you have. What we did here is just added a new line. What I am going to just...  It is just this line here. Now, we have got these unobserved characteristics that are both influencing the outcome and the sorting process. We do not observe them. These become really problematic. We know that and what this means is that the sorting is connected to our outcomes then it is connected to our error terms. 

If you remember back to prior talks. If this biasing our estimates of the treatment groups. The causality is not identified. We have always talked about it in observational studies. But causation or a correlation does not equal causation. This is the key reason why. Now what I am not going to talk about here is the idea of instrumental variables. That is going to come up in a future Cyber Seminar. But just to sort of lay the groundwork. 

Sometimes you have these exogenous factors; whether it is a law. Maybe it is a change in price in one state versus another state. Or a change in taxation rights across states that affects the sorting but otherwise does not affect the outcome. That is the idea really behind instrumental variables. That is in green here. It gives you insights in the causal relationships as it relates to the exogenous factors. Again, I am not going to talk about it here. Let us assume that we do not instrumental variables. What we are really concerned about here is this unobserved characteristics that is affecting the sorting.

Let us move into the propensity score because that is...  I will define this. A lot of people are interested in how do we control for the best selection. The propensity score uses observed information. I should probably underline that because it only uses observed information. Think of the information as being multidimensional. You could have a whole host of reasons why people choose to do different treatments. It could be multidimensional. What it is going to do is it is going to calculate a single variable. 

What we think of as the score. The propensity score, the score is the predicted propensity to get sorted. Typically what you will hear in the link and the literature is people say the propensity to get treatment. Because what we are typically interested in is this idea of this treatment group here to the comparison group. We are using and we are relying observed information to get a sense on that sorting. You can think of it. 

The score here is that we are getting back to this idea of what is expected treatment effects and the propensity score is the probability of getting a conditional on X. typically what you will see time and time again in the literature is people will use a logistic regression model or perhaps a probit regression model to say what is the odds of getting treatments?  Then you will put all sorts of things into their – to their logistic regression to create this predictive probability. 

What it is. What I think of it is at least is another way to correct for observable characteristics. There is no way that it is making anything to do with unobserved unless you are willing to make assumptions. I always want to caution people to say it is not a way of handling unadjusted characteristics. Some people say well, you are looking at controlling for selection here. It is really based on the observed data that you have. 

The only way to make causal claims is to make huge assumptions. That assumption really is this issue of strong ignorability. To make statements about causation, you would need to make assumptions that you have captured everything that is important to the sorting process and fully observe this. In all of the studies that I have worked on, it has just never been a claim that I have been willing to make. 

It is one of those questions on when there is smoke, what you will find in the future is that there will be mirrors. People looking back and saying that was a wrong assumption. Do not use smoke and mirrors. I would say just it is a hard assumption to make. It is very similar to saying things are missing at random again. That is typically not the case. It is also saying that it is equivalent to saying that you have observed everything. 

Let us move into calculating the propensity score. I think most of the people here want to hear about how to calculate the propensity score. How to use it. Then coming back, I will sort of finish up and remind you about the limitations downstream. Let us say you want to calculate a propensity score. You see that one group receives treatment and another group does not. You are going to use a logistic regression typically. It could be a probit, if you have different reasons for using it. Typically there is – and what I find is that the field of economics uses probit which assumes a Gaussian distribution and that the field of healthcare uses logistic, which just assumes a logit distribution. They are very similar. 

But typically people use the logistic regression and then estimate the predicted probability of the person receiving treatments. That predicted probability is the propensity score. What variables do you include in your model?  There is a great paper. A lot of the work here has been done with simulated data. Because you will see at least in the Brookhart paper. Then I am going to talk about one later by John Brooks. When you make your own data you can build in the assumptions you want to test. It is very nice to be able to say what – and how far can you push this?  If it is just a data that you are observing you do not really understand the data generating process. In this case, the question is what variables to include. 

You want to include variables that are unrelated to the exposure but are related to the outcome. If you have variables that are instruments, law changes. Things that effect pricing that would affect just the sorting process or your exposure. You actually do not want to include those in your propensity score. This will decrease the variance of the estimated exposure without increasing the bias. You have got to be really careful about it. Because you are going to inflate things and especially problematic for small – from all small trials. 

Going back to this diagram. I apologize that – I am going to walk you through it. I have changed this but we could not upload the changes because of the bandwidth. What you do not want to include is in the red, the exogenous factors. What you do want to include and it should say observed characteristics and not unobserved. Because if it was unobserved you could not include it. But if you have observed characteristics that affect both the sorting and the outcome, those are the kind of variables that you want to include in your propensity score. Maybe it is gender, or family history of the behavior. You could think of all sorts of things that effect both the sorting and the outcome. You want to include all of those in your propensity score. 

Like I said, you want to exclude variables that are related to the exposure but not to the outcome. If you include those, like I said, you will increase your variance of the estimate exposure without decreasing your bias. What Brookhart found is that in small studies and the, sort of the thumb rule here is studies that have fewer than 500 people. What you are going to do is just inflate your variance estimate to the point where it is actually creating noise that is very problematic for you. 

I will also say that this. I am going to skip that slide. I will also say that people do not spend enough attention on what variables go into propensity score. Time and time again I am on studies with large teams. People that are just create a propensity score. Then let us go to the second stage. We will look at the effects on the outcome. I think you should spend a lot of time really understanding that propensity and the balancing. I am going to show you diagrams here about how do you think about balance?  How do you use a propensity score?  

You first off; you can understand sorting and balance. It is probably the reason we create this score. Remember sorting is multidimensional. If you are interested in things like why do people smoke?  There is a whole host of reasons why people smoke. The nice thing about a propensity score is it reducing this multidimensional nature of the data to a single dimension that is very easy to grasp, and look at, and understand. It is a very simple way of reducing the dimensionality to a single dimension. You can also use it to adjust for a variance that you are interested in getting rid of. But I do not think – and getting back to that first thing. I do not look and think enough people pay attention to that first bullet; and really understanding the sorting balance. 

Was there a question?  

Moderator:No, not right now.

Todd Wagner:  Okay, it is just my feedback. Jean, feel free to jump in if you have – and people ask questions. Here is an question that is – that is an example. Are surgical outcomes worse when the surgeon is a resident?  Now, it is a question that a lot of people would be interested. Many people have family members who get surgery. The question would you rather have an attending or a surgeon or resident do this, the surgery?  Now the resident assignment. You can think about the surgical team here may depend on all sorts of things that are observed and probably some stuff that is not observed. My, in talking to surgeons, often we talk about if the patient is extremely high risk. Often you will see the attending doing it and not the resident. There are often questions about availability of the residents. 

Now, we have these work hour restrictions and so forth. Sometimes it is a low risk patients, but there is no resident available. You might also be wanting to do, for example, a new technique. Maybe you were trying to do minimally invasive vein harvesting on this patient. The resident does not have the skill. You will say now, we will – the attending will do it this time. But there are also these things like local culture that really matters about who gets to see what procedures. Now, I think we also realize that is beneficial for society to train new people to do these procedures. Otherwise, we are going to eventually run out of people to do these procedures. We need to have residents trained. But there are also questions about what are the right patients for doing the training?  

You could look at this?  This is a – this a resident assignment. This is from a clinical trial. We were doing a secondary analysis of it. We were trying to understand which patients were getting assigned to the resident?  Let me just walk through it. This was a heart bypass trial. You see things that I did not – the red lines. Things that do not matter. It did not look like age mattered at all. In fact, the adjusted odds ratio there was one.  But it also does not look like that the number of graphs matter. That they were – residents were equally likely to be assigned to a multi-vessel bypass as it was to a single vessel bypass. But there were some other differences here. Canadian functional class is a way of identifying sort of angina severity if I am remembering correctly. You get to see that does matter. 

The people who have the worst angina frequency or severity are more likely to be assigned to a resident in this case. Then you have some other information that really matters here. What also really matters, notice those site variables. This is – it is a multi-site trial. Site 5, a huge odds ratio; much more likely to use residents. What is going on there?  Maybe it is availability. Maybe it is culture. We have got all sorts of things happening at that level. It is sort of – there is also issue of endovascular harvesting when we are doing that. We are much more likely to use an attending. Clearly, if a multidimensional difference between residents and attendings. You might say well, what about a propensity score?  The propensity score is very nice for understanding the balance. You can actually then – 

Jean Yoon:Todd?

Todd Wagner:  Yes?

Jean Yoon:You could go back to that slide. There are a couple of questions.

Todd Wagner:  Yes?

Jean Yoon:There the first question is how are you defining association of variables of treatment and outcomes?  It looks like you were using the physical significance. He wants to know is it based on a particular test or model, or simply clinical or subject matter knowledge?  Like how did you pick these variables that went into your resident assignment equation?

Todd Wagner:  That is a...  In this case, it is both right. There is some information out there that I, as an economist. I am not a clinician. I am not going to know the detail on creating a functional class and why that might matter. Because we were doing this in a clinical trial setting. We had data from the case report forms in the clinical trials. We are able to include that. Now, clearly if you are using just administrative data, you might not be able to have that variable. That might be the – an admitted variable or an unobserved variable. You are trying to take as much information that you might think would affect the sorting. 

Try to understand the sorting. It is both a clinical intuition as well as sort of data availability. The other thing to keep in mind here is depending on your sample size, there is very little penalty. If you have got a large enough sample to adding another variable. It does not really take anything away from the model except a degree of freedom. Now, if you are working with 20, 30, 40 cases, a very small data set. Clearly you are going to have trouble estimating a regression with many covariates. But in this case we are talking about 1,000 patients that we were able to estimate this. That is a great question.

Jean Yoon:Another question asking about how should the R-Square from a logistic regression be interpreted?  Is it necessarily bad to have a low R-Squared?  

Todd Wagner:  I do not even look at the R-Squared in the logistic regression. In fact the logistic regression is not going to give you an R-Squared estimate. What it is going to – much more is about the model fits. Let me move onto the diagrams if there are no other questions. I think we will sort of highlight why and what I mean by that. Now, if you were thinking about model fit. Model fit matters, but I think the intuition there is much more about how much overlap are these populations between the treated and untreated. Another great question, _____ [00:22:56]?

Jean Yoon:Yes. When excluding variables that are related to the exposure but not to the outcome, what is the exposure?  Is it the sorting variables or the treatments?

Todd Wagner:  In this case it is the sorting variables. If you are interested in – I should give you an example. Let us say you are really interested in smoking and the effect that smoking has on mortality. You would not want to put in your propensity scores things that effect the sorting. For example, tax rates because that is something that could according to Brookhart, inflate your various estimates. You are really interested in things that for example, that would effect both the sorting and the outcomes. Did the parents smoke and years of education?  Things like and along those lines that could affect both. You are trying to get a handle that and observe sorting.

Moderator:This person wants to know. What are the outcomes?  Is it the treatment decision or something downstream?

Todd Wagner:  There are two outcomes we should be careful of. One is the propensity score is trying to measure. What we say is the outcome and the propensity model is the sorting and the treatment. Are you treatment A or treatment B?  The true outcome that you are probably interested in is things like mortality, readmission, things downstream. Keep in mind that the propensity score we are going to talk about is for collecting, and creating, and calculating the propensity. That eventually, we are going to use that in analysis when we are looking at outcomes where we are trying to understand the sorting there. But in the propensity model itself, what we are really interested in assigning is or looking at that is the sorting process. Do you want to say anything more about that, Jean?

Jean Yoon:No, that is great, thanks.

Todd Wagner:  Okay any other questions?

Jean Yoon:No.

Todd Wagner:  Perfect. The question about R-Squared and model fit is really one that I would say is much more about model fit and common support. The goal really is if you wanted to say something about the sorting process. Ideally you have these populations that look very similar on observed characteristics. You are going to get this idea that you can sketch out. In the red line is one of the treatment groups. In the blue line is the other treatment group. These are all their predicted probability of getting the residents versus attendings.  

There are different characteristics. This is just the predictive probability. One of the really nice things I talk about, the multidimensional nature of propensity scores by calculating the propensity score using the probability, you can then sketch it out to the two different groups. That is just what I have done here. You can see also that there are people outside the common supports. There are people where there is – where we do not have overlap. If you wanted to come down here, you could say well really you have got this idea of there is no one in that one group that is that way. There is really no one up here that is that way.  

When you say something about R-Squared or the person asked a question about R-Squared, you really are asking a question about well how well do we understand the sorting here?  I would say well perhaps in the middle we understand the sorting reasonably well. I have a hard time making a judgment on whether that is perfect. But clearly there are people that fall outside of those ranges where we do not have a lot of data. They do not overlap. I have actually seen – one of the nice things about propensity scores. I have actually seen people come back and say based on the propensity score we do not feel like we can continue with this analysis. Because there is no way we can make these groups look similar. That is another advantage of propensity scores.

Here are three scores. My eight disappears.... So 7A.  A is this one up here. Let me write it in. That is the one we were just looking at. Now, these are – B and C are two other scores from actual data. You get to see differences about how similar these groups are. I have a poll question for you. Move onto that. Do any of these distributions concern you?  Now, you can say well, I apologize. But I cannot show you both the distributions and this.  

You could say A, D, C, none of them, all of them. Do any of them concern you?  No one has chosen B. some, I think because if you chose B, you are also choosing all of them or none of them. We have just a couple of people choosing none of them. Some people are very concerned about A. we have a couple of people concerned with C. one person is concerned with B. then it looks like 20 percent of the people are concerned with all of them. Okay.

Now, this is and I am going to end the poll, if I can. We close out, thank you. Let me move and we will go back. Describe where they came from. C is actually data from a randomized trial. Even if you do a randomized trial you might have slight imperfections. Because methodologically, we did the flip of the coin, we might have more faith in the overlap of those distributions. But I just want to highlight that even in a randomized trial it might not be perfectly aligned. 

Now, B is actually a different group. This is comparing people in a trial to the general public who did not enroll in that trial. This is, again the same set of people in group A or in B. But that what it is comparing is those people who enrolled in that bypass trial to all people in VA who has got a bypass. You might say that we look a little bit different than we actually public – the paper shows that where we looked a little bit different than the general population. 

Jean Yoon:Somebody wants the clarification about can you explain what is on the axis in those graphs.

Todd Wagner:  Okay. You are right. This is a common way for me anyway to display information. This is just the density if you will. It is what – this is known as a kernel density diagram. You could use a histogram to present the data across the two populations. I do not like histograms because they bending the data. Depending on the bends that you create, it is not always as intuitive. What this is – think of each person as being a snowflake. Instead of being a dot than they would. If you let the snow fall on the x-axis. If it was purely just a dot, you would have a flat line of people. 

My hand is not very flat here. What kernel density says; well, instead of being just a dot, we are going to make things that have a little distribution. Each person gets the distribution. Now, that is known as the Epanechnikov distribution. You can have the other distributions. Now you can let the snowflakes again. When these folks fall to the ground; and instead of this, one falls there and one falls here. 

You might have and then you traced the outline of their distribution. You end up with this kernel density diagram. For me, it is just a very handy way of understanding distributions and displays of data even though that is sort of the historical way of doing this would just be a plain histogram. Think of it as the same idea without bending the data necessarily. But, a great question; any other?

Jean Yoon:No.

Todd Wagner:  I apologize. All of this stuff for me is done in Stata. That is because of what I use it. This stuff is really easy to do in Stata. I apologize if people are using other programs for this. This is not common. Alright...

Jean Yoon:The x-axis, this is all of the variables that go into your sorting equations?

Todd Wagner:  Yeah, and that is the probability. That is right and that is the probability of treatment. The probability of assignment here. We have people – you will notice down in C anyway, that you will have people – you have people who are predicted out to one and from zero. Whereas when we are looking at for example, in the – up here in A. We have much less overlap, right, in probability. 

Now you might actually have done times that I – and done times where it is very hard to get the balance. That is why I think it is important to spend a lot of time understanding your propensity scores and what is affecting your balance. I have sort of given this away again. What would happen if you used the propensity score data with a randomized trial?  Well, you can – and this is the thing. They share a common support. You can see it. 

Not always does a randomized trial because of a sort – the assignment is random. It does not mean it is always perfect. You can actually see slight imperfections. That is – if you went back to the – one of the first slides where we have this idea sometimes you want to control for baseline covariates even in a randomized trial. You could do the same thing with a propensity score. Instead of using those baseline covariates, you could have a randomized trial that used a propensity score. 

Let us talk about using the propensity score. Now, you have hopefully got the idea about how to make it. You have got this idea that you have got. You see these two groups. One is getting treatment A and treatment B. if you are using the smoking example; you have got these two groups. One group is the smokers and the other group is non-smokers. You are creating a logistic regression. You are predicting the probability of smoking based on observables. There are many ways to use this propensity score once you have calculated it. Perhaps you will want to compare individuals based on similar propensity scores.  

This is in many regards a matched analysis. You could also stratify. You could say well I want to conduct subgroup analysis on similar propensity score groups. That would be a stratification. Perhaps I want to include it as a covariate in my regression model on the outcomes. Maybe you were looking at the effect of smoking on mortality. That mortality regression, you could include it as a covariate. Perhaps I want to weight the regression to take it and place more weight on people with similarities in their propensity score. Then in some sense you can use three and four together and get this idea, if it was a doubly robust. I am going to walk you through these. Matched analysis; the idea is to select controls that resemble the treatment groups in all dimensions except for treatments. Now this is all observed dimensions. It is not unobserved. I will get to a limitation about that later on. You can specifically exclude cases and controls that do not match. 

Now, one of the challenges in doing a matched analysis is that you will -depending on your sample size, you could have very people in the end who end up matching. There are different matching models to run analytically. I do not – I have not seen. I am not a person who spends a lot of time doing propensity scores. I have done a number of models. I have seen the sort of trend is to move to doubly robust. I will talk about, a little more about them. I am not going to spend a whole lot more on the matched analysis except to say there are different ways of calculating the match. 

One is this idea of statistical. Think of yourself in a community. Perhaps you could take a statistical nearest neighbor. We will rank the propensity score and choose the control that is closest to that case statistically. You could also think of this idea as a caliper. You could identify people in the common support that overlap then randomly draw from within. That is the idea of a caliper. There are different ways of doing these things.

Perhaps you just want to use this propensity score as a covariate. This is a -the idea of maybe you have created a whole regression model to look at mortality and bases smoking. You have created your propensity score on smoking. You are going to put everything on your right-hand side variable on the smoking model and mortality model. You are also going to include the propensity score. 

This is the one case where it seems like the propensity score has little advantage. Because you are in some sense already capturing those observed characteristics in your – in your model. Adding the propensity score does not add a whole lot. Now you could say that well it adds a sort of banal linearities and it allows the variables to be a little bit more flexible. That is true. It provides a little bit of flexibility in the functional form. You might also say well, I have a got very tiny sample size. I want to control for a number of covariates. I worry about my sample size if I have a lot of covariates. 

You are right, that may be a propensity score model is helpful there. There are some reading to do there. But this is – we do not see a lot of people. I do not see a lot of people using propensity score as a covariate in a multiple regression model. What I see a lot of people doing now. Jean, feel free to jump in. you have been spending more time on these, too. This idea of a doubly robust estimator. What a brilliant name. I have to say that because it is what we are hear time and time again is that it is trying to account for and allow you to have misspecification in two areas of your model. But I just want to remind people it is only on observables. 

What we are still going to come away with when we get back to the limitation section is that we are only doing these that observed to this data. Here are the steps for a doubly robust estimator. Because we talk about expected treatment effects, I just want to make sure you have that in mind. You are going to fit a logistic regression for the treatment conditional on your baseline values. You are going to predict from this regression and get the estimated propensity – propensity score. Let us say it is at the patient level so that [inaud.] subscript for your patients. 

Now, we are going to fit a regression model for an outcome of the baseline variables for the treatment group. Well in this case it is only group A. obtain the predicted values for that group. We are going to fit the same regression model for the group B. this is – the first one, let us say it is smokers. The second one is non-smokers. We are going to obtain to predict the values for the whole sample. There is a nice formula. It is relatively straightforward that where you use the propensity score and the predicted values of A from number two up there. The predicted values from B from number three up there into a formula. 

Now, that citation I give you there shows the formula. It would not fit on the screen and make sense to walk you through it. Then what you are going to do is you are going to bootstrap the standard errors. In some sense what it doing. I want you to sort of have the intuition here is it is taking the predicted values from A and the predicted values from B. instead of just giving you a mean difference; it is using the propensity score to give you a weighted mean difference. 

Now, the reason that people like propensity scores or this idea of doubly robust is they say that if you have assumptions built into your, the way you are estimating your propensity, this allows some flexibility there, and gives you some leeway there. The same as if you – maybe you misspecify the true relationship between age and your outcome variable. This gives you some flexibility there. It is becoming – it, a more popular model. It has gained favor because it is allowing some misspecification; either the regression or the propensity score models. Here are sort of the classical statistical text on that if you are interested in reading more about these doubly robust estimators.

If there are any questions about the different ways of scoring it, I am happy to answer those. I am going to move into the limitation section.

Jean Yoon:No, there are some questions that can wait until the end.

Todd Wagner:  Okay. I will say that doubly robust, it has become so popular now that there are built in functions and stata for estimated doubly robust estimators. You do not actually have to go through that four step process. You might want to do it once to understand what is going in and making the analysis. But you can actually now just type in VR and specify your regression. Then it calculates your doubly robust estimator. The other reason I would say to be careful of that is it does not allow you to spend a lot of time understanding your balance and your shared support. 

The question here to keep in mind is do the unobservables matter?  Let say you have a data set on smoking. You have a lot of information on why these people smoke. The question still regards is everything we have done with these propensity scores focus on observed characteristics and really not the unobserved characteristics. Now, you could make the argument that they are proxies. But I can also make the counterarguments that we are still missing something that is critical and important there. 

That is an admitted variable that is going to bias your regress – your regression results in the outcome. It seems very practical to me that it is improbable that we fully observe the sorting process. We are building in this bias. Whether you are using multivariable regression or multi-vary – including the propensity, it is still biased. We need some other ways to fix that bias. You can think about instrumental variables being one method. Christine, and she will talk about that and I believe it is the next lecture. 

You can say that fixed effects, if you are willing to make the assumption that the unobserved is fixed over, let us say time. Or you could use a randomize trial and see if there is no other way to handle it. Do the unobservables matter?  Often I am not – and they to be fair. Jean, I would love your take on this. I see different disciplines have different cultural willingnesses or propensity to accept propensity scores. In economics, they are not used widely. They seem to be used more in sort of clinical health and health services. What is your take on that, Jean?

Jean Yoon:Yeah, I mean, I agree with you. I also think whether you use propensity scores or some other method, it depends on lot on practical reasons. It can be difficult to find an instrument. If you cannot find one then it might be easier to just use propensity scores. 

Todd Wagner:  Yeah. The other thing I...  Propensity scores have sort of entered the common vernacular at least for clinical outcomes and health services research. If you put it in a method part of your paper. Many journals are actually asking people to go on multi-variable and do propensity scores. It has become that common. Whereas my sense if you came on and said we did an instrumental variable regression. I think they would still be with a furrowed brow trying to understand what you did. Would you have the same take on that, Jean?

Jean Yoon:I would agree with that.

Todd Wagner:  Okay, thanks. One question, this is the Brooks and Ohsfeldt, they also use this simulated data here. One of the questions they ask which I think is a very interesting question is perhaps you are left with this limitation which is just that the propensity score is great. It is equivalent to multivariate maybe with some additional things going on. But they ask this question of maybe if you are forcing everybody to look the same on observables, maybe their unobserved features are actual getting worse?  Now, the only way to test that would be to make up data and exclude variables that they know to be observed that they made up. 

That is exactly what they did. John Brooks is a health economist at the University of Iowa. Ohsfeldt used to be there but now I believe he is at Texas A&M. They used simulated data. They showed that the propensity score models can create greater imbalance among unobserved variables. This was the first time at least that I was aware of it.  

This is Health Services –  I think it is either; it might be even last month or two months ago where they are showing that if you are forcing everyone to look the same on observables, you are actually – could be creating greater imbalance on the unobserved. I do not know that we fully understand the ramifications for this. Whether there is going to be a backlash and moving away from propensity scores. I think it is worth understanding that the propensity scores are not making things. You are not fully controlling for your observe. It is a very interesting provocative. 

They actually title it. I love the title, which is Squeezing The Balloon. Because it is something we all do. When you grab that balloon and you squeeze part of it assuming it does not part – is the air just goes elsewhere. You might make those where you squeeze look very similar. But you have just exacerbated the divergent elsewhere. I think we are starting to see a little bit of pushback now on what exactly is in the propensity scores. How do we think about balance?

Jean Yoon:There is a question about unobservables.

Todd Wagner:  Sure.

Jean Yoon:It says is not the assignment of treatment A versus B depending on observables and clinical symptoms?  Then why worry about unobservables?

Todd Wagner:  That is a great question. There are many times whether it is a clinical condition. Let us say you are – I will be try to be very clinical. Instead of using my smoking example. I typically use a smoking example because I think we often think about and we can see people smoking. We know that they smoke from a sort of multitude of reasons. Whether it is family history, or where they grew up, or their income, or their education. We cannot always control for all of that. 

There could be generic factors too. But let us even take a situation that is much more clinical. Let us say we are interested in post-operative A-fib, atrial fibrillation. You have got these, the heartbeat irregularities. You see some people who were assigned prophylactic treatments whether it is amiodarone or beta-blockers. Some people who were not. Well, there could be a whole host of reasons why those people, some were assigned and some were not. 

Let us just sort of mentally run through what would make a difference in assignment of prophylactics. Maybe it is the patient was better informed; had a better education; had a parent who was a healthcare provider. When they started sort of pushing it in this direction. Or maybe it affected where they got their surgery in the first place. You could also make a case that perhaps there are clinical. But the hospitals do not only always have the same sort of clinical foundation or knowledge. Some place might be better providers specifically when it comes to this; and have more knowledgeand doo a better job and thinking about patients at risk for afib and assigning this prophylacsis. It is hard in my mind to come up with a model where we can capture all of that detail and fully observe it. If we cannot do that, then our assessment of the effective prophylactics treatment, what we observe is biased. 

That is the trick. We get back this idea of we see this association between prophylactics and A-fibs. But we do not know the causal relationship. Now, if you are only interested in associations, that is fine. You are happy. Go ahead and publish. But I think most people are more interested in the causal understanding and the causal things. Because they want to change it. They want to be able to improve care. They want to be able to...  In my opening slide, I talk about drinking more red wine. I was sort of being tongue and cheek there. But the idea is that what you are really trying to do is improve the system. You are really interested in the causal relationships. 

Sorry, a long winded answer; hopefully that answered it. Any other questions, Jean?

Jean Yoon:No, just some for the end.

Todd Wagner:  Okay. This is a – it is a variance of getting back to this idea of exacerbation of imbalance. It is a very interesting idea that things could get exacerbated by using a propensity score. Just to summarize where we went through. Propensity scores, they offer another way to adjust for compounding by observables. But the key there is the observables. They are actual very useful for reducing the multidimensional nature of compounding. I would really encourage more people to think about that first stage where you are just trying to understand the sorting. You could plot your – what we had on the slides as those kernel propensity diagrams. To understand how much overlap are between these groups. 

Now, if you were doing a trial where they actually do not overlap at all or there is very little overlap. That could give you huge reasons for pause to use them. Or to even make any comparisons whatsoever. Then keep in mind that there are many ways to implement propensity scores. There is a growing interest in doubly robust estimators. They are actual quite straightforward to calculate as we have shown. 

The strengths like I would say.  I am not a huge fan of propensity scores. Maybe I am showing too many of my cards here as a poker player. They allow one check – one to check for balance and balance between controls and treatments. Without balance, this idea of an average treatment effect or this expected difference can be very sensitive to the choice of estimators. 

That is a great paper to read if you are interested in it, and looking on matching. Guido Imbens from Harvard and Jeff Wooldridge from Michigan State. There are strengths to those methods. There are challenges. I think that propensity scores are often misunderstood. I think people are too quick to jump this next bullet which is looking at the score themselves. Thinking about what is going into the propensity score?  What variables do I want to include?  There are a bunch of techniques at least in stata for understanding how you could see how the difference in the populations change when variables are included and excluded. 

I do not think enough people pay enough attention to the robustness checks. If you are getting incredibly different estimates when you are using propensity scores for multivariate regression, I think it is incumbent upon you to figure out why is that the case?  Is it because of the way you are implementing your propensity score?  Are you including things that you should be including in propensity score?  Try to figure out why that is the case. Hopefully, the situation is one where it is providing very similar results. But again, that might give you cause to say well, it – is it valuable in that case where it is just adding time to the analysis?  

Then the other thing to keep in mind. This is where we are just starting to see the edge of this iceberg is while propensity scores can help create balance and observables, they do not control for unobservables or the true unobserved selection by it. With that Brooks and Ohsfeldt, they actually might be making that worse. That might even be a bigger concern. If you are really interested in further reading, alright here is a bunch of things. The first view that I have here. If I come down to about here, it is much more about theory. It is sort of general overviews on propensity scores. 

The Brookhart one I like a lot because it talks about the variable selection and what variables go into the propensity scores; which I do not think gives enough – we give enough attention to. The Emsley of The Stata Journal, it is actually very easy to read. What I like about it is it is hands on. Like here is how you do it. That is it. Everybody can access it. Then the Brooks and Ohsfeldt one, which is this idea, is that you might be making things worse with the propensity scores actually. It just came out in Health Services Research. That is for further reading.

I think that is all of the slides I have. I will let, and probably open up the floor for questions at this point.

Jean Yoon:Okay. You have a long list of questions here, Todd.

Todd Wagner:  Alright, I appreciate your help in sorting them and answering them. Thank you.

Jean Yoon:Okay. The first one refers to the poll that you did.

Todd Wagner:  Yes.

Jean Yoon:I just wanted clarification on what was the answer to the quiz?

Todd Wagner:  There was no answer. That is the reality of propensity scores is that there is no. You as a judgment maker have to figure out which one gives you pause. Methodologically, I would be less concerned about data from a randomized trial especially one that where the randomization was doubly blind. But you still might see imperfection in the distribution. It is surely based on the coin flip. It is one of the reasons that people talk about replicating studies. It is to make sure that we get these results time and time again. 

Now, if you are between A and B in the polls, I think that you would agree that the distribution in A gives you more reason for pause than the distribution in D. but I was trying to answer different questions with those. One is between attending and resident, that was A. B was much more about trying to understand where the people in our trial generalizable to the whole population of the people getting the surgery. I apologize for not having a true answer.

Jean Yoon:Okay. The next question asks what if you have a very large sample of 200,000 people, but only one percent get treated?  The propensity model only had very little values for treatment for everybody. Is this an issue?

Todd Wagner:  Think of it from a multivariate regression standpoint. The main issue has to be is that you are comparing a – you will have a huge amount of information on the untreated group. Very precise estimates on their sort of confidence interval, if you will. Because you have such a large number of people in that group. What you are going to struggle with is you have very limited precision in your...  I think the question was about and you get a small treated group, is that correct?

Jean Yoon:Right. 

Todd Wagner:  You are going to have a very small precision in your treated group. Now, that same intuition is going to carry over into your propensity score model. You might have very little information about which to understand the selection. Maybe it is a very new treatment. Only certain sites have that treatment. Even there it is hard to understand what is going on. Or it is just a very rare treatment. 

Yeah, you might end up having this situation in your propensity score where they just do not share any common support. You might with the 200,000 people, you might not have ended up with subgroups where you say wow, I really. This, none of the people in this subgroup ever got treatment. Let us cut them out. They are not really applicable for the comparison group here. Let us say you have 10,000 people in the treating group and about 190,000 people in the untreated group. 

You might figure out that there are people in the untreated group that you can cut off because they are never even appearing in the treating group. Start reducing sample... And that will help with your noise, and your estimates, and your variance as well even though you are shrinking your sample size. 

Jean Yoon:Okay. Somebody asked was the Framingham score built using a propensity score?  If so, that is used as a covariate all of the time.

Todd Wagner:  I tried and I do not know the Framingham enough to answer that question. I could not answer you. Do you know Jean?

Jean Yoon:No and neither do I. If the person could maybe give more of an explanation about what the Framingham score.

[Crosstalk] 

Todd Wagner:  I know it is a – yeah, I just do not know enough about that. Underlying data and that was one where I would have to go back to the source document you mentioned to try to figure out exactly how it is calculated and so forth. I think we used risk covariates all of the time. Whether they are calculated by propensity score, or I am not sure.

Jean Yoon:Right. I think the Framingham score might be more of a co-morbidity measure and not a treatment and whether or not you got treatment or not. But maybe the person is right and has more information. Somebody else...

[Crosstalk] 

Todd Wagner:  Yes.

Jean Yoon:Sorry, go ahead.

Todd Wagner:  Just that there is a lot of sort of risk adjustment methods that one could include in the propensity score. Because you could imagine that they would both effect the sorting and the outcomes. You could think about the Charlson Index. I do not believe it was ever created for propensity score but you could think about you have got these different groups. You want to include them; or the Elixhauser. Or maybe you are using a big sampling. You have the HCCs from some risk adjustment model.

Jean Yoon:Okay. Somebody wanted to tell about the stata code, if you could share that especially for the density graphs and the doubly robust progression. 

Todd Wagner:  Sure. I am happy to share any of that stuff. I do not know about our current platform for sharing. I would say the easiest way is just todd dot wagner at VA dot gov. I will happily send you back the codes that I am using to calculate that and show those things. Those are very easy.

Jean Yoon:Okay. Does somebody want to ask a question about the doubly robust estimator and whether it is similar to inverse probability weighted treatment effects?

Todd Wagner:  It is similar to that. It is using – so that the mean difference between the two populations that you are comparing. Let me go back to that slide, if I may. Okay. The predicted A, so the steps two and three are used – or kind of deal with a multivariate progression. But we are not including the propensity score. Then you have also calculated it as propensity score in step one. Then you used these three pieces of information to calculate a weighted mean difference. 

It provides some – if you have got some specification problems that are present in two or three. Let us say you misspecified. You have included age of the linear variables. But really age is non-linear. This four provides some ways of handling it. It is a little bit different than an inverted probability or treatment weights. Because that would just place more people on; even if you have misspecified it. Just try to make the weights the same for those folks. A little bit different on how to do it; but, the Emsley walks you through what it is trying to do. I think it is actually quite clear on how it is doing it. 

Jean Yoon:There were several questions asking about how do you do robustness checks for propensity scores regressions?

Todd Wagner:  Yeah. I guess when I say robustness checks; I am no so meaning statistical or empirical ways of testing it. It is not like there is a Hausman estimator to test and give you a key value on the robustness of it. What I am really suggesting is that I think you are incumbent upon yourself as being analysts to sort of really kick it. If this was a used car that you were buying, you would really kick the tires so to speak. Make sure that changing your specification model whether you are thinking the age of linear or non-linear. 

It does not really matter. That the key results that you are getting are not sensitive to these types of effects. If they are, perhaps they and that is the most important thing. Maybe it is you have figured there is – from one group it is highly significant. Another group it is not very significant at all. That is often really critical information. I just when I say robustness, it is much more of the truly try to understand your data and the underlying data generated process that just sort of quickly doing the analysis and publishing the results. That is all I mean.

Jean Yoon:Okay. The next... I see several comments, if not questions.

Todd Wagner:  Okay. 

Jean Yoon:His first comment is please state out loud that Stata has recently produced the command for calculating a propensity score previously that were user written commands made available like PS Match 2.

Todd Wagner:  That is correct. The PS, score, PS Match 2 now embedded in it is this idea of DR. that is what I have been using more recently is this doubly robust estimator with the bootstrap standard errors. 

Jean Yoon:Okay. This second comment is instead of one to one matching of treated and controls, you can choose to perform one to one – one to many matchings when there are many more controls compared to treated.

Todd Wagner:  That is true, too. You could do a ten to one. We are doing a different study where we are doing an ALS trial. We are doing a ten to one. Because ALS, Lou Gehrig’s disease. It is a neurological disease that is very rare. We have a lot more controls than we do for the people who with the diseases. We are doing a ten to one. Essentially it is you take the average of the ten. You compare it to that in the person, right.

Jean Yoon:Does it give you a more precise estimate then if you do many to one matching?

Todd Wagner:  I would have to get my...  I work on that project with an epidemiologist. I am not a technical expert on matching. I do not understand implicitly if there is a statistical advantage. I am assuming there are. The reason we are doing it and why she is wanting to do it is because there are. But I could not point you to the math and say here is where it is affecting and whether ten or eleven or five or two. I do not know the exact number there; which or why we should – or she chose ten. I apologize.

Jean Yoon:Okay. This person had the final comment that Peter Austin has published many manuscripts using simulated data which compared different propensity score methods and using them for different types of effects. This person just wanted to point that out to the audience in case they needed more references.  

Todd Wagner:  Yeah. Austin, you are right, it is not a huge amount. At some point I have to stop doing this lecture. I know that today we had over 200 people on it. Because it is, the literature is growing so fast right now both in the application of propensity scores and on their weakness. That it is very hard for me to keep up with the literature. You are right that there are a bunch of people who are very – are thinking very hard about these issues. Thank you for that comment. That is great. 

Jean Yoon:Okay. It is 12 o’clock. I do not know if you would rather have people e-mail you these questions. Or whether you wanted to continue answering some.

Todd Wagner:  I have a call coming up in two minutes. I will take probably one more quick if there is – you see one more question. Then the rest I think will have to – sure, I will answer it by e-mail. I apologize for not being able to get through them all.

Jean Yoon:Okay. Is it practical to execute a matching process if there are too many cases rather than controls?

Todd Wagner:  The problem there is that you are going to end up assigning controls to multiple cases. Then you are – one of your assumptions that is going to come back to in your analysis is that you believe that you have got these independence of your error terms. You will just have to figure out if you have got people that are in the data multiple times. You will have to figure out a way of correcting for that. Because it does pose a problem, yes.

Jean Yoon:Okay. There are a handful of other questions we did not have time to go through. If you want to go ahead and e-mail Todd.

[Crosstalk] 

Todd Wagner:  Can you figure out a way of snipping those and sending them to me and their asker so that they do not have to retype them in?

Moderator:I actually already have done that. I will get those forwarded over to you this afternoon.

Todd Wagner:  Okay, thank you. You do not have to re-email me your questions. If you have other questions, feel free to e-mail me. I will do my best to answer them or point you to the literature if I cannot specifically answer them. Thank you all for coming.  It sounds like it is a topic that a lot of people are interested in. thank you again, Jean, and Heidi for all your help making this possible.

Moderator:Thank you everyone for joining us as we shut down the meeting here. You will be prompted with a feedback survey, if you could take a few moments to fill that out, we would appreciate it. Thank you everyone for joining us for today’s HSR&D Cyber Seminar. Thank you.

[END OF TAPE] 

Page | 1

Page 1 of 20