**Transcript of Cyberseminar**

**HERC Econometrics with Observational Data**

**Propensity Scores**

**Presenter: Todd Wagner**

**7/25/2012**

Todd:Hi. This is Todd Wagner. I just want to welcome everybody to the propensity scores. And people have answered or asked questions about the weather. So I appreciate that. If you have questions as the talk goes on feel free to punch in questions. It is a little hard for me, as the presenter, to keep track of those so Paul Barnett will be here shortly and he will manage the questions for us. I will try to pause as we go through and make sure that we have time to answer the questions. I know that there is – I see 122 people right now as attendees so that is fantastic. I know this is a very popular topic these days and many people are being asked by reviewers to do propensity scores. Let me jump into it. 

Here is the outline for today’s talk. I want to talk a little bit about understanding and assessing causation and why randomized trials are the gold standard for this and ways to think about confounding observational studies and what is it propensity scores are trying to do and what they are not trying to do. And then I will talk a lot about calculating a propensity score. It is incredibly simple sort of idea that is very simple to do. It is actually much harder in practice to do and create a good, valid propensity score. So I will talk about that and then the limitations will come last. 

Causality. Most of us are interested here in trying to use observational data or even randomized trials to understand causal relationships. That is a sort of why we are here in VA and what we are trying to do. Maybe you aren’t a red wine but maybe you might be interested in does drinking red wine affect health. More specifically, we might be interested in for example, does a new treatment improve mortality or reduce mortality. And you can create randomized trials that provide a venue for understanding causation. And these really are the gold standards for doing that. Here is a diagram that is Matt Maciejewski, Steve Pizer created, and I am kind of stealing it from them. You recruit participants. You randomly sort them. And that is done by a flip of a coin or some sort of algorithm for sorting them that is known into treatment A or treatment B. Then you follow the same outcome. So that is outcome Y. Really the only difference here is the random assortment. And because the patients didn’t choose whether to go into treatment A or treatment B we get a pretty good estimate of what is the causal effect. 

Through just random chance you can have unbalance. So there is a lot of techniques that clinical trial is used to make sure that they are maintaining the balance over time. But you also get that classic table one in a randomized trial paper that shows the balance. They sort of want to convince us that they have achieved that balance. 

So the expected difference of the treatment the expected you sort of treat it as the average of Y. the effect of the treatment on outcome it is just the subtraction of being treatment of group A versus group B. You might hear that the idea is that this is a mean difference. This is really just what is the difference on average being in group A versus being in group B. 

Now you might have in real life that there is some slight imbalance in your group and you can change this model if you want to adjust for this. If we think about this basic analysis in terms of equation and I know some of you don’t like equations. If you like equations here is the linear model equation that you can run. Y is your outcome, Alpha is your interceptor, B is your effect of your treatment and E is your sketaskic error term. To walk you through that – the key here is that there is an assumption implicit that we talk about a couple of sessions ago that the relationship between beta, this random assortment and the error term is independent. And many times for example if you were to follow people on observational data and ask the question does smoking or drinking of red wine affect health people choose that. So there is an implicit relationship between the beta and the error term. Here because of the randomization that is not the case it is a good randomization that is independent and allows us the ability to look at causation. In this case the trial is just a simple trial of patients of the subscript I as denoting the patient. 

Like I said you can have slight imbalance and you can expand this basic linear model to control for basic characteristics in this case we are just sort of adding this vector Z which is a predetermined prior randomization of baseline characteristics and it gives us information again. Beta is still independent of I and we still have the idea that this is a causal relationship based on the trial. 

So just to review the classic linear model assumes that the right hand side variables and that is the beta are measured without noise. They are sort of considered fixed or repeated samples. If they are measured with noise we assume that the noise is perfectly captured in that error term. If there is some sort of implicit... the way you are doing things is different for the treatment group than the control group then it is not purely... it is not fixed in the repeated samples. You are building in a confounder so to speak and a bias. 

There is no correlation. And we are assuming there is no correlation between the right hand side variables and the error term. The expectation of that X1 or XI to the error term is 0. If these conditions hold we say that the beta is an unbiased estimate of the causal effect. And so we say it is based on all of our clinical trials here is the effect of your treatment. And you go off and you try to publish it in JAMA, in the New England Journal. Wherever you can find these great studies done. 

Randomized trials we all know that they are expensive. They are slow. I have been working on a number of trials where they take many years to do. So it might be impractical to do especially if you want a quick answer may be impractical to get the information through a randomized trial in a short period of time. It may be infeasible. You might be very interested, for example in the effects of testosterone treatment on whether it affects bone density but you are really worried about the effect on potential cancer as an adverse effect. So you could have this huge trial with 25,000 people and it is expensive and its just infeasible to do it. 

There could be times where it is unethical. You can’t randomize patients to withhold treatment in most cases. You can’t randomize them to drink red wine to smoke cigarettes. That sort of causal relationship between drinking and smoking we might not be able to observe in a randomized trial. And then there is many terms where it is just not scientifically justified. And there are terms about [inaud.]  and so forth here where you just might say it is not really a good idea to do that kind of observation. Or a randomized trial in observational study. 

It is clear. One of the goals in setting up QUERI is the ideas that not all the money was going to go into randomized trials. That some of the money was going to go into using observational data. We have a great wealth for observational data. It is a real strength of being in VA in fact we have this amazing data. The question is what kind of questions can we ask and how do we understand  causal pathways and observational data. 

Let’s talk about the sorting without randomization. So the real question I have gone back a couple of slides and showed you that the randomized trial the nice thing is that that sorting mechanism is determined by the investigator. Here when we are using observational data we have patient characteristics. We can observe some of them. We  can have provider characteristics we can observe some of them. For example, we might have staffing and congestion and backed up facility and so forth. These effect how people sort themselves. So you might be particularly in how patients at the VA are using specialty care. Well patients choose whether to have specialty care based on a whole set of criteria. Some of which we see and some of which we don’t. so you might want to compare, for example, patients use specialty care and those who don’t and that would be your treatment group and comparison group and then you could say well do people who access specialty care have better outcomes. You can model that but it is not a randomized trial. And that sorting is the problem here. That confounds our relationship and can bias the outcome. 

So if everything is fully observed the results are not biased because you fully understand that sorting mechanism. It is like sort of saying you fully understood the flip of the coin or the algorithm that put them into the randomized trial. In this case you exactly understand that sorting mechanism and why people do certain things. In red I put it never really happens. We really never understand sorting. It is just that sorting mechanism it is different for everybody. It might be multi-factorial, it might be complex, it might change within person over time. It just never really happens. 

Now there is sorting without randomization and sometimes you will see some unobserved characteristics and observational data. You will often hear people concerned about unobservables. And these are the things we believe are confounding this relationship that we can’t observe. So even with great data sets we are often missing something. So for example, let’s say you’re interested in whether surgical quality affects patient outcomes. Well sometimes we can look at things like volumes and outcomes. The surgeon volume, but we often don’t observe things like the teamwork. How well is that anesthesiologist and the surgeon. And perhaps you have got some nurses in the room as well. How are they all working together and communicating. For example, you might also be interested in patient education and you might never observe that. We don’t have great data in our VA data sets on education. It is not always fully observed without how people with education are using the system or using joint systems. 

These unobserved factors can affect the outcome. If they don’t affect the sorting process you can think about ways of controlling for them. One way is a fixed affect. For example, if you had a – if you know that the sorting was done at facilities and it was just the facility level sorting was different at each place you can include a dummy variable for that facility. And that controls for that effect. Typically fixed affects aren’t the only solution. We often use them in econometrics. There are challenges with fixed affects. When you put in a fixed affect it absorbs typically a lot of variation that is sometimes of interest to you and correlated with your independent ability you are trying to assess. Typically, this is more often the case. You got these unobserved characteristics that also affect sorting. And this is a very challenging – they affect sorting and outcomes. It may be multi factorial again. The unobserved factors affect outcome in sorting the treatment here is biased or the treatment affect is biased. And we really don’t understand the relationship of causality here. And there is no fix. There is no magic bullet to fix this. Now you are going to hear in a future talk about instrumental variables. 

Sometimes we hear these things exogenous factors and you can think of things like laws that happen differentially in one state versus another state or changes in prices that affect patients and perhaps their demands,  taxes for example, are ones that are often state levied. Cigarette taxes are state levied or alcohol taxes. That diiferentiate people depending on those states. You can think of those as being exogenous in many cases and then you can try to use those as an instrumental variable to tease out the affect and to isolate the effect of the sorting based on the exogenous relationship. 

This is the potential fix and for those of you as not just spoiled Patsy’s talking internal variables in a couple of weeks but it is tough to find instrumental variables that are strongly correlated with sorting and really help us understand that sorting mechanism. What we are really trying to do with the instrumental variables if you are not familiar with that technique is really just break sorting into those two buckets. One that is based on sorting that is exogenous, people’s relationships and how it changes the prices, things they didn’t choose and things that are their choice. Then we sort of focus on the piece that is exogenous and we discard the rest. 

In this talk today I am going to be talking about propensity scores. We are going to put the idea of instrumental variables on hold for right now. What is propensity scores? It is really another way to correct for observable characteristics. We will talk about if you even go on Wikipedia and try to understand about propensity scores my sense is you will get the wrong impression about it. It is trying to correct for selection. My belief, my two cents is that it is not trying to correct for unobserved characteristics. It is trying to do a better job of correcting for observed characteristics. Now you can make an assumption. Assumption that there is a perfect relationship between observed and unobserved. You may differ in how that assumption sits with you. It doesn’t sit well with me. I don’t like this assumption it is a sort of strong ignorability assumption. I don’t believe that propensity scores were developed to handle non-random sorting. To make statements about causation you need to then make assumptions about the treatment assignment is strongly ignorable. I just don’t feel comfortable making that. I don’t want to say that you should discard and throw out propensity scores as a method because the rest of the talk is going to show you ways that they can be beneficial and show you ways that you can actually calculate them. But I do want to say that you have to be very careful going in that don’t expect that these are going to be sort of the magic bullet in solving the problems here about unobserved characteristics. 

So I’m going to start moving in to the calculation of propensity scores. So at this point I sort of want to stop and take two seconds. If there is questions that Paul has that I can answer that would be great. Paul, I think you’re muted. 

Paul:Are you able to hear me? 

Todd:I am now, yes. Thank you. 

Paul:Okay, great. There was a question way back and I tried to answer it with text, but this is when you had the X and Y the classical regression model up and you said Y and then someone asked why is X the outcome difference? Shouldn’t it be the characteristics that are unbalanced between the groups? 

Todd:So X is your flipping of the coin so to speak and sort of tells you the difference in what you are randomizing. 

Paul:So it is the treatment. 

Todd:It is the treatment. I might have misspoken there about the statement. So it is really a treatment. Yes. 

Paul:It says X is the mean difference in outcome. 

Todd:Relative to treatment B. 

Paul:It would be beta is that if X is an indicator variable. So the slide has got an error in it we need to – 

Todd:I will fix that. Thank you. 

Paul:So X is the treatment the person receives. Or it could also be the characteristics of the patient. But Y is the outcome. 

Todd:Correct. It is a little bit easier for me to think of this slide where Z is characteristic where you don’t have perfect balance. Where you might want to control for some baseline characteristics. 

Paul:But then again this one should also say beta is the added value treatment. A relative to treatment B if X is the – measure of.... x is the indicator

Todd:Yes. Because you will get out of your statistical package your beta estimate of beta which is the treatment affect, yes. 

Paul:Thanks, Wen Yu for finding the error in the slides. Appreciate that. I thought when you had the instrumental variable slide up – there you go with the exogenous factor. So I think important thing for me to realize is that there is no arrow going from exogenous factors into outcome. 

Todd:That is correct. 

Paul:And that is what makes it a good instrumental variable. It doesn’t have any affect except through its effect on the treatment sorting. 

Todd:Correct. And that is also what makes it very hard to come by. And some people even argue that changes in state taxation rates were passed because people have in those states have predilections to things like use of alcohol or smoking perceptions and are willing to pass those laws so that people even draw a slight dotted line between some of those taxes and outcomes. It has been challenged even at that level. Any other questions, Paul? 

Paul:No. 

Todd:Great. So calculating the propensity score. So typically you have a treatment group or you are trying to look at let’s say a question about whether two groups are different and you want to understand something about the causal relationship. So one group receives the treatment the other one doesn’t. If you are interested in smoking say somebody smokes and one doesn’t. if you are interested in sort of the red wine example you can say you have got people who drink wine and people who don’t and you are interested in the health effects of red wine, moderate red wine let’s say. So you have you can have a dummy variable that says X is the moderate red wine and 0 if they are not. 

You are then going to use a logistic regression model to estimate the probability that a person received the treatment. And received I should probably put in quotes because it is really the probability that the person does this treatment. So the probability that the person drinks red wine in moderation or does not. You are then going to calculate in the statistical packages like R and Stata and Cass, can easily calculate the predicted probability and that this is your propensity score. 

So variables to include. So this really is an important slide. And it is something that I struggled with and there is some counter intuitiveness here that I will try to walk you through. I found the American journal of Epidemiology paper to be helpful. They actually created simulations and went through this. So the of interest here predominantly is the relationship between the exposure variable and outcome. They denote exposure as A and the outcome as Y. They see three different types of confounders. They see confounders first that affect not only the exposure and the outcome so those are the X1s. They see confounders that mostly affect people’s use of the outcomes. And they see confounders that affect the exposure. Whether they are going to get the exposure or not. Your choice of that sorting. 

What they say is most important to include are actually the X2 confounders. What you really want in your propensity score is the X2 confounders. The intuition about it is if you have a lot of variables that are X3 confounders that are multi collinear in some sense with A is that you are weakening your power to identify the effect of A on outcome and you are not necessarily making your groups conditionally look more similar. 

So they go through a bunch of simulations to try to keep this apart so it is a worthwhile read. When I first read it it was a little bit counterintuitive. I’m so used to thinking about instrumental variables that I’m so used to thinking about what we really need to do is identify that exposure characteristic very, very carefully. What they say is the other way around is that you are trying to actually make the folks very similar in outcome so that you have a good understanding of your exposure group. Go ahead, Paul. 

Paul:Todd, I was just going to say that – that is – it is hard to understand. And I was just wondering if you could say it again why we prefer X2 to X3. 

Todd:So the key is that you are trying to make the two treatment groups conditional. Conditional means that on all of the characteristics that you are going to include on your regression model that these are very similar people and the only difference really is exposure. Now if you include a bunch of things that are very good predictors and highly correlated with exposure you are going to have a bunch of things in your model that actually break apart the effect of A. you are going to have a much weaker signal that you are trying to detect in your propensity score and it is going to make it harder for your propensity score to actually calculate the effect of your exposure. These X3 covariants are highly correlated with A. they are not correlated with the outcome. You are weakening your ability to see A. 

Conversely, if you include a lot of these covariants that are correlated with the outcome you are making your populations, your samples, look conditionally more similar but you haven’t reduced your power to feed the exposure of A. so that is the intuition they go through and do some simulation to show it. They say to exclude those variables that are correlated with the exposure but not the outcome. So if we go back they are saying to exclude the X3s. 

These variables will increase the variance of the estimated exposure affect without decreasing the buyout. So the increase in the variance means you have weakened your ability, your precision to see sorry – your ability to see A. So if you exclude extra you have weakened your ability to see A. and then variable selection is particularly important in small studies. 

Cases. Each case has more weight in a smaller study. Think of standard error which is the standard deviation divided by the square root of your sample size. In very small studies cases have more leverage. So if you have an outlier it has potentially much more leverage in a smaller study than in a bigger study. They use 500 as their sort of simulated small sample size. Think of it as just as a rule of thumb. I have never heard anyone say that is the gold standard for small. Just keep in mind if you are doing a small study that your variable selection can really be important in how you develop a propensity score. 

Some other things that I have been reading on propensity scores is that very few people report to their model fit statistics. So I expect in the next two to five years just like what happened in instrumental variables there is going to be a push for greater clarity is in the propensity score so if you want to be on the front edge of that when you are submitting your paper describe how you actually tested the goodness of fits. How you tested for functional form in your logistic regression. For example, if you have age did you just include it as a linear variable or did you break it into dummy variables? Those things are going to be increasingly important for the clarity and understanding of the use and utility of the propensity score. 

I am going to – 

Paul:I have two questions. One says this is completely counter intuitive. Aren’t we talking about the propensity to receive the intervention, the X3s? 

Todd:The propensity – yes. So if the exposure is the treatment. So A is your treatment  affect. And so what you are wanting to do is make the groups look as similar as they can on outcomes with diluting your exposure affect. It is counter intuitive. You are right. The way that I think is you want to control for everything you can possibly control for and what they show is if you can include just things that are strong predictors of this exposure that you can actually weaken your propensity score. 

Paul:So the other question is can you provide an example of you know an X2 and an X3 and – one idea I have for exposure and I am not sure whether that helps you with an example is the exposure could be say somebody gets the strong drug versus the weak drug for particular treatment. 

Todd:Yea, I’m trying to think of a good example. I don’t have one. That is a good question. I’ll put that on my list of things to try to figure out a good answer to that. 

Paul:I can think of plenty of X1s.

Todd: X1s are very common. You are correct. 

Paul:So in that case a strong drug, weak drug you can think the sicker patient is going to get the stronger drug and have the worst outcome. So because they are sicker. So you would think that maybe the strong drug produces worse outcomes when it is just that – it is the people who are going to have the worse outcomes have the stronger drug. This is the bias that we deal with in these kinds of studies. 

Todd:Yeah. And it may be hard to theoretically think of ones that only affect exposure versus outcome. I am trying to think of surgical cases where you do things that are related to surgical techniques that really affect your exposure but not your outcome. I’m sorry that I don’t have a worked out example on that to sort of nail that home. I see that there are some more questions popping up Paul. 

Paul:Oh yeah. A proximity to a wine shop is X3. Somebody said. Although that is more of an instrumental variable. But, yes. 

Todd:Yeah. You can think of distance as an instrumental variable rather than as long as someone didn’t choose to live near the wine shop because they wanted easier access to wine. Most people don’t do that. I meant he classic Mytholen paper was you know, distance to the hospital and it was their first time having a heart attack. So you had this plausibility that they didn’t choose to live there thinking that they might have a heart attack in the future and needed to be close to the hospital. 

Paul:So someone else offers again that the randomization to have greater access to a treatment is a kind of X3, but it is really back to another instrumental variable. And then someone asks is there a minimum number of X2s to be included in the propensity score equation for the model to be identified? 

Todd:That is a great question. When we get to the let me see if I have a slide on it and I don’t. there are some rules of thumb out there. I haven’t seen anything that is set in stone. When you think about the model fits if you are looking at something that is a larger sample and it is a more common outcome you are typically doing okay. If you are looking at something that is rare we often have harder times fitting models to rare events anyways. And so you are already going to be on that sort of narrow side of the distribution looking at rare events. What people have said is that you want eight to 10 exposures per variable in your model. But there is no hard and fast rule on it. I think this is just getting back to this we are at the sort of cutting edge of where propensity scores are and you just need to be careful about how you are specifying your model being clear to the reviewers and the readers here is how you built your model. I wish I had more answers on that. A  lot of these papers are the ones that are just getting published now and in the last five years. 

Paul:Yea. Then we are also asked that the proof of the pudding was after adjusting or stratifying by propensity that the groups would be more balanced with respect to measured characteristics. If you used X2s to create the propensity score there is no reason to believe that it will balance the groups. Is Brookheart missing the forest for the trees?

Todd:That is a good question you will have to ask Brookheart and his colleagues on that. I mean they were running simulations so one of the keys about simulations is it is not observable data they actually created the data with these types of confounders. And then figured out how that affects the propensity score. So their goal was to say and it sounded in reading their article that the X3 types of propensity would initially be the better ones to have. They found it was counter intuitive that even in the Ruben paper that he talks about sort of this intuition that you don’t want to decrease your power of identifying the exposure. So you don’t everything that is collinear to that in your propensity score. It dilutes and expands your standard error so to speak on the exposure variable. You are trying to make these groups look as similar as possible. 

Anything else, Paul you want to add? 

Paul:Well, I think we should move along or we are going to get we are too far off track here. 

Todd:Fair enough. 

Paul:There was a question does X3 have an effect on Y. I guess indirectly is the answer. 

Todd:If X3 does have an effect on Y directly then it would be reclassified as an X1. But you are right X3 is only affecting the exposure. So I am going to walk you through – this is one that we actually did for surgical trial. I have been involved in a CSP trial. So an 11 site trial looking at patients who are randomized to radial artery or saphenous vein for cardiac bypass surgery. The primary outcome was patency, but one of the questions that we asked or are curious about is whether outcomes are different for patients who had the surgery when it was conducted by an attending versus patients who had it conducted by a resident. Now as you – many people are sort of concerned about surgical training that the data is often not available in the observational data sets. You are often sort of using these data from clinical trials. So we track the primary surgeon in CSP 474. 

Is resident assignment random? Paul, I am hearing some feedback from your phone if you wouldn’t mind just – thank you. Is it random? No, it’s not random. The assignment may depend on patient risk, you can think of a whole slew of things that happen in the operating room that determine who is going to conduct the surgery. So we talk to the surgeons about this and it differs at different places. There are different cultures about what you do about this. But some cases are if it is a very risky patient and it is a new trainee, a new surgical resident that you are more likely to assign that to the attending. That makes a lot of sense. There are times where there is just no availability of the residents and so it needs to go to the attending. There might be resident’s skill that is dependent on it. So you think you got a great surgical resident so you want them to handle the complex cases because this is their time to shine and get experience with it. Then there is the local culture. So in CFP 474, 23% so 167 out of the 725 cases were led by a resident. Now, keep in mind we have no information. All we see is that they were led by a resident. We have no information on the training on the resident, all sorts of other things that you would like to know about the skills of the resident. We don’t know that information. We just know that the surgery was led by the resident. 

Use of resident varies by site. So these are state 3Ns. You can see that there is a bunch of stations where no one, no resident did any surgery with any of the patients in the study. Now you could keep these patients, these cases in your analysis but they really only provide information on your control group. And, in fact, if you put in a side effect, a fixed effect for that site these folks are going to drop out because there is no variance within that site. For example if you put a dummy variable in for States 3N 501 to say were they in 501 or not you are not going to be able to assess the effect of the resident because there is no variance in residents. Everybody there was seen by an attending. 

So here is the resident assignment in CFP 474 slight is a very strong predictor. It is perhaps our strongest predictor of why people were doing a surgery with a resident. It is unclear why. Even after talking to the surgeons all of those factors come up. It is the culture, it is the training, it is the supply of the residents and so forth. We see other differences here as well. We see, for example, assignment is not associated with the age or number of graphs. We hypothesize that residents could be assigned to sort of the simpler cases. But again we aren’t able to tease out the experience versus the junior residents because that is really based on this idea that junior residents get the easy cases. Senior residents get the hard cases. 

The assignment is associated with angina symptoms and planned harvesting techniques. There are things then whey think about the surgery they clearly think we are going to do this endovascular harvesting. So when we pull the saphenous vein out of the leg we are going to make minimal incisions. We are going to endovascularly harvest that vein. That is a more complicated technique. In that case they are more likely to use the attending. So you can see those things come up. Urgent priority was not significant. So clearly the sorting is non-random. If sorting was fully observed we could estimate the unbiased effect of the resident surgeon affect. So you could just in a multi variant sense if you fully understood why people were getting sorted to resident and attendee you could control for it in your statistical model.

Improbable is probably an understatement that we fully observe the sorting process.  I say we don’t understand it. And so you could use a multi variant regression. The effects on your betas coefficient on it whether it is attending versus resident is biased. And we need instrumental variables. There is no instrument here. There is no metric instrument that tells us why people are choosing one versus the other. That is an exogenous factor. So again we are trying to do is there any way to do a better job than multi variant to say something about the association here that we are seeing? 

One of the nice things about propensity scores is that it helps us deal with the dimensionality. So when we deal with the multi variant regression things are happening on M dimensions. It is multi-dimensional. With propensity scores we are able to narrow that dimensionality to one dimension. We are creating this propensity score we get to see how the patients are different from one another based on this one dimension. So it reduces this dimensionality to a single dimension. It allows us to create pretty graphs like this. This is called common support. This is the overlap. This is the probability density of having sort of the predictive probability of having a resident surgery versus a non-resident surgery. 

And you can see that there is a considerable overlap. There are times where it is not an overlap between your two tales. So where it says common support it shows that here is where there is overlap between these two populations. The farther apart these graphs get means the more dissimilar the two treated populations are. And you are going to have a harder time using propensity scores to make them look statistically similar. 

So I am going to show you in a little bit a different graph, which is based on a randomized trial hopefully you are thinking about if you were to do a propensity score on a randomized trial what do you expect that graph would look like. So think about that and I will get to there. 

Using a propensity score there are different ways to do it. You can match individuals. You can say well we now have this propensity score we are going to match the individuals and we are going to compare match cases and controls. That is probably the most common approach to doing it. You can include the propensity score for example in quintiles in a regression model with all of your other  covariants. There is quite a bit of data that shows that that doesn’t add a whole lot. But you could do it. 

There are some people that say you can use a weighted regression and that propensity score now is your weight. What you are basically saying is that you are giving more weight to people who look statistically alike and you are giving very little weight to people who are on the edges of your distribution. So that is what that third option is. You can also do subgroup analysis. 

Paul:Todd, if you go back. Sorry, Todd. If you go back we ask if you can just define what common support means. 

Todd:So common support is the area under the probability densities where they share – where there are cases that share the sort of statistical characters that they overlap. Do you have a better answer for that, Paul?  I guess not. Okay. So they share – 

Paul:They are similar. 

Todd:Yea, so they overlap in their characteristics on that common support. Sorry I cut you off, Paul. Alright, let me move on. So you can also conduct subgroup analyses on similar groups. You can stratify and you can say well let’s look at this one quintile of a propensity score. Are they similar and so forth. That would help you understand is this affect that you are seeing in this association consistent across the different propensity score quintiles for example. 

Matched analysis. Let me sort of walk you through some of the basics of the matched analysis. The idea is to select controls that resemble the treatment group on all dimensions except for the treatments. That is the intuition. So you are selecting cases and controls that match on all dimensions except for the treatments. You can exclude cases and controls that don’t match that reduce your sample of size and power. For example, if you go back two slides you can say well there is no overlap between the people at the two extremes of the distribution. Maybe we just don’t want to include those folks because they are not similar. There is no way to make them statistically similar so let’s just exclude them. You can do that. It will reduce your sample size and your power depending on how many you cut out of your sample in your original sample size. And there are different matching methods. 

For example, one is your nearest neighbor. You can just rank propensity. And you can say that Bill has a 79% probability of being in the control group and Jane has a 62% and you can rank everybody. And you can just choose the cases and controls that have the closest propensity. You can also choose your common support. You can vary that common support and then you can randomly draw within each one. That is known as a caliber. There are different ways to do this. And again, there is like I said there are other ways to do it. I don’t mean to say this is all just radial matching there is other types of matching methods here. But they are all trying to figure out what the best way statistically to pick cases and controls. 

Paul:So Todd, there is a question here it seems to confuse people a little bit that the statement about matching seems to contradict what you previously told us about X2. So when you say propensity to be in the control group. What you really mean is to have characteristics like people in the control group, right? 

Todd:Correct. 

Paul:It is a little bit slippery, the concept. 

Todd:Thank you for pointing that out. 

Paul:Your statement about matching seems to contradict what you previously told us about X2 so how do you respond to that? 

Todd:Use the X3 you want a control for your X3s to make your groups as similar as possible so that you have a strong ability to look at your exposure variable. This slide on slide 29 is confusing use my previous slide and I apologize for the confusion. 

There are questions about whether to just use multi variant regression models or propensity scores. Now again, they are both using observable characteristics. They are not trying to use unobservable characteristics. There seems to be little advantage to using propensity scores in most cases over multi variant analysis. And it really gets back to this issue of the common support. There has only been one case and I don’t think Matt Maciejewski is on this, but I read an article where the results differed and he was doing a study on bariatric surgery, where the results differed when he used the propensity score. And in talking to him what happened was when you get back to this common score the groups were so far apart they were so separated that when you tried to use the propensity score there was very little common support. And so he said it basically broke down and said when you look at the people that are most similar there was no effect but in general the groups are so dissimilar to each other you could easily come up with this idea that there is a big association with bariatric surgery and outcomes. So one has to be very careful about it. So the farther the groups are apart the harder you are going to have to sort of use these multi variant cases or propensity scores to make these groups similar. 

What I like about propensity scores is that it reduces the dimensionality. And so it allows you to sort of see how similar are the groups. If you just do multi variant this dimensionality is much more hard to understand. The propensity scores also provide flexibility in how you want to understand functional form of your control variables. So you can say in the first quintile or the first decile how the association and is it consistent across the deciles. You might have situations where the affect is very different in one group than another group and I like that ability. 

Here is the silk purse out of the sow’s ear, I think propensity scores have a lot of uses. But I would strongly encourage people to think about there are ways of understanding the observed characteristics not the unobserved. I am still uncomfortable personally with this idea of that the unobserved are perfectly correlated with the observed and this issue of strong ignorability that I talked about earlier. To me it is just improbable, I sort of take the negative sort of I generally am overly optimistic. Paul will probably agree with that. But in this case I’m pessimistic. I think we rarely see and understand sorting. In observational data it is really hard to understand causality. Propensity scores isn’t the solution here. Now we would say that I would say that instrumental variables may be a solution but you have to be very careful about your instruments and the context in which you are using it and we will talk more about that in an upcoming lecture. That is statistically trying to do something very different than propensity scores. 

How is our time and are there other questions? I have a second example here, but I wasn’t sure if we were going to run out of time, Paul. I think we are doing okay. We have 15 questions. 

Paul:The time is 15 minutes. There was a question about whether there are SAS or STATA.... about how they handle unmatched observations when using near stable matching. I think that is a little bit downstream a bit from where we are in the presentation. 

Todd:To be very quick on this; most of the things that you are seeing here that have been done I’ve done in STATA and there are algorithms in STATA I believe, ARC has algorithms for it. I just don’t use SAS much for analysis. I use SAS for data management. I apologize for not knowing if there is any embedded SAS routines. But it is not very hard to create them yourself as well. 

Here is the second example and this example is really going to show the dimensionality issue that I talked about before is to understand on multi-dimensional you might think these populations are very different and how do we blend that down the propensity score can be useful in that. 

The question here getting back to the CSP is that same trial I was talking about earlier. We randomized patients in 11 sites. They were randomized to two different types of bypass the saphenous vein versus the radial artery and one of the questions that we had is: are the participants in our study generalizable? The Ruby trial which is another heart bypass trial that was published in New England Journal done in VA had just been published in 2009 and one of the major concerns that people voiced in the literature was that the sample was just generalizable neither to the VA nor to the full nation. So we were curious about whether the sample generalized and if it didn’t then why not. 

We identified all of our bypass patients in our sample and across the VA. Because we were enrolling between 2003 and 2008 that was the sample that we used. We are going to compare participants with non-participants within participating sites. We are then curious about whether the participating sites differ from non-participating sites and at the time we were interested in whether the participants are similar to all non-participants across all sites in VA. Sort of gets at this question about generalizable and if they are not generalizable in what dimensions are they not? 

So the propensity score and the reviewer we were trying to do this in a multi variant sense and a reviewer says you should really just use the propensity score sort of out of my thinking about ways to use the propensity score but it is a great idea. So we estimate  a logistic regression for participation so in STATA there you can see there is a PS score and a PS test for this command. Here is the group comparison. We get to see that they are similar in most dimensions but there is not complete overlap. And here is you can see when you do kernel density estimates you have to be careful about the bandwidth of your densities and the type of kernel that you are using so I just put that information there. 

It shows you these matched and unmatched, this is a very common set of outputs you are going to receive when you do propensity scores to tell you a little bit about the balances. Let me walk you through it. So MS is the marital status in group one, MS is the marital status in group three, MS2 is what we omitted. So MS1 means your single, MS3 means you are divorced or separated, MS2 means you are married. Males versus females, you’re A’s are your ages. The reason for including this partial list is this is where we were most different. So I just want to be very sort of forward on that. So in the unmatched sample in the treated and controlled group we had approximately 9.7% of the people in marital status one group. When we controlled for it multi variantly we can see that or the control group is about 10%. When we matched them we can improve that. You can see the change in the buy outs from the treated and control. You can say what is the percent reduction in bias. And you can actually create key statistic for these things. 

You get to see that the bias does change. We can improve the fit here. If I were to sort of focus you in on certain things here that matter is that we tended to recruit more married folks. We weren’t getting as many unmarried folks and we tended to get younger folks and were somehow not enrolling as much of the older folks. Now it does look like you can say that the standardized difference to be clear here the standardized difference is your reduction and the percent change in bias. Let me see if I can pull up a pen here. Highlighter. Not very good here. 

You can look at this percent change in your reduction in your bias and say okay this is really important. You can then see your percent bias and you can get a sense on whether you got strong imbalances. You would say that this is a stronger imbalance here. You have started off unmatched at 12% and that indicates an imbalance. Here we have a 19%, an 18% imbalance. You can say that is approaching severe. These are not hard and fast numbers. So we were severely under representing the older folks as we were saying. Over representing the younger folks. There was less balance in the marital status. We tended to get people who are married and you can think about that as just being easier for them enroll. It is easier for them to come back for follow-up because that is when they are married and have someone who can actually drive them there. You can use this information to actually look at the balance and whether that improves over time. So these are useful statistics to look at. 

You can look at the distribution of the absolute bias overall before matching and after matching. So for folks not familiar with STATA this is your median over here. Here is your mean. After matching the difference has gone way down. Now it is not perfect. It hasn’t gotten to zero but it has gone way down after matching. You can say that the absolute bias has decreased substantially after matching. 

What we concluded is that the participants tended to be slightly healthier and younger. Sites that enrolled participants for different provider and patient characteristics and non-participating sites. It was one of those we weren’t completely generalizable and we could explain why we weren’t generalizable. Now what do you do with that result when you are not generalizable? So we had – let me just sort of walk through – we had 38 covariants in the propensity score model. Balance could be achieved using a propensity score. They were generally similar after matching but not perfect. So what do you do with those results? 

One is they could say just talking about the generalizability is the study generalizable and we could say we tend to enroll younger people, we tended to not enroll older people or people over 75. We tended to enroll healthier people. You can also then keep this propensity score and you could reweight your intent to treat analysis with the propensity score. So you are just putting more weight on the generalizable folks. Here is a citation from some statisticians from Harvard who talk about this with hierarchical data. So you could do that. We had done this study after we had already published in JAMA the main outcomes paper. We absolutely didn’t go back. I had to rerun some of my analysis using this technique and it didn’t change the affect at all. So but you could then think of this as a new weight. And you want to put more weight on people who are generalizable than the people who are less generalizable. 

So here is the question that I sort of want to post to the group that I talked to we actually didn’t go back. I have rerun some of my analysis using this technique and it didn’t change the affect at all. But you could then think of this as a newer weight. You want to put more weight on the people that are generalizable than the people who are less generalizable. 

So here is the question that I want to pose to the group that I talked about earlier .what would happen if you used or you conducted a randomized trial and then you said does it improve at all if I use a propensity score with my randomized trial. Now I don’t have a whiteboard here in the past we have done that and we got a lot of attendees so I am not going to open that up to typing on the screen and so forth. So here I an actual data from a randomized trial. What you are talking about is a common port. They share almost identically the common support. It is a good randomized trial and the randomization worked out well and we struck a good balance. Generally speaking implicitly speaking the propensity score is not going to add anything here. Now you could say well they differ in certain ways. Often through randomization just by chance you differ on some characteristics. 

In that case the propensity score doesn’t add anything to the randomized trial because it was a well done randomized trial. You could think about situations where there is imbalance and you could try to correct for that imbalance. So propensity scores I just want to summarize what we have gone through. They offer another way to adjust for confounding base unobservables. There is really not an attempt here to get at the underlying issue of unobservables. They are very handy at reducing the multi-dimensional nature of confounding and they can help you understand where your group is different and where they are similar. If you wanted to do matched analysis you can use the calipers or your neighbor matching and you can rerun your analysis and say when I use these matching techniques what is the affect. But please don’t say that this is causal. Propensity scores do not attempt to adjust for the unobserved here. They are really just another very useful technique for attempting to adjust for the observed characteristics. I should be careful that is my belief in this. So I would encourage people if you are believing my talk here is to not step up the bounds on that one. 

Don’t believe that this is somehow a magic fix. If you believe that this is – you are going to hear reviewers coming time and time again saying I like your multi variant regressions but I would like you to go a step farther and use propensity scores. There may be very valid reasons for doing that. You might say well we did the propensity scores we see where the inbalance is we can break it into quintiles. We see that we are imbalanced on these different groups. Or you might come back and say we did the propensity score and this is typically what I often see. It made no difference. We had very similar affects for across the quintiles when we stratified we feel very comfortable about the multi variant. So just don’t believe that this is a sort of magic bullet that is going to solve your problem just by doing this you can fix the science. 

Propensity scores I believe are often misunderstood. I think we are still pushing the boundaries on where to do a better job on them. In particular and I apologize for letting you guys down. The variable selection is the key issue here on how you build your propensity model and understanding that. So I encourage people to read more about that. That model building is critically important. I think the articles that I’ve read especially the applied articles and even my own articles we don’t talk enough about the propensity score and how actually we built that propensity score. What variables we included and what we didn’t include, why we didn’t include them and so forth. 

While we can use these methods to create balanced observables I will say it again they are not really fixing inherent selection by us or an observable. We are still left by saying we have this association in the end. 

The strengths. They allow you to check for balance. If you just do multi variant you really don’t know how much balance you have. Without balance the average treatment affects can be very sensitive to the choice of your estimators that you are using. The estimator being are you using a linear regression, are you using perhaps a log linear or maybe you are using an least absolute lineation model and so forth. So there is a great set of slides and I know it is a couple of years old that was done for NBER and it is Guido Imbens and Jeff Wooldridge. Guido is at Harvard and Jeff is at Michigan State and they put together this whole set of lectures for economists. The first one the lecture number one is about matching. They go through the techniques. Heckman has done a lot on matching. You can get into this very in depth if you are interested in this area. 

Further reading. So here is the link for that. There is a book out there. There are many books out there that the American Journal of Epidemiology article I would recommend. And partly because it is using simulated data. And so that really helps us understand when you build and implicitly create these biases that affect when we are doing it. It is much harder when you don’t know the implicit biases that are built into the data and you are just using observational data.

That is the end of my slides. I see a bunch more questions have come in. Paul, do you want to open the floor here? 

Paul:Sure. This goes back I think the comparison of trials. It says sorry this is new and I am a bit confused. The means before and after matching are they the means of the outcome variable or the means of the propensity score? I think this was the one that the slide that you drew on. 

Todd:Let’s go back. Oh. There is my drawing went away. I didn’t know it would do that. 

Paul:Are they the means of the outcome variable or the means of the propensity score. 

Todd:These are the means of your propensity score. So it is really if I’m interpreting that correctly it is how your marital status changed when you are using your matched propensity score versus your unmatched propensity score. 

Paul:But the column that is rated mean is actually the percentage of the folks that are in that group, right? 

Todd:That is correct. The percentage. So we have here if you say .097 if you times that by 100 you know that 9.7% of your folks in the unmatched group were in marital status group one. If we look in the first this age group here we know that approximately 13% of our people were in our treated groups. That is younger about 9% of the folks were in our control group. So after using the propensity score we see that this didn’t change, but we can increase the rate at which based on the propensity score of the match we approximately have 11.9% of the people looked at that age group statistically. 

Paul:Great. And then is there a baseline severe imbalance in randomized  control that we can use propensity score methods? 

Todd:You can definitely... people do this and that is why you end up with that table one in most journal articles that shows you we don’t take it at face value that randomized trials have always done well and led to perfect balance you want to then show where do they differ based on the characteristics when they enter the trial. So you could clearly if you felt like there was imbalance you could control for using analysis of covariants. You could use those variants in the affects. I go way back. I don’t know if I can get there early enough but I talk to them about randomized trials and I had a slide – 

Paul:Disease. Disease that were in there, right? 

Todd:You can control for those. You can also use propensity scores in a randomized trial if you felt that there was imbalance. You could use them and you could say about matching them based on match cases. Or you might say well keep this intent to treat framework and we will use a weighted – so we will put more weight on folks who have very similar characteristics. 

Paul:And then someone asked if you could briefly comment on the IPW approach and how it differs from matching. 

Todd:IPW I am not familiar with that approach. 

Paul:So that is inverse- wait a minute. 

Todd:I think it is probability weighting. Is that? 

Paul:Yes. Exactly. I think that is what they mean. That’s what I think it means. 

Todd:Yea that sounds – that sounds right. You’re right there are different ways to use this propensity score once you have it. I haven’t seen any simulated studies to say that there is a right way to do it versus a wrong way to do it. In economics we often see people use this inverted probability weight. Say let’s place more weight on people who are more generalizable for example or are more similar in cases or controls. We don’t want to throw out the cases that are less generalizable we just want to put less weight on them. Versus a matching technique, for example. Epidemiology might have a different –they might use more of the matching techniques. I haven’t seen any that put those head to head. If somebody is aware of that I would be very interested in seeing that paper. 

Paul:So then someone asked if we used propensity scores to weight the regression how can we correct the standard errors to take into account the predicted nature of propensity scores? 

Todd:I’m not even sure that – 

Paul:Similar question I guess. 

Todd:I am not even sure with the inverse probability weights that analytical standard errors would be incorrect or above standard errors would be incorrect. If you are concerned you got incorrect standard errors bootstrapping is an empirical way of trying to correct your standard errors. That should be fine. I can’t imagine – although I haven’t done those simulations myself and I could be wrong, I haven’t seen any papers on this I can’t imagine why bootstrapping wouldn’t come after that. 

Paul:Then someone asked in the Brookheart framework if one has substantial – oops. I just lost it. Excuse me. If one has substantial imbalance both in unmatched and matched estimates from a covariant X2 but inclusion of X3 covariants correlated with X2 potentially adjust the amount next to the variable. 

Todd:No. So let’s if I can go back there. That doesn’t necessarily – what you are left with if you have got severe imbalance extreme severe imbalance you sort of have to report that. Statistically you can’t make these group similar. There is nothing you can do to make these groups similar to give them sufficient overlap. And so you are sort of left in this situation where and get back to Matt Maciejewski’s paper that there may be very small groups that are similar and if you focus on that group there is no difference but on a bigger group there is a difference and you just have to report that. Be very honest and report that. And that is exactly what Matt does. So that is a paper in JAMA on bariatric surgery, effective bariatric surgery using that data. You can think of it – when I showed you the support the common support in the randomized trial think about the most unbalanced observational studies where there is very little common support if any. You would say well, no matter what we can do we can’t make these groups look alike. 

Paul:Right. So then it was asked is it safe to say that variable selection may be based on the theoretical model but a variable may be useful in propensity score even if the P value is not significant. 

Todd:Absolutely. So there are people that talk about whether to use standard model building techniques. You can think about in those standard model building techniques forwards, backwards you know certain statistics that help you build your epidemiological models you can have variables that are not significant but are confounders. Or are collinear with groups of variables or important as a set. You have to be very careful. 

Paul:Not significant in predicting the outcome I guess is the – 

Todd:You shouldn’t drop a variable from the propensity score just because it is not significant in the propensity score the general rule of thumb is – getting back to this diagram that you have got good X2s put all of your X2s in that you got. There is very little harm in putting in extra X2s. 

Paul:Why don’t those percent in treatment versus percent in control – wait a minute. I’m not sure I understand this. Why don’t those percentage in treatment versus in control do not change but the control percents do from the bias table? 

Todd:It is just that when they use the scores the question becomes they have to hold one constant. So here they just have to hold one constant. So to say under the mask conditions what does the control group look like they could have held the control group constant at 10.6% and buried the treated condition. It is just this is what the statistics chose to hold. 

Paul:This is relevant to your reference group which is the married people. 

Todd:Yes. They just had to hold one constant and say well now under our new regime of matched what does it look like. So they just had to hold one constant. 

Paul:So we are six minutes past the hour. I’m not sure when we are supposed to wrap up. 

Todd:We are done. I apologize for running over. I just know that this is a topic that is hot on people’s radar. So feel free to contact us if you have other questions and thank you for hanging with us for six additional minutes. 

Heidi, do you want any closing words here ? And Patsy is coming up next about instrumental variables. So if you find this provocative and you are dismayed about our inability to control or unobserved come back to Patsy’s talk. 

Heidi:Patsy will be presenting on August 8th and will be sending registration information out to everyone on that about a week before her session. So keep your eyes open for that. Thank you, Todd and Paul. We really appreciate the time you put into this. It looks like we may have some pending questions out here, Todd. I will get those sent over to you probably on Monday,  I’m out for the rest of the week. 

Todd:That sounds great. I know a lot of people have vacations coming up. So hopefully everyone enjoys their vacations including you, Heidi. Thank you so much. 

Heidi: Thank you. We will talk to you later. 

Todd:Alright, thanks. 

Paul:So people accept our apologies. There is one person who asked can we get back to them by email we don’t have any way to copy the questions out of the system here, but Heidi does. It will be a week before we get back to you. 

Heidi:But I will get everything over to them first thing next week. 

Paul:Some people asked for a copy of questions and answers. We haven’t written out too many of the answers yet. You may want to just you can see this on the archive too. You can see it again if you want to see the repeat. 

Heidi:If we do have any that we respond to we usually do postals in the archives. Lots of resource. 

Todd:And they do a lot of editing so the archive is always better than the first, so. 

Heidi:You make it sound like we do bad stuff in the first one and it is our chance to make it sound better. 

Paul:He is talking about us. 

Todd:It is like we could voice this over with George Clooney you know make these very convincing talks in the archives. 

Heidi:You are going to make people think I have a huge budget here now. Okay wrap it up and I will talk to you guys later. To the audience, thank you so much for joining us. We look forward to seeing you at a future HSR&D cyberseminar. 

Paul:Thank you. 

Page 1 of 21