**Transcript of Cyberseminar**

**HERC Econometrics with Observational Data**

**Instrumental Variables Regression**

**Presenter: Christine Pal Chee**

**October 30, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact HERC at herc@va.gov.*

Todd Wagner:  I just wanted to welcome everybody to today's HERC cyber course on Instrumental Variables Regression. I'm pleased to introduce Christine Pal Chee who's one of the new health economists here. She's been here about a year. And she gave a talk a couple weeks ago and so it's  - we're very pleased to have her back and talk about this important area. Thanks. 

Christine Pal Chee:  Thanks Todd. As Heidi and Todd mentioned today's lecture will be discussing instrumental variables regression. To start, it's  helpful to think about instrumental variables regression within the larger context of estimating causal effects. The estimation of causal effects is a common aim in health services research. Our research questions often look something like this. What is the effect of blank treatment on blank outcome? Ideally we'd estimate this effect  using a randomized controlled trial but we know that conducting a randomized controlled trial is often very difficult and impossible. So an alternative then is to perform multiple regression analysis using observational data. In order for us to estimate an accurate causal treatment effect using multiple regression on observation data it must be the case that treatment is exogenous. We discussed this concept in greater detail in the research design lecture a few weeks ago, and we'll briefly review it in today's lecture. But what it means is that whether or not someone receives treatment is uncorrelated with all other factors that may affect our outcome variable. Now with treatment it's  not exogenous. That is if treatment is endogenous then our estimated affects will be inaccurate and we'll need another method or design in order to estimate our causal affect. And one possibility is instrumental variables regression which we'll spend the rest of our time today discussing. 

To start off I thought it'd be nice to get a sense of everyone's familiarity with instrumental variables regression and we had a poll here that Heidi  is putting up. It'd be great if you could let us know if you have an advanced knowledge of instrumental variables regressions, you're somewhat familiar with instrumental variables regression, or if you're new to instrumental variables regression. 

Moderator:  And it looks like responses have slowed down. If you want to read through those. 

Christine Pal Chee:    Okay. It looks like about two percent of the audience has an advanced of instrumental variables regression. Almost half is somewhat familiar with instrumental variables regression, and half is new to instrumental variables regression. Thanks Heidi. Which is great because the purpose of today's lecture is to provide an introduction to instrumental variables regression. And we won't be focusing on technical details or procedures but rather the key concepts and the intuition be instrumental variables regression. To begin we'll review the basic linear regression model and when we might need to consider an alternative method like instrumental variables regression. We'll discuss necessary conditions for a valid instrument and look at why and how instrumental variables regression works. To see instrumental variables regression in action we'll walk through a well-known McClellan, McNeil, New House paper that uses distance to evaluate the effective intensive treatment on mortality. And then we'll briefly discuss a few other examples if we have enough time. And finally, we'll wrap with some brief comments about limitations of instrumental variables regression. And I'd like to leave sufficient time at the end to take questions but if people have questions throughout the lecture you can feel free to send them in and Todd can bring those up. 

Before we jump into our discussion of instrumental variables regression we'll briefly review the linear regression model to see how instrumental variables regression fits in. Our basic linear regression model usually looks something like this where on the left hand size of the equation we have Y which is our outcome variable of interest or our dependent variable. And on the right hand size we have X which is our explanatory variable of interest. Here we'd like to understand the effect that X has on Y. Also, in the right hand side is E, which is our error term. Our error term contains all other factors besides X that determine the value of Y. And in general the coefficient we're most interested in is Beta-1 which corresponds to the change in Y that's associated with a unit change in X. 

Now in order for our estimate Beta-1 hat to be an accurate estimate of the causal effect of X on Y it must be the case that X is exogenous. Otherwise all we have is a correlation between X and Y. Now what does it mean for X to be exogenous? Formally it means that the conditional mean of the error given X is zero. So for a given value of X the mean or average value of the error, which is the difference between the predictive value and the observed value of Y is zero. This essentially means that additional information in E, and these are all the other factors that may determine Y, do not help us better predict Y. And so the error, all these other factors, are essentially noise but all we need is information in X. When this is true we say that X is exogenous and this implies that X in the error term cannot be correlated. If X and the error term, all the other factors that may affect Y are correlated then we say X is endogenous. And if X is endogenous then our estimate of the causal effect, beta-1, is biased. 

In the research design lecture we saw how X and the error term are correlated when there is omitted variable bias, sample selection, or simultaneous causality. Each of these creates a problem of endogeneity in X. Sometimes it's  possible to overcome this problem of endogeneity within multiple regression analysis. But when it's  not we need to find another method or research design to overcome it. And this is where instrumental variables regression comes in. The idea behind instrumental variables regression is actually really simple. It starts with the insight that variation in X has two components. One component is correlated with the error term, and this is the problematic component that causes endogeneity in X. The other component is uncorrelated with the error term which poses no problems to estimating beta-1 accurately. We refer to this component as exogenous variable in X. In instrumental variables regression the aim is to use only exogenous variation in X to estimate beta-1. 

Now how do we isolate the exogenous variable in X? And this is where our instrumental variables come in. Instrumental variables or instruments can be used to isolate the exogenous variation in X or endogenous variable that is uncorrelated with the error term. In order for an instrument to be valid two conditions must be satisfied. They're instrument relevance and instrument exogeneity and we'll talk more about each of these conditions. But before we do that let's take another look at our regression model so we have a framework in mind. So again, we'd like to estimate the effect of X on Y using this basic regression model. But we have a problem and that is our explanatory variable of interest, X, is endogenous which leads our estimate of beta-1 to be biased. This is true when X and the error term are correlated. And remember that the error term contains all other factors besides X that determine the value of Y. But now we have a potential instrument Z and we'll evaluate this instrument within the context of this model. Returning to the two conditions for valid instrument the first that must be satisfied is that our instrument must be relevant. This means that X must - or instrument X must be correlated with our endogenous variable X. Sorry, I think I got that mixed up. Our instrument Z should be correlated with our endogenous variable X. This means that variation in X explains variation. I'm sorry. I keep making this up. Variation in Z explains variation in X. It means that if we know the value of Z, our instrument, we can predict the value of X. In other words, Z affects X so it must be the case that our instrument affects our variable X. When this is true we say that Z is relevant. 

The second condition that must be satisfied for an instrument to be valid is instrument exogeneity. This requires our instrument Z to be uncorrelated with our error term. This means that Z is uncorrelated with all other factors besides X that determine Y.  It also means that Z does not directly affect Y except through X. When this is true we say that Z is exogenous. To see how these two conditions, instrument relevance and instrument exogeneity come together and help us estimate beta-1 our causal effect of X on Y let's return to our regression model. Here we're interested in the effect that X has on Y. The key insight behind instrumental variables regression is that variation in X has two components. One component is uncorrelated with the error term while the other component is correlated with the error. A valid instrument affects X though a valid instrument is relevant but is also uncorrelated with the error term, so it's  exogenous. In instrumental variables regression we use this instrument to isolate exogenous variation in X that is uncorrelated with the error term. And in this way we disregard the problematic variable that is correlated with the error term. It's important to note here that Z does not directly affect Y. The only way that Z affects Y is by affecting X, which in turn affects Y. 

The main point here is that our instrument Z only captures the exogenous variation in X that is uncorrelated with our error. To provide more intuition on what an instrument might look like and how it works let's consider an example where we'd like to estimate the affect of some medical treatment on a particular outcome like mortality. And let's say that treatment is assigned through a coin flip. If a patient flips heads he gets treatment. And if he flips tails he does not get treatment. Now let's consider whether this coin flip is a valid instrument for treatment. First, does it affect whether or not a patient receives treatment? Actually it does. The value of the coin flip whether it's  heads or tails directly affects whether or not a patient receives treatment. So the coin flip is relevant. Second does the coin flip directly affect the outcome mortality? In this case it's  probably very unlikely that flipping a heads or tails in and of itself will cause a patient to live longer or shorter. So it's  likely that the coin flip is exogenous. Now if we were to take a different scenario where the outcome variable we're interested in were headaches and when we flipped our coin we actually threw it at the patient's head as hard as we possibly could then it's  possible that the coin flip in and of itself would affect outcomes. And in that case the coin flip would be endogenous. But in our example here where our outcome variable is mortality and we have an honest coin flip it's  probably the case that the coin flip is exogenous. 

Now in the scenario that I just described where the coin flip effectively randomized patients to treatment without actually affecting outcomes we have a situation that looks very much like a randomized controlled trial. Now in a similar way to how the coin flip randomized patients to treatment variation in an instrument mimics randomization of patients with different likelihoods of receiving treatment. And here we refer to the likelihood of receiving treatment because it does not need to be the case that an instrument directly causes treatment or perfectly predicts whether or not someone receives treatment. It just needs to be the case that the instrument affects the likelihood of treatment.

Now that we've discussed what's required of a valid instrument and what it does let's see how instrumental variables regression works. Before we do that let's formulate the instrumental variables model. So here we have our usual regression model. We're interested in the affect of X on Y, and endogenous variable X that's correlated with the error term. However, we also have a valid instrument Z our  disposal. This instrument is both relevant. It's  correlated with the endogenous variable X and its  exogenous. It is uncorrelated with the error term. There are several different ways to estimate the instrumental variables estimator of beta-1, our causal affect of X on Y. But a common method that also nicely estimates how instrumental variables regressions works is two stage least squares which as its  name suggests is implemented in two stages. 

In the first stage we regress X on Z. When we look at the regression equation we can see how the variation in X can be broken down into two components. This first component relies on information from X, sorry, from Z. This first component relies on variation from Z, our instrument. And because our instrument Z is exogenous and uncorrelated with the error term this first component is also uncorrelated with our error term. While the second component, which does not use information from Z, can be correlated with our error term. And we'd like to estimate the variation in X that comes from this first component which is uncorrelated with the error term. And in order to do that we predict X using information, using the coefficients from the first stage regressions of X on Z, pi 0 hat and pi 01. These are the regression coefficients that come from this first stage regression. We used these coefficients and our instrument Z to predict X. This predicted X has been cleaned of variation that is correlated with E, with our error term. And it only contains variation that is uncorrelated with the error term because it only uses information from Z. 

In the second stage, we regress Y, our outcome variable, on our new predicted X. And we estimate the two stage least squares estimator of beta-1. This will give us our causal effect of X on Y. Because X hat is uncorrelated with the error term from the original regression model we have up to two stage least squares estimate of beta-1 is in an unbiased estimate of the true causal effect of X on Y. One thing to note about two stage least squares is that the standard errors in the second stage, two stage least squares regression  needs to be adjusted to account for the fact that we are using predicted X hat rather than the actually observed X. But many statistical software packages will estimate both stages simultaneously and automatically adjust the standard errors for us. So it's  usually not something we have to worry too much about. 

To simplify our discussion I focused on the case where there is only one endogenous regressor and one instrument. But the instrumental variables model can be generalized to include K endogenous regressors so we can include X1 to XK endogenous regressors that are correlated with the error term. Our exogenous regressors or control variables, W1 to WR, and M instrumental variable. So it's  possible to have multiple instrumental variables, Z1 to ZM. The important thing to note is that there must be at least as many instruments as there are endogenous variables. So it must be the case that M is greater than or equal to K. 

Another thing that is important to note is that instrument variables regression estimates the local average treatment affect, which is the weighted average of individual causal affects. With individuals who are most influenced by the instrument receiving the most weight. So we're basically estimating the causal effect of treatment on the people who are most influence by the instrument. These are the people who would not otherwise receive the instrument. Sorry. These are the people who would not otherwise receive the treatment were it not for the instrument. In that way instrumental variables regression often estimates the marginal effect of treatment. In general the local average treatment effect differs from the average treatment effect over the entire population. This doesn’t bias our estimates in any way. It just changes the interpretation of our estimates which is something we need to be mindful of when drawing conclusions from instrumental variables regression. 

To see instrumental variables regression in action we'll walk through an example from a well-known paper by McClellan, McNeil, and New House that asks does more intensive treatment of acute myocardial infarction, AMI or heart attack, in the elderly reduce mortality? More specifically we want to estimate the effect of intensive treatment on heart attacks, and the authors look specifically at cardiac catheterization on mortality. This research question corresponds to this basic regression model. Here we'd like to understand the effect of treatment on mortality. The authors find that when you estimate a simple equation, a simple regression model like this that there is a very large effect of catheterization. Catheterization reduces mortality by 31 percentage points at one year. But there is a problem  and that's whether or not a patient receives more intensive treatment is correlated with many unobserved factors that may also affect mortality. Examples include health status, or patient or physician preferences. And for the big concern here is about health status. It's  likely to be the case that patients who are most likely to benefit from surgery are most able to recover perhaps the most healthiest are probably more likely to receive treatment. If that's the case then our estimates from comparing patients who receive treatment versus patients who do not receive treatment are likely to be bias. Actually when we take a closer look at the patients who do and do not receive catheterization we see that in general patients who do receive catheterization these correspond to the far right hand column tend to be younger and they tend to be healthier. They're less likely to have comorbid disease conditions. 

Now when we control for these observable demographic and comorbid differences we find that the estimated affect of catheterization is reduced by about 20 percent. Now instead of a 31 percentage point decrease in mortality at one year we find that catheterization is associated with a 24-percentage point decrease in mortality. So because we know that patients who receive catheterization are different than those who don’t we have an evidence of selection bias. If we don’t account for this selection our estimates will be bias. Now if the two groups of patients different only in the observable demographic and comorbidity dimensions that we just saw and that are controlled for the second panel we can just explicitly control for them and solve the problem of selection. But the concern is that these patients may also differ in unobserved ways say in their appropriateness for treatment or their ability to recover that we cannot observe from the data and therefore cannot control for. And the fact that patients differ significantly on observable dimensions leads us to believe that they may also differ in unobservable dimensions. So we have a problem but the authors have an idea that may be able to solve this problem. And if that - patients who closer to hospitals that have the capacity to perform more intensive treatments are more likely to receive those treatments. So basically if you live closer to a hospital that does catheterization you're probably more likely to get catheterization. 

Now if this is true then distance is relevant. Next, the authors speculate that the distance a patient lives from a given hospital should be independent of his health status. If this is true then distance is also exogenous. If these two conditions, instrument relevance and instrument exogeneity are met then we've found a potential instrument for intensive heart attack treatment and that's differential distance to catheterization and revascularization hospitals. In this table the authors compare characteristics of patients who live closer which corresponds to the left column and farther which corresponds to the right column from catheterization and revascularization hospitals. In the top panel we see that patients who live closer and farther to these hospitals are actually pretty similar in terms of their observable heath characteristics. This suggests that they may also be similar in unobserved dimensions or at least provide some support for it. If this is true then distance is exogenous. Now in the second column of our panel here we see that patients who live closer to catheterization and revascularization hospital are more likely to be admitted to these hospitals and they're more likely to receive cardiac catheterization. And they're also slightly more likely to receive CABG or angioplasty. 

Now because patients who live closer to these hospitals are more likely to receive cardiac catheterizations the authors argue that distance is relevant. Using instrumental variables regression to account for unobserved heterogeneity the authors find that cardiac catheterization reduces mortality by five percentage points at one year. This is in contrast to earlier results that showed a 31-percentage point decrease in mortality. And even after adjusting for observable demographic and comorbid differences the authors estimated a 24-percentage point decrease in mortality. So this new estimate of five-percentage point's decrease in mortality is very different from these other results. And they suggest that unobserved heterogeneity or patient characteristics play a very important role when estimating the causal effect of catheterization on mortality. 

So in this paper, in the instrumental variables regression results we find that catheterization reduces mortality by five percentage points at one to four years. There are  a few important points to note. The first is that the validity of these results hinge on the validity of the instrument. These results are only as good as the instrument is. And we will only believe the results if we believe that the instrument is both relevant and exogenous. The next is that instrumental variables regression estimates the local average treatment effect. So this five-percentage point decrease in mortality is an estimate of the marginal effect of catheterization. For patients for whom distance actually matters when it comes to receiving cardiac catheterization these are the patients who would not have otherwise received treatment had they lived relatively far from the catheterization or revascularization hospital. This does not include everyone who receives treatment with cardiac catheterization because it's  likely that regardless of how far you live from a catheterization hospital if you are very clearly appropriate for catheterization you'll probably receive catheterization regardless of the distance you live from the hospital. So these are patients who are probably the marginal cases where it's  unclear whether or not they should receive catheterization, which explains why this estimate is relatively small. 

Last, this estimate is in upper bound of the effect of catheterization. If catheterization or revascularization hospitals offer better care other than more intensive procedures these might include a more available beds, more specialists in ICU, then mortality should be lower at these hospitals. And the effects that we estimate may also pick up this effect of better care at these hospitals. And actually when we look at the effect of catheterization at one day we find that catheterization reduces mortality by nine percentage points. Interestingly in the sample the authors used, most people do not receive catheterization in the first day after a heart attack, which suggest that these effects are probably driven by something else, something other than catheterization itself. So that’s just a caveat to keep in mind. But that said in this paper we seem to have found a great instrument for intensive treatment for heart attack that deals with the problem of endogeneity of treatment. Now does this mean that distance is always a good instrument for treatment? We'll look at two examples to see whether or not this is the case. 

First, let's say we're interested in estimating the effect of primary care use on health outcomes. We have a problem with endogeneity because people usually see a doctor when they're sick. So if we were to just simply compare the health outcomes of people who see a primary care doctor versus patients do not see a primary care doctor we'll probably find that patients who do see a primary care provider generally tend to be sicker and have poorer health outcomes. Now can we use distance to the nearest primary care clinic as an instrument for primary care use? So patients who  live closer to primary care clinics are probably more likely to see a primary care provider. This seems plausible. But if you have to drive really far, a really long distance to see a doctor you probably might not go, you might go less frequently. If that's the case then distance is relevant. However, patients who need to see a doctor often might move to live closer to healthcare facilities. For example, if a patient is on renal dialysis and knows that they have to get receive dialysis frequently and see a doctor for checkups frequently they might actually move to live closer to a healthcare facility. If that’s the case then distance is not exogenous and in this context we cannot use distance as an instrument. 

Second, let's say we are interested in the effect of emergency department services for car accident injuries on mortality. So if you see - if you go to an emergency department after a car accident is your mortality lower? The issue here is that only seriously injured passengers are taken to the emergency department. So if we were to compare mortality for patients who are taken to the ED to the mortality of patients who are not taken to the ED we'll likely find that patients who are taken to the ED have much higher mortality. But it doesn’t necessarily mean that going to the emergency department reduces mortality. It's  just that patients who go to the emergency department generally tend to be more severely injured. So can we use distance to the nearest emergency department as an instrument for emergency department services? So first it seems likely that distance to the nearest ED is probably uncorrelated with accident severity. It's  possible - it seems likely that your accident severity is independent of how far the accident occurred from a hospital. If that's the case then distance is exogenous. However, it's  still the case that only people who need medical care are taken to the ED regardless of distance. So a person gets into an accident relatively far from an ED but they're very seriously injured they'll likely be taken to the ED whereas if you were to receive a little bump on your back bumper in front of a hospital you probably wouldn't be taken to the emergency room. Now that means that distance is not relevant. It does not predict whether or not someone is taken to the ED. And so in this case distance also cannot be used as an instrument for treatment. And in general, whether or not something is a good instrument completely depends on the context. And our judgment on whether or not it is relevant and exogenous. 

To make sure we have enough time for questions I'm going to just briefly discuss a few recent papers that utilized instrumental variables regression. The first is by Bhattacharya, Bundorf , Pace, and Sood who investigate the effect of insurance coverage on body weight. Because insurance coverage is a choice to some extent and related to employment it's  likely to be correlated with other factors that may affect someone's health or body weight. And in that case if we were to just compare people with insurance coverage to people without insurance coverage we probably would have biased estimates of the effect of insurance coverage. The idea that these authors have in this paper is that individuals who live in states with a higher portion of large firms and with a higher share of Medicaid coverage are more likely to be insured. And in using the distribution of firm size and Medicaid coverage for each state and year, they find that health insurance coverage actually increases body weight. And in the work that I discussed in the research design lecture I was interested in estimating the affect of treatment with antiretroviral therapy on substance use. So in other words is a patient more likely to use substances or to engage in substance use after they've received treatment with antiretroviral therapy for HIV/AIDS. The problem with comparing patients who receive antiretroviral therapy to patients who do not receive antiretroviral therapy is that only patients who have a certain level of disease progression are appropriate for antiretroviral therapy. So only the sickest patients will be on antiretroviral therapy. And if a patient is really sick they're less likely to engage in a lot of activities, they're less likely to go to work, they're less likely to engage in social activities, and even engage in substance use. 

In this paper I make use of state Medicaid policies or variation in the generosity of state Medicaid policies. The idea here is that patients who live in states with more generous Medicaid policies or more generous Medicaid eligibility policies are more likely to receive antiretroviral therapy. And using state Medicaid policies as an instrument for antiretroviral therapy I find that treatment with antiretroviral therapy increases the likelihood of substance use. In another recent paper, Joe Doyle investigates the effect of foster care on long and short-term outcomes which include juvenile delinquency and emergency department visits. We know that children who are placed in foster care have poorer outcomes across the board but this may be due to foster care or it may just be due to the residual effects of abuse and neglect to the reasons why they were put into foster care to begin with. In this paper, Doyle makes use of random assignments to investigators. So in the state of Illinois cases are assigned to investigators following a set rotation schedule. So cases are essentially randomly assigned and these investigators differ in their propensity to add the case for foster care placements. Some investigators simply are more likely to place kids in foster care, maybe because they really believe in the benefit of foster care and some - while some investigators are less likely to place kids in foster care. So using  a random assignment to investigators as an instrument for placement in foster care Doyle finds that foster care actually increases the likelihood of delinquency and emergency healthcare episodes. 

Before I wrap up I'd like to make a few comments about invalid instruments. First we'll discuss instruments that fail to satisfy instrument relevance. These are weak instruments. They explain little variation in X. Instrumental variable regression with weak instruments provide unreliable estimates. There's a rule of thumb to check for weak instruments when there is only one endogenous regressor. And that is a test of the joint significance of the instruments in the first stage regression of two page least squares. So here what we do is we confuse the F-statistic testing the hypothesis that the coefficients and all of the instruments are equal to zero. An F-statistic greater than ten indicates that instruments are not weak. But it's  important to note that this is just a rule of thumb and we still need a convincing argument that the instrument is relevant or strong. 

Next, I'll talk about endogenous instruments. So instruments that are correlated with the error term or they're correlated with other factors that affect the outcome variable are endogenous and they fail to satisfy instrument exogeneity. Instrumental variables regression with endogenous instruments provide meaningless estimates. The whole point of instrumental variables regression is to isolate and utilize exogenous variation in X that's uncorrelated with the error term. An instrument that is also correlated with the error term will not allow us to do that. When there are more instruments than there are endogenous regressors it's  possible to test over identifying restrictions using a J-statistic. But since it's  actually hard enough to find one good instrument I don’t go into this in detail, but if anyone would like more information about this test I can send more information over email. What is most important here is also that we need a convincing argument that the instruments are exogenous. For both conditions instrument relevance and instrument exogenous was absolutely necessary is a good story and good reasons for why the instrument is relevant and exogenous. 

Now I'll wrap up now with a few comments about instrumental variables regression. The first is that instrumental variables regression is a very powerful tool to estimate causal effects. There are two conditions that must be satisfied for a valid instrument. But the first is relevance, the instrument must effect treatment. Second the instrument must be exogenous. It must  be uncorrelated with all other factors that may affect the outcome variable. These instruments are actually very difficult to find and using an invalid,  a weak or endogenous instrument will give meaningless results. Although there are some tests available to check for instrument validity, what is most important and absolutely necessary is a good story for why an instrument is relevant and exogenous. And I've also listed a few resources. The first is textbook, introduction to Econometrics by Stock and Watson. Chapter 12 on instrumental variables regression actually provides an excellent introduction to the topic. And the last paper listed, Joe Doyle's paper also includes a really nice discussion that sets up the background and intuition for instrumental variables regression for anyone who would like more information. And now I think we have time for questions. 

Todd Wagner:  Sounds great! So when you say there's a good story, you need a good story, what constitutes a good story? Is it clinical relevance, subject matter, knowledge? Can you say more about the good story? 

Christine Pal Chee:    Yeah. Actually the good story, depending on the context, will require clinical knowledge or background, expert knowledge, but what is necessary is to convince yourself and others that the instrument is actually relevant and exogenous. And oftentimes that requires a deeper understanding of the context. So in the case of the McClellan paper with looking at heart attack treatments I thought it was necessary to have a clinical background to understand who actually receives cardiac catheterization, someone who is more familiar with that setting would be more likely to be able to figure out what would actually be plausible and what would not be plausible. 

Todd Wagner:  And just to follow-up on that little bit, so the two conditions for the valid instrument, the first one is relevant. That's empirically testable as Christine talks about, the real [inaud.] on F-statistic that's greater than ten. The other one really is something that you believe that this is exogenous and so that's the question about when you're thinking about when I say driving time or differential driving distance, could people be moving themselves closer because they know that  they want more medical care? One of the great stories behind McClellan's paper was that these were otherwise asymptomatic healthy people who had a heart attack. So it's  not something that was planned in any regard. It just happened. And so when they're picked up by the ambulance they're taken typically to the closest facility that has space in the emergency department. So there's a very nice, both health services, clinical, the whole story fits together well there. But if you're talking about, for example, primary care as you mentioned Christine you can imagine people moving all over just to get better access to their primary care physician especially in retirement. A lot of people move from the upper northeast to Florida. You can imagine that that choice of where to live in Florida is driven in large part based on their health conditions. 

Christine Pal Chee:  Yep, that's true. And the point you made Todd about being able to test for instrument relevance is true but not instrument exogeneity is also totally true. It's  possible there are witness tests for instrument strength or instrument relevance but largely for instrument exogeneity it requires that deeper clinical knowledge or background or expert judgment to establish exogeneity. 

Todd Wagner:  Alright, so we have two more questions, one's  I think a shorter one so I'm going to get that one first. If you're endogenous variable is categorical or binary can you use instrumental variables? 

Christine Pal Chee:   Yes, you can. All you would have in the - is your endogenous variable. Yeah, so that means in the first stage your dependent variable will binary in the case of binary endogenous variable, and you probably use probit or logit in that first stage. There are the statistical software packages. We'll usually have different options for these different cases. But it is possible with both binary and categorical variables. 

Todd Wagner:  Alright, so that's a great answer. Here's the more challenging question. In different fields economics we often talk about instrumental variables. In psychology they talk about structural equation modeling or SEM, and there's entire software packages that develop around that. Can you say anything more about how structural equation modeling compares to instrumental variables? 

Christine Pal Chee:   I'm afraid I don’t know much about structural equation modeling so it's  very new to me. I don’t know if you do Todd. I could look more into it. 

Todd Wagner:  Yeah, so I'm married to a psychologist. 

Christine Pal Chee:    Oh okay. 

Todd Wagner:   Although she doesn't use it we often talk about these kinds of things. I think the intuition is the same idea, that you're trying to come up with this - they often use what is [inaud.] They come up with these pathways but at some point they're saying something is exogenous that starts the process in motion. So I would argue that what I'd like to see more in the structural equation-modeling world is, and they typically provide very good stories, is more information on the relevance of those instruments. And so often you'll see things like age or genders being the underlying exogenous structure and I just worry that it's  not very relevant. But I'm by far - by any means an expert. That's just sort of spousal information. We have three more questions. How distinct should the IV estimate differ from an RCT estimate? Does this provide an effectiveness estimate in the observational world similar to the goal of propensity score estimates? 

Christine Pal Chee:  Sorry, could you repeat the main question Todd? 

Todd Wagner:  Sure, yeah, so it's  two part. So how distinct should the IV estimated differ from an RCT estimate? 

Christine Pal Chee:  Okay, that's a great question. So it depends. So in the case where the instrument does not perfectly predict or perfectly determine treatment the IV estimator will estimate the marginal effective treatment whereas in a randomized controlled trial you have a group of people who receive treatment and a group of people who do not receive treatment. And there you're able to get an estimate of the average causal effect at least for that population included in the study. So those two numbers should differ if it's  the case that the instrument does not perfectly predict treatment. However, if the instrument does perfectly predict treatment - let me stop to think about this. 

Todd Wagner:  I'm going to jump in if I can for a second. 

Christine Pal Chee:    Yep

Todd Wagner:  So there's two things that are going on here, on is how well your instrument works compared to the truth of the RCT. But the other thing that the underlying populations that were enrolled typically with observational data you're getting very generalizable populations and the RCTs typically enroll a very select population. So you could have either one being off, one being the instrument, or you could have such a select RCT population that it's  hard to make the connections. 

Christine Pal Chee:   Actually that's true Todd. With RCTs often - from my understanding often times patients who are enrolled in the first place are generally good candidates for the treatment or the procedure. And that actually explains the discrepancy between the McClellan estimates of the effect of catheterization on estimates that come from randomized controlled trials. It's  that - the McClellan paper estimates  the marginal effect of treatment. So these are for patients who would not otherwise receive catheterization if they did not live close, very close to catheterization hospital versus an RCT where the patients are probably more likely to benefit from the procedure and it's  across the whole population of patients enrolled in the clinical trial. 

Todd Wagner:  Alright, so I've gotten some comments back that my structural equation modeling is off and they say you can do instrumental variables within structural equation modeling. So I'll beg ignorance at this point and say I need to learn more about structural equation modeling. We've got two people  who would like us - and the second part of that question was about propensity score. And so I want to come back to that because there are two people who want us to compare instrumental variables and propensity scores. 

Christine Pal Chee:   Sure. With - so Todd you covered instruments or propensity scores, was it one or two weeks ago, so you can please jump in an add anything you feel is relevant. But from my understanding propensity scores only account for what you can control for, what you can match by. They account for observable differences and if there are any remaining unobservable differences between the two patient groups, the two treatment groups then your estimates will still be biased. The beauty behind instrumental variables regression is that it allows you to account for unobserved heterogeneity if you have a valid instrument, one that is both relevant and exogenous.

Todd: Yeah,  can you go back to - I don’t know if you can get there easily. You had a slide that compared McClellan's paper where it is his three estimates. 

Christine Pal Chee:   Yes. 

Todd Wagner:  That  one right there, perfect. So in some sense what you're looking at on the bottom panel is no, it's  unadjusted differences gets you the 30 percentage point change in mortality rate. When you just use multi-varied adjustment you get 24 percent. So the question then would become if you use propensity scores could you improve on that? I didn't - there's mixed opinions if you were at my class, I forget, was it two weeks ago or a week ago, there was mixed opinions on whether that would improve it or not. There's a recent article by Brooks and Oschfelt [ph] that says you might actually make that worse. What's striking here is that when you take the instrumental variables account you end up with a much smaller effect that’s no longer statistically significant. So that five percentage points, you notice that the air goes up, the standard goes up. So there's a very different interpretation and it's  because what you're trying to do is not just control for observables but you're trying to think about what's truly exogenous here. So there's a different conceptual thinking about this. Do you want to add anything else? 

Christine Pal Chee:   No, I agree. I think the main point is that the object of instrumental variables regression is to control for unobserved differences and we can do that if we have a valid instrument. 

Todd Wagner:  So hopefully we got - we answered the questions about propensity score although it may be not completely satisfied because there's not a right or wrong answer. And then some people provided some information on the structural equation modeling. So we'll take a look at that. Thank you very much for those of you who did that? Any other questions...? Now you mentioned running these in statistical software. Do you want to say anything more about STATA and how you get...

Christine Pal Chee:    Sure, yeah, in STATA - it's  very nice packages that will run instrumental variables regression for you. And the command I think in the newest versions of STATA is just IV regress. So in STATA or just on the internet if you were to Google STATA, help IV regress you'll get the documentation for that and it'll tell you exactly how the - what to put into the command and the different options. So you can choose to do two stage least squares, or use other estimators. I think in older versions of STATA the command is IV reg. 

Todd Wagner:  Correct. And my head is - I'm still old school. Every time I type IV reg it tells me I'm old. 

Christine Pal Chee:    Actually it does that to me too Todd. 

Todd: And then in terms of other - if people are doing - there was a question about a binary endogenous variable so your first stage model is binary. There are also programs in STATA for running those as well. I think they're called treat reg if I'm not mistaken, or maybe they've been subsumed by the IV regress. I'm just getting to know STATA 13, so. 

Christine Pal Chee:   In general though STATA documentation is actually really good. And so it'll refer to other commands and other related resources. 

Todd Wagner:  So hopefully people found this helpful. We're wrapping up. We're right at the top of the hour. It was a great presentation. Probably one of the best presentations I've heard on intuition instrumental variables so hopefully everybody else enjoyed it too. If you're interested in slides, Heidi, did you want to say a little bit more about how to collect the slides? 

Moderator: There was a link included in the reminder that was sent out this morning. I can try really quick here to bring the opening slide back up. And on here there is actually a link, click here for today's handout. It will open up the slides in a separate browser window and you can just save them or print them from there. So those are your options for getting today's handouts. 

Todd: And then if you - when you close out your system you're going to be asked for comments and we really do take those comments seriously so if you could answer those and provide specifically any text fields, we read them all. We take them to heart, and we try to improve these each year, so thank you so much. And thank  you Christine. 

Christine Pal Chee:    Oh, you're welcome. 

Moderator:  Thank you everyone for joining us and we will be sending registration information out for our next upcoming HERC econometrics course. Thank you everyone for joining us today. 

Page 1 of 13