HERC Econometrics with Observational Data- 17 -Department of Veterans Affairs

**Department of Veterans Affairs**

**HERC Econometrics with Observational Data**

**Econometrics Course: Introduction and Identification**

**Todd Wagner**

**April 4,  2012**

Todd Wagner:  I just wanted to welcome everybody to the first class of the HERC Econometrics course.  We've taught this course a couple times before and so it's a pleasure to teach it again.  We've got a lot of people registered and it's always a very diverse course.  The goals for the course, what we're really trying to do is enable researchers to conduct careful analyses of existing VA data sets and if you will, non-VA data sets too.  I know that there are some folks that are VA-affiliated or affiliated with other federal agencies, they're welcome as well.

The goal really is to give an introduction and to provide description of econometric tools and their strengths and limitations and use examples to reinforce learning.  

Yesterday that was an article in JAMA by [Mary S. Vaughan] Sarrazin and Gary Rosenthal, there's no substitute when you're doing good econometrics for also asking the right clinical questions and so as much as we're going to talk about the econometric side of the equation here, please keep in mind that you have to be pulling the right data and asking the right types of clinical questions.  If you have any questions about that, I thought that was a really nice overview where they described data limitations and how people look at pneumonia trends over time, so just in yesterday's JAMA.

For today's class what I want to do is sort of start out with a little description about the terminology and I know that that can be a huge stumbling block for some people, just because epidemiologists, economists, biostaticians, psychologists, many of you are on the call today—all have different terms for how we think of these things and just that alone can pose a huge stumbling block.

Then I want to talk a little bit about understanding causation with observational data and then what I'm going to use to motivate that is a little bit of discussion about clinical trials and why people tend to gravitate towards clinical trials and then the limitations of clinical trials and what people are hoping to do with observational data.

I tend to think of the work that the CORI, the new outcomes research initiative associated with the ACIA is really trying to look at a lot more that has to do with head-to-head observational trials and can we get more out of the data that we already have.

What I'm going to then do is—because I know people love equations so much, I'm going to describe the elements of an equation, it helps throughout the course, not only in this lecture, but in future lectures, when it comes to talking about what's going on with the model and why we have misspecification.  I'm going to give an example—this might bring you back to high school algebra, but hopefully it's a good memory, not a bad memory, and then I'm going to walk through the assumptions of the classic linear model.

Terminology—like I said, there's—confusing terminology can be a major barrier and Matt Maciejewski and Paul Hebert did a paper in 2002 and I'll have the cite later on in this talk and we provided that paper in the past.  They actually just updated that paper in 2011 and there's the citation's for it, but it really gives you an idea of if you're trying to figure out is this multivariable, multivariate, endogeneity, confounding, interaction, moderations, right, wrong, yin, yang?  You can come up with all these different things, but if you're in an interdisciplinary setting and at many VAs the research is interdisciplinary.  

The questions become—if you're saying, for example, "I want to look at moderation and the moderating effect of age," the economist might say, "Well, can you be more specific when you say moderation, do you mean interactions?"  So we can get through there and I would recommend people to read that paper.  If you have any trouble getting a hold of it, Matt is at the Durham VA and I'm sure he'd be happy to send out—and Paul is at the Puget Sound VA.

I have a couple polls because what's going to happen here is—given this is our first class, it's really helpful for me to understand how diverse the audience is, so I have three polls.  Can you tell me a little bit about your academic training and please choose your answer that best describes your background.  I know there are clinicians with fellowship training in HSR, whether you're a nurse, or a physician or physical therapist with specific training in HSR.  Clinicians, if you have a PhD, a Master's, Bachelor's and feel free to just choose those, I'm interested to see the spread.  Not only will this be helpful for me in my talk today, but I'll also be feeding this information back to the other presenters, so that we can see it and tailor the trainings accordingly.  

All right.  So we have 42 percent have a PhD, not many clinicians out there, 39 percent have a Master's degree and seven percent have a BA.  That's amazing, that's awesome.  Okay.  So the next poll.  So, for those of you, for example, the 42 percent that have PhDs, my guess is they're all not in the same field, they're probably not all in economics, we don't have that many economists, I don't think, in the VA.  I'd love to know your training and same with Master's, Bachelor's and clinicians, if you feel like you have—or you took your statistics course work.  

Todd Wagner:  All right, Heidi, I think you can see what we got here.

Heidi:  There's your results.

Todd Wagner:  Fifteen percent psychology, eighteen percent econ, some math, engineering, that's great.  Five percent never took a—that's also great—and then the last question, if you will, Heidi.  Your level of expertise.  So, for example, if you're a PhD in economics, and you often review for other econ journals or are asked to do statistical analyses, put yourself down as a 5.  If you're just sort of understanding what the average is and you know what the median is in standard deviation, but maybe not as much about [keritosis], you can put yourself down as a 1 and if you feel like you fall somewhere in between feel free to put yourself in there too. 

So a pretty normal distribution, that's pretty amazing, so we got 12 percent out there who say that they're beginners, so welcome to you and we have 13 percent—I wish I knew exactly who those were so that I could call on them to present.  So feel free to—if you're one of those advanced people and would like to send in questions or suggestions as we go along, I'll be more than happy to hear about that, as well as from moderately or No. 4 groups as well.  

This is going to be a challenge to teach this course, for two reasons, one is we have a huge spread here and the second reason is I can't see your faces and I always find it a little bit challenging to teach a statistics course when you don't really know whether you've lost half the crowd, so you'll have to give me a little bit of patience as we go through this.  Please raise your hands, please ask questions if you feel like you are totally lost, because the goal is not to present this in a way that is complete and economic jargon and at a fast pace, the goal is to get everybody there.  So I apologize in advance if some things are confusing, but feel free to ask questions and we'll try to do our best.

Throughout the class, I'll also try to pause and we have another one of the health economists here, Jean Yoon, who's helping me handle questions, so as questions come in, she's going to be responding to those, but also she'll pipe up or interrupt me, if I don't stop and you can ask questions.

In many regards we often hear that randomized clinical trials or RCTs are the gold-standard research design for assessing causality.  Think about it, what's unique about a randomized trial?  In some of the previous classes, we've been able to open up the phone lines.  I see that we have 170 people right now on, so there's no way that I can actually open it up and have you guys answer that question.  So clearly, what's unique about a randomized trial is the experimental design that the researcher is randomly assigning somebody to get a treatment or not to get that treatment and it can be more than two treatments, it can be three treatments, four treatments, but that experimental design, as long as it's done well, helps somebody say something about causation.  

So the treatment exposure is randomly assigned, so think about this, whether it's a drug trial or a procedure trial.  Then when you do this, the benefits of randomization—with a little bit of luck you have some information on the causal inferences.  I say a little bit of luck because if by chance you have flipped the coin and it just—luck is against you, your two different groups are not going to be equal.  

The random assignment really distinguishes between experimental and non-experimental designs.  At our Palo Alto VA we have a lot of psychologists here, so a lot of them have training in experimental design and have thought a lot about how to balance trials and so forth and do it very well.  We also have a large number of people who have no experience in clinical trials, but have a large experience in sort of non-experimental observational studies.

I just want to make sure that when people think about random assignment—we're not confusing random assignment with selection or random selection and so when we think about national surveys, you'll often hear about the sample is randomly selected.  That's very different from a random assignment and really if you want to understand causation in a meaningful way, you also have to think about the random assignment is required.  We'll get into it in—not today's class, but later in the course, we have a class on some instrumental variables, which is the statistical technique that it is trying to mimic, random assignment.  So stick around for that class later in the year, if you want to hear about that.

The limitations of randomized clinical trials: One is generalizability.  For example, if you're interested in looking at whether off-pump versus on-pump cabbage surgery—and there's a big trial in VA that did this, the questions are: Is it generalizable?  Your inclusion criteria may have resulted in a select sample, such that your result may not generalize to the world or even to VA and when that main paper was published in New England Journal, the main criticism was that the VA facilities that were conducting the surgery aren't high enough volume to really have shown the real benefits of off-pump.  So there is this question of generalizability.  You could have this Hawthorne effect that when people are observed, they change their behavior.  

Undeniably randomized clinical trials are expensive and slow. Many of the trials that we work are multimillion dollars, take many years to complete.  You can also think about a case where it would be unethical to randomize people to certain treatments or to conditions.  So there's a question that we've been interested in about whether access or use of specialty care improves patient outcomes.  Well, you couldn't imagine a randomized trial in which case you randomize people to specialty care and other people cannot get specialty care.  Probably the classic study in this case is a study that Mark [McClellen] did, looking at heart attacks, and he suggested that there would never be a way to do a randomized trial, looking at what he was interested in, so he developed this instrumental variables model to sort of simulate the randomized trial.  So this quasi-experimental design, depending on the design can fill an important role.

So can secondary data help us understand causation?  Clearly they can confuse us and for those you who know me, I love coffee.  I'm sort of a coffee fanatic, I roast my own coffee, so don't get me started about coffee, but here are just some headlines from newses about coffee.  Coffee may make you lazy, it's not linked to psoriasis.  It may decrease the risk of skin cancer.  It poses no threats, although we've heard it also makes threats.  Here's another that may make high achievers slack off.  So I don't know if that's a good thing or a bad thing.  An effective weight loss tool, so clearly secondary data can be problematic in the way you analyze it, especially if people try to infer causation from it, but there are things that secondary data are good for and that's what we're going to talk about.

One of the things about observational data is they're widely available, especially in VA.  So if you have access to the medical fast data sets, it's amazing that quite quickly you can see all the utilization in VA for over five million encounters, five million veterans, I should say, on an outpatient system that's—I think it's over 100 million encounters now.  Then on the inpatient side, it's just shy of a million encounters.  So you can do quick data analysis at a low cost.  It can be realistic and generalizable, you could say something about what's happened for all cabbage surgeries or percutaneous interventions in VA and what's happened in trends.

The challenge in many of these is that you often are faced with a lot of dependent variables, questions that are endogenous and key independent variables may be lacking.  I'll talk a lot more about this question of exogeneity and endogeneity as we go here.  Just to be clear, a variable said to endogenous, when it is correlated with the error term and I apologize for being statistical jargony there—but think of it this way, if there is a loop between the independent and dependent variables, such that you're not really sure what's causing what, then there's the problem of endogeneity and here's a case, where if I were looking at your faces, I could probably see who understands this concept and who doesn't.  

I often rely on a medical analogy that I think drives it home for most people and it has to do with hormone replacement.  We all think about the different hormones that we have in our body, those are your endogenous hormones.  One see these studies—looking at, for example, a sample of men and could say, "Well, let's look at the relationship between endogenous testosterone and muscle volume and you could say, "Wow, higher rates of endogenous testosterone is linked to muscle volume," but that doesn't necessarily address the question about what's the effect of exogenous testosterone or a testosterone injection?  With hormones, it's quite clear that there's an endogenous and an exogenous.  When we delve into the social sciences, try to keep that in mind, too, there might be situations where there are endogenous variables [inaudible] that you're really interested in, that you're really trying to figure out what's the exogenous role there.

Endogeneity happens for many reasons: There can be measurement error.  There can be autoregression with autocorrelated errors.  There can be simultaneity, such that the dependent variable and the independent variable happen at the same time or are sort of co-determined.  You can have omitted variables in your model and your off-sample selection.

Now I apologize, I'm already jumping into jargon and I haven't even described the equation, so I'll get to the description of the equation, but if you're one of those people who's just starting off in statistics and you might be feeling like you're lost already, just hang on and we'll get you there.

So let's get on to the equation: So let me just talk a little bit about the terms here, so you have univariate and what I'll talk about is a univariate is a statistical expression of one variable.  Often we'll have histograms to depict a univariate distribution.  A bivariate is we're looking at two variables, hence the "bi", and then what I say is multivariatal is the expression of more than one variable and some use the term: just univariate and multivariate and don't use the term bivariate.  Many people think of multivariate as more than two, but multi is just more than one really.  

So here's your equation—if you were a stellar or even not so stellar algebra student in high school, you might remember the equation of the line, which is on the bottom of the screen, it's the Y=MX plus B, you should see similarities here.  So Y is your dependent variable, B is the nod at your intercept and then you can have a covariate or a right-hand side variable.  We say right-hand side variable because it's on the right-hand side of the equal sign.  You could say it's a predictor, independent variable.  Independent variable, I don't usually use because that means it's exogenous, and in cases that there is endogeneity that gets a little confusing.  

Then clearly the distinction between a determined line, the y=mx+B, and this statistical equation is the addition of the error term.  So the error term is important, just to go through this in more detail, we see that there's a small subscript "I" here, "I" as in index.  If we're analyzing people, then this typically refers to the person or unit of analysis, but if you're analyzing medical centers, this could also refer to a medical center, so it doesn't have to be a person.  There can also be additional indexes, I didn't include it here, but the most common one that we often see is Yit, which might be, for example, people over time and the "t" would be the time index, "t" is often used to refer to time.  Your dependent variable is your "Y", your intercept—let's say we're extending this now to two covariates, so now we have X and Z and then your error term.

We can extend this, if we want to with "J" covariates and you could say we're really interested in X and Z and then there's all these other covariates that we want to control for and those are in that notation of the summative sign, it just means it continues on for this other vector of covariates.  So you might see these types of equations as you read journals and I would suggest that it just takes a little bit of time to parse your way through it and it's very helpful to understand the basics here.

Still the error term—the error term exists because important variables might be omitted by accident typically because they're not measured.  There might be just measurement error, there are a lot of things that we try to measure that we may measure imperfectly.  So even if we're thinking about, for example, body temperature, something that most of us have measured on ourselves or our kids, there are measurement errors and you'll recognize quite quickly that if you think your child has a temperature, you take their temperature under the arm or in the mouth, orally, that even if you take it five minutes later or two minutes later, you'll get different readings, different units will give you different readings and there's often a sort of a measurement error complicit there.

There's also this idea that there's just human indeterminancy, which is really a—it's sort of come up in the philosophical world, but it also has a physics meaning, so if you're a physicist, and you're familiar, for example, with the Heisenberg uncertainty principle, or you're really interested in the position of momentum, there's really these things that are really hard to figure out and be deterministic about.  In philosophy, it really has to get down to this relationship of we're all really interconnected in very complex ways and you might not be able to say determine that A determines B.  Things could be very subtle and this can create what we think of as error.  

Understanding the error structure is a huge challenge for econometrics and one of the things that we're frequently trying to do is understand that structure and then to minimize your error.  Your error can be additive, multiplicative, there can be different ways to think about that error term.  So that error term is really important in the way we think about our observational data.

So I'm just going to walk through a very, very basic example of: is height associated with income?  This is just a made-up example, but it's really trying to help people who are just trying to get a sense of what do we mean here by these different terms and what does it look like?  So here is the equation—what we're trying to figure out, does height determine your income?  Is it associated with your income?  So Y is your income, so that's your dependent variable, and we're going to be regressing that on this equation and that equation is including your height, so beta not becomes your average and then you have your error term.

So the hypothesis that we're testing is that height is not related to income.  So if beta 1 equals zero then what is beta not?  The answer and it might not be immediate—again this is one of those questions that I would have opened to it, but we have 180 people, so I can't.  It's just the average income, but there's no relationship, it's just your average income.

So here is the description, here is your diagram and so here's the question, here's your height and income on your X axis you've got height and that's in inches.  On your Y axis, you've got income and typically I would say that's your annual income, although it could be monthly income for some people.  Let's just think of it as the annual income, so how do we want to describe the data.

You might say, "How would you like to describe the data?"  Well, one way to describe the data is with an estimator and just think estimators are really simple statistics that provide information on a parameter of interest, in this case, height.  We're typically applying a function to the data and there are many common estimators that people use all the time.  So when I use the term "estimator", don't freak out and say, "Oh, my God he's using a term 'estimator' I've never heard that before."  You often use estimators, you talk about means and medians.  You often probably use things like ordinary least squares, if you're familiar with that term or linear regression, if you're familiar with that term.  So we use estimators all the time.  So here's your ordinary least squares estimator, by default, what it's trying to do is minimize the vertical distance between those red dots and that blue line. I thought at one point I had green arrows, but I think my green arrow has disappeared.  I was actually showing that vertical distance.  So by default ordinary least squares is minimizing, hence the term least squares.  It's minimizing the distance between the red dot and the blue line, but there could be other estimators here that one might be interested in.  

You could think of the least absolute deviation, the maximum likelihood.  You could have a non linear estimator, for example that's graphed at the right.  You could think about very different estimators here.  One of the challenges you have as an analyst is trying to figure out what's the right estimator to describe this population or this sample, that becomes incredibly hard when you have cost data and you've got these very unusual distributions with some long right-hand side tails.  So this choice of the estimator is important.  

So there are ways that we think about choosing estimators.  One is you could say, "We want to have least squares"—you could say something about we want to make sure we choose an estimator that's unbiased, that's always good.  We don't want to have bias," that's like saying something is wasteful, well wasteful is another way of efficiency, minimum variance.  You could say, "Well, there's other properties, there's asymptotic properties."  You could have maximum likelihood, you could have goodness of fit statistics that will help you choose your estimator.

Throughout the whole course, we're going to talk about the way to choose your estimator and specifically the next two classes are talking about analyzing cost data and Paul Barnett is doing those and he'll talk a lot about the choice of estimator, largely of least squares, general linear models, with some other types of distribution for data or [inaudible] laws models and trying to come up with the right estimator.  It's very important for describing a very challenging distribution.

Now here's those green arrows.  How is the OLS fit in this case between height and weight.  You may say, "Well, that looks pretty good," the taller you are, the more money you make.  I'd like to say I'm six feet tall, but I'm 5'10", so that puts me at 70 inches, so that puts me out here—I wish I could make more money, so that means I need to grow.  Again, it just sort of highlights the fact that is this a causal?  Maybe not causal, it might be related to other things.  But you might also say, is that the right fit, we see different things going on, it seems to fit the data better at the right-hand side of the distribution, then the left-hand side.  We can figure green arrows on the left-hand side.

What about gender?  Could gender affect this relationship between height and income?  So you might say, "Let's generate a gender-specific intercept.  Let's do different lines for men and women."  You could also say, "Maybe there's an interaction here," different slopes.  So what does that look like in terms of the equation?  So if you just want to say there's different lines for men and women, you're just going to put a gender intercept in here, so beta 2 Z is your gender and Beta 1 X is your height.  So what does that look like?  Now we have two lines, the red line is the women's line and the blue line is the men's line.  It's just the slope because by construction they're having the same slope, but the only thing that's different is the gender in intercept, so beta 2 is at gender intercept that moves it from beta not up to beta 2. 

You might say, "Well, that still doesn't quite look right to me, what about the interaction term?"  What an interaction term is, we still have that intercept, we have to keep the main effect in there for height and gender, but we're also going to multiply the two for an interaction.  

Now, if you're a psychologist, you might have heard the term effect modification or modifier.  We're interested in that question.  Do they have different slopes.  Here's the question: Do they have different slopes?  It looks like they have different slopes and you could actually test this in a statistical model, we don't do that, but you might say for men it looks much more deterministical, much more related height and income and I would be very disappointed here, because now I'm on the left-hand side of the male line, whereas it means less for women.  I know there's actual research that's looked at height and income, but please don't infer that this is a causal relationship.  I just made up the data to show this example.

I'm going to stop before I move on and make sure that there's no questions before I continue to lose people.  Jean, how are we doing?

Jean Yoon:  No questions yet, Todd

Todd Wagner:  I have sufficiently confused everybody and they're all checking their e-mail and Blackberry right now.  I'm going to move on to the classic linear regression and we're going to talk about the assumptions.  Hopefully, people are thinking about how are these statistical assumptions built into models and how do we violate these and what do we do when we violate these assumptions?  What I'm setting up here is basically setting up the discussions for the future classes, I'm going to walk through the five common assumptions and ways that those are broken and then I'm going to talk a little bit about who's going to be teaching the class that's really going to address this and why it's an important issue.

The first thing to keep in mind is that there's no superestimator, there's no estimator out there that is always the best estimator, but think about it this way, the classic linear model is used as the starting point for analyses and we often find in cases that they worked out really well.  They often are much better off than we think they're going to be.  

There are five assumptions underlying them and then the variations in the assumptions will guide your choice of estimator, sort of your happiness and the happiness of the reviewers for your paper at journals and so forth, so these are important assumptions.

Assumption 1, is that the dependent variable can be calculated as a linear function of a specific set of independent variables plus an error term, so we've shown this equation before and so you're making an assumption that there's a linear function that is appropriately fitting your dependent variable.  So there are times when this assumption can be violated.  There are times when the violation is a minor violation, there are times when it's a major violation.  So you can have omitted variables that caused this assumption to be a problem.  You can have non-linearity and there are different ways of handling non-linearities, one is by transforming the independent variables.  For example, if you're really interested in cost, you might say, "Well, the relationship is not linear it's log linear," so that one way of handling it would be to take the natural log rhythm across and working with that.  You can think about these different types of functions.

You can have theory-based functions.  There are a couple different theories out there.  For example, if you're familiar with Prospect's theory, which is a psychological theory about how people make decisions that could guide your thinking about how this function should look and it might say, for example, that it's not perfect to have a linear function.  You might also, for example—you're interested in discount rates and you might say, "Well, the real belief here is that it's an exponential function," so you might have a model that's exponential.  You can also have empirically-based transformations when you do your different transformations and you go through a certain set of them, you're looking for ones that, for example, that meet empirical values, for example, best fit.  You might say something about common sense.  Common sense—I often think about it as it makes sense to you, but it's also very easy to explain and it's intuitive for a lot of people to understand, that's not always bad, especially if you think about these much more complicated models and if you show that those much more complicated models add very little or add nothing.  Then it may make sense to go with the common sense easiest one.  The Pregibon Link test is a very simple test that's trying to understand the functional form of your dependent variable.  There's also what's known as the Ramsey RESET test and there are statistics for it.  

Keep in mind that many of these tests tend not to be particularly powerful and you often need larger sample sizes to run them, but we often use the Pregibon Link test when we're looking at, for example, whether we should be using OLS for cost regressions or semi log or GLM for cost regression, it's often used there.

I just wanted to take a brief detour, if you will, I know some people are going to disagree with me when I talk about stepwise regression.  When I was trained in multivariate analysis, I took a class in epidemiology in public health and it was a very interesting, eye-opening experience for me and very different from economics because the questions were often about you've got an E-coli breakout or you've got a disease breakout and you're trying to come up and you have limited data sets and you're trying to come up with a parsimonious model that identifies the cause of that outbreak.  To me, when I think about that, there's a lot of reasons for using stepwise and the stepwise is that you're trying to use your statistical model to kick out variables that are not associated.  Typically it's the P value of less than two, those values are [inaudible] greater than two, they're thrown out and so you can end up with this very parsimonious model that's great if you have a very small data set, but it's also nice in epidemiology when you've got these outbreaks that are ostensibly exogenous, that really aren't related to the system.  

In health services research, I have a really hard time with stepwise regression and there's a statistical reason for it.  One is the indeterminancy that we talked about before that there's these very complex things and I think it's often a disservice to just throw out variables out of your model.  There's little penalty for adding a nuisance variable in your model, there's a big penalty if you drop something that's really important and there can be reasons, for example, high colinearity between variables that you might drop one, when it's actually important to keep it within the model.  It could be important for theoretical reasons and whatnot.

In VA, we have these huge data sets and so I often say it's very easy to keep a lot of these nuisance variables in the model and there are ways to make your table smaller if you need to do so for reviewers, but I think that it's beneficial to have it.  

I've also done a meta-analysis in my life and it's very hard to combine studies when people do stepwise and have very different covariates in the model and it's not always clear what they're controlling for.  So that's another thing that I would say is a reason for being careful about stepwise regression and assumption No. 1.

Assumption 2—the expected value of the error term, so keep in mind the error term, we're expecting this error term to have this—that it's independent, identically distributed—we often think of like this little normal curve of error and we expect that to be centered around zero.  If we have violations to this, it can lead to biased intercepts and this is particularly true when analyzing cost data because no matter how many parameters we tend to have in our model, the error terms are still really weird for cost data.  You have these—what I think of as train wrecks, people who have had extremely high costs stays that are so different from any other person in the data set and that really changes your distribution as well as your error term.  So keep this in mind, Paul will be talking about the error term assumption when he talks about analyzing cost data in the next two lectures.  

Assumption 3 is that these error terms are independent so that if we see one person in the data set that that person's error term is not related to somebody else's error term.  You can think about reasons why you might have dependence, so if you're interested in— for example, comparing across facilities—the common one here is doctors.  That doctors can be treating patients in certain ways because of their preferences or selection of patients to physicians, that causes this clustering at a doctor level, so that makes this dependent. 

We also assume that there are identically distributed error terms and often we'll have what's known as heteroskedasticity.  We're assuming this is homoskedasticity, this is not an infectious disease, although it sounds like it, but here's a diagram of heteroskedasticity, so here's length of stay in a bed section, bed section is a VA term for like a ward, if you will, and here's your cost, your national cost and this is a typical relationship we see with cost data is that the longer you stay, the more dispersed the data become in terms of the error, and that's true there.  So you'll often see these heteroskedastic patterns in the data.

What happens if you violate Assumption 3?  Well, one of the first things to think about is that the OLS coefficients, for most of the time, the linear regression, doesn't bias them per se, but it means that they're inefficient.  So you could be saying that there's no effect, when there really is an effect.  It can also create biased standard errors, it can make you leap to the wrong conclusions and plotting is often very helpful.  There are different statistical tests trying to figure out if you've got heteroskedasticity going on.  The most general test for heteroskedasticity are probably the least powerful, the most specific are more powerful, but, again, it's that question about how specific can you be and how general can you be.  They all have some limited powers, so you have to be very careful about it.

There are fixes for this, one is to transform the dependent variable such that you're not necessarily looking at a cause, but you're looking at a different function, maybe you're looking at daily rates, the average daily cost.

Another way to think about it is there are a set of standard errors, called robust standard errors, and they also go by Huber White or sandwich estimators that are very helpful and are robust for heteroskedasticity [inaudible].

Assumption No. 4—what I think of is the endogeneity assumption, what it really is saying is that observations on independent variables are considered fixed in repeated samples.  If you were to re-sample the observations are independent.  The statistical mathematics of this is you're Xi from your regression is uncorrelated with your [ui] from your error term.  So the expectation is zero.  So no correlation between your covariates and your error term.  We often have—because of endogeneity, this implicit correlation between something that's unmeasured and something that's measured on your right-hand side.  

So the classic one I think of is smoking and if you were to do a regression and say, "We're interested in whether smoking is associated with health," you would regress health on smoking behavior.  Well, we know that implicitly there are many things that we're not controlling for when it comes to smoking.  It could be family love that caused you to smoke or lack of family love that caused you to smoke.  It could have been your parents were smokers.  It could have been a school you went to.  It could have been your friends were smokers because you're often not capturing all of that information, there is an implicit correlation between smoking behavior and your error terms.  Some of that information on smoking is picked up on your error terms and that's what causes that endogeneity and that's what biases that variable, so it's a very important assumption.  There's other ways that you can get this, there's errors in variables, there's autoregression and simultaneity and there's simultaneously determined and so forth, but this is a really key assumption because if you screw up, implicitly it makes a bias in your estimate.

The measurement of the error term of the dependent variable is maintained in the error term, we know that.  OLS assumes that other covariates are measured without error, but here's the errors in variables.  The one that I like to talk about here, for example, I try to give more of a psychological explanation, but let's say you're interested in patient outcomes and whether someone's locus of control or self-efficacy affects their outcomes.  Well, no matter how many good measures of self-efficacy or locus of control that we have implicitly there is some misestimation or measured inaccurately and that can cause problems specifically—the level of that inaccuracy varies across the population, sort of this unobserved heterogeneity.  So that can be a big problem as well.

Another common violation is if, for example, if we're including lagged dependent variables as a covariate that often creates problems right off the bat.  

You can have contemporaneous correlation.  You can have tests for this, the Hausman test is a very famous test, but it's very weak in small samples and then instrumental variables offers a possible solution.  Many of you have heard instrumental variables or at least that term, and it's very distinct from propensity scores.  We're going to have classes later on, talking both about propensity scores and instrumental variables.  Propensity scores is just trying to make the most out of your variates that you have measured, instrumental variables is trying to mimic randomized control trials, using variables that are plausibly exogenous.  I wish—I saw Mike Pesca's name on the attendee list, so I'm going to give a shout-out to him.  He is a researcher who has done some great research on smoking and taxation and [inaudible] at CDC.  So at some point, I'd love to have him give a cyber seminar, talking about his work, looking at instrumental variables, but again that's an issue where you're trying to figure out these plausibly exogenous issues like taxation does that affect somebody's smoking propensity and their outcome?

Assumption No. 5—that you have more observations than covariates.  We're not going to spend colossal extra time talking about  this assumption.  This is a very small assumption, but clearly if you've got small data—let's say you have 20 cases, 20 observations, you cannot put 25 covariates in your model, you'd have more covariates than you would have observations.  You also have to make assumptions about multicollinearity that you can't have perfect collinearity, [inaudible], they'll drop a variable that's perfectly collinear stat, that sort of hangs up on that and says you can't estimate it.  What it's telling you is that it can't invert the matrix because it's all based on matrix algebra, it can't invert that matrix.  

What typically people will say if you've got high collinearity, you can have problems and that's true.  You can have two variables that are highly collinear, both look non significant, both are important variables and the best answer to get around that solution is to either remove the perfectly collinear variables or increase your sample size.  The more sample size you get the more power you have to disentangle these collinear effects.

Remember the standard error is the standard deviation over the square root of your sample size.  So if that square root goes up, your ability to de-tangle these things increases.  Any questions?

Jean Yoon:  We have one question if I can pull it up.  "How would you interpret highly significant effect sizes when the sample size is huge?  So if the control group is much bigger than the treatment group?

Todd Wagner:  There's a couple questions embedded in this.  One is getting back to that issue of standard error, so keep in mind that your significant testing is often based on your standard of error.  So you got your point estimate, your standard errors generating your confidence intervals.  If your standard error is your standard deviation of the square root of N, you can quite quickly see that if you have millions and millions of observations, your standard error is going to be very, very small.  It gets you huge power to detect meaningless differences.

So then the question is: when you've got large sample sizes—there's no golden rule to tell you about what is the right size that makes it large.  I've seen people say 30-, 40,000, but clearly if you have hundreds of thousands of cases, millions of cases, you've got a lot of power and you're going to be protecting effects that may not be clinically neutral.  Those are important to work through with sets of researchers to figure out—okay, so what does that effect really mean in terms of changes in behavior or changes in outcome?  So it's sort of predicting your dependent variables, saying are we really just talking about like a half a percentage point here or are we really talking about like a 20 percentage point difference here?

Now, of course, depending on where you are in your distribution—let's say you've got one group that is 90 percent of your distribution and one group that's 10 percent of your distribution, you can see how the power is differential [net].  The statistical methods are pretty good at that.  Again, if you get really rare cases, you just have less power to detect really rare cases, so, for example, if you're interested in discharge mortality for heart attacks, in the 1980s, you might have been able to, with a couple thousand cases, been able to look at that.  Now, with medical care being so good and discharge mortality being so low, probably even if you had a sample size of 5,000, you're probably only going to have three or four people in that category who died at discharge or died in the surgery. 

So the only answer to that really is to either you need a much, much bigger sample size or you need to change your definition to one that's more meaningful, maybe to a one-year mortality, that's more meaningful to look at this group or you could say, for example, I'm much more interested in blood flow and [patency] through the heart, that one might be predictive of mortality down the line, which we don't have time to observe in the study.  Do you want to add to that, Jean?

Jean Yoon:  No, not right now.  There are several more questions here, so I can go ahead and give you the next one.  There were several questions that wanted you to clear up some of the terms that you've used?

Todd Wagner:  Sure.

Jean Yoon:  They wanted you to clear up what the meaning of asymptotic properties, lag dependent variable and contemporaneous correlation.

Todd Wagner:  Okay.  

Jean Yoon:  The first one was asymptotic properties.

Todd Wagner:  Right.  Okay.  Asymptotic properties means as your sample size goes from finite, from small samples, to incredibly large samples, you—the question then becomes how does the properties of the estimator change as you increase your sample size and go to sort of this infinite limit, so it's a statistical term for moving to infinity or the probability limit of moving to infinity.

Jean Yoon:  The next one was lag dependent variable?  

Todd Wagner:  Lag dependent variable is a good one.  For example, you might be interested in somebody's smoking behavior today—well, probably the best predictor of their smoking today is their smoking behavior yesterday, that would probably be a very strong predictor, but it's clearly—it's correlated with the error terms in your model.  Yesterday's behavior is still related to all sorts of things that you're not really observing.  So you've got to be very careful about putting in lag dependent variables in your model, so just be careful about it because they're not always the answer that you think it is.  Feel free to expand on these, Jean.

Jean Yoon:  Yeah.  So you can have lag independent variables, where you use last year's drinking, to predict next year's drinking, so that could be another example.  

Todd Wagner:  You can, yes.  You just have to be very careful about it.  There's a whole set of [Grainger] causality tests that—if you believe that lags are important, you can also imagine leads being ways to test that out.  If you're interested in medical center spending, you could say, "Well, a good predictor of spending this year is last year's spending, but last year's spending could be endogenous.  So what about future spending is predictive of this year's spending and that's a Grainger causality test and if you find that it's not, then it helps you understand what's going on here with your leads and lags.  

Jean Yoon:  And the final term was contemporaneous correlation?

Todd Wagner:  I think of those as simultaneity, so that the contemporaneous means that it happens at the same time and so often we think in economics of supply and demand as being simultaneously determined or contemporaneous, so that can cause problems in your model.  If you're sort of saying well, which came first the chicken or the egg?  And you're saying well, they both came at the exact same time, so they're both happening, depending on each other and there are statistical ways to try to figure that out—simultaneous models [inaudible] variables.

Jean Yoon:  The next question asks: "Is there a distinction to be made between the distribution assumption of the error at each X and the distribution assumption on the error across all values of X?  Once X is correctly known can the distribution across all values of X change?

Todd Wagner:  [Pause]

Jean Yoon:  I think it might be helpful to have a little bit more clarification on this question?  Todd, are you still with us?  Todd, we can't hear you.  Okay.  So we'll get some more clarification from the questioner about this question, I'm not sure what this is asking exactly.  Hopefully we can get Todd back on the line.  So the next question asks: "Do economists ever use cluster analysis techniques to deal with heteroskedasticity?"  Okay.  So Todd just messaged me, "Help".  I guess Todd's microphone is [inaudible] working right now, so hopefully maybe we'll get Todd back on the line, maybe he can sign out and then sign in again.  

[Heidi]:  I apologize to our audience.  We're trying to figure out the best way to do the audio for these sessions and we've been testing out USB headsets.  Yes, Todd is calling in right now, so he should be back with us in just a second.  I apologize to our audience.  We are trying to figure out how some of this stuff works again.  Unfortunately, it looks like the headset was not a successful test today.  We've used it a couple times very well, but obviously not today—it wasn't successful.  Todd, do we have you back on the line yet?  Todd, do we have you?

Todd Wagner:  Yeah.  I'm back.  

Jean Yoon:  Thank you.  I asked for clarification on one of the questions.  Another quick question in the meantime asked about, "Do the economists ever use cluster analysis techniques to deal with heteroskedasticity?

Todd Wagner:  Yeah.  We do it all the time.  The cluster analysis routine is specifically with large clusters.  Let's say you had a study across ten medical centers or five medical centers, your clustering breaks down if you have very small numbers of clusters and clustering is actually predictably much more useful and powerful as your clusters go up.  So if you had, for example, 150 clusters or 1,000 clusters, then the statistical technique for identifying the cluster is much easier and much more robust.  If you're interested in that specific, there is a paper by Guido [inaudible] and Wooldridge that was an NBER workshop that talked about clustering and it walks through the cases when you have small numbers, large numbers and medium numbers of clusters and why clustering can be useful, but why it can also breakdown.  Clustering is an option specifically if you're doing sort of the time series data sets, if you're looking at patients over time or medical centers over time, the clustering option is useful.  Not only does it control for the heteroskedasticity, but it can also handle and help your other properties and your variables [inaudible] across the facilities do.  So it can be a very useful, powerful technique. 

Jean Yoon:  I think of clustering as patients or providers might be clustered in certain units like the hospital or the clinic, so that's one way of dealing with this correlation between units within these larger groups.  Heteroskedasticity also refers to the error can change with your variable, so, for example, if your Y variable—the variants get larger as the Y increases.  So there are other techniques using the robust options [inaudible].

Todd Wagner:  The funny thing—just as an aside.  I was working on CSP trial once, we were doing a secondary analysis and I was wanting to see—it was a cabbage surgery trial and we wanted to see if outcomes were better for patients who had their surgery done by the attending surgeon rather than the resident surgeon.  The funny thing I did this whole thing and I was like, "Oh, obviously we got a cluster, use a cluster here to us back to what's going on in the medical center, but we had a very small number of trial sites and even fewer when you looked at the actual number of sites that were participating both in the resident and the attending surgeons and so it was very confusing to all of us, we had four PhD statisticians working it, including myself, and the funny thing is it was not significant until you controlled for clustering and then once you controlled for clustering, it was very significant and the outcomes were much worse if you were getting a resident.  So everybody was sort of scratching their head, why does this matter?  Somebody pointed me out to the Guido [inaudible]'s paper and walked me through it, realizing that in that small number, the clustering option was actually causing problems.  So it's really particularly helpful when you have larger data sets.

Jean Yoon:  The next question asks: "Any comments about stratified surveys to increase survey power and decrease data set sizes?  Are stratified surveys useful in your work?"

Todd Wagner:  I'm going to also request a little bit more clarity.  I guess the distinction here between—there's stratified sampling techniques.  There's stratified randomization techniques.  I often will do stratified analyses when I do my analyses as a way of trying to figure out whether an interaction effect is going on.  So I'm not exactly clear, I understand the concept to that question, maybe you get it, Jean?

Jean Yoon:  I think this also might refer back to the clustering question, so if you're looking at patients in clinics and you look at the correlation of the patients between each other within the clinics, if that's high, then it doesn't matter how many patients you have, it matters more about how many clinics you have.  So that will increase your power.  If it's is the opposite situation, if there's very little correlation between patients within the clinic, then it doesn't matter about how many clinics you have, if you want to increase your power, you need to increase the number of patients.  So I think of stratisfying in terms of individuals nested within other units.  So hopefully that answers your question, if not—does that answer your question?  

Todd Wagner:  I hope so, nicely said.

Jean Yoon:  Okay.  So the next question says, "In what way can measurement error in a covariate lead to endogeneity in the model?"

Todd Wagner:  There are different ways that it can lead to the problem, but the clearest way is that there is measurement error, but it's systematic measurement error, such that you can imagine it being correlated with a dependent variable, so that if you're interested in, for example, the relationship between homoglobin A-1C and patient outcomes, so hemoglobin A-1C is a measure of blood sugar and this is used in diabetes, if that's your independent variable and let's say there's some error in your measurement, but the error in your measurement gets worse with the healthier patients are—for some reason.  So you can imagine correlations of that magnitude would then create endogenous biases in your data sets.  So there's a lot of concern about calibrating tests to make sure that they're not only accurate, but precise and they're not sort of biased, particularly for certain subgroups, but that's also why psychologists spend a lot of time with the psychometrics.  If you try to do the same thing as you would on a diagnostic test, trying to make sure that it's not somehow picking up something that's going to be error-term related.

Jean Yoon:  Okay.  The next question asks about stepwise regression.  "There's a lot of people seem to hate stepwise, do you recommend data mining methods and topic knowledge to shrink models or do you not shrink models at all?

Todd Wagner:  I can't say that I don't shrink models.  There is a balancing act between having very large models and what is acceptable for what journals will accept.  Journals will always push me back on why do you have all these things?  For example, in a latest model I must have had 40 covariates, well, 40 covariates with their standard errors can go on for two and half to three pages.  For example, there was many dummie variables for medical centers, so I just reported that it included the fixed effect for a medical center, but I didn't actually report the beta coefficient for each medical center.  

Getting back to the broader issue of stepwise versus non-stepwise, it's been years since I've done a stepwise, probably more than ten years since I've done a stepwise, I just don't do it anymore, but that's just me and my field and I could be odd there.  Do you use it, Jean?

Jean Yoon:  No.  I know that it's commonly taught in biostatistics courses, in econometrics it's more about trying to figure out what your true conceptual model is and then putting in parameters to estimate the parameters in your model, whereas biostatistics just sort of throws out things that aren't significant, so it's just a different methodological approach to building a model.

Todd Wagner:  I think that's a very wise way of thinking about it, if you were doing a genomics study, you might have to throw things out that weren't significant, just so that you could actually invert the matrix and move on.  So then the question is: how do you build your models, using this data mining?  I guess my issue is I'm not often involved in data mining purely, maybe that's the way I should have thought about that.  

Jean Yoon:  Okay.  Another question asks if Y is highly skewed shouldn't the right model reduce the skewness of the errors or eliminate it?

Todd Wagner:  If Y is cost because costs are almost always skewed shouldn't the right model—there are models for cost variables that I have never been able to fix the skewness.  You can improve the skewness by taking the natural logarithm of it that pulls in that skewness and pulls it in and helps fix it, but is it always going to fix?  No, it's never always going to fix.  You want to add to it, Jean?

Jean Yoon:  Well, Paul Barnett is going to go into more detail about models to predict costs, so he'll go into all different ways to address the skewness of cost, so I just wanted to point that out.

Todd Wagner:  How much do you deal with skewness when you do your analysis?  Do you focus on it a lot?

Jean Yoon:  Yeah.  I mean in looking at cost that's a common dependent variable in the studies that I work on.  So I would say that I use a lot of similar methods that Paul uses and there are different ways of dealing with skewness like you can create categories for variables, ordinal categories, [inaudible] dependent variable that might work better to building your model.

Todd Wagner:  Right, right.  Good point.

Jean Yoon:  I think maybe we can just answer one more question here and 

that's—somebody just wanted to point out stepwise is more useful for predictive models, whereas explanatory models could use variables based in theory or a conceptual model.  So it wasn't a question, it was just more of a comment.

Todd Wagner:  Okay.

Jean Yoon:  So that's it for the questions.

Todd Wagner:  I appreciate the comments and the questions that people raised.  So the next class is going to be like Jean said, it's going to be Paul Barnett speaking on the first of two lectures on modeling cost data.  It's broken into two lectures just because there's a lot of material to cover and much of the things that we actually do with the VA data is analyzing the cost data, so it's valuable information, we just want to make sure we go through it thoroughly, but it's really getting back to that question about the distribution of your error terms and whatnot.  So special thanks to Heidi and for Jean especially when I got cut off and helping me hold down the fort, I appreciate that.  Hopefully people enjoyed it, if they have other questions, they're welcome to e-mail us—either us directly or at herc@va.gov and we'll get back to them.

Heidi:  If our audience could take a moment as you're leaving, you'll have a feedback survey come up on your screen, if you could take a moment and fill that out, we would very much appreciate getting some feedback from you.  Thank you all.  Thank you, Todd and Jean, we appreciate the time that you put into today's session and we will see our audience, we hope to see you at our next econometrics session.  Thank you all.

Todd Wagner:  Thank you, Heidi.

**[End of Recording]**