**Transcript of Cyberseminar**

**HERC Econometrics with Observational Data**

**Cost as the Dependent Variable (Part II)**

**Presenter:  Paul G. Barnett, PhD**

**December 4, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or HERC at herc@va.gov.*

*Dr. Barnett:*Welcome to the second part of my presentation on using and analyzing with econometric techniques cost when cost is the dependent variable.  I wanted to start by talking a little bit about what we covered last time just to refresh your memory from two weeks ago.

It’s possible to evaluate costs using the ordinarily least squares, that classic linear model.  The dependent variable that is assumed in that classic linear model can be expressed as a function, linear function, of the independent variables in an error term.  Y in this case is cost and is a function of some X or combination of X’s.  X might be an indicator of group membership.  It could be some covariates like gender or age or presence of a chronic disease.  That epsilon, that E, that thing—the Greek E is the error term, and so we don’t exactly predict perfectly the cost, but we estimate a model that minimizes the sum of squared epsilon, sum of squared errors.

That’s the classic linear model, but it turns out for costs we have to make a lot of strong assumptions, and those strong assumptions—one is that we—our expected value of all of those epsilons that is—they take the mean of those—is going to be zero.  The errors are independent.  The idea that one error term from one observation is independent of an error term from another observation.  The errors all have an identical variance, the variance of the error in the population, which is shown here as sigma squared.  The errors are normally distributed and are not correlated with the X’s, those independent variables.

These are the five assumptions of ordinarily least squares, and it turns out that when cost is our dependent variable, one or many of these violations may be—excuse me—these assumptions may be violated, and so what we talk about is what to do when that happens.  Last time we just observed that cost is a difficult variable.  It’s not normally distributed.  It has a skewed distribution, skewed to the right by rare but extremely high-cost events.  Most people don’t get hospitalized in a year, but a few people will, and there’s even a fewer still that will have very high-cost hospital stays.

The other issue is on the left-hand side there aren’t any negative values, so we’re bound in by zero, and sometimes there’s a whole bunch of observations that are piled up at zero of people who are enrolled in the healthcare system but had no utilization.  Those are all challenges, and applying ordinarily least squares in these kinds of situations can result in biased parameters.  One very obvious problem with ordinarily least squares is the parameters in an ordinarily least squares regression.  If you use them to predict the costs of a particular observation, they actually predict negative costs, and so that’s obviously—nobody can incur negative costs, so that’s a problem.  We’re predicting something that can’t occur.

Now, if we take the log of cost, that results in a variable that is much more normally distributed.  Of course, this has the limitation of that we can’t deal with those zero variables.  We can’t take the log of zero.  That’s just not defined.  Then the other issue is if we do ordinarily least squares of log costs as the dependent variable and then we want to predict cost, there is a problem called retransformation bias.

It’s possible to correct that retransformation bias and come up with a predicted cost.  One method that is pretty robust is the smearing estimator, which we defined last time, but it requires the assumption that the error terms are all the same, that assumption of homoscedasticity, or that the variance of the error is all sigma squared.  That turns out to be not such a good assumption.

What we’re going to talk about today are these four topics about this problem of heteroscedasticity.  We’ll define it, and then suggest what needs to be done about it.  What can be done when we have a data set that has a whole bunch of zero cost values, people who don’t use health care?  How can we test differences between groups without making any assumptions about how our cost data are distributed, and then finally just some advice about how to choose among alternate methods that are available.

The first question is: What is heteroscedasticity, and what should be done about it?  A good 50-cent word, heteroscedasticity, and it’s simply a violation of one of those assumptions that the errors have identical variance that is expected to have a value of sigma.  In heteroscedasticity that’s not true.  The variance in the errors depends on X or perhaps on the predicted value of Y, that is, what is our predicted cost.

We can graphically illustrate homoscedasticity.  Here on our axis we have what’s the variance, and then on our X-axis we have the predicted cost, and so in this case the variance is identical across all levels of cost.  This is homoscedasticity, but a more common view in cost is the—actually the variance or our ability to predict is not so good as we get up into the higher cost region, and so this is heteroscedasticity, and violating the assumption of homoscedasticity causes some problems when we use the log cost model.

What do we do about this problem of heteroscedasticity?  The answer is—well, first, why do we worry about it—is that in using an OLS model even the log as a dependent variable can result in biased predictions, and the retransformation assumes homoscedastic errors, so we really can’t use the smearing estimator to make that retransformation.  That can bias our estimates of predicted cost.

The answer about what do we do is to apply a generalized linear model or GLM.  The GLM involves some sort of link function.  There’s a lot of different link functions.  We’ll talk about what those are.  We also specify a variance function, and so given the link function and the variance function are things that you could plug into the standard statistical packages, which have ways of estimating GLM, and we’ll talk a little bit about this.

This paper by Mullahy and Manning in the Journal of Health Economics is a real seminal one that will help you understand the GLM model for conducting the cost regression, and a lot of what I have to say is really drawn from their work.

The link function I’ve highlighted here—that’s the G in red that’s the link function—is just assuming that we can take this function of the expectation of Y conditional on X and express it as a linear combination of the X’s, those independent variable or variables.  In this illustration, or example of this, our link function is log, so we’re taking log of the expected value of Y conditional on X.  As before, when we used log as the link function, then our data, that is our parameter that we’re estimating, tells a percent change for each unit of change in X, what’s the percentage change and why.  It has the same—the parameters have the same interpretation.  Not such an easy interpretation when other linked functions are used.

In ordinarily least squares we are taking—when we’re doing this log transformation—we are taking the expected value of the log of cost.  In the GLM we’re actually taking the log of the expected value.  That log of the expected value is not the same thing as the expectation of the log.  They are different and in an important way.  First is that with GLM when we want to simulate or predict the value of Y or of cost we don’t have that problem of retransformation bias, so the smearing estimator is not needed.  The other advantage of the GLM of taking the log of the expected value is we can take—we can have zero cost observations in our estimate and the GLM will work with them, whereas that’s not possible in the taking ordinarily squares of log.

The other thing that we need in doing a generalized linear model is assumption about variance.  GLM assumes there is some function that explains the relationship between variance in our X’s or our mean values.  Some common assumptions are the gamma distribution, which is that the variance is proportional to the square of the mean, and the Poisson distribution, which is the variance, is proportional to the mean.  There’s actually a way to specify or to identify which is the appropriate assumption to be made, and I’ll be talking about that test in just a moment.

I just want to bring this down to earth a little bit and talk about how do you actually specify this in a statistical package.  We’re going to describe how do you do a general linearized regression when you have—you want to use a log link function, assume the gamma distribution, and you have a dependent variable Y, say your cost variable is called Y, and you have these independent variables X1, 2, 3.  How would you go about doing this?

In Stata you—it’s the GLM command.  If you have your data set set up in Stata, Y is your cost variable, you have three predictors, independent variables, X1, X2, X3—you simply enter it in a standard way that you give any static command, so rather than say the regress command you’re going to use the GML command, specify your model, and then that comma—it says—after the comma are the options, and it says, “Use the variance family of gamma,” and “Use the link function of log.”  It’s pretty straightforward how to specify one of these.

In SAS it’s a little more complicated.  Their basic syntax is just as simple as in Stata.  You can see this PROC GENMOD rather than GML is the command, and we have a model statement, which is very similar to the Stata model except this we need to specify the word, the key word MODEL and the Y=, and I guess there actually should be a semicolon between the GENMOD and the MODEL.  The MODEL’s on a separate line.  After the slash, our forward slash, is our assumptions that the distribution is a gamma distribution and our link function is a log.

That’s pretty straightforward.  The problem is that if we have zero cost observations in our data set, SAS—the designers of SAS have decided that it’s not appropriate to keep those when you have a gamma distribution, and so they drop them.  The code that I have listed below, this refined syntax, is actually kind of a work around that allows you to run PROC GENMOD with a gamma distribution.  You don’t specify gamma distribution.  You use this code that I’ve given here, the five lines between the PROC GENMOD and the MODEL statement that are basically—specify gamma distribution but allow you to include zero cost observations.  If you try this refined syntax out, it gives you the same answer and keeps all the zero cost observations as Stata does.  That’s a little bit of trick in how to work around that limitation in SAS.

The question is: Well, should I use the GLM, or should I use ordinarily least squares of log.  The advantages of the generalized linear model is that it can correct—have standard errors that are corrected for heteroscedasticity, and that if I’m simulating the cost, my predicting cost, using the model parameters, it does not cause retransformation errors, so I don’t have to use any correction for retransformation.  Those are some good advantages.  I guess one other should be listed, which is that in GLM I can have observations with zeros in them.  In the ordinarily least squares of the log of cost is more efficient, that is the standard errors are smaller with GLM, but I have to be concerned about bias from heteroscedasticity, and I have the problem of retransformation here, so that’s the disadvantages.

Up until now I’ve assumed that we should be doing a log transformation, that that’s the way to get most approximately normal and the best approach.  There’s actually an empiric way to test which is the best link function, and there are other link functions available, square root, or we could take squared, or—we’ll talk about those.  The Box-Cox regression is a method in Stata.  You use the Box-Cost—excuse me—Box-Cox command, and what we’re trying to do here is not find—at the bottom is the regression.  We’re trying to find that parameter for theta.  If theta were one, that would just be cost equals alpha plus beta, X plus E.  

If theta—well, we take different values, and I’ll show them.  *[Inaudible 15:51]*run the Box-Cox regression, and theta could take any of these values or something between them.  If theta is zero then we—approximating a log transformation—maybe we would estimate the Box-Cox regression and find that it was much closer to point five than to one or zero, we use the square root—and we use the specified square root as our link function.  There are these different possible link functions.  I have to say in my experience of doing cost regressions the square root is—the log is by—usually turns out that you use a log specification and occasionally a square root is what is yield, and that’s just my practical experience.  Maybe others have other data sets, other experiences, but I’ve done it in a few different studies, and it usually turns out to be a log, but this is what you estimate.  It’s kind of good to say that you checked, that your assumption that log was a link function was appropriate, that you actually did this test to show that it was close to zero, the Box-Cox parameter was close to zero.

Now the other assumption we made was that we would do a gamma—that the gamma distribution was the appropriate variance assumption.  That too is an assumption that we can evaluate.  If we first run the regression with a log link and a gamma regression like we specified before, we can examine the pattern of the residuals.  What we do is we take—so the residuals are the difference between our actual value of cost and the value that’s predicted by the regression.  Most programs have a very easy way to save your residuals into a new data set.  You can then square them, and then you use them in another regression where you have your—as your dependent variable is the squared residuals, and your independent variable is that predicted value of cost.

When you do this you have a parameter there.  We have this gamma 0, gamma 1, and the gamma 1, that is the parameter that says how to—as the predicted cost—predict the squared residuals.  That gamma 1 is the key to knowing which distributional assumption is appropriate.  When we do that, here’s that same regression again, that gamma 1 parameter.  If it’s zero then that’s saying that it would have been fine to assume—that shouldn’t say norma—but normal variance would be appropriate.  Poisson or gamma if the parameter is one or two, or the inverse *[inaudible 18:49]* it’s three.  My experience is that most of the times I’ve run this GML family test that I get a value near two, occasionally one, closer to one, so then I use the Poisson assumption.

That’s how you choose using the—empirically those tests.  There is a more sophisticated approach here, which is described by Basu in this paper that I believe is cited at the end of the talk.  We have some cites listed.  The idea that Basu said, “Well, why go through all these steps to estimate them?  We can build up a model where everything’s estimated at once, both the link function, the distribution, and the parameters.  They all get estimated in a single model.”  Unfortunately there’s not yet any canned program in a standard statistical package that does this, so it’s a little bit more challenging to do this approach.  I’m not sure that you really get that much more from it, but I would say this is now the state of the art.

I think that if you—on the one hand if you use—are doing cost and you’re using ordinarily least squares with untransformed cost, reviewers are not going to like that.  If you do a generalized linear model, that’s going to be acceptable in the modern world, and this generalized gamma model is something that is pretty much on the cusp, cutting edge.  Who knows?  In a few years this may become what is the accepted practice and the minimum standard of what you need to do.  Not yet.

I thought we’d just take a pause here and ask if there’s any questions.  I was just seeing there was already a question about the sound.  Hopefully that person got their needs met by going to the *VANTS* line rather than trying to use their USB headphones or their speaker on their computer.

Anybody have any questions on what we’ve covered so far?  Generalized linear models.  

*Moderator:*There aren’t any questions right now.

*Dr. Barnett:*No questions yet.  I don’t know whether that means that I’m making a very clear explanation, Jean, or—

*Moderator:*Well, that could be it *[laughter]*.

*Dr. Barnett:*- I’m just mystifying people.  *[Laughter]*Well, that’s optimistic.  I fear that maybe I’ve mystified people a little bit.  Well, we’ll go back over this a little bit.  I want to move—

*Moderator:*There’s actually a question right now, Paul,—

*Dr. Barnett:*There is a question.

*Moderator:*- that’s asking, “Can you put up the Box-Cox equation again?”

*Dr. Barnett:*Sure.  The idea here is that we want to estimate theta because if theta is a zero, then it really is a log function, and if it’s theta’s one it’s normal distribution or linear function, and there are some other—square root if theta’s point five.  We are really interested in the value of theta in this.  Just to know what link function.  It’s really about the choice of link function.  Any other questions?

*Moderator:*No, just the question about the recording of this presentation, which should be available on the HSR&D website later.

*Dr. Barnett:*Right.  That’s how to choose a link function, and this is how to choose the variance structures and away you go.  All right, so the next question is, “Well, what happens when there are many zero values?”  Well, we can us a GLM model when there are zero values, but sometimes we want to delve into that about why do some people have costs and other people don’t.  What are those sort of situations?  We would have examples where people are enrollees but don’t have any utilization.  We gave this one example of people who we identified as being users of Veterans’ health care in fiscal ’10 and what their costs were in the prior year, and there was a significant fraction of people—looks like somewhere like 17 percent—who didn’t have costs in the prior year.  That truncates the distribution.

We could estimate that with the GLM, but there is another approach, and that is the two-part model.  This is sometimes called a hurdle model.  The first part is—the question is—estimate a regression to determine whether the person incurred any costs.  We create a new dependent variable that takes a value of one if cost is incurred and zero if no cost is incurred.  This is an indicator of whether or not the person had incurred cost, and then the second part of the two-part model is a regression of how much cost did this person incur given that they had—that they’re one of the persons that did have some cost.  It’s called a hurdle—part one is sort of the hurdle.  Did you get over the hurdle and incur cost, and part two is this conditional cost regression.  I don’t know if hurdle’s the greatest metaphor, but that’s what they call them.  You’ll see that sometimes in the literature.

Our two-part model can be expressed in this sense that our expectation for Y conditional on X is we have part one, the probability that Y is greater than zero, that is the person incurred any costs, and part two, just how much cost they incurred conditional on being greater than zero, that is having any cost.  That’s given their value of X, so both of these are given this particular person’s value of X.  We can use a two-part model to simulate just this same setup again that we have if we can predict the probability that the cost is being incurred given this person’s values of X, say their group membership or their age or their gender, and then we multiply it by their expected cost conditional on having incurred cost.  That would give us our predicted value in the two-part model.

Let’s think for a moment about that first part, this probability that the person incurred any cost.  Our dependent variable in this case is a zero, one, or a dichotomous indicator variable.  Takes a value of one if cost is incurred or zero if no cost is incurred.  The question on this is—Heidi, we are hoping for the whiteboard to address the question—for people to tell me what type of regression you think should be used when you have a dependent variable that’s dichotomous.  It takes a value of either zero or one.  Oops.  I almost gave away the answer.

*Moderator:*Here we have the whiteboard to type on there.  At the top of your screen you have a capital T.  Just click on that and go down to your whiteboard, and you’ll be able to type right on the screen there.  Once you hit the enter button then we’ll be able to see what you typed.

*Dr. Barnett:*We have some people are saying logistic regression, and that’s a right answer but not the only right answer, or sometimes called logit—logistic.  Another idea.  Lots of votes for that one.

Probit.  That’s really the other right answer.  You could use a probit, which is another way of coping with the zero, one variables.  Gives probably very similar answers.  Thanks for that feedback that is—and someone drew a nice picture there too I see.  Dimensioning returns.  It looks like that curve. 

Back to the slides, so this is set up now.  Here is the logistic regression.  I’ve mentioned probit, but I’m not going to put the formula for probit, but the logistic regression takes that log odds ratio.  The odds ratio is the probability that they had costs divided by the probability that they didn’t incur costs, and we take the log of that and express that as a linear function of the parameters.  That X is assumed to be related to the log odds ratio in a linear fashion, and that’s how we estimate a regression with a dichotomous variable.  The desirable property is that we can estimate that P value, that probability conditional on X, and that’s exactly what we need to do for our simulation.

Here in SAS is how we do a logistic.  Hopefully a review for many of you.  I put in here with a descending option of—SAS has a way of giving you the inverse of the parameters assuming for some reason that what you want to predict the probability of Y being zero and when every other statistical package gives you the parameter for Y being one, and so descendings flips it back to the normal way.  We can save our results from the logistic regression, which is what we want to do.  Our output statement we use the out equals to specify the name of the data set where we’ll keep our result and then prob—we’d name the variable, which is the probability.  For each observation in the data set given its X’s will predict the probability that they incurred any cost.

We want to use this because SAS will otherwise estimate the probability that the dependent variable equals zero.  We want to estimate the probability that the dependent variable equals one, that is that the person incurred cost.  In Stata it’s a similar logistic regression, and I think that I’ve made a mistake there in putting an equal sign because Stata does not use the equal sign.  My bad.  You can tell this is written by someone who more often does things in SAS.  Then you predict and assign a variable name, and the comma is the option that is to save the probability that Y equals one.  This predict statement generates that predicted probability.

Then the second part of the model is that conditional cost model, that is the regression of only the observations where cost was incurred.  We call it conditional because it’s conditional on having incurred any cost, and we could just do it in the methods that we’ve talked about previously using a generalized linear model with a link function or taking log cost and using ordinarily least squares.  Either of those would work with all of the same caveats that we had about those.

The two-part models are really about having separate parameters for participation or getting over the hurdle.  What’s the probability of incurring costs and separate parameters for the conditional quantity of care—conditional cost I should say?  For example, we might be interested in say age, that older people are more likely to incur—go use the healthcare system in any given year or gender, that women are more likely than men to use—incur any cost or say some chronic illness maybe.  People with diabetes are more likely to incur cost than people who don’t have diabetes.  It’s all about participation in care, and that is kind of a separate consideration than how much cost someone incurred given that they did go to the healthcare system and did incur costs.  That’s the advantages that you get separate parameters. 

The disadvantage of a two-part model is that if you’re using it for simulation trying to figure out what cost did someone incur given they had a certain X.  They were in a certain group, had a certain gender, age, chronic disease, whatever.  It’s hard to exactly—you can predict the cost, that’s the probability for the given, that X times the conditional cost given that X, but it’s hard to get a confidence interval around that to say how certain you are that that’s the cost.  Really, if you’re trying to predict the cost, you’re probably better off with a GLM model.  The GLM models seem to work pretty well even with a substantial number of zero values.  Ten, twenty percent of observations with zero values don’t seem to phase the GLM model, but we’ll talk a little bit about what’s a good enough model in the fourth part of the talk.

The alternate to the two-part model is just what we talk about before.  Ordinarily least squares with untransformed costs, that is if we just use cost as a dependent—raw cost as a dependent variable except that there’s some zero values in it.  Well, yes, we could do that, but if we’ve got very many zero values, we’re going to start predicting some people have negative costs.  That’s probably not a great alternative.  Ordinarily least square with log cost, and we could use a small positive value in place of the zero.  If we have zero observations, we could—we can’t take the log of zero, but if we say, “Oh, well, ten cents is close enough to zero, so we’ll just take the log of ten cents.”  Well, the problem with that is that the results will be sensitive to what our small positive value is.  That choice has a big influence on the parameters.  That’s not such a great alternative, and we talked about that last week.  You can actually see some graphic illustrations if you missed that lecture.  You see that in the archive.  Then the GLM models do perform pretty well when you have zero costs.  Those are some of the alternatives to a two-part or hurdle model.

I think now we’ll turn on to the third topic, which is: What about these methods that don’t require us to make any assumptions about the distribution?  You’ll remember that when we did the GLM we were making some assumptions about is it a gamma or is it a Poisson distribution.  We have some empiric tests to think the assumption’s good or bad, but still we are making an assumption.  There is a possibility of a non-parametric test, which doesn’t make any assumptions about the distributions, that is it doesn’t make any assumptions about the variance.

The classic example of this is the Wilcoxon rank-sum test.  Every observation in our data set is rank ordered, that is we start with the highest cost data set in position number one, and the second highest cost observation stays in position two, and so on all the way down.  Then we look at our groups and their rankings, and there is a way of using probabilities to figure out, “Well, did this ranking occur just by chance alone, or does it really reflect—is it so improbable that say one group ends up with many more at the higher ranks and the other group many more at the lower ranks?”  Chance alone would deem that that’s improbable, and then we say by this rank-sum test that this is actually statistically significant.  That it didn’t occur by chance alone.  That it really represents some significant result.

The rank-sum test can be used when there are more than two groups, and this is the Kruskall Wallis test.  What Kruskall Wallis does—it gives you a statistic that tells you that if you’ve assigned your observations to more than two groups, say you have three or four but everybody’s in one of several mutually exclusive groups.  You’ve ranked them, and then it looks at it, and Kruskall Wallis will tell you is some group different from another group.  It doesn’t tell you which groups are different.  It just tells you that something is statistically significant.  It’s just like an analysis of variance tells you that something is different, and then you have to do some sort of post-hoc test.

In the Kruskall Wallis test if you find that somebody’s different, then you can start using Wilcoxon tests to compare pairs of groups as a kind of a post-hoc comparison.  That allows you to use non-parametric methods when you have more than two groups.  There is that extension to the method.

Now the non-parametric test has a limitation.  It’s pretty conservative.  The fact that you don’t make any assumptions and compare ranks it ignores the influential outliers, and so if we think about this, we can have two interventions that give the same result with the Wilcoxon.  In the one case the top rank observation is $1,000,000.00 more costly.  In the second case the top rank observation is just $1.00 more costly than the second ranked observation.  Wilcoxon would give the same result, whereas if that extra $1,000,000.00 or $999,000.00 in the first example is actually very important to us, the Wilcoxon is just ignoring it because it did—that top cost was the top cost whether it was $1.00 more or $1,000,000.00 more.  The Wilcoxon is very conservative in that regard.  That’s one disadvantage.

My experience is—I have seen examples in studies where I’ve done—looked at—is that the Wilcoxon has shown that groups are not significantly different, whereas if I use some sort of GLM model with a indicator as the independent variable, the groups will shown to be significantly different because I’m taking into account some of those outlier observations and the information that they provide.  Wilcoxon is a bit conservative.

The other disadvantage of the Wilcoxon is that it doesn’t allow for any explanatory variables.  You can compare Group A to Group B, but say that they’re different in some respect.  Say Group A has more older people, and you would like to compare those groups while controlling for age.  You could do that in a GLM model or a regression, but you can’t do that in a Wilcoxon.  Not in any way that’s convenient in any way.  You’ve got to—have to acknowledge that limitation that it’s non-parametric compare groups, but it doesn’t allow for additional explanatory variables.

The final section of the talk is: How do we synthesize all of this information?  How do we know that we’ve got a method that’s good or good enough?  What’s the convincing evidence in that?  The answer is it really has to do with the predictive accuracy.  If your model does a good job of taking your X’s and predicting your Y’s, that is taking your independent variables and predicting cost, then your—to feel that it’s a good model of the relationship between the independent variables and cost, that is it’s really telling you what’s going on with the world.

Typically what you see—people who are demonstrating their goodness of fit estimate their regression with half of the data in their sample, and then test the predictive accuracy on the other half of the data.  There are other approaches, but this is kind of the classic one to evaluate model fit.  Then, when I say test the predictive accuracy, it means they’re finding statistics like these two I’ve just put up here.  Mean absolute error and root mean square error.  I’ll define those for you.

Mean absolute error is you take each observation and find the difference between observed and predicted cost, that is the residual in the regression.  You take its absolute value, and then you find the mean of that absolute value.  It’s basically getting at the distance to the regression line.  The model that has the smallest value would be the—or the least mean absolute error—would be regarded as the best.  The root mean square error is a similar idea, but instead of taking the absolute value we square them and take the—so that makes a positive—always a positive number, and then we take the square root of that.  Again, the best model has the smallest value of that root mean square error.

Those two statistics, mean absolute error and root mean square error, are useful.  Mean absolute error—excuse me—and root mean square error are useful, but not the only method.  We can also—really what is more interesting is to look at the residuals and whether we’re simulating over the entire range of cost.  Usually where we get into trouble with models is not in the—for the typical observation but for the outliers—we don’t do a very good job of predicting those very extreme cases where the costs are very high or the very low-cost cases, or zero-cost cases.  This evaluation of residuals gets at that how well do we simulate over the entire range of observations?  We can look at the residual or the ratio of predicted to observed and calculate those separately for each decile of the cost, that is the 10 percent lowest cost observations all the way on up to the 10 percent highest cost observations.

A good model should have equal residuals or equal mean ratio over all those deciles, that we do just as well at the middle of the distribution of costs as at the top and the bottom.  Usually where you get into trouble is at the top and the bottom.  There is actually some formal tests here, and Manning, Basu, & Mullahy in 2005— their paper really gets into how you do these formal tests.  I think this is convincing if you’ve got your Hosmer-Lemeshow test—test where the residuals in each decile are essentially the same.  Then if that F test—if that F statistic is small, then you’re saying that each decile’s got a very similar pattern for the residuals, and that’s convincing that the model’s performing quite well.  Pregibon’s link test tests if the linearity assumption is violated.  

I direct you to this paper for the details on this.  Also, Will Manning, Willard Manning, at University of Chicago has on his personal page—you can google him there—he has some Stata code that shows you how to program these.  It’s not that complicated to do, but he’s done the heavy lifting.  If you have trouble finding that link, just send us an email.  Send an email to me or to HERC at the a.gov, and I’ll send you the link to Will’s Stata code where he’s coded these in Stata, these tests.  His code also does those other, those GLM tests where we were testing what the appropriate distributional assumption is.  He’s got code for doing that too.

This is a time to entertain any questions that you might have about the talk.  I see we’ve got a bunch of them now.

*Moderator:*A bunch of questions and some comments.  The first question asks: I’m interested to know if this framework is possible to look at within differences when we assume a distribution other than the normal one?  I think when she says “within differences” she means within differences between patients over time, but if that person could clarify that’s now what they’re asking.

*Dr. Barnett:*Exactly the point is is that we don’t have to assume costs are normal.  We can assume they’re log normal, or we can make other assumptions about the distribution.  The other issue is is not only do we not have to assume they’re log normal, but we can relax the assumption that the errors are homoscedastic, that is that they’re— possibly the errors vary over the range of the value of the dependent variable.  Yes, these are pretty flexible specifications that get us around—avoid some of the pitfalls from having to make too strong assumptions about distribution.  At one extreme we make absolutely no assumptions, and that’s the non-parametrics.

*Moderator:*Okay, so the person followed up and asked, “Can we do fixed effects when it’s not a normal distribution?”

*Dr. Barnett:*Yes.  You can do fixed-end effects and random effects models with the GLM.  You can do repeated measures models.  Say you had annual costs for a group of people for several years running.  You’d want to have a term for the person or whatever.  All of those things are—there’s certainly embellishments in all this that are possible. 

*Moderator:*Okay.  The next few are some comments, so I don’t know if you want to respond to them or not.  The first comment is: “I believe a hurdle model is more of a term that is used when the outcome is a count.”  The second comment is: “Please mention that the Stata command TPM allows you to predict the conference interval around a predicted Y given X.  This command is new within the past year.”

*Dr. Barnett:*I’m sorry.  Say the—okay, so there’s—

*Moderator:*There’s a command in Stata called TPM, which allows you to predict the conference interval around a predicted Y.

*Dr. Barnett:*Well, we’ll have to—we’re always learning something, and I think Tom was the one who helped me learn something when he saw this last year.  Maybe we need to get him to teach *[laughter]* the class *[laughter]*.  Sounds like he’s on top of it *[laughter]*. 

*Moderator:**[Laughter]*

*Dr. Barnett:*TPM.  We’ll look that one up.  Well, we’ll have to ask him exactly what that does.  Is that a post-hoc command, or is that an estimation command?  We’ll look it up.

*Moderator:*The next comment is: “The old classic method was to start with a small cost value and to see how the model changed as it was brought to zero.”  It’s just a comment.  The next question asks: “What if there are more than 20 percent zero values?”  This is usually common in private health insurance data.

*Dr. Barnett:*Well, so I think that then it becomes—the question really revolves on what are you trying to accomplish?  Are you trying to simulate costs?  Are you trying to answer policy questions about what predicts utilization?  What you do has to be fashioned in the context of what you’re—what’s the important thing you’re trying to figure out?

I think the issue is is it becomes more difficult as you begin to have lots of zeros.  If you have a significant number of people that don’t have any healthcare utilization, don’t incur any cost, then I think you’re probably going to find it more difficult to use a GLM model to predict that.  Then you’re going to have to go to a two-part model.

*Moderator:*Next question asks: “Why did you save the probabilities from part one of the part two model if you only include observations with non-zero costs in the part-two regression?”

*Dr. Barnett:*The idea is that if you want to use the two-part model to simulate costs, you need to have information from both parts.  If I’ve got a 50 percent chance of incurring any cost given my age and gender, then the question is—well, then I multiply that probability times what the expected cost of a utilizer, given that age and gender.  You really need both of those to predict cost.  You need both to see both the probability and to estimate the conditional cost.  The probability times the conditional cost is the expected cost.

*Moderator:*Okay.  The next question asks about outliers.  “Sometimes very few observations incur extremely high costs.”

*Dr. Barnett:*I’m sorry.  I was just noticing that someone posted that TPM is a command for estimating two-part models, so that’s something new.  I don’t know whether it’s a Stata—built into Stata or—Stata allows users to develop new routines, and I’m not sure whether it’s one of those new user routines or something that’s now built into Stata.  That’s very good information.  We’ll have to put that two-part model in the next talk when we give it again.  Hopefully we’ll have a—when we gave this class before we had opportunity to demonstrate these things on the desktop.  Unfortunately, the new interface doesn’t allow us to do that.  We’ll try to get a little more creative here in the future.  I’m sorry.  What was the question?

*Moderator:*I just want to follow up.  There’s a clarification that TPM is a user-written command, and you can find it by just typing in “find it TPM” in your Stata command line.

*Dr. Barnett:*Yeah, so often these things—once they get—a user develops and then Stata says, “Oh, yeah.  That’s a good idea,” and they incorporate it in some subsequent upgrade.  I’m sorry.  The question that I got off the track—

*Moderator:*The original question was asking about how to handle outliers, and so the models that you talked about basically addressed the fact that there are very high outliers.

*Dr. Barnett:*Yeah, so some people like to say, “Oh, I’m going to drop the outlier, and that’s how I’ll deal with them.”  I think that’s not a good practice because often the outlier has the information that you really want.  Now there are bootstrapping and jackknifing methods that do that kind of systematically, but the idea of dropping the—some people will drop the top 1 percent or 3 percent observations, and they should do the same on the bottom.  I actually think that’s throwing out important information and is not a good practice.  

The RAND Healthcare Experiment—I remember it was saying last week that the—Will Manning said that in one of the health plans there was one patient that had incurred 17 percent of the *[laughter]* cost of the whole health plan.  Well, if we throw that observation out maybe we’re ignoring some important thing that that health plan caused.  The thing to do is to have methods that are robust enough that will allow you to have—include outlier observations and see whether it’s—represents a statistically significant effect of the things you’re interested in.

*Moderator:*Okay.  Next question asks: “Can you still estimate mean differences in cost in the parameter coefficients when using the various link functions that were described?”

*Dr. Barnett:*Yes.  That’s exactly right.  We can predict cost—so say we have our—we estimate our cost regression with a generalized linear model, and our independent variable, our X, is a dichotomous variable indicating their person is in a group or not in that group.  It takes a value of one if they’re in the group and zero if they’re not.  Well, we could predict the cost with X set to one, and we can predict the cost again with X set to zero, and that might be evaluating a regression with all the other X’s taken at their mean value, and then we’ve simulated the cost, or we’ve predicted what’s the mean cost of being in that group.

That’s exactly what we want to do is to be able to—because sometimes that parameter doesn’t have a very satisfying—if we say, “Oh, the parameter was .12.  There’re 12 percent extra costs in the people in the group.”  What people probably are more interested in knowing is that it’s—what the dollar amount is, and what the dollar difference is in groups.  That would allow you, by doing those simulations, to predict that.

One approach is to actually evaluate the parameters at the mean.  Another way is to—you find the predicted value of cost of everybody in your sample, given their characteristics, and then take the mean of that.  That has some desirable properties, that it seems to be less subject to the problems of what non-linearities can do.  It’s a little bit hard to talk about in the abstract, to answer the question this way, but it is exactly what you want to do—is to use your model to predict cost.  That’s the emphasis on how well do we do at the extremes.

If you predict the costs, say with your X set to one, and then you go and use your parameters again and it predicts your costs with X set to zero, and then take the mean over observations in each case, you have an idea of what the affect of group membership is on cost.  That’s really an important use of these cost regressions.

*Moderator:*Okay.  Somebody was asking if you could clarify the difference between the Pregibon link test and the Park test.

*Dr. Barnett:*Here we’re down to two minutes, so I want to just be sure, before other people log off, that if you—if you’re ready to go since we’re at the top of the hour—I will answer that question in just a moment—but since we’re very near the top of the hour I want to encourage people very much to please complete the two-question evaluation.  We’re asking for your feedback and hopefully to make this talk better.  We don’t get a chance to talk to you directly.  We don’t see your faces, and so this is really our chance to find out how we’re doing, so please complete the questionnaire.

The question again was about the Pregibon link test.

*Moderator:*Pregibon link test and the Park test.

*Dr. Barnett:*They’re really two different things.  The link test is about testing the residuals and about the goodness of fit of the model.  The other one is the Park test.  The Park test is about figuring out which distributional assumption is appropriate.  One is how do you choose the best—how do you choose the distribution.  Should you use gamma or Poisson or some other distributional assumption?  The Pregibon test is to say, “Gee, how well did we do?”  How well and especially at the extremes.

*Moderator:*Okay.  The next question asks: “Some people suggest that as long as a sample is very large we can violate the normality assumption and just run an OLS regression.  Do you have any comment on that?”

*Dr. Barnett:*I think that you get in less trouble in a large sample, but I don’t think you’re going to dodge the problems altogether.

*Moderator:*Okay.

*Dr. Barnett:*That’s more my intuition than *[laughter]*—I don’t think I could prove it, but I think that’s right.  I do know that the comments about the problems with OLS from the papers that I’ve read say, “And you’re going to get in a special problem when you have a small sample,” but I think that the same thing is going to occur.  If you have skewness in a big sample, you’re still violating the assumptions of OLS, and you still should be thinking about a GLM model.

You’re still going to get in situations where you’re predicting—some observations are going to be negative costs, so they’re not—at the bottom they’re not going to be very well fit, and at the top a few observations are going to be—the most extremes are going to be very influential.  It doesn’t really matter how many observations.  If it’s not normal, it’s not normal.  If it’s heteroscedastic, it’s heteroscedastic.

Maybe one more, Jean, if we’ve got one more.

*Moderator:*Yeah.  Can you give further details on the Manning package that estimates GLM link and variance structure?

*Dr. Barnett:*Well, so I just know that the last time I looked that Willard Manning at University of Chicago—if you google him, you will find that you’re going to get to his biography page at the university, and on that page there’s a link to the slides that he presents at his workshop at the health economics meetings and also some sample Stata code that they hand out to the students at those presentations.  That presentation’s about two or three times as complicated as what I’ve been doing in the last—these two sessions.

I had a review here.  I don’t think I’m going to go back over this.  I do want to—this is just restating what we said.  I do want to mention these sources on general linearized models, that these are important papers.  The ones that I put the star by I think are especially useful.  The second one there is about log models, and the third one, the 2005 paper, is about the generalized linear models.  I would say that’s probably currently the state of the art.  The Basu paper is going beyond what I think most people would do currently.  There are these papers on two-part models that are very useful in understanding.  There are some worked examples, so these people show exactly how they evaluated the fit of their model.  These four papers I think here.  If you can’t find any of these references and you have a VA email address, we’ll be glad to send you the papers.

I will also mention that Maria Montez-Rath several years ago did an evaluation of a statistical model that she used in a mental health substance abuse treatment cost in PA.  I think it’s a very interesting—sorry that the links there are a little faded, but they’re clickable if you download the slides.  You can go straight to there and hear her talk.  She really explains about using the mean absolute error, mean squared error, and root mean squared error, and doing some of these tests to see how good the model fit was.  I think that’s a very—a great seminar that she provided for us.  Please take the—provide your feedback if you haven’t already, and I don’t know.  Any last—one last question.

*Moderator:*Actually, Paul, I’m not able to put the feedback from them up until I actually close the meeting.

*Dr. Barnett:*Oh, dear.  Let’s close it then.  I thought every person got it when they left.  I had forgotten that’s the old system.

*Moderator:*Unfortunately that’s the old one, so if we can wrap things up that way we’ll get as much as we can.

*Dr. Barnett:*Okay, well, let’s bid folks adieu and thank them for their attention.

*Moderator:*Thank you everyone for joining us, and we would appreciate you filling out the feedback form.  Thank you.

*[End of Audio]*

Page 1 of 14