**Transcript of Cyberseminar**

**HERC Health Economics Seminar**

**Risk Adjustment for Cost Analyses: The comparative effectiveness of two systems**

**Presenters: Todd Wagner, PhD; Anjali Upadhyay, MS; Theodore Stefos, PhD; Elizabeth Cowgill, MPH; Eileen Moran, MHA; Peter Almenoff, MD**

**June 19, 2013**

**Risk Adjustment for Cost Analyses: The comparative effectiveness of two systems**

**June 18, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact: herc@va.gov*

Todd Wagner:I just wanted to welcome everybody today to HERC Health Care Economics Cyberseminar where I will be talking about risk adjustment. I appreciate everybody’s patience today. I am in Maryland doing sort of a tour of the east coast VAs, and so I am sitting in an office with an air card and phone and trying to make this work, so this is sort of a new one for me. Heidi,   can you see this live, is that okay? 

Moderator:Yes, you are coming through just fine. 

Todd Wagner:Sounds great. What I am going to be talking today, this is research in progress and so we definitely will value your input and ideas as we are going forward here. It is a team effort and we are really standing on the shoulders of giants. If you like what you are seeing today, it is the results of the team. If you do not like what you are seeing, it is my fault. 

Working with us closely is Anjali Upadhyay, who is a programmer, was at HERC, she moved to southern California and is now with Kaiser so I miss her dearly. Theodore Stefos, Eileen Moran, Elizabeth Cowgill is with me, and then Peter Almenoff. There are a bunch of other folks as well. So you can see this is an awfully large group putting in different kinds of inputs. Bruce and Amy are both researchers, Bruce is at UPenn, Amy is at Boston; and Maria is a former graduate student of Amy’s now at Stanford out here. So, they have given us some terrific help. Mei-Ling and Yu-Fang up in Seattle; and Steve Asch, as always with helpful comments. 

Here’s the outline of what I’m hoping to present today and get feedback on. One is just an introduction on why I think risk adjustments are going to be even more important than it has been in the past few years. The second one is how do we compute the scores and how do they compare and we are largely going to look at two systems. 

The second aim is a question about what is gained about recalibrating the risk models to accept VA data. When we use these algorithms and compute the risk scores, they are based on non-VA data, so the question is can we improve this with VA data. There are, of course, limitations, and then sort of  [inaud]  in the findings. 

So, what is risk adjustment? Really, it is just a statistical method to adjust for the observable differences between patients. There are, as you can imagine, many, many different observable differences between patients as they use care over the course of an episode or a year. We can think of that largely based on diagnostic information, but age and sex obviously fit into that as well. The goal is when you are doing statistical analysis, is you want to classify, or you want to compare homogenous groups. That unobserved heterogeneity can really cause problems with your analysis, so risk adjustment, many of the systems that you use are trying to classify patients into what they think of as homogenous clinical categories. Then many of them will calculate a single dimensional risk score using these clinical categories, which is quite helpful, especially if you are working with a limited number of data, a finite sample and you just want to have a single dimension risk score that uses just one degree of freedom. 

So why risk adjust? Clearly, everybody is hearing ideas about big data. Big data is probably not new to VA. In fact, we have had millions, if not billions of records for many years now, but the questions are, there is limited research budget, limited time to do large randomized clinical trials, and yet people want to use these administrative data to inform policy and inform decision-making. So, the question of how to best use these observable, these administrative data to take into account what we can observe. Whether you are just interested in comparative effectiveness... let’s say you were just interested in the comparison of two treatment groups, but you can also be interested in efficiency of care and you’re interested in the delivery of care across VA medical centers. You might also be interested in high value care or how does the care that you are providing affect the outcomes of the patients. Risk adjustment is necessary to address the difference, the heterogeneity across the populations. 

Right now, risk adjustment at VA is used by operations and research to assess medical center efficiency and productivity; and to do health services research. If you are looking, for example, at hospital readmissions and you are looking across time, or across medical centers, you will need to do some sort of risk adjustment to control as best you can. Now, you might have other statistical methods for controlling some things, fixed effects and so forth, but in many cases, we are taking a large population of Veterans and we are wanting to control for their differences, their clinical differences. 

Historically, VA has contracted with Verisk, which is a company, to obtain the calculated risk scores for VA data. The idea is that we obtain software, we process the software, the VA data, and out pops these HCCs, these hierarchical condition categories, as well as risk scores. There is a history to these things and I think HCCs, that term hierarchical condition categories, is a little bit like calling something a Xerox. There are many different software’s that produce that, there are also improvements over time, even within a company’s software, so there are risk versions, just to be transparent here, is the risk smart algorithm created a hundred and eighty-four HCCs in risk scores. Verisk is phasing out that version and is moving toward a new version called risk solution, which creates three-hundred and ninety-four HCCs in risk scores. So, you have to be very careful about what versions you’re talking about. 

There are other software’s out there that also use the same terminology of HCCs and so forth, so one has to be very careful about what we are talking about. In this study, we focused on the latter which is this risk solutions in the three hundred and ninety-four HCCs; and so if you are familiar, you are using a lot of the data that have been put on Austin and on the HCCs and risk scores, that’s the risk smart version. I apologize if I am sort of shifting the game on you. 

From here on out, just to make my life easier, hopefully if I say DCGs or DXCGs, just refer to this risk solution model with the greater number of HCCs. The model produces, when you run it, three risk scores. One is a prospective risk score without any pharmacy information. The other is a concurrent risk score without pharmacy information. Both of those are based on Medicare data. The last one is Medicaid prospective risk, with information based on pharmacy. You can imagine these different scores produce different levels of risk information that you might want to use. We are going to compare all three. Now, just to be consistent, some people have trouble between the concurrent and the prospective and the reason really is just a circularity argument. If you are trying to understand risk and you are using concurrent data, some people say well, it is not fair if you are using the concurrent data because obviously, that should predict better than a prospective model. In a prospective model, how well does this year’s risk predict next year’s cost? I am not going to get into the weeds and whether I prefer one or the other. I actually do have a preference one way or the other, but I am not going to let you know what my preference is. 

The operational question given the transition from risk smart to risk solutions, the question was should VA continue to contract with Verisk and move to risk solutions. The question could be framed as well, there are many other softwares out there, why should we be doing a sole source contract? In the spirit of equity in contracting, the real question is maybe there are other systems that we should be comparing. So, as Peter Almanoff like to call it, it was a bake off. We are going to compare these different systems, so we started talking to different companies. You can imagine, some of these are publicly available systems. Charleson is actually a very, mostly simple system; and it’s a comorbidity index that has been updated over time. Steve Finn’s group in Seattle has created this CAN score, clinical assessment of needs score to look at things like mortality and readmission. I should say that the Charleson was mostly to look at mortality. There’s these ACG groups, there’s CDPS which is out of San Diego and publicly available, and then there’s this CMS risk adjustment model, version twenty-one also known as the Pace model that CMS uses or was hoping to use for its payments for medicare. There have been many versions over time and so we chose the V21 based on discussions with CMS as well as our TI, which is their contractor, to make this model. 

Let me be specific. I should say, I think I actually have a slide about where we head with this, but that bake off became too challenging. It is too hard to compare every different software, all the different models, so we are really going to focus on two. One is the risk solutions model and the other is this CMS Pace V21 model. It generates a hundred and eighty-nine HCCs, it produces three perspective risk scores. These are all perspectives so there is no concurrent risk score. One is a community, one is an institution, so for example if you are in a long-term nursing home, and the other one is a new enrollee model. 

Keep in mind that what they are using this for, and why they use perspective is they are using this for prospective payments for the next year for the medicare advantage plans. There is no concurrent risk score. I just want to stress that if you are weighed to this idea that you need this concurrent risk score, this model may not be the model to use. 

We simplified our aims. Originally, like I said, we were interested in this bake off. Eventually we simplified it. How does the DCG model and the V21 risk scores compare? That’s aim one, and aim two is going to be what’s gained by adding variables that we have in VA data and recalibrating the risk scores to fit the VA data. 

How do the computed risk scores compare? There are six study samples that we created for this analysis. One was a general random sample of two million Veterans, VA users; I should be very specific here. Why two million? Well, it seems like a relatively good number. We knew that we were going to be doing a lot of GLM models and there was a concern about over fitting the data. In discussions with [inaud] in previous iterations, we talked about what size is it likely to over fit and there was discussion about if you’re under a million, you’re likely to over fit, so we just chose two million. Relatively arbitrary, I will be honest there. 

The second one is high cost Veterans. These are Veterans who are the top five most expensive users, and I will go through separate slides on how we define these study samples. What we are trying to do here, the motivation for the different study samples is the belief that the VA has sub populations that are particularly interesting and noteworthy, and we wanted to make sure that the risk models were fitting those populations adequately. 

The third one is Veterans with mental health or substance use disorders. 

The fourth one is Veterans over the age of sixty-five, particularly because these are Veterans who are dually eligible. There are a handful of Veterans under sixty-five who are disabled who are also dually eligible, but we know all of them over sixty-five are; Veterans with multi-morbidity; that is the idea that they have multiple systems involved in chronic diseases; and then perhaps our hardest sample to identify is healthy Veterans. Keep in mind; if they are truly healthy Veterans, they are not using VA care, so it is hard to observe them, so that is a real hard one to observe. 

Let me just walk you through each of these samples and how we computed it because this does effect how we think about downstream. 

Molly or Elizabeth, if you have questions feel free to chime in if they are questions that would help clarify what’s going on here. 

Moderator:Thank you, none so far. 

Todd Wagner: Sounds great. So the high cost users, there is a question about what data set to use for high cost users. If you use DSS data, for example, the DSS data are local and you would have to geographically wage adjust for them, otherwise you’re going to disproportionately sample from high wage areas, for example Palo Alto, Boston... so we use the HERC national cost to remove geographic wage variations, so this is the most costly five percent of VA users. 

The next one is the mental health/substance use. We actually worked with mental health operations, and these are all patients with mental health or substance use diagnostic codes in VA. We used the same diagnostic codes that mental health operations used, so if they were interested in how our results would apply to them, they can use these results. Perhaps our simplest identifiers, over age sixty-five, relatively simple. Multi-morbidity is also challenging. There are different ways, and none of them are standardized. We used AHRQ has a chronic conditions indicator and part of that is a body indicator. What it really is trying to do is identify the systems that your body is involved with. So here is an example, the body system indicator has infectious and parasitic diseases, neoplasms, endocrine, disease for the blood, mental disorders, and what we were really looking for here, and I’ve grayed out some which have basically no observations in VA because they are basically pregnancy and kids. What you are really looking for is multi system use at least that is how we have defined it here. 

So we said two or more body systems are indicators. There are a couple other nuances that we put into the panel... or into the paper, I should say, but I will get to that for people who are particularly interested. Then healthy; like I said, that was really challenging. What we said is they could not be multi-morbid. They had to have just one body system indicator and they had to have a V code for a physical. The idea being is that these folks, we wanted to be able to observe people who were using VA for hopefully most of their care, so it was not just a missing data problem. We didn’t want to classify them as healthy just because they were using mostly Medicare, and the belief that if they are getting a physical in VA, that means generally speaking, the thought is that they would be getting most of their care at VA, or would think of VA as their primary provider. 

Here are the outcomes that we are going to use. We are going to use total costs; and these are DSS costs for FY10 and FY11. We have actually gone through the models and re run them with all the HERC data and interesting there, you  might think that HERC data, being national, you might get a slightly better fit with HERC over DSS, you actually get a slightly better fit with DSS over HERC. We have suppositions about why that is true, but we are going to use DSS and provide that. We are also including fee basis, which is purchased care. So if a Veteran shows up at a non VA provider, the VA pays for that. That is covered under fee basis care. So we are interested in two years of data, FY11 and FY10. 

For the diagnostic information... yes. 

Moderator:There is one question about samples. Were the study samples two to five subsets of sample one, the general sample? Or were they independently drawn from the VA user population? 

Todd Wagner:That is a great question. Only the healthy population was drawn from the general sample. All other... the four others, the high cost, the mental health/substance use, over age sixty-five and multi-morbidity, are actually all patients with those diagnoses, so they are not random samples. They are all patients with those diagnoses. If that makes sense, but the healthy Veterans, because we really struggled to define it... by the time we defined it, we just said to make life easier, let’s just go back and pull it from our general sample. So that’s what we did. Thanks, are there any other questions? 

Moderator:Not so far. 

Todd Wagner:Okay, so the data that we are going to use for VA care, we are going to use utilization, cost and diagnostic information from the national care database, largely the XE file, the patient treatment file, the DSS file and the HERC average cost data. As I said, for the non-VA care, we are getting fee basis files. One area you are not going to see here is the Medicare data and we are in the process of identifying and getting those files as well, because you can imagine that you might systematically under code certain people if they are using Medicare more. 

Here are the descriptive statistics. You will notice, for example, I said there were two million people in the general sample, and right off the bat we are slightly under two million and you might ask why. We just had to make some decisions because we are using DSS pharmacy costs; and we have had discussions with the DSSO folks about this. They know that there are some high cost pharmacies that are probably beyond what are accurate and they will tell the medical centers about it, but they don’t have the authority to remove the workload from the files. So, we just made a... if the patient had a pharmacy cost exceeding fifty thousand dollars, we dropped that record. So you can see it’s not dropping a tremendous number of records, but we just want to make sure that we weren’t influencing our model fit by what we believe to be outliers. And I should be specific, outliers where we had a suspicion there might be an error in the data. 

Let’s walk through these statistics. The general sample, the mean cost is eight thousand-eight hundred dollars per person per year. You can see the standard deviation. The maximum is one point six, so there are some very high cost patients. Now, keep in mind that is a general random sample of two million, so when you get over to the high cost users, this is the complete five percent, the top five percent, and you can see there is actually a more expensive person there. That is, there is a person who has a high cost that is almost three million dollars. 

In the mental health/substance use, there is a two point four million dollar... so those high cost patients matter a huge amount in almost all of the statistical models we do because you can imagine there’s a very, very long white tail, and that’s going to be challenging for fitting data. 

Here is the healthy sample, so when you look at sample sizes, again the top row, you will see that we are doing reasonably well on sample sizes. You could still be worried about over fitting, especially in the high cost where there is two hundred and sixty-one thousand. But the healthy is considerably smaller, seventy-eight thousand. When you look at the costs of the healthy, they are still using some care, but the average is two-point... two hundred... twenty four hundred dollars and the maximum is two hundred and seventy five. So what we were hoping to capture in the healthy population would be people, I’m assuming like many of us on the phone call, who maybe one of them got into a car accident. That would be the two hundred and seventy-five thousand dollar one, but they are relatively healthy otherwise. Remember, they can have one body system indicated. 

When you are comparing risk scores, there are many regression models. If you just use ordinary least squares, you’re going to assume a functional form depending on how you include your variable. That’s probably just a linear form. You don’t want to necessarily do that, especially as you notice that these distributions are non-normal, so the questions are the model might have a big impact on how we think about these risk scores, so we tried many different models. We tried ordinary least squares, logged OLS, square root OLS, general linear model with a gamma distribution, a log-link, we also tried the general linear model with a gamma distribution or square root link. And in the basic sets, this is aim one here, so it’s just the comparison of the data that the algorithms... those risk scores that they give out, we are just interested in some basic co-variants, age, age-squared and gender. 

Let me just walk you through the different risk scores here and hopefully this table is not overwhelming. You have, right off the bat, and I am going to try to keep this format relatively similar across the tables to try to minimize your headaches. In the general population, we have got the V21 has an average risk score of point-seven-five and you can see the standard deviation in the parentheses. You can then walk across the DCG models prospective without pharmacy, point-six-six, or slightly lower. Concurrent without RX it is even lower, and then when you include the RX it goes up considerably. Now, we are going to see this time and time again. The pharmacy matters. There is an underlying theme, or thread in this whole story that the pharmacy matters in part because people who are relatively healthy might still be on one or two pharmacy, or prescriptions. We can do a better job on the edges of the distribution when we include pharmacy information. Hopefully, I didn’t give away too much of the story here, but keep that in mind. 

When you get over sixty-five, the risk scores go up a little bit. Clearly when you go to the high cost, we are considerably higher. Notice the heterogeneity across the different risk scores there, so for the V21 the average is two-point-two, for the DCG-Medicaid with pharmacy, it is seven-point-two. You can see it is quite variable with the mental health-substance use, multi-morbidity as well, and then the healthy we are seeing... what we would expect to see is relatively low-risk scores. The idea here is that our risk scores are generally centered around one, so a person... the average risk score for a population would be one. 

How do risk scores fit the VA data? Well, there is many metrics or criteria that we can look to, to judge fit. I’m going to show you all of these at first, and then I’m going to start just because of time, and I think because of your ability to stay with it, I’m going to... and my ability to present it, I’m going to start narrowing down what I present. There is a natural tendency, for me at least, to focus on the R-squared and the Hosmer-Lemeshow. You might be particularly interested in the other ones, and I apologize if I am culling out data here, and I just think there is an interpretation that is easy for the R-squared, as well as for the Hosmer-Lemeshow test. 

Here is the R-squared statistic of the basic models. The V21 model, your R-squared... the idea here with a point four-two or a point-four-two- eight, or a point-four-three, is that you are accounting for about forty-three percent of the variants. You can see that is very, very similar on almost all of these. Those first two columns are very similar. If you were just making the decision, and we eventually came to this conclusion, the decision between prospective V21 or prospective DCG model, they look almost identical. The correlations are very high, if you actually compute the difference score, ninety-five percent of the time, they are very, very tightly together, so they seem very, very similar across the majority of these dimensions, in which case you make the decision, well let’s use the V21 which is free versus the paid software. 

Where they differ here is really in the other columns. You can say what is the V21 compared to the concurrent? Well, if you like concurrent models, you would expect a better fit because the data is more temporal. In fact, you generally see that. We see about a ten-percentage point improvement in fit, and a better fit even yet when we are using the pharmacy. Now, keep in mind the pharmacy is Medicaid data with a prospective, but generally speaking we are doing much better. Especially look at the healthy population. We go from not explaining much of the variance, to explaining about a quarter of the variant. If you are working with healthy samples, that could be quite important for your analysis. 

What you are going to look here is the root mean squared error. Generally speaking, what you’re wanting to see is lower numbers indicate a better fit, and it’s largely the same story here. The very similar numbers between the first two columns, and that there do seem to be some differences, especially when you include the pharmacy data. Here it is less dramatic for the healthy population. What I am showing here, again we are running many different models here, I can’t have all the tables here. What I am showing is the square root OLS model which was this preferred, best fitting model across the range of models. 

Then, the mean absolute error, again you’re going to see very similar numbers across the first two columns and then this slight improvement with the concurrents, then again you see some better fits with the last column. The Hosmer-Lemeshow, I think this is also a very useful test. What it is doing is breaking the sample into deciles, and for those of you who run a lot of logistic regressions, this is the way that people think about assessing models fit for logistic regressions. Well, you can run Hosmer-Lemeshow tests, the same style of test, with different types of data. What you are doing is breaking the data into deciles and you are saying how well does the model fit in that decile. What you are seeing here is average minus expected, so you are seeing that we are underestimating in the low cost deciles and we are overestimating in the high cost deciles. Remember, these are wildly non-normal distribution, so that is expected. What you are hoping to see is lower numbers. What you would like to see is zeros across the deciles. That would be a perfectly fitted model if you had zeros across the deciles. 

The V21 model gives you some pause. We have some not particularly great fit in the tenth decile. Remember, that is a very hard decile to fit because it is so expensive. Again, you are seeing similar stories across the first two columns and then as we get better fits when you include the pharmacy data. 

Our basic summary of findings is if you are interested in concurrent risk scores, the DCG offers concurrent risk scores; the V21 does not. So if you’re interested in concurrent risk scores, you have to go with DCG and that’s the comparative advantage of the DCG model. Concurrent risk scores tend to produce slightly better fit statistics. We talked about the ten percentage points, than the prospective risk model, and one could argue that is because of the temporal relationship to the data. 

If you believe in and want to use prospective risk scores, both DCG and V21 produce prospective risk scores. If you are interested in just the basic output, they produce, in those first two columns; they produce similar results across a range of samples in regression specifications. Our group actually came to the conclusion that if you were just making a choice between those two models, you would probably end up, and cost is a factor, you’re purchasing here and you have to go through VA contracting, so it is not a simple procedure of the VA paying, it is also the hassle of going through contracting. We would probably choose the V21model. 

What gave us a lot of pause when we came to this conclusion was the benefit... the added benefit of the pharmacy data. You saw noticeable improvements across the fits when you said there’s pharmacy data included. We started pondering about what does this mean for VA, especially because VA has pharmacy data. We could make use of the pharmacy data even if the V21 model does not; so this led us naturally to aim two. 

We started saying well, what happens if we started monkeying with this risk score and adding variables, and is there a way we could add pharmacy information for VA that would improve our models. And you might say, well maybe it is going to improve the fit for everything, or you might find a scenario where it is [inaud] improvement, so you improve the fit, but that we start to look more and more like that DCG Medicaid with pharmacy model, so that is what we are trying to figure out here. 

Any questions about aim one before I jump into aim two? 

Moderator:Not at this time. 

Todd Wagner:I am hopefully overwhelming everybody with so many numbers that they cannot keep track. Aim two... re calibration. We are going to add more co-variants. We are going ot add race, marital status, other health insurance variables that we have in the VA data, administrative data, the Veteran priority level. There are a number of registries that identify Veteran exposure; for example, Agent Orange; and these registries can be important. We also talked to Amy Rosen and she has developed, and keeps active a psychiatric condition category risk adjustment model, that has forty-six HCCs for psychiatric conditions, so we pull that in. Now, you might say, and we have questions raised about this, well doesn’t your other model have psychiatric conditions? It does and we are not overly worried about colinearity. What we are really worried about is the end model fit. We are not particularly interested in any specific HCC. Then we are also going to try to take advantage of the pharmacy data, you know VA is a big pharmacy provider. 

We are going to re run every analytical model from aim one, and then we are going to estimate the new risk scores for each patient. The true hero in all of... well, there’s a number of heroes. One of the true heroes in this was Anjali who was able to code this very, very slickly and rerun this. There is a lot of behind the scenes stuff in SAS and then a lot of work in STATUS to make this work efficiently. 

So pharmacy, you can imagine many different ways of pharmacy data being used to inform risk adjustment. As an economist, I thought wow; the easiest way is we have the pharmacy spending. Why if we just use last year’s spending as an indicator of this year’s, or you can imagine this year’s as an indicator of next year’s spending for pharmacy. Turns out that it did not help at all. The model improved just fractionally, just a small amount. So we are like so it’s not an intensiveness of pharmacy, maybe it has much more to do with the extensiveness of the pharmacy. Maybe it is just a different category of pharmacy, so PBM has a drug class category and you can actually see this on their web site on the internet, it is not even on Tredmed, it is internet. We are interested in these twenty-six drug class categories, which let me tell you a little bit more about because they become important. 

So PBM maintains an alpha numeric list of five hundred and eighty drug types within twenty-nine drug classes. When we pulled our data, we saw that three classes were not frequently used, so we came up with just twenty-six. Here is an example of what these drug classes look like. Antimicrobials, that is things like penicillin, central nervous system agents, antidepressants, diagnostic agents, rectal solutions. Well, it turns out that rectal solutions, you would not expect, it turns out to be important. Those are things that, for example if you are getting constipated that might be important and you might think to yourself why does constipation matter. I raised that question and Steve Asch jumped in and said, well if you’re in chronic pain and you’re using a lot of opioids, you’re going to get stopped up, so that is one way to try to fix that problem. Then vitamins, so you get to see that there is a potential for a lot of overlap here, but we are measuring across these twenty-six, and we are only interested in any or no use in these drug classes; and it is a very small litmus test. If they had one prescription for example in CN category, they got a flag of one there. 

Here is our... I am going to start throwing the same type of results here with our goosed up models, so this if you know turbo models or calibrated model. Here is the V21 model and we are going to present you with two different columns here. One is... the first column is without the pharmacy information and the second one is with the pharmacy information. This is just the drug class indicators. In the general sample if you remember we were down around point-four-two-eight-seven if my memory was correct in risk scores for the first general sample perspective, and boy did that improve a lot, so we are now at point-five-seven-nine-three. 

If you include the pharmacy information, we are incredibly higher than that, so we are now up into an R-square of point-six-nine. How do these compare? You might say if you are actually looking quite similar now to the DCG model with the pharmacy, and in some cases we are predicting higher R-squares than that model, so healthy, just opposite, trying to remind you where we were. Originally they healthy, the first column, the number was point one-six. What we are seeing now is R-squares of point two-nine, so a thirteen percent improvement and if you include the drug class, it’s point four-five, so that’s considerably better. Then you can see how we compare to the different models, so you get to see the improvement when we include the perspective drug class indicators. 

We were all pretty psyched I have to admit when we saw it. we didn’t know if we could actually do this, and we seem to be matching the DCG model with the V21 model. Now, this is a very data intensive process because you have to process in the algorithm every piece of pharmacy data, in-patient data, outpatient data and you are running through that with the HCCs. This also includes Andy Rosen’s psych HCCs in there and the registry variable, so that is the other reason why I am breaking out that first column and why you see that improvement. And just... reminds me... when I said the psych HCCs are in there, that model improved from point three-nine-eight-five to point five-eight, so a twenty percentage point in improvement and fit. 

I am going to skip the... 

Moderator:Tom? We just lost your audio; it went very, very quiet. Did you click a button? 

Todd Wagner:I did not do anything, am I back? 

Moderator:Yeah, just speak up a little bit, thanks. 

Todd Wagner:All right, sorry. 

Moderator:Also, we have pending questions; do you want to touch those now or a little bit later? 

Todd Wagner:Let’s take them right now. 

Moderator:Okay, Elizabeth, would you like to moderate them? 

Elizabeth:Sure. The first question refers back to aim one. The R-squares are much higher than I have seen for observations on individuals, you listed each age, age-squared in gender as co-variants, did you get those R-squares with only those co-variants?

Todd Wagner:Yes, we got those R-squares as only those co-variants with the risk score in the model. To be very specific, you can imagine a linear regression, which has the risk score, age, age-squared, and gender. 

ElizabethClarification, are the forty-six conditions from Rosen the condition in the psychiatric system? 

Todd Wagner:Yes, that she published in Medicare, or Medical Care is the journal and we just contacted Amy and she was able to provide us with the SAS code for those indicators. 

ElizabethThere are questions about how to get the V21 data and the DXCG data, do you want to address those at the end? 

Todd Wagner:Yeah, let me hold those off if I may about how to get those data. That is probably best at the end. Thank you so much Elizabeth. Is my voice loud enough for people to hear? 

Moderator:Yeah, you are fine now. 

Todd Wagner:Okay, great, I am worried that I am losing you. I have this view overlooking the Chesapeake, which I am not used to, so... it is an odd place to be giving a cyber seminar. Okay, so we are back on R-squared. I am going to move onto Hosmer-Lemeshow, I am going to skip the mean absolute error; and the root mean squared error and what I sense in some of the questions is a desire to look at even further under the hood. We actually have regression outputs. If you are particularly interested in working and seeing regression with the beta coefficients of the actual HCCs and the risk scores and so forth, we can provide you and work with you on that. It is a little overwhelming I must say. 

Here is just the Hosmer-Lemeshow, keep in mind that what you’re ideally trying to see across the spectrum is zeros and it’s... you never do see zeros, and you get to see where we are fitting. Here is the place where it gave us a little bit of pause. When you include the VA drug class, we actually seem to be... we have improved our fit dramatically on the low end of the spectrum, so the low cost end of the spectrum, but we seem to have done a little bit worse on the high cost end of the spectrum. We are not doing as well there, so it is not entirely a win situation here where everything came out better. We do not know exactly how the DCG pharmacy coding is happening, to us it is a black box, but we have seen that there are still some advantages to there. If you are particularly interested in, for example, that tenth decile, the highest cost users, you can imagine a situation where you still wanted the DCG risk score. 

Then the idea is that you can take these regression models, you can predict the person’s cost, then divide it by the average predicted cost and it comes out with a risk score that’s centered around one. And you can do this with split samples, you can take a million of the two million, and typically we talk to a lot of people about what’s the right split. A lot of people use a fifty-fifty split for historical reasons more than anything else. You can think of an eighty-twenty split, we just chose a fifty-fifty, but you can run these models to make sure you are not over fitting the data, that is why people do these split sample validation. The data looked pretty good in all those cases, so here is the end result in the predicted risk score. Again, this is the general sample, the first row, it has a mean of one, so what we are going to do here is take the risk scores for all of VA, we are going to predict it, so that’s why you’re not seeing means of one. They are not getting recalibrated for every sub sample. They are getting recalibrated at the VA level for the whole population and then each sub sample has different average scores. 

The max risk score that you see in the general sample is a forty-one. The over sixty-five, we see a slightly lower risk score, but again, there’s a fair distribution here and if you walk down the samples, the high cost makes a lot of sense about five times the risk score of the general sample, and the highest is forty-five. Healthy, about point-three-eighths of these in many regards have a lower mean risk score across the sample. 

The model really matters and many people use OLS. It works very well; we had a number of problems with our GLM models, not so much over fitting, actually more on the convergence problems. So let me just show you... one of the things that matters here... the model matters, so I circled two things, one is the OLS model and the square root OLS model. This is just a model with prospective risk score in it and you get to see that there is actually considerable improvement in the square root OLS model, and that is why we used it across the board. Now the square root... you are taking the square root of your cost data, and then you are running OLS on it. So it does different things with the data, and then to get back to risk scores you have to use a Smeering estimator for the square root OLS on that as well. It is slightly different from the Log-Smeering estimator. 

Notable gain from pharmacy, so here is the V21 model prospective. I have highlighted a couple of these and again, showed you the differences across the model fit. If you look at the prospective without pharmacy, and the prospective with pharmacy, the pharmacy data matter a lot. That is a recurring theme that we see throughout the work that we have been doing, is that the pharmacy data matter, and if you are really wanting to do risk adjustment depending on your sample, it seems to make sense to try to take advantage of the pharmacy information in VA. 

Recalibration is possible and it yields an improved fit, so in the first row here is the basic, so this was aim one. In the second row is the recalibrate, this was aim two, and so I am going to get to show you how things change. That point four-two-eight-seven with the new variables in the model move to point five-nine-three. Now, you should be careful, this is R-squared values. I did not note that, but I should have noted that; and then you also see with the pharmacy. Originally, we did not have it because CMS does not have that information. It was only through the R-work, but you get to see what happens now, we get a point-six-nine-two-four and you get to see how that compares with the Medicaid VCG model with pharmacy, point six-three-five-one. So the re calibration is possible and yields notably improved fit. 

At its core, there are different numbers of HCCs. They seem to be working in different ways. If your particular interest is in specific HCCs, we have not done any analyses looking at that, so that is clearly one limitation. Another limitation is we have this grandiose idea that we were going to compare all the risk adjustment systems and we could be done in six months. Even wearing operation tags, we were only able to get two comparisons done. It is a lot of data, it’s a little bit overwhelming to start doing analyses and looking at the different criteria for judging model fits as well as the six different samples we pulled, as well as all the different regression specifications that we tried. 

We then thought about... I should also be very clear, when the commercial world develops risk adjustment systems; they have to pay particular attention to gaming. If these systems are being used by Medicare Advantage, you can imagine a system, a healthcare plan, that thought about how it wanted to code its data or getting people care so they would maximize reimbursements. In the VA that’s a little bit less of a concern because all of our results here, although they might be used by operations and research, aren’t affecting VERA which is the Veteran’s Equitable Resource Allocation and how medical centers divisions are given money for patients. 

These risk models are not to be used for payment, so a little bit less of a concern. When we reach back to RTI and said Hey, would you be interested in these results, and we talked about them with them, and those conversations are ongoing, their main concern, for example is that rectal suppository and when we showed that really made a difference was, boy that could easily be used in gaming. They were worried that health plans would start prescribing people at a higher probability, rectal suppositories to get higher reimbursement rates, so we do not have that concern, where the commercial world does. 

There is a lot... this is a huge literature... and it is hard for me to jump in and stand on the shoulders of giants here; but clearly, when you go back in time, there’s Randy Ellis and Arlene Ashe, have done just a tremendous amount. Greg Pope, who used to be with them and is now with RTI, has done a tremendous amount. In the VA, it is just tremendously helpful to have Amy Rosen to call on and Bruce Kinosian to call on. My neighbor, Karen Phibbs has done some stuff on this, so there is just some tremendous resources in this area for that. Perhaps this is the time to get to the questions, and if there are clarifications or broader questions about the data, I get to get there. The thing I also wanted to... about two months ago, I posted a question on HSR Data saying who is using HDCs? I should have been a little bit more careful about that because a lot of people wrote back in. So, one of the questions we are trying to gather now, is who is using HDCs risk smart software and what would it take if they were to migrate to a different algorithm, whether that’s the risk solutions algorithm, or to our recalibrated V21 model algorithm. So, we have had a number of conversations about that so if people have questions about migrating to a new system that we might be building, I would really love those questions as well. 

Moderator:All right, we will start with some of the questions we have. The first is, could you describe re calibration in a little more detail? 

Todd Wagner:Sure, the basic model that we are looking at here, for example, when you run the basic software systems, out pops a risk code. You have provided a number of parameters, you have told the software this is my field for diagnostic information, this is my age/gender information, and that goes through and it calculates the risk code. In our basic models comparing these risk systems, we are just using that risk estimate plus age and gender. In the re calibration models, what we are doing is we are adding more variables to it and then we are centering the average risk score around one. So it’s just retransforming the data around a central value of one. So if you saw a value of one, you’d say this is the average risk score for the VA population. That is the idea behind this. 

Moderator:Often, data mining projects set aside a portion of the data set for testing the models and other fits, for final validation of the selected model, for example, eighty percent for training, ten percent testing and ten percent validation. The cross validation and Hosmer-Lemeshow evaluate the reproducibility, the liability and reliability, but may not fully assess accuracy. Could you comment? 

Todd Wagner:That is exactly right. You have to be, especially with the GLM models. The gamma model has more parameters and is susceptible to over fitting. If you were just to do your model on everybody, you might end up with a model that works well with the data you built it on, because it was fit to that, but it might not work well on other iterations of the data whether that is a different year or other alternative universe of data. We did hold that aside, and we did rerun our models. I was not accustomed to using the square root OLS model, and Maria Rath-Montez  has done some work, and Amy Rosen has done some work, she’s actually the one who cued me into this model, so we actually had to do some research on what is this model, how does this model work. The other concern you might have with a square root transformation, if you end up with risk scores that are negative, when you Smeer them back, you could end up with what’s called a spike meta [inaud] you could imagine a little plot distribution because you’ve reoriented, or retransformed the data. When you square something, when you square a negative number it becomes positive, but a square that is point negative two-five, the resulting number is different if it is point-oh-oh-four; and you can imagine a reordering of the data. So we test it to make sure we were not reordering the data or if it was, it was small. We actually ended up not reordering the data at all, so there is a bunch of these other types of specification tests. I was actually very pleased with the square root OLS model, so hats off to Maria and Amy for helping me navigate that. 

We did run the models with split samples. The question was about eighty-twenty, eighty-ten-ten. We used fifty-fifty; we asked questions about if there was a best way to do this. There are many alternatives and people have even suggested running it across ten different samples and comparing across the tens... we just did a fifty-fifty for expediency. Hopefully, that answered the question. 

Moderator:Are there any plans to compare the ten score to the recalibrated risk algorithm?

Todd Wagner:Another great question. People are probably much more, or becoming more familiar with the CAN score we had on a prior HERC cyber seminar. Steve Finn was gracious enough to talk about the CAN score, and I know there are other talks going on. I think there is one coming up if I am not mistaken, about the CAN score. The CAN score was designed to my knowledge based on the idea of trying to predict mortality and readmission. What we are trying to do here is predict cost. You can imagine a scenario where the two of them do not function the same way. Cost and readmission, although if you readmit a lot, you are going to have higher costs, is not necessarily the same thing. Originally, we thought we were going to compare it to the CAN score in the bake off. For simple fits of trying to keep the project tractable, we decided not to go into all these other competing risk scores. One of the reasons for that is that the V21 model from CMS is publicly available, and so Eileen, her eyes perk up and her ears perk up when she sees that she does not have to go through contracting to keep this software going. The other thing with the V21 model is we can go back in time and we can repopulate data sets. We can go back, for example, I talked to Matt in ITF, he said he would want ten years data going back and I said we could do it. It is just data, it is just processing. It is not functionally hard. 

Moderator:How do you proceed with such cost analyses for ongoing quality improvement projects? 

Todd Wagner:If I am interpreting the question correctly, let’s say you’re doing a comparative effectiveness of two different treatment types. You believe there is clinical heterogeneity that you need to control for and you might have different statistical methods, you might want to say, well let’s include some fixed effects but you might also decide that you want to do a Charleson comorbidity score, or you might want to use one of these methods. What we are hoping to do is these data would be then posted. The risk scores for every Veteran across multiple years would be posted on for example CDW. Whether you’re accessing CDW directly or through VINCI,  you would get the risk scores and then cross walk it using social security number and you would say here’s the risk scores, and you could include that risk score in your regression or your analysis for controlling for it. The long-term goal is to do that. Whether we continue with the contracting of DCG, or we decide to build our own model, so this user’s group, folks on the phone call, we value your input on this. 

Moderator:Could you briefly describe where people can get the DCG data and the V21 data and is there a cost associated with it?

Todd Wagner:The V21 data, there is a cost with it. You have to contract with Verisk to get the algorithms in the software to run and my memory, although I could be wrong, is that you pay per record that you run it through, or run through it. It is not... for individuals it is expensive. For VA it is not overly expensive, but it is not free either. The V21 model, Elizabeth I’m not the one who downloaded the software, but it’s downloadable from CMS’s web site and I believe we have all those links in our records, but I don’t remember off hand where they are. 

Moderator:I can send the link to be added to the notes. 

Todd Wagner:Okay, that would be great and you know, we are in the process of improving these re calibrated models, so if we get to the point where we start using that, or that is the suggestion, we would be happy... we have nothing to hide here. We would be happy to repost those models and show people what we have. If people want to improve upon our models, that is great. 

Any other questions? 

Moderator:Just a moment, I am trying to piece together a question. To do a cost analysis, do you need to partner with CDW? 

Todd Wagner:Yes, that is a good question and I have to recognize there may be people on this phone call who are non-VA or primarily at the academic affiliate, so CDW is a... think of it as a server, a data repository that VA is maintaining. You have to have appropriate access to get access to that server, and that if you are operations, you can touch it directly. If you are research, you touch it through the Vinci portal. If you are non-VA, you cannot touch it. There is no way to get access to it. You can work with VA researchers, and I would encourage that. If that did not answer the question, feel free to clarify or send us another question or contact us off line. 

Moderator:I think, at the moment, that’s it. 

Todd Wagner:Well, we are rapidly approaching the top of the hour, so hopefully people got a flavor for what can be done with risk adjustments and where we are hoping to go. What I would really ask people to do, and I know people are starting to sign off the call, if you have opinions on where we should go, I would love to hear them. From our group, we are starting to think about maybe we should do the re calibrated model and use the V21 model for VA, and go back and create assessments from ten years of data and post them in places that people can access them with appropriate permissions. If people have strong feelings one way or the other, we are definitely all ears. 

Moderator:Thank you Todd, we really appreciate you coming on to share your expertise. I do want to take just a moment to plug the next health economics seminar, which will be taking place on July 17, at 2 p.m. eastern, and it’s on estimating readmission rates using incomplete data, using implications for two methods of hospital profiling, and that will be presented by Doctor William O’Brien. 

Once again, Todd... 

Todd Wagner:Do you have the information... there is two people, I think it is Tamara Box and Steve Fihn are talking about the CAN score. It is probably worth plugging that one as well if people are interested in the CAN score. I think it is coming up, I do not think we have missed it. 

Moderator:You are right; it is the 27th of this month at 1 p.m. eastern; it is on care assessment needs, also known as CAN, care assessment needs score and the patient care assessment system, PCAS; tools for care management. That is actually being presented by VIReC, the clinical informatics cyberseminar session, and again as you mentioned, it’s Tamara Box and Steven Fihn, so feel free to go to our web site and register for that session as well. 

Thank you Todd, thank you Elizabeth and thank you to our audience. 

Todd Wagner:Sounds great. Hopefully, people will contact us if they have further questions. 

Moderator:I think we already got some confirmation that you will be hearing from some people. 

Todd Wagner:Okay, great. 

Moderator:Thank you, have a good day everyone; and please fill out the survey at the end of the session. 

Page 1 of 16

Page 16 of 16