**Transcript of Cyberseminar**

**HERC Health Economics Seminar**

**Estimating Readmission Rates using Incomplete Data: Implications for Two Methods of Hospital Profiling**

**Presenter:  William J. O’Brien**

**July 17, 2013**

**Estimating Readmission Rates using Incomplete Data: Implications for Two Methods of Hospital Profiling**

**July 17, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact:**herc@va.gov**.*

Todd Wagner:I just wanted to welcome everybody to the July issue of the HERC Cyber seminar. I hope everybody is having a good summer and I wanted to thank Bill O’Brien for his willingness to present today. Bill is an analyst at the VA Boston Center for Organization Leadership and Management Research, COLMR, he has a master’s degree in economics from Suffolk University; and prior to working at the VA, he has done a bunch of work estimating time series and forecasting models for an economics consulting firm. So it is actually a great link to his topic today, which is of great interest to the VA and also to Medicare, which is Estimating Readmission Rates Using Incomplete Data and the Implications for Two Methods of Hospital Profiling. So with that, hopefully I can turn it over to you, Bill. Thank you so much.

William J. O’Brien:Thank you, Todd, for the introduction and thanks so much for inviting me to do this talk today. I will be presenting a recent study we did that involves examining the effects of including Medicare data in an assessment of VA hospital readmission rates. Specifically, we are looking at how the introduction of Medicare data affects hospital profiling results.

Okay. Before we start with the content, I would like to do a poll to see who is in the audience. The poll question is, what is your primary professional role? Researcher, clinician, quality manager, hospital administration or other. And so we will wait just a moment for answers to come in.

Moderator:And there are your results.

William J. O'Brien:Okay. Okay, so about half of the audience is involved in research, 9 percent clinicians, 7 percent quality manager and 7 percent hospital administration and almost one-third other. Okay, great. Thank you.

Okay. So hospital readmission is a hot topic in the health services literature, especially in the past few years. For several years now, hospital level readmission rates have been publicly reported on two consumer-oriented websites and those are CMS Hospital Compare and VA Hospital Compare, and we will take a look at those. In addition, you probably know that CMS in October of 2012 began to penalize Medicare-reimbursed hospitals under the hospital-readmission reduction program, and those penalties in the first year of the program were approximately $280 million.

Both of these measures in CMS and the VA potentially made important information about dual usage. Among the Medicare population there is certainly a large proportion of patients that use outside healthcare from, for example, Medicaid or a spouse’s commercial insurance. Among the VA population, it is certainly known that a large proportion also use outside healthcare from Medicare and Medicaid and other sources.

The purpose of our study is to determine study changes in VA hospital readmission rates and especially hospital profiling results after including Medicare fee-for-service records.

Okay. So before we get into the study, I just want to give a little bit of background about what is out there in terms of publicly available information about hospital readmission rates. This is a screenshot from the CMS Hospital Compare website. Anybody can go to this website. You can pick a geographic location and a medical condition. This is Boston area for heart attack patients, and you can pick up to three hospitals to compare. You will notice that it has VA hospitals as well as the vast majority are non-federal hospitals.

And here I have picked Boston Medical Center, which is a Safety Net Hospital; Mass General, which is world-class teaching hospital; and the VA Boston Medical Center. So there is some good information once you get to this detail page. You see that the National Readmission Rates for AMI patients is 19.7 percent. 

You can compare the hospital profiling results for up to three hospitals. In this example, BMC has a risk-adjusted readmission rate that is worse than the U.S. national rate and MGH and the VA are no different from the U.S. national rate. So this is what we mean in terms of hospital profiling. It is simply labeling hospitals at least in hospital compared as good, bad or average.

There is some other interesting information at the bottom of the page. There are 4,500 hospitals that are being looked at in this methodology. Twenty-one hundred do not have enough volume to make a determination, and out of the remaining 2,400, the vast, vast majority are no different from the national rate. Thirty hospitals out of 2,400 are better than the U.S. rate and 41 are worse than the U.S. national rate, and we will see why that is.

So here is a different example of hospital profiling. This is also available as public information from the CMS website. So I mentioned that the HRRP will reduce base GRG payments in FY13 by about $280 million and this is the data that drives that figure. 

Each row in this spreadsheet is a hospital and in column D it gives the payment adjustment factor. A value of 1 means no adjustment to base GRG payments and a value of less than 1 indicates a reduction in payments for the fiscal year.

The range of adjustments in FY13 is a max of 1 percent of payment penalty, and it will increase to 3 percent in FY15.

The rest of the columns deal with condition-specific readmission rates. You have the number of cases and the excess readmission ratio for patients in each condition cohort.

Well, look at this hospital, for instance. It has a reduction in GRG payments, so it is labeled or profiled as having excess readmissions. And the reason for that is this hospital had excess readmissions among pneumonia patients, even though it had fewer than expected readmissions among heart failure patients and it had too little volume to make a determination for AMI patients.

In contrast, this hospital on row eight of the spreadsheet had no payment adjustment factor because in all three condition cohorts, its excess readmission ratio was less than one, so it had fewer than expected readmissions. So we are replicating the hospital compare and the payment penalty profiling method in our study.

This is another example of a hospital compare website. This is for VA. It uses the same methodology as the CMS Hospital Compare website. It shows some similar information such as the national readmission rate and you can pick a U.S. state and a medical condition and the UB readmission performance as well as other outcome and process measures in a table like this. It gives you the medical center name, the readmission rate; and importantly, it gives the interval estimate and that comes into play and I will demonstrate that later. And each hospital is labeled as lower than, within, or higher than the national VA rate.

Okay. So that brings us to a new poll question. I have been really curious about this lately. Do you know anyone who has used VA or CMS Hospital Compare to guide personal healthcare decisions?

Okay. So 27 percent yes and 73 percent no. I guess that is pretty much what we expected. Okay. And thank you for answering that.

Okay. So getting into our study, we looked at index admissions MZA during fiscal years 2008 through ’10. The patient sample was veterans who were dually eligible for Medicare because of their age in 65 and older. For data sources, for inpatient data we used the VA Patient Treatment File as well as the MedPAR datasets for Medicare Inpatient claims. And on the outpatient side, we used the Outpatient Encounter File and the Carrier and Hospital Outpatient Datasets for Medicare outpatient data. And we obtained Medicare outpatient and inpatient data through VIReC. 

A few definitions. These are very consistent with the CMS methodology. We defined an index as an acute care hospitalization where the patient leaves not AMA to a non-acute setting. The principal diagnosis had to be AMI, heart failure or pneumonia; and we created three separate, independent cohorts based on principal diagnosis during the index. And the patient could not have had another index discharge in these 30 days prior to the current index discharge.

Readmission was the first acute care admission during the 30-day post-discharge period. And in order to not flag likely planned hospitalizations, we looked at the procedure codes and potential readmissions and we excluded any that had procedure codes for things like revascularizations  that were likely planned. And the hospitalization could not have been both an index and a readmission within the same model. It had to be either one or the other.

Why is it important to look at dual use when looking at readmissions? This represents our study cohort for three years of data. For all index admissions we looked at the proportion of those having at least one Medicare inpatient claim during the study period. And we found that it is—this is already well known, but it is not uncommon for a patient to use Medicare inpatient services, and these are all again, 65 years of age and older, so the dual-use rates by this measure ranged from 39 to 50 percent.

Okay, now we will go through the mechanics of how we identified index admissions and readmissions and how we worked in the Medicare data.

For the baseline analysis, we looked at only VA administrative and patient and outpatient data. So we identified VA index admissions, looked ahead 30 days and determined whether or not the patient had a subsequent readmission to a VA hospital. So it is a dichotomous outcome that we were looking for.

Once we did that, we included Medicare utilization to see how that would affect things, and there were a few different important ways that Medicare claims records fit into this picture. The patient may have had a Medicare admission just prior to a VA index admission, or the patient might have had a readmission to a Medicare hospital but not to a VA hospital in the 30-day post-discharge period. A patient might have been actually transferred to a Medicare-reimbursed hospital from the initially identified VA index admission and that has consequences as well. So in the next few slides I will go through each one of these cases and say why it is important.

So of finding new readmissions, if when we looked at VA-only data and saw that the patient did not have a readmission to a VA hospital, that index admission would have been flagged as having a negative or no-readmission outcome. If the patient, it turns out actually did have a Medicare readmission but to a Medicare hospital, that is important because the readmission outcome for the VA index changes from no to yes.

We also had to exclude some initially identified VA index admissions, and there were two reasons for this. The first case was when the patient had a Medicare admission immediately prior to a VA index admission. In cases where they were both for the same condition, say they were two AMI potential index admissions, we would exclude the VA index admission.

And in the second case, we found that it is not uncommon for a patient to be transferred from an initially-identified VA index admission to a Medicare hospital. In this case we do not want to attribute any readmission outcome to the first hospitalization at the VA. If anything, we would want to attribute readmission to the Medicare-reimbursed hospital. So we decided to exclude those as well in our study.

These two effects, the conditional readmissions and the exclusions, had a noticeable effect on observed readmission rates. In the AMI cohort, the observed readmission rate went from 20.7 to 24.2 percent; 22.5 percent to 26.5 percent for heart failure patients; and for pneumonia patients, 17.7 to 20.8 percent. So observed rates increased by a 3.1 to 4.0 percentage points from finding extra readmissions to Medicare.

You will also notice that the number of index admissions at risk for readmission changed slightly between the two analyses and that was due to the exclusions that I mentioned.

So we have talked about exclusions and finding new outcomes. Now we will move on to identifying risk factors, which will be used in risk adjustment models. 

To identify patient risk factors for readmission, we looked at several different things. We looked at secondary diagnoses during the VA index admission and we also looked back at the one-year pre-index admission period at VA inpatient and outpatient administrative data, and we flagged risk factors based on the presence of certain ICD9 diagnosis and procedure codes. 

When we obtained the Medicare data, we included those claims in the one-year pre-index period, and these were potentially a source of more and richer risk adjustment diagnoses. In cases where the patient had certain diagnoses coded only in the Medicare data but not in the VA data, that would increase the prevalence of risk factors overall.

So the next poll question is, will the additional Medicare clinical data increase the prevalence of risk factors for readmission? Another way of putting this is, are there a lot of diagnoses coded in Medicare data but not VA data for these patients? And the options are slight increase, significant increase, no change or not sure.

Todd Wagner:Bill, can you hear me?

William J. O'Brien:Mm hm.

Todd Wagner:This is Todd. So there is a question that came in that I will ask as people are answering your question here, which is, in general when people think about readmission: is it all-cause readmission or is it same diagnostic information readmission, so they are returning from their pneumonia or ...

William J. O'Brien:Yeah. It is almost always for all-cause readmission. I think CMS has it in to looking at condition-specific readmissions because then it might be open to gaming. So the overall feeling is that any readmission outcome, no matter what the diagnosis, counts as a readmission. So the diagnosis during the readmission does not come into play during this.

Todd Wagner:Right. It is not so hard, I think, clinically to identify if it is truly an MI sometimes. You take some of these data points ...

William J. O'Brien:Oh, yeah.

Todd Wagner:... an MI being one of them, and we think of a severe MI and there are clinical ways of diagnosing it. But on some margins, it is actually very hard to clinically define whether it is an MI or not.

William J. O'Brien:It is. And in our study that we are doing right now, we are doing chart review for MI patients. And I am not a clinician so I cannot comment with too much detail on this, but you are right. It is very difficult based on administrative data codes to identify true MIs sometimes.

Todd Wagner:Thanks.

William J. O'Brien:It is one of the limitations of using administrative data in general. Okay. 

Okay. So we have about half of the people expecting a slight increase, one-fifth expecting a significant increase and 8 percent no change and one-quarter of the people not sure. Okay.

So let us go to the results for pneumonia. There are really no right or wrong answers—oh, I am sorry. I am showing my screen. This shows the prevalence of pneumonia risk factors. Some risk factors—and I should start off by saying that these are CMS condition category based risk factors, so it rolls up ICD-9 diagnosis and procedure codes to broader condition categories. And these are judged by CMS to be clinically relevant to risk of readmission for pneumonia patients, and they are condition-specific.

I have shown the top ten and the bottom ten risk factors in terms of their relative percent increase and Medicare data is included. Some risk factors such as – an example would be cardiorespiratory failure or shock, condition category 79, increases from 9.6 percent of index admissions to 14.6 percent of index admissions for a 50 percent relative increase. Septicemia and shock condition category 2 increases from 2.8 to 4.9 percent; it is a 74 percent relative increase, although from a low baseline. 

If you look at the bottom section, most of these had single digit increase. There are several cancer categories; COPD; diabetes, probably not surprising; drug and alcohol abuse that had very small increases.

So we have identified the risk factors for readmission. We have identified the outcomes, hopefully getting better information from Medicare data. Then we ran risk adjustment models to be able to do a fair and meaningful comparison between hospitals that have a different case mix.

We followed the methodology of the CMS readmission measures. We used HGLMs to account for clustering of patients within hospitals and we estimated 30-day all-cause readmission as a function of patient demographic and clinical characteristics. The patient demographics were age and gender. We specifically did not include things like income or race, and the clinical characteristics were mostly condition category based and there are a few others, for example, the location of the MI, anterior versus other. We expected that Medicare data would affect risk-adjusted rates by changing readmission outcomes from no to yes and adding potentially better information about patient risk factors.

So this is our last poll. What is the effect of the additional outcome and risk information on models’ predictive ability of 30-day readmission? And if you have a staff background, we are specifically going to be looking at the change in the C statistic. And the responses are slight improvement, significant improvement, no change or not sure.

Todd Wagner:And while people answer that, Bill, one more question came in, and it has to do with the idea of moving between centers that are categorized as excessive versus acceptable. The question is: Did the new model increase the variability admission rate in the national cohort?

William J. O'Brien:The variability of readmission rate.

Todd Wagner:I am not – it is a little bit vague so I am struggling, and perhaps that person could write more specifically if there is a slide number that [overlapping voice].

William J. O'Brien:I would ask for clarification on that one, Todd.

Todd Wagner:Okay. Will do.

William J. O'Brien:Okay. Thank you.

Okay, great. So we have the results in and 40 percent—oh, this is tied, this is great. Forty percent, slight improvement, 40 percent significant improvement, and a few for no change or not sure.

And we will take a look at the change in C statistics. The answer is it made virtually no different. In the AMI model, the C statistic went from 0.614 to 0.621 and in heart failure and pneumonia the discrimination ability of the models actually went down very slightly. So overall there was no change in predictive ability from getting more accurate outcome information and risk information. And we are not really sure why that is. I would love to hear from people.

If you have been keeping up with the literature on comparing models in terms of C statistics, there is an additional statistic that we ran called the Integrated Discrimination Improvement Index. That allows for a better comparison between C statistics for competing models, and that result was either zero or slightly negative for all three cohorts.

Once we ran the risk adjustment models, the main output of interest from these was the P/E ratio. We saw the P/E ratios earlier in the presentation. That was in the spreadsheet from CMS. The P is the predicted probability of readmission uses both fixed effects and hospital random effects, and the expected probability uses only fixed effects.

And an interpretation of the P/E ratio is, did a hospital have more or fewer readmissions than would be expected from a typical VA hospital controlling for case mix? So this is not real essential to the study, but still I thought kind of interesting. 

These are the P/E ratios for every hospital in the pneumonia cohort. The dots in the lower left quadrant had better-than-expected readmission rates whether or not Medicare data was included. The hospitals in the upper right quadrant had higher-than-expected readmissions whether or not Medicare data was included.

In the bottom right quadrant, these hospitals appeared to be better performers once Medicare data was added; and in the upper left quadrant, there are some hospitals whose P/E ratio got worse when Medicare data was added. And for some of these, the change was quite large in terms of payment penalties.

And it is worth pointing out at this point, since I had mentioned the method, we are using this CMS payment methodology to profile hospitals—of course this does not really apply to the VA—the payment penalty methodology is just – we are looking at it as a way of profiling hospitals. A way of weighing them as better than or worse than expected. And the payment penalties that CMS levies against some hospitals are simply a consequence of having more than expected readmissions.

So let us look at hospital compared profiling. This is a very basic idea of how CMS and GA get to the profiling results for hospitals.

We looked at the P/E ratio for every hospital within each cohort. We then calculated 95 percent confidence intervals for the point estimate of P/E ratios. And for hospitals having a confidence interval strictly less than 1, those would be labeled as better than expected. If the confidence interval crossed 1, it would be labeled as, as expected or within the national rate. And hospitals with a confidence interval greater than 1 would be labeled as worse than expected. So that is what we are doing to VA hospitals in our study and these are the results.

So it turns out the additional VA/Medicare outcome and risk data does not make that much of a difference in terms of what would get reported on VA hospital comparisons. There was one hospital in the AMI hospital that was rated discordantly between data sources and there were two in the heart failure and pneumonia models that were rated discordantly. So this represents less than 1 percent of VA hospitals that would see any change in hospital profile results as a result of incorporating Medicare data into the analysis. So that was the first profiling method. 

The second profiling method is based on the CMS payment penalty profiling methodology and here is the basic idea behind that, and this reflects what we saw on the spreadsheet at the beginning as well.

Hospital A here would be labeled as having no excess readmissions because in each of the three cohorts, its P/E ratio point estimate is less than 1. Hospital B would be labeled as having excess readmissions in at least one cohort because in this example its heart failure P/E ratio is greater than 1. And so we are applying this to the P/E ratios that we computed from our VA data and Medicare data.

These are the results. It turns out it makes a lot bigger difference with this profiling method. Okay, there 11 VA hospitals that had excess readmissions in the VA-only scenario, and those turned out to have no excess readmissions in the VA-Medicare analysis.

There were six hospitals that had no excess readmissions in the VA-only analysis, but then when we introduced Medicare data, they turned out to have excess readmissions. So that is 17 out of 130 hospitals, about 13 percent of hospitals would be labeled differently depending on whether Medicare data were included.

So to summarize the results, the incorporation of Medicare data did not make that much of a difference in terms of hospital profiling in the hospital-compare method. But if we simulated the HRRP profiling methodology, we would see that 13 percent of VA hospitals were classified discordantly. And as an aside, this was not again really essential to the study, but it is of interest: the additional risk and outcomes that we gleaned from Medicare data did not improve the discrimination ability of the risk adjustment models. We were kind of surprised by this.

So I think our study suggests that inclusion of Medicare data when calculating GA readmission rates is important. We think it makes sense. It provides a more complete view of the care the patients receive overall.

In terms of policy implications, we think that an assessment of readmission rates should include as much data as possible about patients’ risk and readmission outcomes. A readmission to a different hospital or a different hospital system or paid for by a different payer is essentially the same as a readmission to the original healthcare system.

So that is kind of on a macro scale, and on a smaller scale we think that hospital quality initiatives should also be based on information about all the admissions and all patient care to the extent possible with outside providers.

Here are some avenues of future research that we are thinking about. Are there ways to improve model performance? So our model performance did not improve from this exercise and we are wondering if additional data sources might improve model discrimination. It is pretty clear that adding social support and socioeconomic status data would improve predictability, but that comes at a cost. And are hospital characteristics associated with Medicare dual use? So it might be interesting to look at why VA patients are readmitted to outside hospitals. Is it patient preference? Do they have a choice? Does urban versus rural location of the index hospital matter? And does it matter if the VA hospital is close to other hospitals? And it might also depend on the emergent nature of the medical condition, for example. When a patient has a heart attack, for example, he would probably go to the closest hospital, not the closest VA hospital. So there are a lot of interesting things to look at with VA readmissions.

Thank you very much for attending this talk. This study was funded by VA HSR&D and Amy Rosen was the PI on this study. Thank you.

Todd Wagner:And thanks, Bill. As you might expect, there has been a number of questions that came in because it is a hot topic, as you point out.

William J. O'Brien:Right.

Todd Wagner:One of the questions that did come in has to do with the timing of the data. And as you rightly point out, in the ideal world, you would use all information whether it is Medicare or Medicaid to sort of build these models and identify readmission. Now Medicare has a two-year lag. There is often an even greater lag for Medicaid. There are data quality problems that are different from Medicare, Medicaid and VA. How do you reconcile the three? Is it important to do real-time estimates obviously if you are doing payments? How do we do this, I guess.

William J. O'Brien:That is a good question, yes. So as VA researchers, we have fairly easy access to Medicare data through VIReC. It is a pretty painless process. But yes, there is a lag to it. We started this study oh, three years ago. The larger grant we started three years ago, not this particular paper. And we were originally doing 2007 through 2009 and then as time went on, we decided to update it to 2008 through ’10 because 2010 had just become available in 2012. So as far as getting real-time data, oh boy, that is a tough one. I am not sure [laughter] what to say about that other than ...

Todd Wagner:So, so ...

William J. O'Brien:... doing the best you can. So how do you [overlapping voice].

Todd Wagner:But let – I guess I sort of push you a little bit on that. So let us say, since we have the luxury of research of time often, but if you are in the group that is creating VA Hospital Compare ...

William J. O'Brien:Mmm.

Todd Wagner:... and that might be Steve Finn’s group if I am not mistaken. But in any case, where you are actually having to generate these relatively real-time ...

William J. O'Brien:Yeah.

Todd Wagner:... it has been misclassification predictable. Or do you just present the data and say, we know we are off, but we are doing the best we can.

William J. O'Brien:Yeah. Well, they know they might be off a little bit in terms of missing some readmissions. I would not really suggest there is a mad rush for IPAC [PH] to go out and somehow get Medicare data quicker. I think that the results of this study show that that is not terribly important. So yeah. So Medicare data, the time lag is an issue and I am not sure how much we can do about that. The Medicaid lag is even worse. There is something like a three-year lag. So it would be wonderful to have this data available in real-time or with a one- or two-month lag. I am honestly not sure how realistic it is right now.

Todd Wagner:Okay. We had a question about slide 18, which says: Are the chronic condition categories described in slide 18, there you go, a standard way to classify reasons for readmission, and do you have any particular reasons for readmission that might be addressed to reduce readmission rates.

William J. O'Brien:Okay. So I just add a little clarification. So these are not actually the reasons for readmission. These are risk factors for readmission used in prediction models that were gathered from secondary diagnoses during the index admission as well as the one-year preadmission period. So these are not reasons for readmission, they are just predictors of readmission. And these were validated by CMS when they developed the readmission measures in 2008.

And the second part of the question, could you remind me what the later .... ?

Todd Wagner:Sure. It says: Has the presenter identified—yeah. Has the presenter identified any particular reasons for readmission that might ...

William J. O'Brien:Right.

Todd Wagner:... that might be addressed.

William J. O'Brien:Great. So as part of the study, we did not really look at the reasons for readmission, but that is a really good question that I think is relevant to this project. So I will go back a little bit more and show one of these pictures here. So in cases where the patient had a VA index admission and had a readmission to a VA hospital that we can identify with VA-only data, there is another scenario I kind of left out. What if the patient had a first readmission to a Medicare reimbursed hospital? Well, that is another way that Medicare data could be useful here. So it is probably important to look at the reasons for readmission, and that has been done by other people in our group, Qi Chen, for surgical readmissions. And if you were able to have as much data as possible about what happens in the post-discharge period, and to have the Medicare data and Medicaid data or anything else you can get, you could more reliably pin down more accurately the patients are readmitted, which is important. 

Todd Wagner:Okay. There is another question that came in, I will just read it, and then we can try to parse it. I am confused. The predicted rate is the output of a predictive model with fixed and random effects.

William J. O'Brien:Right.

Todd Wagner:The expected readmission rate is the result of a predictive model using only fixed effects. So the P/E ...

William J. O'Brien:Yeah.

Todd Wagner:... ratio is the ratio of predicted outcomes using two models.

William J. O'Brien:Oh, okay.

Todd Wagner:Where do the actual readmissions come in?

William J. O'Brien:So I will clarify that a little bit. The Ps and the Es come from the exact same model specification, but when we are fitting the predicted value with the expected probability of readmission, we are not taking into account the hospital-specific intercepts. So it is actually the same model. It is not two different models. We are not running one logistic model then one HGLM, for example, if that is what the question represented.

And what was the second part of the question, Todd?

Todd Wagner:If the P/E ratio – yeah. So the P/E ratio is the ratio of predicted outcomes using two models, where do the actual readmissions come in? So in some sense the predicted based on [overlapping voice].

William J. O'Brien:The actual, oh, well [overlapping voice].

Todd Wagner:Great. Go ahead.

William J. O'Brien:Yeah, so where do the actual readmissions come in? That is on the left-hand side of the equation. So the HGLMs estimate readmission outcome, which is either yes or no. That is on the left-hand side of the model. And on the right-hand side are the patient demographic and clinical characteristics that are thought to be predictive of risk of readmission. So in the VA-only analysis, we are looking at only readmissions to the hospitals. And in the VA-Medicare analysis, we are looking at either readmission to a VA medical center or to a Medicare-reimbursed hospital. So the zeros and ones on the left-hand side of the equation will be different in between analyses depending on whether Medicare data is used or not.

Todd Wagner:And I guess another way of saying it is just because we see the actual readmissions, but what we often do not see is the clinical differences that are important between the patients getting readmitted and we want to statistically control for them. So we create this model or you create this model that is based on actual readmission. But we are trying to statistically adjust for all their clinical differences that we observe. And so in some sense these predictions are based on actuals is what we are using, which is quite standard in the literature. Is that ...

William J. O'Brien:Yes.

Todd Wagner:... hope that would answer the ...

William J. O'Brien:Right.

Todd Wagner:Because we do not want to just use the actuals because the clinical difference is some people are very sick, other people are less sick. So.

William J. O'Brien:Exactly.

Todd Wagner:There are a number of other questions. Right. I cannot—and this has to do with the readmissions and relatedness question that we addressed before. I cannot understand how an individual with a history of active heart failure readmitted to the hospital within 30 days for something like a broken leg or diabetes complications and not heart failure gets included as a heart failure readmission. So there is a policy answer to this, Bill, that I am hoping you can address as well as sort of maybe more than a clinical/statistical question.

William J. O'Brien:That is a classic case that we would – you might get a truckload of patients leave the hospital, then get hit by a car. Do you really attribute a readmission outcome to the discharging hospital? And that is a tough one. In the CMS models as currently implemented, that patient would count as a readmission and within the hospital that discharged him. I would not really argue that that makes all that much sense. However, we are doing this on kind of a wide scale, so hopefully there will not be too many of those.

An alternative to doing the CMS readmission measures—and again, I did not invent any of these models; these are CMS and Harlan Krumholtz’ groups validated and developed these and they are endorsed by NQF, so they were not developed by our group. But there is an alternative to looking at 30-day all-cause readmission. 

One alternative might be 3-Ms potentially preventable readmission software, and that is pretty interesting. We are actually validating that right now as part of our grant. It actually looks at the DRG in both the index admission and the readmission. And the main thing it does is that it excludes readmissions that are not plausibly related to the index admission. So if the patient is discharged with chronic – with congestive heart failure and then suffers a broken leg, the software would flag that and the initial admission would therefore not be at risk of a readmission. And that is one thing we are hoping might be an improvement to the all-cause readmissions that CMS is currently using.

Todd Wagner:So I actually have had some conversations with the CMS about this issue. The one, and I appreciate your response, the one thing that dominates the discussion is that having a very simple policy that is easy to observe and easy to understand and to avoid the gaming. So you brought up the gaming issue before, Bill, which is that is: is the hospital is able to recode this in a way that it makes it look unrelated and makes them look favorable so they do not get dinged. And so they have to keep in mind that there are some very bizarre things. Let us say the patient leaves the hospital after having had a heart attack. They get into their car. They get into a car accident but it was because they were dizzy and they were discharged too soon. You could think of all different ways that it would be coded that it would not be related when it fact it clinically was. And so there is this tremendously gray area and the belief is that we have a confidence interval that would just pick that up.

William J. O'Brien:Exactly. That is the hope. So like I mentioned before, we are doing chart review for a lot of these readmitted cases and hopefully one of the clinicians on the study will write about some of these anecdotes that we see. Basically every case is different. Every reason for readmission is different, and a lot of these readmitting cases are just very, very messy. And it is really not possible to account for every – the crazy scenario that can happen after a patient is discharged from a hospital. So yeah, you are right. Hopefully the confidence intervals will take care of that to some extent.

Todd Wagner:Okay. So with that, three questions have popped up as clarifying questions. So let me just take these, then, in order. Can you provide a reference on slide 18? You mentioned those CC categories. Do you have a reference for that? You mentioned that they were Medicare categories.

William J. O'Brien:Yes. These are ...

Todd Wagner:Exactly. The CCs.

William J. O'Brien:Okay. So a little background on these. CMS has – actually not CMS. I believe it is Verisk has hierarchical condition categories and those are proprietary. CMS uses a simplified version of the HCCs without the hierarchy and those are referred to as CCs. These are publicly available and they are also use in CMS’ PACE model, which I believe is used for risk adjustment for Medicare Advantage, I think, Todd, if you could correct me if I am wrong.

Todd Wagner:That is correct. I actually did the Cyber seminar last month on these. I just call them the HBCs. But you are right. They are CCs. Yes.

William J. O'Brien:CCs. Right. Yeah, so these are just rolled up from ICD9 CM diagnosis codes and there is a simple mapping that is available from CMS and it gives me the mapping from many ICD-9 codes to one CC code. And I believe that it is probably on the readmission measure methodology documents at qualitymet.org and I believe it is also probably available at cms.gov.

Todd Wagner:We can also provide the reference because we have been using it heavily here. So.

William J. O'Brien:Yeah.

Todd Wagner:So that is fine, too. So if I can reach back out to that person. Another question has come up and it is related. In the terms of future research where you talk about future research, you mention to improve model performance. Is the goal to more precisely risk adjust admissions? Is that the goal when you say performance?

William J. O'Brien:Model performance. So can the models discriminate between a randomly selected readmitted case and the randomly selected non-readmitted case? In other words, would the model assign a higher likelihood of probability to the case that was actually readmitted higher than the non-readmitted case? So is having a better discrimination ability an extremely important feature of these risk adjustment models? 

I think CMS would probably argue no. They acknowledge that the discrimination ability as it is now is modest to poor and there is very likely a good reason for that. It is probably because the models omit race and socioeconomic status, and I have seen some other papers posted here just recently that show that when you include SES and race and income that the C-statistic does improve significantly. But there are conceptual issues with that. You do not want to imply a lower standard of care for safety in that hospital just because they serve low-income hospitals. And you can argue either way on that. I do not think there is any clear agreement.

But is it an essential feature for this type of model to have high discrimination? Maybe. I think that the more important thing—it is kind of like looking at the R-squared of an OLS model and judging it just by that. I think you have to take into account whether or not the model controls for what is important and that is the disease burden of patients at a given hospital. And I think that is more what CMS is going for. So I do not think getting a better C-statistic is the highest priority, but it probably would be a good thing regardless just to have a model that can discriminate between high and low risk.

Todd Wagner:Great. And you mentioned the 3M software, the readmission software. I am somewhat interested in the specifics on that. Do you have the actual name of that software and my guess is it is like [overlapping voice] ...

William J. O'Brien:Yeah, sure ...

Todd Wagner:... is that right that you contacted?

William J. O'Brien:Well, the contact person would be Norbert Goldfield and he is at 3M Health Information Systems and the name of the software is PPR, which stands for Potentially Preventable Readmissions. And it is a proprietary software package and Norbert was kind enough to provide us with a research license in order to validate that software against VA data for potential adoption – recommendation for adoption in the VA as an improvement to what we currently have.

Todd Wagner:And I will say that we had originally last month when we presented on the risk adjustment models, we had contacted 3M because they have another software package for that, and they have been very open to letting VA researchers use their software. So. I expect that there is a benefit to them. They are a small player right now in the field and there is a benefit for getting people to use it. But nonetheless, they have been very open.

William J. O'Brien:Yeah. I think it is a very good concept. Like I discussed, the main benefit that I see currently as we are using it, is that it rules out a lot of index admissions that really should not be indexed admissions. And so they have kind of – Norbert would say that it is going for the low-hanging fruit.

Todd Wagner:Fair enough.

William J. O'Brien:Okay.

Todd Wagner:Well, this is fascinating work. That is all the questions we have and we are actually quickly approaching the top of the hour. So let us just hold on for one more minute and perhaps, Heidi, you can put up the – or they just get it when they exist. That is correct.

Moderator:Yes, it actually pops up when they complete the session.

Todd Wagner:That is right. I forget that all the time. I just wanted to publicly thank you, Bill, for a great presentation on readmissions. It is definitely one of the hottest topics right now in health policy, so this is fantastic.

William J. O'Brien:Great, thank you, Todd.

Todd Wagner:And I will just hold on for one second and we will see if any more questions come in. I have not seen any, and then I will get back to the individual who asked about the risk adjustment software.

William J. O'Brien:Okay.

Todd Wagner:That looks like it.

Moderator:It looks like that is it. Bill, I [overlapping voice].

Todd Wagner:I will let you know if other questions come in.

Moderator:And so I would [overlapping voice].

William J. O'Brien:Okay, thank you so much, Todd.

Moderator:I want to thank you for presenting today and for our audience, as Todd was mentioning, when you leave the session today, you will be prompted with a feedback form. If you could take just a few moments to fill that out, we would definitely appreciate it. We will not be holding a Cyber seminar in August. Our next sessions will be in September and we will be sending registration information on that out to everyone as we get a little bit closer to that date. Thank you, everyone, for joining us for today’s HSR&D Cyber seminar, and we hope to see you at a future session. Thank you.

Page 1 of 15

Page 16 of 16