**Transcript of Cyberseminar**

**HERC Health Economics Seminar**

**Hospital Profiling with Enhanced Risk Adjustment Based on Laboratory Tests and Vital Signs Data**

**Presenters: Ann Borzecki, MD MPH, and Amresh D. Hanchate, PhD**

**February 19, 2014**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact herc@va.gov.*

*Paul Barnett:*My name is Paul Barnett.  I’m happy to welcome folks to the HERC Health Economics Seminar, our monthly session.  I’m very excited about today’s presentation.  I think that we’re in a very interesting new world, where have really an exponential expansion in the amount of data that are available to us because of electronic health records.  

The seminar that we’re going to hear today is really on the cutting edge of how we can use that to do more important stuff in health services research.  

I’d like to introduce our speakers.  Amresh Hanchate, an economist at VA Boston and an assistant professor at the Boston University School of Medicine and does research on improvement and risk adjustment for hospital profiling, which we’re talking about today, but also looks at impactive health reform on VA and geographic differences in care for stroke and effect of geographic differences on cost in VA.

Ann Borzecki is a research associate professor at BU Public Health and Medicine and general internist trained in Canada.  She has worked as a clinician researcher in VA for more than 10 years and is attending physician in the primary care clinic at Bedford VA and on the general medical board in West Roxbury VA.  She’s a research health scientist at the Center for Health Organization and Implementation Research, CHOIR, in the Boston area and divides her time between Bedford and Boston sites.  Her research focuses on using administrative data to measure quality of care.  Take it away, please. Very interested in hearing what you have to say.

*Ann Borzecki, MD, MPH:*Thank you and thank you all for joining us.  Today we’re going to present some results from a project that’s noted on the title slide.  The main goal of this project was to develop enhanced risk adjusted models for mortality prediction by adding labs, vital signs, pharmacy and Medicare data to VA administrative data for nine clinical cohorts.

The current presentation will be limited to the enhanced models using labs and vital signs for three of the cohorts, AMI, heart failure and pneumonia.  Before we get into the meat of the project, I just wanted to acknowledge our funding source, the VA HSR&D program and our co-investigators and study team members.

We’ll review results of two specific project aims in today’s talk.  For the first aim, we examined the impact of adding labs and vital signs to administrative data for hospital profiling.  In the second aim, we evaluated whether this addition led to improved measures of hospital quality.

In terms of background, a number of studies using different populations and data sources have examined the impact of adding labs and vital signs to administrative data in models predicting mortality.  These include three non-VA studies and two VA studies.  For the VA studies, Render et al used IPEC data, which was derived from the VISTAs at local sites, first for studying mortality models in an ICU population and more recently, for studying AMI heart failure and pneumonia discharges.

These studies all found sizeable improvement in model performance, as defined by the model discrimination C statistics.  The more recent Render study also found discordance in hospitals identified as above or below average.  However, we know of no studies that have examined whether adding clinical data leads to better measures of hospital quality.

In terms of our methods, to economize on time we’ll focus on the details related to adding the clinical data.  We used standard VA databases.  Namely, we used the inpatient treatment files and the outpatient encounter files from the VA national patient care database.  For lab results, we used data from the DSS LAR files.  For vital signs, we used data from the CDW vital signs files and we used VA vital status files to establish 30 day mortality.

I just wanted to mention that much of our methodology related to administrative data and hospital profiling comes from the CMS VA hospital compare program.  We also followed the hospital compare protocol for identifying discharges for AMI, heart failure, and pneumonia and applied the same exclusion criteria, except for two main differences.  We included all patients aged 18 and older and we only kept hospitals with at least 50 discharges.

This next slide shows the total counts of hospitals and discharges for our three cohorts over the four year period, from fiscal year ’07 through 2010.  It also shows the median number of discharges per hospital and the median observed mortality rate per hospital.  As you can see, there’s a fairly wide range in the observed 30 day mortality rate across the three admission positions.  

To identify our candidate lab tests and vital signs for model inclusion, we first identified all relevant tests in the data sources.  We excluded tests which weren’t recorded for a majority of the study population and this left us with the final list of 16 lab tests and vital signs shown on this slide.

For each item, we then identified all lab tests or vital signs performed within 24 hours of the index admission times.  This included either the 24 hour period before or the 24 hour period after admissions, as well as tests performed in either a VA inpatient or outpatient setting.

If a patient had multiple recordings of the same test or vital signs within a 24 hour period, we selected the most extreme results.  We then defined cut offs for each test, based on a combination of clinical judgment, as well as results of bivariate analyses examining the association between deciles of each test value with the outcome of 30 day mortality.

Based on these cutoffs and the odds ratios obtained from the bivariate analyses, we then grouped each test into the five categories shown.  We considered missing test results separately because we found that for many tests, this was significantly associated with higher or lower mortality.  This suggests that missing it was not random and could result from high or low patient severity.

We then estimated preliminary multivariate risk adjustment models and if none of the categories of test measure were correlated with 30 day mortality, that test measure was excluded.  The final model for each admission cohort included a subset of the 16 lab tests and six vital signs.

This next table shows an example of a test measure and how we categorized it for use in our multivariate models.  We demonstrate this with how serum creatinine was defined for the AMI cohort.  As I mentioned, the categorization is based on the odds ratios we obtained from the bivariate analyses.  We used the same process for vital signs measures.

With respect to missingness, we define this as the proportion of patients for whom a particular test was not recorded within 24 hours of the admission time.  When we looked at lab tests and vital signs data from 2005 onward, we found that for the first one or two years, missingness was actually quite high for some measures, but by 2007, this had stabilized for most of the measures.  For the majority of the risk measures in the final models for each condition, missingness rates were less than 5 percent.

We did, however, find variation in missingness rates across conditions and across hospitals.  For example, for the heart failure model, we found the highest missingness rate for bilirubin at 29 percent.

We next estimated two models for each admission cohort.  The first used only administrative database variables, which was identical to the CMS VA hospital compare models.  The second was an enhanced model, which included variables based on lab tests and vital signs.

For each model, we estimated a hierarchical logistic regression model with unobserved hospital effects.  We then used the results from these regressions to obtain hospital level risk adjusted mortality rates, using the same metric as hospital compared.

For each condition cohort, the enhanced model you can see performed slightly better, in terms of the C statistic.  We also found this was consistent using other measures, including R-squares and calibrations.  For example, for heart failure, the C statistic went from .473 to .81 when we added labs and vital signs.

The next slide compares our findings to those of the previous VA study by Render and all from 2010.  As you can see, our findings are quite comparable, even though she’d used a different data source for her labs and didn’t include vital signs.  She actually used data that was automatically extracted from local vistas and not DSS.  And another difference with her study was that she imputed missing values as normal.

At this point, I’ll hand things over to Dr. Hanchate.

*Amresh D. Hanchate, PhD:*Thanks, Ann.  Hello.  I’ll continue, having seen that adding clinical data—by that I mean the lab tests and vital signs leads to model—the risk adjustment model improvement.  

Then we looked at two follow up questions.  One is how much does this affect the hospital profiles, measured in terms of the adjusted mortality, what I’m going to say RSMRs, both in absolute terms and in relative terms?  Secondly, after all this, do we hopefully have a better measure of hospital quality, since ultimately I guess that’s our aim?

First the kind of changes we see at the hospital level, first the absolute changes in RSMRs.  The first column is for AMI.  We had 91 hospitals and we see that for 75 percent of them, the change in RSMR, both either an increase or a decrease of up to 10 percent.  Seventy-five percent of the hospitals had about 10 or less than 10 points change, meaning that a quarter of the hospitals saw a change of between 10 and 20 percent, most of them.

Across the other two columns too, the 128 hospitals in the heart failure cohort and the 131 in the pneumonia cohort, we kind of saw a similar pattern with about a quarter of the hospitals seeing at least a 10 point change.

We can also look at this relatively.  Here we are looking at the heart failure numbers.  We saw similar patterns across the other cohorts.  We grouped the RSMRs into quartiles.  We have the rows where the quartiles are based on the—without clinical data, meaning the base data, just the administrative data and the columns are based on the quartiles, with the enhanced model, with the clinical data.

The first cell says that 24—there were 24 hospitals that were concordant using both the models.  They were categorized as good hospitals with the lowest risk adjusted mortality.  Along the same role, if you look at the next number there were eight hospitals that, using the administrative data, have been classified as good hospitals or the lowest mortality hospitals, but got bumped up in terms of mortality using the enhanced model.

If you just focus on the off diagonal, the discordant cells, adding all those up gives us, for the heart failure, about 42 percent of the hospitals ended up changing their relative quartile position after the model enhancement.  At the bottom you see the similar corresponding numbers for AMI.  Forty-eight percent of the hospitals switched positions.  For pneumonia it was 42 percent.

A third metric, for those familiar with the hospital compare reporting, is based on not just the RSMR itself, but on the confidence interval, the 95 percent confidence interval of the RSMR.  The hospitals get grouped into either average, meaning no different than the VA national rate.  The table here now is for the pneumonia cohort and the national observe rate, observe 30 day mortality rate, was 10 percent.

A hospital whose RSMR and the confidence interval was below 10 percent would be categorized as better than national rate.  When the confidence interval was about at 10 percent, it would be worse than the national rate.

As is commonly known, this grouping tends to be rather conservative, where most of the hospitals tend to be in the middle, as indicating average performance.  Here we see that after enhancement, again, if we have the off-diagnosed terms, there are 14 out of the 131 which switch positions.  The corresponding number for AMI was five out of 91, for heart failure about 16 out of 128.  

The more interesting bit was the direction of change.  We saw that in all the three cases, all the three cohorts, the enhancement led to a larger proportion of the hospitals being grouped as average.  We can see here in this table the baseline model, the administrative model, classified 110 as no different than average, where if we look at the middle column, I guess that’s 122 hospitals being categorized as average.

This is a different view of the same information.  Again, the pneumonia cohort now represented graphically.  Each of the two color bars represent one hospital.  The blue bar is the confidence interval for the base model RSMR and the red one for the corresponding enhanced model RSMR.  We have 14 of the hospitals that switched places, hospital compare designations, I should say, in the pneumonia story.

The flat line you see at around 10 percent is the national VA rate.  This is just to give you an idea of the nature of changes that happened to bump the hospitals into a different category.  You could look at the very first one on the left and say the hospital was very close to 10 percent.  And so a modest change in RSMR could change its designation.  It looks like that happened, to some extent.

If you look at the second one, it goes quite far from the average.  It required quite a bit of movement to make it go from, in this case worse than average to an average status.  If you look across the 14, roughly, by my count, about half the hospitals saw a kind of sizeable change for the change in status.  As for the remaining, it was one of those modest borderline changes.

Just to kind of summarize this whole bit of changes story, what it says is for some hospitals there was a sizeable change in RSMRs, both absolutely and in relative terms, from the  model enhancement.  

The next question is does adding the clinical data, does it lead to a better measure of quality?  As we all know, there’s voluminous literature pointing to the limitations of relying on administrative data.  One of the often cited limitations is the lack of clinical detail, particularly the severity of the patient or the status of the patient at the admission time, because after all, we are looking at that as the outcome.

The second segment of the literature also indicates that we have these multiple measures of hospital quality, besides mortality we look at process measures.  We also look at hospital 30 day readmissions.  Again, there is a growing literature indicating that the concordance between these measures is rather poor.  For example, the correlation between mortality and process measures and between mortality and readmission measures.

Our next step was to check if adding clinical data gives us an improved measure of quality.  Our approach was to see if the concordance between this improved mortality measure and the other quality measures actually increased over time.

In this, we are following a recent study in health affairs by Press and colleagues, which examined the concordance between the CMS, not the VA, hospital compare readmission measures with a host of other quality indicators.

Again, we look at a series of comparisons here, concordances.  The first is mortality with process measures.  We used the ORYX VA composite course that reported for the same three admission cohorts, AMI, heart failure and pneumonia.  The first cell here, minus .01, which is the kappa statistic, is between the process course, the ORYX course for AMI, turned into quartiles with the mortality rate, risk-adjusted mortality rate using the base model, again, grouped into quartiles.  In the cell below that, again, minus .01, is for the corresponding concordance and now the RSMR’s coming from the enhanced model.

As you can see in both cases, not only is there no change in concordance, but in fact there appears to be no concordance.  If you look at the next two columns for heart failure and pneumonia, again, we see a similar pattern.  The changes are by no means significant and the kappa statistic itself is not significantly different from zero.

I should mention that here we are looking at concordance by quartiles.  We have done this in terms of the simple correlation of the measures themselves, as well as rank or relations.  We did not see any significant difference in the nature of this relationship.

The next concordance we look at is between mortality and a 30 day readmission rate.  Here we obtained the VA hospital compare 30 day readmission rate, again, for the corresponding cohorts, AMI, heart failure and pneumonia.  Again, the layout is the same as before.  For AMI, we see that kappa statistic is almost near zero and does not change after enhancement.  Virtually the same is true for heart failure.  The change for pneumonia is not statistically significant.

The third concordant is based on the premise that hospital quality, to the extent that’s what we are measuring here with mortality rate is unlikely to be changing very quickly.  Based on that assumption, we have four years of data.  We split that into two parts of two years each and we calculated the RSMRs using the base and enhanced model separately.  And then we looked at the—to the extent that the stability and the measure of quality does the enhanced model give us a more stable measure of quality?

So the first number here, .09, is for the AMI cohort.  Again, here we are looking at 30 day mortality in both contexts.  We are just comparing the RSMR for a hospital based on its 2007, 2008 data versus its 2009, 2010 data.  If you had used the base model, then the concordance between the two periods is .09.  If we had used the enhanced model, it’s .03.  Neither is a significant amount and the change itself is not significant.

The heart failure looks promising.  It’s a change from .06 to .16.  The .16 is statistically significant, testing for whether it’s zero or not.  It’s different from zero, but the two are not statistically distinguishable.  The difference is not significant.  For pneumonia, it is, again, the same case.

Just to not belabor the point, we looked at several other concordances.  Just to list them quickly, the first was to treat the three cohorts that mortality for heart failure, for pneumonia and for AMI as indicators of overall hospital quality, we looked at concordance between pairs of this.  We looked at 30 day mortality for AMI and for pneumonia and checked if model enhancement improved the concordance between pairs and we did not see any significant improvements.

Then we looked at the vascular surgical outcome measures that are available at the facility level with our mortality measures, again, with no significant difference in outcomes, and volume.  Often people use facility volume or hospital volume as a sort of proxy indicator for performance with higher volume indicating potentially better quality.  Again, we found no such improvement in correlation from the enhanced models.

Finally, going back to what I presented earlier on, we saw that adding lab and vital signs caused the hospital RSMRs to change and change substantially for some hospitals.  You know, one possible inference from that is, well, we are adding new risk measures based on the patient’s status and admissions, based on the labs and vital signs.  We have more information and that additional information, you know, can be expected to change the profiles of the patients in each hospital and therefore change the RSMR.

What we are doing in some sense is comparing hospital profiles in two settings, using first the administrative model and then the enhanced model.  Whenever we make these *[inaudible 27:41]* comparisons of the same measure, in this case the hospital quality, as you all know we need to be worried about something called regression to the mean.

To the extent that these changes may well reflect changes in patient status by adding information, we kind of cursedly performed this test of regression to the mean.  You know, just to emphasize, a regression to the mean is when some hospitals, you know, because these two measures, the RSMR measures, both the base and the enhanced, are statistical estimates and therefore, have some randomness included in them.  

Regression to the mean is when hospitals get assigned to higher or lower RSMRs purely out of randomness and then when we re-measure the quality using another RSMR metric, in our case the enhanced model, hospitals with previously high RSMRs experience a reduction in the risk adjusted mortality and hospitals with previously low RSMR experience an increase in RSMR.  That’s kind of the statistical phenomenon, where it’s just the randomness induces pattern and changes.

What we’re seeing here in the graph is—well, first the axes.  The horizontal axis measures the RSMR from the base model, what it was, using the administrative data.  The vertical axis is the change in RSMR, percentage in RSMR after enhancement.

What we see is that, if you look at the note there on the first graph, our square is 0.51.  That is 51 percent of the variation in the RSMR change is encountered simply by what RSMR was in the base mark.  Looking at pneumonia, 26 percent of the variation is explained simply by what its RSMR was in the previous administrative model.

Strongly suggesting that there is some of this confounding, basically, that’s going on, again, suggesting that we kind of use caution in how we interpret the changes that we see at the hospital level in RSMRs.

Just to summarize, first we found that VA has fairly complete data on lab tests and vital signs, although missingness rates varied quite a bit by hospitals.  Second, adding risk measures from lab tests and vital signs improved model performance and caused sizeable change in hospital performance metrics.  Finally, correlation of enhanced model mortality performance measures with other hospital quality indicators remains poor.

We recognize several limitations in what we’ve done.  Firstly, in all our comparisons we don’t have a gold standard measure of hospital quality to compare with.  Secondly, the different measures of hospital quality we looked at may each be capturing or representing different dimensions of quality, some of which may be not overlapping.

Also, there are limitations in some of the comparison measures we use.  For example, we all know that the audit measure, the performance measures in the VA setting almost give us no radiation.  Virtually all hospitals are in the 90-plus range for these measures.

Also, here we are using a hospital as the unit of observation.  Being the VA, this limits our aim and likely our power to distinguish what may be of some significant changes.  Nevertheless, absence of concordance with a wide range of quality indicators still remains an issue that probably will need to be resolved.  Also, we need to take any kind of changes we talk about, take the confounding from the regression to the mean also to work on.

In terms of implications, although the case for adding clinical data for risk adjustment is reasonable and, well, justified.  We need for the research on the ability of the enhanced models to distinguish what we may consider as true hospital quality, meaning that for a veteran who looks at a number of these metrics and may be conflicted, say, good mortality, performance but poor readmissions performance for a certain hospital, unfortunately using the enhanced model like the kind we have does not seem to provide much of a resolution to this conflict.

In conclusion, use of data on labs and tests—lab tests and vital signs enables to better account for differences in patient status, in measuring mortality differences.  Unfortunately, in spite of the imprudent models, concordance with other hospital quality measures remains rather poor.  Thank you.

*Paul Barnett:*We have an opportunity for—people have questions.  We have one from Todd.  I don’t know, Ann and Amresh, if you can see the question and answer box.

*Ann Borzecki, MD, MPH:*I can see a box.  I don’t see any question in it.

*Paul Barnett:*I think if you click on it, it will say Todd Wagner’s listing.  It says, “This raises interesting questions about”— so is everyone seeing that now, Heidi?

Moderator:What Amresh and Ann would need to do, on Q&A if you just go straight over to the right from where it says Q&A there are two buttons, presenter view and participant view.  You’re probably on participant view.  If you click the presenter view button, you’ll be able to see the questions that are in that pane.

*Paul Barnett:*Todd’s question is, “What might be the best information today might not be the best information tomorrow.  Should sites like Hospital Compare include a 95 percent confidence interval?  How do we teach managers that improvements can be made and can change ranking?”

*Amresh D. Hanchate, PhD:*Yeah, I mean, certainly, especially in the VA setting, we have quite a few hospitals that are relatively small, in terms of volumes of cases and some sense of the noise and the measure should be included, meaning the margin of error, so to speak.  I guess that was the question? 

*Paul Barnett:*Yes, well, so we should inform people about the uncertainty and how do we teach managers that improvements can be made and can change ranking?

*Ann Borzecki, MD, MPH:*That gets to the issue of how actionable is mortality as a quality measure?  I suppose there’s some debate about that.  You know, we try to improve the processes associated with it, but if there’s poor concordance, I’m not sure how to get their buy-in, other than intuitively or has face validity to do that.

*Paul Barnett:*Well, there are a couple of other questions, but I just want sort of a follow-up on that.  You compared quartiles, so this particular measure with other measures, it consistently assigned hospitals to the same quartile.  I was just wondering, don’t what we probably care most about is that low group?  In fact, in Hospital Compare it’s the ones that they say are significantly worse.  There were relatively few hospitals that fell in that lowest group.  I’m just wondering—and in fact that’s, in a certain sense, all that you can identify from this risk adjusted model.

I’m just wondering, are those poorer performers consistently identified in the measures?  I mean, it seems like the manager would care the most about, identifying the poor performers and are they poor by every metric?

*Amresh D. Hanchate, PhD:*Right.  I mean, I guess in some sense that probably should be the message, if you were to ask me, given that there is so much difference across these metrics.  You know, if you can find hospitals, even a single hospital which would kind of get flagged on different measures, you’d probably then have some real information to address.

*Paul Barnett:*I don’t care too much if a hospital is one standard deviation above the mean or one standard deviation below the mean as much as I do knowing the hospital’s consistently two standard deviations below the mean.

*Amresh D. Hanchate, PhD:*Right and consistently over time and consistently across some other measures—

*Paul Barnett:*Yes.

*Amresh D. Hanchate, PhD:*--that we believe are, you know, also capturing quality.

*Paul Barnett:*I wonder whether you’ve been too hard on yourself in saying it.  You know, it’s not consistent with other measures, if you had a lower threshold of what you’re looking for.

*Amresh D. Hanchate, PhD:*Right.  I mean, our next—it’s kind of from a policy perspective, I guess.  We don’t want to kind of flag hospitals that are categorized, as you put it, as below average if it’s sort of by chance.  At least consistency over time would probably be another sort of—a less demanding requirement.

*Paul Barnett:*Well, we do have some other questions.  Mae Hu asks were all the lab tests and all the vital signs significant in the enhanced model or were some insignificant and then dropped?

*Ann Borzecki, MD, MPH:*The latter was the case.  Not all were significant and some did get dropped.

*Paul Barnett:*Did that depend on the clinical condition?

*Ann Borzecki, MD, MPH:*Yes.  There were some cases where we made a clinical decision to keep a variable in a lab test in the model, even though it wasn’t significant.  I think there were a few models where hematocrit or hemoglobin didn’t end up being significant, I think maybe for heart failure, but we chose to keep it because clinically it made sense.

*Paul Barnett:*There is another question from someone who signed in as Guest.  They say, “The lab and vitals measured at a single time point of 24 hours may just be a reflection of disease severity or patient acuity, so I’m not surprised it better predicts 30 day mortality risk.  But this may not affect hospital quality.  Maybe a better clinical measure would be change, normalization of vital signs in the 24 hours with treatment.”

*Ann Borzecki, MD, MPH:*Well, do you want to answer that?  I mean, that’s the whole issue of separating out patient severity from quality of care.  I mean, we made the decision to use the first abnormal vitals because we wanted to incorporate patient severity and not mask that.

*Amresh D. Hanchate, PhD:*Right.  That’s right, yeah.  Yeah, I mean, we wanted to just capture patient status, if you will, at the point of admission.  I guess, you know, if one were interested in the interventions then you could get into the process measure, so to speak, how they differed in normalizing any kind of acuity.

*Paul Barnett:*Well, I wonder if we have any other questions.  People can type them in their question and answer box, which I believe is on the lower right hand corner of your screen, at least if your screen’s like mine.

What do you plan to do next to follow up on this?

*Amresh D. Hanchate, PhD:*A kind of other interesting issue out there is the readmissions measures, to see if they benefit any more than the mortality measures in this kind of enhancement.  At least for readmissions, we can actually do a side-by-side concordance to compare the enhanced mortality versus the enhanced readmissions model.  

That’s what we have planned for the near future.  I guess at different levels, this raises a number of questions depending on who you are.  If there are people interested in just the data part of it, the lab and vital signs, it seems quite promising for other model purposes or coding model developments or whatever the reason is.  It looks like the—just from looking at the stability of data it seems to be much more, much less random and much more complete.

*Paul Barnett:*Readmissions as a quality measure?

*Amresh D. Hanchate, PhD:*No, I just meant use of the labs and the vital signs data in the VA, from the lab science from the DSS files and the vital signs from the CDW files, for whatever purpose, seem to be—

*Paul Barnett:*The 30 day mortality, is that kind of a rare event?

*Amresh D. Hanchate, PhD:*Well, it’s not that rare.

*Paul Barnett:*Less than 5 percent though, maybe, huh?

*Amresh D. Hanchate, PhD:*Less than 10 percent.

*Ann Borzecki, MD, MPH:*Yeah, 10 percent for AMI, 10 percent for pneumonia.  I mean, it varied.  For AMI, for example, by site it varied from 3 to 26 percent.  These are observed rates, so it’s not that rare.

*Paul Barnett:*Yeah, interesting.  I was just wondering, if it’s too rare, then it might be—that would explain why it’s unstable.  I guess the other issue is whether the hospital—anything the hospital does really affects it or anything the hospital does differently from other hospitals, I guess that’s the question, isn’t it?

*Amresh D. Hanchate, PhD:*Well, I guess the fact that we are going to be penalizing at least in the CMS world based on the mortality metrics.  That is easier.

*Ann Borzecki, MD, MPH:*They’re not already penalized?

*Amresh D. Hanchate, PhD:*Right now I think it’s just the readmissions, but I guess 2014, I don’t know when, I believe the value-based system is going to bring in the mortality metrics.  I guess the current thinking is that these are, whatever, valuable but capture quality as pointed out, making them more significantly and consistently. 

*Paul Barnett:*There is one more question, saying basically that maybe your hierarchical logistic model result is too conservative, that it has too wide a standard error and should you try a regular logistic model instead?

*Amresh D. Hanchate, PhD:*Yes.  In fact, we have done that.  Part of our effort was to also focus on the methodology itself, the statistical methodology of the hospital compare method, which kind of tends to shrink for the smaller volume facilities.  It tends to assign these hospitals to the average range, simply by re-rating their outcomes.  

We did what is, I guess, the tradition approach, the observed to expected ratios is what is used typically and still used in many of the state reporting, New York State and so on, based on a simple logistic model, where they report the observed rate—the ratio between the observed rate and the expected rate, based on the logistic model.  We have done that throughout.

I didn’t present it here, but we did see—I don’t recall if we did all the type of analysis I presented using only the 02E 48:32 models.  The analysis I recall is we contrasted the Hospital Compare profiling against the more traditional profiling method, the 02E method.  If that changed because of enhancement, so if the enhanced data metrics, using the Hospital Compare and the 02E method would show more concordance and we did not see that.

*Paul Barnett:*Well, I think we don’t have any more questions.  We’re getting close to the 50 minute mark here and I want to give Heidi a chance to make sure that people have a chance to give their feedback to the seminar.  It’s helpful to us in planning new seminars.  I want to thank you both for a very provocative, interesting talk, some thoughts about how hard it is to measure hospital performance.  We’ll be very interested to see what you come up with next.

*Amresh D. Hanchate, PhD:*Thank you.

*Ann Borzecki, MD, MPH:*Thank you for the opportunity.

Moderator:Thank you both for presenting.  We very much appreciate it.  I’m going to close out the session in just a moment.  When I do that, a feedback form will pop up on your screen.  We would really appreciate you taking a few moments to fill that out.  

First, our next session in the series will be on March 19th and Ming Tai-Seale will be presenting Mental Health Care During Periodic Health Exams: How Do They Occur and How Long Do They Last?  We’ll be sending registration information out to everyone on that shortly.

Thank you everyone for joining us for today’s HSR&D Cyber Seminar and we hope to see you at a future session.  Thank you.

*[End of Audio]*

Page 1 of 13