ESP 12/13/11 Risk Prediction Models for Hospital Readmission

----------

>> I would like to introduce our presenters, Dr. Devan Kansagara, Dr.Honora Englander, and Dr. David Kagan and Dr. Amanda Salanitro is an instructor at GRECC which is part of the Tennessee Valley healthcare system and she is also an instructor in the section of hospital medicine at Vanderbilt University. We are very thankful to have all these presenters with us today. I'm ready to turn it over to you Devan if you're all set.

>> Good morning or good afternoon depending on where you are. We are going to talk today about predicting the risk of hospital readmission and a lot of what we'll focus on is a systematic review we recently completed on hospital readmission risk prediction models. And before we dive into the topic, we thought we'd get a sense of why people are interested in this topic.

>> On your screen right now, you should see the poll question and go ahead and click the circle next to the answer that best fits you. And we will share the results with everybody in just a moment. I still see the results flooding in. I want to make sure everybody has a chance to answer before I reveal the results to everyone. The poll is still in progress. I do see the answers starting to slow down. We will review those in just a second. Looks about 80% of people have voted. I'm going to close the polls and share the results with everybody. As you can see we have about 16% responding they are involved in implementing a transitional care intervention, about 40% are involved in using readmission rates as a quality metric and about 27% are researchers interesting studying readmission risk prediction. And about 18% are just curious. Thank you to all of you.

>> Wonderful, thanks. It seems like a pretty good representation of all groups here. Our talk will cover several different things here. I will start briefly with an overview of the evidence-based synthesis program for which we did this review. And then we will move quickly into the meat of the talk, we will describe why people are interested in readmission risk prediction and it largely has to do two reasons. One would be quality of reporting and the other has to do with clinical applications. And then we will move into describing in depth our systematic review of readmission risk prediction model, we will review our methods and findings. We evaluated 26 models so obviously we don't have time to describe all the models. But we have chosen three models to describe in a little bit more depth which we will do later in the talk just to give you a sense of types of models are out there. A bit of a spoiler alert. The models don't perform very well, so we will discuss reasons for the poor performance and then we will hopefully leave you with some lessons learned from all this. So the ESP program is a HSR&D funded VA organization which has been in existence for several years and it's been established to provide timely and accurate syntheses for VA stakeholders. The four centers, Portland which we're a part of, and then the others are listed on the slides. 

>> If you have any questions about the ESP program in general, we can answer them toward the end of the question session, and that there is a link to the ESP topic nomination form on this slide.

>> To move into the meat of the talk. Why are people interested in readmission risk prediction? As many of you probably know, readmissions are common and costly. This slide is taken from a study that came out a couple of years ago looking at Medicare fee for service patients nationally and  their rates of read hospitalization. This particular map shows thirty-day readmission after hospital discharge. The map shows a variation in rates from state to state, but they're relatively high all across the country. They range as high as 23% and as low as 13%, but roughly on average about one in five patients are readmitted within 30 days after hospital discharge. This is estimated to cost Medicare over $17 billion. Interestingly, a report just came out comparing VA to non-VA hospitals. And our thirty-day readmission rates in the VA are very similar to readmission rates in non-VA hospitals. For instance, for congestive heart failure patiets, about one quarter of all patients are rehospitalized within 30 days. About one in five acute mycardial infarction and pneumonia patients are rehospitalized within 30 days. This comes from a quote from the Commonwealth fund website. "Hospital readmissions are frequent and costly events, which can be reduced by systemic changes to the healthcare system, including improved transition planning, quick follow-up care, and persistent treatment of chronic homelessness." This really encapsulates the thrust of interest in readmission rates in general, and as we will see, the target of reducing readmission rates, people have become interested in trying to predict who will and who won't be readmitted to the hospital for a couple different for reasons. The first reason is because risk standardized readmission rates have become a quality metric. This metric is currently being used for public reporting purposes to compare hospitals, and as we will talk about, will soon be used to inform financial penalties of hospitals that have high risk standardized readmission rates. So the risk standardization part of this phrase is where the risk prediction models come in. Before we get into what it means, we have another poll question here.

>> Go ahead and take just a moment and fill in your responses. We have had about 70% of people vote already. We will give it just another couple of seconds. And wait for a few more. We've had about 82% of people vote. I will go ahead and close the poll now, and share with everyone the results. It looks like about 58% think yes, and 18% no, and 23% don't know or not sure. Thank you for those responses.

>> Thanks. To move on, the rationale for risk standardization. If you consider two hypothetical hospitals. Hospital A and B. Hospital A is a midsized hospital in an affluent suburb. Its patients have relatively few comorbidities. They are younger, and many are insured. It is located in a health system with good access to out outpatient care and the hospital has a good track record of care coordination. Hospital B on the other hand as an urban tertiary care center, whose patience -- many of whom are transferred into the hospital, have multiple co-morbidities, they have complex illnesses, and many are uninsured. It is located within a system with limited access to outpatient care, and limited peridischarge services. Is it fair to directly compare hospitals A. and B.? The answer is probably not without doing some statistical adjustment. You have to adjust for the patient case mix. It wouldn't be fair to directly compare Hospital A to B. Because B has the sicker patients so you have to account for that. If we pull up the hospitals again, what we would ideally want to do is control for these patient comorbidity factors that the hospital really has no control over. While on the other hand, you would not want to control for some of the  system-level factors that are the targets for change. Things like improving care coordination and innovating better peridischarge care services are the types of things the public reporting and financial penalties are trying to get hospitals to innovate on, so you would not want to obscure that variability from hospital to hospital.

>> How the CMS actually calculate risk standardized readmission rates? It compares a hospitals performance given its patients case mix with the average hospitals performance giving the same case mix. In other words, on the top part of that ratio, you would take the number of thirty-day readmissions, predicted for a given hospital, based on its patient case mix and also based on its baseline readmission risk. That value is really determined by the hospitals track record on readmission rates. The bottom part of the ratio, it is the number of thirty-day readmissions for an average hospital with the same patient case mix. And that's multiplied by the US national readmission rate. A hospital with a high baseline readmission rate may have the top part of that value may be 20%, whereas the average hospital with the same patient case mix might have an expected readmission rate at 30 days of 10%. That ratio is two, if the US national average for that group of patients is 12%, then the risk-adjusted readmission rate is 24%. Obviously, that number would be lower if the hospital in question had good baseline performance, that ratio would be less than one and they would have a risk standardized readmission rate less than the national average. 

>>These risk standardized readmission rate are currently being reported on the Hospital compare website, Hospitalcompare.HHS.gov. You can type in a ZIP code and pull up these readmission rates. And that is where that VA and non-VA data partially came from, that was reported in Kaiser health. For this hospital comparison purposes, currently, who is counted? These are patients age 65 or older, enrolled in Medicare fee-for-service programs, enrolled for at least a year, discharged alive, they exclude patients that left against medical advice and currently we are looking at acute MI, heart failure, pneumonia patients. As I have said before, CMS plans on penalizing hospitals with high readmission rates beginning in 2013. Payments will be cut, if risk standardized readmission rates for these three conditions, MI, heart failure, pneumonia are in the highest quartile, the penalties will rise each successive year and they may very well add additional diagnoses as listed here in the future. Here is another poll question. 

>>Do you think hospitals should be financially penalized for high readmission rates? Please select the answer that best fits your opinion. And we've had about half the people vote so far. We will leave it open for another few seconds. We've had about 80% respond, so I'll go ahead and close the poll and share the results with everybody. About 37% say yes, they should be penalized and 40% say no and 23% they don't know, not sure. Thank you to those respondents.

>> Interesting, a larger proportion of participants thought that hospitals should not be financially penalized compared to the numbers that thought should have hospital comparisons based on readmission rates. To switch gears, the second reason for interest in readmission risk prediction is for clinical purposes. You'd want to identify high-risk patients in your intervention. The transitional care interventions have become a hot topic in the last decade. Transitional care has been defined as a set of actions designed to ensure the coordination and continuity of health care as patients transfer between different locations or different levels of care. There've been several examples of transitional care interventions that have reduced readmission rates at least on a small scale. And these are just a few examples. Many of the interventions involve some sort of in-hospital component, for instance, the care transition intervention features a nurse who visits with  a patient in the hospital and does some patient coaching and patient education and then does follow-up in the patient's home after a period of 30 days. Some of these interventions are somewhat resource intensive, and ideally they would be applied to be higher-risk patients and not the lower-risk patients to make this an efficient use of resources. Here is another poll question having to do with programs in your own facility.

>> Is there a transitional care program at your facility? If the answer is yes, how are the patients identified for the intervention? Yes, based on disease, yes- based on clinical referral, yes, based on risk assessment model, yes, but I don't know how they are identified. Or no, there is no transitional care program. We can only have -- have five multiple choice so I had to cut off the don't know/not sure. We've had about half the people response so far so we will leave it open for a few more seconds. The responses seem to be slowing down, and I am going to go ahead and close the poll and share the results with everybody. Looks like 18% report yes, and it is based on the disease. 20% report yes based on a clinical referral, 8% percent yes based on risk assessment  model and 20% yes but I don't know how the patients identified and the majority, 35% say no transitional care program. Thank you to those responded.

>> Thanks a lot. Here's the schematic of what a peridischarge operation might look like. Patients admitted there would be some sort of medication assessment and reconciliation and on down the line. On into the post discharge period. We have circled the risk assessment portion, which would happen sometime after hospital admission before discharge and might identify a higher risk group of patients for whom a different type of service is provided. In this instance it might be home visits for higher risk patients. So in thinking about creating risk prediction models, the characteristic of these models might differ if they are designed for hospital comparison versus clinical application purposes. Hospital comparison models need reliable data that is easily obtained, deployable in large populations that uses variables clinically related to  a validated and target population and has good predictive value. A clinical application model ideally would  probably provide data before discharge since many of these interventions begin at or before discharge. They should distinguish very high from very low risk patients so you are not wasting resources on patients who really do need it. They should be not overly complex, and adapted to settings and populations in which use is intended. OK so our review, given all this background, we wanted to synthesize the available literature on validated risk prediction models, described thier  performance and assess their suitability for clinical or administrative use. 

>>We will talk a little bit about the methods here. We searched MEDLINE, CINAHL, and the Cochrane Library through March 2011 and MED based through August 2011. We included studies of statistical models that were designed to predict hospital readmission risk in medical populations. And we only included validated models which means that a model would have been derived in a given cohort but then tested and validated in another cohort. We excluded models testing focused on non medical populations, so we didn't look at pediatric, surgical, psychiatric or obstetric populations. We included non English-language studies and studies in developing nations. This is a somewhat busy slide and I will walk through it step-by-step. We wanted to characterize models to get a better sense of whether they were designed for hospital comparison or clinical purposes. This is how we characterize the model. The first step was to determine the data source that was used to gather variables for the model. The data source could be administrative or primary. Administrative data sources are those that are generally claims based data though they can also include EMR automated, electronic medical record data. That is as opposed to primary data collection which involves some sort of data gathering which could be survey or chart review. We then looked at the timing of data collection. If data was available at or after discharge, we considered it retrospective. We classified all claims data as retrospective as we assumed that claims even from indexed admission wouldn't all be posted and reliably available until at or after discharge. Data available before discharge, we classified as real time. Model category, you can have four model categories. Retrospective administrative database models are those that are probably best designed for hospital comparison. Models that use real-time administrative data could be used for clinical purposes.  And models that use real-time primary data collection could also be used for clinical purposes. And models using retrospective primary data collection methods, we put question mark clinical here depending on the timing of intervention. If the data wasn't available until discharge, you couldn't use it for intervention that began before discharge.

>> How do we assess model performance? One of the main ways to look at model discrimination. Model discrimination, we reported through the c-statistic which measures the models ability to discriminate between those who get readmitted and those who don't. The c-statistic is equivalent to the area under the receiver operating curve. An example here would be a c-statistic of 0.7, means that a model will correctly sort a pair of patients one of whom is high-risk and low-risk that c-stat of .7 means the model will correctly sort that pair, 70% of the time and I will get it wrong 30% of the time. The range of values can range from .5 which is no better than flipping a coin to 1 which is perfect. C-Statistics of .5 to .7 are considered poor .7 to .8 are acceptable or modest and greater than .8 are good. We also look at model calibration which assesses the degree to which predicted rates are similar to those observed in the population. We reported simply the range of Observed readmission rates from the predicted lowest to highest risk groupings. We also assessed some the methodological characteristics of the model. We looked at how models define the cohort they're studying. How completely they were able to gather follow-up data on that cohort, the adequacy of prognostic and outcome variable measurement and the method of validation. We didn't exclude models based on how they chose to validate their models. We look at all validation methods. And validation methods did differ from model to model. We wont so much of the methodolical assessment but if anybody is interested, I can send it after the talk.

>> We will switch gears to results here. I will hand it over to Dave Kagan.

>> I'm going to give a broad and brief overview of our results and then Dr. Englander is going to come on and drill down and give some statistics about a few of the individual models. Our original searches identified almost 8000 different studies that would potential he be relevant. Which we widdled down quickly to 286 potentially eligible articles for fulltext review and finally identified 30 primary studies of 26 unique models. Of these 26 models, about half for just over half, 14 were categorized as retrospective administrative data models. Again just to remind you retrospective means the data became available after discharge or shortly before, but not around time of admission. The administrative data just means that this was claims data or some other data that was available in an automated fashion. The other not quite half, 12 studies were either real-time meaning data were available at or near the time of admission and or required primary data collection in the form of survey or chart review. Only one of the models attempted to evaluate preventable readmission which are really the ones we'd be most interested in. The rest focused on all cause readmission and the most commonly used time frame was 30 days. Of the 14 retrospective administrative data models, i.e. the ones we think were most appropriately used for hospital comparison, nine were tested in large US populations that would be most relevant to us. And three of these nine were the models derived for CMS, in the heart failure, acute MI and pneumnomia populations. The C-stat demonstrate that the performance of each of these studies is not particularly good. There was a range of C-stats from .55 to .65, they did a poor job really of discriminating and this was true as well for the CMS model, which had a c-stat of .6 for heart failure and .63 for MI and pneumonia.Three of the better performing models potentially relevant to hospital comparison were derived outside of the United States. Even these had modest discriminatory ability.

>> Of the other 12 remaining models that we thought were perhaps best used clinically, three use real-time administrative data with c-statistics that again weren't terrible impressive. At best, were in the modest discriminatory ability range. Four used Primary data available before hospital discharge and five used primary data available at or after discharge. This included the model that has the best performance but again, primary data available at or after discharge will be harder to use to target an intervention if the data isn't available until after the patient is at home. In some, while performance from a discriminitive f capacity was often poor, there were actually still some meaningful gradients across the different risk stata in some of the individual studies. In other words, the studies that did a pretty decent job of identifying low versus high risk and the Observed readmission often differed by 20% to 30% among patients who were categorized as low versus high risk. I will turn it over to Dr. Englander to go through some the specific models.

>> I'm going to delve into three of the different types of models, and again as characterized by either the kind of data, administrative or survey or primary data collection, and also the timing of data collection. Part of the reason to delve into the specifics of the model is it allows us to see how different data elements infomr the practical utility of these models and also highlights some of the limitations of the model. To start, we look at a model using retrospective administrative data and this is the model developed with CMS, the CMS heart failure model, and they looked at 30 the readmissions amongst fee-for-service Medicare beneficiaries 65 and older. The data was from 2003 and 2004. It is a comorbidity-based model. They took co-morbidities from Medicare claims data using both the index admission as well as claims data from the 12 months prior to index admission and this including a mix of inpatient and outpatient data. What they did was they had 189 condition categories that they've been boiled down into 37 variables using logistic regression  and the 37 variables included age, gender, cardiovascular variables, like the presence of coronary disease, arithmia and also a number of comorbidities. You can imagine if somebody have failure to thrive or malignancy that that would increase the risk. Their model used 37 variables, and the performance of the model had a c-stat of .6 with a discriminative ability or calibration of between 15% and 37%. This is really a large administrative database intended to be used for hospital comparison. 

>> The next model is a model using real-time administrative data. And this was a model developed using electronic medical record data. They have a cohort of patients with congestive heart failure and they looked at 30 day readmission, all cause readmission for these patients. This is done at a single urban center in Dallas Texas, with a large socioeconomically disadvantaged patient population. For their EMR data elements, they have to be capable of extraction from the hospital EMR, they also had to be available within the first 24 hours of hospital presentation. Another criteria that they used in terms of selecting their data elements was that they had to be reasonably available to most hospitals to improve the generalizability of their model. And interestingly, this model used a Tabec mortality score, which is more typical, includes more typical variables including labs and vital signs but then they also incorporated certain social behavioral utilization variables including the number of address changes that their patients had in the EMR, their marital status, socioeconomic status, history of axienty or depression as well as confirmed cocaine use by urine screen and other utilization variables. This model relative to the others performed for fairly well with a c-stat 0.72 and again the calibration of 12.2 to 45.7. And then the last model that we will talk about is the probability of readmission.

>> It was a model with primary data collection collected in real-time, and the model was developed using four-year all cause readmission data. However, -- actually, the survey was performed with patients when they were outpatients as part of the longitudinal study on aging. But then looked prospectively at readmissions. It was developed in a Medicare population 1984 however subsequently it has been validated looking at 30 day and also 41 day readmissions with poor performance. They looked at eight different factors including age, sex, self-rated health, the presence of an informal caregiver, as well as other comorbidities and utilization data. You can see how these factors such as self-rated health and informal caregiver would not be available through administrative data but through real-time primary data collection. This model also performed roughly as well as the others with a C-stat of .6. There were several models that compare different models within a single population, and this can be helpful because it gives us a sense of the added or incremental benefits of certain variables in predicting readmission risks. The EMR model that I referenced earlier as I said, used the Tabek mortality model at baseline and at that point, just with the tabek mortality model, the statistic was .61. But when they added the social behavioral and utilization variables, the c-stat went up to .72 suggesting that these variables improve the  predictive ability of the model. Similarly Coleman and colleagues compared an administrative model and then added self-rated health as well as need for assistance with ADL and visual impairment as well as functional status and again, the addition of these additional variables improve the predictive ability of their model.

>> Interestingly, very few of the studies that we looked at incorporated these social determinant and functional status variables. Just going to take a moment to describe some the variables that were included in the model. Again as an attempt to understand better where some of the shortcomings are and perhaps for some further opportunity are for these models. Most of the models included -- this is divided up into the models that were included these variables in the final model, those that were evaluated but then not included, and the variable not considered for use in the model. Most of the models used medical diagnosis or some kind of a core morbidity index. And many of them also used prior hospitalizations. But when we looked at social factors, you can see that these were largely not considered in the model development. This was quite possible that this explains some of the poor performance of the models. Now I will pass on to Dr. Salanitro at Vanderbilt. First the poll question.

>> We will go ahead and pull up our last poll question for today. I will launch that now. Based on your own observations at your facility, which contributes often to preventable readmissions? You can choose more than one option. The options are lack of access to timely outpatient follow up, poor quality of inpatient care, lack of patient self-management training, lack of access to palliative care/hospice services, or patient factors such as lack of social support, compliance, mental health, substance use disorder. And the responses are streaming in so we will give people a few more seconds to respond to that. We have about 55% response rates so far. We will give people another 10 seconds or so. We've had about two thirds of the people respond. I am going to go ahead close the poll and show the results. It looks like 46% have of observed that it's lack of access to timely outpatient follow up. 8% percent, poor quality of inpatient care, 58% lack of patient self-management training, 12% lack of access to palliative care, 85% paitent factors and as you can see people have the option to choose more than one so that is way over 100%. Thank you to our respondants.

>> This is Amanda Salanitro and I will be picking up where Dr. Englander eft off. I just want to make sure that I can advance the slides. We are going to go into a little bit more discussion about why most of these models that have been created so far have problems in predicting readmission risk. If you think about readmissions, it's not just one cause that usually brings a patient back to the hospital. It is more complicated than that. And the factors that may be contributing to readmission help us understand how we can improve transitions of care. There is certainly a variety of factors that contribute, ranging from patient factors like comorbidies, but also things like social supports, their literacy, both literacy and health literacy, and then ranging also to things like inpatient quality of care, the Post discharge care, and their access to care and the Post discharge period as well as even the supply of beds in the hospital or the area in which the patient lives. Transitions of care, improvements need to respond to these complexities and not be unidimensional. They need to respond to different patient needs and bridge important patient care gaps around the time of discharge. 

>> And so what we are focusing on for the rest of the discussion is how do these social determinents play into rehospitalization and they may be mediated by their access to care in the Post discharge period. Not surprisingly, the variables that have been included in many of the models that you saw,and those that we reviewed were usually found in administrative data sets. They include diagnoses or comorbidity indexes like the Charleston or the Elix Howser index, prior utilization certainly predicts much of future utilization, and then easily obtained demographic very bowls like age, sex, race, or ethnicity. Some of the models have explored the additional predictive value of other social determinants thought to influence readmission. Those include things like illness severity, that might use Apache scores or even lab values that are determined during the hospitalization. Other models have looked at the presence of mental health, both depression, anxiety and substance abuse or use. And of particular interest is the impact of depression on self-management for illnesses such as coronary disease, congestive heart failure and diabetes in particular. Some of the models have looked at overall self rated health again from the patient's perspective and their functional status including such things as their -- if they have visual or hearing impairment or even cognitive impairments that may play into their self-management skills. Socioeconomic status is often a measure of that incorporates several social factors including their insurance status, employment status, their income and their educational attainment. Social support as several of you indicated in the poll question can also include things like the need for and the availability of caregivers in the post discharge period. Another variable of interest is access to care, and that can include also aspects of how close patients are to facilities and the rurality of their residence. We've also been interested in the role of health literacy and numeracy and self-management issues like self-efficacy and coping skills as to how they contribute to readmission risk.

>> From the articles we found in the systematic review we are looking -- dedicating a whole nother paper to delve into the measures used for the social determinants in the gaps in our knowledge with regard to how much they contribute to readmission risk. Although some of these variables are not currently collected on a regular basis, during hospitalization or shortly thereafter, nor are they incorporated into electronic medical records on a regular basis. The national Institute of health and the Society of behavioral medicine have made recommendations earlier this year as to how to incorporate specific patient reported measures of some of these variables into electronic medical records, often with having a screening question and follow up questions to determine the extent to which these social factors are playing into the utilization of health care for the patients. And since these models were developed thus far have pretty limited perdictive ability I think the use of these variables reflecting social determinants will grow and be more rigorously tested in the future.

>> I just wanted to touch on as was mentioned before, a couple of the transition care programs, interventions that have been implemented fairly widely now at different hospitals, some of you may be familiar with project boost, which stands for better outcomes for older adults through safe transitions, this is a program that is sponsored by and supported by the Society of Hospital medicine in which hospitals have a mentored implementation of a care transition intervention to help reduce readmissions. They focus on the eight P's as they've designated. So that's Problem medications, or high risk medicines that may contribute to readmissions, psychological issues, especially depression, as a factor influencing self-management skills, principal diagnoses, as you can see several of them like congestive heart failure, that are already high-profile diagnoses. Polypharmacy, patients that are taking more than five medications, whether they have poor health literacy and they cannot explain what has been taught to them on the inpatient side. Patients may be lacking social support and they also might have prior hospitalizations or a need for palliative care.

>> Another care transitions intervention that has been implemented is the best call project red which is stands for reengineered discharge. This was a project out of the Boston University group and although it was not based on any formal model in predicting readmission, they do include targeting risk factors such as depressive symptoms, patients with limited healh literacy, and frequent hospital admissions. And then some access to care things like unstable housing and substance abuse. In summary, I just wanted to recap and say that the risk prediction models that we looked at have been developed mainly for two different reasons. One for hospital comparison and to for clinical intervention purposes. Most models in both of the categories performed fairly poorly, only slightly better than chance. And most of them have relied on comorbidity and utilization data, whereas few have looked at the additive benefit of social determinants variables in the models. With that, I will turn it back over to Dr. Kansagara.

>> Just in the last few minutes, we will try to sum up -- what does this all mean? Are there any practical lessons here? As we have seen, the hospital comparison models don't work very well. One implication might be that broad-based comparisons of risks standardized rates, especially when tied to reimbursement may be problematic and could be associated with unintended consequences. Here's a paper that came out this summer looking at risk-adjusted hospital readmission rates across the country. Divided into quartiles of readmission risk. And they adk the question which hospitals have the highest remission rate? We circled two things as an example. The first circle highlights that the hospitals with the highest readmission rate tended to be located in the poorest areas. And they also tended, the second circle shows they tended to be the hospitals with the fewest resources, the lowest nurse to census ratio, fewer cardiac services and so forth. This highlighted a little bit of concern that financial penalization could exacerbate disparities potentially. For clinical purposes, the perfect does not have to be the enemy of the good. Even modest incremental knowledge of risk can improve the cost effectiveness of interventions. We talked about before the calibration issue in that patients in the lowest risk categories, say the lowest 10% of perdictive risk versus the highest 10% of predicted risk, for many models there really was a clinically meaningful difference in observed to readmission rates between the two. Even if you don't get it perfect for clinical purposes, sorting out some of the patients that are very low risk and don't need intervention can add to the cost effectiveness of intervention, and there have been a couple of modeling studies showing that, and we don't have time to show those to your now, but if you're interested, I will show them to you later. The third implication. And as we have described through the talk, there are models can be designed differently depending on the intended use. It's important to match the model to its intended use. Models designed for measuring quality are probably not well-suited for clinical use and vice versa. We should also think carefully about the local population to which the model is being applied. The risk factors that lead to readmission, for instance, may differ in a socio-economically disadvantaged urban population compared to an older medically complex population in an affluent suburb. The models you might use might differ in both those situations.

>> Given the lack of an existing risk prediction, in other words we have not found a single model that works well on a broad-based, in the interim, we have a chance to incorporate clinically informative variables in risk assessment that are not routinely captured as a part of clinical assessment when patients are admitted to the hospital. Dr. Salanitro highlighted some of these things and these are things that we can certainly gather from our patients and it may be in the future that these are incorporated into EMR's. It's important to think about workflow and feasibility of data collection when adapting risk assessment tools. Avoid overly complex models that impede workflow. The data must be easily available in real-time to inform clinical interventions. Dr. Englander talked about one study that incorporated its real-time data collection into EMR and develop software to directly pull those variables  from their EMR are so it could be available in real-time. And on the other hand there are other studies rather that just use surveys, which are resource intensive but relatively easy to employ. Finally, we don't know how many readmissions are preventable. There was a recent systematic review looking at studies that try to assess the potential preventability of readmission. Most of the studies would use the consensus process to determine whether a readmission was -- could be reasonably prevented if better peridischarge services were provided. And the various studies ranged broadly in their estimates of how many readmissions were preventable. They ranged from 7% to 79%, which gives you a sense of our uncertainty and knowing how may readmissions are truly preventable. Therefore, aside from readmission rates, it may be important to think about using additional metrics to measure peridischarge care. These could be things like assessing the adequacy of patient education, completeness of discharge summaries and other process variables, access to outpatient care and so forth.

>> Finally, since this is a VA talk, I thought I would bring it back to the VA data we had shown before. Again, the VA hopsitals and non-VA hospitals have very similar 30 day readmission rates, which is interesting because many non-VA hospitals are trying to get to a level of integration that the VA already has. This raises the question, can we make improvements in an already integrated system? I think there are two ways of looking at this. One is to say well we've already got an integrated system and yet our readmission rates are very similar to not integrated systems. Can we really lower these rates even more? Another way to look at this is to say we have room for improvement in the VA and in improving the quality of our peridischarge care. That is the end of our talk. We would be happy to answer any questions, and at the bottom of the slide is a link to the report itself and these slides will be available on that page as well. Thanks.

>> Thank you everybody. Let's go ahead and leave that slide up, and we will leave it on the screen while we are doing the Q&A. We do have quite a few questions that have come in. The first one, what is the difference between predicted and expected? Aren't you comparing with observed?

>> That's a good question. We had struggled with that language a little bit. That slide describing how CMS is calculating risk standardize readmission rates is a little bit tricky. It would've been easier to understand if they had just-- it was taken directly from their page so this is the language they are using-- it would have been easier to understand if ratio had been observed to expected. Essentially conceptually what they are doing. They use the term predicted because they're using the hospital's baseline readmission rate to make assumptions about their future performance. Technically, it is a redicted value but conceptually it's really based on baseline readmission rates. 

>> Thank you for that response. The next question is referencing poll question number three. Is more of a comment. The problem with the question is that it doesn't allow for nuance. For example, penalizing for poor performance maybe okay until all institutions achieve a threshold level of quality. In the current system 25% of hospitals would be in the worst quartile even if their score is extrememly high, for axample 95% or if readmission rate is low. 

>> I think that that's a good point, and the poll questions were just meant to be -- these are tested are well-developed survey questions. These are just meant to get a rough sense of what people are thinking out there. I think there is a lot of nuance in the whole discussion of reengineering financial incentives to produce better outcomes in the discussion of that is outside the scope of our review I think we wanted to highlight that some of the decision-making is based on models that are --. It's probably the case that poor performers are truly poor performers, and there have been studies suggesting that public reporting in hospital comparisons have led to innovation and so they do spur people to innovate. We have even seen that here locally. They do make changes. It's also important to consider the potential unintended consequences.

>> Thank you for that reply. The next question, why would you require a validated study if it was done on the full population?

>> There were models that we included that were done on large general population data sets but it's important even for those models that they take the extra step of validating. They don't necessarily need to validate and we didn't exclude studies that did not validate a separate cohort. Many models took a large population and did a slipt sample derivation validation. For instance, there is a British model that basically takes the entire British population and divided it into derivation and validation cohorts and tested the models in both.

>> Thank you. The next question, did any of the models look at readmission at time intervals less than 30 days? Arguably, people who are readmitted within 5 days are different than those readmited after 25 days. 

>> There was a range of outcomes that were examined, and 30 days was certainly the most common. There weren't many models the look at shorter intervals. In fact, I believe the only model that looked at a shortened interval is that Thomas model which looked at 15 days and a variety of other outcomes as well. We don't have information on the performance or very good information on the performance of models for very short time frames. It's probably true the shorter the time interval you're looking at, the more closely related is going to be to both the inpatient admission itself and the peridischarge care.

>> Thank you for that reply. The next question we have is were the models developed at the patient or hospitalization level or at the hospital level? Were there any that looked at readmission over time?

>> The models were developed did and tested looking at patient populations. The outcomes were at a patient level, and for instance there were some models that just looked at a single center. They weren't even comparing different hospitals. I'm not sure understand the question totally. In general, the model used data and outcomes at the patient level. 

>> Thank you. If the person who wrote in wants to clarify, they're welcome to do so. The next question we have is actually a comment, models don't seem to take into account reason for readmission. Readmission may be unrelated to a diagnosis which is unrelated to the original diagnosis.

>> That's where this whole issue of trying to assess the potential prevent ability of readmission comes in. As we had said, most of these models looked at all cause readmission but there was a Swiss study that went through a fairly ornate development process, and came up with an algorithm for defining potentially preventable readmission based largely on a chart review methodology. I think that's completely right. It's certainly not going to be the case with 100% readmissions are  preventable with better care after discharge. In many cases, patients are just sick and need to come back to the hospital and the readmission is perfectly appropriate and not preventable. The number of readmissions that are potentially preventable through better peridischarge care is simply not known at this point.

>> Thank you for that reply. The next question, why did you exclude social support or availability of caregiver?

>> We may have not explained this clearly. We didn't exclude them. In fact, I think what we are trying to say is that most of the models we considered very few evaluated those variables. We identified that as a gap in research and an area for future development. Potentially one of the reasons why models may not perform as well as they did, we certainly didn't exclude it from a review standpoint. We were interested in those variables, it's just the models that we looked at, many of them didn't look at those variables.

>> Thank you. The next question, can you discuss preventable readmissions, that label seems very subjective. Is that defined or measured by CMS or do they look at all the readmission?

>> They look at all readmissions for those populations-- they are only looking at three populations right now. People try to define preventability. Example of there is now the 3M company, 3M software PPR pr potentially preventable readmissions and the way they came up with came up with their definition of potentially preventable readmissions was to look at 317 DRGs, and they lined up these DRGs on the x axis and then again on the y axis. So the x axis would be index admission and the y axis would be readmission. So there's 96,000 cells there where you would have a DRG for the index admission and a DRG for the repeated admission. And they would have investigators independently assess each cell and say are these two things reasonably related. And so they would define say for instance you have somebody with the DRG on index admission of diabetes who is readmitted with a myocardial infarction. They would define that as a potentially preventable readmission. I think that's a fairly rough way of defining potentially preventable readmission and probably many would say clinically you have no way of knowing if that readmission was truly preventable unless you look at the chart and really looked at the circumstances of the particaular readmission. I think that's a fairly rough way of defining preventable readmission and their reported rates of potential preventability are fairly high as a consequence. Other studies have tried to get at this issue by doing the fairly laborious chart review process, they will take a sample of readmitted patients and have investigators independently look through the charts and try to -- according to different alogrithms - figure out if these things are potentially preventable or not. The short answer is to say no, there is no standard for defining potential preventability yet.

>> Thank you. The questions keep pouring in. We've got several more. The next one, Doesn't look like the transition, self-care variables were tested in most prediction models, but they are the focus of efforts to reduce readmissions. Your thoughts about whether planned interventions are off-base?

>> It is a good question. One of the things we are trying to -- with many of these models, if you're using them for hospital comparison purposes, you couldn't really control for those health system factors such as care coordination and so forth, because those are the very things that you are trying to target. If you control for those factors, you would obscure the very variability between programs that you are trying to get them to improve on. To a certain extent, the exclusion of those system-level barriers is purposeful, and certainly one reason why they models don't perform as well as they could. You just don't want to account for them because you would be able to find the differences between hospital performance. I think there are other things like self-efficacy and self-care those types of things are really interesting things to think about conceptually. I think some people would argue that you can change self efficacy through self management training and patient coaching and things like that. That's a potential target for quality improvement interventions. You wouldn't want to control for them. As to whether or not those are good ways of intervening, and doing transitional care interventions, I think that is really addressed by intervention studies, rolling these interventions out and seeing if they actually reduce whatever outcomes is, in many cases the outcome is readmission rates. So the care Transitions intervention used is very focused on a patient coaching model and home healthcare and they reduce readmission rates at least in the population that they studied, and efforts are being made to do similar types of interventions on a broader scale. They have not been studied on a broader scale. 

>>If you're interested, there is a systematic review of the annals of internal medicine. Within the last two months, in October, the first author's last name is Hansen, and they did a systematic review of transitional care interventions and looked at a broad variety I think about 46 different types of interventions. Different studies, and found that many didn't work, and some of the ones we highlighted did work, but many of the ones that they looked at did not work. So that'll give you a better sense the kind of universe of things that people of studied and tried.

>> Thank you for that reply. The next question, excellent talk thank you. How effective our interventions to reduce readmissions?

>> I would point to the same review- I'm sure the question got written while I was talking-- it is Hansen annals of internal medicine from October, and that's the most recent systematic review of transitional care intervention. In a nutshell, they basically found that a few these intereventions, like the Coleman intervention and a few others worked, but many did not. And they tried to get at what components of the intervention tended to be included and models that worked versus models that didn't. The few models that have been studied and shown reductions in readmission rates, again e.g. Coleman's work, Mary Elnman's work-- all involve some sort of a bridging component wherer there's often nursing care provided in the hospital, then follow-up after discharge. Often in the home care setting. Whether that is applicable for broad populations of patients is still not known. 

>>, Thank you, you are correct a lot of these questions came in before you finish presenting the material. A next question. One model has a c-stat of .83. Can you briefly describe this model? Feel free to go back in your slides if you need to.

>> That model is not in one of our slides. That was an Eric Coleman model and so that was the only model that used as its outcome- I think they called it- a complex care transition outcome. The outcome wasn't just 30 they readmission. It also included -- move to a higher level of care. It could be moving from a home to a nursing home or from a nursing home to a hospital. And so the outcome was slightly different and also, they used -- their survey data came from the Medicare beneficiary survey and the answers to those questions weren't necessarily -- the survey wasn't necessarily performed during hospital admission. It may have been performed sometime after  hospital discharge. Then they kind of retroactively applied that in a model of predicting readmission risk and that's why we classify it as retrospective primary data collection. They had the advantage of using variables well after hospital discharge which you could imagine you have a fuller picture of some of the clinical factors-- but I think that model does also highlight that they looked at -- this was one of the models Dr. Englander was talking about where they looked at administrative data and then they added on this  survey component and the survey component improve the performance of the model and included things like functional status, visual impairment and things like that. This was by the way ederly cohorts for whom those types of factors might really be conceptually important in thinking about readmission rates in those populations.

>> Thank you. We have reached the scheduled time limit for the session, but if you are available to continue answering questions, it would be great to do so now and include them in the recording. Are you available for a few more minutes?

>> I would love to. But we actually have to run downtown to a meeting. 

>> Okay, what I will do is go ahead and send you these remaining questions off-line, and then if I can get written responses I will post them with the archive recording.

>> Great, thanks so much.

>> Thank you all for presenting. Thank you for the attendees for joining us. That does formally concluded today's HSR&D Cyberseminar.