**Transcript of Cyberseminar**

**HERC Health Economics Seminar**

**Are Misinterpreted, Hospital-level Relationships Between Process Performance Measures and Outcomes Undermining Evidence-based Patient Care?**

**Presenter: John Finney, Ph.D.**

**May 22, 2013**

**Are Misinterpreted, Hospital-level Relationships Between Process Performance Measures and Outcomes Undermining Evidence-based Patient Care?**

**May 22, 2013**

*This is an unedited transcript of this session. As such, it may contain omissions or errors due to sound quality or misinterpretation. For clarification or verification of any points in the transcript, please refer to the audio version posted at www.hsrd.research.va.gov/cyberseminars/catalog-archive.cfm or contact: herc@va.gov.*

Dr. John Finney:Good morning, or good afternoon everyone. I am John Finney. I am a research health science specialist at the Center Health Care Evaluation at the Palo Alto VA. Just to go right into my presentation, the title of my presentation is somewhat provocative. I hope everyone can see the slides now. But, I hope by the end of my presentation you will agree that the answer to the question that is posed in the title is yes.

The work I will be talking about was, and I am having trouble, there it is, okay. The work that I will be talking about today was supported by a couple of HSR&D grants, one to Alex Sox-Harris, one to me a number of years ago, and of course, the views presented are mine and not anyone else’s.

The talk today is based on an article that Keith Humphreys, Dan Kivlahan, and Alex Sox-Harris and I published back in 2011 in the *American Journal of Public Health,*and it was really an article on the ecological fallacy as it relates to examining performance measure outcome relationships. But, a few of the things, at least, that I will be talking about today are not in the article.

So, what I want to do today is to provide an introduction, talk a little bit about the ecological fallacy, illustrate with a continuing care performance measure for VA patients with substance use disorders, that one can get different relationships at the facility and patient levels. And, talk about how differences can occur at those two levels and then talk about why those differences can occur. And, finally, discuss a few implications for health care quality management and research.

So, probably everyone on the call knows performance measure is one factor that has been pointed to as driving substantial improvement in health care. And, many performance measures assess the extent to which processes of care that have been shown to cause, or at least relate to positive outcomes among research participants are applied in a health care facility. And, so that is what I am referring to by process performance measure, or PPM. And, those PPMs are used to rate health care providers and facilities and larger health care systems, and states or provinces, and even countries on quality of care, but they are usually, most often used to rate the quality of hospital care as illustrated by this Hospital Compare website.

And, those process measures of health care quality typically are implemented on the assumption that patient level care processes aggregated to the facility level are associated with positive facility or aggregated outcomes in the same way as was found at the patient or participant level in the supporting research. So, the basic idea is there is research that shows that receipt of performance measure specified care is associated with positive outcomes. The idea is that if one aggregates the performance measure care to the facility level and talks about the rate or the percentage of patients with designated conditions who are getting that performance measure specified care at different facilities. If one aggregates facility outcomes and talks about the percentage or rate of patients with a designated condition who have positive outcomes at the facility level, that relationship should be at least similar to the relationship found at the patient or research participant level.

And, that is an example of the homology assumption, what Michael Hannon referred to as the assumption that things work the same at different levels of organization or analysis. So, linking facility performance rates to facility outcomes is thought of as one way to validate process of care measures, or as a form of what Brian Mittman has referred to as post-implementation surveillance. But, investigators examining facility-level PPM-outcome relationships sometimes have found that those relationships are weak or non-existent. 

So, one example is this article that was published in JAMA in 2006 by Elizabeth Bradley and her colleagues, and they were looking at the 7 CMS and Joint Commission core processes for managing acute myocardial infarctions. So, they were looking at beta blockers at admission and discharge, aspirin at admission and discharge, ACE inhibitors at admission, smoking cessation therapy for those patients who were smoking, timely reperfusion therapy, and then a composite of those seven indicators. So, across 962 hospitals, they looked at the relationship of each of those indicators and the composite, too. If you look at the bottom row, in-hospital all-cause risk-adjusted mortality rates and found no significant relationships at all. In terms of 30-day mortality rate they found that three of the seven indicators were related to mortality rates at reduced mortality rate, but those relationships were pretty weak and even the relationship of the composite of the seven indicators to mortality was pretty weak. 

Another example, also published in JAMA in 2006, was offered by Werner and Bradlow, and they were looking at the CMS composite performance measures for myocardial infarction, heart failure, and pneumonia. And, they compared the absolute reduction in risk-adjusted mortality between hospitals performing in the 75th percentile and above on those composite performance measures, and the hospitals performing at the 25th percentile and below. So, they looked at over 3,650 hospitals and they were looking at not only in-hospital or inpatient mortality, but 30-day mortality and one-year mortality. And, if you look at those absolute reductions in mortality rates, you can see that not all were significant, but all were very small differences. 

And, the last example, published in 2010 in the *Archives of Surgery*, is this article by Nicholas, et al. What they did was to compare surgical care improvement project compliance data in terms of things such as the timely administration of prophylactic antibiotics before surgery and the timely discontinuation of antibiotics after surgery, and other surgical care performance measures. And, they looked at the odds of surgical mortality rate and also, venous thromboembolism and surgical site infection, neither of which is shown in this slide. But, they looked at the odds of, in this case, surgical mortality for the hospitals and it was 2000 hospitals in the U.S. The hospitals in the highest quintile versus the hospitals in the middle quintile, and then they also compared mortality for hospitals in the lowest quintile versus those in the middle quintile. And, they found no significant differences in the odds of mortality in those comparisons, either contemporaneous comparisons in the same year, or lag comparisons performance in one year with mortality in a later year.

So, there is certainly nothing wrong and I think actually those hospital-level analyses are valuable, but the interesting thing about them is the reactions to them, and in some cases the misinterpretation of what those hospital-level analyses are actually showing. So, just in terms of a range of reactions to the findings, Bradley, et al, and Werner and Bradlow notice a variety of factors that might have reduced the relationships between those facility-level process performance measure rates and facility-level mortality rates. So, they pointed to a high level of compliance for some of the practices like aspirin at admission for AMI. They talked about the insensitivity of in-hospital and 30-day mortality as outcomes.

A much more dramatic reaction was in a commentary on the Nicholas, et al, article by Mabry, and he said if those findings of Nicholas, et al, those null funding are true, they call into serious question the increased time, labor and effort currently expended by hospitals and surgeons across the U.S. to comply with the SCIP program process measures. “How can it be that the National Quality Form, CMS, and others got it wrong?” And, to be fair to Dr. Mabry, he posed that question and then he provided some reasons why, perhaps, the analyses should not be taken too seriously, such as they were early in the SKIP program process and some hospitals that were reported poorer practices might also have been less rigorous in reporting, say surgical complications. But, in any event, called the results into question.

I have been told by a person in the VA Office of Quality and Performance, and by a surgeon at Stanford, that surgeons in both of those systems have questioned why they should be held accountable to surgical practice performance measures that are not related to patient outcomes in studies such as that by Nicholas. And, there is another study by Ingraham. And, I did not hear that those surgeons were too worried about any methodological problems in those analyses. 

Werner and Bradlow, another reaction, said that, “Efforts should be made to develop performance measures that are tightly linked to patient outcomes”, but neither they nor any of the other authors that I cited earlier raised the possibility that performance measures specified practices might relate differently to outcomes at the patient level than they do at the hospital level.

So, in writing that *American Journal of Public Health* article, we wanted to provide a non-technical overview of the relevant multi-level issues in a way that would be accessible to a wide range of stakeholders. And, we felt well-qualified to write a non-technical overview in that, with the exception of Alex Sox-Harris, none of the other three of us had any particular statistical training beyond the usual courses in psychology graduate school programs. But, drawing on Alex’s expertise and on relevant epidemiological and sociological and statistical literature, we wrote the article. 

And, as I mentioned, we focused a lot on the ecological fallacies, and that notion was introduced most prominently to researchers by Robinson in an article in 1950. And, Robinson provided a few examples in that article, and one was with 1930 census data. And, he was interested in the relationship between being foreign born and being illiterate in the English language. And, so he looked at the state level relationship and he found, perhaps surprisingly to some people, that the proportion of foreign born state residents correlated negatively, minus .53, with state English illiteracy rates. And, so states that have higher proportions of foreign born residents had lower English illiteracy rates. And, then he looked at the relationship at the individual level in the census data and this was before techniques were available to adjust for clustering of individuals within states, so he just looked at the individual relationship across all the individuals in the census date, and he found a positive correlation between being foreign born and illiterate in English. So, individuals who were foreign born were at least slightly more likely to be illiterate in the English language than those who were not foreign born. And, what Robinson concluded was that one cannot infer relationships for individuals from relationships for higher level units. In doing so, later became known as committing the ecological fallacy.

The later methodological issue literature on this issue has been more even-handed in addressing problems of inference and moving from lower to higher-level units, as well as from higher to lower level units. It addresses the incompleteness of many single-level analyses and it stresses the need for multi-level analyses.

So, we wanted to explore this issue with some performance measure data. It is one thing, perhaps, to find say unusual or unexpected relationships when one is dealing with kind of arbitrary aggregates, geographically defined like states and the United States. It might be something different to find different relationships between aggregated units that might be considered more integrated or organized like hospitals versus relationships at the patient level. So, we explored this issue with continuing care performance measure for substance use disorder patients in the VA. There is a citation to some of the work at the bottom of the slide. And, to meet this performance measure a patient must have had at least two VA substance use disorder clinic visits, in each of three consecutive 30-day periods following a qualifying discharge from a residential or inpatient substance use disorder program, or a qualifying visit to an outpatient clinic, really, the third visit within 30 days. 

So, for simplicity, again, we were just trying to illustrate that different relationships could arise. We randomly selected one of the five datasets with imputed missing data that had been averaged in that Harris, et al, article. We ignored the fact that at some facilities, patients were drawn from more than one substance use disorder and at other facilities patients were not sampled from all the substance use disorder programs. We ignored providers and programs that, as intervening levels of aggregate, and we did not control for covariates. Again, we wanted to focus in on just the difference at different levels. But, I will talk about confounding later.

So, we ended up with a subgroup of 1,485 patients who had been non-abstinent from alcohol and drugs, and or drugs in the 30 days prior to a baseline assessment. And, we selected that group for a reason. We knew from prior analyses within that group at the patient level there was a positive relationship between meeting the performance measure and abstinence. So, that was a subgroup of patients from non-methadone VA SUD programs that were located at 72 different facilities. The patients had been followed up an average of 7.3 months later, at which point abstinence for the past 30 days was assessed and the follow-up rate was a little over 35% and outcome data were imputed using a multiple imputation approach for those who were not successfully followed.

So, we ran three analyses to examine the relationships between that continuing care performance measure in abstinence. One was a facility-level analysis. The second was a mixed-effects analysis with facility as a random factor, so an individual patient-level analysis, but controlling for clustering of patient outcomes within facilities. And, the third was another mixed-effect or multi-level analysis in which not only receipt or non-receipt of continuing care was a predictor, but also facility rate of compliance on the performance measure, the percentage of patients, qualifying patients receiving the specified continuing care also was included as a predictor.

So, what did we find? In the first facility-level analysis we found no relationship. You can see this regression coefficient in the bottom row and the second column. No relationship at the facility level between facility-level performance on this continuing care measure, and facility-level abstinence. In the second analysis, the mixed-effects analysis controlling for clustering of patients within facilities, there was a positive relationship, and a significant relationship between receiving the specified continuing care and abstinence. So, the odds of abstinence for patients receiving continuing care were 1.7 times greater than the odds of abstinence for those not receiving continuing care. And, the third analysis we found a negative relationship in the bottom row here, between rate of continuing care and abstinence, controlling for individual receipt of continuing care, but it was not quite significant. However, the relationship at the patient level was positive and even stronger than that in the second analysis, so now controlling for rate of facility continuing care, the odds of abstinence for those receiving continuing care, patient-level relationship, the odds were 1.9 times greater than the odds of abstinence for those not receiving continuing care.

So, this analysis raises the question of how to interpret this coefficient for facility-level rate of continuing care. And, in the absence of more data, one really does not know what to make out of it. One would need to see what would be associated with that residual facility-level rate of continuing care after individual receipt of continuing care was controlled. We speculated that perhaps the facilities that put more resources into having high rates of continuing care might have had fewer resources available to provide high quality continuing care. But, that is just speculation. The point is, though, that one can get quite different relationships at the hospital level, and at the patient level in either individual analyses or in the multi-level analysis of the type that you are looking at now, in examining performance measure relationships to outcomes. 

So, the question then arises of how do differences occur between facility- and patient-level PPM-outcome relationships. And, that, fortunately, Greenland and Morgenstern, and Greenland, 2001, provided some tabular examples that really illustrate how those different kinds of relationships can occur. And, I think those tabular examples are extremely important because this literature is highly technical, and if one does not have those kinds of more simple, tabular examples. 

So, we provided a hypothetical example drawing on the Greenland and Morgenstern, and Greenland examples. We made it even simpler. And, so we talk about three hypothetical facilities that were doing quite differently on the continuing care performance measure. So, at Facility A 33% of the qualifying patients received continuing care. At Facility B 52% of the patients received continuing care, and that was just above the threshold of the expected continuing care in this performance measure around the time that we started working on this article. And, in hypothetical Facility C they were doing extremely well in almost three quarters of the patients, 72% were receiving continuing care. But, unfortunately, the abstinence rate was only 25% in each of the three facilities. 

So, if you graph those data, there is no relationship at the facility level between receipt of continuing care, at least the rate of receipt of continuing care and the rate of abstinence. So, those data are drawn from these marginal distributions. Here we have the 33% performance for Facility A, 52% for Facility B, 72% for Facility C, and then the 25% abstinence rates at each of the three hypothetical facilities. And, what is missing from this picture are the join distributions within these facilities that indicate the relationship at the patient level within each facility, between receipt of continuing care and non-receipt of continuing care.

So, we wanted to examine a possible joint distribution within each of these facilities. So, what we did was we multiplied these percentages by two, so now we have two hundred patients at each of the facilities that, again, in Facility A 33% of the patients met the performance measures, 25% of the patients were abstinent, same thing for the other three programs. And, we plugged in some data to see if we could show a positive relationship within facilities given no relationship across facilities.  

So, here are some numbers we plugged in and there is a positive relationship now within each facility at the patient level between receiving continuing care and outcome. So, just to look at Facility A, 26 out of 66 patients who received continuing care were abstinent in these hypothetical data versus only 24 of 134 patients who did not receive continuing care. Again, the same thing at the other two facilities. Graph those data and now we have our zero relationship between continuing care rate and abstinence rate at the facility level, but within each of those facilities we have a positive relationship between receiving continuing care and abstinence, and those different slopes, if you will, for the individual programs, they intersect this line at 25% abstinence at the 33%, 52%, and 72% facility performance rates.

Well, marginal distributions set constraints on what can be, what the joint distributions can be, but within those constraints, one can get quite different results. So, we want to see – and this is not in the article – we wanted to see if we could get negative relationships given the same hospital-level marginal distribution. So, again, plugged in some numbers and now in Facility A, 8 out of 66 patients who received continuing care were abstinent versus 42 of the patients who did not, of the 134 patients who did not receive continuing care were abstinent.

So, now one can graph those data and now we have a negative relationship. Again, the same hospital-level relationship, but within hospitals at the patient level, now there is a negative relationship between receiving continuing care and abstinence.

So, the take-home message, I think, from these examples are that one can indeed get different relationships at the hospital and patient levels. And, secondly, that having the hospital-level relationship tells you nothing about whether within hospitals the patient-level relationship is going to be positive or negative, or non-existent. Sorry, again, those slopes intersected at the performance rate points.

So, I have talked a little bit about how differences can arise, and the interesting question is why do differences arise between facility- and patient-level relationships. And in the epidemiological literature in particular, they talk about effect modifications, so in this context there might be something at different facilities that would cause the patient-level relationship to be different across facilities. They talk about the differential effects of measurement error, especially measurement error in the exposure variable, in this case whether or not people got the performance measure of specified care. But, the reason that is talked about most often, and the reason that I think is most readily grasped is that one might have different confounding variables at the patient and facility levels.

So, I think everyone knows about confounding variables, so at the patient level you might have a confounding variable Z and it is associated both with receipt of performance measure specified care and with outcome. And, to try to get at the actual impact, causal impact of performance measure care and outcome, one would want to control for that confounding factor. The committed ecological analyst might say, “Well, no problem. We’ll just get the rate of patient, or percentage of patients say who are positive on that confounder at each hospital, and we’ll control for its relationship to performance measure care rate and to the outcome rate for the hospital, and everything should be fine.” But, in actuality, we have already seen that this relationship between performance measure care rate and outcome rate at the hospital level does not tell you anything about this relationship at the patient level. So, there is no reason to think that controlling for these confounding relationships at the hospital level should do anything similar to controlling for the ‘same’ confounder at the patient level. So, even with control of the confounders, there is no reason to think that the relationships would be the same at the patient and facility levels. The good news is that effects of at least aggregated confounders can be disaggregated into there between and within facility or patient-level effects.

So, when one groups data or only collects data at the health care facility level, that may allow new variables to affect the relationship between a process performance measure and outcomes. In other words, variables that may not come into play, or have different relationships or different effects on that relationship at the patient level. So, actually, back in 1964, Blalock wrote that the key to the problem – the problem being different relationships at different levels of analysis – “the key to that problem [and thinking about that problem] may come with the realization that in shifting units we may be affecting the degree to which other unknown or unmeasured variables are influencing the picture.” 

And Susser pointed out that higher level units like health care facilities have integral properties, properties that are not the aggregated characteristics of the individuals comprising those higher level units. So, for example, different health care facilities have different leaders, local policies, structural properties, such as staff-patient ratios, and they may operate in different environmental context. And, those factors may confound facility-level relationships between performance measure, performance on a particular quality indicator and outcomes, but they could have quite different impacts or no impact on patient-level performance measure outcome relationships.

So, the fact that different variables may or may not related to process performance measures at the patient versus the hospital level implies that a performance measure at the hospital level may be assessing something quite different than the performance measure at the patient level. So, it is a basic notion of construct validity that what a measure is measuring is defined in part by what that measure of the other things with which that measure may or may not be correlated. 

Anyway, that different constructs at different levels of aggregation or analysis argument has been made by a number of observers. So, for example, the state rate of English illiteracy among foreign-born residents in Robinson’s example that I mentioned earlier, turned out to be a proxy for, or confounded with, the literacy rate among native-born state residents. So, people who had immigrated to the U.S. from other countries tended to settle in the northeastern states where the literacy rates among the native-born residents were higher than in other parts of the country, where maybe fewer foreign-born individuals have settled.

So, in addition to process performance measures might be confounded with, I think it is also important to think about what they might not be confounded with. And, I think part of the confidence in hospital-level process performance measure outcome analyses stems from the believe that various indicators of quality are positively correlated with each other. But Bradley and her colleagues noted that hospital mortality rates, even when they are risk-standardized are likely to be influenced by many factors that are independent of those core process measures, including processes that involve patient safety, staffing, response to emergency and clinical strategies, all of which may, in addition to compliance with the performance measure, may contribute to a hospital’s outcome performance. And, in fact, Isaac and Jha found inconsistent and usually poor associations among patient safety indicators and other hospital quality measures

So, that is kind of long-winded talk, but what are the implications of all of that for quality management? One implication we thought is that quality managers, who only have findings on facility-level PPM-outcome relationships should view those relationships with caution. It is possible that they reflect PPM-outcome relationships at the patient level, but it is likely that they do not. In fact, facility-level PPM-outcome relationships may be in the opposite direction as we saw with the example I presented of the relationship at the patient level.

So, Pinatadosi concluded that with aggregate data, “We not only lose all ability to extend inferences reliably to less aggregated data but we even lose the ability to estimate the direction and magnitude of bias.” Bias meaning difference from individual-level relationships. “We cannot rely on the addition of more grouped data to eliminate the bias.” 

So, in addition to Robinson’s findings, Greenland and Robins noted that ecologic analyses have been conducted that, if taken at face value, would support the conclusion that radon exposure has a protective effect for lung cancer and cigarette smoking has protective effect for oesophageal cancer.

So, if I were a hospital quality manager and say a surgeon came to me and said these hospital-level analyses show no relationship between these evidence-based surgical care practices and outcomes, so, I do not think we, the surgeons should be held accountable to them, I would say, “Well, the evidence you are citing is the same evidence one could cite for arguing that inhaling radon gas can have a protective effect for lung cancer and cigarette smoking can help protect one from oesophageal cancer.”

In any event, I think quality managers should keep in mind Naylor’s recommendation concerning ecologic analyses of treatment effects, which was caveat emptor or buyer beware. A second implication that if a process performance measure is associated with positive outcomes at the patient level, but not at the facility level, health care system leaders should encourage the use of that practice through facility-level performance measures. And, a third implication we thought is that a performance measure is associated with positive outcomes at the patient level, but not at the facility level, then health care system leaders should try to determine what at the facility level is cancelling out the patient level effect of the performance measure specified care.

On the other hand, it might be that the proportion of facility patients meeting the PPM, what is positively related to desired outcomes with a negative or no relationship between patient receipt of that specified care in the outcome. So, for example, within hospitals, patients who receive a particular surgical procedure might be more likely to die. However, at facilities at which that surgical procedure was performed for higher percentages of patients, those facilities might also tend to have better infection control systems or send patients to safer extended care facilities. And, if so, then the challenge would be to ensure a safer surgical procedure at all facilities, while preserving or enhancing infection control and patient safety measures.

So, overall I think that false negative hospital-level PPM-outcome findings are of more concern that false positive results, if the patient-level evidence from randomized control trials and other sources supports that process performance measure. The multi-level findings provide more useful, but also more nuanced information to quality managers on how outcomes might be improved with interventions targeting patients and their care, as well as the health care facility and, perhaps, the broader community context.

So, instead of just that inner circle where one is looking at performance measures or composites of performance measures in relation to outcome, quality managers and researchers might also need to think about the hospital context, say, in terms of infection control, processes and the broader community context with the quality of extended care facilities being one example.

So, in terms of research on performance measures, we think two questions capture the inherently multi-level nature of process performance measures and their relationships to clinical outcomes. And, one of those questions is, independent of the proportion of patients for whom performance measure specified care is provided at their hospitals, is the performance measure specified care linked to better patient outcomes within hospitals? So, that is the patient-level question. The other question is independent of the relationship of the performance measure specified care to outcome for patients within hospitals, is the proportion of patients receiving that care across facilities linked to better outcomes? 

And, the following types of analyses do not address those two questions. So, the facility-level relationship between a PPM and outcome, an analysis no one would do today, but similar to what Robinson did, an analysis of the relationship between a PPM and outcome across all patients in the system ignoring the facility in which they received care. And, also then the patient-level relationship between a PPM and outcome controlling for the clustering of patients on outcomes with facilities. And, the reason that those analyses do not address those two questions is because findings from those analyses reflect a mixture of between or hospital level and within facility or patient level relationships.

So, again, continuing with implications for research, Firebaugh noted that “Single-level analyses are subject to severe omitted-variable bias in the presence of multilevel effects.” The strong implication for researchers is that studies of performance measures or quality indicators should be guided by multilevel conceptual models and use multilevel analyses to examine them whenever possible. And, this type of information that would come from that multilevel analyses where in contract to my earlier slides, you have different intercepts and different slopes within facilities, but you also know something about the patient-level relationship. That type of information could be very useful.

I am going to skip over these next few slides and go to an implication for quality management organizations, which is that they should consider making de-identified patient data on both processes of care and outcomes available to researchers so that researchers can do the multilevel analyses of the type that I have been talking about. Or, they refuse to do that, they should conduct and report multilevel analyses of process performance measure outcome relationships themselves.

And, as we were thinking about that, I did not really know whether these, some of these organizations would have individual level data, but it turns out that they do, at least according to this figure from the Hospital Quality Alliance website. So, it seems like a very useful thing would be the provision of de-identified data to researchers so they could do the type of multilevel analyses that I have been talking about.

So, in terms of conclusions, one is that I hope that you will agree that misinterpreted, and that is the key word, misinterpreted hospital-level analyses of relationships between process performance measures and clinical outcomes can undermine evidence-based patient care. Again, I think these hospital-level analyses are very important in pointing out that if one wants to improve outcomes at the hospital level, one probably has to do more than just increase a few specific clinical practices. A second conclusion is that consideration of multilevel conceptual and methodological issues reinforces the wisdom of Donabedian’s early call for a focus on what we would call a multilevel system of structure, process and outcome performance indicators.

So, I will stop at that point and see what questions there are, Heidi.

Heidi:We do have a couple of questions, but I think Elizabeth Calgo [PH] will be handling the questions.

Elizabeth:Can anyone hear me?

Heidi:We can hear you, yes.

Elizabeth:Great. So, the first question is, so in summary, homogeneity of PPM outcome relationships at the patient level might be exceeded by heterogeneity at the facility level?

Dr. Finney:Well, I think the relationships can be heterogeneous at both levels. I did not really get into that. I did not, except for, excuse me, one slide toward the end where I was showing heterogeneous relationships at the patient level, and a homogeneous relationship at the facility level. But, certainly there could be moderators of relationships at either level, and again, I think that is useful information if one has the data and time and resources to try to uncover what some of those moderators might be.

Elizabeth:The other question is, are the administrative stakeholders in the PM applications aware of this type of data?

Dr. Finney:I did not quite hear all of the question. So, the administrators and what entity?

Elizabeth:I am sorry. Hold on one moment. Are the administrative stakeholders in the PM applications aware of this type of data?

Dr. Finney:I do not think they are sufficiently aware of it and that was one of the reasons that we wrote the article, and in fact, one the reviewers of the manuscript at the *American Journal of Public Health* wrote that he or she had been involved in this field for fifteen years and was aware of this issue, but felt it was exceedingly underappreciated in most discussions of performance measures and their relationships to outcomes. And, in terms of the VA, you know, I mentioned that someone in the Office of Quality and Performance had mentioned that surgeons, based on some of the hospital-level analyses were questioning why they should be held accountable to some of these surgical process performance measure. And, that person was Joe Francis, and we thought an initial draft of this article or manuscript might be of interest to Joe. And, when I emailed him, he said he had been thinking about raising the ecological fallacy in talking to some of the surgeons, and in presenting data on performance measure outcome relationships. So, a) he was happy to see our discussion , a more thorough, I suppose, discussion of the issues, but, b), it showed that certainly a stakeholder in OQP was aware of the issue. 

Elizabeth:Great. Next question, are there any PPM metrics that do not have a discrepancy between the patient level and the facility level?

Dr. Finney:I do not think we know the answer to that question. I think because of the absence of a lot of multilevel analyses in the greater prevalence, I think, and I do not really consider myself an expert in this field, but I think there is a greater prevalence of facility-level analyses. So, I do not think we know to what extent. And, in a way, that is the whole point. It is that given the hospital-level analyses, it does not tell us anything about whether or not at the patient level the performance measure specified care is associated with positive outcomes.

Elizabeth:All right. How does this fallacy relate to the identification of outliers in terms of facility performance?

Dr. Finney:Well, I mean, outliers will drive relationships. Again, I cannot think of a reason why outliers at the facility level, again, would tell you anything about what is going on within facilities at the patient level. I do not know if that answers the question or not.

Elizabeth:Okay. The next question is, what is the value of studies of relationships at the hospital level?

Dr. Finney:Well, as I mentioned toward the end, I think what they show is that in many cases, simply focusing on a particular practice or a small set of practices is not enough to elevate hospital outcomes, is not enough to really drive hospital outcomes. And that, to do that one needs a broader perspective, and one needs to think about the pathways to positive outcomes for patients with particular conditions. And, there might be multiple pathways to positive outcomes, but I think those negative, or sorry, the null relationships or weak relationships in many of the hospital-level analyses raise that issue very prominently. So, that is one great value, I think, in those analyses. Again, it is the misinterpretation, it is jumping from no or weak relationships at the hospital level, to saying, “Well, it doesn’t matter whether we do perform this particular practice for my individual patient or not.” That is the ecological fallacy. Ecological analyses are not fallacious, it is the jump from that higher level of analysis to a lower level of analysis and assuming that the same relationship will obtain at the lower level.

Elizabeth:Great. The next question, is it fair to say that when adherents bundles for processes of terror are created, such as the prevention of ventilator-associated pneumonia, contextual factors of the facility level are often ignored?

Dr. Finney:Well, as I mentioned, I am a socio-psychologist by training and I have focused on substance use disorder treatment evaluation for a number of years, so I do not pretend to be an expert in particular medical practices. But, I think, in general that that is a reasonable assumption, that particular bundles of practice could be undermined by something else that is going on in the system, or something else that is going on in the next system of care to which patients are transferred. 

Elizabeth:Okay. Do you think these issues also correspond to facility versus patient level analyses in cost and utilization?

Dr. Finney:That is an interesting question. [Laughs] You know, I have not thought about that and I am certainly not an economist. You know, I think, though, that the, I cannot see why these issues would not apply, that there are issues in aggregation and disaggregation and I do not know why they would not apply.

Elizabeth:Okay. Austin, there’s is concerned about making inferences about causality from patient-level PM analysis, because of confounding by indication in which sicker patients are often more likely to be recommended treatments. Excuse me. Please comment.

Dr. Finney:Can people still see my slides? I assume they can. So, that was actually one issue I was going to talk about, but skipped over because of time. So, Johnston has recommended that, because of confounding by indication, which is the notion that based on their condition, patients may be more or less likely to get the recommended practice and that will distort relationships between the causal impact of the practice and patient outcomes. So, what Johnston recommended was to use the facility performance rate on the practice as kind of a pseudo-instrumental variable, and then he and his colleagues and also Joe Selby and some of his colleagues, when Dr. Selby was at the Division of Research at Kaiser Permanente in Oakland. They have used that approach, and it is an interesting idea in that the basic idea is that some other factors like condition may affect not only receipt or non-receipt of the performance measure specified care, but the outcome. 

And, the idea is with a pseudo-instrumental variable, the ID, which is the proportion of patients receiving care, that one would have to make the assumption like for any instrumental variable that proportion of patients receiving the care is related to the actual receipt of care, probably a reasonable assumption, fairly strong related. And, that the effect of the proportion of patients receiving the performance measure specified care to outcome is only through receipt of the care. And, in other words, that the pseudo-instrumental variable is unrelated to other factors. And, that, I think is the flawed assumption. One cannot test that assumption. 

But, I think it is likely that the proportion of patients receiving care at the facility level might be associated with other facility-level variables. One does not know what they might be. But, to me, it is a situation of substituting confounding by indication, substituting confounding by indication for confounding by the unknown. And, I once heard Hal Sox say – I do not think the notion was original to him – but, that the cure for confounding by indications is to measure the indications better. 

Elizabeth:Great. One last question, do you think that plotting adjusted associations of patients within each site, similar to your unadjusted example and comparing the slopes can offer another way to examine the patterns of associations across sites?

Dr. Finney:Yes, and I have given this talk about six times and I have often wondered if someone would raise that issue. And, I wondered it, because I frankly did not have a really good way and did not want to get into data enough to plot adjusted relationships. So, I do not know to what extent covariant adjusted relationships would show a different picture. I have no reason to think, though, that covariant adjusted relationships would be any less likely to show differences in relationships at the facility and patient levels. So, I do not think the issue is covariant adjustment. I think the issue is you get different relationships at the facility and patient levels. Or, you can get different relationships, and I would say are likely to get different relationships.

Elizabeth:All right. Well, it looks like we are at the top of the hour. John, thank you very much for presenting at today’s HERC Cyberseminar. And, Heidi...

Heidi:Yes, John. Thank you very much taking to put this together and present for us today. We really appreciate it. We are getting a lot of great feedback through the Q&A screen, so the audience really, really was into it. For the audience, I know a lot of you are leaving right now, you have somewhere else to get to, but on your way out you are going to be prompted with a feedback form, if you could take just a couple of seconds to fill that out we would definitely appreciate that. We do read through all of your feedback, so if you could take just a few seconds to do that, we would appreciate it. Our next session is scheduled for June 19 at 2:00PM Eastern. Todd Wagner will be presenting Risk Adjustment for Cost Analyses: The Comparative Effectiveness of Two Systems, and we will be getting further registration information out to everyone on that as we get a little closer to that time. Thank you everyone for joining us for today’s HERC Health Economic Seminar and we hope to see you at a future session. Thank you.

[End of audio]

Page 1 of 13

Page 13 of 13