Are Misinterpreted, Hospital-level 

Relationships Between Process Performance 

Measures and Outcomes Undermining 

Evidence-based Patient Care?

John W. Finney, PhD

HSR&D Center for Health Care Evaluation

VA Palo Alto Health Care System, Palo Alto, CA

May 22, 2013

1 
Acknowledgements

Supported by the Department of Veterans Affairs Office 

of Research and Development’s Health Services 

Research and Development Service (HSR&D) Grants 

SUS 99-015 and IIR-07-092-1, and by the HSR&D 

Substance Use Disorders Quality Enhancement 

Research Initiative.

Views are those of the presenter and do not 

necessarily represent those of the Department of 

Veterans Affairs or any other entity of the U. S. 

Government.

2 
Finney JW, Humphreys K, Kivlahan DR, 

and Harris AHS. 2011. Why process 

performance measure-outcome 

relationships can differ at the patient and 

facility levels: Understanding the 

“ecological fallacy.” American Journal of 

Public Health 101: 1635-1642.

Based on:

3 
Outline of Presentation

• Background: The “Ecological Fallacy”

• Introduction

• Continuing Care PM for Patients with Substance Use 

Disorders – Illustrative Data

• How Differences Occur Between Facility- and PatientLevel PPM-Outcome Relationships

• Why Differences Occur Between Facility- and PatientLevel PPM-Outcome Relationships

• Implications for Health Care Quality Management and 

Research
• Performance measurement – one factor pointed to as 

driving substantial improvement in health care

• Many performance measures (PMs) assess the extent 

to which processes of care, that have been shown to 

“cause” or at least to relate to positive outcomes 

among research participants, are applied in a health 

care facility (process performance measures - PPMs)

• Used to rate health care providers, facilities (e.g., 

hospitals), health care systems, states, and countries 

on quality of care

Introduction

5 
6 
Introduction

7 

Process measures of health care quality 

typically are implemented on the 

assumption that:

• Patient-level care processes 

aggregated to the facility level are 

associated with positive facility-level 

(aggregated) outcomes in the same 

way as was found at the patient-level 

in supporting research. 
Introduction

Facilities: PM Care Rate Outcome Rate

Patients: PM Care Y/N Outcome
Introduction

9 

An example of the “homology” 

assumption – the assumption 

that things work the same at 

different levels of organization 

or analysis (Hannon, 1970)
Introduction

Linking facility performance rates to 

facility outcomes is thought of as one 

way to validate process of care 

performance measures or as a form 

of “post-implementation surveillance”:

But, investigators examining facilitylevel PPM-outcome linkages 

sometimes have found weak or nonexistent relationships. 10
11
Bradley et al. (2006)

mortality 

rate −0.04 0.07 −0.12 0.10 0.06 −0.01 −0.04 0.02

30-day,

all-cause 

risk-adj.

mortality 

rate −0.03 −0.16† −0.06 −0.18† −0.10 −0.03 −0.18† −0.25†

In-hospital,

all-cause 

risk-adj.

_______________________________________________________________________________

 

 Beta Beta Timely 

 Blocker Blocker Aspirin Aspirin ACE Inhib Smoking Reper- Composite

 Admit Dischg. Admit Dischg. Admit Cessation fusion

 Therapy

_______________________________________________________________________________

Centers for Medicare & Medicaid Services (CMS) and Joint Commission on 

Accreditation of Healthcare Organizations “Core” Processes for AMI

12

______________________________________________________________________ 

†p<.001

N=962 hospitals
13
Werner and Bradlow (2006)

Absolute Reduction in Risk-adjusted Mortality Between Hospitals

Performing in the 75th versus 25th Percentiles (N=3,657 Acute Care Hospitals)

________________________________________________________________

CMS Composite

Performance Inpatient 30-day 1-year

Measure for: Mortality Mortality Mortality

 __________________________________________

Myocardial 0.005 (p=.003) 0.007 (p=.002) 0.012 (p<.001)

Infarction

Heart Failure 0.001 (p=.08) 0.001 (p=.14) 0.002 (p=.23)

Pneumonia 0.005 (p=.01) 0.003 (p=.15) 0.007 (p=.02) 

______________________________________________________________________ 

(Italicized from a correction in JAMA, 2007, 297(7), p. 700.)

14
15
Nicholas et al. (2010)

CMS Surgical Care Improvement Project (SCIP) data

for 2,000 U.S. hospitals: 

Odds of Risk-Adjusted Surgical Mortality Rate in High and Low 

SCIP Compliant Hospitals, 2005-2006

Overall Contemporaneous Lagged

Highest quintile quality 0.98 (0.92-1.05) 0.96 (0.86-1.06)

Lowest quintile quality 1.06 (0.97-1.16) 1.09 (0.97-1.22)

16

Nicholas, LH, Osborne NH, Birkmeyer JD, and Dimick JB. 2010. 

Hospital process compliance and surgical outcomes in Medicare 

beneficiaries. Archives of Surgery 145: 999-1004. 
Bradley et al. (2006) and Werner and Bradlow

(2006) noted a variety of factors that might have 

reduced relationships between facility-level PPM 

rates and facility-level mortality rates: 

Reactions to Findings

E.g., high level of compliance (restricted 

range) for some practices (e.g., aspirin at 

admission for AMI).

Insensitivity of in-hospital and 30-day 

mortality as outcomes.

17
Reactions to Findings

Mabry (2010), commenting on Nicholas et al. 

(2010):

“These findings, if true, call into serious question 

the increased time, labor, and effort currently 

expended by hospitals and surgeons across the 

United States to comply with the SCIP program 

process measures. . . . How can it be that the 

National Quality Forum, CMS, and others got it 

wrong?’” (p. 1104).

Mabry CD. 2010. “Say it ain’t so, Joe.” Archives of Surgery, 145(10), 1004-1005.

18
Reactions to Findings

Surgeons in the VA and at Stanford 

University have questioned why they 

should be held accountable to 

surgical practice performance 

measures that are not related to 

patient outcomes.

19
“Efforts should be made to develop 

performance measures that are tightly linked to 

patient outcomes” (Werner & Bradlow, 2006, p. 

2,694).

Reactions to Findings

But neither they, nor any of the authors cited 

earlier, raised the possibility that PM-specified 

practices might relate differently to outcomes 

at the patient level than at the hospital level.

Werner RM, and Bradlow ET. 2006. Relationship between Medicare’s Hospital Compare 

performance measures and mortality rates. JAMA 296(22): 2694-2702. 

20
Provide a non-technical overview of relevant 

multilevel issues that would be accessible to a 

wide range of stakeholders:

• Clinicians

• Researchers

• Quality managers

• Health care leaders

• Members of performance measure bodies

Aim of AJPH Article

Drew on Alex’s statistical expertise and on the 

relevant epidemiological, sociological and 

statistical literature.

21
Background: The “Ecological Fallacy”

1930 Census Data

State level: Proportion foreign-born state 

residents correlated -.53 with state English 

illiteracy rate

Individual level: Correlation +.12

Conclusion: Cannot infer relationships for 

individuals from relationships for higher-level 

units. 

Robinson WS. 1950. Ecological correlations and the behavior of 

individuals. American Sociological Review 15:351-357. 
More recent considerations of this issue (e.g., Firebaugh 

1978; Greenland 2001; 2002; Greenland and 

Morgenstern 1982; Morgenstern 1982; Schwartz 1994; 

Subramanian et al. 2009; Susser 1994):

• Have been more even-handed: addressing problems 

of inference in moving from lower- to higher-level 

units, as well as from higher- to lower-level units

• Addressed the incompleteness of many single-level 

analyses

• Stressed the need for multilevel analyses (e.g., 

Duncan et al., 1998; Greenland 2002; Localio et al., 

2001)

Later Methodological Literature

23
Continuing Care PM for Patients with Substance

Use Disorders: Illustrative Data

To meet the PPM, a patient must have 

had at least 2 SUD clinic visits in each 

of three consecutive 30-day periods 

following a “qualifying” discharge or 

visit.

24

Harris, AHS, Humphreys K, Bowe T, Kivlahan D, and Finney JW. 2009. 

Measuring the quality of substance use disorder treatment: Evaluating the 

validity of the VHA continuity of care performance measure. Journal of 

Substance Abuse Treatment 36: 294-305. 
Continuing Care PM for Patients with Substance

Use Disorders: Illustrative Data

For simplicity, we:

• Randomly selected one of five datasets with 

imputed missing data that were averaged in Harris 

et al. (2009)

• Ignored the fact that at some facilities’ patients were 

drawn from more than one SUD program, and at 

other facilities patients were not sampled from all 

the SUD programs

• Ignored providers and programs/clinics as 

intervening levels

• Did not control for covariates (will discuss 

confounding later)

25
• Subgroup of 1,485 patients in VHA SUD treatment 

identified by Harris et al. (2009) as non-abstinent (alcohol 

or drugs) in the 30 days prior to a baseline assessment – 

selected for a reason.

• Subgroup from non-methadone SUD programs at 72 

facilities. 

• Followed-up an average of 7.3 months later, at which point 

abstinence for the past 30 days was assessed. 

• Follow-up rate was a little over 65%.

• Outcome data imputed for those individuals not followed 

(see Harris et al., 2009, for more details on the sample and 

data collection and imputation methods). 

Continuing Care PM for Patients with Substance

Use Disorders: Illustrative Data

26
Continuing Care PM for Patients with Substance

Use Disorders: Illustrative Data

Ran three analyses to examine CC 

PM-abstinence relationships:

1. Facility-level analysis

2. Mixed-effects analysis with facility 

as random factor

3. Multilevel analysis

27
Variable Regression Standard Significance 

 Coefficient Error Level

 ________________________________

Intercept .447 .023 <.001

Rate of CC .045 .065 .489 (ns)

______________________________________________

Analysis 1: Regression Analysis Linking Facility CC 

Performance Rate to Facility Abstinence Rate 

(N=72 Facilities)

28
Variable Regression Standard Significance OR CI

 Coefficient Error Level

 ________________________________________________

Intercept -.323 .063 <.001

CC (Y/N) .558 .115 <.001 1.75 1.40-2.19 

________________________________________________________________

Analysis 2: Mixed-effects Patient-level Analysis 

with Facility as a Random Factor (N=1,485 Patients)

29
 Variable Regression Standard Significance OR CI

 Coefficient Error Level

 _______________________________________________

 Intercept -.193 .100 .055

 CC (Y/N) 

 Rate of CC -.509 .305 <.100 .60 .33-1.09

____________________________________________________________

Analysis 3: Multilevel Analysis of Between- and 

Within-facility (Patient-level) Relationships

30
 Variable Regression Standard Significance OR CI

 Coefficient Error Level

 _______________________________________________

 Intercept -.193 .100 .055

 CC (Y/N) .647 .127 <.001 1.91 1.49-2.45

 Rate of CC -.509 .305 <.100 .60 .33-1.09

____________________________________________________________

Analysis 3: Multilevel Analysis of Between- and 

Within-facility (Patient-level) Relationships

31
How Do Differences Occur Between 

Facility- and Patient-Level PPM-Outcome 

Relationships?

32

(Greenland & Morgenstern, 1989; Greenland, 2001)
 Facility A Facility B Facility C

_______________________________________________________________________

 

 CC PM Rate 33% 52% 72%

 

Three Hypothetical Facilities with the Different CC PM 

Rates and the Same Abstinence Rate 

Abstinence 25% 25% 25% 

 Rate

34
Zero Facility-level Relationship of CC PM to Abstinence 

for Three Hypothetical Facilities

0 

5 

10

15

20

25

30

35

40

45

50

No PM PM

Percent Abstinent

Percent Receiving PM CC

35
 Facility A Facility B Facility C

_______________________________________________________________________

 

 Abstinence Abstinence Abstinence

 Y N Y N Y N 

Y 33% 52% 72%

 PM 

N 67% 48% 28%

 

 25% 75% 100% 25% 75% 100% 25% 75% 100%

 ______________________________________________________________________________

Three Hypothetical Facilities with the Different CC PM 

Rates and the Same Abstinence Rate - 

Marginal Distributions

36
 Facility A Facility B Facility C

_______________________________________________________________________

 

 Abstinence Abstinence Abstinence

 Y N Y N Y N 

Y --- --- 33% --- --- 52% --- --- 72%

 PM 

N --- --- 67% --- --- 48% --- --- 28%

 

 25% 75% 100% 25% 75% 100% 25% 75% 100%

 ______________________________________________________________________________

Three Hypothetical Facilities with the Different CC PM 

Rates and the Same Abstinence Rate - 

Marginal and Joint Distributions

37
 Facility A Facility B Facility C

_______________________________________________________________________

 

 Abstinence Abstinence Abstinence

 Y N Y N Y N 

Y --- --- 66 --- --- 104 --- --- 144

 PM 

N --- --- 134 --- --- 96 --- --- 56

 

 50 150 200 50 150 200 50 150 200

 ______________________________________________________________________________

Three Hypothetical Facilities with the Different CC PM 

Rates and the Same Abstinence Rate - 

Marginal and Joint Distributions

38
 Facility A Facility B Facility C

_______________________________________________________________________

 

 Abstinence Abstinence Abstinence

 Y N Y N Y N 

Y 26 40 66 36 68 104 44 100 144

 PM 

N 24 110 134 14 82 96 6 50 56

 

 50 150 200 50 150 200 50 150 200

 ______________________________________________________________________________

Three Hypothetical Facilities with the Different CC PM 

Rates and the Same Abstinence Rate - 

Marginal and Joint Distributions

39
Zero Facility-level Relationship of CC PM to Abstinence with Similar Positive 

PPM-Abstinence Relationships for Patients Within Three Hypothetical Facilities

40
Zero Facility-level Relationship of CoC PM to Abstinence with Similar Positive 

PPM-Abstinence Relationships for Patients Within Three Hypothetical Facilities
 Facility A Facility B Facility C

_______________________________________________________________________

 

 Abstinence Abstinence Abstinence

 Y N Y N Y N 

Y --- --- 66 --- --- 104 --- --- 144

 PM 

N --- --- 134 --- --- 96 --- --- 56

 

 50 150 200 50 150 200 50 150 200

 ______________________________________________________________________________

Three Hypothetical Facilities with the Different CC PM 

Rates and the Same Abstinence Rate - 

Marginal and Joint Distributions

42
 Facility A Facility B Facility C

_______________________________________________________________________

 

 Abstinence Abstinence Abstinence

 Y N Y N Y N 

Y 8 58 66 16 88 104 28 116 144

 PM 

N 42 92 134 34 62 96 22 34 56

 

 50 150 200 50 150 200 50 150 200

 ______________________________________________________________________________

Three Hypothetical Facilities with the Different CC PM 

Rates and the Same Abstinence Rate - 

Marginal and Joint Distributions

43
Zero Facility-level Relationship of CC PM to Abstinence with Similar Positive 

PPM-Abstinence Relationships for Patients Within Three Hypothetical Facilities

44

0 

5 

10

15

20

25

30

35

40

45

No PM PM

Percent Abstinent

Percent Receiving PM CoC

A

B

C

Means
Zero Facility-level Relationship of CC PM to Abstinence with Similar Positive 

PPM-Abstinence Relationships for Patients Within Three Hypothetical Facilities

45

0 

5 

10

15

20

25

30

35

40

45

No PM PM

Percent Abstinent

Percent Receiving PM CoC

A

B

C

Means
Why Do Differences Occur Between 

Facility- and Patient-Level PPM-Outcome 

Relationships?

46

• Effect modification

• Differential effects of measurement error, 

 especially in the “exposure” variable
Why Do Differences Occur Between 

Facility- and Patient-Level PPM-Outcome 

Relationships?

Different confounding variables at patient 

and facility levels (Morgenstern, 1982; 

Greenland and Morgenstern, 1989)

47
Different Confounding Variables

Confounder Z Rate

Facility: PM Care Rate Outcome Rate

Patient: PM Care Outcome

Confounder Z
Different Confounding Variables

Effects of aggregated confounders can 

be disaggregated into their betweenand within-facility (patient-level) effects 

(Neuhaus & Kalbfleisch 1998).

Neuhaus JM, and Kalbfleisch JD. 1998. Between- and within-cluster 

covariate effects in the analysis of clustered data. Biometrics 54: 638-645. 

49
Grouping data or only collecting data at the health 

care facility level may allow new variables to affect 

the relationship between a PPM and outcomes - i.e., 

variables that may not come into play or have 

different “effects” on the PPM-outcome relationship at 

the patient level. 

Blalock (1964): “The key to the problem may come 

with the realization that in shifting units we may be 

affecting the degree to which other unknown or 

unmeasured variables are influencing the picture” (p. 

99). 

“Integral” or “Structural” Properties of 

Higher-level Units

50
Higher level units, such as health care facilities, have 

“integral” properties (Susser, 1994) – not aggregated 

characteristics of the individuals comprising them. 

E.g., different health care facilities have different 

leaders, local policies, and structural properties, such 

as staff-patient ratios, and may operate in different 

environmental contexts. 

“Integral” or “Structural” Properties of 

Higher-level Units

These factors may confound facility-level relationships 

between PM performance and outcomes, but could 

have quite different impacts or no impact on patientlevel PM-outcome relationships. 
The fact that different variables may or 

may not relate to “PPMs” at the patient 

versus the hospital level implies that:

Implications of Different Confounders 

at Different Levels

A performance measure at the 

hospital level may be assessing 

something quite different than 

the performance measure at the 

patient level.
This “different constructs at different levels of 

aggregation/analysis” argument has been 

made by many observers (e.g., Firebaugh, 

1978; Glick, 1980).

Implications of Different Confounders 

at Different Levels

E.g., state rate of English illiteracy among 

foreign-born residents turned out to be a proxy 

for, or confounded with, the literacy rate among 

native-born state residents (Robinson, 1950).
Lack of Confounding

What PPMs are not confounded with 

may be as important or more important 

as the factors with which they are 

confounded.

Part of the confidence in hospital-level 

PPM-outcome analyses stems from a 

belief that various indicators of quality 

are positively correlated.

54
Lack of Confounding

“[H]ospital mortality rates, even riskstandardized, are likely influenced by 

many factors that are independent of the

core measures, including processes that 

involve patient safety, staffing, response

to emergencies, and clinical strategies 

that may contribute to a hospital’s 

outcome performance” (Bradley et al., 

2006, p. 77).

55
Lack of Confounding

Isaac and Jha (2008) found 

“inconsistent and usually poor 

associations among the PSIs 

[Patient Safety Indicators] and other 

hospital quality measures . . . .”

Isaac T, and AK Jha. 2008. Are patient safety indicators related to widely 

used measures of hospital quality? Journal of General Internal Medicine

23(9): 1373-1378.

56
1. Quality managers who only have findings on 

facility-level PPM-outcome relationships 

should view those relationships with caution. 

Possible that they reflect the PPM-outcome 

relationships at the patient level within 

facilities, but likely that they do not.

Facility-level PPM-outcome relationship may 

even be in the opposite direction of the 

relationship at the (within-facility) patient level. 

Implications for Quality Management

57
Piantadosi (1994) concluded that with 

aggregated data:

Implications for Quality Management

58

“[W]e not only lose all ability to extend 

inferences reliably to less aggregated data 

but we even lose the ability to estimate the 

direction and magnitude of bias. We cannot 

rely on the addition of more grouped data to 

eliminate the bias” (p. 763).
In addition to Robinson’s (1950) findings, 

Greenland and Robins (1991) note that ecologic 

analyses have been conducted that, if taken at 

face value, would support the conclusions that:

Implications for Quality Management

• Radon exposure has a protective effect for 

lung cancer (Cohen, 1990; 1,729 counties in 

the U.S). 

• Cigarette smoking has a protective effect for 

oesophageal cancer (Richardson et al. 1987; 

22 administrative regions [départements] in 

France). 
Quality managers should keep in 

mind Naylor’s (1999) 

recommendation concerning 

ecologic analyses of treatment 

effects: 

Implications for Quality Management

60
Quality managers should keep in 

mind Naylor’s (1999) 

recommendation concerning 

ecologic analyses of treatment 

effects: “caveat emptor.”

Implications for Quality Management

61
Implications for Quality Management

62

2. If a PPM is associated with positive 

outcomes at the patient level, but not 

the facility level, health care system 

leaders should encourage the use of 

that practice through facility-level 

performance measures. 
Implications for Quality Management

63

3. If a PPM is associated with positive 

outcomes at the patient level, but not 

the facility level, health care system 

leaders should try to determine what, 

at the facility level, is “canceling out” 

the patient-level “effect” of the PMspecified care. 
Implications for Quality Management

On the other hand:

The proportion of facility patients 

meeting the PPM may be positively 

related to desired outcomes, with a 

negative or no relationship between 

patient receipt of the PM-specified 

care and outcome (e.g., Berlin et al., 

1999). 

64
Implications for Quality Management

E.g., within hospitals, patients who receive a 

particular surgical procedure might be more likely 

to die. 

However, facilities at which that surgical procedure 

is performed for higher percentages of patients 

might also tend to provide better infection control 

or send patients to safer extended care facilities. 

If so, the challenge would be to ensure a safer 

surgical procedure at all facilities, while 

preserving or enhancing infection control and 

patient safety measures. 
Implications for Quality Management

Overall, “false negative” hospitallevel PPM-outcome findings are of 

more concern than “false positive” 

results, if patient-level evidence 

from RCTs and other sources 

supports the PPM. 

66
Implications for Quality Management

Multilevel findings provide more useful, 

but also more nuanced, information to 

quality managers on how outcomes 

might be improved with interventions 

targeting patients and their care, as well 

as the health care facility and, perhaps, 

the broader community context.

67
Broader Conceptual Model

Community Context

PM+PM+PM 

Care

Hospital Context

Outcome

Patient

Factors
Implications for Research on Performance Measures

Two questions capture the inherently multilevel 

nature of PPMs and their relationships to clinical 

outcomes: 

1. Independent of the proportion of patients for 

whom PM-specified care is provided at their 

hospitals, is the PM-specified care linked to 

better patient outcomes within hospitals? 

2. Independent of the relationship of PM-specified 

care to outcome for patients within hospitals, is 

the proportion of patients receiving the PMspecified care across facilities linked to better 

outcomes?

69
Implications for Research on Performance Measures

The following types of analyses do not address 

those questions: Analyses of:

• The facility-level relationship between a PPM and 

outcome.

• The relationship between a PPM and outcome across 

all patients in the system ignoring the facility in which 

they received care.

• The patient-level relationship between a PPM and 

outcome controlling for clustering of patients (on 

outcomes) within facilities.

Why not? Because findings reflect a mixture of 

between- and within-facility relationships. 
Implications for Research on Performance Measures

Firebaugh (2009) notes: “[S]ingle-level 

analyses are subject to severe omittedvariable bias in the presence of multilevel 

effects” (p. 369). 

Strong implication for researchers:

• Studies of performance measures or 

quality indicators should be guided by 

multilevel conceptual models and use 

multilevel analyses to examine them 

whenever possible (Duncan et al., 1998; 

Localio et al., 2001).

71
Varied Within-facility Relationships of PPM to Abstinence with Flat Facility-level 

PM-Abstinence Relationship for Patients Within Three Hypothetical Facilities

0 

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

No PM PM

Abstinence

% CC Care

A

B

C

Mean 

72
But . . . .

Why not use facility-level 

performance as a “pseudoinstrumental variable” in examining 

relationships between PM-specified 

practices and patient outcomes to 

counteract the confounding by 

indication that can occur at the 

patient level?

73
Performance Rate as an Instrumental Variable

Johnston SC. 2000. Combining ecological and individual 

variables to reduce confounding by indication: Case study – 

subarachnoid hemorrhage treatment. Journal of Clinical 

Epidemiology 53: 1236-1241.

Johnston SC, Henneman T, McCulloch CE, and van der Laan M. 

2002. Modeling treatment effects on binary outcomes with 

grouped-treatment variables and individual covariates. American 

Journal of Epidemiology 156 (8): 753-60.

Selby JV., Uratsu CS, Fireman B, Schmittdiel JA, Peng T, 

Rodondi N, Karter AJ, and Kerr EA. 2009. Treatment 

intensification and risk factor control: Toward more clinically 

relevant quality measures. Medical Care 47(40): 395-402.

74
PM 

Care

Outcome

Other

Factors

Performance Rate as an Instrumental Variable

75
IV

Proportion 

Patients 

Receiving

PM Care

PM 

Care

Outcome

Other

Factors

Performance Rate as an Instrumental Variable

i 

ii

iii

76
IV

Proportion 

Patients 

Receiving

PM Care

PM 

Care

Outcome

Other

Factors?

Performance Rate as an Instrumental Variable

i 

ii

iii

77
Implications for Quality Management Organizations

Health care quality organizations (e.g., CMS, 

Hospital Quality Alliance) should consider: 

• Making de-identified patient data on both 

processes of care and outcomes 

(assuming a sufficiently large N within 

hospitals) available to researchers so that 

multilevel relationships could be examined 

and reported;

• Or conducting and reporting multilevel 

analyses of process performance measureoutcome relationships themselves.

78
Hospital N

Medical 

Records 

Abstract

Compute 

Measures

Patient 

Level 

Datastore

Vendor 

Computes 

Measures

Individual 

Patient 

Data Hospital

Compare

Joint 

Commission

Quality

Check

Hospital 1

• Extract Patient 

Records

• Transform to 

“Standard”

• Lead

National Goals/

Measure Framework

Measure Development

& Maintenance

Operational 

Data Store

Measures 

Data Store

•

•

•

Results

PatientLevel

Data

Vendor 1

Joint 

Commission 

Vendor N

•

•

•

CMSJCAHO

National 

Data Store

Audit / 

Validation Quality 

Assurance

Aggregate 

& PatientLevel

Data

Direct Data Submission

Collect &

Transmit Data

Aggregate Data & 

Calculate Results

Distribute

Results

NQF Endorsement

HQA Measure Selection

Measure 

Implementation

Audit / Validation

Quality Measurement Enterprise

http://www.hospitalqualityalliance.org/hospitalqualityalliance/resources/resources_page.html 79
Conclusion

Misinterpreted hospital-level 

analyses of relationships between 

process performance measures 

and clinical outcomes can 

undermine evidence-based patient 

care. 

80
Conclusion

Consideration of multilevel conceptual 

and methodological issues reinforces 

the wisdom of Donabedian’s (1980) 

early call to focus on a [multilevel] 

system of structure, process, and 

outcome performance indicators.

81
Questions? 

John.Finney@va.gov

82