BENEFIT‐OF‐THE‐DOUBT

APPROACHES FOR CALCULAT

A COMPOSITE MEASURE MEASURE O

QUALITY

By Michael Michael Shwartz, Shwartz, James F.

Burgess, Jr. (Presenting), and Da

Berlowitz

Funded by VA Health Services Research and Development grant IIR 06‐260
Context and Background Background 

• Standard Approaches for Creation of Com

Measures of Quality (Quality Indicators ‐

– Equal Weighting– Prevalence Based Weights– Judgment Based Weights

• Concept Concept of Benefit Benefit‐of‐the‐Doubt Approa

Approa

– Relative Performance represents a Measure 

R ld evea ed P f re erences by the O i ti l rganizational 

Relative Importance
Context and Background Background 

• Distinguish two types of Composite M

– Reflective Measures ( 

manifestations of c

– Formative Measures (defined by individu

• Ill t t h t t F 

Illustrate some approaches to create Fo

Measures from QIs

• QIs are not Highly Correlated and Exp

are Added to Include More QI Dimensare Added to Include More QI Dimens
Benefit‐of‐the‐Doubt Meas

Meas

• Nardo et al. (OECD‐2005) Review of M

• Benefit Benefit‐of‐the‐Doubt Approaches Approaches Rec

Rec

Revealed Preferences w/Higher Weigh

• Ch h erc ye et al. (JO SR ‐200 ) 7 and S 

emp 

(EJOR‐1996) note this is the Natural O

of Nash evaluation game: Regulator v. 

• Mostly used to date to Compare Compare Coun

Coun

(e.g. Lovell (IJPE‐1995), Despotis (JORS
Criticism and Intuition

• If Wei hg ts are O ii rgan zation‐S ifi pec c are

Comparisons Across Units Possible?

– Dropping Lowest Grade Example

– Data Envelopment Analysis (DEA) does this

• If Final Comparisons are made on Relativ

then what happens in practice?

– No one knows in advance who benefits mos

– Actual rankings may not change much

– Dropping or downweighting lower scores m

good will from the organization/student at 
Purpose Statement

• Imagine we have a fi d xe set of QIs wi ht a

reporting period just ended

• Goals for the Regulator might be:

– Facilitating consumer choice with gestalt va– Pay‐for‐Performance to reward high perform

– Quality improvement learning to spread val

• Comparative Approaches

– DEA (here all QIs are reported on the same – Simple LP Optimizing subject to constraint t

weights sum to 1 (needs QIs on the same sc
Example: VA Nursing Nursing Homes 

• 35 Nursing Homes in VA ( l Ber owitz et • Five QIs Reflecting Patient Change Ove

– Pressure Ulcer Development– Functional Decline– Behavioral Decline– Mortality– Preventable Hospitalization

• All QIs are Ri ks Adj d uste w/P bli h d ublished 

• 32 Nursing Homes with no Missing Da
Calculating the QIs

• Many ways can be used to calculate a 

of importance in this example

• Model generates Predicted Probability 

month adverse adverse event given initial initial risk

• Add up observed adverse events (O)• Add up predicted probabilities (E)• We create O/E R ti a os whi hc are wid l e 
Comparisons of Composit

• Equal Weights Model

• Facility‐Specific Prevalence Weights Mod

• Overall Prevalence‐Based Weights Mode• Si l mp e LP Mdl o e ( i ht we g s sum to 1)• Weight Constrained DEA Model

– Employ Rachel Allen/Thanassoulis Constrain

of the Weights Measure

– This does not permit some QIs to drop weig

zero (the student drop the lowest grade mo
Table 2: Composite scores and facility ranks for high and low ranked facilities

Composite Score Facility R

facilityspecific

prevalencefacility- specific prevalence- overall prevalence- overall prevalence equal simple LP prevalence- 

based weights

prevalence

based weights

prevalence

based weights

prevalence

based weights

equal

weights

simple LP

model*

facility DEA* 

10 0.576 0.495 0.509 0.351 1.000 1 1 

6 0.654 0.530 0.495 0.451 1.000 2 2 

28 0 662 . 0 875 . 0 445 . 0 412 . 1 000 . 3 7 19 0.754 0.790 0.755 0.596 1.000 4 4 

8 0.762 0.957 0.599 0.479 1.000 5 15 

24 0.779 0.848 0.621 0.636 0.969 6 6 11 0.805 0.782 0.917 0.680 1.000 7 3 17 0.857 0.846 0.728 0.566 0.990 9 5 

4 0.997 1.002 0.914 0.868 0.900 21 20 

15 1 035 . 1 037 . 1 190 . 0 936 . 0 939 . 23 23 16 1.047 1.056 1.091 0.951 0.895 24 24 13 1.086 1.086 1.198 1.001 0.903 26 26 31 1.118 1.134 0.958 1.023 0.881 27 27 30 1.150 1.402 1.236 0.855 0.952 28 30 32 1.158 1.323 1.162 0.858 0.930 29 29 18 1.272 1.303 1.321 1.187 0.837 30 28 20 1.385 1.556 1.383 1.107 0.889 31 32 

2 1.443 1.520 1.717 0.923 0.949 32 31 

*: results for the benefit-of-the-doubt approaches are for allowable weight adjustments of + 0.75 of overall prevalence-based weights
Weight Constrained Models T

• We Test Differences in Ranks/Correlation

• Levels: Allowable Weight Adjustments

• 1: + 0.25*overall prevalence‐based w• 2: + 0 50* . overall prevalence‐b d ase 

• 3: + 0.75*overall prevalence‐based w

• 4: + 0.90*overall prevalence‐based w• 5: no constraints constraints
Figure 1: Comparison of ranks using overall prevale

wei

g g hts to ranks usin

g each of the benefit-of-the-dou

approaches with different amounts of allowable weigadjustments (previous slide)P t A A diff i k Par

t

A: 

Average difference in ran

k

s

7.005 006.00

anks

4.005.00

erence in Ra 

2.003.00

verage Diffe

0.001.00

Av

12345

Allow able Weight Adjustments
Part B: Correlation

1.201.000.80

tion

0 400.60

Correla

0.200.400.00

1 2

3

4

Allowable Weight Adjustments
Outcomes Outcomes of Benefit Benefit‐of‐the‐

• There is no gold stand d ar for wei hi g t ng

• But “equal weighting” is a choice and may ge

“these wei ht g s do not refl t ec wh t a is i t

mpor a

patients”

• Face validity? validity? A moving concept? concept?• Post Hoc Discussion of Weights can only be S• B t u if true preferences are refl t d ec e in perfor

this approach should lessen tensions and imp

and engagement

• No Need to Blame the Messenger!
Other Outcomes and Bene

Bene

• We know Ri ks Adj t t Adjustment is i ft mper ect, s

adjustment is made

• Using Weight Constraints Constraints in DEA Allows

Policymakers to Choose how far to go

– We used simple constraints constraints but others poss

poss

• DEA has been used before and has favor

properties properties (Nash outcome outcome, flexible flexible to sco

sco

• DEA also has negatives (best with large a

of data to set benchmarks) benchmarks)

• Simple LP must be normalized but may b

transparent than DEA – Simp y licit a Virtu
Incentive Effects and Gam

• If a organization performs simil l ar y on 

gains no value from the approach

– Unless scoring “high” relative performan

• Managers will focus on QIs where the

improve & which are most important 

• P4P Programs Programs now leaning leaning toward rew

rew

attainment and improvement (to bala

incentives) incentives), this method can combine combine 

regulator weights between them for t
Limitations and Improvem

• Simple O/E ratios can be improved up

– (O‐E)/variance (O) or z‐scores– Hierarchical modeling results (Bayesian o

• More data, more measures, measures, more rece

more data over time all can be incorp

• CMS Nursing Nursing Home Compare Compare has a re

re

complex algorithm while CMS Hospita

Compare currently currently using simpler simpler meth

meth

– Concept of “Five Star” systems
Final Thoughts

• Explosion of Quality Measures ( ) QIs in 

years

• Measurement of Composites is going 

continue to be debated

• Inherent limitations (safety net faciliti

incomplete risk adjustment) adjustment) support support f

f

to generate trust and buy‐in

• B fit ene ‐of‐the‐D bt ou Measures sh l

ou 

of the discussion