# Presentation: 489238

## Massachusetts Quality e-Measure Validation Study(MQeVS)

- Eric C. Schneider, MD, MSc
- Brigham and Women’s Hospital
- Harvard School of Public Health

- Sponsor: AHRQ (R18 HS017048)

## Performance Measurement: Fundamental to Improvement

- Performance
- Visibility

- Performance
- Rewards

- Peers

- Patients

- Public

- Purchasers

- Performance
- Feedback

- Market Share

- Payments &
- Penalties

- “Report Cards”

- “P4P”

## Quality Measurement and Reporting: 1997

- Limited public demand
- Few standardized quality measures
- Few organizations engaged
- Few physicians aware
- Public disclosure rare
- Few patients aware

## 2007: Dramatic Increase in Standardized Measures

- Standard-setting organizations
- National Quality Forum (NQF)
- Ambulatory Quality Alliance (AQA)
- Engagement of Medical Profession
- Physician Consortium for Performance Improvement
- AHRQ’s National Quality Measures Clearinghouse: (www.qualitymeasures.ahrq.gov)
- _[Access](http://www.qualitymeasures.ahrq.gov/browse/processofcare.aspx?num=20&type=8)_ - 22 measures
- _[Outcome](http://www.qualitymeasures.ahrq.gov/browse/processofcare.aspx?num=20&type=7)_ - 204 measures
- _[Patient Experience](http://www.qualitymeasures.ahrq.gov/browse/processofcare.aspx?num=20&type=9)_ - 298 measures
- _[Population Health](http://www.qualitymeasures.ahrq.gov/browse/processofcare.aspx?num=20&type=838)_ - 33 measures
- _[Process](http://www.qualitymeasures.ahrq.gov/browse/processofcare.aspx?num=20&type=6)_ - 636 measures
- _[Structure](http://www.qualitymeasures.ahrq.gov/browse/processofcare.aspx?num=20&type=734)_ - 44 measures
- _[Use of Services](http://www.qualitymeasures.ahrq.gov/browse/processofcare.aspx?num=20&type=837)_ - 33 measures

## Implementation of Measurement and Reporting Remains Controversial

- Measurement programs based on
- Administrative data
- Hybrid method (supplemental med record review)
- Patient/enrollee surveys
- Skepticism about validity of performance results
- Burdensome, costly data collection

## Colorectal Cancer Screening: Results by Method

- Schneider et al, Under Review

## Envisioning the EHR forPerformance Measurement

- Detailed, structured clinical data
- Unobtrusive data collection
- Performance data aggregated across care settings to enable sophisticated measures (e.g. care coordination, safety)
- Performance results at physician group rather than health plan level

- Schneider et al, Enhancing performance
- measurement: NCQA’s Roadmap for a Health
- Information Framework.  JAMA 1999;282:1184

## Studies of Performance Measurement Using EHR and HIE

- Single institution
- Single care setting
- Single IT platform
- Limited number of performance measures
- Local, rather than national standards

## Massachusetts e-Health Collaborative (MAeHC)

- 2004 Demonstration Project
- $50 million from Blue Cross Blue Shield of MA
- Universal EHR adoption in 3 MA communities
- Intra- and inter-community data exchange (HIE)
- Integrated performance measurement/reporting
- Massachusetts Health Quality Partners (MHQP)
- AQA ambulatory performance measure starter set (26 measures)
- Quality Data Warehouse
- Receives HIE data as needed to calculate measure results

## Courtesy of Micky Tripathi

## MA Quality e-Measure Validation Study (MQeVS)

**To compare a quality measurement method using structured, coded EHR data with...**
**1) a “hybrid method” involving a combination of aggregated claims data and medical record review.**
**2) a “claims-only method” based on a novel database that aggregates claims data from commercial health plans and Medicare.**

## MQeVS Highlights

- Implementation
- MAeHC Pilot (www.maehc.org)
- MHQP (_[www.mhqp.org](http://www.mhqp.org/)_)
- Evaluation
- Community ambulatory practices “similar to” U.S.
- Broad clinical and research expertise among research partners
- Harvard School of Public Health
- MHQP
- Partners Healthcare
- Harvard Medical School
- Center for Survey Research (U Mass, Boston)

## MQeVS Sample

- Two Specific Aims assure that study addresses broad populations and measures
- Aim 1: 900 patients with EHR-HIE data, patient survey, medical record review, health plan administrative data
- Aim 2: All “measure eligible” patients with EHR-HIE data and health plan administrative data

## Data Collection Protocol: Privacy/Confidentiality

- Analysis
- Team

- Survey
- Team
- (CSR)

- Patients

- Med Record
- Review Team
- (Partners)

- EHR
-  VENDORS

- Physician
- Offices

- CSC/
- QDW

**Medical Record Copies**

**Pre-notification**

**Re-identified**
**Study Cohort:**
**Pt Contact Info**

**Study Consent, Survey, and Med Record Review Consent**

**Medical Record Requests**

**Medical Record Consent Cohort (Study IDs)**

**Completed **
**Surveys and **
**Consents**

**Med Record Extract (Aim 1)**

**Survey **
**Data Extract**
**(Aim 1)**

**Opt-out**
**Step**

**EHR Quality Measure Extracts (Aims 1 & 2)**

**De-identified Study Cohort List (Aim 1)**

- MHQP

**Claims Data Extract (Aims 1&2)**

**EHR Data**

- Mass E-Health
- Collaborative

**#1**

**#2**

**#9**

**#10**

**#8**

**#5**

**#7**

**#6**

**#3**

**#4**

## Analysis: Availability of Data for Measure Components

- Five key steps comprise a quality measure calculation algorithm, determining whether a patient meets criteria ...
- 1. for inclusion in the preliminary denominator
- 2. to be excluded from the preliminary denominator
- 3. for membership in the final denominator (after exclusions)
- 4. for membership in the measure numerator
- 5. for selection for both the numerator and denominator

## Quality Measures: Deconstructing Data NeedsE=exclusion criteria; D=denominator inclusion; N=numerator inclusion; Var=varies

|  | Age/
Sex | Den 
Time 
Window | Num 
Time
Window | Enc
Data | Dx
Data | Rx 
Data | Proc
Data | Test | Test 
Result |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Colorectal Screening | D | 2 yr | 10 yr | N | E |  | N | N |  |
| Beta-blocker after MI | D | 1 yr | 7 d | D | D,E | N |  |  |  |
| HbA1C Control  | D | 1 yr | 1 yr |  | D,E |  |  | D,
N | N |
| Eye exam | D | 1 yr | 1 yr | D,N | D | E | N,E |  | E |

## Data Adequacy Assessment 

- (1) _Data Present, Event Confirmed_: a lipid lowering medication is recorded in the patient’s data;
- (2) _Data Present, Event Not Confirmed_: there is no lipid lowering medication, but data indicate that the patient is taking other medications;
- (3) _Data Not Present_: there are no data regarding medications making it uncertain whether the patient is taking a lipid lowering medication.

## Analysis

| Availability of Inclusion Criteria Data for Colonoscopy? |  |  |  |
| --- | --- | --- | --- |
|  |  | Through EHR Data Method |  |
|  |  | Yes | No |
| Through Hybrid Method | Yes | a | b |
|  | No | c | d |

- Where:  Availability through the EHR =  (a+c) / (a+b+c+d) = 92%
-          And: Availability through Hybrid method = (a+b) / (a+b+c+d) = 98%

## Challenges 

- Logistical
- HIE implementation
- Data sharing (privacy/confidentiality)
- Analytic
- Lack of a “gold standard”
- Complex correlation among data sources
- Identifying and interpreting “missing” data
- Small sample sizes for some measures

## Opportunities

- NQF-endorsed national standard measure set
- Measure set creates need for broad range of data types
- Direct comparison of HIE to two widely-used measurement methods
- “Deconstruction” of measure components identifies data problems that may affect future performance measures

## “Crossing the Quality Chasm?”