Research and applications Primary Care Information Project, New York City Department of Health and Mental Hygiene, New York, New York, USA Correspondence to Dr Amanda Parsons, Health Care Access & Improvement, New York City Department of Health and Mental Hygiene, 42-09 28th Street, 12th Floor, Queens, NY 11101, USA; aparsons@health.nyc.gov Received 18 August 2011 Accepted 21 December 2011 Published Online First 16 January 2012 This paper is freely available online under the BMJ Journals unlocked scheme, see http:// jamia.bmj.com/site/about/ unlocked.xhtml 604 Validity of electronic health record-derived quality measurement for performance monitoring Amanda Parsons, Colleen McCullough, Jason Wang, Sarah Shih ABSTRACT Background Since 2007, New York City’s primary care information project has assisted over 3000 providers to adopt and use a prevention-oriented electronic health record (EHR). Participating practices were taught to re- adjust their workﬂows to use the EHR built-in population health monitoring tools, including automated quality measures, patient registries and a clinical decision support system. Practices received a comprehensive suite of technical assistance, which included quality improvement, EHR customization and conﬁguration, privacy and security training, and revenue cycle optimization. These services were aimed at helping providers understand how to use their EHR to track and improve the quality of care delivered to patients. Materials and Methods Retrospective electronic chart reviews of 4081 patient records across 57 practices were analyzed to determine the validity of EHR-derived quality measures and documented preventive services. Results Results from this study show that workﬂow and documentation habits have a profound impact on EHR-derived quality measures. Compared with the manual review of electronic charts, EHR-derived measures can undercount practice performance, with a disproportionately negative impact on the number of patients captured as receiving a clinical preventive service or meeting a recommended treatment goal. Conclusion This study provides a cautionary note in using EHR-derived measurement for public reporting of provider performance or use for payment. The American Recovery and Reinvestment Act of 2009 authorized US$19 billion in funding for the deployment and meaningful use of electronic health records (EHR), and introduced a national framework for the adoption of health information technology.1 The Center for Medicare and Medicaid Services has offered eligible providers ﬁnancial incentives for demonstrating meaningful use of EHR and reporting on the quality of care.2 Starting with stage 1 of meaningful use, the Center for Medicare and Medicaid Services calls for the submission of provider-level quality measures, initially by attestation but then through electronic submission, starting as early as 2012. Many stake- holders, including payers, independent physician associations, and consumers have a vested interest in accessing and utilizing EHR-derived quality measures for purposes of accountability or rank- ings. However, quality measures derived from EHR have yet to be validated as representative of provider performance for incentives or comparative purposes. Unlike most claims-based quality measurement, measures derived from EHR can incorporate clinical ﬁndings, allowing for the tracking of intermediate outcomes such as blood pressure and body mass index. However, documentation habits by providers can vary, and the necessary data entered into the EHR may not be interpreted or recognized by standard EHR software programming. This may lead to undercounting the patients eligible for a preventive service (eg, diagnosis of ischemic cardiovascular disease) or receiving a recommended treatment (eg, screening or medication) or meeting a recommended target (eg, control of blood pressure to less than 140/90 mm Hg). Formed in 2005, the New York City Primary Care Information Project (PCIP) has assisted over 3000 providers to adopt and use a prevention-oriented EHR as a means to improve the delivery of primary care. Nearly 40% of the participating providers are operating in small (fewer than 10 providers), physician-owned practices. PCIP selected an EHR vendor through a competitive process and co- developed prevention-oriented functionality and population health monitoring tools, including automated quality reporting and a clinical decision support system (CDSS). The quality reporting tool displays by measure, for each eligible patient, whether the practice has or has not met the recommended preventive service. In addition, at the point of care, the CDSS function displays the preventive services a patient is eligible for and has not yet received, allowing the provider to take action (eg, order a mammogram, adjust medications, discuss smoking cessation aids) during the visit. Providers were trained by both the EHR vendor ’s training staff and practice consultants employed by PCIP, who provided on-site technical assistance. Providers were taught to re-adjust the practice’s workﬂows to document diagnoses and key preventive services in structured ﬁelds that are searchable and capable of generating the quality measures and preventive service reminders. Providers were also shown how to view their EHR calculated quality measures both within the EHR and through monthly reports generated by PCIP staff and emailed to individual providers. In addi- tion, efforts were made to create alignment between payment and improved quality of care by informing providers of the various available incen- tive programmes as well as launching PCIP designed programmes. Through these synergistic changes: (1) prevention-oriented EHR; (2) practice workﬂow redesign; and (3) payment that rewards prevention, PCIP worked with primary care providers to prioritize prevention and facilitate management of chronic disease.3 This report provides an assessment of the validity of quality measures derived from J Am Med Inform Assoc 2012;19:604e609. doi:10.1136/amiajnl-2011-000557