information entered into the EHR and describes the issues contributing to variations in the results of automated EHR quality measurement. METHODS Practice selection A subset of 82 practices enrolled in a pilot rewards and recog- nition programme were invited to participate in the data vali- dation study, as they had all implemented the eClinicalWorks EHR software before January 2009 and received technical assistance through the PCIP programme, and had a majority of their patient panel recorded in the EHR. Practices were required to have a minimum of 200 electronic patient records with a diagnosis of diabetes, hypertension, dyslipidemia, or ischemic cardiovascular disease. All 82 practices received a software upgrade between February and August 2009 to implement automated quality measurement reporting and the CDSS func- tionality. Participating practices signed a letter of consent allowing independent medical reviewers to conduct abstraction of the EHR and received an honorarium of US$500 for completing the study. This study was approved by the Department of Health and Mental Hygiene institutional review board no 09-067. Electronic chart reviews Medical reviewers randomly sampled 120 electronic patient charts per practice for established patients between 18 and 75 years of age, with at least one ofﬁce visit since the practice implemented the EHR. For this study, data from the manual review of the electronic chart (e-chart) were analyzed if the patient had an ofﬁce visit during the 6-month period after the activation date of the quality measurement reporting tool and implementation of the CDSS. For each e-chart, reviewers abstracted the patient’s age and gender, along with vitals, diagnoses, medications, laboratory results, diagnostic images, vaccinations, and receipt of or referral to counselling for the most recent visit. Depending on the data element, reviewers were instructed to search in multiple loca- tions of the EHR: problem list, medical history, social history, progress notes (chief complaint, history of present illness, assessment), procedures, diagnostic images, vitals, and laboratory tests. Each reviewer was trained and tested by a standardized approach to ensure interrater reliability. If there was uncertainty whether documentation would meet the quality measure criteria a senior reviewer and PCIP staff would determine whether to include or exclude the observation. Analytical methods For each patient, each data element was coded based on whether it was documented in a structured ﬁeld recognized by the existing EHR software for automated quality measurement (1¼location recognized and 0¼not recognized for quality measurement reporting). In addition, two sets of numerator and denominator counts were generated for each of the 11 quality measures. The ﬁrst set of counts included only those patients whose information was documented in structured ﬁelds recog- nized by the existing software (EHR automated). The second set of counts incorporated all information about patients docu- mented in the EHR (e-chart review). Data element coding and all counts were calculated using Microsoft Access structured query language. Simple frequencies and descriptive statistics were generated to determine data element documentation patterns and estimate population-level numerators and denominators for each quality J Am Med Inform Assoc 2012;19:604e609. doi:10.1136/amiajnl-2011-000557 Research and applications measure. Wilcoxon non-parametric tests were used to compare practice-level numerators and denominators calculated for each measure. All descriptive statistics and tests were conducted using SAS V.9.2 analytical software. Practice distribution of documentation and calculated proportions of elements documented in various locations of the electronic chart were generated in MS Excel and MS Access. RESULTS Of the 82 practices eligible for the study, 57 practices agreed to allow PCIP to conduct the e-chart review. Based on self-reported information and practice registries, practices that agreed to participate in the study had a higher average percentage of Medicaid insured patients (42.7% vs 30.7%), and a higher number of patients with diabetes (209.9 vs 110.0) and hypertension (477.8 vs 353.2) (table 1). A total of 4081 e-charts was available for this analysis. An additional 2759 e-charts were reviewed, but were excluded from the analysis because the patients did not have a qualifying ofﬁce visit during the 6-month study period. More than half of the ﬁnal study sample of patients were women (59.4%), and the average patient age was 48.1 years. Participating practices varied in their distribution of patients with diagnoses of diabetes, ischemic cardiovascular disease, hypertension, dyslipidemia and patients who were current smokers (table 1). The majority (89.9%) of participating providers were primary care providers (ie, internal medicine, family medicine, pediatrics, obstetrics/ gynecology). Non-primary care providers specialized in cardi- ology, pulmonology, endocrinology, allergy, gastroenterology, or did not specify a specialty (data not shown). We looked across the 11 clinical quality measures to assess where information was documented. The presence of data recognized for automated quality measurement varied widely, ranging from 10.7% to 99.9% (table 2). Measure components Table 1 Characteristics of practices and patient charts reviewed Practice characteristics No of providers per practice No of full-time equivalent Percentage of patients with Medicaid insurance or uninsured Estimated population* Patients >18 years of age with at least one ofﬁce visit in the past year Diabetes Hypertension Months using EHR by 1 July 2009 Patient characteristics of charts reviewed Medical charts reviewed (total 4081) %Female Patient age, years Percentage of records with diagnoses Diabetes Ischemic cardiovascular disease Hypertension Dyslipidemia Current smokers Participating Not participating in chart in chart review (N[57) review (N[28) Mean (minLmax) Mean (minLmax) 2.9 (1 24) 1.7 (0.7 9.2) 42.7 (0.0 90.0) 2.4 (1 13) 1.9 (1.0 10.8) 30.7 (2.0 81.0) 1936.2 (0 9117) 1905.5 (12 9723) 209.9 (3 1480) 110.0 (1 450) 477.8 (2 2460) 353.2 (1 1121) 14.3 (7.5 39.1) 13.7 (3.7 22.4) Mean (Min Max) Not applicable 71.6 (19 111) 59.4 (35.0 80.7) 48.1 (32.8 61.5) 17.6 (0.0 48.1) 7.3 (0.0 39.0) 41.2 (8.2e87.0) 36.0 (3.3 85.4) 10.3 (1.4 37.9) e e e e e e e e *From automated reporting from the electronic health record (EHR)dseparate from electronic chart reviews. 605