Research and applications Table 3 Comparison of numerator and denominator counts for EHR automated quality measurement and electronic chart review (n¼57 practices) Measure A1c control A1c screening Antithrombotic therapy Body mass index Blood pressure control Breast cancer screening Cholesterol control Cholesterol screening Inﬂuenza vaccination Smoking cessation intervention Smoking status recorded *p<0.01 **p<0.001. EHR, electronic health record. Denominator EHR query Mean 4.9 11.8 13.1 71.4 25.7 28.7 15.7 41.7 34.4 4.3 71.5 SD 5.2 8.6 10.3 18.1 14.4 9.9 14.3 15.2 15.1 3.8 18.1 e-Chart review e-Chart review Mean 8.2** 12.9 14.7 71.4 29.3 28.7 29.1** 44.2 34.4 7.2** 71.5 SD 6.1 8.6 10.6 18.1 14.4 9.9 15.7 15.8 15.1 4.8 18.1 Numerator EHR query Mean 1.8 4.9 7.5 54.6 14.1 1.0 13.0 15.7 8.1 1.5 31.5 SD 2.4 5.2 7.8 23.2 9.6 2.3 12.5 14.3 8.7 2.7 24.0 Mean 2.7* 8.2** 8.2 54.8 16.2 9.5** 23.8** 29.1** 8.3 2.3 59.5** SD 2.6 6.1 7.8 23.2 10.0 8.2 13.5 15.7 8.9 3.5 18.9 Data from this study suggest that clinical diagnoses for the majority of patients are documented in structured ﬁelds needed for automated quality reporting and point-of-care reminders, and that there is not a signiﬁcant amount of ‘denominator loss’ when using EHR-derived measurement versus e-chart reviews. However, EHR-derived quality measurement can result in signiﬁcant numerator loss, resulting in underestimates of the delivery of clinical preventive services. It is important to understand the impact of workﬂows and documentation habits on EHR-derived quality measures. In this study, the majority of practices correctly documented diagnoses of hypertension and diabetes over 80% of the time, but rates of appropriate documentation for dyslipidemia and ischemic cardiovascular disease were substantially lower. Providers may be more likely to overlook chronic diseases that are documented elsewhere in the chart, a ﬁnding we commonly see in the case of obesity and active smoking. Anecdotally, providers have reported to us that they limit the number of assessments assigned to a patient at any given visit due to the historical limitation related to paper claims (some payers limit the number of assessments that can be reported in paper claims to four or fewer), which may have the unintended consequence of ‘underdocumentation’. For laboratory tests, the presence or absence of laboratory interfaces and the appropriate logical observation identiﬁers names and codes creates signiﬁcant variation in practice scores. Even if a practice has an electronic interface, reference labora- tories do not consistently provide EHR vendors with compen- diums that have logical observation identiﬁers names and codes codes for each test, and therefore, some test orders and results remain undetected by EHR software. For practices with no electronic laboratory interface, the quality measures that rely on laboratory data can only be addressed by manually entering the results into structured ﬁelds, which few providers do. In addition, for quality measures that rely on tests or procedures performed in a different ofﬁce setting, the difﬁculty of getting the results back, and in structured form, makes it challenging to satisfy those measures. In this study, the mammography quality measure was only satisﬁed by the presence of a structured test result, yet most mammography results come back in the form of faxed results, necessitating an additional step by the practice to codify the results in structured form. Eventually, developments in natural language processing may help convert unstructured text results into their structured counterparts needed to satisfy quality measures and trigger clinical decision support. 608 Finally, some quality measures may be impacted by incorrect or imprecise logic used to code the measure. For instance, with mammography rates, the relatively low practice performance seen in this study is largely attributable to a speciﬁc ﬂaw in the design of the EHR’s quality measure. For over a year, this particular measure would only allow a mammogram result to satisfy the measure if the results were structured and the test was ordered as a ‘procedure’, and not a ‘diagnostic image’, the latter being the more consonant with provider preference. Providers uniformly scored well on documentation of aspirin therapy, inﬂuenza vaccination and blood pressure, probably due to the relative lack of options in the EHR to document these data in ﬁelds other than the designated locations. For some other measures, such as smoking status, documentation can vary widely provider to provider, including a variety of notations (eg, ‘+ smoker ’, ‘+ cig’, ‘2 ppd’, ‘smoker ’, and ‘+ tobacco’) and locations (eg, smart form, preventive medicine, social history). This variability in documentation preference has been shown to lead to signiﬁcant variations in quality measurement depending on which ﬁelds are chosen and how much granularity is provided. The bimodal distribution of practice documentation for smoking status is probably due to the difference in whether practices used the structured ﬁelds in the smart form to satisfy the quality measure. This study has several limitations. Several practices refused to participate in the chart review. The majority that did not participate stated they lacked sufﬁcient physical space for the chart reviewers to conduct their reviews. Practices electing not to participate in the study did not differ in their population characteristics, as shown in table 1. This study also limited its chart reviews to practices that were using eClinicalWorks; therefore, the ﬁndings may not be generalizable to other EHR. In addition, the study focused on available documentation in the electronic record, and we did not conduct an audit of whether providers or practice staff actually delivered the services recorded. Separate studies will need to be conducted to ascertain whether information recorded in the EHR may not reﬂect the actual care delivered. Finally, this study did not assess how practice characteristics, interventions by PCIP, or the use of speciﬁc EHR functionality may have impacted differences in documentation variation. These studies are being conducted separately. Until recently, quality measures were largely derived from manual paper chart reviews, secondary analysis of claims J Am Med Inform Assoc 2012;19:604e609. doi:10.1136/amiajnl-2011-000557