Can Electronic Health Records Be Used for Population Health Surveillance?
Validating Population Health Metrics Against Established Survey Data
Authors: McVeigh Katharine H. , Newton-Dame Remle , Chan Pui Ying , Thorpe Lorna E. , Schreibstein Lauren , Tatem Kathleen S. , Chernov Claudia , Lurie-Moroni Elizabeth , Perlman Sharon E. 
Published: December 2016

Introduction: Electronic health records (EHRs) offer potential for population
health surveillance but EHR-based surveillance measures require validation prior to use.
We assessed the validity of obesity, smoking, depression, and influenza vaccination
indicators from a new EHR surveillance system, the New York City (NYC) Macroscope. This
report is the second in a 3-part series describing the development and validation of the
NYC Macroscope. The first report describes in detail the infrastructure underlying the
NYC Macroscope; design decisions that were made to maximize data quality;
characteristics of the population sampled; completeness of data collected; and lessons
learned from doing this work. This second report, which addresses concerns related to
sampling bias and data quality, describes the methods used to evaluate the validity and
robustness of NYC Macroscope prevalence estimates; presents validation results for
estimates of obesity, smoking, depression and influenza vaccination; and discusses the
implications of our findings for NYC and for other jurisdictions embarking on similar
work. The third report applies the same validation methods described in this report to
metabolic outcomes, including the prevalence, treatment and control of diabetes,
hypertension and hyperlipidemia.Methods: NYC Macroscope prevalence estimates, overall
and stratified by sex and age group, were compared to reference survey estimates for
adult New Yorkers who reported visiting a doctor in the past year. Agreement was
evaluated against 5 a priori criteria. Sensitivity and specificity were assessed by
examining individual EHR records in a subsample of 48 survey participants.Results: Among
adult New Yorkers in care, the NYC Macroscope prevalence estimate for smoking (15.2%)
fell between estimates from NYC HANES (17.7 %) and CHS (14.9%) and met all 5 a
priori criteria. The NYC Macroscope obesity prevalence estimate (27.8%) also fell
between the NYC HANES (31.3%) and CHS (24.7%) estimates, but met only 3 a
priori criteria. Sensitivity and specificity exceeded 0.90 for both the smoking and
obesity indicators. The NYC Macroscope estimates of depression and influenza vaccination
prevalence were more than 10 percentage points lower than the estimates from either
reference survey. While specificity was > 0.90 for both of these indicators,
sensitivity was < 0.70.Discussion: Through this work we have demonstrated that EHR
data from a convenience sample of providers can produce acceptable estimates of smoking
and obesity prevalence among adult New Yorkers in care; gained a better understanding of
the challenges involved in estimating depression prevalence from EHRs; and identified
areas for additional research regarding estimation of influenza vaccination prevalence.
We have also shared lessons learned about how EHR indicators should be constructed and
offer methodologic suggestions for validating them.Conclusions: This work adds to a
rapidly emerging body of literature about how to define, collect and interpret EHR-based
surveillance measures and may help guide other jurisdictions.