Amirfar et al. BMC Public Health 2011, 11:753 http://www.biomedcentral.com/1471-2458/11/753 In addition to reconciling with national standards, we invited DOHMH subject matter experts on sexually transmitted diseases, tobacco, asthma, and mental health to get their input on the respective measures and to devise the best method of data collection. We met with health organizations within New York City to get their recommendations on a practical set of measure alerts. For example, PCIP worked with subject matter experts, informatics specialists, and clinical providers from the Institute for Family Health, a community health center based in New York City, to develop our measures. These measures needed to incorporate clinical guide- lines, population level epidemiologic information, and optimal design based on extensive clinical experience to ensure that they were not only clinically appropriate and useful to clinicians but were also consonant with exist- ing workflows to ensure their use. At the end of this one year process and after incorporating the recommen- dations of all these subject matter experts, we had trans- formed the 10 TCNY general health areas into 40 specific quality measures at the point-of-care CDSS alerts (see Table 3). Once these 40 clinical quality measures were agreed upon, the PCIP development team undertook the task of translating these 40 measures into logic models that could be processed and interpreted by a computer pro- gram. The logic models were specific actions and pro- cesses we required the computer program to run for every patient depending on a variety of demographic and clinical content entered by the provider in the EHR. For example, if the patient was over 18 years old, we required the program to identify if smoking status was documented for the patient and if it had, to determine if the last status of “never”, “former”, or “current smoker” had been updated within the past year. The creation of the logic models was a good method that allowed the team to identify how the EHR system could determine the particular measures a patient qualified for. It also helped formulate in our minds the necessary data ele- ments that were crucial for a particular measure. In addition, the process of creating the logic models was successful in building consensus across subject experts on how a measure can be interpreted proactively at the point of care and how to reconcile best practices with small practice workflow. However, translating guidelines at the point of care often resulted in these logic models becoming extremely complicated. For example, the EHR software would have to keep track of each patient through the logic diagram, save their progress, and retrieve it quickly at the next clinical encounter. Moreover, the system was then required to keep track of several conditions and to use data for multiple “if-then” statements. It quickly became apparent that this longitudinal examination of the Page 4 of 8 patient was too complicated and these difficulties with complex coding would result in slower system perfor- mance. A limitation of the logic models was that its use did not allow for comparative analysis across patients and providers. With logic models, it was simply a matter of where a patient was placed in the flow diagram; there was no inherent “good” or “bad” state for the patient to be evaluated on. While logic models are very good for providing decision support for one patient at the point of care, there was no easy way to determine a group of patients’ compliance to a particular measure. In other words, while the alerts were appropriate and timely, it was difficult to evaluate the compliance rate to these alerts by patient and provider. Despite the complexity and ultimate abandoning of the logic model concept, the method helped to elucidate the data elements neces- sary to calculate quality measures solely based on EHR derived data. In response to the challenges presented by the logic model, the team shifted our philosophy to aggregate counting and developed compliant binary counts. Com- pliant binary counts are defined as true/false assess- ments of whether a patient meets or does not meet the defined criteria, thus indicating that a patient is either compliant or non-compliant for a particular measure. If a patient meets the criteria for disease evaluation, then a “1” is placed in the denominator. If a patient meets the criteria for measure satisfaction, then a “1” is placed in the numerator. Therefore, every patient for each applic- able measure is either, “0/1” or 0% for non-compliant or “1/1” or 100% for compliant. For example, one of our measures asks that every patient with diabetes mellitus have their hemoglobin A1c checked at least once every 6 months and the result to be entered in the EHR. If a patient is documented in the EHR as having diabetes mellitus as a medical condition in their problem list, then they are added to the hemoglobin A1c testing mea- sure and the denominator of that measure increases by 1. If that provider then orders a hemoglobin A1c test for the patient and the result of the test enters into the EHR, the numerator increases by 1 and the patient is fully compliant (1/1) for this measure. If not, the patient is not compliant (0/1). If the provider has 500 diabetic patients, this number can be summed for a compliance rate for this measure. If 400 hemoglobin A1c lab tests results are eventually entered for these patients, then the provider’s compliance rate will be 80%, indicating that 80% (400/500) of the provider’s diabetic patients have had the appropriate intervention for this measure. In this way, results from a group of patients can be aggregated by provider, thus providing a quality assess- ment of the provider’s patient panel which for this parti- cular provider for this particular CDSS alert would be a compliance rate of 80%. The major advantage of using