Amirfar et al. BMC Public Health 2011, 11:753 http://www.biomedcentral.com/1471-2458/11/753 indicators, PCIP began the development phase in which we created an overall plan for customizing the electronic health record to meet the needs of New York City pro- viders. We developed a timeline for the project and defined our measures for success, namely 2000 providers being live with a full public health oriented EHR with actionable CDSS in two years. Our long term goal has always been to improve clinical outcomes for New Yor- kers and all our project plans were produced with this overarching goal in mind. The next crucial step in this phase was producing our CDSS alerts within the software by translating the mea- sures with the binary counts we had produced with our DOHMH and community partners into definitions that computer software could understand. To do this, we had to create a PCIP team dedicated to producing this new system. Our development team consists of quality assurance specialists who test the new product, EHR super users who go to the providers and provide tailored training, physicians who provide clinical input and prac- tice workflow experience, epidemiologists who analyze the resulting data, and project managers who coordinate all the activities. Once the team had been established, the operational structure for collaborative development between PCIP and eCW was created. We agreed that meetings between ourselves and eCW software develo- pers had to be frequent and lengthy in order to solidify relationships and to make sure we had similar expecta- tions of the final product. During these meetings, we spent the majority of the time defining the scope for each new proposed functionality as well as the rationale behind its introduction. We felt it was important that the vendor had “buy-in” for each functionality and felt that the proposed changes were necessary as well. Hav- ing senior executive leaders with both vision and con- tent matter expertise closely involved was critical because it created an environment where both the DOHMH and vendor could overcome the inevitable obstacles. The cooperation and shared vision helped both organizations to succeed and achieve their respec- tive goals. Even with the time spent describing each new func- tionality during the collaborative development meetings, there were many follow-up discussions for clarifying or redeveloping functionalities because items that eCW developed were not always what we had in mind. We quickly came to realize another important aspect of our relationship. While we might be agreeing on the facts, the interpretation can also make or break a function in terms of usefulness, effectiveness, and adoption. The software developer approaches each solution with a practical engineering viewpoint that must be countered with a clinical or public health viewpoint for balance. Because of this, it is very important to get frequent Page 6 of 8 visual updates for each functionality via demonstrations to make sure the vision does not veer off the path from what is expected. One of the issues that our collaborative development team needed to discuss was the use of technological standards. Standards are necessary in an environment where the software needs to exchange medical content entered in the progress notes across many users. The software vendor needs standards so that their system is able to communicate between different users. For the DOHMH, our scope is broader since we need standards to satisfy our larger health information technology needs in order to receive aggregate data transmissions of clini- cal information both among all electronic health users in New York City and to the DOHMH. Since the deci- sion support tools were implemented for all our provi- ders, it was necessary to discuss the standard set of data the software would utilize to produce the alerts with the EHR vendor. Some standards already existed and we simply used them in our CDSS programming. For exam- ple, we used the International Classification of Diseases (ICD-9) codes as the standard for codifying the patient’s medical problems and Logical Observation Identifiers Names and Codes (LOINC) codes for lab results return- ing from commercial laboratories. For other data ele- ments where no standard exists, it was necessary to make our own. Laboratory ordering was one area where there are no established standards and we made our own numerical classification system called community IDs. For example, an HIV test ordered from the system is the same irrespective of the commercial laboratory where the test is ordered. Results Once the different CDSS functionalities had been devel- oped in the electronic health record, testing the func- tionality began in earnest. We had underestimated the time and effort that testing of the functionalities would require. In fact, testing of the measures required more resources and was more complex than the development phase. Between physically entering test patients, ensur- ing that CDSS measures were functional, and keeping track of the bugs and versions developed, we quickly realized we had underestimated this portion of CDSS development. Testing of the various bugs within the system compro- mised two main stages. The first stage involved making sure that the functionalities of the CDSS system were working. This included alpha testing which entailed making sure new CDSS functions in the system worked such as quick links to labs and medications, and that there were no gross and obvious system errors. One of the most common errors we discovered during this part of the testing were usually HTML or JavaScript errors