Amirfar et al. BMC Public Health 2011, 11:753 http://www.biomedcentral.com/1471-2458/11/753 where the CDSS alerts were supposed to be. As these areas in the system were slowly corrected, we focused our attention on the second stage of quality assurance. In the second stage, we primarily focused upon correct- ing any clinical content errors such as clinical alerts that appeared for the wrong patients using use cases. For example, if a test hypertension patient in the system had a blood pressure of 135/85, a blood pressure alert would not appear for this particular patient. However if a diag- nosis of diabetes was entered for this patient, their blood pressure would qualify them for the correspond- ing more stringent blood pressure alert for a diabetic patient. Conclusions Building meaningful measures as CDSS and incorporat- ing them effectively into an electronic health record was more challenging than anticipated. The steps involved are difficult and long and really cannot be shortchanged. All view points, including those of public health, clinical, and technical must be represented so that there is con- sensus. Since these measures will affect and direct provi- der workflow, the chosen measures must be clinically significant. There needs to be some understanding of where these measures fit into the other state and national measures that have already been developed. Once the measures are finalized, the difficult part is to translate them into a format that a computer program can understand and utilize effectively. Public health and quality measurement experts must find their counter- parts in software engineers who share a public health vision and will prioritize the development and imple- mentation of these measures. During their development, there need to be countless conversations to facilitate transparent translation of the measures from the clinical perspective to an engineering/programming perspective. Finally, testing of the measures needs to be done to make sure that they work at the right time for the right patient to give the right message. Having completed this process, there are some recom- mendations we can make to improve the chances of suc- cess for others seeking to build CDSS within their EHR. First, organize a workgroup of various clinical experts to help define the scope and build consensus. Sometimes, you will not know how big a project you are facing until you can see the challenge from every perspective. A large pool of experts will help with this. Not only will a large pool of experts allow for a wider viewpoint on clinical measures complying with both domestic and interna- tional standards but will also help the project to become a success since more people will have a vested interest in the project. Second, get support for your activities from executive leadership. The full support of two sequential City Health Commissioners and the Chief Executive Page 7 of 8 Officer of our partner software company has been invalu- able for getting help and cooperation from not only our respective organizations but from the community as well. Third, find an enthusiastic and energetic software partner who shares your vision for your project and is well pre- pared to undertake a complex task such as this. The energy in a good relationship can be used to pull both of you through during the inevitable tough times. Fourth, implement a transparent operational strategy so both sides can understand what the other is developing. The same scope and function may be agreed upon in meet- ings but its actual implementation will inevitably vary. An engineer will always find ways of interpreting facts differently from what a clinician envisions. Do not assume anything except that they will see things differ- ently. It would be unfortunate to find out several months into development that the vendor is producing a func- tionality that one cannot use. Fifth, create and share use cases. You need to make sure that the right alerts show up for the right patients at the appropriate time. Testing by users and developers guarantees that the new func- tionality is working properly, but only a clinician can ulti- mately decide if the software “makes sense” in a clinical setting. Lastly, respect your time and effort in doing this project and ask for help if you need it. Others have gone through the process, so there is no need to reinvent the wheel. Guidance and advice from others can go a long way to getting things back on track. By following these recommendations, one can improve their chances for developing a clinically useful CDSS alerting system while minimizing the inevitable obstacles and frustrations. We have followed these recommendations ourselves in a later phase while developing new functionalities. The process has been much improved: shorter, more focused, and more satisfying for both parties. These efforts of develop- ing a public health oriented CDSS has allowed us to com- municate more effectively with providers when they are seeing patients and has advanced the relationship between local government agency and the providers of healthcare into the 21st century. Abbreviations BP: blood pressure; PCG: primary care giver; DM: diabetes mellitus; HTN: hypertension; IVD: ischemic vascular disease; HbA1c: hemoglobin A1c. Acknowledgements We would like to thank the New York City Department of Health and Mental Hygiene, Primary Care Information Project, and eClinicalWorks for all their help during this development process. This project is funded through New York City funds and New York State grant dollars. Authors’ contributions SA literature review and drafted the majority of the manuscript. SA drafted parts of the manuscript and performed major editing. JT performed literature review and wrote parts of the manuscript. JS drafted parts of the manuscript and performed editing. All authors were involved in the