Board 421 - Research Abstract A Systematic Approach to Design Clinical Performance Checklists (Submission #227)
2013-12-01

Introduction/Background Assessing performance helps to identify the abilities of clinicians and potential performance gaps, augments debriefings and is essential for scientific studies investigating factors influencing clinical performance.1 Checklists are widespread tools to assess performance. But while the development process of such a performance checklist is essential for its quality, existing studies rarely describe their checklist development in detail. Current methodological recommendations2,3 fail to provide a systematic, step-by-step approach to develop clinical performance checklists. Such a systematic approach would support researchers in evaluating the suitability of the checklists for different contexts, designing performance assessment tools for specific clinical scenarios reflecting precisely the task demands on the clinician and in either adapting existing checklists or generating new ones. Thus, the aim of this study was to provide an overall systematic approach to develop clinical performance checklists. Using the example of a simulated sepsis scenario we illustrate our five development steps. Methods Step 1 - Draft Checklist: Based on the literature and own clinical experience we designed a draft checklist. Step 2 – The Delphi-Review-Rounds: We sent out the draft checklist to five experts for reviewing using an adapted Delphi-Method.4 Step 3 – Design of the final checklist and pilot testing: Every checklist item was then divided into three scoring categories: task not performed (0 points); task performed partially (1 point), and task performed completely (2 points). Then the checklist was tested by rating video clips of simulation trainings and a few adjustments have been made. This step is indispensable; by applying the checklist to a set of different examples, the raters experience the applicability of the items and the usability of the rating scale. Step 4 – Final Delphi-Review-Round: To assure that the changes made after the pilot testing are generally valid the checklist resulted from step three was sent out again to the five experts. Step 5 – Items weighting: In the last step we sent out the checklist to 30 pediatricians and instructed them to rate all actions in terms of their importance for the success of the treatment. The mean importance score serves as a weighting factor for every item. This way we get a more accurate assessment of performance because the checklist differentiates more between essential and less important items. Validity testing – Six videos of septic shock simulation training were independently rated from two raters. Interrater reliability was calculated using Cronbach’s α; criterion validity was tested by investigating the relationship between the checklist score and three external criterion: team experience level, experience level of the leader and a global performance rating (rating from 1-10). Results We successfully applied our five step approach and we developed a performance checklist including 33 items for a simulated paediatric sepsis scenario. Cronbach’s α ranged from acceptable (αα = .6) to very good (α = .9). Criterion validity is given: Significant correlation between the checklist score and i) mean experience level of team (r = .37, p= .05) ii) leader experience level (r = .44, p= .05) iii) global performance rating score (r = .54, p= .05). Conclusion We described a systematic approach to design clinical performance checklists that integrates the published evidence and the knowledge of domain experts. The validity of the checklist has been confirmed. A structured development process is a necessary prerequisite of a valid checklist. Only if a widely recognized standard for developing performance checklists is established we can design appropriate measurement tools and move the field of performance assessment in healthcare forward. References 1. Boulet JR, Murray D. Review article: Assessment in anesthesiology education. Canadian Journal of Anesthesia/Journal canadien d’anesthésie. 2011:1-11. 2. Stufflebeam DL. Guidelines for developing evaluation checklists: the checklists development checklist (CDC). [monograph on the Internet]. 2000; http://www.wmich.edu/evalctr/archive_checklists/guidelines_cdc.pdf. Accessed Dezember 17, 2012. 3. Scriven M. The logic and methodology of checklists. Retrieved on. 2000;11:02-07. 4. Clayton MJ. Delphi: a technique to harness expert opinion for critical decisionÃ¢â‚¬Â making tasks in education. Educational Psychology. 1997;17(4):373-386. Disclosures Salary Support from Center for Medical Simulation to teach on simulation courses none Per dien honoraria from PAEDSIM e.V. to teach on pediatric simulation courses non-profit organization PAEDSIM e.V.