A an investigation of interrater reliability among athletic training accreditation site visitors.

Accreditation plays an important role in many allied health professions. The main purpose of this study was to determine interrater reliability among athletic training accreditation site visitors assigned to interpret and judge a college or university's educational program in accordance with published standards. Ten scenarios were sent electronically to 135 site visitors, of which 93 (68%) responded. Respondents rated whether hypothetical situations described in each scenario were compliant (i.e., meets standard) or noncompliant (i.e., does not meet standard). A comment section was included for site visitors to justify their responses and/or to clarify their selections. The results of this study showed that there was poor interrater reliability in the judgments made by the site visitors. Moreover, a majority of the respondents responded to two of the 10 scenarios incorrectly (scenarios 5 and 10). The average score was 6.77 ± 1.57, or 68%. Site visitors who misinterpret standards or apply personal values pose serious threats to the accreditation process because they may judge educational programs compliant when in fact programs are noncompliant and vice versa. Furthermore, this study suggests that there may be a great deal of variability across academic programs in applying accreditation standards because of the low agreement among site visitors. Therefore, educational methods and training procedures for accreditation site visitors should be continually reexamined in an attempt to improve and ensure consistency in allied health professions. J Allied Health 2005; 34:65-75. ACCREDITATION STANDARDS are driving curriculum development and assessment strategies on college campuses across the country. However, accreditation is not new. More than a century ago, accreditation agencies formed as self-regulatory and evaluative forces for higher education in the United States.1-6 Today, there are regional and specialized accreditation agencies. Regional agencies grant institutional accreditation to an entire college or university, whereas specialized accreditation agencies such as the National League for Nursing Accrediting Commission, the Commission on Dental Accreditation, and the Commission on Accreditation of Allied Health Education Programs (CAAHEP) grant accreditation to specific educational programs. Across the United States, there are six regional accreditation agencies representing different geographic locations (i.e., New England, Middle States, North Central, Southern, Northwest, and Western). The primary goal of regional accreditation is to ensure quality teaching and learning visa-vis established and accepted standards.7 Specialized agencies vary widely and represent various professions. For instance, CAAHEP reviews and accredits more than 2,000 educational programs in 21 health science occupations such as emergency medical technician/paramedic, orthotist/prosthetist, respiratory therapist, and surgical technologist.8 The concept of specialized accreditation for athletic training education programs has been important in the preparation of certified athletic trainers (ATC) for more than 40 years.9-11 Accreditation standards provide a comprehensive framework in which the clinical instruction of students is monitored. Program directors and clinical instructors rely on accreditation standards to help them maintain stability when faced with budget cuts or declining student enrollments. In addition, accreditation helps to bring legitimacy to the profession, especially as ATCs seek third-party reimbursement for their services from medical insurance companies and health maintenance organizations.12 Only accredited programs may sponsor students for the National Athletic Trainers' Association Board of Certification (NATABOC) examination. Accreditation is indeed an important strategy for assuring quality control not only in athletic training but also in many allied health professions. For the past 10 years, academic programs in athletic training have been accredited by CAAHEP. …