Towards a systems approach to assessment

In her plenary at the AMEE conference 2009 in Malaga, Professor Janet Grant labeled the assessment of competence field ‘‘the crown jewels’’ of medical education. Indeed, assessment has a very rich history in our field. In a recent analysis of the medical education literature from 1988 to 2010, assessment was found to be the most popular topic with approximately 26% of the total number of papers dedicated to assessment (Rotgans 2011). Over these years, we see four major developments. One is the plethora of methods that have been proposed and investigated. Within a few decades, we have in essence been able to ‘climb’ Miller’s (1990) pyramid with numerous assessment instruments. One might say that in the arena of standardized assessment technology (first three layers of the pyramid), we could speak of an established technology which is heavily used in assessment practices in our training programs across the whole spectrum of the training continuum in medical education. Currently, we are in the midst of developing non-standardized methods, assessing performance in the authentic setting, either in the educational environment or in the professional workplace. Associated with this authentic assessment is the prominence of complex competencies, sometimes called domain-independent skills, generic skills, or soft skills, such as professionalism, communication, and collaboration. These skills are increasingly being considered to be essential for optimal professional functioning. These complex competencies can hardly be assessed with standardized assessment technology. A second development is related to the first one and that is the well-developed methodology around assessment. In areas of item and test construction, stimulus and response formats, scoring, itemand test analysis, standard setting, and validation strategies, a lot of ‘technology’ is available. A lot of these methods and associated technologies have impacted training programs and are even becoming part of accreditation requirements. A third development is the notion of assessment for learning as opposed to assessment of learning. In the latter notion, certification, decision making, and promotion are central concepts, basically to ascertain if a person assessed has acquired (minimum) mastery of a certain domain. In assessment for learning, the learning function of assessment is emphasized. Feedback and learning support for students to promote growth and excellence development become central notions. We think we are still at the start of a whole new area of assessment here. Finally, the fourth development is the move beyond the individual assessment method toward a more system-oriented approach, also called programmatic assessment. This is the view that an assessment program is a deliberate arrangement of a set of assessment activities. Individual methods do not stand alone in a programmatic approach, but may have their own meaning in relation to the total program. In a well-arranged assessment program, one tries to apply as many insights from all previous developments into a whole, reconciling local traditions, regulations, educational philosophies, and resources. We think the assessment field requires urgent progression in the development of the systems approach. Although the assessment literature is filled with information on how to design individual assessment instruments, very little information is found on assessment programs. Except for the work of Baartman et al. (2007) and the initial work of Dijkstra et al. (2010), very little can be found on designing assessment programs and virtually nothing about their implementations and their functioning. With the following four papers, we wish to make some first steps in this direction. The first paper by van der Vleuten and coworkers is a theoretical one. It describes a model for an assessment program that tries to optimize both assessment for learning and of learning. The model is described generically in terms of learning activities, assessment activities, and learner support activities and has periodic progression evaluation moments. Strategies are described on how the combination of these activities makes the program as a whole fit for purpose. Fit for purpose in terms of being educationally meaningful, while at the same time allowing robust decision making on learner progress. The model is also generic for the setting, but assumes a learner-centered program. The presented model in the first paper is abstract and rather idealistic. Two questions that probably come to mind when confronted with this model is how might it function in actual practice and how is it then evaluated. These questions are addressed in the following three papers. Dannefer and coworkers provide a description of the assessment program of the Cleveland Clinic Lerner College of Medicine. It very much functions in conformity with the theoretical model as in the first paper. It relies heavily on the gathering of feedback from all kinds of assessment sources. Students use this feedback to direct their learning and select evidence used for progress decisions. The paper also evaluates the program by looking at the strength of evidence that learners provide for their summative performance review. In a second paper,