Getting beyond learner reactions: elevating our scholarship about teaching and learning

The scholarship of teaching and learning (SoTL) has come a long, long way in the past two decades. More pharmacy educators than ever are publishing about innovations in teaching practices, residency program design, and professional development activities. We now have numerous peer-reviewed journals, including the Journal of the American College of Clinical Pharmacy, dedicated to publishing research and reviews related to a wide range of topics relevant to pharmacy education. Most national pharmacy meetings now have invited podium and peer-reviewed poster presentations related to teaching and learning. And the scholarship of teaching and learning is seriously considered when it comes time for promotion at many academic institutions. Similar to pharmacy practice and patient care, teaching practices should be informed by evidence. The fact that we are generating more evidence is outstanding. But it's time to up our game. It's time to get beyond learner reactions. We need to do a better job of rigorously measuring and reporting the outcomes of our educational efforts. To explain why we need better evidence, let us use an analogy that clinicians will understand. Most clinicians use the biomedical literature to inform their treatment decisions every day. Drug therapy is an intervention intended to address specific needs and achieve specific goals. To make good decisions about what drug will likely work best in a specific situation, we need studies regarding the efficacy, safety, and cost-effectiveness of the available treatments. In these studies, we look at a wide range of measures of effect to give us a well-rounded sense of the likely outcomes if the treatment were used in a variety of circumstances. We'd want to know about the patients' experiences when using the medication, including symptoms, quality of life, and the development of adverse effects. We'd want to gather information about the medication's impact on physiological responses (eg, surrogate markers, like blood pressure) as well as the health outcomes that matter most (eg, strokes and death). Moreover, we'd want to know how the treatment compares to alternative approaches. And we'd want to know how much it costs and whether it's “worth it. ” All of these pieces of information are important—one piece is not sufficient alone. Finally, we'd want these studies to have strong internal validity by following rigorous methodological standards. Similar to drug therapy, teaching is an intervention intended to address specific needs and achieve specific goals. While the learner's experience is critical to understanding whether a specific teaching strategy might be useful, it's not the whole picture. Arguably the outcome of greatest interest is learning and behavior change, not how the learner feels about the experience. If an educational intervention does not result in learning, does it matter if learners liked it or report feeling more confident? Several frameworks for measuring the outcomes of instruction have been proposed and pharmacy educators should use these models to inform the design and conduct of their educational research. See Table 1 for three widely referenced outcome frameworks for evaluating education and training programs. First described by Donald E Kirkpatrick in the 1950s, Kirkpatrick's four levels of evaluation is perhaps the simplest and most widely used framework: reaction, learning, behavior, and results. More recently, it has been modified to conceptualize the outcomes that should be measured when evaluating continuing medical education (Moore's Framework) and patient education (Rankin and Stallings) programs. Reaction data are the most commonly reported “outcome” reported in SoTL papers in pharmacy, and other health professions, journals. Reaction data include measures of participant satisfaction, the participants' judgment about the quality of the program, or changes in participants' confidence or self-perceived knowledge or skill. While these data are no doubt critical to collect, they are subjective and prone to significant bias, most particularly social desirability bias and a cognitive bias known as the Dunning-Kruger Effect. Participants often provide answers on surveys that they believe will be viewed favorably by the researcher, particularly an instructor that's well-liked. Moreover, most novices aren't very good at judging their skill level and grossly over-estimate their competence. Thus, it is not uncommon for learners to report enormous gains in self-perceived knowledge and confidence after relatively brief instructional activities. For these reasons, authors should report at least one objective measure of learning when reporting the results of an educational intervention. Measures of learning include changes in declarative knowledge, procedural knowledge, or competence measured in a variety of ways. In the absence of a control group, pre-intervention and postintervention assessments should be administered to compare performance before and immediately after the instructional activities. All too often, only post-intervention learning data from a single cohort are reported. It's simply not possible to determine if the learning activity was effective when there is no baseline or control group data, and preferably both, to compare. The goal of most educational programs is not merely for participants to be able to demonstrate their knowledge or skill on some test administered immediately after the program. The real goal is to change behavior. When the behavior is applied consistently and persistently, it leads to results. The adoption of new habits, improvements in health, and changes in organizational efficiency are admittedly more difficult to measure and report. None-the-less, we Received: 13 November 2020 Accepted: 16 November 2020