Evidence In Practice: A Study Of "Evidence-Based" Non-Formal Education

Non-formal education programs are under increasing pressure to be “evidence-based,” where evidence derived from randomized controlled trials is seen as the most credible type of evidence—the “gold standard.” This qualitative study explores the politics of evidence enacted in the practices of the “evidencebased” education movement, focusing on three cases. Educational programs should be based on evidence. On its face, this statement seems obvious and platitudinous. All non-formal education—like all human action—is based on some evidence, in the vernacular sense of the term. Educators, administrators, program planners, scholars and other people engaged in non-formal education base their actions on a wide array of evidentiary information. What’s more, people engaged in non-formal and community-based education tend to not be entirely unreflective about what informs their practice—the 2000 edition of the Handbook of Adult and Continuing Education is centered on questions of critically reflexive practice, highlighting the multiplicity of frameworks, theories of action, and ways of knowing that guide professional non-formal educational praxis (Wilson & Hayes, 2000). However, the “evidence-based” education movement—one of many related attempts to “bridge the research-practice gap” that has gained prevalence in recent decades—is predicated on more formal and conscribed definitions of evidence, where certain research and evaluation approaches are valued more highly than others. In the current “era of accountability,” some policy-makers, funding agencies, and scholars position “scientific” evidence derived from randomized controlled trials (RCTs) as the “gold standard” for establishing proof of which programs “work” and which do not (Coalition for Evidence-Based Policy, 2003; Mosteller & Boruch, 2002; U.S. Department of Education, 2003). According to Trochim: The gold standard debate is one of the most important controversies in contemporary evaluation and applied social sciences. It’s at the heart of how we go about trying to understand the world around us. It is integrally related to what we think science is and how it relates to practice. There is a lot at stake. (Unpublished speech transcript, September 10, 2007) In his critique of the RCT design—in which he concludes that the RCT has “essentially zero practical application to the field of human affairs” (2008, p. 12)—Scriven reiterates the point that much is at stake, claiming, “This issue is not a mere academic dispute, and should be treated as one involving the welfare of very many people, not just the egos of a few” (2008, p. 24). In this study, I leverage both empirical and theoretical perspectives on the “evidence-based” movement to provide examples of the ways in which tightly circumscribed definitions of “evidence-based programs” affect non-formal education praxis, showing in concrete terms what is at stake. Purpose and Research Questions The purpose of this study is to better understand how efforts to make non-formal education more “evidence-based” actually unfold in practice. I focus specifically on three cases, two of which involve mandates to implement evidence-based programs (i.e., tightly-scripted