bluuebunny/crossref_metadata_2025_split huggingface dataset:
Elements to Enhance the Successful Start and Completion of Program and Policy Evaluations: The Injury & Violence Prevention (IVP) Program & Policy Evaluation Institute
Authors: Porter Jamila M. , Brennan Laura K. , Fine Mighty , Robinson Ina I. 
Published: April 2023

Background: Public health practitioners, including injury and violence prevention (IVP) professionals, are responsible for implementing evaluations, but often lack formal evaluation training. Impacts of many practitioner-focused evaluation trainings—particularly their ability to help participants successfully start and complete evaluations—are unknown.
Objectives: We assessed the impact of the Injury and Violence Prevention (IVP) Program & Policy Evaluation Institute (“Evaluation Institute”), a team-based, multidisciplinary, and practitioner-focused evaluation training designed to teach state IVP practitioners and their cross-sector partners how to evaluate program and policy interventions.
Design: Semi-structured interviews were conducted with members of 13 evaluation teams across eight states at least one year after training participation (24 participants in total). Document reviews were conducted to triangulate, supplement, and contextualize reported improvements to policies, programs, and practices.
Intervention: Teams of practitioners applied for and participated in the Evaluation Institute, a five-month evaluation training initiative that included a set of online training modules, an in-person workshop, and technical support from evaluation consultants.
Main Outcome Measure(s): The successful start and/or completion of a program or policy evaluation focused on an IVP intervention.
Results: Of the 13 teams studied, a total of 12 teams (92%) reported starting or completing an evaluation. Four teams (31%) reported fully completing their evaluations; eight teams (61%) reported partially completing their evaluations. Teams identified common facilitators and barriers that impacted their ability to start and complete their evaluations. Nearly half of the 13 teams (46%) – whether or not they completed their evaluation – reported at least one common improvement made to a program or policy as a result of engaging in an evaluative process.
Conclusion: Practitioner-focused evaluation trainings are essential to build critical evaluation skills among public health professionals and their multidisciplinary partners. The process of evaluating an intervention—even if the evaluation is not completed—has substantial value and can drive improvements to public health interventions. The Evaluation Institute can serve as a model for training public health practitioners and their partners to successfully plan, start, complete, and utilize evaluations to improve programs and policies.
Keywords: Evaluation; injury; multidisciplinary partnerships; practitioner-focused evaluation training; professional development; program and policy evaluation; public health; technical assistance; violence
Use of Geographic Information Systems by American Evaluation Association Members in their Professional Practice
Authors: Kates Aaron W. , Coryn Chris L. 
Published: April 2023

Background: As geographic information systems (GIS) technology continues to develop and expand in its capacity and applications, it is becoming increasingly useful to many disciplines. Even so, little has been written about the place of GIS technology in evaluation practice, and there is a paucity of information as to the extent to and applications for which evaluation practitioners use such technology.
Purpose: In this investigation, the prevalence and common applications of GIS technology in professional evaluation practice are examined. Particularly, the study was designed to estimate what proportion of American Evaluation Association (AEA) members who self-identify as evaluation practitioners use GIS in their practice, if at all, and, if so, to what extent. For those who use GIS in their evaluation practice, the specific GIS software packages and applications used also are explored.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: A simple random sample of American Evaluation Association (AEA) members were surveyed, with an emphasis on evaluation practitioners.
Findings: Less than less than half (41.04% ±6.09%) of AEA members who consider themselves evaluation practitioners have ever used GIS in their evaluation practice and less than one-third (31.47% ±5.75%) have received some form of training in GIS methods. Data visualization is, by far, the most frequent application of GIS in evaluation practice.
Keywords: American Evaluation Association; geographic information systems; technology in evaluation; evaluation practice; research on evaluation
Cultural Humility: A Collaborative Approach to Recruiting Patients with Deliberate Self-Harm into a Multi-Hospital Randomized Controlled Trial
Authors: Brownhill Suzanne H. , Stevens Garry J. , Hammond Trent E. , Baldacchino Richard , Maposa Rose , Makoni Bonyface , Sheb'a Aviva , Andepalli Jagadeesh , Kotak Anita , D'Souza Olav , Atashnama Azadeh , Thayil Anish , Jones Alison 
Published: April 2023

Objectives: The ‘SMS SOS’ Deliberate Self-Harm (DSH) Aftercare Study was conducted in Western Sydney, Australia (October 2017 to December 2020) across three large public hospitals. During this randomized controlled trial (RCT), it was observed that knowledge exchange between key stakeholders and their ‘cultural’ perspectives (for example, Mental Health Clinicians, Lived Experience Mental Health Consultants—Patient Representatives, Administrative Officers, and Researchers) was essential to effective recruitment of patients experiencing DSH. Knowledge exchange within and between cultural groups was maximised and assessed using a communication matrix. This process, transferable to other trials engaging multiple ‘cultures’, aimed to promote the early identification of wider-team strengths as well as active management of emergent issues that would otherwise impede patient recruitment, and to maximise funding and human resources.
Methods: A descriptive study was conducted with a convenience sample of team members who represented different cultures in the study. Qualitative data were elicited from a ‘know and tell’ matrix. Through an iterative process, themes were generated that encapsulated what team members needed to know from and tell to their colleagues concerning the study.
Results: Factors that impacted participation in the study included clinician workload, the level of motivation/ commitment/confidence of clinicians to recruit patients, clinician-patient engagement, perception and expectations of study involvement, inter-cultural communication, and clinician training and support. The findings of this multidisciplinary consultation informed a composite model of knowledge exchange and the development of educational briefing/ orientation modules that make explicit team members’ roles and responsibilities to foster group member participation and enhance patient recruitment.
Conclusions: It is incumbent upon multidisciplinary team members of large-scale studies to adopt a similar ‘knowledge exchange’ strategy early in the planning and design stage. Adoption of such a strategy has the potential to mitigate risk of delay in project timelines, improve project outcomes, and ensure the efficient use of research funding, particularly in newly established research teams within clinical settings and with members newer to formal research collaborations.
Keywords: cultural humility; deliberate self-harm; engagement; participant recruitment; participatory research; randomized controlled trial
Measuring Mental Health Literacy: Development of the Mental Health Awareness and Advocacy Assessment Tool
Authors: Aller Ty B. , Fauth Elizabeth B. , Novak Joshua R. , Schwartz Sarah 
Published: April 2023

Background: Mental health literacy programs are a common community-based approach used to address the prevention of mental health issues on college campuses. Current assessment strategies used to evaluate the effectiveness of these programs often lack strong theoretical rational and psychometric rigor.
Purpose: The purpose of this study was twofold. First, based upon extant literature, theory, and standard clinical practice, we propose a process-based model of mental health literacy that includes three macro factors—identifying mental health issues, locating empirically based resources, and responding to mental health issues—and three micro processes of how they unfold—acquiring knowledge, building self-efficacy, and applying skills (behavior). The second aim was to test the psychometric properties of a new tool created to evaluate this process-based model—the Mental Health Awareness and Advocacy Assessment Tool (MHAA-AT).
Setting: Not applicable.
Intervention: Not applicable.
Research Design: A national sample of 296 college attending participants were recruited from Amazon’s Mechanical Turk. Participants responded to a demographic questionnaire and the newly developed MHAA-AT. Psychometric properties were examined through item response theory, exploratory factor analyses, and bivariate correlations.
Findings: Results suggest the MHAA-AT is a sound measure and demonstrates appropriate item, person, and trait characteristics on declarative knowledge items, and single factor structures on self-efficacy and behavior items with moderate to high reliability and validity. While additional testing is need among other samples, results suggest that the MHAA-AT is a quality assessment tool.
Keywords: College students; mental health literacy; item response theory; measurement
Emergent Developmental Evaluation Developments
Authors: Patton Michael Q. 
Published: April 2023

Background: Developmental Evaluation (DE) continues to develop. After reviewing the niche and history of DE, this article identifies five new and emergent purposes for DE to add to the original five – and forecasts future developments for diverse, innovative, and adaptive DE applications. An extended example of DE in supporting pivoting in the pandemic illustrates DE as adaptation to crisis as distinct from supporting social innovation.
Purpose: To update DE.
Setting: Global.
Data Collection and Analysis: Not applicable.
Findings: The applications and uses of DE have expanded beyond the original framing.
Keywords: adaptation; developmental evaluation; pivot; social innovation
Choosing an Evaluation Theory: A Supplement to Evaluation Roots (3rd Edition)
Authors: Alkin Marvin C. , Christie Christina A. , Stephen Naomi A. 
Published: April 2023

Background: Unlike scientific theories, evaluation theories are prescriptive: a set of actions and approaches that should be followed when conducting an evaluation. While evaluation theorists have offered a variety of writings describing their theories and approaches, few have offered a specific outline of what the theory looks like in practice. Thus, Alkin and Christie formulated a book to aid evaluators in how to apply theories in evaluations (Alkin & Christie, forthcoming). This book culminates in a series of prototypes that outline each theory’s goals, appropriate contexts, prescriptions, and observable actions in application.
Purpose: In order to aid evaluators in applying theories, this article seeks to provide a basis for comparison that can be used to help evaluators select which theory would be most appropriate in their practice.
Setting: This comparison can be applied in any setting where evaluations fit the context prescribed by each of the theories.
Intervention: Not applicable.
Research Design: Not applicablre.
Data Collection and Analysis: Not applicable.
Findings: In order for theories to influence practice effectively, theories must be displayed in a way that allows for easy comparison. This comparison of three theory prototypes demonstrates that prototypes can be an effective way for selecting a prescriptive theory when conducting an evaluation.
Keywords: prescriptive theories; practice; empowerment evaluation; learning centered model; developmental evaluation
Beyond GDP: Tracking and Evaluating National Contributions to Social and Environmental Sustainability
Authors: Picciotto Robert 
Published: April 2023

Background: The Gross Domestic Product (GDP) emerged as a convenient measure of national economic activity during the Great Depression. It was subsequently adopted by international development economists to track developing countries’ progress so that, despite its severe deficiencies, it became ‘locked in’ by habit, convenience, and policy makers’ preferences.
Purpose: This article conceives of GDP as a social intervention fit for evaluation. It shows that the GDP has had a pervasive and pernicious influence on policy making. Since past strategies aimed at dethroning the GDP have failed, it proposes new, evaluator-driven approaches designed to undermine the GDP’s dominance in the global market economy.
Setting: The Stiglitz report commissioned in the wake of the 2008 financial crisis launched a ‘Beyond GDP’ movement. Since then, public alarm about the GDP growth addiction has escalated: the drawbacks of GDP as a free-market policy tool have become self-evident as the rich get richer, the ranks of the poor swell and the future of the planet hangs in the balance.
Research Design: Not applicable.
Data Collection and Analysis: For the twenty largest economies in the world, the article estimates climate change discounts to the GDP based on official CO2 emissions statistics and a social cost of carbon estimate derived from a 2015 survey of eminent climatologists. It also draws on composite indexes generated by four reputable social research organizations to rank countries for their contributions to the 5 Ps of the Sustainable Development Goals (SDGs): people, planet, prosperity, peace, and partnership.
Findings: Pending the results of on-going efforts to upgrade worldwide statistics focused on the 169 SDG targets, the proposed GDP discounts help track progress towards the SDGs. But monitoring is not enough. In a policy world dominated by vested interests, the new ‘Beyond GDP’ indicators should be combined with principled, evaluator-directed evaluations.
Keywords: Beyond GDP; climate change; evaluator-directed evaluation; Gross Domestic Product; indicators; Sustainable Development Goals
Empowerment Evaluation of Programs Involving Youth
Authors: Heath Sarah , Moreau Katherine 
Published: April 2023

Background: Participatory and collaborative evaluation approaches, including Empowerment Evaluation (EE), are useful for evaluating programs involving youth. Empowerment evaluation involves stakeholders in the evaluation process through a set of structured steps. It is primarily concerned with empowering, illuminating, and building program beneficiaries’ self-determination. Given the emphasis that EE places on inclusivity of stakeholders, it appears to be a good fit for evaluating programs that involve youth.
Purpose: To explore the extent to which evaluators use EE to evaluate programs involving youth as well as what factor(s) facilitate and hinder their use of EE in these programs.
Setting: The study involved evaluators associated with the Collaborative, Participatory and Empowerment Evaluation and Youth-Focused Evaluation Targeted Interest Groups (TIGs) of the American Evaluation Association (AEA) who are involved in evaluating programs targeted at youth.
Intervention: Not applicable.
Research Design: We used a two-phase sequential mixed-methods research design. In Phase 1, we surveyed evaluators. In Phase 2, we interviewed a sample of evaluators from Phase 1.
Findings: In Phase 1, 41 (53.9%) respondents indicated not using EE to evaluate programs involving youth, 30 (39.5%) had used EE and 5 (6.6%) were unsure. Of those who used EE, they used it to teach youth program stakeholders about evaluation (n=8, 24.2%), produce more authentic results by engaging youth as experts of their lived experience (n=7, 21.2%) or produce more useful results for stakeholders to use (n=6, 18.2%), as well as other less popular reasons. In Phase 2, 12 interviewees raised five factors that facilitate or hinder the use of EE to evaluate programs involving youth including, evaluator perceptions, type of evaluation experience, evaluator knowledge and professional training, guidelines from organizations and funders, and stakeholders and time. Factors that some interviewees viewed as facilitators others viewed as hinderances.

Keywords: empowerment evaluation, program evaluation, youth-focused evaluation
The Contribution of Metaevaluation to Program Evaluation: Proposition of a Model
Authors: Hedler Helga , Ribeiro Namara G. 
Published: April 2023

Background: This theoretical article points to the fundamental difference between meta-analysis and metaevaluation. A model of metaevaluation for social programs is presented based on prior practical research.
Purpose: The purpose is to present a model for metaevaluation as a tool that can be used in other studies. Theory points to the need of a qualitative framework to go beyond the understanding of meta-analysis for program evaluation.
Setting: This theoretical article is based on an empirical research conducted at a Brazilian Governmental audit agency.
Subjects: The Government agency where the practical research was conducted is responsible for the effectiveness and accountability of social programs through audits that occurred from 2003 to 2006.
Intervention: Meetings and interviews were held with auditors that participated in the evaluation process going from planning to final reports as the model proposes
Research Design: The model for metaevaluation has a qualitative approach used to evaluate prior evaluations for social programs.
Data Collection and Analysis: Data collection included structured interview with the chief manager of the agency in charge of evaluating governmental programs. Documents and reports were analyzed using qualitative method for content analysis. Synthesis of categories was applied to compare different analysis and summarize findings.
Findings: Metaevaluation and meta-analysis are different research methods with a different approach. Meta evaluation is a qualitative method useful when evaluating prior evaluations. Yet the quantitative approach of meta-analysis applies better for first evaluations. Meta evaluation may include other methods to help strengthen the evaluation results.
Conclusions: Metaevaluation aligns theory and practice for program evaluation. The proposed model for metaevaluation may hold value for future theoretical and empirical work.
Inclusiveness and Social Justice in Evaluation: Can the Transformative Agenda Really Alter the Status Quo? A Conversation with Donna M. Mertens
Authors: Tarsilla Michele 
Published: April 2023

Background: Evaluation is sometimes viewed as a professional practice rather than a discipline corresponding to a well defined set of theories. However, Shadish, Cook and Leviton (1991) were able to demonstrate that evaluators’ work does have theoretical foundations. In particular, the authors identified five main elements for evaluation theory and described the contribution made to each of them by seven of the most influential scholars in the field over the last five decades. Purpose: This paper intends to further the discussion on evaluation theory, by examining some of the contributions made by Donna Mertens. The main focus of the paper is on her innovative ideas on each of the five main elements of evaluation theory. Setting: Not applicable. Subjects: Not applicable. Research Design: Not applicable. Data Collection and Analysis: The paper is the result of both a desk review of her work on transformative evaluation, inclusiveness, and social justice, and a phone interview with her. For the sake of accuracy, the text of the interview and the corresponding analysis were submitted to Mertens for review prior to publication. Findings: The author demonstrates how Mertens has incorporated the five principles into her own work and practice. According to Mertens, evaluators need first to be cognizant of the plurality of values held by the communities where they work and second to let those very same values guide the design, implementation, and use of evaluation. Based on her belief that knowledge is not neutral but is influenced by human interests, Mertens calls upon evaluators to play a more active role in the construction of new and “socially compensatory” knowledge. In doing so, Mertens particularly stresses how evaluation practice could address and challenge the social, cultural and economic inequities perpetuated by the status quo. Conclusions: Pursuing inclusiveness and social justice in evaluation is possible. Transformative evaluation is capable of altering the status quo. Transformative evaluation is instrumental in bringing about not only social change but also social transformation. This article also shows that the debate on evaluation’s main theoretical foundations is still relevant. Such theoretical categories are especially beneficial in that provide common ground for understanding. Keywords: transformative evaluation, social justice, cultural competence, inclusiveness, valuing, social programming, knowledge, practice, use
An In-Depth International Comparison of Major Donor Agencies: How Do They Systematically Conduct Country Program Evaluation?
Authors: Sasaki Ryo 
Published: April 2023

Background: This paper presents an in-depth international comparison of systems and procedures of aid evaluation, focusing on Country Program Evaluation among major donor agencies. The original client of this study is Ministry of Foreign Affairs, Japan (MOFAJ). Purpose: The purposes of this paper are set as follows: (1) to understand how aid agencies conduct Country Program Evaluation; and (2) to make recommendations for improvement of the current practice of Country Program Evaluation in the aid evaluation community. Setting: The examined donors include: the World Bank (WB), the Asian Development Bank (ADB), the Inter-American Development Bank (IADB), the United Nations Development Programme (UNDP), the U.S. (USAID), Canada (CIDA), the U.K. (DFID), the Netherlands (IOB), Germany (BMZ), France (Foreign Ministry), and Japan (Ministry of Foreign Affairs (MOFAJ)). In addition, aid agencies conducting respective project evaluation are also examined, and they are JICA (Japan), GTZ and KfW (Germany) and AFD (France). Intervention: This study presents the result of comparative analysis among those donor agencies in terms of the following viewpoints: (1) evaluation criteria employed; (2) approaches to evaluate “effectiveness” and “impact”; (3) attribution issue; (4) the use of a rating system; and (5) overall evaluative conclusion and integrating methods. All viewpoints are focusing on Country Program Evaluation. One conclusion is that most agencies have been struggling with how to judge the degree and value of their country programs. Data Collection and Analysis: Mixed methodologies were employed to collect data from the said donor agencies. The analysis was conducted by a systematic procedure consisting of: (i) summarizing information in a comparative table; (ii) trying to make groups/categories based on common characteristics if possible; and (iii) examining and concluding basic thoughts/philosophy which make their differences. Findings: This study made some new knowledge about how aid agencies conduct Country Program Evaluation and identified several issues remained. Varieties of their practices are observed and it is far from the unified methods agreed. Some remarkable points identified in this study are:(1) Most aid agencies invoke the DAC five evaluation criteria for Country Program Evaluation. (Major exception was USAID); (2) “Strategic relevance” and “coherence/complementarity” are the emerging new criteria; (3) Attribution is still the issue that aid agencies have struggled; and (4) The attitude for introduction of rating system is clearly divided among aid agencies.