bluuebunny/crossref_metadata_2025_split huggingface dataset:
Learning How to Learn in Sustainability Transitions Projects: The Potential Contribution of Developmental Evaluation
Authors: Mitchell Andrew S. , Lemon Mark 
Published: April 2023

Background: Community-based sustainability transitions projects are increasingly being considered for their potential as policy delivery vehicles for the UK government Climate Change Act commitments. At the same time, project funders seek reassurances that their investments are relevant in helping communities mitigate, and adapt to, the effects of climate change. Despite this increased pressure, recent research suggests that evaluations of such sustainability transitions projects have, on the one hand variable impacts, or impacts that are of an inadequate scale, duration, or type, or on the other, that project staff lack the capacity or resources to undertake monitoring and evaluation to the degree of rigour expected by policy makers and funders.
Purpose: This article reports on an extended case study of a fully-funded five year community-based sustainability transitions project in Leicestershire, England. In particular, it reviews the deployment of developmental evaluation (DE) methods in an attempt to capture the project team’s learning about doing community-based sustainability work.
Setting: A funded community-based sustainability transitions project in a south Leicestershire market town.
Intervention: Developmental evaluation methods were used to capture project-based learning as a resource for project innovation and adaptation.
Research Design: Phronetic case study.
Data Collection and Analysis: Participant-observation, action research, focus and special issue group facilitation.
Findings: Use of a developmental evaluation method identified key learning points for the project actors; these focused on how the project had adapted to the complexities of the operating environment through innovations in second-order learning or learning how to learn. The paper makes recommendations for the design and funding arrangements of community-based sustainability transitions initiatives and developmental evaluation is endorsed as a viable and promising adjunct to more traditional impact, economic, and process evaluation methodologies.
Keywords: developmental evaluation; second-order learning; community-based sustainability; phronesis; action research; project design.
A Critical-Historical Review of Program Evaluation and the Emerging Motif ‘Evaluation Science’
Authors: Fear William James 
Published: April 2023

Background: It is important to distinguish between evaluation as an inherent, automatic, affective process and Program Evaluation (Evaluation, with capitalised ‘E’) as an institution, and equally important to consider what a good understanding of evaluation tells us about Evaluation. Evaluation is an established social institution whose modern roots can be traced back to 16th century France. Since the early 1900s the institution has developed within and across a range of scientific disciplines with interests in perceived social problems and efforts to resolve the said problems. This can be demonstrated objectively by the number and scale of relevant publications within relevant disciplines. This, in turn, helps us understand more about Evaluation as an institution. Set in this context is the question of Evaluation Science: is this simply a fashionable institutional motif or is it a potential new era for Evaluation?
Purpose: Commentary on the history and development of Program Evaluation.
Setting: Not applicable.
Intervention: Not applicable.
Research design: Not applicable.
Data collection & analysis: Not applicable.
Findings: Not applicable.
Keywords: program evaluation; evaluation; history; evaluation science.
Evaluation Policy and Organizational Evaluation Capacity Building: Application of an Ecological Framework across Cultural Contexts
Authors: Al Hudib Hind , Cousins J. Bradly 
Published: April 2023

Background: Research on the role and effects of evaluation policy is limited. Some research on the policy’s role in enhancing organizational evaluation capacity (EC) is beginning to accrue but to date it has been limited largely to global Western evaluation contexts.
Purpose: We employed an ecological conceptual framework arising from our own empirical research to explore the interface between evaluation policy and EC in non-western contexts. We asked—To what extent does this framework resonate across these contexts? In the selected non-Western context, what are the salient variables moderating the relationship between policy and EC in the selected contexts? Are there differences across countries?
Setting: The present research is focused on perceptions about evaluation culture and experiences in two countries situated in the Middle East and North Africa (MENA) region, namely Turkey and Jordan.
Intervention: Not applicable.
Research design: We conducted focus groups within the respective countries with a combined total of 18 participants associated with country-level voluntary organizations for professional evaluation (VOPE). Participants worked in government, non-governmental aid agencies, universities and private sector organizations.
Data collection and analysis: We introduced the focus group participants to our ecological framework and then guided the conversation using semi-structured questions. Data were audio-recorded, transcribed and subsequently thematically analyzed using NVivo.
Findings: The ecological framework was found to resonate well but the findings were weighted heavily toward macro-level contextual variables. Even though important contextual and cultural differences between Turkey and Jordan were evident, leadership emerged as a significant meso-level moderating variable in both settings. The discussion of the results included implications for ongoing research.
Keywords: Evaluation capacity building; evaluation policy; program evaluation; cultural context
Using Cognitive Interviewing to Test Youth Survey and Interview Items in Evaluation: A Case Example
Authors: LaPietra Elisa , Brown Urban Jennifer , Linver Miriam R. 
Published: April 2023

Background: Cognitive interviewing is a pretesting tool used by evaluators to increase item and response option validity. Cognitive interviewing techniques are used to assess the cognitive processes utilized by participants to respond to items. This approach is particularly appropriate for testing items with children and adolescents who have more limited cognitive capacities than adults, vary in their cognitive development, and have a unique perspective on their life experiences and context.
Purpose: This paper presents a case example of cognitive interviewing with youth as part of a national program evaluation, and aims to expand the use of cognitive interviewing as a pretesting tool for both quantitative and qualitative items in evaluation studies involving youth.
Setting: Youth participants were located in four regions of the United States: Northeast, Central, Southern, and Western. Interviewers were located at Montclair State University.
Intervention: Not applicable.
Research design: A cognitive interview measure was designed to include a subset of survey items, interview questions, and verbal probes, to evaluate if these items and questions would be understood as intended by both younger and older youth participants. An iterative design was used with cognitive interviewing testing rounds, analysis, and revisions.
Data Collection and Analysis: The cognitive interview was administered by phone to 10 male youth, five from the 10-13-year-old age range and five from the 15-17-year-old age range. Interviews were audio-recorded, transcribed, reviewed, and coded. Survey items and interview questions were revised based on feedback from the participants and consensus agreement among the evaluation team. Item revisions were included in further testing rounds with new participants.
Findings: As a result of using cognitive interviewing to pretest survey and interview items with youth, response errors were identified. Participants did not understand some of the items and response options as intended, indicating problems with validity. These findings support the use of cognitive interviewing for testing and modifying survey items adapted for use with youth, as well as qualitative interview items. Additionally, the perspective of the youth participants was valuable for informing decisions to modify items and helping the evaluators learn the participants’ program culture and experiences. Based on the findings and limitations of the study, we give practice recommendations for future studies using cognitive interviewing with a youth sample.
Keywords: cognitive interviewing; item validity; response error; verbal probes; pre-testing surveys; qualitative evaluation; interviewing children and adolescents; survey development
Dialogic and Generative Reflection: An Application of the Critical Appreciative Process in Program Evaluation
Authors: He Ye , Smith Tiffany L. , Moteane Malitsitso 
Published: April 2023

Background: Even though the positive potential of reflective practice is widely acknowledged across professional fields, it has been recognized that reflective practice may be carried out primarily as an individual-based exercise, and at the technical or descriptive level without generative impact. Dialogic reflective processes involving both evaluators and program directors are far from being systematically implemented or examined.
Purpose: The purpose of this article is to share our experiences engaging in dialogic and generative reflections as the project director and program evaluators of a K-12 teacher education program using the critical appreciative process. Building upon the reflective practice traditions in both disciplinary areas, we introduce the use of the critical appreciative process as a promising model to guide dialogic and generative reflection to support the co-design and improvement of the program and accompanying evaluation efforts.
Setting: The project director and evaluators are engaged in a grant-funded teacher preparation project designed to prepare teachers for K-12 English learners and dual language learners. The project builds upon partnerships between the university teacher preparation program and two local school districts. The evaluation plan was designed based on culturally responsive, collaborative, and use-focused evaluation approaches and theory. In 2020, the project team faced critical decisions in the context of the COVID-19 pandemic.
Intervention: Not applicable.
Research Design: We applied self-study methodology to guide data collection and analysis in this study. The primary data source included individual written reflections and group critical friend dialogues guided by the critical appreciative process. Both the reflections and meeting notes were analyzed to identify convergent and divergent perspectives shared throughout the critical appreciative process and to highlight implications for both the evaluation and the program moving forward.
Findings: Convergent and divergent perspectives from both the project director and the evaluators were shared based on the 4-D critical appreciative process: Discover, Dream, Design, and Deliver. Based on this shared experience, we illustrate how the dialogic reflective process entails reflexivity and requires pausing; how reflective practice in program evaluation situates our dialogues as learning-oriented rather than a mere accountability discussion; and how reflective action can create a dialogic and generative virtuous cycle.
Keywords: reflective practice; evaluation; critical appreciative process
Program Evaluation Development in the Newly Independent States
Authors: Karimov Afar , Borovykh Alexander , Kuzmin Alexey , Abdykadyrova Asel , Efendiev Djahangir , Greshnova Ekaterina , Konovalova Elena , Frants Inessa , Palivoda Liubov , Usifli Seymour , Balakirev Vladimir 
Published: April 2023

This paper provides a general overview of the development of program evaluation in CIS countries. We start by telling a story that describes how evaluation appeared in the scene, how it developed and who the key players were in its development. We discuss the issue of demand for and supply of evaluation services. In the conclusion, we describe our view of the stages in the emergence of evaluation in the CIS and the perspective of each. The four stages we identify are: “An External Phenomenon” (1991–1995), “Initiation” (1996–2000), “Assimilation” (2000–2004), and “Implementation” (2005 to the present).
Analysis of Paired Dichotomous Data: A Gentle Introduction to the McNemar Test in SPSS
Authors: Adedokun Omolola A. , Burgess Wilella D. 
Published: April 2023

Background: Although McNemar Test is the most appropriate tool for analyzing pre-post differences in dichotomous items (e.g., “yes” or “no”, “correct” or “incorrect”, etc.), many scholars have noted the inappropriate use of Pearson’s Chi-square Test by researchers, including social scientists and evaluators, for the analysis of related or dependent dichotomous variables. Purpose: The goal of this paper is to promote the use of McNemar Test among evaluators by providing a gentle introduction to the method. Setting: Not applicable. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Using data from 506 6th grade students’ responses to a pre-post science test; this contribution illustrates how to conduct McNemar Test in SPSS.. Findings: This contribution provides a non-technical introduction to McNemar test and illustrates its use in an applied research/evaluation context. Keywords: McNemar Test; paired dichotomous data; Chi-square test; equality of proportions.
‘Policing Schools’ Strategies: A Review of the Evaluation Evidence
Authors: Petrosino Anthony , Guckenburg Sarah , Fronius Trevor 
Published: April 2023

Background: Schools experience a wide range of crime and disorder, victimizing students and staff, and undermining attempts to create a safe and orderly environment for student learning. Police have long established programs with schools, but there has been no systematic review of evaluations of these programs, outside of police-led prevention classroom curriculum programs such as D.A.R.E. Purpose: This paper documents a systematic search to identify experimental and quasi-experimental evaluations that assess the effectiveness of non-educational policing strategies and programs in schools. Setting: Included studies took place in or around K-12 schools in the United States, Canada, and the United Kingdom. Intervention: Studies were included if they reported on a specific school-based strategy that heavily involved police and did not exclusively involve the police teaching a curriculum or program such as Drug Abuse Resistance Education (D.A.R.E.). Research Design: Systematic review of experimental or quasi-experimental evaluations Data Collection and Analysis: Only those impact studies that used experimental or quasi-experimental design, had at least one outcome measure of school crime or disorder, and were available through December 2009 were eligible. Electronic searches and other methods were used to identify published and unpublished evaluation reports. Findings: The searches identified a total of eleven quasi-experimental studies. Ten of the eleven studies would likely have received a “3” on the Maryland Scientific Methods Rating Scale, a common approach to classifying studies on the basis of internal validity. If evidence rating criteria from the U.S. Department of Education’s What Works Clearinghouse (WWC) were applied, only one study would likely receive a grade of “Level 2” evidence (acceptable with reservations) and the other ten studies would likely not meet WWC evidence screening criteria. Keywords: systematic review; police and schools; crime prevention
Methodological Pathways to Improvements of Evaluation Approaches: The Case of Irrigated Agriculture Performance Evaluation
Authors: Chaponnière Anne , Marlet Serge , Perret Sylvain R. , Bouleau Gabrielle , Zaïri Abdelaziz 
Published: April 2023

Background: Irrigated agriculture is often evaluated but few reviews of evaluation methodology adapted to this object are available in the literature. Besides, recommendations to improve evaluation in this field are lacking. Purpose: The purpose of the paper is to contribute filling this gap. Setting: Not applicable. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Desk review. Findings: This review shows the evolution of evaluation methodology in the field of irrigated agriculture pointing out a trend towards more comprehensive methodologies. The review also suggests some methodological tools to improve evaluation process. Keywords: irrigated agriculture; evaluation methodology; plurality of evaluation; intended users; viewpoint
Utilizing Generalizability Theory to Investigate the Reliability of Grades Assigned to Undergraduate Research Papers
Authors: Gugiu Mihaiela R. , Gugiu Paul C. , Baldus Robert 
Published: April 2023

Background: Educational researchers have long espoused the virtues of writing with regard to student cognitive skills. However, research on the reliability of the grades assigned to written papers reveals a high degree of contradiction, with some researchers concluding that the grades assigned are very reliable whereas others suggesting that they are so unreliable that random assignment of grades would have been almost as helpful. Purpose: The primary purpose of the study was to investigate the reliability of grades assigned to written reports. The secondary purpose was to illustrate the use of Generalizability Theory, specifically the fully-crossed two-facet model, for computing interrater reliability coefficients. Setting: The participants for this study were 29 undergraduate students enrolled in an introductory-level course on Political Behavior in Spring 2011 at a Midwest university. Intervention: Not applicable. Research Design: Students were randomly assigned to one of nine groups. Two-facet fully crossed G-study and D-study designs were used wherein two raters graded four assignments for 9 student groups—72 evaluations in total. The universe of admissible observations was deemed to be random for both raters and assignments, whereas the universe of generalization was deemed to be mixed (random for two raters but fixed for four assignments). Data Collection and Analysis: The semester-long project was assigned to groups consisting of an annotated bibliography, survey development, sampling design, and analysis and final report. Four grading rubrics were developed and utilized to evaluate the quality of each written report. Two-facet generalizability analyses were conducted to assess interrater reliability using software developed by one of the authors. Findings: This study found a very high interrater reliability coefficient (0.929) for only two raters who received no training in how to use the four grading rubrics. Keywords: grading; reliability; Generalizability Theory; writing
What’s in a Scriven Number?
Authors: Roy Abhik R. , Hobson Kristin A. , Coryn Chris L. S. 
Published: April 2023

Background: Other than Leonhard Euler, one of the greatest physicists and mathematicians ever to have lived and dubbed “…c'est notre maître à tous” or “…the master of us all” by the French mathematician and astronomer Pierre-Simon Laplace, Paul Erdös was a mathematician who published more papers in mathematics than anyone else (Hoffman, 1998). Because of this feat, friends and colleagues created what has been known as an Erdös number. An Erdös number describes a person’s degree of separation from Erdös through collaborations directly with him or with others who have collaborated with him. Purpose: In this paper, we present reasoning similar to that of an Erdös number for Michael Scriven who is widely considered to be one of the leading theorists in evaluation. With his numerous publications and influence on the theory and practice of evaluation, we present not only the hope, but also the need for a Scriven number. Setting: Not applicable. Intervention: Scriven number. Research Design: Not applicable. Data Collection and Analysis: Not applicable. Findings: A collaborative count such as a Scriven number would centralize evaluation. A Scriven number would form new connections and collaborations, thus yielding a robust connectivity. Keywords: Scriven number
Parental willingness to have children vaccinated against COVID-19 in Geneva, Switzerland: a cross-sectional population-based study
Authors: Baysson Hélène , Pullen Nick , De Mestral Carlos , Semaani Claire , Pennacchio Francesco , Zaballa María-Eugenia , L’Huillier Arnaud G. , Lorthe Elsa , Guessous Idris , Stringhini Silvia , The Specchio-COVID19 study group  
Published: April 2023

OBJECTIVE: We aimed to examine factors associated with parental willingness to vaccinate their children against COVID-19.
METHODS: We surveyed adults included in a digital longitudinal cohort study composed of participants in previous SARS-CoV-2 serosurveys conducted in Geneva, Switzerland. In February 2022, an online questionnaire collected information on COVID-19 vaccination acceptance, parental willingness to vaccinate their children aged ≥5 years and reasons for vaccination preference. We used multivariable logistic regression to assess the demographic, socioeconomic and health-related factors associated with being vaccinated and with parental intention to vaccinate their children.
RESULTS: We included 1,383 participants (56.8% women; 69.3% aged 35–49 years). Parental willingness to vaccinate their children increased markedly with the child’s age: 84.0%, 60.9% and 21.2%, respectively, for parents of adolescents aged 16–17 years, 12–15 years and 5–12 years. For all child age groups, unvaccinated parents more frequently indicated not intending to vaccinate their children than vaccinated parents. Refusal to vaccine children was associated with having a secondary education (1.73; 1.18–2.47) relative to a tertiary education and with middle (1.75; 1.18–2.60) and low (1.96; 1.20–3.22) household income relative to high income. Refusal to vaccine their children was also associated with only having children aged 12–15 years (3.08; 1.61–5.91), aged 5–11 years (19.77; 10.27–38.05), or in multiple age groups (6.05; 3.22–11.37), relative to only having children aged 16–17 years.
CONCLUSION: Willingness to vaccinate children was high for parents of adolescents aged 16–17 years but decreased significantly with decreasing child age. Unvaccinated, socioeconomically disadvantaged parents and those with younger children were less likely to be willing to vaccinate their children. These results are important for vaccination programs and developing communication strategies to reach vaccine-hesitant groups, both in the context of COVID-19 and in the prevention of other diseases and future pandemics.