bluuebunny/crossref_metadata_2025_split huggingface dataset:
Using Graphical Perception Principles to Improve the ST Tools’ Data Visualization: Revisiting the Systems Dynamics Model
Authors: Vaca Sara , Vidueira Pablo 
Published: April 2023

Background: Systems Thinking (ST) is the new paradigm in Evaluation. It represents a significant mind-set shift for the evaluation field and it is a powerful tool to tackle complex environments. Heir to the systems concepts of the engineering field, and especially regarding the hard systems tools, ST in evaluation uses the same visual tools that were created many years ago. All these tools already incorporate data visualization features: they depict ideas, relationships and concepts relying in shapes and figures more than a textual explanation. Revisiting these tools and applying the latest data visualization principles, they could be optimised in order to provide with more information within the same concept. Purpose: To provide ST practitioners with more informative tools in order to facilitate:- ST experts and users can optimise the application of the tools to real life models beyond the initial set up of their visual representations.- Audiences of evaluations using ST as part of the toolkit can find the outputs more apprehensible and easy to understand . Setting: Not applicable. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Not applicable. Findings: Improving ST representations of reality and systems can help both enhance ST applications and make it more accessible and comprehensible for evaluation practitioners' and users. Six ways for improving the understanding of the current stock and flow diagrams were identified. The tools proposed consist of: customizing the colours and shape of the variables and their relationships to make them more informative; highlighting the existing subsystems within the model; and providing the specific sequence for reading the main causal chains.
Adaptation and Use of a Five-Task Model for Evaluability Assessment
Authors: Watts Brad R. , Washington Heather M. 
Published: April 2023

Background: Over four decades have passed since the concept of evaluability was introduced; however, the availability and accessibility of methodology, frameworks, checklists and other guidance on evaluability assessments remains limited (Smith, 2005). Evaluators who wish to conduct an evaluability assessment must adopt one of few existing models or operate without guidance. This case provides an example of one model and how it was utilized to conduct an evaluability assessment of an ongoing program intervention. Purpose: This article provides a real-world example of how an evaluability assessment was conducted using a five-task model from the criminal justice field. Setting: A criminal justice program operating with at-risk adolescent youth and police officers who patrol their neighborhoods. Intervention: NA Research Design: NA Data Collection and Analysis: The original evaluability assessment collected data through interviews with program personnel, observations by the evaluability researchers, and analysis of survey data provided by the program (i.e., program participants’ responses to survey questions about the program). Findings: The evaluability assessment conducted in this case study example identified multiple changes necessary for the program to be evaluable. Technical assistance was provided in order to support the efforts of the program to prepare for evaluation, though, the evaluability assessment identified risks and costs for the program to consider prior to proceeding. The findings were developed through a five-task process, which evaluators may be able to adapt for use with other types of programs undergoing evaluability assessment.
Re-Structuring Evaluation Findings into Useful Knowledge
Authors: Houston Danielle , Wright Bernadette , Wallis Steven E. 
Published: April 2023

Background: A long research stream has shown that when knowledge is more structured it is more likely to be effective in practical application. Building on that research, the authors applied Integrative Proposition Analysis to visualize, integrate, and assess the quality and usefulness of knowledge gained from the NMAC (formerly National Minority AIDS Council) Strong Communities evaluation.
Purpose: Demonstrate an innovative method to rigorously integrate and strengthen knowledge gained from evaluation and to encourage discussion of future directions for developing stronger theories for more effective evaluation and more effective action.
Setting: Birmingham, Alabama
Intervention: A project to identify local strategies for community-based organizations and community health centers that serve African American and Latinx gay and bisexual men and transgender women to collaboratively meet HIV-related community needs.
Research Design: The researchers applied Integrative Propositional Analysis to integrate and map concepts and causal connections emerging from the evaluation findings. The authors then analyzed the resulting map to identify top-mentioned concepts, better understood concepts, reinforcing loops, and knowledge gaps.
Data Collection and Analysis: Integrative Propositional Analysis applied to a literature review and stakeholder interview transcripts collected for the evaluation.
Findings: Integrating literature and interview results helped to identify several actions where providers of HIV-related services could increase their impact on combating the HIV epidemic among the communities they serve. The authors also identified a reinforcing loop; this shows opportunity to improve two desired outcomes by increasing one. In addition, the authors identified blank spots on the map; these show where additional research could strengthen the quality and usefulness of the mapped knowledge.
Examining Factors Impacting Online Survey Response Rates in Educational Research: Perceptions of Graduate Students
Authors: Saleh Amany , Bista Krishna 
Published: April 2023

Background: In educational research, online survey has become one of the most popular methods of data collection. Academic researchers, including faculty and students, expect and require a good response rate to their research projects for reliable results.
Purpose: In this paper, the authors examine a wide range of factors related to survey response rates in academic research. Examples include email checking habits, survey design, and attitudes toward research.
Setting: An online survey environment
Intervention: Not applicable.
Research Design: A cross-sectional quantitative research method was used to analyze the factors that influence participants’ email survey response rate. Data were collected at a single point in time. The authors did not directly measure changes that come over time in this study.
Data Collection and Analysis: After receiving the Institutional Research Board’s approval, the researchers distributed the survey via the American Educational Research Association (AERA) Graduate Student Discussion List subscribers. A sample of 454 responses was used in the final analysis-- with a 78.9 % response rate. The authors used descriptive statistics (percentage, average mean) and inferential statistics (chi-square and correlations) to report the data analysis and findings.
Findings: Results indicated that research survey response rate was highly influenced by interests of participants, survey structure, communication methods, and assurance of privacy and confidentiality. The findings also suggested that male participants were more likely to respond to surveys if they received a reminder, and older participants were more likely to respond if they were promised a reward.
Refining Theories of Change
Authors: Dhillon Lovely , Vaca Sara 
Published: April 2023

Background: Despite the disparities in how they are defined and what elements are included, most Theories of Change remain consistent in one way – their visual format. Typically, Theories of Change are presented as a one-page visual in a flowchart style with lines and boxes of uniform size. In addition, Theories of Change are often created as stand-alone tools that are rarely linked effectively to other organizational tools.
Purpose: The authors: (1) propose the essential elements that contribute to robust Theories of Change and clarify the characteristics that distinguish Theories of Change from other organizational tools and formats; (2) suggest additional elements for inclusion in the Theory of Change; (3) present graphic alternatives that allow for an evolution in representing their complexity and depth; and (4) provide ways to link Theories of Change to other organizational tools to increase organizational alignment, efficiency, and, most importantly, impact.
Research Design: NA
Data Collection and Analysis: NA
Setting: NA
Intervention: NA
Findings: NA
Using Program Theory to Evaluate a Graduate College Student Development Program
Authors: Gus Shannon , Keener Ashley , Bullard Chelsea E. , Gordon Sarah 
Published: April 2023

Background: The “3 Minute Presentation” is a graduate student competition based off the more popular “3 Minute Thesis” competition. The program aims to help graduate students learn to inform others of their research in a quick and accessible manner. Programs to engage graduate students more deeply in their education require evaluation to determine if they are useful and effective at meeting their intended goals. Evaluation literature in graduate educational programs is currently limited, but increasingly needed for both the field and the students served.

Purpose: Development and testing of a program-theory evaluation to understand participation, recruitment, preparation, training, skills, and confidence of graduate students engaging in a “3 Minute Presentation” competition at a state university.

Setting: Institution of Higher Education

Intervention: 3 Minute Presentation competition

Research Design: Mixed-method program-theory evaluation

Data Collection and Analysis: Direct observations and closed-ended survey analyzed through qualitative coding, descriptive statistics, group comparisons, and correlation analysis.

Findings: Overall, the program evaluation found, with a possible lack of diversity in participants, that the program components of recruitment, preparation, and skill development work as expected. Additionally, engagement in preparation was associated with competition scores and the perceived helpfulness of preparation was related to students’ confidence in their presentation skills. This evaluation was deemed useful for program improvement and capacity building in the program’s continuation at the university.
Method for Using Rubric Ratings on Fishbone Diagrams to Compare Case Studies
Authors: Davey Melinda E. , Morell Jonathan A. 
Published: April 2023

Background: In multi-case study program evaluations, the large amount of qualitative data that are generated from interviews can be difficult to utilize. This is particularly so when inference must be made as to why some cases succeed and some fail.
Purpose: This paper shows a method for comparing multiple evaluation sites by using a rubric to define ratings for relevant factors, and an Ishikawa fishbone diagram as a model to show relationships among those factors. We show how this technique identified reasons for differences in outcomes among the sites.
Setting: The evaluation setting was a large-scale safety innovation in the U.S. railroad industry. Four cases were considered—two passenger railroads and two freight railroads.
Intervention: The Confidential Close Call Reporting System (C3RS) program allowed railroad workers to confidentially submit “close calls” which were reviewed by a team made up of labor, management, and the Federal Railroad administration to determine ways to improve safety.
Research design: Multiple comparative case study, Ishikawa root and contributing cause modeling, evaluative rubric scoring, and data visualization techniques.
Data collection & analysis: Interview data were collected from four pilot railroad sites, each of which participated in a five-year test of C3RS. Testing periods overlapped, with the entire evaluation lasting about 12 years.
Findings: The method of using Ishikawa fishbone diagrams with ratings from an evaluative rubric was an effective method to summarize, analyze, and present large quantities of qualitative data. The approach succeeded in explaining degrees of success and failure across the sites. The sponsor and industry stakeholders were able to understand the analysis and the findings, and to develop deep insight into how to promote successful implementation.
Keywords: Multiple comparative case studies; qualitative methodology; qualitative coding; data visualization; fishbone diagrams; Ishikawa diagrams; evaluative rubrics; close calls; near miss; data visualization.
Mixed Methods Research in Designing an Instrument for Consumer-Oriented Evaluation
Authors: Gallant Dorinda J. , Luthy Nicole 
Published: April 2023

Background: The educational product market has been gradually shifting from primarily print to primarily digital content. Educators must make quick decisions when selecting materials that will assist students in their learning.
Purpose: Purposes of this study were to describe the application of a two-stage sequential mixed-method, mixed-model design in designing an instrument for consumer-oriented evaluation and to describe implications of using mixed methods research in developing a rubric to evaluate prekindergarten through Grade 12 digital content.
Setting: The Ohio State University, Columbus, OH.

Intervention: N/A.
Research design: A two-stage sequential mixed-method, mixed-model design.
Data collection & analysis: In Stage 1, a modified electronic Delphi survey technique was implemented with US geographically dispersed subject matter experts. In Stage 2, cross-sectional focus group interviews were conducted with local teachers, administrators, and textbook publishers.
Findings: Inclusion of multiple perspectives and viewpoints from teachers, administrators, textbook publishers, and experts on importance, clarity, and appropriateness of criteria to evaluate digital content resulted in a final version of the rubric that can be used by teachers and administrators to evaluate digital content that supports students’ learning in prekindergarten through Grade 12.
Refining and Measuring the Construct of Evaluative Thinking: An Exploratory Factor Analysis of the Evaluative Thinking Inventory
Authors: McIntosh Jason S. , Buckley Jane , Archibald Thomas 
Published: April 2023

Background: Evaluative thinking has emerged as a key construct in evaluation, especially for evaluation practitioners and researchers interested in evaluation capacity building (ECB). Yet, despite increasing calls for more research on evaluation and, more specifically, for more research on ECB, until recently little empirical inquiry on the dimensions of evaluative thinking has been conducted.
Purpose: To address that lack, the purpose of the study presented in this paper is to refine the construct of evaluative thinking by exploring its underlying dimensions and to ascertain the internal consistency of an instrument developed to measure evaluative thinking, the Evaluative Thinking Inventory (ETI).
Setting: The ETI was developed as part of an ECB initiative focused on non-formal science, engineering, technology, and math (STEM) education in the United States, and was tested as part of a study focused on evaluating gifted education programs, also in the United States.
Intervention: Not applicable.
Research design: Survey research and exploratory factor analysis (EFA).
Data collection & analysis: The ETI was administered to participants in a study measuring the effectiveness of a tool used to conduct internal evaluations of gifted education programs. SPSS was used to conduct an EFA on 96 completed ETIs. Cronbach’s alpha was used to estimate the internal consistency of the instrument.
Findings: The analysis of the ETI revealed a two-factor model of evaluative thinking (i.e., believe in and practice evaluation and pose thoughtful questions and seek alternatives). This study also provided internal consistency evidence for the ETI showing alpha reliabilities for the two factors ranging from 0.80 to 0.82. The ETI has potentially wide applicability in research and practice in ECB and in the field of evaluation more generally.
Keywords: evaluative thinking; evaluation capacity building; research on evaluation; exploratory factor analysis.
Using PhotoVoice as an Evaluation Method
Authors: Hunter Ouen , Leeburg Emma , Harnar Michael 
Published: April 2023

Background: Engaging with youth through PhotoVoice is beneficial as a program evaluation method and functions as a method of inquiry to understand youths’ perceptions of a college preparation program. The students used PhotoVoice to respond to prompts about how they learn and their opinions of the college preparation program.
Purpose: This reflection of practice article provides an example of PhotoVoice as an evaluation method.
Setting: This evaluation was conducted during the summer college preparation programming.
Intervention: The combination of student photography, photo gallery walk, and group discussion as an evaluation method.
Research Design: A qualitative reflective design.
Data Collection and Analysis: The student photographs, narratives, and observational notes were analyzed thematically.
Findings: We found that the youths enjoyed the unique experience of PhotoVoice. The combination of student photography, narratives, gallery walk, and group discussion, were useful as an inquiry tool for this youth program.
Process Evaluation to Document Crucial Moments in Development of the National Neurological Conditions Surveillance System at the U.S. Centers for Disease Control and Prevention
Authors: MacDonald Goldie , Mercer Shawna L , Fisher Angela H , Grigorescu Violanda , Mac Kenzie William R , Anderson Mindelyn 
Published: April 2023

Background: Neurological conditions or disorders strike roughly 50 million Americans annually but accurate and comprehensive national estimates for many of these conditions are not available. In 2019, Congress provided $5 million to Centers for Disease Control and Prevention (CDC) to establish the National Neurological Conditions Surveillance System (NNCSS). CDC focused initial activities on multiple sclerosis and Parkinson’s disease.
Purpose: We conducted a process evaluation to document and understand multifaceted work to implement a new surveillance activity for two neurological conditions.
Setting: We conducted this evaluation with government personnel internal to the Center for Surveillance, Epidemiology, and Laboratory Services at the Centers for Disease Control and Prevention in Atlanta, GA.
Intervention: A new public health surveillance activity for two neurological conditions, multiple sclerosis and Parkinson’s disease, that uses existing data resources and systems.
Research design: The evaluation included interviews with CDC personnel and review of administrative and programmatic information. Data were analyzed and interpreted to identify crucial moments in the first year of funded work on NNCSS. The study revealed that this surveillance activity required diverse contributions and collaboration within the federal government and with non-governmental organizations. The findings can be used to guide work to enhance surveillance for many neurological conditions.
Findings: The study revealed that this surveillance activity required diverse contributions and collaboration within the federal government and with non-governmental organizations. While collaboration is a cornerstone of public health practice, it is not always well-documented in planning or implementation of surveillance or other data-related activities.
Keywords: program evaluation; surveillance; neurological conditions; neurological disorders; multiple sclerosis; Parkinson’s disease.
Journey “Back Over the Line”: Critical Pedagogies of Curriculum Evaluation
Authors: Tintiangco-Cubales Allyson , Halagao Patricia Espiritu , Cordova JoanMay Timtiman 
Published: April 2023

Background: We re-trace our liberatory journey in developing a Critical Framework of Review to evaluate K-12 Filipina/x/o American curricula. Our framework is rooted in our positionality and epistemology as Filipina educational scholars engaged in confronting oppression that impacts our community. It responds to the need for evaluation methods grounded in culturally responsive and critical pedagogies.
Purpose: The purpose is to provide a critical and cultural method of evaluation to assess curriculum and pedagogy of, by, and about our communities.
Setting: The research takes place in the Filipinx/a/o American community in the United States. The authors are from three academic institutions in California, Hawai‘i and the Philippines.
Intervention: Our Critical Framework of Review attempts to counter the predominance of Eurocentric, male, objective, and uncritical models of curricula evaluation.
Research design: This research deconstructs how we developed and applied our framework, which was used to evaluate thirty-three Filipina/x/o American K-12 curricula in critical content, critical instruction, and critical impact, by asking 20 questions that reflected critical and cultural theories and pedagogies.
Data collection and analysis: We asked: Who and what informed our evaluation framework? How was it developed? How do we use it? How could our framework be further applied? We referenced diverse scholars and used critical race, feminist, indigenous, and deolonizing pedagogies as guidelines to establish our evaluation framework and standards.
Findings: The framework is an example of standards-based and responsive-based evaluation with a checklist of indicators to evaluate curricula for culture, race, positionality, and social justice. Although created for Filipina/x/o, the framework can be used to evaluate curriculum for other marginalized groups.
Keywords: critical pedagogy; critical evaluation; framework of review; curriculum; curriculum evaluation