bluuebunny/crossref_metadata_2025 huggingface dataset:
One Sherwood Street: realising the benefits of a modular approach
Authors: Goold Matt
Published: April 2023

One Sherwood Street is a major mixed-use development on one of London's busiest thoroughfares. The architectural concept for the building includes a feature roof comprising a series of geometrically varying triangles and quadrilaterals which intersect at differing angles and pitches. Originally designed as a traditionally built, hot-rolled steel roof with individual infill members, the complexity of the design and detailing and site constraints saw the contractor seek an alternative approach. This article describes the off-site, modular approach adopted which allowed the roof to be prefabricated as a series of steel panels that could be assembled in a controlled factory environment. The modular approach brought significant benefits to the project, including tighter tolerances, better design coordination, less waste, and a reduction in site works, temporary works, deliveries and crane time, leading to a faster construction programme.
Innovative composite floors for sustainable schools
Authors: Chapman Daniel , McRobie Allan , Winder Sam , Papastavrou Panayiotis , Smith Simon
Published: April 2023

This article describes a project to investigate the dynamic behaviour of a prefabricated timber-concrete composite floor slab developed for the Department for Education's GenZero sustainable schools programme. The GenZero initiative involves a new building system made of a kit of parts that could be manufactured off site and then assembled on site to create diferent confgurations of secondary school. The slab is constructed of a precast concrete deck on glulam ribs, with two interlayers. The project investigated the effect of both the interlayers and different shear connectors on the performance of the slab.
CUMHURİYET DÖNEMİNDE TÜRK LEHÇELERİ ÜZERİNDEKİ KAYNAK ÇALIŞMALARINA GENEL BAKIŞ
Authors: KİRİŞÇİOĞLU Mehmet Fatih
Published: April 2023

Özet: Dünyada akademik olarak Türk lehçeleri üzerine çalışmalar 18 yy.ın 2. yarısında başlamıştır. Ancak bu dönemler Osmanlı İmparatorluğu’nun sıkıntılı dönemleridir. 1730’da İran’da yenilen Osmanlı arkasından Kerkük, Gence, Tiflis, Kırım gibi bölgeleri kaybetmiş ve iyice kendi içine dönmüştür. Muhakkak ki bu dönemde diğer Türk boylarıyla ilgilenmesi mümkün olamazdı. Buna rağmen az da olsa Türk lehçeleriyle ilgili yayınlar basılıyordu. I. Dünya Savaşı’ndan yenik çıkan devlet arkasından Kurtuluş Savaşı da yapmak zorunda kalmıştı. Bütün bu sıkıntılı dönemde Orhun Abideleri bulunuyor ve çözülüyor, Kutadgu Bilig neşrediliyordu. 1923’de kurulan yeni Türkiye Cumhuriyeti kurumlarını oluşturduktan sonra soydaşlarının dil ve edebiyatlarıyla ilgilenmeye 1924 yılında başlamıştır. Bu makale Cumhuriyet’in 100. Yılında Türkiye’de Türk lehçeleri üzerindeki çalışmaları belirtmektedir.

Türkiye’de Türk lehçeleri üzerindeki çalışmaların sadece künyelerini vermek bile bir kitap boyutunda olacağından makalemizde ilk önemli çalışmalar ve önemli gördüğümüz kaynak çalışmalar belirtilmeye çalışılmıştır
A Student-Led Methodology for Evaluating Curricular Redundancy
Authors: Royal Kenneth D. , Gilliland Kurt O. , Dent Georgette A.
Published: April 2023

Background: Curricular redundancy can be a significant problem for any educational curriculum. Redundancy can be both desirable and undesirable, but differentiating the two can be quite challenging. Further, pinpointing undesirable redundancy and quantifying it so as to produce an estimate of inefficiency is even more difficult. Purpose: The purpose of this research is to describe a student-led strategy for evaluating redundancy in a highly integrated medical school curriculum. It is our hope that the methodology presented here will serve as a useful evaluation model for persons attempting similar work in various educational arenas. Setting: A highly-integrated medical school at a large public university. Intervention: This research did not require an intervention. Research Design: We identified two advanced medical students and asked them to identify redundant material across the first two years of the medical school curriculum. The students had to operationalize ‘redundancy’, develop an evaluation plan/framework, and evaluate the extent to which undesirable redundancy was prevalent in the current curriculum. Data Collection and Analysis: Students reviewed course syllabi, notes, and materials and documented the amount of redundant material they found in the curriculum. Findings: A total of approximately 167 hours, or 8.35 weeks, could be eliminated from the curriculum; the vast majority of the redundancy occurred as a result of small group activities.
Using Graphical Perception Principles to Improve the ST Tools’ Data Visualization: Revisiting the Systems Dynamics Model
Authors: Vaca Sara , Vidueira Pablo
Published: April 2023

Background: Systems Thinking (ST) is the new paradigm in Evaluation. It represents a significant mind-set shift for the evaluation field and it is a powerful tool to tackle complex environments. Heir to the systems concepts of the engineering field, and especially regarding the hard systems tools, ST in evaluation uses the same visual tools that were created many years ago. All these tools already incorporate data visualization features:  they depict ideas, relationships and concepts relying in shapes and figures more than a textual explanation. Revisiting these tools and applying the latest data visualization principles, they could be optimised in order to provide with more information within the same concept. Purpose: To provide ST practitioners with more informative tools in order to facilitate:-  ST experts and users can optimise the application of the tools to real life models beyond the initial set up of their visual representations.-  Audiences of evaluations using ST as part of the toolkit can find the outputs more apprehensible and easy to understand .  Setting: Not applicable. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Not applicable. Findings: Improving ST representations of reality and systems can help both enhance ST applications and make it more accessible and comprehensible for evaluation practitioners' and users. Six ways for improving the understanding of the current stock and flow diagrams were identified. The tools proposed consist of: customizing the colours and shape of the variables and their relationships to make them more informative; highlighting the existing subsystems within the model; and providing the specific sequence for reading the main causal chains.
Adaptation and Use of a Five-Task Model for Evaluability Assessment
Authors: Watts Brad R. , Washington Heather M.
Published: April 2023

Background: Over four decades have passed since the concept of evaluability was introduced; however, the availability and accessibility of methodology, frameworks, checklists and other guidance on evaluability assessments remains limited (Smith, 2005). Evaluators who wish to conduct an evaluability assessment must adopt one of few existing models or operate without guidance. This case provides an example of one model and how it was utilized to conduct an evaluability assessment of an ongoing program intervention. Purpose: This article provides a real-world example of how an evaluability assessment was conducted using a five-task model from the criminal justice field. Setting: A criminal justice program operating with at-risk adolescent youth and police officers who patrol their neighborhoods. Intervention: NA Research Design:  NA Data Collection and Analysis: The original evaluability assessment collected data through interviews with program personnel, observations by the evaluability researchers, and analysis of survey data provided by the program (i.e., program participants’ responses to survey questions about the program). Findings: The evaluability assessment conducted in this case study example identified multiple changes necessary for the program to be evaluable. Technical assistance was provided in order to support the efforts of the program to prepare for evaluation, though, the evaluability assessment identified risks and costs for the program to consider prior to proceeding. The findings were developed through a five-task process, which evaluators may be able to adapt for use with other types of programs undergoing evaluability assessment.
Re-Structuring Evaluation Findings into Useful Knowledge
Authors: Houston Danielle , Wright Bernadette , Wallis Steven E.
Published: April 2023

Background: A long research stream has shown that when knowledge is more structured it is more likely to be effective in practical application. Building on that research, the authors applied Integrative Proposition Analysis to visualize, integrate, and assess the quality and usefulness of knowledge gained from the NMAC (formerly National Minority AIDS Council) Strong Communities evaluation.
Purpose: Demonstrate an innovative method to rigorously integrate and strengthen knowledge gained from evaluation and to encourage discussion of future directions for developing stronger theories for more effective evaluation and more effective action.
Setting: Birmingham, Alabama
Intervention: A project to identify local strategies for community-based organizations and community health centers that serve African American and Latinx gay and bisexual men and transgender women to collaboratively meet HIV-related community needs.
Research Design:  The researchers applied Integrative Propositional Analysis to integrate and map concepts and causal connections emerging from the evaluation findings. The authors then analyzed the resulting map to identify top-mentioned concepts, better understood concepts, reinforcing loops, and knowledge gaps.
Data Collection and Analysis: Integrative Propositional Analysis applied to a literature review and stakeholder interview transcripts collected for the evaluation.
Findings: Integrating literature and interview results helped to identify several actions where providers of HIV-related services could increase their impact on combating the HIV epidemic among the communities they serve. The authors also identified a reinforcing loop; this shows opportunity to improve two desired outcomes by increasing one. In addition, the authors identified blank spots on the map; these show where additional research could strengthen the quality and usefulness of the mapped knowledge.
Examining Factors Impacting Online Survey Response Rates in Educational Research: Perceptions of Graduate Students
Authors: Saleh Amany , Bista Krishna
Published: April 2023

Background: In educational research, online survey has become one of the most popular methods of data collection. Academic researchers, including faculty and students, expect and require a good response rate to their research projects for reliable results.
Purpose: In this paper, the authors examine a wide range of factors related to survey response rates in academic research. Examples include email checking habits, survey design, and attitudes toward research.
Setting: An online survey environment
Intervention: Not applicable.
Research Design:  A cross-sectional quantitative research method was used to analyze the factors that influence participants’ email survey response rate. Data were collected at a single point in time. The authors did not directly measure changes that come over time in this study.
Data Collection and Analysis: After receiving the Institutional Research Board’s approval, the researchers distributed the survey via the American Educational Research Association (AERA) Graduate Student Discussion List subscribers. A sample of 454 responses was used in the final analysis-- with a 78.9 % response rate. The authors used descriptive statistics (percentage, average mean) and inferential statistics (chi-square and correlations) to report the data analysis and findings.
Findings: Results indicated that research survey response rate was highly influenced by interests of participants, survey structure, communication methods, and assurance of privacy and confidentiality. The findings also suggested that male participants were more likely to respond to surveys if they received a reminder, and older participants were more likely to respond if they were promised a reward.
Refining Theories of Change
Authors: Dhillon Lovely , Vaca Sara
Published: April 2023

Background: Despite the disparities in how they are defined and what elements are included, most Theories of Change remain consistent in one way – their visual format. Typically, Theories of Change are presented as a one-page visual in a flowchart style with lines and boxes of uniform size. In addition, Theories of Change are often created as stand-alone tools that are rarely linked effectively to other organizational tools.
Purpose: The authors: (1) propose the essential elements that contribute to robust Theories of Change and clarify the characteristics that distinguish Theories of Change from other organizational tools and formats; (2) suggest additional elements for inclusion in the Theory of Change; (3) present graphic alternatives that allow for an evolution in representing their complexity and depth; and (4) provide ways to link Theories of Change to other organizational tools to increase organizational alignment, efficiency, and, most importantly, impact.
Research Design: NA
Data Collection and Analysis: NA
Setting: NA
Intervention: NA
Findings: NA
Customizing the Standard to the Purpose of the Assessment
Authors: Monteiro Sandra , Sibbald Debra
Published: April 2023

No abstract (commentary).
Evaluation Essentials: From A to Z (2nd ed.)
Authors: Verburg Mark Edward
Published: April 2023

A review of the book Evaluation Essentials: From A to Z (2nd ed.) by Marvin C. Alkin and Anne T. Vo, published in 2018 by Guilford Press.
Using Program Theory to Evaluate a Graduate College Student Development Program
Authors: Gus Shannon , Keener Ashley , Bullard Chelsea E. , Gordon Sarah
Published: April 2023

Background: The “3 Minute Presentation” is a graduate student competition based off the more popular “3 Minute Thesis” competition. The program aims to help graduate students learn to inform others of their research in a quick and accessible manner. Programs to engage graduate students more deeply in their education require evaluation to determine if they are useful and effective at meeting their intended goals. Evaluation literature in graduate educational programs is currently limited, but increasingly needed for both the field and the students served.

 

Purpose: Development and testing of a program-theory evaluation to understand participation, recruitment, preparation, training, skills, and confidence of graduate students engaging in a “3 Minute Presentation” competition at a state university.

 

Setting: Institution of Higher Education

 

 

Intervention: 3 Minute Presentation competition

 

Research Design:  Mixed-method program-theory evaluation

 

Data Collection and Analysis: Direct observations and closed-ended survey analyzed through qualitative coding, descriptive statistics, group comparisons, and correlation analysis.  

 

Findings: Overall, the program evaluation found, with a possible lack of diversity in participants, that the program components of recruitment, preparation, and skill development work as expected.  Additionally, engagement in preparation was associated with competition scores and the perceived helpfulness of preparation was related to students’ confidence in their presentation skills. This evaluation was deemed useful for program improvement and capacity building in the program’s continuation at the university.
Method for Using Rubric Ratings on Fishbone Diagrams to Compare Case Studies
Authors: Davey Melinda E. , Morell Jonathan A.
Published: April 2023

Background: In multi-case study program evaluations, the large amount of qualitative data that are generated from interviews can be difficult to utilize. This is particularly so when inference must be made as to why some cases succeed and some fail.

Purpose: This paper shows a method for comparing multiple evaluation sites by using a rubric to define ratings for relevant factors, and an Ishikawa fishbone diagram as a model to show relationships among those factors. We show how this technique identified reasons for differences in outcomes among the sites.

Setting: The evaluation setting was a large-scale safety innovation in the U.S. railroad industry. Four cases were considered—two passenger railroads and two freight railroads.

Intervention: The Confidential Close Call Reporting System (C3RS) program allowed railroad workers to confidentially submit “close calls” which were reviewed by a team made up of labor, management, and the Federal Railroad administration to determine ways to improve safety.

Research design: Multiple comparative case study, Ishikawa root and contributing cause modeling, evaluative rubric scoring, and data visualization techniques.

Data collection & analysis: Interview data were collected from four pilot railroad sites, each of which participated in a five-year test of C3RS. Testing periods overlapped, with the entire evaluation lasting about 12 years.

Findings: The method of using Ishikawa fishbone diagrams with ratings from an evaluative rubric was an effective method to summarize, analyze, and present large quantities of qualitative data. The approach succeeded in explaining degrees of success and failure across the sites. The sponsor and industry stakeholders were able to understand the analysis and the findings, and to develop deep insight into how to promote successful implementation.

Keywords: Multiple comparative case studies; qualitative methodology; qualitative coding; data visualization; fishbone diagrams; Ishikawa diagrams; evaluative rubrics; close calls; near miss; data visualization.
Mixed Methods Research in Designing an Instrument for Consumer-Oriented Evaluation
Authors: Gallant Dorinda J. , Luthy Nicole
Published: April 2023

Background: The educational product market has been gradually shifting from primarily print to primarily digital content. Educators must make quick decisions when selecting materials that will assist students in their learning.

Purpose: Purposes of this study were to describe the application of a two-stage sequential mixed-method, mixed-model design in designing an instrument for consumer-oriented evaluation and to describe implications of using mixed methods research in developing a rubric to evaluate prekindergarten through Grade 12 digital content.

Setting: The Ohio State University, Columbus, OH.

 

Intervention: N/A.

Research design: A two-stage sequential mixed-method, mixed-model design.

Data collection & analysis: In Stage 1, a modified electronic Delphi survey technique was implemented with US geographically dispersed subject matter experts. In Stage 2, cross-sectional focus group interviews were conducted with local teachers, administrators, and textbook publishers.

Findings: Inclusion of multiple perspectives and viewpoints from teachers, administrators, textbook publishers, and experts on importance, clarity, and appropriateness of criteria to evaluate digital content resulted in a final version of the rubric that can be used by teachers and administrators to evaluate digital content that supports students’ learning in prekindergarten through Grade 12.
Refining and Measuring the Construct of Evaluative Thinking: An Exploratory Factor Analysis of the Evaluative Thinking Inventory
Authors: McIntosh Jason S. , Buckley Jane , Archibald Thomas
Published: April 2023

Background: Evaluative thinking has emerged as a key construct in evaluation, especially for evaluation practitioners and researchers interested in evaluation capacity building (ECB). Yet, despite increasing calls for more research on evaluation and, more specifically, for more research on ECB, until recently little empirical inquiry on the dimensions of evaluative thinking has been conducted.

Purpose: To address that lack, the purpose of the study presented in this paper is to refine the construct of evaluative thinking by exploring its underlying dimensions and to ascertain the internal consistency of an instrument developed to measure evaluative thinking, the Evaluative Thinking Inventory (ETI).

Setting: The ETI was developed as part of an ECB initiative focused on non-formal science, engineering, technology, and math (STEM) education in the United States, and was tested as part of a study focused on evaluating gifted education programs, also in the United States.

Intervention: Not applicable.

Research design: Survey research and exploratory factor analysis (EFA).

Data collection & analysis: The ETI was administered to participants in a study measuring the effectiveness of a tool used to conduct internal evaluations of gifted education programs. SPSS was used to conduct an EFA on 96 completed ETIs. Cronbach’s alpha was used to estimate the internal consistency of the instrument.

Findings: The analysis of the ETI revealed a two-factor model of evaluative thinking (i.e., believe in and practice evaluation and pose thoughtful questions and seek alternatives). This study also provided internal consistency evidence for the ETI showing alpha reliabilities for the two factors ranging from 0.80 to 0.82. The ETI has potentially wide applicability in research and practice in ECB and in the field of evaluation more generally.

Keywords: evaluative thinking; evaluation capacity building; research on evaluation; exploratory factor analysis.
Using PhotoVoice as an Evaluation Method
Authors: Hunter Ouen , Leeburg Emma , Harnar Michael
Published: April 2023

Background: Engaging with youth through PhotoVoice is beneficial as a program evaluation method and functions as a method of inquiry to understand youths’ perceptions of a college preparation program. The students used PhotoVoice to respond to prompts about how they learn and their opinions of the college preparation program.

Purpose: This reflection of practice article provides an example of PhotoVoice as an evaluation method.

Setting: This evaluation was conducted during the summer college preparation programming.

Intervention: The combination of student photography, photo gallery walk, and group discussion as an evaluation method.

Research Design: A qualitative reflective design.

Data Collection and Analysis: The student photographs, narratives, and observational notes were analyzed thematically.

Findings: We found that the youths enjoyed the unique experience of PhotoVoice. The combination of student photography, narratives, gallery walk, and group discussion, were useful as an inquiry tool for this youth program.
Process Evaluation to Document Crucial Moments in Development of the National Neurological Conditions Surveillance System at the U.S. Centers for Disease Control and Prevention
Authors: MacDonald Goldie , Mercer Shawna L , Fisher Angela H , Grigorescu Violanda , Mac Kenzie William R , Anderson Mindelyn
Published: April 2023

Background: Neurological conditions or disorders strike roughly 50 million Americans annually but accurate and comprehensive national estimates for many of these conditions are not available. In 2019, Congress provided $5 million to Centers for Disease Control and Prevention (CDC) to establish the National Neurological Conditions Surveillance System (NNCSS). CDC focused initial activities on multiple sclerosis and Parkinson’s disease.

Purpose: We conducted a process evaluation to document and understand multifaceted work to implement a new surveillance activity for two neurological conditions.

Setting: We conducted this evaluation with government personnel internal to the Center for Surveillance, Epidemiology, and Laboratory Services at the Centers for Disease Control and Prevention in Atlanta, GA.

Intervention: A new public health surveillance activity for two neurological conditions, multiple sclerosis and Parkinson’s disease, that uses existing data resources and systems.

Research design: The evaluation included interviews with CDC personnel and review of administrative and programmatic information. Data were analyzed and interpreted to identify crucial moments in the first year of funded work on NNCSS. The study revealed that this surveillance activity required diverse contributions and collaboration within the federal government and with non-governmental organizations. The findings can be used to guide work to enhance surveillance for many neurological conditions.

Findings: The study revealed that this surveillance activity required diverse contributions and collaboration within the federal government and with non-governmental organizations. While collaboration is a cornerstone of public health practice, it is not always well-documented in planning or implementation of surveillance or other data-related activities.

Keywords: program evaluation; surveillance; neurological conditions; neurological disorders; multiple sclerosis; Parkinson’s disease.
Journey “Back Over the Line”: Critical Pedagogies of Curriculum Evaluation
Authors: Tintiangco-Cubales Allyson , Halagao Patricia Espiritu , Cordova JoanMay Timtiman
Published: April 2023

Background: We re-trace our liberatory journey in developing a Critical Framework of Review to evaluate K-12 Filipina/x/o  American curricula. Our framework is rooted in our positionality and epistemology as Filipina educational scholars engaged in confronting oppression that impacts our community. It responds to the need for evaluation methods grounded in culturally responsive and critical pedagogies.

Purpose: The purpose is to provide a critical and cultural method of evaluation to assess curriculum and pedagogy of, by, and about our communities.

Setting: The research takes place in the Filipinx/a/o American community in the United States. The authors are from three academic institutions in California, Hawai‘i and the Philippines.

Intervention: Our Critical Framework of Review attempts to counter the predominance of Eurocentric, male, objective, and uncritical models of curricula evaluation.

Research design: This research deconstructs how we developed and applied our framework, which was used to evaluate thirty-three Filipina/x/o American K-12 curricula in critical content, critical instruction, and critical impact, by asking 20 questions that reflected critical and cultural theories and pedagogies.

Data collection and analysis: We asked: Who and what informed our evaluation framework? How was it developed? How do we use it?  How could our framework be further applied? We referenced diverse scholars and used critical race, feminist, indigenous, and deolonizing pedagogies as guidelines to establish our evaluation framework and standards.

Findings: The framework is an example of standards-based and responsive-based evaluation with a checklist of indicators to evaluate curricula for culture, race, positionality, and social justice. Although created for Filipina/x/o, the framework can be used to evaluate curriculum for other marginalized groups.

Keywords: critical pedagogy; critical evaluation; framework of review; curriculum; curriculum evaluation
Elements to Enhance the Successful Start and Completion of Program and Policy Evaluations: The Injury & Violence Prevention (IVP) Program & Policy Evaluation Institute
Authors: Porter Jamila M. , Brennan Laura K. , Fine Mighty , Robinson Ina I.
Published: April 2023

Background: Public health practitioners, including injury and violence prevention (IVP) professionals, are responsible for implementing evaluations, but often lack formal evaluation training. Impacts of many practitioner-focused evaluation trainings—particularly their ability to help participants successfully start and complete evaluations—are unknown.

Objectives: We assessed the impact of the Injury and Violence Prevention (IVP) Program & Policy Evaluation Institute (“Evaluation Institute”), a team-based, multidisciplinary, and practitioner-focused evaluation training designed to teach state IVP practitioners and their cross-sector partners how to evaluate program and policy interventions.

Design: Semi-structured interviews were conducted with members of 13 evaluation teams across eight states at least one year after training participation (24 participants in total). Document reviews were conducted to triangulate, supplement, and contextualize reported improvements to policies, programs, and practices.

Intervention: Teams of practitioners applied for and participated in the Evaluation Institute, a five-month evaluation training initiative that included a set of online training modules, an in-person workshop, and technical support from evaluation consultants.

Main Outcome Measure(s): The successful start and/or completion of a program or policy evaluation focused on an IVP intervention.

Results: Of the 13 teams studied, a total of 12 teams (92%) reported starting or completing an evaluation. Four teams (31%) reported fully completing their evaluations; eight teams (61%) reported partially completing their evaluations. Teams identified common facilitators and barriers that impacted their ability to start and complete their evaluations. Nearly half of the 13 teams (46%) – whether or not they completed their evaluation – reported at least one common improvement made to a program or policy as a result of engaging in an evaluative process.

Conclusion: Practitioner-focused evaluation trainings are essential to build critical evaluation skills among public health professionals and their multidisciplinary partners. The process of evaluating an intervention—even if the evaluation is not completed—has substantial value and can drive improvements to public health interventions. The Evaluation Institute can serve as a model for training public health practitioners and their partners to successfully plan, start, complete, and utilize evaluations to improve programs and policies.

Keywords: Evaluation; injury; multidisciplinary partnerships; practitioner-focused evaluation training; professional development; program and policy evaluation; public health; technical assistance; violence
Use of Geographic Information Systems by American Evaluation Association Members in their Professional Practice
Authors: Kates Aaron W. , Coryn Chris L.
Published: April 2023

Background: As geographic information systems (GIS) technology continues to develop and expand in its capacity and applications, it is becoming increasingly useful to many disciplines. Even so, little has been written about the place of GIS technology in evaluation practice, and there is a paucity of information as to the extent to and applications for which evaluation practitioners use such technology.

Purpose: In this investigation, the prevalence and common applications of GIS technology in professional evaluation practice are examined. Particularly, the study was designed to estimate what proportion of American Evaluation Association (AEA) members who self-identify as evaluation practitioners use GIS in their practice, if at all, and, if so, to what extent. For those who use GIS in their evaluation practice, the specific GIS software packages and applications used also are explored.

Setting: Not applicable.

Intervention: Not applicable.

Research Design: A simple random sample of American Evaluation Association (AEA) members were surveyed, with an emphasis on evaluation practitioners.

Findings: Less than less than half (41.04% ±6.09%) of AEA members who consider themselves evaluation practitioners have ever used GIS in their evaluation practice and less than one-third (31.47% ±5.75%) have received some form of training in GIS methods. Data visualization is, by far, the most frequent application of GIS in evaluation practice.

Keywords: American Evaluation Association; geographic information systems; technology in evaluation; evaluation practice; research on evaluation
Cultural Humility: A Collaborative Approach to Recruiting Patients with Deliberate Self-Harm into a Multi-Hospital Randomized Controlled Trial
Authors: Brownhill Suzanne H. , Stevens Garry J. , Hammond Trent E. , Baldacchino Richard , Maposa Rose , Makoni Bonyface , Sheb'a Aviva , Andepalli Jagadeesh , Kotak Anita , D'Souza Olav , Atashnama Azadeh , Thayil Anish , Jones Alison
Published: April 2023

Objectives: The ‘SMS SOS’ Deliberate Self-Harm (DSH) Aftercare Study was conducted in Western Sydney, Australia (October 2017 to December 2020) across three large public hospitals. During this randomized controlled trial (RCT), it was observed that knowledge exchange between key stakeholders and their ‘cultural’ perspectives (for example, Mental Health Clinicians, Lived Experience Mental Health Consultants—Patient Representatives, Administrative Officers, and Researchers) was essential to effective recruitment of patients experiencing DSH. Knowledge exchange within and between cultural groups was maximised and assessed using a communication matrix. This process, transferable to other trials engaging multiple ‘cultures’, aimed to promote the early identification of wider-team strengths as well as active management of emergent issues that would otherwise impede patient recruitment, and to maximise funding and human resources.

Methods: A descriptive study was conducted with a convenience sample of team members who represented different cultures in the study. Qualitative data were elicited from a ‘know and tell’ matrix. Through an iterative process, themes were generated that encapsulated what team members needed to know from and tell to their colleagues concerning the study.

Results: Factors that impacted participation in the study included clinician workload, the level of motivation/ commitment/confidence of clinicians to recruit patients, clinician-patient engagement, perception and expectations of study involvement, inter-cultural communication, and clinician training and support. The findings of this multidisciplinary consultation informed a composite model of knowledge exchange and the development of educational briefing/ orientation modules that make explicit team members’ roles and responsibilities to foster group member participation and enhance patient recruitment.

Conclusions: It is incumbent upon multidisciplinary team members of large-scale studies to adopt a similar ‘knowledge exchange’ strategy early in the planning and design stage. Adoption of such a strategy has the potential to mitigate risk of delay in project timelines, improve project outcomes, and ensure the efficient use of research funding, particularly in newly established research teams within clinical settings and with members newer to formal research collaborations.

Keywords: cultural humility; deliberate self-harm; engagement; participant recruitment; participatory research; randomized controlled trial
Measuring Mental Health Literacy: Development of the Mental Health Awareness and Advocacy Assessment Tool
Authors: Aller Ty B. , Fauth Elizabeth B. , Novak Joshua R. , Schwartz Sarah
Published: April 2023

Background: Mental health literacy programs are a common community-based approach used to address the prevention of mental health issues on college campuses. Current assessment strategies used to evaluate the effectiveness of these programs often lack strong theoretical rational and psychometric rigor.

Purpose: The purpose of this study was twofold. First, based upon extant literature, theory, and standard clinical practice, we propose a process-based model of mental health literacy that includes three macro factors—identifying mental health issues, locating empirically based resources, and responding to mental health issues—and three micro processes of how they unfold—acquiring knowledge, building self-efficacy, and applying skills (behavior). The second aim was to test the psychometric properties of a new tool created to evaluate this process-based model—the Mental Health Awareness and Advocacy Assessment Tool (MHAA-AT).

Setting: Not applicable.

Intervention: Not applicable.

Research Design: A national sample of 296 college attending participants were recruited from Amazon’s Mechanical Turk. Participants responded to a demographic questionnaire and the newly developed MHAA-AT. Psychometric properties were examined through item response theory, exploratory factor analyses, and bivariate correlations.

Findings: Results suggest the MHAA-AT is a sound measure and demonstrates appropriate item, person, and trait characteristics on declarative knowledge items, and single factor structures on self-efficacy and behavior items with moderate to high reliability and validity. While additional testing is need among other samples, results suggest that the MHAA-AT is a quality assessment tool.

Keywords: College students; mental health literacy; item response theory; measurement
Emergent Developmental Evaluation Developments
Authors: Patton Michael Q.
Published: April 2023

Background: Developmental Evaluation (DE) continues to develop. After reviewing the niche and history of DE, this article identifies five new and emergent purposes for DE to add to the original five – and forecasts future developments for diverse, innovative, and adaptive DE applications. An extended example of DE in supporting pivoting in the pandemic illustrates DE as adaptation to crisis as distinct from supporting social innovation.

Purpose: To update DE.

Setting: Global.

Data Collection and Analysis: Not applicable.

Findings: The applications and uses of DE have expanded beyond the original framing.

Keywords: adaptation; developmental evaluation; pivot; social innovation
Choosing an Evaluation Theory: A Supplement to Evaluation Roots (3rd Edition)
Authors: Alkin Marvin C. , Christie Christina A. , Stephen Naomi A.
Published: April 2023

Background: Unlike scientific theories, evaluation theories are prescriptive: a set of actions and approaches that should be followed when conducting an evaluation. While evaluation theorists have offered a variety of writings describing their theories and approaches, few have offered a specific outline of what the theory looks like in practice. Thus, Alkin and Christie formulated a book to aid evaluators in how to apply theories in evaluations (Alkin & Christie, forthcoming). This book culminates in a series of prototypes that outline each theory’s goals, appropriate contexts, prescriptions, and observable actions in application.

Purpose: In order to aid evaluators in applying theories, this article seeks to provide a basis for comparison that can be used to help evaluators select which theory would be most appropriate in their practice.

Setting: This comparison can be applied in any setting where evaluations fit the context prescribed by each of the theories.

Intervention: Not applicable.

Research Design: Not applicablre.

Data Collection and Analysis: Not applicable.

Findings: In order for theories to influence practice effectively, theories must be displayed in a way that allows for easy comparison. This comparison of three theory prototypes demonstrates that prototypes can be an effective way for selecting a prescriptive theory when conducting an evaluation.

Keywords: prescriptive theories; practice; empowerment evaluation; learning centered model; developmental evaluation
Beyond GDP: Tracking and Evaluating National Contributions to Social and Environmental Sustainability
Authors: Picciotto Robert
Published: April 2023

Background: The Gross Domestic Product (GDP) emerged as a convenient measure of national economic activity during the Great Depression. It was subsequently adopted by international development economists to track developing countries’ progress so that, despite its severe deficiencies, it became ‘locked in’ by habit, convenience, and policy makers’ preferences.

Purpose: This article conceives of GDP as a social intervention fit for evaluation. It shows that the GDP has had a pervasive and pernicious influence on policy making. Since past strategies aimed at dethroning the GDP have failed, it proposes new, evaluator-driven approaches designed to undermine the GDP’s dominance in the global market economy.

Setting: The Stiglitz report commissioned in the wake of the 2008 financial crisis launched a ‘Beyond GDP’ movement. Since then, public alarm about the GDP growth addiction has escalated: the drawbacks of GDP as a free-market policy tool have become self-evident as the rich get richer, the ranks of the poor swell and the future of the planet hangs in the balance.

Research Design: Not applicable.

Data Collection and Analysis: For the twenty largest economies in the world, the article estimates climate change discounts to the GDP based on official CO2 emissions statistics and a social cost of carbon estimate derived from a 2015 survey of eminent climatologists. It also draws on composite indexes generated by four reputable social research organizations to rank countries for their contributions to the 5 Ps of the Sustainable Development Goals (SDGs): people, planet, prosperity, peace, and partnership.

Findings: Pending the results of on-going efforts to upgrade worldwide statistics focused on the 169 SDG targets, the proposed GDP discounts help track progress towards the SDGs. But monitoring is not enough. In a policy world dominated by vested interests, the new ‘Beyond GDP’ indicators should be combined with principled, evaluator-directed evaluations.

Keywords: Beyond GDP; climate change; evaluator-directed evaluation; Gross Domestic Product; indicators; Sustainable Development Goals
Empowerment Evaluation of Programs Involving Youth
Authors: Heath Sarah , Moreau Katherine
Published: April 2023

Background: Participatory and collaborative evaluation approaches, including Empowerment Evaluation (EE), are useful for evaluating programs involving youth. Empowerment evaluation involves stakeholders in the evaluation process through a set of structured steps. It is primarily concerned with empowering, illuminating, and building program beneficiaries’ self-determination. Given the emphasis that EE places on inclusivity of stakeholders, it appears to be a good fit for evaluating programs that involve youth.

Purpose: To explore the extent to which evaluators use EE to evaluate programs involving youth as well as what factor(s) facilitate and hinder their use of EE in these programs.

Setting: The study involved evaluators associated with the Collaborative, Participatory and Empowerment Evaluation and Youth-Focused Evaluation Targeted Interest Groups (TIGs) of the American Evaluation Association (AEA) who are involved in evaluating programs targeted at youth.

Intervention: Not applicable.

Research Design: We used a two-phase sequential mixed-methods research design. In Phase 1, we surveyed evaluators. In Phase 2, we interviewed a sample of evaluators from Phase 1. 

Findings: In Phase 1, 41 (53.9%) respondents indicated not using EE to evaluate programs involving youth, 30 (39.5%) had used EE and 5 (6.6%) were unsure. Of those who used EE, they used it to teach youth program stakeholders about evaluation (n=8, 24.2%), produce more authentic results by engaging youth as experts of their lived experience (n=7, 21.2%) or produce more useful results for stakeholders to use (n=6, 18.2%), as well as other less popular reasons. In Phase 2, 12 interviewees raised five factors that facilitate or hinder the use of EE to evaluate programs involving youth including, evaluator perceptions, type of evaluation experience, evaluator knowledge and professional training, guidelines from organizations and funders, and stakeholders and time. Factors that some interviewees viewed as facilitators others viewed as hinderances.

 

Keywords: empowerment evaluation, program evaluation, youth-focused evaluation
Intangible Outcomes
Authors: Wilson Kurt
Published: April 2023

Human life – and therefore the scope of human goals – includes dimensions that are both visible and countable (e.g., money, weight, attendance or tested proficiency) as well the invisible and intangible (e.g., hope, trust, faith, love, joy, peace). Furthermore, the visible and tangible aspects of life are intrinsically connected to and dependent on the invisible and intangible aspects - much as the visible branches and fruit of a tree are connected to and dependent on an underlying and hidden root structure. While the importance of intangibles can be understood intuitively, it can also be illustrated: 73% of all chartable giving in the U.S. goes to organizations that are explicitly religious, and 118,280 nonprofit organizations are so strongly identified with the intangibles of hope, trust, faith, love, joy, peace that they included one of these words in their name. While the intangible realities of human life are explicitly relevant to a large proportion of organizations we seek to serve, it is essentially ignored by current evaluation practice: only 10 articles within the American Journal of Evaluation, New Directions in Evaluation and Journal of Multidisciplinary Evaluation included even minimal reference to the most common intangibles. New evaluation theory and methodology to address this gap will be needed, and cross-disciplinary exploration with psychology, philosophy and sociology should guide this development. In the meantime, useful questions about intangibles can be drawn from the AEA guiding principles and addressing these can provide a useful starting point for evaluators seeking to consider intangibles within their evaluations.
Editor's Note
Authors: Harnar Michael A.
Published: April 2023

I would like to start off this editorial note by offering my sincere thanks and appreciation to all the authors and reviewers that have entrusted the Journal of MultiDisciplinary Evaluation (JMDE) with your scholarship, especially this year as we’ve worked to transition our journal. Your patience and encouragement have meant the world to those of us on the editorial team producing this important resource.
The State of Evaluation in Canada
Authors: Coryn Chris L. S.
Published: April 2023

The article discusses the state of evaluation in Canada, highlighting the Canadian Evaluation Society and its publication, the Canadian Journal of Program Evaluation.
New Directions for Evaluation
Authors: Risley John S.
Published: April 2023

The two most recent issues of New Directions for Evaluation each cover international perspectives in the field. The Fall 2004 issue (Rugg, Peersman, and Carael) addressed “Global Advances in HIV/AIDS Monitoring and Evaluation” while the Winter 2004 issue (Russon and Russon) concerned “International Perspectives on Evaluation Standards.”
Evaluation in Canada
Authors: Coryn Chris L. S.
Published: April 2023

The Canadian provinces continue to be a source of evaluation-related activities and events such as Evaluation 2005, Beaulac, Goodine, and Aubry’s work on a reportcard of homelessness in Ottawa, the 2005 International Program for Development Evaluation jointly sponsored by the World Bank Group and Carleton University, and the Canadian Evaluation Society Student Case Competition and Paper Contest, to name but a few. For those interested in detailed information on these and other Canadian evaluation news and events please visit the Canadian Evaluation Society Website.
Evaluation in Germany: An Overview
Authors: Struhkamp Gerlinde
Published: April 2023

Even though often considered being a “late-starter” concerning evaluation, the beginnings for systematic inquiry into impacts of governmental programs in Germany parallel the developments in the U.S. where evaluation became particularly prominent during the 1960s and from then on.
Review of New Directions for Evaluation, Volume 106: Theorists' Models in Action
Authors: Coryn Chris L. S.
Published: April 2023

The summer 2005 issue of New Directions for Evaluation, “Theorists’ Models in Action”, edited by Marvin C. Alkin and Christina A. Christie is, in part, inspired by the “Radnor Middle School” case study conducted in the early 1980s (Brandt, 1981). In the Radnor case study notable theorists including Scriven, Stake, Eisner, and Popham were asked to explain how they would evaluate Radnor’s humanities curriculum.
In this issue Alkin and Christie asked four contemporary theorists—Jennifer C. Greene, Gary T. Henry, Stewart I. Donaldson, and Jean A. King—to describe how they would evaluate the case of the “Bunche-Da Vinci Learning Partnership Academy.” The Bunche-Da Vinci Learning Partnership Academy is essentially a “unique partnership between the [school] district and a nonprofit educational company specializing in innovative school interventions for low-performing students” (Eisenberg, Winters, & Alkin, 2005, p. 5).