bluuebunny/crossref_metadata_2025_split huggingface dataset:
Communicating About Evaluation: A Conceptual Model and Case Example
Authors: Montrosse-Moorhead Bianca , Bellara Aarti P. , Gambino Anthony J. 
Published: April 2023

Background: Despite consensus within the evaluation community about what is distinctive about evaluation, confusion among stakeholders and other professions abounds. The evaluation literature describes how those in the social sciences continue to view evaluation as applied social science and part of what they already know how to do, with the implication that no additional training beyond the traditional social sciences is needed. Given the lack of broader understanding of the specialized role of evaluation, the field struggles with how best to communicate about evaluation to stakeholders and other professions.
Purpose: This paper addresses the need to clearly communicate what is distinctive about evaluation to stakeholders and other professions by offering a conceptual tool that can be used in dialogue with others. Specifically, we adapt a personnel evaluation framework to map out what is distinctive about what evaluators know and can do. We then compare this map with the knowledge and skill needed in a related profession (i.e., assessment) in order to reveal how the professions differ.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Not applicable.
Findings: We argue that using a conceptual tool such as the one presented in this paper with comparative case examples would clarify for outsiders the distinct work of evaluators. Additionally, we explain how this conceptual tool is flexible and could be extended by evaluation practitioners in a myriad of ways.
Raising Aid Efficiency with International Development Aid Monitoring and Evaluation Systems
Authors: Boehmer Hans Martin , Zaytsev Yury K. 
Published: April 2023

Background: Since the adoption of the Millennium Development Goals (MDGs), or more specifically perhaps the Monterrey Consensus, there has been a distinct shift in the conversation around assessing the development effectiveness of international aid programs. Initially, the focus had been on establishing monitoring and evaluation systems that served the needs to bilateral and multilateral donors to demonstrate the effectiveness of their assistance vis-à-vis their own constituencies. Today, there is an increasing recognition that recipient countries should be equally concerned with the effectiveness of donor resources as they are with the use of national resources devoted to development programs.

Purpose: This article reveals the current efforts of the sovereign new and traditional donors in establishing and operationalizing the international development assistance (IDA) monitoring and evaluation (M&E) systems to raise the efficiency of IDA programs.

Setting: N/A

Intervention: N/A

Research Design: A review of previous studies and international practicies of IDA M&E.

Data Collection and Analysis: Employing game theory the authors identify the choice of particular IDA M&E system by traditional and new donors and analyze key elements and factors affecting M&E systems operationalization. They elaborate a set of policy recommendations on how to use M&E systems both in donor and recipient countries to raise aid effectiveness.

Findings: The authors conclude that in spite of donor and recipient countries having different purposes and approaches in implementing IDA projects, the use of the M&E is usually helpful to bridge their interests and to increase aid effectiveness.
Big Shoes to Fill: An Evaluation Journey in the Footsteps of Daniel L. Stufflebeam
Authors: Camilli Sherrie-Ann 
Published: April 2023

Background: Evaluation has evolved remarkably since the early 1960s, largely due to the innovative contributions of Daniel Stufflebeam and his colleagues. As a pioneer of evaluation methods, some of the notable achievements arising from Stufflebeam’s work include the context-input-process-product (CIPP) model, evaluation standards, and evaluation checklists.

Purpose: The purpose of this paper is to explore Daniel Stufflebeam’s journey beginning with the early days of evaluation through to his retirement and unfortunate passing at 80 years old in 2017. Key features of the CIPP model are considered within a context of other popular models for comparison with the goal of finding relevance for use of CIPP evaluation in education settings.

Setting: Not applicable.

Intervention: Not applicable.

Research Design: Literature review.

Data Collection and Analysis: Not applicable.

Findings: Stufflebeam’s CIPP model and evaluation standards remain prominent in evaluation practices and his legacy will lay the foundation for future evaluators through continued professional development.
Equity Implications in Evaluating Development Aid: The Italian Case
Authors: Forestieri Marina 
Published: April 2023

Background: In the field of development aid, social equity is an emerging issue that concerns the evaluation community in its theoretical and practical dimensions.
Purpose: A widely held belief is that evaluators do not apply theory. In this paper, we intend to verify this statement about equity in the field of cooperation projects.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: We considered equity-focused approaches and found three common factors: stakeholder participation, attention to context, and focus on marginalized groups. These elements operate as screening criteria in identifying equity issues in a case study.
Data Collection and Analysis: The paper examines a practical experience of Italian cooperation. This involved a review of evaluations reports completed between 2013 and 2014. The reports are analyzed according to the three screening criteria.
Findings: The use of the three criteria has proved its worth in grasping the issues of equity neglected and often not recognized in reports. Once again a gap emerges between theory and practice. The availability of theoretical approaches is not sufficient. The paper, therefore, proposes a reflection on the responsibility of evaluation towards social justice.
Keywords: social equity; development aid; theory and practice gap.
Meta-Evaluation Revisited
Authors: Scriven Michael 
Published: April 2023

Next month will mark the passage of forty years since I published my first article about “meta-evaluation” (Scriven, 1969), a term I had invented somewhat earlier in a report to the Urban Institute, who had asked me for help in dealing with the noncomparability of the evaluations they had commissioned for several housing projects. This year also marks the elevation of the concept to the status of an independent category in the latest edition of the Program Evaluation Standards (Joint Committee, 1994). So the editors of JMDE thought it might be time to take another look at this process and its products, and, as promised, we have provided a special section in this issue on the topic. Many thanks for the much-appreciated contributions that were sent in for that section! Before reading them, and hence without implying any criticism of them for ignoring or disagreeing with what follows, I’m going to mention some reflections on the notion that now seem to me worth stressing. They—and the other articles in this issue—may inspire reactions from you: Please put them on paper or screen, and we’d be happy to consider them for the (nonrefereed) discussion section of a later issue of JMDE.
How do Evaluators Communicate Cultural Competence? Indications of Cultural Competence through an Examination of the American Evaluation Association’s Career Center
Authors: Evergreen Stephanie D. H. , Robertson Kelly N. 
Published: April 2023

Background: Cultural competency is an important but under-adopted skill among professional evaluators. Yet in the transactions around job seeking and hiring in evaluation, cultural competency is a practical and common concept. How cultural competency gets communicated in those transactions may provide insights for the field.
Purpose: The purpose of this article is to identify ways job seekers and employers discuss cultural competency in order to move toward a more widely accepted way of operationalizing the concept.
Setting: The American Evaluation Association’s (AEA) Career Center Webpage.
Intervention: Not applicable.
Research Design: A nonexperimental design was used in the study.
Data Collection and Analysis: Document review was the main form of data collection, where resumes and job postings on AEA’s webpage were systematically collected. They were then rated by the two study authors on cultural competency and interrater reliabilities were calculated. Content analysis was also used to identify themes in ways through which cultural competency is expressed or communicated in the documents.
Findings: Indicators of cultural competency are identified, in the context of employment seeking. The study also highlights the conditions which may have contributed to low interrater reliabilities and a larger need to develop a practical, operationalized definition of cultural competency, despite inherent flaws.
The Context of Evaluation: Balancing Rigor and Relevance
Authors: Uehara Denise Lea , Tom Tammy 
Published: April 2023

Background: The context in which an evaluation is undertaken impacts not only the core evaluation activities but also the ethical standards that guide our work. Theoretical constructs and ethical decision-making frameworks often may not support us in ethical dilemmas given certain evaluation settings.
Purpose: The purpose of this article is to present an example of how a well-intentioned, responsive yet rigorous evaluation provided opportunities to expand our experiences. We conclude this paper with a set of recommendations for organizations/institutions, evaluators, project developers/ implementers, and grantors.
Setting: Not applicable
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Desk review.
Findings: The complexities of conducting an external evaluation of a study that relied on a public-private partnership coupled with a complex intervention that was not fully developed resulted in unintended study limitations and necessitated adapting evaluation strategies and making ad hoc adjustments midstream. Through our recommendations we share our lessons learned and how challenges can be creatively and ethically addressed at the onset of an evaluation.
The Pragmatist in Context of a National Science Foundation Supported Grant Program Evaluation: Guidelines and Paradigms
Authors: Ross Margaret E. , Narayanan N. Hari , Hendrix Theron Dean , Myneni Lakshman Sundeep 
Published: April 2023

Background: The philosophical underpinnings of evaluation guidelines set forth by a funding agency can sometimes seem inconsistent with that of the intervention.

Purpose: Our purpose is to introduce questions pertaining to the contrast between the instructional program’s underlying philosophical beliefs and assumptions and those underlying our evaluation approach. Drawing heavily on Scriven, we discuss these from a pragmatist evaluation stance in light of issues defined by Lincoln and Guba (2000). The discussion is couched in the evaluation of an innovative approach to teaching computer science.
Setting: Auburn University, Auburn, AL
Intervention: The evaluation is designed to investigate the effects of a studio-based teaching approach in computer science education. The evaluation framework employs a rigorous design that seeks to provide evidence to support or refute some assumed truth about the object (or construct) investigated. The program evaluated is steeped in a constructivist framework which assumes that no universal truth or reality exists, but rather, is constructed by the individual.
Research Design: Our evaluation design, to a good extent, reflects a post-positivist, quasi-experimental position. We also include a qualitative component using student interviews.
Data Collection and Analysis: Evidence of the effectiveness of the instructional approach for learning is assessed quantitatively using pre- and post-test and pre- and post-survey data group comparisons (mixed design ANOVA). Interviews provide the basis for qualitative theme analysis.
Findings: Quantitative results were somewhat weak but consistent in support of the studio-based teaching. Interview data suggest that most students did find working in groups enjoyable and a valuable experience.
Integrating the Funder-Fundee Relationship into a Model of Evaluation Utilization
Authors: Mayhew Fred 
Published: April 2023

Background: Evaluation is a tool that can promote accountability and enhance organizational improvement. For these reasons funding entities from government to foundations are increasingly relying on program evaluation as a key instrument to determine effectiveness and hold recipient organizations accountable. What has ensued is an environment of increasing evaluation efforts, where findings often weigh heavily on future funding decisions.
Purpose: The purpose of this article is to examine how the nature of the funder-fundee interorganizational relationship influences evaluation utilization by the fundee organization. Within this context, attention is paid to the role the evaluator plays in the evaluation process and the skills needed to promote utilization.
Setting: A statewide multi-tier public/private initiative addressing early childhood education.
Intervention: Not applicable.
Research Design: A model is designed to test the influence of the nature of the funder-fundee interorganizational relationship on factors associated with utilization, and ultimately on use itself.
Data Collection and Analysis: A survey was administered online and data analysis was conducted using principal components analysis (PCA) and structural equation modeling (SEM).
Findings: The results suggest that the funder-fundee relationship plays a critical role in the evaluation process, and consequently influences utilization. However, the findings indicate that the relationship among the mediating factors is more complex than originally hypothesized.
Theory Creation, Modification, and Testing: An Information Processing Model and Theory of the Anticipated and Unanticipated Consequences of Research and Development
Authors: Perla Rocco J. , Carifio James M. 
Published: April 2023

Background: Extending Merton’s (1936) work on the consequences of purposive social action, the model, theory and taxonomy outlined here incorporates and formalizes both anticipated and unanticipated research findings in a unified theoretical framework. The model of anticipated research findings was developed initially by Carifio (1975, 1977) and was followed by the addition of the unanticipated findings component by Perla (2006). This is the first formal model, theory and synthesis of anticipated and unanticipated research findings developed to date. The wide-ranging consequences and implications of the model are discussed.
Purpose: To the extent that educational researchers, philosophers and scholars reduce unanticipated findings solely to chance, whimsy, or inspiration they declare these occurrences impossible to effectively predict, model or understand. This article provides a way to conceptualize and formally model anticipated and unanticipated research findings. Many concrete examples from the history of science and research and evaluation methodology are provided to illustrate the model as well as various details of an application of the model in developing instructional materials.
Setting: Nature of science instructional materials development effort.
Intervention: Not applicable.
Research Design: Theory and model development using quantitative and qualitative methods including literature review and original model evaluations.
Data Collection and Analysis: Content analysis and modified Q-sorts of research related documents, journals, logs, literature and emails as well as various theory construction and modification techniques.
Findings: This article demonstrates that a formal model and theory of anticipated and unanticipated research findings can be developed and that such models should inform a broad range of research and evaluation efforts that are conducted daily worldwide. Nine key research and evaluation principles were derived to supplement the formal model and its operations that should be helpful to novice as well as experienced researchers regardless of the research methodology or strategies they are employing.
Responding to the UNDP Evaluations Unit
Authors: Lempert David H. 
Published: April 2023

Abstract: The author of an indicator that tests evaluation system compliance with good governance principles addresses the UNDP’s response to his article by offering an empirical test of the UNDP’s commitment to reform. While the UNDP Evaluations Office claims to be working on reforms, the test exposes the unwillingness of the UNDP Evaluations Office to take even a simple step that would correct the problems they admit exist. When presented with the chance to support reform on a real case – a de-politicized evaluation contract that protects professional standards and offers public accountability for the Spanish government’s MDG-Fund to the UNDP – the Vice Director of UNDP’s Evaluations Unit failed to support a single change. The author believes that public oversight of international organizations has been so weakened that not even their evaluation units understand anymore what democratic, public control means or how it works and they have now become part of an embedded culture that is unable to even conceive of how to apply simple public solutions.
Crowd-Sourced Evaluation: A Qualitative Study of User-Generated Product Review Videos on ExpoTV.com
Authors: Wilson Kurt 
Published: April 2023

Background: While user-generated videos are typically associated with humorous or shocking videos far removed from the rigorous world of evaluation, this paper explores the potential for utilizing user-generated videos as evaluation data. This topic is addressed through a qualitative pilot study of product review videos available on the ExpoTV.com website. The findings from this important subset of the evaluation field are analyzed with the goal of identifying themes and insights that could be useful to the broad field of evaluation. Purpose: The primary purpose of this study is to identify characteristics or insights about the user-generated content that could serve as a guide for future studies and the development of new theory or methodology to enlarge the scope and relevance of data used in evaluations. Setting: The ExpoTV.com website Intervention: Not applicable. Research Design: Grounded theory. Data Collection and Analysis: User-generated videos downloaded from the ExpoTV.com website. Analyzed in three stages – In Vivo coding, Axial coding and theme identification. Findings: This article finds intriguing strengths to user-generated video as an evaluation data source because user-generated product review videos consistently demonstrate three important elements of evaluation: they provide a description of the product, explain the broader context for both the product and nature of the specific review, and provide an evaluative conclusion that is logically related to specific evaluative descriptions. Keywords: user-generated video; grounded theory; video data; evaluation methodology
Evaluation Roots: An International Perspective
Authors: Carden Fred , Alkin Marvin C. 
Published: April 2023

Background: The second edition of Evaluation Roots has expanded to more global coverage, but no chapter emerged on development theory in low and middle income countries (LMICs). Purpose: The purpose of this article is initiate a conversation on filling this gap. Setting: Not applicable. Intervention: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Desk review. Findings: Two important distinctions emerged in evaluation roots in LMICs. The first is that much evaluation fits effectively on the Theory Tree as it is presented, with perhaps some nuance, but we see a collective rather than individual origin of the evaluation theory writings we have uncovered. The second is the nature of the formal evaluation work in LMICs; its practical origins are not yet formalized into full prescriptive theory. The paper notes the prominence of external actors (such as the donor community) in formalizing evaluation practice, while at the same time observes important threads are developing in different regions of the world. This paper proposes a way to thinking about evaluation in LMICs that is based on practice. The paper suggests a need for evaluation analysts, especially those in LMICs, to build a more comprehensive knowledge and documentation on development evaluation and, more broadly on building the field of evaluation in LMICs. Keywords: development evaluation; Evaluation Roots; Evaluation Theory Tree; evaluation field building
Acquired haemophilia A in southern Switzerland from 2013 to 2019: a case series
Authors: Ruberti Andrea , Kremer Hovinga Johanna A. , Nappi Federico , Vettese Aurora , Bianchi Elena , Fernandes Eliana , Galfetti Elena , Monotti Rita , Paul Pamella , Regazzoni Stefano , Valente Daniela , Rossi Davide , Stussi Georg , Gerber Bernhard 
Published: April 2023

AIMS OF THE STUDY: Acquired haemophilia A is a rare disease with an annual incidence of 1.48 per million. Based on clinical observations, we suspect a higher incidence in southern Switzerland, and aimed at providing local epidemiological data, and clinical information regarding diagnosis, treatment and outcome in our region.
METHODS: All adult patients with acquired haemophilia A treated between 2013 and 2019 in our facility were included in the present retrospective analysis.
RESULTS: We treated 11 patients with acquired haemophilia A between 2013 and 2019, resulting in an annual incidence of 4.5 per million (95% confidence interval [CI] 0–9.0). Median delay from first symptoms to diagnosis was 4.5 days, and the median age at diagnosis was 79 years (range 23–87). Possible causative conditions were: pregnancy (n = 1), polyarteritis nodosa (n = 1), myelodysplastic syndrome (n = 1), chronic human immunodeficiency virus (HIV) (n = 1), and HIV postexposure prophylaxis (n = 1). In five patients no underlying or associated condition was identified. Median activated partial thromboplastin time (aPTT)) at baseline was 79 seconds (65–117; ref. value <38 sec), and FVIII:C 2.15% (<1–3.75%). A FVIII:C <1% was present in 4/10 patients. Median FVIII-inhibitor titre was 10.3 BU/ml (2.4–75.0 BU/ml). All patients had bleeding symptoms, 5/10 patients had major bleedings, and 7/10 patients were treated with bypassing agents. All patients received corticosteroids; 7/10 patients received immunosuppressive combination therapy. FVIII levels of ≥50% were achieved after a median of 40 days (8–62). One patient had a severe immunosuppressive therapy-related infection. An 87-years-old woman died for reasons not related to acquired haemophilia A or immunosuppressive therapy.
CONCLUSIONS: Acquired haemophilia A is a rare disease, but manageable despite the advanced patient age and comorbidities. Its incidence in Southern Switzerland is higher than previously suspected.