bluuebunny/crossref_metadata_2025_split huggingface dataset:
Impact Evaluation Based on Buddhist Principles
Authors: Russon Craig , Russon Karen 
Published: April 2023

Background: This is one in a series of articles in which the authors attempt to relate Eastern philosophy to contemporary programme planning and evaluation. Setting: Not applicable. Purpose: The authors examine impact evaluation through the lens of Eastern Philosophy. Subjects: Not applicable. Research Design: Not applicable. Data Collection and Analysis: The authors examine the basic causal statements from the approach to impact evaluation commonly used by The World Bank and from Buddhist philosophy. Second, they examine the econometric assumptions on which impact evaluation is often based and propose alternative Buddhist principles. Lastly, they speculate what impact evaluation might look like using the alternative principles that were identified. Findings: Not applicable. Conclusions: There is no such thing as impact in and of itself. Rather, a combination of conditions comes together in a certain way, at a certain time, and we call it an impact. Impact is, therefore, the result of conditionality (Salzberg, & Goldstein, 2001). Evaluation that examined the conditionality of impact would be in a position to make statements about patterns of relationships. Instead of experimental and quasi-experimental approaches, the Insight Evaluation approach (Russon and Russon, 2011) might be more appropriate.
The Quality of Mathematics Education Technology Literature
Authors: Ronau Robert N. , Rakes Christopher R. , Bush Sarah B. , Driskell Shannon O. , Niess Margaret L. , Pugalee David 
Published: April 2023

Background: The present study evaluated the quality of 1,165 scholarly literature papers about mathematics education technology literature.Purpose: The purpose of the present study was to determine the extent to which mathematics education technology literature reports the information needed to support the scientific basis of a study.Setting: N/AIntervention: N/AResearch Design: A systematic review was used to organize the data collection and analysis processes Data Collection and Analysis: A literature search was conducted to identify scholarly papers that addressed the use of technology in mathematics education. A coding process was developed to record descriptive information about each paper. The Quality Framework developed for this process provided a structure to identify key information across research types based on types of analyses conducted, assigning a certain number of possible points based on the type of research conducted.Findings: Dissertations accounted for a surprisingly high portion of the literature and research: 39.7% of the available literature and 57.0% of the research studies. The overall quality of the mathematics education technology literature was lower than we expected, averaging only 48.9% of the points possible. We noted that the quality of research papers, with respect to possible point values averaged 54.6% over four decades. For mathematics education technology researchers, manuscript reviewers, and editors, these results suggest that more attention is needed on the information being included and excluded from scholarly papers, especially with regard to connections to theoretical frameworks and research designs.
Propensity Scores: A Practical Introduction Using R
Authors: Olmos Antonio , Govindasamy Priyalatha 
Published: April 2023

Background: This paper provides an introduction to propensity scores for evaluation practitioners. Purpose: The purpose of this paper is to provide the reader with a conceptual and practical introduction to propensity scores, matching using propensity scores, and its implementation using statistical R program/software. Setting: Not applicable Intervention: Not applicable Research Design: Not applicable Data Collection and Analysis: Not applicable Findings: In this demonstration paper, we describe the context in which propensity scores are used, including the conditions under which the use of propensity scores is recommended, as well as the basic assumptions needed for a correct implementation of the technique. Next, we describe some of the more common techniques used to conduct propensity score matching. We conclude with a description of the recommended steps associated with the implementation of propensity score matching using several packages developed in R, including syntax and brief interpretations of the output associated with every step. Keywords: propensity score analysis; propensity score matching; R.
Conference Evaluation Focusing on Learning and Transfer
Authors: Andersen Michael , Wahlgren Bjarne 
Published: April 2023

Background: The article presents the results of using a model for evaluation of conferences; a model focusing on learning and transfer. Many conferences are evaluated using participant-satisfaction surveys, but satisfaction is a diffuse concept and is linked to many factors other than learning. Purpose: The purpose of the present study was to examined how participants of four national conferences of adult learning assessed the relevance of the content of a number of workshops, how much they had learned from the workshops and whether they had used what they had learned. And to test a tool to measure this. Setting: The subject matters of the study is four evaluations based on surveys of national conferences in Denmark in 2010, 2011, 2012 and 2013. The overall purpose of the conferences was to communicate and discuss new knowledge within the area of general adult education and vocational education and training, and to help ensure that this knowledge is subsequently used in the participants' daily practices at work. Intervention: Testing a tool to measure learning and transfer, which is not too resource-intensive to use for managers of workshops as well as conferences to strengthen the potential for learning and transfer by participants. Research Design: Using the same survey-based evaluation-tool in four different conferences in order to compare the results and try to find general knowledge. Data Collection and Analysis: Collecting and analysing data on the basis of survey-data of the participant’s self-reported perception of relevance, learning and use (transfer) of the content of the workshops. Findings: The study shows that the percentage of conference participants who experience relevance, learning and transfer from the conferences only varies marginally from year to year, while this percentage varies particularly between the individual workshops. Another finding is that participants’ assessment of relevance is more associated than satisfaction with learning and transfer. Keywords: evaluation; conference; workshop; learning; transfer; adult learning.
Failing Forward Quickly as a Developmental Evaluator: Lessons from Year One of the LiveWell Kershaw Journey
Authors: Hayes Holly G. , Witkowski Susan , Smith Lillian 
Published: April 2023

Background: Learning to be a developmental evaluator is challenging because it is relatively new and sparsely documented in the scholarly literature. Developmental evaluation is intended to support the ever-changing and adaptive nature of complex environments. In a developmental, systems-oriented evaluation framework, the evaluator is embedded in the process to support and generate learnings from ongoing findings. Purpose: This article presents the lessons learned through one case study and how the concept of failing forward can guide the evaluator’s reflective process through a developmental evaluation.Setting: Free clinic and community setting in Kershaw County, South Carolina Intervention: NA Research Design: NA Data Collection and Analysis: The Evaluator documented mistakes and lessons learned during the beginning, planning and implementation stages of a complex community health initiative. Findings: The evaluation team shares five mistakes made along during the journey and lessons learned. It’s important for teams to understand what differentiates developmental evaluation from other types of evaluation and the role of the evaluator. The “critical friend” boundary can be easily crossed and the evaluator needs to have a strong understanding of the needs of the initiative.
Are All Biases Bad? Collaborative Grounded Theory in Developmental Evaluation of Education Policy
Authors: Anderson Ross , Guerreiro Meg , Smith Jo 
Published: April 2023

Background: Using two researchers as independent instruments for interpretation in education policy evaluation, this study applies a collaborative grounded theory approach to qualitative data analysis and theory generation. Purpose: This study argues that varied perspectives should be a critical component in the methodological and analytical choices of education research, especially when the sought after outcome is deeper understanding of the impact, both positive and negative, of an education program or policy. In this study, rather than using one researcher to confirm the reliability of the other, the study explores the outcome of drawing on the positional reflexivity of two researchers, each with a distinct perspective, as a potential strength to co-generate themes and theory in the evaluation of complex policy or programs.Setting: The data for this analysis originated from interviews of education leaders (n = 13) from two states with contrasting approaches to teacher evaluation: Kentucky and California. Intervention: NA Research Design: Qualitative developmental evaluation Data Collection and Analysis: Semi-structured interviews and grounded theory coding Findings: Results suggest that more robust theory and analysis may result from independent thematic development and converged theory generation when working in a research team, as opposed to early application of inter-rater reliability. The reflexivity of different perspectives was, in part, reflexive of the self—their own biased perspective and prior experience. When merged, their joint interpretation may have unearthed greater dimensionality. Findings from this study can inform future strategies for evaluating qualitative research data, especially within a developmental evaluation approach aimed at understanding system complexity in education policy and practice.
On the Feasibility of Extending Social Experiments to Wider Applications
Authors: Bell Stephen H. , Peck Laura R. 
Published: April 2023

Background: When deciding how to allocate limited funds for social programs, policymakers and program managers increasingly ask for evidence of effectiveness based on studies that rely on solid methodology, providing credible scientific evidence. The basic claim for the “social experiment”—that the “coin flip” of randomization creates two statistically equivalent groups that do not diverge except through an intervention’s effects—makes resulting estimates unbiased. Despite the transparency and conceptual strength of the experimental strategy for revealing the causal connection between an intervention and the outcomes of its participants, the wisdom or feasibility of conducting social experiments is often questioned on a variety of grounds. Purpose: This article defines 15 common concerns about the viability and policy reliability of social experiments, in order to assess how much these issues need constrain the use of the method in providing policy evidence. Setting: NA Intervention: NA Research Design: The research uses the authors’ experience designing and conducting dozens of social experiments to examine the basis for and soundness of each concern. It provides examples from the scholarly literature and evaluations in practice of both the problems posed and responses to each issue. Data Collection and Analysis: NA Findings: We conclude that none of the 15 concerns precludes substantially extending the use of randomized experiments as a means of evaluating the impacts of government and foundation social policies and programs.
Retrospective Pretest and Counterfactual Self-Report: Different or Same?
Authors: Lam Tony C.M. , Valencia Edgar 
Published: April 2023

Purpose: To examine discriminant validity of treatment participants’ self-report of the state they would be in had they not received treatment (counterfactual); specifically, the distinction between self-report of counterfactual and self-report of preintervention state (retrospective pretest).

Setting: An education department of a large University in North America.

Intervention: Methods of self-reporting research self-efficacy with counterfactual items and with retrospective pretest items.

Research design: A randomized comparison group design with two treatments that were defined by the version of the survey used in each. In the survey for the counterfactual condition, items about research self-efficacy without the influence of their program of studies were included. The survey in the retrospective pretest condition contained items regarding research self-efficacy before participating in their program of study. The same items about research self-efficacy at the current time (posttest) were included in both treatment conditions.

Data collection & analysis: Participants were graduate students recruited via email who answered an online survey about research self-efficacy. These students were randomly assigned to one of the two aforementioned treatments. Responses were analyzed using a mixed 2 by 2 randomized factorial ANOVA design with self-report method (counterfactual or retrospective pretest) as the between-subjects factor and time (pre and post intervention) as the within-subjects factor.

Findings: Our findings show that counterfactual and retrospective pretest scores and treatment effects computed based on these two sets of scores are virtually identical, casting doubt on participants’ ability to differentiate between a state of no treatment and a state at treatment commencement after they have received treatment.
Towards a Complexity Framework for Transformative Evaluation
Authors: Picciotto Robert 
Published: April 2023

Background: Complexity ideas originating in mathematics and the natural sciences have begun to inform evaluation practice. A new wave in evaluation history is about to break. A new mindset, new methods, and new evaluation processes are being summoned to explore and address the challenges of global pandemics, growing inequities, and existential environmental risks. This is part of a broader paradigm shift underway in science where interdisciplinarity has become the norm rather than the exception.
Purpose: This article explores the utility of a complexity framework for a more effective evaluation function. It unearths the antecedents of complexity thinking; explores its relevance to evolving knowledge paradigms; provides a bird’s eye view of complexity concepts; uses the logic of complex adaptive systems to unpack the role of evaluation in society; and draws the implications of contemporary social challenges for evaluation policy directions.
Setting: Not applicable.
Intervention: Not applicable.
Research design: Not applicable.
Findings: The evaluation complexity challenge coincides with an urgent imperative: social transformation. The on-going pandemic has brought to light the disproportionate effects of health emergencies on disadvantaged groups and emphasized the urgency of improving the interface between humans and nature. It has also demonstrated the importance of modelling for policy making – as well as its limitations. Evaluation, a complex adaptive system, should be transformed to serve society.
Keywords: complexity; computers; disciplines; emergence; modelling; paradigm, systems
Evaluating in a Fragmented Society
Authors: House Ernest R. 
Published: April 2023

Background: Over decades American society has become increasingly fragmented, distrusting, and unequal. Distrust and inequality interact with institutions performing improperly to weaken the society.
Purpose: To suggest ways to strengthen evaluation’s role in a changing society
Setting: Evaluation has entered a post normal phase where evaluations are losing credibility and effectiveness.
Intervention: Analyze the changing society and suggest adjustments that evaluators might make.
Research design: Collate and synthesize empirical studies about society and the implications for evaluators.
Data collection and analysis: Collect and interpret seminal empirical economic, sociological, and political studies of beliefs and inequality in the United States.
Findings: To strengthen the potency of evaluations of any type, evaluators could act as moral fiduciaries, practice transparency, cultivate cognitive empathy, focus on deep stories and deep values, and mitigate inequalities in the evaluation space. They can act as critics of evaluation practices inside and outside the evaluation space. They should avoid technical, social, and situational biases, including racism, sexism, and conflicts of interest, to increase the honesty and credibility of evaluations. They should not allow career concerns to prevent them from doing the right thing. These professional ethics and practices can be applied singly or collectively to most evaluation approaches to strengthen the evaluator’s role and address major societal problems.
Keywords: moral fiduciary; cognitive empathy; post normal; inequality; transparency; distrust; deep stories; values
Equity “On the Sideline”
Authors: Gates Emily F , Williamson Eric , Madres Joseph , Benitez Alvarez Kayla , Hall Jori N 
Published: April 2023

Background: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice.
Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study.
Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education).
Intervention: Not applicable
Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis.
Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation.
How the I Ching or Book of Changes can Inform Western Notions of Theory of Change
Authors: Russon Craig , Russon Karen 
Published: April 2023

Background: This article is the third part of a trilogy. The Eastern paradigm article (Russon, 2008) dealt with ontological issues (the nature of reality). The insight evaluation article (Russon & Russon, 2009) dealt with epistemological issues (how we can know reality).
Purpose: This article deals with a methodological issue—specifically it explores how ancient Chinese philosophy might influence the way in which modern day evaluators think about theories of change.
Setting: Not applicable.
Subjects: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: The paper is the result of a desk review that compared and contrasted modern theory of change literature with the writings contained in the I Ching or Book of Changes.
Findings: The authors believe that the theories of change expressed in the systems diagrams of the I Ching would be considered to be conducive to recurrent change that occurs due to cycles in the natural world. They further believe that the change which harnesses the power of naturally occurring cycles offers the best hope for long-term sustainability. This is because such change takes advantage of an intuitive cooperation with the natural order.
Conclusions: The authors believe that using the diagrams of the I Ching as a theory of change template offers modern day evaluators a number of advantages. The advantage of greater sustainability has already mentioned. The authors believe that organic theories of change also encourage evaluators to think about the contribution from many factors instead of attribution of a few factors.
What Does it Take to do Evaluation in Communities and Cultural Contexts Other Than Our Own?
Authors: Wehipeihana Nan , Davidson E. Jane , McKegg Kate , Shanker Vidhya 
Published: April 2023

Background: This paper builds on the growing body of evaluation literature around the importance of culture and cultural context in evaluation (e.g. Greene, 2005; Hood, Hopson and Frierson 2005; Hopson, 2009; Kirkhart, 1995 and 2005; La France, 2001).
Purpose: The place of language, culture, cultural context, and leadership roles in evaluation is explored through consideration of the question, “What does it take to do evaluation in communities and cultural contexts other than our own?”
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Findings: Attention to the location of power and privilege in evaluation, and to community engagement and ‘sense-making’ processes are the conversational starting points to begin to explore what it takes to do evaluation in communities, where the language, culture, and cultural context are different from one’s own.
The Use of Kōans for Insight Evaluation Approach Training
Authors: Russon Craig , Russon Karen 
Published: April 2023

Background: Since publishing a trilogy of articles relating Eastern mysticism to evaluation, the authors have received many inquiries about training. Purpose: The article explains one promising new method under development—adapting the Zen practice of kōans for evaluation training. Setting: Not applicable. Subjects: Not applicable. Research Design: Not applicable. Data Collection and Analysis: Not applicable. Findings: Not applicable. Conclusions: The authors believe that kōans can be used as a poetic technology for recalibrating evaluation practioners’ attitudes and actions about evaluation. Keywords: Insight evaluation approach, Eastern mysticism, kōans, training, Zen, enlightenment
Using Readability Tests to Improve the Accuracy of Evaluation Documents Intended for Low-Literate Participants
Authors: Kouamé Julien B. 
Published: April 2023

Background: Readability tests are indicators that measure how easy a document can be read and understood. Simple, but very often ignored, readability statistics cannot only provide information about the level of difficulty of the readability of particular documents but also can increase an evaluator’s credibility. Purpose: The purpose of this article is two-fold: (1) to provide readers with logical reasons for using readability tests and (2) how to choose the right test for a project. Setting: United States. Research Design: A comparative framework is used to present the need for readability testing. Keywords: Readability tests, evaluation instruments, survey research, low-literacy survey