bluuebunny/crossref_metadata_2025 huggingface dataset:
Propensity Scores: A Practical Introduction Using R
Authors: Olmos Antonio , Govindasamy Priyalatha
Published: April 2023

Background: This paper provides an introduction to propensity scores for evaluation practitioners.  Purpose: The purpose of this paper is to provide the reader with a conceptual and practical introduction to propensity scores, matching using propensity scores, and its implementation using statistical R program/software. Setting: Not applicable Intervention: Not applicable Research Design: Not applicable   Data Collection and Analysis: Not applicable Findings: In this demonstration paper, we describe the context in which propensity scores are used, including the conditions under which the use of propensity scores is recommended, as well as the basic assumptions needed for a correct implementation of the technique. Next, we describe some of the more common techniques used to conduct propensity score matching. We conclude with a description of the recommended steps associated with the implementation of propensity score matching using several packages developed in R, including syntax and brief interpretations of the output associated with every step. Keywords: propensity score analysis; propensity score matching; R.
Conference Evaluation Focusing on Learning and Transfer
Authors: Andersen Michael , Wahlgren Bjarne
Published: April 2023

Background: The article presents the results of using a model for evaluation of conferences; a model focusing on learning and transfer. Many conferences are evaluated using participant-satisfaction surveys, but satisfaction is a diffuse concept and is linked to many factors other than learning. Purpose: The purpose of the present study was to examined how participants of four national conferences of adult learning assessed the relevance of the content of a number of workshops, how much they had learned from the workshops and whether they had used what they had learned. And to test a tool to measure this. Setting: The subject matters of the study is four evaluations based on surveys of national conferences in Denmark in 2010, 2011, 2012 and 2013. The overall purpose of the conferences was to communicate and discuss new knowledge within the area of general adult education and vocational education and training, and to help ensure that this knowledge is subsequently used in the participants' daily practices at work. Intervention: Testing a tool to measure learning and transfer, which is not too resource-intensive to use for managers of workshops as well as conferences to strengthen the potential for learning and transfer by participants. Research Design: Using the same survey-based evaluation-tool in four different conferences in order to compare the results and try to find general knowledge. Data Collection and Analysis: Collecting and analysing data on the basis of survey-data of the participant’s self-reported perception of relevance, learning and use (transfer) of the content of the workshops. Findings: The study shows that the percentage of conference participants who experience relevance, learning and transfer from the conferences only varies marginally from year to year, while this percentage varies particularly between the individual workshops. Another finding is that participants’ assessment of relevance is more associated than satisfaction with learning and transfer. Keywords: evaluation; conference; workshop; learning; transfer; adult learning.
Failing Forward Quickly as a Developmental Evaluator: Lessons from Year One of the LiveWell Kershaw Journey
Authors: Hayes Holly G. , Witkowski Susan , Smith Lillian
Published: April 2023

Background: Learning to be a developmental evaluator is challenging because it is relatively new and sparsely documented in the scholarly literature. Developmental evaluation is intended to support the ever-changing and adaptive nature of complex environments. In a developmental, systems-oriented evaluation framework, the evaluator is embedded in the process to support and generate learnings from ongoing findings. Purpose: This article presents the lessons learned through one case study and how the concept of failing forward can guide the evaluator’s reflective process through a developmental evaluation.Setting: Free clinic and community setting in Kershaw County, South Carolina Intervention: NA Research Design: NA Data Collection and Analysis: The Evaluator documented mistakes and lessons learned during the beginning, planning and implementation stages of a complex community health initiative. Findings: The evaluation team shares five mistakes made along during the journey and lessons learned.  It’s important for teams to understand what differentiates developmental evaluation from other types of evaluation and the role of the evaluator.  The “critical friend” boundary can be easily crossed and the evaluator needs to have a strong understanding of the needs of the initiative.
Are All Biases Bad? Collaborative Grounded Theory in Developmental Evaluation of Education Policy
Authors: Anderson Ross , Guerreiro Meg , Smith Jo
Published: April 2023

Background: Using two researchers as independent instruments for interpretation in education policy evaluation, this study applies a collaborative grounded theory approach to qualitative data analysis and theory generation. Purpose: This study argues that varied perspectives should be a critical component in the methodological and analytical choices of education research, especially when the sought after outcome is deeper understanding of the impact, both positive and negative, of an education program or policy. In this study, rather than using one researcher to confirm the reliability of the other, the study explores the outcome of drawing on the positional reflexivity of two researchers, each with a distinct perspective, as a potential strength to co-generate themes and theory in the evaluation of complex policy or programs.Setting: The data for this analysis originated from interviews of education leaders (n = 13) from two states with contrasting approaches to teacher evaluation: Kentucky and California. Intervention: NA Research Design:  Qualitative developmental evaluation Data Collection and Analysis: Semi-structured interviews and grounded theory coding Findings: Results suggest that more robust theory and analysis may result from independent thematic development and converged theory generation when working in a research team, as opposed to early application of inter-rater reliability. The reflexivity of different perspectives was, in part, reflexive of the self—their own biased perspective and prior experience. When merged, their joint interpretation may have unearthed greater dimensionality. Findings from this study can inform future strategies for evaluating qualitative research data, especially within a developmental evaluation approach aimed at understanding system complexity in education policy and practice.
On the Feasibility of Extending Social Experiments to Wider Applications
Authors: Bell Stephen H. , Peck Laura R.
Published: April 2023

Background: When deciding how to allocate limited funds for social programs, policymakers and program managers increasingly ask for evidence of effectiveness based on studies that rely on solid methodology, providing credible scientific evidence. The basic claim for the “social experiment”—that the “coin flip” of randomization creates two statistically equivalent groups that do not diverge except through an intervention’s effects—makes resulting estimates unbiased. Despite the transparency and conceptual strength of the experimental strategy for revealing the causal connection between an intervention and the outcomes of its participants, the wisdom or feasibility of conducting social experiments is often questioned on a variety of grounds. Purpose: This article defines 15 common concerns about the viability and policy reliability of social experiments, in order to assess how much these issues need constrain the use of the method in providing policy evidence. Setting: NA Intervention: NA Research Design: The research uses the authors’ experience designing and conducting dozens of social experiments to examine the basis for and soundness of each concern.  It  provides examples from the scholarly literature and evaluations in practice of both the problems posed and responses to each issue. Data Collection and Analysis: NA Findings: We conclude that none of the 15 concerns precludes substantially extending the use of randomized experiments as a means of evaluating the impacts of government and foundation social policies and programs.
Retrospective Pretest and Counterfactual Self-Report: Different or Same?
Authors: Lam Tony C.M. , Valencia Edgar
Published: April 2023

Purpose: To examine discriminant validity of treatment participants’ self-report of the state they would be in had they not received treatment (counterfactual); specifically, the distinction between self-report of counterfactual and self-report of preintervention state (retrospective pretest).

 

Setting: An education department of a large University in North America.

 

Intervention: Methods of self-reporting research self-efficacy with counterfactual items and with retrospective pretest items.

 

Research design: A randomized comparison group design with two treatments that were defined by the version of the survey used in each. In the survey for the counterfactual condition, items about research self-efficacy without the influence of their program of studies were included. The survey in the retrospective pretest condition contained items regarding research self-efficacy before participating in their program of study. The same items about research self-efficacy at the current time (posttest) were included in both treatment conditions.

 

Data collection & analysis: Participants were graduate students recruited via email who answered an online survey about research self-efficacy. These students were randomly assigned to one of the two aforementioned treatments. Responses were analyzed using a mixed 2 by 2 randomized factorial ANOVA design with self-report method (counterfactual or retrospective pretest) as the between-subjects factor and time (pre and post intervention) as the within-subjects factor.

 

Findings: Our findings show that counterfactual and retrospective pretest scores and treatment effects computed based on these two sets of scores are virtually identical, casting doubt on participants’ ability to differentiate between a state of no treatment and a state at treatment commencement after they have received treatment.
Towards a Complexity Framework for Transformative Evaluation
Authors: Picciotto Robert
Published: April 2023

Background:  Complexity ideas originating in mathematics and the natural sciences have begun to inform evaluation practice. A new wave in evaluation history is about to break. A new mindset, new methods, and new evaluation processes are being summoned to explore and address the challenges of global pandemics, growing inequities, and existential environmental risks. This is part of a broader paradigm shift underway in science where interdisciplinarity has become the norm rather than the exception.

Purpose: This article explores the utility of a complexity framework for a more effective evaluation function. It unearths the antecedents of complexity thinking; explores its relevance to evolving knowledge paradigms; provides a bird’s eye view of complexity concepts; uses the logic of complex adaptive systems to unpack the role of evaluation in society; and draws the implications of contemporary social challenges for evaluation policy directions.

Setting: Not applicable.

Intervention: Not applicable.

Research design: Not applicable.

Findings: The evaluation complexity challenge coincides with an urgent imperative: social transformation. The on-going pandemic has brought to light the disproportionate effects of health emergencies on disadvantaged groups and emphasized the urgency of improving the interface between humans and nature. It has also demonstrated the importance of modelling for policy making – as well as its limitations. Evaluation, a complex adaptive system, should be transformed to serve society.

Keywords: complexity; computers; disciplines; emergence; modelling; paradigm, systems
Evaluating in a Fragmented Society
Authors: House Ernest R.
Published: April 2023

Background: Over decades American society has become increasingly fragmented, distrusting, and unequal. Distrust and inequality interact with institutions performing improperly to weaken the society.

Purpose: To suggest ways to strengthen evaluation’s role in a changing society

Setting: Evaluation has entered a post normal phase where evaluations are losing credibility and effectiveness.

Intervention: Analyze the changing society and suggest adjustments that evaluators might make.

Research design: Collate and synthesize empirical studies about society and the implications for evaluators.

Data collection and analysis: Collect and interpret seminal empirical economic, sociological, and political studies of beliefs and inequality in the United States.

Findings: To strengthen the potency of evaluations of any type, evaluators could act as moral fiduciaries, practice transparency, cultivate cognitive empathy, focus on deep stories and deep values, and mitigate inequalities in the evaluation space. They can act as critics of evaluation practices inside and outside the evaluation space. They should avoid technical, social, and situational biases, including racism, sexism, and conflicts of interest, to increase the honesty and credibility of evaluations. They should not allow career concerns to prevent them from doing the right thing. These professional ethics and practices can be applied singly or collectively to most evaluation approaches to strengthen the evaluator’s role and address major societal problems.

Keywords: moral fiduciary; cognitive empathy; post normal; inequality; transparency; distrust; deep stories; values
Equity “On the Sideline”
Authors: Gates Emily F , Williamson Eric , Madres Joseph , Benitez Alvarez Kayla , Hall Jori N
Published: April 2023

Background: Centering equity in evaluations is increasingly recognized as an important professional responsibility of evaluators. While some theoretical and practical guidance exists, the evaluation field has limited empirical research on equity within evaluation practice.
Purpose: This paper explores whether and how evaluators address inequities and advance equity throughout evaluation phases drawing on select findings from a larger study.
Setting: The study focuses on American Evaluation Association-affiliated evaluators in the New England region of the United States who work in a variety of areas (e.g., health, education).  
Intervention: Not applicable
Research Design: The study uses a complementarity, sequential mixed methods design comprised of a researcher-developed online questionnaire administered to a census and snowball sample of practicing evaluators (n=82) and individual, semi-structured interviews with a subset of this sample selected to maximize variation (n=21). Quantitative data were analyzed using descriptive statistics (i.e., means and standard deviations, frequencies). Qualitative data were analyzed using a collaborative process of deductive and inductive coding followed by thematic analysis.
Findings: Eight overarching findings suggest that despite evaluators’ attempts to center equity, it remains largely “on the sideline.” This is due to evaluators’ need to work against some conventional professional and methodological norms, within contractual and contextual constraints, and with limited professional preparation.
Competitive champions versus cooperative advocates
Authors: Rogers Alison Frances , Gullickson Amy M , King Jean A , McKinley Elizabeth
Published: April 2023

Abstract
Background: Evaluation offers non-profit organizations an opportunity to improve their services, demonstrate achievements, and be accountable. The extant literature identifies individuals who can enhance the uptake of evaluation as evaluation champions. However, a paucity of detail is available regarding how to identify them and the support they require.
Purpose: This research investigated the characteristics and motivations of evaluation champions and examined how they promoted and embedded evaluation in an organizational system.
Setting: Australian human and social service non-profit organizations.
Research design: Drawing upon the literature and social interdependence theory, the research took an interpretivist perspective to collaboratively generate knowledge about evaluation champions. The aim was to understand and develop a reconstruction of the characteristics of individuals. This article constitutes a component of a larger research project.
Data Collection and Analysis: This research used purposive sampling to recruit champions working in Australian non-profit organizations, who were identified via descriptive criteria gleaned from a literature review. The research involved interviewing 17 champions, four of whom also participated in organizational case studies. Analysis of the semi-structured interviews and case studies generated information about the activities, strategies, motivations, and attributes of individuals who championed and advocated for evaluation.
Findings: This article argues that evaluation advocates is a preferable descriptor when attempting to embed evaluation by cultivating mutually beneficial interactions and cooperative working relationships. This research defines evaluation advocates as individuals who motivate others and provide energy, interest, and enthusiasm by connecting evaluation with colleagues’ personal aspirations and the organizational goals to make judgements about effectiveness. This article includes a field guide to facilitate evaluation advocates’ identification, recruitment, support, and development.
Zen and the Art of Everyday Evaluation
Authors: Scriven Michael
Published: April 2023

The article draws a parallel between Zen Buddhism's aims and evaluation's goals of extending everyday reasoning into science. It announces an essay competition around the question of what values to use when evaluating revolutions. A humorous section points readers to the Journal of Nondestructive Evaluation.
Evaluation in Europe: An Overview
Authors: Schröter Daniela C.
Published: April 2023

The article presents an overview of evaluation in Europe, highlighting the activities and publications of the European Evaluation Society.
Evaluation in Eastern Europe and the Middle East
Authors: Gugiu P. Cristian
Published: April 2023

The article briefly examines the current state of evaluation in Europe and the Middle East.
New Directions for Evaluation (Vol. 102)
Authors: Keating Mary
Published: April 2023

A summary of issue 102 of New Directions for Evaluation entitled "In Search of Cultural Competence in Evaluation - Toward Principles and Practices" published in summer 2004.
Marketing Evaluation as a Profession and Discipline
Authors: Davidson E. Jane
Published: April 2023

It can be a bit like pushing sand uphill with a pointy stick, as they say here in New Zealand. One of the great challenges in developing evaluation as a discipline is getting it recognised as being distinct from the various other disciplines to which it applies. In this piece, I offer a few reflections on the challenges with this, recount a story where a group of practitioners from outside the discipline actually sat up and took notice, and propose some possible solutions for moving us forward.
Is Cost Analysis Underutilized in Decision Making?
Authors: Persaud Nadini
Published: April 2023

Is cost analysis underutilized in decision making? Research suggests it is.
An Update on Evaluation in Canada
Authors: Coryn Chris L. S.
Published: April 2023

The most pertinent news on the Canadian front is of course the 2005 joint Canadian Evaluation Society (CES)/American Evaluation Association (AEA)conference: Crossing Borders, Crossing Boundaries.
Canadian Journal of Program Evaluation, Volume 19(2), Fall 2004
Authors: Coryn Chris L. S.
Published: April 2023

The Canadian Journal of Program Evaluation's (CJPE) most recent issue contains 7 articles (5 in English and 2 in French), a research and practice note, and 3 book reviews.
National and Regional Evaluation Networks
Authors: IOCE
Published: April 2023

The following list of national and regional evaluation networks was obtained from the International Organization for Cooperation in Evaluation (IOCE) at http://ioce.net. The IOCE is an organization for evaluation networks and societies that is committed to building a worldwide evaluation community.
The Evaluation Capacity Development Group
Authors: Russon Craig , Russon Karen
Published: April 2023

Last February, Craig Russon served as a resource person at The World Bank’s Global Grantmaking for Small Grants Workshop (Russon, 2005). During his presentation on tracking results and measuring effectiveness, he advanced the idea that the evaluation needs of projects funded by small grants and those funded by large grants are essentially the same in that they both seek to answer two fundamental questions: “Are we doing the right things?”, and “Are we doing things right?”
Review of Evaluation: The International Journal of Theory, Research, and Practice, Volumes 11(2), 11(3), and 11(4)
Authors: Schröter Daniela C.
Published: April 2023

The following contribution provides a summary of Volumes 11(2), 11(3), and 11(4) of Evaluation: The International Journal of Theory, Research and Practice (2005). The central motif of issue 11(2) is learning, that is, learning from theories of change, “accountability (f)or learning,” learning aided by cluster evaluation, facets of organizational learning, and learning from the use of certain research designs. Issue 11(3) centers on issues and approaches of (primarily policy) evaluation in different countries (Belgium, the United Kingdom, Norway, and Japan) and contexts (development evaluation, educational evaluation, social service evaluation). Issue 11(4) stresses evaluation designs, methodological approaches, evaluator competencies, and problems of bottom-up versus top-down evaluation. The following briefly summarized the contributions of each volume.
Review of the Japanese Journal of Evaluation Study
Authors: Sasaki Ryoh
Published: April 2023

The Japan Evaluation Society was established in 2000. Since then, its professional journal, called Japanese Journal of Evaluation Study (JJES), has been published twice a year. These issues can be downloaded from the Society’s website (http://www.idcj.or.jp/JES). We can clearly observe several hot issues recently discussed by Japanese evaluation professionals in the journal. In this paper, three of them discussed in the 2005 issues are explained briefly.
The Evaluation of Research Merit versus the Evaluation of Research Funding
Authors: Scriven Michael
Published: April 2023

The evaluation of research and researchers is an example of a fairly basic kind of evaluation. It normally involves either a ranking or a grading (aka., rating) of research projects or personnel for merit, worth, or significance, and these are tasks that we know a good deal about doing.
But the evaluation of research funding is another kind of animal altogether. It aims for an apportionment or allocation decision, which is either something essentially different from evaluation or, with a stretch, a highly complex kind of evaluation decision. It is certainly a decision that depends on more than one kind of basic evaluation, but it depends on them in a way that has never been reduced to a formula or computer program.
The Fundamental Characteristics of Research
Authors: Coryn Chris L. S.
Published: April 2023

One of the most serious problems encountered in evaluating research, or researchers, is the dispute about what is and is not research. That is, how is research defined? Or, can it be satisfactorily defined? Of course, there are numerous competing definitions of the term and little apparent consensus. However, common usage holds that research is a truth-seeking activity which contributes to knowledge, aimed at describing or explaining the world, conducted and governed by those with a high level of proficiency or expertise.
Converting Perspective to Practice
Authors: Scriven Michael
Published: April 2023

As a critic, not of RCTs but only of RCT imperialism, I am delighted to be able to agree with most of what Tom Cook says in his extremely valuable paper. But I also note here a few minor caveats and a few major disagreements, which I hope will lead to a constructive dialog and a better balance of research funding.
Using Hierarchical Linear Modeling for Proformative Evaluation: A Case Example
Authors: Coryn Chris L. S.
Published: April 2023

Proformative evaluation—first introduced in Scriven’s (2006) The great enigma: An evaluation design puzzle—“is motivated, like formative evaluation, by the intention to improve something that is still developing, but unlike formative, the improvement is only possible by taking action, hence proactive instead of reactive, hence both, hence proformative” (M. Scriven, personal communication, March 9, 2006). An exploratory study was conducted to proformatively assess the effects of a pilot intervention intended to reduce negative attitudes toward the poor using a non-probability, convenience sample of 202 (149 women, 53 men) predominately White college students from a midsized Midwestern university. The study was conducted so that the program, prior to full-scale implementation, could be refined and more effectively designed. Hierarchical linear modeling was used to fit a two-level baseline (unconditional means) and growth model. The baseline model regressed the within individuals (nested) measures of attitudes toward the poor on time. The growth model regressed attitudes toward the poor on level two individual characteristics of gender, receipt of public assistance, race, and just-world beliefs.
Review of Evaluation Journal of Australasia: Volume 6, No. 1
Authors: Martz Wes
Published: April 2023

Editors Rosalind Hurworth and Delwyn Goodrick include four refereed articles, two book reviews, and the keynote address from the AES 2005 International Conference in this issue of the Evaluation Journal of Australasia.
Why an Active Control Group Makes a Difference and What to Do About It
Authors: Datta Lois-ellin
Published: April 2023

The Randomized Control Trials (RCT) design and its quasi-experimental kissing cousin, the Comparison Group Trials (CGT), are golden to some and not even silver to others. At the center of the affection, at the vortex of the discomfort, are beliefs about what it takes to establish causality. These designs are considered primarily when the purpose of the evaluation is establishing whether there are outcomes associated with a program and, if so, how confidently the results can be attributed to the program. If one concludes these designs are superior to alternatives for establishing causality, and have no more bad habits than the alternatives, then the RCT and the CGT are the methods of choice.
Programme Evaluation Theory: The Next Step Toward a Synthesis of Logic Models and Organizational Theory
Authors: Fear William J.
Published: April 2023

This paper argues that the formulation of policy, at whatever level, to whatever scale or scope, is any different to the myriad of processes involved in strategic planning within and between organisations, and the attendant decision making processes that abound in such an environment (Hage, 1980; Hickson, 1987; Thompson, 1967; Weick, 1976). Those forces that impact upon organisations are precisely the same forces that impact upon policy making groups who are themselves ‘organisations’ (or institutions), whether actual or representative. This provides a startign point for considering a synthesis between various models or paradigms. The argument continues, that program theory and the program logic model can incorporate and synthesise theories from elsewhere that, in turn, usefully inform and develop programme theory and program logic. A ‘hierarchical’ model is presented to explore the linkages between the components and ‘simple’ processes of theory, programme theory, logic modelling, organisational functioning, policy, and the consequent impact. It is hoped that this model can be used as a framework to: (a) determine which components are essential to give the policy external validity, credibility, implementability, and so on; and (b) provide a common point of reference for policy makers, stakeholders, and evaluators.
The 'Holy Trinity' of Methodological Rigor: A Skeptical View
Authors: Coryn Chris L. S.
Published: April 2023

Rigor in research is normally conceived of as the means by which integrity and competence are confirmed (Tobin & Begley, 2004).1 That is, a way of demonstrating the legitimacy or soundness of the research process. Without rigor, it is argued, there is a danger that research may become fictional journalism and therefore worthless as contributing to knowledge (Morse, Barrett, Mayan, Olson, & Spiers, 2002).
Using Strong Evaluation Designs in Developing Countries: Experience and Challenges
Authors: Bamberger Michael , White Howard
Published: April 2023

In his article in the November 2006 JMDE, Thomas Cook reviewed the current debate on the role of randomized control trials (RCT). While his article focused on educational research, the issues raised, and the responses by Michael Scriven (2006) and Jane Davidson (2006), highlighted the broader issues currently being debated on the need for more ‘rigorous’ program evaluation in all sectors. The purpose of this article is to extend the discussion to the field of international development evaluation, reviewing the different approaches which can be adopted to rigorous evaluation methodology and their applicability in a development setting.
Evaluation in the Context of the Government Market Place: Implications for the Evaluation of Research
Authors: Della-Piana Connie K. , Della-Piana Gabriel M.
Published: April 2023

The evaluation community has concentrated on examining and explicating implications of the choice of methods for evaluating federal programs, as described in the New Directions for Evaluation edited by Julnes and Rog (2007), placing the policy debate in historical and contemporary contexts. In that volume and elsewhere we find that there are several mechanisms described for supporting and/or conducting program evaluation at the federal level. In the Julnes and Rog volume, Chelimsky (2007) describes evaluation activities conducted within the federal government by the Government Accountability Office (GAO). Both grants and contracts supported the work of Yin (Yin and Davis, 2007) in the evaluation of large comprehensive reforms in K-12 science and mathematics education. Other evaluation activities come under the authority of the Office of the Inspector General which conducts performance audits of government programs that draw on program evaluation and its methods (see the Yellow Book http://www.gao.gov/govaud/ybk01.htm).
Systems Concepts in Evaluation
Authors: Gullickson Amy M.
Published: April 2023

This anthology provided not only an eye opener, but also an excellent initiation to the variety of concepts inherent in systems.
An Association to Improve Evaluation of Development Aid
Authors: Clements Paul
Published: April 2023

This paper argues that development aid could be greatly improved if the management of development programs and projects could be governed by an effective orientation to cost-effectiveness. This in turn could substantially be achieved by changes the paper proposes to aid evaluation—through a particular kind of association—and a few lesser management reforms.
Youth Participatory Evaluation: Strategies for Engaging Young People
Authors: Cullen Anne
Published: April 2023

A review of the book Youth Participatory Evaluation: Strategies for Engaging Young People by Kim Sabo Flores, published in 2008 by Jossey-Bass.
Commentary on "An Association to Improve Evaluation of Development Aid"
Authors: Eggers Hellmut
Published: April 2023

This piece includes general and specific comments on “An Association to Improve Evaluation of Development Aid.”
Reply to Hellmut Eggers’ Comments on “An Association to Improve Evaluation of Development Aid”
Authors: Clements Paul
Published: April 2023

I would like to thank Hellmut Eggers for his thoughtful comments on my paper. I hope we can collaborate on the project we both hold dear, improving evaluation of development aid! I am particularly grateful for the opportunity to reply to his comments.
The Insight Evaluation Approach
Authors: Russon Craig , Russon Karen
Published: April 2023

Projects and programmes that are often the objects of our inquiry are actually no more than mental constructs. They are the result of numerous conditions that happen to come together for one brief moment. Their funding, activities, and results arise, dwell in a state of constant flux for a period of time, and then pass away. Insight Evaluation provides an approach to see projects and programmes clearly, without false impression.
The views expressed in this manuscript are those of the authors and do not necessarily reflect the policy of the International Labour Organization, the United Nations, or the Evaluation Capacity Development Group
Promoting Student Ownership of Learning Through High-Impact Formative Assessment Practices
Authors: Brookhart Susan M. , Moss Connie M. , Long Beverly A.
Published: April 2023

The most important instructional decisions, those with the greatest influence on student success, are made by learners themselves (Stiggins, 2008). Formative assessment, done well, contributes to student ownership of learning more than any other classroom-based instructional or assessment practice (Bloom, 1984). It is an economical, highly effective, and uniquely flexible method that can improve learning (Leahy, Lyon, Thompson, & Wiliam, 2005). Simply put, the teacher’s purpose in formative assessment is to give students the means, motive, and opportunity to take control of their own learning. And, through their involvement in formative assessment, students develop self-efficacy for specific learning and, more generally, they develop skills that contribute to increased self-regulation and self-assessment of learning. In order for students to be meaningfully involved in formative assessment, they must be guided by teachers who hold the beliefs, knowledge and skills that engender active student engagement in the learning process. This paper highlights interim findings from a five-year professional development initiative involving the Armstrong School District, a large, rural school district in Western Pennsylvania, and the Center for Advancing the Study of Teaching and Learning at the Duquesne University School of Education. The initiative rests on the fusion of formative assessment, teacher-student communication, and student ownership of learning. The professional development program employs online modules, peer study groups, classroom walkthroughs, and teacher inquiry into their classroom practices and the beliefs that drive them. The program explores seven formative assessment components: 1) Identifying and Clearly Communicating Learning Targets, .) Feedback that Feeds Forward, 3) Student Goal-Setting, 4) Student Self- Assessment, 6) Strategic Questioning, and 7) Formative Discourse. All components are linked to specific aspects of student motivation: intrinsic motivation, self-efficacy, self-regulation, goal-setting, and student attributions. The paper describes the impacts of formative assessment on student ownership of learning, student achievement, motivation, and active engagement as well as provides insights into teachers’ experiences with student involvement. Findings show that not only have the teachers come to value and promote student ownership of learning using high impact formative assessment strategies, but that their efforts have resulted in high student engagement in learning and increased student achievement.
How the I Ching or Book of Changes can Inform Western Notions of Theory of Change
Authors: Russon Craig , Russon Karen
Published: April 2023

Background: This article is the third part of a trilogy. The Eastern paradigm article (Russon, 2008) dealt with ontological issues (the nature of reality). The insight evaluation article (Russon & Russon, 2009) dealt with epistemological issues (how we can know reality). 
Purpose: This article deals with a methodological issue—specifically it explores how ancient Chinese philosophy might influence the way in which modern day evaluators think about theories of change.  
Setting: Not applicable. 
Subjects: Not applicable. 
Research Design: Not applicable. 
Data Collection and Analysis: The paper is the result of a desk review that compared and contrasted modern theory of change literature with the writings contained in the I Ching or Book of Changes.
Findings: The authors believe that the theories of change expressed in the systems diagrams of the I Ching would be considered to be conducive to recurrent change that occurs due to cycles in the natural world. They further believe that the change which harnesses the power of naturally occurring cycles offers the best hope for long-term sustainability. This is because such change takes advantage of an intuitive cooperation with the natural order.
Conclusions: The authors believe that using the diagrams of the I Ching as a theory of change template offers modern day evaluators a number of advantages. The advantage of greater sustainability has already mentioned. The authors believe that organic theories of change also encourage evaluators to think about the contribution from many factors instead of attribution of a few factors.
Moving to Genuine: Credible Cultural Competence and Stakeholder Believability
Authors: Evergreen Stephanie D. H. , Cullen Anne
Published: April 2023

Background: Cultural competency is becoming an increasingly important concept in evaluation. The developing discussions tend to revolve around how to become culturally competent, why it is important, and how to know when it is attained. More problematically, cultural competency seems bound by dimensions of race, even though culture represents a broader scope of characteristics.
Purpose: We review the current usage of cultural competence to point out its limitations and we suggest alternative ideas that can better facilitate communication about this essential topic. We look beyond the evaluation field to learn how cultural competence is handled in other disciplines. Particularly seeking to support communication between evaluation clients and evaluators, we offer strategies to engage in a dialogue about cultural issues.
Setting: Not applicable.
Intervention: Not applicable.
Research Design: Analytic essay.
Data Collection and Analysis: Not applicable.
Findings: Cultural competence, as a term, is inherently limiting in its connotations, implications, and ability to bring about change. Cultural humility may be a more appropriate term. Regardless of the semantics, the basic need to communicate about the topic remains. Asking for cultural competency, or humility, as we suggest, in Requests for Proposals may be one way to start the conversation. Positioning statements and focused interview questions also may serve to generate discussion between client and evaluator.
What Does it Take to do Evaluation in Communities and Cultural Contexts Other Than Our Own?
Authors: Wehipeihana Nan , Davidson E. Jane , McKegg Kate , Shanker Vidhya
Published: April 2023

Background: This paper builds on the growing body of evaluation literature around the importance of culture and cultural context in evaluation (e.g. Greene, 2005; Hood, Hopson and Frierson 2005; Hopson, 2009; Kirkhart, 1995 and 2005; La France, 2001).
Purpose: The place of language, culture, cultural context, and leadership roles in evaluation is explored through consideration of the question, “What does it take to do evaluation in communities and cultural contexts other than our own?” 
Setting: Not applicable.
Intervention: Not applicable.
Research Design:  Not applicable.
Findings: Attention to the location of power and privilege in evaluation, and to community engagement and ‘sense-making’ processes are the conversational starting points to begin to explore what it takes to do evaluation in communities, where the language, culture, and cultural context are different from one’s own.
The Use of Kōans for Insight Evaluation Approach Training
Authors: Russon Craig , Russon Karen
Published: April 2023

Background: Since publishing a trilogy of articles relating Eastern mysticism to evaluation, the authors have received many inquiries about training.   Purpose: The article explains one promising new method under development—adapting the Zen practice of kōans for evaluation training.   Setting: Not applicable.   Subjects: Not applicable.   Research Design: Not applicable. Data Collection and Analysis: Not applicable.   Findings: Not applicable.   Conclusions: The authors believe that kōans can be used as a poetic technology for recalibrating evaluation practioners’ attitudes and actions about evaluation.   Keywords: Insight evaluation approach, Eastern mysticism, kōans, training, Zen, enlightenment
Using Readability Tests to Improve the Accuracy of Evaluation Documents Intended for Low-Literate Participants
Authors: Kouamé Julien B.
Published: April 2023

Background: Readability tests are indicators that measure how easy a document can be read and understood. Simple, but very often ignored, readability statistics cannot only provide information about the level of difficulty of the readability of particular documents but also can increase an evaluator’s credibility.     Purpose: The purpose of this article is two-fold: (1) to provide readers with logical reasons for using readability tests and (2) how to choose the right test for a project. Setting: United States.   Research Design: A comparative framework is used to present the need for readability testing.   Keywords: Readability tests, evaluation instruments, survey research, low-literacy survey
The Calorie Count of Evaluation
Authors: Scriven Michael
Published: April 2023

We all know one kind of answer to the question of the value of evaluations; they can help to improve the evaluands, and/or establish accountability for the expenditure that created or supports or buys the evaluands, and/or increase our knowledge about the evaluands’ merit, worth, or significance, etc. Those answers are about the useful functions of evaluation; they’re like saying that food can help build bone structure and/or muscle mass and/or brain mass; these are all functions that food performs. But there’s a more fundamental and more specific level at which we say that a particular food has a certain calorie count, a certain fat content, a certain sugar content, has some iron or peanut oil or salt in it, etc. That level of analysis—a kind of nitty-gritty or component level—is very valuable for the nutritionist and for many consumer concerns. What is the equivalent of this kind of answer for evaluations? Is this a level of analysis we have been overlooking?
Demands on Users for Interpretation of Achievement Test Scores: Implications for the Evaluation Profession
Authors: Della-Piana Gabriel Mario , Gardner Michael
Published: April 2023

Background:  Professional standards for validity of achievement tests have long reflected a consensus that validity is the degree to which evidence and theory support interpretations of test scores entailed by the intended uses of tests.  Yet there are convincing lines of evidence that the standards are not adequately followed in practice, that standards alone are not sufficient guides to action, and that reviewers of tests do not call attention to important kinds of validity evidence that might support the demanding process of making sense of test scores or reasoning from test scores.
Purpose: The intent of this article is to make more transparent the demands of achievement test interpretation on users in instructional contexts and to open up a dialogue on implications for the evaluation profession for improvement of practice along lines already set out by evaluation theorists.
Setting:  Not applicable.
Intervention: Not applicable.
Research Design: Not applicable.
Data Collection and Analysis: Review of current practice.
Findings:  The article makes transparent the lack of attention to validation of achievement tests to support inferences relevant to intended uses in instruction and project evaluation. Elements of a model for the process of reasoning from test scores are articulated. The cognitive demands on the test score user are illustrated in achievement test contexts in writing, science, and mathematics. Implications are drawn for deliberation on issues and for the development of casebooks to guide practice.
Evaluating the Cost Effectiveness of Heifer International Country Programs
Authors: Clements Paul
Published: April 2023

Background: In the 1950s and ‘60s cost effectiveness was the primary focus of evaluation of international development aid, but in recent decades this has been increasingly neglected. The most recent decade saw great interest in impact evaluation, but the step from impacts to cost effectiveness was often neglected. This article explains how a multi-year evaluation of a major international NGO that was designed to estimate country program impacts was expanded to include estimates of cost effectiveness.    Purpose: The article explains the importance and illustrates the practicability of evaluation for cost effectiveness. It describes the evaluation’s methodology and presents its major finding, that Heifer International is highly cost effective in improving the income, assets and nutrition of poor households in Albania, Nepal and Uganda.   Setting: The article focuses mainly on the 2011 evaluation of the Heifer International country program in Uganda.    Intervention: The evaluations focused on usually three to five year projects of usually a few dozen to a few hundred households, but the evaluation itself did not conduct interventions (beyond its interviews). To add evaluation of cost effectiveness, the focus was changed from the project to the household level and evaluators estimated changes in income, assets and nutrition due to the project. Estimates of income impacts were then used as the primary basis for estimating the cost effectiveness of the respective country program (although assets and nutrition were also considered).   Data Collection and Analysis: Data were collected through two and a half to three week country program evaluations by two or three evaluators and their translators. Group and household interviews were based on questionnaires, but they also required evaluators to pursue lines of inquiry to logical conclusions. Quantitative and qualitative factors were considered as a basis for impact estimates on scales from zero to five in the original evaluation and in terms of economic values for income and assets. Analysis was carried out largely with Excel spreadsheets.   Findings: Due to Heifer International’s expenditure of about $7 million over six years, about 8,500 Ugandan families are likely to experience income gains exceeding $8.5 million a year on an ongoing basis and asset gains of about $17 million. About 5,500 of their children are likely to avoid stunting due to nutritional shortfalls. For each $1 expenditure by country programs in Albania, Nepal and Uganda, households can be expected to gain about $2.35, $1.19 and $1.25 in the respective country programs on an ongoing basis once the projects reach the maturity profile of those included in the evaluation.