bluuebunny/crossref_metadata_2025 huggingface dataset:
Insights into Evaluation Capacity Building: Motivations, Strategies, Outcomes, and Lessons Learned
Authors: Preskill Hallie , Boyle Shanelle
Published: April 2023

Abstract: Evaluation capacity building (ECB) is a topic of great interest to many organizations as they face increasing demands for accountability and evidence-based practices. While many evaluators are engaged in evaluation capacity building activities and processes with a wide variety of organizations, we still know very little about whose capacity is being built, what strategies are being used, and the overall effectiveness of these efforts. To explore these issues, a research study was conducted with 15 organizations that have been involved in ECB efforts during the last few years. The findings reported in this article are part of a larger study, and represent interviews with 25 evaluators and 13 clients (n = 38), who have facilitated and supported an organization’s ECB effort. We specifically focus on the participants’ motivations for engaging in ECB, the teaching and learning strategies used to facilitate capacity building, their perceived outcomes of this effort, and their lessons learned.
Understanding Organization Capacity for Evaluation: Synthesis and Integration
Authors: Bourgeois Isabelle , Chouinard Jill Anne , Cousins J. Bradley
Published: April 2023

Abstract: The special issue is devoted to the examination of organizational capacity for evaluation and evaluation capacity building (ECB) through empirical inquiry. The compilation consists of two quantitative surveys of evaluators and seven single or multiple case studies across a broad array of organizations in a diverse contexts (e.g., east-central Ontario, California, Hawaii, Minnesota, and Israel). In this final article, the authors look across the collection of studies to identify emerging themes and trends with implications for ECB. The emergent themes are defining ECB; conceptualizing ECB outcomes; organizational context; ECB implementation issues; and enabling factors and barriers to organizational evaluation capacity development.
Common Measures to Advance the Science and Practice of Population Intervention for Chronic Disease Prevention: The Promise and Two Early Experiences in Tobacco Control
Authors: Riley Barbara L. , Cameron Roy , Campbell H. Sharon , Manske Steve R. , Lamers-Bellio Kim , Czukar Donna
Published: April 2023

Abstract: Pertinent evidence to inform population interventions for chronic disease prevention is sparse. The use of common measures across multiple jurisdictions is a promising approach to study “natural experiments” that can dually advance research/knowledge development and evaluation/practice improvement for population intervention. Early experiences with provincial tobacco control strategies and North American quitlines reveal the importance of (a) sustained collaboration across research, evaluation, policy, and practice communities; (b) honouring different perspectives; and (c) stable institutional support for the creation and implementation of common measures. The promise of common measures will be better understood as mature examples of their use are explored.
Evaluating School-Based Tobacco Control Programs and Policies: An Opportunity Gained and Many Opportunities Lost
Authors: Leatherdale Scott T.
Published: April 2023

Abstract: In the domain of school-based tobacco control, there is a large gap between the research being performed and the type of research needed to inform school-based prevention activities. The Tobacco Module of the School Health Action Planning and Evaluation System is a data collection and feedback system designed to support school-based tobacco control planning and evaluation. Despite numerous opportunities to evaluate interventions and natural experiments that have been occurring within schools, school boards, provinces, and even nationally within the last few years, a lack of systematic data collection means that the impact of these initiatives has not been robustly evaluated.
Can Tobacco Control Advocacy Work be Evaluated?
Authors: Minian Nadia , Katz Danielle , Sante Emily Di , Pathammavong Ratsamy
Published: April 2023

Abstract: Tobacco control advocates have played an important role in Ontario’s tobacco control efforts. As with other advocacy work, their efforts have been mostly unexamined. Based on Ontario’s tobacco control advocacy work, this article highlights the challenges of evaluating the work of advocates (e.g., having multiple aims, multilayered structures, shifting time frames, measuring long-term goals, dealing with external factors, having goals change at midstream, having different approaches working simultaneously) and summarizes some theories and tools (e.g., Bellwether Methodology, Agenda Setting Theory, Context-in approaches, Pathways of Influence, and logic models) that have been used to conduct effective advocacy evaluations.
Advocacy Evaluation Theory as a Tool for Strategic Conversation: A 25-Year Review of Tobacco Control Advocacy at the Canadian Cancer Society
Authors: Montague Steve , Lamers-Bellio Kim
Published: April 2023

Abstract: Advocacy evaluation has been considered difficult, even in specific and uncomplicated situations. This article describes a short evaluative review of a series of tobacco control policy advocacy activities involving multiple parties over 25 years. Despite its obvious limitations, the effort suggests that the retrospective development of a structured contribution analysis and “key event” stories from documentary and key informant information can generate compelling circumstantial evidence for impact. Perhaps more importantly, the study was found to provide what appear to be valuable insights and lessons for the planning and management of future advocacy initiatives.
Evaluation, Valuation, Negotiation: Some Reflections Towards a Culture of Evaluation
Authors: Rodríguez-Bilella Pablo , Monterde-Díaz Rafael
Published: April 2023

Abstract: Although the evaluation of public policies is a subject of growing interest in Latin America, there are problems with the design and implementation of evaluations, as well as with the limited use of their results. In many cases, the evaluations have more to do with generating descriptions and less with assessing these activities and using those assessments to improve planning and decision making. These points are explored in a case study of the evaluation of a rural development program in Argentina, emphasizing the process of negotiation and consensus building between the evaluators and the official in charge of approving the evaluation report. The lessons learned from the experience point to the generation and consolidation of a culture of evaluation in the region.
Management Challenges in M&E: Thoughts from South Africa
Authors: Naidoo Indran A.
Published: April 2023

Abstract: This impressionistic narrative focuses on experiences in setting up South Africa’s first government monitoring and evaluation (M&E) unit. The high political expectations for the program required that M&E provide a progress overview and diagnosis of the ambitious land reform program. Key contextual factors that influence success and that may pose challenges are considered in this account, which shows both how strategies had to be changed to address varied demands and the mixed results these produced. Based on the practical experiences, a set of suggestions are provided to emerging evaluators as they sharpen their evaluation acumen.
Learning from Evaluation Misadventures: The Importance of Good Communication
Authors: Perrin Burt
Published: April 2023

Abstract: Good communication is a key factor affecting the success of evaluation in many ways. This article presents four examples from a variety of government and international organizations illustrating communication-gone-wrong, and often evaluation management-gone-wrong. The article discusses what can be learned from these situations in order to provide for more effective communication in other evaluation contexts, and thereby enhancing the relevance, quality, and use of evaluation. This article highlights the implications of these experiences for effective project management of overall evaluation functions as well as of specific evaluations.
L’approche Realist à l’épreuve du réel de l’évaluation des programmes
Authors: Ridde Valéry , Robert Emilie , Guichard Anne , Blaise Pierre , Olmen Josefien Van
Published: April 2023

Abstract: The Realist approach suggests a way to grasp the complexity of social interventions, through questioning what works, for whom, and under what circumstances. We present here some challenges and discussion on the use of this methodological innovation, based on an analysis of our own experience and an examination of the scientific literature. Although the Realist approach seems intellectually attractive, building its “Context-Mechanism-Outcome” configurations proves problematic. Finding the necessary information about context and outcome, differentiating contextrelated elements from those related to mechanisms, and adopting an accurate level of analysis are among the challenges in using the Realist approach.
Logic Analysis: Testing Program Theory to Better Evaluate Complex Interventions
Authors: Rey Lynda , Brousselle Astrid , Dedobbeleer Nicole
Published: April 2023

Abstract: Evaluating complex interventions requires an understanding of the program’s logic of action. Logic analysis, a specific type of program theory evaluation based on scientific knowledge, can help identify either the critical conditions for achieving desired outcomes or alternative interventions for that purpose. In this article, we outline the principles of logic analysis and its roots. We then illustrate its use with an actual evaluation case. Finally, we discuss the advantages of conducting logic analysis prior to other types of evaluations. This article will provide evaluators with both theoretical and practical information to help them in conceptualizing their evaluations.
Trois conceptions de la nature des programmes : implications pour l’évaluation de programmes complexes en santé publique
Authors: Potvin Louise , Bilodeau Angèle , Gendron Sylvie
Published: April 2023

Abstract: This article discusses three perspectives on the nature of programs to clarify the purposes of an evaluation. An empirical realist program design views programs as real objects that allow dysfunctional and problematic objects to be repaired. In that view, the purpose of the evaluation is to determine the efficacy of programs to solve problems. An idealist design views both programs and the problems they address as representations, which leads to evaluation intended to reconcile the players’ representations. In a critical realist design both programs and problems are events resulting from causal mechanisms activated by the program players. From that perspective, the purpose of the evaluation is to support innovation and program transformation.
Meta-evaluation: Evaluating the Evaluation of the Paris Declaration
Authors: Patton Michael Quinn
Published: April 2023

Abstract: It has become a standard in major high-stakes evaluations to commission an independent review to determine whether the evaluation meets generally accepted standards of quality. This is called a meta-evaluation. Given the historic importance of the Evaluation of the Paris Declaration, the Management Group commissioned a meta-evaluation of the evaluation. The meta-evaluation concluded that the findings, conclusions, and recommendations presented in the Paris Declaration Evaluation adhered closely and rigorously to the evaluation evidence collected and synthesized. The meta-evaluation included an assessment of the evaluation’s strengths, weaknesses, and lessons. This article describes how the meta-evaluation was designed and implemented, the data collected, and the conclusions reached.
Lessons Learned and the Contributions of the Paris Declaration Evaluation to Evaluation Theory and Practice
Authors: Dabelstein Niels , Patton Michael Quinn
Published: April 2023

Abstract: The final event of the Paris Declaration Evaluation was a lessons-learned workshop. This article first highlights the lessons about joint evaluations identified by participants in that workshop with the resulting report being a model of how to bring closure to a major evaluation. The article then presents 10 contributions of the Paris Declaration Evaluation to evaluation theory and practice. The article closes by recognizing that the Paris Declaration Evaluation received the 2012 American Evaluation Association (AEA) Outstanding Evaluation Award, which noted, “The success of the Evaluation required an unusually skilled, knowledgeable and committed evaluation team; a visionary, well-organized, and well-connected Secretariat to manage the logistics, international stakeholder meetings, and financial accounts; and a highly competent and respected Management Group to provide oversight and ensure the Evaluation’s independence and integrity. This was an extraordinary partnership where all involved understood their roles and carried out their responsibilities fully and effectively.”
An Idiographic Evaluation of an Integrated, Team Case Management Program: Hospitalization, Client Problems and Stages of Change
Authors: Dalrymple Andrew J. , Perron Denise M.
Published: April 2023

Abstract: The authors describe the evaluation of an integrated case management and addictions treatment program for individuals who have the dual challenge of coping with both mental illness and substance abuse problems—the dually diagnosed. This program, modeled after the New Hampshire Specialized Services Program, combined assertive case management and a four-stage addictions treatment model in which individuals are conceived as moving through the processes of engagement, persuasion, active group treatment, and relapse prevention. The method consisted of a quasi-experimental A-B design, with an idiographic analysis of patterns of hospitalization across small groups of individuals. Replication of the effect across these groups was accomplished as a natural byproduct of continuous enrollment into the program. Two new measurement devices were developed to assess functioning and to determine the level of change accomplished by the participants: the Client Problems Checklist and the Stages of Change rating scale respectively. Reliability data on each device indicate moderate to good levels of reliability. The results indicate that during its first year the program was successful in reducing hospitalization, increasing commitment to substance abuse treatment, and improving functioning in several life areas. Program participants expressed high levels of satisfaction. Further component analysis research is needed to better define the effects and appropriate targets for treatment.
L’évaluation au sein des régions françaises et de la Commission Européenne: son rôle dans le cadre du partage des responsabilités
Authors: Monnier Eric
Published: April 2023

Abstract: With the example of the French government before them, European and regional administrations are establishing policy evaluation procedures, sometimes jointly and sometimes competitively. The interest these administrations have in evaluation is related to their relative youth. They are facing a double necessity: developing a formula for action appropriate to the new public entity whose interests they have to protect, and defending their own point of view vis-à-vis the machinery of the national administration, which remains powerful and takes a dim view of the emergence of competing definitions of the public interest. There are frequent negotiations between the three levels of authority in the context of arrangements referred to as partnership policies, which are increasing. Few projects undertaken by the regions and the European Community are not jointly financed by the French government. The contractual approach to defining common goals and resources is often accompanied by an obligation to conduct an evaluation.
Evaluating a Three-Day Exercise to Obtain Convergence of Opinion
Authors: Uhl Norman , Wentzel Carolyn
Published: April 2023

Abstract: A department of the Government of Canada wished to identify the most serious difficulties associated with partnerships between the department and other agencies and to develop ways of improving future partnerships. To accomplish this task, a workshop was organized for a group of senior staff from the department who were familiar with these partnerships. This study reports the process employed in the very limited time available and evaluates its effectiveness.
Impact de l’entente cartierchamplain sur certains indicateurs relatifs à la sécurité routière
Authors: Armanville François
Published: April 2023

Abstract: In this article, the impact of a Cartier-Champlain police service agreement with a federal agency and the Sûreté du Québec is evaluated. The purpose of this agreement was to improve road safety by increasing police patrols in the area involved. In order to quantify the impact of the agreement, an evaluation model with control groups and a chronological series was used. This model involves taking a series of monthly measures in the area covered by the agreement and comparing them to two control areas. The results of the evaluation lead us to question the positive impact of the agreement and whether external factors could have been responsible for the reduction in accidents.
Application of Nominal Group Technique to Evaluate a Community Health Status Report
Authors: Lee-Han Hyewon , Dwyer John J. M. , Johnson Ian L.
Published: April 2023

Abstract: In 1992, the City of North York Public Health Department released the Community Health Status Report and Social Profile Report on the residents in North York, Ontario, Canada. These reports are a resource that is used by departmental staff to assess local needs, plan and evaluate programs, and assist in departmental strategic planning. This paper describes the application of the nominal group technique (NGT) among staff in a public health department to identify, discuss, and prioritize ideas to improve the reports.
Un état des lieux théoriques de l’évaluation: une discipline à la remorque d’une révolution scientifique qui n’en finit pas
Authors: Dubois Nathalie , Marceau Richard
Published: April 2023

Abstract: This article examines the present state of the theoretical bases of program evaluation by applying key analytical criteria proposed by Kuhn and by Lakatos for the study of scientific disciplines. The study identifies the main theoretical and empirical components of the discipline, clarifies their interactions, and discusses their influence on the development of the discipline. The application of the analytical model leads to the acknowledgement that evaluation is currently at the third stage of development of the disciplines, scientific revolution, which assumes an upcoming change in the discipline’s dominant foundations. However, the structuring of knowledge, the relationship between theory and empirical research, and the epistemological schism prevailing in the social sciences contribute to keeping evaluation in a never-ending state of scientific revolution.
Exploration of the Validity and Usefulness of an Integrated Performance Indicator for Postgraduate Scholarship Programs
Authors: Bourgeois Isabelle
Published: April 2023

Abstract: Performance indicators used to evaluate postgraduate scholarship programs in the natural sciences and engineering are usually long-term in nature and focus on such measures as time to completion, employment status, employment sector, and annual income. These measures are time-consuming and costly to collect and analyze. The information provided by these measures, therefore, should truly be useful to both program staff and senior managers responsible for administering these programs. Based on an earlier study conducted by two Canadian researchers, this article presents the results of a study on the validity of an integrated indicator for postgraduate scholarship programs. The findings of the study reveal that the integrated indicator offers no added value compared to traditional survey analysis. The article concludes with some suggestions for future performance measurement studies.
Impacts du pirs en milieu scolaire
Authors: Simon Marielle , Turcotte Catherine , Forgette-Giroux Renée
Published: April 2023

Abstract: This article examines the impact of the School Achievement Indicators Program (SAIP) on the educational system. SAIP is a Canada-wide program involving the large-scale assessment of student achievement in mathematics, science, reading, and writing. Data were collected using semi-structured interviews conducted with the SAIP’s jurisdictional coordinators (n = 20) and from surveys sent to participating school boards across Canada (n = 147). SAIP’s impact is described using two dimensions: intrinsic versus extrinsic and positive versus negative. Results revealed that poor communication between the coordinators and various educational stakeholders plays an important role in the level of impact. Recommendations are made for learning evaluation programs.
Value-for-Money Analysis of Active Labour Market Programs
Authors: Mason Greg , Tereraho Maximilien
Published: April 2023

Abstract: Accountability requirements by central agencies in government have imposed expectations on management to show results for resources used — in other words, “value for money.” While demonstrating value for money means showing that the program has relevance and a rationale and that the program logic and theory make sense, the core of value for money lies in showing that a program is cost-effective. Unfortunately, many public programs and policies do not provide quantifiable outcomes, and this limits conclusions on value for money. However, labour market training programs are amenable to cost-effectiveness analysis (CEA), provided that the evaluation methodology meets certain conditions. This article reviews CEA in the context of labour market training, especially programs directed to economically disadvantaged groups. After reviewing the data availability and the analytical methods commonly used to support value-for-money analysis of training programs, the authors present several practice improvements that would increase the “value” and validity of value-for-money analysis.
Youth Voices: Evaluation of Participatory Action Research
Authors: Chen Sandra , Poland Blake , Skinner Harvey A.
Published: April 2023

Abstract: When conducted with sensitivity and reflexivity, participatory action research (PAR) can be an empowering process that is particularly relevant for engaging young people in reflection and dialogue for social change. As the theory and practice of PAR evolve, researchers have evaluated the experiences of community participants, using both qualitative and quantitative approaches. However, only a limited number of evaluations have focused on PAR processes undertaken with youth, and few published papers have reported on involving youth in the evaluation. This article addresses the process of enabling youth to participate to their fullest ability in an evaluation of a PAR project called Youth Voices. The analysis draws on feedback questionnaires from community evaluators, minutes and notes from team meetings, and the researchers’ experiences and observations. The authors reflect on lessons learned that can be helpful to others considering participatory evaluation research with youth. The study revealed limitations in employing participatory evaluation with at-risk youth, including challenges posed by their psychosocial development and maintaining participants’ engagement throughout the processes of participatory evaluation. These lessons shed light on key tensions in using participatory evaluation and challenge the implicit assumption that a higher level of participation is necessarily better when working with youth. A central question is posed: What level of participation is optimal to ensure authentic community decision-making in a PAR project without overwhelming youth participants?
L’utilisation de l’évaluation fondée sur la théorie du programme comme stratégie d’application des connaissances issues de la recherche
Authors: Chagnon François , Daigle Marc , Gervais Marie-Joëlle , Houle Janie , Béguet Véronique
Published: April 2023

Abstract: Research on knowledge application and program theory-driven evaluation are two types of activity associated with research which involve close collaboration between researchers-evaluators and stakeholders and also facilitate maximal use of research results. In this article, first we succinctly present theoretical models of knowledge application emphasizing the conceptual bases and strategic elements common to both approaches. Specifically we examine the crucial role of tacit knowledge and distinguish between the individual and organizational levels in the utilization of knowledge generated by research and evaluation. We examine and discuss the advantages of program theory-driven evaluation as a knowledge application strategy. We suggest that conceptual developments in the field of knowledge application research can also contribute to the advancement of program evaluation.
Reconnecting Knowledge Utilization and Evaluation Utilization Domains of Inquiry
Authors: Amo Courtney , Cousins J. Bradley
Published: April 2023

Abstract: This article provides commentary for the thematic segment titled “Applying a variety of methods to the evaluation of various efforts aimed at transferring knowledge generated from research.” The authors revisit arguments supporting inquiry that takes up the challenge of connecting cognate fields of evaluation utilization and the broader domain of knowledge utilization. The central contribution of each of the foregoing articles is identified and situated within the context of ongoing inquiry in this domain.
Professional Identity of Evaluators in Israel
Authors: Levin-Rozalis Miri , Shochot-Reich Efrat
Published: April 2023

Abstract: Professional identity is a precondition for the establishment of a profession. This examines the professional identity of program evaluators in Israel. The field of evaluation in Israel has developed differently than in most Western countries—from the ground and with minimal governmental interference—and thus it is an interesting case study. In spite of the diversity of the backgrounds of evaluators, there is strong agreement among them on the core of evaluation as an interdisciplinary profession whose aim is mainly as an advisory tool that serves for learning. They also strongly agree that the borders and essence of evaluation are not clear to evaluators, evaluees, and the public. While half of the respondents practicing evaluation do not identify themselves as evaluators, a professional community is important to them. Evaluators in Israel are not well connected to professional activities and developments outside of the country. They do not participate in international conferences and do not publish in scientific journals, yet they are very active in professional activities in Israel. The context of Israeli society is analyzed for a better understanding of these findings.
Evaluating the Early Implementation of a Community Crisis Bed Program for People Experiencing a Psychiatric Emergency Using Key Component Scaling
Authors: Sylvestre John , Sundar Purnima , Jamshidi Parastoo , Manion Matthew
Published: April 2023

Abstract: This article describes an evaluation of the early implementation of a community crisis bed program for people experiencing a psychiatric emergency. A Key Component Scaling approach was used (a) to provide detailed and specific data on the implementation of particular program components, (b) to facilitate a comparison of perceptions of program implementation across program partners, and (c) to increase program understanding by these various partners. Three stakeholder groups (mobile crisis team staff, bed provider staff, and community partners) provided quantitative and qualitative data that helped to identify well-implemented program components, challenges in implementation, and suggestions for improvements.
Research on Evaluation: A Needs Assessment
Authors: Szanyi Michael , Azzam Tarek , Galen Matthew
Published: April 2023

Abstract: This survey study attempts to understand the research questions that evaluators were most interested in answering. The findings suggested that there is a great deal of interest in research efforts that (a) explore factors that increase the impact of evaluation, (b) help develop new methodologies, (c) examine the influence of context on evaluations, and (d) help to address ethical dilemmas. Respondents also provided research questions for each topic, revealing a diverse body of concerns and issues. The study also indicated that research on evaluation is viewed as an important endeavour with strong support from the community.
Propositions théoriques et pratiques pour l’évaluation de programmes en négligence
Authors: Bérubé Annie , Coutu Sylvain , Dubeau Diane , Lafantaisie Vicky , Devault Annie
Published: April 2023

Abstract: There is as yet no clear consensus about the effectiveness of interventions targetting child neglect. This article presents proposals for what to consider when evaluating those programs. First of all, child neglect is more an ecosystemic than a parental problem. An evaluation should therefore focus on changes within and outside the family. The individualized, multimodal nature of the programs makes an implementation fidelity evaluation necessary. Further, the evaluation should examine the factors that support the implementation, sustainability, and distribution of the program. Finally, an evaluation must be based on the outcomes perceived by a variety of players to provide a complete picture of the direct and indirect impact of the intervention.
The Social Discount Rate
Authors: Watson Kenneth
Published: April 2023

Abstract: Recently the discount rate recommended by Treasury Board Canada for use in program evaluations and project assessments has been challenged. This article reviews the theory and evidence supporting various estimates of the Canadian discount rate, and includes a comparison of rates used by the U.S. government and the World Bank. The article was written with the support of Training and Development Canada.
The Use of Holistic versus Analytic Scoring for Large-Scale Assessment of Writing
Authors: Hunter Darryl M. , Jones Richard M. , Randhawa Bikkar S.
Published: April 2023

Abstract: This article discusses a variety of issues associated with the use of holistic and analytic scoring methods in large-scale student assessments. The article reviews the education literature on the subject, describes the results of a recent Saskatchewan study comparing the results of the two scoring approaches, and discusses some of the study’s implications for future large-scale assessments.
A Theory-Driven Approach to Evaluating a Communication Intervention
Authors: McGilton Katherine S. , Fox Mary T. , Sidani Souraya
Published: April 2023

Abstract: Evaluating interventions in the practice setting calls for an alternative approach to the traditional randomized controlled trial (RCT), as its feasibility and generalizability, and clinical utility of its results are being questioned. The theory-driven approach (TDA) to evaluation, as an alternative to the RCT, attempts to account for the realities of practice. The TDA specifies the causal processes underlying the intervention effects, and identifies its expected outcomes as well as factors that affect treatment processes, such as patient, intervener, and setting characteristics. In this article, the TDA to intervention evaluation is presented as a means of designing and conducting evaluation. The TDA is discussed at the conceptual level and illustrated with examples from a pilot study that examined the effectiveness of a communication enhancement intervention designed to improve the communication skills of nursing staff in a complex continuing care (CCC) facility.
Réflexions autour d’une typologie des dispositifs institutionnels d’évaluation
Authors: Jacob Steve
Published: April 2023

Abstract: Recent developments in research on the institutionalization of policy evaluation offer new perspectives for understanding evaluation capacity development. This theoretical article focusses on mechanisms that contribute to ensuring the development and the stability of evaluative practice and suggests a typology for institutional evaluation methods. By combining reflections borrowed from the comparative approach in the social sciences and new institutionalism theories, it opens new avenues for research in this field of analysis. It also identifies the major characteristics of an evaluation method, focusing on two dimensions (openness and results) to outline an ideal-typical model.
Evaluator Competencies in University-Based Evaluation Training Programs
Authors: Stevahn Laurie , King Jean A. , Ghere Gail , Minnema Jane
Published: April 2023

Abstract: In this article we revisit the comprehensive taxonomy of Essential Competencies for Program Evaluators and explore its utility in university-based evaluation training programs. We begin by briefly summarizing the development of the taxonomy, then elaborate on how it can be used to enhance evaluation training through six decision points in graduate degree or certificate programs. We then discuss the challenges of credentialing/licensure and accreditation within the field of program evaluation and end with a proposal for the development of standards for program evaluation training programs.
Certification, Credentialing, Licensure, Competencies, and the Like: Issues Confronting the Field of Evaluation
Authors: Altschuld James W.
Published: April 2023

Abstract: After a brief review of recent history framing the debate about certification, the discussion begins by defining key terms (certification, credentialing, licensure, accreditation, and professional development). It then briefly summarizes the three main articles on the topic in this journal issue, analyzing their content in terms of similarities and differences. Lastly, implications are provided for the future of certifying or credentialing evaluators.
Simulating or Imputing Non-participant Intervention Durations Using a Flexible Semi-Parametric Model
Authors: Cahill Ian , Folkes Paula , Szabo Les
Published: April 2023

Abstract: In the evaluation of labour market training programs using matching, evaluators must decide when to start comparing participant outcomes against non-participant outcomes. Measurement relative to an intervention period permits the separation of training opportunity costs from possible benefits, but an equivalent period for the comparison group must be determined. One method imputes the timing of the intervention for comparisons from that of the participant match. However, with Propensity Score Matching, this may produce biased outcome estimates. Instead, the authors develop and apply semi-parametric duration models using Human Resources and Social Development Canada data to simulate the positioning and duration of the intervention for non-participants.
Performance Studies: The Missing Link?
Authors: Mayne John
Published: April 2023

Abstract: Typically, a good measurement strategy to support results-based management includes both ongoing performance measures and periodic evaluations. It is argued that this is a too limited set of measurement tools, resulting, not infrequently, in less useful and costly ongoing performance measures. It is proposed that, in addition to ongoing performance measures and periodic evaluations, an alternative measurement tool called a performance study should be used in many situations, and further, that in a number of circumstances, performance studies should replace specific ongoing performance measures.
Methodological and Conceptual Challenges in Studying Evaluation Process Use
Authors: Taut Sandy
Published: April 2023

Abstract: This article discusses methodological and conceptual challenges in empirically studying process use. The main difficulty lies in disentangling cause (here the evaluation process) and effects (here indicators of process use). The evaluation researcher not only needs to take into account all relevant factors regarding the evaluator, the participants, the evaluation context, and the evaluation approach and implementation, but also needs to base the research on a valid operationalization of process use. The article was inspired by the author’s experiences in conducting an exploratory study of process use in the context of two expert-facilitated self-evaluation projects involving five program staff. Before larger-scale studies can establish more generalizable knowledge on process use, evaluation researchers should engage in high-quality, real-time, in-depth qualitative studies to better understand the complex interactions at play and to help build a solid operationalization of this relevant construct.
Collaborative Evaluation in a Community Change Initiative: Dilemmas of Control Over Technical Decision Making
Authors: Brisson Daniel
Published: April 2023

Abstract: Collaborative evaluation brings community stakeholders and evaluators together with the opportunity to provide significant meaning to their work. In a community change initiative, collaborative evaluation also affords the opportunity to build community capacity. This article provides a framework for how collaborative evaluation can help achieve the goals of a community change initiative. The manuscript then explores one process dimension of collaborative evaluation—control over technical decision making—and explicates five dilemmas in its navigation. A case study is provided to illustrate the five dilemmas and to offer strategies for overcoming the pitfalls involved in the process of control over technical decision making. The article concludes with two primary lessons learned for evaluators trying to navigate the control over technical decision-making dilemmas in collaborative evaluation.
Increasing Research Skills in Rural Health Boards: An Evaluation of a Training Program from Western Newfoundland
Authors: Mathews Maria , Lynch Anne
Published: April 2023

Abstract: Rural health boards face barriers to increasing evaluation skills, such as fewer opportunities for continuing education and limited access to training resources. In this project, we set out to develop and evaluate a research skills training program suitable for a rural health board. Participants attended five one-day workshops in their home region and completed a research project with ongoing mentoring from their instructor. Post-workshop surveys found that the workshops were highly rated (mean: 4.5 out of 5). We surveyed participants before and after the training. We did not find an increase in participants’ knowledge scores but found significant increases in self-rated ability to carry out research tasks. Qualitative data suggest that, while most staff were familiar with research terminology, application-focused training was beneficial.
Reframing Evaluation: Defining an Indigenous Evaluation Framework
Authors: LaFrance Joan , Nichols Richard
Published: April 2023

Abstract: The American Indian Higher Education Consortium (AIHEC), comprising 34 American Indian tribally controlled colleges and universities, has undertaken a comprehensive effort to develop an “Indigenous Framework for Evaluation” that synthesizes Indigenous ways of knowing and Western evaluation practice. To ground the framework, AIHEC engaged in an extensive consultation process including conducting a number of focus groups in major regions of the United States. Cultural experts, Indian educators, and evaluators shared their concerns regarding evaluation and described how evaluation fits within a cultural framework. This article summarizes the focus group discussions and describes how the framework developed using the key principles of Indigenous ways of knowing and four core values common to tribal communities.
Moments of Truth: An Unexplored Dimension to Communicate Effectiveness
Authors: Ramírez Ricardo
Published: April 2023

Abstract: The settings for this article are rural and remote communities in the province of Ontario, Canada, where the advent of high-speed Internet has brought about new opportunities for the provision of public health and information services. This article proposes that public funding agencies and service providers will gain planning and evaluation insight from the notion of “moments of truth” as an additional dimension to capture and communicate program effectiveness. This is an evaluation dimension that is seldom appreciated as valid in the public sector, and yet it is at the heart of private sector behaviour.
The Essential Skills Series in Evaluation: Assessing the Validity of the ESS Participant Workshop Evaluation Questionnaire
Authors: Dagenais Christian , Dargis-Damphousse Luc , Dutil Julie
Published: April 2023

Abstract: Since 2003, the Essential Skills Series training program developed by the Canadian Evaluation Society has been offered to more than 15 groups in the province of Québec. The evaluations of these workshops were based on the participants’ reactions collected by a Participant Feedback Questionnaire. This article describes the process used to assess the structure of the questionnaire and document its psychometric properties in order to determine whether it covered all subjects addressed by the training program and the extent to which it measured the items it was intended to measure. The results suggest that the questionnaire is effective in measuring participant responses to all the relevant components of the training program. This procedure may interest professional evaluators who want to ensure that questionnaires used to evaluate training programs are suitable for their intended purpose.
The Art of the Nudge: Five Practices for Developmental Evaluators
Authors: Langlois Marc , Blanchet-Cohen Natasha , Beer Tanya
Published: April 2023

Abstract: This article focuses on developmental evaluation, based on an action research study involving a group of developmental evaluators in a three-year comprehensive community initiative on youth and community change. The study presents five practices found central to the art of the nudge: (a) practicing servant leadership, (b) sensing program energy, (c) supporting common spaces, (d) untying knots iteratively, and (e) paying attention to structure. These practices can help developmental evaluators detect and support opportunities for learning and adaptation leading to right-timed feedback.
The Key Functions of Collaborative Logic Modelling: Insights from the British Columbia Early Childhood Dental Programs
Authors: Poon Brenda , Leung Jannie Wing-sea , Louie Amber , Dios Celina Vergel de
Published: April 2023

Abstract: As part of a government partnership to evaluate British Columbia Early Childhood Dental Programs, the development of a provincial logic model provided an effective tool to integrate regional variations of dental public health programming and also foster collaborative processes with program stakeholders. Specifically, logic modelling provided a program documentation tool, a validity feedback loop, a means to collaborate across different levels of organization, and a forum for cross-health authority decision-making. This article highlights our experiences using logic models to enhance and provide structure to a collaborative approach in evaluation.
Informed Consent of 16- to 18-Year-Old Participants in Evaluations
Authors: Scott Susan
Published: April 2023

Abstract: Ethics policies require parental consent for “children” less than 18 years old. This article examines whether parental consent should be required for youth aged 16 to 18 years. It examines the current position of youth vis-à-vis services, informed-consent requirements, the quality of parental consent, and youths’ legal and developmental capacity to consent. It concludes that youth have the capacity to consent. Recommendations are made to revise the parental-consent policy and to address the impacts of stress, emotion, and inexperience on youth’s decision making at all evaluation process.
Obstetric Violence and Vulnerability: A Bioethical Approach
Authors: Berzon Corinne , Shabot Sara Cohen
Published: April 2023

At healthcare facilities worldwide, women during childbirth undergo medical procedures they haven’t consented to and experience mistreatment and disrespect. This phenomenon is recognized as obstetric violence (OV), a distinct form of gender violence. The resulting trauma carries both immediate and long-term implications, making it vital to address for promoting women’s health. OV is partly shaped by a narrow, paternalistic conception of vulnerability. A flawed conception of the vulnerability of pregnant women and fetuses has opened the door to medical control and coercion during childbirth. In this paper we examine what role notions of vulnerability play in perpetuating OV and consider recent attitudinal shifts in research ethics as a model for addressing it.
Creating the Conditions for Trust Around PrEP as HIV Prevention: The Relationships of MSM with Sexual and Romantic Partners and Healthcare Providers
Authors: Montess Michael
Published: April 2023

In this paper, I consider how trust affects the decisions of men who have sex with men (MSM) around using pre-exposure prophylaxis (PrEP) as HIV prevention in their sexual and romantic relationships, and how the use of PrEP affects their relationships with healthcare providers. MSM have to trust their sexual and romantic partners as well as their healthcare providers for PrEP to be successful as a relatively new HIV prevention strategy. This trust includes both interpersonal trust and institutional trust and it is complicated by different kinds of relationship dynamics and the history of prejudice against MSM in healthcare institutions.
Presumed Consent for Pelvic Exams Under Anesthesia Is Medical Sexual Assault
Authors: Tillman Stephanie
Published: April 2023

Unconsented pelvic exams under anesthesia are assaults cloaked in defense of healthcare education. Preemptive linguistic qualifiers “presumed” or “implied” attempt to justify such violations with flippancy toward their oxymoronic implications: to suggest a priori that consent can be assumed undermines its otherwise standalone social, ethical, and medico-legal reverence. In this paper I conceptualize “medical sexual assault” and argue that presumed consent for intimate exams exemplifies its definition. By bluntly describing pelvic exams as “penetration,” this work aims to reify the intimate reality of the clinical label “pelvic exam” and to call attention to cisheteronormative and androcentric assumptions involved in its practice. Additionally, this scholarship seeks to create a foundation toward broader work in conceptualizing clinical rape culture. Given recent national survey data indicating a surprising frequency of unconsented intimate exams, detailed language as to their problematics is necessary for ongoing legal and ethical efforts. Explicit consent for intimate exams must be the standard of care for conscious and unconscious patients.
الرياضة وحقل العلاقات الدولية: التفكير في الرياضة عبر عدسات النظريات الوضعية
Authors: حسين أحمد قاسم
Published: April 2023

​نسعى في هذه الدراسة لتسليط الضوء على تأثير الظاهرة الرياضية في السياسة الدولية وكيف يمكن التفكير فيها عبر عدسات المقاربات الوضعية في حقل العلاقات الدولية، لا سيّما أنّ الرياضة في عالم اليوم لم تعد مجرّد هواية أو ترفيه، بل صارت ظاهرة معولمة جرى إضفاء الطابع المؤسسي العالمي عليها، ما يستدعي التوقّف مليًّا أمام تأثيرها في حركة التنظير داخل حقل العلاقات الدولية مستقبلًا، لما تحمله في طيّاتها من تأثير في جموع الجماهير في مختلف أرجاء العالم الذي لم يعد، كما بشّرت العولمة، قريةً صغيرة، بل صار اليوم أشبه ببرجٍ سكني كبير يقيم فيه سكّان المعمورة نتيجة ثورة تكنولوجيات الاتصال والإعلام (ICTs) الجديدة، وما يرتبط بها من الانتشار الآخذ في الاتّساع لاستخدام وسائل التواصل "الاجتماعي".