bluuebunny/crossref_metadata_2025 huggingface dataset:
The Conjunction Fallacy: Confirmation or Relevance?
Authors: Chung WooJin , Dorst Kevin Matthew , Mandelkern Matthew , Mascarenhas Salvador
Published: April 2023

The conjunction fallacy is the well-documented reasoning error on which people rate a conjunction A&B as more probable than one of its conjuncts, A. Many explanations appeal to the fact that B has a high probability in the given scenarios, but Tentori et al. (2013) have challenged such approaches. They report experiments suggesting that degree of confirmation—rather than probability—is the central determinant of the conjunction fallacy. In this paper, we have two goals. First, we address a confound in Tentori et al.’s (2013) experiments: they failed to control for the fact that in their stimuli where B is confirmed, it is also conversationally relevant in the sense that it fits with the topic or question under discussion (Roberts, 2012). Conversely, when B has a high probability but is not confirmed, it is conversationally irrelevant. Consequently, it is possible that conversational relevance, rather than confirmation, is responsible for the differences they found between confirmed and probable hypotheses. Second, inspired by recent theoretical work, we aim to give the first empirical investigation of the hypothesis that this type of conversational relevance on its own—independently of degree of confirmation—can be an important factor in the conjunction fallacy. We report on two experiments that vary Tentori et al.’s (2013) design by making B relevant without changing its degree of probability or confirmation. We found that doing so increases the rate of the conjunction fallacy, suggesting that relevance plays an important role in the conjunction fallacy.
Parental education and children's cognitive development: A prospective approach
Authors: Klein Markus , Kühhirt Michael
Published: April 2023

This study uses nationally representative data from the 1970 British Cohort Study (BCS70) to estimate the impact of parental education on children’s cognitive development using a prospective approach. The analysis focuses on a subsample of participants and their first-born children (n = 1,042, aged 3 to 16). Previous research often overlooked selective patterns of family formation, potentially introducing endogenous selection bias, and did not fully account for genetic confounding in the association between parental education and children’s cognitive outcomes. To address these limitations, we used the BCS70’s multigenerational design and inverse probability of treatment and censoring weighting to correct for non-random selection into parenthood and control for confounding parental and grandparental factors. Our findings show that accounting for endogenous selection bias increases the estimated effect of parental education, while adjusting for confounding—primarily driven by parental cognitive ability—substantially reduces this effect. These results underscore the importance of disentangling the direct effects of parental education from the broader intergenerational factors that shape children’s cognitive outcomes.
Complete blood counts parameters and neonatal sepsis
Authors: K.C. Shiva Raj , Gyawali Purnima , Dhakal Ajaya Kumar , K C Geetika , Shrestha Devendra
Published: April 2023

Background: Neonatal sepsis is one of the common causes of neonatal mortality. Blood culture is the gold standard procedure to confirm sepsis. However, the rapid rate of disease progression makes its timely diagnosis and prompt management very crucial before the availability of culture. The study aimed to determine the significance of complete blood count parameters including platelet indices in diagnosing neonatal sepsis.
Materials and Methods: This study included 129 suspected cases of neonatal sepsis admitted to KIST Medical College Teaching Hospital from 15th April 2018 to 14th April 2019. Complete blood count, platelet indices, micro erythrocyte sedimentation rate, C-reactive protein, and blood culture were performed and analyzed.
Results: Of 129 suspected neonatal sepsis, abnormal leukocyte count was encountered in 31 (24.0%) patients. Culture-proven sepsis was seen in 25 (19.1%) patients. The sensitivity and specificity of Neonatal Sepsis Screening Criteria were 10.0% and 80.0% respectively.
Among platelet-derived indices, PDW (full form) had a sensitivity of 80% and specificity of 56% with a cut of 17.02 CV. MPV (full form), with a cut-off value of 9.78 fL had a sensitivity of 88 % with a specificity of 59%. Both MPV and PDW had low positive predictive values (33.3%) but a high negative predictive value (95.2%).
Conclusions: Along with total leukocyte count, platelet indices have shown their significance in determining neonatal sepsis with significant negative predictive value. The addition of MPV and PDW to the Neonatal Sepsis Screening Criteria might help the treating physician to rule in or rule out neonatal sepsis and manage it accordingly.
Study on Potential and Practices of Rooftop Rainwater Harvesting System in Oxford College of Engineering and Management
Authors: KC Manila , Bhandari Min Prasad , Timilsena Sambriddhi , Lamichhane Sangina , Adhikari Basanta Prasad , Poudel Ashish , Kafle Sarad Chandra
Published: April 2023

The objective of this study was to evaluate the possibility of promoting rooftop rainwater harvesting to address future water scarcity in the Oxford College of Engineering and Management Nawalparasi district. AutoCAD software drew accepted plans and maps; manual measurement was done with measuring tape since some parts were missing.

The results show that the linear forecast 1558.139, 1581.57mm,1548.63mm, 1598.14mm and 1570.64mm. The volume of water is 5135.54, 5028.16, 5188.94, 5099.68 respectively. It also shows the calculation of water demand which is 5,820,000 liters per year, and the water harvested is 5188946.58 liters per year. Finally, with this ratio, we realized we could fulfil 89.15% of water demand through water harvesting. The time series relevant results indicated that the A.R.I.M.A. model of forecasting rainfall was based on (0.1.0), signifying the R Square value is .631 and the p-value is insignificant (p > 250), which is better for this forecasting model.
Gotowość szkolna w perspektywie wczesnodziecięcych doświadczeń
Authors: Michalak Renata
Published: April 2023

School readiness includes the readiness of each individual child, the school’s readiness for children, and the ability of families and communities to support optimal early childhood development. Current knowledge about early childhood development, particularly in the field of neuropsychology, provides empirically verified evidence that the factors determining early childhood experiences can be modified, which may affect the trajectory of children’s development and learning, and thus their school readiness. Early identification of the needs and resources of the children and their families, as the primary development environment, can significantly reduce or eliminate the deficits in development and functioning with which many children enter formal education. Optimal support includes health, educational and preventive practices that promote the child’s school readiness in all of its areas, i.e. physical, socio-emotional, cognitive. The child and their family should be given special care in order to recognise irregularities as soon as possible and to intervene at an early stage, and to prevent problems and difficulties from developing and accumulating. The main aim of the article is to demonstrate the necessity of considering school readiness as a result of the support received by the child from their environment from the moment of their birth. It is assumed that their functioning has a decisive impact on the child’s cognitive, socioemotional and motor functioning, making them ready to undertake educational challenges to varying degrees. To focus solely on the child’s readiness to start formal education is to ignore the importance of their early development and to diminish the role of their experiences. This belief has been supported by a wealth of empirical evidence cited in this article.
The Trend Scenario Method in Cost-Benefit Analysis: A Model for the Evaluation of Occupational Safety and Health Regulations
Authors: Berljawsky Alex G.
Published: April 2023

Abstract: This article outlines the trend scenario method of analyzing costs and benefits in Canadian federal government occupational safety and health regulations. The trend scenario enables a comprehensive cost-benefit evaluation model involving past trends and parameter projections, which improve reliability of and control over the predicted results, as long as the assumptions and parameters are carefully documented and qualified. Inherent biases and limitations of this trend scenario method are also discussed.
Collectif pour un Québec sans pauvreté : un processus d’évaluation continue des « fruits » obtenus et des « chemins » parcourus
Authors: Paradis-Pelletier Marie-Anne , Pedneault Evelyne
Published: April 2023

Abstract: From the start of Projet AVEC, the Collectif pour un Québec sans pauvreté adopted the obligation to track the project’s development. Far from the traditional methods of evaluation, principles were established and constantly revisited to ensure a rigorous evaluation process appropriate for the project involving individuals in a situation of poverty. Measurement tools were developed with those principles in mind. One, called the “fruits” tool, was created to evaluate activities and another, the “route” tool, was designed to facilitate the evaluation of the project in relation to its objectives.
Achievement Assessment Programs in Reading and Writing: What Lessons Can Be Learned?
Authors: Simon Marielle
Published: April 2023

Abstract: Canada and its provinces and territories regularly subscribe to student achievement programs, primarily for social accountability purposes. In addition to administering their own provincial and territorial programs, they have participated for the past 30 years in the Third International Mathematics and Science Study (TIMSS), Progress in International Reading Literacy Study (PIRLS), Programme for International Student Assessment (PISA), and School Achievement Indicators Program (SAIP), recently replaced by the Pan-Canadian Assessment Program (PCAP). These programs assess the achievement of primary and secondary students in mathematics, science, reading, and writing. The PIRLS, PISA and PCAP all have a reading component, but only SAIP has a section on writing.
The Road to Evaluation Capacity Building: A Case Study from Israel
Authors: Rosenstein Barbara , Englert Pnina Elal
Published: April 2023

Abstract: We present an empirical case study of an Evaluation Capacity Building (ECB) initiative in a school in Israel. First, we tell the story of the school’s 10-year journey toward the successful integration of evaluation through ECB. Then we examine the case according to King’s (2002) four elements of ECB: teachers, students, the curriculum, and the context. We present four concrete examples of evaluations conducted at the school illustrating how these elements influenced the results of ECB. We follow this presentation with a discussion of the case in relation to other ECB checklists. After thorough and systematic investigation, we believe that through this case we can learn more about the factors that contribute to or inhibit the successful outcome of an ECB effort in schools.
Participatory Evaluation as Seen in a Vygotskian Framework
Authors: Higa Terry Ann F. , Brandon Paul R.
Published: April 2023

Abstract: In participatory evaluations of K–12 programs, evaluators develop school faculty’s and administrators’ evaluation capacity by training them to conduct evaluation tasks and providing consultation while the tasks are conducted. A strong case can be made that the capacity building in these evaluations can be examined using a Vygotskian approach. We conducted participatory evaluations at 9 Hawaii public schools and collected data on the extent to which various factors affected participating school personnel’s learning about program evaluation. After the evaluations were completed, a trained interviewer conducted standardized interviews eliciting the participating school personnel’s opinions about the methods and effects of the capacity building. Two reviewers used codes representing Vygotskian concepts to categorize the interview results. We present the results of the coding and provide conclusions about the value of using a Vygotskian framework to examine capacity building in participatory evaluations.
L’influence de l’évaluation sur les suites des projets d’expérimentation : l’exemple des projets d’informatisation au Québec
Authors: Fortin Jean-Paul , Lamothe Lise , Gagnon Marie-Pierre
Published: April 2023

Abstract: The dissemination and sustainability of many health-care IT projects in Quebec have hardly been influenced by their evaluation, even when the evaluation was participative, pluralist, comprehensive, formative, and user-focussed. This article aims to improve our understanding of why that has been the case and to suggest courses of action and reflection. Human and organizational factors and other factors related to the evaluation itself are considered. To increase the impact of evaluation on project outcomes, the authors recommend knowing the project better, adding to the action plan usually intended to promote knowledge and learning, and further developing frameworks for integrating practices and theories to improve our understanding of the conditions for their use in action.
La figure de l’évaluateur, mi-ingénieur mi-bricoleur
Authors: Jabot Françoise , Bauchet Murielle
Published: April 2023

Abstract: This article describes an evaluation experience in health promotion. Caught in a wide range of representations, challenges, and expectations, the evaluator adapts her practice to provide a quality evaluation to address the users’ needs. The authors analyse the evaluation process with a focus on the French Evaluation Society’s principles and suggest adjustments specific to the field of health promotion. The evaluator’s ability to take on several roles and to adapt to the context is fundamental here.
Je me souviens … de t’avoir trop longtemps cherché
Authors: Dumaine François
Published: April 2023

Abstract: In its simplest expression, the purpose of program evaluation is to explore the logic that links a set of related activities to their expected results. Hence, program evaluation serves to determine the extent to which the logic of a program has unfolded as expected. A prerequisite to this assessment is to have a program (evaluand) that is adequately defined. This article explores the potential impacts of having an ill-defined or incomplete definition of a program that is about to be evaluated. It discusses some strategies that evaluators can explore to avoid what may well turn out to be a fatal flaw in what is already a rather challenging research process.
The Paris Declaration Evaluation Process and Methods
Authors: Betts Julia , Wood Bernard
Published: April 2023

Abstract: The Paris Declaration Evaluation faced the challenge of assessing the effects of a policy compact between nations, an area in which evaluation thinking and practice is still emergent. It also confronted the challenge of attempting to link development results to the implementation of a broad reform agenda in countries of widely differing circumstances. This article describes how the Core Evaluation Team, in consultation with the management and governance structures of the Evaluation and the evaluation teams involved, addressed these challenges. It describes the development of the evaluation questions, the methodological framework, and how the methodology worked in practice. Highlights include ensuring the use of evidence through evaluation teams; the analysis and synthesis processes; and processes for ensuring independence, integrity, quality, and ethics. Limitations and risks encountered are discussed. Finally, the article sets out lessons learned for complex transnational studies from the Core Evaluation Team perspective.
A Professional Grounding and History of the Development and Formal Use of Evaluator Competencies
Authors: Wilcox Yuanjing , King Jean A.
Published: April 2023

Abstract: This article provides background for the special issue. The authors first review the history of competency development in general and then in evaluation specifically. To date, the Canadian Evaluation Society and the Japanese Evaluation Society are the only professional organizations that have launched credentialing systems. However, increasing numbers of evaluation organizations and associations worldwide have developed lists of evaluator competencies, moving the field one step closer to professionalization. Although there are many viewpoints on the value of developing sets of evaluator competencies, there is currently a lack of empirical studies linking them to useful or sound evaluations.
Evaluator Competencies: The Canadian Experience
Authors: Buchanan Heather , Kuji-Shikatani Keiko
Published: April 2023

Abstract: This article examines the development and adoption of competencies in Canada, created as a key foundation of the Credentialed Evaluator designation under the auspices of the Canadian Evaluation Society (CES). Following a brief description of the Canadian evaluation context and issues that led to the competencies’ development, this article reviews the development process. The approved Competencies for Canadian Evaluation Practice are presented with concluding comments on the future of competencies in Canada.
Evaluator Competencies: The Aotearoa New Zealand Experience
Authors: Wehipeihana Nan , Bailey Robyn , Davidson E. Jane , McKegg Kate
Published: April 2023

Abstract: Why would you place valuing, culture, and cultural values at the centre of an evaluation competencies framework? What would it look like if you did? This new set of evaluator competencies from the Aotearoa New Zealand Evaluation Association (ANZEA) is distinct from other competency and credentialing frameworks developed around the world. Why do we believe this approach is key to promoting high quality, culturally sound, responsive, and ethical evaluation practice in Aotearoa New Zealand? This article outlines the process for developing the competencies as well as the reasoning and powerful exchange of ideas underpinning their development and subsequently infused in the competencies.
Evaluability Assessment of a Community-Based Program
Authors: Meeres Sandra L. , Fisher Ron , Gerrard Nikki
Published: April 2023

Abstract: Evaluability assessments have increased the usefulness and meaningfulness of evaluation studies through in-depth program analyses to determine which program elements are amenable to further evaluations and which are not. This is especially important for community-based programs, whose structure, activities, and goals are often too broadly defined to allow one to accurately measure their effects. An example of such a program is the Rural Quality of Life (RQL) program designed to address the human consequences of the present rural crisis in Saskatchewan. The evaluability assessment found that the program’s structure and description of its activities and goals required additional clarification before further evaluation research could be considered. The recommendations were directed toward further program development and planning for future evaluations. Overall, the process of the evaluability assessment was helpful for both the program staff and evaluators involved in determining the present nature and functioning of the RQL program in a practical way.
Use of a Stakeholder Advisory Group to Facilitate the Utilization of Evaluation Results
Authors: Shea Michael P. , Lewko John H.
Published: April 2023

Abstract: This article describes the steps taken by the authors to establish a stakeholder advisory group for a major evaluation project. The authors argue that use of the advisory group enhanced utilization of the evaluation project’s findings.
Evaluating an Indian and Métis Education Staff Development Program
Authors: Jones Richard M. , Mazurkie Evelyn A.
Published: April 2023

Abstract: The purpose of this study was to provide formative information to the Indian and Métis Education Branch of Saskatchewan Education, Training and Employment on its Staff Development Program. The evaluation was designed in cooperation with the Indian and Métis Education Branch and with input from the provincial Indian and Métis Education Advisory Committee. Data for the study were collected by means of questionnaires, telephone surveys, group interviews, on-site visits, and document analyses. The evaluation provided information on the extent to which the program has met its objectives.
Political Context and Program Evaluation: The Inextricable Link
Authors: Knox Colin
Published: April 2023

Abstract: Program evaluations have assumed a much greater importance as pressure on public expenditure increases. This article eschews evaluation designs that are politically vacuous by drawing attention to the importance of contextual variables in a case study of a community relations program based in Northern Ireland. The research illustrates the dilemma now faced by evaluators in the United Kingdom. The demands of short-term, context-stripped, value-for-money evaluations are often in conflict with context-bound qualitative enquiries that can uncover the unique mix of social, political, and cultural influences on program delivery. The dominance of macro-political variables are illustrated as one example of a unique mix within this program.
Setting Achievement Standards/Expectations for Large-Scale Student Assessments
Authors: Jones Richard M. , Hunter Darryl M.
Published: April 2023

Abstract: This article discusses a variety of issues associated with establishing formal standards or expectations for student achievement on large-scale assessment programs. The article reviews the educational literature on the subject of standard-setting methods, describes two approaches to standard setting that have been used in province-wide assessments in Saskatchewan, and evaluates the effectiveness of these approaches. Although the two procedures described were generally successful, numerous issues are raised and recommendations are made to support future standard-setting activity and research.
The Changing Role of the Evaluator in the Process of Organizational Learning
Authors: Levin-Rozalis Miri , Rosenstein Barbara
Published: April 2023

Abstract: In this article we examine the role of the evaluator in the process of organizational learning, and discuss the conditions necessary to facilitate the productive execution of such a role and the consequent ramifications for evaluation. First, we describe the process of organizational learning as presented in the literature of organizational learning. Second, we examine the demands that process presents to evaluators. Third, we discuss organizational learning within the context of participatory evaluation, and then explore the role of the external learning agent. Finally, we present some major changes in the role of the evaluator, changes that stem from the very nature of the organizational learning process. The focus on organizational learning transforms the role of the evaluator to one of knowledgeable facilitator who returns responsibility of the operation, development, and evaluation back to the project/program or organization. We conclude by acknowledging the difficulties involved in changing the traditional role of the evaluator, particularly in giving up control of the evaluation to the stakeholders and letting the organization become the “owner” of the evaluation process and knowledge, leaving the evaluator the important role of facilitator. The evaluator is responsible for the procedures of learning — providing tools and monitoring the learning that goes on. The learning content is the responsibility of the organization and not of the evaluator. While we do not preclude the traditional role of the evaluator, we do suggest a significant change in the procedures involved in evaluation, in the skills required to conduct effective evaluations within the organizational context, and in the ownership of the knowledge that emerges from such evaluation.
Pragmatism or Transformation? Participatory Evaluation of a Humanitarian Education Project in Sierra Leone
Authors: Maclure Richard
Published: April 2023

Abstract: Although participatory evaluation (PE) is now widely acknowledged as a potentially useful way to assess international development assistance programs, there is little documented evidence of participatory approaches being incorporated into evaluations of humanitarian aid for populations living in emergency situations. Two principal streams, practical participatory evaluation (P-PE) and transformative participatory evaluation (T-PE), are pertinent to international aid programs. Yet because these two approaches to PE are rooted in different goals and procedures, it is unclear how pragmatic and transformative dynamics can be integral to participatory evaluations of humanitarian aid. The example of an evaluation of a humanitarian education project for displaced children in wartorn Sierra Leone reveals the practical benefits that can accrue from even limited stakeholder participation in the inquiry process. In addition, while the evaluation of rapid education was not a transformative intervention, it nonetheless generated insights into the challenges of fostering incremental social transformation in a post-war context.
Developing a Tool to Measure Knowledge Exchange Outcomes
Authors: Skinner Kelly
Published: April 2023

Abstract: This article describes the process of developing measures to assess knowledge exchange outcomes using the dissemination of a best practices in type 2 diabetes document as a specific example. A best practices model consists of knowledge synthesis, knowledge exchange (dissemination/adoption), and evaluation stages. Best practices are required at each stage. An extensive literature review found no previous knowledge syntheses of concrete tools and models for evaluating dissemination or exchange strategies. This project developed a practical and usable tool to measure the reach and uptake of disseminated innovations. The instrument itself facilitates an opportunity for knowledge exchange to occur between producers and adopters. At this point the tool has a strong theoretical basis. Initial pilot-testing has begun; however, the accumulation of evidence of validity and reliability is only in the planning stages. The instrument described here can be adapted to other areas of population health and evaluation research.
Navigating Uncharted Waters: Project Monitoring at Cida
Authors: Maneepong Chuthatip , Stiles J. Mark
Published: April 2023

Abstract: The Canadian International Development Agency (CIDA) has employed external monitors for many years to assist in measuring the performance of its projects. At first, this role was one of surveillance, with monitors expected to keep a distance from the implementing organizations. Today, in keeping with international trends in monitoring and evaluation, the monitoring role is, in theory, more participatory and improvement-oriented, requiring of monitors a different set of knowledge, skills, and attitudes. The role is nevertheless poorly defined, open to individual interpretation and made even more challenging when the monitoring involves measuring performance in relation to cross-cutting themes such as gender equality. This article presents many of the challenges inherent in monitoring and makes a case for a participatory approach aimed at learning and at program improvement. The authors call upon the evaluation community to undertake research and scholarly discourse in this area to guide successful practice.
What an Eight-Year-Old Can Teach Us About Logic Modelling and Mainstreaming
Authors: Renger Ralph , Page Melissa , Renger Jessica
Published: April 2023

Abstract: This article presents a short case narrative, the purpose of which is to illustrate that complex evaluation methodologies such as logic modelling can be simplified to the point where a child can be guided through the process quickly. However, the case narrative also serves to highlight the potential consequences to program development and evaluation activities when the process is oversimplified. Like a double-edged sword, simplifying the process encourages more organizations to use a logic model to develop and evaluate programs, but, in hindsight, the simplicity may lead to program architectures that have little opportunity of demonstrating success or to evaluations that may be off the mark.
Evaluating Inter-, Multi-, and Transdisciplinary Research in the European Research Area
Authors: Schmidt Evanthia Kalpazidou , Siune Karen
Published: April 2023

Abstract: Europe faces challenges in creating an integrated European Research Area. This article focuses on the use of research evaluations in a framework where knowledge production processes are changing and increasingly take place within inter-, multi-, and transdisciplinary settings. European research policies and instruments such as the Framework Programmes for Research and Technological Development and the recently established European Research Council are discussed in view of new knowledge production processes and evaluation practices. The article concludes with an explanation of the need to rethink evaluation approaches in the European Research Area.
The BC Cancer Agency Primary Prevention Program: Using Negotiated Accountability to Evaluate a Community-Determined Health Initiative
Authors: Chomik Treena , Baillie Lynne , McLeod Heather , Broughton Sandra
Published: April 2023

Abstract: The British Columbia Cancer Agency Centre for the Southern Interior Prevention Program operated from 1997 to 2005, resulting in community-determined cancer prevention initiatives specific to unique local needs. These initiatives included community events and forums, school food and nutrition activities, physical exercise initiatives, environment and air quality programs, early childhood education, tobacco counselling programs, and cancer prevention-related publications. The external evaluation of the program used the concept of negotiated accountability to strengthen collaborative evaluative practices. In this article, we present evaluation findings and provide a detailed example from one partner community.
The Lay of the Land: Evaluation Practice in Canada in 2009
Authors: Gauthier Benoît , Barrington Gail V. , Bozzo Sandra L. , Chaytor Kaireen , Dignard Alice , Lahey Robert , Malatest Robert , McDavid James C. , Mason Greg , Mayne John , Porteous Nancy L. , Roy Simon
Published: April 2023

Abstract: A group of 12 evaluation practitioners and observers takes stock of the state of program evaluation in Canada. Each contributor provides a personal viewpoint, based on his or her own experience in the field. The selection of contributors constitutes a purposive sample aimed at providing depth of view and a variety of perspectives. Each presentation highlights one strength of program evaluation practiced in Canada, one weakness, one threat, and one opportunity. It is concluded that Canadian evaluation has matured in many ways since 2003 (when a first panel scan was conducted): professional designation is a reality; the infrastructure is stronger than ever; organizations are more focused on results. Still, evaluation is weakened by lacunas in advanced education and professional development, limited resources, lack of independence, rigidity in evaluation approaches, and lack of self-assessment. While the demand for evaluation and evaluators appears on the rise, the supply of evaluators and the financial resources to conduct evaluations are not. The collective definition of the field of evaluation still lacks clarity. There is also reassurance in looking toward the future. With increased appetite for evaluation, evaluators could make a real difference, especially if evaluators adopt a more systemic view of program action to offer a global understanding of organizational effectiveness. The implementation of a Certified Evaluator designation by CES is a major opportunity to position evaluation as a more credible discipline.
Obstacles à l’implantation d’un programme de boire contrôlé dans le contexte organisationnel des services en dépendance
Authors: Simoneau Hélène , Cournoyer Louis-George , Favrod-Coune Thierry , Landry Michel , Tremblay Joël , Patenaude Catherine
Published: April 2023

Abstract: Implementation is a primary step in disseminating innovation because, without successful implementation, even the most effective program cannot show results. Many barriers were reported in the implementation evaluation of Alcochoix+, a Quebec program for managing alcohol consumption intended for non-dependent excessive drinkers. In addition to barriers already identified in the literature, such as lack of advertising, staff turnover, and financial resource problems, the results of this study point out a more specific barrier: a lack of understanding of the program’s underlying approach. Moreover, the relative importance attributed to the barriers seems to change over time. These results illustrate the importance of conducting implementation studies for new programs.
Addressing the Challenges Encountered During a Developmental Evaluation: Implications for Evaluation Practice
Authors: Poth Cheryl-Anne , Pinto Dorothy , Howery Kathy
Published: April 2023

Abstract: This article describes three challenges encountered during a developmental evaluation and explains how these were addressed from the evaluators’ perspective. The evaluation was conducted to support the implementation of a three-year educational technology leadership project funded by the Alberta provincial government. The developmental evaluation responded to two purposes identified by the evaluation client: inform ongoing programmatic decisions and measure change in practice. The implications for evaluation practice related to the challenges of introducing a new evaluation approach, defining the boundaries of evaluator roles, and integrating technological resources are discussed and related to the Canadian Evaluator Competencies (Canadian Evaluation Society, 2010) and the Program Evaluation Standards (Yarborough, Shulha, Hopson, & Carruthers, 2011).
Evaluation-Capacity Building: The Three Sides of the Coin
Authors: Smits Pernelle
Published: April 2023

Abstract: I share my experience as a Canadian evaluator who is starting out in the field and new to the process of evaluation capacity building. I draw on three metaphors to describe what I believe are abilities of an effective evaluator. The first is an ability to be like playdough—to mould to external requirements. The second is an ability to be like a spider—to build webs or networks based on an understanding of the global context of the intervention. The third is an ability to be like Buddha—to cultivate a Zen-like attitude during stormy times. I illustrate each metaphor with some of my own evaluation experiences. I also discuss the possible disadvantages of each of these abilities.
Evaluation of Research and Development Accomplishments: Northern Forestry Centre
Authors: Moore W. , Newstead R.
Published: April 2023

Abstract: A retrospective, or hindsight, model was used to evaluate impacts and benefits of completed research and development (R&D) at the Northern Forestry Centre (NoFC) over a recent 10-year period. The evaluation included: (1) an inventory of R&D accomplishments in use with estimates of benefits; (2) client (forestry users) validation; (3) management rating of 62 accomplishments; (4) time and cost assessments; and, (5) quantified benefit compilations for a sample of six accomplishments. A conservative three-to-one benefit-to-cost ratio, over the entire NoFC R&D program costs (including services), for the 10-year period has been identified for the sample alone. The most successful R&D was client-driven, with emphasis on applied R&D and technology transfer.
Une stratégie de triangulation pour l’évaluation des effets d’un programme de soutien aux nouveaux parents
Authors: Ouellet Francine , Durand Danièle , Massé Raymond
Published: April 2023

Abstract: Combined use of qualitative and quantitative approaches to program evaluation is a strategy which is attracting some interest in evaluation circles. However, its applications are few and its rules, related to the choice of devices and data analysis, are still poorly defined. In this article, the authors present an application of a triangulation strategy evaluating the effects of a child abuse and neglect prevention program. Using objectivation and generalisation techniques, the qualitative data then become comparable to the quantitative data. This compari son makes it possible to better interpret the results, raise new hypotheses and identify some unanticipated results.
Évaluation des effets d’un système d’information clinique sur l’activite médico-hospitalière
Authors: Sicotte Claude , Pineault Raynald , Tilquin Charles , Contandriopoulos André-Pierre
Published: April 2023

Abstract: We have analyzed the impact of a clinical information system designed to give feedback on profiles of medical practice in two hospitals. The system was planned to use group influence to induce change among deviant physicians. The information system failed to modify the physician’s practice profiles. A large part of this failure is linked to the small variation recorded in utilization patterns among physicians. Medical practice appears to be quite homogeneous in a setting such as a hospital where the clinical behavior of physicians is highly visible to colleagues.
Assessing Quality in R&D Laboratories: Bibliometrics, Peer Review and Organizational Surveys
Authors: Miller Roger , Lortie Pierre
Published: April 2023

Abstract: A review of R&D evaluation outlines six approaches: strategic audits, constituency reviews, peer reviews, cost-benefit analyses, organizational assessments, and bibliometric indicators. To study the quality of research outputs, the authors present a comparative research plan combining peer reviews, bibliometric indicators, and organizational assessments. They provide results of peer reviews, bibliometric analyses, and organizational assessments. Using data collected from 53 laboratories around the world, they advance a taxonomy of laboratories as an additional guide to the selection of appropriate criteria for R&D performance evaluation. A clustering technique identifies six unique types or profiles of laboratories: regulation/control, project management, science frontier, service, devices, and survey.
Preparing School Evaluators: Hiroshima Pilot Test of the Japan Evaluation Society’s Accreditation Project
Authors: Nagao Masafumi , Kuji-Shikatani Keiko , Love Arnold J.
Published: April 2023

Abstract: This article reports on the efforts of the Japan Evaluation Society (JES), in collaboration with the Canadian Evaluation Society, to develop and pilot test an accreditation and certification scheme for school evaluators. The purpose of the JES accreditation model is to support evaluation capacity building and promote high quality evaluation by developing functional evaluation competencies. The article describes the theory and practice of the JES approach to evaluation training and accreditation, including its overall rationale, the influence of Japan’s socio-political context, the content of the school evaluator training program, and the findings of the initial “test of concept” pilot test in Hiroshima. Based on a six-month follow-up evaluation, the article also provides an assessment of the acceptance, early results, and potential sustainability of the evaluator training program. These findings have encouraged the JES to establish the accreditation scheme for school evaluation, followed by a similar system for the evaluation of international development assistance programs and government policy evaluation. The development of the JES accreditation scheme should be of interest to other evaluation societies and also to public/nonprofit organizations that must use brief training courses or evaluation “toolkits” for building evaluation competencies quickly among staff.
Suggestions d’améliorations d’un cadre conceptuel de l’évaluation participative
Authors: Ridde Valéry
Published: April 2023

Abstract: Due to the diversity of practices in participatory evaluation, it is not always easy to teach or practice this evaluation approach. This article aims to provide a preliminary attempt to validate the Weaver and Cousins (2005) framework in order to characterize three forms of participatory evaluation (stakeholder-based, practical, empowerment). The application of this framework to three case studies resulted in two recommendations to enhance the framework validity (detailing the values of the scale and reflecting on the relevance of stakeholder-based evaluation) and one research proposal recommendation (verifying the impact of empowerment evaluation).
Using Public Databases to Study Relative Program Impact
Authors: Azzam Tarek , Christie Christina
Published: April 2023

Abstract: Evaluators are under increasing pressure to answer the “compared to what” question when examining the impact of the programs they study. Program contexts and other restraints often make it impossible to study impact using some of our more rigorous methods such as randomized control trials. Alternative methods for studying impact under extreme contextual constraints should be explored and shared for use by others. This article presents a method for studying program impact using existing public datasets as a means for deriving comparison groups and assessing relative impact.
Using Technology to Enhance Aboriginal Evaluations
Authors: Johnston Andrea L. K.
Published: April 2023

Abstract: With a focus on the use of technology when evaluating programs for Aboriginal people, this article explores the possibility of using visual and oral computer technology to enhance the incorporation of Aboriginal worldviews in program evaluation. The author situates Aboriginal worldviews, including methods of communication and transmission of knowledge, within a unique evaluation framework that also considers Western methods of data collection. Examples of the author’s framework are offered in the context of evaluations of Aboriginal programs. Based on her experiences, the author concludes that it is possible to join the traditional knowledge of Aboriginal people with digital technology in program evaluation.
Drawing on Indigenous Ways of Knowing: Reflections from a Community Evaluator
Authors: Scott Sheryl
Published: April 2023

Abstract: The clash between Western and Indigenous ways of knowing has been epitomized by the “parachuting model” of the Western researcher who drops onto the reservation, collects data, and leaves, never to be heard from again. The strengths of indigenous science, for example, observation and contextual factors, are either ignored or appropriated. These past (and sometimes present) wrongs committed by academic researchers continue to be a contentious issue in Native communities, where, despite the research dollars flowing into the community to “solve” health problems, disparities between Native health status and that of the general population persist. This article shares reflections from a community-based evaluator who, along with a Lakota health educator, served as “cultural translators” in a community participatory process led by a community agency. We recognized the need to work with/in two cultures—both the academic research world and the Native community—and drew on collaborative evaluation principles and indigenous ways of knowing to conduct formative evaluation research on smoking cessation issues for pregnant Native women.
Bureaucratic Competence as an Essential Factor in Cross-Cultural/Multicultural Program Evaluations
Authors: Wilson H. T.
Published: April 2023

Abstract: While we may all agree in principle that both implementers and evaluators should be culturally sensitive and ethical as well as instrumentally effective in their work practices, we often ignore the extent to which these practice goals may conflict with one another in achieving bureaucratic competence, particularly in a multicultural society. Reconciling them requires us to acknowledge the indispensable role of responsible program evaluation in this effort, one that addresses: both the employees and the recipients of programs; the need for evaluators to be open to both theoretical and operational contributions to the field; the signal role of bureaucratic, as well as electoral, modes of representation; the indispensable function of affirmative action and pay equity programs in reconciling diversity and fairness; and the principle that subjects in evaluation and implementation processes should play a more significant role than the passive status assigned them by traditional bureaucracy and applied social science.
Coûts de transaction et politiques agri-environnementales
Authors: Vollet Dominique , Callois Jean-Marc , Aznar Olivier , Yelkouni Martin
Published: April 2023

Abstract: This article proposes an analytical framework and method for measuring transaction costs associated with the application of French agri-environmental policies. Transaction cost theory attempts to take into consideration the effect of costs involved in drawing up and securing contracts for economic organization and efficiency. Originally developed to understand forms of industrial organization, it has proven relevant in the assessment of public policies, in particular territorial and environmental policies involving a large number of players, often with diverging interests, which raises important issues of information asymmetry. The proposed method is applied to two highly different initiatives, one fairly standard («grass premium») and the other localized («local farming contract»). The diverse forms transaction costs may take for the various players involved (government departments, professional organizations, farmers) are shown, as is the usefulness of analysing these costs for an accurate evaluation of the effectiveness of the policies.
The Downward Trend of Survey Response Rates: Implications and Considerations for Evaluators
Authors: Bladon Teresa L.
Published: April 2023

Abstract: Rapidly declining response rates and the associated threat of nonresponse bias call into question the validity of data obtained through telephone surveys, a tool often used in evaluation. This article explores changes in nonresponse bias over time by examining three data points (1991, 1996, and 2002) from an annual household telephone survey conducted by the University of Alberta’s Population Research Lab. Results demonstrate a substantial decline in response rates accompanied by an increasing level of bias in variables related to respondent education. Implications of these results are investigated through regression analyses and suggest that declining representation of individuals with less education could significantly impact a variety of survey variables, thus creating opportunity for opinions of the more educated to become more heavily weighted in evaluation results. In turn, such results could be used to inform government policies and programs in ways that advantage the educated middle class.
Neutral Assessment of the National Research Council Canada Evaluation Function
Authors: Fraser Melissa A. , Tremblay Ghislaine H. , Bourgeois Isabelle , Lahey Robert
Published: April 2023

Abstract: Federal government departments and agencies are required to conduct a neutral assessment of their evaluation function once every five years under the Treasury Board Secretariat’s Policy on Evaluation (2009). This article describes the National Research Council’s experience conducting the first neutral assessment of its evaluation function. Based on learning from this first assessment, best practices that NRC intends on replicating, as well as lessons learned for future assessments, are discussed. This article may be of interest to both federal and non-federal organizations seeking to conduct a neutral assessment in an effort to improve their evaluation services and products.