bluuebunny/crossref_metadata_2025_split huggingface dataset:
Cultivating Communities of Practice to Build Organisational Capacity: A Case Study of the Philippines-Australia Basic Education Assistance for Mindanao (BEAM) Project
Authors: Cawrse Scott , Walsh Ian D'arcy 
Published: January 2018

This article examines how the latent potential of communities of practice (CoPs) can be harnessed as a strategic resource for building capacity and improving organisational performance. In particular it focuses on a case study of developing communities of practice in the Philippines Department of Education (DepEd) in three Regions (XI, XII and the Autonomous Region of Muslim Mindanao (ARMM)), which have been supported under the Philippines-Australia Basic Education Assistance for Mindanao (BEAM) Project. This article describes community-building activities sponsored by BEAM to strengthen local Department of Education (DepEd) management capacity to assume greater responsibilities following the passing of the Governance of Basic Education Act 2001. It maps and examines the process of connecting staff who were previously performing the same function independently, to come together to form knowledge sharing networks. Real examples of the changes that emerging CoPs are delivering are also provided. Discussion then turns to the new challenges CoPs face in sustaining the momentum and benefits that these professional networks are yielding.
A Stagecraft of New Zealand Evaluation
Authors: Lunt Neil T , Trotman Ian G 
Published: January 2018

Since the 1960s there has been a growing interest in evaluation shown by most Western countries. Alongside discussion of practical and theoretical issues of evaluation, such as methodological developments, best practice, and cross-cultural practice, there has also been increased interest in mapping the history of evaluation activity. Historical discussions are significant for three reasons; first, in providing a record for future generations of evaluators. Second, they provide a consideration of the domestic and international context that has shaped evaluation development, giving each country its distinct institutional make-up and brand of evaluation activity. Third, they assist a country's evaluation capacity development by building on its strengths and compensating for the weaknesses of its history. This article traces the emergence of evaluation within New Zealand using the metaphor of dramaturgy to introduce the settings and actors that we consider to have been constituent of what was played out in the New Zealand situation. Our remit is a broad one of attempting to describe and explain the range of evaluation activities, including program evaluation, organisational review, performance management, and process and policy evaluation. Within this article a broad overview only is possible. As an example of a more in-depth study, a comprehensive article could be prepared on the history of performance management in the public service. Our comments cover developments in the public sector, tertiary sector, and private and professional organisations. It is a companion paper to one on the history of evaluation in Australia, prepared by Colin A Sharp in a recent issue of this journal (Sharpe 2003).
Empowerment Evaluation: A Practical Method for Evaluating a National School Breakfast Program
Authors: Miller Wayne , Lennie June 
Published: January 2018

Over the past decade, empowerment evaluation has demonstrated its worth as a practical and valuable alternative to traditional program evaluation methodologies. Following a brief overview of the development of empowerment evaluation, we outline its 10 underlying principles and the three key steps involved in conducting an empowerment evaluation. A preliminary case study on our use of this methodology to evaluate the Good Start Breakfast Club program is presented. The Australian Red Cross operates this program in nearly 100 primary schools around Australia with sponsorship and support from the Sanitarium Health Food Company and other organisations. This case study highlights the potential effectiveness of empowerment evaluation in improving and assessing the impacts of other public health and community-based programs in Australia and some of the lessons we have learned so far from using this methodology. We identify strengths and limitations of empowerment evaluation and argue that a more critical approach is required to avoid the idealism that tends to underpin this approach. We suggest that greater attention needs to be paid to the communicative and relational dimensions of empowerment evaluation in order to increase its effectiveness.
Fly High, Go Deep, Add Value: Characteristics of ‘Black Belt’ Evaluators
Authors: Lake Robert 
Published: January 2018

In this paper I discuss the characteristics of an expert evaluator. My thesis is that the expert evaluator—the ‘black belt’ in the field—is characterised by taking a broad view that sees patterns where others see points; by delving deep for root causes where others see surface explanations; and by emphasising worth and value. This paper is a thought piece and a reflection—its aim is to stimulate debate and discussion rather than to offer definitive conclusions. The intent of this paper is to explore what it is that makes a ‘black belt’ evaluator; that is, someone who is a recognised expert in the area with demonstrated mastery. The thesis of the paper is that expertise/mastery in evaluation involves more that just the superior acquisition of the same set of skills, attributes or competencies that define ‘an evaluator’; rather expertise/mastery involves an additional set of qualities. These additional qualities are taking the macro perspective and a focus on value for the client. The structure of this paper is as follows. After discussing why I think it matters to consider ‘black belt’ evaluators, and what ‘black belt’ means, I describe the work in Australia and New Zealand on evaluator competencies, Next, I consider from a theoretical perspective the nature of competence and expertise before considering how competence models fit within an expert–novice framework. I then draw on the literature on expert–novice differences to hypothesise areas in which expert and novice evaluators might differ.
The fate of recommendations
Authors: Wheeler Glenn 
Published: January 2018

This article examines a case of recommendation implementation in evaluation practice. It summarizes the results of a performance audit, Management of Programs for First Nations, completed by the Office of the Auditor General of Canada and reported to the Canadian Parliament in May 2006. The performance audit took an innovative approach to examining First Nations programs that included applying a causal lens to identify and understand factors critical to the successful use of recommendations in complex government decision-making. The performance audit assessed the progress of federal departments in implementing recommendations that the Auditor General had made in audits reported between 2000 and 2003 on First Nations issues. The Office of the Auditor General's performance audits usually make recommendations and sometimes follow-up audits report on their implementation, but typically do not address the reasons behind the progress of adoption; this audit was different in that it attempted to ascertain some of the reasons for progress or the lack of progress. This innovative approach resulted in the identification of several factors that appear to be critical to the successful implementation of recommendations, and to the successful design and delivery of programs.
2008 AES Awards for Excellence in Evaluation
Authors: Bycroft Peter 
Published: January 2018

The aim of the AES Awards for Excellence in Evaluation is to encourage the development of evaluation and high-quality evaluation practice in Australasia. In 2008, seven awards were offered by the Australasian Evaluation Society. Two of the awards recognise individual evaluators nominated by their peers: the Outstanding Contribution to Evaluation (ET&S) Award and the Emerging New Talent Award. Five awards recognise organisations, project teams, specific evaluation projects, programs or systems nominated by the evaluators and/or the organisations involved: the Best Evaluation Publication Award (the Caulley Tulloch Award), the Best Evaluation Study Award, the Best Evaluation Policy and Systems Award, the Community Development Evaluation Award and the Indigenous Evaluation Award. The following awards were presented at the 2008 AES International Conference dinner in Perth in September.
Additionality: A Useful Way to Construct the Counterfactual Qualitatively?
Authors: Hind Julie 
Published: January 2018

One concern about impact evaluation is whether a program has made a difference (Owen 2006). Essentially, in trying to understand this, an evaluation seeks to establish causality—the causal link between the social program and the outcomes (Mohr 1999; Rossi & Freeman 1993). However, attributing causality to social programs is difficult because of the inherent complexity and the many and varied factors at play (House 2001; Mayne 2001; Pawson & Tilley 1998; White 2010). Evaluators can choose from a number of theories and methods to help address causality (Davidson 2000). Measuring the counterfactual—the difference between actual outcomes and what would have occurred without the intervention—has been at the heart of traditional impact evaluations. While this has traditionally been measured using experimental and quasi-experimental methods, a counterfactual does not always need a comparison group (White 2010) and can be constructed qualitatively (Cummings 2006). With these in mind, this article explores the usefulness of the concept of additionality, a mixed-methods framework developed by Buisseret et al. (cited in Georghiou 2002) as a means of evaluative comparison of the counterfactual.
The ‘Problem’ of Gender and Development Evaluation in the Pacific
Authors: Newton Karen 
Published: January 2018

Gender equality is generally accepted within the international development community as being critical for achieving the Millennium Development Goals (MDGs) agreed to by member nations of the United Nations in 2000. However, this article suggests that there are tensions between the MDGs and the strategies to achieve gender equality on the one hand, and international development policies, such as the Paris Declaration on Aid Effectiveness (2005), on the other. Furthermore, rather than gender equality being central to achieving the MDGs it appears that since the Paris Declaration, gender equality has ‘dropped off’ the mainstream international development agenda. This article reviews the policy and practices of NZAID (New Zealand's international aid and development agency) to help illustrate the difficulty of keeping a focus on gender within the current international development policy context. The impact of this issue on the effectiveness of official development assistance (ODA) in helping to achieve the MDGs in the Pacific is examined, as are the implications for evaluative policy and practice of a weakened focus on gender. In conclusion, it is suggested that, although evaluators do not set the development agenda of either aid agencies or developing countries, it is still possible to conduct evaluations that are ‘gender sensitive’ and that have the potential to transform gender relations of those women and men that development assistance is intended to benefit.
Closing the Gap through Respect, Relevance, Reciprocity and Responsibility: Issues in the Evaluation of Programs for Indigenous Communities in Australia
Authors: Markiewicz Anne 
Published: January 2018

This article outlines key considerations and issues inherent in undertaking effective evaluations of programs developed for Indigenous Australians, written from the perspective of a non-Indigenous evaluator. The considerations identified include a number of professional practice areas, namely the need for evaluators to: operate with an understanding and appreciation of the historical and systemic context experienced by Indigenous Australians; work from the basis of a solid code of ethics and practice standards; and operate with high levels of cultural sensitivity and the ability to appreciate Indigenous world views and differences. Another central consideration identified in the article is that evaluators need to hold a commitment to produce useful and useable evaluation findings that can inform future program design and social policy in relation to improving the health and wellbeing of Indigenous Australians. The article develops a number of principles for evaluators to follow in evaluating programs developed for Indigenous Australians. These include: having respect for the importance of historical, socioeconomic and psychological context; commitment to ensuring relevance in methodologies and approaches used; reciprocity in considering the benefits for participating Indigenous communities; and responsibility in undertaking effective communication and consultation. While most evaluators would agree with the above principles and values, the latter can be more challenging to apply in practice and examples of their application to evaluation, and the inherent challenges in the Indigenous context, are described. Such challenges include: operating within the restrictions of government silos with difficulties portraying the complex interrelationships that reflect the lives of Indigenous Australians; reaching consensus on evaluation design and implementation; working within restricted time frames with lack of opportunity to appreciate and reflect the unique differences among Indigenous communities across Australia; limitations inherent in engaging a broad and representative group of Indigenous stakeholders; and lack of control by the evaluator over the dissemination and influence of evaluation findings and results that link with subsequent decision-making processes.
Results-Based Accountability: Evaluating Program Outcomes in a Social Services Organisation in New Zealand
Authors: Weir Annie , Watts Rod 
Published: January 2018

In times of financial constraint, social services organisations reliant on external funding from government, community and philanthropic sources are under increasing pressure to demonstrate to their stakeholders the difference they are making to their clients' lives. This can present a challenge for organisations, encouraging them to move beyond traditional methods of financial accountability. From collecting data on the level and extent of their services, they need to explore new accountability tools that best demonstrate client outcomes to stakeholders. This article presents a case study of a two-year evaluation that explored the challenges for Presbyterian Support Northern, a large New Zealand social services organisation at the forefront of the sector, as they attempted to improve the quality of measurement and evaluation of organisational and service performance through the adoption of Results-Based Accountability, a client outcomes-focused approach. The evaluation aimed to investigate the impact of Results-Based Accountability on the organisation's culture and performance, using a participatory action research mixed methods approach. Data from interviews and focus groups were used to look at changes in culture. Senior management of Presbyterian Support Northern are committed to Results-Based Accountability and steady progress has been made in embedding it into the organisational culture. An ongoing challenge for the organisation is to help staff, in particular those on the front-line, to understand the dimension related to making a difference in client outcomes. Implications for evaluation practice are based on the lessons learnt by the organisation in implementing Results-Based Accountability.
Testing Policy Evaluation Methods for Smoke-free Outdoor Area Policies
Authors: Cortés Leanne Toledo , Thomson George , Edwards Richard 
Published: January 2018

We aimed to test evaluation methods for smoke-free outdoor area policies using a long-term evaluation of a smoke-free parks and playgrounds policy as a case study. We documented the results and an appraisal of a multifaceted evaluation (playground and sports field user surveys, observations, butt collections and key stakeholder in-depth interviews) of a New Zealand local council's smoke-free policy in 2011. We also drew upon previous evaluations completed before (2008) and immediately after (2009) the implementation of the policy. Butt collection was a particularly efficient method of data collection due to the in-field data time (five hours) compared with observations (31 hours) or surveys (46 hours). Surveys and key stakeholder interviews yielded important information to inform effective recommendations to best promote the policy. We ensured effective replicable and accurate data could be collected by training and supervising volunteers for data collection, and documenting all methods. Each method we tested helped us to evaluate the impact of the policy, but not all methods were resource efficient.
Rethinking the Roles of Evaluation in Learning how to Solve ‘Wicked’ Problems: The Case of Anticipatory Techniques used to Support Climate Change Mitigation and Adaptation
Authors: McGrail Stephen 
Published: January 2018

Many of the most pressing policy and social challenges of the modern era are widely viewed to be ‘wicked’ problems, that is, complex, largely intractable, contested problems. A good example is climate change mitigation and adaptation. The dual context for this article is: the recent increase in evaluative questions being asked by practitioners and policymakers who are seeking to address wicked socio-environmental problems (like climate change); and the recognition that such problems and related interventions present major challenges to evaluative inquiry. To effectively answer these evaluative questions, both practitioners and policymakers need the assistance of evaluators who recognise the limitations of traditional accountability-oriented evaluation and adequately understand the complexity of programs, the contexts in which they are embedded, and the socio-environmental problems they seek to address. This article reviews and builds on discussion of wicked environmental and policy problems in the evaluation literature to consider the evaluation challenges (and associated guiding principles) and potential roles for evaluative inquiry. The article argues that evaluations of interventions into wicked problems should be grounded in the realist paradigm. Drawing on the review and this theory, four roles for evaluative inquiry are identified: (1) testing and refining transferable intervention theories; (2) enabling learning (rather than demonstrating accountability); (3) providing decision support; and (4) mapping program complexity and prompting adequate consideration of complexity.
Evaluating as an Outsider or an Insider: A Two-Way Approach Guided by the Knowers of Culture
Authors: Rossingh Bronwyn , Yunupingu Yalmay 
Published: January 2018

Evaluating programs in remote Indigenous communities requires contextual consideration and a degree of connection that goes beyond the usual focus on output measurement and system monitoring. Evaluators who are experienced with working in remote communities become more and more cognisant of the issues and complexities that impact the quality of evaluations. As this reality presents itself, evaluators rely more and more on the help of the local community to explain and reveal such realities. A learning journey takes place that allows ‘outsiders’ to be accepted into the community and be part of a productive matrix of support and ‘insiders’ begin to assume their legitimate rights to participate, advise, educate and guide evaluators. Without a participative and collaborative styled process, evaluation risks being non-reality based without a guarantee of quality and has the potential to impact the community in a negative way. This paper presents the views and experiences of ‘insiders’ and ‘outsiders’ who have been involved in evaluations in remote communities. Their collective views demonstrate the need for a two-way approach that establishes middle ground for a respectful and quality evaluation process. This approach ensures evaluation remains relevant and purposeful for those who are impacted by the program outcomes, providing a focus on the everyday reality that depends on cultural responsiveness and ensures legitimacy for Indigenous people.
Making Formative Assessment Feasible: A Collaborative Approach Supporting Grant Funded Projects
Authors: Gullickson Amy M. , Al-Nawab Hadeel F. 
Published: January 2018

Formative assessment is a means of unlocking the ‘black box’ of an intervention to explore and understand cause and effect, and is, therefore, vital for improvements that lead to success. However, most funders focus on summative evaluation, and projects with limited resources struggle to meet that requirement, leaving formative investigations ideal, but infeasible. Yet, formative assessment is essential when conducting an intervention that must create results with a small budget. This paper discusses the Formative Assessment Systems for Advanced Technological Education (FAS4ATE) project, which used a high level, theory based participatory approach to address these issues in small grant funded projects. FAS4ATE brought the leaders and evaluators from a variety of small projects together with experts in formative assessment and evaluation in two webinars and a workshop. Logic models were the core activity and teaching tool used to identify formative assessment needs throughout the life cycle of individual projects. The logic models were then analysed across projects to provide specifications for the development of tools and resources to support formative assessment. The paper describes the project's process, key strategy, products and recommendations. It will be particularly valuable for those who are interested in evaluation capacity building, both in terms of individual learning and creating evaluation systems.
Transcranial magnetic stimulation (TMS) safety: a practical guide for psychiatrists
Authors: Taylor Rohan , Galvez Veronica , Loo Colleen 
Published: January 2018

Objectives: Repetitive transcranial magnetic stimulation (rTMS) is increasingly being utilised as a treatment option for depression, and with this comes a need for a practical review of safety issues intended for clinicians. This article provides an overview of the current literature regarding safety issues with rTMS for depression, and provides recommendations for clinical practice. Conclusions: Overall, rTMS is a well-tolerated treatment with common side effects (such as headache or local pain at the site of stimulation) being mild. Severe adverse effects, such as seizures, hearing impairment or mania, are uncommon. Certain populations, including adolescents, pregnant women, older adults and those with metal/electronic implants, require special consideration when prescribing and monitoring treatment courses. With adequate assessment and monitoring processes, rTMS can be administered safely in a large proportion of depressed patients.