bluuebunny/crossref_metadata_2025_split huggingface dataset:
Using an Empowerment Evaluation Approach with Community-based Programs: Reflections from the Front Line
Authors: Newell Sallie , Graham Anne 
Published: January 2018

Identifying the most appropriate approach for small-scale program evaluation remains a vexed issue for both the researchers who design and implement evaluations, and for the service provider organisations that seek to ensure that they can use the findings as evidence for further program funding. This article reflects on our experiences and the issues encountered in attempting to undertake useful small-scale, community-based program evaluation, particularly in regional settings, where the evaluation capacity, practices and cultures of organisations are still evolving. Cognisant of increasing funder-fundee tensions arising from more externally controlled social program evaluations and considering varied organisational, program, personnel, funding and broader political needs and influences, we have focused on tailoring evaluation approaches to ensure they are robust, relevant and responsive to the varied organisational contexts in which we are endeavouring to strengthen evaluation capacity. The approach that has emerged over the past eight years fits most closely with an empowerment evaluation framework that, although theoretically well suited, has had to be adapted in order to respect, and work productively with the frequently competing imperatives of the different stakeholders involved.
Building Evaluation Capability in the Public Health Workforce
Authors: Dickinson Pauline , Adams Jeffery 
Published: January 2018

Building evaluation capability and capacity is a complex task involving enhancing and building technical skills and working closely with individuals and groups to ensure evaluation learning is embedded in organisational philosophy and practice. In this article, we present the evaluation findings of the ‘Easy Evaluation’ initiative, which provides evaluation training workshops and organisational coaching and support, in order to address our central question: Are evaluation training workshops effective and what else is needed? The findings are organised in line with three levels of Kirkpatrick's four-level evaluation model: trainee reactions, learning and behaviour. The ‘reaction’ and ‘learning’ data indicate that workshops were found to be effective in developing positive responses and enhancing trainee learning for most participants. Evaluation data relating to ‘behaviour’ indicated that just over half the training participants, at least three months following the training, were implementing evaluation activities. In summary, to build sustainable evaluation practice within organisations, training needs to be supplemented with other evaluation capacity-building strategies.
Developing Symbiotic Clinical Educators: Using Program Logic to Evaluate a Clinical Education Course
Authors: Berryman Carolyn , Sweet Linda , Wearne Susan , Greenhill Jennene 
Published: January 2018

Developing health professional clinical educators able to meet the shifting health care requirements of this century is a current government priority. To change practices in the clinic, those who teach need to change and that is best achieved by specific curricula designed for faculty development. Principles revealed during an investigation of a successful community-based medical education program underpin the Master of Clinical Education (MCE) course, a new faculty development curriculum. After six years, it is timely to evaluate whether the educational goals of the MCE course are being met. The MCE faculty employed an independent research officer to conduct the study. A two-phase approach used semi-structured interviews and content analysis to develop and assess outcomes against an outcomes logic model. The program achieved all short-term goals: understanding the symbiotic model; commitment to self as teacher; and changes in the way to give feedback. It achieved two of three medium-term goals: belief of transformation; and relationship development. One medium-term goal was not met: developing a sense of community. This evaluation of the MCE course shows the program develops clinicians as committed educators and also as change agents who see building relationships as the key to operating effectively within complex health services and systems.
Bridging the Divide between Upward Accountability and Learning-Based Approaches to Development Evaluation: Strategies for an Enabling Environment
Authors: Lennie June , Tacchi Jo 
Published: January 2018

A more complex and unstable development context and major shifts in development theory and practice over the past 20 years have created major challenges for evaluation. There remains a vast gap between the theory and practice of development evaluation that needs to be resolved. This indicates a vital need to bridge the divide between dominant results-based, upward accountability evaluation approaches and emergent learning, participatory and complexity-based approaches to more effectively achieve development goals. In this article we explore this divide and contrast these approaches, highlighting some of the key challenges that each face. Drawing on our recent research, we identify key challenges, tensions and contradictions in evaluating communication for development (C4D) that need to be better understood and addressed. We outline a new conceptual framework for evaluating C4D and other complex development initiatives as a strategy to address these challenges. We argue that achieving change requires reconceptualising accountability and learning, developing evaluation capacities that enable learning and multiple forms of accountability, and drawing on local knowledge, ideas and innovation to develop and implement more appropriate and effective evaluation approaches. These strategies can create an enabling environment in which these new ideas and approaches can flourish and complex development issues can be better addressed.
Combining Realism with Rigour: An Evaluation of a National Kitchen Garden Program in Australian Primary Schools
Authors: Yeatman Heather , Quinsey Karen , Dawber James , Nielsen Wendy , Condon-Paoloni Deanne , Eckermann Simon , Morris Darcy , Grootemaat Pam , Fildes David 
Published: January 2018

School garden and kitchen-based activities have potential as health-promoting school initiatives but lack well-designed evaluation frameworks. This article outlines an evaluation of a national kitchen garden program conducted in Australian primary schools. The aim was to evaluate the implementation and impact of a national garden and kitchen program in Australia, including child and health-related impacts of the program, and program implementation factors. The evaluation was undertaken using a multi-component, hierarchical evaluation framework that incorporated a modified case-control design and mixed methods. A cross-section of initiative (N=28) and comparison (N=14) schools was included. The Stephanie Alexander Kitchen Garden model achieved overall improvements in students’ food choices (as reported by students; t=2.26, p=0.024) and kitchen lifestyle behaviours (as reported by parents; t=2.35, p=0.019); female students and students from provincial schools had statistically greater improvements in food choices than other students and girls showed higher scores than boys (t=6.19, p<0.001). The evaluation framework generated data that addressed the evaluation questions. Program integrity was maintained. Factors related to the program's implementation, sustainability, generalisability and return on investment were reported. Factors impacting the evaluation included lack of baseline data, specific data collection methods and time issues. The main conclusion drawn from the study is that structured hierarchical, multi-component evaluation is needed to provide practical insights into effective health promotion initiatives in schools.
Finding the common thread: The charms and challenges of evaluation for a consortium-run international development program
Authors: Dinh Kathryn , Keys Tricia , Thomson Naomi 
Published: January 2018

This article explores the opportunities and challenges in designing and implementing a monitoring and evaluation (M&E) framework for a consortium-run, multi-country international development program. The East Asia Vision Program (EAVP), a three-year program of the Vision 2020 Australia Global Consortium, aims to improve capacity in the delivery of eye health and vision care services in Vietnam, Cambodia, and Timor-Leste. Funded by the Australian Government, the EAVP began in 2013 and consists of five Australian organisations working with government and other stakeholders in each country. The Consortium members have been working in collaboration to support national government planning, health professional training, patient treatment, and research capacity through monitoring and evaluation. An M&E Reference Group helped with initial drafts of an M&E framework. Australian and in-country staff were then consulted to ensure availability of data and understanding of the Framework. Early drafts of data entered into the framework were shared between the participating organisations and the evaluator in order to detect errors and share good practice examples using online program management software. Monitoring the program provided an opportunity for Consortium members to improve program implementation and strengthen their capacity for M&E, by sharing examples of evaluation tools and expertise in the monitoring of cross-cutting issues and data collection. The combined work of the organisations within each country provided a rich dataset of outcomes at a health systems strengthening level. Challenges faced during the evaluation included: aligning the evaluation systems of all organisations (including data that were feasible to collect); monitoring a large range of activities; and developing an evaluation tool that was usable by a diverse range of staff. This article reports the perspectives of the M&E Advisor for EAVP and those of a program implementer to share learnings regarding the M&E for a consortium-run program. Consortia are used globally to implement international development programs and they present a unique challenge for evaluators. These consortia also provide the opportunity to build the monitoring and evaluation capacity of participating organisations, leading to improved data quality and better-informed program implementation.
Working Both-Ways: Using Participatory and Standardised Methodologies with Indigenous Australians in a Study of Remote Community Safety and Wellbeing
Authors: Sutton Sue , Baxter Nathalie , Grey Kim , Putt Judy 
Published: January 2018

This paper offers reflections on our experience and learning arising from implementing a study design that used evaluation research to pursue multiple benefits. The Community Safety and Wellbeing Study adopted a mixed methods approach, referred to as a ‘both-ways’ (or two-ways) research model, that addressed decision maker's needs and heard the people's voice. The study design was inspired by a both-ways learning model and attempted to address both needs together. The aim of the study was to involve local people in communities and encourage them to share their views about changes in community safety. Through systematic research it provided a voice for Indigenous Australians affected by the Northern Territory Emergency Response (NTER), often called ‘the intervention’. Regardless of views about the intervention, this paper aims to share the lessons learned from conducting this study. The mixed method approach involved a community based standardised survey and qualitative data collection techniques. The study was undertaken in a representative sample of 17 NTER communities with over 1300 local residents, around five percent of the relevant population. Over 60 Indigenous people were employed in conducting the study; the majority lived in or had strong ties with remote communities in the study and around 10 had significant experience with social research projects. The research had many objectives in addition to providing evidence on outcomes on a multi-faceted and sometimes controversial government policy. Ethically the research had to have benefit for the people and communities involved. In addition, the community had to be able to see there were benefits from the research for them, not just for government. This paper documents how these objectives were achieved in relation to the methodology, content, data collection and reporting aspects of the research, and discusses what worked and what could be done differently in the future.
A Case for the Advancement of Utilisation of Realist Evaluation in International Development: An Oxfam Focused Illustration
Authors: Mukova-Moses Maud 
Published: January 2018

Grounded in scientific realist philosophy and expounded by Pawson and Tilley (1997), realist evaluation is founded on the assertion that community development programs seek to address real problems, and not social constructs. It is therefore a realistic approach to evaluation that informs policy and decision makers of the effectiveness of programs, examining the program's specific mechanisms, contexts and outcomes. Pawson and Tilley (1997) place people at the centre of the realist approach to evaluation, emphasising that the effectiveness of programs is dependent on the respondents’ willingness to engage, as well as the existence of an environment that enables them to participate. This paper seeks to reinforce the significance of realist evaluation in international development programs, illustrating its value in informing programs that are effective and responsive, and employing contextually appropriate mechanisms that enhance communities’ benefits. Realist philosophy asserts that, ‘entities exist independently of being perceived, or independently of our theories about them’ (Phillips, 1987, p. 205). It is this reality, outside of our knowledge and experiences, that we seek to define, as best we can and never absolutely, through scientific approaches and social inquiry (Ball, MUller & Nelson, 2014). It is within this framework that realist evaluation exists. As championed by Pawson and Tilley (2004), realist evaluation is theory based and seeks to answer the question, ‘what works for whom in what circumstances and in what respects, and how’ (p. 2). Pawson and Tilley (1994) note that unlike experimental research which focuses primarily on addressing what works using a successionist approach1, realistic evaluation takes a generative approach and examines underlying causal linkages to changes observed (p. 291). Examining mechanisms, contexts and outcomes, realistic evaluation presents a methodological approach that amplifies the role of people in bringing about change. For Pawson and Tilley (1994), ‘programmes ‘work, if subjects choose to make them work and are placed in the right conditions to enable them to do so’ (p. 294). This is useful to Oxfam Australia's (2014) focus on ‘a just world without poverty, where people influence decisions that affect their lives, enjoy their rights, and assume their responsibilities—a world in which everyone is valued, and everyone is treated equally’ (p. 6). Inspired by the Australasian Evaluation Society's (2016) priority to influence and promote evaluation practice, this article seeks to contribute to the discourse on realist evaluation in an endeavour to advance its application in evaluation practice within the international development sector.
The Capacity Development Evaluation Framework: Tested on Three Initiatives
Authors: Kotvojs Fiona 
Published: January 2018

Adoption of effective capacity development has long been considered critical to reducing poverty through international development assistance activities (World Bank, 2005). Consequently, annual global investment in capacity development now exceeds US$30 billion (Ling & Roberts, 2012). However, capacity development results have been questionable and evaluation findings have contributed little to addressing this due to their poor quality and inability to provide the information users require. Using evaluation of capacity development to address these weaknesses, this research sought to determine whether the application of the Capacity Development Evaluation Framework (referred to hereafter as the Framework) provided users with the information, and demonstrated the characteristics, they required. The research adopted a case study approach to three initiatives that applied the Framework. Data was collected from 37 documents and through repeated interviews with 50 users over at least 18 months. The research also considered alternative explanations for the findings. The research found that application of the Framework identified: the changes to which the initiative contributed, progress toward the objectives, effectiveness of different capacity development strategies, and lessons learnt. The Framework did not identify unintended changes or changes in the environment that may influence outcomes. Users identified that application of the Framework provided numerous immediate and future benefits and consequently, many had since applied the Framework elsewhere. The research concluded that because application of the Framework provided users with the information they required, its use should contribute to strengthened capacity development outcomes.
Building Evaluation Capital in Government: A Queensland Departmental Approach
Authors: Goswami Sarah , Lane Vicki 
Published: January 2018

Increasingly, government departments are being held accountable for investment in public services. In Queensland the Financial Accountability Act 2009 (Queensland Treasury, 2016) requires that accountable officers and statutory bodies ‘achieve reasonable value for money by ensuring the operations of the department or statutory body are carried out efficiently, effectively and economically’ (Section 61). Whilst there is a directive for agencies to evaluate and demonstrate value for money, it has in practice been difficult to embed long term, as many systems and decision makers have neglected the role of organisation-wide evaluation capital. This paper will outline the work being undertaken in the Queensland Department of Agriculture and Fisheries (DAF) to implement an Impact and Investment Framework, which will support and embed evaluation in a multidisciplinary setting. A central tenant of this framework is ‘business empowerment and learning'—building the evaluation culture in the organisation by first establishing evaluation building blocks, through business empowerment, support and utility. The framework is comprised of five key elements and is built on the principles of evaluation and evaluation capacity building disciplines. It has been designed to be low-cost, effective and efficient, whilst enabling business improvement, meeting accountability needs and allowing the department to demonstrate the value of its work.
Trends and repetition of non-fatal suicidal behaviour: analyses of the Gold Coast University Hospital’s Emergency Department
Authors: Kõlves Kairi , Crompton David , Turner Kathryn , Stapelberg Nicolas JC , Khan Ashar , Robinson Gail , de Leo Diego 
Published: January 2018

Objective The aim of the current paper is to analyse time trends of non-fatal suicidal behaviour (NFSB) and its repetition at the Gold Coast in 2005–2015. Methods Data on presentations for NFSB were obtained from the Emergency Department (ED) Information System. Potential cases were identified through keyword searches, which were further scrutinised and coded. Annual person-based age-standardised rates for NFSB were calculated. Chi-square test, Poisson regression and Cox proportional hazards regression were used. Results: There was a significant increase in the age-standardised rates of NFSB for males (incidence Rate Ratio = 1.05; 95% confidence interval (CI): 1.04–1.07) and females (iRR = 1.06; 95% CI: 1.04–1.07). Age-specific rates showed significant increases for all age groups, except 25–34 and 55+ for females. Different types of poisoning were the predominant method of NFSB (poisoning only – 61.7% of episodes), followed by cutting (23%). Within the first year after the index episode, 13.4% of subjects repeated NFSB. Multivariate Cox regression model showed that sex, age and method predicted repetition. Conclusion: The increasing trends of NFSB and relatively high repetition rates emphasise the need for preventative actions. Monitoring of NFSB at the ED level should be further extended in Australia.
A New Technique in Alveolar Cleft Bone Grafting for Dental Implant Placement in Patients With Cleft Lip and Palate
Authors: Van Nhan Vo , Van Son Le , Tuan Ta Anh , Son Nguyen Tai , Hai Trinh Dinh , Lanh Le Duc , Ha Nguyen Manh , Phong Lam Dai 
Published: January 2018

Objective: To evaluate 2 iliac corticocancellous-block grafting techniques for dental implant placement in residual alveolar clefts. Design: Nonrandomized prospective clinical trial between March 2010 and December 2014. Setting: National Hospital of Odonto-Stomatology, Hanoi, Vietnam. Participants: Thirty-two patients (23 female, 9 male; mean age, 21.28 years; range, 16-31 years) with unilateral complete alveolar cleft after reconstructive surgery for cleft lip and palate (CLP). Interventions: Harvested iliac crest bone was cut into 2 corticocancellous blocks. The smaller block was adapted against the sutured nasal mucoperiosteum and overlaid with cancellous bone; the larger one overlapped the labial cleft margin and was fixed with screws. Endosteal dental implants were placed after 4 to 6 months, and final restorations were delivered 6 months later. Main Outcome Measures: Flap statuses were assessed clinically. Bone formation was assessed using the Enemark scale. Cone-beam computed tomography was used for graft height and width measurements. Implant health was assessed by the Misch criteria. Results: The mean postgrafting follow-up period was 36.7 ± 10.4 (range, 18-53) months. Three patients (9.4%) showed flap dehiscence but no infection 7 days after bone grafting. Twenty-nine patients (90.6%) had 75% to 100% bone fill (Enemark score of 1). The mean graft height and width were 11.4 ± 2.4 and 6.1 ± 1.0 mm, respectively. Sufficient bone for implant placement was noted in 29 patients (90.6%); the others required partially fixed prostheses. All implants functioned for at least 18 months. Conclusion: The proposed technique is reliable to reconstruct the alveolar cleft for implant placement in CLP patients.