bluuebunny/crossref_metadata_2025 huggingface dataset:
Protective mechanisms of 6-gingerol in dextran sulfate sodium-induced chronic ulcerative colitis in mice
Authors: Ajayi BO , Adedara IA , Farombi EO
Published: January 2018

Ulcerative colitis (UC) is a relapsing and remitting inflammatory disease of the colon, with an increasing incidence worldwide. 6-Gingerol (6G) is a bioactive constituent of Zingiber officinale, which has been reported to possess various biological activities. This study was designed to evaluate the role of 6G in chronic UC. Chronic UC was induced in mice by three cycles of 2.5% dextran sulfate sodium (DSS) in drinking water. Each cycle consisted of 7 days of 2.5% DSS followed by 14 days of normal drinking water. 6G (100 mg/kg) and a reference anti-colitis drug sulfasalazine (SZ) (100 mg/kg) were orally administered daily to the mice throughout exposure to three cycles of 2.5% DSS. Administration of 6G and SZ significantly prevented disease activity index and aberrant crypt foci formation in DSS-treated mice. Furthermore, 6G and SZ suppresses immunoexpression of tumor necrosis factor alpha, interleukin-1β, inducible nitric oxide synthase, Regulated on activation, normal T cell expressed and secreted (RANTES), and Monocyte chemoattractant protein-1 (MCP-1) in the DSS-treated mice. 6G effectively protected against colonic oxidative damage by augmenting the antioxidant status with marked decrease in lipid peroxidation levels in DSS-treated mice. Moreover, 6G significantly inhibited nuclear factor kappa B (P65), p38, cyclooxygenase-2, and β-catenin whereas it enhanced IL-10 and adenomatous polyposis coli expression in DSS-treated mice. In conclusion, 6G prevented DSS-induced chronic UC via anti-inflammatory and antioxidative mechanisms and preservation of the Wnt/β-catenin signaling pathway.
Building a performance-based monitoring and evaluation system: The challenges facing developing countries
Authors: Kusak Jody Zall , Rist Ray C.
Published: January 2018

There are growing pressures in countries throughout the world to improve the performance of their public sectors. One strategy now being employed in both developed and developing countries is to design and construct performance-based monitoring and evaluation (M&E) systems so as to be able to track the results produced (or not) by governments. This paper describes a 10-step approach to the design and construction of such M&E systems that is currently being deployed in a number of developing countries. The overall strategy builds on the experiences of developed countries – especially those in the OECD – but also reflects the particular challenges and difficulties faced by developing countries that can range from a lack of skill capacity to poor governance structures to systems that are far from transparent. The last section of the paper addresses in detail the first of the 10 steps – that of assessing whether a country is, in fact, ready to begin the process of M&E system design and construction.
NORM software review: Handling missing values with multiple imputation methods
Authors: Darmawan I Gusti Ngurah
Published: January 2018

Evaluation studies often lack sophistication in their statistical analyses, particularly where there are small data sets or missing data. Until recently, the methods used for analysing incomplete data focused on removing the missing values, either by deleting records with incomplete information or by substituting the missing values with estimated mean scores. These methods, though simple to implement, are problematic. However, recent advances in theoretical and computational statistics have led to more flexible techniques with sound statistical bases. These procedures involve multiple imputation (MI), a technique in which the missing values are replaced by m > 1 estimated values, where m is typically small (e.g. 3-10). Each of the resultant m data sets is then analysed by standard methods, and the results are combined to produce estimates and confidence intervals that incorporate missing data uncertainty. This paper reviews the key ideas of multiple imputation, discusses the currently available software programs relevant to evaluation studies, and demonstrates their use with data from a study of the adoption and implementation of information technology in Bali, Indonesia.
Revealing the hidden costs: Research experiences from a case study evaluation project
Authors: Burton Jo , Rajan Rashmi
Published: January 2018

Aftermath: the social and economic consequences of workplace injury and illness was a case study evaluation of the experiences of 15 individuals injured or made ill at work, their families, friends, workplaces, colleagues and government officials. 1 It was a unique research experience for those on the project team, as it brought the researchers into close contact with the pain and suffering of the participants. This article discusses the methodology of the study, the experience of the researchers, and what lessons can be taken from these research experiences.

1 Copies of the report are available online at: www.dol.govt.nz .
‘Death by evaluation’? Reflections on monitoring and evaluation in Australia and New Zealand
Authors: Ryan Bill
Published: January 2018

This paper reviews the state of play in outcome-oriented monitoring and evaluation in New Zealand in the light of recent Australian experience. Given issues arising from a ‘big bang’ approach to evaluation adopted with the Commonwealth of Australia's 1988 Evaluation Strategy, and the lead times involved in building capacity, it argues that New Zealand should focus in the short-term on creating ‘evaluative management cultures’ inside organisations rather than opting exclusively for building formal and technical evaluation capacity and expectations – although that too should be developed, even though it will take some years to mature. Actually, a practice rather than institutional approach to capacity building that tries to ‘build from within’ would also be consistent with current thinking in management and organisational development. More may be gained by developing a learning approach to public management, strategic emphasis on internal evaluation, and focusing on immediate and intermediate outcomes especially in relation to monitoring.
Evaluation culture: A definition and analysis of its development within organisations
Authors: Owen John M
Published: January 2018

Discussions about decision-making and the kinds of knowledge that could and should be used for this purpose within the workplace have become prominent in current organisational literature. These issues are high on the agenda of the business community, and among not-for-profit agencies.

Research and observation suggests that a good deal of organisational decision-making has, in the past, been based on the judgements of authority figures. It was long assumed that senior managers had the sole right to make decisions, relying on a combination of experience, political know-how and the advice of trusted others in making choices about the present and future of the agencies for which they were responsible.

This situation is changing. Most organisations recognise that important executive-level decisions should involve others besides senior managers, and I have previously argued that evaluators should work cooperatively in providing ‘just in time’ information for leaders (Owen & Lambert 1998). This implies that evidence and empirically based knowledge has the capacity to enhance decision-making and the effectiveness of organisations, for example by making managers and other practitioners more aware of the context in which they operate, understanding the needs of their clients, determining the effects of major initiatives, and being accountable to funding sources.

In this paper, I argue that the creation of an evaluation culture leads to a change and improvement in the ‘mix’ of ‘working knowledge’ that is used by those responsible for applying information to solve organisational problems.

An evaluation culture can be regarded as a commitment to roles for evaluation in decision-making within an organisation (Owen & McDonald 1999). This is systematic enquiry which is initiated and controlled by members of the organisation, and is carried out with the explicit purpose of contributing to the stock of its working knowledge. Enquiry of this nature is not undertaken routinely, but in response to the need for empirically based knowledge to contribute to issues regarded as strategic.
Reflections on a decade in the life of the Australasian Evaluation Society: 1990-1999
Authors: Trotman Ian G
Published: January 2018

The 2003 AES Conference1 was the 21st birthday of conferences held by a group of evaluators who after a few such efforts organised themselves more formally in 1996 to become the AES, which was officially registered in Victoria in 1987. Ours was the third such regional professional society of evaluators, after the Canadian Evaluation Society (CES) started in 1981 and the American Evaluation Association (AEA) formed in 1986, as the consolidation of two earlier evaluation associations: the Evaluation Network and the Evaluation Research Society.

This article is not a history of the AES. It lacks the academic rigour and research. It particularly overlooks much of the activity of the regions which are really the heart of the Society. It reflects my experiences, my perceptions, and my analysis over the ten years when I was most actively involved in the Society. Despite these deficiencies or potential biases, I hope it provides an insight into what goes into making a professional society as effective and highly regarded as the AES is, despite having had its share of trials along the way.
A Fundamental Choice: Internal or External Evaluation?
Authors: Conley-Tyler Melissa
Published: January 2018

An organisation wishes to evaluate one of its programs. It can ask a staff member or hire someone outside the organisation. Which should it choose?

Surprisingly little guidance is available for this common scenario. A review of 30 texts dealing with organisational performance and evaluation shows that too often the issue is assumed one way or the other.

Management texts aimed at business and organisational audiences tend to presume that evaluation is conducted by internal evaluators, usually managers. By contrast the specialist evaluation literature almost always proceeds from the opposite assumption: that evaluation is undertaken by external evaluators.

This paper proposes a series of measures for comparing the strengths and weaknesses of internal and external evaluators. These include cost, knowledge, flexibility, objectivity, accountability, willingness to criticise, ethics and utilisation of results.

A set of guidelines is offered to assist organisations in choosing between internal and external evaluation in each particular case.
Evaluating a Model of Service Integration for Older People with Complex Health Needs
Authors: Bird Steve , Kurowski William , Dickman Gillian
Published: January 2018

Background

Older people with multiple chronic conditions and complex health care needs require a comprehensive, accessible and well-coordinated system of services. To address this growing problem, a consortium of acute and community-based health care organisations implemented a ‘Patients First’ model of service integration for the target population. The project evaluation utilised a combination of quantitative and qualitative methods in an action research framework.

Findings

The evaluation process not only demonstrated the benefits of the project to patients and the health care system, but also contributed to the identification of pivotal components in the model, aspects requiring attention and consequently their refinement. It was also a vehicle for the development of a sense of ownership amongst staff and has evolved into an integral part of the model.
The Use of Photo-interviewing: Three Examples from Health Evaluation and Research
Authors: Hurworth Rosalind , Clark Eileen , Martin Jenepher , Thomsen Steve
Published: January 2018

This article reviews the use of photographs as data within the social sciences as well as defining related terminology used over the past century. It then examines the use of photos as stimuli for talking about health settings before presenting three recent case studies where photo-interviewing has been used successfully in health evaluation and research. Advantages and limitations of the method are considered.
The Development of a Logic Model for the Protection against Family Violence Act: An Incremental Approach
Authors: Laycock Janice
Published: January 2018

On 1 April 2005 the Northwest Territories (NWT) became the seventh jurisdiction in Canada to implement family violence legislation. The NWT Protection Against Family Violence Act (PAFVA) is civil legislation designed to protect victims of family violence, improve access to the justice system, and provide a wider range of remedies than were available under existing civil legislation. It also provides a proactive framework that can be used when the grounds to lay a criminal charge may not exist.

This article describes a process that was followed during the implementation period to clarify the intentions and delivery of the PAFVA program. This process involved the iterative development of a logic model. The process of development had two unique features; first it involved the clarification of a program from legislation, and second, it involved an incremental approach to logic development. Despite attention given to program clarification in the evaluation literature, there is little information on how to handle the translation of legislation through such processes. Thus, this paper is a contribution to this gap in evaluation knowledge by detailing the process followed and the lessons learned. The following section presents a contextual overview of the PAFVA.
Aes Fellows and Awards 2004
Authors: Neale Jenny , Owen John
Published: January 2018

The annual conference of the Australasian Evaluation Society for 2004 was held in Adelaide in October. Under the able leadership of Susan Dawe, pictured left, the event offered a blend of workshops, paper and panel sessions and opportunities for networking, and entertainment which began at the welcome reception.

An additional important function associated with the conference is the presentation of Fellowships to distinguished evaluators, and Awards to those who were successful in categories which have been struck by the Society.
The Comparative Value of Three Different Methodologies for Measuring the Return on Investment from a Leadership Program
Authors: Mitchell John , Hamilton Margaret , Hayman Catherine
Published: January 2018

An evaluation was conducted in 2003 of a major leadership program within Centrelink—the Senior Executive Leadership Program (SELP). The evaluation consisted of three separate studies of SELP: a Return on Investment (ROI) study, a Return on Training Investment (ROTI) study and a Value for Money (VFM) study. This paper reports on the comparative value of each of these three different approaches. Specifically, it aims to highlight the different nature and benefits of each approach in order to assist organisational decision-makers to make informed decisions about whether they should use any one, or more, of these three methodologies in the future.
Exploring program sustainability: Identifying factors in two educational initiatives in Victoria
Authors: Harvey Graeme , Hurworth Rosalind
Published: January 2018

This paper examines two recent successful school-based health initiatives in Victoria, particularly in relation to factors that seem to foster program sustainability. These programs, dealing with drug education and healthy eating, are described before presenting two different methods (individual and group) used to determine elements that allow for the continuation of such projects. The findings on sustainability from each program are discussed using the broad areas of factors associated with the programs themselves; those associated with the context in which the programs were implemented; and finally, those factors external to the programs and their implementation contexts. These results indicate a strong congruence with factors identified in the literature but also highlight the influence of the use of change theory in strengthening sustainability approaches in program development as well as the need to focus on funding options in forward planning. The possible roles for evaluators in assisting program development and supporting the integration of factors supporting sustained use are also discussed.
The joint outcomes of a ‘citizens’ jury’ workshop in North Sulawesi, Indonesia
Authors: Straton Anna
Published: January 2018

The Bunaken National Park (BNP) is a multiple-use marine national park in North Sulawesi, Indonesia. Decisions about how to balance the conservation, tourism, extractive and subsistence needs of different stakeholder groups of the BNP are challenging and often undermined by institutional failure. The main official organisation charged with managing the BNP is supported and shadowed by an NGO-facilitated multi-stakeholder collaboration that can buffer the impacts of institutional failure. One goal of this collaboration is to build decision-making capacity within the community. An evaluation process—a ‘citizens’ jury’—was undertaken for the BNP with the primary purpose of providing a comparison with an environmental evaluation exercise. This evaluation exercise yielded joint outcomes expressed through a declaration by participants to make use of citizens’ juries in future decisions about matters of public importance. The citizens’ jury process is described, and the joint outcomes of the exercise as an evaluation process are discussed. These include community empowerment, the introduction of a community-level deliberative tool, and potential capacity building in decision-making.
Contribution analysis
Authors: Kotvojs Fiona , Shrimpton Bradley
Published: January 2018

This article examines AusAID's shift to contribution analysis and a system of outcome-based monitoring and evaluating. It looks at the method of contribution analysis, its implementation in the Fiji Education Sector Program, an assessment of its use, and the challenges faced in the application of contribution analysis.
Making sense of experience
Authors: Roberts Bridget
Published: January 2018

Society's response to alcohol and drug problems must continually evolve. The author expresses optimism about evaluation's ability to guide change from the ground up through developing a culture of ongoing learning. It is argued that the alcohol and other drug treatment sector has a particular need for empowering professional development processes. The article describes the background to and personal experience of a new unit within a Graduate Diploma in Alcohol and Other Drug Studies course.

This article is an edited version of a paper presented on 7 September 2007 at the conference of the Australasian Evaluation Society, ‘Doing Evaluation Better’, Melbourne. Bridget Roberts introduced the presentation by acknowledging the people of the Kulin nation, specifically the Wurundjeri (the first people to occupy the Melbourne area and the custodians of this land), including their elders past and present.
Promoting evaluation culture
Authors: Hanwright John , Makinson Sharnie
Published: January 2018

This article describes the development and implementation of a comprehensive Program Evaluation Strategy by the Queensland Department of Education, Training and the Arts (DETA). It discusses the key stages adopted in this change management strategy by drawing upon characterisations of effective innovations.

The Program Evaluation Strategy adopted by DETA was based on a change management model rather than a compliance approach. The evaluation strategy incorporated a number of elements: the overall strategy, an evaluation manual providing supporting materials and templates, evaluation training for staff, and specific provision of advice and assistance in evaluation. The article argues that this multifaceted approach has fostered an evaluative culture within the agency and been critical in supporting the influence and use of evaluation.
2007 AES Awards for Excellence in Evaluation
Authors: Bycroft Peter
Published: January 2018

The aim of the AES Awards for Excellence in Evaluation is to encourage the development of evaluation and high-quality evaluation practice in Australasia. In 2007, seven awards were offered by the Australasian Evaluation Society, which included two new awards, the Emerging New Talent Award and the Indigenous Evaluation Award.

Two of the awards recognise individual evaluators nominated by their peers: the Outstanding Contribution to Evaluation (ET&S) Award and the Emerging New Talent Award. Five awards recognise organisations, project teams, specific evaluation projects, programs or systems nominated by the evaluators and/or the organisations involved: the Best Evaluation Publication Award (the Caulley Tulloch Award), the Best Evaluation Study Award, the Best Evaluation Policy and Systems Award, the Community Development Evaluation Award and the Indigenous Evaluation Award.

The following awards were presented at the 2007 AES International Conference dinner in Melbourne in September. Dr Zita Unger announced the award winners as Acting Chair of the AES Awards for Excellence in Evaluation Committee at the presentation.
Improving Project, Program and Policy Performance in Developing Countries through Managing for Development Results
Authors: Leonard R Keith , Bayley Scott
Published: January 2018

This article seeks to answer five questions: (1) What is ‘Managing for Development Results (MfDR)’? (2) What are the known conditions for the successful adoption of MfDR?, (3) Do these conditions apply to developing countries?, (4) Does MfDR produce better results in developing countries?, and (5) What are some of the emerging challenges in adopting MfDR? In noting that that MfDR is results-based management expressed in the language of development, the reasons for the adoption of the new term and the possible consequences of this are explored. While the perils of transferring management practices from one cultural context to another are noted, the Asian Development Bank's experience supports the view that the known conditions for successful adoption of MfDR generally apply to developing countries. Many developing countries also face special challenges not faced by rich countries and they need to deal with the particular cultural context that may support or impede successful adoption. Surprisingly little research has been conducted to test the relationship between MfDR and improved development results. However, the evidence presented in this article supports the view that applying the principles of MfDR can bring about significant improvements in project, program and policy performance. Paradoxically though, those countries that need the benefits the most are the least likely to be able to apply whole-of-government MfDR. Notwithstanding this, adoption where the conditions are favorable offers hope for better results from projects, programs and policies in the future.
Adding Value through Community Participation: Evaluating Capacity in Community Health Committees
Authors: Entwistle Phil
Published: January 2018

This article describes the procedure involved in evaluating the effectiveness of the capacity-building process within the Community Health Committees (CHCs) established by the Sunrise Health Service Aboriginal Corporation (SHSAC) as part of their service governance structure. SHSAC itself was set up as the initial stage of the rollout of the Katherine East Coordinated Care Trial in the Northern Territory. A purposive sample of three CHCs was selected in consultation with SHSAC for the evaluation.

The development of an evaluation instrument and the subsequent evaluation of the capacity of the health committees was one part of the overall evaluation of the Katherine East Coordinated Care Trial. The criteria for measuring capacity were developed in close collaboration with members of the CHCs and the SHSAC Community Development Unit. The evaluation framework was based on the Community Capacity Index (Bush, Dower & Mutch 2002), which assesses four areas of competence. This framework was suitably modified in collaboration with the Community Development Unit to reflect the Sunrise Health Service situation. The result was an evaluation that increased knowledge and capacity for the CHCs and Community Development Unit, in addition to evaluating the capacity of the CHCs to function in their designated role.
Assessment, Evaluation and Improvement of University Council Performance
Authors: Armstrong Anona , Unger Zita
Published: January 2018

Higher education is the third-biggest export industry in Australia. Hence, the quality of the governance of educational institutions is of major concern to the Australian Government. Although evaluation of board performance is now used widely in both the private and public sectors, little attention has been given to its application in university contexts. This article addresses this gap. It describes a comparative analysis of various criteria used to conduct board assessments with a framework developed in Australia to guide evaluation of a university council's performance. In particular, it describes the components of the University Council Assessment Questionnaire. This article concludes by making some recommendations about how the use of assessment can contribute to improving a council's performance.
Evaluating Anti-Money Laundering Initiatives: A Country Perspective1
Authors: Barako Dulacha Galgallo , Brown Alistair
Published: January 2018

Kenya faces a number of challenges in countering anti-money laundering (AML) activities and developing appropriate policy measures to avert and contain the adverse socioeconomic consequences of money laundering. Through textual documentation, direct observation and semi-structured interviews, this article outlines the findings of a study about the awareness and attitudes of key Kenyan stakeholder groups towards recent anti-money laundering legislation. Invoking Brown, Tower and Taplin's (2004) typology of financially classified stakeholder groups, this study finds that the proposed anti-money laundering legislation raises a number of diverse views on Kenyan AML initiatives that makes carrying out the results of an evaluation of the AML legislation highly problematic.
Issues concerning the Delivery of Surveys: An Institutional View
Authors: Nair Chenicheri S , Adams Phillip , Mertova Patricie
Published: January 2018

This article presents a case study that investigates several issues concerning the delivery of evaluation surveys in a large research-intensive university in Australia. The article touches on the issue of ‘survey fatigue’, which has been highlighted in relation to the conduct of an increased number and frequency of surveys in tertiary institutions. Despite these warnings, the university described in this case study has recently increased the number and frequency of evaluation surveys, while its survey response rates have been gradually decreasing since the late 1990s.

The case study further outlines two strategies developed in the institution with the aim of engaging students better, subsequently increasing student survey response rates and ultimately improving educational provision in the institution. These strategies were supported further by adopting a more user-friendly platform for submission of online evaluation surveys and by providing students with alternative means of survey submission. The ease and coherence of the evaluation process in the institution was assisted further by the introduction of a new Survey Management System (SMS).

All these strategies and initiatives have proved successful in the institution. Therefore, the article aims to provide some suggestions for other tertiary institutions dealing with similar issues. At the same time, it highlights the fact that individual institutional circumstances ought to be considered and such strategies need to be planned carefully according to the needs of the individual institution.
Maximising Evaluation Influence in an International Development Donor Agency
Authors: Tennant Stacey
Published: January 2018

This article examines how current theories of evaluation use and evaluation influence can assist an international development donor agency, namely the Australian Agency for International Development (AusAID), maximise the outcomes and track the consequences of evaluations of its country programs. It considers those theories most relevant to AusAID's evaluation interests and contexts, that is: instrumental use; conceptual (or enlightenment) use; process use; and evaluation influence. It finds that evaluation influence has much to offer in terms of linking evaluation outcomes to the achievement of program outcomes. By considering the mechanisms that can mediate evaluation influence, the framework underpinning evaluation influence theory builds on many of the central tenets identified in evaluation use theories and encourages the consideration of an evaluation as an intervention in itself. The principal benefit of doing so is to understand when, and why, some use of evaluation occurs and when and why it does not.

Evaluation influence theory holds the potential for AusAID to maximise its evaluation efforts and strengthen its country programs. The systematic consideration of which mechanisms can influence the intermediate and long-term outcomes that lead to social betterment is the key to realising this potential. Empirical research into these mechanisms could both assist and be assisted by AusAID embracing an approach of valuing evaluations as an intervention in themselves.
Maximising Participation in International Community-level Project Evaluation: A Strength-based Approach
Authors: Donnelly John
Published: January 2018

The evaluation of international aid-funded development projects implemented primarily by NGOs usually relies heavily on input from community members. However, the evaluation design and conduct generally does not provide the time, or the opportunity, for the participation of all community members. Informants are often selected according to a statistical sampling method or they are the ‘leaders’ of the community or groups within the community.

In addition, methods used for data collection, such as questionnaires and surveys, can be intimidating to people who are illiterate or who have limited literacy skills, thus alienating large numbers of potential community stakeholders further. The community can be alienated from the evaluation process even more when the data collected from the community, or its representatives, at each evaluation site, is taken away for collation and analysis and is rarely seen again by those who provided it.

This article demonstrates that the situation outlined above need not be the case and that evaluations can be both inclusive and informing for all stakeholders, including donors/commissioners, implementing agencies and the community. The article also contends that this approach to project evaluation, as well as being inclusive and transparent, is also empowering to communities and enhances the sustainability of project outcomes.
Reflections on the Use of a Māori Conceptual Framework to Evaluate Complex Health Policy: The Case of New Zealand's Healthy Eating, Healthy Action Strategy Evaluation
Authors: Boulton Amohia , Kingi Te Kani
Published: January 2018

Obesity among Māori is of growing concern in New Zealand with rates for Māori being one-and-a-half times that of the total population. The New Zealand government's Healthy Eating—Healthy Action: Oranga Kai—Oranga Pumau (HEHA) Strategy has, in the past seven years, been a significant policy mechanism through which the health sector has been able to address the increasing prevalence of obesity and associated chronic diseases.
Negotiating and Planning an Evaluation: An Example from a Youth Services Work for the Dole Program Activity
Authors: Scott Theresa L , Darlington Yvonne
Published: January 2018

This article describes the negotiating and planning of an evaluation proposal for a client, a job network agency situated in Northern New South Wales. The evaluand is a training-based Work for the Dole program activity targeted at unemployed youth and overseen by a not-for-profit community organisation.

The article begins by examining the social, political and historical context in which the Work for the Dole program is embedded, and considers the potential impact of context on the evaluation process and utilisation of the results. The evaluation proposal itself covers negotiation, ethical issues, key evaluation questions and methodology. The last section of the article provides a critical review of the proposed plan and discusses potential process issues.

Throughout, it is emphasised that developing an understanding of the competing values of the many stakeholders involved is fundamental to successful negotiation and planning with all those involved in the evaluation process.
The Use of Appreciative Inquiry in Organisational Change
Authors: Friedman Ruth
Published: January 2018

The introduction of organisational change presents a number of challenges for managers. These can include staff perception of the power of managers, the need to achieve measurable outcomes from the change process, and the tension between using outside experts and internal knowledge. Participatory evaluation methods are suitable for organisational change as they can be used at various stages of program development (Owen 2006, p. 234). They focus on questions such as: ‘What is this program trying to achieve? How could delivery be changed to make it more effective? How could this organisation be changed to make it more effective?’ (Owen 2006, p. 218). This article presents the key features of one participatory method, the Appreciative Inquiry (AI) approach, with particular emphasis on its positive approach to organisational change.
Goal Attainment Scaling in Early Childhood Intervention
Authors: Sharp Colin , Read Phillip
Published: January 2018

One of the problems about evaluations of outcomes of therapeutic services is the need for quantitative and/or qualitative measurement of attainment of intended goals that is sensitive to small changes over short periods. Goal Attainment Scaling (GAS) was developed by Dr Tom Kiresuk and colleagues (in the United States) in the 1960s to address this issue. While there are many variations and applications of GAS in education and community services, there is little available literature on its application in the field of therapeutic interventions with visually impaired young children.

The project described in this article entailed research on the efficacy of GAS in an evaluation of family-centred early intervention programs in an organisation focusing on young children with a vision impairment. It emerged that the preferred version of GAS was different in early intervention from the main applications of this instrument. In addition to the initial focus on GAS, was the need to develop a complementary global assessment measure.

A repeated measures comparison of three intervention groups of visually impaired children showed a statistically significant effect, indicating that GAS was sufficiently clinically sensitive to detect small changes in individuals and small groups over several weeks of intervention and observation. Meanwhile, the qualitative data revealed a positive appreciation by all stakeholders of the opportunity for parents to become involved in goal setting and evaluation of their child's program.
Kaupapa Māori Theory-based Evaluation
Authors: Kerr Sandy
Published: January 2018

In an environment where Māori approaches to evaluation are developing quickly, with ever-widening influence, this article is an attempt to capture the theoretical roots of Kaupapa1 Māori evaluation approaches. From a range of Kaupapa Māori theorists, six principles are drawn and their relevance to evaluation theory and practice is discussed. These principles are then mapped to major movements in evaluation theory, illustrating how Kaupapa Māori theory-based evaluation, arising as a unique praxis within the context of Aotearoa2 New Zealand, has strong alignment with international developments.
Editorial
Authors: Harvey Graeme
Published: January 2018

For this edition we start by acknowledging the contribution of Associate Professor Rosalind Hurworth as Editor of the Evaluation Journal of Australasia (EJA) who passed away in December 2012 after a short struggle with ovarian cancer. Ros's contribution as Editor of EJA has been a longstanding one of over 10 years and during that period her strong stewardship has helped to build the professional standing and quality of the journal. This edition, which Ros continued to assist with during her illness, is dedicated to her.
Using Root Cause Analysis for Evaluating Program Improvement
Authors: Coşkun Rebekah , Akande Abigail , Renger Ralph
Published: January 2018

A common evaluation purpose is to determine whether a policy or program was implemented as intended: referred to as formative evaluation, process evaluation, or evaluating program improvement. A well-designed formative evaluation is important in: detecting program drift; providing timely feedback to program staff to make cost-saving mid-course corrections; reassuring the sponsor that quality assurance measures are implemented to protect investments; and interpreting impact/outcome evaluation. A formative evaluation should not just gather data on deviations from an anticipated course of action, but provide recommendations for improvement. Current methods for program improvement vary in their ability to solicit targeted recommendations. Root cause analysis (RCA) is a well-established, robust methodology used in a variety of disciplines. RCA has been primarily used by evaluators operating from a theory-driven orientation to evaluate the merit and worth of a program or policy. Surprisingly, a review of the literature suggests that RCA's utility as a program improvement tool has remained largely unrecognised in evaluation. This article illustrates the application of RCA in evaluating program improvement. The conditions under which RCA might be preferred over other formative evaluation methods are discussed.
Creative Clinical Data Mining: Glimpsing the Engagement of Families with an Integrated Service System
Authors: Earles Wendy
Published: January 2018

This article illustrates the potential of clinical data mining (CDM) for exploring families' engagement with integrated early intervention and prevention (EIP) service systems. New approaches to EIP under the Communities for Children program seek to achieve systemic change towards more integrated service provision in an area. To date, program monitoring has focused on contacts by families with individual services, not necessarily engagement across an emerging system. As part of an evaluative capacity-building (ECB) process, practitioners asked the question: What can we learn about users' engagement with the service system from existing records of use? The approach taken drew from CDM methodology. EIP does not usually involve significant user database systems as the work entails community development methodologies and users can be variously the community, groups, families and individuals. A partial dataset, from routinely recorded contacts, was available for one area. The dataset relied on a family number system created as a membership club for consciousness-raising on child-friendly communities. The de-identified dataset was mined for ‘connection’ with, and ‘movement’ within, the service system. There was evidence beyond the intuitive knowing of practitioners that the contractual obligation to engage families and children with services across a system was achieved. CDM provided a means to demonstrate this within a broader evaluative capacity-building process.
Reframing Evaluation of Learning and Teaching: An Approach to Change
Authors: Alderman Lyn , Towers Stephen , Bannah Sylvia , Phan Le Hoa
Published: January 2018

This article is the second in a series1 of three that describes the journey on which the Queensland University of Technology (QUT) embarked to redevelop our evaluation practices in the interest of improving the quality of learning and teaching. The article provides insights into: how the previous evaluation system operated and why it needed to change; the emergence and early development of the new project; and formulation of a conceptual framework identifying key dimensions of evaluation. It then compares the draft framework with other conceptualisations and models of evaluation identified in the literature to determine its validity and suitability for supporting QUT's plans for the future. Given that the project followed Seldin's evidence-based, practice-led process, the findings are presented in the belief that QUT's experience is broadly applicable to other institutions that may be contemplating change in relation to the evaluation of learning and teaching.
Editorial
Authors: Alderman Lyn , Clinton Janet , Harvey Graeme
Published: January 2018

Welcome all members and readers to the second online issue of the Evaluation Journal of Australasia (EJA) for 2014. EJA is a flagship initiative of the Australasian Evaluation Society (AES) whose mission is to build capacity in evaluators across our region. EJA plays an essential part in this task and from 2015 will be published four times a year. At this time, the Editors would like to acknowledge the fantastic work that members and others contribute to the publication of EJA as well as the annual conference and national workshops, which work in a symbiotic relationship to advocate, disseminate and recognise good theory and practice in evaluation.
Learning from Evaluations of School–Family Strengthening Programs: Lessons for All
Authors: Guenther John , Galbraith Mal
Published: January 2018

For more than 10 years, a program called Families And Schools Together (FAST) has been run in schools across the Northern Territory. These programs have always had an evaluation component built in. However, over time, the evaluations have changed. Initially, they were summative, built around a quantitative psychometric tool (with a positivist research paradigm). The intent of the summative evaluation was to demonstrate the effectiveness of the program. However, as the program was rolled out in remote contexts, the need for adaptation was recognised. Changes were made, but it was soon recognised that other outcomes, not captured or explored in the methodology, were emerging. After six years of working together, the evaluator and the program manager felt that it was time to explore effectiveness in a different way with a more qualitative evaluation process (based on naturalistic and pragmatic paradigms).

This article explores the six-year learning journey through evaluation that the evaluator and program manager have undertaken. It describes tensions between the need for reliable and generalisable objective quantitative data and the need for authentic and credible data based on participant experience.
Editorial
Authors: Alderman Lyn , Clinton Janet
Published: January 2018

‘Evaluation is a profession that serves all other professions and draws from a full range of disciplines’ (Coryn & Stufflebeam 2014, p. 40). This idea that evaluation is pervasive is an interesting one, and if correct, makes evaluation a discipline that is difficult to define and one whose power is difficult to harness. Stufflebeam and Coryn also argue that society and most of its institutions require formal and systematic evaluation to function successfully. Of course, evaluation needs to be rigorous and make a worthwhile contribution. It is unfortunate that evaluation is not yet embedded or manifested across our society, and often the quality and impact is questionable.
Indicators of buy-in to gauge evaluation success
Authors: Becker Karin L. , Renger Ralph , McPherson Makenzie
Published: January 2018

In order for novice evaluators to gauge the success of their evaluations, the ability to identify and recognize indicators of buy-in is an essential skill. While much is written in the evaluation literature about strategies to build buy-in, the indicators of buy-in are largely overlooked. This article shares strategies used for building buy-in from a multi-state evaluation of cardiac care. Indicators of the success of these strategies are then shared with an emphasis on simple, unobtrusive, robust measures that novice evaluators can use to assess stakeholder engagement and responsiveness. Four indicators of buy-in include: timeliness to evaluation team requests; quality and quantity of feedback received; interaction with decision-makers; and investment of in-kind contributions. Awareness of these indicators can boost confidence and authority for students who are new to the field of evaluation. A tracking tool is also showcased which serves as a systematic, responsive prompt to monitor buy-in. The value of the tracking tool is realized in its ability to alert evaluators to potential problems for which corrective actions can be made, assist evaluators in creating realistic expectations, and enhance confidence in evaluation proceedings.
Cost-benefit analysis explained
Authors: Liddle John , Wright Matt , Koop Bronte
Published: January 2018

In every evaluation context, the question of worth underpins the motivation or role of the evaluation, even if it is not explicitly stated.

Often, this question cannot be answered fully, simply because people or parties may make varying subjective judgments about the value and likelihood of different outcomes—it is at this juncture that cost–benefit analysis (CBA) can be of use. For evaluations where most significant inputs and outcomes can be valued in monetary terms, CBA provides a systematic methodology for reaching conclusions on ‘worth’.

CBA is best used in conjunction with traditional evaluation approaches rather than as a replacement approach. The use of CBA can help an evaluator to be more confident in stating the evaluation findings, particularly in social policy settings where the use and importance of CBA is growing.

This article aims to help evaluators learn more about CBA by demonstrating its use to analyse an intervention designed to help young people disengaged from learning.
Editorial
Authors: Alderman Lyn , Clinton Janet
Published: January 2018

A common topic for discussion amongst those engaged in evaluation including commissioners, donors, providers, evaluators, and participants is—'what is evaluation?’ However, an answer to this question often requires more than just a definition, rather, an appropriate response should consider what evaluation does, how evaluation works, and most importantly, why we need evaluation.
Evaluation of Pacific cancer care navigation patient facilitation and support service
Authors: Tanuvasa Ausaga Faasalele , Neale Jenny
Published: January 2018

From 2010, a not-for-profit community services’ provider has delivered a Pacific cancer navigation patient facilitation and support service (hereafter ‘the Service’) for Pacific peoples residing in an area of West Auckland, New Zealand. The Service was evaluated in 2013.

As part of the evaluation, both group and individual interviews were held with sixteen cancer patients, eight family members and seven key informants; in conjunction, observations and informal discussions were held with two navigators. Service personnel were closely involved from the inception of the evaluation, and this practice will be discussed as part of building an evaluation culture in the organisation.

There was general agreement that the implementation of the facilitation and support Service had proceeded well, this was characterised by effective engagement with health professionals; the establishment of a support group; the provision of transport; assistance in dealings with Work and Income New Zealand (WINZ); and the provision of timely and accurate information to assist patients and their families. From this common consensus, suggestions were made around increasing and ensuring ongoing funding to sustain the supporting roles of navigators and the services offered.

Other invaluable features of the Service included the ability of navigators to translate from English to Samoan, and from medical to lay language, as well as the provision of the Service beyond central metro areas.