bluuebunny/crossref_metadata_2025 huggingface dataset:
Improving Structural Policy Under Conditions of Hybrid Governance:
Authors: Lang Jochen
Published: July 2003

As compared to national policies, European structural policy is evaluated extensively. Although the last decade shows considerable success in institutionalizing evaluation processes, the effectiveness of evaluations in terms of the use of their results for policy development could be improved. The existing problems stem from a tension generated by the hybrid governance of structural funds implementation, i.e. the mixture of management by results and bureaucratic rule steering. Against this background, institutional and alternative means for enabling and ensuring the effectiveness of evaluation of the structural funds are discussed. The analysis shows that such an input of evaluation findings into the processes of policy development and improvement can be supported by regulatory requirements. However, they should be complemented by incentives that address the needs of de-central implementing actors.
The Merits of Mixing Methods in Evaluation
Authors: Greene Jennifer C. , Benjamin Lehn , Goodyear Leslie
Published: July 2003

Evaluators are challenged to understand human behavior in all of its natural complexity and individuality. Our work is conducted in natural settings, where history and context matter, where human behavior traces complex patterns of influence and relationship, where what is meaningful to those in the setting is both phenomenological and structural, arising from both lived experiences and the societal institutions that frame and shape those experiences. Engaging this complexity requires not a privileging of just one way of knowing and valuing, but rather a marshalling of all of our ways of understanding in a framework that honors diversity and respects difference. This framework is advanced in this article as a mixed-method way of thinking. A presentation of key concepts in the framework is followed by case examples from the US.
What is the Policy Problem?
Authors: Hanberger Anders
Published: July 2003

Because when a policy process starts, nobody knows what line of action will eventually be implemented, policy evaluation has to continuously examine the content of different policy components. In order to understand and explain public policy, different stakeholders' perceptions of the policy problem need to be scrutinized. A policy evaluation should also facilitate the interpretation of policy in a broader context. What values and order does the policy or programme promote? Using an open evaluation framework and a mix of criteria can facilitate a broader interpretation of the policy process. In this article, problems undertaking policy evaluation are discussed in relation to a Swedish medical informatics programme.
The Impact of Development Co-operation in Conflict Situations
Authors: Klingebiel Stephan
Published: July 2003

Within a debate about the contribution that development co-operation (DC) can make to stable and peaceful development in conflict situations, development agencies are trying to gain a better understanding of their options. This article deals with the experience and findings of German DC in six countries.

The direct influence of DC on conflict situations is limited. Conflict recognition and the flow of information are important and this has been sometimes deficient or even highly problematic. DC actors are often insufficiently aware of conflict situations and there are no real 'conflict strategies'. German DC pursued very different strategies in the countries studied: sometimes German governmental DC tried to act independently of the actual or potential conflict, whilst on other occasions efforts were made to help prevent crises or resolve conflicts. The six country studies differ in their conclusions on the overall impact of German DC on potential and actual conflict situations: some positive and some negative. All the studies identify opposing tendencies (ambivalence); in each case German DC may exacerbate potential or actual conflict situations as well as make positive contributions.
Considerations in the Design of a Mixed-Method Cluster Evaluation of a Community Programme for 'At-Risk' Young People
Authors: Lucke Jayne C. , Donald Maria , Dower Jo , Raphael Beverley
Published: July 2003

This article discusses the design of a comprehensive evaluation of a community development programme for young people 'at-risk' of self-harming behaviour. It outlines considerations in the design of the evaluation and focuses on the complexities and difficulties associated with the evaluation of a community development programme. The challenge was to fulfil the needs of the funding body for a broad, outcome-focused evaluation while remaining close enough to the programme to accurately represent its activities and potential effects at a community level. Specifically, the strengths and limitations of a mixed-method evaluation plan are discussed with recommendations for future evaluation practice.
Opening Thoughts
Authors: Abma Tineke A.
Published: July 2003

Today many disciplines including organization and management, public administration, social psychology, educational research and political science are becoming interested in dialogue. Evaluators are only just beginning to recognize how valuable dialogue can be for their work. Dialogue may have the potential to improve the democratic quality of decision making and judgement in postmodern societies and enhance (mutual) understanding between people. In this special issue we aim to disentangle some of the complexities, meanings and aims of dialogue and dialogical evaluation. A number of evaluation scholars and practitioners will discuss different notions of the concept and its relevance for evaluation practice. They will also deal with some practical considerations: what are the essential characteristics of evaluation dialogue; who should participate and what should the role and responsibility of evaluators be. We will also identify areas where we think caution should be exercised. Examples of dialogical evaluation projects in various policy settings and countries – including Sweden, Holland and the USA – together with comments by other scholars are meant to stimulate an exchange of ideas.
Dialogue as a Democratizing Evaluation Method
Authors: Ryan Katherine , Destefano Lizanne
Published: July 2003

Looking at dialogue from the intersection of theory and practice, we present our view of dialogue as a democratizing practice. Using a constructivist grounded theory approach, we compared and contrasted our notions against data we collected. Our analyses of the two case studies suggest power relations, the identity of the evaluator, and the skills needed to facilitate dialogue, are problematic, including virtual dialogue. The evaluator needs to be more than an expert. Knowing the theory of dialogue in evaluation was not sufficient to understand these two cases. Instead we saw in each instance, the evaluator drew on tacit skills to keep the dialogue going. These skills were developed in the ‘thick of practice’. But more importantly, these cases show how the evaluator must privilege the dialogical process to keep it going.
The Missing Link: Processing Variation in Dialogical Evaluation
Authors: Edelenbos Jurian , Van Eeten Michel
Published: July 2003

This article is a comment on ‘Dialogue as a Democratizing Evaluation Method’ by Katherine Ryan and Lizanne DeStefano. Their insightful paper prompted us to reflect on several difficult issues at the core of the ‘dialogical paradigm’ of evaluation. Our reflections focus on four issues: (1) dialogue as a democratizing process, (2) representativeness and participation, (3) the missing link between variation and selection in dialogical evaluation, and (4) the role of the evaluator. Our main argument is that dialogical democracy is excellent for raising awareness and a rich variation of views and ideas, but it does not help in retaining that variation for subsequent policy processes. Research into dialogical and participatory democracy must pay more attention to this missing link. The paradox that needs to be addressed, we argue, is that variation can be retained only when dialogical evaluation also supports selection and condensation processes. The article concludes with an outline of the evaluator's role and tools in providing the missing link.
Dialogue in Evaluation: A Hermeneutic Perspective
Authors: Widdershoven Guy A. M.
Published: July 2003

Dialogue is crucial to evaluation. Through dialogue, people are invited to develop new, shared ways of seeing and acting. From a hermeneutic perspective, dialogue implies consensus, based upon shared meanings and oriented towards reaching agreement. Dialogue also implies disagreement, in the form of a conflict between different, rival stories. The role of the evaluator is to enter into dialogue with people participating in practice. It is also to stimulate a dialogue between participants. This requires support for marginalized voices, but also attention to the voices of those who are in power. Only if the evaluator tries to understand the values endorsed in a practice and is prepared to listen to the stories of all parties involved, can evaluation become a practice to which all participants contribute.
Great Expectations: Can Social Science Evaluate New Labour's Policies?
Authors: Walker Robert
Published: July 2003

The UK Labour government has offered a hand of friendship to the academic research community and to social science in particular. It has rejected anti-intellectualism in government and sought to promote policy made on the basis of evidence. In particular, it has introduced piloting of policies prior to national implementation. This article draws on the experience of four of the largest pilot evaluations undertaken to date, and considers the extent to which the research community has been able to contribute effectively to determining whether Labour's policies work. Five sets of inhibitors are identified and discussed relating to the policy-making process, the characteristics of particular policies, the mechanisms for commissioning and also designing evaluations and the orientation of the social science community. The article concludes that success to date has been severely limited and that to continue evaluating new policies as at present is unsustainable. Evidence-based policy cannot be limited to the use of evidence; policy making has to accommodate the production of evidence. An alternative scenario is presented that might facilitate such an accommodation.
From Programme Theory to Constructivism
Authors: Dahler-Larsen Peter
Published: July 2003

Theory-based evaluation utilizes terms such as ‘moderators’ and ‘CMO configurations’ to describe how the validity of a programme theory depends on the context in which it operates. From a constructivist perspective, however, contexts are influenced by ongoing social (re)construction. The truth value of a programme theory is contingent upon this construction. This helps explain why tests of programme theories provided by theory-based evaluators are often unconvincing to programme advocates. A typology of ‘magic’, ‘tragic’ and ‘competing’ programmes, each of which influence their contexts in different ways, is described and the implications for evaluation discussed.
Devising the Evaluation Strategy for a Mental Health Training Programme
Authors: Bailey Di , Littlechild Rosemary
Published: July 2003

What follows is an evaluation of the definitions and methods of evaluation that can be developed into an evaluation strategy. The strategy is derived from the use of a case-study research methodology applied to a UK postgraduate multi-disciplinary Programme in Community Mental Health. This chosen methodology is coupled with a model adapted from the training literature in an attempt to critically review how the strategy might be applied in practice.
Assessing USAID's Assistance for Democratic Development
Authors: Mcmahon Edward R.
Published: July 2003

The focus of this article is on efforts undertaken within the United States government, especially the US Agency for International Development, to develop a coherent approach to determining the effectiveness of its democracy support programming. The main four priority areas are to support: civil society development; competitive political processes; the rule of law; and governance.

The article begins by asking the question of why this issue is important. It then provides context, and briefly assesses varying perspectives and motivations of different stakeholders. It examines and critiques current approaches being taken to ‘measure performance’ and concludes by suggesting some potentially useful policy and research directions. These include instituting structural reforms to enhance the independence of the evaluation process, disaggregating approaches to assessing the impact of programming, focussing on priority countries, and expanding use of integrative methodologies.
Investigating Consumer Perspectives on Evaluating Health Technologies
Authors: Oliver Sandy , Milne Ruairidh , Bradburn Jane , Buchanan Phyll , Kerridge Lynn , Walley Tom , Gabbay John
Published: July 2003

In a short pilot study we systematically involved healthcare consumers in identifying research topics, and prioritizing, commissioning and reporting evaluations of health technologies. We analysed records of consumers' suggestions for their content and, wherever possible, compared them with suggestions from academics and staff of the UK National Health Service. Consumers tended to highlight issues about patients' views, social contexts, information and support needs, long-term outcomes, and dissemination of research findings to consumers. These issues were usually not addressed by professionals who tended to focus more on technical and economic aspects of the work.

Consumers made unique contributions and brought fresh ideas and skills, which were readily incorporated, into the programme. Their direct involvement promises to better satisfy the information needs of those who use the NHS, and similar approaches may be applied to evaluating other public services.
Evaluating Healthcare Reform: The Challenge of Evaluating Changing Policy Environments
Authors: Van Eyk Helen , Baum Fran , Blandford John
Published: July 2003

Evaluators in the health sector struggle to develop effective mechanisms with which to evaluate healthcare reform programmes. Evaluation of these programmes is difficult because they are entangled in complex, inherently political processes and often shift away from their original aims and objectives as a result of policy changes within the health system. This article describes the evaluation of a healthcare reform process occurring in South Australia and discusses some of the methodological challenges encountered in evaluating a case study of continuous change with shifting policy objectives. It describes how the evaluators have attempted to address these challenges through a flexible and dynamic action-research approach. The article concludes by discussing the role of the evaluator, as intelligent observer, in reflecting on and documenting change in the highly political and complex field of healthcare reform.
Evaluating New Regional Policies
Authors: Diez Maria-Angeles
Published: July 2003

Regional development theories and policies have now moved into new arenas. Today, the competitive advantages of regions put a premium on social capital, hence, the significant role played by localized learning and the construction of knowledge networks and institutional capacity. These are necessary and complementary assets to factors traditionally thought to influence regional development. The evaluation of these new regional policies is an important challenge, but classical evaluation models do not adapt well to the specific characteristics of these policies. As a consequence, it is necessary to seek new evaluation approaches to assess the regional impact of these new policies, understand how they work and why, generate knowledge and, also, mobilize regional communities to act, empowering local agents and enhancing learning capacity.
Solving Methodological Challenges Using a Theory-driven Evaluation in the Study of Complex Patient Care
Authors: Grocott Patricia , Cowley Sarah , Richardson Alison
Published: July 2003

This article draws on two research designs, which were used in a study of the palliative management of malignant wounds, to demonstrate the inherent challenges in producing generalizable knowledge from a complex subject. The designs included quasi-experimentation and a form of theory- driven evaluation. The focus of the study was the performance of dressings to reduce the impact of the wounds on daily life. The convention for evaluating the outcomes of interventions is a controlled study design. This approach is considered the valid way of producing evidence that is generalizable. An experimental design was therefore adopted to conduct a series of n = 1 quasi-experiments on wound dressing performance with qualitative methods to explore individual experiences. Problems with the methods, in particular the failure to capture complex inter-relationships between clinical problems and patient experiences, forced the researcher to abandon the experimental design. An alternative approach to measurement was adopted to maintain the study focus, the TELER® system of treatment evaluation, which includes indicators of patient-centred outcomes of care. The philosophical position of the study was reviewed. Consensus emerged as an unalterable, major in uence on the design and paradigmatic assumptions of the study. A `system of reasoning' was adopted to overcome the inherent relativist position of knowledge derived from this approach. The system abstracts general issues from case study data to construct theoretical explanations that may be consistent with, or challenge, current knowledge. This article is based on the evolution of one particular study. However, it makes a more general contribution to evaluation research by explaining the rationale for a form of theory-driven evaluation that uses evidence, reason and theory to develop generalizable explanations from complex, individual case study data. The methodology is proposed for other complex situations where speci c and rigorous evidence, capable of generalization is needed.
Promoting Evaluation through Collaboration
Authors: O'Sullivan Rita G. , D'Agostino Anne
Published: July 2003

Collaborative evaluation engages key program stakeholders actively in the evaluation process. Distance between external evaluators and program staff is often minimal, based on the assumption that a collaborative stance will strengthen evaluation results and increase utilization of evaluation findings. Convincing empirical evidence to support these assumptions is scarce. This study uses evaluation findings from a county-wide, comprehensive, early childhood education initiative to support the contention that a collaborative approach can substantively improve evaluation findings.
Managing Quality within the Swedish Social Services
Authors: Bergström Maria
Published: July 2003

This research represents a first attempt to measure to what extent Swedish Social Services were familiar with quality assurance processes at the time of the introduction of a new article in the Social Services Act which stipulated that `measures within social services shall be of good quality'. The philosophy of total quality management is drawn upon as a reference point from which to assess the preparedness of the social services to fulfil the requirements of the new law, especially from a managerial point of view. The social service management's ability to assess the quality of organizational performance, the use of evaluation and perceived obstacles to evaluation practice, are explored and tested against evaluation activities undertaken within the social services.
On the Principles of Comparative Evaluation
Authors: Vartiainen Pirkko
Published: July 2003

This article discusses both the methodological and theoretical principles involved in comparative evaluation. The term comparative evaluation refers to research in which an evaluation and the findings of the evaluation process are set in a comparative framework. The principles on which the comparative evaluation process should be based, however, are not always clear. This article analyses the content of these principles and begins with a discussion of the purpose and function of comparative evaluation. This is followed by an analysis of four different principles common to the method of comparative evaluation: selection of the evaluation object; the level of comparison; conceptual comprehension; and the analysis of the findings of an evaluation. When these principles are applied to the comparative evaluation study, the study most likely produces comparable and reliable information for organizational development work.
A Mutual Catalytic Model of Formative Evaluation
Authors: Chacón-Moscoso Salvador , Anguera-Argilaga M. Teresa , Pérez-Gil José Antonio , Holgado-Tello F. Pablo
Published: July 2003

This article describes the `mutual catalytic model' of programme design and formative evaluation. The model presumes two conditions under which it can be useful. Firstly, those responsible for a programme at the point of service delivery should participate equally in a programme's design and in its evaluation. This collaboration will: 1) increase the clarity of programme objectives and the level of consensus regarding them; 2) improve the evaluation design and process by connecting the evaluation with the realities at the point of service delivery; and 3) increase the likelihood that staff will use the results of the evaluation to improve programme design. Secondly, evaluators are required relatively frequently to facilitate the review of programme objectives and procedures, since reviews of this kind rarely take place without an extra stimulus and sometimes need a practical hand to guide them. In the mutual catalytic model of formative evaluation, local practitioners help to specify programme goals and improve evaluation, while evaluators help to elaborate programme goals and evaluation needs. This model is exemplified using experiences with homeless-child care centres in Seville in Spain.
Use of Evaluations and the Evaluation of their Use
Authors: Feinstein Osvald N.
Published: July 2003

All evaluations have a cost but not necessarily a value. Their value does not depend on their cost but on their use, and this article discusses factors affecting the use of evaluations. These factors could be taken into account in order to increase and improve the use made of evaluations and, consequently, their value. Two key issues (lags and the attribution problem) for the evaluation of the use of evaluations are discussed and a `possibilist' approach to evaluation use is presented.
Pro-active Evaluators: Enabling Research to Be Useful, Usable and Used
Authors: Boaz Annette , Hayden Carol
Published: July 2003

Evidence-based policy making presents challenges for researchers as the pragmatic imperative of delivering `the right information, at the right time, for the right people' can appear to compromise traditional academic roles and responsibilities. Using the recent evaluation of the UK Cabinet Office-led `Better Government for Older People' programme as a case study, we discuss attempts to meet these challenges in practice. Our experience confirms that there has been a shift from the assumption of rational models (i.e. that evidence will be used as part of a linear planning process) to a more `realistic' perspective in which evidence from research requires advocates to promote its potential contribution. In turn, this suggests that it is in the interest of the research community to try to understand the policy-making process, and so design, undertake and disseminate their research in a way that maximizes the likelihood of its use in actively contributing to policy development.
A Theoretical Framework and Grid for Analysis of Programme-evaluation Practices
Authors: Demarteau Michel
Published: July 2003

This article proposes a theoretical framework and a grid to unify models and to describe programme evaluation. It identifies three components in the evaluation process: the information-collecting process; the value-judgement product; and the use of the latter in decision making. It defines three differentiation axes (or dimensions): the social enunciation, the external or internal position of actors, and the formal or informal character of the evaluation components. The relevance for the practitioner and the researcher is discussed and briefly illustrated with two examples from the health promotion sector: a project selection process and a retrospective study of programme-evaluation practices.
Policies and Programmes to Address Disadvantage among Young People: Issues for Evaluation
Authors: Pitcher Jane
Published: July 2003

This article discusses some of the issues facing young people experiencing disadvantage in the labour market and associated implications for evaluation of programmes to address disadvantage. Drawing on the experiences of an evaluation study undertaken in an English county, the article explores alternative approaches to evaluation and the need for additional indicators to accompany quantitative outputs such as numbers into training or employment. The article is structured as follows: introduction to issues facing young people in the labour market; comment on evaluation of social programmes; a case study example of a local evaluation study; and discussion of evaluation issues arising from the study.
Evaluation as a Tool in the Development of Social Work Discourse
Authors: Cheetham Juliet , Mullen Edward J. , Soydan Haluk , Tengvald Karin
Published: July 2003

This article reports reflections from an international conference on evaluation as a tool in the development of social work discourse. The conference was organized with the purpose of presenting the state of the art to the Swedish social work research community. In this article, a number of major issues that were discussed at the conference are reported, including evaluation paradigms and preoccupations, the relationship between the social research community and practitioners, and some of the methods of evaluation research. The authors assess their understanding of the outcomes of the conference from national perspectives, and attempt to discuss similarities and differences between Sweden, the United Kingdom and the United States.
New Developments and Challenges of Health Status Measurement for Outcomes Evaluation
Authors: Rodriguez Eunice , Bowen Kathryn A.
Published: July 2003

Increasing interest is now being directed towards the assessment of patient outcomes, both in evaluating medical interventions and in endeavors to increase the quality of care patients receive. While it is argued that outcomes measurement has the potential to positively influence medical care, when carried out without due consideration of its limitations and weaknesses it could lead to data that are misleading. In light of health outcomes data being viewed as central to monitoring and evaluation, we believe that evaluators have the potential to bring expertise and valuable experience into this rapidly evolving field of inquiry. The following article will discuss recent developments in the field of health status measurement for outcomes evaluation and analyse some of the most important challenges in the field.
Issues in the Evaluation of Innovation and Technology Policy
Authors: Georghiou Luke
Published: July 2003

The efforts by policymakers to help firms become more innovative have created a strong desire to know which policies work. This has placed high expectations upon evaluation. The development of evaluation in this sphere has mirrored the evolution of policy, beginning with a focus on large-scale collaborative technology programmes and gradually moving towards an examination of measures seeking to enhance the environment for innovation. Evaluations in this field may be divided into those assessing direct and indirect financial support for research and design and those addressing opportunity-enhancing innovation policies such as technology transfer networks. Several tensions and challenges for evaluation are identified, including the lack of comparative and systemic innovation policy evaluations. Evaluation needs to follow the same adaptive learning approach as innovation policy itself.
Renomo—a Design Tool for Evaluations
Authors: Beywl Wolfgang , Potter Philip
Published: July 2003

Evaluators designing evaluation studies that are responsive to the interests of the different stakeholding audiences have to cope with conflicts and constraints from the very beginning of their work. This article describes a tool for a stakeholder-orientated evaluation approach which combines Nominal Group Techniques and the Moderation Method. The tool is presented in a detailed scenario for an evaluation design context. The paper concludes with an assessment of the utility and areas of application of the RENOMO instrument.
Caring Communities, Paradigm Polemics, Design Debates
Authors: Pawson Ray , Tilley Nick
Published: July 2003

In a previous issue of this journal (Vol. 3, no. 2) David Farrington introduced—‘in the interests of stimulating discussions’—an outline research design for Evaluating a Community Crime Prevention Program. This article takes up the challenge, offers a critique of his ‘quasi-experimental’ approach and insists that such programs are better evaluated using designs based on ‘scientific realist’ principles. The paper also draws attention to benefits of a ‘theories of change’ approach to evaluation, pioneered in the US in conjunction with such comprehensive community initiatives.
Evaluation in Italy
Authors: Stame Nicoletta
Published: July 2003

The management of public affairs in Italy today is characterized by a demand for transparency and accountability, and by a desire to learn to manage the instruments of intervention today available to society. This emphasizes the need to evaluate the effectiveness and the results of policies and programs, which is not, however, matched by adequate knowledge of the methods or by sufficiently creative approaches. This social deficit in evaluation has both a cultural and an organizational origin.
Designing Investigative Evaluations
Authors: Smith Nick L.
Published: July 2003

The predominant purpose of most investigative social program evaluations has been the amelioration of social problems; however, increasing client demands, shifting evaluator roles and methodological innovations have all influenced evaluators to become increasingly responsive to client interests. Concerns have been raised that evaluations cannot simultaneously emphasize both problem solutions and client responsiveness, and that to emphasize client interests is to necessarily undermine the quality of information needed to solve social problems. Three case studies are reported here to illustrate alternative ways by which evaluation practitioners have successfully balanced client responsiveness with problem solution: evaluations of a medical school curriculum in Hawaii, an industrial technology transfer program in Norway, and a fisheries employment-support program in Newfoundland.
Moderation of Policy-Making?
Authors: Kuhlmann Stefan
Published: July 2003

In the field of science and technology policies, for the most part, evaluation procedures are utilized as a way of measuring the scientific and technological quality or the socio-economic impacts of publicly funded research. Beyond this practice, could evaluation procedures be used as a medium for the ‘moderation’ of struggles, controversies and negotiations in the science and technology policy arena? The present article addresses this question using the German evaluation practice—moulded by a relatively high degree of institutional differentiation and autonomy of the major policy actors—as a background. After some theoretical considerations, a case study is presented illustrating the ‘moderation approach’: a multi-annual monitoring evaluation of eight newly created, publicly funded interdisciplinary clinical research centres at German university hospitals.
Spanish Evaluation Practice Versus Program Evaluation Theory
Authors: Ballart Xavier
Published: July 2003

In this article, a contrast is made between evaluation theory and practice. For this purpose, a typology is constructed that is based on seven authors and nine dimensions, and presents examples of evaluations in various policy areas in Spain. It is anticipated that there will be a mismatch between theory and practice. This can be explained by the difficulties government evaluators have finding the data, time and resources to apply the techniques and conduct the types of evaluation that scholars consider academically excellent. It can also be explained by the fact that government evaluators and decision-makers, in a national context where there is no pressure for experimental methods, choose to produce information on a broader set of measures that gives them insights on both program effectiveness and organizational performance, but does not inform them on whether there is a causal relation between government programs and observed effects. From the five examples of the practice of evaluation in Spain, it is observed that practical evaluation can provide a good ‘picture’ of the relative effectiveness and efficiency of government programs. It is also observed that evaluation results fit the type of information which program managers and stakeholders expect, increasing the chances of evaluation studies being effectively used. Finally, for these five examples, the evaluators performed very much as controllers would do in the private sector, where there is no concern for the type of causality academics try to find between program features and the consequences of programs.
Evaluation in Partnership
Authors: Toulemonde Jacques , Fontaine Céline , Laudren Emilé , Ncke Philippevi
Published: July 2003

Co-funded policies are proliferating in the public sector, thus creating the need for evaluations in partnership. In this context, evaluation faces specific difficulties: objectives are multiple and ambiguous, indicators may not be perceived as relevant, data collection is complex, and partners' views legitimately differ. The article describes the evaluation of an urban development policy co-decided and co-funded by the French government and the Poitou-Charentes Region. It demonstrates that it was possible to overcome the difficulties of joint evaluations through a series of technical innovations: a concept map structured the evaluation; scoring sheets were used to construct qualitative impact indicators, data about impacts was collected via 20 case studies, and, finally, a multicriteria-multijudge analysis was used to conclude the evaluation while respecting different points of view. In addition to its operational interest, this experience seeks to make a theoretical contribution to the question of the final synthesis of evaluations.
Evaluating ‘Communities that Care’
Authors: Farrington David P.
Published: July 2003

I am very grateful to the editor for the opportunity to reply to Pawson and Tilley's (1998) critical comments on my proposals (Farrington, 1997) for evaluating Communities that Care (CTC) in the UK. Of course, I agree that it is valuable to compare alternative research designs. However, I think that most of Pawson and Tilley's comments are based on assumptions about the nature of CTC that differ greatly from the assumptions on which my article was based. Briefly, whereas I assumed that CTC was a well-developed technology that was ready to be tested, Pawson and Tilley assumed that it was an iterative, evolving, ill-defined, highly variable procedure that essentially required further exploratory and developmental work. However, it seems to me that these comments apply far more to a ‘scientific realism’ evaluation than to CTC. I should perhaps add at the outset that I have no personal stake in CTC and suggested it mainly because I thought that it was the best-developed vehicle for delivering risk-focused prevention, which I strongly recommended.
A Cultural Revolution in the Universities
Authors: Niklasson Lars
Published: July 2003

Audits were introduced in Sweden in an effort to foster self-regulation by the universities when central regulation was decreased in 1993. It was believed that internal systems of quality assurance were the optimal means for increased quality and efficiency of Swedish universities and university colleges. This paper claims that an unspoken intermediary was to change the internal culture of the universities, from ‘cloisterism’ to ‘new collegiality’. In line with neo-institutionalism, new behaviour was to be stimulated by new institutions. In other words, audits were to change the incentives and increase the rewards for proponents of change. The struggle for change could be understood as a game-like situation, where audits affect pay-offs. The adoption of new means of internal quality assurance is used as a measurement for the spread of the new culture. Rough data from six institutional audits indicate that a ‘cultural revolution’ is taking place and that audits are enhancing it. Further explanations are discussed, as well as possible means of improvement.
The Politics of Evaluation Reconsidered
Authors: Schwartz Robert
Published: July 2003

Recent experience suggests that the effect of political considerations on the incidence of evaluation is not constant across programs. The article attempts to refine the politics of evaluation theory by identifying affective variables when political considerations are more likely to inhibit the conduct of evaluation. Evaluation activity for 63 programs was measured over the course of an 11-year period. Two important refinements emerge: (1) contrary to the theory, program directors demonstrate a sincere interest in the conduct of evaluations. At the same time, interest in program evaluation from senior management, ministers and members of parliament is almost non-existent. (2) Within the context of internally focused evaluations, not all program directors avoid evaluation. Some agencies have, mainly because of professional staff backgrounds and resource availability, a higher proclivity to conduct evaluation. Within agencies, there is a tendency to focus evaluation efforts on large and direct provision programs.
Evaluation Capacity Building
Authors: Khan M. Adil
Published: July 2003

Most developing countries now have monitoring and evaluation systems in place. However, most systems are concerned with the progress of implementation, rather than assessing the social, economic and environmental impacts of projects. Also, there seem to be no systems that assess the impact of policy interventions emerging from recent macro-level measures, such as liberalization, privatization, and the preservation of women's rights. In developing countries, donor agencies have played a role in planning, implementing and financing various socio-economic development programs and projects. In many cases, the outcomes of these interventions do not match the intended objectives. It has been argued that due to the lack of ongoing evaluation many governments fail to learn, in time, the way a project is unfolding and the manner in which it is generating benefits. There are also many who simply do not see the benefits of evaluation and consider it to be a donor-driven activity of no management use. Those donors who do see evaluation as an important tool to improve investment quality are now initiating evaluation capacity building activities. The success of these initiatives seems to have been constrained, among other things, by the lack of a unified approach; inadequate appreciation and analysis of governmental culture; confusion about concepts and methodologies; lack of long-term commitment; and lack of either interest or resources—or both—from the recipient governments. Future evaluation capacity building work will need to make a careful analysis of these constraints and approach the subject with far greater sensitivity and technical knowledge.
Constructivist Capacity Building and Systemic Evaluation of Adult Basic Education and Training in South Africa
Authors: Bhola H. S.
Published: July 2003

What follows is a discourse of coherence: first through an epistemic triangle conjoining three epistemological traditions of systems thinking, dialectical thinking and constructivist thinking; and, second, in a praxeological triangle that unites planning, implementation and evaluation in one single act of reflective action with purpose. Working from within these two mutually embedded triangles of coherence, a discourse on the Adult Basic Education and Training (ABET) initiative of South Africa was developed in necessary and sufficient detail, to capture the calculus of means and ends of ABET as well as the dynamics between its intended objectives and its actual effects. This one general discourse is subsequently used to generate two specific scripts: one on capacity building, and another on evaluation. There are implications here for theory and practice for both program development and program evaluation. Understandings of how the social world works, holistic educational planning, functional specialization without fragmentation, reflective action for implementation, critical management, participative learning, inbuilt evaluation, and thereby economy in the deployment and use of materials and professional resources, all become possible.
Teaching Evaluation Using Evaluability Assessment
Authors: Leviton Laura C. , Collins Charles B. , Laird Beverly L. , Kratt Polly P.
Published: July 2003

A case is made for using the technique of evaluability assessment to introduce students to the field of evaluation. Procedures for directing, instructing, and supervising evaluability assessments in the classroom setting are described, along with techniques for adapting the method to the students' level of training, mix of experiences, and duration of the class. Evaluability assessment is a useful exercise that assists students in applying their research skills to real program needs. Twenty local evaluability assessments, course evaluations, and subsequent contact indicate that this approach is beneficial for student professional development and is a useful service for program stakeholders.
Goal Focused Evaluation
Authors: Peled Einat , Spiro Shimon E.
Published: July 2003

This article revisits a longstanding controversy regarding the place of organizational goals in evaluation. While early writers saw goals as the yardstick against which outcomes are measured, others argued for a goal free evaluation. We propose a Goal Focused Evaluation, which acknowledges the problematic relationship that may exist between a program's declared and operative goals. It assumes that a comprehensive evaluation of a program's effectiveness is often best achieved by identifying both types of goals and possible gaps between them, thus enabling organizational stakeholders to refocus program goals, rechannel resources and, if desired, contract a ‘second stage’ evaluation of the newly reshaped goals. The article examines the processes, benefits and limitations of a Goal Focused approach to the evaluation of social programs, as it was applied in a study of a shelter for homeless youth in Tel Aviv.
Towards an Economics of Evaluation
Authors: Picciotto Robert
Published: July 2003

This article probes the contrasting traditions of economics and evaluation and shows that intensified collaboration would benefit both professions. While evaluators have long been examining the impact of economic advice on public policy, economists have neglected evaluation both as a platform for testing economic policy hypotheses and as a topic for economic research in its own right. To encourage convergence between economics and evaluation, this article sketches the elements of an institutional economics of evaluation, considers the ongoing debate about the proper function of evaluation in society and presents cost benefit concepts and a supply demand framework to scrutinize the value and prospects of evaluation activities. Finally, it identifies performance measurement as a priority area for cooperation between evaluators and economists.
Cultural Codes – Who Holds the Key?
Authors: Hyatt Jenny , Simons Helen
Published: July 2003

This article examines the impact of cultural codes on the conduct and outcomes of evaluation. Cultural codes are defined as symbols and systems of meaning that are relevant to members of a particular culture (or subculture). These codes can be utilized to facilitate communication within the ‘inside group’ and also to obscure the meaning to ‘outside groups’. Drawing on the authors' combined experiences in Central and Eastern Europe, the article explores how cultural codes emerged in these contexts and the challenges they presented to the conduct and utilization of evaluation and to the evaluators themselves. Evaluators must be prepared, it is suggested, to unlock their own internal codes in relation to those of other cultures in order to create the appropriate socio-political relationships that are a prerequisite for learning and change. The article concludes with a discussion of the kind of partnerships and trust it is necessary to establish to underpin a mutual unlocking of codes.
Active Qualitative Evaluation
Authors: Kuipers Herman , Richardson Rudy
Published: July 2003

This article discusses active qualitative evaluation and modern sociotechnical design as parts of an approach for the evaluation of organizational change. ‘Intervention in interactions’ is the most important similarity between these parts when considering the evaluation of organizational change. The two parts are discussed in terms of their theoretical core elements, practical consequences for research and dilemmas. Because, in practice, intervening in a changing organization is always accompanied by some important paradoxes, the different parts of the approach are presented as supplementary in practical evaluation of organizational change. The leitmotif on the basis of the case of the Royal Netherlands Military Academy here is that a paradoxical situation such as intervening in a changing organization needs a paradoxical intervention strategy.
The Evaluation of Course Quality Through a Service Template
Authors: Gilroy Peter , Long Peter , Rangecroft Margaret , Tricker Tony
Published: July 2003

This article reports on initial findings of the Template Project, a project researching ways of evaluating post-graduate courses offered through the medium of distance education. It has taken an evaluation tool from industry and re-designed it so that it can apply to distance education courses. The question as to whether or not a tool initially located in the service industries is appropriate for education is examined as are the alternative paradigms of evaluation. A Popperian approach to evaluation is offered as the paradigm within which the Template Project's approach to evaluation should be located.
Purposes and Use
Authors: Eggers Hellmut W. , Chelimsky Eleanor
Published: July 2003

In Volume 3 (1) we published a keynote speech delivered by Eleanor Chelimsky at the December 1996 Conference of the UK Evaluation Society. Hellmut Eggers, an active member of the evaluation community in Europe, wrote to partly challenge some of Eleanor Chelimsky's arguments. Dr Eggers is no stranger to the problems of making evaluations purposeful and useful: he was for many years responsible for evaluation within the General Development Directorate of the European Commission. We are pleased to publish below both Hellmut Eggers' contribution and Eleanor Chelimsky's response. Readers who wish to take this debate further may want to re-read Eleanor Chelimsky's keynote address published in this journal and also consider a more recent article by the same author in the American Journal of Evaluation 19 (1) 1998.