# Saylor.org's Comparative Politics/Evaluation and Feedback

NOTE: The kind of evaluation and feedback one can expect when implementing public policy is discussed in this paper from the perspective of privatization policy.

# Researching privatisation: some notes about public policy evaluation

by Luciana de Oliveira Miranda Gomes

## ABSTRACT

The objective of this paper is to present a literature review about public policy evaluation, with specific reference to privatisation. The basic research goal - still in process - is to evaluate the purpose and reach of public policy objectives relating to privatisation, particularly in relation to their socio-economic outcomes at regional and community levels. In order to achieve this objective, the author discusses some approaches to privatisation produces in United Kingdom over the last ten years. Indeed, she presents some key concepts that underline her reflections and submits some key methodologies about researching public policy evaluation. Finally, the author presents the gap in this theoretical framework discussing that public policy evaluation does not contemplate the social environment in which this policy should be directed. In this way, she proposes further research that will contribute to improving state of the art about public policy evaluation.

## INTRODUCTION

The objective of this paper is to present state of the art about public policy evaluation, with specific reference to privatisation. The basic research goal is to evaluate the purpose and reach of the public policy objectives relating to privatisation, particularly in relation to their socio-economic outcomes at regional and community levels.
What are the concerns that underpin the research focus on socio-economic effects? Jackson and Price (1994) comment that the success (or otherwise) of privatisation should not only be judged in terms of improvements in corporate performance indicators such as productivity and profitability. Such measures focus upon the distribution of benefits primarily to shareholders. They suggest, instead, that "there are other stakeholders in the benefits of privatisation, especially consumers, who need to be considered" (Jackson and Price, 1994, p. 17). According to authors, consumers can benefit from privatisation by paying less for their service, by facing more stable prices, or by receiving an improvement in the quality of the service.
The present discussion seeks to go further and address issues of a wider socioeconomic impact. What are the consequences of privatisation of an industry for the inhabitants of a certain place? Was privatisation of state corporations advantageous or not to stake-holding communities' quality of life?

## ABOUT PRIVATISATION: SOME APPROACHES TO DISCUSS

This section will discuss some approaches to privatisation and studies that try to demonstrate this impact.
The policy of privatisation has been adopted by an ever-increasing number of countries since the 1980s, starting in the UK but spreading widely over the ensuing years. Seeking to reduce its asset base in the economy and to concentrate on the so-called functions of State, the State transfers its assets and patrimonies to the private sector. This is a quasi-permanent arrangement, as distinct from the contracting out of public interests - known as temporary privatisation.
The wisdom of privatisation exercises has been hotly contested by researchers around the world (Bishop and Green, 1995; Hossain and Malbon, 1998), despite some research evidence on potential efficiency gains (Parker, 1998; Chisari et al., 1999).
Three distinct tactical approaches have been adopted in the process of privatisation (Hossain and Malbon, 1998). They suggest that these tactics have been pursued with different levels of vigour in differing regions of the world economy. The big-bang approach is being pursued in Latin America and Caribbean region, the go-slow and institutional approach is characteristic of the Asian and East Asian nations, and the marketisation political approach of the Eastern Europe. The more developed countries have been directing their strategies towards industries and utilities such as electricity, water and gas. Regardless of these distinctions, a momentum to privatise state industries has been adopted throughout the world. What are the claimed benefits that underpin privatisation? Gould (apud Hossain and Malbon, 1998), said that privatisation will:
. raise funds and reduce borrowing to improve a nation's effective fiscal and macroeconomic management;
. increase efficiency at the enterprise level;
. reduce government and bureaucratic interference in the day-to-day business of an enterprise;
. increase the share in enterprise ownership;
. create competition in the market;
. promote discipline in the market place.
The term privatisation can be defined in an inclusive manner, encompassing reductions in the welfare state, promoting at the same time entrepreneurial and voluntary activity. More narrow definitions conceive privatisation as the substitution of a private labour force for what is effectively an entirely identical group of public employees, to carry out a defined task.
Bishop and Green (1995) present the concept of privatisation as a political mainstream. In 1978, a policy document prepared for the then opposition Conservative party in England proposed state a sell-off as a possible means of reducing the power of public sector trade unions. Analysing the privatisation that took place in UK in the 1980s/90s and the situation of firms transferring to competitive markets, the authors concluded that these firms took advantage of commercial and managerial freedoms that had been denied them prior to privatisation.
In particular, some firms were able to restructure their operations and reduce their workforce in anticipation of the onset of recession whilst others were able to limit their exposure to recession by diversifying into less cyclical business. Bishop and Green (1995), however, fail to address at any point the consequences of the privatisation for the local and regional economy, nor do they address consumers' or customers' opinions.
Hossain and Malbon's (1998) edited contribution is explicitly dedicated to analysing the distribution of benefits from privatisation. This book attempts to comment on the impact of privatisation and some questions from it for the purpose of regulatory reform. The contributors analyse the losses and benefits of privatisation in a variety of countries from various perspectives. In many nations, regulators have been established for the privatisation process and subsequent performance of the privatised firms. Their remit is to protect the interests and activities of three major players, namely consumers, privately-owned new companies and governments. Despite the research showing undoubted benefits in some privatised industries, as Malbon (1998) comments, the overall impacts of privatisation remain unclear.
Malbon (1998) argues that privatisation should not be confused with deregulation. For example, the UK consumer was promised a great deal from privatisation: competition would force prices down and consumers would benefit from increased allocative efficiency. The idea was that competition would bring choice, promote efficiency within companies and provide the strongest possible incentive to keep retail prices as low as possible. However, as Meek (1998) observes, privatisation did not necessarily bring great competition in the UK. Analysing the privatisation of utility industries, he argues that, while consumers wanted a new market to develop benefits for them, the shareholder obviously sought healthy returns from a monopolistic player through the earning of economic rents.
Meek (1998) defends the lack of competition associated with the British privatisation experience in the utility industries. He contends that greater competition was not a central policy objective when the industries were sold and, at the time of writing, the only competition that domestic consumers had seen is in the market for telephone and communications. On this basis, the regulators may be judged to have failed in their efforts to extract a proportion of the net surplus from privatisation for domestic consumers.
Brown (1998), examining the economic issues relating to privatisation of telecommunications, in particular of the Telstra Corporation in Australia, concludes that the implications of privatisation are complex and depend upon a number of factors. These include whether or not the enterprise will be more profitable under private than continued public ownership, and the amount of underpricing and transactional costs. Longitudinal studies like these certainly shed light on the dynamic efficiency and impact of the privatisation process on monopolistic state firms. However, the socio-economic effects of privatisation in firms that were already operating in deregulated and competitive markets appear not to have been explored in the same way.
Wilding (1990) analysing the experience of the three Thatcher administrations in the UK suggest that privatisation policy can be analysed under three headings: economic, political and social factors. The economic focus was predicated on the belief that market provision is always more efficient than public sector provision. Two further propositions underpinned this idea:
. that management practice in the private sector is superior to that in the public sector and that such practices are transferable to newly privatised industries;
. that in market situations, the consumer has much more power relative to producers. This implies that economic freedom is enhanced when individuals buy and sell goods and services in market relationships and reduced when services are publicly provided.
The contention of the beneficence of the free market lay at the heart of government's privatisation policy in United Kingdom in the 1980s. Wilding's (1990) analysis of the political factors focuses on 1982, which he perceives as a crucial year. The Conservative Party needed a cause, to displace other problems with its strategy. Macroeconomic policy appeared in serious disarray. In addition, there was a more pervasive belief in the dead hand of the state, in the inherent inefficiency of public activity and public enterprise, the belief that what the state touches becomes relatively inefficient quite rapidly. The political agenda in this case pushes in two directions: a displacement of other potential failures elsewhere in economic policy; and a very conscious effort to roll back the interventionism of the Keynesian consensus. Finally, Wilding's (1990) social factors concern the moral benefits to a society where individual responsibility is clearly established, and this is to be achieved through expanding private ownership of homes and firms and liberating people from trade union restrictions. The author argues: how can we choose between public or private provision in differing, context-specific, socio-economic environments?
Privatisation policies change the structure of incentives and the criteria used to judge success. Jackson and Price (1994) are surely correct to argue that the literature on privatisation has tended to focus upon technical efficiency and ignore allocative efficiency - i.e. who gains and who loses. Technical efficiency requires that the firms produce the output demanded by consumers at the lowest possible cost subject to the technical constraints of production. The input mix is, therefore, efficient. On the other hand, allocative efficiency requires that firms produce the level, mix and quality of output at a price where it is not possible to re-arrange the outputs of an economy and make one consumer better off without making another worse off. The singular focus on technical efficiency may result in policies that are directed at cost-cutting while policies that might result in price reflecting costs more closely are often ignored. Besides, Jackson and Price (1994) ask the following question: has privatisation resulted in improvements in efficiency as was hoped for by the advocates of the programme?
Aylens' (1988) study of the UK steel industry concludes that the performance of the British Steel Corporation have been improved since privatisation. In 1980, British Steel was one of the lowest performing UK companies producing steel at prices significantly higher than its competitors. By the middle of the 1980s this had been reversed and the privatised British Steel Corporation was among the world's leading steel producers. This turnaround was not only achieved by changing ownership. Aylens (1988) describes the radical managerial and organisational reforms that were introduced. Decentralised business units with clear commercial objectives, which focused on outcomes, were established. Productivity bonuses were paid which, in 1987, represented about 20% of steelworkers' total earnings. The improvement in performance reflects the changes that were made to the incentive structure facing both management and the work force.
Raju and Thanassoulis (1994) researched a number of privatisations of large publicly owned UK companies. Their analysis identified the factors that explained in large measure, the dividend yield, market capitalisation and the change in the market value of the company on the first day of trading of its shares, after privatisation. The objective was to see how the experience gained from past privatisations might be used to the taxpayer's advantage. Their research provides guidance for any future privatisations on how best to price assets and state enterprises to maximise the benefits to taxpayers.
Parker and Martin (1996) examine the effects of privatisation on the use of inputs and hence on labour, capital and other supplies in eleven firms which were privatised in the United Kingdom in the 1980s. Their findings indicate that privatisation does seem to have been associated with higher profitability and labour does seem to have lost out in terms of income shares, though to a lesser degree in terms of employment and wage relativities and in purchases and out-sourcing. However, both sets of results need further investigation. Parker and Martin (1996) also suggest that the performance of each privatised company should be partly determined by the environment in which it operates. For example, in 1985 the UK steel industry had surpassed the steel industries of Germany, France and the USA in terms of technical efficiency. However, since then it has become more lacklustre. The authors suggest that the change in the capital market did not lead to obvious performance gains, although whether British Steel would have weathered the economic recession after 1989 worse (or better) under state ownership is impossible to determine. This observation, that performance appears environmentally contingent, underscores the potential relevance of the research programme being pursued here.
In a further study, Parker and Wu (1998) analysed the performance of British Steel before and after privatisation in December of 1988. They measure performance in terms of trends in labour and total factor productivity and profitability, on the one hand, and by comparing the technical efficiency in the UK steel industry with that in six other major steel producing countries, on the other. Their research method uses data envelopment analysis (DEA). The results confirm the existence of efficiency in the British Steel gains before privatisation but since then performance has been more lacklustre once again.
Parker (1998) reviews privatisation in each of the European Union member states, identifying differences in the level of privatisation activity between countries. He also explains why privatisation may not lead to efficiency gains because of the specific form that privatisations are taking and the nature of capital market within the European Union. The author argues that privatisation implies a redistribution of income and wealth and therefore of economic power. However, these effects have yet to be widely discussed and researched. Parker (1998) suggests concentrating on the long-term effects on the competitiveness of European industry and the implications for social welfare.
This brief review indicates the range of research methods being used by many authors to try to establish the impact of privatisation on the performance of the firms concerned over time. It also suggests a continuing concern with the impact of privatisation on differing interest groups.

## KEY CONCEPTS

The literature on privatisation is full of so many complementary definitions of key terms in regard to it. Hossain and Malbon's (1998) definition seems to be the most complete. According to the authors, privatisation is the desire to strengthen and expand the market at the expense of the state and increase the exposure of the public sector to market forces. Privatisation involves the selling of state-owned enterprises (SOEs), government business enterprises (GBEs), and others government activities to the private sector. The most common privatisation mechanism includes direct sale of a GBE, public offer of a GBE, concession sale, joint venture, management buyouts, liquidation and lease. In many nations, regulators have been established to oversee the privatisation process and to protect the interest and activities of three major players: namely consumers, privately owned new companies and governments.
Defining the policy is a complex problem and several authors discuss this theme. Palfrey et al. (1992) outlines some of the more important differences. Academic researches contrast policy as an aspiration of a political party, as a general purpose and a particular programme. In this way, public policy should be understood as a process by which proposals are transformed into activities that will produce some kind of output (effects and impacts) in the society directly affected.
The evaluation of public policy could be understood as the whole process of checking afterwards how far policy objectives have been achieved and how effectively and economically. Evaluation frequently relates to existent, measurable and most often segmental aspects of the policy process. However, subjective values are admissible in some cases. Evaluations try to provide answers to causal questions (impact or efficiency), and are conducted post facto (Geva-May and Pal, 1999). This perspective is denominated a summative evaluation. This discussion will be presented later in this paper.
The several stages in the public policy cycle, agenda setting, policy formulation, policy legitimating, policy implementation, evaluation and policy reconsideration could be comprehended as a system where inputs, outputs, effects and impacts have a particular importance and must be distinguished as a part of the chain of value creation.
Inputs are the resources required for formulating and executing the policy. So, outputs are specific activities that directly result from inputs. Effects (or outcomes) should be defined as what actually happened. It refers to direct consequences or results that follow from an activity or process, the resulting change in previous conditions. Ultimately, the impact is the consequence of a policy, both good and bad, expected and unexpected, reflecting those implications for society as whole, for example, the long-range improvements in attractiveness or the improvement of the quality of life in the community.
Evaluation's literature has been giving emphasis at, firstly, the importance of the shareholders and stakeholders. Who should command the evaluators' attentions first? The answer to this question depends on the objectives of evaluation. For example, if the aim is to analyse the financial performance of a privatised company, the focus will be the shareholders and financial assets. On the other hand, if the intent is to understand the various interests that encircle the whole process of public policy making, then it is necessary to identify persons or groups who make decisions or desire information about that program, which means the stakeholders (Patton, 1997).

## PRIVATISATION AS A PUBLIC POLICY

The authors have discussed above how to approach privatisation from a public policy perspective, seeking to analyse the broad range of impacts arising from the privatisation process. Besides, the study of public policy requires attention to its initiation, formulation, implementation and impact.
Hogwood (1995) traces the growth of interest in a policy focus in Britain during the 1970s and early 1980s and its subsequent partial displacement by the emphasis on public management. Despite this partial displacement, the policy focus is now institutionalised in academic research, textbooks, journals and teaching. One area in which the British debate has assumed a leading role has been implementation planning and management. However, the literature about evaluation continues to be dominated by the huge volume of American material.
Pollitt (1996) confirms this conclusion about the development of Britain's research about public policy. He suggests that British public administration has endured radical anti-state reforms since 1979. However, he concludes that "UK academic public administration is still more that of a North American Satellite than a core European State" (Pollitt, 1996).
Nevertheless, Geva-May and Pal (1999), describing the differences about policy evaluation and policy analysis, argue that evaluation is a subset or phase of policy analysis that is primarily research-oriented. Evaluation of a public policy tries to provide answers to causal questions, like impact or efficiency. It is conducted post facto, and concentrates its analysis in process and costs issues. On the other hand, policy analysis is about making recommendations for future improvement and will be based in part on research evidence derived from previous evaluation studies. Defined in these terms, policy analysis is intended to assist policymakers in choosing the best policy alternative among a set of alternatives, with the aid of evidence and reason, based on multiple methods of inquiry and argumentation.
The act of evaluating a public policy may thus be thought of as the compiling of a retrospective argument. It relates to existent, measurable and often artificially segmented aspects of the policy process. Frequently, evaluators are required to use evidence and present policy makers with data that may assist them in the decision-making process. This informs (but never determines) the choice of the best policy among a set of alternatives, which is ultimately made on the basis of subjective reasoning. The target information of evaluators is goals, programme design, the implementation process and the outcomes of the evaluation process.
The study methods are another area of controversy and considerable experimentation. Evaluation typically uses strict and objectivist social science research methods to assess the worth of the various organisational structures, ongoing programs, personnel, budgets and operating procedures. Policy analysis could be compared to journalism. It uses empirical methods and social science theories to figure out the expected consequences of anticipated alternatives; it requires data gathering and communication skill similar to those practised in journalism. After all, the policy analyst must provide good information to convince that his conclusion is the best.
Evaluation does make use of case studies, mostly experimental or comparative studies whether quantitative or qualitative, or a combination of the two. It sets strict criteria and emphasises reliability and validity measurements. On the other hand, the data used by policy analysis are derivative and have been gathered, created or developed by others, in different settings and times, and for different needs.
Therefore, what is the evaluation's problem definition? Geva-May and Pal (1999) says that it is to determine the research method, target population and research tools, but particularly avoid predicting results. The analyst of public policy should finish this part of the work. Since evaluation is primarily retrospective, it looks at context in terms of factors that facilitated or impeded policy success.
When evaluators suggest policy options based on their evaluation research, they are using faithful data to make unfaithful recommendations. That is because the selection of options depends on a much wider range of information than can be provided through standard evaluation methods. This is the domain of the policy analyst. But, under certain circumstances, the boundaries can be crossed. The big question is how to recognise the transition from one mode to another.

## RESEARCHING PUBLIC POLICY EVALUATION: SOME KEY METHODOLOGIES

Geva-May and Pal (1999) propose a theoretical distinction between policy analysis and evaluation concepts, marking the potential divergence between them. They suggest some methods to evaluate a public policy:
. cost-benefit analysis;
. system analysis;
. the use of case studies (experimental or comparative);
. the use of qualitative and quantitative research, with the objective being to emphasise reliability and validity of measurements.
Beneath each of these methods lies a panoply of specific tools and procedures that are spelt out in considerable detail in the Centre European Evaluation Expertise (1999). Geva-May and Pal (1999) also suggest that the act of evaluation should be oriented to the particular problem under investigation and avoid predicting results.
Goldman, Nakazawa and Taylor's (1997) article proposes eight steps to evaluate the economic impact of a public policy. For them, economic impact studies need to provide information about the impact a project will have on the community's overall environment as well as on jobs, income and housing. The idea is not only to estimate total benefits and costs, but also to gauge the differences in distribution of income and costs arising from an intervention in the community. They argue that the evaluator must analyse the effects of each alternative on the individual economic areas being considered and also the indirect effects (or cross-impacts, as they term them) among the economic areas. It is also necessary to identify both the aggregated and the distributional impacts of each alternative.
Reese and Fasenfest (1997) consider which kind of evaluation works best. The article discusses the values underpinning the evaluation of public policy, specifically economic programme. The authors argue that, firstly one must identify what the locality is trying to achieve, particularly what they think development is. The answer emerges between economic growth and a large systematic change. Economic growth objectives use indicators like job creation, less tax. The large systematic change defines development rather as a quantum shift development than simply growth, but the distributional impacts, that is, those that affect some individuals, groups, institutions, business and communities more than others.
A proposal to narrow the gap between generating evaluation findings and actually using those findings for program decision making and improvement is demonstrated by Patton (1997), with his utilisation-focused evaluation methodology. In this approach, the first question is what has to be done to get results that are appropriately and meaningfully used. For this, the author presents four standards for evaluation: utility, feasibility, propriety and accuracy.
Utility is intended to ensure that an evaluation will serve the practical information needs of intended users. Feasibility is intended to ensure that an evaluation will be realistic, prudent, diplomatic, and frugal. Propriety focuses on evaluation practice (the process) in order to ensure that an evaluation will be conducted legally, ethically, and with due regard for the welfare of those involved in the evaluation, as well as those affected by its results. Finally, accuracy is intended to ensure that an evaluation will reveal and convey technically adequate information about the features that determine the worth or merit of the program being evaluated.
The first challenge in evaluation is therefore to answer the following question: who needs evaluation? Who cares about its findings? Following this, the task is to localise and identify the primary users, narrowing the list of potential stakeholders to a much shorter, more specific group of primary intended users. Their information needs, that is, their intended uses, focus the evaluation. Another important observation is that a particular evaluation may involve multiple levels of stakeholders and therefore need multiple levels of stakeholder involvement.
Patton (1997) identifies three primary uses of evaluation findings: first, to judge merit or worth, for example, summative evaluation, accountability, audits, quality control, cost-benefit decisions, decide a program's future, and accreditation or licensing. The second use is to improve programs. He highlights this in terms of specific approaches, formative evaluation, the identification of strengths and weaknesses, continuous improvement, quality enhancement, constructing a learning organisation, managing more effectively, local adaptation of national model. Lastly, with the objective of generating knowledge, Patton (1997) indicates the following examples of evaluation:
. generalisations concerning effectiveness;
. extrapolation of principles about what works;
. theory building;
. synthesising of patterns across programs;
. the production of research output.
Where the answers to the evaluator's questions indicate a major decision about a programme's merit or worth, then the evaluation should be designed to render an overall judgement: in other words, to conduct a summative appraisal. Where dialogue with primary intended users indicates, however, an interest in identifying strengths and weaknesses, clarifying the program's model, and generally working at increased effectiveness, the evaluation should be framed to support improvement-oriented decision making. In pursuit of this, the evaluator must employ skills in offering formative feedback. Beyond this, where the intended users are more concerned about generating knowledge for formulating future programs than with making decisions about current programs, then some form of synthesis or cluster evaluation will be most appropriate to discover generic principles of effectiveness.
Pawson and Tilley (1997) present proposals for realistic evaluation. Firstly, they discuss the following proposition: programmes work (achieved desired outcomes) only in so far as they introduce the appropriate ideas and opportunities (mechanisms) to groups in the appropriate social and cultural conditions (contexts). These three concepts inform the authors' overall rationalist theoretical contribution. The choice of method has to be carefully tailored to the exact form of hypothesis developed earlier in the realist evaluation cycle.
Pawson and Tilley's (1997) objective of constructing realistic data leads them to analyse four different interpretative currents of key informants (the stakeholders): policy-makers, subjects, practitioners and evaluators. According to the authors, doing a cross fertilisation analysis with these items of information will design a more significant data, because the researcher's theory is the subject matter of the interview, and the stakeholder (the subject) is there to confirm, to falsify and, above all, to refine that theory.
Lichfield (1996) presents proposals for community impact evaluation, an interesting discussion about how to devise planning interventions that can be clearly justified in terms of need, cost of planning, the impact on individual freedoms and the bureaucratic implications. In other words, what options are available to produce better results in terms of value for money? Starting the discussion with cost-benefit analysis, the author sets out several questions to criticise this methodology:
. Whose cost and benefits are to be taken into account?
. Which costs and benefits in geographical terms?
. Should the decision relate simply to efficiency or also to equity and social justice?
To answer these reflections, Lichfield (1996) proposes a methodology to show the importance of analysing not only the impact as output, but also the effect of that on people (the community). The author shows the gulf between effects and impacts. The effects could be defined as the physical and natural changes resulting, directly or indirectly, from development. On the other hand, impacts are the consequences or end products of those effects represented by attributes of the environment on which we can place an objective or subjective value. There are several categories of impact, but the main ones are direct and indirect.
Lichfield (1996) argues that the direct (or first order) impact manifests directly on the environment as a result of the project impact. However, the indirect (or second order, induced or secondary) impact is generated by activities that result from the project. Other impact categories highlighted by the author are cumulative; short, medium or long-term; permanent or temporary; reversible or irreversible; positive or negative effects. Finally, the impact could be evaluated ex ante, with the objective to predict the scale, magnitude and significance of the impacts or ex pos, after the event.

## THE LACUNA: THE PROPOSAL CONTRIBUTION TO THE PUBLIC POLICY EVALUATION

Following Palfrey et al.'s (1992) idea, the policy evaluation literature has been dominated by American contributions, in two kinds of publications: a general texts on the process of evaluation research; and titles relating to the methods for evaluating particular social programmes. In Britain, after the development of cost-benefit analysis in the early 1960s, the main preoccupation since the 1970s seems to have been the construction of performance indicators.
However, the lacuna highlighted by Palfrey and others is that targets may be met and objectives achieved but at what cost to considerations of equity and social justice. According to the authors, a properly conducted piece of evaluation research has ultimately to relate the findings not just to specified, measurable objectives but to prior considerations about the values involving particular policies. The relevance and importance of processes in determining both efficiency and effectiveness are too often ignored.
Therefore, succeeding the Hossain and Malbon's (1998) backing, let us return to the initial question: who benefits from privatisation? If privatisation can be understood as a public policy programme, the literature suggests a number of further questions:
. Have privatisation programmes shown any sensitivity to socio-economic impacts on local and regional communities?
. What research methodologies have been used to analyse this impact?
. Are these research methodologies serving their purpose?
The answers to these questions above will contribute to public policy evaluation as a discipline and as a comprehensive view on reality.
For a further study, the author wishes to demonstrate a research proposal that could supply the gap described above with new reflections. The proposition is to research Brazil and England's privatisation experiences, and answer the questions implied for the following reasons.

## CONCLUSIONS BUT NOT THE ENDINGS

The first question will analyse an important public policy programme which took place in both countries. In the case of England, the especial importance is that it was the first country in the world to start this kind of economic public policy on a large scale. Brazil's study figures as an interesting case because it is one of the first countries in Latin America to follow the neo-liberalism perspective of Mrs. Thatcher on a government level. Another reason is that, in Latin America, Brazil had greatly different kinds of business which were state-owed in the earlier 1990's. For example, metallurgy, petrochemical, fertilisers, electric, rail, mining, port, financial, computer systems, and others sectors have been privatised since 1991.
The second question needs to understand if the usually research methodologies have been discuss the difference shows above in this paper between outcomes and impacts. For in such a way, one of the proposals of this research is to made a vigorous literature review on public policy evaluation, mainly in privatisation programmes.
The third question is to reflect on the possible gaps in the research methodologies that propose to analyse and evaluate public policy impacts. The results of the second question will be the data of these considerations. The final aim of the study will contribute to public policy theory in terms not only of evaluation, but also of policy formulation.
REFERENCES
AYLENS, J. Privatisation of the British Steel Corporation. Fiscal Studies, v. 9, n. 3, p. 1-26, 1988.
BISHOP, M.; GREEN, M. Privatisation and recession: the miracle tested. Discussion Paper n.10. Centre for the Study of Regulated Industries, 1995.
BROWN, A. The economics of privatisation: case study of Australian Telecommunications. In: HOSSAIN, M.; MALBON, J. (Eds.). Who benefits from privatisation? Routledge Studies in the Modern World Economy, n. 16, 1998.
CENTRE FOR EUROPEAN EVALUATION EXPERTISE. Evaluating socio-economic programmes: principal evaluation techniques and tools. Means Collection, v. 3, 1999.
GEVA-MAY, I.; PAL, L. Good fences make good neighbours: policy evaluation and policy analysis - exploring the differences. Evaluation, v. 5, n. 3, p. 259-277, 1999.
GOLDMAN, G.; NAKAZAWA, A.; TAYLOR, D. Determining economic impacts for a community. Economic Development Review, p. 48-51, Spring 1997.
HOGWOOD, B. W. Public policy. Public Administration, v. 73, p. 59-73, Spring 1995.
HOSSAIN, M.; MALBON, J. (Eds.). Who benefits from privatisation? Routledge Studies in the Modern World Economy, n. 16, 1998.
JACKSON, P. M.; PRICE, C. Privatisation and regulation: a review of the issues. Longman Group, 1994.
LICHFIELD, N. Community impact evaluation. London: UCL Press Limited, 1996.
MALBON, J. Gaining balance on the regulatory tightrope. In: HOSSAIN, M.; MALBON, J. (Eds.). Who benefits from privatisation? Routledge Studies in the Modern World Economy, n. 16, 1998.
MEEK, C. Privatisation doesn't necessarily equal competition: the UK experience. In: HOSSAIN, M.; MALBON, J. (Eds.). Who benefits from privatisation? Routledge Studies in the Modern World Economy, n. 16, 1998.
PALFREY, C. et al. Policy evaluation in the public sector: approaches and methods. Averby: Ashgate Publishing Limited, 1992.
PARKER, D. Privatisation in the European Union: a critical assessment of its development, rationale and consequences. Aston Business School, Research Paper Series, n. 5, 1998.
PARKER, D.; MARTIN, S. The impact of UK privatisation on employment, profits and the distribution of business income. Public Money & Management, p. 31-38, Jan./Mar. 1996.
PARKER, D.; WU, H. Privatisation and performance: a study of British steel industry under public and private ownership. Aston Business School, Research Paper Series, n. 15, 1998.
PATTON, M. Q. Utilisation-focused evaluation: the new century text. 3. ed. London: Sage Publications, 1997.
PAWSON, R.; TILLEY, N. Realistic evaluation. London: Sage Publications, 1997.
POLLITT, C. Antistatist reforms and new administrative directions: public administration in the United Kingdom. Public Administration Review, v. 56, n. 1, p. 81-87, Jan./Feb. 1996.
RAJU, S.; THANASSOULIS, E. Privatisation of UK public companies: factors influencing flotation values and the lessons to be drawn for future privatisations. Warwick Business School, Research Bureau, n. 123, 1994.
REESE, L.; FASENFEST, D. What works best? Values and the evaluation of local economic development policy. Economic Development Quarterly, v. 11, n. 3, p. 195-207, Aug. 1997.
WILDING, P. Privatisation: an introduction and a critique. In: PERRY, R. (Ed.). Privatisation. Research Highlights in Social Work, n. 18, 1990.