SPECIAL REPORT No 15/98 on the assessment of Structural Fund intervention for the 1989-1993 and 1994-1999 periods together with the Commission's replies (submitted pursuant to Article 188c(4)(2) of the EC Treaty) (98/C 347/01)

1. INTRODUCTION

1.1. Under the 1988 reform of the Structural Funds (SFs), the Commission and the Member States were required to evaluate European structural policies. In 1993 this requirement was strengthened by the amendments to the Regulations that were introduced in the context of the revision of the SF reform (1994-1999) and the standard clauses of the new Community Support Frameworks (CSFs) and Single Programming Documents (SPDs).

1.2. Evaluation, an essential tool for the sound management of the SFs, is defined in the main provisions of the Regulations (1) as the evaluation of effectiveness on the following three levels:

(a) the overall impact of the structural measures on the objectives set out in Article 130A of the Treaty;

(b) the overall impact of the measures laid down in the CSFs;

(c) the specific impact of each operational intervention (programmes and other forms of intervention).

1.3. There are three main phases to this evaluation:

(a) prior appraisals (2), which enable a correct identification of the socio-economic problems that need to be resolved and the objectives to be achieved, on the one hand, and, on the other hand, of the strategy adopted and the means available for this purpose;

(b) ongoing assessments, which are carried out during implementation of a programme and make it possible to check whether the objectives are still appropriate and the desired results on the way to being achieved;

(c) ex post evaluation, which is carried out after the programme has been completed. Its aim is to analyse whether the results and impact obtained are to be considered positive, bearing in mind that the long-term effects of structural measures can only be appreciated several years after their completion.

1.4. The 1993 versions of the Regulations reiterate the responsibilities of both the Member States and the Commission with regard to assessment and stress that it is to be carried out within the partnership system (3). In this context, the Commission has taken the initiative concerning the supervision and general coordination of the implementation of evaluation and, in accordance with Article 16 of Regulation (EEC) No 2052/88, submits every three years an evaluation report on the progress achieved towards economic and social cohesion and, in particular, the contribution made by the Funds and national co-financing.

2. THE SCOPE OF THE AUDIT

2.1. The general aim of the Court's audit (4) was to check the progress achieved in evaluating the Community's structural interventions, in terms of systems and evaluation procedures and at a methodological level. The audit looked at the methods used to evaluate structural interventions both by the Commission and under the partnership system. It also examined the legality and regularity of the utilisation of the appropriations earmarked for evaluation in the budget of the European Communities and the quality of the financial management of these appropriations.

2.2. The audit also involved an appraisal of the techniques used for evaluating the socio-economic impact of structural measures. Although it did not aim to evaluate the programmes directly with the intention of producing new scientific results, it nevertheless made use of the results of work by experts so as to obtain a better understanding of the relationship between the economic policy instruments used and the results obtained.

2.3. Specific audits were carried out on the ex post evaluations of the CSFs and Operational Programmes (OPs) for Objectives Nos 1 and 2 for the 1989-1993 programming period and on the ongoing assessments of the CSFs, SPDs and OPs for the 1994-1999 programming period. All the ex post evaluations for the 1989-1993 Objective No 1 CSF were taken into account, as well as approximately 20 % of the ex post evaluation reports for the 1989-1993 Objective No 2 CSF and 15 % of the ongoing assessment reports for the 1994-1999 CSFs, selected on the basis of their geographical distribution and the nature of the interventions (CSFs, SPDs or OPs). In addition, the Court analysed 15 out of the 25 evaluation reports that dealt mainly with macroeconomic aspects. The audit was carried out at the Commission and in eight Member States (Belgium, Greece, Spain, France, Ireland, Italy, the Netherlands and Portugal), where the auditors consulted the assessors and national authorities concerned.

3. SUMMARY OF THE MAIN OBSERVATIONS

3.1. Evaluation developed and spread, at the Commission's initiative, mainly during the 1994-1999 programming period, in particular in the context of the ERDF, leading to its recognition in the field of management and programming. As from the beginning of this programming period, evaluation has been systematically applied to the first two phases (prior appraisal and ongoing assessment) and will be applied to the third at the end of the period (ex post evaluation) (see paragraphs 4.2-4.8).

3.2. Although the Commission has the figures regarding its own direct expenditure on evaluation, it does not always have access to all the information for expenditure on evaluations carried out under the partnership system (see paragraphs 4.9-4.11).

3.3. The Commission made a considerable contribution towards spreading the idea of evaluation, even though it did not introduce an adequate framework for evaluation indicators and methods in its programme on 'Methods for Evaluating Activities of a Structural Nature` (MEANS). In view of the complex nature of the methodological aspects of evaluation, there is no single methodology that can be applied to all specific situations. It is, however, important that the Commission should be in a position to be able to supply methodological frameworks that provide appropriate guidelines for the implementation of evaluations suited to specific situations (see paragraphs 4.12-4.13).

3.4. The quality of the indicators is closely linked to the reliability of the information gained from the monitoring of existing interventions and statistical bases. The Member States are responsible for these latter and they are often inadequate. The Member States should improve their quality (see paragraphs 4.14-4.21).

3.5. Many evaluations are completed late, which may proportionately delay the adjustment of programmes. Furthermore, the decision-making system could take evaluation results concerning the performance or effectiveness of intervention schemes better into account for future programmes (see paragraphs 4.22-4.27).

3.6. With regard to the objectivity of evaluations, the Commission makes no on-the-spot checks of the consistency and reliability of indicators or the systems that produce them (see paragraphs 4.29-4.30).

3.7. There are still significant shortcomings in the Commission's procedures for awarding evaluation contracts to consultants. The choice of consultants is not always clearly justified. Contract specifications are very general in terms of methodological techniques, indicators and the need to achieve results. This leads to delays and disturbances in the submission of evaluation reports, qualitatively variable analyses, and, sometimes, vague recommendations (see paragraphs 4.31-4.36).

3.8. The implementation of the ex post evaluation methodology for Objectives Nos 1 and 2 (1989-93) came up against statistical shortcomings, a lack of objectives, an absence of quantified forward indicators, an ineffective monitoring of impact and was also not sufficiently stringent. Nevertheless, these evaluations revealed that the SFs did indeed contribute Community added value, both in terms of the achievements, results and effects of the CSFs, OPs and measures and from the point of view of programming, organisation, monitoring, coordination, partnership and additionality (see paragraphs 4.15-4.19, 5.1, 5.4, 5.10-5.16).

3.9. With regard to the ongoing assessment of the 1994-1999 SF interventions carried out in partnership under the aegis of the Monitoring Committees, an examination of the physical and financial results always plays an important role in mid-term assessment, even though there may be variations in the priorities granted to the various evaluation objectives. The evaluation of the effects is hampered in particular by problems with indicators and the modelling of effects. This type of evaluation is still often only of relative value in terms of the decision-making process, the management of projects, reprogramming and the preparation of future interventions (see paragraphs 4.16-4.21, 6.5, 6.7, 6.14-6.26).

3.10. In the specific case of Objective No 2, the evaluation revealed the following weaknesses:

(a) a mismatch between programme implementation and Community funding, with significant carry-overs of appropriations from one programme to another (about 12 % on average for the shift from the 1994-1996 period to the 1997-1999 one)(see paragraph 6.27);

(b) the succession of short programming periods does not permit an optimum use of evaluation results (see paragraph 4.23);

(c) indirect effects are still not measured sufficiently to determine the impact of structural interventions, in particular as regards employment (see paragraphs 5.12-5.15).

3.11. Despite failings in the programming, implementing and monitoring systems, the macroeconomic evaluations revealed positive results, in particular in respect of the impact of structural measures (increases in GDP, effects on supply and demand, improvements in terms of employment and convergence and the effectiveness of the CSF instruments), (see paragraphs 7.1-7.14).

3.12. Evaluations of sub-programmes and measures continue to be inadequate. Furthermore, they come up against technical difficulties caused by the dispersion and inconsistency of many measures and sub-programmes (see paragraph 7.17).

4. GENERAL ISSUES

Introduction

4.1. The aim of evaluation is to determine the socioeconomic effects of the measures implemented, i.e. to examine the results obtained, compare them with the resources invested and analyse their cost. Furthermore, all these points must also be looked at in relation to forecasts. However, this evaluation task presupposes a good statistical basis, quantified objectives established on the basis of a system of indicators and the development of a methodological and analytical framework. At the same time, it must be supported by an analysis of the relationships between the means used, final results and intermediate results. The Court found failings at all these levels, and, despite real progress that has helped consolidate the system of evaluation, these have so far limited the usefulness of the evaluations carried out in connection with the Structural Funds.

4.2. Since the 1988 reform, evaluation has been explicitly included within the partnership system (Article 26 of Regulation (EEC) No 4253/88 - described more fully in the amended Regulation of 20 July 1993). During the first programming period and up until 1995, evaluations were nevertheless mostly carried out at the Commission's initiative (5). These evaluations - in particular the ex-post ones - were hampered by a lack of specific development strategies and inadequate monitoring. Moreover, generally speaking, they were not really taken into consideration and discussed in detail by the Monitoring Committees.

4.3. The Commission initially concentrated mainly on setting up its own evaluation mechanism for all the CSFs covering the reform's five priority objectives (6). Among the Commission's initiatives in connection with the development of evaluation and evaluation methods, mention must be made, in particular, of the 1992 creation of the technical group on the 'Evaluation of Regional Policies` and, at a methodological level, the launching of the programme on 'Methods for Evaluating Activities of a Structural Nature` (MEANS) in 1994.

4.4. For the 1994-1999 period, the Commission was responsible for a new evaluation system, which is now systematically applied to the first two phases (prior appraisal and ongoing assessment) and will later be applied to the ex post phase.

4.5. Although ongoing assessment is not explicitly provided for in the SF Regulations, as of 1995-1996 it has been applied to all the CSFs, SPDs and OPs for the 1994-1999 programming period. In concrete terms, it falls under the partnership system, allowing the Monitoring Committees to turn it into an integrated management tool and thus contributing towards a very broad awareness of the idea of evaluation. The Commission set up a system for the phased implementation of ongoing assessment in the Member States (7).

4.6. However, this system did not specify the actual procedures to be used (the definition of project specifications, the criteria for the selection of assessors, the structure of evaluation reports, data collection, quality standards, timetables, etc.).

4.7. At the same time, the Commission organised a series of events accompanied by a number of publications (8). By bringing the SF managers together with experts in the field of evaluation, these events contributed to the development of the idea of evaluation and helped raise the managers' awareness of this subject. Evaluation was presented primarily as a process to be learnt by experience on the job.

4.8. In May 1998, in the context of the preliminary draft budget of the European Communities for the financial year 1999, the Commission presented its 'evaluation report 1996/1997`. In this working paper, it describes the evaluation activities carried out in 1996-1997, gives a list of evaluations already implemented and currently underway and supplies a summary of the results of the evaluations implemented in the 1996-1997 period.

4.9. Expenditure on evaluation work may either be financed directly by the Commission, or co-financed with the Member States. Tables 1 and 2 show expenditure directly settled by DG XVI in the field of evaluation for the 1989-1993 and 1994-1999 programming periods respectively.

4.10. During the 1989-1993 period, as Table 1 shows, around 10 Mio ECU was spent on evaluation. Around two thirds of this expenditure occurred in two fields which come directly under the Commission's jurisdiction and for which it is directly responsible. These were, on the one hand, the Community initiatives under Article 11 of Regulation (EEC) No 4253/88, amounting to 3,1 Mio ECU, and, on the other hand, cross-disciplinary and thematic evaluations amounting to 3,2 Mio ECU, such as those concerning SMEs, telecommunications, the environment and human resources. However, the amounts earmarked for the evaluation of Objectives Nos 1 and 2 of the Structural Funds were relatively small, given the fact that, during this period, partnership was not very well developed in the field of evaluation. As is shown in Table 2, the amount earmarked for evaluation for the 1994-1999 period amounted to 25,2 Mio ECU in April 1998. This increase over the previous period was due to the development of ongoing assessment and its methodology within the partnership system.

4.11. With regard to spending in this field by the Commission and the Member States, which has increased considerably since 1996, the Commission is not always fully informed, in particular where the evaluations are included under the 'technical assistance` measures of Operational Programmes, which may also involve other forms of expenditure. A specific allocation to cover the needs of evaluation could be provided right from the initial programming phase, which would make expenditure more transparent.

Reference framework

4.12. The MEANS programme (1995-1998) (2,8 Mio ECU) has played a specific role in the process of the development of evaluation (9). The procedure applied aims to encourage the development of a uniform and consistent methodology, via an inventory of the methods used in the Member States and the identification of valid examples. This essentially empirical and exhortatory approach does not enable the Commission to decide on the actual procedures to be used for the implementation of evaluation methods or dictate a choice of indicators, in particular those for measuring the socioeconomic impact in the field of regional development. In view of the complexity of the methodological aspects of evaluation, there is no single method applicable to every specific situation. It is nevertheless important that the Commission should be in a position to supply methodological frameworks to enable appropriate guidelines to be drawn up for the implementation of evaluations suited to specific situations.

4.13. So far, the results of the MEANS programme have been used to lay the foundations of a new evaluation procedure, in particular by introducing a common language, and have enabled the preparation of ongoing assessments within the partnership system. Tables 3 and 4 show the generally accepted evaluation procedure. However, the MEANS programme has not yet provided detailed answers to the questions regarding the practical and operational aspects of evaluation. Thus, two fundamental questions remain unanswered: firstly, what is the most appropriate system of indicators for use in formulating an overall assessment of a given Community structural programme; and, secondly, what evaluation approach is most suitable for measuring the socio-economic impact of a given structural programme in the context of regional development? The Commission is not yet able to draw up precise instructions regarding the actual techniques to be applied in each specific evaluation situation. The absence of such instructions is felt particularly during the ongoing assessments of intervention for the 1994-1999 programming period.

Statistical bases and information concerning objectives

4.14. Evaluations (whether prior appraisals, ongoing assessments or ex post evaluations) are often confronted with objectives that have been formulated too generally, even with regard to the objectives of the measures in question.

4.15. The level of development of existing statistical bases is an impediment to the monitoring and evaluation of programmes and measures co-financed by the Structural Funds. Shortcomings are evident at various points, including, in particular, the lack of statistical coverage limited to the areas concerned, the absence of statistics regarding certain aspects and the collection of information at intervals that are inappropriate in relation to the needs of monitoring and evaluation.

Indicators

4.16. The existence of reliable and appropriate systems of indicators is an essential element in the planning process. For evaluation purposes, the structural programmes must contain information on financial and physical achievement as well as on results and performance. These indicators should provide managers with information on the immediate or long-term effects of action taken and also provide a measurement of the progress achieved in relation to the overall objectives of the programmes (e.g. regional growth or the reduction of unemployment).

4.17. The reliability of the financial indicators is closely linked to that of the expenditure certificates sent to the Commission prior to the relevant Community payments. Physical indicators are often considered to be a source of supplementary information and their reliability is not consistent. Existing databases do not always provide a record of actual physical progress and final recipients do not always provide managers with periodic information about the physical state of operations. The ongoing assessment of the Lisboa e Vale de Tejo OP in Portugal reported physical indicators in strict proportion to expenditure without mentioning any variations linked to delays in implementation or other causes. The Member States should improve their monitoring systems and the databases on which their indicators are based.

4.18. Both the measurement of results and impact at sub-programme and measure level and the measurement of impact at programme level are hampered by the fact that neither the causal relationships between the various levels of objectives and between objectives and measures/projects nor the horizontal effects of complementarity and synergy between measures and between objectives can be identified with any certainty. It is therefore difficult to define the specific effect of a given measure on regional development or the impact of investment on demand or, a fortiori, on supply. In this context, with regard to the 1994-1999 CSF, the search for indicators that will serve as an appropriate tool for evaluating results and impact has still not been completed (see paragraph 6.23).

4.19. The prior appraisal of the CSFs is essential to subsequent evaluations. It ensures that all the result and impact indicators have been specified and quantified, thus enabling a judgement to be made on the estimated impact of the programme concerned and comparisons to be drawn with actual effects. In this context, the management by objectives approach has not always been applied in a uniform or satisfactory fashion, which has had an impact in terms of the indicators necessary for evaluation. Thus, the Italian 1994-1999 CSF contains a small number of indicators, a fact which could hamper subsequent evaluation. On the other hand, the 1994-1995 Portuguese CSF, which was approved by the Commission without any major changes, includes too many indicators (about 300), which might also complicate evaluation. This variation in the establishment of indicators from one CSF to another highlights the lack of consistency in programming. Where the system of indicators approved by the Commission during prior appraisal contains a number of general and unquantifiable objectives, they are sometimes revised for purposes of ongoing assessment because they are found to be badly defined and insufficiently functional.

4.20. The structural interventions and the socioeconomic situations upon which they are brought to bear give rise to a wide variety of indicators that are often developed locally. Although partial lists of indicators have been sent to certain regional or national authorities and have already been the subject of debate, the Commission has not drawn up any lists of the indicators that it considers to be both essential and representative in terms of regional development and in connection with the various types of measure.

4.21. The quantification of result and impact indicators should be developed further in connection with monitoring, and, at the same time, this quantification should also be reliable. The fact is that it has been known to be unreliable, thus failing to prevent cases where new jobs were counted twice or to allow a distinction to be made between the gross and net effects on employment. This was discovered by comparing the ongoing assessment report of the Brandenburg OP with the statistics concerning the people registered at the Social Security office.

Taking evaluation into account

4.22. The contribution of evaluations in terms of a more efficient utilisation of Community funds largely depends on the quality of their content, and on when the results are received and decisions are made regarding changes in strategy. However, a number of ongoing assessments for the CSFs, SPDs and OPs for the 1994-99 programming period were finalised rather late, thus delaying possible changes.

4.23. With regard to Objective No 2, the succession of shorter programming periods (as compared with Objective No 1) means that it is particularly unlikely that the dates when programming decisions need to be taken will coincide with those when evaluation results become available, thus making it impossible to learn from the past and adapt the subsequent interventions. The Objective No 2 programming and evaluation cycles are, to a certain extent, out of step. For example, the final version of the summary report on the ex post evaluation of Objective No 2 for the 1989-1993 period was submitted to the Commission in the summer of 1997, while the prior appraisal for the 1997-1999 programming period was underway and the ex post evaluation for the 1994-1996 period had not started.

4.24. Budgetary appropriations for structural programmes and subsequent programmes are rarely influenced by the programme results shown up by the evaluation reports. During the first programming period the decision-taking system very rarely took any account of evaluations, and changes in programmes were generally based primarily on the degree of financial absorption. This was the case for the useful recommendations made by the head consultant in the context of the ex post evaluation of the 1989-1993 Objective No 1 CSF (see paragraph 5.7).

4.25. The more qualitative aspects of performance or programme effectiveness identified in the ongoing assessments or implementation reports did not have sufficient impact on the operational management of the SFs or on the development of the sense of responsibility of the partners affected by the development of the programmes.

4.26. Just as for monitoring, the results of the ongoing assessments of the 1994-1996 Objective No 2 SPDs, CSFs and OPs reveal a divergence between the financing plan and the implementation of the programmes. Because of this situation, the need to utilise appropriations becomes the main concern and as a result the importance of the role of evaluation in the analysis of the area in question may be under-estimated, especially as regards consistency between the objectives, the resources allocated and the impact of structural interventions.

4.27. The evaluations' conclusions and recommendations are often self-evident to the intervention managers, although they may be useful for other parties involved. This is the case for both ex post evaluations (see paragraph 5.8) and ongoing assessments under the partnership system.

The objectivity of evaluations

4.28. The lack of a precise frame of reference is a hindrance to the application of suitable techniques that can be objectively applied in specific evaluation situations so as to produce easily verifiable results and guarantee the independence of the assessors. On occasion, however, the recommended method is not objectively justified, or assessors are all too willing to adapt their approaches to the difficulties encountered or select indicators on the basis of the information available.

4.29. Furthermore, independence from managers does not always appear to be guaranteed, particularly in the field of information (see paragraph 6.11), while evaluation reports do not always give the source of the information on which their observations are based. This failing may raise doubts as to the reliability and validity of the information submitted in the evaluation reports.

4.30. Objectivity should be guaranteed by a verification system based on the quality of the indicators (of implementation, results and impact) used, the reliability of the basic information and the degree of implementation of the methodology. However, the Commission departments analyse the information and results submitted under the evaluation procedure, but do not carry out any on-the-spot checks or verification as to how well the evaluation systems used in the Member States actually work.

The evaluation contracts awarded by the Commission

4.31. The audit of the Commission's procedures for awarding evaluation contracts showed up a number of shortcomings - unsatisfactory selection procedures, excessively vague specifications, delays and low quality reports.

4.32. The minutes of the meetings concerning calls for tender do not always specify the judgemental and analytical factors that determined the Commission's choice of consultants. The Court's audit did not provide evidence to show that the best experts from the most appropriate fields of competence were selected.

4.33. In the case of the ex post evaluation of Objective No 2 and all the 1989-1993 CSFs for Objective No 1, the selection procedure was used to appoint a chief consultant accompanied by other national consultants and coordinating consultants, who then subcontracted the evaluation work to other consultants. In this chain of consultants, the Commission may lose its control of the evaluation procedure, which is supposed to guarantee the consistency, uniformity and quality of the results.

4.34. Specifications are sometimes vague in terms of the description of the work to be carried out and the obligation to achieve results. The wording of the contracts is very general. Often, procedures for carrying out the evaluation work are left open pending a further agreement with the Commission on the methodological framework.

4.35. There are often delays in respect of the schedule for submitting reports. Thus, in the case of the evaluation of all the 1989-1993 Objective No 1 CSFs, the contract provided for the report to be submitted on 28 February 1995, when, in actual fact, it was not completed until November 1995. The Commission provided for no penalties for failure to comply with contractual obligations on deadlines and, on the contrary, made the final 40 % payment on 11 July 1995 under the terms of Article 4 of the contract.

4.36. The contracts do not contain clear, precise guidelines as to the methodology to be followed and the results expected. In consequence, the Commission departments may react to the reports in different ways. For example, some of the reports had been approved, even though they were no more than simple descriptions or vague analyses of the usefulness of the work carried out, were based on general statements by the persons questioned, or produced over-general recommendations. In the case of the ex post evaluation of the 1989-1993 Ireland CSF, one department recommended terminating the consultants' contract because of the bad quality of their report, whereas another department approved it. In the case of the ex post evaluation of the 1989-1993 Portuguese CSF, the departments concerned agreed that, rather than providing an assessment, the report did no more than reiterate the aspects of physical implementation already presented by the national authorities. Similarly, it was pointed out that the ex post evaluation report for the 1989-1993 Spanish CSF was too descriptive and that it was not sufficiently analytical. The ex post evaluation of the 1989-1993 Northern Ireland CSF was judged to be incomplete. The report on the ex post evaluation of the 1989-1993 Italian CSF was considered to be badly structured and organised. Although the evaluation report for the 1989-1993 Greek CSF has chapter by chapter conclusions, there is no general conclusion.

5. THE EX POST EVALUATION OF OBJECTIVE Nos 1 AND 2 CSFs (1989-1993)

Introduction

5.1. Assessors are often faced with the following difficulties:

(a) initial programming that has often been carried out hastily and without a prior appraisal of the programmes, in particular as regards intrinsic economic fundamentals, especially with regard to Objective No 1 regions.

(b) the absence of quantified objectives and result indicators;

(c) gaps in the statistics with regard to the socio-economic situation both before and after Objective No 2 SF intervention, in particular with regard to the industrial fabric, investments and gross regional product. The statistical tools used do not always correspond to the area selected for analysis (boroughs, districts, employment catchment areas);

(d) the fact that certain measures have not been completed;

(e) the existence of exogenous factors that help explain the development of the socio-economic situation;

(f) where the allocation of aid is not linked closely to regional or local needs, the heterogeneous nature of the projects and their objectives;

(g) it is hard to identify the intrinsic effects of the measures and programmes, because of the barrage of measures aimed at the same region or area;

(h) the difficulty of measuring the indirect effects of a programme (in particular as regards employment), because no system of observation was set up for this purpose when the programmes were launched and no detailed inquiry has been carried out on this subject since their completion.

The ex post evaluation of the Objective No 1 CSFs (1989-1993)

5.2. The ex post evaluation of the Objective No 1 CSFs was coordinated by a central consultant and comprises the ex post evaluations of the CSFs concerned that were carried out by national consultants between January 1994 and June 1995. The consultant responsible for coordination developed the methodology that was to be applied, which made use of analytical methods, at both macroeconomic and measure/project levels. The approach followed aimed, firstly, to analyse the strategy and intrinsic logic of the CSFs, OPs and measures, secondly, to provide a preliminary evaluation of physical progress and impact, and, finally, to examine the implementation systems. This procedure is summarised in Annex I.

5.3. Although generally consistent, the proposed methodology does have shortcomings. Thus, it provides that analysis of the intrinsic logic and strategy of the CSFs, OPs and measures should be based on a uniform structure of objectives, linked to a standard classification of activities. This approach might allow comparison between Member States, but it does not necessarily allow objectives to be listed in order of importance on the basis of both the Member States' priorities and the criteria of hierarchy, complementarity, substitutability and synergy. Nor does it allow the identification of all inconsistencies in the strategies specific to each of the CSFs and OPs. The methodology is supposed to provide answers to relevant questions about intentional and unintentional effects, but, in many cases, the physical and impact indicators given in the methodological paper are unable to provide them.

5.4. The Court's audit also found the following shortcomings:

(a) as regards the development of the socioeconomic environment of the CSFs and OPs, a comparison was not always made between the initial situation upon which the Community intervention was to act (1988-89) and the situation at the end of it (1993-94). Data were sometimes out of date, particularly those concerning regional differences;

(b) there was often insufficient analysis for the ranking of objectives, which is required by the methodological guidelines. Either it was not done at all, or it was only done partially -either just for those OPs and measures that were selected for specific examination or without systematically applying the structuring criteria. As a result, the ranking of objectives and instruments sometimes did not produce any improvement in the identification of problems of consistency and the relevance and relative importance of priorities;

(c) adjustments are sometimes made to CSFs and OPs without a thorough analysis of the causes;

(d) financial implementation, implementation systems and physical implementation are generally well dealt with in the evaluation reports. However, in certain cases, either the effects of programmes or measures were not examined except in terms of job creation, or the results were formulated in non-quantitative terms. Sometimes the only quantification of effects is provided by applying macroeconomic models and is concerned with increases in GDP, job creation and competitiveness. This shows the inadequacy of the data provided by managers in the context of monitoring systems;

(e) bottom-up analyses did not generally allow conclusions to be drawn in terms of impact at programme level;

(f) Community added value is sometimes only expressed in qualitative terms (for example, the taking into account of the programme logic, the implementation of an integrated strategy or the effect on the organisation process). Furthermore, it was not always investigated systematically;

(g) anticipated long-term effects were not discussed or were discussed only in qualitative terms.

5.5. The Spanish Objective No 1 CSF was the subject of both an ex post evaluation ordered by the Commission and another one organised by the national authorities and co-financed by the Communities. Both studies were drawn up during the same period and pursued the same objectives but were conducted independently and separately.

5.6. With regard to results and impact, the ex post evaluations highlight the physical achievements of the Objective No 1 CSFs, the macroeconomic impact of the CSFs observed in certain national evaluations, and the progress made in the field of planning -although in this field, the programme-based approach encountered certain obstacles (10). They also stress the CSFs' contributions in terms of innovative organisational structures, financial management, monitoring, information systems, coordination and partnership, although with substantial progress still being needed at each of these levels.

5.7. The chief consultant's 1995 summary included some useful recommendations that the Commission could not follow up completely. One of these was that, when programmes were being drawn up, a clearer distinction should be made between Community structural interventions designed to encourage endogenous growth factors and those aimed at the restoration of a regional equilibrium or a raising of living standards. Another concerned striking a balance between the priority given to infrastructure investments and specific regional characteristics that needed development or finding an equilibrium between the fight against unemployment and the need to increase productivity and competitiveness. Finally, it recommended that regional policies should pay more attention to employment.

5.8. The ex post evaluation of all the 1989-93 Objective No 1 CSFs carried out by the national assessors mostly concentrated on the improvement of the Member States' monitoring and management systems rather than on a real exercise in the assessment of the impact of the SFs.

Ex-post evaluations of Objective No 2 (1989-1993)

5.9. The ex post evaluations of the Objective No 2 programmes for the 1989-1993 period were coordinated by an external assessor in the 60 eligible regions. A general analysis was made for each of the regions, while detailed studies were made of 22 regions. The main aim of these evaluations was to analyse the impact of these structural interventions on the industrial redeployment process in these regions and draw lessons for the next programming phase. A summary of this evaluation approach appears in Annex II.

5.10. Although a common framework was envisaged, the approaches adopted are very heterogeneous. This concerns particularly the priorities adopted and the depth of the analysis, whether it is descriptive or critical, and the methods and techniques used.

5.11. Analysing strategies and the appropriateness of the CSFs and OPs and the changes that they have undergone in relation to the socio-economic environment seldom leads to a reformulation of the structure of the objectives, sub-programmes and measures in accordance with the logic of regional development or redeployment and on the basis of the criteria of ranking, complementarity and substitutability. Whereas many strategic analyses based on the structure of the CSFs and OPs are partial and uncritical with regard to the strategy concerned, a reformulation of this kind should, after due elaboration, make it possible systematically to identify the strategy's shortcomings in terms of consistency, the weighting of priorities, synergy, complementarity and effectiveness vis-à-vis the socio-economic environment.

5.12. As for the impact of the SF interventions, the assessors are not able, on the basis of a sample of projects and an examination of their results, added value and additionality, to aggregate their observations in terms of individual measures and the socio-economic impact of the programme concerned.

5.13. Few impact evaluations were based on a genuine combination of top-down and bottom-up approaches. This combination was, in particular, used for the evaluation of the Objective No 2 interventions in Tuscany. In this case, the top-down approach used a bi-regional (Tuscany/rest of Italy) econometric model to estimate the overall impact of an increase in demand on production by sector, consumption levels and added value, both in Tuscany and the rest of Italy. This model also makes it possible to estimate the multiplier effects by type of intervention (infrastructures, services for businesses, aid for private investment) and the effects on employment (11). The direct effects on employment were estimated using a technique of standard coefficients concerning the various types of intervention. This top-down approach therefore regarded direct short-term effects related to demand. The bottom-up approach, which is based on a sample of projects and measures, is more capable of reflecting long-term, more formative and indirect effects, relating especially to supply and external savings that enable reductions in production costs. Even though these effects are more difficult to isolate and quantify and this approach does not enable an aggregation of the effects in terms of individual measures and programmes (which, however, was the objective of detailed evaluation), qualitative considerations as to the effectiveness of the measures and the degree of additionality of Community interventions constitute a useful complement to the top-down approach. A combination of both approaches can provide real added value in terms of analysis, in particular if it is based on the techniques of modelling or formalisation.

5.14. Impact evaluations that make use of systematic inquiries (such as a number of investigations concerning the additionality of measures or the application of a specific method for estimating net measure-related job creation by comparing an experimental group with a control group) are also rare. However, they do provide real added value in terms of impact evaluation. The effectiveness of measures and programmes would have been more accurately evaluated if assessors had actually conducted the statistical analyses of certain result indicators that they had sometimes said that they wished to carry out. Thus, it is not always possible to find data on the number of companies located in new industrial zones set up with the help of SF aid, or on technological transfers from research establishments to businesses. The meetings with measure and project managers that were generally provided for in the evaluation procedures only seldom made it possible to obtain precise information on the physical implementation, results and impact of the operations.

5.15. In the field of job creation, the results are not clear in terms of the durability of the jobs and the specific contribution ascribable to the structural measures. In addition, the data produced in the evaluation reports is sometimes contradictory.

5.16. With regard to the impact of the Objective No 2 programmes, we can draw the following conclusions from the ex post evaluations:

(a) although the Objective No 2 OPs do not often bring about an improvement in the socio-economic situation, they at least contribute towards avoiding further deterioration (in particular as regards the maintenance of employment (12)) and preparing the ground for industrial redeployment. Indeed, most of the regions were seeking to diversify their economies and their industrial bases and, where possible, cast off their reputations as exclusively industrial areas;

(b) despite the integrated procedure, based on the programme concept, the OPs sometimes consist of a modification of national and regional policies that have already been set up and do not always constitute a programming tool;

(c) the simultaneous existence of the Objective No 2 OPs and the Community initiatives gave rise to confusion and to the overlapping of aid;

(d) SF intervention had positive effects in terms of programming, implementation methods, partnership, synergy, coordination and evaluation. However, the lack of synergy and communication between ERDF and ESF managers and measures was often stressed;

(e) as regards additionality, the SFs make it possible to tackle new target areas, carry out projects which would have remained unrealised without Community aid, speed up implementation, carry out more ambitious projects or give a formative character to initial projects. The additionality of different types of measure (aid for infrastructures, business services, aid for private investment) may vary considerably;

(f) the SFs do not often act as a lever for the involvement of private money or, at least, the evaluations omit this aspect;

(g) the evaluations stress that studies, preparatory negotiations, administrative procedures relating to the adoption and implementation of certain projects and the integration of new national policies all involve delays that are often underestimated.

5.17. With regard to management, the evaluations show that certain techniques used for monitoring the programmes were inadequate, and that the internal changes made during the programming period mainly concerned the maximisation of the rate of absorption of appropriations rather than the revision of regional development priorities.

6. ONGOING ASSESSMENTS (1994-1999)

The development and integration of the principles of ongoing assessment

6.1. Ongoing assessments are provided for in the chapter on implementing provisions, dealing with the improvement of monitoring (included in each of the CSFs, SPDs and other forms of intervention). It is stressed that ongoing assessments involve a critical analysis of the data collected during monitoring, explain any differences in relation to the original objectives and assess the intended results of the measure in question. They also contain an opinion on the validity of the current intervention and the relevance of its objectives.

6.2. In the letters that it sent to the Member States during the financial year 1995, the Commission stressed that ongoing assessments were to provide the Monitoring Committees with appropriate information as to whether the programmes were running smoothly, put forward possible reprogramming measures and present an independent, objective picture of the progress made. In the initial phase, it was also proposed that they should concentrate on the operating conditions of the interventions and, in particular, on the indicators, the management abilities of the persons in charge and the procedures used for selecting projects. Subsequently, with a view to proposing necessary changes and adapting programming, evaluation reports were to look at the relevance of the strategies employed, the validity of intervention objectives, the effectiveness of the programmes in terms of the work carried out, the initial results in relation to objectives, the efficiency of the programmes, the correct application of Community policies (environment, public contracts, etc.) and the measurement of results as regards employment and SMEs.

6.3. In addition, the Commission suggested that evaluations should be general and that thematic evaluations concerning one particular sector or geographical area should be developed in a later phase.

6.4. Certain Member States, in particular Spain (13), France (14), Ireland (15) and the Netherlands (16), also drew up frameworks for ongoing assessment. They all feel that ongoing assessments should look at the following:

(a) the development of the socioeconomic context;

(b) strategies and internal and external consistency;

(c) the implementation conditions, including programming and selection criteria, administrative organisation, the monitoring system and project financing channels;

(d) implementation and the results achieved, taking into account intended results;

(e) initial effects obtained, together with an appraisal of expected effects;

(f) costs, with a view to measuring efficiency, i.e. the relationship between expenditure and results.

It was also felt that, where necessary, ongoing assessment, should also lead to proposals for changes in respect of priorities or intended measures.

6.5. The partnership system was introduced in 1988 as a means for ensuring the effectiveness of the SFs, but it has only played its full role in the system of decentralised management since the start of the ongoing assessment of all the 1994-1999 CSFs in 1995. On this occasion, it was decided that the Monitoring Committees should be responsible for establishing the objectives of evaluations and the required quality standards needed to guarantee the overall consistency of the process, adapting specifications to the characteristics of the regions in question whilst respecting the objectives of the exercise and selecting assessors on the basis of shortlists prepared by the persons responsible for evaluation (members of the Monitoring Committee and Community rapporteurs) after a public call for tenders. Ongoing assessments have made a real contribution to strengthening the partnership system, by allowing it to participate in the causal analysis of structural interventions. Tables 5 and 6 show the operational implementation of ongoing assessments and the significant work carried out by Member States under the partnership system in terms of identifying the impact of the SFs.

6.6. The partnership approach to ongoing assessment has also strongly influenced the institutional structures and the national/regional traditions of the public bodies responsible. It has made for more widespread knowledge of the idea of multiannual programming, the evaluation of which is essential to drawing up critical assessments of structural interventions and preparing any necessary adjustments. In the field of evaluation, partnership is particularly well developed in connection with Objective No 1.

6.7. At present, the aim of ongoing assessment work is to supply overall solutions to all involved, from programme designers to managers and political leaders. This work results in very bulky reports containing information from the programmes themselves, prior appraisals, annual implementation reports and ongoing or ex post evaluations. However, according to information gleaned orally during the audits, only some of the results and recommendations are of any real use in terms of assisting decision-making, managing projects, reprogramming or preparing the next generation of interventions.

Implementation procedures

6.8. Of the 389 intermediate evaluations undertaken, 320 concerned Objectives Nos 1, 2, 5b and 6 (17). The implementation of these evaluations is subject to significant delays in several Member States.

6.9. The intermediate evaluation process was originally envisaged as comprising two distinct stages. The first stage involved an assessment of the scope for evaluation and the identification of related problems. The second consisted of the evaluation itself and the subsequent drafting of a report. In practice, these arrangements were seldom applied.

6.10. As a rule, the Commission insisted on intermediate evaluations being carried out by external experts so as to guarantee not only the credibility of individual know-how but also the impartiality of conclusions about the legitimacy of the strategy, the consistency of socio-economic analyses and the objective quantification of the intended effects. Evaluation requirements in the area of the Structural Funds have created in the space of a few years a network of external assessors known to the Commission and to the Member States, which is advantageous insofar as it generates greater expertise in this area but which focuses on a specific market.

6.11. In practice, the Court's on-the-spot checks highlighted the limitations on the independence of the external consultant. Given the limited time available to the assessors and the consultant's de facto dependency on programme managers for the necessary information, the separation of the respective duties is not very rigorous. Too often, the results obtained in terms of performance merely reproduce and update information from the monitoring system while presenting it as an external contribution.

6.12. Furthermore, the usual procedure of an open invitation to tender based on publication in the Official Journal was not always applied with a view to selecting a consultant to carry out assessments. In the Netherlands, the procedure consisted of asking a limited number of consultants on a list of potential assessors to submit a bid. The selection was then made from these bids.

6.13. Although the Commission regularly participates in monitoring committee meetings and is able to win acceptance for its criteria for selecting consultants, its involvement in this process is inconsistent. Thus, in Belgium, Greece and Portugal the Commission is actively involved in the procedure for selecting assessors within the framework of the technical groups or the ad hoc technical accompanying committee which prepare the choice of consultant and monitor the corresponding implementation and evaluation. By contrast, in France the Commission is officially informed only once the two bids deemed to be the best have been selected; these latter are accompanied by a final proposal in respect of which the Prefect of the Region, in his capacity as contracting authority, asks for any observations to be brought to his attention. As regards the Netherlands, the Commission, which takes part in the work of the national accompanying committee which pilots ex ante, intermediate and ex post evaluations of the regions covered by Objectives Nos 1 and 2 and in the work of the monitoring committees, reacted only belatedly to the content of the draft intermediate evaluation report on the Flevoland Objective No 1 SPD.

Priority given to evaluation objectives

6.14. As regards the implementation of ongoing evaluations, the priority given to evaluation objectives varies, with the exception of the examination of the physical and financial results of the measures, which is an important aspect of any ongoing evaluation. Priority developments often correspond to a genuine regional need, such as in Corsica, where the emphasis is placed mainly on dealing with problems of management, monitoring and the links between evaluation and monitoring. Moreover, the pursuit of certain evaluation objectives is hampered by technical problems, such as the measurement of evaluation and impact indicators.

6.15. Very often, ongoing evaluations do not accord sufficient importance to critical appreciation of the regional development strategy chosen, to the viability of the projects financed by the Structural Funds, to the criteria for selecting projects, and to the effects and impact of the programmes' specific and general objectives.

6.16. Community added value, the analysis of which was one of the objectives of the ex post evaluations of the Objective No 1 and Objective No 2 CSFs for the 1989-1993 programming period, is covered only in exceptional cases by ongoing evaluations.

6.17. Changes in the socio-economic context, where Community assistance takes effect and in respect of which evaluation contracts generally provide for monitoring and updating, are dealt with systematically in certain cases (e.g. by means of very detailed progress charts in Wallonia) and in an unsatisfactory manner in other cases, which has a direct influence on the quality and updating of the analyses of strengths and weaknesses which are generally carried out as part of ongoing evaluations.

6.18. The extent and maintenance over time of the internal and external coherence of all of a programme's components (the objectives of the programme, sub-programmes and measures, and the degree of complementarity and synergy between objectives and measures) are examined only rarely in a systematic manner. Thus, in the case of the Brandenburg OP, the financial commitments in respect of priorities 1.1 (productive investments) and 1.2 (infrastructure) account over the 1994-96 period for 138 % and 242 % respectively of the initial estimate, and the assessor provided no justification for the regional development strategy that was implemented. Synergies are sometimes subject to analyses, which are often incomplete. This aspect warrants more systematic treatment, in particular with regard to synergies between ERDF and ESF activities.

6.19. Similarly, where the reorientation of a programme's priorities and of the allocation of resources is an explicit objective of the evaluation, the evaluation does not always produce a consistent overall presentation of the proposed amendments or an explanatory memorandum. Moreover, the implications of the extensions or curtailments proposed in respect of the measures are not always specified.

6.20. Many of the evaluations examined showed that the assessor had not been able to achieve all of the objectives envisaged in the contract, in particular as regards the evaluation of the measure's impact. Moreover, the time and resources available were not always proportionate to the objectives.

Approaches, methods and techniques

6.21. As contracts often formulate evaluation objectives and approaches in general terms, the choice of approaches, methods and techniques is often left to assessors, who sometimes adapt them during implementation to the difficulties they have encountered. Thus, the assessors occasionally make indicators out of the only information at their disposal.

6.22. Techniques and methods have not always been developed to meet evaluation objectives. Moreover, no genuine consensus exists as to the appropriate operational methodology for evaluating regional development; the Commission merely circulates 'best practices` in general terms without actually coordinating or harmonising their implementation in a concrete way. This imprecision, together with disparities between the Member States as regards programming, implementation and evaluation, results in evaluation reports of variable quality and makes it very difficult to compare results at a European level.

6.23. Measuring the impact of measures thus reveals shortcomings in the techniques applied, in particular as regards the characteristics and complexity of the effects to be evaluated:

(a) a detailed search for even more relevant indicators has not always been carried out, and the envisaged indicators sometimes serve only to measure physical and financial results;

(b) impact indicators have not always been given substance;

(c) evaluation of the socio-economic or regional development impact appears to be hampered by the increasing number of measures, sub-programmes and measures (Objective No 1), whilst many programmes for assisting the conversion and/or revitalisation of industry (OPs, Community initiatives and national programmes) may be implemented jointly. Furthermore, Community and national measures sometimes differ in their application, as do the dates on which measures are launched. Lastly, objectives are often unquantified and are formulated too generally (18). In this context, the links between the diagnosis of the socio-economic problems to be resolved and the objectives of the series of measures, sub-programmes and projects implemented, as well as the links between the objectives of this series, do not always appear obvious, especially when one remembers that the physical and impact indicators attributed to the various levels of a programme do not wholly reflect the programme's evaluation requirements. Similarly, it is difficult to identify the results of each programme and to separate them from the results of other private, regional or national measures;

(d) models for estimating the expected effects and the impact of measures on the basis of mid-term or anticipated results are rarely used. The ongoing evaluation of certain national CSFs was based on an input-output model (Portugal) or on a macro-econometric model (Spain, Ireland), whilst the use of regional models continued to be very marginal. The example of the development model for the province of Hainault (Belgium) used in the context of intermediate evaluations is worthy of mention in this connection. The establishment of models describing the effects of the measures is not very widespread either, except in the field of transport infrastructure. Mention should be made of the application of the input-output model to the Expo 1998 project (Portugal) as part of prior and ongoing evaluations, although the estimates should be considered with caution;

(e) estimates of the effects of measures on employment are also still precarious. The application of models and other analysis and enquiry techniques does not supply adequate information concerning, on the one hand, the net number of jobs created (taking into consideration, in particular the number of jobs that would have been created without the measure or the effects of job substitutions), and, on the other hand, the degree of permanence of the jobs created. Moreover, the application of such models and techniques usually provided no details about new jobs which could not be attributed to the implementation of the programme.

6.24. Some evaluation contracts provide, as part of the bottom-up approach, for enquiries in respect of local economic development partners and final beneficiaries, including companies which receive Community aid, with a view to making a quantitative or qualitative assessment of the contribution made by the Community programme. The enquiries' findings are often unrepresentative in the light of the number of participating companies and partners, and are of limited reliability insofar as they are not always subject to scrutiny.

6.25. As regards result and impact indicators, it was observed that:

(a) the time taken to obtain data and statistics is often long;

(b) the existing data cover geographical areas which do not correspond to the areas in question;

(c) data on companies are not always recorded by the establishments concerned, a fact which distorts local statistics if a single establishment is present in the area but the company itself has premises all over the country;

(d) many statistical bodies are unable to subdivide overall data with sufficient precision to target an area or a given series of problems.

6.26. Where the assessor provides more adequate indicators with a view to measuring impact, whether to compensate for a shortcoming or to replace those originally envisaged, he does not substantiate them and leaves the situation as to their operational nature uncertain. Furthermore, he does not always mention the enquiries still to be carried out which will give concrete form to the indicators, nor does he estimate their cost. It should also be stressed that it is difficult to quantify indicators concerning the programme's aims in the absence of information on historical series.

Implementation of Objective No 2 programmes (1994-1996)

6.27. The results not only of the ongoing evaluation of SPDs, CSFs and OPs during the 1994-1996 period but also of the monitoring procedure highlight the imbalance between the financing originally envisaged and the funds actually used to implement the programmes. Despite numerous adaptations of the programmes during the last programming phase (1994-1996), in order to absorb the financial aid, appropriations were transferred from the 1994-1996 programming period to the 1997-1999 one at an average rate of 12 %. The risk of this situation is that it may lead to an overriding emphasis being placed on the need to use up appropriations and an underestimation of the role of evaluation, in particular as regards consistency between objectives, resources allocated and the impact of structural measures.

7. IMPACT OF THE STRUCTURAL FUNDS AND EVALUATION MODELS

Impact of CSFs and programmes

Results of the models as regards the impact of the Structural Funds

7.1. Given their specific characteristics, the models used for macroeconomic evaluation in the field of the Structural Funds (see Annex III) make differentiated assessments of the impact of the Structural Funds as regards:

(a) the implementation of the Structural Funds' objectives in terms of GDP growth, actual convergence of per capita GDP, employment and unemployment;

(b) observance of the conditions for implementing the Structural Funds, such as additionality and durable development with particular reference to controlling public debt and the balance of payments, to income distribution and to respect for the environment;

(c) the effects in terms of supply and demand;

(d) the effectiveness of Structural Fund instruments, such as public investment, investment in human resources and aid for private companies.

GDP growth

7.2. As Table 7 shows, the results of the principal models used to evaluate the impact of the Structural Funds (HERMIN, OGM and DIO) indicate a positive increase in GDP for Cohesion Fund countries and consequently greater convergence. Thus, cumulative growth in the economies of the Cohesion Fund countries from 1994 to 1999 as a result of structural measures under the CSFs appears, in terms of real GDP, to be 5,5 % for Portugal, 5,9 % for Greece, 2,6 % for Spain and 3,3 % for Ireland. These results assume that the principle of additionality is fully observed; they may thus be modified where the principle is not observed (see paragraphs 8.11 to 8.13). Furthermore, the various macroeconomic models produced sufficiently consistent results for the impact of the 1994-1999 CSFs on real annual GDP growth to be considered plausible. Nevertheless, demand-related effects continue to be predominant in the short term, as the effects on supply become apparent only much later. Similarly, the HELM regional model used for the Hainault region evaluates the impact of the SPD in terms of annual growth in total added value as accounting for an increase of 1,7 % to 2,5 % over the 1994-2002 period. By contrast, in the case of the Italian Mezzogiorno, the impact of the Structural Funds on growth seems very limited, or even close to zero. The Commission's QUEST model, by introducing hypotheses on the expectations of businesses and consumers, produces lower estimates of the effects on growth and employment than was the case with the previous models. The gains shown by this model with regard to the potential for growth and the reduction of unemployment only become apparent in the longer term.

7.3. As regards the average acceleration in annual growth due to structural measures implemented as part of CSFs, Table 8 shows that 1 % of growth in GDP generated by the multiplier effects of the CSFs leads, over the 1994-1999 period, to additional annual growth of 0,2 % to 0,3 % for demand and of no more than 0,2 % for supply. Economically speaking, the predominance, as highlighted by the HERMIN model, of the effects on demand, which would disappear if Community aid were discontinued, contributes to achieving the objectives of Article 130A of the Treaty only if the effects on supply are lasting, thereby increasing the productive capacity of the regions concerned. Insofar as the effects on supply become apparent much later than those on demand, the corresponding multiplier effects may be underestimated.

Real convergence in terms of per capita GDP

7.4. The models used within the framework of the Structural Funds show that convergence is not always guaranteed and that some CSF programming may at best reduce divergence rather than develop convergence.

7.5. In the case of Ireland, Bradley (HERMIN 1994) and Honohan (HERMIN 1997) envisage the possible effect of aid under the 1989-1993 and 1994-1999 CSFs on per capita GDP. Bradley considers that the 1994-1999 CSF's contribution to bringing the Irish economy closer to the EU average is modest whereas, in Honohan's view, the CSFs play a significant role in the convergence process. Beutel (1997) predicts that Ireland will catch up with the rest of Europe even without the 1994-1999 CSF, which would increase the annual growth rate by 0,6 %. Similarly, the annual growth rates of Spain, Portugal and Greece would increase for the 1994-1999 period by 0,4 %, 1,1 % and 1 % respectively due to the Structural Funds, which, in any case, has had a positive effect on convergence in terms of per capita GDP.

7.6. Forecasting simulations based on the OGM model of convergence in real per capita GDP in relation to the EU average provide estimates that stress the fact that, even if the trends projected in 1995 are too pessimistic in the case of Ireland (see Table 9), the CSFs' net contribution to convergence remains modest for the Cohesion Fund countries.

7.7. In the case of Hainault, the HELM model takes aspects of real convergence into consideration for the purposes of ongoing evaluation and considers that the province's estimated annual growth rate would be approximately 1,4 % to 1,6 % higher than the European average because of the Structural Funds; these figures would indeed make it possible for Hainault to catch up with other regions.

7.8. These estimates highlight the fact that, given the differences between Objective No 1 Member States' regional or sectoral productive structures, Structural Fund aid produces variable effects. They also confirm that Structural Fund programming must be improved in order for the corresponding assistance to improve the economy's productive capacity with a view to convergence.

Employment and unemployment

7.9. Table 10 summarises the findings of evaluations of the impact of the Structural Funds on employment and unemployment. All of the studies conclude that the CSFs generate net jobs. These new jobs are often created in the non-traded services sector, whereas the productivity effects of the CSFs are mainly concentrated in the industrial sector. Lastly, some studies note a reduction in the number of industrial jobs due to an increase in productivity.

Effectiveness of CSF instruments

7.10. The HERMIN evaluations for Spain, Ireland and Portugal show that aid for public investment, especially in infrastructure, is more productive in terms of growth than aid for investment in human resources (except for Spain) and for private investment. This trend is illustrated in Table 11 which shows, for each instrument, the increase in GDP due to the 1 % increase in total GDP generated by the CSF over the 1994-1999 period. In the case of Greece, the HERMIN study also concludes that investment in 'hard infrastructure` covered by the CSFs contributes more to GDP growth than investment in human resources. The use of the OGM model (Pereira 1991) to evaluate the effects of replacing aid for private investment or for investment in human resources by additional aid for investment in infrastructure shows an improvement in GDP growth and in the convergence process as a result of the Structural Funds.

Additionality and joint-financing requirements

7.11. The aforementioned models also made it possible to evaluate the significance of additionality and of public joint financing. Thus, Honahan (HERMIN, 1997) considers that, without the public joint-financing requirement, the growth rate of the Irish economy would have been 1 % lower and employment would have experienced a smaller increase. By contrast, Honahan envisages a more appreciable reduction in public debt as a proportion of GDP and in the balance of payments surplus if the public joint financing requirement did not apply. In the case of Portugal, the joint financing requirement leads, according to the HERMIN evaluation, to a total additional gain of 1,3 % in terms of GDP, but the development is less positive in terms of public debt.

7.12. The HELM evaluation for Hainault attaches great importance to additionality. According to the evaluation, the average rates of growth in value added for the 1994-1999 period based on the three envisaged scenarios (regional policy including the SPD, the continuation of regional policy without Community aid and regional policy from which the SPD has been deducted) are 2,5 %, 1,7 % and 0,1 % respectively, and the growth rate in terms of employment would fall from 0,8 % to 0,2 %. The industrial situation would also deteriorate considerably. The HELM evaluation also concluded that, if the rate of subsidy varies, aid does not stimulate investment to the same extent and that considerable investment would take place if levels of aid were lower.

Durable development

7.13. Table 10 shows that the effects of the Structural Funds on the balance of trade, the current-account balance or public debt are not always evaluated. Moreover, the various studies reach different conclusions. Thus, the economic expansion generated by the CSFs leads, in certain cases, to the improvement of public financing through taxation whereas, according to other models, the CSFs create budget deficits.

Role of macroeconomic evaluation in the Structural Funds' decision-making process

7.14. Given the short-term effects of the Structural Funds on demand, ongoing evaluations tend to over-estimate the difficulties experienced by the political authorities in using up Community funds and to underestimate the longer-term effects on supply. Reorientation or reprogramming of the OPs or CSFs may also be biased. The fact is that measures or projects which do not use up Community appropriations quickly enough may be discontinued, especially in the fields of innovation or new technology where implementation is complex, in favour of existing structures which are much less productive. However, in such cases the macroeconomic aspects of sustainable development are not always considered.

7.15. In this context, additional macroeconomic evaluations which take account of sustainability, growth or convergence are essential before any decision is taken regarding initial programming and any necessary changes. However, on several occasions in Spain, Greece and the new German Länder, measures were rescheduled on the basis of ongoing evaluation without serious overall macroeconomic analysis. With regard to Objective 1 regions in Germany, no macroeconomic analysis of the impact of the measures was performed when the intermediate evaluation was made.

Impact of sub-programmes, measures and projects

7.16. Very few evaluations are made of sub-programmes and measures, even though they act as the driving force of Structural Fund programming. The fragmentation, lack of definition and inconsistency of many sub-programmes and measures, and the inadequate quantification of their specific objectives, may help to explain the difficulty of measuring their impact.

7.17. This shortcoming complicates any attempt to identify and measure synergies and the degree of complementarity between individual measures or between one programme and another. Furthermore, the links between a programme's general objectives and the specific objectives of sub-programmes or measures are very often unclear. Thus, in several cases, problems of consistency in the Objective No 1 1994-1999 CSF programmes were noted in the ongoing evaluation reports.

7.18. Furthermore, techniques are seldom fashioned to measure the impact of a sub-programme or specific measure. Thus, the macroeconomic models described above evaluate the effectiveness of the CSFs' principal instruments (public investment, private investment and investment in human resources) only in terms of their respective contributions to GDP growth. Expenditure on infrastructure seems to increase the yield of the private capital which it complements, and would appear to speed up convergence more than other forms of investment (see Table 11).

7.19. Nevertheless, the choice and financing of measures or sub-programmes which make up the programme are not neutral in terms of the impact of regional development on lasting endogenous growth. The evaluations performed by the Commission or by the Member States in the context of their partnership as regards sub-programmes or measures are also insufficient to make it possible to examine the consequences of a specific measure on other areas or sectors or to compare it with other measures with similar objectives. Furthermore, analysis of the impact of the measures or sub-programmes must also take into consideration the initial attempts to offset growth against inter-regional equality which determined the choices made, so as to compare expected gains in efficiency with actual results.

7.20. In principle, the projects' relative costs and advantages are analysed with a view to approving financing. Pursuant to Article 5 of Regulation (EEC) No 2083/93 (ERDF), this analysis is compulsory in the case of major projects the total cost of which is more than 25 Mio ECU for investment in infrastructure and 15 Mio ECU for productive investment. These prior assessments of major projects are part of the examination of applications for Community aid and determine the amount of financing. However, ex post evaluation of them is not provided for in the SF Regulations and may not be required by the Commission for the purpose of establishing whether the socio-economic advantages actually obtained reflect the Community resources originally provided. This deficiency in the process for evaluating major projects renders the very concept of evaluation in this area ineffectual, as a prior assessment can be justified only if an ex post evaluation is actually made. Furthermore, these microeconomic and sectoral studies could also consider the potential impact of the Structural Funds on productivity and thus become an effective instrument for analysing this impact in the short or medium term.

7.21. Synergies and complementarity must also be taken into consideration when projects are evaluated. Current evaluations in this connection are still inadequate or superficial. The harmonisation of synergies and complementarity between projects is essential so as not to weaken their desired impact.

7.22. Although ongoing evaluations often tackle problems relating to programme management, they do not often take account of the criteria which determined the selection of projects, such as their sustainability or their contribution to regional development. However, the programme management system must be able to guarantee that the criteria for selecting projects are applied in such a way as to ensure that the programme produces the best possible results. In particular, projects providing very little added value in terms of development should be avoided even if the fact that they consume appropriations very quickly is an advantage. The project selection system must be more transparent for the purposes of inspection and evaluation.

8. CONCLUSION

8.1. The evaluation of Community structural measures is a legal requirement stemming from Council Decisions and is currently extensively implemented in partnership with the Member States. Evaluation has become a system of aid for the design and management of intervention measures, and contributes to improving understanding of their effects and usefulness. For the 1994-1999 period, more than 250 mid-term evaluations were completed by Monitoring Committees (19) in 1997.

8.2. With the revision of the reform of the Structural Funds, evaluation is entering a phase of maturity which should be consolidated without delay by improving the methodology. Much work still needs to be done to determine the most appropriate evaluation methods for measuring the socio-economic impact of the programmes within the framework of regional development, as the MEANS programme has been unable to provide a satisfactory solution.

8.3. Particular attention should also be paid to indicators at all levels as regards their relevance, the scrutiny of information and the attendant costs. In addition, qualitative aspects should be formalised. Without further progress in these areas, the evaluation process may very well stagnate.

8.4. The same is true of statistical bases, the weakness of which casts doubt on the validity of the indicators and has a major influence on the monitoring and evaluation process.

8.5. The current plethora of indicators of varying relevance is detrimental to the quality of evaluation and prevents it from becoming operational. A small number of impact indicators which are essential for justifying or reorientating a structural policy should be carefully selected bearing in mind the constraints imposed by the regionalisation of data and the material possibilities for gathering information. The reliability of these indicators should be checked. The comparability of indicators and their aggregation on a European scale should be improved.

8.6. In order for full account to be taken of evaluations in the decision-making process and in the operational management of the Structural Funds, steps should be taken to ensure that the evaluations are transparent and objective, that their objectives correspond to requirements and that the finalisation date is adequate.

8.7. In order for evaluation findings to be objective, assessors must remain independent of the managers and public authorities concerned.

8.8. The use of a common frame of reference would make it easier to read and compare the evaluation reports and to share practical experience.

8.9. For each evaluation, more operational objectives should be defined. The terms of reference and the technical specifications, which are generally approved by the Commission, should clearly indicate the most appropriate methodology for dealing with regional, sectoral or national aspects or for specifically covering short- or long-term effects. The strengths and weaknesses of any other methodology should be compared with the Commission's methods. Similarly, the frame of reference for each evaluation should include a minimum list of indicators proposed by the Commission for measuring the performance of the programmes' objectives so as to guarantee a degree of consistency which is essential for the purposes of analysis. Other socio-economic and structural indicators may, of course, be used to complement and refine this analysis.

8.10. Given the diversity of approaches to evaluation, the Commission must require assessors to explain their approach much more explicitly when calls for tenders are being issued, in particular as regards the type of effect or impact which is supposed to be measured and the period concerned, with a view to assessing the exhaustiveness of the methods used. In addition, evaluations should contain a detailed analysis of the links between the programmes' objectives and the priorities and measures implemented to achieve them, and of the corresponding inter-relationship between the objectives and results of the programmes, priorities and measures.

8.11. In this context, the sub-division of evaluations between the beneficiaries concerned needs to be improved so as to increase efficiency and to meet specific expectations, in particular in the area of ongoing evaluations. Similarly, greater control over costs should be sought, as the Commission is not always aware of the volume of expenditure allocated to evaluation within the partnership framework.

8.12. The Commission should also indicate which particular aspects of regional policy are to be evaluated: additionality, convergence, partnership, concentration, employment, endogenous economic development or social welfare. The per capita GDP should not be the only criterion for judging the effectiveness of structural measures.

8.13. It should be possible to base estimates of the programme's effects, impact and Community added value on national and regional development models and on models of the impact of the individual measures.

8.14. The macroeconomic results of structural measures should be refined by taking account not only of the imperfections of the underlying models but also of the side-effects of the measures and their cumulative effect on social welfare.

8.15. Macroeconomic models make a fundamental distinction between the effects on demand in the shorter term and structural adjustments in terms of supply. The programming of Structural Fund assistance should place greater emphasis on improving the productive capacity of the economy so as to ensure that the effects of assistance are permanent and to avoid excessive dependency on demand, which would lead to a situation where Community aid would remain essential in the long term.

8.16. As the effects on supply become apparent only over a longer period, ongoing evaluations generally pay more attention to the effects on demand. The Commission should aim to develop the instruments needed to monitor and evaluate effects on the structure of the economy.

8.17. Macroeconomic models also show that a policy of economic growth is not always compatible with durable development. The relationship between the impact on growth and the effects on durable development should be analysed in greater detail and systematic account should be taken of this duality in the programming of Structural Fund assistance.

8.18. The effects of the additionality and public joint-financing requirement are not easy to discern and are not always examined. More systematic and detailed analyses should be carried out in this area.

8.19. Evaluation per se does not profess to replace a political decision-making process which involves broader considerations, but aims to provide an essential tool for the sound and effective management of Community structural measures which takes account of the corresponding socio-economic results.

This report was adopted by the Court of Auditors in Luxembourg at its meeting of 23 September 1998.

For the Court of Auditors

Bernhard FRIEDMANN

President

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

>TABLE>

(1) In particular, these are:

(a) Article 6 of the framework Regulation (Regulation EEC No 2052/88 as amended by Regulation EEC No 2081/93) on the tasks of the Structural Funds and their effectiveness and on the coordination of their activities;

(b) Article 26 of the Funds' coordinating Regulation (Regulation EEC No 4253/88 as amended by Regulation EEC No 2082/93).

(2) 'Prior appraisal` is also sometimes called 'prior assessment`. The Court prefers to adopt the expression 'prior appraisal` as it appears in the Regulations.

(3) Community operations are to complement corresponding national operations or contribute to them. They are set up via close consultation, known as 'partnership`, between the Commission, the Member State concerned and the competent regional and local bodies, social partners and other competent bodies. 'Partnership` is conducted in full compliance with the respective institutional, legal and financial powers of each of the parties concerned.

(4) In its Annual Reports concerning, in particular, the financial years 1990 (chapter 7), 1993 (chapters 6, 7, 8 and 9) and 1994 (chapter 4), the Court had already reported on the numerous shortcomings in the field of evaluation for the 1989 programming period.

(5) This concerns the following evaluations:

(a) For the 1989-1993 programming period:

- prior evaluations of the O1 and O5b CSFs (1990-1991)

ex post evaluations of the O1 CSFs (1994-1995)

- ex post evaluations of the O2 CSFs (1996-1997)

- thematic evaluations (SMEs, telecommunications);

(b) for the 1994-199 programming period:

- prior appraisals of the O1 regional development plans

- prior appraisals of the O5b programmes

- prior appraisals of the 1994-1996 O2 SDPs

- prior appraisals of the 1997-1998 O2 SDPs

- critical analyses of the O1 SDPs and CSFs (1995).

The Commission also requested evaluations by Member State of the impact of European structural policies on the economic and social cohesion of the Union.

(6) DG XXII's memo of 14 March 1991.

(7) (a) The drawing up, in 1994, of the first 'terms of reference` for the on-going evaluation of the CSFs and the Objective 1 forms of intervention. These confirmed the role of ongoing assessment as an aid to decision making for the Monitoring Committees and its implementation within the partnership system. They also specified the procedures for its implementation, in particular the composition and role of the evaluation team;

(b) the 1995 distribution of the 'Common guidelines for monitoring and ongoing assessment` for the Community SFs, which lay down the factors for analysis (overall and specific objectives) and the link between monitoring and ongoing assessment;

(c) the sending of letters to the Member States (1995) reminding them that they were required to carry out ongoing assessment and again specifying the desired implementing procedures.

(8) Such as:

(a) the Brussels conference on the evaluation of European regional policies (16 November 1995) concerning mainly the results of the ex post evaluations of Objective 1 (1989-1993) and prior appraisals for 1994-1999;

(b) the European conference on evaluation methods for SF interventions (Berlin 2 and 3 December 1996) mainly concerning the problems involved in assessing impact at the micro-economic, macro-economic and thematic levels;

(c) the Seville conference of March 1998 aimed at promoting exchanges of experience in the field of evaluation under the partnership system.

(9) This programme is managed jointly by the Commission and an external consultant, whose involvement is financed from budget heading B2-1 8 2 0, for an annual amount of 932 000 ECU.

(10) The programme by programme approach was hampered by the following:

i) the short planning deadlines implying, in particular, the absence of any prior appraisal;

ii) the lack of tangible objectives, quantified indicators for implementation, results and impact;

iii) administrative constraints, in particular where the sub-programmes of a given programme fall under the responsibility of different authorities, or the sectoral and regional approaches overlap;

iv) weaknesses in the Community rules, resulting in particular, in projects rejected for the Objective 1 OPS being accepted in the context of the Community initiatives;

v) the project by project approach of the Cohesion Fund, which also did not contribute towards a harmonious application of the programme by programme approach.

(11) This does not, however, mean that these are job creations, as the econometric model is based on the hypothesis of full employment and companies with production overcapacity during down-turns.

(12) The 8th annual report on the SFs mentions important leverage effects which refer to overall results, such as the creation of jobs or companies.

(13) Evaluación intermedia de programas operativos regionales del objetivo n° 2 (fase 1994-1996). Orientaciones sobre contenidos comunes a recoger en los pliegos de condiciones técnicas, 23 May 1996.

(14) DATAR, Regulatory and administrative timetable.

(15) General criteria to underpin Terms of Reference for Mid-Term Evaluation of Operational Programmes/Community Initiatives.

(16) Evaluatiekader voor de programma's van de Europese Structuurfondsen, Terp Advies- CBEA Pieter Van Run, December 1997.

(17) 36 of which were examined on the spot and/or at the Commission.

(18) E.g. the revitalisation of the countryside, the strengthening of the economic fabric of the region and structural modifications related to regional development.

(19) The Monitoring Committees include representatives of the national authorities concerned and of the Commission.

ANNEX I

Summary of the methodology to be applied as part of the ex post evaluation of Objective 1 CSFs (1989 to 1993)

1. The explicit aim of the ex post evaluation of the Objective 1 CSFs (1989 to 1993) was to:

(a) assess the strategy and objectives selected in the CSFs and to draw the appropriate conclusions for future action. To analyse the consequences for the initial objectives of redefining and adjusting interventions during the implementation period;

(b) take provisional stock of the concrete results of the CSFs and OPs as regards significant measures and make an initial assessment of their impact;

(c) analyse the effectiveness of the implementation mechanisms and draw the necessary conclusions with a view to making future adjustments.

2. Methodological approaches were prepared by the principal consultant, who was responsible for coordinating evaluations made by other consultants on behalf of the Member States concerned:

(a) for the first objective, in order to evaluate the logic intrinsic to the CSFs and OPs and to the adjustments made to the CSFs and OPs since 1989 - and, more specifically, to analyse the causes of these adjustments, the Community added value, internal consistency, external consistency, the relevance of the strategy to existing problems and the appropriateness of financing - the following tasks, based on a top-down approach, were to be carried out:

- collection of information and key data: financial data, ex ante objectives, anticipated output/effects, target groups and areas, main instruments used, number of projects/measures approved and number of projects completed,

- description of the environment and development of the programme (initial situation -current situation); identification of external factors influencing the development of the programme and its success/failure; analysis of the basic problems and causes; formulation of key regional development indicators, including indicators of changes in disparities,

- identification of policy instruments (loans, subsidies and technical assistance),

- prioritisation of objectives, identification of levels of priority and of the links between objectives and instruments;

(b) for the second objective, the analyses to be performed were based on top-down and bottom-up approaches:

- analysis of the output and effects of OPs, measures and projects, account being taken of 'net losses`, and substitution and multiplication effects,

- analysis of the relative effectiveness of measures and combinations of measures; analysis of the causes of successes/failures,

- comparison of output and input,

the aim was also to formulate key and optional indicators;

(c) for the third objective, the aim was to analyse the effectiveness of CSF and OP implementation measures and of a small number of measures, and to identify existing and planned improvements in the planning, implementation, monitoring and evaluation processes; for each Member State, the examination would cover decision-making, planning, financial control, implementation, monitoring and inspection mechanisms, financial channels, partnership, additionality, fund coordination, and greater flexibility and transparency in the simplification and harmonisation process; implementation and monitoring would also be analysed in depth for a small sample of measures/major projects;

(d) ad hoc samples of programmes, measures and projects were chosen with a view to substantiating a small number of physical and impact indicators and to carrying out in-depth analyses.

ANNEX II

Summary of the ex post evaluation approach in respect of Objective 2 (1989 to 1993)

The ex post evaluation of Community assistance under Objective 2 for the 1989 to 1993 planning period, which generally included ERDF and ESF components, comprised:

(a) general comparisons in all Objective 2 regions and areas of the socio-economic situation prior to Objective 2 Structural Fund assistance and of the socio-economic situation after assistance; analysis of the strategy employed in terms of the prioritisation of objectives, priorities and measures, and of indicators, planned and unanticipated effects, synergies between programme components and the utilisation of the various funds. These analyses were to be based on the content of the CSFs and OPs, on annual implementation reports or reports submitted to the Monitoring Committee and on prior and ex post evaluations, where available;

(b) detailed studies covering 22 regions, the aim of which was to:

- study in depth the CSFs, OPs and types of measures adapted,

- analyse institutional structures and programme management procedures,

- evaluate the impact and added value effects attributable to the CSFs and to the OPs,

- evaluate the cost and effectiveness of these effects,

- study the Objective 2 programme's contribution to the regional development process.

These in-depth evaluations employed the bottom-up approach, the aim being - using an examination of operations covering the selection process, typology, management process, the impact and effect in terms of added value, and synergies and additionality - to work upwards from one level to the next, compiling information first about individual measures and ultimately about the whole programme.

ANNEX III

Characteristics of the macroeconomic models used to evaluate the Structural Funds

1. The principal models used in the macroeconomic evaluation of the Structural Funds comprise:

(a) econometric models, such as HERMIN (1) and HELM (2) and QUEST II (3);

(b) calculable general equilibrium models, such as OGM (4) and BBM (5);

(c) input-output models, such as DIO (6) and FIDIMI (7);

(d) structural-residual analysis.

The characteristics of the application of these models in the cases examined are shown in diagram form in Tables III 1 and III 2. Table III 3 reproduces a model of the characteristics of the CSFs.

2. Furthermore, by simulating the evolution of the economies of Objective 1 countries over a period of more than 10 years and on multi-sectoral bases, it is possible to perceive the structural change brought about by the Structural Funds and to evaluate the reliability of the catching-up process which Community assistance is supposed to support and consolidate.

3. As part of its enquiry, the Court had an analysis performed (8) to assess the appropriateness of these macroeconomic techniques for measuring the impact of structural measures and to establish whether they may contribute effectively to the decision-making and strategic aspects of programming.

4. The analysis showed that explanations of the endogenous or exogenous factors upon which the success of the structural programmes depends vary according to the methods and conceptual approaches employed. The fact is that macroeconomic models aim to describe the mechanisms of the economic system of a country or region and to capture or simulate the effects of an exogenous shock such as the assistance provided by the structural programmes. However, no single description of the economic facts is definitive.

5. The HERMIN econometric model takes into account the specific characteristics of each Member State and aggregates effects at a national level. It assumes that economies actually utilise their full production capacity and that the accelerator and multiplier effects related to increases in investment consequently create an expansion in production and employment. This model seems to give satisfactory results in terms of effects on demand. With regard to the effects on supply, taking the Phillips curve into account, the model suggests that a fall in unemployment will bring about an increase in wages whilst limiting the effect on inflation. Also, the effects on productivity are explicitly taken into account by modelling the external saving effects that increases and improvements in human capital and public and private physical capital have on productivity. They are, however taken into account differently from one country to another. The model provides only partial information on durable development, and, in particular, does not consider variations in exchange rates aimed at guar