Special report No 10/2006 on ex post evaluations of Objectives 1 and 3 programmes 1994 to 1999 (Structural Funds), together with the Commission's replies
Date: 2006-07-20
Institutions: European Court of Auditors
Subjects: Structural Funds, aid system, audit, cost-effectiveness analysis, project evaluation, structural expenditure

12.12.2006

EN

Official Journal of the European Union

C 302/1

I

(Information)

COURT OF AUDITORS

SPECIAL REPORT No 10/2006

on ex post evaluations of Objectives 1 and 3 programmes 1994 to 1999
(Structural Funds), together with the Commission’s replies

(pursuant to Article 248(4), second subparagraph, EC)

(2006/C 302/01)

TABLE OF CONTENTS

Paragraph

Page

Glossary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Framework for the evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The basic objectives established for each evaluation . . . . . . . . . . . . . . . . . . . . . . . . .

AUDIT SCOPE . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

EX POST ASSESSMENT OF OBJECTIVE 1 BY DG REGIO . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The Court’s assessment of the quality of the evaluation . . . . . . . . . . . . . . . . . . . . . . . . . .

Preliminary remarks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Evolution of issues confronting the regions since 1994 . . . . . . . . . . . . . . . . . . . . . .

Appropriateness of strategies adopted . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Evaluation of effectiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Evaluation of efficiency . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Evaluation of the impact of Structural Funds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Evaluation of the management systems . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Adequacy of the framework for evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The terms of reference . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Adequacy of supervision by the Commission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Relevance of evaluation recommendations for future programming periods . . . . . . . . . .

EX POST ASSESSMENT OF ESF OPERATIONS (UNDER OBJECTIVES 1 AND 3) BY DG EMPL

The Court’s assessment of the quality of the evaluation . . . . . . . . . . . . . . . . . . . . . . . . . .

Review of shortcomings in the data and in the analysis . . . . . . . . . . . . . . . . . . . . . .

Assessing the impact of ESF operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

I-XI

1-9

6-9

7-9

10-14

15-80

16-66

16

17-20

21-28

29-40

41-44

45-53

54-66

67-75

67-70

71-75

76-80

81-113

82-113

82-87

88

3

4

5

6

6

6

7

7

7

8

8

10

12

12

14

15

15

16

17

17

18

18

19

C 302/2

EN

Official Journal of the European Union

12.12.2006

Paragraph

Page

Evaluation of equal opportunities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Other aspects examined . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Comments on the overall assessment by the Commission . . . . . . . . . . . . . . . . . . . .

89

90-93

94-96

Other insights from the national reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

97-102

Adequacy of the framework for evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

103-110

Recommendations concluding the evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

111-113

CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

114-118

RECOMMENDATIONS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

119-125

ANNEXES:

Annex I — Breakdown of aid under the Structural Funds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Annex II — GDP and the employment rate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Annex III — GDP growth and the Structural Funds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Annex IV — HERMIN simulations and unemployment

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The Commission’s replies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

20

20

20

20

21

22

22

23

25

26

29

30

32

12.12.2006

EN

Official Journal of the European Union

C 302/3

ADAPT

ALMP

CSF

Deadweight

GLOSSARY

Adaptation of the Workforce to Industrial Change (a Community initiative to
promote employment and the adaptation of the workforce to industrial change)

Active Labour Market Policies

Community Support Framework

Change observed among direct addressees following the public intervention, or
reported by direct addressees as a consequence of the public intervention, that
would have occurred, even without the intervention.

Displacement effect

Effect obtained in an eligible area at the cost of another area.

ERDF

ESF

FIFG

European Regional Development Fund

European Social Fund

Financial Instrument for Fisheries Guidance

HERMIN model

Macroeconomic model used to simulate the impact of SF interventions

Inception report

Report containing methodological questions such as the evaluation objectives,
the methodological framework, the questionnaires and the organisations to be
contacted

Monitoring Committee Committee to oversee the strategic implementation of the funding programme

OP

RTD

SFs

SME

Operational programme

Research and technical development

Structural Funds

Small and medium-sized enterprises

Soft outcome

Result difficult to measure quantitatively

SPD

Single programming document

Synthesis Report

Final reports of DG REGIO (concerning Objective 1) and of DG EMPL (concern-
ing Objective 3) about the ex post evaluation of operations financed by the ERDF
and the ESF

ToR

Terms of reference

C 302/4

EN

Official Journal of the European Union

12.12.2006

SUMMARY

I.
The audit concerned the Commission’s ex post evaluations of Structural Funds interventions in Objec-
tive 1 and 3 regions over the 1994 to 1999 programming period totalling in excess of 245 billion euro, includ-
ing matching private and public sector investments.

II.
Ex post evaluation seeks to compare actual expenditure outcomes against the original objectives. It is
the mechanism whereby insights gained during one programming period are exploited to make future SF
expenditure more effective. The ex post evaluations were carried out by the Commission which relied on con-
tracted external consultants.

The audit objectives were to examine if, at the conclusion of the ex post evaluation exercise, the Com-
III.
mission was able to estimate the effects of the interventions relative to the objectives and if the results and rec-
ommendations have been helpful for the revision of the programmes from 2000 onwards.

The starting point for the audit was the final Synthesis Report for each fund. These reports summarise
IV.
the overall assessment of the impact of the funds and capture the elements that should be of relevance when
taking decisions concerning subsequent planning periods. The next step in the audit was to assess the extent
to which any identified deficiencies could be traced back to weaknesses in the ToR or problems during imple-
mentation of the evaluation process.

Evaluating Structural Fund expenditure is an intrinsically complex process. However, the audit identi-

V.
fied weaknesses in the evaluation process that cannot be attributed merely to such inherent constraints.

The Court identified the following weaknesses in the evaluation of SF interventions in Objective 1

VI.
regions:

— there were discrepancies in the underlying data,

— the assessment of the effectiveness of SF expenditure was not robust enough due inter alia to limited data

on outputs and results collected by Member States,

— the evaluation did not fully succeed in comparing the unit costs of projects receiving assistance under the

SFs,

— the evaluation was largely based on a macroeconomic model (HERMIN) which suffered from significant

limitations.

VII.

The Court found that the evaluation of ESF operations under Objectives 1 and 3:

— was in part based on unreliable or incomplete data,

— reached some conclusions which were not adequately supported by the evidence, thereby shedding doubt

on the methodology employed to reach them.

VIII.
The Commission does not appear to have anticipated some of the difficulties involved in the evalu-
ations in the terms of reference which it formulated for them. Nor did the Commission fully address the short-
comings in the ex post evaluation exercise. The Court’s findings indicate that there were weaknesses in the over-
sight of the evaluations.

On the other hand the evaluations yielded useful insights concerning the management systems for

IX.
ERDF and ESF-funded interventions.

12.12.2006

EN

Official Journal of the European Union

C 302/5

However, while some worthwhile and useful recommendations were made as input for future program-
X.
ming periods, the Commission could not benefit fully from the experiences of the 1994 to 1999 programmes.

XI.

The Court recommends that

— the Commission should, for future evaluation exercises, ensure that available data and allocated resources
are in line with set objectives, which must be realistic; and that the terms of reference must be established
accordingly,

— the Commission should give particular attention to the techniques used for measuring the economic
impact of the funds. If a macro-economic model is used it must take proper account of the characteristics
of the economies under review. Ideally it should be able to benefit from micro-data generated at the project
level,

— proper account should be taken of linkages with other studies relevant to the evaluation of the Structural

Funds,

— a number of specific issues might usefully be included in future evaluations, relating for example to pri-
vate sector contributions, ‘financial engineering’ measures, assessing project costs, project application and
approval processes and internal evaluations by national or regional authorities,

— the Commission’s oversight of the evaluation process should be improved inter alia through collaboration

with research institutes and universities.

INTRODUCTION

(b)

interim evaluations,
to improve the management of the intervention;

conducted during implementation

1.
Since 1996, the Commission has been developing the
evaluation of Structural Funds within a single framework. The
term ‘evaluation’ is defined by the Commission as the ‘judgement
of interventions according to their results, impacts and needs they
aim to satisfy’ (1). The main purpose of the evaluation activity is
to support decision making by providing information and judge-
ments about the relevance, economy, efficiency, effectiveness,
utility and sustainability of the impacts of EU interventions. Such
information is crucial for improving existing, or designing new,
interventions and for setting political priorities and allocating
resources.

Evaluations are carried out at various levels and stages in
2.
the programming cycle (six years) the most important of which
are:

(a)

ex ante evaluations, conducted prior to the implementation
of an intervention to contribute to its design and cost-effec-
tiveness;

(c)

ex post evaluations, conducted after the completion of an
intervention to assess the results achieved and lessons to be
learned.

3.
The Court’s audit concerned ex post evaluations which the
Commission carried out in connection with SF interventions in
Objective 1 and 3 regions over the 1994 to 1999 programming
period. For the purpose of this audit, the scope of ex post evalua-
tion is taken as that defined in the MEANS collection (2), namely:
‘Ex post evaluation recapitulates and judges the entire programme,
particularly its impacts. Its aim is to account for the use of
resources and to report on the effectiveness and efficiency of
interventions and the extent to which expected effects were
achieved. It focuses on factors of success or failure, and on the
sustainability of results and impacts. It tries to draw conclusions
that can be generalised and applied to other programmes and
regions’ (3). It is against this background that the ex post evalua-
tion exercise has been assessed.

(1) SEC(2000) 1051 of 26 July 2000: Communication to the Commis-
sion ‘Focus on Results: Strengthening Evaluation of Commission
Activities.’

(2) The MEANS programme was initiated by the Commission as a
response to the demand for developing evaluation methodology. The
MEANS Collection consists of six volumes on the evaluation of socio-
economic programmes.

(3) MEANS Collection, No 1, p. 53.

C 302/6

EN

Official Journal of the European Union

12.12.2006

The ex post evaluation task is part of a ‘process’ that also
4.
draws on the results of ex ante assessment and mid-term review.
Furthermore, evaluations are undertaken at various levels both
national and thematic and culminate in a Synthesis Report.
Together they constitute an important element in the manage-
ment of the Structural Funds and, in particular, the programming
and review of Community assistance. Responsibility for the evalu-
ation process rests with the Commission (in partnership with the
Member States). It is up to the Commission to decide which
aspects of the evaluation to carry out in-house and which it needs
to subcontract. In either case, overall and final responsibility still
rests with the Commission.

5.
Over the 1994 to 1999 period, the aid scheduled under
the Structural Funds (1) amounted to 136,4 billion euro for ‘objec-
tives’ and to 14,5 billion euro for ‘Community initiatives’. Annex I
provides a breakdown of the former amount by Member State
and objective and of the latter by Community initiative. These
funds were supplemented by at least a further 95 billion euro of
matching private or public sector investment (2) bringing the total
investment to over 245 billion euro.

Framework for the evaluation

6.
The ex post evaluations were carried out on behalf of and
under the direction of the Commission. Following a public ten-
dering procedure, a principal consultant was selected for each
specified set of objectives and Community initiatives. In most
cases, the consultant was supported by a team of evaluators in the
individual Member States (3) whose role was to undertake the
necessary evaluations of the respective programmes on the basis
of a common methodological framework. Moreover, the evalua-
tion process was monitored by a steering group. The evaluations
resulted in national reports prepared by the national evaluators
and a Synthesis Report prepared by the contracted consultants for
each specified set of objectives. These reports were completed
between early 2003 and mid 2004.

The basic objectives established for each evaluation

(b) analyse the impact and effectiveness of the Community
actions on specific structural problems and, as far as possible,
to ascertain the Community value added of assistance.

8.
To meet these overall objectives, the terms of reference
(ToRs) established a number of key questions around the themes
of appropriateness of strategy, effectiveness, efficiency, impact,
management and implementation systems, Community added
value and lessons from current and for future programming. To
structure the work, the terms also established a number of speci-
fied tasks and identified a certain number of techniques and
sources of information.

9.
The ToR for each objective, as compiled by the Commis-
sion, thus provided the basis for the work of the evaluators. These
were complemented by the inception report and the methodol-
ogy guide for Objective 1, in January 2002. For ESF operations,
they were complemented by the methodological progress report,
in December 2002, and the evaluation questions, included in
each national evaluation report. For this purpose, the evaluators
relied on literature reviews, questionnaires, stakeholder inter-
views, the analysis of case studies and other field research as
deemed necessary.

AUDIT SCOPE

10.
The general audit objectives were to examine if the evalu-
ations provided the Commission with adequate information to
estimate the effects of the interventions relative to the objectives
and if the results and recommendations have been helpful for the
revision of
the programmes for subsequent programming
periods.

7.
The DGs involved opted for a wide-ranging approach cov-
ering a broad list of issues. The overall objectives of the evalua-
tions were to:

(a) assess the results and impacts of the operations in the Mem-
ber States vis-à-vis their objectives and impacts on persons
and systems;

(1) Comprising the European Regional Development Fund (ERDF), the
European Social Fund (ESF), the European Agriculture Guidance and
Guarantee Fund (EAGGF) and the Financial Instrument for Fisheries
Guidance (FIFG).

(2) Synthesis Report, p. 53.
(3) Hereafter referred to as ‘national evaluators’.

11.
The audit focused on the ex post evaluations concerning
ERDF interventions under Objective 1 and ESF interventions
under Objectives 1 and 3, undertaken respectively by the
Directorate-General for Regional Policy (DG REGIO) and the
Directorate-General for Employment, Social Affairs and Equal
Opportunities (DG EMPL). The relevant Synthesis Reports and a
sample of national evaluations were examined corresponding to
these selected objectives and instruments (4) which, together,
accounted for 124 billion euro, that is 82 % of the total sched-
uled aid under the SFs for the 1994 to 1999 programming
period (5).

(4) National evaluations examined relate to: Objective 1 Belgium, Greece,
Spain, Germany, Italy, Ireland, Portugal, and Objectives 1 and 3 (ESF):
Spain, France, Italy, Germany, United Kingdom.

(5) See paragraph 5 for the overall figures.

12.12.2006

EN

Official Journal of the European Union

C 302/7

For each evaluation, the audit sought to answer three

12.
questions:

(a) Was the evaluation of good quality?

(b) To what extent were identified weaknesses due to deficien-
cies in the framework and supervision of the evaluation
exercise?

(c) Were the resulting recommendations well-based, useful and
well-suited to be taken into account in the following pro-
gramming periods?

13.
In particular, the audit assessed the quality and complete-
ness of the reports in terms of their evaluation of the appropri-
ateness of the policies, and the effectiveness, efficiency and impact
of the measures, as well as how the measures impacted on the
management systems. It also assessed the adequacy of the Com-
mission’s role in the prepartion, monitoring and supervision of
the evaluations.

14.
The first part of this report is dedicated to the ex post
evaluation of ERDF interventions in Objective 1 regions (para-
graphs 15 to 80) while the second part covers the ex post evalu-
ation of ESF operations under Objectives 1 and 3 (paragraphs 81
to 113).

EX POST ASSESSMENT OF OBJECTIVE 1 BY DG REGIO

As an overall assessment of the appropriateness of the strategies
adopted, the report concludes that the strategic approach to Objec-
tive 1 taken in 1994 must be ‘viewed as broadly appropriate in each
of the Member States’. In its view: ‘The most significant criticism of
the strategies was that they did not constitute much more than the
sum of their parts. Whilst each was an adequate response to the iden-
tified needs in particular fields, there was less indication as to how
these different elements contributed towards an overarching develop-
ment strategy.’

In terms of effectiveness, the report indicated that the analysis relied
to a substantial extent on ‘qualitative judgements rather than quanti-
fied comparisons of results with targets’ due to shortcomings in the
available data. As a general comment, the report states that: ‘Many of
the apparently most effective actions have been those of a largely
“mainstream character.” Funding has largely tended to reinforce exist-
ing patterns of economic development related activity, rather than
providing a basis for developing new activities to meet identified
objectives or stimulating more innovative approaches.’

In terms of impact, the report relied, for most regions, on a macro-
modelling approach to assess the impact of the SFs on economic and
social cohesion. It recognised the fact that:
‘The emerging results
inevitably flow to some extent from assumptions made within the
modelling process.’ The report noted that the scale of their impact, in
different cases, reflected the relative importance and allocation of the
CSF resources and factors such as initial factor endowments, eco-
nomic structures, levels of competitiveness and the degree of ‘open-
ness’ of the economy concerned. It drew general conclusions concern-
ing the impact of the structural funds relative to other external factors
and indicated that, while job creation has been substantial, dead-
weight and displacement was also likely to be substantial.

The Synthesis Report also included a series of observations concern-
ing management and implementation. Its overall conclusion concern-
ing these aspects was that: ‘Management and implementation arrange-
ments, though often characterised by weaknesses, were generally fit
for the purpose, and significant benefits were achieved through the
partnership approach. Decision-making structures typically failed to
establish strong strategic direction. However, important benefits in
terms of wider “added value” were achieved through the program-
ming approach and the development of public sector management
capacity and practices.’

15.
Before undertaking the detailed assessment of the quality
of the evaluation, it is useful to provide a brief overview of the
conclusions of the Commission’s evaluation. These are sum-
marised hereunder (see Text box 1).

The Court’s assessment of the quality of the evaluation

Text box 1
The overall conclusions of the evaluation by DG REGIO

The Synthesis report recognised that the Structural Funds have had a
positive impact on the GDP of the Objective 1 regions and that their
overall performance, relative to the EU as a whole, has improved. It
considered that the extent of the beneficial impact has been heavily
dependent on both institutional capacities and factors such as the
structure and openness of the economy and that ‘these aspects are
clearly crucial in particular to the capacity of the economy to respond
to the challenges and opportunities of the process of economic inte-
gration’. It noted, however, that given the disparity of countries
and contexts and because of limitations of the available monitoring
data and lack of clearly defined objectives, it was not possible to
aggregate the results to give a reliable picture of the overall Objec-
tive 1 achievements.

Preliminary remarks

16.
It is important to acknowledge that the broad approach
adopted faced some intrinsic constraints, the principal ones being:
time limitations, a limited range of quantified objectives and a
lack of detailed result/impact indicators. Furthermore, the impact
that can be attributed directly to the programmes and related
measures was often difficult to isolate since the target regions fre-
quently benefitted from other operations that are instrumental in
achieving similar objectives. It was also difficult to identify and
measure the indirect effects of the Structural Funds expenditure.

C 302/8

EN

Official Journal of the European Union

12.12.2006

However, the Court considers that certain weaknesses in the
evaluation process cannot be attributed merely to inherent con-
straints but were due to deficiencies in the evaluation process
which need to be addressed, particularly so before the ex post
assessment of the 2000 to 2006 programming period.

data was not sufficiently analysed and indicate an important rela-
tionship between increases in GDP and increases in the employ-
ment rate which the Synthesis Report did not highlight.

Evolution of issues confronting the regions since 1994

Appropriateness of strategies adopted

The Synthesis Report starts with a brief analysis of devel-
17.
opments in GDP per capita, unemployment and employment
rates in the various regions concerned (1). The report concludes
that: ‘Increased levels of GDP per head have generally not been
the result of increased employment rates resulting from job cre-
ation’ (2). Such a conclusion would be a potentially important one
for policy makers, suggesting that the main goal of cohesion
policy of closing the GDP gap has not been significantly helped
by improving labour participation rates (3), a conclusion of major
relevance to the SFs.

The second chapter of the Synthesis Report examines the
21.
appropriateness of strategies adopted in terms of geographical
and programme balance, as well as the balance between the funds.
The planned and actual programme expenditures (4) of the Struc-
tural Funds are analysed by Member State and by type of fund but
not by the relevant regions — in contrast to what is done in the
report on GDP per capita and employment and unemployment
rates in the various Objective 1 regions. This limits the analysis
that can be performed, as is noted by the report itself in giving the
results of a regression analysis which attempts to relate EU funds
per capita by Member State to the GDP per head of population (5).
Accordingly, the results given in the report are not statistically
significant and, consequently, are of limited value.

From the information presented in the Synthesis Report,
18.
it is difficult to judge the validity of the statement quoted above.
There are many regions where GDP per capita went up, as did the
participation rates. However, there are some regions where either
GDP per capita or the participation rate declined or increased
only marginally. The Court has sought to assess the claim quoted
in paragraph 17 by applying statistical techniques to the data
published in the Synthesis Report. The technical details of this
analysis are given in Annex II and the essential points are briefly
summarised hereunder.

19.
The results obtained indicate that changes in employ-
ment had a statistically significant impact on real GDP per capita.
In particular, a one percentage point increase in the employment
rate was associated with a 0,24 percentage point increase in GDP
per capita over the period under assessment. This result is incon-
sistent with the conclusion of the Synthesis Report quoted above.
It also suggests that there could indeed be some potential to close
the GDP gap by improving labour participation rates — a rel-
evant result from the policy perspective.

To explore further the robustness of the above result, the
20.
Court carried out an identical analysis of 145 regions for which
data was consistently available (i.e. not just Objective 1 regions).
The results, also reported in Annex II, show very similar results
across the Union. These results show that information from basic

(1) Synthesis Report, p. 61.
(2) Synthesis Report, p. 62.
(3) Here referring to the active labour force as a percentage of the work-

ing age population.

However, the Court applied slightly different statistical
22.
techniques, using the same data as in the report. The technical
details of this analysis are given in Annex III and the essential
points are briefly summarised hereunder.

23.
The results obtained support the intuitive conclusion that
the higher the EU funds provided per capita, the higher is the GDP
growth per capita in Member States. This result shows that with
a broader approach, relevant assessments can be made even with
the unnecessarily limited data base used in this important area
under consideration.

24.
A central part of the assessment in this section of the
Synthesis Report is based on a comparison of various categories
of planned expenditure in 1994 and the estimated actual expen-
diture in 1999. However, the data presented suffers from a num-
ber of serious errors or discrepancies which limits the validity of
the conclusions which can be drawn. In particular, in the case of
five Member States, the figure for actual expenditure given under
the ‘total’ column (in the relevant table published in the Synthesis
Report) is different from the sum of the figures which appear
under the individual columns that denote EU, public and private

(4) Synthesis Report, Table 2.1, p. 75, Tables 2.5 and 2.6, p. 88 and 89.
(5) Synthesis Report, p. 75.

12.12.2006

EN

Official Journal of the European Union

C 302/9

sector expenditure respectively (as they appear in the very same
table). These discrepancies are summarised in Table 1. The per-
centage discrepancy, which is shown in the last column of this
table, is significant and implies an inadequate quality control in
the formulation of the report content and results.

Notwithstanding these discrepancies, transforming the
25.
data given on a per capita basis (1) provides some useful observa-
tions which in some contexts differ from what was concluded in
the Synthesis Report. Table 2 gives the actual per capita expendi-
ture ranked from highest to lowest for each category.

Table 1

Summary of discrepancies noted in paragraph 24

Sum of columns (1)

Published total (1)

Difference (1)

Percentage discrepancy

Germany

Italy

United Kingdom

France

Austria

(1) Million euro.

41 015

34 804

3 961

3 339

1 116

48 243

30 547

4 803

4 019

1 040

Table 2

Actual expenditure (1)

7 228

– 4 257

842

680

– 76

– 15

14

– 18

– 17

7

Total Actual/capita

EU per capita

Public Actual/capita

Private Actual/capita

Austria

3 866

Ireland

1 653

Netherlands

1 822

Austria

Netherlands

3 438

Greece

1 580

Austria

1 033

Germany

Germany

2 933

Portugal

1 403

Greece

2 926

Spain

1 143

Ireland

Ireland

Greece

Portugal

Spain

Belgium

France

Italy

2 716

Germany

2 472

Austria

1 725

Italy

1 587

France

1 579

UK

1 445

Netherlands

884

657

652

621

605

578

427

Portugal

Spain

France

Belgium

UK

Germany

Italy

United Kingdom

1 407

Belgium

Average/capita

2 104

1 033

Netherlands

Belgium

Ireland

Portugal

Greece

France

UK

Spain

Italy

739

703

621

582

485

456

381

311

na

419

2 458

1 299

1 039

705

570

448

396

205

174

na

na

377

(1) The same data as published in the report has been used in the compilation of Table 2. As a result, it can be seen that, for the Member

States listed in Table 1, the sum of the three columns in this table do not add up to the total column.

(1) Per capita figures, in this context, have been calculated with reference

to the whole national population.

C 302/10

EN

Official Journal of the European Union

12.12.2006

The table shows that the total actual funds average was
26.
2 104 euro per person. The EU Structural Funds contributed
approximately half of this, while three of the four cohesion Mem-
ber States managed around 50 % more than the average of
1 033 euro per person. Of particular importance is the contribu-
tion of the private sector, especially in the light of the emphasis
on long-run supply side impacts on the regions. It appears that
some Objective 1 regions in Member States, such as Austria, Ger-
many and the Netherlands, have been particularly successful in
combining private investment with the Structural Funds, even
surpassing, significantly in the case of Austria and Germany, the
average contribution from the EU Structural Funds.

added value on a ‘yes’ or ‘no’ basis. Such appraisals and opinions
are often classified too broadly and are not always objectively
verifiable. There was no attempt to set more concrete objectives
by identifying criteria which could have been useful for this pur-
pose and no additional data was gathered for the purpose of
micro analysis. Important considerations for a fuller assessment
of effectiveness, such as deadweight and displacement effects,
were often not considered, and when these aspects were raised,
they were not analysed in sufficient detail.

27.
However, the factors contributing to a higher participa-
tion by the private sector remain unexplored in the report. For
example, was this a result of the nature of the projects chosen, the
skills of the human resources available or indeed a different kind
of assistance that is given to the private sector from the funds? In
the Court’s view, these aspects merited further examination.

31.
Although there are a number of sections which have
valid and useful observations, in general the Court finds this cru-
cial chapter (2) in the Synthesis Report lacking robustness, with
missing data and often weak conclusions and an insufficient or
missing assessment of the results of other thematic studies com-
missioned by DG REGIO for the SFs for the 1994 to 1999
programme.

28.
The assessment of appropriateness was seldom based on
formalised techniques (1) and the practices followed by the
national evaluators were quite different and were generally largely
descriptive. The matter of appropriateness concerning the degree
to which SF interventions address ‘market failures’ was seldom the
explicit subject of ex post evaluation. Similarly, the changes in
strategy during the programming period have not been examined
thoroughly in terms of their appropriateness.

Evaluation of effectiveness

For example, in the ex post assessment section on trans-
32.
port, in the tables dealing with actual and target kilometres of
roads improved by country, about half the target information is
either missing or incomplete, making it difficult to get a robust
conclusion on achievements as compared to targets. Yet, the The-
matic Evaluation (3) of the Impact of Structural Funds on Trans-
port Infrastructures for the same programme period, which is
quoted frequently in the report, provides a more complete and
rigorous assessment of the results by Member State.

29.
Specific chapters in the national evaluations and in Chap-
ter 3 of the Synthesis Report tackle the effectiveness of Structural
Funds, based on the extent to which expected effects have been
obtained and objectives have been achieved. Five themes are
explored, namely Transport, SMEs, R&D, Education and Training,
and Rural Development. Two horizontal themes dealing with the
environment and equal opportunities are also examined.

33.
Of particular relevance to the ex post assessment is the
estimated employment impact of transport programmes that is
given in the Thematic Evaluation Report on Transport. The total
direct and indirect employment impact is estimated at 2,3 mil-
lion person years (4). Furthermore the economic benefits of the
transport programmes given in the study are judged to be signifi-
cant. In Ireland, the reported economic rate of return for road
projects is given as 13 %, while that of Spain is given as 23 % (5).

30.
The analysis is based on a sample of 70 % of the value of
Objective 1 programmes. However, a great deal of emphasis was
placed on qualitative appraisals, for example rating community

(1) This assessment needs identification and examination within a global
framework of the causal links between the needs, the strategic objec-
tives, the objectives of programmes/sub-programmes/measures and
the allocation of resources. It needs also in principle a ranking of the
needs and unsatisfactory socio-economic situations which the CSF
aim to address and a hierarchy of inherent factors to be considered in
order to ensure that resources are optimally targeted towards meeting
needs.

(2) Chapter 3.
(3) A thematic evaluation is an evaluation which transversally analyses a
particular point (a theme) in the context of several interventions
within a single programme or of several programmes implemented in
different countries or regions.

(4) Thematic Evaluation of the Impact of Structural Funds on Transport

Infrastructures, Final Report, November 2000, p. 8.

(5) For example, in the case of Ireland, this result means that for every
100 euro invested in terms of construction and operating costs, the
value of journey time savings, vehicle operating cost savings and safety
benefits, was 13 euro for each year. Thus, the investment effectively
paid for itself in eight years. (Thematic Evaluation of the Impact of
Structural Funds on Transport Infrastructures, p. 8).

12.12.2006

EN

Official Journal of the European Union

C 302/11

These two relevant conclusions of the Thematic Evalua-
34.
tion Report on Transport are not taken into consideration in the
Synthesis Report. Furthermore, the employment impact estimate
in the thematic evaluation is inconsistent with that of the ex post
study. Based on calculations made by the Court, the total employ-
ment impact of Structural Funds over the period 1994 to 1999,
implied by the study (1) (using the HERMIN model and the assess-
ment of the other regions) is 1,62 million jobs, which is incon-
sistent with the estimate of 2,3 million jobs in the thematic
report, especially since transport only accounts for approximately
20 % of all Structural Fund expenditure.

deadweight or multiplier effects. The overall effectiveness in terms
of net effects is unknowable’ (5). This conclusion is not given suf-
ficient attention, especially in the light of the estimate provided in
the thematic study on SMEs. Furthermore, the table in the ex post
study, giving a breakdown of employment created by Member
State, provides no details for Spain, so that the estimate given is
clearly incomplete (6). This aspect merited a more in-depth exami-
nation in the ex post evaluation.

35.
In contrast, the results of the Thematic Evaluation of the
Impact of the Structural Funds on SMEs are used in the Synthesis
Report which states: ‘The Thematic Evaluation of the Impact of
Structural Funds’ on SMEs recommends a shift from grant expen-
diture to financial engineering measures such as seed and venture
capital funds, loans, interest rate subsidies in the future, arguing
that these methods are more sustainable in the long term (2). This
is an important conclusion within the thematic study which the
ex post assessment does well to bring out (3). However, the report
does not develop this issue further despite the important impli-
cations that could be drawn for future programmes.

The Synthesis Report quotes the Thematic Evaluation on
36.
the Impact of Structural Funds on SMEs as having estimated that
‘in the absence of Community support, 70 % of investment
projects would have either not taken place at all, or been smaller
in scale or postponed’. The thematic evaluation ‘further estimates
such assistance contributed to creating more than
that
300 000 additional jobs, even after taking account of “dead-
weight” and substitution effects. The ex post evaluation of Objec-
tive 1 is not able to replicate these figures owing to the lack of
comparable and robust data on which to build such estimates’ (4).
Given the significance of the different results and that the two
studies were prepared for the same contractor, treated the same
subject, dealt with the same period and were published only two
or three years apart, a fuller treatment of their differences was
warranted.

37.
Following the above statement, in a section dealing with
an ‘overall analysis’, the Synthesis Report does give an estimate of
the number of jobs created in SMEs as a result of the Structural
Funds. ‘From the evidence available it appears that Objective 1
supported the development of some 800 000 jobs. This is in
terms of gross jobs and does not take into account substitution,

(1) See section on impact.
(2) Synthesis Report, p. 105.
(3) The specific comment made was: ‘Whilst this would seem a logical
recommendation, the context in which it is applied emerges as an
important consideration’. Synthesis Report, p. 105.

(4) Synthesis Report, p. 108.

38.
In the ‘other topics’ which the Synthesis Report covers
under Effectiveness, there is a useful if brief summary of various
pertinent issues under a number of headings. However, one
underlying limitation is ‘weak comparative data which has forced
a qualitative assessment rather than a detailed project-based
assessment’ (7). Some of this qualitative assessment could have
been better linked to other thematic studies undertaken, such as
the thematic evaluation on RTD. In the ToR, it is stated that: ‘A
broad scoped literature review will be undertaken in order to iden-
tify key issues in the implementation of the Structural Funds’ (8).
However, there are shortcomings in the way in which such impli-
cations are reported, and are linked to subsequent findings. In the
few cases where there is such integration with the assessment, the
benefits on the quality of the report are clearly evident.

39.
For example, the ex post assessment reports that the the-
matic evaluation on RTD notes that the focus of activities would
be better directed at developing region specific innovation strat-
egies to exploit existing RTD capacity, rather than based on per-
ceived ‘Technology Gaps’. The ex post study reports that this was
so in a number of cases, with results being much more successful
when focused on local strengths (9). This example clearly high-
lights the relevance of building on what other reports and studies
have demonstrated, as emphasised in the criteria developed in the
MEANS programme and its update, which was included as part
of the ToR of this evaluation exercise (10).

(5) Synthesis Report, p. 130.
(6) To quote again from the study: ‘The most effective mechanism for the
creation and maintenance of jobs is argued to be direct support for
productive investment, although as this takes no account of dead-
weight and substitution effects, its overall effectiveness cannot be
judged’. Synthesis Report, p. 130.

(7) The implications of weak data and some general comments on this

will be discussed in a later section.

(8) ToR, p. 90.
(9) Synthesis Report, p. 111.
(10) ToR, p. 11.

C 302/12

EN

Official Journal of the European Union

12.12.2006

The national evaluations were seldom based on a forma-
40.
lised or quantitative analysis of effects. Furthermore, the national
evaluators rarely used the results of quantitative analysis carried
out by other parties.

observations, providing little or no assessment of the quality of
management by the final beneficiaries as required in the ToR (5).
Furthermore, the relative efficiency of public private partnerships
and of the tendering procedures employed were not evaluated.

Evaluation of efficiency

41.
The aim of the Synthesis Report under this heading
was the examination of the efficiency in the implementation of
large projects supported in Objective 1 regions. The analysis was
based on a geographically and thematically representative sample
of approximately 60 large projects (1). Efficiency is understood to
refer to the effects being obtained at reasonable cost. ‘The main
objective is to establish unit costs, i.e. the cost per output achieved.
The analysis should also address the quality of the management
of large projects by the final beneficiaries. At EU level, unit costs
should be analysed across Objective 1 regions. Significant differ-
ences in unit costs across regions should be explained where pos-
sible and a range of appropriate costs should be established,
which may be used as the basis for benchmarking exercises in the
future’ (2). These objectives are relevant and form a critical ele-
ment in the efficient management of projects across the EU.

42.
However these objectives were largely not met by the
ex post assessment. This is admitted in the Synthesis Report, with
the explanation that project managers found it virtually impos-
sible to determine the individual component parts of the project
and subsequently could not calculate unit costs (3). Such a broad
statement implies that there is such a poor state of knowledge on
major projects funded by the SFs as to make this objective in the
ToR unattainable. As a minimum, it should have been possible to
arrive at comparable costs for typical outputs, e.g the cost per
square metre of building a factory, or per kilometre of road built.

43.
Some partial results are presented in three tables dealing
with road construction in Member States, environment projects
in Spain and costs per job created in industrial projects (4). How-
ever, there is little evaluation or analysis of these results, for
example to factor in the impact of different labour costs and par-
ticular geographic characteristics in say road construction.

44.
The Synthesis Report reached firm conclusions about the
timing of projects and respect for budget limits. These were, first,
that only one third of the 60 projects reviewed were completed
within the originally planned timescale, with over a third a year
late. Secondly, approximately two thirds of projects examined ran
over budget, with 20 % costing over 30 % more than originally
planned. However, the analysis is limited to a few general

(1) Greater than 25 million euro for infrastructure and greater than

15 million euro for productive investment projects.

(2) ToR, p. 13.
(3) Synthesis Report, p. 143.
(4) Synthesis Report, Tables 4.6, 4.7 and 4.8.

Evaluation of the impact of SFs

45.
The Synthesis Report claims that the primary aim of
Structural Funds is to influence the long-run supply potential of
the economy (6). These supply side effects work through a num-
ber of channels that are designed to: (a) increase investment in
order to improve physical infrastructure as an input to private
increase human capital, due
sector productive activity;
to investment in training, as an input to private sector productive
activity; and (c) channel public funding assistance to the private
sector to stimulate investment, thus increasing factor productiv-
ity and reducing sectoral costs of production and of capital (7).

(b)

46.
The report goes on to discuss the importance of what are
called positive externalities — ‘conditions where private firms
enjoy the use of additional productive factors at no cost to them-
selves’, thereby improving the productivity and cost competitive-
ness of the economy. These considerations are relevant to an
ex post assessment. However, private sector co-financing was
excluded from the analysis because of uncertainties about its
‘driving mechanisms’ (8). This limited approach leads to a severe
understatement of the impact of certain expenditures in the Struc-
tural Funds by excluding one of the major long-term supply side
effects. One clear demonstration of this is the example of a devel-
opment company presented in the Synthesis Report (9) whose co
financed expenditures have attracted significant additional private
sector investment. Instead of focusing on the more directly dis-
cernible impact of SF expenditure on private sector investment,
the study concentrates on attempting to assess the so-called exter-
nalities, with rather limited techniques of estimation as a result of
a lack of studies providing measures of elasticities which are rel-
evant to the regions under consideration.

(5) As noted above.
(6) Synthesis Report, p. 71.
(7) Synthesis Report, p. 72.
(8) ‘Inclusion of private sector co-financing is of questionable value for an
impact evaluation that is based on the use of a formal macro model
and since considerable uncertainty and ambiguity surrounds the driv-
ing mechanisms behind the private sector CSF co-finance expendi-
tures, we exclude them from our analysis’. Synthesis Report, p. 71.
(9) UK-Speke Garston project; in this particular example, additional pri-
vate sector investment of 223 million GBP was attracted, generating
4 600 jobs in direct employment. Synthesis Report, p. 204.

12.12.2006

EN

Official Journal of the European Union

C 302/13

To assess the impact of the Structural Funds on economic
47.
and social cohesion, a macroeconomic modelling approach based
on the HERMIN model was applied to four Member States
(Greece, Ireland, Portugal and Spain) (1) as well as for the regions
of the new German Länder and Northern Ireland (2). This approach
was intended to measure the impact of Objective 1 progammes
in terms of the effects on GDP and employment.

(c) Use is made of planned rather than actual expenditures (3)
which is more reasonably to be expected in an ex ante pro-
jection rather than in an ex post evaluation.

48.
An alternative assessment scenario would be to use
microeconomic data from projects to build up to the total or
aggregate impact of the Structural Funds on a region or country
basis. However, despite the use of standardised data requirements
across Objective 1 regions, there are difficulties in collecting
microeconomic data. This results in attempts to measure impacts
solely from macroeconometric models, giving rise to many
restrictions and assumptions which, for the reasons set out below,
can restrict the assessment significantly, thereby limiting the
policy implications that can be derived.

49.
The principal difficulties identified with the specific appli-
cation of the HERMIN macromodel used in the ex post assessment
under review are:

(a) The structure of the model puts too much emphasis on the
manufacturing sector while the economies of most of the
regions or countries under consideration have seen greater
development in the past 10 years in traded market services,
such as tourism and, in some cases, the financial sector. The
driving mechanisms of these services are different from
manufacturing, which has become more susceptible to the
impact of globalisation. As a result, the orientation of the
macro model needs to be shifted towards this aspect. The
model used did not provide for such a consideration and
there is little evidence in the report that this aspect has been
recognised as a serious deficiency despite the fact that tour-
ism is so important to the economies of quite a number of
the regions concerned.

(b) The model used is based on annual data starting in 1980.
However, due to the fast-changing structure of the regions
concerned over the intervening time frame, structural insta-
bility of the parameters estimated may be a significant limi-
tation of the macro model-based approach adopted. This
aspect was also not taken into account.

(1) The four countries which qualify, in their totality, as Objective 1

regions.

(2) In the case of Italy the planned construction of a HERMIN model fore-
seen in the ToR, was not done due to ‘contracting difficulties’. For the
remaining regions, which did not have a HERMIN model available, a
so-called ‘bottom-up’ estimation was undertaken.

(d)

Impact estimates are significantly biased downwards due to
the exclusion of the private sector co-financing from the
model (see also paragraph 46) as well as the flow of private
sector investment that results from the Structural Fund inter-
ventions. For example, if the SFs provide assistance in the
building of a factory, the impact on the region’s economy is
not limited to the construction of the building but relates
also to the permanent jobs it gives rise to as a result of the
investment by the private sector that takes place following
the construction of the factory.

(e) The impact that Structural Funds have in the long term on
the stock of public infrastructure and human capital (posi-
tive externalities) plays a crucial role in assessing the impact
of the Structural Funds in the Synthesis Report. The HERMIN
model requires externality elasticities to be supplied from
other specific studies. However, as these elasticities are not
available for any of the relevant regions in the EU, use was
made of empirical literature largely based on USA regions
which, as noted within the Report itself,
‘is somewhat
ambiguous about the appropriate magnitude of the externali-
ties’ (4). As a consequence it is not clear to what extent the
estimated impacts are based on judgement or on fact.

(f) As regards additionality, the HERMIN model simulations
compare the situations with and without CSF funding. The
scenario without CSF funding assumes that neither the EU
funds nor the co-financed amounts are spent. The validity of
such an assumption is debatable (5).

50.
The HERMIN model, in the manner in which it has been
applied, thus appears unsuited to the task of estimating the effects
of SF interventions in Objective 1 regions.

(3) The Report states: ‘Ideally we should use the actual ex post realised CSF
expenditures. But these were not available for every country or region,
disaggregated by priority and on an annual basis. In the interests of
uniformity, we have used the planned CSF expenditure data as con-
tained in the CSF 94 to 99 treaty documents. While these give a fairly
accurate total for expenditure, they do not always give an accurate pic-
ture of the ex post scheduling of expenditures.’ (Synthesis Report,
p. 150).

(4) Synthesis Report, p. 161.
(5) As a matter of fact, some studies claim that EU funding crowds out
national investment such that the growth path of the economy with-
out CSF funding would look better than the picture obtained from the
HERMIN simulations.

C 302/14

EN

Official Journal of the European Union

12.12.2006

The Court analysed the results of HERMIN simuilations
51.
concerning changes in the rate of unemployment, against a set of
pre-established hypotheses (1). The outcome is
shown in
Annex IV. The results show significant variations regarding unem-
ployment from those available from other sources.

Finally, in many cases, the degree to which the interme-

52.
diate objectives were attained was not evaluated.

53.
As regards the strategic objective of convergence, the
only mention in the Synthesis Report is ‘that the overall conver-
gence between Member States is progressing but that this is not
occurring to the same extent between regions across Europe.
Whether the convergence is due to the Structural Funds is a
debatable point’ (2).

Evaluation of the management systems

54.
The management and implementation arrangements
adopted for the Objective 1 programmes have a crucial role to
play in contributing to the effectiveness, efficiency and impact of
the Structural Funds. This aspect was addressed in both the ToR
and the Methodology Guide and was taken up in the national
evaluations (3). The national evaluation reports described the
impact on management systems but also identified situations
where the effects on management systems could have been more
pronounced.

55.
The importance of this topic is also recognised in the
Synthesis Report and, in the recommendations section (4), no less
than 13 out of 19 recommendations deal with management
issues.

The Synthesis Report provides a number of relevant
56.
insights into the management of EU Structural Funds and refers
to several conclusions of the Thematic Evaluation of the Partner-
ship Principle that was conducted in 1999. This provides for con-
tinuity in assessment, and some evidence in support or otherwise
of observations made in the earlier study. However, in a number
of areas, the conclusions reached are in conflict with the thematic
evaluation.

57.
The Synthesis Report provides a summary of the average
size of membership of Monitoring Committees(MCs) by Member
State. The overall average was 44, but there was a wide variation
from a low of 15 to a high of 85. Indeed, the report concludes
that the high membership level was regarded as a key constraint
on the efficiency of the Monitoring Committees, and that this was

(1) This sensitivity analysis was carried out for eastern Germany, Spain,

Greece, Ireland and Portugal.

(2) Synthesis Report, p. 183.
(3) These demonstrate the positive influence of the SFs on the develop-
ment of multiannual programming, the application of the programme
concept and of objective-based management, the establishment of
management structures and procedures and in particular systems of
monitoring and indicators, systems of quality assurance in public
works, tendering procedures for government aid, the development of
evaluation systems, capacity and culture, the project selection proce-
dures and control systems. SFs stimulate the development of more
appropriate structures and capacities to handle development projects.

common across all Member States (5). In contrast, the thematic
study concluded that there was no evidence to indicate a loss in
efficiency through extending inclusiveness. Given the significant
difference in the conclusions of the different reports on this
aspect, it would have been appropriate to analyse this aspect more
thoroughly in the Synthesis Report, and to draw more concrete
recommendations on the way forward.

58.
Another area which was linked to the thematic evalua-
tion concerns the management of different funds. Both evalua-
tions concluded that the funds operated according to different
principles and with different financial requirements and that this
restricted synergy between the funds. This important observation
echoes the views already expressed by the Court (6) and under-
lines the need to strive for greater synergy between the funds.

The assessment that is made about the effectiveness of
59.
the overall management systems also demonstrates some weak-
nesses. The Synthesis Report states that: ‘The management sys-
tems were thought to be adequate by the national evaluations in
just under half of the eligible member states’. This is an impor-
tant result and a detailed assessment of both the success cases and
the inadequacies identified could have been expected. Although
some points raised are developed, there is an overall lack of clar-
ity in the Report about what was actually found.

The Thematic Evaluation of the Partnership Principle in
60.
relation to the Structural Funds (1999) found that this principle
provided a number of positive advantages to Structural Funds
implementation (7). The Synthesis Report states that ‘Broadly,
these advantages were also recognised by the Member States
involved in the implementation of the 1994 to 1999 Objective 1
programme, and horizontal partnership working was welcomed
as one of the “added value” aspects of European programming’ (8).
This is a relevant result, and although some highlights of various
cases are presented, including brief comments on negative expe-
riences or difficulties encountered, the discussion could have been
more directed at highlighting improvements that could be made,
on the basis of the experiences mentioned, so as to make the part-
nership principle more effective.

(5) Synthesis Report, p. 196.
(6) Opinion No 2/2005 on the proposal for a Council Regulation laying
down general provisions on the European Regional Development
Fund, the European Social Fund and the Cohesion Fund (COM(2004)
492 final of 14 July 2004) (OJ C 121, 20.5.2005, p. 18); Special
Report No 7/2003 on the implementation of assistance programming
for the period 2000 to 2006 within the framework of the Structural
Funds, paragraph 90 (OJ C 174, 23.7.2003, p. 22).

(7) These included: greater effectiveness in programme development and
monitoring, more effective project selection, greater legitimacy and
transparency in decisions and decision-making processes, greater
commitment and ownership of programme outputs, etc. See Thematic
Evaluation of the Partnership Principle, p. II.

(4) Synthesis Report, section 9.1.

(8) Synthesis Report, p. 188.

12.12.2006

EN

Official Journal of the European Union

C 302/15

The project application/selection process is identified as
61.
one of the two most frequently cited processes that caused admin-
istrative problems. A number of examples are given, indicating a
lack of clarity in the procedures and an urgent need for the Com-
mission to issue a best-practice benchmark. A table is provided
giving the duration of the project selection process by country,
although no data is provided for three Member States. The dura-
tion of the process for the remaining Member States ranges from
two to nine months with an average of five months. The evidence
provided implies that there is still work to be done in order to
streamline the application process. In this context, no thorough
analysis has been conducted concerning the respective merits
and/or effectiveness in the choice between the competitive pro-
cess and the queuing process.

Due consideration must also be given to the monitoring
62.
and control process. It is claimed, in the Synthesis Report, that the
‘Financial control systems appear to have functioned relatively
well within the Member States under the Objective 1 1994
to 1999 although some adjustments needed to be made to
accommodate the new Financial Regulations in 1997. In the
majority of Member States financial control systems were felt to
be fit for the purpose and operated relatively efficiently. Financial
control appears to have been generally reliable and respected the
5 % rule’ (1). This finding does not coincide with the various
reports of the Court (2) which indicate that the 1994 to1999
Structural Funds programme suffered from significant financial
control deficiencies.

63.
The other element of the monitoring process that gives
rise to administrative problems relates to the non-financial moni-
toring and evaluation systems. Monitoring comes in for heavy
criticism. The Synthesis Report concludes that
‘monitoring
remained a major weakness despite the emphasis on developing
its role since the first programming period. The available data is
fragmented, inconsistent, overly focused on physical outputs and
often of doubtful reliability’ (3). It also states: ‘Across all of the
Member States there was a lack of quantification to the objectives
defined at the start of the programme’ (4). Of the 10 Member
States listed, seven are classified as having a poor level of quanti-
fication, one as variable and two as adequate only at the start of
the programme. Brief summaries are given for each Member State.
However, no further insight is given.

managing authorities to achieve an oversight on progress within
the programmes, as well as posing a problem for later evalua-
tions’ (5). These findings must also cast doubt on the validity of
the earlier conclusion in the report that the financial management
of the programme performed well if monitoring information,
which is such an important input into the management process,
was deemed to have been ‘fragmented, inconsistent and of doubt-
ful reliability’ (3).

65.
The Synthesis Report also provides some brief comments
on so-called ‘internal evaluations’ (6). The following observation is
made: ‘These internal evaluations were carried out by the minis-
tries or regional authorities, with assistance from universities and
external agencies. The degree to which evaluation was carried out
may, perhaps, be surprising in the light of the weakness of the
monitoring systems. This perhaps is a reflection of the perceived
requirement to undertake evaluation activity, although the results
did often feed through into programme management’ (7). In view
of this statement, it would seem that it would have been useful to
focus more on these internal evaluations, and to build on the
results achieved by the relevant ministries or regional authorities.
In contrast, the Synthesis Report only gives some very general
observations with limited relevance to the overall ex post
assessment.

66.
With regard to the 2000 to 2006 programming period,
the Synthesis Report notes that attempts have been made to
address the monitoring problem experienced in the 1994 to 1999
period (8).

Adequacy of the framework for evaluation

The terms of reference

In the light of the various problems identified in the
67.
analysis of the Synthesis Report it is necessary to focus specifi-
cally on the original terms of reference. These, together with the
inception report and the Methodology Guide, suffered from cer-
tain weaknesses. In particular:

64.
It is also stated that the computerised systems set up for
monitoring the 1994 to 1999 programme ‘were frequently criti-
cised due to their lack of integration across programmes and
between funds. This had an impact on the ongoing ability of

(a) They did not address the issue of the lack of quantified objec-
tives. Furthermore, the ToR and the Methodology Guide did
not sufficiently address the paucity of adequate indicators.

(1) Synthesis Report, p. 218.
(2) Annual Report concerning the financial year 2003, paragraphs 5.22,
5.32(d), 5.33 and 5.34 (OJ C 293, 30.11.2004, p. 162). Annual
Report concerning the financial year 2004, paragraph 5.30 (OJ C 301,
30.11.2005, p. 143).
(3) Synthesis Report, p. 224.
(4) Synthesis Report, p. 219.

(5) Synthesis Report, p. 221.
(6) Dealing with ongoing evaluations in this context.
(7) Synthesis Report, p. 222.
(8) For example: ‘There is clear evidence that the partnership arrange-
ments have deepened and widened and that there has been increasing
delegation of responsibility for decision making to the programme
partnership’ (Synthesis Report, p. 246).

C 302/16

EN

Official Journal of the European Union

12.12.2006

(b) Similarly, they did not require the evaluator to collect addi-
tional data on performance indicators. The ToR (1) did fore-
see the carrying out of field review ‘as necessary, if data is not
available in the Member States’. However, this point was not
further specified in the Methodology Guide (2).

(c)

Insufficient emphasis was put on the need to assess net
effects (3), with the exception of some economic impacts.

(d) The ToRs under-emphasised the question of effectiveness
and deadweight effects in particular. This is regrettable given
that deadweight is often considered to be a relevant factor in
important areas such as business investment grants, direct
aid for employment and vocational training facilitating
access to employment.

(e)

Insufficient attention was paid to the analysis of the poten-
tial
links between strategy appropriateness, effectiveness
and impact (4). Similarly, there was little attempt to relate the
identified objectives, even if not quantified, to some ranking
of the needs which the CSF aim to address.

(f)

Except for the application of the HERMIN model and the ref-
erence to the critical incidence analysis which was suggested
for the ‘case studies’ (5), no other evaluation techniques were
proposed within the ToR and inception report.

(g) With regard to the ‘country-specific questions’ identified in
the ToR (6), no clear indication was given concerning the
methodology to be used.

68.
In addition, the strengthening of economic and social
cohesion in the EU and the convergence of the less-developed
Member States were not accorded sufficient priority in the ToR
even though they are amongst the main objectives of structural
measures. The inception report did include some questions about
the development models used, the constraints on growth which
are to be addressed and the channels through which an impact on
the regional economy can be expected (7). However, the ex post
evaluation should ideally have included empirical studies on the
convergence between regions and the reduction of disparities as

(1) ToR, p. 16.
(2) Methodology Guide, p. 16.
(3) ‘Net’ effects are estimated by subtracting from gross effects any dead-

weight and displacement/substitution effects.

(4) Especially the links between the results of the analyses of impact and
the analysis of effectiveness and appropriateness of the strategy.

(5) Inception Report, p. 7.
(6) ToR, p. 102.
(7) Inception Report, page 13. Similarly, the Methodology Guide includes
key questions as: ‘What was the overall impact of the structural funds
on supporting regional convergence during the period 1994 to 1999?
What were the economic effects of the Structural Funds investments
in terms of convergence of income levels?’ (p. 24).

a result of such factors as infrastructure investment, develop-
ments in the education system and in R&D capacity.

Furthermore, as an evaluation objective, it would have
69.
been appropriate to identify and analyse specific case-studies rep-
resenting benchmarks for regional development policies, for
instance, in the area of clustering or growth hubs.

70.
At the launch of the evaluation exercise, the question of
appropriateness of the strategy was addressed by means of a list
of issues and questions. However, the evaluation guidelines pro-
vided by the ToR (8) and the Methodology Guide were not suffi-
ciently precise or concrete.

Adequacy of supervision by the Commission

71.
The (general) contract between the Commission and the
consultants selected to carry out the evaluation stipulated that
technical assistance with the evaluation was to be provided under
the responsibility and supervision of Commission officials. The
quality of the evaluation was to be assessed by the Commission
the criteria developed under the MEANS
on the basis of
programme.

72.
The steering group chaired by DG REGIO which included
representatives of other DGs was involved in drafting the ToR,
approving the methodology and detailed work plan (in the Incep-
tion Report) and examining the national reports and the Synthe-
sis Report. A panel of four experts was appointed to assist with
their principal objective being that of ensuring that high-quality
evaluations were produced. However, their intervention came so
late so that the process was unable to deliver significant improve-
ments in evaluation quality.

73.
The contractually agreed schedule was not adhered to.
The submission of the different reports was delayed, with the
result that the final Synthesis Report and the national reports
were not submitted to the Commission within the planned
12-month period, i.e. by 12 November 2002, but only in April
2003, and were published in May 2003. It would appear that the
timetable was excessively tight from the outset and did not
include sufficient time reserves for treating the relatively large
number of changes/amendments to the reports requested by
experts and representatives in the steering group. This has impor-
tant consequences for future ex post evaluations.

74.
The ToR (6) required the examination of two specific
themes by Member State and a list of such themes was proposed.
However none of the national evaluation reports included a spe-
cific focus on this aspect. Although individual chapters some-
times provide statistics or a partial exploration of these themes,
they have not been subject, as required by the ToR, to specific
research.

(8) ToR, p. 94.

12.12.2006

EN

Official Journal of the European Union

C 302/17

As a general conclusion, the oversight of the evaluation
75.
process, by the Commission, needs to be improved. In particular,
given that the lack of reliable data and performance indicators
and of precise objectives was foreseeable from the outset, the
Commission should have ensured that appropriate measures were
taken to address this critical constraint. Furthermore, given the
tight timetable and the inherent constraints, the evaluation should
have been concentrated on essential elements rather than attempt-
ing to undertake such a wide overall assessment of the program-
ming period.

Relevance
programming periods

of

evaluation recommendations

for

future

Given the problems identified in the preceding sections
76.
of this report, the ability of the ex post evaluation process to yield
worthwhile recommendations was limited. While the process
yielded few recommendations about programme content, it deliv-
ered a number of relevant and sensible suggestions on how
their
to improve the management of
implementation (1).

the SFs and of

The report recommends that: ‘The role of the Monitor-
77.
ing Committees needs to be shifted towards one of strategic direc-
tion rather than issues of financial management’ (2). However, the
Monitoring Committees should be able to do both functions, as
required by the regulations.

78.
The evaluations have led to recommendations for the
programming period 2000 to 2006 which mainly address the
questions of management systems. A wide range of themes is
covered and these are generally well suited to the specific situa-
tion of the country concerned.

79.
With respect to the 2000 to 2006 programmes, the key
lessons to be drawn from the ex post evaluation of the preceding
period relate to implementation, in particular with reference to
the case study work undertaken to evaluate the effectiveness and
efficiency of Structural Fund operations.

As regards post 2006, the recommendations proposed
80.
within the Synthesis Report are rather limited and of a general
nature. However, they are still relevant and, as such, are useful.
For example,
it is stated that ‘More use could be made of
independent, ongoing evaluations, providing an objective exter-
nal
is a sensible recommendation,

input’ (3). While this

(1) For example, a case was made for: ‘Providing much greater discretion
to those Member States which have effective domestic monitoring sys-
tems to utilise their own monitoring requirements in relation to
EU-supported initiatives. This would reduce the costs involved and the
data which emerges would probably be more meaningful’. Synthesis
Report, p. 249.

(2) Synthesis Report, p. 249.
(3) Synthesis Report, p. 251.

it would have been of benefit if it had been adopted before the
launch of the ex post evaluation exercise under review.

EX POST ASSESSMENT OF ESF OPERATIONS (UNDER
OBJECTIVES 1 AND 3) BY DG EMPL

81.
Together with the evaluation of Objective 1 regions,
under the direction of DG REGIO, another evaluation exercise
was carried out, in this case under the direction of DG EMPL,
to cover ESF operations under Objectives 1, 3 and 4 and the
Community Initiatives, Employment and Adapt. The ToR for the
latter ex post evaluation anticipated the utilisation of the results
obtained from the evaluation of the Objective 1 regions. How-
ever, despite organisational arrangements intended to facilitate an
effective coordination between the two evaluations, there is little
evidence that this was achieved in practice.

Text box 2
The overall conclusions of the evaluation by DG Employment

The evaluation aimed at assessing the impact of ESF operations in the
Member States, at the level of individuals and of systems. It also
sought to analyse the impact and effectiveness of Community action
on specific structural problems and to ascertain the Community value
added. In the methodology adopted to address these objectives, evalu-
ators were asked inter alia to indicate whether the cases which they
were asked to assess provided support for or contradicted a number
of hypotheses.

The report noted a number of strengths which can be summarised as
follows: Much of the ESF funding was targeted at the long-term
unemployed which was deemed appropriate given the persistence of
the latter. The improvements in the labour market position for the
beneficiaries were commensurate with the resources involved. There
were major improvements in employment services and educational
and training provisions in several national contexts. The resources
used enabled a strengthening of certain policy priorities, such as equal
opportunities, adaptation of the workforce and targeting those with
disabilities. Through the ESF, there was an increased involvement of
social partners and regional authorities. The funds contributed
to improvements in capacities to manage and implement labour mar-
ket interventions at all levels.

However, the report also noted a number of weaknesses. Deployment
was largely driven by availability of resources rather than policy and
took place in parallel and separate from the relevant policy debates.
Interventions were largely concentrated on training and mainly allo-
cated to service providers which is not, in isolation, the most effec-
tive means of helping the long-term unemployed. Consequently, it is
likely that there was a high degree of deadweight associated with the
intervention. Finally, administrative arrangements were perceived as
complex while evaluation arrangements were weak and unstructured.

C 302/18

EN

Official Journal of the European Union

12.12.2006

The Court’s assessment of the quality of the evaluation

Review of shortcomings in the data and in the analysis

82.
The influence of ESF interventions often takes the form
of a contribution or of a potential effect that is intangible and
therefore not easily quantifiable (1). Notwithstanding this intrin-
sic problem, it must still be a prime objective of an evaluation
exercise to assess the impact and, in conjunction with it, also the
question of effectiveness and efficiency. This is what the ex post
evaluation exercise was required to do.

In seeking to analyse the impact of ESF interventions, the
83.
Synthesis Report provides estimations of the number of ESF ben-
eficiaries per country and per objective (2). As a complement to
the national evaluation reports, the ex post evaluation also
attempted to provide evidence for a number of pre-established
hypotheses (3). These considerations are discussed in the follow-
ing paragraphs.

84.
One of the most basic requirements of the ex post assess-
ment of ESF operations relates to the number of beneficiaries of
the programme over the period 1994 to 1999. The report pro-
vides a table which gives the estimated number of beneficiaries as
52 million. However, the additional notes to this table cast seri-
ous doubts on the figures presented (4). The uncertainty about the
accuracy of the data detracts from some of the useful and relevant
observations resulting in other parts of the assessment.

The Synthesis Report (3) provides estimates of impacts
85.
on individuals from ESF interventions. This information is poten-
tially important in a number of aspects, including the calculation
of unit costs of programmes and estimates of beneficiaries still in
employment after 12 months. However, there are factors which
seriously limit the usefulness of the information, notably the lack
of explanation as to how the data has been derived. The report
simply states that: ‘These ratios applied to generate these esti-
mates have been informed by the individual cases examined and
the review of the literature’ (2). Such a statement cannot be con-
sidered as being sufficient and more detailed information on the
derivation of the various estimates is necessary to understand bet-
ter the applicability of these numbers, such as benchmarks of unit
costs and success rate of programmes.

(1) For example, increased labour force ‘flexibility’ as a result of training.
(2) Synthesis Report, p. 124.
(3) Synthesis Report, p. 125.
(4) For instance, one can read in these notes that the figures for France,
which amount to 17 million, were being ‘double-checked’ for ‘double
counting’ while the figures for the Netherlands, which amount
to 8,7 million, are also being ‘double checked with national teams’.
Furthermore, it is indicated that data are largely missing for Germany,
Ireland and the United Kingdom — although the later is listed as hav-
ing 7,9 million beneficiaries and Ireland as having 680 000 beneficia-
ries.

A second unsatisfactory aspect is the analysis accompa-
86.
nying the data. There is a mismatch between what is stated in the
assessment and what is found in the accompanying table. For
example, the text states that ‘in broad terms unit costs were lower
for the unemployed and young unemployed, around 6 000 euro
per worker in employment, and considerably higher and wide-
ranging for those with disabilities and long-term unemployed’ (2).
Yet the table gives the unit costs for training the unemployed as
the highest (at 6 000 euro) and the integration of the disabled as
one of the lowest (at 4 000 euro). Furthermore, it is common
practice to give the range of variation when there is what is
claimed to be ‘wide variation’ between and within programmes.
The analysis in this section is of limited value because it relies
overly on reducing the assessment to a single number rather than
defining a relevant range of values and providing a more com-
prehensive analysis of these ranges found in the various cases
examined.

87.
This mismatch between the commentary and the figures
given occurs in other parts of Chapter 3 (see Text box 3 for some
examples).

Text box 3

For instance, it is stated that: ‘The ESF per capita expenditure was
highest in Ireland, Portugal and Greece (fully Objective 1 countries)
and in Spain (1)’. Yet Table 3.1 gives an ESF expenditure per capita per
year for Portugal of 2 086 euro and for Germany of 405 euro, while
Ireland is given as 12 euro and Greece as 17 euro. Clearly, the figures
in the table do not match the discussion in the text. Even more sig-
nificant is the statement: ‘The ESF expenditure on education relative
to national expenditure was in all countries lower than the relative
expenditure to ALMP. It was highest in Greece and Portugal (2)’. Yet
in Table 3.1, Greece is classified as having no ESF expenditure in the
relevant column giving ESF expenditure on education relative to
national expenditure.

For example in paragraph 3.2.3 (3) it is stated that: ‘The national
expenditure on labour market training as a proportion of GDP ranged
from 0,2 % in Luxembourg to 0,58 % in Sweden (4)’. Yet in Table 3.3,
the expenditure on training in Denmark is shown as 0,94 % of GDP,
almost twice that of Sweden. As another example, consider para-
graph 3.2.6 (5), dealing with expenditure on the integration of the dis-
abled. In the text, one finds the following statement: ‘The ESF was par-
ticularly important in Portugal, Spain, Greece and Austria.’ Yet in
Table 3.6, Spain and Greece are listed as having no ESF expenditure
on the disabled. A further example is found in paragraph 3.2.7 deal-
ing with expenditure on education. It is stated that: ‘The ESF contri-
bution was most significant in Greece, Ireland and Portugal (6)’. But
Greece is listed in Table 3.7 as having no ESF expenditure on educa-
tion and training.

Synthesis Report: (1) p. 78; (2) p. 79; (3) p. 81; (4) p. 81; (5) p. 84; (6) p. 85.

12.12.2006

EN

Official Journal of the European Union

C 302/19

Assessing the impact of ESF operations

88.
The evaluators who took part in the study were asked
to indicate whether the cases assessed provided support for
or contradicted a number of hypotheses that had been formulated
on the basis of an initial review of the literature and the findings
of the inception phase of the evaluation. The study claims ‘strong

supporting evidence for four hypotheses’ (see Table 3 (1)). How-
ever, the data presented does not support the claim of strong sup-
porting evidence. For example, only seven of the 30 evaluators
said there was ‘strong supporting evidence’ for the first hypoth-
esis. A more objective evaluation of these results would have been
warranted.

Observations on the hypotheses addressed in the examination of cases of impact on individuals

Table 3

Strong
supporting
evidence

Some
supporting
evidence

No
evidence to
support
or contradict

Some
evidence
to
contradict

Strong
evidence
to
contradict

No
opinion
at this
stage

1.
The level of impact achieved is
related to the underlying national and
regional market conditions (e.g. results
depend upon the economic cycle and
level of demand and unemployment).

The level of impact is related to

2.
the characteristics of the beneficiary
groups targeted. (e.g. training measures
are more effective for women
re-entering the labour market and edu-
cated immigrants than for low-skilled
workers and youth).

The level of impact is related to

3.
the type of measure. Some measures
show high levels of deadweight and
some are effective at a small scale but
do not provide general solutions.

The level of impact is closely

4.
related to aspects of the structure and
management of programmes including
the quality of analyses that define the
precise scope of interventions. (e.g,
small targeted programmes may be
more effective than large-scale pro-
grammes).

7

14

8

8

18

3

5

17

6

0

0

1

1

0

0

1

0

1

3

17

7

1

0

1

(1) Table 3 reproduces part of Table 4.8 in the Synthesis Report.

C 302/20

EN

Official Journal of the European Union

12.12.2006

Evaluation of equal opportunities

89.
The terms of reference concerning the impact of ESF
1994 to 1999 also required an evaluation of the extent to which
ESF operations provide support to the ‘equal opportunities (1)’
objective. This aspect is tackled in Chapter 7 of the report. How-
ever, it is claimed that: ‘A major finding is that gender disaggre-
gated data were not collected systematically for each of the ESF
Objectives and Community Initiatives across the EU’ (2). This has
limited significantly the ability of the study to evaluate this topic.
It is difficult to accept this result as a major finding of this report
since a preliminary analysis would have revealed this shortcom-
ing, necessitating field work and sample surveys to overcome this
restriction, rather than simply concluding that disaggregated data
is not available.

Other aspects examined

As regards the impact of ESF operations on individuals,
90.
the interventions ‘combined measures’ (3) have not been specifi-
cally examined in spite of the substantial percentage of ESF
resources having been allocated (18 %) to them. As a result, the
conclusions of the Synthesis Report are rather modest: ‘Most of
the evaluation work points to positive outcomes. However, the
evaluation methods tend to focus on identifying the components
of the programmes that are most effective rather than judging the
performance of combined measures compared with single mea-
sure interventions’ (4).

91.
As regards the impact of Active Labour Market Policies
(ALMP), especially with regard to Objective 1 countries, the Syn-
thesis Report draws some conclusions without adequately
explaining how they were reached.

92.
As regards the relative financial contribution of the ESF,
the Synthesis Report (5) concludes: ‘that the combined allocation
of national expenditure and ESF resources does not appear to
have led to a convergence of the patterns of expenditure between
the countries, nor the adoption of norms linking specific eco-
nomic and labour market conditions to resource allocations’ (6).
It is difficult to see how this conclusion can be sustained given the
level of errors in the relevant tables and/or mistakes in the dis-
cussion. Similarly, another important conclusion in this part of
the report is that: ‘These circumstances suggest that not all of the
ESF interventions are likely to have been additional. Additional-
ity,
in the sense of the ESF co-financed interventions being
unlikely to otherwise have taken place, is more likely to have
occurred where ESF resources were focused and concentrated on
particular types of interventions and where the related national

(1) Here refers primarily to gender equality.
(2) Synthesis Report, p.173.
(3) Combined measures enclose intensive counselling and training, work
experience and training and employment incentives and training.

(4) Synthesis Report, p. 122.
(5) Paragraph 3.2.8.
(6) Synthesis Report, p. 86.

expenditure and programmes were relatively small and less devel-
oped compared with countries with similar labour market con-
ditions.’ Here again, it is difficult to see how this conclusion is
reached from the evidence presented in this chapter, especially
given the mismatch that there is between the analysis and the data
given in the tables.

With regard to the hypotheses advanced on the impact
93.
of ESF on systems, a problem arises similar to that described ear-
lier. Strong supporting evidence is claimed this time for seven
hypotheses (7). However, an analysis of the data presented also
indicates that the conclusion is not warranted.

Comments on the overall assessment by the Commission

94.
Giving due weight to the constraints faced by the evalu-
ators, the following critical comments must nevertheless be made
with regard to the Commission’s own assessment of the ex post
evaluation which concluded with a satisfactory overall appraisal.

95.
The evaluation of the report by DG EMPL is based on
eight performance criteria and four possible quality levels,
namely: excellent, good, acceptable and unacceptable. None of
the applied criteria are rated as excellent or unacceptable. Four
criteria are rated as good, namely; meeting needs, relevant scope,
defensible design and clear report. The four other criteria consid-
ered are ranked as acceptable. These relate to reliable data, sound
analysis, credible results and impartial conclusions.

96.
It is significant that DG EMPL assessed the latter four cri-
teria as being less satisfactory than the previous four. However
given the problems encountered as regards reliabilty of the data
and the inconsistencies of the commentary thereon, as high-
lighted in this report, the Court considers that the evaluation
report was unsatisfactory, especially with regard to reliable data,
sound analysis, credible results and impartial conclusions.

Other insights from the national reports

97.
Some pertinent comments can also be made with refer-
ence to the national evaluation reports, with regard to the evalu-
ation objectives and results:

(a) The evaluation results in relation to the impact on individu-
als were often of a qualitative nature; deadweight was not
always estimated.

(7) Synthesis Report, p. 144.

12.12.2006

EN

Official Journal of the European Union

C 302/21

(b)

Impact aspects of a more qualitative nature (1), the so-called
soft outcomes, have not been systematically, examined.

(c) Except for integrated or mixed approaches, few general state-
ments concerning the performance of individual measures
can be deduced from the evaluations.

Concerning the European Employment Strategy, the
100.
evaluation reports highlighted the positive influence of the ESF in
some Member States such as Greece, France, Italy, Portugal and
Spain. However, it was noted in the case of Germany that the top-
down approach associated with the European Employment Strat-
egy had the potential to conflict with the ESF bottom-up
approach (4). At the more general level, the Synthesis Report con-
cludes that the EES has provided a stronger strategic framework
within which to programme ESF interventions (5).

(d)

Important factors such as the contribution to sustainable
development and to socio-economic cohesion have not been
systematically assessed.

(e) Target groups such as the long-term unemployed and young
people seeking to enter the labour force have not systemati-
cally been the subject of specific conclusions in the evalua-
tion of the co-financed measures.

(f) Only in a few cases was it possible to compare expectations
with results; the evaluation was generally hampered by the
lack of specific initial objectives and indicators and the
unavailability of accurate information.

(g) The complementarity between ESF and other actions has
only rarely been taken into consideration in order to explain
certain results. The significance of the productivity increase
of individual ESF beneficiaries, in terms of the benefits to
the firms employing them, was almost never examined
directly.

The sampled national reports do not always present a
98.
thorough analysis of the impact of ESF interventions on national
policies. However, they generally do suggest that the ESF had an
impact on the development of ALMPs and, in some cases, a con-
crete, direct effect was stated. However, in some other instances,
a different assessment was made, namely that it is difficult to
show that the ESF has led to changes in the ALMPs.

99.
Each national evaluation report and the Synthesis Report
describe in a more or less extended way the impacts on systems
as regards employment services; initial education and training;
continuing education and training; accreditation systems and
mechanisms for anticipating needs. The impacts on systems often
concerned a more specific policy context. It was generally diffi-
cult to identify the net impacts of ESF operations (2); only in
exceptional cases, was the net impact effectively delimited (3).

(1) For example, the achievement of qualifications and the soft outcomes
presented in the UK evaluation report (representing ‘distance travelled
toward the labour market’).

(2) This was also stated by some evaluators (Spain and Italy).
(3) For example, in Spanish cases.

As regards the Community value added, the national
101.
evaluation reports have contributed some relevant results (6),
although some of the specified questions have not been dealt with
in some national reports.

The financial implication of policy choices has rarely
102.
been addressed in detail. The Synthesis Report includes a calcu-
lation of unit costs concerning some measures but this is based
on unsatisfactory data, Where cost issues have been addressed,
they would seem to highlight potentially serious findings that
merit further analysis. For example, for Spain, it is claimed that
‘This roughly doubling of resources due to the ESF has led,
according to some experts, to a substantial amount of wasteful
spending. The larger source of funds has been used to expand
operations and not so much to introduce new measures.’ Simi-
larly for Italy, it is claimed that there have been relevant transac-
tion costs associated with the ESF, referring to implementation
issues and the costs associated with partnership building and new
management practices (7).

Adequacy of the framework for evaluation

103.
The following main evaluation topics were specified in
the ToR: the principal characteristics and the accomplishments of
the 1994 to 1999 programming; the impact of the ESF vis-à-vis
national policies and the effectiveness and impact of ESF opera-
tions on individuals and on systems.

(4) One such example was quoted in the case of Germany.
(5) In particular, the EES promotes stronger links between EU programme
funding mechanisms and other policy instruments, regulations and
national programmes including passive labour market policies.
(6) For example, in the case of Germany: support to the objectives that
target disadvantaged sections of the public, sustainable development,
equal opportunities, financial leverage effects for the new Länder. Simi-
larly, various other topics were highlighted in the other national
reports.

(7) National report, p. 95.

C 302/22

EN

Official Journal of the European Union

12.12.2006

A methods report was compiled by the Commission
104.
and published a year later to provide a critical review of the meth-
odologies adopted for the assessment of results and of their
impacts on individuals and systems. This report was intended to
prepare the ground for future evaluations but it is questionable if
this objective has been effectively achieved. This report suffers
from various weaknesses. Its preparation was not based on a suf-
ficiently thorough enquiry as regards to the approaches applied
in assessing the net impact on individuals. As concerns the evalu-
ation of the impact on systems, it does propose a model for the
classification of ESF expenditures and of their anticipated associ-
ated impacts (capacity, capability and responsiveness) (1) though
some issues remain to be addressed (see page 35 of the report).

105.
In the Methods Report, within the context of the ‘Meth-
ods for policy and strategic evaluation in the ESF field’ (2), no con-
crete recommendations have been presented as regards policy
analysis, with the report merely referring to MEANS. Similarly,
there was no reference to the approach to be followed for the
selection of appropriate indicators at different strategic levels.

106.
It should be remarked that the relevant elements for
some evaluation tasks are not always clearly specified in the ToR.
For example: the description of the tasks as concerns effective-
ness is rather abstract, especially with regard to the underlying
concepts, the selection of mechanisms for impact measurement
and the evaluation techniques to be used. In particular, no con-
crete indication has been given as regards the methods of gener-
alising, at the EU level, the direct impact on individuals from the
results of the activities in the different Member States. The for-
mulation of the tasks has not given due regard to the possible
mismatch between quantitative objectives and the data available.
Furthermore, no concrete action has been foreseen to evaluate the
impact on individuals when the indicators are not concretely
established, or when specific statistical evaluations have not been
carried out recently. The other methodological documents, com-
piled subsequent to the ToR, do not contain the desired explana-
tions either.

107.
Changes in programming have not always been
adequately examined. For example, this subject was not addressed
in the studies concerning Northern Ireland, Merseyside and Spain.
Even where changes in programming have been indicated, their
treatment is far from being optimal. In some cases the evaluator
has not provided a clear assessment of the relevant changes and
of their appropriateness (3).

108.
The Commission did not carry out a pre-evaluation sur-
vey or reconnaissance exercise and this has made it more difficult
for it to anticipate the problems that have been encountered by
the evaluators.

There is no clear evidence that the steering group was
109.
sufficiently aware of the lack of data available to the national

(1) Methods Report, p. 24.
(2) Methods Report, p. 44.
(3) See evaluations concerning France, Spain and Italy.

evaluators, nor the impact that this would have on the quality of
the final evaluations or the problems regarding delays. No reme-
dial action was taken by the steering group to address this issue.

110.
As was the case for the Objective 1 evaluation, the
Commission’s oversight of the evaluation process needs to be
improved (see paragraph 75).

Recommendations concluding the evaluation

111.
For most Member States, the recommendations derived
from the 1994 to 1999 ex post assessment have mainly dealt with
the management systems and procedures. They relate to a variety
of issues and are generally well adapted to the specific situations.

112.
Fewer recommendations for 2000 to 2006 have con-
cerned the content of the co-financed policies. Where such rec-
ommendations have been made,
to be of
relevance (4).

they appear

113.
The Synthesis Report brought up a list of strengths and
weaknesses for each Objective (5) and main recommendations for
the 2000 to 2006 period and post 2006 (6). These statements are
rather general although they do appear to be valid in certain cases,
e.g. in highlighting potential differencies between EU-15 and
EU-10 and in the policy response to territorial differencies.

CONCLUSIONS

The evaluation process is of critical importance in terms
114.
of providing information and judgements about the appropriate-
ness, relevance, economy, efficiency, effectiveness and the impacts
of EU interventions. By doing so, it can support the decision-
making process for setting political priorities and allocating
resources. In order for the process to be successful, however,
there are certain prerequisites to be respected, namely, clear objec-
tives for policies and programmes, and performance indicators
which are produced on a regular basis and which are complete,
reliable and relative to the objectives set. In addition, it is neces-
sary to ensure that the resources available to carry out the pro-
cess are adequate for the purpose, both in terms of time and tech-
nical competencies.

(4) Examples are: the need for a stronger link between the planned objec-
tives and the local socio-economic context (Italy); a better targeting of
beneficiaries (France);
integrated intervention
the expansion of
approaches (United Kingdom); to widen the scope of the programmes
to more innovative and risky actions (Spain); to strengthen anticipa-
tion actions in the training domain (Spain); to pay more attention to
the less-favoured regions by putting more weight on variables such as
unemployment,
long term unemployment and social exclusion
(Spain).

(5) Synthesis Report, p. 188 for Objectives 1 and 3.
(6) Synthesis Report, p. 198.

12.12.2006

EN

Official Journal of the European Union

C 302/23

The Court’s audit of the ex post evaluations under review
115.
identified a number of significant shortcomings in the overall
approach adopted and in the quality of the assessments made:

economy to calculate external elasticities in lieu of actual data
for the relevant European regions, which were not available).

(a) Objectives set during the 1994 to 1999 programming period

lacked clarity and coherence.

(b) While there were few performance indicators available, few
measures were taken to overcome this deficiency by, for
example, collecting additional quantitative data.

(c) As a result, an appropriate balance between quantitative and

qualitative analysis was not always achieved.

(d) This situation resulted in conclusions being drawn which

were not supported by adequate analysis.

(e) At the same time, a number of themes which merited being
treated in an ex post evaluation, such as deadweight, conver-
gence and the proposed themes per Member State, were not
always dealt with in the evaluation.

116.
With regard to the various themes of the evaluations,
the following conclusions can be drawn concerning the Objec-
tive 1 evaluation:

(a)

In assessing the appropriateness of the strategies adopted, the
analysis suffers from some serious discrepancies in the data,
as published in the Synthesis Report, concerning the com-
parison between planned and estimated actual expenditure.
Even allowing for limitations in the data available, a more
in-depth analysis could have been attempted to shed more
light on private sector investment linked to SF expenditure.

(b) The examination of effectiveness lacked robustness, inter alia
due to inadequate or insufficient analysis and a failure to take
into account some important and relevant conclusions in
thematic studies.

(c) The examination of efficiency was confined to some general
observations, while the evaluation objectives of establishing
cost benchmarks and addressing the quality of the manage-
ment of large projects were not met.

(d) As regards the evaluation of the impact of the measures, the
evaluation was based primarily on the use of aggregate
macro-models which presents certain limitations. The spe-
cific macroeconomic model used (HERMIN) did not permit
the necessary adjustments to take account of the specificities
of the economies being evaluated (such as the existence of an
important tourism sector). In addition, there were critical
limitations in the estimation of certain key variables (such as
was the choice of proxy data taken from studies of the US

As regards the Objective 3 evaluation, it was in part based on
unreliable or incomplete data while there were significant mis-
matches between the data provided and the commentary thereon.
Furthermore, there was a lack of convincing support for some of
the hypotheses advanced thereby shedding doubts on the meth-
odology employed to reach them.

117.
Many of the shortcomings of the evaluations can be
traced to the circumstances in which they were carried out, nota-
bly the limited information available and the tight time con-
straints set. It should have been possible to anticipate these con-
straints and the evaluation exercise should either have been given
the necessary resources or the scale of the exercise should have
been limited to what was more feasible. Furthermore, the Com-
mission’s oversight of
the evaluation process needs to be
improved.

The ex post evaluations have still provided some worth-
118.
while insights, notably into the management systems underlying
the implementation of SFs. However, the Commission encoun-
tered serious difficulties in delivering the desired in-depth analy-
sis of the effectiveness, efficiency and impact of the funds, largely
due to limitations in the approach adopted and restrictions in the
availability of data. The latter constraint had not been adequately
anticipated at the start of the exercise. These problems were com-
pounded by certain weaknesses in the Commision’s oversight of
the process. Consequently, while some worthwile and useful rec-
ommendations were made as input for future programming peri-
ods, the Commission could not benefit fully from the experiences
of the 1994 to 1999 programming.

RECOMMENDATIONS

119.
A reappraisal of the scope, procedures and approach
used in ex post assessments needs to be done with some urgency,
before the next batch of contracts for ex post evaluations are issued
at the end of the current programming period 2000 to 2006.

120.
To improve the evaluation process, better quality con-
trol procedures need to be introduced and effectively applied by
the Commission so that the problems encountered in the evalu-
ations under review do not recur in future ex post assessments.
Such procedures should ensure:

(a)

that relevant and reliable data is collected regularly and is
available at each stage of the evaluation process;

(b)

that terms of reference are adequate and provide for the
application of appropriate methodologies by the contractors;

(c)

that adequate resources and time are allocated to the evalu-
ation processes;

C 302/24

EN

Official Journal of the European Union

12.12.2006

(d)

that there is appropriate monitoring and supervision by the
Commission so as to ensure the overall quality of the process.

121.
Secondly, particular attention needs to be reserved to
the choice of appropriate techniques for assessing and measuring
the economic impact. If a macroeconomic modelling approach is
used to analyse the impact of the structural funds, it must be
adapted to take due account of the characteristics of the econo-
mies under review. This consideration is particularly relevant in
the situation where the evaluation will need to be extended
to cover the 10 new Member States. In this light, the Court con-
siders that there should be a re-appraisal of the appropriateness
of an aggregate macroeconomic modelling approach (e.g. such as
one based on the HERMIN model). The Commission should
examine whether greater emphasis could be placed on models
that rely on the use of micro-data derived from projects. It is also
likely that a more in-depth evaluation of certain, specific themes
could better lead to the identification of best practices and pro-
vide additional insight into particular issues such as for instance
the impact of clustering, the significance of growth hubs in
regional development
the overall objective of
convergence.

vis-à-vis

122.
Thirdly, greater emphasis needs to be placed on estab-
lishing linkages between the ex post assessments and thematic
studies, and to ensure that results are consistent with each other. If
results appear to be incompatible, then this aspect needs to be
specifically tackled. In the evaluations under review, where link-
ages were developed by the evaluators, the quality of the assess-
ment improved markedly.

Fourthly, a number of recommendations for future
123.
assessments more particularly for Objective 1 Regions may be
made, namely:

(a) There needs to be specific focus on the reasons why private
sector contributions to the structural funds projects differ

markedly between the Members States. This may cast impor-
tant light on the different drivers to economic development
in the various Objective 1 regions.

(b) Another worthwhile topic to explore is whether there should
be a shift in the future from grant expenditure to financial
engineering measures, such as seed and venture capital funds,
loans and interest rate subsidies, which may prove to be more
sustainable and efficient in the long term.

(c) Greater attention needs to be given to the establishment of
unit costs and benchmarks for various types of projects.

(d) Additional attention also needs to be given to the project
application and approval process as the evaluators pointed
out.

(e) There should be a greater focus on internal evaluations by
the relevant ministries or regional authorities in future ex post
assessments.

124.
Finally, it is important to strengthen the capacity of the
Commission to oversee the evaluation process, including the
application of complex macroeconometric models. One possibil-
ity to consider, in this context, would be for the relevant DGs to
seek a closer collaboration with research institutes and universi-
ties specialised in this area of assessment. Appropriate and timely
supervision of contracted consultants is crucial in ensuring that
objectives are met and that value for money is derived from such
ex post assessments.

125.
In conclusion, the ex post evaluation exercise must go
beyond the compilation of distinct assessments, carried out at
given intervals of time, but should be envisaged and managed as
an integral part of an overall process. This is another aspect where
a closer, and long term collaboration with research institutes or
universities could be of significant benefit in providing a more
effective framework within which to integrate the specific reports
and recommendations generated at each stage of the evaluation
process.

This report was adopted by the Court of Auditors in Luxembourg at its meeting of 20 July 2006.

For the Court of Auditors
Hubert WEBER
President

ANNEX I

Breakdown of aid under the Structural Funds — Available resources related to CSF 1994-1999

.

1
2
1
2
2
0
0
6

.

Belgium

Denmark

Germany

Greece

Spain

France

Ireland

Italy

Luxembourg

The
Netherlands

Portugal

United
Kingdom

Austria

Finland

Sweden

Total

E
N

(million euro)

13 985,30 14 333,90 26 965,80

2 245,40

5 762,30

15 236,20

153,80

14 333,90

2 419,70

O1

O2

O3/O4

O5a Fisheries

O5b

O6

748,50

349,40

476,50

25,10

78,90

123,00

308,40

143,20

1 604,80

1 990,00

76,30

55,30

1 257,30

2 474,80

3 866,40

1 888,60

3 282,20

122,20

194,40

680,40

2 293,30

1 498,20

1 757,40

137,60

923,30

15,40

23,60

1,10

6,10

666,20

1 105,70

47,70

153,70

4 693,40

3 460,50

90,80

171,50

104,40

408,30

2,10

837,20

424,90

165,40

354,10

23,80

201,00

475,90

189,30

537,60

41,40

142,70

260,70

96 356,30

15 750,70

15 592,90

905,70

7 054,10

736,60

Total

1 678,40

629,90

18 913,70 14 333,90 32 131,80 11 881,70

5 762,30

19 552,70

46,20

2 127,10

14 333,90 11 501,60

1 111,20

1 220,20

1 171,70

136 396,30

Source: Special Report No 16/98 (OJ C 347, 16.11.1998).

Breakdown of aid under the Structural Funds —
Available resources related to Community Initiatives 1994 to 1999 (1)

(million euro)

Interreg II

Leader II

ADAPT

PME

URBAN

PESCA

Rechar II

Resider II

RETEX

EMPLOI

Konver

REGIS II

PEACE

3 562,3

1 777,2

1 646,4

1 092,5

894,5

301,5

465,0

610,9

609,2

1 858,5

734,9

615,0

303,1

14 471,0

Total

(1)

Names of programmes related to Structural Funds; further references can be found in the
Commission budget, subsection B2.

Source: Special Report No 16/98 (OJ C 347, 16.11.1998).

O

f
f
i
c
i
a
l

J
o
u
r
n
a
l

o
f

t
h
e

E
u
r
o
p
e
a
n
U
n
o
n

i

C
3
0
2
/
2
5

C 302/26

EN

Official Journal of the European Union

12.12.2006

ANNEX II

GDP and the employment rate

A multiple regression analysis was conducted to test the statement in the Synthesis Report that ‘increased levels of

1.
GDP per capita have generally not been the result of increased employment rates resulting from job creation’ (1).

2.
For strict comparability, the same data published in the Synthesis Report has been used in this analysis. In the case of
the dependent variable the percentage change in GDP per capita for each Objective 1 region between 1993 and 2000 was
used, while as main explanatory variables, the percentage change in the employment rate as well as that of the unemploy-
ment rate was used for the same regions, and over the two years 1993 and 1999 as published in the report. The results are
given in Table 1.

Table 1 (1)

Regression of the change in GDP per capita on change in employment (CEMPL) and change in unemployment
(CUNEMP)

Regressor

Coefficient

Standard error

T-ratio

[Prob]

C1

CEMPL

CUNEMP

BE

IR

DX

1,7615

0,24563

0,051927

– 16,8803

38,2350

26,9660

1,2361

0,14237

0,027418

6,2156

6,5314

2,9504

1,4251

1,7252

1,8939

– 2,7158

5,8540

9,1398

R-squared

S.E. of Regression

Mean of dependent variable

0,74449 R-bar-squared

6,1411

F-stat. F (5,44)

25,6414

6,3647

S.D. of dependent
variable

Residual sum of squares

1 659,4

Equation log-likelihood

Akaike information
criterion

DW-statistic

– 164,5015

Schwarz Bayesian
criterion

2,3210

[0,161]

[0,092]

[0,065]

[0,009]

[000]

[000]

0,71546

[000]

11,5126

– 158,5015

– 170,2376

(1) 50 observations used for estimation. BE, IR and DX are dummy variables for Belgium, Ireland and Germany. C1 is the intercept.

3.
From a preliminary assessment of the data, it emerged that three particular regions exhibited significantly different
patterns of change in GDP per capita over the period as compared to the other regions, and which required specific treat-
ment in the regression, as otherwise it would be difficult to draw general conclusions about underlying links. These were;
Ireland, which saw its GDP per capita rise from 81 % to 115 % of the EU average over the period; the Hainaut region in
Belgium, which saw its GDP per capita decline from 82 % to 71 %; and German Objective 1 regions which saw the GDP per
capita rise from a range around 50 % to close to 70 % of the EU average by the end of the period (for example, Thuringen
saw its GDP per capita rise from 52 % in 1993 to 70 % in 2000). To take account of such large relative changes in GDP per
capita, specific statistical (or dummy) variables where included for this purpose which allowed the function to shift for these
three broad regions.

4.
The regression results shown in Table 1 provide a number of relevant points worth noting. Overall, 74 % of the varia-
tion in the change in GDP per capita is accounted for by the ‘explanatory’ variables. Both main explanatory variables, that is
the change in employment rates (CEMPL) and change in unemployment rates (CUNMP) are statistically significant. In the
case of CEMPL, the coefficient estimated is 0,24, suggesting that each one percentage point increase in the employment or
participation rate is associated with an improvement in GDP per capita of Objective 1 regions by a quarter of a percentage
point — thereby, if nothing else changes, closing the GDP gap by this amount.

(1) Synthesis Report, p. 62.

12.12.2006

EN

Official Journal of the European Union

C 302/27

This result directly contradicts the conclusion reached in the synthesis report, and provides a useful indicator of the
5.
potential there is, from an analysis of the results of Objective 1 regions over this period, to close the GDP gap by improving
participation rates — a most relevant result from the policy perspective.

6.
A second feature of note in the results in Table 1 relates to the impact of the changes in unemployment rates on
changes in the GDP per capita. This impact (also discussed in the next section) is also statistically significant, but is rather
small quantitatively, almost of an irrelevant value. Unemployment declined marginally overall in Objective 1 regions, and
this may be the reason for this result.

7.
To explore further the robustness of the main result in Table 1 relating to the impact of changes in the employment
rate and changes in GDP per capita, Table 2 provides the regression results for all regions in the EU-15 and for which data
was consistently available from Eurostat, and not just for Objective 1 regions. The results from 145 regions where derived
from slightly different data than in Table 1. Rather than taking just the rate of change between the beginning and end of the
relevant period, the average rate of change was computed for the whole period. This was judged to give a more represen-
tative summary of developments over a number of years, in this case over the five year period between 1996 and 2000.
(Due to data limitations in the available statistics, an earlier starting point was not feasible for a number of regions.)

Table 2 (1)

Regression of average change in GDP per capita on average change in employment (ACEMPL) and average
change in unemployment (ACUNEMP)

Regressor

Coefficient

Standard error

T-ratio

[Prob]

C

ACEMPL

ACUNEMP

ID

LD

BED

DED

5,2110

0,24119

– 0,0019479

4,7009

5,4309

– 0,64276

– 1,3747

0,12999

0,63540

0,012260

0,85873

0,84670

0,25885

0,17517

40,086

3,7959

– 0,1588

5,4743

6,4142

– 2,4831

– 7,8479

R-squared

S.E. of regression

Mean of dependent
variable

Residual sum of squares

0,62600 R-bar-squared

0,84193

F-stat. F (6,138)

5,2359

S.D. of dependent
variable

97,8204

Equation log-
likelihood

Akaike information criterion

– 184,2101

Schwarz Bayesian crite-
rion

DW-statistic

1,6353

[000]

[000]

[0,874]

[000]

[000]

[0,014]

[000]

0,60974

38,4979
[000]

1,3477

– 177,2101

– 194,6286

(1) 145 observations used for estimation. ID, LD, BED, DED are dummy variables for Ireland, Luxembourg, Belgium and Germany. C is the

intercept.

8.
The results indicate that the coefficient for the average change in employment is practically the same as in the pre-
vious case, that is 0,24, indicating that for all regions, a 1 % increase in the employment rate tends to be associated with a
0,24 % increase in GDP per capita. This result is also statistically significant, as is the whole regression equation. (An addi-
tional dummy variable for Luxembourg has been added, again because the increase in GDP per capita from 161 % in 1996
to 199 % in 2000, is markedly different from the average performance of the other regions).

C 302/28

EN

Official Journal of the European Union

12.12.2006

The impact of the change in the unemployment rate variable is again minimal and in this case is statistically insig-
9.
nificant and negative. The implications from these results are that the impact of changes in the unemployment rate on
changes in GDP per capita does not appear strong — possibly because the unemployed are already accounted for in the
labour force and in the measure of GDP that is used, and any change from unemployed to employed has a less easy impact
to capture in statistical analysis of this type. Certainly, more research in this area seems warranted.

12.12.2006

EN

Official Journal of the European Union

C 302/29

ANNEX III

GDP growth and the structural funds

A multiple regression analysis was conducted to explore the link between the average annual change in GDP per
1.
capita for the Objective 1 area (the dependent variable) vis-à-vis the actual EU funds per capita expenditure in the Objective
1 regions of each member state. The same data and time period (1993 to 1999) is used as that found in the Synthesis Report.

2.
It should be noted that modelling rates of change is a statistically demanding exercise, and often gives low levels of
explained variation in the dependent variable. Nonetheless, the results in Table 1 show that 53 % of the variation in the
dependent variable is accounted for by the different allocations in EU Funds on a per capita basis. This is a statistically sig-
nificant result, and suggests that regions which have had a higher actual EU Structural Fund expenditure on a per capita basis,
also tended to have a higher average GDP growth per capita over the period.

Although the sample is small, and this makes the statistical assessment difficult, each of the 11 observations is an
3.
amalgam of the information in many regions and thus represents a wider information base than the mere 11 observations
would suggest. This result contrasts with that given in the report and suggests that further research in this area is warranted.
The plot of the actual and fitted values from this regression is also shown in Figure 1.

Table 1

Regression of average change in GDP per capita on average change in EU structural funds per capita (EUSFPC)

Regressor

Coefficient

Standard error

T-ratio

[Prob]

C
EUSFPC

– 2,0780
2,8192

0,90194
0,88559

– 2,3040
3,1834

10,1340

R-squared
S.E. of regression

Mean of dependent variable

Residual sum of squares

Akaike information criterion

1,2355

0,53688

0,52963 R-bar-squared
F-stat. F (1,9)
S.D. of dependent
variable
Equation log-likelihood
Schwarz Bayesian
criterion

13,7371

– 18,8304

DW-statistic

1,8219

Figure 1

Plot of actual and fitted values

[0,047]
[0,011]

0,47737
[0,011]

1,7089

– 16,8304

– 19,2283

C 302/30

EN

Official Journal of the European Union

12.12.2006

ANNEX IV

HERMIN simulations and unemployment

1.
The HERMIN model was applied on the basis of an assumed ‘average’ hypothesis (medium-medium) concerning sup-
ply effects or elasticities with respect to human capital and infrastructure. The robustness of these results were analysed by
means of a sensitivity analysis on the basis of two further hypotheses:

zero-zero:

— minor neo-classical effects (via changes in relative prices),

— no supply effects (in terms of improvements in infrastructure and human capital) or small output and

factor productivity elasticities with respect to physical infrastructure and human capital;

high-high:

— supply effects much more pronounced (in terms of physical infrastructure and human capital)

and continue to apply following the CSF period.

2.
As regards to unemployment, the HERMIN simulations for eastern Germany, Spain, Greece, Ireland and Portugal have
led to changes in the unemployment rate which varies according to the three hypotheses. To assess better the appropriate-
ness of the results, the Court converted the falls (or increases) in the unemployment rate into net jobs created (or lost). The
results of converting falls (or increases) in unemployment into net jobs created (or lost) are given in Table 1.

Table 1

HERMIN simulations

Eastern Germany

Spain

Greece

Ireland

Portugal

Zero-zero hypothesis

Medium-medium hypothesis

High-high hypothesis

Net jobs created

822 308

1 051 135

303 034

86 679

497 445

729 923

241 789

190 825

64 621

250 502

635 755

– 806 026

65 969

36 335

– 253 299

3.

The validity of the results needs to be assessed in the light of the following observations:

(a)

In the case of the new German Länder, although the bottom-up data are still incomplete, it can be concluded that the
figures produced using HERMIN for net jobs created appear, in all the hypotheses, substantially to exceed the data pro-
duced by monitoring directly the projects themselves (about 500 000 created jobs or maintained) (1).

(b) As regards Ireland, the comparison with other data show that the net figure of 64 621 that is attributed to the CSF
appears somewhat understated compared with the other evidence. Monitoring figures show that the ‘industry’ OP led
to about 212 000 gross jobs created and 90 000 net jobs created. Furthermore, the total number of jobs, across the
whole economy, increased by 395 075 (2). The evaluators also note that it is important to look beyond the CSF in order
to take account of policy elements, such as incentives for FDI and the policy of curbing wage inflation, that accompany
the CSF. In other words, the model does not take account of these elements, which increase the impact attributed to
the CSF.

(c) The Spanish O1 CSF, which is the largest of the CSFs, by applying the average hypothesis, is far less successful than the
new Länder O1 CSF in terms of job creation and outperforms the latter in this area only if it is supposed that demand
effects predominate. However, under that hypothesis, the net figure for jobs created during 1994 to 1999 as a result of
the O1 CSF would account for around 40 % of employment growth in Spain between 1995 and 2000, which appears
rather optimistic.

(d) Applying the average hypothesis and the hypothesis of dominant demand effects to Portugal gives net job creation esti-

mates that are far higher than those produced by the national authorities (3). However, this has not been addressed.

(1) Evaluation report for Germany (O1), (pp. 42, 48, 70, 72, 85, 86).
(2) Evaluation report for Ireland (pp. 61, 92).
(3) Department for Forecasts and Planning (DPP).

12.12.2006

EN

Official Journal of the European Union

C 302/31

(e) The same is true for Greece, where the national authority’s net job creation estimates substantially exceed the estimates
obtained from the HERMIN model (under all hypotheses) concerning falls in unemployment. According to the national
report, the CSF has contributed to an annual net increase in employment of 1,7 % of the labour force (1) which trans-
lates as a net figure of more than 400 000 jobs created during 1994 to 1999.

(f)

The figures for jobs lost under the high-high hypothesis for the Spanish and Portuguese CSFs do not appear credible.

(1) National report, p. 210.

C 302/32

EN

Official Journal of the European Union

12.12.2006

THE COMMISSION’S REPLIES

SUMMARY

VI.
The Commission is fully aware of the limitations of the
study, due to a large extent to data constraints, and will take steps
to improve the feasibility of future ex post evaluations.

VII.
Although some difficulties with data were encountered,
the evaluation used a number of types of information to reach its
conclusions (member state mid-term and final evaluation reports,
interviews, national workshops, and case studies).

VIII.
The Commission considers that the terms of reference
were adequate for the purpose, but the evaluation faced major
challenges, especially of data availability. The Commission took
into account both positive and negative aspects of the work car-
ried out.

Nevertheless, the Commission accepts that a preparatory analysis
would have helped to identify some of the information difficul-
ties that the Court has described.

X.
Although difficulties were encountered, the results of the
evaluations have been considered during the mid-term review
process as foreseen.

XI.
The Commission considers that the future ex post evalu-
ations should be more focused and take full account of data and
resource constraints.

The Commission considers that macroeconomic models are a rel-
evant tool for measuring the economic impact of large scale inter-
ventions. The HERMIN models developed for the various coun-
tries concerned took account as far as possible of the national
specificities of the recipient economies. However, as for any
evaluation tool, further improvements can be made to address
limitations, in particular on micro data.

The Commission agrees that ex post assessments should take full
account of existing thematic studies.

Future evaluations should cover fewer evaluation questions. This
will make it possible to deepen the analysis in certain fields and
will facilitate the oversight carried out by the Commission.

INTRODUCTION

1.
The Court has made observations on the evaluation in
general and ex post evaluation in particular on several previous
occasions, notably in Special Report No 15/1998 on the assess-
ment of Structural Fund intervention for the 1989 to 1993
and 1994 to 1999 periods and Special Report No 7/2003 on the
implementation of assistance programming for the period 2000
to 2006 within the framework of the Structural Funds. In this ear-
lier work the Court acknowledged the specific constraints of such

evaluations and came to an overall satisfactory assessment, while
pointing out aspects needing improvement. The Commission
welcomes the Court’s constructive criticism on this topic.

4.
For the future ex post evaluation the Commission will
review the organisation of the exercise, with particular reference
to data collection and the preparation of the synthesis report.

The ESF ex post evaluation used the evaluation reports carried out
by Member States at both the mid term and ‘final’ stages of the
programming period 1994 to 1999.

EX POST ASSESSMENT OF OBJECTIVE 1 BY DG REGIO

15. Text box 1.
The ex post evaluation is not an official report
of the Commission. Nevertheless, the Commission supports the
overall conclusion of the report, which is borne out by evidence
from the Commission’s Third Report on Economic and Social
Cohesion.

16.
The Court acknowledges the particular challenges facing
ex post evaluation, and in particular the lack of quantified indica-
tor data. The Commission agrees that there is room for improve-
ment in future ex post evaluations. To this end, the Commission
intends to initiate a broader methodological debate with Member
states and the academic world before the next round of ex post
evaluations. A major issue that needs to be explored is whether
to focus on a small number of relevant evaluation questions. The
exercise will be less subject to data constraints.

17.
Although the convergence rate differs between regions,
growth in Objective 1 regions is driven not only by employment
increases but also significantly by increases in productivity. Other
empirical studies suggest that productivity is the key factor deter-
mining the growth rate that is sustainable in the long run (unless
there is a permanent inflow of migrants).

18.
The Commission welcomes the regression analysis
undertaken by the Court. However, it is based on only one of the
explanatory variables. To avoid biased results, a control analysis
for the other explanatory variables would be necessary.

Other studies undertaken by the Commission,

for
19.
example for the Third Cohesion Report, have also led to different
conclusions from the Synthesis Report. The picture is in fact
mixed, with increases in employment appearing to be the main
factor in growth in some Member States, and productivity growth
in other Member States (see Third Cohesion Report, p. 3).

12.12.2006

EN

Official Journal of the European Union

C 302/33

The Commission agrees that the data available on par-
20.
ticipation rates, gross domestic product (GDP) and productivity
could have been better exploited by the evaluator. However, other
studies have also explored these relationships.

The two evaluations have a different scope and coverage.
32.
For example, the thematic evaluation of transport is also based on
the Cohesion Fund and covers all objectives.

21.
Planned and actual expenditure data cannot be broken
down by region, as a significant share of the assistance is deliv-
ered through national programmes such as transport, research,
national aid schemes or human resource programmes, and Mem-
ber States do not keep statistics on how it is allocated by region.
Therefore, the regression could necessarily only cover part of EU
expenditure. This was an inherent constraint on this part of the
evaluation.

23.
In its Third Cohesion Report (p. 147) the Commission
showed that there was some relationship between the amount of
structural aid provided and real growth of GDP in Objective 1
regions.

33.
Estimating the impact of Structural Fund interventions
on employment is notoriously difficult and depending on the
methodology can produce different results. These should there-
fore be interpreted with caution. The Commission is doing fur-
ther methodological work on the estimation of employment
effects to increase the consistency of results.

The methods used for estimating the employment effects
34.
in the Synthesis Report and the thematic evaluation report are not
comparable. The HERMIN model
is based on more robust
assumptions and also captures supply-side effects associated with
the investment in physical and human capital, while the study on
transport estimates the direct and indirect short term effects.

24.
As already pointed out, the evaluation was faced with
severe data constraints, particularly regarding national data on
both public and private co-financed expenditure. The data pre-
sented were mostly based on estimation procedures, which gave
rise in some cases to errors or discrepancies.

35.-36.
As indicated above, estimating the impact of Struc-
tural Fund interventions on employment is notoriously difficult
and depending on the methodology can come to different results.
The thematic study on SMEs carried out a more in-depth analysis
on SME interventions and also covered other interventions out-
side Objective 1 areas.

26.
The participation of the private sector depends on the
general economic conditions of the country as a whole and the
region concerned. The leverage effect of EU Structural Funds var-
ies greatly from one country to another.

37.
Monitoring data of the Member States was a necessary
input for the study. Given the time and budget constraints, it
would have been impossible to make up for the gaps and defi-
ciencies in the data.

Examination of this issue was hampered by the uncer-

27.
tainty surrounding figures for the private sector.

28.
Market failure is a crucial issue in the context of ex ante
evaluation. Although it is mentioned in the methodology guide
drawn up by the evaluator, a formal and detailed examination of
appropriateness was not required by the terms of reference.

30.
The evaluators had to base their judgements largely on
information and data available in the Member States. If data, for
example on deadweight, was not available, the terms of reference
did not require the evaluators to collect such data themselves,
given the limited budget available for the exercise.

Concerning the concept of Community value added, the Com-
mission considers that the content of the concept depends on the
context in the different Member States. For example, the pro-
gramming approach can be new in one Member State but not in
another. A qualitative assessment seems the most appropriate in
these circumstances.

Given the broad scope of the evaluation, it was impos-
38.
sible to address all issues in depth by undertaking ad hoc surveys
or other field work.

39.
The Commission agrees that evaluations should use exist-
ing studies and literature as a starting point for the analysis. How-
ever, the thematic studies had been undertaken in another con-
text, namely as an input for preparing 2000 to 2006 programmes,
whereas the ex post evaluation came later.

40.
The use of formalised quantitative methods requires the
existence of robust and sufficient data from the Member States.
As the Court points out, such data was often missing for the 1994
to 1999 programming period.

The objective set by the terms of reference (ToR) for this

41.
area was fairly limited in scope.

31.
The limitations are largely due to the data and budget
constraints. While a consideration of the results of other thematic
studies may well have given added weight to the conclusions, the
ex post evaluation was necessarily broader in scope.

42.
Despite the difficulties encountered in obtaining data due
to the fact that different types of building work were intercon-
nected, the evaluators attempted to establish comparable unit
costs for some fields such as road construction, environmental
projects and jobs created in industrial projects.

C 302/34

EN

Official Journal of the European Union

12.12.2006

The Commission agrees that a more detailed analysis is
43.
desirable. The subsequent ex post evaluation on the Cohesion
Fund delivered further insights on this subject. In 1998 the Com-
mission prepared a guide entitled ‘Understanding and monitor-
ing the cost-determining factors of infrastructure projects — A
user’s guide’.

44.
The wording of the ToR made it clear that these tasks
were especially difficult and could only yield partial conclusions.
An assessment of public-private partnerships and of tendering
procedures was not required by the ToR. The issue of public-
private partnerships has been addressed in the ex post evaluation
of the Cohesion Fund.

46.
The evaluator excluded private cofinancing in the mac-
roeconomic modelling, as the model sought to quantify the over-
all
impact of EU and public cofinanced expenditure on the
economy including private investment. Private sector investment
is regarded as a result and not as a separate policy input. The
Commisssion refers to points 24 to 27 of its replies. The Com-
mission shares the concerns of the Court on the difficulty of
specifying externality parameters. The choice made was based on
international studies from which a range of elasticity values could
be derived. At the time the study was conducted, there was no
real alternative for conducting the impact analysis.

48.
Some of the Court’s observations in points 48 and 49
were made in its Special Report No 7/2003 on the implementa-
tion of assistance programming for the period 2000 to 2006
within the framework of the Structural Funds. The Commission
refers to the replies given in that report.

The alternative suggested by the Court was not feasible. The
requirements for collecting micro-data were not standardised to
the extent that would have been necessary for this type of analy-
sis. The Commission agrees that future macroeconomic evalua-
tions should rely more on microeconomic data analysis.

49.

considers different scenarios based on high, medium and low
values. This sensitivity test results in a range of impact values.

All macroeconomic models face data problems. The model-
ling exercise relied on the best data available. The Commis-
sion agrees that additional work is needed to further improve
the elasticities data used by the model.

(f)

In the HERMIN model, the counterfactual scenario is defined
as the situation with no CSF in order to be able to estimate
the deviation relative to the baseline. However, other addi-
tionality scenarios can be envisaged, e.g. only structural
and cohesion funds (without national co-financing).

50.
In the Commission’s opinion, despite possible shortcom-
ings, HERMIN-type models are a valuable instrument for under-
standing and measuring the effects of cohesion policy. The use of
the model by a growing number of Member States and the
absence of convincing alternative approaches confirm this view.
The HERMIN model can be improved and the Commission is
working on its development for the programming period 2007
to 2013.

51.
The Commission considers that it is difficult to establish
any linear correspondence between a fall or increase in unem-
ployment and the number of jobs created or lost. Labour market
mechanisms are more complex and should be examined carefully
before drawing firm conclusions.

52.
Again, the evaluation was dependent on the quality of
data and setting of objectives by the Member States. The absence
of such data made the evaluation in many cases difficult
or impossible.

(a) These were the assumptions of the HERMIN model at the
time of the evaluation. However, further development of the
model should pay greater attention to the importance of the
service sector in Objective 1 economies.

Partially conflicting or differing recommendations in dif-
56.
ferent evaluations with different questions and methods are a nor-
mal and productive phenomenon in economic and sociological
research.

(b) The issue of structural instability of the model will be exam-

ined carefully in future macroeconomic work.

(c) The Commission acknowledges that the modelling exercise
would have been more accurate if data sets on actual expen-
diture had been available. The commitment data used were
the only data available across all regions at the time of the
evaluation.

(e) The elasticity values associated with physical and human
capital investment which are used in the HERMIN model are
based on a broad range of academic studies. The model

57.
The recommendation in the ex post evaluation to pay
attention to an appropriate, workable size of monitoring commit-
tees is not inconsistent with the recommendation to ensure inclu-
siveness in their composition made in the thematic study.

The ex post evaluation was based here only on partial
59.
studies. The Commission agrees that the findings concerning the
management systems were interesting enough to deserve more
attention. A specific evaluation of this issue was undertaken by
the Commission services in 2003 (The efficiency of the implementa-
tion methods for Structural Funds, December 2003).

12.12.2006

EN

Official Journal of the European Union

C 302/35

Like the other thematic evaluation studies, that of the
60.
partnership principle was designed to inform the programming of
the 2000 to 2006 period. The programming had been completed
by the time of the ex post evaluation. Therefore, greater discus-
sion of this subject in the ex post evaluation, while it may have
been desirable, was not essential.

The tasks and budget of the evaluation were necessarily

61.
limited.

The issues raised by the Court were the subject of the study
referred to in the reply to point 59. Project selection procedures
are also one of the aspects of management and control systems
that are checked in audits, as they are an important condition of
sound financial management.

The Commission tends to the view that future ex post evaluations
should cover less questions.

62.
The Commission agrees that Member States’ manage-
ment and control systems for the Structural Funds during the
1994 to 1999 period contained weaknesses. However, they
improved towards the end of the period with the introduction of
the control regulation 2064/97, and the improvement has con-
tinued in the 2000 to 2006 period.

Member States and Commission agree that the monitor-
63.
ing of Structural Funds programmes was a weak point in the
1994 to 1999 programming period. In the 2000 to 2006 period,
substantial efforts were undertaken to address this issue. For the
first time systematic evaluations (including ex ante evaluations and
mid-term evaluations) were carried out for all Structural Funds
programmes. This led to a substantial improvement of the indi-
cator and monitoring system for the Structural Funds.

64.
The financial data produced by the systems were broadly
reliable. This was not, however, the case for monitoring data,
although the availability of such data improved in the later stages
of the programme period.

65.
‘Internal evaluations’ were not legally required by the
regulations but were performed in response to specific needs felt
by the Member States. This was a positive by-product of the ex
post evaluation, which is now being widely imitated.

For the programming period 2000 to 2006 the regula-
66.
tion established for the first time detailed rules and requirements.
Based on that, the Commission elaborated detailed working
papers on the elements of monitoring and evaluation. The ex ante
evaluations, carried out sytematically for all programmes and
objectives, were a major opportunity for the Member States
to invest in the establishment of indicators and quantified objec-
tives. Most Member states introduced IT solutions to support
regular and systematic reporting based on quantified information.
The mid-term evaluation was an opportunity to revise and
to improve the indicators and their use.

67.

(a)

(b)

(c)

(d)

The Terms of Reference stated that ‘data available in the
Member States may not be complete, particularly if the
Member State does not plan to undertake ex post
evaluations…’.

The extent of further data collection was constrained by
the time and resources available.

There are methodological difficulties in assessing net
effects through microeconomic methods. However,
further work should be undertaken in this area.

The Commission agrees that this is an important issue
in assessing effectiveness. However,
it would have
required specific further studies, which are expensive.
The Commission, however, encourages Member States
to undertake such work.

(e)

The potential
and impacts are analysed in the ex ante evaluations.

links between strategy, effectiveness

(f) and (g) In the Commission’s view, it is better in terms of refer-
ence to set only the evaluation questions, leaving a
methodological proposal to the bidders.

68.
A great deal of research on convergence has been done.
The results of recent such work are summarised in the Commis-
sion’s Cohesion Reports.

This could have been another interesting approach, but
69.
was not necessary. In the Commission’s view, future evaluation
must set clear priorities, limiting the scope of questions.

The Commission considers that the ToRs and the meth-

70.
odological guide were adequate for the purpose.

72.
Learning from the ex post evaluation, the Commision cur-
rently uses expert advice in methodologically demanding evalua-
tions at an earlier stage.

The ex post evaluation 1994 to 1999 faced specific chal-

73.
lenges, especially of data availability.

75.
The Commission applied in this evaluation the usual
management techniques such as a formalised interim reporting
system, oversight by a steering committee, regular working meet-
ings and the involvement of experts. A report of acceptable qual-
ity was the outcome of this process.

C 302/36

EN

Official Journal of the European Union

12.12.2006

The complexity of the ex post evaluation was significant. The
Commission will use the experience gained to address this issue
better in future.

The ex post evaluation did not rely on single sources of informa-
tion in order to reach conclusions and make recommendations.

The Commission agrees with the view of the Court that future
evaluations should cover fewer evaluation questions. This will
make it possible to deepen the analysis in certain fields and will
facilitate the oversight carried out by the Commission.

76.
The evaluation, both in the Executive Summary
and Chapter 9 Recommendations, goes far beyond management
remmendations. The recommendations are aimed at Structural
Fund interventions in all Objective 1 regions and are necessarily
of a general nature. The role of research and development is
stressed, as are the need to pay more attention to differences
within large Objective 1 regions, the need to coordinate better
Structural Funds interventions across funds in rural areas, etc.

77.
The Commission agrees with the evaluator that Monitor-
ing Committees should devote more attention to strategic issues.
This must not detract from the other tasks Committees have to
perform under the regulations.

80.
The recommendation specifically referred to by the Court
has been taken up by the Commission for the 2007 to 2013 pro-
gramming period.

EX POST ASSESSMENT OF ESF OPERATIONS (UNDER
OBJECTIVES 1 AND 3) BY DG EMPL

81.
A coordination meeting between DG REGIO, DG EMPL
and the evaluators for the ESF ex post evaluation took place in
October 2002. At this meeting it was agreed that the ESF ex post
evaluators would have access to material gathered by the ERDF
ex post evaluators. This later took place. The two managers of the
ex post evaluations were members of each other’s steering groups
and received documentation circulated to other steering group
members. The ESF evaluators used the ERDF evaluation material
available.

Text box 2.
the ESF ex post evaluators.

A variety of sources of information was used by

The synthesis report alone contains a great deal of infor-
82.
mation concerning the use of ESF resources (nature, quantity
and impact) across different target groups and by types of mea-
sure across the Member States (see in particular Chapter 2 of the
Synthesis Report).

84.
The number of beneficiaries estimated in the table men-
tioned is based on the information provided by the Member
States’ monitoring systems. The difficulties encountered in the
accuracy of some data did not have a significant influence on the
useful observations issued.

85.
The Commission recognises that there were problems in
estimating the number of ESF beneficiaries. Nevertheless, it con-
siders that the table contains useful data for policy makers.

While the ex post evaluator has already, quite correctly, cited the
two sources of information which have been used to develop the
table, a fuller explanation of how the figures were derived would
nonetheless have been useful.

The comments in the text refer to more detailed infor-
86.
mation than that presented in the accompanying table. The table
presents averages while the text gives a breakdown by
subcategory.

The Commission holds the view that the fact that a wide range of
values is not provided does not detract from the value of the
analysis.

87.
The aim of Chapter 3 is to provide contextual informa-
tion for ESF actions and it is in fact Chapter 2 that provides the
bulk of numerical information concerning the ESF.

Text box 3.
This limited number of textual mismatches does
not invalidate the accuracy and usefulness of the data in the table.

The table clearly shows overall support for the hypoth-

88.
eses identified.

Examination of Table 3 shows that:

(a)

the highest number of cases which contradict any hypoth-
esis is one out of 30;

(b) at least two thirds of the scores are in the categories ‘strong
supporting evidence’ or ‘some supporting evidence’ (20/30,
21/30, 22/30 and 26/30).

It is also a legitimate aspect of evaluation work to iden-

89.
tify deficiencies in data.

The terms of reference clearly state that the evaluation should be
primarily desk-based using secondary data (data and information
available in the mid term and final evaluations and in the man-
aging authorities’ reports and monitoring systems) and that the
evaluators should not engage in extensive primary information
gathering. Significant additional field work and survey samples
would have required extra resources and would have engendered
significant additional costs especially given the need to collect
information concerning the entire programming period. The
forthcoming ESF ex post evaluation for 2000 to 2006 will be pre-
ceded by a preparatory analysis.

The beneficiary totals only play a role in relation to (i) account-
ability and (ii) the limited inpout/output relationships analysis.

The synthesis report contains several references to the

90.
significance of ‘combined measures’ in terms of effectiveness.

12.12.2006

EN

Official Journal of the European Union

C 302/37

Firstly, the evaluators choose their words with care and
92.
emphasise that the conclusions they draw are tentative ‘The com-
bined allocation of national expenditure and ESF resources does
not appear to have led to a convergence …’. Secondly, the figures
(tables in Chapter 3 and Annex 2) do show that there is not a con-
vergence of expenditure patterns between countries in combined
allocation of national expenditures. Thirdly, as mentioned previ-
ously in reply to point 87 (including Text box 3) the data pre-
sented in the Chapter 3 tables mentioned by the Court are accu-
rate and support the qualified observation provided by the
evaluators. This observation is also based on the analysis of labour
market trends presented in Annex 2 of the Synthesis Report (not
only Chapter 3). The Commission has already indicated that the
examples given concerned a limited number of textual mis-
matches which do not undermine the figures.

Information on performance is also available from the case
studies.

(d) Sustainable development was not yet a political priority for
the Structural Funds at the time when the programmes for
the 1994 to 1999 period were being established and there-
fore it was not identified as a theme in the terms of reference.

Socio-economic cohesion does not appear in the terms of
reference: it was dealt with in the Cohesion Report.

93.
As in the case of Table 3 on individuals mentioned in
point 88, the data clearly shows overall support for the hypoth-
eses identified.

(e) Each of the national reports examined by the Court contains
substantial material on the long-term unemployed and young
people. The synthesis report also contains information on
the above two ‘target groups’.

The Commission considers that its quality assessment
95.
was accurate and balanced, taking into account both positive and
negative aspects of the work carried out (rather than only con-
centrating on a limited number of negative aspects).

Taking into account the above, and since the results and conclu-
sions were based on a variety of sources of information the Com-
mission stands by its quality assessment.

97.

(a)

It is not surprising that some evaluation results are of a quali-
tative nature as the evaluation makes use of secondary
sources which have been summarised and primary sources
which are of a qualitative nature (interviews with stakehold-
ers and national workshops on the evaluation findings In
some cases the description of results can only be of a quali-
tative nature. Estimation of deadweight was not a compul-
sory subject of analysis for Member States during the pro-
gramming period 1994 to 1999, nor was it part of the
evaluation terms of reference. The logical consequence of
this is that the ex post evaluation itself could only report on
deadweight estimates that were available to it.

(f) As has already been stated, the ex post evaluators were in
some respects dependent on information provided by Mem-
ber States. The fact that some programmes lacked ‘specific
initial objectives’ was a further constraining factor for the
evaluators.

(g) The ESF and the ERDF ex post evaluators collaborated and the
ESF ex post evaluators used reports provided to them by the
Objective 1 evaluators. Productivity increases by ESF benefi-
ciaries in terms of the benefits to the firms employing them
were not a required subject for Member States in their evalu-
ation work during the programming period 1994 to 1999,
nor was it part of the terms of reference. As such it is not sur-
prising that this subject occurs only occasionally.

98.
The synthesis report identified numerous factors impact-
ing on the development of Active Labour Market Policies
(ALMPs), such as the level of ESF resources, the Structural Funds
planning processes and organisational arrangements for the ESF.
It also described rather concretely the influence of the ESF in the
context of the European Employment Strategy (EES).

(b) The UK report provides a substantial amount of evidence on

soft outcomes.

Other national reports also make references to the concept.

As already pointed out, net impact analysis was not a compul-
sory subject for Member States during 1994 to 1999. Therefore,
it is normal that the subject does not appear in all Member States’
evaluation reports. Since the evaluators were requested not
to carry out large scale primary data gathering, the ex post evalu-
ators cannot be criticised for not having provided this material.

(c) The synthesis report, and the national reports on which it is
based contain numerous observations about the content of
ESF action (financial, by target group and by types of mea-
sure) and performance, comments derived from national
evaluation reports and comments derived from interviews
with stakeholders).

The EES stronger strategic framework provides targets
100.
against which progress can be judged. Improved institutional
structures have been created to measure progress. Moreover, the
EES promotes stronger links between EU programme funding
mechanisms and other policy instruments, regulations and
national programmes.

C 302/38

EN

Official Journal of the European Union

12.12.2006

The Commission believes that each national report has
101.
dealt with the aspects of Community added value identified in the
terms of reference (financial and leverage effects, policy and insti-
tutional effects, socio cultural effects, and variation across objec-
tives). It recognises, however, that these aspects were sometimes
dealt with in other sections of the national reports.

In addition, the national ex post evaluation reports (and the syn-
thesis ex post evaluation report) integrate these various aspects
into an overall analysis of Community added value.

102.
The national and synthesis reports contain a great deal
of information concerning the financial resources that have been
devoted to ESF action.

In the case of both Spain and Italy, the evaluators arrived at posi-
tive conclusions overall.

The ex post evaluator for Spain reports that: ‘The general picture
that emerges is that the ESF funds have been adequately and effi-
ciently used by Spanish authorities in the 1994 to 1999 period.
Most of the programmes have been fully executed in the way that
had been planned’ (p. 74).

In the case of Italy, the evaluator concludes that: ‘Some of these
costs are necessary to improve performance and transparency in
the implementation process; others could probably be reduced by
simplification of procedure’. (The issue of simplification of pro-
cedures was taken up in the next programming period 2000
to 2006).

104.
The Commission believes that the report does indeed
provide useful material for conducting future ex post evaluations.

The report contains rather comprehensive reviews of approaches
used to evaluate ESF type actions, both in relation to actions tar-
geted at individuals and actions targeted at systems.

With regard to approaches applied in assessing net impact on
individuals, the report contains numerous references to tech-
niques used to estimate net impact on individuals (see especially
p. 14, although additional references can be found on p. 10, 12,
21, etc.).

As regards the recommendations on policy analysis, the
105.
methods report clearly shows that the evaluators have identified
a number of concrete recommendations linked to the area of
policy analysis, for instance close monitoring of the relationships
between ESF interventions and the evolving national and other
regional level interventions.

Furthermore, the methods report discusses the ‘logic models’ for
policy analysis at some length and develops evaluation questions
relating to the justification for ESF action.

programming period in guidance documents issued by the Com-
mission in 2000.

The Commission considers that the evaluation tasks are
106.
adequately specified in the TOR. Conclusions in national ex post
evaluation reports are based on diverse sources (interviews, case
studies, etc.). The synthesis report provides an overall analysis of
the results contained in the national reports. The contractor has
provided an explanation of how these sources of information
were used to generate conclusions in the proposal submitted to
the Commission.

Describing all aspects of ESF action and fixing all aspects of the
evaluation a priori would lead to unmanageable and inflexible
terms of reference.

With regard to ‘concrete action’ to evaluate the impact on indi-
viduals, when the indicators are not concretely established or
when specific statistical evaluations have not been carried out
recently, as already stated, the entire evaluation approach fol-
lowed by the ex post evaluator was designed to draw conclusions
from a variety of sources of information (Member State evalua-
tion reports, national workshops, face-to-face interviews with
programme stakeholders, case studies, etc.).

The extent of the mismatch between quantitative objectives and
available data was not known at the time of the drafting of the
terms of reference, therefore it seems incoherent to expect the
terms of reference to have been drafted differently.

The main contractor developed, in cooperation with national
contractors, operational guidelines for the work to be conducted
at national level.

The terms of reference did not require the evaluators to
107.
assess the ‘appropriateness of the reprogramming choices’, but
only to describe them. In each of the cases cited the ex post evalu-
ator has described reprogramming changes based on the available
information.

108.
sidered for the next exercise.

The possibility of a preparatory analysis is being con-

109.
The Commission considers that the data difficulties
faced by national evaluators did not detract from the overall qual-
ity of the evaluation. The evaluation design foresaw the use of a
variety of sources of information. In addition to reviewing the
content of Member State evaluation reports, the national ex post
evaluation reports include material from closure reports, case
studies, national workshops and face-to-face interviews.

The ex post evaluators were not expected to propose an approach
for the selection of appropriate indicators at different strategic
levels because such indicators had already been set for the entire

In terms of delays, the steering group was not in a position to
speed up the closure of mainstream ESF programmes, which are
dependent on implementation by the managing authorities.

12.12.2006

EN

Official Journal of the European Union

C 302/39

At the start of the evaluation, the ex post evaluator was provided
with all evaluation reports produced by the Member States dur-
ing 1994 to 1999. Furthermore, the ex post evaluator was pro-
vided with letters of introduction in order to facilitate contacts
with national ESF administrations.

111.
In the future the Commission will take all necessary
steps for the successful preparation and close monitoring and
supervision of the exercise.

CONCLUSIONS

115.

(c) Some attempt was made to assess efficiency on the basis of a

limited number of projects.

(d) The HERMIN models did take account of the national speci-
ficities of the economies, however further work will be
undertaken to address the limitations.

The Commission considers that the limited number of drafting
mismatches in Chapter 3 of the report do not call into question
the accuracy, validity and usefulness of the data to support the
conclusions drawn. The data contained in the tables commented
upon by the Court in points 88 and 92 show that there is signifi-
cant overall supporting evidence for the hypotheses identified.

(b) The extent of further data collection was constrained by the

time and resources available.

The difficulties encountered regarding the exhaustiveness of some
data did not have a significant influence on the useful observa-
tions issued.

Nevertheless, a substantial amount of additional qualitative
data was collected, e.g. from interviews and workshops.
National closure reports and the evaluation reports con-
ducted by Member States during the 1994 to 1999 program-
ming period were also reviewed.

(c) A variety of sources of information was used in order to

arrive at balanced, well considered conclusions.

117.
The Commission considers that the evaluation relied on
a robust design using a variety of information sources and that its
conclusions are sound.

In the future, the Commission will take all necessary steps for the
successful preparation and close monitoring and supervision of
the exercise.

In some cases the description of results can only be of a
qualitative nature. But this was not solely the case: the evalu-
ation
quantitative
information.

significant

contain

results

also

(d) The conclusions drawn were based on the evaluators’ use of
secondary sources (which they summarised) and primary
sources of a qualitative nature (interviews with stakeholders
and national workshops on the evaluation findings).

See also reply to points 88, 92 and 96.

(e) An evaluation of deadweight was not foreseen in the ToR.

As a general remark, an evaluation exercise cannot cover all
the topics and has to focus on priority objectives and areas
taking into account regulatory obligations, time and resource
constraints and the intended use of the results. The experi-
ence of this exercise and lessons learned will be used for the
2000 to 2006 ex post evaluation.

116.

(a) Examination of this issue was hampered by the uncertainty

surrounding figures for the private sector.

(b) The analysis was constrained by the availability of data and
lack of comparability resulting from differences in methods
and scope.

See reply to point 111 above.

118.
In the future the Commission will take all necessary
steps for the successful preparation and close monitoring and
supervision of the exercise.

RECOMMENDATIONS

119.
For future ex post evaluations preparatory analyses or
feasibility studies will be carried out to assess the availability of
data and what outputs can reasonably be delivered within the
budget and timeframe.

120.
The Commission agrees that the quality control proce-
dures applied in the 1994 to 1999 period ex post evaluations
should be further improved.

The Commission recognises that for the forthcoming Structural
Funds ex post evaluation exercises a preparatory analysis could
play a useful role in facilitating the work of contractors during the
evaluations proper.

(a) The basic data used in the ex post evaluation are collected and
provided by the Member States through the monitoring sys-
tem. The availability of reliable data may be a constraint on
the evaluation, but in the 2000 to 2006 period there has
been progress in this area.

C 302/40

EN

Official Journal of the European Union

12.12.2006

(b) Terms of reference must enable the Commission to assess the
ability of the candidate to conduct the evaluation and
whether the particular combination of methods and tech-
niques proposed can deliver the required outputs.

The Commission shares the Court’s opinion that terms of ref-
erence play an important role in achieving good quality
evaluations results.

(c) Given budget and time constraints, future ex post evaluations
will be more focused on key evaluation questions and will be
prepared and launched earlier than the 1994 to 1999
exercise.

Based on the experience of 1994 to 1999, monitoring and super-
vision arrangements will be strengthened and the process
launched earlier.

121.
The Commission considers that macroeocnomic mod-
els are relevant for estimating the economic impact of large-scale
interventions. The HERMIN model was designed specifically for
such purpose in the early 1990s. However, as for any economic
model, further improvements will need to be made, in particular
regarding the micro-data for the estimation of the economic
returns of physical and human capital investments. The Commis-
sion considers that the future ex post evaluation should be more
focused.

122.
The Commission agrees that ex post assessments should
take full account of existing thematic studies. When using them,
that ex post assessments and thematic studies serve different pur-
poses and therefore may lead to different estimates.

123.

(a) The Commission agrees that private investment is a key
driver for economic development. However, experience
shows that private sector leverage differs across countries
depending on specific economic conditions.

(b) This approach cannot be universally applied since economic
conditions in Member States and regions differ greatly.

(c) The Commission is aware that further systematic work is

needed in this extremely complex area.

(d) The Commission is aware that further work is also needed in

this area.

(e) Ex post evaluation reports commissioned by Member States
were one of the primary sources of information in the 1994
to 1999 evaluations. Although the Commission is primarily
responsible for ex post evaluation, cooperation with Member
States should be strengthened in the future and internal
evaluations could play a greater role.

124.
sight of the process.

The Commission is taking steps to improve its over-

125.
The Commission agrees that the SF ex post evaluations
should make full use of knowledge gained during an entire pro-
gramming period and for this reason accepts the idea that a pre-
paratory analysis could contribute to facilitating the task of con-
tractors and improving the quality of forthcoming SF ex post
evaluations.

The Commission already collaborates extensively with research
institutes and universities and is willing to envisage further assign-
ments for the purposes referred to by the Court.