Special Report No 15/98 on the assessment of Structural Fund intervention for the 1989-1993 and 1994-1999 periods together with the Commission's replies (submitted pursuant to Article 188c(4)(2) of the EC Treaty)
Date: 1998-09-23
Institutions: European Court of Auditors
Subjects: Community support framework, Structural Funds, impact study, management, operational programme, project evaluation

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/1

I

(Information)

COURT  OF  AUDITORS

SPECIAL  REPORT  No  15/98

on  the  assessment  of  Structural  Fund  intervention  for  the  1989-1993  and  1994-1999  periods
together  with  the  Commission’s  replies

(submitted  pursuant  to  Article  188c(4)(2)  of  the  EC  Treaty)

(98/C  347/01)

TABLE  OF  CONTENTS

Paragraph
number

Page

1.  INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1.1 — 1.4

2.  THE  SCOPE  OF  THE  AUDIT . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

2.1 — 2.3

3.  SUMMARY  OF  THE  MAIN  OBSERVATIONS . . . . . . . . . . . . . . . . . . . . . . .

3.1 — 3.12

4.  GENERAL  ISSUES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.1 — 4.36

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.1 — 4.11

Reference  framework . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.12 — 4.13

Statistical  bases  and  information  concerning  objectives . . . . . . . . . . . . . . . .

4.14 — 4.15

Indicators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.16 — 4.21

Taking  evaluation  into  account . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.22 — 4.27

The  objectivity  of  evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

4.28 — 4.30

The  evaluation  contracts  awarded  by  the  Commission . . . . . . . . . . . . . . . .

4.31 — 4.36

5.  THE  EX  POST  EVALUATION  OF  THE  OBJECTIVE  Nos  1  AND  2  CSFs
. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

(1989-1993)

5.1 — 5.17

Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

5.1

The  ex  post  evaluation  of  the  Objective  No  1  CSFs  (1989-1993) . . . . . . .

5.2 — 5.8

Ex  post  evaluations  of  Objective  No  2  (1989-1993)

. . . . . . . . . . . . . . . . .

5.9 — 5.17

3

3

4

5

5

6

7

7

8

8

9

9

9

10

11

C  347/2

EN

Official  Journal  of  the  European  Communities

16.11.98

Paragraph
number

Page

6.  ONGOING  ASSESSMENTS  (1994-1999)

. . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.1 — 6.27

The  development  and  integration  of  the  principles  of  ongoing  assessment

6.1 — 6.7

Implementation  procedures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.8 — 6.13

Priority  given  to  evaluation  objectives . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.14 — 6.20

Approaches,  methods  and  techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

6.21 — 6.26

Implementation  of  Objective  No  2  programmes  (1994-1996)

. . . . . . . . . .

6.27

7.  IMPACT  OF  THE  STRUCTURAL  FUNDS  AND  EVALUATION  MODELS

7.1 — 7.22

Impact  of  CSFs  and  programmes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7.1 — 7.15

Results  of  the  models  as  regards  the  impact  of  the  Structural  Funds . .

7.1 — 7.13

GDP  growth . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7.2 — 7.3

Real  convergence  in  terms  of  per  capita  GDP . . . . . . . . . . . . . . . . . . . . .

7.4 — 7.8

Employment  and  unemployment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

Effectiveness  of  CSF  instruments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7.9

7.10

Additionality  and  joint-financing  requirements . . . . . . . . . . . . . . . . . . . . .

7.11 — 7.12

Durable  development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

7.13

Role  of  macroeconomic 
the  Structural  Funds’
evaluation 
decision-making  process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

in 

7.14 — 7.15

Impact  of  sub-programmes,  measures  and  projects . . . . . . . . . . . . . . . . . . .

7.16 — 7.22

8.  CONCLUSION . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

8.1 — 8.19

TABLES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

ANNEXES . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

The  Commission’s  replies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

13

13

14

15

15

16

17

17

17

17

17

18

18

18

18

19

19

20

22

31

40

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/3

1.  INTRODUCTION

1.1. Under  the  1988  reform  of  the  Structural  Funds
(SFs),  the  Commission  and  the  Member  States  were
required to evaluate European structural policies. In 1993
this requirement was strengthened by the amendments to
the  Regulations  that  were  introduced  in  the  context  of
the  revision  of  the  SF  reform  (1994-1999)  and  the
standard  clauses  of 
the  new  Community  Support
Frameworks  (CSFs)  and  Single  Programming  Documents
(SPDs).

1.4. The  1993  versions  of  the  Regulations  reiterate  the
responsibilities  of  both  the  Member  States  and  the
Commission  with  regard  to  assessment  and  stress  that  it
is  to  be  carried  out  within  the  partnership  system (3).  In
this  context,  the  Commission  has  taken  the  initiative
concerning  the  supervision  and  general  coordination  of
the implementation of evaluation and, in accordance with
Article  16  of  Regulation  (EEC)  No  2052/88,  submits
every  three  years  an  evaluation  report  on  the  progress
achieved  towards  economic  and  social  cohesion  and,  in
particular,  the  contribution  made  by  the  Funds  and
national co-financing.

1.2. Evaluation,  an  essential 
the  sound
management of the SFs, is defined in the main provisions
of the Regulations (1) as the evaluation of effectiveness on
the  following  three  levels:

tool 

for 

2.  THE  SCOPE  OF  THE  AUDIT

(a)  the  overall  impact  of  the  structural  measures  on  the
objectives  set  out  in  Article  130A  of  the  Treaty;

(b) the  overall  impact  of  the  measures  laid  down  in  the

CSFs;

(c)  the  specific  impact  of  each  operational  intervention

(programmes  and  other  forms  of  intervention).

1.3. There  are  three  main  phases  to  this  evaluation:

in 

evaluating 

the  progress  achieved 

2.1. The  general  aim  of  the  Court’s  audit (4)  was  to
check 
the
Community’s structural interventions, in terms of systems
and evaluation procedures and at a methodological level.
The  audit  looked  at  the  methods  used  to  evaluate
structural  interventions  both  by  the  Commission  and
under  the  partnership  system.  It  also  examined  the
legality  and 
the
appropriations earmarked for evaluation in the budget of
the  European  Communities  and  the  quality  of  the
financial  management  of  these  appropriations.

the  utilisation  of 

regularity  of 

(a)  prior 

enable 

appraisals (2),  which 

correct
identification  of  the  socio-economic  problems  that
need to be resolved and the objectives to be achieved,
on  the  one  hand,  and,  on  the  other  hand,  of  the
strategy  adopted  and  the  means  available  for  this
purpose;

a 

(b) ongoing  assessments,  which  are  carried  out  during
implementation of a programme and make it possible
to  check  whether  the  objectives  are  still  appropriate
and  the  desired  results  on  the  way  to  being
achieved;

(c)  ex  post  evaluation,  which  is  carried  out  after  the
programme  has  been  completed.  Its  aim  is  to  analyse
whether  the  results  and  impact  obtained  are  to  be
considered  positive,  bearing 
the
long-term  effects  of  structural  measures  can  only  be
appreciated  several  years  after  their  completion.

in  mind 

that 

(1)  In  particular,  these  are:

(a)  Article  6  of  the  framework  Regulation  (Regulation  EEC
No  2052/88  as  amended  by  Regulation  EEC  No
2081/93)  on  the  tasks  of  the  Structural  Funds  and  their
effectiveness  and  on  the  coordination  of  their  activities;

(b) Article  26  of 

the  Funds’  coordinating  Regulation
(Regulation  EEC  No  4253/88  as  amended  by  Regulation
EEC  No  2082/93).

(2)  ‘Prior  appraisal’  is  also  sometimes  called  ‘prior  assessment’.
The Court prefers to adopt the expression ‘prior appraisal’ as
it  appears  in  the  Regulations.

2.2. The  audit  also 
involved  an  appraisal  of  the
techniques used for evaluating the socio-economic impact
of  structural  measures.  Although  it  did  not  aim  to
evaluate  the  programmes  directly  with  the  intention  of
producing  new  scientific  results,  it  nevertheless  made  use
of the results of work by experts so as to obtain a better
understanding  of  the  relationship  between  the  economic
policy  instruments  used  and  the  results  obtained.

2.3. Specific  audits  were  carried  out  on  the  ex  post
evaluations  of  the  CSFs  and  Operational  Programmes
(OPs)  for  Objectives  Nos  1  and  2  for  the  1989-1993
programming  period  and  on  the  ongoing  assessments  of
the CSFs, SPDs and OPs for the 1994-1999 programming
period.  All  the  ex  post  evaluations  for  the  1989-1993
Objective  No  1  CSF  were  taken  into  account,  as  well  as

(3)  Community  operations  are  to  complement  corresponding
national  operations  or  contribute  to  them.  They  are  set  up
via  close  consultation,  known  as  ‘partnership’,  between  the
Commission, the Member State concerned and the competent
local  bodies,  social  partners  and  other
regional  and 
full
is  conducted 
competent  bodies. 
‘Partnership’ 
compliance  with  the  respective 
legal  and
institutional, 
financial  powers  of  each  of  the  parties  concerned.

in 

(4)  In its Annual Reports concerning, in particular, the financial
years  1990  (chapter  7),  1993  (chapters  6,  7,  8  and  9)  and
1994  (chapter  4),  the  Court  had  already  reported  on  the
numerous  shortcomings  in  the  field  of  evaluation  for  the
1989  programming  period.

C  347/4

EN

Official  Journal  of  the  European  Communities

16.11.98

approximately 20 % of the ex post evaluation reports for
the  1989-1993  Objective  No  2  CSF  and  15 %  of  the
ongoing  assessment  reports  for  the  1994-1999  CSFs,
selected  on  the  basis  of  their  geographical  distribution
and the nature of the interventions (CSFs, SPDs or OPs).
In  addition,  the  Court  analysed  15  out  of  the  25
evaluation reports that dealt mainly with macroeconomic
aspects.  The  audit  was  carried  out  at  the  Commission
and  in  eight  Member  States  (Belgium,  Greece,  Spain,
France,  Ireland,  Italy,  the  Netherlands  and  Portugal),
where  the  auditors  consulted  the  assessors  and  national
authorities  concerned.

are often inadequate. The Member States should improve
their  quality  (see  paragraphs  4.14-4.21).

3.5. Many  evaluations  are  completed  late,  which  may
proportionately  delay  the  adjustment  of  programmes.
Furthermore,  the  decision-making  system  could  take
the  performance  or
evaluation 
effectiveness  of  intervention  schemes  better  into  account
for  future  programmes  (see  paragraphs  4.22-4.27).

results  concerning 

3.  SUMMARY  OF  THE  MAIN  OBSERVATIONS

3.6. With  regard  to  the  objectivity  of  evaluations,  the
Commission  makes  no  on-the-spot  checks  of 
the
consistency  and  reliability  of  indicators  or  the  systems
that  produce  them  (see  paragraphs  4.29-4.30).

at 

and 

spread, 

leading  to 

its  recognition 

3.1. Evaluation  developed 
the
Commission’s  initiative,  mainly  during  the  1994-1999
programming  period,  in  particular  in  the  context  of  the
ERDF, 
in  the  field  of
management and programming. As from the beginning of
this  programming  period, 
evaluation  has  been
systematically  applied  to  the  first  two  phases  (prior
appraisal and ongoing assessment) and will be applied to
the  third  at  the  end  of  the  period  (ex  post  evaluation)
(see  paragraphs  4.2-4.8).

3.2. Although the Commission has the figures regarding
its  own  direct  expenditure  on  evaluation,  it  does  not
always have access to all the information for expenditure
on  evaluations  carried  out  under  the  partnership  system
(see  paragraphs  4.9-4.11).

3.3. The  Commission  made  a  considerable  contribution
towards  spreading  the  idea  of  evaluation,  even  though  it
did  not  introduce  an  adequate  framework  for  evaluation
indicators  and  methods  in  its  programme  on  ‘Methods
for  Evaluating  Activities  of  a  Structural  Nature’
(MEANS).  In  view  of  the  complex  nature  of  the
methodological  aspects  of  evaluation,  there  is  no  single
methodology that can be applied to all specific situations.
It  is,  however,  important  that  the  Commission  should  be
in  a  position  to  be  able  to  supply  methodological
frameworks  that  provide  appropriate  guidelines  for  the
implementation of evaluations suited to specific situations
(see  paragraphs  4.12-4.13).

3.4. The quality of the indicators is closely linked to the
reliability  of  the  information  gained  from  the  monitoring
of  existing 
interventions  and  statistical  bases.  The
Member  States  are  responsible  for  these  latter  and  they

in  the
3.7. There  are  still  significant  shortcomings 
Commission’s  procedures 
for  awarding  evaluation
contracts to consultants. The choice of consultants is not
always  clearly  justified.  Contract  specifications  are  very
general in terms of methodological techniques, indicators
and  the  need  to  achieve  results.  This  leads  to  delays  and
disturbances  in  the  submission  of  evaluation  reports,
qualitatively  variable  analyses,  and,  sometimes,  vague
recommendations  (see  paragraphs  4.31-4.36).

forward 

3.8. The  implementation  of  the  ex  post  evaluation
methodology for Objectives Nos 1 and 2 (1989-93) came
up  against  statistical  shortcomings,  a  lack  of  objectives,
an  absence  of  quantified 
indicators,  an
ineffective  monitoring  of  impact  and  was  also  not
sufficiently  stringent.  Nevertheless,  these  evaluations
revealed  that  the  SFs  did  indeed  contribute  Community
added  value,  both  in  terms  of  the  achievements,  results
and  effects  of  the  CSFs,  OPs  and  measures  and  from  the
point of view of programming, organisation, monitoring,
coordination, 
(see
partnership 
paragraphs  4.15-4.19,  5.1,  5.4,  5.10-5.16).

additionality 

and 

3.9. With  regard  to  the  ongoing  assessment  of  the
1994-1999  SF  interventions  carried  out  in  partnership
under  the  aegis  of  the  Monitoring  Committees,  an
examination  of  the  physical  and  financial  results  always
plays  an  important  role  in  mid-term  assessment,  even
though  there  may  be  variations  in  the  priorities  granted
to the various evaluation objectives. The evaluation of the
effects  is  hampered  in  particular  by  problems  with
indicators  and  the  modelling  of  effects.  This  type  of
evaluation  is  still  often  only  of  relative  value  in  terms  of
the decision-making process, the management of projects,
reprogramming 
future
interventions 
(see  paragraphs  4.16-4.21,  6.5,  6.7,
6.14-6.26).

preparation 

and 

the 

of 

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/5

In  the  specific  case  of  Objective  No  2,  the

3.10.
evaluation  revealed  the  following  weaknesses:

(a)  a  mismatch  between  programme  implementation  and
Community  funding,  with  significant  carry-overs  of
appropriations  from  one  programme  to  another
(about  12 %  on  average  for  the  shift  from  the
1994-1996  period 
the  1997-1999  one)(see
paragraph 6.27);

to 

(b) the succession of short programming periods does not
permit  an  optimum  use  of  evaluation  results  (see
paragraph 4.23);

(c)  indirect  effects  are  still  not  measured  sufficiently  to
determine  the  impact  of  structural  interventions,  in
particular  as  regards  employment  (see  paragraphs
5.12-5.15).

3.11. Despite failings in the programming, implementing
and  monitoring  systems,  the  macroeconomic  evaluations
revealed  positive  results,  in  particular  in  respect  of  the
impact  of  structural  measures  (increases  in  GDP,  effects
on  supply  and  demand,  improvements  in  terms  of
employment and convergence and the effectiveness of the
CSF  instruments),  (see  paragraphs  7.1-7.14).

4.2. Since 
the  1988  reform,  evaluation  has  been
explicitly  included  within  the  partnership  system  (Article
26  of  Regulation  (EEC)  No  4253/88  —  described  more
fully in the amended Regulation of 20 July 1993). During
the  first  programming  period  and  up  until  1995,
evaluations  were  nevertheless  mostly  carried  out  at  the
in
Commission’s 
particular  the  ex-post  ones  —  were  hampered  by  a  lack
of 
inadequate
monitoring.  Moreover,  generally  speaking,  they  were  not
really  taken  into  consideration  and  discussed  in  detail  by
the  Monitoring  Committees.

initiative (5).  These  evaluations  — 

specific  development 

strategies  and 

4.3. The  Commission  initially  concentrated  mainly  on
setting up its own evaluation mechanism for all the CSFs
covering  the  reform’s  five  priority  objectives (6).  Among
the  Commission’s  initiatives  in  connection  with  the
development  of  evaluation  and  evaluation  methods,
mention  must  be  made,  in  particular,  of  the  1992
creation  of  the  technical  group  on  the  ‘Evaluation  of
Regional  Policies’  and,  at  a  methodological  level,  the
launching  of  the  programme  on  ‘Methods  for  Evaluating
Activities  of  a  Structural  Nature’  (MEANS)  in  1994.

3.12. Evaluations  of  sub-programmes  and  measures
continue  to  be  inadequate.  Furthermore,  they  come  up
against technical difficulties caused by the dispersion and
inconsistency of many measures and sub-programmes (see
paragraph 7.17).

4.4. For  the  1994-1999  period,  the  Commission  was
responsible  for  a  new  evaluation  system,  which  is  now
systematically  applied  to  the  first  two  phases  (prior
appraisal  and  ongoing  assessment)  and  will  later  be
applied  to  the  ex  post  phase.

4.  GENERAL  ISSUES

Introduction

is 

to  determine 

4.1. The  aim  of  evaluation 
the
socioeconomic  effects  of  the  measures  implemented,  i.e.
to  examine  the  results  obtained,  compare  them  with  the
resources invested and analyse their cost. Furthermore, all
these  points  must  also  be  looked  at  in  relation  to
forecasts.  However,  this  evaluation  task  presupposes  a
good statistical basis, quantified objectives established on
the  basis  of  a  system  of  indicators  and  the  development
of  a  methodological  and  analytical  framework.  At  the
same  time,  it  must  be  supported  by  an  analysis  of  the
relationships  between  the  means  used,  final  results  and
intermediate results. The Court found failings at all these
levels,  and,  despite  real  progress  that  has  helped
consolidate  the  system  of  evaluation,  these  have  so  far
limited  the  usefulness  of  the  evaluations  carried  out  in
connection  with  the  Structural  Funds.

4.5. Although  ongoing  assessment 
is  not  explicitly
provided  for  in  the  SF  Regulations,  as  of  1995-1996  it
has  been  applied  to  all  the  CSFs,  SPDs  and  OPs  for  the
1994-1999  programming  period.  In  concrete  terms,  it
falls  under 
the
Monitoring  Committees  to  turn  it  into  an  integrated

system,  allowing 

the  partnership 

(5)  This  concerns  the  following  evaluations:

(a)  For  the  1989-1993  programming  period:

—  prior  evaluations  of 

the  O1  and  O5b  CSFs

(1990-1991)
ex  post  evaluations  of  the  O1  CSFs  (1994-1995)
—  ex  post  evaluations  of  the  O2  CSFs  (1996-1997)
—  thematic  evaluations  (SMEs,  telecommunications);

(b) for  the  1994-199  programming  period:

—  prior  appraisals  of  the  O1  regional  development

plans

—  prior  appraisals  of  the  O5b  programmes
—  prior  appraisals  of  the  1994-1996  O2  SDPs
—  prior  appraisals  of  the  1997-1998  O2  SDPs
—  critical  analyses  of  the  O1  SDPs  and  CSFs  (1995).
The Commission also requested evaluations by Member State
of  the 
impact  of  European  structural  policies  on  the
economic  and  social  cohesion  of  the  Union.

(6)  DG  XXII’s  memo  of  14  March  1991.

C  347/6

EN

Official  Journal  of  the  European  Communities

16.11.98

management  tool  and  thus  contributing  towards  a  very
idea  of  evaluation.  The
broad  awareness  of 
Commission 
the  phased
system 
implementation  of  ongoing  assessment  in  the  Member
States (7).

the 
set  up  a 

for 

to  be  used 

4.6. However,  this  system  did  not  specify  the  actual
procedures 
(the  definition  of  project
specifications,  the  criteria  for  the  selection  of  assessors,
the  structure  of  evaluation  reports,  data  collection,
quality  standards,  timetables,  etc.).

4.7. At  the  same  time,  the  Commission  organised  a
series  of  events  accompanied  by  a  number  of
publications (8).  By  bringing  the  SF  managers  together
with  experts  in  the  field  of  evaluation,  these  events
contributed  to  the  development  of  the  idea  of  evaluation
and helped raise the managers’ awareness of this subject.
Evaluation  was  presented  primarily  as  a  process  to  be
learnt  by  experience  on  the  job.

4.8.
In  May  1998,  in  the  context  of  the  preliminary
draft  budget  of  the  European  Communities  for  the
financial  year  1999,  the  Commission  presented 
its
‘evaluation  report  1996/1997’.  In  this  working  paper,  it
describes 
in
evaluations  already
1996-1997,  gives  a 

the  evaluation  activities  carried  out 

list  of 

(7)  (a)  The drawing up, in 1994, of the first ‘terms of reference’
for the on-going evaluation of the CSFs and the Objective
1  forms  of  intervention.  These  confirmed  the  role  of
ongoing  assessment  as  an  aid  to  decision  making  for  the
Monitoring  Committees  and  its  implementation  within
the  partnership 
the
procedures  for  its  implementation,  in  particular  the
composition  and  role  of  the  evaluation  team;

system.  They  also 

specified 

(b) the  1995  distribution  of  the  ‘Common  guidelines  for
monitoring  and  ongoing  assessment’  for  the  Community
SFs, which lay down the factors for analysis (overall and
specific  objectives)  and  the  link  between  monitoring  and
ongoing  assessment;

(c)  the  sending  of  letters  to  the  Member  States  (1995)
reminding  them  that  they  were  required  to  carry  out
ongoing  assessment  and  again  specifying  the  desired
implementing procedures.

(8)  Such  as:

(a)  the  Brussels  conference  on  the  evaluation  of  European
regional  policies  (16  November  1995)  concerning  mainly
the  results  of  the  ex  post  evaluations  of  Objective  1
(1989-1993)  and  prior  appraisals  for  1994-1999;

(b) the  European  conference  on  evaluation  methods  for  SF
interventions  (Berlin  2  and  3  December  1996)  mainly
concerning  the  problems  involved  in  assessing  impact  at
thematic
the  micro-economic,  macro-economic  and 
levels;

(c)  the Seville conference of March 1998 aimed at promoting
exchanges  of  experience  in  the  field  of  evaluation  under
the  partnership  system.

implemented  and  currently  underway  and  supplies  a
summary of the results of the evaluations implemented in
the  1996-1997  period.

4.9. Expenditure  on  evaluation  work  may  either  be
financed directly by the Commission, or co-financed with
the  Member  States.  Tables  1  and  2  show  expenditure
directly  settled  by  DG  XVI  in  the  field  of  evaluation  for
the  1989-1993  and  1994-1999  programming  periods
respectively.

4.10. During  the  1989-1993  period,  as  Table  1  shows,
around  10  Mio  ECU  was  spent  on  evaluation.  Around
two  thirds  of  this  expenditure  occurred  in  two  fields
which  come  directly  under  the  Commission’s  jurisdiction
and  for  which  it  is  directly  responsible.  These  were,  on
the one hand, the Community initiatives under Article 11
of Regulation (EEC) No 4253/88, amounting to 3,1 Mio
ECU,  and,  on  the  other  hand,  cross-disciplinary  and
thematic evaluations amounting to 3,2 Mio ECU, such as
the
SMEs, 
concerning 
those 
environment  and  human 
the
amounts  earmarked  for  the  evaluation  of  Objectives  Nos
1  and  2  of  the  Structural  Funds  were  relatively  small,
given  the  fact  that,  during  this  period,  partnership  was
not  very  well  developed  in  the  field  of  evaluation.  As  is
shown  in  Table  2,  the  amount  earmarked  for  evaluation
for the 1994-1999 period amounted to 25,2 Mio ECU in
April  1998.  This  increase  over  the  previous  period  was
due  to  the  development  of  ongoing  assessment  and  its
methodology  within  the  partnership  system.

telecommunications, 
resources.  However, 

4.11. With  regard  to  spending  in  this  field  by  the
Commission and the Member States, which has increased
considerably  since  1996,  the  Commission  is  not  always
fully  informed,  in  particular  where  the  evaluations  are
included  under  the  ‘technical  assistance’  measures  of
Operational  Programmes,  which  may  also  involve  other
forms  of  expenditure.  A  specific  allocation  to  cover  the
needs  of  evaluation  could  be  provided  right  from  the
initial  programming  phase,  which  would  make
expenditure  more  transparent.

Reference  framework

4.12. The  MEANS  programme  (1995-1998)  (2,8  Mio
ECU)  has  played  a  specific  role  in  the  process  of  the
development  of  evaluation (9).  The  procedure  applied
aims  to  encourage  the  development  of  a  uniform  and
consistent  methodology,  via  an  inventory  of  the  methods
used in the Member States and the identification of valid

(9)  This  programme  is  managed  jointly  by  the  Commission  and
an  external  consultant,  whose  involvement  is  financed  from
budget  heading  B2-1  8  2  0,  for  an  annual  amount  of
932 000  ECU.

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/7

examples.  This  essentially  empirical  and  exhortatory
approach  does  not  enable  the  Commission  to  decide  on
the  actual  procedures  to  be  used  for  the  implementation
of evaluation methods or dictate a choice of indicators, in
particular  those  for  measuring  the  socioeconomic  impact
in  the  field  of  regional  development.  In  view  of  the
complexity  of  the  methodological  aspects  of  evaluation,
there  is  no  single  method  applicable  to  every  specific
the
situation. 
supply
to 
Commission 
methodological 
enable  appropriate
guidelines  to  be  drawn  up  for  the  implementation  of
evaluations  suited  to  specific  situations.

important 
in  a  position 

is  nevertheless 

frameworks 

should  be 

that 

to 

It 

in  particular  by 

4.13. So far, the results of the MEANS programme have
been  used  to  lay  the  foundations  of  a  new  evaluation
procedure, 
introducing  a  common
language,  and  have  enabled  the  preparation  of  ongoing
assessments  within  the  partnership  system.  Tables  3  and
4  show  the  generally  accepted  evaluation  procedure.
However,  the  MEANS  programme  has  not  yet  provided
detailed  answers  to  the  questions  regarding  the  practical
and  operational  aspects  of  evaluation.  Thus, 
two
fundamental  questions  remain  unanswered:  firstly,  what
is  the  most  appropriate  system  of  indicators  for  use  in
formulating  an  overall  assessment  of  a  given  Community
structural  programme;  and,  secondly,  what  evaluation
approach 
the
suitable 
socio-economic  impact  of  a  given  structural  programme
in the context of regional development? The Commission
is  not  yet  able  to  draw  up  precise  instructions  regarding
the  actual  techniques  to  be  applied  in  each  specific
evaluation  situation.  The  absence  of  such  instructions  is
felt  particularly  during  the  ongoing  assessments  of
intervention  for  the  1994-1999  programming  period.

for  measuring 

is  most 

Statistical  bases  and  information  concerning  objectives

4.14. Evaluations  (whether  prior  appraisals,  ongoing
assessments  or  ex  post  evaluations)  are  often  confronted
with  objectives  that  have  been  formulated  too  generally,
even  with  regard  to  the  objectives  of  the  measures  in
question.

4.15. The  level  of  development  of  existing  statistical
bases is an impediment to the monitoring and evaluation
of  programmes  and  measures  co-financed  by 
the
Structural  Funds.  Shortcomings  are  evident  at  various
points,  including,  in  particular,  the  lack  of  statistical
coverage  limited  to  the  areas  concerned,  the  absence  of
statistics  regarding  certain  aspects  and  the  collection  of
information at intervals that are inappropriate in relation
to  the  needs  of  monitoring  and  evaluation.

Indicators

the 

4.16. The  existence  of  reliable  and  appropriate  systems
of  indicators  is  an  essential  element  in  the  planning
process.  For 
structural
evaluation  purposes, 
programmes  must  contain  information  on  financial  and
physical  achievement  as  well  as  on  results  and
performance.  These  indicators  should  provide  managers
with  information  on  the  immediate  or  long-term  effects
of  action  taken  and  also  provide  a  measurement  of  the
progress  achieved  in  relation  to  the  overall  objectives  of
the programmes (e.g. regional growth or the reduction of
unemployment).

4.17. The  reliability  of  the  financial  indicators  is  closely
linked  to  that  of  the  expenditure  certificates  sent  to  the
Commission  prior  to  the  relevant  Community  payments.
Physical indicators are often considered to be a source of
supplementary  information  and  their  reliability  is  not
consistent.  Existing  databases  do  not  always  provide  a
record  of  actual  physical  progress  and  final  recipients  do
not  always  provide  managers  with  periodic  information
about  the  physical  state  of  operations.  The  ongoing
assessment  of  the  Lisboa  e  Vale  de  Tejo  OP  in  Portugal
reported  physical 
in  strict  proportion  to
expenditure  without  mentioning  any  variations  linked  to
delays  in  implementation  or  other  causes.  The  Member
States  should  improve  their  monitoring  systems  and  the
databases  on  which  their  indicators  are  based.

indicators 

the  horizontal 

4.18. Both  the  measurement  of  results  and  impact  at
sub-programme  and  measure  level  and  the  measurement
of  impact  at  programme  level  are  hampered  by  the  fact
that  neither  the  causal  relationships  between  the  various
levels  of  objectives  and  between  objectives  and
measures/projects  nor 
effects  of
complementarity  and  synergy  between  measures  and
between objectives can be identified with any certainty. It
is therefore difficult to define the specific effect of a given
measure  on  regional  development  or  the  impact  of
investment  on  demand  or,  a  fortiori,  on  supply.  In  this
context,  with  regard  to  the  1994-1999  CSF,  the  search
for  indicators  that  will  serve  as  an  appropriate  tool  for
evaluating results and impact has still not been completed
(see  paragraph  6.23).

4.19. The  prior  appraisal  of  the  CSFs  is  essential  to
subsequent  evaluations.  It  ensures  that  all  the  result  and
impact indicators have been specified and quantified, thus
enabling a judgement to be made on the estimated impact
of  the  programme  concerned  and  comparisons  to  be
drawn  with  actual  effects. 
the
management by objectives approach has not always been
applied  in  a  uniform  or  satisfactory  fashion,  which  has

this  context, 

In 

C  347/8

EN

Official  Journal  of  the  European  Communities

16.11.98

indicators 

the  other  hand, 

had  an  impact  in  terms  of  the  indicators  necessary  for
evaluation.  Thus,  the  Italian  1994-1999  CSF  contains  a
small  number  of  indicators,  a  fact  which  could  hamper
subsequent  evaluation.  On 
the
1994-1995  Portuguese  CSF,  which  was  approved  by  the
Commission  without  any  major  changes,  includes  too
many 
(about  300),  which  might  also
complicate evaluation. This variation in the establishment
of indicators from one CSF to another highlights the lack
of  consistency  in  programming.  Where  the  system  of
indicators  approved  by  the  Commission  during  prior
appraisal 
and
unquantifiable  objectives,  they  are  sometimes  revised  for
purposes  of  ongoing  assessment  because  they  are  found
to  be  badly  defined  and  insufficiently  functional.

a  number  of 

contains 

general 

and 

structural 

interventions 

locally.  Although  partial 

the
4.20. The 
socioeconomic situations upon which they are brought to
bear  give  rise  to  a  wide  variety  of  indicators  that  are
often  developed 
lists  of
indicators  have  been  sent  to  certain  regional  or  national
authorities  and  have  already  been  the  subject  of  debate,
the  Commission  has  not  drawn  up  any  lists  of  the
indicators  that  it  considers  to  be  both  essential  and
representative  in  terms  of  regional  development  and  in
connection  with  the  various  types  of  measure.

4.21. The  quantification  of  result  and  impact  indicators
in  connection  with
further 
should  be  developed 
monitoring,  and,  at  the  same  time,  this  quantification
should also be reliable. The fact is that it has been known
to  be  unreliable,  thus  failing  to  prevent  cases  where  new
jobs  were  counted  twice  or  to  allow  a  distinction  to  be
made  between  the  gross  and  net  effects  on  employment.
This  was  discovered  by  comparing 
the  ongoing
assessment  report  of  the  Brandenburg  OP  with  the
statistics  concerning  the  people  registered  at  the  Social
Security  office.

Taking  evaluation  into  account

4.22. The contribution of evaluations in terms of a more
efficient  utilisation  of  Community  funds  largely  depends
on  the  quality  of  their  content,  and  on  when  the  results
are received and decisions are made regarding changes in
strategy.  However,  a  number  of  ongoing  assessments  for
the  CSFs,  SPDs  and  OPs  for  the  1994-99  programming
period  were  finalised  rather  late,  thus  delaying  possible
changes.

4.23. With  regard  to  Objective  No  2,  the  succession  of
shorter  programming  periods 
(as  compared  with
Objective No 1) means that it is particularly unlikely that
the  dates  when  programming  decisions  need  to  be  taken

will  coincide  with  those  when  evaluation  results  become
available,  thus  making  it  impossible  to  learn  from  the
past  and  adapt 
interventions.  The
the  subsequent 
Objective  No  2  programming  and  evaluation  cycles  are,
to  a  certain  extent,  out  of  step.  For  example,  the  final
version  of  the  summary  report  on  the  ex  post  evaluation
of  Objective  No  2  for  the  1989-1993  period  was
submitted  to  the  Commission  in  the  summer  of  1997,
while the prior appraisal for the 1997-1999 programming
period  was  underway  and  the  ex  post  evaluation  for  the
1994-1996  period  had  not  started.

for 

structural
appropriations 
4.24. Budgetary 
programmes  and  subsequent  programmes  are  rarely
influenced  by  the  programme  results  shown  up  by  the
evaluation  reports.  During  the  first  programming  period
the  decision-taking  system  very  rarely  took  any  account
in  programmes  were
of  evaluations,  and  changes 
generally  based  primarily  on  the  degree  of  financial
absorption.  This  was 
the  useful
the 
recommendations  made  by  the  head  consultant  in  the
context  of  the  ex  post  evaluation  of  the  1989-1993
Objective  No  1  CSF  (see  paragraph  5.7).

case 

for 

identified 

4.25. The  more  qualitative  aspects  of  performance  or
the  ongoing
programme  effectiveness 
assessments  or  implementation  reports  did  not  have
sufficient  impact  on  the  operational  management  of  the
SFs  or  on  the  development  of  the  sense  of  responsibility
of  the  partners  affected  by  the  development  of  the
programmes.

in 

4.26.
Just  as  for  monitoring,  the  results  of  the  ongoing
assessments  of  the  1994-1996  Objective  No  2  SPDs,
CSFs  and  OPs  reveal  a  divergence  between  the  financing
plan and the implementation of the programmes. Because
of  this  situation,  the  need  to  utilise  appropriations
becomes the main concern and as a result the importance
of  the  role  of  evaluation  in  the  analysis  of  the  area  in
question  may  be  under-estimated,  especially  as  regards
consistency  between 
resources
allocated  and  the  impact  of  structural  interventions.

the  objectives, 

the 

are  often 

evaluations’ 

and
4.27. The 
recommendations 
the
intervention  managers,  although  they  may  be  useful  for
other  parties  involved.  This  is  the  case  for  both  ex  post
evaluations  (see  paragraph  5.8)  and  ongoing  assessments
under  the  partnership  system.

conclusions 
self-evident 

to 

The  objectivity  of  evaluations

4.28. The  lack  of  a  precise  frame  of  reference  is  a
hindrance  to  the  application  of  suitable  techniques  that
can be objectively applied in specific evaluation situations
so  as  to  produce  easily  verifiable  results  and  guarantee
the independence of the assessors. On occasion, however,
the  recommended  method  is  not  objectively  justified,  or

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/9

assessors  are  all  too  willing  to  adapt  their  approaches  to
the  difficulties  encountered  or  select  indicators  on  the
basis  of  the  information  available.

information 

4.29. Furthermore,  independence  from  managers  does
not  always  appear  to  be  guaranteed,  particularly  in  the
field  of 
(see  paragraph  6.11),  while
evaluation  reports  do  not  always  give  the  source  of  the
information  on  which  their  observations  are  based.  This
failing  may  raise  doubts  as  to  the  reliability  and  validity
of  the  information  submitted  in  the  evaluation  reports.

information 

4.30. Objectivity  should  be  guaranteed  by  a  verification
system  based  on  the  quality  of  the 
indicators  (of
implementation,  results  and  impact)  used,  the  reliability
the  degree  of
the  basic 
of 
implementation  of 
the
Commission  departments  analyse  the  information  and
results  submitted  under  the  evaluation  procedure,  but  do
not carry out any on-the-spot checks or verification as to
how  well  the  evaluation  systems  used  in  the  Member
States  actually  work.

the  methodology.  However, 

and 

The  evaluation  contracts  awarded  by  the  Commission

4.31. The  audit  of  the  Commission’s  procedures  for
awarding  evaluation  contracts  showed  up  a  number  of
shortcomings  —  unsatisfactory  selection  procedures,
excessively  vague  specifications,  delays  and  low  quality
reports.

4.32. The  minutes  of  the  meetings  concerning  calls  for
tender  do  not  always  specify  the 
judgemental  and
analytical  factors  that  determined  the  Commission’s
choice  of  consultants.  The  Court’s  audit  did  not  provide
evidence  to  show  that  the  best  experts  from  the  most
appropriate  fields  of  competence  were  selected.

4.33.
In  the  case  of  the  ex  post  evaluation  of  Objective
No  2  and  all  the  1989-1993  CSFs  for  Objective  No  1,
the  selection  procedure  was  used  to  appoint  a  chief
consultant  accompanied  by  other  national  consultants
and  coordinating  consultants,  who  then  subcontracted
the evaluation work to other consultants. In this chain of
consultants,  the  Commission  may  lose  its  control  of  the
evaluation procedure, which is supposed to guarantee the
consistency,  uniformity  and  quality  of  the  results.

4.34. Specifications are sometimes vague in terms of the
description  of  the  work  to  be  carried  out  and  the
obligation  to  achieve  results.  The  wording  of  the
contracts  is  very  general.  Often,  procedures  for  carrying
out  the  evaluation  work  are  left  open  pending  a  further
agreement  with  the  Commission  on  the  methodological
framework.

4.35. There  are  often  delays  in  respect  of  the  schedule
for submitting reports. Thus, in the case of the evaluation
of  all  the  1989-1993  Objective  No  1  CSFs,  the  contract
provided  for  the  report  to  be  submitted  on  28  February
1995,  when,  in  actual  fact,  it  was  not  completed  until
November  1995.  The  Commission  provided  for  no
penalties 
to  comply  with  contractual
obligations  on  deadlines  and,  on  the  contrary,  made  the
final  40 %  payment  on  11  July  1995  under  the  terms  of
Article  4  of  the  contract.

failure 

for 

4.36. The  contracts  do  not  contain  clear,  precise
guidelines  as  to  the  methodology  to  be  followed  and  the
the  Commission
In  consequence, 
results  expected. 
departments  may  react  to  the  reports  in  different  ways.
For  example,  some  of  the  reports  had  been  approved,
even  though  they  were  no  more  than  simple  descriptions
or  vague  analyses  of  the  usefulness  of  the  work  carried
out,  were  based  on  general  statements  by  the  persons
questioned,  or  produced  over-general  recommendations.
In  the  case  of  the  ex  post  evaluation  of  the  1989-1993
Ireland  CSF,  one  department  recommended  terminating
the  consultants’  contract  because  of  the  bad  quality  of
their  report,  whereas  another  department  approved  it.  In
the  case  of  the  ex  post  evaluation  of  the  1989-1993
Portuguese  CSF,  the  departments  concerned  agreed  that,
rather  than  providing  an  assessment,  the  report  did  no
physical
of 
aspects 
the 
reiterate 
more 
implementation  already  presented  by 
the  national
authorities. Similarly, it was pointed out that the ex  post
evaluation  report  for  the  1989-1993  Spanish  CSF  was
too descriptive and that it was not sufficiently analytical.
The  ex  post  evaluation  of  the  1989-1993  Northern
Ireland  CSF  was  judged  to  be  incomplete.  The  report  on
the ex post  evaluation of the 1989-1993 Italian CSF was
considered 
to  be  badly  structured  and  organised.
Although  the  evaluation  report  for  the  1989-1993  Greek
CSF  has  chapter  by  chapter  conclusions,  there  is  no
general conclusion.

than 

5.  THE  EX  POST  EVALUATION  OF  OBJECTIVE  Nos  1

AND  2  CSFs  (1989-1993)

Introduction

5.1. Assessors  are  often  faced  with  the  following
difficulties:

(a)  initial  programming  that  has  often  been  carried  out
the
hastily  and  without  a  prior  appraisal  of 
programmes, 
intrinsic
in  particular  as 
economic  fundamentals,  especially  with  regard  to
Objective  No  1  regions.

regards 

(b) the  absence  of  quantified  objectives  and  result

indicators;

(c)  gaps 

in 

the 

the
statistics  with 
socio-economic  situation  both  before  and  after
Objective  No  2  SF  intervention,  in  particular  with
regard  to  the  industrial  fabric,  investments  and  gross

regard 

to 

C  347/10

EN

Official  Journal  of  the  European  Communities

16.11.98

regional  product.  The  statistical  tools  used  do  not
always  correspond  to  the  area  selected  for  analysis
(boroughs,  districts,  employment  catchment  areas);

(d) the 

fact 

that  certain  measures  have  not  been

completed;

(e)  the  existence  of  exogenous  factors  that  help  explain
the  development  of  the  socio-economic  situation;

(f)  where  the  allocation  of  aid  is  not  linked  closely  to
regional  or  local  needs,  the  heterogeneous  nature  of
the  projects  and  their  objectives;

(g)  it  is  hard  to  identify  the  intrinsic  effects  of  the
measures  and  programmes,  because  of  the  barrage  of
measures  aimed  at  the  same  region  or  area;

(h) the  difficulty  of  measuring  the  indirect  effects  of  a
programme  (in  particular  as  regards  employment),
because  no  system  of  observation  was  set  up  for  this
purpose when the programmes were launched and no
detailed  inquiry  has  been  carried  out  on  this  subject
since  their  completion.

The  ex  post  evaluation  of  the  Objective  No  1  CSFs
(1989-1993)

5.2. The  ex  post  evaluation of the Objective No 1 CSFs
was  coordinated  by  a  central  consultant  and  comprises
the  ex  post  evaluations  of  the  CSFs  concerned  that  were
carried  out  by  national  consultants  between  January
1994  and  June  1995.  The  consultant  responsible  for
coordination  developed  the  methodology  that  was  to  be
applied,  which  made  use  of  analytical  methods,  at  both
macroeconomic and measure/project levels. The approach
followed  aimed,  firstly,  to  analyse  the  strategy  and
intrinsic  logic  of  the  CSFs,  OPs  and  measures,  secondly,
to  provide  a  preliminary  evaluation  of  physical  progress
and  impact,  and,  finally,  to  examine  the  implementation
systems.  This  procedure  is  summarised  in  Annex  I.

consistent, 

5.3. Although  generally 
the  proposed
methodology  does  have  shortcomings.  Thus,  it  provides
that  analysis  of  the  intrinsic  logic  and  strategy  of  the
CSFs,  OPs  and  measures  should  be  based  on  a  uniform
structure  of  objectives,  linked  to  a  standard  classification
of  activities.  This  approach  might  allow  comparison
between Member States, but it does not necessarily allow
objectives to be listed in order of importance on the basis
of  both  the  Member  States’  priorities  and  the  criteria  of
hierarchy,  complementarity,  substitutability  and  synergy.
Nor  does  it  allow  the  identification  of  all  inconsistencies
in  the  strategies  specific  to  each  of  the  CSFs  and  OPs.

The  methodology  is  supposed  to  provide  answers  to
relevant  questions  about  intentional  and  unintentional
effects,  but,  in  many  cases,  the  physical  and  impact
indicators  given  in  the  methodological  paper  are  unable
to  provide  them.

5.4. The  Court’s  audit  also 
shortcomings:

found 

the 

following

(a)  as  regards  the  development  of  the  socioeconomic
environment of the CSFs and OPs, a comparison was
not  always  made  between  the  initial  situation  upon
intervention  was  to  act
which  the  Community 
(1988-89)  and  the  situation  at  the  end  of 
it
(1993-94).  Data  were  sometimes  out  of  date,
particularly  those  concerning  regional  differences;

(b) there was often insufficient analysis for the ranking of
objectives,  which  is  required  by  the  methodological
guidelines.  Either  it  was  not  done  at  all,  or  it  was
only  done  partially  —either  just  for  those  OPs  and
measures  that  were  selected  for  specific  examination
or  without  systematically  applying  the  structuring
criteria.  As  a  result,  the  ranking  of  objectives  and
instruments 
sometimes  did  not  produce  any
improvement  in  the  identification  of  problems  of
consistency and the relevance and relative importance
of  priorities;

(c)  adjustments  are  sometimes  made  to  CSFs  and  OPs

without  a  thorough  analysis  of  the  causes;

(d) financial implementation, implementation systems and
physical  implementation  are  generally  well  dealt  with
in  the  evaluation  reports.  However,  in  certain  cases,
either  the  effects  of  programmes  or  measures  were
not  examined  except  in  terms  of  job  creation,  or  the
results  were  formulated  in  non-quantitative  terms.
is
Sometimes  the  only  quantification  of  effects 
provided  by  applying  macroeconomic  models  and  is
concerned  with  increases  in  GDP,  job  creation  and
competitiveness.  This  shows  the  inadequacy  of  the
data  provided  by  managers 
in  the  context  of
monitoring  systems;

(e)  bottom-up 

analyses  did  not 

allow
conclusions  to  be  drawn  in  terms  of  impact  at
programme level;

generally 

(f)  Community  added  value  is  sometimes  only  expressed
in  qualitative  terms  (for  example,  the  taking  into
account  of  the  programme  logic,  the  implementation
of  an 
integrated  strategy  or  the  effect  on  the
organisation process). Furthermore, it was not always
investigated systematically;

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/11

(g)  anticipated  long-term  effects  were  not  discussed  or

were  discussed  only  in  qualitative  terms.

5.8. The ex post evaluation of all the 1989-93 Objective
No  1  CSFs  carried  out  by  the  national  assessors  mostly
concentrated  on  the  improvement  of  the  Member  States’
monitoring  and  management  systems  rather  than  on  a
real  exercise  in  the  assessment  of  the  impact  of  the  SFs.

5.5. The Spanish Objective No 1 CSF was the subject of
both  an  ex  post  evaluation  ordered  by  the  Commission
and  another  one  organised  by  the  national  authorities
and  co-financed  by  the  Communities.  Both  studies  were
drawn  up  during  the  same  period  and  pursued  the  same
objectives  but  were  conducted 
independently  and
separately.

5.6. With  regard  to  results  and  impact,  the  ex  post
evaluations  highlight  the  physical  achievements  of  the
Objective  No  1  CSFs,  the  macroeconomic  impact  of  the
CSFs  observed  in  certain  national  evaluations,  and  the
progress made in the field of planning —although in this
field, the programme-based approach encountered certain
obstacles (10).  They  also  stress  the  CSFs’  contributions  in
terms  of  innovative  organisational  structures,  financial
management,  monitoring, 
systems,
coordination  and  partnership,  although  with  substantial
progress  still  being  needed  at  each  of  these  levels.

information 

5.7. The  chief  consultant’s  1995  summary  included
some useful recommendations that the Commission could
not  follow  up  completely.  One  of  these  was  that,  when
programmes  were  being  drawn  up,  a  clearer  distinction
should  be  made  between  Community 
structural
interventions  designed  to  encourage  endogenous  growth
factors  and  those  aimed  at  the  restoration  of  a  regional
equilibrium  or  a  raising  of  living  standards.  Another
concerned  striking  a  balance  between  the  priority  given
to 
investments  and  specific  regional
characteristics  that  needed  development  or  finding  an
equilibrium between the fight against unemployment and
the  need  to  increase  productivity  and  competitiveness.
Finally, it recommended that regional policies should pay
more  attention  to  employment.

infrastructure 

(10) The  programme  by  programme  approach  was  hampered  by

the  following:
i) 

the  short  planning  deadlines  implying,  in  particular,  the
absence  of  any  prior  appraisal;
the  lack  of  tangible  objectives,  quantified  indicators  for
implementation,  results  and  impact;

ii) 

iii)  administrative  constraints, 

in  particular  where  the
sub-programmes  of  a  given  programme  fall  under  the
responsibility of different authorities, or the sectoral and
regional approaches overlap;

iv)  weaknesses 

in  the  Community  rules,  resulting 

in
particular,  in  projects  rejected  for  the  Objective  1  OPS
being  accepted 
in  the  context  of  the  Community
initiatives;
the  project  by  project  approach  of  the  Cohesion  Fund,
which  also  did  not  contribute  towards  a  harmonious
application  of 
the  programme  by  programme
approach.

v) 

Ex-post  evaluations  of  Objective  No  2  (1989-1993)

5.9. The  ex  post  evaluations  of  the  Objective  No  2
programmes  for  the  1989-1993  period  were  coordinated
by  an  external  assessor  in  the  60  eligible  regions.  A
general  analysis  was  made  for  each  of  the  regions,  while
detailed  studies  were  made  of  22  regions.  The  main  aim
of  these  evaluations  was  to  analyse  the  impact  of  these
structural  interventions  on  the  industrial  redeployment
process  in  these  regions  and  draw  lessons  for  the  next
programming  phase.  A  summary  of  this  evaluation
approach  appears  in  Annex  II.

5.10. Although  a  common  framework  was  envisaged,
the  approaches  adopted  are  very  heterogeneous.  This
concerns particularly the priorities adopted and the depth
of  the  analysis,  whether  it  is  descriptive  or  critical,  and
the  methods  and  techniques  used.

5.11. Analysing strategies and the appropriateness of the
CSFs and OPs and the changes that they have undergone
in  relation  to  the  socio-economic  environment  seldom
leads to a reformulation of the structure of the objectives,
sub-programmes  and  measures  in  accordance  with  the
logic  of  regional  development  or  redeployment  and  on
the  basis  of  the  criteria  of  ranking,  complementarity  and
substitutability.  Whereas  many  strategic  analyses  based
on  the  structure  of  the  CSFs  and  OPs  are  partial  and
uncritical  with  regard  to  the  strategy  concerned,  a
reformulation  of  this  kind  should,  after  due  elaboration,
make  it  possible  systematically  to  identify  the  strategy’s
shortcomings  in  terms  of  consistency,  the  weighting  of
priorities,  synergy,  complementarity  and  effectiveness
vis-à-vis  the  socio-economic  environment.

5.12. As  for  the  impact  of  the  SF  interventions,  the
assessors  are  not  able,  on  the  basis  of  a  sample  of
projects and an examination of their results, added value
and additionality, to aggregate their observations in terms
of  individual  measures  and  the  socio-economic  impact  of
the  programme  concerned.

5.13. Few  impact  evaluations  were  based  on  a  genuine
combination  of  top-down  and  bottom-up  approaches.
in  particular,  used  for  the
This  combination  was, 
in
evaluation  of  the  Objective  No  2 

interventions 

C  347/12

EN

Official  Journal  of  the  European  Communities

16.11.98

services 

intervention 

(infrastructures, 

Tuscany.  In  this  case,  the  top-down  approach  used  a
bi-regional  (Tuscany/rest  of  Italy)  econometric  model  to
estimate  the  overall  impact  of  an  increase  in  demand  on
production  by  sector,  consumption  levels  and  added
value,  both  in  Tuscany  and  the  rest  of  Italy.  This  model
also makes it possible to estimate the multiplier effects by
type  of 
for
businesses,  aid  for  private  investment)  and  the  effects  on
employment (11).  The  direct  effects  on  employment  were
estimated  using  a  technique  of  standard  coefficients
concerning  the  various  types  of 
intervention.  This
top-down  approach  therefore  regarded  direct  short-term
effects  related  to  demand.  The  bottom-up  approach,
which  is  based  on  a  sample  of  projects  and  measures,  is
more capable of reflecting long-term, more formative and
indirect  effects,  relating  especially  to  supply  and  external
savings  that  enable  reductions  in  production  costs.  Even
though  these  effects  are  more  difficult  to  isolate  and
this  approach  does  not  enable  an
quantify  and 
aggregation of the effects in terms of individual measures
and  programmes  (which,  however,  was  the  objective  of
detailed  evaluation),  qualitative  considerations  as  to  the
effectiveness  of 
the  degree  of
the  measures  and 
additionality  of  Community  interventions  constitute  a
useful  complement 
top-down  approach.  A
the 
combination  of  both  approaches  can  provide  real  added
value  in  terms  of  analysis,  in  particular  if  it  is  based  on
the  techniques  of  modelling  or  formalisation.

to 

generally  provided  for  in  the  evaluation  procedures  only
seldom made it possible to obtain precise information on
the  physical  implementation,  results  and  impact  of  the
operations.

5.15.
In  the  field  of  job  creation,  the  results  are  not
clear  in  terms  of  the  durability  of  the  jobs  and  the
structural
specific 
measures. 
the
evaluation  reports  is  sometimes  contradictory.

to 
the  data  produced 

contribution  ascribable 

In  addition, 

the 

in 

5.16. With  regard  to  the  impact  of  the  Objective  No  2
programmes,  we  can  draw  the  following  conclusions
from  the  ex  post  evaluations:

in 

the 

improvement 

(a) although  the  Objective  No  2  OPs  do  not  often  bring
socio-economic
about  an 
situation,  they  at  least  contribute  towards  avoiding
further  deterioration  (in  particular  as  regards  the
maintenance  of  employment (12))  and  preparing  the
ground  for  industrial  redeployment.  Indeed,  most  of
the  regions  were  seeking  to  diversify  their  economies
and their industrial bases and, where possible, cast off
their  reputations  as  exclusively  industrial  areas;

(b) despite 

the 

integrated  procedure,  based  on 

the
programme  concept,  the  OPs  sometimes  consist  of  a
modification  of  national  and  regional  policies  that
have already been set up and do not always constitute
a  programming  tool;

5.14.
Impact  evaluations  that  make  use  of  systematic
inquiries  (such  as  a  number  of  investigations  concerning
the  additionality  of  measures  or  the  application  of  a
specific  method  for  estimating  net  measure-related  job
creation  by  comparing  an  experimental  group  with  a
control  group)  are  also  rare.  However,  they  do  provide
real  added  value  in  terms  of  impact  evaluation.  The
effectiveness  of  measures  and  programmes  would  have
been  more  accurately  evaluated  if  assessors  had  actually
conducted  the  statistical  analyses  of  certain  result
indicators  that  they  had  sometimes  said  that  they  wished
to  carry  out.  Thus,  it  is  not  always  possible  to  find  data
on  the  number  of  companies  located  in  new  industrial
zones set up with the help of SF aid, or on technological
transfers  from  research  establishments  to  businesses.  The
meetings  with  measure  and  project  managers  that  were

(c) the simultaneous existence of the Objective No 2 OPs
and  the  Community  initiatives  gave  rise  to  confusion
and  to  the  overlapping  of  aid;

(d) SF  intervention  had  positive  effects  in  terms  of
programming,  implementation  methods,  partnership,
synergy,  coordination  and  evaluation.  However,  the
lack  of  synergy  and  communication  between  ERDF
and  ESF  managers  and  measures  was  often  stressed;

(e) as  regards  additionality,  the  SFs  make  it  possible  to
tackle  new  target  areas,  carry  out  projects  which
would  have  remained  unrealised  without  Community
aid,  speed  up 
implementation,  carry  out  more
ambitious  projects  or  give  a  formative  character  to
initial  projects.  The  additionality  of  different  types  of
measure  (aid  for  infrastructures,  business  services,  aid
for  private  investment)  may  vary  considerably;

(11) This does not, however, mean that these are job creations, as
the  econometric  model  is  based  on  the  hypothesis  of  full
employment  and  companies  with  production  overcapacity
during  down-turns.

(12) The  8th  annual  report  on  the  SFs  mentions  important
leverage  effects  which  refer  to  overall  results,  such  as  the
creation  of  jobs  or  companies.

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/13

(f)  the SFs do not often act as a lever for the involvement
of private money or, at least, the evaluations omit this
aspect;

measurement  of  results  as  regards  employment  and
SMEs.

(g) the  evaluations  stress 

that  studies,  preparatory
negotiations,  administrative  procedures  relating  to  the
adoption  and  implementation  of  certain  projects  and
the  integration  of  new  national  policies  all  involve
delays  that  are  often  underestimated.

In  addition, 

the  Commission 
6.3.
evaluations  should  be  general  and 
evaluations 
concerning  one  particular 
geographical  area  should  be  developed  in  a  later  phase.

that
thematic
sector  or

suggested 
that 

techniques  used 

5.17. With regard to management, the evaluations show
the
that  certain 
programmes  were  inadequate,  and  that  the  internal
changes  made  during  the  programming  period  mainly
concerned  the  maximisation  of  the  rate  of  absorption  of
appropriations  rather  than  the  revision  of  regional
development  priorities.

for  monitoring 

6.4. Certain  Member  States,  in  particular  Spain (13),
France (14),  Ireland (15)  and  the  Netherlands (16),  also
drew  up  frameworks  for  ongoing  assessment.  They  all
look  at  the
feel  that  ongoing  assessments  should 
following:

(a)  the  development  of  the  socioeconomic  context;

(b) strategies  and  internal  and  external  consistency;

6.  ONGOING  ASSESSMENTS  (1994-1999)

The  development  and  integration  of  the  principles  of
ongoing  assessment

in  the
6.1. Ongoing  assessments  are  provided  for 
chapter  on  implementing  provisions,  dealing  with  the
improvement  of  monitoring  (included  in  each  of  the
CSFs,  SPDs  and  other  forms  of  intervention).  It  is
involve  a  critical
stressed  that  ongoing  assessments 
analysis  of  the  data  collected  during  monitoring,  explain
any  differences  in  relation  to  the  original  objectives  and
assess  the  intended  results  of  the  measure  in  question.
They  also  contain  an  opinion  on  the  validity  of  the
current  intervention  and  the  relevance  of  its  objectives.

the  operating  conditions  of 

6.2.
In  the  letters  that  it  sent  to  the  Member  States
during  the  financial  year  1995,  the  Commission  stressed
that ongoing assessments were to provide the Monitoring
Committees  with  appropriate  information  as  to  whether
the  programmes  were  running  smoothly,  put  forward
possible 
reprogramming  measures  and  present  an
independent,  objective  picture  of  the  progress  made.  In
the  initial  phase,  it  was  also  proposed  that  they  should
concentrate  on 
the
interventions  and,  in  particular,  on  the  indicators,  the
management  abilities  of  the  persons  in  charge  and  the
procedures used for selecting projects. Subsequently, with
a  view  to  proposing  necessary  changes  and  adapting
programming,  evaluation  reports  were  to  look  at  the
relevance  of  the  strategies  employed,  the  validity  of
the
intervention  objectives, 
programmes  in  terms  of  the  work  carried  out,  the  initial
results  in  relation  to  objectives,  the  efficiency  of  the
programmes,  the  correct  application  of  Community
policies  (environment,  public  contracts,  etc.)  and  the

effectiveness  of 

the 

(c)  the 

implementation 

including
programming  and  selection  criteria,  administrative
organisation,  the  monitoring  system  and  project
financing channels;

conditions, 

(d) implementation  and  the  results  achieved,  taking  into

account  intended  results;

(e)  initial  effects  obtained,  together  with  an  appraisal  of

expected  effects;

(f)  costs,  with  a  view  to  measuring  efficiency,  i.e.  the

relationship  between  expenditure  and  results.

It  was  also 
that,  where  necessary,  ongoing
assessment,  should  also  lead  to  proposals  for  changes  in
respect  of  priorities  or  intended  measures.

felt 

6.5. The partnership system was introduced in 1988 as a
means for ensuring the effectiveness of the SFs, but it has
only  played  its  full  role  in  the  system  of  decentralised
management  since  the  start  of  the  ongoing  assessment  of
all the 1994-1999 CSFs in 1995. On this occasion, it was
decided  that  the  Monitoring  Committees  should  be
responsible  for  establishing  the  objectives  of  evaluations
and  the  required  quality  standards  needed  to  guarantee
the  overall  consistency  of 
the  process,  adapting
specifications  to  the  characteristics  of  the  regions  in
question  whilst  respecting  the  objectives  of  the  exercise
and selecting assessors on the basis of shortlists prepared
by the persons responsible for evaluation (members of the

(13) Evaluación  intermedia  de  programas  operativos  regionales
del  objetivo  no  2  (fase  1994-1996).  Orientaciones  sobre
contenidos  comunes  a  recoger  en  los  pliegos  de  condiciones
te´cnicas,  23  May  1996.

(14) DATAR,  Regulatory  and  administrative  timetable.
(15) General  criteria  to  underpin  Terms  of  Reference  for
Operational

of 

Mid-Term 
Programmes/Community Initiatives.

Evaluation 

(16) Evaluatiekader  voor  de  programma’s  van  de  Europese
Structuurfondsen,  Terp  Advies-  CBEA  Pieter  Van  Run,
December  1997.

C  347/14

EN

Official  Journal  of  the  European  Communities

16.11.98

Monitoring  Committee  and  Community  rapporteurs)
after  a  public  call  for  tenders.  Ongoing  assessments  have
made a real contribution to strengthening the partnership
system, by allowing it to participate in the causal analysis
of  structural  interventions.  Tables  5  and  6  show  the
operational  implementation  of  ongoing  assessments  and
the  significant  work  carried  out  by  Member  States  under
the  partnership  system  in  terms  of  identifying  the  impact
of  the  SFs.

know-how but also the impartiality of conclusions about
the 
legitimacy  of  the  strategy,  the  consistency  of
socio-economic  analyses  and  the  objective  quantification
of  the  intended  effects.  Evaluation  requirements  in  the
area  of  the  Structural  Funds  have  created  in  the  space  of
a few years a network of external assessors known to the
Commission  and  to  the  Member  States,  which 
is
advantageous  insofar  as  it  generates  greater  expertise  in
this  area  but  which  focuses  on  a  specific  market.

6.6. The  partnership  approach  to  ongoing  assessment
has  also  strongly  influenced  the  institutional  structures
and  the  national/regional  traditions  of  the  public  bodies
responsible.  It  has  made  for  more  widespread  knowledge
of  the  idea  of  multiannual  programming,  the  evaluation
of which is essential to drawing up critical assessments of
structural  interventions  and  preparing  any  necessary
adjustments.  In  the  field  of  evaluation,  partnership  is
particularly  well  developed  in  connection  with  Objective
No  1.

to  all 

involved, 

6.7. At  present,  the  aim  of  ongoing  assessment  work  is
to  supply  overall  solutions 
from
programme  designers  to  managers  and  political  leaders.
This  work  results  in  very  bulky  reports  containing
information  from  the  programmes  themselves,  prior
appraisals,  annual  implementation  reports  and  ongoing
or  ex  post 
to
information  gleaned  orally  during  the  audits,  only  some
of the results and recommendations are of any real use in
terms  of  assisting  decision-making,  managing  projects,
reprogramming  or  preparing  the  next  generation  of
interventions.

evaluations.  However,  according 

Implementation  procedures

6.8. Of  the  389  intermediate  evaluations  undertaken,
320  concerned  Objectives  Nos  1,  2,  5b  and  6 (17).  The
implementation  of 
to
significant  delays  in  several  Member  States.

these  evaluations 

is  subject 

6.9. The  intermediate  evaluation  process  was  originally
envisaged  as  comprising  two  distinct  stages.  The  first
stage  involved  an  assessment  of  the  scope  for  evaluation
and  the  identification  of  related  problems.  The  second
consisted  of  the  evaluation  itself  and  the  subsequent
drafting of a report. In practice, these arrangements were
seldom applied.

6.10. As a rule, the Commission insisted on intermediate
evaluations being carried out by external experts so as to
individual
guarantee  not  only 

the  credibility  of 

(17) 36  of  which  were  examined  on  the  spot  and/or  at  the

Commission.

In  practice, 

6.11.
the  Court’s  on-the-spot  checks
highlighted  the  limitations  on  the  independence  of  the
external  consultant.  Given  the  limited  time  available  to
the assessors and the consultant’s de facto dependency on
programme  managers  for  the  necessary  information,  the
separation  of  the  respective  duties  is  not  very  rigorous.
Too  often,  the  results  obtained  in  terms  of  performance
merely  reproduce  and  update  information  from  the
monitoring  system  while  presenting  it  as  an  external
contribution.

6.12. Furthermore,  the  usual  procedure  of  an  open
invitation  to  tender  based  on  publication  in  the  Official
Journal was not always applied with a view to selecting a
consultant  to  carry  out  assessments.  In  the  Netherlands,
the  procedure  consisted  of  asking  a  limited  number  of
consultants  on  a  list  of  potential  assessors  to  submit  a
bid.  The  selection  was  then  made  from  these  bids.

6.13. Although the Commission regularly participates in
monitoring  committee  meetings  and  is  able  to  win
acceptance  for  its  criteria  for  selecting  consultants,  its
involvement  in  this  process  is  inconsistent.  Thus,  in
Belgium, Greece and Portugal the Commission is actively
involved  in  the  procedure  for  selecting  assessors  within
the  framework  of  the  technical  groups  or  the  ad  hoc
technical  accompanying  committee  which  prepare  the
choice  of  consultant  and  monitor  the  corresponding
implementation  and  evaluation.  By  contrast,  in  France
the  Commission  is  officially  informed  only  once  the  two
bids deemed to be the best have been selected; these latter
are  accompanied  by  a  final  proposal  in  respect  of  which
the  Prefect  of  the  Region,  in  his  capacity  as  contracting
authority,  asks  for  any  observations  to  be  brought  to  his
attention.  As  regards  the  Netherlands,  the  Commission,
the  national
which 
accompanying 
ante,
intermediate  and  ex  post  evaluations  of  the  regions
covered  by  Objectives  Nos  1  and  2  and  in  the  work  of
the  monitoring  committees,  reacted  only  belatedly  to  the
content of the draft intermediate evaluation report on the
Flevoland  Objective  No  1  SPD.

committee  which  pilots 

the  work  of 

takes  part 

ex 

in 

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/15

Priority  given  to  evaluation  objectives

with  regard  to  synergies  between  ERDF  and  ESF
activities.

the 

regards 

implementation  of  ongoing
6.14. As 
evaluations,  the  priority  given  to  evaluation  objectives
varies,  with  the  exception  of  the  examination  of  the
physical and financial results of the measures, which is an
important  aspect  of  any  ongoing  evaluation.  Priority
developments  often  correspond  to  a  genuine  regional
need,  such  as  in  Corsica,  where  the  emphasis  is  placed
mainly  on  dealing  with  problems  of  management,
monitoring  and 
links  between  evaluation  and
monitoring.  Moreover,  the  pursuit  of  certain  evaluation
objectives is hampered by technical problems, such as the
measurement  of  evaluation  and  impact  indicators.

the 

6.15. Very  often,  ongoing  evaluations  do  not  accord
sufficient  importance  to  critical  appreciation  of  the
regional  development  strategy  chosen,  to  the  viability  of
the  projects  financed  by  the  Structural  Funds,  to  the
criteria  for  selecting  projects,  and  to  the  effects  and
impact  of 
specific  and  general
objectives.

the  programmes’ 

6.16. Community  added  value,  the  analysis  of  which
was  one  of  the  objectives  of  the  ex  post  evaluations  of
the  Objective  No  1  and  Objective  No  2  CSFs  for  the
1989-1993  programming  period,  is  covered  only  in
exceptional  cases  by  ongoing  evaluations.

6.17. Changes  in  the  socio-economic  context,  where
Community  assistance  takes  effect  and  in  respect  of
which  evaluation  contracts  generally  provide 
for
monitoring and updating, are dealt with systematically in
certain  cases  (e.g.  by  means  of  very  detailed  progress
charts  in  Wallonia)  and  in  an  unsatisfactory  manner  in
other  cases,  which  has  a  direct  influence  on  the  quality
and updating of the analyses of strengths and weaknesses
which  are  generally  carried  out  as  part  of  ongoing
evaluations.

(the  objectives  of 

6.18. The  extent  and  maintenance  over  time  of  the
internal  and  external  coherence  of  all  of  a  programme’s
components 
the  programme,
sub-programmes  and  measures,  and  the  degree  of
complementarity  and  synergy  between  objectives  and
measures)  are  examined  only  rarely  in  a  systematic
manner.  Thus,  in  the  case  of  the  Brandenburg  OP,  the
financial  commitments 
in  respect  of  priorities  1.1
(productive  investments)  and  1.2  (infrastructure)  account
over 
for  138 %  and  242 %
respectively  of  the  initial  estimate,  and  the  assessor
provided  no  justification  for  the  regional  development
strategy  that  was  implemented.  Synergies  are  sometimes
subject  to  analyses,  which  are  often  incomplete.  This
aspect  warrants  more  systematic  treatment,  in  particular

the  1994-96  period 

of 

the 

reorientation 

a
6.19. Similarly,  where 
programme’s  priorities  and  of  the  allocation  of  resources
is  an  explicit  objective  of  the  evaluation,  the  evaluation
does not always produce a consistent overall presentation
the  proposed  amendments  or  an  explanatory
of 
the
the 
memorandum.  Moreover, 
extensions  or  curtailments  proposed  in  respect  of  the
measures  are  not  always  specified.

implications  of 

6.20. Many  of  the  evaluations  examined  showed  that
the  assessor  had  not  been  able  to  achieve  all  of  the
objectives  envisaged  in  the  contract,  in  particular  as
regards 
impact.
Moreover,  the  time  and  resources  available  were  not
always  proportionate  to  the  objectives.

the  evaluation  of 

the  measure’s 

Approaches,  methods  and  techniques

in  general 

6.21. As  contracts  often  formulate  evaluation  objectives
and  approaches 
the  choice  of
approaches,  methods  and  techniques  is  often  left  to
assessors,  who 
during
implementation  to  the  difficulties  they  have  encountered.
Thus,  the  assessors  occasionally  make  indicators  out  of
the  only  information  at  their  disposal.

sometimes 

terms, 

adapt 

them 

consensus 

to 
evaluating 

6.22. Techniques  and  methods  have  not  always  been
developed  to  meet  evaluation  objectives.  Moreover,  no
the  appropriate
exists  as 
genuine 
operational  methodology 
regional
for 
development;  the  Commission  merely  circulates  ‘best
practices’  in  general  terms  without  actually  coordinating
or  harmonising  their  implementation  in  a  concrete  way.
This  imprecision,  together  with  disparities  between  the
Member  States  as  regards  programming,  implementation
and  evaluation,  results  in  evaluation  reports  of  variable
quality and makes it very difficult to compare results at a
European level.

6.23. Measuring  the  impact  of  measures  thus  reveals
shortcomings  in  the  techniques  applied,  in  particular  as
regards  the  characteristics  and  complexity  of  the  effects
to  be  evaluated:

(a)  a  detailed  search  for  even  more  relevant  indicators
has  not  always  been  carried  out,  and  the  envisaged
indicators  sometimes  serve  only  to  measure  physical
and  financial  results;

(b) impact 

indicators  have  not  always  been  given

substance;

(c)  evaluation  of 

the 

socio-economic  or 

regional
development  impact  appears  to  be  hampered  by  the
increasing  number  of  measures,  sub-programmes  and
measures  (Objective  No  1),  whilst  many  programmes

C  347/16

EN

Official  Journal  of  the  European  Communities

16.11.98

implemented 

for  assisting  the  conversion  and/or  revitalisation  of
industry  (OPs,  Community  initiatives  and  national
programmes)  may 
jointly.
be 
Furthermore,  Community  and  national  measures
sometimes  differ  in  their  application,  as  do  the  dates
on  which  measures  are  launched.  Lastly,  objectives
are  often  unquantified  and  are  formulated  too
generally (18).  In  this  context,  the  links  between  the
diagnosis  of  the  socio-economic  problems  to  be
resolved  and  the  objectives  of  the  series  of  measures,
sub-programmes and projects implemented, as well as
the  links  between  the  objectives  of  this  series,  do  not
always  appear  obvious, 
especially  when  one
remembers  that  the  physical  and  impact  indicators
attributed  to  the  various  levels  of  a  programme  do
not  wholly  reflect 
the  programme’s  evaluation
requirements.  Similarly,  it  is  difficult  to  identify  the
results of each programme and to separate them from
the  results  of  other  private,  regional  or  national
measures;

or 

on 

(Portugal) 

(d) models  for  estimating  the  expected  effects  and  the
impact  of  measures  on  the  basis  of  mid-term  or
anticipated  results  are  rarely  used.  The  ongoing
evaluation  of  certain  national  CSFs  was  based  on  an
a
input-output  model 
macro-econometric  model  (Spain,  Ireland),  whilst  the
use of regional models continued to be very marginal.
The  example  of  the  development  model  for  the
province of Hainault (Belgium) used in the context of
intermediate  evaluations  is  worthy  of  mention  in  this
connection.  The  establishment  of  models  describing
the  effects  of  the  measures  is  not  very  widespread
either,  except  in  the  field  of  transport  infrastructure.
Mention  should  be  made  of  the  application  of  the
input-output  model 
the  Expo  1998  project
(Portugal)  as  part  of  prior  and  ongoing  evaluations,
although  the  estimates  should  be  considered  with
caution;

to 

(e)  estimates  of  the  effects  of  measures  on  employment
are  also  still  precarious.  The  application  of  models
and  other  analysis  and  enquiry  techniques  does  not
supply  adequate  information  concerning,  on  the  one
hand,  the  net  number  of  jobs  created  (taking  into
consideration,  in  particular  the  number  of  jobs  that
would  have  been  created  without  the  measure  or  the
effects  of  job  substitutions),  and,  on  the  other  hand,
the  degree  of  permanence  of  the 
jobs  created.
Moreover,  the  application  of  such  models  and
techniques usually provided no details about new jobs
which  could  not  be  attributed  to  the  implementation
of  the  programme.

(18) E.g. the revitalisation of the countryside, the strengthening of
structural

the 
fabric  of 
modifications  related  to  regional  development.

region  and 

economic 

the 

6.24. Some  evaluation  contracts  provide,  as  part  of  the
bottom-up  approach,  for  enquiries  in  respect  of  local
economic  development  partners  and  final  beneficiaries,
including  companies  which  receive  Community  aid,  with
a view to making a quantitative or qualitative assessment
of the contribution made by the Community programme.
The  enquiries’  findings  are  often  unrepresentative  in  the
light  of  the  number  of  participating  companies  and
partners,  and  are  of  limited  reliability  insofar  as  they  are
not  always  subject  to  scrutiny.

6.25. As  regards  result  and  impact  indicators,  it  was
observed  that:

(a)  the  time  taken  to  obtain  data  and  statistics  is  often

long;

(b) the  existing  data  cover  geographical  areas  which  do

not  correspond  to  the  areas  in  question;

(c)  data  on  companies  are  not  always  recorded  by  the
establishments  concerned,  a  fact  which  distorts  local
statistics  if  a  single  establishment  is  present  in  the
area  but  the  company  itself  has  premises  all  over  the
country;

(d) many statistical bodies are unable to subdivide overall
data  with  sufficient  precision  to  target  an  area  or  a
given  series  of  problems.

6.26. Where 
the  assessor  provides  more  adequate
indicators  with  a  view  to  measuring  impact,  whether  to
compensate  for  a  shortcoming  or  to  replace  those
originally  envisaged,  he  does  not  substantiate  them  and
leaves  the  situation  as  to  their  operational  nature
uncertain.  Furthermore,  he  does  not  always  mention  the
enquiries  still  to  be  carried  out  which  will  give  concrete
form to the indicators, nor does he estimate their cost. It
should  also  be  stressed  that  it  is  difficult  to  quantify
indicators  concerning  the  programme’s  aims 
in  the
absence  of  information  on  historical  series.

Implementation  of  Objective  No  2  programmes
(1994-1996)

6.27. The  results  not  only  of  the  ongoing  evaluation  of
SPDs,  CSFs  and  OPs  during  the  1994-1996  period  but
also of the monitoring procedure highlight the imbalance
between  the  financing  originally  envisaged  and  the  funds
actually  used  to  implement  the  programmes.  Despite
numerous  adaptations  of  the  programmes  during  the  last
programming  phase  (1994-1996),  in  order  to  absorb  the
financial  aid,  appropriations  were  transferred  from  the
1994-1996 programming period to the 1997-1999 one at
an average rate of 12 %. The risk of this situation is that
it may lead to an overriding emphasis being placed on the

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/17

need to use up appropriations and an underestimation of
the role of evaluation, in particular as regards consistency
between objectives, resources allocated and the impact of
structural  measures.

7.  IMPACT  OF  THE  STRUCTURAL  FUNDS  AND

EVALUATION  MODELS

Impact  of  CSFs  and  programmes

Results  of  the  models  as  regards  the  impact  of  the
Structural  Funds

7.1. Given  their  specific  characteristics,  the  models  used
for  macroeconomic  evaluation 
in  the  field  of  the
Structural  Funds  (see  Annex  III)  make  differentiated
assessments  of  the  impact  of  the  Structural  Funds  as
regards:

(a)  the implementation of the Structural Funds’ objectives
in  terms  of  GDP  growth,  actual  convergence  of  per
capita  GDP,  employment  and  unemployment;

(b) observance  of  the  conditions  for  implementing  the
Structural  Funds,  such  as  additionality  and  durable
development  with  particular  reference  to  controlling
public  debt  and  the  balance  of  payments,  to  income
distribution  and  to  respect  for  the  environment;

(c)  the  effects  in  terms  of  supply  and  demand;

(d) the  effectiveness  of  Structural  Fund  instruments,  such
as  public  investment,  investment  in  human  resources
and  aid  for  private  companies.

GDP  growth

7.2. As  Table  7  shows,  the  results  of  the  principal
models  used  to  evaluate  the  impact  of  the  Structural
Funds  (HERMIN,  OGM  and  DIO)  indicate  a  positive
increase  in  GDP  for  Cohesion  Fund  countries  and
consequently  greater  convergence.  Thus,  cumulative
growth  in  the  economies  of  the  Cohesion  Fund  countries
from  1994  to  1999  as  a  result  of  structural  measures
under  the  CSFs  appears,  in  terms  of  real  GDP,  to  be
5,5 %  for  Portugal,  5,9 %  for  Greece,  2,6 %  for  Spain
and  3,3 %  for  Ireland.  These  results  assume  that  the
principle of additionality is fully observed; they may thus
be  modified  where  the  principle  is  not  observed  (see
paragraphs  8.11  to  8.13).  Furthermore,  the  various
macroeconomic  models  produced  sufficiently  consistent
results  for  the  impact  of  the  1994-1999  CSFs  on  real
to  be  considered  plausible.
annual  GDP  growth 
Nevertheless,  demand-related  effects  continue  to  be
predominant  in  the  short  term,  as  the  effects  on  supply
become  apparent  only  much  later.  Similarly,  the  HELM
regional model used for the Hainault region evaluates the

impact  of  the  SPD  in  terms  of  annual  growth  in  total
added  value  as  accounting  for  an  increase  of  1,7 %  to
2,5 %  over  the  1994-2002  period.  By  contrast,  in  the
case  of  the  Italian  Mezzogiorno,  the  impact  of  the
Structural  Funds  on  growth  seems  very  limited,  or  even
close  to  zero.  The  Commission’s  QUEST  model,  by
introducing  hypotheses  on  the  expectations  of  businesses
and  consumers,  produces  lower  estimates  of  the  effects
on  growth  and  employment  than  was  the  case  with  the
previous  models.  The  gains  shown  by  this  model  with
regard  to  the  potential  for  growth  and  the  reduction  of
unemployment  only  become  apparent  in  the  longer
term.

7.3. As  regards  the  average  acceleration  in  annual
growth  due  to  structural  measures  implemented  as  part
of  CSFs,  Table  8  shows  that  1 %  of  growth  in  GDP
generated by the multiplier effects of the CSFs leads, over
the  1994-1999  period,  to  additional  annual  growth  of
0,2 %  to  0,3 %  for  demand  and  of  no  more  than  0,2 %
for  supply.  Economically  speaking,  the  predominance,  as
highlighted  by  the  HERMIN  model,  of  the  effects  on
demand,  which  would  disappear  if  Community  aid  were
discontinued,  contributes  to  achieving  the  objectives  of
Article  130A  of  the  Treaty  only  if  the  effects  on  supply
are  lasting,  thereby  increasing  the  productive  capacity  of
the  regions  concerned.  Insofar  as  the  effects  on  supply
become  apparent  much  later  than  those  on  demand,  the
corresponding  multiplier  effects  may  be  underestimated.

Real  convergence  in  terms  of  per  capita  GDP

7.4. The  models  used  within  the  framework  of  the
Structural  Funds  show  that  convergence  is  not  always
guaranteed and that some CSF programming may at best
reduce  divergence  rather  than  develop  convergence.

7.5.
In  the  case  of  Ireland,  Bradley  (HERMIN  1994)
and  Honohan  (HERMIN  1997)  envisage  the  possible
effect  of  aid  under  the  1989-1993  and  1994-1999  CSFs
on per capita GDP. Bradley considers that the 1994-1999
CSF’s  contribution  to  bringing  the  Irish  economy  closer
to the EU average is modest whereas, in Honohan’s view,
the  CSFs  play  a  significant  role  in  the  convergence
process.  Beutel  (1997)  predicts  that  Ireland  will  catch  up
with the rest of Europe even without the 1994-1999 CSF,
which  would  increase  the  annual  growth  rate  by  0,6 %.
Similarly, the annual growth rates of Spain, Portugal and
Greece  would  increase  for  the  1994-1999  period  by

C  347/18

EN

Official  Journal  of  the  European  Communities

16.11.98

0,4 %,  1,1 %  and  1 %  respectively  due  to  the  Structural
Funds,  which,  in  any  case,  has  had  a  positive  effect  on
convergence  in  terms  of  per  capita  GDP.

7.6. Forecasting  simulations  based  on  the  OGM  model
of  convergence  in  real  per  capita  GDP  in  relation  to  the
EU  average  provide  estimates  that  stress  the  fact  that,
even if the trends projected in 1995 are too pessimistic in
the  case  of  Ireland  (see  Table  9),  the  CSFs’  net
contribution  to  convergence  remains  modest  for  the
Cohesion  Fund  countries.

7.7.
In  the  case  of  Hainault,  the  HELM  model  takes
aspects  of  real  convergence  into  consideration  for  the
purposes  of  ongoing  evaluation  and  considers  that  the
province’s  estimated  annual  growth  rate  would  be
approximately 1,4 %  to  1,6 %  higher than the European
average  because  of  the  Structural  Funds;  these  figures
would  indeed  make  it  possible  for  Hainault  to  catch  up
with  other  regions.

7.8. These  estimates  highlight  the  fact  that,  given  the
differences  between  Objective  No  1  Member  States’
regional  or  sectoral  productive  structures,  Structural
Fund  aid  produces  variable  effects.  They  also  confirm
that  Structural  Fund  programming  must  be  improved  in
order  for  the  corresponding  assistance  to  improve  the
economy’s  productive 
to
convergence.

capacity  with  a  view 

Employment  and  unemployment

7.9. Table  10  summarises  the  findings  of  evaluations  of
the  impact  of  the  Structural  Funds  on  employment  and
unemployment.  All  of  the  studies  conclude  that  the  CSFs
generate net jobs. These new jobs are often created in the
non-traded  services  sector,  whereas  the  productivity
effects  of  the  CSFs  are  mainly  concentrated  in  the
industrial sector. Lastly, some studies note a reduction in
the  number  of  industrial  jobs  due  to  an  increase  in
productivity.

Effectiveness  of  CSF  instruments

7.10. The  HERMIN  evaluations  for  Spain,  Ireland  and
Portugal  show  that  aid  for  public  investment,  especially
in  infrastructure,  is  more  productive  in  terms  of  growth
than  aid  for  investment  in  human  resources  (except  for
Spain) and for private investment. This trend is illustrated
in  Table  11  which  shows,  for  each  instrument,  the
increase  in  GDP  due  to  the  1 %  increase  in  total  GDP
generated  by  the  CSF  over  the  1994-1999  period.  In  the
case  of  Greece,  the  HERMIN  study  also  concludes  that
investment  in  ‘hard  infrastructure’  covered  by  the  CSFs
contributes  more  to  GDP  growth  than  investment  in

human  resources.  The  use  of  the  OGM  model  (Pereira
1991)  to  evaluate  the  effects  of  replacing  aid  for  private
investment  or  for  investment  in  human  resources  by
additional  aid  for  investment  in  infrastructure  shows  an
improvement  in  GDP  growth  and  in  the  convergence
process  as  a  result  of  the  Structural  Funds.

Additionality  and  joint-financing  requirements

the  public 

that,  without 

financing.  Thus,  Honahan 

7.11. The  aforementioned  models  also  made  it  possible
to evaluate the significance of additionality and of public
(HERMIN,  1997)
joint 
considers 
joint-financing
requirement, the growth rate of the Irish economy would
have  been  1 %  lower  and  employment  would  have
experienced  a  smaller  increase.  By  contrast,  Honahan
envisages  a  more  appreciable  reduction  in  public  debt  as
a  proportion  of  GDP  and  in  the  balance  of  payments
surplus  if  the  public  joint  financing  requirement  did  not
apply.  In  the  case  of  Portugal,  the  joint  financing
requirement leads, according to the HERMIN evaluation,
to a total additional gain of 1,3 %  in  terms  of  GDP,  but
the  development  is  less  positive  in  terms  of  public  debt.

7.12. The HELM evaluation for Hainault attaches great
importance to additionality. According to the evaluation,
the  average  rates  of  growth  in  value  added  for  the
1994-1999 period based on the three envisaged scenarios
(regional  policy  including  the  SPD,  the  continuation  of
regional  policy  without  Community  aid  and  regional
policy from which the SPD has been deducted) are 2,5 %,
1,7 %  and  0,1 %  respectively,  and  the  growth  rate  in
terms  of  employment  would  fall  from  0,8 %  to  0,2 %.
The 
also  deteriorate
situation  would 
considerably. The HELM evaluation also concluded that,
if  the  rate  of  subsidy  varies,  aid  does  not  stimulate
investment  to  the  same  extent  and  that  considerable
investment  would  take  place  if  levels  of  aid  were  lower.

industrial 

Durable  development

7.13. Table  10  shows  that  the  effects  of  the  Structural
Funds  on  the  balance  of  trade,  the  current-account
balance  or  public  debt  are  not  always  evaluated.
Moreover, the various studies reach different conclusions.
Thus,  the  economic  expansion  generated  by  the  CSFs
leads,  in  certain  cases,  to  the  improvement  of  public
financing  through  taxation  whereas,  according  to  other
models,  the  CSFs  create  budget  deficits.

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/19

Role  of  macroeconomic  evaluation  in  the  Structural
Funds’  decision-making  process

the 

tend 

effects  on 

7.14. Given  the  short-term  effects  of  the  Structural
Funds  on  demand,  ongoing  evaluations 
to
over-estimate  the  difficulties  experienced  by  the  political
in  using  up  Community  funds  and  to
authorities 
supply.
longer-term 
underestimate 
Reorientation or reprogramming of the OPs or CSFs may
also  be  biased.  The  fact  is  that  measures  or  projects
which  do  not  use  up  Community  appropriations  quickly
enough  may  be  discontinued,  especially  in  the  fields  of
innovation  or  new  technology  where  implementation  is
complex, in favour of existing structures which are much
less 
the
macroeconomic  aspects  of  sustainable  development  are
not  always  considered.

productive.  However, 

cases 

such 

in 

In 

this 

7.15.
context,  additional  macroeconomic
evaluations  which  take  account  of  sustainability,  growth
or  convergence  are  essential  before  any  decision  is  taken
regarding initial programming and any necessary changes.
However,  on  several  occasions  in  Spain,  Greece  and  the
new  German  Länder,  measures  were  rescheduled  on  the
basis  of  ongoing  evaluation  without  serious  overall
macroeconomic  analysis.  With  regard  to  Objective  1
regions  in  Germany,  no  macroeconomic  analysis  of  the
impact  of  the  measures  was  performed  when  the
intermediate  evaluation  was  made.

above  evaluate  the  effectiveness  of  the  CSFs’  principal
instruments  (public  investment,  private  investment  and
investment  in  human  resources)  only  in  terms  of  their
respective  contributions  to  GDP  growth.  Expenditure  on
infrastructure  seems  to  increase  the  yield  of  the  private
capital which it complements, and would appear to speed
up convergence more than other forms of investment (see
Table  11).

the 

terms  of 

7.19. Nevertheless, the choice and financing of measures
or  sub-programmes  which  make  up  the  programme  are
not  neutral 
impact  of  regional
in 
development  on 
lasting  endogenous  growth.  The
evaluations  performed  by  the  Commission  or  by  the
Member  States  in  the  context  of  their  partnership  as
regards sub-programmes or measures are also insufficient
to  make  it  possible  to  examine  the  consequences  of  a
specific  measure  on  other  areas  or  sectors  or  to  compare
similar  objectives.
it  with  other  measures  with 
Furthermore,  analysis  of  the  impact  of  the  measures  or
sub-programmes  must  also  take  into  consideration  the
initial  attempts  to  offset  growth  against  inter-regional
equality  which  determined  the  choices  made,  so  as  to
compare  expected  gains  in  efficiency  with  actual  results.

Impact  of  sub-programmes,  measures  and  projects

7.16. Very few evaluations are made of sub-programmes
and  measures,  even  though  they  act  as  the  driving  force
of Structural Fund programming. The fragmentation, lack
of  definition  and  inconsistency  of  many  sub-programmes
and  measures,  and  the  inadequate  quantification  of  their
specific  objectives,  may  help  to  explain  the  difficulty  of
measuring  their  impact.

7.17. This  shortcoming  complicates  any  attempt  to
identify  and  measure  synergies  and  the  degree  of
complementarity between individual measures or between
one  programme  and  another.  Furthermore,  the  links
between  a  programme’s  general  objectives  and  the
specific  objectives  of  sub-programmes  or  measures  are
very  often  unclear.  Thus,  in  several  cases,  problems  of
consistency  in  the  Objective  No  1  1994-1999  CSF
programmes  were  noted 
in  the  ongoing  evaluation
reports.

7.20.
In  principle,  the  projects’  relative  costs  and
advantages  are  analysed  with  a  view  to  approving
financing.  Pursuant  to  Article  5  of  Regulation  (EEC)  No
2083/93  (ERDF),  this  analysis  is  compulsory  in  the  case
of major projects the total cost of which is more than 25
Mio  ECU  for  investment  in  infrastructure  and  15  Mio
ECU  for  productive  investment.  These  prior  assessments
of  major  projects  are  part  of  the  examination  of
applications  for  Community  aid  and  determine  the
amount  of  financing.  However,  ex  post  evaluation  of
them  is  not  provided  for  in  the  SF  Regulations  and  may
not  be  required  by  the  Commission  for  the  purpose  of
establishing  whether 
the  socio-economic  advantages
the  Community  resources
actually  obtained  reflect 
originally  provided.  This  deficiency  in  the  process  for
evaluating  major  projects  renders  the  very  concept  of
evaluation  in  this  area  ineffectual,  as  a  prior  assessment
can  be  justified  only  if  an  ex  post  evaluation  is  actually
made.  Furthermore,  these  microeconomic  and  sectoral
studies  could  also  consider  the  potential  impact  of  the
Structural  Funds  on  productivity  and  thus  become  an
effective instrument for analysing this impact in the short
or  medium  term.

7.18. Furthermore,  techniques  are  seldom  fashioned  to
measure  the  impact  of  a  sub-programme  or  specific
measure.  Thus,  the  macroeconomic  models  described

7.21. Synergies and complementarity must also be taken
into  consideration  when  projects  are  evaluated.  Current
evaluations  in  this  connection  are  still  inadequate  or

C  347/20

EN

Official  Journal  of  the  European  Communities

16.11.98

superficial.  The  harmonisation  of 
synergies  and
complementarity between projects is essential so as not to
weaken  their  desired  impact.

8.4. The  same  is  true  of  statistical  bases,  the  weakness
of which casts doubt on the validity of the indicators and
has  a  major  influence  on  the  monitoring  and  evaluation
process.

evaluations  often 

7.22. Although  ongoing 
tackle
problems  relating  to  programme  management,  they  do
not  often  take  account  of  the  criteria  which  determined
the  selection  of  projects,  such  as  their  sustainability  or
their contribution to regional development. However, the
programme  management  system  must  be  able 
to
guarantee  that  the  criteria  for  selecting  projects  are
applied  in  such  a  way  as  to  ensure  that  the  programme
produces  the  best  possible  results.  In  particular,  projects
providing very little added value in terms of development
should  be  avoided  even  if  the  fact  that  they  consume
appropriations  very  quickly  is  an  advantage.  The  project
selection  system  must  be  more  transparent  for  the
purposes  of  inspection  and  evaluation.

8.5. The  current  plethora  of 
indicators  of  varying
relevance  is  detrimental  to  the  quality  of  evaluation  and
prevents  it  from  becoming  operational.  A  small  number
of  impact  indicators  which  are  essential  for  justifying  or
reorientating  a  structural  policy  should  be  carefully
selected  bearing  in  mind  the  constraints  imposed  by  the
regionalisation  of  data  and  the  material  possibilities  for
gathering  information.  The  reliability  of  these  indicators
should  be  checked.  The  comparability  of  indicators  and
their  aggregation  on  a  European  scale  should  be
improved.

8.  CONCLUSION

8.6.
In  order  for  full  account  to  be  taken  of  evaluations
in  the  decision-making  process  and  in  the  operational
management  of  the  Structural  Funds,  steps  should  be
taken  to  ensure  that  the  evaluations  are  transparent  and
objective, 
to
requirements  and  that  the  finalisation  date  is  adequate.

correspond 

objectives 

their 

that 

8.1. The  evaluation  of  Community  structural  measures
is  a  legal  requirement  stemming  from  Council  Decisions
and  is  currently  extensively  implemented  in  partnership
with the Member States. Evaluation has become a system
of  aid  for  the  design  and  management  of  intervention
measures, and contributes to improving understanding of
their  effects  and  usefulness.  For  the  1994-1999  period,
more  than  250  mid-term  evaluations  were  completed  by
Monitoring  Committees (19)  in  1997.

8.7.
In  order  for  evaluation  findings  to  be  objective,
assessors  must  remain  independent  of  the  managers  and
public  authorities  concerned.

8.8. The  use  of  a  common  frame  of  reference  would
make it easier to read and compare the evaluation reports
and  to  share  practical  experience.

8.2. With  the  revision  of  the  reform  of  the  Structural
Funds,  evaluation  is  entering  a  phase  of  maturity  which
should  be  consolidated  without  delay  by  improving  the
methodology.  Much  work  still  needs  to  be  done  to
determine  the  most  appropriate  evaluation  methods  for
measuring the  socio-economic impact  of  the programmes
within  the  framework  of  regional  development,  as  the
MEANS  programme  has  been  unable  to  provide  a
satisfactory  solution.

8.3. Particular  attention  should  also  be  paid 
to
indicators  at  all  levels  as  regards  their  relevance,  the
scrutiny  of  information  and  the  attendant  costs.  In
addition,  qualititative  aspects  should  be  formalised.
Without  further  progress  in  these  areas,  the  evaluation
process  may  very  well  stagnate.

(19) The  Monitoring  Committees  include  representatives  of  the
national  authorities  concerned  and  of  the  Commission.

indicate 

8.9. For  each  evaluation,  more  operational  objectives
should  be  defined.  The  terms  of  reference  and  the
technical  specifications,  which  are  generally  approved  by
the  Commission,  should  clearly 
the  most
appropriate  methodology  for  dealing  with  regional,
sectoral  or  national  aspects  or  for  specifically  covering
short- or long-term effects. The strengths and weaknesses
of  any  other  methodology  should  be  compared  with  the
Commission’s  methods.  Similarly,  the  frame  of  reference
for  each  evaluation  should  include  a  minimum  list  of
indicators  proposed  by  the  Commission  for  measuring
the  performance  of  the  programmes’  objectives  so  as  to
guarantee  a  degree  of  consistency  which  is  essential  for
the  purposes  of  analysis.  Other  socio-economic  and
to
structural 
complement  and  refine  this  analysis.

indicators  may,  of  course,  be  used 

8.10. Given  the  diversity  of  approaches  to  evaluation,
the  Commission  must  require  assessors  to  explain  their
approach much more explicitly when calls for tenders are
being issued, in particular as regards the type of effect or

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/21

impact which is supposed to be measured and the period
concerned, with a view to assessing the exhaustiveness of
the methods used. In addition, evaluations should contain
a  detailed  analysis  of  the  links  between  the  programmes’
objectives and the priorities and measures implemented to
achieve them, and of the corresponding inter-relationship
between  the  objectives  and  results  of  the  programmes,
priorities  and  measures.

8.15. Macroeconomic  models  make  a 
fundamental
distinction  between  the  effects  on  demand  in  the  shorter
term  and  structural  adjustments  in  terms  of  supply.  The
programming  of  Structural  Fund  assistance  should  place
greater emphasis on improving the productive capacity of
the economy so as to ensure that the effects of assistance
are  permanent  and  to  avoid  excessive  dependency  on
demand,  which  would 
to  a  situation  where
lead 
Community  aid  would  remain  essential  in  the  long
term.

8.11.
In  this  context,  the  sub-division  of  evaluations
between the beneficiaries concerned needs to be improved
increase  efficiency  and  to  meet  specific
so  as  to 
expectations, 
in  the  area  of  ongoing
evaluations.  Similarly,  greater  control  over  costs  should
be sought, as the Commission is not always aware of the
volume  of  expenditure  allocated  to  evaluation  within  the
partnership  framework.

in  particular 

8.16. As  the  effects  on  supply  become  apparent  only
over  a  longer  period,  ongoing  evaluations  generally  pay
more  attention 
the  effects  on  demand.  The
Commission  should  aim  to  develop  the  instruments
needed  to  monitor  and  evaluate  effects  on  the  structure
of  the  economy.

to 

indicate  which
8.12. The  Commission  should  also 
particular  aspects  of  regional  policy  are  to  be  evaluated:
additionality,  convergence,  partnership,  concentration,
employment,  endogenous  economic  development  or
social  welfare.  The  per  capita  GDP  should  not  be  the
only  criterion  for  judging  the  effectiveness  of  structural
measures.

8.17. Macroeconomic models also show that a policy of
economic  growth  is  not  always  compatible  with  durable
development.  The  relationship  between  the  impact  on
growth and the effects on durable development should be
analysed  in  greater  detail  and  systematic  account  should
be taken of this duality in the programming of Structural
Fund  assistance.

8.13.
It  should  be  possible  to  base  estimates  of  the
programme’s effects, impact and Community added value
on  national  and  regional  development  models  and  on
models  of  the  impact  of  the  individual  measures.

8.18. The  effects  of 
the  additionality  and  public
joint-financing  requirement  are  not  easy  to  discern  and
are  not  always  examined.  More  systematic  and  detailed
analyses  should  be  carried  out  in  this  area.

8.14. The  macroeconomic  results  of  structural  measures
should  be  refined  by  taking  account  not  only  of  the
imperfections  of  the  underlying  models  but  also  of  the
side-effects of the measures and their cumulative effect on
social welfare.

8.19. Evaluation  per  se  does  not  profess  to  replace  a
political  decision-making  process  which  involves  broader
considerations,  but  aims  to  provide  an  essential  tool  for
the  sound  and  effective  management  of  Community
structural  measures  which 
the
corresponding  socio-economic  results.

takes  account  of 

This  report  was  adopted  by  the  Court  of  Auditors  in  Luxembourg  at  its  meeting  of  23
September  1998.

For  the  Court  of  Auditors

Bernhard  FRIEDMANN

President

Cost  of  evaluation  funded  directly  by  the  Commission  (DG  XVI)  for  the  first  CSF

Table  1

Country

Objective  1

1989-1993

Objective  2

Community  initiatives

Thematic

Others

Total

Prior appraisal

Ex  post

Prior appraisal

Ex  post

Prior appraisal

Ex  post

(in  ECU)

Belgium
Denmark
Germany
Greece
Spain
France
Ireland
Italy
Luxembourg
Netherlands
Portugal
United  Kingdom
EC

35 000
31 000
42 000
35 000
35 000

35 000
53 000

150 172
207 372
97 662
109 572
150 552

193 022
72 572

17 500
35 000
17 500

70 000
35 000

35 000
17 500
35 000

122 500

69 040
25 520
138 570

117 090
253 185

96 580
15 880
47 925

131 500
14 815

65 963
10 514
143 480
526 347
739 274
197 228
201 915
395 955
15 128
46 816
452 706
223 428
98 400

35 395
25 000

593 350
260 977
104 371
464 064
390 156

25 000
625 169
626 694
77 500

151 845
30 000
151 845

151 845

35 000

550 256

187 898
96 034
299 550
1 456 714
1 455 713
881 291
810 551
1 255 088
48 508
154 741
1 340 897
1 229 694
740 971

Total

266 000

980 924

385 000

910 105

3 117 154

3 227 676

1 070 791

9 957 650

Source:  the  European  Commission.

C

3
4
7
2
2

/

E
N

O

f
f
i
c
i
a
l

J
o
u
r
n
a
l

o
f

t
h
e

E
u
r
o
p
e
a
n

C
o
m
m
u
n
i
t
i
e
s

1
6
.
1
1
.
9
8

 
 
 
 
 
 
Cost  of  evaluation  directly  financed  by  the  Commission  (DG  XVI)  for  the  second  CSF  up  until  1  April  1998

Table  2

Country

1994-1999

Objective  1

1994-1996

1997-1999

1994-1999

Objective  2

Objective  2

Cls

Thematic

Cohesion  Studies

Prior appraisal

Ongoing

Prior appraisal

Ongoing

Prior appraisal

Prior appraisal

(En  ECU)

Methodological
development  and
others

Total

Belgium
Denmark
Germany
Greece
Spain
France
Ireland
Italy
Luxembourg
Netherlands
Austria
Portugal
Finland
Sweden
United  Kingdom
EC

24 000

352 527

38 940
108 303
68 995
86 841
56 000
83 809

19 000
17 671
60 000

71 774
49 150

1 030 037
4 171 941
1 382 117
481 092
1 067 557
1 514 311

37 499
11 827
1 631 604
339 214
281 295
70 885

26 530
23 312
94 784

85 102
186 122

105 287
9 143
46 431
33 000

16 750
27 000
154 591
127 300

611 348
9 278
128 450

209 078
807 462

722 690
61 741
104 924
171 201

326 614
247 636

10 600
9 600
29 200

19 000
45 600

31 169
10 600
10 600

30 000

1 559

39 702
23 286
29 725
36 091
4 326
33 185
460
4 406
245
35 247

35 845
4 314

127 986

2 576 126

19 417
33 357
62 217
55 167
48 367
56 967
52 317
49 417
14 597
46 817

38 417

52 122
90 000

4 030 438

1 045 981
75 547
1 551 316
4 358 697
1 842 384
1 700 175
1 180 200
2 539 868
96 541
269 677
233 944
1 765 268
682 578
555 931
415 217
6 877 328

Total

684 483

12 371 906

935 352

3 400 422

196 369

248 391

2 704 112

619 179

4 030 438

25 190 652

Source:  the  European  Commission.

.

1
6
1
1
9
8

.

E
N

O

f
f
i
c
i
a
l

J
o
u
r
n
a
l

o
f

t
h
e

E
u
r
o
p
e
a
n

C
o
m
m
u
n
i
t
i
e
s

C

3
4
7
/
2
3

 
 
 
 
 
 
C  347/24

EN

Official  Journal  of  the  European  Communities

16.11.98

Table 3

Control, Monitoring and Evaluation

Means

RR

Achievements

RR

Impact

Input

Output

Outcome

rr Control RR

rr Monitoring (of management) RR

rr

Evaluation (of effects)

RR

Source: MEANS

Table 4

Judging the value of a programme

Points of Reference

Compliance with
administrative texts

Standards of sound
management

Public satisfaction

Impact

qq

QQ QQ

Society

Needs
What is at stake

QQ QQ

Results

qq

QQ

QQ

RR

Means
Resources

RR

Interventions’
achievements

QQ

Specific
overall
objectives

Relevance

Efficiency

Effectiveness

Usefulness

Public action

Evaluation

Source: MEANS

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/25

Table  5

Number  of  ongoing  assessments  by  country  and  objective

Country

Objective  1

Objective  2

Objective  3

Objective  4

Objective 5a

Objective  5b

Objective  6

Total

Belgium
Denmark
Germany
Greece
Spain
France
Ireland
Italy
Luxembourg
Netherlands
Austria
Portugal
Finland
Sweden
United  Kingdom

Total

1

8
28
27
6
11
24

1
1
25

12

144

4
2
9

9
27

11
1
5
4

1
5
13

91

5
1
13

12
1

1
1
1
1

4
1
1

5
1
1

2
1

1
1
1
1

4
1

42

19

1
1
1

1

1

1
1
1

8

3
1
8

7
20

13
1
5
7

2
5
11

83

19
6
40
28
58
55
11
50
4
13
15
25
13
14
38

389

1
1

2

At CSF level, five ongoing assessments were undertaken — Spain, Greece, Ireland, Italy and Portugal — and a summary of the evaluation work was drawn
up  for  Objective  1  measures  in  Germany.

Source:  the  European  Commission.

Table  6

State  of  progress  of  mid-term  evaluations  of  Objectives  1,  2  and  6

Member  State

Monitoring  Committees

Calls  for  tenders

Body  selected

Number  of  reports  delivered  or  planned

Obj.  1

Cohesion  Fund  Country

Spain

Greece

Ireland

Portugal

1  CSF  Committee
12  regional
subcommittees
7  ERDF  sectoral
committees 
1 regional/central
(Cantabria)

finalised
12  finalised

7  finalised

1  finalised

1  external
12  external

7  external

1  external

1  report  planned  for  07/97
12  reports  planned  for  05/97;  1  report  delivered

7  reports  planned  for  05-06/97;  1  report  delivered

report  planned  for  05/97

1  CSF  committee
13  regional  committees
17  sectoral
subcommittees

selection  underway
13  finalised
7  finalised
3  published

6  offers  received
13  external
Nomination
underway

—
13  reports  delivered  during  06/97
—

1  CSF  committee
9  sectoral  committees

launched  in  1996

1  CSF  committee
7  regional  subcommittees
3  multi-regional
subcommittees
7  sectoral  subcommittees

finalised
finalised
finalised

finalised

1  external
9  external

1  external
7  external
3  external

14  external

1  mid-term  report  delivered  during  03.97
9  mid-term  reports  delivered  during  03.97

report  planned  for  the  end  of  97
7  reports  delivered  during  1st  quarter  of  97
3  reports  delivered  during  1st  quarter  of  97

12  reports  delivered  during  1st  quarter  of  97
—  1  report  planned  for  07/97
—  1  report  planned  for  07/97

Obj.  1

Other  countries

Germany

1  CSF  Committee
6  regional  subcommittees
3  ESF-EAGGF-FIFG
subcommittees

1997  (restricted)
1996
—

1  external
6  external
—

1  final  report  planned  for  08/97
6  reports  delivered  during  05/97
—

Belgium

1  SPD  committee

03/09/94

1  external

final  report  delivered  during  05/97

C

3
4
7
2
6

/

E
N

O

f
f
i
c
i
a
l

J
o
u
r
n
a
l

o
f

t
h
e

E
u
r
o
p
e
a
n

C
o
m
m
u
n
i
t
i
e
s

1
6
.
1
1
.
9
8

 
 
 
 
 
 
Member  State

Monitoring  Committees

Calls  for  tenders

Body  selected

Number  of  reports  delivered  or  planned

France

6  SPD  committees

6  finalised

6  external

Corsica:  mid-term  report  delivered  during  12/96
Hainault:  report  planned  for  07/97
Martinique/Guadeloupe:  reports  planned  for  06/97
Re´union/Guiana:  report  planned  for  the  end  of  97

Italy

1  CSF  Committee

8  regional  subcommittees
12  multi-regional
subcommittees

short  list  available,
selection  planned  for
11/97
1996
publication
underway,  selection
planned  for  summer
97

0

internal  evaluation:  report  planned  for  07/97
external  evaluation:  report  planned  for  06/98

2  out  of  8  external
0

report  planned  for  beginning  of  98
—

The  Netherlands

1  SPD  committee

03/96

1  external

report  planned  for  the  end  of  1997

United  Kingdom

2  SPD  committees

1:02/96  (Merseyside
2:  1996  (H&I)

1  external
1  external

1  report  delivered  at  the  beginning  of  97
1  report  planned  for  the  summer  of  97

Northern  Ierland

1:06/96
1  CSF  committee
9  sectoral  subcommittees May  96

1  external
9  external

report  delivered  in  May  97
9  reports  delivered  in  the  first  half  of  97

Austria

1  committee

finalised

1  external

report  planned  for  autumn  1997

Obj.  6

Finland

1  SPD  committee

finalised

1  external

report  planned  for  10/1997

Sweden

1  SPD  committee

finalised

1  external

report  delivered  during  06/97

Source:  European  Commission.

.

1
6
1
1
9
8

.

E
N

O

f
f
i
c
i
a
l

J
o
u
r
n
a
l

o
f

t
h
e

E
u
r
o
p
e
a
n

C
o
m
m
u
n
i
t
i
e
s

C

3
4
7
/
2
7

 
 
 
 
 
 
C  347/28

EN

Official  Journal  of  the  European  Communities

16.11.98

The  impact  of  the  second  CSF  programmes  (including  national  cofinancing)  on  GDP  growth

Table  7

CSF  1994-1999
as  a  %  of  GDP

HERMIN (1)
(Average  %  annual  growth  rates  for  1994-1999)

demand

supply

Total

OGM (2)
(Average  %  annual
growth  rates  for
1994-1999)

DIO (3)
(Average  %  annual
growth  rates  for
1994-1999)

Greece
Spain(4)
Portugal
Ireland

5,9
2,6
5,5
3,3

0,9
0,6
1,6
0,5

0,9
0,3
0,2
0,2

1,8
0,8
1,8
0,7

0,6
0,2
0,4
0,1

1
0,5
1
0,6

(1)  HERMIN:  Econometric  model  based  on  the  original  model  HERMES  (Harmonised  Econometric  Research  for  Modelling  Economic  Systems),  but  with
the  special  feature  of  being  minimum  size  (Min),  hence  (HER)MIN.  See  Chapter  7  and  in  particular  points  7.2,  7.5,  7.10,  7.11,  and  Annex  III
point  5.

(2)  Optimal  Growth  Model  (OGM)  by  Pereira  (1991  and  1995).  See  Chapter  7  and  in  particular  points  7.2,  7.6,  7.10,  and  Annex  III  point  7.
(3)  Dynamic  input-output  model  (DIO)  by  Beutel  (1995,  1996  et  1997).  See  point  6.23  d)  and  Annex  III  point  9.
(4)  Regarding  Spain,  not  all  regions  are  eligible  for  Objective  1.
Source:  Evaluations  based  on  the  macroeconomic  models  concerned,  cofinanced  by  the  Commission.

Annual  growth  multipliers  of  the  CSF  programme  in  1994-1999

Table  8

HERMIN

demand

supply

Total

OGM

DIO

Greece
Spain
Portugal
Ireland

0,2
0,2
0,3
0,2

0,2
0,1
0
0,1

0,4
0,3
0,3
0,3

0,1
0,1
0,1
0

0,2
0,2
0,2
0,2

Source:  Evaluations  based  on  the  macroeconomic  models  in  question,  cofinanced  by  the  Commission.

Real convergence in GDP per capita in the OGM approach

Table 9

9 0
90

8 0
80

7 0
70

60
6 0

50
5 0

40
4 0

e
g
a
r
e
v
a
U
E

f
o
%

Greece
G r e e c e
I r e l a n d
Ireland
P o r t u g a l
Portugal
S p a i n
Spain

30
3 0
1993 
1 9 9 3

1995 
1 9 9 5

1997 
1 9 9 7

1999 
1 9 9 9

2003 
2 0 0 3

2007
2 0 0 7

Note: With regard to these forward simulations, which were drawn up on the basis of relatively old data, only the trend should
be noted, and not the absolute values of the projections, which, particularly for Ireland, turned out to be too pessimistic. 

Source: Evaluations on the basis of the OGM model (Pereira, 1995)

 
 
 
16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/29

The  impact  of  CSF  programmes  on  employment  and  sustainability  in  the  1989-1999  period

Table  10

Authors

Employment  and  unemployment

Sustainability

Public  finances

Balance  of  payments

Bradley  et  al
[HERMIN]

Honohan  (ed.)
[HERMIN]

Herce  and
Sosvilla-Rivero
[HERMIN]

Christodoulakis
and Kalyvitis
[HERMIN]

Sema  Group
[HELM]

—  Total  employment  increases

—  Public  borrowing

Trade  balance  deteriorates

requirement  increases  at
the  start  of  the
programme  and
decreases  afterwards
when  supply  effects  are
present

—  Growth  effects  with

unchanged  public  sector
borrowing  requirement
are smaller

—  Employment  in  manufacturing  declines
partially  due  to  productivity  increases

—  Employment  in  the  non-traded  sector  increases

—  Total  employment  increases

—  See  previous  study

Balance  of  payments
improves  as  long  as  EU
funding  continues

—  Debt/GNP  ratio  declines

substantially

—  Total  employment  increases

—  See  study  by  Bradley  et

al

Trade  balance  deteriorates
due  to  the  demand  effects

—  Unemployment  declines

—  Debt/GNP  ratio  declines

—  Total  employment  increases

—  Primary  deficit  increases Not  considered

—  Urban  unemployment  falls

—  Total  employment  increases

Not  considered

Not  considered

—  Manufacturing  employment  declines

—  Employment  in  commercial  services  increases

Pereira  (1995)
[OGM]

Full  employment  model

Primary  public  deficit  as
a  %  of  GDP  increases

Current  account  deficit
increases  except  for  Spain  in
1997-1999

Lolos  and  Zonallas
[BBM]

Employment  increases  and  unemployment  declines

Fiscal  deficit  increases  from
1989-1993  and  declines  in
1994-1995

Current  account  deteriorates

Beutel  [DIO]

—  Employment  increases  in  Portugal,  Greece  and

Not  considered

Spain

Exports-imports  increase  in
Portugal  and  Greece  but
decrease  in  Ireland

—  In  Ireland  the  first  CSF  decreases  employment

while  the  second  CSF  creates  new  jobs

FIDIMS

Employment  increases  in  Northern  and  Southern
Italy

Not  considered

Not  considered

C  347/30

EN

Official  Journal  of  the  European  Communities

16.11.98

The accumulated % gain in GDP (GNP) generated by a one % point increase in the share of CSF in GDP
(1994-1999)

Table  11

Physical capital

Human capital

Private capital

Spain
Ireland
Portugal

2
1,4
1,9

3,3
1,2
1,3

1,5
1,1
Not available

Source:  HERMIN  evaluations.

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/31

ANNEX  I

Summary  of  the  methodology  to  be  applied  as  part  of  the  ex  post  evaluation  of  Objective  1  CSFs
(1989  to  1993)

1.  The  explicit  aim  of  the  ex  post  evaluation  of  the  Objective  1  CSFs  (1989  to  1993)  was  to:

(a)  assess the strategy and objectives selected in the CSFs and to draw the appropriate conclusions for
future  action.  To  analyse  the  consequences  for  the  initial  objectives  of  redefining  and  adjusting
interventions  during  the  implementation  period;

(b) take  provisional  stock  of  the  concrete  results  of  the  CSFs  and  OPs  as  regards  significant  measures

and  make  an  initial  assessment  of  their  impact;

(c)  analyse  the  effectiveness  of  the  implementation  mechanisms  and  draw  the  necessary  conclusions

with  a  view  to  making  future  adjustments.

2.  Methodological  approaches  were  prepared  by  the  principal  consultant,  who  was  responsible  for
coordinating  evaluations  made  by  other  consultants  on  behalf  of  the  Member  States  concerned:

(a)  for  the  first  objective,  in  order  to  evaluate  the  logic  intrinsic  to  the  CSFs  and  OPs  and  to  the
adjustments made to the CSFs and OPs since 1989 — and, more specifically, to analyse the causes
of  these  adjustments,  the  Community  added  value,  internal  consistency,  external  consistency,  the
relevance of the strategy to existing problems and the appropriateness of financing — the following
tasks,  based  on  a  top-down  approach,  were  to  be  carried  out:

—  collection  of  information  and  key  data:  financial  data,  ex  ante  objectives,  anticipated
output/effects,  target  groups  and  areas,  main  instruments  used,  number  of  projects/measures
approved  and  number  of  projects  completed,

—  description  of  the  environment  and  development  of  the  programme  (initial  situation  —current
situation);  identification  of  external  factors  influencing  the  development  of  the  programme  and
its  success/failure;  analysis  of  the  basic  problems  and  causes;  formulation  of  key  regional
development  indicators,  including  indicators  of  changes  in  disparities,

—  identification  of  policy  instruments  (loans,  subsidies  and  technical  assistance),

—  prioritisation of objectives, identification of levels of priority and of the links between objectives

and  instruments;

(b) for  the  second  objective,  the  analyses  to  be  performed  were  based  on  top-down  and  bottom-up

approaches:

—  analysis  of  the  output  and  effects  of  OPs,  measures  and  projects,  account  being  taken  of  ‘net

losses’,  and  substitution  and  multiplication  effects,

—  analysis  of  the  relative  effectiveness  of  measures  and  combinations  of  measures;  analysis  of  the

causes  of  successes/failures,

—  comparison  of  output  and  input,

the  aim  was  also  to  formulate  key  and  optional  indicators;

(c)  for  the  third  objective,  the  aim  was  to  analyse  the  effectiveness  of  CSF  and  OP  implementation
measures and of a small number of measures, and to identify existing and planned improvements in
the  planning,  implementation,  monitoring  and  evaluation  processes;  for  each  Member  State,  the
examination would cover decision-making, planning, financial control, implementation, monitoring
and  inspection  mechanisms,  financial  channels,  partnership,  additionality,  fund  coordination,  and
greater flexibility and transparency in the simplification and harmonisation process; implementation
and  monitoring  would  also  be  analysed  in  depth  for  a  small  sample  of  measures/major  projects;

(d) ad hoc samples of programmes, measures and projects were chosen with a view to substantiating a

small  number  of  physical  and  impact  indicators  and  to  carrying  out  in-depth  analyses.

C  347/32

EN

Official  Journal  of  the  European  Communities

16.11.98

ANNEX  II

Summary  of  the  ex  post  evaluation  approach  in  respect  of  Objective  2  (1989  to  1993)

The ex  post  evaluation  of  Community  assistance  under  Objective  2  for  the  1989  to  1993  planning  period,
which  generally  included  ERDF  and  ESF  components,  comprised:

(a)  general  comparisons  in  all  Objective  2  regions  and  areas  of  the  socio-economic  situation  prior  to
Objective  2  Structural  Fund  assistance  and  of  the  socio-economic  situation  after  assistance;  analysis  of
the  strategy  employed  in  terms  of  the  prioritisation  of  objectives,  priorities  and  measures,  and  of
indicators,  planned  and  unanticipated  effects,  synergies  between  programme  components  and  the
utilisation of the various funds. These analyses were to be based on the content of the CSFs and OPs, on
annual implementation reports or reports submitted to the Monitoring Committee and on prior and ex
post  evaluations, where available;

(b) detailed  studies  covering  22  regions,  the  aim  of  which  was  to:

—  study  in  depth  the  CSFs,  OPs  and  types  of  measures  adapted,

—  analyse  institutional  structures  and  programme  management  procedures,

—  evaluate  the  impact  and  added  value  effects  attributable  to  the  CSFs  and  to  the  OPs,

—  evaluate  the  cost  and  effectiveness  of  these  effects,

—  study  the  Objective  2  programme’s  contribution  to  the  regional  development  process.

These  in-depth  evaluations  employed  the  bottom-up  approach,  the  aim  being  —  using  an  examination  of
operations  covering  the  selection  process,  typology,  management  process,  the  impact  and  effect  in  terms  of
added  value,  and  synergies  and  additionality  —  to  work  upwards  from  one  level  to  the  next,  compiling
information  first  about  individual  measures  and  ultimately  about  the  whole  programme.

ANNEX  III

Characteristics  of  the  macroeconomic  models  used  to  evaluate  the  Structural  Funds

1.  The  principal  models  used  in  the  macroeconomic  evaluation  of  the  Structural  Funds  comprise:

(a)  econometric  models,  such  as  HERMIN (1)  and  HELM (2)  and  QUEST  II (3);

(b) calculable  general  equilibrium  models,  such  as  OGM (4)  and  BBM (5);

(c)  input-output  models,  such  as  DIO (6)  and  FIDIMI (7);

(d) structural-residual analysis.

(1)  HERMIN:  Econometric  model  based  on  the  original  HERMES  model  (Harmonised  Econometric
Research  for  Modelling  Economic  Systems),  but  whose  minimum  size  (Min)  distinguishes  it  from  this
latter,  whence  (HER)MIN.

(2)  HELM:  Hainaut  Economic  Lead-in  Model.
(3)  QUEST II: a Multi-Country Business Cycle and Growth Model — Economic Paper No 123 — October

1997.  DG  II  —  Economic  and  Financial  Affairs.

(4)  Optimal  Growth  Model  (OGM);  see  Pereira,  1991  and  1995.
(5)  Bourguignon,  Branson  and  de  Melo  model  (BBM);  see  Lolos  and  Zonzilas,  1992.
(6)  Dynamic  Input-output  model  (DIO);  see  Beutel,  1995,  1996  and  1997.
(7)  FIDIMI  Consulting  Company,  Evaluation  of  the  macroeconomic  impact  of  the  Objective  1  CSF  (Italy)

and  of  the  implementation  mechanisms.

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/33

The characteristics of the application of these models in the cases examined are shown in diagram form
in  Tables  III  1  and  III  2.  Table  III  3  reproduces  a  model  of  the  characteristics  of  the  CSFs.

2.  Furthermore,  by  simulating  the  evolution  of  the  economies  of  Objective  1  countries  over  a  period  of
more than 10 years and on multi-sectoral bases, it is possible to perceive the structural change brought
about  by  the  Structural  Funds  and  to  evaluate  the  reliability  of  the  catching-up  process  which
Community  assistance  is  supposed  to  support  and  consolidate.

3.  As  part  of  its  enquiry,  the  Court  had  an  analysis  performed (8)  to  assess  the  appropriateness  of  these
macroeconomic  techniques  for  measuring  the  impact  of  structural  measures  and  to  establish  whether
they  may  contribute  effectively  to  the  decision-making  and  strategic  aspects  of  programming.

4.  The analysis showed that explanations of the endogenous or exogenous factors upon which the success
of  the  structural  programmes  depends  vary  according  to  the  methods  and  conceptual  approaches
employed.  The  fact  is  that  macroeconomic  models  aim  to  describe  the  mechanisms  of  the  economic
system of a country or region and to capture or simulate the effects of an exogenous shock such as the
assistance provided by the structural programmes. However, no single description of the economic facts
is definitive.

5.  The HERMIN econometric model takes into account the specific characteristics of each Member State
and aggregates effects at a national level. It assumes that economies actually utilise their full production
capacity  and  that  the  accelerator  and  multiplier  effects  related  to  increases  in  investment  consequently
create  an  expansion  in  production  and  employment.  This  model  seems  to  give  satisfactory  results  in
terms of effects on demand. With regard to the effects on supply, taking the Phillips curve into account,
the  model  suggests  that  a  fall  in  unemployment  will  bring  about  an  increase  in  wages  whilst  limiting
the  effect  on  inflation.  Also,  the  effects  on  productivity  are  explicitly  taken  into  account  by  modelling
the  external  saving  effects  that  increases  and  improvements  in  human  capital  and  public  and  private
physical  capital  have  on  productivity.  They  are,  however  taken  into  account  differently  from  one
country  to  another.  The  model  provides  only  partial  information  on  durable  development,  and,  in
particular,  does  not  consider  variations  in  exchange  rates  aimed  at  guaranteeing  the  stability  of  the
balance  of  payments  or  variations  in  fiscal  transfers.

6.  The  HELM  macro-econometric  model,  which  was  used  for  the  ongoing  assessment  of  the  1994  to
1999  Objective  1  SPD  for  the  Province  of  Hainault  (Belgium),  is  one  of  the  few  to  take  the  regional
development approach specifically into account by considering the effects of endogenous growth linked
to  long-term  productivity,  although  improvements  in  productivity  due  to  SF  intervention  are  largely
limited  to  the  industrial  sector  and  only  concern  labour  productivity.  On  the  basis  of  a  similar
assumption to that of the HERMIN model with regard to the utilisation of productive capacity and the
multiplier and accelerator effects of investment on demand, it gives the overall and aggregate effects of
the SPD on the Hainault economy, such as overall investment generated, added value or employment in
the manufacturing industry and commercial service sectors for the period between 1994 and the years
2002  or  2005.  It  should  be  noted  that  the  HELM  model  does  take  regional  exports  into  account,  but
not imports from other regions, thus risking an overstatement of the effects on demand. The figure for
demand  is  all  the  higher  because  neither  wage  adjustments  related  to  the  fall  in  unemployment  nor
higher  costs  or  capacity  constraints  were  taken  into  account.  The  HELM  model  also  gives  little
information as to the trend of unemployment or the demand for labour. Lastly, it is based on historical
statistical  data,  but  these  were  not  cross-checked  against  any  other  series  of  relevant  data.

7.  The computable general equilibrium models apply more particularly at a national level. They show the
long-term effects of Community structural measures on supply. The OMG model, which uses a context
of  inter-temporal  endogenous  growth,  assumes  that  the  production  factor  markets  are  in  equilibrium
and  that  economies  are  working  at  full  production  capacity.  Changes  in  demand  have  no  effect  on

(8)  The  Court  has  had  the  services  of  an  external  assessor  who  was  highly  specialised  in  the  field  of

macroeconomic  and  European  economy  models.

C  347/34

EN

Official  Journal  of  the  European  Communities

16.11.98

production  and  employment  and  the  SF  programmes  can  only  stimulate  growth  by  increasing
productivity. These productivity effects are specified clearly in the production function. The objective of
the  model  is  to  determine  an  optimum  situation  for  the  country  by  maximising  a  function  of
inter-temporal  social  well-being.  On  the  basis  of  the  hypothesis  of  a  situation  without  uncertainty,
economic  agents,  anticipating  perfectly  the  amount  of  money  that  will  be  allocated  to  them  and  the
period  over  which  aid  will  be  spread,  are  supposed  to  take  the  best  possible  consumption  and
investment decisions in relation to these data. Lastly, the OMG model provides for long-term stability
of  the  balance  of  payments  and  budgetary  equilibrium.

8.  For  certain  sectors,  the  BBM  model  expects  that  labour  markets  will  be  in  equilibrium  and  that,
consequently,  demand  will  not  influence  production  and  employment,  while,  for  other  sectors,  an
expansion in demand is supposed to generate growth. This model also includes technological change —
related  productivity  effects  that  result  in  reductions  in  inputs.

9.  The DIO input-output model tries to show the links between all the resources available to an economy
and  all  final  consumption.  In  this  way  it  identifies  interdependencies  between  the  various  sectors  of  a
given  country,  linked  together  on  the  basis  of  intermediate  consumption.  The  input-output  approach
maximises  the  effects  of  structural  interventions  on  demand  and  ignores  those  on  supply  (effects  on
productivity  and  wages  and  inflationary  adjustments).  Durable  development  is  not  taken  into
consideration. The FIDIMI model is a combination of a general equilibrium model and an input-output
model  which  can  be  used  to  analyse  the  effects  of  Community  interventions  in  Southern  Italy  on  the
North  of  the  country.

10.  Among  other  existing  techniques  for  measuring  the  effects  of  regional  policies,  Eurostat  uses
structural/residual analysis to evaluate the impact of the SFs in certain areas eligible under Objectives 1
or  2.  This  analysis  attempts  to  identify  the  residue  resulting  from  the  difference  between  the  actual
situation and the counterfactual situation produced via the application of estimated national tendencies.
However, carried out in this way, the calculation of impact mixes the results of the programme under
evaluation,  other  national  and  European  political  measures  and  the  path  previously  followed  by
regional policy. Moreover, the studies carried out by Eurostat in 1997 include data for indicators, the
most  recent  of  which  date  back  to  1993.  This  considerably  reduces  the  value  of  their  conclusions.

11.  The  QUEST  II  model  attempts  to  analyse  the  economies  of  the  Member  States  and  their  interaction
with  the  rest  of  the  world,  in  particular  with  the  United  States  of  America  and  Japan.  The  model  is
concerned more particularly with the transmission of the effects of economic policy at the national and
international  levels.  It  was  initially  devised  to  serve  as  a  simulation  tool  rather  than  for  predictive
purposes.

The QUEST II model now pays more attention to the effects of demand than its predecessor and takes
into greater consideration interactions in terms of stocks. However, it gives little or no attention to the
implications  in  terms  of  changes  in  the  behaviour  of  economic  agents.

Table  III.1

Fundamental  characteristics  of  the  macroeconometric  models

Authors

Model

Country  or  region

Demand  effects

Sustainability

Wages

Productivity

Supply  effects

Bradley  et  al.  (1994) HERMIN

Ireland  and
Portugal

—  Standard  Keynesian

set-up  investment  demand
stimulates aggregate
demand

—  Unemployment  augmented
Phillips-curve  set-up  links
employment,  wages  and
inflation

—  CSF-related  investments
increase  sectoral  total
factor  productivity  in
Ireland

Balance  of  payments:

—  Demand  effect  increases

imports

—  Labour  markets  are  not

automatically in
equilibrium

—  Inflationary  impact  of
demand  expansion  is
limited

Honohan  (ed.)
(1997)

HERMIN

Ireland

—  Economy  does  not

—  Rising  wages  cause  labour

operate  at  full  capacity

inflow  in  Ireland

—  Monetary  sector  and

—  Investment  in  human

exchange  rates  are  not
modelled

capital  does  not  have  wage
effects

—  In  Portugal  no  productivity

—  Productivity  effect

effects  of  private
investment  and  certain
infrastructure  expenditures

improves  exports  and
attracts  foreign  direct
investment  in  Ireland

—  Modelling  of  productivity
effects  is  debatable  in
particular  for  human
capital

—  Exogenous  exchange  rate

does  not  allow  full
evaluation  of  balance  of
payments

—  Productivity  effect  may
lower  employment

Public  finance

Herce  and
Sosvilla-Rivero
(1994)

HERMIN

Spain

See  model  for  Ireland  but  no
labour migration

See  model  for  Ireland

See  model  for  Ireland  but:

—  Human  capital  investment

increases  labour
productivity  instead  of
total  factor  productivity

—  Aid  to  private  investment
decreases  the  user  cost  of
capital

—  Limited  modelling  of

public  sector

—  Public  sector  borrowing
increases  due  to  CSF

—  Simulation  with  unchanged
public  sector  borrowing  is
considered

See  model  for  Ireland  but  no
link  between  productivity
effect  and  foreign  direct
investment

.

1
6
1
1
9
8

.

E
N

O

f
f
i
c
i
a
l

J
o
u
r
n
a
l

o
f

t
h
e

E
u
r
o
p
e
a
n

C
o
m
m
u
n
i
t
i
e
s

C

3
4
7
/
3
5

 
 
 
 
 
 
Authors

Model

Country  or  region

Demand  effects

Sustainability

Wages

Productivity

Supply  effects

Christodoulakis  and
Kalyvitis
(1995)

HERMIN

Greece

—  See  model  for  Spain

See  models  for  Ireland

—  Exchange  rate  is

modelled

Sema  Group  (1997) HELM

Hainaut

—  Keynesian/Kaldor  set-up

No  wage  and  price  effects

—  Investment  demand
stimulates aggregate
demand

—  Labour  demand  is

modelled

—  Labour  supply  and

unemployment  are  not
modelled

—  Imports  from  other

regions  are  not  modelled

See  model  for  Spain  and
Ireland  but:

—  Human  capital  investments
decrease  unit  labour  cost

—  Aid  to  private  investment

See  model  for  Spain  but:

—  Nominal  exchange  rate
adjusts  to  maintain  a
constant  real  exchange
rate

does  not  have  productivity
effect

—  Fiscal  rule  is  assumed  to
stabilise  public  debt

—  Investment  in  ‘soft

infrastructure’  raises  public
employment

—  CSF-related  programmes
raise  primarily  labour
factor  productivity  in  the
manufacturing  sector

—  In  the  reporting  of  the
results  no  distinction
between  supply  and
demand  effects  is  made

No  evaluation  of
consequences  for  public  debt
or  balance  of  payments

C

3
4
7
3
6

/

E
N

O

f
f
i
c
i
a
l

J
o
u
r
n
a
l

o
f

t
h
e

E
u
r
o
p
e
a
n

C
o
m
m
u
n
i
t
i
e
s

1
6
.
1
1
.
9
8

 
 
 
 
 
 
Fundamental  characteristics  of  the  computable  general  equilibrium  and  input-output  model

Table  III.2

Authors

Model

Country  or  region

Demand  effects

Sustainability

Wages

Productivity

Supply  effects

Pereira  (1991)

Optimal  Growth
model  (OGM)

Portugal

—  No  Keynesian  demand  effects

—  Wages  adjust  to

—  All  factor  markets  are  in

equilibrium

clear  labour  market

Productivity  effects
because  public,  human
and  private  capital  are
in  production

—  Intertemporal  welfare

maximisation
guarantees dynamic
optimal  solution

Pereira  (1995)

Portugal,  Greece,  Ireland
and  Spain

—  The  economy  operates  at  full

capacity

Lolos  and  Zonzilas
(1992)

Bourguignon,
Branson  and  de
Melo  model
(BBM)

Greece

—  Keynesian  demand  effects

—  Wages  adjust  to

—  Labour  markets  clear  in  some
sectors  and  not  in  others

—  Fixed  and  flexible  exchange  rate

are  considered

clear  labour  markets
in  some  sectors

—  Unemployment
augmented
Phillips-curve  in  the
other  sectors

—  Balance  of  payments
and  public  finance
constraints  are  satisfied

—  No  constraint  imposed
on  public  borrowing

—  Measures  of  income

inequality are
considered

—  Investment  in  public
and  private  capital
increases sectoral
total  factor
productivity  in  a
debatable way

—  Investment  in
human capital
increases skilled
labour  input  and
public  employment

Beutel  (1995,  1996,
1997)

Dynamic
input-output
(DIO)

Greece,  Portugal  and
Spain

Keynesian  demand  effects

No  wage  effects

No  productivity  effects No  modelling  of  balance

of  payments  and  public
finance  constraints

.

1
6
1
1
9
8

.

E
N

O

f
f
i
c
i
a
l

J
o
u
r
n
a
l

o
f

t
h
e

E
u
r
o
p
e
a
n

C
o
m
m
u
n
i
t
i
e
s

C

3
4
7
/
3
7

 
 
 
 
 
 
Honohan  (ed:)  (1997)

Herce and Sosvilla-Rivero
(1994)
(HERMIN)

Christodoulakis  and
Kalyvitis
(HERMIN)

Sema  Group
(HELM)

Table  III.3

The  modelling  of  the  characteristics  of  CSF  programmes

Authors

CSF  programme

Sectoral  focus

Instruments

Co-financing  and  additionality

Starting  point  and  end  game

Bradley  et  al  (1994)

1994  to  1999 —  Four  sectors:  traded,

—  Physical  infrastructural  (public)

—  Some  studies  compare  full

non-traded,  public  and
agricultural sectors

—  Traded  sector  is  key

sector  for  supply  effects

—  Non  traded  and  public

sector  are  key  for  demand
effects

investments

—  Human  capital  investment

—  Investments  and  production  aids  to

the  private  sector

domestic  co-financing  with  no  or
partial  domestic  co-financing

—  Forecasts  for  1991  to
2020  except  Honohan
(1993  to  2010)

—  Hard  and  soft  landing

scenarios  are  considered

1994  to  1999

Same  as  previous  studies

—  Hard  infrastructure

—  Soft  infrastructure

—  Aid  to  productive  investment

—  Education  and  training

—  Simulations  assume  projected
co-financing  rates  are  realised

—  Forecasts  for  1991  to

2010

—  Hard  landing  scenario

1994  to  1999 —  Six  sectors,  agriculture,
manufacturing, energy,
construction,  commercial
services, non-commercial
services

—  Investments  in  public  and  private

—  Scenario  where  European  Union

—  Forecasts  for  1994  to

physical capital

—  Investments  in  human  capital

—  Investment  in  R&D  capital

and  regional  aid  of  1989  to  1993
is  continued  afterwards

2005

—  Scenario  where  additional

European  Union  aid  and  domestic
co-financing  for  1994  to  1999  is
added

—  Scenario  where  no  European

Union  aid  is  given  in  1989  to
1999

Lolos  and  Zonzilos
(BBM)

1989  to  1993

Nine  sectors:  agriculture,
energy  and  mining,  food
industry,  light  industry,
intermediate  industry,
construction,  trade  and
commerce,  transport  and
telecommunication services,
other  services  (including
Government)

—  Public  investments  in  selected

sectors

Additionality  is  not  discussed  but,
presumably,  imposed  on  the  model

—  Human  capital  investments

increase  public  employment,  the
number  of  skilled  workers  and
generate  transfers  to  households

—  Support  to  private  firms

—  Forecasts  for  1988  to

1995

—  Hard  landing

C

3
4
7
3
8

/

E
N

O

f
f
i
c
i
a
l

J
o
u
r
n
a
l

o
f

t
h
e

E
u
r
o
p
e
a
n

C
o
m
m
u
n
i
t
i
e
s

1
6
.
1
1
.
9
8

 
 
 
 
 
 
Authors

CSF  programme

Sectoral  focus

Instruments

Co-financing  and  additionality

Starting  point  and  end  game

Beutel  [DIO]

1989  to  1999

25  sectors  based  on
input-output  tables

FIDIMI  (1991)

1989  to  1993

16  sectors  based  on
input-output  tables

—  Sectorally  focused  modelling  of

Additionality  is  imposed  on  the  model —  Forecasts  for  1989  to

public,  private  and  human  capital
investments

—  Investments  in  training  affect

wages  and  salaries  in  the  economy

1994

—  Sectorally  focused  modelling  of

—  Additionality  is  a  key  part  of  the

—  Only  total  effects  are

CSF  programs

study

—  Scenarios  differ  in  the  amounts  of
funds  that  would  be  invested  in
Southern  Italy  with  and  without
CSF  programmes

reported  without  a  yearly
break-up

—  No  discussion  of  hard  and
soft  landing  scenarios

Pereira  (1991)
(OGM)

1989  to  1993

Two  sectors:  traded  and
non-traded

—  Investments  in  public  infrastructure

—  Co-financing  requirements  are

—  Forecasts  for  1990  to

—  Investments  in  human  capital

—  Support  to  private  investment

imposed

2006

—  Model  allows  an  evaluation  of

—  Soft  and  hard  landing

whether  additionality  will  be  met

scenario’s  are  considered

Pereira  (1995)
(OGM)

1994  to  1999
and  comparison
with  1989  to
1993

See  previous  model

See  previous  model

—  Co-financing  requirements  are

—  Forecasts  for  1993  to

imposed

2007

—  Simulations  compare,  cases  of:

—  Hard  landing  scenario

(i)  no-additionality

(ii) official  definition  and

(iii) intertemporal  definition  of

additionality

.

1
6
1
1
9
8

.

E
N

O

f
f
i
c
i
a
l

J
o
u
r
n
a
l

o
f

t
h
e

E
u
r
o
p
e
a
n

C
o
m
m
u
n
i
t
i
e
s

C

3
4
7
/
3
9

 
 
 
 
 
 
C  347/40

EN

Official  Journal  of  the  European  Communities

16.11.98

COMMISSION’S  REPLIES

I.  GENERAL  COMMENTS

The  Commission  accepts  that  the  evaluation  process,
being  something  entirely  new  for  many  parties,  is  still
rather  rough  and  ready  as  regards  its  scope  and  the  use
that  can  be  made  of  it,  but  insists  that  remarkable
progress  has  been  made  in  the  space  of  just  a  few  years.

Conscious  of  the  difficulties  that  still  have  to  be
overcome,  the  Commission  has  redoubled  its  efforts  on
the  organisational  and  methodological  front  to  achieve
the  most  satisfactory  situation  possible,  engaging  in  a
permanent  dialogue  with  all  those 
in  the  different
countries of the Union who are studying and carrying out
research  in  this  area.

There  are  three  main  phases  in  the  development  of  the
assessment  process  for  Structural  Fund  intervention,
corresponding  to  the  three  programming  periods:

1989  to  1993  period

The  regulations  on  the  first  reform  of  the  Structural
Funds  were  not  adopted  by  the  Council  until  December
1988  and  the  national  plans  submitted  at  the  beginning
of  1989  only  partly  reflected  the  requirements  for
strategic  vision,  choice  of  objectives  to  be  achieved  and
quantification  of  expected  results.

The  ex  post  evaluations  therefore  attempted  to  remedy
these  shortcomings  by  comparing  the  results  achieved
with  the  reworked  objectives.  This  considerable  task
suffered  from  its  own  limitations,  which  are  reflected  in
the  reports  dealing  with  the  interventions  in  Objective  1
and Objective 2 areas. Nevertheless, it represents the first
attempt  at  coherent  analysis  of  the  impact  of  this  reform
on  economic  and  social  cohesion.

1994  to  1999  period

At  the  request  of  certain  Member  States  and  of  the
Commission,  this  first  experiment  was  followed  by  a
tightening of the rules on monitoring and evaluation and
the identification of three stages: prior appraisal, ongoing
assessment  and  ex  post  evaluation.  There  can  be  no
doubt  about  the  success  of  the  ongoing  assessment.
Going  beyond  what  was  strictly  required  by 
the
regulations,  the  mid-term  assessments  were  organised  for
the  first  time  by  and  at  the  level  of  each  Monitoring

Committee,  thus  bringing  assessor  and  the  beneficiaries
of  the  assessment  closer  together.  The  reports  produced
were  presented  to  and  debated  in  each  committee,  with
the  result  that  the  process  served  an  educational  and
operational  purpose  that  studies  organised  and  carried
out  by  the  Commission  could  not  hope  to  fulfil.

There  may  have  been  some  discrepancies  between  the
methods  employed,  the  choice  of  assessors,  the  cost  of
the  work  and  the  level  of  detail  of  the  reports,  even
though  coordination  and  general  supervision  were
provided  by  the  Commission’s  evaluation  coordination
unit  and  its  national  contacts  in  the  technical  evaluation
group.  However,  despite  certain  differences  connected
with  the  discretion  allowed  to  each  Member  State  under
the  subsidiarity  principle,  the  Commission  believes  that
this  process  played  an  essential  role  in  propagating  a
genuine  climate  of  evaluation.  It  proved  to  beneficiaries
that  this  was  an  operational  tool  for  improving  the
management of interventions financed from both national
and  Community  sources.  Future  evaluations  can  only
benefit  from  this  attitude  in  that  the  quality  of  their
conclusions  depends  as  much  on  the  quality  of  the
information  in  the  possession  of  and  supplied  by  the
project  managers  as  on  the  expertise  of  the  independent
assessors.

2000  to  2006  period

The  new  proposals  for  regulations  aim  in  particular  to
reduce  still  further  the  factors  limiting  the  scope  of  the
evaluation  process,  out  of  concern  primarily  to  increase
the efficiency of the deployment of Structural Funds. The
following  proposals  are  made  in  this  spirit:

—  the  prior  appraisal  should  be  properly  incorporated
into  the  plans  which  Member  States  are  required  to
submit  containing  a  strict  quantification  of  the
objectives  to  be  attained,

—  the  results  of  evaluations  already  carried  out  in  the
previous period, either on a particular topic (thematic
evaluations)  or  by  a  specific  programme,  must  be
taken  into  account,

—  there  should  be  a  close  correlation  between  mid-term
reprogramming  and  the  conclusions  of  the  ongoing
assessment.

In  a  bid  to  ensure  greater  effectiveness  in  the  use  of
Structural  Funds,  the  Commission  has  proposed  that
10 %  of  the  Structural  Fund  aid  be  disbursed  at
mid-term,  depending  on  performance,  the  criteria  for
assessing  performance  being  closely  linked  to  the  quality

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/41

of  the  programming,  follow-up,  prior  appraisal  and
ongoing  assessments.

II.  SPECIFIC  COMMENTS

Finally, with regard more specifically to methodology, the
Commission  plans  by  the  autumn  to  publish  the  work
carried out under the MEANS programme (1) in order to
ensure  greater  harmonisation  of  evaluation  within  the
Union.  This  will  involve:

—  a  detailed  description  of  specifications  and  the
methodology  for  prior  appraisal,  ongoing  assessment
and  ex  post  evaluations,

—  a  survey  of  the  literature  on  different  assessment
methods  and  the  conditions  for  their  application,

—  an  analysis  of  systems  of  indicators  by  area  of
intervention.  Harmonisation  of  concepts  and  a  list  of
suggested  indicators,

—  methodology  and  organisation  of 

subject-based

evaluations,

—  a  multilingual  glossary  of  the  main  concepts  of

evaluation,

—  an  outline  of  the  most 

innovative  national  or

Community  projects.

2.  THE  SCOPE  OF  THE  AUDIT

2.3. The Court did not include in its audit countries like
Germany  and  the  United  Kingdom  that  have  a  very
different  attitude  to  evaluation.

3.  SUMMARY  OF  THE  MAIN  OBSERVATIONS

3.2. The  Commission  believes  that  the  evaluations  for
which  it  does  not  have  the  figures  on  expenditure  are
marginal in terms of costs. Because ongoing assessment is
a  new  and  growing  field  and  is  carried  out  at  a
decentralised  level,  the  volume  of  expenditure  devoted  to
it is changing. Expenditure on evaluation is broken down
by country, by objective and by type of intervention. The
figures  are  also  published  in  the  Commission’s  annual
evaluation overview.

The  Commission  will  continue  its  efforts  to  obtain  a
systematic  overview  of  the  amount  of  expenditure  set
aside for evaluation out of the sums awarded in technical
assistance  and  managed  under  the  partnership  system.

It  will  be  clear  in  what  follows  that  the  Commission
endorses  the  Court’s  comments  in  some  cases.  But  it  is
important  to  take  account  of  the  difficulties  and
limitations  inherent  in  any  evaluation  exercise,  even  if
these  are  being  gradually  overcome  or  superseded.

3.3. Under the terms of the regulations and in deference
to  the  subsidiarity  principle,  the  Commission  has  no
authority  to  dictate  a  precise  framework  for  evaluation
indicators  and  methods.

Nor  should  we  lose  sight  of  the  fact  that  evaluation  is
first and foremost a technical aid to decision-making and
management;  it  is  not  intended  to  take  the  place  of
political  decision-makers.  It  serves  to  shed  light  on  the
consequences  of  choices  to  be  made  or  decisions  already
taken. Moreover, there is a need for a degree of flexibility
as to the methods used in order to allow for the different
contexts  from  one  evaluation  to  the  next.

Finally,  the  Commission  would  point  out  that  since  the
start  of  the  Fund  reforms  in  1988,  the  respective
responsibilities  of  national  administrations  and 
the
Commission  for  Structural  Fund  management  in  general
and  evaluation 
in  particular  have  been  constantly
shifting.  The  Member  States  are  increasingly  required  to
assume  prime  responsibility  and  to  carry  out  their  own
evaluation.

(1)  MEANS  =  Methods  for  evaluating  activities  of  a  structural
nature:  a  multi-annual  programme  to  analyse  and  propose
methodologies  of  evaluation.

In an effort to develop national practice and improve the
quality  of  evaluation,  the  Commission  has:

—  set  up  the  MEANS  programme  and  distributed  eight

booklets  on  methodological  questions,

—  organised  three  European  conferences  on  evaluation

techniques  and  practice,

—  distributed  documents  such  as  the  frame  of  reference
for  prior  appraisals  and  a  guide  for  evaluating  major
projects,

—  set  up  Technical  Evaluation  Groups  in  which  the
Member  States  participate  and  organised  several
seminars.

is  an  objective  problem 

to
3.5. There 
coordinate  the  ex  post  evaluation  with  the  different
stages  of  planning  structural  interventions.  Ex  post
evaluations  should  really  be  carried  out  two  or  three
years  after  the  end  of  programming.

trying 

in 

C  347/42

EN

Official  Journal  of  the  European  Communities

16.11.98

In  order  to  solve  this  problem  the  Commission  has
promoted ongoing assessments and thematic assessments.
The  thematic  assessments  carried  out  in  1992  and  1993,
for  example,  produced  material  that  could  be  used  when
preparing  the  programming  for  1994  to  1999.

The  ongoing  assessments  will  be  available  for  the
preparation  of  programming  after  2000,  because  they
have  already  been  completed,  and  evaluation  overviews
are  currently  being  introduced  for  each  objective  to
supplement  and  update  the  ongoing  assessments.  Four
thematic  evaluations  have  also  been  started.

The  Commission  has  introduced  this  provision  into  its
proposals  for  amending  the  Regulations.

should  be  made  between 

3.6. A  distinction 
the
objectivity  of  the  evaluations  and  the  adequacy  and
quality  of  the  data.  The  question  of  data  is  often
highlighted  in  evaluations.  The  reports  almost  always
pinpoint this problem. However, there has been a distinct
improvement  in  the  situation  as  regards  data,  largely
thanks  to  the  demands  of  evaluation.

The  Commission 
is  not  satisfied  with  the  current
situation as regards data, but this should not be regarded
as  a  criticism  of  evaluation  but  of  the  monitoring
systems.  The  Commission  would  stress  that  the  problem
of  the  relevance  of  monitoring  data  and  indicators  is  a
matter  primarily  for  the  follow-up  audit.

The  Commission  is  continuing  its  efforts  to  improve  the
situation  and  will  act  on  the  Court’s  comment  about  a
need  for  more  systematic  evaluation  of  the  reliability  of
data.  The  work  carried  out  in  connection  with  the  audit
and control will also help to consolidate the improvement
in  the  situation  which  has  been  apparent  in  the  field  for
several years.

3.7. The  departments  concerned  comply  with 
the
conditions  for  awarding  contracts  prescribed  in  the
relevant  Directives  and  with  the  procedure  laid  down  by
the  ACPC.  However,  it  is  not  possible  to  negotiate  the
remit  before  signing  the  contract.  This  sometimes  means
that  clarifications  have  to  be  made  which  upset  the
timetable,  but  the  problem  is  marginal.

basis,  some  reports  in  a  country  may  be  excellent  while
others  are  unsatisfactory.

3.8. The Commission shares the Court’s views about the
difficulties  experienced  by  the  first  evaluations  organised
in  a  climate  of  indifference  in  the  Member  States.

The  ex  post  evaluations  of  Objectives  1  and  2  were
carried  out  by  consultancies  with  an 
international
reputation  and  by  highly  qualified  experts  using  a
common methodology. The second part of paragraph 3.7
justifies  both  the  measure  and  the  evaluation  exercise,
despite  these  constraints.  The  Court’s  criticisms  refer  to
the  evaluations  carried  out  in  1993  to  1994,  but  on  the
Commission’s  initiative  a  great  deal  of  work  has  been
done  since  then  on  developing  methodologies.  The
Court’s criticism of the ex post evaluation of Objective 2
is  also  valid  in  part.  However,  a  more  realistic  view
should  be  taken  of  the  possibility  of  devising  a  precise
impact  or  Community
quantitative  measurement  of 
added value.

3.9. The  ongoing  assessment  took  the  form  of  an
analysis  of  the  implementation  of  the  interventions.
Effects are more likely to be assessed in the context of the
evaluation  overviews  and  ex  post  evaluations.  These
evaluations,  however,  do  not  always  involve  the  use  of
econometric  models  —  which  tend  to  be  very  expensive
and  are  not  necessarily  the  most  appropriate  way  of
measuring impacts.

Evaluation is a management tool and, as such, just one of
several  aids  to  the  decision-making  process.

3.10. The  Commission  found  that  dividing  Objective  2
programming 
into  two  periods  did  not  correspond
adequately  to  the  needs  of  the  areas  in  question  and
implementation  of  programmes  more
made 
complicated. The Commission’s proposals in March 1988
for a reform of the Structural Funds provided for a single
programming  period  for  Objective  2,  as  is  the  case  for
the  other  objectives.

the 

In  order 

3.11.  —  3.12.
to  avoid  having  an
unmanageable  number  of  evaluations  of  programmes  or
selective  approach  may
sub-programmes  a  more 
sometimes  be  needed  in  certain  key  areas.

The  Commission  alone  cannot  be  blamed  for  this
problem.  The  inadequacies  of  some  evaluation  reports
may  also  be  due  to  incompetence  on  the  part  of  the
consultants  or  the  poor  quality  of  the  data  available.
Even  when  the  same  terms  of  reference  are  used  as  a

The  Commission  shares  the  Court’s  opinion  on  the  need
to  cut  the  number  of  measures  and  sub-programmes  or
even programmes. It made a number of specific proposals
to  this  effect  in  the  course  of  the  Fund  reform.

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/43

4.  GENERAL  ISSUES

Introduction

4.1. On  several  occasions  the  Commission  expressed  its
regret at the failure of certain Member States to quantify
their  objectives  adequately.  The  fact  that  national  public
expenditure is not administered by objective also hampers
the  development  of  management  systems  based  on
indicators.  Consequently  it  is  less  a  matter  of  a  defect  in
the  methodological  or  analytical  framework  than  a
problem  of  programme  design  and  the  sophistication  of
national  statistics.

1  CSFs  and  15  Objective  2  SPDs  at  the  beginning  of  the
period.  In  addition  to  the  thematic  assessments,  which
also  relate  to  the  period  1989  to  1993,  the  Commission
carried  out  an  ex  post  evaluation  of  all  CSFs  and  SPDs
for  Objectives  1  and  2  in  the  period  1989  to  1993.

4.11. The  question  of  knowledge  of  costs  was  raised  in
paragraph  3.2.  The  only  evaluations  that  might  not  be
included  in  the  financial  overview  of  evaluations  drawn
up  by  the  Commission  are  the  small  number  initiated
solely  by  the  Member  State  and  funded  from  technical
assistance. The amounts involved are marginal compared
with  all  the  other  work.

4.3.  The  Commission  would  point  out  that:

Frame  of  reference

—  in  1994  it  sent  Member  States  its  common
ongoing
on  monitoring 

and 

guidelines 
assessments,

(consisting 

—  in  December  1994  it  presented  the  Technical
of 
those
Evaluation  Group 
responsible  for  evaluation 
in  the  Member
States)  with  a  framework  document  on  the
purpose  and  content  of  and  procedure  for
ongoing  assessments.  This  initial  meeting  was
followed  by  four  further  meetings  of  the  same
group,

—  in  1995 

to 

it  distributed 

the  various
departments  copies  of  a  MEANS  document  on
how  to  organise  ongoing  assessments  in  the
context  of  partnership.  Three  of  the  five
deal  with
chapters 
formulating  the  specification  and  invitations  to
tender,  selecting  assessors  and  supervising  the
quality  of  the  evaluation,

document 

this 

of 

—  in 1996, before receiving the first evaluations, it
distributed  to  the  departments 
involved  a
MEANS  document  on  assessing  the  quality  of
evaluation 
a  detailed
schedule.

including 

reports, 

It 

should  be 

recalled,  however, 

some
4.5.
evaluations  were 
carried  out  under  partnership
arrangements  even  in  the  period  1989  to  1993.  Thus
Member  States  and  the  Monitoring  Committees  actively
participated  in  several  thematic  and  ex  post  evaluations
initiated  by  the  Commission.

that 

4.12. — 4.13. The MEANS documents were intended as
educational tools for administrations which are often still
developing  their  technical  expertise.  The  priority  for  the
Commission  was  to  raise  the  average  standard  of
evaluation,  not  the  level  of  experts  who  are  already
specialised.

Statistical  bases  and  information  concerning  objectives

4.14.  —  4.15. The  Commission  shares  the  view  of  the
Court.

Indicators

efforts 

to  develop 

acknowledges 

4.21. The  Court 

the
4.16.  — 
Commission’s 
indicators.  The
regulations were tightened up in 1993, and an exercise to
quantify  disparities  was  launched  with  each  Member
State.  The  question  of  quantifying  objectives  has  been  a
priority  for  the  prior  appraisal  and  some  progress  has
been  made:  the  1994  to  1999  interventions  are  distinctly
better quantified than their predecessors. The first task of
the  ongoing  assessments  has  been  to 
improve  the
indicators  and  their  effectiveness.

4.10. The  Court  comments  on  the  small  amount  of
money  earmarked  for  Objectives  1  and  2  evaluations  in
the  first  programming  period.

Because  almost  none  of  the  Member  States  had  carried
out  an  evaluation,  the  Commission 
itself  took  the
initiative  of  conducting  a  first  evaluation  of  all  Objective

The  Commission  would  stress  that  the  problem  is  not
just  one  of 
increasing  or  reducing  the  number  of
indicators  but  the  difficulty  the  monitoring  systems  have
in regularly quantifying the agreed indicators that already
exist  for  the  programmes.  Nor  is  it  a  question  of  a  lack
of  coherence  in  programming,  for  it  is  the  right  of  the
regional  and  local  authorities,  under  the  principle  of
subsidiarity, to set their own priorities and determine the
scope  of  programming  in  their  countries.

C  347/44

EN

Official  Journal  of  the  European  Communities

16.11.98

Progress is, however, being made, under the pressure and
demands  of  evaluation.

The  objectivity  of  evaluations

The  Commission’s  view  is  that  a  simple  methodological
framework is not enough and it therefore intends to take
more  far-reaching  steps  as  part  of  the  Fund  reform.

In  these  paragraphs  the  Court  does  not
4.28.  —  4.30.
make  sufficient  allowance  for  the  complexity  of  the
partnership  system,  despite  the  positive  comments  in
paragraphs  6.5  to  6.7.

Taking  evaluation  into  account

the 

that 

agrees 

4.23. The  Commission 
dual
programming cycle did not make coordinating evaluation
and  programming  any  easier  and  for  this  and  other
reasons  it  proposed  a  single  programming  period  for  the
future.  Nevertheless,  the  1997  to  1999  programming
drew  on  an  ex  post  evaluation  of  the  1989  to  1993
period  and  on  a  prior  appraisal  of  the  SPDs  carried  out
on  the  Commission’s  initiative  and  relating  to  matters
under  its  authority  and  priorities  for  the  Union.

As for the ex post evaluation of the 1994 to 1996 period,
the  Commission  believes  that  the  evaluations  carried  out
in  the  context  of  reprogramming  and  the  thematic
evaluations  of  SMEs  and  R&D  cover  much  of  this
ground.  The  Commission  is  currently  discussing  with  the
Member  States  the  launch  of  an  evaluation  overview  in
the  second  half  of  1998  covering  the  period  since  1994.
This  overview  would  be  used  for  future  programming.

4.24.  —  4.27. The  Commission  is  currently  drafting  a
report  describing,  for  each  country,  the  quality  of
evaluation  and 
its  contribution  to  modifications  to
interventions. A summary will cover the key points of the
mid-term  review.  The 
information  available  to  the
Commission  does  not  support  the  Court’s  comments
relating  to  widely  divergent  situations  and  hundreds  of
programmes.

The  Commission  will  assess  the  results  when  the  report
on  the  mid-term  review  is  presented.  For  the  time  being,
the  Court’s  position  seems  to  be  premature.

The  Commission  would  also  point  out  that  evaluations
are useful not only for managers but also in the interests
of  greater  accountability  to  the  budgetary  authority,  the
European  Parliament  and  the  public.

to 

the 

is  more 

There 
issue  of  objectivity  and
independence  of  evaluations  than  the  existence  of  a
precise frame of reference for applying the methodologies
and  checking  data  on  the  spot.  The  quality  of  the
evaluation  relates  more  to  degree  of  professionalism,
professional  ethics  and  the  interplay  with  the  interests  of
the  Member  States.

The Court fails to mention that three of the eight criteria
in the MEANS brochure No 5 on assessing the quality of
evaluation  reports  explicitly  refer  to  the  appropriateness
of  the  methodology,  the  reliability  of  the  data  and  the
credibility  of  the  results.

The  Commission  is  constantly  striving  for  improvement
in  the  standards  and  requirements,  bearing  in  mind  the
reality on the ground. It will be turning its attention more
to  assessing  the  operation  of  evaluation  systems  on  the
ground.

The  evaluation  contracts  awarded  by  the  Commission

4.31.  —  4.32. The  Commission  applies  the  standards
and  procedures  for  awarding  contracts  laid  down  by  the
ACPC.  Experts  are  chosen  on  the  basis  of  selection  and
award  criteria  published  in  the  Official  Journal.  Their
application  and  the  results  of  this  exercise  feature  in  the
report  to  the  ACPC.

As  regards  the  degree  of  precision  of  the  contracts  it
should  be  borne  in  mind  that  the  Commission  is  paying
for  expert  advice  and  has  every  interest  in  ensuring  that
the  contracts  are  drafted  in  such  a  way  as  to  enable  the
experts  appointed  to  make  the  best  use  of  their  skills.

for  Objective  1 

is  not
4.33. The  method  used 
appropriate 
for  Objective  2,  which  comprises  60
programmes.  Given  the  expenditure  involved  and  the
number of teams, it would lead to a fragmentation of the
invitation  to  tender  that  would  go  against  the  letter  and
the  spirit  of  the  Directive  on  the  procedure  for  awarding
public  works  contracts.

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/45

The  Commission  complied  with  the  rules  in  these  cases
and  was  careful  not  to  lose  control  of  the  practical
arrangements  for  implementing  the  evaluation.  To  this
end  a  steering  committee  was  set  up  to  accompany  the
evaluation and meetings were organised periodically with
the  evaluation  teams.  The  reports  were  subsequently
checked  with  the  departments  in  charge.

4.34. The  details  of  the  specifications  depend  on  the
needs and information requirements of the requester. The
requester  drafts  them  and  the  evaluation  units  carry  out
checks.  The  terms  of  the  contract,  with  the  exception  of
the description of work to be carried out are standardised
by  the  ACPC.

As  pointed  out  earlier,  there  is  no  generally  accepted
methodology  for  programme  evaluation.  This  being  the
case,  and  in  order  not  to  lose  its  control  over  the
approach  adopted, 
the
methodology, its relevance to the questions raised and the
applicability  of  the  techniques  proposed.  The  general
methodological approaches form part of the specification,
but  the  precise  details  and  techniques  are  left  open  to
discussion.

the  Commission  discusses 

4.35.
In  this  specific  case  the  Commission,  with  the
the
agreement  of 
evaluation,  tried  to  improve  the  quality  of  the  report  by
organising a meeting on the content and the lessons to be
learned  with  all  the  units  and  officials  concerned.

the  consultants  who  produced 

The  final  report  was  submitted  in  May  1995  (three
months  after  the  scheduled  date),  accepted  after  official
consultation in June 1995 and payment was made in July
1995.  The  report  criticised  by  the  Court  was  in  fact  a
clarification  by  the  consultant  involving  no  additional
budget.

that 

4.36. The  Commission  stresses 
terms  of
three  principal,
structured  around 
reference  were 
well-defined  questions:  the  relevance  of  the  strategy,  the
results  and  effectiveness  in  terms  of  output  and  the
effectiveness  of  the  implementing  strategies  (pages  2  to  4
of  the  document).

the 

The Commission explained its position on the question of
methodologies 
various
departments  are  consulted  and  the  lead  department
decides  whether  or  not  to  accept  the  report.

in  paragraph  4.12.  The 

5.  THE  EX  POST  EVALUATION  OF  THE  OBJECTIVES  1

AND  2  CSFs  (1989  to  1993)

Introduction

5.1. The Commission endorses the Court’s comments on
these  evaluations
the  difficulties  experienced  with 
connected  with 
first-generation
programmes.

the  design  of 

the 

The  ex  post  evaluation  of  Objective  1  CSFs  (1989  to
1993)

5.4. Some of the Court’s comments on this point should
be  qualified.

(a)  If data on regional disparities were out of date this is
a  criticism  of  the  statistical  office  rather  than  the
methodology  of  the  evaluation.

(b) The  assessors  used  the  sampling  techniques  generally
employed for evaluations, combined with case studies
for  greater  detail 
instances.  These
techniques,  properly  used,  can  provide  useful
lessons.

specific 

in 

(e)  The  Court  states  that  bottom-up  analyses  do  not
generally  allow  conclusions  to  be  drawn  in  terms  of
impact  on
impact  at  programme 
improving
the 
Community  policies, 
management,  transparency,  etc.  cannot  be  top-down
or  based  on  econometric  models.

impact  on 

level.  The 

(g)  The  Court  criticises  aspects  that  were  not  within  the

remit  of  the  evaluation.

5.7. The  Commission  confirms  that  the  evaluations  in
question  were  fully  used  in  the  context  of  its  duty  of
accountability  to  the  budgetary  authority  and  as  part  of
the  work  on  the  report  on  Economic  and  Social
Cohesion.

5.8. One  of  the  three  objectives  of  the  remit  for  the
evaluation  was  to  suggest  ways  of 
improving  the
monitoring and management system. The relevance of the
strategy was also analysed and a performance assessment
drawn  up  which  enabled  the  effectiveness  of  the
interventions  to  be  analysed.

The  ex  post  evaluations  of  Objective  2  (1989  to  1993)

5.13. The  systematic  use  of  econometric  models  is  not
justified in all cases, either because the level of aid is low
or  because  the  necessary  regional  statistical  series  are
unavailable.

Most  of  the  assessments  of  the  policy  appear  in  the
report  on  Economic  and  Social  Cohesion.  The
Commission  has  suggested  modifications  as  part  of  its
new proposal for the reform of the Structural Funds from
the  year  2000.

C  347/46

EN

Official  Journal  of  the  European  Communities

16.11.98

5.16. The Commission has already included a number of
the  recommendations  of  this  evaluation  in  its  proposals
for  the  new  regulations.

Bottom-up  methods  are  used  for  evaluating  effectiveness
and  efficiency.  In  the  case  of  large  CSFs,  where  the
consequences of reprogramming on global impacts had to
be  determined,  use  was  made  of  econometric  modelling
techniques  in  Ireland,  Spain,  Greece,  Portugal  and  Italy.

6.  ONGOING  ASSESSMENTS  (1994  to  1999)

The  development  and  integration  of  the  principles  of
ongoing  assessment

6.1. — 6.20. The Commission shares the opinion of the
Court  in  this  part  of  the  report.  It  would  clarify  a
number  of  points:

—  Paragraph 6.15: The Commission believes that critical
appreciation  of  the  development  strategy,  the  impact
in  terms  of  objectives,  and  project  viability  are
matters more specifically for the prior appraisal or the
ex  post  evaluation.

Paragraph  6.16:  The  purpose  of 
the  ongoing
assessments is to adjust the interventions and improve
management. Analysis of the Community added value
will  be  part  of  the  evaluation  overview  or  ex  post
evaluation.  However,  the  evaluation  of  the  impact  of
Community  priorities  on 
the  modification  of
interventions  will  in  several  cases  contribute  valuable
information  about  Community  added  value.

summary  of 

in  countries  with 

Paragraph  6.18:  As  far  as  possible,  the  Commission
tried to obtain a picture via the ongoing assessment of
the  synergy  between  operations  financed  by  different
Funds,  particularly 
separate
programming, as in the case of the German Objective
1  SPDs.  To  this  end  the  assessors  appointed  for
ERDF  projects  produced  a 
the
evaluations  carried  out  under  each  Fund.  The
Commission  agrees  that  the  regional  evaluation  cited
by  the  Court  was  particularly  weak,  but  it  is  not
representative of all the evaluations carried out in that
country.  It  is  correct  to  state,  for  example,  that  the
ongoing assessment did not analyse the changes in the
indicative  funding  plan  resulting  from  a  request  by
Brandenburg to change its OP. As for the high rate of
absorption  of  funds  allocated  in  the  case  of  the
projects  referred  to,  a  number  of  changes  were  made
to  the  measures  in  the  Monitoring  Committees  of  12
June  1996  and  5  August  1996.

Approaches,  methods  and  techniques

6.21. The  Court  criticises  the  vague  formulation  of
interim objectives in the contracts. The Commission gave
guidelines in certain cases, but these were not followed at
the  decentralised  level.

6.22. The  Commission  drew  on  the  best  expert  opinion
in  Europe  and  the  United  States  of  America  to  identify
the  most  effective  ways  of  evaluating  programmes.

Objective  methodological  difficulties  and,  in  many  cases,
inadequate monitoring systems, limit the comparability of
results  at  European  level.

6.23. The  aim  of  ongoing  assessments  is  not  usually  to
measure  the  impact  of  programmes.  This  is  a  deliberate
choice  because  the  impact  is  not  quantifiable  in  the  first
phase  of  the 
intervention,  the  sequence  being  first
implementation,  then  results  and  finally  impact.  The
question  of  the  evaluation  of  the  overall  impact  of  large
CSFs  was  discussed  at  paragraph  6.21.

7.  IMPACT  OF  THE 

STRUCTURAL  FUNDS  AND

EVALUATION  MODELS

Impact  of  CSFs  and  programmes

In  this  part  of  the  report  the  Court
7.1.  —  7.15.
presents 
the  results  of  different  econometric  and
macroeconomic  models  but  without  any  explanation  of
their  scope  or  nature,  which  could  lead  to  confusion.
These  data  are  presented  in  a  structured  manner  in  the
report  on  Economic  and  Social  Cohesion,  with
appropriate  background  information.

Impact  of  sub-programmes,  measures  and  projects

7.16. — 7.19. The Court appears to criticise the lack of
evaluation  at  the  level  of  sub-programmes  or  measures,
yet 
is  generally
bottom-up,  is  necessarily  based  on  the  evaluation  of
measures  and  sub-programmes.

the  programme  evaluation,  which 

Similarly, the current thematic evaluations that have been
launched  by  the  Commission  are  concerned  more
specifically  with  certain  categories  of  measure  and/or
sub-programme  across  the  board,  as  in  the  case  of
measures  for  small  businesses,  R&D,  the  environment,
etc.

16.11.98

EN

Official  Journal  of  the  European  Communities

C  347/47

8.  CONCLUSION

8.1.  —  8.19. The  Commission  believes  that  the  Court’s
conclusions provide constructive criticism on issues which
the  Commission  has  been  working  on  for  years  and  will
continue  to  develop.

been made. The steps taken by the Commission under the
aegis  of  the  Structural  Funds,  reinforced  by  the  SEM
2000  exercise,  are  helping  to  develop  a  culture  of
evaluation  which  did  not  exist  before.  There  are  few
regions  that  have  not  carried  out  evaluations  and  that
have  not  committed  themselves  to  this  process.

The  Commission  admits  that  the  Court  has  detected
weaknesses  but  it  has  also  noted  the  progress  that  has

The  subsidiarity  principle  and 
the  difficulties  and
heterogeneity  of  geographic  locations  and  subject  areas
are  both  factors  to  be  taken  into  account.