Report from the Commission - Mid-term review of structural interventions - Objectives 1 and 6 (1994-99) - Developing a management culture through evaluation: towards best practice
Date: 1999-01-07
Institutions: European Commission
Subjects: EU programme, Structural Funds, activity report, project evaluation

COMMISSION  OF THE  EUROPEAN  COMMUNITIES 

Brussels,  07.01.1999 
COM(1998)782  final 

REPORT  FROM  THE  COMMISSION 

MID-TERM  REVIEW  OF 
STRUCTURAL  INTERVENTIONS 
OBJECTIVES 1 AND 6 (1994-99) 

DEVELOPING  A MANAGEMENT  CULTURE 
THROUGH EVALUATION:  TOWARDS BEST PRACTICE 

si 

tSHlfc.' 

s»ëf-. 

mm 

i$fja£ 

m 

•5 

&£$ïk 

E X E C U T I VE  S U M M A RY 

The Mid-Term Review: raising effectiveness 

1.  The  Mid-Term  Review  of  Objective  1 and  6  programmes  is  a  process  informed 
primarily by the various Mid-Term  Evaluations carried out at programme  level. It 
also takes  account  of  other elements  including  national  policy  contexts, the  EU's 
own priorities and other relevant strategic  analyses. The principal  aim of the Mid-
Term  Review 
is  therefore  a  thorough  and  systematic  assessment  of  the 
programmes,  leading  where  necessary  to  the  adjustment  of  priorities  and 
reallocation of budget for each programme. 

2.  The  present  Mid-Term  Review  was  able  to  draw  upon  a  large  number  of 
evaluation  exercises  (more  than  a  hundred)  and  was  greatly  facilitated  by 
discussions in partnership with national authorities. 

3.  The  Mid-Term  Review  covered  virtually  all  Objective  1 and  6  CSFs  and  SPDs. 
The  importance  of  this  process  varied  from  one  Member  State  to  another.  In 
general,  it  has  enabled  a  more  effective  deployment  of  resources  and  provided 
support for specific  strategic priorities. 

Mid-Term Evaluations have proved  useful to inform programme adjustments 

4.  Mid-Term  Evaluations  are not  an end  in  themselves  but  a means  to improve  the 
quality  and  relevance  of  programming.  They  involve  substantial  amounts  of 
human and financial  resources as well as, in some cases, considerable  preparation 
and discussion involving programme partners. 

5.  These  evaluations  were  not  implemented  simply  so  as  to  comply  with  existing 
Community programming rules. They  were  also introduced  because of  a growing 
interest  in evaluation  on the part of national  or regional  authorities.  Increasingly, 
these  authorities  are  coming  to  see  evaluation  as  a  means  of  improving  the 
management of Funds and maximizing the results obtained. 

6.  Mid-Term  Evaluations  offer  an  important  means  of  assessing  the  desirability  of 
particular  changes  to  original  programme  objectives.  Programming  of  structural 
interventions can be influenced  by changing external factors  and by changes in the 
general  policy  framework.  Factors  such  as  these  can  necessitate  a  review  of 
strategic priorities and lead to more systematic and effective  programming. 

7  Benefits  arising  from  the  present  evaluation  include  enhanced  value  for  money 
and a revised  balance of priorities  matched  to changing  needs  and  circumstances. 
These  should  be seen  as part of  a more  fundamental  learning process, one which 
includes benefits  which might not be fully  recognised in the short term such as the 
increase  of  existing  evaluation  capabilities  and  improved  co-ordination  between 
national and regional authorities. 

Assessing performance is  relatively straightforward 

8.  Compared with previous exercises in respect of earlier programming periods, the 
in  the  use  of 
Mid-Term  Evaluations  revealed  significant  developments 
quantitative  indicators  as  a  means  of  monitoring  the  implementation  of 
interventions, assessing their impact and determining progress towards their stated 
objectives.  Although  the  development  of  indicators  has  been  uneven  to  date, 
elements  of  'good  practice'  identified  in  certain  programmes  suggest  that 
measuring  and assessing  performance  is  relatively  straightforward,  and does  not 
necessarily require supplementary work on the part of national administrations. 

9.  The use of formal performance indicators for measuring progress in relation to the 
main  objectives  and  priorities  of  a  programme  can  be  of  greater  use  than  has 
hitherto been recognised. This should lead to more efficient  management capable 
of delivering demonstrable improvements in terms of actual outputs and results. 

10. Assuming that Member States will have to carry out ex-ante evaluations of future 
programmes,  it  is  essential  that  programming  documents  contain  quantified 
objectives  and  indicators  relevant  to  the  Mid-Term  Review  process.  The 
availability  of  this  type  of  information  can  be  genuinely  useful  for  the 
management of the programmes in question and can serve as an objective basis for 
judging performance.  It will, following  proposals by the Commission  in the draft 
Structural Fund Regulations, determine the allocation of a reserve to programmes 
according to their own internal performance. 

11  As regards the operational framework, monitoring systems will have to ensure that 
relevant  indicators  are measurable  in  a short  time  period  and at  low  cost,  using 
adequate  existing  information  sources,  and  that  they  are  subject  to  control  by 
national  and  regional  administrations.  In  addition,  realistic  targets  should  be 
established  in  a  transparent  way  and  initial  objectives  maintained  during  the 
implementation  phase. These  should  be  adjusted  downwards  only  where  this  is 
justified by changing circumstances. 

The Mid-Term  Review  has  been flexible  enough  in  meeting  changing  needs  and 

priorities 

12. The  current  approach  to  Mid  Term  Review  seems,  in  general,  to  be  both 
appropriate and sufficiently  flexible in its response to changing circumstances and 
the  need  to  ensure  that  current  interventions  are  made  more  efficient.  Both  of 
these objectives were able to be pursued within the same operational framework. 

13. Strategic priorities are designed in a particular economic  and institutional  context 
and are  subject  to  both  external  and internal  influences.  Although  continuity  of 
policy  is  desirable,  there  is  always  scope  for  further  improvements,  even  in the 
case of the most successful  interventions. In rapidly growing economies, (e.g. the 
Irish economy), economic  performance  enables  structural  changes  to be made to 
the  programming  framework  so  as  to  address  major  problem  areas  such  as 
inadequate physical infrastructure or skills shortages. 

14. The  Mid-Term  Review  carried  out  for  Objective  1  and  6  programmes  did  not 
radically  affect  their  overall  strategic  orientations.  The  extent  to  which  these 

interventions  have  been  adjusted  varies  between  Member  States  and  regions,  as 
well  as between  CSFs and SPDs. These  latter,  although  smaller  in  scale, involved 
transfer  between 
the  different  measures  which  were  proportionately  more 
substantial. 

15. In  advancing  its  own  priorities  for  adjustments  to  be  made  in  the  context  of  the 
Mid-Term  Review,  the  Commission  gave  particular  emphasis  to  sharpening  the 
focus  on  employment,  especially  by  supporting  more job-intensive  interventions. 
Although  more  could  have  been  done  to  support  this  overriding  priority, 
encouraging signs such as the results of the Mid-Term Review have emerged  from 
the  current  experience.  For  example,  there  has  been  some  reorientation  of  major 
aid  schemes  so  as  to  benefit  initiatives  favouring  SMEs  and  employment.  In 
addition, Territorial Employment  Pacts have been included  within  the  mainstream 
programmes  and  specific  actions  introduced  with  regard  to  Information  Society 
and equal opportunities between men and women. 

Notable progress  in evaluation, but further  improvements are needed 

16. Lessons  drawn  from  the  present  Mid-Term  Review  show  that  it  is  not  a  purely 
academic  exercise,  but  operationally  and  strategically  oriented.  In  many  Member 
States, programme managers  and policy makers took a similar view  and asked, on 
the basis  of  the evaluation  findings,  what kind of  adjustments  needed to be made, 
particularly with regard to the current interventions. 

17. Notable  progress  has  been  made  regarding  the  quality  and  content  of  the  Mid-
Term  Evaluations  and  their  incorporation  into  the  overall  Mid-Term  Process.  In 
time,  this  will  lead  to  an  improved  management  culture.  In  identifying  good 
practice, factors  such as process management (e.g. identifying  key issues, reaching 
consensus, etc.) and more careful  planning of the Mid-Term  Review  process have 
been noted. These will be included in any future  guidance on best practice. 

18. As a result  of the Mid-Term Review, national  and regional  authorities  have taken 
steps  to  improve  further  the  effectiveness  of  their  interventions,  especially  with 
regard to their own delivery systems for Structural Funds. In particular,  significant 
improvements  will  be  made  to  the  monitoring  systems,  notably  by  raising  the 
quality  and  relevance  of  the  indicators  set  out  in  the  programmes.  Other 
arrangements  are  currently  being  sought  to  simplify  the  management  of  small-
scale  programmes  and  establish  more  coherent  selection  criteria  in  line  with 
programming  objectives. 

19. With regard to the next programming period, the main issue will be to consolidate 
evaluation  practice further,  taking stock of the results achieved  so far.  Reinforcing 
evaluation  as  a  genuine  tool  for  decision  making,  strengthening  monitoring  and 
indicator  systems,  supporting  evaluation  methodologies  and  promoting  a 
multilateral  exchange  of  experience  across  the  European  Union  appear,  in  this 
perspective,  as challenging  areas for  further  raising  the  effectiveness  of  structural 
interventions. 

in 

E X E C U T I VE  S U M M A RY 

INTRODUCTION 

/. 

Key elements of the Mid-Term Review process 

1.1. Policy context and evolving needs 

1.2. Commission guidelines 

1.3. Mid-Term Evaluations 

1.4. Financial implementation 

//. 

Lessons from  the Mid-Term Evaluations 

2.1. Implementing the Mid-Term Evaluations 

2.2. Principal findings 

2.3. Utilisation of evaluation results 

///.  Main outcome and adjustments 

3.1. Link between evaluation and reprogramming 

3.2. Overall reallocations of Structural Funds 

3.3. Inclusion of EU priorities 

IV.  Policy implications and future  challenges 

4.1. Short-term implications for current delivery systems 

4.2. Initial reflections  for future programmes 

4.3. The challenge of evaluation 

3 

4 

4 

5 

5 

6 

6 

6 

11 

21 

22 

22 

...24 

26 

29 

29 

30 

30 

2-

INTRODUCTION 
The Mid-Term Review1  aims to ensure a more effective  and efficient  implementation  of 
structural  interventions  delivered through Community  Support  Frameworks  (CSFs) and 
Single  Programming  Documents  (SPDs).  This  process  is  particularly  important  given 
public  budgetary  constraints.  These  latter  make  it  increasingly  necessary  for  Structural 
Funds  resources  to  be  fully  used  and  for  Structural  Funds  interventions  to  be  of  high 
quality in terms of their individual content. 

Evaluation  represents  an important  means  by  which  these  goals  might  be  realised.  An 
important  innovation  of  the  current  Structural  Regulations  is  the  requirement  for 
systematic evaluation to be introduced for all EU programmes. In the context of the SEM 
(Sound and Efficient  Management) 2000 initiative, the Commission regards evaluation as 
a key element for improving the management culture in which  Community  programmes 
operate2. More recently, the Commission's  policy document Agenda 20003 covering the 
period  2000-2006  has  aimed  for  greater  cost-effectiveness 
from  all  structural 
interventions  through  reinforcing  monitoring,  financial  control  and  evaluation  in  the 
context of decentralised management. 

The present report summarises the principal elements of  the Mid-Term  Reviews carried 
out in the Member States and analyses, in particular the role of the Mid-Term Evaluations 
as well as the extent to which EU priorities feature in all Objective 1  and 6 programmes. 

The  Mid-Term  Evaluations  carried  out  by  the  individual  Member  States  are  the  main 
source of  information  for this Report. It attempts to establish  the role  of  the evaluation 
process  in  the  development  of  the  present  Mid-Term  Review  and  to  identify  best 
evaluation  practice  as  a  guide  for  the  future  programming  period.  This  reflects  a 
management culture which is emerging through the development of evaluation practice. 

By identifying  good practice (Annex  1), this report contributes to a better understanding 
of  how  evaluations  were  implemented  and  used  tp  inform  the  Mid-Term  Reviews 
particularly  with  regard  to  how  they  will  be  used  in  future  in  the  context  of  the  new 
Structural Fund regulations. It thus offers some experimental basis for actions to be taken 
after 1999. 

The following four areas are examined in the Report: 

1.  the overall Mid-Term Review process; 

2.  the principal findings of the Mid-Term Evaluations; 

Throughout  this report  a distinction  is made  between  Mid-Term Evaluation  and  Mid-Term  Review. 
Although  the  two  terms  are  sometimes  used  interchangeably,  Mid-term  Review  more  properly 
describes  the  complete  process  comprising  the  evaluation  exercise  and  the  resulting  adjustment  of 
priorities  and  programme  budget  reallocations.  Mid-term  Evaluation  is simply  the actual  evaluation 
exercise  which  informs  the Mid-Term  Review. 

SEM 2000 - Communication  on Evaluation,  8 May  1996. 

Agenda  2000, For a stronger  and  wider Union,  1997. 

- 3-

3.  the main outcomes and adjustments made; 

4.  the implications for programme delivery and priorities as well as future challenges 

I. 

KEY ELEMENTS OF THE MiD-TERM REVIEW PROCESS 

The  Mid-Term  Review  process  draws  on  four  main  elements:  the  policy  context;  the 
Commission  guidelines;  the  Mid-Term  Evaluations;  and  the  state  of  financial 
implementation of Objective 1 and 6 programmes. 

/. /.  Policy context and evolving needs 

The main aim of CSFs and SFDs is to reduce disparities between Member States in terms 
of  their  respective  per  capita  income.  These  interventions  seek  to  effect  a  long-term 
change in the structure and the performance  of recipient economies. They are driven by 
specific  national  priorities  and  strategies  to  tackle  basic  structural  problems,  such  as 
differences  in infrastructure  and human capital. These  are important  factors  influencing 
regional competitiveness and hence growth performance. 

The  four  poorest  Member  States  -  the  "Cohesion  countries"-  have  seen  a  notable 
improvement in their per capita income levels. However, continuing disparities still exist 
among the European regions: in 1995, the 25 most prosperous regions had an average per 
capita  income  three  times  as  high  as  that  of  the  least-developed  regions.  The 
unemployment  gap  is  even. more  acute,  affecting,  in  particular,  the  most  socially 
disadvantaged groups4. 

Changes in the economic, social or political context in the Member States have  affected 
their respective needs and problems, and thereby influenced  the way in which they have 
adjusted their programmes. For example, Ireland's recent economic  success  has altered 
the need for certain types of  intervention,  notably interventions  which  aim to assist the 
productive sector. The Mid-Term Evaluation carried out in respect of that Member State 
has  shown  that  there  is  a need  to redirect  resources  towards  public  infrastructure  (e.g. 
non-urban roads) to keep pace with the rapid economic growth. 

Similarly, the acknowledgement of employment as an overriding EU priority has also led 
to  a  number  of  programme  adjustment  such  as  the  formal  introduction  of  Territorial 
Employment  Pacts.  In  addition,  changes  in  national  employment  policies,  such  as 
structural  reforms  in  the  labour  market  have  contributed  to  the  refocusing  of  some 
training actions. 

Unexpected developments were also addressed in the context of the Structural Funds. For 
example, the earthquake emergency in the Umbria and Marche regions of Italy resulted in 
those  regions  benefiting  from  CSF  resources  an  example  of  mutual  support  between 
Italian regions. 

See  First  Report  on  Economic  and  Social  Cohesion  (1996),  COM  (96)  542  final  of  6  November 
1996. 

1.2. Commission guidelines 

The  changing  economic  and  social  context  at  Member  State  level  has  led  the 
Commission  to  adopt  general  guidelines5  to  foster  the  value-added  of  Community 
interventions.  These  aimed  to  provide  a  general  policy  and  priority  framework  within 
which  adjustments  to their current  1994-1999  Objective  1 and 6  programmes  could be 
made as part of the Mid-Term Review process. 

The  guidelines  set  out  a number  of  thematic  priorities  in  support  of  the  main  aim  of 
Structural  Funds  interventions:  helping  to  establish  the  conditions  for  sustainable 
economic  development,  growth  and  competitiveness  and,  thereby,  lasting-jobs.  This 
overall  aim  was  underpinned  by  the  following  specific  priorities:  Basic  Infrastructure, 
Productive  Environment  (all  types  of  measures  to 
improve  the  growth  and 
competitiveness  of  business  and  industry),  Research  and  Technological  Development, 
Environment  and Sustainable Development,  Human Resources  Development  and Equal 
Opportunities. 

1.3. Mid-Term Evaluations 

In accordance  with Structural  Funds regulations  and the programmes'  standard clauses, 
all  Member States  (with  the exception  of  Italy6)  have  managed  to carry out Mid-Term 
Evaluations to inform the review of their structural interventions. These evaluations were 
designed to contribute to the process of improving the execution and management of the 
programmes for the remainder of the period. 

The guidance document7 published by the Commission stated that Mid-Term Evaluations 
should  involve:  a critical  analysis  of  all  of  the  data  collected  (particularly  monitoring 
data);  some  measurement  of  the  extent  to  which  objectives  were  being  achieved;  an 
explanation  of  any  discrepancies  between  the  actual  and  expected  results  of  the 
intervention;  an  assessment  of  the  rationale  for  the  intervention  and  the  continued 
relevance of its objectives. 

' In addition, evaluations were expected to provide the data necessary to assist Monitoring 
Committees  to  form  an  opinion  and  propose  corrective  measures  to  the  relevant 
authorities so as to enable adjustments to be made to the programmes. 

As a general rule, in the case of programmes lasting longer than three years, a Mid-Term 
Evaluation  is  carried  out  at  the  end  of  the  third  year  of  implementation.  These 

Priorities  for  the  adjustment  of  Structural  Funds  programmes  to  the  end  of  1999*  -  Commission 
Guidelines, May  1997. The formulation  of guidelines had been endorsed by the Member States at an 
informal  meeting  of  Ministers  responsible  for  regional  policy  and  spatial  planning  held  in 
Ballyconnell, Ireland on  14-15 November  1996. The Member States formally  asked the Commission 
to "formulate policy guidelines established in accordance with the principle  of subidiarity, aimed at 
adapting  the  current  Objective  1  and  6 programmes  towards  increased job-intensiveness  for  the 
period to the end of 1999 ". 

The Mid-Term Evaluation  was undertaken on an interim basis by the Italian  authorities  (Evaluation 
unit of Budget Ministry) in the first half of  1997. 

7 

Common Guide on Monitoring and Interim Evaluations (1996). 

- 5-

programmes  are  also  subject  to  ex-post  evaluation  at  the  end  of  the  reference  period. 
Evaluation arrangements should recognise this timescale. 

1.4.  Financial implementation 

Evaluations  deal  normally with  assessing  impacts  and progress  in  relation  to objectives 
and priorities. They rely on financial  data relating to the implementation  of programmes 
which  are  supplied  by  the  relevant  monitoring  systems.  In  addition,  budget 
implementation  at  EU  level  provides  an  overview  of  commitments  and  payments. 
Progress  here is  summarised  in the next  paragraph  for programmes  under Objectives 1 
and 6 (see Annex 2). 

Since  1994, there has been continuous and regular progress in the implementation of the 
Objective  1 programmes. This was sustained during the first half of  1998. By the end of 
1997, 64% of  the funds  had been committed (compared with 46% at the end of  1996). 
Similarly,  payments had been made covering 48% of  the funds  (compared with 33%  at 
the end of  1996). Progress was particularly marked in Spain and Portugal where, over the 
1994-1997  period,  respectively  69% and  70% of  funds  were  committed  and  53% and 
56%  of  funds  were  paid.  Other  Member  States  reported  significant  delays  in 
implementation  in  terms  of  commitments,  (particularly  Netherlands,  France,  Belgium), 
while  for  Italy  and  Austria,  delays  related  mainly  to  payments  made.  With  regard to 
Objective 6, the two Member States offer contrasting situations. Implementation is more 
advanced in Finland than in Sweden. In Finland, by the end of  1997, 59% of funds had 
been committed compared with 32% in Sweden. 

II. 

LESSONS FROM THE MiD-TERM EVALUATIONS 

Mid-Term  Evaluation  has  been  an  important  development  in  the  management  of 
Structural Funds. It offers a valuable framework for reflection on the overall efficiency of 
EU interventions. The main lessons to be drawn concern the implementation process and 
the value of the main findings in terms of policy. 

2.1. Implementing the Mid-Term Evaluations 

a) Mid-Term Evaluation:  a shared responsibility 

One  innovation 
in  the  current  Regulations  governing  Structural  Funds  is  the 
strengthening of requirements regarding the evaluation of efficiency  and effectiveness  of 
the interventions. These requirements are clearly set out in the Regulations.8 

The Commission  and the Member States have worked together to define  the operational 
procedures  to be followed.  Evaluation  is  thus  a shared responsibility.  During  1995, the 
Member States and the Commission were actively involved in trying to reach a consensus 

According to art. 26.2 of the Co-ordination Regulation, "appraisal  and evaluation  shall, according to 
the  circumstances,  be  carried  out  by  contrasting  the  goals  with  the  results  obtained,  where 
applicable,  and  by  reference  to  macroeconomic  and  sectoral  objectives  and  indicators  based  on 
national  and  regional  statistics,  to  information  yielded  by descriptive  and  analytical  studies  and  to 
qualitative analyses". The Standard  Clauses in the CSFs and SPDs, agreed  with the Member States, 
indicate that "monitoring  [...]  is backed  up by interim evaluations  so that  any  adjustments  required 
to the CSFs and to operations in progress may be made". 

on  the  content  of  such  evaluations  and  on  the  operational  procedures  to  be  followed  in 
each case. While the operational  framework  proposed by the Commission  was discussed 
multilaterally, the task of implementing the evaluation  has, in general, been carried out at 
the level of each Monitoring Committee. 

By  and  large,  despite  the  problems  which  accompany  any  new  procedures  and  the 
difficulties  some  Member  States  had  in  defining  and  initiating  an  evaluation  process, 
there  have  been  no  major  differences  of  opinion  between  the  Member  States  and  the 
Commission9. 

b) Setting up the Mid-Term Evaluations 

Beyond the regulations -  Improving management 

Most of the forms  of intervention  (CSFs, SPDs, Operational Programmes, Global Grants, 
e t c .)  programmed  in  each  Member  State  were  subject  to  a  specific  evaluation.  Some 
Member  States  also  instigated  additional  evaluations  to  complement  their  Mid  Term 
Evaluations.  These  were  either  thematic  (examining,  inter  alia,  SMEs  and  Research 
related  measures)  or  operational  (e.g.  selection  criteria,  procedures,  etc.).  The  high 
than  a  hundred  for  Objective  1  & 6 
number  of  Mid-Term  Evaluations  (more 
programmes10)  indicates  that  there  is now  a  widespread  recognition  of  their  importance 
by  the  Member  States.  This  is  shown,  not  just  through  their  compliance  with  the 
regulatory provisions, but also through their perception of evaluation as an instrument  for 
improving the management of funds  and optimising the results obtained thereby. 

The cost of the evaluations 

In  the  SEM  2000  Communication  on  Evaluation  (1996),  it  was  recommended  that  the 
overall budget for evaluation activities should not exceed 0.5% of the programme budget. 
Evaluation  costs  were  generally  financed  under  the  "Technical  Assistance"  budget  of 
each  programme. The  percentage  of  funds  allocated  to  this  activity  has  been  in  general 
below 0.1% of total cost of the programmes.  In any case, fixed  percentages  would have 
been  inappropriate  since the cost of  an evaluation  is not  necessarily  proportionate  to the 
cost of the programme in question. 

The evaluation  procedure 

The starting point for  the Mid-Term Evaluation  process is to analyse the extent to which 
the programme in question  lends itself  to evaluation  (the practicability of carrying out an 
evaluation  or  "evaluability ") and to establish  a methodology  to  apply  in  the  subsequent 
phase. An assessment of how readily a given prograrnme can be evaluated is necessary in 
order  to  address  some  gaps  or  inconsistencies  in  the  information  contained  in  the 
programming  documents  such  as  an  insufficient  quantification  of  targets  for  certain 
measures. Important to this process is the recognition that, before  a successful  evaluation 
can be undertaken, there is a need to ensure that sufficient  arrangements have been put in 

The  Madrid  Informal  meeting  of  Ministers  of  regional  policy  and  spatial  planning  on  30th 
November-1st  December  1995  marked  an  increased  awareness  among  the  Member  States  to  the 
importance of Mid-Term Evaluation and its operational aims. 

10 

See list in Annex 3. 

- 7-

place to allow the programme ir^question can be evaluated. In particular, the evaluation 
of a programme generally requires that relevant  indicators  and targets  have been  set in 
advance and monitored over the programme's lifetime (Box 1). 

Box 1: Assessing the practicability of evaluation - ERDF Andalusia OP (Obj. 1) 

This evaluation  aimed  to  describe  the  rationale for  the programmed  actions,  objectives  and  goals,  the 
degree  of synergy  between them and the adequacy of programme  management. It focused  on three main 
elements: quality of planning;  information systems; management capacity. The approach adopted involved 
an analysis of the programming  documents (using a Logical Framework Scheme) and interviews with key 
programme managers. 

The application of this methodology led  to some practical  conclusions.  With regard to the quality  of the 
programme,  the evaluator observed some  deficiencies in the SWOT (Strengths-Weaknesses-Opportunities 
and  Threats) analysis as well as problems  with some  of the  objectives (e.g.  Tourism) which  were poorly 
targeted. What is more, the information base was deemed to be too reliant on output as opposed to impact 
indicators. An assessment of the management system was also undertaken in order to bring about possible 
improvements, with  particular  regard  to  the  division  of  tasks  and  responsibilities  within  the  regional 
administration. 

Although  not  all  of  the  recommendations  made  were  taken  on  board,  this first  assessment  raised  an 
increased  awareness  among programme  managers  of  the  need  to promote  an  evaluation  culture  and 
establish a better understanding of their own programme. 

The  Mid-Term  Evaluation  itself  is  carried  out  halfway  through  a  programme's 
implementation. It is intended to assess the degree to which the programme has met its 
objectives,  determine  the  initial  impact  of  the  interventions  and,  where  necessary, 
propose recommendations  in order to improve the management  of  the programme and 
optimise its results. Mid-Term Evaluations should, in general, take as a reference  point 
the outcomes of the ex-ante evaluations conducted on the basis of the plans submitted by 
the Member States. 

Independence of the evaluators 

The Member States  agreed  that  it  was particularly  important  to ensure  that  evaluation 
reports were drawn up independent of both the authorities responsible for managing the 
Structural  Funds  and  the executive  bodies  with  responsibility  for  co-financed  actions. 
Mechanisms  have  therefore  been  established  for  appointing  external  evaluators 
(consultants,  specialist  academic  teams,  etc.).  In  those  Member  States  in  which 
evaluation  is  a  well-established  part  of  public  expenditure  programmes,  external 
evaluators have tended to benefit  from continuous advice and support from  the relevant 
authorities. 

Internal evaluations can have some benefits  (e.g. promoting "learning by doing") since 
managing authorities are closely involved in questioning the 'how' and the 'why' of their 
activities. However, for mid-term and ex-post evaluations, internal evaluations may not 
be practical, cost effective  or even  desirable. A notable exception  was Italy  where the 
Mid-Term Evaluation, performed by the Evaluation Unit of the Budget Ministry, raised a 
number of critical  issues, especially on the quality of the programming  documents and 
the lack of appropriate targets and indicators. 

-8 

c) Building up evaluation capacities in the Members States 

Establishing management  structures 

Responsibility  for  the  implementation  of  Mid  Term  Evaluations  usually  rests  with  the 
Monitoring  Committees.  Typically,  these  are  involved  in  defining  the  content  of 
evaluation  reports  and  establishing  a  work  programme  for  the  evaluators.  More 
importantly,  they  have  analysed  and  discussed  the  completed  evaluation  reports,  and, 
where  applicable, proposed  or  advised  on  possible  programme  changes  to maximise  the 
efficiency  and effectiveness  of the Structural Funds allocated. 

In  a  significant  number  of  cases,  Monitoring  Committees  have  established  technical 
groups or sub-committees  (see Box 2) in charge of ensuring ongoing  liaison  between  the 
Committees  themselves  and  the  evaluators.  These  have  then  drawn  up  reports  and 
proposals (based on evaluation results), for discussion  and clearance by the Committee as 
a whole. 

Box 2: Managing evaluations -  the creation of ad-hoc structures. 

In Ireland,  the CSF as a whole and its three largest component Operational Programmes were each served 
by a  dedicated  evaluation  unit providing  on going  advice  and  expertise.  External  evaluators supplied  a 
similar on-call service to five other Operational Programmes. 

The Monitoring  Committees  have  discussed the  Mid-Term Evaluation  reports  at  two  separate meetings. 
Their conclusions  and  recommendations,  along  with  the  content  of  individual  Operational Programme 
evaluations,  have  provided  input  to  the  overall  CSF  Mid-Term  Evaluation.  In  addition,  a  regional 
evaluation  report  informed  the  overall  CSF  evaluation  and  account  has  also  been  taken  of  the 
Commission's guidelines. 

In Portugal, technical  evaluation groups  were set  up to  orient and  control the  evaluation process.  Their 
main tasks included, inter alia, validating terms of reference, assisting in the selection  of evaluators and 
discussing final  reports prior  to  submission  to  the  Monitoring  Committees.  These  groups  consisted  of 
Commission  officials  and  representatives  of  the  relevant  national  and  regional  administrations.  They 
normally  met  at  least  twice  a  year  in  advance  of  the  Monitoring  Committee  meetings.  Constructive 
relations  between the partners  within these groups  have contributed to  an  improvement  in the quality of 
evaluation work. 

Role of the partnerships 

In  general,  programme  partners  have  shown  a  positive  attitude  towards  Mid-Term 
Evaluations  and  willingly  accepted  responsibility  for  taking  these  forward.  However, 
regional  authorities  vary  in  terms  of  their  specific  involvement  in  the  tasks  associated 
with  Mid-Term  Evaluation.  These  differences  reflect  characteristics  of  the  political  and 
institutional  systems (i.e. the degree of decentralisation), the different  forms  of  Structural 
Fund  interventions  (national,  sectoral,  regional  and  sub-regional  programmes)  and 
differing  practices and experiences at the different  Member State level. 

In  some  cases,  regional  authorities  were  responsible  for  setting  up  and  managing  the 
evaluation  process.  They  were  involved  in  selecting  the  evaluators,  defining  and 
facilitating  their  work  and  ensuring  that  the  Monitoring  Committees  discussed  and 
analysed  the  outcome  of  the  evaluations.  In  addition,  they  ensured  that  evaluation 
recommendations  were  applied  at  programme  level.  In  other  cases,  responsibilities  and 
work  were taken on by both  national  and regional  authorities. But normally,  where there 
were Community  Support  Frameworks  (as in the case of  some Objective  1 regions), the 
national authorities were responsible for the evaluation. 

- 9-

A further  situation  is  that  of  Member  States  where  the  regional  authorities  do  not  have 
executive  responsibilities  for  the  programming  and  management  of  Structural  Funds. 
Their  participation  in  the process, consisted  of  facilitating  the  provision  of  the  data  and 
information  needed  to  draw  up  evaluation  reports,  co-operating  in  the  analysis  of  the 
impact that policy measures had made at local level, and, to some extent, participating in 
the 
the 
recommendations of the evaluation reports. Social partners were also associated  with this 
process  to  the  extent  of  their  participation  in  Monitoring  Committees.  In  general, 
partnerships have been developed in a pragmatic and flexible  manner. 

redefinition  of  strategic  and  operational  guidelines 

stemming 

from 

Creating professional  skills in evaluation 

Over  the  past  four  years,  the  development  of  evaluation  activities  across  all  Member 
States  has  been  notable.  Structural  Funds  regulatory  requirements  have  been  a  main 
factor  in  this development.  The provision  of  evaluation  services  remains  largely  outside 
of the various  administrations,  although  Ireland  and Italy  are  notable  exceptions,  having 
created  their  own  specialised  teams.  The  development  of  evaluation  as  a  specialised 
service has led to growing professionalism  and expertise. Although  the evaluation  teams 
appointed for different  programmes tend to have multi-disciplinary  expertise and include 
both  consultancy  companies  and  academic  research  centres,  their  main  strengths  are 
usually in the fields of economics and management. 

d) Improving the quality of evaluations 

The  average  quality  of  Mid-Term  Evaluations  carried  out  during  the  1994-99 
programming  period  has  undoubtedly  improved  compared  with  the  previous  period 
(1989-93)  although  factors  such  as  delays  in  the  launch  of  certain  programmes  or  the 
adoption of inappropriate methodologies meant that some exercises were below standard. 

Evaluation  reports  were  subject  to  assessments  according  to  the  eight  criteria  set  out 
under  the  MEANS  Programme11. These  assessments  suggest  that  most  reports  were of 
reasonably  good  quality  (Box  3).  The  evaluations  frequently  provided  updated 
information  and analysis of the programmes in relation to their objectives  and targets and 
some  evaluations  (e.g.  Northern  Ireland)  have  applied  their  own  quality  assessment  in 
terms  of  strengths  and  weaknesses.  Experienced  evaluators  and  more  extensive 
knowledge  and  research  on  the  practicalities  of  evaluation  have  also  contributed  to 
enhancing the quality of the reports. 

11 

Quality Assessment  of  Evaluation  Reports: A  Framework.  MEANS  (Méthodes  d'Evaluation  des 
Actions de Nature Structurelle)  is a European Commission  programme,  which  aims to improve the 
methodological  tools  available  for  assessing  structural  actions.  They  were  given  by  assessing  the 
evaluation  reports  against the following  8 criteria:  meeting  needs: does the evaluations  adequately 
address  the  information  needs  of  the  MC  and  fit  the Terms  of  Reference?;  relevant  scope:  is  the 
programme's  rationale,  outputs,  and  impacts  fully  covered,  including  unexpected  outcomes?; 
defensible  design: is the evaluation design appropriate to answer  the questions asked?; data: is the 
data  used/collected  appropriate  and  is  its  reliability  duly  taken  account  of?;  sound  analysis:  is  the 
information  available subjected  to  appropriate  analysis?; credible  findings:  do the findings follow 
logically  from,  and  are  justified,  by  the  analysis?;  impartial  conclusions:  are  conclusions  fair, 
unbiased  by stakeholder  views, and operational?; and  clarity:  is the report  written  in a  way that  is 
easy to understand? 

- 1 0-

Box  3: Assessing the quality of an evaluation: the Irish CSF evaluation 

The evaluation report fulfils  the following  criteria 

information needs were adequately addressed; 

the rationale of the programme and its objectives were comprehensively examined; 

a specific methodological framework  was developed  to assess the effectiveness of measures; 

an adequate amount of reliable  data were collected (although gaps and deficiencies still exist); 

a sound analysis of CSF's performance was provided; 

credible findings  were produced and  had practical  effects on the progrgmme's  structure 

objective conclusions were proposed and were sufficiently detailed for  operational use; 

presented to stakeholders in an accessible and clearly written formats. 

The overall quality rating is therefore very high, due to a well designed evaluation framework.  The report 
was published and made available to the public. 

The  assessments  made  of  rural  development  programmes,  based  on  the  interim 
evaluation,  vary  somewhat.  In  some  cases  this  assessment  is  considered  acceptable 
(Spain), and in others satisfactory  (various German Lander) since it has achieved its aim 
of  drawing  attention  to good  results.  Elsewhere,  it is  regarded  as  being  of  a very high 
level  in  view  of  the  experience  and  quality  of  the  independent  assessors  and  a more 
precise  definition  of  aims  and  specifications  (Ireland).  Sometimes,  the  scope  of  the 
evaluations  has  been  more  limited  because  of  their  academic  or  excessively  general 
nature  or because  they  looked  only  at  implementation  and management  rather than at 
results and impact (France, Belgium). 

Some  evaluations  were  also  criticised  as  being  insufficiently  independent  of  the 
administration  which  commissioned  them.  In other cases,  constraints  were  imposed on 
the assessor's work by lack of cooperation from the administrations concerned. 

The  methodologies  applied  by  evaluators  have  varied  in  nature  and  quality  from  one 
programme  to  another.  Although  the  Commission  has  promoted  a number  of  methods 
and tools through the MEANS programme, their implementation is still at an early stage. 
For  example,  impacts  on  employment  are  often  seen  as  being  difficult  to  measure, 
especially when deadweight and displacement effects have been taken into account12. 

Despite a number of qualitative improvements, there remains a need to reinforce, through 
appropriate  guidance,  the  use  of  sound  evaluation  methods  over  the  next  evaluation 
stages. 

2.2. Principal findings 

Amongst their main goals, the Mid-Term Evaluations attempted to assess progress on the 
attainment  of  the  various  programme  objectives,  as well  as their likely  impacts  on,  for 
example, job creation. They also sought to examine the process by which the programmes 
had been managed and their component projects selected and monitored. 

See European  Commission  document,  "Counting the jobs",  1997 and the European Job Challenge 
(1998). 

- 1 1-

The purpose of this section is to review, on the basis of  a representative cross section of 
evaluations,  significant  results  relating  to  mainstream  issues  of  concern  and  to  suggest 
areas where improvement  is  needed.  It will  highlight  some  key features  of  the analyses 
carried out concerning three main themes: 

-  macroeconomic  impacts  for  the  largest  programmes,  notably  with  regard  to growth 

and employment; 

-  microeconomic assessment, focusing on effectiveness issues; 

- 

the  quality  of  delivery  systems,  particularly  monitoring  and  indicators  systems  and 
project selection criteria. 

From this analysis, some general conclusions can be drawn in terms of how these results 
might be used and integrated into the Mid Term Review process. 

a) Macroeconomic impacts: growth and employment effects 

Most  evaluations  of  Objective  1 CSFs  generally  include  a macroeconomic  assessment 
providing  a  number  of  valuable  insights  into  the  overall  Mid-Term  Review  process. 
These  illustrate  the  importance  of  taking  into  account  supply-side,  as  well  as  demand 
side, impacts and provide a first quantification of the likely scale of these impacts. 

For large-scale interventions such as major Objective  1 CSFs, macro-economic effects in 
terms  of  economic  growth  and  employment  are  likely  to  be  visible.  However,  these 
effects are difficult  to quantify due to the need to identify the counterfactual situation, i.e. 
the  situation  without  Structural  Funds.  One  option  is  the  use  of  simulation  techniques 
based  on  models  which  take  into  account  the  macro-economic  interdependence  of 
variables.  Although  there  are  several  models  quantifying  the  short-term  demand  side 
effects  (e.g. input-output  models), only a few  take into account the long-term effects  of 
supply  side  conditions  the  improvement  of  which  is  the  main  objective  of  Structural 
Funds13. 

In  the  context  of  the  Mid-Term  Evaluations,  the  supply-side  effects  of  adding 
infrastructure,  human capital  and productive investment to an economy  are estimated in 
different  ways of varying sophistication. For Ireland and Spain, HERMIN14-type models 
have been applied, while for Greece, Portugal and Italy more general econometric models 
have been used. Many variables are endogenous but "external" influences include interest 
rates,  exchange  rates  and  economic  growth  among  trade  partners.  Due  to  the  non 
availability of  a macro-model  for East Germany, the German evaluation  took  an ad hoc 
approach, applying,  step by step, estimates  for the various  parameters  influencing  GDP 

13 

14 

The  most  recent  attempts  to  quantify  the  combined  demand  and  supply  side  effects  of  Structural 
Funds  have been made using the QUEST II model, the results  of  which  were  presented  in the First 
Cohesion report (1996), the HERMIN model used for the Single Market Review (1996). 

Hermin  is  a  macroeconometric  model  specifically  designed  to  quantifiy  the  medium  to  long-term 
impact  of  the  CSFs.  It  provides  a  common  framework  focusing  on  the  most  important  structural 
features of the  'Cohesion countries'and  explains the mechanisms through  which the CSFs will  affect 
the supply-side of their economies. 

-  12-

and employment; the authors themselves note in the report that the results carrying a high 
degree of uncertainty. 

Two  studies  did  not  provide  an  integrated  supply  side  in  their  model  and  simply  made 
additional  calculations  on  supply  side  effects  based  on  average  values  available  in  the 
literature.  The  Portuguese  evaluation  calculates  an  additional  productivity  effect  of 
between  1.17%  and  2.35%. The  study on  East  Germany  assumes  long-term  supply  side 
effects  on  GDP  to  be  20%  of  the  short-term  demand  side  effect.  For  this  reason,  the 
results  presented  below  for  these  two  countries  (and,  at  this  stage,  for  Greece)  feature 
demand side effects  only. 

A specific  problem of macro-level  modelling is given for  those countries  whose territory 
is not fully  eligible for Objective  1 assistance (D, E, I). Each  of the evaluations  has dealt 
with this problem in a different  way: the German evaluation  analysed the impact on East 
Germany alone whereas the Spanish evaluation  only considered  the impact on Spain  as* a 
whole. The  dualistic  structure  of  the Italian  macro-model  enabled  at the  impact  on  both 
Southern Italy and Italy as a whole to be assessed. 

The results  of  the different  Mid-Term  Evaluations  (see  Graph  1) should  be treated  with 
caution  regarding  their  comparability  since  the  methods  used  varied  across  countries. 
Nevertheless,  there  can  be  no  doubt  on  the  positive  growth  and  employment  effects  of 
Objective  1 programmes. Mainly  due to the impact  of  increased  productivity,  effects  on 
employment  are  usually  estimated  to  be  much  lower  than  those  on  growth.  Graph  1 
shows,  for  example,  in  Spain,  that  the  CSF  is  likely  to  increase  GDP  by  5.1% which 
means  an  additional  annual  growth  of  roughly  0,8%  average;  and  employment  by 2.4% 
by 1999 relative to the baseline position (i.e. the situation without CSF). 

The  challenge  for  future  evaluations  of  the  macroeconomic  impact  of  Structural  Funds 
will  be to make more use of  methodologies  which  represent  the state of  the art. Macro-
models should have integrated demand and supply sides, the latter allowing to distinguish 
the main categories of interventions  such as infrastructure,  human capital  and productive 
investment.  Main  variables  to  give  results  for  are  GDP,  investment,  employment, 
consumer  prices, budget  deficit,  imports  and  exports. The  most  interesting  additionality 
scenarios are EU funding  alone as well as EU funding  and national funding  together, both 
excluding  private  co-financing.  Account  should  also  be  taken  of  opportunity  costs  of 
public  spending,  i.e.  the effects  of  an  alternative  use of  EU  and  national  funds.  Finally, 
the sensitivity of results to changes in CSF spending and in economic policy can provide 
more concrete results in terms of policy conclusions. 

- 1 3-

Graph  1: Mid-Term Evaluation results -percentage deviations from  baseline in 1999 

% 

•  GDP 
•  employment 

Germany  Greece 
(Obj. 1) 

Spain 

Ireland 

Italy 
(Obj. 1) 

Portugal 

Note: 

The  employment  figure  for  Portugal  is  the  reduction  of  the  unemployment  rate  in  percentage 
points. 

Sources:  Ifo  Institut  (D),  KEPE  (EL),  Quasar  (E),  ESRI  (IRL),  Ministero  del  Tesoro-Nucleo  di 

Valutazione (I), CISEP (P). 

Assessing the macroeconomic impact in small areas 

For  small  areas,  specific  models  were  designed  to  estimate  the  long  term  impact  on 
growth, investment and employment. An interesting methodology was developed for the 
Belgian  Hainaut  region  (Objective  1),  notwithstanding  the  difficulties  in  collating 
regional data at a very disaggregated level (Box 4). 

Box 4: Hainaut (Obj.l)  : the HELM model 

HELM  (Hainaut Lead-in  Model)  is  an  econometric  model  assuming  a  key  relationship  between  output 
growth  and productivity growth.  This model  is extended to integrate different components  of productivity 
growth (research and development, physical  capital, skills). The results are presented for  three situations: 
the baseline situation  (A), the SPD without  constraint (B) (measuring the  additional  impact of the SPD), 
the SPD with constraints (C) (measuring the additional  impact but  integrating,  for  the  aid schemes, the 
rate of assistance and selection  criteria adopted).  The comparison between these situations  allows for  an 
estimate of the additional impact of the SPD against the baseline. 

The  results  of  the  simulations  run for  the  SPD  Mid-Term  Evaluation  relate  to  three  main  variables 
(investment, value-added, employment) in  the manufacturing  and  business services  sector.  According  to 
this model, the SPD  will generate  an  additional  investment growth  of  0,7% per  year for  the 1994-2005 
period  and  additional  employment growth  of  0,3% per  year,  amounting  to  5100  net jobs  in  1999  and 
15800 net jobs  in 2005. 

The massive  absorption  of funds  at the  end  of the period  will reduce  the  efficiency  of  implementing the 
SPD. Delays in execution may also have the effect that impacts are more visible at the  end  of the period 
and thereafter. The model shows that any increase in the rate of assistance will have only a moderate effect 
on investments, but these will produce  some beneficial (indirect) effects within the area. Therefore, the risk 
of  returning  to  slower  growth  compared  with  the  EU  average  after  the  completion  of  the  SPD  is 
significant. 

14 

b) Microeconomic assessment 

Assessing effectiveness 

The evaluators were asked to provide an analysis of the effectiveness  of the interventions. 
Most evaluations took a view on the extent to which the programmes were meeting their 
stated objectives15.  As part of  their work, they had to review the existing  indicators, and 
in some cases proposed a number of new or revived core and key indicators, which had to 
be agreed with the regional  authorities. This assessment  has often  required a substantial 
review  of the indicator systems, in terms of  both financial  results  and the monitoring of 
the agreed physical and impact indicators. 

A systematic approach to effectiveness  was adopted for the Irish CSF evaluation, using a 
microeconomic  model  to assess  whether the measures were designed  properly and were 
able to meet their objectives. This approach implies that Structural Funds cannot only be 
judged on spending but rather on the social benefits  they will bring to the economy. The 
methodology adopted by the evaluator aims to compare the effectiveness  of the measures 
in order to ensure an optimal  allocation of funds  (i.e. comparable value of money across 
the  programmes)  and  identify  best  practice.  A  further  more  detailed  description  is 
presented in Box  11. 

At a more operational level, evaluations should seek to measure the extent to which main 
programme objectives  are being met. In the case of  Merseyside,  the evaluation contains 
some  key  indicators  (SMEs,  land  improved,  floor  space),  including  output  data  and 
corresponding  results  of job creation  and private sector leverage.  Quantified  targets are 
available for 1999 and the Mid-Term Evaluation presented figures  measuring progress to 
date.  These  indicators  provided  in  an  aggregate  fashion  at  the  level  of  priorities. 
Subsequent  to  the  Mid-Term  Evaluation,  a  process  of  revision  of  the  Merseyside 
baseline, target and output data was initiated. 

Box  5: Assessing effectiveness through programme  indicators-the case of Mersey side-Objective  1 (1994-
1999) -actual outputs/results for  ERDF interventions 

^ ^^ 

Driver 

ERDF 
expenditure 

Total Gross Jobs 
(direct & indirect) 

Total Floorspace 

Total land 
mproved 

Total Private 
Investment 

1 
2 
3 
4 
5 
All 

15 

£m 

%of 
SPD 

000s 

28.8 
44.4 
15.4 
4.8 
22.1 

115.5 

21 
54 
42 
16 
26 

31 

5.4 
9.2 
0.8 
0.7 
3.0 
19.2 

% of 
SPD 
Target 

15 
142 
16 
64 
454 
39 

000m3 

(Hec) 

%of 
SPD 
Target 

% of 
SPD 
Target 

£m 

%of 
SPD 
Target 

114 
116 

1 
9 
240 

20 
70 
0 
15 
n/a 
24 

255 
17 

32 
210 
513 

52 
12 

n/a 
300 
73 

67 
76 
0 
8 
23 
176 

40 
90 

28 
n/a 
77 

Efficiency  issues  were  rarely  addressed,  being  one  of  the  most  difficult  aspects  of  evaluation. 
Analysing  efficiency  involves comparing  programme  inputs  (financial  resources)  with  outputs  (the 
goods and services  it provides) and results (the initial effects)  to estimate if the same benefits  could 
have  been  produced  using  fewer  inputs  or,  alternatively,  if  the  same  inputs  could  have  produced 
greater benefits. Discussion of efficiency  necessarily entails comparisons with various  counterfactual 
positions. The main difficulty  in this area is therefore the choice of appropriate benchmarks. 

- 1 5-

Note: 

Drivers  (or priorities) for  the  Merseyside  SPD  are  the following;  1)  inward  investment,  2) 
indigenous enterprise, 3) knowledge based measures, 4) cultural, media and leisure, 5) actions 
for  Merseyside. 

Source:  Mid- Term Evaluation report (Nov. 1996). 

Indicators  defined  at  OP  or  SPD  level  provide  a  basis  on  which  effectiveness  can  be 
monitored and assessecl. However, they do not in themselves allow definitive  conclusions 
to be drawn, particularly on impact, because of the difficulty  in establishing causation in 
some areas (i.e.: the indicators may be affected  by external factors). They often  need to be 
supplemented by qualitative analysis and by appropriate evaluation  work. 

Overview of main outputs and results 

Microeconomic  data  on  outputs  and  results  arising  from  the  evaluation  reports  are  not 
easy  to  present  in  an  aggregated  way  because  of  their  specificity  and  lack  of 
comparability across programmes. It is also recognised that the extent to which these data 
can  be quantified  at the Mid-Term  stage will  vary across both  the areas  of  interventions 
(infrastructure,  SMEs,  human  resources,  etc.  ...)  and  regions  concerned.  The  standard 
and coverage in the evaluation  reports is inevitably somewhat  variable as is the extent to 
which  key elements  such  as outputs, results  and changes  in  marked  disparities  could be 
quantified.  In Box  6,  some  achievements  relating  to  Objective  1 and 6 programmes  are 
presented. 

Basic infrastructure -  reducing disparities 

Basic infrastructure  is the area where the impact of Community action is most visible and 
measurable.  The  incidence  of  these  major  investments  is  sometimes  high  in  structural 
terms.  For  example,  in  Spain,  CSF  resources  supporting  transport,  communication  and 
energy networks represent  on average more than  25% of national  infrastructure.  Notable 
progress in reducing disparities with the rest of the Union has been made in most sectors 
(see box 6). 

Box 6: Important achievements in basic infrastructure. 

• 

• 

• 

Substantial  effort  is  continuing  to  support  the  development  of  the  strategic  road  networks  in  the 
Cohesion countries through, for  example, the improvement of four  major corridors in Ireland (reaching 
49% in 1996 against a target level set for  1999 of 53%) and the completion of the Athens-Thessaloniki 
motorway (400 km built to date). In Portugal, the construction of motorways and primary roads to date 
represents  74%  of  the  1999  target.  In  Spain,  the  stock  of  physical  infrastructure for  motorways 
increased by  13% between 1993 and  1995  (latest data available), benefiting  in particular  to the less 
developed regions like Andalusia. 

In  all  Objective  1 regions, major  investment has gone  into  the  telecommunication  sector, providing 
more  modern  systems  such  as  digital  exchanges  and  fibre  optic  links.  This  has  contributed  to  a 
significant reduction of disparities in provision  compared with the rest  of the  Union. In Portugal, the 
average  density  of  telephone  lines  (per  100  inhabitants)  is  aligning 
line  with  the  EU  level;  this 
objective  was  expected  to  be  attained  by  1999.  Similarly,  in  Italy,  the  rate  of  digitalisation  in  the 
Southern  regions has reached the same  level (85%) as that  of the  Centre North.  In Spain, digitalised 
networks  increased by 36% in absolute terms between 1993 and 1995. 

Progress  in energy  diversification, notably  the reduction  of  dependence  on  oil  has  also  been made. 
Deliveries from  the new natural gas distribution should begin in Greece. The high pressure natural gas 
pipeline  (513 km)  has  already been  completed  and  another  major project  (1000 km  of  low pressure 
natural gas network) is nearing completion (84%). In Portugal, more than half of the gas network (600 
km)  had  been  laid  by  1996  and  this  will  account,  by  1999, for  7.5%  of  total  energy  consumption. 
Greece, Spain and Portugal have all seen significant reductions in their energy use relative to GDP in 
_^ 
recent years. 

- 1 6-

As  regards the environmental sector,  support  is being provided  to  improve systems  of water supply 
and  increase the  capacity  of  wastewater  treatment facilities.  In  Greece,  water supply,  sewerage  and 
wastewater treatment systems, serving about 5 million inhabitants (half of the population)  have already 
been completed. Investment has also gone  into smaller  Objective  1 regions, like  the  Northern Ireland 
Sub-programme where the construction of water supply and waste water facilities  have generated some 
2270 jobs  in  the  area.  Significant  environmental  benefits  have  also  arisen from  a  whole  range  of 
specific  interventions,  including the  clearance  of  major  contaminated  industrial  sites,  environmental 
audits ofSMEs,  assistance in the  development  ofeco  products  and reduction  of emissions  due to the 
diversification of energy supply. 

Productive environment -  supporting employment 

In  terms  of  the  productive  sector,  structural  assistance  has  been  directed  at  improving 
conditions for existing firms, thereby encouraging  new  activities. It is  largely  recognised 
that support, especially for SMEs is a major driver for job creation, and this contributes to 
bridging structural gaps in productivity and income levels. Some examples may highlight 
recent trends in Structural Funds impacts in this area (see box 7). 

Box 7: Some achievements in the productive environment 

•  Major  aid  schemes  have  emphasised job  creation, partly  offsetting  the  negative  consequences  of 

industrial restructuring. 

In Hainaut, almost all funds  available to aid schemes, notably SME schemes had been used before the 
end of the period; it has been estimated that job  creation targets for  those measures have already been 
exceeded to date (with more than 2200 jobs  created compared with 1800 jobs  expected). 

In  Germany (Obj.  I),  Structural Funds  assistance contributed actively to support  employment  in, for 
example,  Sachsen-Anhalt  where  18.500 jobs  were  assisted  in  the  chemical  industry  and  a  further 
20.000jobs  in SMEs. 

In Italy, the industry Programme is likely to have created or safeguarded more than  75.000 jobs  over 
the 1994-96 period. 

• 

• 

In the tourism sector,  increased benefits will derive from  new  capacity  being  made  available  in, for 
example,  Sachsen  (additional  capacity  of  11500  new  hotel  beds)  and  in  Ireland,  where  18  large 
projects  which have already been completed providing  additional facilities  to increase the quality and 
standard  of  tourist  attractions. New  employment  opportunities  will  be generated  by  these Structural 
Funds assisted improvements 

Physical  regeneration  of  industrial  sites  will  also  contribute  to  the  improvement  of  the  economic 
environment  and  attract  new  business.  This was  a  main priority  in  some  German  Lander:  in  East 
Berlin, for  example,  where  51000  m2  were  refurbished, creating potentially  600 jobs  or  in Sachsen 
where 620 ha of land were transformed into business parks.  In Nord-Pas de Calais, approximately 40-
45% of industrial land located in the Objective 1 area will have been improved. 

Human resources: quantifying impacts 

Specific  evaluations  conducted  in the  area of  human  resources  included,  as part  of  their 
workplan,  an  impact  analysis  of  training  measures  on  beneficiaries.  The  main  results of 
these studies are presented below in box 8. 

Box 8: Impact of training measures on beneficiaries 

In Spain the evaluator made a comparative analysis of the placement  of trainees by target group, using the 
INEM (The National Employment Institute) database as a quasi  control group.  For  this  analysis only the 
occupational training measures were taken into account. The main conclusions are in line with the results 
of the 1989-93 ex-post  evaluation:  in terms  of gross  impact, the placement  rate  of trainees  is on average 
20%  higher  than  the  control  group;  placement  is  higher for  young  people,  men  and  those  holding  a 
secondary education diploma. However, if net impact is estimated, taking  into account  the characteristics 
of the trainees, the differences vary substantially. For example, in terms of age, trainees over 45 years  old 
perform  32% better in finding a job  than those in the control group.  Training also seems  to increase the 

- 1 7-

chances of women finding jobs:  the net impact is 27.1%,  whilst for  men it is 14.9%. If qualification levels 
are  taken  into  account,  the  effect  of  training on  those holding  a primary  education  diploma  is striking: 
going through a training action increases their chances of finding  a job  to 48.8%. 

Germany: impact of the QfQsj (Gerneinsçhafi Initiative -  Qst) 

In Germany, it has been estimated that the Bund programme  in support  ofGIOst  (i.e. apprentice schools 
for  the young)  has brought about a reduction in youth  unemployment of three percentage points  in 1995. 
At  Lander level,  very few  placement  indicators have so far  been  collected. The combination  of training 
measures with grants generally leads to higher performance, but this is the exception rather than the rule. 
Some  regional  results  can  be  mentioned. For  Berlin-Ost, the  evaluator  concluded  that, for  continuous 
training and unemployed training taken together, 60% of trainees had found  or kept jobs  after six months. 
Training programmes  in Meckelenburg-Vorpommern achieved placement  rates of 40-45%  (21-32% in the 
first  labour market). A programme for  business starters, which basically  tries to help people  ensure that 
they  start  a  viable  business,  resulted  in  about  50%  of  actual  "successful"  starters.  Various continued 
training  and  education  programmes  for  employees  (Objective  4  type  measures)  in  Mecklenburg-
Vorpommern, Thuringen and Sachsen-Anhalt resulted in increases of 35 to  77% in the perception  of job-
security by participants. 

Portugal: impact on the education system 

Impacts in terms of strengthening the Portuguese education system (through PRODEP) have been found  to 
be linked to the diversification of education pathways  and to a focus  on both quantity  and quality issues. 
Training  teachers and other agents appears to be one of the major contributions of the programme,  both 
at  1st  and  2nd  level  (representing  49.3%  of  teaching  staff)  and  at  university  level  (provision  of 
scholarships for  Masters and PhD programmes). As regards the expansion and diversification of education 
/ training pathways,  major changes have included enlarging secondary education, bringing it closer to the 
workplace and increasing guidance  and counselling in schools. In fact,  support  to middle level technician 
training has set up a whole sub-system providing technical skills directly relevant to the labour market and 
an alternative to mainstream education. 

Agriculture and rural development 

Since agriculture and rural development constitute separate programmes at national level, 
they have been  assessed  separately. In the case of  SPDs  and  the  agricultural  sections of 
regional programmes, evaluation formed part of a more general evaluation. 

Some  Member  States  such  as  Ireland  carried  out  detailed  evaluations  of  the  main 
priorities  of  the  programmes  (the  food  industry,  establishment  of  young  farmers, 
compensatory  allowances,  equal  opportunities,  agricultural  training,  etc.). In  such  cases 
evaluation proved a useful  tool for obtaining better knowledge of the situation, analysing 
problems and finding the best ways of solving them. 

Box  9: Evaluation  of  agricultural  interventions -  the  case  of  PAMAF  (modernisation  of agricultural 
structures in Portugal 

The evaluation report found  that financial and physical  implementation had  been slow  in getting started, 
mainly  because  of  delays  in  drafting  the  relevant  legislation  and  management  changes.  However,  the 
initial delay has  been made  up  and  the assessor is  optimistic about  the  chances  of some  adjustments in 
management and the allocation of funding  between measures permitting programme  implementation to get 
back on schedule so that the original goals can be achieved. 

The operational  assistance  measures  concern primarily  support for  investment  on  agricultural holdings 
where the rate of implementation is high, infrastructure such as irrigation, rural roads,  drainage and soil 
conservation, forestry,  the processing  and marketing of agricultural and forestry  products,  compensatory 
allowances in less-favoured  areas, training  and research. Measures  of  these  types  are  intended to make 
agricultural holdings more competitive. Most of them represent a continuation  of the earlier programme, 

hich had proved successful. They also form part  of structural policy  at Union level. 

The assessor stressed in particular  that measures were to some extent scattered, which  affected their real 
impcct as  well  as  management  and  implementation. He  suggested  that  the funding  available  should  be 
rea!" seated to the measures regarded as priorities, particularly aid to agricultural holdings and irrigation, 

- 1 8-

and  insisted on  the  need  to  improve  vocational  training.  He  also  noted  the  effort  made  to  concentrate 
support on the private sector and a non-uniform regional distribution of projects.  The report notes some of 
the  results  of  the  agriculture programme  in  1994-96,  including  the  financing  of  some  20  000 projects, 
supportfor  11 500 agricultural holdings and the establishment of 2 800 young farmers. 

Fisheries 

The  FTFG provides  support  in  five  main  areas  of  assistance:  adjustment  of  the  fishing 
effort  (26%  of  the  budget),  renovation  and  modernisation  of  the  fleet  (24%),  the 
processing  and marketing of products  (23%), the development  of  aquaculture  (11%) and 
facilities  in  fishing  ports  (7%).  It  currently  includes  31  programmes,  of  which  17  are 
under Objective  1, 12 under Objective 5(a) and two under Objective  6. 65% of the budget 
goes to Objective  1 regions. 

The Mid-Term Evaluations  of  these programmes  provided  interesting  information  about 
the  effectiveness  of  assistance  and  reprogramming  (the  reallocation  of  funds  between 
areas  of  assistance  or  between  measures).  They  revealed  some  shortcomings  in 
monitoring, particularly in the provision of socio-economic data. In some cases proposals 
to  improve  the  indicators  were  implemented  and  initial  analyses  of  the  impact  on 
employment carried out. 

Box 10: An example: the Spanish Objective 1 prqgramme 

The operational programme for fisheries,  which has funding  of about ECU 1 billion, is one of the priorities 
in  the  CSF.  The  measures  in  the programme  are  mainly  concerned  with  adjusting  the  fishing  effort, 
modernising  the  fleet,  the  processing  of  products,  aquaculture  and  facilities 
in  fishing  ports.  The 
evaluation report found  that the  average implementation rate  was satisfactory  at  57%  but varied widely 
depending  on  the  area of  intervention: it was  low for  modernisation of the fleet  and  aquaculture (10%-
20%) and high for port facilities  and processing  of products  (60%-100%). 

Effectiveness  was  lowest  in  aquaculture  (2%):  this  measure  has  been  affected  by  the  cumbersome 
administrative  procedures  required  to  implement  projects,  poor  returns  which  discourage  potential 
investors  and  environmental  constraints  on project  selection.  Effectiveness  was  highest  in  the  case  of 
processing  (65%),  where beneficiaries often achieve high growth rates  and so good  investment capacity. 
The assessor proposes  to  reallocate funds  from  adjustment  of  the  fishing  effort  and  aquaculture  to  the 
modernisation of vessels and the processing  of products. 

A  survey  of  beneficiaries  showed  that  35%  of  the projects  receiving  grants  under  the programme  had 
created or preserved jobs  (initial figures suggest  a total  of  780 jobs),  principally  in the  area  of product 
processing.  Proposals have also been made to improve the indictors of achievement and results, to reduce 
the number of specific indicators for  aquaculture, establish impact indicators for  the fleet and employment 
indicators for  all  areas  of  intervention.  The  assessor  noted  that  aquaculture  paid  strict  attention  to 
environmental protection  and recommended that greater account should be taken of this principle  in other 
areas (processing of products  and port facilities). 

c) Quality of delivery mechanisms 

Mid-Term  Evaluations  were  also  required  to  assess  the  way  in  which  the  Funds  are 
managed, particularly in terms of monitoring systems and project selection criteria. 

Monitoring and indicators systems 

Compared  with  the  previous  programming  period,  the  current  programmes  show 
significant  developments  in  the  use of  quantitative  indicators  as  a  means  of  monitoring 
their  implementation,  assessing  their  impact  and  determining  progress  towards  meeting 
overall objectives. 

-19 

The Commission  prepared  a Common Guide for  Monitoring and Evaluation with  a view 
to ensuring  consistency  of  approach  and uniform  standards16.  This  has  been  a primary 
impetus in bringing about these improvements. However, experience  across programmes 
is mixed. In a number of  programmes, the information  available by way of  indicators is 
often  inadequate in terms of measuring performance  or verifying  if  satisfactory  progress 
is being achieved. Other principal shortcomings relate to the determination of appropriate 
and  practical  targets  in  relation  to  the  measures  in  question  as  well  as  the  lack  of  a 
consistent,  regular  system  of  data  collection  relating  to  physical  outputs,  results  or 
impacts. 

Notwithstanding  many  improvements,  the  definition  and  quantification  of  appropriate 
indicators  remains  an  unsatisfactory  area  of  programme  management.  Whilst  further 
work is required on indicator systems to improve monitoring and assist evaluation work, 
this  should  begin  by  identifying  examples  of  good  practice  in  this  area.  Interesting 
attempts  at  building  appropriate  management  information  systems  can  be  found  in  a 
number of programmes. Some examples are provided in Box  11. 

Box 11: Enhancing the monitoring systems 

A well managed information system  is  a necessary  (although not sufficient)  condition for  supporting  the 
evaluation  process.  The  case  of  Nord-Pas-de  Calais  (Objective  1)  is  an  example  of  an  integrated 
monitoring system, which couples financial and physical  indicators at project  level. In Northern  Ireland, 
an important part  of the Mid-Term Evaluation  was an intensive scrutiny  ofNIQUID  (the database  of all 
the indicators agreed for  the SPD) to transform it into an effective management information system for  the 
SPD.  The evaluator proposed  monitoring on a regular basis the progress  of 52 indicators, rather than the 
330  indicators identified in  the SPD.  In  addition, for  each sub-programme,  a set  of  key  indicators was 
developed to capture its specific impact. 

Significant work has also been done in Italy, where a standardised system of indicators was established for 
the ESF  interventions. In Portugal,  indicators have  improved in almost all areas of  intervention, even if 
there is still scope for further  improvements in the quality of data. 

Project selection criteria 

The first two or three years of programme implementation  have, in some cases, seen the 
development  of  relatively  sophisticated  scoring  systems  for  co-financed  projects.  The 
systems  put  in  place  (e.g.  in  Merseyside)  take  account  of  a  wide  range  of  factors 
simultaneously and in addition to assessing eligibility they look at selection criteria both 
at  priority  and measure  level.  For  example,  priority  criteria  which  are  common  to  all 
projects  include  project  outputs,  value  for  money,  private  sector  leverage  and  more 
recently environmental impact. 

Many Mid-Term Evaluations have questioned the effectiveness  of such systems. Having 
transparent and objective criteria, which in itself is considered as good practice, may not 
be a sufficient  condition to ensure that the best projects are selected. Further checks will 

16 

The  Commission  Guide  distinguishes  between  indicators  of  output,  results  and  impact.  Output 
indicators refer to financial and physical implementation (e.g. number of kilometres of road built and 
cost,  number  of  training  courses  provided.  Results  indicators  refer  to  the  immediate  effects  of  an 
intervention  (for  example,  time  savings  in  a  road  project;  the  number  of  people  who  successfully 
complete  a  training  course).  Impact  indicators  refer  to  the  outcomes  of  the  interventions.  A 
distinction can be made between the specific or immediate impact of an intervention (e.g. number of 
people placed into jobs) and the general or final impact, i.e. the socio-economic effects  (e.g. increase 
in employment or GDP). 

- 2 0-

have to be made, for example on the degree of credibility attached to the information and 
output forecast provided by the applicants. 

Box  12: Merseyside -  scoring systems for  ESF7ERDFprojects 

In the Merseyside SPD, the system put  in place for  selecting projects  has three different aspects: eligibility 
or  "core criteria", priority  selection criteria common to all projects  and measure selection criteria (which 
may be different from  the priority criteria). 

The  aim,was  to  develop  sets  of  criteria  which  were  understandable,  transparent  and  which  could  be 
applied to a large number of heterogeneous projects. Main priority  criteria include the expected outputs of 
the projects;  value for  money,  the  nature  of  beneficiaries  (for  ESF);  labour  market  issues;  evidence of 
partnership and linkage; private sector leverage; and environmental impact. 

2.3. Utilisation of evaluation results 

As  described  in  the  previous  section,  Mid-Term  Evaluations  represent  an  important 
source  of  knowledge  for programme  managers  and decision  makers. They  inform them 
on key issues such as the relevance of the aims and objectives of their interventions and 
the effectiveness  of  the wider economic  effects  on the area concerned.  An  authoritative 
evaluation should be supported by rigourous analysis and be sufficiently  operational to be 
fully utilised for policy purposes. 

However,  the  quality  of  the  evaluation  results  does  not  guarantee  that  they  will 
necessarily be utilised. In other words, the evaluations cannot be considered as the only 
input  for  carrying  out  the  Mid-Term  Review.  Institutions  and  policy  actors  are 
continuously involved in exchanges of information. Other factors relating to the political 
context  of  the  macroeconomic  framework  may  also  influence  directly  the  final 
reprogramming decisions. Bearing in mind this wider framework, three different  levels of 
utilisation should be distinguished in the Mid-Term Evaluation process. 

The first level of utilisation involves evaluation as a feedback  mechanism relating to the 
effectiveness  of  policy  measures.  Evaluation  results  and  recommendations  have,  in  a 
significant  number of cases, been largely followed and incorporated in the changes made 
to  the  programmes.  The  best  example  is  the  Irish  CSF  Mid-Term  Evaluation  which 
produced credible  and useful  conclusions  leading  to  the  reallocation  of  funds  between 
measures or sub-programmes (see box 12). 

Secondly, evaluation played a major role in providing new sources of information  or on 
going advice to programme managers and policy makers . For example, the evaluation of 
the  Hainaut  Objective  1 programme,  which  included  a specific  macroeconomic  model 
(see  Box  3),  was  also  recognised  as  useful  for  informing  the  reprogramming  actions. 
Conclusions  here  opened  longer-term  perspectives  on  the  convergence  trend  of  the 
regional economy. Other evaluations, having a narrower focus, stressed the importance of 
certain  critical  aspects  of  programme  management,  rather  than  being  exhaustive  with 
regard to the whole implementation cycle. 

Finally,  the  experience  also  showed  that  substantial  input  from  the  programme's 
partnerships  is  key  to  the  success  and  effectiveness  of  the  evaluation  process.  One 
important  result  is  that  decision  makers  were  involved  in  the  discussion  of  evaluation 
issues  and  a  number  of  lessons  learned  in  terms  of  improving  their  interventions. 
Although  this  learning  process  was  not  always  formally  structured,  it  had a  significant 
impact on the final outcome of the Mid-Term Reviews. 

- 2 1-

III.  MAIN OUTCOMES AND ADJUSTMENTS 

The  evaluations  informed  the  Mid-Term  Review  process  by  providing  an  overall 
assessment  of  what had been  achieved during the first  half  of  the  programming period. 
Taking stock of these elements, this part of the report assesses the main outcomes of this 
process priority in terms of  the adjustments  made to programmes. Account  is also taken 
of the extent to which priorities laid down by the Commission have been incorporated in 
the programmes. 

3.1. Link between evaluation and reprogramming 

The  evaluations  were  delivered  within  the  agreed  time  period  enabling  the  results  to 
influence  the  Mid-Term  Reviews  carried  out  between  1997  and  199817.  A  special 
reference  should be made to the three new  Member States.  Although  their programmes 
started  a  year  later  than  for  those  of  the  other  Member  States,  all  three  made  formal 
arrangements to launch Mid-Term Evaluations. The Commission  promoted an exchange 
of  experience  between  evaluators  which  brought  new  ideas  into  the  conduct  of  the 
evaluations. A key feature of this process was that no significant  adjustments were made 
prior to the evaluations and the formal review of programmes. 

Evaluations have, in general, proved useful in addressing major issues or highlighting the 
kinds of decision to be made. Their ability to do so has depended on the intrinsic quality 
of the analysis as well as on the operational focus of their recommendations. 

In the case of rural development, interim evaluations produced specific  recommendations 
and proposals as requested both by the Commission and the managing authorities. These 
included  the  reallocation  of  resources.  In  Spain  and  Germany  they  coincided  with  the 
proposals  by  the  responsible  authorities  since  the  grounds  (progress  of  measures  as 
shown by the monitoring data) were the same. In other cases, only some were taken into 
consideration  (Portugal). Sometimes, evaluations were carried out after programmes had 
been  amended  and  it  was  subsequently  found  that  there  had  been  no  contradictions 
(France).  In  Italy  the  assessor  concentrated  on  analyses  of  problems  and  obstacles  to 
implementation  and those  responsible  for  the  programmes  proposed  the  reallocation  of 
resources.  In  the  case  of  Ireland,  it  has  already  been  noted  that  the  assessor's 
recommendations  concerning  the  strengthening  of  certain  measures  could  not  be 
implemented because of lack of finance. 

The example of Ireland and Merseyside 

Two examples  serve to highlight  the entire Mid-Term Review  process, from the formal 
evaluation exercise  to the implementation  of  the financial  reallocation  decisions  . These 
relate  to  two  different  levels  of  intervention:  a  large  CSF  (Ireland)  and  an  SPD 
(Merseyside) (see Box  13 and 14). 

* ' 

There  has been some delay  in the negotiations  for  the Mid Term  Review  in France-Obj. 1 (especially 
la  Réunion)  and  for  Greece  (reprogramming  of  ERDF  interventions;  ESF  modifications  scheduled 
for  the  second  half  of  '98).  In  Italy,  a number  of  reprogramming  actions  were  decided  in  April  '98 
and the Mid Term Review  will be finalised  in October,  on the basis of actual  progress  of some slow 
spending  programmes. 

- 2 2-

Box  13: Mid-Term  Review  (I): the case of the Irish  CSF 

The report states  that  the Irish  CSF represents a notable success  and  that  the funds  have  been deployed 
effectively  to  support  and  enhance  what  has  been  a  remarkable  economic  recovery.  Under  the  CSF, 
medium term planning  of public  expenditure has  allowed more systematic  and  effective programming  in 
many areas. 

The CSF evaluation raised questions about the value of devoting public  resources to a number of specific 
measures, notably free  or subsidised provision  of services for  which firms or individuals would be willing 
to pay.  A  core assumption here was that markets are the most effective and  efficient means  of achieving 
economic  and  social  objectives  so  that public  intervention  is justifiable  only  where  the  market  is  not 
working properly  and the intervention in question creates no economic distortions of its own. In the light of 
these  theoretical  assumptions, four  situations  were  identified, where  in  the  opinion  of  the  evaluators, 
public  intervention in a market economy could be justified: 

the provision  of public  goods  and  services  (such  as  transport  infrastructure)  which  would  not  be 
provided  in the absence of public intervention; 

the  introduction of  corrective subsidies  designed to  alter the price  of goods  and services  where the 
market price does not adequately reflect their wider social benefit (e.g. the cost of loans to SMEs) 

the  management  of  targeted  schemes  aimed  at  changing  behaviour  through  correcting  a  lack  of 
knowledge,  awareness  or  understanding  (e.g.  introducing  business  owners  or  employees  to  new 
technology) 

the redistribution of wealth through subsidies or welfare benefits in pursuit  of broadly social aims 

The CSF evaluation categorised the  166 CSF measures using the above headings. For each category, one 
or two  "anchor measures "- measures which  were  large scale  or which  addressed  long-established and 
well-understood elements of public policy -  were selected and used as comparators for  the other measures 
in that same category. On the basis of these comparisons, decisions were taken as to which measures were 
successful and which were not. 

Some 150 recommendations were in turn produced providing  how CSF resources might be deployed more 
effectively, with particular focus  on  those  measures poorly  targeted  or  having  undesirable  effects. The 
evaluation pointed  to  a  clear  case for  more public  spending  on physical  infrastructure,  especially non-
urban roads.  In some  demand-led  measures it was suggested that spending  targets should  be revised in 
line with actual evolution, for  example a reduction of the grant to the Food Industry Sub-programme. 

It  was also recognised, in discussions within the Monitoring Committees,  that  more  emphasis should be 
placed  on youth  and long-term unemployment and that allocations to specific measures for  education and 
the disadvantaged should be increased. 

The CSF evaluation also contains a number of detailed recommendations  in relation  to the management 
and implementation of specific measures. It is argued, for  example, that there is insufficient competition in 
the provision  of  subsidised  services  due  to  the predominant  position  of  state  agencies  and  the  lack  of 
competition. Regarding  monitoring,  the report suggests further  improvements to performance  indicators, 
while  admitting  that  the  current  CSF  represents  the  most  advanced  experiment  in  monitoring  and 
indicators systems. 

The outcome  of  the  Mid-Term  Review  largely followed  these  recommendations.  Accordingly  .the  shifts 
were  mainly  away from  the  productive  sector  towards  human  resources  development  and  economic 
infrastructure. Financial  reallocations  amounted  to  163 Mecu, focusing  on  RTD  (37  Mecu), Transport 
Infrastructure  (36  Mecu)  and  a  range  of  measures  dealing  with  early  school  leavers  and  other 
marginalised groups (46 Mecu). 

Box  14: Mid-Term  Review (II) : the case of Merseyside  (Obj.l) 

Structural Funds assistance to Merseyside comprises 816 MECU delivered through the Merseyside Single 
Programme's  five  "Drivers for  Change":  Inward  investment  and  the  corporate  sector;  Indigenous 
enterprise and  local  business; Knowledge-based  industry and advanced  technology;  Culture, media and 
leisure  industries;  Action for  the  people  of  Merseyside.  The  latter  receives  the  largest  proportion  of 
Programme funds  (more  than  two fifths).  The  inward  investment  and  local  business  drivers received, 
respectively, a quarter and a fifth of Programme funds. 

23 

The advent of the Single Programme coincided with a cyclical upturn in the  UK economy characterised by 
falling  unemployment  and  economic growth,  both  at rates  higher  than  the  EU  average.  Merseyside has 
shared in this upturn with the result that Structural Funds receipts are being deployed in a situation more 
favourable  than that envisaged when the Single Programme was being developed.  Although improvements 
in terms of unemployment have been small relative to the UK as a whole, they should be seen in the context 
of a recent picture  of decline in the principal social and economic indicators. 

The  Mid  Term Evaluation  assessed  the  current  and potential  impact  of  the  Single  Programme  on  the 
people  and  business  life  of  Merseyside, commented  critically  on  existing  monitoring  arrangements and 
made recommendations as to how the Programme should proceed  over the remainder of the programming 
period.  Desk  research  was  supplemented  with  a  review  of  100  separate  projects  along  with extensive 
survey  and  interview  research focusing  on  Programme  beneficiaries.  The  volume  and  quality  of  the 
evaluation research was significant. It compensated significantly for  resource constraints which limited the 
amount of in-depth analysis possible. 

The evaluator  made  a  series  of  detailed  and  reasoned  recommendations,  proposing  in particular  that 
future  monitoring  and  evaluation work should focus  on  the  larger projects  likely  to  deliver the greatest 
impact as  well  as  on  issues such  as displacement  and sustainability.  It  was  also  recommended  that the 
existing  set  of  indicators  be  expanded  to  enhance  the  assessment  of  Programme  impact.  Proposed 
indicators included, inter alia, survival rates for  SMEs and their use of new technology. In addition, it was 
suggested that  insufficient focus  was being given  to synergies  within the five Programme Drivers and to 
themes which cut across several measures. 

Reprogramming changes at the Mid-Term Review stage  broadly followed  the  evaluator's proposals  and 
were much  in  line  with  the  Commission's  Regional  Policy  Priorities  to  the  end  of  1999. A  total  of 97 
MECU was reallocated, primarily to measures most likely to lead to direct job  creation (SMEs,  technology 
and  social  inclusion).  New  measures  were  created for  fish  processing  and for  the  development  and 
application  of  communications  technologies.  The  Commission  also  set  a  deadline for  a comprehensive 
review of baseline data and quantified targets, with particular regard to net employment impacts. 

3.2. 

Overall reallocations of Structural Funds 

Under Objectives  1  and 6, all CSFs, SPDs and other forms of intervention were subject to 
a  Mid-Term  Review.  The  only  exception  was  the  new  German  Lander,  where 
adjustments to the programmes are being made on an on-going basis. All  these reviews 
led  to  financial  reallocations  of  varying  significance  in  order  to  improve  the  overall 
efficiency  of the interventions. 

Due to the financial  amounts involved, the scope of these changes is more wide-ranging 
for the larger and more complex CSFs than the SPDs. But it is not directly proportionate 
to  the  success  or  failure  of  the  programmes  concerned.  Even  a  highly  successful 
programme (e.g. the Irish CSF) can be improved and necessary corrections made without 
altering the main priorities and objectives. 

The  scope  for  reallocations  varied  widely  among  programmes.  Financial  progress  has 
been  the  general  focus  for  these  actions,  especially  with  regard  to  the  under-spending 
programmes.  In  addition,  the  reallocations  also  reflect  the  degree  to  which  the  EU 
priorities  have  been  taken  into  account  (see  4.3).  In general,  limited  adjustments (fine-
tuning) were made to the programmes without affecting  strategic priorities. In some cases 
(e.g.  Italy-Obj.l,  Portugal  CSF),  these  modifications  involved  a  transfer  of  resources 
from  the  slower  spending  programmes  or sub-programmes  to  the  faster  spending ones, 
with  due  consideration  given  to  other  qualitative  criteria.  In  most  cases,  however, 
modifications  were  made  within  programmes  and  between  measures.  Additional 
..  uurces from the deflator  were also used for refocusing  certain  actions, particularly in 
favour of employment creation. Graph 4 indicates the extent of the transfers of resources 
for each Member State. 

24-

The  largest  modifications  were made to the Objective  1 CSF in Italy  (around  700 Mecu, 
including  non-programmed  resources  and  CSF  indexation  for  1998), followed  by  Spain 
(more than 600 Mecu) and Greece (400 Mecu only for the national programmes). Despite 
the large financial  amounts involved, these only represent between 2% (Spain) and 5% of 
the  total  Structural  Funds  allocations.  In  Ireland,  financial  transfers  amounted  to 
160 Mecu  of  Structural  Funds, or 3% of EU  Structural  Funds  for  the  1994-99 period. In 
Northern  Ireland,  no  transfer  of  funds  between  sub-programmes  was  made,  aside  from 
the  allocation  of  resources  deriving  from  the  deflator.  In Finland  and  Sweden,  the most 
significant  changes  were  made  through  the  merger  of  a number  of  measures  in  order to 
simplify  the programme's structure. 

Conversely,  in  some  relatively  small  programmes,  significant  changes  were  made,  in 
relative terms, i.e. as a share of total Structural  allocations. In the Belgian  Hainaut, about 
19% of  available  resources  were shifted  particularly  towards  the  SME aid  schemes. The 
case  of  Flevoland,  despite  a relatively  high  amount  of  Funds  transfered  (13%)  involved 
several  minor  changes  in  the  SPD  measures,  without  affecting  its  overall  aims  and 
objectives. 

It  is  somewhat  difficult  to  present  a  comprehensive  overview  of  the  nature  of  these 
adjustments.  More  qualitative  insights  relating  to  some  key  EU  priorities  only  will 
therefore be presented in the next section. 

Graph  4:  Mid-Term  Review  Reallocation  of  Structural  Funds  by  Member  State  (in 
Mecu) 

Mecu 

D Arhount in Mecu  •  Percentage of S.F allocation 

1 

2 
3 
4 

of  which  Nord-Pas-de-Calais  (19,6  Mecus),  Corse  (10,1  Mecus),  Guyane  (36,1  Mecus)  and 
Martinique (22,4 Mecus). 
no Mid-Term Review 
only ERDF reallocation 
of which Northern Ireland  (69,8 Mecus), Merseyside (97,0) and Highlands and Islands (21 Mecu). 

- 2 5-

3.3. Inclusion of EU priorities 

The extent  to which  the EU priorities  laid down  in the Commission's  Guidelines  for the 
Mid-Term  Review  have  been  addressed  is  not  always  easy  to  assess.  New  measures 
added  or  existing  measures  receiving  additional  funding  may  provide  some  indication 
however. 

Basic infrastructure -  reinforcing TENs 

Reducing  disparities  in  infrastructure  endowment  in  the  areas  of  transport,  energy, 
telecommunications  and environment  has been recognised as a long term commitment of 
Structural  Funds. This is reflected,  in particular,  in the share of total  funding  (more than 
30%) allocated to this area in the current Objective  1  programming documents  1994-99. 

Although  still  a major  priority  for  most  regions,  no  significant  resource  was  devoted  to 
funding  new  infrastructure.  In  general,  the  emphasis  was  placed  on  specific  TENs 
projects with the aim of establishing efficient  connections and systems, particularly in the 
area of transport (box  15). 

Box 15: Airport programme -  Italy Objective 1 

The  10  Mecu  airport programme,  recently  approved  by  the  Commission,  aims  to  improve  or  upgrade 
existing airports in the South of Italy. Following the Mid-Term Review, it was agreed to provide  additional 
funding  (50 Mecu) to this programme,  exclusively for  TENs projects,  in order to further  enhance regional 
endowments in this area. 

Productive environment -  a strengthened focus  on employment 

to  increase  the 
The  results  of  the  Mid-Term  Reviews  show  that  there  has  been  scope 
focus  on  employment  through  a  variety  of  actions  (reinforcement  of  aid  schemes, 
Territorial Employment Pacts, local development  initiatives, etc.). A number of decisions 
already  taken  (e.g.  to  include  Territorial  Employment  Pacts  within  the  mainstream 
programmes,  Box  16)  reveal  a  political  aspiration  and  commitment  to  combat 
unemployment  more effectively.  The increased  focus  on the employment  generating role 
of projects,  particularly  SME projects,  is also in  line with  the direction  of  policy  within 
most Member States. This is reflected in the CSFs and SPDs. 

In  this  context,  the  Mid-Term  Review  has  questioned  the  effectiveness  of  policy 
instruments  in  terms  of  supporting  job  creation  and  responding  to  the  need  for  more 
active employment policies through human resources and education measures. At a more 
operational  level, emphasis  was  also  given  to  the  measurement  of  employment  impacts 
(e.g. UK regions, Nord-Pas de Calais) and to take better account of employment issues in 
project  selection criteria (e.g. Italy). 

Box 16: Territorial Employment Pacts 

In  three Member States  (Spain, Greece, Italy), the Monitoring  Committees allocated CSF resources  to a 
programme  or  a  specific  measure  to  finance  action plans for  Territorial Employment  Pacts.  In  Spain 
(Objective I),  priority  was given to five pacts  which should receive 55 Mecu of EU funding  (deriving from 
the CSF deflator), for  integration into a new multi-regional programme.  In Greece, it is envisaged that the 
resources arising from  the  1997 and  1998 deflator will  be  allocated  to specific  sub-programmes  within 
regional programmes  which  are  currently  under review.  In  Italy,  national  authorities proposed,  in  the 
context of the Mid-Term Review,  a multi-regional and multi-fund programme  of 280  Mecu,  of which 140 
Mecu from Structural Funds will provide  assistance to 9 pacts. 

26-

National aid schemes, which are being cofinanced out of Structural Funds, have also been 
reviewed  in  order to  increase  their focus  on  employment  creation  as  well  as  on  SMEs 
(see Box 17). 

Box  17: SMEs  : the case of the Greek CSF 

The  Greek  authorities  have  refocused  their  Industry  Operational  Programme  towards  SMEs,  with 
additional funding  of 136 Mecu.  The national aid scheme has been modified in line with the Commission's 
guidelines  by  increasing  support  to  business services  and  by providing  indirect  assistance  rather  than 
direct grants.  Intermediary  bodies  have  been  set  up  to  manage  the  SME  actions  and  projects  will  be 
selected on a competitive basis. A prior  appraisal of the SME projects  to be funded  (about  1450 projects 
over 4200 received) was also undertaken. 

Environment and sustainable development 

Environment,  as an element of sustainable development  is one of the wider EU priorities 
most  frequently  addressed  in  the context  of  the  Mid-Term  Review.  The  pursuit  of  this 
objective  is  considered  both  through  the  introduction  of  specific  measures  and  more 
importantly, through horizontal  integration  with other programme  priorities.  Because of 
objective difficulties  (lack of  indicators, methodological  aspects) the results of  the Mid-
Term  Review  could  not  in  all  cases  lead  to  direct  operational  conclusions  for  the 
adjustment  of  programmes.  Specific  attention  was  drawn  to  a  lack  of  systematic 
environmental impact assessment of projects and to the gains in project quality that could 
be  secured  by  systematising  such  assessments.  The  way  has  been  prepared  for 
improvements in environmental  protection and general project selection procedures over 
the new programming period (box 18). 

Box 18: Mainstreaming environment 

Environment is a  long-standing  concern  of most Member States.  For  the Structural  Funds  to contribute 
properly to environmental protection,  they must take account of this issue in all project  decisions, so as to 
minimise damage to the environment and to maximise positive  benefits, by for  example giving preference 
to  projects  with positive  environmental  impact  (preventive  action,  eco-products,  ...).  The  Commission 
services are actively involved in Environmental Impact Assessment actions (specific thematic study already 
launched, a Handbook  on Environmental Assessment  of Regional  Development plans  and  EU Structural 
Funds Programmes prepared, etc) with a view to improving the integration aspect. 

Progress has been made in operationalising environment as a  "core criterion" for  selection of projects, in 
particular  in Finland  and  Sweden.  Recently,  this  criterion has  also  been  introduced  in Italy  within  the 
industry programme, for  the selection of new industrial projects. 

Research and Technological Development 

While  research and development  facilities  are,  in  general,  already  in  place,  the  use  of 
existing  assets  needed  to  be  optimised.  Some  improvements  have  been  made  in 
particular,  in  increasing  RTD  investment  towards  the  local  business  and  industry  (see 
Box 19). 

Box  19: Research and Technological Development: the case of Ireland 

The importance of RTD was recognised  in the original CSF and its position  has been enhanced further  in 
the context of the Mid-Term Review. Some  37 Mecu was devoted to  additional  RTD  measures within the 
Industry  operational  programme.  Elements  of  competitive  bidding  and  reduced  aid  rates  will  be 
introduced. The Monitoring  Committee  also  agreed that  certain services  directed  towards  the industrial 
sector could be provided  on a cost recovery basis. 

27-

Information Society 

With regard to Information society, further resources have been directed to this priority in 
recognition of its importance in helping to reduce the effects  of peripherality, and thereby 
increasing investment and productivity (Box 20). 

Box 20: Information Society (Portugml, Spain) 

In Spmm, a new sub-axis (6.6.) on Information Society was  created within the CSF, following  a specific 
recommendation by  the  evaluator. An  initial funding  of  46  Meat  has  been  earmarked to  support this 
prierity.  ERDF measures will  be  devoted  to  the provision  ef  professional  and  interactive services for 
universities,  tourism and businesses.  ESF measures  wM focus  on specific training actions for  citizem as 
well as for  local authorities. 

A hey innovation brought about in the Portuguese CSF  has been the decision to introduce a new measure 
to support Information Society within the Telecommunications sub-programme. This measure has received 
some 7 Mecufrom ERDF.  In addition, significant efforts have also been made in other CSF interventions, 
in particular in the Local Development Programme (PDDR) where all the local development agencies will 
be  connected or  in  the  Education programme,  with  the  introduction of  Internet access  to  secondary 
schools. 

Human Resources Development 

A  refocusing  on  the  unemployed  was  deemed  necessary  with  regard  to  interventions 
targeting  human  resources  development  following  three  years  of  strong  emphasis  on 
young  people.  This emphasis  on  the unemployed,  in  line  with  the  preventive  approach 
adopted  in  the  Employment  guidelines,  took  various  forms:  strengthening  integrated 
interventions  (guidance  and  counselling,  training,  employment  aids),  boosting  micro-
enterprise  creation  subsidies,  developing  new  forms  of  training,  and  local  employment 
services. Within interventions targeting young people, support was given to moves away 
from conventional education and training programmes towards interventions in favour of 
young  people  at  risk  of  exclusion  and  early-school  leavers.  In  this  regard,  a  closer 
integration between mainstream education systems and training and employment systems 
has been supported, especially with regard to upper-secondary level  professional  training 
(box 21). 

Box 21: Human resources development -  Ireland Objective 1 

The Mid-Term Review  of  the  Human Resources  Development  Operational  Programme  (HRDOP) in 
Ireland was  approved  by  the Monitoring Committee (MC) in October  1997.  The Mid-Term Review was 
informed not only by the HRDOP,  the CSF and the ESF Programme Evaluation Unit reports but also by 
national and Community policy  orientations,  such as Government White Papers  and the EU Employment 
Strategy. 

The Mid-Term  Review package  adopted by the MC consisted of:  a broad policy framework, financial re 
allocations  with  a  revised financial plan  (there  was  approximately  40  MECU  in  changes within the 
programme,  with  a  net  addition  of  14  MECU  to  the  OP)  and  the  identification  of  qualitative 
recommendations of the OP and CSF evaluations requiring action. 

The major reallocation within the OP has been to the Early-school leavers  measure (reinforced with 26 
MECU)  aiming  to  provide  a  minimum  of  1000  additional places  (with  capacity  within the financial 
envelope to provide  1400 plus,  if necessary)y.  Accompanying measures were strengthened,  allowing for  a 
more  flexible  approach.  Provision was  also  made for  the  re-allocation  and  ring-fencing of additional 
progression  places  for  early-school-leavers  in  other  measures.  Qualitative  recommendations  were 
adopted by the MC to refocus on long-term  unemployed;  increase the responsiveness of Industry training 
to employers needs; and reinforce investment in childcare infrastructure. 

- 2 8-

Equal  opportunities 

The theme of  equal opportunities has been strengthened in many programmes, although 
this  priority  often  appears  as  a  horizontal  objective  and  not  systematically  at  measure 
level.  However,  significant  efforts  were  made  to  refocus  some  measures  towards  this 
priority,  especially  under  Human  Resources  interventions.  Under  ERDF  interventions, 
specific  measures  for  female  entrepreneurship  were  introduced,  especially  in  Italy  and 
Greece,  within  the  respective  industrial  programmes  of  those  Member  States(see 
Box 22). 

Box 22: Equal opportunities: reinforcing female  entrepreneurship 

Access for  women  to  enterprise creation schemes  has  been  encouraged  in  a  number  of programmes.  In 
Italy  and Greece, a specific measure on female  entrepreneurship (supported by national  legislation) was 
introduced  within  their  industrial  Operational  Programmes.  In  Finland,  it  is  envisaged  to  guarantee 
access to a  'soft loan ' scheme, which is currently under scrutiny. 

IV.  POLICY IMPLICATIONS AND FUTURE CHALLENGES 

The  elements  of  the  Mid-Term  Review  process  presented  earlier  in  this  report  had 
various implications  for policy purposes. They drew attention to certain  implementation 
and management  issues,  which  could  already be  addressed during the  remainder of  the 
programming period. In addition together with an indication of  future challenges arising 
from  current  evaluation  practice,  they  provided  a  basis  for  reflection  on  strategic 
priorities for future programmes. 

4.1. Short-term implications for  current delivery systems 

In  discussing  the  Mid-Term  Reviews,  the  point  was  frequently  made  that  monitoring 
procedures  tend  to  place  greater  emphasis  on  financial  execution  than  on  physical 
achievements. The majority of Mid-Term Evaluations  have pointed to the need for new 
sets  of  indicators  enabling  better measurement  of  programme  results  and impacts. The 
identification  of  such 
indicators  -  as  well  as  ensuring  reliability  and  proper 
quantification -  will  increase  the  quality  and  scope  of  the  next  round  of  Mid-Term 
Evaluations. 

Some Member states have already taken steps to improve a number of elements  in their 
current delivery systems, inter alia: 

- 

- 

fixing  clear  quantified  targets  and  indicators  for  monitoring  and  evaluation  (e.g. 
Austria, Portugal, Italy, United Kingdom); 

setting  more  appropriate  selection  criteria  for  projects  (including  employment  and 
environmental criteria (e.g. Italy); 

simplifying  management  procedures,  in  particular  for  small  programmes  (e.g. 
Finland, Sweden); 

- 

securing  synergies  between  different  Structural  Fund  interventions  (e.g.  Portugal, 
Spain). 

29 

4.2. Initial reflections for future  programmes 

The  Mid-Term  Review  was  concerned  with  making  necessary  changes 
to  the 
programmes  to  reflect  changing  circumstances  and  needs  arising  in  the  course  of  the 
implementation  phase. In parallel  with  the Mid-Term  Evaluations, both  the  Commission 
and the Member States have conducted  a number of strategic  analyses  which  look  ahead 
to the period beyond the current  interventions. These were designed to prepare for  future 
Structural  Funds  interventions  by  considering  longer  term  issues  which  could  not  be 
addressed  during  the  remainder  of  the  existing  programmes.  For  example,  the 
Commission  has  supplemented  the  limited  information  basis  offered  by  the  interim 
evaluation  in Italy with  a set of thematic  studies relating to key CSF priorities: industry-
related  aid  schemes,  RTD,  environment,  transport  infrastructure  and  human  resources 
development. 

These  studies  set  out  to  address  certain  key  policy  issues  and  also  to  provide 
recommendations  for  amending  future  CSFs. Whilst  necessarily  broad  in  outlook,  these 
exercises  have  nonetheless  usefully  informed  the  Mid-Term  Review  process  on  issues 
such as the future  re-orientation  of spending priorities. 

The process of re-examining  funding  priorities  has already commenced  in the context of 
the Mid-Term  Review  in  some  Member  States. Informal  discussions  and  seminars  have 
also taken  place with  national  authorities  to address  some  key issues  for  the future.  The 
Commission  has  launched  four  thematic  studies  (RTD,  SMEs,  environment  and  the 
partnership  principle)  to  help  identify  the  broad  directions  of  policy  and  facilitate  the 
introduction of new arrangements for the period after  1999. 

4.3. The challenge of evaluation 

This  report  has  shown  that  notable  progress  has  been  achieved  in  evaluation  practice 
throughout the various Objectives  1 and 6 programmes. Important  good practice  features 
of the Mid-Term Review have been highlighted throughout the report including: 

-  The soundness of evaluation (methodology and quality criteria); 

-  The involvement of programme partnerships in the evaluation process; 

-  The organisation and timing of the Mid-Term Review process; 

-  The  feedback  role  of  evaluation  in  supporting  reprogramming  decisions  as  well  as 

anticipating key issues. 

Lessons  arising from  this experience  may represent  a basis for  guidance  on best  practice 
for  the next  programmes  in  order  to improve  further  the  management  of  the funds.  The 
crucial  issue  is,  thus,  how  to  take  stock  of  the  results  to  promote  a  more  systematic 
dissemination  of  good  practice  both  within  and  between  regions.  In  the  light  of 
experience  to date, the Mid-Term  Review  shows that  there is  still  a need  for  continuing 
improvements  to the quality  of  the  process  in  view  of  the  challenges  to  be  faced  in  the 
context of future  programmes. 

Consolidating evaluation as a tool for  decision making 

Evaluation  is increasingly  seen  as a management  tool, which  should  assist  policy  makers 
and programme  managers  in  their tasks by providing  them  with  important  insights  as to 

- 3 0-

the  effectiveness  of  measures  and  programmes.  In  its  organisation,  this  function  should 
link more closely the production  of reliable and operational  results  to their integration  in 
the  decision  making  process.  This  would  imply  the  creation,  where  unavailable,  of 
adequate structures for managing the evaluation activities in the context of partnership. 

Improving monitoring systems and indicators 

Some  significant  actions have already  been  taken  by national  and regional  authorities  to 
address  some gaps or deficiencies  in  their  monitoring  systems  over the remainder  of the 
programming  period.  Improvements  in  the  definition  of  indicators  and  data  collection 
will  be  necessary  to  enable  these  to  perform  their  management  tasks  more  effectively. 
Better integration  between  financial  and physical  indicators  is a further,  notable  concern 
for programme managers. 

For future  programmes, there is a need to ensure  adequate quantification  of baseline  and 
target  indicators  and provide benchmark  data  to allow  for  better comparison  within  and 
between  programmes.  In  this  respect,  the  Commission  will  provide,  in  due  course,  a 
guidance  document  on  methodological  issues,  including  an  indicative  list  of  relevant 
indicators for the main areas of Structural Funds interventions. 

Supporting evaluation methodologies 

Substantial  progress  has  been  achieved  in  the  area  of  evaluation  methodology.  The 
MEANS  Programme,  initiated  by  the  Commission  in  1994,  has  contributed  to  the 
development  of  that  organisation's  expertise,  drawing  on  first  hand  experience  of 
Structural Funds. There is no single and standardised evaluation  methodology but rather a 
wide  range  of  methods  which  should  be  customised  so  that  they  match  the  particular 
needs and situation of each programme. Efforts  will be made, however, to ensure a better 
knowledge  of  evaluation  techniques,  not  only  among  evaluators  but  also  programme 
managers, who will have to appraise the quality of evaluation results. 

Promoting multilateral exchange of experience 

Since evaluation  has been recognised as a major  instrument for decision making, national 
and  regional  authorities  are  generally  keen  to  have  informal  meetings  to  exchange 
experience and opinions on this matter. 

The  Technical  Evaluation  Group  at  EU  level  is  made  up  of  representatives  of  Member 
States  and  Commission  officials  involved  in  the  management  of  Structural  Funds  and 
might  be  considered  a  suitable  forum  in  which  to  discuss  evaluation  issues  and 
disseminate  best  practice.  The  main  issues  of  common  interest  include  the  type  and 
relevance  of  evaluation  methodologies  (e.g.  measuring  employment  effects)  and  the 
deployment of evaluation results for policy purposes. 

Overall  effectiveness  of  Structural  Funds  interventions  will  also  be  influenced  by  the 
extent  to  which  national  and  regional  authorities  take  necessary  steps  to  improve  their 
monitoring  systems and are able to adopt good practice features  to conduct  the next Mid-
Term Review process. 

-31 

Annex I: List of boxes 

Box 1 

Box 2 

Box 3 

Box 4 

Box 5 

Box 6 

Box 7 

Box 8 

Box 9 

Assessing the practicability of evaluation  - ERDF Andalusia OP (Obj.l) 

Managing evaluations -  the creation of ad-hoc structures 

Assessing the quality of an evaluation: the Irish CSF evaluation 

Hainaut (Obj. 1 ) : the HELM model 

Assessing  effectiveness  through  programme  indicators-the  case  of  Merseyside-
Objective  1  (4994-99) -actual outputs/results for ERDF interventions 

Important achievements in basic  infrastructure. 

Some key achievements in the productive environment. 

Impact of training measures on  beneficiaries 

Evaluation  of  agricultural  interventions  the  case  of  PAMAF  (modernisation  of 
agricultural  structures) 

Box  10: An example: the Spanish Objective  1  programme 

Box  11 : Enhancing the monitoring systems 

Box  12: Merseyside -  scoring systems for ESF / ERDF projects 

Box  13: Mid-Term Review (I): the case of the Irish CSF 

Box  14: Mid-Term Review (II) : the case of Merseyside  (Obj.l) 

Box  15: Airport programme -  Italy Objective 1 

Box  16: Territorial Employment Pacts 

Box  17: SMEs : the case of the Greek CSF 

Box  18: Mainstreaming environment 

Box  19: Research and Technological Development: the case of Ireland 

Box 20: Information  Society (Portugal, Spain) 

Box 21:  Human resources development -  Ireland (Objective  1) 

Box 22: Equal opportunities: reinforcing female  entrepreneurship 

32-

Annex  2 : Financial  implementation  of Objective I  programmes 

Commitments/ total SF assistance  under 
Objective 1 

Payments/ total SF assistance  under 
Objective 1 

1994 

1995 

1996 

1997 

1998 (1st 

1994 

1995 

1996 

1997 

I annual 

• cumulated 

I 

semester) 

I annual 

•cumulated 

! 

1998 (1st 
semester) 

i._..: 

Financial implementation of Objective 1 programmes by Member  States (31.12.97) 

% 

Financial  implementation  of Objective  1 Programmes bv Member-State (31/12/97) 

75 

70 

65  ] 

60 

55 

50 

45 

40 

35 

30 

25 

20 

15 

10 

5 

0 

69 

61 

61 

63 

59 

47 

44 

70 

62 

64 

51 

Belgium 

Germany 

Greece 

Spain 

France 

Ireland 

Itaiy 

Netherlands 

Austria 

Portugal 

United- 
Kingdom 

Total 

IE Commitments/total SF assistance  1994-97 £3 Payments/ Total SF assistance  1994-97 

(Austria:  SPD  approuvée! in  1995) 

33 

Financial implementation of Objective 6 programmes Finland 

Commitments/total SF assistance under 
Objective 6 Finland 

57 

57 

Payments/ total SFassistance  under 
Objective 6 Finland 

1994 

1995 

1996 

1997 

1998(1st 

I annual 

-^--curnjteted 

L; 

1997 

1998 (1st 
semester) 

Financial implementation of Objective 6 programmes Sweden 

Commitments/ total SFassistance under 
Objective 6 Sweden 

Payments/total SFassistance  under 
Objective 6 Sweden 

1994 
• annual 

1995 

1996 

1997 

—•— currulated  ~] 

1998 (1st 
semester) 

1996 

1997 

I annual 

•cumulated 

1998 (1st 
semester) 

34 

Annex 3: List of reports 

Objective 1  reports 

Austria 

Belgium 

Germany 

Germany 

Germany 

Germany 

Germany 

Germany 

Germany 

Spain 

Spain 

Spain 

Spain 

Spain 

Spain 

Spain 

BURGENLAND : ZWISCHENEVALUATION  DES 
ZIEL1-PROGRAMMS BURGENLAND:  Endbericht 
HAINAUT : EVALUATION  INTERMEDIATE DU 
DOCUP OBJECTIF  1  HAINAUT - RAPPORT FINAL -
REGION  WALLONNE 
BERLIN: ZWISCHENBEWERTUNG  DER EU-
STRUKTURFONDSINTERVENTIONEN  IM LAND 
BERLIN IM ZEITRAUM  1994 BIS  1996 
BRANDENBURG  : ZWISCHENBEWERTUNG  ZUM 
EINSATE DES EFRE, DES ESF UND DES EAGFL-A IM 
LAND BRANDENBURG  1994-1996 
MECKLENBURG-VORPOMMERN : 
ZWISCHENBEWERTUNG  DES EINSATZES DER 
EUROPAISCHEN  STRUKTURFONDS  (EAGFL, EFRE, 
ESF) FUR DIE PERIODE  1994 BIS  1996 IM 
BUNDESLAND MECKLENBURG-VORPOMMERN 
SACHSEN  : ZWISCHENEVALUIERUNG  DES 
EINSATZES DER EUROPAISCHEN  STRUKTURFONDS 
IM FREISTAAT SACHSEN  1994-1996 
SACHSEN-ANHALT: ZWISCHENBEWERTUNG  DES 
EINSATZES DER EU-STRUKTURFONDS  IN 
SACHSEN-ANHALT  FUR DEN ZEITRAUM  1994 BIS 
1996 
THURINGEN  : DIE EUROPAISCHEN 
STRUKTURFONDS  IN THURINGEN : 
ZWISCHENBEWERTUNG  DES 
MITTELEINSATZESVON  1994 BIS  1996 
NEUE BUNDESLANDER  : DIE EUROPAISCHEN 
STRUKTURFONDS  IN DEN NEUEN 
BUNDESLANDERN  - ZWISCHENBEWERTUNG  DES 
MITTELEINSATZES  VON  1994 BIS  1996 (BERLIN/ 
BRANDENBURG/MECKLENBURG-
VORPOMMERN/SACHSEN/SACHSEN-
ANHALT/THURINGEN) 
PO CANARIES : EVALUACION  INTERMEDIA 
PROGRAMA OPERATIVO DE CANARIAS  (FEDER) 
1994-1999 
OP INCENTIVOS REGIONALES  : EVALUACION 
INTERMEDIA  DEL PROGRAMA OPERATIVO DE 
INCENTIVOS REGIONALES  FEDER (1994-1999) PARA 
LAS REGIONES ESPANOLAS  INCLUIDAS EN EL 
OBJETIVO N°l  DELOS FONDOS  ESTRUCTURALES 
EUROPEOS 
CANTABRIA  : ESTUDIO DE EVALUACION 
INTERMEDIA  DEL PROGRAMA OPERATIVO DE 
CANTABRIA (FEDER)  1994-1999 
CASTILLA Y LEON  : EVALUACION  DE LOS 
PROGRAMAS OPERATIVOS DEL FEDER Y DEL FSE 
1994-1999 DE CASTILLO Y LEON 
EVALUACION  INTERMEDIA  DE LA SUBVENCION 
GLOBAL FEDER-CDTI EN LAS REGIONES 
OBJETIVO  1 (1994-1999) 
ESTUDIO DE EVALUACION  INTERMEDIA  DEL 
PROGRAMA OPERATIVO DE  INFRAESTRUCTURA 
CIENTIFICA  DEI PERIODO  1994-1999, EN REGIONES 
ESPANOLAS INCLUIDAS EN EL OBJ. 1, 
COFINANCIADO CON FEDER  Y COORDINADO POR 
CICYT 
EVALUACI6N INTERMEDIA  DEL MARCO 
COMUNITARIO  DE APOYO-REGIONES OBJETIVO 1-
1994-1999-ESPANA 

35-

Apr-98 

OIR 

Apr-97 

SEMA GROUP/RIDER-
UCL 

May-97 

PROGNOS  BERLIN 

May-97 

TROJE  BERENTUNG 
HANN, MUNCHEN 

Apr-97 

BUSTRO GmbH 
ROSTOCK 

May-97 

DIW BERLIN 

May-97 

ISW HALLEFLEIPZIG 

May-97 

IFO INSTITUT FUR 
WIRTSCHAFTSFOR-
SCHUNG  DRESDEN 

Sep^97 

IFO INSTITUT FOR 
WIRTSCHAFTSFOR-
SCHUNG  DRESDEN 

Jun-97 

SERVICIOS  OMICRON, 
S.A. 

May-97 

SERVICIOS  OMICRON 
SA 

Jun-97 

INFYDE 

Jan-97 

(UNIV.VALLADOLID) 

Jun-97 

INFYDE/CDTI 

Jul-97 

PRICE WATERHOUSE 

Sep-97 

QUASAR 

Spain 

Spain 

Spain 

Spain 

Spain 

Spain 

Spain 

Spain 

Spain 

Spain 

Spain 

France 

France 

France 

France 

Jul-97 

(UNIV.VALLADOLID) 

Jun-97 

GEDESA 

May-97 

AGRO CONSULTING 
INTERNATIONAL 

May-97 

Jul-96 

ARANDA  Y BELTRAN 
S.L.  ESTUDIOS Y 
ANALISIS SOCIO 
ECONOMICOS 
UNIVERSIDAD DE 
OVIEDO 

Dec-97 

ANDERSON ARTHUR 

Aug-97 

ECOTEC 

Jul-97 

Mar-98 

UNIVERSIDAD DE 
CASTILLA-LA 
MANCHA 
INSTITUTO DE 
DESARROLLO 
REGIONAL, 
FUNDACI6N 
UNIVERSITARIA 
ARENAL GRUPO 
CONSULTOR SL 

INSTITUTO DE 
DESARROLLO 
REGIONAL, 
FUNDACION 
UNIVERSITARIA 
FERE CONSULTANTS 

Jun-97 

Feb-98 

ERNST & YOUNG 

Dec-97 

ACT 

Feb-96 

CODE/ATHOS 

CASTILLA Y LEON :  EVALUACION DE LOS 
PROGRAMAS OPERATIVOS DEL FEDER Y DEL FSE 
1994-1999 DE CASTILLO Y LEON 
CEUTA : 2e ESTUDIO DE EVALUACION INTERMEDIA 
DEL MARCO DE APOYO COMUNITARIO DE LAS 
REGIONES OBJETIVO N°l  PARA CEUTA (PO CEUTA 
1994-1999) 
GALICIA: EVALUACION  INTERMEDIA DEL 
PROGRAMA OPERATIVO DE GALICIA  1994-1999 
cofinanciado per el FEDER (PO GALICIA) 
PO MELILLA : EVALUACION INTERMEDIA DEL 
PROGRAMA OPERATIVO FEDER DE MELILLA  1994-
1999 

ASTURIAS  : EVALUACION  DEL PROGRAMA 
OPERATIVO DEL PRINCIPADO DE ASTURIAS 
(FEDER) 1994-1999 - INFORME PREPARATORIO 
OP LOCAL: ESTUDIO DE EVALUACION 
INTERMEDIA DEL PROGRAMA OPERATIVO LOCAL 
(FEDER)  1994-1999 PARA LAS REGIONES 
ESPANOLAS INCLUIDAS EN EL OBJETIVO N°  1  DE 
LOS FONDOS ESTRUCTURALES EUROPEOS 
OP MEDIO AMBIENTE: EVALUACION INTERMEDIA 
DEL PROGRAMA OPERATIVO DEL MEDIO 
AMBIENTE LOCAL (FEDER) 1994-1999 EN REGIONES 
ESPANOLAS INCLUIDAS EN EL OBJETIVO NO 1  DE 
LOS FONDOS ESTRUCTURALES EUROPEOS 
OP CASTILLA-LA-MANCHA:SEGUNDO INFORME DE 
EVALUACION-PROGRAMA OPERATIVO FEDER DE 
CASTILLA-LA MANCHA  1994-1999 
ANDALUCf A : EVALUACION INTERMEDIA DEL P.O. 
ANDALUCIA  1994-1999 

ANDALUCIA  :  SUBVENCI6N GLOBAL DE 
ANDALUCfA  1994-1999  : INFORME DE EVALUACION 
INTERMEDIA 
DONANA : EVALUACI6N INTERMEDIA DEL P.O. 
DONANA Ha PHASE (OBJ.l) 

Mar-98 

May-97 

CORSE : EVALUATION INTERMEDIAIRE DOCUP 
OBJECTIF 1  CORSE 1994-1999 
GUYANE : EVALUATION INTERMEDIAIRE DE LA 
MISE EN OEUVRE EN GUYANE DES 
PROGRAMMATIONS REGIONALES (1994-1999) 
COFINANCEES PAR L'UNION EUROPEENNE (DOCUP 
ET REGIS II) - SYNTHESE DU RAPPORT 
NORD-PAS-DE-CALAIS (AVESNES / DOUAI / 
VALENCIENNES): PROGRAMME OBJECTIF 1  (1994-
1999): EVALUATION INTERMEDIAIRE, 
EVALUATION DENSEMBLE DU PROGRAMME, 
ASPECTS SECTORIELS EVALUATION DES MESURES 
GUADELOUPE.EVALUABILITE DU DOCUP 
GAUDELOUPE : RAPPORT D'ETUDE (DOCUP  1994-
1999) 

36 

Greece 

Greece 

Greece 

Greece 

Greece 

Greece 

Greece 

Greece 

Greece 

Greece 

Greece 

Greece 
Ireland 

Ireland 
Ireland 

Ireland 

Ireland 

Ireland 

Ireland 

Ireland 

Ireland 

Ireland 

Ireland 

Italy 

Netherlands 

Portugal 

Portugal 

PELOPONNESE  : INTERMEDIATE  EVALUATION 
OBJECTIVE  1  - OPERATIONAL PROGRAMME  1994-
1999 (a) VOL. A (b) VOL. B 
GRECE OCCIDENTALE (GRECE OUEST): 
INTERMEDIATE EVALUATION  OBJECTIVE  1  -
OPERATIONAL  PROGRAMME 
CDS P.O. EPIRIUS : INTERMEDIATE EVALUATION 
1994-1999 OBJECTIVE  1  -  OP 
CRETE : INTERMEDIATE EVALUATION OBJECTIVE 1 
- OPERATIONAL PROGRAMME  1994-1999 
THESSALIE: INTERMEDIATE  EVALUATION 
OBJECTIVE  1  -O.P.  1994-1999 
ILES IONIENNES  : INTERMEDIATE  EVALUATION 
OBJECTIVE  1  -  O.P.  1994-1999 
MACEDOINE CENTRALE + THRACE : 
INTERMEDIATE EVALUATION OBJ.  1  - O . P.  1994-99 
MACEDOINE OCCIDENTALE:  INTERMEDIATE 
EVALUATION OBJECTIVE  1  O.P.  1994-1999 
MACfEDOINE CENTRALE:  INTERMEDIATE 
EVALUATION  OBJECTIVE  1  -O.P.  1994-1999 
PO GRECE CONTINENTALE (=CENTR.): 
INTERMEDIATE EVALUATION O.P. OBJ.  1  -  1994-1999 
EGEE DU SUD : INTERMEDIATE  EVALUATION 
OBJECTIVE  1 ILES DE LA MER EGEE SUD -
OPERATIONAL PROGRAMME  1994-1999 
CSFGRECE : OBJ.l  -  1994-1999 
EXTERNAL EVALUATION OF THE OPERATIONAL 
PROGRAMME FOR INDUSTRIAL  DEVELOPMENT 
1994-1999  : MID-TERM  REVIEW 
CSF MID-TERM  EVALUATION 
MID TERM EVALUATION OF OPERATIONAL 
PROGRAMME FOR ECONOMIC  INFRASTRUCTURE 
MID-TERM EVALUATION  OF THE LOCAL, URBAN 
AND RURAL DEVELOPMENT O.P. 
MID-TERM  EVALUATION  OF THE HUMAN 
RESOURCES DEVELOPMENT-O.P.  1994-1999 
MID TERM EVALUATION OF THE O.P. FOR 
ENVIRONMENTAL SERVICES  1994-1999 
OPERATONAL PROGRAMME FOR TRANSPORT - MID 
TERM EVALUATION 
MID-TERM EVALUATION OF THE OPERATIONAL 
PROGRAMME FOR TOURISM  1994-1999 
OPERATIONAL PROGRAMME FOR AGRICULTURE, 
RURAL DEVELOPMENT AND FORESTRY  1994-1999 
MID-TERM  EVALUATION OF OPERATIONAL 
PROGRAMME FOR FISHERIES  1994-1999 
MID-TERM EVALUATION  : REGIONAL IMPACT OF 
THE COMMUNITY SUPPORT FRAMEWORK  FOR 
IRELAND  1994-1999 
VALUTAZIONE DI MEDIO TERMINE QUADRO 
COMUNITARIO DI SOSTEGNO  1994-99 delle REGIONI 
ITALIANE DELL'OBIETTIVO 1 
FLEVOLAND: ON-GOING EVALUATIE 
DOELSTELLING  1 PROGRAMMA 
AMBIENTE:AVALIACAO INTERCALAR DO 
PROGRAMA  AMBIENTE -  EVALUATION  DU SOUS-
PROGRAMME ENVIRONNEMENT  DU CCA II PT 
ALENTEJO:AVALIACAO INTERCALAR DO 
PROGRAMA OPERACIONAL DA REGIAO DO 
ALENTEJO - CCA  1994-1999 

May-97 

OMAS LTD 

Jun-97 

EEO GROUP SA 

Apr-97 

May-97 

May-97 

PROPTIKI  SA. -
METPON 
ASTIKI DIACHIRISI 
S.A. 
YPODOMI LTD 

May-97 

PROOVADISMA  LTD 

Apr-97 

INDECO S.A. 

May-97 

EXANTAS LRD 

May-97 

EUROTEC LTD 

May-97 

LDK 

May-97 

ENVIPLAN 

Mar-98 
Feb-97 

REMACO 
ERNST & YOUNG 

Apr-97 
Feb-97 

ESRI 
INDECON 

Jan-97 

GOODBODY 

Feb-97 

GOODBODY 

Jan-97 

ERM 

Feb-97 

Feb-97 

Jan-97 

Feb-97 

DKM  ECONOMIC 
CONSULTANTS 
FITZPATRICK 
ASSOCIATES 
FITZPATRICK 
ASSOCIATES 
INDECON 

Feb-97 

FITZPATRICK 
ASSOCIATES 

Jul-97 

Jan-97 

Feb-97 

NUCLEO DI 
VALUTAZIONE/INEA E 
ISFOL 
RESEARCH  VOOR 
BELEID 
QUATERNAIRE 
(PINHO PAULO) 

Feb-97 

CESO I&D 

- 3 7-

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

Portugal 

AMBIENTE:ESTUDO DE AVALIACAO INTERCALAR 
DA INTERVENCAO OPERACIONAL-RENOVACAO 
URBANA DO PROGRAMA'AMBIENTE E 
REVITALIZACAO URBANA DO QUADRO 
COMUNITARIO DE APOIO 1994-1999 

SOCIAL:SUB-PROGRAMA INTEGRAR AVALIACAO 
INTERCALAR 
LISBOA E VALE DO TEJO:ESTUDO DE AVALIACAO 
INTERCALAR DO PROGRAMA OPERACIONAL DA 
REGIAO DE LISBOA E VALE DO TEJO 
PPDR:ESTUDO DE AVALIACAO INTERCALAR 
PROGRAMA OPERACIONAL PROMOCAO DO 
POTENCIAL DE DESENVOLVIMENTO REGIONAL -
QUADRO COMUNITARIO DE APOIO 1994-1999 
SAUDE:AVALIACAO INTERCALAR DA 
INTER VENCAO OPERACIONAL DA SAUDE 
ENERGIA:AVALIACAO INTERCALAR DO 
PROGRAMA ENERGIA E DO PROJECTO DE GAS 
NATURAL (INTERREG II - CONCLUSAO DAS REDES 
DE ENERGIA) DO QCA II (1994-1999) 
PO CENTRO:ESTUDO DE AVALIACAO INTERCALAR 
DO P. 0.  REGIAO CENTRO 
PO NORTE:ESTUDO DE AVALIACAO INTERCALAR 
DO PRONORTE 
PEDIP II: AVALIACAO INTERCALAR  DO PEDIP II: 
RELATORIO FINAL INTERCALAR 

PESCA : AVALIACAO INTERCALAR DA 
INTER VENCAO OPERACIONAL DAS PESCAS 
TURISMO: ESTUDO DE AVALIACAO INTERCALAR 
DA INTER VENCAO OPERACIONAL TURISMO E 
PATRIMONII CULTURAL DO PROGRAMA 
MODERNIZACAO DO TECIDO ECONOMICO 
AGRICULTURA : QCA II: ESTUDO DE AVALIACAO 
INTERCALAR DA INTERVENCAO OPERACIONAL 
AGRICULTURA : PROGRAMA MODERNIZACAO DO 
TECIDO ECONOMICO - PERIODO  1994-1996 
PROFAP : PROGRAMA INTEGRADO DE FORMACAO 
PARA A MODERNIZACAO DA ADMINISTRACAO 
PUBLICA - PROFAP 2 : ESTUDO DE AVALIACAO 
INTERCALAR 
ALGARVE :  ESTUDO DE AVALIACAO INTERCALAR 
DO PROGRAMA OPERACIONAL DO ALGARVE 
TELECOMUNICACOES  : ESTUDO DE AVALIACAO 
DA INTERVENCAO OPERACIONAL DAS 
TELECOMUNICACOES DO PROGRAMA INFRA-
ESTRUTURAS DE APOIO AO DESENVOVIMENTO DO 
QCA 
SUB-PROGRAMME TRANSPORTES: AVALIACAO 
INTERCALAR DA INTERVENCAO OPERACIONAL 
DOS TRANSPORTES DO PROGRAMA INFRA-
ESTRUTURAS DE APOIA AO DESENVOLVIMENTO 
DO QUADRO COMUNITARI DE APOIO 1994-1999, NO 
PERIODO ENTRE 1994 E 1996 
SUB-PROGRAMME COMERCIO E SERVICOS 
:ESTUDO DE AVALIACAO INTERCALAR DA 
INTERVENCAO OPERACIONAL COMERCIO E 
SERVICOS 
ACORES=ESTUDO DE AVALIACAO INTERCALAR DO 
PROGRAMA OPERACIONAL DA REGIAO 
AUTONOMA DOS ACORES DO QUADRO 
COMUNITARIO DE APOIO  1994-1999 

38-

Mar-97 

Jan-97 

PARTEX-INSTITUTO 
DE GESTAO E 
ALIENACAO DO 
PATRIMONIO 
HABITACIONAL DO 
ESTADO 
CIES 

Jan-97 

CEDRU 

Jan-97 

CEETA 

Feb-97 

CESO I&D 

Jan-97 

IESE 

Jan-97 

CEDRU 

Jan-97 

QUATERNAIRE 

Mar-97 

Apr-97 

UNIVERSIDADE 
CATOLICA 
PORTUGUESA 
CESO I&D 

Feb-97 

DELOITTE & TOUCHE 

Jun-97 

AGRO.GES 

Mar-97 

QUATERNAIRE 

Jan-97 

CEDRU 

Mar-97 

SILICON 

Apr-97 

CISED 

Nov-97 

CESO I&D 

Feb-97 

CESO I&D 

Portugal 

Portugal 

Portugal 

Portugal 

United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
(IRN) 
United 
Kingdom 
United 
Kingdom 

SUB-PROGRAMME  EDUCACAO:AVALIACAO 
INTERCALAR  DE PRODEP II:  EDUCACAO 
CIENCIA  E TECNOLOGIA:ESTUDO DE AVALIACAO 
INTERCALAR  DO SUB-PROGRAMA CIENCIA E 
TECNOLOGIA 
OP PESSOA: ESTUDO DE AVALIACAO  INTERCALAR 
PROGRAMA OPERACIONAL EMPREGO E 
FORMACAO PROFISSIONAL DO QCA II  1994-1999 
(PROGRAMMA  PESSOA) 
CSF PORTUGAL-OBJ.l:AVALIACAO INTERCALAR 
DO QUADRO COMUNITARIO DE APOIO II 
NORTHERN  IRELAND SINGLE PROGRAMME 1994-
1999 - MID TERM  REVIEW - EXTERNAL 
EVALUATION 
MID-TERM  REVIEW OF THE ENVIRONMENTAL 
SERVICES AND PROTECTION  SUB-PROGRAMME 

MEASURES 4.1.8 AND 4.19. OF THE SUB-
PROGRAMME FOR AGRICULTURE AND RURAL 
DEVELOPMENT  : MID TERM EVALUATION 
MID-TERM EVALUATION OF THE NORTHERN 
IRELAND SINGLE PROGRAMME FOR AGRICULTURE 
AND RURAL DEVELOPMENT (SPARD) 
ENERGY SUB-PROGRAMME  1994-1999 : MID-TERM 
REVIEW 

MID TERM EVALUATION OF THE 
TRANSPORTATION  SUB-PROGRAMME,  1994-1999 

MID-TERM EVALUATION OF THE INVESTMENT IN 
THE DEVELOPMENT OF PEOPLE SUB-PROGRAMME, 
NORTHERN  IRELAND SINGLE PROGRAMME  1995-99 
MID TERM REVIEW OF PHYSICAL & SOCIAL 
ENVIRONMENT PROGRAMME (EU S. F.) 

Scp-97 

QUATERNAIRE 

Feb-98 

IESE/GEOIDEIA 

Mar-97 

IESE (INSTITUTO DE 
ESTUDOS SOCIAIS E 
ECONOMICOS 

jun-98 

CISEP 

May-97 

COLIN STUTT 
CONSULTING 

Dec-96 

PIEDA CONSULTANTS 

Feb-97 

COOPERS  & LYBRAND 

Mar-97 

ERM 

Jan-97 

ERM 

Jan-97 

ERM 

Dec-96 

LRDP 

Jan-97 

COOPERS & LYBRAND 

MID-TERM  EVALUATION OF THE TOURISM 
SUBPROGRAMME  1994-1999 

Fcb-97 

NIERC 

MERSEYSIDE. MID-TERM EVALUATION OF 
MERSEYSIDE OBJECTIVE ONE PROGRAMME 
HIGHLANDS AND ISLANDS -  OBJECTIVE 1 
PROGRAMME INTERMEDIATE  ASSESSMENT 

Nov-96 

PIEDA PLC 

Jun-97 

SQW 

Objective 6 report 

Finland 

SUOMEN TAVOITE 6 -  OHJELMAN  ARVIOINNIN 
VALIRAPORTTI [FI] 

Jan-98 

Sweden 

HALVTIDSUTVARDERING  AV SVERIGES NAL-6-
PROGRAM 

Nov-97 

HELSINGIN 
YLIOPISTON 
TIETOPALVELUT 
OY/ETLATIETO OY/OY 
FINNAGRO 
AB/INT.DEVEL.IRL/PE 
LLERVON 
NORD-REGIO-EPRC 

-39 

ISSN 0254-1475 

COM(98) 782 final 

DOCUMENTS 

EN 

04  13  09  16 

Catalogue number  :  CB-CO-98-786-EN-C 

Office  for Official  Publications of the European Communities 

L-2985 Luxembourg 

ÙO